Rigid-body engines leave the calling thread on its own card; the MR search runs on its transform's card

Every RigidBodyGPUEngine call did cudaSetDevice(engine's card) and never set the caller back. Molecular
replacement's group threads refine copies on the pool's engines - on any card where every engine on this one
is taken - and then run the next translation function on the same thread. MolecularReplacementGPU ran its
calls on whatever card the thread was on, so that function's buffers and cuFFT plan were made on the
engine's card, beside the transform and the stream on the search's card: an illegal memory access, 9s02 with
four cards visible, every run (the further-copies loop: refine a trial on another card's engine, then
translation_function on the group thread).

- Each engine call (and the constructor) now holds a DeviceGuard: on the engine's card for the call, back on
  the caller's after it.
- MolecularReplacementGPU remembers the card its transform was uploaded to and runs every call (and its
  destructor) there, whichever card the calling thread is on.
- RigidBodyGPU_CallsLeaveTheCallerOnItsCard: one engine per card, all taken from one thread; the thread's card
  is unchanged after each call (meaningful with two or more cards).

No number changes: where every call runs is the same card as before for each object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-17 09:24:16 +02:00
1 parent 5d8c6f4504
commit a3f30d03c1
3 files changed
+72 -13

No files matched your search

+30
View File
@@ -1655,6 +1655,36 @@ TEST_CASE("RigidBodyGPU_Deterministic", "[ModelValidation][gpu]") {
}
}
// An engine's calls run on the engine's card and leave the calling thread on its own: every engine of a pool with
// one on each card, all taken at once from one thread, evaluated in turn. A thread left on the last engine's card
// would run what it does next - a search's function, memory it allocates - on that card, beside data on another.
TEST_CASE("RigidBodyGPU_CallsLeaveTheCallerOnItsCard", "[ModelValidation][gpu]") {
if (get_gpu_count() == 0)
SKIP("No GPU");
Logger logger("RigidBodyGPU_CallsLeaveTheCallerOnItsCard");
const char *cryst = kRigidBodyCrysts[4];
const gemmi::Structure st = AnisoCluster(cryst);
const gemmi::SpaceGroup *sg = st.find_spacegroup();
const auto fobs = OwnAmplitudes(cryst, st, 3.5, logger);
const size_t cards = get_gpu_count();
std::vector<gemmi::Model> models(cards, st.models[0]);
auto pool = TestPool(models[0], st.cell, *sg, 3.5, fobs.v.size(), cards, logger);
const int caller = RigidBodyGPUEngine::CurrentDevice();
std::vector<std::unique_ptr<RigidBodyTargetGPU>> targets;
for (size_t k = 0; k < cards; k++)
targets.push_back(std::make_unique<RigidBodyTargetGPU>(*pool, models[k], st.cell, *sg, 4));
CHECK(RigidBodyGPUEngine::CurrentDevice() == caller);
const double q0[6] = {0.12, -0.09, 0.07, 0.15, -0.10, 0.05};
for (auto &t : targets) {
t->SetZone(fobs, 3.5);
CHECK(RigidBodyGPUEngine::CurrentDevice() == caller);
std::vector<double> r(t->NumObservations()), j(6 * t->NumObservations());
REQUIRE(t->Residuals(q0, r.data()));
REQUIRE(t->Jacobian(q0, j.data()));
CHECK(RigidBodyGPUEngine::CurrentDevice() == caller);
}
}
// The model's structure factors on the GPU against the CPU path model validation takes - gemmi's density and
// mask on the symmetrized grid, transformed by FFTW and read off by prepare_asu_data() - in the five groups,
// at a resolution where the mask's shrink step does nothing and at one where it does. The same reflections