Rigid-body engines leave the calling thread on its own card; the MR search runs on its transform's card
Every RigidBodyGPUEngine call did cudaSetDevice(engine's card) and never set the caller back. Molecular replacement's group threads refine copies on the pool's engines - on any card where every engine on this one is taken - and then run the next translation function on the same thread. MolecularReplacementGPU ran its calls on whatever card the thread was on, so that function's buffers and cuFFT plan were made on the engine's card, beside the transform and the stream on the search's card: an illegal memory access, 9s02 with four cards visible, every run (the further-copies loop: refine a trial on another card's engine, then translation_function on the group thread). - Each engine call (and the constructor) now holds a DeviceGuard: on the engine's card for the call, back on the caller's after it. - MolecularReplacementGPU remembers the card its transform was uploaded to and runs every call (and its destructor) there, whichever card the calling thread is on. - RigidBodyGPU_CallsLeaveTheCallerOnItsCard: one engine per card, all taken from one thread; the thread's card is unchanged after each call (meaningful with two or more cards). No number changes: where every call runs is the same card as before for each object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
This commit is contained in:
1 parent
0f461a65a2
commit
fbdf036e6f
3 files changed
+72
-13
No files matched your search
@@ -1692,6 +1692,36 @@ TEST_CASE("RigidBodyGPU_Deterministic", "[ModelValidation][gpu]") {
|
||||
}
|
||||
}
|
||||
|
||||
// An engine's calls run on the engine's card and leave the calling thread on its own: every engine of a pool with
|
||||
// one on each card, all taken at once from one thread, evaluated in turn. A thread left on the last engine's card
|
||||
// would run what it does next - a search's function, memory it allocates - on that card, beside data on another.
|
||||
TEST_CASE("RigidBodyGPU_CallsLeaveTheCallerOnItsCard", "[ModelValidation][gpu]") {
|
||||
if (get_gpu_count() == 0)
|
||||
SKIP("No GPU");
|
||||
Logger logger("RigidBodyGPU_CallsLeaveTheCallerOnItsCard");
|
||||
const char *cryst = kRigidBodyCrysts[4];
|
||||
const gemmi::Structure st = AnisoCluster(cryst);
|
||||
const gemmi::SpaceGroup *sg = st.find_spacegroup();
|
||||
const auto fobs = OwnAmplitudes(cryst, st, 3.5, logger);
|
||||
const size_t cards = get_gpu_count();
|
||||
std::vector<gemmi::Model> models(cards, st.models[0]);
|
||||
auto pool = TestPool(models[0], st.cell, *sg, 3.5, fobs.v.size(), cards, logger);
|
||||
const int caller = RigidBodyGPUEngine::CurrentDevice();
|
||||
std::vector<std::unique_ptr<RigidBodyTargetGPU>> targets;
|
||||
for (size_t k = 0; k < cards; k++)
|
||||
targets.push_back(std::make_unique<RigidBodyTargetGPU>(*pool, models[k], st.cell, *sg, 4));
|
||||
CHECK(RigidBodyGPUEngine::CurrentDevice() == caller);
|
||||
const double q0[6] = {0.12, -0.09, 0.07, 0.15, -0.10, 0.05};
|
||||
for (auto &t : targets) {
|
||||
t->SetZone(fobs, 3.5);
|
||||
CHECK(RigidBodyGPUEngine::CurrentDevice() == caller);
|
||||
std::vector<double> r(t->NumObservations()), j(6 * t->NumObservations());
|
||||
REQUIRE(t->Residuals(q0, r.data()));
|
||||
REQUIRE(t->Jacobian(q0, j.data()));
|
||||
CHECK(RigidBodyGPUEngine::CurrentDevice() == caller);
|
||||
}
|
||||
}
|
||||
|
||||
// The model's structure factors on the GPU against the CPU path model validation takes - gemmi's density and
|
||||
// mask on the symmetrized grid, transformed by FFTW and read off by prepare_asu_data() - in the five groups,
|
||||
// at a resolution where the mask's shrink step does nothing and at one where it does. The same reflections
|
||||
|
||||
Reference in new issue
Block a user