diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 5144111cd..24003342e 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -271,7 +271,7 @@ The refinement walks a coarse-to-fine ladder, 6 Å → 4.5 Å → 3.5 Å, each z The refinement sees **only the working reflections**. The step is then committed only if it lowers **R-free**, computed at full resolution on the free set it never saw; otherwise the model is put back exactly where it was read and the maps are the ones it would have given. The whole addition costs about two seconds. Note that the origin is a gauge in some space groups — free in all three directions in $P1$, and along the unique axis in a polar group — so those components of the translation are undetermined; nothing is done about that beyond the Levenberg–Marquardt damping and the R-free gate, which between them make an undetermined direction harmless rather than unstable. -Where a CUDA GPU is present, the refinement's target is evaluated on it: the same density, the same composition of $F_\mathrm{calc}$ from one copy of the model, the same bulk-solvent mask with its islands removed, the same scale fit and the same Jacobian, computed on the device, where a fit takes a fraction of a second rather than several seconds. The device is chosen once per validation, before anything is computed, and never revisited - the real fit and every replicate of the null of §14.5 on the same one. What goes on the GPU is decided for all parts at once, on one budget: 80% of the card's total memory (judged on the total, so the same input on the same machine always takes the same path), less what a merge still at work holds there. Every engine is reserved up front and kept to the end, so it is their sum that has to fit: the structure factors to the data's resolution (which also make the maps), one rigid-body engine, the null's structure factors to its coarser limit and one engine for the scale fits of §14.2 each take their place in that order where they still fit, and the rest of the budget goes to more rigid-body engines and then more scale-fit engines, which change only how many fits run at once. A part that does not fit, or a cell too small for the GPU's gridding, runs on the CPU, as everything does where there is no GPU. A CUDA failure ends the validation, not the run: the validation reports why it did not finish, decides nothing, and the reflection files are those of a run without a model. The two agree to rounding and not bit for bit: the device measures distances in single precision and transforms with cuFFT instead of FFTW, and GEMMI's scale fit, whose stopping rule leaves it unconverged at about $10^{-4}$ of $|F|$, can respond to that rounding with a step of its own. A rigid-body step accepted or rejected on an R-free difference below about $5\cdot10^{-4}$ can therefore go either way between the two; every kernel is deterministic, so either one gives the same answer on every run. +Where a CUDA GPU is present, the refinement's target is evaluated on it: the same density, the same composition of $F_\mathrm{calc}$ from one copy of the model, the same bulk-solvent mask with its islands removed, the same scale fit and the same Jacobian, computed on the device, where a fit takes a fraction of a second rather than several seconds. The device is chosen once per validation, before anything is computed, and never revisited - the real fit and every replicate of the null of §14.5 on the same one. What goes on the GPU is decided for all parts at once, on one budget: 80% of the card's total memory (judged on the total, so the same input on the same machine always takes the same path), less what a merge still at work holds there. Every engine is reserved up front and kept to the end, so it is their sum that has to fit: the structure factors to the data's resolution (which also make the maps), one rigid-body engine, the null's structure factors to its coarser limit and one engine for the scale fits of §14.2 each take their place in that order where they still fit, and more rigid-body engines and then more scale-fit engines are added only within half of the budget: they change only how many fits run at once, and on a large cell, where one fit already fills the card, they buy little. A part that does not fit, or a cell too small for the GPU's gridding, runs on the CPU, as everything does where there is no GPU. A CUDA failure ends the validation, not the run: the validation reports why it did not finish, decides nothing, and the reflection files are those of a run without a model. The two agree to rounding and not bit for bit: the device measures distances in single precision and transforms with cuFFT instead of FFTW, and GEMMI's scale fit, whose stopping rule leaves it unconverged at about $10^{-4}$ of $|F|$, can respond to that rounding with a step of its own. A rigid-body step accepted or rejected on an R-free difference below about $5\cdot10^{-4}$ can therefore go either way between the two; every kernel is deterministic, so either one gives the same answer on every run. The rotation and translation actually taken are reported (`RIGID_BODY_ROTATION_DEG`, `RIGID_BODY_SHIFT_A`), together with the R-free before it (`R_FREE_BEFORE_RIGID_BODY`), so what the placement bought is visible. diff --git a/image_analysis/structure_refinement/ModelValidation.cpp b/image_analysis/structure_refinement/ModelValidation.cpp index 25b64b63b..46b5d7b9b 100644 --- a/image_analysis/structure_refinement/ModelValidation.cpp +++ b/image_analysis/structure_refinement/ModelValidation.cpp @@ -45,6 +45,7 @@ #include "SigmaA.h" #ifdef JFJOCH_USE_CUDA #include "ModelStructureFactorsGPU.h" +#include "../indexing/CUDAMemHelpers.h" // cuda_trim_allocation_pool #include "RigidBodyGPU.h" #include "RigidBodyGPUEngine.h" // RigidBodyGPUEngine::CurrentDevice #include "../../common/CUDAWrapper.h" @@ -792,6 +793,10 @@ ModelValidationResult Validate(const std::vector &merged, ModelValidationGPUPlan gpu_plan; if (rigid_body_gpu && get_gpu_count() > 0) { const int device = RigidBodyGPUEngine::CurrentDevice(); + // What the stream-ordered pool holds with no buffer in it - freed by the passes before, measured + // at 1-2 GB - goes back to the driver first: the engines below are allocated outside the pool and + // cannot use it, and the budget counts only what is still held. + cuda_trim_allocation_pool(); ModelValidationGPUNeed need; need.card = get_gpu_total_memory(device); need.resident = schedule.gpu_resident_bytes; @@ -1893,7 +1898,8 @@ ModelValidationGPUPlan PlanModelValidationGPU(const ModelValidationGPUNeed &need plan.null_structure_factors = take(need.null_structure_factors, plan.budget); plan.scale_fit_engines = need.scale_fit_engines > 0 && take(need.scale_fit, plan.budget) ? 1 : 0; - const size_t room = plan.budget > need.beside ? plan.budget - need.beside : 0; + // More engines only within half of the budget (see the declaration). + const size_t room = std::min(plan.budget / 2, plan.budget > need.beside ? plan.budget - need.beside : 0); while (plan.rigid_body_engines > 0 && plan.rigid_body_engines < need.rigid_body_engines && take(need.rigid_body, room)) plan.rigid_body_engines++; diff --git a/image_analysis/structure_refinement/ModelValidation.h b/image_analysis/structure_refinement/ModelValidation.h index 5b145c67d..78cfeb972 100644 --- a/image_analysis/structure_refinement/ModelValidation.h +++ b/image_analysis/structure_refinement/ModelValidation.h @@ -314,9 +314,11 @@ struct ModelValidationGPUPlan { // up front and kept to the end - so it is their sum that is held against the budget, 80% of the card // less what is resident. Each part takes one engine where it still fits, in the order of what it saves: // the structure factors to d_min, the rigid body, the null's structure factors, the scale fits; a part -// that does not fit runs on the CPU. The rest of the budget, less what a validation beside holds, goes -// to more rigid-body engines and then more scale-fit engines, which change only how many fits run at -// once. A second validation may run beside where twice the whole of this one's GPU parts, at every +// that does not fit runs on the CPU. More rigid-body engines and then more scale-fit engines are added +// only within half of the budget, less what a validation beside holds: they change only how many fits +// run at once, and buy little - on a large cell a fit already fills the card (measured on a 16 GB card: +// 13.6 s with one rigid-body engine, 13.1 s with two, 12.6 s with five) - so the card keeps headroom +// rather than filling with them. A second validation may run beside where twice the whole of this one's GPU parts, at every // engine it can use, fits the budgets of the cards together. ModelValidationGPUPlan PlanModelValidationGPU(const ModelValidationGPUNeed &need, size_t cards = 1); diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 2e6f46a35..51c9917ea 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -3996,6 +3996,16 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { // for by molecular replacement below. bool placed_by_rigid_body = false; bool twin_chosen_by_model = false; + // Nothing indexes after the merge, so the image pass's indexers - an FFT indexer keeps its transform + // buffers and histograms on the card, 0.8 GB for the two of a rotation run - go before the model is + // validated or searched for, once the leftover-lattice census, the last to use them, has ended. + if (model_run) { + if (census_future.valid()) { + TimingMark census_mark(logger, "leftover-lattice census wait (model)"); + census_future.wait(); + } + p.indexer_pool.reset(); + } if (model_run && !model_predicted && !config_.force_mr) { phase("Validating against model"); Step("Model validation"); diff --git a/tests/ModelValidationTest.cpp b/tests/ModelValidationTest.cpp index 59bc85639..e20b00984 100644 --- a/tests/ModelValidationTest.cpp +++ b/tests/ModelValidationTest.cpp @@ -145,10 +145,15 @@ TEST_CASE("ModelValidation_GPUPlanOnCardSizes", "[ModelValidation]") { last_rb = plan.rigid_body_engines; last_scale = plan.scale_fit_engines; } - // 8 GB, a 6.40 GB budget: 1.40 + 0.89 + 2 x 1.88 + 1 x 0.25 = 6.31 GB. + // 8 GB, a 6.40 GB budget: one engine of each, 4.42 GB, already over half of it, so no more engines. const ModelValidationGPUPlan small = PlanModelValidationGPU(LargeCellNeed(8 * GB)); - CHECK(small.rigid_body_engines == 2); + CHECK(small.rigid_body_engines == 1); CHECK(small.scale_fit_engines == 1); + CHECK(small.bytes == size_t(4424500000)); + // 16 GB, half of the 12.8 GB budget: a second rigid-body engine, 6.31 GB, and no second scale fit. + const ModelValidationGPUPlan mid = PlanModelValidationGPU(LargeCellNeed(16 * GB)); + CHECK(mid.rigid_body_engines == 2); + CHECK(mid.scale_fit_engines == 1); CHECK(!small.may_run_beside); // 128 GB: every engine wanted, and room for a second validation of the same size. const ModelValidationGPUPlan large = PlanModelValidationGPU(LargeCellNeed(128 * GB)); @@ -198,7 +203,7 @@ TEST_CASE("ModelValidation_GPUPlanBesideChangesOnlyEngines", "[ModelValidation]" CHECK(full.structure_factors); CHECK(full.rigid_body_engines == 0); CHECK(!full.null_structure_factors); - CHECK(full.scale_fit_engines == 3); + CHECK(full.scale_fit_engines == 1); } TEST_CASE("ModelValidation_ReadsPdbAndMmcif", "[ModelValidation]") {