From 0d25d765cfffaf5f8bec21cb4af80d3f1d5cc149 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 15 Oct 2026 08:05:50 +0200 Subject: [PATCH] Model validation: one GPU budget for all its parts, 80% of the card's total What a validation puts on its GPU was decided part by part on fixed fractions of the card: the structure factors to d_min on a quarter, the null's structure factors on whatever of that quarter the first left, the rigid-body engines on another quarter, the scale fits on an eighth. On an 8 GB card that sent the null's structure factors of a 4.4 GB validation to the CPU (and held the rigid body to one engine) although everything the validation holds fits the card with room to spare. Now PlanModelValidationGPU decides all of it at once, before anything is reserved, from the card's total memory: every engine is reserved up front and kept to the end, so their sum is held against one budget, 80% of the card less what a merge engine still at work holds there (the finalist ledger's P1 merge, RotationScaleMerge::DeviceBytes). Each part takes one engine where it still fits, in the order structure factors to d_min, rigid body, the null's structure factors, scale fits; a part that does not fit runs on the CPU. What is left goes to more rigid-body engines (up to one per null replicate plus the real fit) and then more scale-fit engines, which only change how many fits run at once. A second validation may run beside where twice this one's whole GPU plan fits the cards' budgets; it gets this one's bytes as gpu_beside_bytes, which limit its engine counts and never its placement, so its numbers are those of the validation it stands in for. One log line states the placement, the bytes held and the budget. - RigidBodyGPUPool::Plan sizes the pool without engines (EngineBytes), Reserve(n) makes them; Create (Plan + engines on a quarter of the card) stays for molecular replacement, whose own rules, and the --mr hand check's structure factors (ReadModelAnomalousMap, a quarter), are unchanged. - The null's resolution is worked out before the plan, so its engine is decided and reserved with the others instead of after the indexing probe. Placement changes only on smaller cards: with the logged footprints of the open-arm sets, 4nwv's null structure factors move to the GPU below ~9 GB (and its rigid body at 6 GB), 8tyy's structure factors to d_min below ~8.7 GB; on 16 GB and larger nothing moves. On the RTX 5080 (16 GB), 4nwv, 8tyy, 6oel and 3inp give byte-identical reflection files, maps and models to rc175 HEAD, with more rigid-body engines where they fit. Co-Authored-By: Claude Opus 5.5 --- docs/CPU_DATA_ANALYSIS_DECISIONS.md | 2 +- .../structure_refinement/ModelValidation.cpp | 338 ++++++++++-------- .../structure_refinement/ModelValidation.h | 49 ++- .../structure_refinement/RigidBodyGPU.cpp | 77 ++-- .../structure_refinement/RigidBodyGPU.h | 24 +- rugnux/RugnuxScaleMerge.cpp | 20 +- tests/ModelValidationTest.cpp | 89 +++++ 7 files changed, 403 insertions(+), 196 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index e976e4498..5144111cd 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -271,7 +271,7 @@ The refinement walks a coarse-to-fine ladder, 6 Å → 4.5 Å → 3.5 Å, each z The refinement sees **only the working reflections**. The step is then committed only if it lowers **R-free**, computed at full resolution on the free set it never saw; otherwise the model is put back exactly where it was read and the maps are the ones it would have given. The whole addition costs about two seconds. Note that the origin is a gauge in some space groups — free in all three directions in $P1$, and along the unique axis in a polar group — so those components of the translation are undetermined; nothing is done about that beyond the Levenberg–Marquardt damping and the R-free gate, which between them make an undetermined direction harmless rather than unstable. -Where a CUDA GPU is present, the refinement's target is evaluated on it: the same density, the same composition of $F_\mathrm{calc}$ from one copy of the model, the same bulk-solvent mask with its islands removed, the same scale fit and the same Jacobian, computed on the device, where a fit takes a fraction of a second rather than several seconds. The device is chosen once per validation, before anything is computed, and never revisited - the real fit and every replicate of the null of §14.5 on the same one. The structure factors go to the GPU where their buffers fit a quarter of the card's total memory (judged on the total, so the same input on the same machine always takes the same path), the rigid-body engines take up to another quarter, and the scale fits of §14.2 run on the GPU too, in engines of their own that take at most an eighth of the card; a part that does not fit, or a cell too small for the GPU's gridding, runs on the CPU, as everything does where there is no GPU. A CUDA failure ends the validation, not the run: the validation reports why it did not finish, decides nothing, and the reflection files are those of a run without a model. The two agree to rounding and not bit for bit: the device measures distances in single precision and transforms with cuFFT instead of FFTW, and GEMMI's scale fit, whose stopping rule leaves it unconverged at about $10^{-4}$ of $|F|$, can respond to that rounding with a step of its own. A rigid-body step accepted or rejected on an R-free difference below about $5\cdot10^{-4}$ can therefore go either way between the two; every kernel is deterministic, so either one gives the same answer on every run. +Where a CUDA GPU is present, the refinement's target is evaluated on it: the same density, the same composition of $F_\mathrm{calc}$ from one copy of the model, the same bulk-solvent mask with its islands removed, the same scale fit and the same Jacobian, computed on the device, where a fit takes a fraction of a second rather than several seconds. The device is chosen once per validation, before anything is computed, and never revisited - the real fit and every replicate of the null of §14.5 on the same one. What goes on the GPU is decided for all parts at once, on one budget: 80% of the card's total memory (judged on the total, so the same input on the same machine always takes the same path), less what a merge still at work holds there. Every engine is reserved up front and kept to the end, so it is their sum that has to fit: the structure factors to the data's resolution (which also make the maps), one rigid-body engine, the null's structure factors to its coarser limit and one engine for the scale fits of §14.2 each take their place in that order where they still fit, and the rest of the budget goes to more rigid-body engines and then more scale-fit engines, which change only how many fits run at once. A part that does not fit, or a cell too small for the GPU's gridding, runs on the CPU, as everything does where there is no GPU. A CUDA failure ends the validation, not the run: the validation reports why it did not finish, decides nothing, and the reflection files are those of a run without a model. The two agree to rounding and not bit for bit: the device measures distances in single precision and transforms with cuFFT instead of FFTW, and GEMMI's scale fit, whose stopping rule leaves it unconverged at about $10^{-4}$ of $|F|$, can respond to that rounding with a step of its own. A rigid-body step accepted or rejected on an R-free difference below about $5\cdot10^{-4}$ can therefore go either way between the two; every kernel is deterministic, so either one gives the same answer on every run. The rotation and translation actually taken are reported (`RIGID_BODY_ROTATION_DEG`, `RIGID_BODY_SHIFT_A`), together with the R-free before it (`R_FREE_BEFORE_RIGID_BODY`), so what the placement bought is visible. diff --git a/image_analysis/structure_refinement/ModelValidation.cpp b/image_analysis/structure_refinement/ModelValidation.cpp index e1148e07b..25b64b63b 100644 --- a/image_analysis/structure_refinement/ModelValidation.cpp +++ b/image_analysis/structure_refinement/ModelValidation.cpp @@ -491,71 +491,39 @@ void read_anomalous_map(const gemmi::Grid &grid, double rms, gemmi::Struc } #ifdef JFJOCH_USE_CUDA -// The model's structure factors to the data's resolution, and the maps, on the GPU - or not, decided here, -// once, before any of them is computed, and never revisited: nothing is moved to the CPU part way through. -// Decided on what the GPU path needs against the card's TOTAL memory and not against what happens to be -// free at the moment, so the same input on the same machine always takes the same path, whatever else is -// running beside it. A validation plans at most half of its card: its structure-factor engines a quarter -// together - `reserved` is what one of them already holds there - and its rigid-body engines a quarter -// (RigidBodyGPUPool::Create). A second validation runs beside it only where twice that plan fits half of -// the cards together (see on_forecast below), so validations never plan more than half of the memory. -std::unique_ptr StructureFactorsGPU(int device, const gemmi::Model &model, - const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, - double d_min, size_t reserved, Logger &logger) { +// An engine for the model's structure factors to d_min (and the maps) on `device`, nothing reserved yet; +// null, logged, where the cell is too small for the GPU's gridding. +std::unique_ptr StructureFactorsEngine(int device, const gemmi::Model &model, + const gemmi::UnitCell &cell, + const gemmi::SpaceGroup &sg, double d_min, + Logger &logger) { std::unique_ptr sf; try { sf = std::make_unique(device, model, cell, sg, d_min); } catch (const JFJochException &e) { throw RigidBodyGPUFailure(e.what()); } - const std::array n = sf->GridSize(); if (!sf->Supported()) { logger.Info("Model validation: structure factors to {:.2f} A on the CPU - the cell is too small for the " "GPU's gridding", d_min); return nullptr; } - const size_t total = sf->DeviceTotalMemory(); - if (reserved + sf->DeviceBytes() > total / 4) { - logger.Info("Model validation: structure factors to {:.2f} A on the CPU - on a {}x{}x{} grid they need " - "{:.2f} GB of GPU memory beside {:.2f} GB already reserved, over a quarter of the card's {:.2f} GB", - d_min, n[0], n[1], n[2], sf->DeviceBytes() / 1e9, reserved / 1e9, total / 1e9); - return nullptr; - } - try { - sf->Reserve(); - } catch (const JFJochException &e) { - throw RigidBodyGPUFailure(e.what()); - } - logger.Info("Model validation: structure factors to {:.2f} A on the GPU - a {}x{}x{} grid, {:.2f} GB of the " - "card's {:.2f} GB", d_min, n[0], n[1], n[2], sf->DeviceBytes() / 1e9, total / 1e9); return sf; } -// The scale fits on the GPU (ModelScaleGPUPool) - or not, decided here, once, on the card's total memory as -// the structure factors are: SCALE_GPU_ENGINES engines, fewer where they would take over an eighth of the -// card, none where not even one fits. -constexpr size_t SCALE_GPU_ENGINES = 4; - -std::unique_ptr ScaleFitsGPU(int device, size_t max_points, Logger &logger) { - const size_t total = get_gpu_total_memory(device); - size_t engines = SCALE_GPU_ENGINES; - while (engines > 0 && ModelScaleGPUPool::DeviceBytes(max_points, engines) > total / 8) - engines--; - if (engines == 0) { - logger.Info("Model validation: scale fits on the CPU - one fit of {} reflections needs {:.2f} GB of GPU " - "memory, over an eighth of the card's {:.2f} GB", max_points, - ModelScaleGPUPool::DeviceBytes(max_points, 1) / 1e9, total / 1e9); - return nullptr; - } - std::unique_ptr pool; +void Reserve(ModelStructureFactorsGPU &sf) { try { - pool = std::make_unique(device, max_points, engines); + sf.Reserve(); } catch (const JFJochException &e) { throw RigidBodyGPUFailure(e.what()); } - logger.Info("Model validation: scale fits on the GPU, {} engine(s), {:.2f} GB of the card's {:.2f} GB", - engines, pool->Bytes() / 1e9, total / 1e9); - return pool; +} + +// The validation's scale fits can use this many engines at once. +constexpr size_t SCALE_GPU_ENGINES = 4; + +std::string GBytes(size_t bytes) { + return fmt::format("{:.2f} GB", bytes / 1e9); } #endif @@ -780,22 +748,130 @@ ModelValidationResult Validate(const std::vector &merged, logger.Info("Model validation: {} atoms, cell a={:.2f} b={:.2f} c={:.2f}, sg {}, to {:.2f} A", gemmi::count_atom_sites(st.models[0]), ucell.a, ucell.b, ucell.c, sg->hm, d_min); - // The structure factors to d_min and the maps on the GPU, where `rigid_body_gpu` allows it and the - // card can take them; decided here, before any is computed. The null's, to its coarser limit, get an - // engine of their own further down, decided the same way before any replicate starts - for all of - // them and the real model's side of the null together, so that every one is computed the same way. + // Whether the model is asked to fit at all (see "does the model fit at all?" below), and the resolution + // its null is scored to, which the null's structure-factor engine is sized for. + const bool decision_pending = schedule.fit_test; + double null_d_min = d_min; + if (decision_pending) { + // The null and the side of the real fit it is compared with are both scored to NULL_D_MIN, + // not to the data's resolution. The placement already stops there (RigidBodyRefine's ladder + // ends at 3.5 A), and whether a model is in the right orientation, or the data in its + // indexing, is decided at low resolution: the finer shells only add reflections to the scale + // fits, which were most of the null's cost. Both sides go through the same fits on the same + // reflections, so the comparison stays like for like. On a small cell 3.5 A leaves too few + // free reflections for a real indexing margin to be told from a random placement's, so the + // limit then moves out to where NULL_MIN_FREE of them are - to the data's own on the + // smallest, which are the cheap ones. + std::vector free_d; + for (const MergedReflection &r : obs) + if (r.rfree_flag && !std::isnan(r.F)) + free_d.push_back(r.d); + std::sort(free_d.begin(), free_d.end(), std::greater<>()); + null_d_min = std::max(d_min, NULL_D_MIN); + if (free_d.size() < NULL_MIN_FREE) + null_d_min = d_min; + else + null_d_min = std::min(null_d_min, free_d[NULL_MIN_FREE - 1]); + } + + // What runs on the GPU, where `rigid_body_gpu` allows it and there is a card: the structure factors to + // d_min and the maps, the null's structure factors to its coarser limit (where that is not d_min), the + // rigid body and the scale fits - all decided here, at once, by PlanModelValidationGPU, before any of + // them is computed, and never revisited: nothing is moved to the CPU part way through. Every engine is + // reserved now and kept to the end, the real fit and every replicate of the null on the same ones, so + // that a verdict never depends on how full the card was when a replicate started. One card for the + // structure factors and the scale fits, the current one; the rigid body's engines count on from it. ModelStructureFactorsGPU *sf_gpu = nullptr, *sf_null = nullptr; + RigidBodyGPUPool *rigid_body_pool = nullptr; + size_t gpu_bytes = 0; + bool may_run_beside = true; #ifdef JFJOCH_USE_CUDA std::unique_ptr sf_engine, sf_null_engine; - // On the current device, as the rigid body's engines take it: one card for the whole validation. - if (rigid_body_gpu && get_gpu_count() > 0) - sf_engine = StructureFactorsGPU(RigidBodyGPUEngine::CurrentDevice(), st.models[0], ucell, *sg, d_min, 0, - logger); - sf_gpu = sf_engine.get(); - // The scale fits beside them, on the same card, for as many reflections as the data have. + std::unique_ptr rigid_body_engines; std::unique_ptr scale_gpu; - if (rigid_body_gpu && get_gpu_count() > 0) - scale_gpu = ScaleFitsGPU(RigidBodyGPUEngine::CurrentDevice(), obs.size(), logger); + ModelValidationGPUPlan gpu_plan; + if (rigid_body_gpu && get_gpu_count() > 0) { + const int device = RigidBodyGPUEngine::CurrentDevice(); + ModelValidationGPUNeed need; + need.card = get_gpu_total_memory(device); + need.resident = schedule.gpu_resident_bytes; + need.beside = schedule.gpu_beside_bytes; + sf_engine = StructureFactorsEngine(device, st.models[0], ucell, *sg, d_min, logger); + if (sf_engine) + need.structure_factors = sf_engine->DeviceBytes(); + if (null_d_min != d_min) { + sf_null_engine = StructureFactorsEngine(device, st.models[0], ucell, *sg, null_d_min, logger); + if (sf_null_engine) + need.null_structure_factors = sf_null_engine->DeviceBytes(); + } + // The rigid body's engines are sized for the most observations a fit gives its finest zone, + // counted as the zones count them: by the resolution of the index once it is in the model's + // ASU, under each indexing the data may be placed in. The model's group need not keep the + // data's metric - a wrong model's cubic group on an orthorhombic cell sends h to an index of + // another resolution - so the data's own d can put a reflection on the wrong side of the zone. + const double finest_zone = RigidBodyLadder(d_min).back(); + const gemmi::GroupOps zone_gops = sg->operations(); + const gemmi::ReciprocalAsu zone_asu(sg); + auto in_finest_zone = [&](const std::vector &v) { + size_t n = 0; + for (const MergedReflection &r : v) + if (!std::isnan(r.F) && ucell.calculate_d(zone_asu.to_asu({r.h, r.k, r.l}, zone_gops).first) >= finest_zone) + ++n; + return n; + }; + size_t zone_observations = in_finest_zone(obs); + for (const gemmi::Op &law : reindex_ops) + zone_observations = std::max(zone_observations, in_finest_zone(ReindexReflections(obs, law))); + rigid_body_engines = RigidBodyGPUPool::Plan(st.models[0], ucell, *sg, d_min, zone_observations, logger); + if (rigid_body_engines) { + need.rigid_body = rigid_body_engines->EngineBytes(); + // One for the real model and one for each replicate of the null placed beside it. + need.rigid_body_engines = std::min(std::max(nthreads, 1), NULL_REPLICATES) + 1; + } + // For as many reflections as the data have. + need.scale_fit = ModelScaleGPUPool::DeviceBytes(obs.size(), 1); + need.scale_fit_engines = SCALE_GPU_ENGINES; + + gpu_plan = PlanModelValidationGPU(need, static_cast(get_gpu_count())); + gpu_bytes = gpu_plan.bytes; + may_run_beside = gpu_plan.may_run_beside; + + std::string parts; + auto part = [&](const std::string &what, bool on_gpu, size_t bytes, size_t engines) { + if (bytes == 0) + return; // no such part, or one the GPU cannot grid (logged where that was found) + parts += fmt::format("{}{} on the ", parts.empty() ? "" : ", ", what); + if (!on_gpu) + parts += fmt::format("CPU ({} do not fit)", GBytes(bytes)); + else if (engines > 0) + parts += fmt::format("GPU ({} x {})", engines, GBytes(bytes)); + else + parts += fmt::format("GPU ({})", GBytes(bytes)); + }; + part(fmt::format("structure factors to {:.2f} A", d_min), gpu_plan.structure_factors, + need.structure_factors, 0); + part(fmt::format("the null's to {:.2f} A", null_d_min), gpu_plan.null_structure_factors, + need.null_structure_factors, 0); + part("rigid body", gpu_plan.rigid_body_engines > 0, need.rigid_body, gpu_plan.rigid_body_engines); + part("scale fits", gpu_plan.scale_fit_engines > 0, need.scale_fit, gpu_plan.scale_fit_engines); + logger.Info("Model validation: {} - {} held of a {} budget, 80% of the card's {}{}{}", parts, + GBytes(gpu_plan.bytes), GBytes(gpu_plan.budget), GBytes(need.card), + need.resident > 0 ? fmt::format(" less {} held by the merge", GBytes(need.resident)) : "", + need.beside > 0 ? fmt::format("; {} held by a validation beside", GBytes(need.beside)) : ""); + + if (!gpu_plan.structure_factors) + sf_engine.reset(); + if (!gpu_plan.null_structure_factors) + sf_null_engine.reset(); + if (sf_engine) + Reserve(*sf_engine); + if (sf_null_engine) + Reserve(*sf_null_engine); + } + sf_gpu = sf_engine.get(); + sf_null = sf_null_engine.get(); +#else + (void) rigid_body_gpu; #endif // --- Fcalc (atomic) via electron density on a grid + FFT, plus a flat bulk-solvent mask -> Fmask. @@ -840,42 +916,23 @@ ModelValidationResult Validate(const std::vector &merged, .prepare_asu_data(dc.d_min, dc.blur, false, false, false); ms.fmask = fmask.get(); }; - // The model's structure factors are made beside the reservation of the GPU engines below, which - // does not need them. + // The model's structure factors are made beside the reservation of the rigid body's and the scale + // fits' engines below, which does not need them. std::future model_factors = std::async(std::launch::async, [&] { compute_model_factors(mdl, d_min); }); - // The rigid body's device, decided once for the whole validation - the real fit and every replicate - // of the null on the same one, so that a verdict never depends on how full the card was when a - // replicate started. The engines are reserved here, up front, and never grow. - RigidBodyGPUPool *rigid_body_pool = nullptr; #ifdef JFJOCH_USE_CUDA - std::unique_ptr rigid_body_engines; - if (rigid_body_gpu) { - // Counted as the zones count them: by the resolution of the index once it is in the model's - // ASU, under each indexing the data may be placed in. The model's group need not keep the - // data's metric - a wrong model's cubic group on an orthorhombic cell sends h to an index of - // another resolution - so the data's own d can put a reflection on the wrong side of the zone. - const double finest_zone = RigidBodyLadder(d_min).back(); - const gemmi::GroupOps zone_gops = sg->operations(); - const gemmi::ReciprocalAsu zone_asu(sg); - auto in_finest_zone = [&](const std::vector &v) { - size_t n = 0; - for (const MergedReflection &r : v) - if (!std::isnan(r.F) && ucell.calculate_d(zone_asu.to_asu({r.h, r.k, r.l}, zone_gops).first) >= finest_zone) - ++n; - return n; - }; - size_t zone_observations = in_finest_zone(obs); - for (const gemmi::Op &law : reindex_ops) - zone_observations = std::max(zone_observations, in_finest_zone(ReindexReflections(obs, law))); - // One engine for the real model and one for each replicate of the null placed beside it. - rigid_body_engines = RigidBodyGPUPool::Create(st.models[0], ucell, *sg, d_min, zone_observations, - std::min(std::max(nthreads, 1), NULL_REPLICATES) + 1, - logger); + if (rigid_body_engines && gpu_plan.rigid_body_engines > 0) { + rigid_body_engines->Reserve(gpu_plan.rigid_body_engines); rigid_body_pool = rigid_body_engines.get(); } -#else - (void) rigid_body_gpu; + if (gpu_plan.scale_fit_engines > 0) { + try { + scale_gpu = std::make_unique(RigidBodyGPUEngine::CurrentDevice(), obs.size(), + gpu_plan.scale_fit_engines); + } catch (const JFJochException &e) { + throw RigidBodyGPUFailure(e.what()); + } + } #endif model_factors.get(); @@ -1092,36 +1149,13 @@ ModelValidationResult Validate(const std::vector &merged, // not place is searched for by molecular replacement instead (Rugnux). Only a validation whose // verdict is taken from another one (KeepModelVerdict), or that is not wanted (a placement already // judged elsewhere, scored for its maps), skips it. - const bool decision_pending = schedule.fit_test; - if (schedule.on_forecast) { - // Two validations at once take twice the engines, and are allowed where twice what this one plans - - // its rigid-body engines, its structure-factor engine and, where a null is coming, the null's, - // which is no larger - fits half of the cards' memory taken together (see StructureFactorsGPU). - bool may_run_beside = true; -#ifdef JFJOCH_USE_CUDA - size_t planned = 0, card = 0; - if (rigid_body_pool != nullptr) { - planned += rigid_body_pool->PlannedBytes(); - card = rigid_body_pool->CardBytes(); - } - if (sf_gpu != nullptr) { - planned += (decision_pending ? 2 : 1) * sf_gpu->DeviceBytes(); - card = sf_gpu->DeviceTotalMemory(); - } - if (scale_gpu) { - planned += scale_gpu->Bytes(); - card = get_gpu_total_memory(RigidBodyGPUEngine::CurrentDevice()); - } - if (planned > 0) - may_run_beside = 2 * planned <= static_cast(get_gpu_count()) * card / 2; -#endif + // A second validation is allowed beside this one where PlanModelValidationGPU says it fits. + if (schedule.on_forecast) schedule.on_forecast({result.change_of_basis_op, indexing.op, result.model_enantiomorph_candidate, - may_run_beside}); - } + may_run_beside, gpu_bytes}); std::vector equivalent; // the orientations the crystal cannot tell from the model's double reach_deg = 0; std::vector null_rotation; - double null_d_min = d_min; std::vector obs_null; if (decision_pending) { // --- the null: this model, this data, in random orientations --- @@ -1181,36 +1215,10 @@ ModelValidationResult Validate(const std::vector &merged, "({:.1f} deg) of an orientation equivalent to the model's, and were drawn again", redraws, reach_deg); } - // The null and the side of the real fit it is compared with are both scored to NULL_D_MIN, - // not to the data's resolution. The placement already stops there (RigidBodyRefine's ladder - // ends at 3.5 A), and whether a model is in the right orientation, or the data in its - // indexing, is decided at low resolution: the finer shells only add reflections to the scale - // fits, which were most of the null's cost. Both sides go through the same fits on the same - // reflections, so the comparison stays like for like. On a small cell 3.5 A leaves too few - // free reflections for a real indexing margin to be told from a random placement's, so the - // limit then moves out to where NULL_MIN_FREE of them are - to the data's own on the - // smallest, which are the cheap ones. - std::vector free_d; - for (const MergedReflection &r : obs) - if (r.rfree_flag && !std::isnan(r.F)) - free_d.push_back(r.d); - std::sort(free_d.begin(), free_d.end(), std::greater<>()); - null_d_min = std::max(d_min, NULL_D_MIN); - if (free_d.size() < NULL_MIN_FREE) - null_d_min = d_min; - else - null_d_min = std::min(null_d_min, free_d[NULL_MIN_FREE - 1]); + // Scored to null_d_min (above), on its own structure-factor engine where that is not d_min. for (const MergedReflection &r : obs) if (r.d >= null_d_min) obs_null.push_back(r); -#ifdef JFJOCH_USE_CUDA - // On the card the d_min engine is on; the replicates' threads may sit on other cards, and each - // call works on the engine's own device. Where the null is scored to d_min it shares that engine. - if (sf_gpu != nullptr && null_d_min != d_min) - sf_null_engine = StructureFactorsGPU(sf_gpu->Device(), st.models[0], ucell, *sg, null_d_min, - sf_gpu->DeviceBytes(), logger); - sf_null = sf_null_engine.get(); -#endif } const gemmi::Structure null_model = st; // the model as read, which the replicates are made from // Each replicate works on its own copy of the model as read and of its structure factors, and the @@ -1264,7 +1272,7 @@ ModelValidationResult Validate(const std::vector &merged, for (size_t t = 0; t < std::min(std::max(nthreads, 1), NULL_REPLICATES); t++) replicate_threads.push_back(std::async(std::launch::async, [&, t] { // Each on a card in turn from the one after the real fit's, which its engines are - // spread the same way over (RigidBodyGPUPool::Create). + // spread the same way over (RigidBodyGPUPool::Reserve). #ifdef JFJOCH_USE_CUDA if (rigid_body_pool != nullptr) pin_gpu((rigid_body_pool->Device() + 1 + static_cast(t)) % get_gpu_count()); @@ -1870,6 +1878,37 @@ ModelValidationResult Validate(const std::vector &merged, } // namespace +ModelValidationGPUPlan PlanModelValidationGPU(const ModelValidationGPUNeed &need, size_t cards) { + ModelValidationGPUPlan plan; + const size_t usable = need.card / 5 * 4; + plan.budget = usable > need.resident ? usable - need.resident : 0; + auto take = [&](size_t bytes, size_t limit) { + if (bytes == 0 || plan.bytes + bytes > limit) + return false; + plan.bytes += bytes; + return true; + }; + plan.structure_factors = take(need.structure_factors, plan.budget); + plan.rigid_body_engines = need.rigid_body_engines > 0 && take(need.rigid_body, plan.budget) ? 1 : 0; + plan.null_structure_factors = take(need.null_structure_factors, plan.budget); + plan.scale_fit_engines = need.scale_fit_engines > 0 && take(need.scale_fit, plan.budget) ? 1 : 0; + + const size_t room = plan.budget > need.beside ? plan.budget - need.beside : 0; + while (plan.rigid_body_engines > 0 && plan.rigid_body_engines < need.rigid_body_engines + && take(need.rigid_body, room)) + plan.rigid_body_engines++; + while (plan.scale_fit_engines > 0 && plan.scale_fit_engines < need.scale_fit_engines + && take(need.scale_fit, room)) + plan.scale_fit_engines++; + + const size_t whole = (plan.structure_factors ? need.structure_factors : 0) + + (plan.null_structure_factors ? need.null_structure_factors : 0) + + (plan.rigid_body_engines > 0 ? need.rigid_body * need.rigid_body_engines : 0) + + (plan.scale_fit_engines > 0 ? need.scale_fit * need.scale_fit_engines : 0); + plan.may_run_beside = 2 * whole <= cards * plan.budget; + return plan; +} + ModelValidationResult ValidateAgainstModel(const std::vector &merged, const UnitCell &cell, const std::string &model_path, @@ -1973,8 +2012,17 @@ ModelValidationResult ReadModelAnomalousMap(const std::vector ModelStructureFactorsGPU *gpu = nullptr; #ifdef JFJOCH_USE_CUDA std::unique_ptr engine; + // Read beside molecular replacement's engines, which keep to a quarter of the card + // (MolecularReplacement.cpp), so on the GPU where it fits a quarter as well. if (get_gpu_count() > 0) - engine = StructureFactorsGPU(RigidBodyGPUEngine::CurrentDevice(), st.models[0], st.cell, *sg, d_min, 0, logger); + engine = StructureFactorsEngine(RigidBodyGPUEngine::CurrentDevice(), st.models[0], st.cell, *sg, d_min, logger); + if (engine && engine->DeviceBytes() > engine->DeviceTotalMemory() / 4) { + logger.Info("Model validation: structure factors to {:.2f} A on the CPU - {} of GPU memory, over a quarter " + "of the card's {}", d_min, GBytes(engine->DeviceBytes()), GBytes(engine->DeviceTotalMemory())); + engine.reset(); + } + if (engine) + Reserve(*engine); gpu = engine.get(); if (gpu != nullptr) { try { diff --git a/image_analysis/structure_refinement/ModelValidation.h b/image_analysis/structure_refinement/ModelValidation.h index 5bf87306f..5b145c67d 100644 --- a/image_analysis/structure_refinement/ModelValidation.h +++ b/image_analysis/structure_refinement/ModelValidation.h @@ -257,9 +257,10 @@ struct ModelFrameForecast { gemmi::Op change_of_basis_op = gemmi::Op::identity(); gemmi::Op indexing_op = gemmi::Op::identity(); bool enantiomorph_candidate = false; - // Whether a second validation of this size may run beside this one: decided on the sizes of its - // GPU engines against the memory of the cards, never on what happens to be free. + // Whether a second validation of this size may run beside this one (ModelValidationGPUPlan), and + // what this one holds on its card meanwhile - which the second is given as gpu_beside_bytes. bool may_run_beside = true; + size_t gpu_bytes = 0; }; struct ModelValidationSchedule { @@ -273,8 +274,52 @@ struct ModelValidationSchedule { // The null that says whether the model fits. False where the verdict is another validation's // (KeepModelVerdict) or was reached elsewhere (a molecular-replacement placement, scored for its maps). bool fit_test = true; + // GPU memory held on the card for as long as the validation runs by what came before it (a merge + // engine still at work beside it): counted against the card when what goes on the GPU is decided. + size_t gpu_resident_bytes = 0; + // GPU memory another validation running beside this one holds there. It limits how many engines + // this one takes and never what runs on the GPU, so no number depends on whether the other runs. + size_t gpu_beside_bytes = 0; }; +// What a model validation needs of its card, part by part: the bytes of one engine of each, how many +// engines of it the validation can use, and what else is on the card. 0 bytes: that part cannot run on +// the GPU (the cell is too small for its gridding), or there is none (the null's structure factors +// where the null is scored to the data's own resolution and shares the engine to d_min). +struct ModelValidationGPUNeed { + size_t card = 0; // the card's total memory + size_t resident = 0; // ModelValidationSchedule::gpu_resident_bytes + size_t beside = 0; // ModelValidationSchedule::gpu_beside_bytes + size_t structure_factors = 0; // to the data's resolution, with the maps + size_t null_structure_factors = 0; // to the null's coarser limit + size_t rigid_body = 0; + size_t rigid_body_engines = 0; + size_t scale_fit = 0; + size_t scale_fit_engines = 0; +}; + +struct ModelValidationGPUPlan { + size_t budget = 0; // 80% of the card, less what is resident there + bool structure_factors = false; + bool null_structure_factors = false; + size_t rigid_body_engines = 0; // 0: the rigid body on the CPU + size_t scale_fit_engines = 0; // 0: the scale fits on the CPU + size_t bytes = 0; // what the plan holds on the card + bool may_run_beside = false; +}; + +// The one rule for what a validation puts on its GPU, decided before anything is reserved, from the +// card's TOTAL memory and never from what is free at the moment, so the same input on the same card +// always runs the same way. Everything the validation holds is alive at once - the engines are reserved +// up front and kept to the end - so it is their sum that is held against the budget, 80% of the card +// less what is resident. Each part takes one engine where it still fits, in the order of what it saves: +// the structure factors to d_min, the rigid body, the null's structure factors, the scale fits; a part +// that does not fit runs on the CPU. The rest of the budget, less what a validation beside holds, goes +// to more rigid-body engines and then more scale-fit engines, which change only how many fits run at +// once. A second validation may run beside where twice the whole of this one's GPU parts, at every +// engine it can use, fits the budgets of the cards together. +ModelValidationGPUPlan PlanModelValidationGPU(const ModelValidationGPUNeed &need, size_t cards = 1); + ModelValidationResult ValidateAgainstModel(const std::vector &merged, const UnitCell &cell, const std::string &model_path, diff --git a/image_analysis/structure_refinement/RigidBodyGPU.cpp b/image_analysis/structure_refinement/RigidBodyGPU.cpp index bc5632942..15103a399 100644 --- a/image_analysis/structure_refinement/RigidBodyGPU.cpp +++ b/image_analysis/structure_refinement/RigidBodyGPU.cpp @@ -127,10 +127,9 @@ bool SameObservations(const gemmi::AsuData> &a, } // namespace -std::unique_ptr RigidBodyGPUPool::Create(const gemmi::Model &model, const gemmi::UnitCell &cell, - const gemmi::SpaceGroup &sg, double d_min, - size_t max_observations, size_t max_engines, - Logger &logger) { +std::unique_ptr RigidBodyGPUPool::Plan(const gemmi::Model &model, const gemmi::UnitCell &cell, + const gemmi::SpaceGroup &sg, double d_min, + size_t max_observations, Logger &logger) { if (get_gpu_count() == 0) return nullptr; try { @@ -155,40 +154,54 @@ std::unique_ptr RigidBodyGPUPool::Create(const gemmi::Model &m cap.observations = max_observations; cap.rows = max_observations; cap.terms = max_observations * ops; - const size_t bytes = RigidBodyGPUEngine::DeviceBytes(cap); - - // A quarter of the card's TOTAL memory - the validation's plan (StructureFactorsGPU) - and not what - // happens to be free: the same input on the same card always gets the same engines. A card that - // is merely busy is waited for by the engines' allocations, never left for the CPU. - const int device = RigidBodyGPUEngine::CurrentDevice(); - const size_t total = get_gpu_total_memory(device); - const size_t budget = total / 4; - const size_t fit = bytes > 0 ? budget / bytes : 0; - const size_t want = std::min(fit, std::max(max_engines, 1)); - // Engine i on the i-th card from the calling thread's, so a validation uses every card there is; - // the engines give the same bits on any card of one model, so this changes only where they run. - const int cards = get_gpu_count(); - pool->device_ = device; - pool->planned_bytes_ = bytes * std::max(max_engines, 1); - pool->card_bytes_ = total; - for (size_t i = 0; i < want; i++) - pool->engines_.push_back(std::make_unique(cap, (device + static_cast(i)) % cards)); - set_gpu(device); // each engine was made on its own card - if (pool->engines_.empty()) { - logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB, over a " - "quarter of the card's {:.0f} MB", bytes / 1e6, total / 1e6); - return nullptr; - } - for (auto &e : pool->engines_) - pool->idle_.push_back(e.get()); - logger.Info("Model validation: the rigid body runs on the GPU, {} engine(s) of {:.0f} MB", pool->Engines(), - bytes / 1e6); + pool->cap_ = std::make_unique(cap); + pool->device_ = RigidBodyGPUEngine::CurrentDevice(); return pool; } catch (const JFJochException &e) { throw RigidBodyGPUFailure(e.what()); } } +size_t RigidBodyGPUPool::EngineBytes() const { + return RigidBodyGPUEngine::DeviceBytes(*cap_); +} + +void RigidBodyGPUPool::Reserve(size_t engines) { + // The engines give the same bits on any card of one model, so where they are changes only where they run. + // A card that is merely busy is waited for by the engines' allocations, never left for the CPU. + try { + const int cards = get_gpu_count(); + for (size_t i = 0; i < engines; i++) + engines_.push_back(std::make_unique(*cap_, (device_ + static_cast(i)) % cards)); + set_gpu(device_); // each engine was made on its own card + } catch (const JFJochException &e) { + throw RigidBodyGPUFailure(e.what()); + } + for (auto &e : engines_) + idle_.push_back(e.get()); +} + +std::unique_ptr RigidBodyGPUPool::Create(const gemmi::Model &model, const gemmi::UnitCell &cell, + const gemmi::SpaceGroup &sg, double d_min, + size_t max_observations, size_t max_engines, + Logger &logger) { + std::unique_ptr pool = Plan(model, cell, sg, d_min, max_observations, logger); + if (!pool) + return nullptr; + const size_t bytes = pool->EngineBytes(); + const size_t total = get_gpu_total_memory(pool->device_); + const size_t fit = bytes > 0 ? total / 4 / bytes : 0; + if (fit == 0) { + logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB, over a " + "quarter of the card's {:.0f} MB", bytes / 1e6, total / 1e6); + return nullptr; + } + pool->Reserve(std::min(fit, std::max(max_engines, 1))); + logger.Info("Model validation: the rigid body runs on the GPU, {} engine(s) of {:.0f} MB", pool->Engines(), + bytes / 1e6); + return pool; +} + RigidBodyGPUPool::~RigidBodyGPUPool() = default; std::shared_ptr RigidBodyGPUPool::ObservedZone( diff --git a/image_analysis/structure_refinement/RigidBodyGPU.h b/image_analysis/structure_refinement/RigidBodyGPU.h index 628e8bc39..072c736dc 100644 --- a/image_analysis/structure_refinement/RigidBodyGPU.h +++ b/image_analysis/structure_refinement/RigidBodyGPU.h @@ -23,6 +23,7 @@ class Logger; class RigidBodyGPUEngine; struct RigidBodyGPUZone; +struct RigidBodyGPUCapacity; // A CUDA failure inside the rigid body, or in model validation's structure factors and maps // (ModelStructureFactorsGPU). Model validation catches it and reports the validation as failed. @@ -39,23 +40,27 @@ public: // radii, which the placement does not change - is worked out once per zone and kept. class RigidBodyGPUPool { public: - // Null where there is no GPU, where not even one engine fits the budget - a quarter of the card's - // total memory - or where the cell is too small for the gather (an atom's box wider than the cell). - // Logged either way. Engines that cannot be set up throw RigidBodyGPUFailure. + // A pool without engines yet, sized for the finest zone of the ladder to d_min: EngineBytes() is + // what each engine will take, so a caller can decide how many first. Null where there is no GPU or + // where the cell is too small for the gather (an atom's box wider than the cell), logged. // `max_observations`: the most working reflections any fit will be given in its finest zone. - // `max_engines`: at most this many, however much memory there is. + static std::unique_ptr Plan(const gemmi::Model &model, const gemmi::UnitCell &cell, + const gemmi::SpaceGroup &sg, double d_min, + size_t max_observations, Logger &logger); + // Plan(), then as many engines as fit a quarter of the card's total memory, at most `max_engines` + // (molecular replacement's rule). Null, logged, where not even one fits. static std::unique_ptr Create(const gemmi::Model &model, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double d_min, size_t max_observations, size_t max_engines, Logger &logger); ~RigidBodyGPUPool(); + size_t EngineBytes() const; + // Reserves `engines` engines, engine i on the i-th card from Device(), so a validation uses every + // card there is. Engines that cannot be set up throw RigidBodyGPUFailure. + void Reserve(size_t engines); size_t Engines() const { return engines_.size(); } // The card the pool was made from, which its first engine is on and the others count on from. int Device() const { return device_; } - // What max_engines engines take, however many fitted, and the memory of the card: the sizes a - // caller decides by whether a second validation may run beside this one. - size_t PlannedBytes() const { return planned_bytes_; } - size_t CardBytes() const { return card_bytes_; } RigidBodyGPUEngine &Acquire(); void Release(RigidBodyGPUEngine &engine); @@ -72,8 +77,7 @@ public: private: RigidBodyGPUPool() = default; int device_ = 0; - size_t planned_bytes_ = 0; - size_t card_bytes_ = 0; + std::unique_ptr cap_; std::vector> engines_; std::vector idle_; std::mutex m_; diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 5fe49147c..2e6f46a35 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -4020,7 +4020,16 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { std::future run; std::promise write; // destroyed before `run`, so a run left waiting on it is released } ahead; + // The merge is done with its engine - nothing below reads it but the finalist ledger's P1 merge, + // which runs on it beside the validation - so the device memory it holds goes back before the + // validation decides, from the card's total, what goes on its GPU. Where the ledger's merge keeps + // it, what it holds is counted against the card by every validation here, the same for each, so + // that one started on the forecast decides as the one it stands in for would. + if (!ledger_ahead.valid()) + rsm.reset(); + const size_t merge_on_card = rsm && rsm->OnGPU() ? rsm->DeviceBytes() : 0; ModelValidationSchedule first_schedule; + first_schedule.gpu_resident_bytes = merge_on_card; first_schedule.on_forecast = [&](const ModelFrameForecast &f) { if (ahead.run.valid() || !config_.reference_data.empty() || !data_sg.has_value() || !f.may_run_beside || f.enantiomorph_candidate @@ -4033,7 +4042,7 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { friedel = experiment_.GetScalingSettings().GetMergeFriedel(), rfree_fraction = experiment_.GetScalingSettings().GetRfreeFraction(), wavelength = experiment_.GetWavelength_A(), - gate = ahead.write.get_future().share()]() mutable { + gate = ahead.write.get_future().share(), beside = f.gpu_bytes]() mutable { if (const int32_t cards = get_gpu_count(); cards > 0) pin_gpu(1 % cards); if (!(ahead.indexing_op == gemmi::Op::identity())) { @@ -4053,6 +4062,8 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { ModelValidationSchedule schedule; schedule.write_gate = gate; schedule.fit_test = false; // the verdict is the first validation's (KeepModelVerdict) + schedule.gpu_resident_bytes = merge_on_card; + schedule.gpu_beside_bytes = beside; ahead.merged = std::move(merged); return ValidateAgainstModel(ahead.merged, cell, config_.model_path, config_.output_prefix, ahead.log, in, /*probe_indexing_ambiguity=*/false, @@ -4072,11 +4083,6 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { // The ledger's P1 merge (started above) is taken before anything below acts on what the // validation decides: relabelling the outcomes and the group is what it must not see happen. // Every such relabelling reaches the P1 merge afterwards, through merge_to_written. - // The merge is done with its engine - nothing below reads it but the finalist ledger's P1 merge, - // which runs on it beside the validation - so the device memory it holds goes back before the - // validation decides, from the card's total, whether its GPU path fits. - if (!ledger_ahead.valid()) - rsm.reset(); ModelValidationResult validation = ValidateAgainstModel(sm.merged, *result.consensus_cell, config_.model_path, config_.output_prefix, logger, data_sg ? &*data_sg : nullptr, @@ -4161,6 +4167,7 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { const auto sg_now = experiment_.GetGemmiSpaceGroup(); ModelValidationSchedule kept_verdict; kept_verdict.fit_test = false; + kept_verdict.gpu_resident_bytes = merge_on_card; remade = ValidateAgainstModel(sm.merged, *result.consensus_cell, config_.model_path, config_.output_prefix, logger, sg_now ? &*sg_now : nullptr, @@ -4204,6 +4211,7 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { return twin_r_free(v) < answer_r_free && (!model_twin_only || v.model_fits); }; ModelValidationSchedule twin_schedule; + twin_schedule.gpu_resident_bytes = merge_on_card; twin_schedule.maps_superseded = [&](const ModelValidationResult &v) { return !maps_from(v); }; phase("Validating against model (twin hypothesis)"); auto twin_validation = ValidateAgainstModel( diff --git a/tests/ModelValidationTest.cpp b/tests/ModelValidationTest.cpp index 2d0c0cbef..59bc85639 100644 --- a/tests/ModelValidationTest.cpp +++ b/tests/ModelValidationTest.cpp @@ -112,6 +112,95 @@ ATOM 2 C . CB GLY A 1 12.000 14.000 16.000 1.00 20.00 // --model used to call gemmi::read_pdb unconditionally, so a deposited model handed over as mmCIF - // which is how the PDB serves coordinates by default - was refused outright. Both formats now go // through the same reader, which decides on the file's content. +namespace { + constexpr size_t GB = 1000000000; + + // A validation's parts, sized as a large-cell case measures: structure factors 1.40 GB to d_min and + // 0.89 GB to the null's limit, rigid-body engines of 1.88 GB (ten wanted), scale fits of 0.25 GB (four). + ModelValidationGPUNeed LargeCellNeed(size_t card) { + ModelValidationGPUNeed need; + need.card = card; + need.structure_factors = 1400000000; + need.null_structure_factors = 890000000; + need.rigid_body = 1882000000; + need.rigid_body_engines = 10; + need.scale_fit = 252500000; + need.scale_fit_engines = 4; + return need; + } +} + +TEST_CASE("ModelValidation_GPUPlanOnCardSizes", "[ModelValidation]") { + size_t last_rb = 0, last_scale = 0; + for (size_t card : {8 * GB, 16 * GB, 24 * GB, 128 * GB}) { + const ModelValidationGPUPlan plan = PlanModelValidationGPU(LargeCellNeed(card)); + CAPTURE(card, plan.rigid_body_engines, plan.scale_fit_engines, plan.bytes); + CHECK(plan.budget == card / 5 * 4); + CHECK(plan.bytes <= plan.budget); + // The parts stay on the GPU on every one of these cards; only the engine counts grow with it. + CHECK(plan.structure_factors); + CHECK(plan.null_structure_factors); + CHECK(plan.rigid_body_engines >= std::max(last_rb, 1)); + CHECK(plan.scale_fit_engines >= std::max(last_scale, 1)); + last_rb = plan.rigid_body_engines; + last_scale = plan.scale_fit_engines; + } + // 8 GB, a 6.40 GB budget: 1.40 + 0.89 + 2 x 1.88 + 1 x 0.25 = 6.31 GB. + const ModelValidationGPUPlan small = PlanModelValidationGPU(LargeCellNeed(8 * GB)); + CHECK(small.rigid_body_engines == 2); + CHECK(small.scale_fit_engines == 1); + CHECK(!small.may_run_beside); + // 128 GB: every engine wanted, and room for a second validation of the same size. + const ModelValidationGPUPlan large = PlanModelValidationGPU(LargeCellNeed(128 * GB)); + CHECK(large.rigid_body_engines == 10); + CHECK(large.scale_fit_engines == 4); + CHECK(large.may_run_beside); +} + +TEST_CASE("ModelValidation_GPUPlanPartThatDoesNotFitGoesToTheCPU", "[ModelValidation]") { + ModelValidationGPUNeed need = LargeCellNeed(8 * GB); + need.structure_factors = 7 * GB; // over the 6.4 GB budget on its own + const ModelValidationGPUPlan plan = PlanModelValidationGPU(need); + CHECK(!plan.structure_factors); + CHECK(plan.null_structure_factors); + CHECK(plan.rigid_body_engines >= 1); + CHECK(plan.scale_fit_engines >= 1); + CHECK(plan.bytes <= plan.budget); + + // A part the GPU cannot grid (0 bytes) is never placed. + need = LargeCellNeed(16 * GB); + need.null_structure_factors = 0; + need.rigid_body = 0; + const ModelValidationGPUPlan no_parts = PlanModelValidationGPU(need); + CHECK(no_parts.structure_factors); + CHECK(!no_parts.null_structure_factors); + CHECK(no_parts.rigid_body_engines == 0); +} + +TEST_CASE("ModelValidation_GPUPlanBesideChangesOnlyEngines", "[ModelValidation]") { + ModelValidationGPUNeed need = LargeCellNeed(24 * GB); + const ModelValidationGPUPlan alone = PlanModelValidationGPU(need); + need.beside = alone.budget; // another validation holding the whole budget + const ModelValidationGPUPlan beside = PlanModelValidationGPU(need); + CHECK(beside.structure_factors == alone.structure_factors); + CHECK(beside.null_structure_factors == alone.null_structure_factors); + CHECK(beside.rigid_body_engines == 1); + CHECK(beside.scale_fit_engines == 1); + CHECK(alone.rigid_body_engines > 1); + + // What is resident does decide: with most of the card held by a merge, 2.2 GB are left, which take + // the structure factors and the scale fits but not the rigid body or the null's structure factors. + need = LargeCellNeed(24 * GB); + need.resident = 17 * GB; + const ModelValidationGPUPlan full = PlanModelValidationGPU(need); + CHECK(full.budget == 24 * GB / 5 * 4 - 17 * GB); + CHECK(full.bytes <= full.budget); + CHECK(full.structure_factors); + CHECK(full.rigid_body_engines == 0); + CHECK(!full.null_structure_factors); + CHECK(full.scale_fit_engines == 3); +} + TEST_CASE("ModelValidation_ReadsPdbAndMmcif", "[ModelValidation]") { Logger logger("ModelValidation_ReadsPdbAndMmcif");