Model validation: one GPU budget for all its parts, 80% of the card's total
Build Packages / Create release (push) Successful in 22s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Failing after 1m31s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 9m12s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 11m39s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 12m39s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 23m36s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m15s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 15m15s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 14m8s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m36s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 17m13s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 17m33s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 22m58s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 24m10s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 22m10s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 18m23s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 22m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 21m53s
Build Packages / Unit tests (push) Successful in 2h5m52s
Build Packages / Validation Battery (push) Successful in 36m32s

What a validation puts on its GPU was decided part by part on fixed fractions of the card: the
structure factors to d_min on a quarter, the null's structure factors on whatever of that quarter
the first left, the rigid-body engines on another quarter, the scale fits on an eighth. On an 8 GB
card that sent the null's structure factors of a 4.4 GB validation to the CPU (and held the rigid
body to one engine) although everything the validation holds fits the card with room to spare.

Now PlanModelValidationGPU decides all of it at once, before anything is reserved, from the card's
total memory: every engine is reserved up front and kept to the end, so their sum is held against
one budget, 80% of the card less what a merge engine still at work holds there (the finalist
ledger's P1 merge, RotationScaleMerge::DeviceBytes). Each part takes one engine where it still fits,
in the order structure factors to d_min, rigid body, the null's structure factors, scale fits; a part
that does not fit runs on the CPU. What is left goes to more rigid-body engines (up to one per null
replicate plus the real fit) and then more scale-fit engines, which only change how many fits run at
once. A second validation may run beside where twice this one's whole GPU plan fits the cards'
budgets; it gets this one's bytes as gpu_beside_bytes, which limit its engine counts and never its
placement, so its numbers are those of the validation it stands in for. One log line states the
placement, the bytes held and the budget.

- RigidBodyGPUPool::Plan sizes the pool without engines (EngineBytes), Reserve(n) makes them; Create
  (Plan + engines on a quarter of the card) stays for molecular replacement, whose own rules, and the
  --mr hand check's structure factors (ReadModelAnomalousMap, a quarter), are unchanged.
- The null's resolution is worked out before the plan, so its engine is decided and reserved with
  the others instead of after the indexing probe.

Placement changes only on smaller cards: with the logged footprints of the open-arm sets, 4nwv's
null structure factors move to the GPU below ~9 GB (and its rigid body at 6 GB), 8tyy's structure
factors to d_min below ~8.7 GB; on 16 GB and larger nothing moves. On the RTX 5080 (16 GB), 4nwv,
8tyy, 6oel and 3inp give byte-identical reflection files, maps and models to rc175 HEAD, with more
rigid-body engines where they fit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 08:05:50 +02:00
co-authored by Claude Opus 5.5
parent 228f0fc00f
commit 0d25d765cf
7 changed files with 403 additions and 196 deletions
@@ -491,71 +491,39 @@ void read_anomalous_map(const gemmi::Grid<float> &grid, double rms, gemmi::Struc
}
#ifdef JFJOCH_USE_CUDA
// The model's structure factors to the data's resolution, and the maps, on the GPU - or not, decided here,
// once, before any of them is computed, and never revisited: nothing is moved to the CPU part way through.
// Decided on what the GPU path needs against the card's TOTAL memory and not against what happens to be
// free at the moment, so the same input on the same machine always takes the same path, whatever else is
// running beside it. A validation plans at most half of its card: its structure-factor engines a quarter
// together - `reserved` is what one of them already holds there - and its rigid-body engines a quarter
// (RigidBodyGPUPool::Create). A second validation runs beside it only where twice that plan fits half of
// the cards together (see on_forecast below), so validations never plan more than half of the memory.
std::unique_ptr<ModelStructureFactorsGPU> StructureFactorsGPU(int device, const gemmi::Model &model,
const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg,
double d_min, size_t reserved, Logger &logger) {
// An engine for the model's structure factors to d_min (and the maps) on `device`, nothing reserved yet;
// null, logged, where the cell is too small for the GPU's gridding.
std::unique_ptr<ModelStructureFactorsGPU> StructureFactorsEngine(int device, const gemmi::Model &model,
const gemmi::UnitCell &cell,
const gemmi::SpaceGroup &sg, double d_min,
Logger &logger) {
std::unique_ptr<ModelStructureFactorsGPU> sf;
try {
sf = std::make_unique<ModelStructureFactorsGPU>(device, model, cell, sg, d_min);
} catch (const JFJochException &e) {
throw RigidBodyGPUFailure(e.what());
}
const std::array<int, 3> n = sf->GridSize();
if (!sf->Supported()) {
logger.Info("Model validation: structure factors to {:.2f} A on the CPU - the cell is too small for the "
"GPU's gridding", d_min);
return nullptr;
}
const size_t total = sf->DeviceTotalMemory();
if (reserved + sf->DeviceBytes() > total / 4) {
logger.Info("Model validation: structure factors to {:.2f} A on the CPU - on a {}x{}x{} grid they need "
"{:.2f} GB of GPU memory beside {:.2f} GB already reserved, over a quarter of the card's {:.2f} GB",
d_min, n[0], n[1], n[2], sf->DeviceBytes() / 1e9, reserved / 1e9, total / 1e9);
return nullptr;
}
try {
sf->Reserve();
} catch (const JFJochException &e) {
throw RigidBodyGPUFailure(e.what());
}
logger.Info("Model validation: structure factors to {:.2f} A on the GPU - a {}x{}x{} grid, {:.2f} GB of the "
"card's {:.2f} GB", d_min, n[0], n[1], n[2], sf->DeviceBytes() / 1e9, total / 1e9);
return sf;
}
// The scale fits on the GPU (ModelScaleGPUPool) - or not, decided here, once, on the card's total memory as
// the structure factors are: SCALE_GPU_ENGINES engines, fewer where they would take over an eighth of the
// card, none where not even one fits.
constexpr size_t SCALE_GPU_ENGINES = 4;
std::unique_ptr<ModelScaleGPUPool> ScaleFitsGPU(int device, size_t max_points, Logger &logger) {
const size_t total = get_gpu_total_memory(device);
size_t engines = SCALE_GPU_ENGINES;
while (engines > 0 && ModelScaleGPUPool::DeviceBytes(max_points, engines) > total / 8)
engines--;
if (engines == 0) {
logger.Info("Model validation: scale fits on the CPU - one fit of {} reflections needs {:.2f} GB of GPU "
"memory, over an eighth of the card's {:.2f} GB", max_points,
ModelScaleGPUPool::DeviceBytes(max_points, 1) / 1e9, total / 1e9);
return nullptr;
}
std::unique_ptr<ModelScaleGPUPool> pool;
void Reserve(ModelStructureFactorsGPU &sf) {
try {
pool = std::make_unique<ModelScaleGPUPool>(device, max_points, engines);
sf.Reserve();
} catch (const JFJochException &e) {
throw RigidBodyGPUFailure(e.what());
}
logger.Info("Model validation: scale fits on the GPU, {} engine(s), {:.2f} GB of the card's {:.2f} GB",
engines, pool->Bytes() / 1e9, total / 1e9);
return pool;
}
// The validation's scale fits can use this many engines at once.
constexpr size_t SCALE_GPU_ENGINES = 4;
std::string GBytes(size_t bytes) {
return fmt::format("{:.2f} GB", bytes / 1e9);
}
#endif
@@ -780,22 +748,130 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
logger.Info("Model validation: {} atoms, cell a={:.2f} b={:.2f} c={:.2f}, sg {}, to {:.2f} A",
gemmi::count_atom_sites(st.models[0]), ucell.a, ucell.b, ucell.c, sg->hm, d_min);
// The structure factors to d_min and the maps on the GPU, where `rigid_body_gpu` allows it and the
// card can take them; decided here, before any is computed. The null's, to its coarser limit, get an
// engine of their own further down, decided the same way before any replicate starts - for all of
// them and the real model's side of the null together, so that every one is computed the same way.
// Whether the model is asked to fit at all (see "does the model fit at all?" below), and the resolution
// its null is scored to, which the null's structure-factor engine is sized for.
const bool decision_pending = schedule.fit_test;
double null_d_min = d_min;
if (decision_pending) {
// The null and the side of the real fit it is compared with are both scored to NULL_D_MIN,
// not to the data's resolution. The placement already stops there (RigidBodyRefine's ladder
// ends at 3.5 A), and whether a model is in the right orientation, or the data in its
// indexing, is decided at low resolution: the finer shells only add reflections to the scale
// fits, which were most of the null's cost. Both sides go through the same fits on the same
// reflections, so the comparison stays like for like. On a small cell 3.5 A leaves too few
// free reflections for a real indexing margin to be told from a random placement's, so the
// limit then moves out to where NULL_MIN_FREE of them are - to the data's own on the
// smallest, which are the cheap ones.
std::vector<float> free_d;
for (const MergedReflection &r : obs)
if (r.rfree_flag && !std::isnan(r.F))
free_d.push_back(r.d);
std::sort(free_d.begin(), free_d.end(), std::greater<>());
null_d_min = std::max(d_min, NULL_D_MIN);
if (free_d.size() < NULL_MIN_FREE)
null_d_min = d_min;
else
null_d_min = std::min<double>(null_d_min, free_d[NULL_MIN_FREE - 1]);
}
// What runs on the GPU, where `rigid_body_gpu` allows it and there is a card: the structure factors to
// d_min and the maps, the null's structure factors to its coarser limit (where that is not d_min), the
// rigid body and the scale fits - all decided here, at once, by PlanModelValidationGPU, before any of
// them is computed, and never revisited: nothing is moved to the CPU part way through. Every engine is
// reserved now and kept to the end, the real fit and every replicate of the null on the same ones, so
// that a verdict never depends on how full the card was when a replicate started. One card for the
// structure factors and the scale fits, the current one; the rigid body's engines count on from it.
ModelStructureFactorsGPU *sf_gpu = nullptr, *sf_null = nullptr;
RigidBodyGPUPool *rigid_body_pool = nullptr;
size_t gpu_bytes = 0;
bool may_run_beside = true;
#ifdef JFJOCH_USE_CUDA
std::unique_ptr<ModelStructureFactorsGPU> sf_engine, sf_null_engine;
// On the current device, as the rigid body's engines take it: one card for the whole validation.
if (rigid_body_gpu && get_gpu_count() > 0)
sf_engine = StructureFactorsGPU(RigidBodyGPUEngine::CurrentDevice(), st.models[0], ucell, *sg, d_min, 0,
logger);
sf_gpu = sf_engine.get();
// The scale fits beside them, on the same card, for as many reflections as the data have.
std::unique_ptr<RigidBodyGPUPool> rigid_body_engines;
std::unique_ptr<ModelScaleGPUPool> scale_gpu;
if (rigid_body_gpu && get_gpu_count() > 0)
scale_gpu = ScaleFitsGPU(RigidBodyGPUEngine::CurrentDevice(), obs.size(), logger);
ModelValidationGPUPlan gpu_plan;
if (rigid_body_gpu && get_gpu_count() > 0) {
const int device = RigidBodyGPUEngine::CurrentDevice();
ModelValidationGPUNeed need;
need.card = get_gpu_total_memory(device);
need.resident = schedule.gpu_resident_bytes;
need.beside = schedule.gpu_beside_bytes;
sf_engine = StructureFactorsEngine(device, st.models[0], ucell, *sg, d_min, logger);
if (sf_engine)
need.structure_factors = sf_engine->DeviceBytes();
if (null_d_min != d_min) {
sf_null_engine = StructureFactorsEngine(device, st.models[0], ucell, *sg, null_d_min, logger);
if (sf_null_engine)
need.null_structure_factors = sf_null_engine->DeviceBytes();
}
// The rigid body's engines are sized for the most observations a fit gives its finest zone,
// counted as the zones count them: by the resolution of the index once it is in the model's
// ASU, under each indexing the data may be placed in. The model's group need not keep the
// data's metric - a wrong model's cubic group on an orthorhombic cell sends h to an index of
// another resolution - so the data's own d can put a reflection on the wrong side of the zone.
const double finest_zone = RigidBodyLadder(d_min).back();
const gemmi::GroupOps zone_gops = sg->operations();
const gemmi::ReciprocalAsu zone_asu(sg);
auto in_finest_zone = [&](const std::vector<MergedReflection> &v) {
size_t n = 0;
for (const MergedReflection &r : v)
if (!std::isnan(r.F) && ucell.calculate_d(zone_asu.to_asu({r.h, r.k, r.l}, zone_gops).first) >= finest_zone)
++n;
return n;
};
size_t zone_observations = in_finest_zone(obs);
for (const gemmi::Op &law : reindex_ops)
zone_observations = std::max(zone_observations, in_finest_zone(ReindexReflections(obs, law)));
rigid_body_engines = RigidBodyGPUPool::Plan(st.models[0], ucell, *sg, d_min, zone_observations, logger);
if (rigid_body_engines) {
need.rigid_body = rigid_body_engines->EngineBytes();
// One for the real model and one for each replicate of the null placed beside it.
need.rigid_body_engines = std::min<size_t>(std::max<size_t>(nthreads, 1), NULL_REPLICATES) + 1;
}
// For as many reflections as the data have.
need.scale_fit = ModelScaleGPUPool::DeviceBytes(obs.size(), 1);
need.scale_fit_engines = SCALE_GPU_ENGINES;
gpu_plan = PlanModelValidationGPU(need, static_cast<size_t>(get_gpu_count()));
gpu_bytes = gpu_plan.bytes;
may_run_beside = gpu_plan.may_run_beside;
std::string parts;
auto part = [&](const std::string &what, bool on_gpu, size_t bytes, size_t engines) {
if (bytes == 0)
return; // no such part, or one the GPU cannot grid (logged where that was found)
parts += fmt::format("{}{} on the ", parts.empty() ? "" : ", ", what);
if (!on_gpu)
parts += fmt::format("CPU ({} do not fit)", GBytes(bytes));
else if (engines > 0)
parts += fmt::format("GPU ({} x {})", engines, GBytes(bytes));
else
parts += fmt::format("GPU ({})", GBytes(bytes));
};
part(fmt::format("structure factors to {:.2f} A", d_min), gpu_plan.structure_factors,
need.structure_factors, 0);
part(fmt::format("the null's to {:.2f} A", null_d_min), gpu_plan.null_structure_factors,
need.null_structure_factors, 0);
part("rigid body", gpu_plan.rigid_body_engines > 0, need.rigid_body, gpu_plan.rigid_body_engines);
part("scale fits", gpu_plan.scale_fit_engines > 0, need.scale_fit, gpu_plan.scale_fit_engines);
logger.Info("Model validation: {} - {} held of a {} budget, 80% of the card's {}{}{}", parts,
GBytes(gpu_plan.bytes), GBytes(gpu_plan.budget), GBytes(need.card),
need.resident > 0 ? fmt::format(" less {} held by the merge", GBytes(need.resident)) : "",
need.beside > 0 ? fmt::format("; {} held by a validation beside", GBytes(need.beside)) : "");
if (!gpu_plan.structure_factors)
sf_engine.reset();
if (!gpu_plan.null_structure_factors)
sf_null_engine.reset();
if (sf_engine)
Reserve(*sf_engine);
if (sf_null_engine)
Reserve(*sf_null_engine);
}
sf_gpu = sf_engine.get();
sf_null = sf_null_engine.get();
#else
(void) rigid_body_gpu;
#endif
// --- Fcalc (atomic) via electron density on a grid + FFT, plus a flat bulk-solvent mask -> Fmask.
@@ -840,42 +916,23 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
.prepare_asu_data(dc.d_min, dc.blur, false, false, false);
ms.fmask = fmask.get();
};
// The model's structure factors are made beside the reservation of the GPU engines below, which
// does not need them.
// The model's structure factors are made beside the reservation of the rigid body's and the scale
// fits' engines below, which does not need them.
std::future<void> model_factors = std::async(std::launch::async, [&] { compute_model_factors(mdl, d_min); });
// The rigid body's device, decided once for the whole validation - the real fit and every replicate
// of the null on the same one, so that a verdict never depends on how full the card was when a
// replicate started. The engines are reserved here, up front, and never grow.
RigidBodyGPUPool *rigid_body_pool = nullptr;
#ifdef JFJOCH_USE_CUDA
std::unique_ptr<RigidBodyGPUPool> rigid_body_engines;
if (rigid_body_gpu) {
// Counted as the zones count them: by the resolution of the index once it is in the model's
// ASU, under each indexing the data may be placed in. The model's group need not keep the
// data's metric - a wrong model's cubic group on an orthorhombic cell sends h to an index of
// another resolution - so the data's own d can put a reflection on the wrong side of the zone.
const double finest_zone = RigidBodyLadder(d_min).back();
const gemmi::GroupOps zone_gops = sg->operations();
const gemmi::ReciprocalAsu zone_asu(sg);
auto in_finest_zone = [&](const std::vector<MergedReflection> &v) {
size_t n = 0;
for (const MergedReflection &r : v)
if (!std::isnan(r.F) && ucell.calculate_d(zone_asu.to_asu({r.h, r.k, r.l}, zone_gops).first) >= finest_zone)
++n;
return n;
};
size_t zone_observations = in_finest_zone(obs);
for (const gemmi::Op &law : reindex_ops)
zone_observations = std::max(zone_observations, in_finest_zone(ReindexReflections(obs, law)));
// One engine for the real model and one for each replicate of the null placed beside it.
rigid_body_engines = RigidBodyGPUPool::Create(st.models[0], ucell, *sg, d_min, zone_observations,
std::min<size_t>(std::max<size_t>(nthreads, 1), NULL_REPLICATES) + 1,
logger);
if (rigid_body_engines && gpu_plan.rigid_body_engines > 0) {
rigid_body_engines->Reserve(gpu_plan.rigid_body_engines);
rigid_body_pool = rigid_body_engines.get();
}
#else
(void) rigid_body_gpu;
if (gpu_plan.scale_fit_engines > 0) {
try {
scale_gpu = std::make_unique<ModelScaleGPUPool>(RigidBodyGPUEngine::CurrentDevice(), obs.size(),
gpu_plan.scale_fit_engines);
} catch (const JFJochException &e) {
throw RigidBodyGPUFailure(e.what());
}
}
#endif
model_factors.get();
@@ -1092,36 +1149,13 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
// not place is searched for by molecular replacement instead (Rugnux). Only a validation whose
// verdict is taken from another one (KeepModelVerdict), or that is not wanted (a placement already
// judged elsewhere, scored for its maps), skips it.
const bool decision_pending = schedule.fit_test;
if (schedule.on_forecast) {
// Two validations at once take twice the engines, and are allowed where twice what this one plans -
// its rigid-body engines, its structure-factor engine and, where a null is coming, the null's,
// which is no larger - fits half of the cards' memory taken together (see StructureFactorsGPU).
bool may_run_beside = true;
#ifdef JFJOCH_USE_CUDA
size_t planned = 0, card = 0;
if (rigid_body_pool != nullptr) {
planned += rigid_body_pool->PlannedBytes();
card = rigid_body_pool->CardBytes();
}
if (sf_gpu != nullptr) {
planned += (decision_pending ? 2 : 1) * sf_gpu->DeviceBytes();
card = sf_gpu->DeviceTotalMemory();
}
if (scale_gpu) {
planned += scale_gpu->Bytes();
card = get_gpu_total_memory(RigidBodyGPUEngine::CurrentDevice());
}
if (planned > 0)
may_run_beside = 2 * planned <= static_cast<size_t>(get_gpu_count()) * card / 2;
#endif
// A second validation is allowed beside this one where PlanModelValidationGPU says it fits.
if (schedule.on_forecast)
schedule.on_forecast({result.change_of_basis_op, indexing.op, result.model_enantiomorph_candidate,
may_run_beside});
}
may_run_beside, gpu_bytes});
std::vector<gemmi::Mat33> equivalent; // the orientations the crystal cannot tell from the model's
double reach_deg = 0;
std::vector<gemmi::Mat33> null_rotation;
double null_d_min = d_min;
std::vector<MergedReflection> obs_null;
if (decision_pending) {
// --- the null: this model, this data, in random orientations ---
@@ -1181,36 +1215,10 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
"({:.1f} deg) of an orientation equivalent to the model's, and were drawn again",
redraws, reach_deg);
}
// The null and the side of the real fit it is compared with are both scored to NULL_D_MIN,
// not to the data's resolution. The placement already stops there (RigidBodyRefine's ladder
// ends at 3.5 A), and whether a model is in the right orientation, or the data in its
// indexing, is decided at low resolution: the finer shells only add reflections to the scale
// fits, which were most of the null's cost. Both sides go through the same fits on the same
// reflections, so the comparison stays like for like. On a small cell 3.5 A leaves too few
// free reflections for a real indexing margin to be told from a random placement's, so the
// limit then moves out to where NULL_MIN_FREE of them are - to the data's own on the
// smallest, which are the cheap ones.
std::vector<float> free_d;
for (const MergedReflection &r : obs)
if (r.rfree_flag && !std::isnan(r.F))
free_d.push_back(r.d);
std::sort(free_d.begin(), free_d.end(), std::greater<>());
null_d_min = std::max(d_min, NULL_D_MIN);
if (free_d.size() < NULL_MIN_FREE)
null_d_min = d_min;
else
null_d_min = std::min<double>(null_d_min, free_d[NULL_MIN_FREE - 1]);
// Scored to null_d_min (above), on its own structure-factor engine where that is not d_min.
for (const MergedReflection &r : obs)
if (r.d >= null_d_min)
obs_null.push_back(r);
#ifdef JFJOCH_USE_CUDA
// On the card the d_min engine is on; the replicates' threads may sit on other cards, and each
// call works on the engine's own device. Where the null is scored to d_min it shares that engine.
if (sf_gpu != nullptr && null_d_min != d_min)
sf_null_engine = StructureFactorsGPU(sf_gpu->Device(), st.models[0], ucell, *sg, null_d_min,
sf_gpu->DeviceBytes(), logger);
sf_null = sf_null_engine.get();
#endif
}
const gemmi::Structure null_model = st; // the model as read, which the replicates are made from
// Each replicate works on its own copy of the model as read and of its structure factors, and the
@@ -1264,7 +1272,7 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
for (size_t t = 0; t < std::min<size_t>(std::max<size_t>(nthreads, 1), NULL_REPLICATES); t++)
replicate_threads.push_back(std::async(std::launch::async, [&, t] {
// Each on a card in turn from the one after the real fit's, which its engines are
// spread the same way over (RigidBodyGPUPool::Create).
// spread the same way over (RigidBodyGPUPool::Reserve).
#ifdef JFJOCH_USE_CUDA
if (rigid_body_pool != nullptr)
pin_gpu((rigid_body_pool->Device() + 1 + static_cast<int>(t)) % get_gpu_count());
@@ -1870,6 +1878,37 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
} // namespace
ModelValidationGPUPlan PlanModelValidationGPU(const ModelValidationGPUNeed &need, size_t cards) {
ModelValidationGPUPlan plan;
const size_t usable = need.card / 5 * 4;
plan.budget = usable > need.resident ? usable - need.resident : 0;
auto take = [&](size_t bytes, size_t limit) {
if (bytes == 0 || plan.bytes + bytes > limit)
return false;
plan.bytes += bytes;
return true;
};
plan.structure_factors = take(need.structure_factors, plan.budget);
plan.rigid_body_engines = need.rigid_body_engines > 0 && take(need.rigid_body, plan.budget) ? 1 : 0;
plan.null_structure_factors = take(need.null_structure_factors, plan.budget);
plan.scale_fit_engines = need.scale_fit_engines > 0 && take(need.scale_fit, plan.budget) ? 1 : 0;
const size_t room = plan.budget > need.beside ? plan.budget - need.beside : 0;
while (plan.rigid_body_engines > 0 && plan.rigid_body_engines < need.rigid_body_engines
&& take(need.rigid_body, room))
plan.rigid_body_engines++;
while (plan.scale_fit_engines > 0 && plan.scale_fit_engines < need.scale_fit_engines
&& take(need.scale_fit, room))
plan.scale_fit_engines++;
const size_t whole = (plan.structure_factors ? need.structure_factors : 0)
+ (plan.null_structure_factors ? need.null_structure_factors : 0)
+ (plan.rigid_body_engines > 0 ? need.rigid_body * need.rigid_body_engines : 0)
+ (plan.scale_fit_engines > 0 ? need.scale_fit * need.scale_fit_engines : 0);
plan.may_run_beside = 2 * whole <= cards * plan.budget;
return plan;
}
ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &merged,
const UnitCell &cell,
const std::string &model_path,
@@ -1973,8 +2012,17 @@ ModelValidationResult ReadModelAnomalousMap(const std::vector<MergedReflection>
ModelStructureFactorsGPU *gpu = nullptr;
#ifdef JFJOCH_USE_CUDA
std::unique_ptr<ModelStructureFactorsGPU> engine;
// Read beside molecular replacement's engines, which keep to a quarter of the card
// (MolecularReplacement.cpp), so on the GPU where it fits a quarter as well.
if (get_gpu_count() > 0)
engine = StructureFactorsGPU(RigidBodyGPUEngine::CurrentDevice(), st.models[0], st.cell, *sg, d_min, 0, logger);
engine = StructureFactorsEngine(RigidBodyGPUEngine::CurrentDevice(), st.models[0], st.cell, *sg, d_min, logger);
if (engine && engine->DeviceBytes() > engine->DeviceTotalMemory() / 4) {
logger.Info("Model validation: structure factors to {:.2f} A on the CPU - {} of GPU memory, over a quarter "
"of the card's {}", d_min, GBytes(engine->DeviceBytes()), GBytes(engine->DeviceTotalMemory()));
engine.reset();
}
if (engine)
Reserve(*engine);
gpu = engine.get();
if (gpu != nullptr) {
try {