Model validation: one GPU budget for all its parts, 80% of the card's total
Build Packages / Create release (push) Successful in 22s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Failing after 1m31s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 9m12s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 11m39s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 12m39s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 23m36s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m15s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 15m15s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 14m8s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m36s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 17m13s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 17m33s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 22m58s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 24m10s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 22m10s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 18m23s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 22m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 21m53s
Build Packages / Unit tests (push) Successful in 2h5m52s
Build Packages / Validation Battery (push) Successful in 36m32s
Build Packages / Create release (push) Successful in 22s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Failing after 1m31s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 9m12s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 11m39s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 12m39s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 23m36s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m15s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 15m15s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 14m8s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m36s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 17m13s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 17m33s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 22m58s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 24m10s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 22m10s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 18m23s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 22m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 21m53s
Build Packages / Unit tests (push) Successful in 2h5m52s
Build Packages / Validation Battery (push) Successful in 36m32s
What a validation puts on its GPU was decided part by part on fixed fractions of the card: the structure factors to d_min on a quarter, the null's structure factors on whatever of that quarter the first left, the rigid-body engines on another quarter, the scale fits on an eighth. On an 8 GB card that sent the null's structure factors of a 4.4 GB validation to the CPU (and held the rigid body to one engine) although everything the validation holds fits the card with room to spare. Now PlanModelValidationGPU decides all of it at once, before anything is reserved, from the card's total memory: every engine is reserved up front and kept to the end, so their sum is held against one budget, 80% of the card less what a merge engine still at work holds there (the finalist ledger's P1 merge, RotationScaleMerge::DeviceBytes). Each part takes one engine where it still fits, in the order structure factors to d_min, rigid body, the null's structure factors, scale fits; a part that does not fit runs on the CPU. What is left goes to more rigid-body engines (up to one per null replicate plus the real fit) and then more scale-fit engines, which only change how many fits run at once. A second validation may run beside where twice this one's whole GPU plan fits the cards' budgets; it gets this one's bytes as gpu_beside_bytes, which limit its engine counts and never its placement, so its numbers are those of the validation it stands in for. One log line states the placement, the bytes held and the budget. - RigidBodyGPUPool::Plan sizes the pool without engines (EngineBytes), Reserve(n) makes them; Create (Plan + engines on a quarter of the card) stays for molecular replacement, whose own rules, and the --mr hand check's structure factors (ReadModelAnomalousMap, a quarter), are unchanged. - The null's resolution is worked out before the plan, so its engine is decided and reserved with the others instead of after the indexing probe. Placement changes only on smaller cards: with the logged footprints of the open-arm sets, 4nwv's null structure factors move to the GPU below ~9 GB (and its rigid body at 6 GB), 8tyy's structure factors to d_min below ~8.7 GB; on 16 GB and larger nothing moves. On the RTX 5080 (16 GB), 4nwv, 8tyy, 6oel and 3inp give byte-identical reflection files, maps and models to rc175 HEAD, with more rigid-body engines where they fit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -112,6 +112,95 @@ ATOM 2 C . CB GLY A 1 12.000 14.000 16.000 1.00 20.00
|
||||
// --model used to call gemmi::read_pdb unconditionally, so a deposited model handed over as mmCIF -
|
||||
// which is how the PDB serves coordinates by default - was refused outright. Both formats now go
|
||||
// through the same reader, which decides on the file's content.
|
||||
namespace {
|
||||
constexpr size_t GB = 1000000000;
|
||||
|
||||
// A validation's parts, sized as a large-cell case measures: structure factors 1.40 GB to d_min and
|
||||
// 0.89 GB to the null's limit, rigid-body engines of 1.88 GB (ten wanted), scale fits of 0.25 GB (four).
|
||||
ModelValidationGPUNeed LargeCellNeed(size_t card) {
|
||||
ModelValidationGPUNeed need;
|
||||
need.card = card;
|
||||
need.structure_factors = 1400000000;
|
||||
need.null_structure_factors = 890000000;
|
||||
need.rigid_body = 1882000000;
|
||||
need.rigid_body_engines = 10;
|
||||
need.scale_fit = 252500000;
|
||||
need.scale_fit_engines = 4;
|
||||
return need;
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("ModelValidation_GPUPlanOnCardSizes", "[ModelValidation]") {
|
||||
size_t last_rb = 0, last_scale = 0;
|
||||
for (size_t card : {8 * GB, 16 * GB, 24 * GB, 128 * GB}) {
|
||||
const ModelValidationGPUPlan plan = PlanModelValidationGPU(LargeCellNeed(card));
|
||||
CAPTURE(card, plan.rigid_body_engines, plan.scale_fit_engines, plan.bytes);
|
||||
CHECK(plan.budget == card / 5 * 4);
|
||||
CHECK(plan.bytes <= plan.budget);
|
||||
// The parts stay on the GPU on every one of these cards; only the engine counts grow with it.
|
||||
CHECK(plan.structure_factors);
|
||||
CHECK(plan.null_structure_factors);
|
||||
CHECK(plan.rigid_body_engines >= std::max<size_t>(last_rb, 1));
|
||||
CHECK(plan.scale_fit_engines >= std::max<size_t>(last_scale, 1));
|
||||
last_rb = plan.rigid_body_engines;
|
||||
last_scale = plan.scale_fit_engines;
|
||||
}
|
||||
// 8 GB, a 6.40 GB budget: 1.40 + 0.89 + 2 x 1.88 + 1 x 0.25 = 6.31 GB.
|
||||
const ModelValidationGPUPlan small = PlanModelValidationGPU(LargeCellNeed(8 * GB));
|
||||
CHECK(small.rigid_body_engines == 2);
|
||||
CHECK(small.scale_fit_engines == 1);
|
||||
CHECK(!small.may_run_beside);
|
||||
// 128 GB: every engine wanted, and room for a second validation of the same size.
|
||||
const ModelValidationGPUPlan large = PlanModelValidationGPU(LargeCellNeed(128 * GB));
|
||||
CHECK(large.rigid_body_engines == 10);
|
||||
CHECK(large.scale_fit_engines == 4);
|
||||
CHECK(large.may_run_beside);
|
||||
}
|
||||
|
||||
TEST_CASE("ModelValidation_GPUPlanPartThatDoesNotFitGoesToTheCPU", "[ModelValidation]") {
|
||||
ModelValidationGPUNeed need = LargeCellNeed(8 * GB);
|
||||
need.structure_factors = 7 * GB; // over the 6.4 GB budget on its own
|
||||
const ModelValidationGPUPlan plan = PlanModelValidationGPU(need);
|
||||
CHECK(!plan.structure_factors);
|
||||
CHECK(plan.null_structure_factors);
|
||||
CHECK(plan.rigid_body_engines >= 1);
|
||||
CHECK(plan.scale_fit_engines >= 1);
|
||||
CHECK(plan.bytes <= plan.budget);
|
||||
|
||||
// A part the GPU cannot grid (0 bytes) is never placed.
|
||||
need = LargeCellNeed(16 * GB);
|
||||
need.null_structure_factors = 0;
|
||||
need.rigid_body = 0;
|
||||
const ModelValidationGPUPlan no_parts = PlanModelValidationGPU(need);
|
||||
CHECK(no_parts.structure_factors);
|
||||
CHECK(!no_parts.null_structure_factors);
|
||||
CHECK(no_parts.rigid_body_engines == 0);
|
||||
}
|
||||
|
||||
TEST_CASE("ModelValidation_GPUPlanBesideChangesOnlyEngines", "[ModelValidation]") {
|
||||
ModelValidationGPUNeed need = LargeCellNeed(24 * GB);
|
||||
const ModelValidationGPUPlan alone = PlanModelValidationGPU(need);
|
||||
need.beside = alone.budget; // another validation holding the whole budget
|
||||
const ModelValidationGPUPlan beside = PlanModelValidationGPU(need);
|
||||
CHECK(beside.structure_factors == alone.structure_factors);
|
||||
CHECK(beside.null_structure_factors == alone.null_structure_factors);
|
||||
CHECK(beside.rigid_body_engines == 1);
|
||||
CHECK(beside.scale_fit_engines == 1);
|
||||
CHECK(alone.rigid_body_engines > 1);
|
||||
|
||||
// What is resident does decide: with most of the card held by a merge, 2.2 GB are left, which take
|
||||
// the structure factors and the scale fits but not the rigid body or the null's structure factors.
|
||||
need = LargeCellNeed(24 * GB);
|
||||
need.resident = 17 * GB;
|
||||
const ModelValidationGPUPlan full = PlanModelValidationGPU(need);
|
||||
CHECK(full.budget == 24 * GB / 5 * 4 - 17 * GB);
|
||||
CHECK(full.bytes <= full.budget);
|
||||
CHECK(full.structure_factors);
|
||||
CHECK(full.rigid_body_engines == 0);
|
||||
CHECK(!full.null_structure_factors);
|
||||
CHECK(full.scale_fit_engines == 3);
|
||||
}
|
||||
|
||||
TEST_CASE("ModelValidation_ReadsPdbAndMmcif", "[ModelValidation]") {
|
||||
Logger logger("ModelValidation_ReadsPdbAndMmcif");
|
||||
|
||||
|
||||
Reference in New Issue
Block a user