MR and model validation: use every card's memory, split MR's functions over the cards

On a multi-GPU host the structure stage ran essentially on one card: the rotation
function's launches all went to the search's card, and the rigid-body engines of
model validation and MR were admitted only within half of that one card's budget,
although RigidBodyGPUPool already placed engines on other cards. On a 4x T4 server
4nwv's MR rotation function took 75.8 s on one card, validation queued its 10 fits
on one engine and MR's rescoring and null got one engine.

- PlanModelValidationGPU / PlanMolecularReplacementGPU: this card is planned
  exactly as on a machine with it alone (GPU-vs-CPU placement unchanged: 80% of the
  total, less what is resident). The rigid-body engines it has no room for go on the
  other cards in turn, each against its own budget (80% of its total, less the
  resident bytes, counted on every card) within half of it. MR also puts a copy of
  the molecular transform on each other card within half of its budget.
- RigidBodyGPUPool::Reserve takes per-card engine counts.
- Validation forwards the most it holds on any card as gpu_bytes to a validation
  beside, and may run one beside only where twice that fits every card's budget.
- MR splits every rotation and translation function into contiguous slices of
  the orientation list over the cards holding a transform copy (and room for the
  call's buffers), one thread per card, gathered back in order. Each orientation
  is scored by itself (one block / one map), nothing is summed across
  orientations, so N cards give the bytes of one.
- ScoreEngines keeps a room per card; score engines always go on the search's
  own card; function buffers are held on the card they run on.

Byte-identical at N=1 (workstation, one RTX 5080): timing benchmark (myob, cytc,
4nwv) and 4nwv, 8tyy, 6oel, 3inp with --model give the same p.hkl, MTZ data,
p_model.pdb, maps and report against rc176. New tests: per-card plans on four
15.6 GB cards (validation 1/3/3/3 engines, MR T+1/T+2/T+2/T+2), and the split
driven over 1-4 virtual cards on one device, bitwise equal to one card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4

tests/ModelValidationTest.cpp

@@ -162,6 +162,43 @@ TEST_CASE("ModelValidation_GPUPlanOnCardSizes", "[ModelValidation]") {
     CHECK(large.may_run_beside);
 }
 
+TEST_CASE("ModelValidation_GPUPlanEnginesOnOtherCards", "[ModelValidation]") {
+    // Four cards of 15.6 GB each: this card cannot take a second rigid-body engine within half of its 12.48 GB
+    // budget (4.42 + 1.88 GB > 6.24 GB), and the other nine go three to a card on the other three.
+    const size_t card = 15600000000;
+    const ModelValidationGPUPlan alone = PlanModelValidationGPU(LargeCellNeed(card));
+    ModelValidationGPUNeed need = LargeCellNeed(card);
+    need.other_cards = {card, card, card};
+    const ModelValidationGPUPlan plan = PlanModelValidationGPU(need);
+    // This card's parts are those of the card alone.
+    CHECK(plan.budget == alone.budget);
+    CHECK(plan.bytes == alone.bytes);
+    CHECK(plan.structure_factors == alone.structure_factors);
+    CHECK(plan.null_structure_factors == alone.null_structure_factors);
+    CHECK(plan.scale_fit_engines == alone.scale_fit_engines);
+    REQUIRE(plan.rigid_body_per_card.size() == 4);
+    CHECK(plan.rigid_body_per_card[0] == alone.rigid_body_engines);
+    CHECK(alone.rigid_body_engines == 1);
+    CHECK(plan.rigid_body_per_card == std::vector<size_t>{1, 3, 3, 3});
+    CHECK(plan.rigid_body_engines == 10);
+    CHECK(plan.most_on_a_card == 3 * need.rigid_body);
+    CHECK(plan.may_run_beside);
+    // Each other card within half of its own budget, less what a validation beside holds there.
+    need.beside = plan.most_on_a_card;
+    const ModelValidationGPUPlan beside = PlanModelValidationGPU(need);
+    for (size_t k = 1; k < 4; k++)
+        CHECK(beside.rigid_body_per_card[k] * need.rigid_body
+              <= std::min(plan.budget / 2, plan.budget - need.beside));
+    // What is resident is counted on every card: a merge holding most of each leaves no room for extra engines.
+    need.beside = 0;
+    need.resident = 8 * GB;
+    const ModelValidationGPUPlan full = PlanModelValidationGPU(need);
+    CHECK(full.rigid_body_per_card == std::vector<size_t>{1, 1, 1, 1});
+    // One card alone: the plan as it was.
+    CHECK(alone.rigid_body_per_card == std::vector<size_t>{1});
+    CHECK(alone.most_on_a_card == alone.bytes);
+}
+
 TEST_CASE("ModelValidation_GPUPlanPartThatDoesNotFitGoesToTheCPU", "[ModelValidation]") {
     ModelValidationGPUNeed need = LargeCellNeed(8 * GB);
     need.structure_factors = 7 * GB;   // over the 6.4 GB budget on its own
@@ -1282,7 +1319,7 @@ namespace {
                                                size_t engines, Logger &logger) {
         auto pool = RigidBodyGPUPool::Plan(model, cell, sg, d_min, observations, logger);
         REQUIRE(pool);
-        pool->Reserve(engines);
+        pool->Reserve({engines});
         return pool;
     }
 }