ModelValidation: use every visible GPU - the null's engines and the second validation

The rigid-body pool put all of its engines on the calling thread's current
card, so a validation used one GPU whatever the machine had. Now, by fixed
rules decided up front and never by momentary free memory:

- RigidBodyGPUPool::Create puts engine i on card (d + i) % count, d being
  the calling thread's card; the pool restores that card afterwards
  (an engine's constructor sets its own) and an engine is released on
  its own card.
- The threads that run the null's replicates are pinned with pin_gpu
  (which also binds them to the card's NUMA node where that is enabled),
  replicate thread t to card (d + 1 + t) % count, and Acquire() hands a
  thread an idle engine on its own card where there is one, any other
  otherwise. With one card this is exactly the previous back-of-the-list
  choice.
- The validation started on the forecast runs on card 1 % count, so with
  two cards or more it is on the other card from the first; the up-front
  memory rule becomes: twice the planned engine bytes within a quarter of
  the cards' total memory taken together.

The engines' kernels are deterministic (no floating-point atomics) and an
engine's result does not depend on which engine it is, so on cards of one
model the numbers are those of one card; what changes is only where the
work runs. On this one-card workstation the multi-card path cannot be
exercised: md5 of p.mtz and the validation outputs are identical to the
base, and the card-count arithmetic was checked by reading only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi

image_analysis/structure_refinement/ModelValidation.cpp

@@ -810,10 +810,11 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
                                   || !(result.change_of_basis_op == gemmi::Op::identity());
     if (schedule.on_forecast) {
         // Two validations at once take twice the engines, and are allowed where twice what this one
-        // asked for fits the share of the card one validation may take (RigidBodyGPUPool::Create).
+        // asked for fits the share of the cards one validation may take (RigidBodyGPUPool::Create).
         bool may_run_beside = true;
         if (rigid_body_pool != nullptr)
-            may_run_beside = 2 * rigid_body_pool->PlannedBytes() <= rigid_body_pool->CardBytes() / 4;
+            may_run_beside = 2 * rigid_body_pool->PlannedBytes()
+                             <= static_cast<size_t>(get_gpu_count()) * rigid_body_pool->CardBytes() / 4;
         schedule.on_forecast({result.change_of_basis_op, indexing.op, result.model_enantiomorph_candidate,
                               may_run_beside});
     }
@@ -953,7 +954,11 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
     std::vector<std::future<void>> replicate_threads;
     if (decision_pending)
         for (size_t t = 0; t < std::min<size_t>(std::max<size_t>(nthreads, 1), NULL_REPLICATES); t++)
-            replicate_threads.push_back(std::async(std::launch::async, [&] {
+            replicate_threads.push_back(std::async(std::launch::async, [&, t] {
+                // Each on a card in turn from the one after the real fit's, which its engines are
+                // spread the same way over (RigidBodyGPUPool::Create).
+                if (rigid_body_pool != nullptr)
+                    pin_gpu((rigid_body_pool->Device() + 1 + static_cast<int>(t)) % get_gpu_count());
                 for (int i = next_replicate++; i < NULL_REPLICATES; i = next_replicate++)
                     run_replicate(i);
             }));