GPU or CPU decided up front from the card's total memory, never from what is free

- Scaling: the 80%-of-card check runs right after the device ingest's sort and cut, where the
  observation and distinct-hkl counts are first known, instead of after the geometry smoothing;
  same counts, same decision, but a run that goes to the CPU no longer does the device smoothing
  (6.9 s on an 8 GB card for a 33 M observation set) or holds its memory meanwhile.
- Rigid body: engines budgeted on a quarter of the card's total (the validation's plan), not
  min(total/4, free - 1 GB). A busy card no longer sends the rigid body to the CPU (measured on an
  8 GB card: ~80 s and a different fit); engine allocations wait for GPU work beside the run, and
  an engine that cannot be set up fails the validation (RigidBodyGPUFailure) instead of falling
  back to the CPU.
- Beam-centre FFT: GPU when its footprint is at most half of the card's total, otherwise CPU,
  logged; no catch-and-rescore on the CPU after a GPU failure.
- FitPerFrameG reserves its per-frame vector (CPU scaling waited on the heap lock on Windows).
- get_gpu_free_memory(): cudaMemGetInfo, except on an integrated GPU under Linux, where free also
  counts MemAvailable less a tenth of the total. Used only where free memory is waited on
  (cuda_memory_available) or quoted in out-of-memory messages; it decides nothing.
  The two card-filling [CUDAMemHelpers] tests skip on integrated GPUs.
- Out-of-memory messages point at rugnux --no-gpu instead of "the CPU build".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-14 21:28:38 +02:00
1 parent f96ee13629
commit 5de7189643
23 files changed
+143 -79

No files matched your search

@@ -157,15 +157,16 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
cap.terms = max_observations * ops;
const size_t bytes = RigidBodyGPUEngine::DeviceBytes(cap);
size_t free = 0, total = 0;
RigidBodyGPUEngine::MemoryInfo(free, total);
constexpr size_t HEADROOM = 1ull << 30;
const size_t budget = std::min(total / 4, free > HEADROOM ? free - HEADROOM : 0);
// A quarter of the card's TOTAL memory - the validation's plan (StructureFactorsGPU) - and not what
// happens to be free: the same input on the same card always gets the same engines. A card that
// is merely busy is waited for by the engines' allocations, never left for the CPU.
const int device = RigidBodyGPUEngine::CurrentDevice();
const size_t total = get_gpu_total_memory(device);
const size_t budget = total / 4;
const size_t fit = bytes > 0 ? budget / bytes : 0;
const size_t want = std::min<size_t>(fit, std::max<size_t>(max_engines, 1));
// Engine i on the i-th card from the calling thread's, so a validation uses every card there is;
// the engines give the same bits on any card of one model, so this changes only where they run.
const int device = RigidBodyGPUEngine::CurrentDevice();
const int cards = get_gpu_count();
pool->device_ = device;
pool->planned_bytes_ = bytes * std::max<size_t>(max_engines, 1);
@@ -174,8 +175,8 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
pool->engines_.push_back(std::make_unique<RigidBodyGPUEngine>(cap, (device + static_cast<int>(i)) % cards));
set_gpu(device); // each engine was made on its own card
if (pool->engines_.empty()) {
logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB and the "
"budget is {:.0f} MB", bytes / 1e6, budget / 1e6);
logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB, over a "
"quarter of the card's {:.0f} MB", bytes / 1e6, total / 1e6);
return nullptr;
}
for (auto &e : pool->engines_)
@@ -184,11 +185,7 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
bytes / 1e6);
return pool;
} catch (const JFJochException &e) {
logger.Warning("Model validation: the rigid body runs on the CPU - the GPU engines could not be set up ({})",
e.what());
cuda_throw_if_context_lost();
cuda_clear_error();
return nullptr;
throw RigidBodyGPUFailure(e.what());
}
}