GPU or CPU decided up front from the card's total memory, never from what is free
- Scaling: the 80%-of-card check runs right after the device ingest's sort and cut, where the observation and distinct-hkl counts are first known, instead of after the geometry smoothing; same counts, same decision, but a run that goes to the CPU no longer does the device smoothing (6.9 s on an 8 GB card for a 33 M observation set) or holds its memory meanwhile. - Rigid body: engines budgeted on a quarter of the card's total (the validation's plan), not min(total/4, free - 1 GB). A busy card no longer sends the rigid body to the CPU (measured on an 8 GB card: ~80 s and a different fit); engine allocations wait for GPU work beside the run, and an engine that cannot be set up fails the validation (RigidBodyGPUFailure) instead of falling back to the CPU. - Beam-centre FFT: GPU when its footprint is at most half of the card's total, otherwise CPU, logged; no catch-and-rescore on the CPU after a GPU failure. - FitPerFrameG reserves its per-frame vector (CPU scaling waited on the heap lock on Windows). - get_gpu_free_memory(): cudaMemGetInfo, except on an integrated GPU under Linux, where free also counts MemAvailable less a tenth of the total. Used only where free memory is waited on (cuda_memory_available) or quoted in out-of-memory messages; it decides nothing. The two card-filling [CUDAMemHelpers] tests skip on integrated GPUs. - Out-of-memory messages point at rugnux --no-gpu instead of "the CPU build". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f96ee13629
commit
5de7189643
23 files changed
+143
-79
No files matched your search
@@ -157,15 +157,16 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
|
||||
cap.terms = max_observations * ops;
|
||||
const size_t bytes = RigidBodyGPUEngine::DeviceBytes(cap);
|
||||
|
||||
size_t free = 0, total = 0;
|
||||
RigidBodyGPUEngine::MemoryInfo(free, total);
|
||||
constexpr size_t HEADROOM = 1ull << 30;
|
||||
const size_t budget = std::min(total / 4, free > HEADROOM ? free - HEADROOM : 0);
|
||||
// A quarter of the card's TOTAL memory - the validation's plan (StructureFactorsGPU) - and not what
|
||||
// happens to be free: the same input on the same card always gets the same engines. A card that
|
||||
// is merely busy is waited for by the engines' allocations, never left for the CPU.
|
||||
const int device = RigidBodyGPUEngine::CurrentDevice();
|
||||
const size_t total = get_gpu_total_memory(device);
|
||||
const size_t budget = total / 4;
|
||||
const size_t fit = bytes > 0 ? budget / bytes : 0;
|
||||
const size_t want = std::min<size_t>(fit, std::max<size_t>(max_engines, 1));
|
||||
// Engine i on the i-th card from the calling thread's, so a validation uses every card there is;
|
||||
// the engines give the same bits on any card of one model, so this changes only where they run.
|
||||
const int device = RigidBodyGPUEngine::CurrentDevice();
|
||||
const int cards = get_gpu_count();
|
||||
pool->device_ = device;
|
||||
pool->planned_bytes_ = bytes * std::max<size_t>(max_engines, 1);
|
||||
@@ -174,8 +175,8 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
|
||||
pool->engines_.push_back(std::make_unique<RigidBodyGPUEngine>(cap, (device + static_cast<int>(i)) % cards));
|
||||
set_gpu(device); // each engine was made on its own card
|
||||
if (pool->engines_.empty()) {
|
||||
logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB and the "
|
||||
"budget is {:.0f} MB", bytes / 1e6, budget / 1e6);
|
||||
logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB, over a "
|
||||
"quarter of the card's {:.0f} MB", bytes / 1e6, total / 1e6);
|
||||
return nullptr;
|
||||
}
|
||||
for (auto &e : pool->engines_)
|
||||
@@ -184,11 +185,7 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
|
||||
bytes / 1e6);
|
||||
return pool;
|
||||
} catch (const JFJochException &e) {
|
||||
logger.Warning("Model validation: the rigid body runs on the CPU - the GPU engines could not be set up ({})",
|
||||
e.what());
|
||||
cuda_throw_if_context_lost();
|
||||
cuda_clear_error();
|
||||
return nullptr;
|
||||
throw RigidBodyGPUFailure(e.what());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user