Merge branch 'modelpar' into gpusf
Both sides kept: modelpar's parallel basis scoring, the second validation started on a forecast beside the first (write gate, schedule parameter), the multi-GPU placement and delete-before-rewrite; gpusf's GPU structure factors, maps and null engine, failure-instead-of-restart, and the merge engine released before the validation (now just before modelpar's ValidateAgainstModel call, after the forecast lambda is set up). Placement: each validation's structure-factor engines (d_min and the null's) are made on the card of the thread that runs it - the main thread's for the first validation, card 1 % count for the speculative second, which pins itself there - so two validations on two cards use both, as the rigid-body pools do. One memory rule for both, per card, from total memory, up front: a validation plans at most half of its card - its structure-factor engines a quarter together (was half for the d_min engine alone), its rigid-body engines a quarter (RigidBodyGPUPool) - and the second validation runs beside the first only where twice the first's plan (rigid-body planned bytes + the d_min engine, twice it where a null is coming, the null's engine being no larger) fits half of all cards' memory together (was: twice the rigid-body plan within a quarter). The card's total is read once when the engine is made; a CUDA error there fails the validation like any other. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This commit is contained in:
commit
278c488c37
9 files changed
+229
-48
No files matched your search
@@ -163,9 +163,16 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
|
||||
const size_t budget = std::min(total / 4, free > HEADROOM ? free - HEADROOM : 0);
|
||||
const size_t fit = bytes > 0 ? budget / bytes : 0;
|
||||
const size_t want = std::min<size_t>(fit, std::max<size_t>(max_engines, 1));
|
||||
// Engine i on the i-th card from the calling thread's, so a validation uses every card there is;
|
||||
// the engines give the same bits on any card of one model, so this changes only where they run.
|
||||
const int device = RigidBodyGPUEngine::CurrentDevice();
|
||||
const int cards = get_gpu_count();
|
||||
pool->device_ = device;
|
||||
pool->planned_bytes_ = bytes * std::max<size_t>(max_engines, 1);
|
||||
pool->card_bytes_ = total;
|
||||
for (size_t i = 0; i < want; i++)
|
||||
pool->engines_.push_back(std::make_unique<RigidBodyGPUEngine>(cap, device));
|
||||
pool->engines_.push_back(std::make_unique<RigidBodyGPUEngine>(cap, (device + static_cast<int>(i)) % cards));
|
||||
set_gpu(device); // each engine was made on its own card
|
||||
if (pool->engines_.empty()) {
|
||||
logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB and the "
|
||||
"budget is {:.0f} MB", bytes / 1e6, budget / 1e6);
|
||||
@@ -216,10 +223,19 @@ const RigidBodyGPUZone &RigidBodyGPUPool::Zone(const gemmi::Model &model, const
|
||||
}
|
||||
|
||||
RigidBodyGPUEngine &RigidBodyGPUPool::Acquire() {
|
||||
// One on the calling thread's own card where one is idle - the threads that drive a validation are
|
||||
// pinned to the cards in turn - and any other otherwise.
|
||||
const int device = RigidBodyGPUEngine::CurrentDevice();
|
||||
std::unique_lock lock(m_);
|
||||
cv_.wait(lock, [this] { return !idle_.empty(); });
|
||||
RigidBodyGPUEngine *e = idle_.back();
|
||||
idle_.pop_back();
|
||||
size_t pick = idle_.size() - 1;
|
||||
for (size_t i = idle_.size(); i-- > 0;)
|
||||
if (idle_[i]->Device() == device) {
|
||||
pick = i;
|
||||
break;
|
||||
}
|
||||
RigidBodyGPUEngine *e = idle_[pick];
|
||||
idle_.erase(idle_.begin() + static_cast<std::ptrdiff_t>(pick));
|
||||
return *e;
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user