Merge branch 'modelpar' into gpusf

Both sides kept: modelpar's parallel basis scoring, the second validation
started on a forecast beside the first (write gate, schedule parameter),
the multi-GPU placement and delete-before-rewrite; gpusf's GPU structure
factors, maps and null engine, failure-instead-of-restart, and the merge
engine released before the validation (now just before modelpar's
ValidateAgainstModel call, after the forecast lambda is set up).

Placement: each validation's structure-factor engines (d_min and the
null's) are made on the card of the thread that runs it - the main
thread's for the first validation, card 1 % count for the speculative
second, which pins itself there - so two validations on two cards use
both, as the rigid-body pools do.

One memory rule for both, per card, from total memory, up front: a
validation plans at most half of its card - its structure-factor engines
a quarter together (was half for the d_min engine alone), its rigid-body
engines a quarter (RigidBodyGPUPool) - and the second validation runs
beside the first only where twice the first's plan (rigid-body planned
bytes + the d_min engine, twice it where a null is coming, the null's
engine being no larger) fits half of all cards' memory together (was:
twice the rigid-body plan within a quarter). The card's total is read
once when the engine is made; a CUDA error there fails the validation
like any other.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-09 13:51:41 +02:00
commit 278c488c37
9 files changed
+229 -48

No files matched your search

@@ -163,9 +163,16 @@ std::unique_ptr<RigidBodyGPUPool> RigidBodyGPUPool::Create(const gemmi::Model &m
const size_t budget = std::min(total / 4, free > HEADROOM ? free - HEADROOM : 0);
const size_t fit = bytes > 0 ? budget / bytes : 0;
const size_t want = std::min<size_t>(fit, std::max<size_t>(max_engines, 1));
// Engine i on the i-th card from the calling thread's, so a validation uses every card there is;
// the engines give the same bits on any card of one model, so this changes only where they run.
const int device = RigidBodyGPUEngine::CurrentDevice();
const int cards = get_gpu_count();
pool->device_ = device;
pool->planned_bytes_ = bytes * std::max<size_t>(max_engines, 1);
pool->card_bytes_ = total;
for (size_t i = 0; i < want; i++)
pool->engines_.push_back(std::make_unique<RigidBodyGPUEngine>(cap, device));
pool->engines_.push_back(std::make_unique<RigidBodyGPUEngine>(cap, (device + static_cast<int>(i)) % cards));
set_gpu(device); // each engine was made on its own card
if (pool->engines_.empty()) {
logger.Info("Model validation: the rigid body runs on the CPU - one GPU engine needs {:.0f} MB and the "
"budget is {:.0f} MB", bytes / 1e6, budget / 1e6);
@@ -216,10 +223,19 @@ const RigidBodyGPUZone &RigidBodyGPUPool::Zone(const gemmi::Model &model, const
}
RigidBodyGPUEngine &RigidBodyGPUPool::Acquire() {
// One on the calling thread's own card where one is idle - the threads that drive a validation are
// pinned to the cards in turn - and any other otherwise.
const int device = RigidBodyGPUEngine::CurrentDevice();
std::unique_lock lock(m_);
cv_.wait(lock, [this] { return !idle_.empty(); });
RigidBodyGPUEngine *e = idle_.back();
idle_.pop_back();
size_t pick = idle_.size() - 1;
for (size_t i = idle_.size(); i-- > 0;)
if (idle_[i]->Device() == device) {
pick = i;
break;
}
RigidBodyGPUEngine *e = idle_[pick];
idle_.erase(idle_.begin() + static_cast<std::ptrdiff_t>(pick));
return *e;
}