MR and model validation on the CPU: workers bounded by their bytes, not the thread count

- The CPU translation function runs as many rotations at once as the host worker pool holds (8 B per
  grid point each, 550 MiB on a 400 A cell; 33 at once at 32 threads was 18 GiB), at most nthreads.
  Counted, not reserved: the rotations run on ParallelFor's pool, whose workers must not wait on memory.
- Null replicates (model validation's and MR's) whose structure factors are made on the CPU reserve
  32 B per point of their grid (ModelGridPoints) against the pool before they start, so the nine
  replicates, and nulls run side by side, hold at most the pool between them.
Every replicate and rotation computes what it did and is stored by index: no result depends on how
many run at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-17 09:31:05 +02:00
1 parent 64a87cdf52
commit 43e836573d
5 files changed
+48 -3

No files matched your search

@@ -33,6 +33,8 @@
#include "../../common/JFJochMath.h" // PI
#include "../../common/Logger.h"
#include "../../common/CUDAWrapper.h" // get_gpu_count
#include "../../common/HostMemory.h"
#include "../../common/ParallelFor.h"
#include "ModelFFT.h"
#include "ModelGrid.h"
@@ -45,7 +47,6 @@
#include "RigidBodyGPUEngine.h"
#include "ModelStructureFactorsGPU.h"
#include "ModelScaleGPU.h"
#include "../../common/CUDAWrapper.h"
#include "../../common/JFJochException.h"
#include "../indexing/CUDAMemHelpers.h" // cuda_trim_allocation_pool
#endif
@@ -766,11 +767,18 @@ double null_sigma(const gemmi::Structure &placed, int copies, const gemmi::Mat33
where = translate(turned);
}
std::vector<double> r_work(NULL_REPLICATES);
// Without a GPU each replicate's structure factors are made on the CPU, a gigabyte or more of grids on a
// large cell, so it reserves them against the host's worker pool (HostWorkerPool) first: as many run at
// once as it holds - with the nulls of other candidates beside it - and each computes the same either way.
const size_t replicate_bytes = get_gpu_count() == 0
? MODEL_FACTORS_BYTES_PER_GRID_POINT * ModelGridPoints(placed.cell, placed.find_spacegroup(), null_d_min)
: 0;
std::atomic<int> next{0};
std::vector<std::future<void>> workers;
for (size_t t = 0; t < std::min<size_t>(std::max<size_t>(nthreads, 1), NULL_REPLICATES); t++)
workers.push_back(std::async(std::launch::async, [&] {
for (int i = next++; i < NULL_REPLICATES; i = next++) {
const HostWorkerPool::Reservation reserved(HostWorkerPool::Instance(), replicate_bytes);
gemmi::Structure rep = placed;
std::vector<gemmi::Chain> &chains = rep.models[0].chains;
for (int k = 0; k < copies; k++) {
@@ -1561,7 +1569,16 @@ std::vector<TranslationPeak> TranslationFunction(const MolecularTransform &ft, c
const std::vector<std::complex<double>> *known, size_t nthreads) {
const TranslationGather gather = make_gather(setup, known != nullptr);
std::vector<TranslationPeak> out(rotations.size());
ParallelFor(static_cast<int>(rotations.size()), nthreads, [&](int i) {
// A rotation's buffers - the grid's coefficients and the function, its terms' factors - are hundreds of
// megabytes on a large cell, so as many rotations run at once as the host's worker pool holds
// (HostWorkerPool), at most nthreads. Counted here rather than reserved in each: the rotations run on
// ParallelFor's pool, whose workers must not wait on one another's memory.
const size_t nu = setup.nu, nv = setup.nv, nw = setup.nw;
const size_t bytes = nu * nv * (nw / 2 + 1) * sizeof(std::complex<float>) + nu * nv * nw * sizeof(float)
+ setup.Terms() * (setup.ops * sizeof(std::complex<double>) + sizeof(double));
const size_t workers = std::clamp<size_t>(HostWorkerPool::Instance().Size() / std::max<size_t>(bytes, 1), 1,
std::max<size_t>(nthreads, 1));
ParallelFor(static_cast<int>(rotations.size()), workers, [&](int i) {
out[i] = translation_function(ft, setup, gather, rotations[i], known);
});
return out;