The null - nine random placements of the model, and the real model's side
of the comparison - is scored to a coarser limit (3.5 A, or where the free
set allows), and those structure factors still ran on the CPU, one model
at a time on each replicate thread. Where the d_min structure factors are
on the GPU, a second ModelStructureFactorsGPU engine at the null's limit
is now made, on the same card, before any replicate starts; every
replicate and the real side go through it, so all of them are computed
the same way. It is decided like the first engine, from the card's total
memory: the two together must stay within half of it, else the null stays
on the CPU (logged). The replicates' threads may sit on other cards
(modelpar's 8d6e9d3a4); each call works on the engine's own device. Where
the null is scored to d_min it shares the d_min engine, as before.
Measured (validation time, before -> after; CPU path for reference):
P3_2 1.55 A: 11.2 -> 9.0 s (CPU 17.7)
P2_1 oblique: 5.1 -> 4.5 s (CPU 7.9)
C2 1.11 A: 12.9 -> 13.2 s (CPU 23.3) - its null time is in the
scale fits, which stay on the CPU
p.mtz md5-identical in all four variants per set (before, after, CPU,
-N 8); maps and map MTZ bit-identical after vs -N 8; the null's sigma
moves only in the second decimal (+113.39 -> +113.38, +86.13 -> +86.14).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A CUDA error on the validation's GPU path (rigid body, structure factors,
maps) used to start the whole validation again on the CPU - a migration
part way through, which makes how long a run takes, and in principle what
its rigid bodies converge to, depend on a device fault. It now ends the
validation with ok = false and a logged reason ("the GPU failed during the
validation (...)"), which the run reports as MODEL_NOT_VALIDATED. A
validation that did not finish decides nothing, so the reflection files
are those of a run without a model. Where there is no GPU at all, or the
up-front rule sends the work to the CPU, nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ModelStructureFactorsGPU computes what compute_model_factors() and
map_from_coefficients() compute on the CPU - F_calc from the model's
density (IT92, Refmac-compatible blur, unblurred as prepare_asu_data()
does) and F_mask from the Refmac bulk-solvent mask, both on the
reflections prepare_asu_data(d_min) lists, in its order; and a map from
ASU coefficients on the grid get_size_for_hkl(coef, 0, 3.0) sizes - on a
device. Made once per cell, group, resolution and model, then evaluated
as often as the coordinates change, so refinement or MR can call it in a
loop. The device is an explicit parameter; every call leaves the calling
thread's current device as it found it.
Pieces:
- ModelDensityGPU: the rigid body's deterministic brick gather, moved
out of RigidBodyGPU.cu into a component of its own (ModelMaskGPU's
pattern); the rigid body uses it unchanged. MAX_BRICKS_PER_AXIS 8 ->
16, so fine grids with high-B atoms (lysozyme at 1.2 A, a 0.9 A P1
cell) are no longer refused; existing zones are gridded identically.
- One copy of the content is gridded and the symmetry composed in
reciprocal space (SymmetryComposition), operators applied on the fly;
the mask is ModelMaskGPU (every image of every atom, islands, shrink).
- Maps: gemmi's get_f_phi_on_grid() in ZYX order on the host (the
coefficients written are the same), in-place cuFFT c2r, transposed back
to XYZ on the device. One map at a time, in the engine's buffers.
Decided once, up front, per card, from its TOTAL memory: the engine's
bytes (16 N + cuFFT work + reflections, N the larger of the structure-
factor and map grids) must be at most half the card - the rigid body's
engines take at most a quarter beside it. Otherwise, or where the gather
cannot grid the cell, the CPU path runs, logged with needed vs total.
Anything to a resolution other than d_min (the null's 3.5 A fits) stays
on the CPU, so all replicates and the real model's side of the null are
computed the same way. A CUDA failure takes the existing path: the
validation restarts on the CPU.
Measured, model validation total per run (CPU path -> GPU), 16 GB card:
F432 215 A cubic, 1.30 A, 500^3 grid: 47.7 -> 15.6 s (two validations;
14.3 -> 2.4 and 33.4 -> 13.2, the rest of the second is writing the
three 0.5 GB maps); F_calc + F_mask 5.7 s -> 0.05 s
C2 1.11 A: 23.9 -> 13.6 s; P3_2 1.55 A: 18.2 -> 10.2 s;
P2_1 1.25 A: 13.4 -> 6.5 s; F4_132 328 A: 13.0 -> 5.5 s;
P6_5: 8.8 -> 4.2 s; P4_3 0.97 A: 4.6 -> 2.5 s; P1 0.92 A: 4.2 -> 2.2 s;
small P1: 3.2 -> 1.3 s; P6_1: 8.8 -> 6.2 s; lysozyme: 1.8 -> 1.4 s.
p.mtz md5-identical on all 13 sets. Against the CPU path: FC within
1e-4 of mean |F|, phases of the strong half within 0.003 deg, maps within
1e-4 (2mFo-DFc) and 7e-4 (mFo-DFc) of their rms; every logged R, CC,
FOM, k_sol and anomalous site list identical at the printed precision,
except where a rigid-body commit sat on an exact R-free tie (0.2155 ->
0.2155) and fell the other way (R-work 0.2127 vs 0.2129). The GPU result
is bit-identical run to run and with -N 8 (maps, map MTZ, placed model).
Peak device memory of the engine: 2.5 GB at 500^3 (process total peaked
at 14.4 GB with what the merge still holds).
Tests: ModelStructureFactorsGPU_MatchesCPU (five groups, 3.5 and 1.5 A:
same reflections, F_calc <= 1e-5 of mean |F|, F_mask 2e-7 rms, repeat
bit-identical), ModelStructureFactorsGPU_MapMatchesCPU (<= 5e-6 of rms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
--model validation (battery-only for users) was 18% of the battery's time. Every
number it produces is unchanged to the bit (p.mtz, maps, placed model and every
model-validation line of the report md5/diff-identical on 11 open sets); only
when and where the work runs changes:
- The bulk-solvent grid fit (FitModelScale, most of the CPU time) fits each
solvent pair on a copy of gemmi::Scaling's target that takes
|Fcalc + k_sol exp(-b_sol s^2) Fmask| once per pair instead of at every
solver evaluation; same expressions, same types (new test checks a grid
point against gemmi's own Scaling fit with ==).
- Fcalc density and the solvent mask are made on two threads; the model's
structure factors beside the GPU engine reservation.
- The indexing probe fits the relabellings concurrently.
- The null's replicates run beside the real model's placement (they start
from a snapshot of the model as read); one GPU engine per replicate plus
one for the real fit instead of a cap of 4 (engines are interchangeable
and deterministic).
- The 2mFo-DFc, mFo-DFc and anomalous maps are made and written
concurrently; the placed model is written beside the reflection files.
- A rigid-body zone whose solvent-mask grid needs gemmi's shrink is sent to
the CPU when the engines are reserved (ModelMaskGPU::ShrinkIsNoOp), instead
of failing on the GPU and validating everything again on the CPU - the
same CPU result, without the wasted first attempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pure move. ModelValidation, RigidBodyRefine, RigidBodyGPU, ModelFFT, ModelGrid,
ModelScaling, ModelMaskGPU, ModelScaleGPU and SigmaA - everything that works on an
atomic model - become the JFJochStructureRefinement library, linked by
JFJochImageAnalysis. WriteModel (the placed-model mmCIF/PDB writer) goes to writer/
as its own small JFJochModelWriter target, so JFJochWriter, which a writer-only build
compiles, does not gain a gemmi dependency. Only include paths and CMake lists change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
leonarski_fandClaude Opus 5.5