4nwv molecular replacement 31.7 -> 20.1 s; outputs identical on myob, cytochrome C and 4nwv on the RTX 5080.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The molecular transform's FFT went to the GPU only where its buffers took at most a quarter of the card
(MapToFPhiGPUFits), the last quarter rule left in molecular replacement. On a large cell that is never:
4nwv's 768^3 box needs 5.45 GB, so on a 16 GB card it ran on the CPU in FFTW, single-threaded - 12.5 s
of the search's 31 s.
Now it is judged, as every other part of the search, against the search's budget (80% of the card less
what is resident; the search holds nothing else yet when it runs), and the device writes only the
cropped transform the search keeps (CropHalfL, moved to ModelFFT), so it needs 4.46 GB instead of 5.45.
cuFFT and FFTW agree to float rounding, not bit for bit, so wherever the transform moves to the GPU the
search's functions can differ in their last bits: on 4nwv on 16 GB and 8.55 GB cards it moves (it now
does on any card above about 5.6 GB, before above 21.8 GB); the small boxes were on the GPU before and
give the same values (the same kernel, cropped).
Measured on the RTX 5080, timing benchmark: 4nwv's transform 14.1 -> 2.2 s, MR 31.6 -> 20.1 s, run
93.0 -> 89.2 s; MR's peak device memory unchanged at 8.7 GB (the FFT's own 5.2 GB comes before anything
else is held). On this card the reflection files, maps, placed models, reports and the MR candidate lines
of myob, cytc and 4nwv still came out byte-identical to rc175-metal-merge. [MolecularReplacement] passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What molecular replacement put on its GPU was decided part by part on a quarter of the card each: every
group's rigid-body pool (up to three engines), every null's (up to nine), the score engines' cache, the
--mr hand check's structure factors. The groups and the nulls run side by side, so the quarters added up:
on 4nwv on a 16 GB card the search held 4 rigid-body engines of 1.88 GB, two score engines of 1.64 GB,
the 1.82 GB molecular transform, two translation functions' 1.2 GB and the 2.4 GB the allocation pool
kept from the validation before - 13.8 GB at its peak, 83% of the card. On an 8.55 GB card that pages.
Now each merge's search is planned once, before it starts, on the budget model validation uses
(ModelGPUBudget: 80% of the card's total, less what a merge engine still at work holds): the transform,
one rigid-body engine and the largest score engine take their place in that order where they still fit,
more rigid-body engines only within half of the budget, and what is left is a room the score engines
(least recently used given up first, as before) and the rotation and translation functions' working
buffers share, each waiting for the other to finish rather than spilling past it
(PlanMolecularReplacementGPU). A part that does not fit runs on the CPU, decided on its size alone.
- One rigid-body pool per merge serves every fit of the merge - the groups' first and further copies, the
nulls, the tNCS shifts, the carried copies - sized for the largest (the most copies Matthews allows, in
any group) and reserved once, instead of a pool made and freed for each. RigidBodyGPUPool::Cover widens
a planned pool to another model or group; its zones are kept by group, atom count and resolution.
RigidBodyGPUPool::Create (the quarter rule) is gone.
- The sizes are worked out on the CPU while the transform is computed.
- The molecular transform keeps only the indices it is read at (|k|, l <= box / d_min + 1): 0.84 GB
instead of 1.82 on 4nwv, host and device. Its rows keep their full length: cut short too, the rotation
function read them 15% slower.
- The translation function's prepared setup is kept on the host and uploaded by each call, so the device
holds it only while a function runs; the cross-term offsets are uploaded only where there are cross terms.
- The allocation pool is trimmed before the search, after the transform's GPU FFT, before a score
engine is made and before the hand check, whose structure factors are now judged against the budget.
- rugnux passes the merge engine's bytes, where the finalist ledger keeps it, to the search, the hand
check and the validation of the placed model.
Measured on the RTX 5080 (16.6 GB), timing benchmark: peak device memory during MR 13.8 -> 8.7 GB on 4nwv
(start 3.0 -> 0.66 GB after the trim), 1.67 -> 1.14 GB on cytc, 6.3 -> 0.76 GB on myob. On 4nwv the plan
holds the transform 0.84 GB, one rigid-body engine of 2.10 GB and leaves 10.35 GB of a 13.29 GB budget
to the score engines and the functions; a second engine would pass half of the budget (one engine
against two: +1.2 s on the rescoring and the null). MR 31.7 -> 31.6 s on 4nwv. Reflection files, maps,
placed models, reports and the MR candidate lines byte-identical to rc175-metal-merge on myob, cytc and
4nwv; [MolecularReplacement], [ModelValidation], RigidBody*, ModelScale*, [CUDAMemHelpers] pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- IndexerThreadPool::ReleaseIndexers(): each worker drops the indexers it has
built (once its current frame is done) and the allocation pool is trimmed, so
their memory goes back to the card, not only to the pool. A later frame builds
them again, as on first use. A rotation pass calls it once its first pass is
done and again when the leftover-lattice census - the last of its indexing -
ends. Measured on the RTX 5080: the 1.24 GB (pre-pass) / 0.62 GB (later
passes) the pass's pool held through the image pass and the scaling is gone;
4nwv's pre-pass image pass peaks 5.97 instead of 7.18 GB of device memory.
- GPUMerge: the main line holds one through ScaleMergeAndSymmetry. The
speculative geometry probe (GPUWorkBeside(after_merges=true)) waits for it on
a card below the take-turns size (12 GB), where the probe's FFT indexers and
image workers beside the merge's arrays would overfill the card - on a
Windows (WDDM) card that does not fail an allocation but pages it and crawls.
The merge waits for nothing, so nothing the merge waits on (the rotation-scale
walk's probes) can wait in a circle with it. No change on a 16 GB card.
Output unchanged: p.hkl byte-identical on cytc_x10sa, myob_x10sa, 4nwv and
8qaw on this card; also identical with the card's total read as 8.55 GB
(10 / 9 image workers instead of 16) and with 4 image workers, so the worker
count does not change the result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The resident ingest reads the reflections twice: first only what the sort
and the cut need (frame position, resolution, background, raw-hkl key),
then - once the GPU/CPU decision has been made on the exact counts - the
observations the cut kept, written straight to their place. The
observation arrays and the sort buffers are never on the card together,
and no observation field is cut (held twice) any more.
- A set too large even for that sort scales on the CPU before anything is
allocated on the card (ScaleOnCPUWhereSortTooLarge, same 80%-of-total rule).
- The predicted detector position never goes to the card: it is taken on the
host, and the combine emits the index of each full's peak partial instead
of its px/py (8 B less per observation and 4 per full).
- A full carries its raw-hkl run index instead of h, k, l (8 B less per
full); the constant-1 partiality array of the fulls is gone.
- The fulls' CSR sorts and the error-model rank sort run on cub::DoubleBuffer.
Measured with per-array device accounting on an RTX 5080, largest engine of
each set, before (c130c7923) -> after, MB:
set obs/hkl ingest peak (B/refl) post-decision peak
4nwv 32.7 M/15.4 M 3277 (100) -> 2336 (72) 4265 -> 3625
8qaw 122.7 M/ 8.6 M 10756 (88) -> 7421 (61) 9327 -> 8156
9v16 54.7 M/ 5.7 M 4838 (89) -> 3419 (63) 4283 -> 3774
7qij 78.1 M/ 2.2 M 6768 (87) -> 4707 (60) 5242 -> 4707
8sa8 50.0 M/ 5.0 M 4107 (82) -> 3134 (63) 4205 -> 3603
The sort before the decision now holds 32 B per reflection (46 on a P1-like
set with many runs); the engine's overall peak is after the decision on all
of them.
GPUScalingPeakBytes refitted on the same fourteen engines: 63 + 115 ->
56 B/obs + 99 B/hkl, within 9%. DeviceBytes as an upper bound: 66/94/52 ->
58/30/86 per obs/event/hkl, ingest 89/56 -> 60/42 per reflection/run.
p.hkl byte-identical to the rc175 baseline on 15 battery sets (4nwv, 8qaw,
9v16, 7qij, 8sa8, 11if, 5reo, 6oel, 9jzo, 9min, metformin, lyso_x06da_ref,
thau_bl1a_3p8keV, insu_I_x06da_5keV_2, myob_x06da); compare shows no change.
[RotationScale] (RotationIngestGPU now also checks the host positions) and
ErrorModel_DeviceFitIsTheHostFit pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the RTX 5080 nothing moves between GPU and CPU and the output is byte-identical; on cards below
about 9-10 GB parts that used to fall back to the CPU now stay on the GPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On an 8.55 GB card (WDDM) the validation planned 6.81 GB of a 6.84 GB budget, but the card already
held more than the 20% headroom allowed, and the driver paged GPU memory to system RAM: every
rigid-body fit ran ~5x slower. Measured on 4nwv on a 16 GB card, at the start of model validation
the device held 1.1 GB beyond a fresh context, and on 6oel 2.7 GB:
- the stream-ordered allocation pool's free-but-reserved memory, left by the image passes and the
scaling's pooled buffers (1.07 GB on 4nwv, 2.1 GB on 6oel); the validation's engines are allocated
outside the pool and cannot reuse it;
- the image pass's FFT indexers (IndexerThreadPool), alive for the whole run (0.8 GB on 4nwv,
0.5 GB on 6oel), although nothing indexes after the merge.
Both are now released: the validation trims the pool (cuda_trim_allocation_pool) before it plans,
and rugnux drops the indexer pool once the leftover-lattice census, its last user, has ended, before
the model is validated or searched for. What is left beside the validation is 0.06 GB over a fresh
context on both sets (streams, idle cuFFT plans).
Extra engines are now added only within half of the budget. They change no number and bought little
where measured (4nwv, 16 GB card: 13.6 s with one rigid-body engine, 13.1 s with two, 12.6 s with
five), so the card keeps headroom instead. On 16 GB 4nwv now plans 2 rigid-body and 2 scale-fit
engines, 6.56 GB (measured peak 7.3 GB with the context); on an 8.55 GB card one of each, 4.42 GB.
Reflection files, maps and models byte-identical to rc175 HEAD on 4nwv, 8tyy, 6oel and 3inp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CUDA engine is untouched: on the RTX 5080 p.hkl is byte-identical to rc175 on 4nwv, lysozyme
and cytochrome C. Under --no-gpu (lysozyme, cytochrome C, myoglobin) intensities move at the ulp
level from the single-precision fit and nothing changes beyond noise in battery compare; wall
44/117/71 s -> 41/105/65 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a validation puts on its GPU was decided part by part on fixed fractions of the card: the
structure factors to d_min on a quarter, the null's structure factors on whatever of that quarter
the first left, the rigid-body engines on another quarter, the scale fits on an eighth. On an 8 GB
card that sent the null's structure factors of a 4.4 GB validation to the CPU (and held the rigid
body to one engine) although everything the validation holds fits the card with room to spare.
Now PlanModelValidationGPU decides all of it at once, before anything is reserved, from the card's
total memory: every engine is reserved up front and kept to the end, so their sum is held against
one budget, 80% of the card less what a merge engine still at work holds there (the finalist
ledger's P1 merge, RotationScaleMerge::DeviceBytes). Each part takes one engine where it still fits,
in the order structure factors to d_min, rigid body, the null's structure factors, scale fits; a part
that does not fit runs on the CPU. What is left goes to more rigid-body engines (up to one per null
replicate plus the real fit) and then more scale-fit engines, which only change how many fits run at
once. A second validation may run beside where twice this one's whole GPU plan fits the cards'
budgets; it gets this one's bytes as gpu_beside_bytes, which limit its engine counts and never its
placement, so its numbers are those of the validation it stands in for. One log line states the
placement, the bytes held and the budget.
- RigidBodyGPUPool::Plan sizes the pool without engines (EngineBytes), Reserve(n) makes them; Create
(Plan + engines on a quarter of the card) stays for molecular replacement, whose own rules, and the
--mr hand check's structure factors (ReadModelAnomalousMap, a quarter), are unchanged.
- The null's resolution is worked out before the plan, so its engine is decided and reserved with
the others instead of after the indexing probe.
Placement changes only on smaller cards: with the logged footprints of the open-arm sets, 4nwv's
null structure factors move to the GPU below ~9 GB (and its rigid body at 6 GB), 8tyy's structure
factors to d_min below ~8.7 GB; on 16 GB and larger nothing moves. On the RTX 5080 (16 GB), 4nwv,
8tyy, 6oel and 3inp give byte-identical reflection files, maps and models to rc175 HEAD, with more
rigid-body engines where they fit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BraggIntegrationEngineMetal is BraggIntegrationEngineCPU with its integer pixel work run as Metal
kernels, one thread per reflection: the neighbour mask, the owner map, pass A's walk over every box
and the background clip. Everything in double, the profile learning and pass B stay on the CPU, in
the CPU engine's own code - Apple GPUs have no double, and no GPU exp matches macOS expf bit for bit.
The sums are integers, so the GPU's order does not matter; the stencil arithmetic is the same
BraggStencil.h, compiled at run time (MSL 3.0, no offline Metal toolchain needed) with fast math off
(sqrt and division correctly rounded, measured) and fp contraction off, as the host already builds
(-ffp-contract=off, now extended to Objective-C++). The image is read in place (unified memory).
The CPU engine is split so both engines share the finishing code: pass A counts into BraggBoxSums
(integer; the double ring sums were exact integers already), the clip threshold is its integer floor,
FinishPassA and ProfilePasses are shared. Verified byte-identical p.hkl and MTZ data before/after the
split and with the Metal engine on lysozyme, cytochrome C, myoglobin, 4nwv and a small-molecule set.
Measured on an Apple M6 (wall, CPU engine -> Metal): 4nwv 180.5 -> 151.2 s, cytochrome C 72.7 ->
59.8 s, lysozyme 29.2 -> 22.5 s, myoglobin 43.3 -> 41.2 s; the sparse small-molecule set +1.5 s
(few reflections per frame leave the GPU idle).
CI: a macOS step runs tests/BraggIntegrationEngineMetalTest.cpp, which requires every float of every
reflection and every count to match the CPU engine over 19 setting combinations on random crowded
frames (negative pixels, exact-pixel ties, hot, masked and overloaded pixels), three frames per
engine; it fails without a Metal device and if the GPU path fell back to the CPU. Ten planted kernel
bugs were each caught. rugnux --no-gpu hides the Metal device too (gpus_are_hidden).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cudaMemcpy from pageable memory may return before the copy lands on the card, and
ImagePreprocessorBufferGPU gathers its squares on a non-blocking stream that does not wait for the
NULL stream, so on a loaded GPU the squares could be cut from the previous frame. Test-only race;
the pre-scan's engines synchronise their own streams before the buffer is read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plan cache of 310222156 never destroyed its plans: a long-lived process would carry every
shape it ever planned, and rugnux's cudaDeviceReset at exit (release_gpu_contexts) left the static
cache holding handles of a reset context. Plans are now kept only while a CudaFFTPlanScope is alive
(reference counted); when the last one ends every kept plan is destroyed on its own device, and
outside any scope a plan is destroyed as soon as its user is done - what every cuFFT user did
before the cache. rugnux opens one scope for the whole run (FFTPlanReuse, a CUDA-free wrapper, in
RunRugnux), so all its modes keep the reuse and the plans go before the device reset.
jfjoch_viewer and jfjoch_broker open none: their plans live exactly as long as the indexer that
holds them, as before, and nothing carries over to the next processing or collection.
Measured with nsys on an RTX 5080: cuModuleLoadData unchanged from 310222156 (cytc_x10sa 303,
4nwv 161, down from 450/458 before the cache); every load is now unloaded at the end of the run
(303/303, 161/161). An idle plan of the indexer shape (3142 x 16384 R2C) holds 1.6 MB of device
and 1.3 MB of host memory, a 3D grid plan 0.26 MB device. Outputs bit-identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every cuModuleLoadData/Unload in a run is cuFFT making or destroying a plan, and each plan loads
its own kernel modules under a process-wide lock: over a second a plan on a busy GB10, and a PTX
compile per load on Windows. The FFT indexers, rigid-body engines, structure-factor engines,
molecular replacement and beam-centre FFT each made their plans afresh and destroyed them again,
and the sizing functions (rigid-body and structure-factor FFTWorkBytes, beam-centre
DeviceMemoryNeeded via cufftEstimate*) made throwaway plans only to read a work size.
CudaFFTPlan now borrows a plan from a per-process cache keyed by device and full plan shape
(CudaFFTShape: n, embeds, strides, distances, type, batch). A plan is lent to one user at a time
(cuFFT is thread safe only across distinct plans), goes back when its owner is done - after a wait
for the owner's stream - and is never destroyed. Kept plans hold no work area: the user passes its
own, or CudaFFTPlan allocates one from the pool while it is borrowed. The sizing functions read the
work size from the kept plan (CudaFFTPlan::WorkBytes), which the engines then reuse. Plan creation
keeps the NULL-stream wait of 90ba15871, which now also covers the rigid-body, structure-factor
and molecular-replacement plans that were raw cufftHandles without it. A rigid-body engine hands
its previous zone's plans back on SetZone. FitsInDeviceMemory answers false where cuFFT cannot
plan the transform, as the ignored cufftEstimate failure did.
Measured with nsys on an RTX 5080: cuModuleLoadData/Unload 450/450 -> 303/51 on cytc_x10sa,
458/458 -> 161/51 on 4nwv (the remaining unloads are cuFFT's own inside plan creation; the
remaining loads are plans that are alive at the same time, mostly the concurrent rigid-body
engines). Wall unchanged on this workstation (ABAB), where a load costs ~1 ms; the gain is for
GB10 and Windows. Outputs bit-identical (p.hkl and the refined model on 4nwv, cytc_x10sa,
lyso_x06da_ref); cufftPlan3d vs cufftMakePlanMany with a caller work area checked bit-identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gpu_direct_host_read() (CUDAWrapper) is true where the device accesses pageable
memory through the host's page tables (cudaDevAttrPageableMemoryAccess and
...UsesHostPageTables: GB10, GH200, GB200), asked once per device;
JFJOCH_FORCE_DIRECT_HOST_READ forces it on for testing with page-locked inputs.
Where it holds:
- BSLZ4DecoderGPU: the LZ4/fused kernels and the tail copy read the reader's
compressed chunk directly, no pinned staging memcpy and no H2D copy;
- ImagePreprocessorGPU: Analyze and the UINT16_WIDENED path hand the host
buffer to preprocess_kernel (if aligned to the pixel type), and nothing is
page-locked;
- RotationScaleMergeGPU::IngestUpload: IngestExtractKernel reads the pinned
stage in place, without device staging buffers; the stage is reused only
after the kernel has run.
Discrete cards take the unchanged path. Every caller synchronises its stream
before the host buffer goes away. DirectHostReadGPUTest checks the three
sites against the CPU on page-locked inputs, in either mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Scaling: the 80%-of-card check runs right after the device ingest's sort and cut, where the
observation and distinct-hkl counts are first known, instead of after the geometry smoothing;
same counts, same decision, but a run that goes to the CPU no longer does the device smoothing
(6.9 s on an 8 GB card for a 33 M observation set) or holds its memory meanwhile.
- Rigid body: engines budgeted on a quarter of the card's total (the validation's plan), not
min(total/4, free - 1 GB). A busy card no longer sends the rigid body to the CPU (measured on an
8 GB card: ~80 s and a different fit); engine allocations wait for GPU work beside the run, and
an engine that cannot be set up fails the validation (RigidBodyGPUFailure) instead of falling
back to the CPU.
- Beam-centre FFT: GPU when its footprint is at most half of the card's total, otherwise CPU,
logged; no catch-and-rescore on the CPU after a GPU failure.
- FitPerFrameG reserves its per-frame vector (CPU scaling waited on the heap lock on Windows).
- get_gpu_free_memory(): cudaMemGetInfo, except on an integrated GPU under Linux, where free also
counts MemAvailable less a tenth of the total. Used only where free memory is waited on
(cuda_memory_available) or quoted in out-of-memory messages; it decides nothing.
The two card-filling [CUDAMemHelpers] tests skip on integrated GPUs.
- Out-of-memory messages point at rugnux --no-gpu instead of "the CPU build".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A still indexed on few spots often gets a lattice that explains them as well as the true one: an axis
multiple, whose reciprocal lattice holds every true node plus the nodes between, or the true lattice in
another setting. Its cell is then not the run's, the merge leaves it out, and its reflections at the true
nodes - as good as any - are lost. On two 25k-frame XFEL runs of a tetragonal crystal these were 26-30%
of the indexed frames; for those holding the true cell as a sublattice, the reflections at true nodes had
the I/sigma of accepted frames and those between them none.
RelabelOnSublattice (new, scale_merge/SublatticeRelabel) compares lattices, not cell parameters: it
looks for integer combinations of the frame's reciprocal axes whose full metric is the run's cell within
the merge's own tolerances, volume ratio 1-4 (another setting, or a 2x-4x supercell). Where several such
bases exist - a nearly symmetric supercell fits along more than one axis - the one with the strongest
mean I/sigma at its nodes is taken, since only the true lattice has signal there. The frame's indices
are re-derived on that basis (h = h' M^-1), reflections that are not nodes are dropped, as are those
finer than any frame already on the run's cell reaches (the wrong lattice was also predicted to a wrong
resolution), and the frame gets the run's basis in its own orientation. It runs before the first stills
merge, and in --mode scale before the resolutions are recomputed from the cell.
--mode scale, both runs (indicative; the mx pairs are queued): 1060 and 920 crystals put back,
observations +8%, I/sigma 10.3 -> 10.6 and 10.7 -> 11.0, CC1/2 0.966 -> 0.969 and 0.970 -> 0.966.
Tests: SublatticeRelabel_DoubledAxisGoesBackToTheTrueCell, _OtherSettingAndUnrelatedLattice,
_SignalPicksTheAxisThatWasDoubled.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The unmerged MTZ took every integrated still. A still whose cell is not close to the run's - the same
test, with the same tolerances, the merge applies - was indexed in another lattice (on few spots,
typically an axis multiple), so its H K L do not name the reflections a reading program takes them
for: aimless or careless would scale them in as wrong reflections. On two 25k-frame XFEL runs such
frames carried 60% of the integrated reflections, and the 824 MB file was the largest single cost
after the merge (about 7 s in --mode scale). Rotation runs are unchanged (one lattice).
The orientation matrix in the batch headers is likewise taken from the first crystal on the run's cell.
Test: Unmerged MTZ: on stills, only the crystals on the run's cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each crystal's (psi_x, psi_y) tilt was refined by a Ceres problem rebuilt for every inner
iteration: one auto-differentiated residual block and one Cauchy loss per reflection. With the
scaling reference cleaned up it was 70% of the stills scale/merge. The same objective - Cauchy
on the sigma-weighted intensity residuals plus the Gaussian prior on the tilt, the tilt bounded -
is now minimised by a small Levenberg-Marquardt on the rho'-reweighted 2x2 normal equations,
the tilt applied to all reflections through one rotation matrix and the Jacobian taken by
central differences.
It reaches the same answer to optimizer precision: on the new synthetic test
(StillsPartialityRefine_RecoversTheTilt) partiality error 0.0559 -> 0.03019 with Ceres and
0.03016 with this, mean tilt 0.02065 vs 0.02067 deg; on two 25k-frame XFEL runs (--mode scale)
CC1/2, I/sigma, R_pim and the CC against a deposited lysozyme model agree to 0.0005, mean
|dpsi| 0.023 deg on both. Scale+merge 6.9 -> 3.0 s (--mode scale, loaded box).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In rugnux people choose the algorithm, not where it runs. -X fft (and the
old -F) now resolves at parse time to the GPU FFT indexer where a GPU is
visible and --no-gpu is not given, and to FFTW otherwise. -X cufft is the
GPU implementation alone, as -X fft was. -X cufft and -X ffbidx without a
GPU now stop at once with a clear message instead of failing later in the
indexer pool. Auto is unchanged and not redundant: on stills with a known
cell and a GPU it takes FFBIDX.
Only rugnux's CLI spelling changes. The internal IndexingAlgorithmEnum, the
broker and its OpenAPI enum keep FFT = the GPU implementation; the CLI
maps onto it, and the command line the viewer writes for a run (RugnuxCommandLine)
gives the enum's FFT back as -X cufft, which re-parses to the same thing.
A run without -X, and -X fft on a machine with a GPU, process exactly as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The stills scale/merge after the image loop was almost entirely serial:
- every scaling-reference merge (MergeAll: the per-image scaling reference and the two
partiality post-refinement references) ran French-Wilson single-threaded on its export,
about 3 s each, and assigned free-R flags - neither is read by a reference, which uses I
only. MergeAll now exports intensities alone (MergeOnTheFly::ExportIntensities).
- the merge accumulated into a std::map; it is now an unordered_map exported in key order,
which is the std::map order.
- AddImage, the error-model collection and the R_meas pass of MergeStats did the
per-observation work (systematic-absence test, ASU key, filters, error-model sigma,
outlier test, merged-I lookup) in one serial loop. They now reduce a block of images in
parallel and consume it in image order (ReduceImagesInOrder), so every floating-point
sum, half-set choice and hash-map insertion is made in the serial order.
Measured on a 25k-frame XFEL stills run (box loaded, load 9-14): merge P1 19.0 -> 7.1 s,
final-group merge 27.7 -> 13.2 s; on a second run 23.4 -> 7.7 s and 29.7 -> 13.6 s.
mtz/unmerged mtz data, hkl, cif and report identical on both runs, mx and --mode scale.
New test MergeOnTheFly_AddImagesIsTheSerialMerge checks the parallel merge against the
serial loop across a block boundary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The measured-centre arm is a whole geometry pre-pass. It now integrates first a third of the pre-pass
frames, in 3k + 1 blocks of about 10 deg over the whole pre-pass arc with every third kept, and the
file's centre merges the same frames (its outcomes for the other frames are moved out while that
engine ingests). Where the measured centre loses on that subset on every line MeasuredCentreWins draws
by more than NEAR_TIE_SIGMA times the line's noise (Poisson for the strong counts, the Fisher standard
error for a CC1/2), it has lost and no full arm is run. Anything else, a near tie or a win, puts the
run back as the file's centre left it and runs the arm over all the pre-pass frames, deciding as before.
The run made again (decide_all_late_) runs the arm on all frames.
The same subset was tried for the metric-symmetry demoted arm and the walk refits and is not used
there: the held-out residual's standard error does not grow on a third of the frames, so a subset
margin overstates the evidence (one set's metric call flipped at 2.07 sigma), and a subset refit moved
one walk to a worse fixed point.
Against rc175 edde4439c, ci + 49 extra + 9 private sets: p.hkl and MTZ data identical on all 86. The
arm lost on the subset on 22 sets (no full arm), was a tie/win on 9 (full arm as before, +0.3-1.7 s).
Alternated runs: one set 62.1 -> 55.2 s, another 46.2 -> 42.4 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Each refit reads a share of the error left, so the walk's steps contract by about the ratio of the
last two. Where the step that ratio predicts moves the end of the sweep by less than half a frame,
the refit (an integration of the pre-pass frames and a post-refinement) and the indexing probe after
it are not made; the scale the validation frames last preferred is adopted, as before.
Measured on the walk sets of the battery against rc175 edde4439c: p.hkl and MTZ data identical; it
fired on one open set (2 refits instead of 3, same adopted scale), about -7 s there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The intermittent "out of GPU memory" on the 3600-frame cbf.gz set (about one run in three on a 16 GB
card, base included) was caught with allocation backtraces: FitErrorModelGPU's 42.8 MB in
MergeAndStats failed while the speculative geometry probe (GPUWorkBeside) held its FFT indexers on
the same card. The engine's own arrays (RotationScaleMergeGPU Impl::Alloc) wait for such work to end
and ask again; the merge's helpers - ErrorModelGPU, WilsonOutliersGPU, PairSortGPU and the cub
scratch of the engine itself - allocated directly and failed at once.
CudaAlloc::SynchronousAfterWorkBeside: Synchronous, and where the card is full after the pool is
trimmed, it waits for the GPU work beside the main line (wait_for_gpu_work_beside, 10 min as for
the engine) and asks once more. The four merge files use it. Only the probe passes register
GPUWorkBeside and they index only, so none of them merges and nothing waits for itself. When a
buffer is allocated is all that changes; what is computed is not.
Catch2 CudaDevicePtr_AfterWorkBesideWaitsForItsMemory: with the card filled by work beside, a
Synchronous buffer fails and a SynchronousAfterWorkBeside one gets its memory once the work ends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A new last column, "disposition", gives each frame's merge disposition as
a number to plot directly: 1 merged, 0.5 downgraded, 0 rejected; nan for
a frame that never got a scale (never reached the merge) and on a run
that merged nothing, as the file writes every value nothing measured.
Existing columns keep their indices.
0.5 is a marker, not a weight. "Downgraded" is a label applied after the
merge (RotationScaleMerge::CountSweepDisposition marks the merged frames
inside a flagged sweep-quality range); the merge applies no factor of its
own to them - they are carried at whatever weight their own scale and
sigmas give - so there is no fixed or per-frame reduction factor that
could honestly be written instead.
Writer-only change: on three open-arm sets with downgraded and rejected
frames p.hkl is identical to the same commit without it, the first ten
columns are unchanged, and the column's 1/0.5/0 counts equal the report's
merged/downgraded/rejected counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A screw candidate whose absent class no reflection contradicts, refused only because its
evidence could not reach min_screw_absence_evidence even with every predicted-absent axial
reflection at the floor (one such reflection reaches 6.9 nats; a row without a control class
reaches nothing), left the symmorphic group adopted and the report saying the group follows from
the absences. Such a row is now SPACE_GROUP_SCREW_UNDETERMINED, with the screw group in
SPACE_GROUP_ALTERNATIVES. Report only: the result is kept apart (unreached_screws) and folded in
after every decision, so the written group and files do not change.
Measured on a monoclinic set whose merge holds five 0k0 reflections at noise level, one of them
predicted absent: P2 was reported as decided by the absences against a deposited P21; it is now
P2 with the b screw open and P21 offered. Over the full battery's candidate tables the condition
holds on that set and one private set (the same P2/P21 case) only. ci tier: no change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
determine_cutoff_radius() starts at the IT92 approximation of the radius. When the density there is
exactly the float cutoff (1e-5f) - which happens for a few B per element, e.g. a nitrogen at
B + blur = 277.64563 - neither loop runs, x1 == x2, and the interpolation returns 0/0 = NaN. Upstream
gemmi master has the same code.
A NaN radius cast to int gave MakeAtomBox() INT_MIN half-widths, so UseAtomBoxesByPlane() sized its
point buffer from a wrapped product (one point) and then wrote past it: the heap overflow (SIGSEGV, or
a crash later in FFTW) that killed molecular replacement after the rigid body of a large deposited
model, turned to imitate another crystal form, had failed (2 of 73476 atoms, in the CPU rigid body of
an MR placement). The GPU density path took the same NaN radius into its box sizes.
The outward search now steps at least once, so that case returns the point at the cutoff. Every other
B gives the same radius bit for bit (checked over ten elements, every 8th float B in 2-600 A^2: 3 NaN
before, none after, no finite radius changed).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Every stage between the socket and the HDF5 write was bounded by a message count and never by
bytes: ZeroMQ receive watermark 100, cbor_fifo 200, outside_fifo 1000, repub_fifo 200 - up to
some 1500 images in flight per writer process, each a whole compressed image, so 6-12 GB for a
JUNGFRAU 4M and 30-60 GB for a 9M, and the writer's memory was a quantity nobody had chosen.
ReceiveRingBuffer is a fixed ring of bytes owned by the ImagePuller base class, so the ZeroMQ
and TCP pullers share it. A message is received straight into it (TCP reads the payload into
the ring; ZeroMQ copies the message in) and travels as a shared_ptr region in ImagePullerOutput;
the deserialised image points into that region, republish sends it as is, and the bytes come
back when the last holder lets go. Space is reclaimed in arrival order: a region released out
of turn - republish finishes with messages in its own order - is marked and waits for the ones
in front of it. Because of that, what republish may hold is capped at a quarter of the ring,
and an image over the cap is not republished, exactly as one that finds the republish queue
full; otherwise a stalled republish consumer would pin the ring and slow the writer to its
rate. A message larger than the whole ring (a START or END of a very long run) goes on the heap,
so it cannot wedge the receiver.
A full ring stalls the receive thread, and what that does depends on the stream. On TCP the
pusher waits behind the writer's BUSY heartbeats and gives up only after a minute in which no
image at all was written. On ZeroMQ the pusher's send blocks for one second, then drops the
image and sends without waiting for the rest of the run - so on ZeroMQ a ring that is smaller
than the writer's stalls costs images, where the old 1500-message queues absorbed them. That
is the trade: the budget is now a number, and it has to be chosen. jfjoch_writer sets it with
-m/--rcv_ring_mb, default 2048 MB per writer process; the ZeroMQ receive watermark default goes
from 100 to 10 messages, since it now only counts what sits beyond the ring. The ring's peak
use is in the end-of-run log line, and docs/JFJOCH_WRITER.md has a Memory section with the
sizing rule.
The puller's own default is no ring at all, which keeps every other user as it was: the
broker's DECTRIS stream2 puller (JFJochServices, JFJochReceiverService) stays zero-copy with
the ZeroMQ message as the buffer, and the broker's direct HDF5 writer (HDF5FilePusher ->
FileWriter) never goes through a puller. The puller threads now also catch std::exception, as
the writer does since the previous commit: deserialising the END message of a million-image
run allocates a vector per per-image array, and a bad_alloc leaving the thread ended the process.
Tests: ReceiveRingBufferTest (reuse, wrap-around, full ring, out-of-order release, heap
fallback, zero size, stop, a region outliving its ring); ZMQImageCommTest_SmallRingBuffer (64
one-megabyte images through a three-image ring, nothing lost, peak within the bound); one of
the two writers in TCPImageCommTest_2Writers_WithAck runs on a three-image ring.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ProcessDataImage, ProcessEndMessage and NotifyReceiverOnFinalizedWrite caught only
JFJochException, while ProcessStartMessage and FinalizeDataCollection already caught
std::exception. Anything else - std::bad_alloc from a per-image array, filesystem_error from
a path check, the JSON library refusing a string that is not UTF-8 while publishing a
file-done notification - left CollectImages, and Run() logged "Exception not properly handled"
and returned with the state never reaching Finalized: no END acknowledgement, no writer
notification, the master's .tmp left behind until the next START. On the TCP image stream the
broker then waited for the END acknowledgement indefinitely, because the writer's heartbeat
thread keeps reporting it alive whatever state it is in.
Now the same catch as at the other sites: error state, fatal ACK, END processed, Finalized.
The new StreamWriter test provokes the JSON case through a sample name that is not UTF-8 with
the file-done socket enabled, and checks the run ends Finalized with no master written.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
None of the eight GPU test cases in the file (BraggPredictionGPU*, the CPU/GPU consistency
cases and BraggPredictionRotGPU_DeviceOrderMatchesHost) checked for a device, so a CUDA build on
a machine without one failed them instead of skipping. Same guard as the other GPU tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
ModelValidation_ReindexesTheDataIntoTheModelsIndexing still expected no null for data already in
the model's indexing, as when the null ran only where a decision was pending. Since the unified
model validation it runs on every validation whose schedule asks for it (fit_test, the default),
and is skipped only where the verdict is another validation's; the test now checks that the null
ran and that the model fits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
A UINT16_WIDENED frame (marCCD) was page-locked where it lay with PinInputRegion, which keeps the
registration until the engine moves on to another frame. The pre-scan hands its kept frames to the
spot measurement and to the defective-pixel search at once, each with its own engines, so one
engine could unregister a frame while another engine's copy from it was still in flight. That
takes the context down ("unspecified launch failure"), seen as a rare crash at the end of the
pre-scan on a 3072x3072 marCCD sweep (1 run in 30 reproduced it).
The stored bytes now go through the engine's own page-locked staging buffer, as the CBF and BSLZ4
decoders already do. The pixels decoded are unchanged.
Test: ImagePreprocessorGPU_SharedStoredFramesManyEngines - four engines upload four shared marCCD
frames concurrently; it aborted with the launch failure before the change and passes after it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The rest of the resident ingest moves to the card: the delta_phi re-solve against each
frame's smoothed lattice, the one exact-Bragg angle per rocking event, the partiality
recomputed from the smoothed mosaicity (with corr brought to it) and the histogram of the
frames an event spans. The host keeps the per-frame work (the smoothed lattices, the
mosaicity window) and reads back only corr; nothing per observation crosses back, and the
host arrays the smoothing worked on are no longer built on this path.
The integer work is the host's to the bit (the event-span histogram). The floating-point
work differs in the last bits: every rounding on the card is explicit (no contraction, so
the same bits on any GPU), the host's compiler fuses some of the same products into sums,
and erf/atan2 come from the CUDA library. This is the one place in the lever where results
change; see the hand-back report for the sets and metrics it moves.
Measured on 7qij (main engine, 78 M reflections): the whole ingest 6.5 s host + 1.3 s device
before this lever -> 2.9 s, of which 2.4 s is the upload (memory-bound, beside the
post-refinement gather) and 0.25 s this smoothing. Same space group, error model,
resolution and reflection count; IMEAN differs at the 1e-7 level except on near-zero
intensities.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
With the GPU resident, the host ingest swept the source reflections four times (background
and h histogram, sort keys, the in-range build, the device staging), bucket-sorted the keys
and cut the raw-hkl runs: 5-6 s of host time per engine on a 78 M-reflection sweep, run
twice beside the quality guard. Now the frames' reflections are copied once, through two
pinned buffers, to the card, where the per-frame background is summed, the reflections
are ordered by two stable radix passes ((image, index), then raw hkl - the one order the
host's total comparator gives), cut into runs by run-length encoding, filtered by the
resolution range and compacted. The observations, the runs and their order are the host
ingest's to the bit; a new GPU test checks the layout against the host's sort and cut.
The geometry smoothing still runs on the host in this step, on arrays read back.
The all-observation arm's engine is cloned after the main engine's device ingest (its
DeviceBytes are known only then) and copies the runs on the card as well. DeviceBytes now
also covers the ingest's own peak (90 bytes per reflection, 64 per run before the cut).
Measured: p.hkl and MTZ data identical on cytc and 7qij.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The marCCD reader read each frame's 16-bit scanlines into scratch and widened them to int32 on the
worker thread (marccd::ReadInto, ~20 % of the image-loop worker CPU on 4nwv), and the 4-byte image was
uploaded. ReadRawImage now reads the scanlines straight into the image's buffer and hands them over as
a new read-only CompressionAlgorithm::UINT16_WIDENED (mode Int32): two bytes a pixel cross PCIe, from
page-locked memory, and the preprocessing kernel widens them itself.
It is not CompressedImageMode::Uint16, which calls 65535 an error pixel; a marCCD's 65535 is its
saturation marker. So the kernel compares each pixel with int32's error marker and the int32 saturation
cap - the comparisons are made on the pixel's value, in int64, rather than in the storage type - and
decides every pixel exactly as for the int32 image the host would have widened it to. The host widening
(JFJochDecompressPtr) is the CPU path.
Test: ImagePreprocessorGPU_StoredEncodingsMatchCPU, with a frame carrying 65535 at a saturation of
65535, against the CPU preprocessor on the host-widened image.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The CBF reader decoded every frame's x-CBF_BYTE_OFFSET stream on the worker thread - 19-30 % of the
image-loop worker CPU on CBF sets (7qij, 4mtk) - and the 4-byte image was then uploaded. ReadRawImage
now hands over the stream as the file stores it, in the image's own buffer, as a new read-only
CompressionAlgorithm::CBF_BYTE_OFFSET (mode Int32, what it decodes to), and ImagePreprocessorGPU
decodes it on the card into the same int32 image and runs the same preprocessing kernel on it.
The device decode (CBFDecoderGPU) cuts the stream into 256-byte tiles. A token is 1, 3, 7 or 15 bytes,
so a tile can be entered at 15 offsets; for each one thread works out where the decode leaves the
tile, how many pixels it emitted and their sum modulo 2^32 (the walks from different entries meet
within a few bytes, after which they are one walk). An exclusive scan composing these maps gives each
tile its true entry, first pixel and running value, and a second walk writes the pixels. A stream
that ends before the image is refused as on the host.
The host decoder moves to compression/ (DecodeCBFByteOffset) and is reached through
JFJochDecompressPtr, so every host consumer - the CPU build, the azimuthal-integration worker, the
pre-scan's shadow and gain passes - decodes exactly as before, now from the stream. It is the CPU
path, not a fallback: a frame the device refuses is not retried on the host. A process file written
from CBF input stores the decoded int32 image, as it always did. .cbf.gz is still inflated on the host.
Tests: CBFDecoderGPU_MatchesHost (every token size, 0x80 inside payloads, escape runs across tiles,
trailing CIF text), CBFDecoderGPU_ShortStreamIsRefused, ImagePreprocessorGPU_StoredEncodingsMatchCPU.
Checked locally against the host decoder on 36 real frames of 9 open-arm CBF sets (signed 32-bit,
signed and unsigned 16-bit, plain and gzip): identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The rotation predictor used to download every predicted reflection into pinned memory, copy it into
the host vector, build sort keys, std::sort them, gather the 88-byte structs into scratch and copy
them back (OrderOutput), and - on a frame over output_limit - partial_sort the structs themselves
(TruncateToOutput). On a large cell that is a quarter to a third of the image-loop worker CPU
(profiles of 4nwv and 7qij: TruncateToOutput 12-16 %, OrderOutput 9-13 %, plus the memmoves).
Now the same order is built on the device with stable cub radix sorts of an index: delta_phi, then
the packed hkl (OrderOutput's h, k, l, delta_phi), then - over the cap only - partiality descending
and excitation error ascending (TruncateToOutput, whose hkl tie-break the stable sort keeps). Only
the kept rows are gathered and downloaded. Floats are keyed by an order-preserving integer map with
-0 made +0, so the key order is the comparators' order; the only possible difference from the host
is between two solutions of the SAME hkl with bit-identical partiality and excitation error, whose
relative order partial_sort leaves to its heap and the stable sort to delta_phi.
The device buffers grow with half again of slack, so a frame predicting a few more than the last
record (the supercell probe's frames above all) no longer reallocates and predicts again; the host
side is sized for what is returned, at most output_limit.
Test: BraggPredictionRotGPU_DeviceOrderMatchesHost holds the device output against OrderOutput's
order and against TruncateToOutput applied on the host, at four caps.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
