The recompute-and-compare check of the first-pass memos (a probe's
validation evidence, the per-frame spot lists, the RotationIndexer outcome)
is now ProcessConfig::verify_first_pass_memo, passed to RotationIndexer with
VerifyMemo(). No command-line option; only tests set it.
A mismatch now throws MemoMismatch, which the passes' catch-and-continue
sites treat as fatal: before, a mismatch inside a rotation-scale probe was
caught and logged as a probe that did not complete, so verification could
pass without having checked anything.
Rugnux_FirstPassMemoMatchesRecomputation ([large], the LFS rotation set,
~15 s on a GPU) runs the two-pass with post-refinement and scaling, which
takes the walk's probes and the canonical pass through every memo, with the
verification on. Checked that a forced mismatch fails it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
ParallelChunks cuts a range into one chunk per worker, so a floating-point
sum folded per chunk and then added up rounds differently at another -N.
ParallelBlocks cuts it by n alone (n / min_per_block blocks, at most 256);
each block folds into its own slot and the slots are added in block order,
so the sum has the same bits at any thread count.
Converted:
- RotationScaleMerge::ApplyCellSurface: the per-cell cross/ref2 sums of the
surface fit (modulation, absorption), previously per-thread partials cut by
ThreadsForWork(idx_all) threads.
- PostRefine: the scale-scan cost grid (per-chunk slots cut by -N).
- PostRefine joint solve: Ceres at a fixed 16 threads (as the rotation
indexer's chain) instead of -N. Ceres sums cost and gradient in
4 * num_threads pieces, so the split no longer follows -N. The pieces are
handed to its threads in scheduling order, which only one thread makes
exact; one thread was measured at +1.0-1.4 s of post-refinement on myob and
lyso (0.5 -> 1.9 s, 1.1 -> 2.1 s), so that channel is left.
- IndexAndRefine supercell probe: per-frame probes kept by image number and
summed in frame order, instead of added under a mutex in completion order;
the primitive is the highest probed frame's instead of the last finisher's.
Integer reductions and per-item passes are unchanged; the three ParallelSort
callers already break ties on the index.
Validation (myob, cytc, lyso, sparse; GPU and CPU builds; -N 8/16/32):
p.hkl, p.mtz, p_P1.mtz and p_unmerged.mtz are md5-identical across -N, and
identical to the base 351be7de0 at every -N. The base was already -N
invariant on these four sets (GPU -N 1..32, CPU lyso -N 2..32), so no output
moved: the converted sums differ from the old ones only in last bits that
never reached a written float.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
BraggPrediction::Calc and BraggPredictionRot::Calc stopped filling at
max_reflections (kPredictionCapacity = 20000), silently, inside the h/k/l
walk - so a frame that predicted more kept whichever reflections the walk
reached first, while the GPU predictors grow their buffer (GrowCapacity) and
keep them all. A large cell therefore merged a different reflection set on a
CPU-only build (the macOS/Windows viewer path, or CUDA_VISIBLE_DEVICES=) than
on a GPU one.
Both CPU predictors now double the buffer when the walk fills it, so they
return the same set as the GPU; output_limit and TruncateToOutput are
unchanged and remain the one cap on what leaves Calc, on both paths.
Tests: BraggPrediction_GrowsPastStartingCapacity ([portable], CPU only) - a
large cell predicting 28k (still) / 54k (rotation) reflections on one frame
gives exactly what a buffer large enough from the outset gives, and an
output_limit keeps the best-recorded half. BraggPrediction_CPU_GPU_
consistency_large_cell - the same frame on CPU and GPU: 28384/28384 and
53671/53671 matched by hkl, positions within 0.1 px. Both fail on the old
code (count == 20000).
Validation (CPU build): four in-house sets md5-identical to the base. A
dense rotation set from the private arm predicted 72.0M (20000 x 3600
frames) before and 235.9M now, against the GPU's 235.9M; it ingests
218.65M partials, as the GPU does, and now reaches the GPU's cell, space
group and merging statistics, where the capped CPU run settled on a
different lattice. Cost on the CPU: 72.5 GB peak RSS (17.5 GB before),
44.5 min wall on a shared machine (7.2 min before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The r2 regions set aside from a reflection's r2..r3 background ring were
those of every prediction in the +-4 sigma rocking window. On a finely
sliced dense pattern most of them are the tails of reflections recorded on
the frames either side, which fill every ring while the frame shows nothing
there. 9ac2ca677 kept the reflections those rings starved by taking the ring
whole, neighbour pixels included, and relying on the high-side clip.
A prediction now masks the ring only where it puts at least 5% of its flux
on the frame (partiality >= 0.05). A ring still starved by those neighbours
has real flux in it, and the reflection is dropped, as before 9ac2ca677:
taking it whole let the neighbours' wings into the background. The CPU mask
holds two levels (tail, flux); the GPU writes them in two launches so the
flux mark wins, with no atomics.
The count the widened-radius guard reads keeps measuring the pattern's
density over every prediction, tails included, so the guard decides on
the quantity its 1.13% bound was read off. Read on the gated mask it let a
cubic set keep r1=6, and R_meas went 36.9 -> 43.2%. Masked but
unreadable ring pixels are no longer counted as the neighbours' doing.
Fixed radius, same code base, --model, the 0.05 deg / 7200-frame set:
pre-9ac2ca677 9ac2ca677 this
rings starved 88.9% 88.9% 21.5%
partials ingested 7.4M 67M 53M
completeness 17.9% 79.7% 85.1%
ISa 12.8 14.3 14.7
R_meas 6.8% 8.4% 7.6%
R_free 0.139 0.176 0.171
radial misfit 0.11 0.185 0.066
R vs model, common hkl to 0.69 A (61k):
0.133 0.130 0.129
(the R_free rise over pre-9ac2ca677 is composition: 63k -> 319k
reflections to 0.69 A, the added ones weaker.) Gate at 0.2 instead:
0.1% starved but radial misfit 0.35, R_free 0.180.
A crystal whose header-geometry pass indexes a 7x supercell:
9ac2ca677's whole rings raised that pass's I/sigma >= 2 count past the
refined pass's by more than 10%. RefinedPassIsWorse then sent the run
back to the supercell. Now: the true cell and space group as before
9ac2ca677, same ISa, 104M partials in the header pass against 219M.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The unmerged export (_unmerged.mtz) was serial apart from its final row
sort: gathering the partials, sorting them by (h,k,l,image), summing the
rocking events and filling the rows all ran on one thread. On a 17 M
partial rotation set that was 2.8 s (sort 1.6 s, event sums 0.6 s,
gather 0.25 s, rows 0.19 s); on a synthetic 122 M partial sweep 23 s.
- The partials are gathered into preallocated slots per outcome, left
uninitialised so the workers first-touch the pages, and sorted with
ParallelSort. The sort key now ends with the part's input position
(outcome, index), so the order is total and the same on any number
of workers. It differs from the previous std::sort order only among
parts with equal (h,k,l,image_number); measured on four in-house
sweeps (464-6138 such positions each), every summed full is bitwise
the same.
- Events are summed in chunks whose bounds move forward to the next
event start, so no event is split; the chunks are returned as pieces
in order rather than concatenated.
- Rows are filled by index into a preallocated table; absences and the
batch range are still decided in one serial pass. The final row
gather after the sort is parallel too.
Measured: the 17 M set 2.82 s -> 0.86 s, the 8 M set 1.37 -> 0.49 s,
all while the P1 cross-check merge shares the pool; a synthetic 122 M
partial sweep 22.7 s -> 7.7 s at 32 workers. What remains is mostly
ParallelSort's pairwise merge levels. The file write (0.1 s for 112 MB)
is I/O and left as it is.
p.hkl, p.mtz, p_P1.mtz and p_unmerged.mtz are md5-identical to the base
on four in-house sets, on the GPU and the CPU build, and at -N 1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Every merge buffer now comes through Impl::Alloc. When one cannot be had
even after the pool is trimmed, the run ends with what it needed and what to
do: "Scaling N partial observations needs more GPU memory than this card
has: X GB for their per-observation arrays alone (67 bytes each), before the
merge's other buffers, on a Y GB card. This data set is too large for GPU
scaling on this card - run it on the CPU (CUDA_VISIBLE_DEVICES= rugnux ...,
or a CPU-only build)". SetPartialsLayout checks the 67 bytes per observation
of its own arrays against the card's total memory before allocating any of
them, so a set that could not fit an empty card fails before the gigabytes
are allocated.
There is deliberately no switch to CPU scaling mid-run: the two paths differ
in the last bits, so the result would depend on the card.
A 218.7M-partial rotation set (a neighbour-starved dense pattern) now fails
with this message 35 s into the run; with CUDA_VISIBLE_DEVICES= it runs to
completion on the CPU path. No arithmetic changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
ca1baffef moved RotationScaleMergeGPU to synchronous cudaMalloc. cudaMalloc
cannot take memory the default pool holds, and after the image loop the pool
still reserves what the per-worker engines freed with cudaFreeAsync (the 1 GB
release threshold only applies at a synchronisation of the pool's streams).
Measured at the merge's construction on a 123.6M-partial rotation set: pool
reserved 5.64 GB, used 0.83 GB - 4.8 GB of the 16.3 GB card idle but
unavailable. The merge's per-observation arrays (about 9.5 GB there) then ran
out at a 494 MB array with 316 MB free, and the run failed with "Failed to
allocate device memory" right after the post-refine gather. The 77710e1ce
binary allocated those arrays from the pool and passed.
A failed synchronous allocation now waits for the device, trims the pool to
zero and tries once more. Memory placement only: no arithmetic changes.
p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz byte-identical on myob, cytc, lyso
and sparse; the failing set now passes (R 3:H, 1.30 A, ISa 10.9).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
rc173 stores refl_mask and owner as 16x16 tiles (cpu-loops-tail), so
the rows of those two can no longer be prefetched as flat rows; the
prefetch now covers the image only, and the box helper uses a local
struct in place of the removed Rect.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
RunImpl on the CPU path is memory-latency bound, not arithmetic bound:
on the 18 Mpx rotation sets ~60% of its cycles sat on the first load of
an image, reflection-mask or owner-map row (each box row is its own
cache line and 4 kB page, in a frame far larger than the caches), and
the function took 59% of all DRAM demand fills of a cytc run (5.0e9).
Pass A now asks for the next reflection's box rows (image and
reflection mask over its outer-ellipse box, owner map over its claim)
while the current one is summed; pass B asks for the fit grid's image
and owner rows as soon as the grid size Rf is known, before the
Gaussian profile is built. A prefetch is only a hint, so every result
is unchanged: p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz byte-identical on
cytc, lyso, myob and sparse; Bragg* and [integration] tests pass.
The prefetch loops are written out in place: GCC 13 treats a function
that only prefetches as having no effect and deletes calls to it (a
lambda version compiled to nothing).
Measured on cytc (CPU build, -march=x86-64-v3, 32 threads, perf at
199 Hz): RunImpl 1645/1665 -> 1375 core-s (-17%); RunImpl DRAM demand
fills 5.0e9 -> 3.3e9. Walking the reflections in detector order instead
(bands of rows) changed neither the fills nor the time, so the misses
are compulsory, not a matter of order - that variant was dropped.
Remaining stalls: the owner-map build and pass B's image rows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Two fixes found in the post-merge profile, where the previous commit's loops did not do what they
were written to do:
- AccumulateRingsBlock: the ring_overflow.emplace_back call inside the loop left no callee-saved
registers for the locals, so GCC kept sum/sum2/count on the stack and the store-reload chain was
still there (29% of the function's samples on one stack add). The azint sums get a loop of their own
over the block, and the out-of-histogram values are listed by a second loop run only when the block
has any, in the same pixel order; now the sums live in registers. Same additions in the same order.
- DetectPass: inlined into DetectPass's main loop the vertical slide was not vectorised (the
vectorised copies GCC made were for the other call sites). It is now a free function,
SlideVertical, which GCC vectorises on its own.
Measured (perf, 75 s of myob, CPU-only, relative to the unchanged FlagRow/AnalyzeBlock):
AccumulateRingsBlock -4..-11%, DetectPass + SlideVertical -7..-14%. Byte-identical p.hkl, p.mtz,
p_P1.mtz, p_unmerged.mtz on myob, cytc, lyso, sparse (CPU) and myob (GPU); ImageSpotFinderCPU*,
AdaptiveSpotFinder, SpotFinding, AzimuthalIntegration and portable tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Conflicts in the three CPU pixel loops the fused pipeline split per block, resolved by carrying this
branch's loop bodies into the new functions with the per-pixel arithmetic unchanged:
- AdaptiveSpotFinderCPU: the register-held ring sums now live in AccumulateRingsBlock (stored at the
end of each block, so the additions stay in pixel order).
- ImagePreprocessorCPU::AnalyzeBlock reads the PixelMask-derived 32-pixel mask words at first + i.
- ImageSpotFinderCPU::DetectPass: rc173's new_row() marking/fill_row calls kept in place, the
vertical update replaced by the vectorised slide with the prev_strong fix-up.
Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz vs rc173 references on myob, cytc, lyso,
sparse (CPU-only build) and myob (GPU build); targeted tests (ImageSpotFinderCPU*, AdaptiveSpotFinder,
SpotFinding, PixelMask, Bragg*, RotationScale, AzimuthalIntegration, portable) pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
refl_mask (uint8) and owner (uint32) were full-frame arrays per engine, ~90 MB at 18 Mpixel, of
which a call writes only the neighbourhoods of its reflections; over a sweep every page of both got
touched in every worker, and the dirty-rectangle clears were row-strided memsets. They are now a
TiledFrame: a per-tile directory plus a pool of 16x16 tiles allocated on first write and dropped by
Clear(); reading a tile nothing wrote gives the empty value (0 / BRAGG_OWNER_NONE). Writes and reads
are exactly those of the flat arrays, so the same ownership decisions.
Measured (perf, 499 Hz, CPU-only build, cytc): RunImpl 1633 -> 1557 core-s, memset 88 -> 32 core-s.
Peak RSS and page faults did not move measurably (myob 15.3 GB; the peak is set elsewhere).
Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz on myob, cytc, lyso, sparse (CPU) and myob,
lyso (GPU); Bragg* tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Each iteration of the partial scaling loop passes over all partials three times (group means,
per-frame G fit, corr update + rescale), each time reading a few fields of the 80-byte Obs, and the
group means gathered whole Obs records through the group permutation. The loop now works on
ScalingObs (frame, group, I, sigma, partiality, prescaling_corr, zeta, on_ice; 36 bytes, partials'
order) with corr in its own float array, and the group means read ScalingRef (I, sigma, partiality)
laid out in the CSR's group order, so only corr is gathered. corr is written back to the partials
(and kept by the memo) when the loop ends. ReferenceWeight takes the fields, FitPerFrameG is a
template over Obs/ScalingObs; the same values, so the same arithmetic in the same order. Transient
memory: 52 bytes per partial during the loop (~0.4 GB on 8M partials).
Measured (perf, 499 Hz, CPU-only build, cytc): update/rescale lambda 104 -> 35, group means
87 -> 24, FitPerFrameG 63 -> 27 core-s (-167 core-s, of which the copy costs ~10). CPU-only path;
the GPU scaling path is untouched. Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz on myob,
cytc, lyso, sparse (CPU) and myob, lyso (GPU); rotation scale/merge tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Most rotation solutions are far from the frame, and the existing phi prefilter rejected them only
after computing phi = atan2(sin, cos). The same test is now asked of the (cos, sin) pair first:
|phi| > phi_limit is cos(phi) < cos(phi_limit), i.e. c < cos(phi_limit) * |(c, s)| in double.
phi_limit is the existing test's threshold at the largest rocking width any reflection can have
(bandwidth term at the resolution limit) plus a 1e-3 rad margin, far above float rounding, so every
solution rejected here is rejected by the existing test too, and the survivors go through it
unchanged. One cos per Calc; min_zeta <= 0 disables it, as it disables the existing test.
Measured (perf, 499 Hz, CPU-only build, cytc): atanf32 + atan2f32 + __atan2f_finite
196 -> 12 core-s, Calc itself 161 -> 199 core-s (the sqrt of the new test), net ~ -150 core-s.
Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz on myob, cytc, lyso, sparse (CPU) and myob,
lyso (GPU); Bragg* tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
AdaptiveSpotFinderCPU::AccumulateRings: consecutive pixels mostly share a ring, so the ring's
sum/sum2/count and the fused azint sums are held in locals while they do and stored when the ring
changes - the same additions in the same order (az_sum2 is still contracted to the same FMA), without
a store-and-reload chain through memory on every pixel.
ImageSpotFinderCPU::DetectPass: the vertical-sum update (add the entering row, take out the leaving
one) is one branch-free loop over the raw image that GCC vectorises (int64 lanes). A pixel strong in
the previous pass used to be substituted per pixel through a bit test, which kept the loop scalar;
it is now added with its row and taken out again from the few set bits of prev_strong. Integer sums,
so the same totals. (A first, fully branch-free version that kept the per-pixel bit test did not
vectorise on the prev_strong path and was measured slower; this is its replacement.)
ImagePreprocessorCPU: the per-engine std::vector<bool> built bit by bit from the 32-bit mask
(~10 core-s per cytc run, one per worker per pass) is replaced by 32-pixel mask words that PixelMask
derives once beside its binary mask; each engine copies 2 MB. A branch-free rewrite of the Analyze
loop was measured and dropped: the loop is bound by reading the decompressed image (330 vs 328
core-s on cytc), so only the mask test changed.
Measured (perf, 499 Hz, CPU-only build, cytc, first version of this change): AccumulateRings
591 -> 539 core-s. Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz on myob, cytc, lyso,
sparse (CPU) and myob, lyso (GPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The CPU-only image loop is DRAM-bound on 16M frames: decode, preprocess, the adaptive finder's plain
ring pass and FlagRings each streamed the whole frame through memory.
- JFJochDecompressHperfBlocks hands each decoded bitshuffle block to a callback; with no output
buffer the block is unshuffled into a reused block-sized scratch (JFJochDecompressBlocks).
- MXAnalysisWithoutFPGA::PreprocessCPU preprocesses each block into the int32 buffer
(ImagePreprocessorCPU::AnalyzeBlock) and, when the fused CPU finder runs, puts it through the plain
ring pass + fused azint (AdaptiveSpotFinderCPU::AccumulateRingsBlock) while it is in cache. Detect()
then starts from those sums. The per-worker decompression buffer is no longer allocated for
bitshuffled data.
- FlagRings becomes FlagRow, called by DetectAt's first pass for row y+NBX just before that row enters
the vertical sums; first_pass_needed is marked from each row's candidates at the same point.
Exact: blocks arrive in pixel order, so the float azint sums see the same pixels in the same order;
the per-pixel expressions are unchanged; everything else is integer. p.hkl, p.mtz, p_P1.mtz and
p_unmerged.mtz byte-identical to rc173 on myob, cytc, lyso, sparse (CPU-only build), GPU myob
identical (the GPU path does not take this route).
CPU-only, 32 workers, under the gpulock on a shared (loaded) machine, base -> fused, two rounds
(second in reversed order):
myob 155.9 -> 97.7 s, 139.9 -> 88.1 s (loop 46.1 -> 26.8 s/pass; user 3690 -> 2221 s)
cytc 220.8 -> 156.6 s, 216.6 -> 155.1 s (user 5160 -> 4128 s)
lyso 134.5 -> 123.3 s, 69.5 -> 64.8 s
peak RSS myob 15.2 -> 10.8 GB, cytc 12.6 -> 10.4 GB, lyso 7.0 -> 6.7 GB
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The r2..r3 background ring excludes every pixel inside any predicted
reflection's r2 region, and a reflection whose ring kept five or fewer
clean pixels was dropped. The mask marks every prediction in the +-4 sigma
rocking window, most of which put a few percent of their flux on the
frame. On a finely sliced, dense pattern (0.05 deg frames, sigma_M 0.067
deg, 0.35 A, 7200 frames) they fill every ring: ~9400 predictions per frame
but only ~900 with partiality above 0.2, and 88.9% of all partials were
dropped at the fixed radius - including the peak frames. The merge was 17.9%
complete.
Such a ring now falls back on all its readable pixels; the existing
high-side clip removes the neighbour cores that are really there. The
starvation counts are unchanged, so the adaptive-radius guard reads the
same numbers. Detector-starved rings are still dropped.
On that set (fixed radius, no --model): 7.4M -> 67M partials ingested,
completeness 17.9 -> 79.7% (88% to 0.65 A), d_min 0.69 -> 0.61 A,
ISa 12.8 -> 14.3, reflections in common with the deposition 60k -> 252k,
rank CC vs |Fc|^2 minus the depositor's, outer shells -0.10 -> +0.07.
A dense 360-frame set: 14.9M -> 17.6M partials, dCC outer -0.116 -> -0.096.
myob_x10sa MTZ byte-identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The GPU finder read the whole image (int32 values + pixel_to_bin) three times per frame for its
per-ring background: a plain pass and two 3-sigma clip passes. As the CPU finder already does
(87cfc879b), the plain pass now also records every valid pixel in a per-ring value histogram
(HIST_VALUES = 1024 bins per ring, global memory, integer atomics aggregated per warp by
(ring, value)), and pixels outside [0, 1024) on an overflow list. Each clip pass is then taken from
the histogram and the list (clip_rings_from_hist, one warp per ring) - integer sums of the same
values, so the ring statistics, thresholds and spots are exactly the image passes'. If the list
overflows its capacity (npix / 32 entries), the clip passes read the image as before; the choice is
made on the device, so there is no extra host sync.
The clip bounds are now written as explicit fused multiply-adds (clip_keep). ptxas already fused
`mean -/+ clip_k * sigma` into FFMA in the image-pass kernel (checked in the sm_120 SASS); spelling it
out keeps the histogram clip and the image clip on exactly the same line.
Memory: nbins * 4 KiB histogram + 8 B * npix / 32 overflow list per engine, only on the
shared-memory path (at most ~1750 rings, so <= 7 MB of histogram); the list is 4.5 MB for a 16M EIGER2.
Exactness: a temporary side-by-side check (histogram clips vs forced image clips on every frame)
gave 0 differing ring sums/counts on myob, cytc, lyso and sparse (overflow list at most 308
entries); p.hkl, p.mtz, p_P1.mtz and p_unmerged.mtz are byte-identical to rc173 on all four.
New test AdaptiveSpotFinderGPU_HistogramClipMatchesImagePasses covers rings above/straddling the
histogram range and the overflow fallback.
Timing (RTX 5080, nsys, GPU locked; myob 16M EIGER2): reduce_rings 230 us x 3 launches/frame
(3.20 s) -> plain pass 308 us + 2 x 4 us early-exit launches + 2 x 12 us clip_rings_from_hist
(1.58 s total); summed kernel time 11.28 -> 9.18 s; image loops 5.89/5.09 -> 5.49/4.69 s. cytc:
kernels 12.86 -> 10.75 s. Small detectors gain less (lyso 0.40 -> 0.32 s, sparse 1.19 -> 1.01 s of
ring reduction), as the histogram costs relatively more in the plain pass there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Exact speed-ups of the per-pass geometry post-refinement; outputs are byte-identical.
Joint fit: the distance-held solve starts from nominal and reads nothing the free solve
produces, so it now runs on its own thread (std::async) beside the free solve and its
covariance, and is used only where the free fit converged, as before. Each solve has its own
copy of the tilt block and one shared, non-owned CauchyLoss instead of one per residual. The
covariance's Jacobian evaluation is threaded (Covariance::Options::num_threads): each residual
block writes its own rows, so the values do not depend on the thread count.
Rotation scale: the all-data fit and the five leave-a-fifth-out folds differ only in which of
the five per-fifth sums they add, so their golden-section searches are stepped together, one
event sweep per step for all of them, and fits whose brackets still coincide share the point.
Each fit takes the same steps on the same sums as before: cost_grid's per-(k, fifth) sums are
bit-identical whatever k it is handed alongside. About 23 event sweeps instead of ~140.
Measured per pass (rc173 baseline -> this, GPU build, box shared with other jobs; post-refine
log timestamps, joint fit / rotation-scale fit):
lyso 0.36 / 0.28 s -> 0.28 / 0.19 s
cytc 0.42 / 0.35 s -> 0.36 / 0.22 s
myob 0.29 / 0.04 s -> 0.23 / 0.04 s
sparse 0.20 / 0.06 s -> 0.14 / 0.07 s
p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz byte-identical to the rc173 outputs on GPU myob, cytc,
lyso, sparse and CPU-only cytc, and the logged post-refinement lines (geometry, residuals, k)
identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
MeasureBatchDeltaCCHalf re-formed the reference and re-measured every open
candidate on every round of its removal loop - including the rounds where the
picked candidate was only marked spent (not corroborated / no longer
convicted), which change nothing. On a 7200-frame sweep that is most of its
cost. All changes below keep every sum in its original order, so every
delta-CC1/2 is bit-identical:
- candidates are re-measured only after frames are actually removed; a
spent candidate leaves the others' measurements standing. The curve is
read off the first round of candidates (the width-1 ones are the batches),
and the redundant build_totals after the loop is gone.
- measure_all measures ranges that start on the same frame as one chain,
shortest first, carrying on the per-reflection sums - the same additions
in the same order, but shared frames are walked once.
- the usable observations are laid out once in frame order as
{group, I, info}, and the per-reflection sums are one struct per
reflection, so a measurement streams contiguous memory instead of three
indirect loads and four arrays per observation.
- locate measures the (up to) four edge moves of a step side by side and
takes them in the original order; the candidate's own measurement and the
conviction's are reused rather than repeated.
Verified with a temporary in-process check that re-ran every measurement
through the old code (sparse 324+825, cytc 68+68 measurements, all
bit-identical), and p.hkl/p.mtz/p_P1.mtz/p_unmerged.mtz plus the logged
delta-CC1/2 lines byte-identical to rc173 on myob/cytc/lyso (GPU), cytc and
sparse (CPU, sparse removes 288 and 72 frames) and 8a1a (vs the r4int
baseline binary).
Measured (perf, loaded box): 8a1a delta-CC1/2 20.1+7.4 s -> 2.2+1.7 s wall
(123 -> 13 CPU-s), run 129.9 -> 107.1 s; lyso (CPU build) 0.76+0.30 ->
0.16+0.11 s; cytc (CPU build) 0.15+0.25 -> 0.10+0.16 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
A sticky error (an illegal address, say) is reported by whichever worker synchronises next - usually
the bslz4 device decode - and MXAnalysisWithoutFPGA::Analyze and Rugnux::MaskDefectivePixels then
logged "falling back to host" and carried on, although the context is gone and the run fails anyway,
later and less clearly. cuda_throw_if_context_lost() now throws a GPUCUDAError ("CUDA device
unusable after an unrecoverable error: ...") there first, which the image loops already treat as
fatal (IsFatalResourceError), like a CUDA out-of-memory.
The last error cannot tell: once cudaGetLastError() has returned a sticky error, it and
cudaPeekAtLastError() both report success. cudaFree(nullptr) frees nothing and does not
synchronise, but returns the sticky error (checked: after an illegal-address kernel it returns
cudaErrorIllegalAddress; with only a pending out-of-memory it returns success), so a handled,
non-sticky decode failure still falls back - MXAnalysis_HandledDeviceDecodeFailureLeavesNoError
passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The engines' kernels run on their own non-blocking stream, which is not ordered after the legacy
NULL stream. The counter reset (cudaMemset, asynchronous to the host) and the radial-background
kernel upload in the Bragg engine constructor, and the direction-grid upload in FFTIndexerGPU (a
pageable cudaMemcpy returns once staged, before the DMA completes) were issued on the NULL stream,
so the first kernel reading them was not ordered after them. Now cudaMemsetAsync/cudaMemcpyAsync on
the engine's stream. In the Bragg constructor that is this->stream: the parameter of the same name
has been moved from.
compute-sanitizer --track-stream-ordered-races all flagged the Bragg ones (16 + 16 reports on
rugnux -e 150 myob; 1 in the [CUDAMemHelpers] tests); 0 after. Outputs byte-identical on myob,
cytc, lyso.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Both queue their work (kernels, cudaMemcpy) on the legacy NULL stream - HotPixelFinderGPU for its
download, its sums otherwise on the workers' streams - while their buffers came from the
stream-ordered pool, whose cudaFreeAsync is ordered on the thread's non-blocking allocation stream
and so after none of that work. Unlike BeamCenterFFTGPU no free has overtaken a read: every entry
point waits for its work on the host (cudaDeviceSynchronize, or a blocking device-to-host copy)
before it returns. But that holds only by convention, not for a free while unwinding from a failed
call, and compute-sanitizer --track-stream-ordered-races cannot see host synchronisation, so it
reported every reassigned merge buffer as a use-after-free - noise that buries a real race like the
BeamCenterFFTGPU one.
compute-sanitizer --tool memcheck --track-stream-ordered-races all:
- rugnux --mode scale on a myob _process.h5: 22 use-after-free reports before, 0 after
(MergeAccum/MergeAccumRange/MergeRmeas buffers freed by MergeAccum's reassignment or ~Impl).
- rugnux -e 150 -N 8 on myob: 92 before, 0 after (together with the next commit).
myob, cytc, lyso: p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz byte-identical to r4-integration. Full-run
wall time unchanged within noise (17.4/25.9/18.6 s before, 17.6/25.8/18.7 s after); the scale/merge
phase alone (--mode scale, myob) 2.23 -> 2.31 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Every kernel, cuFFT plan and copy in the beam-centre FFT engine is queued on the legacy NULL
stream, but its buffers came from the stream-ordered pool, whose cudaFreeAsync is ordered on the
thread's non-blocking allocation stream - not after the NULL stream. The free therefore completed
at once while PointShortlist's last Suppress (and, depending on what the plan teardown syncs, the
last inverse transform) was still writing, and the pool was free to hand those bytes to another
worker or return them to the driver.
The capture runs in the background into the first pass of the main run, exactly when the spot
engines are allocating their buffers, and on a loaded shared card its NULL-stream work lags. That
is the "illegal memory access" seen 0.3-0.5 s after "Processing ... images" in four runs across
three binaries and two datasets: a sticky error, reported by whichever worker synchronised next -
the bslz4 device decode, whose host fallback cannot recover from it.
compute-sanitizer --track-stream-ordered-races all over the [BeamCenterFFTGPU] tests: 16502
use-after-free reports before, 0 after; on a rugnux run it flagged the 294 MB transform buffers
freed under a cuFFT kernel. myob output (hkl, mtz, P1 mtz, unmerged mtz) byte-identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Ingest and every Run allocated per-observation arrays as sized vectors (the ingest sort keys, the
remap, the ingest-time delta_phi / partiality / image number / usability arrays, and per Run the
group ids and the group permutation), each zeroed on the main thread and faulted in page by page
before the parallel loop that writes every element of it: 0.6 s of main-thread memset in a loaded
16M profile of ComputeAsuGroups alone. They are plain arrays now, first touched by the threads that
fill them - the idiom the ingest staging buffers already use. RockingEventFrames' walk over every
raw-hkl run (0.3 s alone on the main thread at ingest) counts per block of runs on all threads;
integer histograms, so the sum is the serial one.
Item 2 of this round - building part of the ingest incrementally while the last image loop runs -
is not done: the per-frame pieces (background mean, h range, h histogram) are one of the five sweeps
over the reflections, and the sort and the grouping need the whole sweep.
Median of 3 paired runs (GPU, no other rugnux running), with the other two commits of this series:
ingest segment 0.94 -> 0.67 s on cytochrome C (17M partials), merge segments down with it. Peak RSS
unchanged (4.74 vs 4.77 GB cytochrome C GPU). Outputs byte-identical on myoglobin, cytochrome C and
lysozyme (GPU) and cytochrome C (CPU-only); logs and reports identical up to line order.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
What the main thread did alone in each written merge (adopted group, pinned pair, P1 cross-check),
each rewritten so that every sum is still taken over the same terms in the same order:
- WilsonOutliers: the observations are gathered once in resolution order and each shell is one task
(the shells are contiguous runs of that order); the reflection sort is on a total order, so it is
a ParallelSort.
- MergeAndStats: the per-group sums (Bijvoet excess, ISa asymptote, the CPU error-model means) and the
per-pair anomalous sums split the GROUPS over the threads, each walking every full and taking only
its own, so each group is summed in index order; the error-model samples, the Wilson records and
the anomalous cells are filled per observation after an in-order compaction.
- FitErrorModel: the per-bin rounds on up to 16 threads instead of 5 (one bin per task already).
- MeasureBatchDeltaCCHalf: the three fields it reads are taken out of the 80-byte fulls once, on all
threads.
- ScaledObservations (anisotropy): the rocking events walked in blocks cut at hkl boundaries and
joined in block order. Its sort stays serial: parts can tie on (hkl, image), and the event sums
depend on introsort's order of the tied ones.
Median of 3 paired runs (GPU, no other rugnux running), with the other two commits of this series:
cytochrome C tail 11.1 -> 8.5 s (P1 cross-check segment 4.0 -> 2.9 s, reports 0.76 -> 0.70),
lysozyme 9.2 -> 8.2 s, myoglobin 2.57 -> 2.40 s. Outputs byte-identical on myoglobin, cytochrome C
and lysozyme (GPU) and cytochrome C (CPU-only); logs and reports identical up to line order.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The transmitting-piece step added in a1816e905 called a pixel dim when it sat below 0.75 of its
ring's median. When the polarization factor the background is divided by is not the beam's, every
outer ring is left with a cos 2phi modulation: on one open-arm marCCD sweep (0.75 A, 90 mm, default
p = 0.99) the corrected background runs 1.5-1.8x the ring median along x and 0.6-0.75x along y at
r = 1600-2200 px. Joined, the y lobes made two pieces of 1.01 M and 0.39 M dim pixels holding only
0.17 % deep pixels (noise and a real ~20 % partial shadow) - past the absolute gate of 200 - and the
mask grew from 16,584 to 1,428,328 pixels, covering the top and bottom of the detector.
The step now fits, per 64 px radial band, the 24 sector medians with m + p cos 2phi + q sin 2phi
(sectors below 0.75 of the fit left out, three times over) and judges dim against min(1, model).
Capped at 1 the model only removes dim pixels, so a sweep with no piece keeps its mask bit for bit
and a piece can only shrink or go.
Shadow pixels (b90b377b8 -> this), same binary otherwise:
that sweep 1,428,328 -> 29,821 (the stop's stick extension kept); arm sets 7atg 724,032 -> 723,708,
6oel 223,087 -> 216,126, 7rji 311,951 -> 309,816, 9fhc 173,597 -> 171,372, 5ebi 793,217 -> 785,477,
6zqr 289,324 -> 251,764, 5j23 530,541 -> 526,115, metformin 341,368 -> 231,123 (two speckled
pieces in the high-angle lobe dropped, the arm below the beam kept).
Merges: 6oel, 7rji, 6zqr, metformin unchanged in space group and resolution, R_meas/ISa within
0.1 %/0.2; 7atg fails either way on this base (the beam-centre defect fixed on atg-arm-centre).
The sweep above: same space group, resolution, CC1/2 and ISa (6.72 -> 6.73); R_meas 21.7 -> 22.7 %,
because the spurious mask also hid part of a real ~20 % partial shadow at the top right that neither
version masks (it is above penumbra_ratio).
New test ShadowFinder_AWrongPolarizationFactorIsNotAnArm (unpolarized background, 0.99 stated,
small absorbing speck in the dim lobe) masks 4547 px before and 0 after; all *Shadow* tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
On a 16 Mpx detector the pre-scan's main thread spent most of its kernel time faulting in memory.
GetMask made some thirty per-pixel arrays (1.6 GB) as value-initialised std::vectors, so the calling
thread zeroed - and took the page faults of - every one of them before the parallel passes ran: about
0.75 s on a 16M sweep. They are now Planes (NoInitAllocator vectors) that the ParallelChunks writing
them first touches; each pass writes every element it used to leave at the initial value, and the
three that are filled piecemeal start from a parallel fill.
The projection was also reduced three times - for the ring-centre fit, for GetMask and for the
beam-centre capture - each a 360 MB Download into fresh memory. It is reduced once on the first read
and kept until the caller releases it after GetMask, and PreScan shares the one mean projection
read-only (shared_ptr<const>) with the ring-centre fit, the diagnostic picture and the capture
instead of copying it into each.
Output identical (p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz) on the three GPU reference sweeps;
ShadowFinder and BeamCenterFromBackground tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
a1816e905 adds the partly transmitting holder-arm pieces to the beam-stop mask, and the
background beam-centre fit read that mask too. A thin arm running into the beam, masked, takes
cells out of the rings it crosses; which radial bins stay above the sector-coverage gate then
changes as the trial centre moves, and a walk seeded beside the arm (the FFT capture sat 48 px off)
limit-cycled on that edge and stopped 44 px short (sigma 0.34 -> 2.27 px). On an open-arm set whose
header centre is 171 px out, the measured centre no longer indexed (60/60 -> 0/60), the run kept the
file's centre and merged P1 at CC1/2 0.004 instead of P212121 at 0.995.
The arm's dimming is multiplicative in azimuth, which the fit's per-sector amplitude already
absorbs, so the fit now uses the mask without those pieces (ShadowFinder marks them TRANSMITTING);
integration still masks them. The fitted centre is back to its a0518abe6 value on every set checked
(the failing one plus four arm sets); the failing set merges P212121, CC1/2 0.995, ISa 22.6.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
One chain runs at a time there, after the parallel candidate solves, on 1-3.5 cores of an otherwise
idle machine. Ceres sums per-thread cost and gradient pieces, so the thread count may move a result at
rounding level; measured, it did not: identical outputs on the four reference sweeps and all 22 smoke
sets, and a sweep that indexes poorly (a chain after every spot-budget rung) went from 96 to 62 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
A pass's P1 merges - the search merge, the all-observations merge, the P1 cross-check - each ran the
partial scaling loop from scratch on identical inputs (about 5 s each on a 16M CPU-only run). The loop
restarts from corr_ingested, fits every frame from its own observations and the group means alone,
and writes G and corr only on the frames it fits; everything else it reads is fixed at Ingest. So
over the same ASU grouping and settings (space group, Friedel, resolution limits, partiality floor,
window, iteration cap) it ends where it ended before, and the frames, G and corr it left are kept for
the next Run to take. Frames it does not fit keep whatever G they had, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Not bit-exact: the radial offset d.rad is ddx*ux + ddy*uy inlined at two sites, which the compiler may
contract to an FMA differently, so the clip's bin (lround(r0 + rad)) moved on a few pixels of one
sweep's CPU run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The MTZ points at the experiment's space group, and the background build used a copy that died with
the task: the file was written through a dangling pointer (a verify-mode run failed with a garbled
space-group name; the ordinary runs happened to read the freed memory intact).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
With candidates, the first pass already tested only the 32-column blocks within reach of a candidate,
but still slid the horizontal window across every column of every row. It now slides it only over
the runs of needed blocks, starting each run from the vertical sums it covers - integers, so the same
window sums - and sets the bits there directly; the rest stay 0 as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The sigma-clip pass recomputed the stencil distances over the whole box to find the same background
ring pixels pass A had just summed. Pass A now keeps their values and radial offsets in its own
order, and the clip runs over them - the same pixels, the same sums.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The four Kabsch reweighting iterations each walked the reflection's grid again with the same bounds,
validity and ownership tests the p_valid pass had just applied. That pass now keeps the profile value
and background-subtracted count of the pixels the fit reads, in grid order, and the iterations run over
them - the same terms summed in the same order (about 6 s of a CPU-only 16M run).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
