Port the four fulls-walking reductions of MergeAndStats to the GPU, over the fulls
group-CSR already resident from scale-fulls: the per-group inv-var mean + leverage-
corrected error-model samples, the merge accumulate (inv-var sums + deterministic
half-sets, error-model-corrected sigma, with outlier rejection), and R_meas + the
per-shell usable count. The host keeps the parts that don't parallelise cleanly or are
tiny: the I2-sort + 16-bin (a,b) median fit, the per-group reject median (a per-group
median is awkward on the GPU - cheap on the host from the GPU cnt), the merged export,
the shells and the gemmi completeness. Only per-group arrays (~55k) + the samples
(~n_fulls, for the fit) come back - the fulls are not re-walked on the host.
Device HalfForImage (splitmix64) + IceRingIndex mirror the host; the corrected-sigma
uses (b*I_for_b)^2 (not b^2*I^2) to match the host rounding; the R_meas usable count
requires finite d (the host counts only fulls with a valid shell, and a group's fulls
share d, so the shell is assigned per group). Gated on fulls_resident (GPU
combine+scale-fulls active); reject is fully supported so it runs for the default
rot3d command.
merge+stats ~0.49 -> ~0.37s, taking RSM on lyso to ~0.78s (was ~0.91). Validated across
the battery: 15/15 deterministic crystals bit-identical to the CPU path (SG / ISa /
CC1.2 / completeness / total-obs, and the exact outlier-reject count), only EP_cs_01-24
noise wobbles. The em-sort + a,b fit are the remaining host floor. Non-CUDA build
unaffected (use_gpu_merge is always false there).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply smooth-G's corr adjustment on the device (a small kernel: corr[i] *=
ratio[frame[i]] for flagged frames, double-then-float, matching host SmoothG) so the
per-image corr never leaves the GPU: it now stays resident through scaling ->
smooth-G -> per-frame CC -> combine, and across the two space-group passes exactly as
the old host round-trip did. The host only builds the tiny per-frame ratio (g/g_smooth
via the extracted ComputeSmoothGWindow) and refreshes host partials[].corr solely for
the CPU-combine path (JFJOCH_RSM_CPU_COMBINE or the diagnostic dump).
This drops the post-scale GetCorr and the two SetCorr re-uploads (~3x25MB/pass) plus the
6.3M host corr-adjust loop: scale-partials ~0.21->~0.10s and the smooth+combine region
shrinks, taking RSM on lyso to ~0.91s (was ~1.47s with phase-1-only, ~1.71s full-CPU) -
under the 1s target for this crystal; merge+stats (~0.49s) is now the dominant chunk.
Bit-identical (GPU smooth-G == host SmoothG on the resident corr); validated across the
battery (15/15 deterministic crystals bit-identical to CPU across default / CPU-combine /
NO_GPU, only EP_cs_01-24 noise wobbles). Non-CUDA build unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-threaded ReduceGroupMeans over the 6.3M partials (~0.07s/2-pass) and the
per-frame diagnostic CC now run on the resident partials on the GPU: after SmoothG,
the smoothed corr is uploaded once (and left resident for the combine, dropping the
combine's redundant re-upload), then the post-smooth group means (reusing the scaling
reduce) and the per-frame Pearson CC (a new one-block-per-frame kernel) run there and
only the tiny per-frame cc/cc_n come back. FinalizePerFrameScale is split into
ComputePerFrameCC (host reference) + the writeback; the GPU path uses ComputePartialCC.
The per-frame CC is diagnostic only (the per-image scaling table), so the tree
reduction's ~ulp difference from the CPU is immaterial and it does not touch merged
intensities. smooth+CC region ~0.10s GPU vs ~0.15s CPU on lyso. Validated across the
battery: 15/15 deterministic crystals run-to-run deterministic and merged output
bit-identical to the CPU path (only EP_cs_01-24, unindexable noise, keeps its benign
error-model-b wobble). CPU fallbacks (JFJOCH_RSM_CPU_COMBINE / _NO_GPU) unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Scale the combined fulls (Unity model) on the device so they no longer round-trip
between the combine and the merge: after the GPU combine, build the fulls' per-frame
and per-ASU-group CSRs on the host from just the small key arrays (f_frame/f_group)
with a deterministic counting sort - no GPU stable-sort - then scale in place and
download once.
The four scaling kernels are reused unchanged except FitPerFrameGKernel, which gains
an optional `perm` argument (null for the partials, whose arrays are already
frame-contiguous; a frame-grouping permutation for the emit-ordered fulls) so the
fulls are scaled without a physical reorder. The Unity model falls out of giving the
fulls all-ones partiality/rlp/zeta (coeff = mean), so no other kernel changes and the
committed phase-1 partial-scaling path is bit-identical (perm == null -> idx == i).
Validated across the rotation battery (JFJOCH_RSM_GPU_COMBINE=1): all 15 deterministic
crystals stay run-to-run deterministic and their merged output is bit-identical to the
CPU path (SG/ISa/CC1.2/completeness). The lone exception is EP_cs_01-24 (CC1/2 2%,
R_meas 379% - unindexable noise): merged intensities/CC/completeness match exactly, but
the ill-conditioned 16-bin error-model b fit amplifies the ~1e-7 scale-fulls rounding
to ISa 10.6 vs 10.8 - benign, same class as the accepted phase-1 GPU rounding. The 3
upstream-nondeterministic crystals vary as before (GPU-prediction overflow, not this).
Scale-fulls drops from ~0.09s to ~0 across the two passes; combine+scale-fulls region
~0.32s GPU vs ~0.46s CPU on lyso. Still opt-in (fulls are downloaded for the host merge;
the win grows once the merge/error-model also stay resident).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port Combine() (partials->fulls) to CUDA, mirroring process_rawrun bit-for-bit:
one thread per raw-hkl run splits its usable partials into rocking events (frame
gap <= 2), pools background, seeds F, runs 3 de-biased Poisson reweights and the
capture-uncertainty term. Emission is deterministic - a count pass, a host
exclusive prefix sum for per-run offsets, then an emit pass at those offsets - so
fulls come out in raw-run-major/event order, identical to the CPU path; both pass
instantiations share the same arithmetic so count == emit exactly. Dmax/Dmin/Fmax
reproduce std::max/min NaN semantics (not fmax) for parity.
Validated across the 18-crystal rotation battery: all 15 deterministic crystals
(P1/P2/C2/H3/I23/P41212/P222/P422) match the CPU combine exactly on SG/ISa/CC1.2/
completeness and run-to-run (fulls count bit-identical); the 3 upstream-nondet
crystals vary from GPU-prediction overflow, not the combine.
Gated opt-in behind JFJOCH_RSM_GPU_COMBINE (default = CPU combine): combine alone
is timing-neutral because the shared 1.2M SortFullsByFrame std::sort dominates and
the fulls round-trip adds a copy - it only pays off once the fulls stay resident
for scale-fulls + merge. Also add JFJOCH_RSM_NO_GPU master switch to force the CPU
fallback (incl. phase-1 scaling) from one binary for A/B parity. SortFullsByFrame
extracted from the Combine tail and shared by both paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First stage of moving the rotation scale/merge onto the GPU. The per-frame partial-scaling loop
(inverse-variance group-mean reduction -> robust per-frame IRLS G -> corr update, x scaling_iter)
now runs in RotationScaleMergeGPU (.cu) when a GPU is present; the CPU loops remain the fallback.
The host keeps the one-time raw-hkl sort and the per-space-group gemmi ASU keying, and hands the
GPU a group-ordered permutation + CSR so the per-group reduction is a DETERMINISTIC segmented
reduction (one thread per group, fixed order, no atomics) - preserving the run-to-run determinism
just won on the CPU path (a float atomicAdd reduction would have re-introduced jitter). Reduction is
one-thread-per-group (groups average tens of obs, so a block-per-group wastes threads); the IRLS is
one block per frame with a deterministic shared-memory reduction.
Validated: bit-identical to the CPU path and deterministic run-to-run on lyso/cytC/Ins_H/pding
(P41212 ISa 7.8 CC1/2 99.7%, etc.). The scaling kernels are ~7x faster than the CPU compute
(~36 ms for 3 iters vs ~0.28 s); end-to-end scale/merge ~2.0 -> ~1.5 s. The remaining gap to the
<1 s target is the per-pass host round-trip (corr down/upload for the CPU combine + per-SG group-CSR
rebuild); phase 2 keeps the data resident by moving the 3D combine and the merge/error-model onto
the GPU too, so nothing round-trips.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
leonarski_fandClaude Opus 4.8