The R_meas accumulators per group and the device's per-group sums were eight
group-length locals plus a 56-byte record per group, value-initialised on one
thread for every merge that reports statistics. They are now members, reset in
parallel, and the device branch's pass that assigns each group its shell and
copies its sums runs on all threads; the per-shell observation counts are
integers, gathered per block. The shells' R_meas sums are still added in group
order, as before.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Which groups are written is decided first (KeptIndices, in group order), then
each written group's MergedReflection is formed on its own, on all threads,
into a result sized once - where the serial loop pushed back up to millions of
records, growing a vector that each merge allocated afresh. The same records in
the same order.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The rejected flags and the observation keys and halves (per full), the merged
intensities (per group) and the Wilson observation records were locals: on a
large cell every one of the run's merges allocated, page-faulted and zeroed
them again on one thread - hundreds of megabytes per merge on 8qaw. They are
now members, only resized, and reset with ParallelChunks where a pass needs
them reset. The error-model samples are no longer cleared before the device
download overwrites them (the host path still clears before it appends).
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The fused adaptive spot finder's azimuthal profile (and the standalone AzIntEngineGPU) summed the
corrected pixel values - pixel times a float correction - with float atomicAdd, in shared memory
and then across blocks. Float addition is not associative, so the per-ring sums depended on the
order the threads and blocks happened to arrive in, and on the grid size chosen for the GPU. The
profile, and the per-image background estimate read from it, moved in their last bits between two
runs of the same command: on a thaumatin JUNGFRAU 4M sweep one bkg value in _plot.txt differed in
the 4th digit between runs. The raw ring sums that drive detection were already exact integers;
only the reported profile was affected.
Each corrected value (and its square) is now quantised to 2^-12 on its own and summed as a signed
64-bit integer, the convention the raw ring sums and the Bragg-integration profile already use.
Integer addition is associative, so the sums are the same whatever the order or grid.
The repeat test of the fused finder now also requires the profile and its std to repeat bit for
bit (it failed on the old kernel at the first repeat); the AzIntEngineGPU test re-runs 20 times.
Fused finder benchmark: 0.313 -> 0.324 ms/frame.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With the fulls resident (the GPU combine ran), ApplyCellSurface no longer builds
its terms on the host and uploads them: SurfaceBuildTerms makes them from the
resident fulls, this surface's cell per full and the current corr (the only two
uploads) - which fulls give a term, the terms in fulls order, the group CSR over
them and each subset's per-(block, cell) segments - and SurfaceShellWeights the
resolution gauge's inputs: the shell edges (order statistics of 1/d^2 read off a
sort), each group's shell and each (cell, shell) bin's weight. Every order the
host made is a stable counting sort, which a stable radix sort of the indices
reproduces; the per-bin weights are summed one bin per thread in fulls order
with the host's rounding. The existing exact surface kernels read them as
before. Without resident fulls (the observation dump) the host path runs.
p.mtz md5 and the surfaces' log lines unchanged on myob, cytc, thau, kdp, 8a1a
and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With two GPUs or more the two-pass quality guard's merge of the pre-pass
frames gets a scaling engine of its own on device 1 (RotationScaleMerge::
SetGpuDevice), and the main engine's device ingest and first merge no
longer wait for it to hand the card back. The first merge waits only for
the guard's ingest, the last time the guard reads the outcomes: that merge
writes per-frame fields (mosaicity among them) back into them.
Neither merge reads what the other writes, and a merge gives the same bits
on any card of one model, so only when the guard runs changes. With one
GPU, and in the CPU build, the order is the one it was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
std::thread::hardware_concurrency() counts every CPU of the node, so a job
given a CPU set (taskset, a cpuset, a Slurm allocation) or a container CPU
quota started one thread per node CPU: the default -N, the shared worker
pool and every "0 = all threads" default. AvailableCpus() (ThreadAffinity)
counts the start affinity mask, capped by the cgroup CPU quota (v2 cpu.max,
v1 cfs_quota/period, the process's own cgroup and then the mounted root),
read once. Outside Linux it is hardware_concurrency(), so the viewer tree
stays portable. It replaces every hardware_concurrency() call outside the
tests and the vendored pocketfft.
Only how many threads run changes; the passes split their work by n alone,
so the results do not depend on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
FitErrorModelGPU puts the samples in the host's rank order with four stable
radix passes, so the equal-count bins hold the same samples; computes every
normalised deviation with the host's rounding, so every bin median is the
host's to the bit; and hands the per-bin terms to the host's own update step,
now ErrorModelUpdate (a pure extraction from FitErrorModel). The one thing
that differs is the order each bin's two sums are added in: a fixed tree on the
device, the order nth_element happened to leave the bin in on the host. a and
b^2 can therefore differ from the host fit in their last bits - measured on two
synthetic pools: relative difference 2e-16 and 3e-13 in a, 5e-16 in b^2. The
device fit is deterministic (same pool, same bits).
p.mtz md5 nevertheless unchanged on myob, cytc, 8a1a and 8qaw (GPU build): the
difference does not survive the float rounding of the merged sigmas there. A
knife-edge decision elsewhere could still see it, which is why this is kept as
its own commit.
The host fit took 0.4-0.6 s per fit on the large merges, about half of it the
initial equal-count split, with only 16-way parallelism in the iterations.
Device peak about 72 bytes per sample (the rank sort; each stage's buffers are
freed when it is done); the free memory is checked first and a device without
it stops with a message naming the CPU build.
Kept last on the branch so that it can be dropped on its own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every full's observation key and its half - the parity of the key's rank among
its group's usable fulls - are integer work, so the device gives the host's
halves exactly: one thread per full for the keys, one per group for the ranks.
The host receives both (the anomalous statistics rank the same keys again within
each Bijvoet cell) instead of walking every full's record for them.
p.mtz md5 and p.cif unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The strong-reflection asymptote is read from per-group sums of I*corr, its
square and the counting variance at the group mean - a walk over every full's
record on the host. In the GPU build a kernel takes them, one thread per group
in index order with the host's rounding (CountingVarianceExact); the host keeps
the strong-group selection and the median.
p.mtz md5, p.cif and the error-model lines (which print the asymptote)
unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The merges of the largest sets run with 12-13.6 GB of a 16 GB device already
taken by the resident observations, so the transient footprint of this pass is
what decides whether it fits. Release the sort keys as soon as the d order is
known and the noise ratios once their per-shell medians are read - on the
device, rather than downloading every sorted value. Peak about 96 bytes per
observation; the free memory is checked first and a device without it stops
with a message naming the CPU build.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The multiplicative spread of the outlier band is the weighted median of
(|ln(I/median)|, weight) pairs, one per usable full, and its sort was the bulk of
that step. SortPairsGPU sorts them on the GPU (by the second key, then stably
by the first) and leaves the sequence std::sort leaves: two pairs that compare
equal are the same two numbers. The values themselves (a logarithm) and the
running sum stay on the host.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build); a Catch2 case
checks the device order against std::sort.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The per-group sums the widening is measured from - each hand's inverse-variance
mean under the error model's sigmas, the group's d and the d range - are a walk
over every full's record on the host. In the GPU build a kernel takes them, one
thread per group in index order, with the host's model sigma to the bit
(ModelSigmaExact: every product rounded on its own); the host keeps the shells
and their medians. The host path is the same sums, now kept per group so both
paths hand the shell step one array.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
MergeEmSamples now forms each sample whole on the device - the counting
variance at the reflection's expected intensity included, every product rounded
on its own as the host build rounds it - and compacts them in fulls order with a
scan, so the host receives exactly the vector it used to build. That replaces
the download of three full-length double arrays and a flag array, and the host
pass that compacted them while reading every full's eighty-byte record.
p.mtz md5 and the error-model lines unchanged on myob, cytc, 8a1a and 8qaw
(GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
WilsonOutliersGPU runs the test of WilsonOutliers.cpp step for step on the
device and returns the same result to the bit: the two orders the host sorts
into are total orders, so a stable radix sort of the indices reaches the same
sequence; every sum is taken by one thread over its shell or reflection in the
host's order; each product the host rounds on its own (-ffp-contract=off) is
rounded with __dmul_rn; medians are read off sorts; and the logarithms (t, z,
u, the tail scale) are taken on the host from downloaded counts and medians.
MergeAndStats calls it where the merge itself is on the device.
8a1a final P61 merge, "rsm merge and statistics" 4.35 -> 2.82 s (the test runs
twice there, once per error-model fit). p.mtz md5 unchanged on myob, cytc,
8a1a and 8qaw (GPU build). A Catch2 case checks the device result against the
host one on a population that reaches every branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The automatic cut fitted its logistic to one Pearson CC1/2 per bin, which
has no bound on what one pair can do to it. In a weak shell a single
reflection with a huge intensity in both halves - an artefact the halves
reproduce, or one observation with a sigma fifty times the shell's - carries
nearly all of the covariance. Each bin's reflections are now dealt into up
to seven groups by their indices (groups of at least ten reflections), and
the bin reads the median of the groups' CC1/2. On a clean shell every group
estimates the same CC1/2, so the median is the Pearson value; an outlier
takes one group with it and the median does not follow. Only the cut
decision changes; the reported shell table is unchanged.
Found on a weak split myoglobin crystal: one reflection carried 50-99 % of
the covariance in most fit bins past 2.6 A (e.g. 43647 in a bin with median
I 1.3), reading them at CC1/2 0.93-0.98 where the median of groups reads
0.2. The cut swung between 1.21 and 2.03 A with --prepass-fraction; it is
now 1.85 A at 0.5 and 1.89 A at 1 (XDS 1.51 A, rc.174 1.96 A).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pixel that reads at most half of its own sector's level on average, by more
than the bound a hot pixel's excess has to clear, is now masked as dead - the
mirror of the hot rule. Nothing but a defect keeps a pixel dark frame after
frame, and every reflection whose disk covers it is integrated short.
The finder now also runs on CCD sweeps (no sensor depth): a dead CCD pixel
reads its offset, far below any ring with background, and the hot rule's
ten-fold ratio does not depend on Poisson counts either.
Found on 5jk4 (ADSC Q4, 2304x2304): the last row and the first two columns
read the offset (~20 ADU against ~60-100 around them) and were unmasked.
Reflections touching them merged to large negative intensities, and one
observation at y=2298 with sigma 10334 swamped the outer shell's half-set
CC1/2 (3.9 %) and the overall CC1/2 (0.786; 0.998 at rc.174). With the
6630 dead pixels masked the fall-off is clean: CC1/2 0.998, cut 0.96 A
(fit 1.01 A), at --prepass-fraction 0.5 and 1 alike. Counting-detector
sets with low background are untouched (myoglobin X06DA: 0 dead).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Rotation prediction enumerated the whole (2h+1)(2k+1)(2l+1) box every frame and kept the ~1% whose
rotation solution lands near the frame. A solution within phi_limit of the frame needs
|f| = ||p0|^2 + 2 S0.p0| <= 2 |S0_perp| |p0| phi_limit, and along an (h, k) column f is a quadratic
in l, so the l that can pass are at most two intervals. Those are walked - widened by 1% in f and by
a whole index at each end - in the same order and with the same arithmetic; the rest the phi test
rejected anyway. Without a phi limit (min_zeta 0) the whole column is walked as before. The GPU
kernel is unchanged.
A new test predicts 1000 random lattices, geometries, axes and rocking widths both ways and
requires the same reflections in the same order, every value bit for bit (30000 cases passed once
locally). CPU build, two interleaved warm runs: 8a1a output loop 241.5 -> 216.3 s, pre-pass loop
90.7 -> 73.8 s, whole run 427.0 -> 384.2 s; 8qaw (with the gzip change before this) loops
116.6 / 251.3 -> 94.4 / 206.8 s, user CPU 14700 -> 12718 s. GPU build within noise.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ApplyCellSurface copied and sorted every term's 1/d^2 to read 19 order statistics off it, for each
of the four surfaces of the final merge. The edges are now selected: one pass counts the values
under each top-16-bit pattern (a non-negative float orders as its bits), a second collects the few
bins the ranks fall in, and nth_element picks the value inside each. An order statistic is a value,
so the edges are the ones the sort gave.
8a1a, GPU build: the shell set-up of the four surfaces 0.55 -> 0.26 s (instrumented run), the
correction-surface window 6.32 -> 6.12 s and the final merge 11.03 -> 10.82 s (two interleaved warm
runs each); whole run 55.78 -> 55.45 s. 8qaw surfaces 2.95 -> 2.83 s.
Tried in the same window and not kept, as they measured nothing: the serial stable counting sorts
(surface group order, decay terms, fulls CSRs, delta-CC1/2 frame order) on all threads; delta-CC1/2
workers uncapped from 8; measuring two rounds of the edge search at once. The delta-CC1/2 window
(2.4 s) is bound by its longest chain of ranges, one serial accumulation, not by the worker count.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged scaled fulls the final merge hands back for the .hkl were gathered by one thread
pushing back over every full. Which fulls is now KeptIndices (the same list, in the same order),
and each record is made side by side.
The same records in the same order: p.mtz and the .hkl byte-identical (myob, thau, cytc, kdp,
cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a, 8tyy, CPU build myob). 8a1a, no other
load, interleaved: the P61 merge's statistics stage 3.94 -> 3.71 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
RotationScaleMerge::Ingest is now IngestHost (read the outcomes, sort, build every host array,
smooth the geometry and recompute the partiality) followed by IngestDevice (allocate the GPU's
arrays and stage them). The host half reads the outcomes and nothing else, as the post-refinement
and the quality guard's merge do, so on the canonical pass it runs on its own thread beside both;
the device half runs where the ingest used to, after the guard has handed the GPU back, so the two
engines never hold the card at once. The device feed now stages the recomputed partiality in the
same slices as every other field, straight from the ingest array, instead of in a pass of its own
after the rest; the values on the device are the ones they were.
8a1a, no other load, interleaved runs: the first merge (P1) ends at 10.9 s after the image pass
against 11.3 with only the guard overlapped and 12.7 on the rc175 base. p.mtz byte-identical on the
gate sets (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a,
8tyy, CPU build myob); --model, stills, calibration, --mode scale (GPU and CUDA_VISIBLE_DEVICES="")
outputs identical; --no-merge / --mode azint / --write-process-h5 _process.h5 differ only in the
datasets that differ base-vs-base (azint sums, bkgEstimate, profiling times, dates, git).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ChooseLogScaleSmoothing picks the smoothing of the fulls' per-frame log scale by leaving out
every fifth block of frames: 25 lambdas x 5 folds, each a banded solve over the whole sweep, run
one after the other on every iteration of every fulls scaling loop. The 125 solves are independent,
so they now run on the engine's threads; the held-out error is still added up per lambda over the
folds and frames in the order the serial loop added it, from the same curves, so the lambda chosen
is the same. nthreads defaults to 1 for other callers (the test).
8a1a: the six "combine and scale fulls" stages of the tail 4.7 -> 3.6 s. p.mtz byte-identical
(myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a); [RotationScale]
and the scale_merge test files pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
OperatorCorrelation flattened, normalised (two serial sorts) and hash-indexed the whole merge for
every operator it was asked about. The metric sub-lattice ask scores each two-fold of the metric
against the same P1 merge, and the twin-law report each law against the same expanded merge, so
both now build an OperatorScorer once and ask it per operator; OperatorCorrelation itself is a
scorer used once. Same prepared merge, same pairing: the scores are the ones it gave.
On 8a1a (hexagonal metric over a 6-fold point group) the stretch between the all-observation
search and the adoption goes from 1.37 to 0.39 s. p.mtz and the twin-law lines byte-identical
(myob, thau, cytc, kdp, 8a1a). The two call sites in rugnux/RugnuxScaleMerge.cpp change by a line
each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The guard's merge of the first prepass_images frames read nothing the post-refinement
writes and the post-refinement reads nothing the guard writes, but the two ran one after
the other at the start of the tail. The guard now merges the first frames of the outcomes
in place (RotationScaleMerge::SetFrameLimit) with the per-frame scale not written back,
instead of moving the reflections into a copy whose write-back was thrown away, and it
runs on its own thread from just before the post-refinement; its log lines are held and
replayed where the guard's result is read.
Same observations in the same order, same settings: p.mtz is byte-identical (myob, thau,
cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a). On 8a1a the 3.7 s
guard is now inside the post-refinement, which itself goes from 4.5 to 6.0 s under the
shared load: about 2 s off the tail.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The P1 cross-check (<prefix>_P1.mtz) and --no-p1-crosscheck are gone. The
tail waited for that merge after the reflection files were written: 12-18 s
on a large rotation set (7200 frames), ~2 s on a small protein. A user who
wants a P1 merge re-runs with -S 1.
--finalist-ledger keeps working: it now starts a P1 merge of its own beside
the analyses (made only when the flag is given) and folds the ledger from it.
The twin-immune zone evidence, which was read on the cross-check merge, moves
with it and is reported only with --finalist-ledger.
The unmerged MTZ was built in the shadow of the cross-check merge; it is now
built beside the reflection files, from the point where the model validation
has settled the indexing.
WriteMtzReflections loses its amplitudes flag (only the cross-check wrote an
intensity-only MTZ). The merged outputs (p.mtz, .hkl, _unmerged.mtz) are
byte-identical to the base on the gate matrix.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A TimingMark (common/Logger.h) logs "TIMING <section> <seconds>" at debug level when a section
ends, so the lines go to <prefix>_log.txt and stay off the terminal. Marked: each pass of
RunPipeline (pre-pass, probe, output) and its first-pass indexing, image pass, scale/merge/symmetry
phase, output files and census wait; inside the scale/merge phase every merge (labelled by its
role: P1, P1 space-group search, P1 all observations, the adopted group, P1 cross-check, quality
guard), the post-refinement, the space-group search and its two SearchSpaceGroup arms, adoption,
analyses, the P1 cross-check wait, model validation and writing; inside RotationScaleMerge the
ingest and, per arm, partial scaling, combine and fulls scaling, correction surfaces and the merge;
the console's steps; and the results report. tools/rugnux_timing.py tabulates the tail of a log.
Only the log changes: the outputs are byte-identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pure move of three method files out of rugnux/: SpotWidth (spot width and
bandwidth, which set the integration radius) joins JFJochBraggIntegration, which now
links JFJochImagePreprocessing for the ImagePreprocessorBuffer it reads;
SpindleCuspLoss and LeftoverLattices join JFJochIndexing. Only include paths and CMake
lists change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pure move. ModelValidation, RigidBodyRefine, RigidBodyGPU, ModelFFT, ModelGrid,
ModelScaling, ModelMaskGPU, ModelScaleGPU and SigmaA - everything that works on an
atomic model - become the JFJochStructureRefinement library, linked by
JFJochImageAnalysis. WriteModel (the placed-model mmCIF/PDB writer) goes to writer/
as its own small JFJochModelWriter target, so JFJochWriter, which a writer-only build
compiles, does not gain a gemmi dependency. Only include paths and CMake lists change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A pure move: HotPixels.{h,cpp} and HotPixelsGPU.{h,cu} are compiled into
JFJochImageAnalysis (like beam_stop/) instead of JFJochRugnux. Only include
paths and the CMake source lists change; the objects are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The online broker cancelled a collection with "Crystal lattice is coplanar and has no reciprocal
cell". The candidate filter in PostIndexingRefinement (used by FFT, FFTW and ffbidx) tests lengths,
angles and spot count only; three rows 120 deg apart in one plane pass all of them, and spots along
the plane normal index as (0,0,0). Such a candidate - from ffbidx, or one the least-squares
refinement walked flat - went on to LatticeSearch and XtalOptimizer, which refuses it and leaves it
in place, and then reached Astar() in AnalyzeIndexing/prediction, which throws; the receiver turns
that into a cancelled acquisition.
- Refine() rejects a candidate below MIN_BASIS_VOLUME_FRACTION, like any other bad candidate.
- XtalOptimizerInternal builds the refined lattice first and counts a coplanar result as a failed
refinement, writing nothing back (the residual's volume clamp lets a solve end there and report
success).
Test: PostIndexingRefinement_CoplanarCandidateIsRejected threw the exact message before the fix.
rugnux p.mtz unchanged on the three in-house reference sets at --prepass-fraction 1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
--estimate-beam-center was opt-in, off by default and used by no battery run. Removed with it: the
half-turn spot-symmetry estimator (image_analysis/geom_refinement/BeamCenterFromSpots and its tests,
used nowhere else), its pre-scan branch and paired-frame sample, the sparse-sweep extra reads, the
BEAM_CENTER_* constants used only there, beam_center_placed_, the spindle fit that only ran inside
it (--fit-spindle / --no-fit-spindle, ProcessConfig::fit_spindle; its estimate was only logged),
the viewer's "Measure beam centre before indexing" job option and the RugnuxCommandLine emission.
The background centre (FindBeamCenter on the pre-scan projection), the asynchronous beam-centre
capture, the beam-centre check that indexes at both centres and the measured centre's bound on the
post-refinement are unchanged, and so is the pre-scan sample for the shadow, the defective pixels
and the spot width. Warnings that pointed at --estimate-beam-center now point at --beam-x/--beam-y.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The ladder stepped the beam centre a pixel at a time after a rotation first pass that indexed fewer
than half the validation frames. A battery with it switched off changed no output, so it goes: the
option, its usage text, ProcessConfig::beam_center_search_pxl, the once-per-run flag that kept
probes, walks and later passes from re-running it, and the docs. The background centre
measurement, the beam-centre check that indexes at the measured centre, the merge-judged arms and
the post-refinement of the centre are unchanged. rugnux now rejects --beam-center-search as an
unknown option.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
When a chain needs more fulls than the arrays hold, fulls and fulls_staging are released before the
resize. The gather overwrites every field, so growing in place only copied the previous chain's
records into the new allocation to be overwritten.
The expected saving (a few tenths of a second on 8tyy) is below the run-to-run noise of the wall
clock: 8tyy median of 3, 127.0 s without vs 129.6 s with, under different machine load. 8qaw max RSS
21.9 -> 21.7 GB.
p.mtz md5-identical to rc175 on myoglobin, cytochrome C (also at -N 4 and -N 8), thaumatin, 8a1a,
8tyy, 9gdj and 8qaw.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The space-group search was almost entirely single-threaded. The per-reflection work now runs in
parallel over fixed chunks, while every floating-point accumulation stays serial and in the same
order as before, so the output is bit-identical.
- AnalyzeLTestUnderMerge takes nthreads (default 1): the usable test, the orbit means and the
partner search run in parallel; the 20 resolution-shell passes become one walk.
- Stage B ClassifyReflection: one byte per reflection, candidates scored in batches of nthreads.
- FindRowModulation: cone test and cosines in parallel.
- PrepareMerge: parallel flatten; the sorts use inline {d, index} keys with the same comparison.
Measured: SearchSpaceGroup 14.7 -> 4.8 s on 8tyy (instrumented build). Wall clock, median of 3
interleaved runs vs rc175, timed inside the GPU lock (-march=x86-64-v3, CUDA): 8tyy 136.9 -> 127.0 s,
8a1a 81.8 -> 80.4 s, cytochrome C 18.8 -> 18.9 s (unchanged). 8qaw max RSS 21.7 -> 21.9 GB.
p.mtz md5-identical to rc175 on myoglobin, cytochrome C, thaumatin, 8a1a, 8tyy, 9gdj and 8qaw
(GPU build), and on myoglobin and cytochrome C in a CUDA-off build.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Floating-point summation order only (tier F): the seven global sums of RefineDecay (slope fits,
held-out disagreements), the per-batch sums of FitRelativeBCurve, and the per-shell and
per-(batch, shell) sums of MeasureRadiationDamageB were single-threaded walks over every full.
They are now FixedBlockSum: the split depends on the number of fulls alone (ReductionBlocks,
ParallelBlocks), the block partials are merged in block order, so the sums have the same bits at
every thread count and on every machine - but not those of the serial walk.
Checked: p.mtz, p_P1.mtz, p.cif, p.hkl and the report identical to production on myob/cytc/thau
(GPU) and myob (CPU), and identical between -N 4/8/default (GPU cytc, myob; CPU myob). On these
sets the decay correction is not adopted (negligible or not cross-validated) and the monitor's
printed values do not move, so the F-tier change is not visible in their outputs; a set where the
decay correction is adopted will move in the last bits of corr.
Clean branch: p.mtz md5 identical on myob/cytc/thau/8a1a (GPU), cytc -N 8, myob -N 4 (CPU).
Measured (prototype with spans, 8a1a, 2 interleaved pairs, load 8-10): decay sections 2.70 -> 1.08 s,
radiation-damage monitor 1.04 -> 0.86 s, RSM total 37.1 -> 35.6 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E): the device gets the same group ids and the same group-ordered permutation
(ascending observation index within a group) the host's counting sort produced.
ComputeAsuGroups still finds the groups on the host (one ASU reduction per raw-hkl run, the
(key, run) sort, the dense ids, the representatives - now unpacked from the packed key instead of
a second ASU reduction). With the GPU resident it then hands the device only the runs in group
order (two arrays as long as the runs, not the observations): BuildGroups stamps every
observation's group from its run (the device evaluates the same finiteness test on the same
uploaded floats), counts each group's observations, and fills each group's segment and puts it in
index order. The host no longer builds or uploads the two observation-length arrays; the device
keeps its per-observation group array across Runs instead of reallocating it beside the old one.
SetGroups (host-built upload) is gone; the CPU path keeps the host histogram CSR.
Measured (prototype, GPU RTX 5080, Run-span sections): ComputeAsuGroups summed over a run's
merges 8tyy 9.0 -> 4.2 s, 8a1a 2.2 -> 0.85 s, cytc 0.51 -> 0.19 s. md5 of p.mtz identical to
production on myob/cytc/thau/8a1a/8tyy and cytc -N 4.
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E): the same observations in the same order, the same verdicts.
- The judged observations are compacted in fixed blocks (count, prefix, fill) instead of a
serial push_back.
- Both sorts (by d, by reflection) sort (key, index) records, so a comparison reads the two
elements it compares instead of two observations elsewhere in memory; same total order.
- Each reflection is decided on its own observations, so the reflections go side by side in
claimed blocks; the rejection count is an integer sum.
Measured (prototype, GPU): Wilson test 8tyy 3.4 -> 3.0 s, 8a1a 2.8 -> 2.6 s per run (the
sort-on-records part is not in the measured binary yet); md5 identical.
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E) for every written file.
- A search merge (for_search) is never written - the run always merges again in the chosen
group - and nothing reads rfree_flag except the writers and --model validation of the written
merge, so RotationScaleMerge no longer assigns a free set to it.
- AssignRfreeFlags takes a thread count: the per-reflection key (the lattice holohedry walk) runs
on fixed chunks, and the distinct Laue-key count is taken off a parallel sort (equal keys are
equal values, so every sort gives the same sequence) instead of an unordered_set. Rugnux passes
its thread count at its two direct call sites; the default stays 1.
Measured (prototype, GPU, 8tyy F432 cubic): the R-free/chi2 section of the merges 4.5 -> 0.65 s;
8a1a 0.76 -> 0.37 s; cytc -0.13 s. md5 of p.mtz/p_P1.mtz/p.cif/p.hkl identical to production on
myob/cytc/thau/8a1a/8tyy.
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E): every per-group and per-pair sum keeps its index-order summation, so p.mtz and
every written file are bit-identical; nothing depends on the thread count.
- The fulls' group CSR (index order within a group) is kept per arm: on the GPU path it is the
one RunArm already builds for SetFullsGroups, on the CPU path BuildFullsGroupCSR.
- The four "every thread walks every full, keeps its own groups" loops (error-model sums on the
CPU path, Bijvoet sums, ISa asymptote, anomalous pair sums) become ParallelBlocks over groups /
pairs walking the CSR - O(n_full) instead of O(threads x n_full) traffic.
- The serial walks that build per-group/per-pair data (outlier-median sets, band spread pairs,
group d, anomalous-variance shells, per-hand R_meas rows, per-frame counts, the error-model
sample list, the Wilson observation list) are ordered-parallel: fixed-block compaction
(KeptIndices), CSR fills, per-block lists whose consumer is order-free (medians).
- CC1/2 and CCanom halves are assigned cell by cell from the CSR (the rank is a set property).
- Friedel pairs are numbered by first appearance through a parallel sort instead of an
unordered_map of millions of keys (NumberByFirstAppearance).
- CPU path: the fulls scaling loop's group means sum over the CSR on all threads
(ReduceFullGroupMeans replaces the serial ReduceGroupMeans), the rescale runs on all threads.
- The error-model sample download goes into uninitialised arrays (no serial zeroing).
Measured (perf-rsm-explore-wip prototypes, GPU RTX 5080, box load 6-14, Run-span medians):
MergeAndStats-side sections 8tyy -3.5 s, 8a1a -2.0 s, cytc -0.25 s. md5 of p.mtz identical
to production rc174 on myob/cytc/thau/8a1a/8tyy (GPU), myob/cytc (CPU), and at -N 4/8 (GPU
cytc, CPU myob).
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every full-statistics rotation merge computed French-Wilson amplitudes - the
search candidates, the pinned low/high pair, the final, and the P1
cross-check - although only the written merge's amplitudes are used, and those
were recomputed anyway with the anisotropic prior. RotationScaleMerge gets
SetFrenchWilson(); Rugnux turns it off for its merges and makes the amplitudes
once after the anisotropy analysis (anisotropic prior when a tensor was
fitted, isotropic otherwise). --mode scale keeps the engine's own amplitudes.
p_P1.mtz is now intensities only (owner decision): the F/SIGF/F(+)/F(-)
columns are omitted; every intensity column is bit-identical to before.
French-Wilson was 29% of the CPU samples in the large-cell tail (8tyy P1
cross-check window, ~84 core-s). p.mtz md5 unchanged on myob, cytc, thau,
kdp, 8tyy (GPU build). Wall: 8tyy 173.9 -> 150.9 s; cytc/thau -0.2..-0.3 s,
myob unchanged (3 interleaved repeats).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
* Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results.
* Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results.
* Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior.
* Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals.
* Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4).
* Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery.
* jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies.
Reviewed-on: #84
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports.
* jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls.
* Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results.
* Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable.
* Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate.
* Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do.
* Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence.
* Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags.
* Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check.
* Rugnux: Clear error messages when a data set needs more GPU or host memory than is available.
Reviewed-on: #83
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection.
* Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100).
* Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused.
* Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image.
Reviewed-on: #82
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF.
* `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots.
* Documentation updates.
Reviewed-on: #81
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked.
* rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results.
* `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry.
* `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel.
Reviewed-on: #80
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
