A pedestal-subtracted JUNGFRAU background is mostly empty pixels; the variance per unit mean leaves
out the zeros and read 6.7 keV per photon on a 12 keV run in keV, 2.4 on a 4.6 keV one, so such
files needed --gain. MeasureGain now also histograms the sampled pixel values and takes the gain from
the single-photon peak (background_gain::PhotonPeak): the highest bin past the zero peak's descent,
standing 5 sigma above the valley before it, narrow (below half height a quarter of its value away),
and repeated by a two-photon peak near twice its value. Where there is no such pair the variance per
unit mean decides as before. Float frames are read for the measurement at a sixteenth of their unit,
so a photon of a few keV is resolved rather than rounded to whole keV.
Measured: float32 keV JUNGFRAU files 11.96 (photon energy 12.04 keV) and 4.53 (4.57 keV); the int16
photon copies show no peak past the first bin and keep the 0.55 variance answer, i.e. stay in
photons; the SACLA MPCCD and PAL Rayonix files show humps, not photon peaks, and measure as before.
A gain the file states or --gain gives still wins. Rotation is untouched (the pre-scan's CCD gain
uses only the variance estimator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Model validation already made the anomalous difference map's coefficients ((F(+) - F(-)) on the model
phases less 90 degrees, where the merge kept the Bijvoet pairs) to read the anomalous sites, then threw
them away. They now go into the map coefficients as ANOM/PHANOM (phenix.maps' labels), missing on the
reflections without a pair, so _maps.mtz opens as an anomalous map in Coot. Nothing in scaling, merging
or the validation's decisions changes; a merge without the Bijvoet split writes the twelve columns as before.
The broker serves the map as /rugnux/map.ccp4?type=anom (404 without Bijvoet pairs), and
rugnux_live_status.anomalous_map says it is there. The Mol* viewer loads it only then, as a magenta mesh
at +4 sigma with its own level slider and switch.
Tests: ANOM/PHANOM appear with Bijvoet pairs (as many as the reported pairs) and not without; the served
anomalous map of a point peaks at it; the auth test covers type=anom. End to end (myoglobin 2M replay):
anomalous_map true, the served map equals gemmi's FFT of ANOM/PHANOM in the written _maps.mtz to 3.4e-6 rms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit c4d6ad74c706181cbb183cfcaa2481653cd7cad6)
The stills geometry bundle (RefineGlobalGeometry) can now refine the detector tilt
together with the beam, the distance, the cell and the modules. With the tilt free the
beam block is the direct beam, from which the PONI is worked out inside the residual: in
PONI terms a tilt moves the whole pattern as a moved beam does (D tan(tilt)/pixel), more
than the beam's 8 px box holds, while the direct beam does not move with it and the tilt
is left with the keystone. The two tilt-like rows of ModuleGauge are gone: within a
module a tilt is a gradient no rigid move makes, and the rows cost held-out accuracy.
The shift, turn and outward gauges stay.
The tilt is a hypothesis (DecideStillsTilt): the strong frames of the last geometry
refinement are split in two halves, each fitted with the tilt held and refined (modules
placed in both where the run placed them), each fit scored on the other half, and the
tilt is kept where the refined fits predict the held-out spots better. No threshold.
It is decided after the stills beam-centre trial and its re-refinement, on frames
indexed at the settled centre: a centre error on frames indexed at it reads as a tilt.
--rot1/--rot2 or --no-refine-tilt hold the tilt on stills.
The stills beam-centre trial now compares direct beams. Its hypotheses (the file's
centre, the background walk, the point-symmetry capture) are taken as the point the
beam lands on and each trial puts the PONI where its distance and the run's tilt land
the direct beam there; the walk, fitted through the geometry's tilt, returns a PONI and
is converted. The pre-scan beam-centre check log compares the walk's direct beam, not
its PONI, with the file's direct beam. Without a tilt nothing changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The lower bound counts each saturated pixel at the saturation limit, so it needs one. An observation
overloaded only by pixels unreadable on this frame (a per-frame bad-pixel flag, data value 0) was kept
as a box sum with those pixels counted as zero, which bounds nothing; it now stays overloaded and is
dropped, per observation.
With the stuck pixels masked by the pre-scan, the run-level gate (kept overloads must be enriched
coarser than 4 A) has nothing left to catch: without it every still set gives the gated result
(clipped CCD and Rayonix sets identical to the fourth decimal, photon-counting sets unchanged), so it
is removed with Reflection::overload_kept. A neighbour-brightness test per saturated pixel was tried
and not kept: after the mask it only dropped 9-20% of the Rayonix overloads and worsened R_split on
one set (0.245 -> 0.278).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A pixel stuck at the saturation limit was never masked: the defective-pixel scan counted only the
error value as "invalid on most frames", and a saturated pixel merely as lit, which the hot rule
does not judge on stills. A Rayonix stills set has 3,627 pixels at 65535 on every frame, at high
resolution; each flagged every reflection whose disk it fell in as overloaded, and the spot finder
found them on every frame. The saturation limit now counts with the error value (CPU and CUDA), so
such a pixel is masked like one holding the error value.
On that set the indexed fraction rises from 10% to 33% (and 7% to 30% on the second lattice of the
same run), R_split 0.120 -> 0.106 and the unrefined R-free against the deposition -0.001 / -0.024.
Rotation merges checked bit-identical (three in-house sets with overloads).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Saturation clips the strongest spots, and those are at low resolution. On a Rayonix stills set the
saturated pixels sat almost entirely past 4 A (0.002% of the reflections coarser than 4 A against
0.59% overall) - flagged or hot pixels, not clipped spots - and keeping those observations as lower
bounds put the saturation limit into high-resolution intensities: R_split 0.12 -> 0.93, CC1/2 0.99
-> 0.33. The run now keeps overloads only where their share coarser than 4 A is at least 1.5x their
overall share; otherwise they are flagged and dropped as before, with a log line saying so.
On clipped CCD stills the share coarser than 4 A is 2.6-5.9x the overall one and nothing changes
(R_split and R_free identical); on the Rayonix set R_split returns to 0.12. A pragmatic gate until
saturation is told from bad pixels per spot. Rotation never keeps overloads and is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The Metal kernel and host code are restored to their panel-s5 content, so neither the background clip
that follows a handed-over radial curve nor the count of saturated pixels in pass A runs on the Metal
path. This is an accepted, temporary platform difference, to be closed once the kernels can be checked
on a Mac: on macOS a still's background clip stays the plain one (the radial correction itself still
uses the handed-over curve, as on the CPU), and an overloaded observation kept as a lower bound counts
its saturated pixels as 0 instead of at the saturation limit.
One host-side change is kept, with no effect on any result: the radial sums are accumulated on the GPU
only where no curve was handed over. StartPassA now sizes those sums for that case alone, so the old
host code would write past them on every still with a curve.
The shared structs need nothing on the Metal side: BraggPassA.h is compiled into the kernel source, the
box-sum kernel zero-initialises BraggBoxSums (n_saturated = 0) and the host zero-initialises
BraggPassAFrame (n_curve = 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The one conflict, tests/BraggIntegrationEngineCPUTest.cpp, where both branches add cases at the same
place: integ-pal's side is taken and stills-sat's test diff applied on top, which keeps all four cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The validation battery of 61def00a1 (20261022-0925_29000d_still-integ-pal-r1clip) shows the r1 rule
buys nothing once the ring clip follows the radial curve: lcls222 at r1 4 (4,6,8) 0.1893 against
0.1856 at r1 6; pal86 0.1998 against 0.1997. The radius scans that motivated it were taken with the
flat clip, whose beam-side cut the smaller ring partly avoided. r1 stays 6 on stills.
The empty apertures' corrected column again clips flat. With the clip following the curve there,
lcls242_s read the correction as a clear gain (z = 11) and the run took it, but dimple on 40,000
frames says it costs 0.013 (0.2461 on against 0.2335 off); judged on the flat clip, as in 07a43a13e,
the decision is off there and unchanged on every other set. The integrator still follows the curve
in its clip wherever the correction is on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills merge dropped every observation with a saturated pixel. On a still the saturated
observations are the fully recorded ones of the strongest low-resolution reflections, so what
survived were their partial observations, whose partiality correction then set the merged
intensity: on clipped CCD stills the merged low-resolution intensities came out at 0.4-0.6x of
the deposited ones, a ramp no Wilson B absorbs, invisible to CC1/2 and R_split.
The pixels are really clipped (flat-topped plateaus a few per mille wide, a pile-up in the
pixel-value histogram from the per-pixel ADC limit), so the auto ceiling is right; the drop was
the defect. A still now keeps the observation: the box sum with every saturated pixel counted at
the saturation limit, unflagged and marked clipped, counted in BraggIntegrationCounts and
reported as OVERLOADED_REFLECTIONS_KEPT. Rotation data (IsRotationIndexing) are unchanged and
keep the flag for the rocking-event drop.
Measured on a clipped-CCD stills subset (dimple R-free, deposited free set): dropped 0.247,
lower bound 0.230 (same as switching saturation off). A Kabsch fit to the unsaturated wings read
lower than the lower bound (0.237), and inflating the lower bound's sigma lost (0.255).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two stills-only changes to the integration stencil.
1. Where the whole-frame radial curve is in use (the correction on), the ring's high-side clip
follows it: a ring pixel is clipped against the threshold plus the curve's rise from the
reflection's radius to its own (BraggIntegrationEngine::RadialClipOffset, rounded to whole counts
so CPU, CUDA and Metal keep the same pixels). On a background falling steeply away from the beam
the flat mean + 4 sqrt(mean) threshold cut the beam-side half of the ring as outliers, so the
background read low and reflections near the beam stop read 3-150x too strong. The empty-aperture
test's corrected column clips the same way, so the per-run correction decision sees it.
Measured with r1 6 (dimple R_free, battery arguments, all frames):
lcls222 0.196 -> 0.1856, pal86 0.216 -> 0.1997, sacla236 0.2559 -> 0.2559, alvra213_int16 0.1986 -> 0.1986.
2. The signal disk is taken from the lattice: it must not reach into the next node's spot. The
closest two nodes can come on the detector is the shortest reciprocal-lattice vector the centring
allows, s = F_px tan(2 asin(lambda / 2 d_max)), and a spot reaches 2 r80 (measured in the
pre-scan); r1 = the largest of 4, 5, 6 px not above s - 2 r80, never below 4, never above the
disk the run starts with (6). The empty apertures are measured for each candidate disk and the
ring is chosen on the chosen disk's tally. Pixel detectors keep 6 (sacla236 s 22 px, alvra213
20 px); the binned CCDs with a long axis shrink to 4 (lcls222 5.4 px, pal86 6.7 px), which the
radius scans there measured as the optimum; pal90 (s 21 px) and lcls242_s (15.5 px) keep 6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The background-ring log line now gives how many standard errors the correction lowers the chosen
ring's mean square by, over the apertures measured both ways. Diagnostic only; the decision is
unchanged. Full pal90: z = -2.7 (correction off), where dimple prefers it on (0.2124 vs 0.2183).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Without a resolution limit the outer d*^2 shells of the width measurement lie beyond the diffraction
limit, where the summed intensity is noise or even negative; one such shell made the whole measurement
fail, and 5 of 11 still-battery sets (sacla237, pal86, pal90, lcls242_s, lcls242_t) fell back to the
per-crystal window fit. A shell now enters the fit only where its summed intensity is measured to 10 %
(sum I / sigma >= 10), and at least three shells are needed.
Weighting the fit by the shells' signal instead was tried and rejected: it lets the strong low-resolution
shells set the width alone, and on the MicroMAX pink set the dimple R-free went 0.2442 -> 0.2748.
The matrix subsets, which are cut at their resolution, are unchanged within 0.0010 dimple R-free
(SwissFEL 0.2453, -C/-S 0.2513, mono 0.3188, pink 0.2442, SACLA 243 0.2455, SACLA 237 0.2327,
LCLS 222 0.2335, PAL 86 0.3628).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The first stills battery of e8145d8 lost on one Rayonix set (lcls242_s, dimple R_free 0.2302 ->
0.2323): there the whole-frame radial correction hurts. On 40,000 of its frames: battery default
0.2462, ring 12 with the correction 0.2444, ring 12 without it 0.2336, ring 14 with it 0.2467. Its
beam is strongly polarised and the azimuthally averaged ring background does not describe the local
background.
So the empty apertures now also carry the corrected difference: the frame's ring background
(RingBackground, factored out of the engine) averaged over the ring pixels minus over the disk pixels
- what the integrator's kernel takes off a reflection - is subtracted from each disk-minus-ring
difference. The ring is still chosen on the plain differences, so its reach does not rest on the
radial model; where the correction was left to auto it is then kept only if it makes that ring predict
the disk better. Decisions on 15-60 pre-scan frames:
pal86 ring 10, correction on (240.6 -> 221.1) lcls222 ring 10, on (6.34e4 -> 3.41e4)
lcls242_s ring 12, off (277.2 -> 285.8) alvra213_int16 ring 14, on (0.01424 -> 0.01422)
Choosing ring and correction jointly instead picked ring 14 with the correction on lcls222, where dimple
reads 0.204 against 0.196 at ring 10; a median instead of the mean square picked ring 14 with the
correction on lcls242_s, the worst arm there. The sequential rule is the one that agrees with dimple on
every set measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
GetConsensusUnitCell averaged every indexed frame within 5% of the -C cell. The given-cell
population filter runs after it, so the MTZ, report and model-validation cell was a mixture of
both forms. Frames whose retained outcome is empty now do not count, and the consensus cell is
read again once the population filter has cleared the other form's frames.
Two-form P21 serial data (LCLS):
- T run: 93.85/93.66/49.98/92.00 -> 95.21/94.50/50.42/91.09 (kept population ~95.5/94.1/50.3/91.2)
- S run: 92.38/93.29/49.59/93.26 -> 91.90/93.06/49.45/93.65 (deposited 92.04/93.02/49.01)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
--capture-uncertainty now reaches the stills merge: each partial carries (c (1 - p) <I>)^2 in its
variance, the rotation combine's capture term per observation, in the merge weights and in the
error-model samples alike. --reject-outliers on stills now uses the rotation merge's band
(OutlierBand.h): exp(+-n s) about the reflection mean for the systematic part, s the weighted median
|ln(I / median)| of the merge's own observations, instead of a symmetric +-n sigma band that reaches
below zero when b is large (stills b 1.4-1.7). A median needs three observations.
Neither is switched on for stills: measured with the run-measured partiality width before its
window-truncation correction (dimple R-free, SwissFEL 4k / SACLA 243 / MicroMAX mono / MicroMAX pink):
no options 0.2567 / 0.2497 / 0.3279 / 0.2477, --capture-uncertainty 1 0.2510 / 0.2470 / 0.3261 /
0.2527, --reject-outliers 6 0.2619 / 0.2487 / 0.3317 / 0.2630 - mixed, and the photon-counting pink
set loses with both. Raising --min-partiality to 0.1 or 0.2 lost 0.008-0.019 on SwissFEL and mono.
Defaults unchanged, so stills output is unchanged; rotation does not use MergeOnTheFly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills partiality width came from a per-crystal least-squares fit of de^2 against d*^2 over the
integrated reflections. Those fill the prediction window evenly, so the fit returned the window's own
width (W^2/3, constant in the linear Ewald distance, gamma_e ~ 0): at low resolution several times the
crystal's rocking width, which mis-corrected every partial there.
The width is now measured once per run from what the reflections RECORD: the intensity-weighted second
moment of the excitation error, sum I de^2 / sum I, which is the width of a Gaussian partiality whatever
the window, in 8 shells of d*^2 over all crystals. Each shell is corrected for the window cutting the
curve's tails (a curve wider than the window is taken as wide as the window), and the shells are fitted
as s^2 = gamma0^2 + (gamma_e d*)^2 + (bw lambda d*^2 / 2)^2 with every term non-negative. The per-crystal
fit stays as the fallback when the run is too small to measure (< 800 reflections). The prediction window
is unchanged (2x the per-frame profile radius).
Measured: on the SwissFEL lysozyme stills the width comes out ~0.035 deg angular at every resolution,
the ~0.03-0.045 deg that the empirical rocking profile of the observations gives; the window-moment
model was 3-4x wider at low resolution.
dimple R-free (deposited free set, matrix subsets/cuts, PARTIALITY_MATRIX.md), shipped -> this:
SwissFEL run0237 4k 0.2692 -> 0.2454
same, -C/-S given 0.2679 -> 0.2513
MicroMAX mono 4k 0.3341 -> 0.3189
MicroMAX pink 4k 0.2415 -> 0.2442
SACLA 243 subset 0.2576 -> 0.2454
SACLA 237 subset 0.2342 -> 0.2331
LCLS 222 subset 0.2344 -> 0.2330
PAL 86 subset 0.3499 -> 0.3629 (binned CCD, stencil-limited)
For comparison --bandwidth 0.003 by hand: 0.2616 / - / 0.3108 / 0.2377 / 0.2460 / 0.2270 / 0.2404 / 0.3469.
Stills only: rotation never runs StillsPartialityRefine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Two changes to the stills integration background, both stills-only (rotation keeps its path bit for bit):
1. The r2..r3 background ring's outer radius is measured per run. The pre-scan lays apertures on a
grid over its frames, away from every found spot, and for each candidate r3 (r2+2 .. 14) takes the
mean square difference between the r1 disk mean and the clipped ring mean. With nothing in the
aperture that is exactly the background error the ring makes (noise + bias); the run keeps the
minimum. On a photon-counting pixel detector the noise wins and the shipped r3=14 is kept; on a
binned CCD the background structure wins and the narrowest ring is taken. r1 stays 6: a smaller
disk (4,6,14) costs 0.0013 dimple R_free on a JUNGFRAU set, so it does not pass the
photon-counting gate. --adaptive-integration-radius now defaults on for stills too (=off restores
the fixed 6/8/14). The pre-scan bandwidth estimate is not applied to stills.
2. The radial background-curvature correction on stills takes its curve from the adaptive spot
finder's peak-excluded per-ring background (whole frame) instead of the predicted reflections'
own ring pixels, and auto applies it to every stills image (no ice-score gate: a solvent/carrier
hump curves the background as much as ice). New engine call RadialBackgroundFromRings; CPU, GPU
and Metal read the handed-over curve. The GPU box sum now sizes the radial bin index separately
from accumulating, so a handed-over curve is read at the right radius.
Measured (dimple R_free, deposited free set, battery arguments, all frames unless noted):
pal86 (Rayonix 4x4, 156 um): 0.354 -> 0.218 (ring 10; at 4,6,10 0.213)
lcls222 (Rayonix 4x4, 177 um): 0.205 -> 0.196 (ring 10)
sacla236 (MPCCD): 0.2588 -> 0.2558 (ring 11)
alvra213_int16 (JUNGFRAU): 0.1986 -> 0.1986 (ring 14, unchanged)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Four changes to the per-run fixed cost of a stills run (Step 2 geometry refinement, Step 3 pre-scan),
none of which changes what is computed:
- Module placement: the four held-out fits and the placement on every strong frame are independent, and
one Ceres fit keeps only ~4.6-6.5 of 32 cores busy (perf, JF16M, 4000 frames / 150888 spots). They
now run side by side on a fifth of the threads each; the placement fit is computed speculatively and
used only where the held-out residuals call for it, as before. Baseline (quiet): held-out 12.3 s +
placement 5.1 s on JF16M, 16.4 + 6.5 s on JF4M, 19.0 + 7.6 s on MPCCD.
- Module scan (+-25 px grid per module ahead of the placement) was one thread: every grid shift is now
scored on the workers and the grid then walked in its old order, so the first best still wins
(3.3 s on JF16M, ~5 s on JF4M/MPCCD before). New test ScanModuleShifts_FindsShiftedModule checks the
shift is found and that 1 and 4 threads give the same answer.
- Stills geometry trial: each hypothesis (centre, distance, frame sample) is indexed once. The distance
grid ran twice on the same centre whenever the file's centre was kept (40 of ~275 hypotheses,
~15% of the trial: 6.2 s on JF16M, 6.7 s on the flat JF8M), and the final here/there check repeated
the first trial. Repeats always gave the same count (checked on three logs).
- The saturation histogram (40 frames, 0.1-2.2 s) reads only frames and the mask: it is taken beside
the geometry refinement and the limit decided where it was, after it.
Plus TIMING marks for the sub-steps (geometry refinement sample, module scan, module placement fits).
Outputs identical to the baseline binary (bb7c9c) where checked: JF16M int16 15 runs (geometry, all
32 module shifts, held-out residuals, cell), JF4M int16, MPCCD 236 (indexing rate, CC1/2, R_split,
R-free with --model identical), PAL 90 and LCLS 222 (identical reports).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
- The NXmx master (and _process.h5) writes module_0..N with the panel table when the start message has one,
besides the whole-image module, so a stored-layout file reads back with its panels
(JFJochReader_PanelId checks the round trip).
- RotationScaleMerge's modulation grid spans PlaneBounds instead of the image rectangle; with no panel
table x0 = 0 and the extent is the image, so the cells are unchanged.
- The beam-stop / pre-scan memo key includes each panel's box and placement; nothing is added with no
table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Everything sparse that took a radial quantity from an image coordinate now takes it from the
reflection's (or spot's) position on the detector plane, through its panel (DetectorModules).
With an empty module table every changed expression reduces to the old one in the same
floating-point order, so flat detectors - every rotation set, EIGER, CCDs, assembled JUNGFRAU -
integrate bit for bit as before.
- Prediction (CPU, GPU, rotation CPU and GPU): bounds test against PlaneBounds instead of the
image rectangle.
- IntegrateOnImage: a prediction in a gap or off every panel is dropped (PlaneToPanel), no longer
passed through onto an unrelated stored pixel. The panel id stays local: the engine and the way
back find it from the image position (the panel boxes tile the image), and a prediction that
rounding to float put on another panel's edge is dropped so the two agree. Nothing is added to
Reflection.
- Bragg engines (CPU, CUDA, Metal) take a BraggPanel per reflection (BraggStencil.h): its plane
position, the panel's pixel axes on the plane and its box. The stencil's radial direction is
taken on the plane and expressed in the panel's pixel axes; r0, the radial background curve,
the Gaussian's elongation, the learned moments, tan(2theta) and Finalize's polarisation use the
plane position; r_max / n_rad come from PlaneBounds. The neighbour mask and the owner map are
clamped to the reflection's own panel; inside the walked box a pixel off the reflection's panel
is read as a gap (unreadable, statically masked - not an overload), which is what the assembled
image has there, so a disk cut by a panel edge is partial rather than silently "full".
- Risk R4: the ProfileEmpirical grid is learned and applied in the plane's axes (the panel axes
taken to the nearest quarter turn, BraggProfileCell), so turned panels pool one shape. Without
it the stacked layout read up to 0.3 % off the assembled one in the harness.
- SpotWidth / SpotFootprint: aperture bounds against the spot's panel box, beam distance,
resolution and the shape's 2theta Jacobian on the plane, moments read in the panel's axes.
- GetDetectorMaxQ_recipA, the far-corner radii (first-pass tilt report, metric-symmetry arm,
footprint table range) from PlaneBounds; pre-scan spot resolution on the plane;
LogHeaderBeamCenter: geometric centre of the panels' plane box, and the masked-pixel check on
the panel the beam lands on (none when it is in a gap).
Harness: PanelLayout_BraggIntegration is now a real [PanelLayout] check, over six settings
(default, bandwidth-elongated ring, radial background, overlap exclude, box sum, empirical
profile, all with polarisation), stacked vs assembled to 1e-5; PanelLayout_BraggIntegrationGPU
runs the same on the CUDA engine. Its spots avoid exact half pixels: the grid centres on the
rounded position, and a half rounds to the other neighbour on a turned panel.
The Metal engine (.metal/.mm) mirrors the CPU change exactly but is UNCOMPILED here (no macOS):
it needs a build and BraggIntegrationEngineMetal_BitIdenticalToCPU on a Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Spot finding works panel by panel, as the FPGA path does per JUNGFRAU module. PanelBoxes (common/) gives
one box per DetectorModule; an empty table gives one box, the whole image, so every clamp reduces to the
old image bounds in the same evaluation order (rotation sets, EIGER, CCDs, our JUNGFRAU unchanged).
- ImageSpotFinder::SetPanels: the CPU local-box pass (dense, count-threshold and the DetectAt/candidate
path of the adaptive finder) slides its windows inside each panel box; host CCL labels panel by panel;
pixels on no panel are folded into the resolution mask.
- GPU: analyze_pixel tiles each panel (warps numbered panel after panel), analyze_candidates and
union_neighbours clip to the pixel's panel, the device spot carries its panel; the GPU list is put in
panel order so it equals the host's.
- DiffractionSpot carries its panel; SpotAnalyze maps by ImageToPlane(panel, ...) (d_A on the plane).
- ImagePreprocessorBuffer::GatherTiles (CPU+GPU) clips each square to the panel of its centre.
- MXAnalysisWithoutFPGA and the pre-scan workers hand the module table to finders and buffers.
Raster algorithms: new common/AssembleToPlane.h (PlaneRasterOf, AssembleToPlane nearest-pixel gather with
gaps, GatherFromPlane back to stored pixels; empty table = copy). With panels, ShadowFinder::GetMask runs on
the projection assembled to the plane raster and gathers the mask back per stored pixel (CPU route);
FindBeamCenterFFT scores the assembled projection and reports plane coordinates; the detector diagnostic
JPEG shows the assembled plane. Without panels all three take their old path.
Harness: PanelLayout_SpotsNeverCrossPanels enabled (count threshold and SNR on the edge scene), new
PanelLayout_SpotFinderParity (CPU vs GPU, local box and adaptive, on the stacked layout),
PanelLayout_AssembleToPlane, PanelLayout_ShadowFinder, PanelLayout_BeamCenterFFT,
PanelLayout_DetectorDiagnostic (stacked vs assembled image cut to the raster).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Every dense map that needs geometry now evaluates each image pixel at its
place on the detector plane (DetectorModules), so it is right for a stacked
panel layout and picks up refined module placements:
- AzimuthalIntegrationMapping: bins, resolution, solid-angle/polarisation
corrections built per pixel through the table; a pixel on no panel is left
as a masked one. The geometry cache key now includes the panel table, so a
mapping built after the stills module placement is rebuilt rather than
served from the cache. ResolutionMaskBits follows (it is a lookup).
- ROIMap / ExportROIMap: azimuthal ROIs on the plane (box and circle ROIs
stay in image pixels).
- ROIIntegration CPU+GPU: the weighted sums are taken per ROI and panel in
image pixels and placed on the plane at the end (exact; the affine commutes
with a weighted mean).
- HotPixelFinder: ring/sector keys and the zeta of the persistence rule on
the plane.
- Background beam-centre walk CPU+GPU: pixels binned at their plane position;
host and device share BackgroundPlanePoint so they stay bit-identical.
New common/PanelRows.h: ForEachPixelInRows, ForEachPixel cut to a band of
rows in index order, for the row-parallel builders.
An empty table takes the old arithmetic in the old order: rotation sets and
every single-panel input are unchanged. [PanelLayout] AzintMapping and
ResolutionMask enabled (+ azimuthal ROI map and ROI centroid on the plane);
new BeamCenterFromBackground_BinsOnThePlane and
ROIIntegrationGPU_MatchesCPU_Panels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Stage 0 of analysing panel detectors in their stored layout. The table of modules already is the panel
table; what the consumers converted in the next stages need from it:
- ImageToPlane / PlaneToImage on a known panel, no search; panel -1 (empty table) is the identity.
- PlaneToPanel: the panel holding a plane point and its image position, nullopt in a gap (never passed
through, unlike the existing PlaneToImage).
- PlaneBounds: the box of all panel footprints on the plane; with an empty table exactly the image
rectangle today's prediction bounds test uses.
- ForEachPixel: f(panel, index, px, py) panel by panel, row by row, gaps not visited; with an empty
table the plain row-major loop with plane == image.
The image preprocessors (CPU, GPU) and the Bragg engines (CPU, GPU; Metal derives from CPU) now refuse
a pixel mask whose size is not the image's, instead of reading a mask of another size out of range.
Every reader sizes the experiment's detector, and so the mask, by the image it reads, so nothing that
runs today is refused. Nothing calls the new functions yet: no output changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto
rc178 with the owner's decisions for this release:
- Inputs are files, never a directory; several files of one format are one dataset (the multi-input
CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles,
ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and
xtc::CollectStreams are gone, CanRead* accept a regular file only.
- One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds:
per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera
length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is
refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit
decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work).
- Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's
place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through
HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib,
bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through
HDF5's filter under the global lock, one frame at a time.
- LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and
binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event
dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all.
- CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path)
and optionally the camera length; everything else is ignored with a note in the log.
- --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored
layout (what a .geom's mask_file used to give).
- Gain measured on the background for panel-table and CXI input that states none (--gain overrides).
- Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with
one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions
are mapped through the modules, the image is not re-assembled.
- Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge
on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only
where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table,
.geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx
NXdetector_module groups), read back with --module-geometry.
- A panel-table file's camera length is nominal: with a reference cell it is refined from the cell.
- Broker and writer output unchanged: masters keep the single NXdetector_module group, the start
message carries no module list.
Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation],
SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Three defects that stopped known-cell stills runs:
- The consensus cell compared every frame's cell to -C axis by axis. A run whose frames index in another
setting of the same lattice (axes in another order, another monoclinic choice) had no frame accepted, no
consensus cell and no output ("No crystal lattice was determined"), although it merged fine. A frame
whose lattice contains the given cell now counts on the given cell's axes (CellOnReference, the same
FindSublatticeBases the merge-stage relabel uses).
- The per-image ambiguity resolver relabelled a frame's reflections but left its lattice on the old axes.
Everything after it predicts from index and lattice: the sublattice relabel put such a frame through a
second transformation, and on a monoclinic cell with a near-symmetry (a+c nearly as long as a) the
merge came out at indices finer than anything measured, shells over 100% complete. The lattice now
follows the indices (ReindexLattice, B' = R^-1 B).
- With -C/-S the stills path replaced the known-cell indexer's lattice, already on the given axes, by
LatticeSearch's conventional choice, which on that near-symmetric cell is the other description on half
the frames. The indexer's lattice is now kept when it is close to the given cell.
Measured (cell + space group given, dimple against the deposited model and free set):
- monoclinic MbCO-type still set: was over-complete/broken with --model; now 2.06 A, R/Rfree 0.233/0.314
at 2.11 A (P1 merge before: 0.275/0.306).
- C-centred orthorhombic set: no MTZ before; now 3.27 A, R/Rfree 0.298/0.403 at 3.68 A (de novo 0.336/0.448).
- monoclinic two-form set, one form: cell P 1 21 1 with beta (was alpha); Rfree 0.310 -> 0.260, 3.05 -> 2.72 A.
- No change on seven other stills sets, nor on two rotation battery sets (5reo, lyso_x10sa_90deg_1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The peak nearest the given cell could be a single stray frame when the given cell is a few percent off
the crystal (a cryo cell given for room-temperature lysozyme stills): the run then merged one frame.
Every population is now found from its own peak and valleys, and of those holding at least a tenth of
the frames the one centred nearest the given cell is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A given cell keeps only the frames whose refined cell belongs to its population (GivenCellPopulation):
the peak of the refined-volume histogram nearest the given cell, out to the valley that separates it
from another population, with the bin set by the data's own spread. A sample of two crystal forms
indexed against one form's cell otherwise merges both; a run with one population keeps every frame.
- The distance grid is run from every centre hypothesis, not only the one decided at the file's
distance, and a local grid of centres follows at the distance decided. A wrong header distance and a
centre several pixels off (both background estimators misread it on a Rayonix set) otherwise leave
each other flat; now the cell alone finds both.
- A distance given with --detector-distance is not searched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On stills a cell and a detector distance are degenerate: a spot's radius fixes only their ratio. The
geometry refinement that -C switches on anchored the cell and refined the distance, so a deposited cell
a percent off this crystal moved a correct distance by as much (MPCCD: 50.47 -> 50.14 mm, R-free
0.259 -> 0.318). Now the distance is held and the cell refined from the spots, starting from the given
one. Only where the frames have rejected the file's distance (DecideStillsGeometry's grid, which is what
-C recovers a bad header distance with) is the cell anchored and the distance refined from it. The grid
itself asks the given cell, not one refined at the file's distance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Foreign stills files state their saturation level wrongly or not at all: an MPCCD Cheetah file is
dark-subtracted per pixel, so each pixel clips at the ADC limit less its own dark, and its
CrystFEL geometry leaves max_adu commented out - the run reported "Overloads: 0" while the inner
panels saturated in most frames.
saturation_ceiling (image_preprocessing/SaturationCeiling) histograms pixel values on a log
scale (64 bins per octave from 256) and returns the lower edge of the lowest bin holding over
twice what any of the 8 bins below it held (and at least 50 values): a bright tail only falls, a
clipped one piles up below where it stops, as a band where the level differs per pixel.
DetectSaturation runs it on 40 frames spread over every stills run and logs it beside the file's
limit; for any input but a Jungfraujoch-written file (ProcessConfig::detect_saturation) a level
below the file's replaces it. The CrystFEL/CXI reader, where the geometry states no max_adu, now
takes the stored integer type's maximum (an int16 Cheetah file clips at 32767).
The geometry-file-free CXI path (previous commit) is limited to square frames: a tiled detector's
Cheetah slab (CSPAD, AGIPD, MPCCD, ...) read as one flat panel would place every pixel wrongly
without a word, and those slabs are not square.
Measured:
MPCCD stills, 1339 frames: pile-up from 1218 photons (by hand 12000 ADU ~ 1200 ph); 5.5% of
reflections coarser than 4 A flagged; lowest-shell CC1/2 0.545 -> 0.674, overall 0.741 ->
0.793; same dimple refinement against the deposited model R-free 0.2586 -> 0.2556
single-panel Rayonix CXI, 1316 frames: no pile-up (20 pixels at the int16 limit in 40 frames);
the int16 limit flags 123 reflections; CC1/2 0.737 -> 0.787, rugnux R-free 0.382 -> 0.358
JUNGFRAU stills (Jungfraujoch files: lysozyme and two private sets): no pile-up below the stated
limit, results bit-identical
Cost 0.5 s per run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Serial crystallography quotes R_split and a CC1/2 whose half-sets keep each
crystal whole (CrystFEL's partialator), and rugnux's stills report had
neither, so its numbers could not be set beside CrystFEL's. MergeOnTheFly now
also sums every frame's observations into the frame's own half (the
existing per-image hash preference), and MergeStats reports from those two
means R_split = (1/sqrt 2) sum|I1-I2| / (1/2 sum(I1+I2)) and their CC1/2.
They are printed as a "Serial statistics" table under the shell table (also
in the --report-resolution table) and as R_SPLIT / CC_HALF_SPLIT and their
REFRES_ keys.
The balanced per-reflection split behind CC_HALF is unchanged, and so is
everything decided on it (resolution cut, space-group search, pass gates).
Rotation output is unchanged: RotationScaleMerge does not fill the new
fields and the table is not printed.
On a 4000-frame jet subset over 30-1.5 A: CC1/2 0.830, R_split 31.8%, against
0.834 / 31.3% computed by hand from two separate even/odd runs and 0.842 /
30.5% from CrystFEL on the same frames.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- WarmUpModelValidation did nothing for a prediction (no crystal cell: "no atoms or no unit cell"), so a broker's
first molecular replacement paid every first use on the GPU. A prediction is now warmed up by a search of its
own (WarmUpMolecularReplacement): its search model in a P 1 21 1 cell four radii on each edge, that crystal's
intensities to 2.5 A, the search, and the validation of what it placed. RunMolecularReplacement waits for a
warm-up still running, as ValidateAgainstModel does. --simulate-live warms the model up as the broker does at
upload. Result-identical.
- fast_live with a prediction (or --mr): the search starts on the geometry pre-pass's merge, averaged into a group
from that merge's own space-group search (to 3 A, standard setting), beside the second pass, on card 1 % cards,
a quarter of the threads, under GPUWorkBeside. At the answer's merge CarryMolecularReplacement takes it over
where it holds - same group and a cell within 1%, every copy refined by the rigid body beside the others
against the answer's merge and its R-free within R_FREE_MARGIN of the one it was found at - and the merge is
searched itself otherwise (and always where the twin hypothesis' merge is searched too). RESULT-CHANGING only
in the refinement end point of the placement.
Sim-live (10 ms/frame, RTX 5080): molecular-replacement step thaumatin 0.96 -> 0.31 s, cytochrome C 1.19 ->
0.25 s; merged reflections identical; thaumatin placed model identical (0.000 A), R-free 0.3765 both;
cytochrome C a symmetry-equivalent placement, CA rmsd 0.03-0.05 A per copy, R-free 0.4443 -> 0.4431.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
The placed candidate of lowest R-free is known once the groups' searches are done. Its null and its
comparisons with the other candidates (the copies carried into another group beside tNCS, the paired R)
used to start only after the tNCS searches again and the null, one after the other. They now start as
soon as the groups are done, beside the searches again, and the choice takes each where it comes to the
same placements (matched by the placement itself); anything the choice does not reach is waited for and
dropped. A null and a comparison are the same number whenever they are computed, so nothing changes but
when they run. All of them are waited for before a merge's rigid-body engines are given back.
The rigid-body pool gets engines for those fits at once (a null's replicates, a carry and a group
searched again: at least 13); the transform's layout is still decided on the search's own fits.
Measured on 4x Tesla T4 (24 threads), MR step, two runs each, byte-identical p.hkl, p_model.pdb,
p_maps.mtz and MR log lines against the base on 1 and 4 cards:
- cytochrome C (AF model, 3 copies, tNCS, enantiomorph decision): 4 cards 2.18 -> 1.74 s, 1 card
2.91 -> 2.71 s;
- thaumatin (AF model, one copy, no tNCS): unchanged (1.62 / 2.49 s), it has no comparisons to overlap;
- 9s02 (AF model, 4 cards): 8.53 -> 8.39 s, identical.
[MolecularReplacement] and RigidBody* pass on 4 cards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
f217f180a tightened SCALING_TOLERANCE to 1e-5 for the per-frame scaling loops, but ApplyCellSurface
used the same constant for its own stop. The surfaces then ran to their 30-round cap ("NOT settled"),
and that - not the per-frame loops - is what lost ISa on the tol-1e-5 full battery
(20261019-2106 vs 20261019-1726): on hepes, myob_x06da, 9rcs and 7bgt the per-frame loops settle on
the same trajectory either way, while the modulation/absorption fits and their half-set gates changed.
The surfaces' own fixed point is the worse one: run to 1e-5 with 300 rounds allowed, the detector-frame
surfaces walk along the resolution direction the gauge projects out (7bgt modulation: 15 -> 283 rounds,
1.13x -> 8.47x ramp removed), their held-out half-set gain drops (hepes +0.026 -> -0.031, 6o2h
-0.28 -> -0.78), and ISa falls (hepes 37.0 -> 18.3, myob_x06da 22.8 -> 19.6, 9rcs 7.4 -> 5.3,
7bgt 17.0 -> 15.3, 6o2h 31.0 -> 18.1). The surfaces' prior and gates were calibrated at 1e-3, so they
get their own SURFACE_TOLERANCE = 1e-3 again; the per-frame loops stay at 1e-5.
Tried and rejected (23 validation sets, runs 20261020-0040/0104/0129 cvfold-*):
- leave-one-out reference in the surface fit: 7bgt R_meas 0.14 -> 0.31, R-free 0.207 -> 0.277,
9rcs CC1/2 0.994 -> 0.956. Regressing the observation on a mean that excludes it puts the rest of
the group's noise in the regressor alone and attenuates the slope; the mean that includes it is
matched by the same noise on both sides.
- leave-one-out reference in the per-frame fit (CPU + GPU): same attenuation; 5reo R_meas
0.23 -> 0.49, 3meb 0.26 -> 0.37, myob_x06da_powder_2 d_min 1.54 -> 1.94 (aspirin and yag gained).
- cross-validating the fulls smoothness on the held-out per-frame fit while scaling on the usual one:
thau_bl1a_4p6keV ISa 34.3 -> 36.4, but kdp 23.9 -> 21.3, aspirin 35.5 -> 33.6, L-cystine R1
0.124 -> 0.137, 6o2h ISa 31.0 -> 25.7; no fewer stalled loops.
Validation, 23 sets (small molecules, the four losers, stall sets, controls; run 20261020-0152
cvfold-final-sets): hepes, myob_x06da, 9rcs, 7bgt back to their tol-1e-3 values; the other 19 equal
to the base within noise (aspirin 20 keV ISa 35.8 -> 35.5, L-cystine 25 keV ISa 8.5 -> 8.2 with SHELXL
R1 0.1152 -> 0.1154). No SG, d_min or verdict change. ci tier (20261020-0152 cvfold-final-ci): 29/29
pass, every row within noise of both earlier full runs. [RotationScale] and AndersonMixing* pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
The floor below which a frame's partial scale is not credible (MIN_CREDIBLE_SCALE_RATIO = 0.02) was
read off the MEDIAN fitted scale - in the loop's smoothing window, in its step, and in the guard
after it (DropCollapsedScales). On a sweep that spent most of its turn out of the beam the median is
itself a dead frame, and it walks: an empty frame's scale has no fixed point above zero, so the dead
frames slide toward zero round after round, the median and the floor slide with them, and frames on
the floor cross it, re-enter the window at their neighbours' scale (3x up in one round) and cross it
again. The loop then has no fixed point, and which frames reach the merge depends on the round it
stops on.
The floor is now read off the run's typical frame (TypicalFrameScale), which is what the loop pins
its gauge to - inside the loop it is the constant 0.02, so it stands still. The fulls' guard keeps
the median (their scales are not gauge-pinned). Dropped frames still carry the median scale, so the
sweep ledger reads them as before.
lyso_x06da_half_image (1800 frames, two thirds weak or out of the beam; median frame at 1/5.7 of the
typical one): traced, the final loop's floor fell 2.4x and every spike in its step was one frame
re-entering the window; the R_meas 287% -> 685% of the rc178 loop fixes was the same non-settling
loop ended by two stopping rules (delta-CC1/2 split the weak stretch differently, 206 frames at
1/20-1/50 of the median merged or not). The pre-pass P1 loop that crept to the cap is the geometry
pre-pass, which commits nothing here.
battery --only, before (rc178-aa-tol5 f217f180a) -> after, GPU:
- lyso_x06da_half_image: R_meas 689% -> 81%, CC1/2 0.932 -> 0.971, REFRES CC1/2 0.484 -> 0.911,
ISa 4.13 -> 4.22, model R-free 0.3302 -> 0.3271, CC_model 0.826 -> 0.829; final loop's smallest
step 5.6e-3 -> 1.8e-4. --no-gpu gives the same numbers.
- lcystine_x10sa_25keV: R_meas 0.396 -> 0.202, ISa 4.45 -> 8.49, CC1/2 0.9987 -> 0.9998.
- lyso_x10sa_strong: R_meas 0.238 -> 0.151, ISa 5.51 -> 5.74, R-free 0.3653 -> 0.3658,
CC_model 0.859 -> 0.857, d_min 1.354 -> 1.344.
- insu_I_x06da_ref: R_meas 0.347 -> 0.290, R-free and ISa unchanged.
- 7mzt: CC1/2 0.967 -> 0.983, CC_model 0.532 -> 0.548, R-free 0.4183 -> 0.4197, d_min 3.18 -> 3.21.
- unchanged: lyso_x06da_ref, myob_x06da_split, myob_x06da_powder_2, thau_bl1a_4p6keV, 6o2h,
3meb, 9z72, 9o0h. Space groups identical on all 13.
Tests: [RotationScale], AndersonMixing* pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
Shutters that open late or close early, and beams that go off, leave frames at the ends of many
sweeps with no exposure at all. Until now those frames went into the scaling like any other: on a
CCD their noise, divided by an incident flux of ~0.005, made the partials loop walk their scales
toward zero until the 50x collapse guard dropped them - or, if the loop stopped first, merged one as
noise, which delta-CC1/2 then removed with its whole 10 deg batch and reported as SWEEP_GAPS (3p85,
frames 338-359). On photon-counting detectors the blank frames fitted G = 0, escaped the guard and
were reported as merged although nothing of them was (cytc_x10sa, myob_x10sa, thau_micromax_pink,
lysoI_micromax_pink, 3p85).
TrimBlankSweepEnds (RotationScaleMerge.h) walks in from each end and clears the reflections of every
frame that
- recorded no diffraction: as many of its reflections stand more than 3 sigma below zero as above
it (a sign count, so no sigma calibration is needed and a slight background bias cannot hide the
strong reflections), and that excess is 3 sigma clear of the run's typical frame - so a crystal
too weak for one frame to show it is never trimmed; and
- had no beam: its mean background is nearer to none than to the run's typical frame. That is what
keeps a crystal leaving the beam, which also records nothing but still scatters, in the sweep
ledger as a finding (insu_I_x06da_ref, 9zm0, 9fhc, 5reo are untouched).
No stretch length: one blank frame is as blank as twenty. Partly exposed first/last frames recorded
diffraction and stay (7tcd, 7q6j, 9z72, lyso_micromax_pink). It runs on the integration outcome
before the overload count, so the merge, post-refinement and unmerged output all see the trimmed
sweep, and in --mode scale on the stored reflections. The report gives FRAMES_BLANK_START/END and a
prose line - information, no warning; the frames count as rejected. The sweep ledger looks for
degraded stretches, and reads its running medians, only between the first and last frame with data.
Delta-CC1/2: a convicted stretch that stops short of a sweep end by less than one rocking event takes
the rest, where those frames read as the stretch does (5reo: 331-396 rejected, 397-399 were left in as
a three-frame weak_diffraction range with its own warning).
Battery --only, 31 sets, tol5 f217f180a -> this commit: no space group, d_min or verdict changes;
merged statistics identical within 0.001 except 5reo (R_meas 0.236 -> 0.231, ISa 23.0 -> 23.3,
R-free 0.2013 -> 0.2004, warnings 3 -> 2). Trimmed: 3p85 0/19 (+3 empty frames; collapse-guard drops
13 -> 0), 6w4h 0/5 (guard 5 -> 0), 7rji 0/4 (delta-CC1/2 no longer removes 3406-3412, two of which
diffracted: 7 -> 0), cytc_x10sa 0/2, thau_x10sa_* 0/1-2, myob_x10sa 0/1, lyso_x10sa_strong 0/1, 8a1a
0/1, lyso_micromax_mono 1/0, lysoI_micromax_pink 2/0, thau_micromax_pink 1/0. Controls (11if, 5f6m,
6moj, 8u0i, 9fhc, 9z44, 9z72, 9zm0, 6iu6, insu_I_x06da_ref, myob_x06da_split, lyso_x06da_half_image,
thau_bl1a_4p6keV) trim nothing and are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
The per-frame scaling tolerance goes from 1e-3 to 1e-5, and the Anderson onset (also the round the
fulls loop keeps its cross-validated lambda) becomes an absolute 1e-2 instead of 10x the tolerance.
Evaluated on 35 merged sets (ci tier + scaling-audit sets), on top of the loop fixes:
- plain 1e-3: 8913 iterations, 38 s of loop time on GPU, 7 loops at the 100-round cap;
- plain 1e-5: 35212 iterations, 120 s, 17 loops at the cap;
- Anderson 1e-3 (previous default): 3715 iterations, 16 s, merged intensities as far from
the converged ones as plain 1e-3 (median 0.13%, max 4.3%);
- Anderson 1e-5 (this commit): 6507 iterations, 27 s, longest loop 89 rounds, merged
intensities within 0.2% of plain 1e-5 and identical SG, d_min, CC1/2, ISa, R_meas, R_free.
No space group or verdict changes in any mode. The converged merge is not measurably better
against the references, but it no longer depends on where a loop happened to stop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
Every cuFFT plan has the driver compile cuFFT's PTX kernels: cufftMakePlanMany ->
cuModuleLoadData -> the PTX JIT, under cuFFT's process-wide lock. A second plan of the same
shape pays again, even while the first is alive. The CUDA JIT cache makes that ~1 ms. Without
one (CUDA_CACHE_DISABLE, no writable $HOME) a plan costs 20-250 ms. That holds for every size
(64^3 and 1D 256 included), every transform type and batch, on sm_75 and sm_120, with cuFFT 11.2,
12.0 and 12.4. No CUFFT_* setting changes it, so a choice of grid sizes cannot avoid it; only
the number of plans made can be cut.
Before this change, a plan was lent to one user at a time, so each concurrent user of a shape
on a card made its own: 9-24 rigid-body engines, the translation slices and the score engines.
Now a plan is made once per card and shape and shared. Each user keeps its own stream and work
area, and CudaFFTPlan::R2C/C2R set both and launch under the plan's mutex. Shared plans give the
same bits as private ones: 8 threads on one plan with their own work areas, 9 shapes x 160
transforms, 0 mismatches on RTX 5080 and T4. A shared work area, the negative control, gives
2-13 mismatches per shape where the plan has a work area.
The rigid-body engine also dropped its batch-3 plan: the Jacobian transforms its three grids one
at a time on the batch-1 plan. That gives the same bits (10 zone-grid shapes, sm_75 and sm_120)
and one plan less per zone and card.
The cytochrome C and thaumatin MR test sets, AF models (MR step, s):
4x T4 cytc cache on 2.22 -> 2.22 cache off 8.41 -> 4.00 plans 86 -> 24
4x T4 thau cache on 1.58 -> 1.61 cache off 11.13 -> 3.08 plans 131 -> 23
1x T4 cytc cache on 2.87 -> 2.81 cache off 8.82 -> 3.84 plans 80 -> 12
1x T4 thau cache on 2.41 -> 2.39 cache off 10.42 -> 3.17 plans 104 -> 11
RTX5080 cytc cache on 1.41 -> 1.12 cache off 6.5-10.7 -> 2.02
RTX5080 thau cache on 0.90 -> 0.85 cache off 8.03 -> 1.60
Outputs are bit-identical before and after, cache on or off: model, merged hkl, maps, and every
MR and validation log line. The mtz/cif differ only in the command line they record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
cuFFT ships its transforms as PTX for every card (cuFFT 11 and 12 alike), so each plan has the driver
compile kernels, and a second plan of the same shape compiles them again. From the driver's JIT cache
that costs ~0.5 ms a plan; without it 45-130 ms, under cuFFT's process-wide lock - which, over the
~100 plans of molecular replacement, matches the stage pattern of the slow MR in the production broker.
The cache is unusable in more ways than CUDA_CACHE_DISABLE: with no HOME at all (a systemd system
service without User= gets none, and the driver does not fall back to the passwd entry), on a read-only
or unwritable directory, on a filesystem whose fcntl locks fail (NFS without locking) - each measured
as slow as no cache - or on a slow network filesystem (each plan opens and locks the cache ~10 times).
- common/CudaJitCache: the path the driver uses (CUDA_CACHE_PATH, else $HOME/.nv/ComputeCache), its
filesystem type, whether it is writable, whether its index opens read-write and takes the driver's
fcntl read lock, CUDA_CACHE_DISABLE and a CUDA_CACHE_MAXSIZE below 64 MiB. One line: info when fine,
a warning with the cause and the remedy (CUDA_CACHE_PATH on a local disk) when not, or when a plan
takes over 10 ms.
- WarmUpCuFFT returns the fastest of three 64^3 R2C plan makes after a first one that pays the
initialisation (and a cold-cache compile): ~2 ms in all with a cache, ~0.3 s without.
- jfjoch_broker runs it after its version line (0.15 s at start, cuFFT init and card 0's context
included); rugnux on its existing background warm-up thread.
- CUDAMemHelpers.h: the modules are PTX on every platform, not only on Windows.
No result changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
The stall rule ('step not below 0.9 x its best for five rounds') called a steady slow fall a stall -
82 of the 245 loops that ended unsettled in the rc176 full battery were still falling at < 2% a round -
and every unsettled loop returned its LAST state: a fulls loop that diverged from round 1 handed on
the state after five rounds of walking. Now a loop is stalled when the median of the pairwise slopes
of log(step) over its last six rounds (Theil-Sen) is not negative - a median, because a least-squares
slope let one round's jump (the smoothness re-chosen five decades away) end a loop falling by a
quarter a round - and however a loop stops (settled, stalled or at the cap) it ends on the plain round
with the smallest step. On a settled loop that is the last round, so nothing changes there; on any
other it is the state closest to a fixed point the loop reached. The partials' loop restores it
through the round's own rescale (ratio fitted G / best G), so the corr bookkeeping of the CPU and GPU
paths is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
ShrinkToRestrained ran only off a fulls loop that reported itself converged, so whether each frame
kept what its own fulls say beyond the smooth curve depended on whether the last step happened to
cross the tolerance: one sweep's ISa read 35.09 or 34.46 by how its loop stopped. The hand-back is a
continuous decision already - each frame keeps the fraction its observations support, read off the
lag-1 covariance of the sweep's deviations - so it is now applied after every fulls loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
The fulls loop re-chose its smoothing lambda by cross-validation every round, so every round was a
different map and the loop had no fixed point: on 149 of the 245 fulls loops that ended unsettled in
the rc176 full battery lambda jumped >= 2x over the last rounds, and one sweep cycled between 1.2e5 and
3.8e7. The choice is now kept from the round whose step falls below SCALING_ANDERSON_ONSET (10x the
tolerance) - the round acceleration starts on, which models one map too. In the audit (env knob
RUGNUX_FREEZE_LAMBDA, non-sticky) this took settled fulls loops 12/30 -> 21/30 on five sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
Both statistics divide the disagreement a promotion's added operators bring by the P1 search
merge's sigmas, which carry the error model the P1 scaling fitted. P1 scaling never ties symmetry
mates together, so the better the per-frame scaling converges, the smaller those sigmas and the
larger the same real systematic floor between symmetry mates reads: the ratios tend to functions
of the sigma-free disagreement H measures directly (chi^2 ratio ~ H ratio^2, b ratio ~ H ratio).
Measured on the weak in-house cubic insulin set (true I23), all-observation arm, 23 over 222:
plain, tol 1e-3 chi^2 2.024 b ratio 1.438 (bound 1.44, rescued) -> I23
plain, tol 3e-4 chi^2 2.072 b ratio 1.453 -> I222
plain, tol 1e-5 chi^2 2.094 b ratio 1.460 -> I222
Anderson 5, 1e-3 chi^2 2.077 b ratio 1.453 -> I222
chi2_ref 4.16-4.18 (> chi2_ref_reliable, so the b ratio alone decided); H ratio 1.46-1.50, every
sigma-free gate passing. The late re-merge comparison (R_meas 0.1388 -> 0.1419, ISa 21.5 -> 19.5)
then kept I222: a genuine cubic step with a 3-fold systematic floor always costs a little.
These two tests exist for the twin law H cannot tell from real symmetry. A twin compresses the
intensity distribution; this crystal's <|L|> is 0.48, the twins they refuse read 0.39-0.43
(including a trigonal twin, the only corpus case so far refused by them alone). So they are now
skipped where the merge reads untwinned (<|L|> >= 0.44, AnalyzeTwinning's bar), and the
sigma-free gates decide there alone. With it the set gives I23 under plain scaling at 1e-3, 3e-4
and 1e-5 and under Anderson at 1e-3 and 1e-5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4
