Saturation clips the strongest spots, and those are at low resolution. On a Rayonix stills set the
saturated pixels sat almost entirely past 4 A (0.002% of the reflections coarser than 4 A against
0.59% overall) - flagged or hot pixels, not clipped spots - and keeping those observations as lower
bounds put the saturation limit into high-resolution intensities: R_split 0.12 -> 0.93, CC1/2 0.99
-> 0.33. The run now keeps overloads only where their share coarser than 4 A is at least 1.5x their
overall share; otherwise they are flagged and dropped as before, with a log line saying so.
On clipped CCD stills the share coarser than 4 A is 2.6-5.9x the overall one and nothing changes
(R_split and R_free identical); on the Rayonix set R_split returns to 0.12. A pragmatic gate until
saturation is told from bad pixels per spot. Rotation never keeps overloads and is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Stills keep overloads as lower bounds on CPU/CUDA but not yet on Metal, an accepted platform difference
to be fixed. On a rotation experiment both engines flag the overloads as before, so the bit-identity
comparison stays meaningful. The engine reads the rotation flag only for that decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The Metal kernel and host code are restored to their panel-s5 content, so neither the background clip
that follows a handed-over radial curve nor the count of saturated pixels in pass A runs on the Metal
path. This is an accepted, temporary platform difference, to be closed once the kernels can be checked
on a Mac: on macOS a still's background clip stays the plain one (the radial correction itself still
uses the handed-over curve, as on the CPU), and an overloaded observation kept as a lower bound counts
its saturated pixels as 0 instead of at the saturation limit.
One host-side change is kept, with no effect on any result: the radial sums are accumulated on the GPU
only where no curve was handed over. StartPassA now sizes those sums for that case alone, so the old
host code would write past them on every still with a curve.
The shared structs need nothing on the Metal side: BraggPassA.h is compiled into the kernel source, the
box-sum kernel zero-initialises BraggBoxSums (n_saturated = 0) and the host zero-initialises
BraggPassAFrame (n_curve = 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The one conflict, tests/BraggIntegrationEngineCPUTest.cpp, where both branches add cases at the same
place: integ-pal's side is taken and stills-sat's test diff applied on top, which keeps all four cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The joint distance+cell fit freed the distance even where the user had given it. A user-given distance
is a hard pin: the cell alone absorbs any error. The file's distance and DecideStillsGeometry's stay
free as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills bundle held whichever of the two it trusted - a given or stated distance, with the cell
refined to absorb any error, or, after DecideStillsGeometry had rejected the file's distance, the
user's cell anchored and the distance refined from it. The two are degenerate only near the beam: the
radius of a spot goes as D tan(2 theta) and the cell scales sin(theta), so the strong frames'
high-angle spots fix both. A distance given 1.4% short (the user's cell was the depositors' starting
cell, about a percent off the crystal) was absorbed into the cell, mispredicted the spots past 2.5 A
and made the merged intensities there negative.
Both are now free (refine_distance, no cell restraint) whatever their source; a --detector-distance
on stills with a reference cell is the fit's starting value, no longer a pin. The flags that chose
between the two regimes (stills_distance_nominal, stills_distance_from_cell_) are gone.
On a Rayonix stills set given 105 mm the fit converges to 106.1 mm from 105 and from 103 mm, and the
deposited model's R_free against the merge drops 0.354 -> 0.283 (all frames); a stills set whose
distance is right holds it (193.43 -> 193.42 mm, R_free unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A manifest entry may carry max_images, applied when no tier is given; both
EuXFEL entries get 2,000 (and a smoke tier of the same size), which keeps
their full-set run time out of the development batteries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The batteries now score the deposited model against a merge one way: model_check.py, REFMAC's first
cycle before any shift (bulk solvent and anisotropic scale by default), the depositor's free set,
d_min_used = max(ours, deposited), compared against the deposited data through the same protocol.
- model_check.py: the restrained-refinement protocol is removed (refine, refined_shared,
rename_misfit_ligands, REFINE_CYCLES, the refined parameter, --no-refine). Both data sets are now
scored on their common reflections only, so a low-resolution or anisotropic cut of either applies
to both (measured on three still sets: <= 0.001 change).
- battery.py / paper_plots.py: refmac_refined_* and refmac_refined_ratio are gone; refmac_rfree_ratio
is the headline; paper_plots draws the first-cycle pair only (--rfree removed).
- still_battery.py: dimple is replaced by model_check.check on the same p.mtz. R_free rugnux is
rfree_depflags, the target is rfree_depdata (no longer min(dimple, published)); the published
R-free stays as an information column. The dimple cache, free-flag writer, deposited-MTZ writer
and the reference verb are removed. This also retires dimple's freerflag COMPLETE renumbering,
which scored R-free on depositor work reflections whenever our merge had hkl the deposition
lacks (model_check carries the depositor's flags per hkl and marks ours-only hkl as work, and
with the common-hkl restriction there are none). `report RUN --out DIR` rescores an older run's
merges without rerunning rugnux.
Refinement cost about 6.4 h per full rotation battery; the 0-cycle check takes seconds per set.
Verified: 9V3D still merge reproduces the 0-cycle common-hkl numbers (0.2461 ours, 0.2367 deposited).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The validation battery of 61def00a1 (20261022-0925_29000d_still-integ-pal-r1clip) shows the r1 rule
buys nothing once the ring clip follows the radial curve: lcls222 at r1 4 (4,6,8) 0.1893 against
0.1856 at r1 6; pal86 0.1998 against 0.1997. The radius scans that motivated it were taken with the
flat clip, whose beam-side cut the smaller ring partly avoided. r1 stays 6 on stills.
The empty apertures' corrected column again clips flat. With the clip following the curve there,
lcls242_s read the correction as a clear gain (z = 11) and the run took it, but dimple on 40,000
frames says it costs 0.013 (0.2461 on against 0.2335 off); judged on the flat clip, as in 07a43a13e,
the decision is off there and unchanged on every other set. The integrator still follows the curve
in its clip wherever the correction is on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills merge dropped every observation with a saturated pixel. On a still the saturated
observations are the fully recorded ones of the strongest low-resolution reflections, so what
survived were their partial observations, whose partiality correction then set the merged
intensity: on clipped CCD stills the merged low-resolution intensities came out at 0.4-0.6x of
the deposited ones, a ramp no Wilson B absorbs, invisible to CC1/2 and R_split.
The pixels are really clipped (flat-topped plateaus a few per mille wide, a pile-up in the
pixel-value histogram from the per-pixel ADC limit), so the auto ceiling is right; the drop was
the defect. A still now keeps the observation: the box sum with every saturated pixel counted at
the saturation limit, unflagged and marked clipped, counted in BraggIntegrationCounts and
reported as OVERLOADED_REFLECTIONS_KEPT. Rotation data (IsRotationIndexing) are unchanged and
keep the flag for the rocking-event drop.
Measured on a clipped-CCD stills subset (dimple R-free, deposited free set): dropped 0.247,
lower bound 0.230 (same as switching saturation off). A Kabsch fit to the unsaturated wings read
lower than the lower bound (0.237), and inflating the lower bound's sigma lost (0.255).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two stills-only changes to the integration stencil.
1. Where the whole-frame radial curve is in use (the correction on), the ring's high-side clip
follows it: a ring pixel is clipped against the threshold plus the curve's rise from the
reflection's radius to its own (BraggIntegrationEngine::RadialClipOffset, rounded to whole counts
so CPU, CUDA and Metal keep the same pixels). On a background falling steeply away from the beam
the flat mean + 4 sqrt(mean) threshold cut the beam-side half of the ring as outliers, so the
background read low and reflections near the beam stop read 3-150x too strong. The empty-aperture
test's corrected column clips the same way, so the per-run correction decision sees it.
Measured with r1 6 (dimple R_free, battery arguments, all frames):
lcls222 0.196 -> 0.1856, pal86 0.216 -> 0.1997, sacla236 0.2559 -> 0.2559, alvra213_int16 0.1986 -> 0.1986.
2. The signal disk is taken from the lattice: it must not reach into the next node's spot. The
closest two nodes can come on the detector is the shortest reciprocal-lattice vector the centring
allows, s = F_px tan(2 asin(lambda / 2 d_max)), and a spot reaches 2 r80 (measured in the
pre-scan); r1 = the largest of 4, 5, 6 px not above s - 2 r80, never below 4, never above the
disk the run starts with (6). The empty apertures are measured for each candidate disk and the
ring is chosen on the chosen disk's tally. Pixel detectors keep 6 (sacla236 s 22 px, alvra213
20 px); the binned CCDs with a long axis shrink to 4 (lcls222 5.4 px, pal86 6.7 px), which the
radius scans there measured as the optimum; pal90 (s 21 px) and lcls242_s (15.5 px) keep 6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Owner's rule: a set is scored against min(R-free of the deposited data through our dimple protocol,
published R-free). The report shows both values, a "target" column naming which one was lower, and
the difference and win/draw/loss verdict against that target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The published R-free comes from the depositors' own refinement (mostly phenix.refine), while the
battery refines with dimple's 8 REFMAC cycles, so the two are not like-for-like: on LCLS 222 (8TDQ)
the deposited intensities reach 0.1918 under dimple against a published 0.2012.
Every set now carries "R-free deposition (dimple)": the deposition's -sf.cif (intensity_meas, else
the mean of the anomalous pair, else F_meas_au squared) refined with the same refine() as our merge -
same deposited model, the deposition's free set, cut at its resolution. It is computed once per entry
and resolution and kept in the PDB cache (<pdb>-dimple-<d_cut>/reference.json); `run` and `report`
fill in a missing one, `reference` computes or lists them up front. The report gains the difference
(ours minus the deposition's) and a quality verdict (win below -0.01, draw within 0.01, loss
otherwise); the published R-free stays as an informational column.
Reproduces the hand-computed references: 8TDQ 0.1918, 9V3D 0.2220, 31JT 0.1945.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
One Ceres fit keeps only ~5-6 of 32 cores busy whatever it is given, and on a loaded machine (load
28-36, JF16M int16, 4000 strong frames) the five side-by-side fits measured 10.9 s with every thread
each, 13.8 s with half, 14.6 s with a fifth, against 18.9 s one after the other (bb7c9c). The
thread split is dropped: each fit gets gr_settings' threads, as before the fits ran side by side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The background-ring log line now gives how many standard errors the correction lowers the chosen
ring's mean square by, over the apertures measured both ways. Diagnostic only; the decision is
unchanged. Full pal90: z = -2.7 (correction off), where dimple prefers it on (0.2124 vs 0.2183).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Without a resolution limit the outer d*^2 shells of the width measurement lie beyond the diffraction
limit, where the summed intensity is noise or even negative; one such shell made the whole measurement
fail, and 5 of 11 still-battery sets (sacla237, pal86, pal90, lcls242_s, lcls242_t) fell back to the
per-crystal window fit. A shell now enters the fit only where its summed intensity is measured to 10 %
(sum I / sigma >= 10), and at least three shells are needed.
Weighting the fit by the shells' signal instead was tried and rejected: it lets the strong low-resolution
shells set the width alone, and on the MicroMAX pink set the dimple R-free went 0.2442 -> 0.2748.
The matrix subsets, which are cut at their resolution, are unchanged within 0.0010 dimple R-free
(SwissFEL 0.2453, -C/-S 0.2513, mono 0.3188, pink 0.2442, SACLA 243 0.2455, SACLA 237 0.2327,
LCLS 222 0.2335, PAL 86 0.3628).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The first stills battery of e8145d8 lost on one Rayonix set (lcls242_s, dimple R_free 0.2302 ->
0.2323): there the whole-frame radial correction hurts. On 40,000 of its frames: battery default
0.2462, ring 12 with the correction 0.2444, ring 12 without it 0.2336, ring 14 with it 0.2467. Its
beam is strongly polarised and the azimuthally averaged ring background does not describe the local
background.
So the empty apertures now also carry the corrected difference: the frame's ring background
(RingBackground, factored out of the engine) averaged over the ring pixels minus over the disk pixels
- what the integrator's kernel takes off a reflection - is subtracted from each disk-minus-ring
difference. The ring is still chosen on the plain differences, so its reach does not rest on the
radial model; where the correction was left to auto it is then kept only if it makes that ring predict
the disk better. Decisions on 15-60 pre-scan frames:
pal86 ring 10, correction on (240.6 -> 221.1) lcls222 ring 10, on (6.34e4 -> 3.41e4)
lcls242_s ring 12, off (277.2 -> 285.8) alvra213_int16 ring 14, on (0.01424 -> 0.01422)
Choosing ring and correction jointly instead picked ring 14 with the correction on lcls222, where dimple
reads 0.204 against 0.196 at ring 10; a median instead of the mean square picked ring 14 with the
correction on lcls242_s, the worst arm there. The sequential rule is the one that agrees with dimple on
every set measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
GetConsensusUnitCell averaged every indexed frame within 5% of the -C cell. The given-cell
population filter runs after it, so the MTZ, report and model-validation cell was a mixture of
both forms. Frames whose retained outcome is empty now do not count, and the consensus cell is
read again once the population filter has cleared the other form's frames.
Two-form P21 serial data (LCLS):
- T run: 93.85/93.66/49.98/92.00 -> 95.21/94.50/50.42/91.09 (kept population ~95.5/94.1/50.3/91.2)
- S run: 92.38/93.29/49.59/93.26 -> 91.90/93.06/49.45/93.65 (deposited 92.04/93.02/49.01)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
--capture-uncertainty now reaches the stills merge: each partial carries (c (1 - p) <I>)^2 in its
variance, the rotation combine's capture term per observation, in the merge weights and in the
error-model samples alike. --reject-outliers on stills now uses the rotation merge's band
(OutlierBand.h): exp(+-n s) about the reflection mean for the systematic part, s the weighted median
|ln(I / median)| of the merge's own observations, instead of a symmetric +-n sigma band that reaches
below zero when b is large (stills b 1.4-1.7). A median needs three observations.
Neither is switched on for stills: measured with the run-measured partiality width before its
window-truncation correction (dimple R-free, SwissFEL 4k / SACLA 243 / MicroMAX mono / MicroMAX pink):
no options 0.2567 / 0.2497 / 0.3279 / 0.2477, --capture-uncertainty 1 0.2510 / 0.2470 / 0.3261 /
0.2527, --reject-outliers 6 0.2619 / 0.2487 / 0.3317 / 0.2630 - mixed, and the photon-counting pink
set loses with both. Raising --min-partiality to 0.1 or 0.2 lost 0.008-0.019 on SwissFEL and mono.
Defaults unchanged, so stills output is unchanged; rotation does not use MergeOnTheFly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills partiality width came from a per-crystal least-squares fit of de^2 against d*^2 over the
integrated reflections. Those fill the prediction window evenly, so the fit returned the window's own
width (W^2/3, constant in the linear Ewald distance, gamma_e ~ 0): at low resolution several times the
crystal's rocking width, which mis-corrected every partial there.
The width is now measured once per run from what the reflections RECORD: the intensity-weighted second
moment of the excitation error, sum I de^2 / sum I, which is the width of a Gaussian partiality whatever
the window, in 8 shells of d*^2 over all crystals. Each shell is corrected for the window cutting the
curve's tails (a curve wider than the window is taken as wide as the window), and the shells are fitted
as s^2 = gamma0^2 + (gamma_e d*)^2 + (bw lambda d*^2 / 2)^2 with every term non-negative. The per-crystal
fit stays as the fallback when the run is too small to measure (< 800 reflections). The prediction window
is unchanged (2x the per-frame profile radius).
Measured: on the SwissFEL lysozyme stills the width comes out ~0.035 deg angular at every resolution,
the ~0.03-0.045 deg that the empirical rocking profile of the observations gives; the window-moment
model was 3-4x wider at low resolution.
dimple R-free (deposited free set, matrix subsets/cuts, PARTIALITY_MATRIX.md), shipped -> this:
SwissFEL run0237 4k 0.2692 -> 0.2454
same, -C/-S given 0.2679 -> 0.2513
MicroMAX mono 4k 0.3341 -> 0.3189
MicroMAX pink 4k 0.2415 -> 0.2442
SACLA 243 subset 0.2576 -> 0.2454
SACLA 237 subset 0.2342 -> 0.2331
LCLS 222 subset 0.2344 -> 0.2330
PAL 86 subset 0.3499 -> 0.3629 (binned CCD, stencil-limited)
For comparison --bandwidth 0.003 by hand: 0.2616 / - / 0.3108 / 0.2377 / 0.2460 / 0.2270 / 0.2404 / 0.3469.
Stills only: rotation never runs StillsPartialityRefine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Two changes to the stills integration background, both stills-only (rotation keeps its path bit for bit):
1. The r2..r3 background ring's outer radius is measured per run. The pre-scan lays apertures on a
grid over its frames, away from every found spot, and for each candidate r3 (r2+2 .. 14) takes the
mean square difference between the r1 disk mean and the clipped ring mean. With nothing in the
aperture that is exactly the background error the ring makes (noise + bias); the run keeps the
minimum. On a photon-counting pixel detector the noise wins and the shipped r3=14 is kept; on a
binned CCD the background structure wins and the narrowest ring is taken. r1 stays 6: a smaller
disk (4,6,14) costs 0.0013 dimple R_free on a JUNGFRAU set, so it does not pass the
photon-counting gate. --adaptive-integration-radius now defaults on for stills too (=off restores
the fixed 6/8/14). The pre-scan bandwidth estimate is not applied to stills.
2. The radial background-curvature correction on stills takes its curve from the adaptive spot
finder's peak-excluded per-ring background (whole frame) instead of the predicted reflections'
own ring pixels, and auto applies it to every stills image (no ice-score gate: a solvent/carrier
hump curves the background as much as ice). New engine call RadialBackgroundFromRings; CPU, GPU
and Metal read the handed-over curve. The GPU box sum now sizes the radial bin index separately
from accumulating, so a handed-over curve is read at the right radius.
Measured (dimple R_free, deposited free set, battery arguments, all frames unless noted):
pal86 (Rayonix 4x4, 156 um): 0.354 -> 0.218 (ring 10; at 4,6,10 0.213)
lcls222 (Rayonix 4x4, 177 um): 0.205 -> 0.196 (ring 10)
sacla236 (MPCCD): 0.2588 -> 0.2558 (ring 11)
alvra213_int16 (JUNGFRAU): 0.1986 -> 0.1986 (ring 14, unchanged)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The geometry refinement's sample (up to 10000 frames: spot finding + indexing on the card, the same
per-frame work as the image pass) and the stills geometry trial's 96-frame spot finding ran -N workers
(32 here), each with its own GPU engines. The image pass caps the workers that drive a card at 16 per
card, fewer where their engines would not fit half the card (GPUWorkersPerCard), because past that a
card stops taking queue work and more workers make the loop slower. The cap is now one function,
ImageWorkers(), used by all three loops. On a 16.9 Mpx JUNGFRAU 16M the 32 sample workers held ~13 GB
of a 16 GB card's engines (24 B/px each) where the cap allows 19.
Which frames the sample examines does not depend on the worker count (fixed stripes), and each trial
frame is found on its own: outputs identical (JF16M int16 15 runs: same 5174 frames sampled, same
module placement, geometry and merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Four changes to the per-run fixed cost of a stills run (Step 2 geometry refinement, Step 3 pre-scan),
none of which changes what is computed:
- Module placement: the four held-out fits and the placement on every strong frame are independent, and
one Ceres fit keeps only ~4.6-6.5 of 32 cores busy (perf, JF16M, 4000 frames / 150888 spots). They
now run side by side on a fifth of the threads each; the placement fit is computed speculatively and
used only where the held-out residuals call for it, as before. Baseline (quiet): held-out 12.3 s +
placement 5.1 s on JF16M, 16.4 + 6.5 s on JF4M, 19.0 + 7.6 s on MPCCD.
- Module scan (+-25 px grid per module ahead of the placement) was one thread: every grid shift is now
scored on the workers and the grid then walked in its old order, so the first best still wins
(3.3 s on JF16M, ~5 s on JF4M/MPCCD before). New test ScanModuleShifts_FindsShiftedModule checks the
shift is found and that 1 and 4 threads give the same answer.
- Stills geometry trial: each hypothesis (centre, distance, frame sample) is indexed once. The distance
grid ran twice on the same centre whenever the file's centre was kept (40 of ~275 hypotheses,
~15% of the trial: 6.2 s on JF16M, 6.7 s on the flat JF8M), and the final here/there check repeated
the first trial. Repeats always gave the same count (checked on three logs).
- The saturation histogram (40 frames, 0.1-2.2 s) reads only frames and the mask: it is taken beside
the geometry refinement and the limit decided where it was, after it.
Plus TIMING marks for the sub-steps (geometry refinement sample, module scan, module placement fits).
Outputs identical to the baseline binary (bb7c9c) where checked: JF16M int16 15 runs (geometry, all
32 module shifts, held-out residuals, cell), JF4M int16, MPCCD 236 (indexing rate, CC1/2, R_split,
R-free with --model identical), PAL 90 and LCLS 222 (identical reports).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The SwissFEL .geom files name their pixel mask with an absolute facility path, so the sets read a
copy beside the original (jf-16M_mask.geom, mbco_march_mask.geom) whose mask_file is relative to the
copy's directory; only that line differs. EXTERNAL_TEST_DATA_STILLS.md says how to make them.
On a 2000-frame subset of the int16 MbCO set, before (geometry without masks + --mask-good) and after
(the geometry's mask and bad regions): indexed 98.2% both, R_FREE (--model) 0.244 -> 0.234.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
- The NXmx master (and _process.h5) writes module_0..N with the panel table when the start message has one,
besides the whole-image module, so a stored-layout file reads back with its panels
(JFJochReader_PanelId checks the round trip).
- RotationScaleMerge's modulation grid spans PlaneBounds instead of the image rectangle; with no panel
table x0 = 0 and the extent is the image, so the cells are unchanged.
- The beam-stop / pre-scan memo key includes each panel's box and placement; nothing is added with no
table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A detector of two or more panels is no longer assembled. The image is the stored frame, and each panel of
the geometry (CrystFEL .geom or the file's own panel table) is a DetectorModule over its box of the frame,
at its true placement on the detector plane: fractional origin, the panel's own axes, no snapping to an
image axis or a whole pixel. Pixels on no panel, edge, bad-region and file-masked pixels are masked in the
stored layout. The block scatter (a transpose for the 90-degree-turned JUNGFRAU and MPCCD panels) is gone
for these inputs. A single panel keeps the old block path, so single-panel CXI files are unchanged.
Signed-integer frames in their own units (gain 1, no per-frame mask) go up with no copy: a bitshuffle/LZ4
chunk still compressed, for the GPU preprocessor's BSLZ4 decoder (int16's smallest value is its error
value there, so the type-min rule holds), a deflated one inflated outside the HDF5 lock straight into the
caller's buffer. Everything else (float, unsigned, a gain, a per-frame mask) is converted once in stored
order. A float NaN/inf now reads as no measurement (INT32_MIN) instead of 0, and values past int32 are
clamped before the cast.
Also: SpotToSave.panel is filled from DiffractionSpot::Panel(), so AssignModules takes the panel the spot
was found on. PreviewImage_PanelLayout expects stage 3's PlaneRasterOf raster (merge fallout).
Main pass, 2000 frames, vs the CCD-reader build: JF4M int16 bslz4 465 -> 1860 frames/s, JF16M 105 -> 1111,
MPCCD 212 -> 252; flat Jungfraujoch data unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The .geom parser now takes the masking information the format defines:
- mask arrays: mask/mask_file/mask_good/mask_bad (mask 0) and maskN_data/maskN_file/maskN_goodbits/
maskN_badbits for N = 0..7; global keys hold for the panels after them, a panel may override. A mask
that is the same on every panel is one whole-frame mask, otherwise one per panel, applied on that
panel's pixels only.
- a relative mask_file is looked for in the working directory (as CrystFEL does), then beside the .geom.
- bad regions (bad*): a lab-plane box (min_x..max_y) or a fs/ss range on a panel (or, old syntax, on
the whole frame); mask_edge_pixels and no_index.
JFJochXFELReader already had the static pixel mask and the per-frame mask path for these; the masks
go through them unchanged. A mask whose path holds % (one per shot group) is applied frame by frame
like a 3D one. Where the geometry names mask arrays, the data file's own Cheetah masks (CXI
mask_shared and per-shot mask with Cheetah's generic bad bits, /metadata/pixelmask) are not added:
the geometry says which bits matter (e.g. an AGIPD mask_bad that includes bit 32). --mask-good /
--mask-bad still add their masks to these.
Tests: [CrystFELGeometry] (new keys, per-panel override, relative mask file), JFJochXFELReader_*
(new GeometryMasks case on a synthetic CXI + .geom), [JFJochMultiReader], JFJochXTCReader*.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Measured values are clipped to [-32767, 32767] so that only a pixel without a measurement holds -32768.
The existing copies are unaffected: none of the converted SwissFEL runs had a non-finite pixel or a
clipped value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The AGIPD sets are run on the files as deposited (Cheetah CXI with the per-ASIC CrystFEL geometry),
and their processing speed is reported as it is on those files; only the SwissFEL float32 sets keep
an int16 photon copy. The reader rule that a signed integer frame's smallest value is no measurement
stays: it is general.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Each set's row now says how many images we use and, where that is a subset, what the entry or its
deposition holds (SACLA 237 ~1.8% of dark_ref, LCLS 222 one of four runs, LCLS 242 runs 256-267, both
AGIPD sets and A2A against their deposited frame counts). The section listing what is not used is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
DecodeInto no longer turns every stored pixel into a float and back:
- the gain is one value (adu_per_photon_), not a float per stored pixel
- it always was one value; only --gain or a measured gain set it;
- a 16-bit signed frame (Cheetah CXI, SACLA) without a gain is handed
up as Int16, in its own type; other integer frames are widened to
int32 without a float; only a float frame or a gain other than one
goes through lround(v / gain), exactly as before;
- placement is per panel (start, size, destination steps) instead of a
64-bit destination per stored pixel; a panel square with the image
is copied row by row (PAL: 1440x1440 into 1444x1444 - the two-pixel
margin is Place()'s, kept so the geometry does not move);
- the decode buffers (chunk, inflated, unshuffled) are reused per
thread, the bitshuffle decode is not copied again;
- a chunk whose HDF5 filter_mask is not zero (a filter skipped) is
read by HDF5 instead of being decoded as if it were not;
- the never-set value flags (flag_lessthan/morethan/equal) are gone
from the layout and from the per-pixel loop.
MeasureGain in the CLI now reads Int16 frames as well, so a SACLA run
still has its gain measured.
Tests: JFJochXFELReader_SACLA/_CXI updated for the mode, new
JFJochXFELReader_CXI_Int16 (deflate without shuffle, a chunk with its
filter skipped) and JFJochXTCReader_StoredPixels (synthetic XTC,
stored hand-up and the offset path).
Measured on two PAL-XFEL Rayonix sets (int16, deflate; RTX 5080,
5950X, page cache warm; reports and .hkl identical to the baseline):
main pass, 2000 frames: 296 -> 799 frames/s (pal86),
302 -> 755 frames/s (pal90)
wall, 2000 frames: 26.4 -> 13.6 s, 32.1 -> 19.4 s
(the geometry-refinement steps, which read frames too, halve)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The main loop's read (ReadRawImage) now hands a camera frame up as the
file stores it - 16-bit pixels, UINT16_WIDENED, as the marCCD reader
does - wherever there is no pixel offset to take off and no gain to
divide by; the GPU preprocessor widens it on the device. The host no
longer widens 3.7 Mpx to int32 per frame (the loop that was ~79% of
the main pass's user cycles). xtc::ReadFrame becomes xtc::ReadPixels,
the stored pixels only; the int32 decode (LoadImage, or an offset or
gain) widens them in one plain loop.
The streams are indexed in parallel (one worker per stream, up to the
hardware threads), merged in the order given: each index is a walk of
small reads, waiting on the disk one at a time.
Measured on a Rayonix MX340 run, 1920x1920 (RTX 5080, 5950X, page
cache warm, other agents' builds running; output identical but for the
known run-to-run flip of one observation on one set, which the
baseline shows between its own runs):
main pass, 2000 frames: 479 -> 789 frames/s (lcls222),
413 -> 477, 388 -> 452 (the two forms of
lcls242, there indexing-bound)
Step 1, 48 streams/82k frames: 34-51 s -> 2.4 s with the headers
warm, 58.2 s -> 5.7 s evicted from cache
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Everything sparse that took a radial quantity from an image coordinate now takes it from the
reflection's (or spot's) position on the detector plane, through its panel (DetectorModules).
With an empty module table every changed expression reduces to the old one in the same
floating-point order, so flat detectors - every rotation set, EIGER, CCDs, assembled JUNGFRAU -
integrate bit for bit as before.
- Prediction (CPU, GPU, rotation CPU and GPU): bounds test against PlaneBounds instead of the
image rectangle.
- IntegrateOnImage: a prediction in a gap or off every panel is dropped (PlaneToPanel), no longer
passed through onto an unrelated stored pixel. The panel id stays local: the engine and the way
back find it from the image position (the panel boxes tile the image), and a prediction that
rounding to float put on another panel's edge is dropped so the two agree. Nothing is added to
Reflection.
- Bragg engines (CPU, CUDA, Metal) take a BraggPanel per reflection (BraggStencil.h): its plane
position, the panel's pixel axes on the plane and its box. The stencil's radial direction is
taken on the plane and expressed in the panel's pixel axes; r0, the radial background curve,
the Gaussian's elongation, the learned moments, tan(2theta) and Finalize's polarisation use the
plane position; r_max / n_rad come from PlaneBounds. The neighbour mask and the owner map are
clamped to the reflection's own panel; inside the walked box a pixel off the reflection's panel
is read as a gap (unreadable, statically masked - not an overload), which is what the assembled
image has there, so a disk cut by a panel edge is partial rather than silently "full".
- Risk R4: the ProfileEmpirical grid is learned and applied in the plane's axes (the panel axes
taken to the nearest quarter turn, BraggProfileCell), so turned panels pool one shape. Without
it the stacked layout read up to 0.3 % off the assembled one in the harness.
- SpotWidth / SpotFootprint: aperture bounds against the spot's panel box, beam distance,
resolution and the shape's 2theta Jacobian on the plane, moments read in the panel's axes.
- GetDetectorMaxQ_recipA, the far-corner radii (first-pass tilt report, metric-symmetry arm,
footprint table range) from PlaneBounds; pre-scan spot resolution on the plane;
LogHeaderBeamCenter: geometric centre of the panels' plane box, and the masked-pixel check on
the panel the beam lands on (none when it is in a gap).
Harness: PanelLayout_BraggIntegration is now a real [PanelLayout] check, over six settings
(default, bandwidth-elongated ring, radial background, overlap exclude, box sum, empirical
profile, all with polarisation), stacked vs assembled to 1e-5; PanelLayout_BraggIntegrationGPU
runs the same on the CUDA engine. Its spots avoid exact half pixels: the grid centres on the
rounded position, and a half rounds to the other neighbour on a turned panel.
The Metal engine (.metal/.mm) mirrors the CPU change exactly but is UNCOMPILED here (no macOS):
it needs a build and BraggIntegrationEngineMetal_BitIdenticalToCPU on a Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Spot finding works panel by panel, as the FPGA path does per JUNGFRAU module. PanelBoxes (common/) gives
one box per DetectorModule; an empty table gives one box, the whole image, so every clamp reduces to the
old image bounds in the same evaluation order (rotation sets, EIGER, CCDs, our JUNGFRAU unchanged).
- ImageSpotFinder::SetPanels: the CPU local-box pass (dense, count-threshold and the DetectAt/candidate
path of the adaptive finder) slides its windows inside each panel box; host CCL labels panel by panel;
pixels on no panel are folded into the resolution mask.
- GPU: analyze_pixel tiles each panel (warps numbered panel after panel), analyze_candidates and
union_neighbours clip to the pixel's panel, the device spot carries its panel; the GPU list is put in
panel order so it equals the host's.
- DiffractionSpot carries its panel; SpotAnalyze maps by ImageToPlane(panel, ...) (d_A on the plane).
- ImagePreprocessorBuffer::GatherTiles (CPU+GPU) clips each square to the panel of its centre.
- MXAnalysisWithoutFPGA and the pre-scan workers hand the module table to finders and buffers.
Raster algorithms: new common/AssembleToPlane.h (PlaneRasterOf, AssembleToPlane nearest-pixel gather with
gaps, GatherFromPlane back to stored pixels; empty table = copy). With panels, ShadowFinder::GetMask runs on
the projection assembled to the plane raster and gathers the mask back per stored pixel (CPU route);
FindBeamCenterFFT scores the assembled projection and reports plane coordinates; the detector diagnostic
JPEG shows the assembled plane. Without panels all three take their old path.
Harness: PanelLayout_SpotsNeverCrossPanels enabled (count threshold and SNR on the edge scene), new
PanelLayout_SpotFinderParity (CPU vs GPU, local box and adaptive, on the stacked layout),
PanelLayout_AssembleToPlane, PanelLayout_ShadowFinder, PanelLayout_BeamCenterFFT,
PanelLayout_DetectorDiagnostic (stacked vs assembled image cut to the raster).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
With a panel table, the viewer and the broker preview gather the stored image
onto the detector plane (AssembleToPlane) and draw on that: spots,
predictions, rings, beam centre and ROIs are plane coordinates, so they now
land on the right pixels for a stored (stacked or moved-module) layout, where
they were drawn as image pixels before. The viewer scene is the plane; its
image item sits at the raster's origin (JFJochImage::origin_, zero for a flat
detector). Mouse readout, pixel labels and the magnifier read the shown
raster; top pixels and saturation crosses are mapped from the stored pixel to
the plane; an ROI turned into a user mask is tested per stored pixel at its
plane position.
A flat detector (empty table) takes the old path verbatim: the shown pixels
alias the image, origin (0, 0) - a JF/EIGER file renders identically.
common/AssembleToPlane.h is a TEMPORARY header-only version so this stage
builds on its own; stage 3 owns the file and its version replaces this one at
merge (callers use AssembleToPlane(modules, image, width, height,
PlaneBounds, gap) -> PlaneRaster{x0, y0, width, height, pixels}).
Test: PreviewImage_PanelLayout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
SpotToSave gains int16_t panel (-1 = no panel table). It sits in the padding
after the two bools, so sizeof(SpotToSave) stays 80. It is serialised only
where it is set: CBOR spot key "panel", HDF5 /entry/MX/peakPanel (data file,
plus a virtual dataset in a VDS master when the start message has panels),
read back by HDF5MetadataSource. For a flat detector none of these is written,
so files and streams for existing data are byte-for-byte unchanged.
The start message carries the panel table (detector_modules, CBOR key absent
when empty), filled from the experiment; JFJochHttpReader takes it, so a
stream viewer knows the panels too.
AssignModules takes the spot's panel id where it has one - the stored pixel box
it was found in - and falls back to the plane position otherwise (every spot
until per-panel spot finding sets the id).
Reflection deliberately gets no panel id (owner decision): it is needed only
between prediction and integration, and the struct feeds GPU layout and
memory budgets.
Tests: CBORSerialize_SpotPanelId, CBORSerialize_StartDetectorModules,
JFJochReader_PanelId (VDS/legacy/integrated, with and without panels).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
