One Ceres fit keeps only ~5-6 of 32 cores busy whatever it is given, and on a loaded machine (load
28-36, JF16M int16, 4000 strong frames) the five side-by-side fits measured 10.9 s with every thread
each, 13.8 s with half, 14.6 s with a fifth, against 18.9 s one after the other (bb7c9c). The
thread split is dropped: each fit gets gr_settings' threads, as before the fits ran side by side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The geometry refinement's sample (up to 10000 frames: spot finding + indexing on the card, the same
per-frame work as the image pass) and the stills geometry trial's 96-frame spot finding ran -N workers
(32 here), each with its own GPU engines. The image pass caps the workers that drive a card at 16 per
card, fewer where their engines would not fit half the card (GPUWorkersPerCard), because past that a
card stops taking queue work and more workers make the loop slower. The cap is now one function,
ImageWorkers(), used by all three loops. On a 16.9 Mpx JUNGFRAU 16M the 32 sample workers held ~13 GB
of a 16 GB card's engines (24 B/px each) where the cap allows 19.
Which frames the sample examines does not depend on the worker count (fixed stripes), and each trial
frame is found on its own: outputs identical (JF16M int16 15 runs: same 5174 frames sampled, same
module placement, geometry and merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Four changes to the per-run fixed cost of a stills run (Step 2 geometry refinement, Step 3 pre-scan),
none of which changes what is computed:
- Module placement: the four held-out fits and the placement on every strong frame are independent, and
one Ceres fit keeps only ~4.6-6.5 of 32 cores busy (perf, JF16M, 4000 frames / 150888 spots). They
now run side by side on a fifth of the threads each; the placement fit is computed speculatively and
used only where the held-out residuals call for it, as before. Baseline (quiet): held-out 12.3 s +
placement 5.1 s on JF16M, 16.4 + 6.5 s on JF4M, 19.0 + 7.6 s on MPCCD.
- Module scan (+-25 px grid per module ahead of the placement) was one thread: every grid shift is now
scored on the workers and the grid then walked in its old order, so the first best still wins
(3.3 s on JF16M, ~5 s on JF4M/MPCCD before). New test ScanModuleShifts_FindsShiftedModule checks the
shift is found and that 1 and 4 threads give the same answer.
- Stills geometry trial: each hypothesis (centre, distance, frame sample) is indexed once. The distance
grid ran twice on the same centre whenever the file's centre was kept (40 of ~275 hypotheses,
~15% of the trial: 6.2 s on JF16M, 6.7 s on the flat JF8M), and the final here/there check repeated
the first trial. Repeats always gave the same count (checked on three logs).
- The saturation histogram (40 frames, 0.1-2.2 s) reads only frames and the mask: it is taken beside
the geometry refinement and the limit decided where it was, after it.
Plus TIMING marks for the sub-steps (geometry refinement sample, module scan, module placement fits).
Outputs identical to the baseline binary (bb7c9c) where checked: JF16M int16 15 runs (geometry, all
32 module shifts, held-out residuals, cell), JF4M int16, MPCCD 236 (indexing rate, CC1/2, R_split,
R-free with --model identical), PAL 90 and LCLS 222 (identical reports).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The SwissFEL .geom files name their pixel mask with an absolute facility path, so the sets read a
copy beside the original (jf-16M_mask.geom, mbco_march_mask.geom) whose mask_file is relative to the
copy's directory; only that line differs. EXTERNAL_TEST_DATA_STILLS.md says how to make them.
On a 2000-frame subset of the int16 MbCO set, before (geometry without masks + --mask-good) and after
(the geometry's mask and bad regions): indexed 98.2% both, R_FREE (--model) 0.244 -> 0.234.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
- The NXmx master (and _process.h5) writes module_0..N with the panel table when the start message has one,
besides the whole-image module, so a stored-layout file reads back with its panels
(JFJochReader_PanelId checks the round trip).
- RotationScaleMerge's modulation grid spans PlaneBounds instead of the image rectangle; with no panel
table x0 = 0 and the extent is the image, so the cells are unchanged.
- The beam-stop / pre-scan memo key includes each panel's box and placement; nothing is added with no
table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A detector of two or more panels is no longer assembled. The image is the stored frame, and each panel of
the geometry (CrystFEL .geom or the file's own panel table) is a DetectorModule over its box of the frame,
at its true placement on the detector plane: fractional origin, the panel's own axes, no snapping to an
image axis or a whole pixel. Pixels on no panel, edge, bad-region and file-masked pixels are masked in the
stored layout. The block scatter (a transpose for the 90-degree-turned JUNGFRAU and MPCCD panels) is gone
for these inputs. A single panel keeps the old block path, so single-panel CXI files are unchanged.
Signed-integer frames in their own units (gain 1, no per-frame mask) go up with no copy: a bitshuffle/LZ4
chunk still compressed, for the GPU preprocessor's BSLZ4 decoder (int16's smallest value is its error
value there, so the type-min rule holds), a deflated one inflated outside the HDF5 lock straight into the
caller's buffer. Everything else (float, unsigned, a gain, a per-frame mask) is converted once in stored
order. A float NaN/inf now reads as no measurement (INT32_MIN) instead of 0, and values past int32 are
clamped before the cast.
Also: SpotToSave.panel is filled from DiffractionSpot::Panel(), so AssignModules takes the panel the spot
was found on. PreviewImage_PanelLayout expects stage 3's PlaneRasterOf raster (merge fallout).
Main pass, 2000 frames, vs the CCD-reader build: JF4M int16 bslz4 465 -> 1860 frames/s, JF16M 105 -> 1111,
MPCCD 212 -> 252; flat Jungfraujoch data unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The .geom parser now takes the masking information the format defines:
- mask arrays: mask/mask_file/mask_good/mask_bad (mask 0) and maskN_data/maskN_file/maskN_goodbits/
maskN_badbits for N = 0..7; global keys hold for the panels after them, a panel may override. A mask
that is the same on every panel is one whole-frame mask, otherwise one per panel, applied on that
panel's pixels only.
- a relative mask_file is looked for in the working directory (as CrystFEL does), then beside the .geom.
- bad regions (bad*): a lab-plane box (min_x..max_y) or a fs/ss range on a panel (or, old syntax, on
the whole frame); mask_edge_pixels and no_index.
JFJochXFELReader already had the static pixel mask and the per-frame mask path for these; the masks
go through them unchanged. A mask whose path holds % (one per shot group) is applied frame by frame
like a 3D one. Where the geometry names mask arrays, the data file's own Cheetah masks (CXI
mask_shared and per-shot mask with Cheetah's generic bad bits, /metadata/pixelmask) are not added:
the geometry says which bits matter (e.g. an AGIPD mask_bad that includes bit 32). --mask-good /
--mask-bad still add their masks to these.
Tests: [CrystFELGeometry] (new keys, per-panel override, relative mask file), JFJochXFELReader_*
(new GeometryMasks case on a synthetic CXI + .geom), [JFJochMultiReader], JFJochXTCReader*.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Measured values are clipped to [-32767, 32767] so that only a pixel without a measurement holds -32768.
The existing copies are unaffected: none of the converted SwissFEL runs had a non-finite pixel or a
clipped value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The AGIPD sets are run on the files as deposited (Cheetah CXI with the per-ASIC CrystFEL geometry),
and their processing speed is reported as it is on those files; only the SwissFEL float32 sets keep
an int16 photon copy. The reader rule that a signed integer frame's smallest value is no measurement
stays: it is general.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Each set's row now says how many images we use and, where that is a subset, what the entry or its
deposition holds (SACLA 237 ~1.8% of dark_ref, LCLS 222 one of four runs, LCLS 242 runs 256-267, both
AGIPD sets and A2A against their deposited frame counts). The section listing what is not used is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
DecodeInto no longer turns every stored pixel into a float and back:
- the gain is one value (adu_per_photon_), not a float per stored pixel
- it always was one value; only --gain or a measured gain set it;
- a 16-bit signed frame (Cheetah CXI, SACLA) without a gain is handed
up as Int16, in its own type; other integer frames are widened to
int32 without a float; only a float frame or a gain other than one
goes through lround(v / gain), exactly as before;
- placement is per panel (start, size, destination steps) instead of a
64-bit destination per stored pixel; a panel square with the image
is copied row by row (PAL: 1440x1440 into 1444x1444 - the two-pixel
margin is Place()'s, kept so the geometry does not move);
- the decode buffers (chunk, inflated, unshuffled) are reused per
thread, the bitshuffle decode is not copied again;
- a chunk whose HDF5 filter_mask is not zero (a filter skipped) is
read by HDF5 instead of being decoded as if it were not;
- the never-set value flags (flag_lessthan/morethan/equal) are gone
from the layout and from the per-pixel loop.
MeasureGain in the CLI now reads Int16 frames as well, so a SACLA run
still has its gain measured.
Tests: JFJochXFELReader_SACLA/_CXI updated for the mode, new
JFJochXFELReader_CXI_Int16 (deflate without shuffle, a chunk with its
filter skipped) and JFJochXTCReader_StoredPixels (synthetic XTC,
stored hand-up and the offset path).
Measured on two PAL-XFEL Rayonix sets (int16, deflate; RTX 5080,
5950X, page cache warm; reports and .hkl identical to the baseline):
main pass, 2000 frames: 296 -> 799 frames/s (pal86),
302 -> 755 frames/s (pal90)
wall, 2000 frames: 26.4 -> 13.6 s, 32.1 -> 19.4 s
(the geometry-refinement steps, which read frames too, halve)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The main loop's read (ReadRawImage) now hands a camera frame up as the
file stores it - 16-bit pixels, UINT16_WIDENED, as the marCCD reader
does - wherever there is no pixel offset to take off and no gain to
divide by; the GPU preprocessor widens it on the device. The host no
longer widens 3.7 Mpx to int32 per frame (the loop that was ~79% of
the main pass's user cycles). xtc::ReadFrame becomes xtc::ReadPixels,
the stored pixels only; the int32 decode (LoadImage, or an offset or
gain) widens them in one plain loop.
The streams are indexed in parallel (one worker per stream, up to the
hardware threads), merged in the order given: each index is a walk of
small reads, waiting on the disk one at a time.
Measured on a Rayonix MX340 run, 1920x1920 (RTX 5080, 5950X, page
cache warm, other agents' builds running; output identical but for the
known run-to-run flip of one observation on one set, which the
baseline shows between its own runs):
main pass, 2000 frames: 479 -> 789 frames/s (lcls222),
413 -> 477, 388 -> 452 (the two forms of
lcls242, there indexing-bound)
Step 1, 48 streams/82k frames: 34-51 s -> 2.4 s with the headers
warm, 58.2 s -> 5.7 s evicted from cache
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Everything sparse that took a radial quantity from an image coordinate now takes it from the
reflection's (or spot's) position on the detector plane, through its panel (DetectorModules).
With an empty module table every changed expression reduces to the old one in the same
floating-point order, so flat detectors - every rotation set, EIGER, CCDs, assembled JUNGFRAU -
integrate bit for bit as before.
- Prediction (CPU, GPU, rotation CPU and GPU): bounds test against PlaneBounds instead of the
image rectangle.
- IntegrateOnImage: a prediction in a gap or off every panel is dropped (PlaneToPanel), no longer
passed through onto an unrelated stored pixel. The panel id stays local: the engine and the way
back find it from the image position (the panel boxes tile the image), and a prediction that
rounding to float put on another panel's edge is dropped so the two agree. Nothing is added to
Reflection.
- Bragg engines (CPU, CUDA, Metal) take a BraggPanel per reflection (BraggStencil.h): its plane
position, the panel's pixel axes on the plane and its box. The stencil's radial direction is
taken on the plane and expressed in the panel's pixel axes; r0, the radial background curve,
the Gaussian's elongation, the learned moments, tan(2theta) and Finalize's polarisation use the
plane position; r_max / n_rad come from PlaneBounds. The neighbour mask and the owner map are
clamped to the reflection's own panel; inside the walked box a pixel off the reflection's panel
is read as a gap (unreadable, statically masked - not an overload), which is what the assembled
image has there, so a disk cut by a panel edge is partial rather than silently "full".
- Risk R4: the ProfileEmpirical grid is learned and applied in the plane's axes (the panel axes
taken to the nearest quarter turn, BraggProfileCell), so turned panels pool one shape. Without
it the stacked layout read up to 0.3 % off the assembled one in the harness.
- SpotWidth / SpotFootprint: aperture bounds against the spot's panel box, beam distance,
resolution and the shape's 2theta Jacobian on the plane, moments read in the panel's axes.
- GetDetectorMaxQ_recipA, the far-corner radii (first-pass tilt report, metric-symmetry arm,
footprint table range) from PlaneBounds; pre-scan spot resolution on the plane;
LogHeaderBeamCenter: geometric centre of the panels' plane box, and the masked-pixel check on
the panel the beam lands on (none when it is in a gap).
Harness: PanelLayout_BraggIntegration is now a real [PanelLayout] check, over six settings
(default, bandwidth-elongated ring, radial background, overlap exclude, box sum, empirical
profile, all with polarisation), stacked vs assembled to 1e-5; PanelLayout_BraggIntegrationGPU
runs the same on the CUDA engine. Its spots avoid exact half pixels: the grid centres on the
rounded position, and a half rounds to the other neighbour on a turned panel.
The Metal engine (.metal/.mm) mirrors the CPU change exactly but is UNCOMPILED here (no macOS):
it needs a build and BraggIntegrationEngineMetal_BitIdenticalToCPU on a Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Spot finding works panel by panel, as the FPGA path does per JUNGFRAU module. PanelBoxes (common/) gives
one box per DetectorModule; an empty table gives one box, the whole image, so every clamp reduces to the
old image bounds in the same evaluation order (rotation sets, EIGER, CCDs, our JUNGFRAU unchanged).
- ImageSpotFinder::SetPanels: the CPU local-box pass (dense, count-threshold and the DetectAt/candidate
path of the adaptive finder) slides its windows inside each panel box; host CCL labels panel by panel;
pixels on no panel are folded into the resolution mask.
- GPU: analyze_pixel tiles each panel (warps numbered panel after panel), analyze_candidates and
union_neighbours clip to the pixel's panel, the device spot carries its panel; the GPU list is put in
panel order so it equals the host's.
- DiffractionSpot carries its panel; SpotAnalyze maps by ImageToPlane(panel, ...) (d_A on the plane).
- ImagePreprocessorBuffer::GatherTiles (CPU+GPU) clips each square to the panel of its centre.
- MXAnalysisWithoutFPGA and the pre-scan workers hand the module table to finders and buffers.
Raster algorithms: new common/AssembleToPlane.h (PlaneRasterOf, AssembleToPlane nearest-pixel gather with
gaps, GatherFromPlane back to stored pixels; empty table = copy). With panels, ShadowFinder::GetMask runs on
the projection assembled to the plane raster and gathers the mask back per stored pixel (CPU route);
FindBeamCenterFFT scores the assembled projection and reports plane coordinates; the detector diagnostic
JPEG shows the assembled plane. Without panels all three take their old path.
Harness: PanelLayout_SpotsNeverCrossPanels enabled (count threshold and SNR on the edge scene), new
PanelLayout_SpotFinderParity (CPU vs GPU, local box and adaptive, on the stacked layout),
PanelLayout_AssembleToPlane, PanelLayout_ShadowFinder, PanelLayout_BeamCenterFFT,
PanelLayout_DetectorDiagnostic (stacked vs assembled image cut to the raster).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
With a panel table, the viewer and the broker preview gather the stored image
onto the detector plane (AssembleToPlane) and draw on that: spots,
predictions, rings, beam centre and ROIs are plane coordinates, so they now
land on the right pixels for a stored (stacked or moved-module) layout, where
they were drawn as image pixels before. The viewer scene is the plane; its
image item sits at the raster's origin (JFJochImage::origin_, zero for a flat
detector). Mouse readout, pixel labels and the magnifier read the shown
raster; top pixels and saturation crosses are mapped from the stored pixel to
the plane; an ROI turned into a user mask is tested per stored pixel at its
plane position.
A flat detector (empty table) takes the old path verbatim: the shown pixels
alias the image, origin (0, 0) - a JF/EIGER file renders identically.
common/AssembleToPlane.h is a TEMPORARY header-only version so this stage
builds on its own; stage 3 owns the file and its version replaces this one at
merge (callers use AssembleToPlane(modules, image, width, height,
PlaneBounds, gap) -> PlaneRaster{x0, y0, width, height, pixels}).
Test: PreviewImage_PanelLayout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
SpotToSave gains int16_t panel (-1 = no panel table). It sits in the padding
after the two bools, so sizeof(SpotToSave) stays 80. It is serialised only
where it is set: CBOR spot key "panel", HDF5 /entry/MX/peakPanel (data file,
plus a virtual dataset in a VDS master when the start message has panels),
read back by HDF5MetadataSource. For a flat detector none of these is written,
so files and streams for existing data are byte-for-byte unchanged.
The start message carries the panel table (detector_modules, CBOR key absent
when empty), filled from the experiment; JFJochHttpReader takes it, so a
stream viewer knows the panels too.
AssignModules takes the spot's panel id where it has one - the stored pixel box
it was found in - and falls back to the plane position otherwise (every spot
until per-panel spot finding sets the id).
Reflection deliberately gets no panel id (owner decision): it is needed only
between prediction and integration, and the struct feeds GPU layout and
memory budgets.
Tests: CBORSerialize_SpotPanelId, CBORSerialize_StartDetectorModules,
JFJochReader_PanelId (VDS/legacy/integrated, with and without panels).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Every dense map that needs geometry now evaluates each image pixel at its
place on the detector plane (DetectorModules), so it is right for a stacked
panel layout and picks up refined module placements:
- AzimuthalIntegrationMapping: bins, resolution, solid-angle/polarisation
corrections built per pixel through the table; a pixel on no panel is left
as a masked one. The geometry cache key now includes the panel table, so a
mapping built after the stills module placement is rebuilt rather than
served from the cache. ResolutionMaskBits follows (it is a lookup).
- ROIMap / ExportROIMap: azimuthal ROIs on the plane (box and circle ROIs
stay in image pixels).
- ROIIntegration CPU+GPU: the weighted sums are taken per ROI and panel in
image pixels and placed on the plane at the end (exact; the affine commutes
with a weighted mean).
- HotPixelFinder: ring/sector keys and the zeta of the persistence rule on
the plane.
- Background beam-centre walk CPU+GPU: pixels binned at their plane position;
host and device share BackgroundPlanePoint so they stay bit-identical.
New common/PanelRows.h: ForEachPixelInRows, ForEachPixel cut to a band of
rows in index order, for the row-parallel builders.
An empty table takes the old arithmetic in the old order: rotation sets and
every single-panel input are unchanged. [PanelLayout] AzintMapping and
ResolutionMask enabled (+ azimuthal ROI map and ROI centroid on the plane);
new BeamCenterFromBackground_BinsOnThePlane and
ROIIntegrationGPU_MatchesCPU_Panels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A synthetic 2 x 2 panel detector, its panels turned 0/180/90/-90 degrees, recorded twice: assembled
with the gaps masked, and stacked one panel under the other with the DetectorModules that place them -
the same value in every physical pixel. What is already layout-correct runs as [PanelLayout]: the
panel table itself, spot centroids on the plane through SpotAnalyze, and predictions going plane ->
panel -> Bragg engine -> plane.
The equivalences the next stages have to deliver are hidden cases tagged [PanelLayoutTodo], failing
today: the azimuthal-integration map and the resolution mask per stored pixel (stage 1), Bragg
integration measuring the same intensity (stage 2), and spot finding that never joins two stacked
panels (stage 3). A stage that makes its case pass retags it [PanelLayout].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Stage 0 of analysing panel detectors in their stored layout. The table of modules already is the panel
table; what the consumers converted in the next stages need from it:
- ImageToPlane / PlaneToImage on a known panel, no search; panel -1 (empty table) is the identity.
- PlaneToPanel: the panel holding a plane point and its image position, nullopt in a gap (never passed
through, unlike the existing PlaneToImage).
- PlaneBounds: the box of all panel footprints on the plane; with an empty table exactly the image
rectangle today's prediction bounds test uses.
- ForEachPixel: f(panel, index, px, py) panel by panel, row by row, gaps not visited; with an empty
table the plain row-major loop with plane == image.
The image preprocessors (CPU, GPU) and the Bragg engines (CPU, GPU; Metal derives from CPU) now refuse
a pixel mask whose size is not the image's, instead of reading a mask of another size out of range.
Every reader sizes the experiment's detector, and so the mask, by the image it reads, so nothing that
runs today is refused. Nothing calls the new functions yet: no output changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
tools/battery/still/agipd_to_int16.py converts an AGIPD 1M Cheetah CXI file into the only form rugnux
supports for AGIPD: each module's eight 64-row ASICs placed 66 rows apart (the 2-pixel ASIC gaps
inserted), one 526 x 128 panel per module; the per-frame mask and mask_shared (bad bits = the
geometry's mask_bad) and the geometry's bad regions folded into the data as -32768; values
round(value / ADU per photon) as int16 (clipped to +-32767; no pixel clipped in either set);
bitshuffle/LZ4, one chunk per frame at /entry_1/data_1/data. It writes the matching CrystFEL geometry
(16 panels, each the rigid fit of its ASICs, module coffset kept) and refuses a module whose ASICs are
not on one straight line at a whole-pixel pitch within 0.5 px (worst here: 0.16 px for CXIDB 83,
0.002 px for 152).
Reader: a stored value equal to the smallest value of its signed integer type (INT16_MIN, INT32_MIN)
is no measurement. The reader decodes to float and divides by the gain, so -32768 used to come out as a
count; the CrystFEL parser takes only module geometry, so a flag_lessthan in the .geom could not say it.
Still battery: euxfel83 / euxfel152 read conv/*_int16.h5 with conv/agipd_int16.geom and --gain 1.
Full sets, -N 16, --refine-geometry=off, --model (other jobs on the machine, load 7-23):
indexed d_min CC1/2 CC_HALF_SPLIT R_split R_FREE main pass wall
CXIDB 83 original 90.9% 1.84 0.641 0.617 0.482 0.280 89.3 s (31/s) 105 s
converted 94.1% 1.84 0.686 0.644 0.470 0.277 6.5 s (429/s) 20 s
CXIDB 152 original 17.5% 5.37 0.521 0.479 0.590 0.308 131.7 s (29/s) 153 s
converted 62.8% 2.92 0.876 0.891 0.354 0.284 13.2 s (287/s) 27 s
Files 4.7 -> 1.3 GB and 5.7 -> 1.1 GB. The 152 gain is the mask: the reader took only Cheetah's
generic bad bits, while the geometry's mask_bad (0xffff) also marks three broken modules (bit 32) and
its bad region a fourth; the converter run with the generic bits instead gives 17.5% / 3.41 A again.
With geometry refinement on (16 rigid modules): 83 94.2%, 1.75 A, CC1/2 0.665, R_FREE 0.284, 49 s;
152 65.1%, 2.94 A, CC1/2 0.843, R_FREE 0.278, 27 s - mixed, the manifest keeps it off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
- still_battery.py run --cold: posix_fadvise(DONTNEED) on a set's input files just before its run, so
every timed row is a cold read without root (the page-cache column confirms it).
- Alvra sets use the depositors' pixel masks (--mask-good); the MbCO int16 glob now finds the copies,
which keep the tarball's directory tree; LCLS 242 states its wavelength (the XTC does not).
- EXTERNAL_TEST_DATA_STILLS.md names the mask files and the one MbCO dark run that holds no frames.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
GCC 12 (the Ubuntu 22.04 CI job) does not pull size_t in through <cstdint>/<vector>, so the header failed
to compile there. Checked: all 601 project sources pass -fsyntax-only with GCC 12.2 (gcc-toolset-12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The direct chunk read opened one HDF5ImageSource::RawFile per file and never released it, so the
MAX_OPEN_FILES bound on the HDF5 handles did not hold for that path: a dataset stored one frame per
file would keep a descriptor per file open. They now follow the same least-recently-used bound; a
reading thread still holding a released handle keeps it through its shared_ptr.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills pipeline figure keeps its layout and adds what rc178 does: the input box lists CXI and
XTC, the geometry box is the beam centre decided on the frames and, with a cell, the distance and the
module placement; the pre-scan reads the saturation level; the P1 merge keeps one crystal form.
RUGNUX_OVERVIEW's stills paragraph says the same in prose. Methods: §1.4 describes the stills centre
and distance hypotheses, a new §1.7 the saturation level read off the frames, and a new §7.7 a given
cell as a starting point, the crystal-form filter and the module placement with its gauge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
- --mask-good / --mask-bad take [<file>:]<dataset>[@<bits>]: without a file the dataset is read from
each data file (a per-frame mask stored beside the frames, as a Cheetah CXI holds it), and <bits>
says which bits mark a pixel bad (or good) - what a CrystFEL geometry's mask/mask_good/mask_bad
said, now that the geometry file gives module geometry only.
- A gain the background shows at or below one detector unit per photon is not applied; such frames
stay in detector units, as a CCD's do. Scaling a low-gain Rayonix's frames up by it cost one PAL
set most of its indexing (69% -> 33%).
- The HDF5 stills readers ask H5Fis_accessible with the error stack silenced before opening a file,
so an XTC stream offered to them no longer prints HDF5's diagnostics.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The viewer reads what rugnux reads: HDF5 files with a /metadata panel table, Cheetah CXI single-panel
stacks and LCLS-I XTC streams, by their content as the raw sweeps are. The open dialog takes several
files; they are one dataset (as on rugnux's command line), and a CrystFEL geometry file among them is
their module geometry. The images are shown in detector units, at the geometry the files state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto
rc178 with the owner's decisions for this release:
- Inputs are files, never a directory; several files of one format are one dataset (the multi-input
CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles,
ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and
xtc::CollectStreams are gone, CanRead* accept a regular file only.
- One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds:
per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera
length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is
refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit
decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work).
- Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's
place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through
HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib,
bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through
HDF5's filter under the global lock, one frame at a time.
- LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and
binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event
dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all.
- CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path)
and optionally the camera length; everything else is ignored with a note in the log.
- --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored
layout (what a .geom's mask_file used to give).
- Gain measured on the background for panel-table and CXI input that states none (--gain overrides).
- Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with
one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions
are mapped through the modules, the image is not re-assembled.
- Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge
on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only
where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table,
.geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx
NXdetector_module groups), read back with --module-geometry.
- A panel-table file's camera length is nominal: with a reference cell it is refined from the cell.
- Broker and writer output unchanged: masters keep the single NXdetector_module group, the start
message carries no module list.
Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation],
SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Every input is a file named on the command line, read in the order given (a shell glob gives them
sorted by name); a directory is refused with the glob form to use instead.
- Several Jungfraujoch/NXmx masters of stills are read as one run (JFJochMultiReader): their images
one after another, the pixel masks combined, the per-image efficiency placed at each input's
images. They must share the detector, wavelength (0.1%), distance (0.1 mm) and beam centre (1 px),
since stills geometry refinement refines one geometry for all frames; rotation sweeps are refused.
The log gives the indexing rate of each input beside the run's (ProcessResult::image_indexed).
- SACLA HDF5 / CXI (JFJochCrystFELReader::ReadFiles, ReadCXI, Open) and XTC
(JFJochXTCReader::ReadFiles) take the list of files instead of scanning a directory;
CanReadCXI and xtc::CanRead accept only a file; ListFiles and xtc::CollectStreams are gone.
- Several inputs must be one format, --mode mx, and stored files (no broker, no --simulate-live);
one frame of a CBF/marCCD/SMV sweep still names the whole sweep.
Validated: two Jungfraujoch lysozyme stills runs together (49,920 frames, known cell) 57.1% indexed (59.3% /
54.9% per run), 1.22 A, CC1/2 0.986, R_split 8.8%, REFMAC R/R-free 0.253/0.269 against 0.257/0.273
for the earlier hand-built combined master. Rotation: 5reo and lyso_x10sa_90deg_1 unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
still_battery.py runs the public serial datasets of still_manifest.json with their deposition's
cell and space group, refines the deposited model against each merge with dimple and the
deposition's own free set, and reports per set: facility, beamline, detector, PDB, images,
indexing rate, R_split, our R-free beside the deposition's published R-free, the wall time of the
rugnux command, ms per image, the input's I/O floor (bytes over 5 GB/s) and how much of the input
was already in the page cache. It shares battery.py's site config (a "still" section), verbs
(run, compare, report, list, keep) and run-directory layout; run directories carry "still-" in
their label.
The datasets (CXIDB 243, 237, 236, 86, 90, 222, 242, 83, 152, 103, 213; dark and ground-state
entries only) are listed with their verified DataCite DOIs in docs/EXTERNAL_TEST_DATA_STILLS.md,
linked from EXTERNAL_TEST_DATA.md; CXIDB is credited in ACKNOWLEDGEMENT.md. The SwissFEL sets also
run as int16 bitshuffle/LZ4 copies, made with still/float32_to_int16.py.
The stray scripts in the repository root move under tools/battery: rugnux_stills_ab.py and its
dataset template to still/, rugnux_anomalous.py beside battery.py; docs/TESTS.md follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
VERSION and the strings update_version.sh rewrites (API spec, python and TypeScript clients, C++ model,
Redoc page, docs/conf.py, FPGA HDL and PCIe driver). The regenerated clients and Redoc page also pick up
rc178's API changes made since the last regeneration (rugnux live settings and model without a bearer
token, the live status fields).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Three defects that stopped known-cell stills runs:
- The consensus cell compared every frame's cell to -C axis by axis. A run whose frames index in another
setting of the same lattice (axes in another order, another monoclinic choice) had no frame accepted, no
consensus cell and no output ("No crystal lattice was determined"), although it merged fine. A frame
whose lattice contains the given cell now counts on the given cell's axes (CellOnReference, the same
FindSublatticeBases the merge-stage relabel uses).
- The per-image ambiguity resolver relabelled a frame's reflections but left its lattice on the old axes.
Everything after it predicts from index and lattice: the sublattice relabel put such a frame through a
second transformation, and on a monoclinic cell with a near-symmetry (a+c nearly as long as a) the
merge came out at indices finer than anything measured, shells over 100% complete. The lattice now
follows the indices (ReindexLattice, B' = R^-1 B).
- With -C/-S the stills path replaced the known-cell indexer's lattice, already on the given axes, by
LatticeSearch's conventional choice, which on that near-symmetric cell is the other description on half
the frames. The indexer's lattice is now kept when it is close to the given cell.
Measured (cell + space group given, dimple against the deposited model and free set):
- monoclinic MbCO-type still set: was over-complete/broken with --model; now 2.06 A, R/Rfree 0.233/0.314
at 2.11 A (P1 merge before: 0.275/0.306).
- C-centred orthorhombic set: no MTZ before; now 3.27 A, R/Rfree 0.298/0.403 at 3.68 A (de novo 0.336/0.448).
- monoclinic two-form set, one form: cell P 1 21 1 with beta (was alpha); Rfree 0.310 -> 0.260, 3.05 -> 2.72 A.
- No change on seven other stills sets, nor on two rotation battery sets (5reo, lyso_x10sa_90deg_1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The peak nearest the given cell could be a single stray frame when the given cell is a few percent off
the crystal (a cryo cell given for room-temperature lysozyme stills): the run then merged one frame.
Every population is now found from its own peak and valleys, and of those holding at least a tenth of
the frames the one centred nearest the given cell is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A given cell keeps only the frames whose refined cell belongs to its population (GivenCellPopulation):
the peak of the refined-volume histogram nearest the given cell, out to the valley that separates it
from another population, with the bin set by the data's own spread. A sample of two crystal forms
indexed against one form's cell otherwise merges both; a run with one population keeps every frame.
- The distance grid is run from every centre hypothesis, not only the one decided at the file's
distance, and a local grid of centres follows at the distance decided. A wrong header distance and a
centre several pixels off (both background estimators misread it on a Rayonix set) otherwise leave
each other flat; now the cell alone finds both.
- A distance given with --detector-distance is not searched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
