On macOS size_t (unsigned long) plus int64_t (long long) is unsigned long long, so std::max/std::min had
two argument types; the template argument is now given. The preprocessor's (masked | error) ? ... is what
the code already did - the parentheses only silence clang's -Wbitwise-conditional-parentheses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The unshuffle of a deflated + shuffled chunk (a panel-table HDF5 run: 8.4 MB per frame) moved one byte at
a time with the pixel size read at run time, and wrote the whole frame before it was converted. It now
assembles each 2- or 4-byte pixel from its byte planes in a loop the compiler vectorises, a 128 kB piece
at a time, and each piece is converted into the image while it is in the cache, as for an unshuffled
chunk.
Bit-identical: the FNV pixel hashes of 2000 frames on the five reader configurations are unchanged.
Reader alone on the panel-table run, 16 threads, 2000 frames: the unshuffle went from 58% of the
reader's cycles to 13%; wall 7.7-8.6 -> 5.1-7.0 s on a loaded machine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The reader inflated a whole deflated chunk (4 MB for a 1 Mpx int32 frame, 2 MB for its uint16 per-frame
mask) into a buffer and then converted it in a second pass; with 16 reading threads those buffers never
stay in the cache, so the conversion ran at memory speed. A deflated, unshuffled chunk is now inflated in
128 kB pieces (InflateInPieces, the zlib stream API - the stream's checksum is checked as before), and
each piece is converted into the image, or marked in the mask, before the next is inflated. Shuffled,
bitshuffle/LZ4, HDF5-read and single-panel (placed) frames are converted as a whole, as before.
Bit-identical: the FNV pixel hashes of 2000 frames on the five reader configurations of the previous
commit are unchanged, and a new test reads 300 x 300 frames (several pieces, a final partial one) with a
per-frame mask and a gain pixel by pixel. Reader alone, 16 threads, 3000 frames: 2.43 -> 1.73 s (the
previous commit's state -> this). rugnux on the 14,445-frame CXI run, against rc178 (A/B, two runs each):
geometry refinement sample 11.6 -> 5.1 s, image pass 27.7 -> 12.9 s, whole run 67.7 -> 46.2 s; the
geometry refinement, module placement and tilt log lines are identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
On a deflated Cheetah CXI with a per-frame mask (AGIPD 1M), half of the reader's time after the inflate
went to turning the stored frame into the image: the mask plane widened into a zeroed uint32 copy before
being tested, MarkBad testing it with a branch per pixel through a reference the store could alias, and
the photon conversion calling lroundf and branching on the mask, the finiteness and the clamp. Now the
mask is tested as stored (uint16 or uint32), and both loops are written so that the compiler vectorises
them: std::lround spelled out exactly (truncate, the exact fraction, a half or more away from zero) and
the selects done as arithmetic (Pick). Plain C++, no intrinsics.
Bit-identical: FNV hash of every image handed out (2000 frames) plus the pixel mask, before and after,
on a CXI with a CrystFEL geometry (with and without a gain), a panel-table HDF5 (with and without a
gain) and a bitshuffle/LZ4 int16 set: all identical. Reader alone, one thread: 14.1 -> 7.0 ms per
frame. rugnux on the 14,445-frame CXI run (16 workers, A/B back to back): geometry refinement sample
11.4 -> 8.9 s, image pass 27.6 -> 21.1 s, whole run 67.4 -> 58.2 s; geometry and refinement log lines
identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A pedestal-subtracted JUNGFRAU background is mostly empty pixels; the variance per unit mean leaves
out the zeros and read 6.7 keV per photon on a 12 keV run in keV, 2.4 on a 4.6 keV one, so such
files needed --gain. MeasureGain now also histograms the sampled pixel values and takes the gain from
the single-photon peak (background_gain::PhotonPeak): the highest bin past the zero peak's descent,
standing 5 sigma above the valley before it, narrow (below half height a quarter of its value away),
and repeated by a two-photon peak near twice its value. Where there is no such pair the variance per
unit mean decides as before. Float frames are read for the measurement at a sixteenth of their unit,
so a photon of a few keV is resolved rather than rounded to whole keV.
Measured: float32 keV JUNGFRAU files 11.96 (photon energy 12.04 keV) and 4.53 (4.57 keV); the int16
photon copies show no peak past the first bin and keep the 0.55 variance answer, i.e. stay in
photons; the SACLA MPCCD and PAL Rayonix files show humps, not photon peaks, and measure as before.
A gain the file states or --gain gives still wins. Rotation is untouched (the pre-scan's CCD gain
uses only the variance estimator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The Cheetah CXI per-frame mask (N x H x W uint16, one gzip chunk per frame) was read with
ReadVector<uint32_t> under hdf5_mutex: the inflate and HDF5's element-wise type conversion serialised
every reading thread and held an AGIPD run to ~40 frames/s. It now goes through ReadStored, as the
data does: only the chunk lookup is under the lock; the read, inflate and unshuffle are not, and the
stored uint16/uint32 values are widened to uint32 exactly as HDF5 converted them.
The direct path is taken where the mask is plain unsigned integers of the frame's size, stored alike
in every file (checked once, in Place); a chunk stored with a filter skipped is read by HDF5 as for
the data, and any other mask keeps the locked HDF5 read. Images and pixel mask are bit-identical to
before (hash of every pixel of 2000 images on two AGIPD runs and one MPCCD run).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A detector of two or more panels is no longer assembled. The image is the stored frame, and each panel of
the geometry (CrystFEL .geom or the file's own panel table) is a DetectorModule over its box of the frame,
at its true placement on the detector plane: fractional origin, the panel's own axes, no snapping to an
image axis or a whole pixel. Pixels on no panel, edge, bad-region and file-masked pixels are masked in the
stored layout. The block scatter (a transpose for the 90-degree-turned JUNGFRAU and MPCCD panels) is gone
for these inputs. A single panel keeps the old block path, so single-panel CXI files are unchanged.
Signed-integer frames in their own units (gain 1, no per-frame mask) go up with no copy: a bitshuffle/LZ4
chunk still compressed, for the GPU preprocessor's BSLZ4 decoder (int16's smallest value is its error
value there, so the type-min rule holds), a deflated one inflated outside the HDF5 lock straight into the
caller's buffer. Everything else (float, unsigned, a gain, a per-frame mask) is converted once in stored
order. A float NaN/inf now reads as no measurement (INT32_MIN) instead of 0, and values past int32 are
clamped before the cast.
Also: SpotToSave.panel is filled from DiffractionSpot::Panel(), so AssignModules takes the panel the spot
was found on. PreviewImage_PanelLayout expects stage 3's PlaneRasterOf raster (merge fallout).
Main pass, 2000 frames, vs the CCD-reader build: JF4M int16 bslz4 465 -> 1860 frames/s, JF16M 105 -> 1111,
MPCCD 212 -> 252; flat Jungfraujoch data unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The .geom parser now takes the masking information the format defines:
- mask arrays: mask/mask_file/mask_good/mask_bad (mask 0) and maskN_data/maskN_file/maskN_goodbits/
maskN_badbits for N = 0..7; global keys hold for the panels after them, a panel may override. A mask
that is the same on every panel is one whole-frame mask, otherwise one per panel, applied on that
panel's pixels only.
- a relative mask_file is looked for in the working directory (as CrystFEL does), then beside the .geom.
- bad regions (bad*): a lab-plane box (min_x..max_y) or a fs/ss range on a panel (or, old syntax, on
the whole frame); mask_edge_pixels and no_index.
JFJochXFELReader already had the static pixel mask and the per-frame mask path for these; the masks
go through them unchanged. A mask whose path holds % (one per shot group) is applied frame by frame
like a 3D one. Where the geometry names mask arrays, the data file's own Cheetah masks (CXI
mask_shared and per-shot mask with Cheetah's generic bad bits, /metadata/pixelmask) are not added:
the geometry says which bits matter (e.g. an AGIPD mask_bad that includes bit 32). --mask-good /
--mask-bad still add their masks to these.
Tests: [CrystFELGeometry] (new keys, per-panel override, relative mask file), JFJochXFELReader_*
(new GeometryMasks case on a synthetic CXI + .geom), [JFJochMultiReader], JFJochXTCReader*.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
DecodeInto no longer turns every stored pixel into a float and back:
- the gain is one value (adu_per_photon_), not a float per stored pixel
- it always was one value; only --gain or a measured gain set it;
- a 16-bit signed frame (Cheetah CXI, SACLA) without a gain is handed
up as Int16, in its own type; other integer frames are widened to
int32 without a float; only a float frame or a gain other than one
goes through lround(v / gain), exactly as before;
- placement is per panel (start, size, destination steps) instead of a
64-bit destination per stored pixel; a panel square with the image
is copied row by row (PAL: 1440x1440 into 1444x1444 - the two-pixel
margin is Place()'s, kept so the geometry does not move);
- the decode buffers (chunk, inflated, unshuffled) are reused per
thread, the bitshuffle decode is not copied again;
- a chunk whose HDF5 filter_mask is not zero (a filter skipped) is
read by HDF5 instead of being decoded as if it were not;
- the never-set value flags (flag_lessthan/morethan/equal) are gone
from the layout and from the per-pixel loop.
MeasureGain in the CLI now reads Int16 frames as well, so a SACLA run
still has its gain measured.
Tests: JFJochXFELReader_SACLA/_CXI updated for the mode, new
JFJochXFELReader_CXI_Int16 (deflate without shuffle, a chunk with its
filter skipped) and JFJochXTCReader_StoredPixels (synthetic XTC,
stored hand-up and the offset path).
Measured on two PAL-XFEL Rayonix sets (int16, deflate; RTX 5080,
5950X, page cache warm; reports and .hkl identical to the baseline):
main pass, 2000 frames: 296 -> 799 frames/s (pal86),
302 -> 755 frames/s (pal90)
wall, 2000 frames: 26.4 -> 13.6 s, 32.1 -> 19.4 s
(the geometry-refinement steps, which read frames too, halve)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
tools/battery/still/agipd_to_int16.py converts an AGIPD 1M Cheetah CXI file into the only form rugnux
supports for AGIPD: each module's eight 64-row ASICs placed 66 rows apart (the 2-pixel ASIC gaps
inserted), one 526 x 128 panel per module; the per-frame mask and mask_shared (bad bits = the
geometry's mask_bad) and the geometry's bad regions folded into the data as -32768; values
round(value / ADU per photon) as int16 (clipped to +-32767; no pixel clipped in either set);
bitshuffle/LZ4, one chunk per frame at /entry_1/data_1/data. It writes the matching CrystFEL geometry
(16 panels, each the rigid fit of its ASICs, module coffset kept) and refuses a module whose ASICs are
not on one straight line at a whole-pixel pitch within 0.5 px (worst here: 0.16 px for CXIDB 83,
0.002 px for 152).
Reader: a stored value equal to the smallest value of its signed integer type (INT16_MIN, INT32_MIN)
is no measurement. The reader decodes to float and divides by the gain, so -32768 used to come out as a
count; the CrystFEL parser takes only module geometry, so a flag_lessthan in the .geom could not say it.
Still battery: euxfel83 / euxfel152 read conv/*_int16.h5 with conv/agipd_int16.geom and --gain 1.
Full sets, -N 16, --refine-geometry=off, --model (other jobs on the machine, load 7-23):
indexed d_min CC1/2 CC_HALF_SPLIT R_split R_FREE main pass wall
CXIDB 83 original 90.9% 1.84 0.641 0.617 0.482 0.280 89.3 s (31/s) 105 s
converted 94.1% 1.84 0.686 0.644 0.470 0.277 6.5 s (429/s) 20 s
CXIDB 152 original 17.5% 5.37 0.521 0.479 0.590 0.308 131.7 s (29/s) 153 s
converted 62.8% 2.92 0.876 0.891 0.354 0.284 13.2 s (287/s) 27 s
Files 4.7 -> 1.3 GB and 5.7 -> 1.1 GB. The 152 gain is the mask: the reader took only Cheetah's
generic bad bits, while the geometry's mask_bad (0xffff) also marks three broken modules (bit 32) and
its bad region a fourth; the converter run with the generic bits instead gives 17.5% / 3.41 A again.
With geometry refinement on (16 rigid modules): 83 94.2%, 1.75 A, CC1/2 0.665, R_FREE 0.284, 49 s;
152 65.1%, 2.94 A, CC1/2 0.843, R_FREE 0.278, 27 s - mixed, the manifest keeps it off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The direct chunk read opened one HDF5ImageSource::RawFile per file and never released it, so the
MAX_OPEN_FILES bound on the HDF5 handles did not hold for that path: a dataset stored one frame per
file would keep a descriptor per file open. They now follow the same least-recently-used bound; a
reading thread still holding a released handle keeps it through its shared_ptr.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
- --mask-good / --mask-bad take [<file>:]<dataset>[@<bits>]: without a file the dataset is read from
each data file (a per-frame mask stored beside the frames, as a Cheetah CXI holds it), and <bits>
says which bits mark a pixel bad (or good) - what a CrystFEL geometry's mask/mask_good/mask_bad
said, now that the geometry file gives module geometry only.
- A gain the background shows at or below one detector unit per photon is not applied; such frames
stay in detector units, as a CCD's do. Scaling a low-gain Rayonix's frames up by it cost one PAL
set most of its indexing (69% -> 33%).
- The HDF5 stills readers ask H5Fis_accessible with the error stack silenced before opening a file,
so an XTC stream offered to them no longer prints HDF5's diagnostics.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto
rc178 with the owner's decisions for this release:
- Inputs are files, never a directory; several files of one format are one dataset (the multi-input
CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles,
ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and
xtc::CollectStreams are gone, CanRead* accept a regular file only.
- One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds:
per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera
length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is
refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit
decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work).
- Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's
place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through
HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib,
bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through
HDF5's filter under the global lock, one frame at a time.
- LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and
binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event
dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all.
- CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path)
and optionally the camera length; everything else is ignored with a note in the log.
- --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored
layout (what a .geom's mask_file used to give).
- Gain measured on the background for panel-table and CXI input that states none (--gain overrides).
- Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with
one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions
are mapped through the modules, the image is not re-assembled.
- Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge
on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only
where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table,
.geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx
NXdetector_module groups), read back with --module-geometry.
- A panel-table file's camera length is nominal: with a reference cell it is refined from the cell.
- Broker and writer output unchanged: masters keep the single NXdetector_module group, the start
message carries no module list.
Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation],
SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
