61def00a1a0373eede2a4a4f129021929bd7f480
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Opus 5.5
|
7c23b4befe |
CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
The .geom parser now takes the masking information the format defines: - mask arrays: mask/mask_file/mask_good/mask_bad (mask 0) and maskN_data/maskN_file/maskN_goodbits/ maskN_badbits for N = 0..7; global keys hold for the panels after them, a panel may override. A mask that is the same on every panel is one whole-frame mask, otherwise one per panel, applied on that panel's pixels only. - a relative mask_file is looked for in the working directory (as CrystFEL does), then beside the .geom. - bad regions (bad*): a lab-plane box (min_x..max_y) or a fs/ss range on a panel (or, old syntax, on the whole frame); mask_edge_pixels and no_index. JFJochXFELReader already had the static pixel mask and the per-frame mask path for these; the masks go through them unchanged. A mask whose path holds % (one per shot group) is applied frame by frame like a 3D one. Where the geometry names mask arrays, the data file's own Cheetah masks (CXI mask_shared and per-shot mask with Cheetah's generic bad bits, /metadata/pixelmask) are not added: the geometry says which bits matter (e.g. an AGIPD mask_bad that includes bit 32). --mask-good / --mask-bad still add their masks to these. Tests: [CrystFELGeometry] (new keys, per-panel override, relative mask file), JFJochXFELReader_* (new GeometryMasks case on a synthetic CXI + .geom), [JFJochMultiReader], JFJochXTCReader*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Opus 5.5
|
95e51f868d |
XFEL reader: integer frames stay integer, one gain, placement by panel
DecodeInto no longer turns every stored pixel into a float and back:
- the gain is one value (adu_per_photon_), not a float per stored pixel
- it always was one value; only --gain or a measured gain set it;
- a 16-bit signed frame (Cheetah CXI, SACLA) without a gain is handed
up as Int16, in its own type; other integer frames are widened to
int32 without a float; only a float frame or a gain other than one
goes through lround(v / gain), exactly as before;
- placement is per panel (start, size, destination steps) instead of a
64-bit destination per stored pixel; a panel square with the image
is copied row by row (PAL: 1440x1440 into 1444x1444 - the two-pixel
margin is Place()'s, kept so the geometry does not move);
- the decode buffers (chunk, inflated, unshuffled) are reused per
thread, the bitshuffle decode is not copied again;
- a chunk whose HDF5 filter_mask is not zero (a filter skipped) is
read by HDF5 instead of being decoded as if it were not;
- the never-set value flags (flag_lessthan/morethan/equal) are gone
from the layout and from the per-pixel loop.
MeasureGain in the CLI now reads Int16 frames as well, so a SACLA run
still has its gain measured.
Tests: JFJochXFELReader_SACLA/_CXI updated for the mode, new
JFJochXFELReader_CXI_Int16 (deflate without shuffle, a chunk with its
filter skipped) and JFJochXTCReader_StoredPixels (synthetic XTC,
stored hand-up and the offset path).
Measured on two PAL-XFEL Rayonix sets (int16, deflate; RTX 5080,
5950X, page cache warm; reports and .hkl identical to the baseline):
main pass, 2000 frames: 296 -> 799 frames/s (pal86),
302 -> 755 frames/s (pal90)
wall, 2000 frames: 26.4 -> 13.6 s, 32.1 -> 19.4 s
(the geometry-refinement steps, which read frames too, halve)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
|
||
|
leonarski_fandClaude Opus 5.5
|
ebce906025 |
Serial XFEL stills inputs: readers from the files themselves, module geometry and its refinement
Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto rc178 with the owner's decisions for this release: - Inputs are files, never a directory; several files of one format are one dataset (the multi-input CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles, ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and xtc::CollectStreams are gone, CanRead* accept a regular file only. - One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds: per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work). - Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib, bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through HDF5's filter under the global lock, one frame at a time. - LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all. - CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path) and optionally the camera length; everything else is ignored with a note in the log. - --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored layout (what a .geom's mask_file used to give). - Gain measured on the background for panel-table and CXI input that states none (--gain overrides). - Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions are mapped through the modules, the image is not re-assembled. - Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table, .geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx NXdetector_module groups), read back with --module-geometry. - A panel-table file's camera length is nominal: with a reference cell it is refined from the cell. - Broker and writer output unchanged: masters keep the single NXdetector_module group, the start message carries no module list. Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation], SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |