Serial XFEL stills inputs: readers from the files themselves, module geometry and its refinement

Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto
rc178 with the owner's decisions for this release:

- Inputs are files, never a directory; several files of one format are one dataset (the multi-input
  CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles,
  ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and
  xtc::CollectStreams are gone, CanRead* accept a regular file only.
- One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds:
  per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera
  length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is
  refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit
  decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work).
- Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's
  place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through
  HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib,
  bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through
  HDF5's filter under the global lock, one frame at a time.
- LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and
  binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event
  dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all.
- CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path)
  and optionally the camera length; everything else is ignored with a note in the log.
- --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored
  layout (what a .geom's mask_file used to give).
- Gain measured on the background for panel-table and CXI input that states none (--gain overrides).
- Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with
  one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions
  are mapped through the modules, the image is not re-assembled.
- Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge
  on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only
  where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table,
  .geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx
  NXdetector_module groups), read back with --module-geometry.
- A panel-table file's camera length is nominal: with a reference cell it is refined from the cell.
- Broker and writer output unchanged: masters keep the single NXdetector_module group, the start
  message carries no module list.

Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation],
SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-20 23:30:48 +02:00
1 parent 6557ba87f4
commit ebce906025
46 files changed
+3686 -936

No files matched your search

+31 -2
View File
@@ -4,6 +4,7 @@
#include "MXAnalysisWithoutFPGA.h"
#include <algorithm>
#include <cmath>
#include <spdlog/spdlog.h>
#include "spot_finding/StrongPixelSet.h"
@@ -205,14 +206,14 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
int64_t image_number) {
if (count_crowding)
crowding_engine->CountNeighbourCrowding(predicted, npredicted);
return bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
return IntegrateOnImage(predicted, npredicted, image_number);
};
// The supercell probe integrates on the same engine, but what the engine counts is the run's
// measurement of its own stencil, and the probe's reflections are not the run's: keep them out.
const auto probe_integrate_fn = [this](const std::vector<Reflection> &predicted, size_t npredicted,
int64_t image_number) {
const BraggIntegrationCounts before = bragg_engine->Counts();
auto ret = bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
auto ret = IntegrateOnImage(predicted, npredicted, image_number);
const BraggIntegrationCounts after = bragg_engine->Counts();
probe_counts.predicted += after.predicted - before.predicted;
probe_counts.bkg_starved += after.bkg_starved - before.bkg_starved;
@@ -504,3 +505,31 @@ void MXAnalysisWithoutFPGA::UpdateMaskResolution(const SpotFindingSettings &sett
spotFinder->SetResolutionMaskBits(*mask_resolution);
adaptiveSpotFinder->SetResolutionMaskBits(*mask_resolution);
}
std::vector<Reflection> MXAnalysisWithoutFPGA::IntegrateOnImage(const std::vector<Reflection> &predicted,
size_t npredicted, int64_t image_number) {
const auto &modules = experiment.GetModules();
if (modules.empty())
return bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
// Predictions are where the reflections fall on the detector plane; the engine reads the image, so
// each goes to where its module put those pixels, and comes back to the plane once integrated.
std::vector<Reflection> on_image(predicted.begin(), predicted.begin() + static_cast<int64_t>(npredicted));
for (auto &r: on_image) {
const auto [x, y] = modules.PlaneToImage(r.predicted_x, r.predicted_y);
r.predicted_x = static_cast<float>(x);
r.predicted_y = static_cast<float>(y);
}
auto ret = bragg_engine->Run(*preprocessor_buffer, on_image, npredicted, image_number);
for (auto &r: ret) {
const auto [px, py] = modules.ImageToPlane(r.predicted_x, r.predicted_y);
r.predicted_x = static_cast<float>(px);
r.predicted_y = static_cast<float>(py);
if (std::isfinite(r.observed_x) && std::isfinite(r.observed_y)) {
const auto [ox, oy] = modules.ImageToPlane(r.observed_x, r.observed_y);
r.observed_x = static_cast<float>(ox);
r.observed_y = static_cast<float>(oy);
}
}
return ret;
}