Serial XFEL stills inputs: readers from the files themselves, module geometry and its refinement

Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto
rc178 with the owner's decisions for this release:

- Inputs are files, never a directory; several files of one format are one dataset (the multi-input
  CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles,
  ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and
  xtc::CollectStreams are gone, CanRead* accept a regular file only.
- One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds:
  per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera
  length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is
  refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit
  decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work).
- Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's
  place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through
  HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib,
  bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through
  HDF5's filter under the global lock, one frame at a time.
- LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and
  binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event
  dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all.
- CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path)
  and optionally the camera length; everything else is ignored with a note in the log.
- --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored
  layout (what a .geom's mask_file used to give).
- Gain measured on the background for panel-table and CXI input that states none (--gain overrides).
- Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with
  one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions
  are mapped through the modules, the image is not re-assembled.
- Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge
  on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only
  where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table,
  .geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx
  NXdetector_module groups), read back with --module-geometry.
- A panel-table file's camera length is nominal: with a reference cell it is refined from the cell.
- Broker and writer output unchanged: masters keep the single NXdetector_module group, the start
  message carries no module list.

Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation],
SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-20 23:30:48 +02:00
1 parent 6557ba87f4
commit ebce906025
46 files changed
+3686 -936

No files matched your search

+96 -118
View File
@@ -3,43 +3,29 @@
#include "JFJochXTCReader.h"
#include <algorithm>
#include <cmath>
#include <filesystem>
#include <fstream>
#include <nlohmann/json.hpp>
#include <map>
#include <numeric>
#include "../common/JFJochException.h"
#include "../common/Logger.h"
namespace {
using Vec = std::array<double, 3>;
double Dot(const Vec &a, const Vec &b) { return a[0] * b[0] + a[1] * b[1] + a[2] * b[2]; }
Vec Cross(const Vec &a, const Vec &b) {
return {a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]};
}
Vec ReadVec(const nlohmann::json &j) { return {j.at(0).get<double>(), j.at(1).get<double>(), j.at(2).get<double>()}; }
// A DIALS laboratory vector in the internal frame. DIALS uses the imgCIF laboratory frame (z from
// the sample to the source, y up), which differs from the internal one by a half turn about x.
Coord DialsToInternal(const Vec &v) {
return {static_cast<float>(v[0]), static_cast<float>(-v[1]), static_cast<float>(-v[2])};
}
// The DIALS experiment list beside the streams, or in the directory above them.
std::string FindExpt(const std::string &stream) {
const auto dir = std::filesystem::absolute(stream).parent_path();
for (const auto &d: {dir, dir.parent_path()}) {
std::error_code ec;
for (const auto &e: std::filesystem::directory_iterator(d, ec))
if (e.is_regular_file() && e.path().extension() == ".expt")
return e.path().string();
}
return {};
// The pixel size of a Rayonix image, from the model name - an MX340-HS is 340 mm across - and the
// image's width. Only where the width times the binning is a whole readout of 3840, 5760 or 7680
// pixels, i.e. no region of interest was read out.
std::optional<double> RayonixPixel_um(const std::string &model, uint32_t binning, uint32_t width) {
if (!model.starts_with("MX") || binning == 0)
return std::nullopt;
const auto digits = model.substr(2, model.find_first_not_of("0123456789", 2) - 2);
if (digits.empty())
return std::nullopt;
const uint64_t native = static_cast<uint64_t>(width) * binning;
if (native != 3840 && native != 5760 && native != 7680)
return std::nullopt;
return std::stod(digits) * 1000.0 / width;
}
} // namespace
@@ -48,108 +34,92 @@ bool JFJochXTCReader::CanRead(const std::string &path) {
return xtc::CanRead(path);
}
void JFJochXTCReader::ReadGeometry(const std::string &expt) {
std::ifstream in(expt);
const auto j = nlohmann::json::parse(in);
const auto &beam = j.at("beam").at(0);
const auto &panels = j.at("detector").at(0).at("panels");
if (panels.size() != 1)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
expt + " describes a detector of " + std::to_string(panels.size())
+ " panels; only a single-panel detector is read from XTC");
const auto &panel = panels.at(0);
const double wavelength_A = beam.at("wavelength").get<double>();
// DIALS states the direction from the sample to the source; the beam travels the other way.
const Vec direction = ReadVec(beam.at("direction"));
const Vec b = {-direction[0], -direction[1], -direction[2]};
const Vec fast = ReadVec(panel.at("fast_axis")), slow = ReadVec(panel.at("slow_axis"));
const Vec origin = ReadVec(panel.at("origin")); // mm, corner of the first pixel
const double pixel_mm = panel.at("pixel_size").at(0).get<double>();
// Where the beam meets the detector plane, in pixels along fast and slow, and how far along the
// detector normal the plane is.
const Vec normal = Cross(fast, slow);
const double t = Dot(origin, normal) / Dot(b, normal);
const Vec hit = {t * b[0] - origin[0], t * b[1] - origin[1], t * b[2] - origin[2]};
if (std::fabs(Dot(b, normal)) < 0.9999)
Logger("XTCReader").Warning("{}: the detector is tilted by {:.2f} deg against the beam; it is read "
"as square-on", expt, std::acos(std::fabs(Dot(b, normal))) * 180.0 / M_PI);
auto &experiment = dataset_->experiment;
const auto &f0 = frames_.front();
DetectorSetup detector = DetDECTRIS(f0.width, f0.height, "XTC camera", {});
detector.PixelSize_um(static_cast<float>(pixel_mm * 1000.0));
// A CCD: the phosphor converts at the surface, so there is no depth to correct parallax for.
detector.SensorThickness_um(0);
detector.SaturationLimit(65535);
detector.BitDepthImage(32);
const auto orientation = DetectorOrientation::Match(DialsToInternal(fast), DialsToInternal(slow));
if (!orientation.has_value())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
expt + ": the panel's fast and slow axes are not a square-on layout this reader supports");
detector.ImageOrientation(orientation.value());
detector.MinFrameTime(std::chrono::microseconds(0));
detector.MinCountTime(std::chrono::microseconds(0));
detector.ReadOutTime(std::chrono::nanoseconds(0));
experiment.Detector(detector);
experiment.BeamX_pxl(static_cast<float>(Dot(hit, fast) / pixel_mm));
experiment.BeamY_pxl(static_cast<float>(Dot(hit, slow) / pixel_mm));
experiment.DetectorDistance_mm(static_cast<float>(std::fabs(Dot(origin, normal))));
experiment.IncidentEnergy_keV(WVL_1A_IN_KEV / static_cast<float>(wavelength_A));
// The polarization degree is 2f - 1 of the fraction f DIALS states; positive where the plane
// contains the lab x axis (normal along y), as in the SMV reader.
if (beam.contains("polarization_fraction") && beam.contains("polarization_normal")) {
const Coord n = DialsToInternal(ReadVec(beam.at("polarization_normal")));
const float degree = 2.0f * beam.at("polarization_fraction").get<float>() - 1.0f;
experiment.PolarizationFactor(std::fabs(n.x) > std::fabs(n.y) ? -degree : degree);
}
}
void JFJochXTCReader::ReadFiles(const std::vector<std::string> &streams) {
frames_.clear();
Close();
Logger logger("XTCReader");
std::vector<xtc::Frame> all;
std::vector<double> energies;
std::string model;
uint32_t binning = 0;
size_t damaged = 0;
for (const auto &stream: streams) {
auto f = xtc::IndexFrames(stream);
frames_.insert(frames_.end(), f.begin(), f.end());
auto index = xtc::IndexFile(stream);
all.insert(all.end(), index.frames.begin(), index.frames.end());
energies.insert(energies.end(), index.photon_energy_eV.begin(), index.photon_energy_eV.end());
if (!index.rayonix_model.empty()) {
model = index.rayonix_model;
binning = index.rayonix_binning;
}
damaged += index.damaged;
if (index.truncated)
logger.Warning("{} ends inside its last event, which is left out", stream);
}
if (frames_.empty())
// The diffraction detector is the camera with the largest image. Another camera of the same size
// would make that choice a guess, so it is refused; an event without it is left out.
std::map<uint64_t, uint64_t> area;
for (const auto &f: all)
area[f.source] = static_cast<uint64_t>(f.width) * f.height;
if (area.empty())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"No camera images found in the XTC streams given");
const auto largest = std::max_element(area.begin(), area.end(),
[](const auto &a, const auto &b) { return a.second < b.second; });
if (std::count_if(area.begin(), area.end(), [&](const auto &a) { return a.second == largest->second; }) > 1)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"The XTC streams hold images of the same size from more than one camera; which is the "
"diffraction detector is not clear");
for (const auto &f: all)
if (f.source == largest->first)
frames_.push_back(f);
for (const auto &f: frames_)
if (f.width != frames_.front().width || f.height != frames_.front().height)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
f.file + ": the images are not all the same size");
if (damaged > 0)
logger.Warning("{} damaged, compressed or unknown-version images left out", damaged);
const auto &f0 = frames_.front();
if (!energies.empty())
photon_energy_eV_ = std::accumulate(energies.begin(), energies.end(), 0.0) / static_cast<double>(energies.size());
pixel_um_ = RayonixPixel_um(model, binning, f0.width);
const std::string expt = FindExpt(frames_.front().file);
dataset_ = std::make_shared<JFJochReaderDataset>();
dataset_->experiment = default_experiment;
has_geometry_ = !expt.empty();
if (has_geometry_) {
Logger("XTCReader").Info("{} images in XTC, geometry from {}", frames_.size(), expt);
ReadGeometry(expt);
} else {
// XTC states no geometry. One flat panel, square-on, with its middle on the beam - the run
// measures the centre - and the pixel size, distance and wavelength to be given by the caller.
Logger("XTCReader").Info("{} images in XTC, no .expt beside them: one flat panel centred on the beam",
frames_.size());
const auto &f0 = frames_.front();
DetectorSetup detector = DetDECTRIS(f0.width, f0.height, "XTC camera", {});
detector.SensorThickness_um(0);
detector.SaturationLimit(65535);
detector.BitDepthImage(32);
detector.MinFrameTime(std::chrono::microseconds(0));
detector.MinCountTime(std::chrono::microseconds(0));
detector.ReadOutTime(std::chrono::nanoseconds(0));
dataset_->experiment.Detector(detector);
dataset_->experiment.BeamX_pxl(0.5f * static_cast<float>(f0.width));
dataset_->experiment.BeamY_pxl(0.5f * static_cast<float>(f0.height));
}
DetectorSetup detector = DetDECTRIS(f0.width, f0.height, model.empty() ? "XTC camera" : model, {});
// A CCD: the phosphor converts at the surface, so there is no depth to correct parallax for.
detector.SensorThickness_um(0);
detector.SaturationLimit(65535);
detector.BitDepthImage(32);
detector.MinFrameTime(std::chrono::microseconds(0));
detector.MinCountTime(std::chrono::microseconds(0));
detector.ReadOutTime(std::chrono::nanoseconds(0));
if (pixel_um_)
detector.PixelSize_um(static_cast<float>(*pixel_um_));
dataset_->experiment.Detector(detector);
dataset_->experiment.BeamX_pxl(0.5f * static_cast<float>(f0.width));
dataset_->experiment.BeamY_pxl(0.5f * static_cast<float>(f0.height));
if (photon_energy_eV_)
dataset_->experiment.IncidentEnergy_keV(static_cast<float>(*photon_energy_eV_ / 1000.0));
logger.Info("{} images in XTC ({}); {}; {}; one flat panel centred on the beam", frames_.size(),
model.empty() ? "camera" : model,
photon_energy_eV_ ? fmt::format("photon energy {:.1f} eV, the mean of {} beam records",
*photon_energy_eV_, energies.size())
: std::string("no beam records"),
pixel_um_ ? fmt::format("{:.2f} um pixels ({}x{} binning)", *pixel_um_, binning, binning)
: std::string("pixel size not stated"));
dataset_->error_value = -1;
dataset_->experiment.ImagesPerTrigger(static_cast<int64_t>(frames_.size()));
dataset_->pixel_mask = std::make_shared<const PixelMask>(frames_.front().width, frames_.front().height);
dataset_->pixel_mask = std::make_shared<const PixelMask>(f0.width, f0.height);
SetStartMessage(dataset_);
}
void JFJochXTCReader::ApplyGain(float adu_per_photon) {
adu_per_photon_ = adu_per_photon;
counts_in_photons_ = true;
dataset_->experiment.Detector().SaturationLimit(static_cast<int64_t>(std::floor(65535.0f / adu_per_photon)));
SetStartMessage(dataset_);
}
@@ -160,6 +130,10 @@ uint64_t JFJochXTCReader::GetNumberOfImages() const {
void JFJochXTCReader::Close() {
frames_.clear();
dataset_.reset();
photon_energy_eV_.reset();
pixel_um_.reset();
adu_per_photon_ = 1.0f;
counts_in_photons_ = false;
}
template <class Buffer>
@@ -169,7 +143,11 @@ CompressedImage JFJochXTCReader::DecodeInto(int64_t image_number, Buffer &buffer
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Image number out of range");
const auto &f = frames_[image_number];
buffer.resize(static_cast<size_t>(f.width) * f.height * sizeof(int32_t));
xtc::ReadFrame(f, reinterpret_cast<int32_t *>(buffer.data()), scratch);
auto out = reinterpret_cast<int32_t *>(buffer.data());
xtc::ReadFrame(f, out, scratch);
if (adu_per_photon_ != 1.0f)
for (size_t i = 0; i < static_cast<size_t>(f.width) * f.height; i++)
out[i] = static_cast<int32_t>(std::lround(static_cast<float>(out[i]) / adu_per_photon_));
return CompressedImage(buffer.data(), buffer.size(), f.width, f.height,
CompressedImageMode::Int32, CompressionAlgorithm::NO_COMPRESSION);
}