Jungfraujoch/docs/RUGNUX_FORMATS.md

# What Rugnux reads

Which data `rugnux` opens, before anything is typed. The short answer: an HDF5 master (NXmx or
DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep — nothing else
is read, so any other format has to be converted to one of these first.

**Input** is an HDF5 master file, or a directory of PILATUS miniCBF, marCCD or SMV frames. One input is
**one sweep of one crystal** — Rugnux does not combine sweeps or crystals in a run; process each
sweep to its own `_unmerged.mtz` and merge them downstream
(see [Taking the data onward](RUGNUX_INTEGRATION.md#taking-the-data-onward)).

* **HDF5 master (NXmx-based)** — a file written by [`jfjoch_writer`](JFJOCH_WRITER.md), a DECTRIS
  EIGER master, or an NXmx master written by another facility's toolchain. Lengths are taken in the
  unit the file declares, the image size from the image array's own shape, and each data file's
  images at the path the master's own link names, so masters that state single values as one-element
  arrays, compose their images as a virtual dataset over the master itself, or keep them somewhere
  other than `/entry/data/data` all open. Where the NXmx spellings are absent the pre-NXmx ones are
  tried, so a DECTRIS firmware 1.x master opens too — including its goniometer, which used to be
  missed and the sweep read as stills. Images may be `bitshuffle`+LZ4/Zstd or the HDF Group's plain
  LZ4 (filter 32004); any other filter is named in the error.
* **PILATUS miniCBF sweep** — one frame per file, read natively with no conversion and no libcbf.
  Name **any frame of the sweep**, or the directory holding it, and the whole sweep is processed:
  the frames are the ones matching that frame's template (prefix plus digit count), so a directory
  holding two sweeps is not spliced into one crystal, and naming a directory takes the sweep with
  the most frames in it. The geometry, the rotation axis and the detector mounting come from the
  header, including the imgCIF axis table where the header carries one (see
  [Detector geometry](DETECTOR_GEOMETRY.md)). A raw CBF carries no analysis results, so
  `--mode scale` — which re-scales the reflections stored in a `_process.h5` — does not accept one.
  **Gzipped frames (`.cbf.gz`) are read directly**, with no decompression step: EMBL Hamburg's
  beamlines write that form by default, and expanding a sweep first would double the disk it needs.
  The two forms count as separate sweeps.
* **marCCD sweep** — what Rayonix MX-series and mar Mosaic detectors write, and what a decade of
  deposited CCD data is archived as: an uncompressed TIFF with the instrument header in the gap
  before the pixels. One frame per file, read natively. Naming a frame or its directory selects the
  sweep exactly as for miniCBF, under either naming scheme these detectors use — a numbered stem
  (`xtal_1_00042.mccd`) or the frame number as the file extension (`D1.042`). The distance, beam
  centre, pixel size, wavelength and the circle that turned come from the header; the rotation
  axis' direction does not, because the format has nowhere to state it, so the run settles its sign
  from the data as it does for a miniCBF that carries no axis table. A CCD frame marks no untrusted
  pixels, so the sweep starts with nothing masked. Like a raw CBF, it carries no analysis results
  and `--mode scale` does not accept one.
* **SMV sweep** — what ADSC Quantum detectors wrote and what Rayonix and others still write, so most
  archived CCD data from the 2000s is in this form: an ASCII `KEY=value;` block between braces, then
  the pixels. One frame per file, read natively, the sweep selected exactly as above. The beam centre
  it states is in **millimetres** and is converted here; its convention varies between writers, and a
  file that states it transposed indexes nothing until the run's own beam-centre measurement replaces
  it, which it does automatically. SMV states no saturation value at all, so overloads are judged on
  the 16-bit container alone and the reader says so once per sweep.

Support for the two CCD formats — marCCD and SMV — is **very limited and provided as-is**; they
are read so that a CCD sweep does not need a conversion step to be processed. Only the header
fields named above are read, and the headers vary by site and vintage, so no promise is made that
every writer's dialect opens. The HDF5 and miniCBF paths are the maintained ones.

**XFEL data saved as individual panels is not read.** The detectors of serial XFEL endstations —
CSPAD, AGIPD, ePix and their kin — write each panel as its own array, with the panel positions kept
in a separate geometry file rather than in the image file. Rugnux has no reader for those per-panel
containers and no way to take an external geometry, so such data cannot be opened; it belongs in
the pipelines built around panel geometries, such as CrystFEL or cctbx.xfel. An XFEL dataset that
has already been assembled into single images in an NXmx master opens like any other HDF5 input.

Spots are always found by `rugnux` itself, including for the two-pass rotation first pass — the
spot lists a dataset may already carry were found online, at the acquisition's threshold and with
its ice-band spots already discarded, so reusing them would hide the spot-finding settings from
the lattice search.