Jungfraujoch/docs/RUGNUX_FORMATS.md

What Rugnux reads

Which data rugnux opens, before anything is typed. The short answer: an HDF5 master (NXmx or DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep (ADSC or Rigaku d*TREK), and for serial stills the files of the XFEL pipelines (below) — nothing else is read, so any other format has to be converted to one of these first. An atomic model and a reference reflection file may be given among the inputs too, told from the images by their contents.

Input is files, never a directory: an HDF5 master file, or one frame of a PILATUS miniCBF, marCCD or SMV sweep. For rotation data one input is one sweep of one crystal — Rugnux does not combine sweeps or crystals in a run; process each sweep to its own _unmerged.mtz and merge them downstream (see Taking the data onward).

Stills are given as all the files of the dataset, which the shell expands from a glob and which are read in the order given: rugnux run1_master.h5 run2_master.h5 (or rugnux *_master.h5) for several Jungfraujoch/NXmx collections of one setup, rugnux run*.h5 for SACLA, rugnux *.cxi for a Cheetah CXI set and rugnux *.xtc for LCLS streams. Several files must all be of one format; a directory is refused.

• HDF5 master (NXmx-based) — a file written by jfjoch_writer, a DECTRIS EIGER master, or an NXmx master written by another facility's toolchain. Lengths are taken in the unit the file declares, the image size from the image array's own shape, and each data file's images at the path the master's own link names, so masters that state single values as one-element arrays, compose their images as a virtual dataset over the master itself, or keep them somewhere other than /entry/data/data all open. Where the NXmx spellings are absent the pre-NXmx ones are tried, so a DECTRIS firmware 1.x master opens too — including its goniometer, which used to be missed and the sweep read as stills. Images may be bitshuffle+LZ4/Zstd or the HDF Group's plain LZ4 (filter 32004); any other filter is named in the error. The sample temperature (/entry/sample/temperature) is read in the unit the file declares - K, or Celsius (degC, C) converted to K; no declared unit means K, an unknown one is warned about and the temperature not read. It goes into the mmCIF (_diffrn.ambient_temp, K) and the SHELX .ins (TEMP, °C). The other formats record no sample temperature, so their runs write neither: SHELXL then assumes its default of 20 °C, and for a cryo-cooled crystal TEMP has to be added to the .ins by hand (see Small-molecule structures).
• PILATUS miniCBF sweep — one frame per file, read natively with no conversion and no libcbf. Name any frame of the sweep and the whole sweep is processed: the frames are the ones matching that frame's template (prefix plus digit count), so a directory holding two sweeps is not spliced into one crystal. The geometry, the rotation axis and the detector mounting come from the header, including the imgCIF axis table where the header carries one (see Detector geometry). A raw CBF carries no analysis results, so --mode scale — which re-scales the reflections stored in a _process.h5 — does not accept one. Gzipped frames (.cbf.gz) are read directly, with no decompression step: EMBL Hamburg's beamlines write that form by default, and expanding a sweep first would double the disk it needs. The two forms count as separate sweeps.
• marCCD sweep — what Rayonix MX-series and mar Mosaic detectors write, and what a decade of deposited CCD data is archived as: an uncompressed TIFF with the instrument header in the gap before the pixels. One frame per file, read natively. Naming a frame selects the sweep exactly as for miniCBF, under either naming scheme these detectors use — a numbered stem (xtal_1_00042.mccd) or the frame number as the file extension (D1.042). The distance, beam centre, pixel size, wavelength and the circle that turned come from the header; the rotation axis' direction does not, because the format has nowhere to state it, so the run settles its sign from the data as it does for a miniCBF that carries no axis table. A CCD frame marks no untrusted pixels, so the sweep starts with nothing masked. Like a raw CBF, it carries no analysis results and --mode scale does not accept one.
• SMV sweep — what ADSC Quantum detectors wrote and what Rayonix and others still write, so most archived CCD data from the 2000s is in this form: an ASCII KEY=value; block between braces, then the pixels. One frame per file, read natively, the sweep selected exactly as above. The beam centre it states is in millimetres and is converted here; its convention varies between writers, and a file that states it transposed indexes nothing until the run's own beam-centre measurement replaces it, which it does automatically. The saturation value is read where the header states one (CCD_IMAGE_SATURATION or SATURATED_VALUE); where it does not, overloads are judged on the 16-bit container alone and the reader says so once per sweep. Rigaku d*TREK headers — what Saturn CCDs on rotating-anode home sources write, recognised by their DETECTOR_NAMES key — are read in their own vocabulary: the pixel size and the point of normal incidence from the spatial-distortion record, the image directions from the detector and distortion vectors (so a mirrored or turned image keeps its hand), the distance and the detector circles, including a swung 2θ arm, from the detector goniometer, the spindle and its angles from the rotation record, and the wavelength from the scan record. Pixel values above 32767 are the format's encoded overflows and are expanded with the header's compression ratio.

Support for the two CCD formats — marCCD and SMV — is very limited and provided as-is; they are read so that a CCD sweep does not need a conversion step to be processed. Only the header fields named above are read, and the headers vary by site and vintage, so no promise is made that every writer's dialect opens. The HDF5 and miniCBF paths are the maintained ones.

Serial XFEL inputs

Stills from the XFEL serial-crystallography pipelines are read straight from the files those pipelines write, without a geometry file. Each reader takes from the file what it states; what the file does not state is either measured by rugnux or given on the command line. Every run is read at one wavelength (the mean of the per-shot photon energies where the file states them), the beam centre is measured (the file's centre, the background-isotropy measurement and the point-symmetry capture are indexed on a sample of frames and the best is kept), and a SACLA or CXI file that states no gain is read at the gain its background shows. Where the pixel values resolve single photons (a JUNGFRAU file in keV, an AGIPD file) the gain is the spacing of their photon peaks, so a baseline the file left in does not hide it, and the detector is a photon counter: its frames are counted in whole photons as they are read. Otherwise (a CCD, an MPCCD, a Rayonix) the gain is the background's variance per unit mean and the frames are analysed in the units they are stored in: the gain enters only where a photon count is assumed (the spot finder's threshold, the background clip, the integrated intensities' variances, which come out in photons). --gain gives the value; the background still says which of the two the detector is. Float frames analysed in their own units are rounded to whole ones. The saturation level is read off the frames where the file states none.

Format	Read from the file	Measured by rugnux	Given on the command line
Cheetah HDF5 with per-shot groups and a panel table (/tag-N/data, sensors stacked in one frame, their placement in /metadata - as SACLA writes for the MPCCD)	the sensors' positions and in-plane angles, pixel size, nominal camera length and pixel mask (/metadata); each shot's photon energy	beam centre, gain, saturation; with a cell, the camera length (the stated one is nominal) and each sensor's placement	nothing; --detector-distance to give the camera length
Cheetah CXI, single-panel detector (/entry_1/data_1/data, square frames, e.g. a Rayonix at PAL-XFEL)	pixel size, photon energy per shot, camera length, mask_shared	beam centre, gain, saturation; with a cell, the camera length	--detector-distance to hold a camera length
LCLS-I XTC, Rayonix (Camera::FrameV1 events; one or more streams)	photon energy per event (electron-beam record), pixel size (Rayonix model and binning), the camera's pixel offset	beam centre, saturation; with a cell, the distance	--detector-distance without a cell (not recorded in XTC); --pixel-size/--wavelength only where the XTC lacks them

The SACLA sensor positions in /metadata are the detector's nominal metrology, a few pixels from a per-beamtime refined geometry; the stills geometry refinement places each sensor as a module (see Module geometry).

What is refused, with a message rather than a wrong answer:

• a Cheetah CXI file of a tiled detector (CSPAD, AGIPD, ePix, JUNGFRAU, MPCCD stored as a slab of panels): the file does not say where the panels lie;
• multi-panel XTC (CSPAD, ePix, JUNGFRAU at LCLS), stored raw with per-module calibration, and an XTC stream holding images of the same size from two cameras;
• unconverted detector data in general (raw ADC values with gain stages and pedestals);
• per-shot wavelengths: they are averaged, never followed shot by shot;
• files of one run that store their frames differently (size, type, compression) or place the sensors differently.

Damaged or compressed XTC images are left out, and so is a last event that a partly downloaded stream ends inside. An XFEL dataset that has already been assembled into single images in an NXmx master opens like any other HDF5 input.

--crystfel-geometry <file> places the panels of an HDF5/CXI input by a CrystFEL geometry file. Taken from it are the module geometry (panel pixel ranges, fast/slow directions, corners, pixel size, per-panel coffset, the frames' data path), optionally the camera length, and the pixels to leave out: the mask arrays (mask, mask_file, mask_good, mask_bad, or up to eight as maskN_data, maskN_file, maskN_goodbits, maskN_badbits; a panel may state its own), the bad regions (bad*, a box in the laboratory plane or a pixel range on a panel), mask_edge_pixels and no_index. A mask dataset with one plane per frame, or with % in its path, is applied frame by frame. A relative mask_file is looked for in the working directory, as CrystFEL does, and then beside the geometry file. Where the geometry names mask arrays, the data file's own (the CXI mask_shared and per-shot mask, a panel table's /metadata/pixelmask) are not taken: the geometry says which bits matter. Everything else it may hold (gain, saturation, photon energy, value flags) is ignored, with a note in the log. DIALS .expt files are not read.

A mask array the data files do not hold themselves - e.g. a per-run pixel mask beside the frames - is given with --mask-good [<file>:]<dataset>[@<bits>] (a pixel is usable only where the array is non-zero, or has all of <bits>) or --mask-bad [<file>:]<dataset>[@<bits>] (unusable where non-zero, or where it has any of <bits>, e.g. @0xfeff), in the frames' own layout, as stored. Without <file> the dataset is read from each data file - a per-frame mask stored beside the frames. Both may be given more than once, and add to the masks the data files or the geometry file state (the CXI mask_shared and per-shot mask, the /metadata/pixelmask of a panel-table file, a --crystfel-geometry file's masks). A gain the background shows at or below one detector unit per photon is taken as one.

Module geometry

A detector of several modules is described module by module: where each module's pixels are in the image rugnux reads, and where they really are on the detector plane. The modules come from an explicit geometry, never from the image:

• an NXmx file with one NXdetector_module group per module (/entry/instrument/detector/module_0, module_1, ...: what a run writes as <prefix>_modules.h5, and how the masters Jungfraujoch writes state a multi-module JUNGFRAU);
• the panels the XFEL readers know (the SACLA sensors from /metadata, the panels of a --crystfel-geometry file); each panel is put on the image whole, at a whole pixel, and its fraction of a pixel and small turn are its placement;
• --module-geometry <file>: an NXmx file with such groups (what a run writes as <prefix>_modules.h5) or a CrystFEL geometry file whose panels are laid on the image (fs = column, ss = row, corners relative to the beam). It replaces the input's own modules - the way to give the module layout of a master written before the modules were recorded.

Spots are mapped from the image onto the plane, and predictions from the plane back onto the image for integration, so the image itself is never re-assembled. With a cell given, the stills geometry refinement can place each module with enough spots on its frames' lattices as a rigid block (shift and in-plane turn; --panel-refine=off keeps them where they were given, a CrystFEL .geom's panels included) where that predicts held-out spots better, module by module; modules move only relative to one another, the beam centre, distance and crystal orientations carrying what all of them share. The placement is used from there on, and written to <prefix>_modules.h5 (read back with --module-geometry); on stills of a detector of several modules <prefix>_module_placement.json is written whenever placement is not turned off - also where every module was held, as given being a placement too - read back with --module-placement and taken by the broker's /start.

Module placement file (<prefix>_module_placement.json)

{
  "module_placement": [
    {"x0": 0, "y0": 1023, "fast_axis": "Xp", "slow_axis": "Yn",
     "x0_plane": 0.31, "y0_plane": 1022.58,
     "fast_step": [1.0, 0.0002], "slow_step": [0.0002, -1.0]},
    ...
  ]
}

The file is a fragment of the broker's dataset_settings (jfjoch_api.yaml, detector_module_placement), read and written through the same OpenAPI model the broker uses; the keys come in alphabetical order.

One entry per module, in pixels of the image as it is stored (for a JUNGFRAU: as the broker assembles it, after mirror_y; a pixel's centre at its index):

• x0, y0, fast_axis, slow_axis - the module's readout pixel 0 (the first pixel of its fast/slow readout) as image x and y, and which way its fast and slow readout axes run in the image (Xp, Xn, Yp, Yn). They are the words of a broker's custom_geometry, but in the stored image: with mirror_y on, y0 here is H - 1 - y0 there (H the image height) and Yp/Yn swap. For a module of a stored file they come from its NXmx module_NN group (data_origin, data_size, and the readout axes Jungfraujoch writes beside fast_pixel_direction/slow_pixel_direction; a group without them, or a panel of a CrystFEL geometry, is read along its own pixel array, Xp/Yp from its first pixel), or from how a reader turns a panel into the image. They name the module: entries match modules exactly by these four, not by their place in the list, and with the module size they give its box. rugnux's log names modules the same way.
• x0_plane, y0_plane, fast_step, slow_step - where the module lies on the detector plane: readout pixel (i, j) - i along the fast axis, j along the slow axis from pixel 0 - is at (x0_plane, y0_plane) + i fast_step + j slow_step. As assembled, x0_plane = x0, y0_plane = y0, and each step is the unit vector of its axis (Xp = [1, 0], Yn = [0, -1], ...); the refinement's fraction of a pixel and small turn are kept.

A placement is valid only for the image layout it was measured on. --module-placement <file> and /start refuse it, naming the module, where an entry matches no module of the image or a module has no entry - another detector size, other gaps, a binned image or a raw (unassembled) JUNGFRAU stream. The beam centre and distance are not in the file: they belong to each dataset (--beam-x, --detector-distance, or the /start settings), and the placement only moves the modules relative to one another.

With the broker, the file's module_placement is merged into the /start dataset settings as it stands; /start answers 400 where it does not name exactly the detector's modules. The image is still assembled as custom_geometry lays it out, at whole pixels; the placement goes into the start message and the master's module_NN groups, and the broker's own analysis uses it.

Spots are always found by rugnux itself, including for the two-pass rotation first pass — the spot lists a dataset may already carry were found online, at the acquisition's threshold and with its ice-band spots already discarded, so reusing them would hide the spot-finding settings from the lattice search.