Jungfraujoch/docs/HDF5.md

# HDF5 / NeXus data format

Jungfraujoch stores images and on-the-fly analysis results in HDF5 files that aim to be
[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html)-compliant. On top of the
NXmx application definition, Jungfraujoch records a substantial amount of *derived* metadata
(spot finding, indexing, integration, azimuthal integration, per-image statistics, timing). These
extra entries do not exist in NXmx and are documented here so that the layout is unambiguous and
reusable.

This page documents the **file layout and the data fields**. The operational behaviour of the
writer (running, republishing, file finalisation) is described in
[jfjoch_writer](JFJOCH_WRITER.md). The wire format that feeds the writer is described in
[CBOR messages](CBOR.md); fields below frequently correspond one-to-one to CBOR message fields, and
that document is a useful companion for their meaning.

```{contents} On this page
:local:
:depth: 2
```

## 1. Motivation: derived metadata and FAIR data

The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep
them findable, accessible, interoperable and reusable (FAIR). Jungfraujoch is used for both
**rotation** macromolecular crystallography (single- and multi-crystal, including fine-sliced and
helical scans) and **serial** crystallography (stills, grid scans); the same concerns apply to both:

* **Findability.** Raw diffraction images carry almost no descriptive metadata about *content*.
  Quantities such as background level, number of diffraction spots, or indexing outcome let a user
  judge the quality and relevance of a dataset *before* inspecting the raw images.
* **Accessibility at scale.** A single experiment can span tens to hundreds of terabytes. Standard
  retrieval (e.g. HTTP) makes a dataset *available* but not *inspectable* — users would otherwise
  have to download a large fraction of the data just to decide whether it is useful. Compact
  derived representations make discovery, assessment and reuse feasible.

Because Jungfraujoch couples acquisition with real-time analysis used to *steer* experiments,
transparency and reproducibility of that analysis matter. As a minimum the writer therefore
preserves spot-finding and indexing results together with the filters that were applied, and it can
retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse.

### Two complementary layouts: per-image spots vs. a reflection table

Jungfraujoch stores analysis products in two shapes, matching how each is accessed.

**Per-image spot finding / indexing.** Spot finding and indexing are inherently *image-centric* —
the natural query is "give me the spots for image *n*" — and this holds for serial stills and for
rotation frames alike. For these products Jungfraujoch adopts a layout similar to the
[Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the convention
understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties (position,
intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by image number,
with each image allocated room for up to a predefined maximum number of spots. These dense arrays
are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are retrieved
without traversing variable-length structures. The cost is some storage overhead for unused slots
(padded with sentinels), which is acceptable for the access pattern.

**Integrated reflections.** Integrated intensities are naturally a *dataset-wide* table, which is
exactly the model of the NeXus
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class.
This fits rotation crystallography well, and Jungfraujoch uses NXreflections for its integration
results (see §4.2 below). We deliberately do *not* force spot finding/indexing into a single
experiment-wide table: across the hundreds of thousands of patterns typical of serial — or
fine-sliced rotation — experiments, that would require aggregating the whole experiment before the
spots of one image can be read. We encourage the community to develop standardised NeXus application
definitions for image-centric crystallography products that combine NeXus interoperability with the
access patterns and scale of modern high-throughput experiments.

## 2. File layout

A run is written as one **master file** plus, depending on the format, one or more **data files**:

```
<prefix>_master.h5             # NXmx master file (metadata + links / virtual datasets)
<prefix>_data_000001.h5        # data file: images + per-image analysis
<prefix>_data_000002.h5
...
```

The master file is produced by `writer/HDF5NXmx.cpp`; data files by `writer/HDF5DataFile.cpp` and
its plugins (`writer/HDF5DataFilePlugin*.cpp`). Files are written to a temporary `*.<random>.tmp`
name and renamed on successful close.

Three master-file variants exist (set via `file_format`):

| Format | Value | Master ↔ data linking |
|--------|:-----:|------------------------|
| **NXmxLegacy** | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. |
| **NXmxVDS** (default) | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. |
| **NXmxIntegrated** | 3 | No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format. |

In legacy/VDS mode, image-indexed analysis arrays live in the **data files** and are exposed in the
master file through external links or virtual datasets; in integrated mode they are written
directly into the single file. Throughout this document a "✓ in master" column marks entries that
are visible (directly or via link/VDS) from the master file.

Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or
bitshuffle + Zstd. Signed integer image datasets carry `INTx_MIN` as the HDF5 fill value (the
"masked / no-data" sentinel); unsigned ones are left at HDF5's own fill of 0, because every unsigned
code is a legitimate count. In the master's virtual dataset the fill is the error marker for both,
so a data file missing beside a VDS master reads as masked rather than as zero counts.

> **Signed images and the Neggia XDS plugin.** Neggia dispatches on the size of the pixel in bytes
> and always casts to an unsigned type, consulting the signedness of the data only for the pixel
> mask. A **signed 16-bit** image is therefore read wrongly: a count of `-2` reaches XDS as `65534`,
> and the `-32768` error marker as `32768`. Signed 32-bit happens to degrade safely, because every
> negative value ends up above `INT32_MAX` and is mapped to `-1`. Use the Jungfraujoch XDS plugin,
> or the Global Phasing build of Durin, for signed data — see
> [Integration with MX data processing software](SOFTWARE_INTEGRATION.md).

### Reprocessing output: `<prefix>_process.h5`

The offline reprocessing tool [`rugnux`](TOOLS.md) (`rugnux/rugnux_cli.cpp`) re-runs the
full analysis pipeline (spot finding, indexing, refinement, integration, scaling) on an existing
dataset and writes its results to a master file named **`<prefix>_process.h5`**. This file uses the
**integrated** format, but instead of copying the images its `/entry/data/data` is a *virtual
dataset that links back to the original image files* (`hdf5_source_data` →
`NXmx::LinkToData_ProcessingVDS`). The result is a compact, self-describing companion file that
holds *all* the derived analysis (everything in §4) plus a virtual view
of the raw images — without duplicating terabytes of data.

This is a particularly FAIR-friendly artefact: it can be shared or archived alongside (or instead
of) the raw data to convey what is in a dataset and how it processed, while the `/entry/data/data`
VDS still resolves to the original images when they are available. `rugnux` can also process
an equally-spaced *subset* of images (start/end/stride), producing a down-sampled reference set.

## 3. NXmx-standard content

The entries below are part of, or valid base classes for, the
[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html) application definition.
"NXmx" = listed in the application definition; "base" = a valid field of the relevant NeXus base
class (`NXdetector`, `NXsample`, `NXsource`) but not in the NXmx required/recommended subset.

### `/entry` (NXentry)

| Field | Std | Notes |
|-------|:---:|-------|
| `definition` | NXmx | value `"NXmx"` |
| `start_time` | NXmx | arming time |
| `end_time`, `end_time_estimated` | NXmx | approximate end time |

File-level HDF5 attributes `file_name`, `file_time`, `HDF5_Version` are also set.

### `/entry/source` (NXsource), `/entry/instrument` (NXinstrument)

| Field | Std | Units |
|-------|:---:|-------|
| `source/name`, `source/type` | NXmx / base | |
| `source/current` | base | A |
| `instrument/name` | NXmx | |

### `/entry/instrument/beam` (NXbeam)

| Field | Std | Units |
|-------|:---:|-------|
| `incident_wavelength` | NXmx | angstrom |
| `incident_wavelength_spread` | NXmx | angstrom (only if polychromatic) |
| `total_flux` | NXmx | Hz |
| `incident_beam_size` | NXmx | m (two elements, x then y; written only when both beam sizes are given) |

### `/entry/instrument/attenuator` (NXattenuator)

| Field | Std |
|-------|:---:|
| `attenuator_transmission` | NXmx |

### `/entry/instrument/detector` (NXdetector)

| Field | Std | Units |
|-------|:---:|-------|
| `depends_on` | NXmx | → `transformations/rot3` |
| `beam_center_x`, `beam_center_y` | NXmx | pixel (0.0 = centre of the first pixel, see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)). The **PONI**, not the direct beam - see below |
| `distance` | NXmx | m |
| `count_time`, `frame_time` | NXmx | s |
| `sensor_thickness` | NXmx | m |
| `sensor_material` | NXmx | |
| `description` | NXmx | |
| `threshold_energy` | NXmx | eV (EIGER; written only for a single channel) |
| `x_pixel_size`, `y_pixel_size` | base | m |
| `serial_number` | base | |
| `bit_depth_readout` | NXmx | bit depth of the **stored image**, not of the detector electronics - see below |
| `saturation_value` | NXmx | highest valid value. Read inclusively by NXmx, by the DIALS `trusted_range` and by the XDS `OVERLOAD` parameter; a saturated pixel carries the value one above it |
| `underload_value` | NXmx | lowest valid value: `0` for an unsigned image, `INTx_MIN + 1` for a signed one |
| `flatfield_applied` | NXmx | |
| `pixel_mask`, `pixel_mask_applied` | NXmx | `pixel_mask` is `[y, x]`, hard-linked from `detectorSpecific/pixel_mask` |
| `countrate_correction_applied` | NXmx | |
| `number_of_cycles` | base | frame-summation factor |

#### Why `bit_depth_readout` is the image depth

NXmx defines only `bit_depth_readout`, "how many bits the electronics record per pixel", and has no
field for the depth of the image actually stored. The two differ whenever summation is used: the
readout stays at the detector's native width while the summed image must be wider to hold the sum.

Jungfraujoch writes the **stored image depth** into `bit_depth_readout` (and the identical value
into the non-standard `bit_depth_image`). The electronic value is a constant of the detector and
tells a data consumer nothing, whereas readers do use `bit_depth_readout` as the width of the
stored pixel — DIALS, for instance, derives its masking markers from it and cannot read a 32-bit
image without it. Writing the electronic value there would therefore mislead exactly in the case
where the two differ.

Note that `bit_depth_readout` gives the width only. The **sign** is carried solely by the HDF5
element type of `/entry/data/data` (and, on the wire, by `image_dtype`); there is no NXmx field for
it.

### `/entry/instrument/detector/transformations` (NXtransformations)

The NXtransformations *mechanism* (the `depends_on` chain, `transformation_type`, `vector`,
`offset` attributes) is standard. The axis **names** follow the PyFAI PONI convention chosen by
Jungfraujoch (see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)):

| Axis | Type | Units | Vector | Depends on |
|------|------|-------|--------|-----------|
| `rot3` | rotation | rad | `(0, 0, -1)` | `.` |
| `rot2` | rotation | rad | `(1, 0, 0)` | `rot3` |
| `rot1` | rotation | rad | `(0, -1, 0)` | `rot2` |
| `translation` | translation | m | unit vector along the sample→PONI direction | `rot1` |

`/entry/instrument/detector/depends_on` is `translation`, and the module's `fast_pixel_direction`,
`slow_pixel_direction` and `module_offset` depend on it in turn. A chain is applied innermost-first,
so reading it outwards the detector is placed at its distance and beam centre and *then* tilted about
the sample — which is what makes a tilt pivot about the crystal rather than about the panel corner.
The `vector` values are in NXmx's **McStas** frame, which is Jungfraujoch's internal frame with x and
y negated.

The beam centre is encoded in `translation` (its offset from the sample), not only in the
informational `beam_center_x`/`beam_center_y` fields. In a `_process.h5` written by rugnux these axes
carry the **refined** detector geometry — the refined beam centre folds into `translation` and the
refined tilt into `rot1`/`rot2`/`rot3`; the broker writes the user-provided geometry unchanged.

#### `beam_center_x`/`beam_center_y` is the PONI; `direct_beam_x`/`direct_beam_y` is the beam

`beam_center_x`/`beam_center_y` is the **PONI** - the foot of the perpendicular dropped from the
sample onto the detector plane. On a tilted detector that is *not* where the undeflected beam lands:
the two points are `distance * tan(tilt) / pixel_size` apart, which is around 8 px on a real in-house
setup and grows with the distance.

Most programs that ask for "the beam centre" mean the point the beam lands on - XDS's `ORGX`/`ORGY`
among them - so Jungfraujoch writes that point out as well:

| Dataset | Units |
|---------|-------|
| `/entry/instrument/detector/detectorSpecific/direct_beam_x` | pixel |
| `/entry/instrument/detector/detectorSpecific/direct_beam_y` | pixel |

They are computed from the same beam centre, distance and `rot1`/`rot2`/`rot3` written beside them,
so the file cannot disagree with itself; on an untilted detector they equal `beam_center_x`/`_y`.

The *provenance* differs by which program wrote the file, though the field does not. In a master
written by `jfjoch_broker` the geometry is the one the user stated, so `direct_beam_x`/`_y` is a
statement about the user's geometry; in a `_process.h5` written by rugnux it is the refined geometry,
so it is a measurement. A user handing either to XDS should know which of the two they have.

### `/entry/instrument/detector/module` (NXdetector_module)

`data_origin`, `data_size`, `fast_pixel_direction`, `slow_pixel_direction`, `module_offset` — all
NXmx (`fast/slow_pixel_direction` and `module_offset` carry transformation attributes). The two
pixel-direction vectors carry the discrete image orientation (mirror in Y, multiples of 90° about the
beam); for a detector this system assembled itself they are the McStas form of the internal +x and
+y, i.e. `(-1, 0, 0)` and `(0, -1, 0)`.

### `/entry/sample` (NXsample)

| Field | Std | Units / notes |
|-------|:---:|-------|
| `name` | NXmx | |
| `depends_on` | NXmx | points at the innermost axis of the sample chain, or `.` for stills |
| `temperature` | NXmx | K |
| `transformations/` (NXtransformations) | NXmx | the sample chain, written in mounting order; hard-linked as `/entry/sample/goniometer` |
| `unit_cell` | base | `[a, b, c, α, β, γ]` |
| `space_group_number` | base | International Tables number |
| `space_group` | base | extended Hermann-Mauguin name, e.g. `P 43 21 2`, `R 3:H` — this is the field that carries the **setting**, and the one the reader takes the group from; the number alone always reads back as the reference setting |
| `ub_matrix` | base | `[1, 3, 3]`, Angstrom⁻¹ |

The chain is written from the base outwards, so the innermost axis — the one `depends_on` names — is
the one nearest the sample. It may hold, in that order: the grid-scan translations `grid_scan_x` and
`grid_scan_y`, the spindle, and a Smargon head's `chi` and `phi`. A grid scan and a goniometer axis
are **not** alternatives; both can be present.

A **grid scan is collected at a stationary spindle, and the angle it stood at is stated by sending
the goniometer axis with a step of 0** — in `dataset_settings.goniometer`, or as the `goniometer`
map of the CBOR start message. The angle is then written per image as `omega` (or whatever the axis
is named) in the chain above, and read back by `reader/`. Send no axis and the spindle is still
recorded, at 0: NXmx would allow a sample with no goniometer at all (`depends_on = "."`), but a
sample chain of translations alone is not something readers accept, so the placeholder is written
whether or not the angle is known. **A 0 there means "nobody said", not "the spindle was at 0".**

A **Smargon head position is told apart from the spindle** by the `equipment_component` attribute,
which is `"smargon"` on `chi` and `phi` and absent on the spindle. This is load-bearing: a spindle can
itself be named `phi`, and without the attribute a reader would take a head position for the scan
axis. `chi` and `phi` are written with one value per image even though neither turns, because a
reader takes the image count from the innermost axis of the chain: written as scalars, a still
recorded at a head position would read back as a single image however many were collected.

For a rotation scan the goniometer axis carries, beyond the per-image angle array `<axis>`, the
Jungfraujoch conveniences `<axis>_end`, scalar `<axis>_range_average` and `<axis>_range_total`, and
for helical scans `<axis>_helical_x/_y/_z`.

### `/entry/data` (NXdata)

`data` (3-D image stack, `[n_images, y, x]`) with `image_nr_low` / `image_nr_high` attributes.
In legacy mode this group instead contains one external link `data_000001`, … per data file.

## 4. Extensions beyond NXmx

Everything in this section is **outside the NXmx standard**. Each group is declared with
`NX_class = NXcollection` (the NeXus-sanctioned container for non-standardised content) unless noted.
The per-image arrays are indexed by image number, padded to the run length and filled with a
sentinel (`NaN` for floats, `-1`/`0` for integer indices) where a quantity is absent.

### 4.1 `/entry/MX` — spot finding and indexing (CXI-style)

The flagship extension. Spot ("peak") properties are stored as fixed-size `[n_images, max_spots]`
arrays (CXI layout, recognised by CrystFEL); scalar-per-image quantities as `[n_images]` vectors.
In legacy/VDS mode these live in the data files and are linked/virtual-stacked into the master.

**Per-spot arrays `[n_images, max_spots]`:**

| Dataset | Units | Meaning | Indexing only |
|---------|-------|---------|:---:|
| `peakXPosRaw`, `peakYPosRaw` | pixel | spot position (raw detector frame) | |
| `peakTotalIntensity` | photons | spot intensity | |
| `peakIceRingRes` | | spot lies in an ice-ring resolution band | |
| `peakH`, `peakK`, `peakL` | | Miller indices of the (indexed) spot | ✓ |
| `peakDistEwaldSphere` | Å⁻¹ | distance of the spot from the Ewald sphere | ✓ |
| `peakIndexed` | | spot fits the indexing solution | ✓ |
| `peakLattice` | | lattice the spot belongs to (`-1` = unindexed) | ✓ |

**Per-image vectors `[n_images]`:**

| Dataset | Units | Meaning |
|---------|-------|---------|
| `nPeaks` | | number of spots stored for the image (CXI) |
| `strongPixels` | | strong-pixel count (first spot-finding stage) |
| `peakCountUnfiltered` | | spots found before filtering |
| `peakCountLowRes` | | low-resolution spots |
| `peakCountIceRingRes` | | spots inside ice-ring bands |
| `peakCountIceRingControl` | | spots in the ice-free flanks beside those bands, rescaled to their q width - the control for the count above (their ratio, pooled over the run, is the spot-based ice indicator) |
| `peakCountIndexed` | | spots fitting the indexing solution |
| `imageIndexed` | | image was indexed (0/1) |
| `indexingLatticeCount` | | number of lattices found for the image |
| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A* (2016), Vol. A, [Table 3.1.3.1](https://onlinelibrary.wiley.com/iucr/itc/Ac/ch3o1v0001/table3o1o3o1.pdf)) |
| `bravaisLattice` | | Bravais lattice short code, e.g. `aP`, `mC`, `oF`, `tI`, `hP`, `hR`, `cF` |
| `profileRadius` | Å⁻¹ | crystal profile radius |
| `mosaicity` | deg | mosaicity estimate |
| `bFactor` | Ų | per-image B-factor estimate |
| `resolutionEstimate` | Å | resolution the merged data are predicted to reach, from this image's spots alone |
| `integratedReflections` | | number of integrated reflections |
| `bkgEstimate` | photons | mean background in the 3–5 Å resolution band |
| `iceRingScore` | ratio | strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice) |
| `spindleBlindFraction` | fraction (0-1) | how much of a rotation sweep's blind cone this orientation makes unrecoverable, as a lone-2-fold worst-case bound; NaN = the frame could not be assessed, which automation must treat like a value at or above the 0.5 trigger, never as 0 |
| `beam_corr_x`, `beam_corr_y` | pixel | beam-center correction applied during processing |
| `imageScaleFactor` | | on-the-fly per-image scale factor *g* |
| `imageScaleCC` | | on-the-fly scaling correlation coefficient |
| `imageScaleMosaicity` | deg | scaling-model mosaicity |
| `sweepQuality` | | why this image's stretch of the sweep was flagged — see below |

**Per-image lattices:** `latticeIndexed` `[n_images, 9]` (Å) — the real-space lattice (flattened
3×3); `latticeIndexedExtra` `[n_images, max_extra_lattices, 9]` (Å) — additional orientation
variants.

**Run-level summaries** (written into the master `/entry/MX` at finalisation):

| Dataset | Units | Meaning |
|---------|-------|---------|
| `indexing_algorithm` | | `FFBIDX` / `FFT (CUDA)` / `FFT (FFTW)` |
| `geom_refinement_algorithm` | | e.g. `beam_center` |
| `rotationLatticeIndexed` | Å | whole-run rotation-indexing lattice (`[9]`) |
| `rotationLatticeIndexedExtra` | Å | additional whole-run lattices (`[m, 9]`) |
| `rotationLatticeNiggliClass` | | Niggli class of the run lattice |
| `imageIndexedMean` | | mean indexing rate over the run |
| `bkgEstimateMean` | photons | mean background over the run |
| `spindleBlindFractionMean` | fraction (0-1) | mean `spindleBlindFraction` over the frames that had one |
| `spindleLostUniqueFraction` | fraction (0-1) | unique reflections (to the run's resolution limit) the mounting made unmeasurable, exact under the measured point group and indexed orientation; offline (rugnux) only |
| `iceRingScoreMean` | ratio | mean `iceRingScore` over the run — the single "how icy was this dataset" number (1 = no ice) |
| `indexedLatticeCount` | | per-image lattice count summary (master). *Note: data files use `indexingLatticeCount`; readers accept either.* |
| `reindexMatrix` | | change of basis from the setting the per-image data are in to the setting of `/entry/sample/unit_cell` (`[9]`, `int32`, flattened 3×3, row major) — see below |

**Reindex matrix.** The per-image `h`, `k`, `l` and `latticeIndexed` are written as each image is
processed, in the setting that image was *indexed* in. The space group is only settled afterwards, by
the merge, and settling it can re-seat the lattice into the group's conventional setting — so
`/entry/sample/unit_cell`, `/entry/sample/space_group_number` and `rotationLatticeIndexed` can be in a
different setting from the per-image data beside them. `reindexMatrix` **M** is the integral change of
basis between the two: `hkl_cell = M · hkl_written`, and the same **M** takes each per-image lattice
across (`latticeIndexedExtra` is not re-seated and stays as indexed). It is **absent** when the two
settings are the same one, which is the identity — as it is on every file written before rugnux
recorded it. The Jungfraujoch reader applies it, so everything it hands out is already in the cell's
setting; a third-party reader that ignores it will index the reflections in the wrong frame whenever
the dataset is present. Written by the offline `rugnux` path only — the broker never re-seats a
lattice — and not carried on the CBOR stream, in the same way as the other offline-only fields.

A `--model` run can leave the merged reflection files (`.mtz`/`.cif`/`.hkl`) in a *different* frame
from the `_process.h5` beside them: the model settles the alternative indexing where nothing else
did, and that choice is applied to the merged reflections as they are written. It also names the
enantiomorph, but that is a change of space-group label only and moves no reflection.
The process file is not rewritten — its per-image reflections went to disk as they were integrated —
so it keeps the space group the run itself determined and stays self-consistent with its own data.
The two frames describe the same measurements; an enantiomorphic pair merges identically, having the
same Laue class and the same absences.

**Sweep quality.** `sweepQuality` `[n_images]` (`uint8`) says why the stretch of the sweep this
image belongs to was flagged as delivering much less than the rest of the run: **0** means it was
not, and any other value is a **1-based index into `sweepQualityReasons`**, a string vector written
beside it that carries the whole vocabulary, so the codes can be read without this source. The
vocabulary is closed and stable — a code is never renamed and never reused — and currently reads
`no_diffraction`, `crystal_out_of_beam`, `weak_diffraction`, `loss_of_centring`, `radiation_damage`;
[the rugnux results report](RUGNUX_REPORT.md#sweep-quality-and-the-reason-vocabulary) defines what each one
means. Both datasets are **absent** unless the sweep-quality diagnostic ran, which needs scaling and
merging; their absence therefore means "not looked for", *not* "every image clean". Written by the
offline `rugnux` path only — the broker does not merge — and not carried on the CBOR stream, in the
same way as the other offline-only fields (`space_group_number`, the refined geometry). Nothing is
excluded from processing on the strength of it. The condensed, dataset-wide form of the same finding
is in `<prefix>_report.txt`.

CrystFEL can read the spots directly with:

```
peak_list = /entry/MX
peak_list_type = cxi
```

### 4.2 `/entry/reflections` — integrated reflections (NXreflections)

Integrated reflections are stored **per image** as
`/entry/reflections/image_NNNNNN` groups, each declared `NX_class = NXreflections`. The columns map
mostly onto the standard
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class:

| Dataset | Units | NXreflections | Meaning |
|---------|-------|:-------------:|---------|
| `h`, `k`, `l` | | standard | Miller indices |
| `d` | Å | standard | resolution |
| `int_sum` | photons | standard | integrated intensity (summation) |
| `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) |
| `background_mean` | photons | standard | mean background under the peak |
| `background_variance` | photons² | non-standard | non-signal part of σ², carried to the merge. Absent in files written before it existed; the reader then recovers it from σ² − I |
| `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` |
| `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) |
| `observed_frame` | | standard | image number of the reflection |
| `lp` | | standard | the Lorentz-polarization factor, stored as the reciprocal of the multiplier that is applied. Lorentz x polarization only, which is what the NXreflections name means; the sensor efficiency is `qe`, beside it |
| `qe` | | **extension** | the sensor efficiency, in the same reciprocal convention as `lp`, so the whole correction applied to a raw count is `1/lp * 1/qe`. Absent in files written before it existed; the reader then takes the whole of `lp` for Lorentz–polarization, which is what those files mean |
| `flight` | | **extension** | the flight path between sample and pixel, in the same reciprocal convention as `lp` and `qe`, so the whole correction applied to a raw count is `1/lp * 1/qe * 1/flight`. It runs the opposite way to `qe` — at most 1 where `qe` is at least 1 — because the medium attenuates an oblique reflection where the sensor favours it. Exactly 1 under `--flight-path vacuum`, and absent in files written before it existed; the reader then takes it as 1 |
| `partiality` | | standard | recorded fraction of the reflection |
| `delta_phi` | deg | **extension** | XDS Δφ: offset from the centre of the current frame |
| `zeta` | | **extension** | Lorentz ζ factor (reciprocal-space geometry term) |
| `image_scale_corr` | | **extension** | per-image scale correction; `I_true = image_scale_corr · int_sum` |

In the master file these per-image groups are exposed through `/entry/reflections` external links
(VDS/integrated formats).

### 4.3 `/entry/azint` — azimuthal integration

| Dataset | Shape | Units | Meaning |
|---------|-------|-------|---------|
| `bin_to_q` | `[φ_bins, q_bins]` | Å⁻¹ | q value of each bin |
| `bin_to_two_theta` | `[φ_bins, q_bins]` | deg | 2θ of each bin |
| `bin_to_phi` | `[φ_bins, q_bins]` | deg | azimuthal angle of each bin |
| `image` | `[n_images, φ_bins, q_bins]` | | per-image integrated profile (NaN for empty bins) |
| `image_std` | `[n_images, φ_bins, q_bins]` | | per-bin standard deviation |
| `image_count` | `[n_images, φ_bins, q_bins]` | | pixels contributing per bin |
| `map` | `[y, x]` | | pixel→bin mapping (master file only) |

### 4.4 `/entry/roi` — regions of interest (per-image results)

`/entry/roi/<roi_name>` has one sub-group per configured ROI, holding the **per-image result
vectors** `[n_images]`. These are written into the data files; in VDS mode they are exposed from
the master file through virtual datasets, and in integrated mode they are in the single file.
(In legacy mode they remain only in the data files.)

| Dataset | Meaning |
|---------|---------|
| `max` | maximum pixel value in the ROI |
| `sum` | sum of pixel values |
| `sum_sq` | sum of squared pixel values |
| `npixel` | number of valid pixels |
| `x`, `y` | intensity-weighted centroid |

### 4.4.1 `/entry/roi_defs` — ROI definitions (master file)

The **dataset-wide ROI definitions** (geometry, fixed for the whole acquisition) live in the
master file under a *separate* `/entry/roi_defs` group — kept apart from `/entry/roi` above so
that older readers, which iterate `/entry/roi`, are unaffected by these entries. One sub-group
`/entry/roi_defs/<roi_name>` per ROI:

| Dataset | Meaning |
|---------|---------|
| `bit_index` | which bit of `roi_map` (below) marks this ROI |
| `type` | `box`, `circle` or `azim` |
| `min_x_pxl`, `max_x_pxl`, `min_y_pxl`, `max_y_pxl` | box bounds (type `box`) |
| `center_x_pxl`, `center_y_pxl`, `radius_pxl` | circle (type `circle`) |
| `q_min_recipA`, `q_max_recipA` | Q range (type `azim`) |
| `phi_min_deg`, `phi_max_deg` | azimuthal-angle sector (type `azim`, omitted for a full ring) |

`/entry/roi_defs/roi_map` `[y, x]` is a `uint16` per-pixel bitmask: bit `bit_index` is set for
every pixel belonging to that ROI, so an ROI's footprint can be recovered exactly.

### 4.5 `/entry/image` — per-image pixel statistics

`[n_images]` vectors: `max_value`, `min_value` (viable min/max, excluding error/saturated pixels),
`error_pixels`, `saturated_pixels`, `pixel_sum`. Surfaced in the master file under `/entry/image`.

### 4.6 `/entry/profiling` — per-image timing

`[n_images]` vectors in seconds: `spotFindingTime`, `indexingTime`, `integrationTime`,
`refinementTime`, `processingTime`, `braggPredictionTime`, `preprocessingTime`, `compressionTime`,
`azIntTime`, `indexAnalysisTime`, `imageScaleTime`.

### 4.7 `/entry/detector` — acquisition diagnostics (data file)

A convenience NXcollection in the data file (note: distinct from the standard
`/entry/instrument/detector`). In **integrated** format these datasets are written under
`/entry/instrument/detector/detectorSpecific` instead.

| Dataset | Meaning |
|---------|---------|
| `timestamp`, `exptime` | per-image timestamp and exposure time |
| `number` | image number (original number if image rejection was used) |
| `det_info` | JUNGFRAU debug field |
| `storage_cell_image` | storage-cell number |
| `rcv_delay`, `rcv_free_send_buffers` | receiver internal diagnostics |
| `packets_expected`, `packets_received` | UDP packets per image |
| `data_collection_efficiency_image` | received / expected packet ratio |

### 4.8 `/entry/xfel` — pulsed-source metadata

`[n_images]` vectors `pulseID` and `eventCode`, written for pulsed sources (e.g. SwissFEL).

### 4.9 Other collections

| Path | Class | Content |
|------|-------|---------|
| `/entry/instrument/detector/detectorSpecific` | NXcollection | Dectris-style detector metadata + Jungfraujoch fields: `x_pixels_in_detector`, `y_pixels_in_detector`, `nimages`, `ntrigger`, `nimages_collected`, `nimages_written`, `data_collection_efficiency`, `max_receiver_delay`, `storage_cell_number`, `storage_cell_delay` [ns], `software_git_commit`, `software_git_date`, `jfjoch_release`, `jfjoch_writer_release`, `summation_mode`, `detect_ice_rings`, `gain_file_names`, `data_reduction_factor_serialmx`, `adu_histogram/`, `data_collection_efficiency_image`, `direct_beam_x`, `direct_beam_y` |
| `/entry/instrument/detector/calibration` | NXcollection | per-channel pedestal / calibration images (bitshuffle-compressed) |
| `/entry/instrument/fluorescence` | NXcollection | XRF spectrum: `energy` [eV], `data` |
| `/entry/user` | NXcollection | scalar values supplied under `header_appendix.hdf5` |

### 4.10 Non-standard fields inside the NXmx detector group

A few extension scalars are written *inside* the otherwise-standard `/entry/instrument/detector`
group for compatibility with existing tooling:

| Field | Units | Meaning |
|-------|-------|---------|
| `detector_distance` | m | duplicate of `distance` (Dectris/Neggia compatibility) |
| `detector_number` | | detector identifier (Dectris convention) |
| `mirror_y` (in `detectorSpecific`) | | whether the stored image is mirrored in Y relative to the raw readout; true is the MX convention (row 0 at the top) |
| `detector_orientation_mirror_y` (in `detectorSpecific`) | | whether the stored image is mirrored in Y relative to the frame `rot1`/`rot2`/`rot3` are stated in — a different setting from `mirror_y`, and one that changes no pixel |
| `detector_orientation_quarter_turns` (in `detectorSpecific`) | | multiples of 90° about the beam the stored image is turned by, relative to that same frame (0-3) |
| `direct_beam_x`, `direct_beam_y` (in `detectorSpecific`) | pixel | where the undeflected beam lands, as opposed to the PONI in `beam_center_x`/`beam_center_y` - see above |
| `error_value` | | masked/error pixel sentinel: `UINTx_MAX` unsigned, `INTx_MIN` signed (NXmx has no equivalent). NXmx `underload_value` is written too: `INTx_MIN + 1` for signed, `0` for unsigned |
| `bit_depth_image` | | stored image bit depth (DECTRIS convention, not NXmx). Equal to `bit_depth_readout` where that is written, i.e. for unsigned images |
| `acquisition_type` | | always `triggered` (Dectris convention) |
| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied |
| `jungfrau_conversion_factor` | eV | conversion factor |
| `geometry_transformation_applied` | | module→full-detector geometry applied |

NeXus has no concept of a fill or no-data value — it expects bad pixels to be flagged in `pixel_mask`,
which Jungfraujoch also writes. The in-band sentinel above is a DECTRIS compatibility convention:
SIMPLON specifies that masked pixels are flagged with `2^bit_depth_image - 1`.

For an **unsigned** image the sentinel and the saturation code are the same value, so a saturated
pixel and a masked one cannot be told apart — the FPGA collapses both onto `UINTx_MAX`. Signed
images keep them separate: `INTx_MIN` is the marker, `INTx_MAX` is saturation.

`bit_depth_readout` is written for **unsigned** images only. DIALS remaps the top two codes of
`2^bit_depth_readout` to `-1` and `-2` whenever the field is present, regardless of the pixel type:
for an unsigned image those land below `underload_value` and are correctly masked, but for a signed
one they land inside the trusted range and a saturated pixel would be integrated as a count of `-2`.
Signed images are read correctly without the field; unsigned 32-bit cannot be read at all without it.

### 4.11 User-supplied metadata: `header_appendix` and `image_appendix`

Facilities frequently need to attach metadata that Jungfraujoch does not model explicitly. Two
free-form JSON fields in the `/start` request (`broker/jfjoch_api.yaml`) provide this without any
schema change; both accept *any valid JSON*:

| Field | Carried in | Persisted to HDF5? |
|-------|-----------|--------------------|
| `header_appendix` | the **start** message, under `user_data.user` (see [CBOR](CBOR.md)) | no — except the `hdf5` sub-object (below) |
| `image_appendix` | **every image** message, as `user_data` | no |

Both are forwarded verbatim through the ZeroMQ/CBOR stream to every downstream consumer (writer,
republished analysis, viewers), so they are the recommended channel for facility- or
beamline-specific provenance (proposal, operator, optics state, per-image trigger info, …) that has
no dedicated API field.

**Persisting selected values to HDF5.** `header_appendix` is normally *not* written to the master
file. As an exception, if it contains a key `hdf5` whose value is a JSON object of scalars (strings
and numbers — no arrays or nested objects), the writer stores each entry under `/entry/user/<key>`.

For example, a `/start` request containing:

```json
{
  "header_appendix": {
    "proposal": "p20001",
    "operator": "jdoe",
    "hdf5": { "beamline": "X06SA", "ring_mode": "top-up", "attenuator_foils": 2 }
  },
  "image_appendix": { "trigger_source": "external" }
}
```

forwards the whole `header_appendix` as `user_data.user` on the start message and
`{"trigger_source": "external"}` as `user_data` on every image message, and writes three scalars
into the master file:

```
/entry/user/beamline          = "X06SA"
/entry/user/ring_mode         = "top-up"
/entry/user/attenuator_foils  = 2
```

## 5. Notes

* **Units** are written as the HDF5 `units` attribute on the dataset (e.g. `m`, `eV`, `deg`,
  `Angstrom`, `Angstrom^-1`, `Angstrom^2`, `pixel`, `s`).
* **Sentinels.** Missing per-image values are `NaN` (floats) or `-1`/`0` (integer indices); image
  pixels use `INTx_MIN` / `UINTx_MAX`.
* **Master vs data file.** In legacy/VDS formats the analysis arrays physically live in the data
  files; the master file links to them (external links in legacy, virtual datasets in VDS). In the
  integrated format there are no data files and everything is in one place.
* **CXI / CrystFEL.** `/entry/MX` follows the CXI peak-list convention; see
  [CXI file format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf).