Jungfraujoch/docs/HDF5.md

# HDF5 / NeXus data format

Jungfraujoch stores images and on-the-fly analysis results in HDF5 files that aim to be
[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html)-compliant. On top of the
NXmx application definition, Jungfraujoch records a substantial amount of *derived* metadata
(spot finding, indexing, integration, azimuthal integration, per-image statistics, timing). These
extra entries do not exist in NXmx and are documented here so that the layout is unambiguous and
reusable.

This page documents the **file layout and the data fields**. The operational behaviour of the
writer (running, republishing, file finalisation, CBF/TIFF output) is described in
[jfjoch_writer](JFJOCH_WRITER.md). The wire format that feeds the writer is described in
[CBOR messages](CBOR.md); fields below frequently correspond one-to-one to CBOR message fields, and
that document is a useful companion for their meaning.

## 1. Motivation: derived metadata and FAIR data

The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep
them findable, accessible, interoperable and reusable (FAIR). For serial crystallography this is
hard for two practical reasons:

* **Findability.** Raw diffraction images carry almost no descriptive metadata about *content*.
  Quantities such as background level, number of diffraction spots, or indexing outcome let a user
  judge the quality and relevance of a dataset *before* inspecting the raw images.
* **Accessibility at scale.** A single experiment can span tens to hundreds of terabytes. Standard
  retrieval (e.g. HTTP) makes a dataset *available* but not *inspectable* — users would otherwise
  have to download a large fraction of the data just to decide whether it is useful. Compact
  derived representations make discovery, assessment and reuse feasible.

Because Jungfraujoch couples acquisition with real-time analysis used to *steer* experiments,
transparency and reproducibility of that analysis matter. As a minimum the writer therefore
preserves spot-finding and indexing results together with the filters that were applied, and it can
retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse.

### Why a CXI-style per-image layout for spot finding / indexing

Spot-finding and indexing results in serial crystallography are inherently *image-centric*: the
natural query is "give me the spots for image *n*". For these products Jungfraujoch adopts a layout
similar to the [Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the
convention understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties
(position, intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by
image number, with each image allocated room for up to a predefined maximum number of spots. These
dense arrays are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are
retrieved without traversing variable-length structures. The cost is some storage overhead for
unused slots (padded with sentinels), which is acceptable for the access pattern.

We also evaluated the NeXus
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class.
NXreflections models a *dataset-wide* reflection table, which fits integrated rotation data well —
and Jungfraujoch does use it for integration results (see §4.2 below).
But for spot finding/indexing across hundreds of thousands of patterns a single table would force
aggregation over the whole experiment before the spots of one image can be accessed efficiently.
For these intermediate products a per-image representation is more suitable. We encourage the
community to develop standardised NeXus application definitions for image-centric serial
crystallography products that combine NeXus interoperability with the access patterns and scale of
modern experiments.

## 2. File layout

A run is written as one **master file** plus, depending on the format, one or more **data files**:

```
<prefix>_master.h5             # NXmx master file (metadata + links / virtual datasets)
<prefix>_data_000001.h5        # data file: images + per-image analysis
<prefix>_data_000002.h5
...
```

The master file is produced by `writer/HDF5NXmx.cpp`; data files by `writer/HDF5DataFile.cpp` and
its plugins (`writer/HDF5DataFilePlugin*.cpp`). Files are written to a temporary `*.<random>.tmp`
name and renamed on successful close.

Three master-file variants exist (set via `file_format`):

| Format | Value | Master ↔ data linking |
|--------|:-----:|------------------------|
| **NXmxLegacy** (default) | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. |
| **NXmxVDS** | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. |
| **NXmxIntegrated** | 3 | No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format. |

In legacy/VDS mode, image-indexed analysis arrays live in the **data files** and are exposed in the
master file through external links or virtual datasets; in integrated mode they are written
directly into the single file. Throughout this document a "✓ in master" column marks entries that
are visible (directly or via link/VDS) from the master file.

Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or
bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the
"masked / no-data" sentinel), unsigned use `UINTx_MAX`.

## 3. NXmx-standard content

The entries below are part of, or valid base classes for, the
[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html) application definition.
"NXmx" = listed in the application definition; "base" = a valid field of the relevant NeXus base
class (`NXdetector`, `NXsample`, `NXsource`) but not in the NXmx required/recommended subset.

### `/entry` (NXentry)

| Field | Std | Notes |
|-------|:---:|-------|
| `definition` | NXmx | value `"NXmx"` |
| `start_time` | NXmx | arming time |
| `end_time`, `end_time_estimated` | NXmx | approximate end time |

File-level HDF5 attributes `file_name`, `file_time`, `HDF5_Version` are also set.

### `/entry/source` (NXsource), `/entry/instrument` (NXinstrument)

| Field | Std | Units |
|-------|:---:|-------|
| `source/name`, `source/type` | NXmx / base | |
| `source/current` | base | A |
| `instrument/name` | NXmx | |

### `/entry/instrument/beam` (NXbeam)

| Field | Std | Units |
|-------|:---:|-------|
| `incident_wavelength` | NXmx | angstrom |
| `incident_wavelength_spread` | NXmx | angstrom (only if polychromatic) |
| `total_flux` | NXmx | Hz |

### `/entry/instrument/attenuator` (NXattenuator)

| Field | Std |
|-------|:---:|
| `attenuator_transmission` | NXmx |

### `/entry/instrument/detector` (NXdetector)

| Field | Std | Units |
|-------|:---:|-------|
| `depends_on` | NXmx | → `transformations/rot3` |
| `beam_center_x`, `beam_center_y` | NXmx | pixel |
| `distance` | NXmx | m |
| `count_time`, `frame_time` | NXmx | s |
| `sensor_thickness` | NXmx | m |
| `sensor_material` | NXmx | |
| `description` | NXmx | |
| `threshold_energy` | NXmx | eV (EIGER; written only for a single channel) |
| `x_pixel_size`, `y_pixel_size` | base | m |
| `serial_number` | base | |
| `bit_depth_readout` | NXmx | |
| `saturation_value` | NXmx | |
| `flatfield_applied` | NXmx | |
| `pixel_mask`, `pixel_mask_applied` | NXmx | `pixel_mask` is `[y, x]`, hard-linked from `detectorSpecific/pixel_mask` |
| `countrate_correction_applied` | NXmx | |
| `number_of_cycles` | base | frame-summation factor |

### `/entry/instrument/detector/transformations` (NXtransformations)

The NXtransformations *mechanism* (the `depends_on` chain, `transformation_type`, `vector`,
`offset` attributes) is standard. The axis **names** follow the PyFAI PONI convention chosen by
Jungfraujoch (see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)):

| Axis | Type | Units | Depends on |
|------|------|-------|-----------|
| `translation` | translation | m | `.` |
| `rot1` | rotation | rad | `translation` |
| `rot2` | rotation | rad | `rot1` |
| `rot3` | rotation | rad | `rot2` |

### `/entry/instrument/detector/module` (NXdetector_module)

`data_origin`, `data_size`, `fast_pixel_direction`, `slow_pixel_direction`, `module_offset` — all
NXmx (`fast/slow_pixel_direction` and `module_offset` carry transformation attributes).

### `/entry/sample` (NXsample)

| Field | Std | Units / notes |
|-------|:---:|-------|
| `name` | NXmx | |
| `depends_on` | NXmx | points at the last goniometer / grid-scan axis, or `.` for stills |
| `temperature` | NXmx | K |
| `transformations/` (NXtransformations) | NXmx | rotation axis (e.g. `omega`) or grid-scan translation; hard-linked as `/entry/sample/goniometer` |
| `unit_cell` | base | `[a, b, c, α, β, γ]` |
| `ub_matrix` | base | `[1, 3, 3]`, Angstrom⁻¹ |

For a rotation scan the goniometer axis is written as a per-image angle array `<axis>` plus
`<axis>_end`, scalar `<axis>_range_average`, `<axis>_range_total`, and for helical scans
`<axis>_helical_x/_y/_z`. These extra goniometer datasets beyond the bare axis array are Jungfraujoch
conveniences.

### `/entry/data` (NXdata)

`data` (3-D image stack, `[n_images, y, x]`) with `image_nr_low` / `image_nr_high` attributes.
In legacy mode this group instead contains one external link `data_000001`, … per data file.

## 4. Extensions beyond NXmx

Everything in this section is **outside the NXmx standard**. Each group is declared with
`NX_class = NXcollection` (the NeXus-sanctioned container for non-standardised content) unless noted.
The per-image arrays are indexed by image number, padded to the run length and filled with a
sentinel (`NaN` for floats, `-1`/`0` for integer indices) where a quantity is absent.

### 4.1 `/entry/MX` — spot finding and indexing (CXI-style)

The flagship extension. Spot ("peak") properties are stored as fixed-size `[n_images, max_spots]`
arrays (CXI layout, recognised by CrystFEL); scalar-per-image quantities as `[n_images]` vectors.
In legacy/VDS mode these live in the data files and are linked/virtual-stacked into the master.

**Per-spot arrays `[n_images, max_spots]`:**

| Dataset | Units | Meaning | Indexing only |
|---------|-------|---------|:---:|
| `peakXPosRaw`, `peakYPosRaw` | pixel | spot position (raw detector frame) | |
| `peakTotalIntensity` | photons | spot intensity | |
| `peakIceRingRes` | | spot lies in an ice-ring resolution band | |
| `peakH`, `peakK`, `peakL` | | Miller indices of the (indexed) spot | ✓ |
| `peakDistEwaldSphere` | Å⁻¹ | distance of the spot from the Ewald sphere | ✓ |
| `peakIndexed` | | spot fits the indexing solution | ✓ |
| `peakLattice` | | lattice the spot belongs to (`-1` = unindexed) | ✓ |

**Per-image vectors `[n_images]`:**

| Dataset | Units | Meaning |
|---------|-------|---------|
| `nPeaks` | | number of spots stored for the image (CXI) |
| `strongPixels` | | strong-pixel count (first spot-finding stage) |
| `peakCountUnfiltered` | | spots found before filtering |
| `peakCountLowRes` | | low-resolution spots |
| `peakCountIceRingRes` | | spots inside ice-ring bands |
| `peakCountIndexed` | | spots fitting the indexing solution |
| `imageIndexed` | | image was indexed (0/1) |
| `indexingLatticeCount` | | number of lattices found for the image |
| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A*, Table 3.1.3.1) |
| `bravaisLattice` | | Bravais lattice short code, e.g. `aP`, `mC`, `oF`, `tI`, `hP`, `hR`, `cF` |
| `profileRadius` | Å⁻¹ | crystal profile radius |
| `mosaicity` | deg | mosaicity estimate |
| `bFactor` | Ų | per-image B-factor estimate |
| `resolutionEstimate` | Å | diffraction resolution estimate |
| `integratedReflections` | | number of integrated reflections |
| `bkgEstimate` | photons | mean background in the 3–5 Å resolution band |
| `beam_corr_x`, `beam_corr_y` | pixel | beam-center correction applied during processing |
| `imageScaleFactor` | | on-the-fly per-image scale factor *g* |
| `imageScaleCC` | | on-the-fly scaling correlation coefficient |
| `imageScaleMosaicity` | deg | scaling-model mosaicity |
| `imageScaleBFactor` | Ų | scaling-model B-factor |

**Per-image lattices:** `latticeIndexed` `[n_images, 9]` (Å) — the real-space lattice (flattened
3×3); `latticeIndexedExtra` `[n_images, max_extra_lattices, 9]` (Å) — additional orientation
variants.

**Run-level summaries** (written into the master `/entry/MX` at finalisation):

| Dataset | Units | Meaning |
|---------|-------|---------|
| `indexing_algorithm` | | `FFBIDX` / `FFT (CUDA)` / `FFT (FFTW)` |
| `geom_refinement_algorithm` | | e.g. `beam_center` |
| `rotationLatticeIndexed` | Å | whole-run rotation-indexing lattice (`[9]`) |
| `rotationLatticeIndexedExtra` | Å | additional whole-run lattices (`[m, 9]`) |
| `rotationLatticeNiggliClass` | | Niggli class of the run lattice |
| `imageIndexedMean` | | mean indexing rate over the run |
| `bkgEstimateMean` | photons | mean background over the run |
| `indexedLatticeCount` | | per-image lattice count summary (master). *Note: data files use `indexingLatticeCount`; readers accept either.* |

CrystFEL can read the spots directly with:

```
peak_list = /entry/MX
peak_list_type = cxi
```

### 4.2 `/entry/reflections` — integrated reflections (NXreflections)

Integrated reflections are stored **per image** as
`/entry/reflections/image_NNNNNN` groups, each declared `NX_class = NXreflections`. The columns map
mostly onto the standard
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class:

| Dataset | Units | NXreflections | Meaning |
|---------|-------|:-------------:|---------|
| `h`, `k`, `l` | | standard | Miller indices |
| `d` | Å | standard | resolution |
| `int_sum` | photons | standard | integrated intensity (summation) |
| `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) |
| `background_mean` | photons | standard | mean background under the peak |
| `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` |
| `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) |
| `observed_frame` | | standard | image number of the reflection |
| `lp` | | standard | Lorentz–polarization factor (stored as `1/rlp`) |
| `partiality` | | standard | recorded fraction of the reflection |
| `delta_phi` | deg | **extension** | XDS Δφ: offset from the centre of the current frame |
| `zeta` | | **extension** | Lorentz ζ factor (reciprocal-space geometry term) |
| `image_scale_corr` | | **extension** | per-image scale correction; `I_true = image_scale_corr · int_sum` |

In the master file these per-image groups are exposed through `/entry/reflections` external links
(VDS/integrated formats).

### 4.3 `/entry/azint` — azimuthal integration

| Dataset | Shape | Units | Meaning |
|---------|-------|-------|---------|
| `bin_to_q` | `[φ_bins, q_bins]` | Å⁻¹ | q value of each bin |
| `bin_to_two_theta` | `[φ_bins, q_bins]` | deg | 2θ of each bin |
| `bin_to_phi` | `[φ_bins, q_bins]` | deg | azimuthal angle of each bin |
| `image` | `[n_images, φ_bins, q_bins]` | | per-image integrated profile (NaN for empty bins) |
| `image_std` | `[n_images, φ_bins, q_bins]` | | per-bin standard deviation |
| `image_count` | `[n_images, φ_bins, q_bins]` | | pixels contributing per bin |
| `map` | `[y, x]` | | pixel→bin mapping (master file only) |

### 4.4 `/entry/roi/<roi_name>` — regions of interest

One sub-group per configured ROI, each with `[n_images]` vectors:

| Dataset | Meaning |
|---------|---------|
| `max` | maximum pixel value in the ROI |
| `sum` | sum of pixel values |
| `sum_sq` | sum of squared pixel values |
| `npixel` | number of valid pixels |
| `x`, `y` | intensity-weighted centroid |

### 4.5 `/entry/image` — per-image pixel statistics

`[n_images]` vectors: `max_value`, `min_value` (viable min/max, excluding error/saturated pixels),
`error_pixels`, `saturated_pixels`, `pixel_sum`. Surfaced in the master file under `/entry/image`.

### 4.6 `/entry/profiling` — per-image timing

`[n_images]` vectors in seconds: `spotFindingTime`, `indexingTime`, `integrationTime`,
`refinementTime`, `processingTime`, `braggPredictionTime`, `preprocessingTime`, `compressionTime`,
`azIntTime`, `indexAnalysisTime`, `imageScaleTime`.

### 4.7 `/entry/detector` — acquisition diagnostics (data file)

A convenience NXcollection in the data file (note: distinct from the standard
`/entry/instrument/detector`). In **integrated** format these datasets are written under
`/entry/instrument/detector/detectorSpecific` instead.

| Dataset | Meaning |
|---------|---------|
| `timestamp`, `exptime` | per-image timestamp and exposure time |
| `number` | image number (original number if image rejection was used) |
| `det_info` | JUNGFRAU debug field |
| `storage_cell_image` | storage-cell number |
| `rcv_delay`, `rcv_free_send_buffers` | receiver internal diagnostics |
| `packets_expected`, `packets_received` | UDP packets per image |
| `data_collection_efficiency_image` | received / expected packet ratio |

### 4.8 `/entry/xfel` — pulsed-source metadata

`[n_images]` vectors `pulseID` and `eventCode`, written for pulsed sources (e.g. SwissFEL).

### 4.9 Other collections

| Path | Class | Content |
|------|-------|---------|
| `/entry/instrument/detector/detectorSpecific` | NXcollection | Dectris-style detector metadata + Jungfraujoch fields: `x_pixels_in_detector`, `y_pixels_in_detector`, `nimages`, `ntrigger`, `nimages_collected`, `nimages_written`, `data_collection_efficiency`, `max_receiver_delay`, `storage_cell_number`, `storage_cell_delay` [ns], `software_git_commit`, `software_git_date`, `jfjoch_release`, `jfjoch_writer_release`, `summation_mode`, `detect_ice_rings`, `gain_file_names`, `data_reduction_factor_serialmx`, `adu_histogram/`, `data_collection_efficiency_image` |
| `/entry/instrument/detector/calibration` | NXcollection | per-channel pedestal / calibration images (bitshuffle-compressed) |
| `/entry/instrument/fluorescence` | NXcollection | XRF spectrum: `energy` [eV], `data` |
| `/entry/user` | NXcollection | scalar values supplied under `header_appendix.hdf5` |

### 4.10 Non-standard fields inside the NXmx detector group

A few extension scalars are written *inside* the otherwise-standard `/entry/instrument/detector`
group for compatibility with existing tooling:

| Field | Units | Meaning |
|-------|-------|---------|
| `detector_distance` | m | duplicate of `distance` (Dectris/Neggia compatibility) |
| `detector_number` | | detector identifier (Dectris convention) |
| `error_value` | | masked/error pixel sentinel (NXmx standard would be `underload_value`) |
| `bit_depth_image` | | stored image bit depth (NXmx standard is `bit_depth_readout`) |
| `acquisition_type` | | always `triggered` (Dectris convention) |
| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied |
| `jungfrau_conversion_factor` | eV | conversion factor |
| `geometry_transformation_applied` | | module→full-detector geometry applied |

## 5. Notes

* **Units** are written as the HDF5 `units` attribute on the dataset (e.g. `m`, `eV`, `deg`,
  `Angstrom`, `Angstrom^-1`, `Angstrom^2`, `pixel`, `s`).
* **Sentinels.** Missing per-image values are `NaN` (floats) or `-1`/`0` (integer indices); image
  pixels use `INTx_MIN` / `UINTx_MAX`.
* **Master vs data file.** In legacy/VDS formats the analysis arrays physically live in the data
  files; the master file links to them (external links in legacy, virtual datasets in VDS). In the
  integrated format there are no data files and everything is in one place.
* **CXI / CrystFEL.** `/entry/MX` follows the CXI peak-list convention; see
  [CXI file format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf).