leonarski_f
749db470ca
Build Packages / build:rpm (rocky9) (push) Successful in 19m56s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m57s
Build Packages / build:windows:cuda (push) Successful in 19m18s
Build Packages / build:viewer-tgz:cpu (push) Successful in 14m48s
Build Packages / build:viewer-tgz:cuda (push) Successful in 16m18s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 14m19s
Build Packages / build:rugnux:windows (push) Successful in 10m34s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 8m49s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 20m55s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 17m4s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 20m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 19m15s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 24m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 20m32s
Build Packages / build:rpm (rocky8) (push) Successful in 23m39s
Build Packages / Generate python client (push) Successful in 46s
Build Packages / Build documentation (push) Successful in 1m45s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 11m3s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m30s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m10s
Build Packages / XDS test (neggia plugin) (push) Successful in 10m17s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m12s
Build Packages / DIALS test (push) Successful in 20m12s
* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
* The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count.
* `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
* `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so.
* The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.
Reviewed-on: #74
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
577 lines
34 KiB
Markdown
577 lines
34 KiB
Markdown
# HDF5 / NeXus data format
|
||
|
||
Jungfraujoch stores images and on-the-fly analysis results in HDF5 files that aim to be
|
||
[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html)-compliant. On top of the
|
||
NXmx application definition, Jungfraujoch records a substantial amount of *derived* metadata
|
||
(spot finding, indexing, integration, azimuthal integration, per-image statistics, timing). These
|
||
extra entries do not exist in NXmx and are documented here so that the layout is unambiguous and
|
||
reusable.
|
||
|
||
This page documents the **file layout and the data fields**. The operational behaviour of the
|
||
writer (running, republishing, file finalisation) is described in
|
||
[jfjoch_writer](JFJOCH_WRITER.md). The wire format that feeds the writer is described in
|
||
[CBOR messages](CBOR.md); fields below frequently correspond one-to-one to CBOR message fields, and
|
||
that document is a useful companion for their meaning.
|
||
|
||
```{contents} On this page
|
||
:local:
|
||
:depth: 2
|
||
```
|
||
|
||
## 1. Motivation: derived metadata and FAIR data
|
||
|
||
The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep
|
||
them findable, accessible, interoperable and reusable (FAIR). Jungfraujoch is used for both
|
||
**rotation** macromolecular crystallography (single- and multi-crystal, including fine-sliced and
|
||
helical scans) and **serial** crystallography (stills, grid scans); the same concerns apply to both:
|
||
|
||
* **Findability.** Raw diffraction images carry almost no descriptive metadata about *content*.
|
||
Quantities such as background level, number of diffraction spots, or indexing outcome let a user
|
||
judge the quality and relevance of a dataset *before* inspecting the raw images.
|
||
* **Accessibility at scale.** A single experiment can span tens to hundreds of terabytes. Standard
|
||
retrieval (e.g. HTTP) makes a dataset *available* but not *inspectable* — users would otherwise
|
||
have to download a large fraction of the data just to decide whether it is useful. Compact
|
||
derived representations make discovery, assessment and reuse feasible.
|
||
|
||
Because Jungfraujoch couples acquisition with real-time analysis used to *steer* experiments,
|
||
transparency and reproducibility of that analysis matter. As a minimum the writer therefore
|
||
preserves spot-finding and indexing results together with the filters that were applied, and it can
|
||
retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse.
|
||
|
||
### Two complementary layouts: per-image spots vs. a reflection table
|
||
|
||
Jungfraujoch stores analysis products in two shapes, matching how each is accessed.
|
||
|
||
**Per-image spot finding / indexing.** Spot finding and indexing are inherently *image-centric* —
|
||
the natural query is "give me the spots for image *n*" — and this holds for serial stills and for
|
||
rotation frames alike. For these products Jungfraujoch adopts a layout similar to the
|
||
[Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the convention
|
||
understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties (position,
|
||
intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by image number,
|
||
with each image allocated room for up to a predefined maximum number of spots. These dense arrays
|
||
are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are retrieved
|
||
without traversing variable-length structures. The cost is some storage overhead for unused slots
|
||
(padded with sentinels), which is acceptable for the access pattern.
|
||
|
||
**Integrated reflections.** Integrated intensities are naturally a *dataset-wide* table, which is
|
||
exactly the model of the NeXus
|
||
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class.
|
||
This fits rotation crystallography well, and Jungfraujoch uses NXreflections for its integration
|
||
results (see §4.2 below). We deliberately do *not* force spot finding/indexing into a single
|
||
experiment-wide table: across the hundreds of thousands of patterns typical of serial — or
|
||
fine-sliced rotation — experiments, that would require aggregating the whole experiment before the
|
||
spots of one image can be read. We encourage the community to develop standardised NeXus application
|
||
definitions for image-centric crystallography products that combine NeXus interoperability with the
|
||
access patterns and scale of modern high-throughput experiments.
|
||
|
||
## 2. File layout
|
||
|
||
A run is written as one **master file** plus, depending on the format, one or more **data files**:
|
||
|
||
```
|
||
<prefix>_master.h5 # NXmx master file (metadata + links / virtual datasets)
|
||
<prefix>_data_000001.h5 # data file: images + per-image analysis
|
||
<prefix>_data_000002.h5
|
||
...
|
||
```
|
||
|
||
The master file is produced by `writer/HDF5NXmx.cpp`; data files by `writer/HDF5DataFile.cpp` and
|
||
its plugins (`writer/HDF5DataFilePlugin*.cpp`). Files are written to a temporary `*.<random>.tmp`
|
||
name and renamed on successful close.
|
||
|
||
Three master-file variants exist (set via `file_format`):
|
||
|
||
| Format | Value | Master ↔ data linking |
|
||
|--------|:-----:|------------------------|
|
||
| **NXmxLegacy** | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. |
|
||
| **NXmxVDS** (default) | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. |
|
||
| **NXmxIntegrated** | 3 | No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format. |
|
||
|
||
In legacy/VDS mode, image-indexed analysis arrays live in the **data files** and are exposed in the
|
||
master file through external links or virtual datasets; in integrated mode they are written
|
||
directly into the single file. Throughout this document a "✓ in master" column marks entries that
|
||
are visible (directly or via link/VDS) from the master file.
|
||
|
||
Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or
|
||
bitshuffle + Zstd. Signed integer image datasets carry `INTx_MIN` as the HDF5 fill value (the
|
||
"masked / no-data" sentinel); unsigned ones are left at HDF5's own fill of 0, because every unsigned
|
||
code is a legitimate count. In the master's virtual dataset the fill is the error marker for both,
|
||
so a data file missing beside a VDS master reads as masked rather than as zero counts.
|
||
|
||
> **Signed images and the Neggia XDS plugin.** Neggia dispatches on the size of the pixel in bytes
|
||
> and always casts to an unsigned type, consulting the signedness of the data only for the pixel
|
||
> mask. A **signed 16-bit** image is therefore read wrongly: a count of `-2` reaches XDS as `65534`,
|
||
> and the `-32768` error marker as `32768`. Signed 32-bit happens to degrade safely, because every
|
||
> negative value ends up above `INT32_MAX` and is mapped to `-1`. Use the Jungfraujoch XDS plugin,
|
||
> or the Global Phasing build of Durin, for signed data — see
|
||
> [Integration with MX data processing software](SOFTWARE_INTEGRATION.md).
|
||
|
||
### Reprocessing output: `<prefix>_process.h5`
|
||
|
||
The offline reprocessing tool [`rugnux`](TOOLS.md) (`rugnux/rugnux_cli.cpp`) re-runs the
|
||
full analysis pipeline (spot finding, indexing, refinement, integration, scaling) on an existing
|
||
dataset and writes its results to a master file named **`<prefix>_process.h5`**. This file uses the
|
||
**integrated** format, but instead of copying the images its `/entry/data/data` is a *virtual
|
||
dataset that links back to the original image files* (`hdf5_source_data` →
|
||
`NXmx::LinkToData_ProcessingVDS`). The result is a compact, self-describing companion file that
|
||
holds *all* the derived analysis (everything in §4) plus a virtual view
|
||
of the raw images — without duplicating terabytes of data.
|
||
|
||
This is a particularly FAIR-friendly artefact: it can be shared or archived alongside (or instead
|
||
of) the raw data to convey what is in a dataset and how it processed, while the `/entry/data/data`
|
||
VDS still resolves to the original images when they are available. `rugnux` can also process
|
||
an equally-spaced *subset* of images (start/end/stride), producing a down-sampled reference set.
|
||
|
||
## 3. NXmx-standard content
|
||
|
||
The entries below are part of, or valid base classes for, the
|
||
[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html) application definition.
|
||
"NXmx" = listed in the application definition; "base" = a valid field of the relevant NeXus base
|
||
class (`NXdetector`, `NXsample`, `NXsource`) but not in the NXmx required/recommended subset.
|
||
|
||
### `/entry` (NXentry)
|
||
|
||
| Field | Std | Notes |
|
||
|-------|:---:|-------|
|
||
| `definition` | NXmx | value `"NXmx"` |
|
||
| `start_time` | NXmx | arming time |
|
||
| `end_time`, `end_time_estimated` | NXmx | approximate end time |
|
||
|
||
File-level HDF5 attributes `file_name`, `file_time`, `HDF5_Version` are also set.
|
||
|
||
### `/entry/source` (NXsource), `/entry/instrument` (NXinstrument)
|
||
|
||
| Field | Std | Units |
|
||
|-------|:---:|-------|
|
||
| `source/name`, `source/type` | NXmx / base | |
|
||
| `source/current` | base | A |
|
||
| `instrument/name` | NXmx | |
|
||
|
||
### `/entry/instrument/beam` (NXbeam)
|
||
|
||
| Field | Std | Units |
|
||
|-------|:---:|-------|
|
||
| `incident_wavelength` | NXmx | angstrom |
|
||
| `incident_wavelength_spread` | NXmx | angstrom (only if polychromatic) |
|
||
| `total_flux` | NXmx | Hz |
|
||
|
||
### `/entry/instrument/attenuator` (NXattenuator)
|
||
|
||
| Field | Std |
|
||
|-------|:---:|
|
||
| `attenuator_transmission` | NXmx |
|
||
|
||
### `/entry/instrument/detector` (NXdetector)
|
||
|
||
| Field | Std | Units |
|
||
|-------|:---:|-------|
|
||
| `depends_on` | NXmx | → `transformations/rot3` |
|
||
| `beam_center_x`, `beam_center_y` | NXmx | pixel (0.0 = centre of the first pixel, see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)) |
|
||
| `distance` | NXmx | m |
|
||
| `count_time`, `frame_time` | NXmx | s |
|
||
| `sensor_thickness` | NXmx | m |
|
||
| `sensor_material` | NXmx | |
|
||
| `description` | NXmx | |
|
||
| `threshold_energy` | NXmx | eV (EIGER; written only for a single channel) |
|
||
| `x_pixel_size`, `y_pixel_size` | base | m |
|
||
| `serial_number` | base | |
|
||
| `bit_depth_readout` | NXmx | bit depth of the **stored image**, not of the detector electronics - see below |
|
||
| `saturation_value` | NXmx | highest valid value. Read inclusively by NXmx, by the DIALS `trusted_range` and by the XDS `OVERLOAD` parameter; a saturated pixel carries the value one above it |
|
||
| `underload_value` | NXmx | lowest valid value: `0` for an unsigned image, `INTx_MIN + 1` for a signed one |
|
||
| `flatfield_applied` | NXmx | |
|
||
| `pixel_mask`, `pixel_mask_applied` | NXmx | `pixel_mask` is `[y, x]`, hard-linked from `detectorSpecific/pixel_mask` |
|
||
| `countrate_correction_applied` | NXmx | |
|
||
| `number_of_cycles` | base | frame-summation factor |
|
||
|
||
#### Why `bit_depth_readout` is the image depth
|
||
|
||
NXmx defines only `bit_depth_readout`, "how many bits the electronics record per pixel", and has no
|
||
field for the depth of the image actually stored. The two differ whenever summation is used: the
|
||
readout stays at the detector's native width while the summed image must be wider to hold the sum.
|
||
|
||
Jungfraujoch writes the **stored image depth** into `bit_depth_readout` (and the identical value
|
||
into the non-standard `bit_depth_image`). The electronic value is a constant of the detector and
|
||
tells a data consumer nothing, whereas readers do use `bit_depth_readout` as the width of the
|
||
stored pixel — DIALS, for instance, derives its masking markers from it and cannot read a 32-bit
|
||
image without it. Writing the electronic value there would therefore mislead exactly in the case
|
||
where the two differ.
|
||
|
||
Note that `bit_depth_readout` gives the width only. The **sign** is carried solely by the HDF5
|
||
element type of `/entry/data/data` (and, on the wire, by `image_dtype`); there is no NXmx field for
|
||
it.
|
||
|
||
### `/entry/instrument/detector/transformations` (NXtransformations)
|
||
|
||
The NXtransformations *mechanism* (the `depends_on` chain, `transformation_type`, `vector`,
|
||
`offset` attributes) is standard. The axis **names** follow the PyFAI PONI convention chosen by
|
||
Jungfraujoch (see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)):
|
||
|
||
| Axis | Type | Units | Vector | Depends on |
|
||
|------|------|-------|--------|-----------|
|
||
| `rot3` | rotation | rad | `(0, 0, -1)` | `.` |
|
||
| `rot2` | rotation | rad | `(1, 0, 0)` | `rot3` |
|
||
| `rot1` | rotation | rad | `(0, -1, 0)` | `rot2` |
|
||
| `translation` | translation | m | unit vector along the sample→PONI direction | `rot1` |
|
||
|
||
`/entry/instrument/detector/depends_on` is `translation`, and the module's `fast_pixel_direction`,
|
||
`slow_pixel_direction` and `module_offset` depend on it in turn. A chain is applied innermost-first,
|
||
so reading it outwards the detector is placed at its distance and beam centre and *then* tilted about
|
||
the sample — which is what makes a tilt pivot about the crystal rather than about the panel corner.
|
||
The `vector` values are in NXmx's **McStas** frame, which is Jungfraujoch's internal frame with x and
|
||
y negated.
|
||
|
||
The beam centre is encoded in `translation` (its offset from the sample), not only in the
|
||
informational `beam_center_x`/`beam_center_y` fields. In a `_process.h5` written by rugnux these axes
|
||
carry the **refined** detector geometry — the refined beam centre folds into `translation` and the
|
||
refined tilt into `rot1`/`rot2`/`rot3`; the broker writes the user-provided geometry unchanged.
|
||
|
||
### `/entry/instrument/detector/module` (NXdetector_module)
|
||
|
||
`data_origin`, `data_size`, `fast_pixel_direction`, `slow_pixel_direction`, `module_offset` — all
|
||
NXmx (`fast/slow_pixel_direction` and `module_offset` carry transformation attributes).
|
||
|
||
### `/entry/sample` (NXsample)
|
||
|
||
| Field | Std | Units / notes |
|
||
|-------|:---:|-------|
|
||
| `name` | NXmx | |
|
||
| `depends_on` | NXmx | points at the innermost axis of the sample chain, or `.` for stills |
|
||
| `temperature` | NXmx | K |
|
||
| `transformations/` (NXtransformations) | NXmx | the sample chain, written in mounting order; hard-linked as `/entry/sample/goniometer` |
|
||
| `unit_cell` | base | `[a, b, c, α, β, γ]` |
|
||
| `ub_matrix` | base | `[1, 3, 3]`, Angstrom⁻¹ |
|
||
|
||
The chain is written from the base outwards, so the innermost axis — the one `depends_on` names — is
|
||
the one nearest the sample. It may hold, in that order: the grid-scan translations `grid_scan_x` and
|
||
`grid_scan_y`, the spindle, and a Smargon head's `chi` and `phi`. A grid scan and a goniometer axis
|
||
are **not** alternatives; both can be present.
|
||
|
||
A **Smargon head position is told apart from the spindle** by the `equipment_component` attribute,
|
||
which is `"smargon"` on `chi` and `phi` and absent on the spindle. This is load-bearing: a spindle can
|
||
itself be named `phi`, and without the attribute a reader would take a head position for the scan
|
||
axis. `chi` and `phi` are written with one value per image even though neither turns, because a
|
||
reader takes the image count from the innermost axis of the chain: written as scalars, a still
|
||
recorded at a head position would read back as a single image however many were collected.
|
||
|
||
For a rotation scan the goniometer axis carries, beyond the per-image angle array `<axis>`, the
|
||
Jungfraujoch conveniences `<axis>_end`, scalar `<axis>_range_average` and `<axis>_range_total`, and
|
||
for helical scans `<axis>_helical_x/_y/_z`.
|
||
|
||
### `/entry/data` (NXdata)
|
||
|
||
`data` (3-D image stack, `[n_images, y, x]`) with `image_nr_low` / `image_nr_high` attributes.
|
||
In legacy mode this group instead contains one external link `data_000001`, … per data file.
|
||
|
||
## 4. Extensions beyond NXmx
|
||
|
||
Everything in this section is **outside the NXmx standard**. Each group is declared with
|
||
`NX_class = NXcollection` (the NeXus-sanctioned container for non-standardised content) unless noted.
|
||
The per-image arrays are indexed by image number, padded to the run length and filled with a
|
||
sentinel (`NaN` for floats, `-1`/`0` for integer indices) where a quantity is absent.
|
||
|
||
### 4.1 `/entry/MX` — spot finding and indexing (CXI-style)
|
||
|
||
The flagship extension. Spot ("peak") properties are stored as fixed-size `[n_images, max_spots]`
|
||
arrays (CXI layout, recognised by CrystFEL); scalar-per-image quantities as `[n_images]` vectors.
|
||
In legacy/VDS mode these live in the data files and are linked/virtual-stacked into the master.
|
||
|
||
**Per-spot arrays `[n_images, max_spots]`:**
|
||
|
||
| Dataset | Units | Meaning | Indexing only |
|
||
|---------|-------|---------|:---:|
|
||
| `peakXPosRaw`, `peakYPosRaw` | pixel | spot position (raw detector frame) | |
|
||
| `peakTotalIntensity` | photons | spot intensity | |
|
||
| `peakIceRingRes` | | spot lies in an ice-ring resolution band | |
|
||
| `peakH`, `peakK`, `peakL` | | Miller indices of the (indexed) spot | ✓ |
|
||
| `peakDistEwaldSphere` | Å⁻¹ | distance of the spot from the Ewald sphere | ✓ |
|
||
| `peakIndexed` | | spot fits the indexing solution | ✓ |
|
||
| `peakLattice` | | lattice the spot belongs to (`-1` = unindexed) | ✓ |
|
||
|
||
**Per-image vectors `[n_images]`:**
|
||
|
||
| Dataset | Units | Meaning |
|
||
|---------|-------|---------|
|
||
| `nPeaks` | | number of spots stored for the image (CXI) |
|
||
| `strongPixels` | | strong-pixel count (first spot-finding stage) |
|
||
| `peakCountUnfiltered` | | spots found before filtering |
|
||
| `peakCountLowRes` | | low-resolution spots |
|
||
| `peakCountIceRingRes` | | spots inside ice-ring bands |
|
||
| `peakCountIceRingControl` | | spots in the ice-free flanks beside those bands, rescaled to their q width - the control for the count above (their ratio, pooled over the run, is the spot-based ice indicator) |
|
||
| `peakCountIndexed` | | spots fitting the indexing solution |
|
||
| `imageIndexed` | | image was indexed (0/1) |
|
||
| `indexingLatticeCount` | | number of lattices found for the image |
|
||
| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A* (2016), Vol. A, [Table 3.1.3.1](https://onlinelibrary.wiley.com/iucr/itc/Ac/ch3o1v0001/table3o1o3o1.pdf)) |
|
||
| `bravaisLattice` | | Bravais lattice short code, e.g. `aP`, `mC`, `oF`, `tI`, `hP`, `hR`, `cF` |
|
||
| `profileRadius` | Å⁻¹ | crystal profile radius |
|
||
| `mosaicity` | deg | mosaicity estimate |
|
||
| `bFactor` | Ų | per-image B-factor estimate |
|
||
| `resolutionEstimate` | Å | resolution the merged data are predicted to reach, from this image's spots alone |
|
||
| `integratedReflections` | | number of integrated reflections |
|
||
| `bkgEstimate` | photons | mean background in the 3–5 Å resolution band |
|
||
| `iceRingScore` | ratio | strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice) |
|
||
| `beam_corr_x`, `beam_corr_y` | pixel | beam-center correction applied during processing |
|
||
| `imageScaleFactor` | | on-the-fly per-image scale factor *g* |
|
||
| `imageScaleCC` | | on-the-fly scaling correlation coefficient |
|
||
| `imageScaleMosaicity` | deg | scaling-model mosaicity |
|
||
| `sweepQuality` | | why this image's stretch of the sweep was flagged — see below |
|
||
|
||
**Per-image lattices:** `latticeIndexed` `[n_images, 9]` (Å) — the real-space lattice (flattened
|
||
3×3); `latticeIndexedExtra` `[n_images, max_extra_lattices, 9]` (Å) — additional orientation
|
||
variants.
|
||
|
||
**Run-level summaries** (written into the master `/entry/MX` at finalisation):
|
||
|
||
| Dataset | Units | Meaning |
|
||
|---------|-------|---------|
|
||
| `indexing_algorithm` | | `FFBIDX` / `FFT (CUDA)` / `FFT (FFTW)` |
|
||
| `geom_refinement_algorithm` | | e.g. `beam_center` |
|
||
| `rotationLatticeIndexed` | Å | whole-run rotation-indexing lattice (`[9]`) |
|
||
| `rotationLatticeIndexedExtra` | Å | additional whole-run lattices (`[m, 9]`) |
|
||
| `rotationLatticeNiggliClass` | | Niggli class of the run lattice |
|
||
| `imageIndexedMean` | | mean indexing rate over the run |
|
||
| `bkgEstimateMean` | photons | mean background over the run |
|
||
| `iceRingScoreMean` | ratio | mean `iceRingScore` over the run — the single "how icy was this dataset" number (1 = no ice) |
|
||
| `indexedLatticeCount` | | per-image lattice count summary (master). *Note: data files use `indexingLatticeCount`; readers accept either.* |
|
||
| `reindexMatrix` | | change of basis from the setting the per-image data are in to the setting of `/entry/sample/unit_cell` (`[9]`, `int32`, flattened 3×3, row major) — see below |
|
||
|
||
**Reindex matrix.** The per-image `h`, `k`, `l` and `latticeIndexed` are written as each image is
|
||
processed, in the setting that image was *indexed* in. The space group is only settled afterwards, by
|
||
the merge, and settling it can re-seat the lattice into the group's conventional setting — so
|
||
`/entry/sample/unit_cell`, `/entry/sample/space_group_number` and `rotationLatticeIndexed` can be in a
|
||
different setting from the per-image data beside them. `reindexMatrix` **M** is the integral change of
|
||
basis between the two: `hkl_cell = M · hkl_written`, and the same **M** takes each per-image lattice
|
||
across (`latticeIndexedExtra` is not re-seated and stays as indexed). It is **absent** when the two
|
||
settings are the same one, which is the identity — as it is on every file written before rugnux
|
||
recorded it. The Jungfraujoch reader applies it, so everything it hands out is already in the cell's
|
||
setting; a third-party reader that ignores it will index the reflections in the wrong frame whenever
|
||
the dataset is present. Written by the offline `rugnux` path only — the broker never re-seats a
|
||
lattice — and not carried on the CBOR stream, in the same way as the other offline-only fields.
|
||
|
||
A `--model` run can leave the merged reflection files (`.mtz`/`.cif`/`.hkl`) in a *different* frame
|
||
from the `_process.h5` beside them: the model settles the enantiomorph, and the alternative indexing
|
||
where nothing else did, and those choices are applied to the merged reflections as they are written.
|
||
The process file is not rewritten — its per-image reflections went to disk as they were integrated —
|
||
so it keeps the space group the run itself determined and stays self-consistent with its own data.
|
||
The two frames describe the same measurements; an enantiomorphic pair merges identically, having the
|
||
same Laue class and the same absences.
|
||
|
||
**Sweep quality.** `sweepQuality` `[n_images]` (`uint8`) says why the stretch of the sweep this
|
||
image belongs to was flagged as delivering much less than the rest of the run: **0** means it was
|
||
not, and any other value is a **1-based index into `sweepQualityReasons`**, a string vector written
|
||
beside it that carries the whole vocabulary, so the codes can be read without this source. The
|
||
vocabulary is closed and stable — a code is never renamed and never reused — and currently reads
|
||
`no_diffraction`, `crystal_out_of_beam`, `weak_diffraction`, `loss_of_centring`, `radiation_damage`;
|
||
[the rugnux documentation](RUGNUX.md#sweep-quality-and-the-reason-vocabulary) defines what each one
|
||
means. Both datasets are **absent** unless the sweep-quality diagnostic ran, which needs scaling and
|
||
merging; their absence therefore means "not looked for", *not* "every image clean". Written by the
|
||
offline `rugnux` path only — the broker does not merge — and not carried on the CBOR stream, in the
|
||
same way as the other offline-only fields (`space_group_number`, the refined geometry). Nothing is
|
||
excluded from processing on the strength of it. The condensed, dataset-wide form of the same finding
|
||
is in `<prefix>_report.txt`.
|
||
|
||
CrystFEL can read the spots directly with:
|
||
|
||
```
|
||
peak_list = /entry/MX
|
||
peak_list_type = cxi
|
||
```
|
||
|
||
### 4.2 `/entry/reflections` — integrated reflections (NXreflections)
|
||
|
||
Integrated reflections are stored **per image** as
|
||
`/entry/reflections/image_NNNNNN` groups, each declared `NX_class = NXreflections`. The columns map
|
||
mostly onto the standard
|
||
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class:
|
||
|
||
| Dataset | Units | NXreflections | Meaning |
|
||
|---------|-------|:-------------:|---------|
|
||
| `h`, `k`, `l` | | standard | Miller indices |
|
||
| `d` | Å | standard | resolution |
|
||
| `int_sum` | photons | standard | integrated intensity (summation) |
|
||
| `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) |
|
||
| `background_mean` | photons | standard | mean background under the peak |
|
||
| `background_variance` | photons² | non-standard | non-signal part of σ², carried to the merge. Absent in files written before it existed; the reader then recovers it from σ² − I |
|
||
| `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` |
|
||
| `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) |
|
||
| `observed_frame` | | standard | image number of the reflection |
|
||
| `lp` | | standard | Lorentz–polarization factor (stored as `1/rlp`) |
|
||
| `partiality` | | standard | recorded fraction of the reflection |
|
||
| `delta_phi` | deg | **extension** | XDS Δφ: offset from the centre of the current frame |
|
||
| `zeta` | | **extension** | Lorentz ζ factor (reciprocal-space geometry term) |
|
||
| `image_scale_corr` | | **extension** | per-image scale correction; `I_true = image_scale_corr · int_sum` |
|
||
|
||
In the master file these per-image groups are exposed through `/entry/reflections` external links
|
||
(VDS/integrated formats).
|
||
|
||
### 4.3 `/entry/azint` — azimuthal integration
|
||
|
||
| Dataset | Shape | Units | Meaning |
|
||
|---------|-------|-------|---------|
|
||
| `bin_to_q` | `[φ_bins, q_bins]` | Å⁻¹ | q value of each bin |
|
||
| `bin_to_two_theta` | `[φ_bins, q_bins]` | deg | 2θ of each bin |
|
||
| `bin_to_phi` | `[φ_bins, q_bins]` | deg | azimuthal angle of each bin |
|
||
| `image` | `[n_images, φ_bins, q_bins]` | | per-image integrated profile (NaN for empty bins) |
|
||
| `image_std` | `[n_images, φ_bins, q_bins]` | | per-bin standard deviation |
|
||
| `image_count` | `[n_images, φ_bins, q_bins]` | | pixels contributing per bin |
|
||
| `map` | `[y, x]` | | pixel→bin mapping (master file only) |
|
||
|
||
### 4.4 `/entry/roi` — regions of interest (per-image results)
|
||
|
||
`/entry/roi/<roi_name>` has one sub-group per configured ROI, holding the **per-image result
|
||
vectors** `[n_images]`. These are written into the data files; in VDS mode they are exposed from
|
||
the master file through virtual datasets, and in integrated mode they are in the single file.
|
||
(In legacy mode they remain only in the data files.)
|
||
|
||
| Dataset | Meaning |
|
||
|---------|---------|
|
||
| `max` | maximum pixel value in the ROI |
|
||
| `sum` | sum of pixel values |
|
||
| `sum_sq` | sum of squared pixel values |
|
||
| `npixel` | number of valid pixels |
|
||
| `x`, `y` | intensity-weighted centroid |
|
||
|
||
### 4.4.1 `/entry/roi_defs` — ROI definitions (master file)
|
||
|
||
The **dataset-wide ROI definitions** (geometry, fixed for the whole acquisition) live in the
|
||
master file under a *separate* `/entry/roi_defs` group — kept apart from `/entry/roi` above so
|
||
that older readers, which iterate `/entry/roi`, are unaffected by these entries. One sub-group
|
||
`/entry/roi_defs/<roi_name>` per ROI:
|
||
|
||
| Dataset | Meaning |
|
||
|---------|---------|
|
||
| `bit_index` | which bit of `roi_map` (below) marks this ROI |
|
||
| `type` | `box`, `circle` or `azim` |
|
||
| `min_x_pxl`, `max_x_pxl`, `min_y_pxl`, `max_y_pxl` | box bounds (type `box`) |
|
||
| `center_x_pxl`, `center_y_pxl`, `radius_pxl` | circle (type `circle`) |
|
||
| `q_min_recipA`, `q_max_recipA` | Q range (type `azim`) |
|
||
| `phi_min_deg`, `phi_max_deg` | azimuthal-angle sector (type `azim`, omitted for a full ring) |
|
||
|
||
`/entry/roi_defs/roi_map` `[y, x]` is a `uint16` per-pixel bitmask: bit `bit_index` is set for
|
||
every pixel belonging to that ROI, so an ROI's footprint can be recovered exactly.
|
||
|
||
### 4.5 `/entry/image` — per-image pixel statistics
|
||
|
||
`[n_images]` vectors: `max_value`, `min_value` (viable min/max, excluding error/saturated pixels),
|
||
`error_pixels`, `saturated_pixels`, `pixel_sum`. Surfaced in the master file under `/entry/image`.
|
||
|
||
### 4.6 `/entry/profiling` — per-image timing
|
||
|
||
`[n_images]` vectors in seconds: `spotFindingTime`, `indexingTime`, `integrationTime`,
|
||
`refinementTime`, `processingTime`, `braggPredictionTime`, `preprocessingTime`, `compressionTime`,
|
||
`azIntTime`, `indexAnalysisTime`, `imageScaleTime`.
|
||
|
||
### 4.7 `/entry/detector` — acquisition diagnostics (data file)
|
||
|
||
A convenience NXcollection in the data file (note: distinct from the standard
|
||
`/entry/instrument/detector`). In **integrated** format these datasets are written under
|
||
`/entry/instrument/detector/detectorSpecific` instead.
|
||
|
||
| Dataset | Meaning |
|
||
|---------|---------|
|
||
| `timestamp`, `exptime` | per-image timestamp and exposure time |
|
||
| `number` | image number (original number if image rejection was used) |
|
||
| `det_info` | JUNGFRAU debug field |
|
||
| `storage_cell_image` | storage-cell number |
|
||
| `rcv_delay`, `rcv_free_send_buffers` | receiver internal diagnostics |
|
||
| `packets_expected`, `packets_received` | UDP packets per image |
|
||
| `data_collection_efficiency_image` | received / expected packet ratio |
|
||
|
||
### 4.8 `/entry/xfel` — pulsed-source metadata
|
||
|
||
`[n_images]` vectors `pulseID` and `eventCode`, written for pulsed sources (e.g. SwissFEL).
|
||
|
||
### 4.9 Other collections
|
||
|
||
| Path | Class | Content |
|
||
|------|-------|---------|
|
||
| `/entry/instrument/detector/detectorSpecific` | NXcollection | Dectris-style detector metadata + Jungfraujoch fields: `x_pixels_in_detector`, `y_pixels_in_detector`, `nimages`, `ntrigger`, `nimages_collected`, `nimages_written`, `data_collection_efficiency`, `max_receiver_delay`, `storage_cell_number`, `storage_cell_delay` [ns], `software_git_commit`, `software_git_date`, `jfjoch_release`, `jfjoch_writer_release`, `summation_mode`, `detect_ice_rings`, `gain_file_names`, `data_reduction_factor_serialmx`, `adu_histogram/`, `data_collection_efficiency_image` |
|
||
| `/entry/instrument/detector/calibration` | NXcollection | per-channel pedestal / calibration images (bitshuffle-compressed) |
|
||
| `/entry/instrument/fluorescence` | NXcollection | XRF spectrum: `energy` [eV], `data` |
|
||
| `/entry/user` | NXcollection | scalar values supplied under `header_appendix.hdf5` |
|
||
|
||
### 4.10 Non-standard fields inside the NXmx detector group
|
||
|
||
A few extension scalars are written *inside* the otherwise-standard `/entry/instrument/detector`
|
||
group for compatibility with existing tooling:
|
||
|
||
| Field | Units | Meaning |
|
||
|-------|-------|---------|
|
||
| `detector_distance` | m | duplicate of `distance` (Dectris/Neggia compatibility) |
|
||
| `detector_number` | | detector identifier (Dectris convention) |
|
||
| `mirror_y` (in `detectorSpecific`) | | whether the stored image is mirrored in Y relative to the raw readout; true is the MX convention (row 0 at the top) |
|
||
| `error_value` | | masked/error pixel sentinel: `UINTx_MAX` unsigned, `INTx_MIN` signed (NXmx has no equivalent). NXmx `underload_value` is written too: `INTx_MIN + 1` for signed, `0` for unsigned |
|
||
| `bit_depth_image` | | stored image bit depth (DECTRIS convention, not NXmx). Equal to `bit_depth_readout` where that is written, i.e. for unsigned images |
|
||
| `acquisition_type` | | always `triggered` (Dectris convention) |
|
||
| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied |
|
||
| `jungfrau_conversion_factor` | eV | conversion factor |
|
||
| `geometry_transformation_applied` | | module→full-detector geometry applied |
|
||
|
||
NeXus has no concept of a fill or no-data value — it expects bad pixels to be flagged in `pixel_mask`,
|
||
which Jungfraujoch also writes. The in-band sentinel above is a DECTRIS compatibility convention:
|
||
SIMPLON specifies that masked pixels are flagged with `2^bit_depth_image - 1`.
|
||
|
||
For an **unsigned** image the sentinel and the saturation code are the same value, so a saturated
|
||
pixel and a masked one cannot be told apart — the FPGA collapses both onto `UINTx_MAX`. Signed
|
||
images keep them separate: `INTx_MIN` is the marker, `INTx_MAX` is saturation.
|
||
|
||
`bit_depth_readout` is written for **unsigned** images only. DIALS remaps the top two codes of
|
||
`2^bit_depth_readout` to `-1` and `-2` whenever the field is present, regardless of the pixel type:
|
||
for an unsigned image those land below `underload_value` and are correctly masked, but for a signed
|
||
one they land inside the trusted range and a saturated pixel would be integrated as a count of `-2`.
|
||
Signed images are read correctly without the field; unsigned 32-bit cannot be read at all without it.
|
||
|
||
### 4.11 User-supplied metadata: `header_appendix` and `image_appendix`
|
||
|
||
Facilities frequently need to attach metadata that Jungfraujoch does not model explicitly. Two
|
||
free-form JSON fields in the `/start` request (`broker/jfjoch_api.yaml`) provide this without any
|
||
schema change; both accept *any valid JSON*:
|
||
|
||
| Field | Carried in | Persisted to HDF5? |
|
||
|-------|-----------|--------------------|
|
||
| `header_appendix` | the **start** message, under `user_data.user` (see [CBOR](CBOR.md)) | no — except the `hdf5` sub-object (below) |
|
||
| `image_appendix` | **every image** message, as `user_data` | no |
|
||
|
||
Both are forwarded verbatim through the ZeroMQ/CBOR stream to every downstream consumer (writer,
|
||
republished analysis, viewers), so they are the recommended channel for facility- or
|
||
beamline-specific provenance (proposal, operator, optics state, per-image trigger info, …) that has
|
||
no dedicated API field.
|
||
|
||
**Persisting selected values to HDF5.** `header_appendix` is normally *not* written to the master
|
||
file. As an exception, if it contains a key `hdf5` whose value is a JSON object of scalars (strings
|
||
and numbers — no arrays or nested objects), the writer stores each entry under `/entry/user/<key>`.
|
||
|
||
For example, a `/start` request containing:
|
||
|
||
```json
|
||
{
|
||
"header_appendix": {
|
||
"proposal": "p20001",
|
||
"operator": "jdoe",
|
||
"hdf5": { "beamline": "X06SA", "ring_mode": "top-up", "attenuator_foils": 2 }
|
||
},
|
||
"image_appendix": { "trigger_source": "external" }
|
||
}
|
||
```
|
||
|
||
forwards the whole `header_appendix` as `user_data.user` on the start message and
|
||
`{"trigger_source": "external"}` as `user_data` on every image message, and writes three scalars
|
||
into the master file:
|
||
|
||
```
|
||
/entry/user/beamline = "X06SA"
|
||
/entry/user/ring_mode = "top-up"
|
||
/entry/user/attenuator_foils = 2
|
||
```
|
||
|
||
## 5. Notes
|
||
|
||
* **Units** are written as the HDF5 `units` attribute on the dataset (e.g. `m`, `eV`, `deg`,
|
||
`Angstrom`, `Angstrom^-1`, `Angstrom^2`, `pixel`, `s`).
|
||
* **Sentinels.** Missing per-image values are `NaN` (floats) or `-1`/`0` (integer indices); image
|
||
pixels use `INTx_MIN` / `UINTx_MAX`.
|
||
* **Master vs data file.** In legacy/VDS formats the analysis arrays physically live in the data
|
||
files; the master file links to them (external links in legacy, virtual datasets in VDS). In the
|
||
integrated format there are no data files and everything is in one place.
|
||
* **CXI / CrystFEL.** `/entry/MX` follows the CXI peak-list convention; see
|
||
[CXI file format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf).
|