From e8d3eb1b08542ca035b9223965899fa2a6d7391e Mon Sep 17 00:00:00 2001 From: leonarski_f Date: Tue, 16 Jun 2026 19:47:36 +0200 Subject: [PATCH] docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link - Motivation no longer frames the format as serial-only: Jungfraujoch serves both rotation and serial MX, and the two-layout rationale (per-image CXI spots vs. dataset-wide NXreflections) is presented for both. - Document the _process.h5 reprocessing output of jfjoch_process: an integrated-format master whose /entry/data/data is a VDS linking back to the original images (all results, no image copies). - Add a section on the header_appendix / image_appendix (user_data) mechanism: how the free-form JSON flows through the start/image CBOR messages and how an "hdf5" sub-object is persisted to /entry/user, with an example. - Link the niggliClass row to International Tables for Crystallography A, Table 3.1.3.1. Co-Authored-By: Claude Opus 4.8 --- docs/HDF5.md | 107 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 85 insertions(+), 22 deletions(-) diff --git a/docs/HDF5.md b/docs/HDF5.md index 9824b1cd..49e72cbd 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -16,8 +16,9 @@ that document is a useful companion for their meaning. ## 1. Motivation: derived metadata and FAIR data The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep -them findable, accessible, interoperable and reusable (FAIR). For serial crystallography this is -hard for two practical reasons: +them findable, accessible, interoperable and reusable (FAIR). Jungfraujoch is used for both +**rotation** macromolecular crystallography (single- and multi-crystal, including fine-sliced and +helical scans) and **serial** crystallography (stills, grid scans); the same concerns apply to both: * **Findability.** Raw diffraction images carry almost no descriptive metadata about *content*. Quantities such as background level, number of diffraction spots, or indexing outcome let a user @@ -32,28 +33,31 @@ transparency and reproducibility of that analysis matter. As a minimum the write preserves spot-finding and indexing results together with the filters that were applied, and it can retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse. -### Why a CXI-style per-image layout for spot finding / indexing +### Two complementary layouts: per-image spots vs. a reflection table -Spot-finding and indexing results in serial crystallography are inherently *image-centric*: the -natural query is "give me the spots for image *n*". For these products Jungfraujoch adopts a layout -similar to the [Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the -convention understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties -(position, intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by -image number, with each image allocated room for up to a predefined maximum number of spots. These -dense arrays are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are -retrieved without traversing variable-length structures. The cost is some storage overhead for -unused slots (padded with sentinels), which is acceptable for the access pattern. +Jungfraujoch stores analysis products in two shapes, matching how each is accessed. -We also evaluated the NeXus +**Per-image spot finding / indexing.** Spot finding and indexing are inherently *image-centric* — +the natural query is "give me the spots for image *n*" — and this holds for serial stills and for +rotation frames alike. For these products Jungfraujoch adopts a layout similar to the +[Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the convention +understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties (position, +intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by image number, +with each image allocated room for up to a predefined maximum number of spots. These dense arrays +are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are retrieved +without traversing variable-length structures. The cost is some storage overhead for unused slots +(padded with sentinels), which is acceptable for the access pattern. + +**Integrated reflections.** Integrated intensities are naturally a *dataset-wide* table, which is +exactly the model of the NeXus [NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class. -NXreflections models a *dataset-wide* reflection table, which fits integrated rotation data well — -and Jungfraujoch does use it for integration results (see §4.2 below). -But for spot finding/indexing across hundreds of thousands of patterns a single table would force -aggregation over the whole experiment before the spots of one image can be accessed efficiently. -For these intermediate products a per-image representation is more suitable. We encourage the -community to develop standardised NeXus application definitions for image-centric serial -crystallography products that combine NeXus interoperability with the access patterns and scale of -modern experiments. +This fits rotation crystallography well, and Jungfraujoch uses NXreflections for its integration +results (see §4.2 below). We deliberately do *not* force spot finding/indexing into a single +experiment-wide table: across the hundreds of thousands of patterns typical of serial — or +fine-sliced rotation — experiments, that would require aggregating the whole experiment before the +spots of one image can be read. We encourage the community to develop standardised NeXus application +definitions for image-centric crystallography products that combine NeXus interoperability with the +access patterns and scale of modern high-throughput experiments. ## 2. File layout @@ -87,6 +91,22 @@ Images are stored chunked (one image per chunk) and compressed with bitshuffle + bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the "masked / no-data" sentinel), unsigned use `UINTx_MAX`. +### Reprocessing output: `_process.h5` + +The offline reprocessing tool [`jfjoch_process`](TOOLS.md) (`tools/jfjoch_process.cpp`) re-runs the +full analysis pipeline (spot finding, indexing, refinement, integration, scaling) on an existing +dataset and writes its results to a master file named **`_process.h5`**. This file uses the +**integrated** format, but instead of copying the images its `/entry/data/data` is a *virtual +dataset that links back to the original image files* (`hdf5_source_data` → +`NXmx::LinkToData_ProcessingVDS`). The result is a compact, self-describing companion file that +holds *all* the derived analysis (everything in §4) plus a virtual view +of the raw images — without duplicating terabytes of data. + +This is a particularly FAIR-friendly artefact: it can be shared or archived alongside (or instead +of) the raw data to convey what is in a dataset and how it processed, while the `/entry/data/data` +VDS still resolves to the original images when they are available. `jfjoch_process` can also process +an equally-spaced *subset* of images (start/end/stride), producing a down-sampled reference set. + ## 3. NXmx-standard content The entries below are part of, or valid base classes for, the @@ -223,7 +243,7 @@ In legacy/VDS mode these live in the data files and are linked/virtual-stacked i | `peakCountIndexed` | | spots fitting the indexing solution | | `imageIndexed` | | image was indexed (0/1) | | `indexingLatticeCount` | | number of lattices found for the image | -| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A*, Table 3.1.3.1) | +| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A* (2016), Vol. A, [Table 3.1.3.1](https://onlinelibrary.wiley.com/iucr/itc/Ac/ch3o1v0001/table3o1o3o1.pdf)) | | `bravaisLattice` | | Bravais lattice short code, e.g. `aP`, `mC`, `oF`, `tI`, `hP`, `hR`, `cF` | | `profileRadius` | Å⁻¹ | crystal profile radius | | `mosaicity` | deg | mosaicity estimate | @@ -367,6 +387,49 @@ group for compatibility with existing tooling: | `jungfrau_conversion_factor` | eV | conversion factor | | `geometry_transformation_applied` | | module→full-detector geometry applied | +### 4.11 User-supplied metadata: `header_appendix` and `image_appendix` + +Facilities frequently need to attach metadata that Jungfraujoch does not model explicitly. Two +free-form JSON fields in the `/start` request (`broker/jfjoch_api.yaml`) provide this without any +schema change; both accept *any valid JSON*: + +| Field | Carried in | Persisted to HDF5? | +|-------|-----------|--------------------| +| `header_appendix` | the **start** message, under `user_data.user` (see [CBOR](CBOR.md)) | no — except the `hdf5` sub-object (below) | +| `image_appendix` | **every image** message, as `user_data` | no | + +Both are forwarded verbatim through the ZeroMQ/CBOR stream to every downstream consumer (writer, +republished analysis, viewers), so they are the recommended channel for facility- or +beamline-specific provenance (proposal, operator, optics state, per-image trigger info, …) that has +no dedicated API field. + +**Persisting selected values to HDF5.** `header_appendix` is normally *not* written to the master +file. As an exception, if it contains a key `hdf5` whose value is a JSON object of scalars (strings +and numbers — no arrays or nested objects), the writer stores each entry under `/entry/user/`. + +For example, a `/start` request containing: + +```json +{ + "header_appendix": { + "proposal": "p20001", + "operator": "jdoe", + "hdf5": { "beamline": "X06SA", "ring_mode": "top-up", "attenuator_foils": 2 } + }, + "image_appendix": { "trigger_source": "external" } +} +``` + +forwards the whole `header_appendix` as `user_data.user` on the start message and +`{"trigger_source": "external"}` as `user_data` on every image message, and writes three scalars +into the master file: + +``` +/entry/user/beamline = "X06SA" +/entry/user/ring_mode = "top-up" +/entry/user/attenuator_foils = 2 +``` + ## 5. Notes * **Units** are written as the HDF5 `units` attribute on the dataset (e.g. `m`, `eV`, `deg`,