docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link
- Motivation no longer frames the format as serial-only: Jungfraujoch serves both rotation and serial MX, and the two-layout rationale (per-image CXI spots vs. dataset-wide NXreflections) is presented for both. - Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an integrated-format master whose /entry/data/data is a VDS linking back to the original images (all results, no image copies). - Add a section on the header_appendix / image_appendix (user_data) mechanism: how the free-form JSON flows through the start/image CBOR messages and how an "hdf5" sub-object is persisted to /entry/user, with an example. - Link the niggliClass row to International Tables for Crystallography A, Table 3.1.3.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
+85
-22
@@ -16,8 +16,9 @@ that document is a useful companion for their meaning.
|
||||
## 1. Motivation: derived metadata and FAIR data
|
||||
|
||||
The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep
|
||||
them findable, accessible, interoperable and reusable (FAIR). For serial crystallography this is
|
||||
hard for two practical reasons:
|
||||
them findable, accessible, interoperable and reusable (FAIR). Jungfraujoch is used for both
|
||||
**rotation** macromolecular crystallography (single- and multi-crystal, including fine-sliced and
|
||||
helical scans) and **serial** crystallography (stills, grid scans); the same concerns apply to both:
|
||||
|
||||
* **Findability.** Raw diffraction images carry almost no descriptive metadata about *content*.
|
||||
Quantities such as background level, number of diffraction spots, or indexing outcome let a user
|
||||
@@ -32,28 +33,31 @@ transparency and reproducibility of that analysis matter. As a minimum the write
|
||||
preserves spot-finding and indexing results together with the filters that were applied, and it can
|
||||
retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse.
|
||||
|
||||
### Why a CXI-style per-image layout for spot finding / indexing
|
||||
### Two complementary layouts: per-image spots vs. a reflection table
|
||||
|
||||
Spot-finding and indexing results in serial crystallography are inherently *image-centric*: the
|
||||
natural query is "give me the spots for image *n*". For these products Jungfraujoch adopts a layout
|
||||
similar to the [Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the
|
||||
convention understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties
|
||||
(position, intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by
|
||||
image number, with each image allocated room for up to a predefined maximum number of spots. These
|
||||
dense arrays are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are
|
||||
retrieved without traversing variable-length structures. The cost is some storage overhead for
|
||||
unused slots (padded with sentinels), which is acceptable for the access pattern.
|
||||
Jungfraujoch stores analysis products in two shapes, matching how each is accessed.
|
||||
|
||||
We also evaluated the NeXus
|
||||
**Per-image spot finding / indexing.** Spot finding and indexing are inherently *image-centric* —
|
||||
the natural query is "give me the spots for image *n*" — and this holds for serial stills and for
|
||||
rotation frames alike. For these products Jungfraujoch adopts a layout similar to the
|
||||
[Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the convention
|
||||
understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties (position,
|
||||
intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by image number,
|
||||
with each image allocated room for up to a predefined maximum number of spots. These dense arrays
|
||||
are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are retrieved
|
||||
without traversing variable-length structures. The cost is some storage overhead for unused slots
|
||||
(padded with sentinels), which is acceptable for the access pattern.
|
||||
|
||||
**Integrated reflections.** Integrated intensities are naturally a *dataset-wide* table, which is
|
||||
exactly the model of the NeXus
|
||||
[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class.
|
||||
NXreflections models a *dataset-wide* reflection table, which fits integrated rotation data well —
|
||||
and Jungfraujoch does use it for integration results (see §4.2 below).
|
||||
But for spot finding/indexing across hundreds of thousands of patterns a single table would force
|
||||
aggregation over the whole experiment before the spots of one image can be accessed efficiently.
|
||||
For these intermediate products a per-image representation is more suitable. We encourage the
|
||||
community to develop standardised NeXus application definitions for image-centric serial
|
||||
crystallography products that combine NeXus interoperability with the access patterns and scale of
|
||||
modern experiments.
|
||||
This fits rotation crystallography well, and Jungfraujoch uses NXreflections for its integration
|
||||
results (see §4.2 below). We deliberately do *not* force spot finding/indexing into a single
|
||||
experiment-wide table: across the hundreds of thousands of patterns typical of serial — or
|
||||
fine-sliced rotation — experiments, that would require aggregating the whole experiment before the
|
||||
spots of one image can be read. We encourage the community to develop standardised NeXus application
|
||||
definitions for image-centric crystallography products that combine NeXus interoperability with the
|
||||
access patterns and scale of modern high-throughput experiments.
|
||||
|
||||
## 2. File layout
|
||||
|
||||
@@ -87,6 +91,22 @@ Images are stored chunked (one image per chunk) and compressed with bitshuffle +
|
||||
bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the
|
||||
"masked / no-data" sentinel), unsigned use `UINTx_MAX`.
|
||||
|
||||
### Reprocessing output: `<prefix>_process.h5`
|
||||
|
||||
The offline reprocessing tool [`jfjoch_process`](TOOLS.md) (`tools/jfjoch_process.cpp`) re-runs the
|
||||
full analysis pipeline (spot finding, indexing, refinement, integration, scaling) on an existing
|
||||
dataset and writes its results to a master file named **`<prefix>_process.h5`**. This file uses the
|
||||
**integrated** format, but instead of copying the images its `/entry/data/data` is a *virtual
|
||||
dataset that links back to the original image files* (`hdf5_source_data` →
|
||||
`NXmx::LinkToData_ProcessingVDS`). The result is a compact, self-describing companion file that
|
||||
holds *all* the derived analysis (everything in §4) plus a virtual view
|
||||
of the raw images — without duplicating terabytes of data.
|
||||
|
||||
This is a particularly FAIR-friendly artefact: it can be shared or archived alongside (or instead
|
||||
of) the raw data to convey what is in a dataset and how it processed, while the `/entry/data/data`
|
||||
VDS still resolves to the original images when they are available. `jfjoch_process` can also process
|
||||
an equally-spaced *subset* of images (start/end/stride), producing a down-sampled reference set.
|
||||
|
||||
## 3. NXmx-standard content
|
||||
|
||||
The entries below are part of, or valid base classes for, the
|
||||
@@ -223,7 +243,7 @@ In legacy/VDS mode these live in the data files and are linked/virtual-stacked i
|
||||
| `peakCountIndexed` | | spots fitting the indexing solution |
|
||||
| `imageIndexed` | | image was indexed (0/1) |
|
||||
| `indexingLatticeCount` | | number of lattices found for the image |
|
||||
| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A*, Table 3.1.3.1) |
|
||||
| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A* (2016), Vol. A, [Table 3.1.3.1](https://onlinelibrary.wiley.com/iucr/itc/Ac/ch3o1v0001/table3o1o3o1.pdf)) |
|
||||
| `bravaisLattice` | | Bravais lattice short code, e.g. `aP`, `mC`, `oF`, `tI`, `hP`, `hR`, `cF` |
|
||||
| `profileRadius` | Å⁻¹ | crystal profile radius |
|
||||
| `mosaicity` | deg | mosaicity estimate |
|
||||
@@ -367,6 +387,49 @@ group for compatibility with existing tooling:
|
||||
| `jungfrau_conversion_factor` | eV | conversion factor |
|
||||
| `geometry_transformation_applied` | | module→full-detector geometry applied |
|
||||
|
||||
### 4.11 User-supplied metadata: `header_appendix` and `image_appendix`
|
||||
|
||||
Facilities frequently need to attach metadata that Jungfraujoch does not model explicitly. Two
|
||||
free-form JSON fields in the `/start` request (`broker/jfjoch_api.yaml`) provide this without any
|
||||
schema change; both accept *any valid JSON*:
|
||||
|
||||
| Field | Carried in | Persisted to HDF5? |
|
||||
|-------|-----------|--------------------|
|
||||
| `header_appendix` | the **start** message, under `user_data.user` (see [CBOR](CBOR.md)) | no — except the `hdf5` sub-object (below) |
|
||||
| `image_appendix` | **every image** message, as `user_data` | no |
|
||||
|
||||
Both are forwarded verbatim through the ZeroMQ/CBOR stream to every downstream consumer (writer,
|
||||
republished analysis, viewers), so they are the recommended channel for facility- or
|
||||
beamline-specific provenance (proposal, operator, optics state, per-image trigger info, …) that has
|
||||
no dedicated API field.
|
||||
|
||||
**Persisting selected values to HDF5.** `header_appendix` is normally *not* written to the master
|
||||
file. As an exception, if it contains a key `hdf5` whose value is a JSON object of scalars (strings
|
||||
and numbers — no arrays or nested objects), the writer stores each entry under `/entry/user/<key>`.
|
||||
|
||||
For example, a `/start` request containing:
|
||||
|
||||
```json
|
||||
{
|
||||
"header_appendix": {
|
||||
"proposal": "p20001",
|
||||
"operator": "jdoe",
|
||||
"hdf5": { "beamline": "X06SA", "ring_mode": "top-up", "attenuator_foils": 2 }
|
||||
},
|
||||
"image_appendix": { "trigger_source": "external" }
|
||||
}
|
||||
```
|
||||
|
||||
forwards the whole `header_appendix` as `user_data.user` on the start message and
|
||||
`{"trigger_source": "external"}` as `user_data` on every image message, and writes three scalars
|
||||
into the master file:
|
||||
|
||||
```
|
||||
/entry/user/beamline = "X06SA"
|
||||
/entry/user/ring_mode = "top-up"
|
||||
/entry/user/attenuator_foils = 2
|
||||
```
|
||||
|
||||
## 5. Notes
|
||||
|
||||
* **Units** are written as the HDF5 `units` attribute on the dataset (e.g. `m`, `eV`, `deg`,
|
||||
|
||||
Reference in New Issue
Block a user