From 3d30c1df956084a81e44bc54007e6fa6cb72d350 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 25 Aug 2026 07:29:30 +0200 Subject: [PATCH] HDF5 docs: describe the transformation chains this release actually writes The detector depends_on table was the pre-rc.162 chain and was wrong in every row: the file now has rot3 at the root and translation at the leaf, with the detector depending on translation, which is what makes a tilt pivot about the sample. Give the vector values too, and say they are McStas rather than the internal frame - a reader cannot use the chain without that. Three prose paragraphs had been inserted into the middle of the non-standard- fields table, orphaning its last five rows into a second table headed bit_depth_image. Rows first, then the prose. The sample section still described a goniometer and a grid scan as alternatives and never mentioned the Smargon tagging, which is now load-bearing - a spindle can itself be called phi, and only equipment_component separates them. Also corrected: NXmxVDS is the default, not NXmxLegacy (FileWriterSettings.h, and the new SOFTWARE_INTEGRATION page already said so); the unsigned data-file fill is HDF5's own 0, not UINTx_MAX, since every unsigned code is a real count; underload_value is INTx_MIN+1 only for signed; and bit_depth_image equals bit_depth_readout only where the latter is written. The changelog claim about a grid scan reading back as one image is narrowed to the still-at-a-head-position case, which is the one the writer actually fixed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Y5XisyYxmF8mUEQjzpMRe2 --- docs/CHANGELOG.md | 2 +- docs/HDF5.md | 64 +++++++++++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 23 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 0ae576b3..5c2913d2 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -6,7 +6,7 @@ This is an UNSTABLE release. It includes many experimental features, as well as **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. -* HDF5: a grid scan, or a still recorded at a goniometer position, is no longer read back as a single image, and records its stationary spindle so a program that requires a rotation axis can open it. +* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. diff --git a/docs/HDF5.md b/docs/HDF5.md index 116b2e2d..5209f75d 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -78,8 +78,8 @@ Three master-file variants exist (set via `file_format`): | Format | Value | Master ↔ data linking | |--------|:-----:|------------------------| -| **NXmxLegacy** (default) | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. | -| **NXmxVDS** | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. | +| **NXmxLegacy** | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. | +| **NXmxVDS** (default) | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. | | **NXmxIntegrated** | 3 | No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format. | In legacy/VDS mode, image-indexed analysis arrays live in the **data files** and are exposed in the @@ -88,8 +88,10 @@ directly into the single file. Throughout this document a "✓ in master" column are visible (directly or via link/VDS) from the master file. Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or -bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the -"masked / no-data" sentinel), unsigned use `UINTx_MAX`. +bitshuffle + Zstd. Signed integer image datasets carry `INTx_MIN` as the HDF5 fill value (the +"masked / no-data" sentinel); unsigned ones are left at HDF5's own fill of 0, because every unsigned +code is a legitimate count. In the master's virtual dataset the fill is the error marker for both, +so a data file missing beside a VDS master reads as masked rather than as zero counts. > **Signed images and the Neggia XDS plugin.** Neggia dispatches on the size of the pixel in bytes > and always casts to an unsigned type, consulting the signedness of the data only for the pixel @@ -199,12 +201,19 @@ The NXtransformations *mechanism* (the `depends_on` chain, `transformation_type` `offset` attributes) is standard. The axis **names** follow the PyFAI PONI convention chosen by Jungfraujoch (see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)): -| Axis | Type | Units | Depends on | -|------|------|-------|-----------| -| `translation` | translation | m | `.` | -| `rot1` | rotation | rad | `translation` | -| `rot2` | rotation | rad | `rot1` | -| `rot3` | rotation | rad | `rot2` | +| Axis | Type | Units | Vector | Depends on | +|------|------|-------|--------|-----------| +| `rot3` | rotation | rad | `(0, 0, -1)` | `.` | +| `rot2` | rotation | rad | `(1, 0, 0)` | `rot3` | +| `rot1` | rotation | rad | `(0, -1, 0)` | `rot2` | +| `translation` | translation | m | unit vector along the sample→PONI direction | `rot1` | + +`/entry/instrument/detector/depends_on` is `translation`, and the module's `fast_pixel_direction`, +`slow_pixel_direction` and `module_offset` depend on it in turn. A chain is applied innermost-first, +so reading it outwards the detector is placed at its distance and beam centre and *then* tilted about +the sample — which is what makes a tilt pivot about the crystal rather than about the panel corner. +The `vector` values are in NXmx's **McStas** frame, which is Jungfraujoch's internal frame with x and +y negated. The beam centre is encoded in `translation` (its offset from the sample), not only in the informational `beam_center_x`/`beam_center_y` fields. In a `_process.h5` written by rugnux these axes @@ -221,16 +230,27 @@ NXmx (`fast/slow_pixel_direction` and `module_offset` carry transformation attri | Field | Std | Units / notes | |-------|:---:|-------| | `name` | NXmx | | -| `depends_on` | NXmx | points at the last goniometer / grid-scan axis, or `.` for stills | +| `depends_on` | NXmx | points at the innermost axis of the sample chain, or `.` for stills | | `temperature` | NXmx | K | -| `transformations/` (NXtransformations) | NXmx | rotation axis (e.g. `omega`) or grid-scan translation; hard-linked as `/entry/sample/goniometer` | +| `transformations/` (NXtransformations) | NXmx | the sample chain, written in mounting order; hard-linked as `/entry/sample/goniometer` | | `unit_cell` | base | `[a, b, c, α, β, γ]` | | `ub_matrix` | base | `[1, 3, 3]`, Angstrom⁻¹ | -For a rotation scan the goniometer axis is written as a per-image angle array `` plus -`_end`, scalar `_range_average`, `_range_total`, and for helical scans -`_helical_x/_y/_z`. These extra goniometer datasets beyond the bare axis array are Jungfraujoch -conveniences. +The chain is written from the base outwards, so the innermost axis — the one `depends_on` names — is +the one nearest the sample. It may hold, in that order: the grid-scan translations `grid_scan_x` and +`grid_scan_y`, the spindle, and a Smargon head's `chi` and `phi`. A grid scan and a goniometer axis +are **not** alternatives; both can be present. + +A **Smargon head position is told apart from the spindle** by the `equipment_component` attribute, +which is `"smargon"` on `chi` and `phi` and absent on the spindle. This is load-bearing: a spindle can +itself be named `phi`, and without the attribute a reader would take a head position for the scan +axis. `chi` and `phi` are written with one value per image even though neither turns, because a +reader takes the image count from the innermost axis of the chain: written as scalars, a still +recorded at a head position would read back as a single image however many were collected. + +For a rotation scan the goniometer axis carries, beyond the per-image angle array ``, the +Jungfraujoch conveniences `_end`, scalar `_range_average` and `_range_total`, and +for helical scans `_helical_x/_y/_z`. ### `/entry/data` (NXdata) @@ -452,7 +472,12 @@ group for compatibility with existing tooling: | `detector_distance` | m | duplicate of `distance` (Dectris/Neggia compatibility) | | `detector_number` | | detector identifier (Dectris convention) | | `mirror_y` (in `detectorSpecific`) | | whether the stored image is mirrored in Y relative to the raw readout; true is the MX convention (row 0 at the top) | -| `error_value` | | masked/error pixel sentinel: `UINTx_MAX` unsigned, `INTx_MIN` signed (NXmx has no equivalent; `underload_value` is written too, and is one above it) | +| `error_value` | | masked/error pixel sentinel: `UINTx_MAX` unsigned, `INTx_MIN` signed (NXmx has no equivalent). NXmx `underload_value` is written too: `INTx_MIN + 1` for signed, `0` for unsigned | +| `bit_depth_image` | | stored image bit depth (DECTRIS convention, not NXmx). Equal to `bit_depth_readout` where that is written, i.e. for unsigned images | +| `acquisition_type` | | always `triggered` (Dectris convention) | +| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied | +| `jungfrau_conversion_factor` | eV | conversion factor | +| `geometry_transformation_applied` | | module→full-detector geometry applied | NeXus has no concept of a fill or no-data value — it expects bad pixels to be flagged in `pixel_mask`, which Jungfraujoch also writes. The in-band sentinel above is a DECTRIS compatibility convention: @@ -467,11 +492,6 @@ images keep them separate: `INTx_MIN` is the marker, `INTx_MAX` is saturation. for an unsigned image those land below `underload_value` and are correctly masked, but for a signed one they land inside the trusted range and a saturated pixel would be integrated as a count of `-2`. Signed images are read correctly without the field; unsigned 32-bit cannot be read at all without it. -| `bit_depth_image` | | stored image bit depth; equal to `bit_depth_readout` (DECTRIS convention, not NXmx) | -| `acquisition_type` | | always `triggered` (Dectris convention) | -| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied | -| `jungfrau_conversion_factor` | eV | conversion factor | -| `geometry_transformation_applied` | | module→full-detector geometry applied | ### 4.11 User-supplied metadata: `header_appendix` and `image_appendix`