mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 72 Wiki Activity

HDF5 docs: describe the transformation chains this release actually writes

Browse Source 
The detector depends_on table was the pre-rc.162 chain and was wrong in every
row: the file now has rot3 at the root and translation at the leaf, with the
detector depending on translation, which is what makes a tilt pivot about the
sample. Give the vector values too, and say they are McStas rather than the
internal frame - a reader cannot use the chain without that.

Three prose paragraphs had been inserted into the middle of the non-standard-
fields table, orphaning its last five rows into a second table headed
bit_depth_image. Rows first, then the prose.

The sample section still described a goniometer and a grid scan as alternatives
and never mentioned the Smargon tagging, which is now load-bearing - a spindle
can itself be called phi, and only equipment_component separates them.

Also corrected: NXmxVDS is the default, not NXmxLegacy (FileWriterSettings.h,
and the new SOFTWARE_INTEGRATION page already said so); the unsigned data-file
fill is HDF5's own 0, not UINTx_MAX, since every unsigned code is a real count;
underload_value is INTx_MIN+1 only for signed; and bit_depth_image equals
bit_depth_readout only where the latter is written.

The changelog claim about a grid scan reading back as one image is narrowed to
the still-at-a-head-position case, which is the one the writer actually fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y5XisyYxmF8mUEQjzpMRe2
This commit is contained in:
leonarski_f
2026-08-25 07:29:30 +02:00
co-authored by Claude Opus 5
parent 9b2b79aa73
commit 3d30c1df95
2 changed files with 43 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHANGELOG.md
+1 -1
View File
@@ -6,7 +6,7 @@ This is an UNSTABLE release. It includes many experimental features, as well as
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.
* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a grid scan, or a still recorded at a goniometer position, is no longer read back as a single image, and records its stationary spindle so a program that requires a rotation axis can open it.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
docs/HDF5.md
+42 -22
View File
@@ -78,8 +78,8 @@ Three master-file variants exist (set via `file_format`):
| Format | Value | Master ↔ data linking |
|--------|:-----:|------------------------|
| **NXmxLegacy** (default) | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. |
| **NXmxVDS** | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. |
| **NXmxLegacy** | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. |
| **NXmxVDS** (default) | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. |
| **NXmxIntegrated** | 3 | No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format. |
In legacy/VDS mode, image-indexed analysis arrays live in the **data files** and are exposed in the
@@ -88,8 +88,10 @@ directly into the single file. Throughout this document a "✓ in master" column
are visible (directly or via link/VDS) from the master file.
Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or
bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the
"masked / no-data" sentinel), unsigned use `UINTx_MAX`.
bitshuffle + Zstd. Signed integer image datasets carry `INTx_MIN` as the HDF5 fill value (the
"masked / no-data" sentinel); unsigned ones are left at HDF5's own fill of 0, because every unsigned
code is a legitimate count. In the master's virtual dataset the fill is the error marker for both,
so a data file missing beside a VDS master reads as masked rather than as zero counts.
> **Signed images and the Neggia XDS plugin.** Neggia dispatches on the size of the pixel in bytes
> and always casts to an unsigned type, consulting the signedness of the data only for the pixel
@@ -199,12 +201,19 @@ The NXtransformations *mechanism* (the `depends_on` chain, `transformation_type`
`offset` attributes) is standard. The axis **names** follow the PyFAI PONI convention chosen by
Jungfraujoch (see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)):
| Axis | Type | Units | Depends on |
|------|------|-------|-----------|
| `translation` | translation | m | `.` |
| `rot1` | rotation | rad | `translation` |
| `rot2` | rotation | rad | `rot1` |
| `rot3` | rotation | rad | `rot2` |
| Axis | Type | Units | Vector | Depends on |
|------|------|-------|--------|-----------|
| `rot3` | rotation | rad | `(0, 0, -1)` | `.` |
| `rot2` | rotation | rad | `(1, 0, 0)` | `rot3` |
| `rot1` | rotation | rad | `(0, -1, 0)` | `rot2` |
| `translation` | translation | m | unit vector along the sample→PONI direction | `rot1` |
`/entry/instrument/detector/depends_on` is `translation`, and the module's `fast_pixel_direction`,
`slow_pixel_direction` and `module_offset` depend on it in turn. A chain is applied innermost-first,
so reading it outwards the detector is placed at its distance and beam centre and *then* tilted about
the sample — which is what makes a tilt pivot about the crystal rather than about the panel corner.
The `vector` values are in NXmx's **McStas** frame, which is Jungfraujoch's internal frame with x and
y negated.
The beam centre is encoded in `translation` (its offset from the sample), not only in the
informational `beam_center_x`/`beam_center_y` fields. In a `_process.h5` written by rugnux these axes
@@ -221,16 +230,27 @@ NXmx (`fast/slow_pixel_direction` and `module_offset` carry transformation attri
| Field | Std | Units / notes |
|-------|:---:|-------|
| `name` | NXmx | |
| `depends_on` | NXmx | points at the last goniometer / grid-scan axis, or `.` for stills |
| `depends_on` | NXmx | points at the innermost axis of the sample chain, or `.` for stills |
| `temperature` | NXmx | K |
| `transformations/` (NXtransformations) | NXmx | rotation axis (e.g. `omega`) or grid-scan translation; hard-linked as `/entry/sample/goniometer` |
| `transformations/` (NXtransformations) | NXmx | the sample chain, written in mounting order; hard-linked as `/entry/sample/goniometer` |
| `unit_cell` | base | `[a, b, c, α, β, γ]` |
| `ub_matrix` | base | `[1, 3, 3]`, Angstrom⁻¹ |
For a rotation scan the goniometer axis is written as a per-image angle array `<axis>` plus
`<axis>_end`, scalar `<axis>_range_average`, `<axis>_range_total`, and for helical scans
`<axis>_helical_x/_y/_z`. These extra goniometer datasets beyond the bare axis array are Jungfraujoch
conveniences.
The chain is written from the base outwards, so the innermost axis — the one `depends_on` names — is
the one nearest the sample. It may hold, in that order: the grid-scan translations `grid_scan_x` and
`grid_scan_y`, the spindle, and a Smargon head's `chi` and `phi`. A grid scan and a goniometer axis
are **not** alternatives; both can be present.
A **Smargon head position is told apart from the spindle** by the `equipment_component` attribute,
which is `"smargon"` on `chi` and `phi` and absent on the spindle. This is load-bearing: a spindle can
itself be named `phi`, and without the attribute a reader would take a head position for the scan
axis. `chi` and `phi` are written with one value per image even though neither turns, because a
reader takes the image count from the innermost axis of the chain: written as scalars, a still
recorded at a head position would read back as a single image however many were collected.
For a rotation scan the goniometer axis carries, beyond the per-image angle array `<axis>`, the
Jungfraujoch conveniences `<axis>_end`, scalar `<axis>_range_average` and `<axis>_range_total`, and
for helical scans `<axis>_helical_x/_y/_z`.
### `/entry/data` (NXdata)
@@ -452,7 +472,12 @@ group for compatibility with existing tooling:
| `detector_distance` | m | duplicate of `distance` (Dectris/Neggia compatibility) |
| `detector_number` | | detector identifier (Dectris convention) |
| `mirror_y` (in `detectorSpecific`) | | whether the stored image is mirrored in Y relative to the raw readout; true is the MX convention (row 0 at the top) |
| `error_value` | | masked/error pixel sentinel: `UINTx_MAX` unsigned, `INTx_MIN` signed (NXmx has no equivalent; `underload_value` is written too, and is one above it) |
| `error_value` | | masked/error pixel sentinel: `UINTx_MAX` unsigned, `INTx_MIN` signed (NXmx has no equivalent). NXmx `underload_value` is written too: `INTx_MIN + 1` for signed, `0` for unsigned |
| `bit_depth_image` | | stored image bit depth (DECTRIS convention, not NXmx). Equal to `bit_depth_readout` where that is written, i.e. for unsigned images |
| `acquisition_type` | | always `triggered` (Dectris convention) |
| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied |
| `jungfrau_conversion_factor` | eV | conversion factor |
| `geometry_transformation_applied` | | module→full-detector geometry applied |
NeXus has no concept of a fill or no-data value — it expects bad pixels to be flagged in `pixel_mask`,
which Jungfraujoch also writes. The in-band sentinel above is a DECTRIS compatibility convention:
@@ -467,11 +492,6 @@ images keep them separate: `INTx_MIN` is the marker, `INTx_MAX` is saturation.
for an unsigned image those land below `underload_value` and are correctly masked, but for a signed
one they land inside the trusted range and a saturated pixel would be integrated as a count of `-2`.
Signed images are read correctly without the field; unsigned 32-bit cannot be read at all without it.
| `bit_depth_image` | | stored image bit depth; equal to `bit_depth_readout` (DECTRIS convention, not NXmx) |
| `acquisition_type` | | always `triggered` (Dectris convention) |
| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied |
| `jungfrau_conversion_factor` | eV | conversion factor |
| `geometry_transformation_applied` | | module→full-detector geometry applied |
### 4.11 User-supplied metadata: `header_appendix` and `image_appendix`