Jungfraujoch/docs/HDF5.md

HDF5 / NeXus data format

Jungfraujoch stores images and on-the-fly analysis results in HDF5 files that aim to be NXmx-compliant. On top of the NXmx application definition, Jungfraujoch records a substantial amount of derived metadata (spot finding, indexing, integration, azimuthal integration, per-image statistics, timing). These extra entries do not exist in NXmx and are documented here so that the layout is unambiguous and reusable.

This page documents the file layout and the data fields. The operational behaviour of the writer (running, republishing, file finalisation) is described in jfjoch_writer. The wire format that feeds the writer is described in CBOR messages; fields below frequently correspond one-to-one to CBOR message fields, and that document is a useful companion for their meaning.

1. Motivation: derived metadata and FAIR data

The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep them findable, accessible, interoperable and reusable (FAIR). Jungfraujoch is used for both rotation macromolecular crystallography (single- and multi-crystal, including fine-sliced and helical scans) and serial crystallography (stills, grid scans); the same concerns apply to both:

• Findability. Raw diffraction images carry almost no descriptive metadata about content. Quantities such as background level, number of diffraction spots, or indexing outcome let a user judge the quality and relevance of a dataset before inspecting the raw images.
• Accessibility at scale. A single experiment can span tens to hundreds of terabytes. Standard retrieval (e.g. HTTP) makes a dataset available but not inspectable — users would otherwise have to download a large fraction of the data just to decide whether it is useful. Compact derived representations make discovery, assessment and reuse feasible.

Because Jungfraujoch couples acquisition with real-time analysis used to steer experiments, transparency and reproducibility of that analysis matter. As a minimum the writer therefore preserves spot-finding and indexing results together with the filters that were applied, and it can retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse.

Two complementary layouts: per-image spots vs. a reflection table

Jungfraujoch stores analysis products in two shapes, matching how each is accessed.

Per-image spot finding / indexing. Spot finding and indexing are inherently image-centric — the natural query is "give me the spots for image n" — and this holds for serial stills and for rotation frames alike. For these products Jungfraujoch adopts a layout similar to the Coherent X-ray Imaging (CXI) data bank (Maia, 2012) and the convention understood by CrystFEL: spot properties (position, intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by image number, with each image allocated room for up to a predefined maximum number of spots. These dense arrays are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are retrieved without traversing variable-length structures. The cost is some storage overhead for unused slots (padded with sentinels), which is acceptable for the access pattern.

Integrated reflections. Integrated intensities are naturally a dataset-wide table, which is exactly the model of the NeXus NXreflections base class. This fits rotation crystallography well, and Jungfraujoch uses NXreflections for its integration results (see §4.2 below). We deliberately do not force spot finding/indexing into a single experiment-wide table: across the hundreds of thousands of patterns typical of serial — or fine-sliced rotation — experiments, that would require aggregating the whole experiment before the spots of one image can be read. We encourage the community to develop standardised NeXus application definitions for image-centric crystallography products that combine NeXus interoperability with the access patterns and scale of modern high-throughput experiments.

2. File layout

A run is written as one master file plus, depending on the format, one or more data files:

<prefix>_master.h5             # NXmx master file (metadata + links / virtual datasets)
<prefix>_data_000001.h5        # data file: images + per-image analysis
<prefix>_data_000002.h5
...

The master file is produced by writer/HDF5NXmx.cpp; data files by writer/HDF5DataFile.cpp and its plugins (writer/HDF5DataFilePlugin*.cpp). Files are written to a temporary *.<random>.tmp name and renamed on successful close.

Three master-file variants exist (set via file_format):

Format	Value	Master ↔ data linking
NXmxLegacy	1	One external link in /entry/data per data file (data_000001, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0.
NXmxVDS (default)	2	A single virtual dataset /entry/data/data spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+.
NXmxIntegrated	3	No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format.

In legacy/VDS mode, image-indexed analysis arrays live in the data files and are exposed in the master file through external links or virtual datasets; in integrated mode they are written directly into the single file. Throughout this document a "✓ in master" column marks entries that are visible (directly or via link/VDS) from the master file.

Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or bitshuffle + Zstd. Signed integer image datasets carry INTx_MIN as the HDF5 fill value (the "masked / no-data" sentinel); unsigned ones are left at HDF5's own fill of 0, because every unsigned code is a legitimate count. In the master's virtual dataset the fill is the error marker for both, so a data file missing beside a VDS master reads as masked rather than as zero counts.

Signed images and the Neggia XDS plugin. Neggia dispatches on the size of the pixel in bytes and always casts to an unsigned type, consulting the signedness of the data only for the pixel mask. A signed 16-bit image is therefore read wrongly: a count of -2 reaches XDS as 65534, and the -32768 error marker as 32768. Signed 32-bit happens to degrade safely, because every negative value ends up above INT32_MAX and is mapped to -1. Use the Jungfraujoch XDS plugin, or the Global Phasing build of Durin, for signed data — see Integration with MX data processing software.

Reprocessing output: <prefix>_process.h5

The offline reprocessing tool rugnux (rugnux/rugnux_cli.cpp) re-runs the full analysis pipeline (spot finding, indexing, refinement, integration, scaling) on an existing dataset and writes its results to a master file named <prefix>_process.h5. This file uses the integrated format, but instead of copying the images its /entry/data/data is a virtual dataset that links back to the original image files (hdf5_source_data → NXmx::LinkToData_ProcessingVDS). The result is a compact, self-describing companion file that holds all the derived analysis (everything in §4) plus a virtual view of the raw images — without duplicating terabytes of data.

This is a particularly FAIR-friendly artefact: it can be shared or archived alongside (or instead of) the raw data to convey what is in a dataset and how it processed, while the /entry/data/data VDS still resolves to the original images when they are available. rugnux can also process an equally-spaced subset of images (start/end/stride), producing a down-sampled reference set.

3. NXmx-standard content

The entries below are part of, or valid base classes for, the NXmx application definition. "NXmx" = listed in the application definition; "base" = a valid field of the relevant NeXus base class (NXdetector, NXsample, NXsource) but not in the NXmx required/recommended subset.

/entry (NXentry)

Field	Std	Notes
definition	NXmx	value "NXmx"
start_time	NXmx	arming time
end_time, end_time_estimated	NXmx	approximate end time

File-level HDF5 attributes file_name, file_time, HDF5_Version are also set.

/entry/source (NXsource), /entry/instrument (NXinstrument)

Field	Std	Units
source/name, source/type	NXmx / base
source/current	base	A
instrument/name	NXmx

/entry/instrument/beam (NXbeam)

Field	Std	Units
incident_wavelength	NXmx	angstrom
incident_wavelength_spread	NXmx	angstrom (only if polychromatic)
total_flux	NXmx	Hz

/entry/instrument/attenuator (NXattenuator)

Field	Std
attenuator_transmission	NXmx

/entry/instrument/detector (NXdetector)

Field	Std	Units
depends_on	NXmx	→ transformations/rot3
beam_center_x, beam_center_y	NXmx	pixel (0.0 = centre of the first pixel, see DETECTOR_GEOMETRY)
distance	NXmx	m
count_time, frame_time	NXmx	s
sensor_thickness	NXmx	m
sensor_material	NXmx
description	NXmx
threshold_energy	NXmx	eV (EIGER; written only for a single channel)
x_pixel_size, y_pixel_size	base	m
serial_number	base
bit_depth_readout	NXmx	bit depth of the stored image, not of the detector electronics - see below
saturation_value	NXmx	highest valid value. Read inclusively by NXmx, by the DIALS trusted_range and by the XDS OVERLOAD parameter; a saturated pixel carries the value one above it
underload_value	NXmx	lowest valid value: 0 for an unsigned image, INTx_MIN + 1 for a signed one
flatfield_applied	NXmx
pixel_mask, pixel_mask_applied	NXmx	pixel_mask is [y, x], hard-linked from detectorSpecific/pixel_mask
countrate_correction_applied	NXmx
number_of_cycles	base	frame-summation factor

Why bit_depth_readout is the image depth

NXmx defines only bit_depth_readout, "how many bits the electronics record per pixel", and has no field for the depth of the image actually stored. The two differ whenever summation is used: the readout stays at the detector's native width while the summed image must be wider to hold the sum.

Jungfraujoch writes the stored image depth into bit_depth_readout (and the identical value into the non-standard bit_depth_image). The electronic value is a constant of the detector and tells a data consumer nothing, whereas readers do use bit_depth_readout as the width of the stored pixel — DIALS, for instance, derives its masking markers from it and cannot read a 32-bit image without it. Writing the electronic value there would therefore mislead exactly in the case where the two differ.

Note that bit_depth_readout gives the width only. The sign is carried solely by the HDF5 element type of /entry/data/data (and, on the wire, by image_dtype); there is no NXmx field for it.

/entry/instrument/detector/transformations (NXtransformations)

The NXtransformations mechanism (the depends_on chain, transformation_type, vector, offset attributes) is standard. The axis names follow the PyFAI PONI convention chosen by Jungfraujoch (see DETECTOR_GEOMETRY):

Axis	Type	Units	Vector	Depends on
rot3	rotation	rad	(0, 0, -1)	.
rot2	rotation	rad	(1, 0, 0)	rot3
rot1	rotation	rad	(0, -1, 0)	rot2
translation	translation	m	unit vector along the sample→PONI direction	rot1

/entry/instrument/detector/depends_on is translation, and the module's fast_pixel_direction, slow_pixel_direction and module_offset depend on it in turn. A chain is applied innermost-first, so reading it outwards the detector is placed at its distance and beam centre and then tilted about the sample — which is what makes a tilt pivot about the crystal rather than about the panel corner. The vector values are in NXmx's McStas frame, which is Jungfraujoch's internal frame with x and y negated.

The beam centre is encoded in translation (its offset from the sample), not only in the informational beam_center_x/beam_center_y fields. In a _process.h5 written by rugnux these axes carry the refined detector geometry — the refined beam centre folds into translation and the refined tilt into rot1/rot2/rot3; the broker writes the user-provided geometry unchanged.

/entry/instrument/detector/module (NXdetector_module)

data_origin, data_size, fast_pixel_direction, slow_pixel_direction, module_offset — all NXmx (fast/slow_pixel_direction and module_offset carry transformation attributes).

/entry/sample (NXsample)

Field	Std	Units / notes
name	NXmx
depends_on	NXmx	points at the innermost axis of the sample chain, or . for stills
temperature	NXmx	K
transformations/ (NXtransformations)	NXmx	the sample chain, written in mounting order; hard-linked as /entry/sample/goniometer
unit_cell	base	[a, b, c, α, β, γ]
ub_matrix	base	[1, 3, 3], Angstrom⁻¹

The chain is written from the base outwards, so the innermost axis — the one depends_on names — is the one nearest the sample. It may hold, in that order: the grid-scan translations grid_scan_x and grid_scan_y, the spindle, and a Smargon head's chi and phi. A grid scan and a goniometer axis are not alternatives; both can be present.

A Smargon head position is told apart from the spindle by the equipment_component attribute, which is "smargon" on chi and phi and absent on the spindle. This is load-bearing: a spindle can itself be named phi, and without the attribute a reader would take a head position for the scan axis. chi and phi are written with one value per image even though neither turns, because a reader takes the image count from the innermost axis of the chain: written as scalars, a still recorded at a head position would read back as a single image however many were collected.

For a rotation scan the goniometer axis carries, beyond the per-image angle array <axis>, the Jungfraujoch conveniences <axis>_end, scalar <axis>_range_average and <axis>_range_total, and for helical scans <axis>_helical_x/_y/_z.

/entry/data (NXdata)

data (3-D image stack, [n_images, y, x]) with image_nr_low / image_nr_high attributes. In legacy mode this group instead contains one external link data_000001, … per data file.

4. Extensions beyond NXmx

Everything in this section is outside the NXmx standard. Each group is declared with NX_class = NXcollection (the NeXus-sanctioned container for non-standardised content) unless noted. The per-image arrays are indexed by image number, padded to the run length and filled with a sentinel (NaN for floats, -1/0 for integer indices) where a quantity is absent.

4.1 /entry/MX — spot finding and indexing (CXI-style)

The flagship extension. Spot ("peak") properties are stored as fixed-size [n_images, max_spots] arrays (CXI layout, recognised by CrystFEL); scalar-per-image quantities as [n_images] vectors. In legacy/VDS mode these live in the data files and are linked/virtual-stacked into the master.

Per-spot arrays [n_images, max_spots]:

Dataset	Units	Meaning	Indexing only
peakXPosRaw, peakYPosRaw	pixel	spot position (raw detector frame)
peakTotalIntensity	photons	spot intensity
peakIceRingRes		spot lies in an ice-ring resolution band
peakH, peakK, peakL		Miller indices of the (indexed) spot	✓
peakDistEwaldSphere	Å⁻¹	distance of the spot from the Ewald sphere	✓
peakIndexed		spot fits the indexing solution	✓
peakLattice		lattice the spot belongs to (-1 = unindexed)	✓

Per-image vectors [n_images]:

Dataset	Units	Meaning
nPeaks		number of spots stored for the image (CXI)
strongPixels		strong-pixel count (first spot-finding stage)
peakCountUnfiltered		spots found before filtering
peakCountLowRes		low-resolution spots
peakCountIceRingRes		spots inside ice-ring bands
peakCountIceRingControl		spots in the ice-free flanks beside those bands, rescaled to their q width - the control for the count above (their ratio, pooled over the run, is the spot-based ice indicator)
peakCountIndexed		spots fitting the indexing solution
imageIndexed		image was indexed (0/1)
indexingLatticeCount		number of lattices found for the image
niggliClass		Niggli class of the indexed Bravais lattice (see International Tables for Crystallography A (2016), Vol. A, Table 3.1.3.1)
bravaisLattice		Bravais lattice short code, e.g. aP, mC, oF, tI, hP, hR, cF
profileRadius	Å⁻¹	crystal profile radius
mosaicity	deg	mosaicity estimate
bFactor	Ų	per-image B-factor estimate
resolutionEstimate	Å	resolution the merged data are predicted to reach, from this image's spots alone
integratedReflections		number of integrated reflections
bkgEstimate	photons	mean background in the 3–5 Å resolution band
iceRingScore	ratio	strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice)
beam_corr_x, beam_corr_y	pixel	beam-center correction applied during processing
imageScaleFactor		on-the-fly per-image scale factor g
imageScaleCC		on-the-fly scaling correlation coefficient
imageScaleMosaicity	deg	scaling-model mosaicity
sweepQuality		why this image's stretch of the sweep was flagged — see below

Per-image lattices: latticeIndexed [n_images, 9] (Å) — the real-space lattice (flattened 3×3); latticeIndexedExtra [n_images, max_extra_lattices, 9] (Å) — additional orientation variants.

Run-level summaries (written into the master /entry/MX at finalisation):

Dataset	Units	Meaning
indexing_algorithm		FFBIDX / FFT (CUDA) / FFT (FFTW)
geom_refinement_algorithm		e.g. beam_center
rotationLatticeIndexed	Å	whole-run rotation-indexing lattice ([9])
rotationLatticeIndexedExtra	Å	additional whole-run lattices ([m, 9])
rotationLatticeNiggliClass		Niggli class of the run lattice
imageIndexedMean		mean indexing rate over the run
bkgEstimateMean	photons	mean background over the run
iceRingScoreMean	ratio	mean iceRingScore over the run — the single "how icy was this dataset" number (1 = no ice)
indexedLatticeCount		per-image lattice count summary (master). Note: data files use indexingLatticeCount; readers accept either.

Sweep quality. sweepQuality [n_images] (uint8) says why the stretch of the sweep this image belongs to was flagged as delivering much less than the rest of the run: 0 means it was not, and any other value is a 1-based index into sweepQualityReasons, a string vector written beside it that carries the whole vocabulary, so the codes can be read without this source. The vocabulary is closed and stable — a code is never renamed and never reused — and currently reads no_diffraction, crystal_out_of_beam, weak_diffraction, loss_of_centring, radiation_damage; the rugnux documentation defines what each one means. Both datasets are absent unless the sweep-quality diagnostic ran, which needs scaling and merging; their absence therefore means "not looked for", not "every image clean". Written by the offline rugnux path only — the broker does not merge — and not carried on the CBOR stream, in the same way as the other offline-only fields (space_group_number, the refined geometry). Nothing is excluded from processing on the strength of it. The condensed, dataset-wide form of the same finding is in <prefix>_report.txt.

CrystFEL can read the spots directly with:

peak_list = /entry/MX
peak_list_type = cxi

4.2 /entry/reflections — integrated reflections (NXreflections)

Integrated reflections are stored per image as /entry/reflections/image_NNNNNN groups, each declared NX_class = NXreflections. The columns map mostly onto the standard NXreflections base class:

Dataset	Units	NXreflections	Meaning
h, k, l		standard	Miller indices
d	Å	standard	resolution
int_sum	photons	standard	integrated intensity (summation)
int_err	photons	non-standard name	σ of the intensity (standard equivalent: int_sum_errors)
background_mean	photons	standard	mean background under the peak
background_variance	photons²	non-standard	non-signal part of σ², carried to the merge. Absent in files written before it existed; the reader then recovers it from σ² − I
predicted_x, predicted_y	pixel	name standard, units differ	predicted position. NXreflections predicted_x/_y are physical lengths; the pixel datasets are predicted_px_x/_y
observed_x, observed_y	pixel	name standard, units differ	observed centroid (pixels; standard pixel form is observed_px_x/_y)
observed_frame		standard	image number of the reflection
lp		standard	Lorentz–polarization factor (stored as 1/rlp)
partiality		standard	recorded fraction of the reflection
delta_phi	deg	extension	XDS Δφ: offset from the centre of the current frame
zeta		extension	Lorentz ζ factor (reciprocal-space geometry term)
image_scale_corr		extension	per-image scale correction; I_true = image_scale_corr · int_sum

In the master file these per-image groups are exposed through /entry/reflections external links (VDS/integrated formats).

4.3 /entry/azint — azimuthal integration

Dataset	Shape	Units	Meaning
bin_to_q	[φ_bins, q_bins]	Å⁻¹	q value of each bin
bin_to_two_theta	[φ_bins, q_bins]	deg	2θ of each bin
bin_to_phi	[φ_bins, q_bins]	deg	azimuthal angle of each bin
image	[n_images, φ_bins, q_bins]		per-image integrated profile (NaN for empty bins)
image_std	[n_images, φ_bins, q_bins]		per-bin standard deviation
image_count	[n_images, φ_bins, q_bins]		pixels contributing per bin
map	[y, x]		pixel→bin mapping (master file only)

4.4 /entry/roi — regions of interest (per-image results)

/entry/roi/<roi_name> has one sub-group per configured ROI, holding the per-image result vectors [n_images]. These are written into the data files; in VDS mode they are exposed from the master file through virtual datasets, and in integrated mode they are in the single file. (In legacy mode they remain only in the data files.)

Dataset	Meaning
max	maximum pixel value in the ROI
sum	sum of pixel values
sum_sq	sum of squared pixel values
npixel	number of valid pixels
x, y	intensity-weighted centroid

4.4.1 /entry/roi_defs — ROI definitions (master file)

The dataset-wide ROI definitions (geometry, fixed for the whole acquisition) live in the master file under a separate /entry/roi_defs group — kept apart from /entry/roi above so that older readers, which iterate /entry/roi, are unaffected by these entries. One sub-group /entry/roi_defs/<roi_name> per ROI:

Dataset	Meaning
bit_index	which bit of roi_map (below) marks this ROI
type	box, circle or azim
min_x_pxl, max_x_pxl, min_y_pxl, max_y_pxl	box bounds (type box)
center_x_pxl, center_y_pxl, radius_pxl	circle (type circle)
q_min_recipA, q_max_recipA	Q range (type azim)
phi_min_deg, phi_max_deg	azimuthal-angle sector (type azim, omitted for a full ring)

/entry/roi_defs/roi_map [y, x] is a uint16 per-pixel bitmask: bit bit_index is set for every pixel belonging to that ROI, so an ROI's footprint can be recovered exactly.

4.5 /entry/image — per-image pixel statistics

[n_images] vectors: max_value, min_value (viable min/max, excluding error/saturated pixels), error_pixels, saturated_pixels, pixel_sum. Surfaced in the master file under /entry/image.

4.6 /entry/profiling — per-image timing

[n_images] vectors in seconds: spotFindingTime, indexingTime, integrationTime, refinementTime, processingTime, braggPredictionTime, preprocessingTime, compressionTime, azIntTime, indexAnalysisTime, imageScaleTime.

4.7 /entry/detector — acquisition diagnostics (data file)

A convenience NXcollection in the data file (note: distinct from the standard /entry/instrument/detector). In integrated format these datasets are written under /entry/instrument/detector/detectorSpecific instead.

Dataset	Meaning
timestamp, exptime	per-image timestamp and exposure time
number	image number (original number if image rejection was used)
det_info	JUNGFRAU debug field
storage_cell_image	storage-cell number
rcv_delay, rcv_free_send_buffers	receiver internal diagnostics
packets_expected, packets_received	UDP packets per image
data_collection_efficiency_image	received / expected packet ratio

4.8 /entry/xfel — pulsed-source metadata

[n_images] vectors pulseID and eventCode, written for pulsed sources (e.g. SwissFEL).

4.9 Other collections

Path	Class	Content
/entry/instrument/detector/detectorSpecific	NXcollection	Dectris-style detector metadata + Jungfraujoch fields: x_pixels_in_detector, y_pixels_in_detector, nimages, ntrigger, nimages_collected, nimages_written, data_collection_efficiency, max_receiver_delay, storage_cell_number, storage_cell_delay [ns], software_git_commit, software_git_date, jfjoch_release, jfjoch_writer_release, summation_mode, detect_ice_rings, gain_file_names, data_reduction_factor_serialmx, adu_histogram/, data_collection_efficiency_image
/entry/instrument/detector/calibration	NXcollection	per-channel pedestal / calibration images (bitshuffle-compressed)
/entry/instrument/fluorescence	NXcollection	XRF spectrum: energy [eV], data
/entry/user	NXcollection	scalar values supplied under header_appendix.hdf5

4.10 Non-standard fields inside the NXmx detector group

A few extension scalars are written inside the otherwise-standard /entry/instrument/detector group for compatibility with existing tooling:

Field	Units	Meaning
detector_distance	m	duplicate of distance (Dectris/Neggia compatibility)
detector_number		detector identifier (Dectris convention)
mirror_y (in detectorSpecific)		whether the stored image is mirrored in Y relative to the raw readout; true is the MX convention (row 0 at the top)
error_value		masked/error pixel sentinel: UINTx_MAX unsigned, INTx_MIN signed (NXmx has no equivalent). NXmx underload_value is written too: INTx_MIN + 1 for signed, 0 for unsigned
bit_depth_image		stored image bit depth (DECTRIS convention, not NXmx). Equal to bit_depth_readout where that is written, i.e. for unsigned images
acquisition_type		always triggered (Dectris convention)
jungfrau_conversion_applied		JUNGFRAU photon/keV conversion applied
jungfrau_conversion_factor	eV	conversion factor
geometry_transformation_applied		module→full-detector geometry applied

NeXus has no concept of a fill or no-data value — it expects bad pixels to be flagged in pixel_mask, which Jungfraujoch also writes. The in-band sentinel above is a DECTRIS compatibility convention: SIMPLON specifies that masked pixels are flagged with 2^bit_depth_image - 1.

For an unsigned image the sentinel and the saturation code are the same value, so a saturated pixel and a masked one cannot be told apart — the FPGA collapses both onto UINTx_MAX. Signed images keep them separate: INTx_MIN is the marker, INTx_MAX is saturation.

bit_depth_readout is written for unsigned images only. DIALS remaps the top two codes of 2^bit_depth_readout to -1 and -2 whenever the field is present, regardless of the pixel type: for an unsigned image those land below underload_value and are correctly masked, but for a signed one they land inside the trusted range and a saturated pixel would be integrated as a count of -2. Signed images are read correctly without the field; unsigned 32-bit cannot be read at all without it.

4.11 User-supplied metadata: header_appendix and image_appendix

Facilities frequently need to attach metadata that Jungfraujoch does not model explicitly. Two free-form JSON fields in the /start request (broker/jfjoch_api.yaml) provide this without any schema change; both accept any valid JSON:

Field	Carried in	Persisted to HDF5?
header_appendix	the start message, under user_data.user (see CBOR)	no — except the hdf5 sub-object (below)
image_appendix	every image message, as user_data	no

Both are forwarded verbatim through the ZeroMQ/CBOR stream to every downstream consumer (writer, republished analysis, viewers), so they are the recommended channel for facility- or beamline-specific provenance (proposal, operator, optics state, per-image trigger info, …) that has no dedicated API field.

Persisting selected values to HDF5. header_appendix is normally not written to the master file. As an exception, if it contains a key hdf5 whose value is a JSON object of scalars (strings and numbers — no arrays or nested objects), the writer stores each entry under /entry/user/<key>.

For example, a /start request containing:

{
  "header_appendix": {
    "proposal": "p20001",
    "operator": "jdoe",
    "hdf5": { "beamline": "X06SA", "ring_mode": "top-up", "attenuator_foils": 2 }
  },
  "image_appendix": { "trigger_source": "external" }
}

forwards the whole header_appendix as user_data.user on the start message and {"trigger_source": "external"} as user_data on every image message, and writes three scalars into the master file:

/entry/user/beamline          = "X06SA"
/entry/user/ring_mode         = "top-up"
/entry/user/attenuator_foils  = 2

5. Notes

• Units are written as the HDF5 units attribute on the dataset (e.g. m, eV, deg, Angstrom, Angstrom^-1, Angstrom^2, pixel, s).
• Sentinels. Missing per-image values are NaN (floats) or -1/0 (integer indices); image pixels use INTx_MIN / UINTx_MAX.
• Master vs data file. In legacy/VDS formats the analysis arrays physically live in the data files; the master file links to them (external links in legacy, virtual datasets in VDS). In the integrated format there are no data files and everything is in one place.
• CXI / CrystFEL. /entry/MX follows the CXI peak-list convention; see CXI file format.