diff --git a/docs/HDF5.md b/docs/HDF5.md new file mode 100644 index 00000000..9824b1cd --- /dev/null +++ b/docs/HDF5.md @@ -0,0 +1,380 @@ +# HDF5 / NeXus data format + +Jungfraujoch stores images and on-the-fly analysis results in HDF5 files that aim to be +[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html)-compliant. On top of the +NXmx application definition, Jungfraujoch records a substantial amount of *derived* metadata +(spot finding, indexing, integration, azimuthal integration, per-image statistics, timing). These +extra entries do not exist in NXmx and are documented here so that the layout is unambiguous and +reusable. + +This page documents the **file layout and the data fields**. The operational behaviour of the +writer (running, republishing, file finalisation, CBF/TIFF output) is described in +[jfjoch_writer](JFJOCH_WRITER.md). The wire format that feeds the writer is described in +[CBOR messages](CBOR.md); fields below frequently correspond one-to-one to CBOR message fields, and +that document is a useful companion for their meaning. + +## 1. Motivation: derived metadata and FAIR data + +The goal of Jungfraujoch is not only to store high-throughput datasets efficiently, but to keep +them findable, accessible, interoperable and reusable (FAIR). For serial crystallography this is +hard for two practical reasons: + +* **Findability.** Raw diffraction images carry almost no descriptive metadata about *content*. + Quantities such as background level, number of diffraction spots, or indexing outcome let a user + judge the quality and relevance of a dataset *before* inspecting the raw images. +* **Accessibility at scale.** A single experiment can span tens to hundreds of terabytes. Standard + retrieval (e.g. HTTP) makes a dataset *available* but not *inspectable* — users would otherwise + have to download a large fraction of the data just to decide whether it is useful. Compact + derived representations make discovery, assessment and reuse feasible. + +Because Jungfraujoch couples acquisition with real-time analysis used to *steer* experiments, +transparency and reproducibility of that analysis matter. As a minimum the writer therefore +preserves spot-finding and indexing results together with the filters that were applied, and it can +retain an unbiased, down-sampled reference set of unfiltered images for validation and reuse. + +### Why a CXI-style per-image layout for spot finding / indexing + +Spot-finding and indexing results in serial crystallography are inherently *image-centric*: the +natural query is "give me the spots for image *n*". For these products Jungfraujoch adopts a layout +similar to the [Coherent X-ray Imaging (CXI) data bank](https://www.cxidb.org) (Maia, 2012) and the +convention understood by [CrystFEL](https://www.desy.de/~twhite/crystfel/): spot properties +(position, intensity, Miller index, …) are stored in fixed-size two-dimensional arrays indexed by +image number, with each image allocated room for up to a predefined maximum number of spots. These +dense arrays are addressed with ordinary HDF5 hyperslab reads, so the spots of a single image are +retrieved without traversing variable-length structures. The cost is some storage overhead for +unused slots (padded with sentinels), which is acceptable for the access pattern. + +We also evaluated the NeXus +[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class. +NXreflections models a *dataset-wide* reflection table, which fits integrated rotation data well — +and Jungfraujoch does use it for integration results (see §4.2 below). +But for spot finding/indexing across hundreds of thousands of patterns a single table would force +aggregation over the whole experiment before the spots of one image can be accessed efficiently. +For these intermediate products a per-image representation is more suitable. We encourage the +community to develop standardised NeXus application definitions for image-centric serial +crystallography products that combine NeXus interoperability with the access patterns and scale of +modern experiments. + +## 2. File layout + +A run is written as one **master file** plus, depending on the format, one or more **data files**: + +``` +_master.h5 # NXmx master file (metadata + links / virtual datasets) +_data_000001.h5 # data file: images + per-image analysis +_data_000002.h5 +... +``` + +The master file is produced by `writer/HDF5NXmx.cpp`; data files by `writer/HDF5DataFile.cpp` and +its plugins (`writer/HDF5DataFilePlugin*.cpp`). Files are written to a temporary `*..tmp` +name and renamed on successful close. + +Three master-file variants exist (set via `file_format`): + +| Format | Value | Master ↔ data linking | +|--------|:-----:|------------------------| +| **NXmxLegacy** (default) | 1 | One external link in `/entry/data` per data file (`data_000001`, …). HDF5 1.8 compatible — works with Neggia/Durin XDS plugins and Albula 4.0. | +| **NXmxVDS** | 2 | A single virtual dataset `/entry/data/data` spans all data files; spot finding, azimuthal integration and reflections are linked the same way. Requires HDF5 1.10 / Albula 4.1+. | +| **NXmxIntegrated** | 3 | No separate data files — images and all metadata live in one file. Equivalent in content to the VDS format. | + +In legacy/VDS mode, image-indexed analysis arrays live in the **data files** and are exposed in the +master file through external links or virtual datasets; in integrated mode they are written +directly into the single file. Throughout this document a "✓ in master" column marks entries that +are visible (directly or via link/VDS) from the master file. + +Images are stored chunked (one image per chunk) and compressed with bitshuffle + LZ4 or +bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the +"masked / no-data" sentinel), unsigned use `UINTx_MAX`. + +## 3. NXmx-standard content + +The entries below are part of, or valid base classes for, the +[NXmx](https://manual.nexusformat.org/classes/applications/NXmx.html) application definition. +"NXmx" = listed in the application definition; "base" = a valid field of the relevant NeXus base +class (`NXdetector`, `NXsample`, `NXsource`) but not in the NXmx required/recommended subset. + +### `/entry` (NXentry) + +| Field | Std | Notes | +|-------|:---:|-------| +| `definition` | NXmx | value `"NXmx"` | +| `start_time` | NXmx | arming time | +| `end_time`, `end_time_estimated` | NXmx | approximate end time | + +File-level HDF5 attributes `file_name`, `file_time`, `HDF5_Version` are also set. + +### `/entry/source` (NXsource), `/entry/instrument` (NXinstrument) + +| Field | Std | Units | +|-------|:---:|-------| +| `source/name`, `source/type` | NXmx / base | | +| `source/current` | base | A | +| `instrument/name` | NXmx | | + +### `/entry/instrument/beam` (NXbeam) + +| Field | Std | Units | +|-------|:---:|-------| +| `incident_wavelength` | NXmx | angstrom | +| `incident_wavelength_spread` | NXmx | angstrom (only if polychromatic) | +| `total_flux` | NXmx | Hz | + +### `/entry/instrument/attenuator` (NXattenuator) + +| Field | Std | +|-------|:---:| +| `attenuator_transmission` | NXmx | + +### `/entry/instrument/detector` (NXdetector) + +| Field | Std | Units | +|-------|:---:|-------| +| `depends_on` | NXmx | → `transformations/rot3` | +| `beam_center_x`, `beam_center_y` | NXmx | pixel | +| `distance` | NXmx | m | +| `count_time`, `frame_time` | NXmx | s | +| `sensor_thickness` | NXmx | m | +| `sensor_material` | NXmx | | +| `description` | NXmx | | +| `threshold_energy` | NXmx | eV (EIGER; written only for a single channel) | +| `x_pixel_size`, `y_pixel_size` | base | m | +| `serial_number` | base | | +| `bit_depth_readout` | NXmx | | +| `saturation_value` | NXmx | | +| `flatfield_applied` | NXmx | | +| `pixel_mask`, `pixel_mask_applied` | NXmx | `pixel_mask` is `[y, x]`, hard-linked from `detectorSpecific/pixel_mask` | +| `countrate_correction_applied` | NXmx | | +| `number_of_cycles` | base | frame-summation factor | + +### `/entry/instrument/detector/transformations` (NXtransformations) + +The NXtransformations *mechanism* (the `depends_on` chain, `transformation_type`, `vector`, +`offset` attributes) is standard. The axis **names** follow the PyFAI PONI convention chosen by +Jungfraujoch (see [DETECTOR_GEOMETRY](DETECTOR_GEOMETRY.md)): + +| Axis | Type | Units | Depends on | +|------|------|-------|-----------| +| `translation` | translation | m | `.` | +| `rot1` | rotation | rad | `translation` | +| `rot2` | rotation | rad | `rot1` | +| `rot3` | rotation | rad | `rot2` | + +### `/entry/instrument/detector/module` (NXdetector_module) + +`data_origin`, `data_size`, `fast_pixel_direction`, `slow_pixel_direction`, `module_offset` — all +NXmx (`fast/slow_pixel_direction` and `module_offset` carry transformation attributes). + +### `/entry/sample` (NXsample) + +| Field | Std | Units / notes | +|-------|:---:|-------| +| `name` | NXmx | | +| `depends_on` | NXmx | points at the last goniometer / grid-scan axis, or `.` for stills | +| `temperature` | NXmx | K | +| `transformations/` (NXtransformations) | NXmx | rotation axis (e.g. `omega`) or grid-scan translation; hard-linked as `/entry/sample/goniometer` | +| `unit_cell` | base | `[a, b, c, α, β, γ]` | +| `ub_matrix` | base | `[1, 3, 3]`, Angstrom⁻¹ | + +For a rotation scan the goniometer axis is written as a per-image angle array `` plus +`_end`, scalar `_range_average`, `_range_total`, and for helical scans +`_helical_x/_y/_z`. These extra goniometer datasets beyond the bare axis array are Jungfraujoch +conveniences. + +### `/entry/data` (NXdata) + +`data` (3-D image stack, `[n_images, y, x]`) with `image_nr_low` / `image_nr_high` attributes. +In legacy mode this group instead contains one external link `data_000001`, … per data file. + +## 4. Extensions beyond NXmx + +Everything in this section is **outside the NXmx standard**. Each group is declared with +`NX_class = NXcollection` (the NeXus-sanctioned container for non-standardised content) unless noted. +The per-image arrays are indexed by image number, padded to the run length and filled with a +sentinel (`NaN` for floats, `-1`/`0` for integer indices) where a quantity is absent. + +### 4.1 `/entry/MX` — spot finding and indexing (CXI-style) + +The flagship extension. Spot ("peak") properties are stored as fixed-size `[n_images, max_spots]` +arrays (CXI layout, recognised by CrystFEL); scalar-per-image quantities as `[n_images]` vectors. +In legacy/VDS mode these live in the data files and are linked/virtual-stacked into the master. + +**Per-spot arrays `[n_images, max_spots]`:** + +| Dataset | Units | Meaning | Indexing only | +|---------|-------|---------|:---:| +| `peakXPosRaw`, `peakYPosRaw` | pixel | spot position (raw detector frame) | | +| `peakTotalIntensity` | photons | spot intensity | | +| `peakIceRingRes` | | spot lies in an ice-ring resolution band | | +| `peakH`, `peakK`, `peakL` | | Miller indices of the (indexed) spot | ✓ | +| `peakDistEwaldSphere` | Å⁻¹ | distance of the spot from the Ewald sphere | ✓ | +| `peakIndexed` | | spot fits the indexing solution | ✓ | +| `peakLattice` | | lattice the spot belongs to (`-1` = unindexed) | ✓ | + +**Per-image vectors `[n_images]`:** + +| Dataset | Units | Meaning | +|---------|-------|---------| +| `nPeaks` | | number of spots stored for the image (CXI) | +| `strongPixels` | | strong-pixel count (first spot-finding stage) | +| `peakCountUnfiltered` | | spots found before filtering | +| `peakCountLowRes` | | low-resolution spots | +| `peakCountIceRingRes` | | spots inside ice-ring bands | +| `peakCountIndexed` | | spots fitting the indexing solution | +| `imageIndexed` | | image was indexed (0/1) | +| `indexingLatticeCount` | | number of lattices found for the image | +| `niggliClass` | | Niggli class of the indexed Bravais lattice (see *International Tables for Crystallography A*, Table 3.1.3.1) | +| `bravaisLattice` | | Bravais lattice short code, e.g. `aP`, `mC`, `oF`, `tI`, `hP`, `hR`, `cF` | +| `profileRadius` | Å⁻¹ | crystal profile radius | +| `mosaicity` | deg | mosaicity estimate | +| `bFactor` | Ų | per-image B-factor estimate | +| `resolutionEstimate` | Å | diffraction resolution estimate | +| `integratedReflections` | | number of integrated reflections | +| `bkgEstimate` | photons | mean background in the 3–5 Å resolution band | +| `beam_corr_x`, `beam_corr_y` | pixel | beam-center correction applied during processing | +| `imageScaleFactor` | | on-the-fly per-image scale factor *g* | +| `imageScaleCC` | | on-the-fly scaling correlation coefficient | +| `imageScaleMosaicity` | deg | scaling-model mosaicity | +| `imageScaleBFactor` | Ų | scaling-model B-factor | + +**Per-image lattices:** `latticeIndexed` `[n_images, 9]` (Å) — the real-space lattice (flattened +3×3); `latticeIndexedExtra` `[n_images, max_extra_lattices, 9]` (Å) — additional orientation +variants. + +**Run-level summaries** (written into the master `/entry/MX` at finalisation): + +| Dataset | Units | Meaning | +|---------|-------|---------| +| `indexing_algorithm` | | `FFBIDX` / `FFT (CUDA)` / `FFT (FFTW)` | +| `geom_refinement_algorithm` | | e.g. `beam_center` | +| `rotationLatticeIndexed` | Å | whole-run rotation-indexing lattice (`[9]`) | +| `rotationLatticeIndexedExtra` | Å | additional whole-run lattices (`[m, 9]`) | +| `rotationLatticeNiggliClass` | | Niggli class of the run lattice | +| `imageIndexedMean` | | mean indexing rate over the run | +| `bkgEstimateMean` | photons | mean background over the run | +| `indexedLatticeCount` | | per-image lattice count summary (master). *Note: data files use `indexingLatticeCount`; readers accept either.* | + +CrystFEL can read the spots directly with: + +``` +peak_list = /entry/MX +peak_list_type = cxi +``` + +### 4.2 `/entry/reflections` — integrated reflections (NXreflections) + +Integrated reflections are stored **per image** as +`/entry/reflections/image_NNNNNN` groups, each declared `NX_class = NXreflections`. The columns map +mostly onto the standard +[NXreflections](https://manual.nexusformat.org/classes/base_classes/NXreflections.html) base class: + +| Dataset | Units | NXreflections | Meaning | +|---------|-------|:-------------:|---------| +| `h`, `k`, `l` | | standard | Miller indices | +| `d` | Å | standard | resolution | +| `int_sum` | photons | standard | integrated intensity (summation) | +| `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) | +| `background_mean` | photons | standard | mean background under the peak | +| `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` | +| `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) | +| `observed_frame` | | standard | image number of the reflection | +| `lp` | | standard | Lorentz–polarization factor (stored as `1/rlp`) | +| `partiality` | | standard | recorded fraction of the reflection | +| `delta_phi` | deg | **extension** | XDS Δφ: offset from the centre of the current frame | +| `zeta` | | **extension** | Lorentz ζ factor (reciprocal-space geometry term) | +| `image_scale_corr` | | **extension** | per-image scale correction; `I_true = image_scale_corr · int_sum` | + +In the master file these per-image groups are exposed through `/entry/reflections` external links +(VDS/integrated formats). + +### 4.3 `/entry/azint` — azimuthal integration + +| Dataset | Shape | Units | Meaning | +|---------|-------|-------|---------| +| `bin_to_q` | `[φ_bins, q_bins]` | Å⁻¹ | q value of each bin | +| `bin_to_two_theta` | `[φ_bins, q_bins]` | deg | 2θ of each bin | +| `bin_to_phi` | `[φ_bins, q_bins]` | deg | azimuthal angle of each bin | +| `image` | `[n_images, φ_bins, q_bins]` | | per-image integrated profile (NaN for empty bins) | +| `image_std` | `[n_images, φ_bins, q_bins]` | | per-bin standard deviation | +| `image_count` | `[n_images, φ_bins, q_bins]` | | pixels contributing per bin | +| `map` | `[y, x]` | | pixel→bin mapping (master file only) | + +### 4.4 `/entry/roi/` — regions of interest + +One sub-group per configured ROI, each with `[n_images]` vectors: + +| Dataset | Meaning | +|---------|---------| +| `max` | maximum pixel value in the ROI | +| `sum` | sum of pixel values | +| `sum_sq` | sum of squared pixel values | +| `npixel` | number of valid pixels | +| `x`, `y` | intensity-weighted centroid | + +### 4.5 `/entry/image` — per-image pixel statistics + +`[n_images]` vectors: `max_value`, `min_value` (viable min/max, excluding error/saturated pixels), +`error_pixels`, `saturated_pixels`, `pixel_sum`. Surfaced in the master file under `/entry/image`. + +### 4.6 `/entry/profiling` — per-image timing + +`[n_images]` vectors in seconds: `spotFindingTime`, `indexingTime`, `integrationTime`, +`refinementTime`, `processingTime`, `braggPredictionTime`, `preprocessingTime`, `compressionTime`, +`azIntTime`, `indexAnalysisTime`, `imageScaleTime`. + +### 4.7 `/entry/detector` — acquisition diagnostics (data file) + +A convenience NXcollection in the data file (note: distinct from the standard +`/entry/instrument/detector`). In **integrated** format these datasets are written under +`/entry/instrument/detector/detectorSpecific` instead. + +| Dataset | Meaning | +|---------|---------| +| `timestamp`, `exptime` | per-image timestamp and exposure time | +| `number` | image number (original number if image rejection was used) | +| `det_info` | JUNGFRAU debug field | +| `storage_cell_image` | storage-cell number | +| `rcv_delay`, `rcv_free_send_buffers` | receiver internal diagnostics | +| `packets_expected`, `packets_received` | UDP packets per image | +| `data_collection_efficiency_image` | received / expected packet ratio | + +### 4.8 `/entry/xfel` — pulsed-source metadata + +`[n_images]` vectors `pulseID` and `eventCode`, written for pulsed sources (e.g. SwissFEL). + +### 4.9 Other collections + +| Path | Class | Content | +|------|-------|---------| +| `/entry/instrument/detector/detectorSpecific` | NXcollection | Dectris-style detector metadata + Jungfraujoch fields: `x_pixels_in_detector`, `y_pixels_in_detector`, `nimages`, `ntrigger`, `nimages_collected`, `nimages_written`, `data_collection_efficiency`, `max_receiver_delay`, `storage_cell_number`, `storage_cell_delay` [ns], `software_git_commit`, `software_git_date`, `jfjoch_release`, `jfjoch_writer_release`, `summation_mode`, `detect_ice_rings`, `gain_file_names`, `data_reduction_factor_serialmx`, `adu_histogram/`, `data_collection_efficiency_image` | +| `/entry/instrument/detector/calibration` | NXcollection | per-channel pedestal / calibration images (bitshuffle-compressed) | +| `/entry/instrument/fluorescence` | NXcollection | XRF spectrum: `energy` [eV], `data` | +| `/entry/user` | NXcollection | scalar values supplied under `header_appendix.hdf5` | + +### 4.10 Non-standard fields inside the NXmx detector group + +A few extension scalars are written *inside* the otherwise-standard `/entry/instrument/detector` +group for compatibility with existing tooling: + +| Field | Units | Meaning | +|-------|-------|---------| +| `detector_distance` | m | duplicate of `distance` (Dectris/Neggia compatibility) | +| `detector_number` | | detector identifier (Dectris convention) | +| `error_value` | | masked/error pixel sentinel (NXmx standard would be `underload_value`) | +| `bit_depth_image` | | stored image bit depth (NXmx standard is `bit_depth_readout`) | +| `acquisition_type` | | always `triggered` (Dectris convention) | +| `jungfrau_conversion_applied` | | JUNGFRAU photon/keV conversion applied | +| `jungfrau_conversion_factor` | eV | conversion factor | +| `geometry_transformation_applied` | | module→full-detector geometry applied | + +## 5. Notes + +* **Units** are written as the HDF5 `units` attribute on the dataset (e.g. `m`, `eV`, `deg`, + `Angstrom`, `Angstrom^-1`, `Angstrom^2`, `pixel`, `s`). +* **Sentinels.** Missing per-image values are `NaN` (floats) or `-1`/`0` (integer indices); image + pixels use `INTx_MIN` / `UINTx_MAX`. +* **Master vs data file.** In legacy/VDS formats the analysis arrays physically live in the data + files; the master file links to them (external links in legacy, virtual datasets in VDS). In the + integrated format there are no data files and everything is in one place. +* **CXI / CrystFEL.** `/entry/MX` follows the CXI peak-list convention; see + [CXI file format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf). diff --git a/docs/JFJOCH_WRITER.md b/docs/JFJOCH_WRITER.md index 9fbbb384..4cba4066 100644 --- a/docs/JFJOCH_WRITER.md +++ b/docs/JFJOCH_WRITER.md @@ -121,89 +121,15 @@ For example `header_appendix` of `{"param1": "test1", "param2": ["test1", "test2 Notifications for finalized files are optional, if notification port number is omitted this functionality is not enabled. ## HDF5 file structure -Jungfraujoch aims to generate files compliant with NXmx format. -### Master file +Jungfraujoch writes NXmx-compliant HDF5, with substantial derived metadata (spot finding, indexing, +integration, azimuthal integration, per-image statistics and timing) stored *beyond* the NXmx +standard. The complete file layout — master vs data files, the three format variants +(`NXmxLegacy`, `NXmxVDS`, `NXmxIntegrated`), every NXmx field that is populated and every +Jungfraujoch extension — is documented in [HDF5 / NeXus data format](HDF5.md). -There are custom extension to NXmx format. These will be documented in the future. - -Specifically, if data collection was configured with `header_appendix` having key equal to `hdf5` and value as JSON -object with number and string values. These will be added to `/entry/user`. - -There are three versions of master file possible. - -#### Legacy version (NXmxLegacy) -By default, *legacy version* is used. This version is compatible with DECTRIS file writer version 1.0 format. -This ensures the file compatibility of Neggia and Durin XDS plugins, as well as DECTRIS Albula viewer version 4.0. -Distinct feature is that if images are split into data files, there will be multiple links in `/entry/data`, -each corresponding to a data file. -Yet, certain new HDF5 features, like virtual datasets, are not possible in this format since it has to be compatible with HDF5 1.8 features. - -#### VDS format (NXmxVDS) -Therefore, we have enabled format *VDS version*. This will link to all data files via a single virtual dataset `/entry/data/data`. -The same way spot finding, azimuthal integration and others, will be linked between master and data files. -This format allows to display processing results in currently developed Jungfraujoch Viewer. -For the time being it only works with Durin XDS plugin, and require DECTRIS Albula viewer version 4.1+. - -#### Integrated format (NXmxIntegrated) -This is format, where no data files are created, but both images and metadata are stored in the same master file. -This is generally equivalent to VDS format described above. - -### Data file - -Data file has the following structure: - -| Location | Description | Optional | Linked in master file v. 2 | -|--------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|:--------------------------:| -| /entry/data/data | Images | | X | -| /entry/detector/timestamp | Timestamp of the image | | | -| /entry/detector/exptime | Exposure time of the image | | | -| /entry/detector/number | Image number; if image rejection was used this will be the original image number | | | -| /entry/detector/det_info | Debug field of the JF detector | X | | -| /entry/detector/storage_cell_image | Storage cell number | X | X * | -| /entry/detector/rcv_delay | Receiver delay for the image (Jungfraujoch debugging) | X | | -| /entry/detector/rcv_free_send_buffers | Receiver number of free send buffers at the time of sending the image (Jungfraujoch debugging) | X | | -| /entry/detector/data_collection_efficiency_image | Ratio of received and expected UDP packets | X | X * | -| /entry/detector/packets_expected | Number of UDP packets expected for the image | X | | -| /entry/detector/packets_received | Number of UDP packets received for the image | X | | -| /entry/image/max_value | Max viable value of the image (excl. overloads, etc.) | X | | -| /entry/azint/bin_to_q | Azimuthal integration - bin-to-Q mapping | X | | -| /entry/azint/image | Azimuthal integration - per image | X | X | -| /entry/MX/peakXPosRaw | Peak position X (see [CXI format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf)) | X | X | -| /entry/MX/peakYPosRaw | Peak position Y (see [CXI format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf)) | X | X | -| /entry/MX/peakTotalIntensity | Peak total intensity (see [CXI format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf)) | X | X | -| /entry/MX/peakH | Miller index h for each detected (indexed) peak | X | X | -| /entry/MX/peakK | Miller index k for each detected (indexed) peak | X | X | -| /entry/MX/peakL | Miller index l for each detected (indexed) peak | X | X | -| /entry/MX/peakDistEwaldSphere | Distance of the peak from the Ewald sphere (prediction) | X | X | -| /entry/MX/nPeaks | Number of peaks per image (see [CXI format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf)) | X | X | -| /entry/MX/strongPixels | Number of strong pixel per image | X | X | -| /entry/MX/nPeaksRingFiltered | Number of peaks not belonging to rings | X | X | -| /entry/MX/imageIndexed | Image is successfully indexed | X | X | -| /entry/MX/profileRadius | Crystal profile radius for indexed images | X | X | -| /entry/MX/latticeIndexed | Crystal lattice for the image, assuming it is indexed | X | X | -| /entry/MX/bkgEstimate | Mean value of pixels in the radius of 3-5 A | X | X | -| /entry/MX/resolutionEstimate | Resolution estimate based on on-the-fly integration | X | X | -| /entry/MX/beam_corr_x | Beam center correction applied during processing (X) [pixel] | X | X | -| /entry/MX/beam_corr_y | Beam center correction applied during processing (Y) [pixel] | X | X | -| /entry/MX/niggli_class | Niggli class identifier of the indexed Bravais lattice (per image) - see [International Tables for Crystallography A (2016). Vol. A, Table 3.1.3.1](https://onlinelibrary.wiley.com/iucr/itc/Ac/ch3o1v0001/table3o1o3o1.pdf) | X | X | -| /entry/MX/bravais_lattice | Bravais lattice short code (per image), e.g., aP, mC, oF, tI, hP, hR, cF | X | X | -| /entry/roi/{roi_name}/max | Max pixel value for roi named {roi_name} | X | X | -| /entry/roi/{roi_name}/sum | Sum pixel value for roi named {roi_name} | X | X | -| /entry/roi/{roi_name}/sum_sq | Sum pixel values squared for roi named {roi_name} | X | X | -| /entry/roi/{roi_name}/npixel | Number of valid pixel for roi named {roi_name} | X | X | -| /entry/roi/{roi_name}/x | Weighted X-coordinate for roi named {roi_name} | X | X | -| /entry/roi/{roi_name}/y | Weighted Y-coordinate for roi named {roi_name} | X | X | -| /entry/xfel/pulseID | Pulse ID (for XFEL only) | X | X | -| /entry/xfel/eventCode | Event code (for XFEL only) | X | X | - -\* - Datasets from `/entry/detector` in data file are mapped to `/entry/instrument/detector/detectorSpecific` in master file. - -If spot finding is enabled, spots are written in the [CXI format](https://raw.githubusercontent.com/cxidb/CXI/master/cxi_file_format.pdf) and are recognized by CrystFEL. The following has to be added to the CrystFEL geometry file: -``` -peak_list = /opt/MX -peak_list_type = cxi -``` +If data collection was configured with a `header_appendix` containing a key `hdf5` whose value is a +JSON object of numbers and strings, those entries are written to `/entry/user`. ## Other formats (CBF and TIFF) In addition to HDF5 format, Jungfraujoch allows to save images in the Crystallographic Binary File (CBF) format. diff --git a/docs/index.rst b/docs/index.rst index 2e7a0350..b8755c9c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -54,6 +54,7 @@ Jungfraujoch is distributed under the GPLv3 license. OPENAPI OPENAPI_SPECS CBOR + HDF5 IMAGE_STREAM PIXEL_MASK WEB_FRONTEND