leonarski_f
538f3504d3
Build Packages / build:windows:nocuda (push) Successful in 20m4s
Build Packages / Unit tests (push) Skipped
Build Packages / build:viewer-tgz:cpu (push) Successful in 16m5s
Build Packages / build:viewer-tgz:cuda (push) Successful in 17m26s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 27m46s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m17s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 26m13s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m17s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m11s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m30s
Build Packages / build:rpm (rocky8) (push) Successful in 24m34s
Build Packages / build:rpm (rocky9) (push) Successful in 21m30s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m33s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m18s
Build Packages / DIALS test (push) Successful in 18m23s
Build Packages / XDS test (durin plugin) (push) Successful in 11m30s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m2s
Build Packages / Generate python client (push) Successful in 49s
Build Packages / Build documentation (push) Successful in 1m21s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 29m45s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
368 lines
71 KiB
Markdown
368 lines
71 KiB
Markdown
# CBOR messages
|
|
|
|
To communicate between FPGA-equipped receiver system and writers,
|
|
Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel).
|
|
The protocol is based on and compatible with [DECTRIS Stream2](https://github.com/dectris/documentation/tree/main/stream_v2).
|
|
There are minor differences at the moment:
|
|
|
|
* LZ4 alone is not allowed; Bitshuffle+LZ4 and Bitshuffle+Zstandard are allowed
|
|
* Few fields are currently absent
|
|
* Extra fields are present beyond DECTRIS standard
|
|
* There are calibration and metadata messages defined beyond DECTRIS specification
|
|
|
|
## Start message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format |
|
|
|----------------------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
|
|
| type | String | value "start" | X |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
|
|
| detector_distance | float | Detector distance \[m\] | |
|
|
| detector_translation | Array(float) | Detector translation vector \[m\] | X |
|
|
| beam_center_x | float | Beam center in X direction \[pixels\] | X |
|
|
| beam_center_y | float | Beam center in Y direction \[pixels\] | X |
|
|
| countrate_correction_enabled | bool | Countrate correction enabled | X |
|
|
| flatfield_enabled | bool | Flatfield enabled | X |
|
|
| number_of_images | uint64 | Number of images in the series | X |
|
|
| image_size_x | uint64 | Image width \[pixels\] | X |
|
|
| image_size_y | uint64 | Image height \[pixels\] | X |
|
|
| incident_energy | float | X-ray energy \[eV\] | X |
|
|
| incident_wavelength | float | X-ray wavelength \[Angstrom\] | X |
|
|
| incident_wavelength_spread | float (optional) | FWHM of the X-ray wavelength distribution \[Angstrom\] (NXmx incident_wavelength_spread); omitted when the beam is monochromatic | |
|
|
| frame_time | float | Frame time, if multiple frames per trigger \[s\] | X |
|
|
| count_time | float | Exposure time \[s\] | X |
|
|
| saturation_value | int64 | Maximum valid sample value | X |
|
|
| error_value | int64 (optional) | Value used in images to describe pixels that are in error state or missing | |
|
|
| pixel_size_x | float | Pixel width \[m\] | X |
|
|
| pixel_size_y | float | Pixel height \[m\] | X |
|
|
| sensor_thickness | float | Sensor thickness \[m\] | X |
|
|
| sensor_material | string | Sensor material | X |
|
|
| arm_date | date | Approximate date of arming | X |
|
|
| pixel_mask_enabled | bool | Pixel mask applied on images | X |
|
|
| detector_description | string | Name of the detector | X |
|
|
| detector_serial_number | string | Detector serial number | X |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
|
|
| fluorescence | object (optional) | X-ray fluorescence spectrum collected at start | |
|
|
| - energy | Array(float) | Energy of measuring point \[eV\] | |
|
|
| - data | Array(float) | Fluorescence scan result `data` \[arbitrary units\]; must be strictly the same length as energy | |
|
|
| goniometer | Map | Definition of rotation axis (optional) | X |
|
|
| - `AXIS` | string | Rotation axis name (e.g. omega) - only one axis is supported in Jungfraujoch | X |
|
|
| - - increment | float | Rotation axis increment (per image) in degree \[deg\] | X |
|
|
| - - start | float | Rotation axis start angle \[deg\] | X |
|
|
| - - axis | Array(float) | Vector for the rotation axis | |
|
|
| - - helical_step | Array(float) | Translation for helical scan for 1 image \[m\] | |
|
|
| - - screening_wedge | Array(float) | Wedge for screening \[deg\] (increment would correspond to difference between screening points) | |
|
|
| grid_scan | object | Grid scan definition (optional and exclusive with rotation axis) | |
|
|
| - n_fast | uint64 | Number of elements along fast axis | |
|
|
| - n_slow | uint64 | Number of elements along slow axis | |
|
|
| - step_x_axis | float | Step along X axis, can be negative \[m\] | |
|
|
| - step_y_axis | float | Step along Y axis, can be negative \[m\] | |
|
|
| - snake_scan | bool | Snake scan (rows alternate direction) | |
|
|
| - vertical_scan | bool | Vertical scan (enabled: fast direction = Y, disabled: fast direction = X) | |
|
|
| jungfrau_conversion_enabled | bool (optional) | Applying JUNGFRAU pixel conversion (to photons or keV) | |
|
|
| jungfrau_conversion_factor | float (optional) | Factor used for JUNGFRAU conversion \[eV\] | |
|
|
| geometry_transformation_enabled | bool (optional) | Transformation from detector module geometry (512x1024) to full detector geometry | |
|
|
| pixel_mask | Map(string -> Image) | Pixel mask - multiple in case of storage cells | X |
|
|
| channels | Array(string) | List of image channels | X |
|
|
| max_spot_count | uint64 | Maximum number of spots identified in spot finding | |
|
|
| max_extra_lattices | uint64 | Maximum number of extra lattices | |
|
|
| storage_cell_number | uint64 (optional) | Number of storage cells used by JUNGFRAU | |
|
|
| storage_cell_delay | Rational | Delay of storage cells in JUNGFRAU | |
|
|
| threshold_energy | Map(string -> float) | Per-channel threshold energy \[eV\] (map of channel name to value) | |
|
|
| image_dtype | string | Pixel bit type (e.g. uint16) | X |
|
|
| unit_cell | object (optional) | Unit cell of the system: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | |
|
|
| az_int_q_bin_count | uint64 | Number of azimuthal integration bins in the radial direction | |
|
|
| az_int_phi_bin_count | uint64 | Number of azimuthal integration bins in the phi angle direction | |
|
|
| az_int_bin_to_q | Array(float) | Q value for each azimuthal integration bin \[angstrom^-1\] | |
|
|
| az_int_bin_to_two_theta | Array(float) | Two theta angle value for each azimuthal integration bin \[deg\] | |
|
|
| az_int_bin_to_phi | Array(float) | Phi value for each azimuthal integration bin \[deg\] | |
|
|
| az_int_map | Image | Mapping between pixel and bin number | |
|
|
| summation | uint64 | Factor of frame summation | |
|
|
| user_data | string | JSON serialized to string that can contain the following fields (all fields are optional): | X |
|
|
| - file_prefix | string | File prefix | |
|
|
| - images_per_file | uint64 | Number of images written per file | |
|
|
| - images_per_trigger | uint64 | Number of images collected per trigger | |
|
|
| - source_name | string | Facility name | |
|
|
| - source_type | string | Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser") | |
|
|
| - instrument_name | string | Instrument name | |
|
|
| - sample_name | string | Name of the sample | |
|
|
| - user | any valid JSON | Value of header_appendix provided at collection start to Jungfraujoch | |
|
|
| - attenuator_transmission | float | Attenuator transmission \[\] | |
|
|
| - total_flux | float | Total flux \[ph/s\] | |
|
|
| - space_group_number | uint64 | Space group number | |
|
|
| - summation_mode | string | Summation mode (internal\|fpga\|cpu) | |
|
|
| - overwrite | bool | Overwrite existing HDF5 files | |
|
|
| - file_format | int | File writer format: 0 = only data files, 1 = NXmx legacy soft links, 2 = NXmx VDS, 3 = NXmx integrated, 4 = CBF, 5 = TIFF, 6 = no file written | |
|
|
| - roi | Array(object) | ROI configurations; each element is one of: | |
|
|
| | | type "box": xmin, xmax, ymin, ymax (numbers) | |
|
|
| | | type "circle": r, x, y (numbers) | |
|
|
| | | type "azim": qmin, qmax (numbers); optional phi_min, phi_max (numbers, deg) for an angular sector | |
|
|
| - gain_file_names | Array(string) | Names of JUNGFRAU gain files used for the current detector | |
|
|
| - write_master_file | bool | With multiple sockets, it selects which socket will provide master file | |
|
|
| - write_images | bool | Write images in the HDF5 file (if false, will only write metadata) | |
|
|
| - data_reduction_factor_serialmx | uint64 | Data reduction factor for serial MX | |
|
|
| - experiment_group | string | ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number | |
|
|
| - jfjoch_release | string | Jungfraujoch release number | |
|
|
| - socket_number | uint64 | Number of ZeroMQ socket (on `jfjoch_broker` side) used for transmission | |
|
|
| - bit_depth_readout | uint64 | Bit depth of the detector readout | |
|
|
| - writer_notification_zmq_addr | string | ZeroMQ address to inform `jfjoch_broker` about writers that finished operation | |
|
|
| - xfel_pulse_id | uint64 | Pulse IDs are recorded for images | |
|
|
| - ring_current_mA | float | Ring current at the start of the measurement | |
|
|
| - sample_temperature_K | float | Sample temperature \[K\] | |
|
|
| - detect_ice_rings | bool | Ice ring detection feature is enabled | |
|
|
| - indexing_algorithm | string | Indexing algorithm used on-the-fly; allowed values: ffbidx, fft, fftw, none | |
|
|
| - geom_refinement_algorithm | string | Post-indexing detector geometry refinement algorithm; allowed values: none, beam_center | |
|
|
| - poni_rot1 | float | Tilt of the detector rot1 according to PyFAI PONI convention \[rad\] | |
|
|
| - poni_rot2 | float | Tilt of the detector rot2 according to PyFAI PONI convention \[rad\] | |
|
|
| - poni_rot3 | float | Tilt of the detector rot3 according to PyFAI PONI convention \[rad\] | |
|
|
|
|
See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/stream_v2) for definition of Image as MultiDimArray with optional compression.
|
|
|
|
## Image message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format | Optional |
|
|
|-----------------------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:|
|
|
| type | String | value "image" | X | |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
|
|
| image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | |
|
|
| original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | | |
|
|
| real_time | Rational | Exposure time | X | |
|
|
| start_time | Rational | Exposure start time (highly approximate) | X | |
|
|
| end_time | Rational | Exposure end time (highly approximate) | X | |
|
|
| spots | Array(object) | Spots: | | |
|
|
| - x | float | observed position in x (pixels) | | |
|
|
| - y | float | observed position in y (pixels) | | |
|
|
| - I | float | intensity (photons) | | |
|
|
| - maxc | int64 | max count (photons) | | |
|
|
| - ice_ring | bool | spot in resolution range for ice rings | | |
|
|
| - indexed | bool | indexed solution | | |
|
|
| - latt | int64 | Lattice to which the peak belongs (negative number = not indexed) | | |
|
|
| - image | int64 | image number the spot belongs to | | |
|
|
| - h | int64 | Miller index (indexed spots only) | | |
|
|
| - k | int64 | Miller index (indexed spots only) | | |
|
|
| - l | int64 | Miller index (indexed spots only) | | |
|
|
| - dist_ewald | float | distance to Ewald sphere \[Angstrom^-1\] (indexed spots only) | | |
|
|
| reflections | Array(object) | Reflections: | | |
|
|
| - h | int64 | Miller index | | |
|
|
| - k | int64 | Miller index | | |
|
|
| - l | int64 | Miller index | | |
|
|
| - x | float | prediced position in x (pixels) | | |
|
|
| - y | float | predicted position in y (pixels) | | |
|
|
| - obs_x | float | observed position in x (pixels) | | |
|
|
| - obs_y | float | observed position in y (pixels) | | |
|
|
| - d | float | resolution \[Angstrom\] | | |
|
|
| - I | float | integrated intensity (photons) | | |
|
|
| - bkg | float | mean background value (photons) | | |
|
|
| - var_bkg | float | non-signal (background) part of sigma^2, carried to the merge (photons^2) | | |
|
|
| - sigma | float | standard deviation, estimated from counting statistics (photons) | | |
|
|
| - image | float | image number (present for each spot) | | |
|
|
| - rp | float | Distance to Ewald sphere \[Angstrom^-1\] | | |
|
|
| - rlp | float | Reciprocal Lorentz and polarization corrections | | |
|
|
| - partiality | float | Partiality of the reflection | | |
|
|
| - phi | float | phi angle from XDS: difference from middle of current frame, not absolute \[deg\] | | |
|
|
| - zeta | float | Lorentz zeta factor (reciprocal-space geometry term) | | |
|
|
| - image_scale_corr | float | Per-image scale correction; I_true = image_scale_corr * I | | |
|
|
| spot_count | uint64 | Spot count | | |
|
|
| spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | | |
|
|
| spot_count_low_res | uint64 | Number of spots in low resolution (prior to filtering) | | |
|
|
| spot_count_indexed | uint64 | Number of spots which fit indexing solution within a given tolerance | | |
|
|
| az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | |
|
|
| | | NaN is used for empty bins and has to be taken care by the receiver | | |
|
|
| az_int_profile_std | Array(float) | Standard deviation for azimuthal integration. (NaN for less than 2 samples) | | |
|
|
| az_int_profile_count | Array(uint64) | Number of pixels contributing to azimuthal bin | | |
|
|
| indexing_result | bool | Indexing successful | | |
|
|
| indexing_lattice_count | int64 | Number of indexing lattices found for this image | | |
|
|
| indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | | X |
|
|
| indexing_extra_lattices | Array(Array(9*float)) | Additional indexed lattices (orientation variants); present only if found | | |
|
|
| indexing_unit_cell | object | Indexing result unit cell: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\]; present only if indexed | | X |
|
|
| | | Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | | |
|
|
| profile_radius | float | Profile radius of the image - describes distance of observed reflections from the Ewald sphere \[Angstrom^-1\] | | |
|
|
| integrated_reflections | int64 | Count of integrated reflections | | |
|
|
| mosaicity | float | Angular range of spots in image from a rotation scan \[degree\] | | |
|
|
| b_factor | float | Estimated B-factor (Angstrom^2) | | |
|
|
| compression_time | float | Time spent on compression/decompressing image \[s\] | | |
|
|
| preprocessing_time | float | Time spent on preparing the image for analysis \[s\] | | |
|
|
| azint_time | float | Time spent on azimuthal integration \[s\] | | |
|
|
| spot_finding_time | float | Time spent on spot finding \[s\] | | |
|
|
| indexing_time | float | Time spent on indexing \[s\] | | |
|
|
| refinement_time | float | Time spent on refinement of indexing solution and experimental geometry \[s\] | | |
|
|
| index_analysis_time | float | Time spent on analyzing idnexing solution, calculating profile radius and mosaicity \[s\] | | |
|
|
| bragg_prediction_time | float | Time spent on predicting Bragg spots \[s\] | | |
|
|
| integration_time | float | Time spent on Bragg integration \[s\] | | |
|
|
| image_scale_time | float | Time spent on on-the-fly scaling \[s\] | | |
|
|
| processing_time | float | Total processing time \[s\] | | |
|
|
| xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X |
|
|
| xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X |
|
|
| lattice_type | object | Bravais lattice classification of the indexing result (present only if available) | | X |
|
|
| - centering | string | One-letter centering code: P, A, B, C, I, F, or R | | |
|
|
| - niggli_class | int64 | Integer identifier for the Niggli-reduced Bravais class | | |
|
|
| - system | string | Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic | | |
|
|
| jf_info | uint64 | Detector info field | | |
|
|
| receiver_aq_dev_delay | uint64 | Receiver internal delay | | |
|
|
| receiver_free_send_buf | uint64 | Receiver internal number of available buffer locations | | |
|
|
| receiver_buf_in_sending | uint64 | Receiver internal number of buffer locations currently in sending/writing | | |
|
|
| receiver_buf_in_preparation | uint64 | Receiver internal number of buffer locations currently in processing | | |
|
|
| storage_cell | uint64 | Storage cell number | | |
|
|
| saturated_pixel_count | uint64 | Saturated pixel count | | |
|
|
| pixel_sum | uint64 | Sum of all pixels, excl. error and saturation | | |
|
|
| error_pixel_count | uint64 | Error pixel count | | |
|
|
| strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | |
|
|
| min_viable_pixel_value | int64 | Minimal pixel value, excl. error and saturation | | |
|
|
| max_viable_pixel_value | int64 | Maximal pixel value, excl. error and saturation | | |
|
|
| resolution_estimate | float | Diffraction resolution estimation \[Angstrom\] | | X |
|
|
| data_collection_efficiency | float | Image collection efficiency \[\] | | |
|
|
| packets_expected | uint64 | Number of packets expected per image (in units of 2 kB) | | |
|
|
| packets_received | uint64 | Number of packets received per image (in units of 2 kB) | | |
|
|
| bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] | | |
|
|
| ice_ring_score | float | Strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice) | | |
|
|
| spot_count_ice_control | float | Spots in the ice-free flanks beside the hexagonal rings, rescaled to the ring bands' own q width (control for spot_count_ice_rings) | | |
|
|
| beam_corr_x | float | Beam center correction X applied during processing \[pixel\] | | X |
|
|
| beam_corr_y | float | Beam center correction Y applied during processing \[pixel\] | | X |
|
|
| image_scale_factor | float | Scaling result: Image scale factor (g) | | X |
|
|
| image_scale_mosaicity | float | Scaling result: Image scale mosaicity \[deg\] | | X |
|
|
| image_scale_cc | float | Scaling result: Image scale CC | | X |
|
|
| adu_histogram | Array(uint64) | ADU histogram | | |
|
|
| roi_integrals | object | Results of ROI calculation | | X |
|
|
| - sum | int64 | Sum of pixels in ROI area \[photons\] | | |
|
|
| - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | |
|
|
| - pixels | uint64 | Valid pixels in ROI area | | |
|
|
| - max_count | int64 | Highest count in ROI area \[photons\] | | |
|
|
| - x_weighted_sum | int64 | ROI pixel X position multiplied by photon count \[photons * pixels\] | | |
|
|
| - y_weighted_sum | int64 | ROI pixel Y position multiplied by photon count \[photons * pixels\] | | |
|
|
| user_data | string | Optional user defined text information - this is image_appendix serialized to JSON format | X | |
|
|
| data | Map(string -> Image) | Image | X | |
|
|
|
|
## Metadata message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format | Optional |
|
|
|------------------------------|------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:|
|
|
| type | String | value "metadata" | X | |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
|
|
| images | Array(object) | Array of images (order and size of the array are not guaranteed) | X | |
|
|
| - image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | |
|
|
| - original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | | |
|
|
| - real_time | Rational | Exposure time | X | |
|
|
| - start_time | Rational | Exposure start time (highly approximate) | X | |
|
|
| - end_time | Rational | Exposure end time (highly approximate) | X | |
|
|
| - spot_count | uint64 | Spot count | | |
|
|
| - spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | | |
|
|
| - az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | |
|
|
| - indexing_result | bool | Indexing successful | | |
|
|
| - indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | | X |
|
|
| - indexing_unit_cell | object | Indexing result unit cell: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\]; present only if indexed | | X |
|
|
| | | Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | | |
|
|
| - xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X |
|
|
| - xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X |
|
|
| - jf_info | uint64 | Detector info field | | |
|
|
| - receiver_aq_dev_delay | uint64 | Receiver internal delay | | |
|
|
| - receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | | |
|
|
| - storage_cell | uint64 | Storage cell number | | |
|
|
| - saturated_pixel_count | uint64 | Saturated pixel count | | |
|
|
| - error_pixel_count | uint64 | Error pixel count | | |
|
|
| - strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | |
|
|
| - data_collection_efficiency | float | Image collection efficiency \[\] | | |
|
|
| - bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] (with solid angle/polarization corrections, if applied) | | X |
|
|
| - resolution_estimate | float | Diffraction resolution estimation | | X |
|
|
| - adu_histogram | Array(uint64) | ADU histogram | | X |
|
|
| - roi_integrals | object | Results of ROI calculation | | X |
|
|
| - - sum | int64 | Sum of pixels in ROI area \[photons\] | | |
|
|
| - - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | |
|
|
| - - pixels | uint64 | Valid pixels in ROI area | | |
|
|
| - - max_count | int64 | Highest count in ROI area \[photons\] | | |
|
|
|
|
## End message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format |
|
|
|----------------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
|
|
| type | String | value "end" | X |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
|
|
| end_date | string | Approximate end date | |
|
|
| max_image_number | uint64 | Number of image with the highest number; counted from 1 to distinguish zero images and one image | |
|
|
| images_collected | uint64 | Number of images collected | |
|
|
| images_sent_to_write | uint64 | Number of images sent to writer; if writer queues were full, it is possible this is less than images collected | |
|
|
| data_collection_efficiency | float | Overall network packets collected / network packets expected | |
|
|
| az_int_result | Map(text->Array(float)) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | |
|
|
| adu_histogram | Map(text->Array(uint64)) | ADU values histogram | |
|
|
| adu_histogram_bin_width | uint64 | Width of bins in the above histogram \[ADU\] | |
|
|
| max_receiver_delay | uint64 | Internal performance of Jungfraujoch | |
|
|
| bkg_estimate | float | Mean background estimate for the whole run | |
|
|
| indexing_rate | float | Mean indexing rate for the whole run | |
|
|
| unit_cell | object (optional) | Unit cell of the system, based on the actual experiment: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | |
|
|
| rotation_lattice_type | object | Bravais lattice classification of the total rotation solution over the run, if available; same schema as `lattice_type` | |
|
|
| - centering | string | One-letter centering code: P, A, B, C, I, F, or R | |
|
|
| - niggli_class | int64 | Integer identifier for the Niggli-reduced Bravais class | |
|
|
| - system | string | Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic | |
|
|
| rotation_lattice | Array(9 * float) | Real-space lattice basis, flattened 3x3 in row-major order | |
|
|
| rotation_extra_lattices | Array(Array(9*float)) | Additional indexed lattices (orientation variants); present only if found | |
|
|
| data_collection_efficiency_image | Array(float) | Per-image data collection efficiency. Missing values are encoded as 0 or 1 depending on producer context | |
|
|
| spot_count | Array(int32) | Per-image spot count | |
|
|
| spot_count_ice_ring | Array(int32) | Per-image number of spots within identified ice-ring resolution ranges | |
|
|
| spot_count_low_res | Array(int32) | Per-image number of low-resolution spots | |
|
|
| spot_count_indexed | Array(int32) | Per-image number of spots fitting indexing solution | |
|
|
| image_indexed | Array(uint8) | Per-image indexing result; 0 = not indexed, nonzero = indexed | |
|
|
| v_bkg_estimate | Array(float) | Per-image background estimate | |
|
|
| ice_ring_score | Array(float) | Per-image strongest ice-ring intensity over the smooth radial background (1 = no ice) | |
|
|
| spot_count_ice_control | Array(float) | Per-image spot count in the ice-free flanks beside the hexagonal rings, rescaled to the ring bands' q width | |
|
|
| ice_ring_score_mean | float | Mean ice-ring score for the whole run (1 = no ice) | |
|
|
| profile_radius | Array(float) | Per-image profile radius \[Angstrom^-1\] | |
|
|
| mosaicity | Array(float) | Per-image mosaicity \[degree\] | |
|
|
| bFactor | Array(float) | Per-image estimated B-factor \[Angstrom^2\] | |
|
|
| resolution_estimate | Array(float) | Per-image diffraction resolution estimate \[Angstrom\] | |
|
|
| min_viable_pixel_value | Array(int64) | Per-image minimum valid pixel value, excluding error/saturated pixels | |
|
|
| max_viable_pixel_value | Array(int64) | Per-image maximum valid pixel value, excluding error/saturated pixels | |
|
|
| saturated_pixel_count | Array(int32) | Per-image saturated pixel count | |
|
|
| error_pixel_count | Array(int32) | Per-image error pixel count | |
|
|
| image_scale_factor | Array(float) | Per-image scale factor, if scaling/merging was performed | |
|
|
| integrated_reflections | Array(int32) | Per-image count of integrated reflections | |
|
|
| indexed_lattice_count | Array(int32) | Per-image count of indexed lattices | |
|
|
| niggli_class | Array(uint8) | Per-image Niggli class identifier for indexed images; 0 if unavailable | |
|
|
| pixel_sum | Array(int64) | Per-image sum of all valid pixels, excluding error/saturated pixels | |
|
|
| image_scale_mosaicity | Array(float) | Scaling result: Image scale mosaicity \[deg\] | |
|
|
| image_scale_cc | Array(float) | Scaling result: Image scale CC | |
|
|
|
|
End-message vector fields are optional. When present, they provide master-file summary data so readers can inspect scan-level and per-image analysis results without opening every linked data file. Missing optional per-image values are encoded by the producer as zero unless otherwise noted.
|
|
|
|
## Calibration message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format |
|
|
|--------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
|
|
| type | String | value "calibration" | |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
|
|
| data | Map(string -> Image) | Calibration map (only single pedestal array per message) | |
|
|
|
|
## User data
|
|
In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch.
|
|
For this reason two fields can be provided: `header_appendix` (sent with start message) and `image_appendix` (send with image message).
|
|
To increase flexibility, both appendices can contain any valid JSON message.
|
|
These appendices are serialized into string and stored in CBOR messages as `user_data`.
|
|
|
|
Notably for start message, `user_data` can contain more information (non-DECTRIS compliant metadata).
|
|
Therefore `user_data` is serialized by Jungfraujoch as CBOR object. There is member `user` which contains `header_appendix` defined in OpenAPI of Jungfraujoch.
|
|
|
|
### Notes on images and compression
|
|
|
|
- Images are encoded as DECTRIS MultiDimArray with typed array tags:
|
|
- For RGB: shape \[3, height, width\], type: u8
|
|
- For grayscale: shape \[height, width\], type according to bit depth and sign (e.g., uint16 LE)
|
|
- Compression:
|
|
- Uncompressed: raw CBOR byte string
|
|
- Bitshuffle+LZ4: tag with \["bslz4", elem_size, bytes\]
|
|
- Bitshuffle+Zstandard: tag with \["bszstd", elem_size, bytes\]
|
|
|
|
### Notes on typed arrays
|
|
|
|
Jungfraujoch uses RFC 8746-style typed byte-string tags for compact numeric arrays.
|
|
|
|
Common tags used in this protocol include:
|
|
|
|
- float32 little-endian arrays for `Array(float)`
|
|
- uint8 arrays for compact boolean/integer flags such as `image_indexed`
|
|
- int32 little-endian arrays for per-image counts
|
|
- int64 little-endian arrays for large per-image integer values
|
|
- uint64 little-endian arrays for histograms |