**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
72 KiB
CBOR messages
To communicate between FPGA-equipped receiver system and writers, Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel). The protocol is based on and compatible with DECTRIS Stream2. There are minor differences at the moment:
- LZ4 alone is not allowed; Bitshuffle+LZ4 and Bitshuffle+Zstandard are allowed
- Few fields are currently absent
- Extra fields are present beyond DECTRIS standard
- There are calibration and metadata messages defined beyond DECTRIS specification
Start message
| Field name | Type | Description | Present in DECTRIS format |
|---|---|---|---|
| type | String | value "start" | X |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| detector_distance | float | Detector distance [m] | |
| detector_translation | Array(float) | Detector translation vector [m] | X |
| beam_center_x | float | Beam center in X direction [pixels] | X |
| beam_center_y | float | Beam center in Y direction [pixels] | X |
| countrate_correction_enabled | bool | Countrate correction enabled | X |
| flatfield_enabled | bool | Flatfield enabled | X |
| number_of_images | uint64 | Number of images in the series | X |
| image_size_x | uint64 | Image width [pixels] | X |
| image_size_y | uint64 | Image height [pixels] | X |
| mirror_y | bool | Whether the assembled image is mirrored in Y relative to the detector's raw readout order. True is the MX convention - row 0 at the top of the detector seen from the sample - and is what absence of the key means | |
| incident_energy | float | X-ray energy [eV] | X |
| incident_wavelength | float | X-ray wavelength [Angstrom] | X |
| incident_wavelength_spread | float (optional) | FWHM of the X-ray wavelength distribution [Angstrom] (NXmx incident_wavelength_spread); omitted when the beam is monochromatic | |
| frame_time | float | Frame time, if multiple frames per trigger [s] | X |
| count_time | float | Exposure time [s] | X |
| saturation_value | int64 | Maximum valid sample value | X |
| error_value | int64 (optional) | Value used in images to describe pixels that are in error state or missing | |
| pixel_size_x | float | Pixel width [m] | X |
| pixel_size_y | float | Pixel height [m] | X |
| sensor_thickness | float | Sensor thickness [m] | X |
| sensor_material | string | Sensor material | X |
| arm_date | date | Approximate date of arming | X |
| pixel_mask_enabled | bool | Pixel mask applied on images | X |
| detector_description | string | Name of the detector | X |
| detector_serial_number | string | Detector serial number | X |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
| fluorescence | object (optional) | X-ray fluorescence spectrum collected at start | |
| - energy | Array(float) | Energy of measuring point [eV] | |
| - data | Array(float) | Fluorescence scan result data [arbitrary units]; must be strictly the same length as energy |
|
| goniometer | Map | Definition of rotation axis (optional) | X |
- AXIS |
string | Rotation axis name (e.g. omega) - only one axis is supported in Jungfraujoch | X |
| - - increment | float | Rotation axis increment (per image) in degree [deg] | X |
| - - start | float | Rotation axis start angle [deg] | X |
| - - axis | Array(float) | Vector for the rotation axis | |
| - - helical_step | Array(float) | Translation for helical scan for 1 image [m] | |
| - - screening_wedge | Array(float) | Wedge for screening [deg] (increment would correspond to difference between screening points) | |
| grid_scan | object | Grid scan definition (optional). May be sent together with goniometer: a grid is often collected at a given head position, recorded as a stationary axis (step 0) |
|
| - n_fast | uint64 | Number of elements along fast axis | |
| - n_slow | uint64 | Number of elements along slow axis | |
| - step_x_axis | float | Step along X axis, can be negative [m] | |
| - step_y_axis | float | Step along Y axis, can be negative [m] | |
| - snake_scan | bool | Snake scan (rows alternate direction) | |
| - vertical_scan | bool | Vertical scan (enabled: fast direction = Y, disabled: fast direction = X) | |
| jungfrau_conversion_enabled | bool (optional) | Applying JUNGFRAU pixel conversion (to photons or keV) | |
| jungfrau_conversion_factor | float (optional) | Factor used for JUNGFRAU conversion [eV] | |
| geometry_transformation_enabled | bool (optional) | Transformation from detector module geometry (512x1024) to full detector geometry | |
| pixel_mask | Map(string -> Image) | Pixel mask - multiple in case of storage cells | X |
| channels | Array(string) | List of image channels | X |
| max_spot_count | uint64 | Maximum number of spots identified in spot finding | |
| max_extra_lattices | uint64 | Maximum number of extra lattices | |
| storage_cell_number | uint64 (optional) | Number of storage cells used by JUNGFRAU | |
| storage_cell_delay | Rational | Delay of storage cells in JUNGFRAU | |
| threshold_energy | Map(string -> float) | Per-channel threshold energy [eV] (map of channel name to value) | |
| image_dtype | string | Pixel type of the image data: uint8, uint16, uint32 (DECTRIS), plus int8, int16, int32 as a Jungfraujoch extension. Sole wire encoding of both the bit depth and the sign, and must agree with the per-image typed-array tag |
X |
| unit_cell | object (optional) | Unit cell of the system: a, b, c [angstrom] and alpha, beta, gamma [degree] | |
| az_int_q_bin_count | uint64 | Number of azimuthal integration bins in the radial direction | |
| az_int_phi_bin_count | uint64 | Number of azimuthal integration bins in the phi angle direction | |
| az_int_bin_to_q | Array(float) | Q value for each azimuthal integration bin [angstrom^-1] | |
| az_int_bin_to_two_theta | Array(float) | Two theta angle value for each azimuthal integration bin [deg] | |
| az_int_bin_to_phi | Array(float) | Phi value for each azimuthal integration bin [deg] | |
| az_int_map | Image | Mapping between pixel and bin number | |
| summation | uint64 | Factor of frame summation | |
| user_data | string | JSON serialized to string that can contain the following fields (all fields are optional): | X |
| - file_prefix | string | File prefix | |
| - images_per_file | uint64 | Number of images written per file | |
| - images_per_trigger | uint64 | Number of images collected per trigger | |
| - source_name | string | Facility name | |
| - source_type | string | Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser") | |
| - instrument_name | string | Instrument name | |
| - sample_name | string | Name of the sample | |
| - user | any valid JSON | Value of header_appendix provided at collection start to Jungfraujoch | |
| - attenuator_transmission | float | Attenuator transmission [] | |
| - total_flux | float | Total flux [ph/s] | |
| - space_group_number | uint64 | Space group number | |
| - summation_mode | string | Summation mode (internal|fpga|cpu) | |
| - overwrite | bool | Overwrite existing HDF5 files | |
| - file_format | int | File writer format: 0 = only data files, 1 = NXmx legacy soft links, 2 = NXmx VDS, 3 = NXmx integrated, 4 = CBF, 5 = TIFF, 6 = no file written | |
| - roi | Array(object) | ROI configurations; each element is one of: | |
| type "box": xmin, xmax, ymin, ymax (numbers) | |||
| type "circle": r, x, y (numbers) | |||
| type "azim": qmin, qmax (numbers); optional phi_min, phi_max (numbers, deg) for an angular sector | |||
| - gain_file_names | Array(string) | Names of JUNGFRAU gain files used for the current detector | |
| - write_master_file | bool | With multiple sockets, it selects which socket will provide master file | |
| - write_images | bool | Write images in the HDF5 file (if false, will only write metadata) | |
| - data_reduction_factor_serialmx | uint64 | Data reduction factor for serial MX | |
| - experiment_group | string | ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number | |
| - jfjoch_release | string | Jungfraujoch release number | |
| - socket_number | uint64 | Number of ZeroMQ socket (on jfjoch_broker side) used for transmission |
|
| - bit_depth_readout | uint64 | Bit depth of the stored image (see note below), copied to NXmx bit_depth_readout |
|
| - underload_value | int64 | Lowest valid value; copied to NXmx underload_value. 0 for an unsigned image, INTx_MIN + 1 for a signed one |
|
| - writer_notification_zmq_addr | string | ZeroMQ address to inform jfjoch_broker about writers that finished operation |
|
| - xfel_pulse_id | uint64 | Pulse IDs are recorded for images | |
| - ring_current_mA | float | Ring current at the start of the measurement | |
| - sample_temperature_K | float | Sample temperature [K] | |
| - detect_ice_rings | bool | Ice ring detection feature is enabled | |
| - indexing_algorithm | string | Indexing algorithm used on-the-fly; allowed values: ffbidx, fft, fftw, none | |
| - geom_refinement_algorithm | string | Post-indexing detector geometry refinement algorithm; allowed values: none, beam_center | |
| - poni_rot1 | float | Tilt of the detector rot1 according to PyFAI PONI convention [rad] | |
| - poni_rot2 | float | Tilt of the detector rot2 according to PyFAI PONI convention [rad] | |
| - poni_rot3 | float | Tilt of the detector rot3 according to PyFAI PONI convention [rad] |
See DECTRIS documentation for definition of Image as MultiDimArray with optional compression.
Image message
| Field name | Type | Description | Present in DECTRIS format | Optional |
|---|---|---|---|---|
| type | String | value "image" | X | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | ||
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
| image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | |
| original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | ||
| real_time | Rational | Exposure time | X | |
| start_time | Rational | Exposure start time (highly approximate) | X | |
| end_time | Rational | Exposure end time (highly approximate) | X | |
| spots | Array(object) | Spots: | ||
| - x | float | observed position in x (pixels) | ||
| - y | float | observed position in y (pixels) | ||
| - I | float | intensity (photons) | ||
| - maxc | int64 | max count (photons) | ||
| - ice_ring | bool | spot in resolution range for ice rings | ||
| - indexed | bool | indexed solution | ||
| - latt | int64 | Lattice to which the peak belongs (negative number = not indexed) | ||
| - image | int64 | image number the spot belongs to | ||
| - h | int64 | Miller index (indexed spots only) | ||
| - k | int64 | Miller index (indexed spots only) | ||
| - l | int64 | Miller index (indexed spots only) | ||
| - dist_ewald | float | distance to Ewald sphere [Angstrom^-1] (indexed spots only) | ||
| reflections | Array(object) | Reflections: | ||
| - h | int64 | Miller index | ||
| - k | int64 | Miller index | ||
| - l | int64 | Miller index | ||
| - x | float | prediced position in x (pixels) | ||
| - y | float | predicted position in y (pixels) | ||
| - obs_x | float | observed position in x (pixels) | ||
| - obs_y | float | observed position in y (pixels) | ||
| - d | float | resolution [Angstrom] | ||
| - I | float | integrated intensity (photons) | ||
| - bkg | float | mean background value (photons) | ||
| - var_bkg | float | non-signal (background) part of sigma^2, carried to the merge (photons^2) | ||
| - sigma | float | standard deviation, estimated from counting statistics (photons) | ||
| - image | float | image number (present for each spot) | ||
| - rp | float | Distance to Ewald sphere [Angstrom^-1] | ||
| - rlp | float | Reciprocal Lorentz and polarization corrections | ||
| - partiality | float | Partiality of the reflection | ||
| - phi | float | phi angle from XDS: difference from middle of current frame, not absolute [deg] | ||
| - zeta | float | Lorentz zeta factor (reciprocal-space geometry term) | ||
| - image_scale_corr | float | Per-image scale correction; I_true = image_scale_corr * I | ||
| spot_count | uint64 | Spot count | ||
| spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | ||
| spot_count_low_res | uint64 | Number of spots in low resolution (prior to filtering) | ||
| spot_count_indexed | uint64 | Number of spots which fit indexing solution within a given tolerance | ||
| az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | ||
| NaN is used for empty bins and has to be taken care by the receiver | ||||
| az_int_profile_std | Array(float) | Standard deviation for azimuthal integration. (NaN for less than 2 samples) | ||
| az_int_profile_count | Array(uint64) | Number of pixels contributing to azimuthal bin | ||
| indexing_result | bool | Indexing successful | ||
| indexing_lattice_count | int64 | Number of indexing lattices found for this image | ||
| indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | X | |
| indexing_extra_lattices | Array(Array(9*float)) | Additional indexed lattices (orientation variants); present only if found | ||
| indexing_unit_cell | object | Indexing result unit cell: a, b, c [angstrom] and alpha, beta, gamma [degree]; present only if indexed | X | |
| Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | ||||
| profile_radius | float | Profile radius of the image - describes distance of observed reflections from the Ewald sphere [Angstrom^-1] | ||
| integrated_reflections | int64 | Count of integrated reflections | ||
| mosaicity | float | Angular range of spots in image from a rotation scan [degree] | ||
| b_factor | float | Estimated B-factor (Angstrom^2) | ||
| compression_time | float | Time spent on compression/decompressing image [s] | ||
| preprocessing_time | float | Time spent on preparing the image for analysis [s] | ||
| azint_time | float | Time spent on azimuthal integration [s] | ||
| spot_finding_time | float | Time spent on spot finding [s] | ||
| indexing_time | float | Time spent on indexing [s] | ||
| refinement_time | float | Time spent on refinement of indexing solution and experimental geometry [s] | ||
| index_analysis_time | float | Time spent on analyzing idnexing solution, calculating profile radius and mosaicity [s] | ||
| bragg_prediction_time | float | Time spent on predicting Bragg spots [s] | ||
| integration_time | float | Time spent on Bragg integration [s] | ||
| image_scale_time | float | Time spent on on-the-fly scaling [s] | ||
| processing_time | float | Total processing time [s] | ||
| xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | X | |
| xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | X | |
| lattice_type | object | Bravais lattice classification of the indexing result (present only if available) | X | |
| - centering | string | One-letter centering code: P, A, B, C, I, F, or R | ||
| - niggli_class | int64 | Integer identifier for the Niggli-reduced Bravais class | ||
| - system | string | Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic | ||
| jf_info | uint64 | Detector info field | ||
| receiver_aq_dev_delay | uint64 | Receiver internal delay | ||
| receiver_free_send_buf | uint64 | Receiver internal number of available buffer locations | ||
| receiver_buf_in_sending | uint64 | Receiver internal number of buffer locations currently in sending/writing | ||
| receiver_buf_in_preparation | uint64 | Receiver internal number of buffer locations currently in processing | ||
| storage_cell | uint64 | Storage cell number | ||
| saturated_pixel_count | uint64 | Saturated pixel count | ||
| pixel_sum | uint64 | Sum of all pixels, excl. error and saturation | ||
| error_pixel_count | uint64 | Error pixel count | ||
| strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | ||
| min_viable_pixel_value | int64 | Minimal pixel value, excl. error and saturation | ||
| max_viable_pixel_value | int64 | Maximal pixel value, excl. error and saturation | ||
| resolution_estimate | float | Resolution the merged data are predicted to reach, from this image's spots alone [Angstrom] | X | |
| data_collection_efficiency | float | Image collection efficiency [] | ||
| packets_expected | uint64 | Number of packets expected per image (in units of 2 kB) | ||
| packets_received | uint64 | Number of packets received per image (in units of 2 kB) | ||
| bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A [photons] | ||
| ice_ring_score | float | Strongest hexagonal-ice ring intensity over the smooth radial background (1 = no ice) | ||
| spot_count_ice_control | float | Spots in the ice-free flanks beside the hexagonal rings, rescaled to the ring bands' own q width (control for spot_count_ice_rings) | ||
| beam_corr_x | float | Beam center correction X applied during processing [pixel] | X | |
| beam_corr_y | float | Beam center correction Y applied during processing [pixel] | X | |
| image_scale_factor | float | Scaling result: Image scale factor (g) | X | |
| image_scale_mosaicity | float | Scaling result: Image scale mosaicity [deg] | X | |
| image_scale_cc | float | Scaling result: Image scale CC | X | |
| adu_histogram | Array(uint64) | ADU histogram | ||
| roi_integrals | object | Results of ROI calculation | X | |
| - sum | int64 | Sum of pixels in ROI area [photons] | ||
| - sum_square | int64 | Sum of squares of pixels in ROI area [photons] | ||
| - pixels | uint64 | Valid pixels in ROI area | ||
| - max_count | int64 | Highest count in ROI area [photons] | ||
| - x_weighted_sum | int64 | ROI pixel X position multiplied by photon count [photons * pixels] | ||
| - y_weighted_sum | int64 | ROI pixel Y position multiplied by photon count [photons * pixels] | ||
| user_data | string | Optional user defined text information - this is image_appendix serialized to JSON format | X | |
| data | Map(string -> Image) | Image | X |
Metadata message
| Field name | Type | Description | Present in DECTRIS format | Optional |
|---|---|---|---|---|
| type | String | value "metadata" | X | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | ||
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
| images | Array(object) | Array of images (order and size of the array are not guaranteed) | X | |
| - image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | |
| - original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | ||
| - real_time | Rational | Exposure time | X | |
| - start_time | Rational | Exposure start time (highly approximate) | X | |
| - end_time | Rational | Exposure end time (highly approximate) | X | |
| - spot_count | uint64 | Spot count | ||
| - spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | ||
| - az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | ||
| - indexing_result | bool | Indexing successful | ||
| - indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | X | |
| - indexing_unit_cell | object | Indexing result unit cell: a, b, c [angstrom] and alpha, beta, gamma [degree]; present only if indexed | X | |
| Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | ||||
| - xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | X | |
| - xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | X | |
| - jf_info | uint64 | Detector info field | ||
| - receiver_aq_dev_delay | uint64 | Receiver internal delay | ||
| - receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | ||
| - storage_cell | uint64 | Storage cell number | ||
| - saturated_pixel_count | uint64 | Saturated pixel count | ||
| - error_pixel_count | uint64 | Error pixel count | ||
| - strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | ||
| - data_collection_efficiency | float | Image collection efficiency [] | ||
| - bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A [photons] (with solid angle/polarization corrections, if applied) | X | |
| - resolution_estimate | float | Predicted merged resolution, from spots alone | X | |
| - adu_histogram | Array(uint64) | ADU histogram | X | |
| - roi_integrals | object | Results of ROI calculation | X | |
| - - sum | int64 | Sum of pixels in ROI area [photons] | ||
| - - sum_square | int64 | Sum of squares of pixels in ROI area [photons] | ||
| - - pixels | uint64 | Valid pixels in ROI area | ||
| - - max_count | int64 | Highest count in ROI area [photons] |
End message
| Field name | Type | Description | Present in DECTRIS format |
|---|---|---|---|
| type | String | value "end" | X |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
| end_date | string | Approximate end date | |
| max_image_number | uint64 | Number of image with the highest number; counted from 1 to distinguish zero images and one image | |
| transformations | Array(object) (optional) | Sample transformation chain in mounting order, base first. Each element mirrors a NeXus NXtransformations axis: name, transformation_type (rotation/translation), units, vector, offset, depends_on (the axis this one is mounted on, empty for the base), and values - a single number for an axis that does not move, otherwise one per image. An ARRAY because the order matters and a CBOR map has none. Optional: when absent the writer builds the same chain from the start message. It is in the END message because a producer may want to report positions that were measured rather than commanded, which are only known once the run is over |
|
| images_collected | uint64 | Number of images collected | |
| images_sent_to_write | uint64 | Number of images sent to writer; if writer queues were full, it is possible this is less than images collected | |
| data_collection_efficiency | float | Overall network packets collected / network packets expected | |
| az_int_result | Map(text->Array(float)) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | |
| adu_histogram | Map(text->Array(uint64)) | ADU values histogram | |
| adu_histogram_bin_width | uint64 | Width of bins in the above histogram [ADU] | |
| max_receiver_delay | uint64 | Internal performance of Jungfraujoch | |
| bkg_estimate | float | Mean background estimate for the whole run | |
| indexing_rate | float | Mean indexing rate for the whole run | |
| unit_cell | object (optional) | Unit cell of the system, based on the actual experiment: a, b, c [angstrom] and alpha, beta, gamma [degree] | |
| rotation_lattice_type | object | Bravais lattice classification of the total rotation solution over the run, if available; same schema as lattice_type |
|
| - centering | string | One-letter centering code: P, A, B, C, I, F, or R | |
| - niggli_class | int64 | Integer identifier for the Niggli-reduced Bravais class | |
| - system | string | Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic | |
| rotation_lattice | Array(9 * float) | Real-space lattice basis, flattened 3x3 in row-major order | |
| rotation_extra_lattices | Array(Array(9*float)) | Additional indexed lattices (orientation variants); present only if found | |
| data_collection_efficiency_image | Array(float) | Per-image data collection efficiency. Missing values are encoded as 0 or 1 depending on producer context | |
| spot_count | Array(int32) | Per-image spot count | |
| spot_count_ice_ring | Array(int32) | Per-image number of spots within identified ice-ring resolution ranges | |
| spot_count_low_res | Array(int32) | Per-image number of low-resolution spots | |
| spot_count_indexed | Array(int32) | Per-image number of spots fitting indexing solution | |
| image_indexed | Array(uint8) | Per-image indexing result; 0 = not indexed, nonzero = indexed | |
| v_bkg_estimate | Array(float) | Per-image background estimate | |
| ice_ring_score | Array(float) | Per-image strongest ice-ring intensity over the smooth radial background (1 = no ice) | |
| spot_count_ice_control | Array(float) | Per-image spot count in the ice-free flanks beside the hexagonal rings, rescaled to the ring bands' q width | |
| ice_ring_score_mean | float | Mean ice-ring score for the whole run (1 = no ice) | |
| profile_radius | Array(float) | Per-image profile radius [Angstrom^-1] | |
| mosaicity | Array(float) | Per-image mosaicity [degree] | |
| bFactor | Array(float) | Per-image estimated B-factor [Angstrom^2] | |
| resolution_estimate | Array(float) | Per-image predicted merged resolution, from spots alone [Angstrom] | |
| min_viable_pixel_value | Array(int64) | Per-image minimum valid pixel value, excluding error/saturated pixels | |
| max_viable_pixel_value | Array(int64) | Per-image maximum valid pixel value, excluding error/saturated pixels | |
| saturated_pixel_count | Array(int32) | Per-image saturated pixel count | |
| error_pixel_count | Array(int32) | Per-image error pixel count | |
| image_scale_factor | Array(float) | Per-image scale factor, if scaling/merging was performed | |
| integrated_reflections | Array(int32) | Per-image count of integrated reflections | |
| indexed_lattice_count | Array(int32) | Per-image count of indexed lattices | |
| niggli_class | Array(uint8) | Per-image Niggli class identifier for indexed images; 0 if unavailable | |
| pixel_sum | Array(int64) | Per-image sum of all valid pixels, excluding error/saturated pixels | |
| image_scale_mosaicity | Array(float) | Scaling result: Image scale mosaicity [deg] | |
| image_scale_cc | Array(float) | Scaling result: Image scale CC |
End-message vector fields are optional. When present, they provide master-file summary data so readers can inspect scan-level and per-image analysis results without opening every linked data file. Missing optional per-image values are encoded by the producer as zero unless otherwise noted.
Calibration message
| Field name | Type | Description | Present in DECTRIS format |
|---|---|---|---|
| type | String | value "calibration" | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| data | Map(string -> Image) | Calibration map (only single pedestal array per message) |
User data
In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch.
For this reason two fields can be provided: header_appendix (sent with start message) and image_appendix (send with image message).
To increase flexibility, both appendices can contain any valid JSON message.
These appendices are serialized into string and stored in CBOR messages as user_data.
Notably for start message, user_data can contain more information (non-DECTRIS compliant metadata).
Therefore user_data is serialized by Jungfraujoch as CBOR object. There is member user which contains header_appendix defined in OpenAPI of Jungfraujoch.
Notes on images and compression
- Images are encoded as DECTRIS MultiDimArray with typed array tags:
- For RGB: shape [3, height, width], type: u8
- For grayscale: shape [height, width], type according to bit depth and sign (e.g., uint16 LE)
- Compression:
- Uncompressed: raw CBOR byte string
- Bitshuffle+LZ4: tag with ["bslz4", elem_size, bytes]
- Bitshuffle+Zstandard: tag with ["bszstd", elem_size, bytes]
Notes on typed arrays
Jungfraujoch uses RFC 8746-style typed byte-string tags for compact numeric arrays.
Common tags used in this protocol include:
- float32 little-endian arrays for
Array(float) - uint8 arrays for compact boolean/integer flags such as
image_indexed - int32 little-endian arrays for per-image counts
- int64 little-endian arrays for large per-image integer values
- uint64 little-endian arrays for histograms