282 lines
53 KiB
Markdown
282 lines
53 KiB
Markdown
# CBOR messages
|
|
|
|
To communicate between FPGA-equipped receiver system and writers,
|
|
Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel).
|
|
The protocol is based on and compatible with [DECTRIS Stream2](https://github.com/dectris/documentation/tree/main/stream_v2).
|
|
There are minor differences at the moment:
|
|
|
|
* LZ4 alone is not allowed; Bitshuffle+LZ4 and Bitshuffle+Zstandard are allowed
|
|
* Few fields are currently absent
|
|
* Extra fields are present beyond DECTRIS standard
|
|
* There are calibration and metadata messages defined beyond DECTRIS specification
|
|
|
|
## Start message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format |
|
|
|----------------------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
|
|
| type | String | value "start" | X |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
|
|
| detector_distance | float | Detector distance \[m\] | |
|
|
| detector_translation | Array(float) | Detector translation vector \[m\] | X |
|
|
| beam_center_x | float | Beam center in X direction \[pixels\] | X |
|
|
| beam_center_y | float | Beam center in Y direction \[pixels\] | X |
|
|
| countrate_correction_enabled | bool | Countrate correction enabled | X |
|
|
| flatfield_enabled | bool | Flatfield enabled | X |
|
|
| number_of_images | uint64 | Number of images in the series | X |
|
|
| image_size_x | uint64 | Image width \[pixels\] | X |
|
|
| image_size_y | uint64 | Image height \[pixels\] | X |
|
|
| incident_energy | float | X-ray energy \[eV\] | X |
|
|
| incident_wavelength | float | X-ray wavelength \[Angstrom\] | X |
|
|
| frame_time | float | Frame time, if multiple frames per trigger \[s\] | X |
|
|
| count_time | float | Exposure time \[s\] | X |
|
|
| saturation_value | int64 | Maximum valid sample value | X |
|
|
| error_value | int64 (optional) | Value used in images to describe pixels that are in error state or missing | |
|
|
| pixel_size_x | float | Pixel width \[m\] | X |
|
|
| pixel_size_y | float | Pixel height \[m\] | X |
|
|
| sensor_thickness | float | Sensor thickness \[m\] | X |
|
|
| sensor_material | string | Sensor material | X |
|
|
| arm_date | date | Approximate date of arming | X |
|
|
| pixel_mask_enabled | bool | Pixel mask applied on images | X |
|
|
| detector_description | string | Name of the detector | X |
|
|
| detector_serial_number | string | Detector serial number | X |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
|
|
| fluorescence | object (optional) | X-ray fluorescence spectrum collected at start | |
|
|
| - energy | Array(float) | Energy of measuring point \[eV\] | |
|
|
| - data | Array(float) | Fluorescence scan result `data` \[arbitrary units\]; must be strictly the same length as energy | |
|
|
| goniometer | Map | Definition of rotation axis (optional) | X |
|
|
| - `AXIS` | string | Rotation axis name (e.g. omega) - only one axis is supported in Jungfraujoch | X |
|
|
| - - increment | float | Rotation axis increment (per image) in degree \[deg\] | X |
|
|
| - - start | float | Rotation axis start angle \[deg\] | X |
|
|
| - - axis | Array(float) | Vector for the rotation axis | |
|
|
| - - helical_step | Array(float) | Translation for helical scan for 1 image \[m\] | |
|
|
| - - screening_wedge | Array(float) | Wedge for screening \[deg\] (increment would correspond to difference between screening points) | |
|
|
| grid_scan | object | Grid scan definition (optional and exclusive with rotation axis) | |
|
|
| - n_fast | uint64 | Number of elements along fast axis | |
|
|
| - n_slow | uint64 | Number of elements along slow axis | |
|
|
| - step_x_axis | float | Step along X axis, can be negative \[m\] | |
|
|
| - step_y_axis | float | Step along Y axis, can be negative \[m\] | |
|
|
| - snake_scan | bool | Snake scan (rows alternate direction) | |
|
|
| - vertical_scan | bool | Vertical scan (enabled: fast direction = Y, disabled: fast direction = X) | |
|
|
| jungfrau_conversion_enabled | bool (optional) | Applying JUNGFRAU pixel conversion (to photons or keV) | |
|
|
| jungfrau_conversion_factor | float (optional) | Factor used for JUNGFRAU conversion \[eV\] | |
|
|
| geometry_transformation_enabled | bool (optional) | Transformation from detector module geometry (512x1024) to full detector geometry | |
|
|
| pixel_mask | Map(string -> Image) | Pixel mask - multiple in case of storage cells | X |
|
|
| channels | Array(string) | List of image channels | X |
|
|
| max_spot_count | uint64 | Maximum number of spots identified in spot finding | |
|
|
| storage_cell_number | uint64 (optional) | Number of storage cells used by JUNGFRAU | |
|
|
| storage_cell_delay | Rational | Delay of storage cells in JUNGFRAU | |
|
|
| threshold_energy | float | Threshold energy for EIGER detector \[eV\] | |
|
|
| image_dtype | string | Pixel bit type (e.g. uint16) | X |
|
|
| unit_cell | object (optional) | Unit cell of the system: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | |
|
|
| az_int_q_bin_count | uint64 | Number of azimuthal integration bins in the radial direction | |
|
|
| az_int_phi_bin_count | uint64 | Number of azimuthal integration bins in the phi angle direction | |
|
|
| az_int_bin_to_q | Array(float) | Q value for each azimuthal integration bin \[angstrom^-1\] | |
|
|
| az_int_bin_to_two_theta | Array(float) | Two theta angle value for each azimuthal integration bin \[deg\] | |
|
|
| az_int_bin_to_phi | Array(float) | Phi value for each azimuthal integration bin \[deg\] | |
|
|
| summation | uint64 | Factor of frame summation | |
|
|
| user_data | string | JSON serialized to string that can contain the following fields (all fields are optional): | X |
|
|
| - file_prefix | string | File prefix | |
|
|
| - images_per_file | uint64 | Number of images written per file | |
|
|
| - images_per_trigger | uint64 | Number of images collected per trigger | |
|
|
| - source_name | string | Facility name | |
|
|
| - source_type | string | Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser") | |
|
|
| - instrument_name | string | Instrument name | |
|
|
| - sample_name | string | Name of the sample | |
|
|
| - user | any valid JSON | Value of header_appendix provided at collection start to Jungfraujoch | |
|
|
| - attenuator_transmission | float | Attenuator transmission \[\] | |
|
|
| - total_flux | float | Total flux \[ph/s\] | |
|
|
| - space_group_number | uint64 | Space group number | |
|
|
| - summation_mode | string | Summation mode (internal\|fpga\|cpu) | |
|
|
| - overwrite | bool | Overwrite existing HDF5 files | |
|
|
| - file_format | int | File writer format: 0 = no master file, 1 = soft links, 2 = virtual dataset, 3 = CBF, 4 = TIFF | |
|
|
| - roi | Array(object) | ROI configurations; each element is one of: | |
|
|
| | | type "box": xmin, xmax, ymin, ymax (numbers) | |
|
|
| | | type "circle": r, x, y (numbers) | |
|
|
| | | type "azim": qmin, qmax (numbers) | |
|
|
| - gain_file_names | Array(string) | Names of JUNGFRAU gain files used for the current detector | |
|
|
| - write_master_file | bool | With multiple sockets, it selects which socket will provide master file | |
|
|
| - data_reduction_factor_serialmx | uint64 | Data reduction factor for serial MX | |
|
|
| - experiment_group | string | ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number | |
|
|
| - jfjoch_release | string | Jungfraujoch release number | |
|
|
| - socket_number | uint64 | Number of ZeroMQ socket (on `jfjoch_broker` side) used for transmission | |
|
|
| - bit_depth_readout | uint64 | Bit depth of the detector readout | |
|
|
| - writer_notification_zmq_addr | string | ZeroMQ address to inform `jfjoch_broker` about writers that finished operation | |
|
|
| - xfel_pulse_id | uint64 | Pulse IDs are recorded for images | |
|
|
| - ring_current_mA | float | Ring current at the start of the measurement | |
|
|
| - sample_temperature_K | float | Sample temperature \[K\] | |
|
|
| - detect_ice_rings | bool | Ice ring detection feature is enabled | |
|
|
| - indexing_algorithm | string | Indexing algorithm used on-the-fly; allowed values: ffbidx, fft, fftw, none | |
|
|
| - geom_refinement_algorithm | string | Post-indexing detector geometry refinement algorithm; allowed values: none, beam_center, beam_center_tetragonal | |
|
|
| - poni_rot1 | float | Tilt of the detector rot1 according to PyFAI PONI convention \[rad\] | |
|
|
| - poni_rot2 | float | Tilt of the detector rot2 according to PyFAI PONI convention \[rad\] | |
|
|
| - poni_rot3 | float | Tilt of the detector rot3 according to PyFAI PONI convention \[rad\] | |
|
|
|
|
See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/stream_v2) for definition of Image as MultiDimArray with optional compression.
|
|
|
|
## Image message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format | Optional |
|
|
|----------------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:|
|
|
| type | String | value "image" | X | |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
|
|
| image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | |
|
|
| original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | | |
|
|
| real_time | Rational | Exposure time | X | |
|
|
| start_time | Rational | Exposure start time (highly approximate) | X | |
|
|
| end_time | Rational | Exposure end time (highly approximate) | X | |
|
|
| spots | Array(object) | Spots: | | |
|
|
| - x | float | observed position in x (pixels) | | |
|
|
| - y | float | observed position in y (pixels) | | |
|
|
| - I | float | intensity (photons) | | |
|
|
| - maxc | int64 | max count (photons) | | |
|
|
| - ice_ring | bool | spot in resolution range for ice rings | | |
|
|
| - indexed | bool | indexed solution | | |
|
|
| reflections | Array(object) | Reflections: | | |
|
|
| - h | int64 | Miller index | | |
|
|
| - k | int64 | Miller index | | |
|
|
| - l | int64 | Miller index | | |
|
|
| - x | float | position in x (pixels) | | |
|
|
| - y | float | position in y (pixels) | | |
|
|
| - d | float | resolution \[angstrom\] | | |
|
|
| - I | float | integrated intensity (photons) | | |
|
|
| - bkg | float | mean background value (photons) | | |
|
|
| - sigma | float | standard deviation, estimated from counting statistics (photons) | | |
|
|
| - image | float | image number | | |
|
|
| spot_count | uint64 | Spot count | | |
|
|
| spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | | |
|
|
| spot_count_low_res | uint64 | Number of spots in low resolution (prior to filtering) | | |
|
|
| spot_count_indexed | uint64 | Number of spots which fit indexing solution within a given tolerance | | |
|
|
| az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | |
|
|
| indexing_result | bool | Indexing successful | | |
|
|
| indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | | X |
|
|
| indexing_unit_cell | object | Indexing result unit cell: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\]; present only if indexed | | X |
|
|
| | | Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | | |
|
|
| profile_radius | float | Profile radius of the image - describes distance of observed reflections from the Ewald sphere (Angstrom^-1) | | |
|
|
| b_factor | float | Estimated B-factor (Angstrom^2) | | |
|
|
| indexing_time | float | Time spent on indexing \[s\] | | |
|
|
| processing_time | float | Total processing time \[s\] | | |
|
|
| xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X |
|
|
| xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X |
|
|
| lattice_type | object | Bravais lattice classification of the indexing result (present only if available) | | X |
|
|
| - centering | string | One-letter centering code: P, A, B, C, I, F, or R | | |
|
|
| - niggli_class | int64 | Integer identifier for the Niggli-reduced Bravais class | | |
|
|
| - system | string | Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic | | |
|
|
| jf_info | uint64 | Detector info field | | |
|
|
| receiver_aq_dev_delay | uint64 | Receiver internal delay | | |
|
|
| receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | | |
|
|
| storage_cell | uint64 | Storage cell number | | |
|
|
| saturated_pixel_count | uint64 | Saturated pixel count | | |
|
|
| pixel_sum | uint64 | Sum of all pixels, excl. error and saturation | | |
|
|
| error_pixel_count | uint64 | Error pixel count | | |
|
|
| strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | |
|
|
| min_viable_pixel_value | int64 | Minimal pixel value, excl. error and saturation | | |
|
|
| max_viable_pixel_value | int64 | Maximal pixel value, excl. error and saturation | | |
|
|
| resolution_estimate | float | Diffraction resolution estimation \[Angstrom\] | | X |
|
|
| data_collection_efficiency | float | Image collection efficiency \[\] | | |
|
|
| packets_expected | uint64 | Number of packets expected per image (in units of 2 kB) | | |
|
|
| packets_received | uint64 | Number of packets received per image (in units of 2 kB) | | |
|
|
| bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] | | |
|
|
| beam_center_x | float | Beam center X from post-indexing refinement \[pixel\] | | X |
|
|
| beam_center_y | float | Beam center Y from post-indexing refinement \[pixel\] | | X |
|
|
| beam_corr_x | float | Beam center correction X applied during processing \[pixel\] | | X |
|
|
| beam_corr_y | float | Beam center correction Y applied during processing \[pixel\] | | X |
|
|
| adu_histogram | Array(uint64) | ADU histogram | | |
|
|
| roi_integrals | object | Results of ROI calculation | | X |
|
|
| - sum | int64 | Sum of pixels in ROI area \[photons\] | | |
|
|
| - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | |
|
|
| - pixels | uint64 | Valid pixels in ROI area | | |
|
|
| - max_count | int64 | Highest count in ROI area \[photons\] | | |
|
|
| - x_weighted_sum | int64 | ROI pixel X position multiplied by photon count \[photons * pixels\] | | |
|
|
| - y_weighted_sum | int64 | ROI pixel Y position multiplied by photon count \[photons * pixels\] | | |
|
|
| user_data | string | Optional user defined text information - this is image_appendix serialized to JSON format | X | |
|
|
| data | Map(string -> Image) | Image | X | |
|
|
|
|
## Metadata message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format | Optional |
|
|
|------------------------------|------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:|
|
|
| type | String | value "metadata" | X | |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
|
|
| images | Array(object) | Array of images (order and size of the array are not guaranteed) | X | |
|
|
| - image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | |
|
|
| - original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | | |
|
|
| - real_time | Rational | Exposure time | X | |
|
|
| - start_time | Rational | Exposure start time (highly approximate) | X | |
|
|
| - end_time | Rational | Exposure end time (highly approximate) | X | |
|
|
| - spot_count | uint64 | Spot count | | |
|
|
| - spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | | |
|
|
| - az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | |
|
|
| - indexing_result | bool | Indexing successful | | |
|
|
| - indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | | X |
|
|
| - indexing_unit_cell | object | Indexing result unit cell: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\]; present only if indexed | | X |
|
|
| | | Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | | |
|
|
| - xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X |
|
|
| - xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X |
|
|
| - jf_info | uint64 | Detector info field | | |
|
|
| - receiver_aq_dev_delay | uint64 | Receiver internal delay | | |
|
|
| - receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | | |
|
|
| - storage_cell | uint64 | Storage cell number | | |
|
|
| - saturated_pixel_count | uint64 | Saturated pixel count | | |
|
|
| - error_pixel_count | uint64 | Error pixel count | | |
|
|
| - strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | |
|
|
| - data_collection_efficiency | float | Image collection efficiency \[\] | | |
|
|
| - bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] (with solid angle/polarization corrections, if applied) | | X |
|
|
| - resolution_estimate | float | Diffraction resolution estimation | | X |
|
|
| - adu_histogram | Array(uint64) | ADU histogram | | X |
|
|
| - roi_integrals | object | Results of ROI calculation | | X |
|
|
| - - sum | int64 | Sum of pixels in ROI area \[photons\] | | |
|
|
| - - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | |
|
|
| - - pixels | uint64 | Valid pixels in ROI area | | |
|
|
| - - max_count | int64 | Highest count in ROI area \[photons\] | | |
|
|
|
|
## End message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format |
|
|
|----------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
|
|
| type | String | value "end" | X |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
|
|
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
|
|
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
|
|
| end_date | date | Approximate end date | |
|
|
| max_image_number | uint64 | Number of image with the highest number (this is counted from 1 - to distinguish zero images and one image) | |
|
|
| images_collected | uint64 | Number of image collected | |
|
|
| images_sent_to_write | uint64 | Number of image sent to writer; if writer queues were full, it is possible this is less than images collected | |
|
|
| data_collection_efficiency | float | Network packets collected / Network packets expected \[\] | |
|
|
| az_int_result | Map(text->Array(float)) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | |
|
|
| adu_histogram | Map(text->Array(uint64)) | ADU values histogram | |
|
|
| adu_histogram_bin_width | uint64 | Width of bins in the above histogram \[ADU\] | |
|
|
| max_receiver_delay | uint64 | Internal performance of Jungfraujoch | |
|
|
| bkg_estimate | float | Mean background estimate for the whole run | |
|
|
| indexing_rate | float | Mean indexing rate for the whole run | |
|
|
|
|
## Calibration message
|
|
|
|
| Field name | Type | Description | Present in DECTRIS format |
|
|
|--------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
|
|
| type | String | value "calibration" | |
|
|
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
|
|
| data | Map(string -> Image) | Calibration map (only single pedestal array per message) | |
|
|
|
|
## User data
|
|
In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch.
|
|
For this reason two fields can be provided: `header_appendix` (sent with start message) and `image_appendix` (send with image message).
|
|
To increase flexibility, both appendices can contain any valid JSON message.
|
|
These appendices are serialized into string and stored in CBOR messages as `user_data`.
|
|
|
|
Notably for start message, `user_data` can contain more information (non-DECTRIS compliant metadata).
|
|
Therefore `user_data` is serialized by Jungfraujoch as CBOR object. There is member `user` which contains `header_appendix` defined in OpenAPI of Jungfraujoch.
|
|
|
|
### Notes on images and compression
|
|
|
|
- Images are encoded as DECTRIS MultiDimArray with typed array tags:
|
|
- For RGB: shape \[3, height, width\], type: u8
|
|
- For grayscale: shape \[height, width\], type according to bit depth and sign (e.g., uint16 LE)
|
|
- Compression:
|
|
- Uncompressed: raw CBOR byte string
|
|
- Bitshuffle+LZ4: tag with \["bslz4", elem_size, bytes\]
|
|
- Bitshuffle+Zstandard: tag with \["bszstd", elem_size, bytes\] |