27 KiB
27 KiB
Image serialization using CBOR
To communicate between FPGA-equipped receiver system and writers, Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel). The protocol is based on and compatible with DECTRIS Stream2. There are minor differences at the moment:
- Pixel mask can be only sent uncompressed
- Only single channel is allowed for images (this only matters for deserialization)
- LZ4 compression is not allowed; Bitshuffle Zstandard is possible
- Few fields are currently absent
- Extra fields are present beyond DECTRIS standard
- There is calibration message defined to transfer pedestal correction
Start message
| Field name | Type | Description | Present in DECTRIS format |
|---|---|---|---|
| type | String | value "start" | X |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| detector_distance | float | Detector distance [m] | |
| detector_translation | float | Detector translation vector [m] | X |
| beam_center_x | float | Beam center in X direction [pixels] | X |
| beam_center_y | float | Beam center in Y direction [pixels] | X |
| number_of_images | uint64 | Number of images in the series | X |
| image_size_x | uint64 | Image width [pixels] | X |
| image_size_y | uint64 | Image height [pixels] | X |
| incident_energy | float | X-ray energy [eV] | X |
| incident_wavelength | float | X-ray wavelength [Angstrom] | X |
| frame_time | float | Frame time, if multiple frames per trigger [s] | X |
| count_time | float | Exposure time [s] | X |
| saturation_value | int64 | Maximum valid sample value | X |
| error_value | int64 | Value used in images to describe pixels that are in error state or missing | |
| pixel_size_x | float | Pixel width [m] | X |
| pixel_size_y | float | Pixel height [m] | X |
| sensor_thickness | float | Sensor thickness [m] | X |
| sensor_material | string | Sensor material | X |
| arm_data | data | Approximate date of arming | X |
| pixel_mask_enabled | bool | Pixel mask applied on images | X |
| detector_description | string | Name of the detector | X |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
| goniometer | Map | See DECTRIS documentation - only one axis is supported | X |
| pixel_mask | Map(string -> Image) | Pixel mask - multiple in case of storage cells | X |
| channels | Array(string) | List of image channels | X |
| max_spot_count | uint64 | Maximum number of spots identified in spot finding | |
| storage_cell_number | uint64 | Number of storage cells used by JUNGFRAU | |
| image_dtype | string | Pixel bit type (e.g. uint16) | X |
| unit_cell | Array(6 * float) (optional) | Unit cell of the system: a, b, c [angstrom] and alpha, beta, gamma [degree] | |
| az_int_bin_number | uint64 | Number of azimuthal integration bins | |
| az_int_bin_to_q | Array(float) | Q value for each azimuthal integration bin [angstrom^-1] | |
| summation | uint64 | Factor of frame summation | |
| user_data | string | JSON serialized to string that can contain the following fields: | X |
| - user | any valid JSON | Value of header_appendix provided at collection start to Jungfraujoch | |
| - source_name | string | Facility name | |
| - source_type | string (optional) | Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser") | |
| - instrument_name | string | Instrument name | |
| - attenuator_transmission | float (optional) | Attenuator transmission [] | |
| - total_flux | float (optional) | Total flux [ph/s] | |
| - data_file_count | uint64 | Number of used data files | |
| - file_prefix | string | File prefix | |
| - bit_depth_readout | uint64 | Bit depth of the detector readout | |
| - sample_name | string | Name of the sample | |
| - rotation_axis | Array(float) | Rotation axis vector (array of size 3) | |
| - space_group_number | uint64 | Space group number | |
| - write_master_file | bool | With multiple sockets, it selects which socket will provide master file | |
| - writer_notification_zmq_addr | string | ZeroMQ address to inform jfjoch_broker about writers that finished operation |
|
| - socket_number | uint64 | Number of ZeroMQ socket (on jfjoch_broker side) used for transmission |
|
| - experiment_group | string | ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number |
See DECTRIS documentation for definition of Image as MultiDimArray with optional compression.
Image message
| Field name | Type | Description | Present in DECTRIS format | Optional |
|---|---|---|---|---|
| type | String | value "image" | X | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | ||
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
| image_id | uint64 | Number of image within the series | X | |
| real_time | Rational | Exposure time | X | |
| start_time | Rational | Exposure start time (highly approximate) | X | |
| end_time | Rational | Exposure end time (highly approximate) | X | |
| spots | Array(object) | Spots: | ||
| - x | float | position in x (pixels) | ||
| - y | float | position in y (pixels) | ||
| - I | float | intensity (photons) | ||
| - indexed | bool | indexed solution | ||
| az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | ||
| indexing_result | bool | Indexing successful | ||
| indexing_lattice | Array(float * 9) (optional) | Indexing result real lattice; present only if indexed | X | |
| xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | X | |
| xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | X | |
| jf_info | uint64 | Detector info field | ||
| receiver_aq_dev_delay | uint64 | Receiver internal delay | ||
| receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | ||
| storage_cell | uint64 | Storage cell number | ||
| saturated_pixel_count | uint64 | Saturated pixel count | ||
| error_pixel_count | uint64 | Error pixel count | ||
| strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | ||
| data_collection_efficiency | float | Image collection efficiency [] | ||
| bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A [\photons] | ||
| adu_histogram | Array(uint64) | ADU histogram | ||
| user_data | string | Optional user defined text information - this is image_appendix serialized to JSON format | ||
| roi_integrals | object | Results of ROI calculation | X | |
| - sum | int64 | Sum of pixels in ROI area [photons] | ||
| - sum_square | int64 | Sum of squares of pixels in ROI area [photons] | ||
| - pixels | int64 | Valid pixels in ROI area [photons] | ||
| - max_count | int64 | Highest count in ROI area [photons] | ||
| image | Map(string -> Image) | Image |
End message
| Field name | Type | Description | Present in DECTRIS format |
|---|---|---|---|
| type | String | value "end" | X |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
| end_date | date | Approximate end date | |
| max_image_number | uint64 | Number of image with the highest number (this is counted from 1 - to distinguish zero images and one image) | |
| images_collected | uint64 | Number of image collected | |
| images_sent_to_writer | uint64 | Number of image sent to writer; if writer queues were full, it is possible this is less than images collected | |
| data_collection_efficiency | float | Network packets collected / Network packets expected [] | |
| az_int_result | Map(text->Array(float)) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | |
| adu_histogram | Map(text->Array(uint64)) | ADU values histogram | |
| adu_histogram_width | uint64 | Width of bins in the above histogram [ADU] | |
| max_receiver_delay | uint64 | Internal performance of Jungfraujoch |
Calibration message
| Field name | Type | Description | Present in DECTRIS format |
|---|---|---|---|
| type | String | value "end" | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| data | Map(string -> Image) | Calibration map (only single pedestal array per message) |
User data
In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch.
For this reason two fields can be provided: header_appendix (sent with start message) and image_appendix (send with image message).
To increase flexibility, both appendices can contain any valid JSON message.
These appendices are serialized into string and stored in CBOR messages as user_data.
Notably for start message, user_data contain more information (non-DECTRIS compliant metadata).
Therefore user_data is serialized by Jungfraujoch as CBOR object. There is member user which contains header_appendix defined in OpenAPI of Jungfraujoch.