mx/Jungfraujoch
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 42 Wiki Activity
Files
b56d3a14616ae8e778b2f699e42eef1348454f07
Jungfraujoch/docs/CBOR.md
leonarski_f adc13ff33e version 1.0.0-rc.24
2024-11-17 14:55:09 +01:00

147 lines
27 KiB
Markdown
Raw Blame History

# Image serialization using CBOR
To communicate between FPGA-equipped receiver system and writers,
Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel).
The protocol is based on and compatible with [DECTRIS Stream2](https://github.com/dectris/documentation/tree/main/stream_v2).
There are minor differences at the moment:
* Pixel mask can be only sent uncompressed
* Only single channel is allowed for images (this only matters for deserialization)
* LZ4 compression is not allowed; Bitshuffle Zstandard is possible
* Few fields are currently absent
* Extra fields are present beyond DECTRIS standard
* There is calibration message defined to transfer pedestal correction
## Start message
| Field name | Type | Description | Present in DECTRIS format |
|--------------------------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
| type | String | value "start" | X |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| detector_distance | float | Detector distance \[m\] | |
| detector_translation | float | Detector translation vector \[m\] | X |
| beam_center_x | float | Beam center in X direction \[pixels\] | X |
| beam_center_y | float | Beam center in Y direction \[pixels\] | X |
| number_of_images | uint64 | Number of images in the series | X |
| image_size_x | uint64 | Image width \[pixels\] | X |
| image_size_y | uint64 | Image height \[pixels\] | X |
| incident_energy | float | X-ray energy \[eV\] | X |
| incident_wavelength | float | X-ray wavelength \[Angstrom\] | X |
| frame_time | float | Frame time, if multiple frames per trigger \[s\] | X |
| count_time | float | Exposure time \[s\] | X |
| saturation_value | int64 | Maximum valid sample value | X |
| error_value | int64 | Value used in images to describe pixels that are in error state or missing | |
| pixel_size_x | float | Pixel width \[m\] | X |
| pixel_size_y | float | Pixel height \[m\] | X |
| sensor_thickness | float | Sensor thickness \[m\] | X |
| sensor_material | string | Sensor material | X |
| arm_data | data | Approximate date of arming | X |
| pixel_mask_enabled | bool | Pixel mask applied on images | X |
| detector_description | string | Name of the detector | X |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
| goniometer | Map | See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/stream_v2) - only one axis is supported | X |
| pixel_mask | Map(string -> Image) | Pixel mask - multiple in case of storage cells | X |
| channels | Array(string) | List of image channels | X |
| max_spot_count | uint64 | Maximum number of spots identified in spot finding | |
| storage_cell_number | uint64 | Number of storage cells used by JUNGFRAU | |
| image_dtype | string | Pixel bit type (e.g. uint16) | X |
| unit_cell | Array(6 * float) (optional) | Unit cell of the system: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | |
| az_int_bin_number | uint64 | Number of azimuthal integration bins | |
| az_int_bin_to_q | Array(float) | Q value for each azimuthal integration bin \[angstrom^-1\] | |
| summation | uint64 | Factor of frame summation | |
| user_data | string | JSON serialized to string that can contain the following fields: | X |
| - user | any valid JSON | Value of header_appendix provided at collection start to Jungfraujoch | |
| - source_name | string | Facility name | |
| - source_type | string (optional) | Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser") | |
| - instrument_name | string | Instrument name | |
| - attenuator_transmission | float (optional) | Attenuator transmission \[\] | |
| - total_flux | float (optional) | Total flux \[ph/s\] | |
| - data_file_count | uint64 | Number of used data files | |
| - file_prefix | string | File prefix | |
| - bit_depth_readout | uint64 | Bit depth of the detector readout | |
| - sample_name | string | Name of the sample | |
| - rotation_axis | Array(float) | Rotation axis vector (array of size 3) | |
| - space_group_number | uint64 | Space group number | |
| - write_master_file | bool | With multiple sockets, it selects which socket will provide master file | |
| - writer_notification_zmq_addr | string | ZeroMQ address to inform `jfjoch_broker` about writers that finished operation | |
| - socket_number | uint64 | Number of ZeroMQ socket (on `jfjoch_broker` side) used for transmission | |
| - experiment_group | string | ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number | |
See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/stream_v2) for definition of Image as MultiDimArray with optional compression.
## Image message
| Field name | Type | Description | Present in DECTRIS format | Optional |
|----------------------------|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:|
| type | String | value "image" | X | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X | |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | |
| image_id | uint64 | Number of image within the series | X | |
| real_time | Rational | Exposure time | X | |
| start_time | Rational | Exposure start time (highly approximate) | X | |
| end_time | Rational | Exposure end time (highly approximate) | X | |
| spots | Array(object) | Spots: | | |
| - x | float | position in x (pixels) | | |
| - y | float | position in y (pixels) | | |
| - I | float | intensity (photons) | | |
| - indexed | bool | indexed solution | | |
| az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | |
| indexing_result | bool | Indexing successful | | |
| indexing_lattice | Array(float * 9) (optional) | Indexing result real lattice; present only if indexed | | X |
| xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X |
| xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X |
| jf_info | uint64 | Detector info field | | |
| receiver_aq_dev_delay | uint64 | Receiver internal delay | | |
| receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | | |
| storage_cell | uint64 | Storage cell number | | |
| saturated_pixel_count | uint64 | Saturated pixel count | | |
| error_pixel_count | uint64 | Error pixel count | | |
| strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | |
| data_collection_efficiency | float | Image collection efficiency \[\] | | |
| bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A [\photons\] | | |
| adu_histogram | Array(uint64) | ADU histogram | | |
| user_data | string | Optional user defined text information - this is image_appendix serialized to JSON format | | |
| roi_integrals | object | Results of ROI calculation | | X |
| - sum | int64 | Sum of pixels in ROI area \[photons\] | | |
| - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | |
| - pixels | int64 | Valid pixels in ROI area \[photons\] | | |
| - max_count | int64 | Highest count in ROI area \[photons\] | | |
| image | Map(string -> Image) | Image | | |
## End message
| Field name | Type | Description | Present in DECTRIS format |
|----------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
| type | String | value "end" | X |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| series_unique_id | string | Unique text ID of the series (run_name parameter) | X |
| series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X |
| end_date | date | Approximate end date | |
| max_image_number | uint64 | Number of image with the highest number (this is counted from 1 - to distinguish zero images and one image) | |
| images_collected | uint64 | Number of image collected | |
| images_sent_to_writer | uint64 | Number of image sent to writer; if writer queues were full, it is possible this is less than images collected | |
| data_collection_efficiency | float | Network packets collected / Network packets expected \[\] | |
| az_int_result | Map(text->Array(float)) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | |
| adu_histogram | Map(text->Array(uint64)) | ADU values histogram | |
| adu_histogram_width | uint64 | Width of bins in the above histogram \[ADU\] | |
| max_receiver_delay | uint64 | Internal performance of Jungfraujoch | |
# Calibration message
| Field name | Type | Description | Present in DECTRIS format |
|--------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|
| type | String | value "end" | |
| magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | |
| data | Map(string -> Image) | Calibration map (only single pedestal array per message) | |
# User data
In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch.
For this reason two fields can be provided: `header_appendix` (sent with start message) and `image_appendix` (send with image message).
To increase flexibility, both appendices can contain any valid JSON message.
These appendices are serialized into string and stored in CBOR messages as `user_data`.
Notably for start message, `user_data` contain more information (non-DECTRIS compliant metadata).
Therefore `user_data` is serialized by Jungfraujoch as CBOR object. There is member `user` which contains `header_appendix` defined in OpenAPI of Jungfraujoch.
Reference in New Issue View Git Blame Copy Permalink