# CBOR messages To communicate between FPGA-equipped receiver system and writers, Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel). The protocol is based on and compatible with [DECTRIS Stream2](https://github.com/dectris/documentation/tree/main/stream_v2). There are minor differences at the moment: * LZ4 alone is not allowed; Bitshuffle+LZ4 and Bitshuffle+Zstandard are allowed * Few fields are currently absent * Extra fields are present beyond DECTRIS standard * There are calibration and metadata messages defined beyond DECTRIS specification ## Start message | Field name | Type | Description | Present in DECTRIS format | |----------------------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:| | type | String | value "start" | X | | magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | | detector_distance | float | Detector distance \[m\] | | | detector_translation | Array(float) | Detector translation vector \[m\] | X | | beam_center_x | float | Beam center in X direction \[pixels\] | X | | beam_center_y | float | Beam center in Y direction \[pixels\] | X | | countrate_correction_enabled | bool | Countrate correction enabled | X | | flatfield_enabled | bool | Flatfield enabled | X | | number_of_images | uint64 | Number of images in the series | X | | image_size_x | uint64 | Image width \[pixels\] | X | | image_size_y | uint64 | Image height \[pixels\] | X | | incident_energy | float | X-ray energy \[eV\] | X | | incident_wavelength | float | X-ray wavelength \[Angstrom\] | X | | frame_time | float | Frame time, if multiple frames per trigger \[s\] | X | | count_time | float | Exposure time \[s\] | X | | saturation_value | int64 | Maximum valid sample value | X | | error_value | int64 (optional) | Value used in images to describe pixels that are in error state or missing | | | pixel_size_x | float | Pixel width \[m\] | X | | pixel_size_y | float | Pixel height \[m\] | X | | sensor_thickness | float | Sensor thickness \[m\] | X | | sensor_material | string | Sensor material | X | | arm_date | date | Approximate date of arming | X | | pixel_mask_enabled | bool | Pixel mask applied on images | X | | detector_description | string | Name of the detector | X | | detector_serial_number | string | Detector serial number | X | | series_unique_id | string | Unique text ID of the series (run_name parameter) | X | | series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | | fluorescence | object (optional) | X-ray fluorescence spectrum collected at start | | | - energy | Array(float) | Energy of measuring point \[eV\] | | | - data | Array(float) | Fluorescence scan result `data` \[arbitrary units\]; must be strictly the same length as energy | | | goniometer | Map | Definition of rotation axis (optional) | X | | - `AXIS` | string | Rotation axis name (e.g. omega) - only one axis is supported in Jungfraujoch | X | | - - increment | float | Rotation axis increment (per image) in degree \[deg\] | X | | - - start | float | Rotation axis start angle \[deg\] | X | | - - axis | Array(float) | Vector for the rotation axis | | | - - helical_step | Array(float) | Translation for helical scan for 1 image \[m\] | | | - - screening_wedge | Array(float) | Wedge for screening \[deg\] (increment would correspond to difference between screening points) | | | grid_scan | object | Grid scan definition (optional and exclusive with rotation axis) | | | - n_fast | uint64 | Number of elements along fast axis | | | - n_slow | uint64 | Number of elements along slow axis | | | - step_x_axis | float | Step along X axis, can be negative \[m\] | | | - step_y_axis | float | Step along Y axis, can be negative \[m\] | | | - snake_scan | bool | Snake scan (rows alternate direction) | | | - vertical_scan | bool | Vertical scan (enabled: fast direction = Y, disabled: fast direction = X) | | | jungfrau_conversion_enabled | bool (optional) | Applying JUNGFRAU pixel conversion (to photons or keV) | | | jungfrau_conversion_factor | float (optional) | Factor used for JUNGFRAU conversion \[eV\] | | | geometry_transformation_enabled | bool (optional) | Transformation from detector module geometry (512x1024) to full detector geometry | | | pixel_mask | Map(string -> Image) | Pixel mask - multiple in case of storage cells | X | | channels | Array(string) | List of image channels | X | | max_spot_count | uint64 | Maximum number of spots identified in spot finding | | | storage_cell_number | uint64 (optional) | Number of storage cells used by JUNGFRAU | | | storage_cell_delay | Rational | Delay of storage cells in JUNGFRAU | | | threshold_energy | float | Threshold energy for EIGER detector \[eV\] | | | image_dtype | string | Pixel bit type (e.g. uint16) | X | | unit_cell | object (optional) | Unit cell of the system: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\] | | | az_int_q_bin_count | uint64 | Number of azimuthal integration bins in the radial direction | | | az_int_phi_bin_count | uint64 | Number of azimuthal integration bins in the phi angle direction | | | az_int_bin_to_q | Array(float) | Q value for each azimuthal integration bin \[angstrom^-1\] | | | az_int_bin_to_two_theta | Array(float) | Two theta angle value for each azimuthal integration bin \[deg\] | | | az_int_bin_to_phi | Array(float) | Phi value for each azimuthal integration bin \[deg\] | | | summation | uint64 | Factor of frame summation | | | user_data | string | JSON serialized to string that can contain the following fields (all fields are optional): | X | | - file_prefix | string | File prefix | | | - images_per_file | uint64 | Number of images written per file | | | - images_per_trigger | uint64 | Number of images collected per trigger | | | - source_name | string | Facility name | | | - source_type | string | Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser") | | | - instrument_name | string | Instrument name | | | - sample_name | string | Name of the sample | | | - user | any valid JSON | Value of header_appendix provided at collection start to Jungfraujoch | | | - attenuator_transmission | float | Attenuator transmission \[\] | | | - total_flux | float | Total flux \[ph/s\] | | | - space_group_number | uint64 | Space group number | | | - summation_mode | string | Summation mode (internal\|fpga\|cpu) | | | - overwrite | bool | Overwrite existing HDF5 files | | | - file_format | int | File writer format: 0 = no master file, 1 = soft links, 2 = virtual dataset, 3 = CBF, 4 = TIFF | | | - roi | Array(object) | ROI configurations; each element is one of: | | | | | type "box": xmin, xmax, ymin, ymax (numbers) | | | | | type "circle": r, x, y (numbers) | | | | | type "azim": qmin, qmax (numbers) | | | - gain_file_names | Array(string) | Names of JUNGFRAU gain files used for the current detector | | | - write_master_file | bool | With multiple sockets, it selects which socket will provide master file | | | - data_reduction_factor_serialmx | uint64 | Data reduction factor for serial MX | | | - experiment_group | string | ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number | | | - jfjoch_release | string | Jungfraujoch release number | | | - socket_number | uint64 | Number of ZeroMQ socket (on `jfjoch_broker` side) used for transmission | | | - bit_depth_readout | uint64 | Bit depth of the detector readout | | | - writer_notification_zmq_addr | string | ZeroMQ address to inform `jfjoch_broker` about writers that finished operation | | | - xfel_pulse_id | uint64 | Pulse IDs are recorded for images | | | - ring_current_mA | float | Ring current at the start of the measurement | | | - sample_temperature_K | float | Sample temperature \[K\] | | | - detect_ice_rings | bool | Ice ring detection feature is enabled | | | - indexing_algorithm | string | Indexing algorithm used on-the-fly; allowed values: ffbidx, fft, fftw, none | | | - geom_refinement_algorithm | string | Post-indexing detector geometry refinement algorithm; allowed values: none, beam_center, beam_center_tetragonal | | | - poni_rot1 | float | Tilt of the detector rot1 according to PyFAI PONI convention \[rad\] | | | - poni_rot2 | float | Tilt of the detector rot2 according to PyFAI PONI convention \[rad\] | | | - poni_rot3 | float | Tilt of the detector rot3 according to PyFAI PONI convention \[rad\] | | See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/stream_v2) for definition of Image as MultiDimArray with optional compression. ## Image message | Field name | Type | Description | Present in DECTRIS format | Optional | |----------------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:| | type | String | value "image" | X | | | magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | | | series_unique_id | string | Unique text ID of the series (run_name parameter) | X | | | series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | | | image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | | | original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | | | | real_time | Rational | Exposure time | X | | | start_time | Rational | Exposure start time (highly approximate) | X | | | end_time | Rational | Exposure end time (highly approximate) | X | | | spots | Array(object) | Spots: | | | | - x | float | observed position in x (pixels) | | | | - y | float | observed position in y (pixels) | | | | - I | float | intensity (photons) | | | | - maxc | int64 | max count (photons) | | | | - ice_ring | bool | spot in resolution range for ice rings | | | | - indexed | bool | indexed solution | | | | reflections | Array(object) | Reflections: | | | | - h | int64 | Miller index | | | | - k | int64 | Miller index | | | | - l | int64 | Miller index | | | | - x | float | position in x (pixels) | | | | - y | float | position in y (pixels) | | | | - d | float | resolution \[angstrom\] | | | | - I | float | integrated intensity (photons) | | | | - bkg | float | mean background value (photons) | | | | - sigma | float | standard deviation, estimated from counting statistics (photons) | | | | - image | float | image number | | | | spot_count | uint64 | Spot count | | | | spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | | | | spot_count_low_res | uint64 | Number of spots in low resolution (prior to filtering) | | | | spot_count_indexed | uint64 | Number of spots which fit indexing solution within a given tolerance | | | | az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | | | indexing_result | bool | Indexing successful | | | | indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | | X | | indexing_unit_cell | object | Indexing result unit cell: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\]; present only if indexed | | X | | | | Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | | | | profile_radius | float | Profile radius of the image - describes distance of observed reflections from the Ewald sphere (Angstrom^-1) | | | | b_factor | float | Estimated B-factor (Angstrom^2) | | | | indexing_time | float | Time spent on indexing \[s\] | | | | processing_time | float | Total processing time \[s\] | | | | xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X | | xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X | | lattice_type | object | Bravais lattice classification of the indexing result (present only if available) | | X | | - centering | string | One-letter centering code: P, A, B, C, I, F, or R | | | | - niggli_class | int64 | Integer identifier for the Niggli-reduced Bravais class | | | | - system | string | Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic | | | | jf_info | uint64 | Detector info field | | | | receiver_aq_dev_delay | uint64 | Receiver internal delay | | | | receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | | | | storage_cell | uint64 | Storage cell number | | | | saturated_pixel_count | uint64 | Saturated pixel count | | | | pixel_sum | uint64 | Sum of all pixels, excl. error and saturation | | | | error_pixel_count | uint64 | Error pixel count | | | | strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | | | min_viable_pixel_value | int64 | Minimal pixel value, excl. error and saturation | | | | max_viable_pixel_value | int64 | Maximal pixel value, excl. error and saturation | | | | resolution_estimate | float | Diffraction resolution estimation \[Angstrom\] | | X | | data_collection_efficiency | float | Image collection efficiency \[\] | | | | packets_expected | uint64 | Number of packets expected per image (in units of 2 kB) | | | | packets_received | uint64 | Number of packets received per image (in units of 2 kB) | | | | bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] | | | | beam_center_x | float | Beam center X from post-indexing refinement \[pixel\] | | X | | beam_center_y | float | Beam center Y from post-indexing refinement \[pixel\] | | X | | beam_corr_x | float | Beam center correction X applied during processing \[pixel\] | | X | | beam_corr_y | float | Beam center correction Y applied during processing \[pixel\] | | X | | adu_histogram | Array(uint64) | ADU histogram | | | | roi_integrals | object | Results of ROI calculation | | X | | - sum | int64 | Sum of pixels in ROI area \[photons\] | | | | - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | | | - pixels | uint64 | Valid pixels in ROI area | | | | - max_count | int64 | Highest count in ROI area \[photons\] | | | | - x_weighted_sum | int64 | ROI pixel X position multiplied by photon count \[photons * pixels\] | | | | - y_weighted_sum | int64 | ROI pixel Y position multiplied by photon count \[photons * pixels\] | | | | user_data | string | Optional user defined text information - this is image_appendix serialized to JSON format | X | | | data | Map(string -> Image) | Image | X | | ## Metadata message | Field name | Type | Description | Present in DECTRIS format | Optional | |------------------------------|------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:|:--------:| | type | String | value "metadata" | X | | | magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | | | series_unique_id | string | Unique text ID of the series (run_name parameter) | X | | | series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | | | images | Array(object) | Array of images (order and size of the array are not guaranteed) | X | | | - image_id | uint64 | Number of image within the series; for MX lossy compression this is sequential excluding removed frames | X | | | - original_image_id | uint64 | Number of image within the series; for MX lossy compression this includes removed frames in the count | | | | - real_time | Rational | Exposure time | X | | | - start_time | Rational | Exposure start time (highly approximate) | X | | | - end_time | Rational | Exposure end time (highly approximate) | X | | | - spot_count | uint64 | Spot count | | | | - spot_count_ice_rings | uint64 | Number of spots within identified rings (experimental) | | | | - az_int_profile | Array(float) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | | | - indexing_result | bool | Indexing successful | | | | - indexing_lattice | Array(9 * float) | Indexing result real lattice; present only if indexed | | X | | - indexing_unit_cell | object | Indexing result unit cell: a, b, c \[angstrom\] and alpha, beta, gamma \[degree\]; present only if indexed | | X | | | | Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided | | | | - xfel_pulse_id | uint64 | Bunch ID (for pulsed source, e.g., SwissFEL) | | X | | - xfel_event_code | uint64 | Event code (for pulsed source, e.g., SwissFEL) | | X | | - jf_info | uint64 | Detector info field | | | | - receiver_aq_dev_delay | uint64 | Receiver internal delay | | | | - receiver_free_send_buf | uint64 | Receiver internal number of available send buffers | | | | - storage_cell | uint64 | Storage cell number | | | | - saturated_pixel_count | uint64 | Saturated pixel count | | | | - error_pixel_count | uint64 | Error pixel count | | | | - strong_pixel_count | uint64 | Strong pixel count (first stage of spot finding) | | | | - data_collection_efficiency | float | Image collection efficiency \[\] | | | | - bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A \[photons\] (with solid angle/polarization corrections, if applied) | | X | | - resolution_estimate | float | Diffraction resolution estimation | | X | | - adu_histogram | Array(uint64) | ADU histogram | | X | | - roi_integrals | object | Results of ROI calculation | | X | | - - sum | int64 | Sum of pixels in ROI area \[photons\] | | | | - - sum_square | int64 | Sum of squares of pixels in ROI area \[photons\] | | | | - - pixels | uint64 | Valid pixels in ROI area | | | | - - max_count | int64 | Highest count in ROI area \[photons\] | | | ## End message | Field name | Type | Description | Present in DECTRIS format | |----------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:| | type | String | value "end" | X | | magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | | series_unique_id | string | Unique text ID of the series (run_name parameter) | X | | series_id | uint64 | Unique numeric ID of the series (run_number parameter) | X | | end_date | date | Approximate end date | | | max_image_number | uint64 | Number of image with the highest number (this is counted from 1 - to distinguish zero images and one image) | | | images_collected | uint64 | Number of image collected | | | images_sent_to_write | uint64 | Number of image sent to writer; if writer queues were full, it is possible this is less than images collected | | | data_collection_efficiency | float | Network packets collected / Network packets expected \[\] | | | az_int_result | Map(text->Array(float)) | Azimuthal integration results, use az_int_bin_to_q from start message for legend | | | adu_histogram | Map(text->Array(uint64)) | ADU values histogram | | | adu_histogram_bin_width | uint64 | Width of bins in the above histogram \[ADU\] | | | max_receiver_delay | uint64 | Internal performance of Jungfraujoch | | | bkg_estimate | float | Mean background estimate for the whole run | | | indexing_rate | float | Mean indexing rate for the whole run | | ## Calibration message | Field name | Type | Description | Present in DECTRIS format | |--------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------:| | type | String | value "calibration" | | | magic_number | uint64 | Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver | | | data | Map(string -> Image) | Calibration map (only single pedestal array per message) | | ## User data In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch. For this reason two fields can be provided: `header_appendix` (sent with start message) and `image_appendix` (send with image message). To increase flexibility, both appendices can contain any valid JSON message. These appendices are serialized into string and stored in CBOR messages as `user_data`. Notably for start message, `user_data` can contain more information (non-DECTRIS compliant metadata). Therefore `user_data` is serialized by Jungfraujoch as CBOR object. There is member `user` which contains `header_appendix` defined in OpenAPI of Jungfraujoch. ### Notes on images and compression - Images are encoded as DECTRIS MultiDimArray with typed array tags: - For RGB: shape \[3, height, width\], type: u8 - For grayscale: shape \[height, width\], type according to bit depth and sign (e.g., uint16 LE) - Compression: - Uncompressed: raw CBOR byte string - Bitshuffle+LZ4: tag with \["bslz4", elem_size, bytes\] - Bitshuffle+Zstandard: tag with \["bszstd", elem_size, bytes\]