Jungfraujoch/docs/CBOR.md

CBOR messages

To communicate between FPGA-equipped receiver system and writers, Jungfraujoch is using binary CBOR encoding with tinycbor library (Intel). The protocol is based on and compatible with DECTRIS Stream2. There are minor differences at the moment:

• LZ4 alone is not allowed; Bitshuffle+LZ4 and Bitshuffle+Zstandard are allowed
• Few fields are currently absent
• Extra fields are present beyond DECTRIS standard
• There are calibration and metadata messages defined beyond DECTRIS specification

Start message

Field name	Type	Description	Present in DECTRIS format
type	String	value "start"	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
detector_distance	float	Detector distance [m]
detector_translation	Array(float)	Detector translation vector [m]	X
beam_center_x	float	Beam center in X direction [pixels]	X
beam_center_y	float	Beam center in Y direction [pixels]	X
countrate_correction_enabled	bool	Countrate correction enabled	X
flatfield_enabled	bool	Flatfield enabled	X
number_of_images	uint64	Number of images in the series	X
image_size_x	uint64	Image width [pixels]	X
image_size_y	uint64	Image height [pixels]	X
incident_energy	float	X-ray energy [eV]	X
incident_wavelength	float	X-ray wavelength [Angstrom]	X
frame_time	float	Frame time, if multiple frames per trigger [s]	X
count_time	float	Exposure time [s]	X
saturation_value	int64	Maximum valid sample value	X
error_value	int64 (optional)	Value used in images to describe pixels that are in error state or missing
pixel_size_x	float	Pixel width [m]	X
pixel_size_y	float	Pixel height [m]	X
sensor_thickness	float	Sensor thickness [m]	X
sensor_material	string	Sensor material	X
arm_date	date	Approximate date of arming	X
pixel_mask_enabled	bool	Pixel mask applied on images	X
detector_description	string	Name of the detector	X
detector_serial_number	string	Detector serial number	X
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
fluorescence	object (optional)	X-ray fluorescence spectrum collected at start
- energy	Array(float)	Energy of measuring point [eV]
- data	Array(float)	Fluorescence scan result data [arbitrary units]; must be strictly the same length as energy
goniometer	Map	Definition of rotation axis (optional)	X
- AXIS	string	Rotation axis name (e.g. omega) - only one axis is supported in Jungfraujoch	X
- - increment	float	Rotation axis increment (per image) in degree [deg]	X
- - start	float	Rotation axis start angle [deg]	X
- - axis	Array(float)	Vector for the rotation axis
- - helical_step	Array(float)	Translation for helical scan for 1 image [m]
- - screening_wedge	Array(float)	Wedge for screening [deg] (increment would correspond to difference between screening points)
grid_scan	object	Grid scan definition (optional and exclusive with rotation axis)
- n_fast	uint64	Number of elements along fast axis
- n_slow	uint64	Number of elements along slow axis
- step_x_axis	float	Step along X axis, can be negative [m]
- step_y_axis	float	Step along Y axis, can be negative [m]
- snake_scan	bool	Snake scan (rows alternate direction)
- vertical_scan	bool	Vertical scan (enabled: fast direction = Y, disabled: fast direction = X)
jungfrau_conversion_enabled	bool (optional)	Applying JUNGFRAU pixel conversion (to photons or keV)
jungfrau_conversion_factor	float (optional)	Factor used for JUNGFRAU conversion [eV]
geometry_transformation_enabled	bool (optional)	Transformation from detector module geometry (512x1024) to full detector geometry
pixel_mask	Map(string -> Image)	Pixel mask - multiple in case of storage cells	X
channels	Array(string)	List of image channels	X
max_spot_count	uint64	Maximum number of spots identified in spot finding
storage_cell_number	uint64 (optional)	Number of storage cells used by JUNGFRAU
storage_cell_delay	Rational	Delay of storage cells in JUNGFRAU
threshold_energy	float	Threshold energy for EIGER detector [eV]
image_dtype	string	Pixel bit type (e.g. uint16)	X
unit_cell	object (optional)	Unit cell of the system: a, b, c [angstrom] and alpha, beta, gamma [degree]
az_int_q_bin_count	uint64	Number of azimuthal integration bins in the radial direction
az_int_phi_bin_count	uint64	Number of azimuthal integration bins in the phi angle direction
az_int_bin_to_q	Array(float)	Q value for each azimuthal integration bin [angstrom^-1]
az_int_bin_to_two_theta	Array(float)	Two theta angle value for each azimuthal integration bin [deg]
az_int_bin_to_phi	Array(float)	Phi value for each azimuthal integration bin [deg]
summation	uint64	Factor of frame summation
user_data	string	JSON serialized to string that can contain the following fields (all fields are optional):	X
- file_prefix	string	File prefix
- images_per_file	uint64	Number of images written per file
- images_per_trigger	uint64	Number of images collected per trigger
- source_name	string	Facility name
- source_type	string	Type of X-ray source (use NXsource/type values, for example "Synchrotron X-ray Source" or "Free-Electron Laser")
- instrument_name	string	Instrument name
- sample_name	string	Name of the sample
- user	any valid JSON	Value of header_appendix provided at collection start to Jungfraujoch
- attenuator_transmission	float	Attenuator transmission []
- total_flux	float	Total flux [ph/s]
- space_group_number	uint64	Space group number
- summation_mode	string	Summation mode (internal|fpga|cpu)
- overwrite	bool	Overwrite existing HDF5 files
- file_format	int	File writer format: 0 = no master file, 1 = soft links, 2 = virtual dataset, 3 = CBF, 4 = TIFF
- roi	Array(object)	ROI configurations; each element is one of:
		type "box": xmin, xmax, ymin, ymax (numbers)
		type "circle": r, x, y (numbers)
		type "azim": qmin, qmax (numbers)
- gain_file_names	Array(string)	Names of JUNGFRAU gain files used for the current detector
- write_master_file	bool	With multiple sockets, it selects which socket will provide master file
- data_reduction_factor_serialmx	uint64	Data reduction factor for serial MX
- experiment_group	string	ID of instrument user, e.g., p-group (SLS/SwissFEL) or proposal number
- jfjoch_release	string	Jungfraujoch release number
- socket_number	uint64	Number of ZeroMQ socket (on jfjoch_broker side) used for transmission
- bit_depth_readout	uint64	Bit depth of the detector readout
- writer_notification_zmq_addr	string	ZeroMQ address to inform jfjoch_broker about writers that finished operation
- xfel_pulse_id	uint64	Pulse IDs are recorded for images
- ring_current_mA	float	Ring current at the start of the measurement
- sample_temperature_K	float	Sample temperature [K]
- detect_ice_rings	bool	Ice ring detection feature is enabled
- indexing_algorithm	string	Indexing algorithm used on-the-fly; allowed values: ffbidx, fft, fftw, none
- geom_refinement_algorithm	string	Post-indexing detector geometry refinement algorithm; allowed values: none, beam_center, beam_center_tetragonal
- poni_rot1	float	Tilt of the detector rot1 according to PyFAI PONI convention [rad]
- poni_rot2	float	Tilt of the detector rot2 according to PyFAI PONI convention [rad]
- poni_rot3	float	Tilt of the detector rot3 according to PyFAI PONI convention [rad]

See DECTRIS documentation for definition of Image as MultiDimArray with optional compression.

Image message

Field name	Type	Description	Present in DECTRIS format	Optional
type	String	value "image"	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
image_id	uint64	Number of image within the series; for MX lossy compression this is sequential excluding removed frames	X
original_image_id	uint64	Number of image within the series; for MX lossy compression this includes removed frames in the count
real_time	Rational	Exposure time	X
start_time	Rational	Exposure start time (highly approximate)	X
end_time	Rational	Exposure end time (highly approximate)	X
spots	Array(object)	Spots:
- x	float	observed position in x (pixels)
- y	float	observed position in y (pixels)
- I	float	intensity (photons)
- maxc	int64	max count (photons)
- ice_ring	bool	spot in resolution range for ice rings
- indexed	bool	indexed solution
reflections	Array(object)	Reflections:
- h	int64	Miller index
- k	int64	Miller index
- l	int64	Miller index
- x	float	position in x (pixels)
- y	float	position in y (pixels)
- d	float	resolution [angstrom]
- I	float	integrated intensity (photons)
- bkg	float	mean background value (photons)
- sigma	float	standard deviation, estimated from counting statistics (photons)
- image	float	image number
spot_count	uint64	Spot count
spot_count_ice_rings	uint64	Number of spots within identified rings (experimental)
spot_count_low_res	uint64	Number of spots in low resolution (prior to filtering)
spot_count_indexed	uint64	Number of spots which fit indexing solution within a given tolerance
az_int_profile	Array(float)	Azimuthal integration results, use az_int_bin_to_q from start message for legend
indexing_result	bool	Indexing successful
indexing_lattice	Array(9 * float)	Indexing result real lattice; present only if indexed		X
indexing_unit_cell	object	Indexing result unit cell: a, b, c [angstrom] and alpha, beta, gamma [degree]; present only if indexed		X
		Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided
profile_radius	float	Profile radius of the image - describes distance of observed reflections from the Ewald sphere (Angstrom^-1)
b_factor	float	Estimated B-factor (Angstrom^2)
indexing_time	float	Time spent on indexing [s]
processing_time	float	Total processing time [s]
xfel_pulse_id	uint64	Bunch ID (for pulsed source, e.g., SwissFEL)		X
xfel_event_code	uint64	Event code (for pulsed source, e.g., SwissFEL)		X
lattice_type	object	Bravais lattice classification of the indexing result (present only if available)		X
- centering	string	One-letter centering code: P, A, B, C, I, F, or R
- niggli_class	int64	Integer identifier for the Niggli-reduced Bravais class
- system	string	Crystal system: triclinic, monoclinic, orthorhombic, tetragonal, trigonal, hexagonal, cubic
jf_info	uint64	Detector info field
receiver_aq_dev_delay	uint64	Receiver internal delay
receiver_free_send_buf	uint64	Receiver internal number of available send buffers
storage_cell	uint64	Storage cell number
saturated_pixel_count	uint64	Saturated pixel count
pixel_sum	uint64	Sum of all pixels, excl. error and saturation
error_pixel_count	uint64	Error pixel count
strong_pixel_count	uint64	Strong pixel count (first stage of spot finding)
min_viable_pixel_value	int64	Minimal pixel value, excl. error and saturation
max_viable_pixel_value	int64	Maximal pixel value, excl. error and saturation
resolution_estimate	float	Diffraction resolution estimation [Angstrom]		X
data_collection_efficiency	float	Image collection efficiency []
packets_expected	uint64	Number of packets expected per image (in units of 2 kB)
packets_received	uint64	Number of packets received per image (in units of 2 kB)
bkg_estimate	float	Mean value for pixels in resolution range from 3.0 to 5.0 A [photons]
beam_center_x	float	Beam center X from post-indexing refinement [pixel]		X
beam_center_y	float	Beam center Y from post-indexing refinement [pixel]		X
beam_corr_x	float	Beam center correction X applied during processing [pixel]		X
beam_corr_y	float	Beam center correction Y applied during processing [pixel]		X
adu_histogram	Array(uint64)	ADU histogram
roi_integrals	object	Results of ROI calculation		X
- sum	int64	Sum of pixels in ROI area [photons]
- sum_square	int64	Sum of squares of pixels in ROI area [photons]
- pixels	uint64	Valid pixels in ROI area
- max_count	int64	Highest count in ROI area [photons]
- x_weighted_sum	int64	ROI pixel X position multiplied by photon count [photons * pixels]
- y_weighted_sum	int64	ROI pixel Y position multiplied by photon count [photons * pixels]
user_data	string	Optional user defined text information - this is image_appendix serialized to JSON format	X
data	Map(string -> Image)	Image	X

Metadata message

Field name	Type	Description	Present in DECTRIS format	Optional
type	String	value "metadata"	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
images	Array(object)	Array of images (order and size of the array are not guaranteed)	X
- image_id	uint64	Number of image within the series; for MX lossy compression this is sequential excluding removed frames	X
- original_image_id	uint64	Number of image within the series; for MX lossy compression this includes removed frames in the count
- real_time	Rational	Exposure time	X
- start_time	Rational	Exposure start time (highly approximate)	X
- end_time	Rational	Exposure end time (highly approximate)	X
- spot_count	uint64	Spot count
- spot_count_ice_rings	uint64	Number of spots within identified rings (experimental)
- az_int_profile	Array(float)	Azimuthal integration results, use az_int_bin_to_q from start message for legend
- indexing_result	bool	Indexing successful
- indexing_lattice	Array(9 * float)	Indexing result real lattice; present only if indexed		X
- indexing_unit_cell	object	Indexing result unit cell: a, b, c [angstrom] and alpha, beta, gamma [degree]; present only if indexed		X
		Unit cell is redundant to lattice - yet to simplify downstream programs to analyze results, both are provided
- xfel_pulse_id	uint64	Bunch ID (for pulsed source, e.g., SwissFEL)		X
- xfel_event_code	uint64	Event code (for pulsed source, e.g., SwissFEL)		X
- jf_info	uint64	Detector info field
- receiver_aq_dev_delay	uint64	Receiver internal delay
- receiver_free_send_buf	uint64	Receiver internal number of available send buffers
- storage_cell	uint64	Storage cell number
- saturated_pixel_count	uint64	Saturated pixel count
- error_pixel_count	uint64	Error pixel count
- strong_pixel_count	uint64	Strong pixel count (first stage of spot finding)
- data_collection_efficiency	float	Image collection efficiency []
- bkg_estimate	float	Mean value for pixels in resolution range from 3.0 to 5.0 A [photons] (with solid angle/polarization corrections, if applied)		X
- resolution_estimate	float	Diffraction resolution estimation		X
- adu_histogram	Array(uint64)	ADU histogram		X
- roi_integrals	object	Results of ROI calculation		X
- - sum	int64	Sum of pixels in ROI area [photons]
- - sum_square	int64	Sum of squares of pixels in ROI area [photons]
- - pixels	uint64	Valid pixels in ROI area
- - max_count	int64	Highest count in ROI area [photons]

End message

Field name	Type	Description	Present in DECTRIS format
type	String	value "end"	X
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
series_unique_id	string	Unique text ID of the series (run_name parameter)	X
series_id	uint64	Unique numeric ID of the series (run_number parameter)	X
end_date	date	Approximate end date
max_image_number	uint64	Number of image with the highest number (this is counted from 1 - to distinguish zero images and one image)
images_collected	uint64	Number of image collected
images_sent_to_write	uint64	Number of image sent to writer; if writer queues were full, it is possible this is less than images collected
data_collection_efficiency	float	Network packets collected / Network packets expected []
az_int_result	Map(text->Array(float))	Azimuthal integration results, use az_int_bin_to_q from start message for legend
adu_histogram	Map(text->Array(uint64))	ADU values histogram
adu_histogram_bin_width	uint64	Width of bins in the above histogram [ADU]
max_receiver_delay	uint64	Internal performance of Jungfraujoch
bkg_estimate	float	Mean background estimate for the whole run
indexing_rate	float	Mean indexing rate for the whole run

Calibration message

Field name	Type	Description	Present in DECTRIS format
type	String	value "calibration"
magic_number	uint64	Number used to describe version of the Jungfraujoch data interface - to allow to detect inconsistency between sender and receiver
data	Map(string -> Image)	Calibration map (only single pedestal array per message)

User data

In many cases there is an interest from facilities to forward more metadata, than available explicitly in the Jungfraujoch. For this reason two fields can be provided: header_appendix (sent with start message) and image_appendix (send with image message). To increase flexibility, both appendices can contain any valid JSON message. These appendices are serialized into string and stored in CBOR messages as user_data.

Notably for start message, user_data can contain more information (non-DECTRIS compliant metadata). Therefore user_data is serialized by Jungfraujoch as CBOR object. There is member user which contains header_appendix defined in OpenAPI of Jungfraujoch.

Notes on images and compression

• Images are encoded as DECTRIS MultiDimArray with typed array tags:
  • For RGB: shape [3, height, width], type: u8
  • For grayscale: shape [height, width], type according to bit depth and sign (e.g., uint16 LE)
• Compression:
  • Uncompressed: raw CBOR byte string
  • Bitshuffle+LZ4: tag with ["bslz4", elem_size, bytes]
  • Bitshuffle+Zstandard: tag with ["bszstd", elem_size, bytes]