Jungfraujoch/docs/ZEROMQ_STREAM.md

ZeroMQ socket

Jungfraujoch process (jfjoch_broker) operates three ZeroMQ outputs. All three can be operated/enabled independently. These are:

• Image - all the images including metadata (PUSH socket)
• Preview - images with metadata at a reduced frame rate (PUB socket)
• Metadata - only metadata for all the images, bundled into packages (PUB socket)

Image stream

Images (with metadata) are serialized as CBOR image message. The stream will also include CBOR start message, calibration messages and end message with run metadata.

Image stream can be split into multiple sockets to increase performance, in this case images will be split according to file number to which the image belongs. All sockets will forward start and end messages. Only first socket will forward calibration messages and will be marked to write master file.

This is using PUSH ZeroMQ socket(s). It should be strictly avoided to have multiple receivers connected to one PUSH ZeroMQ socket. ZeroMQ will send the images in a round-robin basis to the receivers. In this case start and end messages will end up only with one receiver. Instead, Jungfraujoch feature of multiple sockets should be used.

Image stream can be replaced with direct HDF5 writer and CBOR dump image pushers, it can be disabled by select "None" image pusher for all the measurements.

If file_prefix is not provided for a data collection, images won't be sent to image stream (or its HDF5/CBOR replacements).

Behavior is as following:

• Start message is sent with timeout of 5s. If within the time the message cannot be put in the outgoing queue or there is no connected puller exception is thrown - stop data collection with error due to absence of a writer.
• Images are sent in non-blocking way and without timeout.
• End message is sent with timeout of 5s. No error is reported.

Writer notification socket

Normally ZeroMQ is asynchronous. When jfjoch_broker is sending messages via ZeroMQ image stream, it doesn't know if these were properly handled downstream, e.g., written to disk. For this reason a writer notification socket is introduced. It allows to downstream processing code to notify 'jfjoch_broker' that all images were handled properly.

To use writer notification socket, it has to be first enabled in the JSON configuration file of broker with writer_notification_socket entry:

{
  "writer_notification_socket":"tcp://192.168.0.1:*"
}

Such entry will create PULL socket on 192.168.0.1 network interface listening on one, random TCP port. When data processing is started, the image stream will send CBOR start message. This message will include information on writer_notification_zmq_addr, which needs to be used by downstream code. Since the start message must reference the address of jfjoch_broker host, notification socket should always listen on a particular network interface, and should not be configured with placeholder address 0.0.0.0. It is, however, OK to use placeholder :* for network port, as it will be substituted for the one chosen by ZeroMQ.

For every image stream, downstream code must send the following message to the PULL socket:

{
  "run_number":135,
  "run_name": "lysozyme_1",
  "socket_number": 1,
  "processed_images":250,
  "ok": true
}

Here run_number, run_name and socket_number must match information from the start message. ok is boolean confirming if the writing process was OK. processed_images is number of images that were written/processed, this is to track how many images were ignored by non-blocking ZeroMQ procedures. If not, it is possible to include error message:

{
  "run_number":135,
  "run_name": "lysozyme_1",
  "socket_number": 1,
  "processed_images": 0,
  "ok": false,
  "error": "Permission error"
}

This way errors from the downstream code are propagated to jfjoch_broker.

If writer notification socket is configured, but downstream code doesn't send proper notification, jfjoch_broker will time out after 60 seconds producing an error message.

Preview stream

Jungfraujoch can also send images (with metadata) at a reduced frame rate for preview purpose. Images are serialized as CBOR image message. The stream will also include CBOR start message and end message with run metadata. Only start and image messages are sent.

This is using PUB socket with conflate option. I.e., only the last message is kept by ZeroMQ, so if receiver cannot cope with the messages, it will always receive the last generated message (no backlog). For this reason it is also recommended to use the same option on receiver side.

Given PUB socket properties, it is possible to connect multiple viewers to a single socket --- all the viewers should receive all the images sent.

Metadata stream

Jungfrajoch can also send pure metadata for the purpose of archiving such information. Metadata are serialized as CBOR metadata message. This is very similar as image message, but excludes the actual image array and spot positions. As metadata are relatively small, to avoid large number of messages, Jungfraujoch bundles metadata of many images in one message. Order of images within bundle, as well a size of the bundle, are not guaranteed The stream will also include CBOR start message and end message with run metadata.

This is using PUB socket with watermark, so there is some queuing of messages with ZeroMQ. Multiple receivers can be connected.