Jungfraujoch/docs/JFJOCH_WRITER.md

jfjoch_writer

jfjoch_writer is a NeXus-compliant HDF5 file writer.

Acknowledgements

Thanks to Zdenek Matej (MAX IV) and Felix Engelmann (MAX IV) for testing and multiple improvement suggestions.

Running directory

The writer needs to run in the base directory for writing files - file_prefix is always relative to the writer's running directory. The writer detects and protects against basic security issues, like file_prefix starting with a slash, or starting with ../, or containing /../.

Usage

Writer needs to be started as a background service, with the following command:

jfjoch_writer {options} <address to connect via ZeroMQ to DCU>

Options:
-T        | --tcp                      Use raw TCP/IP instead of ZeroMQ
-j<int>   | --nproc=<int>              Number of forks (only with -T)
-d<path>  | --root_dir=<path>          Root directory for file writing (-R is a deprecated alias)
-r<int>   | --zmq_repub_port=<int>     ZeroMQ port for PUSH socket to republish images
-f<int>   | --zmq_file_port=<int>      ZeroMQ port for PUB socket for notifications on finalized files
-w<int>   | --rcv_watermark=<int>      Receiving ZeroMQ socket watermark in messages (default = 10)
-m<int>   | --rcv_ring_mb=<int>        Receive ring buffer in MB; holds images until written and republished (default = 2048)
-W<int>   | --repub_watermark=<int>    Republish ZeroMQ socket watermark (default = 1000)
-b<int>   | --rcv_buffer=<int>         Receiving TCP socket buffer in bytes (only with -T; disables autotuning, capped by net.core.rmem_max)
-v        | --verbose                  Verbose output
-h                                     This message

for example:

jfjoch_writer -d /data tcp://dcu-address:5400 

Memory

Every message the writer receives is placed in a ring buffer of a fixed size (-m, default 2048 MB, per writer process - -j 8 is eight rings) and stays there until it has been written to HDF5 and, if republish is on, republished. The ring is the writer's memory budget for buffered images, and the peak use is printed at the end of every run. Size it as the number of images you can afford to have in flight times their compressed size: at 2048 MB that is some 250-500 JUNGFRAU 4M images but only 50-100 JUNGFRAU 9M images, so a 9M site will want more.

When the ring is full the writer stops reading from the socket and the sender has to wait. How long it is prepared to wait depends on the stream:

• ZeroMQ. The socket queue fills up to its watermark (-w, a count of messages, kept small for that reason) and the broker's send blocks for one second. After that the broker drops the image, logs the fact, and sends without waiting for the rest of the run, so every later image that finds the queue full is dropped too. A writer that stalls for longer than its ring covers therefore loses images on ZeroMQ; the ring has to be large enough for the stalls of the filesystem it writes to. (Before the ring the same stall was absorbed by queues of about 1500 messages, whatever their size - a budget nobody had chosen.)
• TCP/IP. The broker sees the writer's BUSY heartbeats and waits, giving up only after a minute in which not a single image was written.

Republish holds its own images in the same ring. Because the ring is reclaimed in arrival order, an image kept for a slow republish consumer would hold back everything received after it, so republish may hold at most a quarter of the ring; beyond that images are simply not republished, exactly as when its queue is full. A single message larger than the whole ring - a START or END message of a very long run - is held outside it.

Status and cancellation

When a data collection is finalized, each writer reports its outcome back to jfjoch_broker over the writer notification socket — a ZeroMQ address the broker passes in the START message (writer_notification_socket in the broker configuration) — as a JSON message with the socket number, run name and number, processed image count, throughput, and on failure an error string. That is how the broker learns that a writer could not write. On the TCP/IP image stream, failures additionally come back in-band as negative acknowledgements (see Data streams).

To stop a writer, send it SIGINT, SIGQUIT, SIGTERM or SIGHUP: it closes the HDF5 files it is writing and exits. This is only for the case where the broker was terminated or disconnected — it is not the normal way to end a data collection, which the broker finishes on its own.

Republish

Republish creates a PUSH socket on the writer, where all the messages are republished for further use by a data analysis pipeline. Republish is non-blocking, so if there is no receiver on other end or the sending queue is full - images won't be republished. In case of START/END messages republishing will attempt sending for 100 ms, but if send times out it won't be retried.

Republish functionality is optional, if republish port number is omitted this functionality is not enabled.

Overwriting files

When jfjoch_writer creates a HDF5 file, it first adds suffix .<random>.tmp. Random value depends on current time-stamp and likely will be different from each file of the particular series. After file is all saved and closed, it is renamed to remove the suffix. By default, renaming won't happen if this would overwrite existing file. However, this behavior can be changed by setting overwrite parameter to true in the file writer configuration.

When the overwrite conflict is reported

An existing output file is a fatal condition (unless overwrite is true). When it is detected depends on whether the transport between the broker and the writer has a back-channel to report the failure before acquisition starts:

• Direct HDF5 pusher and TCP writer (back-channel available). The conflict is detected at start: the writer that owns the master file checks whether it already exists and refuses to start. The direct pusher raises the error in-process; the TCP writer returns a START-failure acknowledgement. Either way the broker learns immediately and aborts the data collection before the detector is armed — no images are taken and nothing is written. Only the master file is checked up front: in a multi-writer setup the per-image data files are staggered across writers, and checking them at start would make each writer inspect files it never writes (and race the writers that do). Data-file conflicts are instead caught by their owning writer at the final rename, which for the TCP path surfaces as a write-failure acknowledgement to the broker.
• ZeroMQ writer (no back-channel). The ZeroMQ image stream is fire-and-forget: the writer has no way to tell the broker to stop, and the broker would keep streaming images regardless. The writer therefore does not fail at start. It writes the whole series to the .<random>.tmp files as usual and only fails at the final rename, leaving the .tmp files on disk. This is deliberate: the acquired images are preserved (in .tmp form) rather than being dropped by a writer that aborted mid-stream. Rename the .tmp files by hand, or re-run with overwrite set, to recover them.

Finalized files information

Creates PUB socket to inform about finalized data files. For each closed file, the socket will send a JSON message, with the following structure:

{
  "filename": <string>: HDF5 data file name (relative to writer root directory),
  "nimages": <int> number of images in the file (counting from 1!),
  "file_number": <int> number of file within the acquisition,
  "sample_name": <string> name of sample,
  "run_name": <string> name of run,
  "run_number": <int> number of run,
  "experiment_group": <string> number of p-group / proposal (optional),
  "user_data": <any json> user_data,
  "beam_x_pxl": <float> beam center (X) in pixels,
  "beam_y_pxl": <float> beam center (Y) in pixels,
  "detector_distance_m": <float> detector distance in m,
  "detector_height_pxl": <int> detector size (Y) in pixels,
  "detector_width_pxl": <int> detector size (X) in pixels,
  "incident_energy_eV": <float> photon energy of the X-ray beam,
  "pixel_size_m": <float> pixel size in meter (assuming pixel X == Y),
  "saturation": <int> this count and higher mean saturation,
  "space_group_number": <int> space group number (optional),
  "underload": <int> lowest valid count; anything below it is invalid,
  "unit_cell": <optional> unit cell dimensions in Angstrom/degree {
    "a": <float>, "b": <float>, "c": <float>,
    "alpha": <float>, "beta": <float>, "gamma": <float>
  },
}

user_data is defined as header_appendix in the /start operation in the jfjoch_broker. Other metadata are also carried over from /start operation.

If the header_appendix is a string with valid JSON meaning, it will be embedded as JSON, otherwise it will be escaped as string. For example header_appendix of {"param1": "test1", "param2": ["test1", "test2"]}, then the example message will look as follows:

{
  "filename": "dataset_name_data_000001.h5",
  "nimages": 1000,
  "file_number": 0,
  "sample_name": "my_sample",
  "run_name": "my_run",
  "run_number": 25,
  "experiment_group": "p00001",
  "beam_x_pxl": 1200,
  "beam_y_pxl": 1500,
  "detector_distance_m": 0.155,
  "detector_height_pxl": 2164,
  "detector_width_pxl": 2068,
  "incident_energy_eV": 12400.0,
  "pixel_size_m": 7.5e-05,
  "saturation": 32766,
  "space_group_number": 96,
  "underload": -32767,
  "unit_cell": {
    "a": 78.0,
    "alpha": 90.0,
    "b": 78.0,
    "beta": 90.0,
    "c": 39.0,
    "gamma": 90.0
  },
  "user_data": {
    "param1": "test1", 
    "param2": ["test1", "test2"]
  }
}

Notifications for finalized files are optional, if notification port number is omitted this functionality is not enabled.

HDF5 file structure

Jungfraujoch writes NXmx-compliant HDF5, with substantial derived metadata (spot finding, indexing, integration, azimuthal integration, per-image statistics and timing) stored beyond the NXmx standard. The complete file layout — master vs data files, the three format variants (NXmxLegacy, NXmxVDS, NXmxIntegrated), every NXmx field that is populated and every Jungfraujoch extension — is documented in HDF5 / NeXus data format.

If data collection was configured with a header_appendix containing a key hdf5 whose value is a JSON object of numbers and strings, those entries are written to /entry/user.

Other formats (CBF and TIFF)

Earlier versions could also write Crystallographic Binary File (CBF, miniCBF) and TIFF images. These writers have been removed: Jungfraujoch now writes only NXmx HDF5. The CBF and TIFF values are retained in the file-format enum for wire back-compatibility, but a request to write either format is rejected.

No file option(s)

There are two options to disable writing of files by the writer:

• Setting file_prefix to empty string - this will disable sending files on ZeroMQ image socket.
• Setting file format to NoFile - files are streamed over ZeroMQ socket, but jfjoch_writer will not write anything. This can be useful for debugging purposes, or if you only rely on republishing functionality of the jfjoch_writer