Jungfraujoch/docs/JFJOCH_BROKER.md

jfjoch_broker

jfjoch_broker is the main service for the Jungfraujoch application. It is responsible for:

• Providing user interface via HTTP and OpenAPI
• Configuring FPGA firmware
• Building images from FPGA output and forwarding the results over ZeroMQ

External interfaces

Broker operates four external interfaces.

Image stream ZeroMQ PUSH socket with CBOR serialization is used to send images, metadata and processing results for writing or downstream processing. See details here.

Preview stream ZeroMQ PUB socket, as above but limited to subset of frames (1 image/s by default). See details here.

Metadata stream ZeroMQ PUB socket, contains metadata for all the images, with bundling. See details here.

Configuration, status and results interface HTTP/REST interface described in the OpenAPI format. Description of the API is presented in the OpenAPI specification.

A dataset can be protected: /start takes an optional tokens list, and while the current dataset has any, its statistics (/statistics/data_collection, /result/scan), buffered images (/image_buffer/*.cbor|jpeg|tiff) and plots (/preview/plot*) need Authorization: Bearer <token> and answer 401 otherwise; /statistics omits its measurement block instead. See Security.

Broker configuration

jfjoch_broker requires JSON configuration files. The file is described by OpenAPI structure jfjoch_settings defined in jfjoch_api.yaml file. It is recommended to go through example files in the etc/.

Example configuration (not every section is shown):

{
  "pcie": [
    {
      "blk": "/dev/jfjoch0",
      "ipv4": "10.1.1.7"
    },
    {
      "blk": "/dev/jfjoch1",
      "ipv4": "10.1.1.8"
    }
  ],
  "zeromq": {
    "send_watermark": 100,
    "send_buffer_size": 1024,
    "image_socket": [
      "tcp://1.2.3.4:5000",
      "tcp://1.2.3.4:5001"
    ],
    "writer_notification_socket": "tcp://1.3.4.6:7000"
  },
  "instrument": {
    "source_name": "Swiss Light Source",
    "source_type": "Synchrotron X-ray Source",
    "instrument_name": "X06SA",
    "pulsed_source": false,
    "electron_source": false
  },
  "detector": [
    {
      "description": "EIGER 1M",
      "serial_number": "E1M-01",
      "type": "EIGER",
      "high_voltage_V": 150,
      "udp_interface_count": 1,
      "module_sync": true,
      "sensor_thickness_um": 320,
      "calibration_file": [
        "gainMaps.bin"
      ],
      "hostname": [
        "e1m-01",
        "e1m-02"
      ],
      "readout_time_us": 3,
      "sensor_material": "Si",
      "tx_delay": [
        0,1
      ],
      "base_data_ipv4_address": "10.10.10.50",
      "standard_geometry": {
        "nmodules": 1,
        "gap_x": 8,
        "gap_y": 36,
        "modules_in_row": 1
      },
      "custom_geometry": [
        {
          "x0": 0,
          "y0": 0,
          "fast_axis": "Xp",
          "slow_axis": "Yp"
        }
      ],
      "mirror_y": true
    }
  ],
  "detector_settings": {
    "frame_time_us": 450,
    "count_time_us": 0,
    "internal_frame_generator": false,
    "internal_frame_generator_images": 1,
    "detector_trigger_delay_ns": 0,
    "timing": "auto",
    "eiger_threshold_keV": 6.0,
    "jungfrau_pedestal_g0_frames": 2000,
    "jungfrau_pedestal_g1_frames": 300,
    "jungfrau_pedestal_g2_frames": 300,
    "jungfrau_pedestal_g0_rms_limit": 100,
    "jungfrau_pedestal_min_image_count": 128,
    "jungfrau_storage_cell_count": 1,
    "jungfrau_storage_cell_delay_ns": 5000,
    "jungfrau_fixed_gain_g1": false,
    "jungfrau_use_gain_hg0": false
  },
  "azim_int": {
    "polarization_factor": -1,
    "solid_angle_corr": true,
    "high_q_recipA": 0,
    "low_q_recipA": 0,
    "q_spacing": 0
  },
  "image_format": {
    "summation": true,
    "geometry_transform": true,
    "jungfrau_conversion": true,
    "jungfrau_conversion_factor_keV": 0.001,
    "bit_depth_image": 16,
    "signed_output": true,
    "mask_module_edges": true,
    "mask_chip_edges": true
  },
  "image_buffer_MiB": 2048,
  "receiver_threads": 64,
  "frontend_directory": "/usr/share/jfjoch/frontend",
  "image_pusher": "ZeroMQ",
  "zeromq_metadata": {
    "enabled": true,
    "period_ms": 1000,
    "socket_address": "tcp://0.0.0.0:4357"
  },
  "zeromq_preview": {
    "enabled": true,
    "period_ms": 1000,
    "socket_address": "tcp://0.0.0.0:4356"
  }
}

Following a collection

A reader can process a collection while it is being written - rugnux does, given the path the master file will have and the broker's address (rugnux <prefix>_master.h5 http://<broker>:<port>/), started any time after /start was accepted. The images still travel as files; the broker says which files exist. GET /live/events, protected by the dataset's bearer tokens like the other dataset routes, streams server-sent events for the current collection from the moment /start accepted it: its history first, then every change. start comes first, once the start message is known - at once for a JUNGFRAU, when the detector sends its own for a DECTRIS - with that start message (CBOR, in base64); a reader builds the dataset from it. Then file (file number, file name relative to the collection's root, first image, image count) for each data file a writer has closed and renamed into place, and end (images collected, and an error if any) once every writer has acknowledged the end, so the data files and the master file are in place - or, with an error, when the measurement ended any other way. A comment line every 10 s keeps the connection alive. One reader at a time; a second one gets 409.

The writers report each closed data file to the broker (TCP protocol version 5, an acknowledgement for FILE_CLOSED); the in-process HDF5 writer reports to it directly. The ZeroMQ writer has no way back to the broker, so with it the route answers 409.

Rugnux live mode

With rugnux live mode on, rotation collections from a DECTRIS detector are processed by rugnux inside the broker while they arrive. It is off by default; the configuration file switches it on with "rugnux": {"enable": true, "reserved_cpus": 4}, where reserved_cpus is the number of CPU threads left to the receiver and the image pusher.

Rugnux reads the images back from the broker's image buffer, so the buffer must hold the whole collection - one slot per image. Where it holds fewer, the collection is only forwarded to the writer, as with the mode off; the live status reports it as skipped and the collection's statistics carry a warning saying how many images the buffer holds against how many the collection has.

The results - the merged data, the map coefficients and, with a model, the placed model - travel to the writer in the end message, which writes them beside the master file as <file_prefix>.mtz, <file_prefix>_maps.mtz, <file_prefix>_model.cif and <file_prefix>_model.pdb. An end message that would be too large for the pushers' buffer (256 MiB) goes without them, with an error in the log and a warning in the live status; the collection itself is not affected.

The same settings are read with GET /config/rugnux and changed with PUT /config/rugnux. A model for molecular replacement and validation (PDB or mmCIF, optionally gzipped, at most 64 MiB) is uploaded as the raw file with PUT /config/rugnux/model (application/octet-stream; a larger body gets 413, an empty one or one that cannot be read as a model 400). The broker parses it at upload and keeps it in memory - nothing is written to disk - for every following collection until DELETE /config/rugnux/model removes it or another upload replaces it; a broker restart drops it. GET /config/rugnux reports whether a model is loaded and its size, never its contents. Changes are refused during a collection. Like the rest of the configuration these routes need no bearer token: they prepare the next collection and expose nothing of the current one.

The end message, and with it the end of the collection, waits for the analysis. deadline_s (default 60 s; 0 for no cutoff) bounds that wait: this long after the last image the analysis is cut. POST /cancel is the other way to cut it: while images still arrive it cancels the collection as with the mode off, and the analysis with it; after the last image it ends only the analysis. A cut analysis gives what it has established: a pass that is past its images still merges and makes its outputs, the canonical pass cut among its images merges those it has read, only the placed model is left out, and any other pass cut short gives way to the last one that ran to its end. The end message goes out at once with that, and the live status records the cut (deadline or cancel) and when.

GET /rugnux/status reports the live run of the last collection, kept until the next one starts: its state, the images received, the step it is on and how far into it, the steps so far with what each found, and once it is over the verdict, the summary, warnings and the time from the last image to every result written. The web frontend shows it on the dashboard and in its "Rugnux live" tab, which also changes the settings and uploads the model. It needs the dataset's bearer token like the dataset routes (/statistics/data_collection, /image_buffer/*, /preview/plot*). GET /result/scan carries the rotation lattice, unit cell and Bravais symbol rugnux found. A collection in which rugnux finds no lattice still gets the spot count, resolution estimate and background of every image in GET /result/scan and the master file, marked as not indexed.

Setting up a local test for Jungfraujoch

For development, it is possible to set up a local installation of Jungfraujoch. This will work without FPGA installed in the computer and allows testing the Jungfraujoch software layer, including ZeroMQ streaming and file writing.

The workflow simulates FPGA behavior, by running high-level synthesis code on the CPU - the performance is therefore very low, as fixed-point calculations have a large performance penalty on the CPU. In the CPU simulation mode, one can simulate using only a single FPGA device.

To run the test:

Compile Jungfraujoch with frontend

mkdir build
cd build
cmake ..
make jfjoch_broker
make frontend

Alternatively, on a RHEL8 system, you can use the RPMs generated by the automated pipeline. The jfjoch package alone is enough. In this case - it is necessary to update etc/broker_local.json file with frontend path in /usr/share/jfjoch/frontend.

Start service

Start broker:

cd build/broker
./jfjoch_broker ../../etc/broker_local.json 5232

Run tests

To run the test, a Python script is provided:

cd tests/test_data
python jfjoch_broker_test.py

The script will initialize Jungfraujoch, import test image and start data collection.

Expected result

You can observe online data analysis by opening the following web page: http://localhost:5232. Also, a dataset with images should be written in the build/broker directory.