Jungfraujoch/docs/JFJOCH_BROKER.md

# jfjoch_broker

`jfjoch_broker` is the main service for the Jungfraujoch application. It is responsible for:

* Providing user interface via HTTP and OpenAPI
* Configuring FPGA firmware
* Building images from FPGA output and forwarding the results over ZeroMQ

## External interfaces
Broker operates four external interfaces.

**Image stream** ZeroMQ PUSH socket with CBOR serialization is used to send images, metadata and processing results for writing or downstream 
processing. See details [here](IMAGE_STREAM.md#image-stream).

**Preview stream** ZeroMQ PUB socket, as above but limited to subset of frames (1 image/s by default). See details [here](IMAGE_STREAM.md#preview-stream).

**Metadata stream** ZeroMQ PUB socket, contains metadata for all the images, with bundling. See details [here](IMAGE_STREAM.md#metadata-stream).

**Configuration, status and results interface** HTTP/REST interface described in the OpenAPI format.
Description of the API is presented in the [OpenAPI specification](OPENAPI_SPECS.rst). 

A dataset can be protected: `/start` takes an optional `tokens` list, and while the current dataset
has any, its statistics (`/statistics/data_collection`, `/result/scan`), buffered images
(`/image_buffer/*.cbor|jpeg|tiff`) and plots (`/preview/plot*`) need `Authorization: Bearer <token>`
and answer `401` otherwise; `/statistics` omits its `measurement` block instead. See
[Security](SECURITY.md#3-authenticated-read-access--per-dataset-bearer-tokens).

## Broker configuration
`jfjoch_broker` requires JSON configuration files. The file is described by OpenAPI structure `jfjoch_settings` defined in `jfjoch_api.yaml` file.
It is recommended to go through example files in the `etc/`.

Example configuration (not every section is shown):

```json
{
  "pcie": [
    {
      "blk": "/dev/jfjoch0",
      "ipv4": "10.1.1.7"
    },
    {
      "blk": "/dev/jfjoch1",
      "ipv4": "10.1.1.8"
    }
  ],
  "zeromq": {
    "send_watermark": 100,
    "send_buffer_size": 1024,
    "image_socket": [
      "tcp://1.2.3.4:5000",
      "tcp://1.2.3.4:5001"
    ],
    "writer_notification_socket": "tcp://1.3.4.6:7000"
  },
  "instrument": {
    "source_name": "Swiss Light Source",
    "source_type": "Synchrotron X-ray Source",
    "instrument_name": "X06SA",
    "pulsed_source": false,
    "electron_source": false
  },
  "detector": [
    {
      "description": "EIGER 1M",
      "serial_number": "E1M-01",
      "type": "EIGER",
      "high_voltage_V": 150,
      "udp_interface_count": 1,
      "module_sync": true,
      "sensor_thickness_um": 320,
      "calibration_file": [
        "gainMaps.bin"
      ],
      "hostname": [
        "e1m-01",
        "e1m-02"
      ],
      "readout_time_us": 3,
      "sensor_material": "Si",
      "tx_delay": [
        0,1
      ],
      "base_data_ipv4_address": "10.10.10.50",
      "standard_geometry": {
        "nmodules": 1,
        "gap_x": 8,
        "gap_y": 36,
        "modules_in_row": 1
      },
      "custom_geometry": [
        {
          "x0": 0,
          "y0": 0,
          "fast_axis": "Xp",
          "slow_axis": "Yp"
        }
      ],
      "mirror_y": true
    }
  ],
  "detector_settings": {
    "frame_time_us": 450,
    "count_time_us": 0,
    "internal_frame_generator": false,
    "internal_frame_generator_images": 1,
    "detector_trigger_delay_ns": 0,
    "timing": "auto",
    "eiger_threshold_keV": 6.0,
    "jungfrau_pedestal_g0_frames": 2000,
    "jungfrau_pedestal_g1_frames": 300,
    "jungfrau_pedestal_g2_frames": 300,
    "jungfrau_pedestal_g0_rms_limit": 100,
    "jungfrau_pedestal_min_image_count": 128,
    "jungfrau_storage_cell_count": 1,
    "jungfrau_storage_cell_delay_ns": 5000,
    "jungfrau_fixed_gain_g1": false,
    "jungfrau_use_gain_hg0": false
  },
  "azim_int": {
    "polarization_factor": -1,
    "solid_angle_corr": true,
    "high_q_recipA": 0,
    "low_q_recipA": 0,
    "q_spacing": 0
  },
  "image_format": {
    "summation": true,
    "geometry_transform": true,
    "jungfrau_conversion": true,
    "jungfrau_conversion_factor_keV": 0.001,
    "bit_depth_image": 16,
    "signed_output": true,
    "mask_module_edges": true,
    "mask_chip_edges": true
  },
  "image_buffer_MiB": 2048,
  "receiver_threads": 64,
  "frontend_directory": "/usr/share/jfjoch/frontend",
  "image_pusher": "ZeroMQ",
  "zeromq_metadata": {
    "enabled": true,
    "period_ms": 1000,
    "socket_address": "tcp://0.0.0.0:4357"
  },
  "zeromq_preview": {
    "enabled": true,
    "period_ms": 1000,
    "socket_address": "tcp://0.0.0.0:4356"
  }
}
```

## Following a collection

A reader can process a collection while it is being written - rugnux does, given the path the master
file will have and the broker's address (`rugnux <prefix>_master.h5 http://<broker>:<port>/`), started
any time after `/start` was accepted. The images still travel as files; the broker says which files
exist. `GET /live/events`, protected by the dataset's bearer tokens like the other dataset routes,
streams server-sent events for the current collection from the moment `/start` accepted it: its
history first, then every change. `start` comes first, once the start message is known - at once
for a JUNGFRAU, when the detector sends its own for a DECTRIS - with that start message (CBOR, in
base64); a reader builds the dataset from it. Then `file` (file number, file name relative to the
collection's root, first image, image count) for each data file a writer has closed and renamed into
place, and `end` (images collected, and an error if any) once every writer has acknowledged the end,
so the data files and the master file are in place - or, with an error, when the measurement ended
any other way. A comment line every 10 s keeps the connection alive. One reader at a time; a second
one gets 409.

The writers report each closed data file to the broker (TCP protocol version 5, an acknowledgement
for `FILE_CLOSED`); the in-process HDF5 writer reports to it directly. The ZeroMQ writer has no way
back to the broker, so with it the route answers 409.

## Rugnux live mode

With rugnux live mode on, rotation collections from a DECTRIS detector are processed by rugnux inside
the broker while they arrive. It is off by default; the configuration file switches it on with
`"rugnux": {"enable": true, "reserved_cpus": 4}`, where `reserved_cpus` is the number of CPU threads
left to the receiver and the image pusher.

Rugnux reads the images back from the broker's image buffer, so the buffer must hold the whole
collection - one slot per image. Where it holds fewer, the collection is only forwarded to the writer,
as with the mode off; the live status reports it as skipped and the collection's statistics carry a
warning saying how many images the buffer holds against how many the collection has.

The results - the merged data, the map coefficients and, with a model, the placed model - travel to
the writer in the end message, which writes them beside the master file as `<file_prefix>.mtz`,
`<file_prefix>_maps.mtz`, `<file_prefix>_model.cif` and `<file_prefix>_model.pdb`. An end message
that would be too large for the pushers' buffer (256 MiB) goes without them, with an error in the log
and a warning in the live status; the collection itself is not affected.

The same settings are read with `GET /config/rugnux` and changed with `PUT /config/rugnux`. A model
for molecular replacement and validation (PDB or mmCIF, optionally gzipped, at most 64 MiB) is
uploaded as the raw file with `PUT /config/rugnux/model` (`application/octet-stream`; a larger body
gets `413`, an empty one or one that cannot be read as a model `400`). The broker parses it at upload
and keeps it in memory - nothing is written to disk - for every following collection until
`DELETE /config/rugnux/model` removes it or another upload replaces it; a broker restart drops it.
`GET /config/rugnux` reports whether a model is loaded and its size, never its contents. Changes are
refused during a collection. Like the rest of the configuration these routes need no bearer token:
they prepare the next collection and expose nothing of the current one.

The end message, and with it the end of the collection, waits for the analysis. `deadline_s` (default
60 s; 0 for no cutoff) bounds that wait: this long after the last image the analysis is cut. `POST /cancel` is
the other way to cut it: while images still arrive it cancels the collection as with the mode off,
and the analysis with it; after the last image it ends only the analysis. A cut analysis gives what it
has established: a pass that is past its images still merges and makes its outputs, the canonical pass
cut among its images merges those it has read, only the placed model is left out, and
any other pass cut short gives way to the last one that ran to its end. The end message goes out at
once with that, and the live status records the cut (`deadline` or `cancel`) and when.

`GET /rugnux/status` reports the live run of the last collection, kept until the next one starts: its
state, the images received, the step it is on and how far into it, the steps so far with what each
found, and once it is over the verdict, the summary, warnings and the time from the last image to
every result written. The web frontend shows it on the dashboard and in its "Rugnux live" tab, which
also changes the settings and uploads the model. It needs the dataset's bearer token
like the dataset routes (`/statistics/data_collection`, `/image_buffer/*`, `/preview/plot*`). `GET /result/scan` carries the rotation lattice, unit cell and Bravais symbol
rugnux found. A collection in which rugnux finds no lattice still gets the spot count, resolution
estimate and background of every image in `GET /result/scan` and the master file, marked as not indexed.

## Setting up a local test for Jungfraujoch
For development, it is possible to set up a local installation of Jungfraujoch. 
This will work without FPGA installed in the computer and allows testing the Jungfraujoch software layer, including
ZeroMQ streaming and file writing. 

The workflow simulates FPGA behavior, by running high-level synthesis code on the CPU - the performance is therefore 
very low, as fixed-point calculations have a large performance penalty on the CPU. In the CPU simulation mode, one can simulate 
using only a single FPGA device. 

To run the test:

### Compile Jungfraujoch with frontend
```
mkdir build
cd build
cmake ..
make jfjoch_broker
make frontend
```
Alternatively, on a RHEL8 system, you can use the RPMs generated by the automated pipeline.
The `jfjoch` package alone is enough.
In this case - it is necessary to update `etc/broker_local.json` file with frontend path in `/usr/share/jfjoch/frontend`.

### Start service
Start broker:
```
cd build/broker
./jfjoch_broker ../../etc/broker_local.json 5232
```

### Run tests
To run the test, a Python script is provided:
```
cd tests/test_data
python jfjoch_broker_test.py
```
The script will initialize Jungfraujoch, import test image and start data collection.

### Expected result
You can observe online data analysis by opening the following web page: [http://localhost:5232](http://localhost:5232).
Also, a dataset with images should be written in the `build/broker` directory.