# jfjoch_broker `jfjoch_broker` is the main service for the Jungfraujoch application. It is responsible for: * Providing user interface via HTTP and OpenAPI * Configuring FPGA firmware * Building images from FPGA output and forwarding the results over ZeroMQ ## External interfaces Broker operates four external interfaces. **Image stream** ZeroMQ PUSH socket with CBOR serialization is used to send images, metadata and processing results for writing or downstream processing. See details [here](IMAGE_STREAM.md#image-stream). **Preview stream** ZeroMQ PUB socket, as above but limited to subset of frames (1 image/s by default). See details [here](IMAGE_STREAM.md#preview-stream). **Metadata stream** ZeroMQ PUB socket, contains metadata for all the images, with bundling. See details [here](IMAGE_STREAM.md#metadata-stream). **Configuration, status and results interface** HTTP/REST interface described in the OpenAPI format. Description of the API is presented in the [OpenAPI specification](OPENAPI_SPECS.rst). A dataset can be protected: `/start` takes an optional `tokens` list, and while the current dataset has any, its statistics (`/statistics/data_collection`, `/result/scan`), buffered images (`/image_buffer/*.cbor|jpeg|tiff`) and plots (`/preview/plot*`) need `Authorization: Bearer ` and answer `401` otherwise; `/statistics` omits its `measurement` block instead. See [Security](SECURITY.md#3-authenticated-read-access--per-dataset-bearer-tokens). ## Broker configuration `jfjoch_broker` requires JSON configuration files. The file is described by OpenAPI structure `jfjoch_settings` defined in `jfjoch_api.yaml` file. It is recommended to go through example files in the `etc/`. Example configuration (not every section is shown): ```json { "pcie": [ { "blk": "/dev/jfjoch0", "ipv4": "10.1.1.7" }, { "blk": "/dev/jfjoch1", "ipv4": "10.1.1.8" } ], "zeromq": { "send_watermark": 100, "send_buffer_size": 1024, "image_socket": [ "tcp://1.2.3.4:5000", "tcp://1.2.3.4:5001" ], "writer_notification_socket": "tcp://1.3.4.6:7000" }, "instrument": { "source_name": "Swiss Light Source", "source_type": "Synchrotron X-ray Source", "instrument_name": "X06SA", "pulsed_source": false, "electron_source": false }, "detector": [ { "description": "EIGER 1M", "serial_number": "E1M-01", "type": "EIGER", "high_voltage_V": 150, "udp_interface_count": 1, "module_sync": true, "sensor_thickness_um": 320, "calibration_file": [ "gainMaps.bin" ], "hostname": [ "e1m-01", "e1m-02" ], "readout_time_us": 3, "sensor_material": "Si", "tx_delay": [ 0,1 ], "base_data_ipv4_address": "10.10.10.50", "standard_geometry": { "nmodules": 1, "gap_x": 8, "gap_y": 36, "modules_in_row": 1 }, "custom_geometry": [ { "x0": 0, "y0": 0, "fast_axis": "Xp", "slow_axis": "Yp" } ], "mirror_y": true } ], "detector_settings": { "frame_time_us": 450, "count_time_us": 0, "internal_frame_generator": false, "internal_frame_generator_images": 1, "detector_trigger_delay_ns": 0, "timing": "auto", "eiger_threshold_keV": 6.0, "jungfrau_pedestal_g0_frames": 2000, "jungfrau_pedestal_g1_frames": 300, "jungfrau_pedestal_g2_frames": 300, "jungfrau_pedestal_g0_rms_limit": 100, "jungfrau_pedestal_min_image_count": 128, "jungfrau_storage_cell_count": 1, "jungfrau_storage_cell_delay_ns": 5000, "jungfrau_fixed_gain_g1": false, "jungfrau_use_gain_hg0": false }, "azim_int": { "polarization_factor": -1, "solid_angle_corr": true, "high_q_recipA": 0, "low_q_recipA": 0, "q_spacing": 0 }, "image_format": { "summation": true, "geometry_transform": true, "jungfrau_conversion": true, "jungfrau_conversion_factor_keV": 0.001, "bit_depth_image": 16, "signed_output": true, "mask_module_edges": true, "mask_chip_edges": true }, "image_buffer_MiB": 2048, "receiver_threads": 64, "frontend_directory": "/usr/share/jfjoch/frontend", "image_pusher": "ZeroMQ", "zeromq_metadata": { "enabled": true, "period_ms": 1000, "socket_address": "tcp://0.0.0.0:4357" }, "zeromq_preview": { "enabled": true, "period_ms": 1000, "socket_address": "tcp://0.0.0.0:4356" } } ``` ## Following a collection A reader can process a collection while it is being written - rugnux does, given the path the master file will have and the broker's address (`rugnux _master.h5 http://:/`), started any time after `/start` was accepted. The images still travel as files; the broker says which files exist. `GET /live/events`, protected by the dataset's bearer tokens like the other dataset routes, streams server-sent events for the current collection from the moment `/start` accepted it: its history first, then every change. `start` comes first, once the start message is known - at once for a JUNGFRAU, when the detector sends its own for a DECTRIS - with that start message (CBOR, in base64); a reader builds the dataset from it. Then `file` (file number, file name relative to the collection's root, first image, image count) for each data file a writer has closed and renamed into place, and `end` (images collected, and an error if any) once every writer has acknowledged the end, so the data files and the master file are in place - or, with an error, when the measurement ended any other way. A comment line every 10 s keeps the connection alive. One reader at a time; a second one gets 409. The writers report each closed data file to the broker (TCP protocol version 5, an acknowledgement for `FILE_CLOSED`); the in-process HDF5 writer reports to it directly. The ZeroMQ writer has no way back to the broker, so with it the route answers 409. ## Rugnux live mode With rugnux live mode on, rotation collections from a DECTRIS detector are processed by rugnux inside the broker while they arrive. It is off by default; the configuration file switches it on with `"rugnux": {"enable": true, "reserved_cpus": 4}`, where `reserved_cpus` is the number of CPU threads left to the receiver and the image pusher. Rugnux reads the images back from the broker's image buffer, so the buffer must hold the whole collection - one slot per image. Where it holds fewer, the collection is only forwarded to the writer, as with the mode off; the live status reports it as skipped and the collection's statistics carry a warning saying how many images the buffer holds against how many the collection has. The results - the merged data, the map coefficients and, with a model, the placed model - travel to the writer in the end message, which writes them beside the master file as `.mtz`, `_maps.mtz`, `_model.cif` and `_model.pdb`. An end message that would be too large for the pushers' buffer (256 MiB) goes without them, with an error in the log and a warning in the live status; the collection itself is not affected. The same settings are read with `GET /config/rugnux` and changed with `PUT /config/rugnux`. A model for molecular replacement and validation (PDB or mmCIF, optionally gzipped, at most 64 MiB) is uploaded as the raw file with `PUT /config/rugnux/model` (`application/octet-stream`; a larger body gets `413`, an empty one or one that cannot be read as a model `400`). The broker parses it at upload and keeps it in memory - nothing is written to disk - for every following collection until `DELETE /config/rugnux/model` removes it or another upload replaces it; a broker restart drops it. `GET /config/rugnux` reports whether a model is loaded and its size, never its contents. Changes are refused during a collection. Like the rest of the configuration these routes need no bearer token: they prepare the next collection and expose nothing of the current one. The end message, and with it the end of the collection, waits for the analysis. `deadline_s` (default 60 s; 0 for no cutoff) bounds that wait: this long after the last image the analysis is cut. `POST /cancel` is the other way to cut it: while images still arrive it cancels the collection as with the mode off, and the analysis with it; after the last image it ends only the analysis. A cut analysis gives what it has established: a pass that is past its images still merges and makes its outputs, the canonical pass cut among its images merges those it has read, only the placed model is left out, and any other pass cut short gives way to the last one that ran to its end. The end message goes out at once with that, and the live status records the cut (`deadline` or `cancel`) and when. `GET /rugnux/status` reports the live run of the last collection, kept until the next one starts: its state, the images received, the step it is on and how far into it, the steps so far with what each found, and once it is over the verdict, the summary, warnings and the time from the last image to every result written. The web frontend shows it on the dashboard and in its "Rugnux live" tab, which also changes the settings and uploads the model. It needs the dataset's bearer token like the dataset routes (`/statistics/data_collection`, `/image_buffer/*`, `/preview/plot*`). `GET /result/scan` carries the rotation lattice, unit cell and Bravais symbol rugnux found. A collection in which rugnux finds no lattice still gets the spot count, resolution estimate and background of every image in `GET /result/scan` and the master file, marked as not indexed. ## Setting up a local test for Jungfraujoch For development, it is possible to set up a local installation of Jungfraujoch. This will work without FPGA installed in the computer and allows testing the Jungfraujoch software layer, including ZeroMQ streaming and file writing. The workflow simulates FPGA behavior, by running high-level synthesis code on the CPU - the performance is therefore very low, as fixed-point calculations have a large performance penalty on the CPU. In the CPU simulation mode, one can simulate using only a single FPGA device. To run the test: ### Compile Jungfraujoch with frontend ``` mkdir build cd build cmake .. make jfjoch_broker make frontend ``` Alternatively, on a RHEL8 system, you can use the RPMs generated by the automated pipeline. The `jfjoch` package alone is enough. In this case - it is necessary to update `etc/broker_local.json` file with frontend path in `/usr/share/jfjoch/frontend`. ### Start service Start broker: ``` cd build/broker ./jfjoch_broker ../../etc/broker_local.json 5232 ``` ### Run tests To run the test, a Python script is provided: ``` cd tests/test_data python jfjoch_broker_test.py ``` The script will initialize Jungfraujoch, import test image and start data collection. ### Expected result You can observe online data analysis by opening the following web page: [http://localhost:5232](http://localhost:5232). Also, a dataset with images should be written in the `build/broker` directory.