Jungfraujoch

Application to receive data from the JUNGFRAU detector.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.

The project is supported by :

• Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
• ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)

License

Operating Jungfraujoch requires license from the Paul Scherrer Institute. Sharing the code requires explicit permission from the Paul Scherrer Institute.

Hardware requirements

1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with AMD/Xilinx Alveo U55C cards and optionally Nvidia GPUs

At the moment only U55C card is supported by Jungfraujoch. Nvidia L4 GPU is recommended and used for performance tuning.

FPGA bitstream

Instructions see here

Software

Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux versions 8 and 9 (both are tested on a regular basis). Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended, but likely possible with providing multiple packages from external repositories. There are some limited tests with recent Ubuntu and Fedora distributions, though these are not systematic. Other linux platforms should work, but no tests were done so far.

Dependencies

Required:

• C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
• CMake version 3.21 or newer + GNU make tool
• HDF5 library version 1.10 or newer
• TIFF library (with C++ headers)
• JPEG library (turbo-jpeg is also OK)

Optional:

• CUDA compiler version 11 or newer - required for MX indexing and ML resolution estimation
• NUMA library - to pin threads to nodes/CPUs
• Node.js - to make frontend
• libtorch - for resolution estimation using model from Stanford - see below

Provided as GIT submodules:

• SLS Detector Package - see github.com/slsdetectorgroup/slsDetectorPackage
• tinycbor (Intel) - see github.com/intel/tinycbor
• Zstandard (Facebook) - see github.com/facebook/zstd
• Pistache webserver - see github.com/pistacheio/pistache
• Fast feedback indexer (Hans-Christian Stadler, PSI) - see github.com/paulscherrerinstitute/fast-feedback-indexer
• fast replacement for Bitshuffle pre-compression filter (Kal Cutter, DECTRIS) - see github.com/kalcutter/bitshuffle

For license check LICENSE file in respective directory

Directly included in the repository:

• JSON parser/writer from N. Lohmann - see github.com/nlohmann/json
• Catch2 testing library - see github.com/catchorg/Catch2
• Xilinx arbitrary precision arithmetic headers - see github.com/Xilinx/HLS_arbitrary_Precision_Types
• Bitshuffle filter from K. Masui - see github.com/kiyo-masui/bitshuffle
• LZ4 compression by Y.Collet - see github.com/lz4/lz4
• Spdlog logging library - see github.com/gabime/spdlog
• ZeroMQ library (through slsDetectorPackage) - see github.com/zeromq/libzmq

For license check LICENSE file in respective directory

Software components

• jfjoch_broker in broker - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition; Example configuration jfjoch_broker for the modules is given in configuration files present in etc directory.
• jfjoch_writer in writer - HDF5 writer; HDF5 writer is designed to work on the same or separate server system. It has rather limited requirements in terms of performance and memory. The goal is to separate data acquisition node with custom FPGA hardware and file system node with stronger security/stability requirements.

Compilation

Use the following commands:

git submodule update --init --recursive
mkdir build
cd build
cmake ..
make jfjoch

Compilation (writer only)

In most use cases it is better to have a separate machine, with access to distributed file system, for writing. Such machine needs only a HDF5 writer service with less dependencies. For compilation use the following commands:

git submodule update --init --recursive
mkdir build
cd build
cmake -DJFJOCH_WRITER_ONLY=ON ..
make jfjoch

Versions

FPGA release is as hexadecimal number indicted in the jfjoch_fpga.h as JFJOCH_FPGA_RELEASE constant. It is also included in the name of FPGA firmware file (.mcs) and kernel driver archive. This number indicated breaking changes in the FPGA firmware interface. FPGA release has to be consistent between FPGA firmware, kernel driver and jfjoch_broker - both kernel driver and software won't work in case of version mismatch. Commits to main branch with the same FPGA release version are OK to mix between components.

Web Frontend

Jungfraujoch is equipped with React-based web frontend for user-friendly experience. Frontend has the following options:

• Presenting current state of the detector
• Plotting results of online quality calculations
• Showing live view images from the detector
• JUNGFRAU calibration numbers
• Configuring the detector, as well as pedestal/initialization operations

Frontend is written in TypeScript. For details see frontend_ui/ directory.

Jungfraujoch Cmake scripts have an option to start frontend build with the following command:

git submodule update --init --recursive
mkdir build
cd build
cmake ..
make frontend

Contrary to standard CMake way, frontend will be built in "source" frontend_ui/build directory, not in build/ subdirectory.

Tests

Automated test routine is then accessible as tests/CatchTest. There are also benchmark routines:

• CompressionBenchmark to measure compression bandwidth (single threaded)
• HDF5DatasetWriteTest to measure HDF5 dataset writing speed (single threaded)
• jfjoch_spot_finding_test to apply spot finding and indexing routines in Jungfraujoch to an example dataset - this is equivalent to FPGA spot finding algorithm, but NOT performance equivalent as it is particularly not-efficient
• jfjoch_action_test to test quality/performance of FPGA card(s) and software routines

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin, XDS Neggia plygin and CrystFEL. Input files for these programs are placed in xds_durin, xds_neggia and crystfel folders. See .gitlab-ci.yml for details.

Resolution estimation

Resolution estimation can be done with a recent deep learning model by D. Mendez et al. (see Acta Cryst D, 80, 26-43), adapted to Jungfraujoch. Model used in the original paper is located in the resonet/ directory, after converting to TorchScript format.

To use the feature it is necessary to install libtorch library, preferably in /opt/libtorch location. The C++11 ABI version needs to be chosen. For RHEL 8 systems, please download older version 2.1.0, as version 2.2.0 requires newer glibc library than available with the operating system. Version 2.1.0 can be downloaded with the following command:

wget https://download.pytorch.org/libtorch/cu121/libtorch-cxx11-abi-shared-with-deps-2.1.0%2Bcu121.zip