Jungfraujoch

Application to receive data from the JUNGFRAU detector.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.

The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG).

License

Operating Jungfraujoch requires license from the Paul Scherrer Institute. Sharing the code requires explicit permission from the Paul Scherrer Institute.

Hardware requirements

1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
3. 100G fiber-optic switch between JUNGFRAU and server

FPGA bitstream

Instructions see here

Software

Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux version 9, but version 8 is also supported. Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended, but likely possible with providing multiple packages from external repositories. Other linux platforms should work, but no tests were done so far.

Dependencies

Required:

• C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
• CMake version 3.21 or newer + GNU make tool
• HDF5 library version 1.10 or newer
• TIFF library (with C++ headers)
• JPEG library (turbo-jpeg is also OK)

Optional:

• CUDA compiler version 11 or newer - MX indexing won't work without it
• NUMA library
• Node.js - to make frontend
• libtorch - for resolution estimation using model from Stanford - see below

provided as GIT submodules: SLS Detector Package, tinycbor (Intel), Zstandard (Facebook), Pistache webserver and fast replacement for bitshuffle filter (DECTRIS).

Directly included in the repository:

• JSON parser/writer from N. Lohmann - see github.com/nlohmann/json
• Catch2 testing library - see github.com/catchorg/Catch2
• Xilinx arbitrary precision arithmetic headers - see github.com/Xilinx/HLS_arbitrary_Precision_Types
• Bitshuffle filter from K. Masui - see github.com/kiyo-masui/bitshuffle
• LZ4 compression by Y.Collet - see github.com/lz4/lz4
• Spdlog logging library - see github.com/gabime/spdlog
• ZeroMQ library (through slsDetectorPackage) For license check LICENSE file in respective directory

Software components

• jfjoch_broker in broker - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition
• jfjoch_writer in writer - HDF5 writer

Configuration for the modules is given in configuration files present in etc directory.

Compilation

Use the following commands:

git submodule update --init --recursive
mkdir build
cd build
cmake ..
make jfjoch

Compilation (writer only)

In most use cases it is better to have a separate machine, with access to distributed file system, for writing. Such machine needs only a HDF5 writer service with less dependencies. For compilation use the following commands:

git submodule update --init --recursive
mkdir build
cd build
cmake -DJFJOCH_WRITER_ONLY=ON ..
make jfjoch

Tests

Automated test routine is then accessible as tests/CatchTest. There are also benchmark routines:

• CompressionBenchmark to measure compression bandwidth (single threaded)
• HDF5DatasetWriteTest to measure HDF5 dataset writing speed (single threaded)

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in xds_durin and crystfel folders. See .gitlab-ci.yml for details.

Resolution estimation

Resolution estimation can be done with a recent deep learning model by D. Mendez et al. (see Acta Cryst D, 80, 26-43), adapted to Jungfraujoch. Model used in the original paper is located in the resonet/ directory, after converting to TorchScript format.

To use the feature it is necessary to install libtorch library, preferably in /opt/libtorch location. The C++11 ABI version needs to be chosen. For RHEL 8 systems, please download older version 2.1.0, as version 2.2.0 requires newer glibc library than available with the operating system. Version 2.1.0 can be downloaded with the following command:

wget https://download.pytorch.org/libtorch/cu121/libtorch-cxx11-abi-shared-with-deps-2.1.0%2Bcu121.zip