Jungfraujoch

Application to receive data from the JUNGFRAU detector.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.

The project is supported by :

• Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
• ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)
• AMD University Program with donation of licenses of Ethernet IP cores and Vivado software

License

Operating Jungfraujoch requires license from the Paul Scherrer Institute. Sharing the code requires explicit permission from the Paul Scherrer Institute.

Hardware requirements

1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with AMD/Xilinx Alveo U55C cards and optionally Nvidia GPUs

At the moment only U55C card is supported by Jungfraujoch. Nvidia L4 GPU is recommended and used for performance tuning.

FPGA bitstream

Instructions see here

Software

Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux versions 8 and 9 (both are tested on a regular basis). Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended, but likely possible with providing multiple packages from external repositories. There are some limited tests with recent Ubuntu and Fedora distributions, though these are not systematic. Other linux platforms should work, but no tests were done so far.

Dependencies

Required:

• C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
• CMake version 3.21 or newer + GNU make tool
• JPEG library (turbo-jpeg is also OK)

Optional:

• CUDA compiler version 11 or newer - required for MX fast feedback indexer
• NUMA library - to pin threads to nodes/CPUs
• Node.js - to make frontend

Automatically downloaded by CMake and statically linked:

• SLS Detector Package - see github.com/slsdetectorgroup/slsDetectorPackage
• Zstandard (Facebook) - see github.com/facebook/zstd
• Pistache webserver - see github.com/pistacheio/pistache
• Fast feedback indexer (Hans-Christian Stadler, PSI) - see github.com/paulscherrerinstitute/fast-feedback-indexer
• Catch2 testing library - see github.com/catchorg/Catch2
• HDF5 library - see github.com/HDFGroup/hdf5
• TIFF library - see gitlab.com/libtiff/libtiff

Please follow the link provided above to check for LICENSE file. Building code with dependencies above requires access from the build system to github.com.

Directly included in the repository:

• JSON parser/writer from N. Lohmann - see github.com/nlohmann/json
• Xilinx arbitrary precision arithmetic headers - see github.com/Xilinx/HLS_arbitrary_Precision_Types
• Bitshuffle filter from K. Masui - see github.com/kiyo-masui/bitshuffle
• Fast replacement for Bitshuffle pre-compression filter (Kal Cutter, DECTRIS) - see github.com/kalcutter/bitshuffle
• Tinycbor (Intel) - see github.com/intel/tinycbor
• LZ4 compression by Y.Collet - see github.com/lz4/lz4
• Spdlog logging library - see github.com/gabime/spdlog
• ZeroMQ library (through slsDetectorPackage) - see github.com/zeromq/libzmq

For license check LICENSE file in respective directory

Software components

• jfjoch_broker in broker - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition; Example configuration jfjoch_broker for the modules is given in configuration files present in etc directory. See details.
• jfjoch_writer in writer - HDF5 writer; HDF5 writer is designed to work on the same or separate server system. It has rather limited requirements in terms of performance and memory. The goal is to separate data acquisition node with custom FPGA hardware and file system node with stronger security/stability requirements. See details.

Compilation

Use the following commands:

mkdir build
cd build
cmake ..
make jfjoch

Compilation (writer only)

In most use cases it is better to have a separate machine, with access to distributed file system, for writing. Such machine needs only a HDF5 writer service with fewer dependencies. For compilation use the following commands:

mkdir build
cd build
cmake -DJFJOCH_WRITER_ONLY=ON ..
make jfjoch

Versions

FPGA release is as hexadecimal number indicted in the jfjoch_fpga.h as JFJOCH_FPGA_RELEASE constant. This number indicated breaking changes in the FPGA firmware interface. FPGA release has to be consistent between FPGA firmware, kernel driver and jfjoch_broker - both kernel driver and software won't work in case of version mismatch. Commits to main branch with the same FPGA release version are OK to mix between components.

Web Frontend

Jungfraujoch is equipped with React-based web frontend for user-friendly experience. Frontend has the following options:

• Presenting current state of the detector
• Plotting results of online quality calculations
• Showing live view images from the detector
• JUNGFRAU calibration numbers
• Configuring the detector, as well as pedestal/initialization operations

Frontend is written in TypeScript. For details see frontend_ui/ directory.

Jungfraujoch Cmake scripts have an option to start frontend build with the following command:

mkdir build
cd build
cmake ..
make frontend

Contrary to standard CMake way, frontend will be built in "source" frontend_ui/build directory, not in build/ subdirectory.

Tests

Automated test routine is then accessible as tests/jfjoch_test. There are also benchmark routines:

• HDF5DatasetWriteTest to measure HDF5 dataset writing speed (single threaded)
• jfjoch_spot_finding_test to apply spot finding and indexing routines in Jungfraujoch to an example dataset - this is equivalent to FPGA spot finding algorithm, but NOT performance equivalent as it is particularly not-efficient
• jfjoch_action_test to test quality/performance of FPGA card(s) and software routines

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin, XDS Neggia plygin and CrystFEL. Input files for these programs are placed in xds_durin, xds_neggia and crystfel folders. See .gitlab-ci.yml for details.