Jungfraujoch

Application to receive data from the JUNGFRAU detector.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.

The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG).

License

Operating Jungfraujoch requires license from the Paul Scherrer Institute. Sharing the code requires explicit permission from the Paul Scherrer Institute.

Hardware requirements

1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
3. 100G fiber-optic switch between JUNGFRAU and server

FPGA bitstream

Instructions see here

Software

Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux version 9, but version 8 is also supported. Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended, but likely possible with providing multiple packages from external repositories. Other linux platforms should work, but no tests were done so far.

Dependencies

Required:

• C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
• CMake version 3.21 or newer + GNU make tool
• HDF5 library version 1.10 or newer
• ZeroMQ library
• Google Remote Procedure Call (gRPC) - see notes below

Optional:

• CUDA compiler version 11 or newer - image analysis features won't work without it
• NUMA library
• Node.js - to make frontend

provided as GIT submodules: SLS Detector Package, tinycbor (Intel), Zstandard (Facebook) and fast replacement for bitshuffle filter (DECTRIS).

Directly included in the repository:

• JSON parser/writer from N. Lohmann - see github.com/nlohmann/json
• Catch2 testing library - see github.com/catchorg/Catch2
• Xilinx arbitrary precision arithmetic headers - see github.com/Xilinx/HLS_arbitrary_Precision_Types
• Bitshuffle filter from K. Masui - see github.com/kiyo-masui/bitshuffle
• LZ4 compression by Y.Collet - see github.com/lz4/lz4
• Spdlog logging library - see github.com/gabime/spdlog

For license check LICENSE file in respective directory

Software components

• jfjoch_receiver in fpga/host - the main component of Jungfraujoch, used to receive data with FPGA smart network cards, compress images and send them over ZeroMQ
• jfjoch_broker in broker - gRPC based broker, controlling all Jungfraujoch components
• jfjoch_writer in writer - HDF5 writer
• jfjoch_detector in detector - wrapper over detector control

Configuration for the modules is given in configuration files present in etc directory.

Installation of gRPC

RHEL/Rocky 9

Please use Protobuf and gRPC from the Extra Packages for Enterprise Linux (EPEL) repository. To enable EPEL, follow instructions on the official website.

RHEL 8

gRPC need to be compiled before, as there is no official/semi-official repository for RHEL8. This also applies to Protocol Buffer, which is provided in RHEL8, but version is old and incompatible with functionality needed in Jungfraujoch. It is recommended to install gRPC in own directory (not in system path), as there is no easy way to uninstall it. To compile gRPC:

git clone https://github.com/grpc/grpc
cd grpc
git checkout v1.53.0 # This can be usually substituted by the current stable release
git submodule update --init
mkdir build
cd build
cmake -DgRPC_ZLIB_PROVIDER="package" -DCMAKE_INSTALL_PREFIX=/opt/grpc ..
make
sudo make install

Currently, handling paths for gRPC and its dependencies gives sometimes troubles and few environmental variables need to be setup before compilation (assuming gRPC installed in /opt/grpc):

export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/opt/grpc/lib/cmake:/opt/grpc/lib64/cmake
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/grpc/lib:/opt/grpc/lib64
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/grpc/lib/pkgconfig:/opt/grpc/lib64/pkgconfig

Compilation

Use the following commands:

git submodule update --init --recursive
mkdir build
cd build
cmake ..
make jfjoch

Tests

To enable compiling test routines use parameter -DJFJOCH_COMPILE_TESTS=ON for cmake. Automated test routine is then accessible as tests/CatchTest. There are also benchmark routines:

• CompressionBenchmark to measure compression bandwidth (single threaded)
• HDF5DatasetWriteTest to measure HDF5 dataset writing speed (single threaded)
• DataAnalysisPerfTest to measure data analysis performance (single threaded)
• JFCalibrationPerfTest to measure pedestal calculation and online conversion performance

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in xds_durin and crystfel folders. See .gitlab-ci.yml for details.

Web user interface

Building

To build web interface:

cd frontend_ui
npm install
npm run build