Jungfraujoch

Application to receive data from the JUNGFRAU detector at Swiss Light Source macromolecular crystallography beamlines.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.

License

Software components are licensed with GNU Public License version 3.

Hardware components are licensed with Strongly-reciprocal CERN Open Hardware Licence version 2.

Hardware requirements

1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
3. 100G fiber-optic switch between JUNGFRAU and server

FPGA bitstream

Dependencies

To build FPGA image you need:

1. Xilinx Vivado 2022.1 or newer
2. License for Ultrascale+ 100G core - license is available at no cost from Xilinx, but needs to be seperatly requested, see Xilinx website.

Instructions see here

Software

Dependencies

1. C++20 compiler and C++20 standard library (NOT provided by default RHEL 7 installation, need to install Developer Tools, tested with devtools-11)
2. CMake version 3.21 or newer + GNU make tool
3. HDF5 library version 1.10 or newer
4. ZeroMQ library
5. Google Remote Procedure Call (gRPC) - see notes below
6. CUDA compiler version 11 or newer (compulsory for receiver/tests)
7. Mellanox OFED - Infinibands Verbs (optional)
8. NUMA library (optional)
9. Node.js (optional) - to make frontend

Additional dependencies: SLS Detector Package, tinycbor (Intel) and Zstandard (Facebook) are provided as GIT submodules.

Directly included in the repository:

• JSON parser/writer from N. Lohmann - see github.com/nlohmann/json
• Catch2 testing library - see github.com/catchorg/Catch2
• Xilinx arbitrary precision arithmetic headers - see github.com/Xilinx/HLS_arbitrary_Precision_Types
• Bitshuffle filter from K. Masui - see github.com/kiyo-masui/bitshuffle
• LZ4 compression by Y.Collet - see github.com/lz4/lz4
• Spdlog logging library - see github.com/gabime/spdlog

For license check LICENSE file in respective directory

Software components

• jfjoch_receiver in fpga/host - the main component of Jungfraujoch, used to receive data with FPGA smart network cards, compress images and send them over ZeroMQ
• jfjoch_broker in broker - gRPC based broker, controlling all Jungfraujoch components
• jfjoch_writer in writer - HDF5 writer
• jfjoch_detector in detector - wrapper over detector control

Configuration for the modules is given in configuration files present in etc directory.

Installation of gRPC

By default, gRPC will be cloned and compiled automatically, when compiling Jungfraujoch with cmake, however this significantly increases installation time. Alternative is to compile gRPC beforehand. This is very useful especially, when one expects to compile Jungfraujoch multiple times on the same machine. It is recommended to install gRPC in own directory (not in system path), as there is no easy way to uninstall it. To compile gRPC:

git clone https://github.com/grpc/grpc
cd grpc
git checkout v1.53.0
git submodule update --init
mkdir build
cd build
cmake -DgRPC_ZLIB_PROVIDER="package" -DCMAKE_INSTALL_PREFIX=/opt/grpc ..
make
sudo make install

Currently, handling paths for gRPC and its dependencies gives sometimes troubles and few environmental variables need to be setup before compilation (assuming gRPC installed in /opt/grpc):

export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/opt/grpc/lib/cmake:/opt/grpc/lib64/cmake
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/grpc/lib:/opt/grpc/lib64
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/grpc/lib/pkgconfig:/opt/grpc/lib64/pkgconfig

Compilation

Use the following commands (use cmake instead of cmake3 in non-RHEL systems):

git submodule update --init --recursive
mkdir build
cd build
cmake3 ..
make jfjoch

Tests

To enable compiling test routines use parameter -DJFJOCH_COMPILE_TESTS=ON for cmake. Automated test routine is then accessible as tests/CatchTest. There are also benchmark routines:

• CompressionBenchmark to measure compression bandwidth (single threaded)
• HDF5DatasetWriteTest to measure HDF5 dataset writing speed (single threaded)
• DataAnalysisPerfTest to measure data analysis performance (single threaded)
• JFCalibrationPerfTest to measure pedestal calculation and online conversion performance

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in xds_durin and crystfel folders. See .gitlab-ci.yml for details.

Web user interface

Building

To build web interface:

cd frontend_ui
npm install
npm build