117 lines
5.1 KiB
Markdown
117 lines
5.1 KiB
Markdown
# Jungfraujoch
|
|
|
|
Application to receive data from the JUNGFRAU detector at Swiss Light Source macromolecular crystallography beamlines.
|
|
|
|
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).
|
|
|
|
## License
|
|
|
|
Operating Jungfraujoch, as well as sharing sources code requires explicit license from PSI.
|
|
|
|
## Hardware requirements
|
|
1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
|
|
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
|
|
3. 100G fiber-optic switch between JUNGFRAU and server
|
|
|
|
## FPGA bitstream
|
|
### Dependencies
|
|
To build FPGA image you need:
|
|
1. Xilinx Vivado 2022.1 or newer
|
|
2. License for Ultrascale+ 100G core - license is available at no cost from Xilinx, but needs to be seperatly requested, see [Xilinx website](https://www.xilinx.com/products/intellectual-property/cmac_usplus.html).
|
|
|
|
Instructions see [here](receiver/README.md)
|
|
|
|
## Software
|
|
|
|
### Dependencies
|
|
Required:
|
|
* C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
|
|
* CMake version 3.21 or newer + GNU make tool
|
|
* HDF5 library version 1.10 or newer
|
|
* ZeroMQ library
|
|
* Google Remote Procedure Call (gRPC) - see notes below
|
|
|
|
Optional:
|
|
* CUDA compiler version 11 or newer - image analysis features won't work without it
|
|
* Mellanox OFED - Infinibands Verbs
|
|
* NUMA library
|
|
* Node.js - to make frontend
|
|
|
|
provided as GIT submodules: SLS Detector Package, tinycbor (Intel) and Zstandard (Facebook).
|
|
|
|
Directly included in the repository:
|
|
* JSON parser/writer from N. Lohmann - see [github.com/nlohmann/json](https://github.com/nlohmann/json)
|
|
* Catch2 testing library - see [github.com/catchorg/Catch2](https://github.com/catchorg/Catch2)
|
|
* Xilinx arbitrary precision arithmetic headers - see [github.com/Xilinx/HLS_arbitrary_Precision_Types](https://github.com/Xilinx/HLS_arbitrary_Precision_Types)
|
|
* Bitshuffle filter from K. Masui - see [github.com/kiyo-masui/bitshuffle](https://github.com/kiyo-masui/bitshuffle)
|
|
* LZ4 compression by Y.Collet - see [github.com/lz4/lz4](https://github.com/lz4/lz4)
|
|
* Spdlog logging library - see [github.com/gabime/spdlog](https://github.com/gabime/spdlog)
|
|
|
|
For license check LICENSE file in respective directory
|
|
|
|
### Software components
|
|
|
|
* `jfjoch_receiver` in `fpga/host` - the main component of Jungfraujoch, used to receive data with FPGA smart network cards, compress images and send them over ZeroMQ
|
|
* `jfjoch_broker` in `broker` - gRPC based broker, controlling all Jungfraujoch components
|
|
* `jfjoch_writer` in `writer` - HDF5 writer
|
|
* `jfjoch_detector` in `detector` - wrapper over detector control
|
|
|
|
Configuration for the modules is given in configuration files present in `etc` directory.
|
|
|
|
### Installation of gRPC
|
|
By default, gRPC will be cloned and compiled automatically, when compiling Jungfraujoch with `cmake`, however this significantly increases installation time.
|
|
Alternative is to compile gRPC beforehand. This is very useful especially, when one expects to compile Jungfraujoch multiple times on the same machine.
|
|
It is recommended to install gRPC in own directory (not in system path), as there is no easy way to uninstall it.
|
|
To compile gRPC:
|
|
```
|
|
git clone https://github.com/grpc/grpc
|
|
cd grpc
|
|
git checkout v1.53.0
|
|
git submodule update --init
|
|
mkdir build
|
|
cd build
|
|
cmake -DgRPC_ZLIB_PROVIDER="package" -DCMAKE_INSTALL_PREFIX=/opt/grpc ..
|
|
make
|
|
sudo make install
|
|
```
|
|
Currently, handling paths for gRPC and its dependencies gives sometimes troubles and few environmental variables need to be setup before compilation (assuming gRPC installed in /opt/grpc):
|
|
```
|
|
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/opt/grpc/lib/cmake:/opt/grpc/lib64/cmake
|
|
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/grpc/lib:/opt/grpc/lib64
|
|
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/grpc/lib/pkgconfig:/opt/grpc/lib64/pkgconfig
|
|
```
|
|
|
|
### Compilation
|
|
Use the following commands (use `cmake` instead of `cmake3` in non-RHEL systems):
|
|
|
|
```
|
|
git submodule update --init --recursive
|
|
mkdir build
|
|
cd build
|
|
cmake3 ..
|
|
make jfjoch
|
|
```
|
|
|
|
## Tests
|
|
|
|
To enable compiling test routines use parameter `-DJFJOCH_COMPILE_TESTS=ON` for `cmake`.
|
|
Automated test routine is then accessible as `tests/CatchTest`. There are also benchmark routines:
|
|
|
|
* `CompressionBenchmark` to measure compression bandwidth (single threaded)
|
|
* `HDF5DatasetWriteTest` to measure HDF5 dataset writing speed (single threaded)
|
|
* `DataAnalysisPerfTest` to measure data analysis performance (single threaded)
|
|
* `JFCalibrationPerfTest` to measure pedestal calculation and online conversion performance
|
|
|
|
In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in `xds_durin` and `crystfel` folders. See `.gitlab-ci.yml` for details.
|
|
|
|
## Web user interface
|
|
|
|
### Building
|
|
To build web interface:
|
|
```
|
|
cd frontend_ui
|
|
npm install
|
|
npm build
|
|
```
|
|
|