# Jungfraujoch

Application to receive data from the JUNGFRAU detector at Swiss Light Source macromolecular crystallography beamlines.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).

## License

Operating Jungfraujoch, as well as sharing sources code requires explicit license from PSI.

## Hardware requirements
1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
3. 100G fiber-optic switch between JUNGFRAU and server

## FPGA bitstream
### Dependencies
To build FPGA image you need:
1. Xilinx Vivado 2022.1 or newer
2. License for Ultrascale+ 100G core - license is available at no cost from Xilinx, but needs to be seperatly requested, see [Xilinx website](https://www.xilinx.com/products/intellectual-property/cmac_usplus.html).

Instructions see [here](receiver/README.md)

## Software

### Dependencies
Required:
* C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
* CMake version 3.21 or newer + GNU make tool 
* HDF5 library version 1.10 or newer
* ZeroMQ library
* Google Remote Procedure Call (gRPC) - see notes below

Optional:
* CUDA compiler version 11 or newer - image analysis features won't work without it
* Mellanox OFED - Infinibands Verbs 
* NUMA library
* Node.js - to make frontend

provided as GIT submodules: SLS Detector Package, tinycbor (Intel) and Zstandard (Facebook). 

Directly included in the repository:
* JSON parser/writer from N. Lohmann - see [github.com/nlohmann/json](https://github.com/nlohmann/json)
* Catch2 testing library - see [github.com/catchorg/Catch2](https://github.com/catchorg/Catch2)
* Xilinx arbitrary precision arithmetic headers - see [github.com/Xilinx/HLS_arbitrary_Precision_Types](https://github.com/Xilinx/HLS_arbitrary_Precision_Types)
* Bitshuffle filter from K. Masui - see [github.com/kiyo-masui/bitshuffle](https://github.com/kiyo-masui/bitshuffle)
* LZ4 compression by Y.Collet - see [github.com/lz4/lz4](https://github.com/lz4/lz4)
* Spdlog logging library - see [github.com/gabime/spdlog](https://github.com/gabime/spdlog)

For license check LICENSE file in respective directory

### Software components

* `jfjoch_receiver` in `fpga/host` - the main component of Jungfraujoch, used to receive data with FPGA smart network cards, compress images and send them over ZeroMQ
* `jfjoch_broker` in `broker` - gRPC based broker, controlling all Jungfraujoch components
* `jfjoch_writer` in `writer` - HDF5 writer
* `jfjoch_detector` in `detector` - wrapper over detector control

Configuration for the modules is given in configuration files present in `etc` directory.

### Installation of gRPC
By default, gRPC will be cloned and compiled automatically, when compiling Jungfraujoch with `cmake`, however this significantly increases installation time.
Alternative is to compile gRPC beforehand. This is very useful especially, when one expects to compile Jungfraujoch multiple times on the same machine.
It is recommended to install gRPC in own directory (not in system path), as there is no easy way to uninstall it.
To compile gRPC:
```
git clone https://github.com/grpc/grpc
cd grpc
git checkout v1.53.0
git submodule update --init
mkdir build
cd build
cmake -DgRPC_ZLIB_PROVIDER="package" -DCMAKE_INSTALL_PREFIX=/opt/grpc ..
make
sudo make install
```
Currently, handling paths for gRPC and its dependencies gives sometimes troubles and few environmental variables need to be setup before compilation (assuming gRPC installed in /opt/grpc):
```
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/opt/grpc/lib/cmake:/opt/grpc/lib64/cmake
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/grpc/lib:/opt/grpc/lib64
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/grpc/lib/pkgconfig:/opt/grpc/lib64/pkgconfig
```

### Compilation
Use the following commands (use `cmake` instead of `cmake3` in non-RHEL systems):

```
git submodule update --init --recursive
mkdir build
cd build
cmake3 ..
make jfjoch
```

## Tests

To enable compiling test routines use parameter `-DJFJOCH_COMPILE_TESTS=ON` for `cmake`. 
Automated test routine is then accessible as `tests/CatchTest`. There are also benchmark routines:

* `CompressionBenchmark` to measure compression bandwidth (single threaded)
* `HDF5DatasetWriteTest` to measure HDF5 dataset writing speed (single threaded)
* `DataAnalysisPerfTest` to measure data analysis performance (single threaded)
* `JFCalibrationPerfTest` to measure pedestal calculation and online conversion performance

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in `xds_durin` and `crystfel` folders. See `.gitlab-ci.yml` for details.

## Web user interface

### Building
To build web interface:
```
cd frontend_ui
npm install
npm build
```