Jungfraujoch/README.md

# Jungfraujoch

Application to receive data from the JUNGFRAU detector.

Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).

The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG).

## License

Operating Jungfraujoch requires license from the Paul Scherrer Institute.
Sharing the code requires explicit permission from the Paul Scherrer Institute.

## Hardware requirements
1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
3. 100G fiber-optic switch between JUNGFRAU and server

## FPGA bitstream
Instructions see [here](fpga/README.md)

## Software
Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux version 9, but version 8 is also supported.
Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended, 
but likely possible with providing multiple packages from external repositories. 
Other linux platforms should work, but no tests were done so far.

### Dependencies
Required:
* C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
* CMake version 3.21 or newer + GNU make tool 
* HDF5 library version 1.10 or newer
* ZeroMQ library
* Google Remote Procedure Call (gRPC) - see notes below

Optional:
* CUDA compiler version 11 or newer - image analysis features won't work without it
* NUMA library
* Node.js - to make frontend

provided as GIT submodules: SLS Detector Package, tinycbor (Intel), Zstandard (Facebook) and fast replacement for bitshuffle filter (DECTRIS). 

Directly included in the repository:
* JSON parser/writer from N. Lohmann - see [github.com/nlohmann/json](https://github.com/nlohmann/json)
* Catch2 testing library - see [github.com/catchorg/Catch2](https://github.com/catchorg/Catch2)
* Xilinx arbitrary precision arithmetic headers - see [github.com/Xilinx/HLS_arbitrary_Precision_Types](https://github.com/Xilinx/HLS_arbitrary_Precision_Types)
* Bitshuffle filter from K. Masui - see [github.com/kiyo-masui/bitshuffle](https://github.com/kiyo-masui/bitshuffle)
* LZ4 compression by Y.Collet - see [github.com/lz4/lz4](https://github.com/lz4/lz4)
* Spdlog logging library - see [github.com/gabime/spdlog](https://github.com/gabime/spdlog)

For license check LICENSE file in respective directory

### Software components

* `jfjoch_broker` in `broker` - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition
* `jfjoch_writer` in `writer` - HDF5 writer

Configuration for the modules is given in configuration files present in `etc` directory.

### Installation of gRPC
#### RHEL/Rocky 9
Please use Protobuf and gRPC from the Extra Packages for Enterprise Linux (EPEL) repository. 
To enable EPEL, follow instructions on [the official website](https://docs.fedoraproject.org/en-US/epel/).

#### RHEL 8
gRPC need to be compiled before, as there is no official/semi-official repository for RHEL8. 
This also applies to Protocol Buffer, which is provided in RHEL8, but version is old and incompatible with functionality needed in Jungfraujoch.
It is recommended to install gRPC in own directory (not in system path), as there is no easy way to uninstall it.
To compile gRPC:
```
git clone https://github.com/grpc/grpc
cd grpc
git checkout v1.53.0 # This can be usually substituted by the current stable release
git submodule update --init
mkdir build
cd build
cmake -DgRPC_ZLIB_PROVIDER="package" -DCMAKE_INSTALL_PREFIX=/opt/grpc ..
make
sudo make install
```
Currently, handling paths for gRPC and its dependencies gives sometimes troubles and few environmental variables need to be setup before compilation (assuming gRPC installed in /opt/grpc):
```
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/opt/grpc/lib/cmake:/opt/grpc/lib64/cmake
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/grpc/lib:/opt/grpc/lib64
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/grpc/lib/pkgconfig:/opt/grpc/lib64/pkgconfig
```

### Compilation
Use the following commands:

```
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make jfjoch
```

### Compilation (writer only)
In most use cases it is better to have a separate machine, with access to distributed file system, for writing.
Such machine needs only a HDF5 writer service with less dependencies. For compilation use the following commands:

```
git submodule update --init --recursive
mkdir build
cd build
cmake -DJFJOCH_WRITER_ONLY=ON ..
make jfjoch
```

## Tests

To enable compiling test routines use parameter `-DJFJOCH_COMPILE_TESTS=ON` for `cmake`. 
Automated test routine is then accessible as `tests/CatchTest`. There are also benchmark routines:

* `CompressionBenchmark` to measure compression bandwidth (single threaded)
* `HDF5DatasetWriteTest` to measure HDF5 dataset writing speed (single threaded)
* `DataAnalysisPerfTest` to measure data analysis performance (single threaded)
* `JFCalibrationPerfTest` to measure pedestal calculation and online conversion performance

In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in `xds_durin` and `crystfel` folders. See `.gitlab-ci.yml` for details.

## Web user interface

### Building
To build web interface:
```
cd frontend_ui
npm install
npm run build
```