leonarski_f
d315506633
* Enhancements for EIGER * Writer is more flexible and capable of handling DECTRIS data
105 lines
4.9 KiB
Markdown
105 lines
4.9 KiB
Markdown
# Jungfraujoch
|
|
|
|
Application to receive data from the JUNGFRAU detector.
|
|
|
|
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).
|
|
|
|
The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG).
|
|
|
|
## License
|
|
|
|
Operating Jungfraujoch requires license from the Paul Scherrer Institute.
|
|
Sharing the code requires explicit permission from the Paul Scherrer Institute.
|
|
|
|
## Hardware requirements
|
|
1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
|
|
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
|
|
3. 100G fiber-optic switch between JUNGFRAU and server
|
|
|
|
## FPGA bitstream
|
|
Instructions see [here](fpga/README.md)
|
|
|
|
## Software
|
|
Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux version 9, but version 8 is also supported.
|
|
Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended,
|
|
but likely possible with providing multiple packages from external repositories.
|
|
Other linux platforms should work, but no tests were done so far.
|
|
|
|
### Dependencies
|
|
Required:
|
|
* C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
|
|
* CMake version 3.21 or newer + GNU make tool
|
|
* HDF5 library version 1.10 or newer
|
|
* TIFF library (with C++ headers)
|
|
* JPEG library (turbo-jpeg is also OK)
|
|
|
|
Optional:
|
|
* CUDA compiler version 11 or newer - MX indexing won't work without it
|
|
* NUMA library
|
|
* Node.js - to make frontend
|
|
* libtorch - for resolution estimation using model from Stanford - see below
|
|
|
|
provided as GIT submodules: SLS Detector Package, tinycbor (Intel), Zstandard (Facebook), Pistache webserver and fast replacement for bitshuffle filter (DECTRIS).
|
|
|
|
Directly included in the repository:
|
|
* JSON parser/writer from N. Lohmann - see [github.com/nlohmann/json](https://github.com/nlohmann/json)
|
|
* Catch2 testing library - see [github.com/catchorg/Catch2](https://github.com/catchorg/Catch2)
|
|
* Xilinx arbitrary precision arithmetic headers - see [github.com/Xilinx/HLS_arbitrary_Precision_Types](https://github.com/Xilinx/HLS_arbitrary_Precision_Types)
|
|
* Bitshuffle filter from K. Masui - see [github.com/kiyo-masui/bitshuffle](https://github.com/kiyo-masui/bitshuffle)
|
|
* LZ4 compression by Y.Collet - see [github.com/lz4/lz4](https://github.com/lz4/lz4)
|
|
* Spdlog logging library - see [github.com/gabime/spdlog](https://github.com/gabime/spdlog)
|
|
* ZeroMQ library (through slsDetectorPackage)
|
|
For license check LICENSE file in respective directory
|
|
|
|
### Software components
|
|
|
|
* `jfjoch_broker` in `broker` - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition
|
|
* `jfjoch_writer` in `writer` - HDF5 writer
|
|
|
|
Configuration for the modules is given in configuration files present in `etc` directory.
|
|
|
|
### Compilation
|
|
Use the following commands:
|
|
|
|
```
|
|
git submodule update --init --recursive
|
|
mkdir build
|
|
cd build
|
|
cmake ..
|
|
make jfjoch
|
|
```
|
|
|
|
### Compilation (writer only)
|
|
In most use cases it is better to have a separate machine, with access to distributed file system, for writing.
|
|
Such machine needs only a HDF5 writer service with less dependencies. For compilation use the following commands:
|
|
|
|
```
|
|
git submodule update --init --recursive
|
|
mkdir build
|
|
cd build
|
|
cmake -DJFJOCH_WRITER_ONLY=ON ..
|
|
make jfjoch
|
|
```
|
|
|
|
## Tests
|
|
|
|
Automated test routine is then accessible as `tests/CatchTest`. There are also benchmark routines:
|
|
|
|
* `CompressionBenchmark` to measure compression bandwidth (single threaded)
|
|
* `HDF5DatasetWriteTest` to measure HDF5 dataset writing speed (single threaded)
|
|
|
|
In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in `xds_durin` and `crystfel` folders. See `.gitlab-ci.yml` for details.
|
|
|
|
## Resolution estimation
|
|
|
|
Resolution estimation can be done with a recent deep learning model by D. Mendez et al.
|
|
(see [Acta Cryst D, 80, 26-43](https://doi.org/10.1107/S2059798323010586)), adapted to Jungfraujoch.
|
|
Model used in the original paper is located in the [resonet/](resonet) directory, after converting to TorchScript format.
|
|
|
|
To use the feature it is necessary to install [libtorch](https://pytorch.org/) library, preferably in `/opt/libtorch` location. The C++11 ABI version needs to be chosen.
|
|
For RHEL 8 systems, please download older version 2.1.0, as version 2.2.0 requires newer `glibc` library than available with the operating system.
|
|
Version 2.1.0 can be downloaded with the following command:
|
|
```
|
|
wget https://download.pytorch.org/libtorch/cu121/libtorch-cxx11-abi-shared-with-deps-2.1.0%2Bcu121.zip
|
|
```
|