mx/Jungfraujoch
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 38 Wiki Activity

Merge branch '2404-doc' into 'main'

Browse Source 
Updates to documentation

See merge request jungfraujoch/nextgendcu!50
This commit is contained in:
leonarski_f
2024-04-26 10:29:00 +02:00
parent 06feb8ea30 97b0d765f1
commit c8680dda9f
2 changed files with 44 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
README.md
View File

@@ -4,7 +4,9 @@ Application to receive data from the JUNGFRAU detector.
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).
The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG).
The project is supported by :
* Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
* ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)
## License
@@ -13,18 +15,21 @@ Sharing the code requires explicit permission from the Paul Scherrer Institute.
## Hardware requirements
1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs
3. 100G fiber-optic switch between JUNGFRAU and server
2. Server system with AMD/Xilinx Alveo U55C cards and optionally Nvidia GPUs
At the moment only U55C card is supported by Jungfraujoch.
Nvidia L4 GPU is recommended and used for performance tuning.
## FPGA bitstream
Instructions see [here](fpga/README.md)
## Software
Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux version 9, but version 8 is also supported.
Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux versions 8 and 9 (both are tested on a regular basis).
Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended,
but likely possible with providing multiple packages from external repositories.
There are some limited tests with recent Ubuntu and Fedora distributions, though these are not systematic.
Other linux platforms should work, but no tests were done so far.
### Dependencies
Required:
* C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC)
@@ -34,12 +39,20 @@ Required:
* JPEG library (turbo-jpeg is also OK)
Optional:
* CUDA compiler version 11 or newer - MX indexing won't work without it
* NUMA library
* CUDA compiler version 11 or newer - required for MX indexing and ML resolution estimation
* NUMA library - to pin threads to nodes/CPUs
* Node.js - to make frontend
* libtorch - for resolution estimation using model from Stanford - see below
provided as GIT submodules: SLS Detector Package, tinycbor (Intel), Zstandard (Facebook), Pistache webserver and fast replacement for bitshuffle filter (DECTRIS).
Provided as GIT submodules:
* SLS Detector Package - see [github.com/slsdetectorgroup/slsDetectorPackage](https://github.com/slsdetectorgroup/slsDetectorPackage)
* tinycbor (Intel) - see [github.com/intel/tinycbor](https://github.com/intel/tinycbor)
* Zstandard (Facebook) - see [github.com/facebook/zstd](https://github.com/facebook/zstd)
* Pistache webserver - see [github.com/pistacheio/pistache](https://github.com/pistacheio/pistache)
* Fast feedback indexer (Hans-Christian Stadler, PSI) - see [github.com/paulscherrerinstitute/fast-feedback-indexer](https://github.com/paulscherrerinstitute/fast-feedback-indexer)
* fast replacement for Bitshuffle pre-compression filter (Kal Cutter, DECTRIS) - see [github.com/kalcutter/bitshuffle](https://github.com/kalcutter/bitshuffle)
For license check LICENSE file in respective directory
Directly included in the repository:
* JSON parser/writer from N. Lohmann - see [github.com/nlohmann/json](https://github.com/nlohmann/json)
@@ -48,15 +61,16 @@ Directly included in the repository:
* Bitshuffle filter from K. Masui - see [github.com/kiyo-masui/bitshuffle](https://github.com/kiyo-masui/bitshuffle)
* LZ4 compression by Y.Collet - see [github.com/lz4/lz4](https://github.com/lz4/lz4)
* Spdlog logging library - see [github.com/gabime/spdlog](https://github.com/gabime/spdlog)
* ZeroMQ library (through slsDetectorPackage)
* ZeroMQ library (through slsDetectorPackage) - see [github.com/zeromq/libzmq](https://github.com/zeromq/libzmq)
For license check LICENSE file in respective directory
### Software components
* `jfjoch_broker` in `broker` - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition
* `jfjoch_writer` in `writer` - HDF5 writer
Configuration for the modules is given in configuration files present in `etc` directory.
* `jfjoch_broker` in `broker` - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition;
Example configuration `jfjoch_broker` for the modules is given in configuration files present in `etc` directory.
* `jfjoch_writer` in `writer` - HDF5 writer; HDF5 writer is designed to work on the same or separate server system. It has rather limited requirements in terms of performance and memory.
The goal is to separate data acquisition node with custom FPGA hardware and file system node with stronger security/stability requirements.
### Compilation
Use the following commands:
@@ -80,6 +94,15 @@ cd build
cmake -DJFJOCH_WRITER_ONLY=ON ..
make jfjoch
```
## Web Frontend
Jungfraujoch is equipped with React-based web frontend for user-friendly experience. Frontend has the following options:
* Presenting current state of the detector
* Plotting results of online quality calculations
* Showing live view images from the detector
* JUNGFRAU calibration numbers
* Configuring the detector, as well as pedestal/initialization operations
Frontend is written in TypeScript. For details see [frontend_ui/](frontend_ui) directory.
## Tests
@@ -87,8 +110,11 @@ Automated test routine is then accessible as `tests/CatchTest`. There are also b
* `CompressionBenchmark` to measure compression bandwidth (single threaded)
* `HDF5DatasetWriteTest` to measure HDF5 dataset writing speed (single threaded)
* `jfjoch_spot_finding_test` to apply spot finding and indexing routines in Jungfraujoch to an example dataset - this is equivalent to FPGA spot finding algorithm, but NOT performance equivalent as it is particularly not-efficient
* `jfjoch_action_test` to test quality/performance of FPGA card(s) and software routines
In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in `xds_durin` and `crystfel` folders. See `.gitlab-ci.yml` for details.
In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin, XDS Neggia plygin and CrystFEL.
Input files for these programs are placed in `xds_durin`, `xds_neggia` and `crystfel` folders. See `.gitlab-ci.yml` for details.
## Resolution estimation

13
frame_serialize/README.md
View File

@@ -8,7 +8,7 @@ There are minor differences at the moment:
* Only single channel is allowed for images (this only matters for deserialization)
* LZ4 compression is not allowed; Bitshuffle Zstandard is possible
* Few fields are currently absent
* Few extra fields are present beyond DECTRIS standard
* Extra fields are present beyond DECTRIS standard
* There is calibration message defined to transfer pedestal correction
## Start message
@@ -97,6 +97,9 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s
| data_collection_efficiency | float | Image collection efficiency \[\] | | |
| bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A [\photons\] | | |
| adu_histogram | Array(uint64) | ADU histogram | | |
| corr_beam_x_pxl | float | Difference between provided and calculated beam center X (pixels) - requires indexing | | X |
| corr_beam_y_pxl | float | Difference between provided and calculated beam center Y (pixels) - requires indexing | | X |
| corr_det_dist_mm | float | Difference between provided and calculated detector distance (mm) - requires indexing | | X |
| user_data | string | Optional user defined text information | | |
| roi_sum | int64 | Sum of ROI rectangle \[photons\] | | X |
| image | Map(string -> Image) | Image | | |
@@ -142,13 +145,5 @@ Pushes images using Stream2 serialization. This is using PUSH ZeroMQ sockets. Be
Same as above, but multiple ZeroMQ sockets are used and images are distributed on a round-robin basis between the sockets based on the image number.
Behavior for errors is the same as for `ZMQStream2Pusher`.
## ZMQStream2PreviewPublisher
Similar to sockets above, though with minor differences:
* PUB ZeroMQ socket is used (i.e., messages might be skipped by receiver)
* Conflate options is used to ensure that the socket does not save "old" images
* No start and end messages are sent
* Communication is fully nonblocking - no exceptions are raised in case of communication issues and `true` is set always returned.
## TestImagePusher
This interface is used for automated tests. It reads incoming images and compares one image with the reference data provided.