From 97b0d765f1cb1d7eac5e4148d32417eae5eaf6c1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 26 Apr 2024 10:28:31 +0200 Subject: [PATCH] Updates to documentation --- README.md | 54 +++++++++++++++++++++++++++++---------- frame_serialize/README.md | 13 +++------- 2 files changed, 44 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 38ef9f66..70094ad8 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,9 @@ Application to receive data from the JUNGFRAU detector. Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268). -The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG). +The project is supported by : +* Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025). +* ETH Domain via Open Research Data Contribute project (Jan - Dec 2023) ## License @@ -13,18 +15,21 @@ Sharing the code requires explicit permission from the Paul Scherrer Institute. ## Hardware requirements 1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards) -2. Server system with Xilinx Alveo U55C cards and Nvidia GPUs -3. 100G fiber-optic switch between JUNGFRAU and server +2. Server system with AMD/Xilinx Alveo U55C cards and optionally Nvidia GPUs + +At the moment only U55C card is supported by Jungfraujoch. +Nvidia L4 GPU is recommended and used for performance tuning. ## FPGA bitstream Instructions see [here](fpga/README.md) ## Software -Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux version 9, but version 8 is also supported. +Recommended operating system is Red Hat Enterprise Linux (RHEL) / Rocky Linux versions 8 and 9 (both are tested on a regular basis). Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended, but likely possible with providing multiple packages from external repositories. +There are some limited tests with recent Ubuntu and Fedora distributions, though these are not systematic. Other linux platforms should work, but no tests were done so far. - + ### Dependencies Required: * C++20 compiler and C++20 standard library; recommended GCC 11+ or clang 14+ (Intel OneAPI, AMD AOCC) @@ -34,12 +39,20 @@ Required: * JPEG library (turbo-jpeg is also OK) Optional: -* CUDA compiler version 11 or newer - MX indexing won't work without it -* NUMA library +* CUDA compiler version 11 or newer - required for MX indexing and ML resolution estimation +* NUMA library - to pin threads to nodes/CPUs * Node.js - to make frontend * libtorch - for resolution estimation using model from Stanford - see below -provided as GIT submodules: SLS Detector Package, tinycbor (Intel), Zstandard (Facebook), Pistache webserver and fast replacement for bitshuffle filter (DECTRIS). +Provided as GIT submodules: +* SLS Detector Package - see [github.com/slsdetectorgroup/slsDetectorPackage](https://github.com/slsdetectorgroup/slsDetectorPackage) +* tinycbor (Intel) - see [github.com/intel/tinycbor](https://github.com/intel/tinycbor) +* Zstandard (Facebook) - see [github.com/facebook/zstd](https://github.com/facebook/zstd) +* Pistache webserver - see [github.com/pistacheio/pistache](https://github.com/pistacheio/pistache) +* Fast feedback indexer (Hans-Christian Stadler, PSI) - see [github.com/paulscherrerinstitute/fast-feedback-indexer](https://github.com/paulscherrerinstitute/fast-feedback-indexer) +* fast replacement for Bitshuffle pre-compression filter (Kal Cutter, DECTRIS) - see [github.com/kalcutter/bitshuffle](https://github.com/kalcutter/bitshuffle) + +For license check LICENSE file in respective directory Directly included in the repository: * JSON parser/writer from N. Lohmann - see [github.com/nlohmann/json](https://github.com/nlohmann/json) @@ -48,15 +61,16 @@ Directly included in the repository: * Bitshuffle filter from K. Masui - see [github.com/kiyo-masui/bitshuffle](https://github.com/kiyo-masui/bitshuffle) * LZ4 compression by Y.Collet - see [github.com/lz4/lz4](https://github.com/lz4/lz4) * Spdlog logging library - see [github.com/gabime/spdlog](https://github.com/gabime/spdlog) -* ZeroMQ library (through slsDetectorPackage) +* ZeroMQ library (through slsDetectorPackage) - see [github.com/zeromq/libzmq](https://github.com/zeromq/libzmq) + For license check LICENSE file in respective directory ### Software components -* `jfjoch_broker` in `broker` - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition -* `jfjoch_writer` in `writer` - HDF5 writer - -Configuration for the modules is given in configuration files present in `etc` directory. +* `jfjoch_broker` in `broker` - main service running on the Jungfraujoch server, responsible for control of the detector and data acquisition; + Example configuration `jfjoch_broker` for the modules is given in configuration files present in `etc` directory. +* `jfjoch_writer` in `writer` - HDF5 writer; HDF5 writer is designed to work on the same or separate server system. It has rather limited requirements in terms of performance and memory. + The goal is to separate data acquisition node with custom FPGA hardware and file system node with stronger security/stability requirements. ### Compilation Use the following commands: @@ -80,6 +94,15 @@ cd build cmake -DJFJOCH_WRITER_ONLY=ON .. make jfjoch ``` +## Web Frontend +Jungfraujoch is equipped with React-based web frontend for user-friendly experience. Frontend has the following options: +* Presenting current state of the detector +* Plotting results of online quality calculations +* Showing live view images from the detector +* JUNGFRAU calibration numbers +* Configuring the detector, as well as pedestal/initialization operations + +Frontend is written in TypeScript. For details see [frontend_ui/](frontend_ui) directory. ## Tests @@ -87,8 +110,11 @@ Automated test routine is then accessible as `tests/CatchTest`. There are also b * `CompressionBenchmark` to measure compression bandwidth (single threaded) * `HDF5DatasetWriteTest` to measure HDF5 dataset writing speed (single threaded) +* `jfjoch_spot_finding_test` to apply spot finding and indexing routines in Jungfraujoch to an example dataset - this is equivalent to FPGA spot finding algorithm, but NOT performance equivalent as it is particularly not-efficient +* `jfjoch_action_test` to test quality/performance of FPGA card(s) and software routines -In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin and CrystFEL. Input files for these programs are placed in `xds_durin` and `crystfel` folders. See `.gitlab-ci.yml` for details. +In addition, tests are executed to verify that datasets written by Jungfraujoch are readable with XDS Durin plugin, XDS Neggia plygin and CrystFEL. +Input files for these programs are placed in `xds_durin`, `xds_neggia` and `crystfel` folders. See `.gitlab-ci.yml` for details. ## Resolution estimation diff --git a/frame_serialize/README.md b/frame_serialize/README.md index 4d971f13..3902a961 100644 --- a/frame_serialize/README.md +++ b/frame_serialize/README.md @@ -8,7 +8,7 @@ There are minor differences at the moment: * Only single channel is allowed for images (this only matters for deserialization) * LZ4 compression is not allowed; Bitshuffle Zstandard is possible * Few fields are currently absent -* Few extra fields are present beyond DECTRIS standard +* Extra fields are present beyond DECTRIS standard * There is calibration message defined to transfer pedestal correction ## Start message @@ -97,6 +97,9 @@ See [DECTRIS documentation](https://github.com/dectris/documentation/tree/main/s | data_collection_efficiency | float | Image collection efficiency \[\] | | | | bkg_estimate | float | Mean value for pixels in resolution range from 3.0 to 5.0 A [\photons\] | | | | adu_histogram | Array(uint64) | ADU histogram | | | +| corr_beam_x_pxl | float | Difference between provided and calculated beam center X (pixels) - requires indexing | | X | +| corr_beam_y_pxl | float | Difference between provided and calculated beam center Y (pixels) - requires indexing | | X | +| corr_det_dist_mm | float | Difference between provided and calculated detector distance (mm) - requires indexing | | X | | user_data | string | Optional user defined text information | | | | roi_sum | int64 | Sum of ROI rectangle \[photons\] | | X | | image | Map(string -> Image) | Image | | | @@ -142,13 +145,5 @@ Pushes images using Stream2 serialization. This is using PUSH ZeroMQ sockets. Be Same as above, but multiple ZeroMQ sockets are used and images are distributed on a round-robin basis between the sockets based on the image number. Behavior for errors is the same as for `ZMQStream2Pusher`. -## ZMQStream2PreviewPublisher -Similar to sockets above, though with minor differences: - -* PUB ZeroMQ socket is used (i.e., messages might be skipped by receiver) -* Conflate options is used to ensure that the socket does not save "old" images -* No start and end messages are sent -* Communication is fully nonblocking - no exceptions are raised in case of communication issues and `true` is set always returned. - ## TestImagePusher This interface is used for automated tests. It reads incoming images and compares one image with the reference data provided. \ No newline at end of file