leonarski_fandClaude Fable 5
70c4d871b6
- Capitalize Rugnux as a proper noun throughout the prose; the command stays lowercase `rugnux` in code font. RUGNUX_OVERVIEW.md is retitled "What Rugnux does". - ACKNOWLEDGEMENT.md cites the raw-data repositories only: dataset counts and DOI prefixes moved out (EXTERNAL_TEST_DATA.md owns them), the ESRF data portal gains its citation (Dimper et al. 2019), and MXRDR remains name + link - it has no canonical citation paper. - RUGNUX_FORMATS.md: the CCD formats (marCCD, SMV) are supported as-is with very limited scope, and per-panel XFEL data is not read. - Fix wrong facts a reader would act on: nonexistent `make jfjoch` targets, invalid udev rules, PUSH sockets documented as PULL, swapped writer width/height, underload semantics, the transposed pixel-mask numpy example (the server checks width and height separately), the Durin/Neggia mask-bit table, FPGA threshold register addresses and the mailbox bit field, the I2C core's document number (PG090), an inverted MODEL_FIT_SIGMA formula, a self-inconsistent worked report example, and 11 cross-page anchors whose slugs carry MyST section numbers. - Unify CC1/2 spelling in prose; math notation and report keys unchanged. - Sweep grammar, typos and editing residue across the FPGA, deployment, streaming and analysis pages, including historical CHANGELOG typos. - rugnux_cli.cpp: the -S usage/error examples pair 96 with P43212; 92 names a different group. - Root THIRD_PARTY_NOTICES.md: scope the GPL-compatibility claim (CUDA EULA) and the vendored-table intro (traccc); the docs copy regenerates via update_version.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
61 lines
3.4 KiB
Markdown
61 lines
3.4 KiB
Markdown
# Hardware requirements
|
|
Operating Jungfraujoch requires the following:
|
|
|
|
1. High performance server
|
|
2. FPGA board(s) installed in the server
|
|
3. (optionally) GPU boards
|
|
4. (optionally) 100G switch to connect FPGA and the detector
|
|
|
|
Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include
|
|
AMD FPGA boards. Therefore, the two have to be purchased separately. This might have impact on the warranty for the hardware
|
|
and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility
|
|
for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.
|
|
|
|
## High performance server
|
|
PSI is using HPE DL380 Gen11 servers at the moment to operate Jungfraujoch systems. However, this is because of general
|
|
preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system
|
|
from any other vendor with similar specification should work as well.
|
|
|
|
At PSI, the configuration of HPE DL380 Gen11 used to operate 9M pixel detectors at 2 kHz is as follows:
|
|
* 2 x Intel Xeon 8558P
|
|
* 512 GB RAM
|
|
* 2 x Nvidia L4 GPU (for indexing)
|
|
* 1 x Nvidia ConnectX-6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
|
|
* Copper 1G/10G network
|
|
|
|
### PCI slots
|
|
When ordering the system, check that it can accommodate enough PCIe cards.
|
|
In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2 x GPU, 1 x network.
|
|
|
|
Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.
|
|
|
|
## FPGA
|
|
Jungfraujoch is built for [AMD/Xilinx U55C](https://www.amd.com/en/products/accelerators/alveo/u55c/a-u55c-p00g-pq-g.html)
|
|
(A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.
|
|
|
|
Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz.
|
|
For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires
|
|
using TX delay functionality in the detector.
|
|
|
|
## GPUs
|
|
Operating fast-feedback indexer code requires a graphics processing unit from Nvidia.
|
|
For practical reasons, i.e. power consumption and cost, we chose the inference-grade Nvidia L4 card.
|
|
In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.
|
|
|
|
Offline processing with [`rugnux`](RUGNUX.md) has a requirement of its own, unrelated to the frame
|
|
rate the acquisition side is sized for: one run wants 3-7 GB on the card and up to 14 GB of host
|
|
RAM, so a server that also reprocesses data needs headroom beyond the online pipeline's. The
|
|
sizing table is in [Installing Rugnux ▸ Memory](RUGNUX_INSTALL.md#memory).
|
|
|
|
## Network switch
|
|
Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs `8x10g` variant
|
|
of the Jungfraujoch FPGA image, which allows 4 JUNGFRAU modules (two 10 Gbit/s links each) to be connected directly to one U55C card.
|
|
|
|
Such configuration is however
|
|
impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujoch server.
|
|
In this case one needs a network switch.
|
|
|
|
We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors.
|
|
A switch with only 100G ports must support splitting them into 4x10G ports to connect the detector.
|
|
|