leonarski_fandClaude Fable 5
70c4d871b6
- Capitalize Rugnux as a proper noun throughout the prose; the command stays lowercase `rugnux` in code font. RUGNUX_OVERVIEW.md is retitled "What Rugnux does". - ACKNOWLEDGEMENT.md cites the raw-data repositories only: dataset counts and DOI prefixes moved out (EXTERNAL_TEST_DATA.md owns them), the ESRF data portal gains its citation (Dimper et al. 2019), and MXRDR remains name + link - it has no canonical citation paper. - RUGNUX_FORMATS.md: the CCD formats (marCCD, SMV) are supported as-is with very limited scope, and per-panel XFEL data is not read. - Fix wrong facts a reader would act on: nonexistent `make jfjoch` targets, invalid udev rules, PUSH sockets documented as PULL, swapped writer width/height, underload semantics, the transposed pixel-mask numpy example (the server checks width and height separately), the Durin/Neggia mask-bit table, FPGA threshold register addresses and the mailbox bit field, the I2C core's document number (PG090), an inverted MODEL_FIT_SIGMA formula, a self-inconsistent worked report example, and 11 cross-page anchors whose slugs carry MyST section numbers. - Unify CC1/2 spelling in prose; math notation and report keys unchanged. - Sweep grammar, typos and editing residue across the FPGA, deployment, streaming and analysis pages, including historical CHANGELOG typos. - rugnux_cli.cpp: the -S usage/error examples pair 96 with P43212; 92 names a different group. - Root THIRD_PARTY_NOTICES.md: scope the GPL-compatibility claim (CUDA EULA) and the vendored-table intro (traccc); the docs copy regenerates via update_version.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3.4 KiB
Hardware requirements
Operating Jungfraujoch requires the following:
- High performance server
- FPGA board(s) installed in the server
- (optionally) GPU boards
- (optionally) 100G switch to connect FPGA and the detector
Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include AMD FPGA boards. Therefore, the two have to be purchased separately. This might have impact on the warranty for the hardware and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.
High performance server
PSI is using HPE DL380 Gen11 servers at the moment to operate Jungfraujoch systems. However, this is because of general preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system from any other vendor with similar specification should work as well.
At PSI, the configuration of HPE DL380 Gen11 used to operate 9M pixel detectors at 2 kHz is as follows:
- 2 x Intel Xeon 8558P
- 512 GB RAM
- 2 x Nvidia L4 GPU (for indexing)
- 1 x Nvidia ConnectX-6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
- Copper 1G/10G network
PCI slots
When ordering the system, check that it can accommodate enough PCIe cards. In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2 x GPU, 1 x network.
Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.
FPGA
Jungfraujoch is built for AMD/Xilinx U55C (A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.
Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz. For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires using TX delay functionality in the detector.
GPUs
Operating fast-feedback indexer code requires a graphics processing unit from Nvidia. For practical reasons, i.e. power consumption and cost, we chose the inference-grade Nvidia L4 card. In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.
Offline processing with rugnux has a requirement of its own, unrelated to the frame
rate the acquisition side is sized for: one run wants 3-7 GB on the card and up to 14 GB of host
RAM, so a server that also reprocesses data needs headroom beyond the online pipeline's. The
sizing table is in Installing Rugnux ▸ Memory.
Network switch
Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs 8x10g variant
of the Jungfraujoch FPGA image, which allows 4 JUNGFRAU modules (two 10 Gbit/s links each) to be connected directly to one U55C card.
Such configuration is however impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujoch server. In this case one needs a network switch.
We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors. A switch with only 100G ports must support splitting them into 4x10G ports to connect the detector.