leonarski_f
30b6800289
* Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
3.4 KiB
Hardware requirements
Operating Jungfraujoch requires the following:
- High performance server
- FPGA board(s) installed in the server
- (optionally) GPU boards
- (optionally) 100G switch to connect FPGA and the detector
Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include AMD FPGA boards. Therefore, the two have to be purchased separately. This might have impact on the warranty for the hardware and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.
High performance server
PSI is using HPE DL380 Gen11 servers at the moment to operate Jungfraujoch systems. However, this is because of general preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system from any other vendor with similar specification should work as well.
At PSI, the configuration of HPE DL380 Gen11 used to operate 9M pixel detectors at 2 kHz is as follows:
- 2 x Intel Xeon 8558P
- 512 GB RAM
- 2 x Nvidia L4 GPU (for indexing)
- 1 x Nvidia ConnectX-6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
- Copper 1G/10G network
PCI slots
When ordering the system, check that it can accommodate enough PCIe cards. In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2 x GPU, 1 x network.
Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.
FPGA
Jungfraujoch is built for AMD/Xilinx U55C (A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.
Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz. For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires using TX delay functionality in the detector.
GPUs
Operating fast-feedback indexer code requires a graphics processing unit from Nvidia. For practical reasons, i.e. power consumption and cost, we chose the inference-grade Nvidia L4 card. In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.
Offline processing with rugnux has a requirement of its own, unrelated to the frame
rate the acquisition side is sized for: one run wants 3-7 GB on the card and up to 14 GB of host
RAM, so a server that also reprocesses data needs headroom beyond the online pipeline's. The
sizing table is in Installing Rugnux ▸ Memory.
Network switch
Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs 8x10g variant
of the Jungfraujoch FPGA image, which allows 4 JUNGFRAU modules (two 10 Gbit/s links each) to be connected directly to one U55C card.
Such configuration is however impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujoch server. In this case one needs a network switch.
We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors. A switch with only 100G ports must support splitting them into 4x10G ports to connect the detector.