Jungfraujoch/docs/HARDWARE.md

Hardware requirements

Operating Jungfraujoch requires the following:

1. High performance server
2. FPGA board(s) installed in the server
3. (optionally) GPU boards
4. (optionally) 100G switch to connect FPGA and the detector

Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include AMD FPGA boards. Therefore, the two has to be purchases separately. This might have impact on the warranty for the hardware and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.

High performance server

PSI is using HPE DL380 Gen11 servers are the moment to operate Jungfraujoch systems. However, this is because of general preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system from any other vendor with similar specification should work as well.

At PSI, we use the following configuration of HPE DL380 Gen11 to operate 9M pixel detectors at 2 kHz is as follows:

• 2 x Intel Xeon 8558P
• 512 GB RAM
• 2 x Nvidia L4 GPU (for indexing)
• 1 x Nvidia Connect-X 6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
• Copper 1G/10G network

PCI slots

When ordering the system it is important to ensure enough PCIe cards can be accommodated in the system. In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2x GPU, 1x network

Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.

FPGA

Jungfraujoch is built for AMD/Xilinx U55C (A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.

Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz. For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires using TX delay functionality in the detector.

GPUs

Operating fast-feedback indexer code requires operation of a graphic processing unit from Nvidia. For practical reasons, i.e. power consumption and cost, we choose inference grade card Nvidia L4. In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.

Network switch

Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs 8x10g variant of the Jungfraujoch FPGA image, which allows to directly connect 4 JUNGFRAU modules to one U55C card.

Such configuration is however impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujochs server. In this case one needs a network switch.

We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors. For switches with only 100G ports it is important to ensure, that these can be split into 4x10G ports to connect the detector.