56 lines
3.1 KiB
Markdown
56 lines
3.1 KiB
Markdown
# Hardware requirements
|
|
Operating Jungfraujoch requires the following:
|
|
|
|
1. High performance server
|
|
2. FPGA board(s) installed in the server
|
|
3. (optionally) GPU boards
|
|
4. (optionally) 100G switch to connect FPGA and the detector
|
|
|
|
Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include
|
|
AMD FPGA boards. Therefore, the two has to be purchases separately. This might have impact on the warranty for the hardware
|
|
and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility
|
|
for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.
|
|
|
|
## High performance server
|
|
PSI is using HPE DL380 Gen11 servers are the moment to operate Jungfraujoch systems. However, this is because of general
|
|
preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system
|
|
from any other vendor with similar specification should work as well.
|
|
|
|
At PSI, we use the following configuration of HPE DL380 Gen11 to operate 9M pixel detectors at 2 kHz is as follows:
|
|
* 2 x Intel Xeon 8558P
|
|
* 512 GB RAM
|
|
* 2 x Nvidia L4 GPU (for indexing)
|
|
* 1 x Nvidia Connect-X 6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
|
|
* Copper 1G/10G network
|
|
|
|
### PCI slots
|
|
When ordering the system it is important to ensure enough PCIe cards can be accommodated in the system.
|
|
In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2x GPU, 1x network
|
|
|
|
Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.
|
|
|
|
## FPGA
|
|
Jungfraujoch is built for [AMD/Xilinx U55C](https://www.amd.com/en/products/accelerators/alveo/u55c/a-u55c-p00g-pq-g.html)
|
|
(A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.
|
|
|
|
Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz.
|
|
For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires
|
|
using TX delay functionality in the detector.
|
|
|
|
## GPUs
|
|
Operating fast-feedback indexer code requires operation of a graphic processing unit from Nvidia.
|
|
For practical reasons, i.e. power consumption and cost, we choose inference grade card Nvidia L4.
|
|
In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.
|
|
|
|
## Network switch
|
|
Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs `8x10g` variant
|
|
of the Jungfraujoch FPGA image, which allows to directly connect 4 JUNGFRAU modules to one U55C card.
|
|
|
|
Such configuration is however
|
|
impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujochs server.
|
|
In this case one needs a network switch.
|
|
|
|
We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors.
|
|
For switches with only 100G ports it is important to ensure, that these can be split into 4x10G ports to connect the detector.
|
|
|