leonarski_f
a5ba1f8e95
Measured on rc.169 over rotation sweeps from 2.5 to 18 megapixels: both peaks fall in scaling and merging, and both track the number of integrated observations rather than the detector size - two sweeps on the same detector differ five-fold. Device memory is linear in -N at about 8 bytes per detector pixel per worker over a fixed floor, so the 16 GB card that fits four light runs fits one heavy one. --no-merge and -N lower the peak; --no-export-unmerged was measured and does not - it saves disk, not RAM. Also records what happens when the card is too small: device decode, the beam-stop projection and an indexer thread degrade to the host, everything else fails the run with a non-zero exit rather than returning a short dataset.
3.4 KiB
Hardware requirements
Operating Jungfraujoch requires the following:
- High performance server
- FPGA board(s) installed in the server
- (optionally) GPU boards
- (optionally) 100G switch to connect FPGA and the detector
Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include AMD FPGA boards. Therefore, the two have to be purchased separately. This might have impact on the warranty for the hardware and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.
High performance server
PSI is using HPE DL380 Gen11 servers at the moment to operate Jungfraujoch systems. However, this is because of general preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system from any other vendor with similar specification should work as well.
At PSI, we use the following configuration of HPE DL380 Gen11 to operate 9M pixel detectors at 2 kHz is as follows:
- 2 x Intel Xeon 8558P
- 512 GB RAM
- 2 x Nvidia L4 GPU (for indexing)
- 1 x Nvidia Connect-X 6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
- Copper 1G/10G network
PCI slots
When ordering the system, check that it can accommodate enough PCIe cards. In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2x GPU, 1x network
Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.
FPGA
Jungfraujoch is built for AMD/Xilinx U55C (A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.
Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz. For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires using TX delay functionality in the detector.
GPUs
Operating fast-feedback indexer code requires operation of a graphic processing unit from Nvidia. For practical reasons, i.e. power consumption and cost, we choose inference grade card Nvidia L4. In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.
Offline processing with rugnux has a requirement of its own, unrelated to the frame
rate the acquisition side is sized for: one run wants 3-7 GB on the card and up to 14 GB of host
RAM, so a server that also reprocesses data needs headroom beyond the online pipeline's. The
sizing table is in Installing rugnux ▸ Memory.
Network switch
Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs 8x10g variant
of the Jungfraujoch FPGA image, which allows to directly connect 4 JUNGFRAU modules to one U55C card.
Such configuration is however impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujochs server. In this case one needs a network switch.
We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors. A switch with only 100G ports must support splitting them into 4x10G ports to connect the detector.