leonarski_f
b00c668e3a
* Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast. * Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU). * Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors. * Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer. Reviewed-on: #90 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
3.5 KiB
Hardware requirements
Operating Jungfraujoch requires the following:
- High performance server
- FPGA board(s) installed in the server
- (optionally) GPU boards
- (optionally) 100G switch to connect FPGA and the detector
Unfortunately, at the moment it is not possible to purchase server configuration from a major vendor that would include AMD FPGA boards. Therefore, the two have to be purchased separately. This might have impact on the warranty for the hardware and has to be clarified with the vendor. PSI only supports the system on the best effort basis and doesn't take any responsibility for warranty limitations for operating FPGA boards in the server. Having said this - we didn't encounter any hardware issues so far.
High performance server
PSI is using HPE DL380 Gen11 servers at the moment to operate Jungfraujoch systems. However, this is because of general preference for this vendor, there is no Jungfraujoch-specific reason to buy from this vendor. We do expect that system from any other vendor with similar specification should work as well.
At PSI, the configuration of HPE DL380 Gen11 used to operate 9M pixel detectors at 2 kHz is as follows:
- 2 x Intel Xeon 8558P
- 512 GB RAM
- 2 x Nvidia L4 GPU (for indexing)
- 1 x Nvidia ConnectX-6 200G ethernet/IB network (for outgoing traffic; this can be substituted according to facility needs)
- Copper 1G/10G network
PCI slots
When ordering the system, check that it can accommodate enough PCIe cards. In case of our system we need to put at least seven PCIe cards: 4 x FPGA, 2 x GPU, 1 x network.
Note - for FPGA x8 lane electrically/x16 lane mechanically PCIe slots are OK.
FPGA
Jungfraujoch is built for AMD/Xilinx U55C (A-U55C-P00G-PQ-G) card. Other FPGA cards are currently not supported.
Single U55C card supports roughly 5 detector modules (2.5M pixels) at 2 kHz and 10 detector modules (5M pixels) at 1 kHz. For detectors operating at lower frame rates (e.g., 100 Hz) larger detectors can be supported by a single U55C card, though it requires using TX delay functionality in the detector.
GPUs
Running the fast-feedback indexer at the detector's frame rate requires a graphics processing unit from Nvidia (without one it runs on the CPU, at a much lower rate). For practical reasons, i.e. power consumption and cost, we chose the inference-grade Nvidia L4 card. In the past we have also used T4 cards. So, in principle any recent CUDA compatible GPU should work.
Offline processing with rugnux has a requirement of its own, unrelated to the frame
rate the acquisition side is sized for: one run wants 3-7 GB on the card and up to 14 GB of host
RAM, so a server that also reprocesses data needs headroom beyond the online pipeline's. The
sizing table is in Installing Rugnux ▸ Memory.
Network switch
Small detectors (up to 4M pixel) can be in principle operated without switch. In this case one needs 8x10g variant
of the Jungfraujoch FPGA image, which allows 4 JUNGFRAU modules (two 10 Gbit/s links each) to be connected directly to one U55C card.
Such configuration is however impractical for larger systems or more complex deployments, like multiple detectors operated from one Jungfraujoch server. In this case one needs a network switch.
We currently use Nvidia/Mellanox SN2100 switch, though there is no reason not to use other models/other vendors. A switch with only 100G ports must support splitting them into 4x10G ports to connect the detector.