Jungfraujoch/docs/FPGA.md

FPGA smartNIC

See separate document for installation instructions.

Hardware

Currently supported FPGA is only Xilinx Alveo U55C.

See AMD/Xilinx webpage for card user guide (UG1469). According to the user guide:

Alveo data center accelerator cards are designed to be installed into a data center server, where controlled air flow provides direct cooling.

Card needs to be placed in PCI Express (PCIe) Gen4 x8 slot, though mechanically slot has to accommodate x16 card. There is no need to connect additional power cable, as power of the card is not exceeding 75 W load available from PCIe edge connector. Current power estimation is about 30 W when idle and 45 W in operation. The card has built-in protection, which will cut power to the card if HBM temperature is above 120°C.

Two variants of the card are available:

• 100g - this variant operates one port in 100 Gbit/s mode and should be used when connecting detector via a switch.
• 8x10g - this variant operates both QSFP ports at 4x10 Gbit/s. QSFP+ (40 Gbit/s) transceivers and MTO/MTP harness cables are necessary. It is designed for detector directly connected to the Jungfraujoch server, without switch.

See network documentation for details of network.

Building firmware

The firmware build targets are generated by CMake only when vivado and vitis_hls are detected in the path, and the Vivado version has to match the one below precisely.

Xilinx Vivado

The following procedures require having AMD (Xilinx) Vivado and Vitis HLS toolsets version 2022.2 installed on the machine. Due to the nature of TCL scripts used to generate board designs Vivado version has to exactly match one provided above - specifically newer versions of Vivado will not work.

In additional to Intellectual Property (IP) cores included in Vivado, two additional licenses are necessary:

• Non-cost license for Ultrascale+ 100G core has to be requested from AMD/Xilinx website, see Xilinx website, to build 100g design.
• Paid 10G/25G Subsystem for Ultrascale+ to build 8x10g design. PSI received non-cost licenses from Xilinx University Program for the latter cores. Therefore, usage of bitstreams generated by PSI continuous integration pipeline for 8x10g is only allowed for non-commercial use.

HLS compilation

Make HLS routines:

mkdir build
cd build
cmake ..
make hls

Synthesis

Create PCIe 100g bitstream with the following command:

mkdir build
cd build
cmake ..
make pcie_100g

and 8x10g:

mkdir build
cd build
cmake ..
make pcie_8x10g

When Vivado is not present

During CMake execution, the following executables: vivado and vitis_hls must be present in the path. If not, build targets will not be generated, and such or similar error message will show up:

$ make pcie_100g
make: *** No rule to make target 'pcie_100g'.  Stop.

Firmware releases

The firmware is stable and is carried from version to version: the MCS files attached to a release are normally the ones from the release before it (see Release contents). When it does need to change, it is rebuilt with the targets above on a machine with Vivado.

Frame generator

Jungfraujoch card is equipped with frame generator. It allows to simulate JUNGFRAU detector without having access to such system. It is placed in parallel to Ethernet MAC - so it is placed before the network stack and before any processing happening on the card. In the future a redirection will be possible to send the simulated stream through the 100G TX network link. Frame generator is written in HLS and controlled with AXI-Lite.