Jungfraujoch/docs/FPGA.md

# FPGA smartNIC

See separate document for [installation instructions](DEPLOYMENT.md).

## Hardware
Currently supported FPGA is only **Xilinx Alveo U55C**. 

See AMD/Xilinx webpage for [card user guide (UG1469)](https://docs.xilinx.com/r/en-US/ug1469-alveo-u55c). 
According to the user guide: 
```
Alveo data center accelerator cards are designed to be installed into a data center server, where controlled air flow provides direct cooling.
```

The card needs to be placed in a PCI Express (PCIe) Gen4 x8 slot, though mechanically slot has to accommodate x16 card.
There is no need to connect additional power cable, as power of the card is not exceeding 75 W load available from PCIe edge connector.
Current power estimation is about 30 W when idle and 45 W in operation. The card has built-in protection, which will cut power to the card if HBM temperature is above 120°C.

Two variants of the card are available:
* `100g` - this variant operates one port in 100 Gbit/s mode and should be used when connecting detector via a switch.
* `8x10g` - this variant operates both QSFP ports at 4x10 Gbit/s. QSFP+ (40 Gbit/s) transceivers and MTO/MTP harness cables
are necessary. It is designed for detector directly connected to the Jungfraujoch server, without switch.

See [network documentation](FPGA_NETWORK.md) for details of network.

## Building firmware
The firmware build targets are generated by CMake only when `vivado` and `vitis_hls` are detected in
the path, and the Vivado version has to match the one below precisely.

### Xilinx Vivado
The following procedures require having AMD (Xilinx) Vivado and Vitis HLS toolsets version **2022.2** installed on the machine.
Due to the nature of TCL scripts used to generate board designs Vivado version has to exactly match one provided above - 
specifically newer versions of Vivado will not work. 

In addition to the Intellectual Property (IP) cores included in Vivado, two additional licenses are necessary:
* Non-cost license for Ultrascale+ 100G core has to be requested from AMD/Xilinx website, see [Xilinx website](https://www.xilinx.com/products/intellectual-property/cmac_usplus.html), to build `100g` design.
* A paid license for the 10G/25G Ethernet Subsystem for Ultrascale+ is necessary to build the `8x10g` design.
PSI received non-cost licenses from Xilinx University Program for the latter cores. Therefore, usage of bitstreams 
generated by PSI continuous integration pipeline for `8x10g` is only allowed for non-commercial use.

### HLS compilation
Make HLS routines:
```
mkdir build
cd build
cmake ..
make hls
```

### Synthesis
Create PCIe `100g` bitstream with the following command:
```
mkdir build
cd build
cmake ..
make pcie_100g
```
and `8x10g`:
```
mkdir build
cd build
cmake ..
make pcie_8x10g
```
### When Vivado is not present

During CMake execution, the following executables: `vivado` and `vitis_hls` must be present in the path.
If not, build targets will not be generated, and such or similar error message will show up:
```
$ make pcie_100g
make: *** No rule to make target 'pcie_100g'.  Stop.
```

### Firmware releases
The firmware is stable and is carried from version to version: the MCS files attached to a release
are normally the ones from the release before it (see [Release contents](RELEASE_CONTENTS.md)). When
it does need to change, it is rebuilt with the targets above on a machine with Vivado.

## Frame generator

The Jungfraujoch card is equipped with a frame generator. It allows simulating a JUNGFRAU detector without having access to such a system.
It sits in parallel with the Ethernet MAC, so it is placed before the network stack and before any processing happening on the card.
In the future a redirection will be possible to send the simulated stream through the 100G TX network link.
Frame generator is written in HLS and controlled with AXI-Lite.