Jungfraujoch/fpga/INSTALL.md

Installation of new U55C card for Jungfraujoch

1. Install the xbflash2 tool from Xilinx, which allows to flash the firmware through PCIe connector.

It can be downloaded as RPM/DEB file from Alveo product page. For RHEL9 this needs to built from source - Xilinx/XRT github repository.

2. Check that the card is detected by OS with "lspci |grep Xilinx" and check the PCIe slot number (11:00.0 in this case):

$ lspci |grep Xilinx
23:00.0 Processing accelerators: Xilinx Corporation Device 3450 (rev 58)

Note the device number 3450 that identifies Jungfraujoch device (Jungfraujoch pass is 3450 m above sea level) and rev 58 identifying release of the firmware.
3. Check the speed of the card, that it is detected as PCIe Gen4x8 device (needs to be done as root, otherwise configuration details are not given):

$ sudo lspci -vv -s <PCIe slot number>
23:00.0 Processing accelerators: Xilinx Corporation Device 3450
(...)
LnkSta:     Speed 16GT/s (ok), Width x8 (ok)
(...)

4. Download the MCS image from release files.
5. Flash the card with xbflash2. For fresh card use:

sudo xbflash2 program --spi --image <path to MCS file> --bar-offset 0x1f06000 -d <PCIe slot from above>

For card that was already flashed with Jungfraujoch images:

sudo xbflash2 program --spi --image <path to MCS file> -d <PCIe slot from above>

It is necessary to confirm the operation by pressing Y key. It is safe to run multiple flashing processes in parallel for different cards, for example in separate screen sessions.

6. Cold reboot:

sudo ipmitool chassis power cycle

7. Install PCIe driver. See pcie_driver/README.md for details. For first run it is though recommended to try the driver without installing to the kernel directory:

$ cd fpga/pcie_driver
$ make
$ sudo insmod jfjoch.ko

8. Check with dmesg that the device was properly found:

$ dmesg |grep jfjoch
[  431.624933] jfjoch 0000:23:00.0: enabling device (0140 -> 0142)
[  431.919147] misc jfjoch0: Jungfraujoch FPGA loaded with FW build: 5610030a

9. Configure switch according to network guide - specifically set manual speed and turn off auto-negotiation for the port used to connect U55C card.
10. Connect card to switch.

Card verification

To test that FPGA board is working properly without access to a JUNGFRAU detector, you can use jfjoch_action_test tool.

Card release number

To ensure compatibility of the card with driver and user application, each design is marked by release number. This number is incremented after each change of functionality of the card or interface to communicate with the host. This ensures consistency between the FPGA card, driver and user application. Changes within the design (e.g. size of FIFOs), that are invisible to interactions with host do not require change in release number.

To check release number, look for constant RELEASE_NUMBER in pcie_driver/jfjoch_fpga.h header file. For FPGA design, release number is saved as PCIe revision.

In case there is mismatch in release number between card and kernel driver, the latter will not create the character device and return error (check dmesg).

It is recommended to use the same git commit hash for building the design and user application, though it is not strictly necessary in case care is made to have same release number.