From b022dd0055fe4fbe98c31813f4ab78cdddc3fcc1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 27 Oct 2023 12:07:15 +0200 Subject: [PATCH] Improved FPGA README.md --- README.md | 11 +++++++---- fpga/README.md | 36 ++++++++++++++++++++++++++++-------- fpga/pcie_driver/README.md | 13 ++++++++++++- 3 files changed, 47 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 63219d37..de8699e7 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,15 @@ # Jungfraujoch -Application to receive data from the JUNGFRAU detector at Swiss Light Source macromolecular crystallography beamlines. +Application to receive data from the JUNGFRAU detector. Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268). +The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG). + ## License -Operating Jungfraujoch, as well as sharing sources code requires explicit license from PSI. +Operating Jungfraujoch requires license from the Paul Scherrer Institute. +Sharing the code requires explicit permission from the Paul Scherrer Institute. ## Hardware requirements 1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards) @@ -16,10 +19,10 @@ Operating Jungfraujoch, as well as sharing sources code requires explicit licens ## FPGA bitstream ### Dependencies To build FPGA image you need: -1. Xilinx Vivado 2022.1 or newer +1. Xilinx Vivado 2022.1 2. License for Ultrascale+ 100G core - license is available at no cost from Xilinx, but needs to be seperatly requested, see [Xilinx website](https://www.xilinx.com/products/intellectual-property/cmac_usplus.html). -Instructions see [here](receiver/README.md) +Instructions see [here](fpga/README.md) ## Software diff --git a/fpga/README.md b/fpga/README.md index abd280ac..f9c4e6b7 100644 --- a/fpga/README.md +++ b/fpga/README.md @@ -1,13 +1,17 @@ # FPGA Smart Network Interface Card ## Hardware -Currently supported FPGA is only Xilinx Alveo U55C +Currently supported FPGA is only **Xilinx Alveo U55C**. + +Card needs to be placed in PCI Express (PCIe) Gen4 x8 slot, though mechanically slot has to accommodate x16 card. +There is no need to connect additional power cable, as power of the card is not exceeding 75 W load available from PCIe edge connector. +Current power estimation is about 30 W when idle and 40 W in operation. ## Content of directories CPU Part: -* `pcie_driver` Linux kernel driver for PCIe version of the FPGA board +* `pcie_driver` Linux kernel driver for PCIe version of the FPGA board - see [instructions](pcie_driver/README.md) * `host_library` Library that should be used to access the driver + some simple diagnostic tools FPGA part: @@ -32,7 +36,7 @@ make hls ``` ## Synthesis -Create PCIe bitstream with 2 data stream (200 Gbit/s) and bifurcated 2 x Gen4x8 PCIe design: +Create PCIe bitstream with the following command: ``` mkdir build cd build @@ -40,13 +44,29 @@ cmake3 .. make action_pcie ``` -Create PCIe bitstream with 1 data stream (100 Gbit/s) and single Gen4x8 PCIe interface: +## Flashing of the card + +After successfully building the bitstream, it is possible to load it onto the Alveo card. + +### New Alveo card +For the first operation of the card it is necessary to upload the bitstream via MicroUSB cable using other system running Vivado. +It is impossible to perform the operation using Vivado running on the same machine. + +### Alveo card with Jungfraujoch firmware +First step is to find PCIe device ID from the operating system. Use for example the following command: ``` -mkdir build -cd build -cmake3 .. -make action_pcie_100g +lspci |grep Xilinx ``` +To flash the card it is necessary to use 'xbflash2' utility from AMD/Xilinx: +``` +xbflash2 program --spi -i <.mcs file path> -d +``` +It is necessary to confirm the operation by pressing `Y` key. +It is safe to run multiple flashing processes in parallel for different cards, for example in separate screen sessions. + +### After flashing the card +Irrespective of the method use to upload firmware, it is necessary to do cold reboot the server (with a short power interruption to PCIe devices). +Currently, this is best done by powering the server off and on again. More efficient procedure is yet to be tested. ## Hardware verification diff --git a/fpga/pcie_driver/README.md b/fpga/pcie_driver/README.md index 7bb54cd9..3804bac0 100644 --- a/fpga/pcie_driver/README.md +++ b/fpga/pcie_driver/README.md @@ -5,18 +5,29 @@ To compile kernel module type: ``` make ``` + ## Installation To install kernel module, you need to have root permissions and run: ``` sudo make install ``` +## Loading driver into kernel +After installing the kernel driver, it should be possible to insert it into the kernel via: +``` +modprobe jfjoch +``` + +## CMake file +While CMake file is present in the driver directory, it is only for the purpose of proper detection of the files in CLion IDE. +It is not made for actual compilation of the kernel driver and should not be used for that purpose. + ## Character device For each FPGA device a character device is created called /dev/jfjoch<number of device>. When device is opened two operations are possible: mmap() to map device buffers ioctl() to communicate with the cards -Interfacing should be done through the JungfraujochDevice class. +Interfacing should be done through the JungfraujochDevice class in `fpga/host_library` directory. ## TODO /sys filesystem for diagnostics