Improved FPGA README.md

2023-10-27 12:07:15 +02:00
parent 6d74732bf5
commit b022dd0055
3 changed files with 47 additions and 13 deletions
@@ -1,12 +1,15 @@
 # Jungfraujoch

-Application to receive data from the JUNGFRAU detector at Swiss Light Source macromolecular crystallography beamlines.
+Application to receive data from the JUNGFRAU detector.

 Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).

+The project is supported by Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG).
+
 ## License

-Operating Jungfraujoch, as well as sharing sources code requires explicit license from PSI.
+Operating Jungfraujoch requires license from the Paul Scherrer Institute.
+Sharing the code requires explicit permission from the Paul Scherrer Institute.

 ## Hardware requirements
 1. JUNGFRAU detector (optimally 4M with 2 kHz enabled read-out boards)
@@ -16,10 +19,10 @@ Operating Jungfraujoch, as well as sharing sources code requires explicit licens
 ## FPGA bitstream
 ### Dependencies
 To build FPGA image you need:
-1. Xilinx Vivado 2022.1 or newer
+1. Xilinx Vivado 2022.1
 2. License for Ultrascale+ 100G core - license is available at no cost from Xilinx, but needs to be seperatly requested, see [Xilinx website](https://www.xilinx.com/products/intellectual-property/cmac_usplus.html).

-Instructions see [here](receiver/README.md)
+Instructions see [here](fpga/README.md)

 ## Software

@@ -1,13 +1,17 @@
 # FPGA Smart Network Interface Card

 ## Hardware
-Currently supported FPGA is only Xilinx Alveo U55C
+Currently supported FPGA is only **Xilinx Alveo U55C**.
+
+Card needs to be placed in PCI Express (PCIe) Gen4 x8 slot, though mechanically slot has to accommodate x16 card.
+There is no need to connect additional power cable, as power of the card is not exceeding 75 W load available from PCIe edge connector.
+Current power estimation is about 30 W when idle and 40 W in operation.

 ## Content of directories

 CPU Part:

-* `pcie_driver` Linux kernel driver for PCIe version of the FPGA board
+* `pcie_driver` Linux kernel driver for PCIe version of the FPGA board - see [instructions](pcie_driver/README.md)
 * `host_library` Library that should be used to access the driver + some simple diagnostic tools

 FPGA part:
@@ -32,7 +36,7 @@ make hls
 ```

 ## Synthesis
-Create PCIe bitstream with 2 data stream (200 Gbit/s) and bifurcated 2 x Gen4x8 PCIe design:
+Create PCIe bitstream with the following command:
 ```
 mkdir build
 cd build
@@ -40,13 +44,29 @@ cmake3 ..
 make action_pcie
 ```

-Create PCIe bitstream with 1 data stream (100 Gbit/s) and single Gen4x8 PCIe interface:
+## Flashing of the card
+
+After successfully building the bitstream, it is possible to load it onto the Alveo card. 
+
+### New Alveo card
+For the first operation of the card it is necessary to upload the bitstream via MicroUSB cable using other system running Vivado.
+It is impossible to perform the operation using Vivado running on the same machine.
+
+### Alveo card with Jungfraujoch firmware
+First step is to find PCIe device ID from the operating system. Use for example the following command:
 ```
-mkdir build
-cd build
-cmake3 ..
-make action_pcie_100g
+lspci |grep Xilinx
 ```
+To flash the card it is necessary to use 'xbflash2' utility from AMD/Xilinx:
+```
+xbflash2 program --spi -i <.mcs file path> -d <device id>
+```
+It is necessary to confirm the operation by pressing `Y` key.
+It is safe to run multiple flashing processes in parallel for different cards, for example in separate screen sessions.
+
+### After flashing the card
+Irrespective of the method use to upload firmware, it is necessary to do cold reboot the server (with a short power interruption to PCIe devices).
+Currently, this is best done by powering the server off and on again. More efficient procedure is yet to be tested.

 ## Hardware verification

@@ -5,18 +5,29 @@ To compile kernel module type:
 ```
 make
 ```
+
 ## Installation
 To install kernel module, you need to have root permissions and run:
 ```
 sudo make install
 ```

+## Loading driver into kernel
+After installing the kernel driver, it should be possible to insert it into the kernel via:
+```
+modprobe jfjoch
+```
+
+## CMake file
+While CMake file is present in the driver directory, it is only for the purpose of proper detection of the files in CLion IDE. 
+It is not made for actual compilation of the kernel driver and should not be used for that purpose.
+
 ## Character device
 For each FPGA device a character device is created called /dev/jfjoch&lt;number of device>.
 When device is opened two operations are possible:
 mmap() to map device buffers
 ioctl() to communicate with the cards
-Interfacing should be done through the JungfraujochDevice class.
+Interfacing should be done through the JungfraujochDevice class in `fpga/host_library` directory.

 ## TODO
 /sys filesystem for diagnostics