152 lines
13 KiB
Markdown
152 lines
13 KiB
Markdown
# FPGA Smart Network Interface Card
|
|
|
|
## Hardware
|
|
Currently supported FPGA is only Xilinx Alveo U55C
|
|
|
|
## Content of directories
|
|
|
|
CPU Part:
|
|
|
|
* `pcie_driver` Linux kernel driver for PCIe version of the FPGA board
|
|
* `host_library` Library that should be used to access the driver + some simple diagnostic tools
|
|
|
|
FPGA part:
|
|
|
|
* `scripts` Scripts for FPGA synthesis
|
|
* `xdc` Constraints for FPGA
|
|
* `hdl` FPGA design parts developed in Verilog
|
|
* `hls` FPGA design parts developed in C++ with high-level synthesis
|
|
|
|
Dependencies:
|
|
|
|
* `include` External (Xilinx) headers for high-level synthesis code
|
|
|
|
|
|
## HLS compilation
|
|
Make HLS routines:
|
|
```
|
|
mkdir build
|
|
cd build
|
|
cmake3 ..
|
|
make hls
|
|
```
|
|
|
|
## Synthesis
|
|
Create PCIe bitstream with 2 data stream (200 Gbit/s) and bifurcated 2 x Gen4x8 PCIe design:
|
|
```
|
|
mkdir build
|
|
cd build
|
|
cmake3 ..
|
|
make action_pcie
|
|
```
|
|
|
|
Create PCIe bitstream with 1 data stream (100 Gbit/s) and single Gen4x8 PCIe interface:
|
|
```
|
|
mkdir build
|
|
cd build
|
|
cmake3 ..
|
|
make action_pcie_100g
|
|
```
|
|
|
|
## Hardware verification
|
|
|
|
To test that FPGA board is working properly without access to a JUNGFRAU detector, you can use `jfjoch_action_test` tool.
|
|
|
|
## FPGA reference
|
|
|
|
### Frame generator
|
|
|
|
Jungfraujoch card is equipped with frame generator. It allows to simulate JUNGFRAU detector without having access to such system.
|
|
It is placed in parallel to Ethernet MAC - so it is placed before the network stack and before any processing happening on the card.
|
|
In the future a redirection will be possible to send the simulated stream through the 100G TX network link.
|
|
Frame generator is written in HLS and controlled with AXI-Lite.
|
|
|
|
### Register map
|
|
FPGA setup can be done via 32-bit registers:
|
|
|
|
| Address | Bits | Meaning | Mode | Notes |
|
|
|---------------------|------|---------------------------------------------------------------------------------------------|:-----|----------------------------------------------|
|
|
| 0x00000 - 0x0FFFF | | Reserved (in case using MicroBlaze in the future, this has to reserved for internal memory) | | |
|
|
| 0x010000 | 32 | Action Control Register | | |
|
|
| | | Bit 0 - Action start | R/W | |
|
|
| | | Bit 1 - Action idle | R | |
|
|
| | | Bit 2 - Action cancel | R/W | cleared on reset or action start |
|
|
| | | Bit 3 - Clear network counters | R/W | cleared on reset or action start |
|
|
| | | Bit 4 - Host writer idle | R | cleared on reset |
|
|
| | | Bit 7 - Design number | R | 0 = PCIe #0, 1 = PCIe #1 |
|
|
| | | Bit 16 - AXI Mailbox interrupt 0 | R | |
|
|
| | | Bit 17 - AXI Mailbox interrupt 1 | R | |
|
|
| | | Bits 24-27 - Various errors in host memory writer | R | cleared on reset or action start |
|
|
| 0x010004 | 32 | Reserved | - | |
|
|
| 0x010008 | 32 | Reserved | - | |
|
|
| 0x01000C | 32 | Action GIT SHA1 | R | |
|
|
| 0x010010 | 32 | Action Type | R | |
|
|
| 0x010014 | 32 | Action Release Level | R | |
|
|
| 0x010020 | 32 | Max. number supported detector modules | R | constant |
|
|
| 0x010024 | 32 | Reserved | R | constant |
|
|
| 0x010028 | 64 | Pipeline stalls before writing to host memory | R | reset on action start |
|
|
| 0x010030 | 64 | Pipeline stalls before accessing HBM | R | reset on action start |
|
|
| 0x010038 | 32 | FIFO status (see action_config.v for details) | R | |
|
|
| 0x01003C | 32 | Size of single HBM channel in bytes (default value for the particular card) | R/W | should not be altered for standard operation |
|
|
| 0x010040 | 64 | Packets processed by the action | R | cleared on reset or action start |
|
|
| 0x010048 | 64 | Valid ethernet packets | R | cleared on reset |
|
|
| 0x010050 | 64 | Valid ICMP packets | R | cleared on reset |
|
|
| 0x010058 | 64 | Valid UDP packets | R | cleared on reset |
|
|
| 0x010060 | 64 | Valid detector packets processed by the card | R | cleared on reset |
|
|
| 0x010066 | 64 | Packets flagged as errors by CMAC | R | cleared on reset |
|
|
| 0x010080 | 64 | MAC address of FPGA card | R/W | network byte order |
|
|
| 0x010088 | 32 | IPv4 address of FPGA card | R/W | network byte order |
|
|
| 0x01008C | 32 | Number of detector modules | R/W | |
|
|
| 0x010090 | 32 | Data collection mode | R/W | |
|
|
| | | Bit 0 - Conversion to photons | | |
|
|
| | | Bit 16:31 - Data collection ID (carried with completions) | | |
|
|
| 0x010094 | 32 | One over energy in keV (in fixed-point:12 int. + 24 frac. bit format) | R/W | |
|
|
| 0x010098 | 32 | Number of frames expected in the data collection (defines termination condition) | R/W | |
|
|
| 0x01009C | 32 | Number of storage cells | R/W | |
|
|
| 0x010100 | 32 | Spot finder photon count threshold | R/W | |
|
|
| 0x010104 | 32 | Spot finder signal-to-noise ratio threshold (in fixed-point: 6 int. + 4 frac. bit format) | R/W | |
|
|
| 0x020000 - 0x02FFFF | | CMAC 100G | | See Xilinx PG203 for register map |
|
|
| 0x030000 - 0x03FFFF | | AXI Mailbox for Work Request / Work Completion | | See Xilinx PG114 for register map |
|
|
| 0x040000 - 0x04FFFF | | QuadSPI flash | | See Xilinx PG153 for register map |
|
|
| 0x060000 - 0x060FFF | 64 | Input calibration memory addresses block RAM | | |
|
|
| 0x070000 - 0x07FFFF | | AXI Firewall | | See Xilinx PG293 for register map |
|
|
| 0x080000 - 0x08FFFF | | Frame generator | | |
|
|
| 0x090000 - 0x09FFFF | | PCIe DMA control | | See Xilinx PG195 for register map |
|
|
| 0x0A0000 - 0x0AFFFF | | Transfer between UltraRAM buffer <-> HBM (HLS registers) | | |
|
|
| 0x0C0000 - 0x0FFFFF | | Xilinx Card Management Solution Subsystem management subsystem | | See Xilinx PG348 for register map |
|
|
| 0x100000 - 0x1FFFFF | 16 | Internal packet generator frame | | |
|
|
| 0x200000 - 0x2FFFFF | | UltraRAM buffer for transfers to/from HBM | | |
|
|
|
|
### AXI Mailbox
|
|
|
|
AXI mailbox is used to send work request from host to action, and receive work completions.
|
|
Messages are exchanged through AXI Mailbox IP from Xilinx (see Xilinx PG114).
|
|
Messages are always multiple of 128-bit.
|
|
|
|
Work request has the following structure:
|
|
|
|
| Bit start | Bit end | Meaning |
|
|
|-----------|---------|----------------------------------------------------|
|
|
| 0 | 31 | Work request ID (handle) |
|
|
| 32 | 95 | Address (Virt: OpenCAPI, DMA: PCIe) |
|
|
| 96 | 127 | Includes parity bit, so bits 0-127 are even parity |
|
|
|
|
Work completion has the following structure:
|
|
|
|
| Bit start | Bit end | Meaning |
|
|
|-----------|---------|--------------------------------------------------------------------|
|
|
| 0 | 31 | Work request ID (handle) |
|
|
| 32 | 39 | Module number |
|
|
| 40 | 40 | All packets for the module arrived OK |
|
|
| 41 | 41 | Trigger signal high |
|
|
| 42 | 62 | Reserved |
|
|
| 63 | 63 | Parity bit - bits 0-127 are even parity |
|
|
| 64 | 127 | Frame number |
|
|
| 128 | 191 | JF Timestamp |
|
|
| 192 | 255 | Bunch ID |
|
|
| 256 | 287 | Exptime |
|
|
| 288 | 319 | JF debug |
|
|
| 320 | 351 | Reserved |
|
|
| 352 | 383 | Data collection ID (16-bit) |
|
|
| 384 | 511 | Packet mask (1 bit per packet: 0 packet missing, 1 packet arrived) |
|