Jungfraujoch/fpga

FPGA Smart Network Interface Card

Hardware

Currently supported FPGA is only Xilinx Alveo U55C

Content of directories

CPU Part:

• pcie_driver Linux kernel driver for PCIe version of the FPGA board
• host_library Library that should be used to access the driver + some simple diagnostic tools

FPGA part:

• scripts Scripts for FPGA synthesis
• xdc Constraints for FPGA
• hdl FPGA design parts developed in Verilog
• hls FPGA design parts developed in C++ with high-level synthesis

Dependencies:

• include External (Xilinx) headers for high-level synthesis code

HLS compilation

Make HLS routines:

mkdir build
cd build
cmake3 ..
make hls

Synthesis

Create PCIe bitstream with 2 data stream (200 Gbit/s) and bifurcated 2 x Gen4x8 PCIe design:

mkdir build
cd build
cmake3 ..
make action_pcie

Create PCIe bitstream with 1 data stream (100 Gbit/s) and single Gen4x8 PCIe interface:

mkdir build
cd build
cmake3 ..
make action_pcie_100g

Hardware verification

To test that FPGA board is working properly without access to a JUNGFRAU detector, you can use jfjoch_action_test tool.

FPGA reference

Frame generator

Jungfraujoch card is equipped with frame generator. It allows to simulate JUNGFRAU detector without having access to such system. It is placed in parallel to Ethernet MAC - so it is placed before the network stack and before any processing happening on the card. In the future a redirection will be possible to send the simulated stream through the 100G TX network link. Frame generator is written in HLS and controlled with AXI-Lite.

Register map

FPGA setup can be done via 32-bit registers:

Address	Bits	Meaning	Mode	Notes
0x00000 - 0x0FFFF		Reserved (in case using MicroBlaze in the future, this has to reserved for internal memory)
0x010000	32	Action Control Register
		Bit 0 - Action start	R/W
		Bit 1 - Action idle	R
		Bit 2 - Action cancel	R/W	cleared on reset or action start
		Bit 3 - Clear network counters	R/W	cleared on reset or action start
		Bit 4 - Host writer idle	R	cleared on reset
		Bit 7 - Design number	R	0 = PCIe #0, 1 = PCIe #1
		Bit 16 - AXI Mailbox interrupt 0	R
		Bit 17 - AXI Mailbox interrupt 1	R
		Bits 24-27 - Various errors in host memory writer	R	cleared on reset or action start
0x010004	32	Reserved	-
0x010008	32	Reserved	-
0x01000C	32	Action GIT SHA1	R
0x010010	32	Action Type	R
0x010014	32	Action Release Level	R
0x010020	32	Max. number supported detector modules	R	constant
0x010024	32	Reserved	R	constant
0x010028	64	Pipeline stalls before writing to host memory	R	reset on action start
0x010030	64	Pipeline stalls before accessing HBM	R	reset on action start
0x010038	32	FIFO status (see action_config.v for details)	R
0x01003C	32	Size of single HBM channel in bytes (default value for the particular card)	R/W	should not be altered for standard operation
0x010040	64	Packets processed by the action	R	cleared on reset or action start
0x010048	64	Valid ethernet packets	R	cleared on reset
0x010050	64	Valid ICMP packets	R	cleared on reset
0x010058	64	Valid UDP packets	R	cleared on reset
0x010060	64	Valid detector packets processed by the card	R	cleared on reset
0x010066	64	Packets flagged as errors by CMAC	R	cleared on reset
0x010080	64	MAC address of FPGA card	R/W	network byte order
0x010088	32	IPv4 address of FPGA card	R/W	network byte order
0x01008C	32	Number of detector modules	R/W
0x010090	32	Data collection mode	R/W
		Bit 0 - Conversion to photons
		Bit 16:31 - Data collection ID (carried with completions)
0x010094	32	One over energy in keV (in fixed-point:12 int. + 24 frac. bit format)	R/W
0x010098	32	Number of frames expected in the data collection (defines termination condition)	R/W
0x01009C	32	Number of storage cells	R/W
0x010100	32	Spot finder photon count threshold	R/W
0x010104	32	Spot finder signal-to-noise ratio threshold (in fixed-point: 6 int. + 4 frac. bit format)	R/W
0x020000 - 0x02FFFF		CMAC 100G		See Xilinx PG203 for register map
0x030000 - 0x03FFFF		AXI Mailbox for Work Request / Work Completion		See Xilinx PG114 for register map
0x040000 - 0x04FFFF		QuadSPI flash		See Xilinx PG153 for register map
0x060000 - 0x060FFF	64	Input calibration memory addresses block RAM
0x070000 - 0x07FFFF		AXI Firewall		See Xilinx PG293 for register map
0x080000 - 0x08FFFF		Frame generator
0x090000 - 0x09FFFF		PCIe DMA control		See Xilinx PG195 for register map
0x0A0000 - 0x0AFFFF		Transfer between UltraRAM buffer <-> HBM (HLS registers)
0x0C0000 - 0x0FFFFF		Xilinx Card Management Solution Subsystem management subsystem		See Xilinx PG348 for register map
0x100000 - 0x1FFFFF	16	Internal packet generator frame
0x200000 - 0x2FFFFF		UltraRAM buffer for transfers to/from HBM

AXI Mailbox

AXI mailbox is used to send work request from host to action, and receive work completions. Messages are exchanged through AXI Mailbox IP from Xilinx (see Xilinx PG114).
Messages are always multiple of 128-bit.

Work request has the following structure:

Bit start	Bit end	Meaning
0	31	Work request ID (handle)
32	95	Address (Virt: OpenCAPI, DMA: PCIe)
96	127	Includes parity bit, so bits 0-127 are even parity

Work completion has the following structure:

Bit start	Bit end	Meaning
0	31	Work request ID (handle)
32	39	Module number
40	40	All packets for the module arrived OK
41	41	Trigger signal high
42	62	Reserved
63	63	Parity bit - bits 0-127 are even parity
64	127	Frame number
128	191	JF Timestamp
192	255	Bunch ID
256	287	Exptime
288	319	JF debug
320	351	Reserved
352	383	Data collection ID (16-bit)
384	511	Packet mask (1 bit per packet: 0 packet missing, 1 packet arrived)