Jungfraujoch/docs/FPGA_NETWORK.md

FPGA network

The U55C card is equipped with two network connectors - QSFP0 is the upper port and QSFP1 is lower port (when PCIe connector is on the bottom). The card FPGA design is offered in two variants 100g and 8x10g. These have different behavior regarding the network:

100g this variant operates QSFP0 port in 100 Gbit/s mode and should be used when connecting detector via a switch. QSFP28 transceivers are necessary.

8x10g this variant operates both QSFP ports at 4x10 Gbit/s. QSFP+ (40 Gbit/s) transceivers and MTO/MTP harness cables are necessary. It is designed for detector directly connected to the Jungfraujoch server, without switch.

Transceivers

AMD doesn't provide transceiver compatibility matrix for Alveo U55C. In our experience operating the card we haven't seen issues with transceivers from various providers (FS.com, Mellanox, Finnisar). We have also successfully operated card with correct direct attach cables instead of fiber optics. Given the card doesn't support link training functionality of 100 Gbit/s ethernet, it could result in performance problems with copper cables, though we haven't encountered such a situation.

Switch configuration

Special care has to be taken for switch operation, given the FPGA core doesn't support auto-negotiation. It is necessary to configure switch port to fixed speed (100 Gbit/s or 10 Gbit/s) and to disable auto-negotiation.

Network LEDs

Each QSFP connector is equipped with green and orange LEDs. These LEDs are connected to Ethernet physical layer status port (rx_status). LED on corresponds to having a physical connection to a switch/computer/detector on the other side of the network. For 100 Gbit/s only green is used, for 8x10 Gbit/s green LEDs means all ports connected, orange LEDs at least one of the ports connected.

Network stack

Each Ethernet link has its own basic network stack. Functionality for Ethernet/ARP/IPv4/ICMP is therefore separately handled for each port. Each link will get dedicated MAC address, and IPv4 addresses can be also assigned independently if needed.

The card will send gratuitous ARP messages every 5 seconds to keep its entry in switch MAC table. The card will also reply to ARP requests for its IP and to ICMP ping requests sent with the card IPv4 address. The card won't respond to broadcast ICMP pings.

Each link can be put in direct mode. In this case destination Ethernet MAC and IPv4 addresses are not enforced for incoming UDP packets. This settings should be used for connecting detector modules directly to the FPGA card, so any detector module can be connected to any 10 Gbit/s link on the same card. Currently direct mode is turned OFF for 100g design and ON for 8x10g design. This can be manually adjusted for each link.