# FPGA network

The U55C card is equipped with two network connectors - QSFP0 is the upper port and QSFP1 is lower port (when PCIe connector is on the bottom). 
The card FPGA design is offered in two variants `100g` and `8x10g`. These have different behavior regarding the network:

`100g` this variant operates QSFP0 port in 100 Gbit/s mode and should be used when connecting detector via a **switch**.
QSFP28 transceivers are necessary.

`8x10g` this variant operates both QSFP ports at 4x10 Gbit/s. QSFP+ (40 Gbit/s) transceivers and MTO/MTP harness cables
  are necessary. It is designed for **detector directly connected** to the Jungfraujoch server, without switch.

## Transceivers
AMD doesn't provide transceiver compatibility matrix for Alveo U55C. 
In our experience operating the card we haven't seen issues with transceivers from various providers (FS.com, Mellanox, Finnisar).
We have also successfully operated card with correct direct attach cables instead of fiber optics. Given the card doesn't
support link training functionality of 100 Gbit/s ethernet, it could result in performance problems with copper cables, though we haven't
encountered such a situation.

## Switch configuration
Special care has to be taken for switch operation, given the FPGA core doesn't support auto-negotiation. It is necessary to configure switch port
to fixed speed (100 Gbit/s or 10 Gbit/s) and to disable auto-negotiation.

## Network LEDs
Each QSFP connector is equipped with green and orange LEDs. These LEDs are connected to Ethernet physical layer status port (rx_status).
LED on corresponds to having a physical connection to a switch/computer/detector on the other side of the network. 
For 100 Gbit/s only green is used, for 8x10 Gbit/s green LEDs means all ports connected, orange LEDs at least one of the ports connected. 

## Network stack
Each Ethernet link has its own basic network stack. Functionality for Ethernet/ARP/IPv4/ICMP is therefore separately handled for each port.
Each link will get dedicated MAC address, and IPv4 addresses can be also assigned independently if needed. 

The card will send gratuitous ARP messages every 5 seconds to keep its entry in switch MAC table.
The card will also reply to ARP requests for its IP and to ICMP ping requests sent with the card IPv4 address. 
The card won't respond to broadcast ICMP pings.

Each link can be put in `direct` mode. In this case destination Ethernet MAC and IPv4 addresses are not enforced for incoming UDP packets.
This settings should be used for connecting detector modules directly to the FPGA card, so any detector module can be connected to any 
10 Gbit/s link on the same card. Currently `direct` mode is turned OFF for `100g` design and ON for `8x10g` design. 
This can be manually adjusted for each link.