39 lines
2.7 KiB
Markdown
39 lines
2.7 KiB
Markdown
# FPGA network
|
|
|
|
The U55C card is equipped with two network connectors - QSFP0 is the upper port and QSFP1 is lower port (when PCIe connector is on the bottom).
|
|
The card FPGA design is offered in two variants `100g` and `8x10g`. These have different behavior regarding the network:
|
|
|
|
`100g` this variant operates QSFP0 port in 100 Gbit/s mode and should be used when connecting detector via a **switch**.
|
|
QSFP28 transceivers are necessary.
|
|
|
|
`8x10g` this variant operates both QSFP ports at 4x10 Gbit/s. QSFP+ (40 Gbit/s) transceivers and MTO/MTP harness cables
|
|
are necessary. It is designed for **detector directly connected** to the Jungfraujoch server, without switch.
|
|
|
|
## Transceivers
|
|
AMD doesn't provide transceiver compatibility matrix for Alveo U55C.
|
|
In our experience operating the card we haven't seen issues with transceivers from various providers (FS.com, Mellanox, Finnisar).
|
|
We have also successfully operated card with correct direct attach cables instead of fiber optics. Given the card doesn't
|
|
support link training functionality of 100 Gbit/s ethernet, it could result in performance problems with copper cables, though we haven't
|
|
encountered such a situation.
|
|
|
|
## Switch configuration
|
|
Special care has to be taken for switch operation, given the FPGA core doesn't support auto-negotiation. It is necessary to configure switch port
|
|
to fixed speed (100 Gbit/s or 10 Gbit/s) and to disable auto-negotiation.
|
|
|
|
## Network LEDs
|
|
Each QSFP connector is equipped with green and orange LEDs. These LEDs are connected to Ethernet physical layer status port (rx_status).
|
|
LED on corresponds to having a physical connection to a switch/computer/detector on the other side of the network.
|
|
For 100 Gbit/s only green is used, for 8x10 Gbit/s green LEDs means all ports connected, orange LEDs at least one of the ports connected.
|
|
|
|
## Network stack
|
|
Each Ethernet link has its own basic network stack. Functionality for Ethernet/ARP/IPv4/ICMP is therefore separately handled for each port.
|
|
Each link will get dedicated MAC address, and IPv4 addresses can be also assigned independently if needed.
|
|
|
|
The card will send gratuitous ARP messages every 5 seconds to keep its entry in switch MAC table.
|
|
The card will also reply to ARP requests for its IP and to ICMP ping requests sent with the card IPv4 address.
|
|
The card won't respond to broadcast ICMP pings.
|
|
|
|
Each link can be put in `direct` mode. In this case destination Ethernet MAC and IPv4 addresses are not enforced for incoming UDP packets.
|
|
This settings should be used for connecting detector modules directly to the FPGA card, so any detector module can be connected to any
|
|
10 Gbit/s link on the same card. Currently `direct` mode is turned OFF for `100g` design and ON for `8x10g` design.
|
|
This can be manually adjusted for each link. |