leonarski_f
30b6800289
Build Packages / build:windows:nocuda (push) Successful in 17m20s
Build Packages / build:windows:cuda (push) Successful in 19m52s
Build Packages / build:viewer-tgz:cpu (push) Successful in 9m38s
Build Packages / build:viewer-tgz:cuda (push) Successful in 11m18s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 9m34s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 5m42s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 20m33s
Build Packages / Create release (push) Successful in 33s
Build Packages / build:rugnux:windows (push) Successful in 12m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 15m42s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m59s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m8s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 16m55s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 16m58s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky8) (push) Successful in 15m21s
Build Packages / Build documentation (push) Successful in 54s
Build Packages / build:rpm (rocky9) (push) Successful in 16m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m2s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m6s
Build Packages / Unit tests (push) Successful in 1h10m26s
* Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
39 lines
2.8 KiB
Markdown
39 lines
2.8 KiB
Markdown
# FPGA network
|
|
|
|
The U55C card is equipped with two network connectors - QSFP0 is the upper port and QSFP1 the lower port (when PCIe connector is on the bottom).
|
|
The card FPGA design is offered in two variants `100g` and `8x10g`. These have different behavior regarding the network:
|
|
|
|
`100g` — this variant operates the QSFP0 port in 100 Gbit/s mode and should be used when connecting detector via a **switch**.
|
|
QSFP28 transceivers are necessary.
|
|
|
|
`8x10g` — this variant operates both QSFP ports at 4x10 Gbit/s. QSFP+ (40 Gbit/s) transceivers and MTO/MTP harness cables
|
|
are necessary. It is designed for **detector directly connected** to the Jungfraujoch server, without switch.
|
|
|
|
## Transceivers
|
|
AMD doesn't provide transceiver compatibility matrix for Alveo U55C.
|
|
In our experience operating the card we haven't seen issues with transceivers from various providers (FS.com, Mellanox, Finisar).
|
|
We have also successfully operated card with correct direct attach cables instead of fiber optics. Given the card doesn't
|
|
support link training functionality of 100 Gbit/s ethernet, it could result in performance problems with copper cables, though we haven't
|
|
encountered such a situation.
|
|
|
|
## Switch configuration
|
|
Special care has to be taken for switch operation, given the FPGA core doesn't support auto-negotiation. It is necessary to configure switch port
|
|
to fixed speed (100 Gbit/s or 10 Gbit/s) and to disable auto-negotiation. It is also necessary to enable jumbo frames (MTU of 9000).
|
|
|
|
## Network LEDs
|
|
Each QSFP connector is equipped with green and orange LEDs. These LEDs are connected to Ethernet physical layer status port (rx_status).
|
|
LED on corresponds to having a physical connection to a switch/computer/detector on the other side of the network.
|
|
For 100 Gbit/s only green is used, for 8x10 Gbit/s green LEDs mean all ports connected, orange LEDs at least one of the ports connected.
|
|
|
|
## Network stack
|
|
Each Ethernet link has its own basic network stack. Functionality for Ethernet/ARP/IPv4/ICMP is therefore separately handled for each port.
|
|
Each link will get dedicated MAC address, and IPv4 addresses can be also assigned independently if needed.
|
|
|
|
The card will send gratuitous ARP messages every 5 seconds to keep its entry in switch MAC table.
|
|
The card will also reply to ARP requests for its IP and to ICMP ping requests sent with the card IPv4 address.
|
|
The card won't respond to broadcast ICMP pings.
|
|
|
|
Each link can be put in `direct` mode. In this case destination Ethernet MAC and IPv4 addresses are not enforced for incoming UDP packets.
|
|
This setting should be used for connecting detector modules directly to the FPGA card, so any detector module can be connected to any
|
|
10 Gbit/s link on the same card. Currently `direct` mode is turned OFF for `100g` design and ON for `8x10g` design.
|
|
This can be manually adjusted for each link. |