Jungfraujoch/docs/FPGA_PCIE_DRIVER.md

FPGA PCIe driver

Compilation

To compile kernel module type:

make

Installation

To install kernel module, you need to have root permissions and run:

sudo make install

Loading driver into kernel

After installing the kernel driver, it should be possible to insert it into the kernel via:

modprobe jfjoch

Ownership of the character devices

By default, character devices /dev/jfjoch<device number> are owned by root (user/group) and are not accessible by others. This means that jfjoch_broker must be running as superuser, which might not be optimal for security reasons in most cases. The behavior can be changed by creating udev rules. Create a file called /etc/udev/rules.d/99-jfjoch.rules with the following content:

KERNEL=="jfjoch*" OWNER="<UNIX username>" GROUP="<UNIX group>"

It is OK to provide only group, for example to make the devices accessible by group jungfrau:

KERNEL=="jfjoch*" GROUP="jungfrau"

DKMS

To avoid problems with updating the kernel, it is possible to use DKMS to autobuild Jungfraujoch kernel module, when new kernel is installed. For RHEL 8 it is well tested to use the RPM module built automatically from Jungfraujoch source. For other systems, it is necessary to follow the procedure below, though it is not well tested.

This first requires to install DKMS - for RHEL it is available via EPEL repository:

sudo dnf install dkms

Then use script provided in the driver directory to copy driver code to DKMS directory:

./install_dkms.sh

If upgrading the driver, please first remove current driver from DKMS system:

dkms remove jfjoch -v <version> --all

Driver parameters

Currently, there is one driver parameter nbuffers, that defines count of exchange buffers (see below). This can be adjusted in the modprobe operation, for example:

modprobe jfjoch nbuffers=1024

Exchange buffers

The parameter defines number of buffers used to exchange data between card and host application. Each buffer can hold one detector module (1024x512) in 16-bit or 32-bit mode + associated processing results and metadata. These buffers are used by both card-to-host and host-to-card operations.

Buffers use special allocation, as they are continuous in physical address space, which helps the FPGA card to transfer all data associated with detector module in two DMA transfers (one data, one metadata). Useful buffer size is a bit more than 2 MiB, but given that kernel allocates physical memory in power of two, 4 MiB is safe number for one buffer size. Buffer can be mapped into user space, but performing mmap system call on the /dev/jfjoch<number of device> character device.

Buffer count can be adjusted by setting nbuffers parameter. There are two considerations for setting optimal value:

1. For card-to-host transfers, minimal value is roughly <number of threads in receiver> * <number of modules processed by thread; usually equal to number of modules per card>, this way each thread can have enough data for operation. Default thread count for Jungfraujoch receiver is 64.
2. For host-to-card transfers, full detector calibration has to fit into memory and one buffer accommodates one calibration set for one module. So minimal count is <number of modules> * (3 + 3 * <number of storage cells>).

Based on both rules, optimal number is 512 buffers (2 GiB), though this can be adjusted for particular system and configuration.

Known problems

To avoid inconsistent behavior, this driver won't load if release number differs between the kernel driver and FPGA card.

CMake file

While CMake file is present in the driver directory, it is only for the purpose of proper detection of the files in CLion IDE. It is not made for actual compilation of the kernel driver and should not be used for that purpose.

Character device access

For each FPGA device a character device is created called /dev/jfjoch<number of device>. When device is opened two operations are possible: mmap() to map exchange buffers ioctl() to communicate with the cards Interfacing should be done through the JungfraujochDevice class in fpga/host_library directory.

Sysfs access

Certain performance counters can be read through sysfs mechanism in the kernel. One needs to cat files in /sys/class/misc/jfjoch<number of device>/ directory.

RHEL 9.5+ issue

RedHat Enterprise Linux 9.5 backported modification to settings virtual memory flags from Linux kernel 6.3, while still operating kernel version 5.14. It is complicated to come up with a single rule to select when newer functions should be used, so it works with RHEL 9.5+, while still being compatible with other Linux distributions. It is even more complex given not all RHEL compatible distributions adopted the change at the same version. For the moment the quick fix is to define an environment variable HAVE_VM_FLAGS_SET before making the kernel.