101 lines
4.8 KiB
Markdown
101 lines
4.8 KiB
Markdown
# FPGA PCIe driver
|
|
|
|
## Compilation
|
|
To compile kernel module type:
|
|
```
|
|
make
|
|
```
|
|
|
|
## Installation
|
|
To install kernel module, you need to have root permissions and run:
|
|
```
|
|
sudo make install
|
|
```
|
|
|
|
## Loading driver into kernel
|
|
After installing the kernel driver, it should be possible to insert it into the kernel via:
|
|
```
|
|
modprobe jfjoch
|
|
```
|
|
|
|
## Ownership of the character devices
|
|
By default, character devices `/dev/jfjoch<device number>` are owned by root (user/group) and are not accessible by others.
|
|
This means that `jfjoch_broker` must be running as superuser, which might not be optimal for security reasons in most cases.
|
|
The behavior can be changed by creating `udev` rules. Create a file called `/etc/udev/rules.d/99-jfjoch.rules`
|
|
with the following content:
|
|
```
|
|
KERNEL=="jfjoch*" OWNER="<UNIX username>" GROUP="<UNIX group>"
|
|
```
|
|
It is OK to provide only group, for example to make the devices accessible by group `jungfrau`:
|
|
```
|
|
KERNEL=="jfjoch*" GROUP="jungfrau"
|
|
```
|
|
|
|
## DKMS
|
|
To avoid problems with updating the kernel, it is possible to use DKMS to autobuild Jungfraujoch kernel
|
|
module, when new kernel is installed. For RHEL 8 it is well tested to use the RPM module built automatically from Jungfraujoch source.
|
|
For other systems, it is necessary to follow the procedure below, though it is not well tested.
|
|
|
|
This first requires to install DKMS - for RHEL it is available via EPEL repository:
|
|
```
|
|
sudo dnf install dkms
|
|
```
|
|
Then use script provided in the driver directory to copy driver code to DKMS directory:
|
|
```
|
|
./install_dkms.sh
|
|
```
|
|
If upgrading the driver, please first remove current driver from DKMS system:
|
|
```
|
|
dkms remove jfjoch -v <version> --all
|
|
```
|
|
|
|
## Driver parameters
|
|
Currently, there is one driver parameter `nbuffers`, that defines count of exchange buffers (see below).
|
|
This can be adjusted in the modprobe operation, for example:
|
|
```
|
|
modprobe jfjoch nbuffers=1024
|
|
```
|
|
|
|
## Exchange buffers
|
|
The parameter defines number of buffers used to exchange data between card and host application.
|
|
Each buffer can hold one detector module (1024x512) in 16-bit or 32-bit mode + associated processing results and metadata.
|
|
These buffers are used by both card-to-host and host-to-card operations.
|
|
|
|
Buffers use special allocation, as they are continuous in physical address space, which helps the FPGA card to transfer all
|
|
data associated with detector module in two DMA transfers (one data, one metadata).
|
|
Useful buffer size is a bit more than 2 MiB, but given that kernel allocates physical memory in power of two, **4 MiB** is safe number for one buffer size.
|
|
Buffer can be mapped into user space, but performing `mmap` system call on the `/dev/jfjoch<number of device>` character device.
|
|
|
|
Buffer count can be adjusted by setting `nbuffers` parameter. There are two considerations for setting optimal value:
|
|
1. For card-to-host transfers, minimal value is roughly
|
|
`<number of threads in receiver> * <number of modules processed by thread; usually equal to number of modules per card>`,
|
|
this way each thread can have enough data for operation. Default thread count for Jungfraujoch receiver is 64.
|
|
2. For host-to-card transfers, full detector calibration has to fit into memory and one buffer accommodates one calibration set for one module.
|
|
So minimal count is `<number of modules> * (3 + 3 * <number of storage cells>)`.
|
|
|
|
Based on both rules, optimal number is 512 buffers (2 GiB), though this can be adjusted for particular system and configuration.
|
|
|
|
## Known problems
|
|
To avoid inconsistent behavior, this driver won't load if release number differs between the kernel driver and FPGA card.
|
|
|
|
## CMake file
|
|
While CMake file is present in the driver directory, it is only for the purpose of proper detection of the files in CLion IDE.
|
|
It is not made for actual compilation of the kernel driver and should not be used for that purpose.
|
|
|
|
## Character device access
|
|
For each FPGA device a character device is created called `/dev/jfjoch<number of device>`.
|
|
When device is opened two operations are possible:
|
|
mmap() to map exchange buffers
|
|
ioctl() to communicate with the cards
|
|
Interfacing should be done through the JungfraujochDevice class in `fpga/host_library` directory.
|
|
|
|
## Sysfs access
|
|
Certain performance counters can be read through sysfs mechanism in the kernel.
|
|
One needs to `cat` files in `/sys/class/misc/jfjoch<number of device>/` directory.
|
|
|
|
## RHEL 9.5+ issue
|
|
RedHat Enterprise Linux 9.5 backported modification to settings virtual memory flags from Linux kernel 6.3, while still operating kernel version 5.14.
|
|
It is complicated to come up with a single rule to select when newer functions should be used, so it works with RHEL 9.5+,
|
|
while still being compatible with other Linux distributions. It is even more complex given not all RHEL compatible distributions adopted the change at the same version.
|
|
For the moment the quick fix is to define an environment variable `HAVE_VM_FLAGS_SET` before making the kernel.
|