leonarski_f
dd67cffde6
The changelog section had grown to 22 entries written from the commits
rather than for a user. Collapsed to 13, each saying what is gained
before what changed, with related work merged - the six anisotropy
entries into one, five space-group entries into two, five
integration-radius entries into one - and four things added that had no
entry at all: the packaging split, the statically linked cuFFT, the Ceres
back-end drop and REPORT_VERSION reaching 3.
CPU_DATA_ANALYSIS gains 9.5, the adaptive integration radius: the r80
measurement, the clamp, the constant-ring-area r3, the convergence test,
why it applies to the final pass only and the density fallback. Three
passages there had gone stale against the code: 13.1 still described the
I/sigma quantile fallback that b90fcfb98 deleted, and 13.5 said the gate
tests delta_B when it tests delta_B_linear and that the high-symmetry
caution fires wherever the Laue class leaves one free direction, which
0da593b32 narrowed.
RUGNUX gains an Installation section - there was no page anywhere saying
where rugnux comes from - a synopsis, a worked first run naming the files
that actually appear and the report keys to grep, and sections on the
unmerged export and on diffraction anisotropy. Its report section list
was one section behind: anisotropy is 9, warnings is 10.
The packaging change reached further than the changelog implied, so the
pages describing what a release ships were corrected too: the viewer
tarball no longer carries rugnux or the two developer utilities, the
Linux archives link cuFFT statically rather than shipping it beside the
binary under an $ORIGIN rpath, the aarch64 archive has a higher glibc
floor than the RHEL 8 ones and is untested on hardware, the archives
unpack with no top-level directory, and the notices live per package
under share/doc/jfjoch_<component>. The RHEL 9.5 vm_flags workaround in
FPGA_PCIE_DRIVER is obsolete and now says so.
104 lines
5.4 KiB
Markdown
104 lines
5.4 KiB
Markdown
# FPGA PCIe driver
|
|
|
|
## Compilation
|
|
To compile kernel module type:
|
|
```
|
|
make
|
|
```
|
|
|
|
## Installation
|
|
To install kernel module, you need to have root permissions and run:
|
|
```
|
|
sudo make install
|
|
```
|
|
|
|
## Loading driver into kernel
|
|
After installing the kernel driver, it should be possible to insert it into the kernel via:
|
|
```
|
|
modprobe jfjoch
|
|
```
|
|
|
|
## Ownership of the character devices
|
|
By default, character devices `/dev/jfjoch<device number>` are owned by root (user/group) and are not accessible by others.
|
|
This means that `jfjoch_broker` must be running as superuser, which might not be optimal for security reasons in most cases.
|
|
The behavior can be changed by creating `udev` rules. Create a file called `/etc/udev/rules.d/99-jfjoch.rules`
|
|
with the following content:
|
|
```
|
|
KERNEL=="jfjoch*" OWNER="<UNIX username>" GROUP="<UNIX group>"
|
|
```
|
|
It is OK to provide only group, for example to make the devices accessible by group `jungfrau`:
|
|
```
|
|
KERNEL=="jfjoch*" GROUP="jungfrau"
|
|
```
|
|
|
|
## DKMS
|
|
To avoid problems with updating the kernel, it is possible to use DKMS to autobuild Jungfraujoch kernel
|
|
module, when new kernel is installed. For RHEL 8 it is well tested to use the RPM module built automatically from Jungfraujoch source.
|
|
For other systems, it is necessary to follow the procedure below, though it is not well tested.
|
|
|
|
This first requires to install DKMS - for RHEL it is available via EPEL repository:
|
|
```
|
|
sudo dnf install dkms
|
|
```
|
|
Then use script provided in the driver directory to copy driver code to DKMS directory:
|
|
```
|
|
./install_dkms.sh
|
|
```
|
|
If upgrading the driver, please first remove current driver from DKMS system:
|
|
```
|
|
dkms remove jfjoch -v <version> --all
|
|
```
|
|
|
|
## Driver parameters
|
|
Currently, there is one driver parameter `nbuffers`, that defines count of exchange buffers (see below).
|
|
This can be adjusted in the modprobe operation, for example:
|
|
```
|
|
modprobe jfjoch nbuffers=1024
|
|
```
|
|
|
|
## Exchange buffers
|
|
The parameter defines number of buffers used to exchange data between card and host application.
|
|
Each buffer can hold one detector module (1024x512) in 16-bit or 32-bit mode + associated processing results and metadata.
|
|
These buffers are used by both card-to-host and host-to-card operations.
|
|
|
|
Buffers use special allocation, as they are continuous in physical address space, which helps the FPGA card to transfer all
|
|
data associated with detector module in two DMA transfers (one data, one metadata).
|
|
Useful buffer size is a bit more than 2 MiB, but given that kernel allocates physical memory in power of two, **4 MiB** is safe number for one buffer size.
|
|
Buffer can be mapped into user space, but performing `mmap` system call on the `/dev/jfjoch<number of device>` character device.
|
|
|
|
Buffer count can be adjusted by setting `nbuffers` parameter. There are two considerations for setting optimal value:
|
|
1. For card-to-host transfers, minimal value is roughly
|
|
`<number of threads in receiver> * <number of modules processed by thread; usually equal to number of modules per card>`,
|
|
this way each thread can have enough data for operation. Default thread count for Jungfraujoch receiver is 64.
|
|
2. For host-to-card transfers, full detector calibration has to fit into memory and one buffer accommodates one calibration set for one module.
|
|
So minimal count is `<number of modules> * (3 + 3 * <number of storage cells>)`.
|
|
|
|
Based on both rules, optimal number is 512 buffers (2 GiB), though this can be adjusted for particular system and configuration.
|
|
|
|
## Known problems
|
|
To avoid inconsistent behavior, this driver won't load if release number differs between the kernel driver and FPGA card.
|
|
|
|
## CMake file
|
|
While CMake file is present in the driver directory, it is only for the purpose of proper detection of the files in CLion IDE.
|
|
It is not made for actual compilation of the kernel driver and should not be used for that purpose.
|
|
|
|
## Character device access
|
|
For each FPGA device a character device is created called `/dev/jfjoch<number of device>`.
|
|
When device is opened two operations are possible:
|
|
mmap() to map exchange buffers
|
|
ioctl() to communicate with the cards
|
|
Interfacing should be done through the JungfraujochDevice class in `fpga/host_library` directory.
|
|
|
|
## Sysfs access
|
|
Certain performance counters can be read through sysfs mechanism in the kernel.
|
|
One needs to `cat` files in `/sys/class/misc/jfjoch<number of device>/` directory.
|
|
|
|
## RHEL 9.5+ virtual memory flags
|
|
RedHat Enterprise Linux 9.5 backported the `vm_flags_set` interface from Linux kernel 6.3 while still reporting kernel version 5.14, so a plain kernel-version test picks the wrong branch and the build fails.
|
|
This is now detected automatically from `RHEL_RELEASE_CODE`, so the module builds unaided on RHEL 9.5 and later and on the CentOS Stream, Rocky and AlmaLinux equivalents, as well as on distributions that have not backported it.
|
|
**No user action is needed.** The `HAVE_VM_FLAGS_SET` environment variable that earlier releases required is obsolete; it is still honoured if set, but setting it is no longer necessary and the DKMS packaging never passed it anyway.
|
|
|
|
## Which kernel DKMS builds for
|
|
The DKMS package builds the module for the kernel it is being **installed for**, not the one currently running, so a module built while a kernel update is being applied loads correctly after the reboot.
|
|
Building by hand in `fpga/pcie_driver/` still defaults to the running kernel; pass `KDIR=/lib/modules/<version>/build` (or `KVER=<version>`) to target another one.
|