leonarski_fandClaude Fable 5
70c4d871b6
- Capitalize Rugnux as a proper noun throughout the prose; the command stays lowercase `rugnux` in code font. RUGNUX_OVERVIEW.md is retitled "What Rugnux does". - ACKNOWLEDGEMENT.md cites the raw-data repositories only: dataset counts and DOI prefixes moved out (EXTERNAL_TEST_DATA.md owns them), the ESRF data portal gains its citation (Dimper et al. 2019), and MXRDR remains name + link - it has no canonical citation paper. - RUGNUX_FORMATS.md: the CCD formats (marCCD, SMV) are supported as-is with very limited scope, and per-panel XFEL data is not read. - Fix wrong facts a reader would act on: nonexistent `make jfjoch` targets, invalid udev rules, PUSH sockets documented as PULL, swapped writer width/height, underload semantics, the transposed pixel-mask numpy example (the server checks width and height separately), the Durin/Neggia mask-bit table, FPGA threshold register addresses and the mailbox bit field, the I2C core's document number (PG090), an inverted MODEL_FIT_SIGMA formula, a self-inconsistent worked report example, and 11 cross-page anchors whose slugs carry MyST section numbers. - Unify CC1/2 spelling in prose; math notation and report keys unchanged. - Sweep grammar, typos and editing residue across the FPGA, deployment, streaming and analysis pages, including historical CHANGELOG typos. - rugnux_cli.cpp: the -S usage/error examples pair 96 with P43212; 92 names a different group. - Root THIRD_PARTY_NOTICES.md: scope the GPL-compatibility claim (CUDA EULA) and the vendored-table intro (traccc); the docs copy regenerates via update_version.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
104 lines
5.4 KiB
Markdown
104 lines
5.4 KiB
Markdown
# FPGA PCIe driver
|
|
|
|
## Compilation
|
|
To compile the kernel module, type:
|
|
```
|
|
make
|
|
```
|
|
|
|
## Installation
|
|
To install kernel module, you need to have root permissions and run:
|
|
```
|
|
sudo make install
|
|
```
|
|
|
|
## Loading driver into kernel
|
|
After installing the kernel driver, it should be possible to insert it into the kernel via:
|
|
```
|
|
modprobe jfjoch
|
|
```
|
|
|
|
## Ownership of the character devices
|
|
By default, character devices `/dev/jfjoch<device number>` are owned by root (user/group) and are not accessible by others.
|
|
This means that `jfjoch_broker` must be running as superuser, which might not be optimal for security reasons in most cases.
|
|
The behavior can be changed by creating `udev` rules. Create a file called `/etc/udev/rules.d/99-jfjoch.rules`
|
|
with the following content:
|
|
```
|
|
KERNEL=="jfjoch*", OWNER="<UNIX username>", GROUP="<UNIX group>"
|
|
```
|
|
It is OK to provide only group, for example to make the devices accessible by group `jungfrau`:
|
|
```
|
|
KERNEL=="jfjoch*", GROUP="jungfrau"
|
|
```
|
|
|
|
## DKMS
|
|
To avoid problems with updating the kernel, it is possible to use DKMS to autobuild Jungfraujoch kernel
|
|
module, when new kernel is installed. For RHEL 8 it is well tested to use the RPM module built automatically from Jungfraujoch source.
|
|
For other systems, it is necessary to follow the procedure below, though it is not well tested.
|
|
|
|
This first requires installing DKMS - for RHEL it is available via EPEL repository:
|
|
```
|
|
sudo dnf install dkms
|
|
```
|
|
Then use the script provided in the driver directory to copy driver code to DKMS directory:
|
|
```
|
|
./install_dkms.sh
|
|
```
|
|
If upgrading the driver, please first remove the current driver from DKMS system:
|
|
```
|
|
dkms remove jfjoch -v <version> --all
|
|
```
|
|
|
|
## Driver parameters
|
|
Currently, there is one driver parameter `nbuffers`, that defines count of exchange buffers (see below).
|
|
This can be adjusted in the modprobe operation, for example:
|
|
```
|
|
modprobe jfjoch nbuffers=1024
|
|
```
|
|
|
|
## Exchange buffers
|
|
The parameter defines number of buffers used to exchange data between card and host application.
|
|
Each buffer can hold one detector module (1024x512) in 16-bit or 32-bit mode + associated processing results and metadata.
|
|
These buffers are used by both card-to-host and host-to-card operations.
|
|
|
|
Buffers use special allocation, as they are contiguous in physical address space, which helps the FPGA card to transfer all
|
|
data associated with detector module in two DMA transfers (one data, one metadata).
|
|
Useful buffer size is a bit more than 2 MiB, but given that kernel allocates physical memory in powers of two, **4 MiB** is a safe number for one buffer size.
|
|
A buffer can be mapped into user space by performing the `mmap` system call on the `/dev/jfjoch<device number>` character device.
|
|
|
|
Buffer count can be adjusted by setting `nbuffers` parameter. There are two considerations for setting optimal value:
|
|
1. For card-to-host transfers, minimal value is roughly
|
|
`<number of threads in receiver> * <number of modules processed by thread; usually equal to number of modules per card>`,
|
|
this way each thread can have enough data for operation. Default thread count for Jungfraujoch receiver is 64.
|
|
2. For host-to-card transfers, full detector calibration has to fit into memory and one buffer accommodates one calibration set for one module.
|
|
So minimal count is `<number of modules> * (3 + 3 * <number of storage cells>)`.
|
|
|
|
Based on both rules, optimal number is 512 buffers (2 GiB), though this can be adjusted for particular system and configuration.
|
|
|
|
## Known problems
|
|
To avoid inconsistent behavior, this driver won't load if release number differs between the kernel driver and FPGA card.
|
|
|
|
## CMake file
|
|
While CMake file is present in the driver directory, it is only for the purpose of proper detection of the files in CLion IDE.
|
|
It is not made for actual compilation of the kernel driver and should not be used for that purpose.
|
|
|
|
## Character device access
|
|
For each FPGA device a character device is created called `/dev/jfjoch<device number>`.
|
|
When the device is opened, two operations are possible:
|
|
* `mmap()` to map exchange buffers
|
|
* `ioctl()` to communicate with the card
|
|
Interfacing should be done through the JungfraujochDevice class in `fpga/host_library` directory.
|
|
|
|
## Sysfs access
|
|
Certain performance counters can be read through sysfs mechanism in the kernel.
|
|
One needs to `cat` files in `/sys/class/misc/jfjoch<device number>/` directory.
|
|
|
|
## RHEL 9.5+ virtual memory flags
|
|
RedHat Enterprise Linux 9.5 backported the `vm_flags_set` interface from Linux kernel 6.3 while still reporting kernel version 5.14, so a plain kernel-version test picks the wrong branch and the build fails.
|
|
This is now detected automatically from `RHEL_RELEASE_CODE`, so the module builds unaided on RHEL 9.5 and later and on the CentOS Stream, Rocky and AlmaLinux equivalents, as well as on distributions that have not backported it.
|
|
**No user action is needed.** The `HAVE_VM_FLAGS_SET` environment variable that earlier releases required is obsolete; it is still honoured if set, but setting it is no longer necessary and the DKMS packaging never passed it anyway.
|
|
|
|
## Which kernel DKMS builds for
|
|
The DKMS package builds the module for the kernel it is being **installed for**, not the one currently running, so a module built while a kernel update is being applied loads correctly after the reboot.
|
|
Building by hand in `fpga/pcie_driver/` still defaults to the running kernel; pass `KDIR=/lib/modules/<version>/build` (or `KVER=<version>`) to target another one.
|