leonarski_f
749db470ca
* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
* The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count.
* `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
* `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so.
* The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.
Reviewed-on: #74
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
8.6 KiB
Deployment
To deploy Jungfraujoch, one needs to follow four steps:
- Install main Jungfraujoch code and frontend web interface
- Flash the U55C FPGA card with a proper image and install Linux kernel driver
- Install Jungfraujoch writer
- Install Python OpenAPI client
rugnux, the offline analysis tool, is installed separately and independently of all
four — see Install rugnux at the end of this page.
Installation procedure depend a lot on the operating system. For RedHat Enterprise Linux 8/9, Rocky 8/9, Ubuntu 22.04/24.04 or compatible, installation can be done with prebuilt packages from the package repositories and is relatively straightforward. For other systems one needs to build software from source. Both ways will be presented. What each released package contains, and what it needs on the machine, is described in Release contents.
Install main Jungfraujoch code and frontend web interface
On RHEL 8 systems there is a jfjoch-<version>-1.el8.x86_64.rpm that needs to be installed and contains all the necessary software and web interface.
On other OSes one needs to compile Jungfraujoch from source (from the repo directory):
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=<directory to install>
$ make
$ sudo make install
For manual installation, we recommend to use non-standard directory (like /opt/jfjoch), to facilitate upgrades and removal.
For DKMS to manage kernel module sources it is necessary to copy driver sources to /usr/src/jfjoch-<VERSION> directory. This requires extra flag in cmake -DJFJOCH_INSTALL_DRIVER_SOURCE=ON.
Frontend web user interface has to be built separately with:
$ cd build
$ make frontend
Frontend files (.html and .js) will be placed in frontend/dist (outside of build/ directory!) and has to be copied to a general location, e.g. /usr/local/jfjoch/frontend or /opt/jfjoch/frotend.
Flash the U55C FPGA card with a proper image and install Linux kernel driver.
Firmware flashing
- Check that the card is detected by OS with "lspci |grep Xilinx" and check the PCIe bus/device/function (BDF) number,
11:00.0in this case:
$ lspci |grep Xilinx
23:00.0 Processing accelerators: Xilinx Corporation Device 3450 (rev 2)
Note the device number 3450 that identifies Jungfraujoch device (Jungfraujoch pass is 3450 m above sea level) and rev 2 identifying release of the firmware.
- Check the speed of the card, that it is detected as PCIe Gen4x8 device (needs to be done as root, otherwise configuration details are not given):
$ sudo lspci -vv -s <PCIe slot number>
23:00.0 Processing accelerators: Xilinx Corporation Device 3450
(...)
LnkSta: Speed 16GT/s (ok), Width x8 (ok)
(...)
- Download the MCS image from release files or build it using Vivado (WARNING! building time can be about 8 hours and doesn't allways reach correct timing).
- Flash the card with
xbflash.qspitool (part of Jungfraujoch). For fresh card use:
sudo xbflash.qspi --primary <path to MCS file> --card <PCIe slot from above> --bar-offset 0x1f06000
For card that was already flashed with Jungfraujoch images:
sudo xbflash.qspi --primary <path to MCS file> --card <PCIe slot from above>
It is necessary to confirm the operation by pressing Y key or one can add --force option to avoid confirmation.
It is safe to run multiple flashing processes in parallel for different cards, for example in separate screen sessions.
- Cold reboot:
sudo ipmitool chassis power cycle
Install PCIe driver
For first run it is though recommended to try the driver without installing to the kernel directory:
$ cd fpga/pcie_driver
$ make
$ sudo insmod jfjoch.ko
Check with dmesg that the device was properly found:
$ dmesg |grep jfjoch
[ 431.624933] jfjoch 0000:23:00.0: enabling device (0140 -> 0142)
[ 431.919147] misc jfjoch0: Jungfraujoch FPGA loaded with FW build: 5610030a
If things work, it is recommended to install the driver with DKMS, so it is rebuilt for kernel updates.
Install the prebuilt jfjoch-driver-dkms package from the
Gitea package registry; on other systems follow the procedure in
PCIe driver.
DKMS builds the module for the kernel it is being installed for rather than the running one, so a
module built during a kernel update loads correctly after the reboot. RHEL 9.5 and later — and their
CentOS Stream, Rocky and AlmaLinux equivalents — build unaided; the HAVE_VM_FLAGS_SET workaround
earlier releases needed is obsolete.
NOTE: In case driver is included in the init RAM-disk image, it is necessary to rebuild the RAM-disk if driver is updated:
$ sudo dracut -f
Configure network
Configure switch according to FPGA network guide - specifically set manual speed and turn off auto-negotiation for the port used to connect U55C card and connect card to switch.
Running Jungfraujoch software
Main Jungfraujoch service is called jfjoch_broker. It is responsible for handling data from FPGAs, doing processing, analysis, compression and sending images on ZeroMQ output.
It is recommended to run the service as systemd service.
jfjoch_broker takes two parameters: JSON configuration file and HTTP port (default is 5232).
Example JSON files are placed in etc/ folder. JSON file format is also explained in the OpenAPI definition, as jfjoch_settings data structure.
When running the service can be accessed via HTTP interface from a web browser for configuration and monitoring.
Jungfraujoch automatically uses every GPU visible to the process and spreads the per-image work across all of them. To run more than one jfjoch_broker on a single machine, each confined to a disjoint subset of GPUs, set CUDA_VISIBLE_DEVICES; setting CUDA_DEVICE_ORDER=PCI_BUS_ID keeps the GPU indices stable across reboots. For example, two brokers on a 4-GPU host:
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 jfjoch_broker broker_a.json 5232
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=2,3 jfjoch_broker broker_b.json 5233
To prepare the configuration file one also needs to reference calibration files: gain files for PSI JUNGFRAU and trim-bit files for PSI EIGER. These need to be obtained from the PSI Detector Group.
Card verification
To test that FPGA board is working properly without access to a JUNGFRAU detector, you can use jfjoch_fpga_test tool.
For example to simulate 10M pixel system with 4 FPGA cards and 200k images on a 2 CPU system with 2 GPUs:
jfjoch_fpga_test ~/nextgendcu/ -m20 -s4 -i 200000
Or 1M pixel system with one FPGA card:
jfjoch_fpga_test ~/nextgendcu/ -m2 -s1 -i 200000
Install Jungfraujoch writer
Jungfraujoch writer is an additional service, that can connect to jfjoch_broker ZeroMQ interface and writes files according to NeXus/NXmx HDF5 standard.
At the moment it is better to have a separate machine, with access to distributed file system, for writing images.
Writer can be installed with a dedicated RPM file or compiled from source. For compilation, you can use the following commands:
mkdir build
cd build
cmake -DJFJOCH_WRITER_ONLY=ON -DCMAKE_INSTALL_PREFIX=<directory to install> ..
make jfjoch
Install Jungfraujoch image viewer
Jungfraujoch viewer is X-ray diffraction image viewer, that is optimized to open Jungfraujoch HDF5 files.
The viewer is a Qt application and it requires recent version of the library, therefore it is an optional dependency.
To include it in the building of Jungfraujoch use -DJFJOCH_VIEWER_BUILD=ON directive for CMake:
mkdir build
cd build
cmake -DJFJOCH_VIEWER_BUILD=ON -DCMAKE_INSTALL_PREFIX=<directory to install> ..
make jfjoch
Install Jungfraujoch Python client
Use pip:
pip install jfjoch-client
Install rugnux (offline analysis)
rugnux is not part of the server stack and is installed independently of all of the above. It
needs neither the broker, the writer, Qt nor a CUDA toolkit — only an NVIDIA driver if you want to
use the GPU — and it does not have to run on the acquisition machine at all.
From the package repositories:
sudo dnf install rugnux # RHEL / Rocky
sudo apt install rugnux # Ubuntu
Or, on a machine no repository covers, from the standalone archive:
mkdir -p /opt/rugnux-<version>
tar xzf rugnux-<version>-linux-x86_64-cuda12.tgz -C /opt/rugnux-<version>
/opt/rugnux-<version>/bin/rugnux
The archive has no top-level directory, so the -C is required. See
rugnux ▸ Installation for the Arm and Windows archives, the driver
versions and building from source.