159 lines
6.6 KiB
Markdown
159 lines
6.6 KiB
Markdown
# Deployment
|
|
|
|
To deploy Jungfraujoch, one needs to follow four steps:
|
|
|
|
1. Install main Jungfraujoch code and frontend web interface
|
|
2. Flash the U55C FPGA card with a proper image and install Linux kernel driver
|
|
3. Install Jungfraujoch writer
|
|
4. Install Python OpenAPI client
|
|
|
|
Installation procedure depend a lot on the operating system. For RedHat Enterprise Linux 8, Rocky 8, or compatible
|
|
installation can be done with prebuilt RPMs and is relatively straightforward. For other systems one needs to build
|
|
software from source. Both ways will be presented.
|
|
|
|
|
|
## Install main Jungfraujoch code and frontend web interface
|
|
|
|
On RHEL 8 systems there is a `jfjoch-<version>-1.el8.x86_64.rpm` that needs to be installed and contains all the necessary software and web interface.
|
|
|
|
On other OSes one needs to compile Jungfraujoch from source (from the repo directory):
|
|
```
|
|
$ mkdir build
|
|
$ cd build
|
|
$ cmake .. -DCMAKE_INSTALL_PREFIX=<directory to install>
|
|
$ make
|
|
$ sudo make install
|
|
```
|
|
For manual installation, we recommend to use non-standard directory (like `/opt/jfjoch`), to facilitate upgrades and removal.
|
|
For DKMS to manage kernel module sources it is necessary to copy driver sources to `/usr/src/jfjoch-<VERSION>` directory. This requires extra flag in cmake `-DJFJOCH_INSTALL_DRIVER_SOURCE=ON`.
|
|
|
|
Frontend web user interface has to be built separately with:
|
|
```
|
|
$ cd build
|
|
$ make frontend
|
|
```
|
|
Frontend files (.html and .js) will be placed in `frontend/dist` (outside of `build/` directory!) and has to be copied to a general location, e.g. `/usr/local/jfjoch/frontend` or `/opt/jfjoch/frotend`.
|
|
|
|
## Flash the U55C FPGA card with a proper image and install Linux kernel driver.
|
|
|
|
### Firmware flashing
|
|
1. Check that the card is detected by OS with "lspci |grep Xilinx" and check the PCIe bus/device/function (BDF) number, `11:00.0` in this case:
|
|
```
|
|
$ lspci |grep Xilinx
|
|
23:00.0 Processing accelerators: Xilinx Corporation Device 3450 (rev 2)
|
|
```
|
|
Note the device number `3450` that identifies Jungfraujoch device (Jungfraujoch pass is 3450 m above sea level) and `rev 2` identifying release of the firmware.
|
|
|
|
2. Check the speed of the card, that it is detected as PCIe Gen4x8 device (needs to be done as root, otherwise configuration details are not given):
|
|
```
|
|
$ sudo lspci -vv -s <PCIe slot number>
|
|
23:00.0 Processing accelerators: Xilinx Corporation Device 3450
|
|
(...)
|
|
LnkSta: Speed 16GT/s (ok), Width x8 (ok)
|
|
(...)
|
|
```
|
|
|
|
3. Download the MCS image from release files or build it using Vivado (WARNING! building time can be about 8 hours and doesn't allways reach correct timing).
|
|
4. Flash the card with `xbflash.qspi` tool (part of Jungfraujoch). For fresh card use:
|
|
```
|
|
sudo xbflash.qspi --primary <path to MCS file> --card <PCIe slot from above> --bar-offset 0x1f06000
|
|
```
|
|
For card that was already flashed with Jungfraujoch images:
|
|
|
|
```
|
|
sudo xbflash.qspi --primary <path to MCS file> --card <PCIe slot from above>
|
|
```
|
|
It is necessary to confirm the operation by pressing `Y` key or one can add `--force` option to avoid confirmation.
|
|
It is safe to run multiple flashing processes in parallel for different cards, for example in separate screen sessions.
|
|
|
|
5. Cold reboot:
|
|
```
|
|
sudo ipmitool chassis power cycle
|
|
```
|
|
|
|
### Install PCIe driver
|
|
|
|
For first run it is though recommended to try the driver without installing to the kernel directory:
|
|
```
|
|
$ cd fpga/pcie_driver
|
|
$ make
|
|
$ sudo insmod jfjoch.ko
|
|
```
|
|
|
|
Check with `dmesg` that the device was properly found:
|
|
```
|
|
$ dmesg |grep jfjoch
|
|
[ 431.624933] jfjoch 0000:23:00.0: enabling device (0140 -> 0142)
|
|
[ 431.919147] misc jfjoch0: Jungfraujoch FPGA loaded with FW build: 5610030a
|
|
```
|
|
|
|
If things work, it is recommended to install the driver with DKMS, so it is rebuilt for kernel updates.
|
|
On RHEL 8 you can install prebuilt RPM provided in the Gitlab package registry. On other systems follow procedure in
|
|
[PCIe driver](FPGA_PCIE_DRIVER.md).
|
|
|
|
NOTE: Driver installation procedure on non-RHEL 8 systems is not well understood/optimized at the moment.
|
|
|
|
NOTE: In case driver is included in the init RAM-disk image, it is necessary to rebuild the RAM-disk if driver is updated:
|
|
```
|
|
$ sudo dracut -f
|
|
```
|
|
### Configure network
|
|
Configure switch according to [FPGA network guide](FPGA_NETWORK.md) - specifically set manual speed and turn off auto-negotiation
|
|
for the port used to connect U55C card and connect card to switch.
|
|
|
|
### Running Jungfraujoch software
|
|
Main Jungfraujoch service is called `jfjoch_broker`. It is responsible for handling data from FPGAs, doing processing, analysis, compression and sending images on ZeroMQ output.
|
|
It is recommended to run the service as `systemd` service.
|
|
|
|
`jfjoch_broker` takes two parameters: JSON configuration file and HTTP port (default is 5232).
|
|
Example JSON files are placed in `etc/` folder. JSON file format is also explained in the OpenAPI definition, as `jfjoch_settings` data structure.
|
|
|
|
When running the service can be accessed via HTTP interface from a web browser for configuration and monitoring.
|
|
|
|
To prepare the configuration file one also needs to reference calibration files: gain files for PSI JUNGFRAU and trim-bit files for PSI EIGER.
|
|
These need to be obtained from the PSI Detector Group.
|
|
|
|
### Card verification
|
|
|
|
To test that FPGA board is working properly without access to a JUNGFRAU detector, you can use `jfjoch_fpga_test` tool.
|
|
For example to simulate 10M pixel system with 4 FPGA cards and 200k images on a 2 CPU system with 2 GPUs:
|
|
```
|
|
jfjoch_fpga_test ~/nextgendcu/ -m20 -s4 -i 200000 -Pn2g2
|
|
```
|
|
Or 1M pixel system with one FPGA card:
|
|
```
|
|
jfjoch_fpga_test ~/nextgendcu/ -m2 -s1 -i 200000
|
|
```
|
|
|
|
## Install Jungfraujoch writer
|
|
Jungfraujoch writer is an additional service, that can connect to `jfjoch_broker` ZeroMQ interface and writes files according to NeXus/NXmx HDF5 standard.
|
|
|
|
At the moment it is better to have a separate machine, with access to distributed file system, for writing images.
|
|
|
|
Writer can be installed with a dedicated RPM file or compiled from source. For compilation, you can use the following commands:
|
|
```
|
|
mkdir build
|
|
cd build
|
|
cmake -DJFJOCH_WRITER_ONLY=ON -DCMAKE_INSTALL_PREFIX=<directory to install> ..
|
|
make jfjoch
|
|
```
|
|
|
|
## Install Jungfraujoch image viewer
|
|
Jungfraujoch viewer is X-ray diffraction image viewer, that is optimized to open Jungfraujoch HDF5 files.
|
|
|
|
The viewer is a Qt application and it requires recent version of the library, therefore it is an optional dependency.
|
|
|
|
To include it in the building of Jungfraujoch use `-DJFJOCH_VIEWER_BUILD=ON` directive for CMake:
|
|
```
|
|
mkdir build
|
|
cd build
|
|
cmake -DJFJOCH_VIEWER_BUILD=ON -DCMAKE_INSTALL_PREFIX=<directory to install> ..
|
|
make jfjoch
|
|
```
|
|
|
|
|
|
## Install Jungfraujoch Python client
|
|
Use pip:
|
|
```shell
|
|
pip install jfjoch-client
|
|
``` |