Jungfraujoch/docs/DEPLOYMENT.md

# Deployment

To deploy Jungfraujoch, one needs to follow four steps:

1. Install main Jungfraujoch code and frontend web interface
2. Flash the U55C FPGA card with a proper image and install Linux kernel driver
3. Install Jungfraujoch writer
4. Install Python OpenAPI client

[`rugnux`](RUGNUX.md), the offline analysis tool, is installed separately and independently of all
four — see [Install rugnux](#install-rugnux-offline-analysis) at the end of this page.

Installation procedure depend a lot on the operating system. For RedHat Enterprise Linux 8/9, Rocky 8/9,
Ubuntu 22.04/24.04 or compatible, installation can be done with prebuilt packages from the
[package repositories](REPOSITORIES.md) and is relatively straightforward. For other systems one needs
to build software from source. Both ways will be presented. What each released package contains, and
what it needs on the machine, is described in [Release contents](RELEASE_CONTENTS.md).


## Install main Jungfraujoch code and frontend web interface

On RHEL 8 systems there is a `jfjoch-<version>-1.el8.x86_64.rpm` that needs to be installed and contains all the necessary software and web interface.

On other OSes one needs to compile Jungfraujoch from source (from the repo directory):
```
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=<directory to install>
$ make
$ sudo make install  
```
For manual installation, we recommend to use non-standard directory (like `/opt/jfjoch`), to facilitate upgrades and removal.
For DKMS to manage kernel module sources it is necessary to copy driver sources to `/usr/src/jfjoch-<VERSION>` directory. This requires extra flag in cmake `-DJFJOCH_INSTALL_DRIVER_SOURCE=ON`.

Frontend web user interface has to be built separately with:
```
$ cd build
$ make frontend
```
Frontend files (.html and .js) will be placed in `frontend/dist` (outside of `build/` directory!) and has to be copied to a general location, e.g. `/usr/local/jfjoch/frontend` or `/opt/jfjoch/frotend`.

## Flash the U55C FPGA card with a proper image and install Linux kernel driver.

### Firmware flashing
1. Check that the card is detected by OS with "lspci |grep Xilinx" and check the PCIe bus/device/function (BDF) number, `11:00.0` in this case:
```
$ lspci |grep Xilinx
23:00.0 Processing accelerators: Xilinx Corporation Device 3450 (rev 2)
```
Note the device number `3450` that identifies Jungfraujoch device (Jungfraujoch pass is 3450 m above sea level) and `rev 2` identifying release of the firmware.

2. Check the speed of the card, that it is detected as PCIe Gen4x8 device (needs to be done as root, otherwise configuration details are not given):
```
$ sudo lspci -vv -s <PCIe slot number>
23:00.0 Processing accelerators: Xilinx Corporation Device 3450
(...)
LnkSta:     Speed 16GT/s (ok), Width x8 (ok)
(...)
```

3. Download the MCS image from release files or build it using Vivado (WARNING! building time can be about 8 hours and doesn't allways reach correct timing).
4. Flash the card with `xbflash.qspi` tool (part of Jungfraujoch). For fresh card use:
```
sudo xbflash.qspi --primary <path to MCS file> --card <PCIe slot from above> --bar-offset 0x1f06000 
```
For card that was already flashed with Jungfraujoch images:

```
sudo xbflash.qspi --primary <path to MCS file> --card <PCIe slot from above>
```
It is necessary to confirm the operation by pressing `Y` key or one can add `--force` option to avoid confirmation.
It is safe to run multiple flashing processes in parallel for different cards, for example in separate screen sessions.

5. Cold reboot:
```
sudo ipmitool chassis power cycle
```

### Install PCIe driver

For first run it is though recommended to try the driver without installing to the kernel directory:
```
$ cd fpga/pcie_driver
$ make
$ sudo insmod jfjoch.ko
```

Check with `dmesg` that the device was properly found:
```
$ dmesg |grep jfjoch
[  431.624933] jfjoch 0000:23:00.0: enabling device (0140 -> 0142)
[  431.919147] misc jfjoch0: Jungfraujoch FPGA loaded with FW build: 5610030a
```

If things work, it is recommended to install the driver with DKMS, so it is rebuilt for kernel updates.
Install the prebuilt `jfjoch-driver-dkms` package from the
[Gitea package registry](REPOSITORIES.md); on other systems follow the procedure in
[PCIe driver](FPGA_PCIE_DRIVER.md).

DKMS builds the module for the kernel it is being installed for rather than the running one, so a
module built during a kernel update loads correctly after the reboot. RHEL 9.5 and later — and their
CentOS Stream, Rocky and AlmaLinux equivalents — build unaided; the `HAVE_VM_FLAGS_SET` workaround
earlier releases needed is obsolete.

NOTE: In case driver is included in the init RAM-disk image, it is necessary to rebuild the RAM-disk if driver is updated:
```
$ sudo dracut -f
```
### Configure network
Configure switch according to [FPGA network guide](FPGA_NETWORK.md) - specifically set manual speed and turn off auto-negotiation
for the port used to connect U55C card and connect card to switch.

### Running Jungfraujoch software
Main Jungfraujoch service is called `jfjoch_broker`. It is responsible for handling data from FPGAs, doing processing, analysis, compression and sending images on ZeroMQ output.
It is recommended to run the service as `systemd` service.

`jfjoch_broker` takes two parameters: JSON configuration file and HTTP port (default is 5232). 
Example JSON files are placed in `etc/` folder. JSON file format is also explained in the OpenAPI definition, as `jfjoch_settings` data structure.

When running the service can be accessed via HTTP interface from a web browser for configuration and monitoring.

Jungfraujoch automatically uses every GPU visible to the process and spreads the per-image work across all of them. To run more than one `jfjoch_broker` on a single machine, each confined to a disjoint subset of GPUs, set `CUDA_VISIBLE_DEVICES`; setting `CUDA_DEVICE_ORDER=PCI_BUS_ID` keeps the GPU indices stable across reboots. For example, two brokers on a 4-GPU host:
```
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 jfjoch_broker broker_a.json 5232
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=2,3 jfjoch_broker broker_b.json 5233
```

To prepare the configuration file one also needs to reference calibration files: gain files for PSI JUNGFRAU and trim-bit files for PSI EIGER. 
These need to be obtained from the PSI Detector Group.

### Card verification

To test that FPGA board is working properly without access to a JUNGFRAU detector, you can use `jfjoch_fpga_test` tool.
For example to simulate 10M pixel system with 4 FPGA cards and 200k images on a 2 CPU system with 2 GPUs:
```
jfjoch_fpga_test ~/nextgendcu/ -m20 -s4 -i 200000
```
Or 1M pixel system with one FPGA card:
```
jfjoch_fpga_test ~/nextgendcu/ -m2 -s1 -i 200000
```

## Install Jungfraujoch writer
Jungfraujoch writer is an additional service, that can connect to `jfjoch_broker` ZeroMQ interface and writes files according to NeXus/NXmx HDF5 standard.

At the moment it is better to have a separate machine, with access to distributed file system, for writing images.

Writer can be installed with a dedicated RPM file or compiled from source. For compilation, you can use the following commands:
```
mkdir build
cd build
cmake -DJFJOCH_WRITER_ONLY=ON -DCMAKE_INSTALL_PREFIX=<directory to install> ..
make jfjoch
```

## Install Jungfraujoch image viewer
Jungfraujoch viewer is X-ray diffraction image viewer, that is optimized to open Jungfraujoch HDF5 files.

The viewer is a Qt application and it requires recent version of the library, therefore it is an optional dependency.

To include it in the building of Jungfraujoch use `-DJFJOCH_VIEWER_BUILD=ON` directive for CMake:
```
mkdir build
cd build
cmake -DJFJOCH_VIEWER_BUILD=ON -DCMAKE_INSTALL_PREFIX=<directory to install> ..
make jfjoch
```


## Install Jungfraujoch Python client
Use pip:
```shell
pip install jfjoch-client
```

## Install rugnux (offline analysis)

`rugnux` is not part of the server stack and is installed independently of all of the above. It
needs neither the broker, the writer, Qt nor a CUDA toolkit — only an NVIDIA driver if you want to
use the GPU — and it does not have to run on the acquisition machine at all.

From the [package repositories](REPOSITORIES.md):

```
sudo dnf install rugnux          # RHEL / Rocky
sudo apt install rugnux          # Ubuntu
```

Or, on a machine no repository covers, from the standalone archive:

```
mkdir -p /opt/rugnux-<version>
tar xzf rugnux-<version>-linux-x86_64-cuda12.tgz -C /opt/rugnux-<version>
/opt/rugnux-<version>/bin/rugnux
```

The archive has no top-level directory, so the `-C` is required. See
[rugnux ▸ Installation](RUGNUX.md#installation) for the Arm and Windows archives, the driver
versions and building from source.