leonarski_f
30b6800289
Build Packages / build:windows:nocuda (push) Successful in 17m20s
Build Packages / build:windows:cuda (push) Successful in 19m52s
Build Packages / build:viewer-tgz:cpu (push) Successful in 9m38s
Build Packages / build:viewer-tgz:cuda (push) Successful in 11m18s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 9m34s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 5m42s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 20m33s
Build Packages / Create release (push) Successful in 33s
Build Packages / build:rugnux:windows (push) Successful in 12m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 15m42s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m59s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m8s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 16m55s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 16m58s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky8) (push) Successful in 15m21s
Build Packages / Build documentation (push) Successful in 54s
Build Packages / build:rpm (rocky9) (push) Successful in 16m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m2s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m6s
Build Packages / Unit tests (push) Successful in 1h10m26s
* Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
154 lines
8.2 KiB
Markdown
154 lines
8.2 KiB
Markdown
# Installing Rugnux
|
|
|
|
```{contents} On this page
|
|
:local:
|
|
:depth: 2
|
|
```
|
|
|
|
|
|
`rugnux` is a **single self-contained executable**. It needs no CUDA toolkit, no Qt, and no
|
|
Jungfraujoch service running anywhere; on a machine with an NVIDIA GPU it needs the NVIDIA
|
|
**driver**, and without one it still runs on the CPU.
|
|
|
|
## From the package repositories (RHEL / Rocky / Ubuntu)
|
|
|
|
On a distribution covered by the [package repositories](REPOSITORIES.md), `rugnux` is a package of
|
|
its own:
|
|
|
|
```
|
|
sudo dnf install rugnux # RHEL / Rocky 8 and 9
|
|
sudo apt install rugnux # Ubuntu 22.04 / 24.04
|
|
```
|
|
|
|
It installs `/usr/bin/rugnux` and depends on nothing from the acquisition side — no broker, no
|
|
detector libraries, no Qt — so it can go on a machine that only processes data.
|
|
|
|
> **Upgrading from rc.163 or earlier.** `/usr/bin/rugnux` used to belong to the `jfjoch-viewer`
|
|
> package. The `rugnux` package declares that the file has moved, so installing it upgrades an old
|
|
> `jfjoch-viewer` in the same transaction instead of failing on the duplicate path. If your
|
|
> `jfjoch-viewer` is pinned to an old version, unpin it or remove it first.
|
|
|
|
## From the release archive
|
|
|
|
For a machine no package manager covers — or for Windows and Arm, which have no repository — take
|
|
the archive for your architecture from the Gitea release page:
|
|
|
|
| Archive | For |
|
|
| --- | --- |
|
|
| `rugnux-<version>-linux-x86_64-cuda12.tgz` | 64-bit Intel/AMD Linux. Built on RHEL 8, so it runs on any newer Linux |
|
|
| `rugnux-<version>-linux-aarch64-cuda13.tgz` | 64-bit Arm Linux — NVIDIA GH200 and DGX Spark. Built on Ubuntu 24.04, so it needs glibc 2.39 or newer. Cross-compiled and **not yet exercised on Arm hardware** |
|
|
| `rugnux-<version>-win64-cuda13.zip` | 64-bit Windows |
|
|
|
|
**The archive has no top-level directory** — it unpacks straight into `bin/` and `share/`. Always
|
|
give `tar` a destination of its own, or it will scatter those into whatever directory you are in:
|
|
|
|
```
|
|
mkdir -p /opt/rugnux-1.0.0
|
|
tar xzf rugnux-1.0.0-linux-x86_64-cuda12.tgz -C /opt/rugnux-1.0.0
|
|
/opt/rugnux-1.0.0/bin/rugnux # prints the usage
|
|
```
|
|
|
|
What you get is:
|
|
|
|
```
|
|
bin/rugnux the program
|
|
share/doc/jfjoch_rugnux/LICENSE GPLv3
|
|
share/doc/jfjoch_rugnux/THIRD_PARTY_NOTICES.md
|
|
share/doc/jfjoch_rugnux/licenses/ verbatim licence texts of the bundled dependencies
|
|
```
|
|
|
|
Nothing is written outside that directory, nothing needs root, and several versions can sit side by
|
|
side. To remove it, delete the directory. Put `bin/` on your `PATH` if you want to type `rugnux`
|
|
rather than the full path.
|
|
|
|
> **Mixing the two.** If a `rugnux` package is also installed, `/usr/bin/rugnux` will normally win
|
|
> on `PATH`. Put the archive's `bin/` first, or call it by its full path, to be sure which one you
|
|
> are running — `rugnux` prints its version on every run.
|
|
|
|
## GPU support
|
|
|
|
The released archives are CUDA builds. They need only an NVIDIA **driver** on the host — 525.60.13
|
|
or newer for the CUDA 12 archive, 580.65.06 or newer for the CUDA 13 ones — and no CUDA toolkit,
|
|
because everything CUDA is linked statically. With no GPU or no driver, `rugnux` reports zero CUDA
|
|
devices and falls back to the CPU path, which works but is far slower and offers only the `fftw`
|
|
indexer. A **V100 needs the CUDA 12 archive**; which generations each build covers is in
|
|
[Release contents ▸ GPU generations and the NVIDIA driver](RELEASE_CONTENTS.md#gpu-generations-and-the-nvidia-driver).
|
|
|
|
## Building from source
|
|
|
|
`rugnux` alone, without the server stack or Qt:
|
|
|
|
```
|
|
cmake -S . -B build -DJFJOCH_RUGNUX_ONLY=ON -DCMAKE_BUILD_TYPE=Release \
|
|
-DCMAKE_CXX_FLAGS="-march=x86-64-v3" -DCMAKE_C_FLAGS="-march=x86-64-v3"
|
|
cmake --build build -j$(nproc) --target rugnux
|
|
```
|
|
|
|
The binary lands in `build/rugnux/rugnux`. No library has to come from the system: every dependency
|
|
is downloaded and built during the first configure, which therefore needs network access. `cmake --build build
|
|
--target package` produces the same `.tgz` the release ships. The `-march` flag is not set by the
|
|
build system on purpose, so a plain build is slower than the released one on the CPU-bound stages —
|
|
see the note in `CMakeLists.txt`.
|
|
|
|
## Hardware
|
|
|
|
As with the rest of Jungfraujoch, **serious performance requires an NVIDIA GPU**. The CUDA build
|
|
provides the GPU fast-feedback indexer (`ffbidx`) and the GPU FFT indexer (`fft`); without CUDA
|
|
only the CPU `fftw` indexer is available. With a GPU present most of the per-image pipeline runs on
|
|
the device — bitshuffle+LZ4 decompression, image preprocessing, azimuthal integration, spot finding,
|
|
prediction and Bragg integration — as does rotation scaling and merging, with CPU implementations as
|
|
the fallback where there is no GPU. The thread count (`-N`) governs the CPU side of all of it.
|
|
|
|
The released CUDA builds need only an NVIDIA **driver** on the host, no CUDA toolkit: 525.60.13 or
|
|
newer for the CUDA 12 artefacts (RHEL 8 packages, the x86_64 `rugnux` archive) and 580.65.06 or
|
|
newer for the CUDA 13 ones (RHEL 9, Ubuntu, the aarch64 and Windows `rugnux` archives). Which GPU
|
|
generations each artefact supports — a V100 in particular works only with the CUDA 12 build — is in
|
|
[Release contents ▸ GPU generations and the NVIDIA driver](RELEASE_CONTENTS.md#gpu-generations-and-the-nvidia-driver).
|
|
|
|
### Memory
|
|
|
|
A run's memory is set by **how many reflection observations it integrates**, not by how many pixels
|
|
the detector has. Measured on rc.169 at `-N 6`, over rotation sweeps of 1200-1800 frames:
|
|
|
|
| Detector | Observations integrated | Host (peak RSS) | Device |
|
|
| --- | --- | --- | --- |
|
|
| 1679 x 1475 (2.5 MP) | 25 M | 7.8 GB | 3.8 GB |
|
|
| 2527 x 2463 (6.2 MP) | 13 M | 4.9 GB | 3.1 GB |
|
|
| 3262 x 3108 (10.1 MP) | 7 M | 2.5 GB | 3.0 GB |
|
|
| 4371 x 4150 (18.1 MP) | 13 M | 4.3 GB | 3.2 GB |
|
|
| 4371 x 4150 (18.1 MP) | 20 M | 6.0 GB | 4.4 GB |
|
|
| 3262 x 3108 (10.1 MP) | 50 M | 14.2 GB | 7.0 GB |
|
|
|
|
Two sweeps on the same detector differ by a factor of five, so size the machine on the observation
|
|
count rather than on the detector. As a rule, **0.6 GB + 0.3 GB of host memory per million
|
|
observations, and 2.4 GB + 0.1 GB of device memory per million**. The run reports what it actually
|
|
integrated — `RotationScaleMerge: ingested ... partial observations`. **32 GB of host RAM and an
|
|
8 GB card** cover every dataset measured here at `-N 6`; an ordinary sweep needs 16 GB and 6 GB.
|
|
|
|
Both peaks fall in **scaling and merging**, not in the per-image loop: the loop alone stays under
|
|
3.2 GB on the device at `-N 6`, whatever the detector. The stage that needs the most is the 3D combine.
|
|
|
|
`-N` multiplies the per-image loop only — about **8 bytes of device memory per detector pixel per
|
|
worker** (150 MB per worker at 18 MP, 87 MB at 6 MP) on top of a fixed ~2.6 GB, and it is linear
|
|
with no ceiling: the same 18 MP sweep takes 2.7 GB of device memory at `-N 1`, 3.2 GB at `-N 6` and
|
|
8.0 GB at `-N 32`. Host memory does not move with `-N`, and neither does the merge's own device
|
|
memory. Where several runs share one card, `-N` is the knob that decides how many fit.
|
|
|
|
To use less:
|
|
|
|
- **`-N`** — the only flag that lowers the per-image loop's device memory. Lowering it is what
|
|
rescues a run on a card that is otherwise full.
|
|
- **`--no-merge`** — on the heaviest sweep measured it took the host peak from 14.2 to 8.3 GB and
|
|
the device peak from 7.0 to 2.9 GB. It also gives up the merged output.
|
|
- **`--no-export-unmerged`** does **not** save memory. `<prefix>_unmerged.mtz` is the largest file a
|
|
run writes, but its observations are resident either way; the flag saves disk, not RAM.
|
|
|
|
Nothing measures the free memory on the card before allocating. A run that does not fit degrades
|
|
where it can — device decompression falls back to the host, the beam-stop projection falls back to
|
|
the host, an indexer thread that cannot allocate reports the frame as not indexed — and **fails the
|
|
run otherwise**, with `Processing failed: CUDA (GPU) error (Failed to allocate device memory)` and a
|
|
non-zero exit status. That is deliberate: a CUDA out-of-memory says nothing about the frame in
|
|
flight and everything about the machine, so skipping frames would leave a run that looks complete
|
|
with quietly different merged numbers. A run is never silently short; it either completes or fails.
|
|
On a GPU shared with another job, lower `-N` or wait for the card.
|