Jungfraujoch/docs/RUGNUX.md

# rugnux

`rugnux` is the **offline** crystallographic data-analysis tool of Jungfraujoch — the
data-processing half of the system (see [Naming](NAMING.md) for where the name comes from).
It takes an existing HDF5 dataset, runs the full analysis pipeline — spot finding, indexing,
geometry refinement, Bragg integration and (optionally) scaling and merging — and writes the
results to a `_process.h5` file, plus reflection files (`.mtz`/`.cif`/`.hkl`) when merging is
requested.

It runs the *same* analysis code as the online and interactive tools, just driven from the
command line over a file rather than a live detector stream.

> **Note.** `rugnux` is under very active development. This page describes the tool and
> its options at a high level; the authoritative, always-current list of options is the program's
> own usage message — run `rugnux` with no arguments.

## Where it fits among the three analysis tools

| Tool | Mode | Driven by | Output |
| --- | --- | --- | --- |
| [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) |
| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | Displayed on screen (results not saved to disk) |
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** |

Use `rugnux` to re-analyse data after acquisition, to experiment with processing
parameters, or to produce merged intensities for downstream structure solution.

## Hardware

As with the rest of Jungfraujoch, **serious performance requires an NVIDIA GPU**. The CUDA build
provides the GPU fast-feedback indexer (`ffbidx`) and the GPU FFT indexer (`fft`); without CUDA
only the CPU `fftw` indexer is available. Spot finding, integration and scaling run on the CPU and
scale with the thread count (`-N`).

## Input and output

**Input** is a single Jungfraujoch HDF5 master file (NXmx-based). If the dataset already contains
stored spot lists, two-pass rotation indexing can reuse them instead of re-running spot finding on
the first pass.

**Output** (controlled by `-o, --output-prefix`, default `output`):

- `<prefix>_process.h5` — NXmx-compliant HDF5 with derived metadata (spots, indexing,
  integration, azimuthal integration, per-image statistics). See
  [HDF5 / NeXus data format](HDF5.md) for the layout. Written by default only when **not** merging
  (i.e. under `--no-merge`); add `--write-process-h5` to also write it when merging.
- Merging is **on by default** (`--no-merge` disables it). The merged reflections are written as
  `<prefix>.cif` (mmCIF — the default), or `<prefix>.mtz` / `<prefix>.hkl` depending on
  `--scaling-output`. Both the mmCIF and the MTZ carry the **refined unit cell** (from rotation
  indexing) and the **space group determined from systematic absences** (constrained to the indexed
  lattice symmetry). No-reference scaling additionally emits per-iteration `<prefix>_iterN_scale.dat`.

Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
console. By default the written resolution is trimmed automatically where CC1/2 falls off
(`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the
limit by hand, or `--resolution-cutoff off` to keep the full range.

## Re-scaling and re-merging (`rugnux --scale`)

The `--scale` mode re-scales and merges the *already-integrated* reflections stored in a
`_process.h5` file, without re-running spot finding or integration. Use it to re-merge quickly with a
different space group, resolution limit, anomalous setting or reference MTZ. It reuses the same
`-o/-N/-s/-e/-S/-A/-B/-z/--scaling-*` options as the full run, and (unlike the full pipeline) does
not run a space-group search, so pass `-S` for the correct symmetry.

## Quick start

### Rotation data

Index, integrate, scale and merge a rotation sweep, fully de novo:

```
rugnux rotation_master.h5 \
    -o lyso_rot -N 32 \
    --scaling-high-resolution 1.4
```

Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by
default**: two-pass rotation indexing (index the sweep once, then process every frame against that
lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). Scaling
and merging run **by default** (for both rotation and stills; `--no-merge` turns them off); the unit
cell is taken from the rotation indexer and the space group is determined from systematic absences,
and both are written
into the merged `.cif`.

Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front
tends to *degrade* low-symmetry cases. `--scaling-high-resolution` (set it to your expected
resolution) sharpens both the space-group search and the error model. To tune the first pass use
`--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be
treated as independent stills use `--force-still`.

After the per-frame scale-fulls step, rotation scaling applies two **correction surfaces**, **on by
default** (`--no-scaling-corrections` disables both):

- **Decay** — a global Debye–Waller relative-*B* over the run, for the radiation damage that weakens
  later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame
  scale cannot remove). It only engages when the total relative-*B* exceeds a physical floor (2 Ų).
- **Absorption** — a smooth multiplicative factor over the diffracted-beam direction in the goniometer
  frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at
  low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error
  largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the
  intensities, so it measurably lowers *R*<sub>free</sub>.

Both are **cross-validated** — fitted on even-numbered frames and kept only if they improve the
held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa) — so where the
systematic is absent they are a no-op rather than a source of added noise; that is why they are safe
to leave on. They run on the GPU when one is present, at negligible cost.

### Still / serial data

A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent
stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer,
then merge against a reference structure:

```
rugnux serial_master.h5 \
    -o lyso_serial -N 32 \
    -X ffbidx -C 79,79,38,90,90,90 -S 96 \
    --spot-sigma 4 \
    -z reference.mtz \
    --scaling-high-resolution 1.8
```

`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. For
weak serial data, tightening spot finding with `--spot-sigma 4` typically raises the indexing rate
substantially. If a dataset *does* carry a goniometer axis but you want per-frame stills processing
anyway, add `--force-still`.

## Command-line options

General:

| Option | Description |
| --- | --- |
| `-o, --output-prefix <txt>` | Output file prefix (default: `output`) |
| `-N, --threads <num>` | Number of worker threads (default: all hardware threads) |
| `-s, --start-image <num>` | First image to process (default: 0) |
| `-e, --end-image <num>` | Last image to process (default: all) |
| `-t, --stride <num>` | Process every *n*-th image (default: 1) |
| `-v, --verbose` | Verbose output |

Modes (default: full analysis — spot finding, indexing, integration and merging):

| Option | Description |
| --- | --- |
| `--azint-only` | Only run azimuthal integration (no spot finding/indexing); writes `<prefix>_process.h5` |
| `--scale` | Only re-scale/merge the already-integrated reflections in the input `_process.h5` (no re-integration) |

Spot finding:

| Option | Description |
| --- | --- |
| `--spot-sigma <num>` | Noise sigma level for spot finding (default: 3.0) |
| `--spot-threshold <num>` | Photon-count threshold for spot finding (default: 10) |
| `--spot-high-resolution <num>` | High-resolution limit for spot finding, Å (default: 1.5) |
| `--max-spots <num>` | Maximum spot count (default: 250) |
| `--detect-ice-rings[=on\|off]` | Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling/merging; overrides the dataset/master-file setting (default: use the dataset value) |

Azimuthal integration (the radial profile behind the per-image ice-ring score):

| Option | Description |
| --- | --- |
| `-q, --azim-q-spacing <num>` | Q bin spacing, 1/Å (default: 0.01; finer resolves the narrow ice rings) |
| `--azim-min-q <num>` | Minimum Q, 1/Å |
| `--azim-max-q <num>` | Maximum Q, 1/Å |
| `--azim-phi-bins <num>` | Number of azimuthal (phi) bins (default: 1) |
| `--polarization-correction <on\|off>` | Enable/disable the azimuthal polarization correction |
| `--solid-angle-correction <on\|off>` | Enable/disable the azimuthal solid-angle correction |

Indexing:

A dataset with a **rotation goniometer axis** is processed as rotation data (two-pass rotation
indexing) by default; a dataset without one is processed as independent stills. `--force-still`
overrides the former; the `-R` / `--single-pass-rotation` / `--force-rotation-lattice` flags request
rotation explicitly and pick the pass or lattice.

| Option | Description |
| --- | --- |
| `--force-still` | Treat a rotation (goniometer) dataset as independent stills instead of rotation |
| `-X, --indexing-algorithm <txt>` | `FFBIDX` \| `FFT` \| `FFTW` \| `Auto` \| `None` |
| `-C, --unit-cell <cell>` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`) |
| `-S, --space-group <num>` | Space group number (used for indexing and scaling) |
| `-r, --refine <txt>` | Geometry refinement: `none` \| `orientation` \| `beam_and_lattice` (default) |
| `-R, --two-pass-rotation[=num]` | Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100) |
| `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) |
| `--redo-rotation-spots` | Redo spot finding for the two-pass rotation first pass |
| `--force-rotation-lattice <vec>` | Force rotation lattice (9 floats, Å), skipping the first pass |

Indexer choice in brief: `ffbidx` (GPU) refines toward a **known cell** and is best for sparse
serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rotation data. See the
[CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md) for the algorithms.

Scaling and merging:

| Option | Description |
| --- | --- |
| `--no-merge` | Skip scaling and merging (on by default); write only the per-image `_process.h5` |
| `-A, --anomalous` | Anomalous mode (keep Friedel pairs separate) |
| `-B, --refine-bfactor` | Refine a per-image B-factor (stills only) |
| `--scale-fulls` / `--no-scale-fulls` | rot3d: refit a per-frame scale on the combined fulls (XDS order, Unity model); on by default for rotation data, off for stills |
| `--smooth-g[=deg]` | rot3d: smooth the per-frame scale *G* over a degree range before the 3D combine (XDS DELPHI-like; default 5° for rotation, 0 = off) |
| `--no-scaling-corrections` | rot3d: disable the default-on decay + absorption correction surfaces fitted on the fulls after scale-fulls (see below) |
| `--capture-uncertainty <num>` | rot3d: systematic sigma on under-captured fulls, ~num·(1−captured_fraction)·I (default: 1.0 for rotation, 0 otherwise) |
| `--min-captured-fraction <num>` | rot3d: drop a combined full whose rocking curve was captured below this fraction — edge-of-sweep truncated fulls (default: 0.7 for rotation, 0 otherwise; 0 = off) |
| `--scaling-high-resolution <num>` | High-resolution limit for scaling, Å — manual override (default: no limit; disables the automatic cutoff below) |
| `--resolution-cutoff <txt>` | Automatic high-resolution cutoff for the written reflections and reported shells: `cc-logistic` \| `off` (default: `cc-logistic`; ignored when `--scaling-high-resolution` is set) |
| `--resolution-cc-target <num>` | CC1/2 target defining the `cc-logistic` fall-off (default: 0.30) |
| `--resolution-shells <num>` | Number of resolution shells in the reported statistics table (default: 10) |
| `--min-partiality <num>` | Minimum partiality to accept a reflection (default: 0.02) |
| `--reject-outliers <num>` | Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for `rot3d`, off otherwise) |
| `--reject-delta-cchalf <num>` | Drop images with ΔCC1/2 below mean − N·stddev (default: off) |
| `--min-image-cc <num>` | Per-image CC limit, percent (default: no limit) |
| `--mosaicity <num>` | Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed |
| `--scaling-iterations <num>` | Scaling iterations with no reference data (default: 3) |
| `--scaling-output <txt>` | Reflection output format: `cif` (mmCIF, default) \| `mtz` \| `txt` |
| `-z, --reference-mtz <file>` | Reference MTZ (enables reference-driven scaling) |
| `--reference-column <label>` | Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
| `--write-process-h5` | Also write the (large) `_process.h5` when merging (default: only `.mtz`/`.cif`) |

Integration:

| Option | Description |
| --- | --- |
| `--integrator <txt>` | Spot integrator: `gaussian` (profile-fit, default) \| `empirical` \| `boxsum` (classical fallback) |
| `--integration-radius <r>` | Signal-box radius `r1`, or `r1,r2,r3` (px). One value ⇒ `r2=r1+2`, `r3=r1+4` |
| `--bandwidth <num>` | Relative X-ray bandwidth FWHM (e.g. `0.01` for a 1% DMM); default from file or 0 (monochromatic) |

Geometry overrides (defaults are taken from the input file; override them to reprocess with a corrected geometry):

| Option | Description |
| --- | --- |
| `--beam-x <num>` | Beam centre X (pixel) |
| `--beam-y <num>` | Beam centre Y (pixel) |
| `--detector-distance <num>` | Detector distance (mm) |
| `--wavelength <num>` | Wavelength (Å) |
| `--rot1 <num>` | PONI detector rotation 1 (rad) |
| `--rot2 <num>` | PONI detector rotation 2 (rad) |
| `--polarization <num>` | Polarization factor |