Jungfraujoch/docs/RUGNUX.md

# Rugnux

`rugnux` is the **offline** crystallographic data-analysis tool of Jungfraujoch — the
data-processing half of the system (see [Naming](NAMING.md) for where the name comes from).
It takes a stored dataset, runs the full analysis pipeline — spot finding, indexing, geometry
refinement, Bragg integration, scaling and merging — and writes the merged reflections
(`.mtz`/`.cif`/`.hkl`), the unmerged MTZ and the results report; the per-image `_process.h5` is
written only on request (`--write-process-h5`), or as the only output with `--no-merge`. A rotation run
that finds no lattice still spot-finds every image and writes their per-image analysis there, then exits
as failed.

It runs the *same* analysis code as the online and interactive tools, driven from the command line
over files.

```
rugnux {<options>} <input.h5> {<input.h5>} [<broker URL>]
```

Run it with no arguments to print the usage.

Several Jungfraujoch/NXmx masters of stills, given one after another, are read as one run: their
images are taken in the order given, indexed and merged together, and the log gives the indexing
rate of each input beside the run's. They must be one setup - the same detector, wavelength,
distance and beam centre (to 0.1%, 0.1 mm and 1 pixel), since stills geometry refinement refines
one geometry for all frames - and a pixel masked in any of them is masked for all. Stills in the
other formats are given the same way, as all their files: `rugnux run*.h5` (SACLA), `rugnux *.cxi`,
`rugnux *.xtc`. Inputs are always files, read in the order given (a shell glob gives them sorted by
name); a directory is refused. Rotation sweeps are not read together.

Given the URL of a `jfjoch_broker` (`http://host:port/`) and the path the master file *will* have,
`rugnux` follows the broker's current collection while it is being written: the data files are read
as the broker reports them closed, processing starts on the first images, and the run ends with the
images the collection delivered. A choice made early on a near tie is checked again on every frame
once all are in, and the run is made again where that check ends elsewhere, so the answer is the one
the whole sweep gives. This is for rotation data and `--mode mx`; the collection's token is read from
`JUNGFRAUJOCH_HTTP_TOKEN` and sent only to the broker. With the master file already on disk, the URL
is ignored.

> **Note.** `rugnux` is under very active development. This page describes the tool and
> its options at a high level; the authoritative, always-current list of options is the program's
> own usage message — run `rugnux` with no arguments.

```{contents} On this page
:local:
:depth: 2
```

## Quick start

Four commands cover most of what people ask of `rugnux`. Each takes the **master** file of the
dataset — one written by Jungfraujoch, a DECTRIS EIGER master, or an NXmx master written by another
facility's toolchain; a PILATUS miniCBF, marCCD or SMV sweep works too (see [What Rugnux reads](RUGNUX_FORMATS.md))
— and names its output files from `-o`:

```
# 1. everything from the data - index, integrate, scale and merge with the defaults
rugnux -o myrun dataset_master.h5

# 2. with a reference dataset of the same crystal form: it fixes the space group and the cell,
#    resolves the indexing ambiguity, and hands over its R-free set
rugnux -o myrun -z reference.mtz dataset_master.h5

# 3. with a known structure: R-work / R-free and sigma_A-weighted 2mFo-DFc / mFo-DFc maps
#    (a predicted model, or one the rigid body cannot place, goes to molecular replacement)
rugnux -o myrun --model model.pdb dataset_master.h5

# 4. with the space group and the cell pinned (-S takes either spelling: P43212 or 96)
rugnux -o myrun -S P43212 -C 79,79,38,90,90,90 dataset_master.h5
```

Parallelism needs no asking for: a run already uses the machine's threads. `-N` is there to *limit*
that, or to lift the per-image loop's default ceiling of 16 workers per GPU.

Nothing more is needed to pick the workflow: a dataset carrying a **goniometer axis** that turns is
processed as a rotation sweep, one without (or with an axis that stands still) as **independent
stills**, and scaling and merging run by default
in both. A rotation run that merges — the default — leaves eight files next to each other:

```
myrun.mtz            merged intensities + French-Wilson amplitudes, for CCP4 / phenix
myrun.cif            the same, as mmCIF - the self-describing format, and what to deposit
myrun.hkl            the scaled observations unmerged, as SHELX HKLF 4 - for SHELXL, SHELXC / ANODE
myrun_unmerged.mtz   every observation before scaling, for pointless / aimless / careless -
                     the largest file of the run (--no-export-unmerged skips it)
myrun_report.txt     what the run determined: cell, space group, statistics, warnings
myrun_plot.txt       one row per image, for plotting how the crystal behaved over the sweep
myrun_detector.jpg   the detector with the pixel mask and the beam-stop shadow drawn on it
myrun_log.txt        the full timestamped log of the run (debug level)
```

The two MTZ extras are most of the bytes a run writes — worth knowing when sizing a scratch
directory for a campaign, and both have off switches.

Read `myrun_report.txt` first: it says which space group was chosen and on what evidence, how far
the data go, and anything that needs attention.

### What the terminal shows

The terminal shows the run as numbered steps, each with one-line results and the time it took, the
warnings among them, and a summary at the end; everything else goes to `myrun_log.txt`. A note's
time is how long the step ran since the line above it, and the rule under a step gives its total, so
the steps add up to the wall time. Abridged:

```
Step 3  Indexing
        ... beam centre kept: measured 7.5 px off, same lattice         1.36 s
        ... lattice triclinic P: 28.56 35.34 63.56 106.0 90.3 90.3      0.00 s
        ... 52/60 validation frames, 24% of their spots on it           0.00 s
        ------------------------------------------------------- done in 1.36 s
Step 4  Integration (geometry pre-pass), 1800 images
        ... 1800 images, 836 per second                                 2.15 s
        ------------------------------------------------------- done in 2.16 s
   ...
Step 9  Scaling and merging
        ... merge in P1: 14464 unique                                   0.64 s
        ... space group P 1 21 1 (No. 4)                                0.05 s
        ... merge in P21: 16847 unique                                  0.26 s
        ... resolution 1.57 A (CC1/2 fall-off at 1.64 A)                0.02 s
        ------------------------------------------------------- done in 1.89 s
   ...
==============================================================================
 Result        WARNINGS: Merged to 1.57 A in P 1 21 1. 3 conditions need
               attention: SCALING_NOT_CONVERGED, MULTIPLE_LATTICES, ICE_RINGS.
 Space group   P 1 21 1
 Unit cell     35.337 28.465 62.937 90.000 105.824 90.000
 Resolution    1.57 A written; the CC1/2 fall-off is fitted at 1.64 A
 Signal        I/sigma 6.0   R_meas 17.2 %   CC1/2 0.995   ISa 10.2
 Wall time     11.78 s
==============================================================================
 With any question about this run, send myrun_report.txt and myrun_log.txt.
```

On a terminal an integration pass draws a progress bar in place; written to a file or a pipe (a
batch job's log) every line starts with the time of day instead, there is no bar, and a long pass
says how far it has got once a quarter. Warnings are marked `!!` and errors `** ERROR:`; a warning
that comes back from a later merge is shown once more on one line and then only counted. `-v` puts
the whole log on the terminal in place of the steps. The log file carries what is needed to answer a
question about a run without the data: version, build, GPU, threads, the command line, the header
geometry, every decision with its numbers, and a timestamp on every line.

A few things worth knowing before reaching for more flags:

- **The written reflections stop where CC1/2 falls through 0.30.** Every reflection file is
  resolution-trimmed automatically (`--resolution-cutoff cc-logistic`, one shell past the crossing);
  `--resolution-cutoff off` keeps the full measured range, `--scaling-high-resolution` fixes the
  limit by hand. A Rugnux file reaching less far than another program's on the same data is usually
  this default at work, not lost data.
- **Rotation data are best left de novo.** Pinning the cell and space group (recipe 4) is the
  normal thing to do for **serial stills**, where the `ffbidx` indexer needs a cell; on a rotation
  sweep it tends to *degrade* low-symmetry cases, so prefer recipe 1 and let the run determine both
  (see [Rotation data](RUGNUX_TUTORIAL.md#rotation-data)). `-S` takes a Hermann-Mauguin symbol (`P43212`) or a
  space-group number (`96`), whichever is to hand.
- **Anomalous data are there without `-A`.** A rotation merge always keeps the Bijvoet split: a
  default run's `.mtz` carries `I(+)`/`I(-)` beside `IMEAN`, and its `.hkl` every observation at the index it was measured at —
  `FRIEDELS_LAW= TRUE` in the report says how the *statistics* were counted, not that the signal was
  averaged away. What `-A` changes is the counting basis and the error model: each hand becomes a
  merged observation of its own, so multiplicity, completeness and ⟨I/σ⟩ are counted anomalously and
  the sigmas are refitted on the Bijvoet-separated merge. Reach for it for anomalous statistics; the
  signal itself is in the file either way. (Stills merges carry no split by default — there `-A` is
  what creates one.)
- **A model names the enantiomorph.** Where the data accept the model — it is tested against a
  null of the same model in random orientations, and `MODEL_FIT=` in the report says the verdict —
  `--model` settles which of P4<sub>1</sub>2<sub>1</sub>2 and P4<sub>3</sub>2<sub>1</sub>2 the merged
  reflections are *labelled* with — a choice no merged intensity can make. It is a label and nothing
  more: the two groups have the same rotation operations, so no reflection moves, and in particular
  I(+) and I(-) are left exactly as measured. Whether the model agrees with the data about the hand
  is then a real question, and the anomalous difference map answers it — a run says so when the
  density at the model's atoms comes out inverted.
- **`-z` and `--model` overlap but are not the same.** A reference MTZ steers the processing from
  the start; a model scores the merge and settles the frame it is written in — where the data
  accept it; a model they reject changes nothing. Either resolves an
  [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity), which on serial data decides whether the merged
  intensities are usable at all.
- **Small molecules need no flag either.** Short axes, spots wider than the integration disk and
  glide-plane space groups are handled by the default run, and `myrun.hkl` goes straight to SHELXT /
  SHELXL ([Small-molecule data](RUGNUX_TUTORIAL.md#small-molecule-data)).
- **`--scaling-high-resolution <d>`**, where the resolution is already known, sharpens both the
  space-group search and the error model.
- **A run wants memory in proportion to what it integrates**, not to the detector: 2.5-14 GB of
  host RAM and 3-7 GB on the card over the datasets measured, both peaking in scaling and
  merging. [Installing Rugnux ▸ Memory](RUGNUX_INSTALL.md#memory) has the table and the two
  flags that lower it. A very large cell needs far more, and a CPU-only build most of all — tens of
  GB of host RAM ([Very large unit cells](RUGNUX_INSTALL.md#very-large-unit-cells)).
- Everything else is in [Running Rugnux](RUGNUX_TUTORIAL.md#running-rugnux) and the full
  [Command-line options](RUGNUX_ADVANCED.md#command-line-options).


## The rest of the manual

One page per job, so the answer needed is near the top of a short page:

- [What Rugnux does](RUGNUX_OVERVIEW.md) — the pipeline from images to merged reflections,
  in order. Read this one first.
- [Installing Rugnux](RUGNUX_INSTALL.md) — packages, the release archive, GPU drivers, building
  from source, hardware.
- [What Rugnux reads](RUGNUX_FORMATS.md) — will it open your data: NXmx / EIGER masters, PILATUS
  miniCBF, marCCD and SMV (ADSC, Rigaku d\*TREK) sweeps, one sweep per input, and serial XFEL stills
  from SACLA (Cheetah HDF5), Cheetah CXI single-panel files and LCLS-I XTC.
- [Running Rugnux](RUGNUX_TUTORIAL.md) — a first run in detail, rotation, serial and small-molecule
  data, and every file a run writes.
- [Rugnux with other programs](RUGNUX_INTEGRATION.md) — the reflection-file conventions, the
  unmerged export, and worked command lines for phenix, REFMAC, POINTLESS / AIMLESS, careless,
  Phaser, SHELXC/D/E and, for small molecules, SHELXT / SHELXL.
- [The results report](RUGNUX_REPORT.md) — the `KEY= value` interface, sweep quality and the
  anisotropy section.
- [Advanced usage](RUGNUX_ADVANCED.md) — reference data and the indexing ambiguity, model
  validation, re-merging, and the full command-line option tables.
- [Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md) — every model and reference option,
  and decision by decision what a model can and cannot change.
- [Detector calibration](RUGNUX_CALIBRATION.md) — the geometry from a calibrant's powder rings
  (`--mode calibration`).
- [Measuring Rugnux](RUGNUX_BENCHMARKING.md) — the validation battery, the REFMAC check, the timing
  benchmark, and how to take timings on another machine that compare with ours.
- [CPU/GPU data analysis](CPU_DATA_ANALYSIS.md) — the algorithms behind all of it.

## Where it fits among the three analysis tools

| Tool | Mode | Driven by | Output |
| --- | --- | --- | --- |
| [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) |
| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | On screen; a processing job can write the same files as `rugnux` |
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** |

Use `rugnux` to re-analyse data after acquisition, to experiment with processing
parameters, or to produce merged intensities for downstream structure solution.