Jungfraujoch/docs/RUGNUX.md

# Rugnux

`rugnux` is the **offline** crystallographic data-analysis tool of Jungfraujoch — the
data-processing half of the system (see [Naming](NAMING.md) for where the name comes from). It takes
a stored dataset from the diffraction images to merged data — spot finding, indexing, geometry
refinement, Bragg integration, scaling and merging, written as `.mtz`/`.cif`/`.hkl` with the
unmerged MTZ and the results report — and, with an atomic model, on to a refined structure and maps:
the model is placed against the merge, refined, and written with its map coefficients and the
difference-density blobs it leaves unexplained. The per-image `_process.h5` is written only on
request (`--write-process-h5`), or as the only output with `--no-merge`. A rotation run that finds
no lattice still spot-finds every image and writes their per-image analysis there, then exits as
failed.

It runs the *same* analysis code as the online and interactive tools, driven from the command line
over files.

```
rugnux {<options>} <input.h5> {<input.h5>} [<broker URL>]
```

Run it with no arguments to print the usage.

Several Jungfraujoch/NXmx masters of stills, given one after another, are read as one run: their
images are taken in the order given, indexed and merged together, and the log gives the indexing
rate of each input beside the run's. They must be one setup - the same detector, wavelength,
distance and beam centre (to 0.1%, 0.1 mm and 1 pixel), since stills geometry refinement refines
one geometry for all frames - and a pixel masked in any of them is masked for all. Stills in the
other formats are given the same way, as all their files: `rugnux run*.h5` (SACLA), `rugnux *.cxi`,
`rugnux *.xtc`. Inputs are always files, read in the order given (a shell glob gives them sorted by
name); a directory is refused. Rotation sweeps are not read together.

Given the URL of a `jfjoch_broker` (`http://host:port/`) and the path the master file *will* have,
`rugnux` follows the broker's current collection while it is being written: the data files are read
as the broker reports them closed, processing starts on the first images, and the run ends with the
images the collection delivered. A choice made early on a near tie is checked again on every frame
once all are in, and the run is made again where that check ends elsewhere, so the answer is the one
the whole sweep gives. This is for rotation data and `--mode mx`; the collection's token is read from
`JUNGFRAUJOCH_HTTP_TOKEN` and sent only to the broker. With the master file already on disk, the URL
is ignored.

> **Note.** `rugnux` is under very active development. This page describes the tool and
> its options at a high level; the authoritative, always-current list of options is the program's
> own usage message — run `rugnux` with no arguments.

```{contents} On this page
:local:
:depth: 2
```

## Quick start

Give `rugnux` the images, and nothing else. A dataset carrying a **goniometer axis** that turns is
processed as a rotation sweep, one without (or with an axis that stands still) as **serial stills**;
indexing, the space group, the cell, the resolution limit and the error model are all decided from
the data, and scaling and merging run by default. `-o` names the output files (default `output`).

**We expect no need to tune anything else.** The defaults are meant to be the answer, and a run that
needed a flag to do well is one we want to hear about. The other options exist for unusual cases and
for diagnosis; they are listed under [Command-line options](RUGNUX_ADVANCED.md#command-line-options).

### Rotation data

```
rugnux -o myrun dataset_master.h5          # NXmx / EIGER / Jungfraujoch master file
rugnux -o myrun sweep_00001.cbf            # a CBF, marCCD or SMV sweep: name its first frame
```

### Serial stills

```
rugnux -o serial *_master.h5               # several NXmx masters of one setup, read as one run
rugnux -o serial run*.h5                   # all the files of one dataset (SACLA); *.cxi, *.xtc alike
```

Inputs are always files, never a directory; a shell glob hands them over sorted by name, and they are
read in that order and merged together. Without a cell the stills are indexed de novo.

### Serial stills with a known cell, space group and a CrystFEL geometry file

```
rugnux -o serial -C 79,79,38,90,90,90 -S P43212 --crystfel-geometry detector.geom run*.h5
```

`-C` is the cell (`a,b,c,alpha,beta,gamma`), `-S` the space group as a symbol (`P43212`) or a number
(`96`). With `-S` the run puts every crystal on one indexing where the space group admits two, from
the data alone. `--crystfel-geometry` places the panels (and takes the masks it names) for HDF5/CXI
input that does not place them itself. The wavelength and the detector distance come from the data
files or the geometry file; only where neither states them add `--wavelength <A>` and
`--detector-distance <mm>` (the distance is then held, not refined — without it a run with `-C`
searches and refines it).

### With a model

```
rugnux -o myrun dataset_master.h5 model.cif     # model and images in any order;
rugnux -o myrun --model model.pdb dataset_master.h5   # or with --model
```

A model is told from the images by its contents (PDB, or mmCIF with atoms), gzipped or not. After
merging it is **placed** — as a rigid body, or by molecular replacement where that does not place it or
the model is a prediction (AlphaFold, ModelCIF) — then **refined** against the merge (restrained,
maximum likelihood; seconds on a GPU, typically minutes on the CPU; `--no-refine` skips it), and the
run writes the maps (σ<sub>A</sub>-weighted 2mFo−DFc / mFo−DFc), the placed and refined model, and
the blobs of difference density the model does not explain. R-work / R-free are in the report.

The model **does not change processing decisions**: the merge, its space group and resolution are
the ones the data give without it (it only names the hand of an enantiomorphic pair, see below).
The one exception is serial stills with `-C` and `-S` in a space
group that admits two indexings: there the model picks which of them the crystals are put on, so the
merge comes out in the model's indexing. Without a model the data resolve the ambiguity themselves,
consistently but in an arbitrary one of the two. A reference reflection file (`-z reference.mtz`,
also accepted as a positional input) does the same, and steers the processing from the start (see
[Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md)).

### What comes out

A rotation run that merges — the default — leaves these files next to each other:

```
myrun.mtz            merged intensities + French-Wilson amplitudes, for CCP4 / phenix
myrun.cif            the same, as mmCIF - the self-describing format, and what to deposit
myrun.hkl            the scaled observations unmerged, as SHELX HKLF 4 - for SHELXL, SHELXC / ANODE
myrun.ins            the SHELX instructions to go with it: CELL, ZERR (the cell's e.s.d.s), LATT, SYMM
myrun_unmerged.mtz   every observation before scaling, for pointless / aimless / careless -
                     the largest file of the run (--no-export-unmerged skips it)
myrun_report.txt     what the run determined: cell, space group, statistics, warnings
myrun_plot.txt       one row per image, for plotting how the crystal behaved over the sweep
myrun_detector.jpg   the detector with the pixel mask and the beam-stop shadow drawn on it
myrun_log.txt        the full timestamped log of the run (debug level)
```

With a model, beside them:

```
myrun_maps.mtz            map coefficients on the placed model (Coot and PyMOL open it as maps)
myrun_model.cif / .pdb    the model as placed, in the cell and space group of myrun.mtz
myrun_refined_model.cif   the refined model (and .pdb)
myrun_refined_maps.mtz    map coefficients on the refined model
myrun_blobs.json          the unexplained difference-density blobs
```

A stills run writes the same merged files (its `.hkl` holds the merged reflections) and, on a
detector of several modules, `serial_modules.h5` with where each module was placed.
[Output files](RUGNUX_TUTORIAL.md#output-files) lists every file a run can write.

Parallelism needs no asking for: a run already uses the machine's threads. `-N` is there to *limit*
that, or to lift the per-image loop's default ceiling of 16 workers per GPU.

Read `myrun_report.txt` first: it says which space group was chosen and on what evidence, how far
the data go, and anything that needs attention.

### What the terminal shows

The terminal shows the run as numbered steps, each with one-line results and the time it took, the
warnings among them, and a summary at the end; everything else goes to `myrun_log.txt`. A note's
time is how long the step ran since the line above it, and the rule under a step gives its total, so
the steps add up to the wall time. Abridged:

```
Step 3  Indexing
        ... beam centre kept: measured 7.5 px off, same lattice         1.36 s
        ... lattice triclinic P: 28.56 35.34 63.56 106.0 90.3 90.3      0.00 s
        ... 52/60 validation frames, 24% of their spots on it           0.00 s
        ------------------------------------------------------- done in 1.36 s
Step 4  Integration (geometry pre-pass), 1800 images
        ... 1800 images, 836 per second                                 2.15 s
        ------------------------------------------------------- done in 2.16 s
   ...
Step 9  Scaling and merging
        ... merge in P1: 14464 unique                                   0.64 s
        ... space group P 1 21 1 (No. 4)                                0.05 s
        ... merge in P21: 16847 unique                                  0.26 s
        ... resolution 1.57 A (CC1/2 fall-off at 1.64 A)                0.02 s
        ------------------------------------------------------- done in 1.89 s
   ...
==============================================================================
 Result        WARNINGS: Merged to 1.57 A in P 1 21 1. 3 conditions need
               attention: SCALING_NOT_CONVERGED, MULTIPLE_LATTICES, ICE_RINGS.
 Space group   P 1 21 1
 Unit cell     35.337 28.465 62.937 90.000 105.824 90.000
 Resolution    1.57 A written; the CC1/2 fall-off is fitted at 1.64 A
 Signal        I/sigma 6.0   R_meas 17.2 %   CC1/2 0.995   ISa 10.2
 Wall time     11.78 s
==============================================================================
 With any question about this run, send myrun_report.txt and myrun_log.txt.
```

On a terminal an integration pass draws a progress bar in place; written to a file or a pipe (a
batch job's log) every line starts with the time of day instead, there is no bar, and a long pass
says how far it has got once a quarter. Warnings are marked `!!` and errors `** ERROR:`; a warning
that comes back from a later merge is shown once more on one line and then only counted. `-v` puts
the whole log on the terminal in place of the steps. The log file carries what is needed to answer a
question about a run without the data: version, build, GPU, threads, the command line, the header
geometry, every decision with its numbers, and a timestamp on every line.

A few things worth knowing before reaching for more flags:

- **The written reflections stop where CC1/2 falls through 0.30.** Every reflection file is
  resolution-trimmed automatically (`--resolution-cutoff cc-logistic`, one shell past the crossing);
  `--resolution-cutoff off` keeps the full measured range, `--scaling-high-resolution` fixes the
  limit by hand. A Rugnux file reaching less far than another program's on the same data is usually
  this default at work, not lost data.
- **Rotation data are best left de novo.** Pinning the cell and space group (`-C`, `-S`) is the
  normal thing to do for **serial stills**, where the `ffbidx` indexer needs a cell; on a rotation
  sweep it tends to *degrade* low-symmetry cases, so give the images alone and let the run determine both
  (see [Rotation data](RUGNUX_TUTORIAL.md#rotation-data)). `-S` takes a Hermann-Mauguin symbol (`P43212`) or a
  space-group number (`96`), whichever is to hand.
- **Anomalous data are there without `-A`.** A rotation merge always keeps the Bijvoet split: a
  default run's `.mtz` carries `I(+)`/`I(-)` beside `IMEAN`, and its `.hkl` every observation at the index it was measured at —
  `FRIEDELS_LAW= TRUE` in the report says how the *statistics* were counted, not that the signal was
  averaged away. What `-A` changes is the counting basis and the error model: each hand becomes a
  merged observation of its own, so multiplicity, completeness and ⟨I/σ⟩ are counted anomalously and
  the sigmas are refitted on the Bijvoet-separated merge. Reach for it for anomalous statistics; the
  signal itself is in the file either way. (Stills merges carry no split by default — there `-A` is
  what creates one.)
- **A model names the enantiomorph.** Where the data accept the model — it is tested against a
  null of the same model in random orientations, and `MODEL_FIT=` in the report says the verdict —
  `--model` settles which of P4<sub>1</sub>2<sub>1</sub>2 and P4<sub>3</sub>2<sub>1</sub>2 the merged
  reflections are *labelled* with — a choice no merged intensity can make. It is a label and nothing
  more: the two groups have the same rotation operations, so no reflection moves, and in particular
  I(+) and I(-) are left exactly as measured. Whether the model agrees with the data about the hand
  is then a real question, and the anomalous difference map answers it — a run says so when the
  density at the model's atoms comes out inverted.
- **`-z` and `--model` overlap but are not the same.** A reference MTZ steers the processing from
  the start; a model scores the merge and settles the frame it is written in — where the data
  accept it; a model they reject changes nothing. Either resolves an
  [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity) in its own indexing. On serial data the
  ambiguity decides whether the merged intensities are usable at all; there the space group (`-S`) alone
  already lets the run make the crystals consistent from the data.
- **Small molecules need no flag either.** Short axes, spots wider than the integration disk and
  glide-plane space groups are handled by the default run, and `myrun.hkl` goes straight to SHELXT /
  SHELXL ([Small-molecule data](RUGNUX_TUTORIAL.md#small-molecule-data)).
- **`--scaling-high-resolution <d>`**, where the resolution is already known, sharpens both the
  space-group search and the error model.
- **A run wants memory in proportion to what it integrates**, not to the detector: 2.5-14 GB of
  host RAM and 3-7 GB on the card over the datasets measured, both peaking in scaling and
  merging. [Installing Rugnux ▸ Memory](RUGNUX_INSTALL.md#memory) has the table and the two
  flags that lower it. A very large cell needs far more, and a CPU-only build most of all — tens of
  GB of host RAM ([Very large unit cells](RUGNUX_INSTALL.md#very-large-unit-cells)).
- **When a run fails** it exits non-zero and says why on the terminal; `VERDICT=` and the `WARNING:`
  lines of the report say what needs attention on a run that finished
  ([The results report](RUGNUX_REPORT.md#the-summary-and-what-the-run-decided)). A rotation run that indexes
  nothing most often has a wrong header: the beam centre is measured on every run, but the distance
  and the wavelength are indexed at the values stated — `--detector-distance` and `--wavelength` override them
  (`--beam-x`/`--beam-y` the centre); serial stills that index poorly want the cell (`-C`). A run that ends
  with `Killed` ran out of memory ([When host memory runs out](RUGNUX_INSTALL.md#when-host-memory-runs-out)).
  With any question about a run, send `myrun_report.txt` and `myrun_log.txt`.
- Everything else is in [Running Rugnux](RUGNUX_TUTORIAL.md#running-rugnux) and the full
  [Command-line options](RUGNUX_ADVANCED.md#command-line-options).


## The rest of the manual

One page per job, so the answer needed is near the top of a short page:

- [What Rugnux does](RUGNUX_OVERVIEW.md) — the pipeline from images to merged data and, with a model, a refined structure,
  in order. Read this one first.
- [Download Rugnux](RUGNUX_DOWNLOAD.md) — the archive for Linux, macOS and Windows, and the
  three commands that run it.
- [Installing Rugnux](RUGNUX_INSTALL.md) — packages, the release archive, GPU drivers, building
  from source, hardware.
- [What Rugnux reads](RUGNUX_FORMATS.md) — will it open your data: NXmx / EIGER masters, PILATUS
  miniCBF, marCCD and SMV (ADSC, Rigaku d\*TREK) sweeps, one sweep per input, and serial XFEL stills
  from SACLA (Cheetah HDF5), Cheetah CXI single-panel files and LCLS-I XTC.
- [Running Rugnux](RUGNUX_TUTORIAL.md) — a first run in detail, rotation, serial and small-molecule
  data, and every file a run writes.
- [Rugnux with other programs](RUGNUX_INTEGRATION.md) — the reflection-file conventions, the
  unmerged export, and worked command lines for phenix, REFMAC, POINTLESS / AIMLESS, careless,
  Phaser, SHELXC/D/E and, for small molecules, SHELXT / SHELXL.
- [The results report](RUGNUX_REPORT.md) — the `KEY= value` interface, sweep quality and the
  anisotropy section.
- [Advanced usage](RUGNUX_ADVANCED.md) — reference data and the indexing ambiguity, model
  validation, re-merging, and the full command-line option tables.
- [Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md) — every model and reference option,
  and decision by decision what a model can and cannot change.
- [Detector calibration](RUGNUX_CALIBRATION.md) — the geometry from a calibrant's powder rings
  (`--mode calibration`).
- [Measuring Rugnux](RUGNUX_BENCHMARKING.md) — the validation battery, the REFMAC check, the timing
  benchmark, and how to take timings on another machine that compare with ours.
- [CPU/GPU data analysis](CPU_DATA_ANALYSIS.md) — the algorithms behind all of it.

## Where it fits among the three analysis tools

| Tool | Mode | Driven by | Output |
| --- | --- | --- | --- |
| [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) |
| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | On screen; a processing job can write the same files as `rugnux` |
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`.mtz`/`.cif`/`.hkl` and the report; `_process.h5` with `--no-merge` or `--write-process-h5`** |

Use `rugnux` to re-analyse data after acquisition, to experiment with processing
parameters, or to produce merged intensities for downstream structure solution.