Jungfraujoch/docs/RUGNUX_TUTORIAL.md

# Running Rugnux

```{contents} On this page
:local:
:depth: 2
```

## A first run in detail

The [Quick start](RUGNUX.md#quick-start) command is the whole of it:

```
rugnux -o myrun /path/to/dataset_master.h5
```

`-o myrun` is the prefix every output file is named from and the last argument is the **master** file
of the dataset — any of the formats under [What Rugnux reads](RUGNUX_FORMATS.md), not only one written
by Jungfraujoch; `-N` would set the worker-thread count, which otherwise follows the
machine. Nothing is assumed about the crystal
— the goniometer axis in the file tells Rugnux this is a rotation sweep, the unit cell comes from
indexing the data, the space group from its systematic absences, and the resolution limit from where
CC1/2 falls off. Progress, statistics and timing go to the terminal, and the output files land
next to each other.

`myrun_report.txt` is written for a person, top to bottom: it says which space group was chosen and
on what evidence, how far the data go, and anything that needs attention. To pull one number out of
it in a script, every value is a `KEY= value` line:

```
grep '^SPACE_GROUP_NUMBER= '        myrun_report.txt
grep '^SPACE_GROUP_NAME= '         myrun_report.txt   # the setting, which the number does not name
grep '^UNIT_CELL_CONSTANTS= '      myrun_report.txt
grep '^INCLUDE_RESOLUTION_RANGE= ' myrun_report.txt
grep '^ISA= '                      myrun_report.txt
grep '^WARNING:'                   myrun_report.txt
```

One caveat for a script reading those keys: where the space group is one of an **enantiomorphic
pair**, `SPACE_GROUP_NUMBER=` / `SPACE_GROUP_NAME=` carry one member of the pair by convention, not
by determination — the report says so beside them, in section 2, as `SPACE_GROUP_ALTERNATIVES=`
naming the other hand and `SPACE_GROUP_ENANTIOMORPH= UNDETERMINED`. Treat the two
hands as interchangeable until a model settles it: with `--model`, where the data accept the model
(`MODEL_FIT= ACCEPTED`), the report carries
`MODEL_ENANTIOMORPH_ADOPTED= TRUE` and the model's group instead, and refining an isomorphous model
against the wrong hand of the pair costs nothing subtler than an R factor near 0.55.

**Useful variations**, each independent of the others:

```
# tell it where the data really stop, if you already know - this sharpens the
# space-group search and the error model
rugnux -o myrun --scaling-high-resolution 1.4 dataset_master.h5

# keep Friedel pairs apart, for anomalous work
rugnux -o myrun -A dataset_master.h5

# a quick look at the first 200 images only
rugnux -o quicklook -e 200 dataset_master.h5

# merge as usual, but also keep the per-image file so the data can be re-merged later
rugnux -o myrun --write-process-h5 dataset_master.h5

# skip the unmerged MTZ (written by default), when only the merged data is wanted
rugnux -o myrun --no-export-unmerged dataset_master.h5

# check the merged data against a known structure: R-work / R-free and maps
rugnux -o myrun --model model.pdb dataset_master.h5
```

Re-merging is cheap and does not re-read the images. Ask the full run to keep its per-image file
with `--write-process-h5`, and `--mode scale` will then re-scale and re-merge the reflections
already integrated in it — seconds rather than minutes:

```
rugnux -o myrun --write-process-h5 dataset_master.h5   # integrate and merge once
rugnux --mode scale -o remerged -A myrun_process.h5          # re-merge, here anomalously
```

Use it to try a different resolution limit, anomalous setting or outlier rejection without
paying for integration again. `--mode scale` merges in the space group and cell the file
records, so the second command needs no `-S`. Note that `--no-merge` also writes a `_process.h5`,
but a run that never merged never determined a space group either, so re-merging that file lands in
P1 unless you pass `-S` yourself — `--write-process-h5` is the one to use.

## Rotation data

Index, integrate, scale and merge a rotation sweep, fully de novo:

```
rugnux rotation_master.h5 \
    -o rotation_run \
    --scaling-high-resolution 1.4
```

Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by
default**: two-pass rotation indexing (index the sweep once, then process every frame against that
lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). Scaling
and merging run **by default** (for both rotation and stills; `--no-merge` turns them off); the unit
cell is taken from the rotation indexer and the space group is determined from systematic absences,
and both are written
into the merged files.

Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front
tends to *degrade* low-symmetry cases. A `-S` group whose Bravais lattice the crystal turns out not to
have stops the run and names the cell that was indexed, rather than merging in a frame the reflections
are not in; where the lattice does have that group's setting, the reflections are reindexed into it. `--scaling-high-resolution` (set it to your expected
resolution) sharpens both the space-group search and the error model. To tune the first pass use
`--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be
treated as independent stills use `--force-still`.

By default a rotation run also **post-refines the geometry** in a second pass: the first pass
integrates and merges at the header geometry, then the detector distance + beam centre and the crystal
cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a
small move: under 1 % in distance, and the gauge-weak beam centre within 15 px of the header value
or of the run's own measured centre, whichever is nearer), and the second pass
re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical
`<prefix>_*` output; the header-geometry pass merges only to choose the space group and to judge the
refined pass against, and writes no merged files of its own — no `<prefix>_01.mtz`, `.cif`, `.hkl` or
`_01_image.dat`, and no `_01_plot.txt` or `_01_detector.jpg`. (Where a process file is asked for at all, with `--no-merge` or
`--write-process-h5`, each pass still writes its own, so `<prefix>_01_process.h5` appears beside
`<prefix>_process.h5`.) Disable it with `--rotation-no-postrefine`.

After the per-frame scale-fulls step, rotation scaling applies three **correction surfaces**, **on by
default** (`--no-scaling-corrections` disables all):

- **Decay** — a global Debye–Waller relative-*B* over the run, for the radiation damage that weakens
  later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame
  scale cannot remove). It only engages when the total relative-*B* exceeds a physical floor (2 Ų). An
  optional `--relative-b[=deg]` extends this single global rate to a smooth per-batch relative-*B* curve
  (default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for
  crystals whose decay is non-linear in dose.
- **Absorption** — a smooth multiplicative factor over the diffracted-beam direction in the goniometer
  frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at
  low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error
  largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the
  intensities, so it measurably lowers *R*<sub>free</sub>.
- **Modulation** — a smooth multiplicative factor over the position where a reflection lands on the
  detector (a flat-field: detector-response and geometric systematics that vary across the detector
  plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal
  rotates, which over-determines the surface. Because it lives in the detector frame (not the
  rotation) the same correction concept applies to stills. This is the largest of the three on
  JUNGFRAU data — it lowers *R*<sub>meas</sub> by several to tens of percent on datasets that carry a
  detector systematic, while holding or improving CC1/2 and the anomalous signal.

All three are **cross-validated** — fitted on even-numbered frames and kept only if they improve the
held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is
scored as a σ-independent, *R*<sub>meas</sub>-like fractional deviation, so a surface can never pass
cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op
rather than a source of added noise, which is why they are safe to leave on.

Independently of any correction, a rotation run prints a **radiation-damage report** — the per-image
scale correlation-to-merge and mosaicity versus dose, and the relative *B*-factor change over the run
(first→last) together with a per-batch relative-*B* curve, also written to the merged mmCIF. It is a
data-quality-vs-dose diagnostic and never alters the merged intensities. A batch whose data cannot
support a measurement prints `-` instead of a value, and the first→last number is printed only where a
straight line describes the curve — damage is progressive, so a curve that dips and recovers is a
disturbance of the sweep, not dose, and the report says so and points at the sweep-quality section
(`RADIATION_DAMAGE_RELATIVE_B= NOT_A_TREND`).

## Still / serial data

A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent
stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer,
then merge against a reference structure:

```
rugnux serial_master.h5 \
    -o serial_run \
    -X ffbidx -C 79,79,38,90,90,90 -S 96 \
    -z reference.mtz \
    --scaling-high-resolution 1.8
```

A crystal form with an [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity) — P3, P4, P6 and their
relatives — **needs** either the `-z` above or `--model model.pdb`, and needs it on the run that
integrates: every crystal is indexed in its own hand, and the two are averaged together in the merge
unless each image is put into the same hand as it is integrated.

`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. The
self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for
serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this
combination raises the indexing rate and typically extends resolution over a fixed threshold and
fixed min-pix, at equal or better CC1/2. (You can still pin a fixed threshold with `--spot-sigma` /
`--spot-threshold` and a fixed min-pix with `--min-pix-per-spot`.) If a dataset *does* carry a
goniometer axis but you want per-frame stills processing anyway, add `--force-still`.

## Output files

**Output** (controlled by `-o, --output-prefix`, default `output`):

- `<prefix>_process.h5` — NXmx-compliant HDF5 with derived metadata (spots, indexing,
  integration, azimuthal integration, per-image statistics). See
  [HDF5 / NeXus data format](HDF5.md) for the layout. Written by default only when **not** merging
  (i.e. under `--no-merge`); add `--write-process-h5` to also write it when merging.
  It does not copy the images: `/entry/data/data` is a virtual dataset over the *input* files, so
  the input has to stay where it was for the pictures to be readable, and the pixel metadata
  (`bit_depth_readout`, `underload_value`, the dataset type) describes those files rather than the
  signed 32-bit container Rugnux processes in.
- Merging is **on by default** (`--no-merge` disables it). The merged reflections are written in
  **three** formats — each has its uses downstream:
  - `<prefix>.mtz` — CCP4 MTZ for the CCP4 / phenix reflection tools. The columns are
    `H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) F SIGF F(+) SIGF(+) F(-) SIGF(-) FreeR_flag`
    (`F` is the French–Wilson amplitude); the anomalous columns are present whenever any reflection
    carries a Bijvoet split — a rotation run always does, a stills run only with `-A`.
  - `<prefix>.cif` — mmCIF, for deposition and as the self-describing native format (also carries the
    merging statistics, ISa, twinning and radiation-damage indicators).
  - `<prefix>.hkl` — SHELX **HKLF 4** text (`h k l I σ(I)`, fixed `3I4,2F8.2`), the direct input for
    **SHELXC / ANODE / SHELXD**. Bijvoet mates are written separately (`I(+)` at `+hkl`, `I(-)` at
    `-hkl`) so the anomalous signal is preserved; intensities are put on a common scale so the largest
    value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file
    ends with the `0 0 0` terminator record.

  All three carry the **refined unit cell** (from rotation indexing) and the **space group determined
  from systematic absences** (constrained to the indexed lattice symmetry).
- `<prefix>_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in
  POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**,
  **careless** or `iotbx.merging_statistics` instead of by Rugnux. Written by default, alongside the
  merged files and with `--no-merge` too; `--no-export-unmerged` skips it. See
  [The unmerged export](RUGNUX_INTEGRATION.md#the-unmerged-export). `--export-unmerged-partials` writes
  `<prefix>_unmerged_partials.mtz`, one row per image, instead of summing.
- `<prefix>_P1.mtz` — the **P1 cross-check dataset**: the same observations merged in P1 instead of the
  space group the run determined, so a wrong call can be re-merged, re-solved or re-refined without
  processing the images again. It is a full merge, not the degraded one the search itself runs on, and
  it is what `rugnux --mode scale -S P1` would make from a `_process.h5`. **Every rotation run that
  determines its own space group writes it** — including one that determined P1, where it simply
  repeats the merged output — so a script harvesting results can always expect the file rather than
  having to reproduce the search's decision to know whether it exists. It is **not** written when `-S`
  fixed the space group: prediction then rejects that group's centring absences, so those reflections
  were never integrated and a P1 merge of them would be missing whole classes of reflections.
  `--no-p1-crosscheck` declines it. Stills are not covered yet. Its `FreeR_flag` is drawn by the
  same index hash but over P1's own asymmetric unit, so it is **not** the merged file's test set:
  a re-refinement in P1 is a fresh cross-validation, and its R-free is not comparable with one
  against `<prefix>.mtz`.
- With `--model`, the validation outputs land beside the files above: the σ<sub>A</sub>-weighted
  maps `<prefix>_2fofc.ccp4` and `<prefix>_fofc.ccp4`, the map-coefficient MTZ `<prefix>_maps.mtz`,
  the anomalous difference map `<prefix>_anom.ccp4` (where the merge kept the Bijvoet split), and
  the model **as placed** against the data as `<prefix>_model.cif` **and** `<prefix>_model.pdb` —
  the coordinates that go with the maps, in the cell and space group of `<prefix>.mtz`. Mind the
  names: `<prefix>.cif` is merged *reflections*, `<prefix>_model.cif` is *coordinates*. The PDB is
  there because fragment-screening tools want a `<name>.pdb` beside a `<name>.mtz` — PanDDA's
  per-dataset input layout, and the pair dimple produces — and it is skipped, with a log line, for
  a cell the PDB format cannot hold. See
  [Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model).
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
  a beamline script can read. Always written, next to the files above. See
  [The results report](RUGNUX_REPORT.md#the-results-report).
- `<prefix>_plot.txt` — **the per-image table**, one row per processed image, whitespace-separated,
  with a single `#` legend line over its own columns so gnuplot plots it as it stands
  (`plot '<prefix>_plot.txt' using 1:4`). Rotation runs only. The columns are:

  | Column | Meaning |
  |--------|---------|
  | `image` | Processed-image ordinal, the numbering every per-image array `rugnux` writes uses; with `-s`/`--stride` the source image is `start + ordinal * stride`. |
  | `angle_deg` | Spindle angle at mid-exposure. |
  | `bkg` | Background estimate from the azimuthal profile (the `bkgEstimate` of the HDF5 file). |
  | `resolution_A` | How far the image's spots reach, from spot finding alone. |
  | `spots` | Spots found. |
  | `scale_G` | Per-image scale factor fitted by scaling. |
  | `sigma_M_deg` | Mosaicity as the rocking-curve **standard deviation** σ<sub>M</sub> in degrees — not a FWHM, and not XDS's `REFLECTING_RANGE`. See the note below. |
  | `cc_to_merge` | Correlation of the image's own observations with the merged data. |
  | `merged` | 1 where the image's observations are in the merged data, 0 where they are not — an image that never indexed, one an earlier guard dropped, and one ΔCC<sub>1/2</sub> convicted all read 0. |

  A quantity nothing measured for an image is written `nan`, which gnuplot and numpy both skip; the
  columns never shift and a row is never left out.

  **Reading `sigma_M_deg`.** It is fitted per image — a maximum-likelihood Gaussian rocking width
  over that image's own indexed spots — but it is then **smoothed in frame order** over the same
  rotation window as the per-image scale (`--smooth-g`), and any frame too sparse to fit one of its
  own is given the run's median. So the column varies only on the smoothing window's scale, and on a
  weakly diffracting crystal it is close to, or exactly, a constant: that flatness is the honest
  statement that the run measured no per-frame variation, not a placeholder. The value is the one
  scaling used to recompute partialities. σ<sub>M</sub> here is the width of a Gaussian
  rocking curve and nothing more — it is not a physical crystal mosaic spread, it absorbs beam
  divergence and every other source of angular spread, and it reads systematically **low** against
  XDS's `SIGMAR` (about 0.9× on the regression corpus). Use it to see whether the run's rocking
  width moves across the sweep, not as a number to quote.
- `<prefix>_detector.jpg` — **the detector as the run saw it**, so the beam stop and the module gaps
  can be checked by eye. Rotation runs only, and only where `--detect-beam-stop` was left on (it is
  by default). The picture is the mean projection over the frames the beam-stop detection sampled —
  the very image it tested — coloured with `jfjoch_viewer`'s default map: white to indigo for the
  counts, **grey** for the module and chip gaps, **coral** for the shadow the run detected, and
  **magenta** for everything else the pixel mask holds. A shadow drawn where the background is
  unbroken, or a dark patch the run left uncoloured, is visible at a glance; read it against the
  `Beam stop shadow: N pixels ...` line in the log.
- `<prefix>_image.dat` — the per-image **scaling** table: `image_number scale_G mosaicity_deg
  wedge_deg cc_to_merge cc_n`, whitespace-separated with a `#` header line. It predates `_plot.txt`
  and is kept because the in-tree comparison scripts read it and because it carries the per-image
  mosaicity and smoothing wedge, which `_plot.txt` does not. **For anything new, script against
  `_plot.txt`.**

Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
console. By default the written resolution is trimmed automatically where CC1/2 falls off
(`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the
limit by hand, or `--resolution-cutoff off` to keep the full range.