Jungfraujoch/docs/RUGNUX_TUTORIAL.md

# Running Rugnux

```{contents} On this page
:local:
:depth: 2
```

## A first run in detail

The [Quick start](RUGNUX.md#quick-start) command is the whole of it:

```
rugnux -o myrun /path/to/dataset_master.h5
```

`-o myrun` is the prefix every output file is named from and the last argument is the **master** file
of the dataset — any of the formats under [What Rugnux reads](RUGNUX_FORMATS.md), not only one written
by Jungfraujoch; `-N` would set the worker-thread count, which otherwise follows the
machine. Nothing is assumed about the crystal
— the goniometer axis in the file tells Rugnux this is a rotation sweep, the unit cell comes from
indexing the data, the space group from its systematic absences, and the resolution limit from where
CC1/2 falls off. Progress, statistics and timing go to the terminal, and the output files land
next to each other.

`myrun_report.txt` is written for a person, top to bottom: it says which space group was chosen and
on what evidence, how far the data go, and anything that needs attention. To pull one number out of
it in a script, every value is a `KEY= value` line:

```
grep '^SPACE_GROUP_NUMBER= '        myrun_report.txt
grep '^SPACE_GROUP_NAME= '         myrun_report.txt   # the setting, which the number does not name
grep '^UNIT_CELL_CONSTANTS= '      myrun_report.txt
grep '^INCLUDE_RESOLUTION_RANGE= ' myrun_report.txt
grep '^ISA= '                      myrun_report.txt
grep '^WARNING:'                   myrun_report.txt
```

One caveat for a script reading those keys: where the space group is one of an **enantiomorphic
pair**, `SPACE_GROUP_NUMBER=` / `SPACE_GROUP_NAME=` carry one member of the pair by convention, not
by determination — the report says so beside them, in section 2, as `SPACE_GROUP_ALTERNATIVES=`
naming the other hand and `SPACE_GROUP_ENANTIOMORPH= UNDETERMINED`. Treat the two
hands as interchangeable until a model settles it: with `--model`, where the data accept the model
(`MODEL_FIT= ACCEPTED`), the report carries
`MODEL_ENANTIOMORPH_ADOPTED= TRUE` and the model's group instead, and refining an isomorphous model
against the wrong hand of the pair costs nothing subtler than an R factor near 0.55.

**Useful variations**, each independent of the others:

```
# tell it where the data really stop, if you already know - this sharpens the
# space-group search and the error model
rugnux -o myrun --scaling-high-resolution 1.4 dataset_master.h5

# keep Friedel pairs apart, for anomalous work
rugnux -o myrun -A dataset_master.h5

# a quick look at the first 200 images only
rugnux -o quicklook -e 200 dataset_master.h5

# merge as usual, but also keep the per-image file so the data can be re-merged later
rugnux -o myrun --write-process-h5 dataset_master.h5

# skip the unmerged MTZ (written by default), when only the merged data is wanted
rugnux -o myrun --no-export-unmerged dataset_master.h5

# check the merged data against a known structure: R-work / R-free and maps
rugnux -o myrun --model model.pdb dataset_master.h5
```

Re-merging is cheap and does not re-read the images. Ask the full run to keep its per-image file
with `--write-process-h5`, and `--mode scale` will then re-scale and re-merge the reflections
already integrated in it — seconds rather than minutes:

```
rugnux -o myrun --write-process-h5 dataset_master.h5   # integrate and merge once
rugnux --mode scale -o remerged -A myrun_process.h5          # re-merge, here anomalously
```

Use it to try a different resolution limit, anomalous setting or outlier rejection without
paying for integration again. `--mode scale` merges in the space group and cell the file
records, so the second command needs no `-S`. Note that `--no-merge` also writes a `_process.h5`,
but a run that never merged never determined a space group either, so re-merging that file lands in
P1 unless you pass `-S` yourself — `--write-process-h5` is the one to use.

## Rotation data

Index, integrate, scale and merge a rotation sweep, fully de novo:

```
rugnux rotation_master.h5 \
    -o rotation_run \
    --scaling-high-resolution 1.4
```

Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by
default**: two-pass rotation indexing (index the sweep once, then process every frame against that
lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). Scaling
and merging run **by default** (for both rotation and stills; `--no-merge` turns them off); the unit
cell is taken from the rotation indexer and the space group is determined from systematic absences,
and both are written
into the merged files.

Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front
tends to *degrade* low-symmetry cases. A `-S` group whose Bravais lattice the crystal turns out not to
have stops the run and names the cell that was indexed, rather than merging in a frame the reflections
are not in; where the lattice does have that group's setting, the reflections are reindexed into it. `--scaling-high-resolution` (set it to your expected
resolution) sharpens both the space-group search and the error model. To tune the first pass use
`--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be
treated as independent stills use `--force-still`.

By default a rotation run also **post-refines the geometry** in a second pass: the first pass
integrates and merges at the header geometry, then the detector distance + beam centre and the crystal
cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a
small move: under 1 % in distance, and the gauge-weak beam centre within 15 px of the header value
or of the run's own measured centre, whichever is nearer), and the second pass
re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical
`<prefix>_*` output; the header-geometry pass merges only to choose the space group and to judge the
refined pass against, and writes no merged files of its own — no `<prefix>_01.mtz`, `.cif` or
`.hkl`, and no `_01_plot.txt` or `_01_detector.jpg`. (Where a process file is asked for at all, with `--no-merge` or
`--write-process-h5`, each pass still writes its own, so `<prefix>_01_process.h5` appears beside
`<prefix>_process.h5`.) Disable it with `--rotation-no-postrefine`.

After the per-frame scale-fulls step, rotation scaling applies three **correction surfaces**, **on by
default** (`--no-scaling-corrections` disables all):

- **Decay** — a global Debye–Waller relative-*B* over the run, for the radiation damage that weakens
  later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame
  scale cannot remove). It only engages when the total relative-*B* exceeds a physical floor (2 Ų). An
  optional `--relative-b[=deg]` extends this single global rate to a smooth per-batch relative-*B* curve
  (default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for
  crystals whose decay is non-linear in dose.
- **Absorption** — a smooth multiplicative factor over the diffracted-beam direction in the goniometer
  frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at
  low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error
  largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the
  intensities, so it measurably lowers *R*<sub>free</sub>.
- **Modulation** — a smooth multiplicative factor over the position where a reflection lands on the
  detector (a flat-field: detector-response and geometric systematics that vary across the detector
  plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal
  rotates, which over-determines the surface. Because it lives in the detector frame (not the
  rotation) the same correction concept applies to stills. This is the largest of the three on
  JUNGFRAU data — it lowers *R*<sub>meas</sub> by several to tens of percent on datasets that carry a
  detector systematic, while holding or improving CC1/2 and the anomalous signal.

All three are **cross-validated** — fitted on even-numbered frames and kept only if they improve the
held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is
scored as a σ-independent, *R*<sub>meas</sub>-like fractional deviation, so a surface can never pass
cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op
rather than a source of added noise, which is why they are safe to leave on.

Independently of any correction, a rotation run prints a **radiation-damage report** — the per-image
scale correlation-to-merge and mosaicity versus dose, and the relative *B*-factor change over the run
(first→last) together with a per-batch relative-*B* curve, also written to the merged mmCIF. It is a
data-quality-vs-dose diagnostic and never alters the merged intensities. A batch whose data cannot
support a measurement prints `-` instead of a value, and the first→last number is printed only where a
straight line describes the curve — damage is progressive, so a curve that dips and recovers is a
disturbance of the sweep, not dose, and the report says so and points at the sweep-quality section
(`RADIATION_DAMAGE_RELATIVE_B= NOT_A_TREND`).

## Still / serial data

A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent
stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer,
then merge against a reference structure:

```
rugnux serial_master.h5 \
    -o serial_run \
    -X ffbidx -C 79,79,38,90,90,90 -S 96 \
    -z reference.mtz \
    --scaling-high-resolution 1.8
```

A crystal form with an [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity) — P3, P4, P6 and their
relatives — **needs** either the `-z` above or `--model model.pdb`, and needs it on the run that
integrates: every crystal is indexed in its own hand, and the two are averaged together in the merge
unless each image is put into the same hand as it is integrated.

`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. The
self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for
serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this
combination raises the indexing rate and typically extends resolution over a fixed threshold and
fixed min-pix, at equal or better CC1/2. (You can still pin a fixed threshold with `--spot-sigma` /
`--spot-threshold` and a fixed min-pix with `--min-pix-per-spot`.) If a dataset *does* carry a
goniometer axis but you want per-frame stills processing anyway, add `--force-still`.

## Output files

**Output** (controlled by `-o, --output-prefix`, default `output`):

- `<prefix>_process.h5` — NXmx-compliant HDF5 with derived metadata (spots, indexing,
  integration, azimuthal integration, per-image statistics). See
  [HDF5 / NeXus data format](HDF5.md) for the layout. Written by default only when **not** merging
  (i.e. under `--no-merge`); add `--write-process-h5` to also write it when merging.
  It does not copy the images: `/entry/data/data` is a virtual dataset over the *input* files, so
  the input has to stay where it was for the pictures to be readable, and the pixel metadata
  (`bit_depth_readout`, `underload_value`, the dataset type) describes those files rather than the
  signed 32-bit container Rugnux processes in.
- Merging is **on by default** (`--no-merge` disables it). The merged reflections are written in
  **three** formats — each has its uses downstream:
  - `<prefix>.mtz` — CCP4 MTZ for the CCP4 / phenix reflection tools. The columns are
    `H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) F SIGF F(+) SIGF(+) F(-) SIGF(-) FreeR_flag`
    (`F` is the French–Wilson amplitude); the anomalous columns are present whenever any reflection
    carries a Bijvoet split — a rotation run always does, a stills run only with `-A`.
  - `<prefix>.cif` — mmCIF, for deposition and as the self-describing native format (also carries the
    merging statistics, ISa, twinning and radiation-damage indicators).
  - `<prefix>.hkl` — SHELX **HKLF 4** text (`h k l I σ(I)`, fixed `3I4,2F8.2`), the direct input for
    **SHELXC / ANODE / SHELXD**. Bijvoet mates are written separately (`I(+)` at `+hkl`, `I(-)` at
    `-hkl`) so the anomalous signal is preserved; intensities are put on a common scale so the largest
    value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file
    ends with the `0 0 0` terminator record.

  All three carry the **refined unit cell** (from rotation indexing) and the **space group determined
  from systematic absences** (constrained to the indexed lattice symmetry).
- `<prefix>_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in
  POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**,
  **careless** or `iotbx.merging_statistics` instead of by Rugnux. Written by default, alongside the
  merged files and with `--no-merge` too; `--no-export-unmerged` skips it. See
  [The unmerged export](RUGNUX_INTEGRATION.md#the-unmerged-export). `--export-unmerged-partials` writes
  `<prefix>_unmerged_partials.mtz`, one row per image, instead of summing.
- `<prefix>_P1.mtz` — the **P1 cross-check dataset**: the same observations merged in P1 instead of the
  space group the run determined, so a wrong call can be re-merged, re-solved or re-refined without
  processing the images again. It is a full merge, not the degraded one the search itself runs on, and
  it is what `rugnux --mode scale -S P1` would make from a `_process.h5`. **Every rotation run that
  determines its own space group writes it** — including one that determined P1, where it simply
  repeats the merged output — so a script harvesting results can always expect the file rather than
  having to reproduce the search's decision to know whether it exists. It is **not** written when `-S`
  fixed the space group: prediction then rejects that group's centring absences, so those reflections
  were never integrated and a P1 merge of them would be missing whole classes of reflections.
  `--no-p1-crosscheck` declines it. Stills are not covered yet. Its `FreeR_flag` is drawn by the
  same index hash but over P1's own asymmetric unit, so it is **not** the merged file's test set:
  a re-refinement in P1 is a fresh cross-validation, and its R-free is not comparable with one
  against `<prefix>.mtz`.
- With `--model`, the validation outputs land beside the files above: the σ<sub>A</sub>-weighted
  maps `<prefix>_2fofc.ccp4` and `<prefix>_fofc.ccp4`, the map-coefficient MTZ `<prefix>_maps.mtz`,
  the anomalous difference map `<prefix>_anom.ccp4` (where the merge kept the Bijvoet split), and
  the model **as placed** against the data as `<prefix>_model.cif` **and** `<prefix>_model.pdb` —
  the coordinates that go with the maps, in the cell and space group of `<prefix>.mtz`. Mind the
  names: `<prefix>.cif` is merged *reflections*, `<prefix>_model.cif` is *coordinates*. The PDB is
  there because fragment-screening tools want a `<name>.pdb` beside a `<name>.mtz` — PanDDA's
  per-dataset input layout, and the pair dimple produces — and it is skipped, with a log line, for
  a cell the PDB format cannot hold. See
  [Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model).
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
  a beamline script can read. Always written, next to the files above. See
  [The results report](RUGNUX_REPORT.md#the-results-report).
- `<prefix>_plot.txt` — **the per-image table**, one row per processed image, whitespace-separated,
  with a single `#` legend line over its own columns so gnuplot plots it as it stands
  (`plot '<prefix>_plot.txt' using 1:4`). The columns are:

  | Column | Meaning |
  |--------|---------|
  | `image` | Processed-image ordinal, the numbering every per-image array `rugnux` writes uses; with `-s`/`--stride` the source image is `start + ordinal * stride`. |
  | `angle_deg` | Spindle angle at mid-exposure. |
  | `bkg` | Background estimate from the azimuthal profile (the `bkgEstimate` of the HDF5 file). |
  | `resolution_A` | How far the image's spots reach, from spot finding alone. |
  | `spots` | Spots found. |
  | `scale_G` | Per-image scale factor fitted by scaling. |
  | `sigma_M_deg` | Mosaicity as the rocking-curve **standard deviation** σ<sub>M</sub> in degrees — not a FWHM, and not XDS's `REFLECTING_RANGE`. See the note below. |
  | `cc_to_merge` | Correlation of the image's own observations with the merged data. |
  | `cc_n` | Observations that correlation was computed over; 0 where none was. |
  | `merged` | 1 where the image's observations are in the merged data, 0 where they are not — an image that never indexed, one an earlier guard dropped, and one ΔCC<sub>1/2</sub> convicted all read 0. `nan` on a run that merged nothing, `--no-merge` included, where the question has no answer. |

  A quantity nothing measured for an image is written `nan`, which gnuplot and numpy both skip; the
  columns never shift and a row is never left out. A stills run writes the table too, with `nan` in
  the columns a sweep would fill; `--no-merge` writes it with the scaling columns `nan`.

  **Reading `sigma_M_deg`.** It is fitted per image — a maximum-likelihood Gaussian rocking width
  over that image's own indexed spots — but it is then **smoothed in frame order** over the same
  rotation window as the per-image scale (`--smooth-g`), and any frame too sparse to fit one of its
  own is given the run's median. So the column varies only on the smoothing window's scale, and on a
  weakly diffracting crystal it is close to, or exactly, a constant: that flatness is the honest
  statement that the run measured no per-frame variation, not a placeholder. The value is the one
  scaling used to recompute partialities. σ<sub>M</sub> here is the width of a Gaussian
  rocking curve and nothing more — it is not a physical crystal mosaic spread, it absorbs beam
  divergence and every other source of angular spread, and it reads systematically **low** against
  XDS's `SIGMAR` (about 0.9× on the regression corpus). Use it to see whether the run's rocking
  width moves across the sweep, not as a number to quote.
- `<prefix>_detector.jpg` — **the detector as the run saw it**, so the beam stop and the module gaps
  can be checked by eye. Rotation runs only, and only where `--detect-beam-stop` was left on (it is
  by default). The picture is the mean projection over the frames the beam-stop detection sampled —
  the very image it tested — coloured with `jfjoch_viewer`'s default map: white to indigo for the
  counts, **grey** for the module and chip gaps, **coral** for the shadow the run detected, and
  **magenta** for everything else the pixel mask holds. The scale saturates at eight times the
  median measured pixel, not at the brightest pixels, so the background spans the ramp and the Bragg
  spots clip: it is the background the picture is read for, since a shadow is a place the background
  is missing. A shadow drawn where the
  background is unbroken, or a dark patch the run left uncoloured, is visible at a glance; read it
  against the `Beam stop shadow: N pixels ...` line in the log.

Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
console. By default the written resolution is trimmed automatically where CC1/2 falls off
(`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the
limit by hand, or `--resolution-cutoff off` to keep the full range.