# Running Rugnux ```{contents} On this page :local: :depth: 2 ``` ## A first run in detail The [Quick start](RUGNUX.md#quick-start) command is the whole of it: ``` rugnux -o myrun /path/to/dataset_master.h5 ``` `-o myrun` is the prefix every output file is named from and the last argument is the **master** file of the dataset — any of the formats under [What Rugnux reads](RUGNUX_FORMATS.md), not only one written by Jungfraujoch; `-N` would set the worker-thread count, which otherwise follows the machine. Nothing is assumed about the crystal — the goniometer axis in the file tells Rugnux this is a rotation sweep, the unit cell comes from indexing the data, the space group from its systematic absences, and the resolution limit from where CC1/2 falls off. Progress, statistics and timing go to the terminal, and the output files land next to each other. `myrun_report.txt` is written for a person, top to bottom: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention. To pull one number out of it in a script, every value is a `KEY= value` line: ``` grep '^SPACE_GROUP_NUMBER= ' myrun_report.txt grep '^SPACE_GROUP_NAME= ' myrun_report.txt # the setting, which the number does not name grep '^UNIT_CELL_CONSTANTS= ' myrun_report.txt grep '^INCLUDE_RESOLUTION_RANGE= ' myrun_report.txt grep '^ISA= ' myrun_report.txt grep '^WARNING:' myrun_report.txt ``` One caveat for a script reading those keys: where the space group is one of an **enantiomorphic pair**, `SPACE_GROUP_NUMBER=` / `SPACE_GROUP_NAME=` carry one member of the pair by convention, not by determination — the report says so beside them, in section 2, as `SPACE_GROUP_ALTERNATIVES=` naming the other hand and `SPACE_GROUP_ENANTIOMORPH= UNDETERMINED`. Treat the two hands as interchangeable until a model settles it: with `--model`, where the data accept the model (`MODEL_FIT= ACCEPTED`), the report carries `MODEL_ENANTIOMORPH_ADOPTED= TRUE` and the model's group instead, and refining an isomorphous model against the wrong hand of the pair costs nothing subtler than an R factor near 0.55. **Useful variations**, each independent of the others: ``` # tell it where the data really stop, if you already know - this sharpens the # space-group search and the error model rugnux -o myrun --scaling-high-resolution 1.4 dataset_master.h5 # keep Friedel pairs apart, for anomalous work rugnux -o myrun -A dataset_master.h5 # a quick look at the first 200 images only rugnux -o quicklook -e 200 dataset_master.h5 # merge as usual, but also keep the per-image file so the data can be re-merged later rugnux -o myrun --write-process-h5 dataset_master.h5 # skip the unmerged MTZ (written by default), when only the merged data is wanted rugnux -o myrun --no-export-unmerged dataset_master.h5 # check the merged data against a known structure: R-work / R-free and maps rugnux -o myrun --model model.pdb dataset_master.h5 ``` Re-merging is cheap and does not re-read the images. Ask the full run to keep its per-image file with `--write-process-h5`, and `--mode scale` will then re-scale and re-merge the reflections already integrated in it — seconds rather than minutes: ``` rugnux -o myrun --write-process-h5 dataset_master.h5 # integrate and merge once rugnux --mode scale -o remerged -A myrun_process.h5 # re-merge, here anomalously ``` Use it to try a different resolution limit, anomalous setting or outlier rejection without paying for integration again. `--mode scale` merges in the space group and cell the file records, so the second command needs no `-S`. Note that `--no-merge` also writes a `_process.h5`, but a run that never merged never determined a space group either, so re-merging that file lands in P1 unless you pass `-S` yourself — `--write-process-h5` is the one to use. ## Rotation data Index, integrate, scale and merge a rotation sweep, fully de novo: ``` rugnux rotation_master.h5 \ -o rotation_run \ --scaling-high-resolution 1.4 ``` Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by default**: two-pass rotation indexing (index the sweep once, then process every frame against that lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). Scaling and merging run **by default** (for both rotation and stills; `--no-merge` turns them off); the unit cell is taken from the rotation indexer and the space group is determined from systematic absences, and both are written into the merged files. Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front tends to *degrade* low-symmetry cases. A `-S` group whose Bravais lattice the crystal turns out not to have stops the run and names the cell that was indexed, rather than merging in a frame the reflections are not in; where the lattice does have that group's setting, the reflections are reindexed into it. `--scaling-high-resolution` (set it to your expected resolution) sharpens both the space-group search and the error model. To tune the first pass use `--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be treated as independent stills use `--force-still`. By default a rotation run also **post-refines the geometry** in a second pass: the first pass integrates and merges at the header geometry, then the detector distance + beam centre and the crystal cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a small move: under 1 % in distance, and the gauge-weak beam centre within 15 px of the header value or of the run's own measured centre, whichever is nearer), and the second pass re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical `_*` output; the header-geometry pass merges only to choose the space group and to judge the refined pass against, and writes no merged files of its own — no `_01.mtz`, `.cif`, `.hkl` or `_01_image.dat`. (Where a process file is asked for at all, with `--no-merge` or `--write-process-h5`, each pass still writes its own, so `_01_process.h5` appears beside `_process.h5`.) Disable it with `--rotation-no-postrefine`. After the per-frame scale-fulls step, rotation scaling applies three **correction surfaces**, **on by default** (`--no-scaling-corrections` disables all): - **Decay** — a global Debye–Waller relative-*B* over the run, for the radiation damage that weakens later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame scale cannot remove). It only engages when the total relative-*B* exceeds a physical floor (2 Ų). An optional `--relative-b[=deg]` extends this single global rate to a smooth per-batch relative-*B* curve (default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for crystals whose decay is non-linear in dose. - **Absorption** — a smooth multiplicative factor over the diffracted-beam direction in the goniometer frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the intensities, so it measurably lowers *R*free. - **Modulation** — a smooth multiplicative factor over the position where a reflection lands on the detector (a flat-field: detector-response and geometric systematics that vary across the detector plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, which over-determines the surface. Because it lives in the detector frame (not the rotation) the same correction concept applies to stills. This is the largest of the three on JUNGFRAU data — it lowers *R*meas by several to tens of percent on datasets that carry a detector systematic, while holding or improving CC1/2 and the anomalous signal. All three are **cross-validated** — fitted on even-numbered frames and kept only if they improve the held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is scored as a σ-independent, *R*meas-like fractional deviation, so a surface can never pass cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op rather than a source of added noise, which is why they are safe to leave on. Independently of any correction, a rotation run prints a **radiation-damage report** — the per-image scale correlation-to-merge and mosaicity versus dose, and the relative *B*-factor change over the run (first→last) together with a per-batch relative-*B* curve, also written to the merged mmCIF. It is a data-quality-vs-dose diagnostic and never alters the merged intensities. A batch whose data cannot support a measurement prints `-` instead of a value, and the first→last number is printed only where a straight line describes the curve — damage is progressive, so a curve that dips and recovers is a disturbance of the sweep, not dose, and the report says so and points at the sweep-quality section (`RADIATION_DAMAGE_RELATIVE_B= NOT_A_TREND`). ## Still / serial data A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer, then merge against a reference structure: ``` rugnux serial_master.h5 \ -o serial_run \ -X ffbidx -C 79,79,38,90,90,90 -S 96 \ -z reference.mtz \ --scaling-high-resolution 1.8 ``` A crystal form with an [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity) — P3, P4, P6 and their relatives — **needs** either the `-z` above or `--model model.pdb`, and needs it on the run that integrates: every crystal is indexed in its own hand, and the two are averaged together in the merge unless each image is put into the same hand as it is integrated. `ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. The self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this combination raises the indexing rate and typically extends resolution over a fixed threshold and fixed min-pix, at equal or better CC1/2. (You can still pin a fixed threshold with `--spot-sigma` / `--spot-threshold` and a fixed min-pix with `--min-pix-per-spot`.) If a dataset *does* carry a goniometer axis but you want per-frame stills processing anyway, add `--force-still`. ## Output files **Output** (controlled by `-o, --output-prefix`, default `output`): - `_process.h5` — NXmx-compliant HDF5 with derived metadata (spots, indexing, integration, azimuthal integration, per-image statistics). See [HDF5 / NeXus data format](HDF5.md) for the layout. Written by default only when **not** merging (i.e. under `--no-merge`); add `--write-process-h5` to also write it when merging. It does not copy the images: `/entry/data/data` is a virtual dataset over the *input* files, so the input has to stay where it was for the pictures to be readable, and the pixel metadata (`bit_depth_readout`, `underload_value`, the dataset type) describes those files rather than the signed 32-bit container Rugnux processes in. - Merging is **on by default** (`--no-merge` disables it). The merged reflections are written in **three** formats — each has its uses downstream: - `.mtz` — CCP4 MTZ for the CCP4 / phenix reflection tools. The columns are `H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) F SIGF F(+) SIGF(+) F(-) SIGF(-) FreeR_flag` (`F` is the French–Wilson amplitude); the anomalous columns are present whenever any reflection carries a Bijvoet split — a rotation run always does, a stills run only with `-A`. - `.cif` — mmCIF, for deposition and as the self-describing native format (also carries the merging statistics, ISa, twinning and radiation-damage indicators). - `.hkl` — SHELX **HKLF 4** text (`h k l I σ(I)`, fixed `3I4,2F8.2`), the direct input for **SHELXC / ANODE / SHELXD**. Bijvoet mates are written separately (`I(+)` at `+hkl`, `I(-)` at `-hkl`) so the anomalous signal is preserved; intensities are put on a common scale so the largest value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file ends with the `0 0 0` terminator record. All three carry the **refined unit cell** (from rotation indexing) and the **space group determined from systematic absences** (constrained to the indexed lattice symmetry). - `_unmerged.mtz` — the integrated observations *before* merging, as an unmerged MTZ in POINTLESS's column layout, so the data can be scaled and merged by **aimless**, **pointless**, **careless** or `iotbx.merging_statistics` instead of by Rugnux. Written by default, alongside the merged files and with `--no-merge` too; `--no-export-unmerged` skips it. See [The unmerged export](RUGNUX_INTEGRATION.md#the-unmerged-export). `--export-unmerged-partials` writes `_unmerged_partials.mtz`, one row per image, instead of summing. - `_P1.mtz` — the **P1 cross-check dataset**: the same observations merged in P1 instead of the space group the run determined, so a wrong call can be re-merged, re-solved or re-refined without processing the images again. It is a full merge, not the degraded one the search itself runs on, and it is what `rugnux --mode scale -S P1` would make from a `_process.h5`. **Every rotation run that determines its own space group writes it** — including one that determined P1, where it simply repeats the merged output — so a script harvesting results can always expect the file rather than having to reproduce the search's decision to know whether it exists. It is **not** written when `-S` fixed the space group: prediction then rejects that group's centring absences, so those reflections were never integrated and a P1 merge of them would be missing whole classes of reflections. `--no-p1-crosscheck` declines it. Stills are not covered yet. Its `FreeR_flag` is drawn by the same index hash but over P1's own asymmetric unit, so it is **not** the merged file's test set: a re-refinement in P1 is a fresh cross-validation, and its R-free is not comparable with one against `.mtz`. - With `--model`, the validation outputs land beside the files above: the σA-weighted maps `_2fofc.ccp4` and `_fofc.ccp4`, the map-coefficient MTZ `_maps.mtz`, the anomalous difference map `_anom.ccp4` (where the merge kept the Bijvoet split), and the model **as placed** against the data as `_model.cif` **and** `_model.pdb` — the coordinates that go with the maps, in the cell and space group of `.mtz`. Mind the names: `.cif` is merged *reflections*, `_model.cif` is *coordinates*. The PDB is there because fragment-screening tools want a `.pdb` beside a `.mtz` — PanDDA's per-dataset input layout, and the pair dimple produces — and it is skipped, with a log line, for a cell the PDB format cannot hold. See [Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model). - `_report.txt` — the **results report**: what the run determined, in a form both a person and a beamline script can read. Always written, next to the files above. See [The results report](RUGNUX_REPORT.md#the-results-report). - `_image.dat` — one row per processed image, whitespace-separated, with a `#` header line naming the columns: `image_number scale_G mosaicity_deg wedge_deg cc_to_merge cc_n` — the per-image scale, mosaicity, smoothing wedge and correlation-to-merge behind the sweep-quality table, ready for gnuplot or numpy. Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the console. By default the written resolution is trimmed automatically where CC1/2 falls off (`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the limit by hand, or `--resolution-cutoff off` to keep the full range.