Jungfraujoch/docs/RUGNUX_TUTORIAL.md

Running rugnux

:local:
:depth: 2

A first run in detail

The Quick start command is the whole of it:

rugnux -o myrun /path/to/dataset_master.h5

-o myrun is the prefix every output file is named from and the last argument is the master file of the dataset — any of the formats under What rugnux reads, not only one written by Jungfraujoch; -N would set the worker-thread count, which otherwise follows the machine. Nothing is assumed about the crystal — the goniometer axis in the file tells rugnux this is a rotation sweep, the unit cell comes from indexing the data, the space group from its systematic absences, and the resolution limit from where CC1/2 falls off. Progress, statistics and timing go to the terminal, and the output files land next to each other.

myrun_report.txt is written for a person, top to bottom: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention. To pull one number out of it in a script, every value is a KEY= value line:

grep '^SPACE_GROUP_NUMBER= '        myrun_report.txt
grep '^SPACE_GROUP_NAME= '         myrun_report.txt   # the setting, which the number does not name
grep '^UNIT_CELL_CONSTANTS= '      myrun_report.txt
grep '^INCLUDE_RESOLUTION_RANGE= ' myrun_report.txt
grep '^ISA= '                      myrun_report.txt
grep '^WARNING:'                   myrun_report.txt

One caveat for a script reading those keys: where the space group is one of an enantiomorphic pair, section 4 of the report says so in prose — Best space group: P 41 or P 43 (indistinguishable from these data) — but SPACE_GROUP_NUMBER= / SPACE_GROUP_NAME= carry the pair's lower-numbered member, which is a reporting convention and not a determination. Treat the two hands as interchangeable until a model settles it: with --model, where the data accept the model (MODEL_FIT= ACCEPTED), the report carries MODEL_ENANTIOMORPH_ADOPTED= TRUE and the model's group instead, and refining an isomorphous model against the wrong hand of the pair costs nothing subtler than an R factor near 0.55.

Useful variations, each independent of the others:

# tell it where the data really stop, if you already know - this sharpens the
# space-group search and the error model
rugnux -o myrun --scaling-high-resolution 1.4 dataset_master.h5

# keep Friedel pairs apart, for anomalous work
rugnux -o myrun -A dataset_master.h5

# a quick look at the first 200 images only
rugnux -o quicklook -e 200 dataset_master.h5

# merge as usual, but also keep the per-image file so the data can be re-merged later
rugnux -o myrun --write-process-h5 dataset_master.h5

# skip the unmerged MTZ (written by default), when only the merged data is wanted
rugnux -o myrun --no-export-unmerged dataset_master.h5

# check the merged data against a known structure: R-work / R-free and maps
rugnux -o myrun --model model.pdb dataset_master.h5

Re-merging is cheap and does not re-read the images. Ask the full run to keep its per-image file with --write-process-h5, and --mode scale will then re-scale and re-merge the reflections already integrated in it — seconds rather than minutes:

rugnux -o myrun --write-process-h5 dataset_master.h5   # integrate and merge once
rugnux --mode scale -o remerged -A myrun_process.h5          # re-merge, here anomalously

Use it to try a different resolution limit, anomalous setting or outlier rejection without paying for integration again. --mode scale merges in the space group and cell the file records, so the second command needs no -S. Note that --no-merge also writes a _process.h5, but a run that never merged never determined a space group either, so re-merging that file lands in P1 unless you pass -S yourself — --write-process-h5 is the one to use.

Rotation data

Index, integrate, scale and merge a rotation sweep, fully de novo:

rugnux rotation_master.h5 \
    -o rotation_run \
    --scaling-high-resolution 1.4

Because the dataset carries a rotation goniometer axis, it is processed as rotation data by default: two-pass rotation indexing (index the sweep once, then process every frame against that lattice) with the rot3d partiality model (rotation partials combined into 3D fulls). Scaling and merging run by default (for both rotation and stills; --no-merge turns them off); the unit cell is taken from the rotation indexer and the space group is determined from systematic absences, and both are written into the merged .cif.

Run fully de novo (no -C/-S) for the best result — supplying a cell or space group up front tends to degrade low-symmetry cases. A -S group whose Bravais lattice the crystal turns out not to have stops the run and names the cell that was indexed, rather than merging in a frame the reflections are not in; where the lattice does have that group's setting, the reflections are reindexed into it. --scaling-high-resolution (set it to your expected resolution) sharpens both the space-group search and the error model. To tune the first pass use --two-pass-rotation=100 (or -R100 — the first-pass image count); to force the sweep to be treated as independent stills use --force-still.

By default a rotation run also post-refines the geometry in a second pass: the first pass integrates and merges at the header geometry, then the detector distance + beam centre and the crystal cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a small move: under 1 % in distance, and the gauge-weak beam centre within 15 px of the header value or of the run's own measured centre, whichever is nearer), and the second pass re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical <prefix>_* output; the header-geometry pass merges only to choose the space group and to judge the refined pass against, and writes no merged files of its own — no <prefix>_01.mtz, .cif, .hkl or _01_image.dat. (Where a process file is asked for at all, with --no-merge or --write-process-h5, each pass still writes its own, so <prefix>_01_process.h5 appears beside <prefix>_process.h5.) Disable it with --rotation-no-postrefine.

After the per-frame scale-fulls step, rotation scaling applies three correction surfaces, on by default (--no-scaling-corrections disables all):

• Decay — a global Debye–Waller relative-B over the run, for the radiation damage that weakens later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame scale cannot remove). It only engages when the total relative-B exceeds a physical floor (2 Ų). An optional --relative-b[=deg] extends this single global rate to a smooth per-batch relative-B curve (default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for crystals whose decay is non-linear in dose.
• Absorption — a smooth multiplicative factor over the diffracted-beam direction in the goniometer frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the intensities, so it measurably lowers R_free.
• Modulation — a smooth multiplicative factor over the position where a reflection lands on the detector (a flat-field: detector-response and geometric systematics that vary across the detector plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, which over-determines the surface. Because it lives in the detector frame (not the rotation) the same correction concept applies to stills. This is the largest of the three on JUNGFRAU data — it lowers R_meas by several to tens of percent on datasets that carry a detector systematic, while holding or improving CC_1/2 and the anomalous signal.

All three are cross-validated — fitted on even-numbered frames and kept only if they improve the held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is scored as a σ-independent, R_meas-like fractional deviation, so a surface can never pass cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op rather than a source of added noise, which is why they are safe to leave on.

Independently of any correction, a rotation run prints a radiation-damage report — the per-image scale correlation-to-merge and mosaicity versus dose, and the relative B-factor change over the run (first→last) together with a per-batch relative-B curve, also written to the merged mmCIF. It is a data-quality-vs-dose diagnostic and never alters the merged intensities. A batch whose data cannot support a measurement prints - instead of a value, and the first→last number is printed only where a straight line describes the curve — damage is progressive, so a curve that dips and recovers is a disturbance of the sweep, not dose, and the report says so and points at the sweep-quality section (RADIATION_DAMAGE_RELATIVE_B= NOT_A_TREND).

Still / serial data

A dataset with no goniometer axis (e.g. a serial grid scan) is processed as independent stills automatically — no flag needed. Known-cell indexing with the GPU fast-feedback indexer, then merge against a reference structure:

rugnux serial_master.h5 \
    -o serial_run \
    -X ffbidx -C 79,79,38,90,90,90 -S 96 \
    -z reference.mtz \
    --scaling-high-resolution 1.8

A crystal form with an indexing ambiguity — P3, P4, P6 and their relatives — needs either the -z above or --model model.pdb, and needs it on the run that integrates: every crystal is indexed in its own hand, and the two are averaged together in the merge unless each image is put into the same hand as it is integrated.

ffbidx requires a known cell (-C) and is the indexer of choice for sparse serial stills. The self-calibrating spot finder is on by default for both workflows (--no-adaptive-spots turns it off), and for serial stills leave --min-pix-per-spot unset so it is chosen per image — across the still-target battery this combination raises the indexing rate and typically extends resolution over a fixed threshold and fixed min-pix, at equal or better CC½. (You can still pin a fixed threshold with --spot-sigma / --spot-threshold and a fixed min-pix with --min-pix-per-spot.) If a dataset does carry a goniometer axis but you want per-frame stills processing anyway, add --force-still.

Output files

Output (controlled by -o, --output-prefix, default output):

• <prefix>_process.h5 — NXmx-compliant HDF5 with derived metadata (spots, indexing, integration, azimuthal integration, per-image statistics). See HDF5 / NeXus data format for the layout. Written by default only when not merging (i.e. under --no-merge); add --write-process-h5 to also write it when merging. It does not copy the images: /entry/data/data is a virtual dataset over the input files, so the input has to stay where it was for the pictures to be readable, and the pixel metadata (bit_depth_readout, underload_value, the dataset type) describes those files rather than the signed 32-bit container rugnux processes in.

• Merging is on by default (--no-merge disables it). The merged reflections are written in three formats — each has its uses downstream:

  • <prefix>.mtz — CCP4 MTZ for the CCP4 / phenix reflection tools. The columns are H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) F SIGF F(+) SIGF(+) F(-) SIGF(-) FreeR_flag (F is the French–Wilson amplitude); the anomalous columns are present whenever any reflection carries a Bijvoet split — a rotation run always does, a stills run only with -A.
  • <prefix>.cif — mmCIF, for deposition and as the self-describing native format (also carries the merging statistics, ISa, twinning and radiation-damage indicators).
  • <prefix>.hkl — SHELX HKLF 4 text (h k l I σ(I), fixed 3I4,2F8.2), the direct input for SHELXC / ANODE / SHELXD. Bijvoet mates are written separately (I(+) at +hkl, I(-) at -hkl) so the anomalous signal is preserved; intensities are put on a common scale so the largest value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file ends with the 0 0 0 terminator record.

  All three carry the refined unit cell (from rotation indexing) and the space group determined from systematic absences (constrained to the indexed lattice symmetry).

• <prefix>_unmerged.mtz — the integrated observations before merging, as an unmerged MTZ in POINTLESS's column layout, so the data can be scaled and merged by aimless, pointless, careless or iotbx.merging_statistics instead of by rugnux. Written by default, alongside the merged files and with --no-merge too; --no-export-unmerged skips it. See The unmerged export below. --export-unmerged-partials writes <prefix>_unmerged_partials.mtz, one row per image, instead of summing.

• <prefix>_P1.mtz — the P1 cross-check dataset: the same observations merged in P1 instead of the space group the run determined, so a wrong call can be re-merged, re-solved or re-refined without processing the images again. It is a full merge, not the degraded one the search itself runs on, and it is what rugnux --mode scale -S P1 would make from a _process.h5. Every rotation run that determines its own space group writes it — including one that determined P1, where it simply repeats the merged output — so a script harvesting results can always expect the file rather than having to reproduce the search's decision to know whether it exists. It is not written when -S fixed the space group: prediction then rejects that group's centring absences, so those reflections were never integrated and a P1 merge of them would be missing whole classes of reflections. --no-p1-crosscheck declines it. Stills are not covered yet. Its FreeR_flag is drawn by the same index hash but over P1's own asymmetric unit, so it is not the merged file's test set: a re-refinement in P1 is a fresh cross-validation, and its R-free is not comparable with one against <prefix>.mtz.

• With --model, the validation outputs land beside the files above: the σ_A-weighted maps <prefix>_2fofc.ccp4 and <prefix>_fofc.ccp4, the map-coefficient MTZ <prefix>_maps.mtz, the anomalous difference map <prefix>_anom.ccp4 (where the merge kept the Bijvoet split), and the model as placed against the data as <prefix>_model.cif and <prefix>_model.pdb — the coordinates that go with the maps, in the cell and space group of <prefix>.mtz. Mind the names: <prefix>.cif is merged reflections, <prefix>_model.cif is coordinates. The PDB is there because fragment-screening tools want a <name>.pdb beside a <name>.mtz — PanDDA's per-dataset input layout, and the pair dimple produces — and it is skipped, with a log line, for a cell the PDB format cannot hold. See Validating against a model.

• <prefix>_report.txt — the results report: what the run determined, in a form both a person and a beamline script can read. Always written, next to the files above. See The results report below.

• <prefix>_image.dat — one row per processed image, whitespace-separated, with a # header line naming the columns: image_number scale_G mosaicity_deg wedge_deg cc_to_merge cc_n — the per-image scale, mosaicity, smoothing wedge and correlation-to-merge behind the sweep-quality table, ready for gnuplot or numpy.

Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the console. By default the written resolution is trimmed automatically where CC1/2 falls off (--resolution-cutoff cc-logistic, CC1/2 target 0.30); set --scaling-high-resolution to fix the limit by hand, or --resolution-cutoff off to keep the full range.