From d41723fd8307aefb9e28d4359afcd243c5f8c287 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 28 Oct 2026 09:40:27 +0100 Subject: [PATCH 1/5] rugnux: a short usage - one section per mode, the basic options, and a pointer to every option The usage printed with no arguments was ~190 lines, many of them a paragraph each. It now opens with how to run each mode (rotation and stills under mx - auto-detected from whether the goniometer axis turns - refine, calibration, azint and scale), one to four example command lines each, then the ~18 options most runs need, and ends with where every option is described: the installed HTML (share/doc/jfjoch_rugnux/html/RUGNUX_CLI.html in the release archive) and Read the Docs. 64 lines including the banner, under 100 columns, no option's name or behaviour changed. The full option tables move from RUGNUX_ADVANCED.md to a page of their own, docs/RUGNUX_CLI.md, headed by the same per-mode section and the input rules the old usage carried (file order, several stills inputs, a model or reflection file told by content, the broker URL). Options the docs did not list are added: --no-refine-solvent, --dump-observations and the short -F; --mr, --crystfel-geometry and --refine-modules=diagnose take over the detail the old usage had. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- docs/RUGNUX.md | 12 +- docs/RUGNUX_ADVANCED.md | 204 +------------------------ docs/RUGNUX_CLI.md | 330 ++++++++++++++++++++++++++++++++++++++++ docs/documentation.rst | 1 + rugnux/rugnux_cli.cpp | 237 +++++++---------------------- 5 files changed, 392 insertions(+), 392 deletions(-) create mode 100644 docs/RUGNUX_CLI.md diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index 86b1e921e..319268178 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -39,8 +39,8 @@ the whole sweep gives. This is for rotation data and `--mode mx`; the collection is ignored. > **Note.** `rugnux` is under very active development. This page describes the tool and -> its options at a high level; the authoritative, always-current list of options is the program's -> own usage message — run `rugnux` with no arguments. +> its options at a high level. Run `rugnux` with no arguments for its usage - how to run each mode +> and the basic options; every option is on [Command-line options](RUGNUX_CLI.md). ```{contents} On this page :local: @@ -56,7 +56,7 @@ the data, and scaling and merging run by default. `-o` names the output files (d **We expect no need to tune anything else.** The defaults are meant to be the answer, and a run that needed a flag to do well is one we want to hear about. The other options exist for unusual cases and -for diagnosis; they are listed under [Command-line options](RUGNUX_ADVANCED.md#command-line-options). +for diagnosis; they are listed under [Command-line options](RUGNUX_CLI.md). ### Rotation data @@ -249,7 +249,7 @@ A few things worth knowing before reaching for more flags: with `Killed` ran out of memory ([When host memory runs out](RUGNUX_INSTALL.md#when-host-memory-runs-out)). With any question about a run, send `myrun_report.txt` and `myrun_log.txt`. - Everything else is in [Running Rugnux](RUGNUX_TUTORIAL.md#running-rugnux) and the full - [Command-line options](RUGNUX_ADVANCED.md#command-line-options). + [Command-line options](RUGNUX_CLI.md). ## The rest of the manual @@ -272,8 +272,10 @@ One page per job, so the answer needed is near the top of a short page: Phaser, SHELXC/D/E and, for small molecules, SHELXT / SHELXL. - [The results report](RUGNUX_REPORT.md) — the `KEY= value` interface, sweep quality and the anisotropy section. +- [Command-line options](RUGNUX_CLI.md) — how to run each mode, and every option with what it + does. - [Advanced usage](RUGNUX_ADVANCED.md) — reference data and the indexing ambiguity, model - validation, re-merging, and the full command-line option tables. + validation and re-merging. - [Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md) — every model and reference option, and decision by decision what a model can and cannot change. - [Detector calibration](RUGNUX_CALIBRATION.md) — the geometry from a calibrant's powder rings diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 0981c315b..609ff86d4 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -1,7 +1,7 @@ # Advanced Rugnux The machinery behind a run that needs more than the defaults: external reference data, model -validation, re-merging without re-integration, and the full option tables. +validation and re-merging without re-integration; the option tables are on [Command-line options](RUGNUX_CLI.md). ```{contents} On this page :local: @@ -338,204 +338,4 @@ the broker is not a file here, and has none. ## Command-line options -General: - -| Option | Description | -| --- | --- | -| `-o, --output-prefix ` | Output file prefix (default: `output`) | -| `-N, --threads ` | Number of worker threads (default, and for any value ≤ 0: every CPU the run may use - its affinity and cgroup quota). Some stages take fewer, because past a point more workers make them slower: the per-image loop of `--mode mx` uses at most 16 per GPU unless `-N` was given a positive value, and first-pass spot finding and the beam-stop pre-scan have ceilings of their own that `-N` does not lift. On a small card the two ceilings, and the per-image loop's 16 where `-N` was left at its default, come down to as many workers as keep their buffers (about 24 bytes per detector pixel each) within half of the card's total memory. Scaling, merging and the space-group search use the full count | -| `-s, --start-image ` | First image to process (default: 0) | -| `-e, --end-image ` | End image number, exclusive: `-s 0 -e 100` processes images 0-99 (default: all) | -| `-t, --stride ` | Process every *n*-th image (default: 1) | -| `-v, --verbose` | Show the whole log on the terminal in place of the steps; `_log.txt` is written either way | -| `--simulate-live ` | Read a stored input as the collection it was while being written: image *i* becomes readable (*i*+1)·`ms` ms after the start (0: all at once). The run takes the live path a broker URL takes, with the same result; for testing the live workflow without a broker. `--mode mx` only | -| `--fast-live` | With `--simulate-live` or a broker URL: run as the broker's rugnux live mode does — every decision stands where it was first taken, nothing is re-checked or run again once the sweep is in, the integration radius is fixed, and SIGINT cuts the run to what it has established. A difficult crystal can come out worse | -| `--no-gpu` | Run on the CPU only, as on a machine without a GPU, even where one is present: CUDA is not touched (nor, on macOS, the Apple GPU that Bragg integration, the model density and the molecular-replacement rotation function otherwise run on) and every GPU step takes its CPU path (`-X FFT` indexes with FFTW and `-X FFBIDX` with its CPU implementation; `-X cuFFT` is refused) | - -Mode — `--mode ` (default `mx`): - -| Value | Description | -| --- | --- | -| `mx` | Full analysis — spot finding, indexing, integration and merging | -| `azint` | Only azimuthal integration (no spot finding/indexing); writes `_process.h5` | -| `scale` | Only re-scale/merge the already-integrated reflections in the input `_process.h5` (no re-integration) | -| `calibration` | Determine the detector geometry from powder rings; writes `.poni` and `.json` | -| `refine` | Refine the `--model` against merged structure factors (an MTZ or a structure-factor mmCIF) and process nothing; see [Refining a model against structure factors](RUGNUX_MODEL_AND_REFERENCE.md#refining-a-model-against-structure-factors-rugnux---mode-refine) (`--refine-score-only`, `--refine-d-min`) | - -Refinement (`--mode refine`): the columns are found, not named - amplitudes with their sigmas (F/SIGF, -FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by -French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, -or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none. -The model is taken as placed in the data's cell, and refined in the data's space group. - -| Option | Description | -| --- | --- | -| `--refine-score-only` | Score the model as given and refine nothing: R-work/R-free, the scale, the bulk solvent, the likelihood and the geometry, each term under `REFINE_SCORE_` (`--developer` prints them all) | -| `--refine-d-min ` | Leave out reflections finer than this (Å) | - -Calibration (`--mode calibration`): - -| Option | Description | -| --- | --- | -| `--calibrant ` | Powder standard: `lab6` \| `agbh` \| `ceo2` \| `si` \| `ice` (default `lab6`, case-insensitive). A cell given with `-C` overrides it: that cell is the standard, its rings enumerated from the cell, with the absences of `-S` where given | -| `--calibration ` | How the rings are measured: `rings` \| `spots` (default `rings`; see above). `rings` defaults `--azim-phi-bins` to 32 | -| `--no-refine-tilt` | Do not refine the detector tilt: hold rot1/rot2 at the header value and fit only the beam centre and the distance, for a calibration handed to a program that cannot express a tilted detector (XDS). On stills it holds the tilt in the geometry refinement too | - -Detector mask: - -| Option | Description | -| --- | --- | -| `--detect-beam-stop[=N\|off]` | Find the beam stop and its holder in a projection of N images and add them to the pixel mask as bit 9, so nothing shadowed by them is integrated. **On by default** (60 images); `=off` disables. A reflection whose signal disk reaches into the shadow is not integrated: the rest of such a disk lies in the stop's half-shadow and reads low. Pixels the finder reports that hold the detector's no-data mark (a PILATUS gap) are masked as dead instead | - -Geometry: - -| Option | Description | -| --- | --- | -| `--beam-center-check[=off]` | Measure the beam centre from the isotropy of the scattered background on **every** run, report how far the file's value is from it (against how right this geometry needs it to be), and on rotation data index a **second first pass** at the measured centre. The fit reads the projection `--detect-beam-stop` already builds, so it costs no extra frames. **On by default**; `=off` disables it, and with it the measured centre's part in the post-refinement bound. The measured centre is adopted where the file's centre indexes nothing and the measured one indexes a majority; where the two return cells related by an integer volume factor (2 to 4) and the measured centre's cell, larger or smaller, carries materially more of the pooled validation spots against its own chance level; and, on a two-pass run, where the first pass is run at both centres and the measured one merges better — done where the two give the same cell in different metric symmetries, where they agree only once less of each frame is read, and where they agree but lie further apart than the geometry absorbs and the file's centre merges inconsistently in the lowest-resolution shell. Otherwise the file's centre is kept. On stills, where there is no first pass, the file's centre, the measured one and the point-symmetry capture's are each indexed on a sample of frames and the one putting materially more spots on the lattice is taken; with `-C` the detector distance is searched too (30-3000 mm, the file's value one hypothesis). Still runs when `--beam-x`/`--beam-y` are given. See [§1.4](CPU_DATA_ANALYSIS_IMAGE.md) | - -Spot finding: - -| Option | Description | -| --- | --- | -| `--spot-sigma ` | Noise sigma level for spot finding (default: 4.0) | -| `--spot-threshold ` | Photon-count threshold for spot finding (default: 10) | -| `--adaptive-spots` | Self-calibrating detection (**default**, stills and rotation alike): the strong-pixel threshold comes from each image's own per-resolution-ring noise instead of the fixed `--spot-threshold`, so one setting adapts across datasets (no per-dataset `--spot-threshold`/`--spot-sigma` tuning) | -| `--no-adaptive-spots` | Turn adaptive detection off and use the fixed `--spot-threshold` / `--spot-sigma` finder | -| `--spot-false-pixels ` | Adaptive-detection operating point: expected noise pixels tolerated per frame (default: 100; implies `--adaptive-spots`) | -| `--spot-high-resolution ` | High-resolution limit for spot finding, Å. Omitted (or 0): no resolution clipping — spot finding extends as far as the detector reaches, for rotation data as well as stills | -| `--spot-low-resolution ` | Low-resolution limit for spot finding, Å (default: 50; lower it, e.g. 24, to exclude the direct-beam halo on weak serial data; 0 removes the limit) | -| `--min-pix-per-spot ` | Minimum connected strong pixels per spot. **If omitted, min-pix is chosen per image** (stills indexing): the frame is scored at min-pix 2/3/1 (one lattice search per frame, refined on each list) and the one maximising indexed-spot count × indexed fraction is kept. Give an explicit value to force a fixed min-pix instead. | -| `--max-spots ` | Maximum spots kept per image (the strongest ones) and handed to indexing. **If omitted, the budget is measured** on rotation data: the first pass reads how deep into an image's spot list its spots still lie on the lattice it found, and the run keeps that many (never more than 1000). Give a value to pin it. Stills always use the fixed 1000. | -| `--detect-ice-rings[=on\|off]` | Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling. Default: the master file's `detect_ice_rings`, or — where the file carries no such key — **on for rotation and off for stills** | - -Azimuthal integration (the radial profile behind the per-image ice-ring score). Every *q* here is -*q* = 2π/*d*, in Å⁻¹: - -| Option | Description | -| --- | --- | -| `-q, --azim-q-spacing ` | Q bin spacing, 1/Å (default: 0.01; finer resolves the narrow ice rings) | -| `--azim-min-q ` | Minimum Q, 1/Å | -| `--azim-max-q ` | Maximum Q, 1/Å. Omitted: integration extends to the highest Q the detector reaches. The adaptive spot finder shares these Q bins, so this also sets how far self-calibrating detection can see | -| `--azim-phi-bins ` | Number of azimuthal (phi) bins (default: 1) | -| `--polarization-correction ` | Enable/disable the azimuthal polarization correction | -| `--solid-angle-correction ` | Enable/disable the azimuthal solid-angle correction | - -Indexing: - -A dataset with a **rotation goniometer axis** is processed as rotation data (two-pass rotation -indexing) by default; a dataset without one is processed as independent stills. `--force-still` -overrides the former; the `-R` / `--single-pass-rotation` / `--force-rotation-lattice` flags request -rotation explicitly and pick the pass or lattice. - -The FFT search looks for cell axes between 10 Å (`--fft-min-unit-cell`) and a default **longest -axis of 500 Å**, which has no flag of its own — a reference cell (`-C`) moves both bounds to cover -the cell it names. A de-novo rotation run also tries a second first-pass hypothesis with the short -end lowered to 5 Å, so a small-molecule cell below the 10 Å floor is indexed on its true axes -rather than as a supercell of them; the standard pass's answer stands unless that evidence takes it. An axis beyond the search's reach is **not refused**: the run returns a -plausible shorter sub-cell or an axis harmonic and processes it happily, so a cell that comes out -at a half or a third of the expected long axis should be read as this limit, not as the crystal. -For very long axes the direction grid's angular resolution binds well below 500 Å — see the -[analysis reference](CPU_DATA_ANALYSIS_INDEXING.md#5-fft-indexing-unknown-unit-cell) on FFT indexing. - -| Option | Description | -| --- | --- | -| `--force-still` | Treat a rotation (goniometer) dataset as independent stills instead of rotation | -| `-X, --indexing-algorithm ` | `Auto` (default) \| `FFT` \| `cuFFT` \| `FFTW` \| `FFBIDX` \| `None`, case-insensitive. `FFT` is the de-novo FFT search, on the GPU (cuFFT) where there is one and `--no-gpu` is not given, otherwise on the CPU (FFTW); `cuFFT` and `FFTW` name one implementation each; `cuFFT` needs a GPU and is refused without one. `FFBIDX` (known cell, `-C`) runs on the GPU where there is one and otherwise on the CPU. `Auto` takes `FFBIDX` on stills with a known cell, and `FFT` otherwise; rotation data always use the FFT search | -| `-C, --unit-cell ` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`). On rotation data it also widens the FFT search to cover the cell given, at both ends: the longest axis looked for is raised to reach it, and `--fft-min-unit-cell` is lowered to admit it | -| `--fft-min-unit-cell ` | Shortest cell axis the FFT search accepts, Å (default: 10). A candidate with a shorter axis is discarded — but a de-novo rotation run also runs a **second first-pass hypothesis with the floor lowered to 5 Å by default**, adopted when the standing cell turns out to be an integer supercell of what it finds, so a small-molecule cell is indexed without this flag. `-C` lowers the floor on its own to cover the cell given (and the second hypothesis then stays out of the way). On stills, where neither applies, a crystal below the floor cannot be indexed unless this is lowered | -| `--min-indexed-spots ` | Spots a frame must have on the lattice before it counts as indexed (default: 6, minimum 4). It sets the reported indexing rate and the count the rotation first pass ranks candidate lattices by; whether the run has a lattice at all is decided on the pooled spots instead ([§4.1](CPU_DATA_ANALYSIS_INDEXING.md#41-indexed-spot-decision-inlier-test)), and integration is not gated by it | -| `-S, --space-group ` | Space group number (`96`) or Hermann-Mauguin symbol (`P43212`) — for indexing and scaling. On serial stills whose group admits two indexings, it also lets the crystals be put on one from the data ([the indexing ambiguity](#the-indexing-ambiguity)) | -| `-r, --geom-refine ` | Geometry refinement: `none` \| `orientation` \| `beam_and_lattice` (default) \| `flex` (try all three per image, keep whichever indexes the most spots; alias `multi`) | -| `-R, --two-pass-rotation[=num]` | Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100) | -| `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) | -| `--force-rotation-lattice ` | Force rotation lattice (9 floats, Å), skipping the first pass | -| `--rotation-no-postrefine` | Rotation: disable the default-on two-pass geometry post-refine (see the rotation section) | -| `--prepass-fraction ` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep, but at least 120° of it (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame. The pre-scan and the first-pass indexing sample the same part of the sweep | -| `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (`-C` / `-z`), a stationary goniometer axis counting as stills, `=off` disables | -| `--index-ice-rings[=on\|off]` | Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: **off**; no effect without `--detect-ice-rings`, which does the flagging) | - -Indexer choice in brief: `ffbidx` (GPU or CPU) refines toward a **known cell** and is best for sparse -serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rotation data. See the -[CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS_INDEXING.md) for the algorithms. - -Scaling and merging: - -| Option | Description | -| --- | --- | -| `--no-merge` | Skip scaling and merging, which are on by default; the per-image `_process.h5` is then written instead of the merged files (`_unmerged.mtz` and the results report are still written) | -| `-A, --anomalous` | Anomalous mode: merge each Bijvoet hand as its own unique reflection, so multiplicity, completeness, ⟨I/σ⟩ and the error model are counted anomalously. A default rotation merge already writes `I(+)`/`I(-)` (see [Reflection-file conventions](RUGNUX_INTEGRATION.md#reflection-file-conventions)); `-A` changes the counting basis, not whether the anomalous signal is in the file | -| `--scale-fulls` / `--no-scale-fulls` | rot3d: refit a per-frame scale on the combined fulls (XDS order, Unity model); on by default for rotation data, off for stills | -| `--smooth-g[=deg]` | rot3d: smooth the per-frame scale *G* over a degree range before the 3D combine (XDS DELPHI-like; default 5° for rotation, 0 = off) | -| `--no-scaling-corrections` | rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls (see below) | -| `--relative-b[=deg]` | rot3d: fit a per-batch relative-*B* beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise) | -| `--simple-stills` | Stills: treat every reflection as a full (*p* = 1, single-pass scale/merge) — disables the default-on physical partiality post-refinement | -| `--no-expected-variance-merge` | Stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting | -| `--capture-uncertainty ` | Systematic sigma num·(1−captured fraction)·⟨I⟩: on rot3d fulls the share of the rocking curve not captured, on stills the share of each partial its partiality does not see (default: 1.0 for rotation, 0 otherwise) | -| `--min-captured-fraction ` | rot3d: drop a combined full whose rocking curve was captured below this fraction — edge-of-sweep truncated fulls (default: 0.7 for rotation, 0 otherwise; 0 = off) | -| `--scaling-high-resolution ` | High-resolution limit for scaling, Å — manual override (default: no limit; disables the automatic cutoff below) | -| `--scaling-low-resolution ` | Low-resolution limit of the scale fit, Å (default: 50, which is also XDS's own default; 0 removes the limit). Reflections coarser than it are merged and written but kept out of every fit of the scale model: on a large cell a few dozen of them, measured on a background the beam stop depresses, bent the model enough to move the merged 10–50 Å intensities by tens of percent. Which of them are measured at all is the beam-stop mask's decision - a reflection whose signal disk reaches the shadow is not integrated (`--detect-beam-stop`) | -| `--resolution-cutoff ` | Automatic high-resolution cutoff for the written reflections and reported shells: `cc-logistic` \| `off` (default: `cc-logistic`; ignored when `--scaling-high-resolution` is set) | -| `--resolution-cc-target ` | CC1/2 target defining the `cc-logistic` fall-off (default: 0.30) | -| `--resolution-shells ` | Number of resolution shells in the reported statistics table (default: 9). The bins are equal steps in 1/d² between the lowest- and highest-resolution reflection merged, which is XDS's rule, and 9 is XDS's count — so at the same resolution limits the two tables have the same shells and can be read row for row | -| `--report-resolution [,]` | Also report the merging statistics over this resolution range, Å (either order; `dmax` defaults to the run's own low-resolution limit), as a second table beside the run's own — the `REFRES_*` keys of the [results report](RUGNUX_REPORT.md#the-reference-range-table) — binned from the same merged reflections, so a run can be read against another program's table at that program's range. Report-only: the cut, the scaling, the error model and every decision stay the run's own, and shells finer than the run's own limit are reported as not merged rather than filled from data the run did not keep | -| `--min-partiality ` | Minimum partiality to accept a reflection (default: 0.02 for rotation, 0.5 for stills) | -| `--ice-min-score ` | Ice-presence gate: the measured per-run ice score (1 = no ice) a dataset must reach before **any** ice handling is applied — the flagging and the exclusion from scaling (default: 1.5; 0 = no gate). The fixed hexagonal bands cover 16–26 % of the unique reflections whether or not the crystal has ice, so handling ice on a clean crystal only costs completeness | -| `--ice-min-spot-ratio ` | The second ice-presence channel: found **spots** on the hexagonal rings over the same q width of ice-free flanks beside them (1 = spots spread evenly). Ice in large crystallites diffracts as discrete spots and leaves the radial profile flat, so `--ice-min-score` alone is blind to it (default: 2.0; 0 disables this channel) | -| `--reject-outliers ` | Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for rotation, 10 for stills; 0 = off) | -| `--min-image-cc ` | Per-image CC limit, percent (default: no limit) | -| `--search-min-zeta ` | De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry \|ζ\| reaches this, and report both answers (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law. Where the two searches disagree, the merge of all the observations decides — as it always has for the systematic absences | -| `--mosaicity ` | Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed | -| `--scaling-iterations ` | Cap on the per-frame scaling iterations; the loop stops earlier once the scales settle, and a merge that hits the cap is flagged `SCALING_NOT_CONVERGED` (default: 100) | -| `-z, --reference ` | Reference data of the same crystal form — an MTZ or a PDB structure-factor mmCIF (`-sf.cif`), gzipped or not, recognised by content; `--reference-mtz` is the same option: fixes the space group, its setting and the cell, resolves the [indexing ambiguity](#the-indexing-ambiguity), hands over the R-free set and reports CCref. Not a scale anchor | -| `--reference-column