Jungfraujoch/docs/RUGNUX.md

rugnux

rugnux is the offline crystallographic data-analysis tool of Jungfraujoch — the data-processing half of the system (see Naming for where the name comes from). It takes an existing HDF5 dataset, runs the full analysis pipeline — spot finding, indexing, geometry refinement, Bragg integration and (optionally) scaling and merging — and writes the results to a _process.h5 file, plus reflection files (.mtz/.cif/.hkl) when merging is requested.

It runs the same analysis code as the online and interactive tools, just driven from the command line over a file rather than a live detector stream.

Note. rugnux is under very active development. This page describes the tool and its options at a high level; the authoritative, always-current list of options is the program's own usage message — run rugnux with no arguments.

Where it fits among the three analysis tools

Tool	Mode	Driven by	Output
jfjoch_broker	Online, real-time streaming analysis on FPGA + GPU	HTTP/REST + ZeroMQ	Live results and statistics, images streamed to jfjoch_writer
jfjoch_viewer	Interactive, on-screen exploration	Qt desktop application	On screen; a processing job can write the same files as rugnux
rugnux	Offline batch processing of a stored dataset	Command-line interface	_process.h5, and .mtz/.cif/.hkl when merging

Use rugnux to re-analyse data after acquisition, to experiment with processing parameters, or to produce merged intensities for downstream structure solution.

Hardware

As with the rest of Jungfraujoch, serious performance requires an NVIDIA GPU. The CUDA build provides the GPU fast-feedback indexer (ffbidx) and the GPU FFT indexer (fft); without CUDA only the CPU fftw indexer is available. Spot finding, integration and scaling run on the CPU and scale with the thread count (-N).

Input and output

Input is a single Jungfraujoch HDF5 master file (NXmx-based). Spots are always found by rugnux itself, including for the two-pass rotation first pass — the spot lists a dataset may already carry were found online, at the acquisition's threshold and with its ice-band spots already discarded, so reusing them would hide the spot-finding settings from the lattice search.

Output (controlled by -o, --output-prefix, default output):

• <prefix>_process.h5 — NXmx-compliant HDF5 with derived metadata (spots, indexing, integration, azimuthal integration, per-image statistics). See HDF5 / NeXus data format for the layout. Written by default only when not merging (i.e. under --no-merge); add --write-process-h5 to also write it when merging.

• Merging is on by default (--no-merge disables it). The merged reflections are written in three formats — each has its uses downstream:

  • <prefix>.mtz — CCP4 MTZ (IMEAN/I(+)/I(-), French–Wilson F, FreeR_flag) for the CCP4 / phenix reflection tools.
  • <prefix>.cif — mmCIF, for deposition and as the self-describing native format (also carries the merging statistics, ISa, twinning and radiation-damage indicators).
  • <prefix>.hkl — SHELX HKLF 4 text (h k l I σ(I), fixed 3I4,2F8.2), the direct input for SHELXC / ANODE / SHELXD. Bijvoet mates are written separately (I(+) at +hkl, I(-) at -hkl) so the anomalous signal is preserved; intensities are put on a common scale so the largest value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file ends with the 0 0 0 terminator record.

  All three carry the refined unit cell (from rotation indexing) and the space group determined from systematic absences (constrained to the indexed lattice symmetry). No-reference scaling additionally emits per-iteration <prefix>_iterN_scale.dat.

Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the console. By default the written resolution is trimmed automatically where CC1/2 falls off (--resolution-cutoff cc-logistic, CC1/2 target 0.30); set --scaling-high-resolution to fix the limit by hand, or --resolution-cutoff off to keep the full range.

Validating against a model (rugnux --model)

Given a PDB atomic model of the same structure, --model model.pdb scales the model structure factors to the merged amplitudes — fitting a flat bulk-solvent contribution and an overall anisotropic B — and reports R-work / R-free and the mean 2Fo-Fc density at the atom centres. It also writes <prefix>_2fofc.ccp4, <prefix>_fofc.ccp4 and <prefix>_maps.mtz next to the merged reflections. Nothing about the model is refined; it is only re-fractionalized into the data cell, so a deposited model with a slightly different cell still lines up.

It is a data-quality lens, independent of the internal statistics: R-free measures the merged intensities against external truth, where CC1/2 and R_meas only measure them against themselves. It also settles the two things merged intensities alone cannot: the enantiomorph (data merged in P4₁2₁2 against a P4₃2₁2 model are reindexed into the model's hand), and — when no reference MTZ has already fixed it — a merohedral indexing ambiguity, by keeping the candidate reindexing with the lowest R-free.

Re-scaling and re-merging (rugnux --mode scale)

The scale mode re-scales and merges the already-integrated reflections stored in a _process.h5 file, without re-running spot finding or integration. Use it to re-merge quickly with a different space group, resolution limit, anomalous setting or reference MTZ. It reuses the same -o/-N/-s/-e/-S/-A/-B/-z/--scaling-* options as the full run, and (unlike the full pipeline) does not run a space-group search, so pass -S for the correct symmetry.

Detector calibration from powder rings (rugnux --mode calibration)

The calibration mode determines the detector geometry — PONI x/y, the two tilts rot1/rot2 and the distance — from the powder rings of a calibrant, and writes it as a pyFAI <prefix>.poni file alongside a printed report of how far each parameter moved from the header. Bragg data pin the beam centre worst (it is gauge-coupled to the crystal orientation); a powder ring has no orientation to be coupled to, so this is the measurement that fixes it.

rugnux --mode calibration --calibrant lab6 -N 8 -o det LaB6_master.h5

--calibrant takes lab6, agbh (silver behenate), ceo2, si or ice, case-insensitively. ice calibrates a real experiment against its own ice rings — no calibrant exposure needed — and is the reason a calibrant is a list of ring positions rather than a unit cell: hexagonal ice is P6₃/mmc, so rings enumerated from its cell would include systematically absent ones.

--calibration picks how the rings are measured, and both use the whole dataset — -s/-e/-t select which images:

• rings (default) sums the (q × azimuth) azimuthal profile over every processed image into one map and fits the ring arcs in it. A powder ring is an arc, not a set of spots, and the summed profile measures it at every azimuth with all the run's counts behind it. It needs the profile to be binned in azimuth, so this mode defaults --azim-phi-bins to 32.
• spots pools the found spots of every processed image and fits those. It determines the centre from scratch (a Hough circle vote, which quantises it to a whole pixel) and then refines.

Both routes read the ring position out of a binned profile or a spot centroid, so the radial sampling matters: at a long detector distance the default 0.01 Å⁻¹ q bin is several pixels wide and quantises the rings route accordingly — pass a finer --azim-q-spacing there (the total q × azimuth bin count must stay under 65534).

The report prints the fitted geometry, the scatter of the ring points about the fitted rings and the standard error that implies on the centre. That error is formal: it measures the scatter of the points, not whether the rings themselves are trustworthy, so it stays small when a fit goes wrong for a structural reason — one visible ring, or ice that is textured rather than smooth.

Both the PONI (the point of normal incidence, which is what a .poni file stores) and the direct beam (where the beam lands, which is what most other programs call the beam centre) are printed. They differ by distance × tan(rot) once the detector is tilted, which on a 0.3° tilt at 300 mm is several pixels — enough to look like a disagreement with another program when there is none.

Quick start

Rotation data

Index, integrate, scale and merge a rotation sweep, fully de novo:

rugnux rotation_master.h5 \
    -o rotation_run -N 32 \
    --scaling-high-resolution 1.4

Because the dataset carries a rotation goniometer axis, it is processed as rotation data by default: two-pass rotation indexing (index the sweep once, then process every frame against that lattice) with the rot3d partiality model (rotation partials combined into 3D fulls). Scaling and merging run by default (for both rotation and stills; --no-merge turns them off); the unit cell is taken from the rotation indexer and the space group is determined from systematic absences, and both are written into the merged .cif.

Run fully de novo (no -C/-S) for the best result — supplying a cell or space group up front tends to degrade low-symmetry cases. --scaling-high-resolution (set it to your expected resolution) sharpens both the space-group search and the error model. To tune the first pass use --two-pass-rotation=100 (or -R100 — the first-pass image count); to force the sweep to be treated as independent stills use --force-still.

By default a rotation run also post-refines the geometry in a second pass: the first pass integrates and merges at the header geometry, then the detector distance + beam centre and the crystal cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a small < 1 % move, with the gauge-weak beam centre restrained toward the header), and the second pass re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical <prefix>_* output; the header-geometry pass is kept alongside as <prefix>_01_* for comparison. Disable it with --rotation-no-postrefine.

After the per-frame scale-fulls step, rotation scaling applies three correction surfaces, on by default (--no-scaling-corrections disables all):

• Decay — a global Debye–Waller relative-B over the run, for the radiation damage that weakens later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame scale cannot remove). It only engages when the total relative-B exceeds a physical floor (2 Ų). An optional --relative-b[=deg] extends this single global rate to a smooth per-batch relative-B curve (default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for crystals whose decay is non-linear in dose.
• Absorption — a smooth multiplicative factor over the diffracted-beam direction in the goniometer frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the intensities, so it measurably lowers R_free.
• Modulation — a smooth multiplicative factor over the position where a reflection lands on the detector (a flat-field: detector-response and geometric systematics that vary across the detector plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, which over-determines the surface. Because it lives in the detector frame (not the rotation) the same correction concept applies to stills. This is the largest of the three on JUNGFRAU data — it lowers R_meas by several to tens of percent on datasets that carry a detector systematic, while holding or improving CC_1/2 and the anomalous signal.

All three are cross-validated — fitted on even-numbered frames and kept only if they improve the held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is scored as a σ-independent, R_meas-like fractional deviation, so a surface can never pass cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op rather than a source of added noise, which is why they are safe to leave on.

Independently of any correction, a rotation run prints a radiation-damage report — the per-image scale correlation-to-merge and mosaicity versus dose, and the relative B-factor change over the run (first→last) together with a per-batch relative-B curve, also written to the merged mmCIF. It is a data-quality-vs-dose diagnostic and never alters the merged intensities.

Still / serial data

A dataset with no goniometer axis (e.g. a serial grid scan) is processed as independent stills automatically — no flag needed. Known-cell indexing with the GPU fast-feedback indexer, then merge against a reference structure:

rugnux serial_master.h5 \
    -o serial_run -N 32 \
    -X ffbidx -C 79,79,38,90,90,90 -S 96 \
    -z reference.mtz \
    --scaling-high-resolution 1.8

ffbidx requires a known cell (-C) and is the indexer of choice for sparse serial stills. The self-calibrating spot finder is on by default for both workflows (--no-adaptive-spots turns it off), and for serial stills leave --min-pix-per-spot unset so it is chosen per image — across the still-target battery this combination raises the indexing rate and typically extends resolution over a fixed threshold and fixed min-pix, at equal or better CC½. (You can still pin a fixed threshold with --spot-sigma / --spot-threshold and a fixed min-pix with --min-pix-per-spot.) If a dataset does carry a goniometer axis but you want per-frame stills processing anyway, add --force-still.

Command-line options

General:

Option	Description
-o, --output-prefix <txt>	Output file prefix (default: output)
-N, --threads <num>	Number of worker threads (default: all hardware threads)
-s, --start-image <num>	First image to process (default: 0)
-e, --end-image <num>	Last image to process (default: all)
-t, --stride <num>	Process every n-th image (default: 1)
-v, --verbose	Verbose output

Mode — --mode <name> (default mx):

Value	Description
mx	Full analysis — spot finding, indexing, integration and merging
azint	Only azimuthal integration (no spot finding/indexing); writes <prefix>_process.h5
scale	Only re-scale/merge the already-integrated reflections in the input _process.h5 (no re-integration)
calibration	Determine the detector geometry from powder rings; writes <prefix>.poni

Calibration (--mode calibration):

Option	Description
--calibrant <name>	Powder standard: lab6 | agbh | ceo2 | si | ice (default lab6, case-insensitive)
--calibration <txt>	How the rings are measured: rings | spots (default rings; see above). rings defaults --azim-phi-bins to 32

Spot finding:

Option	Description
--spot-sigma <num>	Noise sigma level for spot finding (default: 4.0)
--spot-threshold <num>	Photon-count threshold for spot finding (default: 10)
--adaptive-spots	Self-calibrating detection (default, stills and rotation alike): the strong-pixel threshold comes from each image's own per-resolution-ring noise instead of the fixed --spot-threshold, so one setting adapts across datasets (no per-dataset --spot-threshold/--spot-sigma tuning)
--no-adaptive-spots	Turn adaptive detection off and use the fixed --spot-threshold / --spot-sigma finder
--spot-false-pixels <num>	Adaptive-detection operating point: expected noise pixels tolerated per frame (default: 100; implies --adaptive-spots)
--spot-high-resolution <num>	High-resolution limit for spot finding, Å. Omitted (or 0): no resolution clipping — spot finding extends as far as the detector reaches, for rotation data as well as stills
--spot-low-resolution <num>	Low-resolution limit for spot finding, Å (default: 50; lower it, e.g. 24, to exclude the direct-beam halo on weak serial data)
--min-pix-per-spot <num>	Minimum connected strong pixels per spot. If omitted, min-pix is chosen per image (stills indexing): the frame is indexed at min-pix 3/2/1 and the one maximising indexed-spot count × indexed fraction is kept. Give an explicit value to force a fixed min-pix instead.
--max-spots <num>	Maximum spots kept per image (the strongest ones) and handed to indexing (default: 1000)
--detect-ice-rings[=on|off]	Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling. Default: the master file's detect_ice_rings, or — where the file carries no such key — on for rotation and off for stills

Azimuthal integration (the radial profile behind the per-image ice-ring score):

Option	Description
-q, --azim-q-spacing <num>	Q bin spacing, 1/Å (default: 0.01; finer resolves the narrow ice rings)
--azim-min-q <num>	Minimum Q, 1/Å
--azim-max-q <num>	Maximum Q, 1/Å. Omitted: integration extends to the highest Q the detector reaches. The adaptive spot finder shares these Q bins, so this also sets how far self-calibrating detection can see
--azim-phi-bins <num>	Number of azimuthal (phi) bins (default: 1)
--azim-sigma-clip <num>	Sigma-clip the azimuthal profile: repeat the integration twice more, each time rejecting pixels further than num standard deviations from their own bin's mean (default: 0 = off; must be ≥ 2). A powder ring is azimuthally smooth and survives the clip, Bragg peaks do not, so the profile becomes the smooth background under the peaks rather than the bin mean — which is what an ice-ring measurement wants and what a ring's integrated intensity does not. Costs one extra pass over the image per clip, ~3× the azimuthal-integration time (measured 0.02 → 0.06 ms per image), and nothing else in the frame
--polarization-correction <on|off>	Enable/disable the azimuthal polarization correction
--solid-angle-correction <on|off>	Enable/disable the azimuthal solid-angle correction

Indexing:

A dataset with a rotation goniometer axis is processed as rotation data (two-pass rotation indexing) by default; a dataset without one is processed as independent stills. --force-still overrides the former; the -R / --single-pass-rotation / --force-rotation-lattice flags request rotation explicitly and pick the pass or lattice.

Option	Description
--force-still	Treat a rotation (goniometer) dataset as independent stills instead of rotation
-X, --indexing-algorithm <txt>	FFBIDX | FFT | FFTW | Auto | None
-C, --unit-cell <cell>	Reference unit cell "a,b,c,alpha,beta,gamma" (required by ffbidx)
-S, --space-group <num|symbol>	Space group number (92) or Hermann-Mauguin symbol (P43212) — for indexing and scaling
-r, --refine <txt>	Geometry refinement: none | orientation | beam_and_lattice (default) | flex (try all three per image, keep whichever indexes the most spots; alias multi)
-R, --two-pass-rotation[=num]	Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100)
--single-pass-rotation[=num]	Online-like single-pass rotation indexing (optional min angular range, deg)
--force-rotation-lattice <vec>	Force rotation lattice (9 floats, Å), skipping the first pass
--rotation-no-postrefine	Rotation: disable the default-on two-pass geometry post-refine (see the rotation section)
--refine-geometry[=N|off]	Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (-C / -z), =off disables
--index-ice-rings[=on|off]	Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: off; no effect without --detect-ice-rings, which does the flagging)

Indexer choice in brief: ffbidx (GPU) refines toward a known cell and is best for sparse serial stills; fft (GPU) / fftw (CPU) index de novo and suit strong rotation data. See the CPU/GPU data-analysis reference for the algorithms.

Scaling and merging:

Option	Description
--no-merge	Skip scaling and merging (on by default); write only the per-image _process.h5
-A, --anomalous	Anomalous mode (keep Friedel pairs separate)
--scale-fulls / --no-scale-fulls	rot3d: refit a per-frame scale on the combined fulls (XDS order, Unity model); on by default for rotation data, off for stills
--smooth-g[=deg]	rot3d: smooth the per-frame scale G over a degree range before the 3D combine (XDS DELPHI-like; default 5° for rotation, 0 = off)
--no-scaling-corrections	rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls (see below)
--relative-b[=deg]	rot3d: fit a per-batch relative-B beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise)
--simple-stills	Stills: treat every reflection as a full (p = 1, single-pass scale/merge) — disables the default-on physical partiality post-refinement
--no-expected-variance-merge	Stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting
--capture-uncertainty <num>	rot3d: systematic sigma on under-captured fulls, ~num·(1−captured_fraction)·I (default: 1.0 for rotation, 0 otherwise)
--min-captured-fraction <num>	rot3d: drop a combined full whose rocking curve was captured below this fraction — edge-of-sweep truncated fulls (default: 0.7 for rotation, 0 otherwise; 0 = off)
--scaling-high-resolution <num>	High-resolution limit for scaling, Å — manual override (default: no limit; disables the automatic cutoff below)
--resolution-cutoff <txt>	Automatic high-resolution cutoff for the written reflections and reported shells: cc-logistic | off (default: cc-logistic; ignored when --scaling-high-resolution is set)
--resolution-cc-target <num>	CC1/2 target defining the cc-logistic fall-off (default: 0.30)
--resolution-shells <num>	Number of resolution shells in the reported statistics table (default: 10)
--min-partiality <num>	Minimum partiality to accept a reflection (default: 0.02)
--ice-min-score <num>	Ice-presence gate: the measured per-run ice score (1 = no ice) a dataset must reach before any ice handling is applied — the flagging and the exclusion from scaling (default: 1.5; 0 = no gate). The eleven fixed hexagonal bands cover 16–26 % of the unique reflections whether or not the crystal has ice, so handling ice on a clean crystal only costs completeness
--ice-min-spot-ratio <num>	The second ice-presence channel: found spots on the hexagonal rings over the same q width of ice-free flanks beside them (1 = spots spread evenly). Ice in large crystallites diffracts as discrete spots and leaves the radial profile flat, so --ice-min-score alone is blind to it (default: 2.0; 0 disables this channel)
--reject-outliers <num>	Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for rot3d, off otherwise)
--min-image-cc <num>	Per-image CC limit, percent (default: no limit)
--search-min-zeta <num>	De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry |ζ| reaches this, and keep whichever search found more symmetry (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law. The point group only — the systematic absences always come from the merge of all the observations
--mosaicity <num>	Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed
--scaling-iterations <num>	Scaling iterations with no reference data (default: 3)
-z, --reference-mtz <file>	Reference MTZ (enables reference-driven scaling)
--reference-column <label>	Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…)
--model <file.pdb>	After merging, validate the merged intensities against this atomic model (see below)
--write-process-h5	Also write the (large) _process.h5 when merging (default: only .mtz/.cif)

Integration:

Option	Description
--integrator <txt>	Spot integrator: gaussian (profile-fit, default) | empirical | boxsum (classical fallback)
--integration-radius <r>	Signal-box radius r1, or r1,r2,r3 (px). One value ⇒ r2=r1+2, r3=r1+4
--background-clip <n>	Monochromatic (rotation + still): high-side clip of the background ring at mean + n·√mean (default 4; 0 = off). The default background estimator — it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Broadband data always clip, at 3σ; ignored by --integrator boxsum
--background-trim <f>	Use the old symmetric trimmed mean for the background ring instead of the clip, 0≤f<0.5 (0.10 was the former default). Switches --background-clip off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; 0 = plain ring mean
--background-radial[=on|off|auto]	Correct the background ring for the curvature of the radial background (default off). Disk and ring are concentric, so a background linear in position cancels between them and only curvature survives — which on a smooth ice ring reaches +26 counts on a single reflection. auto applies it per image where that image's ice score shows a smooth powder ring, since the model is a function of radius alone: on ice made of discrete crystallite spots there is no smooth ring and the correction makes the bias worse. Ignored by --integrator boxsum (no clip pass to take the curve from)
--integration-high-resolution <num>	High-resolution limit for prediction and integration. Omitted (or 0) means integration extends as far as the detector reaches — which is what the predictor can place on the detector anyway, since it rejects reflections that miss it. Set a value to integrate less than the detector offers
--max-hkl <n>	Predict reflections with |h|,|k|,|l| ≤ n (max 511). By default this is derived per crystal from the refined cell as ceil(max(a,b,c)/d_min) + 1, which is the exact bound: the predictor keeps only |q| ≤ 1/d_min and h = a·q, so no reflection can lie outside it and no candidate inside it is wasted on a shorter axis. Set it only to override that
--bandwidth <num>	Relative X-ray bandwidth FWHM (e.g. 0.01 for a 1% DMM); default from file or 0 (monochromatic)

Geometry overrides (defaults are taken from the input file; override them to reprocess with a corrected geometry):

Option	Description
--beam-x <num>	Beam centre X (pixel)
--beam-y <num>	Beam centre Y (pixel)
--detector-distance <num>	Detector distance (mm)
--wavelength <num>	Wavelength (Å)
--rot1 <num>	PONI detector rotation 1 (rad)
--rot2 <num>	PONI detector rotation 2 (rad)
--polarization <num>	Polarization factor