leonarski_f
67dca388bd
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
18 KiB
rugnux
rugnux is the offline crystallographic data-analysis tool of Jungfraujoch — the
data-processing half of the system (see Naming for where the name comes from).
It takes an existing HDF5 dataset, runs the full analysis pipeline — spot finding, indexing,
geometry refinement, Bragg integration and (optionally) scaling and merging — and writes the
results to a _process.h5 file, plus reflection files (.mtz/.cif/.hkl) when merging is
requested.
It runs the same analysis code as the online and interactive tools, just driven from the command line over a file rather than a live detector stream.
Note.
rugnuxis under very active development. This page describes the tool and its options at a high level; the authoritative, always-current list of options is the program's own usage message — runrugnuxwith no arguments.
Where it fits among the three analysis tools
| Tool | Mode | Driven by | Output |
|---|---|---|---|
jfjoch_broker |
Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to jfjoch_writer |
jfjoch_viewer |
Interactive, on-screen exploration | Qt desktop application | Displayed on screen (results not saved to disk) |
rugnux |
Offline batch processing of a stored dataset | Command-line interface | _process.h5, and .mtz/.cif/.hkl when merging |
Use rugnux to re-analyse data after acquisition, to experiment with processing
parameters, or to produce merged intensities for downstream structure solution.
Hardware
As with the rest of Jungfraujoch, serious performance requires an NVIDIA GPU. The CUDA build
provides the GPU fast-feedback indexer (ffbidx) and the GPU FFT indexer (fft); without CUDA
only the CPU fftw indexer is available. Spot finding, integration and scaling run on the CPU and
scale with the thread count (-N).
Input and output
Input is a single Jungfraujoch HDF5 master file (NXmx-based). If the dataset already contains stored spot lists, two-pass rotation indexing can reuse them instead of re-running spot finding on the first pass.
Output (controlled by -o, --output-prefix, default output):
-
<prefix>_process.h5— NXmx-compliant HDF5 with derived metadata (spots, indexing, integration, azimuthal integration, per-image statistics). See HDF5 / NeXus data format for the layout. Written by default only when not merging (i.e. under--no-merge); add--write-process-h5to also write it when merging. -
Merging is on by default (
--no-mergedisables it). The merged reflections are written in three formats — each has its uses downstream:<prefix>.mtz— CCP4 MTZ (IMEAN/I(+)/I(-), French–WilsonF,FreeR_flag) for the CCP4 / phenix reflection tools.<prefix>.cif— mmCIF, for deposition and as the self-describing native format (also carries the merging statistics, ISa, twinning and radiation-damage indicators).<prefix>.hkl— SHELX HKLF 4 text (h k l I σ(I), fixed3I4,2F8.2), the direct input for SHELXC / ANODE / SHELXD. Bijvoet mates are written separately (I(+)at+hkl,I(-)at-hkl) so the anomalous signal is preserved; intensities are put on a common scale so the largest value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file ends with the0 0 0terminator record.
All three carry the refined unit cell (from rotation indexing) and the space group determined from systematic absences (constrained to the indexed lattice symmetry). No-reference scaling additionally emits per-iteration
<prefix>_iterN_scale.dat.
Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
console. By default the written resolution is trimmed automatically where CC1/2 falls off
(--resolution-cutoff cc-logistic, CC1/2 target 0.30); set --scaling-high-resolution to fix the
limit by hand, or --resolution-cutoff off to keep the full range.
Re-scaling and re-merging (rugnux --scale)
The --scale mode re-scales and merges the already-integrated reflections stored in a
_process.h5 file, without re-running spot finding or integration. Use it to re-merge quickly with a
different space group, resolution limit, anomalous setting or reference MTZ. It reuses the same
-o/-N/-s/-e/-S/-A/-B/-z/--scaling-* options as the full run, and (unlike the full pipeline) does
not run a space-group search, so pass -S for the correct symmetry.
Quick start
Rotation data
Index, integrate, scale and merge a rotation sweep, fully de novo:
rugnux rotation_master.h5 \
-o rotation_run -N 32 \
--scaling-high-resolution 1.4
Because the dataset carries a rotation goniometer axis, it is processed as rotation data by
default: two-pass rotation indexing (index the sweep once, then process every frame against that
lattice) with the rot3d partiality model (rotation partials combined into 3D fulls). Scaling
and merging run by default (for both rotation and stills; --no-merge turns them off); the unit
cell is taken from the rotation indexer and the space group is determined from systematic absences,
and both are written
into the merged .cif.
Run fully de novo (no -C/-S) for the best result — supplying a cell or space group up front
tends to degrade low-symmetry cases. --scaling-high-resolution (set it to your expected
resolution) sharpens both the space-group search and the error model. To tune the first pass use
--two-pass-rotation=100 (or -R100 — the first-pass image count); to force the sweep to be
treated as independent stills use --force-still.
By default a rotation run also post-refines the geometry in a second pass: the first pass
integrates and merges at the header geometry, then the detector distance + beam centre and the crystal
cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a
small < 1 % move, with the gauge-weak beam centre restrained toward the header), and the second pass
re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical
<prefix>_* output; the header-geometry pass is kept alongside as <prefix>_01_* for comparison.
Disable it with --rotation-no-postrefine.
After the per-frame scale-fulls step, rotation scaling applies three correction surfaces, on by
default (--no-scaling-corrections disables all):
- Decay — a global Debye–Waller relative-B over the run, for the radiation damage that weakens
later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame
scale cannot remove). It only engages when the total relative-B exceeds a physical floor (2 Ų). An
optional
--relative-b[=deg]extends this single global rate to a smooth per-batch relative-B curve (default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for crystals whose decay is non-linear in dose. - Absorption — a smooth multiplicative factor over the diffracted-beam direction in the goniometer frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the intensities, so it measurably lowers Rfree.
- Modulation — a smooth multiplicative factor over the position where a reflection lands on the detector (a flat-field: detector-response and geometric systematics that vary across the detector plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, which over-determines the surface. Because it lives in the detector frame (not the rotation) the same correction concept applies to stills. This is the largest of the three on JUNGFRAU data — it lowers Rmeas by several to tens of percent on datasets that carry a detector systematic, while holding or improving CC1/2 and the anomalous signal.
All three are cross-validated — fitted on even-numbered frames and kept only if they improve the held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is scored as a σ-independent, Rmeas-like fractional deviation, so a surface can never pass cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op rather than a source of added noise, which is why they are safe to leave on.
Independently of any correction, a rotation run prints a radiation-damage report — the per-image scale correlation-to-merge and mosaicity versus dose, and the relative B-factor change over the run (first→last) together with a per-batch relative-B curve, also written to the merged mmCIF. It is a data-quality-vs-dose diagnostic and never alters the merged intensities.
Still / serial data
A dataset with no goniometer axis (e.g. a serial grid scan) is processed as independent stills automatically — no flag needed. Known-cell indexing with the GPU fast-feedback indexer, then merge against a reference structure:
rugnux serial_master.h5 \
-o serial_run -N 32 \
-X ffbidx -C 79,79,38,90,90,90 -S 96 \
--spot-sigma 4 \
-z reference.mtz \
--scaling-high-resolution 1.8
ffbidx requires a known cell (-C) and is the indexer of choice for sparse serial stills. For
weak serial data, tightening spot finding with --spot-sigma 4 typically raises the indexing rate
substantially. If a dataset does carry a goniometer axis but you want per-frame stills processing
anyway, add --force-still.
Command-line options
General:
| Option | Description |
|---|---|
-o, --output-prefix <txt> |
Output file prefix (default: output) |
-N, --threads <num> |
Number of worker threads (default: all hardware threads) |
-s, --start-image <num> |
First image to process (default: 0) |
-e, --end-image <num> |
Last image to process (default: all) |
-t, --stride <num> |
Process every n-th image (default: 1) |
-v, --verbose |
Verbose output |
Modes (default: full analysis — spot finding, indexing, integration and merging):
| Option | Description |
|---|---|
--azint-only |
Only run azimuthal integration (no spot finding/indexing); writes <prefix>_process.h5 |
--scale |
Only re-scale/merge the already-integrated reflections in the input _process.h5 (no re-integration) |
Spot finding:
| Option | Description |
|---|---|
--spot-sigma <num> |
Noise sigma level for spot finding (default: 3.0) |
--spot-threshold <num> |
Photon-count threshold for spot finding (default: 10) |
--spot-high-resolution <num> |
High-resolution limit for spot finding, Å (default: 1.5) |
--spot-low-resolution <num> |
Low-resolution limit for spot finding, Å (default: 50; lower it, e.g. 24, to exclude the direct-beam halo on weak serial data) |
--min-pix-per-spot <num> |
Minimum connected strong pixels per spot (default: 2; serial data can index better with 1 and a higher --spot-threshold) |
--max-spots <num> |
Maximum spot count (default: 250) |
--detect-ice-rings[=on|off] |
Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling/merging; overrides the dataset/master-file setting (default: use the dataset value) |
Azimuthal integration (the radial profile behind the per-image ice-ring score):
| Option | Description |
|---|---|
-q, --azim-q-spacing <num> |
Q bin spacing, 1/Å (default: 0.01; finer resolves the narrow ice rings) |
--azim-min-q <num> |
Minimum Q, 1/Å |
--azim-max-q <num> |
Maximum Q, 1/Å |
--azim-phi-bins <num> |
Number of azimuthal (phi) bins (default: 1) |
--polarization-correction <on|off> |
Enable/disable the azimuthal polarization correction |
--solid-angle-correction <on|off> |
Enable/disable the azimuthal solid-angle correction |
Indexing:
A dataset with a rotation goniometer axis is processed as rotation data (two-pass rotation
indexing) by default; a dataset without one is processed as independent stills. --force-still
overrides the former; the -R / --single-pass-rotation / --force-rotation-lattice flags request
rotation explicitly and pick the pass or lattice.
| Option | Description |
|---|---|
--force-still |
Treat a rotation (goniometer) dataset as independent stills instead of rotation |
-X, --indexing-algorithm <txt> |
FFBIDX | FFT | FFTW | Auto | None |
-C, --unit-cell <cell> |
Reference unit cell "a,b,c,alpha,beta,gamma" (required by ffbidx) |
-S, --space-group <num|symbol> |
Space group number (92) or Hermann-Mauguin symbol (P43212) — for indexing and scaling |
-r, --refine <txt> |
Geometry refinement: none | orientation | beam_and_lattice (default) | flex (try all three per image, keep whichever indexes the most spots; alias multi) |
-R, --two-pass-rotation[=num] |
Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100) |
--single-pass-rotation[=num] |
Online-like single-pass rotation indexing (optional min angular range, deg) |
--redo-rotation-spots |
Redo spot finding for the two-pass rotation first pass |
--force-rotation-lattice <vec> |
Force rotation lattice (9 floats, Å), skipping the first pass |
--rotation-no-postrefine |
Rotation: disable the default-on two-pass geometry post-refine (see the rotation section) |
--refine-geometry[=N|off] |
Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (-C / -z), =off disables |
Indexer choice in brief: ffbidx (GPU) refines toward a known cell and is best for sparse
serial stills; fft (GPU) / fftw (CPU) index de novo and suit strong rotation data. See the
CPU/GPU data-analysis reference for the algorithms.
Scaling and merging:
| Option | Description |
|---|---|
--no-merge |
Skip scaling and merging (on by default); write only the per-image _process.h5 |
-A, --anomalous |
Anomalous mode (keep Friedel pairs separate) |
-B, --refine-bfactor |
Refine a per-image B-factor (stills only) |
--scale-fulls / --no-scale-fulls |
rot3d: refit a per-frame scale on the combined fulls (XDS order, Unity model); on by default for rotation data, off for stills |
--smooth-g[=deg] |
rot3d: smooth the per-frame scale G over a degree range before the 3D combine (XDS DELPHI-like; default 5° for rotation, 0 = off) |
--no-scaling-corrections |
rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls (see below) |
--relative-b[=deg] |
rot3d: fit a per-batch relative-B beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise) |
--still-partiality |
Experimental (stills): weight reflections by a Gaussian excitation-error partiality instead of treating each as a full |
--partiality-uncertainty <num> |
Stills: extra merge sigma ~num·(1−partiality)·⟨I⟩ on partials (use with --still-partiality; default 0, ~2.5 recommended) |
--stills-modulation |
Experimental (stills): fit a detector-plane modulation (flat-field) surface, cross-validated (default off) |
--capture-uncertainty <num> |
rot3d: systematic sigma on under-captured fulls, ~num·(1−captured_fraction)·I (default: 1.0 for rotation, 0 otherwise) |
--min-captured-fraction <num> |
rot3d: drop a combined full whose rocking curve was captured below this fraction — edge-of-sweep truncated fulls (default: 0.7 for rotation, 0 otherwise; 0 = off) |
--scaling-high-resolution <num> |
High-resolution limit for scaling, Å — manual override (default: no limit; disables the automatic cutoff below) |
--resolution-cutoff <txt> |
Automatic high-resolution cutoff for the written reflections and reported shells: cc-logistic | off (default: cc-logistic; ignored when --scaling-high-resolution is set) |
--resolution-cc-target <num> |
CC1/2 target defining the cc-logistic fall-off (default: 0.30) |
--resolution-shells <num> |
Number of resolution shells in the reported statistics table (default: 10) |
--min-partiality <num> |
Minimum partiality to accept a reflection (default: 0.02) |
--reject-outliers <num> |
Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for rot3d, off otherwise) |
--reject-delta-cchalf <num> |
Drop images with ΔCC1/2 below mean − N·stddev (default: off) |
--min-image-cc <num> |
Per-image CC limit, percent (default: no limit) |
--mosaicity <num> |
Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed |
--scaling-iterations <num> |
Scaling iterations with no reference data (default: 3) |
-z, --reference-mtz <file> |
Reference MTZ (enables reference-driven scaling) |
--reference-column <label> |
Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
--write-process-h5 |
Also write the (large) _process.h5 when merging (default: only .mtz/.cif) |
Integration:
| Option | Description |
|---|---|
--integrator <txt> |
Spot integrator: gaussian (profile-fit, default) | empirical | boxsum (classical fallback) |
--integration-radius <r> |
Signal-box radius r1, or r1,r2,r3 (px). One value ⇒ r2=r1+2, r3=r1+4 |
--background-trim <f> |
Monochromatic (rotation + still): symmetric trimmed-mean fraction for the background ring, 0≤f<0.5 (default 0.10; 0 = plain mean) — removes the high-side bias that over-subtracts weak high-angle spots |
--bandwidth <num> |
Relative X-ray bandwidth FWHM (e.g. 0.01 for a 1% DMM); default from file or 0 (monochromatic) |
Geometry overrides (defaults are taken from the input file; override them to reprocess with a corrected geometry):
| Option | Description |
|---|---|
--beam-x <num> |
Beam centre X (pixel) |
--beam-y <num> |
Beam centre Y (pixel) |
--detector-distance <num> |
Detector distance (mm) |
--wavelength <num> |
Wavelength (Å) |
--rot1 <num> |
PONI detector rotation 1 (rad) |
--rot2 <num> |
PONI detector rotation 2 (rad) |
--polarization <num> |
Polarization factor |