mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 66 Wiki Activity
Files
007e65cdfe0432cf00e7f811bb002f04fa6d2846
Jungfraujoch/docs/RUGNUX.md
T
leonarski_fandClaude Opus 4.8 e7fbeb527f docs: update analysis references to match current code 
Reconcile docs/CPU_DATA_ANALYSIS.md and docs/RUGNUX.md with the current
image_analysis/ and rugnux CLI (verified section-by-section against the code):
unified profile-fit Bragg integration engine, multi-lattice indexing,
azimuthal phi binning, radial parallax/bandwidth profile with sub-pixel
centring, rot3d capture-fraction handling, automatic CC1/2 resolution cutoff,
and the new rugnux options; fix the section numbering and cross-references;
remove the never-implemented French-Wilson and still-partiality descriptions.

Delete the stale in-source design notes (ICE_RING_DETECTION,
BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) and fix the two code comments
that pointed at them; the BraggIntegrationEngine header no longer claims it is
'not yet wired'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 10:59:15 +02:00

221 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# rugnux
`rugnux` is the **offline** crystallographic data-analysis tool of Jungfraujoch — the
data-processing half of the system (see [Naming](NAMING.md) for where the name comes from).
It takes an existing HDF5 dataset, runs the full analysis pipeline — spot finding, indexing,
geometry refinement, Bragg integration and (optionally) scaling and merging — and writes the
results to a `_process.h5` file, plus reflection files (`.mtz`/`.cif`/`.hkl`) when merging is
requested.
It runs the *same* analysis code as the online and interactive tools, just driven from the
command line over a file rather than a live detector stream.
> **Note.** `rugnux` is under very active development. This page describes the tool and
> its options at a high level; the authoritative, always-current list of options is the program's
> own usage message — run `rugnux` with no arguments.
## Where it fits among the three analysis tools
| Tool | Mode | Driven by | Output |
| --- | --- | --- | --- |
| [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) |
| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | Displayed on screen (results not saved to disk) |
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** |
Use `rugnux` to re-analyse data after acquisition, to experiment with processing
parameters, or to produce merged intensities for downstream structure solution.
## Hardware
As with the rest of Jungfraujoch, **serious performance requires an NVIDIA GPU**. The CUDA build
provides the GPU fast-feedback indexer (`ffbidx`) and the GPU FFT indexer (`fft`); without CUDA
only the CPU `fftw` indexer is available. Spot finding, integration and scaling run on the CPU and
scale with the thread count (`-N`).
## Input and output
**Input** is a single Jungfraujoch HDF5 master file (NXmx-based). If the dataset already contains
stored spot lists, two-pass rotation indexing can reuse them instead of re-running spot finding on
the first pass.
**Output** (controlled by `-o, --output-prefix`, default `output`):
- `<prefix>_process.h5` — NXmx-compliant HDF5 with derived metadata (spots, indexing,
integration, azimuthal integration, per-image statistics). See
[HDF5 / NeXus data format](HDF5.md) for the layout. Written by default only when **not** merging
(i.e. under `--no-merge`); add `--write-process-h5` to also write it when merging.
- Merging is **on by default** (`--no-merge` disables it). The merged reflections are written as
`<prefix>.cif` (mmCIF — the default), or `<prefix>.mtz` / `<prefix>.hkl` depending on
`--scaling-output`. Both the mmCIF and the MTZ carry the **refined unit cell** (from rotation
indexing) and the **space group determined from systematic absences** (constrained to the indexed
lattice symmetry). No-reference scaling additionally emits per-iteration `<prefix>_iterN_scale.dat`.
Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
console. By default the written resolution is trimmed automatically where CC1/2 falls off
(`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the
limit by hand, or `--resolution-cutoff off` to keep the full range.
## Re-scaling and re-merging (`rugnux --scale`)
The `--scale` mode re-scales and merges the *already-integrated* reflections stored in a
`_process.h5` file, without re-running spot finding or integration. Use it to re-merge quickly with a
different space group, resolution limit, anomalous setting or reference MTZ. It reuses the same
`-o/-N/-s/-e/-S/-A/-B/-z/--scaling-*` options as the full run, and (unlike the full pipeline) does
not run a space-group search, so pass `-S` for the correct symmetry.
## Quick start
### Rotation data
Index, integrate, scale and merge a rotation sweep, fully de novo:
```
rugnux rotation_master.h5 \
-o lyso_rot -N 32 \
--scaling-high-resolution 1.4
```
Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by
default**: two-pass rotation indexing (index the sweep once, then process every frame against that
lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). Scaling
and merging run **by default** (for both rotation and stills; `--no-merge` turns them off); the unit
cell is taken from the rotation indexer and the space group is determined from systematic absences,
and both are written
into the merged `.cif`.
Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front
tends to *degrade* low-symmetry cases. `--scaling-high-resolution` (set it to your expected
resolution) sharpens both the space-group search and the error model. To tune the first pass use
`--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be
treated as independent stills use `--force-still`.
### Still / serial data
A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent
stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer,
then merge against a reference structure:
```
rugnux serial_master.h5 \
-o lyso_serial -N 32 \
-X ffbidx -C 79,79,38,90,90,90 -S 96 \
--spot-sigma 4 \
-z reference.mtz \
--scaling-high-resolution 1.8
```
`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. For
weak serial data, tightening spot finding with `--spot-sigma 4` typically raises the indexing rate
substantially. If a dataset *does* carry a goniometer axis but you want per-frame stills processing
anyway, add `--force-still`.
## Command-line options
General:
| Option | Description |
| --- | --- |
| `-o, --output-prefix <txt>` | Output file prefix (default: `output`) |
| `-N, --threads <num>` | Number of worker threads (default: all hardware threads) |
| `-s, --start-image <num>` | First image to process (default: 0) |
| `-e, --end-image <num>` | Last image to process (default: all) |
| `-t, --stride <num>` | Process every *n*-th image (default: 1) |
| `-v, --verbose` | Verbose output |
Modes (default: full analysis — spot finding, indexing, integration and merging):
| Option | Description |
| --- | --- |
| `--azint-only` | Only run azimuthal integration (no spot finding/indexing); writes `<prefix>_process.h5` |
| `--scale` | Only re-scale/merge the already-integrated reflections in the input `_process.h5` (no re-integration) |
Spot finding:
| Option | Description |
| --- | --- |
| `--spot-sigma <num>` | Noise sigma level for spot finding (default: 3.0) |
| `--spot-threshold <num>` | Photon-count threshold for spot finding (default: 10) |
| `--spot-high-resolution <num>` | High-resolution limit for spot finding, Å (default: 1.5) |
| `--max-spots <num>` | Maximum spot count (default: 250) |
| `--detect-ice-rings[=on\|off]` | Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling/merging; overrides the dataset/master-file setting (default: use the dataset value) |
Azimuthal integration (the radial profile behind the per-image ice-ring score):
| Option | Description |
| --- | --- |
| `-q, --azim-q-spacing <num>` | Q bin spacing, 1/Å (default: 0.01; finer resolves the narrow ice rings) |
| `--azim-min-q <num>` | Minimum Q, 1/Å |
| `--azim-max-q <num>` | Maximum Q, 1/Å |
| `--azim-phi-bins <num>` | Number of azimuthal (phi) bins (default: 1) |
| `--polarization-correction <on\|off>` | Enable/disable the azimuthal polarization correction |
| `--solid-angle-correction <on\|off>` | Enable/disable the azimuthal solid-angle correction |
Indexing:
A dataset with a **rotation goniometer axis** is processed as rotation data (two-pass rotation
indexing) by default; a dataset without one is processed as independent stills. `--force-still`
overrides the former; the `-R` / `--single-pass-rotation` / `--force-rotation-lattice` flags request
rotation explicitly and pick the pass or lattice.
| Option | Description |
| --- | --- |
| `--force-still` | Treat a rotation (goniometer) dataset as independent stills instead of rotation |
| `-X, --indexing-algorithm <txt>` | `FFBIDX` \| `FFT` \| `FFTW` \| `Auto` \| `None` |
| `-C, --unit-cell <cell>` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`) |
| `-S, --space-group <num>` | Space group number (used for indexing and scaling) |
| `-r, --refine <txt>` | Geometry refinement: `none` \| `orientation` \| `beam_and_lattice` (default) |
| `-R, --two-pass-rotation[=num]` | Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100) |
| `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) |
| `--redo-rotation-spots` | Redo spot finding for the two-pass rotation first pass |
| `--force-rotation-lattice <vec>` | Force rotation lattice (9 floats, Å), skipping the first pass |
Indexer choice in brief: `ffbidx` (GPU) refines toward a **known cell** and is best for sparse
serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rotation data. See the
[CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md) for the algorithms.
Scaling and merging:
| Option | Description |
| --- | --- |
| `--no-merge` | Skip scaling and merging (on by default); write only the per-image `_process.h5` |
| `-A, --anomalous` | Anomalous mode (keep Friedel pairs separate) |
| `-B, --refine-bfactor` | Refine a per-image B-factor |
| `--scale-fulls` / `--no-scale-fulls` | rot3d: refit a per-frame scale on the combined fulls (XDS order, Unity model); on by default for rotation data, off for stills |
| `--smooth-g[=deg]` | rot3d: smooth the per-frame scale *G* over a degree range before the 3D combine (XDS DELPHI-like; default 5° for rotation, 0 = off) |
| `--capture-uncertainty <num>` | rot3d: systematic sigma on under-captured fulls, ~num·(1−captured_fraction)·I (default: 1.0 for rotation, 0 otherwise) |
| `--min-captured-fraction <num>` | rot3d: drop a combined full whose rocking curve was captured below this fraction — edge-of-sweep truncated fulls (default: 0.7 for rotation, 0 otherwise; 0 = off) |
| `--scaling-high-resolution <num>` | High-resolution limit for scaling, Å — manual override (default: no limit; disables the automatic cutoff below) |
| `--resolution-cutoff <txt>` | Automatic high-resolution cutoff for the written reflections and reported shells: `cc-logistic` \| `off` (default: `cc-logistic`; ignored when `--scaling-high-resolution` is set) |
| `--resolution-cc-target <num>` | CC1/2 target defining the `cc-logistic` fall-off (default: 0.30) |
| `--resolution-shells <num>` | Number of resolution shells in the reported statistics table (default: 10) |
| `--min-partiality <num>` | Minimum partiality to accept a reflection (default: 0.02) |
| `--reject-outliers <num>` | Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for `rot3d`, off otherwise) |
| `--reject-delta-cchalf <num>` | Drop images with ΔCC1/2 below mean − N·stddev (default: off) |
| `--min-image-cc <num>` | Per-image CC limit, percent (default: no limit) |
| `--mosaicity <num>` | Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed |
| `--scaling-iterations <num>` | Scaling iterations with no reference data (default: 3) |
| `--scaling-output <txt>` | Reflection output format: `cif` (mmCIF, default) \| `mtz` \| `txt` |
| `-z, --reference-mtz <file>` | Reference MTZ (enables reference-driven scaling) |
| `--reference-column <label>` | Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
| `--write-process-h5` | Also write the (large) `_process.h5` when merging (default: only `.mtz`/`.cif`) |
Integration:
| Option | Description |
| --- | --- |
| `--integrator <txt>` | Spot integrator: `gaussian` (profile-fit, default) \| `empirical` \| `boxsum` (classical fallback) |
| `--integration-radius <r>` | Signal-box radius `r1`, or `r1,r2,r3` (px). One value ⇒ `r2=r1+2`, `r3=r1+4` |
| `--bandwidth <num>` | Relative X-ray bandwidth FWHM (e.g. `0.01` for a 1% DMM); default from file or 0 (monochromatic) |
Geometry overrides (defaults are taken from the input file; override them to reprocess with a corrected geometry):
| Option | Description |
| --- | --- |
| `--beam-x <num>` | Beam centre X (pixel) |
| `--beam-y <num>` | Beam centre Y (pixel) |
| `--detector-distance <num>` | Detector distance (mm) |
| `--wavelength <num>` | Wavelength (Å) |
| `--rot1 <num>` | PONI detector rotation 1 (rad) |
| `--rot2 <num>` | PONI detector rotation 2 (rad) |
| `--polarization <num>` | Polarization factor |
Reference in New Issue View Git Blame Copy Permalink