leonarski_fandClaude Opus 5
abfdb89b79
The centre in the file is often a placeholder, and nothing measures it until post-refinement has already indexed the sweep - by which time a wrong centre has chosen the lattice. Two exact facts about a rotation sweep give it from spot positions alone, with no cell, no orientation matrix and nothing indexed. Rotating 180 degrees about the spindle and taking -h negates a reflection's component along the spindle and leaves the rest, so with the spindle perpendicular to the beam the Laue condition is preserved and the spots recorded half a turn apart are mirror images along the spindle. Those are Friedel mates, not the same reflection. The same reflection appears twice for a different reason: it meets the Ewald sphere on two crossings, generally not half a turn apart, differing only in the sign of the component perpendicular to both the spindle and the beam. The first observable gives the coordinate along the spindle, the second the coordinate across it. Each candidate pairing votes and the true value accumulates while wrong pairings scatter. Both observables need guarding, because a vote is a comb and the tallest tooth is not always the right one. Along the spindle a false pairing cannot fake the equality of Friedel amplitudes. Across it, the two crossings of one reflection are separated by a sweep angle its own position fixes, which no accidental pair reproduces. The mirror is exact in the laboratory frame, so it is only as good as the rotation axis. Every file here states an ideal axis and none of them has one; a skew about the beam spreads the vote instead of shifting it, and past a milliradian it moves an otherwise correct answer by pixels while every internal statistic still looks healthy. It is therefore fitted, not assumed. A tilt of the axis towards the beam is measured and reported but not applied, being confounded with the detector rotation until that is fitted too. Nothing inside the fit can see a wrong tooth - when the vote flips, every frame pair flips with it - so the answer is checked from outside, by asking whether it depends on where the search began. That, and a floor on the angular span the pairs cover, are what refuse the cases this cannot measure: a sweep barely past half a turn is the dangerous one, not the short one, because at exactly half a turn there is nothing to fit and just past it there is almost nothing. Where the sweep is too short for any of this the radial background profile gives a coarser centre from a handful of images, and where neither can measure it the file's value is kept. The beam-stop projection now takes its own frames rather than sharing the sample, so turning this on cannot change the mask; and both samples keep away from the ends of the sweep, where shutter synchronisation spoils an image. Reading twice as many frames as before costs a few seconds once, and is what makes the answer independent of which frames were drawn. Off by default. Over the 38-crystal rotation battery it serves every dataset, agrees with XDS's refined direct beam to 0.116 px in the median against 0.135 for the value in the file, and changes no space group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
511 lines
39 KiB
Markdown
511 lines
39 KiB
Markdown
# rugnux
|
||
|
||
`rugnux` is the **offline** crystallographic data-analysis tool of Jungfraujoch — the
|
||
data-processing half of the system (see [Naming](NAMING.md) for where the name comes from).
|
||
It takes an existing HDF5 dataset, runs the full analysis pipeline — spot finding, indexing,
|
||
geometry refinement, Bragg integration and (optionally) scaling and merging — and writes the
|
||
results to a `_process.h5` file, plus reflection files (`.mtz`/`.cif`/`.hkl`) when merging is
|
||
requested.
|
||
|
||
It runs the *same* analysis code as the online and interactive tools, just driven from the
|
||
command line over a file rather than a live detector stream.
|
||
|
||
> **Note.** `rugnux` is under very active development. This page describes the tool and
|
||
> its options at a high level; the authoritative, always-current list of options is the program's
|
||
> own usage message — run `rugnux` with no arguments.
|
||
|
||
## Where it fits among the three analysis tools
|
||
|
||
| Tool | Mode | Driven by | Output |
|
||
| --- | --- | --- | --- |
|
||
| [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) |
|
||
| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | On screen; a processing job can write the same files as `rugnux` |
|
||
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** |
|
||
|
||
Use `rugnux` to re-analyse data after acquisition, to experiment with processing
|
||
parameters, or to produce merged intensities for downstream structure solution.
|
||
|
||
## Hardware
|
||
|
||
As with the rest of Jungfraujoch, **serious performance requires an NVIDIA GPU**. The CUDA build
|
||
provides the GPU fast-feedback indexer (`ffbidx`) and the GPU FFT indexer (`fft`); without CUDA
|
||
only the CPU `fftw` indexer is available. Spot finding, integration and scaling run on the CPU and
|
||
scale with the thread count (`-N`).
|
||
|
||
## Input and output
|
||
|
||
**Input** is a single Jungfraujoch HDF5 master file (NXmx-based). Spots are always found by `rugnux`
|
||
itself, including for the two-pass rotation first pass — the spot lists a dataset may already carry
|
||
were found online, at the acquisition's threshold and with its ice-band spots already discarded, so
|
||
reusing them would hide the spot-finding settings from the lattice search.
|
||
|
||
**Output** (controlled by `-o, --output-prefix`, default `output`):
|
||
|
||
- `<prefix>_process.h5` — NXmx-compliant HDF5 with derived metadata (spots, indexing,
|
||
integration, azimuthal integration, per-image statistics). See
|
||
[HDF5 / NeXus data format](HDF5.md) for the layout. Written by default only when **not** merging
|
||
(i.e. under `--no-merge`); add `--write-process-h5` to also write it when merging.
|
||
- Merging is **on by default** (`--no-merge` disables it). The merged reflections are written in
|
||
**three** formats — each has its uses downstream:
|
||
- `<prefix>.mtz` — CCP4 MTZ (`IMEAN`/`I(+)`/`I(-)`, French–Wilson `F`, `FreeR_flag`) for the CCP4 /
|
||
phenix reflection tools.
|
||
- `<prefix>.cif` — mmCIF, for deposition and as the self-describing native format (also carries the
|
||
merging statistics, ISa, twinning and radiation-damage indicators).
|
||
- `<prefix>.hkl` — SHELX **HKLF 4** text (`h k l I σ(I)`, fixed `3I4,2F8.2`), the direct input for
|
||
**SHELXC / ANODE / SHELXD**. Bijvoet mates are written separately (`I(+)` at `+hkl`, `I(-)` at
|
||
`-hkl`) so the anomalous signal is preserved; intensities are put on a common scale so the largest
|
||
value fits the fixed-width field (the absolute scale is irrelevant to SHELXC/ANODE), and the file
|
||
ends with the `0 0 0` terminator record.
|
||
|
||
All three carry the **refined unit cell** (from rotation indexing) and the **space group determined
|
||
from systematic absences** (constrained to the indexed lattice symmetry). No-reference scaling
|
||
additionally emits per-iteration `<prefix>_iterN_scale.dat`.
|
||
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
|
||
a beamline script can read. Always written, next to the files above. See
|
||
[The results report](#the-results-report) below.
|
||
|
||
Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
|
||
console. By default the written resolution is trimmed automatically where CC1/2 falls off
|
||
(`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the
|
||
limit by hand, or `--resolution-cutoff off` to keep the full range.
|
||
|
||
### Reflection-file conventions
|
||
|
||
**mmCIF.** Standard items carry their standard meanings — `_refln.intensity_meas` / `_intensity_sigma`,
|
||
the `pdbx_I_plus`/`pdbx_I_minus` and `pdbx_F_plus`/`pdbx_F_minus` anomalous pairs, `_reflns.*` and
|
||
`_reflns_shell.*` for the merging statistics, `_reflns.B_iso_Wilson_estimate` for the Wilson B, and
|
||
`_cell.*` / `_diffrn_radiation_wavelength.wavelength` for the geometry.
|
||
|
||
Anything rugnux reports that has no standard item is written under a **`jfjoch_`** prefix, inside the
|
||
standard category it belongs to. That is a deliberate choice: a reader that does not know these items
|
||
ignores them, and one that does can find them without guessing.
|
||
|
||
| item | meaning |
|
||
|---|---|
|
||
| `_reflns.jfjoch_diffrn_ISa` | Asymptotic I/σ in **XDS's sense**: the whole-range `1/√(a·b)` of the error model, so it can be read directly against a `CORRECT.LP` |
|
||
| `_reflns.jfjoch_diffrn_ISa_asymptotic` | The **strong-reflection** tier — the counting-subtracted scatter of well-measured groups. XDS has no equivalent, and it can only ever be the more optimistic of the two. Rotation path only |
|
||
| `_reflns.jfjoch_error_model_a`, `_b` | The error model in XDS's convention, `σ² = a(σ₀² + b·I²)`, so the ISa above is re-derivable from the file rather than taken on trust |
|
||
| `_reflns.jfjoch_second_moment_I` | Twinning second moment ⟨I²⟩/⟨I⟩² — 2.00 untwinned, 1.50 for a perfect twin |
|
||
| `_reflns.jfjoch_L_test_mean_abs_L`, `_L_test_mean_L_squared` | Padilla–Yeates L-test. ⟨\|L\|⟩ is 0.500 untwinned / 0.375 for a perfect twin; ⟨L²⟩ is 0.333 / 0.200. Written only when the test found pairs |
|
||
| `_reflns.jfjoch_radiation_damage_relative_B` | Relative *B* from the first to the last rotation batch (Ų); positive is the usual direction, high-resolution intensity fading with dose |
|
||
| `_jfjoch_radiation_damage_batch.*` | Per-batch loop: `id`, `rotation_start_deg`, `relative_B` |
|
||
| `_diffrn_detector.jfjoch_distance_mm`, `_jfjoch_beam_center_x_pxl`, `_jfjoch_beam_center_y_pxl` | The refined detector geometry actually used, which is not otherwise recoverable from the reflection file |
|
||
|
||
> **Compatibility note.** Before rc.161, `_reflns.jfjoch_diffrn_ISa` carried the *asymptote*, not the
|
||
> whole-range value. There is no version marker inside the file, so a number taken from an older
|
||
> `.cif` is not comparable with one taken from a newer one.
|
||
|
||
**SHELX HKLF 4** (`<prefix>.hkl`). Fixed-format `3I4,2F8.2` — `h k l I σ(I)`, one record per
|
||
reflection, terminated by a `0 0 0` record — which is what **SHELXC**, **SHELXD** and **ANODE**
|
||
expect. Two properties worth knowing before using it:
|
||
|
||
- **Bijvoet mates are written separately**, `I(+)` at `+hkl` and `I(-)` at `-hkl`, so the anomalous
|
||
differences survive into SHELXC; a reflection with no anomalous split is written once, as its mean.
|
||
- **Intensities are rescaled** by a single global factor so the largest value fits the `F8.2` field.
|
||
`I` and `σ(I)` share that factor, so every ratio — and therefore the anomalous signal — is
|
||
untouched, but the absolute scale is not meaningful. This matters only if you intend to compare
|
||
magnitudes with another file; SHELXC and ANODE use ratios alone.
|
||
|
||
## The results report
|
||
|
||
`<prefix>_report.txt` records **what the run determined**, next to the reflection files. It is
|
||
written on every `--mode mx` and `--mode scale` run that has an output prefix — there is no option
|
||
to enable or disable it. Two cases follow from that:
|
||
|
||
- An **empty output prefix** (`-o ""`, the "compute the statistics, persist nothing" mode) writes
|
||
nothing, the report included.
|
||
- **`--no-merge`** still writes a report. It determined an indexing and a geometry result, and those
|
||
are recorded; the merging section then says `MERGE= NOT_PERFORMED` rather than being omitted, so
|
||
the absence is a statement and not something a reader has to infer.
|
||
|
||
The report is never allowed to fail a run: if it cannot be written (unwritable path, full disk) the
|
||
failure is logged as a warning and the run finishes normally.
|
||
|
||
### Format
|
||
|
||
The model is XDS's `CORRECT.LP`: prose and tables a crystallographer reads top to bottom, with a
|
||
structure a script can consume without parsing prose.
|
||
|
||
- **`KEY= value` assignment lines.** Every number worth extracting is one, so a consumer gets it with
|
||
a single `grep '^ISA= '` and never has to read a sentence. Key names are stable.
|
||
- **Fixed-width tables** with a stable header row for anything that is genuinely tabular — the
|
||
resolution shells, the space-group candidates, the sweep-quality ranges.
|
||
- **`WARNING:` lines**, one per finding, in plain English: `WARNING: Frames 500-600 out of beam
|
||
(10.1 deg, scale 0.12 and CC 0.30 of the run, 2% scaled)`. `grep '^WARNING:'` finds every one.
|
||
- **Section banners** (`***…***` around a numbered title) delimiting the blocks.
|
||
|
||
`REPORT_VERSION=` is the format's own version. Key names, table columns and the reason vocabulary
|
||
below are an interface other software may depend on: they do not change without that number moving.
|
||
|
||
Sections, in order: `1. DATA SET`, `2. INDEXING`, `3. GEOMETRY POST-REFINEMENT` (rotation only),
|
||
`4. SPACE GROUP DETERMINATION`, `5. SCALING AND MERGING`, `6. TWINNING`, `7. RADIATION DAMAGE`,
|
||
`8. SWEEP QUALITY`, `9. WARNINGS`.
|
||
|
||
**Which pass.** A rotation run integrates twice — once at the geometry in the input file
|
||
(`<prefix>_01.*`), then again at the post-refined geometry (`<prefix>.*`) — and can integrate a
|
||
third time if a guard rejects the second pass. There is **one** report, for the pass that became the
|
||
canonical output, and `PASS=` / `PASS_DECISION=` in section 1 say which pass that is and on what
|
||
evidence, so no number in the file is ambiguous about which geometry produced it.
|
||
|
||
**Not in the report:** timing, frame rates, thread counts, per-image progress and library banners.
|
||
Those are process, not result, and stay on stdout.
|
||
|
||
### Sweep quality and the reason vocabulary
|
||
|
||
Section 8 lists the stretches of the sweep over which the crystal delivered much less than the rest
|
||
of the run — the feedback a beamline control system needs to tell an operator that a crystal should
|
||
be recentred or recollected. Nothing is excluded on the strength of it; the frames still carry
|
||
signal, and this is a message for the beamline, not a filter.
|
||
|
||
```
|
||
SWEEP_QUALITY_STATUS= COMPUTED
|
||
SWEEP_QUALITY_COUNT= 1
|
||
SWEEP_QUALITY_REASONS= no_diffraction crystal_out_of_beam weak_diffraction loss_of_centring radiation_damage
|
||
SWEEP_ROTATION= 360.0
|
||
FLUX_PEAK_TO_TROUGH= 1.03
|
||
SCALE_MODULATION_PEAK_TO_TROUGH= 1.00
|
||
|
||
FIRST_IMAGE LAST_IMAGE N_IMAGES ROTATION REASON SEVERITY SCALE CC INDEXED
|
||
----------- ----------- --------- -------- -------------------- -------- ------ ------ --------
|
||
500 600 101 10.1 crystal_out_of_beam 0.83 0.12 0.30 0.02
|
||
----------- ----------- --------- -------- -------------------- -------- ------ ------ --------
|
||
```
|
||
|
||
`SWEEP_QUALITY_STATUS` distinguishes **`COMPUTED`** (the diagnostic ran; a count of 0 means the sweep
|
||
was clean throughout) from **`NOT_COMPUTED`** (it did not run — no scaling and merging, or stills
|
||
data). A consumer must not read a missing table or a zero count as "clean" without checking it.
|
||
`SWEEP_QUALITY_REASONS` lists the whole vocabulary this version can emit, so an unknown code is
|
||
distinguishable from a missing one.
|
||
|
||
| Reason code | Meaning |
|
||
|-------------|---------|
|
||
| `no_diffraction` | The range recorded essentially no diffraction from the indexed lattice. |
|
||
| `crystal_out_of_beam` | Frames were lost: over the range a per-image scale could be fitted far less often than over the run. |
|
||
| `weak_diffraction` | The frames all still index, but with much less intensity — the cause was not determined. |
|
||
| `loss_of_centring` | One cycle of modulation per revolution: the crystal is off the rotation axis. |
|
||
| `radiation_damage` | The range runs to the end of a sweep whose quality was already decaying. |
|
||
|
||
The vocabulary is **closed and stable**: a code is never renamed, and never reused for a different
|
||
meaning. New codes are only ever added, and adding one moves `REPORT_VERSION`.
|
||
|
||
The columns are: `FIRST_IMAGE`/`LAST_IMAGE` — inclusive, in processed-image ordinals (the numbering
|
||
of `<prefix>_image.dat` and of every other per-image array `rugnux` writes; with `-s`/`--stride` the
|
||
source image is `start + ordinal * stride`); `ROTATION` — the width of the range in degrees;
|
||
`SEVERITY` — the fraction of the run's typical diffracting power missing over the range, 0 (as good
|
||
as the run) to 1 (nothing at all); `SCALE` and `CC` — the range's mean per-image scale and
|
||
CC-to-merge relative to the run median; `INDEXED` — the fraction of the range's frames that were
|
||
scaled at all. Every range also appears as a `WARNING:` sentence in section 9.
|
||
|
||
The same finding is written **per image** into the `_process.h5` as `/entry/MX/sweepQuality`, when
|
||
one is written — see [HDF5](HDF5.md#41-entrymx--spot-finding-and-indexing-cxi-style).
|
||
|
||
## Validating against a model (`rugnux --model`)
|
||
|
||
Given a PDB atomic model of the same structure, `--model model.pdb` scales the model structure
|
||
factors to the merged amplitudes — fitting a flat bulk-solvent contribution and an overall
|
||
anisotropic *B* — and reports **R-work / R-free** and the mean 2Fo-Fc density at the atom centres.
|
||
It also writes `<prefix>_2fofc.ccp4`, `<prefix>_fofc.ccp4` and `<prefix>_maps.mtz` next to the
|
||
merged reflections. Nothing about the model is refined; it is only re-fractionalized into the data
|
||
cell, so a deposited model with a slightly different cell still lines up.
|
||
|
||
It is a *data-quality lens*, independent of the internal statistics: R-free measures the merged
|
||
intensities against external truth, where CC1/2 and R<sub>meas</sub> only measure them against
|
||
themselves. It also settles the two things merged intensities alone cannot: the enantiomorph (data
|
||
merged in P4<sub>1</sub>2<sub>1</sub>2 against a P4<sub>3</sub>2<sub>1</sub>2 model are reindexed
|
||
into the model's hand), and — when no reference MTZ has already fixed it — a merohedral indexing
|
||
ambiguity, by keeping the candidate reindexing with the lowest R-free.
|
||
|
||
## Re-scaling and re-merging (`rugnux --mode scale`)
|
||
|
||
The `scale` mode re-scales and merges the *already-integrated* reflections stored in a
|
||
`_process.h5` file, without re-running spot finding or integration. Use it to re-merge quickly with a
|
||
different space group, resolution limit, anomalous setting or reference MTZ. It reuses the same
|
||
`-o/-N/-s/-e/-S/-A/-B/-z/--scaling-*` options as the full run, and (unlike the full pipeline) does
|
||
not run a space-group search, so pass `-S` for the correct symmetry.
|
||
|
||
## Detector calibration from powder rings (`rugnux --mode calibration`)
|
||
|
||
The `calibration` mode determines the detector geometry — PONI *x*/*y*, the two tilts
|
||
rot1/rot2 and the distance — from the powder rings of a calibrant, and writes it as a
|
||
pyFAI **`<prefix>.poni`** file alongside a printed report of how far each parameter moved from
|
||
the header. Bragg data pin the beam centre worst (it is gauge-coupled to the crystal orientation);
|
||
a powder ring has no orientation to be coupled to, so this is the measurement that fixes it.
|
||
|
||
```
|
||
rugnux --mode calibration --calibrant lab6 -N 8 -o det LaB6_master.h5
|
||
```
|
||
|
||
`--calibrant` takes `lab6`, `agbh` (silver behenate), `ceo2`, `si` or `ice`, case-insensitively.
|
||
**`ice`** calibrates a real experiment against its own ice rings — no calibrant exposure needed —
|
||
and is the reason a calibrant is a list of ring positions rather than a unit cell: hexagonal ice
|
||
is P6<sub>3</sub>/mmc, so rings enumerated from its cell would include systematically absent ones.
|
||
|
||
`--calibration` picks how the rings are measured, and **both use the whole dataset** — `-s`/`-e`/`-t`
|
||
select which images:
|
||
|
||
- **`rings`** (default) sums the (*q* × azimuth) azimuthal profile over every processed image into
|
||
one map and fits the ring arcs in it. A powder ring is an arc, not a set of spots, and the summed
|
||
profile measures it at every azimuth with all the run's counts behind it. It needs the profile to
|
||
be binned in azimuth, so this mode defaults `--azim-phi-bins` to 32.
|
||
- **`spots`** pools the found spots of every processed image and fits those. It determines the
|
||
centre from scratch (a Hough circle vote, which quantises it to a whole pixel) and then refines.
|
||
|
||
Both routes read the ring position out of a binned profile or a spot centroid, so the radial
|
||
sampling matters: at a long detector distance the default 0.01 Å⁻¹ *q* bin is several pixels wide
|
||
and quantises the `rings` route accordingly — pass a finer `--azim-q-spacing` there (the total
|
||
*q* × azimuth bin count must stay under 65534).
|
||
|
||
The report prints the fitted geometry, the scatter of the ring points about the fitted rings and the
|
||
standard error that implies on the centre. That error is *formal*: it measures the scatter of the
|
||
points, not whether the rings themselves are trustworthy, so it stays small when a fit goes wrong
|
||
for a structural reason — one visible ring, or ice that is textured rather than smooth.
|
||
|
||
Both the PONI (the point of normal incidence, which is what a `.poni` file stores) and the direct
|
||
beam (where the beam lands, which is what most other programs call the beam centre) are printed.
|
||
They differ by *distance* × tan(rot) once the detector is tilted, which on a 0.3° tilt at 300 mm is
|
||
several pixels — enough to look like a disagreement with another program when there is none.
|
||
|
||
## Quick start
|
||
|
||
### Rotation data
|
||
|
||
Index, integrate, scale and merge a rotation sweep, fully de novo:
|
||
|
||
```
|
||
rugnux rotation_master.h5 \
|
||
-o rotation_run -N 32 \
|
||
--scaling-high-resolution 1.4
|
||
```
|
||
|
||
Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by
|
||
default**: two-pass rotation indexing (index the sweep once, then process every frame against that
|
||
lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). Scaling
|
||
and merging run **by default** (for both rotation and stills; `--no-merge` turns them off); the unit
|
||
cell is taken from the rotation indexer and the space group is determined from systematic absences,
|
||
and both are written
|
||
into the merged `.cif`.
|
||
|
||
Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front
|
||
tends to *degrade* low-symmetry cases. `--scaling-high-resolution` (set it to your expected
|
||
resolution) sharpens both the space-group search and the error model. To tune the first pass use
|
||
`--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be
|
||
treated as independent stills use `--force-still`.
|
||
|
||
By default a rotation run also **post-refines the geometry** in a second pass: the first pass
|
||
integrates and merges at the header geometry, then the detector distance + beam centre and the crystal
|
||
cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a
|
||
small < 1 % move, with the gauge-weak beam centre restrained toward the header), and the second pass
|
||
re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical
|
||
`<prefix>_*` output; the header-geometry pass is kept alongside as `<prefix>_01_*` for comparison.
|
||
Disable it with `--rotation-no-postrefine`.
|
||
|
||
After the per-frame scale-fulls step, rotation scaling applies three **correction surfaces**, **on by
|
||
default** (`--no-scaling-corrections` disables all):
|
||
|
||
- **Decay** — a global Debye–Waller relative-*B* over the run, for the radiation damage that weakens
|
||
later frames more at high resolution (a resolution×time systematic the resolution-flat per-frame
|
||
scale cannot remove). It only engages when the total relative-*B* exceeds a physical floor (2 Ų). An
|
||
optional `--relative-b[=deg]` extends this single global rate to a smooth per-batch relative-*B* curve
|
||
(default 10°-of-rotation batches when bare, off otherwise), cross-validated like the surfaces here, for
|
||
crystals whose decay is non-linear in dose.
|
||
- **Absorption** — a smooth multiplicative factor over the diffracted-beam direction in the goniometer
|
||
frame (path length through the crystal). Negligible at hard X-rays / thin crystals; it matters at
|
||
low photon energy. Its benefit shows up most on model-based metrics: a smooth absorption error
|
||
largely cancels among symmetry mates (little effect on the error model / ISa) but still biases the
|
||
intensities, so it measurably lowers *R*<sub>free</sub>.
|
||
- **Modulation** — a smooth multiplicative factor over the position where a reflection lands on the
|
||
detector (a flat-field: detector-response and geometric systematics that vary across the detector
|
||
plane). Symmetry-equivalents of one reflection land at different detector positions as the crystal
|
||
rotates, which over-determines the surface. Because it lives in the detector frame (not the
|
||
rotation) the same correction concept applies to stills. This is the largest of the three on
|
||
JUNGFRAU data — it lowers *R*<sub>meas</sub> by several to tens of percent on datasets that carry a
|
||
detector systematic, while holding or improving CC<sub>1/2</sub> and the anomalous signal.
|
||
|
||
All three are **cross-validated** — fitted on even-numbered frames and kept only if they improve the
|
||
held-out odd-frame symmetry-equivalent agreement by a clear margin (and vice versa). The agreement is
|
||
scored as a σ-independent, *R*<sub>meas</sub>-like fractional deviation, so a surface can never pass
|
||
cross-validation by merely reshaping the sigmas; where the systematic is absent the surface is a no-op
|
||
rather than a source of added noise, which is why they are safe to leave on.
|
||
|
||
Independently of any correction, a rotation run prints a **radiation-damage report** — the per-image
|
||
scale correlation-to-merge and mosaicity versus dose, and the relative *B*-factor change over the run
|
||
(first→last) together with a per-batch relative-*B* curve, also written to the merged mmCIF. It is a
|
||
data-quality-vs-dose diagnostic and never alters the merged intensities. A batch whose data cannot
|
||
support a measurement prints `-` instead of a value, and the first→last number is printed only where a
|
||
straight line describes the curve — damage is progressive, so a curve that dips and recovers is a
|
||
disturbance of the sweep, not dose, and the report says so and points at the sweep-quality section
|
||
(`RADIATION_DAMAGE_RELATIVE_B= NOT_A_TREND`).
|
||
|
||
### Still / serial data
|
||
|
||
A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent
|
||
stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer,
|
||
then merge against a reference structure:
|
||
|
||
```
|
||
rugnux serial_master.h5 \
|
||
-o serial_run -N 32 \
|
||
-X ffbidx -C 79,79,38,90,90,90 -S 96 \
|
||
-z reference.mtz \
|
||
--scaling-high-resolution 1.8
|
||
```
|
||
|
||
`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. The
|
||
self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for
|
||
serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this
|
||
combination raises the indexing rate and typically extends resolution over a fixed threshold and
|
||
fixed min-pix, at equal or better CC½. (You can still pin a fixed threshold with `--spot-sigma` /
|
||
`--spot-threshold` and a fixed min-pix with `--min-pix-per-spot`.) If a dataset *does* carry a
|
||
goniometer axis but you want per-frame stills processing anyway, add `--force-still`.
|
||
|
||
## Command-line options
|
||
|
||
General:
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `-o, --output-prefix <txt>` | Output file prefix (default: `output`) |
|
||
| `-N, --threads <num>` | Number of worker threads (default: all hardware threads) |
|
||
| `-s, --start-image <num>` | First image to process (default: 0) |
|
||
| `-e, --end-image <num>` | Last image to process (default: all) |
|
||
| `-t, --stride <num>` | Process every *n*-th image (default: 1) |
|
||
| `-v, --verbose` | Verbose output |
|
||
|
||
Mode — `--mode <name>` (default `mx`):
|
||
|
||
| Value | Description |
|
||
| --- | --- |
|
||
| `mx` | Full analysis — spot finding, indexing, integration and merging |
|
||
| `azint` | Only azimuthal integration (no spot finding/indexing); writes `<prefix>_process.h5` |
|
||
| `scale` | Only re-scale/merge the already-integrated reflections in the input `_process.h5` (no re-integration) |
|
||
| `calibration` | Determine the detector geometry from powder rings; writes `<prefix>.poni` |
|
||
|
||
Calibration (`--mode calibration`):
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--calibrant <name>` | Powder standard: `lab6` \| `agbh` \| `ceo2` \| `si` \| `ice` (default `lab6`, case-insensitive) |
|
||
| `--calibration <txt>` | How the rings are measured: `rings` \| `spots` (default `rings`; see above). `rings` defaults `--azim-phi-bins` to 32 |
|
||
|
||
Detector mask:
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--detect-beam-stop[=N\|off]` | Find the beam stop and its holder in a projection of N images and add them to the pixel mask as bit 9, so nothing shadowed by them is integrated. **On by default** (60 images); `=off` disables. Reflections behind the stop are attenuated but not flagged, so they integrate low with a plausible sigma and no existing rejection catches them |
|
||
|
||
Geometry:
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--estimate-beam-center` | Measure the direct beam before indexing, from the symmetry of the spots where the sweep reaches at least half a turn and from the radial background profile where it does not; the value in the file is kept where neither can measure it. Off by default |
|
||
| `--no-fit-spindle` | With the above, keep the rotation axis given in the file instead of fitting its skew about the beam |
|
||
|
||
Spot finding:
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--spot-sigma <num>` | Noise sigma level for spot finding (default: 4.0) |
|
||
| `--spot-threshold <num>` | Photon-count threshold for spot finding (default: 10) |
|
||
| `--adaptive-spots` | Self-calibrating detection (**default**, stills and rotation alike): the strong-pixel threshold comes from each image's own per-resolution-ring noise instead of the fixed `--spot-threshold`, so one setting adapts across datasets (no per-dataset `--spot-threshold`/`--spot-sigma` tuning) |
|
||
| `--no-adaptive-spots` | Turn adaptive detection off and use the fixed `--spot-threshold` / `--spot-sigma` finder |
|
||
| `--spot-false-pixels <num>` | Adaptive-detection operating point: expected noise pixels tolerated per frame (default: 100; implies `--adaptive-spots`) |
|
||
| `--spot-high-resolution <num>` | High-resolution limit for spot finding, Å. Omitted (or 0): no resolution clipping — spot finding extends as far as the detector reaches, for rotation data as well as stills |
|
||
| `--spot-low-resolution <num>` | Low-resolution limit for spot finding, Å (default: 50; lower it, e.g. 24, to exclude the direct-beam halo on weak serial data; 0 removes the limit) |
|
||
| `--min-pix-per-spot <num>` | Minimum connected strong pixels per spot. **If omitted, min-pix is chosen per image** (stills indexing): the frame is indexed at min-pix 3/2/1 and the one maximising indexed-spot count × indexed fraction is kept. Give an explicit value to force a fixed min-pix instead. |
|
||
| `--max-spots <num>` | Maximum spots kept per image (the strongest ones) and handed to indexing (default: 1000) |
|
||
| `--detect-ice-rings[=on\|off]` | Flag ice-ring spots (de-prioritised in indexing) and exclude ice-ring reflections from scaling. Default: the master file's `detect_ice_rings`, or — where the file carries no such key — **on for rotation and off for stills** |
|
||
|
||
Azimuthal integration (the radial profile behind the per-image ice-ring score):
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `-q, --azim-q-spacing <num>` | Q bin spacing, 1/Å (default: 0.01; finer resolves the narrow ice rings) |
|
||
| `--azim-min-q <num>` | Minimum Q, 1/Å |
|
||
| `--azim-max-q <num>` | Maximum Q, 1/Å. Omitted: integration extends to the highest Q the detector reaches. The adaptive spot finder shares these Q bins, so this also sets how far self-calibrating detection can see |
|
||
| `--azim-phi-bins <num>` | Number of azimuthal (phi) bins (default: 1) |
|
||
| `--polarization-correction <on\|off>` | Enable/disable the azimuthal polarization correction |
|
||
| `--solid-angle-correction <on\|off>` | Enable/disable the azimuthal solid-angle correction |
|
||
|
||
Indexing:
|
||
|
||
A dataset with a **rotation goniometer axis** is processed as rotation data (two-pass rotation
|
||
indexing) by default; a dataset without one is processed as independent stills. `--force-still`
|
||
overrides the former; the `-R` / `--single-pass-rotation` / `--force-rotation-lattice` flags request
|
||
rotation explicitly and pick the pass or lattice.
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--force-still` | Treat a rotation (goniometer) dataset as independent stills instead of rotation |
|
||
| `-X, --indexing-algorithm <txt>` | `FFBIDX` \| `FFT` \| `FFTW` \| `Auto` \| `None` |
|
||
| `-C, --unit-cell <cell>` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`) |
|
||
| `-S, --space-group <num\|symbol>` | Space group number (`92`) or Hermann-Mauguin symbol (`P43212`) — for indexing and scaling |
|
||
| `-r, --refine <txt>` | Geometry refinement: `none` \| `orientation` \| `beam_and_lattice` (default) \| `flex` (try all three per image, keep whichever indexes the most spots; alias `multi`) |
|
||
| `-R, --two-pass-rotation[=num]` | Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100) |
|
||
| `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) |
|
||
| `--force-rotation-lattice <vec>` | Force rotation lattice (9 floats, Å), skipping the first pass |
|
||
| `--rotation-no-postrefine` | Rotation: disable the default-on two-pass geometry post-refine (see the rotation section) |
|
||
| `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills with a reference cell (`-C` / `-z`), `=off` disables |
|
||
| `--index-ice-rings[=on\|off]` | Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: **off**; no effect without `--detect-ice-rings`, which does the flagging) |
|
||
|
||
Indexer choice in brief: `ffbidx` (GPU) refines toward a **known cell** and is best for sparse
|
||
serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rotation data. See the
|
||
[CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md) for the algorithms.
|
||
|
||
Scaling and merging:
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--no-merge` | Skip scaling and merging (on by default); write only the per-image `_process.h5` |
|
||
| `-A, --anomalous` | Anomalous mode (keep Friedel pairs separate) |
|
||
| `--scale-fulls` / `--no-scale-fulls` | rot3d: refit a per-frame scale on the combined fulls (XDS order, Unity model); on by default for rotation data, off for stills |
|
||
| `--smooth-g[=deg]` | rot3d: smooth the per-frame scale *G* over a degree range before the 3D combine (XDS DELPHI-like; default 5° for rotation, 0 = off) |
|
||
| `--no-scaling-corrections` | rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls (see below) |
|
||
| `--relative-b[=deg]` | rot3d: fit a per-batch relative-*B* beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise) |
|
||
| `--simple-stills` | Stills: treat every reflection as a full (*p* = 1, single-pass scale/merge) — disables the default-on physical partiality post-refinement |
|
||
| `--no-expected-variance-merge` | Stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting |
|
||
| `--capture-uncertainty <num>` | rot3d: systematic sigma on under-captured fulls, ~num·(1−captured_fraction)·I (default: 1.0 for rotation, 0 otherwise) |
|
||
| `--min-captured-fraction <num>` | rot3d: drop a combined full whose rocking curve was captured below this fraction — edge-of-sweep truncated fulls (default: 0.7 for rotation, 0 otherwise; 0 = off) |
|
||
| `--scaling-high-resolution <num>` | High-resolution limit for scaling, Å — manual override (default: no limit; disables the automatic cutoff below) |
|
||
| `--scaling-low-resolution <num>` | Low-resolution limit for scaling and merging, Å (default: 50, the value XDS configurations use; 0 removes the limit). Reflections coarser than this sit behind or beside the beam stop and are measured on a background it has eaten into |
|
||
| `--resolution-cutoff <txt>` | Automatic high-resolution cutoff for the written reflections and reported shells: `cc-logistic` \| `off` (default: `cc-logistic`; ignored when `--scaling-high-resolution` is set) |
|
||
| `--resolution-cc-target <num>` | CC1/2 target defining the `cc-logistic` fall-off (default: 0.30) |
|
||
| `--resolution-shells <num>` | Number of resolution shells in the reported statistics table (default: 10) |
|
||
| `--min-partiality <num>` | Minimum partiality to accept a reflection (default: 0.02) |
|
||
| `--ice-min-score <num>` | Ice-presence gate: the measured per-run ice score (1 = no ice) a dataset must reach before **any** ice handling is applied — the flagging and the exclusion from scaling (default: 1.5; 0 = no gate). The eleven fixed hexagonal bands cover 16–26 % of the unique reflections whether or not the crystal has ice, so handling ice on a clean crystal only costs completeness |
|
||
| `--ice-min-spot-ratio <num>` | The second ice-presence channel: found **spots** on the hexagonal rings over the same q width of ice-free flanks beside them (1 = spots spread evenly). Ice in large crystallites diffracts as discrete spots and leaves the radial profile flat, so `--ice-min-score` alone is blind to it (default: 2.0; 0 disables this channel) |
|
||
| `--reject-outliers <num>` | Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for `rot3d`, off otherwise) |
|
||
| `--min-image-cc <num>` | Per-image CC limit, percent (default: no limit) |
|
||
| `--search-min-zeta <num>` | De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry \|ζ\| reaches this, and report both answers (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law. Where the two searches disagree, the merge of all the observations decides — as it always has for the systematic absences |
|
||
| `--mosaicity <num>` | Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed |
|
||
| `--scaling-iterations <num>` | Scaling iterations with no reference data (default: 3) |
|
||
| `-z, --reference-mtz <file>` | Reference MTZ (enables reference-driven scaling) |
|
||
| `--reference-column <label>` | Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
|
||
| `--model <file.pdb>` | After merging, validate the merged intensities against this atomic model (see below) |
|
||
| `--write-process-h5` | Also write the (large) `_process.h5` when merging (default: only `.mtz`/`.cif`) |
|
||
|
||
Integration:
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--integrator <txt>` | Spot integrator: `gaussian` (profile-fit, default) \| `empirical` \| `boxsum` (classical fallback) |
|
||
| `--integration-radius <r>` | Signal-box radius `r1`, or `r1,r2,r3` (px). One value ⇒ `r2=r1+2`, `r3=r1+4` |
|
||
| `--integration-stencil <k>` | Push the `r2..r3` background ring out by `k` times the beam's radial streak `bandwidth·R_px`, per reflection (default `0` = the fixed circular ring). A fixed ring otherwise ends up on a streaked reflection's own tails at high resolution and measures them as background. Only the ring moves, and only radially — the `r1` signal box stays a circle — and the growth is capped at `2·r3`. The neighbour exclusion grows with it, so on a crowded pattern a few reflections can be left with too little background and dropped. Needs `--bandwidth`: on a monochromatic beam the streak is zero and this does nothing |
|
||
| `--background-clip <n>` | Monochromatic (rotation + still): high-side clip of the background ring at `mean + n·√mean` (default 4; 0 = off). The default background estimator — it rejects neighbour cores and zingers without the symmetric trim's Poisson skew bias. Broadband data always clip, at 3σ; ignored by `--integrator boxsum` |
|
||
| `--background-trim <f>` | Use the old symmetric trimmed mean for the background ring instead of the clip, 0≤f<0.5 (`0.10` was the former default). Switches `--background-clip` off. A symmetric trim is biased low on Poisson data and adds ~5 counts to every partial, so this is for back compatibility only; `0` = plain ring mean. Rings holding more than 512 pixels fall back to the plain mean (the GPU sorts the ring in shared memory and the CPU now matches it), which the default radii never reach but wide ones do |
|
||
| `--background-radial[=on\|off\|auto]` | Correct the background ring for the **curvature** of the radial background (default **off**). Disk and ring are concentric, so a background linear in position cancels between them and only curvature survives — which on a smooth ice ring reaches +26 counts on a single reflection. `auto` applies it per image where that image's ice score shows a *smooth* powder ring, since the model is a function of radius alone: on ice made of discrete crystallite spots there is no smooth ring and the correction makes the bias worse. Ignored by `--integrator boxsum` (no clip pass to take the curve from) |
|
||
| `--integration-high-resolution <num>` | High-resolution limit for prediction and integration. Omitted (or 0) means integration extends as far as the detector reaches — which is what the predictor can place on the detector anyway, since it rejects reflections that miss it. Set a value to integrate less than the detector offers |
|
||
| `--max-hkl <n>` | Predict reflections with \|h\|,\|k\|,\|l\| ≤ `n` (max 511). By default this is derived per crystal from the refined cell as `ceil(max(a,b,c)/d_min) + 1`, which is the exact bound: the predictor keeps only \|q\| ≤ 1/d_min and `h = a·q`, so no reflection can lie outside it and no candidate inside it is wasted on a shorter axis. Set it only to override that |
|
||
| `--bandwidth <num>` | Relative X-ray bandwidth FWHM (e.g. `0.01` for a 1% DMM); default from file or 0 (monochromatic) |
|
||
|
||
Geometry overrides (defaults are taken from the input file; override them to reprocess with a corrected geometry):
|
||
|
||
| Option | Description |
|
||
| --- | --- |
|
||
| `--beam-x <num>` | Beam centre X (pixel) |
|
||
| `--beam-y <num>` | Beam centre Y (pixel) |
|
||
| `--detector-distance <num>` | Detector distance (mm) |
|
||
| `--wavelength <num>` | Wavelength (Å) |
|
||
| `--rot1 <num>` | PONI detector rotation 1 (rad) |
|
||
| `--rot2 <num>` | PONI detector rotation 2 (rad) |
|
||
| `--polarization <num>` | Polarization factor |
|