From 0c74e6b6668a87f4c8551a68078350ffbfb04c22 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 9 Nov 2026 14:44:13 +0100 Subject: [PATCH] Stills geometry: refine the free cell angles, -C as a prior and a hypothesis, place modules from the fitted geometry Three defects of the stills global geometry/cell fit, each moving the detector distance: - The cell angles were held at the start cell (InitCellAngles, "constant in v1") while the lengths and the distance were free, so a wrong angle went into the lengths and the distance: a monoclinic cell given with beta 4 degrees off refined to lengths and a distance both ~4% long, and a cell given with beta held at 90 on a crystal at ~90.9 left the distance 1.3% short. The angles the crystal system leaves free (monoclinic beta, triclinic all three) are now a refined block. The cell given is a Gaussian prior (1% on a length, 1 degree on an angle) weighed against the spots' ~1 px scatter, replacing the unused length regularizer; the refined-vs-start deviation (scale, lengths, angles) and the distance move are logged. A cell move, not only a distance move, now triggers the re-indexing refit. - The module placement started from the file's geometry: on a run whose header distance was 3% short and centre 6 px off, the module scan absorbed the centre into every module and the placement pushed two second-ring modules 13 px outwards - a radius-dependent push the gauge does not hold, which trades with the distance/cell-scale curvature - and the distance settled 0.8% long. The placement now starts from a fit of the same frames with the modules held, each frame's lattice taken from it (GeometryRefinerResult::lattices). Same data, right cell: the distance went from 1.0% long to the value a reference model's R-free puts it at, the modules moved < 1.4 px. - A cell given with the wrong shape still indexes the frames at a distance of its own and the fit cannot leave it. With -C on stills the geometry refinement is now run both with the cell given and de novo, from the file's geometry, and each is indexed on the same 96-frame sample with its own refined cell and geometry. The cell given is kept where it puts >= 0.9x as many spots on the lattice, or where its fit puts the detector within 2% of the de novo fit (a crystal form of the sample, possibly a minority one the user asked for); otherwise the run is processed as without -C, with a warning. `rugnux -C file` now gives the result of `rugnux file`. Costs one more geometry-refinement sample on -C stills runs (~+30-50 s on a 4000-frame run). Docs: RUGNUX_CLI (-C, --refine-geometry), CPU_DATA_ANALYSIS_INDEXING 7.7, RUGNUX_MODEL_AND_REFERENCE. Test: RefineGlobalGeometry_MonoclinicBeta. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK (cherry picked from commit fe442ff33aeec7569de1fdf90076dba6b39a8e8f) --- docs/CPU_DATA_ANALYSIS_INDEXING.md | 35 +++- docs/RUGNUX_CLI.md | 4 +- docs/RUGNUX_MODEL_AND_REFERENCE.md | 2 +- .../geom_refinement/GeometryRefiner.cpp | 105 +++++----- .../geom_refinement/GeometryRefiner.h | 26 ++- rugnux/ModuleDiagnostics.cpp | 9 +- rugnux/Rugnux.cpp | 8 +- rugnux/Rugnux.h | 4 + rugnux/RugnuxStills.cpp | 181 +++++++++++++++--- tests/ModuleRefinementTest.cpp | 35 ++++ 10 files changed, 313 insertions(+), 96 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index 6564911aa..00407e7f4 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -412,13 +412,28 @@ The extraction window around a ring is capped at half the gap to its neighbour, ### 7.7 Stills: a given cell and the detector modules -**A given cell is a starting point.** On stills a cell and a distance are degenerate - a spot's -radius fixes only their ratio. A cell taken from a deposited structure is routinely a percent off the -crystal at hand (another temperature, another batch), while a facility's stated distance is usually -better than that, so the stills geometry refinement holds the distance and refines the cell. Only where -the frames have rejected the file's distance (§1.4), or the file states the distance only nominally, is -the given cell anchored and the distance refined from it. Frames are matched to the given cell in any -setting of the same lattice, and are described on its axes. +**A given cell is a prior, and a hypothesis.** On stills a cell and a distance are degenerate near the +beam - a spot's radius fixes only their ratio - and are told apart only by the curvature of +$r=D\tan2\theta$ against $\sin\theta$ at high angle. The stills geometry refinement therefore refines +the beam centre, the distance and the cell together over the strong frames: the cell lengths and the +angles its crystal system leaves free (the monoclinic $\beta$, all three triclinic angles), each held to +the given cell by a Gaussian prior of 1 % on a length and 1° on an angle, weighed against the spots' +scatter of about a pixel. On a bundle of strong frames the spots outweigh the prior wherever they +determine the cell, so it decides only what they leave open. Holding the angles instead puts a wrong +angle into the lengths and the distance: a monoclinic cell given with $\beta$ 4° off was taken up as a +cell and a distance both 4 % long. The log states how far the refined cell is from the one it started +from (scale, each length, each angle) and how far the distance moved. A distance given with +`--detector-distance` is held, and the cell alone takes up its error. Frames are matched to the given +cell in any setting of the same lattice, and are described on its axes. + +With `-C` the cell given is also tested as a hypothesis against the cell the frames index to de novo. +Both are fitted from the file's geometry - the cell given as above, the de novo one as without `-C` - and +each, with its refined cell and geometry, indexes the same sample of frames. The cell given is kept where +it puts at least 0.9 times as many of the sample's spots on the lattice, and also where it does not but +its fit puts the detector within 2 % of where the de novo fit puts it: the detector is the same for every +crystal of the run, so such a cell is a crystal form of the sample (possibly a minority one, which the +user may be after). A cell that fits only with a detector of its own describes no crystal of the sample, +and the run is then processed as without `-C`, with a warning. **One crystal form.** A sample can hold two crystal forms, and a known-cell indexer fits both to the given cell's orientation search; their refined cells then fall into two volume populations, and @@ -457,7 +472,11 @@ fixed and with them placed, and scored on the other. The placement is used if it spots better as a whole, and then each module keeps its move only if the held-out spots on that module are predicted better; the others go back to where they were given and the placement is fitted again. On a detector of many modules the outer ones, reached only by a few strong high-resolution spots, are -the ones this decides on. `--panel-refine` keeps every move without the tests, `--panel-refine=off` none. The modules move only relative to one another: a +the ones this decides on. `--panel-refine` keeps every move without the tests, `--panel-refine=off` none. The placement starts from the beam, the distance and the cell fitted on the same frames with the modules held, +each frame's lattice taken from that fit, not from the file's geometry: from a start off in distance and centre +the search below moved every module by the centre's error and the placement then pushed two modules of the +second ring outwards, where the distance's error had left their spots, and the distance settled 0.8 % long. +The modules move only relative to one another: a gauge holds their spot-weighted mean shift, turn and outward push, and with the detector tilt refined also the two tilt-like gradients across the detector, at zero, so the beam centre, distance, tilt and crystal orientations carry what all modules share. A coarse search of each module's position comes diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index 1a17296cb..3c1dc4895 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -264,7 +264,7 @@ For very long axes the direction grid's angular resolution binds well below 500 | `--force-still` | Treat a rotation (goniometer) dataset as independent stills instead of rotation | | `-X, --indexing-algorithm ` | `Auto` (default) \| `FFT` \| `cuFFT` \| `FFTW` \| `FFBIDX` \| `None`, case-insensitive. `FFT` is the de-novo FFT search, on the GPU (cuFFT) where there is one and `--no-gpu` is not given, otherwise on the CPU (FFTW); `cuFFT` and `FFTW` name one implementation each; `cuFFT` needs a GPU and is refused without one. `FFBIDX` (known cell, `-C`) runs on the GPU where there is one and otherwise on the CPU. `Auto` takes `FFBIDX` on stills with a known cell, and `FFT` otherwise; rotation data always use the FFT search | | `-F` | Short for `-X FFT`, kept for old command lines | -| `-C, --unit-cell ` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`). On rotation data it also widens the FFT search to cover the cell given, at both ends: the longest axis looked for is raised to reach it, and `--fft-min-unit-cell` is lowered to admit it. A candidate that is an integer supercell of the cell given (a twin's coincidence lattice, a row taken at a multiple of its length) is stepped down to it | +| `-C, --unit-cell ` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`). On rotation data it also widens the FFT search to cover the cell given, at both ends: the longest axis looked for is raised to reach it, and `--fft-min-unit-cell` is lowered to admit it. A candidate that is an integer supercell of the cell given (a twin's coincidence lattice, a row taken at a multiple of its length) is stepped down to it. On stills the geometry refinement takes it as a prior (1 % on the lengths, 1° on the angles the crystal system leaves free) and tests it against the cell the frames index to de novo: a cell that fits the frames only with a detector distance of its own (more than 2 % from the de novo fit's) and puts fewer than 0.9 times as many sampled spots on the lattice is dropped with a warning, and the run is processed as without `-C` ([§7.7](CPU_DATA_ANALYSIS_INDEXING.md#77-stills-a-given-cell-and-the-detector-modules)) | | `--fft-min-unit-cell ` | Shortest cell axis the FFT search accepts, Å (default: 10). A candidate with a shorter axis is discarded — but on rotation data a row shorter than the floor, down to 3 Å, is recognised under the multiple of it the search reports past the floor, so a small-molecule cell is indexed without this flag. `-C` lowers the floor on its own to cover the cell given. On stills a crystal below the floor cannot be indexed unless this is lowered | | `--min-indexed-spots ` | Spots a frame must have on the lattice before it counts as indexed (default: 6, minimum 4). It sets the reported indexing rate and the count the rotation first pass ranks candidate lattices by; whether the run has a lattice at all is decided on the pooled spots instead ([§4.1](CPU_DATA_ANALYSIS_INDEXING.md#41-indexed-spot-decision-inlier-test)), and integration is not gated by it | | `-S, --space-group ` | Space group number (`96`) or Hermann-Mauguin symbol (`P43212`) — for indexing and scaling. On serial stills whose group admits two indexings, it also lets the crystals be put on one from the data ([the indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity)) | @@ -273,7 +273,7 @@ For very long axes the direction grid's angular resolution binds well below 500 | `--force-rotation-lattice ` | Force rotation lattice (9 floats, Å), skipping the first pass | | `--rotation-no-postrefine` | Rotation: disable the default-on two-pass geometry post-refine (see [Rotation data](RUGNUX_TUTORIAL.md#rotation-data)) | | `--prepass-fraction ` | Rotation: the two-pass geometry pre-pass integrates the first fraction `f` of the sweep, but at least 120° of it (0.05-1, default **0.5**; `1` integrates the whole sweep twice). The second pass always integrates every frame. The pre-scan and the first-pass indexing sample the same part of the sweep | -| `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills, a stationary goniometer axis counting as stills, `=off` disables. The bundle anchors on the reference cell (`-C` / `-z`); without one, on the cell most of the sampled frames index to de novo, which is used for the geometry steps only (the run is still indexed de novo) | +| `--refine-geometry[=N\|off]` | Stills: extra first pass that bundle-adjusts the shared beam/distance/cell from N strongly-indexed frames (default 200) then re-indexes; default ON for stills, a stationary goniometer axis counting as stills, `=off` disables. The cell's lengths and the angles its crystal system leaves free are refined with the distance; the bundle anchors them on the reference cell (`-C` / `-z`) by a prior of 1 % / 1°; without one, on the cell most of the sampled frames index to de novo, which is used for the geometry steps only (the run is still indexed de novo) | | `--index-ice-rings[=on\|off]` | Index on the spots flagged as sitting on an ice ring too, instead of setting them aside (default: **off**; no effect without `--detect-ice-rings`, which does the flagging) | Indexer choice in brief: `ffbidx` (GPU or CPU) refines toward a **known cell** and is best for sparse diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index baa27592f..d68ff1dd5 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -58,7 +58,7 @@ the rest walks through a run step by step. | `--reference-column