rugnux: adaptive spot detection is the default for rotation data too

It was held back because a 33-crystal rotation battery showed it breaking three crystals deterministically - a lost space group, a halved indexing rate and a collapsed merge. None of those causes turned out to be in detection. The extra spots adaptive finds are real. Measured per spot against a finder-neutral local background: 64% recur at the same position on the adjacent frame (chance rate 0.5%) with 2-frame rocking curves, and 0.00% would fail a conventional local SNR >= 4 test, median local SNR 34. What they include is genuine peaks belonging to no lattice the indexer found, and the damage they did scaled with their absolute COUNT (80.6 per frame against 36.8 for the fixed finder), not with their quality - which is why nothing aimed at judging individual spots ever worked. The three failures fell to fixes elsewhere: merge collapsed - a per-frame scale free to collapse toward zero amplified two junk frames by 546x (704098712, ec7a82613). Not a detection problem at all: the fixed-threshold finder trips the same bug on a different frame range. space group lost - the per-image geometry refinement was dragged 2.3-2.8 deg off by the weak-spot tail in an unweighted fit; weighting each spot by how strong it is FOR ITS RESOLUTION fixed it (ae126c3d5), and gained a point group for the fixed finder too. indexing halved - gone with the same two; that crystal is now better under adaptive (CC1/2 92.6 -> 95.9, high-shell 44.9 -> 56.4). Battery, 33 crystals, adaptive vs the fixed finder: exact space group matching XDS 26/33 vs 25/33 point group matching XDS 29/33 vs 30/33 ISa better on 6 crystals recovers a screw axis the other misses (P321 -> P3121) The one point group it loses is a tetragonal crystal where adaptive collects 2.4x the observations at better R_meas (21.4% vs 28.3%) and better ISa (4.58 vs 2.91), and forced to the right group gives CC1/2 99.2% at multiplicity 10.7 - matching XDS. Only the automatic symmetry call fails there, and six candidate causes have been measured and refuted (mixed indexing hands, off-crystal frames, a badly integrated minority, radiation damage, pseudo-tetragonality, uncorrected anisotropy). It is left as the subgroup, which is the recoverable direction: -S gives XDS-quality data from the same run, whereas the failures this unblocks were not recoverable. --no-adaptive-spots reverts to the fixed-threshold finder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:59:37 +02:00
parent 4895dc1018
commit 6f4917dcee
2 changed files with 3 additions and 3 deletions
@@ -187,7 +187,7 @@ Special cases:
 ### 3.2 Adaptive (self-calibrating) detection
-The local-statistics test above still needs a fixed photon/count threshold, and the right value depends on the background level, which varies between datasets — so it has to be tuned per dataset. The **adaptive** mode (`--adaptive-spots`; the default for stills in `rugnux` and in the viewer, `--no-adaptive-spots` reverts) removes that tuning by deriving the threshold from each image's own noise, per resolution ring. Rotation data keeps the fixed-threshold finder by default: across a 33-crystal rotation battery adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge).
+The local-statistics test above still needs a fixed photon/count threshold, and the right value depends on the background level, which varies between datasets — so it has to be tuned per dataset. The **adaptive** mode (`--adaptive-spots`; the default in `rugnux` and in the viewer, `--no-adaptive-spots` reverts) removes that tuning by deriving the threshold from each image's own noise, per resolution ring. It is the default for rotation data as well: it was held back while a 33-crystal battery showed it breaking three crystals (a lost space group, a halved indexing rate, a collapsed merge), but none of those causes lay in detection. Adaptive finds roughly twice as many spots and they are real — 64 % recur at the same position on the adjacent frame against a 0.5 % chance rate, and none would fail a conventional local-SNR test — but they include genuine peaks that belong to no indexed lattice, and the damage they did scaled with their absolute *count*. Guarding the per-frame scale against collapse and weighting each spot by how strong it is for its resolution (§ per-image refinement) removed all three failures; adaptive now matches the exact space group on 26 of 33 crystals against 25 for the fixed-threshold finder, with better ISa on six.
 Pixels are binned into the same resolution rings as the azimuthal integrator (§2). For each ring a robust background is estimated in three passes: one plain pass over all valid pixels, then two $\sigma$-clipping passes that keep only pixels within $\pm 3\sigma$ of the current ring mean (removing the Bragg peaks from the background estimate). This yields a per-ring background mean $\mu_b$ and scatter $\sigma_b$.
@@ -66,7 +66,7 @@ void print_usage() {
    std::cout << "   --spot-sigma <num>               Noise sigma level for spot finding (default: 3.0)" << std::endl;
    std::cout << "   --spot-threshold <num>           Photon count threshold for spot finding (default: 10)" << std::endl;
    std::cout << "   --min-pix-per-spot <num>         Minimum connected strong pixels per spot. If omitted, min-pix is chosen PER IMAGE (stills indexing): the frame is indexed at min-pix 3/2/1 and the one maximising indexed count x indexed fraction is kept. Give an explicit value to force a fixed min-pix instead." << std::endl;
-    std::cout << "   --adaptive-spots                 Self-calibrating detection (DEFAULT for stills): the strong-pixel threshold comes from each image's own per-resolution-ring noise instead of the fixed --spot-threshold, so one setting adapts across datasets (no per-dataset --spot-threshold/--spot-sigma tuning). Rotation data keeps the fixed-threshold finder unless this is given." << std::endl;
+    std::cout << "   --adaptive-spots                 Self-calibrating detection (DEFAULT): the strong-pixel threshold comes from each image's own per-resolution-ring noise instead of the fixed --spot-threshold, so one setting adapts across datasets (no per-dataset --spot-threshold/--spot-sigma tuning)." << std::endl;
    std::cout << "   --no-adaptive-spots              Turn adaptive detection off and use the fixed --spot-threshold / --spot-sigma finder instead" << std::endl;
    std::cout << "   --spot-false-pixels <num>        Adaptive detection operating point: expected noise pixels tolerated per frame (default: 100; implies --adaptive-spots)" << std::endl;
    std::cout << "   --spot-high-resolution <num>     High resolution limit for spot finding. If omitted, stills extend as far as the detector reaches (no resolution clipping) and rotation data keeps a 1.5 A limit." << std::endl;
@@ -1495,7 +1495,7 @@ int main(int argc, char **argv) {
        spot_settings.min_pix_per_spot = 2;
    if (rotation_indexing && !d_min_spot_finding.has_value())
        d_min_spot_finding = 1.5f;
-    spot_settings.adaptive_threshold = adaptive_spots.value_or(!rotation_indexing);
+    spot_settings.adaptive_threshold = adaptive_spots.value_or(true);
    spot_settings.high_resolution_limit = d_min_spot_finding;
    spot_settings.false_pixels_per_frame = false_pixels_per_frame;
    if (d_max_spot_finding > 0.0f)