From 64957d5a5f446cd71d085e465e27f7f87c9340fb Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 26 Aug 2026 01:44:32 +0200 Subject: [PATCH] Space-group search: record what min_pairs_per_operator is actually worth Comment only; the constant stays at 20. f6bfa5462 moved min_operator_cc to 0.30, which is close enough to the null of a sparse operator that the pair minimum deserved measuring rather than assuming. Measured, over 38 crystals and 142 (crystal, pair-count) points, the null of the normalised operator correlation has mean -0.02 to +0.05 and standard deviation 1.08/sqrt(n), flat from 20 pairs to 5000. Its tail is the part worth writing down: log10 P(null CC >= 0.30) = -1.03 z + 0.69, residual sd 0.14 decades One decade per standard deviation, where normal theory gives about five. So a false operator at 20 pairs clears 0.30 on 12.4% of draws, and any calibration of this gate that reasons from 1/sqrt(n-3) with a normal or Fisher-z tail is optimistic by one to two orders of magnitude. It does not bite where the decision is made. Over 314 candidate operators on 36 crystals the sparsest has 520 pairs and none is below 500; the weakest confirmed operator sits 9.1 sigma above its own null and the strongest rejected one 42.7 sigma above. Nothing passes on noise. The search's own cut is what keeps that null flat - reconstructing the same crystals from an unfiltered merge reproduces a null floor of 0.10 to 0.20, which is where the "false CC rises at low pair count" impression came from. That describes a shrinking merge, not a sparse operator. A minimum of 251 pairs would buy a per-operator false rate of 1e-3. It was built and rejected on measurement: SearchSpaceGroup runs twice, once on all observations, which decides, and once on a Lorentz-filtered merge, which only reports - and no log prints the second arm's pair counts. Genuine operators fall to 98 pairs there, so a floor of 251 empties a conjugacy class on one crystal and makes the report say its filtered merge supports triclinic, about a merge whose own correlations put every one of those operators 9 sigma clear at 0.91 to 0.97. Any floor above about 150 does that; any below about 103 is barely better than 20. A count floor rejects on pair count however decisive the correlation is, which is why the shape is wrong rather than the value. The comment records the law, the rejected value, and the form that would work if this is ever revisited - a significance floor rather than a count floor. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P --- image_analysis/scale_merge/SearchSpaceGroup.h | 30 +++++++++++++++++-- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/image_analysis/scale_merge/SearchSpaceGroup.h b/image_analysis/scale_merge/SearchSpaceGroup.h index 91077afb..47210ccc 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.h +++ b/image_analysis/scale_merge/SearchSpaceGroup.h @@ -123,9 +123,33 @@ struct SearchSpaceGroupOptions { // tails - away from the strong crystals, which were being handed margin out of their own noise // floor, and towards the weak ones, which had none. double min_operator_cc = 0.30; - // NOTE this is under-calibrated for any threshold in this range: a Pearson CC over 20 null pairs - // has sd ~0.24, so a 1.3-sigma fluctuation clears the bound. Raising it (or making the bound - // pair-count-aware) is a separate change with its own calibration. + // KNOWN-WEAK, DELIBERATELY LEFT AT 20 - do not raise it without reading this. + // + // 20 pairs is not a test. The null of the normalised CC - shell-matched random pairing over + // pass_cc with symmetry-related pairs vetoed, i.e. what a metrically-allowed but FALSE operator + // looks like - has a mean of -0.02 to +0.05 and a standard deviation C/sqrt(n) with C = 1.08 + // (0.98 to 1.23 across crystals), flat from 20 to 5000 pairs. Its upper tail is far heavier than + // normal theory, because E^2 is exponentially distributed: measured over 142 (crystal, n) points + // the exceedance is log10 P = -1.03 z + 0.69 with z = (min_operator_cc - null mean)/sd, i.e. ONE + // DECADE PER STANDARD DEVIATION where the normal law would give five. At 20 pairs a false + // operator therefore clears 0.30 on 12.4% of draws (and cleared the old bound of 0.5 on 3.2%). + // A bound that held that to 1e-3 on the worst merge measured would be 251 pairs. + // + // It is left at 20 because a pair-count FLOOR is the wrong shape for this problem, and 251 was + // measured to do harm. Nothing in the rotation battery is confirmed on noise: over its 314 + // candidate operators the sparsest carries 511 pairs and the weakest confirmed one stands 9.1 + // sigma above its own null, so there is no live case to fix. Meanwhile SearchSpaceGroup is run + // twice per data set (see Rugnux.cpp) - once on all observations, which decides, and once on a + // Lorentz-filtered merge that is deliberately starved and only reports - and on that second arm + // genuine operators reach 98 pairs. At 251 the sparsest crystal's filtered arm loses 19 of its + // 23 operators and the run prints "the Lorentz-filtered one supports 1" for an arm whose own + // correlations are 0.91-0.97, every one of them 9 sigma clear. A floor discards an operator on + // pair count alone however decisive its CC is, and that is what makes it the wrong instrument. + // + // The right fix, when it is worth its own calibration and battery, is the pair-count-aware bound + // this null hands over directly: require cc >= max(min_operator_cc, mu + C*z/sqrt(n)). That + // rejects the 20-pair noise the floor is aimed at while keeping every operator above, because it + // asks how large the correlation is and not only how many pairs it came from. int min_pairs_per_operator = 20; // Per-operator CC alone cannot tell a real weak operator from a false strong one (a noisy crystal's