diff --git a/image_analysis/scale_merge/SearchSpaceGroup.h b/image_analysis/scale_merge/SearchSpaceGroup.h index 91077afb..47210ccc 100644 --- a/image_analysis/scale_merge/SearchSpaceGroup.h +++ b/image_analysis/scale_merge/SearchSpaceGroup.h @@ -123,9 +123,33 @@ struct SearchSpaceGroupOptions { // tails - away from the strong crystals, which were being handed margin out of their own noise // floor, and towards the weak ones, which had none. double min_operator_cc = 0.30; - // NOTE this is under-calibrated for any threshold in this range: a Pearson CC over 20 null pairs - // has sd ~0.24, so a 1.3-sigma fluctuation clears the bound. Raising it (or making the bound - // pair-count-aware) is a separate change with its own calibration. + // KNOWN-WEAK, DELIBERATELY LEFT AT 20 - do not raise it without reading this. + // + // 20 pairs is not a test. The null of the normalised CC - shell-matched random pairing over + // pass_cc with symmetry-related pairs vetoed, i.e. what a metrically-allowed but FALSE operator + // looks like - has a mean of -0.02 to +0.05 and a standard deviation C/sqrt(n) with C = 1.08 + // (0.98 to 1.23 across crystals), flat from 20 to 5000 pairs. Its upper tail is far heavier than + // normal theory, because E^2 is exponentially distributed: measured over 142 (crystal, n) points + // the exceedance is log10 P = -1.03 z + 0.69 with z = (min_operator_cc - null mean)/sd, i.e. ONE + // DECADE PER STANDARD DEVIATION where the normal law would give five. At 20 pairs a false + // operator therefore clears 0.30 on 12.4% of draws (and cleared the old bound of 0.5 on 3.2%). + // A bound that held that to 1e-3 on the worst merge measured would be 251 pairs. + // + // It is left at 20 because a pair-count FLOOR is the wrong shape for this problem, and 251 was + // measured to do harm. Nothing in the rotation battery is confirmed on noise: over its 314 + // candidate operators the sparsest carries 511 pairs and the weakest confirmed one stands 9.1 + // sigma above its own null, so there is no live case to fix. Meanwhile SearchSpaceGroup is run + // twice per data set (see Rugnux.cpp) - once on all observations, which decides, and once on a + // Lorentz-filtered merge that is deliberately starved and only reports - and on that second arm + // genuine operators reach 98 pairs. At 251 the sparsest crystal's filtered arm loses 19 of its + // 23 operators and the run prints "the Lorentz-filtered one supports 1" for an arm whose own + // correlations are 0.91-0.97, every one of them 9 sigma clear. A floor discards an operator on + // pair count alone however decisive its CC is, and that is what makes it the wrong instrument. + // + // The right fix, when it is worth its own calibration and battery, is the pair-count-aware bound + // this null hands over directly: require cc >= max(min_operator_cc, mu + C*z/sqrt(n)). That + // rejects the 20-pair noise the floor is aimed at while keeping every operator above, because it + // asks how large the correlation is and not only how many pairs it came from. int min_pairs_per_operator = 20; // Per-operator CC alone cannot tell a real weak operator from a false strong one (a noisy crystal's