Space-group search: record what min_pairs_per_operator is actually worth
Comment only; the constant stays at 20.
f6bfa5462 moved min_operator_cc to 0.30, which is close enough to the null of
a sparse operator that the pair minimum deserved measuring rather than
assuming. Measured, over 38 crystals and 142 (crystal, pair-count) points,
the null of the normalised operator correlation has mean -0.02 to +0.05 and
standard deviation 1.08/sqrt(n), flat from 20 pairs to 5000. Its tail is the
part worth writing down:
log10 P(null CC >= 0.30) = -1.03 z + 0.69, residual sd 0.14 decades
One decade per standard deviation, where normal theory gives about five. So a
false operator at 20 pairs clears 0.30 on 12.4% of draws, and any calibration
of this gate that reasons from 1/sqrt(n-3) with a normal or Fisher-z tail is
optimistic by one to two orders of magnitude.
It does not bite where the decision is made. Over 314 candidate operators on
36 crystals the sparsest has 520 pairs and none is below 500; the weakest
confirmed operator sits 9.1 sigma above its own null and the strongest
rejected one 42.7 sigma above. Nothing passes on noise. The search's own
<I/sigma> cut is what keeps that null flat - reconstructing the same crystals
from an unfiltered merge reproduces a null floor of 0.10 to 0.20, which is
where the "false CC rises at low pair count" impression came from. That
describes a shrinking merge, not a sparse operator.
A minimum of 251 pairs would buy a per-operator false rate of 1e-3. It was
built and rejected on measurement: SearchSpaceGroup runs twice, once on all
observations, which decides, and once on a Lorentz-filtered merge, which only
reports - and no log prints the second arm's pair counts. Genuine operators
fall to 98 pairs there, so a floor of 251 empties a conjugacy class on one
crystal and makes the report say its filtered merge supports triclinic, about
a merge whose own correlations put every one of those operators 9 sigma clear
at 0.91 to 0.97. Any floor above about 150 does that; any below about 103 is
barely better than 20. A count floor rejects on pair count however decisive
the correlation is, which is why the shape is wrong rather than the value.
The comment records the law, the rejected value, and the form that would work
if this is ever revisited - a significance floor rather than a count floor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P
This commit is contained in:
@@ -123,9 +123,33 @@ struct SearchSpaceGroupOptions {
|
||||
// tails - away from the strong crystals, which were being handed margin out of their own noise
|
||||
// floor, and towards the weak ones, which had none.
|
||||
double min_operator_cc = 0.30;
|
||||
// NOTE this is under-calibrated for any threshold in this range: a Pearson CC over 20 null pairs
|
||||
// has sd ~0.24, so a 1.3-sigma fluctuation clears the bound. Raising it (or making the bound
|
||||
// pair-count-aware) is a separate change with its own calibration.
|
||||
// KNOWN-WEAK, DELIBERATELY LEFT AT 20 - do not raise it without reading this.
|
||||
//
|
||||
// 20 pairs is not a test. The null of the normalised CC - shell-matched random pairing over
|
||||
// pass_cc with symmetry-related pairs vetoed, i.e. what a metrically-allowed but FALSE operator
|
||||
// looks like - has a mean of -0.02 to +0.05 and a standard deviation C/sqrt(n) with C = 1.08
|
||||
// (0.98 to 1.23 across crystals), flat from 20 to 5000 pairs. Its upper tail is far heavier than
|
||||
// normal theory, because E^2 is exponentially distributed: measured over 142 (crystal, n) points
|
||||
// the exceedance is log10 P = -1.03 z + 0.69 with z = (min_operator_cc - null mean)/sd, i.e. ONE
|
||||
// DECADE PER STANDARD DEVIATION where the normal law would give five. At 20 pairs a false
|
||||
// operator therefore clears 0.30 on 12.4% of draws (and cleared the old bound of 0.5 on 3.2%).
|
||||
// A bound that held that to 1e-3 on the worst merge measured would be 251 pairs.
|
||||
//
|
||||
// It is left at 20 because a pair-count FLOOR is the wrong shape for this problem, and 251 was
|
||||
// measured to do harm. Nothing in the rotation battery is confirmed on noise: over its 314
|
||||
// candidate operators the sparsest carries 511 pairs and the weakest confirmed one stands 9.1
|
||||
// sigma above its own null, so there is no live case to fix. Meanwhile SearchSpaceGroup is run
|
||||
// twice per data set (see Rugnux.cpp) - once on all observations, which decides, and once on a
|
||||
// Lorentz-filtered merge that is deliberately starved and only reports - and on that second arm
|
||||
// genuine operators reach 98 pairs. At 251 the sparsest crystal's filtered arm loses 19 of its
|
||||
// 23 operators and the run prints "the Lorentz-filtered one supports 1" for an arm whose own
|
||||
// correlations are 0.91-0.97, every one of them 9 sigma clear. A floor discards an operator on
|
||||
// pair count alone however decisive its CC is, and that is what makes it the wrong instrument.
|
||||
//
|
||||
// The right fix, when it is worth its own calibration and battery, is the pair-count-aware bound
|
||||
// this null hands over directly: require cc >= max(min_operator_cc, mu + C*z/sqrt(n)). That
|
||||
// rejects the 20-pair noise the floor is aimed at while keeping every operator above, because it
|
||||
// asks how large the correlation is and not only how many pairs it came from.
|
||||
int min_pairs_per_operator = 20;
|
||||
|
||||
// Per-operator CC alone cannot tell a real weak operator from a false strong one (a noisy crystal's
|
||||
|
||||
Reference in New Issue
Block a user