Merging: do not floor the merged sigma at the systematic term

The merged sigma was floored at b*|I|, so I/sigma could never exceed the reported ISa.
On one dataset every merged reflection came out at I/sigma <= 12.96 with a 99th
percentile of 12.77 in every resolution shell alike, while the scatter of the
observations implied about 44 and XDS reported 58.

The floor is wrong in principle. `b` is fitted from the scatter BETWEEN a reflection's
symmetry equivalents, i.e. from the part that is not common to them, so it averages
down with multiplicity exactly like the counting term. 1/sqrt(sum_w) with the
b-inflated per-observation sigma already gives b*I/sqrt(n); flooring at b*|I| puts the
sqrt(n) back. That is the whole effect: 12.96 * sqrt(21.6) = 60, against XDS's 58.

It was introduced on a comparison of our MERGED I/sigma against XDS's UNMERGED
I/sigma. XDS's own merged low-resolution I/sigma exceeds its reported ISa on 30 of the
39 reference datasets here, median ratio 1.78 and up to 4.23.

Merged low-shell I/sigma now lands where XDS's does: 22.4 -> 46.2 against 46.2 on one
crystal, 26.7 -> 115.7 against 96.6 on another, 12.5 -> 45.0 against 58.0 on a third.
Over the 38-crystal battery the space groups, the merged reflection sets, R_meas and
CC1/2 are all unchanged - every one of them is sigma-independent, which is what makes
them the right control - and <I/sigma> rises on 35 crystals with none worse.

The asymptotic estimator that fed the floor stays, for the reported ISa only, and is
repaired in the process: it subtracts a*sigma^2 rather than the raw sigma^2 (at a < 1
the difference is the same size as the b^2 being measured, which is what made it
flip between 10.9 and 62.7 on consecutive passes of the same data), it rescales each
group's variance median-unbiased before subtracting an unbiased counting term, its
I/sigma gate uses the same convention, and it is bounded by the whole-range b - an
asymptote exists to refine 1/b upward, not to report 0.3 because "strong" was selected
on a sigma scale the fit itself rejects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-09 20:58:35 +02:00
co-authored by Claude Opus 5
parent f60768d49c
commit 72efb75a8c
8 changed files with 82 additions and 87 deletions
+4 -4
View File
@@ -76,8 +76,8 @@ namespace {
// model's b matching the systematic scatter that is actually there.
{"sigmas 1.7x too small", 1.7, 0.05, std::nullopt},
// The other way a fitted error model misses: the statistical sigmas come out somewhat too
// LARGE while b - the asymptotic I/sigma ceiling, ISa = 1/b - is fitted 3x too optimistic, so
// the systematic scatter present is 3x what the merged sigmas' floor admits.
// LARGE while b - the asymptotic per-observation I/sigma, ISa = 1/b - is fitted 3x too
// optimistic, so the systematic scatter present is 3x what the merged sigmas admit.
{"ISa 3x too optimistic", 0.6, 0.02, 0.06},
};
@@ -259,8 +259,8 @@ TEST_CASE("SearchSpaceGroup on a perfect merohedral twin returns one of the two
}
}
// THE property this harness exists for. Multiplicity changes only the sigmas - the random part of a
// merged sigma averages down as 1/sqrt(n) while the systematic floor b*|I| does not - so it changes
// THE property this harness exists for. Multiplicity changes only the sigmas - a merged sigma averages
// down as 1/sqrt(n) while the systematic error the crystal carries does not - so it changes
// how well the SAME crystal is measured, never what its symmetry is. A symmetry decision that moves
// when the same crystal is merged 2x instead of 18x is a defect of the criterion, not a property of
// the data.