Merging: do not floor the merged sigma at the systematic term

The merged sigma was floored at b*|I|, so I/sigma could never exceed the reported ISa.
On one dataset every merged reflection came out at I/sigma <= 12.96 with a 99th
percentile of 12.77 in every resolution shell alike, while the scatter of the
observations implied about 44 and XDS reported 58.

The floor is wrong in principle. `b` is fitted from the scatter BETWEEN a reflection's
symmetry equivalents, i.e. from the part that is not common to them, so it averages
down with multiplicity exactly like the counting term. 1/sqrt(sum_w) with the
b-inflated per-observation sigma already gives b*I/sqrt(n); flooring at b*|I| puts the
sqrt(n) back. That is the whole effect: 12.96 * sqrt(21.6) = 60, against XDS's 58.

It was introduced on a comparison of our MERGED I/sigma against XDS's UNMERGED
I/sigma. XDS's own merged low-resolution I/sigma exceeds its reported ISa on 30 of the
39 reference datasets here, median ratio 1.78 and up to 4.23.

Merged low-shell I/sigma now lands where XDS's does: 22.4 -> 46.2 against 46.2 on one
crystal, 26.7 -> 115.7 against 96.6 on another, 12.5 -> 45.0 against 58.0 on a third.
Over the 38-crystal battery the space groups, the merged reflection sets, R_meas and
CC1/2 are all unchanged - every one of them is sigma-independent, which is what makes
them the right control - and <I/sigma> rises on 35 crystals with none worse.

The asymptotic estimator that fed the floor stays, for the reported ISa only, and is
repaired in the process: it subtracts a*sigma^2 rather than the raw sigma^2 (at a < 1
the difference is the same size as the b^2 being measured, which is what made it
flip between 10.9 and 62.7 on consecutive passes of the same data), it rescales each
group's variance median-unbiased before subtracting an unbiased counting term, its
I/sigma gate uses the same convention, and it is bounded by the whole-range b - an
asymptote exists to refine 1/b upward, not to report 0.3 because "strong" was selected
on a sigma scale the fit itself rejects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-09 20:58:35 +02:00
co-authored by Claude Opus 5
parent f60768d49c
commit 72efb75a8c
8 changed files with 82 additions and 87 deletions
+2 -4
View File
@@ -358,8 +358,7 @@ std::vector<MergedReflection> MergeOnTheFly::ExportReflections() {
.k = accum.k,
.l = accum.l,
.I = static_cast<float>(accum.sum_wI / accum.sum_w),
.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(accum.sum_w),
static_cast<float>(accum.sum_wI / accum.sum_w), error_model_b),
.sigma = static_cast<float>(1.0 / std::sqrt(accum.sum_w)),
.I_half = {NAN, NAN},
.sigma_half = {NAN, NAN},
.d = accum.d
@@ -368,8 +367,7 @@ std::vector<MergedReflection> MergeOnTheFly::ExportReflections() {
if (accum.n_half[0] + accum.n_half[1] > 0 && accum.sum_w_half[0] > 0.0 && accum.sum_w_half[1] > 0.0) {
for (int i = 0; i < 2; ++i) {
mr.I_half[i] = static_cast<float>(accum.sum_wI_half[i] / accum.sum_w_half[i]);
mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(accum.sum_w_half[i]),
mr.I_half[i], error_model_b);
mr.sigma_half[i] = static_cast<float>(1.0 / std::sqrt(accum.sum_w_half[i]));
}
}