The merged sigma was floored at b*|I|, so I/sigma could never exceed the reported ISa.
On one dataset every merged reflection came out at I/sigma <= 12.96 with a 99th
percentile of 12.77 in every resolution shell alike, while the scatter of the
observations implied about 44 and XDS reported 58.
The floor is wrong in principle. `b` is fitted from the scatter BETWEEN a reflection's
symmetry equivalents, i.e. from the part that is not common to them, so it averages
down with multiplicity exactly like the counting term. 1/sqrt(sum_w) with the
b-inflated per-observation sigma already gives b*I/sqrt(n); flooring at b*|I| puts the
sqrt(n) back. That is the whole effect: 12.96 * sqrt(21.6) = 60, against XDS's 58.
It was introduced on a comparison of our MERGED I/sigma against XDS's UNMERGED
I/sigma. XDS's own merged low-resolution I/sigma exceeds its reported ISa on 30 of the
39 reference datasets here, median ratio 1.78 and up to 4.23.
Merged low-shell I/sigma now lands where XDS's does: 22.4 -> 46.2 against 46.2 on one
crystal, 26.7 -> 115.7 against 96.6 on another, 12.5 -> 45.0 against 58.0 on a third.
Over the 38-crystal battery the space groups, the merged reflection sets, R_meas and
CC1/2 are all unchanged - every one of them is sigma-independent, which is what makes
them the right control - and <I/sigma> rises on 35 crystals with none worse.
The asymptotic estimator that fed the floor stays, for the reported ISa only, and is
repaired in the process: it subtracts a*sigma^2 rather than the raw sigma^2 (at a < 1
the difference is the same size as the b^2 being measured, which is what made it
flip between 10.9 and 62.7 on consecutive passes of the same data), it rescales each
group's variance median-unbiased before subtracting an unbiased counting term, its
I/sigma gate uses the same convention, and it is bounded by the whole-range b - an
asymptote exists to refine 1/b upward, not to report 0.3 because "strong" was selected
on a sigma scale the fit itself rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
leonarski_fandClaude Opus 5
72efb75a8c