Bragg integration: drop the 2% sigma floor and carry the background variance

Two changes to the same variance chain; they are in one commit because the second
exists to remove an assumption the first was breaking, and separating them leaves a
tree that is correct only by luck.

The reported sigma was floored at 2% of the intensity, a per-partial I/sigma cap of
50. It applied only to the box-sum seed, never to the profile fit, so the shipped
default was unaffected - but the combine back-derives each partial's non-signal
variance as sigma^2 - I, and a floored sigma makes that quantity mean nothing. It
then read corr^2 * (0.0004 I^2 - I), which is not a background variance. Measured on
--integrator boxsum: the reported sigma understated the true scatter by up to 16x at
I ~ 21000 counts per partial, and pooled_I amplified a 1 ct/px background drift into
an 11.5% intensity error on the strongest reflections.

What the floor stood in for - that at high intensity the error is systematic rather
than counting - is already carried downstream, twice: the fitted b in
v = a*sigma^2 + (b*I)^2, measured from the data rather than assumed, and
SigmaWithSystematicFloor on the merged sigma. The floor was that idea applied one
level too early with a hardcoded b of 0.02. It arrived without a test or a setter and
was unreachable from the CLI, the API and the config.

The merge now takes the non-signal variance the integrator actually measured instead
of inverting sigma^2 = I + N. That identity is exact for a box sum once the floor is
gone and was never exact for a profile fit, whose sigma^2 = 1/den + (wsum/den)^2 *
bkg_var is formed against a fitted intensity. The value is carried through
BraggFitResult, Reflection and Obs, both engines, both merges, and the process-file
round trip; files written before this change are read with the term absent, which is
what they had.

Battery, 37 crystals, paired: space groups unchanged, reflection sets unchanged,
median delta zero on R_meas and CC1/2. --integrator boxsum on the reference crystal
goes ISa 8.9 -> 20.2 with a 0.947 -> 1.032.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-09 19:10:57 +02:00
co-authored by Claude Opus 5
parent d565b66916
commit f60768d49c
14 changed files with 89 additions and 35 deletions
+9
View File
@@ -150,6 +150,8 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
auto int_sum = file.ReadOptVector<float>(image_group_name + "/int_sum");
auto int_err = file.ReadOptVector<float>(image_group_name + "/int_err");
auto bkg = file.ReadOptVector<float>(image_group_name + "/background_mean");
// Written since the merge stopped back-deriving it; older _process.h5 do not carry it.
auto var_bkg = file.ReadOptVector<float>(image_group_name + "/background_variance");
auto lp = file.ReadOptVector<float>(image_group_name + "/lp");
auto partiality = file.ReadOptVector<float>(image_group_name + "/partiality");
auto phi = file.ReadOptVector<float>(image_group_name + "/delta_phi");
@@ -177,6 +179,12 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
if (zeta.size() > i)
zeta_val = zeta[i];
// A file written before the merge carried this cannot say what the non-signal variance was;
// 0 leaves the combine with the signal term alone, which is what it had before.
float var_bkg_val = 0.0f;
if (var_bkg.size() > i)
var_bkg_val = var_bkg[i];
float image_scale_corr_val = 1.0f; // Default is 1.0, if we don't know any better
if (image_scale_corr.size() > i)
image_scale_corr_val = image_scale_corr[i];
@@ -202,6 +210,7 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
.d = d.at(i),
.I = int_sum.at(i),
.bkg = bkg.at(i),
.var_bkg = var_bkg_val,
.sigma = int_err.at(i),
.rlp = lp_val,
.partiality = partiality_val,