A _process.h5 written before background_variance existed was read with var_bkg = 0, on the reasoning
that zero leaves the combine with the signal term alone, "which is what it had before". It does not.
Before, the combine back-derived the non-signal variance from sigma itself, and on a weak reflection
that is essentially the whole of sigma^2; zero deletes the dominant term and weights the reflection by
roughly 1/I instead of 1/sigma^2.
Recover it from the integrator's own identity, sigma^2 = I + var_bkg, when the dataset is absent.
Measured by re-scaling a stills _process.h5 with the dataset deleted, against the same file with it
intact: the automatic resolution cutoff was reading 1.66 A where the intact file reads 1.81, with
14731 unique reflections against 11508 - i.e. the zeroed file looked good enough to merge 0.15 A past
its own limit. Reconstructed, it reads 1.78 A and 11926, within 0.03 A of the intact file. The
residue is the profile-fit path, where var_bkg is not exactly sigma^2 - I and only the box-sum
identity is exact; that is recoverable to a closer approximation only by storing it, which is what
files written from now on do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
leonarski_fandClaude Opus 5
52ea727650