From 52ea7276504d813e92a94417c8eb0e41b5ab5824 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 10 Aug 2026 05:52:56 +0200 Subject: [PATCH] Reader: reconstruct the background variance a legacy file does not store A _process.h5 written before background_variance existed was read with var_bkg = 0, on the reasoning that zero leaves the combine with the signal term alone, "which is what it had before". It does not. Before, the combine back-derived the non-signal variance from sigma itself, and on a weak reflection that is essentially the whole of sigma^2; zero deletes the dominant term and weights the reflection by roughly 1/I instead of 1/sigma^2. Recover it from the integrator's own identity, sigma^2 = I + var_bkg, when the dataset is absent. Measured by re-scaling a stills _process.h5 with the dataset deleted, against the same file with it intact: the automatic resolution cutoff was reading 1.66 A where the intact file reads 1.81, with 14731 unique reflections against 11508 - i.e. the zeroed file looked good enough to merge 0.15 A past its own limit. Reconstructed, it reads 1.78 A and 11926, within 0.03 A of the intact file. The residue is the profile-fit path, where var_bkg is not exactly sigma^2 - I and only the box-sum identity is exact; that is recoverable to a closer approximation only by storing it, which is what files written from now on do. Co-Authored-By: Claude Opus 5 (1M context) --- docs/CHANGELOG.md | 4 ++++ docs/HDF5.md | 1 + reader/HDF5MetadataSource.cpp | 9 ++++++--- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 25a6427e..39a51533 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -12,6 +12,10 @@ This is an UNSTABLE release. It includes many experimental features, as well as * Resolution limits: Bragg integration, azimuthal integration and spot finding all default to **as far as the detector reaches**; `--integration-high-resolution` and `--spot-high-resolution` still set one by hand, and 0 means "no limit" at either end. * Bragg prediction: How far the predictor walks the lattice is a setting (`bragg_integration_settings.max_hkl`, `--max-hkl`) instead of a fixed 100, derived per crystal offline and held fixed online. * Rotation data: a set X-ray bandwidth (`--bandwidth`) now widens each reflection's rocking curve by Δλ/λ·tan(θ) in prediction and partiality, and is deconvolved out of the fitted mosaicity; monochromatic data are unaffected. +* Bragg integration: `background_variance` is now written to the CBOR reflection stream, so files written by the broker carry it as well as those written by rugnux. +* Scaling: a `_process.h5` written before `background_variance` existed now has it reconstructed on read instead of taken as zero, which had let such a file merge past its true resolution limit. +* Space-group search: the operator-agreement bound and the resolution cut that feeds the search were both loosened to match measured data, so a genuine symmetry is no longer refused on weak or lopsided sweeps. +* Scaling: the detector modulation surface is refined on a 24×24 grid instead of 16×16. * Bragg integration: The local background ring is made robust with a **high-side sigma clip** (`--background-clip `, default 4) instead of the symmetric trimmed mean; the trim stays reachable with `--background-trim `. * Bragg integration: The **uncertainty of the background estimate** is now propagated into `sigma`, which both engines previously omitted. * Bragg integration: Profile-fit `sigma` is **no longer inflated on weak reflections** - the fit weights take the signal estimate as it is instead of clamping it at zero, and the per-pixel variance floor is 0.01 counts instead of 1/12. diff --git a/docs/HDF5.md b/docs/HDF5.md index a0ac8adc..f9f2deaf 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -302,6 +302,7 @@ mostly onto the standard | `int_sum` | photons | standard | integrated intensity (summation) | | `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) | | `background_mean` | photons | standard | mean background under the peak | +| `background_variance` | photons² | non-standard | non-signal part of σ², carried to the merge. Absent in files written before it existed; the reader then recovers it from σ² − I | | `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` | | `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) | | `observed_frame` | | standard | image number of the reflection | diff --git a/reader/HDF5MetadataSource.cpp b/reader/HDF5MetadataSource.cpp index 574dd6ff..b047b8d8 100644 --- a/reader/HDF5MetadataSource.cpp +++ b/reader/HDF5MetadataSource.cpp @@ -179,9 +179,12 @@ bool ReadReflectionsFromGroup(HDF5Object &file, if (zeta.size() > i) zeta_val = zeta[i]; - // A file written before the merge carried this cannot say what the non-signal variance was; - // 0 leaves the combine with the signal term alone, which is what it had before. - float var_bkg_val = 0.0f; + // A file written before this dataset existed has to have the non-signal variance reconstructed, + // not zeroed. The combine takes var_bkg as the authoritative non-signal term, so a zero would + // leave it the signal alone and weight a weak reflection by ~1/I instead of ~1/sigma^2 - orders + // of magnitude too high, and worst exactly where the reflection is weakest. The integrator's own + // identity sigma^2 = I + var_bkg inverts to recover what the file does not store. + float var_bkg_val = std::max(0.0f, int_err.at(i) * int_err.at(i) - int_sum.at(i)); if (var_bkg.size() > i) var_bkg_val = var_bkg[i];