Reader: reconstruct the background variance a legacy file does not store

A _process.h5 written before background_variance existed was read with var_bkg = 0, on the reasoning
that zero leaves the combine with the signal term alone, "which is what it had before". It does not.
Before, the combine back-derived the non-signal variance from sigma itself, and on a weak reflection
that is essentially the whole of sigma^2; zero deletes the dominant term and weights the reflection by
roughly 1/I instead of 1/sigma^2.

Recover it from the integrator's own identity, sigma^2 = I + var_bkg, when the dataset is absent.

Measured by re-scaling a stills _process.h5 with the dataset deleted, against the same file with it
intact: the automatic resolution cutoff was reading 1.66 A where the intact file reads 1.81, with
14731 unique reflections against 11508 - i.e. the zeroed file looked good enough to merge 0.15 A past
its own limit. Reconstructed, it reads 1.78 A and 11926, within 0.03 A of the intact file. The
residue is the profile-fit path, where var_bkg is not exactly sigma^2 - I and only the box-sum
identity is exact; that is recoverable to a closer approximation only by storing it, which is what
files written from now on do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/CHANGELOG.md

@@ -12,6 +12,10 @@ This is an UNSTABLE release. It includes many experimental features, as well as
 * Resolution limits: Bragg integration, azimuthal integration and spot finding all default to **as far as the detector reaches**; `--integration-high-resolution` and `--spot-high-resolution` still set one by hand, and 0 means "no limit" at either end.
 * Bragg prediction: How far the predictor walks the lattice is a setting (`bragg_integration_settings.max_hkl`, `--max-hkl`) instead of a fixed 100, derived per crystal offline and held fixed online.
 * Rotation data: a set X-ray bandwidth (`--bandwidth`) now widens each reflection's rocking curve by Δλ/λ·tan(θ) in prediction and partiality, and is deconvolved out of the fitted mosaicity; monochromatic data are unaffected.
+* Bragg integration: `background_variance` is now written to the CBOR reflection stream, so files written by the broker carry it as well as those written by rugnux.
+* Scaling: a `_process.h5` written before `background_variance` existed now has it reconstructed on read instead of taken as zero, which had let such a file merge past its true resolution limit.
+* Space-group search: the operator-agreement bound and the resolution cut that feeds the search were both loosened to match measured data, so a genuine symmetry is no longer refused on weak or lopsided sweeps.
+* Scaling: the detector modulation surface is refined on a 24×24 grid instead of 16×16.
 * Bragg integration: The local background ring is made robust with a **high-side sigma clip** (`--background-clip <n>`, default 4) instead of the symmetric trimmed mean; the trim stays reachable with `--background-trim <f>`.
 * Bragg integration: The **uncertainty of the background estimate** is now propagated into `sigma`, which both engines previously omitted.
 * Bragg integration: Profile-fit `sigma` is **no longer inflated on weak reflections** - the fit weights take the signal estimate as it is instead of clamping it at zero, and the per-pixel variance floor is 0.01 counts instead of 1/12.

docs/HDF5.md

@@ -302,6 +302,7 @@ mostly onto the standard
 | `int_sum` | photons | standard | integrated intensity (summation) |
 | `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) |
 | `background_mean` | photons | standard | mean background under the peak |
+| `background_variance` | photons² | non-standard | non-signal part of σ², carried to the merge. Absent in files written before it existed; the reader then recovers it from σ² − I |
 | `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` |
 | `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) |
 | `observed_frame` | | standard | image number of the reflection |

reader/HDF5MetadataSource.cpp

@@ -179,9 +179,12 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
         if (zeta.size() > i)
             zeta_val = zeta[i];
 
-        // A file written before the merge carried this cannot say what the non-signal variance was;
-        // 0 leaves the combine with the signal term alone, which is what it had before.
-        float var_bkg_val = 0.0f;
+        // A file written before this dataset existed has to have the non-signal variance reconstructed,
+        // not zeroed. The combine takes var_bkg as the authoritative non-signal term, so a zero would
+        // leave it the signal alone and weight a weak reflection by ~1/I instead of ~1/sigma^2 - orders
+        // of magnitude too high, and worst exactly where the reflection is weakest. The integrator's own
+        // identity sigma^2 = I + var_bkg inverts to recover what the file does not store.
+        float var_bkg_val = std::max(0.0f, int_err.at(i) * int_err.at(i) - int_sum.at(i));
         if (var_bkg.size() > i)
             var_bkg_val = var_bkg[i];