diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 25a6427e..39a51533 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -12,6 +12,10 @@ This is an UNSTABLE release. It includes many experimental features, as well as * Resolution limits: Bragg integration, azimuthal integration and spot finding all default to **as far as the detector reaches**; `--integration-high-resolution` and `--spot-high-resolution` still set one by hand, and 0 means "no limit" at either end. * Bragg prediction: How far the predictor walks the lattice is a setting (`bragg_integration_settings.max_hkl`, `--max-hkl`) instead of a fixed 100, derived per crystal offline and held fixed online. * Rotation data: a set X-ray bandwidth (`--bandwidth`) now widens each reflection's rocking curve by Δλ/λ·tan(θ) in prediction and partiality, and is deconvolved out of the fitted mosaicity; monochromatic data are unaffected. +* Bragg integration: `background_variance` is now written to the CBOR reflection stream, so files written by the broker carry it as well as those written by rugnux. +* Scaling: a `_process.h5` written before `background_variance` existed now has it reconstructed on read instead of taken as zero, which had let such a file merge past its true resolution limit. +* Space-group search: the operator-agreement bound and the resolution cut that feeds the search were both loosened to match measured data, so a genuine symmetry is no longer refused on weak or lopsided sweeps. +* Scaling: the detector modulation surface is refined on a 24×24 grid instead of 16×16. * Bragg integration: The local background ring is made robust with a **high-side sigma clip** (`--background-clip `, default 4) instead of the symmetric trimmed mean; the trim stays reachable with `--background-trim `. * Bragg integration: The **uncertainty of the background estimate** is now propagated into `sigma`, which both engines previously omitted. * Bragg integration: Profile-fit `sigma` is **no longer inflated on weak reflections** - the fit weights take the signal estimate as it is instead of clamping it at zero, and the per-pixel variance floor is 0.01 counts instead of 1/12. diff --git a/docs/HDF5.md b/docs/HDF5.md index a0ac8adc..f9f2deaf 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -302,6 +302,7 @@ mostly onto the standard | `int_sum` | photons | standard | integrated intensity (summation) | | `int_err` | photons | non-standard name | σ of the intensity (standard equivalent: `int_sum_errors`) | | `background_mean` | photons | standard | mean background under the peak | +| `background_variance` | photons² | non-standard | non-signal part of σ², carried to the merge. Absent in files written before it existed; the reader then recovers it from σ² − I | | `predicted_x`, `predicted_y` | pixel | name standard, units differ | predicted position. NXreflections `predicted_x/_y` are *physical* lengths; the pixel datasets are `predicted_px_x/_y` | | `observed_x`, `observed_y` | pixel | name standard, units differ | observed centroid (pixels; standard pixel form is `observed_px_x/_y`) | | `observed_frame` | | standard | image number of the reflection | diff --git a/reader/HDF5MetadataSource.cpp b/reader/HDF5MetadataSource.cpp index 574dd6ff..b047b8d8 100644 --- a/reader/HDF5MetadataSource.cpp +++ b/reader/HDF5MetadataSource.cpp @@ -179,9 +179,12 @@ bool ReadReflectionsFromGroup(HDF5Object &file, if (zeta.size() > i) zeta_val = zeta[i]; - // A file written before the merge carried this cannot say what the non-signal variance was; - // 0 leaves the combine with the signal term alone, which is what it had before. - float var_bkg_val = 0.0f; + // A file written before this dataset existed has to have the non-signal variance reconstructed, + // not zeroed. The combine takes var_bkg as the authoritative non-signal term, so a zero would + // leave it the signal alone and weight a weak reflection by ~1/I instead of ~1/sigma^2 - orders + // of magnitude too high, and worst exactly where the reflection is weakest. The integrator's own + // identity sigma^2 = I + var_bkg inverts to recover what the file does not store. + float var_bkg_val = std::max(0.0f, int_err.at(i) * int_err.at(i) - int_sum.at(i)); if (var_bkg.size() > i) var_bkg_val = var_bkg[i];