Bragg integration: clip the background ring high side instead of trimming it

The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A
symmetric trim is not a consistent estimator of the mean of a right-skewed
(Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true
mean at every level, and with ~50 signal pixels in the r1 disk that
under-subtraction adds ~5 counts to every partial on every frame. Measured two
independent ways on four rotation datasets - stored background_mean against a
plain ring mean over the same pixels on reflection-free frames, and directly on
apertures that provably hold no reflection. Empty-aperture pedestal, counts:
plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip
+0.02..+0.54.

Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for
monochromatic data. It rejects the same one-sided contamination the trim was
there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim
by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds
~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean
ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The
trim stays reachable with --background-trim for back compatibility; setting
either estimator clears the other, so they can never stack. --integrator boxsum
does not take the clip (matching what the shipped clip already did), so it now
uses the plain ring mean unless --background-trim is given.

The intensities get measurably more accurate: per-shell agreement with an
independent processing of the same images improves on 14 of 16 crystals
(weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR
- absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and
CC1/2 in the outer shell improves on 14/7.

EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by
information-free counts, so both get worse when the bias is removed; neither is
evidence against this change. That fingerprint is exactly how the trimmed mean
was accepted in the first place.

Known cost: over the 37-crystal rotation battery the de-novo space-group count
goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal
whose merge fails the absolute-sanity gate under either background (R_meas
63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the
very operator being scored, so its operator CC is diluted by construction. No
other crystal changes space group, and twin protection is not weakened - the
H-ratio veto that refuses genuinely twinned crystals gets MORE decisive
(1.63 -> 1.84, 2.83 -> 3.99).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-05 18:03:11 +02:00
co-authored by Claude Opus 5
parent 9549caf82b
commit 09fb8e0306
13 changed files with 109 additions and 43 deletions
@@ -55,12 +55,13 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
return Finalize(predicted, npredicted, results, image_number);
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
const bool do_clip = apply_bkg_clip && mode != IntegratorMode::BoxSum;
const bool do_clip = bkg_clip_nsigma > 0.0f && mode != IntegratorMode::BoxSum;
// Symmetric trimmed-mean background fraction (BraggIntegrationSettings, rugnux --background-trim):
// replaces the r2..r3 ring MEAN with an f-trimmed mean, robust to the high-side contamination
// (neighbour wings, tails) that biases the plain mean up and makes it over-subtract weak high-angle
// reflections. Already forced to 0 for broadband/stills by the base ctor (they keep the clip below).
// reflections. The base ctor has already forced it to 0 whenever the clip below is in force, which
// is the default - the two are alternatives.
const double bkg_trim_frac = bkg_trim;
auto grid_idx = [this](int dx, int dy) { return (dy + R) * G + (dx + R); };
@@ -138,9 +139,9 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
out.bkg = s / static_cast<double>(hi - lo);
}
} else if (do_clip) {
// One high-outlier sigma-clip pass on the background ring (stills-only): reject pixels
// above mean + 3*sqrt(mean) to strip a bandwidth-streaked neighbour that biases the mean.
const double thr = out.bkg + 3.0 * std::sqrt(std::max(out.bkg, 1.0));
// One high-outlier sigma-clip pass on the background ring: reject pixels above
// mean + n*sqrt(mean) to strip a neighbour core or a zinger that biases the mean.
const double thr = out.bkg + bkg_clip_nsigma * std::sqrt(std::max(out.bkg, 1.0));
double s = 0.0;
int n = 0;
for (int y = y0; y <= y1; ++y)