Bragg integration: clip the background ring high side instead of trimming it

The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A
symmetric trim is not a consistent estimator of the mean of a right-skewed
(Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true
mean at every level, and with ~50 signal pixels in the r1 disk that
under-subtraction adds ~5 counts to every partial on every frame. Measured two
independent ways on four rotation datasets - stored background_mean against a
plain ring mean over the same pixels on reflection-free frames, and directly on
apertures that provably hold no reflection. Empty-aperture pedestal, counts:
plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip
+0.02..+0.54.

Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for
monochromatic data. It rejects the same one-sided contamination the trim was
there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim
by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds
~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean
ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The
trim stays reachable with --background-trim for back compatibility; setting
either estimator clears the other, so they can never stack. --integrator boxsum
does not take the clip (matching what the shipped clip already did), so it now
uses the plain ring mean unless --background-trim is given.

The intensities get measurably more accurate: per-shell agreement with an
independent processing of the same images improves on 14 of 16 crystals
(weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR
- absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and
CC1/2 in the outer shell improves on 14/7.

EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by
information-free counts, so both get worse when the bias is removed; neither is
evidence against this change. That fingerprint is exactly how the trimmed mean
was accepted in the first place.

Known cost: over the 37-crystal rotation battery the de-novo space-group count
goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal
whose merge fails the absolute-sanity gate under either background (R_meas
63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the
very operator being scored, so its operator CC is diluted by construction. No
other crystal changes space group, and twin protection is not weakened - the
H-ratio veto that refuses genuinely twinned crystals gets MORE decisive
(1.63 -> 1.84, 2.83 -> 3.99).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-05 18:03:11 +02:00
co-authored by Claude Opus 5
parent 9549caf82b
commit 09fb8e0306
13 changed files with 109 additions and 43 deletions
+13 -3
View File
@@ -78,7 +78,10 @@ Scene BuildScene(size_t width, size_t height, int spacing = 60) {
return s;
}
// clip_nsigma 0 selects the OTHER background-ring estimator, the symmetric trim, so the two branches
// the CPU and GPU each implement separately are both covered.
DiffractionExperiment MakeExperiment(IntegratorMode mode, std::optional<float> bandwidth_fwhm,
float clip_nsigma = 4.0f,
const DetectorSetup &det = DetJF(2)) {
DiffractionExperiment experiment(det); // DetJF(2) (small) keeps the correctness test fast
experiment.DetectorDistance_mm(100.0f).IncidentEnergy_keV(WVL_1A_IN_KEV)
@@ -86,12 +89,17 @@ DiffractionExperiment MakeExperiment(IntegratorMode mode, std::optional<float> b
experiment.BandwidthFWHM(bandwidth_fwhm);
BraggIntegrationSettings settings;
settings.Integrator(mode);
if (clip_nsigma > 0.0f)
settings.BackgroundClipNSigma(clip_nsigma);
else
settings.BackgroundTrimFraction(0.10f);
experiment.ImportBraggIntegrationSettings(settings);
return experiment;
}
void CompareCpuVsGpu(IntegratorMode mode, std::optional<float> bandwidth_fwhm) {
const DiffractionExperiment experiment = MakeExperiment(mode, bandwidth_fwhm);
void CompareCpuVsGpu(IntegratorMode mode, std::optional<float> bandwidth_fwhm,
float clip_nsigma = 4.0f) {
const DiffractionExperiment experiment = MakeExperiment(mode, bandwidth_fwhm, clip_nsigma);
const size_t width = experiment.GetXPixelsNum();
const size_t height = experiment.GetYPixelsNum();
const size_t npixel = experiment.GetPixelsNum();
@@ -145,6 +153,7 @@ TEST_CASE("BraggIntegrationEngineGPU_MatchesCPU") {
SECTION("ProfileGaussian mono") { CompareCpuVsGpu(IntegratorMode::ProfileGaussian, std::nullopt); }
SECTION("ProfileGaussian broadband") { CompareCpuVsGpu(IntegratorMode::ProfileGaussian, 0.03f); }
SECTION("ProfileEmpirical") { CompareCpuVsGpu(IntegratorMode::ProfileEmpirical, std::nullopt); }
SECTION("ProfileGaussian mono trim") { CompareCpuVsGpu(IntegratorMode::ProfileGaussian, std::nullopt, 0.0f); }
}
// Hidden ([.]) benchmark: the raison d'etre of the GPU port is < 2 ms/frame (vs ~142 ms on the CPU
@@ -154,7 +163,8 @@ TEST_CASE("BraggIntegrationEngineGPU_Benchmark", "[.][bragg_bench]") {
WARN("No CUDA GPU present. Skipping benchmark");
return;
}
const DiffractionExperiment experiment = MakeExperiment(IntegratorMode::ProfileGaussian, std::nullopt, DetJF4M());
const DiffractionExperiment experiment = MakeExperiment(IntegratorMode::ProfileGaussian, std::nullopt,
4.0f, DetJF4M());
const size_t width = experiment.GetXPixelsNum();
const size_t height = experiment.GetYPixelsNum();
const size_t npixel = experiment.GetPixelsNum();