bragg integration: trimmed-mean background, on by default for rotation
Build Packages / build:windows:nocuda (push) Failing after 1m40s
Build Packages / build:windows:cuda (push) Failing after 1m39s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m50s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m45s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m43s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m54s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m3s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m3s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m50s
Build Packages / build:rpm (rocky8) (push) Successful in 12m1s
Build Packages / XDS test (durin plugin) (push) Successful in 8m9s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky9) (push) Successful in 12m31s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m58s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 13m19s
Build Packages / DIALS test (push) Successful in 14m25s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m45s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m22s
Build Packages / Unit tests (push) Successful in 58m49s

The local Bragg background is the mean over the r2..r3 ring. That mean reads
high because the contaminants that survive the signal-disk mask - neighbour-
spot wings, tails, zingers - are one-sided (positive), so it over-subtracts.
Since a weak intensity is a small difference of large numbers (I = S - nS*b),
a per-pixel background bias is fractionally largest at the resolution edge,
exactly where it hurts most.

Replace the ring mean with a symmetric trimmed mean (sort the ring, drop the
lowest and highest fraction f, average the rest), controlled by a new
BraggIntegrationSettings field and the rugnux `--background-trim <f>` option
(default f=0.10; 0 restores the plain mean). Default on for monochromatic
(rotation) data; broadband (stills) keep their tuned high-side sigma-clip, so
the base engine forces the trim to 0 there. Implemented in both the CPU engine
and the GPU kernel (shared-memory bitonic sort per block, flat-mean fallback
above BKG_TRIM_MAX ring pixels); the two agree.

25-crystal rotation battery (fixed SG/cell): <I/sigma> improved on every
crystal (median +50%), ISa on 20/22, resolution-edge R_meas fell several-fold
(e.g. lyso_ref 1.0 A 108%->43%). Last-shell CC1/2 is rescued where the plain
mean had collapsed to noise (Thau_9 at ~2.0 A 3.8%->64%, ~0.5 A of resolution
regained; cytC_10 0.2%->10%) at a small cost (1-3%) in already-clean shells -
it flattens the CC1/2 fall-off rather than shifting it. Stills unchanged.
Documented in CPU_DATA_ANALYSIS.md section 9.2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-16 18:32:57 +02:00
co-authored by Claude Opus 4.8
parent 118f0833dc
commit c3e877d5ec
9 changed files with 127 additions and 5 deletions
@@ -7,6 +7,7 @@
#include <cmath>
#include <cstdint>
#include <limits>
#include <vector>
#include "../../common/CompressedImage.h"
#include "../../common/JFJochException.h"
@@ -56,6 +57,12 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
const int W = static_cast<int>(xpixel), H = static_cast<int>(ypixel);
const bool do_clip = apply_bkg_clip && mode != IntegratorMode::BoxSum;
// Symmetric trimmed-mean background fraction (BraggIntegrationSettings, rugnux --background-trim):
// replaces the r2..r3 ring MEAN with an f-trimmed mean, robust to the high-side contamination
// (neighbour wings, tails) that biases the plain mean up and makes it over-subtract weak high-angle
// reflections. Already forced to 0 for broadband/stills by the base ctor (they keep the clip below).
const double bkg_trim_frac = bkg_trim;
auto grid_idx = [this](int dx, int dy) { return (dy + R) * G + (dx + R); };
// --- Reflection mask: mark the r2 signal disk of every predicted reflection so a neighbour's
@@ -82,6 +89,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
};
std::vector<Rough> rough(npredicted);
double inv_d2_min = std::numeric_limits<double>::max(), inv_d2_max = 0.0;
std::vector<int32_t> bkg_vals; // reused per reflection for the trimmed-mean background (idea 1)
for (size_t i = 0; i < npredicted; ++i) {
const auto &r = predicted[i];
@@ -94,6 +102,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
int64_t I_sum = 0, I_sum_x = 0, I_sum_y = 0, n_inner = 0, n_inner_valid = 0;
double bkg_sum = 0.0;
int n_bkg = 0;
bkg_vals.clear();
for (int y = y0; y <= y1; ++y)
for (int x = x0; x <= x1; ++x) {
const double d2 = (x - r.predicted_x) * (x - r.predicted_x) + (y - r.predicted_y) * (y - r.predicted_y);
@@ -109,15 +118,28 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
if (refl_mask[y * W + x]) continue;
if (!valid(px)) continue;
bkg_sum += static_cast<double>(px);
if (bkg_trim_frac > 0.0) bkg_vals.push_back(px);
++n_bkg;
}
}
if (n_inner_valid == n_inner && n_bkg > 5) {
out.bkg = bkg_sum / n_bkg;
// One high-outlier sigma-clip pass on the background ring (stills-only): reject pixels
// above mean + 3*sqrt(mean) to strip a bandwidth-streaked neighbour that biases the mean.
if (do_clip) {
if (bkg_trim_frac > 0.0 && bkg_vals.size() > 5) {
// Symmetric trimmed mean over the background ring (idea 1): drop the lowest and highest
// bkg_trim_frac of the pixels, average the rest. Robust to the high-side contamination
// that biases the plain ring mean and makes it over-subtract at high resolution.
std::sort(bkg_vals.begin(), bkg_vals.end());
const size_t lo = static_cast<size_t>(bkg_vals.size() * bkg_trim_frac);
const size_t hi = bkg_vals.size() - lo;
if (hi > lo) {
double s = 0.0;
for (size_t t = lo; t < hi; ++t) s += bkg_vals[t];
out.bkg = s / static_cast<double>(hi - lo);
}
} else if (do_clip) {
// One high-outlier sigma-clip pass on the background ring (stills-only): reject pixels
// above mean + 3*sqrt(mean) to strip a bandwidth-streaked neighbour that biases the mean.
const double thr = out.bkg + 3.0 * std::sqrt(std::max(out.bkg, 1.0));
double s = 0.0;
int n = 0;