Files
Jungfraujoch/common/BraggIntegrationSettings.h
T
leonarski_fandClaude Opus 5 09fb8e0306 Bragg integration: clip the background ring high side instead of trimming it
The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A
symmetric trim is not a consistent estimator of the mean of a right-skewed
(Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true
mean at every level, and with ~50 signal pixels in the r1 disk that
under-subtraction adds ~5 counts to every partial on every frame. Measured two
independent ways on four rotation datasets - stored background_mean against a
plain ring mean over the same pixels on reflection-free frames, and directly on
apertures that provably hold no reflection. Empty-aperture pedestal, counts:
plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip
+0.02..+0.54.

Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for
monochromatic data. It rejects the same one-sided contamination the trim was
there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim
by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds
~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean
ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The
trim stays reachable with --background-trim for back compatibility; setting
either estimator clears the other, so they can never stack. --integrator boxsum
does not take the clip (matching what the shipped clip already did), so it now
uses the plain ring mean unless --background-trim is given.

The intensities get measurably more accurate: per-shell agreement with an
independent processing of the same images improves on 14 of 16 crystals
(weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR
- absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and
CC1/2 in the outer shell improves on 14/7.

EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by
information-free counts, so both get worse when the bias is removed; neither is
evidence against this change. That fingerprint is exactly how the trimmed mean
was accepted in the first place.

Known cost: over the 37-crystal rotation battery the de-novo space-group count
goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal
whose merge fails the absolute-sanity gate under either background (R_meas
63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the
very operator being scored, so its operator CC is diluted by construction. No
other crystal changes space group, and twin protection is not weakened - the
H-ratio veto that refuses genuinely twinned crystals gets MORE decisive
(1.63 -> 1.84, 2.83 -> 3.99).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 18:03:11 +02:00

85 lines
5.0 KiB
C++

// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#pragma once
#include <optional>
// Spot-intensity extraction method used by the Bragg integration engine. ProfileGaussian (default)
// profile-fits with a measured-width Gaussian (Kabsch-style) - more accurate intensities than the
// classical uniform BoxSum; validated on anomalous data (stronger S/Cl peaks vs box-sum). BoxSum is
// the simpler, faster fallback. ProfileEmpirical learns the profile per resolution shell from strong
// spots - see docs/CPU_DATA_ANALYSIS.md (Bragg integration).
enum class IntegratorMode { BoxSum, ProfileGaussian, ProfileEmpirical };
// The hkl half-width the broker bootstraps when a config carries no bragg_integration block. Matches
// the max_hkl default in broker/jfjoch_api.yaml, so an omitting client and an omitting config agree.
constexpr int BRAGG_ONLINE_DEFAULT_MAX_HKL = 100;
class BraggIntegrationSettings {
IntegratorMode integrator_mode = IntegratorMode::ProfileGaussian;
float r_1 = 4;
float r_2 = 6;
float r_3 = 10;
// Integration/prediction resolution limit. Unset means "as far as the detector reaches", resolved
// from the geometry where it is used. The predictor independently rejects any reflection that misses
// the detector, so this is a bound on how far the lattice walk goes rather than a second opinion on
// what is measurable - a fixed default simply truncated every experiment whose detector reached
// past it.
std::optional<float> d_min_limit_A;
std::optional<float> fixed_profile_radius;
float minimum_sigma_in_regards_to_i = 0.02;
// The r2..r3 background ring is estimated with ONE of two robust means, never both: a high-side
// sigma-clip (bkg_clip_nsigma, the default) or a symmetric trimmed mean (bkg_trim_fraction). Setting
// either through its setter clears the other, so whichever was asked for last is the one in force;
// with both at 0 the ring is a plain mean.
//
// Symmetric trimmed-mean fraction: drop the lowest and highest this fraction of ring pixels before
// averaging. Robust to the high-side contamination (neighbour-spot wings, tails, zingers) that
// biases a plain ring mean up, but a symmetric trim is NOT a consistent estimator of the mean of a
// right-skewed (Poisson) sample - it sits ~0.1 ct/px low at every level, which with ~50 ring pixels
// adds ~5 counts to every partial. Kept reachable (rugnux --background-trim) for back compatibility;
// 0.10 was the shipped value.
float bkg_trim_fraction = 0.0f;
// High-side-only sigma clip: reject ring pixels above mean + this many sqrt(mean). Rejects the same
// contamination as the trim - measurably better, in fact - without cutting the low side, so it does
// not carry the trim's skew bias. Measured empty-aperture pedestal, counts: plain mean -0.03..-0.20,
// 10% symmetric trim +5.05..+6.34, 4 sigma clip +0.02..+0.54. Applied to monochromatic data
// (rugnux --background-clip); broadband (non-zero bandwidth) data always clip, at their tuned 3 sigma.
float bkg_clip_nsigma = 4.0f;
// Half-width of the hkl cube the predictor walks: every reflection with |h|,|k|,|l| <= this is
// tested against the Ewald sphere, and nothing outside it can ever be predicted. An axis is
// truncated once a/d_min exceeds this, and the GPU cost is the cube (2n+1)^3 of candidates, so
// neither a small nor a large fixed value is right for every crystal.
//
// Unset (the default) means "take it from the refined cell", which is exact: the predictor keeps
// only |q| <= 1/d_min and h = a.q, so no reflection can have |h| > a/d_min. See MaxHKLForCell.
// Offline that is what is wanted. ONLINE it is not: the broker bootstraps a concrete value
// (BRAGG_ONLINE_DEFAULT_MAX_HKL) so per-image cost stays predictable across samples.
std::optional<int> max_hkl;
public:
BraggIntegrationSettings& R1(float input);
BraggIntegrationSettings& R2(float input);
BraggIntegrationSettings& R3(float input);
BraggIntegrationSettings& DMinLimit_A(std::optional<float> input);
BraggIntegrationSettings& FixedProfileRadius_recipA(std::optional<float> input);
BraggIntegrationSettings& Integrator(IntegratorMode input);
BraggIntegrationSettings& BackgroundTrimFraction(float input);
BraggIntegrationSettings& BackgroundClipNSigma(float input);
BraggIntegrationSettings& MaxHKL(std::optional<int> input);
[[nodiscard]] IntegratorMode GetIntegrator() const;
[[nodiscard]] float GetR1() const;
[[nodiscard]] float GetR2() const;
[[nodiscard]] float GetR3() const;
[[nodiscard]] std::optional<float> GetFixedProfileRadius_recipA() const;
[[nodiscard]] std::optional<float> GetDMinLimit_A() const;
[[nodiscard]] float GetMinimumSigmaInRegardsToI() const;
[[nodiscard]] float GetBackgroundTrimFraction() const;
[[nodiscard]] float GetBackgroundClipNSigma() const;
[[nodiscard]] std::optional<int> GetMaxHKL() const;
};