integration: the background is fitted over the ring that survives, not averaged over it

The signal disk and the background ring are concentric, which is the whole reason a linear
background cancels between them. Within the outer ring radius of the edge of the sensor array
that concentricity is gone: the ring loses its outer part while the disk barely loses anything,
so what is left of the ring sits further into the detector, where the radial background is
higher, and the reflection reads low.

Measured at signal-free positions four pixels from a border: the ring reads 162.35 counts per
pixel against a true background over the disk of 160.57, which over a hundred disk pixels is 182
counts of deficit, against 216 to 239 observed. <I/sigma> runs -1.83, -2.34 and -1.18 at nought
to three, three to six and six to nine pixels from the border, and recovers exactly at the outer
ring radius. The same reflection measured at a border reads 179 counts lower than in the
interior over seven thousand matched pairs.

A masked module gap does the same thing but signed by the direction of the displacement, which is
why nothing has caught this: at a gap the two populations cancel in the mean, while at the sensor
border the truncation is always inward, so the bias is always negative.

The background is now the intercept of a straight line in radial offset over whatever ring pixels
survive, read at the reflection's centre. Three extra sums per ring pixel and no extra reads; the
radial distance was already computed there. It is exact under any truncation and reduces to the
mean when the ring is whole, so it is unconditional rather than a mode: a badly truncated ring
pays in sigma, through the variance the fit honestly reports, rather than in a rejection.

On the crystal where this surfaced the outermost shell's correlation with a deposited model goes
from -0.234 to +0.004, and the shell above it from -0.091 to +0.179. Correcting beats discarding:
dropping every observation within fifteen pixels of a border reached only -0.019 and +0.127,
because the corrected observations still carry signal. Interior reflections do not move.

The cost is one geometry: where a neighbour mask has already truncated the ring almost everywhere,
the fit roughly doubles the variance of the background estimate while finding no gradient worth
removing, and a crowded small detector loses one to two points of CC1/2 in its finest shells.

Also: the MINPK denominator counted only profile mass that lands on the detector, so a reflection
whose peak is off the sensor scored a perfect one and no guard could fire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N

image_analysis/bragg_integration/BraggIntegrationEngineCPU.cpp

@@ -168,6 +168,10 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
         int n_disk = 0, n_own = 0;   // pixels in the signal disk, and how many are this reflection's
         double bkg_sum = 0.0;
         int n_bkg = 0;
+        // Sums for the radial-linear background fit below, over n_fit ring pixels: sum(rad),
+        // sum(rad^2), sum(rad*px), with rad the pixel's offset along the beam->reflection direction
+        // (already computed for the stencil). The clip pass below replaces them with its own.
+        double rad_s1 = 0.0, rad_s2 = 0.0, rad_sp = 0.0;
         // Ring pixels a NEIGHBOUR's signal region occupies. refl_mask marks d.inner < r2_sq and the
         // ring is d.inner >= r2_sq, so the two are complementary and a masked ring pixel is always
         // someone else's - never this reflection's own core. That makes this an exact count of what
@@ -202,12 +206,16 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
                     if (refl_mask[y * W + x]) { ++n_bkg_neighbour; continue; }
                     if (!valid(px)) continue;
                     bkg_sum += static_cast<double>(px);
+                    rad_s1 += d.rad;
+                    rad_s2 += static_cast<double>(d.rad) * d.rad;
+                    rad_sp += static_cast<double>(d.rad) * px;
                     if (bkg_trim_frac > 0.0) bkg_vals.push_back(px);
                     ++n_bkg;
                 }
             }
 
         int n_bkg_used = n_bkg;   // pixels behind the FINAL background value (trim/clip shrink it)
+        int n_fit = n_bkg;        // pixels behind rad_s1/rad_s2/rad_sp (the clip pass shrinks them)
         // A masked, untrusted, gapped or overloaded pixel inside the signal disk used to discard the
         // reflection outright. A profile fit does not need it to: the fit is the amplitude of a
         // NORMALISED profile, so leaving pixels out renormalises the estimator by construction and
@@ -249,6 +257,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
                 const double thr = out.bkg + bkg_clip_nsigma * std::sqrt(std::max(out.bkg, 1.0));
                 double s = 0.0;
                 int n = 0;
+                double c_s1 = 0.0, c_s2 = 0.0, c_sp = 0.0;
                 for (int y = y0; y <= y1; ++y)
                     for (int x = x0; x <= x1; ++x) {
                         const auto d = BraggStencilDistances(st, x - r.predicted_x, y - r.predicted_y);
@@ -259,6 +268,9 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
                         if (static_cast<double>(px) <= thr) {
                             s += px;
                             ++n;
+                            c_s1 += d.rad;
+                            c_s2 += static_cast<double>(d.rad) * d.rad;
+                            c_sp += static_cast<double>(d.rad) * px;
                             if (bkg_radial) {
                                 // The radial curve is binned on the TRUE detector radius, so the
                                 // offset here is the unshrunk radial projection.
@@ -268,7 +280,44 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
                             }
                         }
                     }
-                if (n > 5) { out.bkg = s / n; n_bkg_used = n; }
+                if (n > 5) {
+                    out.bkg = s / n;
+                    n_bkg_used = n;
+                    bkg_sum = s;
+                    rad_s1 = c_s1; rad_s2 = c_s2; rad_sp = c_sp; n_fit = n;
+                }
+            }
+            // --- Fit the background rather than average it. The ring mean is the background AT THE
+            //     RING'S CENTROID, and that is the value under the disk only while the ring is
+            //     whole: the two are concentric, which is the whole reason a background linear in
+            //     position cancels between them. Within r3 of the array edge (or of a large mask)
+            //     the ring loses its outer part while the much smaller disk keeps nearly all of its
+            //     own, so the surviving ring sits off-centre along the radius, where the background
+            //     is different - and the mean carries that difference into every one of the ~n_inner
+            //     pixels it is subtracted from. Measured at a sensor border on a 164 counts/px
+            //     background: +1.8 counts/px over a 100 px disk, -180 counts on the intensity, and
+            //     always the same sign, because the truncation is always inward.
+            //     A straight line in the radial offset, evaluated at rad = 0 (the reflection centre),
+            //     is exact for a linear background under ANY truncation. It reduces to the mean when
+            //     the ring is whole (the centroid is then the centre), needs no threshold and rejects
+            //     nothing. The curvature that survives even an untruncated ring is a separate,
+            //     smaller term and is what --background-radial corrects; a linear fit is in that
+            //     kernel's null space, so the two do not overlap.
+            const double rad_mean = rad_s1 / n_fit;
+            const double Sxx = rad_s2 - rad_s1 * rad_mean;
+            const double Sxy = rad_sp - rad_mean * bkg_sum;
+            // The intercept is estimated from the ring pixels' spread in radius, so its variance is
+            // the mean's inflated by how far the ring's centroid had to be extrapolated back to the
+            // centre. A whole ring has rad_mean ~ 0 and gets bkg/n_bkg_used, as before; a badly
+            // truncated one pays for the extrapolation in sigma instead of in a rejection.
+            // The Poisson scale of that variance stays the ring MEAN, which is an average of counts
+            // and cannot be negative. The fitted intercept can be, on a background of well under one
+            // count per pixel, and a negative variance is not a variance - it reaches sigma as a NaN.
+            const double bkg_mean = out.bkg;
+            double bkg_var_factor = 1.0 / n_bkg_used;
+            if (Sxx > 0.0) {
+                out.bkg -= (Sxy / Sxx) * rad_mean;
+                bkg_var_factor += rad_mean * rad_mean / Sxx;
             }
             // The sum is over the pixels actually READ, so that is the count the background is
             // subtracted with; with nothing missing it is the whole disk, exactly as before.
@@ -277,8 +326,8 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
             // error enters n_inner times over: var(I) = I_sum + n_inner^2 * bkg/n_bkg_used. Leaving
             // the second term out understates sigma by sqrt(1 + n_inner/n_bkg) - 1.109x at the
             // default r1=4/r2=6/r3=10 stencil, on every reflection of every dataset.
-            out.bkg_var = out.bkg / n_bkg_used;
-            out.var_bkg = static_cast<double>(n_inner_valid) * out.bkg
+            out.bkg_var = bkg_mean * bkg_var_factor;
+            out.var_bkg = static_cast<double>(n_inner_valid) * bkg_mean
                           + static_cast<double>(n_inner_valid) * n_inner_valid * out.bkg_var;
             out.I_sum = I_sum;
             out.n_inner = static_cast<int>(n_inner_valid);
@@ -521,10 +570,14 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
             for (int dx = -Rf; dx <= Rf; ++dx) {
                 const double Pp = (*Pvec)[(dy + Rf) * Gf + (dx + Rf)];
                 if (Pp <= 0.0) continue;
+                // Profile mass the array does not reach is mass this fit has LOST, so it belongs in
+                // the denominator. Counted after the bounds test it cancels out of p_valid / p_grid,
+                // which is then exactly 1 for a reflection whose peak is off the sensor entirely -
+                // and no threshold on a ratio that is always 1 can fire.
+                p_grid += Pp;
                 const int x = rh.cx + dx, y = rh.cy + dy;
                 if (x < 0 || y < 0 || x >= W || y >= H) continue;
                 const bool in_disk = dx * dx + dy * dy < r1_sq;
-                p_grid += Pp;
                 p_peak = std::max(p_peak, Pp);
                 if (in_disk) m_all += Pp;
                 if (!valid(img[y * W + x])) {