ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization

The shadow finder divided the background by the Kahn factor computed from the experiment's
polarization degree, plane and detector distance before comparing each pixel with its ring. On a
stills run none of the three is trustworthy at the pre-scan: the degree is not identifiable from
the data, the plane can lie along the image's y axis rather than lab x (a detector mounted turned),
and the distance is still the file's placeholder, so 2 theta is overestimated. On such a run the
division made the ring anisotropy worse instead of removing it: a residual cos 2phi of 0.8 at high
angle, dim lobes over the detector corners sitting on the PENUMBRA_RATIO threshold, and the
transmitting-hardware search turned them into false pieces of 17-21k pixels whose extent followed
the third decimal of p (22973 / 25102 / 26190 pixels masked at p = 0.99 / 0.995 / 0.998, of which
only 5248 were the actual stop).

A polarized background is exactly a second harmonic in azimuth whatever the degree, plane and
2 theta, so it is now measured: the pooled background relative to its ring median, the medians of
24 sectors per 64 px band fitted with 1 + c cos 2phi + s sin 2phi (HarmonicFit, dim sectors left
out), band values interpolated in radius, a band that cannot be fitted taking the nearest fitted
one. The background is divided by that before the ring baseline is taken; the Poisson test still
puts the factor back. The finder no longer reads the polarization at all, on the host and on the
device. The arm search's own capped harmonic, which patched the old residual, is gone: the ratio it
works on is flat now.

On the stills set that showed it, the mask is the 5248-pixel stop at every p and at either plane,
and the merge statistics are the same at p = 0.99, 0.995 and 0.998 except for continuous changes
from the integration's own polarization correction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu

tests/ShadowFinderTest.cpp

@@ -329,11 +329,13 @@ TEST_CASE("ShadowFinder_TheRingsFollowTheCentreTheCallerNames", "[ShadowFinder]"
     CHECK(at_measured[I((PATCH_X0 + PATCH_X1) / 2, (PATCH_Y0 + PATCH_Y1) / 2)] == 1);
 
     // About the true centre the rings are flat and only the patch and its penumbra come back;
-    // about the file's they cut across the fall-off and a large part of the detector does.
+    // about the file's they cut across the fall-off and more of the detector does. (Less than it
+    // used to: the part of the fall-off that is a second harmonic about the wrong centre is taken out
+    // with the background's azimuthal modulation.)
     const auto masked_at_file = std::count(at_file.begin(), at_file.end(), 1u);
     const auto masked_at_measured = std::count(at_measured.begin(), at_measured.end(), 1u);
     CHECK(masked_at_measured < 2 * PATCH_PX);
-    CHECK(masked_at_file > 3 * PATCH_PX);
+    CHECK(masked_at_file > masked_at_measured);
 }
 
 // A detector on a swung 2theta arm. The beam no longer lands on the geometry's centre (the point of
@@ -377,8 +379,7 @@ TEST_CASE("ShadowFinder_ATiltedDetectorIsNotAShadow", "[ShadowFinder]") {
 // several times the dip this class looks for, so with the modulation left in, half of every outer
 // ring reads as shadow with nothing in the beam. The scene here is exactly that and nothing else -
 // a background carrying the Kahn factor of a horizontally polarized source, no hardware anywhere -
-// and it must come back empty when the experiment states its polarization, and does not when it
-// says nothing, which is what the correction is for.
+// and it must come back empty.
 TEST_CASE("ShadowFinder_APolarizedBackgroundIsNotAShadow", "[ShadowFinder]") {
     // Short enough that this small detector reaches 2 theta = 70 degrees in its corner, where
     // the source suppresses its own plane to a fifth. A real geometry gets there with a short
@@ -410,7 +411,11 @@ TEST_CASE("ShadowFinder_APolarizedBackgroundIsNotAShadow", "[ShadowFinder]") {
             frame[I(x, y)] = static_cast<int32_t>(std::lround(LEVEL * factor / at_peak));
         }
 
-    auto mask_for = [&](const std::optional<float> &polarization) {
+    // The modulation is measured from the background, so what the experiment states about the source
+    // - the beam's own value, another degree, none at all, or the plane turned by 90 degrees - is not
+    // read and cannot turn the lobes into a shadow.
+    for (const std::optional<float> polarization : {std::optional<float>(POLARIZATION), std::optional<float>(0.9f),
+                                                    std::optional<float>(), std::optional<float>(-POLARIZATION)}) {
         DiffractionExperiment x = TestExperiment();
         x.DetectorDistance_mm(DISTANCE_MM).PolarizationFactor(polarization);
         const PixelMask pixel_mask(x);
@@ -422,20 +427,65 @@ TEST_CASE("ShadowFinder_APolarizedBackgroundIsNotAShadow", "[ShadowFinder]") {
             finder.AddImage(msg, buffer);
         }
         const auto mask = finder.GetMask();
-        return std::count(mask.begin(), mask.end(), 1u);
-    };
-
-    CHECK(mask_for(POLARIZATION) == 0);
-    CHECK(mask_for(std::nullopt) > static_cast<long>(W) * H / 20);
+        CHECK(std::count_if(mask.begin(), mask.end(), [](uint32_t v) { return v != 0; }) == 0);
+    }
 }
 
-// The other side of that correction: a polarization factor that is not the beam's. Here the
-// background is unpolarized and the experiment states 0.99, so after the division every outer ring is
-// bright along x and dim along y - a smooth lobe below PENUMBRA_RATIO across the top and bottom of the
-// detector with nothing in the beam. On its own the lobe is never deep, but a small absorbing speck
-// inside it is, and the transmitting-hardware step joined the two into one piece "dim everywhere and
-// deep in places" and masked the whole lobe. The step compares a pixel with its ring's own second
-// harmonic, so the lobe is explained away and the speck, far below a shadow's size, stays unmasked.
+// A beam stop with four arms on a polarized background at a short distance: the stop has to come back,
+// two of its arms inside the background's dim lobes, the lobes themselves must not, and the mask must not move when the experiment states another
+// degree of polarization or none. The degree cannot be told from the data, so a mask that followed
+// it would follow a guess.
+TEST_CASE("ShadowFinder_TheMaskDoesNotDependOnTheStatedPolarization", "[ShadowFinder]") {
+    constexpr float DISTANCE_MM = 8.0f;
+    constexpr double TRUE_POLARIZATION = -0.998;   // the plane along the image's y
+    constexpr int LEVEL = 24;
+
+    const float pixel_mm = TestExperiment().GetPixelSize_mm();
+    const auto stop = Scene(/*cross=*/true, false);
+    std::vector<int32_t> frame(static_cast<size_t>(W) * H);
+    for (int y = 0; y < H; y++)
+        for (int xi = 0; xi < W; xi++) {
+            const double dx = xi - C, dy = y - C;
+            const double rho = std::hypot(dx, dy) * pixel_mm;
+            const double cos_2theta_sq = DISTANCE_MM * DISTANCE_MM / (DISTANCE_MM * DISTANCE_MM + rho * rho);
+            const double rr = dx * dx + dy * dy;
+            const double cos_2phi = rr > 0.0 ? (dx * dx - dy * dy) / rr : 0.0;
+            const double factor = 0.5 * (1.0 + cos_2theta_sq - TRUE_POLARIZATION * cos_2phi * (1.0 - cos_2theta_sq));
+            frame[I(xi, y)] = stop[I(xi, y)] == 0 ? 0 : static_cast<int32_t>(std::lround(LEVEL * factor));
+        }
+
+    auto mask_for = [&](const std::optional<float> &polarization) {
+        DiffractionExperiment x = TestExperiment();
+        x.DetectorDistance_mm(DISTANCE_MM).PolarizationFactor(polarization);
+        const PixelMask pixel_mask(x);
+        ShadowFinder finder(x, pixel_mask);
+        std::vector<uint8_t> buffer;
+        for (int f = 0; f < NFRAMES; f++) {
+            DataMessage msg{};
+            msg.image = CompressedImage(frame, W, H);
+            finder.AddImage(msg, buffer);
+        }
+        return finder.GetMask();
+    };
+
+    const auto mask = mask_for(0.99f);
+    CHECK(mask[I(C, C)] == ShadowFinder::SHADOW);              // the stop
+    CHECK(mask[I(W - 3, C)] == ShadowFinder::SHADOW);          // and its arms, along x
+    CHECK(mask[I(C, 3)] == ShadowFinder::SHADOW);              // and along y, inside the suppressed lobe
+    CHECK(mask[I(C + 20, 3)] == 0);                            // but not the lobe itself
+    CHECK(mask[I(C - 20, H - 4)] == 0);
+    CHECK(mask_for(0.998f) == mask);
+    CHECK(mask_for(std::nullopt) == mask);
+    CHECK(mask_for(-0.998f) == mask);
+}
+
+// An unpolarized background in an experiment that states 0.99. When the finder divided by the stated
+// factor, every outer ring came out bright along x and dim along y - a smooth lobe below PENUMBRA_RATIO
+// across the top and bottom of the detector with nothing in the beam. On its own the lobe is never
+// deep, but a small absorbing speck inside it is, and the transmitting-hardware step joined the two
+// into one piece "dim everywhere and deep in places" and masked the whole lobe. The modulation is
+// measured from the background now, so there is no lobe, and the speck, far below a shadow's size,
+// stays unmasked.
 TEST_CASE("ShadowFinder_AWrongPolarizationFactorIsNotAnArm", "[ShadowFinder]") {
     constexpr float DISTANCE_MM = 6.0f;
     constexpr int LEVEL = 24;
@@ -554,9 +604,9 @@ namespace {
 }
 
 // The mask is made on the GPU wherever the projection is there. On scenes where no pixel sits within a
-// rounding of a threshold the two must agree to the pixel: every step but the polarization factor,
-// the Poisson test's logarithm and the arm search's azimuth is exact on both, and these scenes are
-// built so that none of the three decides anything at an edge.
+// rounding of a threshold the two must agree to the pixel: every step but the ring geometry, the
+// Poisson test's logarithm and the sectors' azimuth is exact on both, and these scenes are built so
+// that none of the three decides anything at an edge.
 TEST_CASE("ShadowFinder_DeviceMaskMatchesHost", "[ShadowFinder]") {
     if (get_gpu_count() == 0)
         SKIP("no GPU");