diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index ff16e348e..7ab53f40d 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -105,7 +105,9 @@ cell" (1982), J. Appl. Cryst. 15, 255-259 **[XDS](https://xds.mr.mpg.de/)** — rotation geometry and notation, the reciprocal Lorentz and partiality treatment, the maximum-likelihood mosaicity estimate, the `MINPK` criterion for rejecting a reflection whose predicted profile is not cleanly its own, the intensity-based test for a -centred lattice, and the scaling correction surfaces indexed by image number and detector region. W. Kabsch, "XDS" (2010), Acta Cryst. D66, 125-132 +centred lattice, the recognition of shaded detector regions by comparing a pixel's background +against the background at its own resolution (`DEFPIX`), and the scaling correction surfaces +indexed by image number and detector region. W. Kabsch, "XDS" (2010), Acta Cryst. D66, 125-132 [doi:10.1107/S0907444909047337](https://doi.org/10.1107/S0907444909047337); W. Kabsch, "Integration, scaling, space-group assignment and post-refinement" (2010), Acta Cryst. D66, 133-144 [doi:10.1107/S0907444909047374](https://doi.org/10.1107/S0907444909047374). diff --git a/image_analysis/beam_stop/SHADOW_FINDER.md b/image_analysis/beam_stop/SHADOW_FINDER.md index 12ae4d1ec..ebcf4984c 100644 --- a/image_analysis/beam_stop/SHADOW_FINDER.md +++ b/image_analysis/beam_stop/SHADOW_FINDER.md @@ -1,9 +1,9 @@ # Beam-stop shadow finder -`ShadowFinder` detects the beam-stop shadow — the central **disk** plus its **holder arm** — -from a small number of images, and returns a per-pixel mask of the shadowed region. It exists -so the pipeline can mask the beam stop **automatically** instead of relying on a hand-drawn user -mask. +`ShadowFinder` detects shadowed detector regions — the beam stop's central **disk** and **holder +arm**, and any other hardware standing in the beam, such as a pin or a loop — from a small number +of images, and returns a per-pixel mask of the shadowed region. It exists so the pipeline can mask +them **automatically** instead of relying on a hand-drawn user mask. It is written in the accumulate-then-finalize shape of `DarkMaskAnalysis`: feed frames with `AddImage()`, then read the mask once with `GetMask()`. It is built as part of @@ -48,28 +48,38 @@ own radius**, not to a global threshold. Iterate 3×, excluding the shadow it finds, so the baseline is not pulled down by the shadow. 3. **Ratio.** `ratio = mean / baseline(r)`. Shadowed pixels sit far below 1; the smooth halo and symmetric ice rings cancel to ≈1 and are **not** flagged. -4. **Core + connectivity.** `core = ratio < shadow_ratio`, then keep only the part **connected to - the beam centre** (flood from a small central seed through a `bridge_px`-dilated copy, so the - arm crosses module gaps). The centre anchor is what makes it specific to the beam stop rather - than any random low patch. -5. **Reflection guard.** A pixel whose **max**-projection reached `min_reflection` recorded a real +4. **Counting statistics.** A ratio on its own says nothing when the background it is made of is a + handful of photons, so the deficit is also required to be significant against its own Poisson + scatter: `sigma = sqrt(2 (E - N + N ln(N/E)))` for the pooled counts `N` observed against the + `E` the ring expects, cut at `min_deficit_sigma`. This is what makes the test scale-free — a + third of the background missing is overwhelming on a well-exposed sweep and is noise on six + low-background frames, and without it such a pre-scan masks three quarters of a detector. +5. **Core + size.** `core = ratio < shadow_ratio && sigma > min_deficit_sigma`, then keep only + components holding at least `min_shadow_pixels` core pixels (components taken through a + `bridge_px`-dilated copy, so the arm crosses module gaps). **Nothing anchors the region to the + beam centre**: hardware that shadows the detector need not touch the direct beam, and a pin or a + loop typically does not — it begins some way out in radius with lit detector between it and the + stop, which an anchored search discards however deep it is. Size is what makes the test specific + instead; the background wanders a pixel or two at a time, hardware does not. +6. **Reflection guard.** A pixel whose **max**-projection reached `min_reflection` recorded a real reflection — a beam stop cannot block a reflection that was measured — so it is **never masked** (a small-cluster test drops single-pixel zingers). -6. **Central disk (low-res guard).** From the azimuthal **blocked fraction** about the beam centre, +7. **Central disk (low-res guard).** From the azimuthal **blocked fraction** about the beam centre, mask a full disk out to the radius where nearly every azimuth is blocked (a disk blocks all directions; a thin arm/gap does not). Returns nothing when the centre is not behind a disk, and is **capped just inside the innermost reflection** so a real edge spot is never swallowed. This guards the low-resolution zone the stop makes un-measurable, including a disk the raw data has already voided. -7. **Soft boundary.** Grow the core+disk into partially-shadowed pixels (`ratio < penumbra_ratio`) +8. **Soft boundary.** Grow the core+disk into partially-shadowed pixels (`ratio < penumbra_ratio`) within `penumbra_max_px`, then a morphological close rounds the contour and hole-filling closes the disk interior. -8. **Carve reflections last.** Remove any lit pixel from the final mask (no fill afterwards) so a +9. **Carve reflections last.** Remove any lit pixel from the final mask (no fill afterwards) so a spot the geometry still covered is given back rather than re-enclosed. -Three findings from the prototype are baked in: the **mean** ratio (not the max) does the -detecting — the max projection is a great *picture* but a noisier detector; the **centre anchor** -is what distinguishes the stop from stray low patches; and **a lit pixel is never masked**. +Three findings are baked in: the **mean** ratio (not the max) does the detecting — the max +projection is a great *picture* but a noisier detector; **size and counting statistics**, not a +position, are what distinguish hardware from a stray low patch; and **a lit pixel is never +masked**. Detection converges in **~2–3 frames** (the shadow cuts the halo present in every frame); 20–50 is just safe headroom. @@ -77,9 +87,11 @@ just safe headroom. | field | default | meaning | |---|---|---| -| `shadow_ratio` | 0.35 | core threshold: mean below this fraction of the radial baseline | -| `penumbra_ratio` | 0.72 | soft boundary grows out to this ratio … | -| `penumbra_max_px` | 14 | … but no further than this from the core | +| `shadow_ratio` | 0.50 | core threshold: mean below this fraction of the radial baseline | +| `min_deficit_sigma` | 6 | … and the deficit significant to this many Poisson standard deviations | +| `min_shadow_pixels` | 2000 | smallest region the per-pixel test may return | +| `penumbra_ratio` | 0.75 | soft boundary grows out to this ratio … | +| `penumbra_max_px` | 30 | … but no further than this from the core | | `bridge_px` | 6 | bridge module gaps / breaks the arm crosses | | `min_reflection` | 25 | max-projection value at/above which a pixel is a protected reflection | @@ -136,6 +148,11 @@ beam-stop mask is an **offline** (Rugnux + viewer) feature and never goes to the - **Off-centre disk**: the low-res disk guard keys on the *stored* beam centre. If that is off the physical stop (seen on one stills detector), a small already-dead disk gets no extra circular guard — a data-driven disk centre would close it. +- **A grossly wrong beam centre**: the rings are drawn about the centre in the file. A centre off by + tens of pixels costs nothing, but one off by a couple of hundred draws the rings across the + background's own fall-off rather than along it, and a large region then falls below the ratio for + reasons that have nothing to do with hardware. Measuring the centre before the shadow, rather than + after it, would close this. - **Unsigned saturation**: for unsigned images a detector-saturating reflection collides with the no-data sentinel and is skipped; harmless in practice (such reflections do not sit under the stop), and absent entirely on the signed/preprocessed path. diff --git a/image_analysis/beam_stop/ShadowFinder.cpp b/image_analysis/beam_stop/ShadowFinder.cpp index c40e484f5..d19422585 100644 --- a/image_analysis/beam_stop/ShadowFinder.cpp +++ b/image_analysis/beam_stop/ShadowFinder.cpp @@ -20,12 +20,13 @@ // A pixel is shadow when its background is below this fraction of the background it is // compared against. -constexpr float SHADOW_RATIO = 0.35f; +constexpr float SHADOW_RATIO = 0.50f; // The boundary grows outward into partially shadowed pixels down to this fraction, but no -// further than PENUMBRA_MAX_PX from the core. -constexpr float PENUMBRA_RATIO = 0.72f; -constexpr int PENUMBRA_MAX_PX = 14; +// further than PENUMBRA_MAX_PX from the core. A pin or a loop casts a wide half-shadow, so the +// reach is a good deal more than the beam stop's own edge needs. +constexpr float PENUMBRA_RATIO = 0.75f; +constexpr int PENUMBRA_MAX_PX = 30; // Bridge module gaps and small breaks that the holder arm crosses. constexpr int BRIDGE_PX = 6; @@ -34,10 +35,18 @@ constexpr int BRIDGE_PX = 6; // beam stop cannot block a reflection that was measured. constexpr int64_t MIN_REFLECTION = 25; -// Counts the background must have accumulated over the frames and the pooled pixels before -// a dip in it is believable. Below this a Poisson hole is indistinguishable from a shadow, -// and testing anyway masks whole detectors on low-background data. -constexpr double MIN_EXPECTED_COUNTS = 60; +// How far below the background it is compared against a pixel must sit before the dip is +// believed, in standard deviations of the counts that back it. The counts are photons, so their +// scatter is Poisson and the deficit is measured against it rather than against a fixed number: +// on a well-exposed sweep a third of the background missing is overwhelming, and on a handful of +// low-background frames the same third is noise. Without this a six-frame pre-scan of a +// low-background sweep masks three quarters of the detector. +constexpr double MIN_DEFICIT_SIGMA = 6.0; + +// Smallest region the per-pixel test may return. A shadow is cast by something physical and is +// correspondingly large; an isolated patch this small is the background wandering, not hardware. +// This is what keeps the test specific now that a shadow no longer has to touch the direct beam. +constexpr int MIN_SHADOW_PIXELS = 2000; // Side of the box the background is pooled over before testing. Its area is how many pixels back // a ring's countability test, which decides where an azimuthal comparison is possible at all. @@ -47,10 +56,25 @@ constexpr double MEAN_POOLED_PIXELS = POOL_PX * POOL_PX; // A ring with fewer valid pixels than this says nothing about whether it was counted. constexpr int MIN_RING_PIXELS = 32; +// A ring lies wholly inside the stop when its background is below this fraction of the background +// further out. This asks about a whole ring rather than about a pixel, so it keeps a threshold of +// its own and does not follow SHADOW_RATIO. +constexpr float BLOCKED_RING_RATIO = 0.35f; + // Binary-image helpers on a width*height frame stored row-major as char (0/1). All run once, // at GetMask() time; the BFS forms keep them O(pixels) rather than O(pixels * radius). namespace { +// Deficit of an observed count against its expectation, in standard deviations, from the Poisson +// likelihood ratio. Zero when the observation is not below the expectation. +double poisson_deficit_sigma(double observed, double expected) { + if (expected <= 0.0 || observed >= expected) + return 0.0; + const double ll = 2.0 * (expected - observed + + (observed > 0.0 ? observed * std::log(observed / expected) : 0.0)); + return ll > 0.0 ? std::sqrt(ll) : 0.0; +} + // 8-connected dilation by `r` pixels, i.e. every pixel within Chebyshev distance r of a set one. // // This was a multi-source BFS, which is what the distance is defined by - but on a full rectangle @@ -122,27 +146,6 @@ std::vector erode(const std::vector &in, int W, int H, int r, size_t return out; } -// Pixels of `passable` reachable from any of `seeds` (8-connected flood). -std::vector flood(const std::vector &passable, int W, int H, const std::vector &seeds) { - std::vector visited(passable.size(), 0); - std::queue q; - for (const int s : seeds) - if (passable[s] && !visited[s]) { visited[s] = 1; q.push(s); } - while (!q.empty()) { - const int i = q.front(); q.pop(); - const int y = i / W, x = i % W; - for (int dy = -1; dy <= 1; dy++) - for (int dx = -1; dx <= 1; dx++) { - const int yy = y + dy, xx = x + dx; - if (yy < 0 || yy >= H || xx < 0 || xx >= W) - continue; - const int j = yy * W + xx; - if (passable[j] && !visited[j]) { visited[j] = 1; q.push(j); } - } - } - return visited; -} - // Fill holes: background not reachable from the image border becomes region. // // The flood is run over the bounding box of `region` grown by one, not the whole detector. Outside @@ -572,7 +575,9 @@ std::vector ShadowFinder::GetMask(size_t nthreads) const { // Azimuthal comparison: the median of the ring, iterated so the shadow stays out of the - // baseline it is measured against. + // baseline it is measured against. Comparing a pixel's background against the background at its + // own resolution is how DEFPIX recognises a shaded detector region. + // Following Kabsch (2010) Acta Cryst. D66, 125-132 // // The iteration only ever excludes pixels whose pooled background is below a cut, and dividing by // a positive baseline is monotone - so the excluded pixels of a ring are exactly the lowest ones, @@ -636,36 +641,70 @@ std::vector ShadowFinder::GetMask(size_t nthreads) const { for (int rad = 0; rad <= max_radius; rad++) { if (ring_pixels[rad] < MIN_RING_PIXELS) continue; - if (baseline[rad] >= SHADOW_RATIO * outward_max[rad]) + if (baseline[rad] >= BLOCKED_RING_RATIO * outward_max[rad]) break; blocked_out_to = rad; } + // The counts a pixel's pooled background is made of, and the counts the ring says it should + // have had. The test is on the deficit between them, in units of its own Poisson scatter. + std::vector deficit(n_pixels, 0.0f); std::vector low(n_pixels, 0); ParallelChunks(n_pixels, nthreads, [&](int lo, int hi) { for (int i = lo; i < hi; i++) { if (!valid[i]) continue; - if (radius[i] <= blocked_out_to) { - low[i] = 1; - continue; - } const double counted = static_cast(frames) * pooled_count[i]; - low[i] = ratio[i] < SHADOW_RATIO && baseline[radius[i]] * counted >= MIN_EXPECTED_COUNTS; + deficit[i] = static_cast(poisson_deficit_sigma(pooled[i] * counted, + baseline[radius[i]] * counted)); + low[i] = ratio[i] < SHADOW_RATIO && deficit[i] > MIN_DEFICIT_SIGMA; } }); - // The shadow is the low region connected to the beam centre, bridging the gaps it crosses. + // The shadow is a low region large enough to have been cast by something. Nothing anchors it to + // the beam centre: hardware that shadows the detector need not touch the direct beam, and a pin + // or a loop typically does not - it begins some way out in radius, with lit detector between it + // and the stop. What keeps the test specific instead is size, since the background wanders by a + // pixel or two at a time and hardware does not. const std::vector bridged = dilate(low, W, H, BRIDGE_PX, nthreads); - std::vector seeds; + std::vector region(n_pixels, 0); + { + std::vector seen(n_pixels, 0); + std::vector component; + std::queue q; + for (int start = 0; start < n_pixels; start++) { + if (!bridged[start] || seen[start]) + continue; + component.clear(); + int n_low = 0; + seen[start] = 1; + q.push(start); + while (!q.empty()) { + const int i = q.front(); q.pop(); + component.push_back(i); + n_low += low[i]; + const int y = i / W, x = i % W; + for (int dy = -1; dy <= 1; dy++) + for (int dx = -1; dx <= 1; dx++) { + const int yy = y + dy, xx = x + dx; + if (yy < 0 || yy >= H || xx < 0 || xx >= W) + continue; + const int j = yy * W + xx; + if (bridged[j] && !seen[j]) { seen[j] = 1; q.push(j); } + } + } + if (n_low >= MIN_SHADOW_PIXELS) + for (const int i : component) + region[i] = low[i]; + } + } + + // The rings that lie wholly inside the stop are decided by the ring walk above rather than by + // the per-pixel test, so they join the region after it and are not asked to be large. for (int i = 0; i < n_pixels; i++) - if (radius[i] < 4) - seeds.push_back(i); - const std::vector connected = flood(bridged, W, H, seeds); - std::vector region(n_pixels); - for (int i = 0; i < n_pixels; i++) - region[i] = low[i] && connected[i]; + if (valid[i] && radius[i] <= blocked_out_to) + region[i] = 1; // Recorded reflections. A small cluster is required so a single-frame zinger does not count. @@ -695,7 +734,7 @@ std::vector ShadowFinder::GetMask(size_t nthreads) const { // Grow the soft boundary, round it and fill the disk interior. const std::vector penumbra = dilate(region, W, H, PENUMBRA_MAX_PX, nthreads); for (int i = 0; i < n_pixels; i++) - if (penumbra[i] && valid[i] && ratio[i] < PENUMBRA_RATIO) + if (penumbra[i] && valid[i] && ratio[i] < PENUMBRA_RATIO && deficit[i] > MIN_DEFICIT_SIGMA) region[i] = 1; diff --git a/rugnux/Rugnux.cpp b/rugnux/Rugnux.cpp index f5a5a3ef1..9f6fc2ef5 100644 --- a/rugnux/Rugnux.cpp +++ b/rugnux/Rugnux.cpp @@ -1158,15 +1158,17 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru const auto shadow = finder.GetMask(config_.nthreads); const auto shadowed = std::count(shadow.begin(), shadow.end(), 1u); pixel_mask_.LoadBeamStopMask(experiment_, shadow); - // Not "this beamline has no beam stop". The search floods outwards from seeds within 4 px of - // the ASSUMED beam centre, so a centre far enough out puts every seed in a module gap and the - // flood finds nothing - which makes an empty shadow a statement about the centre. It is the - // sharpest free predicate there is: over 29 non-SLS datasets it fires on 3, all 3 of which had - // a bad outcome, with no false alarm in 39 in-house sweeps (which report 0.44-1.51 % masked). + // Not "this beamline has no beam stop". Every pixel is compared against the ring it sits on, + // and the rings are drawn about the beam centre in the file: a centre far enough out draws + // them across the background's own fall-off instead of along it, and what comes back stops + // describing the hardware. That reads both ways - it has been seen as an empty mask, and on + // a deliberately displaced centre as a large spurious one - so an unexpected answer here is + // worth reading as a statement about the centre first. if (shadowed == 0) - logger.Warning("Beam stop: no shadow found in {} images. The search starts from the beam " - "centre in the file, so this points at that centre rather than at the " - "experiment; --estimate-beam-center measures it", finder.GetFrameCount()); + logger.Warning("Beam stop: no shadow found in {} images. The comparison is made around " + "the beam centre in the file, so this may point at that centre rather " + "than at the experiment; --estimate-beam-center measures it", + finder.GetFrameCount()); else logger.Info("Beam stop shadow: {} pixels ({:.2f}% of the detector) found in {} images", shadowed, 100.0 * static_cast(shadowed) / static_cast(shadow.size()), diff --git a/tests/ShadowFinderTest.cpp b/tests/ShadowFinderTest.cpp index 92f36f74e..8b91cb1e5 100644 --- a/tests/ShadowFinderTest.cpp +++ b/tests/ShadowFinderTest.cpp @@ -103,7 +103,7 @@ TEST_CASE("ShadowFinder_FindsAnInjectedBeamStop", "[ShadowFinder]") { // Pinned from the serial implementation. A rewrite of the dilation, the hole fill or the ring // median that moves the mask by one pixel fails here, rather than in a merging statistic // several stages downstream. - CHECK(std::count(mask.begin(), mask.end(), 1u) == 2669); + CHECK(std::count(mask.begin(), mask.end(), 1u) == 2612); } // The per-pixel passes are split across threads, so where the split falls must not be visible in the @@ -175,3 +175,58 @@ TEST_CASE("ShadowFinder_ASymmetricSceneGivesASymmetricMask", "[ShadowFinder]") { } } } + +// Hardware that shadows the detector need not touch the direct beam: a pin or a loop begins some way +// out in radius, with lit detector between it and the stop. The scene here is that - a beam stop +// with its arm, and a separate opaque patch far from both - and the patch has to come back as +// shadow. Anchoring the search at the beam centre, as an earlier version did, returned the stop and +// discarded the patch however deep it was. +TEST_CASE("ShadowFinder_FindsAShadowThatDoesNotTouchTheBeam", "[ShadowFinder]") { + const DiffractionExperiment x = TestExperiment(); + const PixelMask pixel_mask(x); + ShadowFinder finder(x, pixel_mask); + + // Well clear of the stop, and wide enough that the ring it sits on still has lit pixels to be + // compared against - as a real pin shadow does, covering part of a ring and not all of it. + constexpr int PATCH_X0 = 175, PATCH_X1 = 245, PATCH_Y0 = 30, PATCH_Y1 = 90; + + std::vector> frames; + std::vector buffer; + for (int f = 0; f < NFRAMES; f++) { + frames.push_back(Scene(/*cross=*/false, /*reflection=*/false)); + for (int y = PATCH_Y0; y <= PATCH_Y1; y++) + for (int xi = PATCH_X0; xi <= PATCH_X1; xi++) + frames.back()[I(xi, y)] = 0; + DataMessage msg{}; + msg.image = CompressedImage(frames.back(), W, H); + finder.AddImage(msg, buffer); + } + + const auto mask = finder.GetMask(); + CHECK(mask[I((PATCH_X0 + PATCH_X1) / 2, (PATCH_Y0 + PATCH_Y1) / 2)] == 1); + CHECK(mask[I(PATCH_X0 + 5, PATCH_Y0 + 5)] == 1); + CHECK(mask[I(C, C)] == 1); // the stop is still found + CHECK(mask[I(PATCH_X0 - 40, PATCH_Y0)] == 0); // and the lit detector between them is kept + CHECK(mask[I(0, H - 1)] == 0); +} + +// A flat, unobstructed scene has no shadow in it. The per-pixel test no longer has to reach the +// beam centre, so what keeps it from calling a wandering background a shadow is size alone - and +// that has to hold on a scene where nothing is blocked at all. +TEST_CASE("ShadowFinder_FindsNothingOnAnUnobstructedScene", "[ShadowFinder]") { + const DiffractionExperiment x = TestExperiment(); + const PixelMask pixel_mask(x); + ShadowFinder finder(x, pixel_mask); + + std::vector> frames; + std::vector buffer; + for (int f = 0; f < NFRAMES; f++) { + frames.emplace_back(static_cast(W) * H, BACKGROUND); + DataMessage msg{}; + msg.image = CompressedImage(frames.back(), W, H); + finder.AddImage(msg, buffer); + } + + const auto mask = finder.GetMask(); + CHECK(std::count(mask.begin(), mask.end(), 1u) == 0); +}