Jungfraujoch/image_analysis/beam_stop/SHADOW_FINDER.md

Beam-stop shadow finder

ShadowFinder detects shadowed detector regions — the beam stop's central disk and holder arm, and any other hardware standing in the beam, such as a pin or a loop — from a small number of images, and returns a per-pixel mask of the shadowed region. It exists so the pipeline can mask them automatically instead of relying on a hand-drawn user mask.

It is written in the accumulate-then-finalize shape of DarkMaskAnalysis: feed frames with AddImage(), then read the mask once with GetMask(). It is built as part of JFJochImageAnalysis and wired into rugnux: Rugnux::FindBeamStop() feeds it 60 equally-spaced frames and loads the result into PixelMask bit 9. On by default; --detect-beam-stop[=N|off] controls it. The class is self-contained (common/ only) and unit-testable.

API

ShadowFinder finder(experiment);                 // geometry (beam centre, converted size)
std::vector<uint8_t> scratch;
for (int i : first_N_frames)                      // ~20–50 spread frames is plenty (see below)
    finder.AddImage(read_frame(i), scratch);      // DataMessage with .image, like DarkMaskAnalysis
finder.BeamCenter(x, y);                          // optional: the measured centre, not the file's
std::vector<uint32_t> shadow = finder.GetMask();  // 1 = shadow, 0 = keep; converted geometry

• Input (AddImage): a full converted-geometry image inside a DataMessage, dispatched by pixel type exactly like DarkMaskAnalysis::AnalyzeImage. A pixel equal to its type's sentinel extreme (INT*_MIN for signed, UINT*_MAX for unsigned) is treated as no data (module gap / masked) and skipped. For signed types the opposite extreme (INT*_MAX, a genuine saturation clamp) is kept, so saturated reflections still register as bright. Thread-safe.
• Output (GetMask): a 0/1 vector of size GetPixelsNumConv(), ready to feed PixelMask::LoadMask(..., bit). Recomputed from the accumulators on each call — call it once at the end.
• Settings (ShadowFinderSettings): plain struct with defaults (below). When wired, this can move onto DiffractionExperiment the way DarkMaskSettings does.

Algorithm

The shadow is a stationary, azimuthal anomaly: air scatter makes the background fall off smoothly with radius but stay roughly constant around each ring, so the beam stop is a localized dip at a fixed place. Detection therefore compares each pixel to the typical background at its own radius, not to a global threshold.

1. Projection. Per pixel accumulate the max and the mean (sum + count) over the added frames, skipping the no-data sentinel. The stop is stationary, so it stays dark while spots and the air-scatter halo build up around it.
2. Polarization. The background is not flat around a ring even with nothing in the beam: a polarized source suppresses it in its own plane by 1 - sin^2(2 theta), which is a factor of three at 2 theta = 55 deg and four at 70 deg — several times the dip being looked for, so on a short-distance geometry the two horizontal lobes of every outer ring read as shadow. The mean is therefore divided by the Kahn factor before the comparison, read off the experiment's own geometry so that detector tilt, a quarter-turned image and an in-plane rotation all move the polarization plane correctly, about the centre named in step 3. On a tilted detector the solid angle of a pixel also varies along a ring (by tens of per cent at a 5 deg 2theta swing and 50 mm), so the mean is divided by that too: the pixel's solid angle over the one an untilted detector has at the same 2 theta, (distance / l_z)^3, which is exactly 1 untilted. Detector and air absorption are functions of 2 theta alone, nearly enough, and the ring's own median absorbs them. The Poisson test of step 5 puts the factors back, so the significance is still the significance of the counts that were recorded. No polarization is corrected when the experiment states none.
3. Robust radial baseline. Take the median of the mean-projection in each 1-pixel ring about the beam. A ring is a cone of constant 2 theta, indexed by distance * tan(2 theta) in pixels: a circle about the centre on an untilted detector, but on a tilted one (a swung 2theta arm) a circle about the centre runs across the background's fall-off, and at 5 deg and 50 mm that put a quarter of the detector in the shadow. The median ignores the shadow (a minority of each ring) and the spots. Iterate 3×, excluding the shadow it finds, so the baseline is not pulled down by the shadow. The rings are drawn about the centre the caller names with BeamCenter(), which defaults to the file's. Tens of pixels of error cost nothing, but a couple of hundred draws the rings across the background's own fall-off instead of along it and a large region then reads as shadow for reasons that have nothing to do with hardware — so rugnux fits the centre from this same projection first and names that one (see below).
4. Ratio. ratio = mean / baseline(r). Shadowed pixels sit far below 1; the smooth halo and symmetric ice rings cancel to ≈1 and are not flagged.
5. Counting statistics. A ratio on its own says nothing when the background it is made of is a handful of photons, so the deficit is also required to be significant against its own Poisson scatter: sigma = sqrt(2 (E - N + N ln(N/E))) for the pooled counts N observed against the E the ring expects, cut at min_deficit_sigma. This is what makes the test scale-free — a third of the background missing is overwhelming on a well-exposed sweep and is noise on six low-background frames, and without it such a pre-scan masks three quarters of a detector.
6. Core + size. core = ratio < shadow_ratio && sigma > min_deficit_sigma, then keep only components holding at least min_shadow_pixels core pixels (components taken through a bridge_px-dilated copy, so the arm crosses module gaps). Nothing anchors the region to the beam centre: hardware that shadows the detector need not touch the direct beam, and a pin or a loop typically does not — it begins some way out in radius with lit detector between it and the stop, which an anchored search discards however deep it is. Size is what makes the test specific instead; the background wanders a pixel or two at a time, hardware does not.
7. Reflection guard. A pixel whose max-projection reached min_reflection recorded a real reflection — a beam stop cannot block a reflection that was measured — so it is never masked (a small-cluster test drops single-pixel zingers).
8. Central disk (low-res guard). Inside the stop a whole ring is blocked, so its own median is blocked too and step 4 has nothing to compare against. Walking outward, every ring whose background is below blocked_ring_ratio of what this detector's background typically is - the median over the rings the walk is willing to judge - lies wholly inside the stop and is shadow in its entirety. The comparison is deliberately not against the largest background further out: a sample whose background peaks in a strong ring away from the beam (a powder standard, a strong solvent ring) then reads as a stop the size of that ring.
9. Soft boundary. Grow the core+disk into partially-shadowed pixels (ratio < penumbra_ratio) within penumbra_max_px, then a morphological close rounds the contour and hole-filling closes the disk interior.
10. Carve reflections last. Remove any lit pixel from the final mask (no fill afterwards) so a spot the geometry still covered is given back rather than re-enclosed.
11. Arms that let part of the beam through. A thin holder arm, or one blurred by its distance from the detector, sits at 40-70 % of the background along most of its length and crosses shadow_ratio only in places, so step 6 keeps only deep fragments, each below min_shadow_pixels; under a bright background step 10 also gives back its attenuated pixels, which reach min_reflection. So, after step 10, the dim pixels the mask left out (ratio < penumbra_ratio && sigma > min_deficit_sigma) are joined through a bridge_px-dilated copy and across any module gap whose two sides both belong to them, however wide the gap, and a piece with at least min_shadow_pixels dim pixels of which min_core_pixels are core is added whole. Nothing is given back inside it: the reflections behind transmitting hardware are recorded attenuated. The step only ever adds such pieces - the dim edge of an ordinary stop is left as steps 6-10 drew it, so a sweep without transmitting hardware keeps its mask bit for bit (an earlier version redrew every stop's edge from the dim pixels and moved every mask in the corpus by 0.01-0.7 % of the detector). "Dim" here is against the ring as it varies in azimuth: each 64 px radial band fits its sector medians (24 sectors, hardware sectors left out) with a second harmonic, cos 2φ, and a pixel is dim below penumbra_ratio times that model where the model is below 1. A polarization factor that is not the beam's leaves exactly such a harmonic after the division, dim lobes at high angle over a quarter of the detector, which the median alone read as one enormous transmitting piece. The model is capped at 1, so it only ever removes dim pixels and a sweep without such a piece still keeps its mask bit for bit.

Three findings are baked in: the mean ratio (not the max) does the detecting — the max projection is a great picture but a noisier detector; size and counting statistics, not a position, are what distinguish hardware from a stray low patch; and a lit pixel is never masked. Detection converges in ~2–3 frames (the shadow cuts the halo present in every frame); 20–50 is just safe headroom.

Parameters (ShadowFinderSettings)

field	default	meaning
shadow_ratio	0.50	core threshold: mean below this fraction of the radial baseline
min_deficit_sigma	6	… and the deficit significant to this many Poisson standard deviations
min_shadow_pixels	2000	smallest region the per-pixel test may return
min_core_pixels	200	core pixels a dim piece of step 11 must hold
penumbra_ratio	0.75	soft boundary grows out to this ratio …
penumbra_max_px	30	… but no further than this from the core
bridge_px	6	bridge module gaps / breaks the arm crosses
min_reflection	25	max-projection value, in photons (times the gain on a detector read in ADU), at/above which a pixel is a protected reflection

Complexity & memory

Per AddImage: O(pixels). GetMask: a handful of O(pixels) sweeps (three ring-median passes and a few BFS-based morphology ops), i.e. a couple of seconds on a full ~18 MP detector — a one-time finalize. Accumulators are int32 + int64 + uint32 per pixel (~0.3 GB at 18 MP); the finalize adds a few transient float/char buffers. Fine for the offline (Rugnux/viewer) context it targets; not intended for the online per-frame hot path.

Wiring (done for rugnux; the broker consumer below is still open)

The detector is deliberately reader-agnostic and bit-agnostic: one algorithm, fed frames by each caller, with the caller deciding where the resulting mask lands. Two intended consumers:

1. Build. (Done.) beam_stop/ShadowFinder.cpp + .h are in the JFJochImageAnalysis source list in image_analysis/CMakeLists.txt, beside dark_mask_analysis/DarkMaskAnalysis.*.

2. Mask bit + loader (common/PixelMask.{h,cpp}): add BeamStopShadowBit = 9 (bit 9 is free; PixelMask.h:31-36), a LoadBeamStopMask mirroring LoadDarkBadPixelMask (.cpp:298-315), a beam_stop_pixel counter in PixelMaskStatistics / GetStatistics (.cpp:186-205), and optionally GetBeamStopMask() (mirroring GetUserMask). Leave GetUserMask untouched (keeps the user-mask channel clean). Application is then automatic: the CPU/GPU preprocessor masks any pixel with a non-zero bit (ImagePreprocessorCPU.cpp:10, ImagePreprocessorGPU.cu:104), and HDF5 write/read + CBOR already carry the full bitfield, so bit 9 persists to _process.h5 and reloads with no extra code.

3. Rugnux offline processing → bit 9 (dataset-specific, per run). Add a detect_beam_stop flag to ProcessConfig (rugnux/Rugnux.h) and a --detect-beam-stop CLI flag mirroring --detect-ice-rings in rugnux/rugnux_cli.cpp. In Rugnux::Run(), after geometry/goniometer setup and before the azimuthal mapping (Rugnux.cpp:281) and the start_message.pixel_mask persist (:296), run a ShadowFinder pre-pass over a spread sample (select_equally_spaced_image_ordinals

• reader_.GetRawImage, as RefineStillsGeometry does at :96-157) and pixel_mask_.LoadBeamStopMask(...). Everything downstream then applies and persists it for free.

4. Viewer standalone → user mask (bit 8) and a viewer preview. A "Find beam stop" action reads the first N frames via the worker's file_reader, runs ShadowFinder, and holds the result as bit 9 for a distinct-coloured, toggleable overlay; a "Transfer to user mask" action ORs it into the user mask through the existing chokepoint JFJochImageReadingWorker::UpdateUserMask_i (.cpp:783, same idiom as LoadUserMaskTIFF add-mode :851-858). The distinct colour needs a ColorScaleSpecial::BeamStop + sentinel branch in JFJochReaderImage.cpp:117-124; the on/off toggle mirrors "Show saturated pixels".

5. Docs. Add a "Bit 9 – beam-stop shadow" row to the bit table in docs/PIXEL_MASK.md:9-21.

Not needed: the online broker / receiver / FPGA / REST-API / React-frontend paths — the auto beam-stop mask is an offline (Rugnux + viewer) feature and never goes to the detector.

Known limitations / future refinements

• Boundary smoothing uses a morphological close; the prototype used a Gaussian level-set for a slightly smoother edge. Upgrade if the edge looks blocky.
• Off-centre disk: the low-res disk guard keys on the stored beam centre. If that is off the physical stop (seen on one stills detector), a small already-dead disk gets no extra circular guard — a data-driven disk centre would close it.
• A grossly wrong beam centre: closed in rugnux by ordering. The centre is fitted from the projection before the mask is read, and named with BeamCenter(); the shadow that biases that fit moves it by a few pixels, three orders below what the ring comparison notices. What remains is the case where the fit declines to place a centre at all, where the file's is used as before. Another caller that reads the mask at the file's centre keeps the old exposure.
• The polarization degree is taken from the experiment; the plane, where the background shows one, from the background. The correction follows DiffractionGeometry::CalcAzIntPolarizationCorr, which fixes the plane on the lab x axis and expresses a vertically polarized source as a negative coefficient. The degree cannot be told from the data, but the plane can, and it is not always the one assumed: a detector mounted a quarter turn from another puts the suppressed direction along the image's other axis. Divided by the factor of the wrong plane, the background comes out less flat than it went in - on one stills run the outer rings carried dim lobes over the detector's corners, sitting on the arm search's thresholds, so that the third decimal of the stated degree moved thousands of pixels in and out of the mask. So before the correction the mean projection is compared ring by ring with the profile a polarized source gives each 64 px band (each ring relative to its upper decile, each sector the brighter of itself and the one opposite, since a shadow is rarely on both sides; a band counted where three quarters of the azimuth are seen). The stated degree is applied in the plane the background follows when it follows one clearly - a correlation above 0.5 and at least 0.3 of a fully polarized source's modulation - and as stated otherwise. Over the rotation and stills corpora only the run whose plane lies along the image's y axis is turned; on every other run the correction, and so the mask, is what it was. An experiment that states no polarization is left uncorrected, as before. What was applied is read back with GetAppliedPolarization(), and rugnux's pre-scan writes it into the experiment, so the integration's correction follows the same plane: the plane is decided once, here.
• Unsigned saturation: for unsigned images a detector-saturating reflection collides with the no-data sentinel and is skipped; harmless in practice (such reflections do not sit under the stop), and absent entirely on the signed/preprocessed path.