Jungfraujoch/image_analysis/beam_stop/SHADOW_FINDER.md

Beam-stop shadow finder

ShadowFinder detects shadowed detector regions — the beam stop's central disk and holder arm, and any other hardware standing in the beam, such as a pin or a loop — from a small number of images, and returns a per-pixel mask of the shadowed region. It exists so the pipeline can mask them automatically instead of relying on a hand-drawn user mask.

It is written in the accumulate-then-finalize shape of DarkMaskAnalysis: feed frames with AddImage(), then read the mask once with GetMask(). It is built as part of JFJochImageAnalysis and wired into rugnux: Rugnux::FindBeamStop() feeds it 60 equally-spaced frames and loads the result into PixelMask bit 9. On by default; --detect-beam-stop[=N|off] controls it. The class is self-contained (common/ only) and unit-testable.

API

ShadowFinder finder(experiment);                 // geometry (beam centre, converted size)
std::vector<uint8_t> scratch;
for (int i : first_N_frames)                      // ~20–50 spread frames is plenty (see below)
    finder.AddImage(read_frame(i), scratch);      // DataMessage with .image, like DarkMaskAnalysis
finder.BeamCenter(x, y);                          // optional: the measured centre, not the file's
std::vector<uint32_t> shadow = finder.GetMask();  // 1 = shadow, 0 = keep; converted geometry

• Input (AddImage): a full converted-geometry image inside a DataMessage, dispatched by pixel type exactly like DarkMaskAnalysis::AnalyzeImage. A pixel equal to its type's sentinel extreme (INT*_MIN for signed, UINT*_MAX for unsigned) is treated as no data (module gap / masked) and skipped. For signed types the opposite extreme (INT*_MAX, a genuine saturation clamp) is kept, so saturated reflections still register as bright. Thread-safe.
• Output (GetMask): a 0/1 vector of size GetPixelsNumConv(), ready to feed PixelMask::LoadMask(..., bit). Recomputed from the accumulators on each call — call it once at the end.
• Settings (ShadowFinderSettings): plain struct with defaults (below). When wired, this can move onto DiffractionExperiment the way DarkMaskSettings does.

Algorithm

The shadow is a stationary, azimuthal anomaly: air scatter makes the background fall off smoothly with radius but stay roughly constant around each ring, so the beam stop is a localized dip at a fixed place. Detection therefore compares each pixel to the typical background at its own radius, not to a global threshold.

1. Projection. Per pixel accumulate the max and the mean (sum + count) over the added frames, skipping the no-data sentinel. The stop is stationary, so it stays dark while spots and the air-scatter halo build up around it.
2. Polarization. The background is not flat around a ring even with nothing in the beam: a polarized source suppresses it in its own plane by 1 - sin^2(2 theta), which is a factor of three at 2 theta = 55 deg and four at 70 deg — several times the dip being looked for, so on a short-distance geometry the two horizontal lobes of every outer ring read as shadow. The mean is therefore divided by the Kahn factor before the comparison, read off the experiment's own geometry so that detector tilt, a quarter-turned image and an in-plane rotation all move the polarization plane correctly, about the centre named in step 3. It is the only correction a ring carries that varies along it — solid angle, detector and air absorption are functions of 2 theta alone and the ring's own median absorbs them. The Poisson test of step 5 puts the factor back, so the significance is still the significance of the counts that were recorded. Nothing is corrected when the experiment states no polarization.
3. Robust radial baseline. Take the median of the mean-projection in each 1-pixel ring about the beam centre. The median ignores the shadow (a minority of each ring) and the spots. Iterate 3×, excluding the shadow it finds, so the baseline is not pulled down by the shadow. The rings are drawn about the centre the caller names with BeamCenter(), which defaults to the file's. Tens of pixels of error cost nothing, but a couple of hundred draws the rings across the background's own fall-off instead of along it and a large region then reads as shadow for reasons that have nothing to do with hardware — so rugnux fits the centre from this same projection first and names that one (see below).
4. Ratio. ratio = mean / baseline(r). Shadowed pixels sit far below 1; the smooth halo and symmetric ice rings cancel to ≈1 and are not flagged.
5. Counting statistics. A ratio on its own says nothing when the background it is made of is a handful of photons, so the deficit is also required to be significant against its own Poisson scatter: sigma = sqrt(2 (E - N + N ln(N/E))) for the pooled counts N observed against the E the ring expects, cut at min_deficit_sigma. This is what makes the test scale-free — a third of the background missing is overwhelming on a well-exposed sweep and is noise on six low-background frames, and without it such a pre-scan masks three quarters of a detector.
6. Core + size. core = ratio < shadow_ratio && sigma > min_deficit_sigma, then keep only components holding at least min_shadow_pixels core pixels (components taken through a bridge_px-dilated copy, so the arm crosses module gaps). Nothing anchors the region to the beam centre: hardware that shadows the detector need not touch the direct beam, and a pin or a loop typically does not — it begins some way out in radius with lit detector between it and the stop, which an anchored search discards however deep it is. Size is what makes the test specific instead; the background wanders a pixel or two at a time, hardware does not.
7. Reflection guard. A pixel whose max-projection reached min_reflection recorded a real reflection — a beam stop cannot block a reflection that was measured — so it is never masked (a small-cluster test drops single-pixel zingers).
8. Central disk (low-res guard). Inside the stop a whole ring is blocked, so its own median is blocked too and step 4 has nothing to compare against. Walking outward, every ring whose background is below blocked_ring_ratio of what this detector's background typically is - the median over the rings the walk is willing to judge - lies wholly inside the stop and is shadow in its entirety. The comparison is deliberately not against the largest background further out: a sample whose background peaks in a strong ring away from the beam (a powder standard, a strong solvent ring) then reads as a stop the size of that ring.
9. Soft boundary. Grow the core+disk into partially-shadowed pixels (ratio < penumbra_ratio) within penumbra_max_px, then a morphological close rounds the contour and hole-filling closes the disk interior.
10. Carve reflections last. Remove any lit pixel from the final mask (no fill afterwards) so a spot the geometry still covered is given back rather than re-enclosed.
11. Arms that let part of the beam through. A thin holder arm, or one blurred by its distance from the detector, sits at 40-70 % of the background along most of its length and crosses shadow_ratio only in places, so step 6 keeps only deep fragments, each below min_shadow_pixels; under a bright background step 10 also gives back its attenuated pixels, which reach min_reflection. So, after step 10, the dim pixels the mask left out (ratio < penumbra_ratio && sigma > min_deficit_sigma) are joined through a bridge_px-dilated copy and across any module gap whose two sides both belong to them, however wide the gap, and a piece with at least min_shadow_pixels dim pixels of which min_core_pixels are core is added whole. Nothing is given back inside it: the reflections behind transmitting hardware are recorded attenuated. The step only ever adds such pieces - the dim edge of an ordinary stop is left as steps 6-10 drew it, so a sweep without transmitting hardware keeps its mask bit for bit (an earlier version redrew every stop's edge from the dim pixels and moved every mask in the corpus by 0.01-0.7 % of the detector). "Dim" here is against the ring as it varies in azimuth: each 64 px radial band fits its sector medians (24 sectors, hardware sectors left out) with a second harmonic, cos 2φ, and a pixel is dim below penumbra_ratio times that model where the model is below 1. A polarization factor that is not the beam's leaves exactly such a harmonic after the division, dim lobes at high angle over a quarter of the detector, which the median alone read as one enormous transmitting piece. The model is capped at 1, so it only ever removes dim pixels and a sweep without such a piece still keeps its mask bit for bit.

Three findings are baked in: the mean ratio (not the max) does the detecting — the max projection is a great picture but a noisier detector; size and counting statistics, not a position, are what distinguish hardware from a stray low patch; and a lit pixel is never masked. Detection converges in ~2–3 frames (the shadow cuts the halo present in every frame); 20–50 is just safe headroom.

Parameters (ShadowFinderSettings)

field	default	meaning
shadow_ratio	0.50	core threshold: mean below this fraction of the radial baseline
min_deficit_sigma	6	… and the deficit significant to this many Poisson standard deviations
min_shadow_pixels	2000	smallest region the per-pixel test may return
min_core_pixels	200	core pixels a dim piece of step 11 must hold
penumbra_ratio	0.75	soft boundary grows out to this ratio …
penumbra_max_px	30	… but no further than this from the core
bridge_px	6	bridge module gaps / breaks the arm crosses
min_reflection	25	max-projection value at/above which a pixel is a protected reflection

Complexity & memory

Per AddImage: O(pixels). GetMask: a handful of O(pixels) sweeps (three ring-median passes and a few BFS-based morphology ops), i.e. a couple of seconds on a full ~18 MP detector — a one-time finalize. Accumulators are int32 + int64 + uint32 per pixel (~0.3 GB at 18 MP); the finalize adds a few transient float/char buffers. Fine for the offline (Rugnux/viewer) context it targets; not intended for the online per-frame hot path.

Wiring (done for rugnux; the broker consumer below is still open)

The detector is deliberately reader-agnostic and bit-agnostic: one algorithm, fed frames by each caller, with the caller deciding where the resulting mask lands. Two intended consumers:

1. Build. (Done.) beam_stop/ShadowFinder.cpp + .h are in the JFJochImageAnalysis source list in image_analysis/CMakeLists.txt, beside dark_mask_analysis/DarkMaskAnalysis.*.

2. Mask bit + loader (common/PixelMask.{h,cpp}): add BeamStopShadowBit = 9 (bit 9 is free; PixelMask.h:31-36), a LoadBeamStopMask mirroring LoadDarkBadPixelMask (.cpp:298-315), a beam_stop_pixel counter in PixelMaskStatistics / GetStatistics (.cpp:186-205), and optionally GetBeamStopMask() (mirroring GetUserMask). Leave GetUserMask untouched (keeps the user-mask channel clean). Application is then automatic: the CPU/GPU preprocessor masks any pixel with a non-zero bit (ImagePreprocessorCPU.cpp:10, ImagePreprocessorGPU.cu:104), and HDF5 write/read + CBOR already carry the full bitfield, so bit 9 persists to _process.h5 and reloads with no extra code.

3. Rugnux offline processing → bit 9 (dataset-specific, per run). Add a detect_beam_stop flag to ProcessConfig (rugnux/Rugnux.h) and a --detect-beam-stop CLI flag mirroring --detect-ice-rings in rugnux/rugnux_cli.cpp. In Rugnux::Run(), after geometry/goniometer setup and before the azimuthal mapping (Rugnux.cpp:281) and the start_message.pixel_mask persist (:296), run a ShadowFinder pre-pass over a spread sample (select_equally_spaced_image_ordinals

• reader_.GetRawImage, as RefineStillsGeometry does at :96-157) and pixel_mask_.LoadBeamStopMask(...). Everything downstream then applies and persists it for free.

4. Viewer standalone → user mask (bit 8) and a viewer preview. A "Find beam stop" action reads the first N frames via the worker's file_reader, runs ShadowFinder, and holds the result as bit 9 for a distinct-coloured, toggleable overlay; a "Transfer to user mask" action ORs it into the user mask through the existing chokepoint JFJochImageReadingWorker::UpdateUserMask_i (.cpp:783, same idiom as LoadUserMaskTIFF add-mode :851-858). The distinct colour needs a ColorScaleSpecial::BeamStop + sentinel branch in JFJochReaderImage.cpp:117-124; the on/off toggle mirrors "Show saturated pixels".

5. Docs. Add a "Bit 9 – beam-stop shadow" row to the bit table in docs/PIXEL_MASK.md:9-21.

Not needed: the online broker / receiver / FPGA / REST-API / React-frontend paths — the auto beam-stop mask is an offline (Rugnux + viewer) feature and never goes to the detector.

Known limitations / future refinements

• Boundary smoothing uses a morphological close; the prototype used a Gaussian level-set for a slightly smoother edge. Upgrade if the edge looks blocky.
• Off-centre disk: the low-res disk guard keys on the stored beam centre. If that is off the physical stop (seen on one stills detector), a small already-dead disk gets no extra circular guard — a data-driven disk centre would close it.
• A grossly wrong beam centre: closed in rugnux by ordering. The centre is fitted from the projection before the mask is read, and named with BeamCenter(); the shadow that biases that fit moves it by a few pixels, three orders below what the ring comparison notices. What remains is the case where the fit declines to place a centre at all, where the file's is used as before. Another caller that reads the mask at the file's centre keeps the old exposure.
• The polarization plane is taken from the experiment, not measured. The correction follows DiffractionGeometry::CalcAzIntPolarizationCorr, which fixes the plane on the lab x axis and expresses a vertically polarized source as a negative coefficient — the same assumption azimuthal integration already makes, so a beamline where it is wrong is wrong there first. Measured over the corpus the plane sits within a degree of lab x on every dataset, and the amplitude within tens of per cent of the prediction, which is far inside the margin the 0.50 threshold leaves. An experiment that states no polarization is left uncorrected, as before.
• Unsigned saturation: for unsigned images a detector-saturating reflection collides with the no-data sentinel and is skipped; harmless in practice (such reflections do not sit under the stop), and absent entirely on the signed/preprocessed path.