diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 7ab53f40d..659d707a7 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -232,8 +232,9 @@ Elements Z = 1 to 92 and 48 Additional Substances of Dosimetric Interest" (1995, NIST Standard Reference Database 126 [doi:10.18434/T4D01F](https://doi.org/10.18434/T4D01F). -**Polarization correction** — the azimuthal polarization factor applied to both the azimuthally -integrated profile and the integrated Bragg intensities is the one derived for a partially polarized +**Polarization correction** — the azimuthal polarization factor applied to the azimuthally +integrated profile, to the integrated Bragg intensities and to the ring background the beam-stop +shadow test compares a pixel against is the one derived for a partially polarized synchrotron source by R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas and D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction" (1982), J. Appl. Cryst. 15, 330-337 diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 6d66d1dcf..78843cc1c 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -55,7 +55,7 @@ The methods draw on, and in places reimplement, solutions from: - M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14). - J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and [github.com/jonwright/bslz4decoders](https://github.com/jonwright/bslz4decoders) (device-side decoding of bitshuffle+LZ4 images, §0). - A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", *J. Appl. Cryst.* **44** (2011), 1285-1287 (anomalous difference density read at the model's sites). -- R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", *J. Appl. Cryst.* **15** (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to both the azimuthal profile and the Bragg intensities). +- R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", *J. Appl. Cryst.* **15** (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to the azimuthal profile, the Bragg intensities and the ring background the beam-stop shadow test compares against). - R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", *Acta Cryst.* **A42** (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4). - A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", *Acta Cryst.* **D58** (2002), 1387-1392 (the flat bulk-solvent model, its optimal parameters and the range they are physically meaningful over, used when scaling a model to the data in §14). - P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", *Acta Cryst.* **D61** (2005), 850-855 (the grid search over that range that fits k_sol and b_sol, with the overall scale and anisotropic B refitted at each grid point). diff --git a/image_analysis/beam_stop/SHADOW_FINDER.md b/image_analysis/beam_stop/SHADOW_FINDER.md index 32ea60734..fdcf4fa7f 100644 --- a/image_analysis/beam_stop/SHADOW_FINDER.md +++ b/image_analysis/beam_stop/SHADOW_FINDER.md @@ -44,7 +44,18 @@ own radius**, not to a global threshold. 1. **Projection.** Per pixel accumulate the **max** and the **mean** (sum + count) over the added frames, skipping the no-data sentinel. The stop is stationary, so it stays dark while spots and the air-scatter halo build up around it. -2. **Robust radial baseline.** Take the **median of the mean-projection in each 1-pixel ring** +2. **Polarization.** The background is not flat around a ring even with nothing in the beam: a + polarized source suppresses it in its own plane by `1 - sin^2(2 theta)`, which is a factor of + three at `2 theta = 55 deg` and four at 70 deg — several times the dip being looked for, so on a + short-distance geometry the two horizontal lobes of every outer ring read as shadow. The mean is + therefore divided by the Kahn factor before the comparison, read off the experiment's own + geometry so that detector tilt, a quarter-turned image and an in-plane rotation all move the + polarization plane correctly, about the centre named in step 3. It is the **only** correction a + ring carries that varies along it — solid angle, detector and air absorption are functions of + `2 theta` alone and the ring's own median absorbs them. The Poisson test of step 5 puts the + factor back, so the significance is still the significance of the counts that were recorded. + Nothing is corrected when the experiment states no polarization. +3. **Robust radial baseline.** Take the **median of the mean-projection in each 1-pixel ring** about the beam centre. The median ignores the shadow (a minority of each ring) and the spots. Iterate 3×, excluding the shadow it finds, so the baseline is not pulled down by the shadow. The rings are drawn about the centre the caller names with `BeamCenter()`, which defaults to the @@ -52,34 +63,34 @@ own radius**, not to a global threshold. background's own fall-off instead of along it and a large region then reads as shadow for reasons that have nothing to do with hardware — so `rugnux` fits the centre from this same projection first and names that one (see below). -3. **Ratio.** `ratio = mean / baseline(r)`. Shadowed pixels sit far below 1; the smooth halo and +4. **Ratio.** `ratio = mean / baseline(r)`. Shadowed pixels sit far below 1; the smooth halo and symmetric ice rings cancel to ≈1 and are **not** flagged. -4. **Counting statistics.** A ratio on its own says nothing when the background it is made of is a +5. **Counting statistics.** A ratio on its own says nothing when the background it is made of is a handful of photons, so the deficit is also required to be significant against its own Poisson scatter: `sigma = sqrt(2 (E - N + N ln(N/E)))` for the pooled counts `N` observed against the `E` the ring expects, cut at `min_deficit_sigma`. This is what makes the test scale-free — a third of the background missing is overwhelming on a well-exposed sweep and is noise on six low-background frames, and without it such a pre-scan masks three quarters of a detector. -5. **Core + size.** `core = ratio < shadow_ratio && sigma > min_deficit_sigma`, then keep only +6. **Core + size.** `core = ratio < shadow_ratio && sigma > min_deficit_sigma`, then keep only components holding at least `min_shadow_pixels` core pixels (components taken through a `bridge_px`-dilated copy, so the arm crosses module gaps). **Nothing anchors the region to the beam centre**: hardware that shadows the detector need not touch the direct beam, and a pin or a loop typically does not — it begins some way out in radius with lit detector between it and the stop, which an anchored search discards however deep it is. Size is what makes the test specific instead; the background wanders a pixel or two at a time, hardware does not. -6. **Reflection guard.** A pixel whose **max**-projection reached `min_reflection` recorded a real +7. **Reflection guard.** A pixel whose **max**-projection reached `min_reflection` recorded a real reflection — a beam stop cannot block a reflection that was measured — so it is **never masked** (a small-cluster test drops single-pixel zingers). -7. **Central disk (low-res guard).** From the azimuthal **blocked fraction** about the beam centre, +8. **Central disk (low-res guard).** From the azimuthal **blocked fraction** about the beam centre, mask a full disk out to the radius where nearly every azimuth is blocked (a disk blocks all directions; a thin arm/gap does not). Returns nothing when the centre is not behind a disk, and is **capped just inside the innermost reflection** so a real edge spot is never swallowed. This guards the low-resolution zone the stop makes un-measurable, including a disk the raw data has already voided. -8. **Soft boundary.** Grow the core+disk into partially-shadowed pixels (`ratio < penumbra_ratio`) +9. **Soft boundary.** Grow the core+disk into partially-shadowed pixels (`ratio < penumbra_ratio`) within `penumbra_max_px`, then a morphological close rounds the contour and hole-filling closes the disk interior. -9. **Carve reflections last.** Remove any lit pixel from the final mask (no fill afterwards) so a +10. **Carve reflections last.** Remove any lit pixel from the final mask (no fill afterwards) so a spot the geometry still covered is given back rather than re-enclosed. Three findings are baked in: the **mean** ratio (not the max) does the detecting — the max @@ -159,6 +170,13 @@ beam-stop mask is an **offline** (Rugnux + viewer) feature and never goes to the fit moves it by a few pixels, three orders below what the ring comparison notices. What remains is the case where the fit declines to place a centre at all, where the file's is used as before. Another caller that reads the mask at the file's centre keeps the old exposure. +- **The polarization plane is taken from the experiment, not measured.** The correction follows + `DiffractionGeometry::CalcAzIntPolarizationCorr`, which fixes the plane on the lab x axis and + expresses a vertically polarized source as a negative coefficient — the same assumption azimuthal + integration already makes, so a beamline where it is wrong is wrong there first. Measured over the + corpus the plane sits within a degree of lab x on every dataset, and the amplitude within tens of + per cent of the prediction, which is far inside the margin the 0.50 threshold leaves. An + experiment that states no polarization is left uncorrected, as before. - **Unsigned saturation**: for unsigned images a detector-saturating reflection collides with the no-data sentinel and is skipped; harmless in practice (such reflections do not sit under the stop), and absent entirely on the signed/preprocessed path. diff --git a/image_analysis/beam_stop/ShadowFinder.cpp b/image_analysis/beam_stop/ShadowFinder.cpp index 94b1af50b..34a60d610 100644 --- a/image_analysis/beam_stop/ShadowFinder.cpp +++ b/image_analysis/beam_stop/ShadowFinder.cpp @@ -290,6 +290,8 @@ ShadowFinder::ShadowFinder(const DiffractionExperiment &experiment, const PixelM height(static_cast(experiment.GetYPixelsNumConv())), beam_x(experiment.GetBeamX_pxl()), beam_y(experiment.GetBeamY_pxl()), + geometry(experiment.GetDiffractionGeometry()), + polarization(experiment.GetPolarizationFactor()), pixel_mask(mask.GetMask(experiment)) { if (pixel_mask.size() != static_cast(width) * height) throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, @@ -528,8 +530,17 @@ std::vector ShadowFinder::GetMask(size_t nthreads) const { if (frames == 0) return mask; - // mean projection, usable pixels and radius from the beam centre + // mean projection, usable pixels and radius from the beam centre. The mean is divided by the + // polarization factor, so that what is left varies around a ring only where something is in the + // way; `pol` is kept because the counts the Poisson test is made of are the ones that were + // recorded, not these. The factor is read off the experiment's own geometry - which follows the + // detector's tilt, quarter turns and in-plane rotation - about the centre the rings use. + // Following Kahn, Fourme, Gadet, Janin, Dumas & Andre (1982) J. Appl. Cryst. 15, 330-337 + DiffractionGeometry pol_geometry = geometry; + pol_geometry.BeamX_pxl(beam_x).BeamY_pxl(beam_y); + std::vector mean(n_pixels, 0.0f); + std::vector pol(n_pixels, 1.0f); std::vector valid(n_pixels, 0); std::vector radius(n_pixels, 0); std::atomic max_radius_atomic{0}; @@ -538,11 +549,16 @@ std::vector ShadowFinder::GetMask(size_t nthreads) const { for (int y = ylo; y < yhi; y++) for (int x = 0; x < W; x++) { const int i = y * W + x; + const float dx = x - beam_x, dy = y - beam_y; + if (polarization.has_value()) + pol[i] = pol_geometry.CalcAzIntPolarizationCorr(static_cast(x), + static_cast(y), + polarization.value()); if (valid_count[i] > 0 && pixel_mask[i] == 0) { - mean[i] = static_cast(static_cast(sum_value[i]) / valid_count[i]); + mean[i] = static_cast(static_cast(sum_value[i]) + / valid_count[i] / pol[i]); valid[i] = 1; } - const float dx = x - beam_x, dy = y - beam_y; radius[i] = static_cast(std::lround(std::sqrt(dx * dx + dy * dy))); local_max = std::max(local_max, radius[i]); } @@ -660,7 +676,11 @@ std::vector ShadowFinder::GetMask(size_t nthreads) const { for (int i = lo; i < hi; i++) { if (!valid[i]) continue; - const double counted = static_cast(frames) * pooled_count[i]; + // The deficit is significant or not in the counts that were recorded, so the pooled + // background and what the ring expects of it are both put back on the detector's own + // scale before they are compared - the same factor on each, so it is the units of the + // comparison that change and not its answer. + const double counted = static_cast(frames) * pooled_count[i] * pol[i]; deficit[i] = static_cast(poisson_deficit_sigma(pooled[i] * counted, baseline[radius[i]] * counted)); low[i] = ratio[i] < SHADOW_RATIO && deficit[i] > MIN_DEFICIT_SIGMA; diff --git a/image_analysis/beam_stop/ShadowFinder.h b/image_analysis/beam_stop/ShadowFinder.h index a337ac0a2..5ca1a44bf 100644 --- a/image_analysis/beam_stop/ShadowFinder.h +++ b/image_analysis/beam_stop/ShadowFinder.h @@ -7,10 +7,12 @@ #include #include #include +#include #include #include "../../common/CompressedImage.h" #include "../../common/DiffractionExperiment.h" +#include "../../common/DiffractionGeometry.h" #include "../../common/JFJochMessages.h" #include "../../common/PixelMask.h" #ifdef JFJOCH_USE_CUDA @@ -45,6 +47,19 @@ class ShadowFinder { float beam_x; float beam_y; + // What the background owes to the source rather than to the hardware. The scattered background + // is not flat around a ring: a polarized source suppresses it in its own plane by a factor that + // reaches three at the 2 theta a short detector distance puts in a corner - several times the + // dip this class is looking for - so the comparison divides it out before it compares. Of the + // corrections a ring carries this is the only one that varies along it; solid angle, detector + // and air absorption are all functions of 2 theta alone and the ring's own median absorbs them. + // The geometry is kept whole rather than reduced to a distance and a pixel size because the + // azimuth is the whole point: detector tilt, a quarter-turned image and an in-plane rotation + // all move the polarization plane across the stored image, and the geometry already knows where + // it lies. Only the centre is replaced, by the one the caller measured. + const DiffractionGeometry geometry; + const std::optional polarization; // unset leaves the background as it was measured + std::vector pixel_mask; // pixels already masked carry no background to test // Per-pixel projection over the frames added so far (converted geometry). One set per shard: diff --git a/tests/ShadowFinderTest.cpp b/tests/ShadowFinderTest.cpp index da4c5bd94..cfb489ff1 100644 --- a/tests/ShadowFinderTest.cpp +++ b/tests/ShadowFinderTest.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include "../common/DetectorSetup.h" @@ -279,3 +280,61 @@ TEST_CASE("ShadowFinder_TheRingsFollowTheCentreTheCallerNames", "[ShadowFinder]" CHECK(masked_at_measured < 2 * PATCH_PX); CHECK(masked_at_file > 3 * PATCH_PX); } + +// The background is not flat around a ring: a polarized source suppresses it in its own plane, by +// a factor that reaches four at the 2 theta a short detector distance puts in a corner. That is +// several times the dip this class looks for, so with the modulation left in, half of every outer +// ring reads as shadow with nothing in the beam. The scene here is exactly that and nothing else - +// a background carrying the Kahn factor of a horizontally polarized source, no hardware anywhere - +// and it must come back empty when the experiment states its polarization, and does not when it +// says nothing, which is what the correction is for. +TEST_CASE("ShadowFinder_APolarizedBackgroundIsNotAShadow", "[ShadowFinder]") { + // Short enough that this small detector reaches 2 theta = 70 degrees in its corner, where + // the source suppresses its own plane to a fifth. A real geometry gets there with a short + // crystal-to-detector distance and a large detector. + constexpr float DISTANCE_MM = 5.0f; + constexpr float POLARIZATION = 0.99f; + // Counts per pixel per frame across the azimuth the source does not suppress. Below + // MIN_REFLECTION, so no pixel of this scene is exempt as a recorded reflection. + constexpr int LEVEL = 24; + + DiffractionExperiment reference = TestExperiment(); + const float pixel_mm = reference.GetPixelSize_mm(); + + std::vector frame(static_cast(W) * H); + for (int y = 0; y < H; y++) + for (int x = 0; x < W; x++) { + const double dx = x - C, dy = y - C; + const double rho = std::hypot(dx, dy) * pixel_mm; + const double cos_2theta_sq = DISTANCE_MM * DISTANCE_MM + / (DISTANCE_MM * DISTANCE_MM + rho * rho); + const double rr = dx * dx + dy * dy; + const double cos_2phi = rr > 0.0 ? (dx * dx - dy * dy) / rr : 0.0; + const double factor = 0.5 * (1.0 + cos_2theta_sq + - POLARIZATION * cos_2phi * (1.0 - cos_2theta_sq)); + // Normalised per ring by the azimuth the source does not suppress, so what is left + // varies around a ring and not with radius - the ring median absorbs the rest. + const double at_peak = 0.5 * (1.0 + cos_2theta_sq + + POLARIZATION * (1.0 - cos_2theta_sq)); + frame[I(x, y)] = static_cast(std::lround(LEVEL * factor / at_peak)); + } + + auto mask_for = [&](const std::optional &polarization) { + DiffractionExperiment x = TestExperiment(); + x.DetectorDistance_mm(DISTANCE_MM).PolarizationFactor(polarization); + const PixelMask pixel_mask(x); + ShadowFinder finder(x, pixel_mask); + std::vector buffer; + for (int f = 0; f < NFRAMES; f++) { + DataMessage msg{}; + msg.image = CompressedImage(frame, W, H); + finder.AddImage(msg, buffer); + } + const auto mask = finder.GetMask(); + return std::count(mask.begin(), mask.end(), 1u); + }; + + CHECK(mask_for(POLARIZATION) == 0); + CHECK(mask_for(std::nullopt) > static_cast(W) * H / 20); +} +