Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 8m41s
Build Packages / build:windows:nocuda (push) Successful in 16m50s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 18m22s
Build Packages / build:windows:cuda (push) Successful in 19m40s
Build Packages / build:viewer-tgz:cpu (push) Successful in 21m2s
Build Packages / build:viewer-tgz:cuda (push) Successful in 22m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 23m21s
Build Packages / build:rugnux:windows (push) Successful in 10m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 27m37s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 27m50s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 19m52s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 22m6s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 25m47s
Build Packages / build:rpm (rocky9) (push) Successful in 23m52s
Build Packages / build:rpm (rocky8) (push) Successful in 26m33s
Build Packages / Generate python client (push) Successful in 45s
Build Packages / Build documentation (push) Successful in 1m16s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m57s
Build Packages / DIALS test (push) Successful in 25m7s
Build Packages / XDS test (durin plugin) (push) Successful in 11m18s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 27m24s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m58s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m43s
Build Packages / Unit tests (push) Successful in 1h22m26s
The self-calibrating finder was meant to replace the classic finder's FIXED PHOTON FLOOR with a per-resolution-ring threshold read off the image's own noise. As written it replaced the local-box SNR test as well, and that is the defect: a whole-ring threshold is an ABSOLUTE contour with no feedback from a pixel's own surroundings, so the area a spot puts above it grows as sigma^2*ln(peak/threshold) and never saturates. Measured on a strongly diffracting rotation set, the detected footprint grows by +8.05 pixels per e-fold of peak, so the brightest reflections came out as 100-500 pixel blobs and were then discarded for exceeding the size bound - every one of the ten strongest on an image. Intersecting with the local box gives -0.24 pixels per e-fold, the classic finder's own number to two decimals. WHY the local box is the right partner, rather than merely the incumbent: it is a prominence rule whose reference level is a 961-pixel mean. A spot inflates the box's own variance and the peak divides out of the acceptance test, so it cuts at a fixed FRACTION of the spot's own height. Referring that level to fewer pixels makes it inherit their shot noise - at FIXED footprint, estimating the level from 961 pixels, from 25, and from the single maximum gives centroid residuals of 0.524, 0.539 and 0.656 - so flat growth and a stable centroid turn out to be two ends of one dial. A contour on the bare maximum has the flattest growth of anything tried (+0.1) and merges worst. The two arms bind in different regimes, which is why intersecting beats choosing: on serial stills the ring threshold is 0.6x the classic floor, on this rotation sweep 2.3-6.0x. Stills are a strict no-op - 175 components against 175, identical per frame - so the +40% in stills indexing that the adaptive threshold was introduced for is untouched. What it buys, stated as one fact rather than two. Across five geometry pins spanning 1.1 mm it indexes the most frames of any arm tried, 0.831 against 0.803, and integrates 3.04 to 5.76% more observations - but those are the SAME number: regressing observation count on indexing rate over four arms leaves residuals of +/-0.7 percentage points against swings of -7 to +4.5%, so the extra observations ARE the extra indexed frames, not better data per frame. CC1/2, the only statistic here carrying per-observation quality, is +0.66 at one pin and -0.06 at the other: not harmed, not improved. <I/sigma>, ISa and R_meas cannot arbitrate on this data - across those pins each crosses zero as a monotone function of the pin. WHY an absolute contour indexes fewer frames, when its spot list is equal or better on every axis measured - recall, top-1000 recall, centroid, ice fraction, component count - is the interesting part, and it is not a detection effect at all: ITS OWN SIZE BOUND DELETES THE BRIGHTEST REFLECTIONS ON THE FRAME. A component is discarded because it grew past 200 px, and it grew past 200 px because it was bright, so the deletions are drawn from the head of the indexing budget rather than uniformly from it: they are 11x enriched in the top 250 of the thousand spots handed to the indexer, and the bound's own real deletions sit at MEDIAN RANK 12. Turning the bound off recovers 66% and 50% of the deficit at the two pins, against a bar registered at 33% before the run. Three of us dismissed this for most of a day on the grounds that the gates delete only ~4% of what is detected. That arithmetic was right and the denominator was wrong - a rate is not an impact when the thing being lost is selected for the property that makes it matter. Reworking the bound instead was measured and rejected: it recovers half the deficit, and it cannot be done without re-admitting what the bound is for - 68 components past 200 px, of which 8 are real and 60 are junk, where the intersect gets the 8 without the 60. The residual once the bound is off, +1.08%/+1.70%, is the contour itself. Component merging is ruled out separately: geometrically impossible here, 33.9 px minimum reflection separation against components spanning 10 px. So is a ranking effect - the intersect's lead runs +0.06% at --max-spots 250, +3.46% at 1000 and +14.26% at 2000, which is backwards for a selection artefact. Costs 0.48 ms per image in the finder, and 0.044 px of bright-spot centroid precision - measured convention-free, by fitting a line to a reflection's own centroid across five frames, after an XDS-referenced figure proved to be four fifths aperture convention. It also makes the compactness gate above it safe. On the absolute contour that gate is net damage, deleting 37 genuine reflections per ten frames; once the footprint stops growing nothing reaches its threshold at all. Also fixes a real but unexercised defect in PoissonThreshold, where the exact tail handed over to a normal approximation with a step. It changes nothing here: the clipped ring sigma is over-dispersed 1.2-4.9x against sqrt(mu) because it still contains diffraction, so the Gaussian arm wins every ring above mu=50 and none of the 522 thresholds move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H
255 lines
11 KiB
C++
255 lines
11 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include <cmath>
|
|
#include <algorithm>
|
|
|
|
#include <catch2/catch_all.hpp>
|
|
|
|
#include "../common/AzimuthalIntegrationMapping.h"
|
|
#include "../image_analysis/spot_finding/AdaptiveSpotFinderCPU.h"
|
|
|
|
namespace {
|
|
|
|
SpotFindingSettings AdaptiveSettings() {
|
|
SpotFindingSettings s{};
|
|
s.adaptive_threshold = true;
|
|
s.false_pixels_per_frame = 100.0f;
|
|
s.min_pix_per_spot = 1;
|
|
s.max_pix_per_spot = 50;
|
|
s.high_resolution_limit = 0.0f;
|
|
s.low_resolution_limit = 1.0e6f;
|
|
s.high_res_gap_Q_recipA = std::nullopt;
|
|
return s;
|
|
}
|
|
|
|
} // namespace
|
|
|
|
// Raw (untransformed) geometry: the mapping is built over the raw module layout, which is smaller
|
|
// than the converted image. The finder has to walk the raw image, so a spot planted at a raw pixel
|
|
// comes back at that pixel.
|
|
TEST_CASE("AdaptiveSpotFinderCPU_RawGeometry", "[AdaptiveSpotFinder]") {
|
|
DiffractionExperiment x(DetJF4M());
|
|
x.DetectorDistance_mm(80).BeamX_pxl(1030).BeamY_pxl(1080);
|
|
x.QSpacingForAzimInt_recipA(0.05).QRangeForAzimInt_recipA(0.05, 5.0);
|
|
x.GeometryTransformation(false);
|
|
|
|
PixelMask pixel_mask(x);
|
|
AzimuthalIntegrationMapping mapping(x, pixel_mask);
|
|
|
|
const size_t w = x.GetXPixelsNum();
|
|
const size_t h = x.GetYPixelsNum();
|
|
REQUIRE(w * h == mapping.GetPixelToBin().size());
|
|
REQUIRE(w * h < static_cast<size_t>(x.GetPixelsNumConv()));
|
|
|
|
ImagePreprocessorBuffer buffer(x.GetPixelsNum());
|
|
for (size_t i = 0; i < w * h; i++)
|
|
buffer[i] = 8 + static_cast<int32_t>(i % 5); // background 8..12
|
|
|
|
// A 3x3 blob on a pixel that has a ring - with all of its neighbours on one too.
|
|
const auto &pixel_to_bin = mapping.GetPixelToBin();
|
|
size_t spot_row = 0, spot_col = 0;
|
|
for (size_t row = 100; row < h - 100 && spot_row == 0; row++) {
|
|
for (size_t col = 100; col < w - 100; col++) {
|
|
bool all_binned = true;
|
|
for (int dr = -1; dr <= 1; dr++)
|
|
for (int dc = -1; dc <= 1; dc++)
|
|
all_binned &= pixel_to_bin[(row + dr) * w + col + dc] != UINT16_MAX;
|
|
if (all_binned) {
|
|
spot_row = row;
|
|
spot_col = col;
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
REQUIRE(spot_row > 0);
|
|
|
|
for (int dr = -1; dr <= 1; dr++)
|
|
for (int dc = -1; dc <= 1; dc++)
|
|
buffer[(spot_row + dr) * w + spot_col + dc] = 200;
|
|
|
|
std::vector<bool> res_mask(x.GetPixelsNum(), false);
|
|
AdaptiveSpotFinderCPU finder(mapping);
|
|
finder.SetResolutionMask(res_mask);
|
|
const auto spots = finder.Run(buffer, AdaptiveSettings());
|
|
|
|
REQUIRE(spots.size() == 1);
|
|
CHECK(std::lround(spots[0].RawCoord().x) == static_cast<long>(spot_col));
|
|
CHECK(std::lround(spots[0].RawCoord().y) == static_cast<long>(spot_row));
|
|
}
|
|
|
|
// The property the whole engine exists for: the threshold comes from the image's OWN noise, so the
|
|
// same settings behave the same way on a frame whose background is ten times higher. A frame is built
|
|
// with background spread S around a mean, one pixel planted a few S above it (must stay unfound) and
|
|
// one planted far above (must be found); then the identical frame scaled by ten must give the identical
|
|
// answer. Any threshold that does not track the background - a constant, or one that drops the sigma
|
|
// term - finds the weak pixel in the scaled frame, or loses the strong one.
|
|
TEST_CASE("AdaptiveSpotFinderCPU_ThresholdTracksBackground", "[AdaptiveSpotFinder]") {
|
|
DiffractionExperiment x(DetJF4M());
|
|
x.DetectorDistance_mm(80).BeamX_pxl(1030).BeamY_pxl(1080);
|
|
x.QSpacingForAzimInt_recipA(0.05).QRangeForAzimInt_recipA(0.05, 5.0);
|
|
x.GeometryTransformation(false);
|
|
|
|
PixelMask pixel_mask(x);
|
|
AzimuthalIntegrationMapping mapping(x, pixel_mask);
|
|
const auto &pixel_to_bin = mapping.GetPixelToBin();
|
|
|
|
const size_t w = x.GetXPixelsNum();
|
|
const size_t h = x.GetYPixelsNum();
|
|
|
|
// Two well-separated pixels that carry a ring, so both are seen by the finder.
|
|
std::vector<size_t> planted;
|
|
for (size_t row = 300; row < h - 300 && planted.size() < 2; row += 137)
|
|
for (size_t col = 300; col < w - 300; col += 149)
|
|
if (pixel_to_bin[row * w + col] != UINT16_MAX) {
|
|
planted.push_back(row * w + col);
|
|
break;
|
|
}
|
|
REQUIRE(planted.size() == 2);
|
|
|
|
// Background takes 5 evenly spaced levels one S apart, i.e. mean + 2S and sigma = sqrt(2) S. With
|
|
// ~100 expected noise pixels per frame the cut lands near mean + 4.1 sigma = mean + 5.8 S.
|
|
const auto run_at_scale = [&](int32_t scale) {
|
|
ImagePreprocessorBuffer buffer(x.GetPixelsNum());
|
|
for (size_t i = 0; i < w * h; i++)
|
|
buffer[i] = scale * (10 + static_cast<int32_t>(i % 5));
|
|
buffer[planted[0]] = scale * (10 + 5); // mean + 3 S: below the cut
|
|
buffer[planted[1]] = scale * (10 + 30); // mean + 28 S: well above it
|
|
std::vector<bool> res_mask(x.GetPixelsNum(), false);
|
|
AdaptiveSpotFinderCPU finder(mapping);
|
|
finder.SetResolutionMask(res_mask);
|
|
return finder.Run(buffer, AdaptiveSettings());
|
|
};
|
|
|
|
const auto plain = run_at_scale(1);
|
|
const auto scaled = run_at_scale(10);
|
|
|
|
REQUIRE(plain.size() == 1);
|
|
CHECK(std::lround(plain[0].RawCoord().x) == static_cast<long>(planted[1] % w));
|
|
CHECK(std::lround(plain[0].RawCoord().y) == static_cast<long>(planted[1] / w));
|
|
|
|
// Ten times the background, ten times the noise, ten times the signal - same answer.
|
|
REQUIRE(scaled.size() == plain.size());
|
|
CHECK(std::lround(scaled[0].RawCoord().x) == std::lround(plain[0].RawCoord().x));
|
|
CHECK(std::lround(scaled[0].RawCoord().y) == std::lround(plain[0].RawCoord().y));
|
|
}
|
|
|
|
// The per-ring background the finder hands out is the peak-EXCLUDED one: a handful of very bright
|
|
// pixels planted on a ring must not move it, which is the property that lets the ice score be read
|
|
// off it instead of off the plain azimuthal profile. Rings with too few pixels to be their own
|
|
// background come back as NaN rather than as a stale value from the previous frame.
|
|
TEST_CASE("AdaptiveSpotFinderCPU_RingBackgroundExcludesPeaks", "[AdaptiveSpotFinder]") {
|
|
DiffractionExperiment x(DetJF4M());
|
|
x.DetectorDistance_mm(80).BeamX_pxl(1030).BeamY_pxl(1080);
|
|
x.QSpacingForAzimInt_recipA(0.05).QRangeForAzimInt_recipA(0.05, 5.0);
|
|
x.GeometryTransformation(false);
|
|
|
|
PixelMask pixel_mask(x);
|
|
AzimuthalIntegrationMapping mapping(x, pixel_mask);
|
|
const auto &pixel_to_bin = mapping.GetPixelToBin();
|
|
|
|
const size_t w = x.GetXPixelsNum();
|
|
const size_t h = x.GetYPixelsNum();
|
|
|
|
ImagePreprocessorBuffer buffer(x.GetPixelsNum());
|
|
for (size_t i = 0; i < w * h; i++)
|
|
buffer[i] = 8 + static_cast<int32_t>(i % 5); // background 8..12
|
|
|
|
std::vector<bool> res_mask(x.GetPixelsNum(), false);
|
|
AdaptiveSpotFinderCPU finder(mapping);
|
|
finder.SetResolutionMask(res_mask);
|
|
|
|
finder.Detect(buffer, AdaptiveSettings());
|
|
const std::vector<float> clean = finder.GetRingBackground();
|
|
REQUIRE(clean.size() == mapping.GetBinNumber());
|
|
|
|
std::vector<int64_t> pixels_per_bin(mapping.GetBinNumber(), 0);
|
|
for (size_t i = 0; i < w * h; i++)
|
|
if (pixel_to_bin[i] < mapping.GetBinNumber())
|
|
pixels_per_bin[pixel_to_bin[i]]++;
|
|
|
|
// Plant 200 bright pixels spread over one well-populated ring - far more than a real spot, so a
|
|
// plain mean would move visibly.
|
|
const uint16_t ring = static_cast<uint16_t>(
|
|
std::max_element(pixels_per_bin.begin(), pixels_per_bin.end()) - pixels_per_bin.begin());
|
|
REQUIRE(pixels_per_bin[ring] > 10000);
|
|
int planted = 0;
|
|
for (size_t i = 0; i < w * h && planted < 200; i++)
|
|
if (pixel_to_bin[i] == ring) {
|
|
buffer[i] = 100000;
|
|
planted++;
|
|
}
|
|
REQUIRE(planted == 200);
|
|
|
|
finder.Detect(buffer, AdaptiveSettings());
|
|
const std::vector<float> spiked = finder.GetRingBackground();
|
|
|
|
REQUIRE(std::isfinite(clean[ring]));
|
|
REQUIRE(std::isfinite(spiked[ring]));
|
|
CHECK(spiked[ring] == Catch::Approx(clean[ring]).epsilon(0.01));
|
|
|
|
for (size_t b = 0; b < clean.size(); b++)
|
|
CHECK(std::isfinite(clean[b]) == (pixels_per_bin[b] >= 40));
|
|
}
|
|
|
|
// The footprint of a bright reflection must not grow with its brightness. A whole-ring threshold is
|
|
// an absolute contour, so the area a Gaussian puts above it grows as sigma^2 ln(peak/threshold) - the
|
|
// same spot detected a hundred times brighter comes back tens of pixels larger, and an upper bound on
|
|
// spot size becomes an upper bound on spot INTENSITY. Intersecting with the local-box test removes
|
|
// that: the spot inflates the box's own variance, the peak divides out of the acceptance test, and the
|
|
// contour lands at a fixed fraction of the spot's own height whatever that height is.
|
|
TEST_CASE("AdaptiveSpotFinderCPU_FootprintDoesNotGrowWithBrightness", "[AdaptiveSpotFinder]") {
|
|
DiffractionExperiment x(DetJF4M());
|
|
x.DetectorDistance_mm(80).BeamX_pxl(1030).BeamY_pxl(1080);
|
|
x.QSpacingForAzimInt_recipA(0.05).QRangeForAzimInt_recipA(0.05, 5.0);
|
|
x.GeometryTransformation(false);
|
|
|
|
PixelMask pixel_mask(x);
|
|
AzimuthalIntegrationMapping mapping(x, pixel_mask);
|
|
const auto &pixel_to_bin = mapping.GetPixelToBin();
|
|
|
|
const size_t w = x.GetXPixelsNum();
|
|
const size_t h = x.GetYPixelsNum();
|
|
|
|
size_t spot_row = 0, spot_col = 0;
|
|
for (size_t row = 400; row < h - 400 && spot_row == 0; row++)
|
|
for (size_t col = 400; col < w - 400; col++)
|
|
if (pixel_to_bin[row * w + col] != UINT16_MAX) {
|
|
spot_row = row;
|
|
spot_col = col;
|
|
break;
|
|
}
|
|
REQUIRE(spot_row > 0);
|
|
|
|
std::vector<bool> res_mask(x.GetPixelsNum(), false);
|
|
auto settings = AdaptiveSettings();
|
|
settings.min_pix_per_spot = 2;
|
|
settings.max_pix_per_spot = 100000; // no bound, so the footprint itself is what is measured
|
|
|
|
// One Gaussian of width 1.5 px on a flat background of 10, at three amplitudes a hundred apart.
|
|
auto footprint = [&](double amplitude) {
|
|
ImagePreprocessorBuffer buffer(x.GetPixelsNum());
|
|
for (size_t i = 0; i < w * h; i++)
|
|
buffer[i] = 10;
|
|
constexpr double sigma = 1.5;
|
|
for (int dr = -12; dr <= 12; dr++)
|
|
for (int dc = -12; dc <= 12; dc++) {
|
|
const double r2 = dr * dr + dc * dc;
|
|
buffer[(spot_row + dr) * w + spot_col + dc] =
|
|
10 + static_cast<int32_t>(amplitude * std::exp(-r2 / (2 * sigma * sigma)));
|
|
}
|
|
AdaptiveSpotFinderCPU finder(mapping);
|
|
finder.SetResolutionMask(res_mask);
|
|
const auto spots = finder.Run(buffer, settings);
|
|
REQUIRE(spots.size() == 1);
|
|
return spots[0].PixelCount();
|
|
};
|
|
|
|
const int64_t small = footprint(300.0);
|
|
const int64_t large = footprint(30000.0);
|
|
CHECK(small > 0);
|
|
// A hundredfold in peak is 4.6 e-folds. An absolute contour would add sigma^2 ln(100) ~ 10 pixels
|
|
// per e-fold of AREA here, tens of pixels in all; a peak-relative one adds nothing.
|
|
CHECK(large - small <= 4);
|
|
}
|