Files
Jungfraujoch/image_analysis/beam_stop/ShadowFinder.cpp
T
leonarski_f 67dca388bd
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
v1.0.0-rc.160 (#70)
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-19 09:39:28 +02:00

339 lines
14 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "ShadowFinder.h"
#include <algorithm>
#include <cmath>
#include <limits>
#include <queue>
#include <type_traits>
#include "../../common/JFJochException.h"
// ---------------------------------------------------------------------------------
// Small binary-image helpers on a width*height frame stored row-major as char (0/1).
// All run once, at GetMask() time. The BFS forms keep them O(pixels) rather than
// O(pixels * radius), so a radius-14 dilation is still a single sweep.
// ---------------------------------------------------------------------------------
namespace {
// 8-connected dilation by `r` pixels (Chebyshev), via a multi-source BFS.
std::vector<char> Dilate(const std::vector<char> &in, int W, int H, int r) {
if (r <= 0)
return in;
std::vector<int> dist(in.size(), -1);
std::queue<int> q;
for (size_t i = 0; i < in.size(); i++)
if (in[i]) { dist[i] = 0; q.push(static_cast<int>(i)); }
while (!q.empty()) {
const int i = q.front(); q.pop();
if (dist[i] >= r)
continue;
const int y = i / W, x = i % W;
for (int dy = -1; dy <= 1; dy++)
for (int dx = -1; dx <= 1; dx++) {
const int yy = y + dy, xx = x + dx;
if (yy < 0 || yy >= H || xx < 0 || xx >= W)
continue;
const int j = yy * W + xx;
if (dist[j] < 0) { dist[j] = dist[i] + 1; q.push(j); }
}
}
std::vector<char> out(in.size());
for (size_t i = 0; i < out.size(); i++)
out[i] = (dist[i] >= 0) ? 1 : 0;
return out;
}
// Erosion by `r` = dilation of the complement (image border counts as outside).
std::vector<char> Erode(const std::vector<char> &in, int W, int H, int r) {
std::vector<char> comp(in.size());
for (size_t i = 0; i < in.size(); i++)
comp[i] = !in[i];
const auto grown = Dilate(comp, W, H, r);
std::vector<char> out(in.size());
for (size_t i = 0; i < out.size(); i++)
out[i] = !grown[i];
return out;
}
// Pixels of `passable` reachable from any of `seeds` (8-connected flood).
std::vector<char> Flood(const std::vector<char> &passable, int W, int H, const std::vector<int> &seeds) {
std::vector<char> visited(passable.size(), 0);
std::queue<int> q;
for (const int s : seeds)
if (s >= 0 && s < static_cast<int>(passable.size()) && passable[s] && !visited[s]) {
visited[s] = 1; q.push(s);
}
while (!q.empty()) {
const int i = q.front(); q.pop();
const int y = i / W, x = i % W;
for (int dy = -1; dy <= 1; dy++)
for (int dx = -1; dx <= 1; dx++) {
const int yy = y + dy, xx = x + dx;
if (yy < 0 || yy >= H || xx < 0 || xx >= W)
continue;
const int j = yy * W + xx;
if (passable[j] && !visited[j]) { visited[j] = 1; q.push(j); }
}
}
return visited;
}
// Fill holes: background not reachable from the image border becomes region.
std::vector<char> FillHoles(const std::vector<char> &region, int W, int H) {
std::vector<char> bg_visited(region.size(), 0);
std::queue<int> q;
auto push = [&](int i) { if (!region[i] && !bg_visited[i]) { bg_visited[i] = 1; q.push(i); } };
for (int x = 0; x < W; x++) { push(x); push((H - 1) * W + x); }
for (int y = 0; y < H; y++) { push(y * W); push(y * W + W - 1); }
while (!q.empty()) {
const int i = q.front(); q.pop();
const int y = i / W, x = i % W;
for (int dy = -1; dy <= 1; dy++)
for (int dx = -1; dx <= 1; dx++) {
const int yy = y + dy, xx = x + dx;
if (yy < 0 || yy >= H || xx < 0 || xx >= W)
continue;
const int j = yy * W + xx;
if (!region[j] && !bg_visited[j]) { bg_visited[j] = 1; q.push(j); }
}
}
std::vector<char> out = region;
for (size_t i = 0; i < out.size(); i++)
if (!region[i] && !bg_visited[i])
out[i] = 1;
return out;
}
// Median of `values` per integer radius, over the pixels flagged in `use`.
std::vector<float> RingMedian(const std::vector<float> &values, const std::vector<char> &use,
const std::vector<int> &radius, int max_radius) {
std::vector<std::vector<float>> bins(max_radius + 1);
for (size_t i = 0; i < values.size(); i++)
if (use[i])
bins[radius[i]].push_back(values[i]);
std::vector<float> median(max_radius + 1, 0.0f);
for (int r = 0; r <= max_radius; r++) {
auto &b = bins[r];
if (!b.empty()) {
const size_t k = b.size() / 2;
std::nth_element(b.begin(), b.begin() + k, b.end());
median[r] = b[k];
}
}
return median;
}
// Fraction of each integer-radius ring that is flagged in `blocked`.
std::vector<float> RingFraction(const std::vector<char> &blocked, const std::vector<int> &radius, int max_radius) {
std::vector<int64_t> num(max_radius + 1, 0), den(max_radius + 1, 0);
for (size_t i = 0; i < blocked.size(); i++) {
den[radius[i]]++;
if (blocked[i]) num[radius[i]]++;
}
std::vector<float> frac(max_radius + 1, 0.0f);
for (int r = 0; r <= max_radius; r++)
frac[r] = den[r] ? static_cast<float>(num[r]) / static_cast<float>(den[r]) : 0.0f;
return frac;
}
} // namespace
// ---------------------------------------------------------------------------------
ShadowFinder::ShadowFinder(const DiffractionExperiment &experiment, ShadowFinderSettings in_settings)
: width(static_cast<int>(experiment.GetXPixelsNumConv())),
height(static_cast<int>(experiment.GetYPixelsNumConv())),
beam_x(experiment.GetBeamX_pxl()),
beam_y(experiment.GetBeamY_pxl()),
settings(in_settings),
max_value(static_cast<size_t>(width) * height, 0),
sum_value(static_cast<size_t>(width) * height, 0),
valid_count(static_cast<size_t>(width) * height, 0) {}
template<class T>
void ShadowFinder::Add(const T *ptr) {
// The pixel type's sentinel extreme marks "no data" (module gap / masked): the
// preprocessor/writer stores INT*_MIN for signed and UINT*_MAX for unsigned. For
// signed types the opposite extreme (INT*_MAX) is a genuine saturated value and is
// kept, so a saturated reflection still registers as bright.
T masked;
if constexpr (std::is_signed_v<T>)
masked = std::numeric_limits<T>::min();
else
masked = std::numeric_limits<T>::max();
std::unique_lock ul(m);
for (size_t i = 0; i < max_value.size(); i++) {
const T v = ptr[i];
if (v == masked)
continue;
const int32_t vi = static_cast<int32_t>(v);
if (valid_count[i] == 0 || vi > max_value[i])
max_value[i] = vi;
sum_value[i] += vi;
valid_count[i]++;
}
frames++;
}
void ShadowFinder::AddImage(const DataMessage &data, std::vector<uint8_t> buffer) {
if (static_cast<size_t>(data.image.GetWidth()) * data.image.GetHeight() != max_value.size())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"ShadowFinder: image size does not match the detector");
const auto ptr = data.image.GetUncompressedPtr(buffer);
switch (data.image.GetMode()) {
case CompressedImageMode::Int8: Add(reinterpret_cast<const int8_t *>(ptr)); break;
case CompressedImageMode::Uint8: Add(reinterpret_cast<const uint8_t *>(ptr)); break;
case CompressedImageMode::Int16: Add(reinterpret_cast<const int16_t *>(ptr)); break;
case CompressedImageMode::Uint16: Add(reinterpret_cast<const uint16_t *>(ptr)); break;
case CompressedImageMode::Int32: Add(reinterpret_cast<const int32_t *>(ptr)); break;
case CompressedImageMode::Uint32: Add(reinterpret_cast<const uint32_t *>(ptr)); break;
default:
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"ShadowFinder: unsupported image mode");
}
}
uint32_t ShadowFinder::GetFrameCount() const {
std::unique_lock ul(m);
return frames;
}
std::vector<uint32_t> ShadowFinder::GetMask() const {
std::unique_lock ul(m);
const int W = width, H = height;
const int N = W * H;
const ShadowFinderSettings &S = settings;
std::vector<uint32_t> mask(N, 0);
if (frames == 0)
return mask;
// --- mean projection, per-pixel validity and radius from the beam centre ---
std::vector<float> mean(N, 0.0f);
std::vector<char> valid(N, 0);
std::vector<int> radius(N, 0);
int max_radius = 0;
for (int y = 0; y < H; y++)
for (int x = 0; x < W; x++) {
const int i = y * W + x;
if (valid_count[i] > 0) {
mean[i] = static_cast<float>(static_cast<double>(sum_value[i]) / valid_count[i]);
valid[i] = 1;
}
const double dx = x - beam_x, dy = y - beam_y;
const int r = static_cast<int>(std::lround(std::sqrt(dx * dx + dy * dy)));
radius[i] = r;
if (r > max_radius) max_radius = r;
}
// --- robust radial baseline; iterate to keep the shadow out of its own baseline ---
std::vector<float> ratio(N, 1.0f);
std::vector<char> excluded(N, 0);
for (int iter = 0; iter < 3; iter++) {
std::vector<char> use(N);
for (int i = 0; i < N; i++)
use[i] = valid[i] && !excluded[i];
const auto baseline = RingMedian(mean, use, radius, max_radius);
for (int i = 0; i < N; i++)
if (valid[i])
ratio[i] = mean[i] / std::max(baseline[radius[i]], 1e-6f);
for (int i = 0; i < N; i++)
excluded[i] = valid[i] && ratio[i] < S.shadow_ratio;
}
// --- shadow core: low-ratio pixels connected to the beam centre (bridging gaps) ---
std::vector<char> low(N);
for (int i = 0; i < N; i++)
low[i] = valid[i] && ratio[i] < S.shadow_ratio;
const std::vector<char> grown = Dilate(low, W, H, S.bridge_px);
std::vector<int> seeds; // a small disk at the beam centre
for (int y = 0; y < H; y++)
for (int x = 0; x < W; x++) {
const double dx = x - beam_x, dy = y - beam_y;
if (dx * dx + dy * dy < 4.0 * 4.0)
seeds.push_back(y * W + x);
}
const std::vector<char> connected = Flood(grown, W, H, seeds);
std::vector<char> core(N);
for (int i = 0; i < N; i++)
core[i] = low[i] && connected[i];
// --- real reflections: any pixel that recorded signal is never masked. Require a
// small cluster so a single-frame zinger does not count as a reflection. ---
std::vector<char> lit(N, 0);
for (int i = 0; i < N; i++)
lit[i] = (valid_count[i] > 0) && (max_value[i] >= static_cast<int32_t>(S.min_reflection));
std::vector<char> reflection(N, 0);
for (int y = 0; y < H; y++)
for (int x = 0; x < W; x++) {
const int i = y * W + x;
if (!lit[i]) continue;
int neighbours = 0;
for (int dy = -1; dy <= 1; dy++)
for (int dx = -1; dx <= 1; dx++) {
const int yy = y + dy, xx = x + dx;
if ((dx || dy) && yy >= 0 && yy < H && xx >= 0 && xx < W && lit[yy * W + xx])
neighbours++;
}
reflection[i] = (neighbours >= 2);
}
// --- central low-res disk: the fully-blocked region about the beam centre. Sized by
// the azimuthal blocked fraction (a disk blocks ~every azimuth; a thin arm or
// gap does not), and capped just inside the innermost reflection. ---
std::vector<char> blocked(N);
for (int i = 0; i < N; i++)
blocked[i] = (valid_count[i] == 0) || low[i];
const auto blocked_frac = RingFraction(blocked, radius, max_radius);
int disk_radius = 0;
{
float head = 0.0f; int head_n = 0;
for (int r = 0; r <= std::min(5, max_radius); r++) { head += blocked_frac[r]; head_n++; }
if (head_n > 0 && head / head_n >= 0.65f) { // the beam centre is behind a disk
disk_radius = max_radius;
for (int r = 1; r <= max_radius; r++)
if (blocked_frac[r] < 0.55f) { disk_radius = r; break; }
}
}
int reflection_radius = max_radius + 1; // innermost reflection (ignore the very centre)
for (int i = 0; i < N; i++)
if (reflection[i] && radius[i] > 12 && radius[i] < reflection_radius)
reflection_radius = radius[i];
if (disk_radius > reflection_radius - 4)
disk_radius = reflection_radius - 4;
if (disk_radius < 0)
disk_radius = 0;
// --- assemble: core + disk, grow the soft penumbra, round, fill the disk interior ---
std::vector<char> region(N);
for (int i = 0; i < N; i++)
region[i] = core[i] || (disk_radius > 0 && radius[i] < disk_radius);
const std::vector<char> near = Dilate(region, W, H, S.penumbra_max_px);
for (int i = 0; i < N; i++)
if (near[i] && valid[i] && ratio[i] < S.penumbra_ratio)
region[i] = 1;
region = Erode(Dilate(region, W, H, 2), W, H, 2); // close: round the boundary
region = FillHoles(region, W, H);
// Expose recorded reflections - done last, with no fill afterwards, so a spot the
// geometry still covered is given back rather than re-enclosed.
const std::vector<char> reflection_grown = Dilate(reflection, W, H, 1);
for (int i = 0; i < N; i++)
if (reflection_grown[i])
region[i] = 0;
for (int i = 0; i < N; i++)
mask[i] = region[i] ? 1 : 0;
return mask;
}