mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 67 Wiki Activity

Azimuthal integration: optional sigma clipping of the reported profile

Browse Source 
The profile is the MEAN of each bin, so a few strong reflections landing in a bin
lift it exactly as a smooth powder ring does. That is the wrong quantity whenever
the profile is wanted as a background rather than as a measurement of what is in
the bin - the ice score being the case in point, where reading a plain profile
INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals
had no ice at all.

The adaptive spot finder already computes the right thing, a sigma-clipped
per-resolution-ring background, as a byproduct of its own threshold. Where it
runs, the ice score uses that. Where it does not - --no-adaptive-spots,
--azint-only, and anything reading the profile the broker wrote - there was no way
to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip),
0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian
bin and biases the estimate low rather than removing outliers.

Two clip passes follow the plain one, matching the finder's recipe - the first
pass's standard deviation is itself inflated by the peaks being removed, so one
pass leaves a threshold that is still too generous. A bin with fewer than eight
pixels is left alone: at the detector edge and behind the beam stop there is no
spread to clip on.

Both engines do it. On the GPU the accept range is computed by a small kernel and
stays resident, so a clip pass is one more read of the same pixels and no round
trip; the two accumulation kernels take the range as a pointer that is null on the
plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal
integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict,
against a 0.34 ms per-image total.

Note what the result IS: the smooth background under the peaks, not the bin mean.
It should not be switched on where a ring's integrated intensity is wanted - the
powder-ring geometry fit reads ring peaks, and those are what a clip is designed
to remove. Off by default, so nothing changes unless it is asked for.

Not exposed over the REST API - that needs the generated model regenerated, which
is a separate step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-08-07 00:18:52 +02:00
co-authored by Claude Opus 5
parent 3d4209d803
commit a6be35ccdb
24 changed files with 284 additions and 432 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/scale_merge/RotationScaleMerge.cpp
+14 -31
View File
@@ -165,7 +165,7 @@ namespace {
RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment,
std::vector<IntegrationOutcome> &partial_outcomes,
std::optional<UnitCell> reference_cell,
int scaling_iterations, float ice_ring_half_width_q,
int scaling_iterations,
size_t nthreads, Logger &logger,
std::string observation_dump_path)
: x(experiment), partials_out(partial_outcomes), reference_cell(std::move(reference_cell)),
@@ -197,7 +197,6 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment,
mosaicity_deg = *forced;
else
mosaicity_deg = s.GetDefaultMosaicity();
ice_half_width_q = ice_ring_half_width_q;
}
void RotationScaleMerge::Ingest() {
@@ -682,7 +681,6 @@ int RotationScaleMerge::ComputeAsuGroups(const HKLKeyGenerator &keygen) {
}
void RotationScaleMerge::ReduceGroupMeans(const std::vector<Obs> &obs, int n_groups,
bool exclude_ice, const std::vector<char> &masked,
std::vector<double> &out_mean) const {
// Inverse-variance per-group mean of I*corr = the merge reference (a segmented reduction over the
// groups; the CPU stand-in for a CUDA reduce_by_key). No cell mask here: the scaling reference
@@ -691,11 +689,6 @@ void RotationScaleMerge::ReduceGroupMeans(const std::vector<Obs> &obs, int n_gro
for (const auto &o : obs) {
if (o.group < 0) continue;
if (!(o.corr > 0.0f) || !std::isfinite(o.corr)) continue;
if (exclude_ice && o.on_ice) continue;
if (!masked.empty()) {
const int ring = IceRingIndex(o.d, ice_half_width_q);
if (ring >= 0 && ring < static_cast<int>(masked.size()) && masked[ring]) continue;
}
if (o.partiality < min_partiality) continue;
const float I_corr = o.I * o.corr;
const float sigma_corr = o.sigma * o.corr;
@@ -1649,21 +1642,14 @@ namespace {
}
RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool for_search,
const std::vector<char> &masked,
bool fulls_resident) {
// A full is usable for the merge / error model if it passes AddImage's filters (with the current
// ice/masked-ring context). group >= 0 already encodes "not absent and passes AcceptReflection".
auto masked_ring = [&](const Obs &o) {
if (masked.empty()) return false;
const int ring = IceRingIndex(o.d, ice_half_width_q);
return ring >= 0 && ring < static_cast<int>(masked.size()) && masked[ring];
};
// ice context). group >= 0 already encodes "not absent and passes AcceptReflection".
auto usable_merge = [&](const Obs &o) {
if (o.group < 0) return false;
if (!frame_cell_ok[o.frame]) return false;
if (!(o.corr > 0.0f) || !std::isfinite(o.corr)) return false;
if (for_search && o.on_ice) return false;
if (masked_ring(o)) return false;
if (o.partiality < min_partiality) return false;
const float I_corr = o.I * o.corr, sigma_corr = o.sigma * o.corr;
return std::isfinite(I_corr) && std::isfinite(sigma_corr) && sigma_corr > 0.0f;
@@ -1676,7 +1662,6 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
bool use_gpu_merge = false;
#ifdef JFJOCH_USE_CUDA
use_gpu_merge = fulls_resident && !fulls.empty();
std::vector<uint8_t> gpu_masked(masked.begin(), masked.end());
#endif
// ---- Error model: fit dev2 = a*sigma^2 + b^2*<I>^2 from symmetry-equivalent scatter. ----
@@ -1694,8 +1679,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
const int nf = static_cast<int>(fulls.size());
std::vector<double> gs2(nf), gI2(nf), gdev2(nf);
std::vector<uint8_t> gvalid(nf);
gpu_->MergeEmSamples(for_search, gpu_masked.data(), static_cast<int>(gpu_masked.size()),
ice_half_width_q, min_partiality, em_mean.data(), cnt.data(),
gpu_->MergeEmSamples(for_search, min_partiality, em_mean.data(), cnt.data(),
gs2.data(), gI2.data(), gdev2.data(), gvalid.data());
samples.reserve(nf);
for (int i = 0; i < nf; ++i)
@@ -2002,7 +1986,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
}
}
// R_meas: re-walk the fulls (Mask = cell only; no ice / masked-ring / error-model), accumulate
// R_meas: re-walk the fulls (Mask = cell only; no ice / error-model), accumulate
// |I_i - <I>| per reflection.
struct RmeasObs { double sum_abs_dev = 0, sum_I = 0; int n = 0, shell = -1; };
std::vector<RmeasObs> rmeas(n_groups);
@@ -2018,11 +2002,11 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
const auto shell = shells.GetShell(acc[g].d);
if (!shell || *shell < 0 || *shell >= n_shells) continue;
// Count the MERGED population, not the R_meas one. The R_meas re-walk deliberately
// ignores the ring mask (and, on a search pass, the ice flag), so its count includes
// observations that never entered `unique` - which inflates the reported multiplicity
// of whatever shell they land in. acc[g].nh is what actually went into this group's
// mean, and it is zero for a masked group. (acc[g].d is NaN for such a group, so
// GetShell above already declines it; this is the same statement made where it counts.)
// ignores the ice flag on a search pass, so its count includes observations that never
// entered `unique` - which inflates the reported multiplicity of whatever shell they
// land in. acc[g].nh is what actually went into this group's mean, and it is zero for a
// group the merge dropped entirely. (acc[g].d is NaN for such a group, so GetShell above
// already declines it; this is the same statement made where it counts.)
sa[*shell].total_obs += static_cast<int>(acc[g].nh[0] + acc[g].nh[1]);
if (std::isfinite(merged_I[g]) && rn[g] > 0) {
auto &r = rmeas[g];
@@ -2173,8 +2157,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
return result;
}
RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
const std::vector<char> &masked_ice_rings) {
RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search) {
const int sg_number = x.GetSpaceGroupNumber().value_or(1);
HKLKeyGenerator keygen(merge_friedel, sg_number);
@@ -2208,7 +2191,7 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
#endif
if (!scaled_on_gpu) {
for (int it = 0; it < scaling_iter; ++it) {
ReduceGroupMeans(partials, n_groups, false, {}, partial_mean);
ReduceGroupMeans(partials, n_groups, partial_mean);
FitPerFrameG(partials, frame_start, frame_count, partial_mean, /*unity=*/false, g_partial);
UpdateCorr(partials, g_partial, frame_scaled_scratch);
}
@@ -2297,7 +2280,7 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
}
#endif
if (!cc_on_gpu) {
ReduceGroupMeans(partials, n_groups, false, {}, partial_mean);
ReduceGroupMeans(partials, n_groups, partial_mean);
ComputePerFrameCC(partial_mean, cc, cc_n);
}
FinalizePerFrameScale(cc, cc_n, partial_scaled);
@@ -2427,7 +2410,7 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
if (scale_fulls && !scaled_fulls_on_gpu) {
std::vector<double> full_mean;
for (int it = 0; it < scaling_iter; ++it) {
ReduceGroupMeans(fulls, n_groups, false, {}, full_mean);
ReduceGroupMeans(fulls, n_groups, full_mean);
FitPerFrameG(fulls, fulls_frame_start, fulls_frame_count, full_mean, /*unity=*/true, g_full);
UpdateCorr(fulls, g_full, frame_scaled_scratch);
}
@@ -2465,6 +2448,6 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
#endif
// --- 5. Error model + merge + statistics. ---
auto r = MergeAndStats(n_groups, for_search, masked_ice_rings, combined_on_gpu && scaled_fulls_on_gpu);
auto r = MergeAndStats(n_groups, for_search, combined_on_gpu && scaled_fulls_on_gpu);
return r;
}