mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases 68 Wiki Activity

Bragg integration: fit a reflection over the pixels it has, not only over all of them

Browse Source 
A predicted reflection was discarded outright if ANY pixel of its signal disk was
unreadable - masked, untrusted, in a detector gap, or overloaded. On a battery
crystal that is 11.1% of all predictions, thrown away for a defect in one pixel
of fifty, and the pixels concerned sit at fixed places on the detector, so the
loss is systematic in reciprocal space rather than random.

Neither XDS nor dials does that. Both estimate the missing part from the profile
instead and keep the reflection while enough of it was seen: XDS's MINPK (default
75%, "the missing intensity is estimated from the learned profiles"), dials'
integration.profile.valid_foreground_threshold (default 0.75). MOSFLM is the one
program that rejects by default, and even it relaxes to 50% with PROFILE EDGE.

We already had the argument and the machinery: a profile fit is the amplitude of
a NORMALISED profile, so leaving pixels out renormalises the estimator by
construction - it costs information, which sum P^2/v duly loses and sigma duly
gains, and biases nothing. That is exactly why --overlap exclude drops a
neighbour's pixels from the fit rather than the reflection. Unreadable pixels are
the same case with a different reason, so they take the same treatment, cut on
the same threshold, in the same place: the readable fraction of the expected
profile, measured against the profile mass that lands on the detector at all so a
reflection is judged on the pixels that exist. A box sum has no profile to
renormalise with and keeps the all-or-nothing rule.

Two consequences handled. The summation seed and its variance now count the
pixels actually read, and the runaway guard scales the fit back to that same disk
before comparing - both exactly as before wherever nothing is missing. (Its
fallback then hands back that partial sum unrescaled, which would read low; the
guard fires on 8 of 96 688 recovered reflections, and on none at all on a weak
crystal, so it is not worth a branch.) And the profile, its resolution shells and
their widths are learned from COMPLETE reflections only, as is the box-sum
centroid post-refinement reads as an observed position: a disk with a hole gives
a centroid pulled away from the hole, and the hole does not move between frames.

That sigma gains what the missing pixels carried is the claim the whole change
rests on, and it is measurable. Force the conventional CENTRED cell of a
body-centred crystal in P1: the predictor then enumerates every lattice point,
and the reflections the centring makes systematically absent have a true
intensity of exactly zero, so their scatter about zero must equal their reported
sigma. Over 7.1 M such observations, matched by resolution shell, the trimmed
std(I)/rms(sigma) of the recovered reflections is 0.99 / 1.20 / 2.33 / 1.04 /
1.69 against 0.98 / 1.22 / 2.29 / 1.03 / 1.56 for the reflections that were
complete - the same calibration to a few percent. The lever there is small,
because the typical recovered reflection is missing only 5% of its disk. Lowering
the threshold to 0.50 admits a band missing 25-50%, which is a real lever: there
sigma comes out 8-43% larger than a complete reflection's in the same shell, and
the scatter about zero tracks it, 0.97 / 1.09 / 1.92 / 0.99 / 1.37, at or below
the complete population. Sigma grows, and by the amount it should.

The threshold stays at XDS's and dials' 0.75, on that evidence and on quality.
Below it the estimator starts to run out: on those same zero-intensity
reflections the recovered ones read +0.8 counts high at 0.75 and +1.9 counts high
in the 0.50-0.75 band, against a sigma of 12-17, and at 0.25 the fit degenerates
outright, single reflections carrying sigma in the thousands. Above it there is
nothing to buy: 0.90 leaves a fifth of the recoverable observations behind and
measures no better for them. On the high-multiplicity control, R_rim over
as-shipped / 0.90 / 0.75 / 0.50 runs 4.49% / 4.51% / 4.56% / 4.78% while
<I/sigma> runs 33.47 / 34.02 / 33.89 / 33.43 - 0.50 is where the recovered
observations stop paying for themselves.

Probe against the previous commit, six crystals. The high-multiplicity control
gains 4.2% more observations, 924 803 -> 963 946, which lands it on XDS's 961 379
from the same images, for <I/sigma> 33.47 -> 33.89, R_rim 4.49% -> 4.56% at 4.3%
more multiplicity, CC1/2 unchanged at 0.9998 and ISa 27.80 -> 27.12. Five weaker
crystals gain 3.3-4.8% of their observations and up to 1.0 point of completeness,
for <I/sigma> +0.4 to +3.6%, R_rim between -8.1% and +5.8% relative, CC1/2
+6.6 / +0.3 / +0.2 / -0.0 / -1.2 points, and ISa between +0.3% and -3.4%. Some of
that ISa is the point rather than the price: a reflection integrated over fewer
pixels carries less information, and the absence test above says the sigma that
reports so is honest. The GPU and CPU engines agree as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Full 38-crystal rotation battery, against the same binary without it:

  observations     better 38 / worse  0,  +937 100
  unique refl      better 30 / worse  0,    +9 229
  overall <I/sig>  better 33 / worse  1,     +7.00
  CC1/2            better  5 / worse  1,      +6.2
  space groups     unchanged at 35/38

Every crystal gains observations and not one loses a unique reflection. The two
costs are small and both are understood. Low-resolution R_meas is worse on eight
crystals, by +0.8 pp at most and +3.2 pp summed - a reflection whose own peak
pixel is unreadable loses the part of the profile that carries most of the
amplitude, and that population sits at low resolution; the following commit
handles it. And ISa falls on 32 crystals, by 10.9 summed, which is what admitting
937 000 further observations does to the strong-reflection asymptote: R_meas
excluding the one crystal whose thread-count noise is 1.5 pp is flat.
This commit is contained in:
leonarski_f
2026-08-12 02:03:42 +02:00
parent e0e429ae2f
commit 12ae6c5228
6 changed files with 136 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/bragg_integration/BraggIntegrationEngineCPU.cpp
+54 -36
View File
@@ -128,6 +128,7 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
int k_bin = 0; // which radial-background kernel this reflection's stencil needs
int cx = 0, cy = 0, shell = -1;
bool ok = false, strong = false, has_obs = false;
bool full = false; // every pixel of the signal disk was readable
};
std::vector<Rough> rough(npredicted);
double inv_d2_min = std::numeric_limits<double>::max(), inv_d2_max = 0.0;
@@ -196,7 +197,15 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
}
int n_bkg_used = n_bkg; // pixels behind the FINAL background value (trim/clip shrink it)
if (n_inner_valid == n_inner && n_bkg > 5) {
// A masked, untrusted, gapped or overloaded pixel inside the signal disk used to discard the
// reflection outright. A profile fit does not need it to: the fit is the amplitude of a
// NORMALISED profile, so leaving pixels out renormalises the estimator by construction and
// widens sigma by the information they carried - the same argument Exclude makes for a
// neighbour's pixels. Pass B keeps the reflection only while enough of the expected profile
// survived to constrain the amplitude (XDS's MINPK, dials' valid_foreground_threshold). A box
// sum has no profile to renormalise with, so there it stays all or nothing.
const bool full = n_inner_valid == n_inner;
if ((full || mode != IntegratorMode::BoxSum) && n_bkg > 5) {
out.bkg = bkg_sum / n_bkg;
if (bkg_trim_frac > 0.0 && bkg_vals.size() > 5
&& bkg_vals.size() <= static_cast<size_t>(bragg_engine::BKG_TRIM_MAX)) {
@@ -239,31 +248,39 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
}
if (n > 5) { out.bkg = s / n; n_bkg_used = n; }
}
out.I = static_cast<double>(I_sum) - static_cast<double>(n_inner) * out.bkg;
// The sum is over the pixels actually READ, so that is the count the background is
// subtracted with; with nothing missing it is the whole disk, exactly as before.
out.I = static_cast<double>(I_sum) - static_cast<double>(n_inner_valid) * out.bkg;
// I = I_sum - n_inner*bkg, and bkg is itself estimated from n_bkg_used pixels, so its
// error enters n_inner times over: var(I) = I_sum + n_inner^2 * bkg/n_bkg_used. Leaving
// the second term out understates sigma by sqrt(1 + n_inner/n_bkg) - 1.109x at the
// default r1=4/r2=6/r3=10 stencil, on every reflection of every dataset.
out.bkg_var = out.bkg / n_bkg_used;
out.var_bkg = static_cast<double>(n_inner) * out.bkg
+ static_cast<double>(n_inner) * n_inner * out.bkg_var;
out.var_bkg = static_cast<double>(n_inner_valid) * out.bkg
+ static_cast<double>(n_inner_valid) * n_inner_valid * out.bkg_var;
out.I_sum = I_sum;
out.n_inner = static_cast<int>(n_inner);
out.n_inner = static_cast<int>(n_inner_valid);
out.n_disk = n_disk;
out.n_own = n_own;
const double var_bkg_term = static_cast<double>(n_inner) * n_inner * out.bkg_var;
const double var_bkg_term = static_cast<double>(n_inner_valid) * n_inner_valid * out.bkg_var;
out.sigma = 1.0;
if (I_sum > 0) {
out.sigma = std::max(out.sigma, std::sqrt(static_cast<double>(I_sum) + var_bkg_term));
out.obs_x = static_cast<double>(I_sum_x) / static_cast<double>(I_sum);
out.obs_y = static_cast<double>(I_sum_y) / static_cast<double>(I_sum);
out.has_obs = true;
// A disk with a hole in it gives a centroid pulled away from the hole, and the hole
// sits at a fixed place on the detector - post-refinement would read that as geometry.
out.has_obs = full;
}
out.cx = static_cast<int>(std::lround(r.predicted_x));
out.cy = static_cast<int>(std::lround(r.predicted_y));
out.ok = true;
out.strong = out.sigma > 0.0 && out.I / out.sigma >= STRONG_I_OVER_SIGMA;
if (r.d > 0.0f) {
out.full = full;
// The profile, its resolution shells and their widths are learned from COMPLETE
// reflections: a partial one has no intensity to normalise its grid by and a hole to
// stack into it. So the learning sees exactly what it saw before this rescue existed.
out.strong = full && out.sigma > 0.0 && out.I / out.sigma >= STRONG_I_OVER_SIGMA;
if (full && r.d > 0.0f) {
const double inv_d2 = 1.0 / (static_cast<double>(r.d) * r.d);
inv_d2_min = std::min(inv_d2_min, inv_d2);
inv_d2_max = std::max(inv_d2_max, inv_d2);
@@ -454,29 +471,30 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
const int Gf = 2 * Rf + 1;
// --- How much of the expected profile is cleanly this reflection's own. p_own is the whole
// grid's clean mass, i.e. XDS's MINPK quantity, and it is what Reject cuts on. m_own /
// m_all is the same fraction over the r1 disk alone, which is what the summation seed the
// runaway guard below compares against actually saw; with nothing excluded it is 1 and
// the guard is untouched. ---
double p_own = 1.0, m_all = 0.0, m_own = 0.0;
if (overlap != OverlapMode::Off) {
p_own = 0.0;
for (int dy = -Rf; dy <= Rf; ++dy)
for (int dx = -Rf; dx <= Rf; ++dx) {
const double Pp = (*Pvec)[(dy + Rf) * Gf + (dx + Rf)];
if (Pp <= 0.0) continue;
const int x = rh.cx + dx, y = rh.cy + dy;
if (x < 0 || y < 0 || x >= W || y >= H) continue;
if (!valid(img[y * W + x])) continue;
const bool own = clean(x, y, i);
if (own) p_own += Pp;
if (dx * dx + dy * dy < r1_sq) {
m_all += Pp;
if (own) m_own += Pp;
}
}
}
// --- How much of the expected profile the fit can actually see. p_valid is the readable
// fraction of it - XDS's MINPK, dials' valid_foreground_threshold - measured against the
// mass that falls on the detector at all, so a reflection is judged on the pixels that
// exist and not on where the sensor ends. p_own is the same quantity over neighbour-owned
// pixels, what Reject cuts on. m_read / m_all is the fraction of the r1 disk the
// summation seed the runaway guard compares against actually summed; with nothing missing
// and nothing excluded it is 1 and the guard is untouched. ---
double p_grid = 0.0, p_valid = 0.0, p_own = 0.0, m_all = 0.0, m_read = 0.0;
for (int dy = -Rf; dy <= Rf; ++dy)
for (int dx = -Rf; dx <= Rf; ++dx) {
const double Pp = (*Pvec)[(dy + Rf) * Gf + (dx + Rf)];
if (Pp <= 0.0) continue;
const int x = rh.cx + dx, y = rh.cy + dy;
if (x < 0 || y < 0 || x >= W || y >= H) continue;
const bool in_disk = dx * dx + dy * dy < r1_sq;
p_grid += Pp;
if (in_disk) m_all += Pp;
if (!valid(img[y * W + x])) continue;
p_valid += Pp;
const bool own = clean(x, y, i);
if (own) p_own += Pp;
if (in_disk && (own || !exclude)) m_read += Pp;
}
if (p_valid < overlap_min_peak * p_grid) continue;
if (overlap == OverlapMode::Reject && p_own < overlap_min_peak) continue;
const double B = std::max(rh.bkg, PIXEL_VARIANCE_FLOOR);
@@ -519,10 +537,10 @@ std::vector<Reflection> BraggIntegrationEngineCPU::RunImpl(const Sampler &img,
// fluctuation, and 1/den - I is what does not (1/den grows with I one for one).
double var_bkg = std::max(0.0, 1.0 / den - I
+ (wsum / den) * (wsum / den) * rh.bkg_var);
// The seed is a sum over the disk the box sum actually read, so when Exclude has taken pixels
// out of both, the fit's full-profile intensity has to be scaled down to that same disk
// before the two are comparable. Nothing excluded gives exactly 1.
const double guard_scale = exclude && m_all > 0.0 ? m_own / m_all : 1.0;
// The seed is a sum over the disk the box sum actually read, so when pixels are missing from
// both - excluded to a neighbour, or unreadable - the fit's full-profile intensity has to be
// scaled down to that same disk before the two are comparable. Nothing dropped gives 1.
const double guard_scale = m_all > 0.0 ? m_read / m_all : 1.0;
if (std::abs(I * guard_scale - rh.I) > PROFILE_SUMMATION_MAX_NSIGMA * rh.sigma) {
I = rh.I;
sigma = rh.sigma;