A predicted reflection was discarded outright if ANY pixel of its signal disk was
unreadable - masked, untrusted, in a detector gap, or overloaded. On a battery
crystal that is 11.1% of all predictions, thrown away for a defect in one pixel
of fifty, and the pixels concerned sit at fixed places on the detector, so the
loss is systematic in reciprocal space rather than random.
Neither XDS nor dials does that. Both estimate the missing part from the profile
instead and keep the reflection while enough of it was seen: XDS's MINPK (default
75%, "the missing intensity is estimated from the learned profiles"), dials'
integration.profile.valid_foreground_threshold (default 0.75). MOSFLM is the one
program that rejects by default, and even it relaxes to 50% with PROFILE EDGE.
We already had the argument and the machinery: a profile fit is the amplitude of
a NORMALISED profile, so leaving pixels out renormalises the estimator by
construction - it costs information, which sum P^2/v duly loses and sigma duly
gains, and biases nothing. That is exactly why --overlap exclude drops a
neighbour's pixels from the fit rather than the reflection. Unreadable pixels are
the same case with a different reason, so they take the same treatment, cut on
the same threshold, in the same place: the readable fraction of the expected
profile, measured against the profile mass that lands on the detector at all so a
reflection is judged on the pixels that exist. A box sum has no profile to
renormalise with and keeps the all-or-nothing rule.
Two consequences handled. The summation seed and its variance now count the
pixels actually read, and the runaway guard scales the fit back to that same disk
before comparing - both exactly as before wherever nothing is missing. (Its
fallback then hands back that partial sum unrescaled, which would read low; the
guard fires on 8 of 96 688 recovered reflections, and on none at all on a weak
crystal, so it is not worth a branch.) And the profile, its resolution shells and
their widths are learned from COMPLETE reflections only, as is the box-sum
centroid post-refinement reads as an observed position: a disk with a hole gives
a centroid pulled away from the hole, and the hole does not move between frames.
That sigma gains what the missing pixels carried is the claim the whole change
rests on, and it is measurable. Force the conventional CENTRED cell of a
body-centred crystal in P1: the predictor then enumerates every lattice point,
and the reflections the centring makes systematically absent have a true
intensity of exactly zero, so their scatter about zero must equal their reported
sigma. Over 7.1 M such observations, matched by resolution shell, the trimmed
std(I)/rms(sigma) of the recovered reflections is 0.99 / 1.20 / 2.33 / 1.04 /
1.69 against 0.98 / 1.22 / 2.29 / 1.03 / 1.56 for the reflections that were
complete - the same calibration to a few percent. The lever there is small,
because the typical recovered reflection is missing only 5% of its disk. Lowering
the threshold to 0.50 admits a band missing 25-50%, which is a real lever: there
sigma comes out 8-43% larger than a complete reflection's in the same shell, and
the scatter about zero tracks it, 0.97 / 1.09 / 1.92 / 0.99 / 1.37, at or below
the complete population. Sigma grows, and by the amount it should.
The threshold stays at XDS's and dials' 0.75, on that evidence and on quality.
Below it the estimator starts to run out: on those same zero-intensity
reflections the recovered ones read +0.8 counts high at 0.75 and +1.9 counts high
in the 0.50-0.75 band, against a sigma of 12-17, and at 0.25 the fit degenerates
outright, single reflections carrying sigma in the thousands. Above it there is
nothing to buy: 0.90 leaves a fifth of the recoverable observations behind and
measures no better for them. On the high-multiplicity control, R_rim over
as-shipped / 0.90 / 0.75 / 0.50 runs 4.49% / 4.51% / 4.56% / 4.78% while
<I/sigma> runs 33.47 / 34.02 / 33.89 / 33.43 - 0.50 is where the recovered
observations stop paying for themselves.
Probe against the previous commit, six crystals. The high-multiplicity control
gains 4.2% more observations, 924 803 -> 963 946, which lands it on XDS's 961 379
from the same images, for <I/sigma> 33.47 -> 33.89, R_rim 4.49% -> 4.56% at 4.3%
more multiplicity, CC1/2 unchanged at 0.9998 and ISa 27.80 -> 27.12. Five weaker
crystals gain 3.3-4.8% of their observations and up to 1.0 point of completeness,
for <I/sigma> +0.4 to +3.6%, R_rim between -8.1% and +5.8% relative, CC1/2
+6.6 / +0.3 / +0.2 / -0.0 / -1.2 points, and ISa between +0.3% and -3.4%. Some of
that ISa is the point rather than the price: a reflection integrated over fewer
pixels carries less information, and the absence test above says the sigma that
reports so is honest. The GPU and CPU engines agree as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Full 38-crystal rotation battery, against the same binary without it:
observations better 38 / worse 0, +937 100
unique refl better 30 / worse 0, +9 229
overall <I/sig> better 33 / worse 1, +7.00
CC1/2 better 5 / worse 1, +6.2
space groups unchanged at 35/38
Every crystal gains observations and not one loses a unique reflection. The two
costs are small and both are understood. Low-resolution R_meas is worse on eight
crystals, by +0.8 pp at most and +3.2 pp summed - a reflection whose own peak
pixel is unreadable loses the part of the profile that carries most of the
amplitude, and that population sits at low resolution; the following commit
handles it. And ISa falls on 32 crystals, by 10.9 summed, which is what admitting
937 000 further observations does to the strong-reflection asymptote: R_meas
excluding the one crystal whose thread-count noise is 1.5 pp is flat.
leonarski_f
12ae6c5228