T

leonarski_fandClaude Opus 5 9885d1fc28 Rotation: integrate every frame the sweep's lattice explains

A rotation dataset has ONE lattice. Once the first pass has found it and the
goniometer gives each frame its orientation, every frame of the sweep is a
frame of that crystal - yet integration was gated on each frame re-indexing on
its own, a test that carries an absolute floor of 9 indexed spots. A weakly
diffracting crystal shows a handful of spots per image while the geometry still
puts ~1500 reflections on the detector, so the floor threw away whole frames
that had nothing wrong with them.

Measured on a 360-degree battery crystal: 1484 of its 1800 frames failed that
gate, all of them on the spot-count floor alone and none on the consistency
test - the median failing frame had 4 spots and the lattice indexed all 4.
Integration therefore ran on 17.7% of the sweep and the merge came out 35.7%
complete at multiplicity 1.1, against XDS's 97.7% at 2.81 from the same images.
XDS's own INTEGRATE.LP shows why the floor is the wrong test there: 964 of its
frames have fewer than 9 strong spots and it predicts ~1483 reflections near
the Ewald sphere on every one of them, because INTEGRATE works from the global
orientation and has no per-frame indexing gate at all. Neither does
dials.integrate.

Split the one verdict into the two questions it was answering. "Does this frame
index?" - what the indexing rate reports and what the first pass scores
candidate lattices on - keeps the floor, because a handful of spots sit on
almost any lattice by chance. "Is this frame worth integrating?" keeps only the
consistency part, and only where the lattice does not come from this frame. A
frame whose spots largely MISS the lattice is still refused: on another battery
crystal that is 35% of the sweep, and integrating those collapsed the space
group to P1 - the floor had been shielding the merge from frames the model does
not describe, which is a different defect and not one to paper over here.

Two consequences had to be handled. A frame that is too sparse to index is also
too sparse to fit its own rocking width, and the placeholder it used to predict
with was being reported onward as if measured, into the frame-order average
that recomputes every partiality; report nothing instead, and fill the gaps in
that average with the run's median rather than a fixed default.

Probe (XDS in brackets): the crystal above goes 9 700 -> 81 956 observations,
8 618 -> 23 960 unique [23 576], 35.7% -> 99.4% complete [97.7%], R_meas
21.2% -> 68.6% [76.7%], CC1/2 96.0% -> 86.4% [81.1%], low-shell R_meas
7.2% -> 14.3% [20.6%], ISa unmeasurable -> 13.8 [10.4] - better than XDS on
every statistic, where before it was merging a third of the data. A second
crystal gains 41% more observations with R_meas 12.6% -> 8.5% and ISa
3.3 -> 3.7. The high-multiplicity control is unchanged to 2 observations in
924 782, and four further crystals move within recompilation noise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-11 17:59:13 +02:00