A crystal integrated to the detector corner but merged well short of it carries observations
through the whole merge that the merge then discards. On the heaviest dataset in the rotation test
set that is 63.3 M partials of which 6.4 M are ever used: the other nine tenths are sorted,
uploaded, scaled, combined and error-modelled before anything looks at their resolution. Ingest
copied every one of them unconditionally, and the d_min limit was first applied far downstream, in
the ASU grouping.
They are now dropped at ingest, immediately after the one big sort:
- WHOLE raw-hkl runs are dropped, on the same rawrun_d the ASU grouping already tests. A
per-observation test is not equivalent - a run is in or out today by one member's d - and using a
different rule here would put the two out of step.
- The drop happens AFTER the flux meter, which takes each frame's mean background over every
reflection on it, and after the sort, so neither changes.
- The compaction runs in index order, so a frame's observations stay contiguous and keep their
order, and every per-frame sum keeps its sequence of roundings.
The incident-flux divide goes with it: it was reading one int and dividing one float across 5 GB in
a pass of its own. The per-frame mean it needs is now accumulated by the ingest fill loop - one
frame, one thread, same order, so bit-exact - and the divide rides on the finiteness pass that
already touches that field.
Ported from 2608-performance with two changes. The ingest fill loop there had been parallelised by
an earlier commit that is not being taken, so the mean background is accumulated in the serial loop
this branch still has; it is the same sum in the same order either way. And the post-refinement
sampling that commit also introduced - thinning the fit to 8 M partials by a hash of the raw hkl -
is NOT included. Every consumer of the dropped observations is gated on the ASU group, so dropping
them is a no-op for the science; thinning post-refinement is not, its own measurement puts the cell
scale breaking at 4 M against a pool of 8 to 16 M, and it makes the fit depend on how far
integration ran. That belongs to its own decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011n8riB6X59oRjkrSHzNPAU
