The rotation error-model (a,b) fit built its sample pool from all usable
fulls, while the merge separately dropped symmetry outliers (> Nsigma from
the per-reflection median). So the fitted sigmas were calibrated against a
dirtier pool than the reflections that actually entered the merge.
Refactor the intensity-binned median fit into a `fit_ab` lambda and re-run
it once on the misfit-free pool (samples with dev2 <= reject_nsigma^2 *
model variance), matching the merge's own rejection. It operates on the
shared `samples` array, so the CPU and GPU error-model paths stay
bit-identical.
Neutral on lyso_ref (the median-binned fit is already robust to the few
outliers) - a correctness fix that matters more on dirtier data. Gated on
reject_outliers (default 6 sigma for rot3d).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
leonarski_fandClaude Opus 4.8
dee679380b
