T

leonarski_fandClaude Opus 5 ae126c3d5b Per-image refinement: weight each spot by how strong it is for its resolution

`RefineGeometryIfNeeded` hands XtalOptimizer the WHOLE spot list, not the
indexed subset, and the first pass admits anything within 0.3 fractional-Miller
units of an integer - which is 11.3% of RANDOMLY placed spots, since the
admitted volume is (4/3)*pi*t^3. Every one of them then enters an unweighted L2
fit with an arbitrary rounded index. On images with many detections the
refined orientation ends up 2.3-2.8 degrees from the goniometer-consistent one
and explains 14 of its own 250 spots where the undragged orientation explains
68; mosaicity and profile radius inherit the error and integration follows.

Weight every spot by its intensity divided by the median intensity of its own
equal-count resolution shell, applied as w^2 on the squared residual with
w^2 = r/(1+r). The shell normalisation is the point: refinement needs the
high-resolution spots because they carry the cell and distance, and those are
LEGITIMATELY weaker, so a raw intensity weight would suppress exactly the
spots the fit depends on. Measured, the weight is resolution-neutral - median
exactly 0.707 in every shell, and corr(w, 1/d^2) = -0.20 / -0.11 against
-0.32 / -0.34 for the same function of un-normalised intensity.

This is a PRIOR: it is computed from the spot alone and never looks at the
current residual, so unlike a robust loss it cannot mistake a genuine spot for
an outlier while the starting geometry is still far off and leave the fit
unable to move. That failure is not hypothetical - a CauchyLoss on this same
residual, at the scale the multi-frame GeometryRefiner uses, collapsed one
crystal's indexing rate from 99.89% to 19.83% and was rejected.

It does not work by telling good spots from bad, and it does not need to. No
per-spot property separates spots that index from spots that do not: measured
AUC is 0.53 for peak pixel, 0.53 for total intensity, 0.51 for pixel count,
0.45 for peakedness, and a logistic regression on all twelve available
features with pairwise interactions reaches only 0.64. What the weight does is
halve the EFFECTIVE COUNT of every spot (mean w^2 = 0.517), and the damage
scales with the absolute count of unexplained spots in the objective - 80.6
per frame here against 36.8 for the finder that was never damaged. That is
also why an empirical `--max-spots 66` cap works while leaving the list no
purer than before: it reaches the same operating point by discarding spots.
This reaches it without discarding any, and without a tuned constant.

Rotation battery, 33 crystals, both spot finders:

  finder A   29/33 -> 30/33 point groups   (one crystal P222 -> P4212 = XDS,
                                            its high-shell CC1/2 86.0 -> 98.4)
  finder B   28/33 -> 29/33 point groups   (one crystal I222 -> I23,
                                            its high-shell CC1/2 14.8 -> 38.0)

No crystal lost its point group in either mode and no run failed. On the
meta-stable multi-lattice dataset the CC1/2 spread over four frame ranges
falls 19.7 -> 13.1 for finder B, and the indexing rate rises in 8 of 8
configurations. The crystal that the rejected robust loss destroyed keeps its
99.89% indexing rate exactly.

The cost, stated plainly: ISa falls by 0.2-1.7 on about five crystals (and
rises on two). Point-group correctness is worth more than that - merging in
the wrong symmetry cannot be undone from the output, whereas ISa is a quality
metric of data that remain correct - but it is a real trade and not a free win.

Off by default. The indexers pass a spot list they have already selected, so
their calls are unchanged; only the per-image refinement, which gets the raw
list, turns it on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-28 09:53:13 +02:00