ApplyCellSurface fits one multiplicative factor per cell by least squares with the
OBSERVATION on the regressor side - A = sum w Is Iref / sum w Is^2, the slope that
carries Is onto Iref. A least-squares slope is attenuated by the noise in its own
regressor, here by 1/(1 + (sigma/I)^2), and an observation carries all of a
reflection's noise where the reference carries about 1/n of it. So every cell is
pulled towards zero by an amount set by its own signal-to-noise - and on a detector
that is a function of radius, which is to say of resolution. The gauge fix then
spreads the ramp over the whole surface and the alternating rounds compound it.
Nothing downstream catches it. The cross-validation splits by frame parity, and a
bias that depends only on a cell's signal-to-noise is identical in both halves. And
the held-out score is sum|Is - Iref| / sum Iref over the whole resolution range,
which any resolution-dependent scale lowers without tightening a single reflection:
applying a scale that is purely a function of d to a merged 360 deg sweep leaves
every resolution shell's R_meas unchanged to 0.05 pp and takes the run's overall
R_meas from 42.4% to 27.5%. The surface finds that manipulation because its own bias
points exactly along it, and reports it as a 40% held-out gain.
The cost is large wherever a sweep was taken past its signal. On one such run the
fitted detector-plane "flat field" ran from 0.25 to 4.0 with 37% of its cells pinned
at the low clamp - an 11x centre-to-edge ramp - and on the same combined fulls it
moved the merge: low-resolution R_meas 8.8% -> 15.5%, <I/sigma> 33.2 -> 13.4, CC1/2
99.65 -> 98.87, error model b 5.0e-03 -> 4.3e-02, ISa 11.0 -> 3.5. The program's own
--no-scaling-corrections run agrees on the same fulls and the same space group
(b 5.4e-03 ISa 10.6 against b 4.1e-02 ISa 3.6). Regressing the other way round
restores all of it - 8.6%, 33.6, 99.65, 4.9e-03, ISa 11.1 - and keeps the surface's
real gain in the middle shells, where CC1/2 rises by 1-2 points.
Where the data are well measured everywhere the two estimators are indistinguishable:
on a control crystal the two surfaces agree to 0.02 pp in every shell and 0.02 in
ISa, which is what a flat field should look like. Recovering a synthetic +-20% ripple
imposed on the same fulls: 0.18 rms in log against 0.91 for the shipped form on the
weak-outer-shell crystal, 0.059 against 0.082 on the well-measured one, with the
spurious correlation between the fitted factor and detector radius falling from -0.30
to -0.01.
The same estimator serves the absorption surface and any resolution-indexed surface
fitted through this function, where the bias lands directly on the resolution axis:
on a crystal the program itself reports as having no radiation damage (total dB
0.00 A^2), a batch x resolution-shell surface fitted the old way manufactures a
monotone 12% falloff from low to high resolution out of nothing, and the new way
gives 1%.
The per-frame scale (FitPerFrameG) already regresses this way round.
Full 38-crystal rotation battery against the same binary without it. Read it with
the next commit, which supplies the correction this one stops faking; alone it
is a partial state:
observations better 34 / worse 4, +22 940
R_meas_lo better 4 / worse 8, -5.9
ISa better 20 / worse 14, +32.66
CC1/2 better 2 / worse 10, -11.4
R_meas better 1 / worse 30, +114.3
space groups unchanged at 35/38
Overall R_meas RISES, and that is the artefact leaving rather than arriving: it
is a ratio of sums across every shell, so a resolution-dependent scale lowers it
without one reflection getting tighter - measured, a scale that is purely f(d)
leaves every shell's R_meas unchanged to 0.05 pp while moving the run's overall
value from 42.4 to 27.5 per cent. That is precisely the shape of the bias, which
is why the surface's own held-out score read it as a 40 per cent gain.
CC1/2 falling on ten crystals is not covered by that argument and is the reason
this commit is not defensible on its own: the ramp was partly standing in for a
real time-dependent absorption that nothing else modelled. With that correction
supplied by the following commit the same battery gives CC1/2 -1.4 and
R_meas_lo -26.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
leonarski_fandClaude Opus 5
b8e7e9c8bf
