Jungfraujoch/image_analysis/pixel_refinement/FINDINGS-2026-06.md

PixelRefine — findings, June 2026

A record of the main results from the still-integration investigation, validated on two lysozyme P4₃2₁2 datasets indexed against the same reference (6G8A_refine_001.mtz):

• Crystal 1 — serial jet stills (LysozymeJet5-…, MicroMAX DMM ~1 % bandwidth).
• Crystal 2 — rotation crystal (fixed_master.h5, 1800 × 0.2°, Si mono), run as stills as a stress test, and as rotation for reference. XDS output of this crystal (CORRECT.LP, XDS_ASCII.HKL) is the external ground truth.

---

1. The factored model is a qualitative win (crystal 2, stills, 1.7 Å)

Replacing the per-pixel least squares with the factored Terms 1+2 (intensity residual + measured per-resolution R_1) turns an erratic, high-res-collapsing result into a flat one:

	N_obs	⟨I/σ⟩	CC₁/₂ (per shell)	CCref (per shell)
baseline per-pixel loss	799 k	7.2	75→81→56→32→…→0 %	erratic, →0
factored Terms 1+2	1.22 M	10.7	84–92 %, flat to 1.7 Å	77–92 %, flat

This reaches the proper rotation (-R -P rot) path's quality from the stills path. The mechanism is clean and on-thesis: Term 1 lifts the per-image scale CC vs the reference from 0.09 → 0.40 (median 0.01 → 0.39) — the per-pixel loss produced near-random per-image scales; the factored objective makes each image's scale agree with the reference, so the merge coheres. The reference enters at maximum leverage (it sets both the target and the Fisher weight).

Term 3 (per-spot recentring) is a no-op on both crystals (93.5 vs 93.6 % CC₁/₂): the generous integration aperture already contains the spot, so shifting the spotlight by the sub-pixel prediction offset does not change the integrated total. It was removed; the position residual's value can only come as a geometry term (refine orientation/distance from the centroid), which is XtalOptimizer's job, not a per-spot mask shift.

---

2. The residual centroid offset is a sampling floor, not a recoverable error

On the jet, predictions land on the spot to within a ~0.4 px tangential scatter. Three independent cuts show this is the centroid undersampling floor of the ~2×2 spots, not a geometry error or a shape effect:

• Flat with intensity. Binned by significance (≈6σ vs ≈39σ, a 6× span) the tangential centroid offset is 0.41 → 0.36 px (median). A background-limited centroid would shrink ∝ 1/significance (≈6×); it does not, so it is a real ~0.35 px floor, not counting noise.
• Peak is a worse predictor than the centroid (sub-pixel parabolic mode: 0.52 vs 0.36 px at high signif). If an asymmetric tail were dragging the centroid off a correct peak, the peak would be better — it is not. So there is no coherent shape asymmetry to model; the offset is a zero-mean per-spot scatter from sub-pixel phase aliasing of a spot only ~2 px wide.
• Radial centroid offset ≈ 0.01–0.02 px, flat → no distance/parallax error (parallax is radial and grows outward).

Consequences: recentring cannot rescue weak reflections (their prediction is already centred; the 0.4 px is irreducible sampling scatter), which is why a generous box beats a tight profile mask. A symmetric non-Gaussian shape would not move the centroid at all — it would show up only in Term 2 (R_1), not in a position term.

---

3. The σ gap to XDS is fulls-vs-partials, not intensities

XDS reports ISa = 28.3 (asymptotic relative error 3.5 %); our stills path reports ISa ≈ 1.1 and the rotation path ≈ 1.6. The decisive clue is in XDS_ASCII.HKL: the PEAK column (fraction of each reflection captured) is ≈ 100 % for 97 % of reflections, and the header gives mosaicity 0.091° < 0.2° oscillation. So these are full reflections, recorded over the 1–3 frames each rocking curve spans — not partials.

• XDS profile-fits in 3D (the third axis is the rotation/rocking direction) and sums the rocking curve with profile weights → one full per reflection, counting-limited σ.
• jfjoch -P rot integrates a 2D shoebox per frame and recovers each full by dividing by the rocking-curve fraction R_j — a cheap approximation of 3D integration. That division injects per-observation noise (it amplifies each frame's background noise, pays N independent backgrounds, and carries a random per-observation partiality error).

Crucially, ISa and merged-intensity accuracy are different axes, decoupled by multiplicity. Our merged intensities are correct (§4), because ~60–240× multiplicity averages the per-observation noise down; ISa measures the per-observation precision, which multiplicity cannot improve. So "right intensities, wrong σ" is not a contradiction.

A cheap probe confirms it. Raising --min-partiality from 0.02 → 0.5 on crystal 2 (rotation) lifts ISa 1.6 → 3.8 at zero completeness cost (high multiplicity) and with CCref flat — and the high-res shells improve. The default 0.02 keeps deep-tail partials (2 % of a reflection, scaled back ×50) that were over-weighted and polluting the merge. So of the ~17× rotation gap, ~2.8× is tunable tail-weighting and ~6× is structural (the 2D-divide vs 3D-sum difference) — the structural part needs a real rocking-curve sum, not a knob.

---

4. Our intensities are XDS-grade — the limit is σ, not I

Direct CC of merged intensities (both reduced to the 4/mmm asymmetric unit; 98.8 % of reflections matched, no manual reindex):

vs XDS (CC on merged I, to 1.2 Å)	overall	at 1.2 Å
PixelRefine stills (factored)	95.9 %	96.9 %
rotation -P rot (min_partiality 0.3)	98.8 %	98.5 %

PixelRefine-stills intensities track XDS at 95–98 %, flat to the 1.2 Å diffraction limit, only ~3 % behind full rotation integration. This is exactly what the CC₁/₂→CC_true relation predicts (stills CC₁/₂≈0.88 ⇒ 0.967), so it is real, carried by the huge stills multiplicity averaging per-observation noise down. Conclusion: the intensity estimation is right; the remaining gap to XDS is entirely the per-observation σ / partiality axis — the 3D rocking-curve integration frontier (FACTORED_MODEL.md §4), deliberately parked.

---

What is fixed vs. parked

Landed (and now the default model): mean (not median) background in both integrators; de-biased (background-limited) variance; widened Ewald prediction band; the factored Terms 1+2 as the only PixelRefine objective; the global XDS-form merge error model with ISa.

Diagnosed dead-ends (do not re-litigate): per-image R refinement (degenerate with scale); per-spot recentring (no-op); chasing the 0.4 px centroid floor (sampling, not recoverable). Parked (rotation-side): the 3D rocking-curve sum / one-shot post-refinement, a min_partiality default for high-multiplicity data, and partiality-aware variance weighting — the path to XDS-grade ISa, but a separate axis from the intensity work.