leonarski_f
75e401f0e5
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63
6.9 KiB
PixelRefine — findings, June 2026
A record of the main results from the still-integration investigation, validated on two
lysozyme P4₃2₁2 datasets indexed against the same reference (6G8A_refine_001.mtz):
- Crystal 1 — serial jet stills (
LysozymeJet5-…, MicroMAX DMM ~1 % bandwidth). - Crystal 2 — rotation crystal (
fixed_master.h5, 1800 × 0.2°, Si mono), run as stills as a stress test, and as rotation for reference. XDS output of this crystal (CORRECT.LP,XDS_ASCII.HKL) is the external ground truth.
1. The factored model is a qualitative win (crystal 2, stills, 1.7 Å)
Replacing the per-pixel least squares with the factored Terms 1+2 (intensity residual +
measured per-resolution R_1) turns an erratic, high-res-collapsing result into a flat one:
| N_obs | ⟨I/σ⟩ | CC₁/₂ (per shell) | CCref (per shell) | |
|---|---|---|---|---|
| baseline per-pixel loss | 799 k | 7.2 | 75→81→56→32→…→0 % | erratic, →0 |
| factored Terms 1+2 | 1.22 M | 10.7 | 84–92 %, flat to 1.7 Å | 77–92 %, flat |
This reaches the proper rotation (-R -P rot) path's quality from the stills path. The
mechanism is clean and on-thesis: Term 1 lifts the per-image scale CC vs the reference
from 0.09 → 0.40 (median 0.01 → 0.39) — the per-pixel loss produced near-random per-image
scales; the factored objective makes each image's scale agree with the reference, so the
merge coheres. The reference enters at maximum leverage (it sets both the target and the
Fisher weight).
Term 3 (per-spot recentring) is a no-op on both crystals (93.5 vs 93.6 % CC₁/₂): the generous integration aperture already contains the spot, so shifting the spotlight by the sub-pixel prediction offset does not change the integrated total. It was removed; the position residual's value can only come as a geometry term (refine orientation/distance from the centroid), which is XtalOptimizer's job, not a per-spot mask shift.
2. The residual centroid offset is a sampling floor, not a recoverable error
On the jet, predictions land on the spot to within a ~0.4 px tangential scatter. Three independent cuts show this is the centroid undersampling floor of the ~2×2 spots, not a geometry error or a shape effect:
- Flat with intensity. Binned by significance (≈6σ vs ≈39σ, a 6× span) the tangential centroid offset is 0.41 → 0.36 px (median). A background-limited centroid would shrink ∝ 1/significance (≈6×); it does not, so it is a real ~0.35 px floor, not counting noise.
- Peak is a worse predictor than the centroid (sub-pixel parabolic mode: 0.52 vs 0.36 px at high signif). If an asymmetric tail were dragging the centroid off a correct peak, the peak would be better — it is not. So there is no coherent shape asymmetry to model; the offset is a zero-mean per-spot scatter from sub-pixel phase aliasing of a spot only ~2 px wide.
- Radial centroid offset ≈ 0.01–0.02 px, flat → no distance/parallax error (parallax is radial and grows outward).
Consequences: recentring cannot rescue weak reflections (their prediction is already
centred; the 0.4 px is irreducible sampling scatter), which is why a generous box beats a
tight profile mask. A symmetric non-Gaussian shape would not move the centroid at all — it
would show up only in Term 2 (R_1), not in a position term.
3. The σ gap to XDS is fulls-vs-partials, not intensities
XDS reports ISa = 28.3 (asymptotic relative error 3.5 %); our stills path reports
ISa ≈ 1.1 and the rotation path ≈ 1.6. The decisive clue is in XDS_ASCII.HKL: the
PEAK column (fraction of each reflection captured) is ≈ 100 % for 97 % of reflections,
and the header gives mosaicity 0.091° < 0.2° oscillation. So these are full reflections,
recorded over the 1–3 frames each rocking curve spans — not partials.
- XDS profile-fits in 3D (the third axis is the rotation/rocking direction) and sums the rocking curve with profile weights → one full per reflection, counting-limited σ.
- jfjoch
-P rotintegrates a 2D shoebox per frame and recovers each full by dividing by the rocking-curve fractionR_j— a cheap approximation of 3D integration. That division injects per-observation noise (it amplifies each frame's background noise, pays N independent backgrounds, and carries a random per-observation partiality error).
Crucially, ISa and merged-intensity accuracy are different axes, decoupled by multiplicity. Our merged intensities are correct (§4), because ~60–240× multiplicity averages the per-observation noise down; ISa measures the per-observation precision, which multiplicity cannot improve. So "right intensities, wrong σ" is not a contradiction.
A cheap probe confirms it. Raising --min-partiality from 0.02 → 0.5 on crystal 2
(rotation) lifts ISa 1.6 → 3.8 at zero completeness cost (high multiplicity) and with
CCref flat — and the high-res shells improve. The default 0.02 keeps deep-tail partials
(2 % of a reflection, scaled back ×50) that were over-weighted and polluting the merge. So
of the ~17× rotation gap, ~2.8× is tunable tail-weighting and ~6× is structural (the
2D-divide vs 3D-sum difference) — the structural part needs a real rocking-curve sum, not a
knob.
4. Our intensities are XDS-grade — the limit is σ, not I
Direct CC of merged intensities (both reduced to the 4/mmm asymmetric unit; 98.8 % of reflections matched, no manual reindex):
| vs XDS (CC on merged I, to 1.2 Å) | overall | at 1.2 Å |
|---|---|---|
| PixelRefine stills (factored) | 95.9 % | 96.9 % |
rotation -P rot (min_partiality 0.3) |
98.8 % | 98.5 % |
PixelRefine-stills intensities track XDS at 95–98 %, flat to the 1.2 Å diffraction
limit, only ~3 % behind full rotation integration. This is exactly what the CC₁/₂→CC_true
relation predicts (stills CC₁/₂≈0.88 ⇒ 0.967), so it is real, carried by the huge stills
multiplicity averaging per-observation noise down. Conclusion: the intensity estimation is
right; the remaining gap to XDS is entirely the per-observation σ / partiality axis — the
3D rocking-curve integration frontier (FACTORED_MODEL.md §4), deliberately parked.
What is fixed vs. parked
Landed (and now the default model): mean (not median) background in both integrators; de-biased (background-limited) variance; widened Ewald prediction band; the factored Terms 1+2 as the only PixelRefine objective; the global XDS-form merge error model with ISa.
Diagnosed dead-ends (do not re-litigate): per-image R refinement (degenerate with
scale); per-spot recentring (no-op); chasing the 0.4 px centroid floor (sampling, not
recoverable). Parked (rotation-side): the 3D rocking-curve sum / one-shot
post-refinement, a min_partiality default for high-multiplicity data, and
partiality-aware variance weighting — the path to XDS-grade ISa, but a separate axis from
the intensity work.