Jungfraujoch/image_analysis/pixel_refinement/METHODS.md

# PixelRefine — methods

`PixelRefine` is the still-image integrator. It integrates the Bragg reflections of one
image by **profile fitting against a reference intensity set** $I^\mathrm{ref}$ (e.g.
`F_calc` from a deposited model, or the current merged estimate in an EM-style outer loop)
and returns already-scaled intensities. It is an **intensity-wise** operation: the
detector geometry (orientation, cell, beam, distance) is taken as fixed — it was refined
upstream by `XtalOptimizer` (`IndexAndRefine::RefineGeometryIfNeeded`) — and PixelRefine
only measures the spot shape and fits the per-image scale.

The objective is the factored per-reflection likelihood of `FACTORED_MODEL.md`, **Terms 1
and 2**. This note records the equations and the reasons behind each design choice.

Throughout, a reflection's shoebox is a small box of raw detector pixels $I_p$ with a
local flat background $B$; the area-normalised tangential profile at pixel $p$ is $P_p$,
and $v_p$ is the variance used to weight pixel $p$.

---

## 0. The forward model

The recorded amplitude of a still reflection is the profile-fit amplitude

$$
J \;=\; \frac{\sum_p P_p\,(I_p - B)/v_p}{\sum_p P_p^{2}/v_p},
\qquad
\operatorname{var}(J) = \frac{1}{\sum_p P_p^{2}/v_p}.
$$

The full (rotation-equivalent) intensity is recovered by dividing out the factors a still
does not record,

$$
I \;=\; \frac{J}{p\,B_\mathrm{DW}\,\mathrm{pol}},\qquad
p = \exp\!\left(-\frac{\epsilon_r^{2}}{R_{0,\mathrm{eff}}^{2}}\right),\quad
B_\mathrm{DW}=\exp\!\left(-\frac{B_\mathrm{fac}}{4 d^{2}}\right),
$$

where the **partiality** $p$ is the fraction of the mosaic block crossing the Ewald
sphere, $\epsilon_r$ is the radial excitation error, $R_0$ the radial (rocking) width, and
$\mathrm{pol}$ the polarisation correction. The tangential profile is a separable,
area-normalised Gaussian of width $R_1$:

$$
P_p = \frac{1}{\pi R_1^{2}}\exp\!\left(-\frac{\epsilon_{t,p}^{2}}{R_1^{2}}\right).
$$

A finite **X-ray bandwidth** thickens the Ewald shell radially, adding a fixed,
resolution-dependent term to the radial width, $R_{0,\mathrm{eff}}^2 = R_0^2 +
(b\lambda)^2/(2d^4)$ ($b$ = relative bandwidth; the pink-beam / DMM signature). $b=0$ is a
monochromatic no-op.

---

## 1. De-biased variance (the load-bearing fix)

**Symptom.** Mean intensities went **negative** in the high-resolution shells
($\langle I/\sigma\rangle$ down to $-12$), and the per-image scale $G$ collapsed to $0$ on
most images, dropping ~80 % of observations.

**Cause.** The extraction weighted each pixel by its **observed** count, $v_p = I_p$. A
down-fluctuated background pixel ($I_p < B$) then gets a small $v_p$, hence a large
$w_p=1/v_p$, and its contribution $P_p(I_p-B)/v_p < 0$ is large in magnitude. Summed over
the many empty shoebox pixels this drags $J$ below zero — the *inverse-observed-count*
(Poisson-on-data) bias, worst where the true signal is weakest (high resolution).

**Fix.** For background-limited reflections the correct variance is the local background,
**constant over the shoebox**, $v_p = \max(B,1)$, giving the unbiased uniform-variance
estimator $J = \sum_p P_p (I_p-B)/\sum_p P_p^2$. This turned $\langle I/\sigma\rangle$
positive at all resolutions and stopped the scale collapse.

---

## 2. Prediction band and multiplicity

A reflection is given a shoebox only when it lies within a radial band of the Ewald
sphere, $\bigl|\,|S_{hkl}| - 1/\lambda\,\bigr| \le \delta$. For randomly oriented stills
the number of images on which a given $hkl$ qualifies is $\propto \delta$. The original
$\delta = 5\times10^{-4}\,\text{Å}^{-1}$ was 4–6× tighter than a box integrator, giving 4×
fewer observations per reflection. Widening to $\delta = 2\times10^{-3}\,\text{Å}^{-1}$
(`ewald_dist_cutoff`) restores the multiplicity; the partiality $p$ downweights the
slightly-off-Ewald tails it admits. (Widening is only safe with the de-biased variance of
§1 and the factored objective of §§3–4 — with a per-pixel geometry fit it diverged.)

---

## 3. Term 2 — measured per-resolution profile width $R_1$

$R_1$ is **measured, not fitted**. Fitting $R_1$ inside a per-image least squares is
degenerate with the scale $G$ (a narrower profile and a larger scale trade off), and that
degeneracy slides the per-image scale and wrecks the merge. But a **second moment** is
normalised by the total intensity, so it carries shape information *decoupled from scale*:

$$
R_1^2 = 2\,\langle \epsilon_t^2\rangle,
\qquad
\langle \epsilon_t^2\rangle = \frac{\sum_p (I_p-B)\,\epsilon_{t,p}^2}{\sum_p (I_p-B)} .
$$

We bin the strong spots ($\mathrm{signif}\ge 5$) by resolution ($1/d^2$, 6 bins) and take
the **median** $\langle\epsilon_t^2\rangle$ per bin, so each reflection integrates with the
$R_1$ of its resolution shell (low-res spots are tight; high-res anisotropic streaks are
wider). Weak spots fall back to the global $R_1$. Measuring the width rather than fitting
it is what makes profile-width refinement stable — and it is a selling point: the mask
adapts to the data per shell.

---

## 4. Term 1 — the intensity / scaling residual

The per-pixel least squares is replaced by **one residual per reflection**: the profile-fit
amplitude $J$ (using the Term-2 $R_1$) should equal the scaled reference,

$$
r_h = \frac{J_h - G\,B_\mathrm{DW}\,p_h\,\mathrm{pol}_h\,I^\mathrm{ref}_h}{\sigma_{J,h}},
\qquad
L = \sum_h r_h^2 + \text{(scale prior)} .
$$

Only the per-image scale $G$ and Debye–Waller $B$ are optimised; geometry and $R$ are
fixed. Three consequences:

* **Integration and scaling become one objective.** $J$ *is* the integrated intensity and
  the residual *is* the scaling residual.
* **The empty-pixel problem disappears by construction.** Empty pixels enter only through
  $J$ (with ~zero profile weight); they make no residual of their own and cannot dominate.
* **Fisher weighting puts the reference at maximum leverage.** $\sigma_J$ uses the
  *model-expected* variance $v_p = B + \max(J,0)\,P_p$ (background plus expected signal from
  $I^\mathrm{ref}$), not the observed counts — so a strong *expected* reflection observed
  absent is penalised, and a noise spike with low $I^\mathrm{ref}$ gets no weight.

The scale is regularised towards 1 with a data-scaled weight $w_G=\sqrt{N_\mathrm{refl}/
\sigma_G}$ (mirrors `ScaleOnTheFly`) so weakly-measured images cannot drift and scramble
the merge.

**Geometry is not refined here.** PixelRefine's earlier per-image geometry refinement
(regularised orientation LSQ, signal-weighting, a global orientation/cell sweep) was
removed: on true stills the predictions are already good (radial centroid error ≈ 0,
tangential ≈ a 0.4 px sampling floor that is *not* a recoverable misprediction), and per-image
geometry refinement only overfit the sparse signal. Geometry is the job of `XtalOptimizer`.

---

## 5. Background estimator — mean, not median (both integrators)

**Symptom.** Pushed past the true resolution limit, the no-signal shells reported
$\langle I/\sigma\rangle\approx4\text{–}6$ at $\mathrm{CC}_{1/2}\approx0$ — confident "data"
where there is none. Present in *both* PixelRefine and the classical `BraggIntegrate2D`.

**Cause.** Both used the **median** of the background ring. For a right-skewed (Poisson)
background $\operatorname{median}(B) < \mathbb{E}[B]$, so subtraction under-subtracts by a
tiny but *coherent* per-pixel offset that grows over an $n_\mathrm{pix}$ peak and a
multiplicity-$m$ merge into a fake $\langle I/\sigma\rangle\propto\sqrt{m}$, worst where the
real signal is weakest.

**Fix.** Use the **mean** of the ring (spot cores and saturation sentinels already
excluded). $\langle I/\sigma\rangle$ then collapses to ~0 wherever $\mathrm{CC}\approx0$ and
the honest resolution limit becomes visible. This was the single largest contributor to
untrustworthy σ — a one-line change in each integrator.

---

## 6. Error model (global $a,b$; XDS form)

Counting statistics under-estimate the variance of strong reflections, which carry
systematic errors proportional to $I$, not $\sqrt{I}$. The standard correction inflates the
variance with a **global** two-parameter model, applied at the merge level so both
integrators benefit:

$$
\sigma'^{\,2} = a\,\sigma^{2} + (b\,\langle I\rangle)^{2},
\qquad
\mathrm{ISa} = \frac{1}{b} = \lim_{I\to\infty}\frac{I}{\sigma'} .
$$

The $I^2$ term uses the reflection **mean** $\langle I\rangle$ (not the per-observation
$I_i$, which would bias the merged mean and collapse CC); $a,b$ are fit from the spread of
symmetry equivalents with a relative ($1/\mathrm{dev}^4$) weight so the strong bins (which
fix $b$) do not swamp the weak bins (which fix $a$). `jfjoch_process` prints the model and
ISa.

---

## Results (lysozyme rotation crystal `fixed_master.h5`, treated as stills, 1.7 Å)

| Configuration | N_obs | $\langle I/\sigma\rangle$ | CC$_{1/2}$ | CC$_\mathrm{ref}$ |
|---|---:|---:|---:|---:|
| Baseline per-pixel loss | 799 k | 7.2 | erratic, →0 at 1.7 Å | erratic |
| **Factored Terms 1+2 (this model)** | 1.22 M | 10.7 | **84–92 % flat** | **77–92 % flat** |

The factored objective turns the erratic, high-res-collapsing per-pixel result into flat
~90 % CC$_{1/2}$/CC$_\mathrm{ref}$ to 1.7 Å — matching the proper rotation integration path
from the *stills* path. (See `FINDINGS-2026-06.md`.)

---

## Default recipe

§§1–4 are PixelRefine-specific; §§5–6 act at the integration/merge level and apply to the
classical route too.

| Field / behaviour | Default | Section |
|---|---|---|
| fit/extraction variance | local background $B$ | 1 |
| `ewald_dist_cutoff` | $2\times10^{-3}\,\text{Å}^{-1}$ | 2 |
| tangential width $R_1$ | measured per resolution shell | 3 |
| objective | per-reflection intensity residual, Fisher-weighted | 4 |
| refined parameters | per-image $G$ (and $B$); geometry fixed | 4 |
| `scale_reg_sigma` | 2.0 | 4 |
| local background estimator | **mean** of the ring | 5 |
| merge error model | global $a,b$ (ISa printed) | 6 |

`ewald_dist_cutoff` (multiplicity vs. cost) and `bandwidth` (Si vs. DMM) are the two knobs
worth setting per dataset.