diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 46440e0b..e881e111 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -537,47 +537,37 @@ Per-shell and overall merging statistics are computed on corrected intensities, Completeness requires enumeration of possible reflections given a unit cell and symmetry; where this is not fully available, completeness may be reported as 0 or omitted. +### 10.6 Rotation datasets: combining partials into fulls (3D integration) + +In a rotation scan a reflection is recorded as a series of *partials* spread across the frames its rocking curve crosses. Merging those partials directly would force the merge error model to absorb the rocking-curve slicing as if it were measurement noise, capping the achievable $I/\sigma$. For rotation data Jungfraujoch instead **combines** each reflection's partials into a single *full* intensity first, then scales and merges the fulls — a 3D integration over the rocking curve. + +The combine groups each reflection's partials into rocking events (contiguous runs of frames) and reduces each event to one full: + +- **De-biased weighted sum.** Partials are combined by inverse-variance weighting, where each partial's variance is its background-noise component plus the *model* signal shared across the event (Kabsch profile-fit form). Using the shared model signal rather than the individual down-fluctuating intensity stops weak partials from being over-weighted, which would otherwise inflate the merged error model. The weights depend on the full, so the estimate is iterated. +- **Captured fraction.** The partiality summed over the event, $f=\min(1,\sum_j p_j)$, measures how completely the rocking curve was sampled; it replaces a per-partial minimum-partiality cut, because an event seen over only a few percent of its curve is unreliable however many frames it spans. +- **Capture-aware uncertainty.** A full captured incompletely ($f<1$) is extrapolated and biased high. The unobserved fraction is charged as an extra systematic uncertainty, $\sigma^2 \leftarrow \sigma^2 + \big(c\,(1-f)\,I\big)^2$, so the merge down-weights these extrapolated fulls and the error model treats their scatter as expected. It is enabled by default for the rotation path. + +The fulls are then re-scaled in the XDS sense — a per-image scale refit directly on the complete reflections under the unity partiality model — and merged (§10.4). Because every merged observation is now a counting-statistics-limited full rather than a partiality-divided slice, the error model reaches a far higher asymptotic $I/\sigma$. + --- ## 11. Mosaicity and “profile radius” monitoring -### 11.1 Profile radius (still excitation error width) +### 11.1 Profile radius (intrinsic excitation-error width) -A simple scalar “profile radius” is estimated from indexed spots using the distribution of $\Delta_\mathrm{Ewald}$. Two estimators are available: - -- standard deviation: - $ - R \approx \sqrt{\frac{1}{N}\sum_i \Delta_{\mathrm{Ewald},i}^2}, - $ -- robust MAD-based alternative (median absolute deviation), scaled by 1.4826. - -Operationally, predictions for still data may use a cutoff proportional to this width (e.g. $\Delta_\mathrm{cut}\approx 2R$). - -### 11.2 Mosaicity from rotation data (maximum likelihood) - -For rotation data, Jungfraujoch estimates mosaicity by maximizing a likelihood based on the XDS reflection fraction $R(\tau;\sigma_M/\zeta)$ as described by Kabsch (2010). In brief: - -- for each indexed spot, find its exact Bragg angle and the angular deviation $\tau$ from the frame, -- compute $\zeta$ (the Lorentz/rotation-axis factor) for each reflection, -- maximize $\sum_i \log R(\tau_i;\sigma_M/\zeta_i)$ over $\sigma_M$ (golden-section search). - -This yields a physically meaningful mosaicity estimate that feeds the rotation prediction (§8.3) and the rotation partiality model (§10.2). - -**Search-window caveat (important).** The per-spot Bragg angle is located by searching $\phi$ in a window around the frame. That window must be **wider than the oscillation** — wide enough to admit reflections recorded at large rocking offset (those with $|\tau|$ up to $\sim\sigma_M + \Delta\phi/2$). These tail reflections are exactly the ones that *carry the information about the mosaic width*: if the window is only $\pm\Delta\phi$ (the oscillation), the $\tau$ distribution is truncated at the oscillation width and the MLE **underestimates $\sigma_M$ by roughly $2\times$** (e.g. $0.066°$ instead of the true $0.13°$ on the HEWL test crystal). Jungfraujoch therefore searches $\pm(\Delta\phi + 0.8°)$; the MLE is then insensitive to widening further, because it weights each spot by its recorded fraction $R(\tau)$ which decays for large $|\tau|$. - -**Do not re-refine mosaicity during scaling.** The mosaicity also appears in the per-image scaling fit (§10.1), where it is *degenerate with the per-image scale* $G$ (both rescale the predicted intensity). A free refinement there collapses $\sigma_M$ toward its lower bound, undoing the estimate above. The scaling step therefore holds the indexing mosaicity **fixed**. - -**Why it matters.** A too-small mosaicity has two compounding effects on rotation integration: (i) the prediction (§8.3) admits each reflection on too few frames, truncating its rocking curve; and (ii) the rotation partiality $p$ (§10.2) is over-peaked, so dividing each frame’s partial by $p$ to recover the full amplifies error. The net result is rot3d-combined fulls (§10.4) that are substantially noisier per observation. Correcting the mosaicity is the single largest lever for rotation per-observation precision (asymptotic $I/\sigma$, “ISa”). - -### 11.3 Per-observation precision of rotation fulls (capture-aware weighting) - -In the rot3d combine (§10.4), one reflection’s per-frame partials are weight-summed into a single “full”. A full that was only captured over a fraction $f<1$ of its rocking curve is *extrapolated* — and these under-captured fulls are systematically biased and noisy (the partiality division magnifies error in the tail). Jungfraujoch charges the unobserved fraction as an extra systematic uncertainty, +The “profile radius” is the intrinsic angular width of a reflection — crystal mosaicity plus beam divergence — estimated from the spread of $\Delta_\mathrm{Ewald}$ over indexed spots, $ -\sigma^2 \leftarrow \sigma^2 + \big(c\,(1-f)\,I\big)^2, +R \approx \sqrt{\tfrac{1}{N}\sum_i \Delta_{\mathrm{Ewald},i}^2}. $ -with captured fraction $f=\min(1,\sum_j p_j)$ and coefficient $c$ (`--capture-uncertainty`, default $1.0$ for rot3d). The merge then down-weights the over-extrapolated fulls and the error model treats their scatter as expected, improving both precision (ISa) and accuracy (anomalous peak height). Unlike post-hoc outlier rejection, this corrects a real systematic rather than trading accuracy for $\mathrm{CC}_{1/2}$. +When the beam has a finite energy bandwidth, that bandwidth smears each reflection radially by $\sigma_\mathrm{bw}\approx \mathrm{bandwidth}\cdot\lambda/2d^2$ (largest at high resolution), which also broadens the measured $\Delta_\mathrm{Ewald}$ spread. Since prediction re-applies the bandwidth term per reflection (§8.2), this contribution is deconvolved from the estimate — $R^2 = \langle\Delta_\mathrm{Ewald}^2\rangle - \langle\sigma_\mathrm{bw}^2\rangle$ — so that $R$ is the intrinsic width and bandwidth is not double-counted. Still predictions use an excitation-error cutoff proportional to $R$. -Together with the mosaicity fix (§11.2), this brings the HEWL rotation crystal to roughly the ceiling of XDS’s per-image/absorption correction surfaces; the residual difference to XDS is the finer (azimuthal absorption, per-frame $B$) corrections, which are small here. +### 11.2 Mosaicity from rotation data + +For rotation data the mosaicity $\sigma_M$ is estimated by maximum likelihood from the rocking offsets $\tau$ of indexed spots, using the XDS reflection-fraction model $R(\tau;\sigma_M/\zeta)$ (Kabsch 2010): each spot's exact Bragg angle is located near its frame, $\zeta$ (the rotation-axis Lorentz component) is computed, and $\sigma_M$ is chosen to maximize $\sum_i \log R(\tau_i;\sigma_M/\zeta_i)$. + +The $\phi$ search window for the Bragg angle is set **wider than the oscillation**, so that reflections recorded at large rocking offset are included. These tail reflections carry most of the information about the mosaic width; a window limited to the oscillation range would truncate the $\tau$ distribution and bias $\sigma_M$ low. + +The estimated mosaicity feeds the rotation prediction (how many frames each reflection spans, §8.3) and the rotation partiality (§10.2). It is **held fixed during scaling**: in the per-image scale fit the mosaicity is degenerate with the scale $G$ (both rescale the predicted intensity), so refining it there is unstable. A correct mosaicity matters because it controls both how much of each rocking curve is captured and the partiality used to form fulls (§10.6); too small a value truncates the captured curve and over-peaks the partiality, degrading the combined fulls. ---