Jungfraujoch/docs/CPU_DATA_ANALYSIS.md

# CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for **CPU**- and **GPU**-side real‑time and near‑real‑time data analysis.

**Scope.** The pipeline covered here comprises:

1. geometry mapping and corrections,
2. azimuthal integration (powder/radial profiles),
3. Bragg spot finding (strong pixels → connected components → spot descriptors),
4. indexing (still and rotation modes),
5. Bravais lattice / centering inference,
6. geometry and lattice refinement,
7. reflection prediction (still and rotation),
8. Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
9. scaling and merging,
10. merge-level error modelling and outlier rejection,
11. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
12. amplitude estimation (French–Wilson) and R-free test-set flagging,
13. optional model-based validation: R-free against a supplied model and 2Fo−Fc / Fo−Fc electron-density maps.

## References

The methods are inspired and reuising solutions implemented in:

- W. Kabsch, “XDS”, *Acta Cryst.* **D66** (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
- W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, *Acta Cryst.* **D66** (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
- T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
- J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", *Powder Diffraction* **28** (2013), S339-S350 (detector geometry definition, azimuthal integration)
- H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", *Acta Cryst.* **D55** (1999), 1690-1695 (FFT indexing)
- S. French & K. Wilson, "On the treatment of negative intensity observations", *Acta Cryst.* **A34** (1978), 517-525 (Bayesian amplitude estimation from intensities).
- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free cross-validation).
- M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14).
- J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and [github.com/jonwright/bslz4decoders](https://github.com/jonwright/bslz4decoders) (device-side decoding of bitshuffle+LZ4 images, §0).
(list is not exhaustive)

## 0. Getting the image onto the GPU: device-side bitshuffle+LZ4 decoding

Images arrive bitshuffle+LZ4 compressed (HDF5 filter 32008), and everything from §1 onwards runs on
the GPU when one is present. Instead of decompressing on the host and uploading the image, the
compressed chunk is uploaded — a few MB rather than tens of MB — and decoded on the device. The
approach follows Jon Wright (ESRF); the kernels are Jungfraujoch's own.

Two kernels do the work:

1. **LZ4, one warp per bitshuffle block.** Blocks are independent, so the parallelism is across
   them; within a warp every lane runs the same sequence parser over the same bytes, and the
   literal and match copies are split across the 32 lanes so the stores coalesce. An overlapping
   match is treated as a pattern of period `offset` sourced from bytes that already precede the
   write position, which keeps it parallel rather than a serial byte loop; `offset == 1` (a run of
   one repeated byte, the common case in sparse detector data) and power-of-two offsets avoid the
   modulo altogether. Because the lanes cooperate on the copies, each one is followed by
   `__syncwarp()` — a later match can read bytes another lane wrote, and since Volta that ordering
   is not implicit.
2. **The bitshuffle inverse fused with preprocessing.** One thread owns one group of 8 elements
   across every byte-plane, so once it has transposed its 8 bytes out of each plane it holds 8
   complete elements — and it applies the pixel mask, the error marker and the saturation cap and
   emits 8 finished `int32` pixels directly. The decompressed image is therefore never materialised
   in device memory at all, which removes a frame-sized buffer per worker and a full-frame write
   plus read from the pipeline. Staging nothing in shared memory also means the kernel has no
   dynamic-shared-memory request, so it is indifferent to the bitshuffle block size the file
   declares. For 8-bit images there is a single plane and the assembly degenerates to a copy.

The block offsets inside the container can only be discovered by reading the block lengths in
order, so that scan stays on the host.

Only `BSHUF_LZ4` is decoded on the device. For the zstd variants (`BSHUF_ZSTD`, `BSHUF_ZSTD_RLE`,
`BSHUF_ZSTD_RLE_HUFF`), and for uncompressed or float images, `BSLZ4DecoderGPU::Supports()` returns
false and the pipeline decompresses on the host and uploads as before.

The container arrives off the network or off disk and is not trusted. Everything checkable on the
host — declared sizes, the block scan, a block size that is not a multiple of 8 elements, a block
count the chunk could not hold, trailing bytes — is rejected before any work is queued. What only
the kernel can see is that a block failed to decode to exactly its declared length while consuming
exactly its payload; that raises a device-side flag which becomes an exception once the caller has
synchronised. This matters because the decode buffers are reused frame to frame: a block that
stopped early would otherwise leave the *previous* image's bytes in place, and in the bitshuffled
layout those are the most significant byte-plane, so the result would not look like a missing
corner but like real pixels several powers of two too bright. The kernel reproduces the reference
decoder's length, bounds and parsing-restriction checks; it does not reproduce its exact
fast-loop/safe-loop selection, so a small tail of corrupt streams still decodes here that
`LZ4_decompress_safe` would refuse — as a *complete* decode differing in a few bytes, never as a
partial one.

## 1. Geometry, reciprocal-space mapping, and basic quantities

### 1.1 Coordinate conventions

For a pixel coordinate $(x,y)$ (in pixels), Jungfraujoch converts to a laboratory direction vector via:

1. shift by direct-beam position $(x_\mathrm{beam}, y_\mathrm{beam})$,
2. scale by pixel size $p$ (mm),
3. set detector distance $D$ (mm),
4. apply detector orientation rotation $R_\mathrm{det}$ (PyFAI-like parameterization).

The unnormalized detector coordinate (mm) is:
$
\mathbf{r}_\mathrm{det}(x,y) =
\begin{pmatrix}
(x-x_\mathrm{beam})p\\
(y-y_\mathrm{beam})p\\
D
\end{pmatrix}.
$

The lab-frame vector is:
$
\mathbf{r}_\mathrm{lab} = R_\mathrm{det}\,\mathbf{r}_\mathrm{det}.
$

Let the incident wavevector magnitude be $k = 1/\lambda$ in Å$^{-1}$, and define:
$
\mathbf{S}_0 = (0,0,k).
$

The **reciprocal-space scattering vector** associated with pixel $(x,y)$ is:
$
\mathbf{s}(x,y) = k\,\frac{\mathbf{r}_\mathrm{lab}}{\lVert \mathbf{r}_\mathrm{lab}\rVert} - \mathbf{S}_0.
$

This $\mathbf{s}$ is the fundamental quantity used for spot finding (resolution filters), indexing, and refinement.

### 1.2 Two-theta, azimuth, resolution and $q$

The scattering angle $2\theta$ is computed from $\mathbf{r}_\mathrm{lab}$ via:
$
2\theta = \arctan\!\left(\frac{\sqrt{x_\mathrm{lab}^2 + y_\mathrm{lab}^2}}{z_\mathrm{lab}}\right).
$

Resolution (Å) at a pixel is:
$
d = \frac{\lambda}{2\sin(\theta)} = \frac{\lambda}{2\sin(2\theta/2)}.
$

The magnitude $q = 2\pi/d$ is used for radial binning and ice-ring handling.

### 1.3 Distance from the Ewald sphere

For a reciprocal lattice point $\mathbf{p}$ (Å$^{-1}$), define:
$
\Delta_\mathrm{Ewald}(\mathbf{p}) = \lVert \mathbf{p} + \mathbf{S}_0\rVert - k.
$
Jungfraujoch uses $|\Delta_\mathrm{Ewald}|$ as an operational proxy for excitation error. This appears in:
- still prediction (accept if $|\Delta_\mathrm{Ewald}|\le \Delta_\mathrm{cut}$),
- profile radius estimation (see §11.1),
- still partiality option in scaling/merging (§10.2).

---

## 2. Azimuthal integration (radial profiles)

Azimuthal integration produces a radial profile $I(q)$ or $I(d)$ by histogramming pixels into radial bins. Pixels are **not split** across bins; each pixel contributes wholly to a single bin. By default the profile is purely radial (a single azimuthal bin), but the azimuth can optionally be split into up to 512 $\phi$ sectors (`azim_bins`, `--azim-phi-bins`), giving a **2D $q\times\phi$ profile** that exposes azimuthal anisotropy such as detector shadowing or sample texture.

### 2.1 Histogram estimator

Let bin index $b(x,y)$ be precomputed from $q(x,y)$ (or equivalently from $d(x,y)$) and, when $\phi$ sectors are enabled, the azimuth $\phi(x,y)$ — so $b = b_q + b_\phi B_q$. For each bin $b$:

- accumulate corrected intensity and its square:
  $
  S_b = \sum_{(x,y):\,b(x,y)=b} I(x,y)\,C(x,y),\qquad
  S^{(2)}_b = \sum I(x,y)^2\,C(x,y)^2,
  $
- and count:
  $
  N_b = \#\{(x,y):\,b(x,y)=b \text{ and pixel is valid}\}.
  $

The profile reports both the mean $\bar{I}_b = S_b / N_b$ (when $N_b>0$) and a per-bin sample standard deviation $\sigma_b = \sqrt{(S^{(2)}_b - S_b^2/N_b)/(N_b-1)}$ (a spread/error estimate for each radial point). Invalid pixels (masked, saturated, detector error codes) are excluded.

### 2.2 Corrections applied

Two standard corrections are available:

**(i) Solid angle / geometric correction.** A flat pixel's solid angle falls off with the **incidence angle $\alpha$ between the scattered ray and the detector normal**. With the in-plane detector offsets $u=(x-x_\mathrm{beam})p$ and $v=(y-y_\mathrm{beam})p$ (§1.1) and detector distance $D$,
$
\cos\alpha = \frac{D}{\sqrt{u^2+v^2+D^2}},\qquad
C_\Omega = \cos^3\alpha,
$
applied — like the polarization term below — as a **divisor** (intensities are scaled by $1/\cos^3\alpha$), so pixels at oblique incidence, which subtend a smaller solid angle, are boosted. Because $\alpha$ is evaluated in the detector's own frame it is **invariant under detector tilt** ($\mathrm{rot1}/\mathrm{rot2}/\mathrm{rot3}$), matching PyFAI's `solidAngleArray` and MAX IV azint. It reduces to the commonly quoted $\cos^3(2\theta)$ form only for an untilted detector, where the incidence angle coincides with the scattering angle.

**(ii) Polarization correction.** With polarization coefficient $P$ (beamline dependent) and azimuth $\phi$:
$
C_\mathrm{pol}(2\theta,\phi) =
\frac{1}{2}\left(1+\cos^2(2\theta) - P\cos(2\phi)\left(1-\cos^2(2\theta)\right)\right),
$
applied as a divisor to intensities (i.e. scale by $1/C_\mathrm{pol}$) when enabled.

### 2.3 Background estimate for profiles

A background estimate is derived from the profile as its mean intensity over a fixed low-to-mid $Q$ window (default $2\pi/5$ to $2\pi/3$ Å$^{-1}$). This background is used for monitoring and diagnostics; it is **not** the same as the local Bragg-spot background used in summation integration (§9.2).

---

## 3. Spot finding (strong pixels → Bragg spots)

Spot finding is a two-stage process:

1. **Strong-pixel selection** using intensity and/or local signal-to-noise criteria.
2. **Connected-component labeling (CCL)** to group strong pixels into candidate spots, followed by spot-level filtering and feature extraction.

### 3.1 Strong-pixel detection by local statistics

For each pixel $i$ with value $v_i$, consider a square window (nominally $31\times 31$ pixels) around it. Let the window contain $n$ valid pixels (excluding masked/bad/saturated), and define:
$
\Sigma = \sum v,\qquad \Sigma_2 = \sum v^2.
$

To avoid biasing the local statistics by the test pixel itself, Jungfraujoch evaluates the pixel against the window with the pixel removed:
$
\Sigma' = \Sigma - v_i,\quad \Sigma_2' = \Sigma_2 - v_i^2,\quad n' = n-1.
$

A variance-like quantity proportional to $n'^2$ is formed:
$
V = n'\Sigma_2' - (\Sigma')^2,
$
and the deviation-from-mean quantity:
$
\Delta = v_i n' - \Sigma'.
$

A pixel is considered strong if:
- it is above a photon/count threshold, and
- its window contains enough valid neighbours (more than 100), so the local statistics are meaningful, and
- $\Delta>0$, and
- the squared deviation exceeds a scaled variance:
  $
  \Delta^2 > V\cdot T^2,
  $
  where $T$ is the configured signal-to-noise threshold.

This is equivalent to a local z-score criterion but implemented in integer arithmetic to be robust and fast.

The test is applied in **two passes** over the image. The first is as described above. The second repeats it with every pixel found strong by the first excluded from the local background — it is treated exactly like a saturated pixel, so it contributes to no window it falls into and stays strong itself. This matters for any spot wide enough to reach into its own background box: on a single pass such a spot inflates the mean and variance it is then tested against, and its outer pixels fail the criterion. Excluding the core recovers them, so the spot is reported with its true extent rather than its brightest few pixels. Both the CPU and GPU implementations run these two passes and return the same spot list for the same frame.

Special cases:
- saturated pixels can be forced to “strong” (useful for detecting overloaded Bragg spots),
- invalid pixels are never strong.

### 3.2 Adaptive (self-calibrating) detection

The local-statistics test above needs a fixed photon/count threshold whose correct value depends on the background level, which varies between datasets. The **adaptive** mode (`--adaptive-spots`; the default in `rugnux` and in the viewer for both stills and rotation data, `--no-adaptive-spots` reverts) derives that threshold from each image's own noise, per resolution ring, so no per-dataset value is needed. It admits more spots than the fixed threshold, including genuine reflections that belong to no indexed lattice; these are down-weighted rather than filtered in the per-image geometry fit (§7.4).

Pixels are binned into the same resolution rings as the azimuthal integrator (§2). For each ring a robust background is estimated in three passes: one plain pass over all valid pixels, then two $\sigma$-clipping passes that keep only pixels within $\pm 3\sigma$ of the current ring mean (removing the Bragg peaks from the background estimate). This yields a per-ring background mean $\mu_b$ and scatter $\sigma_b$.

The ring's detection threshold is the larger of two arms,
$
t_b = \max\!\big(\;\mu_b + z\,\sqrt{\sigma_b^2 + \sigma_\mathrm{read}^2}\;,\;\; k_\mathrm{Poisson}(\mu_b, p)\;\big),
$
where $k_\mathrm{Poisson}(\mu_b,p)$ is the smallest count whose Poisson$(\mu_b)$ upper tail is $\le p$. The Poisson arm is correct where the background is countable (a bright low-resolution ring gets a high threshold); the Gaussian arm — floored by a detector-level excess-noise constant $\sigma_\mathrm{read}$ — takes over on near-empty high-resolution rings, where the Poisson arm degenerates to "one photon is significant" and would flood. The operating point $p = E/N$ is set from a single portable knob $E$, the expected number of false pixels tolerated per frame (`--spot-false-pixels`, default 100), with $N$ the number of valid pixels. Because $p$ and every $\mu_b,\sigma_b$ come from the image itself, the same $E$ lands a sensible photon threshold on strong and weak datasets alike, with no per-dataset tuning. Rings too sparse to characterise (fewer than ~40 pixels) fall back to a whole-frame background. A pixel is strong when $v_i \ge t_b$ for its ring (saturated pixels are still forced strong); the strong pixels then feed the same CCL stage (§3.4). The signal-to-noise and photon-count criteria of §3.1 are not used in this mode.

Because detection reads the pixel's ring, a pixel that falls outside the azimuthal-integration $q$ range has no ring and can never be strong: the integration range bounds what adaptive detection can see. Both upper limits are therefore optional and default to the detector itself — the azimuthal integration runs to the highest $q$ any pixel of the detector reaches (`--azim-max-q` unset), and spot finding is not clipped in resolution (`--spot-high-resolution` unset), for rotation data as well as stills. Setting either one narrows detection accordingly — appropriate for weak, high-background data, where the spots admitted at the detector edge are dominated by noise.

**Fused GPU engine.** The per-ring reduction the adaptive threshold needs is the *same* reduction the azimuthal integrator performs. On the GPU path the two are fused into a single image pass (`AdaptiveSpotFinderGPU`): one reduction accumulates the corrected per-ring sums for the azimuthal profile (§2) *and* the raw per-ring statistics for the threshold, after which a light kernel flags the strong pixels. One GPU pass therefore replaces both the separate azimuthal-integration pass and the host-side adaptive spot-finding pass, at a small fraction of the CPU finder's cost per frame and producing the same spot list and azimuthal profile. It is enabled by default in the offline `rugnux` path, the interactive viewer and the online receiver.

**Online.** `spot_finding_settings` in the REST API carries `adaptive_threshold` and `false_pixels_per_frame`, so the mode is reachable from the broker and from the web frontend as well as from `rugnux` and the viewer. It defaults to *off* online, unlike `rugnux` and the viewer, because the broker serves both workflows and only one of them can run it: spots are found in software only on the DECTRIS/SIMPLON path, while the JUNGFRAU and EIGER workflows find them on the FPGA at its own fixed threshold. Setting `adaptive_threshold` on those is refused with an error rather than accepted and ignored, so a detection setting that had no effect cannot be mistaken for one that did.

### 3.3 Resolution and ice-ring handling

Spot finding can be restricted to a resolution range $[d_\mathrm{high}, d_\mathrm{low}]$ by masking pixels outside the range. Optionally, spots in identified ice-ring regions can be tagged so that subsequent indexing/refinement may include or exclude them (see §4 and §6).

A single per-image **ice-ring score** is derived from the azimuthally-integrated radial profile: for each hexagonal-ice powder ring (positions $d$ from Moreau *et al.*, Acta Cryst D77, 2021), the profile intensity at the ring is divided by a smooth background estimated from the *whole* profile — a running median of the non-ice bins, interpolated under each ring — and the strongest ring's ratio is reported (1 = no ice, $>1$ = ice above background). A whole-profile background is used rather than a couple of adjacent shoulder bins so the estimate is robust to the radial binning: at a coarse Q-spacing a local shoulder can be only ~1 bin and would double-count the ring's own edge (offline processing defaults to a fine 0.01 1/Å spacing, `--azim-q-spacing`, so the rings are well resolved). The reported quantity is the ice *magnitude* rather than a significance: with many photons any real ice ring is statistically significant, so significance does not discriminate. It is stored per image (`ice_ring_score`, HDF5 `/entry/MX/iceRingScore`) as a monitoring quantity, distinct from the merge-time ice masking, which is data-driven from the per-ring merged CC1/2.

A further optional safeguard removes isolated high-resolution “spur” spots by detecting large gaps in $1/d$ (or $q$) space and discarding spots beyond the gap. This is intended for macromolecular diffraction where edge-of-detector backgrounds can be extremely low.

### 3.4 Connected-component labeling (CCL)

Strong pixels are grouped into connected components (adjacent strong pixels) using a CCL algorithm. Each component yields a candidate spot with:

- centroid $(x,y)$ (often intensity-weighted),
- pixel count (spot size),
- integrated spot intensity proxy (sum of pixel values),
- resolution $d$ at the centroid (or mean over pixels),
- and quality flags (e.g. ice-ring classification).

Spot-level filters include minimum/maximum pixel count and resolution limits.

The host implementation (`StrongPixelSet::sparseccl`) is the SparseCCL of the ACTS/traccc project: it
runs over the strong pixels sorted row-major, uses a sliding window over the previous line and a
union-find whose root is each component's lowest index. On the GPU the same labelling runs **on the
device** (`SpotExtractorGPU`): the packed strong-pixel bitmask is compacted into that same sorted
list without leaving the card, each pixel finds its at most four earlier 8-neighbours by binary
search, and a lock-free union-find with path halving labels them. Only the finished spot list — a few
hundred entries — comes back to the host, instead of the whole bitmask (2.26 MB per frame at 18 MP).
The two implementations produce the same components, in the same order, with the same pixel counts
and intensities; `tests/SpotExtractorGPUParityTest.cpp` holds them to it. The device version is also
insensitive to frame content: the host sliding window becomes quadratic when many pixels light up in
one detector line — a hot module, or a diffraction ring where it runs tangent to a row — which costs
tens to hundreds of milliseconds on such a frame, while the device version stays under a millisecond.

### 3.5 Adaptive per-image minimum spot size

The minimum-pixels-per-spot filter (§3.4) trades sensitivity against noise: a small value keeps faint one- or two-pixel spots — real signal on strong data, but detector noise on high-background frames — while a larger value keeps only well-formed spots. The best value is dataset-dependent, so for serial-stills indexing it can be chosen **per image** rather than fixed. The frame is indexed three times, at min-pix 3, 2 and 1, and the setting that maximises

$$ \frac{n_\mathrm{indexed}^2}{n_\mathrm{total}} \quad\text{(indexed-spot count weighted by indexed fraction)} $$

is kept; the frame is then integrated once at that min-pix. The fraction factor discounts the extra spots a smaller min-pix admits *unless the lattice actually explains them*, so strong frames keep their real weak spots (extending resolution) while noise-flooded frames stay strict. Because min-pix filters the connected components *after* detection, strong-pixel detection AND the connected-component labelling both run **once** per frame, and the three attempts only repeat the spot-level filter; the azimuthal profile is the one that single detection pass computed. The winning attempt's spot list is kept rather than re-extracted, so the frame that is integrated is exactly the frame that was scored. This is a **stills-only, indexing-path** option — rotation indexing builds one global lattice from all frames and keeps a fixed min-pix. In `rugnux` it is the default; giving an explicit `--min-pix-per-spot` pins a fixed value instead.

---

## 4. Indexing overview

Indexing maps observed reciprocal-space vectors $\mathbf{s}_i$ to a lattice such that:
$
\mathbf{s}_i \approx h_i\mathbf{a}^* + k_i\mathbf{b}^* + l_i\mathbf{c}^*,
$
with integer $(h_i,k_i,l_i)$.

Jungfraujoch supports two complementary indexing strategies:

1. **FFT-based indexing** (Rossmann-type): does not require an a priori unit cell; suitable for unknown samples.
2. **Fast-feedback indexing** (TORO-like): requires an approximate unit cell; optimized for speed and feedback.

Both feed into a common robust refinement/selection stage which maximizes the number of inliers under an indexing tolerance, and which can return **more than one lattice** per image (multi-lattice indexing; see §5.4).

### 4.1 Indexed-spot decision (inlier test)

Given a trial lattice with direct basis vectors $\mathbf{a},\mathbf{b},\mathbf{c}$ (used here as reciprocal-space dot-test vectors), fractional indices are estimated by:
$
h_f = \mathbf{s}\cdot\mathbf{a},\quad
k_f = \mathbf{s}\cdot\mathbf{b},\quad
l_f = \mathbf{s}\cdot\mathbf{c}.
$
Let $(h,k,l)=(\mathrm{round}(h_f),\mathrm{round}(k_f),\mathrm{round}(l_f))$ and define the fractional residual:
$
\delta^2 = (h_f-h)^2 + (k_f-k)^2 + (l_f-l)^2.
$
A spot is indexed if $\delta^2 < \tau^2$, where $\tau$ is the configured tolerance.

For indexed spots, the reciprocal lattice point $\mathbf{p} = h\mathbf{a}^*+k\mathbf{b}^*+l\mathbf{c}^*$ is used to compute $\Delta_\mathrm{Ewald}(\mathbf{p})$ (stored as a diagnostic and later used in profile-radius estimation).

---

## 5. FFT indexing (unknown unit cell)

FFT indexing follows a classical approach: detect dominant periodicities by projecting reciprocal-space points onto many directions and Fourier transforming the resulting 1D histograms.

### 5.1 Directional projections and histograms

Choose a set of unit vectors $\{\mathbf{u}_d\}$ on a half-sphere (a near-uniform distribution generated via a golden-angle construction). For each direction $d$, form a histogram in the scalar projection:
$
t_{id} = \left|\mathbf{u}_d\cdot \mathbf{s}_i\right|.
$

Bin width is chosen approximately as:
$
\Delta t \approx \frac{1}{2 L_\mathrm{max}},
$
where $L_\mathrm{max}$ is the maximum expected real-space unit-cell edge (Å). The histogram extent is tied to the maximum $q$ used (set by a high-resolution cutoff for indexing).

### 5.2 FFT peak picking and candidate vectors

For each direction, the FFT magnitude spectrum is computed; peaks correspond to periodicities along $\mathbf{u}_d$. Each direction yields a candidate real-space length $L$ chosen **not** by raw magnitude but by **maximum prominence above a running-mean local background** (subtracting the broad low-frequency envelope that otherwise dominates on weak or pink-beam frames), subject to $L\ge L_\mathrm{min}$.

Candidate vectors are $\mathbf{v}_d = L_d\,\mathbf{u}_d$.

A collinearity filter removes nearly parallel vectors (e.g. within 5°) and attempts to resolve harmonic ambiguity: shorter “fundamental” vectors may be preferred over longer harmonics if their peak magnitude is sufficiently strong relative to the dominant peak.

### 5.3 Lattice reduction and cell candidates

Triples of candidate vectors are combined to form candidate bases $(\mathbf{A},\mathbf{B},\mathbf{C})$, each reduced to its **Niggli-reduced cell** (Gruber-vector reduction) before comparison, and filtered by allowed length and angle ranges. Two passes are run: a standard pass forms shortest-vector triples from the ~30 strongest filtered directions; if the best cell then indexes fewer than half the spots, a **widened fallback** anchors the two shortest axes and lets the third range over up to ~60 candidate vectors (deduplicated by Niggli cell), catching large, elongated or superstructure cells the first pass misses.

### 5.4 Robust refinement and best-cell selection

Candidate bases are refined against observed spots using an iterative inlier‑focused least‑squares procedure (trimmed/contracting threshold). Candidates are then ranked:
1. more indexed spots wins — **unless** two candidates index within ~10 % of each other, in which case
2. the **smaller-volume** cell is preferred (when the volumes differ by more than ~5 %), avoiding a doubled supercell, then
3. the smaller refinement score, then the spot count again.

Selection is **not limited to a single lattice**: after the best cell is accepted, further lattices are added as separate crystals provided fewer than ~40 % of their indexed spots overlap an already-accepted lattice (up to two extra by default), so split or multi-lattice crystals are indexed rather than discarded.

An optional reference unit cell (if supplied) restricts acceptance to cells within a relative distance tolerance in edge lengths (permutation-invariant).

---

## 6. Bravais lattice / centering inference (“lattice search”)

If the space group is supplied by the user, its lattice constraints are assumed for refinement and subsequent processing.

If not, Jungfraujoch attempts to infer the most plausible Bravais lattice type from the metric tensor after Niggli reduction:

1. **Niggli reduction** is performed to obtain a reduced cell in $G^6$ representation (Gruber vector).
2. The reduced cell is compared against a list of Niggli classes corresponding to Bravais lattices and centerings.
3. The highest-symmetry class that matches within tolerances is selected (relative metric tolerance and angular tolerance).

The output includes:
- a conventional cell,
- crystal system (triclinic, monoclinic, …),
- centering symbol (one of $P, C, I, F, R$; the $A/B$ variants are not emitted here — they are handled only later as prediction absences, §8.4).

This stage provides centering information used for systematic absences in prediction (§8.4) and for reporting.

**Note.** In ambiguous or special cases, forcing space group to $P1$ (no symmetry assumptions) is recommended.

---

## 7. Geometry and lattice refinement

Refinement adjusts experimental geometry and crystal parameters to minimize discrepancies between observed spot reciprocal vectors and those predicted by a lattice model with integer indices.

### 7.1 Parameterization

The refinement jointly optimizes, depending on mode and constraints:

- beam center $(x_\mathrm{beam}, y_\mathrm{beam})$,
- detector distance $D$,
- detector tilt angles (two-angle model; third rotation often held at 0),
- rotation axis direction (for rotation datasets),
- crystal orientation (a global rotation),
- unit-cell parameters, with constraints determined by inferred crystal system.

By default only the beam center, unit cell and crystal orientation are refined; the detector distance, tilt angles and rotation-axis direction are held fixed unless explicitly enabled. A lighter **orientation-only** mode refines just the crystal orientation, for stills whose geometry is already trusted. It carries a weak small-rotation prior penalising the whole angle-axis vector (all three components, at a low weight); what it is there for is the poorly-determined out-of-plane component, which is the one the data barely constrain.

For higher symmetries, constraints are enforced, e.g.
- cubic: $a=b=c,\ \alpha=\beta=\gamma=90^\circ$,
- tetragonal: $a=b$,
- hexagonal: $a=b,\ \gamma=120^\circ$,
- monoclinic (unique axis $b$): $\alpha=\gamma=90^\circ$, $\beta$ refined.

### 7.2 Residuals and objective

For each indexed spot assigned integer $(h,k,l)$, compute:

- observed reciprocal vector $\mathbf{s}_\mathrm{obs}$ from its detector position and current geometry,
- predicted reciprocal vector $\mathbf{s}_\mathrm{pred}(h,k,l;\ \text{lattice params})$.

Residual is:
$
\mathbf{r} = \mathbf{s}_\mathrm{obs} - \mathbf{s}_\mathrm{pred}.
$

A non-linear least squares solver minimizes $\sum \|\mathbf{r}\|^2$ over all selected inlier spots.

### 7.3 Rotation datasets: bringing observations to a common reference frame

For oscillation/rotation data, each image corresponds to a rotation angle $\phi$ about an axis $\mathbf{m}_2$. Observed reciprocal vectors are rotated “back to start” so that all images are refined in a single reference crystal frame:
$
\mathbf{s}_\mathrm{obs,ref} = R(\phi)\,\mathbf{s}_\mathrm{obs},
$
with $R(\phi)$ constructed from the axis-angle representation of the goniometer model. The angle $\phi$ is taken at the centre of each frame's oscillation (the frame angle plus half the oscillation width).

### 7.4 Multi-stage tightening of inlier tolerance

Refinement is performed in stages with decreasing acceptance tolerance for including reflections (three stages, indexing tolerance $0.3\to0.2\to0.1$), which stabilizes convergence when starting from imperfect indexing and approximate geometry.

The loose first stage necessarily admits some spots that are not reflections of this lattice — the fraction of *randomly* placed spots inside a fractional-Miller tolerance $t$ is $\tfrac{4}{3}\pi t^3$, i.e. 11 % at $t=0.3$ — and an unweighted fit lets them pull the orientation. Each residual is therefore weighted by how strong its spot is **for its resolution**: the frame's spots are cut into equal-count resolution shells and each intensity is divided by its shell median, mapped to $w^2=r/(1+r)$. The shell normalisation is what makes this safe — genuine high-resolution spots are legitimately weaker and carry the cell and distance information, so an un-normalised intensity weight would suppress exactly the spots the fit needs. The weight is a property of the spot and never of the current residual, so it does not depend on how far the geometry is from convergence.

### 7.5 Rotation geometry post-refinement (two-pass)

The refinement above (§7.2) runs per image against that image's spots. For rotation data an additional **post-refinement** (on by default; `--rotation-no-postrefine` disables it) improves the detector distance, beam centre and crystal cell/axis using **all** frames at once, then re-integrates:

1. **Pass 1** integrates, scales and merges at the header geometry.
2. From pass-1's integrated reflections, the geometry is refined over all frames (Ceres, robust loss) in **two separate steps** rather than one joint fit:
   - **Step A**: crystal cell scale + goniometer-axis direction, from the observed rotation angles (a distance-independent excitation residual).
   - **Step B**: shared detector distance + beam centre, from the observed spot positions, with the cell held at step A — so the positional residual is no longer degenerate with the cell scale.

   Each step is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual, otherwise left at nominal. The solver bounds the move — distance within ±5 %, beam centre within ±15 px — and detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal.
3. **Pass 2** re-indexes de novo and re-integrates at the committed geometry, reusing pass-1's space group for the merge only. Only the **detector distance and beam centre** carry over: the refined cell and axis are used to make step B well-posed, but pass 2 re-indexes from scratch, so they are not propagated.

The refined pass is written as the canonical `<prefix>_*` output; the pass-1 (header-geometry) result is kept alongside as `<prefix>_01_*` for comparison.

---

## 8. Reflection prediction

Jungfraujoch predicts reflection positions for integration by enumerating Miller indices within a resolution cutoff and accepting those that satisfy a diffraction condition model.

### 8.1 Enumerating reciprocal lattice points

For a maximum resolution $d_\mathrm{min}$, accept $(h,k,l)$ such that:
$
\lVert \mathbf{p}(h,k,l)\rVert^2 = \lVert h\mathbf{a}^* + k\mathbf{b}^* + l\mathbf{c}^*\rVert^2 \le \left(\frac{1}{d_\mathrm{min}}\right)^2.
$

### 8.2 Still prediction (excitation-error cutoff)

For still images, the diffracting condition is approximated by an excitation-error cutoff:
$
\left|\Delta_\mathrm{Ewald}(\mathbf{p})\right| \le \Delta_\mathrm{cut}.
$
Accepted reflections are projected to the detector by intersecting the diffracted direction $\mathbf{S}=\mathbf{S}_0+\mathbf{p}$ with the detector plane, using the current geometry.

When the beam has a finite energy bandwidth, this window is **broadened radially per reflection**: the cutoff is combined in quadrature with a bandwidth smear, $\sqrt{\Delta_\mathrm{cut}^2 + (3\,\sigma_\mathrm{bw})^2}$, where $\sigma_\mathrm{bw}\propto|p_z|$ (the reciprocal-space depth along the beam, growing as $\sim 1/d^2$). This keeps high-resolution reflections — smeared by the bandwidth into radial streaks — from being clipped. The same $\sigma_\mathrm{bw}$ is deconvolved from the measured profile radius (§11.1), so it is not double-counted.

### 8.3 Rotation prediction (Laue equation + partiality model)

For rotation/oscillation datasets, Jungfraujoch solves for rotation angles $\phi$ where the rotated reciprocal lattice point satisfies the Ewald-sphere condition. In an XDS-like notation, define:

- rotation axis unit vector $\mathbf{m}_2$,
- $\mathbf{S}_0$ incident vector,
- $\mathbf{S}(\phi)=\mathbf{S}_0+\mathbf{p}(\phi)$.

A key quantity is:
$
\zeta = \left|\mathbf{m}_2\cdot \mathbf{e}_1\right|,\quad
\mathbf{e}_1 = \frac{\mathbf{S}\times \mathbf{S}_0}{\lVert \mathbf{S}\times \mathbf{S}_0\rVert},
$
which also appears in XDS as the Lorentz component linked to the rotation axis.

A Gaussian mosaicity model yields a partiality fraction over an oscillation width $\Delta\phi$:

$ P(\phi;\sigma_M,\zeta,\Delta\phi) = \frac{1}{2}\left[\mathrm{erf}\!\left(\frac{\phi+\Delta\phi/2}{\sqrt{2}\,\sigma_M/\zeta}\right) - \mathrm{erf}\!\left(\frac{\phi-\Delta\phi/2}{\sqrt{2}\,\sigma_M/\zeta}\right)\right], $

with mosaicity $\sigma_M$ in radians.

Reflections are predicted if they meet minimum $\zeta$ and mosaicity-window criteria, and their predicted detector coordinates fall on the active detector area.

### 8.4 Systematic absences (centering)

Systematic absences are applied at the centering level (prior to full space-group symmetry) **when the space group is supplied by the user**. With no user-fixed space group, prediction runs in $P$ regardless of the centering the lattice search inferred: the centering-absent reflections are integrated so that the space-group search (§13) can confirm or disprove the centering from the measured intensities, and so that a missed superstructure shows up. For centering symbol $C$:

- $I$: absent if $h+k+l$ odd,
- $A$: absent if $k+l$ odd,
- $B$: absent if $h+l$ odd,
- $C$: absent if $h+k$ odd,
- $F$: absent if any of $h+k, h+l, k+l$ is odd,
- $R$: absent if $(-h+k+l)\bmod 3 \ne 0$,
- $P$: no centering absences.

---

## 9. 2D Bragg integration (profile fitting over a three-ring ROI)

Jungfraujoch integrates each predicted reflection in the detector plane over a CrystFEL-inspired “three-ring” region of interest (§9.1). The **default** extraction is **profile fitting** (Kabsch; §9.3), which weights each pixel by a fitted spot profile and so recovers weak reflections far better than plain summation; plain box summation (§9.2) is retained as the seed for the profile and as a fallback. Both methods share the same ROI and background model, and emit the same per-reflection $(I,\sigma,\text{partiality},d)$, so scaling, the rotation combine (§10.6) and merging consume either unchanged.

### 9.1 Regions of interest

For each predicted reflection at $(x_p,y_p)$, define three radii:

- $r_1$: inner signal radius,
- $r_2$: inner background radius,
- $r_3$: outer background radius.

Pixels are classified by their squared distance $r^2=(x-x_p)^2+(y-y_p)^2$:

- **signal region:** $r^2 < r_1^2$,
- **background annulus:** $r_2^2 \le r^2 < r_3^2$.

Invalid pixels (masked/bad/saturated) are excluded from both sums. In addition, pixels lying inside the signal disk ($r<r_2$) of any *other* predicted reflection are removed from this reflection's background annulus, so a neighbouring spot cannot leak into the background estimate.

### 9.2 Box summation (seed and fallback)

Let:
- $S = \sum I(x,y)$ over signal pixels,
- $n_S$ = number of valid signal pixels,
- $B = \sum I(x,y)$ over background pixels,
- $n_B$ = number of valid background pixels.

Background per pixel and integrated intensity:
$
\hat{b} = \frac{B}{n_B},\qquad
\hat{I} = S - n_S \hat{b},
$
with a Poisson-like uncertainty $\sigma(\hat{I})=\max\!\big(1,\ r_\sigma\hat{I},\ \sqrt{S}\big)$, i.e. $\sqrt{S}$ floored both at 1 and at a small fraction $r_\sigma$ of the intensity. A reflection is accepted as “observed” only if all signal pixels were valid and $n_B$ exceeds a minimum. This box sum is the classical estimator; it is used directly with `--integrator boxsum`, and otherwise seeds the profile fit below.

**Trimmed-mean background (monochromatic, default on).** On monochromatic data — rotation *and* still (the discriminator is the beam, not the acquisition mode) — the ring mean $\hat{b}$ is by default replaced with a **symmetric trimmed mean**: the ring pixels are sorted, the lowest and highest fraction $f$ are dropped, and the central $(1-2f)$ are averaged ($f=0.10$ by default, `--background-trim`; $f=0$ restores the plain mean). Because $\hat{I}=S-n_S\hat{b}$ is a small difference of large numbers for weak reflections, a per-pixel background bias $\delta\hat{b}$ becomes a *fractional* intensity bias $\approx n_S\,\delta\hat{b}/\hat{I}$ that grows as $\hat{I}$ shrinks — worst at the resolution edge. The plain mean reads high there because neighbour-spot wings that survive the signal-disk mask, tails and zingers are one-sided (positive) contaminants; dropping the extreme ring pixels removes that bias while a clean Poisson ring is essentially unchanged. The trimmed estimate has a slightly higher variance than the plain mean, so the gain is at the resolution edge and the cost, where any, is in already-clean shells. It is applied to the shared background used by both the box sum and the profile fit.

For the **profile-fit path on broadband (non-zero bandwidth: pink-beam / DMM) data**, the trimmed mean is *not* used; instead the background mean is computed with a single high-outlier reject (drop ring pixels above $\hat{b}+3\sqrt{\hat{b}}$, then recompute): a bandwidth-streaked high-resolution spot or a close neighbour can leak into the ring and bias the mean high, over-subtracting and driving weak high-resolution intensities negative. A clean Poisson background is essentially unchanged by the cut. This $\sigma$-clip is the one robustification that is *not* applied to plain box summation (`--integrator boxsum`); the trimmed mean above is computed in the shared background pass and so applies to box summation too.

### 9.3 Profile-fitted extraction (default)

A fixed signal disk captures a *width-dependent* fraction of each spot, which puts a multiplicative floor on the per-observation precision of strong reflections and weights weak reflections poorly. Profile fitting removes this by extracting each intensity against a fitted spot shape, without needing reference intensities. Per frame:

1. **Seed.** Box-sum every reflection (§9.2) to get a rough intensity and observed centroid, and select strong spots (significance $\ge 5$).
2. **Build the profile.** For `gaussian` (the default) the width is taken **per resolution shell** from the measured second moment of the strong spots (shell-dependent because spot size grows with resolution); the intrinsic spot is treated as round in the detector plane, the crystal's own anisotropy lying in the rocking direction, which a 2D detector-plane profile does not sample. For `empirical` the profile is instead the averaged, background-subtracted pixel grid of the shell's strong spots, accumulated on their **rounded predicted** positions. For `gaussian` only, the profile is then **rebuilt for each reflection**, centred on its **sub-pixel predicted position** (the noise-free geometric centre, not the observed centroid) and, where needed, **elongated only along the radial direction** (away from the beam centre) — because two effects stretch a spot radially but not tangentially:
   - a finite energy **bandwidth** smears each spot by $\sigma_\mathrm{bw}=\text{bandwidth}\cdot R_\mathrm{px}$ ($R_\mathrm{px}$ = distance from the beam centre, large at high resolution), and
   - sensor **parallax** — the depth over which a photon converts in a thick Si/CdTe sensor — adds a term $\propto\tan^2(2\theta)$ (material- and energy-dependent), plus, on the monochromatic path, a small fixed weak-spot capture term.

   These combine as $\sigma^2_\mathrm{radial}=\sigma^2_\mathrm{intrinsic}+\sigma_\mathrm{bw}^2+c_\mathrm{par}\tan^2(2\theta)$ (tangential unchanged), on a grid grown to hold the streak — capturing it without the tangential background an isotropic widening would add. The `empirical` profile keeps the fixed per-shell grid and gets none of this.
3. **Fit (Kabsch).** With profile $P$, background $B$ and the shell variance model, the intensity and its uncertainty are
$
I = \frac{\sum P\,(c-B)/v}{\sum P^2/v},\qquad
\sigma = \sqrt{\frac{1}{\sum P^2/v}},\qquad
v = B + \max(I,0)\,P,
$
where $c$ is the pixel value and the de-biased variance $v$ (background plus model signal, rather than the down-fluctuating observed count) is iterated (a few passes). As a guard, if the profile intensity runs away from the box-sum seed (by more than ~10 box-sum $\sigma$) it falls back to the seed, and the variance floors the background at $1/12$ (the integer-binning pixel-variance floor). The rotation/excitation partiality is carried exactly as in the box-sum path.

The integrator is selected by `--integrator boxsum|gaussian|empirical` (default `gaussian`).

### 9.4 Lorentz–polarization factor handling

For integrated reflections, polarization correction can be applied as a multiplicative correction to the reflection scale via the geometry-based polarization term (§2.2). A Lorentz-like factor is carried as `rlp` in predictions, and used during scaling/merging (§10).

---

## 10. Scaling and merging

After per-image integration, Jungfraujoch scales observations and merges them into unique reflections. The design is intentionally compatible with XDS/XSCALE concepts, and handles both still and rotation data.

### 10.1 Observation model

For an observation $j$ of a unique reflection $h$ on image (or image group) $i$, the predicted measured intensity is modeled as:
$
I_{ij} \approx G_i \, L_{ij}\, P_{ij}\, I_h,
$
where:

- $G_i$ is the image scale factor,
- $L_{ij}$ is a Lorentz-like / geometry factor; predictions carry its **reciprocal** as `rlp`, so $L = 1/\texttt{rlp}$ and the correction below is applied as a multiplication by `rlp`,
- $P_{ij}$ is a partiality term (model-dependent),
- $I_h$ is the merged (true) intensity parameter for that unique reflection.

A least-squares objective is minimized:
$
\sum_{ij} \left(\frac{I_{ij}^{\mathrm{pred}} - I_{ij}^{\mathrm{obs}}}{\sigma_{ij}}\right)^2
$
solved by robust (Cauchy) weighted least squares, with optional post-fit smoothing of the per-frame scales for rotation series (§10.3).

### 10.2 Partiality models

The partiality applied is fixed by the data type and scaling stage, not chosen from a user menu:

1. **Rotation partiality** (XDS-like; see §8.3), used for the per-frame scaling of rotation partials:
   $
   P_{ij} = \frac{1}{2}\left[
   \mathrm{erf}\!\left(\frac{\Delta\phi_{ij}+\Delta\phi/2}{\sqrt{2}\,\sigma_{M,i}/\zeta_{ij}}\right) -
   \mathrm{erf}\!\left(\frac{\Delta\phi_{ij}-\Delta\phi/2}{\sqrt{2}\,\sigma_{M,i}/\zeta_{ij}}\right)
   \right].
   $
   The mosaicity $\sigma_{M,i}$ is **measured once per image at indexing** (MLE, §11.2) and held fixed during scaling — only smoothed in frame order (§10.3), never re-refined (it is degenerate with the scale $G$; §11.2).

2. **Unity** ($P_{ij}=1$): used for the scale-on-fulls refit (§10.6), where each observation is already a complete reflection.

3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius (resolution-independent), the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature. The fit typically drives $\gamma_e\to0$, leaving the resolution-independent $\gamma_0$ as the effective width. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times.

Reflections below a minimum partiality can be rejected from merging to avoid unstable corrections.

### 10.3 Smoothing of per-frame scales

The per-frame scales $G_i$ are fit by robust (Cauchy) inverse-variance-weighted ratios; there is no explicit $G\approx1$ prior. For rotation datasets, optional smoothing enforces the expectation that scale and mosaicity vary slowly across a sweep: **after** the per-frame fit, $\log G_i$ (and the mosaicity) are replaced by a centred **moving average** over a window spanning a configurable rotation range (XDS DELPHI-like; `--smooth-g`, default 5° for rot3d, off otherwise). It is a post-fit smoothing pass, not a curvature penalty inside the least-squares objective.

A per-frame scale enters every intensity as $1/G$, so a frame whose fit is not determined by its data can amplify it without bound — and $\sigma$ is amplified by the same factor, which makes it invisible to any $\sigma$-based outlier test. A fitted $G$ far below the run's median is therefore treated as *undetermined* rather than as a successful fit, both here and in the separate refit on the combined fulls (§10.6). The bound is a ratio to the run's own median because $G$ is not gauge-fixed: it and the merged means have an exact global multiplicative degeneracy, so no absolute value is meaningful.

### 10.4 Merging estimator

After refinement, corrected observations are formed:
$
I^{\mathrm{corr}}_{ij} = \frac{I^{\mathrm{obs}}_{ij}}{G_i L_{ij} P_{ij}},\qquad
\sigma^{\mathrm{corr}}_{ij} = \frac{\sigma^{\mathrm{obs}}_{ij}}{G_i L_{ij} P_{ij}}.
$

Unique intensities are merged by inverse-variance weighted mean:
$
I_h = \frac{\sum_j w_j I^{\mathrm{corr}}_{ij}}{\sum_j w_j},\qquad
w_j = \frac{1}{(\sigma^{\mathrm{corr}}_{ij})^2}.
$

The weights use an **expected** variance: the Poisson signal part of each $\sigma^{\mathrm{corr}}_{ij}$ is rebuilt at the reflection's merged $\langle I\rangle$ rather than at that observation's own intensity. Weighting by an observation's own $\sigma^2$ biases the inverse-variance mean low below about one photon, because an up-fluctuated observation gets a larger sigma and is then down-weighted too hard. The rotation combine already does this; for stills it is on by default, and `--no-expected-variance-merge` restores the observed-sigma weighting.

An internal-consistency term can inflate uncertainties when multiple observations are present, in the spirit of XSCALE.

### 10.5 Merging statistics

Per-shell and overall merging statistics are computed on corrected intensities, including:
- number of observations and of unique reflections, and multiplicity,
- mean $I/\sigma(I)$,
- $R_\mathrm{meas}$ (the redundancy-independent Diederichs–Karplus form) from within‑HKL deviations,
- $\mathrm{CC}_{1/2}$ (half-set correlation) and, when a reference dataset is supplied, $\mathrm{CC}_\mathrm{ref}$,
- completeness against the enumerated reflections for the cell and symmetry,
- the anomalous signal-to-noise $\mathrm{SigAno}$ (below).

The error model is refined as $\sigma_\mathrm{corr}^2 = a\,\sigma^2 + (b\,\langle I\rangle)^2$, with $a$ set by the scatter of weak (counting-limited) reflections and $b$ the intensity-proportional systematic scatter of the strong ones. On the **rotation** path, **ISa** is the asymptotic ($I\to\infty$) signal-to-noise — by definition the reproducibility limit of the strongest reflections (Diederichs, *Acta Cryst.* **D66** (2010) 733) — and is read directly from the strong symmetry equivalents as the counting-subtracted fractional scatter of well-measured reflection groups (a robust median over strong groups; the $I/\sigma$ threshold is relaxed on weak or radiation-damaged data that has few strong reflections), rather than as $1/b$ of the whole-range fit, whose $b$ is raised slightly by an intermediate-intensity excess and so understates the limit. The reported ISa and the merged-intensity systematic floor $\sigma \ge b_\mathrm{ISa}\,|I|$ both use this asymptotic value, so a high-multiplicity merged $I/\sigma$ approaches ISa; the per-observation $\sigma_\mathrm{corr}$ (the merge weights) uses the whole-range $a,b$ and is unchanged. The **stills** path has no asymptotic estimate: it reports $\mathrm{ISa}=1/b$ and floors the merged $\sigma$ with the same whole-range $b$.

**Anomalous signal-to-noise (SigAno).** The strength of the anomalous signal is reported per shell and overall as $\mathrm{SigAno}=\langle|\Delta I|\rangle / \langle\sigma(\Delta I)\rangle$, where $\Delta I = I(+)-I(-)$ over acentric reflections measured in both Bijvoet hands and $\sigma(\Delta I)=\sqrt{\sigma_+^2+\sigma_-^2}$. It is computed from the **full-multiplicity** inverse-variance $I(+)/I(-)$ split (the same one written to the output), i.e. from all observations rather than a half-set. For pure noise $\mathrm{SigAno}$ approaches the half-normal value $\sqrt{2/\pi}\approx0.8$, and it rises above $1$ once a real anomalous difference is present. A half-set anomalous correlation ("$\mathrm{CC}_\mathrm{anom}$") is **not** reported. Its two half estimates $\Delta I_0,\Delta I_1$ are complementary partitions of one observation pool ($\Delta I_0+\Delta I_1=2\,\Delta I_\mathrm{full}$), and subtracting the two Bijvoet hands cancels the large common intensity that keeps $\mathrm{CC}_{1/2}$ non-negative, leaving the small anomalous signal against the per-half split noise; below an anomalous signal-to-noise of $1$ per half that correlation tends towards $-1$ rather than $0$. $\mathrm{SigAno}$ has no such floor. It is emitted only when an anomalous split was made, using the standard PDBx items `_reflns.pdbx_absDiff_over_sigma_anomalous` (overall) and `_reflns_shell.pdbx_absDiff_over_sigma_anomalous` (per shell), and appears as the `SigAno` column of the printed merge-statistics table.

### 10.6 Rotation datasets: combining partials into fulls (3D integration)

In a rotation scan a reflection is recorded as a series of *partials* spread across the frames its rocking curve crosses. Merging those partials directly would force the merge error model to absorb the rocking-curve slicing as if it were measurement noise, capping the achievable $I/\sigma$. For rotation data Jungfraujoch instead **combines** each reflection's partials into a single *full* intensity first, then scales and merges the fulls — a 3D integration over the rocking curve.

The combine groups each reflection's partials into rocking events (contiguous runs of frames) and reduces each event to one full:

- **De-biased weighted sum.** Partials are combined by inverse-variance weighting, where each partial's variance is its background-noise component plus the *model* signal shared across the event (Kabsch profile-fit form). Using the shared model signal rather than the individual down-fluctuating intensity stops weak partials from being over-weighted, which would otherwise inflate the merged error model. The weights depend on the full, so the estimate is iterated.
- **Captured fraction.** The partiality summed over the event, $f=\min(1,\sum_j p_j)$, measures how completely the rocking curve was sampled. A full whose curve was captured below a threshold (`--min-captured-fraction`, default 0.7 for rotation) is dropped — an event seen over only a small fraction of its curve is unreliable however many frames it spans. (The per-partial minimum-partiality cut of §10.2 still applies upstream, in the per-frame scaling.)
- **Per-image rejection (opt-in).** A frame whose observations correlate poorly with the merged reference is not measuring the crystal being merged — it may be off-crystal, or on a *different* crystal where two lattices occupy separate regions of the sample. `--min-image-cc` drops such frames. It has no default: the per-frame correlation measures data quality as much as frame validity, and its typical level varies widely between datasets, so no single absolute bound is generally valid.
- **Capture-aware uncertainty.** A full captured incompletely ($f<1$) is extrapolated and biased high. The unobserved fraction is charged as an extra systematic uncertainty, $\sigma^2 \leftarrow \sigma^2 + \big(c\,(1-f)\,I\big)^2$, so the merge down-weights these extrapolated fulls and the error model treats their scatter as expected. It is enabled by default for the rotation path.

The fulls are then re-scaled in the XDS sense — a per-image scale refit directly on the complete reflections under the unity partiality model — and merged (§10.4). Because every merged observation is now a counting-statistics-limited full rather than a partiality-divided slice, the error model reaches a far higher asymptotic $I/\sigma$.

After scale-fulls, three **correction surfaces** are fitted on the combined fulls (rotation path, **on by default**; disable all with `--no-scaling-corrections`), each an alternating multiplicative refinement of the per-full scale against the merged reference:

- **Decay.** Radiation damage weakens later frames more at higher resolution — a resolution×time (Debye–Waller) systematic the resolution-flat per-image scale cannot capture. A single global relative-$B$ rate is fitted, $\ln(I_\mathrm{ref}/I_\mathrm{obs}) = 2\,(\mathrm{d}B/\mathrm{d}n)\,(n-\bar n)\,s^2$ (frame $n$, $s^2 = 1/4d^2$), and folded into the scale. It engages only when the total relative-$B$ over the run exceeds a physical floor (2 Ų); below that the decay is negligible and "correcting" it only spreads symmetry equivalents (same $s^2$, different frames). An optional **per-batch relative-$B$** (`--relative-b[=deg]`, off unless requested; 10°-of-rotation batches by default) extends the single global rate to a smooth $B(n)$ curve — the same $s^2$-weighted decay fit solved independently over short frame batches, curvature-penalized so it cannot over-fit and cross-validated like the surfaces below — for crystals whose decay is non-linear in dose. Its cross-validation splits on **ASU-group parity**, not the frame parity the surfaces below use: a per-batch parameter owns whole frames and so cannot be scored on a held-out frame, whereas splitting the symmetry equivalents tests whether a batch's $B$ generalises to reflections it was not fitted on.
- **Absorption.** A smooth multiplicative factor over the diffracted-beam direction expressed in the goniometer (crystal) frame: each full's predicted detector position gives the lab diffracted direction, de-rotated by the spindle so a fixed crystal-frame direction is sampled at many rotation angles and its grid cell is well-determined. Negligible at hard X-rays / thin crystals; it matters at low photon energy.
- **Modulation** (detector-plane flat-field). A smooth multiplicative factor over where each reflection lands on the detector (predicted $x,y$): symmetry-equivalents land at different positions as the crystal rotates, over-determining the surface. It absorbs detector-response and geometric systematics that inflate $R_\mathrm{meas}$.

Each surface is **cross-validated**: fitted on even-numbered frames and kept only if it improves the held-out odd-frame agreement by a clear margin (and vice versa), scored by a **σ-independent, $R_\mathrm{meas}$-like** fractional agreement $\sum|I_s-I_\mathrm{ref}|/\sum|I_\mathrm{ref}|$ rather than a studentized $\chi^2$ — so a surface cannot "pass" by reshaping the sigmas instead of tightening the intensities. A surface fitted to noise where its systematic is absent does not generalize and is discarded — a correction never adds scatter.

**Radiation-damage report (rotation, report-only).** Independently of whether any decay correction is applied, rugnux measures and reports the relative Debye–Waller $B$ across the sweep: the per-image scale's correlation to the merge and the per-image mosaicity versus frame (dose), together with a per-batch relative-$B$ curve whose first→last change is a single headline number (measured before any decay correction, against the least-damaged early wedge). It is written to the log and to the merged mmCIF as a data-quality-vs-dose diagnostic and **never** alters the merged intensities — distinct from the decay correction above, which does fold into the scale.

### 10.7 R-free test-set flags

A fraction of the unique reflections (`rfree_fraction`, default 0.05) is flagged as a **free (test) set**, written to the output (MTZ `FreeR_flag`, mmCIF `_refln.status_free`, a text-HKL column) for model validation (§14) and for downstream refinement. The flag is a pure function of the reflection's **Friedel-merged (Laue) ASU index**, which gives three properties:

- all symmetry- and Friedel-equivalent reflections share one flag — in particular a Bijvoet pair $I(+)/I(-)$, kept as two separate merged rows in anomalous mode, is **never split** across the work and free sets (which would bias R-free);
- the free/work decision is a deterministic hash of that key, so the same reflection always lands in the same set — reproducible run-to-run and independent of the order in which observations were merged;
- the hash depends only on the reflection index, **not** on this dataset's resolution range or which reflections it happens to contain, so a uniform draw takes ~`rfree_fraction` of the distinct reflections free and — crucially — **every dataset of one crystal form gets the same free set**. That cross-dataset consistency is what a multi-dataset campaign (ensemble refinement, PanDDA) requires; a per-shell stratification tied to each dataset's own $d_\mathrm{min}$ would break it.

On small data, where `rfree_fraction` (default 0.05) would give too few test reflections for a statistically stable R-free (Brünger's ~500–2000 rule), the fraction is **floored** so at least ~500 distinct reflections are free — capped at 10 % so a large test set never steals working data. For ordinary data this floor is inactive and the fraction stays flat at `rfree_fraction`, preserving the cross-dataset-identical property above; it only lifts the fraction on genuinely small datasets, where per-dataset R-free stability outweighs cross-dataset identity (and a shared reference set is the way to keep exact identity there).

When a reference MTZ (`--reference-mtz`) carries a `FreeR_flag` column, its test set is **imported** instead: every merged reflection whose Laue-ASU index matches the reference takes the reference's flag (reflections absent from the reference keep the hash flag). This lets a whole fragment-screening campaign inherit one shared free set from the apo/reference dataset. The CCP4/refmac convention (test set = flag 0, including the historical 0–19 form) is assumed, with the complement taken automatically if flag 0 would be the majority (a phenix-style file where 1 marks free).

### 10.8 French–Wilson amplitudes

The last step of the merge estimates a Bayesian structure-factor amplitude $|F|$ for each unique reflection from its intensity $I$ and error $\sigma$, so the output carries amplitudes alongside intensities (a naïve $\sqrt{\max(I,0)}$ turns every weak or negative measurement into a biased — or zero — amplitude). With the Wilson prior for the true intensity $J\ge 0$ at that resolution,

$
P_\mathrm{acentric}(J) \propto e^{-J/\Sigma},\qquad
P_\mathrm{centric}(J) \propto J^{-1/2}\,e^{-J/2\Sigma},
$

and a Gaussian likelihood $\mathcal{N}(I;J,\sigma^2)$, the posterior mean amplitude and its uncertainty are

$
\langle |F|\rangle = \frac{\int_0^\infty \sqrt{J}\,\mathcal{N}(I;J,\sigma^2)\,P(J)\,\mathrm{d}J}{\int_0^\infty \mathcal{N}(I;J,\sigma^2)\,P(J)\,\mathrm{d}J},\qquad
\sigma_F = \sqrt{\langle J\rangle - \langle|F|\rangle^2}.
$

The prior mean is $\Sigma = \varepsilon\,\langle I/\varepsilon\rangle_\mathrm{shell}$, where $\varepsilon$ is the reflection's epsilon (symmetry-enhancement) multiplicity and $\langle I/\varepsilon\rangle$ is the Wilson mean in its resolution shell (so reflections on symmetry elements, and each shell, are treated correctly). Strong reflections ($I>4\sigma$) short-circuit to $|F|=\sqrt{I}$, where the French–Wilson bias is negligible; a reflection with an unusable $I/\sigma$ falls back to $\sqrt{\max(I,0)}$. The integral is evaluated numerically with a log-shift for stability.

Amplitudes are written as MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`/`F_meas_sigma_au`, and appended to the text HKL, alongside the intensity columns. The **same** $|F|$ feed the model-validation step (§14), so the reflection file and the maps use one consistent set of amplitudes.

### 10.9 Reference data: fixing the space group and resolving the indexing ambiguity

A reference dataset (`--reference-mtz`) supplies known intensities for the same crystal form, and is used in two ways.

**Fix the space group and cell.** Unless overridden on the command line (`-S` for the space group, `-C` for the cell), the reference's space group is adopted and its cell is used as the soft reference cell — indexing may still drift the cell within tolerance, so a small mismatch between reference and data is absorbed rather than rejected. This applies to both stills and rotation data.

**Resolve the indexing (merohedral) ambiguity.** When the lattice symmetry is higher than the crystal's Laue symmetry (e.g. $P3$, $P4$, $P6$, $C2$), more than one indexing of the same lattice is geometrically valid, and the two solutions produce *different* merged intensities that a self-consistent scale cannot tell apart — only an external reference can. The candidate reindexings are the identity together with the twin-law cosets of the metric symmetry (from the unit-cell metric and the Laue group); each is scored by the intensity correlation $\mathrm{CC}_\mathrm{ref}$ of the reindexed merge against the reference, and the data are re-merged in the best-correlating indexing. The reindex is **metric-preserving** — only the $hkl$ labels change, the cell is unchanged — and it is a no-op for a holohedral crystal, which has no twin laws (the lattice and Laue symmetry coincide). For rotation data this is done once, after the space group is determined; the reference is *not* used to scale the rotation merge, which stays self-consistent (its $\mathrm{ISa}$ comes from the data alone). For stills the reference is the per-image scale target of the on-the-fly scaling (§10.2).

---

## 11. Mosaicity and “profile radius” monitoring

### 11.1 Profile radius (intrinsic excitation-error width)

The “profile radius” is the intrinsic angular width of a reflection — crystal mosaicity plus beam divergence — estimated from the spread of $\Delta_\mathrm{Ewald}$ over indexed spots,
$
R \approx \sqrt{\tfrac{1}{N}\sum_i \Delta_{\mathrm{Ewald},i}^2}.
$
When the beam has a finite energy bandwidth, that bandwidth smears each reflection radially by $\sigma_\mathrm{bw}\approx \mathrm{bandwidth}\cdot\lambda/2d^2$ (largest at high resolution), which also broadens the measured $\Delta_\mathrm{Ewald}$ spread. Since prediction re-applies the bandwidth term per reflection (§8.2), this contribution is deconvolved from the estimate — $R^2 = \langle\Delta_\mathrm{Ewald}^2\rangle - \langle\sigma_\mathrm{bw}^2\rangle$ — so that $R$ is the intrinsic width and bandwidth is not double-counted. Still predictions use an excitation-error cutoff proportional to $R$.

### 11.2 Mosaicity from rotation data

For rotation data the mosaicity $\sigma_M$ is estimated by maximum likelihood from the rocking offsets $\tau$ of indexed spots, using the XDS reflection-fraction model $R(\tau;\sigma_M/\zeta)$ (Kabsch 2010): each spot's exact Bragg angle is located near its frame, $\zeta$ (the rotation-axis Lorentz component) is computed, and $\sigma_M$ is chosen to maximize $\sum_i \log R(\tau_i;\sigma_M/\zeta_i)$.

The $\phi$ search window for the Bragg angle is set **wider than the oscillation**, so that reflections recorded at large rocking offset are included. These tail reflections carry most of the information about the mosaic width; a window limited to the oscillation range would truncate the $\tau$ distribution and bias $\sigma_M$ low.

The estimated mosaicity feeds the rotation prediction (how many frames each reflection spans, §8.3) and the rotation partiality (§10.2). It is **held fixed during scaling**: in the per-image scale fit the mosaicity is degenerate with the scale $G$ (both rescale the predicted intensity), so refining it there is unstable. A correct mosaicity matters because it controls both how much of each rocking curve is captured and the partiality used to form fulls (§10.6); too small a value truncates the captured curve and over-peaks the partiality, degrading the combined fulls.

---

## 12. Auxiliary statistics: ⟨I/σ(I)⟩ and Wilson plot

### 12.1 Per-shell ⟨I/σ(I)⟩

For monitoring integration quality, Jungfraujoch reports mean $\langle I/\sigma(I)\rangle$ in a fixed number of resolution shells. Shelling is performed in $1/d^2$ space (typical of crystallographic practice).

### 12.2 Wilson plot (B-factor proxy)

A Wilson-type analysis is computed by binning intensities by resolution and fitting:
$
\langle I\rangle \propto \exp\!\left(-\frac{B}{2}\frac{1}{d^2}\right),
$
i.e.
$
\log \langle I\rangle = \mathrm{const} - \frac{B}{2}\left(\frac{1}{d^2}\right).
$
A linear regression of $\log\langle I\rangle$ vs $1/d^2$ provides an estimate of $B$, subject to basic quality checks (e.g. $R^2$ threshold).

A **dataset-wide** Wilson $B$ is also estimated over the merged reflections — restricted to the meaningful resolution range (skipping the low-resolution non-linear region below ~4 Å and shells past the signal limit $\langle I/\sigma\rangle < 1$, so it is insensitive to how far the merged data extend) — and written to the merged mmCIF as `_reflns.B_iso_Wilson_estimate`, the analogue of XDS's Wilson-line $B$. It is diagnostic only and is not fed back into scaling. The **per-image** estimate (used for the live radiation-damage plot) is accepted only when the fit is well-correlated and physically plausible ($0 < B < 200$ Ų); on a bad frame (an indexing glitch, too few reflections) the Wilson line runs wildly steep, so an implausible $B$ is reported as NaN rather than a spurious hundreds-of-Ų value.

---

## 13. Practical notes and limitations

- **Bragg integration is profile-fitted by default** (per-shell Gaussian profile, Kabsch extraction; §9.3), with plain box summation available as a fallback (`--integrator boxsum`). The profiles are built per frame from that frame's strong spots, which suits fast-feedback and serial/streaming use; a profile shared across many frames (as in full offline workflows) is not currently formed.
- **Space-group symmetry** beyond centering absences is not necessarily enforced during prediction/integration unless the space group is supplied and used downstream.
- **Resolution masking and ice rings** are controllable; including ice-ring spots in indexing can improve robustness for some samples but may bias refinement in others.
- **Rotation vs still modes** differ substantially in prediction and scaling: partiality is angle-driven in rotation data, while stills are predicted within an excitation-error window and get their partiality from the default-on per-crystal tilt post-refinement (§10.2) — or unit partiality with `--simple-stills`.
- **Space-group determination.** When no space group is supplied, a POINTLESS-like search scores Laue-group symmetry (CC of $I(h)$ vs $I(Rh)$ plus merge self-consistency) and detects screw/centering absences from the $P1$-merged intensities. Three tests gate a promotion to higher symmetry, all aimed at the merohedral twin, whose twin law forces non-equivalent reflections together and so mimics symmetry:

  1. **Merge self-consistency** ($\chi^2$ under the candidate group, relative to the confirmed subgroup). On its own this is not sufficient: it is a ratio to an error model that moves with the *amount* of data — the parent's systematic term grows as $\sigma$ shrinks with $1/\sqrt{N}$, while a twin's is already saturated — so its verdict depends on how much data the search saw.
  2. **Error-model $b$** (the intensity-proportional systematic). A genuine symmetry step gains multiplicity without inflating $b$; merging a twin law's extra operator inflates it. A $\chi^2$-passing promotion is vetoed when $b$ rises past a bound relative to the confirmed subgroup.
  3. **Operator disagreement**, a sigma-free statistic $H=\mathrm{median}\,|I_1-I_2|/(I_1+I_2)$, formed as the ratio of the operators a promotion *adds* to the parent's own, measured on the same reflections. Normalising against the parent divides out the systematic floor that symmetry mates carry on real data, which varies by crystal and by operator; a median is used because a twin perturbs every pair whereas a badly-measured minority perturbs only the tail. Where a candidate has several parents of the same order, it is judged against the worst of them, since a rival subgroup can itself contain the twin laws.

  The Lorentz factor $\zeta$ (§8.3) governs how well a reflection can be measured, so when the spindle lies in a plane of the lattice, an operator permuting the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. The search is therefore run a second time on a merge of only the well-measured observations (`--search-min-zeta`, rotation default 0.85), and **whichever search found more symmetry is kept** — one-way safe, because discarding observations can starve an operator correlation but never invent one. The filtered merge decides the point group only; absences come from the full merge, since they live in the weak reflections the filter removes. A tie (same order, different symmetry) is reported with both candidates named, for trying in molecular replacement.

  **Centering** is accepted when the systematically-absent class is weak relative to the present one by *either* of two floor-independent tests: its mean signed $I/\sigma$ well below the present mean, *or* its rate of individually-significant reflections well below the present class's own significant rate. The second test covers weak and low-energy data, where a positive intensity floor (background and profile leakage) lifts the absent class's mean $I/\sigma$ well above zero and, when the present class is itself weak, carries the plain mean ratio past its bound; a false centering fails both tests, its absent class being as strong as the present one. When several centerings pass, they are ranked by their **net** systematic absences (absent minus violating), not the gross absent count, so a super-centering (e.g. $F$ over a true $C$) whose extra, only-half-populated absent class dilutes the strength ratio does not out-rank the correct lower centering.
- **Twinning check.** A Padilla–Yeates $L$-test ($\langle|L|\rangle$, $\langle L^2\rangle$) and the second moment $\langle I^2\rangle/\langle I\rangle^2$ (taken per resolution shell with noise-only shells skipped and Wilson outliers rejected, so a single strong reflection in a collapsed-mean shell cannot skew it) are written to the merged mmCIF as a twinning diagnostic. Twinning is only flagged in Laue classes where a merohedral twin law can exist; the holohedral high-symmetry classes ($4/mmm$, $6/mmm$, $m\bar{3}m$, and $\bar{3}m$ on a rhombohedral lattice) are exempt, so a low $\langle|L|\rangle$ there is reported as a statistical artefact rather than twinning.
- **Outlier rejection.** Merging applies an optional per-observation median-based $N\sigma$ cut (`--reject-outliers`, default 6σ for `rot3d`, off otherwise). The same $N\sigma$ cut is fed back into the error model: after an initial $a,b$ fit the parameters are re-fit once on the reflections that survive rejection (dropping any whose squared deviation exceeds $N\sigma^2\,[a\,\sigma^2 + (b\,\langle I\rangle)^2]$), so the calibrated errors describe the reflections that actually enter the merge rather than the pre-rejection pool.
- **Automatic resolution cutoff.** By default the reported/written high-resolution limit is trimmed where $\mathrm{CC}_{1/2}$ falls off: a logistic is fitted to $\mathrm{CC}_{1/2}(s)$, and the limit is set **one reported-shell width past** the point where the fit crosses 0.30 — deliberately "one shell too far", so weak-but-real data below the crossing are kept rather than discarded. The extension is measured over the range that is actually kept, not the full measured range, so a detector reaching far past where the crystal diffracts cannot inflate it. `--scaling-high-resolution` overrides the limit and `--resolution-cutoff off` disables it.
- **Amplitudes and intensities.** The merged output carries both intensities (mmCIF `intensity_meas`, MTZ `IMEAN`/`SIGIMEAN`) and French–Wilson amplitudes (mmCIF `F_meas_au`, MTZ `F`/`SIGF`; §10.8), so a downstream program can refine against either.

---

## 14. Model-based validation: R-free against a model and electron-density maps

Offline (`rugnux --model model.pdb`) the merged data can be scored against a supplied atomic model and **initial** electron-density maps computed — enough to confirm that a model fits the data and to inspect the density, not a substitute for refinement. **The structure itself is not refined**; the model is only re-fractionalized into the data unit cell (a rigid cell adjustment, so a deposited model with a slightly different cell still lines up), and the observed amplitudes are the French–Wilson $|F|$ from §10.8, so the R-free and the maps use exactly the same amplitudes as the written reflection file. The model, structure-factor, bulk-solvent and FFT machinery is provided by GEMMI.

### 14.1 Model structure factors

The model electron density is sampled on a grid (IT92 X-ray form factors, with a Refmac-compatible Gaussian blur chosen for the grid spacing) and Fourier-transformed to structure factors $F_\mathrm{calc}(hkl)$ up to the data resolution.

### 14.2 Bulk solvent and scaling

A flat bulk-solvent mask around the model is transformed to $F_\mathrm{mask}$, and the model is scaled to the observed amplitudes by an overall least-squares fit of a scale $k$, an anisotropic $B$, and the flat-solvent parameters $k_\mathrm{sol}, B_\mathrm{sol}$:

$
F_\mathrm{model} = k\,e^{-\mathbf{h}^\top \mathbf{B}\,\mathbf{h}/4}\left(F_\mathrm{calc} + k_\mathrm{sol}\,e^{-B_\mathrm{sol}\,s^2}\,F_\mathrm{mask}\right),\quad s^2 = 1/4d^2.
$

This is the standard, few-parameter scaling model used by refinement programs. No free-form per-resolution-shell rescale is applied: such a rescale is dataset-specific and reshapes each map's radial amplitude profile differently, which would make maps from a multi-dataset campaign no longer directly comparable.

### 14.3 R-work and R-free

Crystallographic R-factors are reported over the work and free sets (the §10.7 flags):

$
R = \frac{\sum \big|\,|F_o| - |F_\mathrm{model}|\,\big|}{\sum |F_o|},
$

with R-free the same sum restricted to the free set. Note that the scaling of §14.2 is fitted over **all** reflections, work and free alike — its few parameters ($k$, an anisotropic $B$, $k_\mathrm{sol}$, $B_\mathrm{sol}$) are far too few to absorb individual reflections, but R-free here is strictly "free of refinement", not free of the scaling fit.

### 14.4 Electron-density maps

Two maps are formed with the model phases $\varphi_\mathrm{model}$: a $2F_o-F_c$ map, coefficients $(2|F_o|-|F_\mathrm{model}|)\,e^{i\varphi_\mathrm{model}}$, and an $F_o-F_c$ difference map, $(|F_o|-|F_\mathrm{model}|)\,e^{i\varphi_\mathrm{model}}$, each inverse-Fourier-transformed to a real-space CCP4 map (`<prefix>_2fofc.ccp4`, `<prefix>_fofc.ccp4`). A map-coefficient MTZ (`<prefix>_maps.mtz`: `FP`, `FC`, `PHIC`, `FWT`/`PHWT`, `DELFWT`/`PHDELWT`, `FREE`) is written alongside so the maps can be reopened or rebuilt in Coot / PyMOL. These are unweighted difference coefficients (no $\sigma_A$ / figure-of-merit weighting), which is why they are described as *initial* maps.

### 14.5 Aligning the data to the model: enantiomorph and indexing ambiguity

The model fixes a definite hand and indexing, but the merged data need not share them, so before comparison the observed reflections are brought into the model's frame.

- **Enantiomorph / screw.** When the data space group is the enantiomorph of the model's (e.g. data $P4_12_12$, model $P4_32_12$; or $P3_1/P3_2$), the two are **indistinguishable from merged intensities** — $|F_\mathrm{calc}|$ is invariant under the change of hand, so R-free cannot choose between them and probing would be meaningless. The hand is therefore taken from the model: the observed reflections are reindexed by the change-of-hand operator into the model's enantiomorph. Only the map phases (the density's hand) depend on this choice.
- **Indexing (merohedral) ambiguity.** When the crystal has a merohedral ambiguity (§10.9), the observed intensities *do* differ between indexings, and the right one is chosen against the best available reference. **If a reference MTZ was supplied, the data were already reindexed to agree with it** (§10.9 — by the reference-intensity correlation, at the merge stage for rotation data or per image in stills scaling), and model validation keeps that authoritative choice. **Only with a model and no reference** does validation resolve the ambiguity itself, as a fallback: the scaled model is fit to each reindexing of the data (identity plus the twin-law cosets) and the one giving the **lowest R-free** is kept. This matters for a multi-dataset campaign — a single shared reference fixes one indexing convention for every dataset, whereas an independent per-dataset lowest-R-free choice could send borderline datasets to different conventions. A no-op either way for a holohedral crystal (no twin laws).