Jungfraujoch/docs/CPU_DATA_ANALYSIS.md

CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for CPU- and GPU-side real‑time and near‑real‑time data analysis.

Scope. The pipeline covered here comprises:

1. geometry mapping and corrections,
2. azimuthal integration (powder/radial profiles),
3. Bragg spot finding (strong pixels → connected components → spot descriptors),
4. indexing (still and rotation modes),
5. Bravais lattice / centering inference,
6. geometry and lattice refinement,
7. reflection prediction (still and rotation),
8. Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
9. scaling and merging,
10. merge-level error modelling, outlier rejection and the resolution cutoff,
11. space-group determination from the merged intensities (Laue group, screw axes, glide planes, centering), the twinning check and the translational pseudo-symmetry check,
12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
13. amplitude estimation (French–Wilson) and R-free test-set flagging,
14. optional model-based validation: rigid-body placement of a supplied model, R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.

The reference is split into four parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites.

• From images to spots (§0–§3) — device-side decoding, geometry and reciprocal-space mapping, azimuthal integration, spot finding.
• Indexing and geometry refinement (§4–§7) — FFT and fast-feedback indexing, the lattice search, geometry refinement, post-refinement and powder calibration.
• Prediction, integration, scaling and merging (§8–§12) — reflection prediction, profile-fitted integration, scaling, merging, mosaicity and the auxiliary statistics.
• Space group and validation (§13–§14) — the space-group search, twinning and translational pseudo-symmetry, the resolution cutoff, diffraction anisotropy, and model-based validation.

Naming: score, ratio, count

Per-image quantities are named for the kind of number they are, because several of them describe the same physics on opposite conventions and the name is the only thing that says which:

Suffix	Range	What 1 means	Examples
*_score	bounded, 0 to 1, saturating	certainly present	protein_score, ice_score
*_ratio	unbounded above	none	ice_ring_ratio, the ice-band spot ratio
*_count	integer	one occurrence	spot_count, spot_count_ice_rings, spot_count_ice_control

So a score is bounded and 1 is certainty; a ratio is unbounded and 1 is nothing. ice_score and ice_ring_ratio both describe ice and their extremes are opposite ends of the scale.

The identifiers, the stored names and the wire names all follow it. The ice ring ratio was called ice_ring_score before; nothing that reads an old file or an old stream lost anything in the rename:

Surface	Now	Retired spelling
HDF5	/entry/MX/iceRingRatio, iceRingRatioMean	iceRingScore, iceRingScoreMean — still read (the per-image array only - no *Mean scalar is read back by anything), so a file written before the rename still opens
CBOR	ice_ring_ratio, ice_ring_ratio_mean	ice_ring_score, ice_ring_score_mean — still decoded, so an older producer's stream still read (the per-image array only - no *Mean scalar is read back by anything)s
REST plot_type	ice_ring_ratio	ice_ring_score — still accepted, and not scheduled for removal; both name the same plot
CLI	--ice-min-score	unchanged, deliberately. It is a threshold on the ratio, and it is the one surface a user types

Writers and producers emit the new spelling only. The old one is read but never written, which is what makes the rename a one-way migration rather than a flag day.

References

The methods draw on, and in places reimplement, solutions from:

• W. Kabsch, “XDS”, Acta Cryst. D66 (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
• W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, Acta Cryst. D66 (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
• T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
• J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", Powder Diffraction 28 (2013), S339-S350 (detector geometry definition, azimuthal integration)
• I. Steller, R. Bolotovsky & M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using Fourier analysis", J. Appl. Cryst. 30 (1997), 1036-1040 (the projection/1D-FFT autoindexing algorithm of §5).
• H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", Acta Cryst. D55 (1999), 1690-1695 (the MOSFLM implementation of it, whose practice is followed)
• P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", J. Appl. Cryst. 57 (2024), 931-944 (the algorithm of the ffbidx fast-feedback indexer, §4).
• I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", Acta Cryst. A32 (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
• J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", Acta Cryst. D59 (2003), 1124-1130 (the L test, §13.2).
• R. J. Read, P. D. Adams & A. J. McCoy, "Intensity statistics in the presence of translational noncrystallographic symmetry", Acta Cryst. D69 (2013), 176-183 (the native-Patterson detection of translational pseudo-symmetry, and the intensity modulation it produces, which the axial-zone screw-absence test scores against).
• A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", J. Appl. Cryst. 47 (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
• A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
• S. French & K. Wilson, "On the treatment of negative intensity observations", Acta Cryst. A34 (1978), 517-525 (Bayesian amplitude estimation from intensities).
• A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free cross-validation).
• M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (model / structure-factor / map machinery used in §14).
• J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and github.com/jonwright/bslz4decoders (device-side decoding of bitshuffle+LZ4 images, §0).
• A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", J. Appl. Cryst. 44 (2011), 1285-1287 (anomalous difference density read at the model's sites).
• R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", J. Appl. Cryst. 15 (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to both the azimuthal profile and the Bragg intensities).
• R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4).
• A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", Acta Cryst. D58 (2002), 1387-1392 (the flat bulk-solvent model, its optimal parameters and the range they are physically meaningful over, used when scaling a model to the data in §14).
• P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", Acta Cryst. D61 (2005), 850-855 (the grid search over that range that fits k_sol and b_sol, with the overall scale and anisotropic B refitted at each grid point).
• K. Shoemake, "Uniform Random Rotations", in Graphics Gems III, ed. D. Kirk, Academic Press (1992), 124-132 (the uniform random rotations the model-fit null of §14.5 is built from).
• Z. Otwinowski & W. Minor, "Processing of X-ray diffraction data collected in oscillation mode", Methods Enzymol. 276 (1997), 307-326 (reweighted, de-biased profile-fit variances).
• G. Winter et al., "DIALS: implementation and evaluation of a new integration package", Acta Cryst. D74 (2018), 85-97, and J. Beilsten-Edmands et al., Acta Cryst. D76 (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).
• P. Evans, "Scaling and assessment of data quality", Acta Cryst. D62 (2006), 72-82, and P. R. Evans, Acta Cryst. D67 (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).
• A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II 245, 41-51 (post-refinement practice: what is refined per image and what over a wedge).
• D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", Acta Cryst. D77 (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).
• K. Röttger, A. Endriss, J. Ihringer, S. Doyle & W. F. Kuhs, "Lattice constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K", Acta Cryst. B50 (1994), 644-648 (the ice Ih cell the ring positions below 1.522 Å are calculated from).
• S. Sheriff & W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals", Acta Cryst. A43 (1987), 118-121 (the overall anisotropic B tensor and its symmetry constraints), and A. N. Popov & G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling", Acta Cryst. D59 (2003), 1145-1153 (the sigma-aware estimation of the anisotropy of the observed intensity distribution, part of that paper's statistic modelling).
• P. R. Evans & G. N. Murshudov, "How good are my data and what is the resolution?", Acta Cryst. D69 (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).
• K. Diederichs & P. A. Karplus, Nat. Struct. Biol. 4 (1997), 269-275, and P. A. Karplus & K. Diederichs, Science 336 (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).
• IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", Acta Cryst. A45 (1989), 63-75, and Acta Cryst. A51 (1995), 565-569 (uncertainty conventions).

(list is not exhaustive; the full citations, with DOIs, are in ACKNOWLEDGEMENT.md)