Jungfraujoch/docs/CPU_DATA_ANALYSIS.md

CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for CPU- and GPU-side real‑time and near‑real‑time data analysis.

Scope. The pipeline covered here comprises:

1. geometry mapping and corrections,
2. azimuthal integration (powder/radial profiles),
3. Bragg spot finding (strong pixels → connected components → spot descriptors),
4. indexing (still and rotation modes),
5. Bravais lattice / centering inference,
6. geometry and lattice refinement,
7. reflection prediction (still and rotation),
8. Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
9. scaling and merging,
10. merge-level error modelling, outlier rejection and the resolution cutoff,
11. space-group determination from the merged intensities (Laue group, screw axes, glide planes, centering, the centre of symmetry), the twinning check and the translational pseudo-symmetry check,
12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
13. amplitude estimation (French–Wilson) and R-free test-set flagging,
14. optional model-based validation: rigid-body placement of a supplied model (molecular replacement where the rigid body does not place it, or for a predicted model), R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.
15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight chosen by R-free within geometry bounds, and the unexplained blobs of the refined difference map.

The reference is split into five parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites.

• From images to spots (§0–§3) — device-side decoding, geometry and reciprocal-space mapping, azimuthal integration, spot finding.
• Indexing and geometry refinement (§4–§7) — FFT and fast-feedback indexing, the lattice search, geometry refinement, post-refinement, powder calibration, and the stills geometry with a given cell and detector modules.
• Prediction, integration, scaling and merging (§8–§12) — reflection prediction, profile-fitted integration, scaling, merging, mosaicity and the auxiliary statistics.
• Space group and validation (§13–§14) — the space-group search, twinning and translational pseudo-symmetry, the resolution cutoff, diffraction anisotropy, model-based validation and molecular replacement.
• Refinement of a placed model (§15) — the maximum-likelihood target and its FFT gradient, the scale, bulk solvent and likelihood parameters, the dictionary and its restraints (metal sites and B included), the minimiser, the weights chosen by trial, the refined maps and their unexplained blobs, and what is not implemented yet.

References

The methods draw on, and in places reimplement, solutions from:

• W. Kabsch, “XDS”, Acta Cryst. D66 (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
• W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, Acta Cryst. D66 (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
• T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
• J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", Powder Diffraction 28 (2013), S339-S350 (detector geometry definition, azimuthal integration)
• G. Bricogne, Proceedings of the EEC Cooperative Workshop on Position-Sensitive Detector Software (Phase III), LURE, Paris (1986), p. 28 (Fourier-transform indexing, proposed as a three-dimensional transform; §5 uses its one-dimensional projection form).
• I. Steller, R. Bolotovsky & M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using Fourier analysis", J. Appl. Cryst. 30 (1997), 1036-1040 (the projection/1D-FFT autoindexing algorithm of §5).
• H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", Acta Cryst. D55 (1999), 1690-1695 (the MOSFLM implementation of it, whose practice is followed)
• P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", J. Appl. Cryst. 57 (2024), 931-944 (the algorithm of the ffbidx fast-feedback indexer, §4).
• I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", Acta Cryst. A32 (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
• J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", Acta Cryst. D59 (2003), 1124-1130 (the L test, §13.2).
• T. O. Yeates, "Detecting and overcoming crystal twinning", Methods Enzymol. 276 (1997), 344-358 (the twinned intensity model and the H statistic behind the twin fraction of the twin hypothesis, §13.2, §14.9).
• A. J. C. Wilson, "The probability distribution of X-ray intensities", Acta Cryst. 2 (1949), 318-321, and E. R. Howells, D. C. Phillips & D. Rogers, "The probability distribution of X-ray intensities. II. Experimental investigation and the X-ray detection of centres of symmetry", Acta Cryst. 3 (1950), 210-214 (the acentric and centric intensity distributions behind the centre-of-symmetry statistics of §13.1 and the Wilson outlier test of §13.3).
• W. H. Baur & D. Kassner, "The perils of Cc: comparing the frequencies of falsely assigned space groups with their general population", Acta Cryst. B48 (1992), 356-369 (writing the centrosymmetric group where the absences cannot decide the centre, §13.1).
• R. J. Read, P. D. Adams & A. J. McCoy, "Intensity statistics in the presence of translational noncrystallographic symmetry", Acta Cryst. D69 (2013), 176-183 (the native-Patterson detection of translational pseudo-symmetry, and the intensity modulation it produces, which the axial-zone screw-absence test scores against).
• A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", J. Appl. Cryst. 47 (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
• A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
• S. French & K. Wilson, "On the treatment of negative intensity observations", Acta Cryst. A34 (1978), 517-525 (Bayesian amplitude estimation from intensities), and CCP4's ctruncate (C. Ballard & N. Stein), whose anisotropic Wilson prior §10.8 follows, cited through M. D. Winn et al., "Overview of the CCP4 suite and current developments", Acta Cryst. D67 (2011), 235-242.
• A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free cross-validation).
• P. H. C. Eilers, "A perfect smoother", Anal. Chem. 75 (2003), 3631-3636, after E. T. Whittaker, "On a new method of graduation", Proc. Edinburgh Math. Soc. 41 (1923), 63-75 (the penalised smoother of the fulls' per-frame scale, §10.6).
• H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", SIAM J. Numer. Anal. 49 (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", J. ACM 12 (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6).
• R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", Biometrika 10 (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared).
• R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", Acta Cryst. A34 (1978), 791-809 (the gradient of the refinement target, §15).
• N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", Acta Cryst. A52 (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", Acta Cryst. D53 (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15).
• V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", Acta Cryst. A51 (1995), 880-887 (the likelihood parameters from the free set, §15).
• A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", Acta Cryst. D60 (2004), 2184-2195 (the restraint dictionary, §15).
• P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", Acta Cryst. D68 (2012), 352-367 (the weight chosen among refinements by R-free, §15).
• D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", Math. Program. 45 (1989), 503-528 (the minimiser, §15).
• J. Nocedal, "Updating quasi-Newton matrices with limited storage", Math. Comp. 35 (1980), 773-782 (the two-loop recursion of the minimiser, §15.6).
• G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", Acta Cryst. D67 (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5).
• A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", Acta Cryst. A45 (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
• P. J. Huber, "Robust estimation of a location parameter", Ann. Math. Statist. 35 (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, "Deterministic edge-preserving regularization in computed imaging", IEEE Trans. Image Process. 6 (1997), 298-311 (the robust, pseudo-Huber scale fit of the refinement's score, §15.3).
• J.-S. Jiang & A. T. Brünger, "Protein hydration observed by X-ray diffraction", J. Mol. Biol. 243 (1994), 100-115 (the flat bulk-solvent mask, §14.2, §15.3).
• A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", J. Comput. Chem. 17 (1996), 1132-1141 (the torsion derivatives, §15.5).
• M. M. Harding, "Geometry of metal-ligand interactions in proteins", Acta Cryst. D57 (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", Acta Cryst. D62 (2006), 678-682, and F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", Nucleic Acids Res. 45 (2017), 987-1004 (the metal-site distances, §15.5).
• P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", Acta Cryst. D66 (2010), 486-501 (unmodelled blobs, §15.9).
• M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (model / structure-factor / map machinery used in §14; the topology, riding hydrogens and blob search of §15).
• J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and github.com/jonwright/bslz4decoders (device-side decoding of bitshuffle+LZ4 images, §0).
• A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", J. Appl. Cryst. 44 (2011), 1285-1287 (anomalous difference density read at the model's sites).
• R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", J. Appl. Cryst. 15 (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to the azimuthal profile, the Bragg intensities and the ring background the beam-stop shadow test compares against).
• R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4).
• A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", Acta Cryst. D58 (2002), 1387-1392 (the flat bulk-solvent model, its optimal parameters and the range they are physically meaningful over, used when scaling a model to the data in §14).
• P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", Acta Cryst. D61 (2005), 850-855 (the grid search over that range that fits k_sol and b_sol, with the overall scale and anisotropic B refitted at each grid point).
• K. Shoemake, "Uniform Random Rotations", in Graphics Gems III, ed. D. Kirk, Academic Press (1992), 124-132 (the uniform random rotations the model-fit null of §14.5 is built from).
• G. H. Golub & V. Pereyra, "The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate", SIAM J. Numer. Anal. 10 (1973), 413-432, and L. Kaufman, "A variable projection method for solving separable nonlinear least squares problems", BIT 15 (1975), 49-57 (the scale re-fit folded into the rigid-body Jacobian of §14.8).
• Z. Otwinowski & W. Minor, "Processing of X-ray diffraction data collected in oscillation mode", Methods Enzymol. 276 (1997), 307-326 (reweighted, de-biased profile-fit variances).
• G. Winter et al., "DIALS: implementation and evaluation of a new integration package", Acta Cryst. D74 (2018), 85-97, and J. Beilsten-Edmands et al., Acta Cryst. D76 (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).
• R. H. Blessing, "An empirical correction for absorption anisotropy", Acta Cryst. A51 (1995), 33-38 (absorption as spherical harmonics of the beam directions).
• P. Evans, "Scaling and assessment of data quality", Acta Cryst. D62 (2006), 72-82, and P. R. Evans, Acta Cryst. D67 (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).
• A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II 245, 41-51 (post-refinement practice: what is refined per image and what over a wedge).
• D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", Acta Cryst. D77 (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).
• K. Röttger, A. Endriss, J. Ihringer, S. Doyle & W. F. Kuhs, "Lattice constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K", Acta Cryst. B50 (1994), 644-648 (the ice Ih cell the ring positions below 1.522 Å are calculated from).
• S. Sheriff & W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals", Acta Cryst. A43 (1987), 118-121 (the overall anisotropic B tensor and its symmetry constraints), and A. N. Popov & G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling", Acta Cryst. D59 (2003), 1145-1153 (the sigma-aware estimation of the anisotropy of the observed intensity distribution, part of that paper's statistic modelling).
• I. J. Tickle, C. Flensburg, P. Keller, W. Paciorek, A. Sharff, C. Vonrhein & G. Bricogne, STARANISO (2016), Global Phasing Ltd, staraniso.globalphasing.org (the inspiration for analysing and reporting diffraction anisotropy on every run, §13.5).
• C. Vonrhein, C. Flensburg, P. Keller et al., "Data processing and analysis with the autoPROC toolbox", Acta Cryst. D67 (2011), 293-302 (the inspiration for not trusting header geometry and for finding the beam centre from the images, §1.4).
• P. R. Evans & G. N. Murshudov, "How good are my data and what is the resolution?", Acta Cryst. D69 (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).
• G. Assmann, W. Brehm & K. Diederichs, "Identification of rogue datasets in serial crystallography", J. Appl. Cryst. 49 (2016), 1021-1028, and G. M. Assmann, M. Wang & K. Diederichs, Acta Cryst. D76 (2020), 636-652 (XDSCC12: sigma-tau CC1/2, delta-CC1/2, the Fisher transformation and the rejection discipline the frame disposition follows).
• W. Brehm & K. Diederichs, "Breaking the indexing ambiguity in serial crystallography", Acta Cryst. D70 (2014), 101-109 (serial stills without a reference: each crystal put into the indexing in which it agrees best with all the others).
• K. Diederichs & P. A. Karplus, Nat. Struct. Biol. 4 (1997), 269-275, and P. A. Karplus & K. Diederichs, Science 336 (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).
• M. S. Weiss, "Global indicators of X-ray data quality", J. Appl. Cryst. 34 (2001), 130-135 (R_pim).
• IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", Acta Cryst. A45 (1989), 63-75, and Acta Cryst. A51 (1995), 565-569 (uncertainty conventions).
• J. Navaza, "AMoRe: an automated package for molecular replacement", Acta Cryst. A50 (1994), 157-163 (the rotation function on an interpolated molecular transform, §14.10).
• R. A. Crowther & D. M. Blow, "A method of positioning a known molecule in an unknown crystal structure", Acta Cryst. 23 (1967), 544-548 (the full-symmetry translation function, §14.10).
• B. W. Matthews, "Solvent content of protein crystals", J. Mol. Biol. 33 (1968), 491-497, and K. A. Kantardjieff & B. Rupp, "Matthews coefficient probabilities", Protein Sci. 12 (2003), 1865-1871 (how many copies the cell holds, §14.10).
• R. J. Read, "Pushing the boundaries of molecular replacement with maximum likelihood", Acta Cryst. D57 (2001), 1373-1382; A. J. McCoy, R. W. Grosse-Kunstleve, L. C. Storoni & R. J. Read, "Likelihood-enhanced fast translation functions", Acta Cryst. D61 (2005), 458-464; A. J. McCoy et al., "Phaser crystallographic software", J. Appl. Cryst. 40 (2007), 658-674; A. J. McCoy et al., Proc. Natl. Acad. Sci. USA 114 (2017), 3637-3641 (the likelihood target, the solution criterion and the expected gain of molecular replacement, §14.10).
• R. D. Oeffner, T. Croll, C. Millán et al., "Putting AlphaFold models to work with phenix.process_predicted_model and ISOLDE", Acta Cryst. D78 (2022), 1303-1314, and J. Jumper et al., "Highly accurate protein structure prediction with AlphaFold", Nature 596 (2021), 583-589 (preparing a predicted model for molecular replacement, §14.10).

(list is not exhaustive; the full citations, with DOIs, are in ACKNOWLEDGEMENT.md)