mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 87 Wiki Activity
Files
07ae590e54f06f27ec00e5c70d1b97f97d139e1d
Jungfraujoch/docs/CPU_DATA_ANALYSIS.md
T
leonarski_f 07ae590e54 Serial stills: the indexing ambiguity resolved from the data when no reference or model is given 
Where the space group is given (-S) and its lattice admits alternative indexings, a stills run without
-z or --model used to merge the crystals' random indexings together (and warned). The crystals are now
made consistent with each other before anything is scaled (ReferenceFreeReindex): each takes the
indexing in which it agrees best with all the others (Brehm & Diederichs 2014), agreement being the
sum over shared reflections of the products of shell-normalised, per-crystal standardised intensities,
weighted by 1/(1+(sigma/sd)^2). The sum over all pairs is (|M|^2 - const)/2 for M the running sum, so a
pass over the crystals is one pass over the reflections rather than over all pairs: coordinate ascent,
four seeded starts, stop when <= 1 crystal in 1000 moves, the overall relabel that moves the fewest
crystals. Deterministic (fixed summation order, sequential passes).

A twin law of a pseudo-merohedral cell whose axes differ by a few per cent is decided by the crystals'
own refined cells, which the indexer has already used: where >= 90% of the crystals' cells fit better as
indexed, those crystals are anchors and only the others are decided by the intensities. Without this,
two public sets got worse (pal90 R-free 0.183 -> 0.207, lcls242_t 0.296 -> 0.350).

With a reference or model the per-image choice stands; the same resolution runs as a check, reported as
AMBIGUITY_DATA_DISAGREES. New report keys (section 2): INDEXING_AMBIGUITY_RESOLVED, AMBIGUITY_CRYSTALS,
AMBIGUITY_REINDEXED, AMBIGUITY_SEPARATION_SIGMA, AMBIGUITY_DECIDED_FRACTION (+ AMBIGUITY_TIME_S, dev).
--mode scale is not changed.

Still battery, no --model (R-free, REFMAC 0-cycle against the deposited model; before -> after; with
--model for comparison): euxfel83 0.321 -> 0.207 (0.204), euxfel152 0.346 -> 0.257 (0.245), alvra213
0.209 -> 0.201 (0.207), alvra213_int16 0.210 -> 0.202 (0.208), pal90 0.183 -> 0.183 (0.188), lcls242_t
0.296 -> 0.294 (0.285); controls sacla237 / lcls222 unchanged (no ambiguity, no-op). The step takes
0.2-1.6 s on 2.7k-54k crystals; synthetic 64k crystals x 200 reflections 0.5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 80d378a1fe92364fd5b99a17f5c9ee8498412d36)
2026-10-25 23:26:32 +01:00

17 KiB
Raw Blame History

CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for CPU- and GPU-side real‑time and near‑real‑time data analysis.

Scope. The pipeline covered here comprises:

  1. geometry mapping and corrections,
  2. azimuthal integration (powder/radial profiles),
  3. Bragg spot finding (strong pixels → connected components → spot descriptors),
  4. indexing (still and rotation modes),
  5. Bravais lattice / centering inference,
  6. geometry and lattice refinement,
  7. reflection prediction (still and rotation),
  8. Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
  9. scaling and merging,
  10. merge-level error modelling, outlier rejection and the resolution cutoff,
  11. space-group determination from the merged intensities (Laue group, screw axes, glide planes, centering, the centre of symmetry), the twinning check and the translational pseudo-symmetry check,
  12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
  13. amplitude estimation (French–Wilson) and R-free test-set flagging,
  14. optional model-based validation: rigid-body placement of a supplied model (molecular replacement where the rigid body does not place it, or for a predicted model), R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.
  15. optional restrained refinement of the placed model (--refine): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight chosen by R-free within geometry bounds, and the unexplained blobs of the refined difference map.

The reference is split into four parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites.

References

The methods draw on, and in places reimplement, solutions from:

  • W. Kabsch, “XDS”, Acta Cryst. D66 (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
  • W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, Acta Cryst. D66 (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
  • T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
  • J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", Powder Diffraction 28 (2013), S339-S350 (detector geometry definition, azimuthal integration)
  • G. Bricogne, Proceedings of the EEC Cooperative Workshop on Position-Sensitive Detector Software (Phase III), LURE, Paris (1986), p. 28 (Fourier-transform indexing, proposed as a three-dimensional transform; §5 uses its one-dimensional projection form).
  • I. Steller, R. Bolotovsky & M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using Fourier analysis", J. Appl. Cryst. 30 (1997), 1036-1040 (the projection/1D-FFT autoindexing algorithm of §5).
  • H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", Acta Cryst. D55 (1999), 1690-1695 (the MOSFLM implementation of it, whose practice is followed)
  • P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", J. Appl. Cryst. 57 (2024), 931-944 (the algorithm of the ffbidx fast-feedback indexer, §4).
  • I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", Acta Cryst. A32 (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
  • J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", Acta Cryst. D59 (2003), 1124-1130 (the L test, §13.2).
  • T. O. Yeates, "Detecting and overcoming crystal twinning", Methods Enzymol. 276 (1997), 344-358 (the twinned intensity model and the H statistic behind the twin fraction of the twin hypothesis, §13.2, §14.9).
  • A. J. C. Wilson, "The probability distribution of X-ray intensities", Acta Cryst. 2 (1949), 318-321, and E. R. Howells, D. C. Phillips & D. Rogers, "The probability distribution of X-ray intensities. II. Experimental investigation and the X-ray detection of centres of symmetry", Acta Cryst. 3 (1950), 210-214 (the acentric and centric intensity distributions behind the centre-of-symmetry statistics of §13.1 and the Wilson outlier test of §13.3).
  • W. H. Baur & D. Kassner, "The perils of Cc: comparing the frequencies of falsely assigned space groups with their general population", Acta Cryst. B48 (1992), 356-369 (writing the centrosymmetric group where the absences cannot decide the centre, §13.1).
  • R. J. Read, P. D. Adams & A. J. McCoy, "Intensity statistics in the presence of translational noncrystallographic symmetry", Acta Cryst. D69 (2013), 176-183 (the native-Patterson detection of translational pseudo-symmetry, and the intensity modulation it produces, which the axial-zone screw-absence test scores against).
  • A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", J. Appl. Cryst. 47 (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
  • A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
  • S. French & K. Wilson, "On the treatment of negative intensity observations", Acta Cryst. A34 (1978), 517-525 (Bayesian amplitude estimation from intensities), and CCP4's ctruncate (C. Ballard & N. Stein), whose anisotropic Wilson prior §10.8 follows, cited through M. D. Winn et al., "Overview of the CCP4 suite and current developments", Acta Cryst. D67 (2011), 235-242.
  • A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free cross-validation).
  • P. H. C. Eilers, "A perfect smoother", Anal. Chem. 75 (2003), 3631-3636, after E. T. Whittaker, "On a new method of graduation", Proc. Edinburgh Math. Soc. 41 (1923), 63-75 (the penalised smoother of the fulls' per-frame scale, §10.6).
  • H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", SIAM J. Numer. Anal. 49 (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", J. ACM 12 (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6).
  • R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", Biometrika 10 (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared).
  • R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", Acta Cryst. A34 (1978), 791-809 (the gradient of the refinement target, §15).
  • N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", Acta Cryst. A52 (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, Acta Cryst. D53 (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15).
  • V. Y. Lunin & T. P. Skovoroda, Acta Cryst. A51 (1995), 880-887 (the likelihood parameters from the free set, §15).
  • A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", Acta Cryst. D60 (2004), 2184-2195 (the restraint dictionary, §15).
  • P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", Acta Cryst. D68 (2012), 352-367 (the weight chosen among refinements by R-free, §15).
  • D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", Math. Program. 45 (1989), 503-528 (the minimiser, §15).
  • M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (model / structure-factor / map machinery used in §14).
  • J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and github.com/jonwright/bslz4decoders (device-side decoding of bitshuffle+LZ4 images, §0).
  • A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", J. Appl. Cryst. 44 (2011), 1285-1287 (anomalous difference density read at the model's sites).
  • R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", J. Appl. Cryst. 15 (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to the azimuthal profile, the Bragg intensities and the ring background the beam-stop shadow test compares against).
  • R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4).
  • A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", Acta Cryst. D58 (2002), 1387-1392 (the flat bulk-solvent model, its optimal parameters and the range they are physically meaningful over, used when scaling a model to the data in §14).
  • P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", Acta Cryst. D61 (2005), 850-855 (the grid search over that range that fits k_sol and b_sol, with the overall scale and anisotropic B refitted at each grid point).
  • K. Shoemake, "Uniform Random Rotations", in Graphics Gems III, ed. D. Kirk, Academic Press (1992), 124-132 (the uniform random rotations the model-fit null of §14.5 is built from).
  • G. H. Golub & V. Pereyra, "The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate", SIAM J. Numer. Anal. 10 (1973), 413-432, and L. Kaufman, "A variable projection method for solving separable nonlinear least squares problems", BIT 15 (1975), 49-57 (the scale re-fit folded into the rigid-body Jacobian of §14.8).
  • Z. Otwinowski & W. Minor, "Processing of X-ray diffraction data collected in oscillation mode", Methods Enzymol. 276 (1997), 307-326 (reweighted, de-biased profile-fit variances).
  • G. Winter et al., "DIALS: implementation and evaluation of a new integration package", Acta Cryst. D74 (2018), 85-97, and J. Beilsten-Edmands et al., Acta Cryst. D76 (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).
  • R. H. Blessing, "An empirical correction for absorption anisotropy", Acta Cryst. A51 (1995), 33-38 (absorption as spherical harmonics of the beam directions).
  • P. Evans, "Scaling and assessment of data quality", Acta Cryst. D62 (2006), 72-82, and P. R. Evans, Acta Cryst. D67 (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).
  • A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II 245, 41-51 (post-refinement practice: what is refined per image and what over a wedge).
  • D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", Acta Cryst. D77 (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).
  • K. Röttger, A. Endriss, J. Ihringer, S. Doyle & W. F. Kuhs, "Lattice constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K", Acta Cryst. B50 (1994), 644-648 (the ice Ih cell the ring positions below 1.522 Å are calculated from).
  • S. Sheriff & W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals", Acta Cryst. A43 (1987), 118-121 (the overall anisotropic B tensor and its symmetry constraints), and A. N. Popov & G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling", Acta Cryst. D59 (2003), 1145-1153 (the sigma-aware estimation of the anisotropy of the observed intensity distribution, part of that paper's statistic modelling).
  • I. J. Tickle, C. Flensburg, P. Keller, W. Paciorek, A. Sharff, C. Vonrhein & G. Bricogne, STARANISO (2016), Global Phasing Ltd, staraniso.globalphasing.org (the inspiration for analysing and reporting diffraction anisotropy on every run, §13.5).
  • C. Vonrhein, C. Flensburg, P. Keller et al., "Data processing and analysis with the autoPROC toolbox", Acta Cryst. D67 (2011), 293-302 (the inspiration for not trusting header geometry and for finding the beam centre from the images, §1.4).
  • P. R. Evans & G. N. Murshudov, "How good are my data and what is the resolution?", Acta Cryst. D69 (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).
  • G. Assmann, W. Brehm & K. Diederichs, "Identification of rogue datasets in serial crystallography", J. Appl. Cryst. 49 (2016), 1021-1028, and G. M. Assmann, M. Wang & K. Diederichs, Acta Cryst. D76 (2020), 636-652 (XDSCC12: sigma-tau CC1/2, delta-CC1/2, the Fisher transformation and the rejection discipline the frame disposition follows).
  • W. Brehm & K. Diederichs, "Breaking the indexing ambiguity in serial crystallography", Acta Cryst. D70 (2014), 101-109 (serial stills without a reference: each crystal put into the indexing in which it agrees best with all the others).
  • K. Diederichs & P. A. Karplus, Nat. Struct. Biol. 4 (1997), 269-275, and P. A. Karplus & K. Diederichs, Science 336 (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).
  • M. S. Weiss, "Global indicators of X-ray data quality", J. Appl. Cryst. 34 (2001), 130-135 (R_pim).
  • IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", Acta Cryst. A45 (1989), 63-75, and Acta Cryst. A51 (1995), 565-569 (uncertainty conventions).
  • J. Navaza, "AMoRe: an automated package for molecular replacement", Acta Cryst. A50 (1994), 157-163 (the rotation function on an interpolated molecular transform, §14.10).
  • R. A. Crowther & D. M. Blow, "A method of positioning a known molecule in an unknown crystal structure", Acta Cryst. 23 (1967), 544-548 (the full-symmetry translation function, §14.10).
  • B. W. Matthews, "Solvent content of protein crystals", J. Mol. Biol. 33 (1968), 491-497, and K. A. Kantardjieff & B. Rupp, "Matthews coefficient probabilities", Protein Sci. 12 (2003), 1865-1871 (how many copies the cell holds, §14.10).
  • R. J. Read, "Pushing the boundaries of molecular replacement with maximum likelihood", Acta Cryst. D57 (2001), 1373-1382; A. J. McCoy, R. W. Grosse-Kunstleve, L. C. Storoni & R. J. Read, "Likelihood-enhanced fast translation functions", Acta Cryst. D61 (2005), 458-464; A. J. McCoy et al., "Phaser crystallographic software", J. Appl. Cryst. 40 (2007), 658-674; A. J. McCoy et al., Proc. Natl. Acad. Sci. USA 114 (2017), 3637-3641 (the likelihood target, the solution criterion and the expected gain of molecular replacement, §14.10).
  • R. D. Oeffner, T. Croll, C. Millán et al., "Putting AlphaFold models to work with phenix.process_predicted_model and ISOLDE", Acta Cryst. D78 (2022), 1303-1314, and J. Jumper et al., "Highly accurate protein structure prediction with AlphaFold", Nature 596 (2021), 583-589 (preparing a predicted model for molecular replacement, §14.10).

(list is not exhaustive; the full citations, with DOIs, are in ACKNOWLEDGEMENT.md)

Reference in new issue View Git Blame Copy Permalink