mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 80 Wiki Activity
Files
54968a245f1dfd0e6fd2780a040e080a826be20f
Jungfraujoch/docs/CPU_DATA_ANALYSIS.md
T
leonarski_fandClaude Opus 5 99910e187e beam stop: a ring is only flat once the polarization is divided out 
The shadow test compares a pixel with the median of its ring, which assumes the background is flat
around a ring with nothing in the beam. It is not: a polarized source suppresses the background in
its own plane by 1 - sin^2(2 theta), a factor of three at 2 theta = 55 degrees and four at 70. That
is several times the dip the test is looking for, so on a short-distance geometry the two horizontal
lobes of every outer ring read as shadow. Measured over the corpus it cost one dataset 5.1 % of its
detector, and another 11.2 %, with nothing visible under either mask.

The mean projection is therefore divided by the Kahn factor before the ring comparison, and the
Poisson deficit multiplies it back in so the significance is still the significance of the counts
that were recorded. The factor is the geometry's own CalcAzIntPolarizationCorr, evaluated about the
centre the caller named rather than the one in the file: the azimuth is the whole point of this
correction, and the geometry is what knows where the polarization plane lies once the detector is
tilted, the stored image quarter-turned or the detector rotated in its own plane. Six corpus
datasets are quarter-turned, and a flat-detector formula on the stored image's own axes gets them
90 degrees out of phase - measured, it takes one of them from 16.5 % masked to 22.8 % where the
geometry takes it to 4.4 %. Polarization is also the only correction a ring carries that varies
along it; solid angle, detector and air absorption are functions of 2 theta and the ring median
absorbs them.

Measured, mask fraction of the detector: a clean 110 mm sweep 0.82 -> 0.51 %, a clean 200 mm 18 Mpx
sweep 0.94 -> 0.80 %, a 150 mm sweep whose header centre is 351 px out 5.05 -> 0.74 %, and the
sweep with a third of the detector behind a pin 32.22 -> 32.83 %. Clean detectors lose mask, a real
obstruction gains it. Across 151 corpus datasets nothing gains more than a tenth of a percent from
the correction and 26 lose some, 6 of them by more than 3 percentage points.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 07:36:57 +02:00

9.8 KiB
Raw Blame History

CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for CPU- and GPU-side real‑time and near‑real‑time data analysis.

Scope. The pipeline covered here comprises:

  1. geometry mapping and corrections,
  2. azimuthal integration (powder/radial profiles),
  3. Bragg spot finding (strong pixels → connected components → spot descriptors),
  4. indexing (still and rotation modes),
  5. Bravais lattice / centering inference,
  6. geometry and lattice refinement,
  7. reflection prediction (still and rotation),
  8. Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
  9. scaling and merging,
  10. merge-level error modelling, outlier rejection and the resolution cutoff,
  11. space-group determination from the merged intensities (Laue group, screw axes, glide planes, centering), the twinning check and the translational pseudo-symmetry check,
  12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
  13. amplitude estimation (French–Wilson) and R-free test-set flagging,
  14. optional model-based validation: rigid-body placement of a supplied model, R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.

The reference is split into four parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites.

References

The methods draw on, and in places reimplement, solutions from:

  • W. Kabsch, “XDS”, Acta Cryst. D66 (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
  • W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, Acta Cryst. D66 (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
  • T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
  • J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", Powder Diffraction 28 (2013), S339-S350 (detector geometry definition, azimuthal integration)
  • I. Steller, R. Bolotovsky & M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using Fourier analysis", J. Appl. Cryst. 30 (1997), 1036-1040 (the projection/1D-FFT autoindexing algorithm of §5).
  • H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", Acta Cryst. D55 (1999), 1690-1695 (the MOSFLM implementation of it, whose practice is followed)
  • P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", J. Appl. Cryst. 57 (2024), 931-944 (the algorithm of the ffbidx fast-feedback indexer, §4).
  • I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", Acta Cryst. A32 (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
  • J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", Acta Cryst. D59 (2003), 1124-1130 (the L test, §13.2).
  • R. J. Read, P. D. Adams & A. J. McCoy, "Intensity statistics in the presence of translational noncrystallographic symmetry", Acta Cryst. D69 (2013), 176-183 (the native-Patterson detection of translational pseudo-symmetry, and the intensity modulation it produces, which the axial-zone screw-absence test scores against).
  • A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", J. Appl. Cryst. 47 (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
  • A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
  • S. French & K. Wilson, "On the treatment of negative intensity observations", Acta Cryst. A34 (1978), 517-525 (Bayesian amplitude estimation from intensities).
  • A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free cross-validation).
  • M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (model / structure-factor / map machinery used in §14).
  • J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and github.com/jonwright/bslz4decoders (device-side decoding of bitshuffle+LZ4 images, §0).
  • A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", J. Appl. Cryst. 44 (2011), 1285-1287 (anomalous difference density read at the model's sites).
  • R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", J. Appl. Cryst. 15 (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to the azimuthal profile, the Bragg intensities and the ring background the beam-stop shadow test compares against).
  • R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4).
  • A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", Acta Cryst. D58 (2002), 1387-1392 (the flat bulk-solvent model, its optimal parameters and the range they are physically meaningful over, used when scaling a model to the data in §14).
  • P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", Acta Cryst. D61 (2005), 850-855 (the grid search over that range that fits k_sol and b_sol, with the overall scale and anisotropic B refitted at each grid point).
  • K. Shoemake, "Uniform Random Rotations", in Graphics Gems III, ed. D. Kirk, Academic Press (1992), 124-132 (the uniform random rotations the model-fit null of §14.5 is built from).
  • Z. Otwinowski & W. Minor, "Processing of X-ray diffraction data collected in oscillation mode", Methods Enzymol. 276 (1997), 307-326 (reweighted, de-biased profile-fit variances).
  • G. Winter et al., "DIALS: implementation and evaluation of a new integration package", Acta Cryst. D74 (2018), 85-97, and J. Beilsten-Edmands et al., Acta Cryst. D76 (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).
  • P. Evans, "Scaling and assessment of data quality", Acta Cryst. D62 (2006), 72-82, and P. R. Evans, Acta Cryst. D67 (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).
  • A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II 245, 41-51 (post-refinement practice: what is refined per image and what over a wedge).
  • D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", Acta Cryst. D77 (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).
  • K. Röttger, A. Endriss, J. Ihringer, S. Doyle & W. F. Kuhs, "Lattice constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K", Acta Cryst. B50 (1994), 644-648 (the ice Ih cell the ring positions below 1.522 Å are calculated from).
  • S. Sheriff & W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals", Acta Cryst. A43 (1987), 118-121 (the overall anisotropic B tensor and its symmetry constraints), and A. N. Popov & G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling", Acta Cryst. D59 (2003), 1145-1153 (the sigma-aware estimation of the anisotropy of the observed intensity distribution, part of that paper's statistic modelling).
  • P. R. Evans & G. N. Murshudov, "How good are my data and what is the resolution?", Acta Cryst. D69 (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).
  • K. Diederichs & P. A. Karplus, Nat. Struct. Biol. 4 (1997), 269-275, and P. A. Karplus & K. Diederichs, Science 336 (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).
  • IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", Acta Cryst. A45 (1989), 63-75, and Acta Cryst. A51 (1995), 565-569 (uncertainty conventions).

(list is not exhaustive; the full citations, with DOIs, are in ACKNOWLEDGEMENT.md)

Reference in New Issue View Git Blame Copy Permalink