Files

T

leonarski_fandClaude Opus 5.5 e0e3e92147 rugnux --model rigid body: Fcalc composed from one P1 transform, analytic translation Jacobian

B+C of the rigid-body speed-up plan (not bit-exact, by design):

- Fcalc is composed from the transform of ONE copy of the model, gridded without
  symmetrization: F(h) = n_cen * sum_S exp(2 pi i h.t) F1(hR), through a per-zone gather table
  (half-l index, Friedel flag, phase, Cartesian hR). Only the indices prepare_asu_data() would
  give a value are composed (ASU, strict d_min, Nyquist box, model-group absences, not 000),
  so the unmatched-observation semantics are unchanged. The mask stays symmetrized and is read
  at h directly.
- Translation columns of the Jacobian are exact (the phase factor's derivative), computed with
  every evaluation; rotation columns are forward differences needing only one copy gridded and
  transformed (3 per Jacobian instead of 6 full evaluations).
- The Jacobian holds the scale and the bulk-solvent mask at the evaluation's; the residuals keep
  both exact. The per-evaluation scale re-fit is folded in by Kaufman's variable projection.
- Composition and projection reductions are per-reflection / serial, so the result does not
  depend on the thread count.

Tests: RigidBodyP1FcalcMatchesSymmetrized (hkl set identical, <= 4.3e-6 of mean |F|),
RigidBodyTranslationDerivativeIsExact (5-point difference, <= 6.3e-4, the grid sampling),
RigidBodyRotationDerivativeMatchesDifferences (q != 0, anisotropic atoms, cos >= 0.9968),
RigidBodyJacobianMatchesNumericScaleRefit (cos >= 0.998 at 0.07 A; >= 0.983 at 0.35 A, the
residual-weighted term Kaufman drops). Existing rigid-body tests pass unchanged.

Validation on 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: 9hnc's real fit, which
moved 0.000 deg / 0.000 A before, now moves 1.46 deg / 0.65 A and R-free 0.5159 -> 0.4651
(MODEL_FIT_SIGMA 15.4 -> 43.3). STOP finding on 9yzk: a null replicate drawn outside the
redraw guard's 10.6 deg reach was walked 23.9 deg onto an equivalent orientation (R-free
0.5578 -> 0.3917), the null SD went 0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED (12.2 sigma)
-> REJECTED (2.6 sigma). The stronger optimiser's capture radius outgrows the null's redraw
guard; not ready to merge until that is addressed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C

2026-09-28 09:53:10 +02:00

11 KiB

Raw Blame History

CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for CPU- and GPU-side real‑time and near‑real‑time data analysis.

Scope. The pipeline covered here comprises:

geometry mapping and corrections,
azimuthal integration (powder/radial profiles),
Bragg spot finding (strong pixels → connected components → spot descriptors),
indexing (still and rotation modes),
Bravais lattice / centering inference,
geometry and lattice refinement,
reflection prediction (still and rotation),
Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
scaling and merging,
merge-level error modelling, outlier rejection and the resolution cutoff,
space-group determination from the merged intensities (Laue group, screw axes, glide planes, centering), the twinning check and the translational pseudo-symmetry check,
auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
amplitude estimation (French–Wilson) and R-free test-set flagging,
optional model-based validation: rigid-body placement of a supplied model, R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.

The reference is split into four parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites.

From images to spots (§0–§3) — device-side decoding, geometry and reciprocal-space mapping, azimuthal integration, spot finding.
Indexing and geometry refinement (§4–§7) — FFT and fast-feedback indexing, the lattice search, geometry refinement, post-refinement and powder calibration.
Prediction, integration, scaling and merging (§8–§12) — reflection prediction, profile-fitted integration, scaling, merging, mosaicity and the auxiliary statistics.
Space group and validation (§13–§14) — the space-group search, twinning and translational pseudo-symmetry, the resolution cutoff, diffraction anisotropy, and model-based validation.

References

The methods draw on, and in places reimplement, solutions from:

W. Kabsch, “XDS”, Acta Cryst. D66 (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, Acta Cryst. D66 (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", Powder Diffraction 28 (2013), S339-S350 (detector geometry definition, azimuthal integration)
I. Steller, R. Bolotovsky & M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using Fourier analysis", J. Appl. Cryst. 30 (1997), 1036-1040 (the projection/1D-FFT autoindexing algorithm of §5).
H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", Acta Cryst. D55 (1999), 1690-1695 (the MOSFLM implementation of it, whose practice is followed)
P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", J. Appl. Cryst. 57 (2024), 931-944 (the algorithm of the ffbidx fast-feedback indexer, §4).
I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", Acta Cryst. A32 (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", Acta Cryst. D59 (2003), 1124-1130 (the L test, §13.2).
R. J. Read, P. D. Adams & A. J. McCoy, "Intensity statistics in the presence of translational noncrystallographic symmetry", Acta Cryst. D69 (2013), 176-183 (the native-Patterson detection of translational pseudo-symmetry, and the intensity modulation it produces, which the axial-zone screw-absence test scores against).
A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", J. Appl. Cryst. 47 (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
S. French & K. Wilson, "On the treatment of negative intensity observations", Acta Cryst. A34 (1978), 517-525 (Bayesian amplitude estimation from intensities).
A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free cross-validation).
R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", Biometrika 10 (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared).
M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (model / structure-factor / map machinery used in §14).
J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and github.com/jonwright/bslz4decoders (device-side decoding of bitshuffle+LZ4 images, §0).
A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", J. Appl. Cryst. 44 (2011), 1285-1287 (anomalous difference density read at the model's sites).
R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", J. Appl. Cryst. 15 (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to the azimuthal profile, the Bragg intensities and the ring background the beam-stop shadow test compares against).
R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4).
A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", Acta Cryst. D58 (2002), 1387-1392 (the flat bulk-solvent model, its optimal parameters and the range they are physically meaningful over, used when scaling a model to the data in §14).
P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", Acta Cryst. D61 (2005), 850-855 (the grid search over that range that fits k_sol and b_sol, with the overall scale and anisotropic B refitted at each grid point).
K. Shoemake, "Uniform Random Rotations", in Graphics Gems III, ed. D. Kirk, Academic Press (1992), 124-132 (the uniform random rotations the model-fit null of §14.5 is built from).
G. H. Golub & V. Pereyra, "The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate", SIAM J. Numer. Anal. 10 (1973), 413-432, and L. Kaufman, "A variable projection method for solving separable nonlinear least squares problems", BIT 15 (1975), 49-57 (the scale re-fit folded into the rigid-body Jacobian of §14.8).
Z. Otwinowski & W. Minor, "Processing of X-ray diffraction data collected in oscillation mode", Methods Enzymol. 276 (1997), 307-326 (reweighted, de-biased profile-fit variances).
G. Winter et al., "DIALS: implementation and evaluation of a new integration package", Acta Cryst. D74 (2018), 85-97, and J. Beilsten-Edmands et al., Acta Cryst. D76 (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).
R. H. Blessing, "An empirical correction for absorption anisotropy", Acta Cryst. A51 (1995), 33-38 (absorption as spherical harmonics of the beam directions).
P. Evans, "Scaling and assessment of data quality", Acta Cryst. D62 (2006), 72-82, and P. R. Evans, Acta Cryst. D67 (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).
A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II 245, 41-51 (post-refinement practice: what is refined per image and what over a wedge).
D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", Acta Cryst. D77 (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).
K. Röttger, A. Endriss, J. Ihringer, S. Doyle & W. F. Kuhs, "Lattice constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K", Acta Cryst. B50 (1994), 644-648 (the ice Ih cell the ring positions below 1.522 Å are calculated from).
S. Sheriff & W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals", Acta Cryst. A43 (1987), 118-121 (the overall anisotropic B tensor and its symmetry constraints), and A. N. Popov & G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling", Acta Cryst. D59 (2003), 1145-1153 (the sigma-aware estimation of the anisotropy of the observed intensity distribution, part of that paper's statistic modelling).
P. R. Evans & G. N. Murshudov, "How good are my data and what is the resolution?", Acta Cryst. D69 (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).
G. Assmann, W. Brehm & K. Diederichs, "Identification of rogue datasets in serial crystallography", J. Appl. Cryst. 49 (2016), 1021-1028, and G. M. Assmann, M. Wang & K. Diederichs, Acta Cryst. D76 (2020), 636-652 (XDSCC12: sigma-tau CC1/2, delta-CC1/2, the Fisher transformation and the rejection discipline the frame disposition follows).
K. Diederichs & P. A. Karplus, Nat. Struct. Biol. 4 (1997), 269-275, and P. A. Karplus & K. Diederichs, Science 336 (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).
IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", Acta Cryst. A45 (1989), 63-75, and Acta Cryst. A51 (1995), 565-569 (uncertainty conventions).

(list is not exhaustive; the full citations, with DOIs, are in ACKNOWLEDGEMENT.md)

11 KiB Raw Blame History Unescape Escape

CPU-side crystallographic data analysis (Jungfraujoch)

References

11 KiB

Raw Blame History