Files

T

leonarski_f c999883eb1 model: --model is a hypothesis, and it decides nothing until it fits

A model supplied with --model rewrote the space group of every reflection
written out on the strength of its file having parsed. Measured on a rotation
dataset merged in P4(1)2(1)2: an unrelated protein and the correct model
rigidly rotated 90 degrees each produced a .mtz, .cif and _unmerged.mtz byte
for byte identical to what the crystal's own model produced - relabelled
P4(3)2(1)2 - at R-free 0.601 and 0.674 against the correct model's 0.591, with
no warning. The adoption was pure space-group-number arithmetic and ran before
the model had been fitted at all.

A model changes exactly two things on the rotation path, and both rewrite the
data: the enantiomorph label and the merohedral indexing. Both now wait for the
fit. Everything else a model produces - R-factors, maps, the rigid-body
placement - is a statement about the MODEL, cannot corrupt a reflection, and is
computed and reported either way.

The gate is not a threshold on R, because no threshold works: the classical
acentric random value is 0.586 at unit scale but 0.550 at the R-minimising
scale, observed nulls land at 0.599-0.615, and the value moves with the model's
atom count and B-factors as much as with the data. Instead the same model is
re-oriented at random about its own centroid five times and run through the
identical path - same scaling, same rigid-body placement, same R - and the real
fit is asked how far above that distribution it sits. R-work carries the
decision: nothing is refined against the working set here, and it has 12615
reflections to R-free's 709. Measured on the case above: the crystal's own model
+15.0 sigma, the unrelated protein +1.8, the 90-degree rotation -1.0, and the
two rejected runs now write files byte-identical to a run with no model.

The nulls are rigid-body refined like the real fit, or the comparison would be
between a placed model and unplaced nulls. That is what the null costs: about
12 s for the five replicates, on a --mode scale run that merges in 2 s.

The merohedral margin gets the same treatment - a random placement also picks a
winner, and measured, by a comparable lead - and the candidate operators are now
enumerated from the DATA's space group. Taking them from the model's enumerated
zero operators wherever the two groups differ, which is exactly the case the
probe exists for: a model in P4(3)2(1)2 against data merged in P4(3) probed
nothing at all, and now probes the twin law.

The anomalous difference map is the only measurement here sensitive to the hand
- inverting the model through the origin moves R-work by less than 1e-4, since
|F(h)| of the inverted structure is |F(-h)| - so where it says the hands
disagree it vetoes the adoption outright, fit or no fit.

Report: MODEL_FIT, MODEL_FIT_SIGMA and the null beside it, MODEL_DECISIONS_TAKEN,
the indexing margin against its null, and MODEL_VALIDATION= PERFORMED as the
counterpart of the failure line. SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL
becomes ASSUMED_FROM_MODEL and is written only where the model was accepted:
nothing here measured the hand, the model asserted it. That is a reason code
changing name and meaning, so REPORT_VERSION is 6.

Stills are untouched: the per-image indexing hand a model can set at integration
time is not reachable on the rotation path and is not gated here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N

2026-09-02 13:34:52 +02:00

8.4 KiB

Raw Blame History

CPU-side crystallographic data analysis (Jungfraujoch)

This document describes the crystallographic algorithms implemented in Jungfraujoch for CPU- and GPU-side real‑time and near‑real‑time data analysis.

Scope. The pipeline covered here comprises:

geometry mapping and corrections,
azimuthal integration (powder/radial profiles),
Bragg spot finding (strong pixels → connected components → spot descriptors),
indexing (still and rotation modes),
Bravais lattice / centering inference,
geometry and lattice refinement,
reflection prediction (still and rotation),
Bragg integration by either 2D box summation or profile fitting (Kabsch, reference-free),
scaling and merging,
merge-level error modelling, outlier rejection and the resolution cutoff,
space-group determination from the merged intensities (Laue group, screw axes, centering) and the twinning check,
auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
amplitude estimation (French–Wilson) and R-free test-set flagging,
optional model-based validation: rigid-body placement of a supplied model, R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.

The reference is split into four parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites.

From images to spots (§0–§3) — device-side decoding, geometry and reciprocal-space mapping, azimuthal integration, spot finding.
Indexing and geometry refinement (§4–§7) — FFT and fast-feedback indexing, the lattice search, geometry refinement, post-refinement and powder calibration.
Prediction, integration, scaling and merging (§8–§12) — reflection prediction, profile-fitted integration, scaling, merging, mosaicity and the auxiliary statistics.
Space group and validation (§13–§14) — the space-group search, twinning, the resolution cutoff, diffraction anisotropy, and model-based validation.

References

The methods draw on, and in places reimplement, solutions from:

W. Kabsch, “XDS”, Acta Cryst. D66 (2010), 125–132 and related XDS papers (rotation geometry, partiality, scaling concepts).
W. Kabsch, “Integration, scaling, space-group assignment and post-refinement”, Acta Cryst. D66 (2010), 133–144 (mosaicity/partiality likelihood treatment; notation such as ζ and rotation factors).
T. A. White et al., CrystFEL method papers (spot finding, three‑ring integration, serial/still diffraction processing concepts).
J. Kieffer & J. P. Wright, "PyFAI: a Python library for high performance azimuthal integration on GPU", Powder Diffraction 28 (2013), S339-S350 (detector geometry definition, azimuthal integration)
I. Steller, R. Bolotovsky & M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using Fourier analysis", J. Appl. Cryst. 30 (1997), 1036-1040 (the projection/1D-FFT autoindexing algorithm of §5).
H. Powell, "The Rossmann Fourier autoindexing algorithm in MOSFLM", Acta Cryst. D55 (1999), 1690-1695 (the MOSFLM implementation of it, whose practice is followed)
P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", J. Appl. Cryst. 57 (2024), 931-944 (the algorithm of the ffbidx fast-feedback indexer, §4).
I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", Acta Cryst. A32 (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", Acta Cryst. D59 (2003), 1124-1130 (the L test, §13.2).
A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", J. Appl. Cryst. 47 (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
S. French & K. Wilson, "On the treatment of negative intensity observations", Acta Cryst. A34 (1978), 517-525 (Bayesian amplitude estimation from intensities).
A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free cross-validation).
M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (model / structure-factor / map machinery used in §14).
J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and github.com/jonwright/bslz4decoders (device-side decoding of bitshuffle+LZ4 images, §0).
A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", J. Appl. Cryst. 44 (2011), 1285-1287 (anomalous difference density read at the model's sites).
R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the sigma_A formalism and the m, D weighting of the map coefficients of §14.4).
K. Shoemake, "Uniform Random Rotations", in Graphics Gems III, ed. D. Kirk, Academic Press (1992), 124-132 (the uniform random rotations the model-fit null of §14.5 is built from).
Z. Otwinowski & W. Minor, "Processing of X-ray diffraction data collected in oscillation mode", Methods Enzymol. 276 (1997), 307-326 (reweighted, de-biased profile-fit variances).
G. Winter et al., "DIALS: implementation and evaluation of a new integration package", Acta Cryst. D74 (2018), 85-97, and J. Beilsten-Edmands et al., Acta Cryst. D76 (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model).
P. Evans, "Scaling and assessment of data quality", Acta Cryst. D62 (2006), 72-82, and P. R. Evans, Acta Cryst. D67 (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test).
A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II 245, 41-51 (post-refinement practice: what is refined per image and what over a wedge).
D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", Acta Cryst. D77 (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant).
K. Röttger, A. Endriss, J. Ihringer, S. Doyle & W. F. Kuhs, "Lattice constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K", Acta Cryst. B50 (1994), 644-648 (the ice Ih cell the ring positions below 1.522 Å are calculated from).
S. Sheriff & W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals", Acta Cryst. A43 (1987), 118-121 (the overall anisotropic B tensor and its symmetry constraints), and A. N. Popov & G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling", Acta Cryst. D59 (2003), 1145-1153 (the sigma-aware estimation of the anisotropy of the observed intensity distribution, part of that paper's statistic modelling).
P. R. Evans & G. N. Murshudov, "How good are my data and what is the resolution?", Acta Cryst. D69 (2013), 1204-1214 (AIMLESS: the anisotropic deltaB as the range of the principal components, and diffraction limits from a cone about each principal direction).
K. Diederichs & P. A. Karplus, Nat. Struct. Biol. 4 (1997), 269-275, and P. A. Karplus & K. Diederichs, Science 336 (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC*).
IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", Acta Cryst. A45 (1989), 63-75, and Acta Cryst. A51 (1995), 565-569 (uncertainty conventions).

(list is not exhaustive; the full citations, with DOIs, are in ACKNOWLEDGEMENT.md)

8.4 KiB Raw Blame History Unescape Escape

CPU-side crystallographic data analysis (Jungfraujoch)

References

8.4 KiB

Raw Blame History