leonarski_fandClaude Opus 5
2222a4085c
RefineAbsorption indexes its surface by the diffracted direction de-rotated into the crystal frame, deliberately without a time axis, and RefineModulation indexes its by detector position, also without one. Nothing is indexed by (rotation, detector position), so the part of the absorption that changes as the crystal turns has no parameter at all. For a rigid absorber illuminating a fixed volume that is the right model: the incident path is a function of the spindle angle alone and the per-image scale takes it, and the exit path is then fixed in the crystal frame. What breaks the factorisation is the diffracting volume moving - a crystal larger than the beam, a mis-centred loop, ice building up. The exit path then depends on the spindle angle as well as the direction, and no time-independent surface reaches it. Measured on 34 rotation datasets, on XDS's own uncorrected intensities, as what is left after the crystal-frame absorption and detector modulation surfaces have taken what they can. The cross-validation gate lets the surface engage on 22 of them. Scored per resolution shell - the gate's whole-range ratio is lowered by any resolution-dependent scale without a reflection getting tighter, so the honest readout is each shell's own ratio, which a per-shell scale leaves unchanged - the median engaged crystal gains 4.1 %, the set gains 117 % summed against 19 % of damage, and 3 of the 22 are hurt. The surface has to be smooth in rotation angle to be absorption at all, and it is: the lag-1 autocorrelation of the fitted factor along the time axis runs +0.32 to +0.71 on the crystals it engages, against -0.08 for the same surface with its time bins shuffled. Where it is not smooth it is fitting something else, and says so - on a sweep whose beam was obstructed for a 70 deg wedge the autocorrelation is +0.16 and the profile is a cliff at the wedge, not a turn. Two null controls. Assign every observation a random cell and the gate refuses it (-1.2 % to -3.8 %). Keep the detector bin and shuffle only the time bin - a surface that cannot contain any time-dependent information - and the gate refuses that too, at +0.04 %, -0.60 % and +0.33 % on three crystals. Against the real surface's +3.7 % to +14.8 % on the same three. 12 time bins x a 10 x 10 detector grid = 1200 factors. On the per-shell score the median gain moves only between 3.1 % and 4.1 % across grids from 216 to 2400 cells, so the grid is second order; 12 x 10 has the largest net and the fewest crystals hurt. Equal-occupancy detector bins, not equal width: an equal-width grid starves the edges and the corners, and a starved cell is where a free surface over-fits. Fitted last, so the two time-independent surfaces get first claim on what they can explain. QUALIFICATION, measured after this was written: the "33 better / 0 worse" above is overall R_meas, which is a ratio of sums across every shell and is therefore lowered by any resolution-dependent scale without a reflection getting tighter - the same property that let the estimator bias pass its own gate. Scored per resolution shell instead, this surface HURTS 6 of 18 crystals under the acceptance gate as it currently stands, because that gate shares the defect and admits the surface where it should not. With a per-shell gate the surface is refused on exactly those crystals and its net over the chain goes from +86.3 to +159.5 per cent with none worse. The correction is right; the gate that decides where to apply it is the next commit's problem, not this one's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery against its own matched baseline - the same binary with the corrected estimator and without this surface: R_meas better 33 / worse 0, -78.7 R_meas_lo better 24 / worse 2, -20.7 CC1/2 better 8 / worse 0, +9.6 ISa better 30 / worse 2, +99.15 space groups unchanged at 35/38 The low-energy datasets gain most, which is what absorption should do: at 5-6 keV one crystal goes ISa 24.68 -> 37.42 and another 14.21 -> 22.08, while the same protein measured at 13 keV moves 13.41 -> 14.83. Taken with the estimator fix it precedes, against a clean baseline: R_meas_lo better 25 / worse 3 summed -26.0, ISa better 31 / worse 2 summed +113.1, outer-shell CC1/2 +83.0, observations +25 880 on 36 crystals of 38, CC1/2 flat at -1.4 and no space group moved. That last number is the point of the pair: the estimator fix alone costs CC1/2 -11.4, because the ramp it removes was partly standing in for this correction. One cost, predicted in advance and still unexplained: outer-shell CC1/2 falls on three of the four low-energy crystals, by 15.8 points on the worst, while every other statistic on those same crystals improves. The fourth goes up. On 5000-9000 That outer-shell fall has since been attributed, and it is not this surface: with the merge's 6-sigma outlier rejection turned off, the sign flips on every crystal that lost, +20.5 and +20.7 where it read -15.8 and -8.0. The surface removes most of the deviants in sample - it is fitted on all the data and applied to it, with no robustness of its own - so the merge's cut stops firing and the survivors land in a shell whose multiplicity is about three. Last-shell CC1/2 is largely made by that cut: one crystal's baseline goes 10.4 to 92.7 purely by dropping 19 per cent of the shell. A second qualification, measured after the numbers above were taken: overall R_meas is a ratio of sums across every shell, so any resolution-dependent scale lowers it without a reflection getting tighter - the same property that let the estimator bias pass its own gate. Scored per resolution shell instead, this surface hurts 6 of 18 crystals under the acceptance gate as it stands, because that gate shares the defect and admits the surface where it should not. Under a per-shell gate it is refused on exactly those crystals and its net over the correction chain goes from +86.3 to +159.5 per cent with none worse. The correction is right; where to apply it is the gate's problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
9.0 KiB
Acknowledgements
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.
The project is supported by :
- Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
- ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)
- AMD University Program with donation of licenses of Ethernet IP cores and Vivado software
Decoding bitshuffle+LZ4 images on the GPU, rather than decompressing them on the host and uploading the result, follows Jon Wright (ESRF): "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and bslz4decoders. The CUDA kernels in Jungfraujoch are its own, but the approach is his.
Spot extraction groups strong pixels into spots with the sparse connected-component labelling of the ACTS traccc project: P. Gessinger, H. M. Gray, A. Krasznahorkay, C. Leggett, J. Niermann, A. Salzburger, S. N. Swatman and B. Yeo, "traccc: GPU track reconstruction library for HEP experiments" (2025), arXiv:2505.22822; traccc. The CPU spot extractor adapts its SparseCCL source, and the CUDA spot extractor follows the design of its GPU counterpart - a backward-neighbour graph over a sorted hit list, resolved by a parallel union-find. traccc is MPL-2.0; see THIRD_PARTY_NOTICES.md.
This software uses Viridis, Magma and Inferno colormaps from Matplotlib under its BSD-compatible license
Crystallographic methods adopted from other packages
The analysis pipeline reimplements methods first published, and in most cases first implemented, by other crystallographic software. The code below is Jungfraujoch's own; the methods are theirs, and are acknowledged here. Where a package's source was consulted this is said explicitly. None of these packages is linked or vendored, with the single exception of GEMMI (see THIRD_PARTY_NOTICES.md).
XDS — rotation geometry and notation, the reciprocal Lorentz and
partiality treatment, the maximum-likelihood mosaicity estimate, the MINPK criterion for rejecting
a reflection whose predicted profile is not cleanly its own, the intensity-based test for a
centred lattice, and the scaling correction surfaces indexed by image number and detector region. W. Kabsch, "XDS" (2010), Acta Cryst. D66, 125-132
doi:10.1107/S0907444909047337; W. Kabsch, "Integration,
scaling, space-group assignment and post-refinement" (2010), Acta Cryst. D66, 133-144
doi:10.1107/S0907444909047374.
Profile fitting with reweighted, de-biased variances is the Kabsch/Otwinowski iteration, from the second XDS paper above and from Z. Otwinowski and W. Minor, "Processing of X-ray diffraction data collected in oscillation mode" (1997), Methods Enzymol. 276, 307-326 doi:10.1016/S0076-6879(97)76066-X.
DIALS — the resolution cutoff from the CC1/2 fall-off, per-observation outlier rejection at merge, the scaling error model, and the treatment of a reflection whose background is contaminated. Its published behaviour, and in places its source, settled several choices here. G. Winter, D. G. Waterman, J. M. Parkhurst et al., "DIALS: implementation and evaluation of a new integration package" (2018), Acta Cryst. D74, 85-97 doi:10.1107/S2059798317017235; D. G. Waterman, G. Winter, R. J. Gildea et al., "Diffraction-geometry refinement in the DIALS framework" (2016), Acta Cryst. D72, 558-575 doi:10.1107/S2059798316002187; J. Beilsten-Edmands, G. Winter, R. Gildea et al., "Scaling diffraction data in the DIALS software package: algorithms and new approaches for multi-crystal scaling" (2020), Acta Cryst. D76, 385-399 doi:10.1107/S2059798320003198; J. M. Parkhurst, G. Winter, D. G. Waterman et al., "Robust background modelling in DIALS" (2016), J. Appl. Cryst. 49, 1912-1921 doi:10.1107/S1600576716013595.
POINTLESS (CCP4) — the space-group search. Stage A scores each candidate rotation operator by the correlation of I(h) with I(Rh); the screw-axis test scores a predicted-absent class against the rest of its own axial row rather than against a global mean or a fixed cut, and lets confidence fall away with the number of axial reflections instead of refusing below a count. P. Evans, "Scaling and assessment of data quality" (2006), Acta Cryst. D62, 72-82 doi:10.1107/S0907444905036693; P. R. Evans, "An introduction to data reduction: space-group determination, scaling and intensity statistics" (2011), Acta Cryst. D67, 282-292 doi:10.1107/S090744491003982X; P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta Cryst. D69, 1204-1214 doi:10.1107/S0907444913000061; J. Agirre, M. Atanasova, H. Bagdonas et al., "The CCP4 suite: integrative software for macromolecular crystallography" (2023), Acta Cryst. D79, 449-461 doi:10.1107/S2059798323003595.
MOSFLM — the Rossmann FFT autoindexing algorithm and post-refinement practice, including which parameters are safe to refine per image and which must be refined over a wedge. A. G. W. Leslie and H. R. Powell, "Processing diffraction data with MOSFLM" (2007), in Evolving Methods for Macromolecular Crystallography, NATO Science Series II, vol. 245, 41-51 doi:10.1007/978-1-4020-6316-9_4; T. G. G. Battye, L. Kontogiannis, O. Johnson, H. R. Powell and A. G. W. Leslie, "iMOSFLM: a new graphical interface for diffraction-image processing with MOSFLM" (2011), Acta Cryst. D67, 271-281 doi:10.1107/S0907444910048675; H. R. Powell, T. G. G. Battye, L. Kontogiannis, O. Johnson and A. G. W. Leslie, "Integrating macromolecular X-ray diffraction data with the graphical user interface iMosflm" (2017), Nat. Protoc. 12, 1310-1325 doi:10.1038/nprot.2017.037.
CrystFEL — spot finding, the three-ring integration
region, the serial/stills processing model, and the per-frame indexing acceptance test
(indexing_peak_check() in peaks.c). T. A. White, R. A. Kirian, A. V. Martin, A. Aquila, K. Nass,
A. Barty and H. N. Chapman, "CrystFEL: a software suite for snapshot serial crystallography" (2012),
J. Appl. Cryst. 45, 335-341 doi:10.1107/S0021889812002312.
GEMMI — symmetry operations, unit-cell and
structure-factor machinery, and MTZ / XDS_ASCII I/O. Vendored in gemmi_gph/, so it also carries a
licence obligation. M. Wojdyr, "GEMMI: A library for structural biology" (2022), J. Open Source
Softw. 7, 4200 doi:10.21105/joss.04200.
Data-quality statistics follow the established conventions rather than any one program: R_meas and R_pim, CC1/2 and CC*, and the reporting of I/sigma(I). K. Diederichs and P. A. Karplus, "Improved R-factors for diffraction data analysis in macromolecular crystallography" (1997), Nat. Struct. Biol. 4, 269-275 doi:10.1038/nsb0497-269; P. A. Karplus and K. Diederichs, "Linking crystallographic model and data quality" (2012), Science 336, 1030-1033 doi:10.1126/science.1218231; K. Diederichs and P. A. Karplus, "Better models by discarding data?" (2013), Acta Cryst. D69, 1215-1222 doi:10.1107/S0907444913001121.
Uncertainty conventions follow the IUCr Commission on Crystallographic Nomenclature: D. Schwarzenbach, S. C. Abrahams, H. D. Flack et al., "Statistical descriptors in crystallography: Report of the IUCr Subcommittee on Statistical Descriptors" (1989), Acta Cryst. A45, 63-75 doi:10.1107/S0108767388009596; D. Schwarzenbach, S. C. Abrahams, H. D. Flack, E. Prince and A. J. C. Wilson, "Statistical descriptors in crystallography. II. Report of a Working Group on Expression of Uncertainty in Measurement" (1995), Acta Cryst. A51, 565-569 doi:10.1107/S0108767395002340.