leonarski_fandClaude Opus 5
799381d027
Where the merge kept the Bijvoet split - which the default Friedel-averaged merge does, so this needs no -A - --model now computes an anomalous difference Fourier, coefficients F(+)-F(-) on the model phase turned back by 90 degrees over the acentric reflections, writes it as <prefix>_anom.ccp4, and reads it at the model's own atom centres. The ten highest are logged and land in the results report as ANOMALOUS_SITE_01..10, each named by the atom, residue and chain it sits on, with its height in map sigma. Reading at the sites rather than searching the map for blobs is what ANODE does, and it says what carries the signal instead of leaving coordinates to look up; a scatterer the model does not contain is by construction absent from the list, which is what the map file is for. The list is always ten long, so its information is in the heights. On an S-SAD dataset at 5 keV the six cysteine sulfurs take ranks 1-6 (10.1 down to 4.2 sigma) and rank 7 falls to 2.2; the same crystal at 13 keV, where sulfur has no anomalous signal to speak of, tops out at 2.7 sigma with a water and a main chain carbon inside the top five. So the same list reads as a signal / no-signal gate, not only as a ranking. The reading is cubic. Grid::interpolate_value defaults to trilinear, and on the d_min/3 map grid that under-reads a peak this sharp by up to a quarter, unevenly enough to reorder the sites; cubic on the same grid is within about 2% of a sample-rate-10 one. The map r.m.s. is unchanged by the sample rate, so the normalisation was never at fault. The 2Fo-Fc reading behind MEAN_ATOM_DENSITY_SIGMA is deliberately left on the default so a number already in existing reports does not move. An anomalous merge keeps each Bijvoet mate as a row of its own and both rows carry the same F(+)/F(-) pair, attached to the + index of the Friedel ASU of the frame the merge was made in, so the differences are read from the merged reflections as they came in - carried into the model's frame here, on the + rows only. Taking the - rows as well would give one reflection both signs of its difference and let the last row written decide. write_ccp4 splits into an FFT and a write so the anomalous map can be both written and sampled from one transform; the 2Fo-Fc atom-centre score, which used to repeat that FFT, now reuses the grid. Cross-checked against ANODE run on this output: identical identification - its three peaks are the three disulfides, each maximum 0.64-0.70 A off a sulfur we name - with our heights about 1.35x lower. Sharpening, grid and sigma normalisation are excluded; the residual is SHELXC's FA preparation, which fed ANODE 2166 reflections against our 2534. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014zTy4Bpi4pPHw4bybf7q2R
12 KiB
Acknowledgements
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 doi:10.1107/S1600577522010268.
The project is supported by :
- Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
- ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)
- AMD University Program with donation of licenses of Ethernet IP cores and Vivado software
Decoding bitshuffle+LZ4 images on the GPU, rather than decompressing them on the host and uploading the result, follows Jon Wright (ESRF): "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and bslz4decoders. The CUDA kernels in Jungfraujoch are its own, but the approach is his.
Spot extraction groups strong pixels into spots with the sparse connected-component labelling of the ACTS traccc project: P. Gessinger, H. M. Gray, A. Krasznahorkay, C. Leggett, J. Niermann, A. Salzburger, S. N. Swatman and B. Yeo, "traccc: GPU track reconstruction library for HEP experiments" (2025), arXiv:2505.22822; traccc. The CPU spot extractor adapts its SparseCCL source, and the CUDA spot extractor follows the design of its GPU counterpart - a backward-neighbour graph over a sorted hit list, resolved by a parallel union-find. traccc is MPL-2.0; see THIRD_PARTY_NOTICES.md.
This software uses Viridis, Magma and Inferno colormaps from Matplotlib under its BSD-compatible license
Crystallographic methods adopted from other packages
The analysis pipeline reimplements methods first published, and in most cases first implemented, by other crystallographic software. The code below is Jungfraujoch's own; the methods are theirs, and are acknowledged here. Where a package's source was consulted this is said explicitly. None of these packages is linked or vendored, with the single exception of GEMMI (see THIRD_PARTY_NOTICES.md).
XDS — rotation geometry and notation, the reciprocal Lorentz and
partiality treatment, the maximum-likelihood mosaicity estimate, the MINPK criterion for rejecting
a reflection whose predicted profile is not cleanly its own, the intensity-based test for a
centred lattice, and the scaling correction surfaces indexed by image number and detector region. W. Kabsch, "XDS" (2010), Acta Cryst. D66, 125-132
doi:10.1107/S0907444909047337; W. Kabsch, "Integration,
scaling, space-group assignment and post-refinement" (2010), Acta Cryst. D66, 133-144
doi:10.1107/S0907444909047374.
Profile fitting with reweighted, de-biased variances is the Kabsch/Otwinowski iteration, from the second XDS paper above and from Z. Otwinowski and W. Minor, "Processing of X-ray diffraction data collected in oscillation mode" (1997), Methods Enzymol. 276, 307-326 doi:10.1016/S0076-6879(97)76066-X.
DIALS — the resolution cutoff from the CC1/2 fall-off, per-observation outlier rejection at merge, the scaling error model, and the treatment of a reflection whose background is contaminated. Its published behaviour, and in places its source, settled several choices here. G. Winter, D. G. Waterman, J. M. Parkhurst et al., "DIALS: implementation and evaluation of a new integration package" (2018), Acta Cryst. D74, 85-97 doi:10.1107/S2059798317017235; D. G. Waterman, G. Winter, R. J. Gildea et al., "Diffraction-geometry refinement in the DIALS framework" (2016), Acta Cryst. D72, 558-575 doi:10.1107/S2059798316002187; J. Beilsten-Edmands, G. Winter, R. Gildea et al., "Scaling diffraction data in the DIALS software package: algorithms and new approaches for multi-crystal scaling" (2020), Acta Cryst. D76, 385-399 doi:10.1107/S2059798320003198; J. M. Parkhurst, G. Winter, D. G. Waterman et al., "Robust background modelling in DIALS" (2016), J. Appl. Cryst. 49, 1912-1921 doi:10.1107/S1600576716013595.
POINTLESS (CCP4) — the space-group search. Stage A scores each candidate rotation operator by the correlation of I(h) with I(Rh) on resolution-normalised intensities (E²), as POINTLESS does — both arms of a symmetry pair sit at the same |s|, so on raw intensities the resolution fall-off is variance shared between them and lifts a false operator's correlation as much as a true one's; the screw-axis test scores a predicted-absent class against the rest of its own axial row rather than against a global mean or a fixed cut, and lets confidence fall away with the number of axial reflections instead of refusing below a count. P. Evans, "Scaling and assessment of data quality" (2006), Acta Cryst. D62, 72-82 doi:10.1107/S0907444905036693; P. R. Evans, "An introduction to data reduction: space-group determination, scaling and intensity statistics" (2011), Acta Cryst. D67, 282-292 doi:10.1107/S090744491003982X; P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta Cryst. D69, 1204-1214 doi:10.1107/S0907444913000061; J. Agirre, M. Atanasova, H. Bagdonas et al., "The CCP4 suite: integrative software for macromolecular crystallography" (2023), Acta Cryst. D79, 449-461 doi:10.1107/S2059798323003595.
MOSFLM — the Rossmann FFT autoindexing algorithm and post-refinement practice, including which parameters are safe to refine per image and which must be refined over a wedge. A. G. W. Leslie and H. R. Powell, "Processing diffraction data with MOSFLM" (2007), in Evolving Methods for Macromolecular Crystallography, NATO Science Series II, vol. 245, 41-51 doi:10.1007/978-1-4020-6316-9_4; T. G. G. Battye, L. Kontogiannis, O. Johnson, H. R. Powell and A. G. W. Leslie, "iMOSFLM: a new graphical interface for diffraction-image processing with MOSFLM" (2011), Acta Cryst. D67, 271-281 doi:10.1107/S0907444910048675; H. R. Powell, T. G. G. Battye, L. Kontogiannis, O. Johnson and A. G. W. Leslie, "Integrating macromolecular X-ray diffraction data with the graphical user interface iMosflm" (2017), Nat. Protoc. 12, 1310-1325 doi:10.1038/nprot.2017.037.
CrystFEL — spot finding, the three-ring integration
region, the serial/stills processing model, and the per-frame indexing acceptance test
(indexing_peak_check() in peaks.c). T. A. White, R. A. Kirian, A. V. Martin, A. Aquila, K. Nass,
A. Barty and H. N. Chapman, "CrystFEL: a software suite for snapshot serial crystallography" (2012),
J. Appl. Cryst. 45, 335-341 doi:10.1107/S0021889812002312.
GEMMI — symmetry operations, unit-cell and
structure-factor machinery, and MTZ / XDS_ASCII I/O. Vendored in gemmi_gph/, so it also carries a
licence obligation. M. Wojdyr, "GEMMI: A library for structural biology" (2022), J. Open Source
Softw. 7, 4200 doi:10.21105/joss.04200.
Hexagonal-ice ring positions — the eleven measured ring d spacings the ice-ring score, the
ice-ring flagging and the ice calibrant are all built on are taken from the measurements of, not
enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in biomolecular
cryocrystallography" (2021), Acta Cryst. D77, 540-554
doi:10.1107/S2059798321001170.
Diffraction anisotropy — the description of the overall fall-off by a single anisotropic displacement tensor, its symmetry constraints, and the fact that only its deviatoric part is determined (the isotropic part being degenerate with the overall scale) are Sheriff and Hendrickson's. The estimator fits that tensor to the observed intensity distribution, taking sigma(I) into account, in the sense of Popov and Bourenkov. The directional diffraction limits - <I/sigma(I)> in a cone about each principal direction, and the reporting of the anisotropic deltaB as the range of the principal components - follow AIMLESS. rugnux reports these; it corrects no intensity and removes no reflection on a directional criterion. S. Sheriff and W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals" (1987), Acta Cryst. A43, 118-121 doi:10.1107/S010876738709977X; A. N. Popov and G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling" (2003), Acta Cryst. D59, 1145-1153 doi:10.1107/S0907444903008163; P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta Cryst. D69, 1204-1214 doi:10.1107/S0907444913000061.
ANODE — reading the anomalous difference map at the atoms of a supplied model and reporting the strongest sites by name, instead of searching the map for blobs. The map itself is the textbook anomalous difference Fourier; what is taken from ANODE is that reading: A. Thorn and G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation" (2011), J. Appl. Cryst. 44, 1285-1287 doi:10.1107/S0021889811041768.
Data-quality statistics follow the established conventions rather than any one program: R_meas and R_pim, CC1/2 and CC*, and the reporting of I/sigma(I). K. Diederichs and P. A. Karplus, "Improved R-factors for diffraction data analysis in macromolecular crystallography" (1997), Nat. Struct. Biol. 4, 269-275 doi:10.1038/nsb0497-269; P. A. Karplus and K. Diederichs, "Linking crystallographic model and data quality" (2012), Science 336, 1030-1033 doi:10.1126/science.1218231; K. Diederichs and P. A. Karplus, "Better models by discarding data?" (2013), Acta Cryst. D69, 1215-1222 doi:10.1107/S0907444913001121.
Uncertainty conventions follow the IUCr Commission on Crystallographic Nomenclature: D. Schwarzenbach, S. C. Abrahams, H. D. Flack et al., "Statistical descriptors in crystallography: Report of the IUCr Subcommittee on Statistical Descriptors" (1989), Acta Cryst. A45, 63-75 doi:10.1107/S0108767388009596; D. Schwarzenbach, S. C. Abrahams, H. D. Flack, E. Prince and A. J. C. Wilson, "Statistical descriptors in crystallography. II. Report of a Working Group on Expression of Uncertainty in Measurement" (1995), Acta Cryst. A51, 565-569 doi:10.1107/S0108767395002340.