leonarski_fandClaude Opus 5
fb18457e6f
The rc.166 changelog was missing fourteen user-visible changes and carried rationale and measurements that belong here instead. Added: the native miniCBF sweep reader, the third-party and firmware-1.x NXmx masters, the plain LZ4 filter, the image orientation taken from the file's module direction vectors, the two beam-centre flags and their rescues, and the FFT reach past 500 A. Trimmed the rest to one line each, moving the numbers out of the user-facing file. RUGNUX.md described the input as a single Jungfraujoch master file, which it has not been since this branch; it now covers the foreign and legacy masters, the accepted compression filters and the miniCBF sweep, including how a sweep is collected from one named frame. Six options existed with no entry in the table - --beam-center-check, --beam-center-search, --fft-min-unit-cell, --min-indexed-spots, --rot3 and --no-p1-crosscheck - and -C now moves both FFT cell bounds, which was not written down anywhere. CPU_DATA_ANALYSIS.md carried two statements this branch made false: 7.5 still said pass 2 reuses pass 1's space group, and 13.1 still said centrings are ranked by net absence count. Both now describe what the code does - the group is determined after pass 2, and centrings are ranked by the same Beta-tail likelihood the screw test uses. Also documents the per-zone screw scoring, the coplanarity volume-fraction guard, the plane-normal transform, the FFT cell bounds and the twelve refined candidates. The miniCBF reader implements the x-CBF_BYTE_OFFSET scheme and reads the imgCIF axis table from the specification alone. No CBF code is vendored or linked, so there is no licence obligation, but reimplementing a published specification carries one of credit: ACKNOWLEDGEMENT.md gains a section and the two algorithms carry a one-line reference each. Both DOIs were resolved before being written. Rottger's initial was wrong where this branch first cited it - K, not A. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5
203 lines
16 KiB
Markdown
203 lines
16 KiB
Markdown
# Acknowledgements
|
|
|
|
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).
|
|
|
|
The project is supported by :
|
|
* Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
|
|
* ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)
|
|
* AMD University Program with donation of licenses of Ethernet IP cores and Vivado software
|
|
|
|
Decoding bitshuffle+LZ4 images on the GPU, rather than decompressing them on the host and uploading
|
|
the result, follows Jon Wright (ESRF): "Experiences with GPU decompression for bitshuffle + LZ4
|
|
data", HDF5 User Group meeting (2021), and [bslz4decoders](https://github.com/jonwright/bslz4decoders).
|
|
The CUDA kernels in Jungfraujoch are its own, but the approach is his.
|
|
|
|
Spot extraction groups strong pixels into spots with the sparse connected-component labelling of the
|
|
ACTS traccc project: P. Gessinger, H. M. Gray, A. Krasznahorkay, C. Leggett, J. Niermann,
|
|
A. Salzburger, S. N. Swatman and B. Yeo, "traccc: GPU track reconstruction library for HEP
|
|
experiments" (2025), [arXiv:2505.22822](https://arxiv.org/abs/2505.22822);
|
|
[traccc](https://github.com/acts-project/traccc). The CPU spot extractor adapts its SparseCCL source,
|
|
and the CUDA spot extractor follows the design of its GPU counterpart - a backward-neighbour graph
|
|
over a sorted hit list, resolved by a parallel union-find. traccc is MPL-2.0; see
|
|
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|
|
|
|
This software uses Viridis, Magma and Inferno colormaps from Matplotlib under its BSD-compatible license
|
|
|
|
## Public diffraction data used for testing
|
|
|
|
Jungfraujoch is tested against diffraction data collected on other people's beamlines, on
|
|
detectors and in file formats we do not produce ourselves - most of it at other facilities, a
|
|
few sets at the Swiss Light Source but not by this system. That data was collected and published
|
|
by other people. Every dataset used, the DOI to cite for it, and
|
|
the deposition it belongs to are listed in [EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md); we thank the
|
|
depositors, and the repositories that make the data findable and citable.
|
|
|
|
**[IRRMC](https://proteindiffraction.org/)**, the Integrated Resource for Reproducibility in
|
|
Macromolecular Crystallography (Minor lab, University of Virginia), is the source of most of them.
|
|
IRRMC releases its data under CC0 and asks that the DOI of the dataset be cited; those DOIs are in
|
|
the table. M. Grabowski, K. M. Langner, M. Cymborowski, P. J. Porebski, P. Sroka, H. Zheng,
|
|
D. R. Cooper, M. D. Zimmerman, M.-A. Elsliger, S. K. Burley and W. Minor, "A public database of
|
|
macromolecular diffraction experiments" (2016), Acta Cryst. D72, 1181-1193
|
|
[doi:10.1107/S2059798316014716](https://doi.org/10.1107/S2059798316014716); M. Grabowski,
|
|
M. Cymborowski, P. J. Porebski, T. Osinski, I. G. Shabalin, D. R. Cooper and W. Minor, "The
|
|
Integrated Resource for Reproducibility in Macromolecular Crystallography: Experiences of the first
|
|
four years" (2019), Struct. Dyn. 6, 064301
|
|
[doi:10.1063/1.5128672](https://doi.org/10.1063/1.5128672).
|
|
|
|
**[SBGrid Data Bank](https://data.sbgrid.org/)** supplied nine of the datasets. P. A. Meyer,
|
|
S. Socias, J. Key, E. Ransey, E. C. Tjon, A. Buschiazzo et al., "Data publication with the
|
|
structural biology data grid supports live analysis" (2016), Nat. Commun. 7, 10882
|
|
[doi:10.1038/ncomms10882](https://doi.org/10.1038/ncomms10882).
|
|
|
|
**[Zenodo](https://zenodo.org/)** hosts eleven, deposited there directly by the groups that
|
|
collected them. European Organization for Nuclear Research and OpenAIRE, "Zenodo" (2013), CERN
|
|
[doi:10.25495/7GXK-RD71](https://doi.org/10.25495/7GXK-RD71). Three of those datasets were
|
|
published as IUCrData Raw Data Letters; the letters are cited on the
|
|
[EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md) page, beside the datasets they describe.
|
|
|
|
The beamline, resolution, space group and unit cell quoted for each dataset are the values
|
|
deposited with the corresponding PDB entry, read from the RCSB PDB data API. H. M. Berman,
|
|
J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov and P. E. Bourne,
|
|
"The Protein Data Bank" (2000), Nucleic Acids Res. 28, 235-242
|
|
[doi:10.1093/nar/28.1.235](https://doi.org/10.1093/nar/28.1.235).
|
|
|
|
## File formats read from a published specification
|
|
|
|
**CBF / imgCIF** - the native miniCBF reader implements the `x-CBF_BYTE_OFFSET` compression scheme
|
|
and reads the imgCIF `_axis` table (the laboratory directions of the image's fast and slow pixel
|
|
directions, of the goniometer axes and of a 2theta arm) from the specification alone; no CBFlib or
|
|
other CBF code is used, so there is no licence obligation, only this credit.
|
|
H. J. Bernstein and A. P. Hammersley, "Specification of the Crystallographic Binary File
|
|
(CBF/imgCIF)" (2006), International Tables for Crystallography Vol. G, 37-43
|
|
[doi:10.1107/97809553602060000729](https://doi.org/10.1107/97809553602060000729);
|
|
A. P. Hammersley, H. J. Bernstein and J. D. Westbrook, "Image dictionary (imgCIF)" (2006),
|
|
International Tables for Crystallography Vol. G, 444-458
|
|
[doi:10.1107/97809553602060000746](https://doi.org/10.1107/97809553602060000746).
|
|
|
|
## Crystallographic methods adopted from other packages
|
|
|
|
The analysis pipeline reimplements methods first published, and in most cases first implemented, by
|
|
other crystallographic software. The code below is Jungfraujoch's own; the methods are theirs, and
|
|
are acknowledged here. Where a package's source was consulted this is said explicitly. None of these
|
|
packages is linked or vendored, with the single exception of GEMMI (see
|
|
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)).
|
|
|
|
**[XDS](https://xds.mr.mpg.de/)** — rotation geometry and notation, the reciprocal Lorentz and
|
|
partiality treatment, the maximum-likelihood mosaicity estimate, the `MINPK` criterion for rejecting
|
|
a reflection whose predicted profile is not cleanly its own, the intensity-based test for a
|
|
centred lattice, and the scaling correction surfaces indexed by image number and detector region. W. Kabsch, "XDS" (2010), Acta Cryst. D66, 125-132
|
|
[doi:10.1107/S0907444909047337](https://doi.org/10.1107/S0907444909047337); W. Kabsch, "Integration,
|
|
scaling, space-group assignment and post-refinement" (2010), Acta Cryst. D66, 133-144
|
|
[doi:10.1107/S0907444909047374](https://doi.org/10.1107/S0907444909047374).
|
|
|
|
**Profile fitting** with reweighted, de-biased variances is the Kabsch/Otwinowski iteration, from the
|
|
second XDS paper above and from Z. Otwinowski and W. Minor, "Processing of X-ray diffraction data
|
|
collected in oscillation mode" (1997), Methods Enzymol. 276, 307-326
|
|
[doi:10.1016/S0076-6879(97)76066-X](https://doi.org/10.1016/S0076-6879%2897%2976066-X).
|
|
|
|
**[DIALS](https://dials.github.io/)** — the resolution cutoff from the CC1/2 fall-off, per-observation
|
|
outlier rejection at merge, the scaling error model, and the treatment of a reflection whose
|
|
background is contaminated. Its published behaviour, and in places its source, settled several
|
|
choices here. G. Winter, D. G. Waterman, J. M. Parkhurst et al., "DIALS: implementation and
|
|
evaluation of a new integration package" (2018), Acta Cryst. D74, 85-97
|
|
[doi:10.1107/S2059798317017235](https://doi.org/10.1107/S2059798317017235); D. G. Waterman,
|
|
G. Winter, R. J. Gildea et al., "Diffraction-geometry refinement in the DIALS framework" (2016),
|
|
Acta Cryst. D72, 558-575 [doi:10.1107/S2059798316002187](https://doi.org/10.1107/S2059798316002187);
|
|
J. Beilsten-Edmands, G. Winter, R. Gildea et al., "Scaling diffraction data in the DIALS software
|
|
package: algorithms and new approaches for multi-crystal scaling" (2020), Acta Cryst. D76, 385-399
|
|
[doi:10.1107/S2059798320003198](https://doi.org/10.1107/S2059798320003198); J. M. Parkhurst,
|
|
G. Winter, D. G. Waterman et al., "Robust background modelling in DIALS" (2016), J. Appl. Cryst. 49,
|
|
1912-1921 [doi:10.1107/S1600576716013595](https://doi.org/10.1107/S1600576716013595).
|
|
|
|
**[POINTLESS](https://www.ccp4.ac.uk/)** (CCP4) — the space-group search. Stage A scores each
|
|
candidate rotation operator by the correlation of I(h) with I(Rh) on **resolution-normalised**
|
|
intensities (E²), as POINTLESS does — both arms of a symmetry pair sit at the same |s|, so on raw
|
|
intensities the resolution fall-off is variance shared between them and lifts a false operator's
|
|
correlation as much as a true one's; the screw-axis test scores a
|
|
predicted-absent class against the rest of its own axial row rather than against a global mean or a
|
|
fixed cut, and lets confidence fall away with the number of axial reflections instead of refusing
|
|
below a count. P. Evans, "Scaling and assessment of data quality" (2006), Acta Cryst. D62, 72-82
|
|
[doi:10.1107/S0907444905036693](https://doi.org/10.1107/S0907444905036693); P. R. Evans, "An
|
|
introduction to data reduction: space-group determination, scaling and intensity statistics" (2011),
|
|
Acta Cryst. D67, 282-292 [doi:10.1107/S090744491003982X](https://doi.org/10.1107/S090744491003982X);
|
|
P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta
|
|
Cryst. D69, 1204-1214 [doi:10.1107/S0907444913000061](https://doi.org/10.1107/S0907444913000061);
|
|
J. Agirre, M. Atanasova, H. Bagdonas et al., "The CCP4 suite: integrative software for macromolecular
|
|
crystallography" (2023), Acta Cryst. D79, 449-461
|
|
[doi:10.1107/S2059798323003595](https://doi.org/10.1107/S2059798323003595).
|
|
|
|
**[MOSFLM](https://www.mrc-lmb.cam.ac.uk/mosflm/)** — the Rossmann FFT autoindexing algorithm and
|
|
post-refinement practice, including which parameters are safe to refine per image and which must be
|
|
refined over a wedge. A. G. W. Leslie and H. R. Powell, "Processing diffraction data with MOSFLM"
|
|
(2007), in *Evolving Methods for Macromolecular Crystallography*, NATO Science Series II, vol. 245,
|
|
41-51 [doi:10.1007/978-1-4020-6316-9_4](https://doi.org/10.1007/978-1-4020-6316-9_4);
|
|
T. G. G. Battye, L. Kontogiannis, O. Johnson, H. R. Powell and A. G. W. Leslie, "iMOSFLM: a new
|
|
graphical interface for diffraction-image processing with MOSFLM" (2011), Acta Cryst. D67, 271-281
|
|
[doi:10.1107/S0907444910048675](https://doi.org/10.1107/S0907444910048675); H. R. Powell,
|
|
T. G. G. Battye, L. Kontogiannis, O. Johnson and A. G. W. Leslie, "Integrating macromolecular X-ray
|
|
diffraction data with the graphical user interface iMosflm" (2017), Nat. Protoc. 12, 1310-1325
|
|
[doi:10.1038/nprot.2017.037](https://doi.org/10.1038/nprot.2017.037).
|
|
|
|
**[CrystFEL](https://www.desy.de/~twhite/crystfel/)** — spot finding, the three-ring integration
|
|
region, the serial/stills processing model, and the per-frame indexing acceptance test
|
|
(`indexing_peak_check()` in `peaks.c`). T. A. White, R. A. Kirian, A. V. Martin, A. Aquila, K. Nass,
|
|
A. Barty and H. N. Chapman, "CrystFEL: a software suite for snapshot serial crystallography" (2012),
|
|
J. Appl. Cryst. 45, 335-341 [doi:10.1107/S0021889812002312](https://doi.org/10.1107/S0021889812002312).
|
|
|
|
**[GEMMI](https://github.com/project-gemmi/gemmi)** — symmetry operations, unit-cell and
|
|
structure-factor machinery, and MTZ / XDS_ASCII I/O. Vendored in `gemmi_gph/`, so it also carries a
|
|
licence obligation. M. Wojdyr, "GEMMI: A library for structural biology" (2022), J. Open Source
|
|
Softw. 7, 4200 [doi:10.21105/joss.04200](https://doi.org/10.21105/joss.04200).
|
|
|
|
**Hexagonal-ice ring positions** — the eleven ring $d$ spacings from 3.895 to 1.522 Å that the
|
|
ice-ring score, the ice-ring flagging and the ice calibrant are all built on are taken from the
|
|
measurements of, not enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in
|
|
biomolecular cryocrystallography" (2021), Acta Cryst. D77, 540-554
|
|
[doi:10.1107/S2059798321001170](https://doi.org/10.1107/S2059798321001170).
|
|
|
|
That list ends at 1.522 Å by its own scope, so the eight bands below it are calculated here rather
|
|
than taken from anyone: ice Ih structure factors on the oxygen sublattice, kept where they reach 3% of
|
|
the strongest line, which reproduces the eleven measured positions exactly. The lattice constants are
|
|
Röttger and co-workers'. K. Röttger, A. Endriss, J. Ihringer, S. Doyle and W. F. Kuhs, "Lattice
|
|
constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K" (1994), Acta Cryst. B50,
|
|
644-648 [doi:10.1107/S0108768194004933](https://doi.org/10.1107/S0108768194004933).
|
|
|
|
**Diffraction anisotropy** — the description of the overall fall-off by a single anisotropic
|
|
displacement tensor, its symmetry constraints, and the fact that only its deviatoric part is
|
|
determined (the isotropic part being degenerate with the overall scale) are Sheriff and Hendrickson's.
|
|
The estimator fits that tensor to the observed intensity distribution, taking sigma(I) into account,
|
|
in the sense of Popov and Bourenkov. The directional diffraction limits - <I/sigma(I)> in a cone about
|
|
each principal direction, and the reporting of the anisotropic deltaB as the range of the principal
|
|
components - follow AIMLESS. rugnux reports these; it corrects no intensity and removes no reflection
|
|
on a directional criterion. S. Sheriff and W. A. Hendrickson, "Description of overall anisotropy in
|
|
diffraction from macromolecular crystals" (1987), Acta Cryst. A43, 118-121
|
|
[doi:10.1107/S010876738709977X](https://doi.org/10.1107/S010876738709977X); A. N. Popov and
|
|
G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling" (2003), Acta
|
|
Cryst. D59, 1145-1153 [doi:10.1107/S0907444903008163](https://doi.org/10.1107/S0907444903008163);
|
|
P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta
|
|
Cryst. D69, 1204-1214 [doi:10.1107/S0907444913000061](https://doi.org/10.1107/S0907444913000061).
|
|
|
|
**[ANODE](https://doi.org/10.1107/S0021889811041768)** — reading the anomalous difference map at the
|
|
atoms of a supplied model and reporting the strongest sites by name, instead of searching the map for
|
|
blobs. The map itself is the textbook anomalous difference Fourier; what is taken from ANODE is that
|
|
reading: A. Thorn and G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation" (2011),
|
|
J. Appl. Cryst. 44, 1285-1287
|
|
[doi:10.1107/S0021889811041768](https://doi.org/10.1107/S0021889811041768).
|
|
|
|
**Data-quality statistics** follow the established conventions rather than any one program: R_meas
|
|
and R_pim, CC1/2 and CC\*, and the reporting of I/sigma(I). K. Diederichs and P. A. Karplus, "Improved
|
|
R-factors for diffraction data analysis in macromolecular crystallography" (1997), Nat. Struct. Biol.
|
|
4, 269-275 [doi:10.1038/nsb0497-269](https://doi.org/10.1038/nsb0497-269); P. A. Karplus and
|
|
K. Diederichs, "Linking crystallographic model and data quality" (2012), Science 336, 1030-1033
|
|
[doi:10.1126/science.1218231](https://doi.org/10.1126/science.1218231); K. Diederichs and
|
|
P. A. Karplus, "Better models by discarding data?" (2013), Acta Cryst. D69, 1215-1222
|
|
[doi:10.1107/S0907444913001121](https://doi.org/10.1107/S0907444913001121).
|
|
|
|
**Uncertainty conventions** follow the IUCr Commission on Crystallographic Nomenclature:
|
|
D. Schwarzenbach, S. C. Abrahams, H. D. Flack et al., "Statistical descriptors in crystallography:
|
|
Report of the IUCr Subcommittee on Statistical Descriptors" (1989), Acta Cryst. A45, 63-75
|
|
[doi:10.1107/S0108767388009596](https://doi.org/10.1107/S0108767388009596); D. Schwarzenbach,
|
|
S. C. Abrahams, H. D. Flack, E. Prince and A. J. C. Wilson, "Statistical descriptors in
|
|
crystallography. II. Report of a Working Group on Expression of Uncertainty in Measurement" (1995),
|
|
Acta Cryst. A51, 565-569 [doi:10.1107/S0108767395002340](https://doi.org/10.1107/S0108767395002340).
|