leonarski_f
511be0c366
Build Packages / build:rpm (rocky8) (push) Successful in 24m0s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m54s
Build Packages / build:windows:cuda (push) Successful in 19m25s
Build Packages / build:viewer-tgz:cpu (push) Successful in 14m44s
Build Packages / build:viewer-tgz:cuda (push) Successful in 16m3s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 13m15s
Build Packages / build:rugnux:windows (push) Successful in 10m45s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 9m34s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 19m7s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 18m9s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 24m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 18m13s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 24m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 22m58s
Build Packages / build:rpm (rocky9) (push) Successful in 21m23s
Build Packages / Generate python client (push) Successful in 1m2s
Build Packages / Build documentation (push) Successful in 1m23s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 9m45s
Build Packages / XDS test (neggia plugin) (push) Successful in 10m19s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m10s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 22m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 17m37s
Build Packages / DIALS test (push) Successful in 17m16s
* `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75
145 lines
12 KiB
Markdown
145 lines
12 KiB
Markdown
# Acknowledgements
|
|
|
|
Citation: F. Leonarski, M. Bruckner, C. Lopez-Cuenca, A. Mozzanica, H.-C. Stadler, Z. Matej, A. Castellane, B. Mesnet, J. Wojdyla, B. Schmitt and M. Wang "Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors" (2023), J. Synchrotron Rad., 30, 227-234 [doi:10.1107/S1600577522010268](https://doi.org/10.1107/S1600577522010268).
|
|
|
|
The project is supported by :
|
|
* Innosuisse via Innovation Project "NextGenDCU high data rate acquisition system for X-ray detectors in structural biology applications" (101.535.1 IP-ENG; Apr 2023 - Sep 2025).
|
|
* ETH Domain via Open Research Data Contribute project (Jan - Dec 2023)
|
|
* AMD University Program with donation of licenses of Ethernet IP cores and Vivado software
|
|
|
|
Decoding bitshuffle+LZ4 images on the GPU, rather than decompressing them on the host and uploading
|
|
the result, follows Jon Wright (ESRF): "Experiences with GPU decompression for bitshuffle + LZ4
|
|
data", HDF5 User Group meeting (2021), and [bslz4decoders](https://github.com/jonwright/bslz4decoders).
|
|
The CUDA kernels in Jungfraujoch are its own, but the approach is his.
|
|
|
|
Spot extraction groups strong pixels into spots with the sparse connected-component labelling of the
|
|
ACTS traccc project: P. Gessinger, H. M. Gray, A. Krasznahorkay, C. Leggett, J. Niermann,
|
|
A. Salzburger, S. N. Swatman and B. Yeo, "traccc: GPU track reconstruction library for HEP
|
|
experiments" (2025), [arXiv:2505.22822](https://arxiv.org/abs/2505.22822);
|
|
[traccc](https://github.com/acts-project/traccc). The CPU spot extractor adapts its SparseCCL source,
|
|
and the CUDA spot extractor follows the design of its GPU counterpart - a backward-neighbour graph
|
|
over a sorted hit list, resolved by a parallel union-find. traccc is MPL-2.0; see
|
|
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|
|
|
|
This software uses Viridis, Magma and Inferno colormaps from Matplotlib under its BSD-compatible license
|
|
|
|
## Crystallographic methods adopted from other packages
|
|
|
|
The analysis pipeline reimplements methods first published, and in most cases first implemented, by
|
|
other crystallographic software. The code below is Jungfraujoch's own; the methods are theirs, and
|
|
are acknowledged here. Where a package's source was consulted this is said explicitly. None of these
|
|
packages is linked or vendored, with the single exception of GEMMI (see
|
|
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)).
|
|
|
|
**[XDS](https://xds.mr.mpg.de/)** — rotation geometry and notation, the reciprocal Lorentz and
|
|
partiality treatment, the maximum-likelihood mosaicity estimate, the `MINPK` criterion for rejecting
|
|
a reflection whose predicted profile is not cleanly its own, the intensity-based test for a
|
|
centred lattice, and the scaling correction surfaces indexed by image number and detector region. W. Kabsch, "XDS" (2010), Acta Cryst. D66, 125-132
|
|
[doi:10.1107/S0907444909047337](https://doi.org/10.1107/S0907444909047337); W. Kabsch, "Integration,
|
|
scaling, space-group assignment and post-refinement" (2010), Acta Cryst. D66, 133-144
|
|
[doi:10.1107/S0907444909047374](https://doi.org/10.1107/S0907444909047374).
|
|
|
|
**Profile fitting** with reweighted, de-biased variances is the Kabsch/Otwinowski iteration, from the
|
|
second XDS paper above and from Z. Otwinowski and W. Minor, "Processing of X-ray diffraction data
|
|
collected in oscillation mode" (1997), Methods Enzymol. 276, 307-326
|
|
[doi:10.1016/S0076-6879(97)76066-X](https://doi.org/10.1016/S0076-6879%2897%2976066-X).
|
|
|
|
**[DIALS](https://dials.github.io/)** — the resolution cutoff from the CC1/2 fall-off, per-observation
|
|
outlier rejection at merge, the scaling error model, and the treatment of a reflection whose
|
|
background is contaminated. Its published behaviour, and in places its source, settled several
|
|
choices here. G. Winter, D. G. Waterman, J. M. Parkhurst et al., "DIALS: implementation and
|
|
evaluation of a new integration package" (2018), Acta Cryst. D74, 85-97
|
|
[doi:10.1107/S2059798317017235](https://doi.org/10.1107/S2059798317017235); D. G. Waterman,
|
|
G. Winter, R. J. Gildea et al., "Diffraction-geometry refinement in the DIALS framework" (2016),
|
|
Acta Cryst. D72, 558-575 [doi:10.1107/S2059798316002187](https://doi.org/10.1107/S2059798316002187);
|
|
J. Beilsten-Edmands, G. Winter, R. Gildea et al., "Scaling diffraction data in the DIALS software
|
|
package: algorithms and new approaches for multi-crystal scaling" (2020), Acta Cryst. D76, 385-399
|
|
[doi:10.1107/S2059798320003198](https://doi.org/10.1107/S2059798320003198); J. M. Parkhurst,
|
|
G. Winter, D. G. Waterman et al., "Robust background modelling in DIALS" (2016), J. Appl. Cryst. 49,
|
|
1912-1921 [doi:10.1107/S1600576716013595](https://doi.org/10.1107/S1600576716013595).
|
|
|
|
**[POINTLESS](https://www.ccp4.ac.uk/)** (CCP4) — the space-group search. Stage A scores each
|
|
candidate rotation operator by the correlation of I(h) with I(Rh) on **resolution-normalised**
|
|
intensities (E²), as POINTLESS does — both arms of a symmetry pair sit at the same |s|, so on raw
|
|
intensities the resolution fall-off is variance shared between them and lifts a false operator's
|
|
correlation as much as a true one's; the screw-axis test scores a
|
|
predicted-absent class against the rest of its own axial row rather than against a global mean or a
|
|
fixed cut, and lets confidence fall away with the number of axial reflections instead of refusing
|
|
below a count. P. Evans, "Scaling and assessment of data quality" (2006), Acta Cryst. D62, 72-82
|
|
[doi:10.1107/S0907444905036693](https://doi.org/10.1107/S0907444905036693); P. R. Evans, "An
|
|
introduction to data reduction: space-group determination, scaling and intensity statistics" (2011),
|
|
Acta Cryst. D67, 282-292 [doi:10.1107/S090744491003982X](https://doi.org/10.1107/S090744491003982X);
|
|
P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta
|
|
Cryst. D69, 1204-1214 [doi:10.1107/S0907444913000061](https://doi.org/10.1107/S0907444913000061);
|
|
J. Agirre, M. Atanasova, H. Bagdonas et al., "The CCP4 suite: integrative software for macromolecular
|
|
crystallography" (2023), Acta Cryst. D79, 449-461
|
|
[doi:10.1107/S2059798323003595](https://doi.org/10.1107/S2059798323003595).
|
|
|
|
**[MOSFLM](https://www.mrc-lmb.cam.ac.uk/mosflm/)** — the Rossmann FFT autoindexing algorithm and
|
|
post-refinement practice, including which parameters are safe to refine per image and which must be
|
|
refined over a wedge. A. G. W. Leslie and H. R. Powell, "Processing diffraction data with MOSFLM"
|
|
(2007), in *Evolving Methods for Macromolecular Crystallography*, NATO Science Series II, vol. 245,
|
|
41-51 [doi:10.1007/978-1-4020-6316-9_4](https://doi.org/10.1007/978-1-4020-6316-9_4);
|
|
T. G. G. Battye, L. Kontogiannis, O. Johnson, H. R. Powell and A. G. W. Leslie, "iMOSFLM: a new
|
|
graphical interface for diffraction-image processing with MOSFLM" (2011), Acta Cryst. D67, 271-281
|
|
[doi:10.1107/S0907444910048675](https://doi.org/10.1107/S0907444910048675); H. R. Powell,
|
|
T. G. G. Battye, L. Kontogiannis, O. Johnson and A. G. W. Leslie, "Integrating macromolecular X-ray
|
|
diffraction data with the graphical user interface iMosflm" (2017), Nat. Protoc. 12, 1310-1325
|
|
[doi:10.1038/nprot.2017.037](https://doi.org/10.1038/nprot.2017.037).
|
|
|
|
**[CrystFEL](https://www.desy.de/~twhite/crystfel/)** — spot finding, the three-ring integration
|
|
region, the serial/stills processing model, and the per-frame indexing acceptance test
|
|
(`indexing_peak_check()` in `peaks.c`). T. A. White, R. A. Kirian, A. V. Martin, A. Aquila, K. Nass,
|
|
A. Barty and H. N. Chapman, "CrystFEL: a software suite for snapshot serial crystallography" (2012),
|
|
J. Appl. Cryst. 45, 335-341 [doi:10.1107/S0021889812002312](https://doi.org/10.1107/S0021889812002312).
|
|
|
|
**[GEMMI](https://github.com/project-gemmi/gemmi)** — symmetry operations, unit-cell and
|
|
structure-factor machinery, and MTZ / XDS_ASCII I/O. Vendored in `gemmi_gph/`, so it also carries a
|
|
licence obligation. M. Wojdyr, "GEMMI: A library for structural biology" (2022), J. Open Source
|
|
Softw. 7, 4200 [doi:10.21105/joss.04200](https://doi.org/10.21105/joss.04200).
|
|
|
|
**Hexagonal-ice ring positions** — the eleven measured ring $d$ spacings the ice-ring score, the
|
|
ice-ring flagging and the ice calibrant are all built on are taken from the measurements of, not
|
|
enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in biomolecular
|
|
cryocrystallography" (2021), Acta Cryst. D77, 540-554
|
|
[doi:10.1107/S2059798321001170](https://doi.org/10.1107/S2059798321001170).
|
|
|
|
**Diffraction anisotropy** — the description of the overall fall-off by a single anisotropic
|
|
displacement tensor, its symmetry constraints, and the fact that only its deviatoric part is
|
|
determined (the isotropic part being degenerate with the overall scale) are Sheriff and Hendrickson's.
|
|
The estimator fits that tensor to the observed intensity distribution, taking sigma(I) into account,
|
|
in the sense of Popov and Bourenkov. The directional diffraction limits - <I/sigma(I)> in a cone about
|
|
each principal direction, and the reporting of the anisotropic deltaB as the range of the principal
|
|
components - follow AIMLESS. rugnux reports these; it corrects no intensity and removes no reflection
|
|
on a directional criterion. S. Sheriff and W. A. Hendrickson, "Description of overall anisotropy in
|
|
diffraction from macromolecular crystals" (1987), Acta Cryst. A43, 118-121
|
|
[doi:10.1107/S010876738709977X](https://doi.org/10.1107/S010876738709977X); A. N. Popov and
|
|
G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling" (2003), Acta
|
|
Cryst. D59, 1145-1153 [doi:10.1107/S0907444903008163](https://doi.org/10.1107/S0907444903008163);
|
|
P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta
|
|
Cryst. D69, 1204-1214 [doi:10.1107/S0907444913000061](https://doi.org/10.1107/S0907444913000061).
|
|
|
|
**[ANODE](https://doi.org/10.1107/S0021889811041768)** — reading the anomalous difference map at the
|
|
atoms of a supplied model and reporting the strongest sites by name, instead of searching the map for
|
|
blobs. The map itself is the textbook anomalous difference Fourier; what is taken from ANODE is that
|
|
reading: A. Thorn and G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation" (2011),
|
|
J. Appl. Cryst. 44, 1285-1287
|
|
[doi:10.1107/S0021889811041768](https://doi.org/10.1107/S0021889811041768).
|
|
|
|
**Data-quality statistics** follow the established conventions rather than any one program: R_meas
|
|
and R_pim, CC1/2 and CC\*, and the reporting of I/sigma(I). K. Diederichs and P. A. Karplus, "Improved
|
|
R-factors for diffraction data analysis in macromolecular crystallography" (1997), Nat. Struct. Biol.
|
|
4, 269-275 [doi:10.1038/nsb0497-269](https://doi.org/10.1038/nsb0497-269); P. A. Karplus and
|
|
K. Diederichs, "Linking crystallographic model and data quality" (2012), Science 336, 1030-1033
|
|
[doi:10.1126/science.1218231](https://doi.org/10.1126/science.1218231); K. Diederichs and
|
|
P. A. Karplus, "Better models by discarding data?" (2013), Acta Cryst. D69, 1215-1222
|
|
[doi:10.1107/S0907444913001121](https://doi.org/10.1107/S0907444913001121).
|
|
|
|
**Uncertainty conventions** follow the IUCr Commission on Crystallographic Nomenclature:
|
|
D. Schwarzenbach, S. C. Abrahams, H. D. Flack et al., "Statistical descriptors in crystallography:
|
|
Report of the IUCr Subcommittee on Statistical Descriptors" (1989), Acta Cryst. A45, 63-75
|
|
[doi:10.1107/S0108767388009596](https://doi.org/10.1107/S0108767388009596); D. Schwarzenbach,
|
|
S. C. Abrahams, H. D. Flack, E. Prince and A. J. C. Wilson, "Statistical descriptors in
|
|
crystallography. II. Report of a Working Group on Expression of Uncertainty in Measurement" (1995),
|
|
Acta Cryst. A51, 565-569 [doi:10.1107/S0108767395002340](https://doi.org/10.1107/S0108767395002340).
|