v1.0.0-rc.166 (#76)
Build Packages / Unit tests (push) Successful in 1h22m15s
Build Packages / build:windows:nocuda (push) Successful in 18m0s
Build Packages / build:windows:cuda (push) Successful in 20m30s
Build Packages / build:viewer-tgz:cpu (push) Successful in 10m32s
Build Packages / build:viewer-tgz:cuda (push) Successful in 11m39s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 8m55s
Build Packages / build:rugnux:windows (push) Successful in 11m25s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 20m6s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m27s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 20m19s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 15m34s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 20m25s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m36s
Build Packages / build:rpm (rocky8) (push) Successful in 17m43s
Build Packages / build:rpm (rocky9) (push) Successful in 13m34s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 21m28s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 18m19s
Build Packages / DIALS test (push) Successful in 12m36s
Build Packages / XDS test (durin plugin) (push) Successful in 6m56s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m48s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m7s
Build Packages / Generate python client (push) Successful in 11s
Build Packages / Build documentation (push) Successful in 36s
Build Packages / Create release (push) Skipped
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 5m11s

* `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands.
* `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion.
* Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants.
* `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing.
* A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed.
* `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing.
* Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences.
* The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to.
* The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after.
* The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution.
* `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have.
* Twinning is no longer reported when the L-test contradicts it.
* The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's.
* `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots.
* The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area.

Reviewed-on: #76
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #76.
This commit is contained in:
leonarski_f authored and leonarski_f committed 2026-09-02 21:17:31 +02:00
1 parent 511be0c366
commit 680c36c20d
383 files changed
+20910 -3936

No files matched your search

+136 -6
View File
@@ -19,10 +19,65 @@ experiments" (2025), [arXiv:2505.22822](https://arxiv.org/abs/2505.22822);
[traccc](https://github.com/acts-project/traccc). The CPU spot extractor adapts its SparseCCL source,
and the CUDA spot extractor follows the design of its GPU counterpart - a backward-neighbour graph
over a sorted hit list, resolved by a parallel union-find. traccc is MPL-2.0; see
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). The SparseCCL algorithm itself is A. Hennequin,
B. Couturier, V. V. Gligorov and L. Lacassagne, "SparseCCL: Connected Components Labeling and
Analysis for sparse images" (2019), DASIP 2019, 65-70
[doi:10.1109/DASIP48288.2019.9049184](https://doi.org/10.1109/DASIP48288.2019.9049184).
This software uses Viridis, Magma and Inferno colormaps from Matplotlib under its BSD-compatible license
## Public diffraction data used for testing
In addition to in-house datasets collected at SLS 2.0, Jungfraujoch is tested against public
diffraction data collected on other people's beamlines, on detectors and in file formats we do not
produce ourselves - most of it at other facilities, a few sets at the Swiss Light Source but not by
this system. That data was collected and published by other people. Every dataset used, the DOI to
cite for it, and the deposition it belongs to are listed in
[EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md); we thank the depositors, and the repositories that make
the data findable and citable.
**[IRRMC](https://proteindiffraction.org/)**, the Integrated Resource for Reproducibility in
Macromolecular Crystallography (Minor lab, University of Virginia), is the source of most of them.
IRRMC releases its data under CC0 and asks that the DOI of the dataset be cited; those DOIs are in
the table. M. Grabowski, K. M. Langner, M. Cymborowski, P. J. Porebski, P. Sroka, H. Zheng,
D. R. Cooper, M. D. Zimmerman, M.-A. Elsliger, S. K. Burley and W. Minor, "A public database of
macromolecular diffraction experiments" (2016), Acta Cryst. D72, 1181-1193
[doi:10.1107/S2059798316014716](https://doi.org/10.1107/S2059798316014716); M. Grabowski,
M. Cymborowski, P. J. Porebski, T. Osinski, I. G. Shabalin, D. R. Cooper and W. Minor, "The
Integrated Resource for Reproducibility in Macromolecular Crystallography: Experiences of the first
four years" (2019), Struct. Dyn. 6, 064301
[doi:10.1063/1.5128672](https://doi.org/10.1063/1.5128672).
**[SBGrid Data Bank](https://data.sbgrid.org/)** supplied nine of the datasets. P. A. Meyer,
S. Socias, J. Key, E. Ransey, E. C. Tjon, A. Buschiazzo et al., "Data publication with the
structural biology data grid supports live analysis" (2016), Nat. Commun. 7, 10882
[doi:10.1038/ncomms10882](https://doi.org/10.1038/ncomms10882).
**[Zenodo](https://zenodo.org/)** hosts eleven, deposited there directly by the groups that
collected them. European Organization for Nuclear Research and OpenAIRE, "Zenodo" (2013), CERN
[doi:10.25495/7GXK-RD71](https://doi.org/10.25495/7GXK-RD71). Three of those datasets were
published as IUCrData Raw Data Letters; the letters are cited on the
[EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md) page, beside the datasets they describe.
The beamline, resolution, space group and unit cell quoted for each dataset are the values
deposited with the corresponding PDB entry, read from the RCSB PDB data API. H. M. Berman,
J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov and P. E. Bourne,
"The Protein Data Bank" (2000), Nucleic Acids Res. 28, 235-242
[doi:10.1093/nar/28.1.235](https://doi.org/10.1093/nar/28.1.235).
## File formats read from a published specification
**CBF / imgCIF** - the native miniCBF reader implements the `x-CBF_BYTE_OFFSET` compression scheme
and reads the imgCIF `_axis` table (the laboratory directions of the image's fast and slow pixel
directions, of the goniometer axes and of a 2theta arm) from the specification alone; no CBFlib or
other CBF code is used, so there is no licence obligation, only this credit.
H. J. Bernstein and A. P. Hammersley, "Specification of the Crystallographic Binary File
(CBF/imgCIF)" (2006), International Tables for Crystallography Vol. G, 37-43
[doi:10.1107/97809553602060000729](https://doi.org/10.1107/97809553602060000729);
A. P. Hammersley, H. J. Bernstein and J. D. Westbrook, "Image dictionary (imgCIF)" (2006),
International Tables for Crystallography Vol. G, 444-458
[doi:10.1107/97809553602060000746](https://doi.org/10.1107/97809553602060000746).
## Crystallographic methods adopted from other packages
The analysis pipeline reimplements methods first published, and in most cases first implemented, by
@@ -44,6 +99,14 @@ second XDS paper above and from Z. Otwinowski and W. Minor, "Processing of X-ray
collected in oscillation mode" (1997), Methods Enzymol. 276, 307-326
[doi:10.1016/S0076-6879(97)76066-X](https://doi.org/10.1016/S0076-6879%2897%2976066-X).
**The two-dimensional integration architecture** — integrating each image in the detector plane
and only afterwards assembling a reflection's partials into a full across images, as against
three-dimensional profile fitting through the image stack — is the architecture of DENZO/SCALEPACK
and MOSFLM, and it is the one rugnux's rotation pipeline follows (per-image profile-fitted
integration, then partials combined into fulls; §9 and §10.6 of the analysis reference). The
Otwinowski & Minor citation above and the MOSFLM citations below carry the credit for the paradigm
as well as for the specifics taken from each.
**[DIALS](https://dials.github.io/)** — the resolution cutoff from the CC1/2 fall-off, per-observation
outlier rejection at merge, the scaling error model, and the treatment of a reflection whose
background is contaminated. Its published behaviour, and in places its source, settled several
@@ -77,7 +140,12 @@ crystallography" (2023), Acta Cryst. D79, 449-461
**[MOSFLM](https://www.mrc-lmb.cam.ac.uk/mosflm/)** — the Rossmann FFT autoindexing algorithm and
post-refinement practice, including which parameters are safe to refine per image and which must be
refined over a wedge. A. G. W. Leslie and H. R. Powell, "Processing diffraction data with MOSFLM"
refined over a wedge. The autoindexing algorithm itself — projecting the reciprocal-space points
onto many directions and Fourier-transforming the 1D projection histograms — is I. Steller,
R. Bolotovsky and M. G. Rossmann, "An algorithm for automatic indexing of oscillation images using
Fourier analysis" (1997), J. Appl. Cryst. 30, 1036-1040
[doi:10.1107/S0021889897008777](https://doi.org/10.1107/S0021889897008777); MOSFLM is the
implementation whose practice is followed. A. G. W. Leslie and H. R. Powell, "Processing diffraction data with MOSFLM"
(2007), in *Evolving Methods for Macromolecular Crystallography*, NATO Science Series II, vol. 245,
41-51 [doi:10.1007/978-1-4020-6316-9_4](https://doi.org/10.1007/978-1-4020-6316-9_4);
T. G. G. Battye, L. Kontogiannis, O. Johnson, H. R. Powell and A. G. W. Leslie, "iMOSFLM: a new
@@ -92,18 +160,66 @@ region, the serial/stills processing model, and the per-frame indexing acceptanc
(`indexing_peak_check()` in `peaks.c`). T. A. White, R. A. Kirian, A. V. Martin, A. Aquila, K. Nass,
A. Barty and H. N. Chapman, "CrystFEL: a software suite for snapshot serial crystallography" (2012),
J. Appl. Cryst. 45, 335-341 [doi:10.1107/S0021889812002312](https://doi.org/10.1107/S0021889812002312).
The self-calibrating spot finder's per-resolution-ring background statistics, with the Bragg peaks
excluded by iterated clipping, follow Cheetah's peakfinder8: A. Barty, R. A. Kirian,
F. R. N. C. Maia, M. Hantke, C. H. Yoon, T. A. White and H. N. Chapman, "Cheetah: software for
high-throughput reduction and analysis of serial femtosecond X-ray diffraction data" (2014),
J. Appl. Cryst. 47, 1118-1131
[doi:10.1107/S1600576714007626](https://doi.org/10.1107/S1600576714007626).
**[fast-feedback-indexer](https://github.com/paulscherrerinstitute/fast-feedback-indexer)** — the
known-cell indexer for serial stills (`-X ffbidx`) is PSI's fast-feedback-indexer library, linked at
build time (BSD-3-Clause; see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)), which implements
the TORO algorithm: P. Gasparotto, L. Barba, H.-C. Stadler, G. Assmann, H. Mendonça, A. W. Ashton,
M. Janousch, F. Leonarski and B. Béjar, "TORO Indexer: a PyTorch-based indexing algorithm for
kilohertz serial crystallography" (2024), J. Appl. Cryst. 57, 931-944
[doi:10.1107/S1600576724003182](https://doi.org/10.1107/S1600576724003182).
**[GEMMI](https://github.com/project-gemmi/gemmi)** — symmetry operations, unit-cell and
structure-factor machinery, and MTZ / XDS_ASCII I/O. Vendored in `gemmi_gph/`, so it also carries a
licence obligation. M. Wojdyr, "GEMMI: A library for structural biology" (2022), J. Open Source
Softw. 7, 4200 [doi:10.21105/joss.04200](https://doi.org/10.21105/joss.04200).
**Hexagonal-ice ring positions** — the eleven measured ring $d$ spacings the ice-ring score, the
ice-ring flagging and the ice calibrant are all built on are taken from the measurements of, not
enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in biomolecular
cryocrystallography" (2021), Acta Cryst. D77, 540-554
**Křivý & Gruber's Niggli reduction, and the lattice-character table** — the reduction that puts
every candidate cell in a comparable form is I. Křivý and B. Gruber, "A unified algorithm for
determining the reduced (Niggli) cell" (1976), Acta Cryst. A32, 297-298
[doi:10.1107/S0567739476000636](https://doi.org/10.1107/S0567739476000636), used through GEMMI's
implementation; the table of lattice characters that maps a reduced cell to Bravais lattices and
centrings follows International Tables for Crystallography Vol. A, Table 9.2.5.1.
**Grosse-Kunstleve, Sauter & Adams's numerically stable cell reduction** - the magnitude-scaled
tolerance that decides the sign of a structurally-zero scalar product, and with it the Niggli type a
reduced cell is presented in. R. W. Grosse-Kunstleve, N. K. Sauter and P. D. Adams, "Numerically
stable algorithms for the computation of reduced unit cells" (2004), Acta Cryst. A60, 1-6
[doi:10.1107/S010876730302186X](https://doi.org/10.1107/S010876730302186X).
**Le Page's metric-symmetry search** - the obliquity of each of the 81 candidate two-folds of a
reduced cell, which is what tells a run that its lattice metric hosts more rotational symmetry than
the group its intensities supported. Used through GEMMI's implementation of it. Y. Le Page, "The
derivation of the axes of the conventional unit cell from the dimensions of the Buerger-reduced
cell" (1982), J. Appl. Cryst. 15, 255-259
[doi:10.1107/S0021889882011959](https://doi.org/10.1107/S0021889882011959).
**Hexagonal-ice ring positions** — the eleven ring $d$ spacings from 3.895 to 1.522 Å that the
ice-ring score, the ice-ring flagging and the ice calibrant are all built on are taken from the
measurements of, not enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in
biomolecular cryocrystallography" (2021), Acta Cryst. D77, 540-554
[doi:10.1107/S2059798321001170](https://doi.org/10.1107/S2059798321001170).
That list ends at 1.522 Å by its own scope, so the eight bands below it are calculated here rather
than taken from anyone: ice Ih structure factors on the oxygen sublattice, kept where they reach 3% of
the strongest line, which reproduces the eleven measured positions exactly. The lattice constants are
Röttger and co-workers'. K. Röttger, A. Endriss, J. Ihringer, S. Doyle and W. F. Kuhs, "Lattice
constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K" (1994), Acta Cryst. B50,
644-648 [doi:10.1107/S0108768194004933](https://doi.org/10.1107/S0108768194004933).
**The twinning L test** is Padilla and Yeates's: pairing each acentric reflection with a
symmetry-independent neighbour and reading the first and second moments of
L = (I1−I2)/(I1+I2) against their untwinned and perfect-twin values. J. E. Padilla and
T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and
pseudo-centering and utility for detecting twinning" (2003), Acta Cryst. D59, 1124-1130
[doi:10.1107/S0907444903007947](https://doi.org/10.1107/S0907444903007947).
**Diffraction anisotropy** — the description of the overall fall-off by a single anisotropic
displacement tensor, its symmetry constraints, and the fact that only its deviatoric part is
determined (the isotropic part being degenerate with the overall scale) are Sheriff and Hendrickson's.
@@ -126,6 +242,20 @@ reading: A. Thorn and G. M. Sheldrick, "ANODE: anomalous and heavy-atom density
J. Appl. Cryst. 44, 1285-1287
[doi:10.1107/S0021889811041768](https://doi.org/10.1107/S0021889811041768).
**sigma_A map coefficients** — the maps written by `--model` are weighted by a maximum-likelihood
sigma_A estimated per resolution shell, giving 2mFo-DFc and mFo-DFc rather than 2Fo-Fc and Fo-Fc.
What is taken is the formalism itself: the Rice and Woolfson likelihoods of |Fo| given |Fc| and
sigma_A, the figure of merit m and the scale D that follow from it, and the result that the
bias-corrected coefficient is 2mFo-DFc for an acentric reflection and mFo for a centric one.
R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with
errors" (1986), Acta Cryst. A42, 140-149
[doi:10.1107/S0108767386099622](https://doi.org/10.1107/S0108767386099622).
**Uniform random rotations** — the null a supplied model is scored against re-orients that model at
random about its own centroid, and the rotations are drawn uniformly from SO(3) through a uniform
random unit quaternion. K. Shoemake, "Uniform Random Rotations", in *Graphics Gems III*, ed. D. Kirk,
Academic Press (1992), 124-132 (no DOI).
**Data-quality statistics** follow the established conventions rather than any one program: R_meas
and R_pim, CC1/2 and CC\*, and the reporting of I/sigma(I). K. Diederichs and P. A. Karplus, "Improved
R-factors for diffraction data analysis in macromolecular crystallography" (1997), Nat. Struct. Biol.