diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index ba9f14c24..dc748edc1 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -22,39 +22,6 @@ The project is supported by: * ETH Domain via Open Research Data Contribute project (Jan - Dec 2023). * AMD University Program with donation of licenses of Ethernet IP cores and Vivado software. -## Automatic processing as an expert system - -**Global Phasing Ltd** — much of what rugnux tries to do was inspired by the work of Gérard Bricogne -and his colleagues at [Global Phasing](https://www.globalphasing.com/): the Fourier-transform -indexing its de-novo indexers descend from, which Bricogne proposed in 1986 (see -[Indexing](#indexing) below), the view of data processing as an expert system, the distrust of -image-header metadata, and the attention paid to diffraction anisotropy, each described below. - -**[autoPROC](https://www.globalphasing.com/autoproc/)** — rugnux's goal is a fully automatic -pipeline that takes the decisions an experienced crystallographer would take while processing a -dataset, and reports each decision together with the evidence it rests on, so that a bare -invocation gives the best answer available. autoPROC is the inspiration and the target for that -behaviour, and two of rugnux's working principles come from it. The first, and the one that has -guided rugnux most, we owe to Clemens Vonrhein: a processing pipeline should not take the -experimental metadata it is given - the beam centre, the detector geometry, the goniometer and its -rotation axis - on trust, but verify it against the images and, where it is wrong, correct it. We -are grateful to him for arguing this so clearly and for so long. The second follows from it: the -beam centre should be determined from the images themselves; rugnux does this in its own way, -described in §1.4 of the [data-analysis reference](CPU_DATA_ANALYSIS_IMAGE.md). The -reasons rugnux gives for a weak stretch of a sweep also use autoPROC's wording ("loss of centring -during crystal rotation"). C. Vonrhein, C. Flensburg, P. Keller, A. Sharff, O. Smart, W. Paciorek, -T. Womack and G. Bricogne, "Data processing and analysis with the autoPROC toolbox" (2011), -Acta Cryst. D67, 293-302 -[doi:10.1107/S0907444911007773](https://doi.org/10.1107/S0907444911007773). - -**[STARANISO](https://staraniso.globalphasing.org/)** — the understanding that diffraction -anisotropy matters, and that every run should analyse and report it, was inspired by STARANISO. -The estimators rugnux uses for it are those credited under -[Scaling, merging and data quality](#scaling-merging-and-data-quality) below. I. J. Tickle, -C. Flensburg, P. Keller, W. Paciorek, A. Sharff, C. Vonrhein and G. Bricogne (2016), STARANISO -([staraniso.globalphasing.org/cgi-bin/staraniso.cgi](https://staraniso.globalphasing.org/cgi-bin/staraniso.cgi)), -Cambridge, United Kingdom: Global Phasing Ltd. - ## Crystallographic methods adopted from other packages The analysis pipeline builds on methods first published, and in most cases first implemented, by @@ -64,6 +31,54 @@ packages are neither linked nor vendored; the three that are - GEMMI, traccc and fast-feedback-indexer - also carry a licence obligation, recorded in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). +### Experimental geometry and physical corrections + +**[autoPROC](https://www.globalphasing.com/autoproc/)** — the principle that a processing pipeline +does not take the experimental metadata it is given - the beam centre, the detector geometry, the +goniometer and its rotation axis - on trust, but verifies it against the images and, where it is +wrong, corrects it, which Clemens Vonrhein has long argued for; and, following from it, that the beam +centre is determined from the images themselves (rugnux does this in its own way, described in §1.4 +of the [data-analysis reference](CPU_DATA_ANALYSIS_IMAGE.md)). autoPROC is also the model for the +behaviour rugnux aims at as a whole: a pipeline that takes the decisions an experienced +crystallographer would take while processing a dataset and reports each decision together with the +evidence it rests on, so that a bare invocation gives the best answer available. The reasons rugnux +gives for a weak stretch of a sweep use its wording ("loss of centring during crystal rotation"). +C. Vonrhein, C. Flensburg, P. Keller, A. Sharff, O. Smart, W. Paciorek, T. Womack and G. Bricogne, +"Data processing and analysis with the autoPROC toolbox" (2011), Acta Cryst. D67, 293-302 +[doi:10.1107/S0907444911007773](https://doi.org/10.1107/S0907444911007773). + +**Sensor absorption at oblique incidence, and the flight path** — the angle-dependent quantum +efficiency of a flat sensor, the radial parallax variance that comes from the same integral, and the +attenuation of a reflection in the air between the sample and its pixel are all the Beer-Lambert law +taken along a ray that crosses t/cos(alpha) of sensor, or D/cos(alpha) of air, and converts at a +random depth. The attenuation coefficients, for silicon, CdTe, dry air and helium alike, are the +NIST tabulation: J. H. Hubbell and S. M. Seltzer, "Tables of X-Ray +Mass Attenuation Coefficients and Mass Energy-Absorption Coefficients from 1 keV to 20 MeV for +Elements Z = 1 to 92 and 48 Additional Substances of Dosimetric Interest" (1995, data updated 2004), +NIST Standard Reference Database 126 +[doi:10.18434/T4D01F](https://doi.org/10.18434/T4D01F). + +**Polarization correction** — the azimuthal polarization factor applied to the azimuthally +integrated profile, to the integrated Bragg intensities and to the ring background the beam-stop +shadow test compares a pixel against is the one derived for a partially polarized +synchrotron source by R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas and D. Andre, "Macromolecular +crystallography with synchrotron radiation: photographic data collection and polarization +correction" (1982), J. Appl. Cryst. 15, 330-337 +[doi:10.1107/S0021889882012060](https://doi.org/10.1107/S0021889882012060). + +**Hexagonal-ice ring positions** — the eleven ring $d$ spacings from 3.895 to 1.522 Å that the +ice-ring score, the ice-ring flagging and the ice calibrant are all built on are taken from the +measurements of Moreau and co-workers, not enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in +biomolecular cryocrystallography" (2021), Acta Cryst. D77, 540-554 +[doi:10.1107/S2059798321001170](https://doi.org/10.1107/S2059798321001170). + +That list ends at 1.522 Å by its own scope, so the eight bands below it are calculated here rather +than taken from anyone: ice Ih structure factors on the oxygen sublattice, kept where they reach 3% of +the strongest line, which reproduces the eleven measured positions exactly. The lattice constants are +Röttger and co-workers'. K. Röttger, A. Endriss, J. Ihringer, S. Doyle and W. F. Kuhs, "Lattice +constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K" (1994), Acta Cryst. B50, +644-648 [doi:10.1107/S0108768194004933](https://doi.org/10.1107/S0108768194004933). + ### Spot finding **[CrystFEL](https://www.desy.de/~twhite/crystfel/)** — spot finding, the three-ring integration @@ -78,16 +93,6 @@ high-throughput reduction and analysis of serial femtosecond X-ray diffraction d J. Appl. Cryst. 47, 1118-1131 [doi:10.1107/S1600576714007626](https://doi.org/10.1107/S1600576714007626). -**Per-crystal post-refinement of serial stills** - refining, for each still, the partiality -parameters the spot positions cannot pin (a small reorientation, the crystal's own rocking width and the -radial scale of its reciprocal lattice, which a shot's photon energy or a crystal's cell moves) against -the merged intensities follows the post-refinement of PRIME and the mosaic-block model of Sauter et al.: -M. Uervirojnangkoorn, O. B. Zeldin, A. Y. Lyubimov, J. Hattne, A. S. Brewster, N. K. Sauter, A. T. Brunger -and W. I. Weis, "Enabling X-ray free electron laser crystallography for challenging biological systems -from a limited number of crystals" (2015), eLife 4, e05421; N. K. Sauter, J. Hattne, A. S. Brewster, -N. Echols, P. H. Zwart and P. D. Adams, "Improved crystal orientation and physical properties from -single-shot XFEL stills" (2014), Acta Cryst. D70, 3299-3309 (DOIs not verified here, so not given). - Spot extraction groups strong pixels into spots with the sparse connected-component labelling of the ACTS traccc project: P. Gessinger, H. M. Gray, A. Krasznahorkay, C. Leggett, J. Niermann, A. Salzburger, S. N. Swatman and B. Yeo, "traccc: GPU track reconstruction library for HEP @@ -362,13 +367,18 @@ The estimator fits that tensor to the observed intensity distribution, taking si in the sense of Popov and Bourenkov. The directional diffraction limits - in a cone about each principal direction, and the reporting of the anisotropic deltaB as the range of the principal components - follow AIMLESS. Rugnux reports these; it corrects no intensity and removes no reflection -on a directional criterion. S. Sheriff and W. A. Hendrickson, "Description of overall anisotropy in +on a directional criterion. That diffraction anisotropy matters, and that every run should analyse +and report it, is inspired by STARANISO. S. Sheriff and W. A. Hendrickson, "Description of overall anisotropy in diffraction from macromolecular crystals" (1987), Acta Cryst. A43, 118-121 [doi:10.1107/S010876738709977X](https://doi.org/10.1107/S010876738709977X); A. N. Popov and G. P. Bourenkov, "Choice of data-collection parameters based on statistic modelling" (2003), Acta Cryst. D59, 1145-1153 [doi:10.1107/S0907444903008163](https://doi.org/10.1107/S0907444903008163); P. R. Evans and G. N. Murshudov, "How good are my data and what is the resolution?" (2013), Acta -Cryst. D69, 1204-1214 [doi:10.1107/S0907444913000061](https://doi.org/10.1107/S0907444913000061). +Cryst. D69, 1204-1214 [doi:10.1107/S0907444913000061](https://doi.org/10.1107/S0907444913000061); +[STARANISO](https://staraniso.globalphasing.org/): I. J. Tickle, C. Flensburg, P. Keller, +W. Paciorek, A. Sharff, C. Vonrhein and G. Bricogne (2016), STARANISO +([staraniso.globalphasing.org/cgi-bin/staraniso.cgi](https://staraniso.globalphasing.org/cgi-bin/staraniso.cgi)), +Cambridge, United Kingdom: Global Phasing Ltd. **Data-quality statistics** follow the established conventions rather than any one program: R_meas and R_pim, CC1/2 and CC\*, the per-shell CC(model, data) between F^2_calc and F^2_obs, and the @@ -445,6 +455,16 @@ consecutive frames and writes the larger of the two estimates: F. H. Herbstein, measurements of unit-cell dimensions from single crystals?" (2000), Acta Cryst. B56, 547-557 [doi:10.1107/S010876810000269X](https://doi.org/10.1107/S010876810000269X). +**Per-crystal post-refinement of serial stills** - refining, for each still, the partiality +parameters the spot positions cannot pin (a small reorientation, the crystal's own rocking width and the +radial scale of its reciprocal lattice, which a shot's photon energy or a crystal's cell moves) against +the merged intensities follows the post-refinement of PRIME and the mosaic-block model of Sauter et al.: +M. Uervirojnangkoorn, O. B. Zeldin, A. Y. Lyubimov, J. Hattne, A. S. Brewster, N. K. Sauter, A. T. Brunger +and W. I. Weis, "Enabling X-ray free electron laser crystallography for challenging biological systems +from a limited number of crystals" (2015), eLife 4, e05421; N. K. Sauter, J. Hattne, A. S. Brewster, +N. Echols, P. H. Zwart and P. D. Adams, "Improved crystal orientation and physical properties from +single-shot XFEL stills" (2014), Acta Cryst. D70, 3299-3309 (DOIs not verified here, so not given). + **Indexing ambiguity of serial data without a reference** — where a serial run's lattice is more symmetric than its crystals and no reference or model is given, each crystal is put into the indexing in which its intensities agree best with those of all the other crystals, crystal by crystal until none @@ -484,40 +504,6 @@ comparison is local in resolution, where xtriage's is a z-score of a shell again that a second, tilted sweep recovers are the data-collection geometry set out by Z. Dauter, "Data-collection strategies" (1999), Acta Cryst. D55, 1703-1717 (DOI not verified here). -### Physical corrections and calibration - -**Sensor absorption at oblique incidence, and the flight path** — the angle-dependent quantum -efficiency of a flat sensor, the radial parallax variance that comes from the same integral, and the -attenuation of a reflection in the air between the sample and its pixel are all the Beer-Lambert law -taken along a ray that crosses t/cos(alpha) of sensor, or D/cos(alpha) of air, and converts at a -random depth. The attenuation coefficients, for silicon, CdTe, dry air and helium alike, are the -NIST tabulation: J. H. Hubbell and S. M. Seltzer, "Tables of X-Ray -Mass Attenuation Coefficients and Mass Energy-Absorption Coefficients from 1 keV to 20 MeV for -Elements Z = 1 to 92 and 48 Additional Substances of Dosimetric Interest" (1995, data updated 2004), -NIST Standard Reference Database 126 -[doi:10.18434/T4D01F](https://doi.org/10.18434/T4D01F). - -**Polarization correction** — the azimuthal polarization factor applied to the azimuthally -integrated profile, to the integrated Bragg intensities and to the ring background the beam-stop -shadow test compares a pixel against is the one derived for a partially polarized -synchrotron source by R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas and D. Andre, "Macromolecular -crystallography with synchrotron radiation: photographic data collection and polarization -correction" (1982), J. Appl. Cryst. 15, 330-337 -[doi:10.1107/S0021889882012060](https://doi.org/10.1107/S0021889882012060). - -**Hexagonal-ice ring positions** — the eleven ring $d$ spacings from 3.895 to 1.522 Å that the -ice-ring score, the ice-ring flagging and the ice calibrant are all built on are taken from the -measurements of Moreau and co-workers, not enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in -biomolecular cryocrystallography" (2021), Acta Cryst. D77, 540-554 -[doi:10.1107/S2059798321001170](https://doi.org/10.1107/S2059798321001170). - -That list ends at 1.522 Å by its own scope, so the eight bands below it are calculated here rather -than taken from anyone: ice Ih structure factors on the oxygen sublattice, kept where they reach 3% of -the strongest line, which reproduces the eleven measured positions exactly. The lattice constants are -Röttger and co-workers'. K. Röttger, A. Endriss, J. Ihringer, S. Doyle and W. F. Kuhs, "Lattice -constants and thermal expansion of H2O and D2O ice Ih between 10 and 265 K" (1994), Acta Cryst. B50, -644-648 [doi:10.1107/S0108768194004933](https://doi.org/10.1107/S0108768194004933). - ### Model-based analysis and maps **Bulk-solvent correction and overall scaling** — the model's structure factors are put on the @@ -643,18 +629,21 @@ and figure of merit. Estimating the likelihood's parameters on free reflections calculated from atomic models" (1995), Acta Cryst. A51, 880-887 [doi:10.1107/S010876739500688X](https://doi.org/10.1107/S010876739500688X); V. Y. Lunin, P. V. Afonine and A. G. Urzhumtsev, "Likelihood-based refinement. I. Irremovable model errors" (2002), Acta Cryst. A58, -270-282 [doi:10.1107/S0108767302001046](https://doi.org/10.1107/S0108767302001046). The gradient with -respect to every atom is taken by the FFT method - the derivatives of the target with respect to the +270-282 [doi:10.1107/S0108767302001046](https://doi.org/10.1107/S0108767302001046). + +**The gradient** with respect to every atom is taken by the FFT method - the derivatives of the target with respect to the structure factors transformed into a map and summed against each atom's density derivatives: R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm" (1978), Acta Cryst. A34, 791-809 [doi:10.1107/S0567739478001618](https://doi.org/10.1107/S0567739478001618). -The positions and B factors are minimised by L-BFGS (rugnux's own, its memory kept from one macro-cycle to + +**The positions and B factors** are minimised by L-BFGS (rugnux's own, its memory kept from one macro-cycle to the next, the B factors' steps scaled by their estimated curvature): D. C. Liu and J. Nocedal, "On the limited memory BFGS method for large scale optimization" (1989), Math. Program. 45, 503-528 [doi:10.1007/BF01589116](https://doi.org/10.1007/BF01589116), whose two-loop recursion is J. Nocedal's, "Updating quasi-Newton matrices with limited storage" (1980), Math. Comp. 35, 773-782 -[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). The geometry restraints are those of the CCP4 -Monomer Library, a subset of which rugnux carries (see THIRD_PARTY_NOTICES.md): A. A. Vagin, +[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). + +**The geometry restraints** are those of the CCP4 Monomer Library, a subset of which rugnux carries (see THIRD_PARTY_NOTICES.md): A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long and G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use" (2004), Acta Cryst. D60, 2184-2195 [doi:10.1107/S0907444904023510](https://doi.org/10.1107/S0907444904023510); applied to the model @@ -664,12 +653,15 @@ library for structural biology" (2022), J. Open Source Softw. 7, 4200 A. Blondel and M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities" (1996), J. Comput. Chem. 17, 1132-1141 [doi:10.1002/(SICI)1096-987X(19960715)17:9<1132::AID-JCC5>3.0.CO;2-T](https://doi.org/10.1002/(SICI)1096-987X(19960715)17:9%3C1132::AID-JCC5%3E3.0.CO;2-T). -The B factors of bonded atoms, and of atoms in contact, are restrained to each other in the + +**The B factors of bonded atoms**, and of atoms in contact, are restrained to each other in the Kullback-Leibler form REFMAC5 uses, whose tolerance grows with B (Murshudov et al. 2011, above; the divergence itself is S. Kullback and R. A. Leibler's, "On information and sufficiency" (1951), Ann. Math. Statist. 22, 79-86 [doi:10.1214/aoms/1177729694](https://doi.org/10.1214/aoms/1177729694)), and the overall isotropic B of the scale is moved into the atoms as phenix.refine moves it (Afonine et al. 2012, -below). The weights of the X-ray term against the geometry and against the B restraints start from the +below). + +**The weights of the X-ray term** against the geometry and against the B restraints start from the ratios of the gradients, as X-PLOR's crystallographic weight was first set: A. T. Brünger, M. Karplus and G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin" (1989), Acta Cryst. A45, 50-61 [doi:10.1107/S0108767388009195](https://doi.org/10.1107/S0108767388009195); and they are chosen @@ -679,7 +671,8 @@ N. W. Moriarty, M. Mustyakimov, T. C. Terwilliger, A. Urzhumtsev, P. H. Zwart an 352-367 [doi:10.1107/S0907444912001308](https://doi.org/10.1107/S0907444912001308). R-free itself: A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures" (1992), Nature 355, 472-475 [doi:10.1038/355472a0](https://doi.org/10.1038/355472a0). -The model the refinement scores is scaled to the data with the overall scale and anisotropic B fitted by + +**The model the refinement scores** is scaled to the data with the overall scale and anisotropic B fitted by robust least squares - iteratively reweighted, with the pseudo-Huber weight of each reflection's residual - so that a few reflections the model cannot explain do not set the scale of the rest: P. J. Huber, "Robust estimation of a location parameter" (1964), Ann. Math. Statist. 35, 73-101 @@ -687,7 +680,8 @@ estimation of a location parameter" (1964), Ann. Math. Statist. 35, 73-101 P. Charbonnier, L. Blanc-Féraud, G. Aubert and M. Barlaud, "Deterministic edge-preserving regularization in computed imaging" (1997), IEEE Trans. Image Process. 6, 298-311 [doi:10.1109/83.551699](https://doi.org/10.1109/83.551699). -The bonds of metal ions to their donor atoms that the model's LINK records name are restrained at the + +**The bonds of metal ions** to their donor atoms that the model's LINK records name are restrained at the distances of the Monomer Library's own statistics of metal sites (its `metals.json`, embedded as `monomers/metals.cif`; per metal, donor element and coordination number), in line with the surveys of M. M. Harding, "Geometry of metal-ligand interactions in proteins" (2001), Acta Cryst. D57, 401-411 @@ -696,7 +690,8 @@ predicted distances around metal sites in proteins" (2006), Acta Cryst. D62, 678 [doi:10.1107/S0907444906014594](https://doi.org/10.1107/S0907444906014594), and, for Mg²⁺, Na⁺ and K⁺ at nucleic acids, F. Leonarski, L. D'Ascenzo and P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?" (2017), Nucleic Acids Res. 45, 987-1004 [doi:10.1093/nar/gkw1175](https://doi.org/10.1093/nar/gkw1175). -The anisotropic displacements: a TLS group's U is that of a rigid body's translation, libration and their + +**The anisotropic displacements**: a TLS group's U is that of a rigid body's translation, libration and their correlation, U = T + A L A^T + A S + S^T A^T, of V. Schomaker and K. N. Trueblood, "On the rigid-body motion of molecules in crystals" (1968), Acta Cryst. B24, 63-76 [doi:10.1107/S0567740868001718](https://doi.org/10.1107/S0567740868001718), refined against the likelihood with a