diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index adf503c4b..8359d48d4 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -598,12 +598,17 @@ Cryst. A34, 791-809 [doi:10.1107/S0567739478001618](https://doi.org/10.1107/S056 memory BFGS method for large scale optimization" (1989), Math. Program. 45, 503-528 [doi:10.1007/BF01589116](https://doi.org/10.1007/BF01589116); its two-loop recursion: J. Nocedal, "Updating quasi-Newton matrices with limited storage" (1980), Math. Comp. 35, 773-782 -[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). The -initial inverse Hessian is rescaled at each iteration by the latest pair, the self-scaling of -S. S. Oren and D. G. Luenberger, "Self-scaling variable metric (SSVM) algorithms. Part I" (1974), -Management Sci. 20, 845-862 [doi:10.1287/mnsc.20.5.845](https://doi.org/10.1287/mnsc.20.5.845), and -D. F. Shanno and K.-H. Phua, "Matrix conditioning and nonlinear optimization" (1978), Math. Program. -14, 149-160 [doi:10.1007/BF01588962](https://doi.org/10.1007/BF01588962). +[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). Its +initial inverse Hessian is the target's sparse normal matrix - the X-ray target's curvature per atom +on the diagonal and the restraints' Gauss-Newton matrix in full - as REFMAC refines on its normal +matrix (Murshudov, Vagin & Dodson 1997, above), solved by conjugate gradients: M. R. Hestenes and +E. Stiefel, "Methods of conjugate gradients for solving linear systems" (1952), J. Res. Natl. Bur. +Stand. 49, 409-436 [doi:10.6028/jres.049.044](https://doi.org/10.6028/jres.049.044). Each atom's +diagonal is raised to hold its step within the step bound, a trust region per atom after the damping +of K. Levenberg, "A method for the solution of certain non-linear problems in least squares" (1944), +Q. Appl. Math. 2, 164-168 [doi:10.1090/qam/10666](https://doi.org/10.1090/qam/10666), and D. W. +Marquardt, "An algorithm for least-squares estimation of nonlinear parameters" (1963), J. Soc. Ind. +Appl. Math. 11, 431-441 [doi:10.1137/0111030](https://doi.org/10.1137/0111030). **Geometry restraints** — those of the CCP4 Monomer Library, a subset of which rugnux carries (see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). A. A. Vagin, R. A. Steiner, A. A. Lebedev, @@ -834,7 +839,8 @@ pseudorandom number generators" (2014), Proc. OOPSLA 2014, 453-472 [doi:10.1145/2660193.2660195](https://doi.org/10.1145/2660193.2660195). **Error-free floating-point arithmetic** — the single-precision exp shared between the CPU and an -Apple GPU (`BraggExp.h`): the argument reduction of W. J. Cody and W. Waite, "Software Manual for the +Apple GPU (`BraggExp.h`) and the double-float sums of the refinement's Gauss-Newton products on the GPU +(`RefinementNewtonGPU.cu`): the argument reduction of W. J. Cody and W. Waite, "Software Manual for the Elementary Functions" (1980), Prentice-Hall; the exact product (with Veltkamp's split) and Fast2Sum of T. J. Dekker, "A floating-point technique for extending the available precision" (1971), Numer. Math. 18, 224-242; and the 2Sum of D. E. Knuth, "The Art of Computer Programming, Vol. 2: diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index bfc7217c1..503030a2f 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -66,12 +66,13 @@ The methods draw on, and in places reimplement, solutions from: - H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", *SIAM J. Numer. Anal.* **49** (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", *J. ACM* **12** (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6). - R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", *Biometrika* **10** (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared). - R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", *Acta Cryst.* **A34** (1978), 791-809 (the gradient of the refinement target, §15). -- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15). +- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target, the B restraints and the minimiser's sparse normal matrix, §15). - V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", *Acta Cryst.* **A51** (1995), 880-887 (the likelihood parameters from the free set, §15). - A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15). - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (the weight chosen among refinements by R-free, §15). - D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15). - J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6). +- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6). - G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5). - P. Roversi, E. Blanc, S. Johnson & S. M. Lea, "Tetartohedral twinning could happen to you too", *Acta Cryst.* **D68** (2012), 418-424 (twin refinement with more than two domains, and the free set closed under the twin domains, §15.12). - K. A. Sultana, I. Alexeev, I. Kursula, P. Mäntsälä, J. Niemi & G. Schneider, "Structure determination by multiwavelength anomalous diffraction of aclacinomycin oxidoreductase: indications of multidomain pseudomerohedral twinning", *Acta Cryst.* **D63** (2007), 149-159 (a six-domain twin of a monoclinic crystal on a hexagonal metric, §15.12). @@ -120,7 +121,6 @@ The methods draw on, and in places reimplement, solutions from: - V. Luzzati, "Traitement statistique des erreurs dans la détermination des structures cristallines", *Acta Cryst.* **5** (1952), 802-810 (the fall-off of $\sigma_A$ with resolution for a model's coordinate error, Luzzati's $D$, in the likelihood of a placement, §14.10). - D. Liebschner, P. V. Afonine, A. G. Urzhumtsev & P. D. Adams, "Implementation of the riding hydrogen model in CCTBX to support the next generation of X-ray and neutron joint refinement in Phenix", *Methods Enzymol.* **634** (2020), 177-199 (the riding hydrogens, placed by GEMMI's code, §15.2). - D. L. Theobald, "Rapid calculation of RMSDs using a quaternion-based characteristic polynomial", *Acta Cryst.* **A61** (2005), 478-480, and P. Liu, D. K. Agrafiotis & D. L. Theobald, *J. Comput. Chem.* **31** (2010), 1561-1563 (the QCP superposition behind the NCS rms criterion, via GEMMI, §15.5). -- S. S. Oren & D. G. Luenberger, "Self-scaling variable metric (SSVM) algorithms. Part I", *Management Sci.* **20** (1974), 845-862, and D. F. Shanno & K.-H. Phua, "Matrix conditioning and nonlinear optimization", *Math. Program.* **14** (1978), 149-160 (the rescaling of the L-BFGS initial inverse Hessian by the latest pair, §15.6). (list is not exhaustive; the full citations, with DOIs, are in [ACKNOWLEDGEMENT.md](ACKNOWLEDGEMENT.md)) diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 0a03eb028..225816d63 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -286,26 +286,51 @@ Whether the restraints are used at all, and how tightly ($\sigma$), the weight t ### 15.6 The minimiser -Positions and B factors are minimised in turn, each by an L-BFGS of rugnux's own (`RefinementMinimiser`), -with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a backtracking line -search from the full step to sufficient decrease, its first trial moving no position by more than 0.3 Å and -no B by more than 10 Ų. What sets it apart from a textbook L-BFGS: +Positions, B factors, anisotropic U and TLS parameters are minimised in turn, each by an L-BFGS of rugnux's own +(`RefinementMinimiser`), with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a +backtracking line search from the full step to sufficient decrease, its first trial moving no position by more +than 0.3 Å and no B by more than 10 Ų. What sets it apart from a textbook L-BFGS is its **initial inverse +Hessian: the target's Gauss-Newton normal matrix**, as REFMAC refines on its sparse normal matrix (Murshudov, +Vagin & Dodson 1997). The matrix holds what is known of the curvature in advance; the L-BFGS pairs add what it +leaves out. -- **Its memory is kept from one macro-cycle to the next.** A macro-cycle is a few dozen iterations on a target - whose scale and likelihood parameters were refitted in between; starting each from what the earlier ones - learnt of the curvature, instead of from a steepest-descent step, is what lets two macro-cycles of a - weight trial get as far as many restarted ones. Each weight trial starts with an empty memory. -- **The B steps are scaled by their curvature.** The initial inverse Hessian of the B factors is a - diagonal: the X-ray target's curvature in each atom's B - its shape from the likelihood's curvature in - $|F_\mathrm{model}|$ and the atom's own scattering, summed in resolution shells, calibrated once per trial - against central differences of the gradient along a random $\pm1$ direction - plus the B restraints' - curvature, each element at least a thousandth of the median. The positions take none: their restraints' - curvature is that of stiff bonds and angles, which move atoms together, and as a diagonal it slows the soft - (torsion and X-ray) directions rather than scaling them. +- **The restraints' part is their Gauss-Newton matrix $J^\mathsf{T}J$, in full.** Every restraint is the square of + a residual (or close to it: the NCS distances' robust loss), so its gradient is $2r\,\nabla r$ and + $J = \nabla E / \sqrt{2E}$ is its row of the Jacobian, $\sqrt2\,\nabla r$; a plane contributes one row per + atom. $J^\mathsf{T}J$ is positive semi-definite where the exact Hessian is not (torsions, repulsions), and near + the ideal geometry it is the exact Hessian. Its products $J^\mathsf{T}(Jv)$ are two gathers - over the rows, + then over each atom's entries - on the slot's threads, or on its GPU. +- **The X-ray part is per atom, on the diagonal.** The X-ray target's curvature in an atom's $x$, $y$ and $z$ is + the likelihood's curvature in $|F_\mathrm{model}|$ summed in resolution shells against the atom's own part of + it, $(2\pi)^2 s^2/3\,f(s)^2\,e^{-Bs^2/2}$ (the reflections taken as spread evenly over directions); in its B, + $(s^2/4)^2 f(s)^2 e^{-Bs^2/2}$, a third of it to each element of a U; and a TLS body's dense $20\times20$ block + is that curvature carried through the derivatives of its atoms' U in the body's parameters. Each is calibrated + once per trial against central differences of the gradient along a random $\pm1$ direction. The coupling of + neighbouring atoms through the X-ray target is not in it; the pairs learn it. +- **The anisotropic U restraints** are quadratic in U, so their Hessian is exact by one difference of their + gradient. +- **A trust floor per atom.** Where nothing but the X-ray target curves a mode - a high-B side chain turning about + a single bond, which the restraints leave free and the X-ray target hardly sees - the matrix's inverse would + make a step of tens of ångströms there, and the step bound would then shrink every other atom's step with it. + So each atom's diagonal is raised, once per macro-cycle, to at least $|g_i|/\Delta$ ($g_i$ its gradient, + $\Delta$ the step bound): its step is held by its own curvature, a trust region per atom in the manner of + Levenberg and Marquardt's damping. A U (per atom) and a TLS body's parameters (each) take the same floor. Without it a rounding difference of $10^{-7}$ in the gradient moved a weight + trial's R-free by up to 0.004; with it the perturbation stays where it was. +- **The matrix is solved, not inverted.** Each L-BFGS direction solves the normal equations by a fixed 20 + conjugate-gradient iterations - not to a tolerance, whose iteration count, and with it the direction, a rounding + difference can flip - preconditioned per residue for the positions (the residue's own $J^\mathsf{T}J$ and X-ray + diagonal, by Cholesky: the stiff bonds and angles inside a residue are solved there, what couples residues is + left to the iterations) and by the diagonal elsewhere. The Jacobian is taken again at each point the minimiser + stands on. +- **Its memory is kept from one macro-cycle to the next.** Each weight trial starts with an empty memory. -The diagonal is rescaled at each iteration by the latest pair (Oren-Shanno scaling in that metric). A solve -stops after its iteration budget (30 for the positions and 20 for the B factors per macro-cycle) or where an -iteration lowers the target by less than $10^{-7}$ of it. +A solve stops after its iteration budget (10 for the positions and 7 for the B factors, U or TLS per +macro-cycle) or where an iteration lowers the target by less than $10^{-7}$ of it. Against the plain L-BFGS this +replaced (a diagonal initial inverse Hessian for the B factors, none for the positions, 30 and 20 iterations), a +trial takes about a third of the evaluations. With it the population (§15.7) no longer gains, and the protocol +is the same on the CPU and the GPU - two rounds, no population - at a third of the evaluations the GPU's protocol +took before and R-free 0.0014 lower in the mean over 19 refinements, and at fewer evaluations than the CPU's +former one-round protocol. ### 15.7 The weights, decided by trial @@ -337,8 +362,10 @@ are: stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model, whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the trajectory ends where it began. -3. **The population**, from where the trajectory ended: twelve members (at most as many as the trial slots, so - that they run in one wave; none below two) refined side by side, each a trial at the +3. **The population** (off by default, `RefinementSettings::population`: with the minimiser of §15.6 its members + took nothing off R-free over 19 refinements that the second round did not), from where the trajectory ended: + up to twelve members (at most as many as the trial slots, so that they run in one wave; none below two) refined + side by side, each a trial at the trajectory's weights but on its own path - the positions as they are or kicked at random by up to 0.2-0.3 Å in each coordinate, on the data as they are or **blurred** along a schedule, 60 then 20 Ų (the landscape of 4 Å, then of 2.2 Å), 100 then 30, or 30 then 10. The blur raises the likelihood's variance of each reflection @@ -741,7 +768,7 @@ What the refinement does not do yet - each a known gap, not a design choice: - L. Piela, J. Kostrowicki & H. A. Scheraga, "On the multiple-minima problem in the conformational analysis of molecules: deformation of the potential energy hypersurface by the diffusion equation method", *J. Phys. Chem.* **93** (1989), 3339-3346, and A. Jack & M. Levitt, "Refinement of large structures by simultaneous minimization of energy and R factor", *Acta Cryst.* **A34** (1978), 931-935 (the members refined on blurred data, §15.7). - J. Praznikar, P. V. Afonine, G. Guncar, P. D. Adams & D. Turk, "Averaged kick maps: less noise, more signal... and probably less bias", *Acta Cryst.* **D65** (2009), 921-931, and J. C. Nwachukwu, M. R. Southern, J. R. Kiefer, P. V. Afonine, P. D. Adams, T. C. Terwilliger & K. W. Nettles, "Improved crystallographic structures using extensive combinatorial refinement", *Structure* **21** (2013), 1923-1930 (the kicked members, their averaged map and the crossover residue by residue, §15.7). - J. Desmet, M. De Maeyer, B. Hazes & I. Lasters, "The dead-end elimination theorem and its use in protein side-chain positioning", *Nature* **356** (1992), 539-542; J. Besag, "On the statistical analysis of dirty pictures", *J. R. Statist. Soc.* **B48** (1986), 259-279; R. P. Joosten, K. Joosten, S. X. Cohen, G. Vriend & A. Perrakis, "Automatic rebuilding and optimization of crystallographic structures in the Protein Data Bank", *Bioinformatics* **27** (2011), 3392-3398 (the side-chain field: states, the conditional-modes solver and its order, §15.5). -- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8). +- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8; the minimiser's sparse normal matrix, §15.6). - G. N. Murshudov, P. Skubák, A. A. Lebedev, N. S. Pannu, R. A. Steiner, R. A. Nicholls, M. D. Winn, F. Long & A. A. Vagin, "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (D from the working set and the variance from the free, §15.3; the Kullback-Leibler form of the B restraints, §15.5; the twinned refinement §15.12 simplifies, its twin laws pruned by fraction and its fractions refined by least squares). - R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", *Acta Cryst.* **A42** (1986), 140-149 (the m, D weighting of the map coefficients, §15.8). - T. O. Yeates, "Detecting and overcoming crystal twinning", *Methods Enzymol.* **276** (1997), 344-358 (the twinned intensity, §15.12). @@ -762,6 +789,8 @@ What the refinement does not do yet - each a known gap, not a design choice: - Z. Dauter & M. Dauter, "Entering a new phase: using solvent halide ions in protein structure determination", *Structure* **9** (2001), R21-R26 (halides at sites with few donors, §15.10). - N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", *Acta Cryst.* **D70** (2014), 1104-1114 (a light ion flagged by its density above the waters', §15.10). - J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782, and D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15.6). +- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6). +- K. Levenberg, "A method for the solution of certain non-linear problems in least squares", *Q. Appl. Math.* **2** (1944), 164-168, and D. W. Marquardt, "An algorithm for least-squares estimation of nonlinear parameters", *J. Soc. Ind. Appl. Math.* **11** (1963), 431-441 (the minimiser's trust floor per atom, §15.6). - A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7). - A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free). diff --git a/image_analysis/structure_refinement/CMakeLists.txt b/image_analysis/structure_refinement/CMakeLists.txt index 873099f93..c51422b37 100644 --- a/image_analysis/structure_refinement/CMakeLists.txt +++ b/image_analysis/structure_refinement/CMakeLists.txt @@ -37,6 +37,9 @@ ADD_LIBRARY(JFJochStructureRefinement STATIC $<$:RefinementGPU.cpp> $<$:RefinementGPU.cu> RefinementGPU.h RefinementGPUEngine.h + $<$:RefinementNewtonGPU.cpp> + $<$:RefinementNewtonGPU.cu> + RefinementNewtonGPU.h ${CMAKE_CURRENT_BINARY_DIR}/MonomerLibraryData.cpp) # The subset of the CCP4 Monomer Library that refinement restrains with (monomers/, chosen by diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index f921b9300..ae6ecbac0 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -47,6 +48,7 @@ #ifdef JFJOCH_USE_CUDA #include "../../common/CUDAWrapper.h" #include "RefinementGPU.h" +#include "RefinementNewtonGPU.h" #endif namespace { @@ -607,7 +609,12 @@ struct Context { // The X-ray target's curvature in each atom's B, up to one factor (CalibrateBCurvature): the likelihood's // curvature in |F_model| of the cycle, 2 (k_total D)^2 / var, summed in resolution shells, against the atom's own // part of it, (s^2 / 4)^2 f(s)^2 exp(-B s^2 / 2). A hydrogen's is its parent's. - std::vector XrayBCurvatureShape() const { + std::vector XrayBCurvatureShape() const { return XrayCurvatureShape(rows, false); } + // The same for each of an atom's x, y and z: its own part of the likelihood's curvature in |F_model| is then + // (2 pi)^2 s^2 / 3 f(s)^2 exp(-B s^2 / 2), the reflections taken as spread evenly over directions - the X-ray + // block of the normal matrix as a multiple of I per atom. `r`: the row model the engine has (blurred or not). + std::vector XrayXyzCurvatureShape(const XrayRowModel &r) const { return XrayCurvatureShape(r, true); } + std::vector XrayCurvatureShape(const XrayRowModel &rows, bool xyz) const { constexpr int SHELLS = 32; double s2_max = 0; for (double s2 : refl.inv_d2) @@ -638,7 +645,10 @@ struct Context { for (int k = 0; k < SHELLS; k++) if (n[k] > 0) { const double s2 = s2_sum[k] / n[k]; - shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * s2 / 16.0; + if (xyz) + shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * (4.0 * M_PI * M_PI / 3.0); + else + shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * s2 / 16.0; } } for (size_t i = 0; i < rider_parent.size(); i++) @@ -675,6 +685,37 @@ struct Context { return den > 0 ? std::max(num / den, 0.0) : 0.0; } + // The factor that turns XrayXyzCurvatureShape() into the X-ray target's curvature, as CalibrateBCurvature() does + // for B: along a random +-1 direction of the positions of `params`, by central differences of the gradient. + // Two evaluations, counted. + double CalibrateXyzCurvature(const std::vector ¶ms, const std::vector &shape) { + constexpr double EPS = 0.01; // A + const std::vector p0 = Positions(); + std::mt19937 rng(7); + std::vector v(params.size()); + for (gemmi::Vec3 &x : v) + x = gemmi::Vec3(rng() % 2 == 0 ? 1.0 : -1.0, rng() % 2 == 0 ? 1.0 : -1.0, rng() % 2 == 0 ? 1.0 : -1.0); + std::array>, 2> g; + for (int side = 0; side < 2; side++) { + for (size_t k = 0; k < params.size(); k++) + atoms[params[k]]->pos = p0[params[k]] + gemmi::Position(v[k] * (side == 0 ? EPS : -EPS)); + MoveRiders(); + engine->Evaluate(Model(), &g[side]); + evaluations++; + FoldRiders(g[side]); + } + SetPositions(p0); + MoveRiders(); + double num = 0, den = 0; + for (size_t k = 0; k < params.size(); k++) { + const int i = params[k]; + num += (v[k].x * (g[0][i][0] - g[1][i][0]) + v[k].y * (g[0][i][1] - g[1][i][1]) + + v[k].z * (g[0][i][2] - g[1][i][2])) / (2 * EPS); + den += 3 * shape[i]; + } + return den > 0 ? std::max(num / den, 0.0) : 0.0; + } + // F_model with the cycle's scale and solvent, for the atoms as they are now. std::vector> FModel() { engine->Fcalc(Model(), fcalc); @@ -792,6 +833,10 @@ public: return std::isfinite(*cost); } + Mode GetMode() const { return mode_; } + double Weight() const { return weight_; } + const std::vector &Atoms() const { return param_atoms_; } + private: const SpecialPosition *Special(int atom) const { const int k = topo_.special_index.empty() ? -1 : topo_.special_index[atom]; @@ -874,6 +919,20 @@ public: } } double MaxStep() const { return MAX_STEP_B; } + // The parameter k of TLS body `body` over its value in the body's own units. + double ParamScale(size_t body, int k) const { return Scale(body, k); } + // Mode::U: the U restraints' gradient alone at p, as Evaluate() makes it (zero where there are none). + void RestraintGradient(const double *p, double *gradient) const { + Set(p); + std::fill(gradient, gradient + 6 * param_atoms_.size(), 0.0); + if (restraints_ == nullptr) + return; + std::vector> gr(ctx_.atoms.size(), {0, 0, 0, 0, 0, 0}); + EvaluateURestraints(*restraints_, ctx_.Positions(), ctx_.UOfAtoms(), &gr); + for (size_t k = 0; k < param_atoms_.size(); k++) + for (int c = 0; c < 6; c++) + gradient[6 * k + c] = gr[param_atoms_[k]][c] / gemmi::u_to_b(); + } bool Evaluate(const double *p, double *cost, double *gradient) const { const RefinementStage stage(mode_ == Mode::U ? "eval.u" : "eval.tls"); @@ -934,40 +993,380 @@ private: std::vector radius_; }; -// The minimiser's diagonal for the B factors of `param_atoms` (RefinementMinimiser::SetDiagonal): weight times the -// X-ray target's curvature in B plus the B restraints', each element at least a thousandth of the median. The -// positions take none: their restraints' curvature is that of stiff bonds and angles, which move atoms together, so -// as a diagonal it slows rather than scales the minimiser. -std::vector BDiagonal(const std::vector &xray, const std::vector &restraints, - const std::vector ¶m_atoms, double weight) { - std::vector h; - for (int i : param_atoms) - h.push_back(weight * xray[i] + restraints[i]); - if (h.empty()) // no atom the dictionary describes - return h; - std::vector sorted = h; +// Each element at least a thousandth of the median (a parameter the target does not see still takes a finite step). +std::vector FlooredDiagonal(std::vector d) { + if (d.empty()) + return d; + std::vector sorted = d; std::nth_element(sorted.begin(), sorted.begin() + sorted.size() / 2, sorted.end()); const double floor = std::max(1e-3 * sorted[sorted.size() / 2], 1e-12); - for (double &v : h) + for (double &v : d) v = std::max(v, floor); - return h; + return d; } -// `minimiser` carries its memory from the call before; `diagonal` is the curvature it scales the steps by (none: 1). +// The minimisers' initial inverse Hessian (RefinementMinimiser::SetInverseHessian) is the Gauss-Newton normal matrix of +// the target, as REFMAC refines on its sparse normal matrix: the X-ray target's curvature per atom on the diagonal and +// the restraints' J^T J in full, solved for each direction by a fixed number of conjugate-gradient iterations - not to a +// tolerance, whose iteration count, and with it the direction, a rounding difference can flip. +// Following Murshudov, Vagin & Dodson (1997) Acta Cryst. D53, 240-255 +constexpr int NORMAL_CG_ITERATIONS = 20; + +// `target` by `minimiser` (its memory from the call before) from where it is, H0 the inverse of the normal matrix: at(x, q) +// takes the matrix at x (where the minimiser stands; q the vector H0 is about to be applied to), then hv = H v +// (product) and z = M^-1 r for a preconditioner M of it (precondition). template -void Minimise(Context &ctx, const Target &target, int iterations, RefinementMinimiser &minimiser, - std::vector diagonal = {}) { +void MinimiseWithNormalMatrix(Context &ctx, const Target &target, RefinementMinimiser &minimiser, int iterations, + const std::function &, const std::vector &)> &at, + const std::function &, std::vector &)> &product, + const std::function &, std::vector &)> &precondition) { std::vector p = target.Get(); if (p.empty()) return; - minimiser.SetDiagonal(diagonal.empty() ? std::vector(p.size(), 1.0) : std::move(diagonal)); + minimiser.SetInverseHessian([&](const std::vector &x, std::vector &q) { + at(x, q); + std::vector z; + ConjugateGradients(product, precondition, q, z, NORMAL_CG_ITERATIONS, 0.0, ctx.nthreads); + q.swap(z); + }); minimiser.Minimise([&](const std::vector &x, double &f, std::vector &g) { return target.Evaluate(x.data(), &f, g.data()); }, p, iterations, MINIMISER_F_TOLERANCE, target.MaxStep(), [&]() { return ctx.Cancelled(); }); + minimiser.SetInverseHessian(nullptr); target.Set(p.data()); // where the minimiser ended, not where its last line search probed } +// The restraints' Gauss-Newton normal matrix J^T J (RestraintGaussNewton) on the x y z or the B of `param_atoms`, at the +// point last given to At(); an atom that is not a parameter (a rider, an atom held) drops out of it. With the engine on +// a GPU, on its card (RestraintNormalMatrixGPU). +class RestraintNormalMatrix { +public: + RestraintNormalMatrix(const Context &ctx, const std::vector ¶m_atoms, bool xyz) + : per_(xyz ? 3 : 1), c0_(xyz ? 0 : 3), n_params_(param_atoms.size()), nthreads_(ctx.nthreads), + param_of_(ctx.atoms.size(), -1) { + for (size_t k = 0; k < param_atoms.size(); k++) + param_of_[param_atoms[k]] = static_cast(k); +#ifdef JFJOCH_USE_CUDA + if (ctx.engine->OnGPU()) + gpu_ = std::make_unique(ctx.device, param_atoms, ctx.atoms.size(), xyz); +#endif + } + + void At(const RestraintSet &set, const std::vector> &xyzb) { + const RefinementStage stage("normal.jacobian"); +#ifdef JFJOCH_USE_CUDA + if (gpu_) + return gpu_->At(set, xyzb); +#endif + jac_ = RestraintGaussNewton(set, xyzb); + const size_t rows = jac_.start.size() - 1; + row_of_.resize(jac_.atom.size()); + param_.resize(jac_.atom.size()); + for (size_t e = 0; e < jac_.atom.size(); e++) + param_[e] = param_of_[jac_.atom[e]]; + for (size_t r = 0; r < rows; r++) + for (int e = jac_.start[r]; e < jac_.start[r + 1]; e++) + row_of_[e] = static_cast(r); + // per parameter atom its entries (CSR), for the transpose as a gather + entry_start_.assign(n_params_ + 1, 0); + for (int a : jac_.atom) + if (param_of_[a] >= 0) + entry_start_[param_of_[a] + 1]++; + for (size_t k = 0; k < n_params_; k++) + entry_start_[k + 1] += entry_start_[k]; + entry_.resize(entry_start_.back()); + std::vector fill(entry_start_.begin(), entry_start_.end() - 1); + for (size_t e = 0; e < jac_.atom.size(); e++) + if (const int k = param_of_[jac_.atom[e]]; k >= 0) + entry_[fill[k]++] = static_cast(e); + t_.resize(rows); + } + + // hv = J^T J v: J v per row, then per parameter atom the sum over its entries + void Multiply(const std::vector &v, std::vector &hv) { + const RefinementStage stage("normal.product"); +#ifdef JFJOCH_USE_CUDA + if (gpu_) + return gpu_->Multiply(v, hv); +#endif + const int rows = static_cast(t_.size()); + ParallelChunks(rows, ThreadsForWork(rows, nthreads_, 16384), [&](int lo, int hi) { + for (int r = lo; r < hi; r++) { + double dot = 0; + for (int e = jac_.start[r]; e < jac_.start[r + 1]; e++) + if (const int k = param_[e]; k >= 0) + for (int c = 0; c < per_; c++) + dot += jac_.j[e][c0_ + c] * v[per_ * k + c]; + t_[r] = dot; + } + }); + hv.assign(per_ * n_params_, 0.0); + const int n = static_cast(n_params_); + ParallelChunks(n, ThreadsForWork(n, nthreads_, 16384), [&](int lo, int hi) { + for (int k = lo; k < hi; k++) + for (int q = entry_start_[k]; q < entry_start_[k + 1]; q++) { + const int e = entry_[q]; + for (int c = 0; c < per_; c++) + hv[per_ * k + c] += t_[row_of_[e]] * jac_.j[e][c0_ + c]; + } + }); + } + + std::vector Diagonal() const { +#ifdef JFJOCH_USE_CUDA + if (gpu_) + return gpu_->Diagonal(); +#endif + std::vector d(per_ * n_params_, 0.0); + for (size_t k = 0; k < n_params_; k++) + for (int q = entry_start_[k]; q < entry_start_[k + 1]; q++) + for (int c = 0; c < per_; c++) + d[per_ * k + c] += jac_.j[entry_[q]][c0_ + c] * jac_.j[entry_[q]][c0_ + c]; + return d; + } + +private: + int per_, c0_; + size_t n_params_, nthreads_; + std::vector param_of_; + RestraintJacobian jac_; + std::vector row_of_, param_, entry_start_, entry_; // per entry its row and its parameter atom (-1: none) + std::vector t_; +#ifdef JFJOCH_USE_CUDA + std::unique_ptr gpu_; +#endif +}; + +// The positions' preconditioner: per residue the dense block of H over its atoms' x y z - the restraints within the +// residue (J^T J of the rows whose atoms it holds, at the point given) and the X-ray diagonal `h` - by its Cholesky +// factor. What couples residues (peptide bonds, NCS, contacts) is left to the conjugate gradients; the stiff bonds +// and angles within a residue, which a diagonal or a block per atom leaves to them too, are solved here. +class ResidueBlocks { +public: + ResidueBlocks(Context &ctx, const std::vector ¶m_atoms, const RestraintSet &set, const std::vector &h) { + std::vector residue_of; // per atom, in model order + for (const gemmi::Chain &ch : ctx.Model().chains) + for (const gemmi::Residue &r : ch.residues) { + residue_of.insert(residue_of.end(), r.atoms.size(), static_cast(params_.size())); + params_.emplace_back(); + } + std::vector local(param_atoms.size()), group(param_atoms.size()), param_of(ctx.atoms.size(), -1); + for (size_t k = 0; k < param_atoms.size(); k++) { + group[k] = residue_of[param_atoms[k]]; + local[k] = static_cast(params_[group[k]].size()); + params_[group[k]].push_back(static_cast(k)); + param_of[param_atoms[k]] = static_cast(k); + } + std::vector block(params_.size()); + for (size_t g = 0; g < params_.size(); g++) { + const int m = 3 * static_cast(params_[g].size()); + block[g] = Eigen::MatrixXd::Zero(m, m); + for (size_t l = 0; l < params_[g].size(); l++) + for (int c = 0; c < 3; c++) + block[g](3 * l + c, 3 * l + c) = h[3 * params_[g][l] + c]; + } + const RestraintJacobian jac = RestraintGaussNewton(set, ctx.XyzB()); + for (size_t row = 0; row + 1 < jac.start.size(); row++) + for (int e1 = jac.start[row]; e1 < jac.start[row + 1]; e1++) + for (int e2 = jac.start[row]; e2 < jac.start[row + 1]; e2++) { + const int k1 = param_of[jac.atom[e1]], k2 = param_of[jac.atom[e2]]; + if (k1 < 0 || k2 < 0 || group[k1] != group[k2]) + continue; + Eigen::MatrixXd &b = block[group[k1]]; + for (int a = 0; a < 3; a++) + for (int c = 0; c < 3; c++) + b(3 * local[k1] + a, 3 * local[k2] + c) += jac.j[e1][a] * jac.j[e2][c]; + } + llt_.resize(params_.size()); + ParallelFor(static_cast(params_.size()), ctx.nthreads, [&](int g) { llt_[g].compute(block[g]); }); + } + + void Apply(const std::vector &r, std::vector &z, size_t nthreads) const { + z.resize(r.size()); + ParallelFor(static_cast(params_.size()), nthreads, [&](int g) { + const std::vector &p = params_[g]; + Eigen::VectorXd rg(3 * p.size()); + for (size_t l = 0; l < p.size(); l++) + for (int c = 0; c < 3; c++) + rg(3 * l + c) = r[3 * p[l] + c]; + const Eigen::VectorXd zg = llt_[g].solve(rg); + for (size_t l = 0; l < p.size(); l++) + for (int c = 0; c < 3; c++) + z[3 * p[l] + c] = zg(3 * l + c); + }); + } + +private: + std::vector> params_; // per residue its parameter atoms (their index among the parameters) + std::vector> llt_; +}; + +// The positions or the B factors: H = diag(xray) (per atom, already times the weight) + the restraints' J^T J of `set`, +// the Jacobian at the point of each direction. The diagonal is raised per atom, once per call, to at least |g_i| over the +// step bound (g_i the atom's part of the first direction's gradient): where nothing else curves the target (a high-B +// side chain turning about a single bond, which the restraints leave free and the X-ray target hardly sees) the step +// is then held to the bound by the atom's own curvature, instead of one such atom scaling down the whole step. +// Preconditioned per residue for the positions (ResidueBlocks), by the diagonal for the B factors. +// The floor a trust region per atom, after Levenberg (1944) Q. Appl. Math. 2, 164-168 and Marquardt (1963) J. SIAM +// 11, 431-441 +void MinimiseAtoms(Context &ctx, const AtomTarget &target, RefinementMinimiser &minimiser, const std::vector &xray, + const RestraintSet &set, int iterations) { + const bool xyz = target.GetMode() == AtomTarget::Mode::Xyz; + const int per = xyz ? 3 : 1; + std::vector x_at = target.Get(); + const size_t n = x_at.size(); + std::vector h(n); + for (size_t k = 0; k < n; k++) + h[k] = xray[target.Atoms()[k / per]]; + RestraintNormalMatrix normal(ctx, target.Atoms(), xyz); + target.Set(x_at.data()); + normal.At(set, ctx.XyzB()); + const std::vector restraint_diagonal = normal.Diagonal(); + std::vector diagonal; + std::unique_ptr residues; + auto at = [&](const std::vector &x, const std::vector &q) { + if (diagonal.empty()) { // the first direction: the floor, then the preconditioner with it + for (size_t a = 0; a < n / per; a++) { + double g2 = 0; + for (int c = 0; c < per; c++) + g2 += q[per * a + c] * q[per * a + c]; + for (int c = 0; c < per; c++) + h[per * a + c] = std::max(h[per * a + c], std::sqrt(g2) / target.MaxStep()); + } + diagonal.resize(n); + for (size_t k = 0; k < n; k++) + diagonal[k] = restraint_diagonal[k] + h[k]; + diagonal = FlooredDiagonal(diagonal); + if (xyz) + residues = std::make_unique(ctx, target.Atoms(), set, h); + } + if (x != x_at) { + x_at = x; + target.Set(x.data()); + normal.At(set, ctx.XyzB()); + } + }; + auto product = [&](const std::vector &v, std::vector &hv) { + normal.Multiply(v, hv); + refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] += h[k] * v[k]; }); + }; + auto precondition = [&](const std::vector &r, std::vector &z) { + if (residues) + residues->Apply(r, z, ctx.nthreads); + else { + z.resize(r.size()); + refine_vec::Each(n, ctx.nthreads, [&](size_t k) { z[k] = r[k] / diagonal[k]; }); + } + }; + MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition); +} + +// Each atom's U (in 8 pi^2 U): H = diag(xray) per element + the U restraints' Hessian, exact by differences of their +// gradient (they are quadratic in U); the diagonal raised to the trust floor as MinimiseAtoms() raises it; +// preconditioned by `diagonal`. +void MinimiseU(Context &ctx, const AdpTarget &target, RefinementMinimiser &minimiser, std::vector xray, + std::vector diagonal, int iterations) { + const size_t n = xray.size(); + std::vector x_at, g_at(n), xp(n), gp(n); + bool first = true; + auto at = [&](const std::vector &x, const std::vector &q) { + if (first) { // the trust floor per atom, as MinimiseAtoms() takes it + first = false; + for (size_t a = 0; a < n / 6; a++) { + double g2 = 0; + for (int c = 0; c < 6; c++) + g2 += q[6 * a + c] * q[6 * a + c]; + const double floor = std::sqrt(g2) / target.MaxStep(); + for (int c = 0; c < 6; c++) { + diagonal[6 * a + c] += std::max(floor - xray[6 * a + c], 0.0); + xray[6 * a + c] = std::max(xray[6 * a + c], floor); + } + } + } + if (x != x_at) { + x_at = x; + target.RestraintGradient(x.data(), g_at.data()); + } + }; + auto product = [&](const std::vector &v, std::vector &hv) { + double v_max = 0; + for (double e : v) + v_max = std::max(v_max, std::fabs(e)); + hv.assign(n, 0.0); + if (v_max > 0) { + const double t = 1.0 / v_max; + refine_vec::Each(n, ctx.nthreads, [&](size_t k) { xp[k] = x_at[k] + t * v[k]; }); + target.RestraintGradient(xp.data(), gp.data()); + refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] = (gp[k] - g_at[k]) / t; }); + } + refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] += xray[k] * v[k]; }); + }; + auto precondition = [&](const std::vector &r, std::vector &z) { + z.resize(n); + refine_vec::Each(n, ctx.nthreads, [&](size_t k) { z[k] = r[k] / diagonal[k]; }); + }; + MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition); +} + +// The TLS bodies: X-ray only, H = sum over each body's atoms of A^T C A, A the derivatives of the atom's U in the body's +// parameters (linear in them), C the X-ray curvature of its U per element (`xray_b`: of its B, a third of it to each); +// one dense block per body, its diagonal raised to the trust floor, preconditioned by its diagonal. +void MinimiseTls(Context &ctx, const AdpTarget &target, RefinementMinimiser &minimiser, const std::vector &xray_b, + int iterations) { + constexpr int P = TlsBody::PARAMETERS; + const size_t nb = ctx.tls.size(); + std::vector> blocks(nb); + for (size_t b = 0; b < nb; b++) { + std::array &m = blocks[b]; + m.fill(0.0); + for (int i : ctx.tls[b].atoms) { + if (!ctx.rider_parent.empty() && ctx.rider_parent[i] >= 0) + continue; // its curvature is its parent's already + const std::array, P> a = ctx.tls[b].Derivatives(ctx.atoms[i]->pos); + const double c = gemmi::u_to_b() * gemmi::u_to_b() * xray_b[i] / 3.0; + for (int k = 0; k < P; k++) + for (int l = 0; l < P; l++) { + double s = 0; + for (int e = 0; e < 6; e++) + s += a[k][e] * a[l][e]; + m[k * P + l] += c * s / (target.ParamScale(b, k) * target.ParamScale(b, l)); + } + } + } + std::vector diagonal(nb * P); + for (size_t b = 0; b < nb; b++) + for (int k = 0; k < P; k++) + diagonal[b * P + k] = blocks[b][k * P + k]; + diagonal = FlooredDiagonal(diagonal); + // the trust floor per parameter, as MinimiseAtoms() takes it per atom + bool first = true; + auto at = [&](const std::vector &, const std::vector &q) { + if (!first) + return; + first = false; + for (size_t b = 0; b < nb; b++) + for (int k = 0; k < P; k++) { + const double floor = std::fabs(q[b * P + k]) / target.MaxStep(); + diagonal[b * P + k] += std::max(floor - blocks[b][k * P + k], 0.0); + blocks[b][k * P + k] = std::max(blocks[b][k * P + k], floor); + } + }; + auto product = [&](const std::vector &v, std::vector &hv) { + hv.assign(v.size(), 0.0); + for (size_t b = 0; b < nb; b++) + for (int k = 0; k < P; k++) + for (int l = 0; l < P; l++) + hv[b * P + k] += blocks[b][k * P + l] * v[b * P + l]; + }; + auto precondition = [&](const std::vector &r, std::vector &z) { + z.resize(r.size()); + for (size_t k = 0; k < r.size(); k++) + z[k] = r[k] / diagonal[k]; + }; + MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition); +} + // Groups of atoms that move as rigid bodies (one per chain) against the X-ray target: per body an angle-axis // rotation about its centroid, scaled by its rms radius so that a unit moves a typical atom by an angstrom as a // unit of translation does, then a translation. @@ -1788,6 +2187,18 @@ double XrayGradientNorm(Context &ctx, const std::vector &refinable) { return n; } +// The X-ray curvature per atom in each of x, y, z at weight w, for the row model `rows` the engine has; `scale` +// calibrated at the first call of the trial (negative: not yet). +std::vector XyzCurvature(Context &ctx, const std::vector &refinable, const XrayRowModel &rows, double w, + double &scale) { + std::vector shape = ctx.XrayXyzCurvatureShape(rows); + if (scale < 0) + scale = ctx.CalibrateXyzCurvature(refinable, shape); + for (double &x : shape) + x *= w * scale; + return shape; +} + TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const std::vector &pos0, const std::vector &b0, const std::vector &aniso0, const Context::Fit *fit, const TrialWeights &tw, int cycles, const RefinementSettings &settings, const NcsCopies *ncs, @@ -1838,7 +2249,8 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s const AdpTarget tls_target(ctx, AdpTarget::Mode::Tls, refinable, 1.0, nullptr); const AdpTarget u_target(ctx, AdpTarget::Mode::U, u_atoms, adp_w, &u_restraints); RefinementMinimiser xyz_minimiser(20, ctx.nthreads), b_minimiser(20, ctx.nthreads), tls_minimiser(20, ctx.nthreads); - double xray_scale = 0; + // the X-ray curvature's calibrations (CalibrateXyzCurvature, CalibrateBCurvature), once per trial + double xyz_scale = -1, b_scale = -1; for (int cycle = 0; cycle < cycles && !ctx.Cancelled(); cycle++) { PlaceRiders(ctx, topo); if (cycle == 0 && fit != nullptr) @@ -1852,8 +2264,8 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s if (adp_w > 0) ctx.MoveOverallB(); if (!hold_positions) { + const RestraintSet set = RestraintsNow(ctx, topo, ncs, ncs_sigma); { - const RestraintSet set = RestraintsNow(ctx, topo, ncs, ncs_sigma); const RefinementStage s1("set_restraints"); ctx.engine->SetRestraints(0, set); } @@ -1870,24 +2282,33 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s ctx.engine->SetRowModel(blurred); const double nh = XrayGradientNorm(ctx, refinable); const AtomTarget xyz_h(ctx, AtomTarget::Mode::Xyz, refinable, nh > 0 ? w * std::sqrt(n0 / nh) : w, 0, topo); - Minimise(ctx, xyz_h, settings.xyz_iterations, xyz_minimiser); + MinimiseAtoms(ctx, xyz_h, xyz_minimiser, XyzCurvature(ctx, refinable, blurred, xyz_h.Weight(), xyz_scale), + set, settings.xyz_iterations); ctx.engine->SetRowModel(ctx.rows); } else - Minimise(ctx, xyz, settings.xyz_iterations, xyz_minimiser); + MinimiseAtoms(ctx, xyz, xyz_minimiser, XyzCurvature(ctx, refinable, ctx.rows, w, xyz_scale), set, + settings.xyz_iterations); } + // the X-ray curvature in B for the atoms as they are, which TLS, U and B take a share of + auto b_curvature = [&]() { + std::vector c = ctx.XrayBCurvatureShape(); + if (b_scale < 0) + b_scale = ctx.CalibrateBCurvature(refinable, c); + for (double &x : c) + x *= b_scale; + return c; + }; if (kind == AdpHypothesis::Kind::Tls) { ctx.ApplyTls(); // the bodies' U at the atoms as they are const RefinementStage s4("minimise.tls"); - Minimise(ctx, tls_target, settings.b_iterations, tls_minimiser); + MinimiseTls(ctx, tls_target, tls_minimiser, b_curvature(), settings.b_iterations); } if (adp_w > 0 && kind == AdpHypothesis::Kind::Anisotropic) { - // each U's six elements take a third of the curvature of its B - std::vector xray = ctx.XrayBCurvatureShape(); - if (cycle == 0) - xray_scale = ctx.CalibrateBCurvature(refinable, xray); - // and the restraints' curvature in 8 pi^2 U: per pair 2 / sigma^2 of the similarity and about a third of - // the rigid bond's, per isotropic atom 2 / sigma^2 - std::vector restraint(xray.size(), 0.0); + const std::vector xray_b = b_curvature(); + // each U's six elements take a third of the curvature of its B; and the restraints' curvature in 8 pi^2 U: + // per pair 2 / sigma^2 of the similarity and about a third of the rigid bond's, per isotropic atom + // 2 / sigma^2 - the preconditioner's diagonal + std::vector restraint(xray_b.size(), 0.0); const double u2 = gemmi::u_to_b() * gemmi::u_to_b(); const double per_pair = (2.0 / (URestraints::SIGMA_SIMILAR * URestraints::SIGMA_SIMILAR) + 2.0 / (3.0 * URestraints::SIGMA_RIGID * URestraints::SIGMA_RIGID)) / u2; @@ -1897,24 +2318,21 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s } for (int i : u_restraints.isotropic) restraint[i] += 2.0 / (URestraints::SIGMA_ISOTROPIC * URestraints::SIGMA_ISOTROPIC) / u2; - for (double &x : xray) - x *= xray_scale / 3.0; - std::vector diagonal; - for (double h : BDiagonal(xray, restraint, u_atoms, adp_w)) - diagonal.insert(diagonal.end(), 6, h); + std::vector h, diagonal; + for (int i : u_atoms) { + h.insert(h.end(), 6, adp_w * xray_b[i] / 3.0); + diagonal.insert(diagonal.end(), 6, adp_w * xray_b[i] / 3.0 + restraint[i]); + } const RefinementStage s3("minimise.u"); - Minimise(ctx, u_target, settings.b_iterations, b_minimiser, diagonal); + MinimiseU(ctx, u_target, b_minimiser, h, FlooredDiagonal(diagonal), settings.b_iterations); } else if (adp_w > 0) { const RestraintSet b_set = BRestraintsNow(ctx, topo); ctx.engine->SetRestraints(1, b_set); - std::vector xray = ctx.XrayBCurvatureShape(); - if (cycle == 0) - xray_scale = ctx.CalibrateBCurvature(refinable, xray); - for (double &x : xray) - x *= xray_scale; + std::vector xray_b = b_curvature(); + for (double &x : xray_b) + x *= adp_w; const RefinementStage s3("minimise.b"); - Minimise(ctx, b, settings.b_iterations, b_minimiser, - BDiagonal(xray, BRestraintCurvature(b_set, ctx.XyzB()), refinable, adp_w)); + MinimiseAtoms(ctx, b, b_minimiser, xray_b, b_set, settings.b_iterations); } } PlaceRiders(ctx, topo); @@ -3050,7 +3468,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(settings.population, trial_slots.size()); + const size_t population = std::min(std::max(settings.population, 0), trial_slots.size()); if (population > 1 && state.ran && !ctx.Cancelled()) { const RefinementStage pop_stage("population"); const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma; @@ -4236,17 +4654,11 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &data = moved > 0 ? closed : merged; - // RefinementSettings::lean_on_cpu #ifdef JFJOCH_USE_CUDA const bool on_cpu = !settings.gpu || get_gpu_count() == 0; #else const bool on_cpu = true; #endif - if (settings.lean_on_cpu && on_cpu) { - settings.population = 0; - settings.rounds = 1; - logger.Info("Refinement on the CPU: no population and one round (half of the evaluations)"); - } // Each arm, untwinned and twinned, is a whole refinement - the solvent built and the displacement model chosen // against its own target - and the decision is taken on where they ended. A model only scored is scored both // ways and decided by the same rule, so the R-free it reports is not an untwinned one by default. @@ -4260,7 +4672,10 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector 1; + // on the CPU each arm on half of the threads - where the lattice admits a twin law, without which the twinned + // arm ends at once and its half would sit idle + const bool halves = on_cpu && settings.population <= 1 && settings.nthreads > 1 && + !RefinementTwinLaws(cell, sg).empty(); RefinementSettings untwinned_settings = settings, twin_settings = settings; if (halves) { twin_settings.nthreads = settings.nthreads / 2; diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index 08861f2a7..82f4ec048 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -48,8 +48,10 @@ struct RefinementSettings { bool score_only = false; // only score the model as given (RefinementResult::score_start), no stage bool group_b = true; // the placement stage's B shift per residue int macro_cycles = 2; // per weight trial, from the model the trial starts from - int xyz_iterations = 30; // L-BFGS iterations per macro-cycle - int b_iterations = 20; + // L-BFGS iterations per macro-cycle (on the Gauss-Newton normal matrix: RefinementMinimiser), of the positions and of + // the B factors, U and TLS + int xyz_iterations = 10; + int b_iterations = 7; // The first trials, from the placed model: the positions' X-ray weight, times the gradient-balanced one // (BalancedWeight), each with the B factors' X-ray weight their own balanced one times each of adp_ladder (0: B // held). @@ -84,8 +86,9 @@ struct RefinementSettings { // The population (RefineOnce, 2a.): this many members at most, refined from where the weight trajectory ended in // one wave, each on the data blurred along a schedule of its own and its positions kicked or not, then crossed // residue by residue on their averaged map and refined once more; the child kept where the free set prefers it. - // Capped at the trial slots; fewer than two: no population. - int population = 12; + // Capped at the trial slots; fewer than two: no population (the default: with the positions minimised on the + // Gauss-Newton matrix, the members took nothing off R-free over 19 sets that the two rounds did not). + int population = 0; // A refinement against the data as twinned by each twin law the lattice admits (RefinementTwinLaws), beside the // untwinned one, and the model of whichever the evidence favours (RefineModel). bool twin = true; @@ -97,11 +100,6 @@ struct RefinementSettings { // model of this crystal scored on a free set drawn here). bool as_given_competes = true; bool gpu = true; - // Where the refinement runs on the CPU (no GPU wanted, or none to run on), where an evaluation costs some thirty - // times a GPU's: no population and one round, for about half of the evaluations, at an R-free 0.001 higher in the - // median of 19 sets (at most 0.015, a placed prediction the population helps). The twinned data are still refined - // against: without that arm a crystal 30% twinned is refined as untwinned. - bool lean_on_cpu = true; size_t nthreads = 1; size_t gpu_resident_bytes = 0; // held on the card by what runs beside // The most weight trials run side by side, 0 for as many as the ladders and the cards take; the result does not diff --git a/image_analysis/structure_refinement/RefinementMinimiser.h b/image_analysis/structure_refinement/RefinementMinimiser.h index 1a19b2677..6bedb58fc 100644 --- a/image_analysis/structure_refinement/RefinementMinimiser.h +++ b/image_analysis/structure_refinement/RefinementMinimiser.h @@ -11,29 +11,58 @@ #include "../../common/ParallelFor.h" -// Limited-memory BFGS whose memory outlives one call, so that a refinement's macro-cycles - each a few dozen -// iterations on a target whose scale and likelihood parameters were refitted in between - go on from what the -// earlier ones learnt of the curvature instead of starting each from a steepest-descent step. The initial inverse -// Hessian is a diagonal the caller gives (SetDiagonal: an estimate of the Hessian's diagonal, so that a parameter -// with ten times the curvature of another takes a tenth of its step), scaled each iteration by the latest pair -// (Oren-Shanno in that metric). The line search backtracks from the full step to sufficient decrease. +// Sweeps over the parameters on `nthreads` threads: each element on its own, and dot products folded in fixed +// blocks (ParallelBlocks), so that the bits do not depend on the number of threads. +namespace refine_vec { +// a pass over fewer than this many parameters per thread costs less than waking the threads +constexpr size_t SWEEP_PER_THREAD = 65536; +template +void Each(size_t n, size_t nthreads, Fn fn) { + ParallelChunks(static_cast(n), ThreadsForWork(n, nthreads, SWEEP_PER_THREAD), [&](int lo, int hi) { + for (int k = lo; k < hi; k++) + fn(static_cast(k)); + }); +} +inline double Dot(const std::vector &a, const std::vector &b, size_t nthreads) { + const int n = static_cast(a.size()); + std::vector part(ReductionBlocks(n, SWEEP_PER_THREAD), 0.0); + ParallelBlocks(n, nthreads, [&](int block, int lo, int hi) { + double s = 0; + for (int k = lo; k < hi; k++) + s += a[k] * b[k]; + part[block] = s; + }, SWEEP_PER_THREAD); + double s = 0; + for (double v : part) + s += v; + return s; +} +} // namespace refine_vec + +// Limited-memory BFGS whose initial inverse Hessian is the caller's (SetInverseHessian: H0 q, here the Gauss-Newton +// normal matrix solved by ConjugateGradients), and whose memory outlives one call, so that a refinement's +// macro-cycles - each a few iterations on a target whose scale and likelihood parameters were refitted in between - +// go on from what the earlier ones learnt. The normal matrix holds what is known of the curvature in advance (the +// restraints exactly, the X-ray target per atom); the pairs add what it leaves out (the X-ray target's coupling of +// neighbouring atoms). The line search backtracks from the full step to sufficient decrease. // Following Nocedal (1980) Math. Comp. 35, 773-782 and Liu & Nocedal (1989) Math. Program. 45, 503-528 class RefinementMinimiser { public: // f(x, g): the target at x, and its gradient in g (sized as x); false where it cannot be evaluated there. using Target = std::function &x, double &f, std::vector &g)>; + // q = H0 q at x. + using InverseHessian = std::function &x, std::vector &q)>; - // `nthreads`: the threads the sweeps over the parameters run on (the dot products stay serial chains). + // `nthreads`: the threads the sweeps over the parameters run on. explicit RefinementMinimiser(size_t memory = 20, size_t nthreads = 1) : memory_(memory), nthreads_(nthreads) {} - // Forget the pairs (the target changed beyond what the memory still describes, e.g. another weight). + // Forget the pairs (the target changed beyond what the memory still describes). void Reset() { s_.clear(); y_.clear(); sy_.clear(); } - // The estimated diagonal of the Hessian; every element > 0. - void SetDiagonal(std::vector h) { h_ = std::move(h); } + void SetInverseHessian(InverseHessian h0) { h0_ = std::move(h0); } struct Outcome { int iterations = 0; @@ -47,8 +76,6 @@ public: Outcome Minimise(const Target &target, std::vector &x, int iterations, double f_tolerance, double max_step, const std::function &stop = nullptr) { const size_t n = x.size(); - if (h_.size() != n) - h_.assign(n, 1.0); Outcome out; double f = 0; std::vector g(n), d(n), x_new(n), g_new(n); @@ -59,11 +86,11 @@ public: for (int it = 0; it < iterations; it++) { if (stop && stop()) break; - Direction(g, d); + Direction(x, g, d); double gd = Dot(g, d); - if (!(gd < 0)) { // not a descent direction: the memory no longer fits, start again from the diagonal + if (!(gd < 0)) { // not a descent direction: the memory no longer fits, start again from H0 Reset(); - Direction(g, d); + Direction(x, g, d); gd = Dot(g, d); if (!(gd < 0)) break; @@ -75,7 +102,7 @@ public: double f_new = 0; bool accepted = false; for (int ls = 0; ls < 10; ls++) { - Sweep(n, [&](size_t k) { x_new[k] = x[k] + alpha * d[k]; }); + refine_vec::Each(n, nthreads_, [&](size_t k) { x_new[k] = x[k] + alpha * d[k]; }); const bool ok = target(x_new, f_new, g_new); out.evaluations++; if (ok && f_new <= f + 1e-4 * alpha * gd) { @@ -88,11 +115,8 @@ public: } if (!accepted) break; - // the pair's vectors are the ones the memory last let go, where it did: made anew, their pages fault in - std::vector s = std::move(spare_s_), y = std::move(spare_y_); - s.resize(n); - y.resize(n); - Sweep(n, [&](size_t k) { + std::vector s(n), y(n); + refine_vec::Each(n, nthreads_, [&](size_t k) { s[k] = x_new[k] - x[k]; y[k] = g_new[k] - g[k]; }); @@ -102,15 +126,10 @@ public: y_.push_back(std::move(y)); sy_.push_back(sy); if (s_.size() > memory_) { - spare_s_ = std::move(s_.front()); - spare_y_ = std::move(y_.front()); s_.pop_front(); y_.pop_front(); sy_.pop_front(); } - } else { - spare_s_ = std::move(s); - spare_y_ = std::move(y); } const double decrease = f - f_new; x.swap(x_new); @@ -125,87 +144,64 @@ public: } private: - static double Dot(const std::vector &a, const std::vector &b) { - double s = 0; - for (size_t k = 0; k < a.size(); k++) - s += a[k] * b[k]; - return s; - } + double Dot(const std::vector &a, const std::vector &b) const { return refine_vec::Dot(a, b, nthreads_); } - // fn(k) for every k below n on the threads, each element on its own: the serial loop's bits. - template - void Sweep(size_t n, Fn fn) const { - ParallelChunks(static_cast(n), ThreadsForWork(n, nthreads_), [&](int lo, int hi) { - for (int k = lo; k < hi; k++) - fn(static_cast(k)); - }); - } - - // d = -H g by the two-loop recursion, H0 = gamma diag(1/h). Every dot product is a serial chain of n - // additions, so each sweep over the parameters takes the next pair's dot product while it applies the - // previous pair's update to q - the same operations on each element in the same order as a sweep for - // each would do, with the update's memory traffic hidden under the chain. - void Direction(const std::vector &g, std::vector &d) const { - const size_t n = g.size(), m = s_.size(); - std::vector &q = q_; // kept from call to call: made anew, its pages fault in every time - q.assign(g.begin(), g.end()); - std::vector a(m), rho(m); - for (size_t i = 0; i < m; i++) - rho[i] = 1.0 / sy_[i]; - double dot = m > 0 ? Dot(s_[m - 1], q) : 0.0, yhy = 0.0; + // d = -H g by the two-loop recursion, H0 the caller's. + void Direction(const std::vector &x, const std::vector &g, std::vector &d) const { + const size_t m = s_.size(), n = g.size(); + std::vector q = g, a(m); for (size_t i = m; i-- > 0;) { - a[i] = rho[i] * dot; - const double ai = a[i]; - const double *y = y_[i].data(); - dot = 0.0; - if (i > 0) { - const double *s = s_[i - 1].data(); - for (size_t k = 0; k < n; k++) { - q[k] -= ai * y[k]; - dot += s[k] * q[k]; - } - } else { - const double *yl = y_[m - 1].data(); - for (size_t k = 0; k < n; k++) { - q[k] -= ai * y[k]; - yhy += yl[k] * yl[k] / h_[k]; - } - } + a[i] = Dot(s_[i], q) / sy_[i]; + refine_vec::Each(n, nthreads_, [&](size_t k) { q[k] -= a[i] * y_[i][k]; }); } - const double gamma = m > 0 ? sy_[m - 1] / yhy : 1.0; - if (m > 0) { - const double *y = y_[0].data(); - for (size_t k = 0; k < n; k++) { - q[k] *= gamma / h_[k]; - dot += y[k] * q[k]; - } - } else - for (size_t k = 0; k < n; k++) - q[k] *= gamma / h_[k]; + h0_(x, q); for (size_t i = 0; i < m; i++) { - // b on its own and (a[i] - b) in the loop, as before: written as a[i] - rho[i] * dot the compiler - // contracts it into a fused multiply-add, and the bits change - const double b = rho[i] * dot; - const double *s = s_[i].data(); - dot = 0.0; - if (i + 1 < m) { - const double *y = y_[i + 1].data(); - for (size_t k = 0; k < n; k++) { - q[k] += s[k] * (a[i] - b); - dot += y[k] * q[k]; - } - } else - for (size_t k = 0; k < n; k++) - q[k] += s[k] * (a[i] - b); + const double b = Dot(y_[i], q) / sy_[i]; + refine_vec::Each(n, nthreads_, [&](size_t k) { q[k] += s_[i][k] * (a[i] - b); }); } - for (size_t k = 0; k < n; k++) - d[k] = -q[k]; + d.resize(n); + refine_vec::Each(n, nthreads_, [&](size_t k) { d[k] = -q[k]; }); } size_t memory_, nthreads_; + InverseHessian h0_; std::deque> s_, y_; std::deque sy_; // s.y of each pair, as taken when it was kept - std::vector spare_s_, spare_y_; // the pair the memory last let go, for the next one to reuse - mutable std::vector q_; // Direction()'s working vector - std::vector h_; }; + +// z with H z = b to `tolerance` of |b| (at most `max_iterations`), by conjugate gradients preconditioned with M: +// hv = H v (product), z = M^-1 r (precondition), both symmetric positive definite. Returns the iterations taken. +// Following Hestenes & Stiefel (1952) J. Res. Natl. Bur. Stand. 49, 409-436 +inline int ConjugateGradients(const std::function &v, std::vector &hv)> &product, + const std::function &r, std::vector &z)> &precondition, + const std::vector &b, std::vector &x, int max_iterations, double tolerance, + size_t nthreads = 1) { + auto dot = [&](const std::vector &u, const std::vector &v) { return refine_vec::Dot(u, v, nthreads); }; + const size_t n = b.size(); + x.assign(n, 0.0); + std::vector r = b, z(n), p, hp(n); + precondition(r, z); + p = z; + double rz = dot(r, z); + const double r0 = std::sqrt(dot(r, r)); + int k = 0; + while (k < max_iterations) { + k++; + product(p, hp); + const double php = dot(p, hp); + if (!(php > 0)) + break; + const double a = rz / php; + refine_vec::Each(n, nthreads, [&](size_t j) { + x[j] += a * p[j]; + r[j] -= a * hp[j]; + }); + if (std::sqrt(dot(r, r)) < tolerance * r0) + break; + precondition(r, z); + const double rz_new = dot(r, z); + refine_vec::Each(n, nthreads, [&](size_t j) { p[j] = z[j] + rz_new / rz * p[j]; }); + rz = rz_new; + } + return k; +} diff --git a/image_analysis/structure_refinement/RefinementNewtonGPU.cpp b/image_analysis/structure_refinement/RefinementNewtonGPU.cpp new file mode 100644 index 000000000..990a3b692 --- /dev/null +++ b/image_analysis/structure_refinement/RefinementNewtonGPU.cpp @@ -0,0 +1,59 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include "RefinementNewtonGPU.h" + +#include "RefinementRestraints.h" + +// The host half: the index of J, as RestraintGaussNewton lays out its rows, with every row kept. +void RestraintNormalMatrixGPU::At(const RestraintSet &set, const std::vector> &xyzb) { + if (!indexed_) { + std::vector record_entry, start{0}, atom; + for (const RestraintRecord &r : set.records) { + record_entry.push_back(static_cast(atom.size())); + switch (static_cast(r.kind)) { + case RestraintKind::Plane: + // one row per atom of the plane, each over all of them + for (int i = 0; i < r.atom[1]; i++) { + for (int k = 0; k < r.atom[1]; k++) + atom.push_back(set.plane_atoms[r.atom[0] + k]); + start.push_back(static_cast(atom.size())); + } + continue; + case RestraintKind::Angle: + atom.insert(atom.end(), r.atom, r.atom + 3); + break; + case RestraintKind::Torsion: + case RestraintKind::Chiral: + case RestraintKind::NcsDistance: + atom.insert(atom.end(), r.atom, r.atom + 4); + break; + default: // Bond, Repulsion, BSimilarity + atom.insert(atom.end(), r.atom, r.atom + 2); + break; + } + start.push_back(static_cast(atom.size())); + } + const size_t rows = start.size() - 1; + std::vector row_of(atom.size()), param(atom.size()); + for (size_t e = 0; e < atom.size(); e++) + param[e] = param_of_[atom[e]]; + for (size_t r = 0; r < rows; r++) + for (int e = start[r]; e < start[r + 1]; e++) + row_of[e] = static_cast(r); + std::vector entry_start(n_params_ + 1, 0); + for (int k : param) + if (k >= 0) + entry_start[k + 1]++; + for (size_t k = 0; k < n_params_; k++) + entry_start[k + 1] += entry_start[k]; + std::vector entry(entry_start.back()); + std::vector fill(entry_start.begin(), entry_start.end() - 1); + for (size_t e = 0; e < param.size(); e++) + if (param[e] >= 0) + entry[fill[param[e]]++] = static_cast(e); + Upload(set.records, set.plane_atoms, record_entry, start, param, row_of, entry_start, entry); + indexed_ = true; + } + Fill(xyzb); +} diff --git a/image_analysis/structure_refinement/RefinementNewtonGPU.cu b/image_analysis/structure_refinement/RefinementNewtonGPU.cu new file mode 100644 index 000000000..a11281e46 --- /dev/null +++ b/image_analysis/structure_refinement/RefinementNewtonGPU.cu @@ -0,0 +1,319 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +// The device half of RestraintNormalMatrixGPU, in float: one thread per restraint for J, one per row of J or per +// parameter atom for the products, each summing its terms in order. J is held in float; the products' v, t and hv in +// float pairs (hi + lo, double-float arithmetic: an error-free sum and product per term), since the soft directions of +// H - those the X-ray term decides, J v the small difference of large terms - need more than float's 24 bits: with +// plain float sums the Newton steps on a model with stiff restraints differed from the CPU's by more than its noise. + +#include "RefinementNewtonGPU.h" + +#include + +#include "../indexing/CUDAMemHelpers.h" +#include "RefinementGPUEngine.h" // MAX_PLANE_ATOMS + +namespace { + +constexpr int THREADS = 256; +using V3f = refine_math::V3; + +void cuda_err(cudaError_t val) { + if (val != cudaSuccess) + throw JFJochException(JFJochExceptionCategory::GPUCUDAError, cudaGetErrorString(val)); +} + +class DeviceGuard { +public: + explicit DeviceGuard(int device) { + cuda_err(cudaGetDevice(&previous_)); + cuda_err(cudaSetDevice(device)); + } + ~DeviceGuard() { cudaSetDevice(previous_); } + DeviceGuard(const DeviceGuard &) = delete; + DeviceGuard &operator=(const DeviceGuard &) = delete; + +private: + int previous_ = 0; +}; + +int GreatestStreamPriority() { + int least = 0, greatest = 0; + cuda_err(cudaDeviceGetStreamPriorityRange(&least, &greatest)); + return greatest; +} + +int GridSize(int n) { + return (n + THREADS - 1) / THREADS; +} + +__device__ V3f xyz(const float4 *p, int i) { + return {p[i].x, p[i].y, p[i].z}; +} + +__device__ float Get(const float4 &j, int c) { + return c == 0 ? j.x : c == 1 ? j.y : c == 2 ? j.z : j.w; +} + +// sum += a (b.x + b.y), sum and b float pairs (hi in x, lo in y). Needs fma and no reassociation (no fast math). +// The exact product and 2Sum following Dekker (1971) Numer. Math. 18, 224-242 and Knuth, TAOCP vol. 2, 4.2.2 +__device__ void AddProduct(float2 &sum, float a, float2 b) { + const float p = a * b.x; + const float p_err = fmaf(a, b.x, -p); // a b.x = p + p_err exactly + const float s = sum.x + p; + const float v = s - sum.x; + const float s_err = (sum.x - (s - v)) + (p - v); // sum.x + p = s + s_err exactly + const float lo = s_err + sum.y + p_err + a * b.y; + const float hi = s + lo; + sum = make_float2(hi, lo - (hi - s)); +} + +// Each restraint's rows of J (RestraintGaussNewton): E = residual^2, so J = g / sqrt(2 E); a plane's atoms each a +// residual of their own, d r_i / d x_k = (delta_ik - 1/m) sqrt(2) normal / sigma. Zero where there is no row. +__global__ void JacobianKernel(const RestraintRecord *records, int n, const int *plane_atoms, const int *record_entry, + const float4 *xyzb, float4 *j) { + const int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i >= n) + return; + const RestraintRecord &r = records[i]; + float4 *out = j + record_entry[i]; + V3f g[4] = {}; + int na = 0; + float e = 0; + switch (static_cast(r.kind)) { + case RestraintKind::Bond: + e = refine_math::BondTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), r.ideal, r.sigma, g); + na = 2; + break; + case RestraintKind::Angle: + e = refine_math::AngleTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]), r.ideal, + r.sigma, g); + na = 3; + break; + case RestraintKind::Torsion: + e = refine_math::TorsionTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]), + xyz(xyzb, r.atom[3]), r.ideal, r.sigma, r.period, g); + na = 4; + break; + case RestraintKind::Chiral: + e = refine_math::ChiralTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]), + xyz(xyzb, r.atom[3]), r.ideal, r.sigma, g); + na = 4; + break; + case RestraintKind::NcsDistance: + e = refine_math::NcsDistanceTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]), + xyz(xyzb, r.atom[3]), r.sigma, r.ideal, g); + na = 4; + break; + case RestraintKind::Repulsion: { + const V3f q = xyz(xyzb, r.atom[1]); + const V3f image{r.rot[0] * q.x + r.rot[1] * q.y + r.rot[2] * q.z + r.tran[0], + r.rot[3] * q.x + r.rot[4] * q.y + r.rot[5] * q.z + r.tran[1], + r.rot[6] * q.x + r.rot[7] * q.y + r.rot[8] * q.z + r.tran[2]}; + e = refine_math::RepulsionTerm(xyz(xyzb, r.atom[0]), image, r.ideal, r.sigma, g); + const V3f gi = g[1]; + g[1] = {r.rot[0] * gi.x + r.rot[3] * gi.y + r.rot[6] * gi.z, + r.rot[1] * gi.x + r.rot[4] * gi.y + r.rot[7] * gi.z, + r.rot[2] * gi.x + r.rot[5] * gi.y + r.rot[8] * gi.z}; + na = 2; + break; + } + case RestraintKind::Plane: { + V3f px[RefinementGPUEngine::MAX_PLANE_ATOMS], pg[RefinementGPUEngine::MAX_PLANE_ATOMS]; + const int m = r.atom[1]; + for (int k = 0; k < m; k++) + px[k] = xyz(xyzb, plane_atoms[r.atom[0] + k]); + refine_math::PlaneTerm(px, m, r.sigma, pg); + V3f nv{0, 0, 0}; + for (int k = 0; k < m; k++) // the normal from the largest of the gradients (each is 2 r_k / sigma n) + if (refine_math::Norm(pg[k]) > refine_math::Norm(nv)) + nv = pg[k]; + const float norm = refine_math::Norm(nv); + const float c = norm > 0 ? sqrtf(2.0f) / (r.sigma * norm) : 0.0f; + for (int a = 0; a < m; a++) + for (int k = 0; k < m; k++) { + const float f = c * ((a == k ? 1.0f : 0.0f) - 1.0f / m); + out[a * m + k] = make_float4(f * nv.x, f * nv.y, f * nv.z, 0.0f); + } + return; + } + case RestraintKind::BSimilarity: { + float gi = 0, gj = 0; + e = refine_math::BSimilarityTerm(xyzb[r.atom[0]].w, xyzb[r.atom[1]].w, r.sigma, gi, gj); + const float f = e > 0 ? 1.0f / sqrtf(2.0f * e) : 0.0f; + out[0] = make_float4(0, 0, 0, f * gi); + out[1] = make_float4(0, 0, 0, f * gj); + return; + } + } + const float f = e > 0 ? 1.0f / sqrtf(2.0f * e) : 0.0f; + for (int k = 0; k < na; k++) + out[k] = make_float4(f * g[k].x, f * g[k].y, f * g[k].z, 0.0f); +} + +// t_r = sum over the row's entries of J_e . v_k +__global__ void RowProduct(const int *start, const int *param, const float4 *j, int per, int c0, const float2 *v, + float2 *t, int rows) { + const int r = blockIdx.x * blockDim.x + threadIdx.x; + if (r >= rows) + return; + float2 dot = make_float2(0, 0); + for (int e = start[r]; e < start[r + 1]; e++) + if (const int k = param[e]; k >= 0) + for (int c = 0; c < per; c++) + AddProduct(dot, Get(j[e], c0 + c), v[per * k + c]); + t[r] = dot; +} + +// hv_k = sum over the parameter atom's entries of t_row J_e +__global__ void TransposeProduct(const int *entry_start, const int *entry, const int *row_of, const float4 *j, int per, + int c0, const float2 *t, float2 *hv, int n) { + const int k = blockIdx.x * blockDim.x + threadIdx.x; + if (k >= n) + return; + for (int c = 0; c < per; c++) { + float2 sum = make_float2(0, 0); + for (int q = entry_start[k]; q < entry_start[k + 1]; q++) { + const int e = entry[q]; + AddProduct(sum, Get(j[e], c0 + c), t[row_of[e]]); + } + hv[per * k + c] = sum; + } +} + +__global__ void DiagonalKernel(const int *entry_start, const int *entry, const float4 *j, int per, int c0, float *d, + int n) { + const int k = blockIdx.x * blockDim.x + threadIdx.x; + if (k >= n) + return; + for (int c = 0; c < per; c++) { + float sum = 0; + for (int q = entry_start[k]; q < entry_start[k + 1]; q++) { + const float x = Get(j[entry[q]], c0 + c); + sum += x * x; + } + d[per * k + c] = sum; + } +} + +template +CudaDevicePtr Up(const std::vector &host, cudaStream_t stream) { + CudaDevicePtr d(std::max(host.size(), 1)); + cuda_err(cudaMemcpyAsync(d.get(), host.data(), host.size() * sizeof(T), cudaMemcpyHostToDevice, stream)); + return d; +} + +} // namespace + +struct RestraintNormalMatrixGPU::Device { + int device; + CudaStream stream; + int records = 0, rows = 0; + CudaDevicePtr record; + CudaDevicePtr plane_atoms, record_entry, start, param, row_of, entry_start, entry; + CudaDevicePtr xyzb, j; + CudaDevicePtr t, v, hv; + CudaDevicePtr diagonal; + PinnedVector host; // v and hv as float pairs + PinnedVector host_xyzb; +}; + +RestraintNormalMatrixGPU::RestraintNormalMatrixGPU(int device, const std::vector ¶m_atoms, size_t n_atoms, + bool xyz) + : per_(xyz ? 3 : 1), c0_(xyz ? 0 : 3), n_params_(param_atoms.size()), param_of_(n_atoms, -1) { + for (size_t k = 0; k < param_atoms.size(); k++) + param_of_[param_atoms[k]] = static_cast(k); + DeviceGuard guard(device); + // At the device's highest priority: a product is a few small kernels the minimiser waits for, queued beside the + // X-ray evaluations of the other trials on the card. + dev_ = std::unique_ptr(new Device{device, CudaStream(cudaStreamNonBlocking, GreatestStreamPriority())}); + const size_t n = std::max(per_ * n_params_, 1); + dev_->xyzb = CudaDevicePtr(std::max(n_atoms, 1)); + dev_->host_xyzb.resize(n_atoms); + dev_->v = CudaDevicePtr(n); + dev_->hv = CudaDevicePtr(n); + dev_->diagonal = CudaDevicePtr(n); + dev_->host.resize(n); +} + +RestraintNormalMatrixGPU::~RestraintNormalMatrixGPU() { + DeviceGuard guard(dev_->device); + dev_.reset(); +} + +void RestraintNormalMatrixGPU::Upload(const std::vector &records, const std::vector &plane_atoms, + const std::vector &record_entry, const std::vector &start, + const std::vector ¶m, const std::vector &row_of, + const std::vector &entry_start, const std::vector &entry) { + DeviceGuard guard(dev_->device); + Device &d = *dev_; + d.records = static_cast(records.size()); + d.rows = static_cast(start.size()) - 1; + d.record = Up(records, d.stream); + d.plane_atoms = Up(plane_atoms, d.stream); + d.record_entry = Up(record_entry, d.stream); + d.start = Up(start, d.stream); + d.param = Up(param, d.stream); + d.row_of = Up(row_of, d.stream); + d.entry_start = Up(entry_start, d.stream); + d.entry = Up(entry, d.stream); + d.j = CudaDevicePtr(std::max(param.size(), 1)); + d.t = CudaDevicePtr(std::max(d.rows, 1)); + cuda_err(cudaStreamSynchronize(d.stream)); +} + +void RestraintNormalMatrixGPU::Fill(const std::vector> &xyzb) { + DeviceGuard guard(dev_->device); + Device &d = *dev_; + for (size_t a = 0; a < xyzb.size(); a++) + d.host_xyzb[a] = make_float4(static_cast(xyzb[a][0]), static_cast(xyzb[a][1]), + static_cast(xyzb[a][2]), static_cast(xyzb[a][3])); + cuda_err(cudaMemcpyAsync(d.xyzb.get(), d.host_xyzb.data(), xyzb.size() * sizeof(float4), cudaMemcpyHostToDevice, + d.stream)); + if (d.records > 0) + JacobianKernel<<>>(d.record, d.records, d.plane_atoms, + d.record_entry, d.xyzb, d.j); + cuda_err(cudaGetLastError()); + cuda_err(cudaStreamSynchronize(d.stream)); +} + +void RestraintNormalMatrixGPU::Multiply(const std::vector &v, std::vector &hv) { + DeviceGuard guard(dev_->device); + Device &d = *dev_; + const int n = static_cast(n_params_); + hv.resize(per_ * n_params_); + if (n == 0) + return; + for (size_t i = 0; i < hv.size(); i++) { + const float hi = static_cast(v[i]); + d.host[i] = make_float2(hi, static_cast(v[i] - hi)); + } + cuda_err(cudaMemcpyAsync(d.v.get(), d.host.data(), hv.size() * sizeof(float2), cudaMemcpyHostToDevice, d.stream)); + if (d.rows > 0) + RowProduct<<>>(d.start, d.param, d.j, per_, c0_, d.v, d.t, d.rows); + TransposeProduct<<>>(d.entry_start, d.entry, d.row_of, d.j, per_, c0_, d.t, + d.hv, n); + cuda_err(cudaGetLastError()); + cuda_err(cudaMemcpyAsync(d.host.data(), d.hv.get(), hv.size() * sizeof(float2), cudaMemcpyDeviceToHost, d.stream)); + cuda_err(cudaStreamSynchronize(d.stream)); + for (size_t i = 0; i < hv.size(); i++) + hv[i] = static_cast(d.host[i].x) + d.host[i].y; +} + +std::vector RestraintNormalMatrixGPU::Diagonal() const { + DeviceGuard guard(dev_->device); + Device &d = *dev_; + const int n = static_cast(n_params_); + std::vector out(per_ * n_params_); + if (n == 0) + return out; + std::vector host(out.size()); + DiagonalKernel<<>>(d.entry_start, d.entry, d.j, per_, c0_, d.diagonal, n); + cuda_err(cudaGetLastError()); + cuda_err(cudaMemcpyAsync(host.data(), d.diagonal.get(), host.size() * sizeof(float), cudaMemcpyDeviceToHost, d.stream)); + cuda_err(cudaStreamSynchronize(d.stream)); + std::copy(host.begin(), host.end(), out.begin()); + return out; +} + diff --git a/image_analysis/structure_refinement/RefinementNewtonGPU.h b/image_analysis/structure_refinement/RefinementNewtonGPU.h new file mode 100644 index 000000000..317cfa8f9 --- /dev/null +++ b/image_analysis/structure_refinement/RefinementNewtonGPU.h @@ -0,0 +1,52 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#pragma once + +// The restraints' Gauss-Newton normal matrix J^T J (RestraintGaussNewton) of the Newton minimisation on a GPU (CUDA +// builds only): RestraintNormalMatrix of Refinement.cpp on the card, in single precision. J is worked out on the device, +// one thread per restraint, with the terms the engine evaluates the restraints with (RefinementMath.h); a product is two +// gathers, J v per row and J^T t per parameter atom over its entries, each summed in a fixed order without atomics, in +// double-float pairs (RefinementNewtonGPU.cu). +// +// Every restraint keeps its rows whatever its E: where RestraintGaussNewton drops a row (E = 0, a plane without a +// normal) its entries here are zero, so the index of J is worked out once, at the first At(), and the set must stay +// the same for the matrix's life. + +#include +#include +#include +#include + +#include "RefinementMath.h" // RestraintRecord + +struct RestraintSet; + +class RestraintNormalMatrixGPU { +public: + // On the x y z (`xyz`) or the B of `param_atoms` of the model's `n_atoms`, on GPU `device`. + RestraintNormalMatrixGPU(int device, const std::vector ¶m_atoms, size_t n_atoms, bool xyz); + ~RestraintNormalMatrixGPU(); + + void At(const RestraintSet &set, const std::vector> &xyzb); + // hv = J^T J v + void Multiply(const std::vector &v, std::vector &hv); + // The diagonal of J^T J. + std::vector Diagonal() const; + +private: + // The set and the index of J (RefinementNewtonGPU.cu): per restraint its first entry, per row its entries + // start[r] ... start[r + 1], per entry its parameter atom (-1: none) and row, per parameter atom its entries. + void Upload(const std::vector &records, const std::vector &plane_atoms, + const std::vector &record_entry, const std::vector &start, const std::vector ¶m, + const std::vector &row_of, const std::vector &entry_start, const std::vector &entry); + // J at `xyzb` (x, y, z, B per atom), in float + void Fill(const std::vector> &xyzb); + + int per_, c0_; + size_t n_params_; + std::vector param_of_; + bool indexed_ = false; + struct Device; + std::unique_ptr dev_; +}; diff --git a/image_analysis/structure_refinement/RefinementRestraints.cpp b/image_analysis/structure_refinement/RefinementRestraints.cpp index d35c89910..73cc05952 100644 --- a/image_analysis/structure_refinement/RefinementRestraints.cpp +++ b/image_analysis/structure_refinement/RefinementRestraints.cpp @@ -397,6 +397,48 @@ double RestraintValue(const RestraintSet &set, const RestraintRecord &r, const s return NAN; } +namespace { +// E of a restraint of the kinds over up to four atoms (not a plane, not a B similarity), its gradient in g[0..n). +double RecordTerm(const RestraintRecord &r, const std::vector> &xyzb, V3 g[4], int &n) { + const double ideal = r.ideal, sigma = r.sigma; + double e = 0; + n = 0; + switch (static_cast(r.kind)) { + case RestraintKind::Bond: + e = refine_math::BondTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), ideal, sigma, g); + n = 2; + break; + case RestraintKind::Angle: + e = refine_math::AngleTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), ideal, sigma, g); + n = 3; + break; + case RestraintKind::Torsion: + e = refine_math::TorsionTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), + At(xyzb, r.atom[3]), ideal, sigma, r.period, g); + n = 4; + break; + case RestraintKind::Chiral: + e = refine_math::ChiralTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), + At(xyzb, r.atom[3]), ideal, sigma, g); + n = 4; + break; + case RestraintKind::NcsDistance: + e = refine_math::NcsDistanceTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), + At(xyzb, r.atom[3]), sigma, ideal, g); + n = 4; + break; + case RestraintKind::Repulsion: + e = refine_math::RepulsionTerm(At(xyzb, r.atom[0]), Image(r, At(xyzb, r.atom[1])), ideal, sigma, g); + g[1] = FromImage(r, g[1]); + n = 2; + break; + default: + break; + } + return e; +} +} // namespace + double EvaluateRestraints(const RestraintSet &set, const std::vector> &xyzb, std::vector> *grad) { double sum = 0; @@ -405,36 +447,8 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector g[4] = {}; int n = 0; double e = 0; - const double ideal = r.ideal, sigma = r.sigma; + const double sigma = r.sigma; switch (static_cast(r.kind)) { - case RestraintKind::Bond: - e = refine_math::BondTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), ideal, sigma, g); - n = 2; - break; - case RestraintKind::Angle: - e = refine_math::AngleTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), ideal, sigma, g); - n = 3; - break; - case RestraintKind::Torsion: - e = refine_math::TorsionTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), - At(xyzb, r.atom[3]), ideal, sigma, r.period, g); - n = 4; - break; - case RestraintKind::Chiral: - e = refine_math::ChiralTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), - At(xyzb, r.atom[3]), ideal, sigma, g); - n = 4; - break; - case RestraintKind::NcsDistance: - e = refine_math::NcsDistanceTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), - At(xyzb, r.atom[3]), sigma, ideal, g); - n = 4; - break; - case RestraintKind::Repulsion: - e = refine_math::RepulsionTerm(At(xyzb, r.atom[0]), Image(r, At(xyzb, r.atom[1])), ideal, sigma, g); - g[1] = FromImage(r, g[1]); - n = 2; - break; case RestraintKind::Plane: { const int m = r.atom[1]; px.resize(m); @@ -460,6 +474,8 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector BRestraintCurvature(const RestraintSet &set, const std::vector> &xyzb) { - std::vector h(xyzb.size(), 0.0); - for (const RestraintRecord &r : set.records) - if (r.kind == static_cast(RestraintKind::BSimilarity)) { - // d2E/dB_i^2 of BSimilarityTerm = 2 B_j / (kappa^2 B_i^3) - const double bi = xyzb[r.atom[0]][3], bj = xyzb[r.atom[1]][3], k2 = double(r.sigma) * r.sigma; - h[r.atom[0]] += 2.0 * bj / (k2 * bi * bi * bi); - h[r.atom[1]] += 2.0 * bi / (k2 * bj * bj * bj); +RestraintJacobian RestraintGaussNewton(const RestraintSet &set, const std::vector> &xyzb) { + RestraintJacobian jac; + jac.start.push_back(0); + auto row = [&](int atom, double x, double y, double z, double b) { + jac.atom.push_back(atom); + jac.j.push_back({x, y, z, b}); + }; + std::vector> px, pg; + for (const RestraintRecord &r : set.records) { + const RestraintKind kind = static_cast(r.kind); + if (kind == RestraintKind::Plane) { + // each atom's distance from the plane a residual of its own, d r_i / d x_j = (delta_ij - 1/n) normal / sigma + // (the normal held), so one row per atom of the plane over all of them + const int m = r.atom[1]; + px.resize(m); + pg.resize(m); + for (int k = 0; k < m; k++) + px[k] = At(xyzb, set.plane_atoms[r.atom[0] + k]); + refine_math::PlaneTerm(px.data(), m, static_cast(r.sigma), pg.data()); + const double c = std::sqrt(2.0) / r.sigma; + V3 nv{0, 0, 0}; + for (int k = 0; k < m; k++) // the normal from the largest of the gradients (each is 2 r_k / sigma n) + if (refine_math::Norm(pg[k]) > refine_math::Norm(nv)) + nv = pg[k]; + if (refine_math::Norm(nv) == 0) + continue; + nv = (1.0 / refine_math::Norm(nv)) * nv; + for (int i = 0; i < m; i++) { + for (int k = 0; k < m; k++) { + const double f = c * ((i == k ? 1.0 : 0.0) - 1.0 / m); + row(set.plane_atoms[r.atom[0] + k], f * nv.x, f * nv.y, f * nv.z, 0); + } + jac.start.push_back(static_cast(jac.atom.size())); + } + continue; } - return h; + // E = residual^2 (or close to it): its gradient is 2 residual grad(residual), so J = g / sqrt(2 E) gives the + // Gauss-Newton term 2 grad(residual) grad(residual)^T = J J^T; nothing where E is 0 + if (kind == RestraintKind::BSimilarity) { + double gi = 0, gj = 0; + const double e = refine_math::BSimilarityTerm(xyzb[r.atom[0]][3], xyzb[r.atom[1]][3], + static_cast(r.sigma), gi, gj); + if (!(e > 0)) + continue; + const double f = 1.0 / std::sqrt(2.0 * e); + row(r.atom[0], 0, 0, 0, f * gi); + row(r.atom[1], 0, 0, 0, f * gj); + } else { + V3 g[4] = {}; + int n = 0; + const double e = RecordTerm(r, xyzb, g, n); + if (!(e > 0)) + continue; + const double f = 1.0 / std::sqrt(2.0 * e); + for (int k = 0; k < n; k++) + row(r.atom[k], f * g[k].x, f * g[k].y, f * g[k].z, 0); + } + jac.start.push_back(static_cast(jac.atom.size())); + } + return jac; } gemmi::Position SpecialPosition::OnSite(const gemmi::Position &p) const { diff --git a/image_analysis/structure_refinement/RefinementRestraints.h b/image_analysis/structure_refinement/RefinementRestraints.h index 0a887a7e7..6fce88e58 100644 --- a/image_analysis/structure_refinement/RefinementRestraints.h +++ b/image_analysis/structure_refinement/RefinementRestraints.h @@ -49,8 +49,14 @@ double RestraintValue(const RestraintSet &set, const RestraintRecord &r, const s double EvaluateRestraints(const RestraintSet &set, const std::vector> &xyzb, std::vector> *grad); -// The diagonal of the Hessian of the set's B similarity restraints with respect to each atom's B, at `xyzb`. -std::vector BRestraintCurvature(const RestraintSet &set, const std::vector> &xyzb); +// The Gauss-Newton approximation of the set's Hessian, sum over rows of J J^T: per row (a restraint's +// residual) its derivatives in (x, y, z, B) of the atoms atom[start[r] ... start[r + 1]). +struct RestraintJacobian { + std::vector start; + std::vector atom; + std::vector> j; +}; +RestraintJacobian RestraintGaussNewton(const RestraintSet &set, const std::vector> &xyzb); // An atom on a crystallographic special position: one that some symmetry operations of the group, each with a // lattice translation, take onto itself - the site's symmetry. Its position moves only within the subspace those diff --git a/tests/RefinementTest.cpp b/tests/RefinementTest.cpp index c933e80bb..d2f5ac074 100644 --- a/tests/RefinementTest.cpp +++ b/tests/RefinementTest.cpp @@ -37,6 +37,7 @@ #include "../image_analysis/structure_refinement/RefinementXray.h" #ifdef JFJOCH_USE_CUDA #include "../image_analysis/structure_refinement/RefinementGPU.h" +#include "../image_analysis/structure_refinement/RefinementNewtonGPU.h" #endif namespace { @@ -787,6 +788,55 @@ TEST_CASE("RefinementRestraints_deposited_ligands", "[Refinement]") { } } +TEST_CASE("RefinementRestraints_gauss_newton", "[Refinement]") { + // near the ideal geometry the restraints' Hessian is their Gauss-Newton J^T J (the residuals' own curvature times + // residuals that are small): v.J^T J v against central differences of the gradient along v, for random v + const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90); + gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5); + st.models[0] = shaken(st.models[0], 0.005, 0.0, 2); + const RefinementTopology topo = BuildRefinementTopology(st); + RestraintSet set = topo.geometry; + set.Finalize(atoms_of(st.models[0]).size()); + std::vector> xyzb; + for (gemmi::Atom *a : atoms_of(st.models[0])) + xyzb.push_back({a->pos.x, a->pos.y, a->pos.z, a->b_iso}); + const RestraintJacobian jac = RestraintGaussNewton(set, xyzb); + REQUIRE(jac.start.size() > 10); + std::mt19937 rng(4); + std::normal_distribution normal(0.0, 1.0); + for (int trial = 0; trial < 5; trial++) { + std::vector> v(xyzb.size(), {0, 0, 0, 0}); + for (auto &e : v) + for (int c = 0; c < 3; c++) + e[c] = normal(rng); + double gn = 0; + for (size_t r = 0; r + 1 < jac.start.size(); r++) { + double jv = 0; + for (int e = jac.start[r]; e < jac.start[r + 1]; e++) + for (int c = 0; c < 3; c++) + jv += jac.j[e][c] * v[jac.atom[e]][c]; + gn += jv * jv; + } + const double h = 1e-5; + std::array>, 2> g; + for (int side = 0; side < 2; side++) { + auto x = xyzb; + for (size_t i = 0; i < x.size(); i++) + for (int c = 0; c < 3; c++) + x[i][c] += (side == 0 ? h : -h) * v[i][c]; + g[side].assign(x.size(), {0, 0, 0, 0}); + EvaluateRestraints(set, x, &g[side]); + } + double exact = 0; + for (size_t i = 0; i < xyzb.size(); i++) + for (int c = 0; c < 3; c++) + exact += v[i][c] * (g[0][i][c] - g[1][i][c]) / (2 * h); + CAPTURE(trial, gn, exact); + CHECK(gn > 0); + CHECK(gn == Catch::Approx(exact).epsilon(0.1)); + } +} + TEST_CASE("RefinementRestraints_gradient", "[Refinement]") { const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90); gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5); @@ -986,6 +1036,82 @@ TEST_CASE("RefinementXray_gradient_CPU", "[Refinement]") { } #ifdef JFJOCH_USE_CUDA +TEST_CASE("RefinementNewton_product_GPU_vs_CPU", "[Refinement]") { + if (get_gpu_count() == 0) + return; + const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90); + gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5); + st.models[0] = shaken(st.models[0], 0.15, 3.0, 2); + const RefinementTopology topo = BuildRefinementTopology(st); + RestraintSet set = topo.geometry; + set.Append(topo.adp); + set.Append(BuildRepulsions(topo, st.models[0], cell, *st.find_spacegroup(), 0.0)); + const std::vector atoms = atoms_of(st.models[0]); + set.Finalize(atoms.size()); + std::vector> xyzb; + for (gemmi::Atom *a : atoms) + xyzb.push_back({a->pos.x, a->pos.y, a->pos.z, a->b_iso}); + // every atom but each fifth a parameter: the rest drop out + std::vector params, param_of(atoms.size(), -1); + for (int a = 0; a < static_cast(atoms.size()); a++) + if (a % 5 != 4) { + param_of[a] = static_cast(params.size()); + params.push_back(a); + } + const RestraintJacobian jac = RestraintGaussNewton(set, xyzb); + REQUIRE(jac.start.size() > 100); + std::mt19937 gen(3); + std::uniform_real_distribution uniform(-1, 1); + // a random v, and a smooth one - the same shift of every atom and a thousandth of it at random, J v then the small + // difference of large terms + for (const auto [xyz, smooth] : {std::pair{true, false}, {true, true}, {false, false}, {false, true}}) { + CAPTURE(xyz, smooth); + const int per = xyz ? 3 : 1, c0 = xyz ? 0 : 3; + const size_t n = per * params.size(); + std::vector v(n); + for (size_t i = 0; i < n; i++) + v[i] = smooth ? 0.3 + 0.2 * (i % per) + 1e-3 * uniform(gen) : uniform(gen); + // J^T J v and the diagonal on the CPU, each entry in turn + std::vector hv(n, 0.0), diagonal(n, 0.0); + for (size_t r = 0; r + 1 < jac.start.size(); r++) { + double t = 0; + for (int e = jac.start[r]; e < jac.start[r + 1]; e++) + if (const int k = param_of[jac.atom[e]]; k >= 0) + for (int c = 0; c < per; c++) + t += jac.j[e][c0 + c] * v[per * k + c]; + for (int e = jac.start[r]; e < jac.start[r + 1]; e++) + if (const int k = param_of[jac.atom[e]]; k >= 0) + for (int c = 0; c < per; c++) + hv[per * k + c] += t * jac.j[e][c0 + c]; + } + for (size_t e = 0; e < jac.atom.size(); e++) + if (const int k = param_of[jac.atom[e]]; k >= 0) { + const std::array &j = jac.j[e]; + for (int c = 0; c < per; c++) + diagonal[per * k + c] += j[c0 + c] * j[c0 + c]; + } + RestraintNormalMatrixGPU gpu(0, params, atoms.size(), xyz); + gpu.At(set, xyzb); + std::vector hv_gpu; + gpu.Multiply(v, hv_gpu); + const std::vector diagonal_gpu = gpu.Diagonal(); + REQUIRE(hv_gpu.size() == n); + REQUIRE(diagonal_gpu.size() == n); + // In float, from float positions: within a few float epsilons of the double product in norm (in refinement + // its error is 2e-7 on average, 7e-6 at most), each element of the diagonal relative to itself. + double num = 0, den = 0, worst_diagonal = 0; + for (size_t i = 0; i < n; i++) { + num += (hv_gpu[i] - hv[i]) * (hv_gpu[i] - hv[i]); + den += hv[i] * hv[i]; + if (diagonal[i] > 0) + worst_diagonal = std::max(worst_diagonal, std::fabs(diagonal_gpu[i] - diagonal[i]) / diagonal[i]); + } + CAPTURE(std::sqrt(num / den), worst_diagonal); + CHECK(std::sqrt(num / den) < 1e-5); + CHECK(worst_diagonal < 1e-5); + } +} + TEST_CASE("RefinementXray_gradient_GPU", "[Refinement]") { if (get_gpu_count() == 0) return; @@ -1062,18 +1188,21 @@ TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") { return d; }; - // the exact diagonal: one step + // the exact inverse Hessian: one step RefinementMinimiser exact; - std::vector h(n); - for (size_t k = 0; k < n; k++) - h[k] = 2 * a[k]; - exact.SetDiagonal(h); + exact.SetInverseHessian([&](const std::vector &, std::vector &q) { + for (size_t k = 0; k < n; k++) + q[k] /= 2 * a[k]; + }); std::vector x(n, 0.0); exact.Minimise(target, x, 1, 0.0, 10.0); CHECK(distance(x) < 1e-9); - // without it, the memory carried from one call to the next does what one long call does + // with the identity, the memory carried from one call to the next does what one long call does RefinementMinimiser split, whole; + const RefinementMinimiser::InverseHessian identity = [](const std::vector &, std::vector &) {}; + split.SetInverseHessian(identity); + whole.SetInverseHessian(identity); std::vector xs(n, 0.0), xw(n, 0.0); for (int call = 0; call < 6; call++) split.Minimise(target, xs, 10, 0.0, 10.0); @@ -1083,6 +1212,44 @@ TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") { CHECK(xs[k] == Catch::Approx(xw[k]).margin(1e-12)); } +TEST_CASE("RefinementMinimiser_conjugate_gradients", "[Refinement]") { + // a tridiagonal positive definite system, solved to the tolerance with and without a (Jacobi) preconditioner + const size_t n = 200; + auto product = [&](const std::vector &v, std::vector &hv) { + hv.assign(n, 0.0); + for (size_t k = 0; k < n; k++) { + hv[k] = (2.0 + 0.01 * k) * v[k]; + if (k > 0) + hv[k] -= v[k - 1]; + if (k + 1 < n) + hv[k] -= v[k + 1]; + } + }; + std::vector b(n); + for (size_t k = 0; k < n; k++) + b[k] = std::sin(0.1 * k); + auto residual = [&](const std::vector &x) { + std::vector hx; + product(x, hx); + double r = 0, b2 = 0; + for (size_t k = 0; k < n; k++) { + r += (hx[k] - b[k]) * (hx[k] - b[k]); + b2 += b[k] * b[k]; + } + return std::sqrt(r / b2); + }; + std::vector x; + ConjugateGradients(product, [](const std::vector &r, std::vector &z) { z = r; }, b, x, 500, 1e-8); + CHECK(residual(x) < 1e-8); + const int jacobi = ConjugateGradients(product, [&](const std::vector &r, std::vector &z) { + z.resize(n); + for (size_t k = 0; k < n; k++) + z[k] = r[k] / (2.0 + 0.01 * k); + }, b, x, 500, 0.1); + CHECK(residual(x) < 0.1); + CHECK(jacobi < 50); +} + TEST_CASE("Refinement_restrained_recovers_model", "[Refinement]") { const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90); const gemmi::SpaceGroup &sg = *gemmi::find_spacegroup_by_name("P 21 21 21");