Merge branch 'rc188-k' into rc188-merge
Refinement: L-BFGS on the Gauss-Newton normal matrix with a per-parameter trust floor and a per-residue preconditioned CG, one protocol on the CPU and the GPU (no population, two rounds), the GPU Gauss-Newton restraint product, and CPU twin arms splitting threads only where a twin law exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
This commit is contained in:
commit
81fb4eafd1
13 files changed
+1353
-237
No files matched your search
+13
-7
@@ -598,12 +598,17 @@ Cryst. A34, 791-809 [doi:10.1107/S0567739478001618](https://doi.org/10.1107/S056
|
||||
memory BFGS method for large scale optimization" (1989), Math. Program. 45, 503-528
|
||||
[doi:10.1007/BF01589116](https://doi.org/10.1007/BF01589116); its two-loop recursion: J. Nocedal,
|
||||
"Updating quasi-Newton matrices with limited storage" (1980), Math. Comp. 35, 773-782
|
||||
[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). The
|
||||
initial inverse Hessian is rescaled at each iteration by the latest pair, the self-scaling of
|
||||
S. S. Oren and D. G. Luenberger, "Self-scaling variable metric (SSVM) algorithms. Part I" (1974),
|
||||
Management Sci. 20, 845-862 [doi:10.1287/mnsc.20.5.845](https://doi.org/10.1287/mnsc.20.5.845), and
|
||||
D. F. Shanno and K.-H. Phua, "Matrix conditioning and nonlinear optimization" (1978), Math. Program.
|
||||
14, 149-160 [doi:10.1007/BF01588962](https://doi.org/10.1007/BF01588962).
|
||||
[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). Its
|
||||
initial inverse Hessian is the target's sparse normal matrix - the X-ray target's curvature per atom
|
||||
on the diagonal and the restraints' Gauss-Newton matrix in full - as REFMAC refines on its normal
|
||||
matrix (Murshudov, Vagin & Dodson 1997, above), solved by conjugate gradients: M. R. Hestenes and
|
||||
E. Stiefel, "Methods of conjugate gradients for solving linear systems" (1952), J. Res. Natl. Bur.
|
||||
Stand. 49, 409-436 [doi:10.6028/jres.049.044](https://doi.org/10.6028/jres.049.044). Each atom's
|
||||
diagonal is raised to hold its step within the step bound, a trust region per atom after the damping
|
||||
of K. Levenberg, "A method for the solution of certain non-linear problems in least squares" (1944),
|
||||
Q. Appl. Math. 2, 164-168 [doi:10.1090/qam/10666](https://doi.org/10.1090/qam/10666), and D. W.
|
||||
Marquardt, "An algorithm for least-squares estimation of nonlinear parameters" (1963), J. Soc. Ind.
|
||||
Appl. Math. 11, 431-441 [doi:10.1137/0111030](https://doi.org/10.1137/0111030).
|
||||
|
||||
**Geometry restraints** — those of the CCP4 Monomer Library, a subset of which rugnux carries (see
|
||||
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). A. A. Vagin, R. A. Steiner, A. A. Lebedev,
|
||||
@@ -834,7 +839,8 @@ pseudorandom number generators" (2014), Proc. OOPSLA 2014, 453-472
|
||||
[doi:10.1145/2660193.2660195](https://doi.org/10.1145/2660193.2660195).
|
||||
|
||||
**Error-free floating-point arithmetic** — the single-precision exp shared between the CPU and an
|
||||
Apple GPU (`BraggExp.h`): the argument reduction of W. J. Cody and W. Waite, "Software Manual for the
|
||||
Apple GPU (`BraggExp.h`) and the double-float sums of the refinement's Gauss-Newton products on the GPU
|
||||
(`RefinementNewtonGPU.cu`): the argument reduction of W. J. Cody and W. Waite, "Software Manual for the
|
||||
Elementary Functions" (1980), Prentice-Hall; the exact product (with Veltkamp's split) and Fast2Sum
|
||||
of T. J. Dekker, "A floating-point technique for extending the available precision" (1971), Numer.
|
||||
Math. 18, 224-242; and the 2Sum of D. E. Knuth, "The Art of Computer Programming, Vol. 2:
|
||||
|
||||
@@ -66,12 +66,13 @@ The methods draw on, and in places reimplement, solutions from:
|
||||
- H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", *SIAM J. Numer. Anal.* **49** (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", *J. ACM* **12** (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6).
|
||||
- R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", *Biometrika* **10** (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared).
|
||||
- R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", *Acta Cryst.* **A34** (1978), 791-809 (the gradient of the refinement target, §15).
|
||||
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15).
|
||||
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target, the B restraints and the minimiser's sparse normal matrix, §15).
|
||||
- V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", *Acta Cryst.* **A51** (1995), 880-887 (the likelihood parameters from the free set, §15).
|
||||
- A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15).
|
||||
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (the weight chosen among refinements by R-free, §15).
|
||||
- D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15).
|
||||
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6).
|
||||
- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6).
|
||||
- G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5).
|
||||
- P. Roversi, E. Blanc, S. Johnson & S. M. Lea, "Tetartohedral twinning could happen to you too", *Acta Cryst.* **D68** (2012), 418-424 (twin refinement with more than two domains, and the free set closed under the twin domains, §15.12).
|
||||
- K. A. Sultana, I. Alexeev, I. Kursula, P. Mäntsälä, J. Niemi & G. Schneider, "Structure determination by multiwavelength anomalous diffraction of aclacinomycin oxidoreductase: indications of multidomain pseudomerohedral twinning", *Acta Cryst.* **D63** (2007), 149-159 (a six-domain twin of a monoclinic crystal on a hexagonal metric, §15.12).
|
||||
@@ -120,7 +121,6 @@ The methods draw on, and in places reimplement, solutions from:
|
||||
- V. Luzzati, "Traitement statistique des erreurs dans la détermination des structures cristallines", *Acta Cryst.* **5** (1952), 802-810 (the fall-off of $\sigma_A$ with resolution for a model's coordinate error, Luzzati's $D$, in the likelihood of a placement, §14.10).
|
||||
- D. Liebschner, P. V. Afonine, A. G. Urzhumtsev & P. D. Adams, "Implementation of the riding hydrogen model in CCTBX to support the next generation of X-ray and neutron joint refinement in Phenix", *Methods Enzymol.* **634** (2020), 177-199 (the riding hydrogens, placed by GEMMI's code, §15.2).
|
||||
- D. L. Theobald, "Rapid calculation of RMSDs using a quaternion-based characteristic polynomial", *Acta Cryst.* **A61** (2005), 478-480, and P. Liu, D. K. Agrafiotis & D. L. Theobald, *J. Comput. Chem.* **31** (2010), 1561-1563 (the QCP superposition behind the NCS rms criterion, via GEMMI, §15.5).
|
||||
- S. S. Oren & D. G. Luenberger, "Self-scaling variable metric (SSVM) algorithms. Part I", *Management Sci.* **20** (1974), 845-862, and D. F. Shanno & K.-H. Phua, "Matrix conditioning and nonlinear optimization", *Math. Program.* **14** (1978), 149-160 (the rescaling of the L-BFGS initial inverse Hessian by the latest pair, §15.6).
|
||||
|
||||
(list is not exhaustive; the full citations, with DOIs, are in [ACKNOWLEDGEMENT.md](ACKNOWLEDGEMENT.md))
|
||||
|
||||
@@ -286,26 +286,51 @@ Whether the restraints are used at all, and how tightly ($\sigma$), the weight t
|
||||
|
||||
### 15.6 The minimiser
|
||||
|
||||
Positions and B factors are minimised in turn, each by an L-BFGS of rugnux's own (`RefinementMinimiser`),
|
||||
with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a backtracking line
|
||||
search from the full step to sufficient decrease, its first trial moving no position by more than 0.3 Å and
|
||||
no B by more than 10 Ų. What sets it apart from a textbook L-BFGS:
|
||||
Positions, B factors, anisotropic U and TLS parameters are minimised in turn, each by an L-BFGS of rugnux's own
|
||||
(`RefinementMinimiser`), with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a
|
||||
backtracking line search from the full step to sufficient decrease, its first trial moving no position by more
|
||||
than 0.3 Šand no B by more than 10 Ų. What sets it apart from a textbook L-BFGS is its **initial inverse
|
||||
Hessian: the target's Gauss-Newton normal matrix**, as REFMAC refines on its sparse normal matrix (Murshudov,
|
||||
Vagin & Dodson 1997). The matrix holds what is known of the curvature in advance; the L-BFGS pairs add what it
|
||||
leaves out.
|
||||
|
||||
- **Its memory is kept from one macro-cycle to the next.** A macro-cycle is a few dozen iterations on a target
|
||||
whose scale and likelihood parameters were refitted in between; starting each from what the earlier ones
|
||||
learnt of the curvature, instead of from a steepest-descent step, is what lets two macro-cycles of a
|
||||
weight trial get as far as many restarted ones. Each weight trial starts with an empty memory.
|
||||
- **The B steps are scaled by their curvature.** The initial inverse Hessian of the B factors is a
|
||||
diagonal: the X-ray target's curvature in each atom's B - its shape from the likelihood's curvature in
|
||||
$|F_\mathrm{model}|$ and the atom's own scattering, summed in resolution shells, calibrated once per trial
|
||||
against central differences of the gradient along a random $\pm1$ direction - plus the B restraints'
|
||||
curvature, each element at least a thousandth of the median. The positions take none: their restraints'
|
||||
curvature is that of stiff bonds and angles, which move atoms together, and as a diagonal it slows the soft
|
||||
(torsion and X-ray) directions rather than scaling them.
|
||||
- **The restraints' part is their Gauss-Newton matrix $J^\mathsf{T}J$, in full.** Every restraint is the square of
|
||||
a residual (or close to it: the NCS distances' robust loss), so its gradient is $2r\,\nabla r$ and
|
||||
$J = \nabla E / \sqrt{2E}$ is its row of the Jacobian, $\sqrt2\,\nabla r$; a plane contributes one row per
|
||||
atom. $J^\mathsf{T}J$ is positive semi-definite where the exact Hessian is not (torsions, repulsions), and near
|
||||
the ideal geometry it is the exact Hessian. Its products $J^\mathsf{T}(Jv)$ are two gathers - over the rows,
|
||||
then over each atom's entries - on the slot's threads, or on its GPU.
|
||||
- **The X-ray part is per atom, on the diagonal.** The X-ray target's curvature in an atom's $x$, $y$ and $z$ is
|
||||
the likelihood's curvature in $|F_\mathrm{model}|$ summed in resolution shells against the atom's own part of
|
||||
it, $(2\pi)^2 s^2/3\,f(s)^2\,e^{-Bs^2/2}$ (the reflections taken as spread evenly over directions); in its B,
|
||||
$(s^2/4)^2 f(s)^2 e^{-Bs^2/2}$, a third of it to each element of a U; and a TLS body's dense $20\times20$ block
|
||||
is that curvature carried through the derivatives of its atoms' U in the body's parameters. Each is calibrated
|
||||
once per trial against central differences of the gradient along a random $\pm1$ direction. The coupling of
|
||||
neighbouring atoms through the X-ray target is not in it; the pairs learn it.
|
||||
- **The anisotropic U restraints** are quadratic in U, so their Hessian is exact by one difference of their
|
||||
gradient.
|
||||
- **A trust floor per atom.** Where nothing but the X-ray target curves a mode - a high-B side chain turning about
|
||||
a single bond, which the restraints leave free and the X-ray target hardly sees - the matrix's inverse would
|
||||
make a step of tens of ångströms there, and the step bound would then shrink every other atom's step with it.
|
||||
So each atom's diagonal is raised, once per macro-cycle, to at least $|g_i|/\Delta$ ($g_i$ its gradient,
|
||||
$\Delta$ the step bound): its step is held by its own curvature, a trust region per atom in the manner of
|
||||
Levenberg and Marquardt's damping. A U (per atom) and a TLS body's parameters (each) take the same floor. Without it a rounding difference of $10^{-7}$ in the gradient moved a weight
|
||||
trial's R-free by up to 0.004; with it the perturbation stays where it was.
|
||||
- **The matrix is solved, not inverted.** Each L-BFGS direction solves the normal equations by a fixed 20
|
||||
conjugate-gradient iterations - not to a tolerance, whose iteration count, and with it the direction, a rounding
|
||||
difference can flip - preconditioned per residue for the positions (the residue's own $J^\mathsf{T}J$ and X-ray
|
||||
diagonal, by Cholesky: the stiff bonds and angles inside a residue are solved there, what couples residues is
|
||||
left to the iterations) and by the diagonal elsewhere. The Jacobian is taken again at each point the minimiser
|
||||
stands on.
|
||||
- **Its memory is kept from one macro-cycle to the next.** Each weight trial starts with an empty memory.
|
||||
|
||||
The diagonal is rescaled at each iteration by the latest pair (Oren-Shanno scaling in that metric). A solve
|
||||
stops after its iteration budget (30 for the positions and 20 for the B factors per macro-cycle) or where an
|
||||
iteration lowers the target by less than $10^{-7}$ of it.
|
||||
A solve stops after its iteration budget (10 for the positions and 7 for the B factors, U or TLS per
|
||||
macro-cycle) or where an iteration lowers the target by less than $10^{-7}$ of it. Against the plain L-BFGS this
|
||||
replaced (a diagonal initial inverse Hessian for the B factors, none for the positions, 30 and 20 iterations), a
|
||||
trial takes about a third of the evaluations. With it the population (§15.7) no longer gains, and the protocol
|
||||
is the same on the CPU and the GPU - two rounds, no population - at a third of the evaluations the GPU's protocol
|
||||
took before and R-free 0.0014 lower in the mean over 19 refinements, and at fewer evaluations than the CPU's
|
||||
former one-round protocol.
|
||||
|
||||
### 15.7 The weights, decided by trial
|
||||
|
||||
@@ -337,8 +362,10 @@ are:
|
||||
stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model,
|
||||
whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the
|
||||
trajectory ends where it began.
|
||||
3. **The population**, from where the trajectory ended: twelve members (at most as many as the trial slots, so
|
||||
that they run in one wave; none below two) refined side by side, each a trial at the
|
||||
3. **The population** (off by default, `RefinementSettings::population`: with the minimiser of §15.6 its members
|
||||
took nothing off R-free over 19 refinements that the second round did not), from where the trajectory ended:
|
||||
up to twelve members (at most as many as the trial slots, so that they run in one wave; none below two) refined
|
||||
side by side, each a trial at the
|
||||
trajectory's weights but on its own path - the positions as they are or kicked at random by up to 0.2-0.3 Å
|
||||
in each coordinate, on the data as they are or **blurred** along a schedule, 60 then 20 Ų (the landscape of
|
||||
4 Å, then of 2.2 Å), 100 then 30, or 30 then 10. The blur raises the likelihood's variance of each reflection
|
||||
@@ -741,7 +768,7 @@ What the refinement does not do yet - each a known gap, not a design choice:
|
||||
- L. Piela, J. Kostrowicki & H. A. Scheraga, "On the multiple-minima problem in the conformational analysis of molecules: deformation of the potential energy hypersurface by the diffusion equation method", *J. Phys. Chem.* **93** (1989), 3339-3346, and A. Jack & M. Levitt, "Refinement of large structures by simultaneous minimization of energy and R factor", *Acta Cryst.* **A34** (1978), 931-935 (the members refined on blurred data, §15.7).
|
||||
- J. Praznikar, P. V. Afonine, G. Guncar, P. D. Adams & D. Turk, "Averaged kick maps: less noise, more signal... and probably less bias", *Acta Cryst.* **D65** (2009), 921-931, and J. C. Nwachukwu, M. R. Southern, J. R. Kiefer, P. V. Afonine, P. D. Adams, T. C. Terwilliger & K. W. Nettles, "Improved crystallographic structures using extensive combinatorial refinement", *Structure* **21** (2013), 1923-1930 (the kicked members, their averaged map and the crossover residue by residue, §15.7).
|
||||
- J. Desmet, M. De Maeyer, B. Hazes & I. Lasters, "The dead-end elimination theorem and its use in protein side-chain positioning", *Nature* **356** (1992), 539-542; J. Besag, "On the statistical analysis of dirty pictures", *J. R. Statist. Soc.* **B48** (1986), 259-279; R. P. Joosten, K. Joosten, S. X. Cohen, G. Vriend & A. Perrakis, "Automatic rebuilding and optimization of crystallographic structures in the Protein Data Bank", *Bioinformatics* **27** (2011), 3392-3398 (the side-chain field: states, the conditional-modes solver and its order, §15.5).
|
||||
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8).
|
||||
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8; the minimiser's sparse normal matrix, §15.6).
|
||||
- G. N. Murshudov, P. Skubák, A. A. Lebedev, N. S. Pannu, R. A. Steiner, R. A. Nicholls, M. D. Winn, F. Long & A. A. Vagin, "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (D from the working set and the variance from the free, §15.3; the Kullback-Leibler form of the B restraints, §15.5; the twinned refinement §15.12 simplifies, its twin laws pruned by fraction and its fractions refined by least squares).
|
||||
- R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", *Acta Cryst.* **A42** (1986), 140-149 (the m, D weighting of the map coefficients, §15.8).
|
||||
- T. O. Yeates, "Detecting and overcoming crystal twinning", *Methods Enzymol.* **276** (1997), 344-358 (the twinned intensity, §15.12).
|
||||
@@ -762,6 +789,8 @@ What the refinement does not do yet - each a known gap, not a design choice:
|
||||
- Z. Dauter & M. Dauter, "Entering a new phase: using solvent halide ions in protein structure determination", *Structure* **9** (2001), R21-R26 (halides at sites with few donors, §15.10).
|
||||
- N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", *Acta Cryst.* **D70** (2014), 1104-1114 (a light ion flagged by its density above the waters', §15.10).
|
||||
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782, and D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15.6).
|
||||
- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6).
|
||||
- K. Levenberg, "A method for the solution of certain non-linear problems in least squares", *Q. Appl. Math.* **2** (1944), 164-168, and D. W. Marquardt, "An algorithm for least-squares estimation of nonlinear parameters", *J. Soc. Ind. Appl. Math.* **11** (1963), 431-441 (the minimiser's trust floor per atom, §15.6).
|
||||
- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
|
||||
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7).
|
||||
- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free).
|
||||
|
||||
@@ -37,6 +37,9 @@ ADD_LIBRARY(JFJochStructureRefinement STATIC
|
||||
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementGPU.cpp>
|
||||
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementGPU.cu>
|
||||
RefinementGPU.h RefinementGPUEngine.h
|
||||
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementNewtonGPU.cpp>
|
||||
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementNewtonGPU.cu>
|
||||
RefinementNewtonGPU.h
|
||||
${CMAKE_CURRENT_BINARY_DIR}/MonomerLibraryData.cpp)
|
||||
|
||||
# The subset of the CCP4 Monomer Library that refinement restrains with (monomers/, chosen by
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
#include <thread>
|
||||
#include <unordered_map>
|
||||
|
||||
#include <Eigen/Dense>
|
||||
#include <ceres/gradient_problem.h>
|
||||
#include <ceres/gradient_problem_solver.h>
|
||||
#include <ceres/jet.h>
|
||||
@@ -47,6 +48,7 @@
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
#include "../../common/CUDAWrapper.h"
|
||||
#include "RefinementGPU.h"
|
||||
#include "RefinementNewtonGPU.h"
|
||||
#endif
|
||||
|
||||
namespace {
|
||||
@@ -607,7 +609,12 @@ struct Context {
|
||||
// The X-ray target's curvature in each atom's B, up to one factor (CalibrateBCurvature): the likelihood's
|
||||
// curvature in |F_model| of the cycle, 2 (k_total D)^2 / var, summed in resolution shells, against the atom's own
|
||||
// part of it, (s^2 / 4)^2 f(s)^2 exp(-B s^2 / 2). A hydrogen's is its parent's.
|
||||
std::vector<double> XrayBCurvatureShape() const {
|
||||
std::vector<double> XrayBCurvatureShape() const { return XrayCurvatureShape(rows, false); }
|
||||
// The same for each of an atom's x, y and z: its own part of the likelihood's curvature in |F_model| is then
|
||||
// (2 pi)^2 s^2 / 3 f(s)^2 exp(-B s^2 / 2), the reflections taken as spread evenly over directions - the X-ray
|
||||
// block of the normal matrix as a multiple of I per atom. `r`: the row model the engine has (blurred or not).
|
||||
std::vector<double> XrayXyzCurvatureShape(const XrayRowModel &r) const { return XrayCurvatureShape(r, true); }
|
||||
std::vector<double> XrayCurvatureShape(const XrayRowModel &rows, bool xyz) const {
|
||||
constexpr int SHELLS = 32;
|
||||
double s2_max = 0;
|
||||
for (double s2 : refl.inv_d2)
|
||||
@@ -638,7 +645,10 @@ struct Context {
|
||||
for (int k = 0; k < SHELLS; k++)
|
||||
if (n[k] > 0) {
|
||||
const double s2 = s2_sum[k] / n[k];
|
||||
shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * s2 / 16.0;
|
||||
if (xyz)
|
||||
shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * (4.0 * M_PI * M_PI / 3.0);
|
||||
else
|
||||
shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * s2 / 16.0;
|
||||
}
|
||||
}
|
||||
for (size_t i = 0; i < rider_parent.size(); i++)
|
||||
@@ -675,6 +685,37 @@ struct Context {
|
||||
return den > 0 ? std::max(num / den, 0.0) : 0.0;
|
||||
}
|
||||
|
||||
// The factor that turns XrayXyzCurvatureShape() into the X-ray target's curvature, as CalibrateBCurvature() does
|
||||
// for B: along a random +-1 direction of the positions of `params`, by central differences of the gradient.
|
||||
// Two evaluations, counted.
|
||||
double CalibrateXyzCurvature(const std::vector<int> ¶ms, const std::vector<double> &shape) {
|
||||
constexpr double EPS = 0.01; // A
|
||||
const std::vector<gemmi::Position> p0 = Positions();
|
||||
std::mt19937 rng(7);
|
||||
std::vector<gemmi::Vec3> v(params.size());
|
||||
for (gemmi::Vec3 &x : v)
|
||||
x = gemmi::Vec3(rng() % 2 == 0 ? 1.0 : -1.0, rng() % 2 == 0 ? 1.0 : -1.0, rng() % 2 == 0 ? 1.0 : -1.0);
|
||||
std::array<std::vector<std::array<double, 4>>, 2> g;
|
||||
for (int side = 0; side < 2; side++) {
|
||||
for (size_t k = 0; k < params.size(); k++)
|
||||
atoms[params[k]]->pos = p0[params[k]] + gemmi::Position(v[k] * (side == 0 ? EPS : -EPS));
|
||||
MoveRiders();
|
||||
engine->Evaluate(Model(), &g[side]);
|
||||
evaluations++;
|
||||
FoldRiders(g[side]);
|
||||
}
|
||||
SetPositions(p0);
|
||||
MoveRiders();
|
||||
double num = 0, den = 0;
|
||||
for (size_t k = 0; k < params.size(); k++) {
|
||||
const int i = params[k];
|
||||
num += (v[k].x * (g[0][i][0] - g[1][i][0]) + v[k].y * (g[0][i][1] - g[1][i][1]) +
|
||||
v[k].z * (g[0][i][2] - g[1][i][2])) / (2 * EPS);
|
||||
den += 3 * shape[i];
|
||||
}
|
||||
return den > 0 ? std::max(num / den, 0.0) : 0.0;
|
||||
}
|
||||
|
||||
// F_model with the cycle's scale and solvent, for the atoms as they are now.
|
||||
std::vector<std::complex<float>> FModel() {
|
||||
engine->Fcalc(Model(), fcalc);
|
||||
@@ -792,6 +833,10 @@ public:
|
||||
return std::isfinite(*cost);
|
||||
}
|
||||
|
||||
Mode GetMode() const { return mode_; }
|
||||
double Weight() const { return weight_; }
|
||||
const std::vector<int> &Atoms() const { return param_atoms_; }
|
||||
|
||||
private:
|
||||
const SpecialPosition *Special(int atom) const {
|
||||
const int k = topo_.special_index.empty() ? -1 : topo_.special_index[atom];
|
||||
@@ -874,6 +919,20 @@ public:
|
||||
}
|
||||
}
|
||||
double MaxStep() const { return MAX_STEP_B; }
|
||||
// The parameter k of TLS body `body` over its value in the body's own units.
|
||||
double ParamScale(size_t body, int k) const { return Scale(body, k); }
|
||||
// Mode::U: the U restraints' gradient alone at p, as Evaluate() makes it (zero where there are none).
|
||||
void RestraintGradient(const double *p, double *gradient) const {
|
||||
Set(p);
|
||||
std::fill(gradient, gradient + 6 * param_atoms_.size(), 0.0);
|
||||
if (restraints_ == nullptr)
|
||||
return;
|
||||
std::vector<std::array<double, 6>> gr(ctx_.atoms.size(), {0, 0, 0, 0, 0, 0});
|
||||
EvaluateURestraints(*restraints_, ctx_.Positions(), ctx_.UOfAtoms(), &gr);
|
||||
for (size_t k = 0; k < param_atoms_.size(); k++)
|
||||
for (int c = 0; c < 6; c++)
|
||||
gradient[6 * k + c] = gr[param_atoms_[k]][c] / gemmi::u_to_b();
|
||||
}
|
||||
|
||||
bool Evaluate(const double *p, double *cost, double *gradient) const {
|
||||
const RefinementStage stage(mode_ == Mode::U ? "eval.u" : "eval.tls");
|
||||
@@ -934,40 +993,380 @@ private:
|
||||
std::vector<double> radius_;
|
||||
};
|
||||
|
||||
// The minimiser's diagonal for the B factors of `param_atoms` (RefinementMinimiser::SetDiagonal): weight times the
|
||||
// X-ray target's curvature in B plus the B restraints', each element at least a thousandth of the median. The
|
||||
// positions take none: their restraints' curvature is that of stiff bonds and angles, which move atoms together, so
|
||||
// as a diagonal it slows rather than scales the minimiser.
|
||||
std::vector<double> BDiagonal(const std::vector<double> &xray, const std::vector<double> &restraints,
|
||||
const std::vector<int> ¶m_atoms, double weight) {
|
||||
std::vector<double> h;
|
||||
for (int i : param_atoms)
|
||||
h.push_back(weight * xray[i] + restraints[i]);
|
||||
if (h.empty()) // no atom the dictionary describes
|
||||
return h;
|
||||
std::vector<double> sorted = h;
|
||||
// Each element at least a thousandth of the median (a parameter the target does not see still takes a finite step).
|
||||
std::vector<double> FlooredDiagonal(std::vector<double> d) {
|
||||
if (d.empty())
|
||||
return d;
|
||||
std::vector<double> sorted = d;
|
||||
std::nth_element(sorted.begin(), sorted.begin() + sorted.size() / 2, sorted.end());
|
||||
const double floor = std::max(1e-3 * sorted[sorted.size() / 2], 1e-12);
|
||||
for (double &v : h)
|
||||
for (double &v : d)
|
||||
v = std::max(v, floor);
|
||||
return h;
|
||||
return d;
|
||||
}
|
||||
|
||||
// `minimiser` carries its memory from the call before; `diagonal` is the curvature it scales the steps by (none: 1).
|
||||
// The minimisers' initial inverse Hessian (RefinementMinimiser::SetInverseHessian) is the Gauss-Newton normal matrix of
|
||||
// the target, as REFMAC refines on its sparse normal matrix: the X-ray target's curvature per atom on the diagonal and
|
||||
// the restraints' J^T J in full, solved for each direction by a fixed number of conjugate-gradient iterations - not to a
|
||||
// tolerance, whose iteration count, and with it the direction, a rounding difference can flip.
|
||||
// Following Murshudov, Vagin & Dodson (1997) Acta Cryst. D53, 240-255
|
||||
constexpr int NORMAL_CG_ITERATIONS = 20;
|
||||
|
||||
// `target` by `minimiser` (its memory from the call before) from where it is, H0 the inverse of the normal matrix: at(x, q)
|
||||
// takes the matrix at x (where the minimiser stands; q the vector H0 is about to be applied to), then hv = H v
|
||||
// (product) and z = M^-1 r for a preconditioner M of it (precondition).
|
||||
template <typename Target>
|
||||
void Minimise(Context &ctx, const Target &target, int iterations, RefinementMinimiser &minimiser,
|
||||
std::vector<double> diagonal = {}) {
|
||||
void MinimiseWithNormalMatrix(Context &ctx, const Target &target, RefinementMinimiser &minimiser, int iterations,
|
||||
const std::function<void(const std::vector<double> &, const std::vector<double> &)> &at,
|
||||
const std::function<void(const std::vector<double> &, std::vector<double> &)> &product,
|
||||
const std::function<void(const std::vector<double> &, std::vector<double> &)> &precondition) {
|
||||
std::vector<double> p = target.Get();
|
||||
if (p.empty())
|
||||
return;
|
||||
minimiser.SetDiagonal(diagonal.empty() ? std::vector<double>(p.size(), 1.0) : std::move(diagonal));
|
||||
minimiser.SetInverseHessian([&](const std::vector<double> &x, std::vector<double> &q) {
|
||||
at(x, q);
|
||||
std::vector<double> z;
|
||||
ConjugateGradients(product, precondition, q, z, NORMAL_CG_ITERATIONS, 0.0, ctx.nthreads);
|
||||
q.swap(z);
|
||||
});
|
||||
minimiser.Minimise([&](const std::vector<double> &x, double &f, std::vector<double> &g) {
|
||||
return target.Evaluate(x.data(), &f, g.data());
|
||||
},
|
||||
p, iterations, MINIMISER_F_TOLERANCE, target.MaxStep(), [&]() { return ctx.Cancelled(); });
|
||||
minimiser.SetInverseHessian(nullptr);
|
||||
target.Set(p.data()); // where the minimiser ended, not where its last line search probed
|
||||
}
|
||||
|
||||
// The restraints' Gauss-Newton normal matrix J^T J (RestraintGaussNewton) on the x y z or the B of `param_atoms`, at the
|
||||
// point last given to At(); an atom that is not a parameter (a rider, an atom held) drops out of it. With the engine on
|
||||
// a GPU, on its card (RestraintNormalMatrixGPU).
|
||||
class RestraintNormalMatrix {
|
||||
public:
|
||||
RestraintNormalMatrix(const Context &ctx, const std::vector<int> ¶m_atoms, bool xyz)
|
||||
: per_(xyz ? 3 : 1), c0_(xyz ? 0 : 3), n_params_(param_atoms.size()), nthreads_(ctx.nthreads),
|
||||
param_of_(ctx.atoms.size(), -1) {
|
||||
for (size_t k = 0; k < param_atoms.size(); k++)
|
||||
param_of_[param_atoms[k]] = static_cast<int>(k);
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
if (ctx.engine->OnGPU())
|
||||
gpu_ = std::make_unique<RestraintNormalMatrixGPU>(ctx.device, param_atoms, ctx.atoms.size(), xyz);
|
||||
#endif
|
||||
}
|
||||
|
||||
void At(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
|
||||
const RefinementStage stage("normal.jacobian");
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
if (gpu_)
|
||||
return gpu_->At(set, xyzb);
|
||||
#endif
|
||||
jac_ = RestraintGaussNewton(set, xyzb);
|
||||
const size_t rows = jac_.start.size() - 1;
|
||||
row_of_.resize(jac_.atom.size());
|
||||
param_.resize(jac_.atom.size());
|
||||
for (size_t e = 0; e < jac_.atom.size(); e++)
|
||||
param_[e] = param_of_[jac_.atom[e]];
|
||||
for (size_t r = 0; r < rows; r++)
|
||||
for (int e = jac_.start[r]; e < jac_.start[r + 1]; e++)
|
||||
row_of_[e] = static_cast<int>(r);
|
||||
// per parameter atom its entries (CSR), for the transpose as a gather
|
||||
entry_start_.assign(n_params_ + 1, 0);
|
||||
for (int a : jac_.atom)
|
||||
if (param_of_[a] >= 0)
|
||||
entry_start_[param_of_[a] + 1]++;
|
||||
for (size_t k = 0; k < n_params_; k++)
|
||||
entry_start_[k + 1] += entry_start_[k];
|
||||
entry_.resize(entry_start_.back());
|
||||
std::vector<int> fill(entry_start_.begin(), entry_start_.end() - 1);
|
||||
for (size_t e = 0; e < jac_.atom.size(); e++)
|
||||
if (const int k = param_of_[jac_.atom[e]]; k >= 0)
|
||||
entry_[fill[k]++] = static_cast<int>(e);
|
||||
t_.resize(rows);
|
||||
}
|
||||
|
||||
// hv = J^T J v: J v per row, then per parameter atom the sum over its entries
|
||||
void Multiply(const std::vector<double> &v, std::vector<double> &hv) {
|
||||
const RefinementStage stage("normal.product");
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
if (gpu_)
|
||||
return gpu_->Multiply(v, hv);
|
||||
#endif
|
||||
const int rows = static_cast<int>(t_.size());
|
||||
ParallelChunks(rows, ThreadsForWork(rows, nthreads_, 16384), [&](int lo, int hi) {
|
||||
for (int r = lo; r < hi; r++) {
|
||||
double dot = 0;
|
||||
for (int e = jac_.start[r]; e < jac_.start[r + 1]; e++)
|
||||
if (const int k = param_[e]; k >= 0)
|
||||
for (int c = 0; c < per_; c++)
|
||||
dot += jac_.j[e][c0_ + c] * v[per_ * k + c];
|
||||
t_[r] = dot;
|
||||
}
|
||||
});
|
||||
hv.assign(per_ * n_params_, 0.0);
|
||||
const int n = static_cast<int>(n_params_);
|
||||
ParallelChunks(n, ThreadsForWork(n, nthreads_, 16384), [&](int lo, int hi) {
|
||||
for (int k = lo; k < hi; k++)
|
||||
for (int q = entry_start_[k]; q < entry_start_[k + 1]; q++) {
|
||||
const int e = entry_[q];
|
||||
for (int c = 0; c < per_; c++)
|
||||
hv[per_ * k + c] += t_[row_of_[e]] * jac_.j[e][c0_ + c];
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
std::vector<double> Diagonal() const {
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
if (gpu_)
|
||||
return gpu_->Diagonal();
|
||||
#endif
|
||||
std::vector<double> d(per_ * n_params_, 0.0);
|
||||
for (size_t k = 0; k < n_params_; k++)
|
||||
for (int q = entry_start_[k]; q < entry_start_[k + 1]; q++)
|
||||
for (int c = 0; c < per_; c++)
|
||||
d[per_ * k + c] += jac_.j[entry_[q]][c0_ + c] * jac_.j[entry_[q]][c0_ + c];
|
||||
return d;
|
||||
}
|
||||
|
||||
private:
|
||||
int per_, c0_;
|
||||
size_t n_params_, nthreads_;
|
||||
std::vector<int> param_of_;
|
||||
RestraintJacobian jac_;
|
||||
std::vector<int> row_of_, param_, entry_start_, entry_; // per entry its row and its parameter atom (-1: none)
|
||||
std::vector<double> t_;
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
std::unique_ptr<RestraintNormalMatrixGPU> gpu_;
|
||||
#endif
|
||||
};
|
||||
|
||||
// The positions' preconditioner: per residue the dense block of H over its atoms' x y z - the restraints within the
|
||||
// residue (J^T J of the rows whose atoms it holds, at the point given) and the X-ray diagonal `h` - by its Cholesky
|
||||
// factor. What couples residues (peptide bonds, NCS, contacts) is left to the conjugate gradients; the stiff bonds
|
||||
// and angles within a residue, which a diagonal or a block per atom leaves to them too, are solved here.
|
||||
class ResidueBlocks {
|
||||
public:
|
||||
ResidueBlocks(Context &ctx, const std::vector<int> ¶m_atoms, const RestraintSet &set, const std::vector<double> &h) {
|
||||
std::vector<int> residue_of; // per atom, in model order
|
||||
for (const gemmi::Chain &ch : ctx.Model().chains)
|
||||
for (const gemmi::Residue &r : ch.residues) {
|
||||
residue_of.insert(residue_of.end(), r.atoms.size(), static_cast<int>(params_.size()));
|
||||
params_.emplace_back();
|
||||
}
|
||||
std::vector<int> local(param_atoms.size()), group(param_atoms.size()), param_of(ctx.atoms.size(), -1);
|
||||
for (size_t k = 0; k < param_atoms.size(); k++) {
|
||||
group[k] = residue_of[param_atoms[k]];
|
||||
local[k] = static_cast<int>(params_[group[k]].size());
|
||||
params_[group[k]].push_back(static_cast<int>(k));
|
||||
param_of[param_atoms[k]] = static_cast<int>(k);
|
||||
}
|
||||
std::vector<Eigen::MatrixXd> block(params_.size());
|
||||
for (size_t g = 0; g < params_.size(); g++) {
|
||||
const int m = 3 * static_cast<int>(params_[g].size());
|
||||
block[g] = Eigen::MatrixXd::Zero(m, m);
|
||||
for (size_t l = 0; l < params_[g].size(); l++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
block[g](3 * l + c, 3 * l + c) = h[3 * params_[g][l] + c];
|
||||
}
|
||||
const RestraintJacobian jac = RestraintGaussNewton(set, ctx.XyzB());
|
||||
for (size_t row = 0; row + 1 < jac.start.size(); row++)
|
||||
for (int e1 = jac.start[row]; e1 < jac.start[row + 1]; e1++)
|
||||
for (int e2 = jac.start[row]; e2 < jac.start[row + 1]; e2++) {
|
||||
const int k1 = param_of[jac.atom[e1]], k2 = param_of[jac.atom[e2]];
|
||||
if (k1 < 0 || k2 < 0 || group[k1] != group[k2])
|
||||
continue;
|
||||
Eigen::MatrixXd &b = block[group[k1]];
|
||||
for (int a = 0; a < 3; a++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
b(3 * local[k1] + a, 3 * local[k2] + c) += jac.j[e1][a] * jac.j[e2][c];
|
||||
}
|
||||
llt_.resize(params_.size());
|
||||
ParallelFor(static_cast<int>(params_.size()), ctx.nthreads, [&](int g) { llt_[g].compute(block[g]); });
|
||||
}
|
||||
|
||||
void Apply(const std::vector<double> &r, std::vector<double> &z, size_t nthreads) const {
|
||||
z.resize(r.size());
|
||||
ParallelFor(static_cast<int>(params_.size()), nthreads, [&](int g) {
|
||||
const std::vector<int> &p = params_[g];
|
||||
Eigen::VectorXd rg(3 * p.size());
|
||||
for (size_t l = 0; l < p.size(); l++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
rg(3 * l + c) = r[3 * p[l] + c];
|
||||
const Eigen::VectorXd zg = llt_[g].solve(rg);
|
||||
for (size_t l = 0; l < p.size(); l++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
z[3 * p[l] + c] = zg(3 * l + c);
|
||||
});
|
||||
}
|
||||
|
||||
private:
|
||||
std::vector<std::vector<int>> params_; // per residue its parameter atoms (their index among the parameters)
|
||||
std::vector<Eigen::LLT<Eigen::MatrixXd>> llt_;
|
||||
};
|
||||
|
||||
// The positions or the B factors: H = diag(xray) (per atom, already times the weight) + the restraints' J^T J of `set`,
|
||||
// the Jacobian at the point of each direction. The diagonal is raised per atom, once per call, to at least |g_i| over the
|
||||
// step bound (g_i the atom's part of the first direction's gradient): where nothing else curves the target (a high-B
|
||||
// side chain turning about a single bond, which the restraints leave free and the X-ray target hardly sees) the step
|
||||
// is then held to the bound by the atom's own curvature, instead of one such atom scaling down the whole step.
|
||||
// Preconditioned per residue for the positions (ResidueBlocks), by the diagonal for the B factors.
|
||||
// The floor a trust region per atom, after Levenberg (1944) Q. Appl. Math. 2, 164-168 and Marquardt (1963) J. SIAM
|
||||
// 11, 431-441
|
||||
void MinimiseAtoms(Context &ctx, const AtomTarget &target, RefinementMinimiser &minimiser, const std::vector<double> &xray,
|
||||
const RestraintSet &set, int iterations) {
|
||||
const bool xyz = target.GetMode() == AtomTarget::Mode::Xyz;
|
||||
const int per = xyz ? 3 : 1;
|
||||
std::vector<double> x_at = target.Get();
|
||||
const size_t n = x_at.size();
|
||||
std::vector<double> h(n);
|
||||
for (size_t k = 0; k < n; k++)
|
||||
h[k] = xray[target.Atoms()[k / per]];
|
||||
RestraintNormalMatrix normal(ctx, target.Atoms(), xyz);
|
||||
target.Set(x_at.data());
|
||||
normal.At(set, ctx.XyzB());
|
||||
const std::vector<double> restraint_diagonal = normal.Diagonal();
|
||||
std::vector<double> diagonal;
|
||||
std::unique_ptr<ResidueBlocks> residues;
|
||||
auto at = [&](const std::vector<double> &x, const std::vector<double> &q) {
|
||||
if (diagonal.empty()) { // the first direction: the floor, then the preconditioner with it
|
||||
for (size_t a = 0; a < n / per; a++) {
|
||||
double g2 = 0;
|
||||
for (int c = 0; c < per; c++)
|
||||
g2 += q[per * a + c] * q[per * a + c];
|
||||
for (int c = 0; c < per; c++)
|
||||
h[per * a + c] = std::max(h[per * a + c], std::sqrt(g2) / target.MaxStep());
|
||||
}
|
||||
diagonal.resize(n);
|
||||
for (size_t k = 0; k < n; k++)
|
||||
diagonal[k] = restraint_diagonal[k] + h[k];
|
||||
diagonal = FlooredDiagonal(diagonal);
|
||||
if (xyz)
|
||||
residues = std::make_unique<ResidueBlocks>(ctx, target.Atoms(), set, h);
|
||||
}
|
||||
if (x != x_at) {
|
||||
x_at = x;
|
||||
target.Set(x.data());
|
||||
normal.At(set, ctx.XyzB());
|
||||
}
|
||||
};
|
||||
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
|
||||
normal.Multiply(v, hv);
|
||||
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] += h[k] * v[k]; });
|
||||
};
|
||||
auto precondition = [&](const std::vector<double> &r, std::vector<double> &z) {
|
||||
if (residues)
|
||||
residues->Apply(r, z, ctx.nthreads);
|
||||
else {
|
||||
z.resize(r.size());
|
||||
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { z[k] = r[k] / diagonal[k]; });
|
||||
}
|
||||
};
|
||||
MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition);
|
||||
}
|
||||
|
||||
// Each atom's U (in 8 pi^2 U): H = diag(xray) per element + the U restraints' Hessian, exact by differences of their
|
||||
// gradient (they are quadratic in U); the diagonal raised to the trust floor as MinimiseAtoms() raises it;
|
||||
// preconditioned by `diagonal`.
|
||||
void MinimiseU(Context &ctx, const AdpTarget &target, RefinementMinimiser &minimiser, std::vector<double> xray,
|
||||
std::vector<double> diagonal, int iterations) {
|
||||
const size_t n = xray.size();
|
||||
std::vector<double> x_at, g_at(n), xp(n), gp(n);
|
||||
bool first = true;
|
||||
auto at = [&](const std::vector<double> &x, const std::vector<double> &q) {
|
||||
if (first) { // the trust floor per atom, as MinimiseAtoms() takes it
|
||||
first = false;
|
||||
for (size_t a = 0; a < n / 6; a++) {
|
||||
double g2 = 0;
|
||||
for (int c = 0; c < 6; c++)
|
||||
g2 += q[6 * a + c] * q[6 * a + c];
|
||||
const double floor = std::sqrt(g2) / target.MaxStep();
|
||||
for (int c = 0; c < 6; c++) {
|
||||
diagonal[6 * a + c] += std::max(floor - xray[6 * a + c], 0.0);
|
||||
xray[6 * a + c] = std::max(xray[6 * a + c], floor);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (x != x_at) {
|
||||
x_at = x;
|
||||
target.RestraintGradient(x.data(), g_at.data());
|
||||
}
|
||||
};
|
||||
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
|
||||
double v_max = 0;
|
||||
for (double e : v)
|
||||
v_max = std::max(v_max, std::fabs(e));
|
||||
hv.assign(n, 0.0);
|
||||
if (v_max > 0) {
|
||||
const double t = 1.0 / v_max;
|
||||
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { xp[k] = x_at[k] + t * v[k]; });
|
||||
target.RestraintGradient(xp.data(), gp.data());
|
||||
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] = (gp[k] - g_at[k]) / t; });
|
||||
}
|
||||
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] += xray[k] * v[k]; });
|
||||
};
|
||||
auto precondition = [&](const std::vector<double> &r, std::vector<double> &z) {
|
||||
z.resize(n);
|
||||
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { z[k] = r[k] / diagonal[k]; });
|
||||
};
|
||||
MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition);
|
||||
}
|
||||
|
||||
// The TLS bodies: X-ray only, H = sum over each body's atoms of A^T C A, A the derivatives of the atom's U in the body's
|
||||
// parameters (linear in them), C the X-ray curvature of its U per element (`xray_b`: of its B, a third of it to each);
|
||||
// one dense block per body, its diagonal raised to the trust floor, preconditioned by its diagonal.
|
||||
void MinimiseTls(Context &ctx, const AdpTarget &target, RefinementMinimiser &minimiser, const std::vector<double> &xray_b,
|
||||
int iterations) {
|
||||
constexpr int P = TlsBody::PARAMETERS;
|
||||
const size_t nb = ctx.tls.size();
|
||||
std::vector<std::array<double, P * P>> blocks(nb);
|
||||
for (size_t b = 0; b < nb; b++) {
|
||||
std::array<double, P * P> &m = blocks[b];
|
||||
m.fill(0.0);
|
||||
for (int i : ctx.tls[b].atoms) {
|
||||
if (!ctx.rider_parent.empty() && ctx.rider_parent[i] >= 0)
|
||||
continue; // its curvature is its parent's already
|
||||
const std::array<std::array<double, 6>, P> a = ctx.tls[b].Derivatives(ctx.atoms[i]->pos);
|
||||
const double c = gemmi::u_to_b() * gemmi::u_to_b() * xray_b[i] / 3.0;
|
||||
for (int k = 0; k < P; k++)
|
||||
for (int l = 0; l < P; l++) {
|
||||
double s = 0;
|
||||
for (int e = 0; e < 6; e++)
|
||||
s += a[k][e] * a[l][e];
|
||||
m[k * P + l] += c * s / (target.ParamScale(b, k) * target.ParamScale(b, l));
|
||||
}
|
||||
}
|
||||
}
|
||||
std::vector<double> diagonal(nb * P);
|
||||
for (size_t b = 0; b < nb; b++)
|
||||
for (int k = 0; k < P; k++)
|
||||
diagonal[b * P + k] = blocks[b][k * P + k];
|
||||
diagonal = FlooredDiagonal(diagonal);
|
||||
// the trust floor per parameter, as MinimiseAtoms() takes it per atom
|
||||
bool first = true;
|
||||
auto at = [&](const std::vector<double> &, const std::vector<double> &q) {
|
||||
if (!first)
|
||||
return;
|
||||
first = false;
|
||||
for (size_t b = 0; b < nb; b++)
|
||||
for (int k = 0; k < P; k++) {
|
||||
const double floor = std::fabs(q[b * P + k]) / target.MaxStep();
|
||||
diagonal[b * P + k] += std::max(floor - blocks[b][k * P + k], 0.0);
|
||||
blocks[b][k * P + k] = std::max(blocks[b][k * P + k], floor);
|
||||
}
|
||||
};
|
||||
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
|
||||
hv.assign(v.size(), 0.0);
|
||||
for (size_t b = 0; b < nb; b++)
|
||||
for (int k = 0; k < P; k++)
|
||||
for (int l = 0; l < P; l++)
|
||||
hv[b * P + k] += blocks[b][k * P + l] * v[b * P + l];
|
||||
};
|
||||
auto precondition = [&](const std::vector<double> &r, std::vector<double> &z) {
|
||||
z.resize(r.size());
|
||||
for (size_t k = 0; k < r.size(); k++)
|
||||
z[k] = r[k] / diagonal[k];
|
||||
};
|
||||
MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition);
|
||||
}
|
||||
|
||||
// Groups of atoms that move as rigid bodies (one per chain) against the X-ray target: per body an angle-axis
|
||||
// rotation about its centroid, scaled by its rms radius so that a unit moves a typical atom by an angstrom as a
|
||||
// unit of translation does, then a translation.
|
||||
@@ -1788,6 +2187,18 @@ double XrayGradientNorm(Context &ctx, const std::vector<int> &refinable) {
|
||||
return n;
|
||||
}
|
||||
|
||||
// The X-ray curvature per atom in each of x, y, z at weight w, for the row model `rows` the engine has; `scale`
|
||||
// calibrated at the first call of the trial (negative: not yet).
|
||||
std::vector<double> XyzCurvature(Context &ctx, const std::vector<int> &refinable, const XrayRowModel &rows, double w,
|
||||
double &scale) {
|
||||
std::vector<double> shape = ctx.XrayXyzCurvatureShape(rows);
|
||||
if (scale < 0)
|
||||
scale = ctx.CalibrateXyzCurvature(refinable, shape);
|
||||
for (double &x : shape)
|
||||
x *= w * scale;
|
||||
return shape;
|
||||
}
|
||||
|
||||
TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const std::vector<gemmi::Position> &pos0,
|
||||
const std::vector<double> &b0, const std::vector<AtomAniso> &aniso0, const Context::Fit *fit,
|
||||
const TrialWeights &tw, int cycles, const RefinementSettings &settings, const NcsCopies *ncs,
|
||||
@@ -1838,7 +2249,8 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
|
||||
const AdpTarget tls_target(ctx, AdpTarget::Mode::Tls, refinable, 1.0, nullptr);
|
||||
const AdpTarget u_target(ctx, AdpTarget::Mode::U, u_atoms, adp_w, &u_restraints);
|
||||
RefinementMinimiser xyz_minimiser(20, ctx.nthreads), b_minimiser(20, ctx.nthreads), tls_minimiser(20, ctx.nthreads);
|
||||
double xray_scale = 0;
|
||||
// the X-ray curvature's calibrations (CalibrateXyzCurvature, CalibrateBCurvature), once per trial
|
||||
double xyz_scale = -1, b_scale = -1;
|
||||
for (int cycle = 0; cycle < cycles && !ctx.Cancelled(); cycle++) {
|
||||
PlaceRiders(ctx, topo);
|
||||
if (cycle == 0 && fit != nullptr)
|
||||
@@ -1852,8 +2264,8 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
|
||||
if (adp_w > 0)
|
||||
ctx.MoveOverallB();
|
||||
if (!hold_positions) {
|
||||
const RestraintSet set = RestraintsNow(ctx, topo, ncs, ncs_sigma);
|
||||
{
|
||||
const RestraintSet set = RestraintsNow(ctx, topo, ncs, ncs_sigma);
|
||||
const RefinementStage s1("set_restraints");
|
||||
ctx.engine->SetRestraints(0, set);
|
||||
}
|
||||
@@ -1870,24 +2282,33 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
|
||||
ctx.engine->SetRowModel(blurred);
|
||||
const double nh = XrayGradientNorm(ctx, refinable);
|
||||
const AtomTarget xyz_h(ctx, AtomTarget::Mode::Xyz, refinable, nh > 0 ? w * std::sqrt(n0 / nh) : w, 0, topo);
|
||||
Minimise(ctx, xyz_h, settings.xyz_iterations, xyz_minimiser);
|
||||
MinimiseAtoms(ctx, xyz_h, xyz_minimiser, XyzCurvature(ctx, refinable, blurred, xyz_h.Weight(), xyz_scale),
|
||||
set, settings.xyz_iterations);
|
||||
ctx.engine->SetRowModel(ctx.rows);
|
||||
} else
|
||||
Minimise(ctx, xyz, settings.xyz_iterations, xyz_minimiser);
|
||||
MinimiseAtoms(ctx, xyz, xyz_minimiser, XyzCurvature(ctx, refinable, ctx.rows, w, xyz_scale), set,
|
||||
settings.xyz_iterations);
|
||||
}
|
||||
// the X-ray curvature in B for the atoms as they are, which TLS, U and B take a share of
|
||||
auto b_curvature = [&]() {
|
||||
std::vector<double> c = ctx.XrayBCurvatureShape();
|
||||
if (b_scale < 0)
|
||||
b_scale = ctx.CalibrateBCurvature(refinable, c);
|
||||
for (double &x : c)
|
||||
x *= b_scale;
|
||||
return c;
|
||||
};
|
||||
if (kind == AdpHypothesis::Kind::Tls) {
|
||||
ctx.ApplyTls(); // the bodies' U at the atoms as they are
|
||||
const RefinementStage s4("minimise.tls");
|
||||
Minimise(ctx, tls_target, settings.b_iterations, tls_minimiser);
|
||||
MinimiseTls(ctx, tls_target, tls_minimiser, b_curvature(), settings.b_iterations);
|
||||
}
|
||||
if (adp_w > 0 && kind == AdpHypothesis::Kind::Anisotropic) {
|
||||
// each U's six elements take a third of the curvature of its B
|
||||
std::vector<double> xray = ctx.XrayBCurvatureShape();
|
||||
if (cycle == 0)
|
||||
xray_scale = ctx.CalibrateBCurvature(refinable, xray);
|
||||
// and the restraints' curvature in 8 pi^2 U: per pair 2 / sigma^2 of the similarity and about a third of
|
||||
// the rigid bond's, per isotropic atom 2 / sigma^2
|
||||
std::vector<double> restraint(xray.size(), 0.0);
|
||||
const std::vector<double> xray_b = b_curvature();
|
||||
// each U's six elements take a third of the curvature of its B; and the restraints' curvature in 8 pi^2 U:
|
||||
// per pair 2 / sigma^2 of the similarity and about a third of the rigid bond's, per isotropic atom
|
||||
// 2 / sigma^2 - the preconditioner's diagonal
|
||||
std::vector<double> restraint(xray_b.size(), 0.0);
|
||||
const double u2 = gemmi::u_to_b() * gemmi::u_to_b();
|
||||
const double per_pair = (2.0 / (URestraints::SIGMA_SIMILAR * URestraints::SIGMA_SIMILAR) +
|
||||
2.0 / (3.0 * URestraints::SIGMA_RIGID * URestraints::SIGMA_RIGID)) / u2;
|
||||
@@ -1897,24 +2318,21 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
|
||||
}
|
||||
for (int i : u_restraints.isotropic)
|
||||
restraint[i] += 2.0 / (URestraints::SIGMA_ISOTROPIC * URestraints::SIGMA_ISOTROPIC) / u2;
|
||||
for (double &x : xray)
|
||||
x *= xray_scale / 3.0;
|
||||
std::vector<double> diagonal;
|
||||
for (double h : BDiagonal(xray, restraint, u_atoms, adp_w))
|
||||
diagonal.insert(diagonal.end(), 6, h);
|
||||
std::vector<double> h, diagonal;
|
||||
for (int i : u_atoms) {
|
||||
h.insert(h.end(), 6, adp_w * xray_b[i] / 3.0);
|
||||
diagonal.insert(diagonal.end(), 6, adp_w * xray_b[i] / 3.0 + restraint[i]);
|
||||
}
|
||||
const RefinementStage s3("minimise.u");
|
||||
Minimise(ctx, u_target, settings.b_iterations, b_minimiser, diagonal);
|
||||
MinimiseU(ctx, u_target, b_minimiser, h, FlooredDiagonal(diagonal), settings.b_iterations);
|
||||
} else if (adp_w > 0) {
|
||||
const RestraintSet b_set = BRestraintsNow(ctx, topo);
|
||||
ctx.engine->SetRestraints(1, b_set);
|
||||
std::vector<double> xray = ctx.XrayBCurvatureShape();
|
||||
if (cycle == 0)
|
||||
xray_scale = ctx.CalibrateBCurvature(refinable, xray);
|
||||
for (double &x : xray)
|
||||
x *= xray_scale;
|
||||
std::vector<double> xray_b = b_curvature();
|
||||
for (double &x : xray_b)
|
||||
x *= adp_w;
|
||||
const RefinementStage s3("minimise.b");
|
||||
Minimise(ctx, b, settings.b_iterations, b_minimiser,
|
||||
BDiagonal(xray, BRestraintCurvature(b_set, ctx.XyzB()), refinable, adp_w));
|
||||
MinimiseAtoms(ctx, b, b_minimiser, xray_b, b_set, settings.b_iterations);
|
||||
}
|
||||
}
|
||||
PlaceRiders(ctx, topo);
|
||||
@@ -3050,7 +3468,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// the members, whose best on the free set would be the best of N.
|
||||
// The members are at most as many as the trial slots, one wave: on a large model, where the card holds
|
||||
// few, more members would cost a wave each.
|
||||
const size_t population = std::min<size_t>(settings.population, trial_slots.size());
|
||||
const size_t population = std::min<size_t>(std::max(settings.population, 0), trial_slots.size());
|
||||
if (population > 1 && state.ran && !ctx.Cancelled()) {
|
||||
const RefinementStage pop_stage("population");
|
||||
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
|
||||
@@ -4236,17 +4654,11 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
|
||||
"reflections in one set - {} reflections changed set", moved);
|
||||
}
|
||||
const std::vector<MergedReflection> &data = moved > 0 ? closed : merged;
|
||||
// RefinementSettings::lean_on_cpu
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
const bool on_cpu = !settings.gpu || get_gpu_count() == 0;
|
||||
#else
|
||||
const bool on_cpu = true;
|
||||
#endif
|
||||
if (settings.lean_on_cpu && on_cpu) {
|
||||
settings.population = 0;
|
||||
settings.rounds = 1;
|
||||
logger.Info("Refinement on the CPU: no population and one round (half of the evaluations)");
|
||||
}
|
||||
// Each arm, untwinned and twinned, is a whole refinement - the solvent built and the displacement model chosen
|
||||
// against its own target - and the decision is taken on where they ended. A model only scored is scored both
|
||||
// ways and decided by the same rule, so the R-free it reports is not an untwinned one by default.
|
||||
@@ -4260,7 +4672,10 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
|
||||
if (!settings.twin)
|
||||
result = RefineOnce(placed, data, cell, sg, wavelength_A, logger, settings);
|
||||
else {
|
||||
const bool halves = on_cpu && settings.population <= 1 && settings.nthreads > 1;
|
||||
// on the CPU each arm on half of the threads - where the lattice admits a twin law, without which the twinned
|
||||
// arm ends at once and its half would sit idle
|
||||
const bool halves = on_cpu && settings.population <= 1 && settings.nthreads > 1 &&
|
||||
!RefinementTwinLaws(cell, sg).empty();
|
||||
RefinementSettings untwinned_settings = settings, twin_settings = settings;
|
||||
if (halves) {
|
||||
twin_settings.nthreads = settings.nthreads / 2;
|
||||
|
||||
@@ -48,8 +48,10 @@ struct RefinementSettings {
|
||||
bool score_only = false; // only score the model as given (RefinementResult::score_start), no stage
|
||||
bool group_b = true; // the placement stage's B shift per residue
|
||||
int macro_cycles = 2; // per weight trial, from the model the trial starts from
|
||||
int xyz_iterations = 30; // L-BFGS iterations per macro-cycle
|
||||
int b_iterations = 20;
|
||||
// L-BFGS iterations per macro-cycle (on the Gauss-Newton normal matrix: RefinementMinimiser), of the positions and of
|
||||
// the B factors, U and TLS
|
||||
int xyz_iterations = 10;
|
||||
int b_iterations = 7;
|
||||
// The first trials, from the placed model: the positions' X-ray weight, times the gradient-balanced one
|
||||
// (BalancedWeight), each with the B factors' X-ray weight their own balanced one times each of adp_ladder (0: B
|
||||
// held).
|
||||
@@ -84,8 +86,9 @@ struct RefinementSettings {
|
||||
// The population (RefineOnce, 2a.): this many members at most, refined from where the weight trajectory ended in
|
||||
// one wave, each on the data blurred along a schedule of its own and its positions kicked or not, then crossed
|
||||
// residue by residue on their averaged map and refined once more; the child kept where the free set prefers it.
|
||||
// Capped at the trial slots; fewer than two: no population.
|
||||
int population = 12;
|
||||
// Capped at the trial slots; fewer than two: no population (the default: with the positions minimised on the
|
||||
// Gauss-Newton matrix, the members took nothing off R-free over 19 sets that the two rounds did not).
|
||||
int population = 0;
|
||||
// A refinement against the data as twinned by each twin law the lattice admits (RefinementTwinLaws), beside the
|
||||
// untwinned one, and the model of whichever the evidence favours (RefineModel).
|
||||
bool twin = true;
|
||||
@@ -97,11 +100,6 @@ struct RefinementSettings {
|
||||
// model of this crystal scored on a free set drawn here).
|
||||
bool as_given_competes = true;
|
||||
bool gpu = true;
|
||||
// Where the refinement runs on the CPU (no GPU wanted, or none to run on), where an evaluation costs some thirty
|
||||
// times a GPU's: no population and one round, for about half of the evaluations, at an R-free 0.001 higher in the
|
||||
// median of 19 sets (at most 0.015, a placed prediction the population helps). The twinned data are still refined
|
||||
// against: without that arm a crystal 30% twinned is refined as untwinned.
|
||||
bool lean_on_cpu = true;
|
||||
size_t nthreads = 1;
|
||||
size_t gpu_resident_bytes = 0; // held on the card by what runs beside
|
||||
// The most weight trials run side by side, 0 for as many as the ladders and the cards take; the result does not
|
||||
|
||||
@@ -11,29 +11,58 @@
|
||||
|
||||
#include "../../common/ParallelFor.h"
|
||||
|
||||
// Limited-memory BFGS whose memory outlives one call, so that a refinement's macro-cycles - each a few dozen
|
||||
// iterations on a target whose scale and likelihood parameters were refitted in between - go on from what the
|
||||
// earlier ones learnt of the curvature instead of starting each from a steepest-descent step. The initial inverse
|
||||
// Hessian is a diagonal the caller gives (SetDiagonal: an estimate of the Hessian's diagonal, so that a parameter
|
||||
// with ten times the curvature of another takes a tenth of its step), scaled each iteration by the latest pair
|
||||
// (Oren-Shanno in that metric). The line search backtracks from the full step to sufficient decrease.
|
||||
// Sweeps over the parameters on `nthreads` threads: each element on its own, and dot products folded in fixed
|
||||
// blocks (ParallelBlocks), so that the bits do not depend on the number of threads.
|
||||
namespace refine_vec {
|
||||
// a pass over fewer than this many parameters per thread costs less than waking the threads
|
||||
constexpr size_t SWEEP_PER_THREAD = 65536;
|
||||
template <typename Fn>
|
||||
void Each(size_t n, size_t nthreads, Fn fn) {
|
||||
ParallelChunks(static_cast<int>(n), ThreadsForWork(n, nthreads, SWEEP_PER_THREAD), [&](int lo, int hi) {
|
||||
for (int k = lo; k < hi; k++)
|
||||
fn(static_cast<size_t>(k));
|
||||
});
|
||||
}
|
||||
inline double Dot(const std::vector<double> &a, const std::vector<double> &b, size_t nthreads) {
|
||||
const int n = static_cast<int>(a.size());
|
||||
std::vector<double> part(ReductionBlocks(n, SWEEP_PER_THREAD), 0.0);
|
||||
ParallelBlocks(n, nthreads, [&](int block, int lo, int hi) {
|
||||
double s = 0;
|
||||
for (int k = lo; k < hi; k++)
|
||||
s += a[k] * b[k];
|
||||
part[block] = s;
|
||||
}, SWEEP_PER_THREAD);
|
||||
double s = 0;
|
||||
for (double v : part)
|
||||
s += v;
|
||||
return s;
|
||||
}
|
||||
} // namespace refine_vec
|
||||
|
||||
// Limited-memory BFGS whose initial inverse Hessian is the caller's (SetInverseHessian: H0 q, here the Gauss-Newton
|
||||
// normal matrix solved by ConjugateGradients), and whose memory outlives one call, so that a refinement's
|
||||
// macro-cycles - each a few iterations on a target whose scale and likelihood parameters were refitted in between -
|
||||
// go on from what the earlier ones learnt. The normal matrix holds what is known of the curvature in advance (the
|
||||
// restraints exactly, the X-ray target per atom); the pairs add what it leaves out (the X-ray target's coupling of
|
||||
// neighbouring atoms). The line search backtracks from the full step to sufficient decrease.
|
||||
// Following Nocedal (1980) Math. Comp. 35, 773-782 and Liu & Nocedal (1989) Math. Program. 45, 503-528
|
||||
class RefinementMinimiser {
|
||||
public:
|
||||
// f(x, g): the target at x, and its gradient in g (sized as x); false where it cannot be evaluated there.
|
||||
using Target = std::function<bool(const std::vector<double> &x, double &f, std::vector<double> &g)>;
|
||||
// q = H0 q at x.
|
||||
using InverseHessian = std::function<void(const std::vector<double> &x, std::vector<double> &q)>;
|
||||
|
||||
// `nthreads`: the threads the sweeps over the parameters run on (the dot products stay serial chains).
|
||||
// `nthreads`: the threads the sweeps over the parameters run on.
|
||||
explicit RefinementMinimiser(size_t memory = 20, size_t nthreads = 1) : memory_(memory), nthreads_(nthreads) {}
|
||||
|
||||
// Forget the pairs (the target changed beyond what the memory still describes, e.g. another weight).
|
||||
// Forget the pairs (the target changed beyond what the memory still describes).
|
||||
void Reset() {
|
||||
s_.clear();
|
||||
y_.clear();
|
||||
sy_.clear();
|
||||
}
|
||||
// The estimated diagonal of the Hessian; every element > 0.
|
||||
void SetDiagonal(std::vector<double> h) { h_ = std::move(h); }
|
||||
void SetInverseHessian(InverseHessian h0) { h0_ = std::move(h0); }
|
||||
|
||||
struct Outcome {
|
||||
int iterations = 0;
|
||||
@@ -47,8 +76,6 @@ public:
|
||||
Outcome Minimise(const Target &target, std::vector<double> &x, int iterations, double f_tolerance, double max_step,
|
||||
const std::function<bool()> &stop = nullptr) {
|
||||
const size_t n = x.size();
|
||||
if (h_.size() != n)
|
||||
h_.assign(n, 1.0);
|
||||
Outcome out;
|
||||
double f = 0;
|
||||
std::vector<double> g(n), d(n), x_new(n), g_new(n);
|
||||
@@ -59,11 +86,11 @@ public:
|
||||
for (int it = 0; it < iterations; it++) {
|
||||
if (stop && stop())
|
||||
break;
|
||||
Direction(g, d);
|
||||
Direction(x, g, d);
|
||||
double gd = Dot(g, d);
|
||||
if (!(gd < 0)) { // not a descent direction: the memory no longer fits, start again from the diagonal
|
||||
if (!(gd < 0)) { // not a descent direction: the memory no longer fits, start again from H0
|
||||
Reset();
|
||||
Direction(g, d);
|
||||
Direction(x, g, d);
|
||||
gd = Dot(g, d);
|
||||
if (!(gd < 0))
|
||||
break;
|
||||
@@ -75,7 +102,7 @@ public:
|
||||
double f_new = 0;
|
||||
bool accepted = false;
|
||||
for (int ls = 0; ls < 10; ls++) {
|
||||
Sweep(n, [&](size_t k) { x_new[k] = x[k] + alpha * d[k]; });
|
||||
refine_vec::Each(n, nthreads_, [&](size_t k) { x_new[k] = x[k] + alpha * d[k]; });
|
||||
const bool ok = target(x_new, f_new, g_new);
|
||||
out.evaluations++;
|
||||
if (ok && f_new <= f + 1e-4 * alpha * gd) {
|
||||
@@ -88,11 +115,8 @@ public:
|
||||
}
|
||||
if (!accepted)
|
||||
break;
|
||||
// the pair's vectors are the ones the memory last let go, where it did: made anew, their pages fault in
|
||||
std::vector<double> s = std::move(spare_s_), y = std::move(spare_y_);
|
||||
s.resize(n);
|
||||
y.resize(n);
|
||||
Sweep(n, [&](size_t k) {
|
||||
std::vector<double> s(n), y(n);
|
||||
refine_vec::Each(n, nthreads_, [&](size_t k) {
|
||||
s[k] = x_new[k] - x[k];
|
||||
y[k] = g_new[k] - g[k];
|
||||
});
|
||||
@@ -102,15 +126,10 @@ public:
|
||||
y_.push_back(std::move(y));
|
||||
sy_.push_back(sy);
|
||||
if (s_.size() > memory_) {
|
||||
spare_s_ = std::move(s_.front());
|
||||
spare_y_ = std::move(y_.front());
|
||||
s_.pop_front();
|
||||
y_.pop_front();
|
||||
sy_.pop_front();
|
||||
}
|
||||
} else {
|
||||
spare_s_ = std::move(s);
|
||||
spare_y_ = std::move(y);
|
||||
}
|
||||
const double decrease = f - f_new;
|
||||
x.swap(x_new);
|
||||
@@ -125,87 +144,64 @@ public:
|
||||
}
|
||||
|
||||
private:
|
||||
static double Dot(const std::vector<double> &a, const std::vector<double> &b) {
|
||||
double s = 0;
|
||||
for (size_t k = 0; k < a.size(); k++)
|
||||
s += a[k] * b[k];
|
||||
return s;
|
||||
}
|
||||
double Dot(const std::vector<double> &a, const std::vector<double> &b) const { return refine_vec::Dot(a, b, nthreads_); }
|
||||
|
||||
// fn(k) for every k below n on the threads, each element on its own: the serial loop's bits.
|
||||
template <typename Fn>
|
||||
void Sweep(size_t n, Fn fn) const {
|
||||
ParallelChunks(static_cast<int>(n), ThreadsForWork(n, nthreads_), [&](int lo, int hi) {
|
||||
for (int k = lo; k < hi; k++)
|
||||
fn(static_cast<size_t>(k));
|
||||
});
|
||||
}
|
||||
|
||||
// d = -H g by the two-loop recursion, H0 = gamma diag(1/h). Every dot product is a serial chain of n
|
||||
// additions, so each sweep over the parameters takes the next pair's dot product while it applies the
|
||||
// previous pair's update to q - the same operations on each element in the same order as a sweep for
|
||||
// each would do, with the update's memory traffic hidden under the chain.
|
||||
void Direction(const std::vector<double> &g, std::vector<double> &d) const {
|
||||
const size_t n = g.size(), m = s_.size();
|
||||
std::vector<double> &q = q_; // kept from call to call: made anew, its pages fault in every time
|
||||
q.assign(g.begin(), g.end());
|
||||
std::vector<double> a(m), rho(m);
|
||||
for (size_t i = 0; i < m; i++)
|
||||
rho[i] = 1.0 / sy_[i];
|
||||
double dot = m > 0 ? Dot(s_[m - 1], q) : 0.0, yhy = 0.0;
|
||||
// d = -H g by the two-loop recursion, H0 the caller's.
|
||||
void Direction(const std::vector<double> &x, const std::vector<double> &g, std::vector<double> &d) const {
|
||||
const size_t m = s_.size(), n = g.size();
|
||||
std::vector<double> q = g, a(m);
|
||||
for (size_t i = m; i-- > 0;) {
|
||||
a[i] = rho[i] * dot;
|
||||
const double ai = a[i];
|
||||
const double *y = y_[i].data();
|
||||
dot = 0.0;
|
||||
if (i > 0) {
|
||||
const double *s = s_[i - 1].data();
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
q[k] -= ai * y[k];
|
||||
dot += s[k] * q[k];
|
||||
}
|
||||
} else {
|
||||
const double *yl = y_[m - 1].data();
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
q[k] -= ai * y[k];
|
||||
yhy += yl[k] * yl[k] / h_[k];
|
||||
}
|
||||
}
|
||||
a[i] = Dot(s_[i], q) / sy_[i];
|
||||
refine_vec::Each(n, nthreads_, [&](size_t k) { q[k] -= a[i] * y_[i][k]; });
|
||||
}
|
||||
const double gamma = m > 0 ? sy_[m - 1] / yhy : 1.0;
|
||||
if (m > 0) {
|
||||
const double *y = y_[0].data();
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
q[k] *= gamma / h_[k];
|
||||
dot += y[k] * q[k];
|
||||
}
|
||||
} else
|
||||
for (size_t k = 0; k < n; k++)
|
||||
q[k] *= gamma / h_[k];
|
||||
h0_(x, q);
|
||||
for (size_t i = 0; i < m; i++) {
|
||||
// b on its own and (a[i] - b) in the loop, as before: written as a[i] - rho[i] * dot the compiler
|
||||
// contracts it into a fused multiply-add, and the bits change
|
||||
const double b = rho[i] * dot;
|
||||
const double *s = s_[i].data();
|
||||
dot = 0.0;
|
||||
if (i + 1 < m) {
|
||||
const double *y = y_[i + 1].data();
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
q[k] += s[k] * (a[i] - b);
|
||||
dot += y[k] * q[k];
|
||||
}
|
||||
} else
|
||||
for (size_t k = 0; k < n; k++)
|
||||
q[k] += s[k] * (a[i] - b);
|
||||
const double b = Dot(y_[i], q) / sy_[i];
|
||||
refine_vec::Each(n, nthreads_, [&](size_t k) { q[k] += s_[i][k] * (a[i] - b); });
|
||||
}
|
||||
for (size_t k = 0; k < n; k++)
|
||||
d[k] = -q[k];
|
||||
d.resize(n);
|
||||
refine_vec::Each(n, nthreads_, [&](size_t k) { d[k] = -q[k]; });
|
||||
}
|
||||
|
||||
size_t memory_, nthreads_;
|
||||
InverseHessian h0_;
|
||||
std::deque<std::vector<double>> s_, y_;
|
||||
std::deque<double> sy_; // s.y of each pair, as taken when it was kept
|
||||
std::vector<double> spare_s_, spare_y_; // the pair the memory last let go, for the next one to reuse
|
||||
mutable std::vector<double> q_; // Direction()'s working vector
|
||||
std::vector<double> h_;
|
||||
};
|
||||
|
||||
// z with H z = b to `tolerance` of |b| (at most `max_iterations`), by conjugate gradients preconditioned with M:
|
||||
// hv = H v (product), z = M^-1 r (precondition), both symmetric positive definite. Returns the iterations taken.
|
||||
// Following Hestenes & Stiefel (1952) J. Res. Natl. Bur. Stand. 49, 409-436
|
||||
inline int ConjugateGradients(const std::function<void(const std::vector<double> &v, std::vector<double> &hv)> &product,
|
||||
const std::function<void(const std::vector<double> &r, std::vector<double> &z)> &precondition,
|
||||
const std::vector<double> &b, std::vector<double> &x, int max_iterations, double tolerance,
|
||||
size_t nthreads = 1) {
|
||||
auto dot = [&](const std::vector<double> &u, const std::vector<double> &v) { return refine_vec::Dot(u, v, nthreads); };
|
||||
const size_t n = b.size();
|
||||
x.assign(n, 0.0);
|
||||
std::vector<double> r = b, z(n), p, hp(n);
|
||||
precondition(r, z);
|
||||
p = z;
|
||||
double rz = dot(r, z);
|
||||
const double r0 = std::sqrt(dot(r, r));
|
||||
int k = 0;
|
||||
while (k < max_iterations) {
|
||||
k++;
|
||||
product(p, hp);
|
||||
const double php = dot(p, hp);
|
||||
if (!(php > 0))
|
||||
break;
|
||||
const double a = rz / php;
|
||||
refine_vec::Each(n, nthreads, [&](size_t j) {
|
||||
x[j] += a * p[j];
|
||||
r[j] -= a * hp[j];
|
||||
});
|
||||
if (std::sqrt(dot(r, r)) < tolerance * r0)
|
||||
break;
|
||||
precondition(r, z);
|
||||
const double rz_new = dot(r, z);
|
||||
refine_vec::Each(n, nthreads, [&](size_t j) { p[j] = z[j] + rz_new / rz * p[j]; });
|
||||
rz = rz_new;
|
||||
}
|
||||
return k;
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
#include "RefinementNewtonGPU.h"
|
||||
|
||||
#include "RefinementRestraints.h"
|
||||
|
||||
// The host half: the index of J, as RestraintGaussNewton lays out its rows, with every row kept.
|
||||
void RestraintNormalMatrixGPU::At(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
|
||||
if (!indexed_) {
|
||||
std::vector<int> record_entry, start{0}, atom;
|
||||
for (const RestraintRecord &r : set.records) {
|
||||
record_entry.push_back(static_cast<int>(atom.size()));
|
||||
switch (static_cast<RestraintKind>(r.kind)) {
|
||||
case RestraintKind::Plane:
|
||||
// one row per atom of the plane, each over all of them
|
||||
for (int i = 0; i < r.atom[1]; i++) {
|
||||
for (int k = 0; k < r.atom[1]; k++)
|
||||
atom.push_back(set.plane_atoms[r.atom[0] + k]);
|
||||
start.push_back(static_cast<int>(atom.size()));
|
||||
}
|
||||
continue;
|
||||
case RestraintKind::Angle:
|
||||
atom.insert(atom.end(), r.atom, r.atom + 3);
|
||||
break;
|
||||
case RestraintKind::Torsion:
|
||||
case RestraintKind::Chiral:
|
||||
case RestraintKind::NcsDistance:
|
||||
atom.insert(atom.end(), r.atom, r.atom + 4);
|
||||
break;
|
||||
default: // Bond, Repulsion, BSimilarity
|
||||
atom.insert(atom.end(), r.atom, r.atom + 2);
|
||||
break;
|
||||
}
|
||||
start.push_back(static_cast<int>(atom.size()));
|
||||
}
|
||||
const size_t rows = start.size() - 1;
|
||||
std::vector<int> row_of(atom.size()), param(atom.size());
|
||||
for (size_t e = 0; e < atom.size(); e++)
|
||||
param[e] = param_of_[atom[e]];
|
||||
for (size_t r = 0; r < rows; r++)
|
||||
for (int e = start[r]; e < start[r + 1]; e++)
|
||||
row_of[e] = static_cast<int>(r);
|
||||
std::vector<int> entry_start(n_params_ + 1, 0);
|
||||
for (int k : param)
|
||||
if (k >= 0)
|
||||
entry_start[k + 1]++;
|
||||
for (size_t k = 0; k < n_params_; k++)
|
||||
entry_start[k + 1] += entry_start[k];
|
||||
std::vector<int> entry(entry_start.back());
|
||||
std::vector<int> fill(entry_start.begin(), entry_start.end() - 1);
|
||||
for (size_t e = 0; e < param.size(); e++)
|
||||
if (param[e] >= 0)
|
||||
entry[fill[param[e]]++] = static_cast<int>(e);
|
||||
Upload(set.records, set.plane_atoms, record_entry, start, param, row_of, entry_start, entry);
|
||||
indexed_ = true;
|
||||
}
|
||||
Fill(xyzb);
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
// The device half of RestraintNormalMatrixGPU, in float: one thread per restraint for J, one per row of J or per
|
||||
// parameter atom for the products, each summing its terms in order. J is held in float; the products' v, t and hv in
|
||||
// float pairs (hi + lo, double-float arithmetic: an error-free sum and product per term), since the soft directions of
|
||||
// H - those the X-ray term decides, J v the small difference of large terms - need more than float's 24 bits: with
|
||||
// plain float sums the Newton steps on a model with stiff restraints differed from the CPU's by more than its noise.
|
||||
|
||||
#include "RefinementNewtonGPU.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "../indexing/CUDAMemHelpers.h"
|
||||
#include "RefinementGPUEngine.h" // MAX_PLANE_ATOMS
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int THREADS = 256;
|
||||
using V3f = refine_math::V3<float>;
|
||||
|
||||
void cuda_err(cudaError_t val) {
|
||||
if (val != cudaSuccess)
|
||||
throw JFJochException(JFJochExceptionCategory::GPUCUDAError, cudaGetErrorString(val));
|
||||
}
|
||||
|
||||
class DeviceGuard {
|
||||
public:
|
||||
explicit DeviceGuard(int device) {
|
||||
cuda_err(cudaGetDevice(&previous_));
|
||||
cuda_err(cudaSetDevice(device));
|
||||
}
|
||||
~DeviceGuard() { cudaSetDevice(previous_); }
|
||||
DeviceGuard(const DeviceGuard &) = delete;
|
||||
DeviceGuard &operator=(const DeviceGuard &) = delete;
|
||||
|
||||
private:
|
||||
int previous_ = 0;
|
||||
};
|
||||
|
||||
int GreatestStreamPriority() {
|
||||
int least = 0, greatest = 0;
|
||||
cuda_err(cudaDeviceGetStreamPriorityRange(&least, &greatest));
|
||||
return greatest;
|
||||
}
|
||||
|
||||
int GridSize(int n) {
|
||||
return (n + THREADS - 1) / THREADS;
|
||||
}
|
||||
|
||||
__device__ V3f xyz(const float4 *p, int i) {
|
||||
return {p[i].x, p[i].y, p[i].z};
|
||||
}
|
||||
|
||||
__device__ float Get(const float4 &j, int c) {
|
||||
return c == 0 ? j.x : c == 1 ? j.y : c == 2 ? j.z : j.w;
|
||||
}
|
||||
|
||||
// sum += a (b.x + b.y), sum and b float pairs (hi in x, lo in y). Needs fma and no reassociation (no fast math).
|
||||
// The exact product and 2Sum following Dekker (1971) Numer. Math. 18, 224-242 and Knuth, TAOCP vol. 2, 4.2.2
|
||||
__device__ void AddProduct(float2 &sum, float a, float2 b) {
|
||||
const float p = a * b.x;
|
||||
const float p_err = fmaf(a, b.x, -p); // a b.x = p + p_err exactly
|
||||
const float s = sum.x + p;
|
||||
const float v = s - sum.x;
|
||||
const float s_err = (sum.x - (s - v)) + (p - v); // sum.x + p = s + s_err exactly
|
||||
const float lo = s_err + sum.y + p_err + a * b.y;
|
||||
const float hi = s + lo;
|
||||
sum = make_float2(hi, lo - (hi - s));
|
||||
}
|
||||
|
||||
// Each restraint's rows of J (RestraintGaussNewton): E = residual^2, so J = g / sqrt(2 E); a plane's atoms each a
|
||||
// residual of their own, d r_i / d x_k = (delta_ik - 1/m) sqrt(2) normal / sigma. Zero where there is no row.
|
||||
__global__ void JacobianKernel(const RestraintRecord *records, int n, const int *plane_atoms, const int *record_entry,
|
||||
const float4 *xyzb, float4 *j) {
|
||||
const int i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (i >= n)
|
||||
return;
|
||||
const RestraintRecord &r = records[i];
|
||||
float4 *out = j + record_entry[i];
|
||||
V3f g[4] = {};
|
||||
int na = 0;
|
||||
float e = 0;
|
||||
switch (static_cast<RestraintKind>(r.kind)) {
|
||||
case RestraintKind::Bond:
|
||||
e = refine_math::BondTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), r.ideal, r.sigma, g);
|
||||
na = 2;
|
||||
break;
|
||||
case RestraintKind::Angle:
|
||||
e = refine_math::AngleTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]), r.ideal,
|
||||
r.sigma, g);
|
||||
na = 3;
|
||||
break;
|
||||
case RestraintKind::Torsion:
|
||||
e = refine_math::TorsionTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]),
|
||||
xyz(xyzb, r.atom[3]), r.ideal, r.sigma, r.period, g);
|
||||
na = 4;
|
||||
break;
|
||||
case RestraintKind::Chiral:
|
||||
e = refine_math::ChiralTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]),
|
||||
xyz(xyzb, r.atom[3]), r.ideal, r.sigma, g);
|
||||
na = 4;
|
||||
break;
|
||||
case RestraintKind::NcsDistance:
|
||||
e = refine_math::NcsDistanceTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]),
|
||||
xyz(xyzb, r.atom[3]), r.sigma, r.ideal, g);
|
||||
na = 4;
|
||||
break;
|
||||
case RestraintKind::Repulsion: {
|
||||
const V3f q = xyz(xyzb, r.atom[1]);
|
||||
const V3f image{r.rot[0] * q.x + r.rot[1] * q.y + r.rot[2] * q.z + r.tran[0],
|
||||
r.rot[3] * q.x + r.rot[4] * q.y + r.rot[5] * q.z + r.tran[1],
|
||||
r.rot[6] * q.x + r.rot[7] * q.y + r.rot[8] * q.z + r.tran[2]};
|
||||
e = refine_math::RepulsionTerm(xyz(xyzb, r.atom[0]), image, r.ideal, r.sigma, g);
|
||||
const V3f gi = g[1];
|
||||
g[1] = {r.rot[0] * gi.x + r.rot[3] * gi.y + r.rot[6] * gi.z,
|
||||
r.rot[1] * gi.x + r.rot[4] * gi.y + r.rot[7] * gi.z,
|
||||
r.rot[2] * gi.x + r.rot[5] * gi.y + r.rot[8] * gi.z};
|
||||
na = 2;
|
||||
break;
|
||||
}
|
||||
case RestraintKind::Plane: {
|
||||
V3f px[RefinementGPUEngine::MAX_PLANE_ATOMS], pg[RefinementGPUEngine::MAX_PLANE_ATOMS];
|
||||
const int m = r.atom[1];
|
||||
for (int k = 0; k < m; k++)
|
||||
px[k] = xyz(xyzb, plane_atoms[r.atom[0] + k]);
|
||||
refine_math::PlaneTerm(px, m, r.sigma, pg);
|
||||
V3f nv{0, 0, 0};
|
||||
for (int k = 0; k < m; k++) // the normal from the largest of the gradients (each is 2 r_k / sigma n)
|
||||
if (refine_math::Norm(pg[k]) > refine_math::Norm(nv))
|
||||
nv = pg[k];
|
||||
const float norm = refine_math::Norm(nv);
|
||||
const float c = norm > 0 ? sqrtf(2.0f) / (r.sigma * norm) : 0.0f;
|
||||
for (int a = 0; a < m; a++)
|
||||
for (int k = 0; k < m; k++) {
|
||||
const float f = c * ((a == k ? 1.0f : 0.0f) - 1.0f / m);
|
||||
out[a * m + k] = make_float4(f * nv.x, f * nv.y, f * nv.z, 0.0f);
|
||||
}
|
||||
return;
|
||||
}
|
||||
case RestraintKind::BSimilarity: {
|
||||
float gi = 0, gj = 0;
|
||||
e = refine_math::BSimilarityTerm(xyzb[r.atom[0]].w, xyzb[r.atom[1]].w, r.sigma, gi, gj);
|
||||
const float f = e > 0 ? 1.0f / sqrtf(2.0f * e) : 0.0f;
|
||||
out[0] = make_float4(0, 0, 0, f * gi);
|
||||
out[1] = make_float4(0, 0, 0, f * gj);
|
||||
return;
|
||||
}
|
||||
}
|
||||
const float f = e > 0 ? 1.0f / sqrtf(2.0f * e) : 0.0f;
|
||||
for (int k = 0; k < na; k++)
|
||||
out[k] = make_float4(f * g[k].x, f * g[k].y, f * g[k].z, 0.0f);
|
||||
}
|
||||
|
||||
// t_r = sum over the row's entries of J_e . v_k
|
||||
__global__ void RowProduct(const int *start, const int *param, const float4 *j, int per, int c0, const float2 *v,
|
||||
float2 *t, int rows) {
|
||||
const int r = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (r >= rows)
|
||||
return;
|
||||
float2 dot = make_float2(0, 0);
|
||||
for (int e = start[r]; e < start[r + 1]; e++)
|
||||
if (const int k = param[e]; k >= 0)
|
||||
for (int c = 0; c < per; c++)
|
||||
AddProduct(dot, Get(j[e], c0 + c), v[per * k + c]);
|
||||
t[r] = dot;
|
||||
}
|
||||
|
||||
// hv_k = sum over the parameter atom's entries of t_row J_e
|
||||
__global__ void TransposeProduct(const int *entry_start, const int *entry, const int *row_of, const float4 *j, int per,
|
||||
int c0, const float2 *t, float2 *hv, int n) {
|
||||
const int k = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (k >= n)
|
||||
return;
|
||||
for (int c = 0; c < per; c++) {
|
||||
float2 sum = make_float2(0, 0);
|
||||
for (int q = entry_start[k]; q < entry_start[k + 1]; q++) {
|
||||
const int e = entry[q];
|
||||
AddProduct(sum, Get(j[e], c0 + c), t[row_of[e]]);
|
||||
}
|
||||
hv[per * k + c] = sum;
|
||||
}
|
||||
}
|
||||
|
||||
__global__ void DiagonalKernel(const int *entry_start, const int *entry, const float4 *j, int per, int c0, float *d,
|
||||
int n) {
|
||||
const int k = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (k >= n)
|
||||
return;
|
||||
for (int c = 0; c < per; c++) {
|
||||
float sum = 0;
|
||||
for (int q = entry_start[k]; q < entry_start[k + 1]; q++) {
|
||||
const float x = Get(j[entry[q]], c0 + c);
|
||||
sum += x * x;
|
||||
}
|
||||
d[per * k + c] = sum;
|
||||
}
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
CudaDevicePtr<T> Up(const std::vector<T> &host, cudaStream_t stream) {
|
||||
CudaDevicePtr<T> d(std::max<size_t>(host.size(), 1));
|
||||
cuda_err(cudaMemcpyAsync(d.get(), host.data(), host.size() * sizeof(T), cudaMemcpyHostToDevice, stream));
|
||||
return d;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
struct RestraintNormalMatrixGPU::Device {
|
||||
int device;
|
||||
CudaStream stream;
|
||||
int records = 0, rows = 0;
|
||||
CudaDevicePtr<RestraintRecord> record;
|
||||
CudaDevicePtr<int> plane_atoms, record_entry, start, param, row_of, entry_start, entry;
|
||||
CudaDevicePtr<float4> xyzb, j;
|
||||
CudaDevicePtr<float2> t, v, hv;
|
||||
CudaDevicePtr<float> diagonal;
|
||||
PinnedVector<float2> host; // v and hv as float pairs
|
||||
PinnedVector<float4> host_xyzb;
|
||||
};
|
||||
|
||||
RestraintNormalMatrixGPU::RestraintNormalMatrixGPU(int device, const std::vector<int> ¶m_atoms, size_t n_atoms,
|
||||
bool xyz)
|
||||
: per_(xyz ? 3 : 1), c0_(xyz ? 0 : 3), n_params_(param_atoms.size()), param_of_(n_atoms, -1) {
|
||||
for (size_t k = 0; k < param_atoms.size(); k++)
|
||||
param_of_[param_atoms[k]] = static_cast<int>(k);
|
||||
DeviceGuard guard(device);
|
||||
// At the device's highest priority: a product is a few small kernels the minimiser waits for, queued beside the
|
||||
// X-ray evaluations of the other trials on the card.
|
||||
dev_ = std::unique_ptr<Device>(new Device{device, CudaStream(cudaStreamNonBlocking, GreatestStreamPriority())});
|
||||
const size_t n = std::max<size_t>(per_ * n_params_, 1);
|
||||
dev_->xyzb = CudaDevicePtr<float4>(std::max<size_t>(n_atoms, 1));
|
||||
dev_->host_xyzb.resize(n_atoms);
|
||||
dev_->v = CudaDevicePtr<float2>(n);
|
||||
dev_->hv = CudaDevicePtr<float2>(n);
|
||||
dev_->diagonal = CudaDevicePtr<float>(n);
|
||||
dev_->host.resize(n);
|
||||
}
|
||||
|
||||
RestraintNormalMatrixGPU::~RestraintNormalMatrixGPU() {
|
||||
DeviceGuard guard(dev_->device);
|
||||
dev_.reset();
|
||||
}
|
||||
|
||||
void RestraintNormalMatrixGPU::Upload(const std::vector<RestraintRecord> &records, const std::vector<int> &plane_atoms,
|
||||
const std::vector<int> &record_entry, const std::vector<int> &start,
|
||||
const std::vector<int> ¶m, const std::vector<int> &row_of,
|
||||
const std::vector<int> &entry_start, const std::vector<int> &entry) {
|
||||
DeviceGuard guard(dev_->device);
|
||||
Device &d = *dev_;
|
||||
d.records = static_cast<int>(records.size());
|
||||
d.rows = static_cast<int>(start.size()) - 1;
|
||||
d.record = Up(records, d.stream);
|
||||
d.plane_atoms = Up(plane_atoms, d.stream);
|
||||
d.record_entry = Up(record_entry, d.stream);
|
||||
d.start = Up(start, d.stream);
|
||||
d.param = Up(param, d.stream);
|
||||
d.row_of = Up(row_of, d.stream);
|
||||
d.entry_start = Up(entry_start, d.stream);
|
||||
d.entry = Up(entry, d.stream);
|
||||
d.j = CudaDevicePtr<float4>(std::max<size_t>(param.size(), 1));
|
||||
d.t = CudaDevicePtr<float2>(std::max(d.rows, 1));
|
||||
cuda_err(cudaStreamSynchronize(d.stream));
|
||||
}
|
||||
|
||||
void RestraintNormalMatrixGPU::Fill(const std::vector<std::array<double, 4>> &xyzb) {
|
||||
DeviceGuard guard(dev_->device);
|
||||
Device &d = *dev_;
|
||||
for (size_t a = 0; a < xyzb.size(); a++)
|
||||
d.host_xyzb[a] = make_float4(static_cast<float>(xyzb[a][0]), static_cast<float>(xyzb[a][1]),
|
||||
static_cast<float>(xyzb[a][2]), static_cast<float>(xyzb[a][3]));
|
||||
cuda_err(cudaMemcpyAsync(d.xyzb.get(), d.host_xyzb.data(), xyzb.size() * sizeof(float4), cudaMemcpyHostToDevice,
|
||||
d.stream));
|
||||
if (d.records > 0)
|
||||
JacobianKernel<<<GridSize(d.records), THREADS, 0, d.stream>>>(d.record, d.records, d.plane_atoms,
|
||||
d.record_entry, d.xyzb, d.j);
|
||||
cuda_err(cudaGetLastError());
|
||||
cuda_err(cudaStreamSynchronize(d.stream));
|
||||
}
|
||||
|
||||
void RestraintNormalMatrixGPU::Multiply(const std::vector<double> &v, std::vector<double> &hv) {
|
||||
DeviceGuard guard(dev_->device);
|
||||
Device &d = *dev_;
|
||||
const int n = static_cast<int>(n_params_);
|
||||
hv.resize(per_ * n_params_);
|
||||
if (n == 0)
|
||||
return;
|
||||
for (size_t i = 0; i < hv.size(); i++) {
|
||||
const float hi = static_cast<float>(v[i]);
|
||||
d.host[i] = make_float2(hi, static_cast<float>(v[i] - hi));
|
||||
}
|
||||
cuda_err(cudaMemcpyAsync(d.v.get(), d.host.data(), hv.size() * sizeof(float2), cudaMemcpyHostToDevice, d.stream));
|
||||
if (d.rows > 0)
|
||||
RowProduct<<<GridSize(d.rows), THREADS, 0, d.stream>>>(d.start, d.param, d.j, per_, c0_, d.v, d.t, d.rows);
|
||||
TransposeProduct<<<GridSize(n), THREADS, 0, d.stream>>>(d.entry_start, d.entry, d.row_of, d.j, per_, c0_, d.t,
|
||||
d.hv, n);
|
||||
cuda_err(cudaGetLastError());
|
||||
cuda_err(cudaMemcpyAsync(d.host.data(), d.hv.get(), hv.size() * sizeof(float2), cudaMemcpyDeviceToHost, d.stream));
|
||||
cuda_err(cudaStreamSynchronize(d.stream));
|
||||
for (size_t i = 0; i < hv.size(); i++)
|
||||
hv[i] = static_cast<double>(d.host[i].x) + d.host[i].y;
|
||||
}
|
||||
|
||||
std::vector<double> RestraintNormalMatrixGPU::Diagonal() const {
|
||||
DeviceGuard guard(dev_->device);
|
||||
Device &d = *dev_;
|
||||
const int n = static_cast<int>(n_params_);
|
||||
std::vector<double> out(per_ * n_params_);
|
||||
if (n == 0)
|
||||
return out;
|
||||
std::vector<float> host(out.size());
|
||||
DiagonalKernel<<<GridSize(n), THREADS, 0, d.stream>>>(d.entry_start, d.entry, d.j, per_, c0_, d.diagonal, n);
|
||||
cuda_err(cudaGetLastError());
|
||||
cuda_err(cudaMemcpyAsync(host.data(), d.diagonal.get(), host.size() * sizeof(float), cudaMemcpyDeviceToHost, d.stream));
|
||||
cuda_err(cudaStreamSynchronize(d.stream));
|
||||
std::copy(host.begin(), host.end(), out.begin());
|
||||
return out;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
#pragma once
|
||||
|
||||
// The restraints' Gauss-Newton normal matrix J^T J (RestraintGaussNewton) of the Newton minimisation on a GPU (CUDA
|
||||
// builds only): RestraintNormalMatrix of Refinement.cpp on the card, in single precision. J is worked out on the device,
|
||||
// one thread per restraint, with the terms the engine evaluates the restraints with (RefinementMath.h); a product is two
|
||||
// gathers, J v per row and J^T t per parameter atom over its entries, each summed in a fixed order without atomics, in
|
||||
// double-float pairs (RefinementNewtonGPU.cu).
|
||||
//
|
||||
// Every restraint keeps its rows whatever its E: where RestraintGaussNewton drops a row (E = 0, a plane without a
|
||||
// normal) its entries here are zero, so the index of J is worked out once, at the first At(), and the set must stay
|
||||
// the same for the matrix's life.
|
||||
|
||||
#include <array>
|
||||
#include <cstddef>
|
||||
#include <memory>
|
||||
#include <vector>
|
||||
|
||||
#include "RefinementMath.h" // RestraintRecord
|
||||
|
||||
struct RestraintSet;
|
||||
|
||||
class RestraintNormalMatrixGPU {
|
||||
public:
|
||||
// On the x y z (`xyz`) or the B of `param_atoms` of the model's `n_atoms`, on GPU `device`.
|
||||
RestraintNormalMatrixGPU(int device, const std::vector<int> ¶m_atoms, size_t n_atoms, bool xyz);
|
||||
~RestraintNormalMatrixGPU();
|
||||
|
||||
void At(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb);
|
||||
// hv = J^T J v
|
||||
void Multiply(const std::vector<double> &v, std::vector<double> &hv);
|
||||
// The diagonal of J^T J.
|
||||
std::vector<double> Diagonal() const;
|
||||
|
||||
private:
|
||||
// The set and the index of J (RefinementNewtonGPU.cu): per restraint its first entry, per row its entries
|
||||
// start[r] ... start[r + 1], per entry its parameter atom (-1: none) and row, per parameter atom its entries.
|
||||
void Upload(const std::vector<RestraintRecord> &records, const std::vector<int> &plane_atoms,
|
||||
const std::vector<int> &record_entry, const std::vector<int> &start, const std::vector<int> ¶m,
|
||||
const std::vector<int> &row_of, const std::vector<int> &entry_start, const std::vector<int> &entry);
|
||||
// J at `xyzb` (x, y, z, B per atom), in float
|
||||
void Fill(const std::vector<std::array<double, 4>> &xyzb);
|
||||
|
||||
int per_, c0_;
|
||||
size_t n_params_;
|
||||
std::vector<int> param_of_;
|
||||
bool indexed_ = false;
|
||||
struct Device;
|
||||
std::unique_ptr<Device> dev_;
|
||||
};
|
||||
@@ -397,6 +397,48 @@ double RestraintValue(const RestraintSet &set, const RestraintRecord &r, const s
|
||||
return NAN;
|
||||
}
|
||||
|
||||
namespace {
|
||||
// E of a restraint of the kinds over up to four atoms (not a plane, not a B similarity), its gradient in g[0..n).
|
||||
double RecordTerm(const RestraintRecord &r, const std::vector<std::array<double, 4>> &xyzb, V3<double> g[4], int &n) {
|
||||
const double ideal = r.ideal, sigma = r.sigma;
|
||||
double e = 0;
|
||||
n = 0;
|
||||
switch (static_cast<RestraintKind>(r.kind)) {
|
||||
case RestraintKind::Bond:
|
||||
e = refine_math::BondTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), ideal, sigma, g);
|
||||
n = 2;
|
||||
break;
|
||||
case RestraintKind::Angle:
|
||||
e = refine_math::AngleTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), ideal, sigma, g);
|
||||
n = 3;
|
||||
break;
|
||||
case RestraintKind::Torsion:
|
||||
e = refine_math::TorsionTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
|
||||
At(xyzb, r.atom[3]), ideal, sigma, r.period, g);
|
||||
n = 4;
|
||||
break;
|
||||
case RestraintKind::Chiral:
|
||||
e = refine_math::ChiralTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
|
||||
At(xyzb, r.atom[3]), ideal, sigma, g);
|
||||
n = 4;
|
||||
break;
|
||||
case RestraintKind::NcsDistance:
|
||||
e = refine_math::NcsDistanceTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
|
||||
At(xyzb, r.atom[3]), sigma, ideal, g);
|
||||
n = 4;
|
||||
break;
|
||||
case RestraintKind::Repulsion:
|
||||
e = refine_math::RepulsionTerm(At(xyzb, r.atom[0]), Image(r, At(xyzb, r.atom[1])), ideal, sigma, g);
|
||||
g[1] = FromImage(r, g[1]);
|
||||
n = 2;
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
return e;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb,
|
||||
std::vector<std::array<double, 4>> *grad) {
|
||||
double sum = 0;
|
||||
@@ -405,36 +447,8 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<
|
||||
V3<double> g[4] = {};
|
||||
int n = 0;
|
||||
double e = 0;
|
||||
const double ideal = r.ideal, sigma = r.sigma;
|
||||
const double sigma = r.sigma;
|
||||
switch (static_cast<RestraintKind>(r.kind)) {
|
||||
case RestraintKind::Bond:
|
||||
e = refine_math::BondTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), ideal, sigma, g);
|
||||
n = 2;
|
||||
break;
|
||||
case RestraintKind::Angle:
|
||||
e = refine_math::AngleTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), ideal, sigma, g);
|
||||
n = 3;
|
||||
break;
|
||||
case RestraintKind::Torsion:
|
||||
e = refine_math::TorsionTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
|
||||
At(xyzb, r.atom[3]), ideal, sigma, r.period, g);
|
||||
n = 4;
|
||||
break;
|
||||
case RestraintKind::Chiral:
|
||||
e = refine_math::ChiralTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
|
||||
At(xyzb, r.atom[3]), ideal, sigma, g);
|
||||
n = 4;
|
||||
break;
|
||||
case RestraintKind::NcsDistance:
|
||||
e = refine_math::NcsDistanceTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
|
||||
At(xyzb, r.atom[3]), sigma, ideal, g);
|
||||
n = 4;
|
||||
break;
|
||||
case RestraintKind::Repulsion:
|
||||
e = refine_math::RepulsionTerm(At(xyzb, r.atom[0]), Image(r, At(xyzb, r.atom[1])), ideal, sigma, g);
|
||||
g[1] = FromImage(r, g[1]);
|
||||
n = 2;
|
||||
break;
|
||||
case RestraintKind::Plane: {
|
||||
const int m = r.atom[1];
|
||||
px.resize(m);
|
||||
@@ -460,6 +474,8 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<
|
||||
}
|
||||
break;
|
||||
}
|
||||
default:
|
||||
e = RecordTerm(r, xyzb, g, n);
|
||||
}
|
||||
sum += e;
|
||||
if (grad != nullptr)
|
||||
@@ -473,16 +489,66 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<
|
||||
return sum;
|
||||
}
|
||||
|
||||
std::vector<double> BRestraintCurvature(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
|
||||
std::vector<double> h(xyzb.size(), 0.0);
|
||||
for (const RestraintRecord &r : set.records)
|
||||
if (r.kind == static_cast<int>(RestraintKind::BSimilarity)) {
|
||||
// d2E/dB_i^2 of BSimilarityTerm = 2 B_j / (kappa^2 B_i^3)
|
||||
const double bi = xyzb[r.atom[0]][3], bj = xyzb[r.atom[1]][3], k2 = double(r.sigma) * r.sigma;
|
||||
h[r.atom[0]] += 2.0 * bj / (k2 * bi * bi * bi);
|
||||
h[r.atom[1]] += 2.0 * bi / (k2 * bj * bj * bj);
|
||||
RestraintJacobian RestraintGaussNewton(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
|
||||
RestraintJacobian jac;
|
||||
jac.start.push_back(0);
|
||||
auto row = [&](int atom, double x, double y, double z, double b) {
|
||||
jac.atom.push_back(atom);
|
||||
jac.j.push_back({x, y, z, b});
|
||||
};
|
||||
std::vector<V3<double>> px, pg;
|
||||
for (const RestraintRecord &r : set.records) {
|
||||
const RestraintKind kind = static_cast<RestraintKind>(r.kind);
|
||||
if (kind == RestraintKind::Plane) {
|
||||
// each atom's distance from the plane a residual of its own, d r_i / d x_j = (delta_ij - 1/n) normal / sigma
|
||||
// (the normal held), so one row per atom of the plane over all of them
|
||||
const int m = r.atom[1];
|
||||
px.resize(m);
|
||||
pg.resize(m);
|
||||
for (int k = 0; k < m; k++)
|
||||
px[k] = At(xyzb, set.plane_atoms[r.atom[0] + k]);
|
||||
refine_math::PlaneTerm(px.data(), m, static_cast<double>(r.sigma), pg.data());
|
||||
const double c = std::sqrt(2.0) / r.sigma;
|
||||
V3<double> nv{0, 0, 0};
|
||||
for (int k = 0; k < m; k++) // the normal from the largest of the gradients (each is 2 r_k / sigma n)
|
||||
if (refine_math::Norm(pg[k]) > refine_math::Norm(nv))
|
||||
nv = pg[k];
|
||||
if (refine_math::Norm(nv) == 0)
|
||||
continue;
|
||||
nv = (1.0 / refine_math::Norm(nv)) * nv;
|
||||
for (int i = 0; i < m; i++) {
|
||||
for (int k = 0; k < m; k++) {
|
||||
const double f = c * ((i == k ? 1.0 : 0.0) - 1.0 / m);
|
||||
row(set.plane_atoms[r.atom[0] + k], f * nv.x, f * nv.y, f * nv.z, 0);
|
||||
}
|
||||
jac.start.push_back(static_cast<int>(jac.atom.size()));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
return h;
|
||||
// E = residual^2 (or close to it): its gradient is 2 residual grad(residual), so J = g / sqrt(2 E) gives the
|
||||
// Gauss-Newton term 2 grad(residual) grad(residual)^T = J J^T; nothing where E is 0
|
||||
if (kind == RestraintKind::BSimilarity) {
|
||||
double gi = 0, gj = 0;
|
||||
const double e = refine_math::BSimilarityTerm(xyzb[r.atom[0]][3], xyzb[r.atom[1]][3],
|
||||
static_cast<double>(r.sigma), gi, gj);
|
||||
if (!(e > 0))
|
||||
continue;
|
||||
const double f = 1.0 / std::sqrt(2.0 * e);
|
||||
row(r.atom[0], 0, 0, 0, f * gi);
|
||||
row(r.atom[1], 0, 0, 0, f * gj);
|
||||
} else {
|
||||
V3<double> g[4] = {};
|
||||
int n = 0;
|
||||
const double e = RecordTerm(r, xyzb, g, n);
|
||||
if (!(e > 0))
|
||||
continue;
|
||||
const double f = 1.0 / std::sqrt(2.0 * e);
|
||||
for (int k = 0; k < n; k++)
|
||||
row(r.atom[k], f * g[k].x, f * g[k].y, f * g[k].z, 0);
|
||||
}
|
||||
jac.start.push_back(static_cast<int>(jac.atom.size()));
|
||||
}
|
||||
return jac;
|
||||
}
|
||||
|
||||
gemmi::Position SpecialPosition::OnSite(const gemmi::Position &p) const {
|
||||
|
||||
@@ -49,8 +49,14 @@ double RestraintValue(const RestraintSet &set, const RestraintRecord &r, const s
|
||||
double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb,
|
||||
std::vector<std::array<double, 4>> *grad);
|
||||
|
||||
// The diagonal of the Hessian of the set's B similarity restraints with respect to each atom's B, at `xyzb`.
|
||||
std::vector<double> BRestraintCurvature(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb);
|
||||
// The Gauss-Newton approximation of the set's Hessian, sum over rows of J J^T: per row (a restraint's
|
||||
// residual) its derivatives in (x, y, z, B) of the atoms atom[start[r] ... start[r + 1]).
|
||||
struct RestraintJacobian {
|
||||
std::vector<int> start;
|
||||
std::vector<int> atom;
|
||||
std::vector<std::array<double, 4>> j;
|
||||
};
|
||||
RestraintJacobian RestraintGaussNewton(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb);
|
||||
|
||||
// An atom on a crystallographic special position: one that some symmetry operations of the group, each with a
|
||||
// lattice translation, take onto itself - the site's symmetry. Its position moves only within the subspace those
|
||||
|
||||
+173
-6
@@ -37,6 +37,7 @@
|
||||
#include "../image_analysis/structure_refinement/RefinementXray.h"
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
#include "../image_analysis/structure_refinement/RefinementGPU.h"
|
||||
#include "../image_analysis/structure_refinement/RefinementNewtonGPU.h"
|
||||
#endif
|
||||
|
||||
namespace {
|
||||
@@ -787,6 +788,55 @@ TEST_CASE("RefinementRestraints_deposited_ligands", "[Refinement]") {
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("RefinementRestraints_gauss_newton", "[Refinement]") {
|
||||
// near the ideal geometry the restraints' Hessian is their Gauss-Newton J^T J (the residuals' own curvature times
|
||||
// residuals that are small): v.J^T J v against central differences of the gradient along v, for random v
|
||||
const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90);
|
||||
gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5);
|
||||
st.models[0] = shaken(st.models[0], 0.005, 0.0, 2);
|
||||
const RefinementTopology topo = BuildRefinementTopology(st);
|
||||
RestraintSet set = topo.geometry;
|
||||
set.Finalize(atoms_of(st.models[0]).size());
|
||||
std::vector<std::array<double, 4>> xyzb;
|
||||
for (gemmi::Atom *a : atoms_of(st.models[0]))
|
||||
xyzb.push_back({a->pos.x, a->pos.y, a->pos.z, a->b_iso});
|
||||
const RestraintJacobian jac = RestraintGaussNewton(set, xyzb);
|
||||
REQUIRE(jac.start.size() > 10);
|
||||
std::mt19937 rng(4);
|
||||
std::normal_distribution<double> normal(0.0, 1.0);
|
||||
for (int trial = 0; trial < 5; trial++) {
|
||||
std::vector<std::array<double, 4>> v(xyzb.size(), {0, 0, 0, 0});
|
||||
for (auto &e : v)
|
||||
for (int c = 0; c < 3; c++)
|
||||
e[c] = normal(rng);
|
||||
double gn = 0;
|
||||
for (size_t r = 0; r + 1 < jac.start.size(); r++) {
|
||||
double jv = 0;
|
||||
for (int e = jac.start[r]; e < jac.start[r + 1]; e++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
jv += jac.j[e][c] * v[jac.atom[e]][c];
|
||||
gn += jv * jv;
|
||||
}
|
||||
const double h = 1e-5;
|
||||
std::array<std::vector<std::array<double, 4>>, 2> g;
|
||||
for (int side = 0; side < 2; side++) {
|
||||
auto x = xyzb;
|
||||
for (size_t i = 0; i < x.size(); i++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
x[i][c] += (side == 0 ? h : -h) * v[i][c];
|
||||
g[side].assign(x.size(), {0, 0, 0, 0});
|
||||
EvaluateRestraints(set, x, &g[side]);
|
||||
}
|
||||
double exact = 0;
|
||||
for (size_t i = 0; i < xyzb.size(); i++)
|
||||
for (int c = 0; c < 3; c++)
|
||||
exact += v[i][c] * (g[0][i][c] - g[1][i][c]) / (2 * h);
|
||||
CAPTURE(trial, gn, exact);
|
||||
CHECK(gn > 0);
|
||||
CHECK(gn == Catch::Approx(exact).epsilon(0.1));
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("RefinementRestraints_gradient", "[Refinement]") {
|
||||
const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90);
|
||||
gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5);
|
||||
@@ -986,6 +1036,82 @@ TEST_CASE("RefinementXray_gradient_CPU", "[Refinement]") {
|
||||
}
|
||||
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
TEST_CASE("RefinementNewton_product_GPU_vs_CPU", "[Refinement]") {
|
||||
if (get_gpu_count() == 0)
|
||||
return;
|
||||
const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90);
|
||||
gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5);
|
||||
st.models[0] = shaken(st.models[0], 0.15, 3.0, 2);
|
||||
const RefinementTopology topo = BuildRefinementTopology(st);
|
||||
RestraintSet set = topo.geometry;
|
||||
set.Append(topo.adp);
|
||||
set.Append(BuildRepulsions(topo, st.models[0], cell, *st.find_spacegroup(), 0.0));
|
||||
const std::vector<gemmi::Atom *> atoms = atoms_of(st.models[0]);
|
||||
set.Finalize(atoms.size());
|
||||
std::vector<std::array<double, 4>> xyzb;
|
||||
for (gemmi::Atom *a : atoms)
|
||||
xyzb.push_back({a->pos.x, a->pos.y, a->pos.z, a->b_iso});
|
||||
// every atom but each fifth a parameter: the rest drop out
|
||||
std::vector<int> params, param_of(atoms.size(), -1);
|
||||
for (int a = 0; a < static_cast<int>(atoms.size()); a++)
|
||||
if (a % 5 != 4) {
|
||||
param_of[a] = static_cast<int>(params.size());
|
||||
params.push_back(a);
|
||||
}
|
||||
const RestraintJacobian jac = RestraintGaussNewton(set, xyzb);
|
||||
REQUIRE(jac.start.size() > 100);
|
||||
std::mt19937 gen(3);
|
||||
std::uniform_real_distribution<double> uniform(-1, 1);
|
||||
// a random v, and a smooth one - the same shift of every atom and a thousandth of it at random, J v then the small
|
||||
// difference of large terms
|
||||
for (const auto [xyz, smooth] : {std::pair{true, false}, {true, true}, {false, false}, {false, true}}) {
|
||||
CAPTURE(xyz, smooth);
|
||||
const int per = xyz ? 3 : 1, c0 = xyz ? 0 : 3;
|
||||
const size_t n = per * params.size();
|
||||
std::vector<double> v(n);
|
||||
for (size_t i = 0; i < n; i++)
|
||||
v[i] = smooth ? 0.3 + 0.2 * (i % per) + 1e-3 * uniform(gen) : uniform(gen);
|
||||
// J^T J v and the diagonal on the CPU, each entry in turn
|
||||
std::vector<double> hv(n, 0.0), diagonal(n, 0.0);
|
||||
for (size_t r = 0; r + 1 < jac.start.size(); r++) {
|
||||
double t = 0;
|
||||
for (int e = jac.start[r]; e < jac.start[r + 1]; e++)
|
||||
if (const int k = param_of[jac.atom[e]]; k >= 0)
|
||||
for (int c = 0; c < per; c++)
|
||||
t += jac.j[e][c0 + c] * v[per * k + c];
|
||||
for (int e = jac.start[r]; e < jac.start[r + 1]; e++)
|
||||
if (const int k = param_of[jac.atom[e]]; k >= 0)
|
||||
for (int c = 0; c < per; c++)
|
||||
hv[per * k + c] += t * jac.j[e][c0 + c];
|
||||
}
|
||||
for (size_t e = 0; e < jac.atom.size(); e++)
|
||||
if (const int k = param_of[jac.atom[e]]; k >= 0) {
|
||||
const std::array<double, 4> &j = jac.j[e];
|
||||
for (int c = 0; c < per; c++)
|
||||
diagonal[per * k + c] += j[c0 + c] * j[c0 + c];
|
||||
}
|
||||
RestraintNormalMatrixGPU gpu(0, params, atoms.size(), xyz);
|
||||
gpu.At(set, xyzb);
|
||||
std::vector<double> hv_gpu;
|
||||
gpu.Multiply(v, hv_gpu);
|
||||
const std::vector<double> diagonal_gpu = gpu.Diagonal();
|
||||
REQUIRE(hv_gpu.size() == n);
|
||||
REQUIRE(diagonal_gpu.size() == n);
|
||||
// In float, from float positions: within a few float epsilons of the double product in norm (in refinement
|
||||
// its error is 2e-7 on average, 7e-6 at most), each element of the diagonal relative to itself.
|
||||
double num = 0, den = 0, worst_diagonal = 0;
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
num += (hv_gpu[i] - hv[i]) * (hv_gpu[i] - hv[i]);
|
||||
den += hv[i] * hv[i];
|
||||
if (diagonal[i] > 0)
|
||||
worst_diagonal = std::max(worst_diagonal, std::fabs(diagonal_gpu[i] - diagonal[i]) / diagonal[i]);
|
||||
}
|
||||
CAPTURE(std::sqrt(num / den), worst_diagonal);
|
||||
CHECK(std::sqrt(num / den) < 1e-5);
|
||||
CHECK(worst_diagonal < 1e-5);
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("RefinementXray_gradient_GPU", "[Refinement]") {
|
||||
if (get_gpu_count() == 0)
|
||||
return;
|
||||
@@ -1062,18 +1188,21 @@ TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") {
|
||||
return d;
|
||||
};
|
||||
|
||||
// the exact diagonal: one step
|
||||
// the exact inverse Hessian: one step
|
||||
RefinementMinimiser exact;
|
||||
std::vector<double> h(n);
|
||||
for (size_t k = 0; k < n; k++)
|
||||
h[k] = 2 * a[k];
|
||||
exact.SetDiagonal(h);
|
||||
exact.SetInverseHessian([&](const std::vector<double> &, std::vector<double> &q) {
|
||||
for (size_t k = 0; k < n; k++)
|
||||
q[k] /= 2 * a[k];
|
||||
});
|
||||
std::vector<double> x(n, 0.0);
|
||||
exact.Minimise(target, x, 1, 0.0, 10.0);
|
||||
CHECK(distance(x) < 1e-9);
|
||||
|
||||
// without it, the memory carried from one call to the next does what one long call does
|
||||
// with the identity, the memory carried from one call to the next does what one long call does
|
||||
RefinementMinimiser split, whole;
|
||||
const RefinementMinimiser::InverseHessian identity = [](const std::vector<double> &, std::vector<double> &) {};
|
||||
split.SetInverseHessian(identity);
|
||||
whole.SetInverseHessian(identity);
|
||||
std::vector<double> xs(n, 0.0), xw(n, 0.0);
|
||||
for (int call = 0; call < 6; call++)
|
||||
split.Minimise(target, xs, 10, 0.0, 10.0);
|
||||
@@ -1083,6 +1212,44 @@ TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") {
|
||||
CHECK(xs[k] == Catch::Approx(xw[k]).margin(1e-12));
|
||||
}
|
||||
|
||||
TEST_CASE("RefinementMinimiser_conjugate_gradients", "[Refinement]") {
|
||||
// a tridiagonal positive definite system, solved to the tolerance with and without a (Jacobi) preconditioner
|
||||
const size_t n = 200;
|
||||
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
|
||||
hv.assign(n, 0.0);
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
hv[k] = (2.0 + 0.01 * k) * v[k];
|
||||
if (k > 0)
|
||||
hv[k] -= v[k - 1];
|
||||
if (k + 1 < n)
|
||||
hv[k] -= v[k + 1];
|
||||
}
|
||||
};
|
||||
std::vector<double> b(n);
|
||||
for (size_t k = 0; k < n; k++)
|
||||
b[k] = std::sin(0.1 * k);
|
||||
auto residual = [&](const std::vector<double> &x) {
|
||||
std::vector<double> hx;
|
||||
product(x, hx);
|
||||
double r = 0, b2 = 0;
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
r += (hx[k] - b[k]) * (hx[k] - b[k]);
|
||||
b2 += b[k] * b[k];
|
||||
}
|
||||
return std::sqrt(r / b2);
|
||||
};
|
||||
std::vector<double> x;
|
||||
ConjugateGradients(product, [](const std::vector<double> &r, std::vector<double> &z) { z = r; }, b, x, 500, 1e-8);
|
||||
CHECK(residual(x) < 1e-8);
|
||||
const int jacobi = ConjugateGradients(product, [&](const std::vector<double> &r, std::vector<double> &z) {
|
||||
z.resize(n);
|
||||
for (size_t k = 0; k < n; k++)
|
||||
z[k] = r[k] / (2.0 + 0.01 * k);
|
||||
}, b, x, 500, 0.1);
|
||||
CHECK(residual(x) < 0.1);
|
||||
CHECK(jacobi < 50);
|
||||
}
|
||||
|
||||
TEST_CASE("Refinement_restrained_recovers_model", "[Refinement]") {
|
||||
const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90);
|
||||
const gemmi::SpaceGroup &sg = *gemmi::find_spacegroup_by_name("P 21 21 21");
|
||||
|
||||
Reference in new issue
Block a user