Merge branch 'rc188-k' into rc188-merge

Refinement: L-BFGS on the Gauss-Newton normal matrix with a per-parameter trust floor and a
per-residue preconditioned CG, one protocol on the CPU and the GPU (no population, two
rounds), the GPU Gauss-Newton restraint product, and CPU twin arms splitting threads only
where a twin law exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-16 21:41:46 +01:00
commit 81fb4eafd1
13 files changed
+1353 -237

No files matched your search

+13 -7
View File
@@ -598,12 +598,17 @@ Cryst. A34, 791-809 [doi:10.1107/S0567739478001618](https://doi.org/10.1107/S056
memory BFGS method for large scale optimization" (1989), Math. Program. 45, 503-528
[doi:10.1007/BF01589116](https://doi.org/10.1007/BF01589116); its two-loop recursion: J. Nocedal,
"Updating quasi-Newton matrices with limited storage" (1980), Math. Comp. 35, 773-782
[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). The
initial inverse Hessian is rescaled at each iteration by the latest pair, the self-scaling of
S. S. Oren and D. G. Luenberger, "Self-scaling variable metric (SSVM) algorithms. Part I" (1974),
Management Sci. 20, 845-862 [doi:10.1287/mnsc.20.5.845](https://doi.org/10.1287/mnsc.20.5.845), and
D. F. Shanno and K.-H. Phua, "Matrix conditioning and nonlinear optimization" (1978), Math. Program.
14, 149-160 [doi:10.1007/BF01588962](https://doi.org/10.1007/BF01588962).
[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). Its
initial inverse Hessian is the target's sparse normal matrix - the X-ray target's curvature per atom
on the diagonal and the restraints' Gauss-Newton matrix in full - as REFMAC refines on its normal
matrix (Murshudov, Vagin & Dodson 1997, above), solved by conjugate gradients: M. R. Hestenes and
E. Stiefel, "Methods of conjugate gradients for solving linear systems" (1952), J. Res. Natl. Bur.
Stand. 49, 409-436 [doi:10.6028/jres.049.044](https://doi.org/10.6028/jres.049.044). Each atom's
diagonal is raised to hold its step within the step bound, a trust region per atom after the damping
of K. Levenberg, "A method for the solution of certain non-linear problems in least squares" (1944),
Q. Appl. Math. 2, 164-168 [doi:10.1090/qam/10666](https://doi.org/10.1090/qam/10666), and D. W.
Marquardt, "An algorithm for least-squares estimation of nonlinear parameters" (1963), J. Soc. Ind.
Appl. Math. 11, 431-441 [doi:10.1137/0111030](https://doi.org/10.1137/0111030).
**Geometry restraints** — those of the CCP4 Monomer Library, a subset of which rugnux carries (see
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). A. A. Vagin, R. A. Steiner, A. A. Lebedev,
@@ -834,7 +839,8 @@ pseudorandom number generators" (2014), Proc. OOPSLA 2014, 453-472
[doi:10.1145/2660193.2660195](https://doi.org/10.1145/2660193.2660195).
**Error-free floating-point arithmetic** — the single-precision exp shared between the CPU and an
Apple GPU (`BraggExp.h`): the argument reduction of W. J. Cody and W. Waite, "Software Manual for the
Apple GPU (`BraggExp.h`) and the double-float sums of the refinement's Gauss-Newton products on the GPU
(`RefinementNewtonGPU.cu`): the argument reduction of W. J. Cody and W. Waite, "Software Manual for the
Elementary Functions" (1980), Prentice-Hall; the exact product (with Veltkamp's split) and Fast2Sum
of T. J. Dekker, "A floating-point technique for extending the available precision" (1971), Numer.
Math. 18, 224-242; and the 2Sum of D. E. Knuth, "The Art of Computer Programming, Vol. 2:
+2 -2
View File
@@ -66,12 +66,13 @@ The methods draw on, and in places reimplement, solutions from:
- H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", *SIAM J. Numer. Anal.* **49** (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", *J. ACM* **12** (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6).
- R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", *Biometrika* **10** (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared).
- R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", *Acta Cryst.* **A34** (1978), 791-809 (the gradient of the refinement target, §15).
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15).
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target, the B restraints and the minimiser's sparse normal matrix, §15).
- V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", *Acta Cryst.* **A51** (1995), 880-887 (the likelihood parameters from the free set, §15).
- A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15).
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (the weight chosen among refinements by R-free, §15).
- D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15).
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6).
- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6).
- G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5).
- P. Roversi, E. Blanc, S. Johnson & S. M. Lea, "Tetartohedral twinning could happen to you too", *Acta Cryst.* **D68** (2012), 418-424 (twin refinement with more than two domains, and the free set closed under the twin domains, §15.12).
- K. A. Sultana, I. Alexeev, I. Kursula, P. Mäntsälä, J. Niemi & G. Schneider, "Structure determination by multiwavelength anomalous diffraction of aclacinomycin oxidoreductase: indications of multidomain pseudomerohedral twinning", *Acta Cryst.* **D63** (2007), 149-159 (a six-domain twin of a monoclinic crystal on a hexagonal metric, §15.12).
@@ -120,7 +121,6 @@ The methods draw on, and in places reimplement, solutions from:
- V. Luzzati, "Traitement statistique des erreurs dans la détermination des structures cristallines", *Acta Cryst.* **5** (1952), 802-810 (the fall-off of $\sigma_A$ with resolution for a model's coordinate error, Luzzati's $D$, in the likelihood of a placement, §14.10).
- D. Liebschner, P. V. Afonine, A. G. Urzhumtsev & P. D. Adams, "Implementation of the riding hydrogen model in CCTBX to support the next generation of X-ray and neutron joint refinement in Phenix", *Methods Enzymol.* **634** (2020), 177-199 (the riding hydrogens, placed by GEMMI's code, §15.2).
- D. L. Theobald, "Rapid calculation of RMSDs using a quaternion-based characteristic polynomial", *Acta Cryst.* **A61** (2005), 478-480, and P. Liu, D. K. Agrafiotis & D. L. Theobald, *J. Comput. Chem.* **31** (2010), 1561-1563 (the QCP superposition behind the NCS rms criterion, via GEMMI, §15.5).
- S. S. Oren & D. G. Luenberger, "Self-scaling variable metric (SSVM) algorithms. Part I", *Management Sci.* **20** (1974), 845-862, and D. F. Shanno & K.-H. Phua, "Matrix conditioning and nonlinear optimization", *Math. Program.* **14** (1978), 149-160 (the rescaling of the L-BFGS initial inverse Hessian by the latest pair, §15.6).
(list is not exhaustive; the full citations, with DOIs, are in [ACKNOWLEDGEMENT.md](ACKNOWLEDGEMENT.md))
+50 -21
View File
@@ -286,26 +286,51 @@ Whether the restraints are used at all, and how tightly ($\sigma$), the weight t
### 15.6 The minimiser
Positions and B factors are minimised in turn, each by an L-BFGS of rugnux's own (`RefinementMinimiser`),
with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a backtracking line
search from the full step to sufficient decrease, its first trial moving no position by more than 0.3 Å and
no B by more than 10 Ų. What sets it apart from a textbook L-BFGS:
Positions, B factors, anisotropic U and TLS parameters are minimised in turn, each by an L-BFGS of rugnux's own
(`RefinementMinimiser`), with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a
backtracking line search from the full step to sufficient decrease, its first trial moving no position by more
than 0.3 Šand no B by more than 10 Ų. What sets it apart from a textbook L-BFGS is its **initial inverse
Hessian: the target's Gauss-Newton normal matrix**, as REFMAC refines on its sparse normal matrix (Murshudov,
Vagin & Dodson 1997). The matrix holds what is known of the curvature in advance; the L-BFGS pairs add what it
leaves out.
- **Its memory is kept from one macro-cycle to the next.** A macro-cycle is a few dozen iterations on a target
whose scale and likelihood parameters were refitted in between; starting each from what the earlier ones
learnt of the curvature, instead of from a steepest-descent step, is what lets two macro-cycles of a
weight trial get as far as many restarted ones. Each weight trial starts with an empty memory.
- **The B steps are scaled by their curvature.** The initial inverse Hessian of the B factors is a
diagonal: the X-ray target's curvature in each atom's B - its shape from the likelihood's curvature in
$|F_\mathrm{model}|$ and the atom's own scattering, summed in resolution shells, calibrated once per trial
against central differences of the gradient along a random $\pm1$ direction - plus the B restraints'
curvature, each element at least a thousandth of the median. The positions take none: their restraints'
curvature is that of stiff bonds and angles, which move atoms together, and as a diagonal it slows the soft
(torsion and X-ray) directions rather than scaling them.
- **The restraints' part is their Gauss-Newton matrix $J^\mathsf{T}J$, in full.** Every restraint is the square of
a residual (or close to it: the NCS distances' robust loss), so its gradient is $2r\,\nabla r$ and
$J = \nabla E / \sqrt{2E}$ is its row of the Jacobian, $\sqrt2\,\nabla r$; a plane contributes one row per
atom. $J^\mathsf{T}J$ is positive semi-definite where the exact Hessian is not (torsions, repulsions), and near
the ideal geometry it is the exact Hessian. Its products $J^\mathsf{T}(Jv)$ are two gathers - over the rows,
then over each atom's entries - on the slot's threads, or on its GPU.
- **The X-ray part is per atom, on the diagonal.** The X-ray target's curvature in an atom's $x$, $y$ and $z$ is
the likelihood's curvature in $|F_\mathrm{model}|$ summed in resolution shells against the atom's own part of
it, $(2\pi)^2 s^2/3\,f(s)^2\,e^{-Bs^2/2}$ (the reflections taken as spread evenly over directions); in its B,
$(s^2/4)^2 f(s)^2 e^{-Bs^2/2}$, a third of it to each element of a U; and a TLS body's dense $20\times20$ block
is that curvature carried through the derivatives of its atoms' U in the body's parameters. Each is calibrated
once per trial against central differences of the gradient along a random $\pm1$ direction. The coupling of
neighbouring atoms through the X-ray target is not in it; the pairs learn it.
- **The anisotropic U restraints** are quadratic in U, so their Hessian is exact by one difference of their
gradient.
- **A trust floor per atom.** Where nothing but the X-ray target curves a mode - a high-B side chain turning about
a single bond, which the restraints leave free and the X-ray target hardly sees - the matrix's inverse would
make a step of tens of ångströms there, and the step bound would then shrink every other atom's step with it.
So each atom's diagonal is raised, once per macro-cycle, to at least $|g_i|/\Delta$ ($g_i$ its gradient,
$\Delta$ the step bound): its step is held by its own curvature, a trust region per atom in the manner of
Levenberg and Marquardt's damping. A U (per atom) and a TLS body's parameters (each) take the same floor. Without it a rounding difference of $10^{-7}$ in the gradient moved a weight
trial's R-free by up to 0.004; with it the perturbation stays where it was.
- **The matrix is solved, not inverted.** Each L-BFGS direction solves the normal equations by a fixed 20
conjugate-gradient iterations - not to a tolerance, whose iteration count, and with it the direction, a rounding
difference can flip - preconditioned per residue for the positions (the residue's own $J^\mathsf{T}J$ and X-ray
diagonal, by Cholesky: the stiff bonds and angles inside a residue are solved there, what couples residues is
left to the iterations) and by the diagonal elsewhere. The Jacobian is taken again at each point the minimiser
stands on.
- **Its memory is kept from one macro-cycle to the next.** Each weight trial starts with an empty memory.
The diagonal is rescaled at each iteration by the latest pair (Oren-Shanno scaling in that metric). A solve
stops after its iteration budget (30 for the positions and 20 for the B factors per macro-cycle) or where an
iteration lowers the target by less than $10^{-7}$ of it.
A solve stops after its iteration budget (10 for the positions and 7 for the B factors, U or TLS per
macro-cycle) or where an iteration lowers the target by less than $10^{-7}$ of it. Against the plain L-BFGS this
replaced (a diagonal initial inverse Hessian for the B factors, none for the positions, 30 and 20 iterations), a
trial takes about a third of the evaluations. With it the population (§15.7) no longer gains, and the protocol
is the same on the CPU and the GPU - two rounds, no population - at a third of the evaluations the GPU's protocol
took before and R-free 0.0014 lower in the mean over 19 refinements, and at fewer evaluations than the CPU's
former one-round protocol.
### 15.7 The weights, decided by trial
@@ -337,8 +362,10 @@ are:
stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model,
whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the
trajectory ends where it began.
3. **The population**, from where the trajectory ended: twelve members (at most as many as the trial slots, so
that they run in one wave; none below two) refined side by side, each a trial at the
3. **The population** (off by default, `RefinementSettings::population`: with the minimiser of §15.6 its members
took nothing off R-free over 19 refinements that the second round did not), from where the trajectory ended:
up to twelve members (at most as many as the trial slots, so that they run in one wave; none below two) refined
side by side, each a trial at the
trajectory's weights but on its own path - the positions as they are or kicked at random by up to 0.2-0.3 Å
in each coordinate, on the data as they are or **blurred** along a schedule, 60 then 20 Ų (the landscape of
4 Å, then of 2.2 Å), 100 then 30, or 30 then 10. The blur raises the likelihood's variance of each reflection
@@ -741,7 +768,7 @@ What the refinement does not do yet - each a known gap, not a design choice:
- L. Piela, J. Kostrowicki & H. A. Scheraga, "On the multiple-minima problem in the conformational analysis of molecules: deformation of the potential energy hypersurface by the diffusion equation method", *J. Phys. Chem.* **93** (1989), 3339-3346, and A. Jack & M. Levitt, "Refinement of large structures by simultaneous minimization of energy and R factor", *Acta Cryst.* **A34** (1978), 931-935 (the members refined on blurred data, §15.7).
- J. Praznikar, P. V. Afonine, G. Guncar, P. D. Adams & D. Turk, "Averaged kick maps: less noise, more signal... and probably less bias", *Acta Cryst.* **D65** (2009), 921-931, and J. C. Nwachukwu, M. R. Southern, J. R. Kiefer, P. V. Afonine, P. D. Adams, T. C. Terwilliger & K. W. Nettles, "Improved crystallographic structures using extensive combinatorial refinement", *Structure* **21** (2013), 1923-1930 (the kicked members, their averaged map and the crossover residue by residue, §15.7).
- J. Desmet, M. De Maeyer, B. Hazes & I. Lasters, "The dead-end elimination theorem and its use in protein side-chain positioning", *Nature* **356** (1992), 539-542; J. Besag, "On the statistical analysis of dirty pictures", *J. R. Statist. Soc.* **B48** (1986), 259-279; R. P. Joosten, K. Joosten, S. X. Cohen, G. Vriend & A. Perrakis, "Automatic rebuilding and optimization of crystallographic structures in the Protein Data Bank", *Bioinformatics* **27** (2011), 3392-3398 (the side-chain field: states, the conditional-modes solver and its order, §15.5).
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8).
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8; the minimiser's sparse normal matrix, §15.6).
- G. N. Murshudov, P. Skubák, A. A. Lebedev, N. S. Pannu, R. A. Steiner, R. A. Nicholls, M. D. Winn, F. Long & A. A. Vagin, "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (D from the working set and the variance from the free, §15.3; the Kullback-Leibler form of the B restraints, §15.5; the twinned refinement §15.12 simplifies, its twin laws pruned by fraction and its fractions refined by least squares).
- R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", *Acta Cryst.* **A42** (1986), 140-149 (the m, D weighting of the map coefficients, §15.8).
- T. O. Yeates, "Detecting and overcoming crystal twinning", *Methods Enzymol.* **276** (1997), 344-358 (the twinned intensity, §15.12).
@@ -762,6 +789,8 @@ What the refinement does not do yet - each a known gap, not a design choice:
- Z. Dauter & M. Dauter, "Entering a new phase: using solvent halide ions in protein structure determination", *Structure* **9** (2001), R21-R26 (halides at sites with few donors, §15.10).
- N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", *Acta Cryst.* **D70** (2014), 1104-1114 (a light ion flagged by its density above the waters', §15.10).
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782, and D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15.6).
- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6).
- K. Levenberg, "A method for the solution of certain non-linear problems in least squares", *Q. Appl. Math.* **2** (1944), 164-168, and D. W. Marquardt, "An algorithm for least-squares estimation of nonlinear parameters", *J. Soc. Ind. Appl. Math.* **11** (1963), 431-441 (the minimiser's trust floor per atom, §15.6).
- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7).
- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free).
@@ -37,6 +37,9 @@ ADD_LIBRARY(JFJochStructureRefinement STATIC
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementGPU.cpp>
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementGPU.cu>
RefinementGPU.h RefinementGPUEngine.h
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementNewtonGPU.cpp>
$<$<BOOL:${JFJOCH_CUDA_AVAILABLE}>:RefinementNewtonGPU.cu>
RefinementNewtonGPU.h
${CMAKE_CURRENT_BINARY_DIR}/MonomerLibraryData.cpp)
# The subset of the CCP4 Monomer Library that refinement restrains with (monomers/, chosen by
@@ -18,6 +18,7 @@
#include <thread>
#include <unordered_map>
#include <Eigen/Dense>
#include <ceres/gradient_problem.h>
#include <ceres/gradient_problem_solver.h>
#include <ceres/jet.h>
@@ -47,6 +48,7 @@
#ifdef JFJOCH_USE_CUDA
#include "../../common/CUDAWrapper.h"
#include "RefinementGPU.h"
#include "RefinementNewtonGPU.h"
#endif
namespace {
@@ -607,7 +609,12 @@ struct Context {
// The X-ray target's curvature in each atom's B, up to one factor (CalibrateBCurvature): the likelihood's
// curvature in |F_model| of the cycle, 2 (k_total D)^2 / var, summed in resolution shells, against the atom's own
// part of it, (s^2 / 4)^2 f(s)^2 exp(-B s^2 / 2). A hydrogen's is its parent's.
std::vector<double> XrayBCurvatureShape() const {
std::vector<double> XrayBCurvatureShape() const { return XrayCurvatureShape(rows, false); }
// The same for each of an atom's x, y and z: its own part of the likelihood's curvature in |F_model| is then
// (2 pi)^2 s^2 / 3 f(s)^2 exp(-B s^2 / 2), the reflections taken as spread evenly over directions - the X-ray
// block of the normal matrix as a multiple of I per atom. `r`: the row model the engine has (blurred or not).
std::vector<double> XrayXyzCurvatureShape(const XrayRowModel &r) const { return XrayCurvatureShape(r, true); }
std::vector<double> XrayCurvatureShape(const XrayRowModel &rows, bool xyz) const {
constexpr int SHELLS = 32;
double s2_max = 0;
for (double s2 : refl.inv_d2)
@@ -638,7 +645,10 @@ struct Context {
for (int k = 0; k < SHELLS; k++)
if (n[k] > 0) {
const double s2 = s2_sum[k] / n[k];
shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * s2 / 16.0;
if (xyz)
shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * (4.0 * M_PI * M_PI / 3.0);
else
shape[i] += w[k] * f[k] * a.occ * a.occ * std::exp(-a.b_iso * s2 / 2.0) * s2 * s2 / 16.0;
}
}
for (size_t i = 0; i < rider_parent.size(); i++)
@@ -675,6 +685,37 @@ struct Context {
return den > 0 ? std::max(num / den, 0.0) : 0.0;
}
// The factor that turns XrayXyzCurvatureShape() into the X-ray target's curvature, as CalibrateBCurvature() does
// for B: along a random +-1 direction of the positions of `params`, by central differences of the gradient.
// Two evaluations, counted.
double CalibrateXyzCurvature(const std::vector<int> &params, const std::vector<double> &shape) {
constexpr double EPS = 0.01; // A
const std::vector<gemmi::Position> p0 = Positions();
std::mt19937 rng(7);
std::vector<gemmi::Vec3> v(params.size());
for (gemmi::Vec3 &x : v)
x = gemmi::Vec3(rng() % 2 == 0 ? 1.0 : -1.0, rng() % 2 == 0 ? 1.0 : -1.0, rng() % 2 == 0 ? 1.0 : -1.0);
std::array<std::vector<std::array<double, 4>>, 2> g;
for (int side = 0; side < 2; side++) {
for (size_t k = 0; k < params.size(); k++)
atoms[params[k]]->pos = p0[params[k]] + gemmi::Position(v[k] * (side == 0 ? EPS : -EPS));
MoveRiders();
engine->Evaluate(Model(), &g[side]);
evaluations++;
FoldRiders(g[side]);
}
SetPositions(p0);
MoveRiders();
double num = 0, den = 0;
for (size_t k = 0; k < params.size(); k++) {
const int i = params[k];
num += (v[k].x * (g[0][i][0] - g[1][i][0]) + v[k].y * (g[0][i][1] - g[1][i][1]) +
v[k].z * (g[0][i][2] - g[1][i][2])) / (2 * EPS);
den += 3 * shape[i];
}
return den > 0 ? std::max(num / den, 0.0) : 0.0;
}
// F_model with the cycle's scale and solvent, for the atoms as they are now.
std::vector<std::complex<float>> FModel() {
engine->Fcalc(Model(), fcalc);
@@ -792,6 +833,10 @@ public:
return std::isfinite(*cost);
}
Mode GetMode() const { return mode_; }
double Weight() const { return weight_; }
const std::vector<int> &Atoms() const { return param_atoms_; }
private:
const SpecialPosition *Special(int atom) const {
const int k = topo_.special_index.empty() ? -1 : topo_.special_index[atom];
@@ -874,6 +919,20 @@ public:
}
}
double MaxStep() const { return MAX_STEP_B; }
// The parameter k of TLS body `body` over its value in the body's own units.
double ParamScale(size_t body, int k) const { return Scale(body, k); }
// Mode::U: the U restraints' gradient alone at p, as Evaluate() makes it (zero where there are none).
void RestraintGradient(const double *p, double *gradient) const {
Set(p);
std::fill(gradient, gradient + 6 * param_atoms_.size(), 0.0);
if (restraints_ == nullptr)
return;
std::vector<std::array<double, 6>> gr(ctx_.atoms.size(), {0, 0, 0, 0, 0, 0});
EvaluateURestraints(*restraints_, ctx_.Positions(), ctx_.UOfAtoms(), &gr);
for (size_t k = 0; k < param_atoms_.size(); k++)
for (int c = 0; c < 6; c++)
gradient[6 * k + c] = gr[param_atoms_[k]][c] / gemmi::u_to_b();
}
bool Evaluate(const double *p, double *cost, double *gradient) const {
const RefinementStage stage(mode_ == Mode::U ? "eval.u" : "eval.tls");
@@ -934,40 +993,380 @@ private:
std::vector<double> radius_;
};
// The minimiser's diagonal for the B factors of `param_atoms` (RefinementMinimiser::SetDiagonal): weight times the
// X-ray target's curvature in B plus the B restraints', each element at least a thousandth of the median. The
// positions take none: their restraints' curvature is that of stiff bonds and angles, which move atoms together, so
// as a diagonal it slows rather than scales the minimiser.
std::vector<double> BDiagonal(const std::vector<double> &xray, const std::vector<double> &restraints,
const std::vector<int> &param_atoms, double weight) {
std::vector<double> h;
for (int i : param_atoms)
h.push_back(weight * xray[i] + restraints[i]);
if (h.empty()) // no atom the dictionary describes
return h;
std::vector<double> sorted = h;
// Each element at least a thousandth of the median (a parameter the target does not see still takes a finite step).
std::vector<double> FlooredDiagonal(std::vector<double> d) {
if (d.empty())
return d;
std::vector<double> sorted = d;
std::nth_element(sorted.begin(), sorted.begin() + sorted.size() / 2, sorted.end());
const double floor = std::max(1e-3 * sorted[sorted.size() / 2], 1e-12);
for (double &v : h)
for (double &v : d)
v = std::max(v, floor);
return h;
return d;
}
// `minimiser` carries its memory from the call before; `diagonal` is the curvature it scales the steps by (none: 1).
// The minimisers' initial inverse Hessian (RefinementMinimiser::SetInverseHessian) is the Gauss-Newton normal matrix of
// the target, as REFMAC refines on its sparse normal matrix: the X-ray target's curvature per atom on the diagonal and
// the restraints' J^T J in full, solved for each direction by a fixed number of conjugate-gradient iterations - not to a
// tolerance, whose iteration count, and with it the direction, a rounding difference can flip.
// Following Murshudov, Vagin & Dodson (1997) Acta Cryst. D53, 240-255
constexpr int NORMAL_CG_ITERATIONS = 20;
// `target` by `minimiser` (its memory from the call before) from where it is, H0 the inverse of the normal matrix: at(x, q)
// takes the matrix at x (where the minimiser stands; q the vector H0 is about to be applied to), then hv = H v
// (product) and z = M^-1 r for a preconditioner M of it (precondition).
template <typename Target>
void Minimise(Context &ctx, const Target &target, int iterations, RefinementMinimiser &minimiser,
std::vector<double> diagonal = {}) {
void MinimiseWithNormalMatrix(Context &ctx, const Target &target, RefinementMinimiser &minimiser, int iterations,
const std::function<void(const std::vector<double> &, const std::vector<double> &)> &at,
const std::function<void(const std::vector<double> &, std::vector<double> &)> &product,
const std::function<void(const std::vector<double> &, std::vector<double> &)> &precondition) {
std::vector<double> p = target.Get();
if (p.empty())
return;
minimiser.SetDiagonal(diagonal.empty() ? std::vector<double>(p.size(), 1.0) : std::move(diagonal));
minimiser.SetInverseHessian([&](const std::vector<double> &x, std::vector<double> &q) {
at(x, q);
std::vector<double> z;
ConjugateGradients(product, precondition, q, z, NORMAL_CG_ITERATIONS, 0.0, ctx.nthreads);
q.swap(z);
});
minimiser.Minimise([&](const std::vector<double> &x, double &f, std::vector<double> &g) {
return target.Evaluate(x.data(), &f, g.data());
},
p, iterations, MINIMISER_F_TOLERANCE, target.MaxStep(), [&]() { return ctx.Cancelled(); });
minimiser.SetInverseHessian(nullptr);
target.Set(p.data()); // where the minimiser ended, not where its last line search probed
}
// The restraints' Gauss-Newton normal matrix J^T J (RestraintGaussNewton) on the x y z or the B of `param_atoms`, at the
// point last given to At(); an atom that is not a parameter (a rider, an atom held) drops out of it. With the engine on
// a GPU, on its card (RestraintNormalMatrixGPU).
class RestraintNormalMatrix {
public:
RestraintNormalMatrix(const Context &ctx, const std::vector<int> &param_atoms, bool xyz)
: per_(xyz ? 3 : 1), c0_(xyz ? 0 : 3), n_params_(param_atoms.size()), nthreads_(ctx.nthreads),
param_of_(ctx.atoms.size(), -1) {
for (size_t k = 0; k < param_atoms.size(); k++)
param_of_[param_atoms[k]] = static_cast<int>(k);
#ifdef JFJOCH_USE_CUDA
if (ctx.engine->OnGPU())
gpu_ = std::make_unique<RestraintNormalMatrixGPU>(ctx.device, param_atoms, ctx.atoms.size(), xyz);
#endif
}
void At(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
const RefinementStage stage("normal.jacobian");
#ifdef JFJOCH_USE_CUDA
if (gpu_)
return gpu_->At(set, xyzb);
#endif
jac_ = RestraintGaussNewton(set, xyzb);
const size_t rows = jac_.start.size() - 1;
row_of_.resize(jac_.atom.size());
param_.resize(jac_.atom.size());
for (size_t e = 0; e < jac_.atom.size(); e++)
param_[e] = param_of_[jac_.atom[e]];
for (size_t r = 0; r < rows; r++)
for (int e = jac_.start[r]; e < jac_.start[r + 1]; e++)
row_of_[e] = static_cast<int>(r);
// per parameter atom its entries (CSR), for the transpose as a gather
entry_start_.assign(n_params_ + 1, 0);
for (int a : jac_.atom)
if (param_of_[a] >= 0)
entry_start_[param_of_[a] + 1]++;
for (size_t k = 0; k < n_params_; k++)
entry_start_[k + 1] += entry_start_[k];
entry_.resize(entry_start_.back());
std::vector<int> fill(entry_start_.begin(), entry_start_.end() - 1);
for (size_t e = 0; e < jac_.atom.size(); e++)
if (const int k = param_of_[jac_.atom[e]]; k >= 0)
entry_[fill[k]++] = static_cast<int>(e);
t_.resize(rows);
}
// hv = J^T J v: J v per row, then per parameter atom the sum over its entries
void Multiply(const std::vector<double> &v, std::vector<double> &hv) {
const RefinementStage stage("normal.product");
#ifdef JFJOCH_USE_CUDA
if (gpu_)
return gpu_->Multiply(v, hv);
#endif
const int rows = static_cast<int>(t_.size());
ParallelChunks(rows, ThreadsForWork(rows, nthreads_, 16384), [&](int lo, int hi) {
for (int r = lo; r < hi; r++) {
double dot = 0;
for (int e = jac_.start[r]; e < jac_.start[r + 1]; e++)
if (const int k = param_[e]; k >= 0)
for (int c = 0; c < per_; c++)
dot += jac_.j[e][c0_ + c] * v[per_ * k + c];
t_[r] = dot;
}
});
hv.assign(per_ * n_params_, 0.0);
const int n = static_cast<int>(n_params_);
ParallelChunks(n, ThreadsForWork(n, nthreads_, 16384), [&](int lo, int hi) {
for (int k = lo; k < hi; k++)
for (int q = entry_start_[k]; q < entry_start_[k + 1]; q++) {
const int e = entry_[q];
for (int c = 0; c < per_; c++)
hv[per_ * k + c] += t_[row_of_[e]] * jac_.j[e][c0_ + c];
}
});
}
std::vector<double> Diagonal() const {
#ifdef JFJOCH_USE_CUDA
if (gpu_)
return gpu_->Diagonal();
#endif
std::vector<double> d(per_ * n_params_, 0.0);
for (size_t k = 0; k < n_params_; k++)
for (int q = entry_start_[k]; q < entry_start_[k + 1]; q++)
for (int c = 0; c < per_; c++)
d[per_ * k + c] += jac_.j[entry_[q]][c0_ + c] * jac_.j[entry_[q]][c0_ + c];
return d;
}
private:
int per_, c0_;
size_t n_params_, nthreads_;
std::vector<int> param_of_;
RestraintJacobian jac_;
std::vector<int> row_of_, param_, entry_start_, entry_; // per entry its row and its parameter atom (-1: none)
std::vector<double> t_;
#ifdef JFJOCH_USE_CUDA
std::unique_ptr<RestraintNormalMatrixGPU> gpu_;
#endif
};
// The positions' preconditioner: per residue the dense block of H over its atoms' x y z - the restraints within the
// residue (J^T J of the rows whose atoms it holds, at the point given) and the X-ray diagonal `h` - by its Cholesky
// factor. What couples residues (peptide bonds, NCS, contacts) is left to the conjugate gradients; the stiff bonds
// and angles within a residue, which a diagonal or a block per atom leaves to them too, are solved here.
class ResidueBlocks {
public:
ResidueBlocks(Context &ctx, const std::vector<int> &param_atoms, const RestraintSet &set, const std::vector<double> &h) {
std::vector<int> residue_of; // per atom, in model order
for (const gemmi::Chain &ch : ctx.Model().chains)
for (const gemmi::Residue &r : ch.residues) {
residue_of.insert(residue_of.end(), r.atoms.size(), static_cast<int>(params_.size()));
params_.emplace_back();
}
std::vector<int> local(param_atoms.size()), group(param_atoms.size()), param_of(ctx.atoms.size(), -1);
for (size_t k = 0; k < param_atoms.size(); k++) {
group[k] = residue_of[param_atoms[k]];
local[k] = static_cast<int>(params_[group[k]].size());
params_[group[k]].push_back(static_cast<int>(k));
param_of[param_atoms[k]] = static_cast<int>(k);
}
std::vector<Eigen::MatrixXd> block(params_.size());
for (size_t g = 0; g < params_.size(); g++) {
const int m = 3 * static_cast<int>(params_[g].size());
block[g] = Eigen::MatrixXd::Zero(m, m);
for (size_t l = 0; l < params_[g].size(); l++)
for (int c = 0; c < 3; c++)
block[g](3 * l + c, 3 * l + c) = h[3 * params_[g][l] + c];
}
const RestraintJacobian jac = RestraintGaussNewton(set, ctx.XyzB());
for (size_t row = 0; row + 1 < jac.start.size(); row++)
for (int e1 = jac.start[row]; e1 < jac.start[row + 1]; e1++)
for (int e2 = jac.start[row]; e2 < jac.start[row + 1]; e2++) {
const int k1 = param_of[jac.atom[e1]], k2 = param_of[jac.atom[e2]];
if (k1 < 0 || k2 < 0 || group[k1] != group[k2])
continue;
Eigen::MatrixXd &b = block[group[k1]];
for (int a = 0; a < 3; a++)
for (int c = 0; c < 3; c++)
b(3 * local[k1] + a, 3 * local[k2] + c) += jac.j[e1][a] * jac.j[e2][c];
}
llt_.resize(params_.size());
ParallelFor(static_cast<int>(params_.size()), ctx.nthreads, [&](int g) { llt_[g].compute(block[g]); });
}
void Apply(const std::vector<double> &r, std::vector<double> &z, size_t nthreads) const {
z.resize(r.size());
ParallelFor(static_cast<int>(params_.size()), nthreads, [&](int g) {
const std::vector<int> &p = params_[g];
Eigen::VectorXd rg(3 * p.size());
for (size_t l = 0; l < p.size(); l++)
for (int c = 0; c < 3; c++)
rg(3 * l + c) = r[3 * p[l] + c];
const Eigen::VectorXd zg = llt_[g].solve(rg);
for (size_t l = 0; l < p.size(); l++)
for (int c = 0; c < 3; c++)
z[3 * p[l] + c] = zg(3 * l + c);
});
}
private:
std::vector<std::vector<int>> params_; // per residue its parameter atoms (their index among the parameters)
std::vector<Eigen::LLT<Eigen::MatrixXd>> llt_;
};
// The positions or the B factors: H = diag(xray) (per atom, already times the weight) + the restraints' J^T J of `set`,
// the Jacobian at the point of each direction. The diagonal is raised per atom, once per call, to at least |g_i| over the
// step bound (g_i the atom's part of the first direction's gradient): where nothing else curves the target (a high-B
// side chain turning about a single bond, which the restraints leave free and the X-ray target hardly sees) the step
// is then held to the bound by the atom's own curvature, instead of one such atom scaling down the whole step.
// Preconditioned per residue for the positions (ResidueBlocks), by the diagonal for the B factors.
// The floor a trust region per atom, after Levenberg (1944) Q. Appl. Math. 2, 164-168 and Marquardt (1963) J. SIAM
// 11, 431-441
void MinimiseAtoms(Context &ctx, const AtomTarget &target, RefinementMinimiser &minimiser, const std::vector<double> &xray,
const RestraintSet &set, int iterations) {
const bool xyz = target.GetMode() == AtomTarget::Mode::Xyz;
const int per = xyz ? 3 : 1;
std::vector<double> x_at = target.Get();
const size_t n = x_at.size();
std::vector<double> h(n);
for (size_t k = 0; k < n; k++)
h[k] = xray[target.Atoms()[k / per]];
RestraintNormalMatrix normal(ctx, target.Atoms(), xyz);
target.Set(x_at.data());
normal.At(set, ctx.XyzB());
const std::vector<double> restraint_diagonal = normal.Diagonal();
std::vector<double> diagonal;
std::unique_ptr<ResidueBlocks> residues;
auto at = [&](const std::vector<double> &x, const std::vector<double> &q) {
if (diagonal.empty()) { // the first direction: the floor, then the preconditioner with it
for (size_t a = 0; a < n / per; a++) {
double g2 = 0;
for (int c = 0; c < per; c++)
g2 += q[per * a + c] * q[per * a + c];
for (int c = 0; c < per; c++)
h[per * a + c] = std::max(h[per * a + c], std::sqrt(g2) / target.MaxStep());
}
diagonal.resize(n);
for (size_t k = 0; k < n; k++)
diagonal[k] = restraint_diagonal[k] + h[k];
diagonal = FlooredDiagonal(diagonal);
if (xyz)
residues = std::make_unique<ResidueBlocks>(ctx, target.Atoms(), set, h);
}
if (x != x_at) {
x_at = x;
target.Set(x.data());
normal.At(set, ctx.XyzB());
}
};
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
normal.Multiply(v, hv);
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] += h[k] * v[k]; });
};
auto precondition = [&](const std::vector<double> &r, std::vector<double> &z) {
if (residues)
residues->Apply(r, z, ctx.nthreads);
else {
z.resize(r.size());
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { z[k] = r[k] / diagonal[k]; });
}
};
MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition);
}
// Each atom's U (in 8 pi^2 U): H = diag(xray) per element + the U restraints' Hessian, exact by differences of their
// gradient (they are quadratic in U); the diagonal raised to the trust floor as MinimiseAtoms() raises it;
// preconditioned by `diagonal`.
void MinimiseU(Context &ctx, const AdpTarget &target, RefinementMinimiser &minimiser, std::vector<double> xray,
std::vector<double> diagonal, int iterations) {
const size_t n = xray.size();
std::vector<double> x_at, g_at(n), xp(n), gp(n);
bool first = true;
auto at = [&](const std::vector<double> &x, const std::vector<double> &q) {
if (first) { // the trust floor per atom, as MinimiseAtoms() takes it
first = false;
for (size_t a = 0; a < n / 6; a++) {
double g2 = 0;
for (int c = 0; c < 6; c++)
g2 += q[6 * a + c] * q[6 * a + c];
const double floor = std::sqrt(g2) / target.MaxStep();
for (int c = 0; c < 6; c++) {
diagonal[6 * a + c] += std::max(floor - xray[6 * a + c], 0.0);
xray[6 * a + c] = std::max(xray[6 * a + c], floor);
}
}
}
if (x != x_at) {
x_at = x;
target.RestraintGradient(x.data(), g_at.data());
}
};
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
double v_max = 0;
for (double e : v)
v_max = std::max(v_max, std::fabs(e));
hv.assign(n, 0.0);
if (v_max > 0) {
const double t = 1.0 / v_max;
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { xp[k] = x_at[k] + t * v[k]; });
target.RestraintGradient(xp.data(), gp.data());
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] = (gp[k] - g_at[k]) / t; });
}
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { hv[k] += xray[k] * v[k]; });
};
auto precondition = [&](const std::vector<double> &r, std::vector<double> &z) {
z.resize(n);
refine_vec::Each(n, ctx.nthreads, [&](size_t k) { z[k] = r[k] / diagonal[k]; });
};
MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition);
}
// The TLS bodies: X-ray only, H = sum over each body's atoms of A^T C A, A the derivatives of the atom's U in the body's
// parameters (linear in them), C the X-ray curvature of its U per element (`xray_b`: of its B, a third of it to each);
// one dense block per body, its diagonal raised to the trust floor, preconditioned by its diagonal.
void MinimiseTls(Context &ctx, const AdpTarget &target, RefinementMinimiser &minimiser, const std::vector<double> &xray_b,
int iterations) {
constexpr int P = TlsBody::PARAMETERS;
const size_t nb = ctx.tls.size();
std::vector<std::array<double, P * P>> blocks(nb);
for (size_t b = 0; b < nb; b++) {
std::array<double, P * P> &m = blocks[b];
m.fill(0.0);
for (int i : ctx.tls[b].atoms) {
if (!ctx.rider_parent.empty() && ctx.rider_parent[i] >= 0)
continue; // its curvature is its parent's already
const std::array<std::array<double, 6>, P> a = ctx.tls[b].Derivatives(ctx.atoms[i]->pos);
const double c = gemmi::u_to_b() * gemmi::u_to_b() * xray_b[i] / 3.0;
for (int k = 0; k < P; k++)
for (int l = 0; l < P; l++) {
double s = 0;
for (int e = 0; e < 6; e++)
s += a[k][e] * a[l][e];
m[k * P + l] += c * s / (target.ParamScale(b, k) * target.ParamScale(b, l));
}
}
}
std::vector<double> diagonal(nb * P);
for (size_t b = 0; b < nb; b++)
for (int k = 0; k < P; k++)
diagonal[b * P + k] = blocks[b][k * P + k];
diagonal = FlooredDiagonal(diagonal);
// the trust floor per parameter, as MinimiseAtoms() takes it per atom
bool first = true;
auto at = [&](const std::vector<double> &, const std::vector<double> &q) {
if (!first)
return;
first = false;
for (size_t b = 0; b < nb; b++)
for (int k = 0; k < P; k++) {
const double floor = std::fabs(q[b * P + k]) / target.MaxStep();
diagonal[b * P + k] += std::max(floor - blocks[b][k * P + k], 0.0);
blocks[b][k * P + k] = std::max(blocks[b][k * P + k], floor);
}
};
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
hv.assign(v.size(), 0.0);
for (size_t b = 0; b < nb; b++)
for (int k = 0; k < P; k++)
for (int l = 0; l < P; l++)
hv[b * P + k] += blocks[b][k * P + l] * v[b * P + l];
};
auto precondition = [&](const std::vector<double> &r, std::vector<double> &z) {
z.resize(r.size());
for (size_t k = 0; k < r.size(); k++)
z[k] = r[k] / diagonal[k];
};
MinimiseWithNormalMatrix(ctx, target, minimiser, iterations, at, product, precondition);
}
// Groups of atoms that move as rigid bodies (one per chain) against the X-ray target: per body an angle-axis
// rotation about its centroid, scaled by its rms radius so that a unit moves a typical atom by an angstrom as a
// unit of translation does, then a translation.
@@ -1788,6 +2187,18 @@ double XrayGradientNorm(Context &ctx, const std::vector<int> &refinable) {
return n;
}
// The X-ray curvature per atom in each of x, y, z at weight w, for the row model `rows` the engine has; `scale`
// calibrated at the first call of the trial (negative: not yet).
std::vector<double> XyzCurvature(Context &ctx, const std::vector<int> &refinable, const XrayRowModel &rows, double w,
double &scale) {
std::vector<double> shape = ctx.XrayXyzCurvatureShape(rows);
if (scale < 0)
scale = ctx.CalibrateXyzCurvature(refinable, shape);
for (double &x : shape)
x *= w * scale;
return shape;
}
TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const std::vector<gemmi::Position> &pos0,
const std::vector<double> &b0, const std::vector<AtomAniso> &aniso0, const Context::Fit *fit,
const TrialWeights &tw, int cycles, const RefinementSettings &settings, const NcsCopies *ncs,
@@ -1838,7 +2249,8 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
const AdpTarget tls_target(ctx, AdpTarget::Mode::Tls, refinable, 1.0, nullptr);
const AdpTarget u_target(ctx, AdpTarget::Mode::U, u_atoms, adp_w, &u_restraints);
RefinementMinimiser xyz_minimiser(20, ctx.nthreads), b_minimiser(20, ctx.nthreads), tls_minimiser(20, ctx.nthreads);
double xray_scale = 0;
// the X-ray curvature's calibrations (CalibrateXyzCurvature, CalibrateBCurvature), once per trial
double xyz_scale = -1, b_scale = -1;
for (int cycle = 0; cycle < cycles && !ctx.Cancelled(); cycle++) {
PlaceRiders(ctx, topo);
if (cycle == 0 && fit != nullptr)
@@ -1852,8 +2264,8 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
if (adp_w > 0)
ctx.MoveOverallB();
if (!hold_positions) {
const RestraintSet set = RestraintsNow(ctx, topo, ncs, ncs_sigma);
{
const RestraintSet set = RestraintsNow(ctx, topo, ncs, ncs_sigma);
const RefinementStage s1("set_restraints");
ctx.engine->SetRestraints(0, set);
}
@@ -1870,24 +2282,33 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
ctx.engine->SetRowModel(blurred);
const double nh = XrayGradientNorm(ctx, refinable);
const AtomTarget xyz_h(ctx, AtomTarget::Mode::Xyz, refinable, nh > 0 ? w * std::sqrt(n0 / nh) : w, 0, topo);
Minimise(ctx, xyz_h, settings.xyz_iterations, xyz_minimiser);
MinimiseAtoms(ctx, xyz_h, xyz_minimiser, XyzCurvature(ctx, refinable, blurred, xyz_h.Weight(), xyz_scale),
set, settings.xyz_iterations);
ctx.engine->SetRowModel(ctx.rows);
} else
Minimise(ctx, xyz, settings.xyz_iterations, xyz_minimiser);
MinimiseAtoms(ctx, xyz, xyz_minimiser, XyzCurvature(ctx, refinable, ctx.rows, w, xyz_scale), set,
settings.xyz_iterations);
}
// the X-ray curvature in B for the atoms as they are, which TLS, U and B take a share of
auto b_curvature = [&]() {
std::vector<double> c = ctx.XrayBCurvatureShape();
if (b_scale < 0)
b_scale = ctx.CalibrateBCurvature(refinable, c);
for (double &x : c)
x *= b_scale;
return c;
};
if (kind == AdpHypothesis::Kind::Tls) {
ctx.ApplyTls(); // the bodies' U at the atoms as they are
const RefinementStage s4("minimise.tls");
Minimise(ctx, tls_target, settings.b_iterations, tls_minimiser);
MinimiseTls(ctx, tls_target, tls_minimiser, b_curvature(), settings.b_iterations);
}
if (adp_w > 0 && kind == AdpHypothesis::Kind::Anisotropic) {
// each U's six elements take a third of the curvature of its B
std::vector<double> xray = ctx.XrayBCurvatureShape();
if (cycle == 0)
xray_scale = ctx.CalibrateBCurvature(refinable, xray);
// and the restraints' curvature in 8 pi^2 U: per pair 2 / sigma^2 of the similarity and about a third of
// the rigid bond's, per isotropic atom 2 / sigma^2
std::vector<double> restraint(xray.size(), 0.0);
const std::vector<double> xray_b = b_curvature();
// each U's six elements take a third of the curvature of its B; and the restraints' curvature in 8 pi^2 U:
// per pair 2 / sigma^2 of the similarity and about a third of the rigid bond's, per isotropic atom
// 2 / sigma^2 - the preconditioner's diagonal
std::vector<double> restraint(xray_b.size(), 0.0);
const double u2 = gemmi::u_to_b() * gemmi::u_to_b();
const double per_pair = (2.0 / (URestraints::SIGMA_SIMILAR * URestraints::SIGMA_SIMILAR) +
2.0 / (3.0 * URestraints::SIGMA_RIGID * URestraints::SIGMA_RIGID)) / u2;
@@ -1897,24 +2318,21 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
}
for (int i : u_restraints.isotropic)
restraint[i] += 2.0 / (URestraints::SIGMA_ISOTROPIC * URestraints::SIGMA_ISOTROPIC) / u2;
for (double &x : xray)
x *= xray_scale / 3.0;
std::vector<double> diagonal;
for (double h : BDiagonal(xray, restraint, u_atoms, adp_w))
diagonal.insert(diagonal.end(), 6, h);
std::vector<double> h, diagonal;
for (int i : u_atoms) {
h.insert(h.end(), 6, adp_w * xray_b[i] / 3.0);
diagonal.insert(diagonal.end(), 6, adp_w * xray_b[i] / 3.0 + restraint[i]);
}
const RefinementStage s3("minimise.u");
Minimise(ctx, u_target, settings.b_iterations, b_minimiser, diagonal);
MinimiseU(ctx, u_target, b_minimiser, h, FlooredDiagonal(diagonal), settings.b_iterations);
} else if (adp_w > 0) {
const RestraintSet b_set = BRestraintsNow(ctx, topo);
ctx.engine->SetRestraints(1, b_set);
std::vector<double> xray = ctx.XrayBCurvatureShape();
if (cycle == 0)
xray_scale = ctx.CalibrateBCurvature(refinable, xray);
for (double &x : xray)
x *= xray_scale;
std::vector<double> xray_b = b_curvature();
for (double &x : xray_b)
x *= adp_w;
const RefinementStage s3("minimise.b");
Minimise(ctx, b, settings.b_iterations, b_minimiser,
BDiagonal(xray, BRestraintCurvature(b_set, ctx.XyzB()), refinable, adp_w));
MinimiseAtoms(ctx, b, b_minimiser, xray_b, b_set, settings.b_iterations);
}
}
PlaceRiders(ctx, topo);
@@ -3050,7 +3468,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// the members, whose best on the free set would be the best of N.
// The members are at most as many as the trial slots, one wave: on a large model, where the card holds
// few, more members would cost a wave each.
const size_t population = std::min<size_t>(settings.population, trial_slots.size());
const size_t population = std::min<size_t>(std::max(settings.population, 0), trial_slots.size());
if (population > 1 && state.ran && !ctx.Cancelled()) {
const RefinementStage pop_stage("population");
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
@@ -4236,17 +4654,11 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
"reflections in one set - {} reflections changed set", moved);
}
const std::vector<MergedReflection> &data = moved > 0 ? closed : merged;
// RefinementSettings::lean_on_cpu
#ifdef JFJOCH_USE_CUDA
const bool on_cpu = !settings.gpu || get_gpu_count() == 0;
#else
const bool on_cpu = true;
#endif
if (settings.lean_on_cpu && on_cpu) {
settings.population = 0;
settings.rounds = 1;
logger.Info("Refinement on the CPU: no population and one round (half of the evaluations)");
}
// Each arm, untwinned and twinned, is a whole refinement - the solvent built and the displacement model chosen
// against its own target - and the decision is taken on where they ended. A model only scored is scored both
// ways and decided by the same rule, so the R-free it reports is not an untwinned one by default.
@@ -4260,7 +4672,10 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
if (!settings.twin)
result = RefineOnce(placed, data, cell, sg, wavelength_A, logger, settings);
else {
const bool halves = on_cpu && settings.population <= 1 && settings.nthreads > 1;
// on the CPU each arm on half of the threads - where the lattice admits a twin law, without which the twinned
// arm ends at once and its half would sit idle
const bool halves = on_cpu && settings.population <= 1 && settings.nthreads > 1 &&
!RefinementTwinLaws(cell, sg).empty();
RefinementSettings untwinned_settings = settings, twin_settings = settings;
if (halves) {
twin_settings.nthreads = settings.nthreads / 2;
@@ -48,8 +48,10 @@ struct RefinementSettings {
bool score_only = false; // only score the model as given (RefinementResult::score_start), no stage
bool group_b = true; // the placement stage's B shift per residue
int macro_cycles = 2; // per weight trial, from the model the trial starts from
int xyz_iterations = 30; // L-BFGS iterations per macro-cycle
int b_iterations = 20;
// L-BFGS iterations per macro-cycle (on the Gauss-Newton normal matrix: RefinementMinimiser), of the positions and of
// the B factors, U and TLS
int xyz_iterations = 10;
int b_iterations = 7;
// The first trials, from the placed model: the positions' X-ray weight, times the gradient-balanced one
// (BalancedWeight), each with the B factors' X-ray weight their own balanced one times each of adp_ladder (0: B
// held).
@@ -84,8 +86,9 @@ struct RefinementSettings {
// The population (RefineOnce, 2a.): this many members at most, refined from where the weight trajectory ended in
// one wave, each on the data blurred along a schedule of its own and its positions kicked or not, then crossed
// residue by residue on their averaged map and refined once more; the child kept where the free set prefers it.
// Capped at the trial slots; fewer than two: no population.
int population = 12;
// Capped at the trial slots; fewer than two: no population (the default: with the positions minimised on the
// Gauss-Newton matrix, the members took nothing off R-free over 19 sets that the two rounds did not).
int population = 0;
// A refinement against the data as twinned by each twin law the lattice admits (RefinementTwinLaws), beside the
// untwinned one, and the model of whichever the evidence favours (RefineModel).
bool twin = true;
@@ -97,11 +100,6 @@ struct RefinementSettings {
// model of this crystal scored on a free set drawn here).
bool as_given_competes = true;
bool gpu = true;
// Where the refinement runs on the CPU (no GPU wanted, or none to run on), where an evaluation costs some thirty
// times a GPU's: no population and one round, for about half of the evaluations, at an R-free 0.001 higher in the
// median of 19 sets (at most 0.015, a placed prediction the population helps). The twinned data are still refined
// against: without that arm a crystal 30% twinned is refined as untwinned.
bool lean_on_cpu = true;
size_t nthreads = 1;
size_t gpu_resident_bytes = 0; // held on the card by what runs beside
// The most weight trials run side by side, 0 for as many as the ladders and the cards take; the result does not
@@ -11,29 +11,58 @@
#include "../../common/ParallelFor.h"
// Limited-memory BFGS whose memory outlives one call, so that a refinement's macro-cycles - each a few dozen
// iterations on a target whose scale and likelihood parameters were refitted in between - go on from what the
// earlier ones learnt of the curvature instead of starting each from a steepest-descent step. The initial inverse
// Hessian is a diagonal the caller gives (SetDiagonal: an estimate of the Hessian's diagonal, so that a parameter
// with ten times the curvature of another takes a tenth of its step), scaled each iteration by the latest pair
// (Oren-Shanno in that metric). The line search backtracks from the full step to sufficient decrease.
// Sweeps over the parameters on `nthreads` threads: each element on its own, and dot products folded in fixed
// blocks (ParallelBlocks), so that the bits do not depend on the number of threads.
namespace refine_vec {
// a pass over fewer than this many parameters per thread costs less than waking the threads
constexpr size_t SWEEP_PER_THREAD = 65536;
template <typename Fn>
void Each(size_t n, size_t nthreads, Fn fn) {
ParallelChunks(static_cast<int>(n), ThreadsForWork(n, nthreads, SWEEP_PER_THREAD), [&](int lo, int hi) {
for (int k = lo; k < hi; k++)
fn(static_cast<size_t>(k));
});
}
inline double Dot(const std::vector<double> &a, const std::vector<double> &b, size_t nthreads) {
const int n = static_cast<int>(a.size());
std::vector<double> part(ReductionBlocks(n, SWEEP_PER_THREAD), 0.0);
ParallelBlocks(n, nthreads, [&](int block, int lo, int hi) {
double s = 0;
for (int k = lo; k < hi; k++)
s += a[k] * b[k];
part[block] = s;
}, SWEEP_PER_THREAD);
double s = 0;
for (double v : part)
s += v;
return s;
}
} // namespace refine_vec
// Limited-memory BFGS whose initial inverse Hessian is the caller's (SetInverseHessian: H0 q, here the Gauss-Newton
// normal matrix solved by ConjugateGradients), and whose memory outlives one call, so that a refinement's
// macro-cycles - each a few iterations on a target whose scale and likelihood parameters were refitted in between -
// go on from what the earlier ones learnt. The normal matrix holds what is known of the curvature in advance (the
// restraints exactly, the X-ray target per atom); the pairs add what it leaves out (the X-ray target's coupling of
// neighbouring atoms). The line search backtracks from the full step to sufficient decrease.
// Following Nocedal (1980) Math. Comp. 35, 773-782 and Liu & Nocedal (1989) Math. Program. 45, 503-528
class RefinementMinimiser {
public:
// f(x, g): the target at x, and its gradient in g (sized as x); false where it cannot be evaluated there.
using Target = std::function<bool(const std::vector<double> &x, double &f, std::vector<double> &g)>;
// q = H0 q at x.
using InverseHessian = std::function<void(const std::vector<double> &x, std::vector<double> &q)>;
// `nthreads`: the threads the sweeps over the parameters run on (the dot products stay serial chains).
// `nthreads`: the threads the sweeps over the parameters run on.
explicit RefinementMinimiser(size_t memory = 20, size_t nthreads = 1) : memory_(memory), nthreads_(nthreads) {}
// Forget the pairs (the target changed beyond what the memory still describes, e.g. another weight).
// Forget the pairs (the target changed beyond what the memory still describes).
void Reset() {
s_.clear();
y_.clear();
sy_.clear();
}
// The estimated diagonal of the Hessian; every element > 0.
void SetDiagonal(std::vector<double> h) { h_ = std::move(h); }
void SetInverseHessian(InverseHessian h0) { h0_ = std::move(h0); }
struct Outcome {
int iterations = 0;
@@ -47,8 +76,6 @@ public:
Outcome Minimise(const Target &target, std::vector<double> &x, int iterations, double f_tolerance, double max_step,
const std::function<bool()> &stop = nullptr) {
const size_t n = x.size();
if (h_.size() != n)
h_.assign(n, 1.0);
Outcome out;
double f = 0;
std::vector<double> g(n), d(n), x_new(n), g_new(n);
@@ -59,11 +86,11 @@ public:
for (int it = 0; it < iterations; it++) {
if (stop && stop())
break;
Direction(g, d);
Direction(x, g, d);
double gd = Dot(g, d);
if (!(gd < 0)) { // not a descent direction: the memory no longer fits, start again from the diagonal
if (!(gd < 0)) { // not a descent direction: the memory no longer fits, start again from H0
Reset();
Direction(g, d);
Direction(x, g, d);
gd = Dot(g, d);
if (!(gd < 0))
break;
@@ -75,7 +102,7 @@ public:
double f_new = 0;
bool accepted = false;
for (int ls = 0; ls < 10; ls++) {
Sweep(n, [&](size_t k) { x_new[k] = x[k] + alpha * d[k]; });
refine_vec::Each(n, nthreads_, [&](size_t k) { x_new[k] = x[k] + alpha * d[k]; });
const bool ok = target(x_new, f_new, g_new);
out.evaluations++;
if (ok && f_new <= f + 1e-4 * alpha * gd) {
@@ -88,11 +115,8 @@ public:
}
if (!accepted)
break;
// the pair's vectors are the ones the memory last let go, where it did: made anew, their pages fault in
std::vector<double> s = std::move(spare_s_), y = std::move(spare_y_);
s.resize(n);
y.resize(n);
Sweep(n, [&](size_t k) {
std::vector<double> s(n), y(n);
refine_vec::Each(n, nthreads_, [&](size_t k) {
s[k] = x_new[k] - x[k];
y[k] = g_new[k] - g[k];
});
@@ -102,15 +126,10 @@ public:
y_.push_back(std::move(y));
sy_.push_back(sy);
if (s_.size() > memory_) {
spare_s_ = std::move(s_.front());
spare_y_ = std::move(y_.front());
s_.pop_front();
y_.pop_front();
sy_.pop_front();
}
} else {
spare_s_ = std::move(s);
spare_y_ = std::move(y);
}
const double decrease = f - f_new;
x.swap(x_new);
@@ -125,87 +144,64 @@ public:
}
private:
static double Dot(const std::vector<double> &a, const std::vector<double> &b) {
double s = 0;
for (size_t k = 0; k < a.size(); k++)
s += a[k] * b[k];
return s;
}
double Dot(const std::vector<double> &a, const std::vector<double> &b) const { return refine_vec::Dot(a, b, nthreads_); }
// fn(k) for every k below n on the threads, each element on its own: the serial loop's bits.
template <typename Fn>
void Sweep(size_t n, Fn fn) const {
ParallelChunks(static_cast<int>(n), ThreadsForWork(n, nthreads_), [&](int lo, int hi) {
for (int k = lo; k < hi; k++)
fn(static_cast<size_t>(k));
});
}
// d = -H g by the two-loop recursion, H0 = gamma diag(1/h). Every dot product is a serial chain of n
// additions, so each sweep over the parameters takes the next pair's dot product while it applies the
// previous pair's update to q - the same operations on each element in the same order as a sweep for
// each would do, with the update's memory traffic hidden under the chain.
void Direction(const std::vector<double> &g, std::vector<double> &d) const {
const size_t n = g.size(), m = s_.size();
std::vector<double> &q = q_; // kept from call to call: made anew, its pages fault in every time
q.assign(g.begin(), g.end());
std::vector<double> a(m), rho(m);
for (size_t i = 0; i < m; i++)
rho[i] = 1.0 / sy_[i];
double dot = m > 0 ? Dot(s_[m - 1], q) : 0.0, yhy = 0.0;
// d = -H g by the two-loop recursion, H0 the caller's.
void Direction(const std::vector<double> &x, const std::vector<double> &g, std::vector<double> &d) const {
const size_t m = s_.size(), n = g.size();
std::vector<double> q = g, a(m);
for (size_t i = m; i-- > 0;) {
a[i] = rho[i] * dot;
const double ai = a[i];
const double *y = y_[i].data();
dot = 0.0;
if (i > 0) {
const double *s = s_[i - 1].data();
for (size_t k = 0; k < n; k++) {
q[k] -= ai * y[k];
dot += s[k] * q[k];
}
} else {
const double *yl = y_[m - 1].data();
for (size_t k = 0; k < n; k++) {
q[k] -= ai * y[k];
yhy += yl[k] * yl[k] / h_[k];
}
}
a[i] = Dot(s_[i], q) / sy_[i];
refine_vec::Each(n, nthreads_, [&](size_t k) { q[k] -= a[i] * y_[i][k]; });
}
const double gamma = m > 0 ? sy_[m - 1] / yhy : 1.0;
if (m > 0) {
const double *y = y_[0].data();
for (size_t k = 0; k < n; k++) {
q[k] *= gamma / h_[k];
dot += y[k] * q[k];
}
} else
for (size_t k = 0; k < n; k++)
q[k] *= gamma / h_[k];
h0_(x, q);
for (size_t i = 0; i < m; i++) {
// b on its own and (a[i] - b) in the loop, as before: written as a[i] - rho[i] * dot the compiler
// contracts it into a fused multiply-add, and the bits change
const double b = rho[i] * dot;
const double *s = s_[i].data();
dot = 0.0;
if (i + 1 < m) {
const double *y = y_[i + 1].data();
for (size_t k = 0; k < n; k++) {
q[k] += s[k] * (a[i] - b);
dot += y[k] * q[k];
}
} else
for (size_t k = 0; k < n; k++)
q[k] += s[k] * (a[i] - b);
const double b = Dot(y_[i], q) / sy_[i];
refine_vec::Each(n, nthreads_, [&](size_t k) { q[k] += s_[i][k] * (a[i] - b); });
}
for (size_t k = 0; k < n; k++)
d[k] = -q[k];
d.resize(n);
refine_vec::Each(n, nthreads_, [&](size_t k) { d[k] = -q[k]; });
}
size_t memory_, nthreads_;
InverseHessian h0_;
std::deque<std::vector<double>> s_, y_;
std::deque<double> sy_; // s.y of each pair, as taken when it was kept
std::vector<double> spare_s_, spare_y_; // the pair the memory last let go, for the next one to reuse
mutable std::vector<double> q_; // Direction()'s working vector
std::vector<double> h_;
};
// z with H z = b to `tolerance` of |b| (at most `max_iterations`), by conjugate gradients preconditioned with M:
// hv = H v (product), z = M^-1 r (precondition), both symmetric positive definite. Returns the iterations taken.
// Following Hestenes & Stiefel (1952) J. Res. Natl. Bur. Stand. 49, 409-436
inline int ConjugateGradients(const std::function<void(const std::vector<double> &v, std::vector<double> &hv)> &product,
const std::function<void(const std::vector<double> &r, std::vector<double> &z)> &precondition,
const std::vector<double> &b, std::vector<double> &x, int max_iterations, double tolerance,
size_t nthreads = 1) {
auto dot = [&](const std::vector<double> &u, const std::vector<double> &v) { return refine_vec::Dot(u, v, nthreads); };
const size_t n = b.size();
x.assign(n, 0.0);
std::vector<double> r = b, z(n), p, hp(n);
precondition(r, z);
p = z;
double rz = dot(r, z);
const double r0 = std::sqrt(dot(r, r));
int k = 0;
while (k < max_iterations) {
k++;
product(p, hp);
const double php = dot(p, hp);
if (!(php > 0))
break;
const double a = rz / php;
refine_vec::Each(n, nthreads, [&](size_t j) {
x[j] += a * p[j];
r[j] -= a * hp[j];
});
if (std::sqrt(dot(r, r)) < tolerance * r0)
break;
precondition(r, z);
const double rz_new = dot(r, z);
refine_vec::Each(n, nthreads, [&](size_t j) { p[j] = z[j] + rz_new / rz * p[j]; });
rz = rz_new;
}
return k;
}
@@ -0,0 +1,59 @@
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "RefinementNewtonGPU.h"
#include "RefinementRestraints.h"
// The host half: the index of J, as RestraintGaussNewton lays out its rows, with every row kept.
void RestraintNormalMatrixGPU::At(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
if (!indexed_) {
std::vector<int> record_entry, start{0}, atom;
for (const RestraintRecord &r : set.records) {
record_entry.push_back(static_cast<int>(atom.size()));
switch (static_cast<RestraintKind>(r.kind)) {
case RestraintKind::Plane:
// one row per atom of the plane, each over all of them
for (int i = 0; i < r.atom[1]; i++) {
for (int k = 0; k < r.atom[1]; k++)
atom.push_back(set.plane_atoms[r.atom[0] + k]);
start.push_back(static_cast<int>(atom.size()));
}
continue;
case RestraintKind::Angle:
atom.insert(atom.end(), r.atom, r.atom + 3);
break;
case RestraintKind::Torsion:
case RestraintKind::Chiral:
case RestraintKind::NcsDistance:
atom.insert(atom.end(), r.atom, r.atom + 4);
break;
default: // Bond, Repulsion, BSimilarity
atom.insert(atom.end(), r.atom, r.atom + 2);
break;
}
start.push_back(static_cast<int>(atom.size()));
}
const size_t rows = start.size() - 1;
std::vector<int> row_of(atom.size()), param(atom.size());
for (size_t e = 0; e < atom.size(); e++)
param[e] = param_of_[atom[e]];
for (size_t r = 0; r < rows; r++)
for (int e = start[r]; e < start[r + 1]; e++)
row_of[e] = static_cast<int>(r);
std::vector<int> entry_start(n_params_ + 1, 0);
for (int k : param)
if (k >= 0)
entry_start[k + 1]++;
for (size_t k = 0; k < n_params_; k++)
entry_start[k + 1] += entry_start[k];
std::vector<int> entry(entry_start.back());
std::vector<int> fill(entry_start.begin(), entry_start.end() - 1);
for (size_t e = 0; e < param.size(); e++)
if (param[e] >= 0)
entry[fill[param[e]]++] = static_cast<int>(e);
Upload(set.records, set.plane_atoms, record_entry, start, param, row_of, entry_start, entry);
indexed_ = true;
}
Fill(xyzb);
}
@@ -0,0 +1,319 @@
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
// The device half of RestraintNormalMatrixGPU, in float: one thread per restraint for J, one per row of J or per
// parameter atom for the products, each summing its terms in order. J is held in float; the products' v, t and hv in
// float pairs (hi + lo, double-float arithmetic: an error-free sum and product per term), since the soft directions of
// H - those the X-ray term decides, J v the small difference of large terms - need more than float's 24 bits: with
// plain float sums the Newton steps on a model with stiff restraints differed from the CPU's by more than its noise.
#include "RefinementNewtonGPU.h"
#include <algorithm>
#include "../indexing/CUDAMemHelpers.h"
#include "RefinementGPUEngine.h" // MAX_PLANE_ATOMS
namespace {
constexpr int THREADS = 256;
using V3f = refine_math::V3<float>;
void cuda_err(cudaError_t val) {
if (val != cudaSuccess)
throw JFJochException(JFJochExceptionCategory::GPUCUDAError, cudaGetErrorString(val));
}
class DeviceGuard {
public:
explicit DeviceGuard(int device) {
cuda_err(cudaGetDevice(&previous_));
cuda_err(cudaSetDevice(device));
}
~DeviceGuard() { cudaSetDevice(previous_); }
DeviceGuard(const DeviceGuard &) = delete;
DeviceGuard &operator=(const DeviceGuard &) = delete;
private:
int previous_ = 0;
};
int GreatestStreamPriority() {
int least = 0, greatest = 0;
cuda_err(cudaDeviceGetStreamPriorityRange(&least, &greatest));
return greatest;
}
int GridSize(int n) {
return (n + THREADS - 1) / THREADS;
}
__device__ V3f xyz(const float4 *p, int i) {
return {p[i].x, p[i].y, p[i].z};
}
__device__ float Get(const float4 &j, int c) {
return c == 0 ? j.x : c == 1 ? j.y : c == 2 ? j.z : j.w;
}
// sum += a (b.x + b.y), sum and b float pairs (hi in x, lo in y). Needs fma and no reassociation (no fast math).
// The exact product and 2Sum following Dekker (1971) Numer. Math. 18, 224-242 and Knuth, TAOCP vol. 2, 4.2.2
__device__ void AddProduct(float2 &sum, float a, float2 b) {
const float p = a * b.x;
const float p_err = fmaf(a, b.x, -p); // a b.x = p + p_err exactly
const float s = sum.x + p;
const float v = s - sum.x;
const float s_err = (sum.x - (s - v)) + (p - v); // sum.x + p = s + s_err exactly
const float lo = s_err + sum.y + p_err + a * b.y;
const float hi = s + lo;
sum = make_float2(hi, lo - (hi - s));
}
// Each restraint's rows of J (RestraintGaussNewton): E = residual^2, so J = g / sqrt(2 E); a plane's atoms each a
// residual of their own, d r_i / d x_k = (delta_ik - 1/m) sqrt(2) normal / sigma. Zero where there is no row.
__global__ void JacobianKernel(const RestraintRecord *records, int n, const int *plane_atoms, const int *record_entry,
const float4 *xyzb, float4 *j) {
const int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i >= n)
return;
const RestraintRecord &r = records[i];
float4 *out = j + record_entry[i];
V3f g[4] = {};
int na = 0;
float e = 0;
switch (static_cast<RestraintKind>(r.kind)) {
case RestraintKind::Bond:
e = refine_math::BondTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), r.ideal, r.sigma, g);
na = 2;
break;
case RestraintKind::Angle:
e = refine_math::AngleTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]), r.ideal,
r.sigma, g);
na = 3;
break;
case RestraintKind::Torsion:
e = refine_math::TorsionTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]),
xyz(xyzb, r.atom[3]), r.ideal, r.sigma, r.period, g);
na = 4;
break;
case RestraintKind::Chiral:
e = refine_math::ChiralTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]),
xyz(xyzb, r.atom[3]), r.ideal, r.sigma, g);
na = 4;
break;
case RestraintKind::NcsDistance:
e = refine_math::NcsDistanceTerm(xyz(xyzb, r.atom[0]), xyz(xyzb, r.atom[1]), xyz(xyzb, r.atom[2]),
xyz(xyzb, r.atom[3]), r.sigma, r.ideal, g);
na = 4;
break;
case RestraintKind::Repulsion: {
const V3f q = xyz(xyzb, r.atom[1]);
const V3f image{r.rot[0] * q.x + r.rot[1] * q.y + r.rot[2] * q.z + r.tran[0],
r.rot[3] * q.x + r.rot[4] * q.y + r.rot[5] * q.z + r.tran[1],
r.rot[6] * q.x + r.rot[7] * q.y + r.rot[8] * q.z + r.tran[2]};
e = refine_math::RepulsionTerm(xyz(xyzb, r.atom[0]), image, r.ideal, r.sigma, g);
const V3f gi = g[1];
g[1] = {r.rot[0] * gi.x + r.rot[3] * gi.y + r.rot[6] * gi.z,
r.rot[1] * gi.x + r.rot[4] * gi.y + r.rot[7] * gi.z,
r.rot[2] * gi.x + r.rot[5] * gi.y + r.rot[8] * gi.z};
na = 2;
break;
}
case RestraintKind::Plane: {
V3f px[RefinementGPUEngine::MAX_PLANE_ATOMS], pg[RefinementGPUEngine::MAX_PLANE_ATOMS];
const int m = r.atom[1];
for (int k = 0; k < m; k++)
px[k] = xyz(xyzb, plane_atoms[r.atom[0] + k]);
refine_math::PlaneTerm(px, m, r.sigma, pg);
V3f nv{0, 0, 0};
for (int k = 0; k < m; k++) // the normal from the largest of the gradients (each is 2 r_k / sigma n)
if (refine_math::Norm(pg[k]) > refine_math::Norm(nv))
nv = pg[k];
const float norm = refine_math::Norm(nv);
const float c = norm > 0 ? sqrtf(2.0f) / (r.sigma * norm) : 0.0f;
for (int a = 0; a < m; a++)
for (int k = 0; k < m; k++) {
const float f = c * ((a == k ? 1.0f : 0.0f) - 1.0f / m);
out[a * m + k] = make_float4(f * nv.x, f * nv.y, f * nv.z, 0.0f);
}
return;
}
case RestraintKind::BSimilarity: {
float gi = 0, gj = 0;
e = refine_math::BSimilarityTerm(xyzb[r.atom[0]].w, xyzb[r.atom[1]].w, r.sigma, gi, gj);
const float f = e > 0 ? 1.0f / sqrtf(2.0f * e) : 0.0f;
out[0] = make_float4(0, 0, 0, f * gi);
out[1] = make_float4(0, 0, 0, f * gj);
return;
}
}
const float f = e > 0 ? 1.0f / sqrtf(2.0f * e) : 0.0f;
for (int k = 0; k < na; k++)
out[k] = make_float4(f * g[k].x, f * g[k].y, f * g[k].z, 0.0f);
}
// t_r = sum over the row's entries of J_e . v_k
__global__ void RowProduct(const int *start, const int *param, const float4 *j, int per, int c0, const float2 *v,
float2 *t, int rows) {
const int r = blockIdx.x * blockDim.x + threadIdx.x;
if (r >= rows)
return;
float2 dot = make_float2(0, 0);
for (int e = start[r]; e < start[r + 1]; e++)
if (const int k = param[e]; k >= 0)
for (int c = 0; c < per; c++)
AddProduct(dot, Get(j[e], c0 + c), v[per * k + c]);
t[r] = dot;
}
// hv_k = sum over the parameter atom's entries of t_row J_e
__global__ void TransposeProduct(const int *entry_start, const int *entry, const int *row_of, const float4 *j, int per,
int c0, const float2 *t, float2 *hv, int n) {
const int k = blockIdx.x * blockDim.x + threadIdx.x;
if (k >= n)
return;
for (int c = 0; c < per; c++) {
float2 sum = make_float2(0, 0);
for (int q = entry_start[k]; q < entry_start[k + 1]; q++) {
const int e = entry[q];
AddProduct(sum, Get(j[e], c0 + c), t[row_of[e]]);
}
hv[per * k + c] = sum;
}
}
__global__ void DiagonalKernel(const int *entry_start, const int *entry, const float4 *j, int per, int c0, float *d,
int n) {
const int k = blockIdx.x * blockDim.x + threadIdx.x;
if (k >= n)
return;
for (int c = 0; c < per; c++) {
float sum = 0;
for (int q = entry_start[k]; q < entry_start[k + 1]; q++) {
const float x = Get(j[entry[q]], c0 + c);
sum += x * x;
}
d[per * k + c] = sum;
}
}
template <typename T>
CudaDevicePtr<T> Up(const std::vector<T> &host, cudaStream_t stream) {
CudaDevicePtr<T> d(std::max<size_t>(host.size(), 1));
cuda_err(cudaMemcpyAsync(d.get(), host.data(), host.size() * sizeof(T), cudaMemcpyHostToDevice, stream));
return d;
}
} // namespace
struct RestraintNormalMatrixGPU::Device {
int device;
CudaStream stream;
int records = 0, rows = 0;
CudaDevicePtr<RestraintRecord> record;
CudaDevicePtr<int> plane_atoms, record_entry, start, param, row_of, entry_start, entry;
CudaDevicePtr<float4> xyzb, j;
CudaDevicePtr<float2> t, v, hv;
CudaDevicePtr<float> diagonal;
PinnedVector<float2> host; // v and hv as float pairs
PinnedVector<float4> host_xyzb;
};
RestraintNormalMatrixGPU::RestraintNormalMatrixGPU(int device, const std::vector<int> &param_atoms, size_t n_atoms,
bool xyz)
: per_(xyz ? 3 : 1), c0_(xyz ? 0 : 3), n_params_(param_atoms.size()), param_of_(n_atoms, -1) {
for (size_t k = 0; k < param_atoms.size(); k++)
param_of_[param_atoms[k]] = static_cast<int>(k);
DeviceGuard guard(device);
// At the device's highest priority: a product is a few small kernels the minimiser waits for, queued beside the
// X-ray evaluations of the other trials on the card.
dev_ = std::unique_ptr<Device>(new Device{device, CudaStream(cudaStreamNonBlocking, GreatestStreamPriority())});
const size_t n = std::max<size_t>(per_ * n_params_, 1);
dev_->xyzb = CudaDevicePtr<float4>(std::max<size_t>(n_atoms, 1));
dev_->host_xyzb.resize(n_atoms);
dev_->v = CudaDevicePtr<float2>(n);
dev_->hv = CudaDevicePtr<float2>(n);
dev_->diagonal = CudaDevicePtr<float>(n);
dev_->host.resize(n);
}
RestraintNormalMatrixGPU::~RestraintNormalMatrixGPU() {
DeviceGuard guard(dev_->device);
dev_.reset();
}
void RestraintNormalMatrixGPU::Upload(const std::vector<RestraintRecord> &records, const std::vector<int> &plane_atoms,
const std::vector<int> &record_entry, const std::vector<int> &start,
const std::vector<int> &param, const std::vector<int> &row_of,
const std::vector<int> &entry_start, const std::vector<int> &entry) {
DeviceGuard guard(dev_->device);
Device &d = *dev_;
d.records = static_cast<int>(records.size());
d.rows = static_cast<int>(start.size()) - 1;
d.record = Up(records, d.stream);
d.plane_atoms = Up(plane_atoms, d.stream);
d.record_entry = Up(record_entry, d.stream);
d.start = Up(start, d.stream);
d.param = Up(param, d.stream);
d.row_of = Up(row_of, d.stream);
d.entry_start = Up(entry_start, d.stream);
d.entry = Up(entry, d.stream);
d.j = CudaDevicePtr<float4>(std::max<size_t>(param.size(), 1));
d.t = CudaDevicePtr<float2>(std::max(d.rows, 1));
cuda_err(cudaStreamSynchronize(d.stream));
}
void RestraintNormalMatrixGPU::Fill(const std::vector<std::array<double, 4>> &xyzb) {
DeviceGuard guard(dev_->device);
Device &d = *dev_;
for (size_t a = 0; a < xyzb.size(); a++)
d.host_xyzb[a] = make_float4(static_cast<float>(xyzb[a][0]), static_cast<float>(xyzb[a][1]),
static_cast<float>(xyzb[a][2]), static_cast<float>(xyzb[a][3]));
cuda_err(cudaMemcpyAsync(d.xyzb.get(), d.host_xyzb.data(), xyzb.size() * sizeof(float4), cudaMemcpyHostToDevice,
d.stream));
if (d.records > 0)
JacobianKernel<<<GridSize(d.records), THREADS, 0, d.stream>>>(d.record, d.records, d.plane_atoms,
d.record_entry, d.xyzb, d.j);
cuda_err(cudaGetLastError());
cuda_err(cudaStreamSynchronize(d.stream));
}
void RestraintNormalMatrixGPU::Multiply(const std::vector<double> &v, std::vector<double> &hv) {
DeviceGuard guard(dev_->device);
Device &d = *dev_;
const int n = static_cast<int>(n_params_);
hv.resize(per_ * n_params_);
if (n == 0)
return;
for (size_t i = 0; i < hv.size(); i++) {
const float hi = static_cast<float>(v[i]);
d.host[i] = make_float2(hi, static_cast<float>(v[i] - hi));
}
cuda_err(cudaMemcpyAsync(d.v.get(), d.host.data(), hv.size() * sizeof(float2), cudaMemcpyHostToDevice, d.stream));
if (d.rows > 0)
RowProduct<<<GridSize(d.rows), THREADS, 0, d.stream>>>(d.start, d.param, d.j, per_, c0_, d.v, d.t, d.rows);
TransposeProduct<<<GridSize(n), THREADS, 0, d.stream>>>(d.entry_start, d.entry, d.row_of, d.j, per_, c0_, d.t,
d.hv, n);
cuda_err(cudaGetLastError());
cuda_err(cudaMemcpyAsync(d.host.data(), d.hv.get(), hv.size() * sizeof(float2), cudaMemcpyDeviceToHost, d.stream));
cuda_err(cudaStreamSynchronize(d.stream));
for (size_t i = 0; i < hv.size(); i++)
hv[i] = static_cast<double>(d.host[i].x) + d.host[i].y;
}
std::vector<double> RestraintNormalMatrixGPU::Diagonal() const {
DeviceGuard guard(dev_->device);
Device &d = *dev_;
const int n = static_cast<int>(n_params_);
std::vector<double> out(per_ * n_params_);
if (n == 0)
return out;
std::vector<float> host(out.size());
DiagonalKernel<<<GridSize(n), THREADS, 0, d.stream>>>(d.entry_start, d.entry, d.j, per_, c0_, d.diagonal, n);
cuda_err(cudaGetLastError());
cuda_err(cudaMemcpyAsync(host.data(), d.diagonal.get(), host.size() * sizeof(float), cudaMemcpyDeviceToHost, d.stream));
cuda_err(cudaStreamSynchronize(d.stream));
std::copy(host.begin(), host.end(), out.begin());
return out;
}
@@ -0,0 +1,52 @@
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#pragma once
// The restraints' Gauss-Newton normal matrix J^T J (RestraintGaussNewton) of the Newton minimisation on a GPU (CUDA
// builds only): RestraintNormalMatrix of Refinement.cpp on the card, in single precision. J is worked out on the device,
// one thread per restraint, with the terms the engine evaluates the restraints with (RefinementMath.h); a product is two
// gathers, J v per row and J^T t per parameter atom over its entries, each summed in a fixed order without atomics, in
// double-float pairs (RefinementNewtonGPU.cu).
//
// Every restraint keeps its rows whatever its E: where RestraintGaussNewton drops a row (E = 0, a plane without a
// normal) its entries here are zero, so the index of J is worked out once, at the first At(), and the set must stay
// the same for the matrix's life.
#include <array>
#include <cstddef>
#include <memory>
#include <vector>
#include "RefinementMath.h" // RestraintRecord
struct RestraintSet;
class RestraintNormalMatrixGPU {
public:
// On the x y z (`xyz`) or the B of `param_atoms` of the model's `n_atoms`, on GPU `device`.
RestraintNormalMatrixGPU(int device, const std::vector<int> &param_atoms, size_t n_atoms, bool xyz);
~RestraintNormalMatrixGPU();
void At(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb);
// hv = J^T J v
void Multiply(const std::vector<double> &v, std::vector<double> &hv);
// The diagonal of J^T J.
std::vector<double> Diagonal() const;
private:
// The set and the index of J (RefinementNewtonGPU.cu): per restraint its first entry, per row its entries
// start[r] ... start[r + 1], per entry its parameter atom (-1: none) and row, per parameter atom its entries.
void Upload(const std::vector<RestraintRecord> &records, const std::vector<int> &plane_atoms,
const std::vector<int> &record_entry, const std::vector<int> &start, const std::vector<int> &param,
const std::vector<int> &row_of, const std::vector<int> &entry_start, const std::vector<int> &entry);
// J at `xyzb` (x, y, z, B per atom), in float
void Fill(const std::vector<std::array<double, 4>> &xyzb);
int per_, c0_;
size_t n_params_;
std::vector<int> param_of_;
bool indexed_ = false;
struct Device;
std::unique_ptr<Device> dev_;
};
@@ -397,6 +397,48 @@ double RestraintValue(const RestraintSet &set, const RestraintRecord &r, const s
return NAN;
}
namespace {
// E of a restraint of the kinds over up to four atoms (not a plane, not a B similarity), its gradient in g[0..n).
double RecordTerm(const RestraintRecord &r, const std::vector<std::array<double, 4>> &xyzb, V3<double> g[4], int &n) {
const double ideal = r.ideal, sigma = r.sigma;
double e = 0;
n = 0;
switch (static_cast<RestraintKind>(r.kind)) {
case RestraintKind::Bond:
e = refine_math::BondTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), ideal, sigma, g);
n = 2;
break;
case RestraintKind::Angle:
e = refine_math::AngleTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), ideal, sigma, g);
n = 3;
break;
case RestraintKind::Torsion:
e = refine_math::TorsionTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
At(xyzb, r.atom[3]), ideal, sigma, r.period, g);
n = 4;
break;
case RestraintKind::Chiral:
e = refine_math::ChiralTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
At(xyzb, r.atom[3]), ideal, sigma, g);
n = 4;
break;
case RestraintKind::NcsDistance:
e = refine_math::NcsDistanceTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
At(xyzb, r.atom[3]), sigma, ideal, g);
n = 4;
break;
case RestraintKind::Repulsion:
e = refine_math::RepulsionTerm(At(xyzb, r.atom[0]), Image(r, At(xyzb, r.atom[1])), ideal, sigma, g);
g[1] = FromImage(r, g[1]);
n = 2;
break;
default:
break;
}
return e;
}
} // namespace
double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb,
std::vector<std::array<double, 4>> *grad) {
double sum = 0;
@@ -405,36 +447,8 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<
V3<double> g[4] = {};
int n = 0;
double e = 0;
const double ideal = r.ideal, sigma = r.sigma;
const double sigma = r.sigma;
switch (static_cast<RestraintKind>(r.kind)) {
case RestraintKind::Bond:
e = refine_math::BondTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), ideal, sigma, g);
n = 2;
break;
case RestraintKind::Angle:
e = refine_math::AngleTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]), ideal, sigma, g);
n = 3;
break;
case RestraintKind::Torsion:
e = refine_math::TorsionTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
At(xyzb, r.atom[3]), ideal, sigma, r.period, g);
n = 4;
break;
case RestraintKind::Chiral:
e = refine_math::ChiralTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
At(xyzb, r.atom[3]), ideal, sigma, g);
n = 4;
break;
case RestraintKind::NcsDistance:
e = refine_math::NcsDistanceTerm(At(xyzb, r.atom[0]), At(xyzb, r.atom[1]), At(xyzb, r.atom[2]),
At(xyzb, r.atom[3]), sigma, ideal, g);
n = 4;
break;
case RestraintKind::Repulsion:
e = refine_math::RepulsionTerm(At(xyzb, r.atom[0]), Image(r, At(xyzb, r.atom[1])), ideal, sigma, g);
g[1] = FromImage(r, g[1]);
n = 2;
break;
case RestraintKind::Plane: {
const int m = r.atom[1];
px.resize(m);
@@ -460,6 +474,8 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<
}
break;
}
default:
e = RecordTerm(r, xyzb, g, n);
}
sum += e;
if (grad != nullptr)
@@ -473,16 +489,66 @@ double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<
return sum;
}
std::vector<double> BRestraintCurvature(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
std::vector<double> h(xyzb.size(), 0.0);
for (const RestraintRecord &r : set.records)
if (r.kind == static_cast<int>(RestraintKind::BSimilarity)) {
// d2E/dB_i^2 of BSimilarityTerm = 2 B_j / (kappa^2 B_i^3)
const double bi = xyzb[r.atom[0]][3], bj = xyzb[r.atom[1]][3], k2 = double(r.sigma) * r.sigma;
h[r.atom[0]] += 2.0 * bj / (k2 * bi * bi * bi);
h[r.atom[1]] += 2.0 * bi / (k2 * bj * bj * bj);
RestraintJacobian RestraintGaussNewton(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb) {
RestraintJacobian jac;
jac.start.push_back(0);
auto row = [&](int atom, double x, double y, double z, double b) {
jac.atom.push_back(atom);
jac.j.push_back({x, y, z, b});
};
std::vector<V3<double>> px, pg;
for (const RestraintRecord &r : set.records) {
const RestraintKind kind = static_cast<RestraintKind>(r.kind);
if (kind == RestraintKind::Plane) {
// each atom's distance from the plane a residual of its own, d r_i / d x_j = (delta_ij - 1/n) normal / sigma
// (the normal held), so one row per atom of the plane over all of them
const int m = r.atom[1];
px.resize(m);
pg.resize(m);
for (int k = 0; k < m; k++)
px[k] = At(xyzb, set.plane_atoms[r.atom[0] + k]);
refine_math::PlaneTerm(px.data(), m, static_cast<double>(r.sigma), pg.data());
const double c = std::sqrt(2.0) / r.sigma;
V3<double> nv{0, 0, 0};
for (int k = 0; k < m; k++) // the normal from the largest of the gradients (each is 2 r_k / sigma n)
if (refine_math::Norm(pg[k]) > refine_math::Norm(nv))
nv = pg[k];
if (refine_math::Norm(nv) == 0)
continue;
nv = (1.0 / refine_math::Norm(nv)) * nv;
for (int i = 0; i < m; i++) {
for (int k = 0; k < m; k++) {
const double f = c * ((i == k ? 1.0 : 0.0) - 1.0 / m);
row(set.plane_atoms[r.atom[0] + k], f * nv.x, f * nv.y, f * nv.z, 0);
}
jac.start.push_back(static_cast<int>(jac.atom.size()));
}
continue;
}
return h;
// E = residual^2 (or close to it): its gradient is 2 residual grad(residual), so J = g / sqrt(2 E) gives the
// Gauss-Newton term 2 grad(residual) grad(residual)^T = J J^T; nothing where E is 0
if (kind == RestraintKind::BSimilarity) {
double gi = 0, gj = 0;
const double e = refine_math::BSimilarityTerm(xyzb[r.atom[0]][3], xyzb[r.atom[1]][3],
static_cast<double>(r.sigma), gi, gj);
if (!(e > 0))
continue;
const double f = 1.0 / std::sqrt(2.0 * e);
row(r.atom[0], 0, 0, 0, f * gi);
row(r.atom[1], 0, 0, 0, f * gj);
} else {
V3<double> g[4] = {};
int n = 0;
const double e = RecordTerm(r, xyzb, g, n);
if (!(e > 0))
continue;
const double f = 1.0 / std::sqrt(2.0 * e);
for (int k = 0; k < n; k++)
row(r.atom[k], f * g[k].x, f * g[k].y, f * g[k].z, 0);
}
jac.start.push_back(static_cast<int>(jac.atom.size()));
}
return jac;
}
gemmi::Position SpecialPosition::OnSite(const gemmi::Position &p) const {
@@ -49,8 +49,14 @@ double RestraintValue(const RestraintSet &set, const RestraintRecord &r, const s
double EvaluateRestraints(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb,
std::vector<std::array<double, 4>> *grad);
// The diagonal of the Hessian of the set's B similarity restraints with respect to each atom's B, at `xyzb`.
std::vector<double> BRestraintCurvature(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb);
// The Gauss-Newton approximation of the set's Hessian, sum over rows of J J^T: per row (a restraint's
// residual) its derivatives in (x, y, z, B) of the atoms atom[start[r] ... start[r + 1]).
struct RestraintJacobian {
std::vector<int> start;
std::vector<int> atom;
std::vector<std::array<double, 4>> j;
};
RestraintJacobian RestraintGaussNewton(const RestraintSet &set, const std::vector<std::array<double, 4>> &xyzb);
// An atom on a crystallographic special position: one that some symmetry operations of the group, each with a
// lattice translation, take onto itself - the site's symmetry. Its position moves only within the subspace those
+173 -6
View File
@@ -37,6 +37,7 @@
#include "../image_analysis/structure_refinement/RefinementXray.h"
#ifdef JFJOCH_USE_CUDA
#include "../image_analysis/structure_refinement/RefinementGPU.h"
#include "../image_analysis/structure_refinement/RefinementNewtonGPU.h"
#endif
namespace {
@@ -787,6 +788,55 @@ TEST_CASE("RefinementRestraints_deposited_ligands", "[Refinement]") {
}
}
TEST_CASE("RefinementRestraints_gauss_newton", "[Refinement]") {
// near the ideal geometry the restraints' Hessian is their Gauss-Newton J^T J (the residuals' own curvature times
// residuals that are small): v.J^T J v against central differences of the gradient along v, for random v
const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90);
gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5);
st.models[0] = shaken(st.models[0], 0.005, 0.0, 2);
const RefinementTopology topo = BuildRefinementTopology(st);
RestraintSet set = topo.geometry;
set.Finalize(atoms_of(st.models[0]).size());
std::vector<std::array<double, 4>> xyzb;
for (gemmi::Atom *a : atoms_of(st.models[0]))
xyzb.push_back({a->pos.x, a->pos.y, a->pos.z, a->b_iso});
const RestraintJacobian jac = RestraintGaussNewton(set, xyzb);
REQUIRE(jac.start.size() > 10);
std::mt19937 rng(4);
std::normal_distribution<double> normal(0.0, 1.0);
for (int trial = 0; trial < 5; trial++) {
std::vector<std::array<double, 4>> v(xyzb.size(), {0, 0, 0, 0});
for (auto &e : v)
for (int c = 0; c < 3; c++)
e[c] = normal(rng);
double gn = 0;
for (size_t r = 0; r + 1 < jac.start.size(); r++) {
double jv = 0;
for (int e = jac.start[r]; e < jac.start[r + 1]; e++)
for (int c = 0; c < 3; c++)
jv += jac.j[e][c] * v[jac.atom[e]][c];
gn += jv * jv;
}
const double h = 1e-5;
std::array<std::vector<std::array<double, 4>>, 2> g;
for (int side = 0; side < 2; side++) {
auto x = xyzb;
for (size_t i = 0; i < x.size(); i++)
for (int c = 0; c < 3; c++)
x[i][c] += (side == 0 ? h : -h) * v[i][c];
g[side].assign(x.size(), {0, 0, 0, 0});
EvaluateRestraints(set, x, &g[side]);
}
double exact = 0;
for (size_t i = 0; i < xyzb.size(); i++)
for (int c = 0; c < 3; c++)
exact += v[i][c] * (g[0][i][c] - g[1][i][c]) / (2 * h);
CAPTURE(trial, gn, exact);
CHECK(gn > 0);
CHECK(gn == Catch::Approx(exact).epsilon(0.1));
}
}
TEST_CASE("RefinementRestraints_gradient", "[Refinement]") {
const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90);
gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5);
@@ -986,6 +1036,82 @@ TEST_CASE("RefinementXray_gradient_CPU", "[Refinement]") {
}
#ifdef JFJOCH_USE_CUDA
TEST_CASE("RefinementNewton_product_GPU_vs_CPU", "[Refinement]") {
if (get_gpu_count() == 0)
return;
const gemmi::UnitCell cell(14, 15, 16, 90, 90, 90);
gemmi::Structure st = residue_crystal({"TYR", "DG"}, cell, "P 21 21 21", 5);
st.models[0] = shaken(st.models[0], 0.15, 3.0, 2);
const RefinementTopology topo = BuildRefinementTopology(st);
RestraintSet set = topo.geometry;
set.Append(topo.adp);
set.Append(BuildRepulsions(topo, st.models[0], cell, *st.find_spacegroup(), 0.0));
const std::vector<gemmi::Atom *> atoms = atoms_of(st.models[0]);
set.Finalize(atoms.size());
std::vector<std::array<double, 4>> xyzb;
for (gemmi::Atom *a : atoms)
xyzb.push_back({a->pos.x, a->pos.y, a->pos.z, a->b_iso});
// every atom but each fifth a parameter: the rest drop out
std::vector<int> params, param_of(atoms.size(), -1);
for (int a = 0; a < static_cast<int>(atoms.size()); a++)
if (a % 5 != 4) {
param_of[a] = static_cast<int>(params.size());
params.push_back(a);
}
const RestraintJacobian jac = RestraintGaussNewton(set, xyzb);
REQUIRE(jac.start.size() > 100);
std::mt19937 gen(3);
std::uniform_real_distribution<double> uniform(-1, 1);
// a random v, and a smooth one - the same shift of every atom and a thousandth of it at random, J v then the small
// difference of large terms
for (const auto [xyz, smooth] : {std::pair{true, false}, {true, true}, {false, false}, {false, true}}) {
CAPTURE(xyz, smooth);
const int per = xyz ? 3 : 1, c0 = xyz ? 0 : 3;
const size_t n = per * params.size();
std::vector<double> v(n);
for (size_t i = 0; i < n; i++)
v[i] = smooth ? 0.3 + 0.2 * (i % per) + 1e-3 * uniform(gen) : uniform(gen);
// J^T J v and the diagonal on the CPU, each entry in turn
std::vector<double> hv(n, 0.0), diagonal(n, 0.0);
for (size_t r = 0; r + 1 < jac.start.size(); r++) {
double t = 0;
for (int e = jac.start[r]; e < jac.start[r + 1]; e++)
if (const int k = param_of[jac.atom[e]]; k >= 0)
for (int c = 0; c < per; c++)
t += jac.j[e][c0 + c] * v[per * k + c];
for (int e = jac.start[r]; e < jac.start[r + 1]; e++)
if (const int k = param_of[jac.atom[e]]; k >= 0)
for (int c = 0; c < per; c++)
hv[per * k + c] += t * jac.j[e][c0 + c];
}
for (size_t e = 0; e < jac.atom.size(); e++)
if (const int k = param_of[jac.atom[e]]; k >= 0) {
const std::array<double, 4> &j = jac.j[e];
for (int c = 0; c < per; c++)
diagonal[per * k + c] += j[c0 + c] * j[c0 + c];
}
RestraintNormalMatrixGPU gpu(0, params, atoms.size(), xyz);
gpu.At(set, xyzb);
std::vector<double> hv_gpu;
gpu.Multiply(v, hv_gpu);
const std::vector<double> diagonal_gpu = gpu.Diagonal();
REQUIRE(hv_gpu.size() == n);
REQUIRE(diagonal_gpu.size() == n);
// In float, from float positions: within a few float epsilons of the double product in norm (in refinement
// its error is 2e-7 on average, 7e-6 at most), each element of the diagonal relative to itself.
double num = 0, den = 0, worst_diagonal = 0;
for (size_t i = 0; i < n; i++) {
num += (hv_gpu[i] - hv[i]) * (hv_gpu[i] - hv[i]);
den += hv[i] * hv[i];
if (diagonal[i] > 0)
worst_diagonal = std::max(worst_diagonal, std::fabs(diagonal_gpu[i] - diagonal[i]) / diagonal[i]);
}
CAPTURE(std::sqrt(num / den), worst_diagonal);
CHECK(std::sqrt(num / den) < 1e-5);
CHECK(worst_diagonal < 1e-5);
}
}
TEST_CASE("RefinementXray_gradient_GPU", "[Refinement]") {
if (get_gpu_count() == 0)
return;
@@ -1062,18 +1188,21 @@ TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") {
return d;
};
// the exact diagonal: one step
// the exact inverse Hessian: one step
RefinementMinimiser exact;
std::vector<double> h(n);
for (size_t k = 0; k < n; k++)
h[k] = 2 * a[k];
exact.SetDiagonal(h);
exact.SetInverseHessian([&](const std::vector<double> &, std::vector<double> &q) {
for (size_t k = 0; k < n; k++)
q[k] /= 2 * a[k];
});
std::vector<double> x(n, 0.0);
exact.Minimise(target, x, 1, 0.0, 10.0);
CHECK(distance(x) < 1e-9);
// without it, the memory carried from one call to the next does what one long call does
// with the identity, the memory carried from one call to the next does what one long call does
RefinementMinimiser split, whole;
const RefinementMinimiser::InverseHessian identity = [](const std::vector<double> &, std::vector<double> &) {};
split.SetInverseHessian(identity);
whole.SetInverseHessian(identity);
std::vector<double> xs(n, 0.0), xw(n, 0.0);
for (int call = 0; call < 6; call++)
split.Minimise(target, xs, 10, 0.0, 10.0);
@@ -1083,6 +1212,44 @@ TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") {
CHECK(xs[k] == Catch::Approx(xw[k]).margin(1e-12));
}
TEST_CASE("RefinementMinimiser_conjugate_gradients", "[Refinement]") {
// a tridiagonal positive definite system, solved to the tolerance with and without a (Jacobi) preconditioner
const size_t n = 200;
auto product = [&](const std::vector<double> &v, std::vector<double> &hv) {
hv.assign(n, 0.0);
for (size_t k = 0; k < n; k++) {
hv[k] = (2.0 + 0.01 * k) * v[k];
if (k > 0)
hv[k] -= v[k - 1];
if (k + 1 < n)
hv[k] -= v[k + 1];
}
};
std::vector<double> b(n);
for (size_t k = 0; k < n; k++)
b[k] = std::sin(0.1 * k);
auto residual = [&](const std::vector<double> &x) {
std::vector<double> hx;
product(x, hx);
double r = 0, b2 = 0;
for (size_t k = 0; k < n; k++) {
r += (hx[k] - b[k]) * (hx[k] - b[k]);
b2 += b[k] * b[k];
}
return std::sqrt(r / b2);
};
std::vector<double> x;
ConjugateGradients(product, [](const std::vector<double> &r, std::vector<double> &z) { z = r; }, b, x, 500, 1e-8);
CHECK(residual(x) < 1e-8);
const int jacobi = ConjugateGradients(product, [&](const std::vector<double> &r, std::vector<double> &z) {
z.resize(n);
for (size_t k = 0; k < n; k++)
z[k] = r[k] / (2.0 + 0.01 * k);
}, b, x, 500, 0.1);
CHECK(residual(x) < 0.1);
CHECK(jacobi < 50);
}
TEST_CASE("Refinement_restrained_recovers_model", "[Refinement]") {
const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90);
const gemmi::SpaceGroup &sg = *gemmi::find_spacegroup_by_name("P 21 21 21");