Refinement minimiser: reorthogonalise the conjugate gradients' residuals

The normal-matrix solve (20 fixed preconditioned CG iterations per L-BFGS direction) was forward
unstable: run in lockstep on right-hand sides 1e-14 apart, two CPU solves in double drifted apart
5-7x per iteration from about the 5th iteration and ended up to 7% apart (myoglobin: 17% of the
position solves above 1e-5). That is the loss of orthogonality of the recurrence in rounding once
its first Ritz values converge, and it is what set the CPU and the GPU apart: the device solve
matched the host to 1e-14 until the first such call and then differed by exactly that spread. It
is not small curvature: an absolute floor on the X-ray diagonal (1e-3 x median, the review's
remedy) changed nothing, and blocks of up to 8 residues in the preconditioner hardly anything.

Each new residual is now made M^-1-orthogonal to all the earlier ones (modified Gram-Schmidt on r
and z together), on the CPU and on the card. Measured on every solve of a refinement run
(myoglobin, cytochrome C): a 1e-12 change of q now moves the result by 2e-12 (max 2e-11), was
median 2e-7 / max 8e-2; device vs host median 4e-14, max 1e-11, was p90 2e-3 / max 6e-2. The
residual after 20 iterations is the same (mean |r20|/|r0| 0.016 vs 0.017).

Costs 2 x 20 stored vectors per solve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-17 11:39:01 +01:00
1 parent a1838b023a
commit 3c59c4fff2
6 files changed
+80 -13

No files matched your search

+6 -1
View File
@@ -603,7 +603,12 @@ initial inverse Hessian is the target's sparse normal matrix - the X-ray target'
on the diagonal and the restraints' Gauss-Newton matrix in full - as REFMAC refines on its normal
matrix (Murshudov, Vagin & Dodson 1997, above), solved by conjugate gradients: M. R. Hestenes and
E. Stiefel, "Methods of conjugate gradients for solving linear systems" (1952), J. Res. Natl. Bur.
Stand. 49, 409-436 [doi:10.6028/jres.049.044](https://doi.org/10.6028/jres.049.044). Each atom's
Stand. 49, 409-436 [doi:10.6028/jres.049.044](https://doi.org/10.6028/jres.049.044), each residual
made orthogonal again to the earlier ones against the loss of orthogonality in rounding described by
C. C. Paige, "Error analysis of the Lanczos algorithm for tridiagonalizing a symmetric matrix" (1976),
J. Inst. Maths Applics 18, 341-349 (DOI not verified here), and A. Greenbaum, "Behavior of
slightly perturbed Lanczos and conjugate-gradient recurrences" (1989), Linear Algebra Appl. 113, 7-63
[doi:10.1016/0024-3795(89)90285-1](https://doi.org/10.1016/0024-3795(89)90285-1). Each atom's
diagonal is raised to hold its step within the step bound, a trust region per atom after the damping
of K. Levenberg, "A method for the solution of certain non-linear problems in least squares" (1944),
Q. Appl. Math. 2, 164-168 [doi:10.1090/qam/10666](https://doi.org/10.1090/qam/10666), and D. W.
+1
View File
@@ -73,6 +73,7 @@ The methods draw on, and in places reimplement, solutions from:
- D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15).
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6).
- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6).
- C. C. Paige, "Error analysis of the Lanczos algorithm for tridiagonalizing a symmetric matrix", *J. Inst. Maths Applics* **18** (1976), 341-349, and A. Greenbaum, "Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences", *Linear Algebra Appl.* **113** (1989), 7-63 (the reorthogonalised conjugate gradients of the minimiser, §15.6).
- G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5).
- P. Roversi, E. Blanc, S. Johnson & S. M. Lea, "Tetartohedral twinning could happen to you too", *Acta Cryst.* **D68** (2012), 418-424 (twin refinement with more than two domains, and the free set closed under the twin domains, §15.12).
- K. A. Sultana, I. Alexeev, I. Kursula, P. Mäntsälä, J. Niemi & G. Schneider, "Structure determination by multiwavelength anomalous diffraction of aclacinomycin oxidoreductase: indications of multidomain pseudomerohedral twinning", *Acta Cryst.* **D63** (2007), 149-159 (a six-domain twin of a monoclinic crystal on a hexagonal metric, §15.12).
+7 -1
View File
@@ -320,7 +320,12 @@ leaves out.
conjugate-gradient iterations - not to a tolerance, whose iteration count, and with it the direction, a rounding
difference can flip - preconditioned per residue for the positions (the residue's own $J^\mathsf{T}J$ and X-ray
diagonal, by Cholesky: the stiff bonds and angles inside a residue are solved there, what couples residues is
left to the iterations) and by the diagonal elsewhere. The Jacobian is taken again at each point the minimiser
left to the iterations) and by the diagonal elsewhere. Each new residual is made orthogonal again (in the
preconditioner's inner product) to all the earlier ones: the plain recurrence loses that orthogonality in
rounding once its first Ritz values converge (Paige; Greenbaum), and from then on a rounding difference grew
several-fold per iteration, so that two solves of right-hand sides $10^{-14}$ apart ended up to 7% apart - which
is what set the CPU and GPU results apart. With it the solve is a smooth function of its input, and the GPU's
float-pair solve agrees with the CPU's to $10^{-13}$. The Jacobian is taken again at each point the minimiser
stands on.
- **Its memory is kept from one macro-cycle to the next.** Each weight trial starts with an empty memory.
@@ -790,6 +795,7 @@ What the refinement does not do yet - each a known gap, not a design choice:
- N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", *Acta Cryst.* **D70** (2014), 1104-1114 (a light ion flagged by its density above the waters', §15.10).
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782, and D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15.6).
- M. R. Hestenes & E. Stiefel, "Methods of conjugate gradients for solving linear systems", *J. Res. Natl. Bur. Stand.* **49** (1952), 409-436 (the normal equations of the minimiser, §15.6).
- C. C. Paige, "Error analysis of the Lanczos algorithm for tridiagonalizing a symmetric matrix", *J. Inst. Maths Applics* **18** (1976), 341-349, and A. Greenbaum, "Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences", *Linear Algebra Appl.* **113** (1989), 7-63 (the loss of orthogonality the minimiser's conjugate gradients are kept from, §15.6).
- K. Levenberg, "A method for the solution of certain non-linear problems in least squares", *Q. Appl. Math.* **2** (1944), 164-168, and D. W. Marquardt, "An algorithm for least-squares estimation of nonlinear parameters", *J. Soc. Ind. Appl. Math.* **11** (1963), 431-441 (the minimiser's trust floor per atom, §15.6).
- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7).
@@ -172,6 +172,12 @@ private:
// z with H z = b to `tolerance` of |b| (at most `max_iterations`), by conjugate gradients preconditioned with M:
// hv = H v (product), z = M^-1 r (precondition), both symmetric positive definite. Returns the iterations taken.
// Following Hestenes & Stiefel (1952) J. Res. Natl. Bur. Stand. 49, 409-436
// Each residual is made M^-1-orthogonal again to all the earlier ones (modified Gram-Schmidt, r and z = M^-1 r
// together). Without it the recurrence loses that orthogonality in rounding once the first Ritz values converge, and
// from then on amplifies a rounding difference several times per iteration: on the refinement's normal matrix two
// solves of right-hand sides 1e-14 apart ended up to 7% apart after 20 iterations, which set CPU and GPU apart.
// The loss of orthogonality after Paige (1976) J. Inst. Maths Applics 18, 341-349 and Greenbaum (1989) Linear
// Algebra Appl. 113, 7-63
inline int ConjugateGradients(const std::function<void(const std::vector<double> &v, std::vector<double> &hv)> &product,
const std::function<void(const std::vector<double> &r, std::vector<double> &z)> &precondition,
const std::vector<double> &b, std::vector<double> &x, int max_iterations, double tolerance,
@@ -183,6 +189,8 @@ inline int ConjugateGradients(const std::function<void(const std::vector<double>
precondition(r, z);
p = z;
double rz = dot(r, z);
std::vector<std::vector<double>> rs{r}, zs{z}; // the residuals so far, and M^-1 of each
std::vector<double> rzs{rz};
const double r0 = std::sqrt(dot(r, r));
int k = 0;
while (k < max_iterations) {
@@ -199,9 +207,19 @@ inline int ConjugateGradients(const std::function<void(const std::vector<double>
if (std::sqrt(dot(r, r)) < tolerance * r0)
break;
precondition(r, z);
for (size_t i = 0; i < rs.size(); i++) {
const double c = dot(r, zs[i]) / rzs[i];
refine_vec::Each(n, nthreads, [&](size_t j) {
r[j] -= c * rs[i][j];
z[j] -= c * zs[i][j];
});
}
const double rz_new = dot(r, z);
refine_vec::Each(n, nthreads, [&](size_t j) { p[j] = z[j] + rz_new / rz * p[j]; });
rz = rz_new;
rs.push_back(r);
zs.push_back(z);
rzs.push_back(rz);
}
return k;
}
@@ -228,7 +228,7 @@ __global__ void DiagonalKernel(const int *entry_start, const int *entry, const f
// The conjugate gradients' scalars, kept on the card between the kernels: r M^-1 r, the step a and beta
struct CGScalars {
float2 rz, a, beta;
float2 rz, a, beta, c; // c: the residual's component along an earlier one
int done; // p H p was not positive: x stays where it is
};
@@ -256,10 +256,11 @@ __global__ void DotPartial(const float2 *u, const float2 *v, int n, float2 *part
partial[blockIdx.x] = block;
}
enum class DotUse { FirstRz, PHp, Rz };
enum class DotUse { FirstRz, PHp, Rz, Component };
// The partial sums' total, taken as the first r M^-1 r, as p H p (a = rz / pHp) or as the next r M^-1 r (beta)
__global__ void DotFinish(const float2 *partial, CGScalars *s, DotUse use) {
// The partial sums' total, taken as the first r M^-1 r, as p H p (a = rz / pHp), as the next r M^-1 r (beta), each
// r M^-1 r kept as rz_of[j], or as r . z_j (c = r . z_j / rz_of[j])
__global__ void DotFinish(const float2 *partial, CGScalars *s, DotUse use, float2 *rz_of, int j) {
__shared__ float2 shared[DOT_BLOCKS];
const float2 dot = BlockSum(shared, partial[threadIdx.x]);
if (threadIdx.x != 0)
@@ -267,6 +268,7 @@ __global__ void DotFinish(const float2 *partial, CGScalars *s, DotUse use) {
switch (use) {
case DotUse::FirstRz:
s->rz = dot;
rz_of[j] = dot;
break;
case DotUse::PHp:
s->a = Div(s->rz, dot);
@@ -276,6 +278,10 @@ __global__ void DotFinish(const float2 *partial, CGScalars *s, DotUse use) {
case DotUse::Rz:
s->beta = Div(dot, s->rz);
s->rz = dot;
rz_of[j] = dot;
break;
case DotUse::Component:
s->c = Div(dot, rz_of[j]);
break;
}
}
@@ -289,6 +295,15 @@ __global__ void Step(const CGScalars *s, const float2 *p, const float2 *hp, floa
r[i] = Add(r[i], Neg(Mul(s->a, hp[i])));
}
// r -= c r_j, z -= c z_j
__global__ void Project(const CGScalars *s, const float2 *rj, const float2 *zj, float2 *r, float2 *z, int n) {
const int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i >= n)
return;
r[i] = Add(r[i], Neg(Mul(s->c, rj[i])));
z[i] = Add(z[i], Neg(Mul(s->c, zj[i])));
}
// p = z + beta p
__global__ void NextDirection(const CGScalars *s, const float2 *z, float2 *p, int n) {
const int i = blockIdx.x * blockDim.x + threadIdx.x;
@@ -340,6 +355,8 @@ struct RestraintNormalMatrixGPU::Device {
// the conjugate gradients: H's X-ray diagonal, M's blocks and their inverses, the vectors, the dot products' partial sums
int elements = 0;
CudaDevicePtr<float2> h, inverse, x, r, z, p, hp, partial;
CudaDevicePtr<float2> rs, zs, rz_of; // the residuals of a solve so far, M^-1 of each, r M^-1 r of each
int kept = 0; // how many of them rs and zs hold room for
CudaDevicePtr<int> block_of, block_start, block_element, inverse_start;
CudaDevicePtr<CGScalars> scalars;
PinnedVector<float2> host; // v and hv as float pairs
@@ -485,9 +502,21 @@ void RestraintNormalMatrixGPU::Solve(const std::vector<double> &b, std::vector<d
BlockApply<<<GridSize(d.elements), THREADS, 0, d.stream>>>(d.block_of, d.block_start, d.block_element,
d.inverse_start, d.inverse, d.r, d.z, d.elements);
};
auto dot = [&](const float2 *u, const float2 *v, DotUse use) {
auto dot = [&](const float2 *u, const float2 *v, DotUse use, int j) {
DotPartial<<<DOT_BLOCKS, THREADS, 0, d.stream>>>(u, v, n, d.partial);
DotFinish<<<1, DOT_BLOCKS, 0, d.stream>>>(d.partial, d.scalars, use);
DotFinish<<<1, DOT_BLOCKS, 0, d.stream>>>(d.partial, d.scalars, use, d.rz_of, j);
};
if (d.kept < iterations) {
d.rs = CudaDevicePtr<float2>(static_cast<size_t>(iterations) * n);
d.zs = CudaDevicePtr<float2>(static_cast<size_t>(iterations) * n);
d.rz_of = CudaDevicePtr<float2>(iterations);
d.kept = iterations;
}
auto keep = [&](int j) { // r and z as the j-th
cuda_err(cudaMemcpyAsync(d.rs.get() + static_cast<size_t>(j) * n, d.r.get(), n * sizeof(float2),
cudaMemcpyDeviceToDevice, d.stream));
cuda_err(cudaMemcpyAsync(d.zs.get() + static_cast<size_t>(j) * n, d.z.get(), n * sizeof(float2),
cudaMemcpyDeviceToDevice, d.stream));
};
for (int i = 0; i < n; i++)
d.host[i] = Split(b[i]);
@@ -496,18 +525,25 @@ void RestraintNormalMatrixGPU::Solve(const std::vector<double> &b, std::vector<d
cuda_err(cudaMemsetAsync(d.scalars.get(), 0, sizeof(CGScalars), d.stream));
precondition();
cuda_err(cudaMemcpyAsync(d.p.get(), d.z.get(), n * sizeof(float2), cudaMemcpyDeviceToDevice, d.stream));
dot(d.r, d.z, DotUse::FirstRz);
dot(d.r, d.z, DotUse::FirstRz, 0);
keep(0);
for (int k = 0; k < iterations; k++) {
if (d.rows > 0)
RowProduct<<<GridSize(d.rows), THREADS, 0, d.stream>>>(d.start, d.param, d.j, per_, c0_, d.p, d.t, d.rows);
TransposeProduct<<<GridSize(static_cast<int>(n_params_)), THREADS, 0, d.stream>>>(
d.entry_start, d.entry, d.row_of, d.j, per_, c0_, d.t, d.h, d.p, d.hp, static_cast<int>(n_params_));
dot(d.p, d.hp, DotUse::PHp);
dot(d.p, d.hp, DotUse::PHp, 0);
Step<<<GridSize(n), THREADS, 0, d.stream>>>(d.scalars, d.p, d.hp, d.x, d.r, n);
if (k + 1 < iterations) { // the last direction would not be used
precondition();
dot(d.r, d.z, DotUse::Rz);
for (int j = 0; j <= k; j++) { // M^-1-orthogonal to the earlier residuals, as ConjugateGradients
const float2 *rj = d.rs.get() + static_cast<size_t>(j) * n, *zj = d.zs.get() + static_cast<size_t>(j) * n;
dot(d.r, zj, DotUse::Component, j);
Project<<<GridSize(n), THREADS, 0, d.stream>>>(d.scalars, rj, zj, d.r, d.z, n);
}
dot(d.r, d.z, DotUse::Rz, k + 1);
NextDirection<<<GridSize(n), THREADS, 0, d.stream>>>(d.scalars, d.z, d.p, n);
keep(k + 1);
}
}
cuda_err(cudaGetLastError());
@@ -40,8 +40,9 @@ public:
// `inverse`, in full, row by row. h and M stay until the next SetSolve().
void SetSolve(const std::vector<double> &h, const std::vector<int> &start, const std::vector<int> &element,
const std::vector<double> &inverse);
// x = H^-1 b by `iterations` of them from x = 0 (fewer only where p H p is not positive), the vectors on the card
// and every sum in float pairs, in a fixed order (RefinementNewtonGPU.cu).
// x = H^-1 b by `iterations` of them from x = 0 (fewer only where p H p is not positive), each residual made
// M^-1-orthogonal to the earlier ones as there; the vectors on the card and every sum in float pairs, in a fixed
// order (RefinementNewtonGPU.cu).
void Solve(const std::vector<double> &b, std::vector<double> &x, int iterations);
private: