Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature

Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every
10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser
keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's
scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking
Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian.

The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution
shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference
probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints'
Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser,
since that stiffness is in directions that move bonded atoms together.

Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the
refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345,
7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393;
REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work
lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung
further). Time unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)

tests/RefinementTest.cpp

@@ -17,6 +17,7 @@
 #include "../image_analysis/structure_refinement/MonomerLibrary.h"
 #include "../image_analysis/structure_refinement/Refinement.h"
 #include "../image_analysis/structure_refinement/RefinementMath.h"
+#include "../image_analysis/structure_refinement/RefinementMinimiser.h"
 #include "../image_analysis/structure_refinement/RefinementRestraints.h"
 #include "../image_analysis/structure_refinement/RefinementXray.h"
 #ifdef JFJOCH_USE_CUDA
@@ -555,6 +556,48 @@ TEST_CASE("RefinementXray_gradient_GPU", "[Refinement]") {
 }
 #endif
 
+TEST_CASE("RefinementMinimiser_quadratic", "[Refinement]") {
+    // f = sum a_k (x_k - 1)^2 with curvatures spread over two decades
+    const size_t n = 50;
+    std::vector<double> a(n);
+    for (size_t k = 0; k < n; k++)
+        a[k] = std::pow(100.0, static_cast<double>(k) / (n - 1));
+    const RefinementMinimiser::Target target = [&](const std::vector<double> &x, double &f, std::vector<double> &g) {
+        f = 0;
+        for (size_t k = 0; k < n; k++) {
+            f += a[k] * (x[k] - 1) * (x[k] - 1);
+            g[k] = 2 * a[k] * (x[k] - 1);
+        }
+        return true;
+    };
+    auto distance = [&](const std::vector<double> &x) {
+        double d = 0;
+        for (double v : x)
+            d = std::max(d, std::fabs(v - 1));
+        return d;
+    };
+
+    // the exact diagonal: one step
+    RefinementMinimiser exact;
+    std::vector<double> h(n);
+    for (size_t k = 0; k < n; k++)
+        h[k] = 2 * a[k];
+    exact.SetDiagonal(h);
+    std::vector<double> x(n, 0.0);
+    exact.Minimise(target, x, 1, 0.0, 10.0);
+    CHECK(distance(x) < 1e-9);
+
+    // without it, the memory carried from one call to the next does what one long call does
+    RefinementMinimiser split, whole;
+    std::vector<double> xs(n, 0.0), xw(n, 0.0);
+    for (int call = 0; call < 6; call++)
+        split.Minimise(target, xs, 10, 0.0, 10.0);
+    whole.Minimise(target, xw, 60, 0.0, 10.0);
+    CHECK(distance(xw) < 1e-3);
+    for (size_t k = 0; k < n; k++)
+        CHECK(xs[k] == Catch::Approx(xw[k]).margin(1e-12));
+}
+
 TEST_CASE("Refinement_restrained_recovers_model", "[Refinement]") {
     const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90);
     const gemmi::SpaceGroup &sg = *gemmi::find_spacegroup_by_name("P 21 21 21");