From 089423067193dc23de3deca295955f10f548b9ca Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 08:18:06 +0200 Subject: [PATCH] Anderson: accelerate a scaling loop only near its fixed point, and never again once its step grows (RESULT-CHANGING) The Anderson commit before this one changed three rc176 battery results, which is why it was reverted there. One of them is the acceleration's own doing: on a split myoglobin sweep (P2_1, ISa 2.5) the partials loops never settle - one walks at a step of ~2e-2 for as long as it runs - and the fulls loops walk from their first round (their cross-validated lambda moves every round, so the map itself is not fixed). Restarting the history whenever the step grew only rebuilt a one- or two-round history and extrapolated along the walk: the stalled loops ended somewhere else, the pre-pass fulls smoothing window went from ~7 frames to ~1, and CC1/2 fell 0.701 -> 0.595. Two rules now confine the acceleration to the regime it is for - a map contracting to its fixed point: - a loop is accelerated only once its plain step is below ten times the tolerance (1e-2); - the first round whose weighted residual grew turns the acceleration off for the rest of the loop (AndersonMixing::Stopped), instead of restarting it. A loop that walks therefore stays the plain loop it was in rc176. The growth rule alone gave CC1/2 0.625; with the onset, 0.701, as rc176. Measured against rc176 bc191bb80 (GPU, workstation): - myob_x06da_split: P2_1, CC1/2 0.701, d 1.81 A - as rc176; 481 -> 257 scaling iterations. - 7qij 434 -> 126 iterations, 9v16 273 -> 101, same space group and statistics. - ci tier (29 sets, no null model): 28 unchanged beyond noise; sum of times 580 -> 585 s (9gdj +9 s, below). The two other changes are NOT Anderson's and stay: - insu_I_x06da_weak I23 -> I222: the all-observations search refuses I23 on a merge chi2 ratio 2.08 against a 1.85 bound; plain scaling converged to 1e-5 gives 2.07-2.09 and refuses it too. Only the plain loop stopped at 1e-3 passed - the bound was set on unconverged scaling. - 9gdj +9 s, same answer: the early beam-centre comparison's "loses beyond noise" margin (-0.002 against 2 sigma 0.0016) moves by ~0.07 between scaling paths, so the converged loop no longer takes the shortcut and both centres are judged on all frames; the file's centre is kept either way. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 +- image_analysis/scale_merge/AndersonMixing.cpp | 6 +++++- image_analysis/scale_merge/AndersonMixing.h | 12 +++++++++--- .../scale_merge/RotationScaleMerge.cpp | 17 ++++++++++++----- tests/AndersonMixingTest.cpp | 9 ++++++--- 5 files changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 9be9ef481..cdd8736a6 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -388,7 +388,7 @@ Before the combine a per-frame scale can also be fitted on the partials themselv How smooth that scale is over the rotation is left to the data rather than to a fixed window. Each round fits every frame on its own fulls against the current reference, giving a scale $G_f$ and its information $D_f=\sum w^2c^2$; the scale is then the curve $x=\log G$ minimising $\sum_f J_f\,(x_f-y_f)^2+\lambda\sum_f(\Delta^2 x)_f^2$, with $y_f$ the frame's own fit in log scale and $J_f$ its information carried there — a penalised (Whittaker–Eilers) smoother, solved as a five-band linear system. $\lambda$ is chosen by cross-validation: blocks of frames one rocking curve wide are left out in turn and predicted from the curve through the rest (neighbours closer than a rocking curve share their measurement, since a full sums those frames). A frame of hundreds of fulls is then followed frame by frame, a frame of two or three is carried by its neighbours, a stretch with none is bridged by a straight line, and a scale that falls a hundredfold over a few degrees — an absorbing crystal turning edge-on — is followed where a window would average across it. Once the curve settles, one free fit is shrunk toward it frame by frame by how much of each frame's deviation its neighbour shares (the lag-1 covariance), which hands back a real per-frame systematic and discards fit noise. -Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the step grows or the set of fitted frames changes. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. +Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the set of fitted frames changes. Acceleration starts only once the plain step has fallen below ten times the tolerance, and stops for the rest of the loop the first time the step grows: Anderson mixing models a map near its fixed point, and a loop further out, or one whose step grows, may be walking rather than converging - accelerated, it walks faster and ends somewhere else. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. After scale-fulls, five **correction surfaces** are fitted on the combined fulls (rotation path, **on by default**; disable all with `--no-scaling-corrections`), each an alternating multiplicative refinement of the per-full scale against the merged reference: diff --git a/image_analysis/scale_merge/AndersonMixing.cpp b/image_analysis/scale_merge/AndersonMixing.cpp index 076a462de..2d307eb3d 100644 --- a/image_analysis/scale_merge/AndersonMixing.cpp +++ b/image_analysis/scale_merge/AndersonMixing.cpp @@ -16,7 +16,11 @@ std::vector AndersonMixing::Next(const std::vector &x, const std for (int i = 0; i < n; ++i) support[i] = w[i] > 0.0 ? 1 : 0; const double r_norm = (wv.array() * r.array().square()).sum(); - if (x_last.size() == n && support == support_last && r_norm <= r_norm_last) { + if (stopped || (x_last.size() == n && r_norm > r_norm_last)) { + stopped = true; + return fx; + } + if (x_last.size() == n && support == support_last) { dx.push_back(xv - x_last); dr.push_back(r - r_last); if (static_cast(dx.size()) > depth) { diff --git a/image_analysis/scale_merge/AndersonMixing.h b/image_analysis/scale_merge/AndersonMixing.h index 8cc18a16d..1f44ea370 100644 --- a/image_analysis/scale_merge/AndersonMixing.h +++ b/image_analysis/scale_merge/AndersonMixing.h @@ -18,9 +18,13 @@ // // Each component carries a weight in the norm r is measured in; a component of weight 0 takes F(x) // unchanged and is not part of the fit. The history is dropped, and the round is a plain one, when -// the residual grew - the history no longer describes the map from here - or when the set of -// components of non-zero weight changed, since the differences of the history are then differences -// of different things. +// the set of components of non-zero weight changed, since the differences of the history are then +// differences of different things. +// +// Acceleration is for a map that contracts. A residual that grows from one round to the next says the +// iteration is not on its way to a fixed point here - it walks, or the map itself moved - and +// extrapolating from its history takes it further along the walk. From that round on every round is +// a plain one. class AndersonMixing { public: explicit AndersonMixing(int depth) : depth(depth) {} @@ -29,10 +33,12 @@ public: const std::vector &w); int Restarts() const { return restarts; } + bool Stopped() const { return stopped; } private: int depth; int restarts = 0; + bool stopped = false; std::deque dx, dr; // x and r differences of consecutive rounds, newest last Eigen::VectorXd x_last, r_last; std::vector support_last; diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index c8606697d..19581fc40 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -65,6 +65,9 @@ namespace { // directions are a few smooth modes over the sweep: measured on a 4nwv-class sweep, 3, 5 and 8 rounds // take its ten loops in 92, 91 and 90 iterations, against 364 without. constexpr int SCALING_ANDERSON_DEPTH = 5; + // The loops are accelerated only once their step is within this of the tolerance: Anderson mixing + // models the map near its fixed point, and a loop still further out is as likely to be walking. + constexpr double SCALING_ANDERSON_ONSET = 10.0 * SCALING_TOLERANCE; // The per-frame scale is smoothed over at least this many rocking curves (see Run). constexpr int SMOOTH_G_MIN_ROCKING_EVENTS = 6; @@ -1858,7 +1861,9 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( // rounds' steps that best cancels this one. That moves no fixed point - only the path to it - and // it takes the loop to its answer in a quarter of the rounds and closer: on one sweep the merged // intensities ended 0.02% rms from the converged point, against 0.12% for the plain loop stopped at - // the same tolerance. A frame below the credible floor (below) takes the plain round's scale. + // the same tolerance. A frame below the credible floor (below) takes the plain round's scale. A + // loop whose step grows is not contracting - it walks, and acceleration would only walk it faster - + // so from that round on its rounds are plain ones. // // The loop stops when the relative scales stop moving - the rms of |log(G_new/G_old)| over the // frames, each weighted by its observation count and nothing else (a weight that vanishes with G @@ -1932,7 +1937,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( } out.step = sw > 0.0 ? std::sqrt(ss / sw) : 0.0; logger->Debug("Per-frame scaling ({}): iteration {}, rms |dlogG| {:.2e} over {} frames, " - "gauge {:.4f}, {} restarts", what, it + 1, out.step, n, g_ref, anderson.Restarts()); + "gauge {:.4f}, {} restarts{}", what, it + 1, out.step, n, g_ref, anderson.Restarts(), + anderson.Stopped() ? ", plain" : ""); if (out.step < SCALING_TOLERANCE) { out.converged = true; stop = true; @@ -1943,7 +1949,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunScalingLoop( out.stalled = true; stop = true; } - if (!stop) { + if (!stop && out.step < SCALING_ANDERSON_ONSET) { // The next round starts from the accelerated scales, gauge-pinned again. const std::vector next = anderson.Next(log_g_prev, log_g, weight); for (int f = 0; f < n_frames; ++f) @@ -2054,7 +2060,8 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( } out.step = sw > 0.0 ? std::sqrt(ss / sw) : 0.0; logger->Debug("Per-frame scaling (fulls): iteration {}, rms |dlogG| {:.2e}, lambda {:.3g}, gauge {:.4f}, " - "{} restarts", it + 1, out.step, lambda, g_ref, anderson.Restarts()); + "{} restarts{}", it + 1, out.step, lambda, g_ref, anderson.Restarts(), + anderson.Stopped() ? ", plain" : ""); bool stop = false; if (out.step < SCALING_TOLERANCE) { out.converged = true; @@ -2066,7 +2073,7 @@ RotationScaleMerge::ScalingLoopOutcome RotationScaleMerge::RunFullsScalingLoop( out.stalled = true; stop = true; } - if (!stop) { + if (!stop && out.step < SCALING_ANDERSON_ONSET) { // The next round starts from the accelerated scales, gauge-pinned again. const std::vector next = anderson.Next(log_g, log_g_new, weight); for (int f = 0; f < n_frames; ++f) diff --git a/tests/AndersonMixingTest.cpp b/tests/AndersonMixingTest.cpp index 255b8b76f..6eac50194 100644 --- a/tests/AndersonMixingTest.cpp +++ b/tests/AndersonMixingTest.cpp @@ -104,12 +104,15 @@ TEST_CASE("AndersonMixing_ZeroWeightTakesThePlainStep", "[RotationScale]") { CHECK(anderson.Restarts() == 0); } -TEST_CASE("AndersonMixing_RestartsWhenTheResidualGrows", "[RotationScale]") { +TEST_CASE("AndersonMixing_PlainOnceTheResidualGrows", "[RotationScale]") { AndersonMixing anderson(5); const std::vector w{1.0}; CHECK(anderson.Next({1.0}, {0.5}, w)[0] == 0.5); // no history: the plain step anderson.Next({0.5}, {0.25}, w); // residual fell: history kept + CHECK(!anderson.Stopped()); + CHECK(anderson.Next({0.0}, {2.0}, w)[0] == 2.0); // residual grew: the plain step + CHECK(anderson.Stopped()); + CHECK(anderson.Next({2.0}, {2.1}, w)[0] == 2.1); // and plain from then on, though it fell again + CHECK(anderson.Next({2.1}, {2.15}, w)[0] == 2.15); CHECK(anderson.Restarts() == 0); - CHECK(anderson.Next({0.0}, {2.0}, w)[0] == 2.0); // residual grew: history dropped, plain step - CHECK(anderson.Restarts() == 1); }