From 4620130f7115a8a6d54cd7cd215e34fbdb3b0133 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 19 Oct 2026 20:24:45 +0200 Subject: [PATCH] RotationScaleMerge: scaling loops run to an rms step of 1e-5, accelerated from 1e-2 (RESULT-CHANGING) The per-frame scaling tolerance goes from 1e-3 to 1e-5, and the Anderson onset (also the round the fulls loop keeps its cross-validated lambda) becomes an absolute 1e-2 instead of 10x the tolerance. Evaluated on 35 merged sets (ci tier + scaling-audit sets), on top of the loop fixes: - plain 1e-3: 8913 iterations, 38 s of loop time on GPU, 7 loops at the 100-round cap; - plain 1e-5: 35212 iterations, 120 s, 17 loops at the cap; - Anderson 1e-3 (previous default): 3715 iterations, 16 s, merged intensities as far from the converged ones as plain 1e-3 (median 0.13%, max 4.3%); - Anderson 1e-5 (this commit): 6507 iterations, 27 s, longest loop 89 rounds, merged intensities within 0.2% of plain 1e-5 and identical SG, d_min, CC1/2, ISa, R_meas, R_free. No space group or verdict changes in any mode. The converged merge is not measurably better against the references, but it no longer depends on where a loop happened to stop. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4 --- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 +- image_analysis/scale_merge/RotationScaleMerge.cpp | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index cdd8736a6..f2eb05d06 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -388,7 +388,7 @@ Before the combine a per-frame scale can also be fitted on the partials themselv How smooth that scale is over the rotation is left to the data rather than to a fixed window. Each round fits every frame on its own fulls against the current reference, giving a scale $G_f$ and its information $D_f=\sum w^2c^2$; the scale is then the curve $x=\log G$ minimising $\sum_f J_f\,(x_f-y_f)^2+\lambda\sum_f(\Delta^2 x)_f^2$, with $y_f$ the frame's own fit in log scale and $J_f$ its information carried there — a penalised (Whittaker–Eilers) smoother, solved as a five-band linear system. $\lambda$ is chosen by cross-validation: blocks of frames one rocking curve wide are left out in turn and predicted from the curve through the rest (neighbours closer than a rocking curve share their measurement, since a full sums those frames). A frame of hundreds of fulls is then followed frame by frame, a frame of two or three is carried by its neighbours, a stretch with none is bridged by a straight line, and a scale that falls a hundredfold over a few degrees — an absorbing crystal turning edge-on — is followed where a window would average across it. Once the curve settles, one free fit is shrunk toward it frame by frame by how much of each frame's deviation its neighbour shares (the lag-1 covariance), which hands back a real per-frame systematic and discards fit noise. -Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the set of fitted frames changes. Acceleration starts only once the plain step has fallen below ten times the tolerance, and stops for the rest of the loop the first time the step grows: Anderson mixing models a map near its fixed point, and a loop further out, or one whose step grows, may be walking rather than converging - accelerated, it walks faster and ends somewhere else. This changes the path, not the fixed point; the loops stop at the same tolerance (an rms step of $10^{-3}$ in $\log G$) in about a quarter of the rounds and closer to the converged scales than the plain iteration stopped there. +Both per-frame scaling loops — on the partials and on the fulls — alternate between the references given the scales and the scales given the references, and converge linearly: a round typically removes only 5–15% of the remaining step, and 2% on the slowest sweeps. Each round is therefore Anderson-accelerated in $\log G$ (type II, five rounds of history, each frame weighted by its observation count): the plain round's scales are corrected by the combination of the last rounds' steps that best cancels the current one, and the history is dropped whenever the set of fitted frames changes. Acceleration starts only once the plain step has fallen below $10^{-2}$, and stops for the rest of the loop the first time the step grows: Anderson mixing models a map near its fixed point, and a loop further out, or one whose step grows, may be walking rather than converging - accelerated, it walks faster and ends somewhere else. This changes the path, not the fixed point. With it the loops are run to an rms step of $10^{-5}$ in $\log G$ - in fewer rounds than a plain loop took to $10^{-3}$ - so the merge no longer depends on where a loop happened to stop. After scale-fulls, five **correction surfaces** are fitted on the combined fulls (rotation path, **on by default**; disable all with `--no-scaling-corrections`), each an alternating multiplicative refinement of the per-full scale against the merged reference: diff --git a/image_analysis/scale_merge/RotationScaleMerge.cpp b/image_analysis/scale_merge/RotationScaleMerge.cpp index 4ce0dea01..e096aeb1c 100644 --- a/image_analysis/scale_merge/RotationScaleMerge.cpp +++ b/image_analysis/scale_merge/RotationScaleMerge.cpp @@ -59,15 +59,17 @@ namespace { constexpr double MIN_EVENTS_PER_FRAME_SCALE_PARTIALS = 50.0; constexpr int64_t MIN_REFLECTIONS_FOR_IMAGE_CC = 20; // below this a frame's CC means nothing // The per-frame scaling loop has settled when the scales moved less than this between two - // iterations, as the rms of |log(G_new/G_old)| over the frames fitted in both. - constexpr double SCALING_TOLERANCE = 1e-3; + // iterations, as the rms of |log(G_new/G_old)| over the frames fitted in both. Tight enough that the + // merge no longer depends on where a loop stopped: with Anderson mixing a loop gets here in fewer + // rounds than a plain one took to 1e-3. + constexpr double SCALING_TOLERANCE = 1e-5; // The rounds of history the per-frame scaling loops are accelerated with (AndersonMixing). Their slow // directions are a few smooth modes over the sweep: measured on a 4nwv-class sweep, 3, 5 and 8 rounds // take its ten loops in 92, 91 and 90 iterations, against 364 without. constexpr int SCALING_ANDERSON_DEPTH = 5; - // The loops are accelerated only once their step is within this of the tolerance: Anderson mixing - // models the map near its fixed point, and a loop still further out is as likely to be walking. - constexpr double SCALING_ANDERSON_ONSET = 10.0 * SCALING_TOLERANCE; + // The loops are accelerated only once their step is below this: Anderson mixing models the map near + // its fixed point, and a loop still further out is as likely to be walking. + constexpr double SCALING_ANDERSON_ONSET = 1e-2; // A scaling loop has stopped settling when the trend of log(step) over its last this many rounds is // not falling (StepsStoppedFalling): a steady slow fall is a loop still contracting, a flat or // rising trend is not.