mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 67 Wiki Activity

rugnux: per-batch relative-B correction + radiation-damage B monitor

Browse Source 
Rotation scaling modelled dose-dependent scattering power with a single
global relative-B decay slope only (RefineDecay); per-frame refinement was
scale-only. Add a per-batch relative Debye-Waller B, both as an opt-in
correction and as an always-on measurement.

- FitRelativeBCurve / SolveCurvatureSmoothedB: per rotation-range batch, fit
  the s^2 slope of ln(Iref/Iobs) that the resolution-flat per-frame G leaves,
  regularised by a second-difference (curvature) penalty so a genuine smooth
  relative-B is kept while resolution-correlated noise (which an unconstrained
  per-batch fit would chase into an oscillating curve) is suppressed.

- Correction (--relative-b[=deg], default off): applied after RefineDecay,
  gated by a physical peak-to-peak floor and cross-validated by ASU-group
  parity (a per-batch parameter owns whole frames, so it is scored on held-out
  equivalents, not held-out frames). On a battery re-scaling identical stored
  reflections it is a no-op on well-behaved data and a small gain on
  radiation-damaged rotation data; never a regression.

- Radiation-damage monitor (always on for rotation, report-only): measure the
  relative-B change from the pristine low-dose start of the run to its end
  (a whole-run-mean reference inverts on damaged data) plus the per-batch
  curve, and report them on the command line (in the radiation-damage report
  and the merge statistics) and in the mmCIF (_reflns.jfjoch_radiation_damage_
  relative_B and a per-batch loop). A large magnitude flags radiation damage,
  complementing the existing per-image CC / mosaicity read.

- --scale now reproduces the de-novo rotation merge (it previously omitted
  scale-fulls and smooth-G, silently using a weaker model), and honours
  --relative-b, so an offline re-scale matches the full pipeline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-07-18 22:08:09 +02:00
co-authored by Claude Opus 4.8
parent e97a610091
commit c5cfdcef7e
9 changed files with 361 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/scale_merge/RotationScaleMerge.cpp
+244 -1
View File
@@ -173,6 +173,7 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment,
refine_decay_b = s.GetCorrectionSurfaces();
absorption_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
modulation_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
relative_b_deg = s.GetRelativeBDegrees(); // opt-in per-batch relative-B (0 = off)
scaling_iter = std::max(1, scaling_iterations);
resolution_cutoff_method = s.GetResolutionCutoff();
resolution_cc_target = s.GetResolutionCCTarget();
@@ -650,6 +651,236 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
0.5 * slope * n_frames, 0.5 * slope);
}
std::vector<double> RotationScaleMerge::SolveCurvatureSmoothedB(const std::vector<double> &num,
const std::vector<double> &den) const {
// Solve for a SMOOTH per-batch relative-B from the per-batch normal-equation data (num_c = sum w s^2 y,
// den_c = sum w s^4). A per-batch regression alone (b_c = num_c/den_c) over-fits: a genuine relative-B is
// a low-curvature function of dose / orientation, but an unconstrained fit chases resolution-correlated
// noise into a rough, oscillating curve. So minimise data-fidelity PLUS a second-difference (curvature)
// penalty sum_c den_c (b_c - num_c/den_c)^2 + mu * sum_c (b_{c-1} - 2 b_c + b_{c+1})^2, which leaves a
// straight RAMP untouched but suppresses oscillation, solving the small SPD normal system A b = num,
// A = diag(den + lambda) + mu * D^T D, by Gauss-Seidel (n_batch <= a few tens). Not anchored - callers set
// the gauge (zero-mean for the correction, low-dose reference for the monitor).
constexpr double RELATIVE_B_MAX = 50.0; // A^2, clamp a single batch's relative-B (generous guard)
constexpr double RELATIVE_B_CURVATURE_K = 8.0; // curvature penalty weight (x the median batch leverage)
const int n_batch = static_cast<int>(num.size());
std::vector<double> dsorted(den);
std::sort(dsorted.begin(), dsorted.end());
const double med_den = dsorted.empty() ? 0.0 : dsorted[dsorted.size() / 2];
const double lambda = 0.1 * med_den; // ridge ~ 10% of the median batch leverage
const double mu = RELATIVE_B_CURVATURE_K * med_den; // curvature penalty
std::vector<double> A(static_cast<size_t>(n_batch) * n_batch, 0.0);
auto Aat = [&](int i, int j) -> double & { return A[static_cast<size_t>(i) * n_batch + j]; };
for (int c = 0; c < n_batch; ++c) Aat(c, c) = den[c] + lambda;
constexpr int st[3] = {-1, 0, 1};
constexpr double sv[3] = {1.0, -2.0, 1.0}; // second-difference stencil
for (int c = 1; c < n_batch - 1; ++c)
for (int a = 0; a < 3; ++a)
for (int b2 = 0; b2 < 3; ++b2)
Aat(c + st[a], c + st[b2]) += mu * sv[a] * sv[b2];
std::vector<double> b(n_batch, 0.0);
for (int it = 0; it < 100; ++it)
for (int c = 0; c < n_batch; ++c) {
if (!(Aat(c, c) > 0.0)) continue;
double s = num[c];
for (int j = 0; j < n_batch; ++j)
if (j != c) s -= Aat(c, j) * b[j];
b[c] = std::clamp(s / Aat(c, c), -RELATIVE_B_MAX, RELATIVE_B_MAX);
}
return b;
}
std::vector<double> RotationScaleMerge::FitRelativeBCurve(int n_groups, int n_batch, int frames_per_batch,
int gparity) const {
// Fit one relative-B per batch over the ASU-group subset {group&1 == gparity} (gparity < 0 = all fulls),
// against an inverse-variance reference built from that same subset (no leakage). Per batch the model is
// y = ln(I_ref / I_obs) = b * s^2 through the origin (the resolution-flat intercept is the per-frame G,
// already applied), so the data term is b_c = sum(w s^2 y) / sum(w s^2 s^2); SolveCurvatureSmoothedB then
// smooths it. Finally subtract the leverage-weighted mean: a constant B is a global Wilson-B degenerate
// with overall scale, and cannot change equivalent agreement (within one ASU group every equivalent sits
// at the same s^2, so only b VARYING between batches moves the merge). Drives the per-batch correction.
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
&& o.partiality >= min_partiality;
};
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity)) continue;
const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
sw[o.group] += w; swI[o.group] += w * static_cast<double>(o.I) * o.corr;
}
std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity) || sw[o.group] <= 0.0) continue;
const double Iref = swI[o.group] / sw[o.group];
const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
const double w = (Is / sc) * (Is / sc);
const double s2 = s2_of(o.d), y = std::log(Iref / Is);
num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
}
std::vector<double> b = SolveCurvatureSmoothedB(num, den);
double dw = 0.0, dbw = 0.0;
for (int c = 0; c < n_batch; ++c) { dw += den[c]; dbw += den[c] * b[c]; }
const double bmean = dw > 0.0 ? dbw / dw : 0.0;
for (auto &v : b) v -= bmean;
return b;
}
void RotationScaleMerge::MeasureRadiationDamageB(int n_groups) {
// Radiation-damage MONITOR (report-only): measure the per-batch relative Debye-Waller B on the scaled
// fulls BEFORE any correction, so it captures how the high-resolution scattering power fades with
// accumulated dose. Crucially the reference is the PRISTINE LOW-DOSE crystal (first ~10% of frames), not
// the whole-run mean: a group measured both early and late then contributes ln(I_early / I_late) vs s^2,
// whose slope is the ADDED B - an absolute, correctly-signed damage read (positive = high-res fades).
// A whole-run-mean reference inverts on heavily damaged data (the surviving late high-res reflections
// look strong against a mean dominated by the frames where they had already faded). Stores the first->last
// relative-B change (the headline "B-factor difference") and the per-batch curve; never touches corr.
rad_damage_delta_b = std::numeric_limits<double>::quiet_NaN();
rad_damage_b_batch.clear();
rad_damage_batch_deg = 0.0;
if (fulls.empty() || n_frames <= 0)
return;
const auto gon = x.GetGoniometer();
const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
if (!(osc_deg > 1e-6))
return;
constexpr double MONITOR_BATCH_DEG = 10.0; // rotation width per batch for the damage curve
const int frames_per_batch = std::max(1, static_cast<int>(std::lround(MONITOR_BATCH_DEG / osc_deg)));
const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
if (n_batch < 2) // a single batch is a global Wilson-B (degenerate with scale) - nothing to monitor
return;
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
&& o.partiality >= min_partiality;
};
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
// Low-dose reference: inverse-variance group means over the first ~10% of frames (>= 1 batch).
const int ref_frames = std::max(frames_per_batch, n_frames / 10);
std::vector<double> sw0(n_groups, 0.0), swI0(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || o.frame >= ref_frames) continue;
const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
sw0[o.group] += w; swI0[o.group] += w * static_cast<double>(o.I) * o.corr;
}
// Per batch: slope of ln(I_ref0 / I_obs) vs s^2, over groups also seen in the low-dose reference.
std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || sw0[o.group] <= 0.0) continue;
const double Iref = swI0[o.group] / sw0[o.group];
const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
const double w = (Is / sc) * (Is / sc);
const double s2 = s2_of(o.d), y = std::log(Iref / Is);
num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
}
std::vector<double> b = SolveCurvatureSmoothedB(num, den);
// Anchor so the reference batches read ~0 (b = added B relative to the low-dose start).
const int n_ref_batch = std::max(1, ref_frames / frames_per_batch);
double ref_mean = 0.0;
for (int c = 0; c < n_ref_batch && c < n_batch; ++c) ref_mean += b[c];
ref_mean /= std::min(n_ref_batch, n_batch);
for (auto &v : b) v -= ref_mean;
// Headline: the first->last relative-B change from a least-squares linear trend of the (smoothed) curve -
// robust to a single noisy end batch, unlike raw endpoints. Positive = high-res fades with dose = damage.
double sx = 0, sy = 0, sxx = 0, sxy = 0;
for (int c = 0; c < n_batch; ++c) { sx += c; sy += b[c]; sxx += double(c) * c; sxy += double(c) * b[c]; }
const double det = n_batch * sxx - sx * sx;
const double slope = det > 0.0 ? (n_batch * sxy - sx * sy) / det : 0.0;
rad_damage_delta_b = slope * (n_batch - 1);
rad_damage_batch_deg = MONITOR_BATCH_DEG;
rad_damage_b_batch.resize(n_batch);
for (int c = 0; c < n_batch; ++c) rad_damage_b_batch[c] = static_cast<float>(b[c]);
}
void RotationScaleMerge::RefineRelativeB(int n_groups) {
// RefineDecay removes the AVERAGE radiation-damage falloff as a single global relative-B slope, but the
// relative scattering power drifts NON-monotonically across a run (absorption path as the crystal
// rotates, slippage, dose bursts, a partly-shadowed wedge). XDS/AIMLESS model this with a per-BATCH
// relative-B on top of the per-batch scale. Here, on the fulls after RefineDecay: fit the smoothed
// per-batch relative-B (FitRelativeBCurve) and apply exp(b_batch * s^2). Same anti-overfit posture as
// RefineDecay - a physical spread floor and a cross-validated held-out gain - but the CV split is by ASU
// GROUP parity, not frame parity: a per-batch parameter owns whole frames, so it cannot be scored on a
// held-out FRAME; splitting the equivalents instead tests whether a batch's B generalises to reflections
// it was not fit on, which is exactly the over-fit risk of a many-parameter model.
if (fulls.empty() || n_frames <= 0 || relative_b_deg <= 0.0)
return;
const auto gon = x.GetGoniometer();
const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
if (!(osc_deg > 1e-6))
return;
const int frames_per_batch = std::max(1, static_cast<int>(std::lround(relative_b_deg / osc_deg)));
const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
if (n_batch < 2) // a single batch is a global Wilson-B (degenerate with overall scale) - nothing relative
return;
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
&& o.partiality >= min_partiality;
};
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
auto b_factor = [&](const Obs &o, const std::vector<double> &b) {
return std::exp(b[batch_of(o)] * s2_of(o.d));
};
// Held-out disagreement over the group subset {group&1 == gparity} with `b` applied: the same
// sigma-INDEPENDENT Rmeas-like metric as RefineDecay (sum|Is-Iref|/sum|Iref|), so a relative-B cannot
// "win" by reshaping sigma. Reference built from the scored subset (no leakage).
auto subset_disagreement = [&](int gparity, const std::vector<double> &b) -> double {
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (o.group & 1) != gparity) continue;
const double f = b_factor(o, b);
const double Is = static_cast<double>(o.I) * o.corr * f, sc = static_cast<double>(o.sigma) * o.corr * f;
const double w = 1.0 / (sc * sc);
sw[o.group] += w; swI[o.group] += w * Is;
}
double num = 0.0, den = 0.0;
for (const auto &o : fulls) {
if (!usable(o) || (o.group & 1) != gparity || sw[o.group] <= 0.0) continue;
const double Iref = swI[o.group] / sw[o.group];
const double Is = static_cast<double>(o.I) * o.corr * b_factor(o, b);
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
num += std::abs(Is - Iref); den += Iref;
}
return den > 0.0 ? num / den : 0.0;
};
// A relative-B whose peak-to-peak is below a couple A^2 is negligible; "correcting" it only spreads
// equivalents (same s^2, different batch) into extra scatter - net harmful, exactly like RefineDecay's
// sub-floor slope. (RELATIVE_B_MIN_SPREAD ~ DECAY_MIN_DELTA_B.)
constexpr double RELATIVE_B_MIN_SPREAD = 2.0; // A^2, minimum peak-to-peak relative-B to engage
const std::vector<double> b_all = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, -1);
const double bmin = *std::min_element(b_all.begin(), b_all.end());
const double bmax = *std::max_element(b_all.begin(), b_all.end());
if (bmax - bmin < RELATIVE_B_MIN_SPREAD) {
logger.Info("Relative-B: negligible variation ({} batches, peak-to-peak {:.2f} A^2 < {:.1f}, skipped)",
n_batch, bmax - bmin, RELATIVE_B_MIN_SPREAD);
return;
}
// Cross-validate: fit on even ASU groups, score the held-out odd equivalents (and vice-versa). Real
// per-batch structure generalises across the equivalent split; over-fit per-batch noise does not.
const std::vector<double> zero(n_batch, 0.0);
const double base = subset_disagreement(0, zero) + subset_disagreement(1, zero);
const std::vector<double> b_even = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 0);
const std::vector<double> b_odd = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 1);
const double gain = base - (subset_disagreement(1, b_even) + subset_disagreement(0, b_odd));
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
logger.Info("Relative-B: not cross-validated ({} batches, peak-to-peak {:.2f} A^2, held-out gain {:.1f}%, skipped)",
n_batch, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
return;
}
for (auto &o : fulls)
if (usable(o))
o.corr = static_cast<float>(o.corr * b_factor(o, b_all));
logger.Info("Relative-B: {} batches of ~{:.1f} deg, peak-to-peak {:.2f} A^2, held-out gain {:.1f}% (cross-validated)",
n_batch, relative_b_deg, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
}
void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
// Absorption / path-length: a smooth multiplicative factor over the diffracted-beam direction in the
// goniometer (crystal) frame. Each full's lab diffracted direction (from its predicted detector
@@ -1567,6 +1798,12 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
logger.Info("Anomalous signal SigAno = {:.2f} ({} acentric pairs)",
overall.abs_diff_over_sigma_anomalous, sig_n);
// Radiation-damage monitor (measured before any correction by MeasureRadiationDamageB): carry the
// first->last relative-B change and the per-batch curve into the reported statistics / mmCIF.
out.radiation_damage_delta_b = rad_damage_delta_b;
out.radiation_damage_b_batch = rad_damage_b_batch;
out.radiation_damage_batch_deg = rad_damage_batch_deg;
// Attach the per-reflection anomalous split so the writer can emit I(+)/I(-) by default (each merged
// reflection maps to its Friedel-ASU key; in an anomalous merge both mates map to the same key).
if (!anom_export.empty()) {
@@ -1757,9 +1994,15 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
// in-symmetry merge, never the P1 space-group search pass (corrections there add risk and can
// perturb the symmetry determination). ---
[[maybe_unused]] const bool corrections =
!for_search && (refine_decay_b || absorption_iter > 0 || modulation_iter > 0);
!for_search && (refine_decay_b || absorption_iter > 0 || modulation_iter > 0 || relative_b_deg > 0.0);
// Radiation-damage monitor: measure the per-batch relative-B on the scaled fulls BEFORE any correction
// (report-only; captures the full damage signature, not a residual). Skipped on the P1 search pass.
if (!for_search)
MeasureRadiationDamageB(n_groups);
if (!for_search && refine_decay_b)
RefineDecay(n_groups);
if (!for_search && relative_b_deg > 0.0)
RefineRelativeB(n_groups); // per-batch relative-B on top of the single decay slope
if (!for_search && absorption_iter > 0)
RefineAbsorption(absorption_iter, n_groups);
if (!for_search && modulation_iter > 0)