v1.0.0-rc.160 (#70)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #70.
This commit is contained in:
@@ -11,6 +11,7 @@
|
||||
#include <future>
|
||||
#include <limits>
|
||||
#include <random>
|
||||
#include <unordered_map>
|
||||
|
||||
#include <gemmi/reciproc.hpp>
|
||||
#include <gemmi/symmetry.hpp>
|
||||
@@ -171,6 +172,8 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment,
|
||||
// so a no-op when their systematic is absent). Decoupled from the stills-only -B / RefineB flag.
|
||||
refine_decay_b = s.GetCorrectionSurfaces();
|
||||
absorption_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
|
||||
modulation_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
|
||||
relative_b_deg = s.GetRelativeBDegrees(); // opt-in per-batch relative-B (0 = off)
|
||||
scaling_iter = std::max(1, scaling_iterations);
|
||||
resolution_cutoff_method = s.GetResolutionCutoff();
|
||||
resolution_cc_target = s.GetResolutionCCTarget();
|
||||
@@ -553,9 +556,10 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
|
||||
// noise / aggregate structure (not real per-reflection decay) would amplify into a high-res
|
||||
// mis-correction and worsen the symmetry-equivalent scatter that sets ISa. An opt-in correction must
|
||||
// never add noise where its systematic is absent (a low-dose reference set), so the slope is
|
||||
// CROSS-VALIDATED: fit on even-frame fulls and score the held-out odd-frame equivalent chi^2, and vice
|
||||
// versa. Real decay generalizes across the split (both held-out scores improve); a noise slope does
|
||||
// not. Apply the full-data slope only if the mean held-out chi^2 strictly improves; else it is a no-op.
|
||||
// CROSS-VALIDATED: fit on even-frame fulls and score the held-out odd-frame equivalent agreement (a
|
||||
// sigma-independent Rmeas-like metric), and vice versa. Real decay generalizes across the split (both
|
||||
// held-out scores improve); a noise slope does not. Apply the full-data slope only if the mean held-out
|
||||
// agreement strictly improves; else it is a no-op.
|
||||
if (fulls.empty() || n_frames <= 0)
|
||||
return;
|
||||
const double fcenter = 0.5 * n_frames;
|
||||
@@ -592,9 +596,14 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
|
||||
// strong damage (slope = 2 dB/dframe, so +-1.0 admits total relative-B up to ~n_frames/2 A^2).
|
||||
return (var > 0.0) ? std::clamp((Sxy - Sx * Sy / Sw) / var, -1.0, 1.0) : 0.0;
|
||||
};
|
||||
// Mean studentized squared deviation over the subset {frame&1 == parity} when `slope` is applied,
|
||||
// scored against that subset's own reference (no leakage). Lower = tighter equivalents.
|
||||
auto subset_chi2 = [&](int parity, double slope) -> double {
|
||||
// Rmeas-like (sigma-INDEPENDENT) disagreement of the equivalents over the subset {frame&1 == parity}
|
||||
// when `slope` is applied, scored against that subset's own reference (no leakage): sum|Is-Iref|/sum|Iref|.
|
||||
// Scoring on the studentized deviation instead would let the slope "improve" the held-out chi^2 by
|
||||
// reshaping sigma - it scales sigma up on late / high-angle frames, down-weighting them - without
|
||||
// actually tightening the intensities; a fractional metric cannot be gamed that way. This mirrors the
|
||||
// absorption / modulation gate in ApplyCellSurface (which was switched off studentized chi^2 for the
|
||||
// same reason); the reference is still the inverse-variance mean.
|
||||
auto subset_disagreement = [&](int parity, double slope) -> double {
|
||||
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || (o.frame & 1) != parity) continue;
|
||||
@@ -603,17 +612,15 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
|
||||
const double w = 1.0 / (sc * sc);
|
||||
sw[o.group] += w; swI[o.group] += w * Is;
|
||||
}
|
||||
double sum = 0.0; size_t n = 0;
|
||||
double num = 0.0, den = 0.0;
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || (o.frame & 1) != parity || sw[o.group] <= 0.0) continue;
|
||||
const double Iref = swI[o.group] / sw[o.group];
|
||||
const double Is = static_cast<double>(o.I) * o.corr * decay_factor(o, slope);
|
||||
const double sc = static_cast<double>(o.sigma) * o.corr * decay_factor(o, slope);
|
||||
if (!std::isfinite(Iref) || !(sc > 0.0)) continue;
|
||||
const double dd = (Is - Iref) / sc;
|
||||
sum += dd * dd; ++n;
|
||||
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
|
||||
num += std::abs(Is - Iref); den += Iref;
|
||||
}
|
||||
return n > 0 ? sum / static_cast<double>(n) : 0.0;
|
||||
return den > 0.0 ? num / den : 0.0;
|
||||
};
|
||||
|
||||
// A meaningful radiation-damage relative-B accrues several A^2 across a dataset. Below a physical floor
|
||||
@@ -630,8 +637,8 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
|
||||
}
|
||||
// And require the slope to cross-validate by a clear margin: fit on even frames, score the held-out
|
||||
// odd equivalents (and vice versa). Real damage generalizes; a fluke does not.
|
||||
const double base = subset_chi2(1, 0.0) + subset_chi2(0, 0.0);
|
||||
const double gain = base - (subset_chi2(1, fit_slope(0)) + subset_chi2(0, fit_slope(1)));
|
||||
const double base = subset_disagreement(1, 0.0) + subset_disagreement(0, 0.0);
|
||||
const double gain = base - (subset_disagreement(1, fit_slope(0)) + subset_disagreement(0, fit_slope(1)));
|
||||
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
|
||||
logger.Info("Decay correction: not cross-validated (dB = {:.2f} A^2, held-out gain {:.1f}%, skipped)",
|
||||
total_delta_B, 100.0 * gain / std::max(base, 1e-30));
|
||||
@@ -644,6 +651,236 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
|
||||
0.5 * slope * n_frames, 0.5 * slope);
|
||||
}
|
||||
|
||||
std::vector<double> RotationScaleMerge::SolveCurvatureSmoothedB(const std::vector<double> &num,
|
||||
const std::vector<double> &den) const {
|
||||
// Solve for a SMOOTH per-batch relative-B from the per-batch normal-equation data (num_c = sum w s^2 y,
|
||||
// den_c = sum w s^4). A per-batch regression alone (b_c = num_c/den_c) over-fits: a genuine relative-B is
|
||||
// a low-curvature function of dose / orientation, but an unconstrained fit chases resolution-correlated
|
||||
// noise into a rough, oscillating curve. So minimise data-fidelity PLUS a second-difference (curvature)
|
||||
// penalty sum_c den_c (b_c - num_c/den_c)^2 + mu * sum_c (b_{c-1} - 2 b_c + b_{c+1})^2, which leaves a
|
||||
// straight RAMP untouched but suppresses oscillation, solving the small SPD normal system A b = num,
|
||||
// A = diag(den + lambda) + mu * D^T D, by Gauss-Seidel (n_batch <= a few tens). Not anchored - callers set
|
||||
// the gauge (zero-mean for the correction, low-dose reference for the monitor).
|
||||
constexpr double RELATIVE_B_MAX = 50.0; // A^2, clamp a single batch's relative-B (generous guard)
|
||||
constexpr double RELATIVE_B_CURVATURE_K = 8.0; // curvature penalty weight (x the median batch leverage)
|
||||
const int n_batch = static_cast<int>(num.size());
|
||||
std::vector<double> dsorted(den);
|
||||
std::sort(dsorted.begin(), dsorted.end());
|
||||
const double med_den = dsorted.empty() ? 0.0 : dsorted[dsorted.size() / 2];
|
||||
const double lambda = 0.1 * med_den; // ridge ~ 10% of the median batch leverage
|
||||
const double mu = RELATIVE_B_CURVATURE_K * med_den; // curvature penalty
|
||||
std::vector<double> A(static_cast<size_t>(n_batch) * n_batch, 0.0);
|
||||
auto Aat = [&](int i, int j) -> double & { return A[static_cast<size_t>(i) * n_batch + j]; };
|
||||
for (int c = 0; c < n_batch; ++c) Aat(c, c) = den[c] + lambda;
|
||||
constexpr int st[3] = {-1, 0, 1};
|
||||
constexpr double sv[3] = {1.0, -2.0, 1.0}; // second-difference stencil
|
||||
for (int c = 1; c < n_batch - 1; ++c)
|
||||
for (int a = 0; a < 3; ++a)
|
||||
for (int b2 = 0; b2 < 3; ++b2)
|
||||
Aat(c + st[a], c + st[b2]) += mu * sv[a] * sv[b2];
|
||||
std::vector<double> b(n_batch, 0.0);
|
||||
for (int it = 0; it < 100; ++it)
|
||||
for (int c = 0; c < n_batch; ++c) {
|
||||
if (!(Aat(c, c) > 0.0)) continue;
|
||||
double s = num[c];
|
||||
for (int j = 0; j < n_batch; ++j)
|
||||
if (j != c) s -= Aat(c, j) * b[j];
|
||||
b[c] = std::clamp(s / Aat(c, c), -RELATIVE_B_MAX, RELATIVE_B_MAX);
|
||||
}
|
||||
return b;
|
||||
}
|
||||
|
||||
std::vector<double> RotationScaleMerge::FitRelativeBCurve(int n_groups, int n_batch, int frames_per_batch,
|
||||
int gparity) const {
|
||||
// Fit one relative-B per batch over the ASU-group subset {group&1 == gparity} (gparity < 0 = all fulls),
|
||||
// against an inverse-variance reference built from that same subset (no leakage). Per batch the model is
|
||||
// y = ln(I_ref / I_obs) = b * s^2 through the origin (the resolution-flat intercept is the per-frame G,
|
||||
// already applied), so the data term is b_c = sum(w s^2 y) / sum(w s^2 s^2); SolveCurvatureSmoothedB then
|
||||
// smooths it. Finally subtract the leverage-weighted mean: a constant B is a global Wilson-B degenerate
|
||||
// with overall scale, and cannot change equivalent agreement (within one ASU group every equivalent sits
|
||||
// at the same s^2, so only b VARYING between batches moves the merge). Drives the per-batch correction.
|
||||
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
|
||||
auto usable = [&](const Obs &o) {
|
||||
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
|
||||
&& o.partiality >= min_partiality;
|
||||
};
|
||||
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
|
||||
|
||||
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity)) continue;
|
||||
const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
|
||||
sw[o.group] += w; swI[o.group] += w * static_cast<double>(o.I) * o.corr;
|
||||
}
|
||||
std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity) || sw[o.group] <= 0.0) continue;
|
||||
const double Iref = swI[o.group] / sw[o.group];
|
||||
const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
|
||||
if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
|
||||
const double w = (Is / sc) * (Is / sc);
|
||||
const double s2 = s2_of(o.d), y = std::log(Iref / Is);
|
||||
num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
|
||||
}
|
||||
std::vector<double> b = SolveCurvatureSmoothedB(num, den);
|
||||
double dw = 0.0, dbw = 0.0;
|
||||
for (int c = 0; c < n_batch; ++c) { dw += den[c]; dbw += den[c] * b[c]; }
|
||||
const double bmean = dw > 0.0 ? dbw / dw : 0.0;
|
||||
for (auto &v : b) v -= bmean;
|
||||
return b;
|
||||
}
|
||||
|
||||
void RotationScaleMerge::MeasureRadiationDamageB(int n_groups) {
|
||||
// Radiation-damage MONITOR (report-only): measure the per-batch relative Debye-Waller B on the scaled
|
||||
// fulls BEFORE any correction, so it captures how the high-resolution scattering power fades with
|
||||
// accumulated dose. Crucially the reference is the PRISTINE LOW-DOSE crystal (first ~10% of frames), not
|
||||
// the whole-run mean: a group measured both early and late then contributes ln(I_early / I_late) vs s^2,
|
||||
// whose slope is the ADDED B - an absolute, correctly-signed damage read (positive = high-res fades).
|
||||
// A whole-run-mean reference inverts on heavily damaged data (the surviving late high-res reflections
|
||||
// look strong against a mean dominated by the frames where they had already faded). Stores the first->last
|
||||
// relative-B change (the headline "B-factor difference") and the per-batch curve; never touches corr.
|
||||
rad_damage_delta_b = std::numeric_limits<double>::quiet_NaN();
|
||||
rad_damage_b_batch.clear();
|
||||
rad_damage_batch_deg = 0.0;
|
||||
if (fulls.empty() || n_frames <= 0)
|
||||
return;
|
||||
const auto gon = x.GetGoniometer();
|
||||
const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
|
||||
if (!(osc_deg > 1e-6))
|
||||
return;
|
||||
constexpr double MONITOR_BATCH_DEG = 10.0; // rotation width per batch for the damage curve
|
||||
const int frames_per_batch = std::max(1, static_cast<int>(std::lround(MONITOR_BATCH_DEG / osc_deg)));
|
||||
const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
|
||||
if (n_batch < 2) // a single batch is a global Wilson-B (degenerate with scale) - nothing to monitor
|
||||
return;
|
||||
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
|
||||
auto usable = [&](const Obs &o) {
|
||||
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
|
||||
&& o.partiality >= min_partiality;
|
||||
};
|
||||
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
|
||||
|
||||
// Low-dose reference: inverse-variance group means over the first ~10% of frames (>= 1 batch).
|
||||
const int ref_frames = std::max(frames_per_batch, n_frames / 10);
|
||||
std::vector<double> sw0(n_groups, 0.0), swI0(n_groups, 0.0);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || o.frame >= ref_frames) continue;
|
||||
const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
|
||||
sw0[o.group] += w; swI0[o.group] += w * static_cast<double>(o.I) * o.corr;
|
||||
}
|
||||
// Per batch: slope of ln(I_ref0 / I_obs) vs s^2, over groups also seen in the low-dose reference.
|
||||
std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || sw0[o.group] <= 0.0) continue;
|
||||
const double Iref = swI0[o.group] / sw0[o.group];
|
||||
const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
|
||||
if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
|
||||
const double w = (Is / sc) * (Is / sc);
|
||||
const double s2 = s2_of(o.d), y = std::log(Iref / Is);
|
||||
num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
|
||||
}
|
||||
std::vector<double> b = SolveCurvatureSmoothedB(num, den);
|
||||
// Anchor so the reference batches read ~0 (b = added B relative to the low-dose start).
|
||||
const int n_ref_batch = std::max(1, ref_frames / frames_per_batch);
|
||||
double ref_mean = 0.0;
|
||||
for (int c = 0; c < n_ref_batch && c < n_batch; ++c) ref_mean += b[c];
|
||||
ref_mean /= std::min(n_ref_batch, n_batch);
|
||||
for (auto &v : b) v -= ref_mean;
|
||||
// Headline: the first->last relative-B change from a least-squares linear trend of the (smoothed) curve -
|
||||
// robust to a single noisy end batch, unlike raw endpoints. Positive = high-res fades with dose = damage.
|
||||
double sx = 0, sy = 0, sxx = 0, sxy = 0;
|
||||
for (int c = 0; c < n_batch; ++c) { sx += c; sy += b[c]; sxx += double(c) * c; sxy += double(c) * b[c]; }
|
||||
const double det = n_batch * sxx - sx * sx;
|
||||
const double slope = det > 0.0 ? (n_batch * sxy - sx * sy) / det : 0.0;
|
||||
rad_damage_delta_b = slope * (n_batch - 1);
|
||||
rad_damage_batch_deg = MONITOR_BATCH_DEG;
|
||||
rad_damage_b_batch.resize(n_batch);
|
||||
for (int c = 0; c < n_batch; ++c) rad_damage_b_batch[c] = static_cast<float>(b[c]);
|
||||
}
|
||||
|
||||
void RotationScaleMerge::RefineRelativeB(int n_groups) {
|
||||
// RefineDecay removes the AVERAGE radiation-damage falloff as a single global relative-B slope, but the
|
||||
// relative scattering power drifts NON-monotonically across a run (absorption path as the crystal
|
||||
// rotates, slippage, dose bursts, a partly-shadowed wedge). XDS/AIMLESS model this with a per-BATCH
|
||||
// relative-B on top of the per-batch scale. Here, on the fulls after RefineDecay: fit the smoothed
|
||||
// per-batch relative-B (FitRelativeBCurve) and apply exp(b_batch * s^2). Same anti-overfit posture as
|
||||
// RefineDecay - a physical spread floor and a cross-validated held-out gain - but the CV split is by ASU
|
||||
// GROUP parity, not frame parity: a per-batch parameter owns whole frames, so it cannot be scored on a
|
||||
// held-out FRAME; splitting the equivalents instead tests whether a batch's B generalises to reflections
|
||||
// it was not fit on, which is exactly the over-fit risk of a many-parameter model.
|
||||
if (fulls.empty() || n_frames <= 0 || relative_b_deg <= 0.0)
|
||||
return;
|
||||
const auto gon = x.GetGoniometer();
|
||||
const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
|
||||
if (!(osc_deg > 1e-6))
|
||||
return;
|
||||
const int frames_per_batch = std::max(1, static_cast<int>(std::lround(relative_b_deg / osc_deg)));
|
||||
const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
|
||||
if (n_batch < 2) // a single batch is a global Wilson-B (degenerate with overall scale) - nothing relative
|
||||
return;
|
||||
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
|
||||
auto usable = [&](const Obs &o) {
|
||||
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
|
||||
&& o.partiality >= min_partiality;
|
||||
};
|
||||
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
|
||||
auto b_factor = [&](const Obs &o, const std::vector<double> &b) {
|
||||
return std::exp(b[batch_of(o)] * s2_of(o.d));
|
||||
};
|
||||
// Held-out disagreement over the group subset {group&1 == gparity} with `b` applied: the same
|
||||
// sigma-INDEPENDENT Rmeas-like metric as RefineDecay (sum|Is-Iref|/sum|Iref|), so a relative-B cannot
|
||||
// "win" by reshaping sigma. Reference built from the scored subset (no leakage).
|
||||
auto subset_disagreement = [&](int gparity, const std::vector<double> &b) -> double {
|
||||
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || (o.group & 1) != gparity) continue;
|
||||
const double f = b_factor(o, b);
|
||||
const double Is = static_cast<double>(o.I) * o.corr * f, sc = static_cast<double>(o.sigma) * o.corr * f;
|
||||
const double w = 1.0 / (sc * sc);
|
||||
sw[o.group] += w; swI[o.group] += w * Is;
|
||||
}
|
||||
double num = 0.0, den = 0.0;
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable(o) || (o.group & 1) != gparity || sw[o.group] <= 0.0) continue;
|
||||
const double Iref = swI[o.group] / sw[o.group];
|
||||
const double Is = static_cast<double>(o.I) * o.corr * b_factor(o, b);
|
||||
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
|
||||
num += std::abs(Is - Iref); den += Iref;
|
||||
}
|
||||
return den > 0.0 ? num / den : 0.0;
|
||||
};
|
||||
|
||||
// A relative-B whose peak-to-peak is below a couple A^2 is negligible; "correcting" it only spreads
|
||||
// equivalents (same s^2, different batch) into extra scatter - net harmful, exactly like RefineDecay's
|
||||
// sub-floor slope. (RELATIVE_B_MIN_SPREAD ~ DECAY_MIN_DELTA_B.)
|
||||
constexpr double RELATIVE_B_MIN_SPREAD = 2.0; // A^2, minimum peak-to-peak relative-B to engage
|
||||
const std::vector<double> b_all = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, -1);
|
||||
const double bmin = *std::min_element(b_all.begin(), b_all.end());
|
||||
const double bmax = *std::max_element(b_all.begin(), b_all.end());
|
||||
if (bmax - bmin < RELATIVE_B_MIN_SPREAD) {
|
||||
logger.Info("Relative-B: negligible variation ({} batches, peak-to-peak {:.2f} A^2 < {:.1f}, skipped)",
|
||||
n_batch, bmax - bmin, RELATIVE_B_MIN_SPREAD);
|
||||
return;
|
||||
}
|
||||
// Cross-validate: fit on even ASU groups, score the held-out odd equivalents (and vice-versa). Real
|
||||
// per-batch structure generalises across the equivalent split; over-fit per-batch noise does not.
|
||||
const std::vector<double> zero(n_batch, 0.0);
|
||||
const double base = subset_disagreement(0, zero) + subset_disagreement(1, zero);
|
||||
const std::vector<double> b_even = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 0);
|
||||
const std::vector<double> b_odd = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 1);
|
||||
const double gain = base - (subset_disagreement(1, b_even) + subset_disagreement(0, b_odd));
|
||||
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
|
||||
logger.Info("Relative-B: not cross-validated ({} batches, peak-to-peak {:.2f} A^2, held-out gain {:.1f}%, skipped)",
|
||||
n_batch, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
|
||||
return;
|
||||
}
|
||||
for (auto &o : fulls)
|
||||
if (usable(o))
|
||||
o.corr = static_cast<float>(o.corr * b_factor(o, b_all));
|
||||
logger.Info("Relative-B: {} batches of ~{:.1f} deg, peak-to-peak {:.2f} A^2, held-out gain {:.1f}% (cross-validated)",
|
||||
n_batch, relative_b_deg, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
|
||||
}
|
||||
|
||||
void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
|
||||
// Absorption / path-length: a smooth multiplicative factor over the diffracted-beam direction in the
|
||||
// goniometer (crystal) frame. Each full's lab diffracted direction (from its predicted detector
|
||||
@@ -661,7 +898,6 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
|
||||
return;
|
||||
|
||||
constexpr int NB = 8;
|
||||
const int ncell = NB * NB;
|
||||
std::vector<int32_t> cell(fulls.size(), -1);
|
||||
for (size_t i = 0; i < fulls.size(); ++i) {
|
||||
const Obs &o = fulls[i];
|
||||
@@ -673,14 +909,46 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
|
||||
const int iy = std::clamp(static_cast<int>((u.y + 1.0f) * 0.5f * NB), 0, NB - 1);
|
||||
cell[i] = ix * NB + iy;
|
||||
}
|
||||
ApplyCellSurface(cell, NB * NB, n_iter, n_groups, "Absorption (goniometer-frame 8x8)");
|
||||
}
|
||||
|
||||
void RotationScaleMerge::RefineModulation(int n_iter, int n_groups) {
|
||||
// Detector-plane modulation (flat-field): a smooth multiplicative factor over the position where a
|
||||
// reflection lands on the detector (predicted px, py). It absorbs detector-response and geometric
|
||||
// systematics that vary across the detector plane; symmetry-equivalents of one reflection land at
|
||||
// different positions as the crystal rotates, over-determining the surface. Because it is a property of
|
||||
// the detector + beam (not the rotation), the same correction concept transfers to stills.
|
||||
if (fulls.empty())
|
||||
return;
|
||||
float pxmin = std::numeric_limits<float>::infinity(), pxmax = -pxmin, pymin = pxmin, pymax = -pxmin;
|
||||
for (const Obs &o : fulls) {
|
||||
if (!std::isfinite(o.px) || !std::isfinite(o.py)) continue;
|
||||
pxmin = std::min(pxmin, o.px); pxmax = std::max(pxmax, o.px);
|
||||
pymin = std::min(pymin, o.py); pymax = std::max(pymax, o.py);
|
||||
}
|
||||
if (!(pxmax > pxmin) || !(pymax > pymin))
|
||||
return;
|
||||
constexpr int NB = 16;
|
||||
const float sx = NB / (pxmax - pxmin), sy = NB / (pymax - pymin);
|
||||
std::vector<int32_t> cell(fulls.size(), -1);
|
||||
for (size_t i = 0; i < fulls.size(); ++i) {
|
||||
const Obs &o = fulls[i];
|
||||
if (!std::isfinite(o.px) || !std::isfinite(o.py)) continue;
|
||||
const int ix = std::clamp(static_cast<int>((o.px - pxmin) * sx), 0, NB - 1);
|
||||
const int iy = std::clamp(static_cast<int>((o.py - pymin) * sy), 0, NB - 1);
|
||||
cell[i] = ix * NB + iy;
|
||||
}
|
||||
ApplyCellSurface(cell, NB * NB, n_iter, n_groups, "Modulation (detector-frame 16x16)");
|
||||
}
|
||||
|
||||
void RotationScaleMerge::ApplyCellSurface(const std::vector<int32_t> &cell, int ncell, int n_iter,
|
||||
int n_groups, const char *name) {
|
||||
auto usable = [&](const Obs &o) {
|
||||
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && o.partiality >= min_partiality;
|
||||
};
|
||||
// Fit the per-cell absorption factor over the subset {frame&1 == parity} (parity < 0 = all fulls),
|
||||
// n_iter alternating rounds against that subset's own reference (Tikhonov pull to 1, gauge-fixed to a
|
||||
// den-weighted geometric mean of 1 so it never drifts the overall scale). Returns the accumulated
|
||||
// per-cell factor A[cell].
|
||||
// Fit the per-cell factor over the subset {frame&1 == parity} (parity < 0 = all fulls), n_iter
|
||||
// alternating rounds against that subset's own reference (Tikhonov pull to 1, gauge-fixed to a
|
||||
// den-weighted geometric mean of 1 so it never drifts the overall scale). Returns the per-cell factor.
|
||||
auto fit_surface = [&](int parity) -> std::vector<double> {
|
||||
std::vector<double> A(ncell, 1.0);
|
||||
for (int it = 0; it < n_iter; ++it) {
|
||||
@@ -725,39 +993,40 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
|
||||
const double sc = static_cast<double>(o.sigma) * o.corr * a, w = 1.0 / (sc * sc);
|
||||
sw[o.group] += w; swI[o.group] += w * Is;
|
||||
}
|
||||
double sum = 0.0; size_t n = 0;
|
||||
// Rmeas-like (sigma-INDEPENDENT) agreement of the held-out equivalents: sum|Is - Iref| / sum|Iref|.
|
||||
// Scoring on the studentized deviation instead lets a surface "improve" the held-out chi^2 by
|
||||
// reshaping sigma (via corr) without tightening the actual intensities - which on mis-indexed / bad
|
||||
// data passes cross-validation yet worsens Rmeas. A fractional metric cannot be gamed that way.
|
||||
double num = 0.0, den = 0.0;
|
||||
for (size_t i = 0; i < fulls.size(); ++i) {
|
||||
const Obs &o = fulls[i];
|
||||
if (!usable(o) || cell[i] < 0 || (o.frame & 1) != parity || sw[o.group] <= 0.0) continue;
|
||||
const double a = A[cell[i]], Is = static_cast<double>(o.I) * o.corr * a;
|
||||
const double sc = static_cast<double>(o.sigma) * o.corr * a;
|
||||
const double Iref = swI[o.group] / sw[o.group];
|
||||
if (!std::isfinite(Iref) || !(sc > 0.0)) continue;
|
||||
const double dd = (Is - Iref) / sc;
|
||||
sum += dd * dd; ++n;
|
||||
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
|
||||
num += std::abs(Is - Iref); den += Iref;
|
||||
}
|
||||
return n > 0 ? sum / static_cast<double>(n) : 0.0;
|
||||
return den > 0.0 ? num / den : 0.0;
|
||||
};
|
||||
|
||||
// Cross-validate: fit the surface on even frames and score the held-out odd equivalents (and vice
|
||||
// versa). A real absorption surface generalizes; an over-fit one (few obs / cell) does not. Apply only
|
||||
// if the held-out equivalent chi^2 improves by a clear margin - so the correction can never worsen the
|
||||
// data or act on marginal noise.
|
||||
// versa). A real surface generalizes; an over-fit one (few obs / cell) does not. Apply only if the
|
||||
// held-out equivalent chi^2 improves by a clear margin - so the correction can never worsen the data.
|
||||
const std::vector<double> ident(ncell, 1.0);
|
||||
const std::vector<double> A_even = fit_surface(0), A_odd = fit_surface(1);
|
||||
const double base = score(1, ident) + score(0, ident);
|
||||
const double gain = base - (score(1, A_even) + score(0, A_odd));
|
||||
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
|
||||
logger.Info("Absorption correction: not cross-validated (held-out gain {:.1f}%, skipped)",
|
||||
100.0 * gain / std::max(base, 1e-30));
|
||||
logger.Info("{} correction: not cross-validated (held-out gain {:.1f}%, skipped)",
|
||||
name, 100.0 * gain / std::max(base, 1e-30));
|
||||
return;
|
||||
}
|
||||
const std::vector<double> A = fit_surface(-1);
|
||||
for (size_t i = 0; i < fulls.size(); ++i)
|
||||
if (cell[i] >= 0)
|
||||
fulls[i].corr = static_cast<float>(fulls[i].corr * A[cell[i]]);
|
||||
logger.Info("Absorption correction: goniometer-frame {}x{} surface (cross-validated, held-out gain {:.1f}%)",
|
||||
NB, NB, 100.0 * gain / std::max(base, 1e-30));
|
||||
logger.Info("{} correction: cross-validated, held-out gain {:.1f}%",
|
||||
name, 100.0 * gain / std::max(base, 1e-30));
|
||||
}
|
||||
|
||||
void RotationScaleMerge::ComputeSmoothGWindow(const std::vector<double> &g, int window,
|
||||
@@ -909,7 +1178,7 @@ void RotationScaleMerge::Combine() {
|
||||
if (dump != nullptr)
|
||||
*dump << full.h << ' ' << full.k << ' ' << full.l << ' ' << full.I << ' '
|
||||
<< full.sigma << ' ' << d << ' ' << n_frames_event << ' ' << sum_partiality << ' '
|
||||
<< static_cast<int>(peak_frame) << '\n';
|
||||
<< static_cast<int>(peak_frame) << ' ' << full.px << ' ' << full.py << '\n';
|
||||
}
|
||||
return ev.size();
|
||||
};
|
||||
@@ -922,7 +1191,9 @@ void RotationScaleMerge::Combine() {
|
||||
std::ofstream dump;
|
||||
if (!observation_dump_path.empty()) {
|
||||
dump.open(observation_dump_path);
|
||||
dump << "# h k l I sigma d n_frames captured_fraction\n";
|
||||
// Columns are enough to rescale/merge the combined fulls in an external program: peak_frame
|
||||
// is a per-full batch (ZD) and px/py the predicted detector position (XD/YD).
|
||||
dump << "# h k l I sigma d n_frames captured_fraction peak_frame px py\n";
|
||||
}
|
||||
fulls.reserve(n_run);
|
||||
for (int r = 0; r < n_run; ++r)
|
||||
@@ -1207,9 +1478,51 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
fit_ab(std::move(kept));
|
||||
}
|
||||
}
|
||||
// Asymptotic I/sigma. ISa is by definition the I -> infinity limit of the signal-to-noise, i.e. the
|
||||
// reproducibility of the strongest reflections (Diederichs, Acta Cryst. D66 (2010), 733-740). The
|
||||
// (a, b) fit above spans the whole intensity range, and a mild excess of scatter at intermediate
|
||||
// intensity raises its systematic term `b`, so 1/b understates that limit. Read it instead directly
|
||||
// from the strong equivalents: for each well-measured reflection group the counting-subtracted
|
||||
// fractional scatter of its symmetry mates estimates the systematic term, and the robust median over
|
||||
// strong groups is the asymptote. The I/sigma threshold that selects "strong" is relaxed on weak /
|
||||
// radiation-damaged data that has few strong reflections (rather than fall back to the higher
|
||||
// whole-range b); it falls back only when even the relaxed set is too small. Host-side over the
|
||||
// merged fulls, so CPU and GPU agree.
|
||||
double error_model_b_asymptotic = error_model_b;
|
||||
if (error_model_active) {
|
||||
struct GroupScatter { double sum = 0, sum_sq = 0, sum_var = 0; int n = 0; };
|
||||
std::vector<GroupScatter> gs(n_groups);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable_merge(o)) continue;
|
||||
const double I_corr = static_cast<double>(o.I) * o.corr;
|
||||
const double sigma_corr = static_cast<double>(o.sigma) * o.corr;
|
||||
auto &g = gs[o.group];
|
||||
g.sum += I_corr; g.sum_sq += I_corr * I_corr; g.sum_var += sigma_corr * sigma_corr; ++g.n;
|
||||
}
|
||||
// Per-group counting-subtracted fractional systematic variance, paired with the group's I/sigma.
|
||||
std::vector<std::pair<double, double>> group_scatter; // (systematic b^2, I/sigma)
|
||||
for (const auto &g : gs) {
|
||||
if (g.n < 5) continue;
|
||||
const double mean = g.sum / g.n;
|
||||
const double counting = g.sum_var / g.n;
|
||||
if (mean <= 0.0 || counting <= 0.0) continue;
|
||||
const double variance = (g.sum_sq - g.sum * g.sum / g.n) / (g.n - 1);
|
||||
group_scatter.push_back({std::max(variance - counting, 0.0) / (mean * mean),
|
||||
mean / std::sqrt(counting)});
|
||||
}
|
||||
auto asymptote_above = [&](double snr_min, size_t min_groups) -> double {
|
||||
std::vector<double> b2;
|
||||
for (const auto &[b2_value, snr] : group_scatter)
|
||||
if (snr >= snr_min) b2.push_back(b2_value);
|
||||
return b2.size() >= min_groups ? std::sqrt(median_of(b2)) : 0.0;
|
||||
};
|
||||
double b_asy = asymptote_above(20.0, 100); // tight threshold on data that supports it
|
||||
if (b_asy <= 0.0) b_asy = asymptote_above(10.0, 50); // relaxed for weak / damaged data
|
||||
if (b_asy > 0.0) error_model_b_asymptotic = b_asy;
|
||||
}
|
||||
if (error_model_active)
|
||||
logger.Info("Error model: a={:.3f} b={:.3f} ISa={:.1f} chi2={:.2f}", error_model_a, error_model_b,
|
||||
error_model_b > 0 ? 1.0 / error_model_b : 0.0, error_model_chi2);
|
||||
error_model_b_asymptotic > 0 ? 1.0 / error_model_b_asymptotic : 0.0, error_model_chi2);
|
||||
|
||||
auto corrected_sigma = [&](float I_corr, float sigma_corr, int g) -> float {
|
||||
if (!error_model_active) return sigma_corr;
|
||||
@@ -1266,7 +1579,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
}
|
||||
// ---- Export merged reflections (+ resolution-shell R-free flags). ----
|
||||
Result result;
|
||||
result.isa = error_model_b > 0 ? 1.0 / error_model_b : 0.0;
|
||||
result.isa = error_model_b_asymptotic > 0 ? 1.0 / error_model_b_asymptotic : 0.0;
|
||||
std::vector<double> merged_I(n_groups, NAN);
|
||||
float d_min = std::numeric_limits<float>::max(), d_max = 0.0f;
|
||||
for (int g = 0; g < n_groups; ++g) {
|
||||
@@ -1275,14 +1588,16 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
MergedReflection mr{};
|
||||
mr.h = group_h[g]; mr.k = group_k[g]; mr.l = group_l[g];
|
||||
mr.I = static_cast<float>(a.swI / a.sw);
|
||||
mr.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(a.sw), mr.I, error_model_b);
|
||||
// The systematic floor caps a high-multiplicity merged sigma at the reproducibility that the
|
||||
// strongest reflections actually reach, i.e. the asymptotic term, so merged I/sigma approaches ISa.
|
||||
mr.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(a.sw), mr.I, error_model_b_asymptotic);
|
||||
mr.I_half[0] = mr.I_half[1] = NAN;
|
||||
mr.sigma_half[0] = mr.sigma_half[1] = NAN;
|
||||
mr.d = a.d;
|
||||
if (a.nh[0] + a.nh[1] > 0 && a.swh[0] > 0.0 && a.swh[1] > 0.0) {
|
||||
for (int i = 0; i < 2; ++i) {
|
||||
mr.I_half[i] = static_cast<float>(a.swIh[i] / a.swh[i]);
|
||||
mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(a.swh[i]), mr.I_half[i], error_model_b);
|
||||
mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(a.swh[i]), mr.I_half[i], error_model_b_asymptotic);
|
||||
}
|
||||
}
|
||||
if (!std::isfinite(a.d) || a.d <= 0.0f) continue;
|
||||
@@ -1301,7 +1616,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
result.merged, d_min_limit, resolution_cutoff_method, resolution_cc_target, for_search, logger);
|
||||
|
||||
AssignRfreeFlags(result.merged, x.GetSpaceGroupNumber().value_or(1), rfree_fraction);
|
||||
ApplyFrenchWilson(result.merged, x.GetSpaceGroupNumber().value_or(1));
|
||||
// French-Wilson (F, and F(+)/F(-) from the anomalous split) is deferred until after the anomalous
|
||||
// accumulator below has attached I(+)/I(-), so the two hands get their amplitudes in one pass.
|
||||
|
||||
if (reject_count > 0)
|
||||
logger.Info("Merge outlier rejection: dropped {} observations", reject_count);
|
||||
@@ -1399,6 +1715,55 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
rmeas_num_all += factor * r.sum_abs_dev; rmeas_den_all += r.sum_I;
|
||||
}
|
||||
|
||||
// ---- Anomalous split (always, even when the merge is Friedel-averaged): for each acentric
|
||||
// reflection, the inverse-variance I(+)/I(-) from the SAME scaled fulls (so it never touches the
|
||||
// Friedel-merged IMEAN, scaling or error model above). Lets I(+)/I(-) and F(+)/F(-) be written by
|
||||
// default without scaling anomalously; a reflection with only one mate, or a centric, is left
|
||||
// without the split. Skipped for the P1 search pass, which has no use for it. ----
|
||||
struct AnomExport { float Ip = NAN, sIp = NAN, Im = NAN, sIm = NAN; };
|
||||
std::unordered_map<uint64_t, AnomExport> anom_export;
|
||||
// SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> accumulated on the standard report shells (num=sum|dI|,
|
||||
// den=sum sigma(dI)); mmCIF pdbx_absDiff_over_sigma_anomalous. NaN'd below when there is no split.
|
||||
std::vector<double> sig_num(n_shells, 0.0), sig_den(n_shells, 0.0);
|
||||
double sig_num_all = 0.0, sig_den_all = 0.0;
|
||||
size_t sig_n = 0;
|
||||
if (!for_search) {
|
||||
const int sg_num = x.GetSpaceGroupNumber().value_or(1);
|
||||
const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, sg_num);
|
||||
const gemmi::GroupOps gops = gemmi::find_spacegroup_by_number(sg_num)->operations();
|
||||
struct AnomAcc { double swI[2] = {}; double sw[2] = {}; int32_t h = 0, k = 0, l = 0; float d = NAN; }; // [hand] 0=I(+) 1=I(-)
|
||||
std::unordered_map<uint64_t, AnomAcc> anom;
|
||||
anom.reserve(result.merged.size() * 2 + 1);
|
||||
for (const auto &o : fulls) {
|
||||
if (!usable_merge(o)) continue;
|
||||
if (rejected_obs[&o - fulls.data()]) continue; // outlier-rejected in the merge (CPU path)
|
||||
const HKLKey ak = anom_keygen(o.h, o.k, o.l);
|
||||
const int hand = ak.plus ? 0 : 1;
|
||||
const float I_corr = o.I * o.corr;
|
||||
const float sigma_corr = corrected_sigma(I_corr, o.sigma * o.corr, o.group);
|
||||
if (!(sigma_corr > 0.0f) || !std::isfinite(sigma_corr)) continue;
|
||||
const double w = 1.0 / (static_cast<double>(sigma_corr) * sigma_corr);
|
||||
AnomAcc &a = anom[HKLKey{ak.h, ak.k, ak.l, true}.pack()];
|
||||
a.h = ak.h; a.k = ak.k; a.l = ak.l; a.d = o.d;
|
||||
a.swI[hand] += w * static_cast<double>(I_corr); a.sw[hand] += w;
|
||||
}
|
||||
for (const auto &[fkey, a] : anom) {
|
||||
// Centrics have I(+)=I(-) by symmetry; leave them without an anomalous split.
|
||||
if (gops.is_reflection_centric(gemmi::Op::Miller{a.h, a.k, a.l})) continue;
|
||||
AnomExport ex;
|
||||
if (a.sw[0] > 0.0) { ex.Ip = static_cast<float>(a.swI[0] / a.sw[0]); ex.sIp = static_cast<float>(1.0 / std::sqrt(a.sw[0])); }
|
||||
if (a.sw[1] > 0.0) { ex.Im = static_cast<float>(a.swI[1] / a.sw[1]); ex.sIm = static_cast<float>(1.0 / std::sqrt(a.sw[1])); }
|
||||
if (std::isfinite(ex.Ip) || std::isfinite(ex.Im)) anom_export[fkey] = ex;
|
||||
if (a.sw[0] > 0.0 && a.sw[1] > 0.0 && a.d > 0.0f) {
|
||||
const double abs_di = std::fabs(a.swI[0] / a.sw[0] - a.swI[1] / a.sw[1]);
|
||||
const double sig_di = std::sqrt(1.0 / a.sw[0] + 1.0 / a.sw[1]);
|
||||
const auto shell = shells.GetShell(a.d);
|
||||
if (shell && *shell >= 0 && *shell < n_shells) { sig_num[*shell] += abs_di; sig_den[*shell] += sig_di; }
|
||||
sig_num_all += abs_di; sig_den_all += sig_di; ++sig_n;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
MergeStatistics &out = result.statistics;
|
||||
out.shells.resize(n_shells);
|
||||
for (int s = 0; s < n_shells; ++s) {
|
||||
@@ -1413,6 +1778,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
ss.cc_half = sa[s].cc_half.GetCC();
|
||||
ss.cc_ref = NAN;
|
||||
ss.r_meas = rmeas_den[s] > 0.0 ? rmeas_num[s] / rmeas_den[s] : NAN;
|
||||
ss.abs_diff_over_sigma_anomalous = sig_den[s] > 0.0 ? sig_num[s] / sig_den[s] : NAN;
|
||||
}
|
||||
auto &overall = out.overall;
|
||||
overall.d_min = sd_min; overall.d_max = sd_max;
|
||||
@@ -1427,6 +1793,33 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
|
||||
overall.cc_half = cc_half_overall.GetCC();
|
||||
overall.cc_ref = NAN;
|
||||
overall.r_meas = rmeas_den_all > 0.0 ? rmeas_num_all / rmeas_den_all : NAN;
|
||||
overall.abs_diff_over_sigma_anomalous = sig_den_all > 0.0 ? sig_num_all / sig_den_all : NAN;
|
||||
if (std::isfinite(overall.abs_diff_over_sigma_anomalous))
|
||||
logger.Info("Anomalous signal SigAno = {:.2f} ({} acentric pairs)",
|
||||
overall.abs_diff_over_sigma_anomalous, sig_n);
|
||||
|
||||
// Radiation-damage monitor (measured before any correction by MeasureRadiationDamageB): carry the
|
||||
// first->last relative-B change and the per-batch curve into the reported statistics / mmCIF.
|
||||
out.radiation_damage_delta_b = rad_damage_delta_b;
|
||||
out.radiation_damage_b_batch = rad_damage_b_batch;
|
||||
out.radiation_damage_batch_deg = rad_damage_batch_deg;
|
||||
|
||||
// Attach the per-reflection anomalous split so the writer can emit I(+)/I(-) by default (each merged
|
||||
// reflection maps to its Friedel-ASU key; in an anomalous merge both mates map to the same key).
|
||||
if (!anom_export.empty()) {
|
||||
const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, x.GetSpaceGroupNumber().value_or(1));
|
||||
for (auto &r : result.merged) {
|
||||
const HKLKey ak = anom_keygen(r.h, r.k, r.l);
|
||||
const auto it = anom_export.find(HKLKey{ak.h, ak.k, ak.l, true}.pack());
|
||||
if (it == anom_export.end()) continue;
|
||||
r.I_plus = it->second.Ip; r.sigma_plus = it->second.sIp;
|
||||
r.I_minus = it->second.Im; r.sigma_minus = it->second.sIm;
|
||||
}
|
||||
}
|
||||
|
||||
// French-Wilson amplitudes for IMEAN and (now that they are attached) each Bijvoet hand.
|
||||
ApplyFrenchWilson(result.merged, x.GetSpaceGroupNumber().value_or(1));
|
||||
|
||||
logger.Info("Merge complete ({} unique reflections)", result.merged.size());
|
||||
return result;
|
||||
}
|
||||
@@ -1600,11 +1993,20 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
|
||||
// the merged reference. Cheap host loops over the downloaded fulls; applied only on the final
|
||||
// in-symmetry merge, never the P1 space-group search pass (corrections there add risk and can
|
||||
// perturb the symmetry determination). ---
|
||||
[[maybe_unused]] const bool corrections = !for_search && (refine_decay_b || absorption_iter > 0);
|
||||
[[maybe_unused]] const bool corrections =
|
||||
!for_search && (refine_decay_b || absorption_iter > 0 || modulation_iter > 0 || relative_b_deg > 0.0);
|
||||
// Radiation-damage monitor: measure the per-batch relative-B on the scaled fulls BEFORE any correction
|
||||
// (report-only; captures the full damage signature, not a residual). Skipped on the P1 search pass.
|
||||
if (!for_search)
|
||||
MeasureRadiationDamageB(n_groups);
|
||||
if (!for_search && refine_decay_b)
|
||||
RefineDecay(n_groups);
|
||||
if (!for_search && relative_b_deg > 0.0)
|
||||
RefineRelativeB(n_groups); // per-batch relative-B on top of the single decay slope
|
||||
if (!for_search && absorption_iter > 0)
|
||||
RefineAbsorption(absorption_iter, n_groups);
|
||||
if (!for_search && modulation_iter > 0)
|
||||
RefineModulation(modulation_iter, n_groups);
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
// The corrections mutate the host fulls' corr; when the merge runs on the resident (GPU) fulls, push
|
||||
// the corrected corr back to the device so the merge reads it.
|
||||
|
||||
Reference in New Issue
Block a user