v1.0.0-rc.160 (#70)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #70.
This commit is contained in:
2026-07-19 09:39:28 +02:00
committed by leonarski_f
parent dd0bffb283
commit 67dca388bd
246 changed files with 5275 additions and 5155 deletions
+442 -40
View File
@@ -11,6 +11,7 @@
#include <future>
#include <limits>
#include <random>
#include <unordered_map>
#include <gemmi/reciproc.hpp>
#include <gemmi/symmetry.hpp>
@@ -171,6 +172,8 @@ RotationScaleMerge::RotationScaleMerge(const DiffractionExperiment &experiment,
// so a no-op when their systematic is absent). Decoupled from the stills-only -B / RefineB flag.
refine_decay_b = s.GetCorrectionSurfaces();
absorption_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
modulation_iter = s.GetCorrectionSurfaces() ? s.GetAbsorptionIter() : 0;
relative_b_deg = s.GetRelativeBDegrees(); // opt-in per-batch relative-B (0 = off)
scaling_iter = std::max(1, scaling_iterations);
resolution_cutoff_method = s.GetResolutionCutoff();
resolution_cc_target = s.GetResolutionCCTarget();
@@ -553,9 +556,10 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
// noise / aggregate structure (not real per-reflection decay) would amplify into a high-res
// mis-correction and worsen the symmetry-equivalent scatter that sets ISa. An opt-in correction must
// never add noise where its systematic is absent (a low-dose reference set), so the slope is
// CROSS-VALIDATED: fit on even-frame fulls and score the held-out odd-frame equivalent chi^2, and vice
// versa. Real decay generalizes across the split (both held-out scores improve); a noise slope does
// not. Apply the full-data slope only if the mean held-out chi^2 strictly improves; else it is a no-op.
// CROSS-VALIDATED: fit on even-frame fulls and score the held-out odd-frame equivalent agreement (a
// sigma-independent Rmeas-like metric), and vice versa. Real decay generalizes across the split (both
// held-out scores improve); a noise slope does not. Apply the full-data slope only if the mean held-out
// agreement strictly improves; else it is a no-op.
if (fulls.empty() || n_frames <= 0)
return;
const double fcenter = 0.5 * n_frames;
@@ -592,9 +596,14 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
// strong damage (slope = 2 dB/dframe, so +-1.0 admits total relative-B up to ~n_frames/2 A^2).
return (var > 0.0) ? std::clamp((Sxy - Sx * Sy / Sw) / var, -1.0, 1.0) : 0.0;
};
// Mean studentized squared deviation over the subset {frame&1 == parity} when `slope` is applied,
// scored against that subset's own reference (no leakage). Lower = tighter equivalents.
auto subset_chi2 = [&](int parity, double slope) -> double {
// Rmeas-like (sigma-INDEPENDENT) disagreement of the equivalents over the subset {frame&1 == parity}
// when `slope` is applied, scored against that subset's own reference (no leakage): sum|Is-Iref|/sum|Iref|.
// Scoring on the studentized deviation instead would let the slope "improve" the held-out chi^2 by
// reshaping sigma - it scales sigma up on late / high-angle frames, down-weighting them - without
// actually tightening the intensities; a fractional metric cannot be gamed that way. This mirrors the
// absorption / modulation gate in ApplyCellSurface (which was switched off studentized chi^2 for the
// same reason); the reference is still the inverse-variance mean.
auto subset_disagreement = [&](int parity, double slope) -> double {
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (o.frame & 1) != parity) continue;
@@ -603,17 +612,15 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
const double w = 1.0 / (sc * sc);
sw[o.group] += w; swI[o.group] += w * Is;
}
double sum = 0.0; size_t n = 0;
double num = 0.0, den = 0.0;
for (const auto &o : fulls) {
if (!usable(o) || (o.frame & 1) != parity || sw[o.group] <= 0.0) continue;
const double Iref = swI[o.group] / sw[o.group];
const double Is = static_cast<double>(o.I) * o.corr * decay_factor(o, slope);
const double sc = static_cast<double>(o.sigma) * o.corr * decay_factor(o, slope);
if (!std::isfinite(Iref) || !(sc > 0.0)) continue;
const double dd = (Is - Iref) / sc;
sum += dd * dd; ++n;
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
num += std::abs(Is - Iref); den += Iref;
}
return n > 0 ? sum / static_cast<double>(n) : 0.0;
return den > 0.0 ? num / den : 0.0;
};
// A meaningful radiation-damage relative-B accrues several A^2 across a dataset. Below a physical floor
@@ -630,8 +637,8 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
}
// And require the slope to cross-validate by a clear margin: fit on even frames, score the held-out
// odd equivalents (and vice versa). Real damage generalizes; a fluke does not.
const double base = subset_chi2(1, 0.0) + subset_chi2(0, 0.0);
const double gain = base - (subset_chi2(1, fit_slope(0)) + subset_chi2(0, fit_slope(1)));
const double base = subset_disagreement(1, 0.0) + subset_disagreement(0, 0.0);
const double gain = base - (subset_disagreement(1, fit_slope(0)) + subset_disagreement(0, fit_slope(1)));
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
logger.Info("Decay correction: not cross-validated (dB = {:.2f} A^2, held-out gain {:.1f}%, skipped)",
total_delta_B, 100.0 * gain / std::max(base, 1e-30));
@@ -644,6 +651,236 @@ void RotationScaleMerge::RefineDecay(int n_groups) {
0.5 * slope * n_frames, 0.5 * slope);
}
std::vector<double> RotationScaleMerge::SolveCurvatureSmoothedB(const std::vector<double> &num,
const std::vector<double> &den) const {
// Solve for a SMOOTH per-batch relative-B from the per-batch normal-equation data (num_c = sum w s^2 y,
// den_c = sum w s^4). A per-batch regression alone (b_c = num_c/den_c) over-fits: a genuine relative-B is
// a low-curvature function of dose / orientation, but an unconstrained fit chases resolution-correlated
// noise into a rough, oscillating curve. So minimise data-fidelity PLUS a second-difference (curvature)
// penalty sum_c den_c (b_c - num_c/den_c)^2 + mu * sum_c (b_{c-1} - 2 b_c + b_{c+1})^2, which leaves a
// straight RAMP untouched but suppresses oscillation, solving the small SPD normal system A b = num,
// A = diag(den + lambda) + mu * D^T D, by Gauss-Seidel (n_batch <= a few tens). Not anchored - callers set
// the gauge (zero-mean for the correction, low-dose reference for the monitor).
constexpr double RELATIVE_B_MAX = 50.0; // A^2, clamp a single batch's relative-B (generous guard)
constexpr double RELATIVE_B_CURVATURE_K = 8.0; // curvature penalty weight (x the median batch leverage)
const int n_batch = static_cast<int>(num.size());
std::vector<double> dsorted(den);
std::sort(dsorted.begin(), dsorted.end());
const double med_den = dsorted.empty() ? 0.0 : dsorted[dsorted.size() / 2];
const double lambda = 0.1 * med_den; // ridge ~ 10% of the median batch leverage
const double mu = RELATIVE_B_CURVATURE_K * med_den; // curvature penalty
std::vector<double> A(static_cast<size_t>(n_batch) * n_batch, 0.0);
auto Aat = [&](int i, int j) -> double & { return A[static_cast<size_t>(i) * n_batch + j]; };
for (int c = 0; c < n_batch; ++c) Aat(c, c) = den[c] + lambda;
constexpr int st[3] = {-1, 0, 1};
constexpr double sv[3] = {1.0, -2.0, 1.0}; // second-difference stencil
for (int c = 1; c < n_batch - 1; ++c)
for (int a = 0; a < 3; ++a)
for (int b2 = 0; b2 < 3; ++b2)
Aat(c + st[a], c + st[b2]) += mu * sv[a] * sv[b2];
std::vector<double> b(n_batch, 0.0);
for (int it = 0; it < 100; ++it)
for (int c = 0; c < n_batch; ++c) {
if (!(Aat(c, c) > 0.0)) continue;
double s = num[c];
for (int j = 0; j < n_batch; ++j)
if (j != c) s -= Aat(c, j) * b[j];
b[c] = std::clamp(s / Aat(c, c), -RELATIVE_B_MAX, RELATIVE_B_MAX);
}
return b;
}
std::vector<double> RotationScaleMerge::FitRelativeBCurve(int n_groups, int n_batch, int frames_per_batch,
int gparity) const {
// Fit one relative-B per batch over the ASU-group subset {group&1 == gparity} (gparity < 0 = all fulls),
// against an inverse-variance reference built from that same subset (no leakage). Per batch the model is
// y = ln(I_ref / I_obs) = b * s^2 through the origin (the resolution-flat intercept is the per-frame G,
// already applied), so the data term is b_c = sum(w s^2 y) / sum(w s^2 s^2); SolveCurvatureSmoothedB then
// smooths it. Finally subtract the leverage-weighted mean: a constant B is a global Wilson-B degenerate
// with overall scale, and cannot change equivalent agreement (within one ASU group every equivalent sits
// at the same s^2, so only b VARYING between batches moves the merge). Drives the per-batch correction.
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
&& o.partiality >= min_partiality;
};
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity)) continue;
const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
sw[o.group] += w; swI[o.group] += w * static_cast<double>(o.I) * o.corr;
}
std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (gparity >= 0 && (o.group & 1) != gparity) || sw[o.group] <= 0.0) continue;
const double Iref = swI[o.group] / sw[o.group];
const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
const double w = (Is / sc) * (Is / sc);
const double s2 = s2_of(o.d), y = std::log(Iref / Is);
num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
}
std::vector<double> b = SolveCurvatureSmoothedB(num, den);
double dw = 0.0, dbw = 0.0;
for (int c = 0; c < n_batch; ++c) { dw += den[c]; dbw += den[c] * b[c]; }
const double bmean = dw > 0.0 ? dbw / dw : 0.0;
for (auto &v : b) v -= bmean;
return b;
}
void RotationScaleMerge::MeasureRadiationDamageB(int n_groups) {
// Radiation-damage MONITOR (report-only): measure the per-batch relative Debye-Waller B on the scaled
// fulls BEFORE any correction, so it captures how the high-resolution scattering power fades with
// accumulated dose. Crucially the reference is the PRISTINE LOW-DOSE crystal (first ~10% of frames), not
// the whole-run mean: a group measured both early and late then contributes ln(I_early / I_late) vs s^2,
// whose slope is the ADDED B - an absolute, correctly-signed damage read (positive = high-res fades).
// A whole-run-mean reference inverts on heavily damaged data (the surviving late high-res reflections
// look strong against a mean dominated by the frames where they had already faded). Stores the first->last
// relative-B change (the headline "B-factor difference") and the per-batch curve; never touches corr.
rad_damage_delta_b = std::numeric_limits<double>::quiet_NaN();
rad_damage_b_batch.clear();
rad_damage_batch_deg = 0.0;
if (fulls.empty() || n_frames <= 0)
return;
const auto gon = x.GetGoniometer();
const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
if (!(osc_deg > 1e-6))
return;
constexpr double MONITOR_BATCH_DEG = 10.0; // rotation width per batch for the damage curve
const int frames_per_batch = std::max(1, static_cast<int>(std::lround(MONITOR_BATCH_DEG / osc_deg)));
const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
if (n_batch < 2) // a single batch is a global Wilson-B (degenerate with scale) - nothing to monitor
return;
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
&& o.partiality >= min_partiality;
};
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
// Low-dose reference: inverse-variance group means over the first ~10% of frames (>= 1 batch).
const int ref_frames = std::max(frames_per_batch, n_frames / 10);
std::vector<double> sw0(n_groups, 0.0), swI0(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || o.frame >= ref_frames) continue;
const double sc = static_cast<double>(o.sigma) * o.corr, w = 1.0 / (sc * sc);
sw0[o.group] += w; swI0[o.group] += w * static_cast<double>(o.I) * o.corr;
}
// Per batch: slope of ln(I_ref0 / I_obs) vs s^2, over groups also seen in the low-dose reference.
std::vector<double> num(n_batch, 0.0), den(n_batch, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || sw0[o.group] <= 0.0) continue;
const double Iref = swI0[o.group] / sw0[o.group];
const double Is = static_cast<double>(o.I) * o.corr, sc = static_cast<double>(o.sigma) * o.corr;
if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0)) continue;
const double w = (Is / sc) * (Is / sc);
const double s2 = s2_of(o.d), y = std::log(Iref / Is);
num[batch_of(o)] += w * s2 * y; den[batch_of(o)] += w * s2 * s2;
}
std::vector<double> b = SolveCurvatureSmoothedB(num, den);
// Anchor so the reference batches read ~0 (b = added B relative to the low-dose start).
const int n_ref_batch = std::max(1, ref_frames / frames_per_batch);
double ref_mean = 0.0;
for (int c = 0; c < n_ref_batch && c < n_batch; ++c) ref_mean += b[c];
ref_mean /= std::min(n_ref_batch, n_batch);
for (auto &v : b) v -= ref_mean;
// Headline: the first->last relative-B change from a least-squares linear trend of the (smoothed) curve -
// robust to a single noisy end batch, unlike raw endpoints. Positive = high-res fades with dose = damage.
double sx = 0, sy = 0, sxx = 0, sxy = 0;
for (int c = 0; c < n_batch; ++c) { sx += c; sy += b[c]; sxx += double(c) * c; sxy += double(c) * b[c]; }
const double det = n_batch * sxx - sx * sx;
const double slope = det > 0.0 ? (n_batch * sxy - sx * sy) / det : 0.0;
rad_damage_delta_b = slope * (n_batch - 1);
rad_damage_batch_deg = MONITOR_BATCH_DEG;
rad_damage_b_batch.resize(n_batch);
for (int c = 0; c < n_batch; ++c) rad_damage_b_batch[c] = static_cast<float>(b[c]);
}
void RotationScaleMerge::RefineRelativeB(int n_groups) {
// RefineDecay removes the AVERAGE radiation-damage falloff as a single global relative-B slope, but the
// relative scattering power drifts NON-monotonically across a run (absorption path as the crystal
// rotates, slippage, dose bursts, a partly-shadowed wedge). XDS/AIMLESS model this with a per-BATCH
// relative-B on top of the per-batch scale. Here, on the fulls after RefineDecay: fit the smoothed
// per-batch relative-B (FitRelativeBCurve) and apply exp(b_batch * s^2). Same anti-overfit posture as
// RefineDecay - a physical spread floor and a cross-validated held-out gain - but the CV split is by ASU
// GROUP parity, not frame parity: a per-batch parameter owns whole frames, so it cannot be scored on a
// held-out FRAME; splitting the equivalents instead tests whether a batch's B generalises to reflections
// it was not fit on, which is exactly the over-fit risk of a many-parameter model.
if (fulls.empty() || n_frames <= 0 || relative_b_deg <= 0.0)
return;
const auto gon = x.GetGoniometer();
const double osc_deg = gon ? std::fabs(gon->GetIncrement_deg()) : 0.0;
if (!(osc_deg > 1e-6))
return;
const int frames_per_batch = std::max(1, static_cast<int>(std::lround(relative_b_deg / osc_deg)));
const int n_batch = std::max(1, (n_frames + frames_per_batch - 1) / frames_per_batch);
if (n_batch < 2) // a single batch is a global Wilson-B (degenerate with overall scale) - nothing relative
return;
auto s2_of = [](float d) { return d > 0.0f ? 1.0 / (4.0 * static_cast<double>(d) * d) : 0.0; };
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && (o.d > 0.0f)
&& o.partiality >= min_partiality;
};
auto batch_of = [&](const Obs &o) { return std::min(n_batch - 1, o.frame / frames_per_batch); };
auto b_factor = [&](const Obs &o, const std::vector<double> &b) {
return std::exp(b[batch_of(o)] * s2_of(o.d));
};
// Held-out disagreement over the group subset {group&1 == gparity} with `b` applied: the same
// sigma-INDEPENDENT Rmeas-like metric as RefineDecay (sum|Is-Iref|/sum|Iref|), so a relative-B cannot
// "win" by reshaping sigma. Reference built from the scored subset (no leakage).
auto subset_disagreement = [&](int gparity, const std::vector<double> &b) -> double {
std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
for (const auto &o : fulls) {
if (!usable(o) || (o.group & 1) != gparity) continue;
const double f = b_factor(o, b);
const double Is = static_cast<double>(o.I) * o.corr * f, sc = static_cast<double>(o.sigma) * o.corr * f;
const double w = 1.0 / (sc * sc);
sw[o.group] += w; swI[o.group] += w * Is;
}
double num = 0.0, den = 0.0;
for (const auto &o : fulls) {
if (!usable(o) || (o.group & 1) != gparity || sw[o.group] <= 0.0) continue;
const double Iref = swI[o.group] / sw[o.group];
const double Is = static_cast<double>(o.I) * o.corr * b_factor(o, b);
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
num += std::abs(Is - Iref); den += Iref;
}
return den > 0.0 ? num / den : 0.0;
};
// A relative-B whose peak-to-peak is below a couple A^2 is negligible; "correcting" it only spreads
// equivalents (same s^2, different batch) into extra scatter - net harmful, exactly like RefineDecay's
// sub-floor slope. (RELATIVE_B_MIN_SPREAD ~ DECAY_MIN_DELTA_B.)
constexpr double RELATIVE_B_MIN_SPREAD = 2.0; // A^2, minimum peak-to-peak relative-B to engage
const std::vector<double> b_all = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, -1);
const double bmin = *std::min_element(b_all.begin(), b_all.end());
const double bmax = *std::max_element(b_all.begin(), b_all.end());
if (bmax - bmin < RELATIVE_B_MIN_SPREAD) {
logger.Info("Relative-B: negligible variation ({} batches, peak-to-peak {:.2f} A^2 < {:.1f}, skipped)",
n_batch, bmax - bmin, RELATIVE_B_MIN_SPREAD);
return;
}
// Cross-validate: fit on even ASU groups, score the held-out odd equivalents (and vice-versa). Real
// per-batch structure generalises across the equivalent split; over-fit per-batch noise does not.
const std::vector<double> zero(n_batch, 0.0);
const double base = subset_disagreement(0, zero) + subset_disagreement(1, zero);
const std::vector<double> b_even = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 0);
const std::vector<double> b_odd = FitRelativeBCurve(n_groups, n_batch, frames_per_batch, 1);
const double gain = base - (subset_disagreement(1, b_even) + subset_disagreement(0, b_odd));
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
logger.Info("Relative-B: not cross-validated ({} batches, peak-to-peak {:.2f} A^2, held-out gain {:.1f}%, skipped)",
n_batch, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
return;
}
for (auto &o : fulls)
if (usable(o))
o.corr = static_cast<float>(o.corr * b_factor(o, b_all));
logger.Info("Relative-B: {} batches of ~{:.1f} deg, peak-to-peak {:.2f} A^2, held-out gain {:.1f}% (cross-validated)",
n_batch, relative_b_deg, bmax - bmin, 100.0 * gain / std::max(base, 1e-30));
}
void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
// Absorption / path-length: a smooth multiplicative factor over the diffracted-beam direction in the
// goniometer (crystal) frame. Each full's lab diffracted direction (from its predicted detector
@@ -661,7 +898,6 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
return;
constexpr int NB = 8;
const int ncell = NB * NB;
std::vector<int32_t> cell(fulls.size(), -1);
for (size_t i = 0; i < fulls.size(); ++i) {
const Obs &o = fulls[i];
@@ -673,14 +909,46 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
const int iy = std::clamp(static_cast<int>((u.y + 1.0f) * 0.5f * NB), 0, NB - 1);
cell[i] = ix * NB + iy;
}
ApplyCellSurface(cell, NB * NB, n_iter, n_groups, "Absorption (goniometer-frame 8x8)");
}
void RotationScaleMerge::RefineModulation(int n_iter, int n_groups) {
// Detector-plane modulation (flat-field): a smooth multiplicative factor over the position where a
// reflection lands on the detector (predicted px, py). It absorbs detector-response and geometric
// systematics that vary across the detector plane; symmetry-equivalents of one reflection land at
// different positions as the crystal rotates, over-determining the surface. Because it is a property of
// the detector + beam (not the rotation), the same correction concept transfers to stills.
if (fulls.empty())
return;
float pxmin = std::numeric_limits<float>::infinity(), pxmax = -pxmin, pymin = pxmin, pymax = -pxmin;
for (const Obs &o : fulls) {
if (!std::isfinite(o.px) || !std::isfinite(o.py)) continue;
pxmin = std::min(pxmin, o.px); pxmax = std::max(pxmax, o.px);
pymin = std::min(pymin, o.py); pymax = std::max(pymax, o.py);
}
if (!(pxmax > pxmin) || !(pymax > pymin))
return;
constexpr int NB = 16;
const float sx = NB / (pxmax - pxmin), sy = NB / (pymax - pymin);
std::vector<int32_t> cell(fulls.size(), -1);
for (size_t i = 0; i < fulls.size(); ++i) {
const Obs &o = fulls[i];
if (!std::isfinite(o.px) || !std::isfinite(o.py)) continue;
const int ix = std::clamp(static_cast<int>((o.px - pxmin) * sx), 0, NB - 1);
const int iy = std::clamp(static_cast<int>((o.py - pymin) * sy), 0, NB - 1);
cell[i] = ix * NB + iy;
}
ApplyCellSurface(cell, NB * NB, n_iter, n_groups, "Modulation (detector-frame 16x16)");
}
void RotationScaleMerge::ApplyCellSurface(const std::vector<int32_t> &cell, int ncell, int n_iter,
int n_groups, const char *name) {
auto usable = [&](const Obs &o) {
return o.group >= 0 && o.corr > 0.0f && std::isfinite(o.corr) && o.partiality >= min_partiality;
};
// Fit the per-cell absorption factor over the subset {frame&1 == parity} (parity < 0 = all fulls),
// n_iter alternating rounds against that subset's own reference (Tikhonov pull to 1, gauge-fixed to a
// den-weighted geometric mean of 1 so it never drifts the overall scale). Returns the accumulated
// per-cell factor A[cell].
// Fit the per-cell factor over the subset {frame&1 == parity} (parity < 0 = all fulls), n_iter
// alternating rounds against that subset's own reference (Tikhonov pull to 1, gauge-fixed to a
// den-weighted geometric mean of 1 so it never drifts the overall scale). Returns the per-cell factor.
auto fit_surface = [&](int parity) -> std::vector<double> {
std::vector<double> A(ncell, 1.0);
for (int it = 0; it < n_iter; ++it) {
@@ -725,39 +993,40 @@ void RotationScaleMerge::RefineAbsorption(int n_iter, int n_groups) {
const double sc = static_cast<double>(o.sigma) * o.corr * a, w = 1.0 / (sc * sc);
sw[o.group] += w; swI[o.group] += w * Is;
}
double sum = 0.0; size_t n = 0;
// Rmeas-like (sigma-INDEPENDENT) agreement of the held-out equivalents: sum|Is - Iref| / sum|Iref|.
// Scoring on the studentized deviation instead lets a surface "improve" the held-out chi^2 by
// reshaping sigma (via corr) without tightening the actual intensities - which on mis-indexed / bad
// data passes cross-validation yet worsens Rmeas. A fractional metric cannot be gamed that way.
double num = 0.0, den = 0.0;
for (size_t i = 0; i < fulls.size(); ++i) {
const Obs &o = fulls[i];
if (!usable(o) || cell[i] < 0 || (o.frame & 1) != parity || sw[o.group] <= 0.0) continue;
const double a = A[cell[i]], Is = static_cast<double>(o.I) * o.corr * a;
const double sc = static_cast<double>(o.sigma) * o.corr * a;
const double Iref = swI[o.group] / sw[o.group];
if (!std::isfinite(Iref) || !(sc > 0.0)) continue;
const double dd = (Is - Iref) / sc;
sum += dd * dd; ++n;
if (!std::isfinite(Iref) || Iref <= 0.0) continue;
num += std::abs(Is - Iref); den += Iref;
}
return n > 0 ? sum / static_cast<double>(n) : 0.0;
return den > 0.0 ? num / den : 0.0;
};
// Cross-validate: fit the surface on even frames and score the held-out odd equivalents (and vice
// versa). A real absorption surface generalizes; an over-fit one (few obs / cell) does not. Apply only
// if the held-out equivalent chi^2 improves by a clear margin - so the correction can never worsen the
// data or act on marginal noise.
// versa). A real surface generalizes; an over-fit one (few obs / cell) does not. Apply only if the
// held-out equivalent chi^2 improves by a clear margin - so the correction can never worsen the data.
const std::vector<double> ident(ncell, 1.0);
const std::vector<double> A_even = fit_surface(0), A_odd = fit_surface(1);
const double base = score(1, ident) + score(0, ident);
const double gain = base - (score(1, A_even) + score(0, A_odd));
if (!(gain > CV_MIN_RELATIVE_GAIN * base)) {
logger.Info("Absorption correction: not cross-validated (held-out gain {:.1f}%, skipped)",
100.0 * gain / std::max(base, 1e-30));
logger.Info("{} correction: not cross-validated (held-out gain {:.1f}%, skipped)",
name, 100.0 * gain / std::max(base, 1e-30));
return;
}
const std::vector<double> A = fit_surface(-1);
for (size_t i = 0; i < fulls.size(); ++i)
if (cell[i] >= 0)
fulls[i].corr = static_cast<float>(fulls[i].corr * A[cell[i]]);
logger.Info("Absorption correction: goniometer-frame {}x{} surface (cross-validated, held-out gain {:.1f}%)",
NB, NB, 100.0 * gain / std::max(base, 1e-30));
logger.Info("{} correction: cross-validated, held-out gain {:.1f}%",
name, 100.0 * gain / std::max(base, 1e-30));
}
void RotationScaleMerge::ComputeSmoothGWindow(const std::vector<double> &g, int window,
@@ -909,7 +1178,7 @@ void RotationScaleMerge::Combine() {
if (dump != nullptr)
*dump << full.h << ' ' << full.k << ' ' << full.l << ' ' << full.I << ' '
<< full.sigma << ' ' << d << ' ' << n_frames_event << ' ' << sum_partiality << ' '
<< static_cast<int>(peak_frame) << '\n';
<< static_cast<int>(peak_frame) << ' ' << full.px << ' ' << full.py << '\n';
}
return ev.size();
};
@@ -922,7 +1191,9 @@ void RotationScaleMerge::Combine() {
std::ofstream dump;
if (!observation_dump_path.empty()) {
dump.open(observation_dump_path);
dump << "# h k l I sigma d n_frames captured_fraction\n";
// Columns are enough to rescale/merge the combined fulls in an external program: peak_frame
// is a per-full batch (ZD) and px/py the predicted detector position (XD/YD).
dump << "# h k l I sigma d n_frames captured_fraction peak_frame px py\n";
}
fulls.reserve(n_run);
for (int r = 0; r < n_run; ++r)
@@ -1207,9 +1478,51 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
fit_ab(std::move(kept));
}
}
// Asymptotic I/sigma. ISa is by definition the I -> infinity limit of the signal-to-noise, i.e. the
// reproducibility of the strongest reflections (Diederichs, Acta Cryst. D66 (2010), 733-740). The
// (a, b) fit above spans the whole intensity range, and a mild excess of scatter at intermediate
// intensity raises its systematic term `b`, so 1/b understates that limit. Read it instead directly
// from the strong equivalents: for each well-measured reflection group the counting-subtracted
// fractional scatter of its symmetry mates estimates the systematic term, and the robust median over
// strong groups is the asymptote. The I/sigma threshold that selects "strong" is relaxed on weak /
// radiation-damaged data that has few strong reflections (rather than fall back to the higher
// whole-range b); it falls back only when even the relaxed set is too small. Host-side over the
// merged fulls, so CPU and GPU agree.
double error_model_b_asymptotic = error_model_b;
if (error_model_active) {
struct GroupScatter { double sum = 0, sum_sq = 0, sum_var = 0; int n = 0; };
std::vector<GroupScatter> gs(n_groups);
for (const auto &o : fulls) {
if (!usable_merge(o)) continue;
const double I_corr = static_cast<double>(o.I) * o.corr;
const double sigma_corr = static_cast<double>(o.sigma) * o.corr;
auto &g = gs[o.group];
g.sum += I_corr; g.sum_sq += I_corr * I_corr; g.sum_var += sigma_corr * sigma_corr; ++g.n;
}
// Per-group counting-subtracted fractional systematic variance, paired with the group's I/sigma.
std::vector<std::pair<double, double>> group_scatter; // (systematic b^2, I/sigma)
for (const auto &g : gs) {
if (g.n < 5) continue;
const double mean = g.sum / g.n;
const double counting = g.sum_var / g.n;
if (mean <= 0.0 || counting <= 0.0) continue;
const double variance = (g.sum_sq - g.sum * g.sum / g.n) / (g.n - 1);
group_scatter.push_back({std::max(variance - counting, 0.0) / (mean * mean),
mean / std::sqrt(counting)});
}
auto asymptote_above = [&](double snr_min, size_t min_groups) -> double {
std::vector<double> b2;
for (const auto &[b2_value, snr] : group_scatter)
if (snr >= snr_min) b2.push_back(b2_value);
return b2.size() >= min_groups ? std::sqrt(median_of(b2)) : 0.0;
};
double b_asy = asymptote_above(20.0, 100); // tight threshold on data that supports it
if (b_asy <= 0.0) b_asy = asymptote_above(10.0, 50); // relaxed for weak / damaged data
if (b_asy > 0.0) error_model_b_asymptotic = b_asy;
}
if (error_model_active)
logger.Info("Error model: a={:.3f} b={:.3f} ISa={:.1f} chi2={:.2f}", error_model_a, error_model_b,
error_model_b > 0 ? 1.0 / error_model_b : 0.0, error_model_chi2);
error_model_b_asymptotic > 0 ? 1.0 / error_model_b_asymptotic : 0.0, error_model_chi2);
auto corrected_sigma = [&](float I_corr, float sigma_corr, int g) -> float {
if (!error_model_active) return sigma_corr;
@@ -1266,7 +1579,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
}
// ---- Export merged reflections (+ resolution-shell R-free flags). ----
Result result;
result.isa = error_model_b > 0 ? 1.0 / error_model_b : 0.0;
result.isa = error_model_b_asymptotic > 0 ? 1.0 / error_model_b_asymptotic : 0.0;
std::vector<double> merged_I(n_groups, NAN);
float d_min = std::numeric_limits<float>::max(), d_max = 0.0f;
for (int g = 0; g < n_groups; ++g) {
@@ -1275,14 +1588,16 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
MergedReflection mr{};
mr.h = group_h[g]; mr.k = group_k[g]; mr.l = group_l[g];
mr.I = static_cast<float>(a.swI / a.sw);
mr.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(a.sw), mr.I, error_model_b);
// The systematic floor caps a high-multiplicity merged sigma at the reproducibility that the
// strongest reflections actually reach, i.e. the asymptotic term, so merged I/sigma approaches ISa.
mr.sigma = SigmaWithSystematicFloor(1.0 / std::sqrt(a.sw), mr.I, error_model_b_asymptotic);
mr.I_half[0] = mr.I_half[1] = NAN;
mr.sigma_half[0] = mr.sigma_half[1] = NAN;
mr.d = a.d;
if (a.nh[0] + a.nh[1] > 0 && a.swh[0] > 0.0 && a.swh[1] > 0.0) {
for (int i = 0; i < 2; ++i) {
mr.I_half[i] = static_cast<float>(a.swIh[i] / a.swh[i]);
mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(a.swh[i]), mr.I_half[i], error_model_b);
mr.sigma_half[i] = SigmaWithSystematicFloor(1.0 / std::sqrt(a.swh[i]), mr.I_half[i], error_model_b_asymptotic);
}
}
if (!std::isfinite(a.d) || a.d <= 0.0f) continue;
@@ -1301,7 +1616,8 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
result.merged, d_min_limit, resolution_cutoff_method, resolution_cc_target, for_search, logger);
AssignRfreeFlags(result.merged, x.GetSpaceGroupNumber().value_or(1), rfree_fraction);
ApplyFrenchWilson(result.merged, x.GetSpaceGroupNumber().value_or(1));
// French-Wilson (F, and F(+)/F(-) from the anomalous split) is deferred until after the anomalous
// accumulator below has attached I(+)/I(-), so the two hands get their amplitudes in one pass.
if (reject_count > 0)
logger.Info("Merge outlier rejection: dropped {} observations", reject_count);
@@ -1399,6 +1715,55 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
rmeas_num_all += factor * r.sum_abs_dev; rmeas_den_all += r.sum_I;
}
// ---- Anomalous split (always, even when the merge is Friedel-averaged): for each acentric
// reflection, the inverse-variance I(+)/I(-) from the SAME scaled fulls (so it never touches the
// Friedel-merged IMEAN, scaling or error model above). Lets I(+)/I(-) and F(+)/F(-) be written by
// default without scaling anomalously; a reflection with only one mate, or a centric, is left
// without the split. Skipped for the P1 search pass, which has no use for it. ----
struct AnomExport { float Ip = NAN, sIp = NAN, Im = NAN, sIm = NAN; };
std::unordered_map<uint64_t, AnomExport> anom_export;
// SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> accumulated on the standard report shells (num=sum|dI|,
// den=sum sigma(dI)); mmCIF pdbx_absDiff_over_sigma_anomalous. NaN'd below when there is no split.
std::vector<double> sig_num(n_shells, 0.0), sig_den(n_shells, 0.0);
double sig_num_all = 0.0, sig_den_all = 0.0;
size_t sig_n = 0;
if (!for_search) {
const int sg_num = x.GetSpaceGroupNumber().value_or(1);
const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, sg_num);
const gemmi::GroupOps gops = gemmi::find_spacegroup_by_number(sg_num)->operations();
struct AnomAcc { double swI[2] = {}; double sw[2] = {}; int32_t h = 0, k = 0, l = 0; float d = NAN; }; // [hand] 0=I(+) 1=I(-)
std::unordered_map<uint64_t, AnomAcc> anom;
anom.reserve(result.merged.size() * 2 + 1);
for (const auto &o : fulls) {
if (!usable_merge(o)) continue;
if (rejected_obs[&o - fulls.data()]) continue; // outlier-rejected in the merge (CPU path)
const HKLKey ak = anom_keygen(o.h, o.k, o.l);
const int hand = ak.plus ? 0 : 1;
const float I_corr = o.I * o.corr;
const float sigma_corr = corrected_sigma(I_corr, o.sigma * o.corr, o.group);
if (!(sigma_corr > 0.0f) || !std::isfinite(sigma_corr)) continue;
const double w = 1.0 / (static_cast<double>(sigma_corr) * sigma_corr);
AnomAcc &a = anom[HKLKey{ak.h, ak.k, ak.l, true}.pack()];
a.h = ak.h; a.k = ak.k; a.l = ak.l; a.d = o.d;
a.swI[hand] += w * static_cast<double>(I_corr); a.sw[hand] += w;
}
for (const auto &[fkey, a] : anom) {
// Centrics have I(+)=I(-) by symmetry; leave them without an anomalous split.
if (gops.is_reflection_centric(gemmi::Op::Miller{a.h, a.k, a.l})) continue;
AnomExport ex;
if (a.sw[0] > 0.0) { ex.Ip = static_cast<float>(a.swI[0] / a.sw[0]); ex.sIp = static_cast<float>(1.0 / std::sqrt(a.sw[0])); }
if (a.sw[1] > 0.0) { ex.Im = static_cast<float>(a.swI[1] / a.sw[1]); ex.sIm = static_cast<float>(1.0 / std::sqrt(a.sw[1])); }
if (std::isfinite(ex.Ip) || std::isfinite(ex.Im)) anom_export[fkey] = ex;
if (a.sw[0] > 0.0 && a.sw[1] > 0.0 && a.d > 0.0f) {
const double abs_di = std::fabs(a.swI[0] / a.sw[0] - a.swI[1] / a.sw[1]);
const double sig_di = std::sqrt(1.0 / a.sw[0] + 1.0 / a.sw[1]);
const auto shell = shells.GetShell(a.d);
if (shell && *shell >= 0 && *shell < n_shells) { sig_num[*shell] += abs_di; sig_den[*shell] += sig_di; }
sig_num_all += abs_di; sig_den_all += sig_di; ++sig_n;
}
}
}
MergeStatistics &out = result.statistics;
out.shells.resize(n_shells);
for (int s = 0; s < n_shells; ++s) {
@@ -1413,6 +1778,7 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
ss.cc_half = sa[s].cc_half.GetCC();
ss.cc_ref = NAN;
ss.r_meas = rmeas_den[s] > 0.0 ? rmeas_num[s] / rmeas_den[s] : NAN;
ss.abs_diff_over_sigma_anomalous = sig_den[s] > 0.0 ? sig_num[s] / sig_den[s] : NAN;
}
auto &overall = out.overall;
overall.d_min = sd_min; overall.d_max = sd_max;
@@ -1427,6 +1793,33 @@ RotationScaleMerge::Result RotationScaleMerge::MergeAndStats(int n_groups, bool
overall.cc_half = cc_half_overall.GetCC();
overall.cc_ref = NAN;
overall.r_meas = rmeas_den_all > 0.0 ? rmeas_num_all / rmeas_den_all : NAN;
overall.abs_diff_over_sigma_anomalous = sig_den_all > 0.0 ? sig_num_all / sig_den_all : NAN;
if (std::isfinite(overall.abs_diff_over_sigma_anomalous))
logger.Info("Anomalous signal SigAno = {:.2f} ({} acentric pairs)",
overall.abs_diff_over_sigma_anomalous, sig_n);
// Radiation-damage monitor (measured before any correction by MeasureRadiationDamageB): carry the
// first->last relative-B change and the per-batch curve into the reported statistics / mmCIF.
out.radiation_damage_delta_b = rad_damage_delta_b;
out.radiation_damage_b_batch = rad_damage_b_batch;
out.radiation_damage_batch_deg = rad_damage_batch_deg;
// Attach the per-reflection anomalous split so the writer can emit I(+)/I(-) by default (each merged
// reflection maps to its Friedel-ASU key; in an anomalous merge both mates map to the same key).
if (!anom_export.empty()) {
const HKLKeyGenerator anom_keygen(/*merge_friedel=*/false, x.GetSpaceGroupNumber().value_or(1));
for (auto &r : result.merged) {
const HKLKey ak = anom_keygen(r.h, r.k, r.l);
const auto it = anom_export.find(HKLKey{ak.h, ak.k, ak.l, true}.pack());
if (it == anom_export.end()) continue;
r.I_plus = it->second.Ip; r.sigma_plus = it->second.sIp;
r.I_minus = it->second.Im; r.sigma_minus = it->second.sIm;
}
}
// French-Wilson amplitudes for IMEAN and (now that they are attached) each Bijvoet hand.
ApplyFrenchWilson(result.merged, x.GetSpaceGroupNumber().value_or(1));
logger.Info("Merge complete ({} unique reflections)", result.merged.size());
return result;
}
@@ -1600,11 +1993,20 @@ RotationScaleMerge::Result RotationScaleMerge::Run(bool for_search,
// the merged reference. Cheap host loops over the downloaded fulls; applied only on the final
// in-symmetry merge, never the P1 space-group search pass (corrections there add risk and can
// perturb the symmetry determination). ---
[[maybe_unused]] const bool corrections = !for_search && (refine_decay_b || absorption_iter > 0);
[[maybe_unused]] const bool corrections =
!for_search && (refine_decay_b || absorption_iter > 0 || modulation_iter > 0 || relative_b_deg > 0.0);
// Radiation-damage monitor: measure the per-batch relative-B on the scaled fulls BEFORE any correction
// (report-only; captures the full damage signature, not a residual). Skipped on the P1 search pass.
if (!for_search)
MeasureRadiationDamageB(n_groups);
if (!for_search && refine_decay_b)
RefineDecay(n_groups);
if (!for_search && relative_b_deg > 0.0)
RefineRelativeB(n_groups); // per-batch relative-B on top of the single decay slope
if (!for_search && absorption_iter > 0)
RefineAbsorption(absorption_iter, n_groups);
if (!for_search && modulation_iter > 0)
RefineModulation(modulation_iter, n_groups);
#ifdef JFJOCH_USE_CUDA
// The corrections mutate the host fulls' corr; when the merge runs on the resident (GPU) fulls, push
// the corrected corr back to the device so the merge reads it.