v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/scale_merge/Merge.cpp

@@ -93,7 +93,7 @@ void MergeOnTheFly::AddImage(const IntegrationOutcome &outcome, int64_t image_id
             continue;
         auto hkl = generator(r);
         auto hkl_key = hkl.pack();
-        sigma_corr = CorrectedSigma(I_corr, sigma_corr, hkl_key);
+        sigma_corr = CorrectedSigma(I_corr, sigma_corr, hkl_key, r.partiality);
 
         // Robust outlier rejection: drop this observation if it sits more than
         // reject_nsigma error-model sigmas from the reflection's median. Needs the active
@@ -131,7 +131,138 @@ void MergeOnTheFly::AddImage(const IntegrationOutcome &outcome, int64_t image_id
     }
 }
 
-float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl_key) const {
+double MergeOnTheFly::RefineModulation(std::vector<IntegrationOutcome> &outcomes) {
+    // A minimum held-out generalizing gain (fraction of the held-out scatter) before the surface is
+    // applied - a margin, so a noise-level "improvement" never engages the correction. Matches the
+    // rotation ApplyCellSurface gate.
+    constexpr double CV_MIN_RELATIVE_GAIN = 0.02;
+    constexpr int NB = 16;
+    const int ncell = NB * NB;
+
+    // One scaled observation reduced to what the surface fit needs. `parity` (image index & 1) drives the
+    // even/odd cross-validation split - the stills analogue of the rotation frame parity.
+    struct MObs { double I, sigma, corr; float px, py; int32_t group; int parity; int cell; };
+    std::vector<MObs> obs;
+
+    // Accept exactly what AddImage merges (systematic absence, scale/resolution/ice/partiality filters),
+    // and additionally require a finite detector position. Returns the dense ASU-group id or -1.
+    std::unordered_map<uint64_t, int> group_of;
+    auto accept = [&](const Reflection &r, MObs &out) -> bool {
+        if (generator.IsSystematicallyAbsent(r)) return false;
+        if (r.image_scale_corr <= 0.0f || !std::isfinite(r.image_scale_corr)) return false;
+        if (!AcceptReflection(r, high_resolution_limit)) return false;
+        if (exclude_ice_rings && r.on_ice_ring) return false;
+        if (IsMaskedRing(r)) return false;
+        if (r.partiality < min_partiality) return false;
+        if (!std::isfinite(r.predicted_x) || !std::isfinite(r.predicted_y)) return false;
+        const float I_corr = r.I * r.image_scale_corr, sigma_corr = r.sigma * r.image_scale_corr;
+        if (!std::isfinite(I_corr) || !std::isfinite(sigma_corr) || sigma_corr <= 0.0f) return false;
+        const uint64_t key = generator(r).pack();
+        auto [it, inserted] = group_of.try_emplace(key, static_cast<int>(group_of.size()));
+        out.I = r.I; out.sigma = r.sigma; out.corr = r.image_scale_corr;
+        out.px = r.predicted_x; out.py = r.predicted_y; out.group = it->second;
+        return true;
+    };
+
+    float pxmin = std::numeric_limits<float>::infinity(), pxmax = -pxmin, pymin = pxmin, pymax = -pxmin;
+    for (size_t i = 0; i < outcomes.size(); ++i) {
+        const int parity = static_cast<int>(i & 1);
+        for (const auto &r : outcomes[i].reflections) {
+            MObs m{};
+            if (!accept(r, m)) continue;
+            m.parity = parity;
+            pxmin = std::min(pxmin, m.px); pxmax = std::max(pxmax, m.px);
+            pymin = std::min(pymin, m.py); pymax = std::max(pymax, m.py);
+            obs.push_back(m);
+        }
+    }
+    const int n_groups = static_cast<int>(group_of.size());
+    if (obs.size() < static_cast<size_t>(8 * ncell) || !(pxmax > pxmin) || !(pymax > pymin))
+        return 0.0;   // too sparse to over-determine a 16x16 surface, or degenerate detector footprint
+
+    const float sx = NB / (pxmax - pxmin), sy = NB / (pymax - pymin);
+    for (auto &m : obs) {
+        const int ix = std::clamp(static_cast<int>((m.px - pxmin) * sx), 0, NB - 1);
+        const int iy = std::clamp(static_cast<int>((m.py - pymin) * sy), 0, NB - 1);
+        m.cell = ix * NB + iy;
+    }
+
+    // Fit the per-cell factor over {parity subset} (parity < 0 = all obs), n_iter alternating rounds against
+    // that subset's own inverse-variance reference: Tikhonov pull to 1, gauge-fixed to a den-weighted
+    // geometric mean of 1 so it never drifts the overall scale. Mirrors RotationScaleMerge::ApplyCellSurface.
+    constexpr int N_ITER = 3;
+    auto fit_surface = [&](int parity) -> std::vector<double> {
+        std::vector<double> A(ncell, 1.0);
+        for (int it = 0; it < N_ITER; ++it) {
+            std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
+            for (const auto &o : obs) {
+                if (parity >= 0 && o.parity != parity) continue;
+                const double a = A[o.cell], sc = o.sigma * o.corr * a, w = 1.0 / (sc * sc);
+                sw[o.group] += w; swI[o.group] += w * o.I * o.corr * a;
+            }
+            std::vector<double> num(ncell, 0.0), den(ncell, 0.0);
+            for (const auto &o : obs) {
+                if ((parity >= 0 && o.parity != parity) || sw[o.group] <= 0.0) continue;
+                const double Iref = swI[o.group] / sw[o.group], a = A[o.cell];
+                const double Is = o.I * o.corr * a, sc = o.sigma * o.corr * a;
+                if (!std::isfinite(Iref) || Iref <= 0.0 || !(Is > 0.0) || !(sc > 0.0)) continue;
+                const double w = 1.0 / (sc * sc);
+                num[o.cell] += w * Is * Iref; den[o.cell] += w * Is * Is;
+            }
+            std::vector<double> dsorted = den;
+            std::nth_element(dsorted.begin(), dsorted.begin() + dsorted.size() / 2, dsorted.end());
+            const double lambda = 0.1 * std::max(1e-30, dsorted[dsorted.size() / 2]);
+            double logsum = 0.0, wsum = 0.0;
+            std::vector<double> upd(ncell, 1.0);
+            for (int c = 0; c < ncell; ++c) upd[c] = (num[c] + lambda) / (den[c] + lambda);
+            for (int c = 0; c < ncell; ++c) if (den[c] > 0.0) { logsum += den[c] * std::log(upd[c]); wsum += den[c]; }
+            const double gm = wsum > 0.0 ? std::exp(logsum / wsum) : 1.0;
+            for (int c = 0; c < ncell; ++c) A[c] = std::clamp(A[c] * upd[c] / gm, 0.25, 4.0);
+        }
+        return A;
+    };
+    // Sigma-independent (R-meas-like) agreement of the held-out equivalents: sum|Is - Iref| / sum|Iref|.
+    // A fractional metric cannot be gamed by a surface that merely reshapes sigma via corr.
+    auto score = [&](int parity, const std::vector<double> &A) -> double {
+        std::vector<double> sw(n_groups, 0.0), swI(n_groups, 0.0);
+        for (const auto &o : obs) {
+            if (o.parity != parity) continue;
+            const double a = A[o.cell], Is = o.I * o.corr * a, sc = o.sigma * o.corr * a, w = 1.0 / (sc * sc);
+            sw[o.group] += w; swI[o.group] += w * Is;
+        }
+        double num = 0.0, den = 0.0;
+        for (const auto &o : obs) {
+            if (o.parity != parity || sw[o.group] <= 0.0) continue;
+            const double a = A[o.cell], Is = o.I * o.corr * a, Iref = swI[o.group] / sw[o.group];
+            if (!std::isfinite(Iref) || Iref <= 0.0) continue;
+            num += std::abs(Is - Iref); den += Iref;
+        }
+        return den > 0.0 ? num / den : 0.0;
+    };
+
+    // Cross-validate: fit on even images, score the held-out odd equivalents (and vice versa). Apply the
+    // full-data surface only if the held-out agreement improves by a clear margin.
+    const std::vector<double> ident(ncell, 1.0);
+    const std::vector<double> A_even = fit_surface(0), A_odd = fit_surface(1);
+    const double base = score(1, ident) + score(0, ident);
+    const double gain = base - (score(1, A_even) + score(0, A_odd));
+    if (!(gain > CV_MIN_RELATIVE_GAIN * base))
+        return 0.0;   // not cross-validated: the correction stays a no-op (caller logs)
+    const std::vector<double> A = fit_surface(-1);
+
+    // Fold the surface into each accepted reflection's image_scale_corr (recompute its cell deterministically).
+    for (auto &outcome : outcomes)
+        for (auto &r : outcome.reflections) {
+            MObs m{};
+            if (!accept(r, m)) continue;
+            const int ix = std::clamp(static_cast<int>((m.px - pxmin) * sx), 0, NB - 1);
+            const int iy = std::clamp(static_cast<int>((m.py - pymin) * sy), 0, NB - 1);
+            r.image_scale_corr = static_cast<float>(r.image_scale_corr * A[ix * NB + iy]);
+        }
+    return gain / std::max(base, 1e-30);   // held-out gain fraction (caller logs)
+}
+
+float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl_key, float partiality) const {
     if (!error_model_active)
         return sigma_corr;
 
@@ -140,8 +271,21 @@ float MergeOnTheFly::CorrectedSigma(float I_corr, float sigma_corr, uint64_t hkl
     const auto it = error_model_mean_I.find(hkl_key);
     const double I_for_b = (it != error_model_mean_I.end()) ? it->second : I_corr;
 
-    const double v = error_model_a * static_cast<double>(sigma_corr) * sigma_corr
-                     + (error_model_b * I_for_b) * (error_model_b * I_for_b);
+    double v = error_model_a * static_cast<double>(sigma_corr) * sigma_corr
+               + (error_model_b * I_for_b) * (error_model_b * I_for_b);
+
+    // Partiality-model uncertainty: a reflection recorded at fraction p carries a systematic intensity
+    // error ~ (dp/p) that is proportional to <I> and grows as p falls - plain counting sigma misses it,
+    // so strong low-p partials would otherwise be over-trusted. This is the stills-partiality analog of
+    // the rotation --capture-uncertainty term ((1-captured_fraction)*I in RotationScaleMerge). Inert when
+    // partiality == 1 (no stills partiality model). Gated on a real systematic (error_model_b > 1, i.e.
+    // ISa < 1): on weak counting-limited data (small b) it would only over-concentrate the merge and hurt.
+    const double c = scaling_settings.GetPartialityUncertaintyCoeff();
+    if (c > 0.0 && error_model_b > 1.0) {
+        const double one_minus_p = std::max(0.0, std::min(1.0, 1.0 - static_cast<double>(partiality)));
+        const double t = c * I_for_b * one_minus_p;
+        v += t * t;
+    }
     return (v > 0.0) ? static_cast<float>(std::sqrt(v)) : sigma_corr;
 }
 
@@ -330,9 +474,6 @@ std::vector<char> MergeOnTheFly::DeltaCChalfReject(const std::vector<Integration
     std::unordered_map<uint64_t, Acc> acc;
     std::vector<int> img_half(outcomes.size(), 0);
 
-    std::mt19937 lrng{2026061600u};
-    std::bernoulli_distribution hd{0.5};
-
     // ---- pass 1: accumulate half-set sums, record each image's half ----
     auto contribution = [&](const Reflection &r, uint64_t &key, double &wI, double &w) -> bool {
         if (generator.IsSystematicallyAbsent(r)) return false;
@@ -349,7 +490,9 @@ std::vector<char> MergeOnTheFly::DeltaCChalfReject(const std::vector<Integration
     };
 
     for (size_t i = 0; i < outcomes.size(); ++i) {
-        const int half = hd(lrng) ? 1 : 0;
+        // Same deterministic half-set as the merge (HalfForImage), so deltaCChalf is measured on the
+        // exact CC1/2 split the statistics report - not an independent, order-dependent RNG draw.
+        const int half = HalfForImage(static_cast<int64_t>(i));
         img_half[i] = half;
         for (const auto &r : outcomes[i].reflections) {
             uint64_t key; double wI, w;
@@ -743,7 +886,7 @@ std::ostream &operator<<(std::ostream &output, const MergeStatisticsShell &in) {
     double multiplicity = in.unique_reflections > 0
         ? static_cast<double>(in.total_observations) / in.unique_reflections : 0.0;
 
-    output << fmt::format("{:8d} {:8d} {:8d} {:7.1f}% {:7.1f} {:8.1f} {:7.1f}% {:7.1f}% {:7.1f}%",
+    output << fmt::format("{:8d} {:8d} {:8d} {:7.1f}% {:7.1f} {:8.1f} {:7.1f}% {:7.1f}% {:7.1f}% {:8.2f}",
                 in.total_observations,
                 in.unique_reflections,
                 in.possible_unique_reflections,
@@ -752,17 +895,18 @@ std::ostream &operator<<(std::ostream &output, const MergeStatisticsShell &in) {
                 in.mean_i_over_sigma,
                 in.r_meas*100.0,
                 in.cc_half*100.0,
-                in.cc_ref*100.0);
+                in.cc_ref*100.0,
+                in.abs_diff_over_sigma_anomalous);
     return output;
 }
 
 std::ostream &operator<<(std::ostream &output, const MergeStatistics &in) {
     output << std::endl;
-    output << fmt::format("  {:>8s} {:>8s} {:>8s} {:>8s} {:>8s} {:>7s} {:>8s} {:>8s} {:>8s} {:>8s}",
-        "d_min", "N_obs", "N_uniq", "N_possib", "Compl", "Mult", "<I/sig>", "R_meas", "CC1/2", "CCref")
+    output << fmt::format("  {:>8s} {:>8s} {:>8s} {:>8s} {:>8s} {:>7s} {:>8s} {:>8s} {:>8s} {:>8s} {:>8s}",
+        "d_min", "N_obs", "N_uniq", "N_possib", "Compl", "Mult", "<I/sig>", "R_meas", "CC1/2", "CCref", "SigAno")
     << std::endl;
-    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s}",
-        "", "", "", "", "", "", "", "", "", "") << std::endl;
+    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s} {:->8s}",
+        "", "", "", "", "", "", "", "", "", "", "") << std::endl;
     for (const auto &sh: in.shells) {
         if (sh.unique_reflections == 0)
             continue;
@@ -770,12 +914,19 @@ std::ostream &operator<<(std::ostream &output, const MergeStatistics &in) {
         output << sh;
         output << std::endl;
     }
-    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s}",
-        "", "", "", "", "", "", "", "", "", "") << std::endl;
+    output << fmt::format("  {:->8s} {:->8s} {:->8s} {:->8s} {:->8s} {:->7s} {:->8s} {:->8s} {:->8s} {:->8s} {:->8s}",
+        "", "", "", "", "", "", "", "", "", "", "") << std::endl;
 
     output << fmt::format("  {:>8s} ", "Overall");
     output << in.overall;
     output << std::endl;
+    if (std::isfinite(in.wilson_b) && in.wilson_b > 0.0)
+        output << fmt::format("  Wilson B-factor estimate: {:.2f} A^2 (correlation {:.3f})",
+                              in.wilson_b, in.wilson_b_correlation) << std::endl;
+    if (std::isfinite(in.radiation_damage_delta_b))
+        output << fmt::format("  Radiation damage: relative B-factor change over run = {:+.2f} A^2 "
+                              "(first->last, {} batches)",
+                              in.radiation_damage_delta_b, in.radiation_damage_b_batch.size()) << std::endl;
     output << std::endl;
     return output;
 }