v1.0.0-rc.159 (#69)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/scale_merge/SearchSpaceGroup.cpp

@@ -366,8 +366,64 @@ SearchSpaceGroupResult SearchSpaceGroup(
         return dof > 0 ? chi2 / static_cast<double>(dof) : std::numeric_limits<double>::quiet_NaN();
     };
 
-    // Operator-CC-confirmed candidates, each with its merge chi^2; chi2_ref = the most consistent.
-    struct PGCand { const PointGroupInfo* pg; int order; double min_class_cc; double chi2; };
+    // Extra intensity-proportional systematic error a point group's merge has to invoke to reconcile
+    // its symmetry equivalents: the smallest b for which sigma^2 + (b I)^2 brings the merged reduced
+    // chi^2 down to 1. A genuine symmetry needs almost none - its equivalents already agree within
+    // their errors, so the extra scatter is random and multiplicity absorbs it. A twin or pseudo-
+    // symmetry forces non-equivalent reflections together, and that disagreement scales with I, so b
+    // has to grow to swallow it (mirroring the merge error model's b / ISa collapse). This isolates
+    // the systematic part of the scatter, which the fixed-sigma chi^2 ratio cannot: a genuine but
+    // imperfectly-scaled high-symmetry merge and a twin can share a chi^2 ratio (~2) yet differ
+    // sharply here (cubic Ins_I_3 b x1.04 vs twin Ins_H_2 b x1.6).
+    auto merge_systematic_b = [&](const std::vector<gemmi::Op>& rotations) -> double {
+        struct Acc { double sw = 0.0, swI = 0.0; int n = 0; };
+        std::unordered_map<HKLKey, Acc, HKLKeyHash> grp;
+        std::vector<HKLKey> rep(n);
+        for (size_t i = 0; i < n; ++i) {
+            if (!pass_cc[i] || !(Sigma[i] > 0.0))
+                continue;
+            HKLKey best = key[i];
+            for (const auto& op : rotations) {
+                const auto m = op.apply_to_hkl(gemmi::Op::Miller{{H[i], K[i], L[i]}});
+                const HKLKey k2 = Canonicalize(m[0], m[1], m[2], opt.merge_friedel);
+                if (std::make_tuple(k2.h, k2.k, k2.l) < std::make_tuple(best.h, best.k, best.l))
+                    best = k2;
+            }
+            rep[i] = best;
+            auto& g = grp[best];
+            const double w = 1.0 / (Sigma[i] * Sigma[i]);
+            g.sw += w; g.swI += w * I[i]; g.n += 1;
+        }
+        std::vector<std::array<double, 3>> obs;   // I, sigma, deviation-from-orbit-mean
+        for (size_t i = 0; i < n; ++i) {
+            if (!pass_cc[i] || !(Sigma[i] > 0.0))
+                continue;
+            const auto& g = grp[rep[i]];
+            if (g.n < 2)
+                continue;
+            obs.push_back({I[i], Sigma[i], I[i] - g.swI / g.sw});
+        }
+        if (obs.size() < 20)
+            return 0.0;
+        auto reduced_chi2 = [&](double b) {
+            double s = 0.0;
+            for (const auto& o : obs)
+                s += o[2] * o[2] / (o[1] * o[1] + (b * o[0]) * (b * o[0]));
+            return s / static_cast<double>(obs.size());
+        };
+        if (reduced_chi2(0.0) <= 1.0)
+            return 0.0;
+        double lo = 0.0, hi = 2.0;   // b is a fraction of I; 2.0 = 200% is far past any real error model
+        for (int it = 0; it < 40; ++it) {
+            const double mid = 0.5 * (lo + hi);
+            (reduced_chi2(mid) > 1.0 ? lo : hi) = mid;
+        }
+        return 0.5 * (lo + hi);
+    };
+
+    // Operator-CC-confirmed candidates, each with its merge chi^2 and systematic-error b; chi2_ref =
+    // the most consistent.
+    struct PGCand { const PointGroupInfo* pg; int order; double min_class_cc; double chi2; double b_extra; };
     std::vector<PGCand> pg_cands;
     double chi2_ref = std::numeric_limits<double>::infinity();
     for (const auto& pg : point_groups) {
@@ -376,7 +432,8 @@ SearchSpaceGroupResult SearchSpaceGroup(
             continue;
         const double ch = pg.rotations.empty() ? std::numeric_limits<double>::quiet_NaN()
                                                : chi2_under(pg.rotations);
-        pg_cands.push_back({&pg, static_cast<int>(pg.rotations.size()) + 1, min_class_cc, ch});
+        const double be = pg.rotations.empty() ? 0.0 : merge_systematic_b(pg.rotations);
+        pg_cands.push_back({&pg, static_cast<int>(pg.rotations.size()) + 1, min_class_cc, ch, be});
         if (!pg.rotations.empty() && std::isfinite(ch))
             chi2_ref = std::min(chi2_ref, ch);
     }
@@ -397,8 +454,28 @@ SearchSpaceGroupResult SearchSpaceGroup(
         // compensated for an under-calibrated error model that inflated real-symmetry ratios with data
         // weakness; the variance-floor fix removed that inflation, and the widening now only let the
         // twin through, so it is gone.)
-        const bool consistent = c.pg->rotations.empty() || !std::isfinite(c.chi2) ||
-                                !std::isfinite(chi2_ref) || c.chi2 <= chi2_ref * opt.max_merge_chi2_ratio;
+        bool consistent = c.pg->rotations.empty() || !std::isfinite(c.chi2) ||
+                          !std::isfinite(chi2_ref) || c.chi2 <= chi2_ref * opt.max_merge_chi2_ratio;
+
+        // Rescue a genuine high-symmetry merge whose chi^2 lands just past the ratio bound because its
+        // data are imperfectly scaled (see max_merge_chi2_rescue). The systematic-error test tells it
+        // apart from a twin: promote only if the extra intensity-proportional error b, relative to the
+        // largest confirmed subgroup (by rotation-set inclusion), stayed within max_systematic_b_ratio -
+        // a genuine step barely moves it, a twin's balloons.
+        if (!consistent && std::isfinite(c.chi2) && std::isfinite(chi2_ref)
+            && c.chi2 <= chi2_ref * opt.max_merge_chi2_rescue) {
+            double parent_b = -1.0;
+            int parent_order = 0;
+            for (const auto& s : pg_cands)
+                if (s.order < c.order && s.order > parent_order
+                    && std::includes(c.pg->rotation_set.begin(), c.pg->rotation_set.end(),
+                                     s.pg->rotation_set.begin(), s.pg->rotation_set.end())) {
+                    parent_order = s.order;
+                    parent_b = s.b_extra;
+                }
+            if (parent_b > 1e-4 && c.b_extra <= parent_b * opt.max_systematic_b_ratio)
+                consistent = true;
+        }
         if (!consistent)
             continue;
         if (c.order > best_pg_order || (c.order == best_pg_order && c.min_class_cc > best_pg_min_cc)) {