v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/scale_merge/ReindexAmbiguity.cpp

@@ -10,6 +10,36 @@
 
 #include "HKLKey.h"
 
+namespace {
+    constexpr size_t MIN_REFLECTIONS = 20;
+
+    struct BestReindex {
+        gemmi::Op op = gemmi::Op::identity();
+        bool is_identity = true;
+        double score = 0.0;
+        double identity_score = 0.0;
+    };
+
+    // Among identity (the baseline, scored `identity_score`) and `ops`, return the operator with the
+    // highest `score_op(op)`. A non-finite score never wins (too little overlap to decide). Shared by the
+    // stills per-image resolver and the rotation post-merge ChooseReindex.
+    BestReindex PickBestReindex(double identity_score, const std::vector<gemmi::Op> &ops,
+                                const std::function<double(const gemmi::Op &)> &score_op) {
+        BestReindex best;
+        best.identity_score = identity_score;
+        best.score = identity_score;
+        for (const auto &op : ops) {
+            const double sc = score_op(op);
+            if (std::isfinite(sc) && (!std::isfinite(best.score) || sc > best.score)) {
+                best.score = sc;
+                best.op = op;
+                best.is_identity = false;
+            }
+        }
+        return best;
+    }
+}
+
 std::vector<gemmi::Op> ReindexAmbiguityOperators(const UnitCell &cell, int space_group_number,
                                                  double max_obliquity_deg) {
     const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(space_group_number);
@@ -37,21 +67,18 @@ ReindexChoice ChooseReindex(const std::vector<MergedReflection> &merged,
                             const UnitCell &cell, int space_group_number,
                             const std::function<double(const std::vector<MergedReflection> &)> &score,
                             double max_obliquity_deg) {
-    ReindexChoice best;
-    best.identity_score = score(merged);
-    best.score = best.identity_score;
-
     const auto ops = ReindexAmbiguityOperators(cell, space_group_number, max_obliquity_deg);
-    best.n_candidates = 1 + static_cast<int>(ops.size());
-    for (const auto &op : ops) {
-        const double s = score(ReindexReflections(merged, op));
-        if (s > best.score) {
-            best.score = s;
-            best.op = op;
-            best.is_identity = false;
-        }
-    }
-    return best;
+    const BestReindex best = PickBestReindex(
+        score(merged), ops,
+        [&](const gemmi::Op &op) { return score(ReindexReflections(merged, op)); });
+
+    ReindexChoice choice;
+    choice.op = best.op;
+    choice.is_identity = best.is_identity;
+    choice.score = best.score;
+    choice.identity_score = best.identity_score;
+    choice.n_candidates = 1 + static_cast<int>(ops.size());
+    return choice;
 }
 
 double ReferenceIntensityCC(const std::vector<MergedReflection> &merged,
@@ -87,3 +114,66 @@ double ReferenceIntensityCC(const std::vector<MergedReflection> &merged,
     const double vy = n * syy - sy * sy;
     return (vx > 0 && vy > 0) ? cov / std::sqrt(vx * vy) : 0.0;
 }
+
+ReindexAmbiguityResolver::ReindexAmbiguityResolver(const DiffractionExperiment &x,
+                                                   const std::vector<MergedReflection> &reference)
+    : s(x.GetScalingSettings()),
+      hkl_key_generator(s.GetMergeFriedel(), x.GetSpaceGroupNumber().value_or(1)) {
+    for (const auto &r : reference)
+        reference_data[hkl_key_generator(r)] = r.I;
+    if (x.GetUnitCell().has_value() && x.GetSpaceGroupNumber().has_value())
+        ops = ReindexAmbiguityOperators(*x.GetUnitCell(), static_cast<int>(*x.GetSpaceGroupNumber()));
+}
+
+bool ReindexAmbiguityResolver::Accept(const Reflection &r) const {
+    if (r.on_ice_ring)   // ice-contaminated intensity would bias the correlation; keep it out
+        return false;
+    return AcceptReflection(r, s.GetHighResolutionLimit_A());
+}
+
+double ReindexAmbiguityResolver::ReferenceCC(const std::vector<Reflection> &reflections,
+                                             const gemmi::Op &op) const {
+    double sx = 0, sy = 0, sxx = 0, syy = 0, sxy = 0;
+    size_t n = 0;
+    for (const auto &r : reflections) {
+        if (!Accept(r) || r.partiality < s.GetMinPartiality())
+            continue;
+        if (!std::isfinite(r.I) || !std::isfinite(r.sigma) || r.sigma <= 0.0f || r.partiality <= 0.0f)
+            continue;
+        const gemmi::Op::Miller h = op.apply_to_hkl({{r.h, r.k, r.l}});
+        const auto it = reference_data.find(hkl_key_generator(h[0], h[1], h[2]));
+        if (it == reference_data.end())
+            continue;
+        const double x = static_cast<double>(r.I) * r.rlp / r.partiality;
+        const double y = it->second;
+        if (!std::isfinite(x) || !std::isfinite(y))
+            continue;
+        sx += x; sy += y; sxx += x * x; syy += y * y; sxy += x * y;
+        ++n;
+    }
+    if (n < MIN_REFLECTIONS)
+        return NAN;
+    const double nd = static_cast<double>(n);
+    const double cov = sxy - sx * sy / nd;
+    const double vx = sxx - sx * sx / nd;
+    const double vy = syy - sy * sy / nd;
+    return (vx > 0 && vy > 0) ? cov / std::sqrt(vx * vy) : NAN;
+}
+
+// Serial stills index each crystal in one of the merohedrally-equivalent hands at random; pick, for this
+// image alone, the reindexing whose intensities correlate best with the external reference and apply it.
+void ReindexAmbiguityResolver::Resolve(std::vector<Reflection> &reflections) const {
+    if (ops.empty())
+        return;
+
+    const BestReindex best = PickBestReindex(
+        ReferenceCC(reflections, gemmi::Op::identity()), ops,
+        [&](const gemmi::Op &op) { return ReferenceCC(reflections, op); });
+    if (best.is_identity)
+        return;
+
+    for (auto &r : reflections) {
+        const gemmi::Op::Miller h = best.op.apply_to_hkl({{r.h, r.k, r.l}});
+        r.h = h[0]; r.k = h[1]; r.l = h[2];
+    }
+}