v1.0.0-rc.159 (#69)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

rugnux/ModelValidation.cpp

@@ -0,0 +1,324 @@
+// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
+// SPDX-License-Identifier: GPL-3.0-only
+
+#include "ModelValidation.h"
+
+#include <cmath>
+#include <complex>
+#include <array>
+#include <vector>
+#include <unordered_map>
+
+#include <gemmi/pdb.hpp>       // read_pdb
+#include <gemmi/gz.hpp>        // MaybeGzipped
+#include <gemmi/it92.hpp>      // IT92 x-ray form factors
+#include <gemmi/dencalc.hpp>   // DensityCalculator
+#include <gemmi/fourier.hpp>   // transform_map_to_f_phi, get_f_phi_on_grid, transform_f_phi_grid_to_map
+#include <gemmi/solmask.hpp>   // SolventMasker
+#include <gemmi/scaling.hpp>   // Scaling (bulk solvent + anisotropic B)
+#include <gemmi/ccp4.hpp>      // Ccp4 map I/O
+#include <gemmi/mtz.hpp>       // Mtz (map-coefficient output)
+
+#include "../common/Logger.h"
+#include "../image_analysis/scale_merge/ReindexAmbiguity.h"   // ReindexReflections
+
+namespace {
+
+using Table = gemmi::IT92<float>;
+
+// Stable key for a Miller index reduced into the ASU (indices are small, well within +/-512).
+long hkl_key(const gemmi::Miller &h) {
+    return (h[0] + 512L) * 1048576 + (h[1] + 512L) * 1024 + (h[2] + 512L);
+}
+
+// Write an FFT of ASU map coefficients as a CCP4 map; return its RMS (for reporting / sigma units).
+double write_ccp4(gemmi::AsuData<std::complex<float>> &coef, const std::string &path) {
+    coef.ensure_sorted();
+    std::array<int, 3> size = gemmi::get_size_for_hkl(coef, {{0, 0, 0}}, 3.0);
+    gemmi::Grid<float> map =
+        gemmi::transform_f_phi_grid_to_map(gemmi::get_f_phi_on_grid<float>(coef, size, true));
+    gemmi::Ccp4<float> ccp4;
+    ccp4.grid = map;
+    ccp4.update_ccp4_header(2);
+    ccp4.write_ccp4_map(path);
+    return ccp4.hstats.rms;
+}
+
+}  // namespace
+
+ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &merged,
+                                           const UnitCell &cell,
+                                           const std::string &model_path,
+                                           const std::string &output_prefix,
+                                           Logger &logger,
+                                           std::optional<int> data_space_group_number,
+                                           bool probe_indexing_ambiguity) {
+    ModelValidationResult result;
+
+    // --- read the atomic model ---
+    gemmi::Structure st;
+    try {
+        st = gemmi::read_pdb(gemmi::MaybeGzipped(model_path));
+    } catch (const std::exception &e) {
+        logger.Error("Model validation: cannot read model {}: {}", model_path, e.what());
+        return result;
+    }
+    if (st.models.empty() || !st.cell.is_crystal()) {
+        logger.Error("Model validation: model {} has no atoms or no unit cell", model_path);
+        return result;
+    }
+    const gemmi::SpaceGroup *sg = st.find_spacegroup();
+    if (!sg) {
+        logger.Error("Model validation: model {} has no usable space group", model_path);
+        return result;
+    }
+
+    // If the data was indexed in the enantiomorph of the model's space group (e.g. data P4(1)2(1)2,
+    // model P4(3)2(1)2 - the merged intensities cannot tell them apart), reindex the observed
+    // reflections into the model's hand so the two settings agree. This does not change the
+    // R-factors (which use |F|), but keeps the observed data consistent with the model.
+    std::vector<MergedReflection> reindexed;
+    const std::vector<MergedReflection> *obs_ptr = &merged;
+    if (data_space_group_number && *data_space_group_number != sg->number) {
+        const gemmi::SpaceGroup *dsg = gemmi::find_spacegroup_by_number(*data_space_group_number);
+        if (dsg && dsg->is_enantiomorphic() && sg->is_enantiomorphic()) {
+            gemmi::GroupOps eops = dsg->operations();
+            eops.change_basis_forward(dsg->change_of_hand_op());
+            const gemmi::SpaceGroup *enant = gemmi::find_spacegroup_by_ops(eops);
+            if (enant && enant->number == sg->number) {
+                reindexed = ReindexReflections(merged, dsg->change_of_hand_op());
+                obs_ptr = &reindexed;
+                logger.Info("Model validation: data space group {} is the enantiomorph of the model {}; "
+                            "reindexed the observed reflections into the model's hand",
+                            dsg->short_name(), sg->hm);
+            }
+        }
+    }
+    const std::vector<MergedReflection> &obs = *obs_ptr;
+
+    // Resolution limit from the data (the merged set is already resolution-trimmed).
+    double d_min = 0.0;
+    for (const MergedReflection &r : obs)
+        if (r.d > 0 && (d_min == 0.0 || r.d < d_min))
+            d_min = r.d;
+    if (d_min <= 0.0) {
+        logger.Error("Model validation: merged reflections carry no resolution");
+        return result;
+    }
+
+    // Re-fractionalize the model into the data cell (rigid cell adjustment; no refinement).
+    const gemmi::UnitCell data_cell = cell;  // UnitCell -> gemmi::UnitCell
+    if (data_cell.is_crystal()) {
+        gemmi::UnitCell old = st.cell;
+        for (gemmi::Model &m : st.models)
+            for (gemmi::Chain &ch : m.chains)
+                for (gemmi::Residue &r : ch.residues)
+                    for (gemmi::Atom &a : r.atoms)
+                        a.pos = data_cell.orthogonalize(old.fractionalize(a.pos));
+        st.cell = data_cell;
+    }
+    st.setup_cell_images();
+    const gemmi::UnitCell &ucell = st.cell;
+
+    logger.Info("Model validation: {} atoms, cell a={:.2f} b={:.2f} c={:.2f}, sg {}, to {:.2f} A",
+                gemmi::count_atom_sites(st.models[0]), ucell.a, ucell.b, ucell.c, sg->hm, d_min);
+
+    // --- Fcalc (atomic) via electron density on a grid + FFT ---
+    gemmi::DensityCalculator<Table, float> dc;
+    dc.d_min = d_min;
+    dc.rate = 1.5;
+    dc.set_grid_cell_and_spacegroup(st);
+    dc.set_refmac_compatible_blur(st.models[0]);
+    dc.put_model_density_on_grid(st.models[0]);
+    gemmi::AsuData<std::complex<float>> fcalc =
+        gemmi::transform_map_to_f_phi(dc.grid, true).prepare_asu_data(dc.d_min, dc.blur, false, false, false);
+
+    // --- flat bulk-solvent mask -> Fmask ---
+    // Refmac radii give a slightly lower R than the Cctbx set on our test cases, at no cost.
+    gemmi::SolventMasker masker(gemmi::AtomicRadiiSet::Refmac);
+    gemmi::Grid<float> mask_grid;
+    mask_grid.unit_cell = dc.grid.unit_cell;
+    mask_grid.spacegroup = dc.grid.spacegroup;
+    mask_grid.set_size_from_spacing(dc.requested_grid_spacing(), gemmi::GridSizeRounding::Up);
+    masker.put_mask_on_grid(mask_grid, st.models[0]);
+    gemmi::AsuData<std::complex<float>> fmask =
+        gemmi::transform_map_to_f_phi(mask_grid, true).prepare_asu_data(dc.d_min, 0);
+
+    gemmi::GroupOps gops = sg->operations();
+    gemmi::ReciprocalAsu asu(sg);
+
+    // --- fit the (scaled, solvent-corrected) model to one observed set and score it ---
+    // Factored into a lambda so we can probe indexing (merohedral) ambiguities: run the same scale +
+    // R computation on each reindexing of the observed reflections and keep the lowest-R-free one.
+    struct Fit {
+        gemmi::AsuData<std::complex<float>> fmodel, map2fofc, mapfofc;
+        std::unordered_map<long, std::pair<double, bool>> obs_by_hkl;  // hkl -> (Fobs, is_free)
+        double r_work = 1, r_free = 1, k_sol = 0, b_sol = 0, k_overall = 0;
+        int n_w = 0, n_f = 0;
+    };
+    auto fit_model = [&](const std::vector<MergedReflection> &obs_in) -> Fit {
+        Fit out;
+        out.fmodel = fcalc;   // copy the atomic structure factors; scaling mutates them in place
+
+        // --- observed amplitudes into the model ASU, keyed by hkl (also remember free flag) ---
+        // Observed amplitudes are the French-Wilson |F| already computed at the end of the merge
+        // (MergedReflection.F), so the model R-free / maps use exactly the same amplitudes as the
+        // written reflection file.
+        gemmi::AsuData<gemmi::ValueSigma<float>> fobs;
+        fobs.unit_cell_ = ucell;
+        fobs.spacegroup_ = sg;
+        for (const MergedReflection &r : obs_in) {
+            if (std::isnan(r.F)) continue;
+            gemmi::Miller h{{r.h, r.k, r.l}};
+            if (!asu.is_in(h)) h = asu.to_asu(h, gops).first;
+            fobs.v.push_back({h, {r.F, 1.0f}});
+            out.obs_by_hkl[hkl_key(h)] = {r.F, r.rfree_flag};
+        }
+        fobs.ensure_asu();
+        fobs.ensure_sorted();
+
+        // --- scale Fmodel(+solvent) to Fobs: k_overall, anisotropic B, k_sol, b_sol ---
+        gemmi::Scaling<float> scaling(ucell, sg);
+        scaling.use_solvent = true;
+        scaling.prepare_points(out.fmodel, fobs, &fmask);
+        scaling.fit_isotropic_b_approximately();
+        scaling.fit_parameters();
+        scaling.scale_data(out.fmodel, &fmask);   // out.fmodel now holds the scaled, solvent-corrected Fmodel
+        out.k_sol = scaling.k_sol;
+        out.b_sol = scaling.b_sol;
+        out.k_overall = scaling.k_overall;
+
+        // The model is scaled to the data with an overall scale, an anisotropic B and a flat bulk
+        // solvent only - the standard, few-parameter model that refinement programs use. A dataset-
+        // specific free-form per-resolution-shell rescale would lower this dataset's R a little, but
+        // it reshapes each map's radial amplitude profile differently, so a batch of maps would no
+        // longer be directly comparable. For a fragment-screening / PanDDA campaign, comparable maps
+        // across datasets matter more than the last bit of per-dataset R, so it is deliberately omitted.
+
+        // --- R-work / R-free and map coefficients (2Fo-Fc and Fo-Fc, model phases) ---
+        out.map2fofc.unit_cell_ = ucell; out.map2fofc.spacegroup_ = sg;
+        out.mapfofc.unit_cell_ = ucell;  out.mapfofc.spacegroup_ = sg;
+        double num_w = 0, den_w = 0, num_f = 0, den_f = 0;
+        for (const auto &hv : out.fmodel.v) {
+            auto it = out.obs_by_hkl.find(hkl_key(hv.hkl));
+            if (it == out.obs_by_hkl.end()) continue;
+            double Fo = it->second.first;
+            double Fc = std::abs(hv.value);
+            double phi = std::arg(hv.value);
+            if (it->second.second) { num_f += std::fabs(Fo - Fc); den_f += Fo; ++out.n_f; }
+            else                   { num_w += std::fabs(Fo - Fc); den_w += Fo; ++out.n_w; }
+            std::complex<float> ph = std::polar(1.0f, static_cast<float>(phi));
+            out.map2fofc.v.push_back({hv.hkl, static_cast<float>(2 * Fo - Fc) * ph});
+            out.mapfofc.v.push_back({hv.hkl, static_cast<float>(Fo - Fc) * ph});
+        }
+        out.r_work = den_w > 0 ? num_w / den_w : 1;
+        out.r_free = den_f > 0 ? num_f / den_f : 1;
+        return out;
+    };
+
+    // --- indexing (merohedral) ambiguity ---
+    // When a reference MTZ was supplied, the data were already reindexed to agree with the reference
+    // intensities (at the merge stage for rotation data, per image in stills scaling), and that
+    // choice is authoritative - we keep it. Only with a model and NO reference do we resolve the
+    // ambiguity here, as a fallback, by fitting each candidate reindexing and keeping the lowest
+    // R-free. A no-op either way for a holohedral crystal (no twin laws), e.g. lysozyme. The
+    // enantiomorph/screw ambiguity is never probed by R-free: |Fcalc| is the same for both hands, so
+    // it cannot distinguish them - that is taken from the model hand above.
+    Fit best = fit_model(obs);
+    if (probe_indexing_ambiguity) {
+        const auto reindex_ops = ReindexAmbiguityOperators(cell, sg->number);
+        bool did_reindex = false;
+        for (const auto &op : reindex_ops) {
+            Fit cand = fit_model(ReindexReflections(obs, op));
+            if (cand.r_free < best.r_free) { best = std::move(cand); did_reindex = true; }
+        }
+        if (!reindex_ops.empty())
+            logger.Info("Model validation: no reference - probed {} indexing solution(s) against the model; "
+                        "{} (R-free {:.4f})", reindex_ops.size() + 1,
+                        did_reindex ? "reindexed to the lower-R-free solution" : "kept the current indexing",
+                        best.r_free);
+    }
+
+    gemmi::AsuData<std::complex<float>> &fmodel = best.fmodel;
+    gemmi::AsuData<std::complex<float>> &map2fofc = best.map2fofc;
+    gemmi::AsuData<std::complex<float>> &mapfofc = best.mapfofc;
+    std::unordered_map<long, std::pair<double, bool>> &obs_by_hkl = best.obs_by_hkl;
+
+    result.r_work = best.r_work;
+    result.r_free = best.r_free;
+    result.n_work = best.n_w;
+    result.n_free = best.n_f;
+    result.k_sol = best.k_sol;
+    result.b_sol = best.b_sol;
+    result.k_overall = best.k_overall;
+
+    // --- write the maps and score the 2Fo-Fc map at atom centres (a real map peaks there) ---
+    const std::string p2 = output_prefix + "_2fofc.ccp4";
+    const std::string pd = output_prefix + "_fofc.ccp4";
+    double rms2 = write_ccp4(map2fofc, p2);
+    write_ccp4(mapfofc, pd);
+
+    // Recompute the 2Fo-Fc real-space grid once more to sample it at atom positions.
+    {
+        map2fofc.ensure_sorted();
+        std::array<int, 3> size = gemmi::get_size_for_hkl(map2fofc, {{0, 0, 0}}, 3.0);
+        gemmi::Grid<float> grid =
+            gemmi::transform_f_phi_grid_to_map(gemmi::get_f_phi_on_grid<float>(map2fofc, size, true));
+        double s = 0; int n = 0;
+        for (gemmi::Model &m : st.models)
+            for (gemmi::Chain &ch : m.chains)
+                for (gemmi::Residue &r : ch.residues)
+                    for (gemmi::Atom &a : r.atoms) { s += grid.interpolate_value(a.pos); ++n; }
+        result.mean_atom_density_sigma = (n > 0 && rms2 > 0) ? (s / n) / rms2 : 0;
+    }
+
+    // --- MTZ of map coefficients so the maps can be re-opened / rebuilt in Coot etc. ---
+    try {
+        gemmi::Mtz mtz(true);
+        mtz.spacegroup = sg;
+        mtz.set_cell_for_all(ucell);
+        mtz.add_dataset("model_validation");
+        mtz.add_column("FP", 'F', -1, -1, false);
+        mtz.add_column("FC", 'F', -1, -1, false);
+        mtz.add_column("PHIC", 'P', -1, -1, false);
+        mtz.add_column("FWT", 'F', -1, -1, false);
+        mtz.add_column("PHWT", 'P', -1, -1, false);
+        mtz.add_column("DELFWT", 'F', -1, -1, false);
+        mtz.add_column("PHDELWT", 'P', -1, -1, false);
+        mtz.add_column("FREE", 'I', -1, -1, false);
+        std::vector<float> data;
+        int nref = 0;
+        for (const auto &hv : fmodel.v) {
+            auto it = obs_by_hkl.find(hkl_key(hv.hkl));
+            if (it == obs_by_hkl.end()) continue;
+            double Fo = it->second.first;
+            double Fc = std::abs(hv.value);
+            double phi_deg = gemmi::phase_in_angles(hv.value);
+            data.insert(data.end(), {static_cast<float>(hv.hkl[0]), static_cast<float>(hv.hkl[1]),
+                                     static_cast<float>(hv.hkl[2]),
+                                     static_cast<float>(Fo), static_cast<float>(Fc),
+                                     static_cast<float>(phi_deg),
+                                     static_cast<float>(2 * Fo - Fc), static_cast<float>(phi_deg),
+                                     static_cast<float>(Fo - Fc), static_cast<float>(phi_deg),
+                                     it->second.second ? 0.0f : 1.0f});
+            ++nref;
+        }
+        mtz.nreflections = nref;
+        mtz.data = std::move(data);
+        mtz.write_to_file(output_prefix + "_maps.mtz");
+    } catch (const std::exception &e) {
+        logger.Warning("Model validation: could not write map MTZ: {}", e.what());
+    }
+
+    result.ok = true;
+    result.maps_prefix = output_prefix;
+    logger.Info("Model validation: R-work={:.4f} ({} refl)  R-free={:.4f} ({} refl)  "
+                "[overall + anisotropic B + bulk solvent]",
+                result.r_work, result.n_work, result.r_free, result.n_free);
+    logger.Info("Model validation: bulk solvent k_sol={:.3f} b_sol={:.1f}, k_overall={:.3f}",
+                result.k_sol, result.b_sol, result.k_overall);
+    logger.Info("Model validation: mean 2Fo-Fc density at atom centres = {:.2f} sigma", result.mean_atom_density_sigma);
+    logger.Info("Model validation: wrote {}_2fofc.ccp4, {}_fofc.ccp4, {}_maps.mtz",
+                output_prefix, output_prefix, output_prefix);
+    return result;
+}