diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index b0a12e309..50d7b697e 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -27,8 +27,18 @@ statistics and its map coefficients go out; the restraint dictionary is compiled ### 15.1 What is refined, and in what order The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed, -and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within -100-300 Ų). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS +and every B inside the range the refinement allows (2 Ų up to twice the model's largest B, within 100-300 Ų; +a model that arrives with B above 300 Ų - a structure refined at 4-5 Å - gets a ceiling of 1.5 times its +largest B instead, so that it is not flattened; a later round of the weights (§15.1, 4) raises the ceiling from +the refined model). + +The model is refined in the data's space group - except where its own group is a proper subgroup of it: a +P2₁ model whose copies are pseudo-symmetric, validated in its own group against a merge in P4₁2₁2 that the +data could not tell from P2₁. Under the data's group the operators the model does not have would overlay its +copies and R would be the null's, so it is refined in its own group, against the merge expanded to that group +(every reflection the data's group made equivalent, at the merged amplitude and with the same free flag), and +no twin law is tested, the data carrying the larger symmetry exactly. The refined model is written in the group +it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest @@ -297,7 +307,11 @@ geometry - is kept). The trials are: 1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors - held and refined at $w_{B,0}$ - six trials. + held and refined at $w_{B,0} \times \{1/16, 1/4, 1\}$ - twelve trials (the B factors are refined first at + the middle rung only, and at the others where that comes within two ties of holding them there). The lower + B weights are for a model whose B factors arrive far from where they refine - a prediction's pLDDT - whose + balanced weight is not the one its B wants: they took 0.002-0.010 off R-free on five such starts, and + nothing off eight deposited models. 2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and @@ -309,9 +323,9 @@ are: was never kept on 55 refinements, and taking it away changed no choice). 4. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U held), each trial as long as a step on the positions where the weights ended, held - only the displacement - parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with - the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held - positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain + parameters refine, so the trials differ in the displacement model alone (where the geometry is out of bounds + there, which held positions cannot mend, each trial refines the positions too, a macro-cycle longer than a + step): TLS groups with a residual B - the model's own, one per chain of at least 50 refinable atoms, and the chains cut into 4, 8 and 16 contiguous segments of at least 10 residues as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r), to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U @@ -327,6 +341,16 @@ are: `REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no TLS group holds a water. + Then the weights and the displacement model are **decided again** from the model as it now is, up to twice: the + B ceiling and the balanced weights from its B factors, the ladder from it with its displacements as they are, the + trajectory, and the displacement hypotheses from where that ends. The first choice took its weights with the + B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT, + the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$), + the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the + refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers + R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the + hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder + and one step. No round runs where the refinement does not displace the model as given. 5. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index f263db765..7da5065ff 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -184,7 +184,8 @@ The columns are found, not named - amplitudes with their sigmas (F/SIGF, FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none. -The model is taken as placed in the data's cell, and refined in the data's space group. +The model is taken as placed in the data's cell, and refined in the data's space group (in its own where that is a +subgroup of the data's, against the data expanded to it). | Option | Description | | --- | --- | diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index 34915dea2..de168046a 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -321,7 +321,8 @@ rugnux --mode refine --model model.cif -o refined data.mtz ``` The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by -content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group. The +content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its +own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 3bcb8e59a..f9b6752bd 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -52,6 +52,12 @@ namespace { constexpr double B_FLOOR = 2.0; // A^2: no atom's B goes below it // nor above twice the model's largest B, within these bounds (the GPU's gather is reserved for it) constexpr double B_CEILING_MIN = 100.0, B_CEILING_MAX = 300.0; +// The ceiling of B for a model whose highest B is `b_max`: twice it, within the bounds - the bounds lifted for a +// model that comes with B above them (a structure refined at 4-5 A, B 400-600 A^2), which the ceiling would +// otherwise flatten (the gather is reserved for it). +double BCeiling(double b_max) { + return std::clamp(2 * b_max, B_CEILING_MIN, std::max(B_CEILING_MAX, 1.5 * b_max)); +} // A B tensor (8 pi^2 U) inside the bounds: its smallest eigenvalue raised to `floor` by a multiple of I, then the // whole tensor scaled down so that its largest is at most `ceiling`. The ceiling comes last because the GPU's @@ -78,6 +84,8 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0; constexpr int SOLVENT_ITERATIONS = 10; // R-free differences smaller than this do not decide between two weights: the tighter geometry is taken. constexpr double R_FREE_TIE = 0.001; +// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each +// after it from the model the one before ended with. // The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this, // relative; the most a position (A) or a B (A^2) moves in the first trial of a line search. constexpr double MINIMISER_F_TOLERANCE = 1e-7; @@ -138,7 +146,7 @@ gemmi::Structure PrepareModel(const gemmi::Structure &placed, const gemmi::UnitC a.charge = ScatteringCharge(r, a); if (a.aniso.nonzero()) a.b_iso = static_cast(gemmi::u_to_b() * a.aniso.trace() / 3.0); - a.b_iso = static_cast(std::clamp(a.b_iso, B_FLOOR, B_CEILING_MAX)); + a.b_iso = static_cast(std::max(a.b_iso, B_FLOOR)); } } gemmi::remove_empty_children(st); @@ -263,8 +271,8 @@ struct Context { continue; // its widest axis no wider than the refinement allows const double top = AnisoRadiusB(at.aniso.scaled(gemmi::u_to_b())); - if (top > B_CEILING_MAX) { - at.aniso = at.aniso.scaled(static_cast(B_CEILING_MAX / top)); + if (top > b_ceiling) { + at.aniso = at.aniso.scaled(static_cast(b_ceiling / top)); at.b_iso = static_cast(gemmi::u_to_b() * at.aniso.trace() / 3.0); } const double t = at.aniso.trace() / 3.0; @@ -1446,7 +1454,29 @@ void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) { for (size_t d = first; d < a.size(); d++) result.twin_fraction_null += a[d]; } - const std::vector> f_null = TwinnedModel(null, rows, ctx.fcalc); + // TwinnedModel with each unrelated mate on its own scale k_total: TwinnedModel puts a mate on the scale of the + // reflection it is the mate of, as the twin domain's reflection at the same spot is measured, but an unrelated + // mate lies in another direction, where the anisotropic scale can differ many times over (22 times in amplitude + // at 2 A, on a model whose U carried the anisotropy opposite to k_total), and on the reflection's scale it put + // the null 0.06 in R-free above the untwinned refinement - a law "ahead" of its null by 7.9 standard errors at a + // fraction of 0.04. + std::vector> f_null(n); + double own_fraction = 1.0; + for (float a : rows.twin_fractions) + own_fraction -= a; + for (size_t m = 0; m < n; m++) { + const std::complex u = std::complex(ctx.fcalc[m]) + std::complex(rows.fmask[m]); + double i_model = own_fraction * rows.k_total[m] * rows.k_total[m] * std::norm(u); + for (size_t d = 0; d < null.Laws(); d++) { + const int j = null.mates[d][m]; + const double k = d < first ? rows.k_total[m] : rows.k_total[j]; + i_model += rows.twin_fractions[d] * k * k * + std::norm(std::complex(ctx.fcalc[j]) + std::complex(rows.fmask[j])); + } + const double amplitude = std::sqrt(std::max(i_model, 0.0)); + f_null[m] = std::norm(u) > 0 ? std::complex(u * (amplitude / std::abs(u))) + : std::complex(static_cast(amplitude), 0); + } result.twin_r_free_null = ComputeRFactors(null, f_null).r_free; // How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin|| // over its standard error. The members of a free twin group share their F_calc, so their terms are summed into @@ -1828,10 +1858,9 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s t.within_bounds = t.geometry.rmsd_bonds <= MAX_RMSD_BONDS && t.geometry.rmsd_angles <= MAX_RMSD_ANGLES; if (ncs != nullptr && !ncs->Empty()) t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB()); - if (hyp != nullptr) { + if (hyp != nullptr) t.adp = hyp->name; - t.nll_free = ctx.Score().nll_free; - } + t.nll_free = ctx.Score().nll_free; out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, " "angles {:.2f} deg{}{}", hyp != nullptr ? hyp->name + ", " : std::string(), w, @@ -1840,8 +1869,7 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s ncs == nullptr || ncs->Empty() ? std::string() : ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd) : fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd), - rf.r_work, rf.r_free, - hyp != nullptr ? fmt::format(", -LL free {:.1f}", t.nll_free) : std::string(), + rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), t.geometry.rmsd_bonds, t.geometry.rmsd_angles, t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal) : std::string(), @@ -2283,7 +2311,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vectorb_iso + gemmi::u_to_b() * body.U(ctx.atoms[i]->pos).trace() / 3.0); - ctx.b_ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX); + ctx.b_ceiling = BCeiling(b_max); if (ctx.atoms.empty()) { result.failure_reason = "the model has no atoms"; return result; @@ -2664,7 +2692,16 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; + std::vector outcomes; + // The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the + // balanced weights as they stand: the trial kept, or one that did not run where none did. + auto ladder_choice = [&](const std::vector &from_pos, const std::vector &from_b, + const std::vector &from_aniso, const Context::Fit *fit) { std::vector ladder; for (double rung : settings.weight_ladder) for (double adp : settings.adp_ladder) { @@ -2672,14 +2709,13 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; - std::vector outcomes(ladder.size()); + outcomes.assign(ladder.size(), TrialOutcome()); auto run_ladder = [&](const std::function &these) { std::vector t; for (const TrialWeights &l : ladder) if (these(l)) t.push_back(l); - std::vector ran = RunTrials(trial_slots, t, refinable, pos0, b0, aniso0, nullptr, + std::vector ran = RunTrials(trial_slots, t, refinable, from_pos, from_b, from_aniso, fit, settings.macro_cycles, settings, ncs_p); for (size_t i = 0, m = 0; i < ladder.size(); i++) if (these(ladder[i])) @@ -2720,10 +2756,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0) - state = std::move(outcomes[k]); + const int k = choose(outcomes, "ladder"); + return k >= 0 ? std::move(outcomes[k]) : TrialOutcome(); + }; + TrialOutcome state = ladder_choice(pos0, b0, aniso0, nullptr); // 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they // are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the @@ -2738,11 +2774,17 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector from_best; // `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since). - auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best) { + // `label`: the steps' name in the log. + auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best, const char *label) { TrialOutcome best = state; from_best.clear(); bool state_is_best = true; - for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) { + // A step taken on R-work alone is followed by one more at most: on a model refined before, R-work falls + // step after step while R-free does not, and the trajectory ends at its best anyway - a model of 490k + // atoms spent six such steps (240 s) on a round it then gave back. (On 28 refinements of 22 deposited + // models and 6 predictions the rule changed nothing.) + int since_best = 0; + for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled() && since_best <= 1; step++) { const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma; std::vector trials = {{w, adp, ncs_s}}; if (step % 2 == 1 && adp > 0) { @@ -2763,7 +2805,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0 && displaces(outcomes[k].trial, state.trial)) state = std::move(outcomes[k]); } @@ -3087,7 +3130,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector &own_tls) { const RefinementStage adp_stage("adp_hypotheses"); ctx.SetPositions(state.pos); ctx.SetB(state.b); @@ -3097,7 +3141,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector hyps = AdpHypotheses(ctx, topo, input_tls); + std::vector hyps = AdpHypotheses(ctx, topo, own_tls); const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0; // the U weight balanced on the model with every refinable atom given a U double u_w0 = adp; @@ -3211,7 +3255,9 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k); } - result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held"; + // (the model where the weights ended keeps the displacement model a round before chose for it) + if (result.adp_model.empty() || state.trial.adp.empty()) + result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held"; if (best >= 0) { result.adp_model = outcomes[best].trial.adp; state = std::move(outcomes[best]); @@ -3232,6 +3278,89 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector own_tls = state.tls; + ctx.SetPositions(state.pos); + ctx.SetB(state.b); + ctx.SetAniso(state.aniso); + ctx.Restore(state.fit); + PlaceRiders(ctx, topo); + // the ceiling of B from the model as it now is, as from the model as given: a placed prediction comes + // with B factors of 6 A^2 under an overall B of 45 (its ceiling 100) and refines at B near 50-70, where + // the TLS bodies and the U per atom were held at the ceiling; where it rises the engines are made again + // (the gather is reserved for it) + double b_max = 0; + for (size_t i = 0; i < ctx.atoms.size(); i++) + b_max = std::max(b_max, ctx.atoms[i]->b_iso); + const double ceiling = BCeiling(b_max); + if (ceiling > ctx.b_ceiling) { + const RefinementStage stage("engine"); + for (std::unique_ptr &o : own) + if (o) + o->ctx.engine.reset(); + ctx.b_ceiling = ceiling; + MakeEngine(ctx, settings.gpu, settings, logger, restraint_records, restraint_slots); + ctx.engine->SetRestraints(0, RestraintsNow(ctx, topo)); + ctx.engine->SetRestraints(1, topo.adp); + for (size_t k = 0; k < own.size(); k++) + if (own[k]) { + Context &o = own[k]->ctx; + o.b_ceiling = ceiling; +#ifdef JFJOCH_USE_CUDA + if (ctx.engine->OnGPU()) + o.engine = MakeRefinementEngineGPU(*ctx.engine, devices[k].device, o.nthreads, devices[k].own_lane); +#endif + if (!o.engine) + o.engine = MakeRefinementEngineCPU(o.cell, *o.sg, o.d_min, o.refl, o.nthreads); + o.engine->SetAniso(&o.aniso); + o.engine->SetRestraints(1, own[k]->topo.adp); + } + ctx.Restore(state.fit); + } + { + const RefinementStage stage("balanced_weight"); + w0 = BalancedWeight(ctx, refinable); + adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable); + } + logger.Info("Refinement, round {}: balanced X-ray weights {:.3g} (positions), {:.3g} (B), B ceiling {:.0f}", round, + w0, adp_w0, ctx.b_ceiling); + TrialOutcome rung = ladder_choice(state.pos, state.b, state.aniso, &state.fit); + if (rung.ran && displaces(rung.trial, state.trial)) + state = std::move(rung); + std::vector unused; + const std::string step = fmt::format("round {} step", round); + state = trajectory(std::move(state), false, unused, step.c_str()); + if (state.trial.r_work == before.trial.r_work && state.trial.r_free == before.trial.r_free) { + logger.Info("Refinement, round {}: the weights leave the model where it was", round); + break; + } + displacements(own_tls.empty() ? input_tls : own_tls); + const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE; + logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free, + kept ? "" : ", not kept"); + if (!kept) { + state = before; + break; + } } // 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every // trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what @@ -3292,7 +3421,6 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector 0 && (null_z >= TWIN_NULL_Z || (data_indicated && null_z >= TWIN_NULL_Z_DATA)); } -RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged, - const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, - Logger &logger, const RefinementSettings &settings) { +bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super) { + if (sub.order() >= super.order()) + return false; + for (const gemmi::Op &op : sub.sym_ops) + if (super.find_by_rotation(op.rot) == nullptr) + return false; + for (const gemmi::Op::Tran &cen : sub.cen_ops) + if (std::find(super.cen_ops.begin(), super.cen_ops.end(), cen) == super.cen_ops.end()) + return false; + return true; +} + +std::vector ExpandToSubgroup(const std::vector &merged, const gemmi::SpaceGroup &from, + const gemmi::SpaceGroup &to) { + const gemmi::GroupOps from_ops = from.operations(), to_ops = to.operations(); + const gemmi::ReciprocalAsu asu(&to); + std::set seen; + std::vector out; + for (const MergedReflection &r : merged) + for (const gemmi::Op &op : from_ops.sym_ops) { // the centring changes no index + const auto [h, isym] = asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), to_ops); + if (!seen.insert(h).second) + continue; + MergedReflection e = r; + e.h = h[0]; + e.k = h[1]; + e.l = h[2]; + if (isym % 2 == 0) { // the Friedel opposite + std::swap(e.I_plus, e.I_minus); + std::swap(e.sigma_plus, e.sigma_minus); + std::swap(e.F_plus, e.F_minus); + std::swap(e.sigmaF_plus, e.sigmaF_minus); + } + out.push_back(e); + } + return out; +} + +RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged_in, + const gemmi::UnitCell &cell, const gemmi::SpaceGroup &data_sg, double wavelength_A, + Logger &logger, const RefinementSettings &settings_in) { + // In the model's own group where that is a proper subgroup of the data's: a model validated in P2(1), its copies + // pseudo-symmetric, against a merge in P4(1)2(1)2 the data could not tell from it. Its copies refine under their + // own symmetry, against the merge expanded to the subgroup - in the data's group the operators the model does not + // have would overlay its copies, and R would be the null's. The data carry the larger group's symmetry exactly, + // so a twin law of the subgroup's lattice has nothing to be tested on. + const gemmi::SpaceGroup *own = placed.find_spacegroup(); + const bool subgroup = own != nullptr && ProperSubgroup(own->operations(), data_sg.operations()); + const gemmi::SpaceGroup &sg = subgroup ? *own : data_sg; + RefinementSettings settings = settings_in; + std::vector expanded; + if (subgroup) { + expanded = ExpandToSubgroup(merged_in, data_sg, sg); + settings.twin = false; + logger.Info("Refinement: in the model's {}, a subgroup of the data's {} - the copies that group would overlay " + "refined under their own symmetry, against the {} reflections expanded to {} in the subgroup; no " + "twin law tested, the data having the larger group's symmetry exactly", + sg.xhm(), data_sg.xhm(), merged_in.size(), expanded.size()); + } + const std::vector &merged = subgroup ? expanded : merged_in; // The free set closed under the lattice's twin domains once, before either arm, so that both share it. std::vector closed; int moved = 0; @@ -3613,6 +3798,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector weight_ladder = {1.0 / 1024, 1.0 / 128, 1.0 / 16}; - std::vector adp_ladder = {0.0, 1.0}; + std::vector adp_ladder = {0.0, 1.0 / 16, 1.0 / 4, 1.0}; // Then at most this many steps of the trajectory, each a set of trials from where the last one ended, int max_steps = 8; // and the final trials from where it ended, at its weights times each of these; @@ -74,6 +74,8 @@ struct RefinementSettings { // groups (the model's own, one per chain, the chains cut into TLSMD segments) with a residual B per atom, and an // anisotropic U per atom with rigid-bond and similarity restraints (false: the isotropic B, or the model's U held). bool adp_hypotheses = true; + // Rounds of the weights and the displacement model decided again from the refined model (RefineOnce, 4b.). + int rounds = 3; // A refinement against the data as twinned by each twin law the lattice admits (RefinementTwinLaws), beside the // untwinned one, and the model of whichever the evidence favours (RefineModel). bool twin = true; @@ -148,6 +150,8 @@ struct RefinementResult { bool on_gpu = false; double seconds = 0; double d_min = 0; + std::string space_group; // the group the model was refined in (xHM): the data's, or the model's own + // where that is a subgroup of it (RefineModel) int n_work = 0, n_free = 0; // The model as given, scaled to these reflections (B made isotropic; the dictionary's hydrogens riding @@ -257,6 +261,17 @@ struct RefinementResult { // As REFMAC5 keeps twin-related reflections in one set (Roversi et al. (2012) Acta Cryst. D68, 418-424) int CloseFreeSet(std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg); +// Whether every operation of `sub` is one of `super`'s (its rotation, and its centring), `super` the larger group: +// `sub` is then a group the data merged in `super` could not tell from it (a P2(1) crystal whose copies are +// pseudo-symmetric, merged in P4(1)2(1)2). +bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super); + +// `merged`, the reflections of the group `from`, as the reflections of its subgroup `to`: each with every mate that +// `from` made equivalent and `to` keeps apart, at the same amplitude, intensity and free flag (its Bijvoet pair +// swapped where the mate is the Friedel opposite). +std::vector ExpandToSubgroup(const std::vector &merged, const gemmi::SpaceGroup &from, + const gemmi::SpaceGroup &to); + // Whether the twinned refinement is taken (RefineModel): a fraction above 0, and the law ahead of its null on the // free set by 3 standard errors - or by 2 where the data's own intensity statistics read twinned as well. bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated); diff --git a/rugnux/RugnuxRefine.cpp b/rugnux/RugnuxRefine.cpp index fe217da2b..436ef657e 100644 --- a/rugnux/RugnuxRefine.cpp +++ b/rugnux/RugnuxRefine.cpp @@ -197,7 +197,9 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m out.space_group, out.free_reflections, out.free_flags_generated ? "drawn here, the file has no free set" : "column " + data.free_column, d_min_limit > 0 ? fmt::format(", to {:.2f} A", d_min_limit) : std::string()); - if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); msg != nullptr && msg->number != sg->number) + // (a model whose group is a subgroup of the data's is refined in its own, RefineModel) + if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); + msg != nullptr && msg->number != sg->number && !ProperSubgroup(msg->operations(), sg->operations())) logger.Warning("The model names space group {}, the data {}: refined in the data's", msg->xhm(), sg->xhm()); try { gemmi::Structure st = model; @@ -217,6 +219,8 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m RefinementSettings refine = settings; refine.as_given_competes = !out.free_flags_generated; out.refinement = RefineModel(st, refl, data.cell, *sg, out.wavelength_A, logger, refine); + if (!out.refinement.space_group.empty()) + out.space_group = out.refinement.space_group; } catch (const std::exception &e) { out.refinement = RefinementResult(); out.refinement.failure_reason = std::string("refinement failed: ") + e.what(); diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 7040e837c..ea07e8a7c 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -5728,7 +5728,10 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { } const PlacedModelOutput &p_final = *placed_model_out; if (refined.ok) { - result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, p_final.space_group}; + // in the group it was refined in: the model's own where that is a subgroup of the data's (RefineModel) + const gemmi::SpaceGroup *refined_in = gemmi::find_spacegroup_by_name(refined.space_group); + result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, + refined_in ? *refined_in : p_final.space_group}; result.outputs->refined_maps = refined.maps; result.outputs->blobs = refined.blobs; result.outputs->solvent = refined.solvent; diff --git a/tests/RefinementTest.cpp b/tests/RefinementTest.cpp index ada536138..92db8c573 100644 --- a/tests/RefinementTest.cpp +++ b/tests/RefinementTest.cpp @@ -966,6 +966,98 @@ TEST_CASE("Refinement_trials_run_ahead", "[Refinement]") { CHECK(same == x.size()); } +// A model whose group is a proper subgroup of the data's: a P2(1) crystal whose two residues are pseudo-symmetric by +// the 2-fold along a, its data merged in P2(1)2(1)2(1). Refined in its own group against the data expanded to it, the +// model scores as it does in the validation; refined in the data's group its contents are overlaid and it does not. +TEST_CASE("Refinement_model_in_a_subgroup_of_the_data", "[Refinement]") { + const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90); + const gemmi::SpaceGroup &sub = *gemmi::find_spacegroup_by_name("P 1 21 1"); + const gemmi::SpaceGroup &super = *gemmi::find_spacegroup_by_name("P 21 21 21"); + CHECK(ProperSubgroup(sub.operations(), super.operations())); + CHECK(!ProperSubgroup(super.operations(), sub.operations())); + CHECK(!ProperSubgroup(sub.operations(), sub.operations())); + CHECK(!ProperSubgroup(gemmi::find_spacegroup_by_name("C 1 2 1")->operations(), super.operations())); + gemmi::Structure truth = residue_crystal({"TRP"}, cell, "P 1 21 1", 5); + { + // the residue's mate by the 2-fold along a, exact: the crystal is then P 2 2(1) 1 (pseudo-P2(1)2(1)2(1)) + gemmi::Residue mate = truth.models[0].chains[0].residues[0]; + mate.seqid = gemmi::SeqId(2, ' '); + for (gemmi::Atom &a : mate.atoms) { + const gemmi::Fractional f = cell.fractionalize(a.pos); + a.pos = cell.orthogonalize(gemmi::Fractional(f.x, -f.y, -f.z)); + } + truth.models[0].chains[0].residues.push_back(mate); + } + const double d_min = 2.0; + // the data: the amplitudes of the truth in its own group, merged in the supergroup (each of its reflections the + // root mean square of the amplitudes it makes equivalent) + std::map amplitude; + { + std::vector own = asu_reflections(cell, sub, d_min); + RefinementReflections refl = PrepareRefinementReflections(own, cell, sub, d_min * 0.999); + auto cpu = MakeRefinementEngineCPU(cell, sub, d_min * 0.999, refl, 4); + cpu->SetBlur(RefinementBlur(cell, sub, d_min, 5.0)); + std::vector> f; + cpu->Fcalc(truth.models[0], f); + for (size_t m = 0; m < refl.Size(); m++) + amplitude[refl.hkl[m]] = std::abs(f[m]); + } + const gemmi::GroupOps super_ops = super.operations(), sub_ops = sub.operations(); + const gemmi::ReciprocalAsu sub_asu(&sub); + std::vector merged = asu_reflections(cell, super, d_min); + for (MergedReflection &r : merged) { + double sum = 0; + int n = 0; + for (const gemmi::Op &op : super_ops.sym_ops) { + const gemmi::Miller h = sub_asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), sub_ops).first; + if (const auto it = amplitude.find(h); it != amplitude.end()) { + sum += it->second * it->second; + n++; + } + } + r.F = n > 0 ? static_cast(std::sqrt(sum / n)) : NAN; + r.sigmaF = 0.03f * r.F + 0.01f; + } + std::erase_if(merged, [](const MergedReflection &r) { return !std::isfinite(r.F); }); + + // the expansion: every reflection of the subgroup's set comes from one of the supergroup's, with its amplitude + // and its flag, and the supergroup's set is reached from it + const std::vector expanded = ExpandToSubgroup(merged, super, sub); + CHECK(expanded.size() > 1.5 * merged.size()); + CHECK(expanded.size() <= 2 * merged.size()); + const gemmi::ReciprocalAsu super_asu(&super); + std::map source; + for (const MergedReflection &r : merged) + source[{{r.h, r.k, r.l}}] = &r; + std::set distinct; + size_t consistent = 0; + for (const MergedReflection &e : expanded) { + distinct.insert({{e.h, e.k, e.l}}); + const auto it = source.find(super_asu.to_asu({{e.h, e.k, e.l}}, super_ops).first); + consistent += it != source.end() && it->second->F == e.F && it->second->rfree_flag == e.rfree_flag; + } + CHECK(distinct.size() == expanded.size()); + CHECK(consistent == expanded.size()); + + Logger logger("RefinementTest"); + RefinementSettings settings; + settings.gpu = false; + settings.nthreads = 4; + settings.score_only = true; + settings.riding_hydrogens = false; + const RefinementResult own_group = RefineModel(truth, merged, cell, super, 1.0, logger, settings); + REQUIRE(own_group.ok); + CHECK(own_group.space_group == "P 1 21 1"); + CHECK(own_group.n_work + own_group.n_free == static_cast(expanded.size())); + CHECK(own_group.r_free_start < 0.1); + gemmi::Structure as_data = truth; + as_data.spacegroup_hm = "P 21 21 21"; + const RefinementResult data_group = RefineModel(as_data, merged, cell, super, 1.0, logger, settings); + REQUIRE(data_group.ok); + CHECK(data_group.space_group == "P 21 21 21"); + CHECK(data_group.r_free_start > 0.3); +} + // A K+ on the 4-fold axis of P4 at occupancy 0.25 (its four coinciding copies one ion), 0.2 A off the axis in the // model as given, with a water 2.8 A from it: the topology puts the ion on the axis, its contact with the water is // listed once (not once per copy) and none with itself; refinement keeps it on the axis at the occupancy given,