diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 90257354f..ca80138b7 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -27,8 +27,18 @@ statistics and its map coefficients go out; the restraint dictionary is compiled ### 15.1 What is refined, and in what order The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed, -and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within -100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS +and every B inside the range the refinement allows (2 Ų up to twice the model's largest B, within 100-300 Ų; +a model that arrives with B above 300 Ų - a structure refined at 4-5 Å - gets a ceiling of 1.5 times its +largest B instead, so that it is not flattened; a later round of the weights (§15.1, 4) raises the ceiling from +the refined model). + +The model is refined in the data's space group - except where its own group is a proper subgroup of it: a +P2₁ model whose copies are pseudo-symmetric, validated in its own group against a merge in P4₁2₁2 that the +data could not tell from P2₁. Under the data's group the operators the model does not have would overlay its +copies and R would be the null's, so it is refined in its own group, against the merge expanded to that group +(every reflection the data's group made equivalent, at the merged amplitude and with the same free flag), and +no twin law is tested, the data carrying the larger symmetry exactly. The refined model is written in the group +it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index f263db765..7da5065ff 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -184,7 +184,8 @@ The columns are found, not named - amplitudes with their sigmas (F/SIGF, FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none. -The model is taken as placed in the data's cell, and refined in the data's space group. +The model is taken as placed in the data's cell, and refined in the data's space group (in its own where that is a +subgroup of the data's, against the data expanded to it). | Option | Description | | --- | --- | diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index 34915dea2..de168046a 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -321,7 +321,8 @@ rugnux --mode refine --model model.cif -o refined data.mtz ``` The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by -content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group. The +content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its +own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index b2dc4f611..8f94870b3 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -3659,9 +3659,66 @@ bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated) { return twin_fraction > 0 && (null_z >= TWIN_NULL_Z || (data_indicated && null_z >= TWIN_NULL_Z_DATA)); } -RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged, - const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, - Logger &logger, const RefinementSettings &settings) { +bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super) { + if (sub.order() >= super.order()) + return false; + for (const gemmi::Op &op : sub.sym_ops) + if (super.find_by_rotation(op.rot) == nullptr) + return false; + for (const gemmi::Op::Tran &cen : sub.cen_ops) + if (std::find(super.cen_ops.begin(), super.cen_ops.end(), cen) == super.cen_ops.end()) + return false; + return true; +} + +std::vector ExpandToSubgroup(const std::vector &merged, const gemmi::SpaceGroup &from, + const gemmi::SpaceGroup &to) { + const gemmi::GroupOps from_ops = from.operations(), to_ops = to.operations(); + const gemmi::ReciprocalAsu asu(&to); + std::set seen; + std::vector out; + for (const MergedReflection &r : merged) + for (const gemmi::Op &op : from_ops.sym_ops) { // the centring changes no index + const auto [h, isym] = asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), to_ops); + if (!seen.insert(h).second) + continue; + MergedReflection e = r; + e.h = h[0]; + e.k = h[1]; + e.l = h[2]; + if (isym % 2 == 0) { // the Friedel opposite + std::swap(e.I_plus, e.I_minus); + std::swap(e.sigma_plus, e.sigma_minus); + std::swap(e.F_plus, e.F_minus); + std::swap(e.sigmaF_plus, e.sigmaF_minus); + } + out.push_back(e); + } + return out; +} + +RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged_in, + const gemmi::UnitCell &cell, const gemmi::SpaceGroup &data_sg, double wavelength_A, + Logger &logger, const RefinementSettings &settings_in) { + // In the model's own group where that is a proper subgroup of the data's: a model validated in P2(1), its copies + // pseudo-symmetric, against a merge in P4(1)2(1)2 the data could not tell from it. Its copies refine under their + // own symmetry, against the merge expanded to the subgroup - in the data's group the operators the model does not + // have would overlay its copies, and R would be the null's. The data carry the larger group's symmetry exactly, + // so a twin law of the subgroup's lattice has nothing to be tested on. + const gemmi::SpaceGroup *own = placed.find_spacegroup(); + const bool subgroup = own != nullptr && ProperSubgroup(own->operations(), data_sg.operations()); + const gemmi::SpaceGroup &sg = subgroup ? *own : data_sg; + RefinementSettings settings = settings_in; + std::vector expanded; + if (subgroup) { + expanded = ExpandToSubgroup(merged_in, data_sg, sg); + settings.twin = false; + logger.Info("Refinement: in the model's {}, a subgroup of the data's {} - the copies that group would overlay " + "refined under their own symmetry, against the {} reflections expanded to {} in the subgroup; no " + "twin law tested, the data having the larger group's symmetry exactly", + sg.xhm(), data_sg.xhm(), merged_in.size(), expanded.size()); + } + const std::vector &merged = subgroup ? expanded : merged_in; // The free set closed under the lattice's twin domains once, before either arm, so that both share it. std::vector closed; int moved = 0; @@ -3727,6 +3784,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg); +// Whether every operation of `sub` is one of `super`'s (its rotation, and its centring), `super` the larger group: +// `sub` is then a group the data merged in `super` could not tell from it (a P2(1) crystal whose copies are +// pseudo-symmetric, merged in P4(1)2(1)2). +bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super); + +// `merged`, the reflections of the group `from`, as the reflections of its subgroup `to`: each with every mate that +// `from` made equivalent and `to` keeps apart, at the same amplitude, intensity and free flag (its Bijvoet pair +// swapped where the mate is the Friedel opposite). +std::vector ExpandToSubgroup(const std::vector &merged, const gemmi::SpaceGroup &from, + const gemmi::SpaceGroup &to); + // Whether the twinned refinement is taken (RefineModel): a fraction above 0, and the law ahead of its null on the // free set by 3 standard errors - or by 2 where the data's own intensity statistics read twinned as well. bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated); diff --git a/rugnux/RugnuxRefine.cpp b/rugnux/RugnuxRefine.cpp index fe217da2b..436ef657e 100644 --- a/rugnux/RugnuxRefine.cpp +++ b/rugnux/RugnuxRefine.cpp @@ -197,7 +197,9 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m out.space_group, out.free_reflections, out.free_flags_generated ? "drawn here, the file has no free set" : "column " + data.free_column, d_min_limit > 0 ? fmt::format(", to {:.2f} A", d_min_limit) : std::string()); - if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); msg != nullptr && msg->number != sg->number) + // (a model whose group is a subgroup of the data's is refined in its own, RefineModel) + if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); + msg != nullptr && msg->number != sg->number && !ProperSubgroup(msg->operations(), sg->operations())) logger.Warning("The model names space group {}, the data {}: refined in the data's", msg->xhm(), sg->xhm()); try { gemmi::Structure st = model; @@ -217,6 +219,8 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m RefinementSettings refine = settings; refine.as_given_competes = !out.free_flags_generated; out.refinement = RefineModel(st, refl, data.cell, *sg, out.wavelength_A, logger, refine); + if (!out.refinement.space_group.empty()) + out.space_group = out.refinement.space_group; } catch (const std::exception &e) { out.refinement = RefinementResult(); out.refinement.failure_reason = std::string("refinement failed: ") + e.what(); diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 7040e837c..ea07e8a7c 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -5728,7 +5728,10 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { } const PlacedModelOutput &p_final = *placed_model_out; if (refined.ok) { - result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, p_final.space_group}; + // in the group it was refined in: the model's own where that is a subgroup of the data's (RefineModel) + const gemmi::SpaceGroup *refined_in = gemmi::find_spacegroup_by_name(refined.space_group); + result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, + refined_in ? *refined_in : p_final.space_group}; result.outputs->refined_maps = refined.maps; result.outputs->blobs = refined.blobs; result.outputs->solvent = refined.solvent; diff --git a/tests/RefinementTest.cpp b/tests/RefinementTest.cpp index ada536138..92db8c573 100644 --- a/tests/RefinementTest.cpp +++ b/tests/RefinementTest.cpp @@ -966,6 +966,98 @@ TEST_CASE("Refinement_trials_run_ahead", "[Refinement]") { CHECK(same == x.size()); } +// A model whose group is a proper subgroup of the data's: a P2(1) crystal whose two residues are pseudo-symmetric by +// the 2-fold along a, its data merged in P2(1)2(1)2(1). Refined in its own group against the data expanded to it, the +// model scores as it does in the validation; refined in the data's group its contents are overlaid and it does not. +TEST_CASE("Refinement_model_in_a_subgroup_of_the_data", "[Refinement]") { + const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90); + const gemmi::SpaceGroup &sub = *gemmi::find_spacegroup_by_name("P 1 21 1"); + const gemmi::SpaceGroup &super = *gemmi::find_spacegroup_by_name("P 21 21 21"); + CHECK(ProperSubgroup(sub.operations(), super.operations())); + CHECK(!ProperSubgroup(super.operations(), sub.operations())); + CHECK(!ProperSubgroup(sub.operations(), sub.operations())); + CHECK(!ProperSubgroup(gemmi::find_spacegroup_by_name("C 1 2 1")->operations(), super.operations())); + gemmi::Structure truth = residue_crystal({"TRP"}, cell, "P 1 21 1", 5); + { + // the residue's mate by the 2-fold along a, exact: the crystal is then P 2 2(1) 1 (pseudo-P2(1)2(1)2(1)) + gemmi::Residue mate = truth.models[0].chains[0].residues[0]; + mate.seqid = gemmi::SeqId(2, ' '); + for (gemmi::Atom &a : mate.atoms) { + const gemmi::Fractional f = cell.fractionalize(a.pos); + a.pos = cell.orthogonalize(gemmi::Fractional(f.x, -f.y, -f.z)); + } + truth.models[0].chains[0].residues.push_back(mate); + } + const double d_min = 2.0; + // the data: the amplitudes of the truth in its own group, merged in the supergroup (each of its reflections the + // root mean square of the amplitudes it makes equivalent) + std::map amplitude; + { + std::vector own = asu_reflections(cell, sub, d_min); + RefinementReflections refl = PrepareRefinementReflections(own, cell, sub, d_min * 0.999); + auto cpu = MakeRefinementEngineCPU(cell, sub, d_min * 0.999, refl, 4); + cpu->SetBlur(RefinementBlur(cell, sub, d_min, 5.0)); + std::vector> f; + cpu->Fcalc(truth.models[0], f); + for (size_t m = 0; m < refl.Size(); m++) + amplitude[refl.hkl[m]] = std::abs(f[m]); + } + const gemmi::GroupOps super_ops = super.operations(), sub_ops = sub.operations(); + const gemmi::ReciprocalAsu sub_asu(&sub); + std::vector merged = asu_reflections(cell, super, d_min); + for (MergedReflection &r : merged) { + double sum = 0; + int n = 0; + for (const gemmi::Op &op : super_ops.sym_ops) { + const gemmi::Miller h = sub_asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), sub_ops).first; + if (const auto it = amplitude.find(h); it != amplitude.end()) { + sum += it->second * it->second; + n++; + } + } + r.F = n > 0 ? static_cast(std::sqrt(sum / n)) : NAN; + r.sigmaF = 0.03f * r.F + 0.01f; + } + std::erase_if(merged, [](const MergedReflection &r) { return !std::isfinite(r.F); }); + + // the expansion: every reflection of the subgroup's set comes from one of the supergroup's, with its amplitude + // and its flag, and the supergroup's set is reached from it + const std::vector expanded = ExpandToSubgroup(merged, super, sub); + CHECK(expanded.size() > 1.5 * merged.size()); + CHECK(expanded.size() <= 2 * merged.size()); + const gemmi::ReciprocalAsu super_asu(&super); + std::map source; + for (const MergedReflection &r : merged) + source[{{r.h, r.k, r.l}}] = &r; + std::set distinct; + size_t consistent = 0; + for (const MergedReflection &e : expanded) { + distinct.insert({{e.h, e.k, e.l}}); + const auto it = source.find(super_asu.to_asu({{e.h, e.k, e.l}}, super_ops).first); + consistent += it != source.end() && it->second->F == e.F && it->second->rfree_flag == e.rfree_flag; + } + CHECK(distinct.size() == expanded.size()); + CHECK(consistent == expanded.size()); + + Logger logger("RefinementTest"); + RefinementSettings settings; + settings.gpu = false; + settings.nthreads = 4; + settings.score_only = true; + settings.riding_hydrogens = false; + const RefinementResult own_group = RefineModel(truth, merged, cell, super, 1.0, logger, settings); + REQUIRE(own_group.ok); + CHECK(own_group.space_group == "P 1 21 1"); + CHECK(own_group.n_work + own_group.n_free == static_cast(expanded.size())); + CHECK(own_group.r_free_start < 0.1); + gemmi::Structure as_data = truth; + as_data.spacegroup_hm = "P 21 21 21"; + const RefinementResult data_group = RefineModel(as_data, merged, cell, super, 1.0, logger, settings); + REQUIRE(data_group.ok); + CHECK(data_group.space_group == "P 21 21 21"); + CHECK(data_group.r_free_start > 0.3); +} + // A K+ on the 4-fold axis of P4 at occupancy 0.25 (its four coinciding copies one ion), 0.2 A off the axis in the // model as given, with a water 2.8 A from it: the topology puts the ion on the axis, its contact with the water is // listed once (not once per copy) and none with itself; refinement keeps it on the axis at the occupancy given,