Refinement: a model whose group is a subgroup of the data's refined in its own group, against the merge expanded to it

A deposited P2(1) model whose eight copies are pseudo-symmetric, validated against a merge in P4(1)2(1)2 (the data
cannot tell the two apart): the validation puts the model through a change of basis into its own group and scores
it there, but the refinement was handed the model under the data's group. The operators the model does not have
overlaid its copies four times over, the solvent and the overall B ran to their bounds, and the "refined" R-free was
the null's (0.33 unrefined -> 0.60 refined; p_model.pdb said P 21 1 1, p_refined_model.pdb P 41 21 2).

RefineModel now refines in the placed model's group where that is a proper subgroup of the data's (every rotation
and centring of the model's group among the data's, the data's group the larger): the merge is expanded to the
subgroup - each reflection standing for every mate the data's group made equivalent, at the merged amplitude, with
its free flag, the Bijvoet pair swapped for a Friedel mate - and no twin law is tested, the data carrying the larger
symmetry exactly. RefinementResult::space_group names the group refined in; the run's refined model and
`--mode refine`'s output are written in it. On the case above the refinement now starts where the validation ended
(as given 0.322 against 75965 reflections in P 21 1 1) and ends at 0.320.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This commit is contained in:
leonarski_fandClaude Fable 5.1 committed 2026-11-14 12:18:49 +01:00
1 parent e0e50f83a9
commit 044dc4deb2
8 files changed
+191 -9

No files matched your search

+12 -2
View File
@@ -27,8 +27,18 @@ statistics and its map coefficients go out; the restraint dictionary is compiled
### 15.1 What is refined, and in what order
The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed,
and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within
100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
and every B inside the range the refinement allows (2 Ų up to twice the model's largest B, within 100-300 Ų;
a model that arrives with B above 300 Ų - a structure refined at 4-5 Š- gets a ceiling of 1.5 times its
largest B instead, so that it is not flattened; a later round of the weights (§15.1, 4) raises the ceiling from
the refined model).
The model is refined in the data's space group - except where its own group is a proper subgroup of it: a
P2₁ model whose copies are pseudo-symmetric, validated in its own group against a merge in P4₁2₁2 that the
data could not tell from P2₁. Under the data's group the operators the model does not have would overlay its
copies and R would be the null's, so it is refined in its own group, against the merge expanded to that group
(every reflection the data's group made equivalent, at the merged amplitude and with the same free flag), and
no twin law is tested, the data carrying the larger symmetry exactly. The refined model is written in the group
it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
+2 -1
View File
@@ -184,7 +184,8 @@ The columns are found, not named - amplitudes with their sigmas (F/SIGF,
FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by
French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set,
or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none.
The model is taken as placed in the data's cell, and refined in the data's space group.
The model is taken as placed in the data's cell, and refined in the data's space group (in its own where that is a
subgroup of the data's, against the data expanded to it).
| Option | Description |
| --- | --- |
+2 -1
View File
@@ -321,7 +321,8 @@ rugnux --mode refine --model model.cif -o refined data.mtz
```
The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by
content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group. The
content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its
own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The
columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and
map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN,
I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from
@@ -3659,9 +3659,66 @@ bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated) {
return twin_fraction > 0 && (null_z >= TWIN_NULL_Z || (data_indicated && null_z >= TWIN_NULL_Z_DATA));
}
RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<MergedReflection> &merged,
const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A,
Logger &logger, const RefinementSettings &settings) {
bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super) {
if (sub.order() >= super.order())
return false;
for (const gemmi::Op &op : sub.sym_ops)
if (super.find_by_rotation(op.rot) == nullptr)
return false;
for (const gemmi::Op::Tran &cen : sub.cen_ops)
if (std::find(super.cen_ops.begin(), super.cen_ops.end(), cen) == super.cen_ops.end())
return false;
return true;
}
std::vector<MergedReflection> ExpandToSubgroup(const std::vector<MergedReflection> &merged, const gemmi::SpaceGroup &from,
const gemmi::SpaceGroup &to) {
const gemmi::GroupOps from_ops = from.operations(), to_ops = to.operations();
const gemmi::ReciprocalAsu asu(&to);
std::set<gemmi::Miller> seen;
std::vector<MergedReflection> out;
for (const MergedReflection &r : merged)
for (const gemmi::Op &op : from_ops.sym_ops) { // the centring changes no index
const auto [h, isym] = asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), to_ops);
if (!seen.insert(h).second)
continue;
MergedReflection e = r;
e.h = h[0];
e.k = h[1];
e.l = h[2];
if (isym % 2 == 0) { // the Friedel opposite
std::swap(e.I_plus, e.I_minus);
std::swap(e.sigma_plus, e.sigma_minus);
std::swap(e.F_plus, e.F_minus);
std::swap(e.sigmaF_plus, e.sigmaF_minus);
}
out.push_back(e);
}
return out;
}
RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<MergedReflection> &merged_in,
const gemmi::UnitCell &cell, const gemmi::SpaceGroup &data_sg, double wavelength_A,
Logger &logger, const RefinementSettings &settings_in) {
// In the model's own group where that is a proper subgroup of the data's: a model validated in P2(1), its copies
// pseudo-symmetric, against a merge in P4(1)2(1)2 the data could not tell from it. Its copies refine under their
// own symmetry, against the merge expanded to the subgroup - in the data's group the operators the model does not
// have would overlay its copies, and R would be the null's. The data carry the larger group's symmetry exactly,
// so a twin law of the subgroup's lattice has nothing to be tested on.
const gemmi::SpaceGroup *own = placed.find_spacegroup();
const bool subgroup = own != nullptr && ProperSubgroup(own->operations(), data_sg.operations());
const gemmi::SpaceGroup &sg = subgroup ? *own : data_sg;
RefinementSettings settings = settings_in;
std::vector<MergedReflection> expanded;
if (subgroup) {
expanded = ExpandToSubgroup(merged_in, data_sg, sg);
settings.twin = false;
logger.Info("Refinement: in the model's {}, a subgroup of the data's {} - the copies that group would overlay "
"refined under their own symmetry, against the {} reflections expanded to {} in the subgroup; no "
"twin law tested, the data having the larger group's symmetry exactly",
sg.xhm(), data_sg.xhm(), merged_in.size(), expanded.size());
}
const std::vector<MergedReflection> &merged = subgroup ? expanded : merged_in;
// The free set closed under the lattice's twin domains once, before either arm, so that both share it.
std::vector<MergedReflection> closed;
int moved = 0;
@@ -3727,6 +3784,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
twin_logger.ReplayInto(logger);
}
result.twin_free_closed = moved;
result.space_group = sg.xhm();
result.map.reset();
return result;
}
@@ -148,6 +148,8 @@ struct RefinementResult {
bool on_gpu = false;
double seconds = 0;
double d_min = 0;
std::string space_group; // the group the model was refined in (xHM): the data's, or the model's own
// where that is a subgroup of it (RefineModel)
int n_work = 0, n_free = 0;
// The model as given, scaled to these reflections (B made isotropic; the dictionary's hydrogens riding
@@ -257,6 +259,17 @@ struct RefinementResult {
// As REFMAC5 keeps twin-related reflections in one set (Roversi et al. (2012) Acta Cryst. D68, 418-424)
int CloseFreeSet(std::vector<MergedReflection> &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg);
// Whether every operation of `sub` is one of `super`'s (its rotation, and its centring), `super` the larger group:
// `sub` is then a group the data merged in `super` could not tell from it (a P2(1) crystal whose copies are
// pseudo-symmetric, merged in P4(1)2(1)2).
bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super);
// `merged`, the reflections of the group `from`, as the reflections of its subgroup `to`: each with every mate that
// `from` made equivalent and `to` keeps apart, at the same amplitude, intensity and free flag (its Bijvoet pair
// swapped where the mate is the Friedel opposite).
std::vector<MergedReflection> ExpandToSubgroup(const std::vector<MergedReflection> &merged, const gemmi::SpaceGroup &from,
const gemmi::SpaceGroup &to);
// Whether the twinned refinement is taken (RefineModel): a fraction above 0, and the law ahead of its null on the
// free set by 3 standard errors - or by 2 where the data's own intensity statistics read twinned as well.
bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated);
+5 -1
View File
@@ -197,7 +197,9 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m
out.space_group, out.free_reflections,
out.free_flags_generated ? "drawn here, the file has no free set" : "column " + data.free_column,
d_min_limit > 0 ? fmt::format(", to {:.2f} A", d_min_limit) : std::string());
if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); msg != nullptr && msg->number != sg->number)
// (a model whose group is a subgroup of the data's is refined in its own, RefineModel)
if (const gemmi::SpaceGroup *msg = model.find_spacegroup();
msg != nullptr && msg->number != sg->number && !ProperSubgroup(msg->operations(), sg->operations()))
logger.Warning("The model names space group {}, the data {}: refined in the data's", msg->xhm(), sg->xhm());
try {
gemmi::Structure st = model;
@@ -217,6 +219,8 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m
RefinementSettings refine = settings;
refine.as_given_competes = !out.free_flags_generated;
out.refinement = RefineModel(st, refl, data.cell, *sg, out.wavelength_A, logger, refine);
if (!out.refinement.space_group.empty())
out.space_group = out.refinement.space_group;
} catch (const std::exception &e) {
out.refinement = RefinementResult();
out.refinement.failure_reason = std::string("refinement failed: ") + e.what();
+4 -1
View File
@@ -5728,7 +5728,10 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) {
}
const PlacedModelOutput &p_final = *placed_model_out;
if (refined.ok) {
result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, p_final.space_group};
// in the group it was refined in: the model's own where that is a subgroup of the data's (RefineModel)
const gemmi::SpaceGroup *refined_in = gemmi::find_spacegroup_by_name(refined.space_group);
result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell,
refined_in ? *refined_in : p_final.space_group};
result.outputs->refined_maps = refined.maps;
result.outputs->blobs = refined.blobs;
result.outputs->solvent = refined.solvent;
+92
View File
@@ -966,6 +966,98 @@ TEST_CASE("Refinement_trials_run_ahead", "[Refinement]") {
CHECK(same == x.size());
}
// A model whose group is a proper subgroup of the data's: a P2(1) crystal whose two residues are pseudo-symmetric by
// the 2-fold along a, its data merged in P2(1)2(1)2(1). Refined in its own group against the data expanded to it, the
// model scores as it does in the validation; refined in the data's group its contents are overlaid and it does not.
TEST_CASE("Refinement_model_in_a_subgroup_of_the_data", "[Refinement]") {
const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90);
const gemmi::SpaceGroup &sub = *gemmi::find_spacegroup_by_name("P 1 21 1");
const gemmi::SpaceGroup &super = *gemmi::find_spacegroup_by_name("P 21 21 21");
CHECK(ProperSubgroup(sub.operations(), super.operations()));
CHECK(!ProperSubgroup(super.operations(), sub.operations()));
CHECK(!ProperSubgroup(sub.operations(), sub.operations()));
CHECK(!ProperSubgroup(gemmi::find_spacegroup_by_name("C 1 2 1")->operations(), super.operations()));
gemmi::Structure truth = residue_crystal({"TRP"}, cell, "P 1 21 1", 5);
{
// the residue's mate by the 2-fold along a, exact: the crystal is then P 2 2(1) 1 (pseudo-P2(1)2(1)2(1))
gemmi::Residue mate = truth.models[0].chains[0].residues[0];
mate.seqid = gemmi::SeqId(2, ' ');
for (gemmi::Atom &a : mate.atoms) {
const gemmi::Fractional f = cell.fractionalize(a.pos);
a.pos = cell.orthogonalize(gemmi::Fractional(f.x, -f.y, -f.z));
}
truth.models[0].chains[0].residues.push_back(mate);
}
const double d_min = 2.0;
// the data: the amplitudes of the truth in its own group, merged in the supergroup (each of its reflections the
// root mean square of the amplitudes it makes equivalent)
std::map<gemmi::Miller, double> amplitude;
{
std::vector<MergedReflection> own = asu_reflections(cell, sub, d_min);
RefinementReflections refl = PrepareRefinementReflections(own, cell, sub, d_min * 0.999);
auto cpu = MakeRefinementEngineCPU(cell, sub, d_min * 0.999, refl, 4);
cpu->SetBlur(RefinementBlur(cell, sub, d_min, 5.0));
std::vector<std::complex<float>> f;
cpu->Fcalc(truth.models[0], f);
for (size_t m = 0; m < refl.Size(); m++)
amplitude[refl.hkl[m]] = std::abs(f[m]);
}
const gemmi::GroupOps super_ops = super.operations(), sub_ops = sub.operations();
const gemmi::ReciprocalAsu sub_asu(&sub);
std::vector<MergedReflection> merged = asu_reflections(cell, super, d_min);
for (MergedReflection &r : merged) {
double sum = 0;
int n = 0;
for (const gemmi::Op &op : super_ops.sym_ops) {
const gemmi::Miller h = sub_asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), sub_ops).first;
if (const auto it = amplitude.find(h); it != amplitude.end()) {
sum += it->second * it->second;
n++;
}
}
r.F = n > 0 ? static_cast<float>(std::sqrt(sum / n)) : NAN;
r.sigmaF = 0.03f * r.F + 0.01f;
}
std::erase_if(merged, [](const MergedReflection &r) { return !std::isfinite(r.F); });
// the expansion: every reflection of the subgroup's set comes from one of the supergroup's, with its amplitude
// and its flag, and the supergroup's set is reached from it
const std::vector<MergedReflection> expanded = ExpandToSubgroup(merged, super, sub);
CHECK(expanded.size() > 1.5 * merged.size());
CHECK(expanded.size() <= 2 * merged.size());
const gemmi::ReciprocalAsu super_asu(&super);
std::map<gemmi::Miller, const MergedReflection *> source;
for (const MergedReflection &r : merged)
source[{{r.h, r.k, r.l}}] = &r;
std::set<gemmi::Miller> distinct;
size_t consistent = 0;
for (const MergedReflection &e : expanded) {
distinct.insert({{e.h, e.k, e.l}});
const auto it = source.find(super_asu.to_asu({{e.h, e.k, e.l}}, super_ops).first);
consistent += it != source.end() && it->second->F == e.F && it->second->rfree_flag == e.rfree_flag;
}
CHECK(distinct.size() == expanded.size());
CHECK(consistent == expanded.size());
Logger logger("RefinementTest");
RefinementSettings settings;
settings.gpu = false;
settings.nthreads = 4;
settings.score_only = true;
settings.riding_hydrogens = false;
const RefinementResult own_group = RefineModel(truth, merged, cell, super, 1.0, logger, settings);
REQUIRE(own_group.ok);
CHECK(own_group.space_group == "P 1 21 1");
CHECK(own_group.n_work + own_group.n_free == static_cast<int>(expanded.size()));
CHECK(own_group.r_free_start < 0.1);
gemmi::Structure as_data = truth;
as_data.spacegroup_hm = "P 21 21 21";
const RefinementResult data_group = RefineModel(as_data, merged, cell, super, 1.0, logger, settings);
REQUIRE(data_group.ok);
CHECK(data_group.space_group == "P 21 21 21");
CHECK(data_group.r_free_start > 0.3);
}
// A K+ on the 4-fold axis of P4 at occupancy 0.25 (its four coinciding copies one ion), 0.2 A off the axis in the
// model as given, with a water 2.8 A from it: the topology puts the ion on the axis, its contact with the water is
// listed once (not once per copy) and none with itself; refinement keeps it on the axis at the occupancy given,