Merge rc187-refine-s1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-14 13:56:05 +01:00
commit 7d6bbb6434
8 files changed
+367 -41

No files matched your search

+30 -6
View File
@@ -27,8 +27,18 @@ statistics and its map coefficients go out; the restraint dictionary is compiled
### 15.1 What is refined, and in what order
The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed,
and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within
100-300 Ų). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
and every B inside the range the refinement allows (2 Ų up to twice the model's largest B, within 100-300 Ų;
a model that arrives with B above 300 Ų - a structure refined at 4-5 Š- gets a ceiling of 1.5 times its
largest B instead, so that it is not flattened; a later round of the weights (§15.1, 4) raises the ceiling from
the refined model).
The model is refined in the data's space group - except where its own group is a proper subgroup of it: a
P2₁ model whose copies are pseudo-symmetric, validated in its own group against a merge in P4₁2₁2 that the
data could not tell from P2₁. Under the data's group the operators the model does not have would overlay its
copies and R would be the null's, so it is refined in its own group, against the merge expanded to that group
(every reflection the data's group made equivalent, at the merged amplitude and with the same free flag), and
no twin law is tested, the data carrying the larger symmetry exactly. The refined model is written in the group
it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
@@ -297,7 +307,11 @@ geometry - is kept). The trials
are:
1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors
held and refined at $w_{B,0}$ - six trials.
held and refined at $w_{B,0} \times \{1/16, 1/4, 1\}$ - twelve trials (the B factors are refined first at
the middle rung only, and at the others where that comes within two ties of holding them there). The lower
B weights are for a model whose B factors arrive far from where they refine - a prediction's pLDDT - whose
balanced weight is not the one its B wants: they took 0.002-0.010 off R-free on five such starts, and
nothing off eight deposited models.
2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights
as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at
one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and
@@ -309,9 +323,9 @@ are:
was never kept on 55 refinements, and taking it away changed no choice).
4. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U
held), each trial as long as a step on the positions where the weights ended, held - only the displacement
parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with
the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held
positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain
parameters refine, so the trials differ in the displacement model alone (where the geometry is out of bounds
there, which held positions cannot mend, each trial refines the positions too, a macro-cycle longer than a
step): TLS groups with a residual B - the model's own, one per chain
of at least 50 refinable atoms, and the chains cut into 4, 8 and 16 contiguous segments of at least 10 residues
as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r),
to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U
@@ -327,6 +341,16 @@ are:
`REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its
solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no
TLS group holds a water.
Then the weights and the displacement model are **decided again** from the model as it now is, up to twice: the
B ceiling and the balanced weights from its B factors, the ladder from it with its displacements as they are, the
trajectory, and the displacement hypotheses from where that ends. The first choice took its weights with the
B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT,
the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$),
the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the
refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers
R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the
hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder
and one step. No round runs where the refinement does not displace the model as given.
5. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the
rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as
it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never
+2 -1
View File
@@ -184,7 +184,8 @@ The columns are found, not named - amplitudes with their sigmas (F/SIGF,
FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by
French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set,
or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none.
The model is taken as placed in the data's cell, and refined in the data's space group.
The model is taken as placed in the data's cell, and refined in the data's space group (in its own where that is a
subgroup of the data's, against the data expanded to it).
| Option | Description |
| --- | --- |
+2 -1
View File
@@ -321,7 +321,8 @@ rugnux --mode refine --model model.cif -o refined data.mtz
```
The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by
content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group. The
content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its
own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The
columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and
map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN,
I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from
@@ -52,6 +52,12 @@ namespace {
constexpr double B_FLOOR = 2.0; // A^2: no atom's B goes below it
// nor above twice the model's largest B, within these bounds (the GPU's gather is reserved for it)
constexpr double B_CEILING_MIN = 100.0, B_CEILING_MAX = 300.0;
// The ceiling of B for a model whose highest B is `b_max`: twice it, within the bounds - the bounds lifted for a
// model that comes with B above them (a structure refined at 4-5 A, B 400-600 A^2), which the ceiling would
// otherwise flatten (the gather is reserved for it).
double BCeiling(double b_max) {
return std::clamp(2 * b_max, B_CEILING_MIN, std::max(B_CEILING_MAX, 1.5 * b_max));
}
// A B tensor (8 pi^2 U) inside the bounds: its smallest eigenvalue raised to `floor` by a multiple of I, then the
// whole tensor scaled down so that its largest is at most `ceiling`. The ceiling comes last because the GPU's
@@ -78,6 +84,8 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0;
constexpr int SOLVENT_ITERATIONS = 10;
// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken.
constexpr double R_FREE_TIE = 0.001;
// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each
// after it from the model the one before ended with.
// The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this,
// relative; the most a position (A) or a B (A^2) moves in the first trial of a line search.
constexpr double MINIMISER_F_TOLERANCE = 1e-7;
@@ -138,7 +146,7 @@ gemmi::Structure PrepareModel(const gemmi::Structure &placed, const gemmi::UnitC
a.charge = ScatteringCharge(r, a);
if (a.aniso.nonzero())
a.b_iso = static_cast<float>(gemmi::u_to_b() * a.aniso.trace() / 3.0);
a.b_iso = static_cast<float>(std::clamp<double>(a.b_iso, B_FLOOR, B_CEILING_MAX));
a.b_iso = static_cast<float>(std::max<double>(a.b_iso, B_FLOOR));
}
}
gemmi::remove_empty_children(st);
@@ -263,8 +271,8 @@ struct Context {
continue;
// its widest axis no wider than the refinement allows
const double top = AnisoRadiusB(at.aniso.scaled(gemmi::u_to_b()));
if (top > B_CEILING_MAX) {
at.aniso = at.aniso.scaled(static_cast<float>(B_CEILING_MAX / top));
if (top > b_ceiling) {
at.aniso = at.aniso.scaled(static_cast<float>(b_ceiling / top));
at.b_iso = static_cast<float>(gemmi::u_to_b() * at.aniso.trace() / 3.0);
}
const double t = at.aniso.trace() / 3.0;
@@ -1446,7 +1454,29 @@ void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) {
for (size_t d = first; d < a.size(); d++)
result.twin_fraction_null += a[d];
}
const std::vector<std::complex<float>> f_null = TwinnedModel(null, rows, ctx.fcalc);
// TwinnedModel with each unrelated mate on its own scale k_total: TwinnedModel puts a mate on the scale of the
// reflection it is the mate of, as the twin domain's reflection at the same spot is measured, but an unrelated
// mate lies in another direction, where the anisotropic scale can differ many times over (22 times in amplitude
// at 2 A, on a model whose U carried the anisotropy opposite to k_total), and on the reflection's scale it put
// the null 0.06 in R-free above the untwinned refinement - a law "ahead" of its null by 7.9 standard errors at a
// fraction of 0.04.
std::vector<std::complex<float>> f_null(n);
double own_fraction = 1.0;
for (float a : rows.twin_fractions)
own_fraction -= a;
for (size_t m = 0; m < n; m++) {
const std::complex<double> u = std::complex<double>(ctx.fcalc[m]) + std::complex<double>(rows.fmask[m]);
double i_model = own_fraction * rows.k_total[m] * rows.k_total[m] * std::norm(u);
for (size_t d = 0; d < null.Laws(); d++) {
const int j = null.mates[d][m];
const double k = d < first ? rows.k_total[m] : rows.k_total[j];
i_model += rows.twin_fractions[d] * k * k *
std::norm(std::complex<double>(ctx.fcalc[j]) + std::complex<double>(rows.fmask[j]));
}
const double amplitude = std::sqrt(std::max(i_model, 0.0));
f_null[m] = std::norm(u) > 0 ? std::complex<float>(u * (amplitude / std::abs(u)))
: std::complex<float>(static_cast<float>(amplitude), 0);
}
result.twin_r_free_null = ComputeRFactors(null, f_null).r_free;
// How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin||
// over its standard error. The members of a free twin group share their F_calc, so their terms are summed into
@@ -1828,10 +1858,9 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
t.within_bounds = t.geometry.rmsd_bonds <= MAX_RMSD_BONDS && t.geometry.rmsd_angles <= MAX_RMSD_ANGLES;
if (ncs != nullptr && !ncs->Empty())
t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB());
if (hyp != nullptr) {
if (hyp != nullptr)
t.adp = hyp->name;
t.nll_free = ctx.Score().nll_free;
}
t.nll_free = ctx.Score().nll_free;
out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, "
"angles {:.2f} deg{}{}",
hyp != nullptr ? hyp->name + ", " : std::string(), w,
@@ -1840,8 +1869,7 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
ncs == nullptr || ncs->Empty() ? std::string()
: ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd)
: fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd),
rf.r_work, rf.r_free,
hyp != nullptr ? fmt::format(", -LL free {:.1f}", t.nll_free) : std::string(),
rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free),
t.geometry.rmsd_bonds, t.geometry.rmsd_angles,
t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal)
: std::string(),
@@ -2283,7 +2311,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
for (const TlsBody &body : tls_bodies)
for (int i : body.atoms)
b_max = std::max(b_max, ctx.atoms[i]->b_iso + gemmi::u_to_b() * body.U(ctx.atoms[i]->pos).trace() / 3.0);
ctx.b_ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX);
ctx.b_ceiling = BCeiling(b_max);
if (ctx.atoms.empty()) {
result.failure_reason = "the model has no atoms";
return result;
@@ -2664,7 +2692,16 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// never kept at any other).
// The B factors are refined first at the middle rung only, and at the others where that comes within two
// ties of holding them there: on 364 refinements of the battery B held was kept by 281, and the rule
// changed the rung kept on 4 (all within 0.006 in R-free).
// changed the rung kept on 4 (all within 0.006 in R-free). Their X-ray weight is tried at the balanced
// one and at a quarter and a sixteenth of it: a model whose B factors arrive far from where they refine
// (a prediction's pLDDT) balances a weight that is not the one its B wants, and the lower rungs took
// 0.002-0.010 off R-free on five such starts, nothing off eight deposited models.
const std::vector<AtomAniso> aniso0 = ctx.aniso;
std::vector<TrialOutcome> outcomes;
// The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the
// balanced weights as they stand: the trial kept, or one that did not run where none did.
auto ladder_choice = [&](const std::vector<gemmi::Position> &from_pos, const std::vector<double> &from_b,
const std::vector<AtomAniso> &from_aniso, const Context::Fit *fit) {
std::vector<TrialWeights> ladder;
for (double rung : settings.weight_ladder)
for (double adp : settings.adp_ladder) {
@@ -2672,14 +2709,13 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
if (ncs_p != nullptr)
ladder.push_back({w0 * rung, adp_w0 * adp, settings.ncs_sigma});
}
const std::vector<AtomAniso> aniso0 = ctx.aniso;
std::vector<TrialOutcome> outcomes(ladder.size());
outcomes.assign(ladder.size(), TrialOutcome());
auto run_ladder = [&](const std::function<bool(const TrialWeights &)> &these) {
std::vector<TrialWeights> t;
for (const TrialWeights &l : ladder)
if (these(l))
t.push_back(l);
std::vector<TrialOutcome> ran = RunTrials(trial_slots, t, refinable, pos0, b0, aniso0, nullptr,
std::vector<TrialOutcome> ran = RunTrials(trial_slots, t, refinable, from_pos, from_b, from_aniso, fit,
settings.macro_cycles, settings, ncs_p);
for (size_t i = 0, m = 0; i < ladder.size(); i++)
if (these(ladder[i]))
@@ -2720,10 +2756,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
for (TrialOutcome &o : with_ncs)
drop_ahead(o);
int k = choose(outcomes, "ladder");
TrialOutcome state;
if (k >= 0)
state = std::move(outcomes[k]);
const int k = choose(outcomes, "ladder");
return k >= 0 ? std::move(outcomes[k]) : TrialOutcome();
};
TrialOutcome state = ladder_choice(pos0, b0, aniso0, nullptr);
// 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they
// are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the
@@ -2738,11 +2774,17 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// of the same weights from the same model is the same computation.
std::vector<TrialOutcome> from_best;
// `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since).
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best) {
// `label`: the steps' name in the log.
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best, const char *label) {
TrialOutcome best = state;
from_best.clear();
bool state_is_best = true;
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) {
// A step taken on R-work alone is followed by one more at most: on a model refined before, R-work falls
// step after step while R-free does not, and the trajectory ends at its best anyway - a model of 490k
// atoms spent six such steps (240 s) on a round it then gave back. (On 28 refinements of 22 deposited
// models and 6 predictions the rule changed nothing.)
int since_best = 0;
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled() && since_best <= 1; step++) {
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
std::vector<TrialWeights> trials = {{w, adp, ncs_s}};
if (step % 2 == 1 && adp > 0) {
@@ -2763,7 +2805,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
settings.macro_cycles, settings, ncs_p);
if (state_is_best)
from_best = outcomes;
const int k = choose(outcomes, fmt::format("step {}", step + 1).c_str());
const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str());
if (k < 0)
break;
const RefinementWeightTrial &t = outcomes[k].trial;
@@ -2775,6 +2817,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
break;
state = std::move(outcomes[k]);
state_is_best = (fresh && step == 0) || displaces(state.trial, best.trial);
since_best = state_is_best ? 0 : since_best + 1;
if (state_is_best) {
best = state;
from_best.clear();
@@ -2782,7 +2825,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
return best;
};
state = trajectory(std::move(state), false, from_best);
state = trajectory(std::move(state), false, from_best, "step");
// 2a. The solvent build (SolventBuild.h), from where the trajectory ended: waters put in at the difference
// map's peaks, taken out where the maps no longer hold them, and waters and ions relabelled, on the maps
@@ -3065,7 +3108,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
for (size_t i = 0, m = 0; i < trials.size(); i++)
if (!outcomes[i].ran)
outcomes[i] = std::move(ran[m++]);
k = choose(outcomes, "final");
const int k = choose(outcomes, "final");
if (k >= 0 && displaces(outcomes[k].trial, state.trial))
state = std::move(outcomes[k]);
}
@@ -3087,7 +3130,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// fits the free set better, the U at its balanced weight only where the U at a quarter of it lowers R-free
// or the -ln L of the free reflections (run ahead where the slots take them), and the U at four times its
// balanced weight not at all (on 49 refinements of 33 structures, no choice taken changed).
if (settings.adp_hypotheses && state.ran && !ctx.Cancelled()) {
// `own_tls`: the TLS groups the model has, tried as a hypothesis of their own.
auto displacements = [&](const std::vector<TlsBody> &own_tls) {
const RefinementStage adp_stage("adp_hypotheses");
ctx.SetPositions(state.pos);
ctx.SetB(state.b);
@@ -3097,7 +3141,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
state.trial.nll_free = ctx.Score().nll_free;
logger.Info("Refinement, displacements: from where the weights ended, R-free {:.4f}, -LL free {:.1f}",
state.trial.r_free, state.trial.nll_free);
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, input_tls);
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, own_tls);
const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0;
// the U weight balanced on the model with every refinable atom given a U
double u_w0 = adp;
@@ -3211,7 +3255,9 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
(t.r_free < trial(best).r_free - R_FREE_TIE && t.nll_free < trial(best).nll_free)))
best = static_cast<int>(k);
}
result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held";
// (the model where the weights ended keeps the displacement model a round before chose for it)
if (result.adp_model.empty() || state.trial.adp.empty())
result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held";
if (best >= 0) {
result.adp_model = outcomes[best].trial.adp;
state = std::move(outcomes[best]);
@@ -3232,6 +3278,89 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
}
logger.Info("Refinement: displacement model {}", result.adp_model);
};
if (settings.adp_hypotheses && state.ran && !ctx.Cancelled())
displacements(input_tls);
// 4b. The weights and the displacement model decided again, from the model as it now is: the balanced
// weights on its B factors, the ladder (1.) from it with its displacements as they are (the U of TLS or
// of the anisotropic model held, B held or refined), the trajectory (2.) from the rung kept, and the
// displacement hypotheses (4.) from where that ends - the TLS bodies refined from the B factors the first
// choice left, which they could not be from the B factors as given. A round is kept where it lowers R-free
// beyond the tie and undone otherwise, RefinementSettings::rounds in all: from a placed prediction, whose B
// factors are its pLDDT (6 A^2 under an overall B of 45), the first round balanced the B factors' weight
// a hundred times too stiff - the B similarity's gradient goes as 1 / B - and chose the middle rung with
// NCS; the second chose the tightest rung without, and from the B factors the first left, its TLS bodies
// 0.02 lower in R-free. Where the ladder and the trajectory leave the model where it was, the hypotheses
// would run the same trials from the same model again, so the round ends there: from a model refined
// before, a round is its ladder and one step. No round where the refinement does not displace the model
// as given (5.).
for (int round = 2; round <= settings.rounds && settings.adp_hypotheses && state.ran && !ctx.Cancelled() &&
(!settings.as_given_competes || displaces(state.trial, given.trial));
round++) {
const TrialOutcome before = state;
const std::vector<TlsBody> own_tls = state.tls;
ctx.SetPositions(state.pos);
ctx.SetB(state.b);
ctx.SetAniso(state.aniso);
ctx.Restore(state.fit);
PlaceRiders(ctx, topo);
// the ceiling of B from the model as it now is, as from the model as given: a placed prediction comes
// with B factors of 6 A^2 under an overall B of 45 (its ceiling 100) and refines at B near 50-70, where
// the TLS bodies and the U per atom were held at the ceiling; where it rises the engines are made again
// (the gather is reserved for it)
double b_max = 0;
for (size_t i = 0; i < ctx.atoms.size(); i++)
b_max = std::max<double>(b_max, ctx.atoms[i]->b_iso);
const double ceiling = BCeiling(b_max);
if (ceiling > ctx.b_ceiling) {
const RefinementStage stage("engine");
for (std::unique_ptr<OwnSlot> &o : own)
if (o)
o->ctx.engine.reset();
ctx.b_ceiling = ceiling;
MakeEngine(ctx, settings.gpu, settings, logger, restraint_records, restraint_slots);
ctx.engine->SetRestraints(0, RestraintsNow(ctx, topo));
ctx.engine->SetRestraints(1, topo.adp);
for (size_t k = 0; k < own.size(); k++)
if (own[k]) {
Context &o = own[k]->ctx;
o.b_ceiling = ceiling;
#ifdef JFJOCH_USE_CUDA
if (ctx.engine->OnGPU())
o.engine = MakeRefinementEngineGPU(*ctx.engine, devices[k].device, o.nthreads, devices[k].own_lane);
#endif
if (!o.engine)
o.engine = MakeRefinementEngineCPU(o.cell, *o.sg, o.d_min, o.refl, o.nthreads);
o.engine->SetAniso(&o.aniso);
o.engine->SetRestraints(1, own[k]->topo.adp);
}
ctx.Restore(state.fit);
}
{
const RefinementStage stage("balanced_weight");
w0 = BalancedWeight(ctx, refinable);
adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable);
}
logger.Info("Refinement, round {}: balanced X-ray weights {:.3g} (positions), {:.3g} (B), B ceiling {:.0f}", round,
w0, adp_w0, ctx.b_ceiling);
TrialOutcome rung = ladder_choice(state.pos, state.b, state.aniso, &state.fit);
if (rung.ran && displaces(rung.trial, state.trial))
state = std::move(rung);
std::vector<TrialOutcome> unused;
const std::string step = fmt::format("round {} step", round);
state = trajectory(std::move(state), false, unused, step.c_str());
if (state.trial.r_work == before.trial.r_work && state.trial.r_free == before.trial.r_free) {
logger.Info("Refinement, round {}: the weights leave the model where it was", round);
break;
}
displacements(own_tls.empty() ? input_tls : own_tls);
const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE;
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free,
kept ? "" : ", not kept");
if (!kept) {
state = before;
break;
}
}
// 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every
// trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what
@@ -3292,7 +3421,6 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
result.cancelled = ctx.Cancelled();
if (result.cancelled)
logger.Warning("Refinement: cancelled, {} weight trials ran", result.trials.size());
finish();
logger.Info("Refinement: done in {:.1f} s, {} target evaluations{}", result.seconds, ctx.evaluations,
result.restrained ? fmt::format("; R-work {:.4f}, R-free {:.4f}, weight {:.3g}", result.r_work,
@@ -3545,9 +3673,66 @@ bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated) {
return twin_fraction > 0 && (null_z >= TWIN_NULL_Z || (data_indicated && null_z >= TWIN_NULL_Z_DATA));
}
RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<MergedReflection> &merged,
const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A,
Logger &logger, const RefinementSettings &settings) {
bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super) {
if (sub.order() >= super.order())
return false;
for (const gemmi::Op &op : sub.sym_ops)
if (super.find_by_rotation(op.rot) == nullptr)
return false;
for (const gemmi::Op::Tran &cen : sub.cen_ops)
if (std::find(super.cen_ops.begin(), super.cen_ops.end(), cen) == super.cen_ops.end())
return false;
return true;
}
std::vector<MergedReflection> ExpandToSubgroup(const std::vector<MergedReflection> &merged, const gemmi::SpaceGroup &from,
const gemmi::SpaceGroup &to) {
const gemmi::GroupOps from_ops = from.operations(), to_ops = to.operations();
const gemmi::ReciprocalAsu asu(&to);
std::set<gemmi::Miller> seen;
std::vector<MergedReflection> out;
for (const MergedReflection &r : merged)
for (const gemmi::Op &op : from_ops.sym_ops) { // the centring changes no index
const auto [h, isym] = asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), to_ops);
if (!seen.insert(h).second)
continue;
MergedReflection e = r;
e.h = h[0];
e.k = h[1];
e.l = h[2];
if (isym % 2 == 0) { // the Friedel opposite
std::swap(e.I_plus, e.I_minus);
std::swap(e.sigma_plus, e.sigma_minus);
std::swap(e.F_plus, e.F_minus);
std::swap(e.sigmaF_plus, e.sigmaF_minus);
}
out.push_back(e);
}
return out;
}
RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<MergedReflection> &merged_in,
const gemmi::UnitCell &cell, const gemmi::SpaceGroup &data_sg, double wavelength_A,
Logger &logger, const RefinementSettings &settings_in) {
// In the model's own group where that is a proper subgroup of the data's: a model validated in P2(1), its copies
// pseudo-symmetric, against a merge in P4(1)2(1)2 the data could not tell from it. Its copies refine under their
// own symmetry, against the merge expanded to the subgroup - in the data's group the operators the model does not
// have would overlay its copies, and R would be the null's. The data carry the larger group's symmetry exactly,
// so a twin law of the subgroup's lattice has nothing to be tested on.
const gemmi::SpaceGroup *own = placed.find_spacegroup();
const bool subgroup = own != nullptr && ProperSubgroup(own->operations(), data_sg.operations());
const gemmi::SpaceGroup &sg = subgroup ? *own : data_sg;
RefinementSettings settings = settings_in;
std::vector<MergedReflection> expanded;
if (subgroup) {
expanded = ExpandToSubgroup(merged_in, data_sg, sg);
settings.twin = false;
logger.Info("Refinement: in the model's {}, a subgroup of the data's {} - the copies that group would overlay "
"refined under their own symmetry, against the {} reflections expanded to {} in the subgroup; no "
"twin law tested, the data having the larger group's symmetry exactly",
sg.xhm(), data_sg.xhm(), merged_in.size(), expanded.size());
}
const std::vector<MergedReflection> &merged = subgroup ? expanded : merged_in;
// The free set closed under the lattice's twin domains once, before either arm, so that both share it.
std::vector<MergedReflection> closed;
int moved = 0;
@@ -3613,6 +3798,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
twin_logger.ReplayInto(logger);
}
result.twin_free_closed = moved;
result.space_group = sg.xhm();
result.map.reset();
return result;
}
@@ -54,7 +54,7 @@ struct RefinementSettings {
// (BalancedWeight), each with the B factors' X-ray weight their own balanced one times each of adp_ladder (0: B
// held).
std::vector<double> weight_ladder = {1.0 / 1024, 1.0 / 128, 1.0 / 16};
std::vector<double> adp_ladder = {0.0, 1.0};
std::vector<double> adp_ladder = {0.0, 1.0 / 16, 1.0 / 4, 1.0};
// Then at most this many steps of the trajectory, each a set of trials from where the last one ended,
int max_steps = 8;
// and the final trials from where it ended, at its weights times each of these;
@@ -74,6 +74,8 @@ struct RefinementSettings {
// groups (the model's own, one per chain, the chains cut into TLSMD segments) with a residual B per atom, and an
// anisotropic U per atom with rigid-bond and similarity restraints (false: the isotropic B, or the model's U held).
bool adp_hypotheses = true;
// Rounds of the weights and the displacement model decided again from the refined model (RefineOnce, 4b.).
int rounds = 3;
// A refinement against the data as twinned by each twin law the lattice admits (RefinementTwinLaws), beside the
// untwinned one, and the model of whichever the evidence favours (RefineModel).
bool twin = true;
@@ -148,6 +150,8 @@ struct RefinementResult {
bool on_gpu = false;
double seconds = 0;
double d_min = 0;
std::string space_group; // the group the model was refined in (xHM): the data's, or the model's own
// where that is a subgroup of it (RefineModel)
int n_work = 0, n_free = 0;
// The model as given, scaled to these reflections (B made isotropic; the dictionary's hydrogens riding
@@ -257,6 +261,17 @@ struct RefinementResult {
// As REFMAC5 keeps twin-related reflections in one set (Roversi et al. (2012) Acta Cryst. D68, 418-424)
int CloseFreeSet(std::vector<MergedReflection> &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg);
// Whether every operation of `sub` is one of `super`'s (its rotation, and its centring), `super` the larger group:
// `sub` is then a group the data merged in `super` could not tell from it (a P2(1) crystal whose copies are
// pseudo-symmetric, merged in P4(1)2(1)2).
bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super);
// `merged`, the reflections of the group `from`, as the reflections of its subgroup `to`: each with every mate that
// `from` made equivalent and `to` keeps apart, at the same amplitude, intensity and free flag (its Bijvoet pair
// swapped where the mate is the Friedel opposite).
std::vector<MergedReflection> ExpandToSubgroup(const std::vector<MergedReflection> &merged, const gemmi::SpaceGroup &from,
const gemmi::SpaceGroup &to);
// Whether the twinned refinement is taken (RefineModel): a fraction above 0, and the law ahead of its null on the
// free set by 3 standard errors - or by 2 where the data's own intensity statistics read twinned as well.
bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated);
+5 -1
View File
@@ -197,7 +197,9 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m
out.space_group, out.free_reflections,
out.free_flags_generated ? "drawn here, the file has no free set" : "column " + data.free_column,
d_min_limit > 0 ? fmt::format(", to {:.2f} A", d_min_limit) : std::string());
if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); msg != nullptr && msg->number != sg->number)
// (a model whose group is a subgroup of the data's is refined in its own, RefineModel)
if (const gemmi::SpaceGroup *msg = model.find_spacegroup();
msg != nullptr && msg->number != sg->number && !ProperSubgroup(msg->operations(), sg->operations()))
logger.Warning("The model names space group {}, the data {}: refined in the data's", msg->xhm(), sg->xhm());
try {
gemmi::Structure st = model;
@@ -217,6 +219,8 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m
RefinementSettings refine = settings;
refine.as_given_competes = !out.free_flags_generated;
out.refinement = RefineModel(st, refl, data.cell, *sg, out.wavelength_A, logger, refine);
if (!out.refinement.space_group.empty())
out.space_group = out.refinement.space_group;
} catch (const std::exception &e) {
out.refinement = RefinementResult();
out.refinement.failure_reason = std::string("refinement failed: ") + e.what();
+4 -1
View File
@@ -5728,7 +5728,10 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) {
}
const PlacedModelOutput &p_final = *placed_model_out;
if (refined.ok) {
result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, p_final.space_group};
// in the group it was refined in: the model's own where that is a subgroup of the data's (RefineModel)
const gemmi::SpaceGroup *refined_in = gemmi::find_spacegroup_by_name(refined.space_group);
result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell,
refined_in ? *refined_in : p_final.space_group};
result.outputs->refined_maps = refined.maps;
result.outputs->blobs = refined.blobs;
result.outputs->solvent = refined.solvent;
+92
View File
@@ -966,6 +966,98 @@ TEST_CASE("Refinement_trials_run_ahead", "[Refinement]") {
CHECK(same == x.size());
}
// A model whose group is a proper subgroup of the data's: a P2(1) crystal whose two residues are pseudo-symmetric by
// the 2-fold along a, its data merged in P2(1)2(1)2(1). Refined in its own group against the data expanded to it, the
// model scores as it does in the validation; refined in the data's group its contents are overlaid and it does not.
TEST_CASE("Refinement_model_in_a_subgroup_of_the_data", "[Refinement]") {
const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90);
const gemmi::SpaceGroup &sub = *gemmi::find_spacegroup_by_name("P 1 21 1");
const gemmi::SpaceGroup &super = *gemmi::find_spacegroup_by_name("P 21 21 21");
CHECK(ProperSubgroup(sub.operations(), super.operations()));
CHECK(!ProperSubgroup(super.operations(), sub.operations()));
CHECK(!ProperSubgroup(sub.operations(), sub.operations()));
CHECK(!ProperSubgroup(gemmi::find_spacegroup_by_name("C 1 2 1")->operations(), super.operations()));
gemmi::Structure truth = residue_crystal({"TRP"}, cell, "P 1 21 1", 5);
{
// the residue's mate by the 2-fold along a, exact: the crystal is then P 2 2(1) 1 (pseudo-P2(1)2(1)2(1))
gemmi::Residue mate = truth.models[0].chains[0].residues[0];
mate.seqid = gemmi::SeqId(2, ' ');
for (gemmi::Atom &a : mate.atoms) {
const gemmi::Fractional f = cell.fractionalize(a.pos);
a.pos = cell.orthogonalize(gemmi::Fractional(f.x, -f.y, -f.z));
}
truth.models[0].chains[0].residues.push_back(mate);
}
const double d_min = 2.0;
// the data: the amplitudes of the truth in its own group, merged in the supergroup (each of its reflections the
// root mean square of the amplitudes it makes equivalent)
std::map<gemmi::Miller, double> amplitude;
{
std::vector<MergedReflection> own = asu_reflections(cell, sub, d_min);
RefinementReflections refl = PrepareRefinementReflections(own, cell, sub, d_min * 0.999);
auto cpu = MakeRefinementEngineCPU(cell, sub, d_min * 0.999, refl, 4);
cpu->SetBlur(RefinementBlur(cell, sub, d_min, 5.0));
std::vector<std::complex<float>> f;
cpu->Fcalc(truth.models[0], f);
for (size_t m = 0; m < refl.Size(); m++)
amplitude[refl.hkl[m]] = std::abs(f[m]);
}
const gemmi::GroupOps super_ops = super.operations(), sub_ops = sub.operations();
const gemmi::ReciprocalAsu sub_asu(&sub);
std::vector<MergedReflection> merged = asu_reflections(cell, super, d_min);
for (MergedReflection &r : merged) {
double sum = 0;
int n = 0;
for (const gemmi::Op &op : super_ops.sym_ops) {
const gemmi::Miller h = sub_asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), sub_ops).first;
if (const auto it = amplitude.find(h); it != amplitude.end()) {
sum += it->second * it->second;
n++;
}
}
r.F = n > 0 ? static_cast<float>(std::sqrt(sum / n)) : NAN;
r.sigmaF = 0.03f * r.F + 0.01f;
}
std::erase_if(merged, [](const MergedReflection &r) { return !std::isfinite(r.F); });
// the expansion: every reflection of the subgroup's set comes from one of the supergroup's, with its amplitude
// and its flag, and the supergroup's set is reached from it
const std::vector<MergedReflection> expanded = ExpandToSubgroup(merged, super, sub);
CHECK(expanded.size() > 1.5 * merged.size());
CHECK(expanded.size() <= 2 * merged.size());
const gemmi::ReciprocalAsu super_asu(&super);
std::map<gemmi::Miller, const MergedReflection *> source;
for (const MergedReflection &r : merged)
source[{{r.h, r.k, r.l}}] = &r;
std::set<gemmi::Miller> distinct;
size_t consistent = 0;
for (const MergedReflection &e : expanded) {
distinct.insert({{e.h, e.k, e.l}});
const auto it = source.find(super_asu.to_asu({{e.h, e.k, e.l}}, super_ops).first);
consistent += it != source.end() && it->second->F == e.F && it->second->rfree_flag == e.rfree_flag;
}
CHECK(distinct.size() == expanded.size());
CHECK(consistent == expanded.size());
Logger logger("RefinementTest");
RefinementSettings settings;
settings.gpu = false;
settings.nthreads = 4;
settings.score_only = true;
settings.riding_hydrogens = false;
const RefinementResult own_group = RefineModel(truth, merged, cell, super, 1.0, logger, settings);
REQUIRE(own_group.ok);
CHECK(own_group.space_group == "P 1 21 1");
CHECK(own_group.n_work + own_group.n_free == static_cast<int>(expanded.size()));
CHECK(own_group.r_free_start < 0.1);
gemmi::Structure as_data = truth;
as_data.spacegroup_hm = "P 21 21 21";
const RefinementResult data_group = RefineModel(as_data, merged, cell, super, 1.0, logger, settings);
REQUIRE(data_group.ok);
CHECK(data_group.space_group == "P 21 21 21");
CHECK(data_group.r_free_start > 0.3);
}
// A K+ on the 4-fold axis of P4 at occupancy 0.25 (its four coinciding copies one ion), 0.2 A off the axis in the
// model as given, with a water 2.8 A from it: the topology puts the ion on the axis, its contact with the water is
// listed once (not once per copy) and none with itself; refinement keeps it on the axis at the occupancy given,