Merge branch 'rc187-refine-speed' into rc187-refine-s1
This commit is contained in:
commit
e0e50f83a9
2 files changed
+143
-19
No files matched your search
@@ -28,7 +28,7 @@ statistics and its map coefficients go out; the restraint dictionary is compiled
|
||||
|
||||
The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed,
|
||||
and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within
|
||||
100-300 Ų). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
|
||||
100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
|
||||
groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the
|
||||
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
|
||||
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
|
||||
@@ -309,9 +309,9 @@ are:
|
||||
was never kept on 55 refinements, and taking it away changed no choice).
|
||||
4. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U
|
||||
held), each trial as long as a step on the positions where the weights ended, held - only the displacement
|
||||
parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with
|
||||
the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held
|
||||
positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain
|
||||
parameters refine, so the trials differ in the displacement model alone (where the geometry is out of bounds
|
||||
there, which held positions cannot mend, each trial refines the positions too, a macro-cycle longer than a
|
||||
step): TLS groups with a residual B - the model's own, one per chain
|
||||
of at least 50 refinable atoms, and the chains cut into 4, 8 and 16 contiguous segments of at least 10 residues
|
||||
as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r),
|
||||
to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U
|
||||
@@ -327,6 +327,16 @@ are:
|
||||
`REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its
|
||||
solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no
|
||||
TLS group holds a water.
|
||||
Then the weights and the displacement model are **decided again** from the model as it now is, up to twice: the
|
||||
B ceiling and the balanced weights from its B factors, the ladder from it with its displacements as they are, the
|
||||
trajectory, and the displacement hypotheses from where that ends. The first choice took its weights with the
|
||||
B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT,
|
||||
the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$),
|
||||
the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the
|
||||
refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers
|
||||
R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the
|
||||
hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder
|
||||
and one step. No round runs where the refinement does not displace the model as given.
|
||||
5. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the
|
||||
rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as
|
||||
it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never
|
||||
|
||||
@@ -78,6 +78,9 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0;
|
||||
constexpr int SOLVENT_ITERATIONS = 10;
|
||||
// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken.
|
||||
constexpr double R_FREE_TIE = 0.001;
|
||||
// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each
|
||||
// after it from the model the one before ended with.
|
||||
constexpr int REFINEMENT_ROUNDS = 3;
|
||||
// The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this,
|
||||
// relative; the most a position (A) or a B (A^2) moves in the first trial of a line search.
|
||||
constexpr double MINIMISER_F_TOLERANCE = 1e-7;
|
||||
@@ -1446,7 +1449,29 @@ void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) {
|
||||
for (size_t d = first; d < a.size(); d++)
|
||||
result.twin_fraction_null += a[d];
|
||||
}
|
||||
const std::vector<std::complex<float>> f_null = TwinnedModel(null, rows, ctx.fcalc);
|
||||
// TwinnedModel with each unrelated mate on its own scale k_total: TwinnedModel puts a mate on the scale of the
|
||||
// reflection it is the mate of, as the twin domain's reflection at the same spot is measured, but an unrelated
|
||||
// mate lies in another direction, where the anisotropic scale can differ many times over (22 times in amplitude
|
||||
// at 2 A, on a model whose U carried the anisotropy opposite to k_total), and on the reflection's scale it put
|
||||
// the null 0.06 in R-free above the untwinned refinement - a law "ahead" of its null by 7.9 standard errors at a
|
||||
// fraction of 0.04.
|
||||
std::vector<std::complex<float>> f_null(n);
|
||||
double own_fraction = 1.0;
|
||||
for (float a : rows.twin_fractions)
|
||||
own_fraction -= a;
|
||||
for (size_t m = 0; m < n; m++) {
|
||||
const std::complex<double> u = std::complex<double>(ctx.fcalc[m]) + std::complex<double>(rows.fmask[m]);
|
||||
double i_model = own_fraction * rows.k_total[m] * rows.k_total[m] * std::norm(u);
|
||||
for (size_t d = 0; d < null.Laws(); d++) {
|
||||
const int j = null.mates[d][m];
|
||||
const double k = d < first ? rows.k_total[m] : rows.k_total[j];
|
||||
i_model += rows.twin_fractions[d] * k * k *
|
||||
std::norm(std::complex<double>(ctx.fcalc[j]) + std::complex<double>(rows.fmask[j]));
|
||||
}
|
||||
const double amplitude = std::sqrt(std::max(i_model, 0.0));
|
||||
f_null[m] = std::norm(u) > 0 ? std::complex<float>(u * (amplitude / std::abs(u)))
|
||||
: std::complex<float>(static_cast<float>(amplitude), 0);
|
||||
}
|
||||
result.twin_r_free_null = ComputeRFactors(null, f_null).r_free;
|
||||
// How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin||
|
||||
// over its standard error. The members of a free twin group share their F_calc, so their terms are summed into
|
||||
@@ -2665,6 +2690,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// The B factors are refined first at the middle rung only, and at the others where that comes within two
|
||||
// ties of holding them there: on 364 refinements of the battery B held was kept by 281, and the rule
|
||||
// changed the rung kept on 4 (all within 0.006 in R-free).
|
||||
const std::vector<AtomAniso> aniso0 = ctx.aniso;
|
||||
std::vector<TrialOutcome> outcomes;
|
||||
// The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the
|
||||
// balanced weights as they stand: the trial kept, or one that did not run where none did.
|
||||
auto ladder_choice = [&](const std::vector<gemmi::Position> &from_pos, const std::vector<double> &from_b,
|
||||
const std::vector<AtomAniso> &from_aniso, const Context::Fit *fit) {
|
||||
std::vector<TrialWeights> ladder;
|
||||
for (double rung : settings.weight_ladder)
|
||||
for (double adp : settings.adp_ladder) {
|
||||
@@ -2672,14 +2703,13 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
if (ncs_p != nullptr)
|
||||
ladder.push_back({w0 * rung, adp_w0 * adp, settings.ncs_sigma});
|
||||
}
|
||||
const std::vector<AtomAniso> aniso0 = ctx.aniso;
|
||||
std::vector<TrialOutcome> outcomes(ladder.size());
|
||||
outcomes.assign(ladder.size(), TrialOutcome());
|
||||
auto run_ladder = [&](const std::function<bool(const TrialWeights &)> &these) {
|
||||
std::vector<TrialWeights> t;
|
||||
for (const TrialWeights &l : ladder)
|
||||
if (these(l))
|
||||
t.push_back(l);
|
||||
std::vector<TrialOutcome> ran = RunTrials(trial_slots, t, refinable, pos0, b0, aniso0, nullptr,
|
||||
std::vector<TrialOutcome> ran = RunTrials(trial_slots, t, refinable, from_pos, from_b, from_aniso, fit,
|
||||
settings.macro_cycles, settings, ncs_p);
|
||||
for (size_t i = 0, m = 0; i < ladder.size(); i++)
|
||||
if (these(ladder[i]))
|
||||
@@ -2720,10 +2750,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
}
|
||||
for (TrialOutcome &o : with_ncs)
|
||||
drop_ahead(o);
|
||||
int k = choose(outcomes, "ladder");
|
||||
TrialOutcome state;
|
||||
if (k >= 0)
|
||||
state = std::move(outcomes[k]);
|
||||
const int k = choose(outcomes, "ladder");
|
||||
return k >= 0 ? std::move(outcomes[k]) : TrialOutcome();
|
||||
};
|
||||
TrialOutcome state = ladder_choice(pos0, b0, aniso0, nullptr);
|
||||
|
||||
// 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they
|
||||
// are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the
|
||||
@@ -2738,7 +2768,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// of the same weights from the same model is the same computation.
|
||||
std::vector<TrialOutcome> from_best;
|
||||
// `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since).
|
||||
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best) {
|
||||
// `label`: the steps' name in the log.
|
||||
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best, const char *label) {
|
||||
TrialOutcome best = state;
|
||||
from_best.clear();
|
||||
bool state_is_best = true;
|
||||
@@ -2763,7 +2794,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
settings.macro_cycles, settings, ncs_p);
|
||||
if (state_is_best)
|
||||
from_best = outcomes;
|
||||
const int k = choose(outcomes, fmt::format("step {}", step + 1).c_str());
|
||||
const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str());
|
||||
if (k < 0)
|
||||
break;
|
||||
const RefinementWeightTrial &t = outcomes[k].trial;
|
||||
@@ -2782,7 +2813,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
}
|
||||
return best;
|
||||
};
|
||||
state = trajectory(std::move(state), false, from_best);
|
||||
state = trajectory(std::move(state), false, from_best, "step");
|
||||
|
||||
// 2a. The solvent build (SolventBuild.h), from where the trajectory ended: waters put in at the difference
|
||||
// map's peaks, taken out where the maps no longer hold them, and waters and ions relabelled, on the maps
|
||||
@@ -3065,7 +3096,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
for (size_t i = 0, m = 0; i < trials.size(); i++)
|
||||
if (!outcomes[i].ran)
|
||||
outcomes[i] = std::move(ran[m++]);
|
||||
k = choose(outcomes, "final");
|
||||
const int k = choose(outcomes, "final");
|
||||
if (k >= 0 && displaces(outcomes[k].trial, state.trial))
|
||||
state = std::move(outcomes[k]);
|
||||
}
|
||||
@@ -3087,7 +3118,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// fits the free set better, the U at its balanced weight only where the U at a quarter of it lowers R-free
|
||||
// or the -ln L of the free reflections (run ahead where the slots take them), and the U at four times its
|
||||
// balanced weight not at all (on 49 refinements of 33 structures, no choice taken changed).
|
||||
if (settings.adp_hypotheses && state.ran && !ctx.Cancelled()) {
|
||||
// `own_tls`: the TLS groups the model has, tried as a hypothesis of their own.
|
||||
auto displacements = [&](const std::vector<TlsBody> &own_tls) {
|
||||
const RefinementStage adp_stage("adp_hypotheses");
|
||||
ctx.SetPositions(state.pos);
|
||||
ctx.SetB(state.b);
|
||||
@@ -3097,7 +3129,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
state.trial.nll_free = ctx.Score().nll_free;
|
||||
logger.Info("Refinement, displacements: from where the weights ended, R-free {:.4f}, -LL free {:.1f}",
|
||||
state.trial.r_free, state.trial.nll_free);
|
||||
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, input_tls);
|
||||
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, own_tls);
|
||||
const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0;
|
||||
// the U weight balanced on the model with every refinable atom given a U
|
||||
double u_w0 = adp;
|
||||
@@ -3232,6 +3264,89 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
}
|
||||
}
|
||||
logger.Info("Refinement: displacement model {}", result.adp_model);
|
||||
};
|
||||
if (settings.adp_hypotheses && state.ran && !ctx.Cancelled())
|
||||
displacements(input_tls);
|
||||
// 4b. The weights and the displacement model decided again, from the model as it now is: the balanced
|
||||
// weights on its B factors, the ladder (1.) from it with its displacements as they are (the U of TLS or
|
||||
// of the anisotropic model held, B held or refined), the trajectory (2.) from the rung kept, and the
|
||||
// displacement hypotheses (4.) from where that ends - the TLS bodies refined from the B factors the first
|
||||
// choice left, which they could not be from the B factors as given. A round is kept where it lowers R-free
|
||||
// beyond the tie and undone otherwise, REFINEMENT_ROUNDS in all: from a placed prediction, whose B
|
||||
// factors are its pLDDT (6 A^2 under an overall B of 45), the first round balanced the B factors' weight
|
||||
// a hundred times too stiff - the B similarity's gradient goes as 1 / B - and chose the middle rung with
|
||||
// NCS; the second chose the tightest rung without, and from the B factors the first left, its TLS bodies
|
||||
// 0.02 lower in R-free. Where the ladder and the trajectory leave the model where it was, the hypotheses
|
||||
// would run the same trials from the same model again, so the round ends there: from a model refined
|
||||
// before, a round is its ladder and one step. No round where the refinement does not displace the model
|
||||
// as given (5.).
|
||||
for (int round = 2; round <= REFINEMENT_ROUNDS && settings.adp_hypotheses && state.ran && !ctx.Cancelled() &&
|
||||
(!settings.as_given_competes || displaces(state.trial, given.trial));
|
||||
round++) {
|
||||
const TrialOutcome before = state;
|
||||
const std::vector<TlsBody> own_tls = state.tls;
|
||||
ctx.SetPositions(state.pos);
|
||||
ctx.SetB(state.b);
|
||||
ctx.SetAniso(state.aniso);
|
||||
ctx.Restore(state.fit);
|
||||
PlaceRiders(ctx, topo);
|
||||
// the ceiling of B from the model as it now is, as from the model as given: a placed prediction comes
|
||||
// with B factors of 6 A^2 under an overall B of 45 (its ceiling 100) and refines at B near 50-70, where
|
||||
// the TLS bodies and the U per atom were held at the ceiling; where it rises the engines are made again
|
||||
// (the gather is reserved for it)
|
||||
double b_max = 0;
|
||||
for (size_t i = 0; i < ctx.atoms.size(); i++)
|
||||
b_max = std::max<double>(b_max, ctx.atoms[i]->b_iso);
|
||||
const double ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX);
|
||||
if (ceiling > ctx.b_ceiling) {
|
||||
const RefinementStage stage("engine");
|
||||
for (std::unique_ptr<OwnSlot> &o : own)
|
||||
if (o)
|
||||
o->ctx.engine.reset();
|
||||
ctx.b_ceiling = ceiling;
|
||||
MakeEngine(ctx, settings.gpu, settings, logger, restraint_records, restraint_slots);
|
||||
ctx.engine->SetRestraints(0, RestraintsNow(ctx, topo));
|
||||
ctx.engine->SetRestraints(1, topo.adp);
|
||||
for (size_t k = 0; k < own.size(); k++)
|
||||
if (own[k]) {
|
||||
Context &o = own[k]->ctx;
|
||||
o.b_ceiling = ceiling;
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
if (ctx.engine->OnGPU())
|
||||
o.engine = MakeRefinementEngineGPU(*ctx.engine, devices[k].device, o.nthreads, devices[k].own_lane);
|
||||
#endif
|
||||
if (!o.engine)
|
||||
o.engine = MakeRefinementEngineCPU(o.cell, *o.sg, o.d_min, o.refl, o.nthreads);
|
||||
o.engine->SetAniso(&o.aniso);
|
||||
o.engine->SetRestraints(1, own[k]->topo.adp);
|
||||
}
|
||||
ctx.Restore(state.fit);
|
||||
}
|
||||
{
|
||||
const RefinementStage stage("balanced_weight");
|
||||
w0 = BalancedWeight(ctx, refinable);
|
||||
adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable);
|
||||
}
|
||||
logger.Info("Refinement, round {}: balanced X-ray weights {:.3g} (positions), {:.3g} (B), B ceiling {:.0f}", round,
|
||||
w0, adp_w0, ctx.b_ceiling);
|
||||
TrialOutcome rung = ladder_choice(state.pos, state.b, state.aniso, &state.fit);
|
||||
if (rung.ran && displaces(rung.trial, state.trial))
|
||||
state = std::move(rung);
|
||||
std::vector<TrialOutcome> unused;
|
||||
const std::string step = fmt::format("round {} step", round);
|
||||
state = trajectory(std::move(state), false, unused, step.c_str());
|
||||
if (state.trial.r_work == before.trial.r_work && state.trial.r_free == before.trial.r_free) {
|
||||
logger.Info("Refinement, round {}: the weights leave the model where it was", round);
|
||||
break;
|
||||
}
|
||||
displacements(own_tls.empty() ? input_tls : own_tls);
|
||||
const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE;
|
||||
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free,
|
||||
kept ? "" : ", not kept");
|
||||
if (!kept) {
|
||||
state = before;
|
||||
break;
|
||||
}
|
||||
}
|
||||
// 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every
|
||||
// trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what
|
||||
@@ -3292,7 +3407,6 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
result.cancelled = ctx.Cancelled();
|
||||
if (result.cancelled)
|
||||
logger.Warning("Refinement: cancelled, {} weight trials ran", result.trials.size());
|
||||
|
||||
finish();
|
||||
logger.Info("Refinement: done in {:.1f} s, {} target evaluations{}", result.seconds, ctx.evaluations,
|
||||
result.restrained ? fmt::format("; R-work {:.4f}, R-free {:.4f}, weight {:.3g}", result.r_work,
|
||||
|
||||
Reference in new issue
Block a user