Merge branch 'rc187-refine-speed' into rc187-refine-s1

This commit is contained in:
leonarski_f committed 2026-11-14 10:59:27 +01:00
commit e0e50f83a9
2 files changed
+143 -19

No files matched your search

+14 -4
View File
@@ -28,7 +28,7 @@ statistics and its map coefficients go out; the restraint dictionary is compiled
The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed,
and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within
100-300 Ų). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
@@ -309,9 +309,9 @@ are:
was never kept on 55 refinements, and taking it away changed no choice).
4. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U
held), each trial as long as a step on the positions where the weights ended, held - only the displacement
parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with
the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held
positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain
parameters refine, so the trials differ in the displacement model alone (where the geometry is out of bounds
there, which held positions cannot mend, each trial refines the positions too, a macro-cycle longer than a
step): TLS groups with a residual B - the model's own, one per chain
of at least 50 refinable atoms, and the chains cut into 4, 8 and 16 contiguous segments of at least 10 residues
as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r),
to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U
@@ -327,6 +327,16 @@ are:
`REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its
solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no
TLS group holds a water.
Then the weights and the displacement model are **decided again** from the model as it now is, up to twice: the
B ceiling and the balanced weights from its B factors, the ladder from it with its displacements as they are, the
trajectory, and the displacement hypotheses from where that ends. The first choice took its weights with the
B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT,
the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$),
the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the
refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers
R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the
hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder
and one step. No round runs where the refinement does not displace the model as given.
5. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the
rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as
it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never
@@ -78,6 +78,9 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0;
constexpr int SOLVENT_ITERATIONS = 10;
// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken.
constexpr double R_FREE_TIE = 0.001;
// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each
// after it from the model the one before ended with.
constexpr int REFINEMENT_ROUNDS = 3;
// The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this,
// relative; the most a position (A) or a B (A^2) moves in the first trial of a line search.
constexpr double MINIMISER_F_TOLERANCE = 1e-7;
@@ -1446,7 +1449,29 @@ void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) {
for (size_t d = first; d < a.size(); d++)
result.twin_fraction_null += a[d];
}
const std::vector<std::complex<float>> f_null = TwinnedModel(null, rows, ctx.fcalc);
// TwinnedModel with each unrelated mate on its own scale k_total: TwinnedModel puts a mate on the scale of the
// reflection it is the mate of, as the twin domain's reflection at the same spot is measured, but an unrelated
// mate lies in another direction, where the anisotropic scale can differ many times over (22 times in amplitude
// at 2 A, on a model whose U carried the anisotropy opposite to k_total), and on the reflection's scale it put
// the null 0.06 in R-free above the untwinned refinement - a law "ahead" of its null by 7.9 standard errors at a
// fraction of 0.04.
std::vector<std::complex<float>> f_null(n);
double own_fraction = 1.0;
for (float a : rows.twin_fractions)
own_fraction -= a;
for (size_t m = 0; m < n; m++) {
const std::complex<double> u = std::complex<double>(ctx.fcalc[m]) + std::complex<double>(rows.fmask[m]);
double i_model = own_fraction * rows.k_total[m] * rows.k_total[m] * std::norm(u);
for (size_t d = 0; d < null.Laws(); d++) {
const int j = null.mates[d][m];
const double k = d < first ? rows.k_total[m] : rows.k_total[j];
i_model += rows.twin_fractions[d] * k * k *
std::norm(std::complex<double>(ctx.fcalc[j]) + std::complex<double>(rows.fmask[j]));
}
const double amplitude = std::sqrt(std::max(i_model, 0.0));
f_null[m] = std::norm(u) > 0 ? std::complex<float>(u * (amplitude / std::abs(u)))
: std::complex<float>(static_cast<float>(amplitude), 0);
}
result.twin_r_free_null = ComputeRFactors(null, f_null).r_free;
// How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin||
// over its standard error. The members of a free twin group share their F_calc, so their terms are summed into
@@ -2665,6 +2690,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// The B factors are refined first at the middle rung only, and at the others where that comes within two
// ties of holding them there: on 364 refinements of the battery B held was kept by 281, and the rule
// changed the rung kept on 4 (all within 0.006 in R-free).
const std::vector<AtomAniso> aniso0 = ctx.aniso;
std::vector<TrialOutcome> outcomes;
// The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the
// balanced weights as they stand: the trial kept, or one that did not run where none did.
auto ladder_choice = [&](const std::vector<gemmi::Position> &from_pos, const std::vector<double> &from_b,
const std::vector<AtomAniso> &from_aniso, const Context::Fit *fit) {
std::vector<TrialWeights> ladder;
for (double rung : settings.weight_ladder)
for (double adp : settings.adp_ladder) {
@@ -2672,14 +2703,13 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
if (ncs_p != nullptr)
ladder.push_back({w0 * rung, adp_w0 * adp, settings.ncs_sigma});
}
const std::vector<AtomAniso> aniso0 = ctx.aniso;
std::vector<TrialOutcome> outcomes(ladder.size());
outcomes.assign(ladder.size(), TrialOutcome());
auto run_ladder = [&](const std::function<bool(const TrialWeights &)> &these) {
std::vector<TrialWeights> t;
for (const TrialWeights &l : ladder)
if (these(l))
t.push_back(l);
std::vector<TrialOutcome> ran = RunTrials(trial_slots, t, refinable, pos0, b0, aniso0, nullptr,
std::vector<TrialOutcome> ran = RunTrials(trial_slots, t, refinable, from_pos, from_b, from_aniso, fit,
settings.macro_cycles, settings, ncs_p);
for (size_t i = 0, m = 0; i < ladder.size(); i++)
if (these(ladder[i]))
@@ -2720,10 +2750,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
for (TrialOutcome &o : with_ncs)
drop_ahead(o);
int k = choose(outcomes, "ladder");
TrialOutcome state;
if (k >= 0)
state = std::move(outcomes[k]);
const int k = choose(outcomes, "ladder");
return k >= 0 ? std::move(outcomes[k]) : TrialOutcome();
};
TrialOutcome state = ladder_choice(pos0, b0, aniso0, nullptr);
// 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they
// are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the
@@ -2738,7 +2768,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// of the same weights from the same model is the same computation.
std::vector<TrialOutcome> from_best;
// `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since).
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best) {
// `label`: the steps' name in the log.
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best, const char *label) {
TrialOutcome best = state;
from_best.clear();
bool state_is_best = true;
@@ -2763,7 +2794,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
settings.macro_cycles, settings, ncs_p);
if (state_is_best)
from_best = outcomes;
const int k = choose(outcomes, fmt::format("step {}", step + 1).c_str());
const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str());
if (k < 0)
break;
const RefinementWeightTrial &t = outcomes[k].trial;
@@ -2782,7 +2813,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
return best;
};
state = trajectory(std::move(state), false, from_best);
state = trajectory(std::move(state), false, from_best, "step");
// 2a. The solvent build (SolventBuild.h), from where the trajectory ended: waters put in at the difference
// map's peaks, taken out where the maps no longer hold them, and waters and ions relabelled, on the maps
@@ -3065,7 +3096,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
for (size_t i = 0, m = 0; i < trials.size(); i++)
if (!outcomes[i].ran)
outcomes[i] = std::move(ran[m++]);
k = choose(outcomes, "final");
const int k = choose(outcomes, "final");
if (k >= 0 && displaces(outcomes[k].trial, state.trial))
state = std::move(outcomes[k]);
}
@@ -3087,7 +3118,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// fits the free set better, the U at its balanced weight only where the U at a quarter of it lowers R-free
// or the -ln L of the free reflections (run ahead where the slots take them), and the U at four times its
// balanced weight not at all (on 49 refinements of 33 structures, no choice taken changed).
if (settings.adp_hypotheses && state.ran && !ctx.Cancelled()) {
// `own_tls`: the TLS groups the model has, tried as a hypothesis of their own.
auto displacements = [&](const std::vector<TlsBody> &own_tls) {
const RefinementStage adp_stage("adp_hypotheses");
ctx.SetPositions(state.pos);
ctx.SetB(state.b);
@@ -3097,7 +3129,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
state.trial.nll_free = ctx.Score().nll_free;
logger.Info("Refinement, displacements: from where the weights ended, R-free {:.4f}, -LL free {:.1f}",
state.trial.r_free, state.trial.nll_free);
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, input_tls);
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, own_tls);
const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0;
// the U weight balanced on the model with every refinable atom given a U
double u_w0 = adp;
@@ -3232,6 +3264,89 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
}
logger.Info("Refinement: displacement model {}", result.adp_model);
};
if (settings.adp_hypotheses && state.ran && !ctx.Cancelled())
displacements(input_tls);
// 4b. The weights and the displacement model decided again, from the model as it now is: the balanced
// weights on its B factors, the ladder (1.) from it with its displacements as they are (the U of TLS or
// of the anisotropic model held, B held or refined), the trajectory (2.) from the rung kept, and the
// displacement hypotheses (4.) from where that ends - the TLS bodies refined from the B factors the first
// choice left, which they could not be from the B factors as given. A round is kept where it lowers R-free
// beyond the tie and undone otherwise, REFINEMENT_ROUNDS in all: from a placed prediction, whose B
// factors are its pLDDT (6 A^2 under an overall B of 45), the first round balanced the B factors' weight
// a hundred times too stiff - the B similarity's gradient goes as 1 / B - and chose the middle rung with
// NCS; the second chose the tightest rung without, and from the B factors the first left, its TLS bodies
// 0.02 lower in R-free. Where the ladder and the trajectory leave the model where it was, the hypotheses
// would run the same trials from the same model again, so the round ends there: from a model refined
// before, a round is its ladder and one step. No round where the refinement does not displace the model
// as given (5.).
for (int round = 2; round <= REFINEMENT_ROUNDS && settings.adp_hypotheses && state.ran && !ctx.Cancelled() &&
(!settings.as_given_competes || displaces(state.trial, given.trial));
round++) {
const TrialOutcome before = state;
const std::vector<TlsBody> own_tls = state.tls;
ctx.SetPositions(state.pos);
ctx.SetB(state.b);
ctx.SetAniso(state.aniso);
ctx.Restore(state.fit);
PlaceRiders(ctx, topo);
// the ceiling of B from the model as it now is, as from the model as given: a placed prediction comes
// with B factors of 6 A^2 under an overall B of 45 (its ceiling 100) and refines at B near 50-70, where
// the TLS bodies and the U per atom were held at the ceiling; where it rises the engines are made again
// (the gather is reserved for it)
double b_max = 0;
for (size_t i = 0; i < ctx.atoms.size(); i++)
b_max = std::max<double>(b_max, ctx.atoms[i]->b_iso);
const double ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX);
if (ceiling > ctx.b_ceiling) {
const RefinementStage stage("engine");
for (std::unique_ptr<OwnSlot> &o : own)
if (o)
o->ctx.engine.reset();
ctx.b_ceiling = ceiling;
MakeEngine(ctx, settings.gpu, settings, logger, restraint_records, restraint_slots);
ctx.engine->SetRestraints(0, RestraintsNow(ctx, topo));
ctx.engine->SetRestraints(1, topo.adp);
for (size_t k = 0; k < own.size(); k++)
if (own[k]) {
Context &o = own[k]->ctx;
o.b_ceiling = ceiling;
#ifdef JFJOCH_USE_CUDA
if (ctx.engine->OnGPU())
o.engine = MakeRefinementEngineGPU(*ctx.engine, devices[k].device, o.nthreads, devices[k].own_lane);
#endif
if (!o.engine)
o.engine = MakeRefinementEngineCPU(o.cell, *o.sg, o.d_min, o.refl, o.nthreads);
o.engine->SetAniso(&o.aniso);
o.engine->SetRestraints(1, own[k]->topo.adp);
}
ctx.Restore(state.fit);
}
{
const RefinementStage stage("balanced_weight");
w0 = BalancedWeight(ctx, refinable);
adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable);
}
logger.Info("Refinement, round {}: balanced X-ray weights {:.3g} (positions), {:.3g} (B), B ceiling {:.0f}", round,
w0, adp_w0, ctx.b_ceiling);
TrialOutcome rung = ladder_choice(state.pos, state.b, state.aniso, &state.fit);
if (rung.ran && displaces(rung.trial, state.trial))
state = std::move(rung);
std::vector<TrialOutcome> unused;
const std::string step = fmt::format("round {} step", round);
state = trajectory(std::move(state), false, unused, step.c_str());
if (state.trial.r_work == before.trial.r_work && state.trial.r_free == before.trial.r_free) {
logger.Info("Refinement, round {}: the weights leave the model where it was", round);
break;
}
displacements(own_tls.empty() ? input_tls : own_tls);
const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE;
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free,
kept ? "" : ", not kept");
if (!kept) {
state = before;
break;
}
}
// 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every
// trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what
@@ -3292,7 +3407,6 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
result.cancelled = ctx.Cancelled();
if (result.cancelled)
logger.Warning("Refinement: cancelled, {} weight trials ran", result.trials.size());
finish();
logger.Info("Refinement: done in {:.1f} s, {} target evaluations{}", result.seconds, ctx.evaluations,
result.restrained ? fmt::format("; R-work {:.4f}, R-free {:.4f}, weight {:.3g}", result.r_work,