Refinement: the free set decides the weights and the displacement model only

The free set now decides only the X-ray weights of the positions and the B
factors, the NCS restraints' sigma, the displacement model (isotropic / TLS
cuts / U per atom), the reading of the model's TLS groups, and whether the
refinement displaces the model as given. Each of these takes a candidate over
the incumbent only where both free measures agree: its -ln L of the free
reflections lower and its R-free not higher by more than the standard error of
the incumbent's, R_free / sqrt(2 N_free) (Tickle, Laskowski & Moss 2000), in
place of the fixed R-free tie of 0.001 (which alone decided most choices). The
approach of requiring free measures to agree follows PDB-REDO (Joosten et al.
2012, 2014).

Every other decision now uses the work-set target - w (-ln L work) + E_geometry
(with NCS at its sigma), plus w_B (-ln L work) + E_B where B is refined - at
the weights the two candidates share, a step taken where it lowers that by
more than 1e-4 relative and the geometry stays within bounds where it was:
the trajectory's steps (which now end where the last step taken ended, not at
the best step on the free set), the final ladder's step, the step after the
displacement choice, the second round, the solvent build's rounds, the
side-chain field's kept test and the population's child. A candidate at other
weights is not compared on a target that changed with them (the positions and
B minimise separate targets, whose sum a weight change can raise while it
lowers each) and is left to the free-set rule. The placement's rigid bodies
and B per residue are kept where they lower -ln L work.

Each weight trial now records -ln L work and the restraint sums
(RefinementWeightTrial::nll_work, e_geometry, e_b) and logs its target.

Not changed: the twin-law choice (a paired free-set z against its null, a
model hypothesis test with its own standard error), MR and model validation.
No k-fold cross-validation yet for free sets under 500 reflections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-18 08:04:28 +01:00
1 parent fdeb43af65
commit 805c3a14c3
2 files changed
+152 -106

No files matched your search

+149 -106
View File
@@ -91,8 +91,16 @@ constexpr double REPULSION_MARGIN = 0.5; // A: pairs this much beyond their co
constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0;
// The minimiser's iterations per macro-cycle where the solvent build refines the solvent alone.
constexpr int SOLVENT_ITERATIONS = 10;
// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken.
constexpr double R_FREE_TIE = 0.001;
// The free set decides the weights and the displacement model alone, and only where both of its measures agree: a
// candidate is preferred to the incumbent where its -ln L of the free reflections is lower and its R-free not higher
// by more than the standard error of the incumbent's, R_free / sqrt(2 N_free).
// Following Tickle, Laskowski & Moss (2000) Acta Cryst. D56, 442-450 (the standard error of R-free), as PDB-REDO
// decides on more than one free measure, Joosten, Long, Murshudov & Perrakis (2014) IUCrJ 1, 213-220
bool FreeSetPrefers(double r_free, double nll_free, double r_free_b, double nll_free_b, int n_free) {
return nll_free < nll_free_b && r_free < r_free_b + r_free_b / std::sqrt(2.0 * n_free);
}
// Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative.
constexpr double TARGET_TOLERANCE = 1e-4;
// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each
// after it from the model the one before ended with.
// The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this,
@@ -1564,9 +1572,10 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
if (body.size() >= 50)
bodies.push_back(body);
}
// Each step is kept only where it lowers R-free beyond the tie: a deposited model was refined atom by atom
// already, and moving its chains or shifting its B factors residue by residue then only makes it worse.
RFactors before = ctx.Refit();
// Each step is kept only where it lowers the -ln L of the working reflections, the target it minimised (the free
// set decides the weights and the displacement model alone).
ctx.Refit();
double before = ctx.Score().nll_work;
if (!bodies.empty()) {
const std::vector<gemmi::Position> pos0 = ctx.Positions();
std::vector<double> q(6 * bodies.size(), 0.0);
@@ -1577,13 +1586,15 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
const CancelCheck cancel_check(ctx);
ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary);
rb->Place(q.data());
const RFactors after = ctx.Refit();
if (after.r_free < before.r_free - R_FREE_TIE) {
ctx.Refit();
const double after = ctx.Score().nll_work;
if (after < before) {
result.rigid_bodies = static_cast<int>(bodies.size());
before = after;
} else {
ctx.SetPositions(pos0);
before = ctx.Refit();
ctx.Refit();
before = ctx.Score().nll_work;
}
}
if (group_b) {
@@ -1596,7 +1607,8 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
const CancelCheck cancel_check(ctx);
ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary);
gb->Place(q.data());
result.group_b = ctx.Refit().r_free < before.r_free - R_FREE_TIE;
ctx.Refit();
result.group_b = ctx.Score().nll_work < before;
if (!result.group_b)
ctx.SetB(b0);
}
@@ -2246,6 +2258,25 @@ struct TrialOutcome {
int evaluations = 0;
};
// The work-set target's terms of the model as it stands (RefinementWeightTrial), at the last Refit()'s fit: what every
// decision but the weights' and the displacement model's is made on (WorkTarget).
void MeasureWorkTarget(Context &ctx, const RefinementTopology &topo, const NcsCopies *ncs, double ncs_sigma,
RefinementWeightTrial &t) {
const RefinementScore sc = ctx.Score();
t.nll_work = sc.nll_work;
t.nll_free = sc.nll_free;
const std::vector<std::array<double, 4>> xyzb = ctx.XyzB();
t.e_geometry = EvaluateRestraints(RestraintsNow(ctx, topo, ncs, ncs_sigma), xyzb, nullptr);
t.e_b = EvaluateRestraints(BRestraintsNow(ctx, topo), xyzb, nullptr);
}
// The work-set target of a trial at the X-ray weights w (positions) and adp (B, 0: held), its NCS restraints at its own
// sigma: what a trial at those weights minimises, the X-ray term of the positions and the geometry, and that of the B
// factors and the B similarity where they are refined.
double WorkTarget(const RefinementWeightTrial &t, double w, double adp) {
return w * t.nll_work + t.e_geometry + (adp > 0 ? adp * t.nll_work + t.e_b : 0.0);
}
// The X-ray target's gradient on the positions, its squared norm over `refinable`, for the atoms as they are.
double XrayGradientNorm(Context &ctx, const std::vector<int> &refinable) {
std::vector<std::array<double, 4>> g;
@@ -2471,16 +2502,16 @@ TrialOutcome RunTrial(TrialSlot slot, XyzCalibrations &calibrations, const std::
t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB());
if (hyp != nullptr)
t.adp = hyp->name;
t.nll_free = ctx.Score().nll_free;
out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, "
"angles {:.2f} deg{}{}",
MeasureWorkTarget(ctx, topo, ncs, ncs_sigma, t);
out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, target {:.6g}, rmsd bonds "
"{:.4f} A, angles {:.2f} deg{}{}",
std::string(tw.blur.empty() ? "" : "blurred, ") + (hyp != nullptr ? hyp->name + ", " : ""), w,
kind == AdpHypothesis::Kind::Anisotropic ? "U" : kind == AdpHypothesis::Kind::Tls ? "residual B" : "B",
adp_w > 0 ? fmt::format("refined at weight {:.3g}", adp_w) : std::string("held"),
ncs == nullptr || ncs->Empty() ? std::string()
: ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd)
: fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd),
rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free),
rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), WorkTarget(t, w, adp_w),
t.geometry.rmsd_bonds, t.geometry.rmsd_angles,
t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal)
: std::string(),
@@ -2859,8 +2890,8 @@ double BalancedUWeight(Context &ctx, const URestraints &restraints, const std::v
// The displacements of the model as given, read where it has TLS groups whose atoms bring no U of their own: their U
// from the groups, its B column taken as the total B (the U's equivalent isotropic B, as phenix.refine writes it) or as
// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - the one of the three whose
// R-free is lowest, beyond the tie in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens
// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - a displacement model, chosen on
// the free set (FreeSetPrefers), in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens
// (which ride on their parents' B). Returns the R-factors of the reading kept.
// Following Winn, Isupov & Murshudov (2001) Acta Cryst. D57, 122-133 (the B column total or residual to the TLS)
RFactors ReadInputDisplacements(Context &ctx, const std::vector<TlsBody> &tls, Logger &logger, RefinementResult &result) {
@@ -2906,6 +2937,7 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector<TlsBody> &tls, L
readings.push_back({result.adp_input == "ANISOU" ? "ANISOU" : "isotropic", b0, aniso0});
int best = -1;
std::vector<RFactors> rf;
std::vector<double> nll_free;
for (size_t k = 0; k < readings.size(); k++) {
ctx.SetB(readings[k].b);
ctx.SetAniso(readings[k].aniso);
@@ -2914,9 +2946,10 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector<TlsBody> &tls, L
ctx.atoms[i]->b_iso = ctx.atoms[ctx.rider_parent[i]]->b_iso;
ctx.ForgetStarts();
rf.push_back(ctx.Refit());
logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}", tls.size(),
tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free);
if (best < 0 || rf[k].r_free < rf[best].r_free - R_FREE_TIE)
nll_free.push_back(ctx.Score().nll_free);
logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}, -LL free "
"{:.1f}", tls.size(), tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free, nll_free[k]);
if (best < 0 || FreeSetPrefers(rf[k].r_free, nll_free[k], rf[best].r_free, nll_free[best], rf[k].n_free))
best = static_cast<int>(k);
}
ctx.SetB(readings[best].b);
@@ -3426,10 +3459,25 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
o = TrialOutcome();
};
// A trial displaces another only where its geometry is within bounds and its R-free lower by more than the
// tie.
auto displaces = [](const RefinementWeightTrial &t, const RefinementWeightTrial &b) {
return t.within_bounds && (!b.within_bounds || t.r_free < b.r_free - R_FREE_TIE);
// A trial at other weights (or the model as given) displaces another only where its geometry is within
// bounds and the free set prefers it (FreeSetPrefers).
const int n_free = result.n_free;
auto displaces = [n_free](const RefinementWeightTrial &t, const RefinementWeightTrial &b) {
return t.within_bounds &&
(!b.within_bounds || FreeSetPrefers(t.r_free, t.nll_free, b.r_free, b.nll_free, n_free));
};
// A trial taken on from `s`: at the weights of s, where the work-set target (WorkTarget) is lower than that of s
// by more than TARGET_TOLERANCE, and the geometry within bounds where that of s is. A trial at other weights
// (X-ray weights or NCS restraints) is not compared on a target that changed with them - the positions and the
// B factors minimise targets of their own, whose sum a change of weight can raise as it lowers each: the
// weights are the free set's to decide (displaces).
auto advances = [&displaces](const RefinementWeightTrial &t, const RefinementWeightTrial &s) {
if (t.weight != s.weight || t.adp_weight != s.adp_weight || t.ncs_sigma != s.ncs_sigma)
return displaces(t, s);
if (t.within_bounds < s.within_bounds)
return false;
const double before = WorkTarget(s, s.weight, s.adp_weight);
return WorkTarget(t, s.weight, s.adp_weight) < before - TARGET_TOLERANCE * std::abs(before);
};
// Of a set of trials, the one kept: they come in order of trust, the first kept unless a later one displaces
// it. -1: none ran.
@@ -3459,9 +3507,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// not change (on 39 refinements of 21 structures with copies it never did; the X-ray weight it does), and
// at the rung they chose and the next looser one (on 198 refinements of the battery with copies, NCS was
// never kept at any other).
// The B factors are refined first at the middle rung only, and at the others where that comes within two
// ties of holding them there: on 364 refinements of the battery B held was kept by 281, and the rule
// changed the rung kept on 4 (all within 0.006 in R-free). Their X-ray weight is tried at the balanced
// The B factors are refined first at the middle rung only, and at the others unless holding them there
// displaces refining them (on 364 refinements of the battery B held was kept by 281). Their X-ray weight is tried at the balanced
// one and at a quarter and a sixteenth of it: a model whose B factors arrive far from where they refine
// (a prediction's pLDDT) balances a weight that is not the one its B wants, and the lower rungs took
// 0.002-0.010 off R-free on five such starts, nothing off eight deposited models.
@@ -3500,8 +3547,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
if (ladder[i].w == mid && ladder[i].ncs == 0)
(ladder[i].adp == 0 ? held : refined) = &outcomes[i];
const bool refine_b = held == nullptr || refined == nullptr ||
(refined->ran && refined->trial.within_bounds &&
(!held->trial.within_bounds || refined->trial.r_free < held->trial.r_free + 2 * R_FREE_TIE));
(refined->ran && refined->trial.within_bounds && !displaces(held->trial, refined->trial));
auto rest_b = [&](const TrialWeights &l) { return l.ncs == 0 && !first(l); };
if (refine_b && !all_ahead)
run_ladder(rest_b);
@@ -3552,24 +3598,17 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// the B factors' divided and multiplied by four; where there are copies, also the NCS restraints changed:
// switched on where they are off, and in turn tightened and loosened (to off beyond NCS_SIGMA_MAX) where
// they are on.
// A step is taken while it lowers R-free or R-work by more than the tie and its R-free stays within the tie
// of the best so far; the trajectory ends at its best step (on a deposited model, whose free set it was
// fitted to, R-work falls while R-free rises, and it ends where it began).
// The trials of the step that went from the best so far: the final ladder goes from there too, and a trial
// of the same weights from the same model is the same computation.
// The weights of a step are chosen on the free set among its trials (displaces, the weights as they are
// first); the step is taken while that trial advances from where the last one ended (advances), and the
// trajectory ends where the last step taken ended.
// The trials of the step that went from where the trajectory ended: the final ladder goes from there too, and
// a trial of the same weights from the same model is the same computation.
std::vector<TrialOutcome> from_best;
// `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since).
// `label`: the steps' name in the log.
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best, const char *label) {
TrialOutcome best = state;
from_best.clear();
bool state_is_best = true;
// A step taken on R-work alone is followed by one more at most: on a model refined before, R-work falls
// step after step while R-free does not, and the trajectory ends at its best anyway - a model of 490k
// atoms spent six such steps (240 s) on a round it then gave back. (On 28 refinements of 22 deposited
// models and 6 predictions the rule changed nothing.)
int since_best = 0;
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled() && since_best <= 1; step++) {
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) {
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
std::vector<TrialWeights> trials = {{w, adp, ncs_s}};
if (step % 2 == 1 && adp > 0) {
@@ -3588,27 +3627,18 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
std::vector<TrialOutcome> outcomes = RunTrials(trial_slots, trials, refinable, state.pos, state.b, state.aniso,
fresh && step == 0 ? nullptr : &state.fit,
settings.macro_cycles, settings, ncs_p);
if (state_is_best)
from_best = outcomes;
from_best = outcomes;
const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str());
if (k < 0)
break;
const RefinementWeightTrial &t = outcomes[k].trial;
// after a change of the model the first step is taken whatever it gains: it is where the change is
// judged from
const bool gains = (fresh && step == 0) || displaces(t, state.trial) ||
(t.within_bounds && t.r_work < state.trial.r_work - R_FREE_TIE);
if (!gains || !((fresh && step == 0) || (t.within_bounds && t.r_free < best.trial.r_free + R_FREE_TIE)))
if (!((fresh && step == 0) || advances(outcomes[k].trial, state.trial)))
break;
state = std::move(outcomes[k]);
state_is_best = (fresh && step == 0) || displaces(state.trial, best.trial);
since_best = state_is_best ? 0 : since_best + 1;
if (state_is_best) {
best = state;
from_best.clear();
}
from_best.clear();
}
return best;
return state;
};
if (!stopped)
state = trajectory(std::move(state), false, from_best, "step");
@@ -3620,9 +3650,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// map averaged over the members with the free reflections left out (PopulationMap), each residue is taken
// from the member whose atoms sit in the most density (following Nwachukwu et al. (2013) Structure 21,
// 1923-1930); the child's junctions are mended by a macro-cycle
// at the ladder's tightest rung and the whole refined by one trial at the state's weights. The child displaces the state by the rule every trial is held to (geometry within
// bounds, R-free lower beyond the tie): the free set decides once, between two candidates - never among
// the members, whose best on the free set would be the best of N.
// at the ladder's tightest rung and the whole refined by one trial at the state's weights. The child is taken where it
// advances on the work-set target, as a step is (the free set does not see the members).
// The members are at most as many as the trial slots, one wave: on a large model, where the card holds
// few, more members would cost a wave each.
const size_t population = std::min<size_t>(std::max(settings.population, 0), trial_slots.size());
@@ -3651,7 +3680,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
blur += fmt::format("{}{:.0f}", blur.empty() ? "" : "/", b);
logger.Info("Refinement, population: blur {}, kick {:.1f} A: {}", blur.empty() ? "none" : blur,
members[k].kick, outcomes[k].line);
if (outcomes[k].trial.within_bounds && (best < 0 || outcomes[k].trial.nll_free < outcomes[best].trial.nll_free))
if (outcomes[k].trial.within_bounds &&
(best < 0 || WorkTarget(outcomes[k].trial, w, adp) < WorkTarget(outcomes[best].trial, w, adp)))
best = static_cast<int>(k);
}
if (best >= 0) {
@@ -3686,11 +3716,11 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
child = RunTrials(trial_slots, {{w, adp, ncs_s}}, refinable, child[0].pos, child[0].b, state.aniso,
&child[0].fit, settings.macro_cycles, settings, ncs_p);
const RefinementWeightTrial &c = child[0].trial;
const bool kept = child[0].ran && displaces(c, state.trial);
logger.Info("Refinement, population: {} members, the best on the free set R-free {:.4f}, -LL free {:.1f}; "
"crossover from {} members: {}; {} (state R-free {:.4f}, -LL free {:.1f})", outcomes.size(),
outcomes[best].trial.r_free, outcomes[best].trial.nll_free, used.size(), child[0].line,
kept ? "kept" : "not kept", state.trial.r_free, state.trial.nll_free);
const bool kept = child[0].ran && advances(c, state.trial);
logger.Info("Refinement, population: {} members, the best on the work-set target {:.6g}; crossover from "
"{} members: {}; {} (state target {:.6g})", outcomes.size(),
WorkTarget(outcomes[best].trial, w, adp), used.size(), child[0].line, kept ? "kept" : "not kept",
WorkTarget(state.trial, w, adp));
if (kept) {
state = std::move(child[0]);
from_best.clear();
@@ -3701,7 +3731,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// 2b. The solvent build (SolventBuild.h), from where the trajectory ended: waters put in at the difference
// map's peaks, taken out where the maps no longer hold them, and waters and ions relabelled, on the maps
// and the solvent audit of that model; then the solvent alone refined, the rest held, at the weights the
// trajectory ended at. A round is kept where that lowers R-free, and undone otherwise (the build ends
// trajectory ended at. A round is kept where that lowers the work-set target at those weights, and undone otherwise (the build ends
// there); at most SOLVENT_BUILD_ROUNDS of them. What they kept is then refined with the rest by the final
// ladder (the trajectory run again from there gained nothing on twelve sets). Each round thus costs a map,
// an audit and a refinement of the solvent, where a refinement of its own cost the whole weight search
@@ -3881,6 +3911,16 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
ctx.Restore(state.fit);
const RFactors built = ctx.Refit();
const Context::Fit built_fit = ctx.Fitted();
RefinementWeightTrial built_trial = state.trial;
built_trial.r_work = built.r_work;
built_trial.r_free = built.r_free;
MeasureWorkTarget(ctx, topo, ncs_p, state.trial.ncs_sigma, built_trial);
// the round judged on the work-set target at the weights the trajectory ended at
const double target_before = WorkTarget(state.trial, state.trial.weight, state.trial.adp_weight);
auto lowers = [&](const RefinementWeightTrial &t) {
return WorkTarget(t, state.trial.weight, state.trial.adp_weight) <
target_before - TARGET_TOLERANCE * std::abs(target_before);
};
// (B refined, at its balanced weight where the trajectory held B: a new water's starting B is a
// guess; fewer iterations, the waters' few parameters converging fast)
RefinementSettings solvent_settings = settings;
@@ -3893,22 +3933,22 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
ncs_p);
logger.Info("Solvent build, round {}: the solvent refined alone: {}", round, local[0].line);
TrialOutcome after = std::move(local[0]);
if (after.ran && !(after.trial.r_free < state.trial.r_free) && built.r_free < state.trial.r_free) {
logger.Info("Solvent build, round {}: kept as built, R-work {:.4f}, R-free {:.4f}", round,
built.r_work, built.r_free);
if (after.ran && !lowers(after.trial) && lowers(built_trial)) {
logger.Info("Solvent build, round {}: kept as built, R-work {:.4f}, R-free {:.4f}, target {:.6g}",
round, built.r_work, built.r_free,
WorkTarget(built_trial, state.trial.weight, state.trial.adp_weight));
after.pos = changed_pos;
after.b = changed_b;
after.aniso = state.aniso;
after.fit = built_fit;
after.trial.r_work = built.r_work;
after.trial.r_free = built.r_free;
after.trial.geometry = state.trial.geometry;
after.trial.within_bounds = state.trial.within_bounds;
const double adp_weight = after.trial.adp_weight;
after.trial = built_trial;
after.trial.adp_weight = adp_weight;
}
if (!after.ran || !(after.trial.r_free < state.trial.r_free)) {
logger.Info("Solvent build, round {}: not kept (R-free {:.4f}, before {:.4f})", round,
after.trial.r_free, state.trial.r_free);
b.stop = fmt::format("round {} raised R-free", round);
if (!after.ran || !lowers(after.trial)) {
logger.Info("Solvent build, round {}: not kept (target {:.6g}, before {:.6g})", round,
WorkTarget(after.trial, state.trial.weight, state.trial.adp_weight), target_before);
b.stop = fmt::format("round {} did not lower the target", round);
// undone, in every slot
for (size_t sl = 0; sl < trial_slots.size(); sl++) {
for (auto u = undo[sl].rbegin(); u != undo[sl].rend(); ++u) {
@@ -3960,10 +4000,9 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// side chain the field can search put in the rotamer of least energy - the -ln L of the working
// reflections at the fit as it stands, plus the refinement's own repulsion term at its X-ray weight -
// given the others, or pruned to CB where no rotamer fits; then one step at the trajectory's weights from
// the field, judged against the same step from the model as it was (the trajectory's end is the best of
// many trials, and a step from it alone regresses from it as often as not): kept where both its R-free
// and its -ln L of the free reflections are lower, the geometry within bounds, undone otherwise. The field
// never sees the free reflections. A model whose side chains the data already hold moves nothing and
// the field, judged against the same step from the model as it was: kept where it advances on the
// work-set target from that step (advances), undone otherwise. Neither the field nor its test sees the
// free reflections. A model whose side chains the data already hold moves nothing and
// costs the sweep alone. The atoms it prunes are left at occupancy 0, as the solvent build's placeholders
// are, and held.
if (settings.side_chain_field && !stopped && state.ran && !ctx.Cancelled() && !ctx.refl.Twinned() &&
@@ -4110,11 +4149,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
if (topo.refinable[i])
refinable.push_back(static_cast<int>(i));
TrialOutcome after = one_step(moved);
const bool kept = after.ran && after.trial.within_bounds && after.trial.r_free < unchanged.trial.r_free &&
after.trial.nll_free < unchanged.trial.nll_free;
logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}; from the model as it "
"was R-free {:.4f}, -LL free {:.1f}: {}", after.trial.r_free, after.trial.nll_free,
unchanged.trial.r_free, unchanged.trial.nll_free, kept ? "kept" : "not kept");
const bool kept = after.ran && unchanged.ran && advances(after.trial, unchanged.trial);
logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}, target {:.6g}; from "
"the model as it was R-free {:.4f}, -LL free {:.1f}, target {:.6g}: {}", after.trial.r_free,
after.trial.nll_free, WorkTarget(after.trial, after.trial.weight, after.trial.adp_weight),
unchanged.trial.r_free, unchanged.trial.nll_free,
WorkTarget(unchanged.trial, after.trial.weight, after.trial.adp_weight), kept ? "kept" : "not kept");
if (kept) {
result.side_chain_field_kept = true;
result.side_chain_field_moved = static_cast<int>(field.moves.size()) - pruned;
@@ -4132,8 +4172,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
}
// 3. The final weights: a short ladder from where the trajectory ended, tighter geometry first, its choice
// kept where it displaces that. A rung the step from there tried already is taken from that step.
// 3. The final weights: a short ladder from where the trajectory ended, the weight as it is first, its choice
// on the free set taken where it advances from there (advances). A rung the step from there tried already is taken from that step.
if (!stopped && state.ran && !ctx.Cancelled()) {
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
std::vector<TrialWeights> trials;
@@ -4156,21 +4196,21 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
if (!outcomes[i].ran)
outcomes[i] = std::move(ran[m++]);
const int k = choose(outcomes, "final");
if (k >= 0 && displaces(outcomes[k].trial, state.trial))
if (k >= 0 && advances(outcomes[k].trial, state.trial))
state = std::move(outcomes[k]);
}
// 4. The displacement model, decided late, on the free set: from where the weights ended, TLS bodies (the
// model's own groups, one per chain, the chains in TLSMD segments) with a residual B, and a U per atom at
// its balanced weight and a quarter of it - each trial as long as a step, on the positions where the weights
// ended, held, so that the trials differ in the displacement model alone; the model chosen then takes a
// step with the positions refined, kept where it displaces its trial (against trials a macro-cycle longer
// step with the positions refined, kept where it advances on the work-set target from its trial (against trials a macro-cycle longer
// that refined the positions too, on 50 refinements of the battery R-free 0.001 lower on average and none
// higher by more than 0.0015, at 78% of the evaluations). Where the geometry is out of bounds there, which
// held positions cannot mend, each trial refines them too, a macro-cycle longer than a step, and no step
// follows. One displaces the one before it (the first
// where the weights ended, the B factors as they are) only where its geometry is within bounds and both its
// R-free (beyond the tie) and the -ln L of the free reflections are lower: the free set is the test of the
// parameters it adds.
// where the weights ended, the B factors as they are) only where its geometry is within bounds and the free
// set prefers it (FreeSetPrefers: its -ln L of the free reflections lower, its R-free not higher beyond its
// standard error): the free set is the test of the parameters it adds.
// Following Merritt (2012) Acta Cryst. D68, 468-477 (the displacement model the data support, decided by test)
// The trials that would not change the choice are left out: the model's own TLS groups are tried only where
// one body per chain or the finest cut lowers R-free, the cuts between those only where also the finest
@@ -4298,8 +4338,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
continue;
logger.Info("Refinement, displacements: {}", outcomes[k].line);
result.adp_trials.push_back(t);
if (t.within_bounds && (!trial(best).within_bounds ||
(t.r_free < trial(best).r_free - R_FREE_TIE && t.nll_free < trial(best).nll_free)))
if (displaces(t, trial(best)))
best = static_cast<int>(k);
}
// (the model where the weights ended keeps the displacement model a round before chose for it)
@@ -4311,7 +4350,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
if (best >= 0 && hold) {
// the model chosen refined on with the positions, a step from where its trial ended, kept where it
// displaces that
// advances on the work-set target from there
AdpHypothesis chosen = *which[best];
chosen.bodies = state.tls;
chosen.tls_b = state.tls_b;
@@ -4320,7 +4359,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
ncs_p, {&chosen});
if (on[0].ran) {
logger.Info("Refinement, displacements, positions refined: {}", on[0].line);
if (displaces(on[0].trial, state.trial))
if (advances(on[0].trial, state.trial))
state = std::move(on[0]);
}
}
@@ -4332,8 +4371,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
// weights on its B factors, the ladder (1.) from it with its displacements as they are (the U of TLS or
// of the anisotropic model held, B held or refined), the trajectory (2.) from the rung kept, and the
// displacement hypotheses (4.) from where that ends - the TLS bodies refined from the B factors the first
// choice left, which they could not be from the B factors as given. A round is kept where it lowers R-free
// beyond the tie and undone otherwise, RefinementSettings::rounds in all: from a placed prediction, whose B
// choice left, which they could not be from the B factors as given. A round is kept where it advances as a step
// does (advances) and undone otherwise, RefinementSettings::rounds in all: from a placed prediction, whose B
// factors are its pLDDT (6 A^2 under an overall B of 45), the first round balanced the B factors' weight
// a hundred times too stiff - the B similarity's gradient goes as 1 / B - and chose the middle rung with
// NCS; the second chose the tightest rung without, and from the B factors the first left, its TLS bodies
@@ -4402,23 +4441,27 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
}
const std::string adp_before = result.adp_model;
displacements(own_tls.empty() ? input_tls : own_tls);
const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE;
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free,
kept ? "" : ", not kept");
const bool kept = advances(state.trial, before.trial);
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}, target {:.6g} -> {:.6g}{}", round,
before.trial.r_free, state.trial.r_free,
WorkTarget(before.trial, state.trial.weight, state.trial.adp_weight),
WorkTarget(state.trial, state.trial.weight, state.trial.adp_weight), kept ? "" : ", not kept");
if (!kept) {
state = before;
result.adp_model = adp_before;
break;
}
}
// 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every
// trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what
// comes out is never worse on the free set than what went in - where the free set is free to the model as
// given (RefinementSettings::as_given_competes).
// 5. The model as given, first in order of trust: the refinement is kept where it displaces it on the free set
// as a weight does (displaces), else the model is given back as it came (the solvent build's edits undone),
// so that what comes out is never worse on the free set than what went in - where the free set is free to
// the model as given (RefinementSettings::as_given_competes). The work-set target cannot make this choice:
// the refinement minimised it from the model as given.
if (settings.as_given_competes && state.ran && !ctx.Cancelled() && !displaces(state.trial, given.trial)) {
logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}), the refinement did not "
"displace it (R-work {:.4f}, R-free {:.4f})", given.trial.r_work, given.trial.r_free,
state.trial.r_work, state.trial.r_free);
logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f}), the "
"refinement did not displace it (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f})",
given.trial.r_work, given.trial.r_free, given.trial.nll_free, state.trial.r_work,
state.trial.r_free, state.trial.nll_free);
size_t a = 0, k = 0;
for (gemmi::Chain &ch : ctx.Model().chains)
for (gemmi::Residue &res : ch.residues) {
@@ -160,6 +160,9 @@ struct RefinementWeightTrial {
double ncs_rmsd = NAN; // the copies' C-alpha (P) rmsd after superposition, where there are copies
double r_work = 1, r_free = 1;
double nll_free = NAN; // -ln L of the free reflections
// The work-set target's terms where the trial ended (RefineOnce's WorkTarget): -ln L of the working reflections,
// the restraints' sum of the geometry (with the NCS restraints at ncs_sigma) and that of the B similarity.
double nll_work = NAN, e_geometry = NAN, e_b = NAN;
GeometryStatistics geometry;
bool within_bounds = false; // the geometry bounds the choice is made inside
std::string adp; // the displacement model: "isotropic", "TLS ..." or "anisotropic"