mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 87 Wiki Activity

Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms

Browse Source 
- B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two
  isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2.
  kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list
  sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these
  equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates
  included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU.
- The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on
  the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held
  and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the
  final ladder leave it.
- Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the
  solvent's B (F_model unchanged), so the two do not drift against each other.
Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the
median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265
vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time.

Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to
convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311),
11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the
model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144)
This commit is contained in:
leonarski_f committed 2026-10-26 01:38:24 +01:00
1 parent 6369a263b1
commit bda90dbdd5
7 files changed
+242 -130
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

image_analysis/structure_refinement/Refinement.cpp
+139 -82
View File
@@ -266,6 +266,34 @@ struct Context {
return ComputeRFactors(refl, fmodel);
}
// The overall isotropic B of the last Refit() moved into the atoms, and into the solvent's B so that F_model stays
// as it is (but where an atom meets a bound of B): left in the scale, it and the atoms' B drift against each
// other as the B factors are refined - and the B restraints, in proportion to B, depend on which holds it.
// As phenix.refine moves the overall B into the atoms (Afonine et al. (2012) Acta Cryst. D68, 352-367)
void MoveOverallB() {
const gemmi::Mat33 b = cell.orth.mat.multiply(b_star.as_mat33()).multiply(cell.orth.mat.transpose());
const double b_iso = (b[0][0] + b[1][1] + b[2][2]) / 3.0;
double b_min = b_ceiling;
for (size_t i = 0; i < atoms.size(); i++)
if (rider_parent.empty() || rider_parent[i] < 0) {
atoms[i]->b_iso = static_cast<float>(std::clamp(atoms[i]->b_iso + b_iso, B_FLOOR, b_ceiling));
b_min = std::min<double>(b_min, atoms[i]->b_iso);
}
MoveRiders();
b_floor = std::max(B_FLOOR, b_min - B_STEP_DOWN);
engine->SetBlur(RefinementBlur(cell, *sg, d_min, b_floor));
// b_iso I in Cartesian is b_iso F F^T in the reciprocal basis
const gemmi::Mat33 f = cell.frac.mat.multiply(cell.frac.mat.transpose());
b_star = {b_star.u11 - b_iso * f[0][0], b_star.u22 - b_iso * f[1][1], b_star.u33 - b_iso * f[2][2],
b_star.u12 - b_iso * f[0][1], b_star.u13 - b_iso * f[0][2], b_star.u23 - b_iso * f[1][2]};
b_sol += b_iso;
for (size_t m = 0; m < refl.Size(); m++) {
rows.k_total[m] *= static_cast<float>(std::exp(b_iso * refl.inv_d2[m] / 4.0));
rows.fmask[m] *= static_cast<float>(std::exp(-b_iso * refl.inv_d2[m] / 4.0));
}
engine->SetRowModel(rows);
}
// Every term of the score of the atoms as they are, at the scale, the solvent and the likelihood parameters
// of the last Refit().
RefinementScore Score() {
@@ -448,8 +476,22 @@ public:
AtomTarget(Context &ctx, Mode mode, const std::vector<int> &param_atoms, double weight, int restraint_set)
: ctx_(ctx), mode_(mode), param_atoms_(param_atoms), weight_(weight), restraint_set_(restraint_set) {}
bool Evaluate(const double *p, double *cost, double *gradient) const {
const RefinementStage stage(mode_ == Mode::Xyz ? "eval.xyz" : "eval.b");
// The parameters of the atoms as they are: x y z per atom, or B per atom.
std::vector<double> Get() const {
std::vector<double> p;
for (int i : param_atoms_) {
const gemmi::Atom &a = *ctx_.atoms[i];
if (mode_ == Mode::Xyz) {
p.push_back(a.pos.x);
p.push_back(a.pos.y);
p.push_back(a.pos.z);
} else
p.push_back(a.b_iso);
}
return p;
}
// The atoms set to the parameters, their riders with them.
void Set(const double *p) const {
for (size_t k = 0; k < param_atoms_.size(); k++) {
gemmi::Atom &a = *ctx_.atoms[param_atoms_[k]];
if (mode_ == Mode::Xyz)
@@ -458,32 +500,36 @@ public:
a.b_iso = static_cast<float>(std::clamp(p[k], ctx_.b_floor, ctx_.b_ceiling));
}
ctx_.MoveRiders();
}
// the most a parameter moves in the first trial of a line search
double MaxStep() const { return mode_ == Mode::Xyz ? MAX_STEP_XYZ : MAX_STEP_B; }
bool Evaluate(const double *p, double *cost, double *gradient) const {
const RefinementStage stage(mode_ == Mode::Xyz ? "eval.xyz" : "eval.b");
Set(p);
ctx_.evaluations++;
std::vector<std::array<double, 4>> gx, gr;
double xray = 0, restraints = 0;
{
const RefinementStage s1("eval.engine");
if (restraint_set_ >= 0)
xray = ctx_.engine->EvaluateWithRestraints(ctx_.Model(), restraint_set_, gradient != nullptr ? &gx : nullptr,
gradient != nullptr ? &gr : nullptr, restraints);
else
xray = ctx_.engine->Evaluate(ctx_.Model(), gradient != nullptr ? &gx : nullptr);
xray = ctx_.engine->EvaluateWithRestraints(ctx_.Model(), restraint_set_, gradient != nullptr ? &gx : nullptr,
gradient != nullptr ? &gr : nullptr, restraints);
}
if (gradient != nullptr)
ctx_.FoldRiders(gx);
*cost = weight_ * xray + restraints;
if (gradient != nullptr)
for (size_t k = 0; k < param_atoms_.size(); k++) {
const int i = param_atoms_[k];
if (mode_ == Mode::Xyz)
for (int c = 0; c < 3; c++)
gradient[3 * k + c] = weight_ * gx[i][c] + (restraint_set_ >= 0 ? gr[i][c] : 0.0);
else {
// held at a bound, B does not move further out
const bool held = (p[k] <= ctx_.b_floor && gx[i][3] > 0) || (p[k] >= ctx_.b_ceiling && gx[i][3] < 0);
gradient[k] = held ? 0.0 : weight_ * gx[i][3] + (restraint_set_ >= 0 ? gr[i][3] : 0.0);
}
if (gradient == nullptr)
return std::isfinite(*cost);
ctx_.FoldRiders(gx);
for (size_t k = 0; k < param_atoms_.size(); k++) {
const int i = param_atoms_[k];
if (mode_ == Mode::Xyz)
for (int c = 0; c < 3; c++)
gradient[3 * k + c] = weight_ * gx[i][c] + gr[i][c];
else {
// held at a bound, B does not move further out
const bool held = (p[k] <= ctx_.b_floor && gx[i][3] > 0) || (p[k] >= ctx_.b_ceiling && gx[i][3] < 0);
gradient[k] = held ? 0.0 : weight_ * gx[i][3] + gr[i][3];
}
}
return std::isfinite(*cost);
}
@@ -495,10 +541,10 @@ private:
int restraint_set_;
};
// The minimiser's diagonal for the B factors of `param_atoms` (RefinementMinimiser::SetDiagonal): weight times
// the X-ray target's curvature in B plus the B restraints', each element at least a thousandth of the median. The
// positions take none: their restraints' curvature is that of stiff bonds and angles, which move atoms together,
// so as a diagonal it slows rather than scales the minimiser.
// The minimiser's diagonal for the B factors of `param_atoms` (RefinementMinimiser::SetDiagonal): weight times the
// X-ray target's curvature in B plus the B restraints', each element at least a thousandth of the median. The
// positions take none: their restraints' curvature is that of stiff bonds and angles, which move atoms together, so
// as a diagonal it slows rather than scales the minimiser.
std::vector<double> BDiagonal(const std::vector<double> &xray, const std::vector<double> &restraints,
const std::vector<int> &param_atoms, double weight) {
std::vector<double> h;
@@ -512,44 +558,18 @@ std::vector<double> BDiagonal(const std::vector<double> &xray, const std::vector
return h;
}
// The curvature of the B factors the minimiser scales its steps by: the X-ray target's (XrayBCurvatureShape() times
// CalibrateBCurvature()) and the B restraints' (BRestraintCurvature()), per atom.
struct BCurvature {
std::vector<double> xray, restraints;
};
// `minimiser` carries its memory from the call before; `diagonal` (B only) the curvature it scales the steps by.
void Minimise(Context &ctx, AtomTarget::Mode mode, const std::vector<int> &param_atoms, double weight,
int restraint_set, int iterations, RefinementMinimiser &minimiser, std::vector<double> diagonal = {}) {
if (param_atoms.empty())
// `minimiser` carries its memory from the call before; `diagonal` is the curvature it scales the steps by (none: 1).
void Minimise(Context &ctx, const AtomTarget &target, int iterations, RefinementMinimiser &minimiser,
std::vector<double> diagonal = {}) {
std::vector<double> p = target.Get();
if (p.empty())
return;
const RefinementStage stage(mode == AtomTarget::Mode::Xyz ? "minimise.xyz" : "minimise.b");
std::vector<double> p;
for (int i : param_atoms) {
const gemmi::Atom &a = *ctx.atoms[i];
if (mode == AtomTarget::Mode::Xyz) {
p.push_back(a.pos.x);
p.push_back(a.pos.y);
p.push_back(a.pos.z);
} else
p.push_back(a.b_iso);
}
const AtomTarget target(ctx, mode, param_atoms, weight, restraint_set);
minimiser.SetDiagonal(diagonal.empty() ? std::vector<double>(p.size(), 1.0) : std::move(diagonal));
minimiser.Minimise([&](const std::vector<double> &x, double &f, std::vector<double> &g) {
return target.Evaluate(x.data(), &f, g.data());
},
p, iterations, MINIMISER_F_TOLERANCE, mode == AtomTarget::Mode::Xyz ? MAX_STEP_XYZ : MAX_STEP_B,
[&]() { return ctx.Cancelled(); });
// the atoms where the solver ended, not where its last line search probed
for (size_t k = 0; k < param_atoms.size(); k++) {
gemmi::Atom &a = *ctx.atoms[param_atoms[k]];
if (mode == AtomTarget::Mode::Xyz)
a.pos = gemmi::Position(p[3 * k], p[3 * k + 1], p[3 * k + 2]);
else
a.b_iso = static_cast<float>(std::clamp(p[k], ctx.b_floor, ctx.b_ceiling));
}
ctx.MoveRiders();
p, iterations, MINIMISER_F_TOLERANCE, target.MaxStep(), [&]() { return ctx.Cancelled(); });
target.Set(p.data()); // where the minimiser ended, not where its last line search probed
}
// Groups of atoms that move as rigid bodies (one per chain) against the X-ray target: per body an angle-axis
@@ -742,6 +762,28 @@ double BalancedWeight(Context &ctx, const std::vector<int> &refinable) {
return nx > 0 ? std::sqrt(ng / nx) : 1.0;
}
// The B factors' X-ray weight that balances the two gradients, as BalancedWeight() does for the positions: the B
// similarity's gradient on the model with its B shaken by 20% (a flat B, as a model may come, gives none) over the
// X-ray target's.
double BalancedADPWeight(Context &ctx, const RestraintSet &b_restraints, const std::vector<int> &refinable) {
std::vector<std::array<double, 4>> gx;
ctx.engine->Evaluate(ctx.Model(), &gx);
ctx.FoldRiders(gx);
std::vector<std::array<double, 4>> xyzb = ctx.XyzB();
std::mt19937 rng(2);
std::normal_distribution<double> shake(0.0, 0.2);
for (std::array<double, 4> &a : xyzb)
a[3] *= std::exp(shake(rng));
std::vector<std::array<double, 4>> gr(xyzb.size(), {0, 0, 0, 0});
EvaluateRestraints(b_restraints, xyzb, &gr);
double nx = 0, ng = 0;
for (int i : refinable) {
nx += gx[i][3] * gx[i][3];
ng += gr[i][3] * gr[i][3];
}
return nx > 0 ? std::sqrt(ng / nx) : 1.0;
}
// The model's map coefficients and its difference map, weighted by the refinement's own likelihood: m its figure
// of merit and D its D (2mFo - DFc, mFo - DFc).
void MakeMaps(Context &ctx, const std::vector<MergedReflection> &merged, double wavelength_A, RefinementResult &result) {
@@ -892,6 +934,15 @@ RestraintSet RestraintsNow(Context &ctx, const RefinementTopology &topo) {
return set;
}
// The B similarity of the model as it is: bonded and angle-related pairs, and atoms in contact.
RestraintSet BRestraintsNow(Context &ctx, const RefinementTopology &topo) {
const RefinementStage stage("b_restraints");
RestraintSet set = topo.adp;
set.Append(BuildBContacts(topo, ctx.Model(), ctx.cell, *ctx.sg, ctx.nthreads));
set.Finalize(ctx.atoms.size());
return set;
}
// Where a weight trial runs: a model with its topology and its engine.
struct TrialSlot {
Context *ctx;
@@ -989,9 +1040,9 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
const RefinementStage stage(adp_w > 0 ? "rung.b" : "rung.xyz");
ctx.SetPositions(pos0);
ctx.SetB(b0);
// one memory for the positions and one for the B factors, kept from macro-cycle to macro-cycle
// the positions and the B factors in turn, each with a memory of its own kept from macro-cycle to macro-cycle
const AtomTarget xyz(ctx, AtomTarget::Mode::Xyz, refinable, w, 0), b(ctx, AtomTarget::Mode::B, refinable, adp_w, 1);
RefinementMinimiser xyz_minimiser, b_minimiser;
BCurvature curvature;
double xray_scale = 0;
for (int cycle = 0; cycle < cycles && !ctx.Cancelled(); cycle++) {
PlaceRiders(ctx, topo);
@@ -999,24 +1050,28 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector<int> &refinable, const s
ctx.Restore(*fit);
else
ctx.Refit();
if (adp_w > 0)
ctx.MoveOverallB();
{
const RestraintSet set = RestraintsNow(ctx, topo);
const RefinementStage s1("set_restraints");
ctx.engine->SetRestraints(0, set);
}
Minimise(ctx, AtomTarget::Mode::Xyz, refinable, w, 0, settings.xyz_iterations, xyz_minimiser);
{
const RefinementStage s2("minimise.xyz");
Minimise(ctx, xyz, settings.xyz_iterations, xyz_minimiser);
}
if (adp_w > 0) {
{
const RefinementStage s2("curvature");
curvature.xray = ctx.XrayBCurvatureShape();
if (cycle == 0)
xray_scale = ctx.CalibrateBCurvature(refinable, curvature.xray);
for (double &x : curvature.xray)
x *= xray_scale;
curvature.restraints = BRestraintCurvature(topo.adp, ctx.XyzB());
}
Minimise(ctx, AtomTarget::Mode::B, refinable, adp_w, 1, settings.b_iterations, b_minimiser,
BDiagonal(curvature.xray, curvature.restraints, refinable, adp_w));
const RestraintSet b_set = BRestraintsNow(ctx, topo);
ctx.engine->SetRestraints(1, b_set);
std::vector<double> xray = ctx.XrayBCurvatureShape();
if (cycle == 0)
xray_scale = ctx.CalibrateBCurvature(refinable, xray);
for (double &x : xray)
x *= xray_scale;
const RefinementStage s3("minimise.b");
Minimise(ctx, b, settings.b_iterations, b_minimiser,
BDiagonal(xray, BRestraintCurvature(b_set, ctx.XyzB()), refinable, adp_w));
}
}
PlaceRiders(ctx, topo);
@@ -1228,14 +1283,15 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
RestraintSet repulsions = BuildRepulsions(topo, ctx.Model(), cell, sg, 0.0, ctx.nthreads);
result.geometry_start = MeasureGeometry(topo.geometry, repulsions, ctx.XyzB(), &topo.metal_bonds);
}
double w0 = 0;
double w0 = 0, adp_w0 = 0;
{
const RefinementStage stage("balanced_weight");
w0 = BalancedWeight(ctx, refinable);
adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable);
}
logger.Info("Refinement: geometry as given rmsd bonds {:.4f} A, angles {:.2f} deg, {} clashes; balanced X-ray "
"weight {:.3g}", result.geometry_start.rmsd_bonds, result.geometry_start.rmsd_angles,
result.geometry_start.clashes, w0);
"weights {:.3g} (positions), {:.3g} (B)", result.geometry_start.rmsd_bonds,
result.geometry_start.rmsd_angles, result.geometry_start.clashes, w0, adp_w0);
const std::vector<gemmi::Position> pos0 = ctx.Positions();
const std::vector<double> b0 = ctx.B();
@@ -1290,11 +1346,11 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
const double w_max = w0 * settings.max_weight;
// 1. The weights from the placed model: the ladder's rungs, tighter geometry first, each with the B factors
// held and refined.
// held and refined at their own balanced weight.
std::vector<std::pair<double, double>> ladder;
for (double rung : settings.weight_ladder)
for (double adp : settings.adp_ladder)
ladder.emplace_back(w0 * rung, w0 * rung * adp);
ladder.emplace_back(w0 * rung, adp_w0 * adp);
std::vector<TrialOutcome> outcomes = RunTrials(trial_slots, ladder, refinable, pos0, b0, nullptr,
settings.macro_cycles, settings);
int k = choose(outcomes, "ladder");
@@ -1303,15 +1359,16 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
state = std::move(outcomes[k]);
// 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they
// are and the X-ray term's weight raised - in turn the positions' (doubled) and the B factors' (times
// four) - until a step lowers neither R-work nor R-free by more than the tie.
// are and, in turn, the positions' X-ray weight doubled and the B factors' divided and multiplied by four,
// until a step lowers neither R-work nor R-free by more than the tie.
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) {
const double w = state.trial.weight, adp = state.trial.adp_weight;
std::vector<std::pair<double, double>> trials = {{w, adp}};
if (step % 2 == 1 && adp > 0)
if (step % 2 == 1 && adp > 0) {
trials.emplace_back(w, adp / 4);
trials.emplace_back(w, 4 * adp);
else if (2 * w <= w_max)
trials.emplace_back(2 * w, 2 * adp);
} else if (2 * w <= w_max)
trials.emplace_back(2 * w, adp);
outcomes = RunTrials(trial_slots, trials, refinable, state.pos, state.b, &state.fit, settings.macro_cycles,
settings);
k = choose(outcomes, fmt::format("step {}", step + 1).c_str());
@@ -1330,7 +1387,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector<M
std::vector<std::pair<double, double>> trials;
for (double f : settings.final_ladder)
if (w * f <= w_max)
trials.emplace_back(w * f, adp * f);
trials.emplace_back(w * f, adp);
outcomes = RunTrials(trial_slots, trials, refinable, state.pos, state.b, &state.fit, settings.macro_cycles,
settings);
k = choose(outcomes, "final");