From 805c3a14c39bb085569f643da5d81e7da39749c4 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 08:04:28 +0100 Subject: [PATCH 01/10] Refinement: the free set decides the weights and the displacement model only The free set now decides only the X-ray weights of the positions and the B factors, the NCS restraints' sigma, the displacement model (isotropic / TLS cuts / U per atom), the reading of the model's TLS groups, and whether the refinement displaces the model as given. Each of these takes a candidate over the incumbent only where both free measures agree: its -ln L of the free reflections lower and its R-free not higher by more than the standard error of the incumbent's, R_free / sqrt(2 N_free) (Tickle, Laskowski & Moss 2000), in place of the fixed R-free tie of 0.001 (which alone decided most choices). The approach of requiring free measures to agree follows PDB-REDO (Joosten et al. 2012, 2014). Every other decision now uses the work-set target - w (-ln L work) + E_geometry (with NCS at its sigma), plus w_B (-ln L work) + E_B where B is refined - at the weights the two candidates share, a step taken where it lowers that by more than 1e-4 relative and the geometry stays within bounds where it was: the trajectory's steps (which now end where the last step taken ended, not at the best step on the free set), the final ladder's step, the step after the displacement choice, the second round, the solvent build's rounds, the side-chain field's kept test and the population's child. A candidate at other weights is not compared on a target that changed with them (the positions and B minimise separate targets, whose sum a weight change can raise while it lowers each) and is left to the free-set rule. The placement's rigid bodies and B per residue are kept where they lower -ln L work. Each weight trial now records -ln L work and the restraint sums (RefinementWeightTrial::nll_work, e_geometry, e_b) and logs its target. Not changed: the twin-law choice (a paired free-set z against its null, a model hypothesis test with its own standard error), MR and model validation. No k-fold cross-validation yet for free sets under 500 reflections. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- .../structure_refinement/Refinement.cpp | 255 ++++++++++-------- .../structure_refinement/Refinement.h | 3 + 2 files changed, 152 insertions(+), 106 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index ac0ffc781..db574ae51 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -91,8 +91,16 @@ constexpr double REPULSION_MARGIN = 0.5; // A: pairs this much beyond their co constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0; // The minimiser's iterations per macro-cycle where the solvent build refines the solvent alone. constexpr int SOLVENT_ITERATIONS = 10; -// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken. -constexpr double R_FREE_TIE = 0.001; +// The free set decides the weights and the displacement model alone, and only where both of its measures agree: a +// candidate is preferred to the incumbent where its -ln L of the free reflections is lower and its R-free not higher +// by more than the standard error of the incumbent's, R_free / sqrt(2 N_free). +// Following Tickle, Laskowski & Moss (2000) Acta Cryst. D56, 442-450 (the standard error of R-free), as PDB-REDO +// decides on more than one free measure, Joosten, Long, Murshudov & Perrakis (2014) IUCrJ 1, 213-220 +bool FreeSetPrefers(double r_free, double nll_free, double r_free_b, double nll_free_b, int n_free) { + return nll_free < nll_free_b && r_free < r_free_b + r_free_b / std::sqrt(2.0 * n_free); +} +// Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative. +constexpr double TARGET_TOLERANCE = 1e-4; // The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each // after it from the model the one before ended with. // The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this, @@ -1564,9 +1572,10 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) { if (body.size() >= 50) bodies.push_back(body); } - // Each step is kept only where it lowers R-free beyond the tie: a deposited model was refined atom by atom - // already, and moving its chains or shifting its B factors residue by residue then only makes it worse. - RFactors before = ctx.Refit(); + // Each step is kept only where it lowers the -ln L of the working reflections, the target it minimised (the free + // set decides the weights and the displacement model alone). + ctx.Refit(); + double before = ctx.Score().nll_work; if (!bodies.empty()) { const std::vector pos0 = ctx.Positions(); std::vector q(6 * bodies.size(), 0.0); @@ -1577,13 +1586,15 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) { const CancelCheck cancel_check(ctx); ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary); rb->Place(q.data()); - const RFactors after = ctx.Refit(); - if (after.r_free < before.r_free - R_FREE_TIE) { + ctx.Refit(); + const double after = ctx.Score().nll_work; + if (after < before) { result.rigid_bodies = static_cast(bodies.size()); before = after; } else { ctx.SetPositions(pos0); - before = ctx.Refit(); + ctx.Refit(); + before = ctx.Score().nll_work; } } if (group_b) { @@ -1596,7 +1607,8 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) { const CancelCheck cancel_check(ctx); ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary); gb->Place(q.data()); - result.group_b = ctx.Refit().r_free < before.r_free - R_FREE_TIE; + ctx.Refit(); + result.group_b = ctx.Score().nll_work < before; if (!result.group_b) ctx.SetB(b0); } @@ -2246,6 +2258,25 @@ struct TrialOutcome { int evaluations = 0; }; +// The work-set target's terms of the model as it stands (RefinementWeightTrial), at the last Refit()'s fit: what every +// decision but the weights' and the displacement model's is made on (WorkTarget). +void MeasureWorkTarget(Context &ctx, const RefinementTopology &topo, const NcsCopies *ncs, double ncs_sigma, + RefinementWeightTrial &t) { + const RefinementScore sc = ctx.Score(); + t.nll_work = sc.nll_work; + t.nll_free = sc.nll_free; + const std::vector> xyzb = ctx.XyzB(); + t.e_geometry = EvaluateRestraints(RestraintsNow(ctx, topo, ncs, ncs_sigma), xyzb, nullptr); + t.e_b = EvaluateRestraints(BRestraintsNow(ctx, topo), xyzb, nullptr); +} + +// The work-set target of a trial at the X-ray weights w (positions) and adp (B, 0: held), its NCS restraints at its own +// sigma: what a trial at those weights minimises, the X-ray term of the positions and the geometry, and that of the B +// factors and the B similarity where they are refined. +double WorkTarget(const RefinementWeightTrial &t, double w, double adp) { + return w * t.nll_work + t.e_geometry + (adp > 0 ? adp * t.nll_work + t.e_b : 0.0); +} + // The X-ray target's gradient on the positions, its squared norm over `refinable`, for the atoms as they are. double XrayGradientNorm(Context &ctx, const std::vector &refinable) { std::vector> g; @@ -2471,16 +2502,16 @@ TrialOutcome RunTrial(TrialSlot slot, XyzCalibrations &calibrations, const std:: t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB()); if (hyp != nullptr) t.adp = hyp->name; - t.nll_free = ctx.Score().nll_free; - out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, " - "angles {:.2f} deg{}{}", + MeasureWorkTarget(ctx, topo, ncs, ncs_sigma, t); + out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, target {:.6g}, rmsd bonds " + "{:.4f} A, angles {:.2f} deg{}{}", std::string(tw.blur.empty() ? "" : "blurred, ") + (hyp != nullptr ? hyp->name + ", " : ""), w, kind == AdpHypothesis::Kind::Anisotropic ? "U" : kind == AdpHypothesis::Kind::Tls ? "residual B" : "B", adp_w > 0 ? fmt::format("refined at weight {:.3g}", adp_w) : std::string("held"), ncs == nullptr || ncs->Empty() ? std::string() : ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd) : fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd), - rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), + rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), WorkTarget(t, w, adp_w), t.geometry.rmsd_bonds, t.geometry.rmsd_angles, t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal) : std::string(), @@ -2859,8 +2890,8 @@ double BalancedUWeight(Context &ctx, const URestraints &restraints, const std::v // The displacements of the model as given, read where it has TLS groups whose atoms bring no U of their own: their U // from the groups, its B column taken as the total B (the U's equivalent isotropic B, as phenix.refine writes it) or as -// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - the one of the three whose -// R-free is lowest, beyond the tie in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens +// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - a displacement model, chosen on +// the free set (FreeSetPrefers), in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens // (which ride on their parents' B). Returns the R-factors of the reading kept. // Following Winn, Isupov & Murshudov (2001) Acta Cryst. D57, 122-133 (the B column total or residual to the TLS) RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, Logger &logger, RefinementResult &result) { @@ -2906,6 +2937,7 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, L readings.push_back({result.adp_input == "ANISOU" ? "ANISOU" : "isotropic", b0, aniso0}); int best = -1; std::vector rf; + std::vector nll_free; for (size_t k = 0; k < readings.size(); k++) { ctx.SetB(readings[k].b); ctx.SetAniso(readings[k].aniso); @@ -2914,9 +2946,10 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, L ctx.atoms[i]->b_iso = ctx.atoms[ctx.rider_parent[i]]->b_iso; ctx.ForgetStarts(); rf.push_back(ctx.Refit()); - logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}", tls.size(), - tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free); - if (best < 0 || rf[k].r_free < rf[best].r_free - R_FREE_TIE) + nll_free.push_back(ctx.Score().nll_free); + logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}, -LL free " + "{:.1f}", tls.size(), tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free, nll_free[k]); + if (best < 0 || FreeSetPrefers(rf[k].r_free, nll_free[k], rf[best].r_free, nll_free[best], rf[k].n_free)) best = static_cast(k); } ctx.SetB(readings[best].b); @@ -3426,10 +3459,25 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vectorran && refined->trial.within_bounds && - (!held->trial.within_bounds || refined->trial.r_free < held->trial.r_free + 2 * R_FREE_TIE)); + (refined->ran && refined->trial.within_bounds && !displaces(held->trial, refined->trial)); auto rest_b = [&](const TrialWeights &l) { return l.ncs == 0 && !first(l); }; if (refine_b && !all_ahead) run_ladder(rest_b); @@ -3552,24 +3598,17 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector from_best; // `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since). // `label`: the steps' name in the log. auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best, const char *label) { - TrialOutcome best = state; from_best.clear(); - bool state_is_best = true; - // A step taken on R-work alone is followed by one more at most: on a model refined before, R-work falls - // step after step while R-free does not, and the trajectory ends at its best anyway - a model of 490k - // atoms spent six such steps (240 s) on a round it then gave back. (On 28 refinements of 22 deposited - // models and 6 predictions the rule changed nothing.) - int since_best = 0; - for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled() && since_best <= 1; step++) { + for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) { const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma; std::vector trials = {{w, adp, ncs_s}}; if (step % 2 == 1 && adp > 0) { @@ -3588,27 +3627,18 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector outcomes = RunTrials(trial_slots, trials, refinable, state.pos, state.b, state.aniso, fresh && step == 0 ? nullptr : &state.fit, settings.macro_cycles, settings, ncs_p); - if (state_is_best) - from_best = outcomes; + from_best = outcomes; const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str()); if (k < 0) break; - const RefinementWeightTrial &t = outcomes[k].trial; // after a change of the model the first step is taken whatever it gains: it is where the change is // judged from - const bool gains = (fresh && step == 0) || displaces(t, state.trial) || - (t.within_bounds && t.r_work < state.trial.r_work - R_FREE_TIE); - if (!gains || !((fresh && step == 0) || (t.within_bounds && t.r_free < best.trial.r_free + R_FREE_TIE))) + if (!((fresh && step == 0) || advances(outcomes[k].trial, state.trial))) break; state = std::move(outcomes[k]); - state_is_best = (fresh && step == 0) || displaces(state.trial, best.trial); - since_best = state_is_best ? 0 : since_best + 1; - if (state_is_best) { - best = state; - from_best.clear(); - } + from_best.clear(); } - return best; + return state; }; if (!stopped) state = trajectory(std::move(state), false, from_best, "step"); @@ -3620,9 +3650,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(std::max(settings.population, 0), trial_slots.size()); @@ -3651,7 +3680,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k); } if (best >= 0) { @@ -3686,11 +3716,11 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(i)); TrialOutcome after = one_step(moved); - const bool kept = after.ran && after.trial.within_bounds && after.trial.r_free < unchanged.trial.r_free && - after.trial.nll_free < unchanged.trial.nll_free; - logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}; from the model as it " - "was R-free {:.4f}, -LL free {:.1f}: {}", after.trial.r_free, after.trial.nll_free, - unchanged.trial.r_free, unchanged.trial.nll_free, kept ? "kept" : "not kept"); + const bool kept = after.ran && unchanged.ran && advances(after.trial, unchanged.trial); + logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}, target {:.6g}; from " + "the model as it was R-free {:.4f}, -LL free {:.1f}, target {:.6g}: {}", after.trial.r_free, + after.trial.nll_free, WorkTarget(after.trial, after.trial.weight, after.trial.adp_weight), + unchanged.trial.r_free, unchanged.trial.nll_free, + WorkTarget(unchanged.trial, after.trial.weight, after.trial.adp_weight), kept ? "kept" : "not kept"); if (kept) { result.side_chain_field_kept = true; result.side_chain_field_moved = static_cast(field.moves.size()) - pruned; @@ -4132,8 +4172,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector trials; @@ -4156,21 +4196,21 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0 && displaces(outcomes[k].trial, state.trial)) + if (k >= 0 && advances(outcomes[k].trial, state.trial)) state = std::move(outcomes[k]); } // 4. The displacement model, decided late, on the free set: from where the weights ended, TLS bodies (the // model's own groups, one per chain, the chains in TLSMD segments) with a residual B, and a U per atom at // its balanced weight and a quarter of it - each trial as long as a step, on the positions where the weights // ended, held, so that the trials differ in the displacement model alone; the model chosen then takes a - // step with the positions refined, kept where it displaces its trial (against trials a macro-cycle longer + // step with the positions refined, kept where it advances on the work-set target from its trial (against trials a macro-cycle longer // that refined the positions too, on 50 refinements of the battery R-free 0.001 lower on average and none // higher by more than 0.0015, at 78% of the evaluations). Where the geometry is out of bounds there, which // held positions cannot mend, each trial refines them too, a macro-cycle longer than a step, and no step // follows. One displaces the one before it (the first - // where the weights ended, the B factors as they are) only where its geometry is within bounds and both its - // R-free (beyond the tie) and the -ln L of the free reflections are lower: the free set is the test of the - // parameters it adds. + // where the weights ended, the B factors as they are) only where its geometry is within bounds and the free + // set prefers it (FreeSetPrefers: its -ln L of the free reflections lower, its R-free not higher beyond its + // standard error): the free set is the test of the parameters it adds. // Following Merritt (2012) Acta Cryst. D68, 468-477 (the displacement model the data support, decided by test) // The trials that would not change the choice are left out: the model's own TLS groups are tried only where // one body per chain or the finest cut lowers R-free, the cuts between those only where also the finest @@ -4298,8 +4338,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k); } // (the model where the weights ended keeps the displacement model a round before chose for it) @@ -4311,7 +4350,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0 && hold) { // the model chosen refined on with the positions, a step from where its trial ended, kept where it - // displaces that + // advances on the work-set target from there AdpHypothesis chosen = *which[best]; chosen.bodies = state.tls; chosen.tls_b = state.tls_b; @@ -4320,7 +4359,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector {:.4f}{}", round, before.trial.r_free, state.trial.r_free, - kept ? "" : ", not kept"); + const bool kept = advances(state.trial, before.trial); + logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}, target {:.6g} -> {:.6g}{}", round, + before.trial.r_free, state.trial.r_free, + WorkTarget(before.trial, state.trial.weight, state.trial.adp_weight), + WorkTarget(state.trial, state.trial.weight, state.trial.adp_weight), kept ? "" : ", not kept"); if (!kept) { state = before; result.adp_model = adp_before; break; } } - // 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every - // trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what - // comes out is never worse on the free set than what went in - where the free set is free to the model as - // given (RefinementSettings::as_given_competes). + // 5. The model as given, first in order of trust: the refinement is kept where it displaces it on the free set + // as a weight does (displaces), else the model is given back as it came (the solvent build's edits undone), + // so that what comes out is never worse on the free set than what went in - where the free set is free to + // the model as given (RefinementSettings::as_given_competes). The work-set target cannot make this choice: + // the refinement minimised it from the model as given. if (settings.as_given_competes && state.ran && !ctx.Cancelled() && !displaces(state.trial, given.trial)) { - logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}), the refinement did not " - "displace it (R-work {:.4f}, R-free {:.4f})", given.trial.r_work, given.trial.r_free, - state.trial.r_work, state.trial.r_free); + logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f}), the " + "refinement did not displace it (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f})", + given.trial.r_work, given.trial.r_free, given.trial.nll_free, state.trial.r_work, + state.trial.r_free, state.trial.nll_free); size_t a = 0, k = 0; for (gemmi::Chain &ch : ctx.Model().chains) for (gemmi::Residue &res : ch.residues) { diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index 5dfece5f1..b3ac73777 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -160,6 +160,9 @@ struct RefinementWeightTrial { double ncs_rmsd = NAN; // the copies' C-alpha (P) rmsd after superposition, where there are copies double r_work = 1, r_free = 1; double nll_free = NAN; // -ln L of the free reflections + // The work-set target's terms where the trial ended (RefineOnce's WorkTarget): -ln L of the working reflections, + // the restraints' sum of the geometry (with the NCS restraints at ncs_sigma) and that of the B similarity. + double nll_work = NAN, e_geometry = NAN, e_b = NAN; GeometryStatistics geometry; bool within_bounds = false; // the geometry bounds the choice is made inside std::string adp; // the displacement model: "isotropic", "TLS ..." or "anisotropic" From f73d9505e1ee22ca9a8890e57b88b61f4933ad9e Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 08:04:28 +0100 Subject: [PATCH 02/10] docs: what the refinement's free set decides, and the work-set target Describe the free-set rule (free -ln L lower and R-free within its standard error), the work-set target every other step is taken on, and credit Tickle, Laskowski & Moss (2000) and PDB-REDO (Joosten et al. 2012, 2014). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- docs/ACKNOWLEDGEMENT.md | 14 +++++ docs/CPU_DATA_ANALYSIS.md | 2 +- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 83 +++++++++++++++++----------- docs/RUGNUX_MODEL_AND_REFERENCE.md | 11 ++-- 4 files changed, 73 insertions(+), 37 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index aa2328e36..e1e2e6af7 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -652,6 +652,20 @@ structure refinement with phenix.refine" (2012), Acta Cryst. D68, 352-367 A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures" (1992), Nature 355, 472-475 [doi:10.1038/355472a0](https://doi.org/10.1038/355472a0). +**What the free set decides** — the free set decides the weights and the displacement model alone, and only +where its two measures agree: a candidate is taken where its free $-\ln L$ is lower and its R-free not higher +by more than R-free's standard error, $R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$, from I. J. Tickle, +R. A. Laskowski and D. S. Moss, "Rfree and the Rfree ratio. II. Calculation of the +expected values and variances of cross-validation statistics in macromolecular least-squares refinement" +(2000), Acta Cryst. D56, 442-450 +[doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868); every other step of the refinement +is taken on the work-set target. The approach of deciding on more than one free measure, and of keeping the +free set for few choices, is PDB-REDO's: R. P. Joosten, K. Joosten, G. N. Murshudov and A. Perrakis, +"PDB_REDO: constructive validation, more than just looking for errors" (2012), Acta Cryst. D68, 484-496 +[doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515), and R. P. Joosten, F. Long, +G. N. Murshudov and A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization" (2014), +IUCrJ 1, 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324). + **Robust scaling of the model** — the overall scale and anisotropic B are fitted by iteratively reweighted least squares with the pseudo-Huber weight. P. J. Huber, "Robust estimation of a location parameter" (1964), Ann. Math. Statist. 35, 73-101 diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 84af30f96..a35c12d3f 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -18,7 +18,7 @@ This document describes the crystallographic algorithms implemented in Jungfrauj 12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref), 13. amplitude estimation (French–Wilson) and R-free test-set flagging, 14. optional model-based validation: rigid-body placement of a supplied model (molecular replacement where the rigid body does not place it, or for a predicted model), R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named. -15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight chosen by R-free within geometry bounds, and the unexplained blobs of the refined difference map. +15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight and the displacement model chosen on the free set (free -ln L and R-free agreeing) within geometry bounds, every other step taken on the work-set target, and the unexplained blobs of the refined difference map. The reference is split into five parts, in pipeline order; the section numbers run continuously diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 6fde51f9f..5cb5495f0 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -42,8 +42,8 @@ no twin law is tested, the data carrying the larger symmetry exactly. The refine it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the -groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest -R-free on the model as given (`REFINE_ADP_INPUT`). Occupancies are kept as they are, and +groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, the free set prefers on +the model as given (§15.7; `REFINE_ADP_INPUT`). Occupancies are kept as they are, and so are alternative conformations, each refined with its own restraints. Two stages follow. @@ -51,12 +51,12 @@ Two stages follow. 1. **Placement.** Each chain of at least 50 non-water atoms is moved as a rigid body - an angle-axis rotation about its centroid, scaled by its rms radius so that a unit of rotation moves a typical atom by about an angstrom, and a translation - against the X-ray target (§15.2), by Ceres' L-BFGS for 30 - iterations. Then one B shift per residue. Each of the two is **kept only where it lowers R-free by more - than 0.001**: a deposited model was refined atom by atom already, and moving its chains or shifting its - B factors residue by residue then only makes it worse. The placement stage needs no dictionary. + iterations. Then one B shift per residue. Each of the two is **kept only where it lowers the $-\ln L$ of the + working reflections**, the target it minimised: the free set decides the weights and the displacement model + alone (§15.7). The placement stage needs no dictionary. 2. **Restrained refinement.** Every atom's position and isotropic B, against the X-ray target and the geometry of the dictionary (§15.4-§15.5) - the built-in one with the user's restraint files over it - (`--restraints`) - in weight trials chosen by R-free (§15.7). Residues the + (`--restraints`) - in weight trials chosen on the free set (§15.7). Residues the dictionary does not describe, and residues with an atom it does not know, are **held fixed**: they stay in the structure factors, in the bulk-solvent mask and in the repulsions the refined atoms see, but they do not move. So do atoms of zero occupancy. @@ -344,29 +344,46 @@ $w_B\,T_\mathrm{X\text{-}ray} + E_B$ for the B factors. Rather than fixing $w$ a runs **short refinements at several weights and keeps the one that does best on the free set** - a weight chosen late, on the evidence, instead of early at a rule. +**What the free set decides.** The free set decides the weights - the X-ray weights of the positions and of the B +factors and the NCS restraints' $\sigma$ - the displacement model (§15.7, 6.), the reading of the model's TLS groups +(§15.1) and whether the refinement displaces the model as given (7.), and nothing else. Each of these choices asks +both of its measures: a candidate is preferred to the incumbent only where its $-\ln L$ of the free reflections is +lower **and** its R-free is not higher by more than the standard error of the incumbent's, +$\sigma(R_\mathrm{free}) = R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$ (Tickle, Laskowski & Moss 2000) - one measure +alone, or a fixed tie, picks the noise of a few hundred reflections as often as a better model (PDB-REDO likewise +takes a change only where its free measures agree, Joosten et al. 2014). Every other step - a step of the +trajectory, a round, the placement's rigid bodies and B per residue, the solvent build's rounds, the side-chain +field, the population's child - is taken on the **work-set target**: the target the trial minimises, +$w\,T_\mathrm{X\text{-}ray} + E_\mathrm{geometry}$ (with the NCS restraints at their $\sigma$), plus +$w_B\,T_\mathrm{X\text{-}ray} + E_B$ where the B factors are refined, $T_\mathrm{X\text{-}ray}$ the $-\ln L$ of the +working reflections - at the weights the two candidates share, and taken where it lowers that by more than +$10^{-4}$ of it, relative, and the geometry stays within bounds where it was. A step at other weights (X-ray weights +or NCS $\sigma$) is not compared on a target that changed with them - the positions and the B factors minimise +targets of their own, whose sum a change of weight can raise while it lowers each - and the free set decides it as +it decides every weight. The free set thus tests a +handful of choices, each between a few candidates, and the R-free reported is not the best of the many steps the +refinement took. + The starting weights balance the gradients: $w_0 = |\nabla E_\mathrm{geometry}| / |\nabla T_\mathrm{X\text{-}ray}|$, the geometry's gradient taken on the model shaken by 0.1 Å (a refined model's geometry would give none), and $w_{B,0}$ likewise from the B restraints on the B factors shaken by 20%. Each **trial** is two macro-cycles (the scale, solvent and likelihood refitted, then the positions, then the B factors) from where it starts. A trial is **within bounds** where its rmsd from the dictionary is at most 0.02 Å for bonds and 2° for -angles; one trial **displaces** another where it is within bounds and its R-free is lower by more than 0.001 -(differences smaller than that do not decide, and the trial listed first - on a ladder, the tighter -geometry - is kept). The trials -are: +angles; one trial **displaces** another where it is within bounds and the free set prefers it as above (the trial +listed first - on a ladder, the tighter geometry - is kept otherwise). The trials are: 1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors held and refined at $w_{B,0} \times \{1/16, 1/4, 1\}$ - twelve trials (the B factors are refined first at - the middle rung only, and at the others where that comes within two ties of holding them there). The lower + the middle rung only, and at the others unless holding them there displaces refining them). The lower B weights are for a model whose B factors arrive far from where they refine - a prediction's pLDDT - whose balanced weight is not the one its B wants: they took 0.002-0.010 off R-free on five such starts, and nothing off eight deposited models. 2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and - multiplied by four. A step is taken while it lowers R-free or R-work by more than 0.001 and its R-free - stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model, - whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the - trajectory ends where it began. + multiplied by four. The step's weights are chosen among its trials on the free set; the step is taken while + that trial lowers the work-set target (or, at other weights, displaces where the last step ended), and the + trajectory ends where the last step taken ended. 3. **The population** (off by default, `RefinementSettings::population`: with the minimiser of §15.6 its members took nothing off R-free over 19 refinements that the second round did not), from where the trajectory ended: up to twelve members (at most as many as the trial slots, so that they run in one wave; none below two) refined @@ -380,8 +397,8 @@ are: members see those around it. Their **crossover** is made on one 2mFo-DFc map averaged over the members with the free reflections left out: each residue is taken from the member whose atoms sit in the most density, the junctions are mended by a macro-cycle at the ladder's tightest rung, and the child is refined by one - trial at the trajectory's weights. The child displaces the state by the rule every trial is held to; the - members themselves are never chosen among on the free set, whose best of twelve would be the best of twelve. On + trial at the trajectory's weights. The child is taken where it lowers the work-set target, as a step is; the + free set never sees the members. On 13 predictions placed by MR the child was kept on four (R-free 0.005-0.011 lower) and on one of 22 refined models (0.009 lower, the rest unchanged); the gain, where there is one, comes mostly in the rounds that follow from the child. @@ -402,14 +419,15 @@ are: sweep would otherwise take more than $4 \times 10^8$ evaluations (states times reflections, about 3 s on one card). An FFT gradient cannot carry a side chain 1.5 Å at 1.7 Å resolution; the field, a search, can, and it is scored by the likelihood rather than by a map, which on an - incomplete model rewards whatever fills unmodelled density. The free reflections are never seen by the field: - one step at the trajectory's weights is then taken from it and judged against the same step from the model as - it was, kept only where both its R-free and its $-\ln L$ of the free reflections are lower and the geometry - within bounds, undone otherwise (a model whose side chains the data already hold moves nothing and costs the + incomplete model rewards whatever fills unmodelled density. The free reflections are never seen by the field + nor by its test: one step at the trajectory's weights is then taken from it and judged against the same step + from the model as it was, kept only where it lowers the work-set target and the geometry stays within bounds, + undone otherwise (a model whose side chains the data already hold moves nothing and costs the sweep alone). The log names each side chain moved with its chi angles before and after and the energy the move took off (`REFINE_SIDE_CHAIN_FIELD_MOVED`, `REFINE_SIDE_CHAIN_FIELD_PRUNED`). Twinned data and a model with anisotropic atoms are refined without the field. -5. **The final ladder**: the trajectory's weights times $\{1, 2\}$, kept where one displaces it (the weight halved +5. **The final ladder**: the trajectory's weights times $\{1, 2\}$, chosen on the free set, the choice taken where it + advances from where the trajectory ended as a step does (the weight halved was never kept on 55 refinements, and taking it away changed no choice). 6. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U held), each trial as long as a step on the positions where the weights ended, held - only the displacement @@ -425,8 +443,7 @@ are: reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving out the U weight $\times 4$ changed a choice taken. A trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds - and both its R-free (beyond the tie) and the $-\ln L$ of the free reflections are lower: the free set tests the - parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). + and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). The refined U are written as ANISOU and the TLS groups as `_pdbx_refine_tls*` and `REMARK 3` (`REFINE_ADP_MODEL`, `REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no @@ -437,12 +454,14 @@ are: B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT, the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$), the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the - refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers - R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the + refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken where it advances as a + step does - on the work-set target where its weights are those it started from, on the free set where it chose + others - and undone otherwise; where its ladder and trajectory leave the model where it was, the hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder and one step. No round runs where the refinement does not displace the model as given. -7. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the - rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as +7. **The model as given** is the first candidate in order of trust: where the refinement does not displace it on the + free set (its geometry within bounds and the free set preferring it; the work-set target cannot choose here, the + refinement having minimised it from the model as given), the model is written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never worse on the free set than what went in. This holds only where the free set is free to the model as given: a model placed as given on a free set drawn by the run (in `--mode refine`, a file without free flags) has @@ -453,7 +472,7 @@ without them, and then with them at $\sigma = 0.1$ Å at the B factors' weight t X-ray weight a rung up as often as not, the B factors' choice it never changed on 39 refinements), and every step of the trajectory also tries them changed - switched on where they are off, and in turn tightened ($\sigma$ halved) and loosened ($\sigma$ doubled, off beyond 0.4 Å) where they are on. Listed after their twin without, the restraints are -kept only where they lower R-free by more than the tie. A model without copies runs exactly the trials it ran before. +kept only where the free set prefers them. A model without copies runs exactly the trials it ran before. $w$ never exceeds $w_0/8$. The trials of one set run side by side - on one GPU, on several, or on the CPU's threads - each from the same start and taking nothing from the one before it, so the result does not depend @@ -576,8 +595,8 @@ and removed and waters and ions relabelled, as phenix.refine builds waters withi trajectory (§15.7) ends, a round reads the maps and the audit of that model and changes its solvent; then the solvent alone - its waters and single-atom ions, the rest held - is refined at the weights the trajectory ended at (its B too, at its balanced weight where the trajectory held B, and 10 iterations a macro-cycle), and the round is kept -where that lowers R-free; where it does not but the round as built does (refining the solvent alone can lose what its -sites gained), it +where that lowers the work-set target at those weights (§15.7); where it does not but the round as built does +(refining the solvent alone can lose what its sites gained), it is kept as built; otherwise it is undone. The rest of the model goes on with its B factors held or refined as the trajectory had them. The rounds stop at one that changes nothing, at one undone, or after two (a third gains nothing measurable); the final ladder of §15.7 then @@ -800,6 +819,8 @@ What the refinement does not do yet - each a known gap, not a design choice: - A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7). - A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free). +- I. J. Tickle, R. A. Laskowski & D. S. Moss, "Rfree and the Rfree ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement", *Acta Cryst.* **D56** (2000), 442-450 [doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868) (the standard error of R-free the free set's choices are tied within, §15.7). +- R. P. Joosten, F. Long, G. N. Murshudov & A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization", *IUCrJ* **1** (2014), 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324), and R. P. Joosten, K. Joosten, G. N. Murshudov & A. Perrakis, "PDB_REDO: constructive validation, more than just looking for errors", *Acta Cryst.* **D68** (2012), 484-496 [doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515) (choices taken only where more than one free measure agrees, §15.7). - G. J. Kleywegt, "Use of non-crystallographic symmetry in protein structure refinement", *Acta Cryst.* **D52** (1996), 842-857; I. Usón, E. Pohl, T. R. Schneider, Z. Dauter, A. Schmidt, H.-J. Fritz & G. M. Sheldrick, "1.7 Å structure of the stabilized REIv mutant T39K. Application of local NCS restraints", *Acta Cryst.* **D55** (1999), 1158-1167; R. A. Nicholls, F. Long & G. N. Murshudov, "Low-resolution refinement tools in REFMAC5", *Acta Cryst.* **D68** (2012), 404-417; J. J. Headd et al., "Flexible torsion-angle noncrystallographic symmetry restraints for improved macromolecular structure refinement", *Acta Cryst.* **D70** (2014), 1346-1356 (local NCS restraints with a robust loss, decided by R-free, and Murshudov et al. 2011 above for the copies found by alignment, §15.5, §15.7). - P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9). - N. Naleem, A. Henning-Knechtel, S. Kirmizialtin & P. Auffinger, "Cat_Wiz: a stereochemistry-guided toolkit for locating, diagnosing, and annotating Mg2+ ions in RNA structures", *Nucleic Acids Res.* **54** (2026), gkag120; F. Leonarski, L. D'Ascenzo & P. Auffinger, *Nucleic Acids Res.* **45** (2017), 987-1004 and *RNA* **25** (2019), 173-192; F. Leonarski, A. Henning-Knechtel, S. Kirmizialtin, E. Ennifar & P. Auffinger, *Nucleic Acids Res.* **53** (2025), gkae1148 (the coordination bands, distances and exclusion zone of the solvent audit, §15.10). diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index 1df1f4d77..fe8a04932 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -287,12 +287,13 @@ element and the number of such records of the ion (Mg-O 2.07 A, Na-O 2.42 A, K-O usual coordination), where the model's distance is within 0.33 A of that (Mg-O up to 2.4 A); a record at an outer-sphere distance, a contact without a record, or one to a symmetry mate, is not restrained as a bond. The B factors of bonded atoms and of atoms in contact (waters included) are restrained to each other in proportion to their size. The weights of -the X-ray term against the geometry and against that B restraint are chosen by R-free among short -refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees: three weights from the placed +the X-ray term against the geometry and against that B restraint are chosen on the free set among short +refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees - a candidate taken only where its free +-ln L is lower and its R-free not higher by more than R-free's standard error: three weights from the placed model, each with the B factors held and refined; then a trajectory of further short refinements from the -best, each trying the weights as they are and changed, taken while R-free or R-work falls beyond 0.001 and -R-free stays within 0.001 of its best; then a short final ladder, kept only where it lowers R-free beyond -0.001. The free set is the merge's own, so the refined R-free judges the refinement against reflections it +chosen one, each trying the weights as they are and changed, taken while it lowers the work-set target (the +refined target at the chosen weights); then a short final ladder. The free set decides the weights and the +displacement model only; every other step is taken on the work-set target. The free set is the merge's own, so the refined R-free judges the refinement against reflections it never fitted - but where the model was deposited, those reflections were most likely its depositor's working reflections, and a refinement against new data then moves R-free up from the unrefined value rather than down. `R_WORK`/`R_FREE` of the model as given are never replaced. On a GPU a refinement takes seconds; the CPU From ce608b5e18af220a5e4af853ba3391a256a5268d Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 09:09:02 +0100 Subject: [PATCH 03/10] Refinement: the trajectory tries the positions' weight halved at every even step With steps now taken on the work-set target, the trajectory goes on from the ladder's choice instead of stopping at its best step on the free set, and the weight halved (tried only at the first step before) is how it can tighten a weight looser than the data want: on the 19-set sweep (GPU) it took one set back to its earlier R-free (+0.0038 -> +0.0003) and changed no other, at 3% more target evaluations. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- image_analysis/structure_refinement/Refinement.cpp | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index db574ae51..643a306d9 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -3593,8 +3593,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector Date: Wed, 18 Nov 2026 10:32:50 +0100 Subject: [PATCH 04/10] Refinement: R-free compared paired; the model as given kept only where better R-free in FreeSetPrefers is now compared as a paired difference: each trial keeps R-free as one term per free twin group (FreeRTerms, the grouping TwinNull's paired z uses, now shared as TwinGroups), and a candidate may be higher than the incumbent by no more than the standard error of the sum of the groups' differences. The single-model standard error R_free / sqrt(2 N_free) it replaces was several times wider than the scatter of two trials from one start on the same reflections, which left -LL free deciding almost alone. The as-given guard now asks whether the model as given displaces the refinement on the free set (rather than whether the refinement displaces it), and never keeps it where the refinement is better on both R-work and R-free: on one guard set the refinement lowered R-free 0.2408 -> 0.2373 and R-work 0.242 -> 0.215 but raised -LL free by 7, and the model as given came out. That -LL free is higher there however D and the variance are fitted (on the free set, on the work set, on both, or crosswise), so it is not a consequence of fitting the variance on the free set. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- .../structure_refinement/Refinement.cpp | 97 ++++++++++++++----- .../structure_refinement/Refinement.h | 1 + 2 files changed, 72 insertions(+), 26 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 643a306d9..71d66f645 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -93,11 +93,21 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0; constexpr int SOLVENT_ITERATIONS = 10; // The free set decides the weights and the displacement model alone, and only where both of its measures agree: a // candidate is preferred to the incumbent where its -ln L of the free reflections is lower and its R-free not higher -// by more than the standard error of the incumbent's, R_free / sqrt(2 N_free). -// Following Tickle, Laskowski & Moss (2000) Acta Cryst. D56, 442-450 (the standard error of R-free), as PDB-REDO -// decides on more than one free measure, Joosten, Long, Murshudov & Perrakis (2014) IUCrJ 1, 213-220 -bool FreeSetPrefers(double r_free, double nll_free, double r_free_b, double nll_free_b, int n_free) { - return nll_free < nll_free_b && r_free < r_free_b + r_free_b / std::sqrt(2.0 * n_free); +// than the incumbent's by more than the standard error of the difference. The two are scored on the same free +// reflections, so the difference is paired: R-free is a sum of one term per free twin group (FreeRTerms), and the +// standard error is that of the sum of the groups' differences. +// As PDB-REDO decides on more than one free measure, Joosten, Long, Murshudov & Perrakis (2014) IUCrJ 1, 213-220 +bool FreeSetPrefers(const std::vector &r_terms, double nll_free, const std::vector &r_terms_b, + double nll_free_b) { + const size_t n = r_terms.size(); + double sum = 0, sum2 = 0; + for (size_t k = 0; k < n; k++) { + const double d = r_terms[k] - r_terms_b[k]; + sum += d; + sum2 += d * d; + } + const double se = n > 1 ? std::sqrt(std::max(sum2 - sum * sum / n, 0.0) * n / (n - 1)) : 0.0; + return nll_free < nll_free_b && sum < se; } // Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative. constexpr double TARGET_TOLERANCE = 1e-4; @@ -1909,6 +1919,39 @@ std::string DomainText(const RefinementReflections &refl, const XrayRowModel &ro // As REFMAC5 drops twin domains of small fraction, Murshudov et al. (2011) Acta Cryst. D67, 355-367 constexpr double TWIN_DOMAIN_MIN = 0.05; +// Each observed reflection's twin group, named by its lowest observed row (itself where there is no twin law). +std::vector TwinGroups(const RefinementReflections &refl) { + const size_t n = refl.Size(); + std::vector group(n); + for (size_t m = 0; m < n; m++) { + group[m] = static_cast(m); + for (size_t d = 0; d < refl.Laws(); d++) + if (refl.mates[d][m] < static_cast(n)) + group[m] = std::min(group[m], refl.mates[d][m]); + } + return group; +} + +// R-free of the model as the last Fcalc left it, as one term per free twin group, in the order of the groups (the +// members of a group share their F_calc, so their terms are summed into one, as TwinNull sums them): the terms +// FreeSetPrefers pairs. +std::vector FreeRTerms(const Context &ctx) { + const std::vector> fmodel = TwinnedModel(ctx.refl, ctx.rows, ctx.fcalc); + const std::vector group = TwinGroups(ctx.refl); + double fo = 0; + std::vector sum(ctx.refl.Size(), 0.0); + for (size_t m = 0; m < ctx.refl.Size(); m++) + if (ctx.refl.free[m]) { + fo += ctx.refl.fo[m]; + sum[group[m]] += std::fabs(ctx.refl.fo[m] - std::abs(fmodel[m])); + } + std::vector terms; + for (size_t m = 0; m < ctx.refl.Size(); m++) + if (ctx.refl.free[m] && group[m] == static_cast(m)) + terms.push_back(static_cast(sum[m] / fo)); + return terms; +} + // The null of a twinned refinement, for the model as the last Refit() left it: its R-free at the same fraction with // every reflection's mate replaced by an unrelated observed one of the same resolution and the same set (free with // free, work with work, as a law's twin pairs are). The reflections of each set are put in order of 1/d^2 with the two @@ -1932,14 +1975,7 @@ constexpr double TWIN_STOP_Z = 0.5; void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) { RefinementReflections null = ctx.refl; const size_t n = null.Size(); - // each observed reflection's twin group, named by its lowest observed row - std::vector group(n); - for (size_t m = 0; m < n; m++) { - group[m] = static_cast(m); - for (size_t d = 0; d < ctx.refl.Laws(); d++) - if (ctx.refl.mates[d][m] < static_cast(n)) - group[m] = std::min(group[m], ctx.refl.mates[d][m]); - } + const std::vector group = TwinGroups(ctx.refl); for (size_t d = first; d < null.Laws(); d++) std::iota(null.mates[d].begin(), null.mates[d].end(), 0); for (const bool free : {false, true}) { @@ -2265,6 +2301,7 @@ void MeasureWorkTarget(Context &ctx, const RefinementTopology &topo, const NcsCo const RefinementScore sc = ctx.Score(); t.nll_work = sc.nll_work; t.nll_free = sc.nll_free; + t.free_r_terms = FreeRTerms(ctx); const std::vector> xyzb = ctx.XyzB(); t.e_geometry = EvaluateRestraints(RestraintsNow(ctx, topo, ncs, ncs_sigma), xyzb, nullptr); t.e_b = EvaluateRestraints(BRestraintsNow(ctx, topo), xyzb, nullptr); @@ -2938,6 +2975,7 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, L int best = -1; std::vector rf; std::vector nll_free; + std::vector> r_terms; for (size_t k = 0; k < readings.size(); k++) { ctx.SetB(readings[k].b); ctx.SetAniso(readings[k].aniso); @@ -2947,9 +2985,10 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, L ctx.ForgetStarts(); rf.push_back(ctx.Refit()); nll_free.push_back(ctx.Score().nll_free); + r_terms.push_back(FreeRTerms(ctx)); logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}, -LL free " "{:.1f}", tls.size(), tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free, nll_free[k]); - if (best < 0 || FreeSetPrefers(rf[k].r_free, nll_free[k], rf[best].r_free, nll_free[best], rf[k].n_free)) + if (best < 0 || FreeSetPrefers(r_terms[k], nll_free[k], r_terms[best], nll_free[best])) best = static_cast(k); } ctx.SetB(readings[best].b); @@ -3194,6 +3233,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector start_r_terms = FreeRTerms(ctx); logger.Info("Refinement{}: {} atoms ({} riding hydrogens), {} reflections to {:.2f} A ({} free), on the {}; as " "given R-work {:.4f}, R-free {:.4f}; k_sol {:.3f}, b_sol {:.1f}, overall B {:.1f}, Wilson B {:.0f}, B " "ceiling {:.0f}, mask grid {:.2f} A", @@ -3211,6 +3251,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector own_tls = state.tls; @@ -4451,12 +4498,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector free_r_terms; // R-free as one term per free twin group (FreeRTerms), for paired comparisons GeometryStatistics geometry; bool within_bounds = false; // the geometry bounds the choice is made inside std::string adp; // the displacement model: "isotropic", "TLS ..." or "anisotropic" From cb4240767e9b018a9f638fa71279da99f2d03596 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 10:32:50 +0100 Subject: [PATCH 05/10] Refinement: REFINE_ADP_MODEL keeps a round's displacement model where its U are held After a second round whose weight trials held the U the first round's TLS or U-per-atom choice made, the label was reset to "isotropic" (or "the model's U held") because the kept trial carried no hypothesis name. It now keeps the first round's model with ", U held"; and a model whose per-atom U are held also drops the input's TLS groups, as one that refined them does. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- image_analysis/structure_refinement/Refinement.cpp | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 71d66f645..3ab866fba 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -3291,7 +3291,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector *groups = ctx.st.meta.get_tls_groups()) @@ -4380,9 +4380,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k); } - // (the model where the weights ended keeps the displacement model a round before chose for it) - if (result.adp_model.empty() || state.trial.adp.empty()) + // (the model where the weights ended keeps the displacement model a round before chose for it, its U held by + // the weight trials since) + if (result.adp_model.empty()) result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held"; + else if (state.trial.adp.empty() && result.adp_model != "isotropic" && !result.adp_model.ends_with("held")) + result.adp_model += ", U held"; if (best >= 0) { result.adp_model = outcomes[best].trial.adp; state = std::move(outcomes[best]); From a7842456270509f42bd52e88e1427ea3952b2d60 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 11:11:20 +0100 Subject: [PATCH 06/10] docs: the free set's R-free compared paired; the as-given rule The R-free part of the free-set rule is a paired difference over the free reflections; the model as given is kept only where it displaces the refinement and the refinement is not better on both R-work and R-free; and why -LL free keeps the refinement's own likelihood parameters rather than D and the variance fitted to the working set. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- docs/ACKNOWLEDGEMENT.md | 8 ++------ docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 28 +++++++++++++++++----------- docs/RUGNUX_MODEL_AND_REFERENCE.md | 3 ++- 3 files changed, 21 insertions(+), 18 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index e1e2e6af7..6d7a77f79 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -654,12 +654,8 @@ structures" (1992), Nature 355, 472-475 [doi:10.1038/355472a0](https://doi.org/1 **What the free set decides** — the free set decides the weights and the displacement model alone, and only where its two measures agree: a candidate is taken where its free $-\ln L$ is lower and its R-free not higher -by more than R-free's standard error, $R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$, from I. J. Tickle, -R. A. Laskowski and D. S. Moss, "Rfree and the Rfree ratio. II. Calculation of the -expected values and variances of cross-validation statistics in macromolecular least-squares refinement" -(2000), Acta Cryst. D56, 442-450 -[doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868); every other step of the refinement -is taken on the work-set target. The approach of deciding on more than one free measure, and of keeping the +than the incumbent's by more than the paired standard error of the difference; every other step of the +refinement is taken on the work-set target. The approach of deciding on more than one free measure, and of keeping the free set for few choices, is PDB-REDO's: R. P. Joosten, K. Joosten, G. N. Murshudov and A. Perrakis, "PDB_REDO: constructive validation, more than just looking for errors" (2012), Acta Cryst. D68, 484-496 [doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515), and R. P. Joosten, F. Long, diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 5cb5495f0..776ab7661 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -346,12 +346,18 @@ chosen late, on the evidence, instead of early at a rule. **What the free set decides.** The free set decides the weights - the X-ray weights of the positions and of the B factors and the NCS restraints' $\sigma$ - the displacement model (§15.7, 6.), the reading of the model's TLS groups -(§15.1) and whether the refinement displaces the model as given (7.), and nothing else. Each of these choices asks +(§15.1) and whether the model as given is kept over the refinement (7.), and nothing else. Each of these choices asks both of its measures: a candidate is preferred to the incumbent only where its $-\ln L$ of the free reflections is -lower **and** its R-free is not higher by more than the standard error of the incumbent's, -$\sigma(R_\mathrm{free}) = R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$ (Tickle, Laskowski & Moss 2000) - one measure -alone, or a fixed tie, picks the noise of a few hundred reflections as often as a better model (PDB-REDO likewise -takes a change only where its free measures agree, Joosten et al. 2014). Every other step - a step of the +lower **and** its R-free is not higher by more than the standard error of the difference. The two are scored on the +same free reflections, so the difference is paired: R-free is a sum of one term per free reflection (per free twin +group under a twin law, whose members share their $F_\mathrm{calc}$), and the standard error is that of the sum of +the terms' differences - two trials from one start differ far less from reflection to reflection than either +scatters about the truth, so this is narrower than the error of one R-free, $R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$. +One measure alone, or a fixed tie, picks the noise of a few hundred reflections as often as a better model (PDB-REDO +likewise takes a change only where its free measures agree, Joosten et al. 2014). $-\ln L$ free is that of the +refinement's own likelihood parameters - the variance fitted to the free reflections; with D and the variance fitted +to the working reflections instead, it kept out a per-atom U that overfits at 2.2 Å, but held back every +refinement (on 17 structures R-free 0.0016 and R-work 0.008 higher on average), and it is not used. Every other step - a step of the trajectory, a round, the placement's rigid bodies and B per residue, the solvent build's rounds, the side-chain field, the population's child - is taken on the **work-set target**: the target the trial minimises, $w\,T_\mathrm{X\text{-}ray} + E_\mathrm{geometry}$ (with the NCS restraints at their $\sigma$), plus @@ -459,11 +465,11 @@ listed first - on a ladder, the tighter geometry - is kept otherwise). The trial others - and undone otherwise; where its ladder and trajectory leave the model where it was, the hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder and one step. No round runs where the refinement does not displace the model as given. -7. **The model as given** is the first candidate in order of trust: where the refinement does not displace it on the - free set (its geometry within bounds and the free set preferring it; the work-set target cannot choose here, the - refinement having minimised it from the model as given), the model is written as - it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never - worse on the free set than what went in. This holds only where the free set is free to the model as given: a model +7. **The model as given** is kept over the refinement where it displaces the refinement on the free set (its geometry + within bounds and the free set preferring it) and the refinement is not better on both R-work and R-free (the + work-set target cannot choose here, the refinement having minimised it from the model as given): the model is + written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes + out is never worse on both free measures than what went in. This holds only where the free set is free to the model as given: a model placed as given on a free set drawn by the run (in `--mode refine`, a file without free flags) has most likely been refined against those reflections, and its R-free as given does not compete. @@ -819,7 +825,7 @@ What the refinement does not do yet - each a known gap, not a design choice: - A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7). - A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free). -- I. J. Tickle, R. A. Laskowski & D. S. Moss, "Rfree and the Rfree ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement", *Acta Cryst.* **D56** (2000), 442-450 [doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868) (the standard error of R-free the free set's choices are tied within, §15.7). +- I. J. Tickle, R. A. Laskowski & D. S. Moss, "Rfree and the Rfree ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement", *Acta Cryst.* **D56** (2000), 442-450 [doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868) (the standard error of one R-free, against which the paired error of §15.7 is set). - R. P. Joosten, F. Long, G. N. Murshudov & A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization", *IUCrJ* **1** (2014), 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324), and R. P. Joosten, K. Joosten, G. N. Murshudov & A. Perrakis, "PDB_REDO: constructive validation, more than just looking for errors", *Acta Cryst.* **D68** (2012), 484-496 [doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515) (choices taken only where more than one free measure agrees, §15.7). - G. J. Kleywegt, "Use of non-crystallographic symmetry in protein structure refinement", *Acta Cryst.* **D52** (1996), 842-857; I. Usón, E. Pohl, T. R. Schneider, Z. Dauter, A. Schmidt, H.-J. Fritz & G. M. Sheldrick, "1.7 Å structure of the stabilized REIv mutant T39K. Application of local NCS restraints", *Acta Cryst.* **D55** (1999), 1158-1167; R. A. Nicholls, F. Long & G. N. Murshudov, "Low-resolution refinement tools in REFMAC5", *Acta Cryst.* **D68** (2012), 404-417; J. J. Headd et al., "Flexible torsion-angle noncrystallographic symmetry restraints for improved macromolecular structure refinement", *Acta Cryst.* **D70** (2014), 1346-1356 (local NCS restraints with a robust loss, decided by R-free, and Murshudov et al. 2011 above for the copies found by alignment, §15.5, §15.7). - P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9). diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index fe8a04932..fb11d3a97 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -289,7 +289,8 @@ outer-sphere distance, a contact without a record, or one to a symmetry mate, is atoms in contact (waters included) are restrained to each other in proportion to their size. The weights of the X-ray term against the geometry and against that B restraint are chosen on the free set among short refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees - a candidate taken only where its free --ln L is lower and its R-free not higher by more than R-free's standard error: three weights from the placed +-ln L is lower and its R-free not higher than the incumbent's by more than the standard error of their paired +difference over the free reflections: three weights from the placed model, each with the B factors held and refined; then a trajectory of further short refinements from the chosen one, each trying the weights as they are and changed, taken while it lowers the work-set target (the refined target at the chosen weights); then a short final ladder. The free set decides the weights and the From e0f1cd6d928a39bb10925f8ffec7578034abe6ee Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 12:36:49 +0100 Subject: [PATCH 07/10] Refinement: a larger displacement model must pass Hamilton's R-ratio test In the displacement-model choice, a candidate with more displacement parameters than the incumbent (TLS bodies over a B per atom, a U per atom over either) now also has to lower R-work by more than its extra parameters account for: R_small / R_large above Hamilton's critical ratio sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the working reflections, m the larger model's displacement parameters (the hypothesis trials hold the positions), b the extra ones, counted unrestrained. The F point is Paulson's normal approximation (within 0.3% of the exact value at 5/100 degrees of freedom and closer above). Each trial records its displacement parameters. On the 2.0-2.8 A sets where a U per atom was taken on -LL free alone, the test now refuses it (1-1.3 working reflections per U parameter); where TLS was taken over the B per atom it passes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- .../structure_refinement/Refinement.cpp | 32 ++++++++++++++++++- .../structure_refinement/Refinement.h | 1 + 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 3ab866fba..6ba0b8e7c 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -109,6 +109,29 @@ bool FreeSetPrefers(const std::vector &r_terms, double nll_free, const st const double se = n > 1 ? std::sqrt(std::max(sum2 - sum * sum / n, 0.0) * n / (n - 1)) : 0.0; return nll_free < nll_free_b && sum < se; } +// The upper alpha point of the F distribution with d1 and d2 degrees of freedom, by Paulson's normal approximation of +// F^(1/3) (z the normal deviate of alpha) - accurate to a few percent at the counts a displacement model has. +// Following Paulson (1942) Ann. Math. Statist. 13, 233-235 +double FQuantile(double d1, double d2, double z) { + const double u = 2.0 / (9.0 * d1), v = 2.0 / (9.0 * d2), a = 1.0 - v, c = 1.0 - u; + const double q = a * a - z * z * v; + const double x = (a * c + std::sqrt(std::max(a * a * c * c - q * (c * c - z * z * u), 0.0))) / q; + return x * x * x; +} + +// Of two displacement models, the one with more parameters is taken only where R-work falls by more than its extra +// parameters account for: R_small / R_large above Hamilton's ratio sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the +// working reflections, m the larger model's displacement parameters (the trials hold the positions) and b its extra +// ones - each counted unrestrained, the conservative count. True where `t` has no more parameters than `s`. +// Following Hamilton (1965) Acta Cryst. 18, 502-510, as Merritt (2012) Acta Cryst. D68, 468-477 and PDB-REDO decide +// the B model by it +bool HamiltonPasses(const RefinementWeightTrial &t, const RefinementWeightTrial &s, int n_work) { + const int b = t.adp_parameters - s.adp_parameters; + if (b <= 0) + return true; + const int dof = n_work - t.adp_parameters; + return dof > 0 && s.r_work / t.r_work > std::sqrt(b / double(dof) * FQuantile(b, dof, 1.6449) + 1.0); +} // Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative. constexpr double TARGET_TOLERANCE = 1e-4; // The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each @@ -2539,6 +2562,13 @@ TrialOutcome RunTrial(TrialSlot slot, XyzCalibrations &calibrations, const std:: t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB()); if (hyp != nullptr) t.adp = hyp->name; + // the displacement parameters: a B per refinable atom, the TLS bodies' 20 each beside it, or a U per atom + int atoms_b = 0; + for (int i : refinable) + atoms_b += !ctx.atoms[i]->is_hydrogen(); + t.adp_parameters = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast(u_atoms.size()) + : kind == AdpHypothesis::Kind::Tls ? atoms_b + 20 * static_cast(hyp->bodies.size()) + : atoms_b; MeasureWorkTarget(ctx, topo, ncs, ncs_sigma, t); out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, target {:.6g}, rmsd bonds " "{:.4f} A, angles {:.2f} deg{}{}", @@ -4377,7 +4407,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k); } // (the model where the weights ended keeps the displacement model a round before chose for it, its U held by diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index ecd8a6acc..c745f254c 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -163,6 +163,7 @@ struct RefinementWeightTrial { // The work-set target's terms where the trial ended (RefineOnce's WorkTarget): -ln L of the working reflections, // the restraints' sum of the geometry (with the NCS restraints at ncs_sigma) and that of the B similarity. double nll_work = NAN, e_geometry = NAN, e_b = NAN; + int adp_parameters = 0; // the displacement model's parameters (HamiltonPasses) std::vector free_r_terms; // R-free as one term per free twin group (FreeRTerms), for paired comparisons GeometryStatistics geometry; bool within_bounds = false; // the geometry bounds the choice is made inside From 93f0bf32b083760d3198e2d421177a6da5f5c7f9 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 12:37:12 +0100 Subject: [PATCH 08/10] docs: Hamilton's R-ratio test in the displacement-model choice Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- docs/ACKNOWLEDGEMENT.md | 7 ++++++- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 6d7a77f79..6382831b4 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -702,7 +702,12 @@ SHELXL" (2015), Acta Cryst. C71, 3-8 [doi:10.1107/S2053229614024218](https://doi.org/10.1107/S2053229614024218) documents them. Which displacement model a refinement keeps is decided on the free set rather than by a rule of resolution: E. A. Merritt, "To B or not to B: a question of resolution?" (2012), Acta Cryst. D68, -468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320). +468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320). A model with more +displacement parameters must in addition pass the R-factor ratio test of W. C. Hamilton, "Significance +tests on the crystallographic R factor" (1965), Acta Cryst. 18, 502-510 (DOI not verified), which Merritt +(2012) and PDB-REDO (Joosten et al. 2012, above) use to choose the B model; its F point is from E. Paulson, +"An approximate normalization of the analysis of variance distribution" (1942), Ann. Math. Statist. 13, +233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608). **Population and crossover** — the members' refinement on blurred data is a continuation (homotopy) on the resolution, the deformation of the landscape of L. Piela, J. Kostrowicki and H. A. Scheraga, "On the diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 776ab7661..bbdde0877 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -449,7 +449,7 @@ listed first - on a ladder, the tighter geometry - is kept otherwise). The trial reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving out the U weight $\times 4$ changed a choice taken. A trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds - and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). + and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). A model with more displacement parameters (TLS over a B per atom, a U per atom over either) must also pass Hamilton's R-ratio test: its R-work below the smaller model's by more than the ratio $\sqrt{b/(n-m)\,F_{b,n-m,0.05} + 1}$, $n$ the working reflections, $m$ its displacement parameters (the positions are held) and $b$ the extra ones, counted unrestrained (Hamilton 1965; the F point by Paulson's approximation). A U per atom then needs several working reflections per U parameter; on 2.0-2.8 Å data with 1-1.3 it is refused even where the free set alone would take it. The refined U are written as ANISOU and the TLS groups as `_pdbx_refine_tls*` and `REMARK 3` (`REFINE_ADP_MODEL`, `REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no @@ -811,6 +811,7 @@ What the refinement does not do yet - each a known gap, not a design choice: - A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long & G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15.4). - M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (the topology, the riding hydrogens and the blob search, §15.4, §15.9). - V. Schomaker & K. N. Trueblood, *Acta Cryst.* **B24** (1968), 63-76, and M. D. Winn, M. N. Isupov & G. N. Murshudov, *Acta Cryst.* **D57** (2001), 122-133 (TLS, and the B column total or residual to it, §15.1, §15.5); J. Painter & E. A. Merritt, *Acta Cryst.* **D62** (2006), 439-450 (TLSMD's segments, §15.7). +- W. C. Hamilton, "Significance tests on the crystallographic R factor", *Acta Cryst.* **18** (1965), 502-510, as E. A. Merritt, "To B or not to B: a question of resolution?", *Acta Cryst.* **D68** (2012), 468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320) applies it to the displacement model, and E. Paulson, "An approximate normalization of the analysis of variance distribution", *Ann. Math. Statist.* **13** (1942), 233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608) (the F point) (§15.7). - F. L. Hirshfeld, *Acta Cryst.* **A32** (1976), 239-244, A. Thorn, B. Dittrich & G. M. Sheldrick, *Acta Cryst.* **A68** (2012), 448-451, and G. M. Sheldrick, *Acta Cryst.* **C71** (2015), 3-8 (the rigid-bond, similarity and isotropy restraints of anisotropic atoms, §15.5); E. A. Merritt, *Acta Cryst.* **D68** (2012), 468-477 (the displacement model decided by test, §15.7). - A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5). - M. M. Harding, "Geometry of metal-ligand interactions in proteins", *Acta Cryst.* **D57** (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", *Acta Cryst.* **D62** (2006), 678-682; F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5). From 7ce5bd77e2d57d4275a36de1847584aa1b70f517 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 14:26:40 +0100 Subject: [PATCH 09/10] Refinement: the displacement model chosen by Hamilton's tests over the restraint weights As Merritt (2012) and PDB-REDO (Joosten et al. 2012) apply Hamilton's R-ratio test: each model's restraints are credited as w R observations, w scanned from 0 to the bound at which they take every parameter they restrain off its count (Merritt's N_atoms / R for a B per atom, 6 N_atoms / R for a U per atom), the two models' weights independently, six values each. The larger model is taken where it passes more than 95% of the tests, the smaller where it passes fewer than 30%, and in between the free set decides (the paired R-free and -LL free rule every other free-set choice uses, rather than PDB-REDO's Z(R-free) and R-free - R gap cut-offs, which are fixed thresholds on one measure). A U per atom is tried only above 18 working reflections per refinable atom, PDB-REDO's bound. Against the single unrestrained test this replaces: a U per atom is taken again at 1.2-1.4 A with 23-33 reflections per atom (the tests inconclusive, the free set preferring it) and still not at 2.0-2.8 A with 6-12.5. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- .../structure_refinement/Refinement.cpp | 62 ++++++++++++++----- .../structure_refinement/Refinement.h | 3 +- 2 files changed, 50 insertions(+), 15 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 6ba0b8e7c..e118c2fbb 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -119,19 +119,32 @@ double FQuantile(double d1, double d2, double z) { return x * x * x; } -// Of two displacement models, the one with more parameters is taken only where R-work falls by more than its extra -// parameters account for: R_small / R_large above Hamilton's ratio sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the -// working reflections, m the larger model's displacement parameters (the trials hold the positions) and b its extra -// ones - each counted unrestrained, the conservative count. True where `t` has no more parameters than `s`. -// Following Hamilton (1965) Acta Cryst. 18, 502-510, as Merritt (2012) Acta Cryst. D68, 468-477 and PDB-REDO decide -// the B model by it -bool HamiltonPasses(const RefinementWeightTrial &t, const RefinementWeightTrial &s, int n_work) { - const int b = t.adp_parameters - s.adp_parameters; - if (b <= 0) - return true; - const int dof = n_work - t.adp_parameters; - return dof > 0 && s.r_work / t.r_work > std::sqrt(b / double(dof) * FQuantile(b, dof, 1.6449) + 1.0); +// Of two displacement models, `large` with more parameters than `small`: the fraction of Hamilton's R-ratio tests it +// passes over the range of weights its restraints may be credited at - R_small / R_large above +// sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the working reflections, m the larger model's displacement parameters +// (the trials hold the positions), b the extra ones. Each model's restraints are credited as w R observations, w from +// 0 up to the bound at which they take every parameter they restrain off its count - Merritt's N_atoms / R for a B +// per atom (as his 3 N_atoms / R for the positions), 6 N_atoms / R for a U per atom; the two weights are scanned +// independently, six values each. +// Following Hamilton (1965) Acta Cryst. 18, 502-510, as Merritt (2012) Acta Cryst. D68, 468-477 and PDB-REDO +// (Joosten et al. 2012 Acta Cryst. D68, 484-496) apply it to the displacement model +double HamiltonPassFraction(const RefinementWeightTrial &large, const RefinementWeightTrial &small, int n_work) { + int pass = 0, tests = 0; + for (int i = 0; i <= 5; i++) + for (int j = 0; j <= 5; j++, tests++) { + const double m_small = small.adp_parameters - small.adp_restrained * i / 5.0; + const double m_large = large.adp_parameters - large.adp_restrained * j / 5.0; + const double b = m_large - m_small, dof = n_work - m_large; + pass += b <= 0 || (dof > 0 && small.r_work / large.r_work > std::sqrt(b / dof * FQuantile(b, dof, 1.6449) + 1.0)); + } + return static_cast(pass) / tests; } +// The displacement model: of the larger and the smaller, the larger where it passes more than this fraction of the +// tests (HamiltonPassFraction), the smaller where it passes fewer than the other, and the free set's choice between +// (as PDB-REDO; Joosten et al. 2012). +constexpr double HAMILTON_LARGER = 0.95, HAMILTON_SMALLER = 0.30; +// A U per atom is tried only above this many working reflections per refinable atom (as PDB-REDO; Joosten et al. 2012). +constexpr double U_MIN_REFLECTIONS_PER_ATOM = 18; // Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative. constexpr double TARGET_TOLERANCE = 1e-4; // The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each @@ -2569,6 +2582,8 @@ TrialOutcome RunTrial(TrialSlot slot, XyzCalibrations &calibrations, const std:: t.adp_parameters = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast(u_atoms.size()) : kind == AdpHypothesis::Kind::Tls ? atoms_b + 20 * static_cast(hyp->bodies.size()) : atoms_b; + // of them the restrained ones: the B or the U of every atom (not the TLS bodies') + t.adp_restrained = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast(u_atoms.size()) : atoms_b; MeasureWorkTarget(ctx, topo, ncs, ncs_sigma, t); out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, target {:.6g}, rmsd bonds " "{:.4f} A, angles {:.2f} deg{}{}", @@ -4299,9 +4314,14 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector hyps = AdpHypotheses(ctx, topo, own_tls); const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0; + // a U per atom only where the data are many times its parameters + int atoms_b = 0; + for (int i : refinable) + atoms_b += !ctx.atoms[i]->is_hydrogen(); + const bool u_tried = result.n_work > U_MIN_REFLECTIONS_PER_ATOM * atoms_b; // the U weight balanced on the model with every refinable atom given a U double u_w0 = adp; - { + if (u_tried) { const std::vector keep = ctx.aniso; std::vector all = keep.empty() ? std::vector(ctx.atoms.size()) : keep; std::vector refines_u(ctx.atoms.size(), 0); @@ -4324,6 +4344,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(trials.size()); for (double f : {0.25, 1.0}) { trials.push_back({w, u_w0 * f, ncs_s}); @@ -4407,7 +4429,19 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector b.adp_parameters; + const double pass = larger ? HamiltonPassFraction(t, b, result.n_work) + : HamiltonPassFraction(b, t, result.n_work); + bool takes = displaces(t, b); + if (t.adp_parameters != b.adp_parameters && (pass > HAMILTON_LARGER || pass < HAMILTON_SMALLER)) + takes = t.within_bounds && (!b.within_bounds || (pass > HAMILTON_LARGER) == larger); + logger.Info("Refinement, displacements: {} against {}: the larger passes {:.0f}% of Hamilton's tests{}", + t.adp.empty() ? "the B factors as they are" : t.adp, + b.adp.empty() ? "the B factors as they are" : b.adp, 100 * pass, takes ? ", taken" : ""); + if (takes) best = static_cast(k); } // (the model where the weights ended keeps the displacement model a round before chose for it, its U held by diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index c745f254c..e7a1073a3 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -163,7 +163,8 @@ struct RefinementWeightTrial { // The work-set target's terms where the trial ended (RefineOnce's WorkTarget): -ln L of the working reflections, // the restraints' sum of the geometry (with the NCS restraints at ncs_sigma) and that of the B similarity. double nll_work = NAN, e_geometry = NAN, e_b = NAN; - int adp_parameters = 0; // the displacement model's parameters (HamiltonPasses) + int adp_parameters = 0; // the displacement model's parameters (HamiltonPassFraction) + int adp_restrained = 0; // of them those the restraints act on, which HamiltonPassFraction credits std::vector free_r_terms; // R-free as one term per free twin group (FreeRTerms), for paired comparisons GeometryStatistics geometry; bool within_bounds = false; // the geometry bounds the choice is made inside From d03c81704cc1001140ad728a4e24abd29716f2c9 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 18 Nov 2026 14:26:58 +0100 Subject: [PATCH 10/10] docs: the displacement model by Hamilton's tests over the restraint weights Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7 --- docs/ACKNOWLEDGEMENT.md | 9 +++++---- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 6382831b4..d2d4598be 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -702,10 +702,11 @@ SHELXL" (2015), Acta Cryst. C71, 3-8 [doi:10.1107/S2053229614024218](https://doi.org/10.1107/S2053229614024218) documents them. Which displacement model a refinement keeps is decided on the free set rather than by a rule of resolution: E. A. Merritt, "To B or not to B: a question of resolution?" (2012), Acta Cryst. D68, -468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320). A model with more -displacement parameters must in addition pass the R-factor ratio test of W. C. Hamilton, "Significance -tests on the crystallographic R factor" (1965), Acta Cryst. 18, 502-510 (DOI not verified), which Merritt -(2012) and PDB-REDO (Joosten et al. 2012, above) use to choose the B model; its F point is from E. Paulson, +468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320). Between a model +with more displacement parameters and one with fewer, the R-factor ratio test of W. C. Hamilton, "Significance +tests on the crystallographic R factor" (1965), Acta Cryst. 18, 502-510 (DOI not verified), decides first, +over a range of restraint weights as Merritt (2012) applies it, with PDB-REDO's (Joosten et al. 2012, above) +95%/30% rule and its bound of 18 reflections per atom for a U per atom; its F point is from E. Paulson, "An approximate normalization of the analysis of variance distribution" (1942), Ann. Math. Statist. 13, 233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608). diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index bbdde0877..14876ac72 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -449,7 +449,7 @@ listed first - on a ladder, the tighter geometry - is kept otherwise). The trial reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving out the U weight $\times 4$ changed a choice taken. A trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds - and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). A model with more displacement parameters (TLS over a B per atom, a U per atom over either) must also pass Hamilton's R-ratio test: its R-work below the smaller model's by more than the ratio $\sqrt{b/(n-m)\,F_{b,n-m,0.05} + 1}$, $n$ the working reflections, $m$ its displacement parameters (the positions are held) and $b$ the extra ones, counted unrestrained (Hamilton 1965; the F point by Paulson's approximation). A U per atom then needs several working reflections per U parameter; on 2.0-2.8 Å data with 1-1.3 it is refused even where the free set alone would take it. + and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). Between a model with more displacement parameters (TLS over a B per atom, a U per atom over either) and one with fewer, Hamilton's R-ratio test decides first, as Merritt (2012) and PDB-REDO apply it: the larger passes where its R-work is below the smaller's by more than the ratio $\sqrt{b/(n-m)\,F_{b,n-m,0.05} + 1}$, $n$ the working reflections, $m$ its displacement parameters (the positions are held) and $b$ the extra ones (Hamilton 1965; the F point by Paulson's approximation). The restraints are credited as $wR$ observations, $w$ unknown: each model's is scanned from 0 to the bound at which its restraints take every parameter they restrain off its count (Merritt's $N_\mathrm{atoms}/R$ for a B per atom, $6N_\mathrm{atoms}/R$ for a U per atom), six values each, independently. The larger model is taken where it passes more than 95% of the 36 tests, the smaller where it passes fewer than 30%, and in between the free set decides as everywhere else (PDB-REDO falls back on Z(R-free) and the R-free - R-work gap at fixed cut-offs instead; the paired rule is the one every other free-set choice here uses). A U per atom is tried only above 18 working reflections per refinable atom, PDB-REDO's bound. The refined U are written as ANISOU and the TLS groups as `_pdbx_refine_tls*` and `REMARK 3` (`REFINE_ADP_MODEL`, `REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no