From 344912139adcc3cba77d91a3c9c6806a4209ed08 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 13 Nov 2026 23:33:46 +0100 Subject: [PATCH 1/8] Refinement twin null: each unrelated mate on its own anisotropic scale The twin null replaces a law's mates by unrelated reflections of the same resolution and asks what averaging alone buys. TwinnedModel puts a mate on the scale k_total of the reflection it is the mate of - right for a twin law, whose domain reflection is measured at the same spot - but an unrelated mate lies in another direction, where the anisotropic scale can differ many times over. On a trigonal crystal with a pseudo-translation, refined from a model whose atomic U carried the anisotropy opposite to k_total, a weak reflection's random mate was put on a scale 22 times its own (amplitude, at 2 A): the null R-free read 0.466 against an untwinned 0.406, and a law at fraction 0.04 was adopted 7.9 standard errors ahead of it. With each mate on its own scale the null reads 0.400 (z 0.8, untwinned); the same crystal's experimental model, adopted twinned at z 3.1 in rc186, reads z 0.6. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK --- .../structure_refinement/Refinement.cpp | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 07f0e089c..cb1e3406e 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -1450,7 +1450,29 @@ void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) { for (size_t d = first; d < a.size(); d++) result.twin_fraction_null += a[d]; } - const std::vector> f_null = TwinnedModel(null, rows, ctx.fcalc); + // TwinnedModel with each unrelated mate on its own scale k_total: TwinnedModel puts a mate on the scale of the + // reflection it is the mate of, as the twin domain's reflection at the same spot is measured, but an unrelated + // mate lies in another direction, where the anisotropic scale can differ many times over (22 times in amplitude + // at 2 A, on a model whose U carried the anisotropy opposite to k_total), and on the reflection's scale it put + // the null 0.06 in R-free above the untwinned refinement - a law "ahead" of its null by 7.9 standard errors at a + // fraction of 0.04. + std::vector> f_null(n); + double own_fraction = 1.0; + for (float a : rows.twin_fractions) + own_fraction -= a; + for (size_t m = 0; m < n; m++) { + const std::complex u = std::complex(ctx.fcalc[m]) + std::complex(rows.fmask[m]); + double i_model = own_fraction * rows.k_total[m] * rows.k_total[m] * std::norm(u); + for (size_t d = 0; d < null.Laws(); d++) { + const int j = null.mates[d][m]; + const double k = d < first ? rows.k_total[m] : rows.k_total[j]; + i_model += rows.twin_fractions[d] * k * k * + std::norm(std::complex(ctx.fcalc[j]) + std::complex(rows.fmask[j])); + } + const double amplitude = std::sqrt(std::max(i_model, 0.0)); + f_null[m] = std::norm(u) > 0 ? std::complex(u * (amplitude / std::abs(u))) + : std::complex(static_cast(amplitude), 0); + } result.twin_r_free_null = ComputeRFactors(null, f_null).r_free; // How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin|| // over its standard error. The members of a free twin group share their F_calc, so their terms are summed into From fabcb0e4d3a1a284213791ba07e6ad713bbaee14 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Fri, 13 Nov 2026 23:51:02 +0100 Subject: [PATCH 2/8] Refinement: run again from the refined model while R-free falls (3 rounds at most) One refinement chooses its X-ray weight and NCS on the B factors as given and its displacement model (TLS / U) only after them. Where the displacements change much - a placed prediction, whose B factors are its pLDDT - the weights stand on B factors the model no longer has. Running the refinement again from its own result (as REFMAC users do by hand) lets the weight ladder and the NCS choice see the refined displacements. Cytochrome C, three copies placed by MR from the AlphaFold model, 1.73 A, live merge: round 2 chose an eight times lower X-ray weight without NCS, R-work 0.367 -> 0.361 at bonds 0.002 -> 0.001 A, R-free 0.418 -> 0.405; with --refine-solvent 0.427 -> 0.409 -> 0.401. REFMAC5 from the same model reaches 0.41-0.43. On 22 battery merges with their placed models no R-free rose; lyso_x06da_ref -0.003, cytc_x10sa -0.005, 9khr -0.003, 6hv2 -0.002, the rest unchanged. End to end from the AlphaFold DB models of the timing benchmark (MR, rugnux mx): myob_x10sa 0.341 -> 0.312, cytc_x10sa 0.419 -> 0.410, 9zlo 0.289 -> 0.283, 9s02 and 9gjx unchanged. Cost: refinement time x1.3-4 (cytochrome C 6 -> 12 s on the GPU; 9s02 22 -> 30 s), the second round being run to be judged. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK --- .../structure_refinement/Refinement.cpp | 59 +++++++++++++++++-- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index cb1e3406e..7e8c81502 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -3332,6 +3332,57 @@ double FinalRWork(const RefinementResult &r) { return r.restrained ? r.r_work : r.r_work_placement; } +// The refinement run again from the model it ended with, while that lowers R-free by more than the tie, up to +// REFINEMENT_ROUNDS in all. One refinement chooses its weights on the B factors as given and its displacement model +// only after them, so where the displacements change much - a placed prediction, whose B factors are its pLDDT - the +// weights were chosen for B factors it no longer has: on a placed prediction of three copies the second round chose +// an eight times lower X-ray weight, R-work and R-free both fell and R-free went 0.418 -> 0.405 (with the solvent +// build 0.427 -> 0.409 -> 0.401), where REFMAC5 reached 0.41-0.43 from the same model. The first round's view of +// the model as given is kept; the solvent build is the first round's. +constexpr int REFINEMENT_ROUNDS = 3; + +RefinementResult RefineRounds(const gemmi::Structure &placed, const std::vector &merged, + const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, + Logger &logger, const RefinementSettings &settings, + const std::vector &twin_laws = {}, size_t null_first = 0, + CardHeld *held = nullptr) { + RefinementResult result = RefineOnce(placed, merged, cell, sg, wavelength_A, logger, settings, twin_laws, + null_first, held); + RefinementSettings again = settings; + again.solvent = false; + for (int round = 2; round <= REFINEMENT_ROUNDS; round++) { + if (!result.ok || !result.restrained || result.as_given || result.cancelled || settings.score_only || + result.model == nullptr) + break; + logger.Info("Refinement, round {}: from the model the round before ended with (R-free {:.4f})", round, + result.r_free); + RefinementResult next = RefineOnce(*result.model, merged, cell, sg, wavelength_A, logger, again, twin_laws, + null_first); + if (!next.ok || !next.restrained || next.as_given || !(next.r_free < result.r_free - R_FREE_TIE)) { + logger.Info("Refinement, round {}: not kept (R-free {:.4f})", round, FinalRFree(next)); + result.seconds += next.seconds; + break; + } + next.seconds += result.seconds; + next.r_work_start = result.r_work_start; + next.r_free_start = result.r_free_start; + next.score_start = result.score_start; + next.adp_input = result.adp_input; + next.adp_input_aniso_atoms = result.adp_input_aniso_atoms; + next.rigid_bodies = result.rigid_bodies; + next.group_b = result.group_b; + next.r_work_placement = result.r_work_placement; + next.r_free_placement = result.r_free_placement; + next.geometry_start = result.geometry_start; + next.ncs_rmsd_start = result.ncs_rmsd_start; + next.solvent_build = result.solvent_build; + if (next.adp_model == "the model's U held") // the round before's displacement model, refined on + next.adp_model = result.adp_model; + result = std::move(next); + } + return result; +} + // The twin hypotheses beside the untwinned refinement `result`, as a ladder. First one law: the placed model refined // again, solvent build and displacement model included, against the data as twinned by a law of the lattice, and the // twinned refinement taken (into `result`) where the law beats its own null (TwinNull) on the free reflections by @@ -3407,7 +3458,7 @@ void DecideTwin(const gemmi::Structure &placed, const std::vector kept{law}; @@ -3590,7 +3641,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector Date: Sat, 14 Nov 2026 03:24:01 +0100 Subject: [PATCH 3/8] Refinement rounds: no placement in rounds 2-3, no maps for a round not kept A further round starts from the model the round before refined, so the rigid-body and per-residue-B placement is skipped there; and a round whose R-free does not beat the one before is returned before its maps, twin null and blobs (it is discarded anyway). The placement skip changes the path of the later rounds, not the rule: on 22 battery merges with their placed models R-free equal to the rounds without it except lyso_x06da_ref 0.2413 -> 0.2430 and cytc_x06da_1 0.3143 -> 0.3128; from the timing benchmark's AlphaFold models (MR, mx): cytc_x10sa 0.410 -> 0.399, 9zlo 0.283 -> 0.284, myob_x10sa 0.312 -> 0.312, 9s02/9gjx equal; cytochrome C live merge, MR model 0.405 -> 0.400. Saves about 1-2 s of the 7-8 s the rounds add on cytochrome C (wall 15.5 -> 13.6 s). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK --- image_analysis/structure_refinement/Refinement.cpp | 11 ++++++++++- image_analysis/structure_refinement/Refinement.h | 4 ++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 7e8c81502..9db12da54 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -2563,7 +2563,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(std::chrono::steady_clock::now() - start).count(); + return result; + } finish(); logger.Info("Refinement: done in {:.1f} s, {} target evaluations{}", result.seconds, ctx.evaluations, @@ -3350,12 +3357,14 @@ RefinementResult RefineRounds(const gemmi::Structure &placed, const std::vector< null_first, held); RefinementSettings again = settings; again.solvent = false; + again.placement = false; // the chains and the B per residue were placed by the first round for (int round = 2; round <= REFINEMENT_ROUNDS; round++) { if (!result.ok || !result.restrained || result.as_given || result.cancelled || settings.score_only || result.model == nullptr) break; logger.Info("Refinement, round {}: from the model the round before ended with (R-free {:.4f})", round, result.r_free); + again.keep_below = result.r_free - R_FREE_TIE; RefinementResult next = RefineOnce(*result.model, merged, cell, sg, wavelength_A, logger, again, twin_laws, null_first); if (!next.ok || !next.restrained || next.as_given || !(next.r_free < result.r_free - R_FREE_TIE)) { diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index c10e6dd84..c5b46c787 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -84,6 +84,10 @@ struct RefinementSettings { // statistic - false where the model has most likely been refined against the free reflections too (a deposited // model of this crystal scored on a free set drawn here). bool as_given_competes = true; + // For a further round of the refinement (RefineRounds): the placement stage skipped, and the maps, the twin null + // and the blobs made only where R-free ends below keep_below - the round is not kept otherwise. + bool placement = true; + double keep_below = INFINITY; bool gpu = true; size_t nthreads = 1; size_t gpu_resident_bytes = 0; // held on the card by what runs beside From f10829dabc17f6e849c63c475325ced417c2a307 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 14 Nov 2026 09:18:09 +0100 Subject: [PATCH 4/8] Refinement: rounds 2 and 3 as the weights and the displacement model decided again on the refined model The rounds (fabcb0e4d, a64d3e3c4) ran the whole refinement again from its written result - setup, placement, ladder, trajectory, final ladder, displacement hypotheses - and a round not kept had cost a full refinement. What a second round changed, read off the logs and by instrumenting the balanced weights and the TLS trials: the ceiling of B (twice the model's largest B: 100 on a placed prediction given with B factors of 6 A^2 under an overall B of 45, which refines at B near 50-70 - the TLS bodies and the U per atom were held at the ceiling, and the TLS hypothesis lost to a U per atom; 200 from the written model), the B factors' weight (balanced on the B factors as given, 2500 against 23 on the refined model: the B similarity's gradient goes as 1/B), the rung (the middle one with NCS on flat B factors, the tightest without NCS once the B factors were refined) and the TLS bodies fitted from refined B factors instead of flat ones. None of this can be decided in the first round - it is the alternation of positions and displacements - so the rounds stay, reduced to what changes. A round is now inside RefineOnce, after the displacement model: the ceiling and the balanced weights from the model as it is (the engines made again where the ceiling rises), the ladder from it with its displacements as they are, the trajectory, and the hypotheses from there; kept where it lowers R-free beyond the tie and undone otherwise, three rounds in all. Where the ladder and the trajectory leave the model where it was, the hypotheses would run identical trials, so the round ends there - an exact rule: a model refined before pays one ladder and one step. No round where the refinement does not displace the model as given. The first round is the one refinement as before, so a model the rounds do not improve comes out as it did. RefineRounds and the placement and keep_below settings go. On the cytochrome C live merge and MR model, R-free 0.4183 (one round) -> 0.4028 (0.3999 with the full rounds, which also re-placed the chains and re-ran the solvent build each round). Tried and rejected: the trajectory continued from the chosen displacement model with positions and U together (no gain: the U per atom overfits), B refined ahead of B held in the ladder's trust (worse), the B weight balanced with the overall B moved in in the first round (fixes the prediction, halves the weight on models with an overall B of 15 and costs them 0.002), and the ceiling raised with the overall B at setup (changes the first round: one AF-start run 0.006 worse). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK --- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 18 +- .../structure_refinement/Refinement.cpp | 190 ++++++++++-------- .../structure_refinement/Refinement.h | 4 - 3 files changed, 125 insertions(+), 87 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index b0a12e309..90257354f 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -28,7 +28,7 @@ statistics and its map coefficients go out; the restraint dictionary is compiled The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed, and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within -100-300 Ų). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS +100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest @@ -309,9 +309,9 @@ are: was never kept on 55 refinements, and taking it away changed no choice). 4. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U held), each trial as long as a step on the positions where the weights ended, held - only the displacement - parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with - the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held - positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain + parameters refine, so the trials differ in the displacement model alone (where the geometry is out of bounds + there, which held positions cannot mend, each trial refines the positions too, a macro-cycle longer than a + step): TLS groups with a residual B - the model's own, one per chain of at least 50 refinable atoms, and the chains cut into 4, 8 and 16 contiguous segments of at least 10 residues as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r), to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U @@ -327,6 +327,16 @@ are: `REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no TLS group holds a water. + Then the weights and the displacement model are **decided again** from the model as it now is, up to twice: the + B ceiling and the balanced weights from its B factors, the ladder from it with its displacements as they are, the + trajectory, and the displacement hypotheses from where that ends. The first choice took its weights with the + B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT, + the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$), + the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the + refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers + R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the + hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder + and one step. No round runs where the refinement does not displace the model as given. 5. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 9db12da54..e20e9c584 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -63,6 +63,9 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0; constexpr int SOLVENT_ITERATIONS = 10; // R-free differences smaller than this do not decide between two weights: the tighter geometry is taken. constexpr double R_FREE_TIE = 0.001; +// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each +// after it from the model the one before ended with. +constexpr int REFINEMENT_ROUNDS = 3; // The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this, // relative; the most a position (A) or a B (A^2) moves in the first trial of a line search. constexpr double MINIMISER_F_TOLERANCE = 1e-7; @@ -2563,8 +2566,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; + std::vector outcomes; + // The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the + // balanced weights as they stand: the trial kept, or one that did not run where none did. + auto ladder_choice = [&](const std::vector &from_pos, const std::vector &from_b, + const std::vector &from_aniso, const Context::Fit *fit) { std::vector ladder; for (double rung : settings.weight_ladder) for (double adp : settings.adp_ladder) { @@ -2695,14 +2703,13 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; - std::vector outcomes(ladder.size()); + outcomes.assign(ladder.size(), TrialOutcome()); auto run_ladder = [&](const std::function &these) { std::vector t; for (const TrialWeights &l : ladder) if (these(l)) t.push_back(l); - std::vector ran = RunTrials(trial_slots, t, refinable, pos0, b0, aniso0, nullptr, + std::vector ran = RunTrials(trial_slots, t, refinable, from_pos, from_b, from_aniso, fit, settings.macro_cycles, settings, ncs_p); for (size_t i = 0, m = 0; i < ladder.size(); i++) if (these(ladder[i])) @@ -2743,10 +2750,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0) - state = std::move(outcomes[k]); + const int k = choose(outcomes, "ladder"); + return k >= 0 ? std::move(outcomes[k]) : TrialOutcome(); + }; + TrialOutcome state = ladder_choice(pos0, b0, aniso0, nullptr); // 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they // are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the @@ -2761,7 +2768,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector from_best; // `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since). - auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best) { + // `label`: the steps' name in the log. + auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best, const char *label) { TrialOutcome best = state; from_best.clear(); bool state_is_best = true; @@ -2786,7 +2794,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0 && displaces(outcomes[k].trial, state.trial)) state = std::move(outcomes[k]); } @@ -3110,7 +3118,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector &own_tls) { const RefinementStage adp_stage("adp_hypotheses"); ctx.SetPositions(state.pos); ctx.SetB(state.b); @@ -3120,7 +3129,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector hyps = AdpHypotheses(ctx, topo, input_tls); + std::vector hyps = AdpHypotheses(ctx, topo, own_tls); const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0; // the U weight balanced on the model with every refinable atom given a U double u_w0 = adp; @@ -3255,6 +3264,89 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector own_tls = state.tls; + ctx.SetPositions(state.pos); + ctx.SetB(state.b); + ctx.SetAniso(state.aniso); + ctx.Restore(state.fit); + PlaceRiders(ctx, topo); + // the ceiling of B from the model as it now is, as from the model as given: a placed prediction comes + // with B factors of 6 A^2 under an overall B of 45 (its ceiling 100) and refines at B near 50-70, where + // the TLS bodies and the U per atom were held at the ceiling; where it rises the engines are made again + // (the gather is reserved for it) + double b_max = 0; + for (size_t i = 0; i < ctx.atoms.size(); i++) + b_max = std::max(b_max, ctx.atoms[i]->b_iso); + const double ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX); + if (ceiling > ctx.b_ceiling) { + const RefinementStage stage("engine"); + for (std::unique_ptr &o : own) + if (o) + o->ctx.engine.reset(); + ctx.b_ceiling = ceiling; + MakeEngine(ctx, settings.gpu, settings, logger, restraint_records, restraint_slots); + ctx.engine->SetRestraints(0, RestraintsNow(ctx, topo)); + ctx.engine->SetRestraints(1, topo.adp); + for (size_t k = 0; k < own.size(); k++) + if (own[k]) { + Context &o = own[k]->ctx; + o.b_ceiling = ceiling; +#ifdef JFJOCH_USE_CUDA + if (ctx.engine->OnGPU()) + o.engine = MakeRefinementEngineGPU(*ctx.engine, devices[k].device, o.nthreads, devices[k].own_lane); +#endif + if (!o.engine) + o.engine = MakeRefinementEngineCPU(o.cell, *o.sg, o.d_min, o.refl, o.nthreads); + o.engine->SetAniso(&o.aniso); + o.engine->SetRestraints(1, own[k]->topo.adp); + } + ctx.Restore(state.fit); + } + { + const RefinementStage stage("balanced_weight"); + w0 = BalancedWeight(ctx, refinable); + adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable); + } + logger.Info("Refinement, round {}: balanced X-ray weights {:.3g} (positions), {:.3g} (B), B ceiling {:.0f}", round, + w0, adp_w0, ctx.b_ceiling); + TrialOutcome rung = ladder_choice(state.pos, state.b, state.aniso, &state.fit); + if (rung.ran && displaces(rung.trial, state.trial)) + state = std::move(rung); + std::vector unused; + const std::string step = fmt::format("round {} step", round); + state = trajectory(std::move(state), false, unused, step.c_str()); + if (state.trial.r_work == before.trial.r_work && state.trial.r_free == before.trial.r_free) { + logger.Info("Refinement, round {}: the weights leave the model where it was", round); + break; + } + displacements(own_tls.empty() ? input_tls : own_tls); + const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE; + logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free, + kept ? "" : ", not kept"); + if (!kept) { + state = before; + break; + } } // 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every // trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what @@ -3315,13 +3407,6 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(std::chrono::steady_clock::now() - start).count(); - return result; - } - finish(); logger.Info("Refinement: done in {:.1f} s, {} target evaluations{}", result.seconds, ctx.evaluations, result.restrained ? fmt::format("; R-work {:.4f}, R-free {:.4f}, weight {:.3g}", result.r_work, @@ -3339,59 +3424,6 @@ double FinalRWork(const RefinementResult &r) { return r.restrained ? r.r_work : r.r_work_placement; } -// The refinement run again from the model it ended with, while that lowers R-free by more than the tie, up to -// REFINEMENT_ROUNDS in all. One refinement chooses its weights on the B factors as given and its displacement model -// only after them, so where the displacements change much - a placed prediction, whose B factors are its pLDDT - the -// weights were chosen for B factors it no longer has: on a placed prediction of three copies the second round chose -// an eight times lower X-ray weight, R-work and R-free both fell and R-free went 0.418 -> 0.405 (with the solvent -// build 0.427 -> 0.409 -> 0.401), where REFMAC5 reached 0.41-0.43 from the same model. The first round's view of -// the model as given is kept; the solvent build is the first round's. -constexpr int REFINEMENT_ROUNDS = 3; - -RefinementResult RefineRounds(const gemmi::Structure &placed, const std::vector &merged, - const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, - Logger &logger, const RefinementSettings &settings, - const std::vector &twin_laws = {}, size_t null_first = 0, - CardHeld *held = nullptr) { - RefinementResult result = RefineOnce(placed, merged, cell, sg, wavelength_A, logger, settings, twin_laws, - null_first, held); - RefinementSettings again = settings; - again.solvent = false; - again.placement = false; // the chains and the B per residue were placed by the first round - for (int round = 2; round <= REFINEMENT_ROUNDS; round++) { - if (!result.ok || !result.restrained || result.as_given || result.cancelled || settings.score_only || - result.model == nullptr) - break; - logger.Info("Refinement, round {}: from the model the round before ended with (R-free {:.4f})", round, - result.r_free); - again.keep_below = result.r_free - R_FREE_TIE; - RefinementResult next = RefineOnce(*result.model, merged, cell, sg, wavelength_A, logger, again, twin_laws, - null_first); - if (!next.ok || !next.restrained || next.as_given || !(next.r_free < result.r_free - R_FREE_TIE)) { - logger.Info("Refinement, round {}: not kept (R-free {:.4f})", round, FinalRFree(next)); - result.seconds += next.seconds; - break; - } - next.seconds += result.seconds; - next.r_work_start = result.r_work_start; - next.r_free_start = result.r_free_start; - next.score_start = result.score_start; - next.adp_input = result.adp_input; - next.adp_input_aniso_atoms = result.adp_input_aniso_atoms; - next.rigid_bodies = result.rigid_bodies; - next.group_b = result.group_b; - next.r_work_placement = result.r_work_placement; - next.r_free_placement = result.r_free_placement; - next.geometry_start = result.geometry_start; - next.ncs_rmsd_start = result.ncs_rmsd_start; - next.solvent_build = result.solvent_build; - if (next.adp_model == "the model's U held") // the round before's displacement model, refined on - next.adp_model = result.adp_model; - result = std::move(next); - } - return result; -} - // The twin hypotheses beside the untwinned refinement `result`, as a ladder. First one law: the placed model refined // again, solvent build and displacement model included, against the data as twinned by a law of the lattice, and the // twinned refinement taken (into `result`) where the law beats its own null (TwinNull) on the free reflections by @@ -3467,7 +3499,7 @@ void DecideTwin(const gemmi::Structure &placed, const std::vector kept{law}; @@ -3650,7 +3682,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector Date: Sat, 14 Nov 2026 12:18:49 +0100 Subject: [PATCH 5/8] Refinement: a model whose group is a subgroup of the data's refined in its own group, against the merge expanded to it A deposited P2(1) model whose eight copies are pseudo-symmetric, validated against a merge in P4(1)2(1)2 (the data cannot tell the two apart): the validation puts the model through a change of basis into its own group and scores it there, but the refinement was handed the model under the data's group. The operators the model does not have overlaid its copies four times over, the solvent and the overall B ran to their bounds, and the "refined" R-free was the null's (0.33 unrefined -> 0.60 refined; p_model.pdb said P 21 1 1, p_refined_model.pdb P 41 21 2). RefineModel now refines in the placed model's group where that is a proper subgroup of the data's (every rotation and centring of the model's group among the data's, the data's group the larger): the merge is expanded to the subgroup - each reflection standing for every mate the data's group made equivalent, at the merged amplitude, with its free flag, the Bijvoet pair swapped for a Friedel mate - and no twin law is tested, the data carrying the larger symmetry exactly. RefinementResult::space_group names the group refined in; the run's refined model and `--mode refine`'s output are written in it. On the case above the refinement now starts where the validation ended (as given 0.322 against 75965 reflections in P 21 1 1) and ends at 0.320. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK --- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 14 ++- docs/RUGNUX_CLI.md | 3 +- docs/RUGNUX_MODEL_AND_REFERENCE.md | 3 +- .../structure_refinement/Refinement.cpp | 64 ++++++++++++- .../structure_refinement/Refinement.h | 13 +++ rugnux/RugnuxRefine.cpp | 6 +- rugnux/RugnuxScaleMerge.cpp | 5 +- tests/RefinementTest.cpp | 92 +++++++++++++++++++ 8 files changed, 191 insertions(+), 9 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 90257354f..ca80138b7 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -27,8 +27,18 @@ statistics and its map coefficients go out; the restraint dictionary is compiled ### 15.1 What is refined, and in what order The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed, -and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within -100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS +and every B inside the range the refinement allows (2 Ų up to twice the model's largest B, within 100-300 Ų; +a model that arrives with B above 300 Ų - a structure refined at 4-5 Å - gets a ceiling of 1.5 times its +largest B instead, so that it is not flattened; a later round of the weights (§15.1, 4) raises the ceiling from +the refined model). + +The model is refined in the data's space group - except where its own group is a proper subgroup of it: a +P2₁ model whose copies are pseudo-symmetric, validated in its own group against a merge in P4₁2₁2 that the +data could not tell from P2₁. Under the data's group the operators the model does not have would overlay its +copies and R would be the null's, so it is refined in its own group, against the merge expanded to that group +(every reflection the data's group made equivalent, at the merged amplitude and with the same free flag), and +no twin law is tested, the data carrying the larger symmetry exactly. The refined model is written in the group +it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index f263db765..7da5065ff 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -184,7 +184,8 @@ The columns are found, not named - amplitudes with their sigmas (F/SIGF, FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none. -The model is taken as placed in the data's cell, and refined in the data's space group. +The model is taken as placed in the data's cell, and refined in the data's space group (in its own where that is a +subgroup of the data's, against the data expanded to it). | Option | Description | | --- | --- | diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index 34915dea2..de168046a 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -321,7 +321,8 @@ rugnux --mode refine --model model.cif -o refined data.mtz ``` The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by -content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group. The +content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its +own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index b2dc4f611..8f94870b3 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -3659,9 +3659,66 @@ bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated) { return twin_fraction > 0 && (null_z >= TWIN_NULL_Z || (data_indicated && null_z >= TWIN_NULL_Z_DATA)); } -RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged, - const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, - Logger &logger, const RefinementSettings &settings) { +bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super) { + if (sub.order() >= super.order()) + return false; + for (const gemmi::Op &op : sub.sym_ops) + if (super.find_by_rotation(op.rot) == nullptr) + return false; + for (const gemmi::Op::Tran &cen : sub.cen_ops) + if (std::find(super.cen_ops.begin(), super.cen_ops.end(), cen) == super.cen_ops.end()) + return false; + return true; +} + +std::vector ExpandToSubgroup(const std::vector &merged, const gemmi::SpaceGroup &from, + const gemmi::SpaceGroup &to) { + const gemmi::GroupOps from_ops = from.operations(), to_ops = to.operations(); + const gemmi::ReciprocalAsu asu(&to); + std::set seen; + std::vector out; + for (const MergedReflection &r : merged) + for (const gemmi::Op &op : from_ops.sym_ops) { // the centring changes no index + const auto [h, isym] = asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), to_ops); + if (!seen.insert(h).second) + continue; + MergedReflection e = r; + e.h = h[0]; + e.k = h[1]; + e.l = h[2]; + if (isym % 2 == 0) { // the Friedel opposite + std::swap(e.I_plus, e.I_minus); + std::swap(e.sigma_plus, e.sigma_minus); + std::swap(e.F_plus, e.F_minus); + std::swap(e.sigmaF_plus, e.sigmaF_minus); + } + out.push_back(e); + } + return out; +} + +RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged_in, + const gemmi::UnitCell &cell, const gemmi::SpaceGroup &data_sg, double wavelength_A, + Logger &logger, const RefinementSettings &settings_in) { + // In the model's own group where that is a proper subgroup of the data's: a model validated in P2(1), its copies + // pseudo-symmetric, against a merge in P4(1)2(1)2 the data could not tell from it. Its copies refine under their + // own symmetry, against the merge expanded to the subgroup - in the data's group the operators the model does not + // have would overlay its copies, and R would be the null's. The data carry the larger group's symmetry exactly, + // so a twin law of the subgroup's lattice has nothing to be tested on. + const gemmi::SpaceGroup *own = placed.find_spacegroup(); + const bool subgroup = own != nullptr && ProperSubgroup(own->operations(), data_sg.operations()); + const gemmi::SpaceGroup &sg = subgroup ? *own : data_sg; + RefinementSettings settings = settings_in; + std::vector expanded; + if (subgroup) { + expanded = ExpandToSubgroup(merged_in, data_sg, sg); + settings.twin = false; + logger.Info("Refinement: in the model's {}, a subgroup of the data's {} - the copies that group would overlay " + "refined under their own symmetry, against the {} reflections expanded to {} in the subgroup; no " + "twin law tested, the data having the larger group's symmetry exactly", + sg.xhm(), data_sg.xhm(), merged_in.size(), expanded.size()); + } + const std::vector &merged = subgroup ? expanded : merged_in; // The free set closed under the lattice's twin domains once, before either arm, so that both share it. std::vector closed; int moved = 0; @@ -3727,6 +3784,7 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg); +// Whether every operation of `sub` is one of `super`'s (its rotation, and its centring), `super` the larger group: +// `sub` is then a group the data merged in `super` could not tell from it (a P2(1) crystal whose copies are +// pseudo-symmetric, merged in P4(1)2(1)2). +bool ProperSubgroup(const gemmi::GroupOps &sub, const gemmi::GroupOps &super); + +// `merged`, the reflections of the group `from`, as the reflections of its subgroup `to`: each with every mate that +// `from` made equivalent and `to` keeps apart, at the same amplitude, intensity and free flag (its Bijvoet pair +// swapped where the mate is the Friedel opposite). +std::vector ExpandToSubgroup(const std::vector &merged, const gemmi::SpaceGroup &from, + const gemmi::SpaceGroup &to); + // Whether the twinned refinement is taken (RefineModel): a fraction above 0, and the law ahead of its null on the // free set by 3 standard errors - or by 2 where the data's own intensity statistics read twinned as well. bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated); diff --git a/rugnux/RugnuxRefine.cpp b/rugnux/RugnuxRefine.cpp index fe217da2b..436ef657e 100644 --- a/rugnux/RugnuxRefine.cpp +++ b/rugnux/RugnuxRefine.cpp @@ -197,7 +197,9 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m out.space_group, out.free_reflections, out.free_flags_generated ? "drawn here, the file has no free set" : "column " + data.free_column, d_min_limit > 0 ? fmt::format(", to {:.2f} A", d_min_limit) : std::string()); - if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); msg != nullptr && msg->number != sg->number) + // (a model whose group is a subgroup of the data's is refined in its own, RefineModel) + if (const gemmi::SpaceGroup *msg = model.find_spacegroup(); + msg != nullptr && msg->number != sg->number && !ProperSubgroup(msg->operations(), sg->operations())) logger.Warning("The model names space group {}, the data {}: refined in the data's", msg->xhm(), sg->xhm()); try { gemmi::Structure st = model; @@ -217,6 +219,8 @@ RefineOnlyResult RunRefineOnly(RugnuxReflections data, const gemmi::Structure &m RefinementSettings refine = settings; refine.as_given_competes = !out.free_flags_generated; out.refinement = RefineModel(st, refl, data.cell, *sg, out.wavelength_A, logger, refine); + if (!out.refinement.space_group.empty()) + out.space_group = out.refinement.space_group; } catch (const std::exception &e) { out.refinement = RefinementResult(); out.refinement.failure_reason = std::string("refinement failed: ") + e.what(); diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index 7040e837c..ea07e8a7c 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -5728,7 +5728,10 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { } const PlacedModelOutput &p_final = *placed_model_out; if (refined.ok) { - result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, p_final.space_group}; + // in the group it was refined in: the model's own where that is a subgroup of the data's (RefineModel) + const gemmi::SpaceGroup *refined_in = gemmi::find_spacegroup_by_name(refined.space_group); + result.outputs->refined_model = PlacedModelOutput{refined.model, p_final.cell, + refined_in ? *refined_in : p_final.space_group}; result.outputs->refined_maps = refined.maps; result.outputs->blobs = refined.blobs; result.outputs->solvent = refined.solvent; diff --git a/tests/RefinementTest.cpp b/tests/RefinementTest.cpp index ada536138..92db8c573 100644 --- a/tests/RefinementTest.cpp +++ b/tests/RefinementTest.cpp @@ -966,6 +966,98 @@ TEST_CASE("Refinement_trials_run_ahead", "[Refinement]") { CHECK(same == x.size()); } +// A model whose group is a proper subgroup of the data's: a P2(1) crystal whose two residues are pseudo-symmetric by +// the 2-fold along a, its data merged in P2(1)2(1)2(1). Refined in its own group against the data expanded to it, the +// model scores as it does in the validation; refined in the data's group its contents are overlaid and it does not. +TEST_CASE("Refinement_model_in_a_subgroup_of_the_data", "[Refinement]") { + const gemmi::UnitCell cell(36, 40, 44, 90, 90, 90); + const gemmi::SpaceGroup &sub = *gemmi::find_spacegroup_by_name("P 1 21 1"); + const gemmi::SpaceGroup &super = *gemmi::find_spacegroup_by_name("P 21 21 21"); + CHECK(ProperSubgroup(sub.operations(), super.operations())); + CHECK(!ProperSubgroup(super.operations(), sub.operations())); + CHECK(!ProperSubgroup(sub.operations(), sub.operations())); + CHECK(!ProperSubgroup(gemmi::find_spacegroup_by_name("C 1 2 1")->operations(), super.operations())); + gemmi::Structure truth = residue_crystal({"TRP"}, cell, "P 1 21 1", 5); + { + // the residue's mate by the 2-fold along a, exact: the crystal is then P 2 2(1) 1 (pseudo-P2(1)2(1)2(1)) + gemmi::Residue mate = truth.models[0].chains[0].residues[0]; + mate.seqid = gemmi::SeqId(2, ' '); + for (gemmi::Atom &a : mate.atoms) { + const gemmi::Fractional f = cell.fractionalize(a.pos); + a.pos = cell.orthogonalize(gemmi::Fractional(f.x, -f.y, -f.z)); + } + truth.models[0].chains[0].residues.push_back(mate); + } + const double d_min = 2.0; + // the data: the amplitudes of the truth in its own group, merged in the supergroup (each of its reflections the + // root mean square of the amplitudes it makes equivalent) + std::map amplitude; + { + std::vector own = asu_reflections(cell, sub, d_min); + RefinementReflections refl = PrepareRefinementReflections(own, cell, sub, d_min * 0.999); + auto cpu = MakeRefinementEngineCPU(cell, sub, d_min * 0.999, refl, 4); + cpu->SetBlur(RefinementBlur(cell, sub, d_min, 5.0)); + std::vector> f; + cpu->Fcalc(truth.models[0], f); + for (size_t m = 0; m < refl.Size(); m++) + amplitude[refl.hkl[m]] = std::abs(f[m]); + } + const gemmi::GroupOps super_ops = super.operations(), sub_ops = sub.operations(); + const gemmi::ReciprocalAsu sub_asu(&sub); + std::vector merged = asu_reflections(cell, super, d_min); + for (MergedReflection &r : merged) { + double sum = 0; + int n = 0; + for (const gemmi::Op &op : super_ops.sym_ops) { + const gemmi::Miller h = sub_asu.to_asu(op.apply_to_hkl({{r.h, r.k, r.l}}), sub_ops).first; + if (const auto it = amplitude.find(h); it != amplitude.end()) { + sum += it->second * it->second; + n++; + } + } + r.F = n > 0 ? static_cast(std::sqrt(sum / n)) : NAN; + r.sigmaF = 0.03f * r.F + 0.01f; + } + std::erase_if(merged, [](const MergedReflection &r) { return !std::isfinite(r.F); }); + + // the expansion: every reflection of the subgroup's set comes from one of the supergroup's, with its amplitude + // and its flag, and the supergroup's set is reached from it + const std::vector expanded = ExpandToSubgroup(merged, super, sub); + CHECK(expanded.size() > 1.5 * merged.size()); + CHECK(expanded.size() <= 2 * merged.size()); + const gemmi::ReciprocalAsu super_asu(&super); + std::map source; + for (const MergedReflection &r : merged) + source[{{r.h, r.k, r.l}}] = &r; + std::set distinct; + size_t consistent = 0; + for (const MergedReflection &e : expanded) { + distinct.insert({{e.h, e.k, e.l}}); + const auto it = source.find(super_asu.to_asu({{e.h, e.k, e.l}}, super_ops).first); + consistent += it != source.end() && it->second->F == e.F && it->second->rfree_flag == e.rfree_flag; + } + CHECK(distinct.size() == expanded.size()); + CHECK(consistent == expanded.size()); + + Logger logger("RefinementTest"); + RefinementSettings settings; + settings.gpu = false; + settings.nthreads = 4; + settings.score_only = true; + settings.riding_hydrogens = false; + const RefinementResult own_group = RefineModel(truth, merged, cell, super, 1.0, logger, settings); + REQUIRE(own_group.ok); + CHECK(own_group.space_group == "P 1 21 1"); + CHECK(own_group.n_work + own_group.n_free == static_cast(expanded.size())); + CHECK(own_group.r_free_start < 0.1); + gemmi::Structure as_data = truth; + as_data.spacegroup_hm = "P 21 21 21"; + const RefinementResult data_group = RefineModel(as_data, merged, cell, super, 1.0, logger, settings); + REQUIRE(data_group.ok); + CHECK(data_group.space_group == "P 21 21 21"); + CHECK(data_group.r_free_start > 0.3); +} + // A K+ on the 4-fold axis of P4 at occupancy 0.25 (its four coinciding copies one ion), 0.2 A off the axis in the // model as given, with a water 2.8 A from it: the topology puts the ion on the axis, its contact with the water is // listed once (not once per copy) and none with itself; refinement keeps it on the axis at the occupancy given, From 13ddef63d344d113e4bb86660106ff42610350d1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 14 Nov 2026 12:19:12 +0100 Subject: [PATCH 6/8] Refinement: the B ceiling lifted for a model that arrives with B above it The ceiling of B was clamp(2 Bmax, 100, 300), and PrepareModel cut every B to 300 before it was set: a structure refined at 4-5 A (mean B 380, most atoms above 300) was flattened, its as-given R-free 0.05 above the validation's on the same model (0.359 against 0.306), and the refinement ended worse than the validation (0.339). The 300 bound stays for every model under it (it sizes the gather); a model above it gets a ceiling of 1.5 times its largest B (BCeiling), PrepareModel keeps only the floor, and an anisotropic U is scaled to the model's ceiling, not the constant. The case above: as given 0.325, refined 0.242. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK --- .../structure_refinement/Refinement.cpp | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 8f94870b3..2dbc0ba8d 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -52,6 +52,12 @@ namespace { constexpr double B_FLOOR = 2.0; // A^2: no atom's B goes below it // nor above twice the model's largest B, within these bounds (the GPU's gather is reserved for it) constexpr double B_CEILING_MIN = 100.0, B_CEILING_MAX = 300.0; +// The ceiling of B for a model whose highest B is `b_max`: twice it, within the bounds - the bounds lifted for a +// model that comes with B above them (a structure refined at 4-5 A, B 400-600 A^2), which the ceiling would +// otherwise flatten (the gather is reserved for it). +double BCeiling(double b_max) { + return std::clamp(2 * b_max, B_CEILING_MIN, std::max(B_CEILING_MAX, 1.5 * b_max)); +} // A B tensor (8 pi^2 U) inside the bounds: its smallest eigenvalue raised to `floor` by a multiple of I, then the // whole tensor scaled down so that its largest is at most `ceiling`. The ceiling comes last because the GPU's @@ -141,7 +147,7 @@ gemmi::Structure PrepareModel(const gemmi::Structure &placed, const gemmi::UnitC a.charge = ScatteringCharge(r, a); if (a.aniso.nonzero()) a.b_iso = static_cast(gemmi::u_to_b() * a.aniso.trace() / 3.0); - a.b_iso = static_cast(std::clamp(a.b_iso, B_FLOOR, B_CEILING_MAX)); + a.b_iso = static_cast(std::max(a.b_iso, B_FLOOR)); } } gemmi::remove_empty_children(st); @@ -266,8 +272,8 @@ struct Context { continue; // its widest axis no wider than the refinement allows const double top = AnisoRadiusB(at.aniso.scaled(gemmi::u_to_b())); - if (top > B_CEILING_MAX) { - at.aniso = at.aniso.scaled(static_cast(B_CEILING_MAX / top)); + if (top > b_ceiling) { + at.aniso = at.aniso.scaled(static_cast(b_ceiling / top)); at.b_iso = static_cast(gemmi::u_to_b() * at.aniso.trace() / 3.0); } const double t = at.aniso.trace() / 3.0; @@ -2308,7 +2314,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vectorb_iso + gemmi::u_to_b() * body.U(ctx.atoms[i]->pos).trace() / 3.0); - ctx.b_ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX); + ctx.b_ceiling = BCeiling(b_max); if (ctx.atoms.empty()) { result.failure_reason = "the model has no atoms"; return result; @@ -3297,7 +3303,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(b_max, ctx.atoms[i]->b_iso); - const double ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX); + const double ceiling = BCeiling(b_max); if (ceiling > ctx.b_ceiling) { const RefinementStage stage("engine"); for (std::unique_ptr &o : own) From 44b905de1d57579b3aa90993364001e0f9ba50ee Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 14 Nov 2026 12:19:28 +0100 Subject: [PATCH 7/8] Refinement: the B factors' X-ray weight tried at a quarter and a sixteenth of the balanced one as well A model whose B factors arrive far from where they refine - a prediction's pLDDT - balances a weight that is not the one its B wants; the lower rungs took 0.002-0.010 off R-free on five such starts (molecular-replacement placements of predictions), nothing off eight deposited models, at 5-30% more refinement time. They run as the balanced weight did: at the middle X-ray rung first, at the others where refining B comes within two ties of holding it. Also: the rounds of the weight and displacement-model choice are a setting (RefinementSettings::rounds); every trial reports the -ln L of its free reflections; and the displacement model named in the result stays the one an earlier round chose where a later round changes nothing (it was reported as isotropic). Measured against: deciding every trial on -ln L free instead of R-free (0.003-0.012 better on the prediction starts, but 0.004-0.02 worse on six of 22 deposited models, so not taken); a hypothesis restarting the isotropic B from flat (never displaced any); 60 L-BFGS iterations per macro-cycle (another 0.004 on the predictions, +60-100% time, mixed on the deposited models - not taken). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK --- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 6 ++++- .../structure_refinement/Refinement.cpp | 22 ++++++++++--------- .../structure_refinement/Refinement.h | 4 +++- 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index ca80138b7..50d7b697e 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -307,7 +307,11 @@ geometry - is kept). The trials are: 1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors - held and refined at $w_{B,0}$ - six trials. + held and refined at $w_{B,0} \times \{1/16, 1/4, 1\}$ - twelve trials (the B factors are refined first at + the middle rung only, and at the others where that comes within two ties of holding them there). The lower + B weights are for a model whose B factors arrive far from where they refine - a prediction's pLDDT - whose + balanced weight is not the one its B wants: they took 0.002-0.010 off R-free on five such starts, and + nothing off eight deposited models. 2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 2dbc0ba8d..c24b3e163 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -86,7 +86,6 @@ constexpr int SOLVENT_ITERATIONS = 10; constexpr double R_FREE_TIE = 0.001; // The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each // after it from the model the one before ended with. -constexpr int REFINEMENT_ROUNDS = 3; // The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this, // relative; the most a position (A) or a B (A^2) moves in the first trial of a line search. constexpr double MINIMISER_F_TOLERANCE = 1e-7; @@ -1859,10 +1858,9 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s t.within_bounds = t.geometry.rmsd_bonds <= MAX_RMSD_BONDS && t.geometry.rmsd_angles <= MAX_RMSD_ANGLES; if (ncs != nullptr && !ncs->Empty()) t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB()); - if (hyp != nullptr) { + if (hyp != nullptr) t.adp = hyp->name; - t.nll_free = ctx.Score().nll_free; - } + t.nll_free = ctx.Score().nll_free; out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, " "angles {:.2f} deg{}{}", hyp != nullptr ? hyp->name + ", " : std::string(), w, @@ -1871,8 +1869,7 @@ TrialOutcome RunTrial(TrialSlot slot, const std::vector &refinable, const s ncs == nullptr || ncs->Empty() ? std::string() : ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd) : fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd), - rf.r_work, rf.r_free, - hyp != nullptr ? fmt::format(", -LL free {:.1f}", t.nll_free) : std::string(), + rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), t.geometry.rmsd_bonds, t.geometry.rmsd_angles, t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal) : std::string(), @@ -2695,7 +2692,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; std::vector outcomes; // The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the @@ -3249,7 +3249,9 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k); } - result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held"; + // (the model where the weights ended keeps the displacement model a round before chose for it) + if (result.adp_model.empty() || state.trial.adp.empty()) + result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held"; if (best >= 0) { result.adp_model = outcomes[best].trial.adp; state = std::move(outcomes[best]); @@ -3278,7 +3280,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector weight_ladder = {1.0 / 1024, 1.0 / 128, 1.0 / 16}; - std::vector adp_ladder = {0.0, 1.0}; + std::vector adp_ladder = {0.0, 1.0 / 16, 1.0 / 4, 1.0}; // Then at most this many steps of the trajectory, each a set of trials from where the last one ended, int max_steps = 8; // and the final trials from where it ended, at its weights times each of these; @@ -74,6 +74,8 @@ struct RefinementSettings { // groups (the model's own, one per chain, the chains cut into TLSMD segments) with a residual B per atom, and an // anisotropic U per atom with rigid-bond and similarity restraints (false: the isotropic B, or the model's U held). bool adp_hypotheses = true; + // Rounds of the weights and the displacement model decided again from the refined model (RefineOnce, 4b.). + int rounds = 3; // A refinement against the data as twinned by each twin law the lattice admits (RefinementTwinLaws), beside the // untwinned one, and the model of whichever the evidence favours (RefineModel). bool twin = true; From 5f8a5814f0ba11f3d0e2ff194ccb114bcd961ca7 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 14 Nov 2026 13:23:10 +0100 Subject: [PATCH 8/8] Refinement: a trajectory step taken on R-work alone is followed by one more at most On a model refined before, R-work falls step after step while R-free does not, and the trajectory ends at its best anyway; each such step is a wave of trials. A model of 490k atoms spent six of them (240 s of a 760 s refinement) on a round it then gave back; with the rule it refines to the same R-free in 620 s. On 28 refinements of 22 deposited models and 6 predictions the rule changed nothing. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK --- image_analysis/structure_refinement/Refinement.cpp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index c24b3e163..f9b6752bd 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -2779,7 +2779,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector trials = {{w, adp, ncs_s}}; if (step % 2 == 1 && adp > 0) { @@ -2812,6 +2817,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector