diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index b0a12e309..90257354f 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -28,7 +28,7 @@ statistics and its map coefficients go out; the restraint dictionary is compiled The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed, and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within -100-300 Ų). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS +100-300 Ų; a later round of the weights (§15.1, 4) raises the ceiling from the refined model). An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest @@ -309,9 +309,9 @@ are: was never kept on 55 refinements, and taking it away changed no choice). 4. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U held), each trial as long as a step on the positions where the weights ended, held - only the displacement - parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with - the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held - positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain + parameters refine, so the trials differ in the displacement model alone (where the geometry is out of bounds + there, which held positions cannot mend, each trial refines the positions too, a macro-cycle longer than a + step): TLS groups with a residual B - the model's own, one per chain of at least 50 refinable atoms, and the chains cut into 4, 8 and 16 contiguous segments of at least 10 residues as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r), to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U @@ -327,6 +327,16 @@ are: `REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no TLS group holds a water. + Then the weights and the displacement model are **decided again** from the model as it now is, up to twice: the + B ceiling and the balanced weights from its B factors, the ladder from it with its displacements as they are, the + trajectory, and the displacement hypotheses from where that ends. The first choice took its weights with the + B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT, + the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$), + the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the + refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers + R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the + hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder + and one step. No round runs where the refinement does not displace the model as given. 5. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 3bcb8e59a..b2dc4f611 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -78,6 +78,9 @@ constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0; constexpr int SOLVENT_ITERATIONS = 10; // R-free differences smaller than this do not decide between two weights: the tighter geometry is taken. constexpr double R_FREE_TIE = 0.001; +// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each +// after it from the model the one before ended with. +constexpr int REFINEMENT_ROUNDS = 3; // The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this, // relative; the most a position (A) or a B (A^2) moves in the first trial of a line search. constexpr double MINIMISER_F_TOLERANCE = 1e-7; @@ -1446,7 +1449,29 @@ void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) { for (size_t d = first; d < a.size(); d++) result.twin_fraction_null += a[d]; } - const std::vector> f_null = TwinnedModel(null, rows, ctx.fcalc); + // TwinnedModel with each unrelated mate on its own scale k_total: TwinnedModel puts a mate on the scale of the + // reflection it is the mate of, as the twin domain's reflection at the same spot is measured, but an unrelated + // mate lies in another direction, where the anisotropic scale can differ many times over (22 times in amplitude + // at 2 A, on a model whose U carried the anisotropy opposite to k_total), and on the reflection's scale it put + // the null 0.06 in R-free above the untwinned refinement - a law "ahead" of its null by 7.9 standard errors at a + // fraction of 0.04. + std::vector> f_null(n); + double own_fraction = 1.0; + for (float a : rows.twin_fractions) + own_fraction -= a; + for (size_t m = 0; m < n; m++) { + const std::complex u = std::complex(ctx.fcalc[m]) + std::complex(rows.fmask[m]); + double i_model = own_fraction * rows.k_total[m] * rows.k_total[m] * std::norm(u); + for (size_t d = 0; d < null.Laws(); d++) { + const int j = null.mates[d][m]; + const double k = d < first ? rows.k_total[m] : rows.k_total[j]; + i_model += rows.twin_fractions[d] * k * k * + std::norm(std::complex(ctx.fcalc[j]) + std::complex(rows.fmask[j])); + } + const double amplitude = std::sqrt(std::max(i_model, 0.0)); + f_null[m] = std::norm(u) > 0 ? std::complex(u * (amplitude / std::abs(u))) + : std::complex(static_cast(amplitude), 0); + } result.twin_r_free_null = ComputeRFactors(null, f_null).r_free; // How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin|| // over its standard error. The members of a free twin group share their F_calc, so their terms are summed into @@ -2665,6 +2690,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; + std::vector outcomes; + // The ladder from `from` (positions, B, anisotropy; `fit` what was fitted there, null to fit it), at the + // balanced weights as they stand: the trial kept, or one that did not run where none did. + auto ladder_choice = [&](const std::vector &from_pos, const std::vector &from_b, + const std::vector &from_aniso, const Context::Fit *fit) { std::vector ladder; for (double rung : settings.weight_ladder) for (double adp : settings.adp_ladder) { @@ -2672,14 +2703,13 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector aniso0 = ctx.aniso; - std::vector outcomes(ladder.size()); + outcomes.assign(ladder.size(), TrialOutcome()); auto run_ladder = [&](const std::function &these) { std::vector t; for (const TrialWeights &l : ladder) if (these(l)) t.push_back(l); - std::vector ran = RunTrials(trial_slots, t, refinable, pos0, b0, aniso0, nullptr, + std::vector ran = RunTrials(trial_slots, t, refinable, from_pos, from_b, from_aniso, fit, settings.macro_cycles, settings, ncs_p); for (size_t i = 0, m = 0; i < ladder.size(); i++) if (these(ladder[i])) @@ -2720,10 +2750,10 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0) - state = std::move(outcomes[k]); + const int k = choose(outcomes, "ladder"); + return k >= 0 ? std::move(outcomes[k]) : TrialOutcome(); + }; + TrialOutcome state = ladder_choice(pos0, b0, aniso0, nullptr); // 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they // are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the @@ -2738,7 +2768,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector from_best; // `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since). - auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best) { + // `label`: the steps' name in the log. + auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best, const char *label) { TrialOutcome best = state; from_best.clear(); bool state_is_best = true; @@ -2763,7 +2794,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0 && displaces(outcomes[k].trial, state.trial)) state = std::move(outcomes[k]); } @@ -3087,7 +3118,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector &own_tls) { const RefinementStage adp_stage("adp_hypotheses"); ctx.SetPositions(state.pos); ctx.SetB(state.b); @@ -3097,7 +3129,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector hyps = AdpHypotheses(ctx, topo, input_tls); + std::vector hyps = AdpHypotheses(ctx, topo, own_tls); const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0; // the U weight balanced on the model with every refinable atom given a U double u_w0 = adp; @@ -3232,6 +3264,89 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector own_tls = state.tls; + ctx.SetPositions(state.pos); + ctx.SetB(state.b); + ctx.SetAniso(state.aniso); + ctx.Restore(state.fit); + PlaceRiders(ctx, topo); + // the ceiling of B from the model as it now is, as from the model as given: a placed prediction comes + // with B factors of 6 A^2 under an overall B of 45 (its ceiling 100) and refines at B near 50-70, where + // the TLS bodies and the U per atom were held at the ceiling; where it rises the engines are made again + // (the gather is reserved for it) + double b_max = 0; + for (size_t i = 0; i < ctx.atoms.size(); i++) + b_max = std::max(b_max, ctx.atoms[i]->b_iso); + const double ceiling = std::clamp(2 * b_max, B_CEILING_MIN, B_CEILING_MAX); + if (ceiling > ctx.b_ceiling) { + const RefinementStage stage("engine"); + for (std::unique_ptr &o : own) + if (o) + o->ctx.engine.reset(); + ctx.b_ceiling = ceiling; + MakeEngine(ctx, settings.gpu, settings, logger, restraint_records, restraint_slots); + ctx.engine->SetRestraints(0, RestraintsNow(ctx, topo)); + ctx.engine->SetRestraints(1, topo.adp); + for (size_t k = 0; k < own.size(); k++) + if (own[k]) { + Context &o = own[k]->ctx; + o.b_ceiling = ceiling; +#ifdef JFJOCH_USE_CUDA + if (ctx.engine->OnGPU()) + o.engine = MakeRefinementEngineGPU(*ctx.engine, devices[k].device, o.nthreads, devices[k].own_lane); +#endif + if (!o.engine) + o.engine = MakeRefinementEngineCPU(o.cell, *o.sg, o.d_min, o.refl, o.nthreads); + o.engine->SetAniso(&o.aniso); + o.engine->SetRestraints(1, own[k]->topo.adp); + } + ctx.Restore(state.fit); + } + { + const RefinementStage stage("balanced_weight"); + w0 = BalancedWeight(ctx, refinable); + adp_w0 = BalancedADPWeight(ctx, BRestraintsNow(ctx, topo), refinable); + } + logger.Info("Refinement, round {}: balanced X-ray weights {:.3g} (positions), {:.3g} (B), B ceiling {:.0f}", round, + w0, adp_w0, ctx.b_ceiling); + TrialOutcome rung = ladder_choice(state.pos, state.b, state.aniso, &state.fit); + if (rung.ran && displaces(rung.trial, state.trial)) + state = std::move(rung); + std::vector unused; + const std::string step = fmt::format("round {} step", round); + state = trajectory(std::move(state), false, unused, step.c_str()); + if (state.trial.r_work == before.trial.r_work && state.trial.r_free == before.trial.r_free) { + logger.Info("Refinement, round {}: the weights leave the model where it was", round); + break; + } + displacements(own_tls.empty() ? input_tls : own_tls); + const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE; + logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free, + kept ? "" : ", not kept"); + if (!kept) { + state = before; + break; + } } // 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every // trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what @@ -3292,7 +3407,6 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector