Merge branch 'rc188-f' into rc188-int
This commit is contained in:
commit
96cd005b21
6 files changed
+371
-165
No files matched your search
+17
-1
@@ -652,6 +652,16 @@ structure refinement with phenix.refine" (2012), Acta Cryst. D68, 352-367
|
||||
A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal
|
||||
structures" (1992), Nature 355, 472-475 [doi:10.1038/355472a0](https://doi.org/10.1038/355472a0).
|
||||
|
||||
**What the free set decides** — the free set decides the weights and the displacement model alone, and only
|
||||
where its two measures agree: a candidate is taken where its free $-\ln L$ is lower and its R-free not higher
|
||||
than the incumbent's by more than the paired standard error of the difference; every other step of the
|
||||
refinement is taken on the work-set target. The approach of deciding on more than one free measure, and of keeping the
|
||||
free set for few choices, is PDB-REDO's: R. P. Joosten, K. Joosten, G. N. Murshudov and A. Perrakis,
|
||||
"PDB_REDO: constructive validation, more than just looking for errors" (2012), Acta Cryst. D68, 484-496
|
||||
[doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515), and R. P. Joosten, F. Long,
|
||||
G. N. Murshudov and A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization" (2014),
|
||||
IUCrJ 1, 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324).
|
||||
|
||||
**Robust scaling of the model** — the overall scale and anisotropic B are fitted by iteratively
|
||||
reweighted least squares with the pseudo-Huber weight. P. J. Huber, "Robust estimation of a location
|
||||
parameter" (1964), Ann. Math. Statist. 35, 73-101
|
||||
@@ -692,7 +702,13 @@ SHELXL" (2015), Acta Cryst. C71, 3-8
|
||||
[doi:10.1107/S2053229614024218](https://doi.org/10.1107/S2053229614024218) documents them. Which
|
||||
displacement model a refinement keeps is decided on the free set rather than by a rule of
|
||||
resolution: E. A. Merritt, "To B or not to B: a question of resolution?" (2012), Acta Cryst. D68,
|
||||
468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320).
|
||||
468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320). Between a model
|
||||
with more displacement parameters and one with fewer, the R-factor ratio test of W. C. Hamilton, "Significance
|
||||
tests on the crystallographic R factor" (1965), Acta Cryst. 18, 502-510 (DOI not verified), decides first,
|
||||
over a range of restraint weights as Merritt (2012) applies it, with PDB-REDO's (Joosten et al. 2012, above)
|
||||
95%/30% rule and its bound of 18 reflections per atom for a U per atom; its F point is from E. Paulson,
|
||||
"An approximate normalization of the analysis of variance distribution" (1942), Ann. Math. Statist. 13,
|
||||
233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608).
|
||||
|
||||
**Population and crossover** — the members' refinement on blurred data is a continuation (homotopy) on the
|
||||
resolution, the deformation of the landscape of L. Piela, J. Kostrowicki and H. A. Scheraga, "On the
|
||||
|
||||
@@ -18,7 +18,7 @@ This document describes the crystallographic algorithms implemented in Jungfrauj
|
||||
12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
|
||||
13. amplitude estimation (French–Wilson) and R-free test-set flagging,
|
||||
14. optional model-based validation: rigid-body placement of a supplied model (molecular replacement where the rigid body does not place it, or for a predicted model), R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.
|
||||
15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight chosen by R-free within geometry bounds, and the unexplained blobs of the refined difference map.
|
||||
15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight and the displacement model chosen on the free set (free -ln L and R-free agreeing) within geometry bounds, every other step taken on the work-set target, and the unexplained blobs of the refined difference map.
|
||||
|
||||
|
||||
The reference is split into five parts, in pipeline order; the section numbers run continuously
|
||||
|
||||
@@ -42,8 +42,8 @@ no twin law is tested, the data carrying the larger symmetry exactly. The refine
|
||||
it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
|
||||
groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the
|
||||
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
|
||||
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
|
||||
R-free on the model as given (`REFINE_ADP_INPUT`). Occupancies are kept as they are, and
|
||||
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, the free set prefers on
|
||||
the model as given (§15.7; `REFINE_ADP_INPUT`). Occupancies are kept as they are, and
|
||||
so are alternative conformations, each refined with its own restraints.
|
||||
|
||||
Two stages follow.
|
||||
@@ -51,12 +51,12 @@ Two stages follow.
|
||||
1. **Placement.** Each chain of at least 50 non-water atoms is moved as a rigid body - an angle-axis
|
||||
rotation about its centroid, scaled by its rms radius so that a unit of rotation moves a typical atom by
|
||||
about an angstrom, and a translation - against the X-ray target (§15.2), by Ceres' L-BFGS for 30
|
||||
iterations. Then one B shift per residue. Each of the two is **kept only where it lowers R-free by more
|
||||
than 0.001**: a deposited model was refined atom by atom already, and moving its chains or shifting its
|
||||
B factors residue by residue then only makes it worse. The placement stage needs no dictionary.
|
||||
iterations. Then one B shift per residue. Each of the two is **kept only where it lowers the $-\ln L$ of the
|
||||
working reflections**, the target it minimised: the free set decides the weights and the displacement model
|
||||
alone (§15.7). The placement stage needs no dictionary.
|
||||
2. **Restrained refinement.** Every atom's position and isotropic B, against the X-ray target and the
|
||||
geometry of the dictionary (§15.4-§15.5) - the built-in one with the user's restraint files over it
|
||||
(`--restraints`) - in weight trials chosen by R-free (§15.7). Residues the
|
||||
(`--restraints`) - in weight trials chosen on the free set (§15.7). Residues the
|
||||
dictionary does not describe, and residues with an atom it does not know, are **held fixed**: they stay
|
||||
in the structure factors, in the bulk-solvent mask and in the repulsions the refined atoms see, but
|
||||
they do not move. So do atoms of zero occupancy.
|
||||
@@ -344,29 +344,52 @@ $w_B\,T_\mathrm{X\text{-}ray} + E_B$ for the B factors. Rather than fixing $w$ a
|
||||
runs **short refinements at several weights and keeps the one that does best on the free set** - a weight
|
||||
chosen late, on the evidence, instead of early at a rule.
|
||||
|
||||
**What the free set decides.** The free set decides the weights - the X-ray weights of the positions and of the B
|
||||
factors and the NCS restraints' $\sigma$ - the displacement model (§15.7, 6.), the reading of the model's TLS groups
|
||||
(§15.1) and whether the model as given is kept over the refinement (7.), and nothing else. Each of these choices asks
|
||||
both of its measures: a candidate is preferred to the incumbent only where its $-\ln L$ of the free reflections is
|
||||
lower **and** its R-free is not higher by more than the standard error of the difference. The two are scored on the
|
||||
same free reflections, so the difference is paired: R-free is a sum of one term per free reflection (per free twin
|
||||
group under a twin law, whose members share their $F_\mathrm{calc}$), and the standard error is that of the sum of
|
||||
the terms' differences - two trials from one start differ far less from reflection to reflection than either
|
||||
scatters about the truth, so this is narrower than the error of one R-free, $R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$.
|
||||
One measure alone, or a fixed tie, picks the noise of a few hundred reflections as often as a better model (PDB-REDO
|
||||
likewise takes a change only where its free measures agree, Joosten et al. 2014). $-\ln L$ free is that of the
|
||||
refinement's own likelihood parameters - the variance fitted to the free reflections; with D and the variance fitted
|
||||
to the working reflections instead, it kept out a per-atom U that overfits at 2.2 Å, but held back every
|
||||
refinement (on 17 structures R-free 0.0016 and R-work 0.008 higher on average), and it is not used. Every other step - a step of the
|
||||
trajectory, a round, the placement's rigid bodies and B per residue, the solvent build's rounds, the side-chain
|
||||
field, the population's child - is taken on the **work-set target**: the target the trial minimises,
|
||||
$w\,T_\mathrm{X\text{-}ray} + E_\mathrm{geometry}$ (with the NCS restraints at their $\sigma$), plus
|
||||
$w_B\,T_\mathrm{X\text{-}ray} + E_B$ where the B factors are refined, $T_\mathrm{X\text{-}ray}$ the $-\ln L$ of the
|
||||
working reflections - at the weights the two candidates share, and taken where it lowers that by more than
|
||||
$10^{-4}$ of it, relative, and the geometry stays within bounds where it was. A step at other weights (X-ray weights
|
||||
or NCS $\sigma$) is not compared on a target that changed with them - the positions and the B factors minimise
|
||||
targets of their own, whose sum a change of weight can raise while it lowers each - and the free set decides it as
|
||||
it decides every weight. The free set thus tests a
|
||||
handful of choices, each between a few candidates, and the R-free reported is not the best of the many steps the
|
||||
refinement took.
|
||||
|
||||
The starting weights balance the gradients: $w_0 = |\nabla E_\mathrm{geometry}| / |\nabla T_\mathrm{X\text{-}ray}|$,
|
||||
the geometry's gradient taken on the model shaken by 0.1 Å (a refined model's geometry would give none), and
|
||||
$w_{B,0}$ likewise from the B restraints on the B factors shaken by 20%. Each **trial** is two macro-cycles
|
||||
(the scale, solvent and likelihood refitted, then the positions, then the B factors) from where it starts.
|
||||
A trial is **within bounds** where its rmsd from the dictionary is at most 0.02 Å for bonds and 2° for
|
||||
angles; one trial **displaces** another where it is within bounds and its R-free is lower by more than 0.001
|
||||
(differences smaller than that do not decide, and the trial listed first - on a ladder, the tighter
|
||||
geometry - is kept). The trials
|
||||
are:
|
||||
angles; one trial **displaces** another where it is within bounds and the free set prefers it as above (the trial
|
||||
listed first - on a ladder, the tighter geometry - is kept otherwise). The trials are:
|
||||
|
||||
1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors
|
||||
held and refined at $w_{B,0} \times \{1/16, 1/4, 1\}$ - twelve trials (the B factors are refined first at
|
||||
the middle rung only, and at the others where that comes within two ties of holding them there). The lower
|
||||
the middle rung only, and at the others unless holding them there displaces refining them). The lower
|
||||
B weights are for a model whose B factors arrive far from where they refine - a prediction's pLDDT - whose
|
||||
balanced weight is not the one its B wants: they took 0.002-0.010 off R-free on five such starts, and
|
||||
nothing off eight deposited models.
|
||||
2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights
|
||||
as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at
|
||||
one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and
|
||||
multiplied by four. A step is taken while it lowers R-free or R-work by more than 0.001 and its R-free
|
||||
stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model,
|
||||
whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the
|
||||
trajectory ends where it began.
|
||||
multiplied by four. The step's weights are chosen among its trials on the free set; the step is taken while
|
||||
that trial lowers the work-set target (or, at other weights, displaces where the last step ended), and the
|
||||
trajectory ends where the last step taken ended.
|
||||
3. **The population** (off by default, `RefinementSettings::population`: with the minimiser of §15.6 its members
|
||||
took nothing off R-free over 19 refinements that the second round did not), from where the trajectory ended:
|
||||
up to twelve members (at most as many as the trial slots, so that they run in one wave; none below two) refined
|
||||
@@ -380,8 +403,8 @@ are:
|
||||
members see those around it. Their **crossover** is made on one 2mFo-DFc map averaged over the members with
|
||||
the free reflections left out: each residue is taken from the member whose atoms sit in the most density,
|
||||
the junctions are mended by a macro-cycle at the ladder's tightest rung, and the child is refined by one
|
||||
trial at the trajectory's weights. The child displaces the state by the rule every trial is held to; the
|
||||
members themselves are never chosen among on the free set, whose best of twelve would be the best of twelve. On
|
||||
trial at the trajectory's weights. The child is taken where it lowers the work-set target, as a step is; the
|
||||
free set never sees the members. On
|
||||
13 predictions placed by MR the child was kept on four (R-free 0.005-0.011 lower) and on one of 22 refined
|
||||
models (0.009 lower, the rest unchanged); the gain, where there is one, comes mostly in the rounds that
|
||||
follow from the child.
|
||||
@@ -402,14 +425,15 @@ are:
|
||||
sweep would otherwise take more than $4 \times 10^8$ evaluations (states times reflections, about 3 s on one
|
||||
card). An FFT gradient cannot carry a side chain 1.5 Å at 1.7 Å
|
||||
resolution; the field, a search, can, and it is scored by the likelihood rather than by a map, which on an
|
||||
incomplete model rewards whatever fills unmodelled density. The free reflections are never seen by the field:
|
||||
one step at the trajectory's weights is then taken from it and judged against the same step from the model as
|
||||
it was, kept only where both its R-free and its $-\ln L$ of the free reflections are lower and the geometry
|
||||
within bounds, undone otherwise (a model whose side chains the data already hold moves nothing and costs the
|
||||
incomplete model rewards whatever fills unmodelled density. The free reflections are never seen by the field
|
||||
nor by its test: one step at the trajectory's weights is then taken from it and judged against the same step
|
||||
from the model as it was, kept only where it lowers the work-set target and the geometry stays within bounds,
|
||||
undone otherwise (a model whose side chains the data already hold moves nothing and costs the
|
||||
sweep alone). The log names each side chain moved with its chi angles before and after and the energy the move
|
||||
took off (`REFINE_SIDE_CHAIN_FIELD_MOVED`, `REFINE_SIDE_CHAIN_FIELD_PRUNED`). Twinned data and a model with
|
||||
anisotropic atoms are refined without the field.
|
||||
5. **The final ladder**: the trajectory's weights times $\{1, 2\}$, kept where one displaces it (the weight halved
|
||||
5. **The final ladder**: the trajectory's weights times $\{1, 2\}$, chosen on the free set, the choice taken where it
|
||||
advances from where the trajectory ended as a step does (the weight halved
|
||||
was never kept on 55 refinements, and taking it away changed no choice).
|
||||
6. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U
|
||||
held), each trial as long as a step on the positions where the weights ended, held - only the displacement
|
||||
@@ -425,8 +449,7 @@ are:
|
||||
reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving
|
||||
out the U weight $\times 4$ changed a choice taken. A
|
||||
trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds
|
||||
and both its R-free (beyond the tie) and the $-\ln L$ of the free reflections are lower: the free set tests the
|
||||
parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012).
|
||||
and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). Between a model with more displacement parameters (TLS over a B per atom, a U per atom over either) and one with fewer, Hamilton's R-ratio test decides first, as Merritt (2012) and PDB-REDO apply it: the larger passes where its R-work is below the smaller's by more than the ratio $\sqrt{b/(n-m)\,F_{b,n-m,0.05} + 1}$, $n$ the working reflections, $m$ its displacement parameters (the positions are held) and $b$ the extra ones (Hamilton 1965; the F point by Paulson's approximation). The restraints are credited as $wR$ observations, $w$ unknown: each model's is scanned from 0 to the bound at which its restraints take every parameter they restrain off its count (Merritt's $N_\mathrm{atoms}/R$ for a B per atom, $6N_\mathrm{atoms}/R$ for a U per atom), six values each, independently. The larger model is taken where it passes more than 95% of the 36 tests, the smaller where it passes fewer than 30%, and in between the free set decides as everywhere else (PDB-REDO falls back on Z(R-free) and the R-free - R-work gap at fixed cut-offs instead; the paired rule is the one every other free-set choice here uses). A U per atom is tried only above 18 working reflections per refinable atom, PDB-REDO's bound.
|
||||
The refined U are written as ANISOU and the TLS groups as `_pdbx_refine_tls*` and `REMARK 3` (`REFINE_ADP_MODEL`,
|
||||
`REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its
|
||||
solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no
|
||||
@@ -437,14 +460,16 @@ are:
|
||||
B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT,
|
||||
the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$),
|
||||
the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the
|
||||
refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers
|
||||
R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the
|
||||
refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken where it advances as a
|
||||
step does - on the work-set target where its weights are those it started from, on the free set where it chose
|
||||
others - and undone otherwise; where its ladder and trajectory leave the model where it was, the
|
||||
hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder
|
||||
and one step. No round runs where the refinement does not displace the model as given.
|
||||
7. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the
|
||||
rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as
|
||||
it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never
|
||||
worse on the free set than what went in. This holds only where the free set is free to the model as given: a model
|
||||
7. **The model as given** is kept over the refinement where it displaces the refinement on the free set (its geometry
|
||||
within bounds and the free set preferring it) and the refinement is not better on both R-work and R-free (the
|
||||
work-set target cannot choose here, the refinement having minimised it from the model as given): the model is
|
||||
written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes
|
||||
out is never worse on both free measures than what went in. This holds only where the free set is free to the model as given: a model
|
||||
placed as given on a free set drawn by the run (in `--mode refine`, a file without free flags) has
|
||||
most likely been refined against those reflections, and its R-free as given does not compete.
|
||||
|
||||
@@ -453,7 +478,7 @@ without them, and then with them at $\sigma = 0.1$ Å at the B factors' weight t
|
||||
X-ray weight a rung up as often as not, the B factors' choice it never changed on 39 refinements), and every step of
|
||||
the trajectory also tries them changed - switched on where they are off, and in turn tightened ($\sigma$ halved) and
|
||||
loosened ($\sigma$ doubled, off beyond 0.4 Å) where they are on. Listed after their twin without, the restraints are
|
||||
kept only where they lower R-free by more than the tie. A model without copies runs exactly the trials it ran before.
|
||||
kept only where the free set prefers them. A model without copies runs exactly the trials it ran before.
|
||||
|
||||
$w$ never exceeds $w_0/8$. The trials of one set run side by side - on one GPU, on several, or on the CPU's
|
||||
threads - each from the same start and taking nothing from the one before it, so the result does not depend
|
||||
@@ -576,8 +601,8 @@ and removed and waters and ions relabelled, as phenix.refine builds waters withi
|
||||
trajectory (§15.7) ends, a round reads the maps and the audit of that model and changes its solvent; then the solvent
|
||||
alone - its waters and single-atom ions, the rest held - is refined at the weights the trajectory ended at (its B
|
||||
too, at its balanced weight where the trajectory held B, and 10 iterations a macro-cycle), and the round is kept
|
||||
where that lowers R-free; where it does not but the round as built does (refining the solvent alone can lose what its
|
||||
sites gained), it
|
||||
where that lowers the work-set target at those weights (§15.7); where it does not but the round as built does
|
||||
(refining the solvent alone can lose what its sites gained), it
|
||||
is kept as built; otherwise it is undone. The rest of the model goes on with its B factors held or refined as the
|
||||
trajectory had them. The rounds stop at one that changes nothing, at one undone, or after two (a third gains
|
||||
nothing measurable); the final ladder of §15.7 then
|
||||
@@ -786,6 +811,7 @@ What the refinement does not do yet - each a known gap, not a design choice:
|
||||
- A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long & G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15.4).
|
||||
- M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (the topology, the riding hydrogens and the blob search, §15.4, §15.9).
|
||||
- V. Schomaker & K. N. Trueblood, *Acta Cryst.* **B24** (1968), 63-76, and M. D. Winn, M. N. Isupov & G. N. Murshudov, *Acta Cryst.* **D57** (2001), 122-133 (TLS, and the B column total or residual to it, §15.1, §15.5); J. Painter & E. A. Merritt, *Acta Cryst.* **D62** (2006), 439-450 (TLSMD's segments, §15.7).
|
||||
- W. C. Hamilton, "Significance tests on the crystallographic R factor", *Acta Cryst.* **18** (1965), 502-510, as E. A. Merritt, "To B or not to B: a question of resolution?", *Acta Cryst.* **D68** (2012), 468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320) applies it to the displacement model, and E. Paulson, "An approximate normalization of the analysis of variance distribution", *Ann. Math. Statist.* **13** (1942), 233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608) (the F point) (§15.7).
|
||||
- F. L. Hirshfeld, *Acta Cryst.* **A32** (1976), 239-244, A. Thorn, B. Dittrich & G. M. Sheldrick, *Acta Cryst.* **A68** (2012), 448-451, and G. M. Sheldrick, *Acta Cryst.* **C71** (2015), 3-8 (the rigid-bond, similarity and isotropy restraints of anisotropic atoms, §15.5); E. A. Merritt, *Acta Cryst.* **D68** (2012), 468-477 (the displacement model decided by test, §15.7).
|
||||
- A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5).
|
||||
- M. M. Harding, "Geometry of metal-ligand interactions in proteins", *Acta Cryst.* **D57** (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", *Acta Cryst.* **D62** (2006), 678-682; F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5).
|
||||
@@ -800,6 +826,8 @@ What the refinement does not do yet - each a known gap, not a design choice:
|
||||
- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
|
||||
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7).
|
||||
- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free).
|
||||
- I. J. Tickle, R. A. Laskowski & D. S. Moss, "R<sub>free</sub> and the R<sub>free</sub> ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement", *Acta Cryst.* **D56** (2000), 442-450 [doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868) (the standard error of one R-free, against which the paired error of §15.7 is set).
|
||||
- R. P. Joosten, F. Long, G. N. Murshudov & A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization", *IUCrJ* **1** (2014), 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324), and R. P. Joosten, K. Joosten, G. N. Murshudov & A. Perrakis, "PDB_REDO: constructive validation, more than just looking for errors", *Acta Cryst.* **D68** (2012), 484-496 [doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515) (choices taken only where more than one free measure agrees, §15.7).
|
||||
- G. J. Kleywegt, "Use of non-crystallographic symmetry in protein structure refinement", *Acta Cryst.* **D52** (1996), 842-857; I. Usón, E. Pohl, T. R. Schneider, Z. Dauter, A. Schmidt, H.-J. Fritz & G. M. Sheldrick, "1.7 Å structure of the stabilized REIv mutant T39K. Application of local NCS restraints", *Acta Cryst.* **D55** (1999), 1158-1167; R. A. Nicholls, F. Long & G. N. Murshudov, "Low-resolution refinement tools in REFMAC5", *Acta Cryst.* **D68** (2012), 404-417; J. J. Headd et al., "Flexible torsion-angle noncrystallographic symmetry restraints for improved macromolecular structure refinement", *Acta Cryst.* **D70** (2014), 1346-1356 (local NCS restraints with a robust loss, decided by R-free, and Murshudov et al. 2011 above for the copies found by alignment, §15.5, §15.7).
|
||||
- P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9).
|
||||
- N. Naleem, A. Henning-Knechtel, S. Kirmizialtin & P. Auffinger, "Cat_Wiz: a stereochemistry-guided toolkit for locating, diagnosing, and annotating Mg2+ ions in RNA structures", *Nucleic Acids Res.* **54** (2026), gkag120; F. Leonarski, L. D'Ascenzo & P. Auffinger, *Nucleic Acids Res.* **45** (2017), 987-1004 and *RNA* **25** (2019), 173-192; F. Leonarski, A. Henning-Knechtel, S. Kirmizialtin, E. Ennifar & P. Auffinger, *Nucleic Acids Res.* **53** (2025), gkae1148 (the coordination bands, distances and exclusion zone of the solvent audit, §15.10).
|
||||
|
||||
@@ -287,12 +287,14 @@ element and the number of such records of the ion (Mg-O 2.07 A, Na-O 2.42 A, K-O
|
||||
usual coordination), where the model's distance is within 0.33 A of that (Mg-O up to 2.4 A); a record at an
|
||||
outer-sphere distance, a contact without a record, or one to a symmetry mate, is not restrained as a bond. The B factors of bonded atoms and of
|
||||
atoms in contact (waters included) are restrained to each other in proportion to their size. The weights of
|
||||
the X-ray term against the geometry and against that B restraint are chosen by R-free among short
|
||||
refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees: three weights from the placed
|
||||
the X-ray term against the geometry and against that B restraint are chosen on the free set among short
|
||||
refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees - a candidate taken only where its free
|
||||
-ln L is lower and its R-free not higher than the incumbent's by more than the standard error of their paired
|
||||
difference over the free reflections: three weights from the placed
|
||||
model, each with the B factors held and refined; then a trajectory of further short refinements from the
|
||||
best, each trying the weights as they are and changed, taken while R-free or R-work falls beyond 0.001 and
|
||||
R-free stays within 0.001 of its best; then a short final ladder, kept only where it lowers R-free beyond
|
||||
0.001. The free set is the merge's own, so the refined R-free judges the refinement against reflections it
|
||||
chosen one, each trying the weights as they are and changed, taken while it lowers the work-set target (the
|
||||
refined target at the chosen weights); then a short final ladder. The free set decides the weights and the
|
||||
displacement model only; every other step is taken on the work-set target. The free set is the merge's own, so the refined R-free judges the refinement against reflections it
|
||||
never fitted - but where the model was deposited, those reflections were most likely its depositor's working
|
||||
reflections, and a refinement against new data then moves R-free up from the unrefined value rather than
|
||||
down. `R_WORK`/`R_FREE` of the model as given are never replaced. On a GPU a refinement takes seconds; the CPU
|
||||
|
||||
@@ -91,8 +91,62 @@ constexpr double REPULSION_MARGIN = 0.5; // A: pairs this much beyond their co
|
||||
constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0;
|
||||
// The minimiser's iterations per macro-cycle where the solvent build refines the solvent alone.
|
||||
constexpr int SOLVENT_ITERATIONS = 10;
|
||||
// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken.
|
||||
constexpr double R_FREE_TIE = 0.001;
|
||||
// The free set decides the weights and the displacement model alone, and only where both of its measures agree: a
|
||||
// candidate is preferred to the incumbent where its -ln L of the free reflections is lower and its R-free not higher
|
||||
// than the incumbent's by more than the standard error of the difference. The two are scored on the same free
|
||||
// reflections, so the difference is paired: R-free is a sum of one term per free twin group (FreeRTerms), and the
|
||||
// standard error is that of the sum of the groups' differences.
|
||||
// As PDB-REDO decides on more than one free measure, Joosten, Long, Murshudov & Perrakis (2014) IUCrJ 1, 213-220
|
||||
bool FreeSetPrefers(const std::vector<float> &r_terms, double nll_free, const std::vector<float> &r_terms_b,
|
||||
double nll_free_b) {
|
||||
const size_t n = r_terms.size();
|
||||
double sum = 0, sum2 = 0;
|
||||
for (size_t k = 0; k < n; k++) {
|
||||
const double d = r_terms[k] - r_terms_b[k];
|
||||
sum += d;
|
||||
sum2 += d * d;
|
||||
}
|
||||
const double se = n > 1 ? std::sqrt(std::max(sum2 - sum * sum / n, 0.0) * n / (n - 1)) : 0.0;
|
||||
return nll_free < nll_free_b && sum < se;
|
||||
}
|
||||
// The upper alpha point of the F distribution with d1 and d2 degrees of freedom, by Paulson's normal approximation of
|
||||
// F^(1/3) (z the normal deviate of alpha) - accurate to a few percent at the counts a displacement model has.
|
||||
// Following Paulson (1942) Ann. Math. Statist. 13, 233-235
|
||||
double FQuantile(double d1, double d2, double z) {
|
||||
const double u = 2.0 / (9.0 * d1), v = 2.0 / (9.0 * d2), a = 1.0 - v, c = 1.0 - u;
|
||||
const double q = a * a - z * z * v;
|
||||
const double x = (a * c + std::sqrt(std::max(a * a * c * c - q * (c * c - z * z * u), 0.0))) / q;
|
||||
return x * x * x;
|
||||
}
|
||||
|
||||
// Of two displacement models, `large` with more parameters than `small`: the fraction of Hamilton's R-ratio tests it
|
||||
// passes over the range of weights its restraints may be credited at - R_small / R_large above
|
||||
// sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the working reflections, m the larger model's displacement parameters
|
||||
// (the trials hold the positions), b the extra ones. Each model's restraints are credited as w R observations, w from
|
||||
// 0 up to the bound at which they take every parameter they restrain off its count - Merritt's N_atoms / R for a B
|
||||
// per atom (as his 3 N_atoms / R for the positions), 6 N_atoms / R for a U per atom; the two weights are scanned
|
||||
// independently, six values each.
|
||||
// Following Hamilton (1965) Acta Cryst. 18, 502-510, as Merritt (2012) Acta Cryst. D68, 468-477 and PDB-REDO
|
||||
// (Joosten et al. 2012 Acta Cryst. D68, 484-496) apply it to the displacement model
|
||||
double HamiltonPassFraction(const RefinementWeightTrial &large, const RefinementWeightTrial &small, int n_work) {
|
||||
int pass = 0, tests = 0;
|
||||
for (int i = 0; i <= 5; i++)
|
||||
for (int j = 0; j <= 5; j++, tests++) {
|
||||
const double m_small = small.adp_parameters - small.adp_restrained * i / 5.0;
|
||||
const double m_large = large.adp_parameters - large.adp_restrained * j / 5.0;
|
||||
const double b = m_large - m_small, dof = n_work - m_large;
|
||||
pass += b <= 0 || (dof > 0 && small.r_work / large.r_work > std::sqrt(b / dof * FQuantile(b, dof, 1.6449) + 1.0));
|
||||
}
|
||||
return static_cast<double>(pass) / tests;
|
||||
}
|
||||
// The displacement model: of the larger and the smaller, the larger where it passes more than this fraction of the
|
||||
// tests (HamiltonPassFraction), the smaller where it passes fewer than the other, and the free set's choice between
|
||||
// (as PDB-REDO; Joosten et al. 2012).
|
||||
constexpr double HAMILTON_LARGER = 0.95, HAMILTON_SMALLER = 0.30;
|
||||
// A U per atom is tried only above this many working reflections per refinable atom (as PDB-REDO; Joosten et al. 2012).
|
||||
constexpr double U_MIN_REFLECTIONS_PER_ATOM = 18;
|
||||
// Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative.
|
||||
constexpr double TARGET_TOLERANCE = 1e-4;
|
||||
// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each
|
||||
// after it from the model the one before ended with.
|
||||
// The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this,
|
||||
@@ -1564,9 +1618,10 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
|
||||
if (body.size() >= 50)
|
||||
bodies.push_back(body);
|
||||
}
|
||||
// Each step is kept only where it lowers R-free beyond the tie: a deposited model was refined atom by atom
|
||||
// already, and moving its chains or shifting its B factors residue by residue then only makes it worse.
|
||||
RFactors before = ctx.Refit();
|
||||
// Each step is kept only where it lowers the -ln L of the working reflections, the target it minimised (the free
|
||||
// set decides the weights and the displacement model alone).
|
||||
ctx.Refit();
|
||||
double before = ctx.Score().nll_work;
|
||||
if (!bodies.empty()) {
|
||||
const std::vector<gemmi::Position> pos0 = ctx.Positions();
|
||||
std::vector<double> q(6 * bodies.size(), 0.0);
|
||||
@@ -1577,13 +1632,15 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
|
||||
const CancelCheck cancel_check(ctx);
|
||||
ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary);
|
||||
rb->Place(q.data());
|
||||
const RFactors after = ctx.Refit();
|
||||
if (after.r_free < before.r_free - R_FREE_TIE) {
|
||||
ctx.Refit();
|
||||
const double after = ctx.Score().nll_work;
|
||||
if (after < before) {
|
||||
result.rigid_bodies = static_cast<int>(bodies.size());
|
||||
before = after;
|
||||
} else {
|
||||
ctx.SetPositions(pos0);
|
||||
before = ctx.Refit();
|
||||
ctx.Refit();
|
||||
before = ctx.Score().nll_work;
|
||||
}
|
||||
}
|
||||
if (group_b) {
|
||||
@@ -1596,7 +1653,8 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
|
||||
const CancelCheck cancel_check(ctx);
|
||||
ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary);
|
||||
gb->Place(q.data());
|
||||
result.group_b = ctx.Refit().r_free < before.r_free - R_FREE_TIE;
|
||||
ctx.Refit();
|
||||
result.group_b = ctx.Score().nll_work < before;
|
||||
if (!result.group_b)
|
||||
ctx.SetB(b0);
|
||||
}
|
||||
@@ -1897,6 +1955,39 @@ std::string DomainText(const RefinementReflections &refl, const XrayRowModel &ro
|
||||
// As REFMAC5 drops twin domains of small fraction, Murshudov et al. (2011) Acta Cryst. D67, 355-367
|
||||
constexpr double TWIN_DOMAIN_MIN = 0.05;
|
||||
|
||||
// Each observed reflection's twin group, named by its lowest observed row (itself where there is no twin law).
|
||||
std::vector<int> TwinGroups(const RefinementReflections &refl) {
|
||||
const size_t n = refl.Size();
|
||||
std::vector<int> group(n);
|
||||
for (size_t m = 0; m < n; m++) {
|
||||
group[m] = static_cast<int>(m);
|
||||
for (size_t d = 0; d < refl.Laws(); d++)
|
||||
if (refl.mates[d][m] < static_cast<int>(n))
|
||||
group[m] = std::min(group[m], refl.mates[d][m]);
|
||||
}
|
||||
return group;
|
||||
}
|
||||
|
||||
// R-free of the model as the last Fcalc left it, as one term per free twin group, in the order of the groups (the
|
||||
// members of a group share their F_calc, so their terms are summed into one, as TwinNull sums them): the terms
|
||||
// FreeSetPrefers pairs.
|
||||
std::vector<float> FreeRTerms(const Context &ctx) {
|
||||
const std::vector<std::complex<float>> fmodel = TwinnedModel(ctx.refl, ctx.rows, ctx.fcalc);
|
||||
const std::vector<int> group = TwinGroups(ctx.refl);
|
||||
double fo = 0;
|
||||
std::vector<double> sum(ctx.refl.Size(), 0.0);
|
||||
for (size_t m = 0; m < ctx.refl.Size(); m++)
|
||||
if (ctx.refl.free[m]) {
|
||||
fo += ctx.refl.fo[m];
|
||||
sum[group[m]] += std::fabs(ctx.refl.fo[m] - std::abs(fmodel[m]));
|
||||
}
|
||||
std::vector<float> terms;
|
||||
for (size_t m = 0; m < ctx.refl.Size(); m++)
|
||||
if (ctx.refl.free[m] && group[m] == static_cast<int>(m))
|
||||
terms.push_back(static_cast<float>(sum[m] / fo));
|
||||
return terms;
|
||||
}
|
||||
|
||||
// The null of a twinned refinement, for the model as the last Refit() left it: its R-free at the same fraction with
|
||||
// every reflection's mate replaced by an unrelated observed one of the same resolution and the same set (free with
|
||||
// free, work with work, as a law's twin pairs are). The reflections of each set are put in order of 1/d^2 with the two
|
||||
@@ -1920,14 +2011,7 @@ constexpr double TWIN_STOP_Z = 0.5;
|
||||
void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) {
|
||||
RefinementReflections null = ctx.refl;
|
||||
const size_t n = null.Size();
|
||||
// each observed reflection's twin group, named by its lowest observed row
|
||||
std::vector<int> group(n);
|
||||
for (size_t m = 0; m < n; m++) {
|
||||
group[m] = static_cast<int>(m);
|
||||
for (size_t d = 0; d < ctx.refl.Laws(); d++)
|
||||
if (ctx.refl.mates[d][m] < static_cast<int>(n))
|
||||
group[m] = std::min(group[m], ctx.refl.mates[d][m]);
|
||||
}
|
||||
const std::vector<int> group = TwinGroups(ctx.refl);
|
||||
for (size_t d = first; d < null.Laws(); d++)
|
||||
std::iota(null.mates[d].begin(), null.mates[d].end(), 0);
|
||||
for (const bool free : {false, true}) {
|
||||
@@ -2246,6 +2330,26 @@ struct TrialOutcome {
|
||||
int evaluations = 0;
|
||||
};
|
||||
|
||||
// The work-set target's terms of the model as it stands (RefinementWeightTrial), at the last Refit()'s fit: what every
|
||||
// decision but the weights' and the displacement model's is made on (WorkTarget).
|
||||
void MeasureWorkTarget(Context &ctx, const RefinementTopology &topo, const NcsCopies *ncs, double ncs_sigma,
|
||||
RefinementWeightTrial &t) {
|
||||
const RefinementScore sc = ctx.Score();
|
||||
t.nll_work = sc.nll_work;
|
||||
t.nll_free = sc.nll_free;
|
||||
t.free_r_terms = FreeRTerms(ctx);
|
||||
const std::vector<std::array<double, 4>> xyzb = ctx.XyzB();
|
||||
t.e_geometry = EvaluateRestraints(RestraintsNow(ctx, topo, ncs, ncs_sigma), xyzb, nullptr);
|
||||
t.e_b = EvaluateRestraints(BRestraintsNow(ctx, topo), xyzb, nullptr);
|
||||
}
|
||||
|
||||
// The work-set target of a trial at the X-ray weights w (positions) and adp (B, 0: held), its NCS restraints at its own
|
||||
// sigma: what a trial at those weights minimises, the X-ray term of the positions and the geometry, and that of the B
|
||||
// factors and the B similarity where they are refined.
|
||||
double WorkTarget(const RefinementWeightTrial &t, double w, double adp) {
|
||||
return w * t.nll_work + t.e_geometry + (adp > 0 ? adp * t.nll_work + t.e_b : 0.0);
|
||||
}
|
||||
|
||||
// The X-ray target's gradient on the positions, its squared norm over `refinable`, for the atoms as they are.
|
||||
double XrayGradientNorm(Context &ctx, const std::vector<int> &refinable) {
|
||||
std::vector<std::array<double, 4>> g;
|
||||
@@ -2471,16 +2575,25 @@ TrialOutcome RunTrial(TrialSlot slot, XyzCalibrations &calibrations, const std::
|
||||
t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB());
|
||||
if (hyp != nullptr)
|
||||
t.adp = hyp->name;
|
||||
t.nll_free = ctx.Score().nll_free;
|
||||
out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, "
|
||||
"angles {:.2f} deg{}{}",
|
||||
// the displacement parameters: a B per refinable atom, the TLS bodies' 20 each beside it, or a U per atom
|
||||
int atoms_b = 0;
|
||||
for (int i : refinable)
|
||||
atoms_b += !ctx.atoms[i]->is_hydrogen();
|
||||
t.adp_parameters = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast<int>(u_atoms.size())
|
||||
: kind == AdpHypothesis::Kind::Tls ? atoms_b + 20 * static_cast<int>(hyp->bodies.size())
|
||||
: atoms_b;
|
||||
// of them the restrained ones: the B or the U of every atom (not the TLS bodies')
|
||||
t.adp_restrained = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast<int>(u_atoms.size()) : atoms_b;
|
||||
MeasureWorkTarget(ctx, topo, ncs, ncs_sigma, t);
|
||||
out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, target {:.6g}, rmsd bonds "
|
||||
"{:.4f} A, angles {:.2f} deg{}{}",
|
||||
std::string(tw.blur.empty() ? "" : "blurred, ") + (hyp != nullptr ? hyp->name + ", " : ""), w,
|
||||
kind == AdpHypothesis::Kind::Anisotropic ? "U" : kind == AdpHypothesis::Kind::Tls ? "residual B" : "B",
|
||||
adp_w > 0 ? fmt::format("refined at weight {:.3g}", adp_w) : std::string("held"),
|
||||
ncs == nullptr || ncs->Empty() ? std::string()
|
||||
: ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd)
|
||||
: fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd),
|
||||
rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free),
|
||||
rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), WorkTarget(t, w, adp_w),
|
||||
t.geometry.rmsd_bonds, t.geometry.rmsd_angles,
|
||||
t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal)
|
||||
: std::string(),
|
||||
@@ -2859,8 +2972,8 @@ double BalancedUWeight(Context &ctx, const URestraints &restraints, const std::v
|
||||
|
||||
// The displacements of the model as given, read where it has TLS groups whose atoms bring no U of their own: their U
|
||||
// from the groups, its B column taken as the total B (the U's equivalent isotropic B, as phenix.refine writes it) or as
|
||||
// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - the one of the three whose
|
||||
// R-free is lowest, beyond the tie in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens
|
||||
// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - a displacement model, chosen on
|
||||
// the free set (FreeSetPrefers), in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens
|
||||
// (which ride on their parents' B). Returns the R-factors of the reading kept.
|
||||
// Following Winn, Isupov & Murshudov (2001) Acta Cryst. D57, 122-133 (the B column total or residual to the TLS)
|
||||
RFactors ReadInputDisplacements(Context &ctx, const std::vector<TlsBody> &tls, Logger &logger, RefinementResult &result) {
|
||||
@@ -2906,6 +3019,8 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector<TlsBody> &tls, L
|
||||
readings.push_back({result.adp_input == "ANISOU" ? "ANISOU" : "isotropic", b0, aniso0});
|
||||
int best = -1;
|
||||
std::vector<RFactors> rf;
|
||||
std::vector<double> nll_free;
|
||||
std::vector<std::vector<float>> r_terms;
|
||||
for (size_t k = 0; k < readings.size(); k++) {
|
||||
ctx.SetB(readings[k].b);
|
||||
ctx.SetAniso(readings[k].aniso);
|
||||
@@ -2914,9 +3029,11 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector<TlsBody> &tls, L
|
||||
ctx.atoms[i]->b_iso = ctx.atoms[ctx.rider_parent[i]]->b_iso;
|
||||
ctx.ForgetStarts();
|
||||
rf.push_back(ctx.Refit());
|
||||
logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}", tls.size(),
|
||||
tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free);
|
||||
if (best < 0 || rf[k].r_free < rf[best].r_free - R_FREE_TIE)
|
||||
nll_free.push_back(ctx.Score().nll_free);
|
||||
r_terms.push_back(FreeRTerms(ctx));
|
||||
logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}, -LL free "
|
||||
"{:.1f}", tls.size(), tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free, nll_free[k]);
|
||||
if (best < 0 || FreeSetPrefers(r_terms[k], nll_free[k], r_terms[best], nll_free[best]))
|
||||
best = static_cast<int>(k);
|
||||
}
|
||||
ctx.SetB(readings[best].b);
|
||||
@@ -3161,6 +3278,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
result.r_work_start = r.r_work;
|
||||
result.r_free_start = r.r_free;
|
||||
result.score_start = ctx.Score();
|
||||
const std::vector<float> start_r_terms = FreeRTerms(ctx);
|
||||
logger.Info("Refinement{}: {} atoms ({} riding hydrogens), {} reflections to {:.2f} A ({} free), on the {}; as "
|
||||
"given R-work {:.4f}, R-free {:.4f}; k_sol {:.3f}, b_sol {:.1f}, overall B {:.1f}, Wilson B {:.0f}, B "
|
||||
"ceiling {:.0f}, mask grid {:.2f} A",
|
||||
@@ -3178,6 +3296,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
given.trial.r_work = r.r_work;
|
||||
given.trial.r_free = r.r_free;
|
||||
given.trial.nll_free = result.score_start.nll_free;
|
||||
given.trial.free_r_terms = start_r_terms;
|
||||
given.pos = ctx.Positions();
|
||||
given.b = ctx.B();
|
||||
given.aniso = ctx.aniso;
|
||||
@@ -3217,7 +3336,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
ctx.st.meta.refinement[0].tls_groups = TlsBodiesToGroups(ctx.Model(), ctx.tls);
|
||||
for (size_t k = 1; k < ctx.st.meta.refinement.size(); k++)
|
||||
ctx.st.meta.refinement[k].tls_groups.clear();
|
||||
} else if (result.adp_model == "anisotropic")
|
||||
} else if (result.adp_model.starts_with("anisotropic"))
|
||||
for (gemmi::RefinementInfo &ref : ctx.st.meta.refinement)
|
||||
ref.tls_groups.clear();
|
||||
if (const std::vector<gemmi::TlsGroup> *groups = ctx.st.meta.get_tls_groups())
|
||||
@@ -3426,10 +3545,24 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
o = TrialOutcome();
|
||||
};
|
||||
|
||||
// A trial displaces another only where its geometry is within bounds and its R-free lower by more than the
|
||||
// tie.
|
||||
// A trial at other weights (or the model as given) displaces another only where its geometry is within
|
||||
// bounds and the free set prefers it (FreeSetPrefers).
|
||||
auto displaces = [](const RefinementWeightTrial &t, const RefinementWeightTrial &b) {
|
||||
return t.within_bounds && (!b.within_bounds || t.r_free < b.r_free - R_FREE_TIE);
|
||||
return t.within_bounds &&
|
||||
(!b.within_bounds || FreeSetPrefers(t.free_r_terms, t.nll_free, b.free_r_terms, b.nll_free));
|
||||
};
|
||||
// A trial taken on from `s`: at the weights of s, where the work-set target (WorkTarget) is lower than that of s
|
||||
// by more than TARGET_TOLERANCE, and the geometry within bounds where that of s is. A trial at other weights
|
||||
// (X-ray weights or NCS restraints) is not compared on a target that changed with them - the positions and the
|
||||
// B factors minimise targets of their own, whose sum a change of weight can raise as it lowers each: the
|
||||
// weights are the free set's to decide (displaces).
|
||||
auto advances = [&displaces](const RefinementWeightTrial &t, const RefinementWeightTrial &s) {
|
||||
if (t.weight != s.weight || t.adp_weight != s.adp_weight || t.ncs_sigma != s.ncs_sigma)
|
||||
return displaces(t, s);
|
||||
if (t.within_bounds < s.within_bounds)
|
||||
return false;
|
||||
const double before = WorkTarget(s, s.weight, s.adp_weight);
|
||||
return WorkTarget(t, s.weight, s.adp_weight) < before - TARGET_TOLERANCE * std::abs(before);
|
||||
};
|
||||
// Of a set of trials, the one kept: they come in order of trust, the first kept unless a later one displaces
|
||||
// it. -1: none ran.
|
||||
@@ -3459,9 +3592,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// not change (on 39 refinements of 21 structures with copies it never did; the X-ray weight it does), and
|
||||
// at the rung they chose and the next looser one (on 198 refinements of the battery with copies, NCS was
|
||||
// never kept at any other).
|
||||
// The B factors are refined first at the middle rung only, and at the others where that comes within two
|
||||
// ties of holding them there: on 364 refinements of the battery B held was kept by 281, and the rule
|
||||
// changed the rung kept on 4 (all within 0.006 in R-free). Their X-ray weight is tried at the balanced
|
||||
// The B factors are refined first at the middle rung only, and at the others unless holding them there
|
||||
// displaces refining them (on 364 refinements of the battery B held was kept by 281). Their X-ray weight is tried at the balanced
|
||||
// one and at a quarter and a sixteenth of it: a model whose B factors arrive far from where they refine
|
||||
// (a prediction's pLDDT) balances a weight that is not the one its B wants, and the lower rungs took
|
||||
// 0.002-0.010 off R-free on five such starts, nothing off eight deposited models.
|
||||
@@ -3500,8 +3632,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
if (ladder[i].w == mid && ladder[i].ncs == 0)
|
||||
(ladder[i].adp == 0 ? held : refined) = &outcomes[i];
|
||||
const bool refine_b = held == nullptr || refined == nullptr ||
|
||||
(refined->ran && refined->trial.within_bounds &&
|
||||
(!held->trial.within_bounds || refined->trial.r_free < held->trial.r_free + 2 * R_FREE_TIE));
|
||||
(refined->ran && refined->trial.within_bounds && !displaces(held->trial, refined->trial));
|
||||
auto rest_b = [&](const TrialWeights &l) { return l.ncs == 0 && !first(l); };
|
||||
if (refine_b && !all_ahead)
|
||||
run_ladder(rest_b);
|
||||
@@ -3547,37 +3678,29 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
}
|
||||
|
||||
// 2. The trajectory: steps of a few macro-cycles from where the last one ended, each trying the weights as they
|
||||
// are and, in turn, the positions' X-ray weight halved and doubled (halved only in the first step, from the
|
||||
// ladder's coarse rung: on 364 refinements of the battery no later step kept it, of 91 that tried it) and
|
||||
// are and, in turn, the positions' X-ray weight halved and doubled (the ladder's rungs are a factor of eight
|
||||
// apart, and a trajectory that goes on from a looser weight than the data want can tighten it only so) and
|
||||
// the B factors' divided and multiplied by four; where there are copies, also the NCS restraints changed:
|
||||
// switched on where they are off, and in turn tightened and loosened (to off beyond NCS_SIGMA_MAX) where
|
||||
// they are on.
|
||||
// A step is taken while it lowers R-free or R-work by more than the tie and its R-free stays within the tie
|
||||
// of the best so far; the trajectory ends at its best step (on a deposited model, whose free set it was
|
||||
// fitted to, R-work falls while R-free rises, and it ends where it began).
|
||||
// The trials of the step that went from the best so far: the final ladder goes from there too, and a trial
|
||||
// of the same weights from the same model is the same computation.
|
||||
// The weights of a step are chosen on the free set among its trials (displaces, the weights as they are
|
||||
// first); the step is taken while that trial advances from where the last one ended (advances), and the
|
||||
// trajectory ends where the last step taken ended.
|
||||
// The trials of the step that went from where the trajectory ended: the final ladder goes from there too, and
|
||||
// a trial of the same weights from the same model is the same computation.
|
||||
std::vector<TrialOutcome> from_best;
|
||||
// `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since).
|
||||
// `label`: the steps' name in the log.
|
||||
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector<TrialOutcome> &from_best, const char *label) {
|
||||
TrialOutcome best = state;
|
||||
from_best.clear();
|
||||
bool state_is_best = true;
|
||||
// A step taken on R-work alone is followed by one more at most: on a model refined before, R-work falls
|
||||
// step after step while R-free does not, and the trajectory ends at its best anyway - a model of 490k
|
||||
// atoms spent six such steps (240 s) on a round it then gave back. (On 28 refinements of 22 deposited
|
||||
// models and 6 predictions the rule changed nothing.)
|
||||
int since_best = 0;
|
||||
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled() && since_best <= 1; step++) {
|
||||
for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) {
|
||||
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
|
||||
std::vector<TrialWeights> trials = {{w, adp, ncs_s}};
|
||||
if (step % 2 == 1 && adp > 0) {
|
||||
trials.push_back({w, adp / 4, ncs_s});
|
||||
trials.push_back({w, 4 * adp, ncs_s});
|
||||
} else {
|
||||
if (step == 0)
|
||||
trials.push_back({w / 2, adp, ncs_s});
|
||||
trials.push_back({w / 2, adp, ncs_s});
|
||||
if (2 * w <= w_max)
|
||||
trials.push_back({2 * w, adp, ncs_s});
|
||||
}
|
||||
@@ -3588,27 +3711,18 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
std::vector<TrialOutcome> outcomes = RunTrials(trial_slots, trials, refinable, state.pos, state.b, state.aniso,
|
||||
fresh && step == 0 ? nullptr : &state.fit,
|
||||
settings.macro_cycles, settings, ncs_p);
|
||||
if (state_is_best)
|
||||
from_best = outcomes;
|
||||
from_best = outcomes;
|
||||
const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str());
|
||||
if (k < 0)
|
||||
break;
|
||||
const RefinementWeightTrial &t = outcomes[k].trial;
|
||||
// after a change of the model the first step is taken whatever it gains: it is where the change is
|
||||
// judged from
|
||||
const bool gains = (fresh && step == 0) || displaces(t, state.trial) ||
|
||||
(t.within_bounds && t.r_work < state.trial.r_work - R_FREE_TIE);
|
||||
if (!gains || !((fresh && step == 0) || (t.within_bounds && t.r_free < best.trial.r_free + R_FREE_TIE)))
|
||||
if (!((fresh && step == 0) || advances(outcomes[k].trial, state.trial)))
|
||||
break;
|
||||
state = std::move(outcomes[k]);
|
||||
state_is_best = (fresh && step == 0) || displaces(state.trial, best.trial);
|
||||
since_best = state_is_best ? 0 : since_best + 1;
|
||||
if (state_is_best) {
|
||||
best = state;
|
||||
from_best.clear();
|
||||
}
|
||||
from_best.clear();
|
||||
}
|
||||
return best;
|
||||
return state;
|
||||
};
|
||||
if (!stopped)
|
||||
state = trajectory(std::move(state), false, from_best, "step");
|
||||
@@ -3620,9 +3734,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// map averaged over the members with the free reflections left out (PopulationMap), each residue is taken
|
||||
// from the member whose atoms sit in the most density (following Nwachukwu et al. (2013) Structure 21,
|
||||
// 1923-1930); the child's junctions are mended by a macro-cycle
|
||||
// at the ladder's tightest rung and the whole refined by one trial at the state's weights. The child displaces the state by the rule every trial is held to (geometry within
|
||||
// bounds, R-free lower beyond the tie): the free set decides once, between two candidates - never among
|
||||
// the members, whose best on the free set would be the best of N.
|
||||
// at the ladder's tightest rung and the whole refined by one trial at the state's weights. The child is taken where it
|
||||
// advances on the work-set target, as a step is (the free set does not see the members).
|
||||
// The members are at most as many as the trial slots, one wave: on a large model, where the card holds
|
||||
// few, more members would cost a wave each.
|
||||
const size_t population = std::min<size_t>(std::max(settings.population, 0), trial_slots.size());
|
||||
@@ -3651,7 +3764,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
blur += fmt::format("{}{:.0f}", blur.empty() ? "" : "/", b);
|
||||
logger.Info("Refinement, population: blur {}, kick {:.1f} A: {}", blur.empty() ? "none" : blur,
|
||||
members[k].kick, outcomes[k].line);
|
||||
if (outcomes[k].trial.within_bounds && (best < 0 || outcomes[k].trial.nll_free < outcomes[best].trial.nll_free))
|
||||
if (outcomes[k].trial.within_bounds &&
|
||||
(best < 0 || WorkTarget(outcomes[k].trial, w, adp) < WorkTarget(outcomes[best].trial, w, adp)))
|
||||
best = static_cast<int>(k);
|
||||
}
|
||||
if (best >= 0) {
|
||||
@@ -3686,11 +3800,11 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
child = RunTrials(trial_slots, {{w, adp, ncs_s}}, refinable, child[0].pos, child[0].b, state.aniso,
|
||||
&child[0].fit, settings.macro_cycles, settings, ncs_p);
|
||||
const RefinementWeightTrial &c = child[0].trial;
|
||||
const bool kept = child[0].ran && displaces(c, state.trial);
|
||||
logger.Info("Refinement, population: {} members, the best on the free set R-free {:.4f}, -LL free {:.1f}; "
|
||||
"crossover from {} members: {}; {} (state R-free {:.4f}, -LL free {:.1f})", outcomes.size(),
|
||||
outcomes[best].trial.r_free, outcomes[best].trial.nll_free, used.size(), child[0].line,
|
||||
kept ? "kept" : "not kept", state.trial.r_free, state.trial.nll_free);
|
||||
const bool kept = child[0].ran && advances(c, state.trial);
|
||||
logger.Info("Refinement, population: {} members, the best on the work-set target {:.6g}; crossover from "
|
||||
"{} members: {}; {} (state target {:.6g})", outcomes.size(),
|
||||
WorkTarget(outcomes[best].trial, w, adp), used.size(), child[0].line, kept ? "kept" : "not kept",
|
||||
WorkTarget(state.trial, w, adp));
|
||||
if (kept) {
|
||||
state = std::move(child[0]);
|
||||
from_best.clear();
|
||||
@@ -3701,7 +3815,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// 2b. The solvent build (SolventBuild.h), from where the trajectory ended: waters put in at the difference
|
||||
// map's peaks, taken out where the maps no longer hold them, and waters and ions relabelled, on the maps
|
||||
// and the solvent audit of that model; then the solvent alone refined, the rest held, at the weights the
|
||||
// trajectory ended at. A round is kept where that lowers R-free, and undone otherwise (the build ends
|
||||
// trajectory ended at. A round is kept where that lowers the work-set target at those weights, and undone otherwise (the build ends
|
||||
// there); at most SOLVENT_BUILD_ROUNDS of them. What they kept is then refined with the rest by the final
|
||||
// ladder (the trajectory run again from there gained nothing on twelve sets). Each round thus costs a map,
|
||||
// an audit and a refinement of the solvent, where a refinement of its own cost the whole weight search
|
||||
@@ -3881,6 +3995,16 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
ctx.Restore(state.fit);
|
||||
const RFactors built = ctx.Refit();
|
||||
const Context::Fit built_fit = ctx.Fitted();
|
||||
RefinementWeightTrial built_trial = state.trial;
|
||||
built_trial.r_work = built.r_work;
|
||||
built_trial.r_free = built.r_free;
|
||||
MeasureWorkTarget(ctx, topo, ncs_p, state.trial.ncs_sigma, built_trial);
|
||||
// the round judged on the work-set target at the weights the trajectory ended at
|
||||
const double target_before = WorkTarget(state.trial, state.trial.weight, state.trial.adp_weight);
|
||||
auto lowers = [&](const RefinementWeightTrial &t) {
|
||||
return WorkTarget(t, state.trial.weight, state.trial.adp_weight) <
|
||||
target_before - TARGET_TOLERANCE * std::abs(target_before);
|
||||
};
|
||||
// (B refined, at its balanced weight where the trajectory held B: a new water's starting B is a
|
||||
// guess; fewer iterations, the waters' few parameters converging fast)
|
||||
RefinementSettings solvent_settings = settings;
|
||||
@@ -3893,22 +4017,22 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
ncs_p);
|
||||
logger.Info("Solvent build, round {}: the solvent refined alone: {}", round, local[0].line);
|
||||
TrialOutcome after = std::move(local[0]);
|
||||
if (after.ran && !(after.trial.r_free < state.trial.r_free) && built.r_free < state.trial.r_free) {
|
||||
logger.Info("Solvent build, round {}: kept as built, R-work {:.4f}, R-free {:.4f}", round,
|
||||
built.r_work, built.r_free);
|
||||
if (after.ran && !lowers(after.trial) && lowers(built_trial)) {
|
||||
logger.Info("Solvent build, round {}: kept as built, R-work {:.4f}, R-free {:.4f}, target {:.6g}",
|
||||
round, built.r_work, built.r_free,
|
||||
WorkTarget(built_trial, state.trial.weight, state.trial.adp_weight));
|
||||
after.pos = changed_pos;
|
||||
after.b = changed_b;
|
||||
after.aniso = state.aniso;
|
||||
after.fit = built_fit;
|
||||
after.trial.r_work = built.r_work;
|
||||
after.trial.r_free = built.r_free;
|
||||
after.trial.geometry = state.trial.geometry;
|
||||
after.trial.within_bounds = state.trial.within_bounds;
|
||||
const double adp_weight = after.trial.adp_weight;
|
||||
after.trial = built_trial;
|
||||
after.trial.adp_weight = adp_weight;
|
||||
}
|
||||
if (!after.ran || !(after.trial.r_free < state.trial.r_free)) {
|
||||
logger.Info("Solvent build, round {}: not kept (R-free {:.4f}, before {:.4f})", round,
|
||||
after.trial.r_free, state.trial.r_free);
|
||||
b.stop = fmt::format("round {} raised R-free", round);
|
||||
if (!after.ran || !lowers(after.trial)) {
|
||||
logger.Info("Solvent build, round {}: not kept (target {:.6g}, before {:.6g})", round,
|
||||
WorkTarget(after.trial, state.trial.weight, state.trial.adp_weight), target_before);
|
||||
b.stop = fmt::format("round {} did not lower the target", round);
|
||||
// undone, in every slot
|
||||
for (size_t sl = 0; sl < trial_slots.size(); sl++) {
|
||||
for (auto u = undo[sl].rbegin(); u != undo[sl].rend(); ++u) {
|
||||
@@ -3960,10 +4084,9 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// side chain the field can search put in the rotamer of least energy - the -ln L of the working
|
||||
// reflections at the fit as it stands, plus the refinement's own repulsion term at its X-ray weight -
|
||||
// given the others, or pruned to CB where no rotamer fits; then one step at the trajectory's weights from
|
||||
// the field, judged against the same step from the model as it was (the trajectory's end is the best of
|
||||
// many trials, and a step from it alone regresses from it as often as not): kept where both its R-free
|
||||
// and its -ln L of the free reflections are lower, the geometry within bounds, undone otherwise. The field
|
||||
// never sees the free reflections. A model whose side chains the data already hold moves nothing and
|
||||
// the field, judged against the same step from the model as it was: kept where it advances on the
|
||||
// work-set target from that step (advances), undone otherwise. Neither the field nor its test sees the
|
||||
// free reflections. A model whose side chains the data already hold moves nothing and
|
||||
// costs the sweep alone. The atoms it prunes are left at occupancy 0, as the solvent build's placeholders
|
||||
// are, and held.
|
||||
if (settings.side_chain_field && !stopped && state.ran && !ctx.Cancelled() && !ctx.refl.Twinned() &&
|
||||
@@ -4110,11 +4233,12 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
if (topo.refinable[i])
|
||||
refinable.push_back(static_cast<int>(i));
|
||||
TrialOutcome after = one_step(moved);
|
||||
const bool kept = after.ran && after.trial.within_bounds && after.trial.r_free < unchanged.trial.r_free &&
|
||||
after.trial.nll_free < unchanged.trial.nll_free;
|
||||
logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}; from the model as it "
|
||||
"was R-free {:.4f}, -LL free {:.1f}: {}", after.trial.r_free, after.trial.nll_free,
|
||||
unchanged.trial.r_free, unchanged.trial.nll_free, kept ? "kept" : "not kept");
|
||||
const bool kept = after.ran && unchanged.ran && advances(after.trial, unchanged.trial);
|
||||
logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}, target {:.6g}; from "
|
||||
"the model as it was R-free {:.4f}, -LL free {:.1f}, target {:.6g}: {}", after.trial.r_free,
|
||||
after.trial.nll_free, WorkTarget(after.trial, after.trial.weight, after.trial.adp_weight),
|
||||
unchanged.trial.r_free, unchanged.trial.nll_free,
|
||||
WorkTarget(unchanged.trial, after.trial.weight, after.trial.adp_weight), kept ? "kept" : "not kept");
|
||||
if (kept) {
|
||||
result.side_chain_field_kept = true;
|
||||
result.side_chain_field_moved = static_cast<int>(field.moves.size()) - pruned;
|
||||
@@ -4132,8 +4256,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
}
|
||||
}
|
||||
|
||||
// 3. The final weights: a short ladder from where the trajectory ended, tighter geometry first, its choice
|
||||
// kept where it displaces that. A rung the step from there tried already is taken from that step.
|
||||
// 3. The final weights: a short ladder from where the trajectory ended, the weight as it is first, its choice
|
||||
// on the free set taken where it advances from there (advances). A rung the step from there tried already is taken from that step.
|
||||
if (!stopped && state.ran && !ctx.Cancelled()) {
|
||||
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
|
||||
std::vector<TrialWeights> trials;
|
||||
@@ -4156,21 +4280,21 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
if (!outcomes[i].ran)
|
||||
outcomes[i] = std::move(ran[m++]);
|
||||
const int k = choose(outcomes, "final");
|
||||
if (k >= 0 && displaces(outcomes[k].trial, state.trial))
|
||||
if (k >= 0 && advances(outcomes[k].trial, state.trial))
|
||||
state = std::move(outcomes[k]);
|
||||
}
|
||||
// 4. The displacement model, decided late, on the free set: from where the weights ended, TLS bodies (the
|
||||
// model's own groups, one per chain, the chains in TLSMD segments) with a residual B, and a U per atom at
|
||||
// its balanced weight and a quarter of it - each trial as long as a step, on the positions where the weights
|
||||
// ended, held, so that the trials differ in the displacement model alone; the model chosen then takes a
|
||||
// step with the positions refined, kept where it displaces its trial (against trials a macro-cycle longer
|
||||
// step with the positions refined, kept where it advances on the work-set target from its trial (against trials a macro-cycle longer
|
||||
// that refined the positions too, on 50 refinements of the battery R-free 0.001 lower on average and none
|
||||
// higher by more than 0.0015, at 78% of the evaluations). Where the geometry is out of bounds there, which
|
||||
// held positions cannot mend, each trial refines them too, a macro-cycle longer than a step, and no step
|
||||
// follows. One displaces the one before it (the first
|
||||
// where the weights ended, the B factors as they are) only where its geometry is within bounds and both its
|
||||
// R-free (beyond the tie) and the -ln L of the free reflections are lower: the free set is the test of the
|
||||
// parameters it adds.
|
||||
// where the weights ended, the B factors as they are) only where its geometry is within bounds and the free
|
||||
// set prefers it (FreeSetPrefers: its -ln L of the free reflections lower, its R-free not higher beyond its
|
||||
// standard error): the free set is the test of the parameters it adds.
|
||||
// Following Merritt (2012) Acta Cryst. D68, 468-477 (the displacement model the data support, decided by test)
|
||||
// The trials that would not change the choice are left out: the model's own TLS groups are tried only where
|
||||
// one body per chain or the finest cut lowers R-free, the cuts between those only where also the finest
|
||||
@@ -4190,9 +4314,14 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
state.trial.r_free, state.trial.nll_free);
|
||||
std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, own_tls);
|
||||
const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0;
|
||||
// a U per atom only where the data are many times its parameters
|
||||
int atoms_b = 0;
|
||||
for (int i : refinable)
|
||||
atoms_b += !ctx.atoms[i]->is_hydrogen();
|
||||
const bool u_tried = result.n_work > U_MIN_REFLECTIONS_PER_ATOM * atoms_b;
|
||||
// the U weight balanced on the model with every refinable atom given a U
|
||||
double u_w0 = adp;
|
||||
{
|
||||
if (u_tried) {
|
||||
const std::vector<AtomAniso> keep = ctx.aniso;
|
||||
std::vector<AtomAniso> all = keep.empty() ? std::vector<AtomAniso>(ctx.atoms.size()) : keep;
|
||||
std::vector<char> refines_u(ctx.atoms.size(), 0);
|
||||
@@ -4215,6 +4344,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
int own = -1, coarsest = -1, finest = -1, u_quarter = -1;
|
||||
for (const AdpHypothesis &h : hyps) {
|
||||
if (h.kind == AdpHypothesis::Kind::Anisotropic) {
|
||||
if (!u_tried)
|
||||
continue;
|
||||
u_quarter = static_cast<int>(trials.size());
|
||||
for (double f : {0.25, 1.0}) {
|
||||
trials.push_back({w, u_w0 * f, ncs_s});
|
||||
@@ -4298,20 +4429,34 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
continue;
|
||||
logger.Info("Refinement, displacements: {}", outcomes[k].line);
|
||||
result.adp_trials.push_back(t);
|
||||
if (t.within_bounds && (!trial(best).within_bounds ||
|
||||
(t.r_free < trial(best).r_free - R_FREE_TIE && t.nll_free < trial(best).nll_free)))
|
||||
// the larger model where Hamilton's tests take it, the smaller where they refuse it, the free set's
|
||||
// choice where they are inconclusive (or the two have as many parameters)
|
||||
const RefinementWeightTrial &b = trial(best);
|
||||
const bool larger = t.adp_parameters > b.adp_parameters;
|
||||
const double pass = larger ? HamiltonPassFraction(t, b, result.n_work)
|
||||
: HamiltonPassFraction(b, t, result.n_work);
|
||||
bool takes = displaces(t, b);
|
||||
if (t.adp_parameters != b.adp_parameters && (pass > HAMILTON_LARGER || pass < HAMILTON_SMALLER))
|
||||
takes = t.within_bounds && (!b.within_bounds || (pass > HAMILTON_LARGER) == larger);
|
||||
logger.Info("Refinement, displacements: {} against {}: the larger passes {:.0f}% of Hamilton's tests{}",
|
||||
t.adp.empty() ? "the B factors as they are" : t.adp,
|
||||
b.adp.empty() ? "the B factors as they are" : b.adp, 100 * pass, takes ? ", taken" : "");
|
||||
if (takes)
|
||||
best = static_cast<int>(k);
|
||||
}
|
||||
// (the model where the weights ended keeps the displacement model a round before chose for it)
|
||||
if (result.adp_model.empty() || state.trial.adp.empty())
|
||||
// (the model where the weights ended keeps the displacement model a round before chose for it, its U held by
|
||||
// the weight trials since)
|
||||
if (result.adp_model.empty())
|
||||
result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held";
|
||||
else if (state.trial.adp.empty() && result.adp_model != "isotropic" && !result.adp_model.ends_with("held"))
|
||||
result.adp_model += ", U held";
|
||||
if (best >= 0) {
|
||||
result.adp_model = outcomes[best].trial.adp;
|
||||
state = std::move(outcomes[best]);
|
||||
}
|
||||
if (best >= 0 && hold) {
|
||||
// the model chosen refined on with the positions, a step from where its trial ended, kept where it
|
||||
// displaces that
|
||||
// advances on the work-set target from there
|
||||
AdpHypothesis chosen = *which[best];
|
||||
chosen.bodies = state.tls;
|
||||
chosen.tls_b = state.tls_b;
|
||||
@@ -4320,7 +4465,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
ncs_p, {&chosen});
|
||||
if (on[0].ran) {
|
||||
logger.Info("Refinement, displacements, positions refined: {}", on[0].line);
|
||||
if (displaces(on[0].trial, state.trial))
|
||||
if (advances(on[0].trial, state.trial))
|
||||
state = std::move(on[0]);
|
||||
}
|
||||
}
|
||||
@@ -4332,17 +4477,24 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
// weights on its B factors, the ladder (1.) from it with its displacements as they are (the U of TLS or
|
||||
// of the anisotropic model held, B held or refined), the trajectory (2.) from the rung kept, and the
|
||||
// displacement hypotheses (4.) from where that ends - the TLS bodies refined from the B factors the first
|
||||
// choice left, which they could not be from the B factors as given. A round is kept where it lowers R-free
|
||||
// beyond the tie and undone otherwise, RefinementSettings::rounds in all: from a placed prediction, whose B
|
||||
// choice left, which they could not be from the B factors as given. A round is kept where it advances as a step
|
||||
// does (advances) and undone otherwise, RefinementSettings::rounds in all: from a placed prediction, whose B
|
||||
// factors are its pLDDT (6 A^2 under an overall B of 45), the first round balanced the B factors' weight
|
||||
// a hundred times too stiff - the B similarity's gradient goes as 1 / B - and chose the middle rung with
|
||||
// NCS; the second chose the tightest rung without, and from the B factors the first left, its TLS bodies
|
||||
// 0.02 lower in R-free. Where the ladder and the trajectory leave the model where it was, the hypotheses
|
||||
// would run the same trials from the same model again, so the round ends there: from a model refined
|
||||
// before, a round is its ladder and one step. No round where the refinement does not displace the model
|
||||
// as given (5.).
|
||||
// before, a round is its ladder and one step. No round where the model as given is kept over the
|
||||
// refinement (5.).
|
||||
// The model as given kept over the refinement: where the free set prefers it, as one weight is preferred to
|
||||
// another (displaces), and the refinement is not better on both R-work and R-free. The work-set target cannot
|
||||
// choose here, the refinement having minimised it from the model as given.
|
||||
auto given_kept = [&]() {
|
||||
return displaces(given.trial, state.trial) &&
|
||||
!(state.trial.r_free < given.trial.r_free && state.trial.r_work < given.trial.r_work);
|
||||
};
|
||||
for (int round = 2; round <= settings.rounds && settings.adp_hypotheses && !stopped && state.ran &&
|
||||
!ctx.Cancelled() && (!settings.as_given_competes || displaces(state.trial, given.trial));
|
||||
!ctx.Cancelled() && (!settings.as_given_competes || !given_kept());
|
||||
round++) {
|
||||
const TrialOutcome before = state;
|
||||
const std::vector<TlsBody> own_tls = state.tls;
|
||||
@@ -4402,23 +4554,25 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
|
||||
}
|
||||
const std::string adp_before = result.adp_model;
|
||||
displacements(own_tls.empty() ? input_tls : own_tls);
|
||||
const bool kept = state.trial.r_free < before.trial.r_free - R_FREE_TIE;
|
||||
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}{}", round, before.trial.r_free, state.trial.r_free,
|
||||
kept ? "" : ", not kept");
|
||||
const bool kept = advances(state.trial, before.trial);
|
||||
logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}, target {:.6g} -> {:.6g}{}", round,
|
||||
before.trial.r_free, state.trial.r_free,
|
||||
WorkTarget(before.trial, state.trial.weight, state.trial.adp_weight),
|
||||
WorkTarget(state.trial, state.trial.weight, state.trial.adp_weight), kept ? "" : ", not kept");
|
||||
if (!kept) {
|
||||
state = before;
|
||||
result.adp_model = adp_before;
|
||||
break;
|
||||
}
|
||||
}
|
||||
// 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every
|
||||
// trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what
|
||||
// comes out is never worse on the free set than what went in - where the free set is free to the model as
|
||||
// given (RefinementSettings::as_given_competes).
|
||||
if (settings.as_given_competes && state.ran && !ctx.Cancelled() && !displaces(state.trial, given.trial)) {
|
||||
logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}), the refinement did not "
|
||||
"displace it (R-work {:.4f}, R-free {:.4f})", given.trial.r_work, given.trial.r_free,
|
||||
state.trial.r_work, state.trial.r_free);
|
||||
// 5. The model as given: where it is kept over the refinement (given_kept), it is given back as it came (the
|
||||
// solvent build's edits undone), so that what comes out is never worse on both free measures than what went
|
||||
// in - where the free set is free to the model as given (RefinementSettings::as_given_competes).
|
||||
if (settings.as_given_competes && state.ran && !ctx.Cancelled() && given_kept()) {
|
||||
logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f}), the "
|
||||
"refinement did not displace it (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f})",
|
||||
given.trial.r_work, given.trial.r_free, given.trial.nll_free, state.trial.r_work,
|
||||
state.trial.r_free, state.trial.nll_free);
|
||||
size_t a = 0, k = 0;
|
||||
for (gemmi::Chain &ch : ctx.Model().chains)
|
||||
for (gemmi::Residue &res : ch.residues) {
|
||||
|
||||
@@ -160,6 +160,12 @@ struct RefinementWeightTrial {
|
||||
double ncs_rmsd = NAN; // the copies' C-alpha (P) rmsd after superposition, where there are copies
|
||||
double r_work = 1, r_free = 1;
|
||||
double nll_free = NAN; // -ln L of the free reflections
|
||||
// The work-set target's terms where the trial ended (RefineOnce's WorkTarget): -ln L of the working reflections,
|
||||
// the restraints' sum of the geometry (with the NCS restraints at ncs_sigma) and that of the B similarity.
|
||||
double nll_work = NAN, e_geometry = NAN, e_b = NAN;
|
||||
int adp_parameters = 0; // the displacement model's parameters (HamiltonPassFraction)
|
||||
int adp_restrained = 0; // of them those the restraints act on, which HamiltonPassFraction credits
|
||||
std::vector<float> free_r_terms; // R-free as one term per free twin group (FreeRTerms), for paired comparisons
|
||||
GeometryStatistics geometry;
|
||||
bool within_bounds = false; // the geometry bounds the choice is made inside
|
||||
std::string adp; // the displacement model: "isotropic", "TLS ..." or "anisotropic"
|
||||
|
||||
Reference in new issue
Block a user