diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md
index aa2328e36..d2d4598be 100644
--- a/docs/ACKNOWLEDGEMENT.md
+++ b/docs/ACKNOWLEDGEMENT.md
@@ -652,6 +652,16 @@ structure refinement with phenix.refine" (2012), Acta Cryst. D68, 352-367
A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal
structures" (1992), Nature 355, 472-475 [doi:10.1038/355472a0](https://doi.org/10.1038/355472a0).
+**What the free set decides** — the free set decides the weights and the displacement model alone, and only
+where its two measures agree: a candidate is taken where its free $-\ln L$ is lower and its R-free not higher
+than the incumbent's by more than the paired standard error of the difference; every other step of the
+refinement is taken on the work-set target. The approach of deciding on more than one free measure, and of keeping the
+free set for few choices, is PDB-REDO's: R. P. Joosten, K. Joosten, G. N. Murshudov and A. Perrakis,
+"PDB_REDO: constructive validation, more than just looking for errors" (2012), Acta Cryst. D68, 484-496
+[doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515), and R. P. Joosten, F. Long,
+G. N. Murshudov and A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization" (2014),
+IUCrJ 1, 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324).
+
**Robust scaling of the model** — the overall scale and anisotropic B are fitted by iteratively
reweighted least squares with the pseudo-Huber weight. P. J. Huber, "Robust estimation of a location
parameter" (1964), Ann. Math. Statist. 35, 73-101
@@ -692,7 +702,13 @@ SHELXL" (2015), Acta Cryst. C71, 3-8
[doi:10.1107/S2053229614024218](https://doi.org/10.1107/S2053229614024218) documents them. Which
displacement model a refinement keeps is decided on the free set rather than by a rule of
resolution: E. A. Merritt, "To B or not to B: a question of resolution?" (2012), Acta Cryst. D68,
-468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320).
+468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320). Between a model
+with more displacement parameters and one with fewer, the R-factor ratio test of W. C. Hamilton, "Significance
+tests on the crystallographic R factor" (1965), Acta Cryst. 18, 502-510 (DOI not verified), decides first,
+over a range of restraint weights as Merritt (2012) applies it, with PDB-REDO's (Joosten et al. 2012, above)
+95%/30% rule and its bound of 18 reflections per atom for a U per atom; its F point is from E. Paulson,
+"An approximate normalization of the analysis of variance distribution" (1942), Ann. Math. Statist. 13,
+233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608).
**Population and crossover** — the members' refinement on blurred data is a continuation (homotopy) on the
resolution, the deformation of the landscape of L. Piela, J. Kostrowicki and H. A. Scheraga, "On the
diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md
index 84af30f96..a35c12d3f 100644
--- a/docs/CPU_DATA_ANALYSIS.md
+++ b/docs/CPU_DATA_ANALYSIS.md
@@ -18,7 +18,7 @@ This document describes the crystallographic algorithms implemented in Jungfrauj
12. auxiliary statistics (Wilson plot, ⟨I/σ(I)⟩, CC1/2, CCref),
13. amplitude estimation (French–Wilson) and R-free test-set flagging,
14. optional model-based validation: rigid-body placement of a supplied model (molecular replacement where the rigid body does not place it, or for a predicted model), R-free against it, sigma_A-weighted 2mFo−DFc / mFo−DFc electron-density maps, and an anomalous difference map with the strongest anomalous sites named.
-15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight chosen by R-free within geometry bounds, and the unexplained blobs of the refined difference map.
+15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight and the displacement model chosen on the free set (free -ln L and R-free agreeing) within geometry bounds, every other step taken on the work-set target, and the unexplained blobs of the refined difference map.
The reference is split into five parts, in pipeline order; the section numbers run continuously
diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md
index 6fde51f9f..14876ac72 100644
--- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md
+++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md
@@ -42,8 +42,8 @@ no twin law is tested, the data carrying the larger symmetry exactly. The refine
it was refined in. An atom's anisotropic U (ANISOU, `_atom_site_anisotrop`) is kept. Where the model has TLS
groups (`_pdbx_refine_tls*`, PDB `REMARK 3`) whose atoms bring no U of their own, their U is made from the
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
-groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
-R-free on the model as given (`REFINE_ADP_INPUT`). Occupancies are kept as they are, and
+groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, the free set prefers on
+the model as given (§15.7; `REFINE_ADP_INPUT`). Occupancies are kept as they are, and
so are alternative conformations, each refined with its own restraints.
Two stages follow.
@@ -51,12 +51,12 @@ Two stages follow.
1. **Placement.** Each chain of at least 50 non-water atoms is moved as a rigid body - an angle-axis
rotation about its centroid, scaled by its rms radius so that a unit of rotation moves a typical atom by
about an angstrom, and a translation - against the X-ray target (§15.2), by Ceres' L-BFGS for 30
- iterations. Then one B shift per residue. Each of the two is **kept only where it lowers R-free by more
- than 0.001**: a deposited model was refined atom by atom already, and moving its chains or shifting its
- B factors residue by residue then only makes it worse. The placement stage needs no dictionary.
+ iterations. Then one B shift per residue. Each of the two is **kept only where it lowers the $-\ln L$ of the
+ working reflections**, the target it minimised: the free set decides the weights and the displacement model
+ alone (§15.7). The placement stage needs no dictionary.
2. **Restrained refinement.** Every atom's position and isotropic B, against the X-ray target and the
geometry of the dictionary (§15.4-§15.5) - the built-in one with the user's restraint files over it
- (`--restraints`) - in weight trials chosen by R-free (§15.7). Residues the
+ (`--restraints`) - in weight trials chosen on the free set (§15.7). Residues the
dictionary does not describe, and residues with an atom it does not know, are **held fixed**: they stay
in the structure factors, in the bulk-solvent mask and in the repulsions the refined atoms see, but
they do not move. So do atoms of zero occupancy.
@@ -344,29 +344,52 @@ $w_B\,T_\mathrm{X\text{-}ray} + E_B$ for the B factors. Rather than fixing $w$ a
runs **short refinements at several weights and keeps the one that does best on the free set** - a weight
chosen late, on the evidence, instead of early at a rule.
+**What the free set decides.** The free set decides the weights - the X-ray weights of the positions and of the B
+factors and the NCS restraints' $\sigma$ - the displacement model (§15.7, 6.), the reading of the model's TLS groups
+(§15.1) and whether the model as given is kept over the refinement (7.), and nothing else. Each of these choices asks
+both of its measures: a candidate is preferred to the incumbent only where its $-\ln L$ of the free reflections is
+lower **and** its R-free is not higher by more than the standard error of the difference. The two are scored on the
+same free reflections, so the difference is paired: R-free is a sum of one term per free reflection (per free twin
+group under a twin law, whose members share their $F_\mathrm{calc}$), and the standard error is that of the sum of
+the terms' differences - two trials from one start differ far less from reflection to reflection than either
+scatters about the truth, so this is narrower than the error of one R-free, $R_\mathrm{free}/\sqrt{2N_\mathrm{free}}$.
+One measure alone, or a fixed tie, picks the noise of a few hundred reflections as often as a better model (PDB-REDO
+likewise takes a change only where its free measures agree, Joosten et al. 2014). $-\ln L$ free is that of the
+refinement's own likelihood parameters - the variance fitted to the free reflections; with D and the variance fitted
+to the working reflections instead, it kept out a per-atom U that overfits at 2.2 Å, but held back every
+refinement (on 17 structures R-free 0.0016 and R-work 0.008 higher on average), and it is not used. Every other step - a step of the
+trajectory, a round, the placement's rigid bodies and B per residue, the solvent build's rounds, the side-chain
+field, the population's child - is taken on the **work-set target**: the target the trial minimises,
+$w\,T_\mathrm{X\text{-}ray} + E_\mathrm{geometry}$ (with the NCS restraints at their $\sigma$), plus
+$w_B\,T_\mathrm{X\text{-}ray} + E_B$ where the B factors are refined, $T_\mathrm{X\text{-}ray}$ the $-\ln L$ of the
+working reflections - at the weights the two candidates share, and taken where it lowers that by more than
+$10^{-4}$ of it, relative, and the geometry stays within bounds where it was. A step at other weights (X-ray weights
+or NCS $\sigma$) is not compared on a target that changed with them - the positions and the B factors minimise
+targets of their own, whose sum a change of weight can raise while it lowers each - and the free set decides it as
+it decides every weight. The free set thus tests a
+handful of choices, each between a few candidates, and the R-free reported is not the best of the many steps the
+refinement took.
+
The starting weights balance the gradients: $w_0 = |\nabla E_\mathrm{geometry}| / |\nabla T_\mathrm{X\text{-}ray}|$,
the geometry's gradient taken on the model shaken by 0.1 Å (a refined model's geometry would give none), and
$w_{B,0}$ likewise from the B restraints on the B factors shaken by 20%. Each **trial** is two macro-cycles
(the scale, solvent and likelihood refitted, then the positions, then the B factors) from where it starts.
A trial is **within bounds** where its rmsd from the dictionary is at most 0.02 Å for bonds and 2° for
-angles; one trial **displaces** another where it is within bounds and its R-free is lower by more than 0.001
-(differences smaller than that do not decide, and the trial listed first - on a ladder, the tighter
-geometry - is kept). The trials
-are:
+angles; one trial **displaces** another where it is within bounds and the free set prefers it as above (the trial
+listed first - on a ladder, the tighter geometry - is kept otherwise). The trials are:
1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors
held and refined at $w_{B,0} \times \{1/16, 1/4, 1\}$ - twelve trials (the B factors are refined first at
- the middle rung only, and at the others where that comes within two ties of holding them there). The lower
+ the middle rung only, and at the others unless holding them there displaces refining them). The lower
B weights are for a model whose B factors arrive far from where they refine - a prediction's pLDDT - whose
balanced weight is not the one its B wants: they took 0.002-0.010 off R-free on five such starts, and
nothing off eight deposited models.
2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights
as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at
one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and
- multiplied by four. A step is taken while it lowers R-free or R-work by more than 0.001 and its R-free
- stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model,
- whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the
- trajectory ends where it began.
+ multiplied by four. The step's weights are chosen among its trials on the free set; the step is taken while
+ that trial lowers the work-set target (or, at other weights, displaces where the last step ended), and the
+ trajectory ends where the last step taken ended.
3. **The population** (off by default, `RefinementSettings::population`: with the minimiser of §15.6 its members
took nothing off R-free over 19 refinements that the second round did not), from where the trajectory ended:
up to twelve members (at most as many as the trial slots, so that they run in one wave; none below two) refined
@@ -380,8 +403,8 @@ are:
members see those around it. Their **crossover** is made on one 2mFo-DFc map averaged over the members with
the free reflections left out: each residue is taken from the member whose atoms sit in the most density,
the junctions are mended by a macro-cycle at the ladder's tightest rung, and the child is refined by one
- trial at the trajectory's weights. The child displaces the state by the rule every trial is held to; the
- members themselves are never chosen among on the free set, whose best of twelve would be the best of twelve. On
+ trial at the trajectory's weights. The child is taken where it lowers the work-set target, as a step is; the
+ free set never sees the members. On
13 predictions placed by MR the child was kept on four (R-free 0.005-0.011 lower) and on one of 22 refined
models (0.009 lower, the rest unchanged); the gain, where there is one, comes mostly in the rounds that
follow from the child.
@@ -402,14 +425,15 @@ are:
sweep would otherwise take more than $4 \times 10^8$ evaluations (states times reflections, about 3 s on one
card). An FFT gradient cannot carry a side chain 1.5 Å at 1.7 Å
resolution; the field, a search, can, and it is scored by the likelihood rather than by a map, which on an
- incomplete model rewards whatever fills unmodelled density. The free reflections are never seen by the field:
- one step at the trajectory's weights is then taken from it and judged against the same step from the model as
- it was, kept only where both its R-free and its $-\ln L$ of the free reflections are lower and the geometry
- within bounds, undone otherwise (a model whose side chains the data already hold moves nothing and costs the
+ incomplete model rewards whatever fills unmodelled density. The free reflections are never seen by the field
+ nor by its test: one step at the trajectory's weights is then taken from it and judged against the same step
+ from the model as it was, kept only where it lowers the work-set target and the geometry stays within bounds,
+ undone otherwise (a model whose side chains the data already hold moves nothing and costs the
sweep alone). The log names each side chain moved with its chi angles before and after and the energy the move
took off (`REFINE_SIDE_CHAIN_FIELD_MOVED`, `REFINE_SIDE_CHAIN_FIELD_PRUNED`). Twinned data and a model with
anisotropic atoms are refined without the field.
-5. **The final ladder**: the trajectory's weights times $\{1, 2\}$, kept where one displaces it (the weight halved
+5. **The final ladder**: the trajectory's weights times $\{1, 2\}$, chosen on the free set, the choice taken where it
+ advances from where the trajectory ended as a step does (the weight halved
was never kept on 55 refinements, and taking it away changed no choice).
6. **The displacement model**, from where the weights ended (the B factors as they are: isotropic, or the model's U
held), each trial as long as a step on the positions where the weights ended, held - only the displacement
@@ -425,8 +449,7 @@ are:
reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving
out the U weight $\times 4$ changed a choice taken. A
trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds
- and both its R-free (beyond the tie) and the $-\ln L$ of the free reflections are lower: the free set tests the
- parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012).
+ and the free set prefers it: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). Between a model with more displacement parameters (TLS over a B per atom, a U per atom over either) and one with fewer, Hamilton's R-ratio test decides first, as Merritt (2012) and PDB-REDO apply it: the larger passes where its R-work is below the smaller's by more than the ratio $\sqrt{b/(n-m)\,F_{b,n-m,0.05} + 1}$, $n$ the working reflections, $m$ its displacement parameters (the positions are held) and $b$ the extra ones (Hamilton 1965; the F point by Paulson's approximation). The restraints are credited as $wR$ observations, $w$ unknown: each model's is scanned from 0 to the bound at which its restraints take every parameter they restrain off its count (Merritt's $N_\mathrm{atoms}/R$ for a B per atom, $6N_\mathrm{atoms}/R$ for a U per atom), six values each, independently. The larger model is taken where it passes more than 95% of the 36 tests, the smaller where it passes fewer than 30%, and in between the free set decides as everywhere else (PDB-REDO falls back on Z(R-free) and the R-free - R-work gap at fixed cut-offs instead; the paired rule is the one every other free-set choice here uses). A U per atom is tried only above 18 working reflections per refinable atom, PDB-REDO's bound.
The refined U are written as ANISOU and the TLS groups as `_pdbx_refine_tls*` and `REMARK 3` (`REFINE_ADP_MODEL`,
`REFINE_ANISO_ATOMS`, `REFINE_TLS_GROUPS`). It is decided after the solvent build (§15.10), on the model with its
solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no
@@ -437,14 +460,16 @@ are:
B factors as given and its displacement model after them; on a placed prediction, whose B factors are its pLDDT,
the B factors' weight balanced on them is a hundred times too stiff (the B similarity's gradient goes as $1/B$),
the middle rung wins where the tightest wins once the B factors are refined, and the TLS bodies fitted from the
- refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken while it lowers
- R-free beyond the tie and undone otherwise; where its ladder and trajectory leave the model where it was, the
+ refined B factors come 0.02 lower in R-free than from the B factors as given. A round is taken where it advances as a
+ step does - on the work-set target where its weights are those it started from, on the free set where it chose
+ others - and undone otherwise; where its ladder and trajectory leave the model where it was, the
hypotheses would repeat the same trials, and it ends there - from a model refined before, a round is one ladder
and one step. No round runs where the refinement does not displace the model as given.
-7. **The model as given** is the first candidate in order of trust: where the refinement does not displace it by the
- rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as
- it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes out is never
- worse on the free set than what went in. This holds only where the free set is free to the model as given: a model
+7. **The model as given** is kept over the refinement where it displaces the refinement on the free set (its geometry
+ within bounds and the free set preferring it) and the refinement is not better on both R-work and R-free (the
+ work-set target cannot choose here, the refinement having minimised it from the model as given): the model is
+ written as it came, scaled to the data, and the solvent build's edits are undone (`REFINE_AS_GIVEN`). What comes
+ out is never worse on both free measures than what went in. This holds only where the free set is free to the model as given: a model
placed as given on a free set drawn by the run (in `--mode refine`, a file without free flags) has
most likely been refined against those reflections, and its R-free as given does not compete.
@@ -453,7 +478,7 @@ without them, and then with them at $\sigma = 0.1$ Å at the B factors' weight t
X-ray weight a rung up as often as not, the B factors' choice it never changed on 39 refinements), and every step of
the trajectory also tries them changed - switched on where they are off, and in turn tightened ($\sigma$ halved) and
loosened ($\sigma$ doubled, off beyond 0.4 Å) where they are on. Listed after their twin without, the restraints are
-kept only where they lower R-free by more than the tie. A model without copies runs exactly the trials it ran before.
+kept only where the free set prefers them. A model without copies runs exactly the trials it ran before.
$w$ never exceeds $w_0/8$. The trials of one set run side by side - on one GPU, on several, or on the CPU's
threads - each from the same start and taking nothing from the one before it, so the result does not depend
@@ -576,8 +601,8 @@ and removed and waters and ions relabelled, as phenix.refine builds waters withi
trajectory (§15.7) ends, a round reads the maps and the audit of that model and changes its solvent; then the solvent
alone - its waters and single-atom ions, the rest held - is refined at the weights the trajectory ended at (its B
too, at its balanced weight where the trajectory held B, and 10 iterations a macro-cycle), and the round is kept
-where that lowers R-free; where it does not but the round as built does (refining the solvent alone can lose what its
-sites gained), it
+where that lowers the work-set target at those weights (§15.7); where it does not but the round as built does
+(refining the solvent alone can lose what its sites gained), it
is kept as built; otherwise it is undone. The rest of the model goes on with its B factors held or refined as the
trajectory had them. The rounds stop at one that changes nothing, at one undone, or after two (a third gains
nothing measurable); the final ladder of §15.7 then
@@ -786,6 +811,7 @@ What the refinement does not do yet - each a known gap, not a design choice:
- A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long & G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15.4).
- M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (the topology, the riding hydrogens and the blob search, §15.4, §15.9).
- V. Schomaker & K. N. Trueblood, *Acta Cryst.* **B24** (1968), 63-76, and M. D. Winn, M. N. Isupov & G. N. Murshudov, *Acta Cryst.* **D57** (2001), 122-133 (TLS, and the B column total or residual to it, §15.1, §15.5); J. Painter & E. A. Merritt, *Acta Cryst.* **D62** (2006), 439-450 (TLSMD's segments, §15.7).
+- W. C. Hamilton, "Significance tests on the crystallographic R factor", *Acta Cryst.* **18** (1965), 502-510, as E. A. Merritt, "To B or not to B: a question of resolution?", *Acta Cryst.* **D68** (2012), 468-477 [doi:10.1107/S0907444911028320](https://doi.org/10.1107/S0907444911028320) applies it to the displacement model, and E. Paulson, "An approximate normalization of the analysis of variance distribution", *Ann. Math. Statist.* **13** (1942), 233-235 [doi:10.1214/aoms/1177731608](https://doi.org/10.1214/aoms/1177731608) (the F point) (§15.7).
- F. L. Hirshfeld, *Acta Cryst.* **A32** (1976), 239-244, A. Thorn, B. Dittrich & G. M. Sheldrick, *Acta Cryst.* **A68** (2012), 448-451, and G. M. Sheldrick, *Acta Cryst.* **C71** (2015), 3-8 (the rigid-bond, similarity and isotropy restraints of anisotropic atoms, §15.5); E. A. Merritt, *Acta Cryst.* **D68** (2012), 468-477 (the displacement model decided by test, §15.7).
- A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5).
- M. M. Harding, "Geometry of metal-ligand interactions in proteins", *Acta Cryst.* **D57** (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", *Acta Cryst.* **D62** (2006), 678-682; F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5).
@@ -800,6 +826,8 @@ What the refinement does not do yet - each a known gap, not a design choice:
- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7).
- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free).
+- I. J. Tickle, R. A. Laskowski & D. S. Moss, "Rfree and the Rfree ratio. II. Calculation of the expected values and variances of cross-validation statistics in macromolecular least-squares refinement", *Acta Cryst.* **D56** (2000), 442-450 [doi:10.1107/S0907444999016868](https://doi.org/10.1107/S0907444999016868) (the standard error of one R-free, against which the paired error of §15.7 is set).
+- R. P. Joosten, F. Long, G. N. Murshudov & A. Perrakis, "The PDB_REDO server for macromolecular structure model optimization", *IUCrJ* **1** (2014), 213-220 [doi:10.1107/S2052252514009324](https://doi.org/10.1107/S2052252514009324), and R. P. Joosten, K. Joosten, G. N. Murshudov & A. Perrakis, "PDB_REDO: constructive validation, more than just looking for errors", *Acta Cryst.* **D68** (2012), 484-496 [doi:10.1107/S0907444911054515](https://doi.org/10.1107/S0907444911054515) (choices taken only where more than one free measure agrees, §15.7).
- G. J. Kleywegt, "Use of non-crystallographic symmetry in protein structure refinement", *Acta Cryst.* **D52** (1996), 842-857; I. Usón, E. Pohl, T. R. Schneider, Z. Dauter, A. Schmidt, H.-J. Fritz & G. M. Sheldrick, "1.7 Å structure of the stabilized REIv mutant T39K. Application of local NCS restraints", *Acta Cryst.* **D55** (1999), 1158-1167; R. A. Nicholls, F. Long & G. N. Murshudov, "Low-resolution refinement tools in REFMAC5", *Acta Cryst.* **D68** (2012), 404-417; J. J. Headd et al., "Flexible torsion-angle noncrystallographic symmetry restraints for improved macromolecular structure refinement", *Acta Cryst.* **D70** (2014), 1346-1356 (local NCS restraints with a robust loss, decided by R-free, and Murshudov et al. 2011 above for the copies found by alignment, §15.5, §15.7).
- P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9).
- N. Naleem, A. Henning-Knechtel, S. Kirmizialtin & P. Auffinger, "Cat_Wiz: a stereochemistry-guided toolkit for locating, diagnosing, and annotating Mg2+ ions in RNA structures", *Nucleic Acids Res.* **54** (2026), gkag120; F. Leonarski, L. D'Ascenzo & P. Auffinger, *Nucleic Acids Res.* **45** (2017), 987-1004 and *RNA* **25** (2019), 173-192; F. Leonarski, A. Henning-Knechtel, S. Kirmizialtin, E. Ennifar & P. Auffinger, *Nucleic Acids Res.* **53** (2025), gkae1148 (the coordination bands, distances and exclusion zone of the solvent audit, §15.10).
diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md
index 1df1f4d77..fb11d3a97 100644
--- a/docs/RUGNUX_MODEL_AND_REFERENCE.md
+++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md
@@ -287,12 +287,14 @@ element and the number of such records of the ion (Mg-O 2.07 A, Na-O 2.42 A, K-O
usual coordination), where the model's distance is within 0.33 A of that (Mg-O up to 2.4 A); a record at an
outer-sphere distance, a contact without a record, or one to a symmetry mate, is not restrained as a bond. The B factors of bonded atoms and of
atoms in contact (waters included) are restrained to each other in proportion to their size. The weights of
-the X-ray term against the geometry and against that B restraint are chosen by R-free among short
-refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees: three weights from the placed
+the X-ray term against the geometry and against that B restraint are chosen on the free set among short
+refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees - a candidate taken only where its free
+-ln L is lower and its R-free not higher than the incumbent's by more than the standard error of their paired
+difference over the free reflections: three weights from the placed
model, each with the B factors held and refined; then a trajectory of further short refinements from the
-best, each trying the weights as they are and changed, taken while R-free or R-work falls beyond 0.001 and
-R-free stays within 0.001 of its best; then a short final ladder, kept only where it lowers R-free beyond
-0.001. The free set is the merge's own, so the refined R-free judges the refinement against reflections it
+chosen one, each trying the weights as they are and changed, taken while it lowers the work-set target (the
+refined target at the chosen weights); then a short final ladder. The free set decides the weights and the
+displacement model only; every other step is taken on the work-set target. The free set is the merge's own, so the refined R-free judges the refinement against reflections it
never fitted - but where the model was deposited, those reflections were most likely its depositor's working
reflections, and a refinement against new data then moves R-free up from the unrefined value rather than
down. `R_WORK`/`R_FREE` of the model as given are never replaced. On a GPU a refinement takes seconds; the CPU
diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp
index ac0ffc781..e118c2fbb 100644
--- a/image_analysis/structure_refinement/Refinement.cpp
+++ b/image_analysis/structure_refinement/Refinement.cpp
@@ -91,8 +91,62 @@ constexpr double REPULSION_MARGIN = 0.5; // A: pairs this much beyond their co
constexpr double MAX_RMSD_BONDS = 0.02, MAX_RMSD_ANGLES = 2.0;
// The minimiser's iterations per macro-cycle where the solvent build refines the solvent alone.
constexpr int SOLVENT_ITERATIONS = 10;
-// R-free differences smaller than this do not decide between two weights: the tighter geometry is taken.
-constexpr double R_FREE_TIE = 0.001;
+// The free set decides the weights and the displacement model alone, and only where both of its measures agree: a
+// candidate is preferred to the incumbent where its -ln L of the free reflections is lower and its R-free not higher
+// than the incumbent's by more than the standard error of the difference. The two are scored on the same free
+// reflections, so the difference is paired: R-free is a sum of one term per free twin group (FreeRTerms), and the
+// standard error is that of the sum of the groups' differences.
+// As PDB-REDO decides on more than one free measure, Joosten, Long, Murshudov & Perrakis (2014) IUCrJ 1, 213-220
+bool FreeSetPrefers(const std::vector &r_terms, double nll_free, const std::vector &r_terms_b,
+ double nll_free_b) {
+ const size_t n = r_terms.size();
+ double sum = 0, sum2 = 0;
+ for (size_t k = 0; k < n; k++) {
+ const double d = r_terms[k] - r_terms_b[k];
+ sum += d;
+ sum2 += d * d;
+ }
+ const double se = n > 1 ? std::sqrt(std::max(sum2 - sum * sum / n, 0.0) * n / (n - 1)) : 0.0;
+ return nll_free < nll_free_b && sum < se;
+}
+// The upper alpha point of the F distribution with d1 and d2 degrees of freedom, by Paulson's normal approximation of
+// F^(1/3) (z the normal deviate of alpha) - accurate to a few percent at the counts a displacement model has.
+// Following Paulson (1942) Ann. Math. Statist. 13, 233-235
+double FQuantile(double d1, double d2, double z) {
+ const double u = 2.0 / (9.0 * d1), v = 2.0 / (9.0 * d2), a = 1.0 - v, c = 1.0 - u;
+ const double q = a * a - z * z * v;
+ const double x = (a * c + std::sqrt(std::max(a * a * c * c - q * (c * c - z * z * u), 0.0))) / q;
+ return x * x * x;
+}
+
+// Of two displacement models, `large` with more parameters than `small`: the fraction of Hamilton's R-ratio tests it
+// passes over the range of weights its restraints may be credited at - R_small / R_large above
+// sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the working reflections, m the larger model's displacement parameters
+// (the trials hold the positions), b the extra ones. Each model's restraints are credited as w R observations, w from
+// 0 up to the bound at which they take every parameter they restrain off its count - Merritt's N_atoms / R for a B
+// per atom (as his 3 N_atoms / R for the positions), 6 N_atoms / R for a U per atom; the two weights are scanned
+// independently, six values each.
+// Following Hamilton (1965) Acta Cryst. 18, 502-510, as Merritt (2012) Acta Cryst. D68, 468-477 and PDB-REDO
+// (Joosten et al. 2012 Acta Cryst. D68, 484-496) apply it to the displacement model
+double HamiltonPassFraction(const RefinementWeightTrial &large, const RefinementWeightTrial &small, int n_work) {
+ int pass = 0, tests = 0;
+ for (int i = 0; i <= 5; i++)
+ for (int j = 0; j <= 5; j++, tests++) {
+ const double m_small = small.adp_parameters - small.adp_restrained * i / 5.0;
+ const double m_large = large.adp_parameters - large.adp_restrained * j / 5.0;
+ const double b = m_large - m_small, dof = n_work - m_large;
+ pass += b <= 0 || (dof > 0 && small.r_work / large.r_work > std::sqrt(b / dof * FQuantile(b, dof, 1.6449) + 1.0));
+ }
+ return static_cast(pass) / tests;
+}
+// The displacement model: of the larger and the smaller, the larger where it passes more than this fraction of the
+// tests (HamiltonPassFraction), the smaller where it passes fewer than the other, and the free set's choice between
+// (as PDB-REDO; Joosten et al. 2012).
+constexpr double HAMILTON_LARGER = 0.95, HAMILTON_SMALLER = 0.30;
+// A U per atom is tried only above this many working reflections per refinable atom (as PDB-REDO; Joosten et al. 2012).
+constexpr double U_MIN_REFLECTIONS_PER_ATOM = 18;
+// Every other step is taken on the work-set target (WorkTarget) where it lowers it by more than this, relative.
+constexpr double TARGET_TOLERANCE = 1e-4;
// The rounds of RefineOnce's weight and displacement-model choice (4b.): the first from the model as given, each
// after it from the model the one before ended with.
// The minimiser (RefinementMinimiser): a solve stops where an iteration lowers the target by less than this,
@@ -1564,9 +1618,10 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
if (body.size() >= 50)
bodies.push_back(body);
}
- // Each step is kept only where it lowers R-free beyond the tie: a deposited model was refined atom by atom
- // already, and moving its chains or shifting its B factors residue by residue then only makes it worse.
- RFactors before = ctx.Refit();
+ // Each step is kept only where it lowers the -ln L of the working reflections, the target it minimised (the free
+ // set decides the weights and the displacement model alone).
+ ctx.Refit();
+ double before = ctx.Score().nll_work;
if (!bodies.empty()) {
const std::vector pos0 = ctx.Positions();
std::vector q(6 * bodies.size(), 0.0);
@@ -1577,13 +1632,15 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
const CancelCheck cancel_check(ctx);
ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary);
rb->Place(q.data());
- const RFactors after = ctx.Refit();
- if (after.r_free < before.r_free - R_FREE_TIE) {
+ ctx.Refit();
+ const double after = ctx.Score().nll_work;
+ if (after < before) {
result.rigid_bodies = static_cast(bodies.size());
before = after;
} else {
ctx.SetPositions(pos0);
- before = ctx.Refit();
+ ctx.Refit();
+ before = ctx.Score().nll_work;
}
}
if (group_b) {
@@ -1596,7 +1653,8 @@ void RefinePlacement(Context &ctx, bool group_b, RefinementResult &result) {
const CancelCheck cancel_check(ctx);
ceres::Solve(cancel_check.Options(30), problem, q.data(), &summary);
gb->Place(q.data());
- result.group_b = ctx.Refit().r_free < before.r_free - R_FREE_TIE;
+ ctx.Refit();
+ result.group_b = ctx.Score().nll_work < before;
if (!result.group_b)
ctx.SetB(b0);
}
@@ -1897,6 +1955,39 @@ std::string DomainText(const RefinementReflections &refl, const XrayRowModel &ro
// As REFMAC5 drops twin domains of small fraction, Murshudov et al. (2011) Acta Cryst. D67, 355-367
constexpr double TWIN_DOMAIN_MIN = 0.05;
+// Each observed reflection's twin group, named by its lowest observed row (itself where there is no twin law).
+std::vector TwinGroups(const RefinementReflections &refl) {
+ const size_t n = refl.Size();
+ std::vector group(n);
+ for (size_t m = 0; m < n; m++) {
+ group[m] = static_cast(m);
+ for (size_t d = 0; d < refl.Laws(); d++)
+ if (refl.mates[d][m] < static_cast(n))
+ group[m] = std::min(group[m], refl.mates[d][m]);
+ }
+ return group;
+}
+
+// R-free of the model as the last Fcalc left it, as one term per free twin group, in the order of the groups (the
+// members of a group share their F_calc, so their terms are summed into one, as TwinNull sums them): the terms
+// FreeSetPrefers pairs.
+std::vector FreeRTerms(const Context &ctx) {
+ const std::vector> fmodel = TwinnedModel(ctx.refl, ctx.rows, ctx.fcalc);
+ const std::vector group = TwinGroups(ctx.refl);
+ double fo = 0;
+ std::vector sum(ctx.refl.Size(), 0.0);
+ for (size_t m = 0; m < ctx.refl.Size(); m++)
+ if (ctx.refl.free[m]) {
+ fo += ctx.refl.fo[m];
+ sum[group[m]] += std::fabs(ctx.refl.fo[m] - std::abs(fmodel[m]));
+ }
+ std::vector terms;
+ for (size_t m = 0; m < ctx.refl.Size(); m++)
+ if (ctx.refl.free[m] && group[m] == static_cast(m))
+ terms.push_back(static_cast(sum[m] / fo));
+ return terms;
+}
+
// The null of a twinned refinement, for the model as the last Refit() left it: its R-free at the same fraction with
// every reflection's mate replaced by an unrelated observed one of the same resolution and the same set (free with
// free, work with work, as a law's twin pairs are). The reflections of each set are put in order of 1/d^2 with the two
@@ -1920,14 +2011,7 @@ constexpr double TWIN_STOP_Z = 0.5;
void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) {
RefinementReflections null = ctx.refl;
const size_t n = null.Size();
- // each observed reflection's twin group, named by its lowest observed row
- std::vector group(n);
- for (size_t m = 0; m < n; m++) {
- group[m] = static_cast(m);
- for (size_t d = 0; d < ctx.refl.Laws(); d++)
- if (ctx.refl.mates[d][m] < static_cast(n))
- group[m] = std::min(group[m], ctx.refl.mates[d][m]);
- }
+ const std::vector group = TwinGroups(ctx.refl);
for (size_t d = first; d < null.Laws(); d++)
std::iota(null.mates[d].begin(), null.mates[d].end(), 0);
for (const bool free : {false, true}) {
@@ -2246,6 +2330,26 @@ struct TrialOutcome {
int evaluations = 0;
};
+// The work-set target's terms of the model as it stands (RefinementWeightTrial), at the last Refit()'s fit: what every
+// decision but the weights' and the displacement model's is made on (WorkTarget).
+void MeasureWorkTarget(Context &ctx, const RefinementTopology &topo, const NcsCopies *ncs, double ncs_sigma,
+ RefinementWeightTrial &t) {
+ const RefinementScore sc = ctx.Score();
+ t.nll_work = sc.nll_work;
+ t.nll_free = sc.nll_free;
+ t.free_r_terms = FreeRTerms(ctx);
+ const std::vector> xyzb = ctx.XyzB();
+ t.e_geometry = EvaluateRestraints(RestraintsNow(ctx, topo, ncs, ncs_sigma), xyzb, nullptr);
+ t.e_b = EvaluateRestraints(BRestraintsNow(ctx, topo), xyzb, nullptr);
+}
+
+// The work-set target of a trial at the X-ray weights w (positions) and adp (B, 0: held), its NCS restraints at its own
+// sigma: what a trial at those weights minimises, the X-ray term of the positions and the geometry, and that of the B
+// factors and the B similarity where they are refined.
+double WorkTarget(const RefinementWeightTrial &t, double w, double adp) {
+ return w * t.nll_work + t.e_geometry + (adp > 0 ? adp * t.nll_work + t.e_b : 0.0);
+}
+
// The X-ray target's gradient on the positions, its squared norm over `refinable`, for the atoms as they are.
double XrayGradientNorm(Context &ctx, const std::vector &refinable) {
std::vector> g;
@@ -2471,16 +2575,25 @@ TrialOutcome RunTrial(TrialSlot slot, XyzCalibrations &calibrations, const std::
t.ncs_rmsd = NcsRmsd(*ncs, ctx.XyzB());
if (hyp != nullptr)
t.adp = hyp->name;
- t.nll_free = ctx.Score().nll_free;
- out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, rmsd bonds {:.4f} A, "
- "angles {:.2f} deg{}{}",
+ // the displacement parameters: a B per refinable atom, the TLS bodies' 20 each beside it, or a U per atom
+ int atoms_b = 0;
+ for (int i : refinable)
+ atoms_b += !ctx.atoms[i]->is_hydrogen();
+ t.adp_parameters = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast(u_atoms.size())
+ : kind == AdpHypothesis::Kind::Tls ? atoms_b + 20 * static_cast(hyp->bodies.size())
+ : atoms_b;
+ // of them the restrained ones: the B or the U of every atom (not the TLS bodies')
+ t.adp_restrained = kind == AdpHypothesis::Kind::Anisotropic ? 6 * static_cast(u_atoms.size()) : atoms_b;
+ MeasureWorkTarget(ctx, topo, ncs, ncs_sigma, t);
+ out.line = fmt::format("{}X-ray weight {:.3g}, {} {}{}: R-work {:.4f}, R-free {:.4f}{}, target {:.6g}, rmsd bonds "
+ "{:.4f} A, angles {:.2f} deg{}{}",
std::string(tw.blur.empty() ? "" : "blurred, ") + (hyp != nullptr ? hyp->name + ", " : ""), w,
kind == AdpHypothesis::Kind::Anisotropic ? "U" : kind == AdpHypothesis::Kind::Tls ? "residual B" : "B",
adp_w > 0 ? fmt::format("refined at weight {:.3g}", adp_w) : std::string("held"),
ncs == nullptr || ncs->Empty() ? std::string()
: ncs_sigma > 0 ? fmt::format(", NCS at {:.3g} A (copies {:.3f} A apart)", ncs_sigma, t.ncs_rmsd)
: fmt::format(", no NCS (copies {:.3f} A apart)", t.ncs_rmsd),
- rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free),
+ rf.r_work, rf.r_free, fmt::format(", -LL free {:.1f}", t.nll_free), WorkTarget(t, w, adp_w),
t.geometry.rmsd_bonds, t.geometry.rmsd_angles,
t.geometry.metal_bonds > 0 ? fmt::format(", metal bonds {:.3f} A", t.geometry.rmsd_metal)
: std::string(),
@@ -2859,8 +2972,8 @@ double BalancedUWeight(Context &ctx, const URestraints &restraints, const std::v
// The displacements of the model as given, read where it has TLS groups whose atoms bring no U of their own: their U
// from the groups, its B column taken as the total B (the U's equivalent isotropic B, as phenix.refine writes it) or as
-// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - the one of the three whose
-// R-free is lowest, beyond the tie in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens
+// the residual to the groups (as REFMAC writes it), and the isotropic B as it stands - a displacement model, chosen on
+// the free set (FreeSetPrefers), in that order. `tls`: the groups, their atoms those of ctx.atoms but the hydrogens
// (which ride on their parents' B). Returns the R-factors of the reading kept.
// Following Winn, Isupov & Murshudov (2001) Acta Cryst. D57, 122-133 (the B column total or residual to the TLS)
RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, Logger &logger, RefinementResult &result) {
@@ -2906,6 +3019,8 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, L
readings.push_back({result.adp_input == "ANISOU" ? "ANISOU" : "isotropic", b0, aniso0});
int best = -1;
std::vector rf;
+ std::vector nll_free;
+ std::vector> r_terms;
for (size_t k = 0; k < readings.size(); k++) {
ctx.SetB(readings[k].b);
ctx.SetAniso(readings[k].aniso);
@@ -2914,9 +3029,11 @@ RFactors ReadInputDisplacements(Context &ctx, const std::vector &tls, L
ctx.atoms[i]->b_iso = ctx.atoms[ctx.rider_parent[i]]->b_iso;
ctx.ForgetStarts();
rf.push_back(ctx.Refit());
- logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}", tls.size(),
- tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free);
- if (best < 0 || rf[k].r_free < rf[best].r_free - R_FREE_TIE)
+ nll_free.push_back(ctx.Score().nll_free);
+ r_terms.push_back(FreeRTerms(ctx));
+ logger.Info("Refinement: the model's {} TLS groups ({} atoms) read as {}: R-work {:.4f}, R-free {:.4f}, -LL free "
+ "{:.1f}", tls.size(), tls_u.size(), readings[k].name, rf[k].r_work, rf[k].r_free, nll_free[k]);
+ if (best < 0 || FreeSetPrefers(r_terms[k], nll_free[k], r_terms[best], nll_free[best]))
best = static_cast(k);
}
ctx.SetB(readings[best].b);
@@ -3161,6 +3278,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector start_r_terms = FreeRTerms(ctx);
logger.Info("Refinement{}: {} atoms ({} riding hydrogens), {} reflections to {:.2f} A ({} free), on the {}; as "
"given R-work {:.4f}, R-free {:.4f}; k_sol {:.3f}, b_sol {:.1f}, overall B {:.1f}, Wilson B {:.0f}, B "
"ceiling {:.0f}, mask grid {:.2f} A",
@@ -3178,6 +3296,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector *groups = ctx.st.meta.get_tls_groups())
@@ -3426,10 +3545,24 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vectorran && refined->trial.within_bounds &&
- (!held->trial.within_bounds || refined->trial.r_free < held->trial.r_free + 2 * R_FREE_TIE));
+ (refined->ran && refined->trial.within_bounds && !displaces(held->trial, refined->trial));
auto rest_b = [&](const TrialWeights &l) { return l.ncs == 0 && !first(l); };
if (refine_b && !all_ahead)
run_ladder(rest_b);
@@ -3547,37 +3678,29 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector from_best;
// `fresh`: the first step fits the scale, the solvent and the likelihood anew (the model changed since).
// `label`: the steps' name in the log.
auto trajectory = [&](TrialOutcome state, bool fresh, std::vector &from_best, const char *label) {
- TrialOutcome best = state;
from_best.clear();
- bool state_is_best = true;
- // A step taken on R-work alone is followed by one more at most: on a model refined before, R-work falls
- // step after step while R-free does not, and the trajectory ends at its best anyway - a model of 490k
- // atoms spent six such steps (240 s) on a round it then gave back. (On 28 refinements of 22 deposited
- // models and 6 predictions the rule changed nothing.)
- int since_best = 0;
- for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled() && since_best <= 1; step++) {
+ for (int step = 0; step < settings.max_steps && state.ran && !ctx.Cancelled(); step++) {
const double w = state.trial.weight, adp = state.trial.adp_weight, ncs_s = state.trial.ncs_sigma;
std::vector trials = {{w, adp, ncs_s}};
if (step % 2 == 1 && adp > 0) {
trials.push_back({w, adp / 4, ncs_s});
trials.push_back({w, 4 * adp, ncs_s});
} else {
- if (step == 0)
- trials.push_back({w / 2, adp, ncs_s});
+ trials.push_back({w / 2, adp, ncs_s});
if (2 * w <= w_max)
trials.push_back({2 * w, adp, ncs_s});
}
@@ -3588,27 +3711,18 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector outcomes = RunTrials(trial_slots, trials, refinable, state.pos, state.b, state.aniso,
fresh && step == 0 ? nullptr : &state.fit,
settings.macro_cycles, settings, ncs_p);
- if (state_is_best)
- from_best = outcomes;
+ from_best = outcomes;
const int k = choose(outcomes, fmt::format("{} {}", label, step + 1).c_str());
if (k < 0)
break;
- const RefinementWeightTrial &t = outcomes[k].trial;
// after a change of the model the first step is taken whatever it gains: it is where the change is
// judged from
- const bool gains = (fresh && step == 0) || displaces(t, state.trial) ||
- (t.within_bounds && t.r_work < state.trial.r_work - R_FREE_TIE);
- if (!gains || !((fresh && step == 0) || (t.within_bounds && t.r_free < best.trial.r_free + R_FREE_TIE)))
+ if (!((fresh && step == 0) || advances(outcomes[k].trial, state.trial)))
break;
state = std::move(outcomes[k]);
- state_is_best = (fresh && step == 0) || displaces(state.trial, best.trial);
- since_best = state_is_best ? 0 : since_best + 1;
- if (state_is_best) {
- best = state;
- from_best.clear();
- }
+ from_best.clear();
}
- return best;
+ return state;
};
if (!stopped)
state = trajectory(std::move(state), false, from_best, "step");
@@ -3620,9 +3734,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(std::max(settings.population, 0), trial_slots.size());
@@ -3651,7 +3764,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(k);
}
if (best >= 0) {
@@ -3686,11 +3800,11 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(i));
TrialOutcome after = one_step(moved);
- const bool kept = after.ran && after.trial.within_bounds && after.trial.r_free < unchanged.trial.r_free &&
- after.trial.nll_free < unchanged.trial.nll_free;
- logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}; from the model as it "
- "was R-free {:.4f}, -LL free {:.1f}: {}", after.trial.r_free, after.trial.nll_free,
- unchanged.trial.r_free, unchanged.trial.nll_free, kept ? "kept" : "not kept");
+ const bool kept = after.ran && unchanged.ran && advances(after.trial, unchanged.trial);
+ logger.Info("Side-chain field: refined from the field R-free {:.4f}, -LL free {:.1f}, target {:.6g}; from "
+ "the model as it was R-free {:.4f}, -LL free {:.1f}, target {:.6g}: {}", after.trial.r_free,
+ after.trial.nll_free, WorkTarget(after.trial, after.trial.weight, after.trial.adp_weight),
+ unchanged.trial.r_free, unchanged.trial.nll_free,
+ WorkTarget(unchanged.trial, after.trial.weight, after.trial.adp_weight), kept ? "kept" : "not kept");
if (kept) {
result.side_chain_field_kept = true;
result.side_chain_field_moved = static_cast(field.moves.size()) - pruned;
@@ -4132,8 +4256,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector trials;
@@ -4156,21 +4280,21 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector= 0 && displaces(outcomes[k].trial, state.trial))
+ if (k >= 0 && advances(outcomes[k].trial, state.trial))
state = std::move(outcomes[k]);
}
// 4. The displacement model, decided late, on the free set: from where the weights ended, TLS bodies (the
// model's own groups, one per chain, the chains in TLSMD segments) with a residual B, and a U per atom at
// its balanced weight and a quarter of it - each trial as long as a step, on the positions where the weights
// ended, held, so that the trials differ in the displacement model alone; the model chosen then takes a
- // step with the positions refined, kept where it displaces its trial (against trials a macro-cycle longer
+ // step with the positions refined, kept where it advances on the work-set target from its trial (against trials a macro-cycle longer
// that refined the positions too, on 50 refinements of the battery R-free 0.001 lower on average and none
// higher by more than 0.0015, at 78% of the evaluations). Where the geometry is out of bounds there, which
// held positions cannot mend, each trial refines them too, a macro-cycle longer than a step, and no step
// follows. One displaces the one before it (the first
- // where the weights ended, the B factors as they are) only where its geometry is within bounds and both its
- // R-free (beyond the tie) and the -ln L of the free reflections are lower: the free set is the test of the
- // parameters it adds.
+ // where the weights ended, the B factors as they are) only where its geometry is within bounds and the free
+ // set prefers it (FreeSetPrefers: its -ln L of the free reflections lower, its R-free not higher beyond its
+ // standard error): the free set is the test of the parameters it adds.
// Following Merritt (2012) Acta Cryst. D68, 468-477 (the displacement model the data support, decided by test)
// The trials that would not change the choice are left out: the model's own TLS groups are tried only where
// one body per chain or the finest cut lowers R-free, the cuts between those only where also the finest
@@ -4190,9 +4314,14 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector hyps = AdpHypotheses(ctx, topo, own_tls);
const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0;
+ // a U per atom only where the data are many times its parameters
+ int atoms_b = 0;
+ for (int i : refinable)
+ atoms_b += !ctx.atoms[i]->is_hydrogen();
+ const bool u_tried = result.n_work > U_MIN_REFLECTIONS_PER_ATOM * atoms_b;
// the U weight balanced on the model with every refinable atom given a U
double u_w0 = adp;
- {
+ if (u_tried) {
const std::vector keep = ctx.aniso;
std::vector all = keep.empty() ? std::vector(ctx.atoms.size()) : keep;
std::vector refines_u(ctx.atoms.size(), 0);
@@ -4215,6 +4344,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector(trials.size());
for (double f : {0.25, 1.0}) {
trials.push_back({w, u_w0 * f, ncs_s});
@@ -4298,20 +4429,34 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector b.adp_parameters;
+ const double pass = larger ? HamiltonPassFraction(t, b, result.n_work)
+ : HamiltonPassFraction(b, t, result.n_work);
+ bool takes = displaces(t, b);
+ if (t.adp_parameters != b.adp_parameters && (pass > HAMILTON_LARGER || pass < HAMILTON_SMALLER))
+ takes = t.within_bounds && (!b.within_bounds || (pass > HAMILTON_LARGER) == larger);
+ logger.Info("Refinement, displacements: {} against {}: the larger passes {:.0f}% of Hamilton's tests{}",
+ t.adp.empty() ? "the B factors as they are" : t.adp,
+ b.adp.empty() ? "the B factors as they are" : b.adp, 100 * pass, takes ? ", taken" : "");
+ if (takes)
best = static_cast(k);
}
- // (the model where the weights ended keeps the displacement model a round before chose for it)
- if (result.adp_model.empty() || state.trial.adp.empty())
+ // (the model where the weights ended keeps the displacement model a round before chose for it, its U held by
+ // the weight trials since)
+ if (result.adp_model.empty())
result.adp_model = result.adp_input == "isotropic" ? "isotropic" : "the model's U held";
+ else if (state.trial.adp.empty() && result.adp_model != "isotropic" && !result.adp_model.ends_with("held"))
+ result.adp_model += ", U held";
if (best >= 0) {
result.adp_model = outcomes[best].trial.adp;
state = std::move(outcomes[best]);
}
if (best >= 0 && hold) {
// the model chosen refined on with the positions, a step from where its trial ended, kept where it
- // displaces that
+ // advances on the work-set target from there
AdpHypothesis chosen = *which[best];
chosen.bodies = state.tls;
chosen.tls_b = state.tls_b;
@@ -4320,7 +4465,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector own_tls = state.tls;
@@ -4402,23 +4554,25 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector {:.4f}{}", round, before.trial.r_free, state.trial.r_free,
- kept ? "" : ", not kept");
+ const bool kept = advances(state.trial, before.trial);
+ logger.Info("Refinement, round {}: R-free {:.4f} -> {:.4f}, target {:.6g} -> {:.6g}{}", round,
+ before.trial.r_free, state.trial.r_free,
+ WorkTarget(before.trial, state.trial.weight, state.trial.adp_weight),
+ WorkTarget(state.trial, state.trial.weight, state.trial.adp_weight), kept ? "" : ", not kept");
if (!kept) {
state = before;
result.adp_model = adp_before;
break;
}
}
- // 5. The model as given, first in order of trust: the refinement is kept where it displaces it by the rule every
- // trial is held to, else the model is given back as it came (the solvent build's edits undone), so that what
- // comes out is never worse on the free set than what went in - where the free set is free to the model as
- // given (RefinementSettings::as_given_competes).
- if (settings.as_given_competes && state.ran && !ctx.Cancelled() && !displaces(state.trial, given.trial)) {
- logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}), the refinement did not "
- "displace it (R-work {:.4f}, R-free {:.4f})", given.trial.r_work, given.trial.r_free,
- state.trial.r_work, state.trial.r_free);
+ // 5. The model as given: where it is kept over the refinement (given_kept), it is given back as it came (the
+ // solvent build's edits undone), so that what comes out is never worse on both free measures than what went
+ // in - where the free set is free to the model as given (RefinementSettings::as_given_competes).
+ if (settings.as_given_competes && state.ran && !ctx.Cancelled() && given_kept()) {
+ logger.Info("Refinement: the model as given kept (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f}), the "
+ "refinement did not displace it (R-work {:.4f}, R-free {:.4f}, -LL free {:.1f})",
+ given.trial.r_work, given.trial.r_free, given.trial.nll_free, state.trial.r_work,
+ state.trial.r_free, state.trial.nll_free);
size_t a = 0, k = 0;
for (gemmi::Chain &ch : ctx.Model().chains)
for (gemmi::Residue &res : ch.residues) {
diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h
index 5dfece5f1..e7a1073a3 100644
--- a/image_analysis/structure_refinement/Refinement.h
+++ b/image_analysis/structure_refinement/Refinement.h
@@ -160,6 +160,12 @@ struct RefinementWeightTrial {
double ncs_rmsd = NAN; // the copies' C-alpha (P) rmsd after superposition, where there are copies
double r_work = 1, r_free = 1;
double nll_free = NAN; // -ln L of the free reflections
+ // The work-set target's terms where the trial ended (RefineOnce's WorkTarget): -ln L of the working reflections,
+ // the restraints' sum of the geometry (with the NCS restraints at ncs_sigma) and that of the B similarity.
+ double nll_work = NAN, e_geometry = NAN, e_b = NAN;
+ int adp_parameters = 0; // the displacement model's parameters (HamiltonPassFraction)
+ int adp_restrained = 0; // of them those the restraints act on, which HamiltonPassFraction credits
+ std::vector free_r_terms; // R-free as one term per free twin group (FreeRTerms), for paired comparisons
GeometryStatistics geometry;
bool within_bounds = false; // the geometry bounds the choice is made inside
std::string adp; // the displacement model: "isotropic", "TLS ..." or "anisotropic"