* jfjoch_broker (experimental): Serial data are merged live, with intermediate results during the collection. * Rugnux: More robust processing of serial data, including CCD detectors. * Rugnux: Parts of a rotation sweep that spoil the merge are removed automatically. * Rugnux: Clearer maps, and anomalous peaks not explained by the model are reported. * Rugnux: SHELX files work with SHELXL; XFEL pulse IDs are kept in `_process.h5`. * Web frontend: Spot-finding settings change only when the user changes them. * Rugnux: Faster processing. Reviewed-on: #94 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
66 KiB
Data analysis: refinement of a placed model (§15)
Part of the CPU/GPU data-analysis reference; the section numbers are continuous across its parts.
:local:
:depth: 2
15. Restrained refinement of a placed model
Wherever a model is placed - it fits where it is (§14.8), or molecular replacement placed it (§14.10) -
rugnux refines it against the merge it was placed on (--no-refine skips this). rugnux --mode refine
runs the same refinement on a model and merged structure factors that already exist. How to use it, and
what it writes, is in Models and reference data; its report
keys are in The results report.
The refinement comes after every processing decision and changes none of them. Nothing it finds is
read by the space-group search, the indexing choice, the resolution cut or the files <prefix>.mtz,
<prefix>_model.cif and <prefix>_maps.mtz; R_WORK and R_FREE stay those of the model as placed
(§14.3). What it produces is written beside them: the refined model, its map coefficients, the blobs of its
difference map and the REFINE_ keys.
The library reads and writes no file. The model and the reflections come in, the refined model, its statistics and its map coefficients go out; the restraint dictionary is compiled into the binary (§15.4).
15.1 What is refined, and in what order
The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed,
and every B clamped to the range the refinement allows (2 Ų up to twice the model's largest B, within
100-300 Ų). An atom's anisotropic U (ANISOU, _atom_site_anisotrop) is kept. Where the model has TLS
groups (_pdbx_refine_tls*, PDB REMARK 3) whose atoms bring no U of their own, their U is made from the
groups, and the B column is read either as the total B (as phenix.refine writes it) or as the residual to the
groups (as REFMAC writes it) - whichever of the two, and of the isotropic B as it stands, scores the lowest
R-free on the model as given (REFINE_ADP_INPUT). Occupancies are kept as they are, and
so are alternative conformations, each refined with its own restraints.
Two stages follow.
- Placement. Each chain of at least 50 non-water atoms is moved as a rigid body - an angle-axis rotation about its centroid, scaled by its rms radius so that a unit of rotation moves a typical atom by about an angstrom, and a translation - against the X-ray target (§15.2), by Ceres' L-BFGS for 30 iterations. Then one B shift per residue. Each of the two is kept only where it lowers R-free by more than 0.001: a deposited model was refined atom by atom already, and moving its chains or shifting its B factors residue by residue then only makes it worse. The placement stage needs no dictionary.
- Restrained refinement. Every atom's position and isotropic B, against the X-ray target and the geometry of the dictionary (§15.4-§15.5), in weight trials chosen by R-free (§15.7). Residues the dictionary does not describe, and residues with an atom it does not know, are held fixed: they stay in the structure factors, in the bulk-solvent mask and in the repulsions the refined atoms see, but they do not move. So do atoms of zero occupancy.
15.2 The X-ray target
The target is the maximum-likelihood amplitude target: for an acentric reflection the Rice distribution of
the observed amplitude F_o given the model's |F_\mathrm{model}|, for a centric one its centric
counterpart,
-\ln L_\mathrm{acentric} = \ln\Sigma + \frac{F_o^2 + D^2|F_\mathrm{model}|^2}{\Sigma} - \ln I_0\!\left(\frac{2F_o D|F_\mathrm{model}|}{\Sigma}\right),\qquad
-\ln L_\mathrm{centric} = \tfrac12\ln\Sigma + \frac{F_o^2 + D^2|F_\mathrm{model}|^2}{2\Sigma} - \ln\cosh\frac{F_o D|F_\mathrm{model}|}{\Sigma},
(the terms that depend on neither the model nor the parameters left out) with D the fraction of the model that is right and \Sigma the variance that the rest of the structure and
the measurement leave (§15.3), summed over the working reflections. \ln I_0 and I_1/I_0 are evaluated
with the polynomial approximations of Abramowitz and Stegun, through the exponentially scaled form for large
arguments so that nothing overflows. The amplitudes are the French-Wilson |F| of §10.8 - the ones the
written reflection file carries - and the free reflections are those of §10.7.
F_\mathrm{model} is the model of §14.2,
F_\mathrm{model} = k\,e^{-\mathbf{s}^\top\mathbf{B}\mathbf{s}/4}\left(F_\mathrm{calc} + k_\mathrm{sol}\,e^{-B_\mathrm{sol}s^2/4}\,F_\mathrm{mask}\right),
with the atoms' F_\mathrm{calc} computed with the dictionary's riding hydrogens included (§15.4), from the IT92
form factors: a single-atom ion (a metal or a halide the dictionary describes) as the ion its dictionary charge
makes it - Mg²⁺ and Na⁺ 10 electrons, K⁺, Ca²⁺ and Cl⁻ 18 - every other atom as the neutral atom, whatever formal
charge the file gives it (a carboxylate's O⁻, a lysine's N⁺); the refined model is written with those charges. The
scale, the anisotropic B, the bulk solvent and the likelihood's parameters are fitted at the start of each
macro-cycle (§15.3) and held fixed while the minimiser runs; the mask is drawn for the model as it is
at that point.
The gradient by the FFT method. The atoms' density is placed on a grid with a Gaussian blur (the one
REFMAC-compatible density calculation uses, chosen for the lowest B a macro-cycle can reach) and transformed
once; the crystal's symmetry is composed in reciprocal space from the transform of one copy, as for the rigid
body (§14.8). The derivative of the target with respect to every structure factor, taken back to real space
by one inverse transform through the adjoint of that composition, is a map; each atom's derivatives with
respect to x, y, z and B are that map summed against the derivatives of its own blurred density. One
evaluation of the target and of the gradient of every atom thus costs a density gather, two FFTs and a
gradient gather, whatever the number of atoms. The derivative of a riding hydrogen is passed to its parent.
15.3 Scale, bulk solvent and the likelihood parameters
Scale and solvent. The flat bulk-solvent mask is drawn with the REFMAC radii (probe 1.0 Å, shrink 0.8 Å)
on a grid of its own, with a spacing of d_\mathrm{min}/3 or 0.4 Å, whichever is finer (coarser only where
a cell would need more than 2^{26} points, and then never coarser than 0.75 Å nor than the density grid,
which a cell that large uses for its mask). On the atoms' density grid the shrink is lost at low
resolution: at 3 Å its 1 Å step leaves no grid point within the shrink radius. k_\mathrm{sol} and
B_\mathrm{sol} are searched over their physical range on the working reflections, and the overall scale
and anisotropic B are fitted at each point of that search (§14.2) - at the points of its fine pass by
robust least squares:
iteratively reweighted (four reweightings), each reflection's residual r = F_o - |F_\mathrm{model}|
weighted by 1/\sqrt{1 + (r/c)^2} with c the median |r| of the unweighted fit. The fit then minimises
the pseudo-Huber loss \sum 2c^2\left(\sqrt{1 + (r/c)^2} - 1\right), quadratic for the bulk of the
reflections and linear for the few a model cannot explain - a beam-stop shadow, an ice ring, a missing part
of the model - which ordinary least squares would let set the scale of all the rest. When the B factors
are refined, the overall isotropic part of the scale's B is moved into the atoms (and into B_\mathrm{sol},
so that F_\mathrm{model} stays as it is); left in the scale, it and the atoms' B drift against each other
as the B factors are refined.
With these three choices - riding hydrogens, the fine mask grid and the robust scale - the R-work and
R-free of a model as given (REFINE_R_WORK_START, REFINE_R_FREE_START) agree with REFMAC 5.8's 0-cycle
values on the same model and data within 0.002 on twelve of fourteen depositions from 1.1 to 3.6 Å, and
within 0.006 on the other two (tools/battery/refine_parity.py).
D and \Sigma. The two parameters of the likelihood are fitted by maximum likelihood in resolution
bins of the free set (100 free reflections to a bin, at most 30 bins; where the free set has fewer than
100 reflections, the working ones join it so that the likelihood still has a scale), and interpolated
linearly in 1/d^2 between the bins' centres:
\betaon the free reflections, as the whole spread ofF_oaboutD F_\mathrm{model}with variance\varepsilon\beta- the measurement error included;Dthen on the working reflections of the bin, with the variance\Sigma = \varepsilon\beta + 2\sigma^2for an acentric reflection and\varepsilon\beta + \sigma^2for a centric one,\sigmathe amplitude's own error.
\Sigma is also the variance of each reflection in the target. Where the measurement error is a large part
of the spread - the weak high-resolution shells - it is counted twice, and D falls towards 0 there, with
the figure of merit. This division of the two parameters between the working and the free set is the one
REFMAC5 describes; measured on REFMAC 5.8's own output it reproduces its D by bin and its overall figure of
merit (the mean over the working reflections) to about 0.01 on most depositions.
15.4 The dictionary and the topology
The restraints are those of the CCP4 Monomer Library, a subset of which is compiled into rugnux
(image_analysis/structure_refinement/monomers/, regenerated by update_monomers.py; nothing is read from
disk, and $CLIBD is never consulted): the 20 standard amino acids and MSE; the DNA and RNA nucleotides
(DA, DC, DG, DT, A, C, G, U); the modified nucleotides of tRNA and rRNA met most often - pseudouridine (PSU),
5-methylcytidine (5MC), 1-methyladenosine (1MA), the 2'-O-methyl nucleotides OMG, OMC and OMU, dihydrouridine
(H2U), N2-methyl- and N2,N2-dimethylguanosine (2MG, M2G), 7-methylguanosine (7MG) and ribothymidine (5MU), polymer
residues joined by the same backbone link; ATP, ADP, AMP, GTP, GDP and GMP; the cofactors NAD⁺ (NAD), NADH (NAI),
NADP⁺ (NAP), NADPH (NDP), FAD, FMN, heme (HEM), heme C (HEC), S-adenosylmethionine (SAM) and
S-adenosylhomocysteine (SAH), and dioxygen (OXY); DMSO; the PEG fragments PEG, PGE, PG4 and 1PE; glycerol,
ethylene glycol, MPD, acetate, sulfate and phosphate; water; and the ions Na, K, Mg, Ca, Mn, Fe²⁺, Fe³⁺, Co, Ni,
Cu, Zn, Cd, Cl, Br and I - 84 compounds - with the library's links and modifications that apply to them (among
them heme C's two thioethers to its cysteines, HEC-CYS1 and HEC-CYS2, and the covalent links of heme and FAD to
the protein) and the energy types those compounds use.
The model's topology is built from it by GEMMI (prepare_topology): the links between residues (the peptide
and cis-peptide, the nucleic-acid backbone, disulfides, and the model's LINK records where the library
describes them), the modifications those links imply, and per residue the bonds, angles, torsions, chiral
volumes and planes. Torsions are restrained where the dictionary gives them an esd, except the main-chain
φ/ψ and the sugar puckers it lists as alternatives. The dictionary's hydrogens are added to every
residue it describes (waters excepted), placed by GEMMI from the dictionary's geometry and moved with their
parents: they take part in the structure factors and in no restraint, and are placed again on their
parents at every macro-cycle. The refined model is written with them.
15.5 Geometry, metal and B restraints
Every restraint is a squared normalised deviation, E = ((x - x_\mathrm{ideal})/\sigma)^2, with its
analytic derivatives:
- bonds and angles, ideal and esd from the dictionary;
- torsions, modulo the dictionary's period, with derivatives in the form of Blondel and Karplus, free of the singularities of the arccos form;
- chiral volumes
(\mathbf{a}_1 - \mathbf{c})\cdot((\mathbf{a}_2 - \mathbf{c})\times(\mathbf{a}_3 - \mathbf{c})),\sigma = 0.2ų; - planes: the distances of the atoms from their least-squares plane (the plane's normal the smallest eigenvector of the atoms' scatter, by Jacobi rotations); the plane is optimal, so the derivative of each distance carries no term from the plane moving;
- repulsions of every pair of atoms closer than its contact distance, symmetry mates included, that no
bond, angle or 1-4 relation joins and that are not alternative conformations of each other, with at least
one of the two refinable:
E = ((d_0 - d)/0.2\,\text{Å})^2below the contact distanced_0and nothing above it,d_0the sum of the energy types' van der Waals radii, 2.6 Å where both atoms can hydrogen-bond and 1.9 Å between a metal and N, O or S. The list is rebuilt at every macro-cycle.
Special positions. An atom closer than 0.5 Å to one of its own symmetry images (the depositions put theirs within 0.03 Å of the site) is on a special position: the symmetry operations, each with its lattice translation, that take it onto itself are its site symmetry (Grosse-Kunstleve & Adams 2002). It is placed exactly on the site - the mean of its position and its images under the site symmetry - and its shifts are confined to the subspace those operations leave unchanged: the gradient is projected by the mean of their rotations (along a 2-fold or 4-fold axis, nowhere on a centre of inversion). Its occupancy is kept as given, the coinciding copies each counting it (0.25 on a 4-fold axis), it makes no contact with its own copies, each of its contacts is listed once rather than once per copy, and it moves with no rigid body.
Metal sites. A metal - an ion, or a metal atom of a ligand such as a heme's iron - is restrained to the
donor atoms (O, N or S) that the model's LINK / _struct_conn records bind it to. Where the library has a link for the pair - its ZN-CYS, ZN-HISND,
ZN-HISNE, FE-CYS, FE-HISND and FE-HISNE - that link is used as it is, with the modification it makes to
the donor's residue (the proton the bond replaces). Every other record becomes a link of one bond at the
distance of the library's own statistics of metal sites (its metals.json, carried as monomers/metals.cif)
for the metal, the donor element and the site's coordination number - but only where the model makes the
contact an inner-sphere one: no further than that distance plus 0.33 Å (Mg-O 2.07 + 0.33 = 2.4 Å, the
inner-sphere bound of Leonarski et al. 2017, the same margin for every metal). A record at an outer-sphere
distance - deposited models carry many, a Mg "bonded" to a phosphate O at 2.9 Å, a Na to a water at 3.0 Å -
would pull the ion onto that atom; it stays a contact instead. The coordination number is the number of
records the metal has within its inner sphere, plus the bonds its own residue's dictionary gives it (a heme iron's four pyrrole nitrogens) - interpolated linearly between two tabulated coordination numbers (rows resting on at least 20 sites
count; σ is the row's standard deviation, at least 0.02 Å). Only the bond is restrained, not
the angles at the metal. A contact without a record stays a repulsion, and a record to a symmetry mate is
left out, as every bond to a symmetry mate is. Which records stay contacts is decided once, on the model as given: one topology serves the whole refinement, its
solvent build included, so a record is not judged again on a distance the refinement has moved (a record at 2.50 Å
refused, the ion slid to 2.39 Å, would have been taken). The records are written out with the refined model as they were
read, without the names of the links made here. The rmsd of these bonds - with the dictionary's own bonds of
a metal within a ligand, a heme iron's to its porphyrin - is reported apart from the covalent bonds'
(REFINE_RMSD_METAL_BONDS).
Anisotropic atoms. An atom's U is held as its equivalent isotropic B and a traceless anisotropy,
U = U_\mathrm{dev} + B/(8\pi^2)\,I, so that everything that refines, restrains and bounds B works on an anisotropic
atom as on an isotropic one, its shape held. Both engines grid it with its full U (an anisotropic Gaussian per
form-factor term, cut at the radius of its widest axis) and give the target's derivatives in its six U elements by the
same FFT method: for \rho_q = a_q \exp(\mathbf{r}^T M_q \mathbf{r}), M_q = -4\pi^2(8\pi^2 U + B_q + B_\mathrm{blur})^{-1},
\partial\rho_q/\partial U = \rho_q\,(M_q + 2 M_q\mathbf{r}\,\mathbf{r}^T M_q). Every U stays positive definite: its
smallest eigenvalue (as B) at least the cycle's floor, its largest at most the ceiling of §15.1.
A TLS group gives each of its atoms U = T + A L A^T + A S + S^T A^T at \mathbf{r} from its origin,
A = [[0, z, -y], [-z, 0, x], [y, -x, 0]] (Schomaker and Trueblood), plus a residual isotropic B per atom; its 20
parameters (the trace of S does not enter U) are refined against the X-ray target alone, the residual B held, with
the derivatives by the chain rule from the atoms' U derivatives (G for T, A^T G A for L, 2 A^T G for S, G an
atom's symmetric matrix of derivatives). An atom refined with a U of its own is restrained, for each pair of
bonded or angle-related atoms, by the rigid bond |(U_i - U_j)\mathbf{n}|^2/\sigma^2, \sigma = 0.004 Ų along the
line \mathbf{n} between them (the components SHELXL's RIGU restrains), and by the similarity
|U_i - U_j|^2/\sigma^2, \sigma = 0.04 Ų (SIMU); an atom in no pair (a water, an ion) by
|U - \mathrm{tr}(U)/3\,I|^2/\sigma^2, \sigma = 0.1 Ų (ISOR).
B restraints. The B factors of two atoms are restrained to each other in proportion to their size, by
the symmetric Kullback-Leibler divergence of two isotropic Gaussians of variances B_i and B_j:
E = \frac{(B_i - B_j)^2}{\kappa^2 B_i B_j},
a difference restraint whose \sigma is \kappa\sqrt{B_iB_j}, so that atoms of high B may differ more than
atoms of low B. \kappa is 0.1 for bonded main-chain atoms and 0.133 for main-chain atoms bonded to a
common atom, 0.2 and 0.3 for the side chains and the rest, and 0.5 for atoms within 4 Å of each other (waters
among them, symmetry mates included) that no bond or angle relates. No atom's B goes below 2 Ų or above
the ceiling of §15.1.
NCS restraints. Where the model holds several copies of one molecule, they are found from the model alone: two polymer chains of one kind (protein, or nucleic acid) are copies where their sequences align with more than 15 residues identical and identical in 80% of the longer chain or more, and their C-alpha (P) atoms lie within 2.5 Å rms of each other after superposition - REFMAC5's criterion, the identity taken on the longer chain. The same sequence in a conformation further away than that, such as a quasi-equivalent subunit of a virus capsid, is another molecule. In the residues aligned to a residue of the same name, the atoms of the same name correspond - not a hydrogen, not an alternative conformation, both refinable. Each copy is paired with the next of its molecule (and the last with the first, where there are three or more), and each pair is restrained locally, on its interatomic distances rather than on a superposition: every distance within a chain shorter than 4.2 Å in either copy, except those a bond or an angle of the dictionary already fixes, against the same distance in the other copy, with the robust loss of Geman and McClure,
E = rac{(\Delta/\sigma)^2}{1 + (\Delta/0.5\, ext{Å})^2},\qquad \Delta = d_\mathrm{copy\,1} - d_\mathrm{copy\,2},
the squared normalised difference where the two distances agree and never more than (0.5\, ext{Å}/\sigma)^2 where
they do not. A region in which the copies really differ - a loop in a crystal contact, a side chain in another
rotamer, a domain turned on a hinge - lets itself go as the refinement moves it, without a rule deciding in advance
which residues to exclude, and rigid motions of whole regions cost nothing, since only distances within 4.2 Å are
compared. Waters and ligands are not restrained, nor are B factors: the copies' B factors are refined independently.
Whether the restraints are used at all, and how tightly (\sigma), the weight trials decide (§15.7).
15.6 The minimiser
Positions and B factors are minimised in turn, each by an L-BFGS of rugnux's own (RefinementMinimiser),
with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a backtracking line
search from the full step to sufficient decrease, its first trial moving no position by more than 0.3 Å and
no B by more than 10 Ų. What sets it apart from a textbook L-BFGS:
- Its memory is kept from one macro-cycle to the next. A macro-cycle is a few dozen iterations on a target whose scale and likelihood parameters were refitted in between; starting each from what the earlier ones learnt of the curvature, instead of from a steepest-descent step, is what lets two macro-cycles of a weight trial get as far as many restarted ones. Each weight trial starts with an empty memory.
- The B steps are scaled by their curvature. The initial inverse Hessian of the B factors is a
diagonal: the X-ray target's curvature in each atom's B - its shape from the likelihood's curvature in
|F_\mathrm{model}|and the atom's own scattering, summed in resolution shells, calibrated once per trial against central differences of the gradient along a random\pm1direction - plus the B restraints' curvature, each element at least a thousandth of the median. The positions take none: their restraints' curvature is that of stiff bonds and angles, which move atoms together, and as a diagonal it slows the soft (torsion and X-ray) directions rather than scaling them.
The diagonal is rescaled at each iteration by the latest pair (Oren-Shanno scaling in that metric). A solve
stops after its iteration budget (30 for the positions and 20 for the B factors per macro-cycle) or where an
iteration lowers the target by less than 10^{-7} of it.
15.7 The weights, decided by trial
The refined target is w\,T_\mathrm{X\text{-}ray} + E_\mathrm{geometry} for the positions and
w_B\,T_\mathrm{X\text{-}ray} + E_B for the B factors. Rather than fixing w and w_B in advance, rugnux
runs short refinements at several weights and keeps the one that does best on the free set - a weight
chosen late, on the evidence, instead of early at a rule.
The starting weights balance the gradients: w_0 = |\nabla E_\mathrm{geometry}| / |\nabla T_\mathrm{X\text{-}ray}|,
the geometry's gradient taken on the model shaken by 0.1 Å (a refined model's geometry would give none), and
w_{B,0} likewise from the B restraints on the B factors shaken by 20%. Each trial is two macro-cycles
(the scale, solvent and likelihood refitted, then the positions, then the B factors) from where it starts.
A trial is within bounds where its rmsd from the dictionary is at most 0.02 Å for bonds and 2° for
angles; one trial displaces another where it is within bounds and its R-free is lower by more than 0.001
(differences smaller than that do not decide, and the trial listed first - on a ladder, the tighter
geometry - is kept). The trials
are:
- The ladder from the placed model:
w = w_0 \times \{1/1024, 1/128, 1/16\}, each with the B factors held and refined atw_{B,0}- six trials. - The trajectory: up to eight steps, each a set of trials from where the last step ended - the weights as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and multiplied by four. A step is taken while it lowers R-free or R-work by more than 0.001 and its R-free stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model, whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the trajectory ends where it began.
- The final ladder: the trajectory's weights times
\{1, 2\}, kept where one displaces it (the weight halved was never kept on 55 refinements, and taking it away changed no choice). - The displacement model, from where the weights ended (the B factors as they are: isotropic, or the model's U
held), each trial as long as a step on the positions where the weights ended, held - only the displacement
parameters refine, so the trials differ in the displacement model alone; the model chosen then takes a step with
the positions refined, kept where it displaces its trial (where the geometry is out of bounds there, which held
positions cannot mend, each trial refines the positions too, a macro-cycle longer than a step): TLS groups with a residual B - the model's own, one per chain
of at least 50 refinable atoms, and the chains cut into 2, 4, 8 and 16 contiguous segments of at least 10 residues
as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r),
to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U
weight balanced as
w_{B,0}is (on the U shaken by 20% of its B) times\{1/4, 1\}. The cuts between one group per chain and the finest run only where the finest fits the free set better (lower-\ln L) than one per chain, and the U at its balanced weight only where the U at a quarter of it lowers R-free or-\ln Lof the free reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving out the U weight\times 4changed a choice taken. A trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds and both its R-free (beyond the tie) and the-\ln Lof the free reflections are lower: the free set tests the parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012). The refined U are written as ANISOU and the TLS groups as_pdbx_refine_tls*andREMARK 3(REFINE_ADP_MODEL,REFINE_ANISO_ATOMS,REFINE_TLS_GROUPS). It is decided after the solvent build (§15.10), on the model with its solvent: a water a round put in refines a U of its own in the per-atom trial as every refinable atom does, and no TLS group holds a water. - The model as given is the first candidate in order of trust: where the refinement does not displace it by the
rule every trial is held to (its geometry within bounds and R-free lower beyond the tie), the model is written as
it came, scaled to the data, and the solvent build's edits are undone (
REFINE_AS_GIVEN). What comes out is never worse on the free set than what went in. This holds only where the free set is free to the model as given: a model placed as given on a free set drawn by the run (MODEL_FREE_SET; in--mode refine, a file without free flags) has most likely been refined against those reflections, and its R-free as given does not compete.
Where the model has copies (§15.5), the NCS restraints are one more weight: every rung of the first ladder is run
without them, and then with them at \sigma = 0.1 Å at the B factors' weight the rungs without chose (NCS moves the
X-ray weight a rung up as often as not, the B factors' choice it never changed on 39 refinements), and every step of
the trajectory also tries them changed - switched on where they are off, and in turn tightened (\sigma halved) and
loosened (\sigma doubled, off beyond 0.4 Å) where they are on. Listed after their twin without, the restraints are
kept only where they lower R-free by more than the tie. A model without copies runs exactly the trials it ran before.
w never exceeds w_0/8. The trials of one set run side by side - on one GPU, on several, or on the CPU's
threads - each from the same start and taking nothing from the one before it, so the result does not depend
on how many run at once.
The refinement can be cut short from outside (the broker's deadline or cancel, Ctrl+C): the check is made between the minimiser's iterations and before every trial, the trials that finished are chosen among as usual, and the report says so. Cut before the first trial, the model is the placement's.
15.8 Maps of the refined model
The refined model's map coefficients, <prefix>_refined_maps.mtz, are 2mF_o - DF_c and mF_o - DF_c as in
§14.4, but with m and D those of the refinement's own likelihood (§15.3) rather than the shell
\sigma_A of §14.4, so the maps carry the weights the target was minimised with; m is
I_1(X)/I_0(X), X = 2F_oD|F_\mathrm{model}|/\Sigma, for an acentric reflection and \tanh(X/2) for a
centric one. As in §14.4, the coefficients in the file have the overall anisotropy of the scale (the anisotropic
part of k_\mathrm{total}) divided out, while the maps the refinement reads itself (the blob finder, the solvent
audit) keep the observed scale. FWT/PHWT is filled at the reflections the data do not have with D|F_\mathrm{model}|
and FWT_no_fill/PHWT_no_fill is not, as in §14.4; F_\mathrm{model} there is the refined model's with the bulk
solvent and scale of the last fit. Where the merge kept the Bijvoet pairs, the anomalous difference map is made on the refined
model's phases too, written in the same file as ANOM/PHANOM (as in <prefix>_maps.mtz; missing where a
reflection has no Bijvoet pair), and read by the blob finder below.
15.9 Unexplained density: blobs
The mF_o - DF_c map of the refined model is searched for what the model leaves unexplained: the connected
regions above 3 σ that peak above 4 σ, found by flood fill (GEMMI's find_blobs_by_flood_fill, the method
of Coot's "unmodelled blobs"), each moved to the copy nearest the model and measured - peak, volume, the
density integrated above the cutoff, the nearest atom, the polar (N, O) atoms at 2.4-3.4 Å and the N, O or S
at 1.8-2.6 Å, and the anomalous map at its peak. Each gets one reading, by rules on what is measured:
| Reading | Rule |
|---|---|
MODEL_ERROR |
within 1.6 Å of a model atom: the model's own atom out of place, or a missing alternative conformation |
ION |
at least 1.8 Å from the model, and an anomalous peak of at least 4 σ, or two metal-like contacts and a peak of at least 8 σ |
LIGAND |
larger than 1.5 times a single atom's blob at this resolution (a sphere of radius d_\mathrm{min}/2) and than 8 ų, at least 2.2 Šfrom the model |
WATER |
at least 2.2 Å from the model with at least one polar contact |
UNASSIGNED |
anything else |
They are listed strongest first (by the integrated density) in <prefix>_blobs.json, and the five strongest
in the report, each with the anomalous difference map's height at its peak where there is that map. A blob
read as LIGAND is density of ligand size where the model has nothing: a buffer molecule or an unmodelled side
chain is that as well as the compound that was soaked. The readings point a
person at the density; they do not identify a compound.
15.10 The solvent audit
Every water and every single-atom ion of the refined model (and, with --refine-score-only, of the model as
given) is measured in its surroundings and its density and scored as each of water, Mg²⁺, Na⁺, K⁺, Ca²⁺ and Cl⁻
(NH₄⁺ is scored too, and only reported). The audit changes nothing; the solvent build below reads it to relabel
sites. Both run only with --refine-solvent (off by default for now).
What is measured. The model's N and O atoms around the site, symmetry mates included, counted in the
coordination bands of Cat_Wiz - 1.9-2.3 Å (Mg-like), 2.3-2.6 Å (Na, Ca), 2.6-3.2 Å (K, water) - and in the
clash shells 3.2-3.4 and 3.4-3.6 Å beyond them. Each neighbour is typed from the dictionary's energy type
(donor, acceptor or both) and from the chemistry of its residue: Lys NZ, Arg NE/NH1/NH2 and a polypeptide's
N-terminal N are positive; Asp/Glu carboxylate O, a C-terminal carboxylate and a nucleotide's phosphate O are
negative; His ring N and the Asn/Gln amide ends are both donor and acceptor, as the model's orientation of
them is a guess. Also measured: the shortest distance between two neighbours closer than 2.6 Å (the
octahedron's cis O…O), the nearest positive and negative group, the other ions within 4.2 and 6 Å, the
bond-valence sum V = \sum_j q_j \exp((R_0 - d_j)/0.37) over the N and O within 3.5 Å for each cation
(R_0 of Brese & O'Keeffe, q_j the neighbour's occupancy) with its normalised vector sum (CheckMyMetal's
nVECSUM), and the 2mFo-DFc, mFo-DFc and anomalous maps at the site, in their rms.
The electrons a site shows. The site's own density is fitted to 2mFo-DFc with its neighbours' held - a local real-space fit of one amplitude at a B tied to its surroundings: the map at points 0.3 Å apart within 1.0 Å of the site, put on the model's scale, less the density the model's other atoms put there (each atom's IT92 density to the data's resolution at its B, to 4.5 Å), against a water's density (O with its two H) at the environment's B (the mean B of the N and O within 3.5 Å, as CheckMyMetal compares a metal's with its ligands'), least squares. The amplitude, in waters, is the electrons it shows, and counted apart from the neighbours' that merge with it at moderate resolution. The map's scale per model unit is the median, over the macromolecule's C, N and O within 6 Å (or all of them where fewer than three are), of each atom's height in the map over what the model puts there - not the waters', whose occupancy is the model's guess (the depositions' waters show half the density their model gives). An ion - whose label's restraints may hold it off its density - is read where that fit puts it, within 0.6 Å of its modelled position (its shell, bands and bond valences are measured from there); a water where it is modelled. On the refine and open arms the depositions' ions read Mg 11, Na 10, Cl 19, K 21-23, Ca 22-26 and Zn 33 (median), the waters 4-6.
The anomalous signal a site would show. For each species, f″ at the data's wavelength (Cromer-Liberman, with the corrections of Kissel & Pratt, as GEMMI computes it) at the environment's B, put on the scale of the anomalous map by least squares on the model's S and P (at least three); without an anomalous map, or nothing to scale it by, the term is left out.
The score. For each species, a plain sum of terms in log units:
| Term | What it is |
|---|---|
| first shell | the neighbours in the species' band against the coordination number it holds (Mg 6, Na 5-6, K 6-10, Ca 6-8, Cl 3-5, NH₄⁺ 4), quadratic; for water only more than four, or none at all, costs; for Mg, Na, Ca and Cl a missing ligand costs little (0.5), as the model may simply not have that water - and a halide binds at hydrophobic surfaces too, with few donors or none (Dauter & Dauter 2001), where its electrons and its anomalous signal tell it from a water |
| distances | for Mg, each neighbour in the band against Mg-O 2.07 ± 0.04 Å, Mg-N 2.19 ± 0.08 Å; for the others, whose distances are spread over their bands, the band is all there is |
| contacts it does not make | 2 per N or O closer than the species' shell (closer than 2.9 Å for Cl), and for Mg per N or O in its exclusion zone 2.3-3.4 Å; for Mg and Ca 2 per first-shell purine N7 the site is more than 1.0 Å out of the plane of, or not 3.5-3.9 Å from the base's O6 or N6 (Leonarski et al. 2017) |
| chemistry | a cation: 2 per donor-only neighbour in its shell and per positive group within 3.5 Å; Cl: 2 per acceptor-only neighbour and per negative group within 3.2 Å; water: 2 per hydrogen bond beyond two donated or two accepted (neighbours at 2.6-3.4 Å) |
| bond valence | for a cation, V against its charge z, Gaussian with a spread of 0.25z widened by the neighbours' distance errors, and the vector sum against CheckMyMetal's outlier bound 0.23 |
| electrons | the electrons the site shows against those the species would (its density's height over a water's at the environment's B: Mg²⁺ and Na⁺ about 11, K⁺, Ca²⁺ and Cl⁻ about 20) - an ion at full occupancy, a water as the model's waters show (their median) - log-normal with the spread of the waters' (an ion's occupancy and order vary as much), at least that of the map's calibration. A site as dense as a full 10-electron scatterer is so more than a typical water, as Echols et al. flag a light ion by a 2mFo-DFc above the waters' mean; where the density is weak the term tells little, and the shell decides |
| anomalous | the anomalous map at the site against what the species would show, Gaussian in the map's rms |
A site bound to another cation (closer than 2.6 Å) shares that cation's other ligands as neighbours - the
cis waters of a Mg(H₂O)₆ are 2.9 Å apart - and those are left out of its score. Coordinate error: each
atom's is taken as its rms displacement \sqrt{B/8\pi^2} times d_\mathrm{min}/10 Š(about 0.1 Šat B = 20 Ų
and 2 Å); a neighbour's band is widened by the error of the distance and its distance term is widened by it
too, so that where the coordinates are poor the species' shells overlap and the score stops telling them
apart instead of telling them apart wrongly. The best of water, Mg, Na, K, Ca and Cl is reported, and
AMBIGUOUS where the runner-up is within 2. The audit counts a site as disagreeing with its label where
the best species is another one, not ambiguously.
Caveats. The refined model's metal-site distances are restrained (§15.5) where its LINK records name
the bonds, which pulls an ion's shell towards the label it carries - the first pitfall of every survey of
ions in the PDB; the local fit reads the ion where its density puts it, but its ligands stay where the restraints
held them. --mode refine --refine-score-only audits the model as given instead. The species are the six above:
a site denser than any of them (a cacodylate's As, a heavier metal) is called the heaviest that fits worst least.
Per site, everything above is in <prefix>_solvent.json (with electrons, b_env, peak_shift_A,
anomalous_expected, n7_ligands, n7_off_plane); the counts are the REFINE_SOLVENT_ keys of the report.
The solvent build. Within the refinement, and to 2.8 Å (phenix.refine's ordered-solvent limit), waters are built
and removed and waters and ions relabelled, as phenix.refine builds waters within its macro-cycles. Where the weight
trajectory (§15.7) ends, a round reads the maps and the audit of that model and changes its solvent; then the solvent
alone - its waters and single-atom ions, the rest held - is refined at the weights the trajectory ended at (its B
too, at its balanced weight where the trajectory held B, and 10 iterations a macro-cycle), and the round is kept
where that lowers R-free; where it does not but the round as built does (refining the solvent alone can lose what its
sites gained: on a deposited model 41 waters lowered R-free by 0.0016 as built and raised it by 0.0036 refined), it
is kept as built; otherwise it is undone. The rest of the model goes on with its B factors held or refined as the
trajectory had them. The rounds stop at one that changes nothing, at one undone, or after two (a third gained at most
0.0004 in R-free where it was kept, on twelve sets, and was refused on the others); the final ladder of §15.7 then
refines what they kept with the rest. The atom list of a refinement is fixed - its engines are made for it - so the
waters a round may add are in the model from the start as placeholders: one water each (as many as a fifteenth of the
model's heavy atoms, at least 64) at occupancy 0, which puts no density, no mask and no contact anywhere and is not
refined until a round puts one at a peak; a water taken out goes to occupancy 0 the same way, and a relabelled site
keeps its place. A round thus costs a map, an audit and a refinement of the solvent: on twelve refine-arm and
open-arm depositions the build adds 0.2-0.9 times the refinement's time (median 0.7), where a refinement of its own
per round added 1-3.7 times (median 2.3), at an R-free within 0.001 of that on eight of them (two 0.0016-0.0028
higher, one 0.0016 lower). --refine-score-only builds nothing, and the build and the audit run only with --refine-solvent
(off by default for now).
-
Relabelled is a water or a single-atom ion (no alternative conformer) whose best-supported species is another one, not ambiguously - a water becomes an ion, an ion water or another ion - once per site. It takes the species' residue name, element and atom name, and scatters as its ion (§15.2); its metal records go, and a metal gets a record for each N, O and S of its inner sphere in the model's own copy (within the library's distance plus 0.33 Å, at the coordination of those within an octahedral site's), which the next refinement restrains as it restrains the model's own (§15.5).
-
Candidates are the local maxima of mFo-DFc from 3 σ (phenix.refine's peak cutoff), each symmetry-equivalent set once, moved to the copy nearest the model - the symmetry operator and lattice translation that put it closest to an atom of the model as written, so a water built there is written beside the model, not beside a symmetry mate or in another unit cell. A peak closer than 1.2 Å to a model atom is that atom's own residual and is passed over.
-
Not solvent - reported in
<prefix>_solvent.jsonwith the reason, never filled: a peak 1.2-1.9 Å from a model atom (a missing or displaced atom); one beside a model atom (within 2.6 Å) that sits in mFo-DFc of -3 σ or lower (the atom is displaced); one within 3 Å of where a truncated residue's missing atoms would be (an atom the dictionary bonds to an atom the model lacks); the centre of three or more other peaks at 1.3-1.7 Å (a tetrahedral group, sulfate or phosphate) or any peak with another 1.1-1.9 Å away (atoms of one molecule); elongated density (the rms length of the density above 3 σ within 1.6 Å along its longest axis over its shortest above 3); and 2mFo-DFc above every S and P of the model (a heavier atom). -
A water is placed where the peak has a hydrogen-bond partner - an N or O that can donate or accept at 2.2-3.4 Å (phenix.refine's 3.2 Å, widened for the coordinate error), a metal within 2.6 Å (its ligand), or a halide at 2.9-3.4 Å - and nothing closer than 2.2 Å to an N or O, nor than 3.0 Å to any other atom (Coot keeps waters 2.4-3.2 Å from the model; a little wider here for the peak's position), strongest first, none within 2.2 Å of one already placed. It goes into a chain of waters of its own, as PDB and mmCIF entries keep them - chain
W(or, where the model's chainWholds more than waters, the first ofX,Y,Z,V,U, ... that does not),HOHresidues numbered on from that chain's last, a water entity and subchain of their own in the mmCIF - with occupancy 1 and the mean B of its partners. The model's own waters stay where they were. A peak on a special position (§15.5) is placed exactly on it with occupancy 1 over the site's multiplicity, its coinciding copies one water (on the refine and open arms this was better or equal in R-free on all five sets with such peaks, against refusing them); one 0.5-2.2 Å from its own copy, which it would clash with, is refused. -
Removed is any water whose mFo-DFc at its centre is -3 σ or lower (the cut Echols et al. use), and a water the build itself added that has no N, O or ion within 3.4 Å any more or whose B reached the refinement's ceiling. phenix.refine's other filter, 2mFo-DFc below 1 σ at a refined water, is not used: on the refine arm it raised R-free wherever it was tried, on the deposited waters (five sets of five) and on the waters built here, and so did taking out deposited waters without a partner. A weak water still explains density, and a deposited one carries its depositor's judgement.
REFINE_SOLVENT_ADDED and REFINE_SOLVENT_REMOVED count the waters added and still there and those of the model
as given taken out, REFINE_SOLVENT_RELABELLED the sites relabelled (each, with its scores, in the report and the
JSON); REFINE_SOLVENT_NOT_SOLVENT the not-solvent peaks of the final model's difference map;
REFINE_SOLVENT_BUILD_ROUNDS the rounds kept. The model as given (REFINE_R_FREE_START) and the placement stage are the first refinement's; the refined model, its R-factors (the REFINE_SCORE_ terms with them), geometry, maps and blobs are the last kept round's.
15.11 GPU and CPU
On a CUDA build the target, the gradient and the restraints are evaluated on the card - the density gather,
cuFFT, the composition of the symmetry and the target per reflection, the adjoint through a CSR list, the inverse transform and one warp per atom for its gradient; the restraints in one kernel whose
per-atom gradient slots are summed in a fixed order; the robust scale fit and the fine solvent mask on the
card as well - with no floating-point atomics, so the result is the same from run to run. The weight trials
of a set run on as many cards as there are, each trial on an engine of its own. Where the case does not fit
the card's budget, or with --no-gpu, or on a build without CUDA (macOS included: the refinement has no
Metal path), the CPU engine computes the same target, to rounding, several times more slowly (seconds on a
GPU, typically 2-15 minutes on the CPU).
15.12 Twinned data
Where the lattice admits a twin law for the group the data are merged in - a rotation of the lattice's metric
symmetry (Le Page two-folds within 3° of obliquity, the search the free set is grouped by, §10.7) that the group
lacks, one per coset of the group, with its square in the group so that it exchanges two domains - the placed
model is refined a second time, against the data as twinned by the law, and the two refinements are compared.
rugnux <file> does this on its own; nothing has to be given.
The target. A crystal twinned by law T at fraction \alpha records (1-\alpha)I(\mathbf{h}) + \alpha I(T\mathbf{h})
(Yeates 1997), so the model an observation is compared with is
|F_\mathrm{twin}(\mathbf{h})| = k\,e^{-\mathbf{s}^\top\mathbf{B}\mathbf{s}/4}\sqrt{(1-\alpha)\,|F_\mathrm{calc}(\mathbf{h}) + F_\mathrm{sol}(\mathbf{h})|^2 + \alpha\,|F_\mathrm{calc}(T\mathbf{h}) + F_\mathrm{sol}(T\mathbf{h})|^2},
and the target is the likelihood of §15.2 with |F_\mathrm{model}| replaced by |F_\mathrm{twin}|, D and \Sigma
fitted against it (§15.3). At \alpha = 0 it is the untwinned target exactly. It is an approximation: a twinned
amplitude is treated as Rice-distributed about D|F_\mathrm{twin}|, while the distribution of a sum of two
intensities is narrower; \Sigma, fitted to the free set, absorbs that, and REFMAC5's twinned-intensity likelihood
(Murshudov et al. 2011) is the treatment this one simplifies. The mates T\mathbf{h} that were not observed are
composed as rows of their own, with no observation: their structure factors come from the same transform, so a
twinned evaluation costs no extra FFT, only one more kernel per evaluation on the GPU (the twinned term per
reflection and the gather of its mates' derivatives, through the converse of the mate table, as a three-fold of
several domains is not an involution).
The fraction, the scale and the solvent are fitted inside the twin sum at every macro-cycle: the scale and the
solvent as in §15.3, then \alpha, then the scale and the solvent again against F_o detwinned by the model as it
stands, F_o|F_\mathrm{model}(\mathbf{h})|/|F_\mathrm{twin}(\mathbf{h})| (at which scale the twinned model meets F_o
as the untwinned one meets the detwinned F_o), then \alpha again. \alpha \in [0, 1/2] comes from the regression
of F_o^2, with an intercept, on the model's intensities at \mathbf{h} and at T\mathbf{h}, all normalised in
resolution shells of 500 reflections: \alpha = b_{T\mathbf{h}}/(b_\mathbf{h} + b_{T\mathbf{h}}), over the working
reflections that are not their own mate. The expectation of F_o^2 given the model, D^2|F_\mathrm{model}|^2 + \Sigma,
is linear in the two intensities, so an unrelated partner takes a coefficient of 0 however poor the model; a
least-squares fit of the twinned amplitudes instead took a fraction of 0.16 for unrelated partners of one model at
R-free 0.31, the average of two predictions being a better prediction than either.
The decision. The twinned refinement is kept where its fraction is above 0 and the law beats its own null
on the free reflections by at least three standard errors - the null being the same twinned model at the same
fraction with every reflection's mate replaced by an unrelated reflection of the same resolution and of the same
set (a free reflection's by a free one, as a law's are): what the averaging of two intensities buys whatever they
are (it lifts a weak reflection's model amplitude off zero). The comparison is paired, reflection by reflection,
|F_o - |F_\mathrm{null}|| - |F_o - |F_\mathrm{twin}|| over the free set, summed over each free twin pair (the two
mates share their two F_\mathrm{calc}, so they are one term, not two independent ones), its mean against its
standard error (REFINE_TWIN_NULL_Z). The R-free of the twinned refinement against the untwinned one's does not
enter: two refinements of one model differ in R-free by a few thousandths from their trajectories alone, and any
twin target lowers R by its averaging, so that comparison favours the twinned one with or without a twin. A pseudo-symmetry the model already carries (non-crystallographic
symmetry parallel to the law) predicts I(T\mathbf{h}) \approx I(\mathbf{h}), so the law changes nothing and is not
taken for a twin.
The data's own evidence. A lead of two to three standard errors is enough where the merge's intensity statistics
read twinned as well - the L-test itself below 0.44 (TWINNING_VERDICT INDICATED with
ESTIMATED_TWIN_FRACTION_FROM L_TEST; not where only the second moment reads low, nor where a pseudo-translation
leaves the L-test unreadable): two independent pieces of evidence, the data's and the model's, then agree.
REFINE_TWIN_RULE says which rule decided. The data-side test is not made a gate - it misses twins a model sees, a
pseudo-merohedral law or a near-perfect twin among them - only further evidence where the null alone is short.
--mode refine has no merge statistics, so there the null alone decides.
The free set is closed under the lattice's domains before either refinement, so that both use one free set
and no law splits a group of twin-related reflections between the working and the free set (REFMAC5 does the same,
Roversi et al. 2012). The groups are the orbits of the merged reflections under every domain operator of the lattice
(below); an orbit the flags already keep in one set stays there, and a split one goes free with the probability of its
own share of free flags, drawn by a hash of the orbit, so the free fraction stays what it was. A majority of each
orbit's flags would empty the free set where the flags were drawn on smaller orbits - a C-centred orthorhombic merge
of a hexagonal metric, whose flags (§10.7) are drawn on the orthorhombic holohedry of the cell's own basis vectors, lost
three quarters of it that way. The free set rugnux draws is already closed for every law that is a rotation of the
cell's own basis vectors, and nothing moves; a law of a centred lattice that is not (its matrix fractional in the
cell's basis), or a free set --mode refine reads from a file, can be split, and the closure is what lets such a law
be tested at all. REFINE_TWIN_FREE_CLOSED counts the reflections that changed set. A law still split (none is,
after the closure, unless its mates fall outside the orbits the data have) is reported (REFINE_TWIN
FREE_SET_SPLIT) rather than tested. Of several laws, the one the placed model asks the largest fraction of is refined. Each of the
two refinements is a whole one - its solvent built (§15.10) and its displacement model chosen (§15.7) against its
own target - and the decision is taken where they ended. --refine-score-only makes the same test on the model
as given: it is scored against the data untwinned and twinned by the law, and the twinned score is taken by the
same rule.
More than two domains. A lattice can hold more domains than two: a monoclinic crystal on a tetragonal metric
four, on a hexagonal one six (Sultana et al. 2007; Roversi et al. 2012). Where the single law is taken and the lattice
has more domain operators - every coset of the group in the lattice's metric symmetry, whatever the operator's order,
three-folds included - a second rung refines every domain at once, each with its own fraction:
|F_\mathrm{twin}(\mathbf{h})|^2 \propto a_0|U(\mathbf{h})|^2 + \sum_d a_d|U(T_d\mathbf{h})|^2, a_0 = 1 - \sum_d a_d,
the fractions from the same intercept regression with one column per domain, a domain whose coefficient comes out
negative taken out and the rest fitted again. It is taken where the added domains beat their own null - the
same model with the added domains' mates replaced by unrelated reflections and the adopted law's kept - by three
standard errors on the free set, summed over each twin group, and each added domain refines to a fraction of 0.05 or
more (as REFMAC5 drops small domains, Murshudov et al. 2011); the domains under it are dropped and the rest refined
again, and with none left the single law stands. REFINE_TWIN_MULTI (MULTI_TWINNED, MULTI_UNTWINNED) says what
was decided, REFINE_TWIN_MULTI_DOMAINS the domains and their fractions; where MULTI_TWINNED, REFINE_TWIN_LAW
lists every law and REFINE_TWIN_FRACTION is their sum, and every other REFINE_ key is of that refinement. The
rung is not tried where the single law was refused: a twin that only several domains show at once is a symmetric
one, whose single-law fraction is already large.
Maps. Against twinned data the map coefficients use F_o detwinned in proportion to the model's own share of
the twinned intensity, F_o|F_\mathrm{model}(\mathbf{h})|/|F_\mathrm{twin}(\mathbf{h})|, and DF_c of the model's own
|F_\mathrm{model}(\mathbf{h})|; FP in <prefix>_refined_maps.mtz is the detwinned amplitude.
The keys are REFINE_TWIN (NO_TWIN_LAW, FREE_SET_SPLIT, ZERO_FRACTION, UNTWINNED, TWINNED),
REFINE_TWIN_LAW, REFINE_TWIN_FRACTION, REFINE_TWIN_R_WORK/_R_FREE, REFINE_TWIN_R_FREE_NULL, REFINE_TWIN_NULL_Z and
REFINE_UNTWINNED_R_WORK/_R_FREE, the last five of the two refinements where they ended. Where REFINE_TWIN is TWINNED, every
other REFINE_ key, REFINE_R_FREE_START and the REFINE_SCORE_ terms included, is of the twinned refinement.
15.13 Not yet
What the refinement does not do yet - each a known gap, not a design choice:
- Custom ligands and other monomer libraries. Only the 84 compounds of §15.4 are restrained. Any other
residue is held fixed (it stays in
F_\mathrm{calc}, the mask and the repulsions); no restraints are generated for a new ligand, and neither a user's restraint CIF nor an external monomer library can be given. - TLS restraints: a TLS group's T and L are not held positive semi-definite (only every atom's total U is kept positive definite); the TLS groups are contiguous segments of one chain, never a domain made of several.
- Global NCS and NCS of B factors: copies are restrained locally, on their interatomic distances (§15.5), never by a superposition, and their B factors are refined independently; ligands and waters are not restrained between copies.
- More than two twin domains (tetartohedral twinning): one law at a time is refined, and laws whose square is not in the group are not considered.
- Ion identification beyond the six species: waters and ions are relabelled among water, Mg²⁺, Na⁺, K⁺, Ca²⁺ and Cl⁻ (§15.10); NH₄⁺ is only reported, transition metals (Zn, Mn, ...) are measured but never relabelled, and no ion is placed at a peak that is not a water first. A site's hypotheses are compared by a local fit of its density, not by refining each in turn.
- Alternative conformations and occupancies: existing conformers are refined in place; no occupancy is refined and no conformer is added.
- Rotamer and peptide-flip search, and Ramachandran restraints.
- Real-space refinement, and jelly-body or reference-model restraints.
- A Metal (Apple GPU) path: on macOS the refinement runs on the CPU.
- Validation beyond geometry: the report gives the bond and angle rmsd, the metal-bond rmsd and the number of clashes, not a MolProbity-style analysis (Ramachandran, rotamer and clashscore statistics).
References
- R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", Acta Cryst. A34 (1978), 791-809 (the gradient by the FFT method, §15.2).
- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", Acta Cryst. A52 (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", Acta Cryst. D53 (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8).
- G. N. Murshudov, P. Skubák, A. A. Lebedev, N. S. Pannu, R. A. Steiner, R. A. Nicholls, M. D. Winn, F. Long & A. A. Vagin, "REFMAC5 for the refinement of macromolecular crystal structures", Acta Cryst. D67 (2011), 355-367 (D from the working set and the variance from the free, §15.3; the Kullback-Leibler form of the B restraints, §15.5; the twinned refinement §15.12 simplifies, its twin laws pruned by fraction and its fractions refined by least squares).
- R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", Acta Cryst. A42 (1986), 140-149 (the m, D weighting of the map coefficients, §15.8).
- T. O. Yeates, "Detecting and overcoming crystal twinning", Methods Enzymol. 276 (1997), 344-358 (the twinned intensity, §15.12).
- A. A. Lebedev, A. A. Vagin & G. N. Murshudov, "Intensity statistics in twinned crystals with examples from the PDB", Acta Cryst. D62 (2006), 83-95 (twin laws from the lattice's metric, §15.12).
- R. Herbst-Irmer & G. M. Sheldrick, "Refinement of twinned structures with SHELXL97", Acta Cryst. B54 (1998), 443-449 (twinned refinement with the fraction refined beside the model, §15.12).
- V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", Acta Cryst. A51 (1995), 880-887, and V. Y. Lunin, P. V. Afonine & A. G. Urzhumtsev, "Likelihood-based refinement. I. Irremovable model errors", Acta Cryst. A58 (2002), 270-282 (the likelihood's parameters from the free set, §15.3).
- J.-S. Jiang & A. T. Brünger, "Protein hydration observed by X-ray diffraction: solvation properties of penicillopepsin and neuraminidase crystal structures", J. Mol. Biol. 243 (1994), 100-115, and A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", Acta Cryst. D58 (2002), 1387-1392 (the flat bulk-solvent mask and its parameters, §15.3).
- P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", Acta Cryst. D61 (2005), 850-855 (the grid search over the solvent parameters, §15.3).
- P. J. Huber, "Robust estimation of a location parameter", Ann. Math. Statist. 35 (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, "Deterministic edge-preserving regularization in computed imaging", IEEE Trans. Image Process. 6 (1997), 298-311 (the robust scale fit and its smooth, pseudo-Huber loss, §15.3).
- A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long & G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", Acta Cryst. D60 (2004), 2184-2195 (the restraint dictionary, §15.4).
- M. Wojdyr, "GEMMI: A library for structural biology", J. Open Source Softw. 7 (2022), 4200 (the topology, the riding hydrogens and the blob search, §15.4, §15.9).
- V. Schomaker & K. N. Trueblood, Acta Cryst. B24 (1968), 63-76, and M. D. Winn, M. N. Isupov & G. N. Murshudov, Acta Cryst. D57 (2001), 122-133 (TLS, and the B column total or residual to it, §15.1, §15.5); J. Painter & E. A. Merritt, Acta Cryst. D62 (2006), 439-450 (TLSMD's segments, §15.7).
- F. L. Hirshfeld, Acta Cryst. A32 (1976), 239-244, A. Thorn, B. Dittrich & G. M. Sheldrick, Acta Cryst. A68 (2012), 448-451, and G. M. Sheldrick, Acta Cryst. C71 (2015), 3-8 (the rigid-bond, similarity and isotropy restraints of anisotropic atoms, §15.5); E. A. Merritt, Acta Cryst. D68 (2012), 468-477 (the displacement model decided by test, §15.7).
- A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", J. Comput. Chem. 17 (1996), 1132-1141 (the torsion derivatives, §15.5).
- M. M. Harding, "Geometry of metal-ligand interactions in proteins", Acta Cryst. D57 (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", Acta Cryst. D62 (2006), 678-682; F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", Nucleic Acids Res. 45 (2017), 987-1004 (the metal-site distances, §15.5).
- R. W. Grosse-Kunstleve & P. D. Adams, "Algorithms for deriving crystallographic space-group information. II. Treatment of special positions", Acta Cryst. A58 (2002), 60-65 (the site symmetry and the constraint of an atom on a special position, §15.5).
- D. T. Cromer & D. A. Liberman, "Anomalous dispersion calculations near to and on the long-wavelength side of an absorption edge", Acta Cryst. A37 (1981), 267-268, and L. Kissel & R. H. Pratt, "Corrections to tabulated anomalous-scattering factors", Acta Cryst. A46 (1990), 170-175 (f″ at the data's wavelength, §15.10).
- Z. Dauter & M. Dauter, "Entering a new phase: using solvent halide ions in protein structure determination", Structure 9 (2001), R21-R26 (halides at sites with few donors, §15.10).
- N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", Acta Cryst. D70 (2014), 1104-1114 (a light ion flagged by its density above the waters', §15.10).
- J. Nocedal, "Updating quasi-Newton matrices with limited storage", Math. Comp. 35 (1980), 773-782, and D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", Math. Program. 45 (1989), 503-528 (the minimiser, §15.6).
- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", Acta Cryst. A45 (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7).
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", Acta Cryst. D68 (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7).
- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", Nature 355 (1992), 472-475 (R-free).
- G. J. Kleywegt, "Use of non-crystallographic symmetry in protein structure refinement", Acta Cryst. D52 (1996), 842-857; I. Usón, E. Pohl, T. R. Schneider, Z. Dauter, A. Schmidt, H.-J. Fritz & G. M. Sheldrick, "1.7 Å structure of the stabilized REIv mutant T39K. Application of local NCS restraints", Acta Cryst. D55 (1999), 1158-1167; R. A. Nicholls, F. Long & G. N. Murshudov, "Low-resolution refinement tools in REFMAC5", Acta Cryst. D68 (2012), 404-417; J. J. Headd et al., "Flexible torsion-angle noncrystallographic symmetry restraints for improved macromolecular structure refinement", Acta Cryst. D70 (2014), 1346-1356 (local NCS restraints with a robust loss, decided by R-free, and Murshudov et al. 2011 above for the copies found by alignment, §15.5, §15.7).
- P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", Acta Cryst. D66 (2010), 486-501 (unmodelled blobs, §15.9).
- N. Naleem, A. Henning-Knechtel, S. Kirmizialtin & P. Auffinger, "Cat_Wiz: a stereochemistry-guided toolkit for locating, diagnosing, and annotating Mg2+ ions in RNA structures", Nucleic Acids Res. 54 (2026), gkag120; F. Leonarski, L. D'Ascenzo & P. Auffinger, Nucleic Acids Res. 45 (2017), 987-1004 and RNA 25 (2019), 173-192; F. Leonarski, A. Henning-Knechtel, S. Kirmizialtin, E. Ennifar & P. Auffinger, Nucleic Acids Res. 53 (2025), gkae1148 (the coordination bands, distances and exclusion zone of the solvent audit, §15.10).
- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", Acta Cryst. D68 (2012), 352-367, and P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", Acta Cryst. D66 (2010), 486-501 (the peak cutoff, the partner and clash distances and the resolution limit of the water build, §15.10).
- N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", Acta Cryst. D70 (2014), 1104-1114, and N. Morshed, N. Echols & P. D. Adams, "Using support vector machines to improve elemental ion identification in macromolecular crystal structures", Acta Cryst. D71 (2015), 1147-1158 (the ligand sphere and the halide rules, §15.10).
- H. Zheng et al., "CheckMyMetal: a macromolecular metal-binding validation tool", Acta Cryst. D73 (2017), 223-233 (the vector sum, §15.10); O. Carugo, "Buried chloride stereochemistry in the Protein Data Bank", BMC Struct. Biol. 14 (2014), 19 (the chloride's shell, §15.10).
- I. D. Brown & D. Altermatt, Acta Cryst. B41 (1985), 244-247, and N. E. Brese & M. O'Keeffe, "Bond-valence parameters for solids", Acta Cryst. B47 (1991), 192-197 (the bond-valence sums, §15.10).
- M. Abramowitz & I. A. Stegun, Handbook of Mathematical Functions, National Bureau of Standards (1964), 9.8.1-9.8.4 (the Bessel functions of the target, §15.2).
The full citations, with DOIs, are in ACKNOWLEDGEMENT.md.