From e406eb17ea2fc0d8872e86704474ca4a929e7d39 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 26 Oct 2026 11:02:10 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20refinement=20chapter=20(=C2=A715),=20RE?= =?UTF-8?q?FINE=5F=20report=20keys,=20acknowledgements=20and=20in-source?= =?UTF-8?q?=20credits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10) --- docs/ACKNOWLEDGEMENT.md | 25 +- docs/CPU_DATA_ANALYSIS.md | 16 +- docs/CPU_DATA_ANALYSIS_DECISIONS.md | 4 +- docs/CPU_DATA_ANALYSIS_IMAGE.md | 2 +- docs/CPU_DATA_ANALYSIS_INDEXING.md | 2 +- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 +- docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 330 ++++++++++++++++++ docs/GPU_DATA_ANALYSIS.md | 23 ++ docs/RUGNUX_ADVANCED.md | 18 +- docs/RUGNUX_MODEL_AND_REFERENCE.md | 20 ++ docs/RUGNUX_REPORT.md | 52 +++ docs/TECHNICAL.md | 1 + docs/THIRD_PARTY_NOTICES.md | 12 +- .../structure_refinement/DensityBlobs.cpp | 1 + .../structure_refinement/ModelScaling.cpp | 1 + .../structure_refinement/Refinement.cpp | 5 +- .../RefinementMinimiser.h | 2 +- .../structure_refinement/RefinementXray.h | 1 + 18 files changed, 496 insertions(+), 21 deletions(-) create mode 100644 docs/CPU_DATA_ANALYSIS_REFINEMENT.md diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index c33ca9378..5ac0c8bc3 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -418,7 +418,11 @@ observed scale with an overall factor, an anisotropic B and a flat bulk-solvent model of A. Fokine and A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters" (2002), Acta Cryst. D58, 1387-1392 [doi:10.1107/S0907444902010284](https://doi.org/10.1107/S0907444902010284), which is also the source -of the starting values and of the range those two parameters are physically meaningful over. The +of the starting values and of the range those two parameters are physically meaningful over. The mask itself - +the region a probe cannot reach from the atoms, shrunk back - is J.-S. Jiang and A. T. Brünger's, "Protein +hydration observed by X-ray diffraction: solvation properties of penicillopepsin and neuraminidase crystal +structures" (1994), J. Mol. Biol. 243, 100-115 +[doi:10.1006/jmbi.1994.1633](https://doi.org/10.1006/jmbi.1994.1633), drawn with the radii REFMAC5 uses. The procedure that fits them — a grid search over that range for the solvent pair, with the overall scale and the anisotropic B refitted at every grid point — follows P. V. Afonine, R. W. Grosse-Kunstleve and P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling @@ -540,7 +544,9 @@ R. C. Agarwal, "A new least-squares refinement technique based on the fast Fouri The positions and B factors are minimised by L-BFGS (rugnux's own, its memory kept from one macro-cycle to the next, the B factors' steps scaled by their estimated curvature): D. C. Liu and J. Nocedal, "On the limited memory BFGS method for large scale optimization" (1989), Math. Program. 45, 503-528 -[doi:10.1007/BF01589116](https://doi.org/10.1007/BF01589116). The geometry restraints are those of the CCP4 +[doi:10.1007/BF01589116](https://doi.org/10.1007/BF01589116), whose two-loop recursion is J. Nocedal's, "Updating +quasi-Newton matrices with limited storage" (1980), Math. Comp. 35, 773-782 +[doi:10.1090/S0025-5718-1980-0572855-7](https://doi.org/10.1090/S0025-5718-1980-0572855-7). The geometry restraints are those of the CCP4 Monomer Library, a subset of which rugnux carries (see THIRD_PARTY_NOTICES.md): A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long and G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use" (2004), Acta Cryst. D60, @@ -552,11 +558,15 @@ A. Blondel and M. Karplus, "New formulation for derivatives of torsion angles an in molecular mechanics: elimination of singularities" (1996), J. Comput. Chem. 17, 1132-1141 [doi:10.1002/(SICI)1096-987X(19960715)17:9<1132::AID-JCC5>3.0.CO;2-T](https://doi.org/10.1002/(SICI)1096-987X(19960715)17:9%3C1132::AID-JCC5%3E3.0.CO;2-T). The B factors of bonded atoms, and of atoms in contact, are restrained to each other in the -Kullback-Leibler form REFMAC5 uses, whose tolerance grows with B (Murshudov et al. 2011, above), and the +Kullback-Leibler form REFMAC5 uses, whose tolerance grows with B (Murshudov et al. 2011, above; the divergence itself +is S. Kullback and R. A. Leibler's, "On information and sufficiency" (1951), Ann. Math. Statist. 22, 79-86 +[doi:10.1214/aoms/1177729694](https://doi.org/10.1214/aoms/1177729694)), and the overall isotropic B of the scale is moved into the atoms as phenix.refine moves it (Afonine et al. 2012, below). The weights of the X-ray term against the geometry and against the B restraints start from the -ratios of the gradients and are chosen among short refinements by R-free with the geometry held within -bounds, as phenix.refine chooses them: P. V. Afonine, R. W. Grosse-Kunstleve, N. Echols, J. J. Headd, +ratios of the gradients, as X-PLOR's crystallographic weight was first set: A. T. Brünger, M. Karplus and +G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin" (1989), Acta Cryst. +A45, 50-61 [doi:10.1107/S0108767388009195](https://doi.org/10.1107/S0108767388009195); and they are chosen +among short refinements by R-free with the geometry held within bounds, as phenix.refine chooses them: P. V. Afonine, R. W. Grosse-Kunstleve, N. Echols, J. J. Headd, N. W. Moriarty, M. Mustyakimov, T. C. Terwilliger, A. Urzhumtsev, P. H. Zwart and P. D. Adams, "Towards automated crystallographic structure refinement with phenix.refine" (2012), Acta Cryst. D68, 352-367 [doi:10.1107/S0907444912001308](https://doi.org/10.1107/S0907444912001308). R-free itself: @@ -566,7 +576,10 @@ The model the refinement scores is scaled to the data with the overall scale and robust least squares - iteratively reweighted, with the pseudo-Huber weight of each reflection's residual - so that a few reflections the model cannot explain do not set the scale of the rest: P. J. Huber, "Robust estimation of a location parameter" (1964), Ann. Math. Statist. 35, 73-101 -[doi:10.1214/aoms/1177703732](https://doi.org/10.1214/aoms/1177703732). +[doi:10.1214/aoms/1177703732](https://doi.org/10.1214/aoms/1177703732), in the smooth (pseudo-Huber) form of +P. Charbonnier, L. Blanc-Féraud, G. Aubert and M. Barlaud, "Deterministic edge-preserving regularization in +computed imaging" (1997), IEEE Trans. Image Process. 6, 298-311 +[doi:10.1109/83.551699](https://doi.org/10.1109/83.551699). The bonds of metal ions to their donor atoms that the model's LINK records name are restrained at the distances of the Monomer Library's own statistics of metal sites (its `metals.json`, embedded as `monomers/metals.cif`; per metal, donor element and coordination number), in line with the surveys of diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 5aa0e067b..04e8c1046 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -21,7 +21,7 @@ This document describes the crystallographic algorithms implemented in Jungfrauj 15. restrained refinement of the placed model (wherever a model is placed): rigid bodies and group B, then positions and B against a maximum-likelihood amplitude target with the dictionary's geometry, the X-ray weight chosen by R-free within geometry bounds, and the unexplained blobs of the refined difference map. -The reference is split into four parts, in pipeline order; the section numbers run continuously +The reference is split into five parts, in pipeline order; the section numbers run continuously across them and are the ones the rest of the documentation cites. - [From images to spots (§0–§3)](CPU_DATA_ANALYSIS_IMAGE.md) — device-side decoding, geometry and @@ -34,6 +34,10 @@ across them and are the ones the rest of the documentation cites. - [Space group and validation (§13–§14)](CPU_DATA_ANALYSIS_DECISIONS.md) — the space-group search, twinning and translational pseudo-symmetry, the resolution cutoff, diffraction anisotropy, model-based validation and molecular replacement. +- [Refinement of a placed model (§15)](CPU_DATA_ANALYSIS_REFINEMENT.md) — the maximum-likelihood target and + its FFT gradient, the scale, bulk solvent and likelihood parameters, the dictionary and its restraints + (metal sites and B included), the minimiser, the weights chosen by trial, the refined maps and their + unexplained blobs, and what is not implemented yet. ## References @@ -66,7 +70,15 @@ The methods draw on, and in places reimplement, solutions from: - A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15). - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (the weight chosen among refinements by R-free, §15). - D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15). -- M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14). +- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6). +- G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5). +- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). +- P. J. Huber, "Robust estimation of a location parameter", *Ann. Math. Statist.* **35** (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, *IEEE Trans. Image Process.* **6** (1997), 298-311 (the robust, pseudo-Huber scale fit of the refinement's score, §15.3). +- J.-S. Jiang & A. T. Brünger, "Protein hydration observed by X-ray diffraction", *J. Mol. Biol.* **243** (1994), 100-115 (the flat bulk-solvent mask, §14.2, §15.3). +- A. Blondel & M. Karplus, *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5). +- M. M. Harding, *Acta Cryst.* **D57** (2001), 401-411 and **D62** (2006), 678-682, and F. Leonarski, L. D'Ascenzo & P. Auffinger, *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5). +- P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9). +- M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14; the topology, riding hydrogens and blob search of §15). - J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and [github.com/jonwright/bslz4decoders](https://github.com/jonwright/bslz4decoders) (device-side decoding of bitshuffle+LZ4 images, §0). - A. Thorn & G. M. Sheldrick, "ANODE: anomalous and heavy-atom density calculation", *J. Appl. Cryst.* **44** (2011), 1285-1287 (anomalous difference density read at the model's sites). - R. Kahn, R. Fourme, A. Gadet, J. Janin, C. Dumas & D. Andre, "Macromolecular crystallography with synchrotron radiation: photographic data collection and polarization correction", *J. Appl. Cryst.* **15** (1982), 330-337 (the azimuthal polarization factor of §2.2, applied to the azimuthal profile, the Bragg intensities and the ring background the beam-stop shadow test compares against). diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 46457e451..e0c1a399e 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -1,6 +1,6 @@ # Data analysis: space group and validation (§13–§14) -Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its four parts. +Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its parts. ```{contents} On this page :local: @@ -124,7 +124,7 @@ Everything lands in `_report.txt` section 4 (`ANISOTROPY_*` keys), in th ## 14. Model-based validation: R-free against a model and electron-density maps -Offline (`rugnux --model model.pdb`) the merged data can be scored against a supplied atomic model and electron-density maps computed — enough to confirm that a model fits the data and to inspect the density, not a substitute for refinement. **The structure itself is not refined**; the model is re-fractionalized into the data unit cell and then **placed as one rigid body** (§14.8) - or, where the rigid body does not place it and for a predicted model, by molecular replacement (§14.10) - and the observed amplitudes are the French–Wilson $|F|$ from §10.8, so the R-free and the maps use exactly the same amplitudes as the written reflection file. The model, structure-factor, bulk-solvent and FFT machinery is provided by GEMMI. +Offline (`rugnux --model model.pdb`) the merged data can be scored against a supplied atomic model and electron-density maps computed — enough to confirm that a model fits the data and to inspect the density, not a substitute for refinement. **The structure itself is not refined here** - a placed model is refined afterwards, on the side and deciding nothing (§15, [Refinement of a placed model](CPU_DATA_ANALYSIS_REFINEMENT.md)); the model is re-fractionalized into the data unit cell and then **placed as one rigid body** (§14.8) - or, where the rigid body does not place it and for a predicted model, by molecular replacement (§14.10) - and the observed amplitudes are the French–Wilson $|F|$ from §10.8, so the R-free and the maps use exactly the same amplitudes as the written reflection file. The model, structure-factor, bulk-solvent and FFT machinery is provided by GEMMI. ### 14.1 Model structure factors diff --git a/docs/CPU_DATA_ANALYSIS_IMAGE.md b/docs/CPU_DATA_ANALYSIS_IMAGE.md index d7c57ad1b..e9d9e99d2 100644 --- a/docs/CPU_DATA_ANALYSIS_IMAGE.md +++ b/docs/CPU_DATA_ANALYSIS_IMAGE.md @@ -1,6 +1,6 @@ # Data analysis: from images to spots (§0–§3) -Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its four parts. +Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its parts. ```{contents} On this page :local: diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index a83665b43..bf2836230 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -1,6 +1,6 @@ # Data analysis: indexing and geometry (§4–§7) -Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its four parts. +Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its parts. ```{contents} On this page :local: diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index eb4dbc5ab..a58ec949d 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -1,6 +1,6 @@ # Data analysis: integration, scaling and merging (§8–§12) -Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its four parts. +Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its parts. ```{contents} On this page :local: diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md new file mode 100644 index 000000000..6abf3ebe2 --- /dev/null +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -0,0 +1,330 @@ +# Data analysis: refinement of a placed model (§15) + +Part of the [CPU/GPU data-analysis reference](CPU_DATA_ANALYSIS.md); the section numbers are continuous across its parts. + +```{contents} On this page +:local: +:depth: 2 +``` + +## 15. Restrained refinement of a placed model + +Wherever a model is placed - it fits where it is (§14.8), or molecular replacement placed it (§14.10) - +rugnux refines it against the merge it was placed on (`--no-refine` skips this). `rugnux --mode refine` +runs the same refinement on a model and merged structure factors that already exist. How to use it, and +what it writes, is in [Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md#refinement); its report +keys are in [The results report](RUGNUX_REPORT.md#refinement-refine_). + +The refinement comes **after every processing decision and changes none of them**. Nothing it finds is +read by the space-group search, the indexing choice, the resolution cut or the files `.mtz`, +`_model.cif` and `_maps.mtz`; `R_WORK` and `R_FREE` stay those of the model as placed +(§14.3). What it produces is written beside them: the refined model, its map coefficients, the blobs of its +difference map and the `REFINE_` keys. + +The library reads and writes no file. The model and the reflections come in, the refined model, its +statistics and its map coefficients go out; the restraint dictionary is compiled into the binary (§15.4). + +### 15.1 What is refined, and in what order + +The model is taken as the refinement can use it: the first model of the file only, its hydrogens removed, +every anisotropic B replaced by its isotropic equivalent, and every B clamped to the range the refinement +allows (2 Ų up to twice the model's largest B, within 100-300 Ų). Occupancies are kept as they are, and +so are alternative conformations, each refined with its own restraints. + +Two stages follow. + +1. **Placement.** Each chain of at least 50 non-water atoms is moved as a rigid body - an angle-axis + rotation about its centroid, scaled by its rms radius so that a unit of rotation moves a typical atom by + about an angstrom, and a translation - against the X-ray target (§15.2), by Ceres' L-BFGS for 30 + iterations. Then one B shift per residue. Each of the two is **kept only where it lowers R-free by more + than 0.001**: a deposited model was refined atom by atom already, and moving its chains or shifting its + B factors residue by residue then only makes it worse. The placement stage needs no dictionary. +2. **Restrained refinement.** Every atom's position and isotropic B, against the X-ray target and the + geometry of the dictionary (§15.4-§15.5), in weight trials chosen by R-free (§15.7). Residues the + dictionary does not describe, and residues with an atom it does not know, are **held fixed**: they stay + in the structure factors, in the bulk-solvent mask and in the repulsions the refined atoms see, but + they do not move. So do atoms of zero occupancy. + +### 15.2 The X-ray target + +The target is the maximum-likelihood amplitude target: for an acentric reflection the Rice distribution of +the observed amplitude $F_o$ given the model's $|F_\mathrm{model}|$, for a centric one its centric +counterpart, + +$$ +-\ln L_\mathrm{acentric} = \ln\Sigma + \frac{F_o^2 + D^2|F_\mathrm{model}|^2}{\Sigma} - \ln I_0\!\left(\frac{2F_o D|F_\mathrm{model}|}{\Sigma}\right),\qquad +-\ln L_\mathrm{centric} = \tfrac12\ln\Sigma + \frac{F_o^2 + D^2|F_\mathrm{model}|^2}{2\Sigma} - \ln\cosh\frac{F_o D|F_\mathrm{model}|}{\Sigma}, +$$ + +(the terms that depend on neither the model nor the parameters left out) with $D$ the fraction of the model that is right and $\Sigma$ the variance that the rest of the structure and +the measurement leave (§15.3), summed over the working reflections. $\ln I_0$ and $I_1/I_0$ are evaluated +with the polynomial approximations of Abramowitz and Stegun, through the exponentially scaled form for large +arguments so that nothing overflows. The amplitudes are the French-Wilson $|F|$ of §10.8 - the ones the +written reflection file carries - and the free reflections are those of §10.7. + +$F_\mathrm{model}$ is the model of §14.2, + +$$ +F_\mathrm{model} = k\,e^{-\mathbf{s}^\top\mathbf{B}\mathbf{s}/4}\left(F_\mathrm{calc} + k_\mathrm{sol}\,e^{-B_\mathrm{sol}s^2/4}\,F_\mathrm{mask}\right), +$$ + +with the atoms' $F_\mathrm{calc}$ computed with the dictionary's riding hydrogens included (§15.4). The +scale, the anisotropic B, the bulk solvent and the likelihood's parameters are fitted at the start of each +**macro-cycle** (§15.3) and held fixed while the minimiser runs; the mask is drawn for the model as it is +at that point. + +**The gradient by the FFT method.** The atoms' density is placed on a grid with a Gaussian blur (the one +REFMAC-compatible density calculation uses, chosen for the lowest B a macro-cycle can reach) and transformed +once; the crystal's symmetry is composed in reciprocal space from the transform of one copy, as for the rigid +body (§14.8). The derivative of the target with respect to every structure factor, taken back to real space +by one inverse transform through the adjoint of that composition, is a map; each atom's derivatives with +respect to $x, y, z$ and $B$ are that map summed against the derivatives of its own blurred density. One +evaluation of the target and of the gradient of every atom thus costs a density gather, two FFTs and a +gradient gather, whatever the number of atoms. The derivative of a riding hydrogen is passed to its parent. + +### 15.3 Scale, bulk solvent and the likelihood parameters + +**Scale and solvent.** The flat bulk-solvent mask is drawn with the REFMAC radii (probe 1.0 Å, shrink 0.8 Å) +on a grid of its own, with a spacing of $d_\mathrm{min}/3$ or 0.4 Å, whichever is finer (coarser only where +a cell would need more than $2^{26}$ points). On the atoms' density grid the shrink is lost at low +resolution: at 3 Å its 1 Å step leaves no grid point within the shrink radius. $k_\mathrm{sol}$ and +$B_\mathrm{sol}$ are searched over their physical range on the working reflections, and the overall scale +and anisotropic B are fitted at each point of that search (§14.2) - at the points of its fine pass by +**robust least squares**: +iteratively reweighted (four reweightings), each reflection's residual $r = F_o - |F_\mathrm{model}|$ +weighted by $1/\sqrt{1 + (r/c)^2}$ with $c$ the median $|r|$ of the unweighted fit. The fit then minimises +the pseudo-Huber loss $\sum 2c^2\left(\sqrt{1 + (r/c)^2} - 1\right)$, quadratic for the bulk of the +reflections and linear for the few a model cannot explain - a beam-stop shadow, an ice ring, a missing part +of the model - which ordinary least squares would let set the scale of all the rest. When the B factors +are refined, the overall isotropic part of the scale's B is moved into the atoms (and into $B_\mathrm{sol}$, +so that $F_\mathrm{model}$ stays as it is); left in the scale, it and the atoms' B drift against each other +as the B factors are refined. + +With these three choices - riding hydrogens, the fine mask grid and the robust scale - the R-work and +R-free of a model as given (`REFINE_R_WORK_START`, `REFINE_R_FREE_START`) agree with REFMAC 5.8's 0-cycle +values on the same model and data within 0.002 on twelve of fourteen depositions from 1.1 to 3.6 Å, and +within 0.006 on the other two (`tools/battery/refine_parity.py`). + +**$D$ and $\Sigma$.** The two parameters of the likelihood are fitted by maximum likelihood in resolution +bins of the free set (100 free reflections to a bin, at most 30 bins; where the free set has fewer than +100 reflections, the working ones join it so that the likelihood still has a scale), and interpolated +linearly in $1/d^2$ between the bins' centres: + +- $\beta$ on the **free** reflections, as the whole spread of $F_o$ about $D F_\mathrm{model}$ with variance + $\varepsilon\beta$ - the measurement error included; +- $D$ then on the **working** reflections of the bin, with the variance + $\Sigma = \varepsilon\beta + 2\sigma^2$ for an acentric reflection and $\varepsilon\beta + \sigma^2$ for a + centric one, $\sigma$ the amplitude's own error. + +$\Sigma$ is also the variance of each reflection in the target. Where the measurement error is a large part +of the spread - the weak high-resolution shells - it is counted twice, and $D$ falls towards 0 there, with +the figure of merit. This division of the two parameters between the working and the free set is the one +REFMAC5 describes; measured on REFMAC 5.8's own output it reproduces its $D$ by bin and its overall figure of +merit (the mean over the working reflections) to about 0.01 on most depositions. + +### 15.4 The dictionary and the topology + +The restraints are those of the **CCP4 Monomer Library**, a subset of which is compiled into rugnux +(`image_analysis/structure_refinement/monomers/`, regenerated by `update_monomers.py`; nothing is read from +disk, and `$CLIBD` is never consulted): the 20 standard amino acids and MSE; the DNA and RNA nucleotides +(DA, DC, DG, DT, A, C, G, U); ATP, ADP, AMP, GTP, GDP and GMP; DMSO; the PEG fragments PEG, PGE, PG4 and 1PE; +glycerol, ethylene glycol, MPD, acetate, sulfate and phosphate; water; and the ions Na, K, Mg, Ca, Mn, Fe²⁺, +Fe³⁺, Co, Ni, Cu, Zn, Cd, Cl, Br and I - 62 compounds - with the library's links and modifications and the +energy types those compounds use. + +The model's topology is built from it by GEMMI (`prepare_topology`): the links between residues (the peptide +and cis-peptide, the nucleic-acid backbone, disulfides, and the model's LINK records where the library +describes them), the modifications those links imply, and per residue the bonds, angles, torsions, chiral +volumes and planes. Torsions are restrained where the dictionary gives them an esd, except the main-chain +φ/ψ and the sugar puckers it lists as alternatives. The dictionary's **hydrogens** are added to every +residue it describes (waters excepted), placed by GEMMI from the dictionary's geometry and moved with their +parents: they take part in the structure factors and in no restraint, and are placed again on their +parents at every macro-cycle. The refined model is written with them. + +### 15.5 Geometry, metal and B restraints + +Every restraint is a squared normalised deviation, $E = ((x - x_\mathrm{ideal})/\sigma)^2$, with its +analytic derivatives: + +- **bonds and angles**, ideal and esd from the dictionary; +- **torsions**, modulo the dictionary's period, with derivatives in the form of Blondel and Karplus, free of + the singularities of the arccos form; +- **chiral volumes** $(\mathbf{a}_1 - \mathbf{c})\cdot((\mathbf{a}_2 - \mathbf{c})\times(\mathbf{a}_3 - \mathbf{c}))$, + $\sigma = 0.2$ ų; +- **planes**: the distances of the atoms from their least-squares plane (the plane's normal the smallest + eigenvector of the atoms' scatter, by Jacobi rotations); the plane is optimal, so the derivative of each + distance carries no term from the plane moving; +- **repulsions** of every pair of atoms closer than its contact distance, symmetry mates included, that no + bond, angle or 1-4 relation joins and that are not alternative conformations of each other, with at least + one of the two refinable: $E = ((d_0 - d)/0.2\,\text{Å})^2$ below the contact distance $d_0$ and nothing + above it, $d_0$ the sum of the energy types' van der Waals radii, 2.6 Å where both atoms can hydrogen-bond + and 1.9 Å between a metal and N, O or S. The list is rebuilt at every macro-cycle. + +**Metal sites.** A metal ion is restrained to the donor atoms (O, N or S) that the model's LINK / +`_struct_conn` records bind it to. Where the library has a link for the pair - its ZN-CYS, ZN-HISND, +ZN-HISNE, FE-CYS, FE-HISND and FE-HISNE - that link is used as it is, with the modification it makes to +the donor's residue (the proton the bond replaces). Every other record becomes a link of one bond at the +distance of the library's own statistics of metal sites (its `metals.json`, carried as `monomers/metals.cif`) +for the metal, the donor element and the site's coordination number - the number of such records the ion +has - interpolated linearly between two tabulated coordination numbers (rows resting on at least 20 sites +count; σ is the row's standard deviation, at least 0.02 Å). Only the bond is restrained, not +the angles at the metal. A contact without a record stays a repulsion, and a record to a symmetry mate is +left out, as every bond to a symmetry mate is. The rmsd of these bonds is reported apart from the covalent +bonds' (`REFINE_RMSD_METAL_BONDS`). + +**B restraints.** The B factors of two atoms are restrained to each other in proportion to their size, by +the symmetric Kullback-Leibler divergence of two isotropic Gaussians of variances $B_i$ and $B_j$: + +$$ +E = \frac{(B_i - B_j)^2}{\kappa^2 B_i B_j}, +$$ + +a difference restraint whose $\sigma$ is $\kappa\sqrt{B_iB_j}$, so that atoms of high B may differ more than +atoms of low B. $\kappa$ is 0.1 for bonded main-chain atoms and 0.133 for main-chain atoms bonded to a +common atom, 0.2 and 0.3 for the side chains and the rest, and 0.5 for atoms within 4 Å of each other (waters +among them, symmetry mates included) that no bond or angle relates. No atom's B goes below 2 Ų or above +the ceiling of §15.1. + +### 15.6 The minimiser + +Positions and B factors are minimised in turn, each by an L-BFGS of rugnux's own (`RefinementMinimiser`), +with the two-loop recursion of Nocedal and of Liu and Nocedal, a memory of 20 pairs and a backtracking line +search from the full step to sufficient decrease, its first trial moving no position by more than 0.3 Å and +no B by more than 10 Ų. What sets it apart from a textbook L-BFGS: + +- **Its memory is kept from one macro-cycle to the next.** A macro-cycle is a few dozen iterations on a target + whose scale and likelihood parameters were refitted in between; starting each from what the earlier ones + learnt of the curvature, instead of from a steepest-descent step, is what lets two macro-cycles of a + weight trial get as far as many restarted ones. Each weight trial starts with an empty memory. +- **The B steps are scaled by their curvature.** The initial inverse Hessian of the B factors is a + diagonal: the X-ray target's curvature in each atom's B - its shape from the likelihood's curvature in + $|F_\mathrm{model}|$ and the atom's own scattering, summed in resolution shells, calibrated once per trial + against central differences of the gradient along a random $\pm1$ direction - plus the B restraints' + curvature, each element at least a thousandth of the median. The positions take none: their restraints' + curvature is that of stiff bonds and angles, which move atoms together, and as a diagonal it slows the soft + (torsion and X-ray) directions rather than scaling them. + +The diagonal is rescaled at each iteration by the latest pair (Oren-Shanno scaling in that metric). A solve +stops after its iteration budget (30 for the positions and 20 for the B factors per macro-cycle) or where an +iteration lowers the target by less than $10^{-7}$ of it. + +### 15.7 The weights, decided by trial + +The refined target is $w\,T_\mathrm{X\text{-}ray} + E_\mathrm{geometry}$ for the positions and +$w_B\,T_\mathrm{X\text{-}ray} + E_B$ for the B factors. Rather than fixing $w$ and $w_B$ in advance, rugnux +runs **short refinements at several weights and keeps the one that does best on the free set** - a weight +chosen late, on the evidence, instead of early at a rule. + +The starting weights balance the gradients: $w_0 = |\nabla E_\mathrm{geometry}| / |\nabla T_\mathrm{X\text{-}ray}|$, +the geometry's gradient taken on the model shaken by 0.1 Å (a refined model's geometry would give none), and +$w_{B,0}$ likewise from the B restraints on the B factors shaken by 20%. Each **trial** is two macro-cycles +(the scale, solvent and likelihood refitted, then the positions, then the B factors) from where it starts. +A trial is **within bounds** where its rmsd from the dictionary is at most 0.02 Å for bonds and 2° for +angles; one trial **displaces** another where it is within bounds and its R-free is lower by more than 0.001 +(differences smaller than that do not decide, and the trial listed first - on a ladder, the tighter +geometry - is kept). The trials +are: + +1. **The ladder from the placed model**: $w = w_0 \times \{1/1024, 1/128, 1/16\}$, each with the B factors + held and refined at $w_{B,0}$ - six trials. +2. **The trajectory**: up to eight steps, each a set of trials from where the last step ended - the weights + as they are, and in turn the positions' weight halved and doubled (halved because the geometry loosens at + one weight as the model improves, until it leaves the bounds) or the B factors' weight divided and + multiplied by four. A step is taken while it lowers R-free or R-work by more than 0.001 and its R-free + stays within 0.001 of the best so far, and the trajectory ends at its best step. On a deposited model, + whose free set the depositor's refinement has already fitted, R-work falls while R-free rises, and the + trajectory ends where it began. +3. **The final ladder**: the trajectory's weights times $\{0.5, 1, 2\}$, kept where one displaces it. + +$w$ never exceeds $w_0/8$. The trials of one set run side by side - on one GPU, on several, or on the CPU's +threads - each from the same start and taking nothing from the one before it, so the result does not depend +on how many run at once. + +The refinement can be cut short from outside (the broker's deadline or cancel, Ctrl+C): the check is made +between the minimiser's iterations and before every trial, the trials that finished are chosen among as +usual, and the report says so. Cut before the first trial, the model is the placement's. + +### 15.8 Maps of the refined model + +The refined model's map coefficients, `_refined_maps.mtz`, are $2mF_o - DF_c$ and $mF_o - DF_c$ as in +§14.4, but with $m$ and $D$ those of the refinement's own likelihood (§15.3) rather than the shell +$\sigma_A$ of §14.4, so the maps carry the weights the target was minimised with; $m$ is +$I_1(X)/I_0(X)$, $X = 2F_oD|F_\mathrm{model}|/\Sigma$, for an acentric reflection and $\tanh(X/2)$ for a +centric one. Where the merge kept the Bijvoet pairs, the anomalous difference map is made on the refined +model's phases too, and read by the blob finder below. + +### 15.9 Unexplained density: blobs + +The $mF_o - DF_c$ map of the refined model is searched for what the model leaves unexplained: the connected +regions above 3 σ that peak above 4 σ, found by flood fill (GEMMI's `find_blobs_by_flood_fill`, the method +of Coot's "unmodelled blobs"), each moved to the copy nearest the model and measured - peak, volume, the +density integrated above the cutoff, the nearest atom, the polar (N, O) atoms at 2.4-3.4 Å and the N, O or S +at 1.8-2.6 Å, and the anomalous map at its peak. Each gets one reading, by rules on what is measured: + +| Reading | Rule | +|---|---| +| `MODEL_ERROR` | within 1.6 Å of a model atom: the model's own atom out of place, or a missing alternative conformation | +| `ION` | at least 1.8 Å from the model, and an anomalous peak of at least 4 σ, or two metal-like contacts and a peak of at least 8 σ | +| `LIGAND` | larger than 1.5 times a single atom's blob at this resolution (a sphere of radius $d_\mathrm{min}/2$) and than 8 ų, at least 2.2 Å from the model | +| `WATER` | at least 2.2 Å from the model with at least one polar contact | +| `UNASSIGNED` | anything else | + +They are listed strongest first (by the integrated density) in `_blobs.json`, and the five strongest +in the report. A blob read as `LIGAND` is density of ligand size where the model has nothing: a buffer +molecule or an unmodelled side chain is that as well as the compound that was soaked. The readings point a +person at the density; they do not identify a compound. + +### 15.10 GPU and CPU + +On a CUDA build the target, the gradient and the restraints are evaluated on the card - the density gather, +cuFFT, the composition of the symmetry and the target per reflection, the adjoint through a CSR list, the inverse transform and one warp per atom for its gradient; the restraints in one kernel whose +per-atom gradient slots are summed in a fixed order; the robust scale fit and the fine solvent mask on the +card as well - with no floating-point atomics, so the result is the same from run to run. The weight trials +of a set run on as many cards as there are, each trial on an engine of its own. Where the case does not fit +the card's budget, or with `--no-gpu`, or on a build without CUDA (macOS included: the refinement has no +Metal path), the CPU engine computes the same target, to rounding, several times more slowly (seconds on a +GPU, typically 2-15 minutes on the CPU). + +### 15.11 Not yet + +What the refinement does not do yet - each a known gap, not a design choice: + +- **Custom ligands and other monomer libraries.** Only the 62 compounds of §15.4 are restrained. Any other + residue is held fixed (it stays in $F_\mathrm{calc}$, the mask and the repulsions); no restraints are + generated for a new ligand, and neither a user's restraint CIF nor an external monomer library can be given. +- **TLS** refinement. +- **Anisotropic ADPs**: an anisotropic B is replaced by its isotropic equivalent, and only isotropic B is + refined. +- **NCS restraints**: copies are refined independently. +- **Twin refinement**: the target is not a twinned one, also where the twin hypothesis' merge was written. +- **Water building**: waters are neither picked nor removed; the blob finder only points at candidates. +- **Alternative conformations and occupancies**: existing conformers are refined in place; no occupancy is + refined and no conformer is added. +- **Rotamer and peptide-flip search**, and **Ramachandran restraints**. +- **Real-space refinement**, and **jelly-body or reference-model restraints**. +- **A Metal (Apple GPU) path**: on macOS the refinement runs on the CPU. +- **Validation beyond geometry**: the report gives the bond and angle rmsd, the metal-bond rmsd and the + number of clashes, not a MolProbity-style analysis (Ramachandran, rotamer and clashscore statistics). + +## References + +- R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", *Acta Cryst.* **A34** (1978), 791-809 (the gradient by the FFT method, §15.2). +- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and its figure of merit, §15.2, §15.8). +- G. N. Murshudov, P. Skubák, A. A. Lebedev, N. S. Pannu, R. A. Steiner, R. A. Nicholls, M. D. Winn, F. Long & A. A. Vagin, "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (D from the working set and the variance from the free, §15.3; the Kullback-Leibler form of the B restraints, §15.5). +- R. J. Read, "Improved Fourier coefficients for maps using phases from partial structures with errors", *Acta Cryst.* **A42** (1986), 140-149 (the m, D weighting of the map coefficients, §15.8). +- V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", *Acta Cryst.* **A51** (1995), 880-887, and V. Y. Lunin, P. V. Afonine & A. G. Urzhumtsev, "Likelihood-based refinement. I. Irremovable model errors", *Acta Cryst.* **A58** (2002), 270-282 (the likelihood's parameters from the free set, §15.3). +- J.-S. Jiang & A. T. Brünger, "Protein hydration observed by X-ray diffraction: solvation properties of penicillopepsin and neuraminidase crystal structures", *J. Mol. Biol.* **243** (1994), 100-115, and A. Fokine & A. Urzhumtsev, "Flat bulk-solvent model: obtaining optimal parameters", *Acta Cryst.* **D58** (2002), 1387-1392 (the flat bulk-solvent mask and its parameters, §15.3). +- P. V. Afonine, R. W. Grosse-Kunstleve & P. D. Adams, "A robust bulk-solvent correction and anisotropic scaling procedure", *Acta Cryst.* **D61** (2005), 850-855 (the grid search over the solvent parameters, §15.3). +- P. J. Huber, "Robust estimation of a location parameter", *Ann. Math. Statist.* **35** (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, "Deterministic edge-preserving regularization in computed imaging", *IEEE Trans. Image Process.* **6** (1997), 298-311 (the robust scale fit and its smooth, pseudo-Huber loss, §15.3). +- A. A. Vagin, R. A. Steiner, A. A. Lebedev, L. Potterton, S. McNicholas, F. Long & G. N. Murshudov, "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15.4). +- M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (the topology, the riding hydrogens and the blob search, §15.4, §15.9). +- A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5). +- M. M. Harding, "Geometry of metal-ligand interactions in proteins", *Acta Cryst.* **D57** (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", *Acta Cryst.* **D62** (2006), 678-682; F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5). +- J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782, and D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15.6). +- A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). +- P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7). +- A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free). +- P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9). +- M. Abramowitz & I. A. Stegun, *Handbook of Mathematical Functions*, National Bureau of Standards (1964), 9.8.1-9.8.4 (the Bessel functions of the target, §15.2). + +The full citations, with DOIs, are in [ACKNOWLEDGEMENT.md](ACKNOWLEDGEMENT.md). diff --git a/docs/GPU_DATA_ANALYSIS.md b/docs/GPU_DATA_ANALYSIS.md index 61c69f57c..98b78452c 100644 --- a/docs/GPU_DATA_ANALYSIS.md +++ b/docs/GPU_DATA_ANALYSIS.md @@ -206,6 +206,29 @@ structure factors above; the likelihood scoring runs on the host. The search is [§14.10](CPU_DATA_ANALYSIS_DECISIONS.md#1410-molecular-replacement-placing-a-model-the-rigid-body-cannot), its keys and decision rules in [Molecular replacement](RUGNUX_REPORT.md#molecular-replacement). +## Refinement + +The restrained refinement of a placed model +([§15](CPU_DATA_ANALYSIS_REFINEMENT.md)) evaluates its target and gradients on the card where its engine fits +the card's budget, and on the CPU otherwise; the two compute the same target, to rounding, and the GPU's +result is the same from run to run (no floating-point atomics). + +- **Target and gradient** (`RefinementGPU.cu`) - the atoms' density gathered on the grid, cuFFT, the + crystal's symmetry composed and the likelihood target evaluated per reflection, the derivative map formed + through the adjoint of that composition and transformed back, and each atom's derivatives gathered from it, + one warp per atom. [§15.2](CPU_DATA_ANALYSIS_REFINEMENT.md#152-the-x-ray-target) +- **Restraints** (`RefinementGPU.cu`) - the geometry and B restraints in one kernel, each writing one gradient + slot per atom it involves, summed per atom in a fixed order. + [§15.5](CPU_DATA_ANALYSIS_REFINEMENT.md#155-geometry-metal-and-b-restraints) +- **Robust scale and fine solvent mask** (`ModelScaleGPU.cu`, `ModelMaskGPU.cu`) - the reweighted scale fit and + the mask on its own fine grid, as model validation's engines compute them. + [§15.3](CPU_DATA_ANALYSIS_REFINEMENT.md#153-scale-bulk-solvent-and-the-likelihood-parameters) + +The weight trials of one set run side by side, each on an engine of its own, spread over every card of the +machine; the outcome does not depend on how many run at once +([§15.7](CPU_DATA_ANALYSIS_REFINEMENT.md#157-the-weights-decided-by-trial)). The refinement has no Metal +engine: on macOS it runs on the CPU. + ## Apple GPUs (Metal) A macOS build has no CUDA, and three engines run on the Apple GPU instead (none under diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 0d1966043..c2709e461 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -172,7 +172,8 @@ maps. Where the merge kept the Bijvoet split (a rotation merge always does), an difference map is made too and read, not written: the report names its strongest sites (`ANOMALOUS_SITE_01`…`10`) and the map's mean height at the model's anomalous scatterers - every atom from phosphorus up - as one number for the anomalous signal (`ANOMALOUS_SCATTERER_MEAN_SIGMA`). -The structure itself is not refined; the model is re-fractionalized into +The model is not refined for these numbers (a placed model is refined afterwards, on the side - see +[Refinement](RUGNUX_MODEL_AND_REFERENCE.md#refinement)); it is re-fractionalized into the data cell and then placed as **one rigid body**, so a deposited model from a crystal that is not quite isomorphous still sits where the density is. The placement runs on the GPU where there is one and on the CPU otherwise, with nothing to set; the two agree to rounding, not bit for bit. @@ -355,6 +356,17 @@ Mode — `--mode ` (default `mx`): | `calibration` | Determine the detector geometry from powder rings; writes `.poni` and `.json` | | `refine` | Refine the `--model` against merged structure factors (an MTZ or a structure-factor mmCIF) and process nothing; see [Refining a model against structure factors](RUGNUX_MODEL_AND_REFERENCE.md#refining-a-model-against-structure-factors-rugnux---mode-refine) (`--refine-score-only`, `--refine-d-min`) | +Refinement (`--mode refine`): the columns are found, not named - amplitudes with their sigmas (F/SIGF, +FP/SIGFP, ...) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by +French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, +or the minority value where 0 is the majority, as phenix writes it), drawn by rugnux where the file has none. +The model is taken as placed in the data's cell, and refined in the data's space group. + +| Option | Description | +| --- | --- | +| `--refine-score-only` | Score the model as given and refine nothing: R-work/R-free, the scale, the bulk solvent, the likelihood and the geometry, each term under `REFINE_SCORE_` (`--developer` prints them all) | +| `--refine-d-min ` | Leave out reflections finer than this (Å) | + Calibration (`--mode calibration`): | Option | Description | @@ -470,9 +482,9 @@ Scaling and merging: | `--scaling-iterations ` | Cap on the per-frame scaling iterations; the loop stops earlier once the scales settle, and a merge that hits the cap is flagged `SCALING_NOT_CONVERGED` (default: 100) | | `-z, --reference ` | Reference data of the same crystal form — an MTZ or a PDB structure-factor mmCIF (`-sf.cif`), gzipped or not, recognised by content; `--reference-mtz` is the same option: fixes the space group, its setting and the cell, resolves the [indexing ambiguity](#the-indexing-ambiguity), hands over the R-free set and reports CCref. Not a scale anchor | | `--reference-column