From 85e0d2da954641efc839135d2d7f14d333634957 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 27 Oct 2026 13:20:01 +0100 Subject: [PATCH] Refinement: the solvent audit of the refined model, as REFINE_SOLVENT_ keys and _solvent.json RefineModel runs AnalyseSolventSites on the refined model (and with --refine-score-only on the model as given) after the blob search, in the pipeline after a merge and in --mode refine alike. The maps step now also makes the 2mFo-DFc grid, which the audit reads at each site and which is released straight after. Nothing reads the audit: the refined model, maps, blobs and every R are unchanged (1HR2 --mode refine: refined model file identical to the build without it). The report gets REFINE_SOLVENT_SITES, _LABELS, _BEST, _DISAGREE, _AMBIGUOUS, _IONS and _IONS_DISAGREE and a paragraph listing the ions (label -> best-supported species, bands, neighbours, the first 40); rugnux writes every site to _solvent.json beside _blobs.json. The broker's live path does not carry it to the writer yet. Docs: CPU_DATA_ANALYSIS_REFINEMENT.md 15.10 describes what is measured and every term of the score; RUGNUX_REPORT.md the keys and the JSON; ACKNOWLEDGEMENT.md credits the Cat_Wiz bands and the Auffinger-group surveys, Echols 2014, Morshed 2015, Carugo 2014, Brown & Altermatt 1985, Brese & O'Keeffe 1991 (R0 as tabulated by Zheng et al. 2017), Mueller et al. 2003 and CheckMyMetal. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37 --- docs/ACKNOWLEDGEMENT.md | 39 +++++++++++++ docs/CPU_DATA_ANALYSIS_REFINEMENT.md | 57 ++++++++++++++++++- docs/RUGNUX.md | 1 + docs/RUGNUX_REPORT.md | 15 ++++- .../structure_refinement/Refinement.cpp | 18 ++++-- .../structure_refinement/Refinement.h | 8 +++ rugnux/ResultReport.cpp | 48 ++++++++++++++++ rugnux/Rugnux.h | 1 + rugnux/RugnuxOutputFiles.cpp | 46 +++++++++++++++ rugnux/RugnuxScaleMerge.cpp | 1 + 10 files changed, 226 insertions(+), 8 deletions(-) diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 064924338..26414f48f 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -601,6 +601,45 @@ a cutoff, found by flood fill as Coot's "unmodelled blobs" finds them, in GEMMI' (`find_blobs_by_flood_fill`): P. Emsley, B. Lohkamp, W. G. Scott and K. Cowtan, "Features and development of Coot" (2010), Acta Cryst. D66, 486-501 [doi:10.1107/S0907444910007493](https://doi.org/10.1107/S0907444910007493). +**Solvent audit** — the waters and ions of the refined model are scored by their surroundings with the +stereochemical criteria of the surveys of ions at nucleic acids by P. Auffinger's group: the coordination bands +1.9-2.3 / 2.3-2.6 / 2.6-3.2 Å, the Mg²⁺ exclusion zone and the clash shells of N. Naleem, A. Henning-Knechtel, +S. Kirmizialtin and P. Auffinger, "Cat_Wiz: a stereochemistry-guided toolkit for locating, diagnosing, and +annotating Mg2+ ions in RNA structures" (2026), Nucleic Acids Res. 54, gkag120 +[doi:10.1093/nar/gkag120](https://doi.org/10.1093/nar/gkag120); the Mg-O and Mg-N distances and the +2.3-3.4 Å exclusion zone of F. Leonarski, A. Henning-Knechtel, S. Kirmizialtin, E. Ennifar and P. Auffinger, +"Principles of ion binding to RNA inferred from the analysis of a 1.55 Å resolution bacterial ribosome +structure – Part I: Mg2+" (2025), Nucleic Acids Res. 53, gkae1148 +[doi:10.1093/nar/gkae1148](https://doi.org/10.1093/nar/gkae1148); the Na⁺, K⁺ and water distances and +coordination numbers of F. Leonarski, L. D'Ascenzo and P. Auffinger, "Nucleobase carbonyl groups are poor +Mg2+ inner-sphere binders but excellent monovalent ion binders—a critical PDB survey" (2019), RNA 25, 173-192 +[doi:10.1261/rna.068437.118](https://doi.org/10.1261/rna.068437.118), and the distance bands of their +2017 survey (above). The ligand sphere of the bond-valence sums and the reading of halides by the donors +around them follow N. Echols, N. Morshed, P. V. Afonine, A. J. McCoy, M. D. Miller, R. J. Read, +J. S. Richardson, T. C. Terwilliger and P. D. Adams, "Automated identification of elemental ions in +macromolecular crystal structures" (2014), Acta Cryst. D70, 1104-1114 +[doi:10.1107/S1399004714001308](https://doi.org/10.1107/S1399004714001308), and the halide's positive and +negative neighbours N. Morshed, N. Echols and P. D. Adams, "Using support vector machines to improve elemental +ion identification in macromolecular crystal structures" (2015), Acta Cryst. D71, 1147-1158 +[doi:10.1107/S1399004715004241](https://doi.org/10.1107/S1399004715004241); the chloride's shell of donors +at up to 3.4 Å is O. Carugo's, "Buried chloride stereochemistry in the Protein Data Bank" (2014), BMC +Struct. Biol. 14, 19 [doi:10.1186/s12900-014-0019-8](https://doi.org/10.1186/s12900-014-0019-8). The +bond-valence sums are I. D. Brown and D. Altermatt's, "Bond-valence parameters obtained from a systematic +analysis of the Inorganic Crystal Structure Database" (1985), Acta Cryst. B41, 244-247 +[doi:10.1107/S0108768185002063](https://doi.org/10.1107/S0108768185002063), with the R₀ of N. E. Brese and +M. O'Keeffe, "Bond-valence parameters for solids" (1991), Acta Cryst. B47, 192-197 +[doi:10.1107/S0108768190011041](https://doi.org/10.1107/S0108768190011041), as tabulated by H. Zheng, +K. M. Langner, G. P. Shields, J. Hou, M. Kowiel, F. H. Allen, G. Murshudov and W. Minor, "Data mining of +iron(II) and iron(III) bond-valence parameters, and their relevance for macromolecular crystallography" +(2017), Acta Cryst. D73, 316-325 [doi:10.1107/S2059798317000584](https://doi.org/10.1107/S2059798317000584), +weighted by occupancy as P. Müller, S. Köpke and G. M. Sheldrick weight them, "Is the bond-valence method +able to identify metal atoms in protein structures?" (2003), Acta Cryst. D59, 32-37 +[doi:10.1107/S0907444902018000](https://doi.org/10.1107/S0907444902018000); their normalised vector sum and +its outlier bound are CheckMyMetal's: H. Zheng, D. R. Cooper, P. J. Porebski, I. G. Shabalin, K. B. Handing +and W. Minor, "CheckMyMetal: a macromolecular metal-binding validation tool" (2017), Acta Cryst. D73, 223-233 +[doi:10.1107/S2059798317001061](https://doi.org/10.1107/S2059798317001061). The score that combines them is +our own. + ## Software and computing methods Decoding bitshuffle+LZ4 images on the GPU, rather than decompressing them on the host and uploading diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index e63f0e730..f88f3ac4d 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -281,7 +281,53 @@ in the report. A blob read as `LIGAND` is density of ligand size where the model molecule or an unmodelled side chain is that as well as the compound that was soaked. The readings point a person at the density; they do not identify a compound. -### 15.10 GPU and CPU +### 15.10 The solvent audit + +Every water and every single-atom ion of the refined model (and, with `--refine-score-only`, of the model as +given) is measured in its surroundings and scored as each of water, Mg²⁺, Na⁺, K⁺, Ca²⁺ and Cl⁻ (NH₄⁺ is +scored too, and only reported). It is an audit: the model is not changed, and nothing else reads it. + +**What is measured.** The model's N and O atoms around the site, symmetry mates included, counted in the +coordination bands of Cat_Wiz - 1.9-2.3 Å (Mg-like), 2.3-2.6 Å (Na, Ca), 2.6-3.2 Å (K, water) - and in the +clash shells 3.2-3.4 and 3.4-3.6 Å beyond them. Each neighbour is typed from the dictionary's energy type +(donor, acceptor or both) and from the chemistry of its residue: Lys NZ, Arg NE/NH1/NH2 and a polypeptide's +N-terminal N are positive; Asp/Glu carboxylate O, a C-terminal carboxylate and a nucleotide's phosphate O are +negative; His ring N and the Asn/Gln amide ends are both donor and acceptor, as the model's orientation of +them is a guess. Also measured: the shortest distance between two neighbours closer than 2.6 Å (the +octahedron's cis O…O), the nearest positive and negative group, the other ions within 4.2 and 6 Å, the +bond-valence sum $V = \sum_j q_j \exp((R_0 - d_j)/0.37)$ over the N and O within 3.5 Å for each cation +($R_0$ of Brese & O'Keeffe, $q_j$ the neighbour's occupancy) with its normalised vector sum (CheckMyMetal's +nVECSUM), and the 2mFo-DFc, mFo-DFc and anomalous maps at the site, in their rms. + +**The score.** For each species, a plain sum of terms in log units: + +| Term | What it is | +|---|---| +| first shell | the neighbours in the species' band against the coordination number it holds (Mg 6, Na 5-6, K 6-10, Ca 6-8, Cl 3-5, NH₄⁺ 4), quadratic; for water only more than four, or none at all, costs; for Mg, Na and Ca a missing ligand costs little (0.5), as the model may simply not have that water | +| distances | each neighbour in the band against the species' own distance (Mg-O 2.07 ± 0.04 Å, Mg-N 2.19 ± 0.08 Å, Na 2.41, Ca 2.37, K and water 2.8, NH₄⁺ 2.9, Cl 3.2 Å) | +| contacts it does not make | 2 per N or O closer than the species' shell (closer than 2.9 Å for Cl), and for Mg per N or O in its exclusion zone 2.3-3.4 Å | +| chemistry | a cation: 2 per donor-only neighbour in its shell and per positive group within 3.5 Å; Cl: 2 per acceptor-only neighbour, per negative group within 3.2 Å, and 2 where there is neither a positive group nor a donor N within 3.5 Å; water: 2 per hydrogen bond beyond two donated or two accepted (neighbours at 2.6-3.4 Å) | +| bond valence | for a cation, $V$ against its charge $z$, Gaussian with a spread of $0.25z$ widened by the neighbours' distance errors, and the vector sum against CheckMyMetal's outlier bound 0.23 | +| electrons | where the difference map at the site is above +3 σ, +1 for the species with more electrons than the label (18 against 10) and -1 for the others; below -3 σ the reverse; in between -0.5 for a species of another electron count, as B and occupancy can hide eight electrons | +| anomalous | an anomalous peak of 4 σ or more: +2 for K, Ca and Cl, -2 for the others | + +A site bound to another cation (closer than 2.6 Å) shares that cation's other ligands as neighbours - the +cis waters of a Mg(H₂O)₆ are 2.9 Å apart - and those are left out of its score. **Coordinate error**: each +atom's is taken as its rms displacement $\sqrt{B/8\pi^2}$ times $d_\mathrm{min}/10$ Å (about 0.1 Å at B = 20 Ų +and 2 Å); a neighbour's band is widened by the error of the distance and its distance term is widened by it +too, so that where the coordinates are poor the species' shells overlap and the score stops telling them +apart instead of telling them apart wrongly. The best of water, Mg, Na, K, Ca and Cl is reported, and +`AMBIGUOUS` where the runner-up is within 2. The audit counts a site as disagreeing with its label where +the best species is another one, not ambiguously. + +**Caveats.** The refined model's metal-site distances are restrained (§15.5) where its LINK records name +the bonds, which pulls an ion's shell towards the label it carries - the first pitfall of every survey of +ions in the PDB. `--mode refine --refine-score-only` audits the model as given instead. The electron count +is read only from the residual difference density, which a B factor or an occupancy can absorb, and the map +values are not yet compared with the neighbouring atoms'. Per site, everything above is in +`_solvent.json`; the counts are the `REFINE_SOLVENT_` keys of the report. + +### 15.11 GPU and CPU On a CUDA build the target, the gradient and the restraints are evaluated on the card - the density gather, cuFFT, the composition of the symmetry and the target per reflection, the adjoint through a CSR list, the inverse transform and one warp per atom for its gradient; the restraints in one kernel whose @@ -292,7 +338,7 @@ the card's budget, or with `--no-gpu`, or on a build without CUDA (macOS include Metal path), the CPU engine computes the same target, to rounding, several times more slowly (seconds on a GPU, typically 2-15 minutes on the CPU). -### 15.11 Not yet +### 15.12 Not yet What the refinement does not do yet - each a known gap, not a design choice: @@ -304,7 +350,8 @@ What the refinement does not do yet - each a known gap, not a design choice: refined. - **NCS restraints**: copies are refined independently. - **Twin refinement**: the target is not a twinned one, also where the twin hypothesis' merge was written. -- **Water building**: waters are neither picked nor removed; the blob finder only points at candidates. +- **Water building and ion identification**: waters are neither picked nor removed and no ion is relabelled; + the blob finder points at candidates and the solvent audit (§15.10) scores the waters and ions there are. - **Alternative conformations and occupancies**: existing conformers are refined in place; no occupancy is refined and no conformer is added. - **Rotamer and peptide-flip search**, and **Ramachandran restraints**. @@ -332,6 +379,10 @@ What the refinement does not do yet - each a known gap, not a design choice: - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (weights chosen among refinements by R-free within geometry bounds, and the overall B moved into the atoms, §15.3, §15.7). - A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures", *Nature* **355** (1992), 472-475 (R-free). - P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9). +- N. Naleem, A. Henning-Knechtel, S. Kirmizialtin & P. Auffinger, "Cat_Wiz: a stereochemistry-guided toolkit for locating, diagnosing, and annotating Mg2+ ions in RNA structures", *Nucleic Acids Res.* **54** (2026), gkag120; F. Leonarski, L. D'Ascenzo & P. Auffinger, *Nucleic Acids Res.* **45** (2017), 987-1004 and *RNA* **25** (2019), 173-192; F. Leonarski, A. Henning-Knechtel, S. Kirmizialtin, E. Ennifar & P. Auffinger, *Nucleic Acids Res.* **53** (2025), gkae1148 (the coordination bands, distances and exclusion zone of the solvent audit, §15.10). +- N. Echols et al., "Automated identification of elemental ions in macromolecular crystal structures", *Acta Cryst.* **D70** (2014), 1104-1114, and N. Morshed, N. Echols & P. D. Adams, "Using support vector machines to improve elemental ion identification in macromolecular crystal structures", *Acta Cryst.* **D71** (2015), 1147-1158 (the ligand sphere and the halide rules, §15.10). +- H. Zheng et al., "CheckMyMetal: a macromolecular metal-binding validation tool", *Acta Cryst.* **D73** (2017), 223-233 (the vector sum, §15.10); O. Carugo, "Buried chloride stereochemistry in the Protein Data Bank", *BMC Struct. Biol.* **14** (2014), 19 (the chloride's shell, §15.10). +- I. D. Brown & D. Altermatt, *Acta Cryst.* **B41** (1985), 244-247, and N. E. Brese & M. O'Keeffe, "Bond-valence parameters for solids", *Acta Cryst.* **B47** (1991), 192-197 (the bond-valence sums, §15.10). - M. Abramowitz & I. A. Stegun, *Handbook of Mathematical Functions*, National Bureau of Standards (1964), 9.8.1-9.8.4 (the Bessel functions of the target, §15.2). The full citations, with DOIs, are in [ACKNOWLEDGEMENT.md](ACKNOWLEDGEMENT.md). diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index 07aea9754..28c5b0eba 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -137,6 +137,7 @@ myrun_model.cif / .pdb the model as placed, in the cell and space group of my myrun_refined_model.cif the refined model (and .pdb) myrun_refined_maps.mtz map coefficients on the refined model myrun_blobs.json the unexplained difference-density blobs +myrun_solvent.json the refined model's waters and ions, each scored by its surroundings ``` A stills run writes the same merged files (its `.hkl` holds the merged reflections) and, on a diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index d88d3e004..92241d051 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -1202,13 +1202,26 @@ Where a model was placed and refined ([Refinement](RUGNUX_MODEL_AND_REFERENCE.md | `REFINE_BULK_SOLVENT_K_SOL`, `REFINE_BULK_SOLVENT_B_SOL` | *dev*. The flat bulk solvent of the refined model | | `REFINE_BLOBS` | The blobs of the refined mFo-DFc map (above 3 σ, peaking above 4 σ) | | `REFINE_BLOBS_LIGAND`, `REFINE_BLOBS_WATER`, `REFINE_BLOBS_ION`, `REFINE_BLOBS_MODEL_ERROR` | How many of them are read as each ([§15.9](CPU_DATA_ANALYSIS_REFINEMENT.md#159-unexplained-density-blobs)); the rest are unassigned | +| `REFINE_SOLVENT_SITES` | The waters and single-atom ions of the refined model the solvent audit scored ([§15.10](CPU_DATA_ANALYSIS_REFINEMENT.md#1510-the-solvent-audit)) | +| `REFINE_SOLVENT_LABELS`, `REFINE_SOLVENT_BEST` | How many of them carry each label (`HOH 265 MG 42`), and how many each species is the best-supported one for | +| `REFINE_SOLVENT_DISAGREE`, `REFINE_SOLVENT_AMBIGUOUS` | The sites whose best-supported species is not their label (water, Mg, Na, K, Ca or Cl), not ambiguously; the sites where the runner-up is within 2 of the best | +| `REFINE_SOLVENT_IONS`, `REFINE_SOLVENT_IONS_DISAGREE` | The same for the sites not labelled water | -A prose paragraph after them lists the five strongest blobs with their reading, peak, volume, position and +A prose paragraph after the blob keys lists the five strongest blobs with their reading, peak, volume, position and nearest atom; all of them are in `_blobs.json`, strongest first, one object per blob: `kind`, `centroid` (Å, Cartesian, the copy nearest the model), `peak_sigma`, `volume_A3`, `score_sigma_A3` (the density above the cutoff integrated), `nearest_atom`, `nearest_atom_distance_A`, `polar_contacts`, `metal_contacts` and `anomalous_sigma`. +A second paragraph, after the solvent keys, lists the ions, label → best-supported species, their N and O in the bands +1.9-2.3/2.3-2.6/2.6-3.2 Å and their neighbours closer than 3.4 Å. Every site is in `_solvent.json`, +in model order, one object per site: `label`, `chain`, `seqid`, `altloc`, `xyz`, `occupancy`, `b_iso`, +`sigma_xyz_A` (the coordinate error assumed), `bands` (N and O by distance), `donors`, `acceptors`, +`donor_acceptors` (closer than 3.4 Å), `min_ligand_distance_A`, `nearest_positive_A`, `nearest_negative_A`, +`positive_within_3.5`, `negative_within_3.2`, `nearest_ion_A`, `ions_within_4.2`, `ions_within_6`, `bvs` and +`nvecsum` (per cation: `MG`, `NA`, `K`, `CA`), `map_sigma`, `difference_sigma`, `anomalous_sigma` (null where +there is no such map), `score` (per species), `best`, `ambiguous`, `agrees` and `neighbours`. + **The score of the model as given** follows, every term under `REFINE_SCORE_`: `REFINE_SCORE_R_WORK`, `REFINE_SCORE_R_FREE`, `REFINE_SCORE_N_WORK` and `REFINE_SCORE_N_FREE` by default, and with `--developer` the overall scale (`REFINE_SCORE_K_OVERALL`), its isotropic and anisotropic B (`REFINE_SCORE_B_ISO`, diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index 86a4d72a8..7178e1943 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -869,9 +869,9 @@ void MakeMaps(Context &ctx, const std::vector &merged, double {"PHDELWT", 'P'}, {"FOM", 'W'}, {"FREE", 'I'}}; for (const auto &[label, type] : columns) mtz.columns.push_back({label, type, type == 'H' ? 0 : 1, {}}); - gemmi::AsuData> diff; - diff.unit_cell_ = ctx.cell; - diff.spacegroup_ = ctx.sg; + gemmi::AsuData> diff, fwt_data; + diff.unit_cell_ = fwt_data.unit_cell_ = ctx.cell; + diff.spacegroup_ = fwt_data.spacegroup_ = ctx.sg; for (size_t m = 0; m < ctx.refl.Size(); m++) { const double fo = ctx.refl.fo[m], fc = std::abs(fmodel[m]); const double mw = fom[m], d = ctx.rows.d[m]; @@ -889,13 +889,16 @@ void MakeMaps(Context &ctx, const std::vector &merged, double for (size_t c = 0; c < mtz.columns.size(); c++) mtz.columns[c].values.push_back(row[c]); diff.v.push_back({ctx.refl.hkl[m], std::polar(static_cast(delfwt), static_cast(phi))}); + fwt_data.v.push_back({ctx.refl.hkl[m], std::polar(static_cast(fwt), static_cast(phi))}); } result.maps = maps; // The difference map beside the anomalous one below: neither reads the other. - std::future difference_map = std::async(std::launch::async, [&diff, &result] { + std::future difference_map = std::async(std::launch::async, [&diff, &fwt_data, &result] { const std::array size = gemmi::get_size_for_hkl(diff, {{0, 0, 0}}, 3.0); result.difference_map = std::make_shared>(MapFromFPhi(gemmi::get_f_phi_on_grid(diff, size, true))); result.difference_map_rms = gemmi::calculate_data_statistics(result.difference_map->data).rms; + result.map = std::make_shared>(MapFromFPhi(gemmi::get_f_phi_on_grid(fwt_data, size, true))); + result.map_rms = gemmi::calculate_data_statistics(result.map->data).rms; }); // The anomalous difference map: F(+) - F(-) on the model's phase turned back by 90 degrees, read on the + @@ -1301,6 +1304,13 @@ RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector(ctx.st); diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index f9eadd055..c33d2e1e4 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -20,6 +20,7 @@ #include "DensityBlobs.h" #include "ModelValidation.h" // ModelMaps #include "RefinementRestraints.h" // GeometryStatistics +#include "SolventSites.h" class Logger; @@ -143,6 +144,10 @@ struct RefinementResult { std::shared_ptr model; // the refined model, in the data's cell and frame std::shared_ptr maps; // its 2mFo-DFc and mFo-DFc coefficients (as _maps.mtz holds them) + // The 2mFo-DFc map of the refined model and its rms, for reading the solvent sites in (SolventSites.h); released + // once they are read. + std::shared_ptr> map; + double map_rms = 0; // The mFo-DFc map of the refined model and its rms, for finding what it leaves unexplained (DensityBlobs.h). std::shared_ptr> difference_map; double difference_map_rms = 0; @@ -151,6 +156,9 @@ struct RefinementResult { double anomalous_map_rms = 0; // What the refined model leaves unexplained in the difference map (FindDensityBlobs). std::vector blobs; + // The refined model's waters and ions and the species their surroundings support (AnalyseSolventSites): an + // audit, nothing in the model is changed by it. + std::vector solvent; }; // `placed`: the model in the data's cell and space group. `merged`: the merge, with French-Wilson amplitudes diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 0596612a9..00c10b6cd 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -455,6 +456,52 @@ namespace { return h; } + // The solvent audit of the refined model (SolventSites.h): REFINE_SOLVENT_ keys. Counts by label as modelled + // and by the species the surroundings support best; a site disagrees where the best is another species, not + // ambiguously. + void AddSolventKeys(ReportSection &s, const std::vector &sites, const std::string &output_prefix) { + std::map labels, best; + constexpr int LISTED_IONS = 40; // the rest are in the JSON + int disagree = 0, ambiguous = 0, ions = 0, ions_disagree = 0; + std::string ion_rows; + for (const SolventSite &site : sites) { + labels[site.label]++; + best[SolventSpeciesName(site.best)]++; + ambiguous += site.ambiguous; + const bool differs = SolventLabelScored(site) && !site.ambiguous && !SolventLabelAgrees(site); + disagree += differs; + if (!SolventLabelIsWater(site)) { + ions++; + ions_disagree += differs; + if (ions <= LISTED_IONS) + ion_rows += fmt::format("\n {:<4} {}{:<6} -> {:<3}{} bands {}/{}/{}, {}", site.label, site.chain, + site.seqid, SolventSpeciesName(site.best), + site.ambiguous ? " (ambiguous)" : "", site.n_19_23, site.n_23_26, + site.n_26_32, site.neighbours.empty() ? "no N/O within 3.4 A" : site.neighbours); + } + } + if (ions > LISTED_IONS) + ion_rows += fmt::format("\n ... and {} more", ions - LISTED_IONS); + auto counts = [](const std::map &m) { + std::string t; + for (const auto &[name, n] : m) + t += fmt::format("{}{} {}", t.empty() ? "" : " ", name, n); + return t; + }; + Add(s, KeyInt("REFINE_SOLVENT_SITES", static_cast(sites.size()))); + Add(s, KeyText("REFINE_SOLVENT_LABELS", counts(labels))); + Add(s, KeyText("REFINE_SOLVENT_BEST", counts(best))); + Add(s, KeyInt("REFINE_SOLVENT_DISAGREE", disagree)); + Add(s, KeyInt("REFINE_SOLVENT_AMBIGUOUS", ambiguous)); + Add(s, KeyInt("REFINE_SOLVENT_IONS", ions)); + Add(s, KeyInt("REFINE_SOLVENT_IONS_DISAGREE", ions_disagree)); + Add(s, Prose(fmt::format( + " The waters and ions of the refined model, each scored by its surroundings (an audit: the\n" + " model is not changed). Per site in {}_solvent.json; the ions, label -> best-supported\n" + " species, N/O in the bands 1.9-2.3/2.3-2.6/2.6-3.2 A, and their neighbours:{}", + output_prefix, ion_rows.empty() ? std::string(" none") : ion_rows))); + } + // The refinement's keys (Refinement.h): REFINE_ of the refinement of a placed model and of --mode refine, with every term of the // score of the model as given under REFINE_SCORE_. void AddRefinementKeys(ReportSection &s, const RefinementResult &rf, const std::string &output_prefix) { @@ -518,6 +565,7 @@ namespace { " R_FREE above remain the model as given. The refinement decided nothing about the\n" " processing. The strongest unexplained blobs of its difference map ({}_blobs.json):{}", output_prefix, output_prefix, top.empty() ? std::string(" none") : top))); + AddSolventKeys(s, rf.solvent, output_prefix); } if (rf.ok) { const RefinementScore &sc = rf.score_start; diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 8ce4222a2..9c8277b65 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -418,6 +418,7 @@ struct RugnuxOutputs { std::optional refined_model; // _refined_model.cif/.pdb (ProcessConfig::refine) std::shared_ptr refined_maps; // _refined_maps.mtz std::optional> blobs; // _blobs.json + std::optional> solvent; // _solvent.json std::optional unmerged; // _unmerged.mtz std::optional unmerged_partials; // _unmerged_partials.mtz std::string detector_jpeg; // _detector.jpg; empty: none drawn diff --git a/rugnux/RugnuxOutputFiles.cpp b/rugnux/RugnuxOutputFiles.cpp index e7d3e7a4e..596fe2385 100644 --- a/rugnux/RugnuxOutputFiles.cpp +++ b/rugnux/RugnuxOutputFiles.cpp @@ -3,9 +3,12 @@ #include "RugnuxOutputFiles.h" +#include #include #include +#include + #include "../writer/FileWriter.h" #include "../writer/WriteModel.h" #include "../writer/WriteRugnuxFiles.h" @@ -23,6 +26,46 @@ void WriteDensityBlobs(const std::vector &blobs, const std::string logger.Info("Difference-map blobs written to {}", path); } +// The solvent audit of the refined model (SolventSites.h), one entry per site in model order. +void WriteSolventSites(const std::vector &sites, const std::string &path, Logger &logger) { + std::ofstream f(path, std::ofstream::out | std::ofstream::trunc); + if (!f) { + logger.Warning("Cannot open {} for writing", path); + return; + } + // NaN and infinity (no such map, no such neighbour) as null + auto num = [](double v) { return std::isfinite(v) ? nlohmann::json(v) : nlohmann::json(); }; + const char *cations[BVS_CATIONS] = {"MG", "NA", "K", "CA"}; + nlohmann::json list = nlohmann::json::array(); + for (const SolventSite &s : sites) { + nlohmann::json bvs, nvecsum, score; + for (int c = 0; c < BVS_CATIONS; c++) { + bvs[cations[c]] = s.bvs[c]; + nvecsum[cations[c]] = s.nvecsum[c]; + } + for (int k = 0; k < SOLVENT_SPECIES; k++) + score[SolventSpeciesName(static_cast(k))] = s.score[k]; + list.push_back({{"label", s.label}, {"chain", s.chain}, {"seqid", s.seqid}, + {"altloc", s.altloc ? std::string(1, s.altloc) : std::string()}, + {"xyz", {s.pos.x, s.pos.y, s.pos.z}}, {"occupancy", s.occupancy}, {"b_iso", s.b_iso}, + {"sigma_xyz_A", s.sigma_xyz}, + {"bands", {{"below_1.9", s.n_below_19}, {"1.9-2.3", s.n_19_23}, {"2.3-2.6", s.n_23_26}, + {"2.6-3.2", s.n_26_32}, {"3.2-3.4", s.n_32_34}, {"3.4-3.6", s.n_34_36}}}, + {"donors", s.donors}, {"acceptors", s.acceptors}, {"donor_acceptors", s.donor_acceptors}, + {"min_ligand_distance_A", num(s.min_ligand_distance)}, + {"nearest_positive_A", num(s.nearest_positive)}, {"nearest_negative_A", num(s.nearest_negative)}, + {"positive_within_3.5", s.positive_contacts}, {"negative_within_3.2", s.negative_contacts}, + {"nearest_ion_A", num(s.nearest_ion)}, {"ions_within_4.2", s.ions_within_42}, + {"ions_within_6", s.ions_within_6}, {"bvs", bvs}, {"nvecsum", nvecsum}, + {"map_sigma", num(s.map_sigma)}, {"difference_sigma", num(s.difference_sigma)}, + {"anomalous_sigma", num(s.anomalous_sigma)}, {"score", score}, + {"best", SolventSpeciesName(s.best)}, {"ambiguous", s.ambiguous}, + {"agrees", SolventLabelAgrees(s)}, {"neighbours", s.neighbours}}); + } + f << nlohmann::json{{"sites", list}}.dump(1) << "\n"; + logger.Info("Solvent sites written to {}", path); +} + // The files of one output pass at `prefix`, all but its _process.h5. void WriteOutputs(const RugnuxOutputs &o, const std::string &prefix, const OutputProvenance &provenance, size_t nthreads, Logger &logger) { @@ -48,6 +91,8 @@ void WriteOutputs(const RugnuxOutputs &o, const std::string &prefix, const Outpu WriteModelMaps(*o.refined_maps, prefix + "_refined", logger); if (o.blobs) WriteDensityBlobs(*o.blobs, prefix + "_blobs.json", logger); + if (o.solvent) + WriteSolventSites(*o.solvent, prefix + "_solvent.json", logger); if (o.unmerged) { WriteMtz(*o.unmerged, prefix + "_unmerged.mtz"); logger.Info("Unmerged observations written to {}_unmerged.mtz", prefix); @@ -194,6 +239,7 @@ ReportDocument WriteRefineOutputs(const RefineOnlyResult &result, const std::str WritePlacedModel(*rf.model, cell, *sg, prefix + "_refined", logger); WriteModelMaps(*rf.maps, prefix + "_refined", logger); WriteDensityBlobs(rf.blobs, prefix + "_blobs.json", logger); + WriteSolventSites(rf.solvent, prefix + "_solvent.json", logger); } try { std::ofstream file(prefix + "_report.txt"); diff --git a/rugnux/RugnuxScaleMerge.cpp b/rugnux/RugnuxScaleMerge.cpp index fc1cc42de..542a7462e 100644 --- a/rugnux/RugnuxScaleMerge.cpp +++ b/rugnux/RugnuxScaleMerge.cpp @@ -5114,6 +5114,7 @@ bool Rugnux::ScaleMergeAndSymmetry(PipelineLocals &p) { result.outputs->refined_model = PlacedModelOutput{refined.model, p.cell, p.space_group}; result.outputs->refined_maps = refined.maps; result.outputs->blobs = refined.blobs; + result.outputs->solvent = refined.solvent; Note(fmt::format("refined{}: R-work {:.3f}, R-free {:.3f}, rmsd bonds {:.3f} A, {} blobs", refined.cancelled ? " (cut short)" : "", refined.r_work, refined.r_free, refined.geometry.rmsd_bonds, refined.blobs.size()));