From 8364cfbc40bddb30a31bf4fc8c28d5fa4fe77243 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 21 Sep 2026 14:09:39 +0200 Subject: [PATCH] rugnux: an R that compares between runs, beside the one that makes the maps R_WORK and R_FREE score the data against a model scaled by k_overall, a symmetry-constrained anisotropic B and a flat bulk solvent. That scale is deliberate - a free per-shell rescale would reshape each map's radial amplitude profile and a batch of maps would no longer be comparable - but it can only bend one way with resolution, so whatever a dataset's own radial profile does that k*exp(-B s^2) cannot follow is reported as R. Two reductions of one crystal whose amplitudes differ radially therefore differ in R_FREE for a reason that has nothing to do with either fitting the model: measured on one such pair of runs, R_FREE moved a median +0.042 over eleven sets where phenix.model_vs_data, which scales per shell, moved +0.002. Report a second reading of the same fit: R_MODEL_SHELL_SCALED, one free scale per resolution shell of the merge table, with R_MODEL as the same sum without it and MODEL_RADIAL_MISFIT (RMS of ln k_shell) as the size of the rescale, so the reason a comparison is or is not readable is on the page. Over all the reflections, not the free 5%: nothing here is refined, so work and free estimate the same quantity and the split only costs precision. Nothing is written from it. On two sets, every output file - both maps, the anomalous map, the map coefficients, the MTZ, the HKL and the CIF - is byte for byte what the same commit produced without this change, and R_WORK and R_FREE are unchanged. On the pair above, the new R moves a median +0.0005 and holds every set inside 0.0065 except the one that is a real regression, which it puts first at +0.0174 where the old metric ranked it seventh. The battery reports it beside R_free, and scores neither. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT --- docs/CPU_DATA_ANALYSIS_DECISIONS.md | 2 + docs/RUGNUX_ADVANCED.md | 4 +- docs/RUGNUX_REPORT.md | 32 ++++++++++++++ rugnux/ModelValidation.cpp | 66 +++++++++++++++++++++++++++++ rugnux/ModelValidation.h | 23 ++++++++++ rugnux/ResultReport.cpp | 32 ++++++++++++++ tests/ResultReportTest.cpp | 31 +++++++++++++- tools/battery/README.md | 15 +++++++ tools/battery/report.py | 27 +++++++++++- tools/battery/score.py | 7 +++ 10 files changed, 235 insertions(+), 4 deletions(-) diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index ecc059048..f248d736d 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -126,6 +126,8 @@ $ This is the standard, few-parameter scaling model used by refinement programs. No free-form per-resolution-shell rescale is applied: such a rescale is dataset-specific and reshapes each map's radial amplitude profile differently, which would make maps from a multi-dataset campaign no longer directly comparable. +That decision has a cost, and it is paid by the R-factors rather than by the maps. $k\exp(-\mathbf{s}^\mathsf{T} B \mathbf{s})$ can only bend one way with resolution, so whatever a dataset's radial amplitude profile does that this shape cannot follow is reported as R — which makes R a reading of *this* dataset and not a quantity comparable with another reduction of the same crystal. Both uses are wanted, so both are served, separately: the maps and `R_WORK` / `R_FREE` keep the scale above, and a second reading, `R_MODEL_SHELL_SCALED`, applies one free scale per resolution shell to the *same* fit and is reported beside them, with `MODEL_RADIAL_MISFIT` saying how much that rescale had to do. Nothing is written from the second reading; it never reaches a map, a map coefficient or a decision (§14.3, and RUGNUX_REPORT). + The fit sees the **working reflections only**; the parameters it returns are then applied to every reflection, free ones included, so that R-free (§14.3) is computed against them. The parameters are few, but they are fitted by minimising the very sum R is made of, and a free reflection that helped choose them is no longer held out. ### 14.3 R-work and R-free diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 9df974f79..077cdf1fd 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -143,7 +143,9 @@ cell or no usable space group, does not fail the run: it is logged, and the resu maps cannot be mistaken for one that was never given `--model`. Either way the results report carries a **`5. MODEL VALIDATION`** section: `R_WORK=` / `R_FREE=` -with their reflection counts, the bulk-solvent and overall scale parameters, the mean 2mFo−DFc density +with their reflection counts, `R_MODEL_SHELL_SCALED=` and `MODEL_RADIAL_MISFIT=` beside them (the R +that compares between two runs, and why the other two do not — see +[RUGNUX_REPORT](RUGNUX_REPORT.md)), the bulk-solvent and overall scale parameters, the mean 2mFo−DFc density at the atom centres, the reindexing operators the written reflections were brought into the model's frame with, and `MAPS_PREFIX=`; or `MODEL_VALIDATION= NOT_PERFORMED` with `MODEL_VALIDATION_REASON=` when the model could not be used. A run given no `--model` has no such diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 9b50f8697..cc0ffa34b 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -551,6 +551,38 @@ The null's own numbers — `MODEL_FIT_STATISTIC`, `MODEL_FIT_VALUE`, the `MODEL_ the `MODEL_INDEXING_MARGIN*` keys — are written with `--developer`; the default report carries the verdict (`MODEL_FIT`, `MODEL_FIT_SIGMA`, `MODEL_DECISIONS_TAKEN` and the two decisions). +### Comparing two runs: `R_MODEL_SHELL_SCALED` + +`R_WORK` and `R_FREE` describe *this* dataset against *this* model, and that is all they describe. +They are **not comparable with another run's**. The model is scaled to the data by an overall factor +and a symmetry-constrained anisotropic *B* — a shape that can only bend one way with resolution, kept +that way on purpose so that a batch of maps stays on one scale (see +[CPU_DATA_ANALYSIS_DECISIONS](CPU_DATA_ANALYSIS_DECISIONS.md)). Whatever the amplitudes' own radial +profile does that this shape cannot follow is then reported as R. Two reductions of one crystal whose +merged amplitudes have different radial profiles therefore differ in `R_FREE` for a reason that has +nothing to do with either fitting the model better, and by more than a real change in the data moves it. + +`R_MODEL_SHELL_SCALED` is the same sum with that taken out: one free scale per resolution shell — the +merge table's own shells — fitted on the shell's reflections, so only the agreement *inside* each +shell is left. `R_MODEL` is the same sum without the per-shell scale, so the gap between the two is +what the radial profile cost, and `MODEL_RADIAL_MISFIT` is the size of the rescale that closed it +(the RMS of ln *k*shell about its mean). When the misfit moves between two runs, `R_FREE` +between those two runs cannot be read; `R_MODEL_SHELL_SCALED` can. + +Both R values are over **all** the reflections, not the free 5%. Nothing here is refined — the +coordinates, the B-factors and the occupancies are the model's own, and only the scale and the +placement are fitted — so work and free estimate the same quantity (they differ by a median 0.001 +over our corpus) and the split buys no cross-validation while costing a factor of √20 in precision. + +None of this touches the maps: the map coefficients, the `.ccp4` files and `R_WORK` / `R_FREE` are +computed from the scale described above and are unchanged by these keys. + +| key | meaning | +|---|---| +| `R_MODEL`, `R_MODEL_REFLECTIONS` | R over every reflection, on the scale the maps use | +| `R_MODEL_SHELL_SCALED` | The same, with one free scale per resolution shell — **the number to compare between runs** | +| `MODEL_RADIAL_MISFIT` | How large that per-shell rescale had to be: RMS of its logarithm. Near zero means the maps' own scale already described the radial profile, and then the two R values agree | + ### CC(model, data) Beside the R-factors, section 5 carries the **correlation of the merged intensities with the placed, diff --git a/rugnux/ModelValidation.cpp b/rugnux/ModelValidation.cpp index 783b19849..df9ac2c7c 100644 --- a/rugnux/ModelValidation.cpp +++ b/rugnux/ModelValidation.cpp @@ -916,6 +916,72 @@ ModelValidationResult ValidateAgainstModel(const std::vector & result.cc_model_overall, result.cc_model_n, result.cc_model_shells.back().d_min, result.cc_model_shells.back().cc, result.cc_model_shells.back().n, result.cc_model_shells.back().sigma); + + // --- the same R with the radial profile taken out, and how much of it there was --- + // R above is what the maps are made of and is read WITHIN a run; this is the one to read + // BETWEEN two. The scale the maps use can only bend as k*exp(-B s^2) (see the note in + // fit_model, and the reason it stays that way), so where two reductions of one crystal differ + // in the radial profile of their amplitudes, the part of the difference that shape cannot + // follow lands in R - measured on this corpus, enough to move R by more than a real change in + // the data does. Giving the scale one free factor per shell removes the radial profile and + // only the radial profile: what is left is the agreement inside each shell, which is the + // question "do these data fit this model better or worse than those did". + // + // On ALL the reflections, not the free set. Nothing here is refined - the coordinates, the B + // factors and the occupancies are the model's own - so work and free estimate the same + // quantity (measured: the two differ by a median 0.001 over the corpus), and the 5% split + // therefore buys no cross-validation while costing a factor of sqrt(20) in precision. Read on + // all of them, the number is worth about +-0.002 per set instead of +-0.010, which is the + // difference between seeing a real change and seeing the sampling of a free set. + // + // Nothing is written from any of this. fmodel, the maps, the MTZ and the R-factors above are + // untouched; it is a second reading of the fit that has already happened. + double num = 0, den = 0, plain_num = 0, plain_den = 0; + int n_scored = 0; + std::vector sum_oc(report_shell_d_min.size(), 0), sum_cc(report_shell_d_min.size(), 0); + for (const auto &hv : fmodel.v) { + const auto it = obs_by_hkl.find(hkl_key(hv.hkl)); + if (it == obs_by_hkl.end()) continue; + const size_t bin = shell_of(it->second.d); + if (bin >= sum_oc.size()) continue; + sum_oc[bin] += it->second.F * std::abs(hv.value); + sum_cc[bin] += static_cast(std::abs(hv.value)) * std::abs(hv.value); + } + // The shell's scale by least squares on its own reflections. One parameter per shell against + // the thousands of reflections in it, so scoring the same reflections it was fitted on lowers + // R by far less than the difference it is there to expose. + std::vector ln_k; + ln_k.reserve(report_shell_d_min.size()); + for (size_t i = 0; i < sum_oc.size(); i++) + if (sum_cc[i] > 0 && sum_oc[i] > 0) + ln_k.push_back(std::log(sum_oc[i] / sum_cc[i])); + for (const auto &hv : fmodel.v) { + const auto it = obs_by_hkl.find(hkl_key(hv.hkl)); + if (it == obs_by_hkl.end()) continue; + const size_t bin = shell_of(it->second.d); + if (bin >= sum_oc.size()) continue; + const double Fo = it->second.F, Fc = std::abs(hv.value); + const double k = (sum_cc[bin] > 0 && sum_oc[bin] > 0) ? sum_oc[bin] / sum_cc[bin] : 1.0; + num += std::fabs(Fo - k * Fc); + den += Fo; + plain_num += std::fabs(Fo - Fc); + plain_den += Fo; + ++n_scored; + } + if (den > 0 && ln_k.size() >= 2) { + result.r_model = plain_num / plain_den; + result.r_model_shell_scaled = num / den; + result.n_model = n_scored; + result.shell_scale_bins = static_cast(ln_k.size()); + const double mean_ln_k = std::accumulate(ln_k.begin(), ln_k.end(), 0.0) / ln_k.size(); + double var = 0; + for (double v : ln_k) var += (v - mean_ln_k) * (v - mean_ln_k); + result.radial_misfit = std::sqrt(var / ln_k.size()); + logger.Info("Model validation: R(all reflections) {:.4f}, with a per-shell scale {:.4f} " + "over {} shells; radial misfit {:.3f}", + result.r_model, result.r_model_shell_scaled, result.shell_scale_bins, + result.radial_misfit); + } } // --- sigma_A weighting: the maps are 2mFo-DFc and mFo-DFc, not 2Fo-Fc and Fo-Fc --- diff --git a/rugnux/ModelValidation.h b/rugnux/ModelValidation.h index 7fc6831a8..4aceab134 100644 --- a/rugnux/ModelValidation.h +++ b/rugnux/ModelValidation.h @@ -3,6 +3,7 @@ #pragma once +#include #include #include #include @@ -31,6 +32,28 @@ struct ModelValidationResult { double k_overall = 0.0; int n_work = 0; int n_free = 0; + + // R over ALL the reflections - r_model as the maps are scaled, r_model_shell_scaled with one free + // scale per resolution shell instead - and how far those per-shell scales had to depart from a + // flat one. The pair exists to be read BETWEEN runs, where r_work / r_free cannot be. + // + // The scale the maps carry can only bend as k*exp(-B s^2), deliberately (see ModelValidation.cpp), + // so where two reductions of one crystal differ in the radial profile of their amplitudes, the + // part of the difference that shape cannot follow is reported as R - by more, measured, than a + // real change in the data moves it. A per-shell scale removes the radial profile and only the + // radial profile, leaving the agreement inside each shell. + // + // All the reflections and not the free set, because nothing here is refined: the coordinates, the + // B factors and the occupancies are the model's own, so work and free estimate the same quantity + // and the 5% split buys no cross-validation while costing a factor of sqrt(20) in precision. + double r_model = NAN; + double r_model_shell_scaled = NAN; + int n_model = 0; + // The RMS of ln(k_shell) about its own mean. Near zero says k*exp(-B s^2) already described this + // dataset's radial profile, and then the two R-factors agree; a large value says it did not, and + // is worth seeing on its own - it is the gap between the two R-factors, named. + double radial_misfit = NAN; + int shell_scale_bins = 0; double mean_atom_density_sigma = 0.0; // mean 2mFo-DFc value at atom centres, in sigma std::string maps_prefix; // where the .ccp4 / _maps.mtz were written diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 9e9960675..08b3d8ff3 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -34,6 +34,9 @@ namespace { // whether choosing the partner reflections differently could repair it. // 13: CC_MODEL_* in section 5 - the correlation of the merged intensities with the placed model, // by resolution shell, which only a run given a model can report. + // 14: R_MODEL / R_MODEL_SHELL_SCALED / MODEL_RADIAL_MISFIT in section 5 - R over all the + // reflections, with and without one free scale per shell, and the size of that rescale. + // R_WORK and R_FREE are unchanged and still describe the model the maps were made from. // 8: every line that is not `KEY= value` and not blank starts with `#`, warnings included // (`# WARNING:`); the header gains RUGNUX_DOWNLOAD and BUILD_CXX_FLAGS, and RUGNUX_GIT is // the full build-time hash. @@ -1631,6 +1634,15 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, Add(s, KeyReal("R_FREE", mv.r_free, "{:.4f}")); Add(s, KeyInt("R_WORK_REFLECTIONS", mv.n_work)); Add(s, KeyInt("R_FREE_REFLECTIONS", mv.n_free)); + // The pair to read BETWEEN two runs, where R_WORK / R_FREE cannot be - see the prose at + // the end of this section and the note in ModelValidation.h. + if (std::isfinite(mv.r_model_shell_scaled)) { + Add(s, KeyReal("R_MODEL", mv.r_model, "{:.4f}")); + Add(s, KeyReal("R_MODEL_SHELL_SCALED", mv.r_model_shell_scaled, "{:.4f}")); + Add(s, KeyInt("R_MODEL_REFLECTIONS", mv.n_model)); + Add(s, KeyReal("MODEL_RADIAL_MISFIT", mv.radial_misfit, "{:.3f}")); + Add(s, KeyInt("MODEL_RADIAL_SHELLS", mv.shell_scale_bins, true)); + } Add(s, KeyReal("BULK_SOLVENT_K_SOL", mv.k_sol, "{:.3f}", true)); Add(s, KeyReal("BULK_SOLVENT_B_SOL", mv.b_sol, "{:.1f}", true)); Add(s, KeyReal("SCALE_OVERALL", mv.k_overall, "{:.4f}", true)); @@ -1746,6 +1758,26 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, " structure's and are a data-quality reading, not a refinement result. The scale and the\n" " placement are both fitted on the working reflections alone, so the free set is held out\n" " of the fit and not only out of the sum.")); + if (std::isfinite(mv.r_model_shell_scaled)) + Add(s, Prose(fmt::format( + " R_WORK and R_FREE above describe THIS dataset against this model, and that is what they\n" + " are for. They are not comparable with another dataset's, or with the same dataset\n" + " reduced differently: the model is scaled to the data by an overall factor and an\n" + " anisotropic B, a shape that can only bend one way with resolution, and whatever the\n" + " amplitudes' own radial profile does that this shape cannot follow is reported as R.\n" + " Two reductions with different radial profiles therefore differ in R for a reason that\n" + " has nothing to do with how well either fits the model.\n" + " R_MODEL_SHELL_SCALED ({:.4f}) is the same R with that removed - one free scale per\n" + " resolution shell of the table above, so only the agreement INSIDE each shell is left -\n" + " and it is the number to compare between runs. R_MODEL ({:.4f}) is the same sum without\n" + " the per-shell scale, so the gap between the two is what the radial profile cost.\n" + " Both are over every reflection, not the free set: nothing here is refined, so work and\n" + " free measure the same thing and using all of them is simply more precise.\n" + " MODEL_RADIAL_MISFIT ({:.3f}) is the size of that per-shell rescale, as the RMS of its\n" + " logarithm over {} shells: near zero means the scale already described the profile and\n" + " the two R values agree. Nothing is written from any of this - the maps, the map\n" + " coefficients and the R-factors above are untouched by it.", + mv.r_model_shell_scaled, mv.r_model, mv.radial_misfit, mv.shell_scale_bins))); if (mv.fit_tested) Add(s, Prose(fmt::format( " MODEL_FIT says whether these data accept the model: the same model was refitted from\n" diff --git a/tests/ResultReportTest.cpp b/tests/ResultReportTest.cpp index 12d61a22b..037a92118 100644 --- a/tests/ResultReportTest.cpp +++ b/tests/ResultReportTest.cpp @@ -421,8 +421,11 @@ TEST_CASE("ResultReport_ModelValidationSection", "[Diagnostics]") { CHECK(dev_text.find("\nMODEL_FIT_NULL_REPLICATES= 5\n") != std::string::npos); } CHECK(text.find("\nMODEL_FIT_SIGMA= +4.12\n") != std::string::npos); - // A run that measured no CC(model, data) writes no key for it. + // A run that measured no CC(model, data) writes no key for it, and the shell-scaled R shares + // its shells, so it is absent too. CHECK(text.find("CC_MODEL_OVERALL=") == std::string::npos); + CHECK(text.find("R_MODEL_SHELL_SCALED=") == std::string::npos); + CHECK(text.find("MODEL_RADIAL_MISFIT=") == std::string::npos); CHECK(text.find("CC_MODEL_CONFIRMED_TO_D_MIN=") == std::string::npos); CHECK(text.find("\nMODEL_DECISIONS_TAKEN= ENANTIOMORPH\n") != std::string::npos); // The hand is ASSUMED from the model, never determined: merged intensities cannot see it. @@ -473,6 +476,32 @@ TEST_CASE("ResultReport_ModelValidationSection", "[Diagnostics]") { CHECK(cc_text.find(" 1.80 0.0412 2110 +1.9") != std::string::npos); CHECK(cc_text.find("The table reads in ONE direction only") != std::string::npos); + // The comparable R rides on the same shells. R_WORK and R_FREE are what they always were - the + // maps' own scale - and the new keys sit beside them rather than in place of them. + ModelValidationResult with_r = with_cc; + with_r.r_model = 0.2011; + with_r.r_model_shell_scaled = 0.1734; + with_r.n_model = 17610; + with_r.radial_misfit = 0.312; + with_r.shell_scale_bins = 3; + result.model_validation = with_r; + const auto r_text = RenderResultReport("p", "in.h5", x, result); + CHECK(r_text.find("\nR_WORK= 0.1823\n") != std::string::npos); + CHECK(r_text.find("\nR_FREE= 0.2145\n") != std::string::npos); + CHECK(r_text.find("\nR_MODEL= 0.2011\n") != std::string::npos); + CHECK(r_text.find("\nR_MODEL_SHELL_SCALED= 0.1734\n") != std::string::npos); + CHECK(r_text.find("\nR_MODEL_REFLECTIONS= 17610\n") != std::string::npos); + CHECK(r_text.find("\nMODEL_RADIAL_MISFIT= 0.312\n") != std::string::npos); + CHECK(r_text.find("the number to compare between runs") != std::string::npos); + // The shell count is a working detail of the rescale, not part of the reading. + CHECK(r_text.find("MODEL_RADIAL_SHELLS=") == std::string::npos); + { + RunProvenance dev; + dev.developer = true; + CHECK(RenderResultReport("p", "in.h5", x, result, dev) + .find("\nMODEL_RADIAL_SHELLS= 3\n") != std::string::npos); + } + // Nothing significant anywhere: the key still has to be written, saying so. ModelValidationResult no_signal = with_cc; for (auto &sh : no_signal.cc_model_shells) diff --git a/tools/battery/README.md b/tools/battery/README.md index b4d932462..a9b8eeffc 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -57,6 +57,20 @@ arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as XDS's would be with half of its observations. +**R_free does not compare between runs; `rmodel_shell_scaled` does.** rugnux scales the model to +the data with an overall factor and a symmetry-constrained anisotropic B, which can only bend one +way with resolution - deliberately, so that the maps of a batch stay comparable. Whatever the +amplitudes' own radial profile does that this shape cannot follow is reported as R. So two runs +whose merged amplitudes have different radial profiles have R_free values that differ for a reason +that has nothing to do with the data fitting the model: measured on one such pair, R_free moved a +median +0.042 over eleven sets where `phenix.model_vs_data`, which scales per shell, moved +0.002. +`rmodel_shell_scaled` is the same R with one free scale per resolution shell, over every reflection +rather than the free 5% (nothing is refined here, so the free set buys no cross-validation and only +costs precision): on that same pair its median was +0.001, and it still flagged the one set that +was a real regression. `radial_misfit` is the size of that per-shell rescale, the RMS of its +logarithm; when it moves, R_free beside it cannot be read. All three are reported for a human, +never scored. + **Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's `CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells' @@ -292,6 +306,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. | `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref | | `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none | | `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited | +| `rmodel`, `rmodel_shell_scaled`, `radial_misfit` | open arm with a model: R_MODEL, R_MODEL_SHELL_SCALED and MODEL_RADIAL_MISFIT - R over all the reflections, with and without one free scale per resolution shell, and how big that rescale was. `rmodel_shell_scaled` is the R that compares between runs (below) | | `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm) | | `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line | diff --git a/tools/battery/report.py b/tools/battery/report.py index aef3ddd89..a15089300 100644 --- a/tools/battery/report.py +++ b/tools/battery/report.py @@ -18,6 +18,8 @@ NOISE = { "completeness": ("abs", 1.0), # percentage points "cell_dev_pct": ("abs", 0.2), # percentage points "rfree": ("abs", 0.005), + "rmodel_shell_scaled": ("abs", 0.005), + "radial_misfit": ("abs", 0.05), "refres_r_meas": ("rel", 0.05), "refres_isa": ("rel", 0.05), "refres_lowres_r_meas": ("rel", 0.05), @@ -188,6 +190,11 @@ def changes(ra, rb): for name in ("d_min", "isa", "r_meas", "cc_half", "completeness", "cell_dev_pct", "rfree"): if beyond_noise(name, ra.get(name), rb.get(name)): out.append(name) + # The R that is comparable between runs, and the radial profile that decides whether `rfree` + # beside it can be read at all (both only where both runs carry them). + for name in ("rmodel_shell_scaled", "radial_misfit"): + if ra.get(name) is not None and rb.get(name) is not None and beyond_noise(name, ra[name], rb[name]): + out.append(name) # the like-for-like table only where both runs have one (a pre-schema-3 run has none) for name in ("refres_r_meas", "refres_isa", "refres_lowres_r_meas"): if ra.get(name) is not None and rb.get(name) is not None and beyond_noise(name, ra[name], rb[name]): @@ -215,7 +222,8 @@ def arrow(x, y, fmt=lambda v: str(v)): def compare_table(doc, rows, only_changed=True): head = ["set", "arm", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2", - "cell dev %", "R_free", "ref-range R_meas", "ref-range low-res R_meas", "time s", "beyond noise"] + "cell dev %", "R_free", "R_model (shell-scaled)", "radial misfit", + "ref-range R_meas", "ref-range low-res R_meas", "time s", "beyond noise"] out = [] for (arm, s), ra, rb, ch in rows: if only_changed and not ch: @@ -229,6 +237,8 @@ def compare_table(doc, rows, only_changed=True): arrow(ra.get("cc_half"), rb.get("cc_half"), lambda v: f"{v:.3f}"), arrow(ra.get("cell_dev_pct"), rb.get("cell_dev_pct"), lambda v: f"{v:.2f}"), arrow(ra.get("rfree"), rb.get("rfree"), lambda v: f"{v:.3f}"), + arrow(ra.get("rmodel_shell_scaled"), rb.get("rmodel_shell_scaled"), lambda v: f"{v:.3f}"), + arrow(ra.get("radial_misfit"), rb.get("radial_misfit"), lambda v: f"{v:.2f}"), arrow(ra.get("refres_r_meas"), rb.get("refres_r_meas"), lambda v: f"{100 * v:.1f}%"), arrow(ra.get("refres_lowres_r_meas"), rb.get("refres_lowres_r_meas"), lambda v: f"{100 * v:.1f}%"), @@ -342,6 +352,8 @@ def build(run_dir, baseline=None, allow_incomplete=False): ("lowres_r_meas_ratio", "ref-range low-res R_meas / XDS", "{:.3f}"), ("cc_half_noise_ratio", "ref-range CC1/2 noise / XDS", "{:.2f}"), ("rfree", "R_free (placement)", "{:.3f}"), + ("rmodel_shell_scaled", "R_model, shell-scaled", "{:.3f}"), + ("radial_misfit", "radial misfit", "{:.2f}"), ("rfree_ratio", "R_free ratio", "{:.3f}"), ("wall_s", "time s", "{:.0f}")): q = quartiles(r.get(name) for r in rs) if q: @@ -456,7 +468,18 @@ def build(run_dir, baseline=None, allow_incomplete=False): doc.h(3, "Per set") doc.p("Space group: on the open arm the data's own determination. Where --model put the model's " "enantiomorph on the label, the label is in the note and the group scored is the search's.") - model = [("R_free", "rfree"), ("R_work", "rwork"), ("CC model", "cc_model"), ("dep R_free", "rfree_deposited"), + if any(r.get("rmodel_shell_scaled") for r in res): + doc.p("R_free and R_work describe one run against its model and do not carry across runs: the " + "model is scaled to the data by an overall factor and an anisotropic B, so whatever the " + "amplitudes' radial profile does that this shape cannot follow is reported as R, and a " + "change in the profile alone moves R_free further than a real change in the data does. " + "R_model (shell-scaled) is the same R with one free scale per resolution shell removing " + "exactly that, over every reflection rather than the free 5%, and it is the column to " + "read in a delta table. Radial misfit is how big that per-shell rescale had to be (the " + "RMS of its logarithm): when it moves between two runs, R_free between them means " + "little. Both are reported for a human to judge; nothing is scored on them.") + model = [("R_free", "rfree"), ("R_work", "rwork"), ("R_model (shell-scaled)", "rmodel_shell_scaled"), + ("radial misfit", "radial_misfit"), ("CC model", "cc_model"), ("dep R_free", "rfree_deposited"), ("R_free ratio", "rfree_ratio")] if any(r.get("rfree") for r in res) else [] model += [("REFMAC R_free", "refmac_rfree_depflags"), ("REFMAC dep data", "refmac_rfree_depdata"), ("REFMAC ratio", "refmac_rfree_ratio")] if any(r.get("refmac_status") for r in res) else [] diff --git a/tools/battery/score.py b/tools/battery/score.py index 8742d1bab..c44b3bb40 100644 --- a/tools/battery/score.py +++ b/tools/battery/score.py @@ -197,6 +197,13 @@ def judge(entry, rep, run_note): # on rugnux's own free set, so a trend number, not the depositor's R-free "rfree": fnum(rep.get("R_FREE")), "rwork": fnum(rep.get("R_WORK")), "model_fit": rep.get("MODEL_FIT"), "cc_model": fnum(rep.get("CC_MODEL_OVERALL")), + # R over ALL the reflections, with and without one free scale per resolution shell, and the + # size of that rescale. rfree above describes one run and is not comparable with another's: + # the model's scale can only bend as k*exp(-B s^2), so a change in the amplitudes' radial + # profile alone moves it. rmodel_shell_scaled has that taken out and IS comparable; it is + # the R to read in a delta table. radial_misfit says how much there was to take out. + "rmodel": fnum(rep.get("R_MODEL")), "rmodel_shell_scaled": fnum(rep.get("R_MODEL_SHELL_SCALED")), + "radial_misfit": fnum(rep.get("MODEL_RADIAL_MISFIT")), "sg_label": None, "accepted_alt": None, "accepted_alt_why": None, } # the same merge binned over the reference's range (--report-resolution, XDS arms)