rugnux: an R that compares between runs, beside the one that makes the maps
R_WORK and R_FREE score the data against a model scaled by k_overall, a symmetry-constrained anisotropic B and a flat bulk solvent. That scale is deliberate - a free per-shell rescale would reshape each map's radial amplitude profile and a batch of maps would no longer be comparable - but it can only bend one way with resolution, so whatever a dataset's own radial profile does that k*exp(-B s^2) cannot follow is reported as R. Two reductions of one crystal whose amplitudes differ radially therefore differ in R_FREE for a reason that has nothing to do with either fitting the model: measured on one such pair of runs, R_FREE moved a median +0.042 over eleven sets where phenix.model_vs_data, which scales per shell, moved +0.002. Report a second reading of the same fit: R_MODEL_SHELL_SCALED, one free scale per resolution shell of the merge table, with R_MODEL as the same sum without it and MODEL_RADIAL_MISFIT (RMS of ln k_shell) as the size of the rescale, so the reason a comparison is or is not readable is on the page. Over all the reflections, not the free 5%: nothing here is refined, so work and free estimate the same quantity and the split only costs precision. Nothing is written from it. On two sets, every output file - both maps, the anomalous map, the map coefficients, the MTZ, the HKL and the CIF - is byte for byte what the same commit produced without this change, and R_WORK and R_FREE are unchanged. On the pair above, the new R moves a median +0.0005 and holds every set inside 0.0065 except the one that is a real regression, which it puts first at +0.0174 where the old metric ranked it seventh. The battery reports it beside R_free, and scores neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
This commit is contained in:
Claude Opus 5
@@ -126,6 +126,8 @@ $
|
||||
|
||||
This is the standard, few-parameter scaling model used by refinement programs. No free-form per-resolution-shell rescale is applied: such a rescale is dataset-specific and reshapes each map's radial amplitude profile differently, which would make maps from a multi-dataset campaign no longer directly comparable.
|
||||
|
||||
That decision has a cost, and it is paid by the R-factors rather than by the maps. $k\exp(-\mathbf{s}^\mathsf{T} B \mathbf{s})$ can only bend one way with resolution, so whatever a dataset's radial amplitude profile does that this shape cannot follow is reported as R — which makes R a reading of *this* dataset and not a quantity comparable with another reduction of the same crystal. Both uses are wanted, so both are served, separately: the maps and `R_WORK` / `R_FREE` keep the scale above, and a second reading, `R_MODEL_SHELL_SCALED`, applies one free scale per resolution shell to the *same* fit and is reported beside them, with `MODEL_RADIAL_MISFIT` saying how much that rescale had to do. Nothing is written from the second reading; it never reaches a map, a map coefficient or a decision (§14.3, and RUGNUX_REPORT).
|
||||
|
||||
The fit sees the **working reflections only**; the parameters it returns are then applied to every reflection, free ones included, so that R-free (§14.3) is computed against them. The parameters are few, but they are fitted by minimising the very sum R is made of, and a free reflection that helped choose them is no longer held out.
|
||||
|
||||
### 14.3 R-work and R-free
|
||||
|
||||
@@ -143,7 +143,9 @@ cell or no usable space group, does not fail the run: it is logged, and the resu
|
||||
maps cannot be mistaken for one that was never given `--model`.
|
||||
|
||||
Either way the results report carries a **`5. MODEL VALIDATION`** section: `R_WORK=` / `R_FREE=`
|
||||
with their reflection counts, the bulk-solvent and overall scale parameters, the mean 2mFo−DFc density
|
||||
with their reflection counts, `R_MODEL_SHELL_SCALED=` and `MODEL_RADIAL_MISFIT=` beside them (the R
|
||||
that compares between two runs, and why the other two do not — see
|
||||
[RUGNUX_REPORT](RUGNUX_REPORT.md)), the bulk-solvent and overall scale parameters, the mean 2mFo−DFc density
|
||||
at the atom centres, the reindexing operators the written reflections were brought into the model's
|
||||
frame with, and `MAPS_PREFIX=`; or `MODEL_VALIDATION= NOT_PERFORMED` with
|
||||
`MODEL_VALIDATION_REASON=` when the model could not be used. A run given no `--model` has no such
|
||||
|
||||
@@ -551,6 +551,38 @@ The null's own numbers — `MODEL_FIT_STATISTIC`, `MODEL_FIT_VALUE`, the `MODEL_
|
||||
the `MODEL_INDEXING_MARGIN*` keys — are written with `--developer`; the default report carries the
|
||||
verdict (`MODEL_FIT`, `MODEL_FIT_SIGMA`, `MODEL_DECISIONS_TAKEN` and the two decisions).
|
||||
|
||||
### Comparing two runs: `R_MODEL_SHELL_SCALED`
|
||||
|
||||
`R_WORK` and `R_FREE` describe *this* dataset against *this* model, and that is all they describe.
|
||||
They are **not comparable with another run's**. The model is scaled to the data by an overall factor
|
||||
and a symmetry-constrained anisotropic *B* — a shape that can only bend one way with resolution, kept
|
||||
that way on purpose so that a batch of maps stays on one scale (see
|
||||
[CPU_DATA_ANALYSIS_DECISIONS](CPU_DATA_ANALYSIS_DECISIONS.md)). Whatever the amplitudes' own radial
|
||||
profile does that this shape cannot follow is then reported as R. Two reductions of one crystal whose
|
||||
merged amplitudes have different radial profiles therefore differ in `R_FREE` for a reason that has
|
||||
nothing to do with either fitting the model better, and by more than a real change in the data moves it.
|
||||
|
||||
`R_MODEL_SHELL_SCALED` is the same sum with that taken out: one free scale per resolution shell — the
|
||||
merge table's own shells — fitted on the shell's reflections, so only the agreement *inside* each
|
||||
shell is left. `R_MODEL` is the same sum without the per-shell scale, so the gap between the two is
|
||||
what the radial profile cost, and `MODEL_RADIAL_MISFIT` is the size of the rescale that closed it
|
||||
(the RMS of ln *k*<sub>shell</sub> about its mean). When the misfit moves between two runs, `R_FREE`
|
||||
between those two runs cannot be read; `R_MODEL_SHELL_SCALED` can.
|
||||
|
||||
Both R values are over **all** the reflections, not the free 5%. Nothing here is refined — the
|
||||
coordinates, the B-factors and the occupancies are the model's own, and only the scale and the
|
||||
placement are fitted — so work and free estimate the same quantity (they differ by a median 0.001
|
||||
over our corpus) and the split buys no cross-validation while costing a factor of √20 in precision.
|
||||
|
||||
None of this touches the maps: the map coefficients, the `.ccp4` files and `R_WORK` / `R_FREE` are
|
||||
computed from the scale described above and are unchanged by these keys.
|
||||
|
||||
| key | meaning |
|
||||
|---|---|
|
||||
| `R_MODEL`, `R_MODEL_REFLECTIONS` | R over every reflection, on the scale the maps use |
|
||||
| `R_MODEL_SHELL_SCALED` | The same, with one free scale per resolution shell — **the number to compare between runs** |
|
||||
| `MODEL_RADIAL_MISFIT` | How large that per-shell rescale had to be: RMS of its logarithm. Near zero means the maps' own scale already described the radial profile, and then the two R values agree |
|
||||
|
||||
### CC(model, data)
|
||||
|
||||
Beside the R-factors, section 5 carries the **correlation of the merged intensities with the placed,
|
||||
|
||||
@@ -916,6 +916,72 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
result.cc_model_overall, result.cc_model_n,
|
||||
result.cc_model_shells.back().d_min, result.cc_model_shells.back().cc,
|
||||
result.cc_model_shells.back().n, result.cc_model_shells.back().sigma);
|
||||
|
||||
// --- the same R with the radial profile taken out, and how much of it there was ---
|
||||
// R above is what the maps are made of and is read WITHIN a run; this is the one to read
|
||||
// BETWEEN two. The scale the maps use can only bend as k*exp(-B s^2) (see the note in
|
||||
// fit_model, and the reason it stays that way), so where two reductions of one crystal differ
|
||||
// in the radial profile of their amplitudes, the part of the difference that shape cannot
|
||||
// follow lands in R - measured on this corpus, enough to move R by more than a real change in
|
||||
// the data does. Giving the scale one free factor per shell removes the radial profile and
|
||||
// only the radial profile: what is left is the agreement inside each shell, which is the
|
||||
// question "do these data fit this model better or worse than those did".
|
||||
//
|
||||
// On ALL the reflections, not the free set. Nothing here is refined - the coordinates, the B
|
||||
// factors and the occupancies are the model's own - so work and free estimate the same
|
||||
// quantity (measured: the two differ by a median 0.001 over the corpus), and the 5% split
|
||||
// therefore buys no cross-validation while costing a factor of sqrt(20) in precision. Read on
|
||||
// all of them, the number is worth about +-0.002 per set instead of +-0.010, which is the
|
||||
// difference between seeing a real change and seeing the sampling of a free set.
|
||||
//
|
||||
// Nothing is written from any of this. fmodel, the maps, the MTZ and the R-factors above are
|
||||
// untouched; it is a second reading of the fit that has already happened.
|
||||
double num = 0, den = 0, plain_num = 0, plain_den = 0;
|
||||
int n_scored = 0;
|
||||
std::vector<double> sum_oc(report_shell_d_min.size(), 0), sum_cc(report_shell_d_min.size(), 0);
|
||||
for (const auto &hv : fmodel.v) {
|
||||
const auto it = obs_by_hkl.find(hkl_key(hv.hkl));
|
||||
if (it == obs_by_hkl.end()) continue;
|
||||
const size_t bin = shell_of(it->second.d);
|
||||
if (bin >= sum_oc.size()) continue;
|
||||
sum_oc[bin] += it->second.F * std::abs(hv.value);
|
||||
sum_cc[bin] += static_cast<double>(std::abs(hv.value)) * std::abs(hv.value);
|
||||
}
|
||||
// The shell's scale by least squares on its own reflections. One parameter per shell against
|
||||
// the thousands of reflections in it, so scoring the same reflections it was fitted on lowers
|
||||
// R by far less than the difference it is there to expose.
|
||||
std::vector<double> ln_k;
|
||||
ln_k.reserve(report_shell_d_min.size());
|
||||
for (size_t i = 0; i < sum_oc.size(); i++)
|
||||
if (sum_cc[i] > 0 && sum_oc[i] > 0)
|
||||
ln_k.push_back(std::log(sum_oc[i] / sum_cc[i]));
|
||||
for (const auto &hv : fmodel.v) {
|
||||
const auto it = obs_by_hkl.find(hkl_key(hv.hkl));
|
||||
if (it == obs_by_hkl.end()) continue;
|
||||
const size_t bin = shell_of(it->second.d);
|
||||
if (bin >= sum_oc.size()) continue;
|
||||
const double Fo = it->second.F, Fc = std::abs(hv.value);
|
||||
const double k = (sum_cc[bin] > 0 && sum_oc[bin] > 0) ? sum_oc[bin] / sum_cc[bin] : 1.0;
|
||||
num += std::fabs(Fo - k * Fc);
|
||||
den += Fo;
|
||||
plain_num += std::fabs(Fo - Fc);
|
||||
plain_den += Fo;
|
||||
++n_scored;
|
||||
}
|
||||
if (den > 0 && ln_k.size() >= 2) {
|
||||
result.r_model = plain_num / plain_den;
|
||||
result.r_model_shell_scaled = num / den;
|
||||
result.n_model = n_scored;
|
||||
result.shell_scale_bins = static_cast<int>(ln_k.size());
|
||||
const double mean_ln_k = std::accumulate(ln_k.begin(), ln_k.end(), 0.0) / ln_k.size();
|
||||
double var = 0;
|
||||
for (double v : ln_k) var += (v - mean_ln_k) * (v - mean_ln_k);
|
||||
result.radial_misfit = std::sqrt(var / ln_k.size());
|
||||
logger.Info("Model validation: R(all reflections) {:.4f}, with a per-shell scale {:.4f} "
|
||||
"over {} shells; radial misfit {:.3f}",
|
||||
result.r_model, result.r_model_shell_scaled, result.shell_scale_bins,
|
||||
result.radial_misfit);
|
||||
}
|
||||
}
|
||||
|
||||
// --- sigma_A weighting: the maps are 2mFo-DFc and mFo-DFc, not 2Fo-Fc and Fo-Fc ---
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <cmath>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
@@ -31,6 +32,28 @@ struct ModelValidationResult {
|
||||
double k_overall = 0.0;
|
||||
int n_work = 0;
|
||||
int n_free = 0;
|
||||
|
||||
// R over ALL the reflections - r_model as the maps are scaled, r_model_shell_scaled with one free
|
||||
// scale per resolution shell instead - and how far those per-shell scales had to depart from a
|
||||
// flat one. The pair exists to be read BETWEEN runs, where r_work / r_free cannot be.
|
||||
//
|
||||
// The scale the maps carry can only bend as k*exp(-B s^2), deliberately (see ModelValidation.cpp),
|
||||
// so where two reductions of one crystal differ in the radial profile of their amplitudes, the
|
||||
// part of the difference that shape cannot follow is reported as R - by more, measured, than a
|
||||
// real change in the data moves it. A per-shell scale removes the radial profile and only the
|
||||
// radial profile, leaving the agreement inside each shell.
|
||||
//
|
||||
// All the reflections and not the free set, because nothing here is refined: the coordinates, the
|
||||
// B factors and the occupancies are the model's own, so work and free estimate the same quantity
|
||||
// and the 5% split buys no cross-validation while costing a factor of sqrt(20) in precision.
|
||||
double r_model = NAN;
|
||||
double r_model_shell_scaled = NAN;
|
||||
int n_model = 0;
|
||||
// The RMS of ln(k_shell) about its own mean. Near zero says k*exp(-B s^2) already described this
|
||||
// dataset's radial profile, and then the two R-factors agree; a large value says it did not, and
|
||||
// is worth seeing on its own - it is the gap between the two R-factors, named.
|
||||
double radial_misfit = NAN;
|
||||
int shell_scale_bins = 0;
|
||||
double mean_atom_density_sigma = 0.0; // mean 2mFo-DFc value at atom centres, in sigma
|
||||
std::string maps_prefix; // where the .ccp4 / _maps.mtz were written
|
||||
|
||||
|
||||
@@ -34,6 +34,9 @@ namespace {
|
||||
// whether choosing the partner reflections differently could repair it.
|
||||
// 13: CC_MODEL_* in section 5 - the correlation of the merged intensities with the placed model,
|
||||
// by resolution shell, which only a run given a model can report.
|
||||
// 14: R_MODEL / R_MODEL_SHELL_SCALED / MODEL_RADIAL_MISFIT in section 5 - R over all the
|
||||
// reflections, with and without one free scale per shell, and the size of that rescale.
|
||||
// R_WORK and R_FREE are unchanged and still describe the model the maps were made from.
|
||||
// 8: every line that is not `KEY= value` and not blank starts with `#`, warnings included
|
||||
// (`# WARNING:`); the header gains RUGNUX_DOWNLOAD and BUILD_CXX_FLAGS, and RUGNUX_GIT is
|
||||
// the full build-time hash.
|
||||
@@ -1631,6 +1634,15 @@ ReportDocument BuildReportDocument(const std::string &output_prefix,
|
||||
Add(s, KeyReal("R_FREE", mv.r_free, "{:.4f}"));
|
||||
Add(s, KeyInt("R_WORK_REFLECTIONS", mv.n_work));
|
||||
Add(s, KeyInt("R_FREE_REFLECTIONS", mv.n_free));
|
||||
// The pair to read BETWEEN two runs, where R_WORK / R_FREE cannot be - see the prose at
|
||||
// the end of this section and the note in ModelValidation.h.
|
||||
if (std::isfinite(mv.r_model_shell_scaled)) {
|
||||
Add(s, KeyReal("R_MODEL", mv.r_model, "{:.4f}"));
|
||||
Add(s, KeyReal("R_MODEL_SHELL_SCALED", mv.r_model_shell_scaled, "{:.4f}"));
|
||||
Add(s, KeyInt("R_MODEL_REFLECTIONS", mv.n_model));
|
||||
Add(s, KeyReal("MODEL_RADIAL_MISFIT", mv.radial_misfit, "{:.3f}"));
|
||||
Add(s, KeyInt("MODEL_RADIAL_SHELLS", mv.shell_scale_bins, true));
|
||||
}
|
||||
Add(s, KeyReal("BULK_SOLVENT_K_SOL", mv.k_sol, "{:.3f}", true));
|
||||
Add(s, KeyReal("BULK_SOLVENT_B_SOL", mv.b_sol, "{:.1f}", true));
|
||||
Add(s, KeyReal("SCALE_OVERALL", mv.k_overall, "{:.4f}", true));
|
||||
@@ -1746,6 +1758,26 @@ ReportDocument BuildReportDocument(const std::string &output_prefix,
|
||||
" structure's and are a data-quality reading, not a refinement result. The scale and the\n"
|
||||
" placement are both fitted on the working reflections alone, so the free set is held out\n"
|
||||
" of the fit and not only out of the sum."));
|
||||
if (std::isfinite(mv.r_model_shell_scaled))
|
||||
Add(s, Prose(fmt::format(
|
||||
" R_WORK and R_FREE above describe THIS dataset against this model, and that is what they\n"
|
||||
" are for. They are not comparable with another dataset's, or with the same dataset\n"
|
||||
" reduced differently: the model is scaled to the data by an overall factor and an\n"
|
||||
" anisotropic B, a shape that can only bend one way with resolution, and whatever the\n"
|
||||
" amplitudes' own radial profile does that this shape cannot follow is reported as R.\n"
|
||||
" Two reductions with different radial profiles therefore differ in R for a reason that\n"
|
||||
" has nothing to do with how well either fits the model.\n"
|
||||
" R_MODEL_SHELL_SCALED ({:.4f}) is the same R with that removed - one free scale per\n"
|
||||
" resolution shell of the table above, so only the agreement INSIDE each shell is left -\n"
|
||||
" and it is the number to compare between runs. R_MODEL ({:.4f}) is the same sum without\n"
|
||||
" the per-shell scale, so the gap between the two is what the radial profile cost.\n"
|
||||
" Both are over every reflection, not the free set: nothing here is refined, so work and\n"
|
||||
" free measure the same thing and using all of them is simply more precise.\n"
|
||||
" MODEL_RADIAL_MISFIT ({:.3f}) is the size of that per-shell rescale, as the RMS of its\n"
|
||||
" logarithm over {} shells: near zero means the scale already described the profile and\n"
|
||||
" the two R values agree. Nothing is written from any of this - the maps, the map\n"
|
||||
" coefficients and the R-factors above are untouched by it.",
|
||||
mv.r_model_shell_scaled, mv.r_model, mv.radial_misfit, mv.shell_scale_bins)));
|
||||
if (mv.fit_tested)
|
||||
Add(s, Prose(fmt::format(
|
||||
" MODEL_FIT says whether these data accept the model: the same model was refitted from\n"
|
||||
|
||||
@@ -421,8 +421,11 @@ TEST_CASE("ResultReport_ModelValidationSection", "[Diagnostics]") {
|
||||
CHECK(dev_text.find("\nMODEL_FIT_NULL_REPLICATES= 5\n") != std::string::npos);
|
||||
}
|
||||
CHECK(text.find("\nMODEL_FIT_SIGMA= +4.12\n") != std::string::npos);
|
||||
// A run that measured no CC(model, data) writes no key for it.
|
||||
// A run that measured no CC(model, data) writes no key for it, and the shell-scaled R shares
|
||||
// its shells, so it is absent too.
|
||||
CHECK(text.find("CC_MODEL_OVERALL=") == std::string::npos);
|
||||
CHECK(text.find("R_MODEL_SHELL_SCALED=") == std::string::npos);
|
||||
CHECK(text.find("MODEL_RADIAL_MISFIT=") == std::string::npos);
|
||||
CHECK(text.find("CC_MODEL_CONFIRMED_TO_D_MIN=") == std::string::npos);
|
||||
CHECK(text.find("\nMODEL_DECISIONS_TAKEN= ENANTIOMORPH\n") != std::string::npos);
|
||||
// The hand is ASSUMED from the model, never determined: merged intensities cannot see it.
|
||||
@@ -473,6 +476,32 @@ TEST_CASE("ResultReport_ModelValidationSection", "[Diagnostics]") {
|
||||
CHECK(cc_text.find(" 1.80 0.0412 2110 +1.9") != std::string::npos);
|
||||
CHECK(cc_text.find("The table reads in ONE direction only") != std::string::npos);
|
||||
|
||||
// The comparable R rides on the same shells. R_WORK and R_FREE are what they always were - the
|
||||
// maps' own scale - and the new keys sit beside them rather than in place of them.
|
||||
ModelValidationResult with_r = with_cc;
|
||||
with_r.r_model = 0.2011;
|
||||
with_r.r_model_shell_scaled = 0.1734;
|
||||
with_r.n_model = 17610;
|
||||
with_r.radial_misfit = 0.312;
|
||||
with_r.shell_scale_bins = 3;
|
||||
result.model_validation = with_r;
|
||||
const auto r_text = RenderResultReport("p", "in.h5", x, result);
|
||||
CHECK(r_text.find("\nR_WORK= 0.1823\n") != std::string::npos);
|
||||
CHECK(r_text.find("\nR_FREE= 0.2145\n") != std::string::npos);
|
||||
CHECK(r_text.find("\nR_MODEL= 0.2011\n") != std::string::npos);
|
||||
CHECK(r_text.find("\nR_MODEL_SHELL_SCALED= 0.1734\n") != std::string::npos);
|
||||
CHECK(r_text.find("\nR_MODEL_REFLECTIONS= 17610\n") != std::string::npos);
|
||||
CHECK(r_text.find("\nMODEL_RADIAL_MISFIT= 0.312\n") != std::string::npos);
|
||||
CHECK(r_text.find("the number to compare between runs") != std::string::npos);
|
||||
// The shell count is a working detail of the rescale, not part of the reading.
|
||||
CHECK(r_text.find("MODEL_RADIAL_SHELLS=") == std::string::npos);
|
||||
{
|
||||
RunProvenance dev;
|
||||
dev.developer = true;
|
||||
CHECK(RenderResultReport("p", "in.h5", x, result, dev)
|
||||
.find("\nMODEL_RADIAL_SHELLS= 3\n") != std::string::npos);
|
||||
}
|
||||
|
||||
// Nothing significant anywhere: the key still has to be written, saying so.
|
||||
ModelValidationResult no_signal = with_cc;
|
||||
for (auto &sh : no_signal.cc_model_shells)
|
||||
|
||||
@@ -57,6 +57,20 @@ arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1)
|
||||
range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as
|
||||
XDS's would be with half of its observations.
|
||||
|
||||
**R_free does not compare between runs; `rmodel_shell_scaled` does.** rugnux scales the model to
|
||||
the data with an overall factor and a symmetry-constrained anisotropic B, which can only bend one
|
||||
way with resolution - deliberately, so that the maps of a batch stay comparable. Whatever the
|
||||
amplitudes' own radial profile does that this shape cannot follow is reported as R. So two runs
|
||||
whose merged amplitudes have different radial profiles have R_free values that differ for a reason
|
||||
that has nothing to do with the data fitting the model: measured on one such pair, R_free moved a
|
||||
median +0.042 over eleven sets where `phenix.model_vs_data`, which scales per shell, moved +0.002.
|
||||
`rmodel_shell_scaled` is the same R with one free scale per resolution shell, over every reflection
|
||||
rather than the free 5% (nothing is refined here, so the free set buys no cross-validation and only
|
||||
costs precision): on that same pair its median was +0.001, and it still flagged the one set that
|
||||
was a real regression. `radial_misfit` is the size of that per-shell rescale, the RMS of its
|
||||
logarithm; when it moves, R_free beside it cannot be read. All three are reported for a human,
|
||||
never scored.
|
||||
|
||||
**Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own
|
||||
table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's
|
||||
`CORRECT.LP` (`lowres_r_meas_ref`, stored in the manifest as `r_meas_low`), with the shells'
|
||||
@@ -292,6 +306,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way.
|
||||
| `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref |
|
||||
| `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none |
|
||||
| `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited |
|
||||
| `rmodel`, `rmodel_shell_scaled`, `radial_misfit` | open arm with a model: R_MODEL, R_MODEL_SHELL_SCALED and MODEL_RADIAL_MISFIT - R over all the reflections, with and without one free scale per resolution shell, and how big that rescale was. `rmodel_shell_scaled` is the R that compares between runs (below) |
|
||||
| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm) |
|
||||
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |
|
||||
|
||||
|
||||
+25
-2
@@ -18,6 +18,8 @@ NOISE = {
|
||||
"completeness": ("abs", 1.0), # percentage points
|
||||
"cell_dev_pct": ("abs", 0.2), # percentage points
|
||||
"rfree": ("abs", 0.005),
|
||||
"rmodel_shell_scaled": ("abs", 0.005),
|
||||
"radial_misfit": ("abs", 0.05),
|
||||
"refres_r_meas": ("rel", 0.05),
|
||||
"refres_isa": ("rel", 0.05),
|
||||
"refres_lowres_r_meas": ("rel", 0.05),
|
||||
@@ -188,6 +190,11 @@ def changes(ra, rb):
|
||||
for name in ("d_min", "isa", "r_meas", "cc_half", "completeness", "cell_dev_pct", "rfree"):
|
||||
if beyond_noise(name, ra.get(name), rb.get(name)):
|
||||
out.append(name)
|
||||
# The R that is comparable between runs, and the radial profile that decides whether `rfree`
|
||||
# beside it can be read at all (both only where both runs carry them).
|
||||
for name in ("rmodel_shell_scaled", "radial_misfit"):
|
||||
if ra.get(name) is not None and rb.get(name) is not None and beyond_noise(name, ra[name], rb[name]):
|
||||
out.append(name)
|
||||
# the like-for-like table only where both runs have one (a pre-schema-3 run has none)
|
||||
for name in ("refres_r_meas", "refres_isa", "refres_lowres_r_meas"):
|
||||
if ra.get(name) is not None and rb.get(name) is not None and beyond_noise(name, ra[name], rb[name]):
|
||||
@@ -215,7 +222,8 @@ def arrow(x, y, fmt=lambda v: str(v)):
|
||||
|
||||
def compare_table(doc, rows, only_changed=True):
|
||||
head = ["set", "arm", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2",
|
||||
"cell dev %", "R_free", "ref-range R_meas", "ref-range low-res R_meas", "time s", "beyond noise"]
|
||||
"cell dev %", "R_free", "R_model (shell-scaled)", "radial misfit",
|
||||
"ref-range R_meas", "ref-range low-res R_meas", "time s", "beyond noise"]
|
||||
out = []
|
||||
for (arm, s), ra, rb, ch in rows:
|
||||
if only_changed and not ch:
|
||||
@@ -229,6 +237,8 @@ def compare_table(doc, rows, only_changed=True):
|
||||
arrow(ra.get("cc_half"), rb.get("cc_half"), lambda v: f"{v:.3f}"),
|
||||
arrow(ra.get("cell_dev_pct"), rb.get("cell_dev_pct"), lambda v: f"{v:.2f}"),
|
||||
arrow(ra.get("rfree"), rb.get("rfree"), lambda v: f"{v:.3f}"),
|
||||
arrow(ra.get("rmodel_shell_scaled"), rb.get("rmodel_shell_scaled"), lambda v: f"{v:.3f}"),
|
||||
arrow(ra.get("radial_misfit"), rb.get("radial_misfit"), lambda v: f"{v:.2f}"),
|
||||
arrow(ra.get("refres_r_meas"), rb.get("refres_r_meas"), lambda v: f"{100 * v:.1f}%"),
|
||||
arrow(ra.get("refres_lowres_r_meas"), rb.get("refres_lowres_r_meas"),
|
||||
lambda v: f"{100 * v:.1f}%"),
|
||||
@@ -342,6 +352,8 @@ def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
("lowres_r_meas_ratio", "ref-range low-res R_meas / XDS", "{:.3f}"),
|
||||
("cc_half_noise_ratio", "ref-range CC1/2 noise / XDS", "{:.2f}"),
|
||||
("rfree", "R_free (placement)", "{:.3f}"),
|
||||
("rmodel_shell_scaled", "R_model, shell-scaled", "{:.3f}"),
|
||||
("radial_misfit", "radial misfit", "{:.2f}"),
|
||||
("rfree_ratio", "R_free ratio", "{:.3f}"), ("wall_s", "time s", "{:.0f}")):
|
||||
q = quartiles(r.get(name) for r in rs)
|
||||
if q:
|
||||
@@ -456,7 +468,18 @@ def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
doc.h(3, "Per set")
|
||||
doc.p("Space group: on the open arm the data's own determination. Where --model put the model's "
|
||||
"enantiomorph on the label, the label is in the note and the group scored is the search's.")
|
||||
model = [("R_free", "rfree"), ("R_work", "rwork"), ("CC model", "cc_model"), ("dep R_free", "rfree_deposited"),
|
||||
if any(r.get("rmodel_shell_scaled") for r in res):
|
||||
doc.p("R_free and R_work describe one run against its model and do not carry across runs: the "
|
||||
"model is scaled to the data by an overall factor and an anisotropic B, so whatever the "
|
||||
"amplitudes' radial profile does that this shape cannot follow is reported as R, and a "
|
||||
"change in the profile alone moves R_free further than a real change in the data does. "
|
||||
"R_model (shell-scaled) is the same R with one free scale per resolution shell removing "
|
||||
"exactly that, over every reflection rather than the free 5%, and it is the column to "
|
||||
"read in a delta table. Radial misfit is how big that per-shell rescale had to be (the "
|
||||
"RMS of its logarithm): when it moves between two runs, R_free between them means "
|
||||
"little. Both are reported for a human to judge; nothing is scored on them.")
|
||||
model = [("R_free", "rfree"), ("R_work", "rwork"), ("R_model (shell-scaled)", "rmodel_shell_scaled"),
|
||||
("radial misfit", "radial_misfit"), ("CC model", "cc_model"), ("dep R_free", "rfree_deposited"),
|
||||
("R_free ratio", "rfree_ratio")] if any(r.get("rfree") for r in res) else []
|
||||
model += [("REFMAC R_free", "refmac_rfree_depflags"), ("REFMAC dep data", "refmac_rfree_depdata"),
|
||||
("REFMAC ratio", "refmac_rfree_ratio")] if any(r.get("refmac_status") for r in res) else []
|
||||
|
||||
@@ -197,6 +197,13 @@ def judge(entry, rep, run_note):
|
||||
# on rugnux's own free set, so a trend number, not the depositor's R-free
|
||||
"rfree": fnum(rep.get("R_FREE")), "rwork": fnum(rep.get("R_WORK")),
|
||||
"model_fit": rep.get("MODEL_FIT"), "cc_model": fnum(rep.get("CC_MODEL_OVERALL")),
|
||||
# R over ALL the reflections, with and without one free scale per resolution shell, and the
|
||||
# size of that rescale. rfree above describes one run and is not comparable with another's:
|
||||
# the model's scale can only bend as k*exp(-B s^2), so a change in the amplitudes' radial
|
||||
# profile alone moves it. rmodel_shell_scaled has that taken out and IS comparable; it is
|
||||
# the R to read in a delta table. radial_misfit says how much there was to take out.
|
||||
"rmodel": fnum(rep.get("R_MODEL")), "rmodel_shell_scaled": fnum(rep.get("R_MODEL_SHELL_SCALED")),
|
||||
"radial_misfit": fnum(rep.get("MODEL_RADIAL_MISFIT")),
|
||||
"sg_label": None, "accepted_alt": None, "accepted_alt_why": None,
|
||||
}
|
||||
# the same merge binned over the reference's range (--report-resolution, XDS arms)
|
||||
|
||||
Reference in New Issue
Block a user