v1.0.0-rc.166 (#76)
Build Packages / Unit tests (push) Successful in 1h22m15s
Build Packages / build:windows:nocuda (push) Successful in 18m0s
Build Packages / build:windows:cuda (push) Successful in 20m30s
Build Packages / build:viewer-tgz:cpu (push) Successful in 10m32s
Build Packages / build:viewer-tgz:cuda (push) Successful in 11m39s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 8m55s
Build Packages / build:rugnux:windows (push) Successful in 11m25s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 20m6s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m27s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 20m19s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 15m34s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 20m25s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m36s
Build Packages / build:rpm (rocky8) (push) Successful in 17m43s
Build Packages / build:rpm (rocky9) (push) Successful in 13m34s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 21m28s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 18m19s
Build Packages / DIALS test (push) Successful in 12m36s
Build Packages / XDS test (durin plugin) (push) Successful in 6m56s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m48s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m7s
Build Packages / Generate python client (push) Successful in 11s
Build Packages / Build documentation (push) Successful in 36s
Build Packages / Create release (push) Skipped
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 5m11s
Build Packages / Unit tests (push) Successful in 1h22m15s
Build Packages / build:windows:nocuda (push) Successful in 18m0s
Build Packages / build:windows:cuda (push) Successful in 20m30s
Build Packages / build:viewer-tgz:cpu (push) Successful in 10m32s
Build Packages / build:viewer-tgz:cuda (push) Successful in 11m39s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 8m55s
Build Packages / build:rugnux:windows (push) Successful in 11m25s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 20m6s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m27s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 20m19s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 15m34s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 20m25s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m36s
Build Packages / build:rpm (rocky8) (push) Successful in 17m43s
Build Packages / build:rpm (rocky9) (push) Successful in 13m34s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 21m28s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 18m19s
Build Packages / DIALS test (push) Successful in 12m36s
Build Packages / XDS test (durin plugin) (push) Successful in 6m56s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m48s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m7s
Build Packages / Generate python client (push) Successful in 11s
Build Packages / Build documentation (push) Successful in 36s
Build Packages / Create release (push) Skipped
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 5m11s
* `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #76.
This commit is contained in:
1 parent
511be0c366
commit
680c36c20d
383 files changed
+20910
-3936
No files matched your search
+505
-104
@@ -7,7 +7,10 @@
|
||||
#include <cmath>
|
||||
#include <complex>
|
||||
#include <array>
|
||||
#include <numeric>
|
||||
#include <random>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
#include <unordered_map>
|
||||
|
||||
@@ -23,7 +26,10 @@
|
||||
|
||||
#include "../common/JFJochMath.h" // PI (M_PI is not standard, and MSVC does not define it)
|
||||
#include "../common/Logger.h"
|
||||
#include "../common/ParallelFor.h" // ParallelFor
|
||||
#include "../image_analysis/scale_merge/ReindexAmbiguity.h" // ReindexReflections
|
||||
#include "RigidBodyRefine.h"
|
||||
#include "SigmaA.h"
|
||||
|
||||
namespace {
|
||||
|
||||
@@ -63,6 +69,75 @@ constexpr double ANOMALOUS_INVERSION_SIGMA = 5.0;
|
||||
// signal is there and what carries it; a full site list is what the map file is for.
|
||||
constexpr size_t MAX_ANOMALOUS_SITES = 10;
|
||||
|
||||
// How many random placements of the same model the real fit is compared against. The verdict is
|
||||
// (real - mean)/sd of this sample, so what matters is not the mean but how well the SPREAD is
|
||||
// pinned: the relative error on an sd from n draws is 1/sqrt(2(n-1)), and a sample that happens to
|
||||
// come out narrow is what turns a model that does not fit into one that appears to. Measured on the
|
||||
// case that sits closest to the gate - a model of an unrelated protein, 1.83 sigma against a
|
||||
// threshold of 3 - the chance of it reading over the gate on a different seed is 31% at n=3, 17% at
|
||||
// n=5, 6% at n=9.
|
||||
//
|
||||
// Nine rather than five because the replicates run concurrently: the null costs the SLOWEST of them
|
||||
// rather than their sum, and the slowest of nine is barely above the slowest of five, so the extra
|
||||
// four are close to free in wall clock (measured 2.5 s against 2.6 s) up to the thread count.
|
||||
constexpr int NULL_REPLICATES = 9;
|
||||
|
||||
// Fixed, so the same data and the same model give the same verdict on every run.
|
||||
constexpr unsigned NULL_SEED = 20260902;
|
||||
|
||||
// How far above its own null a fit has to sit before the model is allowed to decide anything. The cut
|
||||
// is in sigma of that null and not in R: measured, the R a model that explains nothing reaches moves
|
||||
// with the model's atom count and B-factors as much as with the data, so no value of R separates the
|
||||
// two on its own. Measured at nine replicates, the crystal's own model reads +17.7 sigma and an
|
||||
// unrelated protein +1.7, so the cut sits in a gap an order of magnitude wider than the sd it is
|
||||
// measured in.
|
||||
constexpr double MODEL_FIT_SIGMA = 3.0;
|
||||
|
||||
// Mean and sample standard deviation of a small sample.
|
||||
std::pair<double, double> mean_sd(const std::vector<double> &v) {
|
||||
if (v.size() < 2)
|
||||
return {v.empty() ? 0.0 : v.front(), 0.0};
|
||||
const double mean = std::accumulate(v.begin(), v.end(), 0.0) / static_cast<double>(v.size());
|
||||
double s2 = 0;
|
||||
for (double x : v)
|
||||
s2 += (x - mean) * (x - mean);
|
||||
return {mean, std::sqrt(s2 / static_cast<double>(v.size() - 1))};
|
||||
}
|
||||
|
||||
// A rotation drawn uniformly from SO(3), through a uniform random unit quaternion.
|
||||
// Following Shoemake (1992) Graphics Gems III, 124-132
|
||||
gemmi::Mat33 random_rotation(std::mt19937 &rng) {
|
||||
std::uniform_real_distribution<double> u(0.0, 1.0);
|
||||
const double u1 = u(rng), t2 = 2 * PI * u(rng), t3 = 2 * PI * u(rng);
|
||||
const double r1 = std::sqrt(1 - u1), r2 = std::sqrt(u1);
|
||||
const double x = r1 * std::sin(t2), y = r1 * std::cos(t2), z = r2 * std::sin(t3), w = r2 * std::cos(t3);
|
||||
return {1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w),
|
||||
2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w),
|
||||
2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)};
|
||||
}
|
||||
|
||||
// Everything one fit moves: the model itself, and the structure factors that follow it. The real
|
||||
// model has one of these and every null replicate gets a copy of its own, which is what lets the
|
||||
// replicates run at the same time without stepping on each other.
|
||||
struct ModelState {
|
||||
gemmi::Structure st;
|
||||
gemmi::AsuData<std::complex<float>> fcalc, fmask;
|
||||
};
|
||||
|
||||
// Turn the model about its own centroid, so it keeps its place in the cell and loses its orientation.
|
||||
void reorient_about_centroid(gemmi::Model &model, const gemmi::Mat33 &rot) {
|
||||
std::vector<gemmi::Position> pos = ModelPositions(model);
|
||||
if (pos.empty())
|
||||
return;
|
||||
gemmi::Vec3 centre;
|
||||
for (const gemmi::Position &p : pos)
|
||||
centre += p;
|
||||
centre *= 1.0 / static_cast<double>(pos.size());
|
||||
for (gemmi::Position &p : pos)
|
||||
p = gemmi::Position(rot.multiply(gemmi::Vec3(p) - centre) + centre);
|
||||
SetModelPositions(model, pos);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &merged,
|
||||
@@ -70,13 +145,16 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
const std::string &model_path,
|
||||
const std::string &output_prefix,
|
||||
Logger &logger,
|
||||
std::optional<int> data_space_group_number,
|
||||
bool probe_indexing_ambiguity) {
|
||||
const gemmi::SpaceGroup *data_space_group,
|
||||
bool probe_indexing_ambiguity,
|
||||
size_t nthreads,
|
||||
double wavelength_A) {
|
||||
ModelValidationResult result;
|
||||
result.model_path = model_path;
|
||||
|
||||
// --- read the atomic model ---
|
||||
gemmi::Structure st;
|
||||
ModelState mdl;
|
||||
gemmi::Structure &st = mdl.st; // the real model, the one the maps and the report describe
|
||||
try {
|
||||
// Detect, not the default: without it GEMMI picks the format from the extension and only
|
||||
// falls back to the content when it does not recognise one. A model arrives named however
|
||||
@@ -101,8 +179,11 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
result.model_space_group_number = sg->number;
|
||||
|
||||
// If the data was indexed in the enantiomorph of the model's space group (e.g. data P4(1)2(1)2,
|
||||
// model P4(3)2(1)2 - the merged intensities cannot tell them apart), adopt the model's group as
|
||||
// a LABEL and leave the reflections exactly where they are.
|
||||
// model P4(3)2(1)2 - the merged intensities cannot tell them apart), the model's group is a
|
||||
// CANDIDATE for the label the reflections are written under. Only a candidate: this is arithmetic
|
||||
// on two group numbers and says nothing about whether the model belongs to this crystal, and a
|
||||
// model that does not is exactly as capable of rewriting the label as one that does. Whether it is
|
||||
// taken up is settled at the end, on the fit against its own null and on the anomalous map.
|
||||
//
|
||||
// It is tempting to reindex by the change-of-hand operator instead, and that is wrong. The two
|
||||
// groups of an enantiomorphic pair have the same rotation operations - only their translations
|
||||
@@ -116,18 +197,18 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
// disagreement but manufactures agreement. The anomalous difference map below is the only honest
|
||||
// arbiter, and it is used to report the disagreement rather than to bury it.
|
||||
const std::vector<MergedReflection> &obs = merged;
|
||||
if (data_space_group_number && *data_space_group_number != sg->number) {
|
||||
const gemmi::SpaceGroup *dsg = gemmi::find_spacegroup_by_number(*data_space_group_number);
|
||||
if (dsg && dsg->is_enantiomorphic() && sg->is_enantiomorphic()) {
|
||||
gemmi::GroupOps eops = dsg->operations();
|
||||
eops.change_basis_forward(dsg->change_of_hand_op());
|
||||
if (data_space_group && data_space_group->number != sg->number) {
|
||||
if (data_space_group->is_enantiomorphic() && sg->is_enantiomorphic()) {
|
||||
gemmi::GroupOps eops = data_space_group->operations();
|
||||
eops.change_basis_forward(data_space_group->change_of_hand_op());
|
||||
const gemmi::SpaceGroup *enant = gemmi::find_spacegroup_by_ops(eops);
|
||||
if (enant && enant->number == sg->number) {
|
||||
result.adopted_model_enantiomorph = true;
|
||||
result.model_enantiomorph_candidate = true;
|
||||
logger.Info("Model validation: data space group {} is the enantiomorph of the model {}; "
|
||||
"taking the model's group as the label, with no reindexing - the two groups "
|
||||
"index identically, and reindexing would flip the anomalous differences",
|
||||
dsg->short_name(), sg->hm);
|
||||
"the model's group is a candidate for the label, pending the fit - a change "
|
||||
"of label only, since the two groups index identically and reindexing would "
|
||||
"flip the anomalous differences",
|
||||
data_space_group->short_name(), sg->hm);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -160,26 +241,29 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
logger.Info("Model validation: {} atoms, cell a={:.2f} b={:.2f} c={:.2f}, sg {}, to {:.2f} A",
|
||||
gemmi::count_atom_sites(st.models[0]), ucell.a, ucell.b, ucell.c, sg->hm, d_min);
|
||||
|
||||
// --- Fcalc (atomic) via electron density on a grid + FFT ---
|
||||
gemmi::DensityCalculator<Table, float> dc;
|
||||
dc.d_min = d_min;
|
||||
dc.rate = 1.5;
|
||||
dc.set_grid_cell_and_spacegroup(st);
|
||||
dc.set_refmac_compatible_blur(st.models[0]);
|
||||
dc.put_model_density_on_grid(st.models[0]);
|
||||
gemmi::AsuData<std::complex<float>> fcalc =
|
||||
gemmi::transform_map_to_f_phi(dc.grid, true).prepare_asu_data(dc.d_min, dc.blur, false, false, false);
|
||||
// --- Fcalc (atomic) via electron density on a grid + FFT, plus a flat bulk-solvent mask -> Fmask.
|
||||
// A lambda because the rigid-body step below moves the model and then needs both again, and it
|
||||
// takes the state to work on so that a null replicate can run it on its own copy. ---
|
||||
auto compute_model_factors = [&](ModelState &ms) {
|
||||
gemmi::DensityCalculator<Table, float> dc;
|
||||
dc.d_min = d_min;
|
||||
dc.rate = 1.5;
|
||||
dc.set_grid_cell_and_spacegroup(ms.st);
|
||||
dc.set_refmac_compatible_blur(ms.st.models[0]);
|
||||
dc.put_model_density_on_grid(ms.st.models[0]);
|
||||
ms.fcalc = gemmi::transform_map_to_f_phi(dc.grid, true)
|
||||
.prepare_asu_data(dc.d_min, dc.blur, false, false, false);
|
||||
|
||||
// --- flat bulk-solvent mask -> Fmask ---
|
||||
// Refmac radii give a slightly lower R than the Cctbx set on our test cases, at no cost.
|
||||
gemmi::SolventMasker masker(gemmi::AtomicRadiiSet::Refmac);
|
||||
gemmi::Grid<float> mask_grid;
|
||||
mask_grid.unit_cell = dc.grid.unit_cell;
|
||||
mask_grid.spacegroup = dc.grid.spacegroup;
|
||||
mask_grid.set_size_from_spacing(dc.requested_grid_spacing(), gemmi::GridSizeRounding::Up);
|
||||
masker.put_mask_on_grid(mask_grid, st.models[0]);
|
||||
gemmi::AsuData<std::complex<float>> fmask =
|
||||
gemmi::transform_map_to_f_phi(mask_grid, true).prepare_asu_data(dc.d_min, 0);
|
||||
// Refmac radii give a slightly lower R than the Cctbx set on our test cases, at no cost.
|
||||
gemmi::SolventMasker masker(gemmi::AtomicRadiiSet::Refmac);
|
||||
gemmi::Grid<float> mask_grid;
|
||||
mask_grid.unit_cell = dc.grid.unit_cell;
|
||||
mask_grid.spacegroup = dc.grid.spacegroup;
|
||||
mask_grid.set_size_from_spacing(dc.requested_grid_spacing(), gemmi::GridSizeRounding::Up);
|
||||
masker.put_mask_on_grid(mask_grid, ms.st.models[0]);
|
||||
ms.fmask = gemmi::transform_map_to_f_phi(mask_grid, true).prepare_asu_data(dc.d_min, 0);
|
||||
};
|
||||
compute_model_factors(mdl);
|
||||
|
||||
gemmi::GroupOps gops = sg->operations();
|
||||
gemmi::ReciprocalAsu asu(sg);
|
||||
@@ -188,20 +272,21 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
// Factored into a lambda so we can probe indexing (merohedral) ambiguities: run the same scale +
|
||||
// R computation on each reindexing of the observed reflections and keep the lowest-R-free one.
|
||||
struct Fit {
|
||||
gemmi::AsuData<std::complex<float>> fmodel, map2fofc, mapfofc;
|
||||
gemmi::AsuData<std::complex<float>> fmodel;
|
||||
gemmi::AsuData<gemmi::ValueSigma<float>> fobs; // what it was fitted to
|
||||
std::unordered_map<long, std::pair<double, bool>> obs_by_hkl; // hkl -> (Fobs, is_free)
|
||||
double r_work = 1, r_free = 1, k_sol = 0, b_sol = 0, k_overall = 0;
|
||||
int n_w = 0, n_f = 0;
|
||||
};
|
||||
auto fit_model = [&](const std::vector<MergedReflection> &obs_in) -> Fit {
|
||||
auto fit_model = [&](ModelState &ms, const std::vector<MergedReflection> &obs_in) -> Fit {
|
||||
Fit out;
|
||||
out.fmodel = fcalc; // copy the atomic structure factors; scaling mutates them in place
|
||||
out.fmodel = ms.fcalc; // copy the atomic structure factors; scaling mutates them in place
|
||||
|
||||
// --- observed amplitudes into the model ASU, keyed by hkl (also remember free flag) ---
|
||||
// Observed amplitudes are the French-Wilson |F| already computed at the end of the merge
|
||||
// (MergedReflection.F), so the model R-free / maps use exactly the same amplitudes as the
|
||||
// written reflection file.
|
||||
gemmi::AsuData<gemmi::ValueSigma<float>> fobs;
|
||||
gemmi::AsuData<gemmi::ValueSigma<float>> &fobs = out.fobs;
|
||||
fobs.unit_cell_ = ucell;
|
||||
fobs.spacegroup_ = sg;
|
||||
for (const MergedReflection &r : obs_in) {
|
||||
@@ -217,10 +302,10 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
// --- scale Fmodel(+solvent) to Fobs: k_overall, anisotropic B, k_sol, b_sol ---
|
||||
gemmi::Scaling<float> scaling(ucell, sg);
|
||||
scaling.use_solvent = true;
|
||||
scaling.prepare_points(out.fmodel, fobs, &fmask);
|
||||
scaling.prepare_points(out.fmodel, fobs, &ms.fmask);
|
||||
scaling.fit_isotropic_b_approximately();
|
||||
scaling.fit_parameters();
|
||||
scaling.scale_data(out.fmodel, &fmask); // out.fmodel now holds the scaled, solvent-corrected Fmodel
|
||||
scaling.scale_data(out.fmodel, &ms.fmask); // out.fmodel now holds the scaled, solvent-corrected Fmodel
|
||||
out.k_sol = scaling.k_sol;
|
||||
out.b_sol = scaling.b_sol;
|
||||
out.k_overall = scaling.k_overall;
|
||||
@@ -231,28 +316,83 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
// it reshapes each map's radial amplitude profile differently, so a batch of maps would no
|
||||
// longer be directly comparable. For a fragment-screening / PanDDA campaign, comparable maps
|
||||
// across datasets matter more than the last bit of per-dataset R, so it is deliberately omitted.
|
||||
// (The sigma_A weighting further down is a different thing and does not reopen this: it never
|
||||
// rescales Fobs, and it leaves the R-factors below untouched. It does weight the map
|
||||
// coefficients per shell and per dataset - see the note where it is computed.)
|
||||
|
||||
// --- R-work / R-free and map coefficients (2Fo-Fc and Fo-Fc, model phases) ---
|
||||
out.map2fofc.unit_cell_ = ucell; out.map2fofc.spacegroup_ = sg;
|
||||
out.mapfofc.unit_cell_ = ucell; out.mapfofc.spacegroup_ = sg;
|
||||
// --- R-work / R-free ---
|
||||
double num_w = 0, den_w = 0, num_f = 0, den_f = 0;
|
||||
for (const auto &hv : out.fmodel.v) {
|
||||
auto it = out.obs_by_hkl.find(hkl_key(hv.hkl));
|
||||
if (it == out.obs_by_hkl.end()) continue;
|
||||
double Fo = it->second.first;
|
||||
double Fc = std::abs(hv.value);
|
||||
double phi = std::arg(hv.value);
|
||||
if (it->second.second) { num_f += std::fabs(Fo - Fc); den_f += Fo; ++out.n_f; }
|
||||
else { num_w += std::fabs(Fo - Fc); den_w += Fo; ++out.n_w; }
|
||||
std::complex<float> ph = std::polar(1.0f, static_cast<float>(phi));
|
||||
out.map2fofc.v.push_back({hv.hkl, static_cast<float>(2 * Fo - Fc) * ph});
|
||||
out.mapfofc.v.push_back({hv.hkl, static_cast<float>(Fo - Fc) * ph});
|
||||
}
|
||||
out.r_work = den_w > 0 ? num_w / den_w : 1;
|
||||
out.r_free = den_f > 0 ? num_f / den_f : 1;
|
||||
return out;
|
||||
};
|
||||
|
||||
// --- the model placed against one observed set: fit, then a rigid-body placement ---
|
||||
// The real model and every null replicate below go through this same lambda, so the two are
|
||||
// comparable - a null that was not placed would be the null of a weaker procedure than the one it
|
||||
// is there to judge. It leaves the model where it ends up, and the caller puts it back.
|
||||
struct Placement {
|
||||
Fit fit;
|
||||
bool rb_applied = false;
|
||||
double rb_angle_deg = 0, rb_shift_A = 0, r_free_before_rb = 0;
|
||||
};
|
||||
auto place_and_fit = [&](ModelState &ms, const std::vector<MergedReflection> &obs_in) -> Placement {
|
||||
Placement out;
|
||||
out.fit = fit_model(ms, obs_in);
|
||||
|
||||
// --- rigid-body placement of the model in the data cell ---
|
||||
// Re-fractionalizing the model above puts it in the right box but not in the right place: a
|
||||
// non-isomorphous cell squeezes the box without moving the body inside it, and the body's own
|
||||
// position differs from crystal to crystal anyway. Six parameters recover that - a rotation about
|
||||
// the model's centroid and a translation - which is all a fragment-screening model needs, since it
|
||||
// arrives already solved. The step is committed only if the FREE reflections, which the refinement
|
||||
// never saw, agree that it helped; on data the model cannot be placed against, the model stays
|
||||
// exactly where it was read.
|
||||
gemmi::AsuData<gemmi::ValueSigma<float>> fobs_work;
|
||||
fobs_work.unit_cell_ = ucell;
|
||||
fobs_work.spacegroup_ = sg;
|
||||
for (const auto &hv : out.fit.fobs.v) {
|
||||
const auto it = out.fit.obs_by_hkl.find(hkl_key(hv.hkl));
|
||||
if (it != out.fit.obs_by_hkl.end() && !it->second.second)
|
||||
fobs_work.v.push_back(hv);
|
||||
}
|
||||
const std::vector<gemmi::Position> before = ModelPositions(ms.st.models[0]);
|
||||
const RigidBodyRefineResult rb =
|
||||
RefineRigidBody(ms.st.models[0], ucell, *sg, fobs_work, d_min, logger);
|
||||
if (!rb.converged) {
|
||||
// RefineRigidBody leaves the model wherever the solver left it, usable answer or not, so
|
||||
// the restore cannot be conditional on the same flag the re-fit is.
|
||||
SetModelPositions(ms.st.models[0], before);
|
||||
return out;
|
||||
}
|
||||
{
|
||||
compute_model_factors(ms);
|
||||
Fit moved = fit_model(ms, obs_in);
|
||||
const bool commit = moved.r_free < out.fit.r_free;
|
||||
logger.Info("Model validation: rigid body held-out R-free {:.4f} -> {:.4f} => {}",
|
||||
out.fit.r_free, moved.r_free, commit ? "committed" : "rejected, model put back");
|
||||
if (commit) {
|
||||
out.rb_applied = true;
|
||||
out.rb_angle_deg = rb.angle_deg;
|
||||
out.rb_shift_A = rb.shift_A;
|
||||
out.r_free_before_rb = out.fit.r_free;
|
||||
out.fit = std::move(moved);
|
||||
} else {
|
||||
SetModelPositions(ms.st.models[0], before);
|
||||
compute_model_factors(ms);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
// --- indexing (merohedral) ambiguity ---
|
||||
// When a reference MTZ was supplied, the data were already reindexed to agree with the reference
|
||||
// intensities (at the merge stage for rotation data, per image in stills scaling), and that
|
||||
@@ -261,30 +401,204 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
// R-free. A no-op either way for a holohedral crystal (no twin laws). The
|
||||
// enantiomorph/screw ambiguity is never probed by R-free: |Fcalc| is the same for both hands, so
|
||||
// it cannot distinguish them - that is taken from the model hand above.
|
||||
Fit best = fit_model(obs);
|
||||
if (probe_indexing_ambiguity) {
|
||||
const auto reindex_ops = ReindexAmbiguityOperators(cell, sg->number);
|
||||
std::vector<double> candidate_r_free{best.r_free}; // identity first, then the twin laws
|
||||
//
|
||||
// The candidates are a property of the DATA's lattice and point group: it is the observed
|
||||
// intensities that are relabelled. Asking the model's group instead enumerates nothing at all
|
||||
// wherever the two groups differ - measured, a model in P4(3)2(1)2 against data merged in P4(3)
|
||||
// probes zero operators - which is exactly the case the probe exists for.
|
||||
const gemmi::SpaceGroup &ambiguity_sg = data_space_group != nullptr ? *data_space_group : *sg;
|
||||
const std::vector<gemmi::Op> reindex_ops =
|
||||
probe_indexing_ambiguity ? ReindexAmbiguityOperators(cell, ambiguity_sg)
|
||||
: std::vector<gemmi::Op>{};
|
||||
|
||||
struct IndexingProbe {
|
||||
gemmi::Op op = gemmi::Op::identity(); // the lowest-R-free relabelling
|
||||
double margin = 0; // by how much in R-free it leads the runner-up
|
||||
};
|
||||
auto probe_indexing = [&](ModelState &ms) {
|
||||
IndexingProbe out;
|
||||
std::vector<double> r_free{fit_model(ms, obs).r_free}; // identity first, then the twin laws
|
||||
double best_r_free = r_free.front();
|
||||
for (const auto &op : reindex_ops) {
|
||||
Fit cand = fit_model(ReindexReflections(obs, op));
|
||||
candidate_r_free.push_back(cand.r_free);
|
||||
if (cand.r_free < best.r_free) { best = std::move(cand); result.indexing_op = op; }
|
||||
const double cand = fit_model(ms, ReindexReflections(obs, op)).r_free;
|
||||
r_free.push_back(cand);
|
||||
if (cand < best_r_free) {
|
||||
best_r_free = cand;
|
||||
out.op = op;
|
||||
}
|
||||
}
|
||||
if (!reindex_ops.empty()) {
|
||||
// The runner-up as well as the winner: the margin between them is what says whether the
|
||||
// choice was made on evidence, and on weak data the two can come out within noise.
|
||||
std::sort(candidate_r_free.begin(), candidate_r_free.end());
|
||||
logger.Info("Model validation: no reference - probed {} indexing solution(s) against the model; "
|
||||
"{} (R-free {:.4f}, runner-up {:.4f})", candidate_r_free.size(),
|
||||
result.indexing_op == gemmi::Op::identity() ? "kept the current indexing"
|
||||
: "reindexed to the lower-R-free solution",
|
||||
candidate_r_free[0], candidate_r_free[1]);
|
||||
// The runner-up as well as the winner: the margin between them is what says whether the choice
|
||||
// was made on evidence, and on weak data the two can come out within noise.
|
||||
std::sort(r_free.begin(), r_free.end());
|
||||
out.margin = r_free[1] - r_free[0];
|
||||
return out;
|
||||
};
|
||||
|
||||
const std::vector<gemmi::Position> as_read = ModelPositions(st.models[0]);
|
||||
|
||||
IndexingProbe indexing;
|
||||
std::vector<MergedReflection> reindexed;
|
||||
const std::vector<MergedReflection> *obs_model = &obs;
|
||||
if (!reindex_ops.empty()) {
|
||||
indexing = probe_indexing(mdl);
|
||||
if (!(indexing.op == gemmi::Op::identity())) {
|
||||
reindexed = ReindexReflections(obs, indexing.op);
|
||||
obs_model = &reindexed;
|
||||
}
|
||||
}
|
||||
Placement real = place_and_fit(mdl, *obs_model);
|
||||
|
||||
// --- is there anything for the model to decide? ---
|
||||
// There are only two: the space-group label, where the model asserts the other enantiomorph, and
|
||||
// the indexing, where a relabelling of the data fits it better. A model that asserts neither -
|
||||
// already in the group the data were merged in, and preferring their indexing - has made no claim
|
||||
// that needs arbitrating, and the null below would be several seconds spent gating a decision
|
||||
// nobody is making. That is the isomorphous case a screening campaign is made of, and it is the
|
||||
// one that has to be fast. Nothing else changes: the R-factors, the maps and the placement are
|
||||
// computed and reported exactly as in every other case.
|
||||
const bool decision_pending = result.model_enantiomorph_candidate
|
||||
|| !(indexing.op == gemmi::Op::identity());
|
||||
// A null that cannot be built is a question that could not be put, and this design already has a
|
||||
// state for that. Without this the run dies here - model validation runs BEFORE the reflection
|
||||
// files are written, so one failed replicate would take the .mtz, .cif, .hkl and _unmerged.mtz
|
||||
// with it, after the merge has already been paid for. The replicates are rotated models fed to a
|
||||
// scaling path that throws on data it cannot pair up, so this is the adversarial input for it.
|
||||
try {
|
||||
if (decision_pending) {
|
||||
// --- the null: this model, this data, in random orientations ---
|
||||
// An R-factor on its own says nothing about whether a model belongs to a crystal. What a model
|
||||
// that explains nothing reaches against these data depends on its atom count and its B-factors as
|
||||
// much as on the data, so a threshold on R calibrated on one dataset misjudges the next; and the
|
||||
// classical random-structure value does not apply either, because the scale here is fitted to
|
||||
// minimise the very sum the R is made of. The only null that fits is made of the same model: it is
|
||||
// reoriented about its own centroid and run through the identical fit and rigid-body placement,
|
||||
// and the real fit is asked how far above the resulting distribution it sits.
|
||||
//
|
||||
// R-work, not R-free. Not because nothing is refined against it - the placement's six parameters
|
||||
// are, and the scale's four - but because every null replicate is placed and scaled the same
|
||||
// way, so whatever that optimism is worth is bought on both sides and cancels in
|
||||
// (mean - real)/sd. R-work is then decided on an order of magnitude more reflections.
|
||||
//
|
||||
// The replicates are fitted to the data as merged even where a reindexing won above: a reindexing
|
||||
// is a relabelling of the same intensities, and a model in a random orientation has no more to do
|
||||
// with one labelling than with the other.
|
||||
logger.Info("Model validation: scoring the fit against a null of {} random placements of the same "
|
||||
"model", NULL_REPLICATES);
|
||||
// The orientations are drawn here, up front and in order, and not inside the loop: the
|
||||
// replicates run at the same time below, and a draw taken on whichever thread reached the
|
||||
// generator first would make the answer depend on the scheduling. Replicate i gets rotation i
|
||||
// on every run, so the sigma reported is a property of the data and not of the machine.
|
||||
std::vector<gemmi::Mat33> null_rotation;
|
||||
{
|
||||
std::mt19937 rng(NULL_SEED);
|
||||
for (int i = 0; i < NULL_REPLICATES; i++)
|
||||
null_rotation.push_back(random_rotation(rng));
|
||||
}
|
||||
// Each replicate works on its own copy of the model and of its structure factors, and the
|
||||
// replicates share nothing else, so they run concurrently and the real model - which the
|
||||
// maps and the atom-density readout below are still taken from - is never moved.
|
||||
std::vector<double> null_r_work(NULL_REPLICATES), null_margin(NULL_REPLICATES);
|
||||
ParallelFor(NULL_REPLICATES, nthreads, [&](int i) {
|
||||
ModelState rep;
|
||||
rep.st = mdl.st;
|
||||
SetModelPositions(rep.st.models[0], as_read);
|
||||
reorient_about_centroid(rep.st.models[0], null_rotation[i]);
|
||||
compute_model_factors(rep);
|
||||
if (!reindex_ops.empty())
|
||||
null_margin[i] = probe_indexing(rep).margin;
|
||||
null_r_work[i] = place_and_fit(rep, obs).fit.r_work;
|
||||
});
|
||||
const auto [null_mean, null_sd] = mean_sd(null_r_work);
|
||||
result.fit_tested = true;
|
||||
result.null_replicates = NULL_REPLICATES;
|
||||
result.null_r_work_mean = null_mean;
|
||||
result.null_r_work_sd = null_sd;
|
||||
|
||||
auto record_fit = [&](const Placement &p) {
|
||||
// Guarded at a floor, not at zero: sd == 0 and sd == 1e-7 are one ulp apart and land on
|
||||
// opposite verdicts, and a null with no spread - a model too small or too symmetric for a
|
||||
// rotation about its own centroid to move |Fcalc| - is the case that produces it. Measured
|
||||
// nulls sit near 0.60 with sd ~0.009, so this is nowhere near anything genuine.
|
||||
constexpr double NULL_SD_FLOOR = 1e-3;
|
||||
result.r_work_sigma =
|
||||
null_sd >= NULL_SD_FLOOR ? (null_mean - p.fit.r_work) / null_sd : 0.0;
|
||||
result.model_fits = result.r_work_sigma >= MODEL_FIT_SIGMA;
|
||||
};
|
||||
record_fit(real);
|
||||
|
||||
// --- does the model get to decide anything? ---
|
||||
// The R-factors, the maps and the rigid-body placement are statements about the MODEL and are
|
||||
// always computed and always reported, fit or no fit: a model that does not describe these data
|
||||
// still has an R against them, and that is the negative result. What the gate below controls is
|
||||
// the two decisions that rewrite the DATA - the space-group label and the indexing - which a
|
||||
// wrong model must not be able to make.
|
||||
if (!reindex_ops.empty()) {
|
||||
result.indexing_probed = true;
|
||||
result.indexing_margin = indexing.margin;
|
||||
const auto [margin_mean, margin_sd] = mean_sd(null_margin);
|
||||
result.indexing_margin_null_mean = margin_mean;
|
||||
result.indexing_margin_null_sd = margin_sd;
|
||||
constexpr double MARGIN_SD_FLOOR = 1e-4;
|
||||
result.indexing_margin_sigma =
|
||||
margin_sd >= MARGIN_SD_FLOOR ? (indexing.margin - margin_mean) / margin_sd : 0.0;
|
||||
// The margin is judged against the margin a random placement produces, not against a value.
|
||||
// A random model also picks a winner, and on these data it picks one by a comparable lead
|
||||
// (measured), so the raw margin says nothing on its own.
|
||||
const bool decided = result.model_fits && result.indexing_margin_sigma >= MODEL_FIT_SIGMA;
|
||||
if (decided)
|
||||
result.indexing_op = indexing.op;
|
||||
logger.Info("Model validation: probed {} indexing solution(s) against the model; the winner "
|
||||
"leads the runner-up by {:.4f} in R-free, against {:.4f} +- {:.4f} for a random "
|
||||
"placement of the same model ({:+.2f} sigma) => {}",
|
||||
reindex_ops.size() + 1, indexing.margin, margin_mean, margin_sd,
|
||||
result.indexing_margin_sigma,
|
||||
!decided ? "not decided; the data keep the indexing they were merged in"
|
||||
: (result.indexing_op == gemmi::Op::identity()
|
||||
? "kept the current indexing" : "reindexed to the model's"));
|
||||
if (!decided && !(indexing.op == gemmi::Op::identity())) {
|
||||
// The R-factors and the maps have to describe the reflections the files carry, and those
|
||||
// are now the ones the merge produced, so the fit is remade on them.
|
||||
SetModelPositions(st.models[0], as_read);
|
||||
compute_model_factors(mdl);
|
||||
real = place_and_fit(mdl, obs);
|
||||
record_fit(real);
|
||||
}
|
||||
}
|
||||
logger.Info("Model validation: R-work {:.4f} against a null of {:.4f} +- {:.4f} = {:+.2f} sigma "
|
||||
"=> the model {} these data",
|
||||
real.fit.r_work, null_mean, null_sd, result.r_work_sigma,
|
||||
result.model_fits ? "FITS" : "DOES NOT FIT");
|
||||
} else {
|
||||
logger.Info("Model validation: the model asserts no other enantiomorph for these data and "
|
||||
"prefers the indexing they were merged in, so it claims nothing about the written "
|
||||
"reflections and no null was run; R-work {:.4f}, R-free {:.4f}",
|
||||
real.fit.r_work, real.fit.r_free);
|
||||
}
|
||||
} catch (const std::exception &e) {
|
||||
// The null could not be built, so no decision is licensed and none is taken - which is the
|
||||
// NOT_TESTED state, not a failure of the run. If the indexing probe had already won on a
|
||||
// relabelling, the fit that survives describes those reflections and the files will not carry
|
||||
// them, so it is remade on the data as merged.
|
||||
logger.Warning("Model validation: the null could not be built ({}), so the model decides "
|
||||
"nothing; the reflections keep the group and the indexing they were merged in",
|
||||
e.what());
|
||||
result.fit_tested = false;
|
||||
result.model_fits = false;
|
||||
result.indexing_probed = false;
|
||||
result.indexing_op = gemmi::Op::identity();
|
||||
result.r_work_sigma = 0.0;
|
||||
SetModelPositions(st.models[0], as_read);
|
||||
compute_model_factors(mdl);
|
||||
real = place_and_fit(mdl, obs);
|
||||
}
|
||||
|
||||
Fit &best = real.fit;
|
||||
result.rigid_body_applied = real.rb_applied;
|
||||
result.rigid_body_angle_deg = real.rb_angle_deg;
|
||||
result.rigid_body_shift_A = real.rb_shift_A;
|
||||
result.r_free_before_rigid_body = real.r_free_before_rb;
|
||||
|
||||
gemmi::AsuData<std::complex<float>> &fmodel = best.fmodel;
|
||||
gemmi::AsuData<std::complex<float>> &map2fofc = best.map2fofc;
|
||||
gemmi::AsuData<std::complex<float>> &mapfofc = best.mapfofc;
|
||||
std::unordered_map<long, std::pair<double, bool>> &obs_by_hkl = best.obs_by_hkl;
|
||||
|
||||
result.r_work = best.r_work;
|
||||
@@ -295,7 +609,53 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
result.b_sol = best.b_sol;
|
||||
result.k_overall = best.k_overall;
|
||||
|
||||
// --- write the maps and score the 2Fo-Fc map at atom centres (a real map peaks there) ---
|
||||
// --- sigma_A weighting: the maps are 2mFo-DFc and mFo-DFc, not 2Fo-Fc and Fo-Fc ---
|
||||
// m and D come from a maximum-likelihood sigma_A per resolution shell, so a shell the model
|
||||
// describes badly is damped rather than carried into the map at full weight, and the difference
|
||||
// map is correspondingly less biased towards the model that made its phases.
|
||||
//
|
||||
// m and D are estimated on THIS dataset, so two datasets of one crystal form get slightly
|
||||
// different weights, and to that extent their maps are no longer scaled identically - the same
|
||||
// property the scaling above deliberately protects. It is kept anyway: the difference between two
|
||||
// datasets' sigma_A curves is the difference in how well the model explains each of them, which is
|
||||
// real and is what a screening campaign is looking for, and a PanDDA-style analysis consumes
|
||||
// 2mFo-DFc maps and normalizes each dataset's map against the ensemble before comparing them. The
|
||||
// per-reflection FOM is written to the MTZ so the weighting can be undone.
|
||||
// Following Read (1986) Acta Cryst. A42, 140-149
|
||||
struct MapTerm { gemmi::Miller hkl; double fo, fc, phi; bool free, centric; };
|
||||
std::vector<MapTerm> terms;
|
||||
std::vector<SigmaAReflection> sa_input;
|
||||
for (const auto &hv : fmodel.v) {
|
||||
const auto it = obs_by_hkl.find(hkl_key(hv.hkl));
|
||||
if (it == obs_by_hkl.end()) continue;
|
||||
const double Fo = it->second.first;
|
||||
const double Fc = std::abs(hv.value);
|
||||
const bool centric = gops.is_reflection_centric(hv.hkl);
|
||||
terms.push_back({hv.hkl, Fo, Fc, std::arg(hv.value), it->second.second, centric});
|
||||
sa_input.push_back({Fo, Fc, ucell.calculate_1_d2(hv.hkl), gops.epsilon_factor(hv.hkl),
|
||||
centric, it->second.second});
|
||||
}
|
||||
const SigmaAResult sigma_a = EstimateSigmaA(sa_input, ucell);
|
||||
result.mean_fom = sigma_a.mean_fom;
|
||||
result.sigma_a_shells = sigma_a.shells;
|
||||
|
||||
gemmi::AsuData<std::complex<float>> map2fofc, mapfofc;
|
||||
map2fofc.unit_cell_ = ucell; map2fofc.spacegroup_ = sg;
|
||||
mapfofc.unit_cell_ = ucell; mapfofc.spacegroup_ = sg;
|
||||
std::vector<float> fwt(terms.size()), delfwt(terms.size());
|
||||
for (size_t i = 0; i < terms.size(); i++) {
|
||||
const double m = sigma_a.weight[i].m, D = sigma_a.weight[i].d;
|
||||
// A centric reflection's phase is either exactly right or 180 degrees wrong, never in
|
||||
// between, so its bias-free coefficient is mFo and not 2mFo - DFc.
|
||||
fwt[i] = static_cast<float>(terms[i].centric ? m * terms[i].fo
|
||||
: 2 * m * terms[i].fo - D * terms[i].fc);
|
||||
delfwt[i] = static_cast<float>(m * terms[i].fo - D * terms[i].fc);
|
||||
const std::complex<float> ph = std::polar(1.0f, static_cast<float>(terms[i].phi));
|
||||
map2fofc.v.push_back({terms[i].hkl, fwt[i] * ph});
|
||||
mapfofc.v.push_back({terms[i].hkl, delfwt[i] * ph});
|
||||
}
|
||||
|
||||
// --- write the maps and score the 2mFo-DFc map at atom centres (a real map peaks there) ---
|
||||
const gemmi::Grid<float> grid2fofc = map_from_coefficients(map2fofc);
|
||||
const double rms2 = write_ccp4(grid2fofc, output_prefix + "_2fofc.ccp4");
|
||||
write_ccp4(map_from_coefficients(mapfofc), output_prefix + "_fofc.ccp4");
|
||||
@@ -320,10 +680,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
// Friedel ASU of that frame, so an anomalous merge - which keeps each mate as a row of its own,
|
||||
// both carrying the same pair - is read on its + rows only. Taking the - rows as well would
|
||||
// give one reflection both signs of its difference, and the last row written would decide.
|
||||
const gemmi::SpaceGroup *data_sg = data_space_group_number
|
||||
? gemmi::find_spacegroup_by_number(*data_space_group_number) : nullptr;
|
||||
if (data_sg == nullptr)
|
||||
data_sg = sg;
|
||||
const gemmi::SpaceGroup *data_sg = data_space_group != nullptr ? data_space_group : sg;
|
||||
const gemmi::ReciprocalAsu data_asu(data_sg);
|
||||
const gemmi::GroupOps data_gops = data_sg->operations();
|
||||
|
||||
@@ -391,12 +748,42 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
}
|
||||
}
|
||||
|
||||
// --- the enantiomorph, now that there is something to decide it on ---
|
||||
// Two things have to hold before the model's hand is written on these data. The model has to
|
||||
// describe them at all - the label is an assertion about the crystal, and a model that fits no
|
||||
// better than its own random placements is in no position to make one. And the anomalous map, the
|
||||
// only measurement here that is sensitive to the hand at all, must not contradict it: R-free
|
||||
// cannot (measured, inverting the model through the origin moves R-work by less than 1e-4, because
|
||||
// |F(h)| of the inverted structure is |F(-h)| = |F(h)| on Friedel-averaged data), so where the
|
||||
// anomalous differences do say something they say it alone, and they get a veto.
|
||||
result.adopted_model_enantiomorph = result.model_enantiomorph_candidate && result.model_fits
|
||||
&& !result.anomalous_hands_disagree;
|
||||
if (result.model_enantiomorph_candidate && !result.adopted_model_enantiomorph)
|
||||
logger.Warning("Model validation: the model asserts the enantiomorph {} against the data's {}, "
|
||||
"and that assertion is NOT taken up: {}. The reflections are written in the "
|
||||
"group they were merged in",
|
||||
sg->hm, data_space_group != nullptr ? data_space_group->hm : "?",
|
||||
result.anomalous_hands_disagree
|
||||
? "the anomalous density at the model's atoms is inverted"
|
||||
: "the model does not fit these data");
|
||||
|
||||
// --- MTZ of map coefficients so the maps can be re-opened / rebuilt in Coot etc. ---
|
||||
try {
|
||||
gemmi::Mtz mtz(true);
|
||||
mtz.spacegroup = sg;
|
||||
// The group the REFLECTIONS end up in, which is the model's only where its hand was adopted -
|
||||
// AdoptModelFrame decides this the same way a few lines below. An enantiomorphic pair indexes
|
||||
// identically, so the coefficients are the same numbers either way and only the label moves;
|
||||
// but the two groups have different screw translations, so a reader that expands symmetry out
|
||||
// of this file works in the wrong one if the label disagrees with the .mtz beside it.
|
||||
mtz.spacegroup = result.adopted_model_enantiomorph
|
||||
? sg
|
||||
// No data group means the caller merged in P1, and P1 is what the
|
||||
// reflection files beside this one carry - not the model's group.
|
||||
: (data_space_group != nullptr ? data_space_group
|
||||
: gemmi::find_spacegroup_by_number(1));
|
||||
mtz.set_cell_for_all(ucell);
|
||||
mtz.add_dataset("model_validation");
|
||||
mtz.datasets.back().wavelength = wavelength_A;
|
||||
mtz.add_column("FP", 'F', -1, -1, false);
|
||||
mtz.add_column("FC", 'F', -1, -1, false);
|
||||
mtz.add_column("PHIC", 'P', -1, -1, false);
|
||||
@@ -404,25 +791,26 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
mtz.add_column("PHWT", 'P', -1, -1, false);
|
||||
mtz.add_column("DELFWT", 'F', -1, -1, false);
|
||||
mtz.add_column("PHDELWT", 'P', -1, -1, false);
|
||||
// The figure of merit the coefficients carry, so the weighting can be read off - and undone -
|
||||
// from the file rather than having to be taken on trust.
|
||||
mtz.add_column("FOM", 'W', -1, -1, false);
|
||||
mtz.add_column("FREE", 'I', -1, -1, false);
|
||||
std::vector<float> data;
|
||||
int nref = 0;
|
||||
for (const auto &hv : fmodel.v) {
|
||||
auto it = obs_by_hkl.find(hkl_key(hv.hkl));
|
||||
if (it == obs_by_hkl.end()) continue;
|
||||
double Fo = it->second.first;
|
||||
double Fc = std::abs(hv.value);
|
||||
double phi_deg = gemmi::phase_in_angles(hv.value);
|
||||
data.insert(data.end(), {static_cast<float>(hv.hkl[0]), static_cast<float>(hv.hkl[1]),
|
||||
static_cast<float>(hv.hkl[2]),
|
||||
static_cast<float>(Fo), static_cast<float>(Fc),
|
||||
static_cast<float>(phi_deg),
|
||||
static_cast<float>(2 * Fo - Fc), static_cast<float>(phi_deg),
|
||||
static_cast<float>(Fo - Fc), static_cast<float>(phi_deg),
|
||||
it->second.second ? 0.0f : 1.0f});
|
||||
++nref;
|
||||
for (size_t i = 0; i < terms.size(); i++) {
|
||||
// [0, 360), the convention the MTZ 'P' columns carried before sigma_A weighting: std::arg
|
||||
// returns (-180, 180] and a P column is not supposed to.
|
||||
const double phi_wrapped = terms[i].phi * 180.0 / PI;
|
||||
const auto phi_deg = static_cast<float>(phi_wrapped < 0 ? phi_wrapped + 360.0 : phi_wrapped);
|
||||
data.insert(data.end(), {static_cast<float>(terms[i].hkl[0]), static_cast<float>(terms[i].hkl[1]),
|
||||
static_cast<float>(terms[i].hkl[2]),
|
||||
static_cast<float>(terms[i].fo), static_cast<float>(terms[i].fc),
|
||||
phi_deg,
|
||||
fwt[i], phi_deg,
|
||||
delfwt[i], phi_deg,
|
||||
static_cast<float>(sigma_a.weight[i].m),
|
||||
terms[i].free ? 0.0f : 1.0f});
|
||||
}
|
||||
mtz.nreflections = nref;
|
||||
mtz.nreflections = static_cast<int>(terms.size());
|
||||
mtz.data = std::move(data);
|
||||
mtz.write_to_file(output_prefix + "_maps.mtz");
|
||||
} catch (const std::exception &e) {
|
||||
@@ -431,12 +819,18 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
|
||||
result.ok = true;
|
||||
result.maps_prefix = output_prefix;
|
||||
// The placed coordinates, for the caller to write out beside the maps once the frame is settled.
|
||||
result.placed_model = std::make_shared<gemmi::Structure>(st);
|
||||
logger.Info("Model validation: R-work={:.4f} ({} refl) R-free={:.4f} ({} refl) "
|
||||
"[overall + anisotropic B + bulk solvent]",
|
||||
result.r_work, result.n_work, result.r_free, result.n_free);
|
||||
logger.Info("Model validation: bulk solvent k_sol={:.3f} b_sol={:.1f}, k_overall={:.3f}",
|
||||
result.k_sol, result.b_sol, result.k_overall);
|
||||
logger.Info("Model validation: mean 2Fo-Fc density at atom centres = {:.2f} sigma", result.mean_atom_density_sigma);
|
||||
logger.Info("Model validation: mean 2mFo-DFc density at atom centres = {:.2f} sigma", result.mean_atom_density_sigma);
|
||||
logger.Info("Model validation: sigma_A weighting over {} resolution shell(s) (estimated on the {} "
|
||||
"free reflections): sigma_A {:.2f} at low resolution, {:.2f} at high, mean FOM {:.3f}",
|
||||
sigma_a.shells, sigma_a.free_reflections, sigma_a.sigma_a_lowest_shell,
|
||||
sigma_a.sigma_a_highest_shell, sigma_a.mean_fom);
|
||||
if (!result.anomalous_sites.empty()) {
|
||||
std::string sites;
|
||||
for (const auto &s : result.anomalous_sites)
|
||||
@@ -455,6 +849,11 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
|
||||
output_prefix, output_prefix,
|
||||
result.anomalous_sites.empty() ? "" : ", " + output_prefix + "_anom.ccp4",
|
||||
output_prefix);
|
||||
if (result.fit_tested && !result.model_fits)
|
||||
logger.Info("Model validation: the model was rejected, so nothing downstream moved - the "
|
||||
"reflection files are byte for byte the ones a run with no model would have "
|
||||
"written. The R-factors and the maps above still describe this model against "
|
||||
"these data, and are the negative result rather than a failure");
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -471,14 +870,17 @@ std::string hkl_triplet(const gemmi::Op &op) {
|
||||
|
||||
} // namespace
|
||||
|
||||
int AdoptModelFrame(const ModelValidationResult &validation,
|
||||
std::vector<MergedReflection> &merged,
|
||||
int data_space_group_number,
|
||||
bool merge_friedel,
|
||||
Logger &logger) {
|
||||
int space_group_number = data_space_group_number;
|
||||
if (!validation.ok)
|
||||
return space_group_number;
|
||||
const gemmi::SpaceGroup *AdoptModelFrame(const ModelValidationResult &validation,
|
||||
std::vector<MergedReflection> &merged,
|
||||
const gemmi::SpaceGroup &data_space_group,
|
||||
bool merge_friedel,
|
||||
Logger &logger) {
|
||||
const gemmi::SpaceGroup *space_group = &data_space_group;
|
||||
// A model that could not be read, or that did not fit, decides nothing: the reflections are
|
||||
// written exactly as a run with no model at all would have written them. Both flags are already
|
||||
// folded into the fields below, and the guard states the rule where the frame is actually adopted.
|
||||
if (!validation.ok || !validation.model_fits)
|
||||
return space_group;
|
||||
|
||||
// Adopting the model's enantiomorph is a change of the space-group LABEL and nothing else. The
|
||||
// two groups have the same rotation operations, so the same reflections, indexed the way they
|
||||
@@ -486,24 +888,24 @@ int AdoptModelFrame(const ModelValidationResult &validation,
|
||||
// group that agrees with the model it will be refined against. Reindexing here would swap the
|
||||
// Bijvoet mates and so change the data - see the note in ValidateAgainstModel.
|
||||
if (validation.adopted_model_enantiomorph && validation.model_space_group_number > 0) {
|
||||
space_group_number = validation.model_space_group_number;
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(space_group_number);
|
||||
space_group = gemmi::find_spacegroup_by_number(validation.model_space_group_number);
|
||||
logger.Info("Model validation: the written reflections take the model's enantiomorph, {} ({}), "
|
||||
"as a label - no reflection moved", sg ? sg->short_name() : "?", space_group_number);
|
||||
"as a label - no reflection moved",
|
||||
space_group ? space_group->short_name() : "?", validation.model_space_group_number);
|
||||
}
|
||||
|
||||
// The alternative indexing, by contrast, is metric- and group-preserving: only the labels move.
|
||||
if (!(validation.indexing_op == gemmi::Op::identity())) {
|
||||
merged = ReindexMergedIntoAsu(merged, validation.indexing_op, space_group_number, merge_friedel);
|
||||
merged = ReindexMergedIntoAsu(merged, validation.indexing_op, *space_group, merge_friedel);
|
||||
logger.Info("Model validation: the written reflections take the model's indexing, reindexed by {}",
|
||||
hkl_triplet(validation.indexing_op));
|
||||
}
|
||||
return space_group_number;
|
||||
return space_group;
|
||||
}
|
||||
|
||||
std::vector<MergedReflection> ModelReferenceIntensities(const std::string &model_path,
|
||||
const std::optional<UnitCell> &cell,
|
||||
std::optional<int> space_group_number,
|
||||
const gemmi::SpaceGroup *space_group,
|
||||
double d_min,
|
||||
Logger &logger) {
|
||||
std::vector<MergedReflection> out;
|
||||
@@ -542,9 +944,8 @@ std::vector<MergedReflection> ModelReferenceIntensities(const std::string &model
|
||||
st.cell = target;
|
||||
}
|
||||
}
|
||||
if (space_group_number.has_value())
|
||||
if (const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_number(*space_group_number))
|
||||
st.spacegroup_hm = sg->xhm();
|
||||
if (space_group != nullptr)
|
||||
st.spacegroup_hm = space_group->xhm();
|
||||
const gemmi::SpaceGroup *sg = st.find_spacegroup();
|
||||
if (!sg) {
|
||||
logger.Error("Model reference: model {} has no usable space group", model_path);
|
||||
|
||||
Reference in new issue
Block a user