mx/Jungfraujoch
3
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 81 Wiki Activity

rugnux --model: a wrong model confirms no shell, fits no hand, and fits the GPU reserve

Browse Source 
Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins):

CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections,
but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these
very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24
wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells).
The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the
data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth
in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in
Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser
shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on
the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the
deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL
0.05-0.13 in every shell). The report no longer says a model cannot agree by accident.

MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated
P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and
the best random placement fits as well as it does, and drawing the null about the cell axes instead
changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma
allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up.
The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and
a deposited model settles one at +4.15.

Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones
count them by the d of the index in the model's ASU, and a model's group need not keep the data's
metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died
with "zone over the engine's reserve". Counted the zones' way, under every indexing probed.

Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated
model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric
(fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
This commit is contained in:
leonarski_f
2026-10-11 21:30:34 +02:00
co-authored by Claude Opus 5.5
parent bd3c4efab6
commit ee350f1f74
7 changed files with 255 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ModelValidationTest.cpp
+75 -2
View File
@@ -76,9 +76,9 @@ ATOM 2 C . CB GLY A 1 12.000 14.000 16.000 1.00 20.00
// The same cell in a polar group: b is the unique axis, so the origin is free along y alone.
const char *kPolarCryst = "CRYST1 30.000 34.000 38.000 90.00 90.00 90.00 P 1 2 1 2\n";
std::string ClusterPdb(const char *cryst = kCryst) {
std::string ClusterPdb(const char *cryst = kCryst, unsigned seed = 20260902) {
std::string pdb = cryst;
std::mt19937 rng(20260902);
std::mt19937 rng(seed);
std::uniform_real_distribution<double> x(2, 14), y(2, 16), z(2, 18);
char line[96];
for (int i = 1; i <= 150; i++) {
@@ -720,6 +720,79 @@ TEST_CASE("ModelValidation_CCModelFollowsTheSignalByShell", "[ModelValidation]")
// The model path's structure factors come from FFTW; they must be gemmi's own transform to float
// precision, in the same layout, so prepare_asu_data() reads the same reflections from either.
// A wrong model may come in a group that does not keep the data's metric - a cubic group on an
// orthorhombic cell - and then its ASU holds an observation at another resolution than the data's own:
// more of them land inside the rigid body's finest zone than the data put there. The GPU engines are
// reserved for the zone as it is counted, so the validation runs instead of failing on the device.
TEST_CASE("ModelValidation_WrongModelInAGroupOffTheDataMetric", "[ModelValidation]") {
Logger logger("ModelValidation_WrongModelInAGroupOffTheDataMetric");
const auto data_path = WriteTemp("off_metric_data.pdb", ClusterPdb().c_str());
const auto wrong_path = WriteTemp("off_metric_wrong.pdb",
ClusterPdb("CRYST1 40.000 40.000 40.000 90.00 90.00 90.00 I 21 3 24\n", 7).c_str());
auto obs = ModelReferenceIntensities(data_path, {}, {}, 2.0, logger);
REQUIRE(obs.size() > 1000);
for (size_t i = 0; i < obs.size(); i++) {
obs[i].sigma = std::max(1.0f, 0.05f * obs[i].I);
obs[i].F = std::sqrt(std::max(0.0f, obs[i].I));
obs[i].rfree_flag = (i % 20) == 0;
}
const std::string prefix = (std::filesystem::temp_directory_path() / "off_metric_test").string();
const auto result =
ValidateAgainstModel(obs, UnitCell{.a = 30, .b = 34, .c = 38, .alpha = 90, .beta = 90, .gamma = 90},
wrong_path, prefix, logger, gemmi::find_spacegroup_by_name("P 21 21 21"));
CHECK(result.ok);
CHECK(!result.adopted_model_enantiomorph);
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(data_path);
std::filesystem::remove(wrong_path);
}
// A model of something else entirely, scaled to the data, shares their falloff across every shell, and
// by Fisher's transform on independent reflections that alone reads as a significant correlation. Against
// the null of the data at h with the model at h + d - which shares the falloff too - it must not.
TEST_CASE("ModelValidation_CCModelUnrelatedModelConfirmsNothing", "[ModelValidation]") {
Logger logger("ModelValidation_CCModelUnrelatedModelConfirmsNothing");
const auto data_path = WriteTemp("cc_null_data.pdb", ClusterPdb().c_str());
const auto wrong_path = WriteTemp("cc_null_wrong.pdb", ClusterPdb(kCryst, 7).c_str());
auto obs = ModelReferenceIntensities(data_path, {}, {}, 2.0, logger);
std::erase_if(obs, [](const MergedReflection &r) { return r.d > 8.0f; });
REQUIRE(obs.size() > 1000);
for (size_t i = 0; i < obs.size(); i++) {
// An anisotropic falloff, B 30 A^2 more along a* than along b* and c*: the validation fits an
// anisotropic B to the data and puts it on the model, whatever the model is.
const double s_a = obs[i].h / 30.0;
obs[i].I = static_cast<float>(obs[i].I * std::exp(-30.0 * s_a * s_a / 2.0));
obs[i].sigma = std::max(1.0f, 0.05f * obs[i].I);
obs[i].F = std::sqrt(std::max(0.0f, obs[i].I));
obs[i].rfree_flag = (i % 20) == 0;
}
const std::vector<float> shells{4.0f, 3.0f, 2.5f, 2.2f, 2.0f};
const std::string prefix = (std::filesystem::temp_directory_path() / "cc_null_test").string();
const auto result =
ValidateAgainstModel(obs, UnitCell{.a = 30, .b = 34, .c = 38, .alpha = 90, .beta = 90, .gamma = 90},
wrong_path, prefix, logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
/*probe_indexing_ambiguity=*/false, 1, 1.0, shells);
REQUIRE(result.ok);
REQUIRE(result.cc_model_shells.size() == shells.size());
double fisher_max = 0;
for (const auto &sh : result.cc_model_shells) {
const double fisher = std::atanh(sh.cc) * std::sqrt(sh.n - 3.0);
fisher_max = std::max(fisher_max, fisher);
logger.Info("unrelated model {:.2f} A: CC {:.3f} (null {:.3f}) on {}: {:+.1f} sigma, {:+.1f} by Fisher alone",
sh.d_min, sh.cc, sh.cc_null, sh.n, sh.sigma, fisher);
CHECK(sh.sigma < 3.0);
}
// The trap is there: on independent reflections some shell would have read as significant.
CHECK(fisher_max > 3.0);
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(data_path);
std::filesystem::remove(wrong_path);
}
TEST_CASE("ModelValidation_MapToFPhiMatchesGemmi", "[ModelValidation]") {
// An even grid, and an odd one on every axis (FFTW and pocketfft split odd lengths differently).
const auto size = GENERATE(std::array<int, 3>{20, 24, 30}, std::array<int, 3>{15, 21, 27});