mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 75 Wiki Activity

model validation: the placement target carries a bulk solvent that means something

Browse Source 
The rigid-body target refits the scale at every evaluation, deliberately - held fixed, the target
would measure the scale as much as the placement, and the body would translate to repair a scale
error instead of moving to where the density is. That refit was gemmi's unbounded fit, the one
already replaced for the reported scale, and here it was worse: measured over a corpus of
deposited models, 40% of the evaluations that decide where the model goes came out with a b_sol
outside 10-80 A^2, on 57 of 89 datasets, ranging from -8072 to +1721. A negative b_sol is a
solvent term that GROWS with resolution. One crystal ran its entire committed placement between
290 and 430 A^2, and that placement went into the reported maps.

The bulk solvent is fitted once per zone instead, inside the same physical box the reported fit
searches, and then held while the overall scale and the anisotropic B keep following the body.
That split is the point: k_sol and b_sol describe the crystal's disordered solvent, not the fit of
one placement, and measured across a whole zone they drift by a single grid step. Fitting them at
every evaluation costs three times the wall clock, makes the scaler 82% of the run, and moves
discontinuously under a forward difference - which is poison for a numerical Jacobian. This costs
4%, leaves no evaluation outside the box, and lands the body within 0.37 degrees of the expensive
version, against 4.33 degrees for what it replaces.

Placements change on fourteen crystals in eighty-nine. R-free is a wash on the mean; the step
buys more total R-free from fewer commits, and the gain sits where the runaway actually bit.

Two things found while auditing the file and left as they were, because they are right: the
refinement sees working reflections only, end to end, and the gauge that removes the origin-free
directions of a polar group carries more than half the whole refined translation in ninety-four
of a hundred and thirty-nine polar zones. It had no test. It has one now.

An observation with no calculated amplitude gets a zero residual, which drops it from the target
rather than scoring it as a perfect fit, and is now counted and reported - a large count says the
model's reflection conditions do not match the data's, which is a statement about the model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
This commit is contained in:
leonarski_f
2026-09-07 15:27:30 +02:00
co-authored by Claude Opus 5
parent 004fa5a781
commit e5c96cd41b
4 changed files with 149 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ModelValidationTest.cpp
+102 -6
View File
@@ -53,12 +53,16 @@ ATOM 2 C . CB GLY A 1 12.000 14.000 16.000 1.00 20.00
"ATOM 2 CB GLY A 1 12.000 14.000 16.000 1.00 20.00 C\n"
"END\n";
// A synthetic "protein": carbons filling one asymmetric unit of a small P2(1)2(1)2(1) cell. The
// space group matters - in P1 the origin is free in all three directions, so |F| does not change
// when the whole content is translated and there is no translation to recover. No specimen is
// involved; the positions come out of a fixed seed.
std::string ClusterPdb() {
std::string pdb = "CRYST1 30.000 34.000 38.000 90.00 90.00 90.00 P 21 21 21 4\n";
// A synthetic "protein": carbons filling one asymmetric unit of a small cell. The space group
// matters - in P1 the origin is free in all three directions, so |F| does not change when the
// whole content is translated and there is no translation to recover. No specimen is involved;
// the positions come out of a fixed seed.
const char *kCryst = "CRYST1 30.000 34.000 38.000 90.00 90.00 90.00 P 21 21 21 4\n";
// The same cell in a polar group: b is the unique axis, so the origin is free along y alone.
const char *kPolarCryst = "CRYST1 30.000 34.000 38.000 90.00 90.00 90.00 P 1 2 1 2\n";
std::string ClusterPdb(const char *cryst = kCryst) {
std::string pdb = cryst;
std::mt19937 rng(20260902);
std::uniform_real_distribution<double> x(2, 14), y(2, 16), z(2, 18);
char line[96];
@@ -185,6 +189,98 @@ TEST_CASE("ModelValidation_RigidBodyRecoversASmallShift", "[ModelValidation]") {
std::filesystem::remove(path);
}
// A polar space group leaves the origin free along one direction: moving the whole cell content
// along it multiplies every structure factor by a phase and changes no amplitude, so the data
// cannot say where the body sits along it and the refinement must not pretend otherwise. The check
// is closed - the "observed" amplitudes are the model's own - and the model is displaced in all
// three directions at once, so the same run says both what is recovered and what is left alone.
TEST_CASE("ModelValidation_RigidBodyLeavesThePolarDirectionAlone", "[ModelValidation]") {
Logger logger("ModelValidation_RigidBodyLeavesThePolarDirectionAlone");
const auto path = WriteTemp("rigid_body_polar_test.pdb", ClusterPdb(kPolarCryst).c_str());
gemmi::Structure st = gemmi::read_structure_gz(path, gemmi::CoorFormat::Detect);
const gemmi::SpaceGroup *sg = st.find_spacegroup();
REQUIRE(sg != nullptr);
st.setup_cell_images();
const auto ref = ModelReferenceIntensities(path, {}, {}, 3.0, logger);
REQUIRE_FALSE(ref.empty());
gemmi::AsuData<gemmi::ValueSigma<float>> fobs;
fobs.unit_cell_ = st.cell;
fobs.spacegroup_ = sg;
for (const auto &r : ref)
fobs.v.push_back({{{r.h, r.k, r.l}}, {std::sqrt(r.I), 1.0f}});
fobs.ensure_sorted();
// b is the unique axis of P 1 2 1, so y is the free direction and x and z are determined.
const std::vector<gemmi::Position> original = ModelPositions(st.models[0]);
std::vector<gemmi::Position> displaced;
for (const gemmi::Position &p : original)
displaced.emplace_back(p.x + 0.35, p.y + 0.50, p.z - 0.30);
SetModelPositions(st.models[0], displaced);
const RigidBodyRefineResult result =
RefineRigidBody(st.models[0], st.cell, *sg, fobs, 3.0, logger);
CHECK(result.converged);
const std::vector<gemmi::Position> refined = ModelPositions(st.models[0]);
gemmi::Vec3 left;
for (size_t i = 0; i < original.size(); i++)
left += gemmi::Vec3(refined[i]) - gemmi::Vec3(original[i]);
left *= 1.0 / static_cast<double>(original.size());
logger.Info("Rigid-body polar test: left over ({:.3f}, {:.3f}, {:.3f}) A", left.x, left.y, left.z);
CHECK(std::fabs(left.x) < 0.10);
CHECK(std::fabs(left.z) < 0.10);
// Along b nothing was refined away, because there is nothing there to refine.
CHECK(left.y == Catch::Approx(0.50).margin(0.02));
std::filesystem::remove(path);
}
// The bulk solvent the placement is scored through has to stay inside the range a flat solvent model
// means anything in. gemmi's own scaler is an unbounded Levenberg-Marquardt and reaches b_sol of
// hundreds or thousands of A^2, which does not corrupt a reported number here but distorts the target
// that decides where the model goes, at every one of the hundreds of evaluations.
TEST_CASE("ModelValidation_RigidBodySolventStaysPhysical", "[ModelValidation]") {
Logger logger("ModelValidation_RigidBodySolventStaysPhysical");
const auto path = WriteTemp("rigid_body_solvent_test.pdb", ClusterPdb().c_str());
gemmi::Structure st = gemmi::read_structure_gz(path, gemmi::CoorFormat::Detect);
const gemmi::SpaceGroup *sg = st.find_spacegroup();
REQUIRE(sg != nullptr);
st.setup_cell_images();
const auto ref = ModelReferenceIntensities(path, {}, {}, 3.0, logger);
REQUIRE_FALSE(ref.empty());
gemmi::AsuData<gemmi::ValueSigma<float>> fobs;
fobs.unit_cell_ = st.cell;
fobs.spacegroup_ = sg;
for (const auto &r : ref)
fobs.v.push_back({{{r.h, r.k, r.l}}, {std::sqrt(r.I), 1.0f}});
fobs.ensure_sorted();
// Turned right around, so the model explains nothing: that is where an unbounded solvent fit
// has nothing to hold it and runs away.
std::vector<gemmi::Position> turned = ModelPositions(st.models[0]);
gemmi::Vec3 centre;
for (const gemmi::Position &p : turned)
centre += p;
centre *= 1.0 / static_cast<double>(turned.size());
for (gemmi::Position &p : turned)
p = gemmi::Position(centre - (gemmi::Vec3(p) - centre));
SetModelPositions(st.models[0], turned);
const RigidBodyRefineResult result =
RefineRigidBody(st.models[0], st.cell, *sg, fobs, 3.0, logger);
logger.Info("Rigid-body solvent test: k_sol {:.3f}, b_sol {:.1f} A^2", result.k_sol, result.b_sol);
CHECK(result.k_sol >= 0.10);
CHECK(result.k_sol <= 0.60);
CHECK(result.b_sol >= 10.0);
CHECK(result.b_sol <= 80.0);
std::filesystem::remove(path);
}
// sigma_A is what says how much of the model to believe, so the two ends of its range are what the
// weighting has to get right: a model that explains the data completely, and one that explains none
// of it.