mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 92 Wiki Activity

rugnux --model rigid body: Fcalc composed from one P1 transform, analytic translation Jacobian

Browse Source 
B+C of the rigid-body speed-up plan (not bit-exact, by design):

- Fcalc is composed from the transform of ONE copy of the model, gridded without
  symmetrization: F(h) = n_cen * sum_S exp(2 pi i h.t) F1(hR), through a per-zone gather table
  (half-l index, Friedel flag, phase, Cartesian hR). Only the indices prepare_asu_data() would
  give a value are composed (ASU, strict d_min, Nyquist box, model-group absences, not 000),
  so the unmatched-observation semantics are unchanged. The mask stays symmetrized and is read
  at h directly.
- Translation columns of the Jacobian are exact (the phase factor's derivative), computed with
  every evaluation; rotation columns are forward differences needing only one copy gridded and
  transformed (3 per Jacobian instead of 6 full evaluations).
- The Jacobian holds the scale and the bulk-solvent mask at the evaluation's; the residuals keep
  both exact. The per-evaluation scale re-fit is folded in by Kaufman's variable projection.
- Composition and projection reductions are per-reflection / serial, so the result does not
  depend on the thread count.

Tests: RigidBodyP1FcalcMatchesSymmetrized (hkl set identical, <= 4.3e-6 of mean |F|),
RigidBodyTranslationDerivativeIsExact (5-point difference, <= 6.3e-4, the grid sampling),
RigidBodyRotationDerivativeMatchesDifferences (q != 0, anisotropic atoms, cos >= 0.9968),
RigidBodyJacobianMatchesNumericScaleRefit (cos >= 0.998 at 0.07 A; >= 0.983 at 0.35 A, the
residual-weighted term Kaufman drops). Existing rigid-body tests pass unchanged.

Validation on 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: 9hnc's real fit, which
moved 0.000 deg / 0.000 A before, now moves 1.46 deg / 0.65 A and R-free 0.5159 -> 0.4651
(MODEL_FIT_SIGMA 15.4 -> 43.3). STOP finding on 9yzk: a null replicate drawn outside the
redraw guard's 10.6 deg reach was walked 23.9 deg onto an equivalent orientation (R-free
0.5578 -> 0.3917), the null SD went 0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED (12.2 sigma)
-> REJECTED (2.6 sigma). The stronger optimiser's capture radius outgrows the null's redraw
guard; not ready to merge until that is addressed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-09-28 09:53:10 +02:00
1 parent 351be7de0f
commit e0e3e92147
7 files changed
+718 -245
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/ModelValidationTest.cpp
+260
View File
@@ -880,3 +880,263 @@ TEST_CASE("ModelValidation_ParallelGriddingMatchesGemmi", "[ModelValidation]") {
}
}
}
namespace {
// The five groups of ModelValidation_ParallelGriddingMatchesGemmi: none, a centring, screws, a
// cubic body centring and a cubic face centring with a diamond glide.
const char *kRigidBodyCrysts[] = {
"CRYST1 40.000 50.000 60.000 90.00 90.00 90.00 P 1 1\n",
"CRYST1 40.000 50.000 60.000 90.00 100.00 90.00 C 1 2 1 4\n",
"CRYST1 40.000 50.000 60.000 90.00 90.00 90.00 P 21 21 21 4\n",
"CRYST1 60.000 60.000 60.000 90.00 90.00 90.00 I 2 3 24\n",
"CRYST1 80.000 80.000 80.000 90.00 90.00 90.00 F 41 3 2 96\n",
};
// ClusterPdb() in `cryst`, every third atom anisotropic.
gemmi::Structure AnisoCluster(const char *cryst) {
const auto path = WriteTemp("rigid_body_composition_test.pdb", ClusterPdb(cryst).c_str());
gemmi::Structure st = gemmi::read_structure_gz(path, gemmi::CoorFormat::Detect);
std::filesystem::remove(path);
st.setup_cell_images();
int i = 0;
for (gemmi::Chain &ch : st.models[0].chains)
for (gemmi::Residue &r : ch.residues)
for (gemmi::Atom &a : r.atoms)
if (i++ % 3 == 0)
a.aniso = {0.30f, 0.25f, 0.20f, 0.02f, -0.01f, 0.03f};
return st;
}
// The zone's density calculator, as the rigid body sets it up for `model`.
gemmi::DensityCalculator<gemmi::IT92<float>, float> ZoneDensity(const gemmi::Structure &st,
const gemmi::Model &model, double d_min) {
gemmi::DensityCalculator<gemmi::IT92<float>, float> dc;
dc.d_min = d_min;
dc.rate = 1.5;
dc.grid.unit_cell = st.cell;
dc.grid.spacegroup = st.find_spacegroup();
dc.set_refmac_compatible_blur(model);
return dc;
}
}
// The rigid body's Fcalc is composed from the transform of one copy of the model instead of being
// taken from the symmetrized grid. The two are the same sum rearranged, so they must agree to
// rounding, and must give a value to exactly the reflections prepare_asu_data() does - the
// systematic absences of the group and a reflection sitting exactly on d_min (the (10 0 0) of the
// 60 A cubic cell at 6 A) are left out by both.
TEST_CASE("ModelValidation_RigidBodyP1FcalcMatchesSymmetrized", "[ModelValidation]") {
for (const char *cryst : kRigidBodyCrysts) {
const gemmi::Structure st = AnisoCluster(cryst);
const gemmi::Model &model = st.models[0];
for (double d_min : {6.0, 3.5}) {
auto sym = ZoneDensity(st, model, d_min);
sym.put_model_density_on_grid(model);
gemmi::FPhiGrid<float> sym_f = MapToFPhi(sym.grid);
const auto ref = sym_f.prepare_asu_data(d_min, sym.blur, false, false, false);
// Every index of the ASU in the grid's box, whatever its resolution, absences and 000 too.
std::vector<gemmi::Miller> candidates;
for (const auto &hv : sym_f.prepare_asu_data(0, 0, true, true, false).v)
candidates.push_back(hv.hkl);
auto copy = ZoneDensity(st, model, d_min);
PutModelDensityOnGrid(copy, model, {}, 4);
const SymmetryComposition composition(copy.grid, d_min, candidates);
std::vector<std::complex<double>> f;
composition.Compose(MapToFPhi(copy.grid), copy.blur, f, nullptr, 4);
REQUIRE(composition.Hkl().size() == ref.v.size());
double mean = 0, worst = 0;
for (size_t i = 0; i < ref.v.size(); i++) {
REQUIRE(composition.Hkl()[i] == ref.v[i].hkl);
mean += std::abs(ref.v[i].value) / static_cast<double>(ref.v.size());
worst = std::max(worst, std::abs(f[i] - std::complex<double>(ref.v[i].value)));
}
INFO(cryst << " at " << d_min << " A: worst " << worst / mean << " of the mean |F|");
CHECK(worst <= 1e-4 * mean);
}
}
}
// The derivative of the composed Fcalc with respect to a translation of the body is a phase factor
// per term, exact - against a central difference of the regridded, recomposed Fcalc. Along a
// direction the origin is free in (all three in P1, b in C2) the amplitude does not change at all.
TEST_CASE("ModelValidation_RigidBodyTranslationDerivativeIsExact", "[ModelValidation]") {
for (const char *cryst : kRigidBodyCrysts) {
const gemmi::Structure st = AnisoCluster(cryst);
const double d_min = 3.5;
const gemmi::DensityCalculator<gemmi::IT92<float>, float> zone = ZoneDensity(st, st.models[0], d_min);
gemmi::Grid<float> grid;
grid.unit_cell = zone.grid.unit_cell;
grid.spacegroup = zone.grid.spacegroup;
grid.set_size_from_spacing(zone.requested_grid_spacing(), gemmi::GridSizeRounding::Up);
std::vector<gemmi::Miller> candidates;
const gemmi::ReciprocalAsu asu(grid.spacegroup);
for (int h = -25; h <= 25; h++)
for (int k = -25; k <= 25; k++)
for (int l = -25; l <= 25; l++)
if (asu.is_in({{h, k, l}}))
candidates.push_back({{h, k, l}});
const SymmetryComposition composition(grid, d_min, candidates);
REQUIRE(composition.Hkl().size() > 200);
auto fcalc = [&](const gemmi::Vec3 &shift, std::vector<std::array<std::complex<double>, 3>> *df_dt) {
gemmi::Model model = st.models[0];
for (gemmi::Chain &ch : model.chains)
for (gemmi::Residue &r : ch.residues)
for (gemmi::Atom &a : r.atoms)
a.pos += gemmi::Position(shift);
auto dc = ZoneDensity(st, model, d_min);
PutModelDensityOnGrid(dc, model, {}, 4);
std::vector<std::complex<double>> f;
composition.Compose(MapToFPhi(dc.grid), dc.blur, f, df_dt, 4);
return f;
};
std::vector<std::array<std::complex<double>, 3>> df_dt;
const std::vector<std::complex<double>> f = fcalc({}, &df_dt);
// A five-point difference, whose truncation error goes as eps^4: the grid is float, and at the
// 1e-3 A of a plain central difference its rounding alone reads as 3e-3 of the derivative. What
// is left, about 5e-4, is the sampling - an atom gridded after a move is the moved atom only to
// the accuracy of the grid - which the exact derivative does not have and the difference does.
const double eps = 0.05;
const bool p1 = st.find_spacegroup()->number == 1, c2 = st.find_spacegroup()->number == 5;
for (int k = 0; k < 3; k++) {
gemmi::Vec3 e;
e.at(k) = eps;
const auto p_1 = fcalc(e, nullptr), m_1 = fcalc(-e, nullptr);
const auto p_2 = fcalc(2 * e, nullptr), m_2 = fcalc(-2 * e, nullptr);
double diff = 0, norm = 0, amplitude = 0;
for (size_t m = 0; m < f.size(); m++) {
const std::complex<double> numeric = (8.0 * (p_1[m] - m_1[m]) - (p_2[m] - m_2[m])) / (12 * eps);
diff += std::norm(numeric - df_dt[m][k]);
norm += std::norm(df_dt[m][k]);
amplitude += gemmi::sq(std::real(std::conj(f[m]) * df_dt[m][k]) / std::abs(f[m]));
}
INFO(cryst << " axis " << k << ": relative error " << std::sqrt(diff / norm)
<< ", amplitude part " << std::sqrt(amplitude / norm));
CHECK(std::sqrt(diff / norm) <= 1e-3);
if (p1 || (c2 && k == 1))
CHECK(std::sqrt(amplitude / norm) <= 1e-5);
}
}
}
namespace {
// The rigid body's Jacobian at q against a central difference of its own residuals - which
// re-fit the scale at every evaluation - with the bulk-solvent mask held, as the Jacobian holds
// it. Per column (those in `columns`): the cosine between the two and the ratio of their norms.
// Also logged, not checked: how far the difference moves once the mask is let move with the body,
// which is what holding it costs.
void CheckRigidBodyJacobian(const char *cryst, const double q0[6], const std::vector<int> &columns,
double min_cosine, double max_norm_error) {
Logger logger("CheckRigidBodyJacobian");
const gemmi::Structure st = AnisoCluster(cryst);
const gemmi::SpaceGroup *sg = st.find_spacegroup();
// "Observed" amplitudes: the model's own, as placed at q = 0.
const auto path = WriteTemp("rigid_body_jacobian_test.pdb", ClusterPdb(cryst).c_str());
const auto ref = ModelReferenceIntensities(path, {}, {}, 3.0, logger);
std::filesystem::remove(path);
REQUIRE_FALSE(ref.empty());
for (double zone : {6.0, 3.5}) {
gemmi::AsuData<gemmi::ValueSigma<float>> fobs;
fobs.unit_cell_ = st.cell;
fobs.spacegroup_ = sg;
for (const auto &r : ref)
if (st.cell.calculate_d({{r.h, r.k, r.l}}) >= zone)
fobs.v.push_back({{{r.h, r.k, r.l}}, {std::sqrt(r.I), 1.0f}});
fobs.ensure_sorted();
gemmi::Model model = st.models[0];
RigidBodyTarget target(model, st.cell, *sg, 4);
target.SetZone(fobs, zone);
const size_t n = target.NumObservations();
std::vector<double> r(n), jacobian(n * 6);
REQUIRE(target.Residuals(q0, r.data()));
REQUIRE(target.Jacobian(q0, jacobian.data()));
auto difference = [&](int j, double h) {
double qp[6], qm[6];
std::copy(q0, q0 + 6, qp);
std::copy(q0, q0 + 6, qm);
qp[j] += h;
qm[j] -= h;
std::vector<double> rp(n), rm(n), d(n);
REQUIRE(target.Residuals(qp, rp.data()));
REQUIRE(target.Residuals(qm, rm.data()));
for (size_t i = 0; i < n; i++)
d[i] = (rp[i] - rm[i]) / (2 * h);
return d;
};
std::vector<std::vector<double>> held(6), moving(6);
target.hold_mask = true;
REQUIRE(target.Residuals(q0, r.data())); // the mask the difference holds is q0's
for (int j : columns)
held[j] = difference(j, 0.005);
// The mask is binary on the grid, so with it moving the difference is taken at the step the
// Jacobian had when every column moved it (0.01 of the zone's resolution): at 0.005 A it
// mostly measures which grid points a few atoms happened to cross.
target.hold_mask = false;
for (int j : columns)
moving[j] = difference(j, 0.01 * zone);
double largest = 0;
for (int j : columns) {
double s = 0;
for (size_t i = 0; i < n; i++)
s += gemmi::sq(held[j][i]);
largest = std::max(largest, std::sqrt(s));
}
for (int j : columns) {
double dot = 0, ours = 0, theirs = 0, dot_moving = 0, norm_moving = 0;
for (size_t i = 0; i < n; i++) {
dot += jacobian[i * 6 + j] * held[j][i];
ours += gemmi::sq(jacobian[i * 6 + j]);
theirs += gemmi::sq(held[j][i]);
dot_moving += moving[j][i] * held[j][i];
norm_moving += gemmi::sq(moving[j][i]);
}
ours = std::sqrt(ours);
theirs = std::sqrt(theirs);
norm_moving = std::sqrt(norm_moving);
const double cosine = dot / (ours * theirs);
logger.Info("{} at {:.1f} A, column {}: cosine {:.5f}, norm {:.4f} of the difference's; "
"with the mask moving, cosine {:.4f} and norm {:.4f}",
std::string(cryst).substr(55, 11), zone, j, cosine, ours / theirs,
dot_moving / (norm_moving * theirs), norm_moving / theirs);
INFO(cryst << " at " << zone << " A, column " << j << ": cosine " << cosine
<< ", norm ratio " << ours / theirs);
if (theirs < 1e-3 * largest) {
// A direction the origin is free in: |F| does not change, so neither may the column.
CHECK(ours < 1e-2 * largest);
continue;
}
CHECK(cosine >= min_cosine);
CHECK(std::fabs(ours / theirs - 1) <= max_norm_error);
}
}
}
}
// The rotation columns at a placement already rotated (about 1.4 deg, so the step is taken from a
// rotated body and not from the model as read) with anisotropic atoms, whose U the placement does not
// turn with the body - the columns must differentiate exactly the function the residuals evaluate.
TEST_CASE("ModelValidation_RigidBodyRotationDerivativeMatchesDifferences", "[ModelValidation]") {
const double q0[6] = {0.12, -0.09, 0.07, 0.15, -0.10, 0.05};
for (const char *cryst : kRigidBodyCrysts)
CheckRigidBodyJacobian(cryst, q0, {0, 1, 2}, 0.99, 0.03);
}
// The whole Jacobian - analytic translation, forward-difference rotation, the scale re-fit folded in by
// projection - against the central difference of the residuals it describes. The projection is
// Kaufman's, which leaves out a term that grows with the residuals (the scale's derivatives depend on
// the placement too, weighted by how badly the model fits): with the body 0.07 A off it agrees with the
// difference to about 2e-3 in cosine, with it 0.35 A off to about 2e-2, and the bounds follow.
TEST_CASE("ModelValidation_RigidBodyJacobianMatchesNumericScaleRefit", "[ModelValidation]") {
const double close[6] = {0.0, 0.0, 0.0, 0.05, -0.04, 0.03};
const double far[6] = {0.0, 0.0, 0.0, 0.25, -0.20, 0.15};
for (const char *cryst : kRigidBodyCrysts) {
CheckRigidBodyJacobian(cryst, close, {0, 1, 2, 3, 4, 5}, 0.99, 0.02);
CheckRigidBodyJacobian(cryst, far, {0, 1, 2, 3, 4, 5}, 0.975, 0.03);
}
}