leonarski_fandClaude Opus 5
91edacdf71
The translational-NCS vector was refined only on d >= 4 A, the band its detection gate reads, where an error of 0.02 costs the contrast almost nothing. The L-test's partner steps and the twin-immune zone's per-class normalisation then assign a class, cos(2 pi h.u), to reflections at full resolution, where the phase error is |h| times the error in the vector: past about |h| = 12 the classes were no longer the physical ones, and on one corpus crystal they were anti-correlated with them over a whole band. The vector is now re-refined against all the intensities, up a ladder that doubles the number of reflections read at each step and starts each step from the previous one's answer, so the phase is never extrapolated further than it is known. The objective is the correlation between E^2 and cos(2 pi h.u) - neither the gate's max/min bin ratio nor the fitted amplitude survives a full-resolution population, both being ratios that run away where the cosine has little variance. The gate itself is untouched: which crystals are called is unchanged, only where the vector points. Measured as that correlation in bands of |h|, before against after: 0.57/0.61, 0.42/0.52, 0.19/0.30, 0.14/0.18 on one crystal and 0.14/0.52, -0.15/0.42, 0.00/0.20, -0.02/0.09 on another. The acentric control of the twin-immune zone moves towards its analytic 0.736 where the classes change (0.671 -> 0.748 on one), and no space-group or twinning verdict moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
801 lines
42 KiB
C++
801 lines
42 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "TranslationalNCS.h"
|
|
|
|
#include <algorithm>
|
|
#include <cmath>
|
|
#include <cstring>
|
|
#include <iomanip>
|
|
#include <limits>
|
|
#include <numbers>
|
|
#include <numeric>
|
|
#include <random>
|
|
#include <sstream>
|
|
#include <unordered_set>
|
|
|
|
#include <fftw3.h>
|
|
|
|
#include "../../common/FFTWPlannerLock.h"
|
|
|
|
namespace {
|
|
// The Patterson is measured in a band, not over the whole dataset: below ~20 A the bulk solvent
|
|
// and the beam stop dominate, above ~5 A the map answers a question about atoms rather than about
|
|
// the arrangement of molecules, which is what a pseudo-translation is.
|
|
constexpr double kPattersonDMin = 5.0;
|
|
constexpr double kPattersonDMax = 20.0;
|
|
// A pseudo-translation shorter than this is a vector inside one molecule, not a translation
|
|
// between two of them; the origin peak's own shoulder also reaches out this far.
|
|
constexpr double kMinPeakDistanceA = 15.0;
|
|
// A peak this high is not a pseudo-symmetry: the origin peak is the total, so a peak at this
|
|
// fraction of it means essentially every reflection is invariant under the translation, which
|
|
// makes it a lattice vector. Measured over 137 datasets the two populations do not overlap - the
|
|
// strongest genuine pseudo-translation reaches 62% of the origin, while an undeclared centring
|
|
// (a C-, I- or F-centred crystal merged in P1) reads 83-102%.
|
|
constexpr double kLatticeTranslationPct = 75.0;
|
|
constexpr double kGridSampling = 2.5; // grid spacing = d_min / this
|
|
constexpr int kPattersonShells = 20; // equal-count shells for the E^2 normalisation
|
|
constexpr int kMinPattersonReflections = 50;
|
|
|
|
constexpr double kModulationDMin = 4.0; // coarse enough that a small error in u does not
|
|
// scramble the phase h.u
|
|
constexpr int kModulationShells = 10;
|
|
constexpr int kPhaseBins = 12;
|
|
constexpr int kMinModulationReflections = 500;
|
|
constexpr double kWilsonOutlierE2 = 8.0;
|
|
constexpr double kMinShellISig = 1.0;
|
|
// 20 rather than a handful: the null is the median of that many greedy refinements from random
|
|
// starts, and with 6 draws its scatter was enough to move datasets sitting near the gate across
|
|
// it from one RNG seed to the next (measured: three of them). The refinement is a dot product
|
|
// per trial, so the extra draws cost a few ms.
|
|
constexpr int kModulationNullDraws = 20;
|
|
|
|
// Gate constants. Read off a corpus of 128 merged datasets, on which the pair fires on 7.0% of
|
|
// them - a pathology rate, and close to the ~6% of PDB entries Read, Adams & McCoy report with a
|
|
// large off-origin Patterson peak. Either criterion ALONE fires on roughly twice as many: a 40%
|
|
// peak can carry no modulation at all, and on a 70,000-reflection dataset a 4% peak reaches
|
|
// z = 18 while modulating nothing.
|
|
constexpr double kPeakZThreshold = 5.0;
|
|
constexpr double kModulationRatioThreshold = 2.5;
|
|
// A modulation this deep means the suppressed class carries under 5% of the other, i.e. it is
|
|
// very nearly systematically absent - which is what a LATTICE translation of a smaller cell looks
|
|
// like measured in the larger one. Over 137 datasets exactly one crystal reaches it (at 89x, the
|
|
// next-deepest genuine pseudo-symmetry being 14x), and its deposited entry is indeed a cell of
|
|
// half the volume, whose axis length this detector reports to within 0.2 A.
|
|
constexpr double kNearExtinctModulation = 20.0;
|
|
|
|
// Above this many grid points the null is cut from 30 draws to 10. A deterministic function of
|
|
// the cell, so the answer does not depend on the machine; no dataset in the corpus reaches it
|
|
// (the largest grid measured was 2.5 M points on a 545 A cell).
|
|
constexpr size_t kLargeGridPoints = 4u << 20;
|
|
|
|
struct Vec3 { double x, y, z; };
|
|
|
|
// Round up to the next even number whose only prime factors are 2, 3 and 5 - the sizes FFTW has
|
|
// codelets for.
|
|
int SmoothSize(int v) {
|
|
v = std::max(2, v);
|
|
for (;; ++v) {
|
|
if (v % 2 != 0)
|
|
continue;
|
|
int x = v;
|
|
for (int p : {2, 3, 5})
|
|
while (x % p == 0)
|
|
x /= p;
|
|
if (x == 1)
|
|
return v;
|
|
}
|
|
}
|
|
|
|
// Origin-equivalent points of the lattice: the origin plus the centring translations, which
|
|
// produce a large Patterson peak by construction and are not pseudo-translations.
|
|
std::vector<Vec3> OriginEquivalents(const gemmi::SpaceGroup *sg) {
|
|
std::vector<Vec3> out{{0.0, 0.0, 0.0}};
|
|
if (!sg)
|
|
return out;
|
|
for (const auto &t : sg->operations().cen_ops) {
|
|
const Vec3 v{t[0] / 24.0, t[1] / 24.0, t[2] / 24.0};
|
|
if (v.x != 0.0 || v.y != 0.0 || v.z != 0.0)
|
|
out.push_back(v);
|
|
}
|
|
return out;
|
|
}
|
|
|
|
// Shortest distance in A from a fractional position to any origin-equivalent lattice point.
|
|
double DistanceToOrigin(const gemmi::UnitCell &cell, const Vec3 &frac,
|
|
const std::vector<Vec3> &equivalents) {
|
|
double best = std::numeric_limits<double>::infinity();
|
|
for (const auto &c : equivalents)
|
|
for (int dx = -1; dx <= 1; ++dx)
|
|
for (int dy = -1; dy <= 1; ++dy)
|
|
for (int dz = -1; dz <= 1; ++dz) {
|
|
const gemmi::Position p = cell.orthogonalize(
|
|
gemmi::Fractional{frac.x - c.x + dx, frac.y - c.y + dy, frac.z - c.z + dz});
|
|
best = std::min(best, p.length());
|
|
}
|
|
return best;
|
|
}
|
|
|
|
// How well a vector explains the intensities of the first `n` reflections under the physical
|
|
// model <E^2> = A (1 + f cos(2 pi h.u)): the correlation between E^2 and cos(2 pi h.u).
|
|
//
|
|
// The gate above scores a vector by the max/min ratio of the phase-bin means instead. That is the
|
|
// right statistic for the gate - it is what the null is drawn against - but it is the wrong one to
|
|
// REFINE on a full-resolution population, because a ratio is maximised by emptying one bin: on a
|
|
// 118,000-reflection merge the max/min search walked to a vector 0.015 away from the correlation's
|
|
// answer while reporting 1.2e4. The fitted amplitude f itself is no better, for the mirror reason:
|
|
// f is a slope divided by an intercept, so it runs away wherever the cosine has little variance
|
|
// over the data, and the search drifts to exactly such a vector (measured: 0.06 off a synthetic
|
|
// half-integer translation). The correlation has neither failure - it is 1 only where the cosine
|
|
// follows the intensities.
|
|
double CosineCorrelation(const std::vector<std::array<int, 3>> &hkl, const std::vector<double> &e2,
|
|
const std::array<double, 3> &u, size_t n) {
|
|
double sc = 0.0, scc = 0.0, sy = 0.0, syy = 0.0, scy = 0.0;
|
|
for (size_t i = 0; i < n; ++i) {
|
|
const double c = std::cos(2.0 * std::numbers::pi
|
|
* (hkl[i][0] * u[0] + hkl[i][1] * u[1] + hkl[i][2] * u[2]));
|
|
sc += c;
|
|
scc += c * c;
|
|
sy += e2[i];
|
|
syy += e2[i] * e2[i];
|
|
scy += c * e2[i];
|
|
}
|
|
const double nn = static_cast<double>(n);
|
|
const double vc = scc - sc * sc / nn, vy = syy - sy * sy / nn;
|
|
if (!(vc > 0.0) || !(vy > 0.0))
|
|
return 0.0;
|
|
return (scy - sc * sy / nn) / std::sqrt(vc * vy);
|
|
}
|
|
}
|
|
|
|
TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector<MergedReflection> &merged,
|
|
const gemmi::UnitCell &cell,
|
|
const gemmi::SpaceGroup *space_group,
|
|
int null_draws, uint32_t seed) {
|
|
TranslationalNCSResult result;
|
|
if (cell.a <= 0.0 || cell.b <= 0.0 || cell.c <= 0.0) {
|
|
result.refusal = "no unit cell";
|
|
return result;
|
|
}
|
|
|
|
const gemmi::GroupOps gops = space_group ? space_group->operations() : gemmi::GroupOps{};
|
|
auto acentric = [&](const MergedReflection &r) {
|
|
return !space_group || !gops.is_reflection_centric(gemmi::Op::Miller{{r.h, r.k, r.l}});
|
|
};
|
|
|
|
// ------------------------------------------------------------------ (a) the Patterson peak
|
|
// Following the native-Patterson construction of Read, Adams & McCoy (2013) Acta Cryst. D69,
|
|
// 176-183, with the fixed peak-height table replaced by a per-dataset permutation null: the
|
|
// noise floor of this statistic runs from ~1% of the origin on a 74,000-reflection merge to ~18%
|
|
// on a 138-reflection one, so no fixed percentage can serve both.
|
|
std::vector<int> idx;
|
|
for (size_t i = 0; i < merged.size(); ++i) {
|
|
const auto &r = merged[i];
|
|
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kPattersonDMin && r.d <= kPattersonDMax)
|
|
idx.push_back(static_cast<int>(i));
|
|
}
|
|
result.n_reflections = static_cast<int>(idx.size());
|
|
if (result.n_reflections < kMinPattersonReflections) {
|
|
std::ostringstream os;
|
|
os << "only " << result.n_reflections << " reflections between " << kPattersonDMax << " and "
|
|
<< kPattersonDMin << " A - too few for a Patterson";
|
|
result.refusal = os.str();
|
|
return result;
|
|
}
|
|
|
|
// E^2: divide each intensity by the mean of an equal-count resolution shell, so the map is not
|
|
// dominated by the Wilson fall-off and the peak percentage is comparable between datasets.
|
|
std::sort(idx.begin(), idx.end(),
|
|
[&](int a, int b) { return merged[a].d != merged[b].d ? merged[a].d > merged[b].d : a < b; });
|
|
const int n = result.n_reflections;
|
|
std::vector<double> e2(n, 0.0);
|
|
std::vector<int> shell_of(n, 0);
|
|
for (int s = 0; s < kPattersonShells; ++s) {
|
|
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
|
|
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
|
|
if (hi <= lo)
|
|
continue;
|
|
double sum = 0.0;
|
|
for (int i = lo; i < hi; ++i)
|
|
sum += merged[idx[i]].I;
|
|
const double mean = sum / (hi - lo);
|
|
for (int i = lo; i < hi; ++i) {
|
|
e2[i] = mean != 0.0 ? merged[idx[i]].I / mean : 0.0;
|
|
shell_of[i] = s;
|
|
}
|
|
}
|
|
|
|
// Grid. Sampled at d_min / 2.5, rounded up to an FFT-friendly size. A very large cell is measured
|
|
// on a coarser map rather than refused: the map only has to resolve a peak 15 A from the origin.
|
|
double patterson_dmin = kPattersonDMin;
|
|
std::array<int, 3> g{};
|
|
const double axis[3] = {cell.a, cell.b, cell.c};
|
|
for (;;) {
|
|
for (int i = 0; i < 3; ++i)
|
|
g[i] = SmoothSize(static_cast<int>(std::ceil(axis[i] * kGridSampling / patterson_dmin)));
|
|
const size_t points = static_cast<size_t>(g[0]) * g[1] * g[2];
|
|
if (points <= (64u << 20) || patterson_dmin >= 12.0)
|
|
break;
|
|
patterson_dmin *= 1.25;
|
|
}
|
|
result.grid = g;
|
|
const int nx = g[0], ny = g[1], nz = g[2];
|
|
const int nzh = nz / 2 + 1;
|
|
const size_t n_points = static_cast<size_t>(nx) * ny * nz;
|
|
const size_t n_complex = static_cast<size_t>(nx) * ny * nzh;
|
|
if (n_points > kLargeGridPoints)
|
|
null_draws = std::min(null_draws, 10);
|
|
|
|
// Expand the asymmetric unit onto the full sphere: every rotation of the space group and both
|
|
// Friedel mates, deduplicated. Only the half-space l' <= nz/2 is stored - the Patterson
|
|
// coefficients are real and centrosymmetric, so the other half is what the c2r transform infers.
|
|
std::vector<std::pair<size_t, int>> fill; // (index into the half-space grid, source reflection)
|
|
fill.reserve(idx.size() * 8);
|
|
{
|
|
std::unordered_set<int64_t> seen;
|
|
seen.reserve(idx.size() * 16);
|
|
std::vector<gemmi::Op> ops;
|
|
if (space_group)
|
|
ops = gops.sym_ops;
|
|
else
|
|
ops.push_back(gemmi::Op::identity());
|
|
for (const auto &op : ops)
|
|
for (int sign : {1, -1})
|
|
for (int i = 0; i < n; ++i) {
|
|
const auto &r = merged[idx[i]];
|
|
const gemmi::Op::Miller m = op.apply_to_hkl({{r.h, r.k, r.l}});
|
|
const int64_t h = sign * m[0], k = sign * m[1], l = sign * m[2];
|
|
const int64_t key = ((h + 1048576) << 42) | ((k + 1048576) << 21) | (l + 1048576);
|
|
if (!seen.insert(key).second)
|
|
continue;
|
|
const int gl = static_cast<int>(((l % nz) + nz) % nz);
|
|
if (gl > nz / 2)
|
|
continue; // its Friedel mate is stored instead
|
|
const int gh = static_cast<int>(((h % nx) + nx) % nx);
|
|
const int gk = static_cast<int>(((k % ny) + ny) % ny);
|
|
fill.emplace_back((static_cast<size_t>(gh) * ny + gk) * nzh + gl, i);
|
|
}
|
|
}
|
|
|
|
float *coeff = fftwf_alloc_real(2 * n_complex);
|
|
float *map = fftwf_alloc_real(n_points);
|
|
if (coeff == nullptr || map == nullptr) {
|
|
if (coeff) fftwf_free(coeff);
|
|
if (map) fftwf_free(map);
|
|
result.refusal = "could not allocate the Patterson grid";
|
|
return result;
|
|
}
|
|
fftwf_plan plan;
|
|
{
|
|
std::lock_guard lock(FFTWPlannerMutex());
|
|
plan = fftwf_plan_dft_c2r_3d(nx, ny, nz, reinterpret_cast<fftwf_complex *>(coeff), map,
|
|
FFTW_ESTIMATE);
|
|
}
|
|
|
|
// The near-origin exclusion, as a set of grid points rather than a per-point distance test: only
|
|
// the points inside a 15 A ball around each origin-equivalent point can be excluded, and there
|
|
// are a few thousand of those however large the grid is.
|
|
std::vector<uint8_t> near(n_points, 0);
|
|
std::vector<Vec3> equivalents = OriginEquivalents(space_group);
|
|
auto build_near_mask = [&]() {
|
|
std::fill(near.begin(), near.end(), 0);
|
|
const gemmi::UnitCell &c = cell;
|
|
// Fractional half-width of a sphere of radius R along axis i is R * |a*_i|, so the ball is
|
|
// enumerated directly instead of testing every grid point: the cost is a few thousand points
|
|
// however large the grid is.
|
|
const double rstar[3] = {c.ar, c.br, c.cr};
|
|
for (const auto &e : equivalents) {
|
|
const int span[3] = {static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[0] * nx)) + 1,
|
|
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[1] * ny)) + 1,
|
|
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[2] * nz)) + 1};
|
|
const int c0[3] = {static_cast<int>(std::lround(e.x * nx)),
|
|
static_cast<int>(std::lround(e.y * ny)),
|
|
static_cast<int>(std::lround(e.z * nz))};
|
|
for (int i = c0[0] - span[0]; i <= c0[0] + span[0]; ++i)
|
|
for (int j = c0[1] - span[1]; j <= c0[1] + span[1]; ++j)
|
|
for (int k = c0[2] - span[2]; k <= c0[2] + span[2]; ++k) {
|
|
const gemmi::Position p = cell.orthogonalize(gemmi::Fractional{
|
|
static_cast<double>(i) / nx - e.x, static_cast<double>(j) / ny - e.y,
|
|
static_cast<double>(k) / nz - e.z});
|
|
if (p.length() >= kMinPeakDistanceA)
|
|
continue;
|
|
const size_t gi = static_cast<size_t>(((i % nx) + nx) % nx);
|
|
const size_t gj = static_cast<size_t>(((j % ny) + ny) % ny);
|
|
const size_t gk = static_cast<size_t>(((k % nz) + nz) % nz);
|
|
near[(gi * ny + gj) * nz + gk] = 1;
|
|
}
|
|
}
|
|
};
|
|
build_near_mask();
|
|
|
|
auto transform = [&](const std::vector<double> &values) {
|
|
std::memset(coeff, 0, 2 * n_complex * sizeof(float));
|
|
for (const auto &[at, src] : fill)
|
|
coeff[2 * at] += static_cast<float>(values[src]);
|
|
fftwf_execute(plan);
|
|
};
|
|
auto max_far = [&]() {
|
|
double best = -std::numeric_limits<double>::infinity();
|
|
for (size_t i = 0; i < n_points; ++i)
|
|
if (!near[i] && map[i] > best)
|
|
best = map[i];
|
|
return best;
|
|
};
|
|
|
|
transform(e2);
|
|
const double origin = map[0];
|
|
|
|
// The vector to report is the largest off-origin LOCAL maximum, which is not always the largest
|
|
// off-origin grid value: the tail of the origin peak can be higher right at the 15 A boundary,
|
|
// and that is a shoulder, not a peak.
|
|
auto top_local_max = [&]() {
|
|
double best = -std::numeric_limits<double>::infinity();
|
|
std::array<double, 3> at_frac{{0.0, 0.0, 0.0}};
|
|
for (int i = 0; i < nx; ++i)
|
|
for (int j = 0; j < ny; ++j)
|
|
for (int k = 0; k < nz; ++k) {
|
|
const size_t at = (static_cast<size_t>(i) * ny + j) * nz + k;
|
|
if (near[at] || map[at] <= best)
|
|
continue;
|
|
bool is_max = true;
|
|
for (int dx = -1; dx <= 1 && is_max; ++dx)
|
|
for (int dy = -1; dy <= 1 && is_max; ++dy)
|
|
for (int dz = -1; dz <= 1; ++dz) {
|
|
if (dx == 0 && dy == 0 && dz == 0)
|
|
continue;
|
|
const size_t nb = (static_cast<size_t>((i + dx + nx) % nx) * ny
|
|
+ (j + dy + ny) % ny) * nz + (k + dz + nz) % nz;
|
|
if (map[nb] > map[at]) { is_max = false; break; }
|
|
}
|
|
if (is_max) {
|
|
best = map[at];
|
|
at_frac = {static_cast<double>(i) / nx, static_cast<double>(j) / ny,
|
|
static_cast<double>(k) / nz};
|
|
}
|
|
}
|
|
return std::pair<std::array<double, 3>, double>{at_frac, best};
|
|
};
|
|
|
|
// Absorb any translation the data are invariant under - a centring the space group in use does
|
|
// not declare - into the exclusion, and look again underneath it. Without this, a C-centred
|
|
// crystal merged in P1 reports its own centring vector as a pseudo-symmetry (measured: 11 of the
|
|
// 95 P1 merges in the corpus), and a genuine pseudo-translation hiding below a centring vector is
|
|
// never reached.
|
|
double observed = 0.0;
|
|
for (int pass = 0; pass < 5; ++pass) {
|
|
observed = max_far();
|
|
result.peak_percent = origin > 0.0 ? 100.0 * observed / origin : 0.0;
|
|
const auto [frac, height] = top_local_max();
|
|
result.peak_frac = frac;
|
|
if (result.peak_percent < kLatticeTranslationPct || !std::isfinite(height))
|
|
break;
|
|
result.undeclared_lattice_translations.push_back(frac);
|
|
equivalents.push_back(Vec3{frac[0], frac[1], frac[2]});
|
|
build_near_mask();
|
|
}
|
|
|
|
// Permutation null: the same reflections, the same map, with the intensities shuffled inside each
|
|
// resolution shell. Everything that could make a Patterson peak except the correlation between
|
|
// reflections is preserved, so this is the noise floor of THIS dataset.
|
|
{
|
|
std::mt19937 rng(seed);
|
|
std::vector<double> permuted = e2;
|
|
std::vector<double> nulls;
|
|
nulls.reserve(null_draws);
|
|
for (int draw = 0; draw < null_draws; ++draw) {
|
|
permuted = e2;
|
|
for (int s = 0; s < kPattersonShells; ++s) {
|
|
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
|
|
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
|
|
for (int i = hi - 1; i > lo; --i)
|
|
std::swap(permuted[i], permuted[lo + static_cast<int>(rng() % (i - lo + 1))]);
|
|
}
|
|
transform(permuted);
|
|
if (map[0] > 0.0)
|
|
nulls.push_back(100.0 * max_far() / map[0]);
|
|
}
|
|
if (!nulls.empty()) {
|
|
const double mean = std::accumulate(nulls.begin(), nulls.end(), 0.0) / nulls.size();
|
|
double var = 0.0;
|
|
for (double v : nulls)
|
|
var += (v - mean) * (v - mean);
|
|
var /= nulls.size();
|
|
result.null_mean = mean;
|
|
result.null_sd = std::sqrt(var);
|
|
result.peak_z = result.null_sd > 0.0 ? (result.peak_percent - mean) / result.null_sd : 0.0;
|
|
}
|
|
}
|
|
|
|
{
|
|
std::lock_guard lock(FFTWPlannerMutex());
|
|
fftwf_destroy_plan(plan);
|
|
}
|
|
fftwf_free(coeff);
|
|
fftwf_free(map);
|
|
result.measurable = true;
|
|
|
|
// ------------------------------------------------------------------ (b) the modulation depth
|
|
// A pure translation u between two copies multiplies the intensity by |1 + exp(2 pi i h.u)|^2, so
|
|
// <I> depends on the phase h.u: strong where it is near an integer, extinct near a half-integer.
|
|
// The Patterson peak says a vector exists; this says whether it modulates the data.
|
|
std::vector<std::array<int, 3>> mod_hkl;
|
|
std::vector<double> mod_e2;
|
|
// Where each kept resolution shell ends in the two arrays above. The null below shuffles the
|
|
// intensities inside a shell, exactly as the Patterson null does, and the shells are written
|
|
// consecutively, so their boundaries are all it needs.
|
|
std::vector<size_t> mod_shell_bounds{0};
|
|
{
|
|
std::vector<int> mod_idx;
|
|
for (size_t i = 0; i < merged.size(); ++i) {
|
|
const auto &r = merged[i];
|
|
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kModulationDMin && acentric(r))
|
|
mod_idx.push_back(static_cast<int>(i));
|
|
}
|
|
if (!mod_idx.empty()) {
|
|
// Shells linear in 1/d^2, and the same defensive normalisation the twinning statistics
|
|
// use: noise-only shells skipped, Wilson outliers rejected, the shell mean re-fitted once
|
|
// so the outlier does not corrupt the mean it is measured against.
|
|
double min_s = std::numeric_limits<double>::infinity(), max_s = -min_s;
|
|
for (int i : mod_idx) {
|
|
const double s = 1.0 / (merged[i].d * merged[i].d);
|
|
min_s = std::min(min_s, s);
|
|
max_s = std::max(max_s, s);
|
|
}
|
|
if (max_s > min_s) {
|
|
std::vector<std::vector<int>> bins(kModulationShells);
|
|
for (int i : mod_idx) {
|
|
const double t = (1.0 / (merged[i].d * merged[i].d) - min_s) / (max_s - min_s);
|
|
bins[std::clamp(static_cast<int>(t * kModulationShells), 0, kModulationShells - 1)]
|
|
.push_back(i);
|
|
}
|
|
for (const auto &b : bins) {
|
|
if (b.empty())
|
|
continue;
|
|
double isig = 0.0;
|
|
int n_isig = 0;
|
|
for (int i : b)
|
|
if (std::isfinite(merged[i].sigma) && merged[i].sigma > 0.0) {
|
|
isig += merged[i].I / merged[i].sigma;
|
|
++n_isig;
|
|
}
|
|
if (n_isig == 0 || isig / n_isig < kMinShellISig)
|
|
continue;
|
|
double sum = 0.0;
|
|
for (int i : b)
|
|
sum += merged[i].I;
|
|
double mean = sum / b.size();
|
|
if (mean <= 0.0)
|
|
continue;
|
|
sum = 0.0;
|
|
int kept = 0;
|
|
for (int i : b)
|
|
if (merged[i].I / mean <= kWilsonOutlierE2) { sum += merged[i].I; ++kept; }
|
|
if (kept == 0)
|
|
continue;
|
|
mean = sum / kept;
|
|
if (mean <= 0.0)
|
|
continue;
|
|
for (int i : b) {
|
|
const double v = merged[i].I / mean;
|
|
if (v > kWilsonOutlierE2)
|
|
continue;
|
|
mod_hkl.push_back({merged[i].h, merged[i].k, merged[i].l});
|
|
mod_e2.push_back(v);
|
|
}
|
|
mod_shell_bounds.push_back(mod_e2.size());
|
|
}
|
|
}
|
|
}
|
|
}
|
|
result.n_modulation_reflections = static_cast<int>(mod_e2.size());
|
|
if (result.n_modulation_reflections < kMinModulationReflections)
|
|
return result;
|
|
|
|
const int floor_count = std::max(50, result.n_modulation_reflections / (4 * kPhaseBins));
|
|
auto contrast = [&](const std::vector<double> &e2v, const std::array<double, 3> &u,
|
|
std::vector<double> *profile) {
|
|
std::array<double, kPhaseBins> sum{}, cnt{};
|
|
sum.fill(0.0);
|
|
cnt.fill(0.0);
|
|
for (size_t i = 0; i < e2v.size(); ++i) {
|
|
const double t = mod_hkl[i][0] * u[0] + mod_hkl[i][1] * u[1] + mod_hkl[i][2] * u[2];
|
|
const double f = t - std::floor(t);
|
|
const int b = std::min(static_cast<int>(f * kPhaseBins), kPhaseBins - 1);
|
|
sum[b] += e2v[i];
|
|
cnt[b] += 1.0;
|
|
}
|
|
double lo = std::numeric_limits<double>::infinity(), hi = -lo;
|
|
int populated = 0;
|
|
if (profile)
|
|
profile->assign(kPhaseBins, std::numeric_limits<double>::quiet_NaN());
|
|
for (int b = 0; b < kPhaseBins; ++b) {
|
|
if (cnt[b] < floor_count)
|
|
continue;
|
|
const double m = sum[b] / cnt[b];
|
|
if (profile)
|
|
(*profile)[b] = m;
|
|
lo = std::min(lo, m);
|
|
hi = std::max(hi, m);
|
|
++populated;
|
|
}
|
|
if (populated < 2 || lo <= 0.0)
|
|
return std::numeric_limits<double>::quiet_NaN();
|
|
return hi / lo;
|
|
};
|
|
|
|
// Coarse-to-fine local search for the vector that modulates the data most strongly, started from
|
|
// the Patterson peak. The Patterson grid is coarse and a real pseudo-translation is rarely at an
|
|
// exactly rational position, so the peak position alone understates the modulation.
|
|
const std::array<double, 3> step0{1.0 / nx, 1.0 / ny, 1.0 / nz};
|
|
auto refine = [&](const std::vector<double> &e2v, std::array<double, 3> u) {
|
|
std::array<double, 3> s = step0;
|
|
double best = contrast(e2v, u, nullptr);
|
|
for (int round = 0; round < 3; ++round) {
|
|
for (bool improved = true; improved;) {
|
|
improved = false;
|
|
for (int i = 0; i < 3; ++i)
|
|
for (double gstep : {-1.0, -0.5, 0.0, 0.5, 1.0}) {
|
|
std::array<double, 3> v = u;
|
|
v[i] += gstep * s[i];
|
|
const double c = contrast(e2v, v, nullptr);
|
|
if (std::isfinite(c) && c > best + 1e-6) {
|
|
best = c;
|
|
u = v;
|
|
improved = true;
|
|
}
|
|
}
|
|
}
|
|
for (double &x : s)
|
|
x /= 2.5;
|
|
}
|
|
return std::pair<std::array<double, 3>, double>{u, best};
|
|
};
|
|
|
|
const auto [u_refined, best_contrast] = refine(mod_e2, result.peak_frac);
|
|
if (!std::isfinite(best_contrast))
|
|
return result;
|
|
for (int i = 0; i < 3; ++i)
|
|
result.vector_frac[i] = u_refined[i] - std::floor(u_refined[i]);
|
|
result.modulation_measured = true;
|
|
result.modulation = best_contrast;
|
|
contrast(mod_e2, u_refined, &result.phase_bin_mean_e2);
|
|
result.vector_length_A = DistanceToOrigin(cell, Vec3{result.vector_frac[0], result.vector_frac[1],
|
|
result.vector_frac[2]},
|
|
std::vector<Vec3>{{0.0, 0.0, 0.0}});
|
|
|
|
// The modulation's own null: the SAME search, from the SAME start, on the same reflections with
|
|
// the intensities shuffled inside each resolution shell - the permutation the Patterson null
|
|
// above already uses, applied to the second half of the evidence. A greedy search over three
|
|
// free parameters can manufacture contrast out of nothing, and this measures how much rather
|
|
// than assuming it is negligible; the shuffle leaves the reflection count, the E^2 distribution
|
|
// and the phase-bin populations exactly as they were and removes only the one thing the
|
|
// modulation claims to see, the correlation between a reflection's intensity and h.u.
|
|
//
|
|
// Drawing the null from the unshuffled data instead - random starting vectors, the crystal's own
|
|
// intensities - measured the search AND whatever modulation the data carry, because a greedy
|
|
// search started anywhere walks into a real modulation's basin. Measured over 110 merged
|
|
// datasets: single draws reached 199x and 6162x where the crystal's own modulation read 4.0x and
|
|
// 67x, and 28 of the 110 had the control at or above the signal it was controlling for. The
|
|
// median was robust enough that this changes no verdict anywhere in that corpus - what it
|
|
// removes is a control that reads the same signal it is subtracted from, and with it the one
|
|
// way an upstream change can move the control while the modulation stands still.
|
|
{
|
|
std::mt19937 rng(seed + 7);
|
|
std::vector<double> permuted = mod_e2;
|
|
std::vector<double> nulls;
|
|
nulls.reserve(kModulationNullDraws);
|
|
for (int draw = 0; draw < kModulationNullDraws; ++draw) {
|
|
permuted = mod_e2;
|
|
for (size_t s = 1; s < mod_shell_bounds.size(); ++s) {
|
|
const size_t lo = mod_shell_bounds[s - 1], hi = mod_shell_bounds[s];
|
|
for (size_t i = hi; i > lo + 1; --i)
|
|
std::swap(permuted[i - 1], permuted[lo + rng() % (i - lo)]);
|
|
}
|
|
const auto [v, c] = refine(permuted, result.peak_frac);
|
|
(void) v;
|
|
nulls.push_back(std::isfinite(c) ? c : 1.0);
|
|
}
|
|
std::sort(nulls.begin(), nulls.end());
|
|
const size_t m = nulls.size();
|
|
result.modulation_null = m % 2 ? nulls[m / 2] : 0.5 * (nulls[m / 2 - 1] + nulls[m / 2]);
|
|
}
|
|
|
|
result.detected = result.peak_z >= kPeakZThreshold && result.modulation_null > 0.0 &&
|
|
result.modulation / result.modulation_null >= kModulationRatioThreshold;
|
|
|
|
result.near_extinct_class = result.detected && result.modulation >= kNearExtinctModulation;
|
|
|
|
// ------------------------------------------------ the vector the phase CLASSES are assigned with
|
|
//
|
|
// Everything above is measured on d >= 4 A, where |h| stays under about 15 and an error of 0.02 in
|
|
// u costs the contrast almost nothing - the gate's vector is refined only to the accuracy the gate
|
|
// needs. But the consumers of this vector (the L-test's partner steps, and the twin-immune zone's
|
|
// per-class normalisation) assign a CLASS, cos(2 pi h.u), to reflections at FULL resolution, and
|
|
// the phase error there is |h| times the error in u. At |h| = 20 an error of 0.02 is a third of a
|
|
// turn and the classes are no longer the physical ones. Measured as the correlation between E^2
|
|
// and cos(2 pi h.u) in bands of |h| on two of this corpus's pseudo-translations, the gate's
|
|
// vector against this one: 0.57/0.61 at |h| < 8, 0.42/0.52, 0.19/0.30, 0.14/0.18 past |h| = 22 on
|
|
// the first; 0.14/0.52, -0.15/0.42, 0.00/0.20, -0.02/0.09 on the second, whose classes the gate's
|
|
// vector gets ANTI-correlated with the real ones over a whole band.
|
|
//
|
|
// So u is re-refined on the whole data, up a ladder: each step reads twice as many reflections as
|
|
// the one before, in order of |h|, and starts from the previous step's answer with a search step
|
|
// of a quarter of the phase ambiguity 1/|h| of the highest order it reads. The vector is never
|
|
// asked to extrapolate further than it is known, which is what lets a greedy search cross a
|
|
// resolution range whose objective has a local maximum every 1/|h| in each direction. Refining on
|
|
// the whole data in one go instead walks into the nearest of those.
|
|
//
|
|
// Nothing above is re-read: `modulation`, `modulation_null` and `detected` stay the gate's, so
|
|
// which crystals are called stays exactly as it was. Only where the vector points changes.
|
|
if (result.detected) {
|
|
std::vector<std::array<int, 3>> cls_hkl;
|
|
std::vector<double> cls_e2;
|
|
{
|
|
std::vector<int> idx;
|
|
for (size_t i = 0; i < merged.size(); ++i)
|
|
if (std::isfinite(merged[i].I) && std::isfinite(merged[i].d) && merged[i].d > 0.0
|
|
&& acentric(merged[i]))
|
|
idx.push_back(static_cast<int>(i));
|
|
// Equal-count shells sorted by |h|, so that every stage of the ladder reads a population
|
|
// that is already normalised and can be taken as a prefix.
|
|
std::sort(idx.begin(), idx.end(), [&](int a, int b) {
|
|
return std::max({std::abs(merged[a].h), std::abs(merged[a].k), std::abs(merged[a].l)})
|
|
< std::max({std::abs(merged[b].h), std::abs(merged[b].k), std::abs(merged[b].l)});
|
|
});
|
|
std::vector<int> by_d = idx;
|
|
std::sort(by_d.begin(), by_d.end(), [&](int a, int b) { return merged[a].d > merged[b].d; });
|
|
std::vector<double> e2_of(merged.size(), std::numeric_limits<double>::quiet_NaN());
|
|
const size_t n_shells = std::max<size_t>(1, by_d.size() / 500);
|
|
for (size_t s = 0; s < n_shells; ++s) {
|
|
const size_t lo = s * by_d.size() / n_shells, hi = (s + 1) * by_d.size() / n_shells;
|
|
double sum = 0.0;
|
|
for (size_t i = lo; i < hi; ++i)
|
|
sum += merged[by_d[i]].I;
|
|
const double mean = sum / static_cast<double>(hi - lo);
|
|
if (mean <= 0.0)
|
|
continue;
|
|
for (size_t i = lo; i < hi; ++i)
|
|
e2_of[by_d[i]] = merged[by_d[i]].I / mean;
|
|
}
|
|
for (int i : idx)
|
|
if (std::isfinite(e2_of[i]) && e2_of[i] <= kWilsonOutlierE2) {
|
|
cls_hkl.push_back({merged[i].h, merged[i].k, merged[i].l});
|
|
cls_e2.push_back(e2_of[i]);
|
|
}
|
|
}
|
|
// The steps of the ladder double the number of reflections read rather than the |h| ceiling,
|
|
// so that the first step always has a population to refine on whatever the cell is and no
|
|
// step is skipped for being empty.
|
|
std::array<double, 3> u = u_refined;
|
|
for (size_t n = kMinModulationReflections; !cls_hkl.empty(); n *= 2) {
|
|
n = std::min(n, cls_hkl.size());
|
|
const int h_top = std::max({std::abs(cls_hkl[n - 1][0]), std::abs(cls_hkl[n - 1][1]),
|
|
std::abs(cls_hkl[n - 1][2])});
|
|
// A quarter of the phase ambiguity 1/|h| of the highest order this step reads: large
|
|
// enough to walk to the maximum, small enough that the first probe cannot land in the
|
|
// neighbouring one.
|
|
double step = h_top > 0 ? 0.25 / h_top : std::max({step0[0], step0[1], step0[2]});
|
|
double best = CosineCorrelation(cls_hkl, cls_e2, u, n);
|
|
for (int round = 0; round < 8; ++round) {
|
|
for (bool improved = true; improved;) {
|
|
improved = false;
|
|
// All 26 neighbours, not the 6 along the axes: the phase is h.u, so a vector
|
|
// whose class comes mostly from one combination of the axes - h/3 + 2k/3, say -
|
|
// sits in a valley that runs diagonally, and a search that only moves one
|
|
// component at a time stalls on its wall (measured: 0.005 short in two
|
|
// components at once, a quarter of a turn by |h| = 60).
|
|
for (int dx = -1; dx <= 1; ++dx)
|
|
for (int dy = -1; dy <= 1; ++dy)
|
|
for (int dz = -1; dz <= 1; ++dz) {
|
|
if (dx == 0 && dy == 0 && dz == 0)
|
|
continue;
|
|
const std::array<double, 3> v{u[0] + dx * step, u[1] + dy * step,
|
|
u[2] + dz * step};
|
|
const double f = CosineCorrelation(cls_hkl, cls_e2, v, n);
|
|
if (f > best + 1e-9) {
|
|
best = f;
|
|
u = v;
|
|
improved = true;
|
|
}
|
|
}
|
|
}
|
|
step *= 0.5;
|
|
}
|
|
if (n == cls_hkl.size())
|
|
break;
|
|
}
|
|
// A greedy climb up the ladder can still end on an alias: a stage whose population is small
|
|
// and whose objective is flat over the step it searches with can hand the next stage a start
|
|
// one phase turn out, and from there the finer steps cannot come back. The two candidates are
|
|
// carried to the end and the whole data decide between them, which is the one place where
|
|
// both are directly comparable.
|
|
if (!cls_hkl.empty()
|
|
&& CosineCorrelation(cls_hkl, cls_e2, u_refined, cls_hkl.size())
|
|
> CosineCorrelation(cls_hkl, cls_e2, u, cls_hkl.size()))
|
|
u = u_refined;
|
|
for (int i = 0; i < 3; ++i)
|
|
result.vector_frac[i] = u[i] - std::floor(u[i]);
|
|
contrast(mod_e2, u, &result.phase_bin_mean_e2);
|
|
result.vector_length_A = DistanceToOrigin(cell, Vec3{result.vector_frac[0], result.vector_frac[1],
|
|
result.vector_frac[2]},
|
|
std::vector<Vec3>{{0.0, 0.0, 0.0}});
|
|
}
|
|
|
|
// Is the vector a rational translation of the reported cell? If it is, the cell contains a
|
|
// sub-lattice: the pseudo-translation is a candidate for a REAL lattice translation the indexing
|
|
// missed, and the reported cell is a supercell candidate rather than a settled result.
|
|
for (int q = 2; q <= 6 && !result.commensurate; ++q) {
|
|
std::array<int, 3> p{};
|
|
double err = 0.0;
|
|
bool fractional = false;
|
|
for (int i = 0; i < 3; ++i) {
|
|
p[i] = static_cast<int>(std::lround(result.vector_frac[i] * q));
|
|
err = std::max(err, std::fabs(result.vector_frac[i] * q - p[i]));
|
|
if (p[i] % q != 0)
|
|
fractional = true;
|
|
}
|
|
if (err / q <= 0.05 && fractional) {
|
|
result.commensurate = true;
|
|
result.commensurate_denominator = q;
|
|
result.commensurate_numerator = p;
|
|
}
|
|
}
|
|
return result;
|
|
}
|
|
|
|
std::string TranslationalNCSToText(const TranslationalNCSResult &result) {
|
|
std::ostringstream os;
|
|
os << "Translational pseudo-symmetry\n";
|
|
if (!result.measurable) {
|
|
os << " Not measured: " << result.refusal << ".\n"
|
|
<< " => This is not a statement that the crystal has none.\n";
|
|
return os.str();
|
|
}
|
|
os << std::fixed;
|
|
for (const auto &t : result.undeclared_lattice_translations)
|
|
os << " The data are invariant under (" << std::setprecision(3) << t[0] << ", " << t[1]
|
|
<< ", " << t[2] << ") - the Patterson is as high there as at the origin, so that is a\n"
|
|
<< " LATTICE translation, not a pseudo-symmetry, and the space group in use does not\n"
|
|
<< " declare it. Either the lattice is centred and the merge is not, or the cell is a\n"
|
|
<< " supercell. Excluded, and the search continued past it.\n";
|
|
os << " Native Patterson off-origin peak " << std::setprecision(1) << result.peak_percent
|
|
<< "% of the origin";
|
|
if (result.vector_length_A > 0.0)
|
|
os << " at (" << std::setprecision(3) << result.vector_frac[0] << ", " << result.vector_frac[1]
|
|
<< ", " << result.vector_frac[2] << "), " << std::setprecision(1) << result.vector_length_A
|
|
<< " A";
|
|
os << "\n [z = " << std::setprecision(1) << result.peak_z << " against a permutation null of "
|
|
<< result.null_mean << " +- " << std::setprecision(2) << result.null_sd << "%]\n";
|
|
if (result.modulation_measured)
|
|
os << " Intensity modulation along that vector: " << std::setprecision(1) << result.modulation
|
|
<< "x between the strongest and weakest phase class [random-vector control "
|
|
<< result.modulation_null << "x]\n";
|
|
if (!result.modulation_measured) {
|
|
os << " The modulation test could not be run - too few acentric reflections beyond 4 A - so\n"
|
|
<< " the peak above stands alone.\n"
|
|
<< " => Inconclusive. A Patterson peak on its own is not enough to call a pseudo-symmetry\n"
|
|
<< " (peaks of this size occur without one), and this dataset cannot supply the second\n"
|
|
<< " half of the evidence.\n";
|
|
return os.str();
|
|
}
|
|
if (!result.detected) {
|
|
os << " => No translational pseudo-symmetry indicated.\n";
|
|
return os.str();
|
|
}
|
|
os << " => Translational pseudo-symmetry. Molecular replacement will need it declared - Phaser\n"
|
|
<< " reads a tNCS vector and corrects the likelihood for it; without that the search can\n"
|
|
<< " fail on data that are otherwise good.\n";
|
|
if (result.near_extinct_class)
|
|
os << " => The class this vector suppresses is not weak but very nearly EXTINCT, which is\n"
|
|
<< " what a lattice translation of a smaller cell looks like measured in a larger one.\n"
|
|
<< " The reported cell is therefore a supercell candidate rather than a settled\n"
|
|
<< " result, and the rival hypothesis is a cell with this translation as a lattice\n"
|
|
<< " vector.\n";
|
|
else if (result.commensurate)
|
|
os << " => The vector is a 1/" << result.commensurate_denominator
|
|
<< " translation of the cell, so the crystal is pseudo-centred. The cell itself is not in\n"
|
|
<< " question - the suppressed class is weak, not absent, and a smaller cell would\n"
|
|
<< " contradict it.\n";
|
|
return os.str();
|
|
}
|