leonarski_fandClaude Opus 5.5
8a8c992cca
Report-layer fixes from the warnings audit of the rc173 full battery. Processing is unchanged (merged MTZ/HKL bit-identical on the sets checked). - SUPERCELL_POSSIBLE stays a key and a summary line; a warning only where the rocking part reaches 20% (sub-cells that refine normally rocked at 2.1-16.2%, the accepted doubled cells at 4.2% and 29.1%). The summary says where further lattice domains may put spots on the half-integer nodes. - LATTICE_TRANSLATION only where the Patterson at the vector reaches 90% of the origin (UNDECLARED_LATTICE_TRANSLATION_PCT, new); the two admitted vectors on the battery read 79% and 85% on crystals refining in the deposited cell, and are now reported as a very strong pseudo-translation. PSEUDO_TRANSLATION warns from a 20% peak (xtriage convention; detected peaks were 8.1% on one set, 24.8-63.2% on the rest). - <|L|> outside 0.375-0.55 (on the adopted or the pre-search merge) is TWINNING_VERDICT= NOT_READABLE with no warning, instead of a twin or SYMMETRY_SUSPECT reading. - NO_LATTICE no longer fires on a rotation run that indexed the sweep with no frame indexed on its own; INDEXING_RATE is --developer on rotation. - A rotation run that finds no lattice writes a report (VERDICT= FAILED, NO_LATTICE) before exiting 1, instead of an input-parameter error and no report (NoLatticeFound). - Powder and Ice summary lines, ICE_* keys, and a POWDER_RINGS flag from a 0.5 spot fraction. - SWEEP_GAPS warns only where the degraded ranges cover 10% of the sweep (77 -> 48 sets). - A single rotation sweep no longer carries an INDEXING_AMBIGUITY warning (one orientation matrix, consistent hand); INDEXING_AMBIGUITY_OPERATORS and an Indexing choice line instead. - A DOMAIN more than 10 deg away is a second crystal; the MULTIPLE_LATTICES sum is documented. - Mosaicity line and docs name XDS's suggested REFLECTING_RANGE_E.S.D. as the comparable number (0.90x median over 34 in-house sets; per-image SIGMAR runs ~1.35x higher). - HARMONIC_CONTAMINATION, SCALING_NOT_CONVERGED, POWDER_RINGS in the documented vocabulary. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
808 lines
42 KiB
C++
808 lines
42 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "TranslationalNCS.h"
|
|
|
|
#include <algorithm>
|
|
#include <cmath>
|
|
#include <cstring>
|
|
#include <iomanip>
|
|
#include <limits>
|
|
#include <numbers>
|
|
#include <numeric>
|
|
#include <random>
|
|
#include <sstream>
|
|
#include <unordered_set>
|
|
|
|
#include <fftw3.h>
|
|
|
|
#include "../../common/FFTWPlannerLock.h"
|
|
|
|
namespace {
|
|
// The Patterson is measured in a band, not over the whole dataset: below ~20 A the bulk solvent
|
|
// and the beam stop dominate, above ~5 A the map answers a question about atoms rather than about
|
|
// the arrangement of molecules, which is what a pseudo-translation is.
|
|
constexpr double kPattersonDMin = 5.0;
|
|
constexpr double kPattersonDMax = 20.0;
|
|
// A pseudo-translation shorter than this is a vector inside one molecule, not a translation
|
|
// between two of them; the origin peak's own shoulder also reaches out this far.
|
|
constexpr double kMinPeakDistanceA = 15.0;
|
|
// A peak this high is not a pseudo-symmetry: the origin peak is the total, so a peak at this
|
|
// fraction of it means essentially every reflection is invariant under the translation, which
|
|
// makes it a lattice vector. Measured over 137 datasets the two populations do not overlap - the
|
|
// strongest genuine pseudo-translation reaches 62% of the origin, while an undeclared centring
|
|
// (a C-, I- or F-centred crystal merged in P1) reads 83-102%.
|
|
constexpr double kLatticeTranslationPct = 75.0;
|
|
constexpr double kGridSampling = 2.5; // grid spacing = d_min / this
|
|
constexpr int kPattersonShells = 20; // equal-count shells for the E^2 normalisation
|
|
constexpr int kMinPattersonReflections = 50;
|
|
|
|
constexpr double kModulationDMin = 4.0; // coarse enough that a small error in u does not
|
|
// scramble the phase h.u
|
|
constexpr int kModulationShells = 10;
|
|
constexpr int kPhaseBins = 12;
|
|
constexpr int kMinModulationReflections = 500;
|
|
constexpr double kWilsonOutlierE2 = 8.0;
|
|
constexpr double kMinShellISig = 1.0;
|
|
// 20 rather than a handful: the null is the median of that many greedy refinements from random
|
|
// starts, and with 6 draws its scatter was enough to move datasets sitting near the gate across
|
|
// it from one RNG seed to the next (measured: three of them). The refinement is a dot product
|
|
// per trial, so the extra draws cost a few ms.
|
|
constexpr int kModulationNullDraws = 20;
|
|
|
|
// Gate constants. Read off a corpus of 128 merged datasets, on which the pair fires on 7.0% of
|
|
// them - a pathology rate, and close to the ~6% of PDB entries Read, Adams & McCoy report with a
|
|
// large off-origin Patterson peak. Either criterion ALONE fires on roughly twice as many: a 40%
|
|
// peak can carry no modulation at all, and on a 70,000-reflection dataset a 4% peak reaches
|
|
// z = 18 while modulating nothing.
|
|
constexpr double kPeakZThreshold = 5.0;
|
|
constexpr double kModulationRatioThreshold = 2.5;
|
|
// A modulation this deep means the suppressed class carries under 5% of the other, i.e. it is
|
|
// very nearly systematically absent - which is what a LATTICE translation of a smaller cell looks
|
|
// like measured in the larger one. Over 137 datasets exactly one crystal reaches it (at 89x, the
|
|
// next-deepest genuine pseudo-symmetry being 14x), and its deposited entry is indeed a cell of
|
|
// half the volume, whose axis length this detector reports to within 0.2 A.
|
|
constexpr double kNearExtinctModulation = 20.0;
|
|
|
|
// Above this many grid points the null is cut from 30 draws to 10. A deterministic function of
|
|
// the cell, so the answer does not depend on the machine; no dataset in the corpus reaches it
|
|
// (the largest grid measured was 2.5 M points on a 545 A cell).
|
|
constexpr size_t kLargeGridPoints = 4u << 20;
|
|
|
|
struct Vec3 { double x, y, z; };
|
|
|
|
// Round up to the next even number whose only prime factors are 2, 3 and 5 - the sizes FFTW has
|
|
// codelets for.
|
|
int SmoothSize(int v) {
|
|
v = std::max(2, v);
|
|
for (;; ++v) {
|
|
if (v % 2 != 0)
|
|
continue;
|
|
int x = v;
|
|
for (int p : {2, 3, 5})
|
|
while (x % p == 0)
|
|
x /= p;
|
|
if (x == 1)
|
|
return v;
|
|
}
|
|
}
|
|
|
|
// Origin-equivalent points of the lattice: the origin plus the centring translations, which
|
|
// produce a large Patterson peak by construction and are not pseudo-translations.
|
|
std::vector<Vec3> OriginEquivalents(const gemmi::SpaceGroup *sg) {
|
|
std::vector<Vec3> out{{0.0, 0.0, 0.0}};
|
|
if (!sg)
|
|
return out;
|
|
for (const auto &t : sg->operations().cen_ops) {
|
|
const Vec3 v{t[0] / 24.0, t[1] / 24.0, t[2] / 24.0};
|
|
if (v.x != 0.0 || v.y != 0.0 || v.z != 0.0)
|
|
out.push_back(v);
|
|
}
|
|
return out;
|
|
}
|
|
|
|
// Shortest distance in A from a fractional position to any origin-equivalent lattice point.
|
|
double DistanceToOrigin(const gemmi::UnitCell &cell, const Vec3 &frac,
|
|
const std::vector<Vec3> &equivalents) {
|
|
double best = std::numeric_limits<double>::infinity();
|
|
for (const auto &c : equivalents)
|
|
for (int dx = -1; dx <= 1; ++dx)
|
|
for (int dy = -1; dy <= 1; ++dy)
|
|
for (int dz = -1; dz <= 1; ++dz) {
|
|
const gemmi::Position p = cell.orthogonalize(
|
|
gemmi::Fractional{frac.x - c.x + dx, frac.y - c.y + dy, frac.z - c.z + dz});
|
|
best = std::min(best, p.length());
|
|
}
|
|
return best;
|
|
}
|
|
|
|
// How well a vector explains the intensities of the first `n` reflections under the physical
|
|
// model <E^2> = A (1 + f cos(2 pi h.u)): the correlation between E^2 and cos(2 pi h.u).
|
|
//
|
|
// The gate above scores a vector by the max/min ratio of the phase-bin means instead. That is the
|
|
// right statistic for the gate - it is what the null is drawn against - but it is the wrong one to
|
|
// REFINE on a full-resolution population, because a ratio is maximised by emptying one bin: on a
|
|
// 118,000-reflection merge the max/min search walked to a vector 0.015 away from the correlation's
|
|
// answer while reporting 1.2e4. The fitted amplitude f itself is no better, for the mirror reason:
|
|
// f is a slope divided by an intercept, so it runs away wherever the cosine has little variance
|
|
// over the data, and the search drifts to exactly such a vector (measured: 0.06 off a synthetic
|
|
// half-integer translation). The correlation has neither failure - it is 1 only where the cosine
|
|
// follows the intensities.
|
|
double CosineCorrelation(const std::vector<std::array<int, 3>> &hkl, const std::vector<double> &e2,
|
|
const std::array<double, 3> &u, size_t n) {
|
|
double sc = 0.0, scc = 0.0, sy = 0.0, syy = 0.0, scy = 0.0;
|
|
for (size_t i = 0; i < n; ++i) {
|
|
const double c = std::cos(2.0 * std::numbers::pi
|
|
* (hkl[i][0] * u[0] + hkl[i][1] * u[1] + hkl[i][2] * u[2]));
|
|
sc += c;
|
|
scc += c * c;
|
|
sy += e2[i];
|
|
syy += e2[i] * e2[i];
|
|
scy += c * e2[i];
|
|
}
|
|
const double nn = static_cast<double>(n);
|
|
const double vc = scc - sc * sc / nn, vy = syy - sy * sy / nn;
|
|
if (!(vc > 0.0) || !(vy > 0.0))
|
|
return 0.0;
|
|
return (scy - sc * sy / nn) / std::sqrt(vc * vy);
|
|
}
|
|
}
|
|
|
|
TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector<MergedReflection> &merged,
|
|
const gemmi::UnitCell &cell,
|
|
const gemmi::SpaceGroup *space_group,
|
|
int null_draws, uint32_t seed) {
|
|
TranslationalNCSResult result;
|
|
if (cell.a <= 0.0 || cell.b <= 0.0 || cell.c <= 0.0) {
|
|
result.refusal = "no unit cell";
|
|
return result;
|
|
}
|
|
|
|
const gemmi::GroupOps gops = space_group ? space_group->operations() : gemmi::GroupOps{};
|
|
auto acentric = [&](const MergedReflection &r) {
|
|
return !space_group || !gops.is_reflection_centric(gemmi::Op::Miller{{r.h, r.k, r.l}});
|
|
};
|
|
|
|
// ------------------------------------------------------------------ (a) the Patterson peak
|
|
// Following the native-Patterson construction of Read, Adams & McCoy (2013) Acta Cryst. D69,
|
|
// 176-183, with the fixed peak-height table replaced by a per-dataset permutation null: the
|
|
// noise floor of this statistic runs from ~1% of the origin on a 74,000-reflection merge to ~18%
|
|
// on a 138-reflection one, so no fixed percentage can serve both.
|
|
std::vector<int> idx;
|
|
for (size_t i = 0; i < merged.size(); ++i) {
|
|
const auto &r = merged[i];
|
|
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kPattersonDMin && r.d <= kPattersonDMax)
|
|
idx.push_back(static_cast<int>(i));
|
|
}
|
|
result.n_reflections = static_cast<int>(idx.size());
|
|
if (result.n_reflections < kMinPattersonReflections) {
|
|
std::ostringstream os;
|
|
os << "only " << result.n_reflections << " reflections between " << kPattersonDMax << " and "
|
|
<< kPattersonDMin << " A - too few for a Patterson";
|
|
result.refusal = os.str();
|
|
return result;
|
|
}
|
|
|
|
// E^2: divide each intensity by the mean of an equal-count resolution shell, so the map is not
|
|
// dominated by the Wilson fall-off and the peak percentage is comparable between datasets.
|
|
std::sort(idx.begin(), idx.end(),
|
|
[&](int a, int b) { return merged[a].d != merged[b].d ? merged[a].d > merged[b].d : a < b; });
|
|
const int n = result.n_reflections;
|
|
std::vector<double> e2(n, 0.0);
|
|
std::vector<int> shell_of(n, 0);
|
|
for (int s = 0; s < kPattersonShells; ++s) {
|
|
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
|
|
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
|
|
if (hi <= lo)
|
|
continue;
|
|
double sum = 0.0;
|
|
for (int i = lo; i < hi; ++i)
|
|
sum += merged[idx[i]].I;
|
|
const double mean = sum / (hi - lo);
|
|
for (int i = lo; i < hi; ++i) {
|
|
e2[i] = mean != 0.0 ? merged[idx[i]].I / mean : 0.0;
|
|
shell_of[i] = s;
|
|
}
|
|
}
|
|
|
|
// Grid. Sampled at d_min / 2.5, rounded up to an FFT-friendly size. A very large cell is measured
|
|
// on a coarser map rather than refused: the map only has to resolve a peak 15 A from the origin.
|
|
double patterson_dmin = kPattersonDMin;
|
|
std::array<int, 3> g{};
|
|
const double axis[3] = {cell.a, cell.b, cell.c};
|
|
for (;;) {
|
|
for (int i = 0; i < 3; ++i)
|
|
g[i] = SmoothSize(static_cast<int>(std::ceil(axis[i] * kGridSampling / patterson_dmin)));
|
|
const size_t points = static_cast<size_t>(g[0]) * g[1] * g[2];
|
|
if (points <= (64u << 20) || patterson_dmin >= 12.0)
|
|
break;
|
|
patterson_dmin *= 1.25;
|
|
}
|
|
result.grid = g;
|
|
const int nx = g[0], ny = g[1], nz = g[2];
|
|
const int nzh = nz / 2 + 1;
|
|
const size_t n_points = static_cast<size_t>(nx) * ny * nz;
|
|
const size_t n_complex = static_cast<size_t>(nx) * ny * nzh;
|
|
if (n_points > kLargeGridPoints)
|
|
null_draws = std::min(null_draws, 10);
|
|
|
|
// Expand the asymmetric unit onto the full sphere: every rotation of the space group and both
|
|
// Friedel mates, deduplicated. Only the half-space l' <= nz/2 is stored - the Patterson
|
|
// coefficients are real and centrosymmetric, so the other half is what the c2r transform infers.
|
|
std::vector<std::pair<size_t, int>> fill; // (index into the half-space grid, source reflection)
|
|
fill.reserve(idx.size() * 8);
|
|
{
|
|
std::unordered_set<int64_t> seen;
|
|
seen.reserve(idx.size() * 16);
|
|
std::vector<gemmi::Op> ops;
|
|
if (space_group)
|
|
ops = gops.sym_ops;
|
|
else
|
|
ops.push_back(gemmi::Op::identity());
|
|
for (const auto &op : ops)
|
|
for (int sign : {1, -1})
|
|
for (int i = 0; i < n; ++i) {
|
|
const auto &r = merged[idx[i]];
|
|
const gemmi::Op::Miller m = op.apply_to_hkl({{r.h, r.k, r.l}});
|
|
const int64_t h = sign * m[0], k = sign * m[1], l = sign * m[2];
|
|
const int64_t key = ((h + 1048576) << 42) | ((k + 1048576) << 21) | (l + 1048576);
|
|
if (!seen.insert(key).second)
|
|
continue;
|
|
const int gl = static_cast<int>(((l % nz) + nz) % nz);
|
|
if (gl > nz / 2)
|
|
continue; // its Friedel mate is stored instead
|
|
const int gh = static_cast<int>(((h % nx) + nx) % nx);
|
|
const int gk = static_cast<int>(((k % ny) + ny) % ny);
|
|
fill.emplace_back((static_cast<size_t>(gh) * ny + gk) * nzh + gl, i);
|
|
}
|
|
}
|
|
|
|
float *coeff = fftwf_alloc_real(2 * n_complex);
|
|
float *map = fftwf_alloc_real(n_points);
|
|
if (coeff == nullptr || map == nullptr) {
|
|
if (coeff) fftwf_free(coeff);
|
|
if (map) fftwf_free(map);
|
|
result.refusal = "could not allocate the Patterson grid";
|
|
return result;
|
|
}
|
|
fftwf_plan plan;
|
|
{
|
|
std::lock_guard lock(FFTWPlannerMutex());
|
|
plan = fftwf_plan_dft_c2r_3d(nx, ny, nz, reinterpret_cast<fftwf_complex *>(coeff), map,
|
|
FFTW_ESTIMATE);
|
|
}
|
|
|
|
// The near-origin exclusion, as a set of grid points rather than a per-point distance test: only
|
|
// the points inside a 15 A ball around each origin-equivalent point can be excluded, and there
|
|
// are a few thousand of those however large the grid is.
|
|
std::vector<uint8_t> near(n_points, 0);
|
|
std::vector<Vec3> equivalents = OriginEquivalents(space_group);
|
|
auto build_near_mask = [&]() {
|
|
std::fill(near.begin(), near.end(), 0);
|
|
const gemmi::UnitCell &c = cell;
|
|
// Fractional half-width of a sphere of radius R along axis i is R * |a*_i|, so the ball is
|
|
// enumerated directly instead of testing every grid point: the cost is a few thousand points
|
|
// however large the grid is.
|
|
const double rstar[3] = {c.ar, c.br, c.cr};
|
|
for (const auto &e : equivalents) {
|
|
const int span[3] = {static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[0] * nx)) + 1,
|
|
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[1] * ny)) + 1,
|
|
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[2] * nz)) + 1};
|
|
const int c0[3] = {static_cast<int>(std::lround(e.x * nx)),
|
|
static_cast<int>(std::lround(e.y * ny)),
|
|
static_cast<int>(std::lround(e.z * nz))};
|
|
for (int i = c0[0] - span[0]; i <= c0[0] + span[0]; ++i)
|
|
for (int j = c0[1] - span[1]; j <= c0[1] + span[1]; ++j)
|
|
for (int k = c0[2] - span[2]; k <= c0[2] + span[2]; ++k) {
|
|
const gemmi::Position p = cell.orthogonalize(gemmi::Fractional{
|
|
static_cast<double>(i) / nx - e.x, static_cast<double>(j) / ny - e.y,
|
|
static_cast<double>(k) / nz - e.z});
|
|
if (p.length() >= kMinPeakDistanceA)
|
|
continue;
|
|
const size_t gi = static_cast<size_t>(((i % nx) + nx) % nx);
|
|
const size_t gj = static_cast<size_t>(((j % ny) + ny) % ny);
|
|
const size_t gk = static_cast<size_t>(((k % nz) + nz) % nz);
|
|
near[(gi * ny + gj) * nz + gk] = 1;
|
|
}
|
|
}
|
|
};
|
|
build_near_mask();
|
|
|
|
auto transform = [&](const std::vector<double> &values) {
|
|
std::memset(coeff, 0, 2 * n_complex * sizeof(float));
|
|
for (const auto &[at, src] : fill)
|
|
coeff[2 * at] += static_cast<float>(values[src]);
|
|
fftwf_execute(plan);
|
|
};
|
|
auto max_far = [&]() {
|
|
double best = -std::numeric_limits<double>::infinity();
|
|
for (size_t i = 0; i < n_points; ++i)
|
|
if (!near[i] && map[i] > best)
|
|
best = map[i];
|
|
return best;
|
|
};
|
|
|
|
transform(e2);
|
|
const double origin = map[0];
|
|
|
|
// The vector to report is the largest off-origin LOCAL maximum, which is not always the largest
|
|
// off-origin grid value: the tail of the origin peak can be higher right at the 15 A boundary,
|
|
// and that is a shoulder, not a peak.
|
|
auto top_local_max = [&]() {
|
|
double best = -std::numeric_limits<double>::infinity();
|
|
std::array<double, 3> at_frac{{0.0, 0.0, 0.0}};
|
|
for (int i = 0; i < nx; ++i)
|
|
for (int j = 0; j < ny; ++j)
|
|
for (int k = 0; k < nz; ++k) {
|
|
const size_t at = (static_cast<size_t>(i) * ny + j) * nz + k;
|
|
if (near[at] || map[at] <= best)
|
|
continue;
|
|
bool is_max = true;
|
|
for (int dx = -1; dx <= 1 && is_max; ++dx)
|
|
for (int dy = -1; dy <= 1 && is_max; ++dy)
|
|
for (int dz = -1; dz <= 1; ++dz) {
|
|
if (dx == 0 && dy == 0 && dz == 0)
|
|
continue;
|
|
const size_t nb = (static_cast<size_t>((i + dx + nx) % nx) * ny
|
|
+ (j + dy + ny) % ny) * nz + (k + dz + nz) % nz;
|
|
if (map[nb] > map[at]) { is_max = false; break; }
|
|
}
|
|
if (is_max) {
|
|
best = map[at];
|
|
at_frac = {static_cast<double>(i) / nx, static_cast<double>(j) / ny,
|
|
static_cast<double>(k) / nz};
|
|
}
|
|
}
|
|
return std::pair<std::array<double, 3>, double>{at_frac, best};
|
|
};
|
|
|
|
// Absorb any translation the data are invariant under - a centring the space group in use does
|
|
// not declare - into the exclusion, and look again underneath it. Without this, a C-centred
|
|
// crystal merged in P1 reports its own centring vector as a pseudo-symmetry (measured: 11 of the
|
|
// 95 P1 merges in the corpus), and a genuine pseudo-translation hiding below a centring vector is
|
|
// never reached.
|
|
double observed = 0.0;
|
|
for (int pass = 0; pass < 5; ++pass) {
|
|
observed = max_far();
|
|
result.peak_percent = origin > 0.0 ? 100.0 * observed / origin : 0.0;
|
|
const auto [frac, height] = top_local_max();
|
|
result.peak_frac = frac;
|
|
if (result.peak_percent < kLatticeTranslationPct || !std::isfinite(height))
|
|
break;
|
|
result.undeclared_lattice_translations.push_back(frac);
|
|
result.undeclared_lattice_translation_pct.push_back(100.0 * height / origin);
|
|
equivalents.push_back(Vec3{frac[0], frac[1], frac[2]});
|
|
build_near_mask();
|
|
}
|
|
|
|
// Permutation null: the same reflections, the same map, with the intensities shuffled inside each
|
|
// resolution shell. Everything that could make a Patterson peak except the correlation between
|
|
// reflections is preserved, so this is the noise floor of THIS dataset.
|
|
{
|
|
std::mt19937 rng(seed);
|
|
std::vector<double> permuted = e2;
|
|
std::vector<double> nulls;
|
|
nulls.reserve(null_draws);
|
|
for (int draw = 0; draw < null_draws; ++draw) {
|
|
permuted = e2;
|
|
for (int s = 0; s < kPattersonShells; ++s) {
|
|
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
|
|
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
|
|
for (int i = hi - 1; i > lo; --i)
|
|
std::swap(permuted[i], permuted[lo + static_cast<int>(rng() % (i - lo + 1))]);
|
|
}
|
|
transform(permuted);
|
|
if (map[0] > 0.0)
|
|
nulls.push_back(100.0 * max_far() / map[0]);
|
|
}
|
|
if (!nulls.empty()) {
|
|
const double mean = std::accumulate(nulls.begin(), nulls.end(), 0.0) / nulls.size();
|
|
double var = 0.0;
|
|
for (double v : nulls)
|
|
var += (v - mean) * (v - mean);
|
|
var /= nulls.size();
|
|
result.null_mean = mean;
|
|
result.null_sd = std::sqrt(var);
|
|
result.peak_z = result.null_sd > 0.0 ? (result.peak_percent - mean) / result.null_sd : 0.0;
|
|
}
|
|
}
|
|
|
|
{
|
|
std::lock_guard lock(FFTWPlannerMutex());
|
|
fftwf_destroy_plan(plan);
|
|
}
|
|
fftwf_free(coeff);
|
|
fftwf_free(map);
|
|
result.measurable = true;
|
|
|
|
// ------------------------------------------------------------------ (b) the modulation depth
|
|
// A pure translation u between two copies multiplies the intensity by |1 + exp(2 pi i h.u)|^2, so
|
|
// <I> depends on the phase h.u: strong where it is near an integer, extinct near a half-integer.
|
|
// The Patterson peak says a vector exists; this says whether it modulates the data.
|
|
std::vector<std::array<int, 3>> mod_hkl;
|
|
std::vector<double> mod_e2;
|
|
// Where each kept resolution shell ends in the two arrays above. The null below shuffles the
|
|
// intensities inside a shell, exactly as the Patterson null does, and the shells are written
|
|
// consecutively, so their boundaries are all it needs.
|
|
std::vector<size_t> mod_shell_bounds{0};
|
|
{
|
|
std::vector<int> mod_idx;
|
|
for (size_t i = 0; i < merged.size(); ++i) {
|
|
const auto &r = merged[i];
|
|
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kModulationDMin && acentric(r))
|
|
mod_idx.push_back(static_cast<int>(i));
|
|
}
|
|
if (!mod_idx.empty()) {
|
|
// Shells linear in 1/d^2, and the same defensive normalisation the twinning statistics
|
|
// use: noise-only shells skipped, Wilson outliers rejected, the shell mean re-fitted once
|
|
// so the outlier does not corrupt the mean it is measured against.
|
|
double min_s = std::numeric_limits<double>::infinity(), max_s = -min_s;
|
|
for (int i : mod_idx) {
|
|
const double s = 1.0 / (merged[i].d * merged[i].d);
|
|
min_s = std::min(min_s, s);
|
|
max_s = std::max(max_s, s);
|
|
}
|
|
if (max_s > min_s) {
|
|
std::vector<std::vector<int>> bins(kModulationShells);
|
|
for (int i : mod_idx) {
|
|
const double t = (1.0 / (merged[i].d * merged[i].d) - min_s) / (max_s - min_s);
|
|
bins[std::clamp(static_cast<int>(t * kModulationShells), 0, kModulationShells - 1)]
|
|
.push_back(i);
|
|
}
|
|
for (const auto &b : bins) {
|
|
if (b.empty())
|
|
continue;
|
|
double isig = 0.0;
|
|
int n_isig = 0;
|
|
for (int i : b)
|
|
if (std::isfinite(merged[i].sigma) && merged[i].sigma > 0.0) {
|
|
isig += merged[i].I / merged[i].sigma;
|
|
++n_isig;
|
|
}
|
|
if (n_isig == 0 || isig / n_isig < kMinShellISig)
|
|
continue;
|
|
double sum = 0.0;
|
|
for (int i : b)
|
|
sum += merged[i].I;
|
|
double mean = sum / b.size();
|
|
if (mean <= 0.0)
|
|
continue;
|
|
sum = 0.0;
|
|
int kept = 0;
|
|
for (int i : b)
|
|
if (merged[i].I / mean <= kWilsonOutlierE2) { sum += merged[i].I; ++kept; }
|
|
if (kept == 0)
|
|
continue;
|
|
mean = sum / kept;
|
|
if (mean <= 0.0)
|
|
continue;
|
|
for (int i : b) {
|
|
const double v = merged[i].I / mean;
|
|
if (v > kWilsonOutlierE2)
|
|
continue;
|
|
mod_hkl.push_back({merged[i].h, merged[i].k, merged[i].l});
|
|
mod_e2.push_back(v);
|
|
}
|
|
mod_shell_bounds.push_back(mod_e2.size());
|
|
}
|
|
}
|
|
}
|
|
}
|
|
result.n_modulation_reflections = static_cast<int>(mod_e2.size());
|
|
if (result.n_modulation_reflections < kMinModulationReflections)
|
|
return result;
|
|
|
|
const int floor_count = std::max(50, result.n_modulation_reflections / (4 * kPhaseBins));
|
|
auto contrast = [&](const std::vector<double> &e2v, const std::array<double, 3> &u,
|
|
std::vector<double> *profile) {
|
|
std::array<double, kPhaseBins> sum{}, cnt{};
|
|
sum.fill(0.0);
|
|
cnt.fill(0.0);
|
|
for (size_t i = 0; i < e2v.size(); ++i) {
|
|
const double t = mod_hkl[i][0] * u[0] + mod_hkl[i][1] * u[1] + mod_hkl[i][2] * u[2];
|
|
const double f = t - std::floor(t);
|
|
const int b = std::min(static_cast<int>(f * kPhaseBins), kPhaseBins - 1);
|
|
sum[b] += e2v[i];
|
|
cnt[b] += 1.0;
|
|
}
|
|
double lo = std::numeric_limits<double>::infinity(), hi = -lo;
|
|
int populated = 0;
|
|
if (profile)
|
|
profile->assign(kPhaseBins, std::numeric_limits<double>::quiet_NaN());
|
|
for (int b = 0; b < kPhaseBins; ++b) {
|
|
if (cnt[b] < floor_count)
|
|
continue;
|
|
const double m = sum[b] / cnt[b];
|
|
if (profile)
|
|
(*profile)[b] = m;
|
|
lo = std::min(lo, m);
|
|
hi = std::max(hi, m);
|
|
++populated;
|
|
}
|
|
if (populated < 2 || lo <= 0.0)
|
|
return std::numeric_limits<double>::quiet_NaN();
|
|
return hi / lo;
|
|
};
|
|
|
|
// Coarse-to-fine local search for the vector that modulates the data most strongly, started from
|
|
// the Patterson peak. The Patterson grid is coarse and a real pseudo-translation is rarely at an
|
|
// exactly rational position, so the peak position alone understates the modulation.
|
|
const std::array<double, 3> step0{1.0 / nx, 1.0 / ny, 1.0 / nz};
|
|
auto refine = [&](const std::vector<double> &e2v, std::array<double, 3> u) {
|
|
std::array<double, 3> s = step0;
|
|
double best = contrast(e2v, u, nullptr);
|
|
for (int round = 0; round < 3; ++round) {
|
|
for (bool improved = true; improved;) {
|
|
improved = false;
|
|
for (int i = 0; i < 3; ++i)
|
|
for (double gstep : {-1.0, -0.5, 0.0, 0.5, 1.0}) {
|
|
std::array<double, 3> v = u;
|
|
v[i] += gstep * s[i];
|
|
const double c = contrast(e2v, v, nullptr);
|
|
if (std::isfinite(c) && c > best + 1e-6) {
|
|
best = c;
|
|
u = v;
|
|
improved = true;
|
|
}
|
|
}
|
|
}
|
|
for (double &x : s)
|
|
x /= 2.5;
|
|
}
|
|
return std::pair<std::array<double, 3>, double>{u, best};
|
|
};
|
|
|
|
const auto [u_refined, best_contrast] = refine(mod_e2, result.peak_frac);
|
|
if (!std::isfinite(best_contrast))
|
|
return result;
|
|
for (int i = 0; i < 3; ++i)
|
|
result.vector_frac[i] = u_refined[i] - std::floor(u_refined[i]);
|
|
result.modulation_measured = true;
|
|
result.modulation = best_contrast;
|
|
contrast(mod_e2, u_refined, &result.phase_bin_mean_e2);
|
|
result.vector_length_A = DistanceToOrigin(cell, Vec3{result.vector_frac[0], result.vector_frac[1],
|
|
result.vector_frac[2]},
|
|
std::vector<Vec3>{{0.0, 0.0, 0.0}});
|
|
|
|
// The modulation's own null: the SAME search, from the SAME start, on the same reflections with
|
|
// the intensities shuffled inside each resolution shell - the permutation the Patterson null
|
|
// above already uses, applied to the second half of the evidence. A greedy search over three
|
|
// free parameters can manufacture contrast out of nothing, and this measures how much rather
|
|
// than assuming it is negligible; the shuffle leaves the reflection count, the E^2 distribution
|
|
// and the phase-bin populations exactly as they were and removes only the one thing the
|
|
// modulation claims to see, the correlation between a reflection's intensity and h.u.
|
|
//
|
|
// Drawing the null from the unshuffled data instead - random starting vectors, the crystal's own
|
|
// intensities - measured the search AND whatever modulation the data carry, because a greedy
|
|
// search started anywhere walks into a real modulation's basin. Measured over 110 merged
|
|
// datasets: single draws reached 199x and 6162x where the crystal's own modulation read 4.0x and
|
|
// 67x, and 28 of the 110 had the control at or above the signal it was controlling for. The
|
|
// median was robust enough that this changes no verdict anywhere in that corpus - what it
|
|
// removes is a control that reads the same signal it is subtracted from, and with it the one
|
|
// way an upstream change can move the control while the modulation stands still.
|
|
{
|
|
std::mt19937 rng(seed + 7);
|
|
std::vector<double> permuted = mod_e2;
|
|
std::vector<double> nulls;
|
|
nulls.reserve(kModulationNullDraws);
|
|
for (int draw = 0; draw < kModulationNullDraws; ++draw) {
|
|
permuted = mod_e2;
|
|
for (size_t s = 1; s < mod_shell_bounds.size(); ++s) {
|
|
const size_t lo = mod_shell_bounds[s - 1], hi = mod_shell_bounds[s];
|
|
for (size_t i = hi; i > lo + 1; --i)
|
|
std::swap(permuted[i - 1], permuted[lo + rng() % (i - lo)]);
|
|
}
|
|
const auto [v, c] = refine(permuted, result.peak_frac);
|
|
(void) v;
|
|
nulls.push_back(std::isfinite(c) ? c : 1.0);
|
|
}
|
|
std::sort(nulls.begin(), nulls.end());
|
|
const size_t m = nulls.size();
|
|
result.modulation_null = m % 2 ? nulls[m / 2] : 0.5 * (nulls[m / 2 - 1] + nulls[m / 2]);
|
|
}
|
|
|
|
result.detected = result.peak_z >= kPeakZThreshold && result.modulation_null > 0.0 &&
|
|
result.modulation / result.modulation_null >= kModulationRatioThreshold;
|
|
|
|
result.near_extinct_class = result.detected && result.modulation >= kNearExtinctModulation;
|
|
|
|
// ------------------------------------------------ the vector the phase CLASSES are assigned with
|
|
//
|
|
// Everything above is measured on d >= 4 A, where |h| stays under about 15 and an error of 0.02 in
|
|
// u costs the contrast almost nothing - the gate's vector is refined only to the accuracy the gate
|
|
// needs. But the consumers of this vector (the L-test's partner steps, and the twin-immune zone's
|
|
// per-class normalisation) assign a CLASS, cos(2 pi h.u), to reflections at FULL resolution, and
|
|
// the phase error there is |h| times the error in u. At |h| = 20 an error of 0.02 is a third of a
|
|
// turn and the classes are no longer the physical ones. Measured as the correlation between E^2
|
|
// and cos(2 pi h.u) in bands of |h| on two of this corpus's pseudo-translations, the gate's
|
|
// vector against this one: 0.57/0.61 at |h| < 8, 0.42/0.52, 0.19/0.30, 0.14/0.18 past |h| = 22 on
|
|
// the first; 0.14/0.52, -0.15/0.42, 0.00/0.20, -0.02/0.09 on the second, whose classes the gate's
|
|
// vector gets ANTI-correlated with the real ones over a whole band.
|
|
//
|
|
// So u is re-refined on the whole data, up a ladder: each step reads twice as many reflections as
|
|
// the one before, in order of |h|, and starts from the previous step's answer with a search step
|
|
// of a quarter of the phase ambiguity 1/|h| of the highest order it reads. The vector is never
|
|
// asked to extrapolate further than it is known, which is what lets a greedy search cross a
|
|
// resolution range whose objective has a local maximum every 1/|h| in each direction. Refining on
|
|
// the whole data in one go instead walks into the nearest of those.
|
|
//
|
|
// Nothing above is re-read: `modulation`, `modulation_null` and `detected` stay the gate's, so
|
|
// which crystals are called stays exactly as it was. Only where the vector points changes.
|
|
if (result.detected) {
|
|
std::vector<std::array<int, 3>> cls_hkl;
|
|
std::vector<double> cls_e2;
|
|
{
|
|
std::vector<int> idx;
|
|
for (size_t i = 0; i < merged.size(); ++i)
|
|
if (std::isfinite(merged[i].I) && std::isfinite(merged[i].d) && merged[i].d > 0.0
|
|
&& acentric(merged[i]))
|
|
idx.push_back(static_cast<int>(i));
|
|
// Equal-count shells sorted by |h|, so that every stage of the ladder reads a population
|
|
// that is already normalised and can be taken as a prefix.
|
|
std::sort(idx.begin(), idx.end(), [&](int a, int b) {
|
|
return std::max({std::abs(merged[a].h), std::abs(merged[a].k), std::abs(merged[a].l)})
|
|
< std::max({std::abs(merged[b].h), std::abs(merged[b].k), std::abs(merged[b].l)});
|
|
});
|
|
std::vector<int> by_d = idx;
|
|
std::sort(by_d.begin(), by_d.end(), [&](int a, int b) { return merged[a].d > merged[b].d; });
|
|
std::vector<double> e2_of(merged.size(), std::numeric_limits<double>::quiet_NaN());
|
|
const size_t n_shells = std::max<size_t>(1, by_d.size() / 500);
|
|
for (size_t s = 0; s < n_shells; ++s) {
|
|
const size_t lo = s * by_d.size() / n_shells, hi = (s + 1) * by_d.size() / n_shells;
|
|
double sum = 0.0;
|
|
for (size_t i = lo; i < hi; ++i)
|
|
sum += merged[by_d[i]].I;
|
|
const double mean = sum / static_cast<double>(hi - lo);
|
|
if (mean <= 0.0)
|
|
continue;
|
|
for (size_t i = lo; i < hi; ++i)
|
|
e2_of[by_d[i]] = merged[by_d[i]].I / mean;
|
|
}
|
|
for (int i : idx)
|
|
if (std::isfinite(e2_of[i]) && e2_of[i] <= kWilsonOutlierE2) {
|
|
cls_hkl.push_back({merged[i].h, merged[i].k, merged[i].l});
|
|
cls_e2.push_back(e2_of[i]);
|
|
}
|
|
}
|
|
// The steps of the ladder double the number of reflections read rather than the |h| ceiling,
|
|
// so that the first step always has a population to refine on whatever the cell is and no
|
|
// step is skipped for being empty.
|
|
std::array<double, 3> u = u_refined;
|
|
for (size_t n = kMinModulationReflections; !cls_hkl.empty(); n *= 2) {
|
|
n = std::min(n, cls_hkl.size());
|
|
const int h_top = std::max({std::abs(cls_hkl[n - 1][0]), std::abs(cls_hkl[n - 1][1]),
|
|
std::abs(cls_hkl[n - 1][2])});
|
|
// A quarter of the phase ambiguity 1/|h| of the highest order this step reads: large
|
|
// enough to walk to the maximum, small enough that the first probe cannot land in the
|
|
// neighbouring one.
|
|
double step = h_top > 0 ? 0.25 / h_top : std::max({step0[0], step0[1], step0[2]});
|
|
double best = CosineCorrelation(cls_hkl, cls_e2, u, n);
|
|
for (int round = 0; round < 8; ++round) {
|
|
for (bool improved = true; improved;) {
|
|
improved = false;
|
|
// All 26 neighbours, not the 6 along the axes: the phase is h.u, so a vector
|
|
// whose class comes mostly from one combination of the axes - h/3 + 2k/3, say -
|
|
// sits in a valley that runs diagonally, and a search that only moves one
|
|
// component at a time stalls on its wall (measured: 0.005 short in two
|
|
// components at once, a quarter of a turn by |h| = 60).
|
|
for (int dx = -1; dx <= 1; ++dx)
|
|
for (int dy = -1; dy <= 1; ++dy)
|
|
for (int dz = -1; dz <= 1; ++dz) {
|
|
if (dx == 0 && dy == 0 && dz == 0)
|
|
continue;
|
|
const std::array<double, 3> v{u[0] + dx * step, u[1] + dy * step,
|
|
u[2] + dz * step};
|
|
const double f = CosineCorrelation(cls_hkl, cls_e2, v, n);
|
|
if (f > best + 1e-9) {
|
|
best = f;
|
|
u = v;
|
|
improved = true;
|
|
}
|
|
}
|
|
}
|
|
step *= 0.5;
|
|
}
|
|
if (n == cls_hkl.size())
|
|
break;
|
|
}
|
|
// A greedy climb up the ladder can still end on an alias: a stage whose population is small
|
|
// and whose objective is flat over the step it searches with can hand the next stage a start
|
|
// one phase turn out, and from there the finer steps cannot come back. The two candidates are
|
|
// carried to the end and the whole data decide between them, which is the one place where
|
|
// both are directly comparable.
|
|
if (!cls_hkl.empty()
|
|
&& CosineCorrelation(cls_hkl, cls_e2, u_refined, cls_hkl.size())
|
|
> CosineCorrelation(cls_hkl, cls_e2, u, cls_hkl.size()))
|
|
u = u_refined;
|
|
for (int i = 0; i < 3; ++i)
|
|
result.vector_frac[i] = u[i] - std::floor(u[i]);
|
|
contrast(mod_e2, u, &result.phase_bin_mean_e2);
|
|
result.vector_length_A = DistanceToOrigin(cell, Vec3{result.vector_frac[0], result.vector_frac[1],
|
|
result.vector_frac[2]},
|
|
std::vector<Vec3>{{0.0, 0.0, 0.0}});
|
|
}
|
|
|
|
// Is the vector a rational translation of the reported cell? If it is, the cell contains a
|
|
// sub-lattice: the pseudo-translation is a candidate for a REAL lattice translation the indexing
|
|
// missed, and the reported cell is a supercell candidate rather than a settled result.
|
|
for (int q = 2; q <= 6 && !result.commensurate; ++q) {
|
|
std::array<int, 3> p{};
|
|
double err = 0.0;
|
|
bool fractional = false;
|
|
for (int i = 0; i < 3; ++i) {
|
|
p[i] = static_cast<int>(std::lround(result.vector_frac[i] * q));
|
|
err = std::max(err, std::fabs(result.vector_frac[i] * q - p[i]));
|
|
if (p[i] % q != 0)
|
|
fractional = true;
|
|
}
|
|
if (err / q <= 0.05 && fractional) {
|
|
result.commensurate = true;
|
|
result.commensurate_denominator = q;
|
|
result.commensurate_numerator = p;
|
|
}
|
|
}
|
|
return result;
|
|
}
|
|
|
|
std::string TranslationalNCSToText(const TranslationalNCSResult &result) {
|
|
std::ostringstream os;
|
|
os << "Translational pseudo-symmetry\n";
|
|
if (!result.measurable) {
|
|
os << " Not measured: " << result.refusal << ".\n"
|
|
<< " => This is not a statement that the crystal has none.\n";
|
|
return os.str();
|
|
}
|
|
os << std::fixed;
|
|
for (size_t i = 0; i < result.undeclared_lattice_translations.size(); ++i) {
|
|
const auto &t = result.undeclared_lattice_translations[i];
|
|
os << " The Patterson at (" << std::setprecision(3) << t[0] << ", " << t[1] << ", " << t[2]
|
|
<< ") is " << std::setprecision(0) << result.undeclared_lattice_translation_pct[i]
|
|
<< "% of the origin: nearly every reflection is invariant under it,\n"
|
|
<< " so it is a LATTICE translation the space group in use does not declare (a centred\n"
|
|
<< " lattice merged without its centring, or a supercell) - or, short of ~90%, a\n"
|
|
<< " pseudo-translation so strong it is nearly one. Excluded, and the search continued past it.\n";
|
|
}
|
|
os << " Native Patterson off-origin peak " << std::setprecision(1) << result.peak_percent
|
|
<< "% of the origin";
|
|
if (result.vector_length_A > 0.0)
|
|
os << " at (" << std::setprecision(3) << result.vector_frac[0] << ", " << result.vector_frac[1]
|
|
<< ", " << result.vector_frac[2] << "), " << std::setprecision(1) << result.vector_length_A
|
|
<< " A";
|
|
os << "\n [z = " << std::setprecision(1) << result.peak_z << " against a permutation null of "
|
|
<< result.null_mean << " +- " << std::setprecision(2) << result.null_sd << "%]\n";
|
|
if (result.modulation_measured)
|
|
os << " Intensity modulation: " << std::setprecision(1) << result.modulation
|
|
<< "x between the strongest and weakest phase class [random-vector control "
|
|
<< result.modulation_null << "x]\n"
|
|
<< " [read on the search's own vector over d >= 4 A, which is what the control is drawn\n"
|
|
<< " against; the vector reported above is that one re-refined on the whole data, so it\n"
|
|
<< " points more accurately and this number is not re-read off it]\n";
|
|
if (!result.modulation_measured) {
|
|
os << " The modulation test could not be run - too few acentric reflections beyond 4 A - so\n"
|
|
<< " the peak above stands alone.\n"
|
|
<< " => Inconclusive. A Patterson peak on its own is not enough to call a pseudo-symmetry\n"
|
|
<< " (peaks of this size occur without one), and this dataset cannot supply the second\n"
|
|
<< " half of the evidence.\n";
|
|
return os.str();
|
|
}
|
|
if (!result.detected) {
|
|
os << " => No translational pseudo-symmetry indicated.\n";
|
|
return os.str();
|
|
}
|
|
os << " => Translational pseudo-symmetry. Molecular replacement will need it declared - Phaser\n"
|
|
<< " reads a tNCS vector and corrects the likelihood for it; without that the search can\n"
|
|
<< " fail on data that are otherwise good.\n";
|
|
if (result.near_extinct_class)
|
|
os << " => The class this vector suppresses is not weak but very nearly EXTINCT, which is\n"
|
|
<< " what a lattice translation of a smaller cell looks like measured in a larger one.\n"
|
|
<< " The reported cell is therefore a supercell candidate rather than a settled\n"
|
|
<< " result, and the rival hypothesis is a cell with this translation as a lattice\n"
|
|
<< " vector.\n";
|
|
else if (result.commensurate)
|
|
os << " => The vector is a 1/" << result.commensurate_denominator
|
|
<< " translation of the cell, so the crystal is pseudo-centred. The cell itself is not in\n"
|
|
<< " question - the suppressed class is weak, not absent, and a smaller cell would\n"
|
|
<< " contradict it.\n";
|
|
return os.str();
|
|
}
|