Files
Jungfraujoch/image_analysis/scale_merge/TranslationalNCS.cpp
T
leonarski_f a39fd29f77
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m4s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 17m46s
Build Packages / build:windows:cuda (push) Successful in 20m20s
Build Packages / build:viewer-tgz:cpu (push) Successful in 15m56s
Build Packages / build:viewer-tgz:cuda (push) Successful in 17m57s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 14m10s
Build Packages / build:rugnux:windows (push) Successful in 11m12s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 7m14s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m13s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 19m17s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 21m21s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 17m26s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m56s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 20m48s
Build Packages / build:rpm (rocky8) (push) Successful in 23m43s
Build Packages / build:rpm (rocky9) (push) Successful in 20m38s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 24m57s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m58s
Build Packages / XDS test (durin plugin) (push) Successful in 10m43s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / Build documentation (push) Successful in 1m5s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (neggia plugin) (push) Successful in 8m57s
Build Packages / DIALS test (push) Successful in 18m40s
v1.0.0-rc.167 (#77)
* `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each.
* `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale.
* The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off.
* The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning.
* `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens.
* The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps.
* `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted.
* The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off.
* rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`).
* The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed.
* Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group.
* Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured.
* A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage.
* A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it.
* `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice.
* Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets.
* Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was.
* A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for.
* `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file.
* Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report.
* Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0.
* The results report's `REPORT_VERSION` is 7.

Reviewed-on: #77
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-09-09 07:25:13 +02:00

617 lines
30 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "TranslationalNCS.h"
#include <algorithm>
#include <cmath>
#include <cstring>
#include <iomanip>
#include <limits>
#include <numeric>
#include <random>
#include <sstream>
#include <unordered_set>
#include <fftw3.h>
namespace {
// The Patterson is measured in a band, not over the whole dataset: below ~20 A the bulk solvent
// and the beam stop dominate, above ~5 A the map answers a question about atoms rather than about
// the arrangement of molecules, which is what a pseudo-translation is.
constexpr double kPattersonDMin = 5.0;
constexpr double kPattersonDMax = 20.0;
// A pseudo-translation shorter than this is a vector inside one molecule, not a translation
// between two of them; the origin peak's own shoulder also reaches out this far.
constexpr double kMinPeakDistanceA = 15.0;
// A peak this high is not a pseudo-symmetry: the origin peak is the total, so a peak at this
// fraction of it means essentially every reflection is invariant under the translation, which
// makes it a lattice vector. Measured over 137 datasets the two populations do not overlap - the
// strongest genuine pseudo-translation reaches 62% of the origin, while an undeclared centring
// (a C-, I- or F-centred crystal merged in P1) reads 83-102%.
constexpr double kLatticeTranslationPct = 75.0;
constexpr double kGridSampling = 2.5; // grid spacing = d_min / this
constexpr int kPattersonShells = 20; // equal-count shells for the E^2 normalisation
constexpr int kMinPattersonReflections = 50;
constexpr double kModulationDMin = 4.0; // coarse enough that a small error in u does not
// scramble the phase h.u
constexpr int kModulationShells = 10;
constexpr int kPhaseBins = 12;
constexpr int kMinModulationReflections = 500;
constexpr double kWilsonOutlierE2 = 8.0;
constexpr double kMinShellISig = 1.0;
// 20 rather than a handful: the null is the median of that many greedy refinements from random
// starts, and with 6 draws its scatter was enough to move datasets sitting near the gate across
// it from one RNG seed to the next (measured: three of them). The refinement is a dot product
// per trial, so the extra draws cost a few ms.
constexpr int kModulationNullDraws = 20;
// Gate constants. Read off a corpus of 128 merged datasets, on which the pair fires on 7.0% of
// them - a pathology rate, and close to the ~6% of PDB entries Read, Adams & McCoy report with a
// large off-origin Patterson peak. Either criterion ALONE fires on roughly twice as many: a 40%
// peak can carry no modulation at all, and on a 70,000-reflection dataset a 4% peak reaches
// z = 18 while modulating nothing.
constexpr double kPeakZThreshold = 5.0;
constexpr double kModulationRatioThreshold = 2.5;
// A modulation this deep means the suppressed class carries under 5% of the other, i.e. it is
// very nearly systematically absent - which is what a LATTICE translation of a smaller cell looks
// like measured in the larger one. Over 137 datasets exactly one crystal reaches it (at 89x, the
// next-deepest genuine pseudo-symmetry being 14x), and its deposited entry is indeed a cell of
// half the volume, whose axis length this detector reports to within 0.2 A.
constexpr double kNearExtinctModulation = 20.0;
// Above this many grid points the null is cut from 30 draws to 10. A deterministic function of
// the cell, so the answer does not depend on the machine; no dataset in the corpus reaches it
// (the largest grid measured was 2.5 M points on a 545 A cell).
constexpr size_t kLargeGridPoints = 4u << 20;
struct Vec3 { double x, y, z; };
// Round up to the next even number whose only prime factors are 2, 3 and 5 - the sizes FFTW has
// codelets for.
int SmoothSize(int v) {
v = std::max(2, v);
for (;; ++v) {
if (v % 2 != 0)
continue;
int x = v;
for (int p : {2, 3, 5})
while (x % p == 0)
x /= p;
if (x == 1)
return v;
}
}
// Origin-equivalent points of the lattice: the origin plus the centring translations, which
// produce a large Patterson peak by construction and are not pseudo-translations.
std::vector<Vec3> OriginEquivalents(const gemmi::SpaceGroup *sg) {
std::vector<Vec3> out{{0.0, 0.0, 0.0}};
if (!sg)
return out;
for (const auto &t : sg->operations().cen_ops) {
const Vec3 v{t[0] / 24.0, t[1] / 24.0, t[2] / 24.0};
if (v.x != 0.0 || v.y != 0.0 || v.z != 0.0)
out.push_back(v);
}
return out;
}
// Shortest distance in A from a fractional position to any origin-equivalent lattice point.
double DistanceToOrigin(const gemmi::UnitCell &cell, const Vec3 &frac,
const std::vector<Vec3> &equivalents) {
double best = std::numeric_limits<double>::infinity();
for (const auto &c : equivalents)
for (int dx = -1; dx <= 1; ++dx)
for (int dy = -1; dy <= 1; ++dy)
for (int dz = -1; dz <= 1; ++dz) {
const gemmi::Position p = cell.orthogonalize(
gemmi::Fractional{frac.x - c.x + dx, frac.y - c.y + dy, frac.z - c.z + dz});
best = std::min(best, p.length());
}
return best;
}
}
TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector<MergedReflection> &merged,
const gemmi::UnitCell &cell,
const gemmi::SpaceGroup *space_group,
int null_draws, uint32_t seed) {
TranslationalNCSResult result;
if (cell.a <= 0.0 || cell.b <= 0.0 || cell.c <= 0.0) {
result.refusal = "no unit cell";
return result;
}
const gemmi::GroupOps gops = space_group ? space_group->operations() : gemmi::GroupOps{};
auto acentric = [&](const MergedReflection &r) {
return !space_group || !gops.is_reflection_centric(gemmi::Op::Miller{{r.h, r.k, r.l}});
};
// ------------------------------------------------------------------ (a) the Patterson peak
// Following the native-Patterson construction of Read, Adams & McCoy (2013) Acta Cryst. D69,
// 176-183, with the fixed peak-height table replaced by a per-dataset permutation null: the
// noise floor of this statistic runs from ~1% of the origin on a 74,000-reflection merge to ~18%
// on a 138-reflection one, so no fixed percentage can serve both.
std::vector<int> idx;
for (size_t i = 0; i < merged.size(); ++i) {
const auto &r = merged[i];
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kPattersonDMin && r.d <= kPattersonDMax)
idx.push_back(static_cast<int>(i));
}
result.n_reflections = static_cast<int>(idx.size());
if (result.n_reflections < kMinPattersonReflections) {
std::ostringstream os;
os << "only " << result.n_reflections << " reflections between " << kPattersonDMax << " and "
<< kPattersonDMin << " A - too few for a Patterson";
result.refusal = os.str();
return result;
}
// E^2: divide each intensity by the mean of an equal-count resolution shell, so the map is not
// dominated by the Wilson fall-off and the peak percentage is comparable between datasets.
std::sort(idx.begin(), idx.end(),
[&](int a, int b) { return merged[a].d != merged[b].d ? merged[a].d > merged[b].d : a < b; });
const int n = result.n_reflections;
std::vector<double> e2(n, 0.0);
std::vector<int> shell_of(n, 0);
for (int s = 0; s < kPattersonShells; ++s) {
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
if (hi <= lo)
continue;
double sum = 0.0;
for (int i = lo; i < hi; ++i)
sum += merged[idx[i]].I;
const double mean = sum / (hi - lo);
for (int i = lo; i < hi; ++i) {
e2[i] = mean != 0.0 ? merged[idx[i]].I / mean : 0.0;
shell_of[i] = s;
}
}
// Grid. Sampled at d_min / 2.5, rounded up to an FFT-friendly size. A very large cell is measured
// on a coarser map rather than refused: the map only has to resolve a peak 15 A from the origin.
double patterson_dmin = kPattersonDMin;
std::array<int, 3> g{};
const double axis[3] = {cell.a, cell.b, cell.c};
for (;;) {
for (int i = 0; i < 3; ++i)
g[i] = SmoothSize(static_cast<int>(std::ceil(axis[i] * kGridSampling / patterson_dmin)));
const size_t points = static_cast<size_t>(g[0]) * g[1] * g[2];
if (points <= (64u << 20) || patterson_dmin >= 12.0)
break;
patterson_dmin *= 1.25;
}
result.grid = g;
const int nx = g[0], ny = g[1], nz = g[2];
const int nzh = nz / 2 + 1;
const size_t n_points = static_cast<size_t>(nx) * ny * nz;
const size_t n_complex = static_cast<size_t>(nx) * ny * nzh;
if (n_points > kLargeGridPoints)
null_draws = std::min(null_draws, 10);
// Expand the asymmetric unit onto the full sphere: every rotation of the space group and both
// Friedel mates, deduplicated. Only the half-space l' <= nz/2 is stored - the Patterson
// coefficients are real and centrosymmetric, so the other half is what the c2r transform infers.
std::vector<std::pair<size_t, int>> fill; // (index into the half-space grid, source reflection)
fill.reserve(idx.size() * 8);
{
std::unordered_set<int64_t> seen;
seen.reserve(idx.size() * 16);
std::vector<gemmi::Op> ops;
if (space_group)
ops = gops.sym_ops;
else
ops.push_back(gemmi::Op::identity());
for (const auto &op : ops)
for (int sign : {1, -1})
for (int i = 0; i < n; ++i) {
const auto &r = merged[idx[i]];
const gemmi::Op::Miller m = op.apply_to_hkl({{r.h, r.k, r.l}});
const int64_t h = sign * m[0], k = sign * m[1], l = sign * m[2];
const int64_t key = ((h + 1048576) << 42) | ((k + 1048576) << 21) | (l + 1048576);
if (!seen.insert(key).second)
continue;
const int gl = static_cast<int>(((l % nz) + nz) % nz);
if (gl > nz / 2)
continue; // its Friedel mate is stored instead
const int gh = static_cast<int>(((h % nx) + nx) % nx);
const int gk = static_cast<int>(((k % ny) + ny) % ny);
fill.emplace_back((static_cast<size_t>(gh) * ny + gk) * nzh + gl, i);
}
}
float *coeff = fftwf_alloc_real(2 * n_complex);
float *map = fftwf_alloc_real(n_points);
if (coeff == nullptr || map == nullptr) {
if (coeff) fftwf_free(coeff);
if (map) fftwf_free(map);
result.refusal = "could not allocate the Patterson grid";
return result;
}
fftwf_plan plan = fftwf_plan_dft_c2r_3d(nx, ny, nz, reinterpret_cast<fftwf_complex *>(coeff), map,
FFTW_ESTIMATE);
// The near-origin exclusion, as a set of grid points rather than a per-point distance test: only
// the points inside a 15 A ball around each origin-equivalent point can be excluded, and there
// are a few thousand of those however large the grid is.
std::vector<uint8_t> near(n_points, 0);
std::vector<Vec3> equivalents = OriginEquivalents(space_group);
auto build_near_mask = [&]() {
std::fill(near.begin(), near.end(), 0);
const gemmi::UnitCell &c = cell;
// Fractional half-width of a sphere of radius R along axis i is R * |a*_i|, so the ball is
// enumerated directly instead of testing every grid point: the cost is a few thousand points
// however large the grid is.
const double rstar[3] = {c.ar, c.br, c.cr};
for (const auto &e : equivalents) {
const int span[3] = {static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[0] * nx)) + 1,
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[1] * ny)) + 1,
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[2] * nz)) + 1};
const int c0[3] = {static_cast<int>(std::lround(e.x * nx)),
static_cast<int>(std::lround(e.y * ny)),
static_cast<int>(std::lround(e.z * nz))};
for (int i = c0[0] - span[0]; i <= c0[0] + span[0]; ++i)
for (int j = c0[1] - span[1]; j <= c0[1] + span[1]; ++j)
for (int k = c0[2] - span[2]; k <= c0[2] + span[2]; ++k) {
const gemmi::Position p = cell.orthogonalize(gemmi::Fractional{
static_cast<double>(i) / nx - e.x, static_cast<double>(j) / ny - e.y,
static_cast<double>(k) / nz - e.z});
if (p.length() >= kMinPeakDistanceA)
continue;
const size_t gi = static_cast<size_t>(((i % nx) + nx) % nx);
const size_t gj = static_cast<size_t>(((j % ny) + ny) % ny);
const size_t gk = static_cast<size_t>(((k % nz) + nz) % nz);
near[(gi * ny + gj) * nz + gk] = 1;
}
}
};
build_near_mask();
auto transform = [&](const std::vector<double> &values) {
std::memset(coeff, 0, 2 * n_complex * sizeof(float));
for (const auto &[at, src] : fill)
coeff[2 * at] += static_cast<float>(values[src]);
fftwf_execute(plan);
};
auto max_far = [&]() {
double best = -std::numeric_limits<double>::infinity();
for (size_t i = 0; i < n_points; ++i)
if (!near[i] && map[i] > best)
best = map[i];
return best;
};
transform(e2);
const double origin = map[0];
// The vector to report is the largest off-origin LOCAL maximum, which is not always the largest
// off-origin grid value: the tail of the origin peak can be higher right at the 15 A boundary,
// and that is a shoulder, not a peak.
auto top_local_max = [&]() {
double best = -std::numeric_limits<double>::infinity();
std::array<double, 3> at_frac{{0.0, 0.0, 0.0}};
for (int i = 0; i < nx; ++i)
for (int j = 0; j < ny; ++j)
for (int k = 0; k < nz; ++k) {
const size_t at = (static_cast<size_t>(i) * ny + j) * nz + k;
if (near[at] || map[at] <= best)
continue;
bool is_max = true;
for (int dx = -1; dx <= 1 && is_max; ++dx)
for (int dy = -1; dy <= 1 && is_max; ++dy)
for (int dz = -1; dz <= 1; ++dz) {
if (dx == 0 && dy == 0 && dz == 0)
continue;
const size_t nb = (static_cast<size_t>((i + dx + nx) % nx) * ny
+ (j + dy + ny) % ny) * nz + (k + dz + nz) % nz;
if (map[nb] > map[at]) { is_max = false; break; }
}
if (is_max) {
best = map[at];
at_frac = {static_cast<double>(i) / nx, static_cast<double>(j) / ny,
static_cast<double>(k) / nz};
}
}
return std::pair<std::array<double, 3>, double>{at_frac, best};
};
// Absorb any translation the data are invariant under - a centring the space group in use does
// not declare - into the exclusion, and look again underneath it. Without this, a C-centred
// crystal merged in P1 reports its own centring vector as a pseudo-symmetry (measured: 11 of the
// 95 P1 merges in the corpus), and a genuine pseudo-translation hiding below a centring vector is
// never reached.
double observed = 0.0;
for (int pass = 0; pass < 5; ++pass) {
observed = max_far();
result.peak_percent = origin > 0.0 ? 100.0 * observed / origin : 0.0;
const auto [frac, height] = top_local_max();
result.peak_frac = frac;
if (result.peak_percent < kLatticeTranslationPct || !std::isfinite(height))
break;
result.undeclared_lattice_translations.push_back(frac);
equivalents.push_back(Vec3{frac[0], frac[1], frac[2]});
build_near_mask();
}
// Permutation null: the same reflections, the same map, with the intensities shuffled inside each
// resolution shell. Everything that could make a Patterson peak except the correlation between
// reflections is preserved, so this is the noise floor of THIS dataset.
{
std::mt19937 rng(seed);
std::vector<double> permuted = e2;
std::vector<double> nulls;
nulls.reserve(null_draws);
for (int draw = 0; draw < null_draws; ++draw) {
permuted = e2;
for (int s = 0; s < kPattersonShells; ++s) {
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
for (int i = hi - 1; i > lo; --i)
std::swap(permuted[i], permuted[lo + static_cast<int>(rng() % (i - lo + 1))]);
}
transform(permuted);
if (map[0] > 0.0)
nulls.push_back(100.0 * max_far() / map[0]);
}
if (!nulls.empty()) {
const double mean = std::accumulate(nulls.begin(), nulls.end(), 0.0) / nulls.size();
double var = 0.0;
for (double v : nulls)
var += (v - mean) * (v - mean);
var /= nulls.size();
result.null_mean = mean;
result.null_sd = std::sqrt(var);
result.peak_z = result.null_sd > 0.0 ? (result.peak_percent - mean) / result.null_sd : 0.0;
}
}
fftwf_destroy_plan(plan);
fftwf_free(coeff);
fftwf_free(map);
result.measurable = true;
// ------------------------------------------------------------------ (b) the modulation depth
// A pure translation u between two copies multiplies the intensity by |1 + exp(2 pi i h.u)|^2, so
// <I> depends on the phase h.u: strong where it is near an integer, extinct near a half-integer.
// The Patterson peak says a vector exists; this says whether it modulates the data.
std::vector<std::array<int, 3>> mod_hkl;
std::vector<double> mod_e2;
{
std::vector<int> mod_idx;
for (size_t i = 0; i < merged.size(); ++i) {
const auto &r = merged[i];
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kModulationDMin && acentric(r))
mod_idx.push_back(static_cast<int>(i));
}
if (!mod_idx.empty()) {
// Shells linear in 1/d^2, and the same defensive normalisation the twinning statistics
// use: noise-only shells skipped, Wilson outliers rejected, the shell mean re-fitted once
// so the outlier does not corrupt the mean it is measured against.
double min_s = std::numeric_limits<double>::infinity(), max_s = -min_s;
for (int i : mod_idx) {
const double s = 1.0 / (merged[i].d * merged[i].d);
min_s = std::min(min_s, s);
max_s = std::max(max_s, s);
}
if (max_s > min_s) {
std::vector<std::vector<int>> bins(kModulationShells);
for (int i : mod_idx) {
const double t = (1.0 / (merged[i].d * merged[i].d) - min_s) / (max_s - min_s);
bins[std::clamp(static_cast<int>(t * kModulationShells), 0, kModulationShells - 1)]
.push_back(i);
}
for (const auto &b : bins) {
if (b.empty())
continue;
double isig = 0.0;
int n_isig = 0;
for (int i : b)
if (std::isfinite(merged[i].sigma) && merged[i].sigma > 0.0) {
isig += merged[i].I / merged[i].sigma;
++n_isig;
}
if (n_isig == 0 || isig / n_isig < kMinShellISig)
continue;
double sum = 0.0;
for (int i : b)
sum += merged[i].I;
double mean = sum / b.size();
if (mean <= 0.0)
continue;
sum = 0.0;
int kept = 0;
for (int i : b)
if (merged[i].I / mean <= kWilsonOutlierE2) { sum += merged[i].I; ++kept; }
if (kept == 0)
continue;
mean = sum / kept;
if (mean <= 0.0)
continue;
for (int i : b) {
const double v = merged[i].I / mean;
if (v > kWilsonOutlierE2)
continue;
mod_hkl.push_back({merged[i].h, merged[i].k, merged[i].l});
mod_e2.push_back(v);
}
}
}
}
}
result.n_modulation_reflections = static_cast<int>(mod_e2.size());
if (result.n_modulation_reflections < kMinModulationReflections)
return result;
const int floor_count = std::max(50, result.n_modulation_reflections / (4 * kPhaseBins));
auto contrast = [&](const std::array<double, 3> &u, std::vector<double> *profile) {
std::array<double, kPhaseBins> sum{}, cnt{};
sum.fill(0.0);
cnt.fill(0.0);
for (size_t i = 0; i < mod_e2.size(); ++i) {
const double t = mod_hkl[i][0] * u[0] + mod_hkl[i][1] * u[1] + mod_hkl[i][2] * u[2];
const double f = t - std::floor(t);
const int b = std::min(static_cast<int>(f * kPhaseBins), kPhaseBins - 1);
sum[b] += mod_e2[i];
cnt[b] += 1.0;
}
double lo = std::numeric_limits<double>::infinity(), hi = -lo;
int populated = 0;
if (profile)
profile->assign(kPhaseBins, std::numeric_limits<double>::quiet_NaN());
for (int b = 0; b < kPhaseBins; ++b) {
if (cnt[b] < floor_count)
continue;
const double m = sum[b] / cnt[b];
if (profile)
(*profile)[b] = m;
lo = std::min(lo, m);
hi = std::max(hi, m);
++populated;
}
if (populated < 2 || lo <= 0.0)
return std::numeric_limits<double>::quiet_NaN();
return hi / lo;
};
// Coarse-to-fine local search for the vector that modulates the data most strongly, started from
// the Patterson peak. The Patterson grid is coarse and a real pseudo-translation is rarely at an
// exactly rational position, so the peak position alone understates the modulation.
const std::array<double, 3> step0{1.0 / nx, 1.0 / ny, 1.0 / nz};
auto refine = [&](std::array<double, 3> u) {
std::array<double, 3> s = step0;
double best = contrast(u, nullptr);
for (int round = 0; round < 3; ++round) {
for (bool improved = true; improved;) {
improved = false;
for (int i = 0; i < 3; ++i)
for (double gstep : {-1.0, -0.5, 0.0, 0.5, 1.0}) {
std::array<double, 3> v = u;
v[i] += gstep * s[i];
const double c = contrast(v, nullptr);
if (std::isfinite(c) && c > best + 1e-6) {
best = c;
u = v;
improved = true;
}
}
}
for (double &x : s)
x /= 2.5;
}
return std::pair<std::array<double, 3>, double>{u, best};
};
const auto [u_refined, best_contrast] = refine(result.peak_frac);
if (!std::isfinite(best_contrast))
return result;
for (int i = 0; i < 3; ++i)
result.vector_frac[i] = u_refined[i] - std::floor(u_refined[i]);
result.modulation_measured = true;
result.modulation = best_contrast;
contrast(u_refined, &result.phase_bin_mean_e2);
result.vector_length_A = DistanceToOrigin(cell, Vec3{result.vector_frac[0], result.vector_frac[1],
result.vector_frac[2]},
std::vector<Vec3>{{0.0, 0.0, 0.0}});
// The modulation's own null: the SAME search from random starting vectors. A greedy search over
// three free parameters can manufacture contrast out of nothing, and this measures how much
// rather than assuming it is negligible.
{
std::mt19937 rng(seed + 7);
std::uniform_real_distribution<double> uni(0.0, 1.0);
std::vector<double> nulls;
for (int i = 0; i < kModulationNullDraws; ++i) {
const auto [v, c] = refine({uni(rng), uni(rng), uni(rng)});
(void) v;
nulls.push_back(std::isfinite(c) ? c : 1.0);
}
std::sort(nulls.begin(), nulls.end());
const size_t m = nulls.size();
result.modulation_null = m % 2 ? nulls[m / 2] : 0.5 * (nulls[m / 2 - 1] + nulls[m / 2]);
}
result.detected = result.peak_z >= kPeakZThreshold && result.modulation_null > 0.0 &&
result.modulation / result.modulation_null >= kModulationRatioThreshold;
result.near_extinct_class = result.detected && result.modulation >= kNearExtinctModulation;
// Is the vector a rational translation of the reported cell? If it is, the cell contains a
// sub-lattice: the pseudo-translation is a candidate for a REAL lattice translation the indexing
// missed, and the reported cell is a supercell candidate rather than a settled result.
for (int q = 2; q <= 6 && !result.commensurate; ++q) {
std::array<int, 3> p{};
double err = 0.0;
bool fractional = false;
for (int i = 0; i < 3; ++i) {
p[i] = static_cast<int>(std::lround(result.vector_frac[i] * q));
err = std::max(err, std::fabs(result.vector_frac[i] * q - p[i]));
if (p[i] % q != 0)
fractional = true;
}
if (err / q <= 0.05 && fractional) {
result.commensurate = true;
result.commensurate_denominator = q;
result.commensurate_numerator = p;
}
}
return result;
}
std::string TranslationalNCSToText(const TranslationalNCSResult &result) {
std::ostringstream os;
os << "Translational pseudo-symmetry\n";
if (!result.measurable) {
os << " Not measured: " << result.refusal << ".\n"
<< " => This is not a statement that the crystal has none.\n";
return os.str();
}
os << std::fixed;
for (const auto &t : result.undeclared_lattice_translations)
os << " The data are invariant under (" << std::setprecision(3) << t[0] << ", " << t[1]
<< ", " << t[2] << ") - the Patterson is as high there as at the origin, so that is a\n"
<< " LATTICE translation, not a pseudo-symmetry, and the space group in use does not\n"
<< " declare it. Either the lattice is centred and the merge is not, or the cell is a\n"
<< " supercell. Excluded, and the search continued past it.\n";
os << " Native Patterson off-origin peak " << std::setprecision(1) << result.peak_percent
<< "% of the origin";
if (result.vector_length_A > 0.0)
os << " at (" << std::setprecision(3) << result.vector_frac[0] << ", " << result.vector_frac[1]
<< ", " << result.vector_frac[2] << "), " << std::setprecision(1) << result.vector_length_A
<< " A";
os << "\n [z = " << std::setprecision(1) << result.peak_z << " against a permutation null of "
<< result.null_mean << " +- " << std::setprecision(2) << result.null_sd << "%]\n";
if (result.modulation_measured)
os << " Intensity modulation along that vector: " << std::setprecision(1) << result.modulation
<< "x between the strongest and weakest phase class [random-vector control "
<< result.modulation_null << "x]\n";
if (!result.modulation_measured) {
os << " The modulation test could not be run - too few acentric reflections beyond 4 A - so\n"
<< " the peak above stands alone.\n"
<< " => Inconclusive. A Patterson peak on its own is not enough to call a pseudo-symmetry\n"
<< " (peaks of this size occur without one), and this dataset cannot supply the second\n"
<< " half of the evidence.\n";
return os.str();
}
if (!result.detected) {
os << " => No translational pseudo-symmetry indicated.\n";
return os.str();
}
os << " => Translational pseudo-symmetry. Molecular replacement will need it declared - Phaser\n"
<< " reads a tNCS vector and corrects the likelihood for it; without that the search can\n"
<< " fail on data that are otherwise good.\n";
if (result.near_extinct_class)
os << " => The class this vector suppresses is not weak but very nearly EXTINCT, which is\n"
<< " what a lattice translation of a smaller cell looks like measured in a larger one.\n"
<< " Treat the reported cell as a supercell candidate rather than a settled result: a\n"
<< " cell with this translation as a lattice vector is the rival to check.\n";
else if (result.commensurate)
os << " => The vector is a 1/" << result.commensurate_denominator
<< " translation of the cell, so the crystal is pseudo-centred. The cell itself is not in\n"
<< " question - the suppressed class is weak, not absent, and a smaller cell would\n"
<< " contradict it.\n";
return os.str();
}