diagnostics: a translational pseudo-symmetry is detected and named
A pseudo-translation makes one class of reflections systematically strong and the complementary class weak. It is the classic predictor of a failed molecular replacement, it raises the second moment where twinning lowers it so each masks the other's test, and rugnux did not look for it at all. Look for it: a native Patterson from the merged intensities, the largest off-origin peak taken as a fraction of the origin, and the vector refined against the data. Both halves are scored against a null computed for that crystal rather than a bound taken from elsewhere - the peak against intensities permuted within resolution shells, the modulation depth against the same search started from random vectors - because the noise floor of the first runs from 1.35 to 17.7 per cent across the corpus, so no fixed threshold could mean the same thing twice. Requiring both is what keeps the false-positive rate down: either alone calls one crystal in ten. A translation the data are exactly invariant under is a lattice vector, not a pseudo-symmetry. Recognising that removes eleven false calls from ninety-five merges folded in P1 - each reading 83 to 102 per cent of origin against a genuine maximum of 62 - and adds one true positive whose pseudo-translation lies underneath its own centring and is unreachable without it. Report-only: it gates nothing and changes no reflection, no scale and no group. It fires on eight of a hundred and thirty-seven crystals, and on those the verdict moves from OK to WARNINGS, which is the point of it. Median cost 55 ms, worst 1.9 s on the largest merge here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
This commit is contained in:
1 parent
5b8ce26c83
commit
a4be5e6cd0
15 files changed
+1271
-5
No files matched your search
@@ -229,6 +229,16 @@ T. O. Yeates, "A statistic for local intensity differences: robustness to anisot
|
||||
pseudo-centering and utility for detecting twinning" (2003), Acta Cryst. D59, 1124-1130
|
||||
[doi:10.1107/S0907444903007947](https://doi.org/10.1107/S0907444903007947).
|
||||
|
||||
**Translational pseudo-symmetry** is detected from the native Patterson computed from the merged
|
||||
intensities, and the interpretation of an off-origin peak as a pseudo-translation between copies of
|
||||
the contents of the asymmetric unit - together with the modulation it puts on the intensities, which
|
||||
is the second half of the test here - is Read, Adams and McCoy's. Their fitted peak-height table is
|
||||
not used: the peak is scored against a per-dataset within-shell permutation null instead, because the
|
||||
noise floor of the statistic depends strongly on how many reflections a dataset has. R. J. Read,
|
||||
P. D. Adams and A. J. McCoy, "Intensity statistics in the presence of translational
|
||||
noncrystallographic symmetry" (2013), Acta Cryst. D69, 176-183
|
||||
[doi:10.1107/S0907444912045374](https://doi.org/10.1107/S0907444912045374).
|
||||
|
||||
**Diffraction anisotropy** — the description of the overall fall-off by a single anisotropic
|
||||
displacement tensor, its symmetry constraints, and the fact that only its deviatoric part is
|
||||
determined (the isotropic part being degenerate with the overall scale) are Sheriff and Hendrickson's.
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
* The rugnux results report is shorter and its sections are renumbered 1-5 with no gaps; `REPORT_VERSION` is 8.
|
||||
* `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file.
|
||||
* The rugnux results report omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off, instead of asserting a resolution the shell table refutes.
|
||||
* The rugnux results report measures translational pseudo-symmetry and reports it as `TNCS_DETECTED=` and the `TNCS_*` keys under the `PSEUDO_TRANSLATION` flag, reporting a translation the merged data are exactly invariant under as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead; `REPORT_VERSION` is 10.
|
||||
|
||||
* `rugnux` corrects each reflection for the sensor's quantum efficiency at the angle the diffracted beam meets the detector.
|
||||
* `rugnux` applies the sensor quantum-efficiency correction to still images processed on a GPU, which were left uncorrected while the same data on a CPU-only machine was corrected.
|
||||
|
||||
@@ -46,6 +46,7 @@ The methods draw on, and in places reimplement, solutions from:
|
||||
- P. Gasparotto, L. Barba, H.-C. Stadler et al., "TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography", *J. Appl. Cryst.* **57** (2024), 931-944 (the algorithm of the `ffbidx` fast-feedback indexer, §4).
|
||||
- I. Křivý & B. Gruber, "A unified algorithm for determining the reduced (Niggli) cell", *Acta Cryst.* **A32** (1976), 297-298, and International Tables for Crystallography Vol. A, Table 9.2.5.1 (the Niggli reduction and the lattice-character table of §5.3/§6).
|
||||
- J. E. Padilla & T. O. Yeates, "A statistic for local intensity differences: robustness to anisotropy and pseudo-centering and utility for detecting twinning", *Acta Cryst.* **D59** (2003), 1124-1130 (the L test, §13.2).
|
||||
- R. J. Read, P. D. Adams & A. J. McCoy, "Intensity statistics in the presence of translational noncrystallographic symmetry", *Acta Cryst.* **D69** (2013), 176-183 (the native-Patterson detection of translational pseudo-symmetry, and the intensity modulation it produces).
|
||||
- A. Barty, R. A. Kirian, F. R. N. C. Maia et al., "Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data", *J. Appl. Cryst.* **47** (2014), 1118-1131 (peakfinder8: the per-resolution-ring background statistics of §3.2).
|
||||
- A. Hennequin, B. Couturier, V. V. Gligorov & L. Lacassagne, "SparseCCL: Connected Components Labeling and Analysis for sparse images", DASIP 2019, 65-70 (the connected-component labelling of §3.4, used via ACTS/traccc).
|
||||
- S. French & K. Wilson, "On the treatment of negative intensity observations", *Acta Cryst.* **A34** (1978), 517-525 (Bayesian amplitude estimation from intensities).
|
||||
|
||||
+45
-2
@@ -116,8 +116,9 @@ and a run that produced a textbook data set read identically for their first thr
|
||||
- **`PATHOLOGY_FLAGS=`** is the *type* of each condition that fired, from a closed vocabulary, so a
|
||||
consumer switches on a code rather than parsing a sentence: `NO_LATTICE`, `INDEXING_AMBIGUITY`,
|
||||
`SYMMETRY_AMBIGUITY`, `CENTERING_UNTESTED`, `UNUSABLE_MERGE`, `LOW_COMPLETENESS`, `SWEEP_GAPS`,
|
||||
`GONIO_SCALE`, `SPINDLE_CAP`, `ANISOTROPY`, `TWINNING`, `MODEL_HAND`, `MODEL_NOT_VALIDATED`,
|
||||
`CANCELLED`, `RESOLUTION_FIT`. `NONE` when nothing fired. A code appears if and only if its
|
||||
`GONIO_SCALE`, `SPINDLE_CAP`, `ANISOTROPY`, `TWINNING`, `PSEUDO_TRANSLATION`,
|
||||
`LATTICE_TRANSLATION`, `MODEL_HAND`, `MODEL_NOT_VALIDATED`, `CANCELLED`, `RESOLUTION_FIT`,
|
||||
`FLIGHT_PATH`. `NONE` when nothing fired. A code appears if and only if its
|
||||
warning fired, so the flags and the `WARNING:` lines are two renderings of one list — the closed
|
||||
type for machinery, the open sentence for a person.
|
||||
- **`WARNING_COUNT=`** and the **`WARNING:`** lines follow, in the same section. They are what they
|
||||
@@ -187,6 +188,48 @@ scaled at all. Every range also appears as a `WARNING:` sentence in the SUMMARY.
|
||||
The same finding is written **per image** into the `_process.h5` as `/entry/MX/sweepQuality`, when
|
||||
one is written — see [HDF5](HDF5.md#entry-mx-spot-finding-and-indexing-cxi-style).
|
||||
|
||||
## Translational pseudo-symmetry
|
||||
|
||||
Two copies of the contents of the asymmetric unit related by a pure translation that is not a lattice
|
||||
vector. It is the pathology that most reliably breaks molecular replacement, because the modulation it
|
||||
puts on the intensities is not in the search model. Section 4 reports it beside twinning, because the
|
||||
two interact.
|
||||
|
||||
**`TNCS_DETECTED=`** is `TRUE`, `FALSE`, `INCONCLUSIVE` or `NOT_MEASURED`. The last two are not
|
||||
`FALSE`: `NOT_MEASURED` means the merge has too few reflections in 20–5 Å to compute the statistic at
|
||||
all, `INCONCLUSIVE` means the Patterson was measured but too few acentric reflections remain to test
|
||||
whether the vector it names modulates the intensities. Neither is a statement that the crystal has no
|
||||
pseudo-symmetry.
|
||||
|
||||
`TRUE` requires **both** of two tests, because either alone over-calls by about a factor of two:
|
||||
|
||||
- **`TNCS_PATTERSON_PEAK_PCT=`** — the largest off-origin peak of the native Patterson, as a
|
||||
percentage of the origin peak, counting only peaks farther than 15 Å from any origin-equivalent
|
||||
lattice point. **`TNCS_PATTERSON_PEAK_Z=`** scores it against **`TNCS_PATTERSON_NULL_PCT=`**, the
|
||||
same map recomputed with the intensities permuted within resolution shells. The null is per dataset
|
||||
and not a table: the noise floor of this statistic runs from about 1% on a large merge to about 18%
|
||||
on a small one, so no fixed percentage separates the two populations.
|
||||
- **`TNCS_MODULATION=`** — the ratio of the strongest to the weakest bin mean of ⟨E²⟩ over the phase
|
||||
frac(**h**·**u**), with **u** the refined peak vector (**`TNCS_VECTOR=`**, fractional, and
|
||||
**`TNCS_VECTOR_LENGTH=`** in Å). **`TNCS_MODULATION_NULL=`** is the same search started from random
|
||||
vectors, so the contrast the search itself can manufacture is measured rather than assumed.
|
||||
|
||||
The vector is good to about 0.05 fractional. It is a starting point for a program that refines it,
|
||||
not a refined result: the refinement maximises the modulation, not the accuracy of the vector.
|
||||
|
||||
**`TNCS_PSEUDO_CENTRED=`** and **`TNCS_SUBLATTICE=`** are separate claims and are deliberately not
|
||||
merged. A vector that is a rational translation `1/q` of the cell means the crystal is pseudo-centred;
|
||||
the cell itself is not in question, because the suppressed class is weak rather than absent and a
|
||||
smaller cell would contradict it. Only `TNCS_SUBLATTICE= NEAR_EXTINCT_CLASS` — the suppressed class
|
||||
almost gone — says the reported cell may be a supercell.
|
||||
|
||||
**`UNDECLARED_LATTICE_TRANSLATION=`** is a different finding, and is reported instead of a
|
||||
pseudo-symmetry rather than as one. The merged data are *exactly* invariant under the translation, and
|
||||
a translation the data are exactly invariant under is a lattice vector by definition — so the centring
|
||||
or the cell is wrong, not the packing. It is what a centred lattice merged in P1 looks like, which
|
||||
`--mode scale` on a file with no space group produces by design. The pseudo-symmetry search continues
|
||||
underneath it, so a real pseudo-translation sitting under an undeclared centring is still found.
|
||||
|
||||
## Diffraction anisotropy
|
||||
|
||||
Section 4 also reports how much the fall-off with resolution depends on **direction**, and whether that is
|
||||
|
||||
@@ -23,9 +23,11 @@ ADD_LIBRARY(JFJochScaleMerge
|
||||
FrenchWilson.h
|
||||
ReindexAmbiguity.cpp
|
||||
ReindexAmbiguity.h
|
||||
TranslationalNCS.cpp
|
||||
TranslationalNCS.h
|
||||
ScalingResult.h
|
||||
ScalingResult.cpp)
|
||||
TARGET_LINK_LIBRARIES(JFJochScaleMerge Ceres::ceres Eigen3::Eigen JFJochCommon)
|
||||
TARGET_LINK_LIBRARIES(JFJochScaleMerge Ceres::ceres Eigen3::Eigen JFJochCommon fftw3f)
|
||||
|
||||
IF (JFJOCH_CUDA_AVAILABLE)
|
||||
TARGET_SOURCES(JFJochScaleMerge PRIVATE ../indexing/CUDAMemHelpers.h
|
||||
|
||||
@@ -0,0 +1,616 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
#include "TranslationalNCS.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cstring>
|
||||
#include <iomanip>
|
||||
#include <limits>
|
||||
#include <numeric>
|
||||
#include <random>
|
||||
#include <sstream>
|
||||
#include <unordered_set>
|
||||
|
||||
#include <fftw3.h>
|
||||
|
||||
namespace {
|
||||
// The Patterson is measured in a band, not over the whole dataset: below ~20 A the bulk solvent
|
||||
// and the beam stop dominate, above ~5 A the map answers a question about atoms rather than about
|
||||
// the arrangement of molecules, which is what a pseudo-translation is.
|
||||
constexpr double kPattersonDMin = 5.0;
|
||||
constexpr double kPattersonDMax = 20.0;
|
||||
// A pseudo-translation shorter than this is a vector inside one molecule, not a translation
|
||||
// between two of them; the origin peak's own shoulder also reaches out this far.
|
||||
constexpr double kMinPeakDistanceA = 15.0;
|
||||
// A peak this high is not a pseudo-symmetry: the origin peak is the total, so a peak at this
|
||||
// fraction of it means essentially every reflection is invariant under the translation, which
|
||||
// makes it a lattice vector. Measured over 137 datasets the two populations do not overlap - the
|
||||
// strongest genuine pseudo-translation reaches 62% of the origin, while an undeclared centring
|
||||
// (a C-, I- or F-centred crystal merged in P1) reads 83-102%.
|
||||
constexpr double kLatticeTranslationPct = 75.0;
|
||||
constexpr double kGridSampling = 2.5; // grid spacing = d_min / this
|
||||
constexpr int kPattersonShells = 20; // equal-count shells for the E^2 normalisation
|
||||
constexpr int kMinPattersonReflections = 50;
|
||||
|
||||
constexpr double kModulationDMin = 4.0; // coarse enough that a small error in u does not
|
||||
// scramble the phase h.u
|
||||
constexpr int kModulationShells = 10;
|
||||
constexpr int kPhaseBins = 12;
|
||||
constexpr int kMinModulationReflections = 500;
|
||||
constexpr double kWilsonOutlierE2 = 8.0;
|
||||
constexpr double kMinShellISig = 1.0;
|
||||
// 20 rather than a handful: the null is the median of that many greedy refinements from random
|
||||
// starts, and with 6 draws its scatter was enough to move datasets sitting near the gate across
|
||||
// it from one RNG seed to the next (measured: three of them). The refinement is a dot product
|
||||
// per trial, so the extra draws cost a few ms.
|
||||
constexpr int kModulationNullDraws = 20;
|
||||
|
||||
// Gate constants. Read off a corpus of 128 merged datasets, on which the pair fires on 7.0% of
|
||||
// them - a pathology rate, and close to the ~6% of PDB entries Read, Adams & McCoy report with a
|
||||
// large off-origin Patterson peak. Either criterion ALONE fires on roughly twice as many: a 40%
|
||||
// peak can carry no modulation at all, and on a 70,000-reflection dataset a 4% peak reaches
|
||||
// z = 18 while modulating nothing.
|
||||
constexpr double kPeakZThreshold = 5.0;
|
||||
constexpr double kModulationRatioThreshold = 2.5;
|
||||
// A modulation this deep means the suppressed class carries under 5% of the other, i.e. it is
|
||||
// very nearly systematically absent - which is what a LATTICE translation of a smaller cell looks
|
||||
// like measured in the larger one. Over 137 datasets exactly one crystal reaches it (at 89x, the
|
||||
// next-deepest genuine pseudo-symmetry being 14x), and its deposited entry is indeed a cell of
|
||||
// half the volume, whose axis length this detector reports to within 0.2 A.
|
||||
constexpr double kNearExtinctModulation = 20.0;
|
||||
|
||||
// Above this many grid points the null is cut from 30 draws to 10. A deterministic function of
|
||||
// the cell, so the answer does not depend on the machine; no dataset in the corpus reaches it
|
||||
// (the largest grid measured was 2.5 M points on a 545 A cell).
|
||||
constexpr size_t kLargeGridPoints = 4u << 20;
|
||||
|
||||
struct Vec3 { double x, y, z; };
|
||||
|
||||
// Round up to the next even number whose only prime factors are 2, 3 and 5 - the sizes FFTW has
|
||||
// codelets for.
|
||||
int SmoothSize(int v) {
|
||||
v = std::max(2, v);
|
||||
for (;; ++v) {
|
||||
if (v % 2 != 0)
|
||||
continue;
|
||||
int x = v;
|
||||
for (int p : {2, 3, 5})
|
||||
while (x % p == 0)
|
||||
x /= p;
|
||||
if (x == 1)
|
||||
return v;
|
||||
}
|
||||
}
|
||||
|
||||
// Origin-equivalent points of the lattice: the origin plus the centring translations, which
|
||||
// produce a large Patterson peak by construction and are not pseudo-translations.
|
||||
std::vector<Vec3> OriginEquivalents(const gemmi::SpaceGroup *sg) {
|
||||
std::vector<Vec3> out{{0.0, 0.0, 0.0}};
|
||||
if (!sg)
|
||||
return out;
|
||||
for (const auto &t : sg->operations().cen_ops) {
|
||||
const Vec3 v{t[0] / 24.0, t[1] / 24.0, t[2] / 24.0};
|
||||
if (v.x != 0.0 || v.y != 0.0 || v.z != 0.0)
|
||||
out.push_back(v);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Shortest distance in A from a fractional position to any origin-equivalent lattice point.
|
||||
double DistanceToOrigin(const gemmi::UnitCell &cell, const Vec3 &frac,
|
||||
const std::vector<Vec3> &equivalents) {
|
||||
double best = std::numeric_limits<double>::infinity();
|
||||
for (const auto &c : equivalents)
|
||||
for (int dx = -1; dx <= 1; ++dx)
|
||||
for (int dy = -1; dy <= 1; ++dy)
|
||||
for (int dz = -1; dz <= 1; ++dz) {
|
||||
const gemmi::Position p = cell.orthogonalize(
|
||||
gemmi::Fractional{frac.x - c.x + dx, frac.y - c.y + dy, frac.z - c.z + dz});
|
||||
best = std::min(best, p.length());
|
||||
}
|
||||
return best;
|
||||
}
|
||||
}
|
||||
|
||||
TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector<MergedReflection> &merged,
|
||||
const gemmi::UnitCell &cell,
|
||||
const gemmi::SpaceGroup *space_group,
|
||||
int null_draws, uint32_t seed) {
|
||||
TranslationalNCSResult result;
|
||||
if (cell.a <= 0.0 || cell.b <= 0.0 || cell.c <= 0.0) {
|
||||
result.refusal = "no unit cell";
|
||||
return result;
|
||||
}
|
||||
|
||||
const gemmi::GroupOps gops = space_group ? space_group->operations() : gemmi::GroupOps{};
|
||||
auto acentric = [&](const MergedReflection &r) {
|
||||
return !space_group || !gops.is_reflection_centric(gemmi::Op::Miller{{r.h, r.k, r.l}});
|
||||
};
|
||||
|
||||
// ------------------------------------------------------------------ (a) the Patterson peak
|
||||
// Following the native-Patterson construction of Read, Adams & McCoy (2013) Acta Cryst. D69,
|
||||
// 176-183, with the fixed peak-height table replaced by a per-dataset permutation null: the
|
||||
// noise floor of this statistic runs from ~1% of the origin on a 74,000-reflection merge to ~18%
|
||||
// on a 138-reflection one, so no fixed percentage can serve both.
|
||||
std::vector<int> idx;
|
||||
for (size_t i = 0; i < merged.size(); ++i) {
|
||||
const auto &r = merged[i];
|
||||
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kPattersonDMin && r.d <= kPattersonDMax)
|
||||
idx.push_back(static_cast<int>(i));
|
||||
}
|
||||
result.n_reflections = static_cast<int>(idx.size());
|
||||
if (result.n_reflections < kMinPattersonReflections) {
|
||||
std::ostringstream os;
|
||||
os << "only " << result.n_reflections << " reflections between " << kPattersonDMax << " and "
|
||||
<< kPattersonDMin << " A - too few for a Patterson";
|
||||
result.refusal = os.str();
|
||||
return result;
|
||||
}
|
||||
|
||||
// E^2: divide each intensity by the mean of an equal-count resolution shell, so the map is not
|
||||
// dominated by the Wilson fall-off and the peak percentage is comparable between datasets.
|
||||
std::sort(idx.begin(), idx.end(),
|
||||
[&](int a, int b) { return merged[a].d != merged[b].d ? merged[a].d > merged[b].d : a < b; });
|
||||
const int n = result.n_reflections;
|
||||
std::vector<double> e2(n, 0.0);
|
||||
std::vector<int> shell_of(n, 0);
|
||||
for (int s = 0; s < kPattersonShells; ++s) {
|
||||
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
|
||||
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
|
||||
if (hi <= lo)
|
||||
continue;
|
||||
double sum = 0.0;
|
||||
for (int i = lo; i < hi; ++i)
|
||||
sum += merged[idx[i]].I;
|
||||
const double mean = sum / (hi - lo);
|
||||
for (int i = lo; i < hi; ++i) {
|
||||
e2[i] = mean != 0.0 ? merged[idx[i]].I / mean : 0.0;
|
||||
shell_of[i] = s;
|
||||
}
|
||||
}
|
||||
|
||||
// Grid. Sampled at d_min / 2.5, rounded up to an FFT-friendly size. A very large cell is measured
|
||||
// on a coarser map rather than refused: the map only has to resolve a peak 15 A from the origin.
|
||||
double patterson_dmin = kPattersonDMin;
|
||||
std::array<int, 3> g{};
|
||||
const double axis[3] = {cell.a, cell.b, cell.c};
|
||||
for (;;) {
|
||||
for (int i = 0; i < 3; ++i)
|
||||
g[i] = SmoothSize(static_cast<int>(std::ceil(axis[i] * kGridSampling / patterson_dmin)));
|
||||
const size_t points = static_cast<size_t>(g[0]) * g[1] * g[2];
|
||||
if (points <= (64u << 20) || patterson_dmin >= 12.0)
|
||||
break;
|
||||
patterson_dmin *= 1.25;
|
||||
}
|
||||
result.grid = g;
|
||||
const int nx = g[0], ny = g[1], nz = g[2];
|
||||
const int nzh = nz / 2 + 1;
|
||||
const size_t n_points = static_cast<size_t>(nx) * ny * nz;
|
||||
const size_t n_complex = static_cast<size_t>(nx) * ny * nzh;
|
||||
if (n_points > kLargeGridPoints)
|
||||
null_draws = std::min(null_draws, 10);
|
||||
|
||||
// Expand the asymmetric unit onto the full sphere: every rotation of the space group and both
|
||||
// Friedel mates, deduplicated. Only the half-space l' <= nz/2 is stored - the Patterson
|
||||
// coefficients are real and centrosymmetric, so the other half is what the c2r transform infers.
|
||||
std::vector<std::pair<size_t, int>> fill; // (index into the half-space grid, source reflection)
|
||||
fill.reserve(idx.size() * 8);
|
||||
{
|
||||
std::unordered_set<int64_t> seen;
|
||||
seen.reserve(idx.size() * 16);
|
||||
std::vector<gemmi::Op> ops;
|
||||
if (space_group)
|
||||
ops = gops.sym_ops;
|
||||
else
|
||||
ops.push_back(gemmi::Op::identity());
|
||||
for (const auto &op : ops)
|
||||
for (int sign : {1, -1})
|
||||
for (int i = 0; i < n; ++i) {
|
||||
const auto &r = merged[idx[i]];
|
||||
const gemmi::Op::Miller m = op.apply_to_hkl({{r.h, r.k, r.l}});
|
||||
const int64_t h = sign * m[0], k = sign * m[1], l = sign * m[2];
|
||||
const int64_t key = ((h + 1048576) << 42) | ((k + 1048576) << 21) | (l + 1048576);
|
||||
if (!seen.insert(key).second)
|
||||
continue;
|
||||
const int gl = static_cast<int>(((l % nz) + nz) % nz);
|
||||
if (gl > nz / 2)
|
||||
continue; // its Friedel mate is stored instead
|
||||
const int gh = static_cast<int>(((h % nx) + nx) % nx);
|
||||
const int gk = static_cast<int>(((k % ny) + ny) % ny);
|
||||
fill.emplace_back((static_cast<size_t>(gh) * ny + gk) * nzh + gl, i);
|
||||
}
|
||||
}
|
||||
|
||||
float *coeff = fftwf_alloc_real(2 * n_complex);
|
||||
float *map = fftwf_alloc_real(n_points);
|
||||
if (coeff == nullptr || map == nullptr) {
|
||||
if (coeff) fftwf_free(coeff);
|
||||
if (map) fftwf_free(map);
|
||||
result.refusal = "could not allocate the Patterson grid";
|
||||
return result;
|
||||
}
|
||||
fftwf_plan plan = fftwf_plan_dft_c2r_3d(nx, ny, nz, reinterpret_cast<fftwf_complex *>(coeff), map,
|
||||
FFTW_ESTIMATE);
|
||||
|
||||
// The near-origin exclusion, as a set of grid points rather than a per-point distance test: only
|
||||
// the points inside a 15 A ball around each origin-equivalent point can be excluded, and there
|
||||
// are a few thousand of those however large the grid is.
|
||||
std::vector<uint8_t> near(n_points, 0);
|
||||
std::vector<Vec3> equivalents = OriginEquivalents(space_group);
|
||||
auto build_near_mask = [&]() {
|
||||
std::fill(near.begin(), near.end(), 0);
|
||||
const gemmi::UnitCell &c = cell;
|
||||
// Fractional half-width of a sphere of radius R along axis i is R * |a*_i|, so the ball is
|
||||
// enumerated directly instead of testing every grid point: the cost is a few thousand points
|
||||
// however large the grid is.
|
||||
const double rstar[3] = {c.ar, c.br, c.cr};
|
||||
for (const auto &e : equivalents) {
|
||||
const int span[3] = {static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[0] * nx)) + 1,
|
||||
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[1] * ny)) + 1,
|
||||
static_cast<int>(std::ceil(kMinPeakDistanceA * rstar[2] * nz)) + 1};
|
||||
const int c0[3] = {static_cast<int>(std::lround(e.x * nx)),
|
||||
static_cast<int>(std::lround(e.y * ny)),
|
||||
static_cast<int>(std::lround(e.z * nz))};
|
||||
for (int i = c0[0] - span[0]; i <= c0[0] + span[0]; ++i)
|
||||
for (int j = c0[1] - span[1]; j <= c0[1] + span[1]; ++j)
|
||||
for (int k = c0[2] - span[2]; k <= c0[2] + span[2]; ++k) {
|
||||
const gemmi::Position p = cell.orthogonalize(gemmi::Fractional{
|
||||
static_cast<double>(i) / nx - e.x, static_cast<double>(j) / ny - e.y,
|
||||
static_cast<double>(k) / nz - e.z});
|
||||
if (p.length() >= kMinPeakDistanceA)
|
||||
continue;
|
||||
const size_t gi = static_cast<size_t>(((i % nx) + nx) % nx);
|
||||
const size_t gj = static_cast<size_t>(((j % ny) + ny) % ny);
|
||||
const size_t gk = static_cast<size_t>(((k % nz) + nz) % nz);
|
||||
near[(gi * ny + gj) * nz + gk] = 1;
|
||||
}
|
||||
}
|
||||
};
|
||||
build_near_mask();
|
||||
|
||||
auto transform = [&](const std::vector<double> &values) {
|
||||
std::memset(coeff, 0, 2 * n_complex * sizeof(float));
|
||||
for (const auto &[at, src] : fill)
|
||||
coeff[2 * at] += static_cast<float>(values[src]);
|
||||
fftwf_execute(plan);
|
||||
};
|
||||
auto max_far = [&]() {
|
||||
double best = -std::numeric_limits<double>::infinity();
|
||||
for (size_t i = 0; i < n_points; ++i)
|
||||
if (!near[i] && map[i] > best)
|
||||
best = map[i];
|
||||
return best;
|
||||
};
|
||||
|
||||
transform(e2);
|
||||
const double origin = map[0];
|
||||
|
||||
// The vector to report is the largest off-origin LOCAL maximum, which is not always the largest
|
||||
// off-origin grid value: the tail of the origin peak can be higher right at the 15 A boundary,
|
||||
// and that is a shoulder, not a peak.
|
||||
auto top_local_max = [&]() {
|
||||
double best = -std::numeric_limits<double>::infinity();
|
||||
std::array<double, 3> at_frac{{0.0, 0.0, 0.0}};
|
||||
for (int i = 0; i < nx; ++i)
|
||||
for (int j = 0; j < ny; ++j)
|
||||
for (int k = 0; k < nz; ++k) {
|
||||
const size_t at = (static_cast<size_t>(i) * ny + j) * nz + k;
|
||||
if (near[at] || map[at] <= best)
|
||||
continue;
|
||||
bool is_max = true;
|
||||
for (int dx = -1; dx <= 1 && is_max; ++dx)
|
||||
for (int dy = -1; dy <= 1 && is_max; ++dy)
|
||||
for (int dz = -1; dz <= 1; ++dz) {
|
||||
if (dx == 0 && dy == 0 && dz == 0)
|
||||
continue;
|
||||
const size_t nb = (static_cast<size_t>((i + dx + nx) % nx) * ny
|
||||
+ (j + dy + ny) % ny) * nz + (k + dz + nz) % nz;
|
||||
if (map[nb] > map[at]) { is_max = false; break; }
|
||||
}
|
||||
if (is_max) {
|
||||
best = map[at];
|
||||
at_frac = {static_cast<double>(i) / nx, static_cast<double>(j) / ny,
|
||||
static_cast<double>(k) / nz};
|
||||
}
|
||||
}
|
||||
return std::pair<std::array<double, 3>, double>{at_frac, best};
|
||||
};
|
||||
|
||||
// Absorb any translation the data are invariant under - a centring the space group in use does
|
||||
// not declare - into the exclusion, and look again underneath it. Without this, a C-centred
|
||||
// crystal merged in P1 reports its own centring vector as a pseudo-symmetry (measured: 11 of the
|
||||
// 95 P1 merges in the corpus), and a genuine pseudo-translation hiding below a centring vector is
|
||||
// never reached.
|
||||
double observed = 0.0;
|
||||
for (int pass = 0; pass < 5; ++pass) {
|
||||
observed = max_far();
|
||||
result.peak_percent = origin > 0.0 ? 100.0 * observed / origin : 0.0;
|
||||
const auto [frac, height] = top_local_max();
|
||||
result.peak_frac = frac;
|
||||
if (result.peak_percent < kLatticeTranslationPct || !std::isfinite(height))
|
||||
break;
|
||||
result.undeclared_lattice_translations.push_back(frac);
|
||||
equivalents.push_back(Vec3{frac[0], frac[1], frac[2]});
|
||||
build_near_mask();
|
||||
}
|
||||
|
||||
// Permutation null: the same reflections, the same map, with the intensities shuffled inside each
|
||||
// resolution shell. Everything that could make a Patterson peak except the correlation between
|
||||
// reflections is preserved, so this is the noise floor of THIS dataset.
|
||||
{
|
||||
std::mt19937 rng(seed);
|
||||
std::vector<double> permuted = e2;
|
||||
std::vector<double> nulls;
|
||||
nulls.reserve(null_draws);
|
||||
for (int draw = 0; draw < null_draws; ++draw) {
|
||||
permuted = e2;
|
||||
for (int s = 0; s < kPattersonShells; ++s) {
|
||||
const int lo = static_cast<int>(static_cast<int64_t>(s) * n / kPattersonShells);
|
||||
const int hi = static_cast<int>(static_cast<int64_t>(s + 1) * n / kPattersonShells);
|
||||
for (int i = hi - 1; i > lo; --i)
|
||||
std::swap(permuted[i], permuted[lo + static_cast<int>(rng() % (i - lo + 1))]);
|
||||
}
|
||||
transform(permuted);
|
||||
if (map[0] > 0.0)
|
||||
nulls.push_back(100.0 * max_far() / map[0]);
|
||||
}
|
||||
if (!nulls.empty()) {
|
||||
const double mean = std::accumulate(nulls.begin(), nulls.end(), 0.0) / nulls.size();
|
||||
double var = 0.0;
|
||||
for (double v : nulls)
|
||||
var += (v - mean) * (v - mean);
|
||||
var /= nulls.size();
|
||||
result.null_mean = mean;
|
||||
result.null_sd = std::sqrt(var);
|
||||
result.peak_z = result.null_sd > 0.0 ? (result.peak_percent - mean) / result.null_sd : 0.0;
|
||||
}
|
||||
}
|
||||
|
||||
fftwf_destroy_plan(plan);
|
||||
fftwf_free(coeff);
|
||||
fftwf_free(map);
|
||||
result.measurable = true;
|
||||
|
||||
// ------------------------------------------------------------------ (b) the modulation depth
|
||||
// A pure translation u between two copies multiplies the intensity by |1 + exp(2 pi i h.u)|^2, so
|
||||
// <I> depends on the phase h.u: strong where it is near an integer, extinct near a half-integer.
|
||||
// The Patterson peak says a vector exists; this says whether it modulates the data.
|
||||
std::vector<std::array<int, 3>> mod_hkl;
|
||||
std::vector<double> mod_e2;
|
||||
{
|
||||
std::vector<int> mod_idx;
|
||||
for (size_t i = 0; i < merged.size(); ++i) {
|
||||
const auto &r = merged[i];
|
||||
if (std::isfinite(r.I) && std::isfinite(r.d) && r.d >= kModulationDMin && acentric(r))
|
||||
mod_idx.push_back(static_cast<int>(i));
|
||||
}
|
||||
if (!mod_idx.empty()) {
|
||||
// Shells linear in 1/d^2, and the same defensive normalisation the twinning statistics
|
||||
// use: noise-only shells skipped, Wilson outliers rejected, the shell mean re-fitted once
|
||||
// so the outlier does not corrupt the mean it is measured against.
|
||||
double min_s = std::numeric_limits<double>::infinity(), max_s = -min_s;
|
||||
for (int i : mod_idx) {
|
||||
const double s = 1.0 / (merged[i].d * merged[i].d);
|
||||
min_s = std::min(min_s, s);
|
||||
max_s = std::max(max_s, s);
|
||||
}
|
||||
if (max_s > min_s) {
|
||||
std::vector<std::vector<int>> bins(kModulationShells);
|
||||
for (int i : mod_idx) {
|
||||
const double t = (1.0 / (merged[i].d * merged[i].d) - min_s) / (max_s - min_s);
|
||||
bins[std::clamp(static_cast<int>(t * kModulationShells), 0, kModulationShells - 1)]
|
||||
.push_back(i);
|
||||
}
|
||||
for (const auto &b : bins) {
|
||||
if (b.empty())
|
||||
continue;
|
||||
double isig = 0.0;
|
||||
int n_isig = 0;
|
||||
for (int i : b)
|
||||
if (std::isfinite(merged[i].sigma) && merged[i].sigma > 0.0) {
|
||||
isig += merged[i].I / merged[i].sigma;
|
||||
++n_isig;
|
||||
}
|
||||
if (n_isig == 0 || isig / n_isig < kMinShellISig)
|
||||
continue;
|
||||
double sum = 0.0;
|
||||
for (int i : b)
|
||||
sum += merged[i].I;
|
||||
double mean = sum / b.size();
|
||||
if (mean <= 0.0)
|
||||
continue;
|
||||
sum = 0.0;
|
||||
int kept = 0;
|
||||
for (int i : b)
|
||||
if (merged[i].I / mean <= kWilsonOutlierE2) { sum += merged[i].I; ++kept; }
|
||||
if (kept == 0)
|
||||
continue;
|
||||
mean = sum / kept;
|
||||
if (mean <= 0.0)
|
||||
continue;
|
||||
for (int i : b) {
|
||||
const double v = merged[i].I / mean;
|
||||
if (v > kWilsonOutlierE2)
|
||||
continue;
|
||||
mod_hkl.push_back({merged[i].h, merged[i].k, merged[i].l});
|
||||
mod_e2.push_back(v);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
result.n_modulation_reflections = static_cast<int>(mod_e2.size());
|
||||
if (result.n_modulation_reflections < kMinModulationReflections)
|
||||
return result;
|
||||
|
||||
const int floor_count = std::max(50, result.n_modulation_reflections / (4 * kPhaseBins));
|
||||
auto contrast = [&](const std::array<double, 3> &u, std::vector<double> *profile) {
|
||||
std::array<double, kPhaseBins> sum{}, cnt{};
|
||||
sum.fill(0.0);
|
||||
cnt.fill(0.0);
|
||||
for (size_t i = 0; i < mod_e2.size(); ++i) {
|
||||
const double t = mod_hkl[i][0] * u[0] + mod_hkl[i][1] * u[1] + mod_hkl[i][2] * u[2];
|
||||
const double f = t - std::floor(t);
|
||||
const int b = std::min(static_cast<int>(f * kPhaseBins), kPhaseBins - 1);
|
||||
sum[b] += mod_e2[i];
|
||||
cnt[b] += 1.0;
|
||||
}
|
||||
double lo = std::numeric_limits<double>::infinity(), hi = -lo;
|
||||
int populated = 0;
|
||||
if (profile)
|
||||
profile->assign(kPhaseBins, std::numeric_limits<double>::quiet_NaN());
|
||||
for (int b = 0; b < kPhaseBins; ++b) {
|
||||
if (cnt[b] < floor_count)
|
||||
continue;
|
||||
const double m = sum[b] / cnt[b];
|
||||
if (profile)
|
||||
(*profile)[b] = m;
|
||||
lo = std::min(lo, m);
|
||||
hi = std::max(hi, m);
|
||||
++populated;
|
||||
}
|
||||
if (populated < 2 || lo <= 0.0)
|
||||
return std::numeric_limits<double>::quiet_NaN();
|
||||
return hi / lo;
|
||||
};
|
||||
|
||||
// Coarse-to-fine local search for the vector that modulates the data most strongly, started from
|
||||
// the Patterson peak. The Patterson grid is coarse and a real pseudo-translation is rarely at an
|
||||
// exactly rational position, so the peak position alone understates the modulation.
|
||||
const std::array<double, 3> step0{1.0 / nx, 1.0 / ny, 1.0 / nz};
|
||||
auto refine = [&](std::array<double, 3> u) {
|
||||
std::array<double, 3> s = step0;
|
||||
double best = contrast(u, nullptr);
|
||||
for (int round = 0; round < 3; ++round) {
|
||||
for (bool improved = true; improved;) {
|
||||
improved = false;
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (double gstep : {-1.0, -0.5, 0.0, 0.5, 1.0}) {
|
||||
std::array<double, 3> v = u;
|
||||
v[i] += gstep * s[i];
|
||||
const double c = contrast(v, nullptr);
|
||||
if (std::isfinite(c) && c > best + 1e-6) {
|
||||
best = c;
|
||||
u = v;
|
||||
improved = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
for (double &x : s)
|
||||
x /= 2.5;
|
||||
}
|
||||
return std::pair<std::array<double, 3>, double>{u, best};
|
||||
};
|
||||
|
||||
const auto [u_refined, best_contrast] = refine(result.peak_frac);
|
||||
if (!std::isfinite(best_contrast))
|
||||
return result;
|
||||
for (int i = 0; i < 3; ++i)
|
||||
result.vector_frac[i] = u_refined[i] - std::floor(u_refined[i]);
|
||||
result.modulation_measured = true;
|
||||
result.modulation = best_contrast;
|
||||
contrast(u_refined, &result.phase_bin_mean_e2);
|
||||
result.vector_length_A = DistanceToOrigin(cell, Vec3{result.vector_frac[0], result.vector_frac[1],
|
||||
result.vector_frac[2]},
|
||||
std::vector<Vec3>{{0.0, 0.0, 0.0}});
|
||||
|
||||
// The modulation's own null: the SAME search from random starting vectors. A greedy search over
|
||||
// three free parameters can manufacture contrast out of nothing, and this measures how much
|
||||
// rather than assuming it is negligible.
|
||||
{
|
||||
std::mt19937 rng(seed + 7);
|
||||
std::uniform_real_distribution<double> uni(0.0, 1.0);
|
||||
std::vector<double> nulls;
|
||||
for (int i = 0; i < kModulationNullDraws; ++i) {
|
||||
const auto [v, c] = refine({uni(rng), uni(rng), uni(rng)});
|
||||
(void) v;
|
||||
nulls.push_back(std::isfinite(c) ? c : 1.0);
|
||||
}
|
||||
std::sort(nulls.begin(), nulls.end());
|
||||
const size_t m = nulls.size();
|
||||
result.modulation_null = m % 2 ? nulls[m / 2] : 0.5 * (nulls[m / 2 - 1] + nulls[m / 2]);
|
||||
}
|
||||
|
||||
result.detected = result.peak_z >= kPeakZThreshold && result.modulation_null > 0.0 &&
|
||||
result.modulation / result.modulation_null >= kModulationRatioThreshold;
|
||||
|
||||
result.near_extinct_class = result.detected && result.modulation >= kNearExtinctModulation;
|
||||
|
||||
// Is the vector a rational translation of the reported cell? If it is, the cell contains a
|
||||
// sub-lattice: the pseudo-translation is a candidate for a REAL lattice translation the indexing
|
||||
// missed, and the reported cell is a supercell candidate rather than a settled result.
|
||||
for (int q = 2; q <= 6 && !result.commensurate; ++q) {
|
||||
std::array<int, 3> p{};
|
||||
double err = 0.0;
|
||||
bool fractional = false;
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
p[i] = static_cast<int>(std::lround(result.vector_frac[i] * q));
|
||||
err = std::max(err, std::fabs(result.vector_frac[i] * q - p[i]));
|
||||
if (p[i] % q != 0)
|
||||
fractional = true;
|
||||
}
|
||||
if (err / q <= 0.05 && fractional) {
|
||||
result.commensurate = true;
|
||||
result.commensurate_denominator = q;
|
||||
result.commensurate_numerator = p;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
std::string TranslationalNCSToText(const TranslationalNCSResult &result) {
|
||||
std::ostringstream os;
|
||||
os << "Translational pseudo-symmetry\n";
|
||||
if (!result.measurable) {
|
||||
os << " Not measured: " << result.refusal << ".\n"
|
||||
<< " => This is not a statement that the crystal has none.\n";
|
||||
return os.str();
|
||||
}
|
||||
os << std::fixed;
|
||||
for (const auto &t : result.undeclared_lattice_translations)
|
||||
os << " The data are invariant under (" << std::setprecision(3) << t[0] << ", " << t[1]
|
||||
<< ", " << t[2] << ") - the Patterson is as high there as at the origin, so that is a\n"
|
||||
<< " LATTICE translation, not a pseudo-symmetry, and the space group in use does not\n"
|
||||
<< " declare it. Either the lattice is centred and the merge is not, or the cell is a\n"
|
||||
<< " supercell. Excluded, and the search continued past it.\n";
|
||||
os << " Native Patterson off-origin peak " << std::setprecision(1) << result.peak_percent
|
||||
<< "% of the origin";
|
||||
if (result.vector_length_A > 0.0)
|
||||
os << " at (" << std::setprecision(3) << result.vector_frac[0] << ", " << result.vector_frac[1]
|
||||
<< ", " << result.vector_frac[2] << "), " << std::setprecision(1) << result.vector_length_A
|
||||
<< " A";
|
||||
os << "\n [z = " << std::setprecision(1) << result.peak_z << " against a permutation null of "
|
||||
<< result.null_mean << " +- " << std::setprecision(2) << result.null_sd << "%]\n";
|
||||
if (result.modulation_measured)
|
||||
os << " Intensity modulation along that vector: " << std::setprecision(1) << result.modulation
|
||||
<< "x between the strongest and weakest phase class [random-vector control "
|
||||
<< result.modulation_null << "x]\n";
|
||||
if (!result.modulation_measured) {
|
||||
os << " The modulation test could not be run - too few acentric reflections beyond 4 A - so\n"
|
||||
<< " the peak above stands alone.\n"
|
||||
<< " => Inconclusive. A Patterson peak on its own is not enough to call a pseudo-symmetry\n"
|
||||
<< " (peaks of this size occur without one), and this dataset cannot supply the second\n"
|
||||
<< " half of the evidence.\n";
|
||||
return os.str();
|
||||
}
|
||||
if (!result.detected) {
|
||||
os << " => No translational pseudo-symmetry indicated.\n";
|
||||
return os.str();
|
||||
}
|
||||
os << " => Translational pseudo-symmetry. Molecular replacement will need it declared - Phaser\n"
|
||||
<< " reads a tNCS vector and corrects the likelihood for it; without that the search can\n"
|
||||
<< " fail on data that are otherwise good.\n";
|
||||
if (result.near_extinct_class)
|
||||
os << " => The class this vector suppresses is not weak but very nearly EXTINCT, which is\n"
|
||||
<< " what a lattice translation of a smaller cell looks like measured in a larger one.\n"
|
||||
<< " Treat the reported cell as a supercell candidate rather than a settled result: a\n"
|
||||
<< " cell with this translation as a lattice vector is the rival to check.\n";
|
||||
else if (result.commensurate)
|
||||
os << " => The vector is a 1/" << result.commensurate_denominator
|
||||
<< " translation of the cell, so the crystal is pseudo-centred. The cell itself is not in\n"
|
||||
<< " question - the suppressed class is weak, not absent, and a smaller cell would\n"
|
||||
<< " contradict it.\n";
|
||||
return os.str();
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <array>
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "../../common/Reflection.h"
|
||||
#include "gemmi/symmetry.hpp"
|
||||
#include "gemmi/unitcell.hpp"
|
||||
|
||||
// Translational non-crystallographic symmetry (tNCS): two copies of the contents of the asymmetric
|
||||
// unit related by a pure translation that is not a lattice vector. It is the pathology that most
|
||||
// reliably breaks molecular replacement, because the modulation it puts on the intensities is not in
|
||||
// the search model, and it is invisible in every statistic this program reported until now.
|
||||
//
|
||||
// TWO numbers, and BOTH must fire. Measured over a corpus of merged datasets, either one alone
|
||||
// over-calls by roughly a factor of two.
|
||||
//
|
||||
// - PATTERSON PEAK. The largest off-origin peak of the native Patterson computed from the merged
|
||||
// intensities (shell-normalised E^2 coefficients over 20-5 A, one inverse FFT), as a percentage
|
||||
// of the origin peak, counting only peaks farther than 15 A from ANY origin-equivalent lattice
|
||||
// point so the origin's own shoulder and the centring vectors cannot be mistaken for a
|
||||
// pseudo-translation. Answers: is there a peak above this dataset's own noise floor?
|
||||
//
|
||||
// - MODULATION DEPTH. With that peak vector refined against the intensities, the acentric
|
||||
// reflections binned by the phase frac(h.u): a real pseudo-translation makes <E^2> follow a
|
||||
// cosine with a deep minimum near phase 1/2, an artefact leaves it flat. Reported as the ratio
|
||||
// of the strongest to the weakest bin mean. Answers: does that vector actually modulate the
|
||||
// intensities - i.e. is it a translation between real copies, or a feature of the map?
|
||||
//
|
||||
// NULLS ARE PER DATASET, not a table. The Patterson peak is scored against the same map recomputed
|
||||
// with the intensities permuted WITHIN resolution shells (same count, same completeness, same
|
||||
// resolution range, same intensity multiset; the inter-reflection correlation that makes a Patterson
|
||||
// peak destroyed), which is the only way to compare datasets whose noise floor runs from ~1% to ~18%
|
||||
// depending on how many reflections they have. The modulation is scored against the same refinement
|
||||
// started from random vectors, so the contrast the search itself can manufacture is measured rather
|
||||
// than assumed.
|
||||
//
|
||||
// REPORTING ONLY. Nothing here removes a reflection, changes a merge, or gates a candidate. Its one
|
||||
// effect on another verdict is to hand the twinning analysis the vector it needs in order not to
|
||||
// misread its own L-test - see the partner-step note in TwinningAnalysis.cpp. That direction only
|
||||
// ever REMOVES a spurious twin veto; it never adds one.
|
||||
//
|
||||
// Method credit is at the algorithm in the .cpp.
|
||||
|
||||
struct TranslationalNCSResult {
|
||||
// ---- was the statistic computable at all -------------------------------------------------
|
||||
// False is NOT "no tNCS": small-molecule cells have single-digit reflection counts in the 20-5 A
|
||||
// band and cannot answer the question either way. `refusal` says which case this is.
|
||||
bool measurable = false;
|
||||
std::string refusal;
|
||||
|
||||
int n_reflections = 0; // reflections entering the Patterson, 20-5 A
|
||||
std::array<int, 3> grid{{0, 0, 0}};
|
||||
|
||||
// ---- (a) Patterson peak -------------------------------------------------------------------
|
||||
double peak_percent = 0.0; // largest off-origin peak, % of the origin peak
|
||||
double null_mean = 0.0; // permutation-null mean of the same quantity
|
||||
double null_sd = 0.0;
|
||||
double peak_z = 0.0; // (peak - null_mean) / null_sd
|
||||
std::array<double, 3> peak_frac{{0.0, 0.0, 0.0}}; // grid-resolution peak position
|
||||
|
||||
// Translations at which the Patterson is as high as at the origin: every reflection is invariant
|
||||
// under them, which is the definition of a LATTICE translation, not of a pseudo-symmetry. The
|
||||
// centring vectors of the space group are excluded from the search to begin with; anything found
|
||||
// here is a lattice translation the space group in use does NOT declare - a centred lattice
|
||||
// merged in P1, or a cell that is a supercell of the true one. They are absorbed into the
|
||||
// exclusion and the search continues past them, so a real pseudo-translation sitting underneath
|
||||
// one is still found.
|
||||
std::vector<std::array<double, 3>> undeclared_lattice_translations;
|
||||
|
||||
// ---- (b) modulation depth -----------------------------------------------------------------
|
||||
// False when the Patterson was measured but the modulation test could not run (too few acentric
|
||||
// reflections beyond 4 A). `detected` is then FALSE for want of the second half of the evidence,
|
||||
// which is not the same claim as "the modulation was measured and was flat" - the report says
|
||||
// INCONCLUSIVE rather than FALSE.
|
||||
bool modulation_measured = false;
|
||||
int n_modulation_reflections = 0;
|
||||
std::array<double, 3> vector_frac{{0.0, 0.0, 0.0}}; // peak vector refined against the intensities
|
||||
double vector_length_A = 0.0; // shortest lattice-equivalent length of that vector
|
||||
double modulation = 0.0; // max/min of <E^2> over the phase bins
|
||||
double modulation_null = 0.0; // same search from random vectors (median)
|
||||
std::vector<double> phase_bin_mean_e2; // the profile itself, for the report
|
||||
|
||||
// ---- verdict -------------------------------------------------------------------------------
|
||||
bool detected = false; // both criteria fire
|
||||
|
||||
// The refined vector is a rational translation p/q of the reported cell: the crystal is
|
||||
// pseudo-centred. This is NOT by itself a statement that the cell is wrong - the suppressed class
|
||||
// is present, merely weak, and a smaller cell would contradict it. `near_extinct_class` below is
|
||||
// the one that says the cell may be a supercell.
|
||||
bool commensurate = false;
|
||||
int commensurate_denominator = 0; // q
|
||||
std::array<int, 3> commensurate_numerator{{0, 0, 0}}; // p
|
||||
|
||||
// The weak phase class is not weak but very nearly EXTINCT. A translation that leaves essentially
|
||||
// no intensity in the class it suppresses is very nearly a lattice translation, whether or not it
|
||||
// lands on a simple fraction of the reported axes - which it need not, if the true sub-cell is
|
||||
// triclinic. Same conclusion as `commensurate` and reported the same way: the cell is a supercell
|
||||
// candidate.
|
||||
bool near_extinct_class = false;
|
||||
};
|
||||
|
||||
// The vector to hand to AnalyzeTwinning when `detected`, so the L-test can choose partner steps that
|
||||
// preserve this pseudo-translation's class. Null when there is nothing to declare.
|
||||
inline const std::array<double, 3> *TranslationalNCSVectorForLTest(const TranslationalNCSResult &r) {
|
||||
return r.detected ? &r.vector_frac : nullptr;
|
||||
}
|
||||
|
||||
TranslationalNCSResult AnalyzeTranslationalNCS(const std::vector<MergedReflection> &merged,
|
||||
const gemmi::UnitCell &cell,
|
||||
const gemmi::SpaceGroup *space_group = nullptr,
|
||||
int null_draws = 30,
|
||||
uint32_t seed = 0);
|
||||
|
||||
std::string TranslationalNCSToText(const TranslationalNCSResult &result);
|
||||
@@ -37,6 +37,11 @@ namespace PathologyCode {
|
||||
constexpr const char *CANCELLED = "CANCELLED"; // the run did not finish
|
||||
constexpr const char *RESOLUTION_FIT = "RESOLUTION_FIT"; // the CC1/2 fall-off is not a fall-off
|
||||
constexpr const char *FLIGHT_PATH = "FLIGHT_PATH"; // the assumed flight-path medium matters here
|
||||
constexpr const char *PSEUDO_TRANSLATION = "PSEUDO_TRANSLATION"; // translational pseudo-symmetry indicated
|
||||
// Both warnings under this code say the same thing to a consumer - a translation the cell in use
|
||||
// does not declare is present in the data, so the centring or the cell is wrong - and differ only
|
||||
// in how exact it is. ANISOTROPY and RESOLUTION_FIT are already shared this way.
|
||||
constexpr const char *LATTICE_TRANSLATION = "LATTICE_TRANSLATION"; // a translation the cell does not declare
|
||||
}
|
||||
|
||||
// A closed type plus an open description. The type is what machinery switches on; the text is what a
|
||||
|
||||
+135
-1
@@ -15,6 +15,7 @@
|
||||
#include "../image_analysis/scale_merge/AnisotropyAnalysis.h"
|
||||
#include "../image_analysis/scale_merge/Merge.h"
|
||||
#include "../image_analysis/scale_merge/SearchSpaceGroup.h"
|
||||
#include "../image_analysis/scale_merge/TranslationalNCS.h"
|
||||
#include "../image_analysis/scale_merge/TwinningAnalysis.h"
|
||||
|
||||
#include "ResultReport.h"
|
||||
@@ -26,7 +27,10 @@ namespace {
|
||||
// pipeline-internal keys and the long explanatory blocks moved behind --developer.
|
||||
// 9: FLIGHT_PATH / FLIGHT_PATH_WILSON_B in section 1 - the medium the beam flew through, which
|
||||
// no file states and the run therefore assumes, and what assuming it was worth.
|
||||
constexpr int REPORT_VERSION = 9;
|
||||
// 10: the TNCS_* block in section 4 - translational pseudo-symmetry, which nothing measured
|
||||
// before, and the UNDECLARED_LATTICE_TRANSLATION it reports instead where the data are
|
||||
// exactly invariant under the vector it found.
|
||||
constexpr int REPORT_VERSION = 10;
|
||||
|
||||
const char *BANNER = " ******************************************************************************";
|
||||
|
||||
@@ -684,6 +688,131 @@ ReportDocument BuildReportDocument(const std::string &output_prefix,
|
||||
ReportSection s;
|
||||
s.title = "4. DIAGNOSTICS";
|
||||
|
||||
// ---- translational pseudo-symmetry
|
||||
// Printed immediately before the twinning block, and in the same section, because the two
|
||||
// interact: a pseudo-translation that is not a half-integer vector biases <|L|> upwards and
|
||||
// so, left unstated, silently overrules the twin verdict below. They are deliberately NOT
|
||||
// combined into one statistic - measured over a corpus of merged datasets the correlation
|
||||
// between the Patterson peak and <I^2>/<I>^2 is 0.03, and the second moment in this pipeline
|
||||
// runs above 2.0 for unrelated reasons.
|
||||
if (merged) {
|
||||
const auto &t = result.tncs;
|
||||
if (!t.measurable) {
|
||||
Add(s, KeyEnum("TNCS_DETECTED", "NOT_MEASURED",
|
||||
{"TRUE", "FALSE", "INCONCLUSIVE", "NOT_MEASURED"}));
|
||||
Add(s, Blank());
|
||||
Add(s, Prose(fmt::format(
|
||||
" Translational pseudo-symmetry: NOT MEASURED{}{}. A refusal is not a\n"
|
||||
" statement that this crystal has none - the question was not answered.",
|
||||
t.refusal.empty() ? "" : " - ", t.refusal)));
|
||||
} else {
|
||||
Add(s, KeyEnum("TNCS_DETECTED",
|
||||
t.detected ? "TRUE"
|
||||
: t.modulation_measured ? "FALSE"
|
||||
: "INCONCLUSIVE",
|
||||
{"TRUE", "FALSE", "INCONCLUSIVE", "NOT_MEASURED"}));
|
||||
// A different finding from a pseudo-symmetry, and reported as one: the data are
|
||||
// EXACTLY invariant under these, so the centring or the cell is wrong rather than the
|
||||
// molecules being packed unusually.
|
||||
for (const auto &v : t.undeclared_lattice_translations)
|
||||
Add(s, KeyText("UNDECLARED_LATTICE_TRANSLATION",
|
||||
fmt::format("{:.3f} {:.3f} {:.3f}", v[0], v[1], v[2])));
|
||||
Add(s, KeyReal("TNCS_PATTERSON_PEAK_PCT", t.peak_percent, "{:.1f}"));
|
||||
Add(s, KeyReal("TNCS_PATTERSON_PEAK_Z", t.peak_z, "{:.1f}"));
|
||||
Add(s, KeyText("TNCS_PATTERSON_NULL_PCT",
|
||||
fmt::format("{:.1f} +- {:.2f}", t.null_mean, t.null_sd)));
|
||||
if (t.modulation_measured) {
|
||||
Add(s, KeyText("TNCS_VECTOR",
|
||||
fmt::format("{:.3f} {:.3f} {:.3f}", t.vector_frac[0],
|
||||
t.vector_frac[1], t.vector_frac[2])));
|
||||
Add(s, KeyReal("TNCS_VECTOR_LENGTH", t.vector_length_A, "{:.1f}"));
|
||||
Add(s, KeyReal("TNCS_MODULATION", t.modulation, "{:.2f}"));
|
||||
Add(s, KeyReal("TNCS_MODULATION_NULL", t.modulation_null, "{:.2f}"));
|
||||
}
|
||||
// Two different claims, deliberately not merged. PSEUDO_CENTRED says the vector is a
|
||||
// rational translation of a cell that is not itself in doubt - the suppressed class is
|
||||
// weak rather than absent, and a smaller cell would contradict it. SUBLATTICE is the
|
||||
// one that says the cell may be a supercell, and only a near-extinct class earns it.
|
||||
if (t.detected && t.commensurate)
|
||||
Add(s, KeyText("TNCS_PSEUDO_CENTRED",
|
||||
fmt::format("1/{}", t.commensurate_denominator)));
|
||||
if (t.detected && t.near_extinct_class)
|
||||
Add(s, KeyEnum("TNCS_SUBLATTICE", "NEAR_EXTINCT_CLASS", {"NEAR_EXTINCT_CLASS"}));
|
||||
Add(s, KeyInt("TNCS_REFLECTIONS", t.n_reflections, true));
|
||||
Add(s, Blank());
|
||||
|
||||
if (t.detected)
|
||||
Add(s, Prose(fmt::format(
|
||||
" Translational pseudo-symmetry: INDICATED. The native Patterson has an off-origin peak\n"
|
||||
" at {:.1f}% of the origin at ({:.3f}, {:.3f}, {:.3f}), {:.1f} A, and the intensities\n"
|
||||
" modulate {:.1f}x along that vector. Declare it to the molecular-replacement program:\n"
|
||||
" the modulation is not in the search model, and without it the search can fail on data\n"
|
||||
" that are otherwise good.",
|
||||
t.peak_percent, t.vector_frac[0], t.vector_frac[1], t.vector_frac[2],
|
||||
t.vector_length_A, t.modulation)));
|
||||
else if (t.modulation_measured)
|
||||
Add(s, Prose(fmt::format(
|
||||
" Translational pseudo-symmetry: no indication. The largest off-origin Patterson peak is\n"
|
||||
" {:.1f}% of the origin, against a permutation null of {:.1f} +- {:.2f}% for these data.",
|
||||
t.peak_percent, t.null_mean, t.null_sd)));
|
||||
else
|
||||
Add(s, Prose(fmt::format(
|
||||
" Translational pseudo-symmetry: INCONCLUSIVE. The Patterson was measured ({:.1f}% of the\n"
|
||||
" origin, z = {:.1f}) but too few acentric reflections remain to test whether that vector\n"
|
||||
" modulates the intensities, so only half the evidence exists. This is not a statement\n"
|
||||
" that the crystal has no pseudo-symmetry.",
|
||||
t.peak_percent, t.peak_z)));
|
||||
|
||||
if (t.detected && t.near_extinct_class)
|
||||
Add(s, ProseDefaultOnly(
|
||||
"\n The class this translation suppresses is not weak but very nearly EXTINCT, which is what\n"
|
||||
" a lattice translation of a smaller cell looks like when it is measured in a larger one.\n"
|
||||
" Treat the reported cell as a supercell candidate rather than a settled result: a cell\n"
|
||||
" with this translation as a lattice vector is the rival to check."));
|
||||
|
||||
if (!t.undeclared_lattice_translations.empty()) {
|
||||
const auto &v = t.undeclared_lattice_translations.front();
|
||||
Add(s, ProseDefaultOnly(fmt::format(
|
||||
"\n The merged data are EXACTLY invariant under ({:.3f}, {:.3f}, {:.3f}), which the space\n"
|
||||
" group in use does not declare. A translation the data are exactly invariant under is a\n"
|
||||
" lattice vector by definition, so it is the centring or the cell that is wrong, not the\n"
|
||||
" packing. The pseudo-symmetry search continues underneath it, so the numbers above still\n"
|
||||
" describe the crystal and not the missing centring.", v[0], v[1], v[2])));
|
||||
}
|
||||
|
||||
Add(s, Prose("\n" + TranslationalNCSToText(t), true));
|
||||
if (t.modulation_measured && !t.detected)
|
||||
Add(s, Prose(
|
||||
"\n TNCS_DETECTED= FALSE means the two tests disagreed with each other or neither fired, not\n"
|
||||
" that a Patterson peak was absent: a peak is called only when it stands above THIS\n"
|
||||
" dataset's own permutation noise floor (TNCS_PATTERSON_NULL_PCT, which runs from ~1% on a\n"
|
||||
" large merge to ~18% on a small one) AND the vector it names is shown to modulate the\n"
|
||||
" intensities. Either test alone over-calls by about a factor of two on a corpus of merged\n"
|
||||
" datasets.", true));
|
||||
|
||||
if (t.detected)
|
||||
Warn(doc, PathologyCode::PSEUDO_TRANSLATION, fmt::format(
|
||||
"Translational pseudo-symmetry: Patterson off-origin peak {:.1f}% of the origin at "
|
||||
"({:.3f}, {:.3f}, {:.3f}), {:.1f} A - declare it to the molecular-replacement program, "
|
||||
"which will otherwise be searching against a modulation it does not model",
|
||||
t.peak_percent, t.vector_frac[0], t.vector_frac[1], t.vector_frac[2],
|
||||
t.vector_length_A));
|
||||
if (!t.undeclared_lattice_translations.empty())
|
||||
Warn(doc, PathologyCode::LATTICE_TRANSLATION, fmt::format(
|
||||
"The merged data are exactly invariant under the translation ({:.3f}, {:.3f}, {:.3f}), "
|
||||
"which the space group in use does not declare - the lattice centring or the unit cell "
|
||||
"is wrong",
|
||||
t.undeclared_lattice_translations[0][0],
|
||||
t.undeclared_lattice_translations[0][1],
|
||||
t.undeclared_lattice_translations[0][2]));
|
||||
if (t.detected && t.near_extinct_class)
|
||||
Warn(doc, PathologyCode::LATTICE_TRANSLATION, fmt::format(
|
||||
"The pseudo-translation at {:.1f} A suppresses its phase class almost completely, so the "
|
||||
"reported cell may be a supercell - check the indexing before treating the cell as settled",
|
||||
t.vector_length_A));
|
||||
}
|
||||
}
|
||||
|
||||
// ---- twinning
|
||||
if (merged && result.twinning.l_test_pairs > 0) {
|
||||
const auto &tw = result.twinning;
|
||||
@@ -1235,6 +1364,11 @@ ReportDocument BuildReportDocument(const std::string &output_prefix,
|
||||
: fmt::format("{}: deltaB {:.1f} A^2", sev, an.delta_b));
|
||||
}
|
||||
}
|
||||
if (result.tncs.measurable)
|
||||
row("Pseudo-symmetry", result.tncs.detected
|
||||
? fmt::format("INDICATED (Patterson peak {:.1f}% of origin, {:.1f} A)",
|
||||
result.tncs.peak_percent, result.tncs.vector_length_A)
|
||||
: std::string("no indication"));
|
||||
if (result.twinning.l_test_pairs > 0)
|
||||
row("Twinning", result.twinning.twinning_suspected
|
||||
? fmt::format("INDICATED (<|L|> {:.3f})", result.twinning.mean_abs_l)
|
||||
|
||||
@@ -57,6 +57,7 @@
|
||||
#include "../image_analysis/lattice_search/LePageLattice.h"
|
||||
#include "../image_analysis/scale_merge/AnisotropyAnalysis.h"
|
||||
#include "../image_analysis/scale_merge/TwinningAnalysis.h"
|
||||
#include "../image_analysis/scale_merge/TranslationalNCS.h"
|
||||
#include "../image_analysis/scale_merge/HKLKey.h"
|
||||
#include "../image_analysis/scale_merge/ScaleOnTheFly.h"
|
||||
#include "../image_analysis/scale_merge/StillsPartialityRefine.h"
|
||||
@@ -4761,6 +4762,18 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
|
||||
// written reflections, and neither survives the run. The promotion flag below is a different
|
||||
// thing - it is what the SEARCH did, the second pass reads it, and it is set either way.
|
||||
if (!geometry_prepass) {
|
||||
// Translational pseudo-symmetry of the same merge. Reporting only - no reflection and no
|
||||
// merge depends on the outcome.
|
||||
if (result.consensus_cell)
|
||||
result.tncs = AnalyzeTranslationalNCS(sm.merged, gemmi::UnitCell(*result.consensus_cell),
|
||||
twin_sg);
|
||||
if (result.tncs.detected)
|
||||
logger.Warning("Translational pseudo-symmetry: Patterson off-origin peak {:.1f}% of the "
|
||||
"origin at ({:.3f}, {:.3f}, {:.3f}), {:.1f} A - declare it to the "
|
||||
"molecular-replacement program",
|
||||
result.tncs.peak_percent, result.tncs.vector_frac[0],
|
||||
result.tncs.vector_frac[1], result.tncs.vector_frac[2],
|
||||
result.tncs.vector_length_A);
|
||||
result.twinning = AnalyzeTwinning(sm.merged, twin_sg);
|
||||
// Set on the struct the text below is rendered from. AnalyzeTwinning returns a fresh result
|
||||
// with this flag clear, so setting it only afterwards left the statistics text asserting
|
||||
@@ -4769,6 +4782,8 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
|
||||
// flat one landing on exactly the case the flag exists for. (Assigning it before the call
|
||||
// would not do: the call replaces the whole struct.)
|
||||
result.twinning.laue_class_was_chosen_by_promotion = promoted_point_group;
|
||||
if (result.tncs.measurable)
|
||||
stats_text << TranslationalNCSToText(result.tncs) << "\n";
|
||||
stats_text << TwinningAnalysisToText(result.twinning) << "\n";
|
||||
}
|
||||
// Mark the conclusion as non-authoritative when the Laue class was reached by a promotion the
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
#include "../image_analysis/spot_finding/SpotFindingSettings.h"
|
||||
#include "../image_analysis/scale_merge/Merge.h" // MergeStatistics
|
||||
#include "../image_analysis/scale_merge/TwinningAnalysis.h" // TwinningAnalysisResult
|
||||
#include "../image_analysis/scale_merge/TranslationalNCS.h" // TranslationalNCSResult
|
||||
#include "ModelValidation.h" // ModelValidationResult
|
||||
#include "ReportDocument.h" // ReportWarning, PathologyCode
|
||||
#include "../image_analysis/scale_merge/SearchSpaceGroup.h" // SearchSpaceGroupResult
|
||||
@@ -270,6 +271,10 @@ struct ProcessResult {
|
||||
|
||||
// Twinning analysis of the final merged intensities (l_test_pairs == 0 when not computed).
|
||||
TwinningAnalysisResult twinning;
|
||||
// Translational pseudo-symmetry of the same merge. Measured BEFORE the twinning analysis,
|
||||
// because a non-half-integer pseudo-translation contaminates the L-test and the twin verdict
|
||||
// has to know that.
|
||||
TranslationalNCSResult tncs;
|
||||
|
||||
// The same analysis measured BEFORE the space group was decided, on the subgroup merge the search
|
||||
// was given. It is the only twinning number a promotion cannot contaminate: once a higher Laue
|
||||
|
||||
@@ -41,6 +41,7 @@
|
||||
#include "../image_analysis/scale_merge/ResolutionCutoff.h"
|
||||
#include "../image_analysis/scale_merge/AnisotropyAnalysis.h"
|
||||
#include "../image_analysis/scale_merge/TwinningAnalysis.h"
|
||||
#include "../image_analysis/scale_merge/TranslationalNCS.h"
|
||||
#include "../image_analysis/scale_merge/SearchSpaceGroup.h"
|
||||
#include "Rugnux.h"
|
||||
#include "RugnuxDefaults.h"
|
||||
@@ -1794,6 +1795,12 @@ static int RunRugnux(int argc, char **argv) {
|
||||
|
||||
const auto &twin_sg_opt = experiment.GetGemmiSpaceGroup();
|
||||
const gemmi::SpaceGroup *twin_sg = twin_sg_opt ? &*twin_sg_opt : nullptr;
|
||||
// Same order as the full pipeline.
|
||||
TranslationalNCSResult tncs;
|
||||
if (const auto uc = experiment.GetUnitCell())
|
||||
tncs = AnalyzeTranslationalNCS(merged_reflections, gemmi::UnitCell(*uc), twin_sg);
|
||||
if (tncs.measurable)
|
||||
std::cout << std::endl << TranslationalNCSToText(tncs);
|
||||
const auto twinning = AnalyzeTwinning(merged_reflections, twin_sg);
|
||||
std::cout << std::endl << TwinningAnalysisToText(twinning) << std::endl;
|
||||
|
||||
|
||||
@@ -95,6 +95,7 @@ ADD_EXECUTABLE(jfjoch_test
|
||||
XDSPluginTest.cpp
|
||||
MergeScaleTest.cpp
|
||||
AnisotropyAnalysisTest.cpp
|
||||
TranslationalNCSTest.cpp
|
||||
RfreeFlagsTest.cpp
|
||||
FrenchWilsonTest.cpp
|
||||
ReindexAmbiguityTest.cpp
|
||||
|
||||
@@ -70,7 +70,7 @@ TEST_CASE("ResultReport_Render", "[Diagnostics]") {
|
||||
|
||||
// The stable keys a consumer greps for. The version is pinned on purpose: a key added to the
|
||||
// report is a contract change, and this line is where it has to be acknowledged.
|
||||
CHECK(text.find("\nREPORT_VERSION= 9\n") != std::string::npos);
|
||||
CHECK(text.find("\nREPORT_VERSION= 10\n") != std::string::npos);
|
||||
// The flight-path medium is an ASSUMPTION - no file states it - so the report has to say which
|
||||
// one was made and what it was worth, or the user cannot know a choice was taken on their behalf.
|
||||
CHECK(text.find("\nFLIGHT_PATH= AIR\n") != std::string::npos);
|
||||
@@ -540,3 +540,70 @@ TEST_CASE("ResultReport_CompletenessTwoRanges", "[Diagnostics]") {
|
||||
CHECK(same.find(" % to ") == std::string::npos);
|
||||
CHECK(same.find("over the full written range") != std::string::npos);
|
||||
}
|
||||
|
||||
// The TNCS_* block is an interface in the same way the keys above are, and the distinction between
|
||||
// FALSE, INCONCLUSIVE and NOT_MEASURED is the whole reason it is an enum rather than a boolean: a
|
||||
// merge the detector could not run on must not read as a crystal that was tested and found clean.
|
||||
TEST_CASE("ResultReport_TranslationalNCS", "[Diagnostics]") {
|
||||
DiffractionExperiment x(DetJF(1));
|
||||
x.ImagesPerTrigger(600);
|
||||
|
||||
ProcessResult result;
|
||||
result.images_processed = 600;
|
||||
result.consensus_cell = UnitCell{.a = 79.0f, .b = 79.0f, .c = 38.0f,
|
||||
.alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
|
||||
result.space_group = *gemmi::find_spacegroup_by_number(96);
|
||||
result.has_merge_statistics = true;
|
||||
|
||||
{
|
||||
const auto text = RenderResultReport("prefix", "in.h5", x, result);
|
||||
CHECK(text.find("\nTNCS_DETECTED= NOT_MEASURED\n") != std::string::npos);
|
||||
CHECK(text.find("not a\n statement that this crystal has none") != std::string::npos);
|
||||
}
|
||||
|
||||
result.tncs.measurable = true;
|
||||
result.tncs.n_reflections = 4000;
|
||||
result.tncs.peak_percent = 54.2;
|
||||
result.tncs.peak_z = 42.3;
|
||||
result.tncs.null_mean = 9.9;
|
||||
result.tncs.null_sd = 1.05;
|
||||
result.tncs.modulation_measured = true;
|
||||
result.tncs.vector_frac = {0.0, 0.5, 0.25};
|
||||
result.tncs.vector_length_A = 40.7;
|
||||
result.tncs.modulation = 12.50;
|
||||
result.tncs.modulation_null = 1.48;
|
||||
result.tncs.detected = true;
|
||||
result.tncs.near_extinct_class = true;
|
||||
|
||||
const auto text = RenderResultReport("prefix", "in.h5", x, result);
|
||||
CHECK(text.find("\nTNCS_DETECTED= TRUE\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_PATTERSON_PEAK_PCT= 54.2\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_PATTERSON_PEAK_Z= 42.3\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_PATTERSON_NULL_PCT= 9.9 +- 1.05\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_VECTOR= 0.000 0.500 0.250\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_VECTOR_LENGTH= 40.7\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_MODULATION= 12.50\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_MODULATION_NULL= 1.48\n") != std::string::npos);
|
||||
CHECK(text.find("\nTNCS_SUBLATTICE= NEAR_EXTINCT_CLASS\n") != std::string::npos);
|
||||
// A detection is a condition a consumer switches on, not only a sentence to read.
|
||||
CHECK(text.find("\nPATHOLOGY_FLAGS= PSEUDO_TRANSLATION LATTICE_TRANSLATION\n")
|
||||
!= std::string::npos);
|
||||
CHECK(text.find("\nVERDICT= WARNINGS\n") != std::string::npos);
|
||||
// TNCS_REFLECTIONS is a pipeline internal and stays out of the default report.
|
||||
CHECK(text.find("\nTNCS_REFLECTIONS=") == std::string::npos);
|
||||
RunProvenance dev;
|
||||
dev.developer = true;
|
||||
CHECK(RenderResultReport("prefix", "in.h5", x, result, dev).find("\nTNCS_REFLECTIONS= 4000\n")
|
||||
!= std::string::npos);
|
||||
|
||||
// A translation the data are EXACTLY invariant under is a lattice vector, and is reported as one
|
||||
// rather than as a pseudo-symmetry - this is the centred-lattice-merged-in-P1 case.
|
||||
result.tncs.detected = false;
|
||||
result.tncs.near_extinct_class = false;
|
||||
result.tncs.undeclared_lattice_translations.push_back({0.5, 0.5, 0.0});
|
||||
const auto lat = RenderResultReport("prefix", "in.h5", x, result);
|
||||
CHECK(lat.find("\nTNCS_DETECTED= FALSE\n") != std::string::npos);
|
||||
CHECK(lat.find("\nUNDECLARED_LATTICE_TRANSLATION= 0.500 0.500 0.000\n") != std::string::npos);
|
||||
CHECK(lat.find("\nPATHOLOGY_FLAGS= LATTICE_TRANSLATION\n") != std::string::npos);
|
||||
CHECK(lat.find("\nTNCS_SUBLATTICE=") == std::string::npos);
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
#include <catch2/catch_all.hpp>
|
||||
|
||||
#include <array>
|
||||
#include <cmath>
|
||||
#include <numbers>
|
||||
#include <vector>
|
||||
|
||||
#include "../image_analysis/scale_merge/TranslationalNCS.h"
|
||||
#include "gemmi/symmetry.hpp"
|
||||
#include "gemmi/unitcell.hpp"
|
||||
|
||||
namespace {
|
||||
// For a half-integer pseudo-translation the classes carry (1 +- f) times the mean intensity, and
|
||||
// the Patterson peak at that vector comes out at exactly f of the origin. f = 0.6 therefore models
|
||||
// a 60% peak - as strong as the strongest genuine pseudo-symmetry in a 137-dataset corpus. f = 1
|
||||
// would mean the weak class is EXTINCT, which is a lattice translation, not a pseudo-symmetry.
|
||||
constexpr double kRealisticDepth = 0.6;
|
||||
|
||||
gemmi::UnitCell Cell(double a, double b, double c) {
|
||||
gemmi::UnitCell out;
|
||||
out.set(a, b, c, 90.0, 90.0, 90.0);
|
||||
return out;
|
||||
}
|
||||
|
||||
// A synthetic merge over the reciprocal asymmetric unit: Wilson-distributed intensities from a
|
||||
// hash of the index (so the crystal is reproduced bit for bit), optionally multiplied by the
|
||||
// factor a translational pseudo-symmetry u imposes, |1 + exp(2 pi i h.u)|^2 / 2 written with a
|
||||
// depth f so the weak class is suppressed rather than extinguished.
|
||||
std::vector<MergedReflection> Synthetic(const gemmi::UnitCell &cell, const char *space_group,
|
||||
const std::array<double, 3> *u, double f,
|
||||
double d_min = 3.0) {
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name(space_group);
|
||||
REQUIRE(sg != nullptr);
|
||||
const gemmi::GroupOps ops = sg->operations();
|
||||
const gemmi::ReciprocalAsu asu(sg);
|
||||
std::vector<MergedReflection> out;
|
||||
const int hmax = static_cast<int>(cell.a / d_min) + 1;
|
||||
const int kmax = static_cast<int>(cell.b / d_min) + 1;
|
||||
const int lmax = static_cast<int>(cell.c / d_min) + 1;
|
||||
for (int h = -hmax; h <= hmax; ++h)
|
||||
for (int k = -kmax; k <= kmax; ++k)
|
||||
for (int l = -lmax; l <= lmax; ++l) {
|
||||
const gemmi::Op::Miller hkl{{h, k, l}};
|
||||
if ((h == 0 && k == 0 && l == 0) || !asu.is_in(hkl)
|
||||
|| ops.is_systematically_absent(hkl))
|
||||
continue;
|
||||
const double d2 = cell.calculate_1_d2(hkl);
|
||||
if (d2 <= 0 || 1.0 / std::sqrt(d2) < d_min)
|
||||
continue;
|
||||
const uint32_t seed = static_cast<uint32_t>(h * 73856093 ^ k * 19349663
|
||||
^ l * 83492791);
|
||||
const double uni = ((seed * 2654435761u) >> 8) / static_cast<double>(1 << 24);
|
||||
double I = 1000.0 * -std::log(std::max(1e-6, uni)) * std::exp(-0.5 * 20.0 * d2);
|
||||
if (u != nullptr) {
|
||||
const double phase = 2.0 * std::numbers::pi
|
||||
* (h * (*u)[0] + k * (*u)[1] + l * (*u)[2]);
|
||||
I *= 1.0 + f * std::cos(phase);
|
||||
}
|
||||
MergedReflection r;
|
||||
r.h = h; r.k = k; r.l = l;
|
||||
r.d = static_cast<float>(1.0 / std::sqrt(d2));
|
||||
r.I = static_cast<float>(I);
|
||||
r.sigma = static_cast<float>(0.02 * std::fabs(r.I) + 1.0);
|
||||
out.push_back(r);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
}
|
||||
|
||||
// The detector must find a pseudo-translation that is there, and name it.
|
||||
TEST_CASE("A pseudo-translation is detected and its vector recovered", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(70.0, 85.0, 110.0);
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 21 21 21");
|
||||
const std::array<double, 3> u{0.5, 0.5, 0.5};
|
||||
const auto merged = Synthetic(cell, "P 21 21 21", &u, kRealisticDepth);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, sg);
|
||||
|
||||
REQUIRE(r.measurable);
|
||||
CHECK(r.detected);
|
||||
CHECK(r.peak_percent > 20.0);
|
||||
CHECK(r.undeclared_lattice_translations.empty());
|
||||
CHECK(r.peak_z > 5.0);
|
||||
CHECK(r.modulation > 2.5 * r.modulation_null);
|
||||
// The vector is refined to maximise the modulation, not to minimise its distance from the truth,
|
||||
// and the greedy search will trade a little accuracy for a little contrast. It is a starting
|
||||
// point for a molecular-replacement program (which refines it), not a measured constant, so the
|
||||
// tolerance here is the honest one - about 0.05 fractional, roughly 5 A on a 100 A axis.
|
||||
for (int i = 0; i < 3; ++i)
|
||||
CHECK(std::fabs(r.vector_frac[i] - 0.5) < 0.05);
|
||||
CHECK(r.commensurate);
|
||||
CHECK(r.commensurate_denominator == 2);
|
||||
// A half-integer pseudo-translation is pseudo-centring, not a wrong cell: the suppressed class is
|
||||
// weak, not absent, so the claim "your cell may be a supercell" must NOT be made here.
|
||||
CHECK_FALSE(r.near_extinct_class);
|
||||
CHECK(TranslationalNCSToText(r).find("supercell candidate") == std::string::npos);
|
||||
}
|
||||
|
||||
// The false-positive rate is the whole product: the same crystal without a pseudo-translation must
|
||||
// say nothing.
|
||||
TEST_CASE("A crystal without a pseudo-translation stays silent", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(70.0, 85.0, 110.0);
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 21 21 21");
|
||||
const auto merged = Synthetic(cell, "P 21 21 21", nullptr, 0.0);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, sg);
|
||||
|
||||
REQUIRE(r.measurable);
|
||||
CHECK_FALSE(r.detected);
|
||||
}
|
||||
|
||||
// The case that would make the detector useless if it were got wrong. A centred lattice has a real,
|
||||
// full-height off-origin Patterson peak at its centring translation BY CONSTRUCTION - every C-, I-
|
||||
// and F-centred crystal has one, and none of them has a pseudo-symmetry because of it. The centring
|
||||
// vectors are origin-equivalent points and must be excluded along with the origin.
|
||||
TEST_CASE("A genuinely centred lattice is not called", "[tncs]") {
|
||||
struct Case { const char *sg; gemmi::UnitCell cell; };
|
||||
const std::vector<Case> cases{
|
||||
{"C 1 2 1", Cell(120.0, 75.0, 90.0)},
|
||||
{"I 2 2 2", Cell(80.0, 95.0, 115.0)},
|
||||
{"F 2 2 2", Cell(110.0, 125.0, 140.0)},
|
||||
};
|
||||
for (const auto &c : cases) {
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name(c.sg);
|
||||
REQUIRE(sg != nullptr);
|
||||
const auto merged = Synthetic(c.cell, c.sg, nullptr, 0.0);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, c.cell, sg);
|
||||
INFO(c.sg << ": peak " << r.peak_percent << "% z " << r.peak_z << " modulation "
|
||||
<< r.modulation << " vs null " << r.modulation_null);
|
||||
REQUIRE(r.measurable);
|
||||
CHECK_FALSE(r.detected);
|
||||
}
|
||||
}
|
||||
|
||||
// A centred lattice that ALSO has a pseudo-translation must still be called - excluding the centring
|
||||
// vectors must not blind the test to a real one somewhere else.
|
||||
TEST_CASE("A centred lattice with a real pseudo-translation is still called", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(120.0, 75.0, 90.0);
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("C 1 2 1");
|
||||
const std::array<double, 3> u{0.0, 0.0, 0.5};
|
||||
const auto merged = Synthetic(cell, "C 1 2 1", &u, kRealisticDepth);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, sg);
|
||||
INFO("peak " << r.peak_percent << "% z " << r.peak_z << " u " << r.vector_frac[0] << " "
|
||||
<< r.vector_frac[1] << " " << r.vector_frac[2]);
|
||||
REQUIRE(r.measurable);
|
||||
CHECK(r.detected);
|
||||
CHECK(std::fabs(r.vector_frac[2] - 0.5) < 0.03);
|
||||
}
|
||||
|
||||
// A cell too small to have anything in the 20-5 A band must say so, and must NOT say "no tNCS".
|
||||
TEST_CASE("A small-molecule cell reports that it could not be measured", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(9.0, 11.0, 13.0);
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 21 21 21");
|
||||
const auto merged = Synthetic(cell, "P 21 21 21", nullptr, 0.0, 0.8);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, sg);
|
||||
CHECK_FALSE(r.measurable);
|
||||
CHECK_FALSE(r.detected);
|
||||
CHECK_FALSE(r.refusal.empty());
|
||||
CHECK(TranslationalNCSToText(r).find("not a statement") != std::string::npos);
|
||||
}
|
||||
|
||||
// P1 - no symmetry to expand with, and no centric reflections. The commonest case for a merge the
|
||||
// space-group search declined to promote, so it must work rather than divide by zero.
|
||||
TEST_CASE("P1 data are handled with no space group at all", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(70.0, 85.0, 110.0);
|
||||
const std::array<double, 3> u{0.5, 0.5, 0.5};
|
||||
const auto merged = Synthetic(cell, "P 1", &u, kRealisticDepth);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, nullptr);
|
||||
REQUIRE(r.measurable);
|
||||
CHECK(r.detected);
|
||||
}
|
||||
|
||||
// The failure this detector would be useless without. A C-centred crystal MERGED IN P1 - which is
|
||||
// what --mode scale does on a file with no space group, and what the pre-promotion merge is - still
|
||||
// carries its centring vector, and the Patterson there is as high as at the origin. With no space
|
||||
// group to name the centring it cannot be excluded from a list, so it has to be recognised for what
|
||||
// it is: a translation the data are exactly invariant under is a LATTICE vector, not a pseudo-
|
||||
// symmetry. Measured before this was handled, 11 of 95 P1 merges in the corpus were falsely called.
|
||||
TEST_CASE("A centred lattice merged in P1 is reported as a lattice translation, not tNCS", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(120.0, 75.0, 90.0);
|
||||
// Generated with C-centring (so the absences are real), then handed over as if it were P1.
|
||||
const auto merged = Synthetic(cell, "C 1 2 1", nullptr, 0.0);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, nullptr);
|
||||
REQUIRE(r.measurable);
|
||||
REQUIRE_FALSE(r.undeclared_lattice_translations.empty());
|
||||
const auto &t = r.undeclared_lattice_translations.front();
|
||||
CHECK(std::fabs(t[0] - 0.5) < 0.03);
|
||||
CHECK(std::fabs(t[1] - 0.5) < 0.03);
|
||||
CHECK_FALSE(r.detected);
|
||||
CHECK(TranslationalNCSToText(r).find("LATTICE translation") != std::string::npos);
|
||||
}
|
||||
|
||||
// ...and a real pseudo-translation hiding underneath that centring vector must still be found, or
|
||||
// the guard above would have traded one blind spot for another.
|
||||
TEST_CASE("A pseudo-translation under an undeclared centring is still found", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(120.0, 75.0, 90.0);
|
||||
const std::array<double, 3> u{0.0, 0.0, 0.5};
|
||||
const auto merged = Synthetic(cell, "C 1 2 1", &u, kRealisticDepth);
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, nullptr);
|
||||
REQUIRE(r.measurable);
|
||||
REQUIRE_FALSE(r.undeclared_lattice_translations.empty());
|
||||
INFO("peak " << r.peak_percent << "% u " << r.vector_frac[0] << " " << r.vector_frac[1] << " "
|
||||
<< r.vector_frac[2]);
|
||||
CHECK(r.detected);
|
||||
CHECK(std::fabs(r.vector_frac[2] - 0.5) < 0.05);
|
||||
}
|
||||
|
||||
// The other end of the same scale: a translation the data are EXACTLY invariant under - the weak
|
||||
// class extinct rather than weak - is a lattice vector however it arose, and must be reported as one
|
||||
// even when the space group declares no centring at all.
|
||||
TEST_CASE("An exact translation is a lattice vector, not a strong pseudo-symmetry", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(70.0, 85.0, 110.0);
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 21 21 21");
|
||||
const std::array<double, 3> u{0.5, 0.5, 0.5};
|
||||
const auto merged = Synthetic(cell, "P 21 21 21", &u, 1.0); // weak class exactly zero
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, sg);
|
||||
REQUIRE(r.measurable);
|
||||
REQUIRE_FALSE(r.undeclared_lattice_translations.empty());
|
||||
CHECK_FALSE(r.detected);
|
||||
}
|
||||
|
||||
// The most valuable thing this detector can say is not "there is a pseudo-symmetry" but "your cell
|
||||
// is twice the size it should be". That case does NOT land on a simple fraction of the reported
|
||||
// axes when the true sub-cell is triclinic, so it is recognised from the depth of the modulation
|
||||
// instead: a class that is nearly extinct rather than merely weak is a lattice translation in
|
||||
// disguise.
|
||||
TEST_CASE("A near-extinct class is reported as a sub-lattice", "[tncs]") {
|
||||
const gemmi::UnitCell cell = Cell(70.0, 85.0, 110.0);
|
||||
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name("P 1 2 1");
|
||||
const std::array<double, 3> u{0.0, 0.425, 0.5}; // no p/q with q <= 6 fits the whole vector
|
||||
const auto merged = Synthetic(cell, "P 1 2 1", &u, 0.96); // weak class at 2% of the strong
|
||||
const auto r = AnalyzeTranslationalNCS(merged, cell, sg);
|
||||
REQUIRE(r.measurable);
|
||||
REQUIRE(r.detected);
|
||||
CHECK_FALSE(r.commensurate);
|
||||
CHECK(r.near_extinct_class);
|
||||
CHECK(TranslationalNCSToText(r).find("supercell candidate") != std::string::npos);
|
||||
}
|
||||
Reference in new issue
Block a user