diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 4069e4629..a4db67fbc 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -140,7 +140,7 @@ a cut calibrated on one pair misjudges the next. In one measured arm an unrelate The only null that fits both the model and the data is therefore **made out of them**: the same model is re-oriented at random about its own centroid and run through the identical path — the same scaling, -the same rigid-body placement, the same R — five times, and the real fit is asked how far above the +the same rigid-body placement, the same R — nine times, and the real fit is asked how far above the resulting distribution it sits (`MODEL_FIT_SIGMA`, accepted at 3σ). The replicates are placed as well as fitted, or the comparison would be between a placed model and unplaced nulls and the margin would be inflated by the placement rather than by the model. Measured on one rotation data set: the crystal's @@ -166,12 +166,20 @@ one that has to deliver a map seconds after the last image — and it is why the indexing probe is a handful of scaling fits and runs first anyway, so whether the expensive part is worth paying for is known before it starts. +The count is nine, and it is the *spread* that sets it rather than the mean: the verdict is +(real − mean)/sd of the sample, the relative error on an sd from $n$ draws is $1/\sqrt{2(n-1)}$, and a +sample that happens to come out narrow is what turns a model that does not fit into one that appears +to. Measured on the case nearest the gate — an unrelated protein at 1.83σ against a threshold of 3 — +the chance of reading over the gate on a different seed is 31 % at $n=3$, 17 % at $n=5$ and 6 % at +$n=9$. Nine is affordable only because the replicates run at once: the null costs the slowest of them +rather than their sum, and the slowest of nine is barely above the slowest of five. + The null costs one full fit and one rigid-body placement per replicate. The replicates share nothing — each is the same model under a different rotation, scored the same way — so each takes a copy of the model and of its structure factors and they run **concurrently**, on the run's own thread budget -(`-N`); the real model is not touched by any of them. Measured on an idle machine, the five cost -2.5 s together where running them one after another costs 10 s, on a `--mode scale` run that merges -in 2 s. What remains is the cost of the *slowest* replicate: how many placement evaluations an +(`-N`); the real model is not touched by any of them. Measured on an idle machine, five replicates +cost 2.5 s together where running them one after another costs 10 s, on a `--mode scale` run that +merges in 2 s. What remains is the cost of the *slowest* replicate: how many placement evaluations an orientation needs varies by nearly half between them, so the concurrency saturates around 4×, and more threads than replicates buy nothing. The orientations are drawn up front, in order, from the fixed seed, so replicate $i$ gets the same orientation whatever order the threads run in — a σ that diff --git a/rugnux/ModelValidation.cpp b/rugnux/ModelValidation.cpp index 1836e87e0..2cf79b608 100644 --- a/rugnux/ModelValidation.cpp +++ b/rugnux/ModelValidation.cpp @@ -69,10 +69,18 @@ constexpr double ANOMALOUS_INVERSION_SIGMA = 5.0; // signal is there and what carries it; a full site list is what the map file is for. constexpr size_t MAX_ANOMALOUS_SITES = 10; -// How many random placements of the same model the real fit is compared against. Five is enough to -// put a mean and a spread on a distribution whose arms are several sigma apart in the cases that -// matter, and each replicate costs a full fit and a rigid-body placement. -constexpr int NULL_REPLICATES = 5; +// How many random placements of the same model the real fit is compared against. The verdict is +// (real - mean)/sd of this sample, so what matters is not the mean but how well the SPREAD is +// pinned: the relative error on an sd from n draws is 1/sqrt(2(n-1)), and a sample that happens to +// come out narrow is what turns a model that does not fit into one that appears to. Measured on the +// case that sits closest to the gate - a model of an unrelated protein, 1.83 sigma against a +// threshold of 3 - the chance of it reading over the gate on a different seed is 31% at n=3, 17% at +// n=5, 6% at n=9. +// +// Nine rather than five because the replicates run concurrently: the null costs the SLOWEST of them +// rather than their sum, and the slowest of nine is barely above the slowest of five, so the extra +// four are close to free in wall clock (measured 2.5 s against 2.6 s) up to the thread count. +constexpr int NULL_REPLICATES = 9; // Fixed, so the same data and the same model give the same verdict on every run. constexpr unsigned NULL_SEED = 20260902;