mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 67 Wiki Activity

Space-group search: the merge with all the observations decides

Browse Source 
A rotation run makes the space-group decision twice: once on a merge of only
the well-measured observations (--search-min-zeta) and once on all of them, and
the rule was to keep whichever search found more symmetry. Its justification
was that the filtered arm can only ever LOSE an operator - discarding 40-80% of
the observations starves the operator correlations - and never invent one.

That premise was checked at one set of geometries, and there it holds exactly:
over 1232 stored battery runs (38 geometries x 32 code variants) the two arms
disagree 59 times and the all-observation arm is the higher one every single
time. Away from those geometries it fails. Over 68 runs whose first pass was
given a displaced beam centre, the filtered arm confirms an operator the full
merge refuses 14 times, and the two arms never once both confirm the promotion.
On one rhombohedral crystal the all-observation merge refuses a 3 -> 32
promotion at the correct beam centre, on the same twin-law statistic the search
uses everywhere (1.78 against a bound of 1.70), and the filtered merge - which
had thrown two thirds of the observations away - overrides it into the wrong
space group. The rule was validated on the only data that cannot test it.

So the filtered arm no longer promotes. Where the two disagree the answer is
the one the merge with all the observations supports, as the systematic
absences already were, and both are still reported. The filter keeps the job it
was added for - stopping a near-tangential measurement from making a real
operator look like a twin law - it simply cannot outvote the merge that has
every observation in it.

Costless on the 38-crystal battery, as predicted: the changed branch is never
taken there (41 searches: 39 agree, 2 with the all-observation arm higher, none
the other way), the space groups stay at 35/38 with the same three misses, and
the crystals the second search does rescue - where the all-observation merge is
the one finding the higher symmetry - are untouched. Injecting the post-refined
beam centre into the crystal above now yields the right space group, 27900
unique reflections at R_meas 12.6% against 14078 at 21.3% before.
This commit is contained in:
leonarski_f
2026-08-12 15:07:34 +02:00
parent 7ce47bd4d1
commit 1f2dad530c
5 changed files with 31 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHANGELOG.md
+1
View File
@@ -3,6 +3,7 @@
### 1.0.0-rc.161
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* Space-group search: where the search on the well-measured observations (`--search-min-zeta`) and the search on all of them disagree, the answer is now always the one the merge of all the observations supports, instead of whichever of the two found more symmetry.
* Scaling: rotation merging now also fits a correction surface over **rotation angle × detector position**, which reaches the absorption left behind when the diffracting volume drifts through the beam; like the other surfaces it is cross-validated and applied only where it improves reflections it was not fitted on.
* Scaling: the rotation correction surfaces (decay, absorption, detector-plane modulation) no longer pull each cell's factor towards zero in proportion to that cell's own noise, which had put a spurious centre-to-edge ramp on runs whose outer resolution shells carry no signal.
* rugnux: The radiation-damage relative *B*-factor is now measured on resolution-shell means instead of single observations, a batch whose data cannot support a measurement prints `-` rather than a value, and no first→last number is reported for a curve that no straight line describes — a disturbance confined to part of a sweep is no longer reported as dose.
docs/CPU_DATA_ANALYSIS.md
+1 -1
View File
@@ -868,7 +868,7 @@ A **dataset-wide** Wilson $B$ is also estimated over the merged reflections —
Several space groups may share an absence pattern exactly. Where they do, the search scores them identically and **all of them are named** in the result rather than one being reported as the answer: some are enantiomorph pairs, which merged intensities cannot distinguish in principle, and others differ only by a screw condition that the centering condition already implies, so the screw has no observable signature at all. The representative reported first is the lowest space-group number, which is a convention and not a measurement.
The Lorentz factor $\zeta$ (§8.3) governs how well a reflection can be measured, so when the spindle lies in a plane of the lattice, an operator permuting the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. The search is therefore run a second time on a merge of only the well-measured observations (`--search-min-zeta`, rotation default 0.85), and **whichever search found more symmetry is kept** — one-way safe, because discarding observations can starve an operator correlation but never invent one. The filtered merge decides the point group only; absences come from the full merge, since they live in the weak reflections the filter removes. A tie (same order, different symmetry) is reported with both candidates named, for trying in molecular replacement.
The Lorentz factor $\zeta$ (§8.3) governs how well a reflection can be measured, so when the spindle lies in a plane of the lattice, an operator permuting the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. The search is therefore run a second time on a merge of only the well-measured observations (`--search-min-zeta`, rotation default 0.85), both answers are reported, and **where they disagree the merge of all the observations decides**. The filter discards 40–80 % of the observations, which can starve an operator correlation the full merge confirms and can equally leave an operator confirmed that the full merge refuses, so the decision — the point group as well as the absences, which live in the weak reflections the filter removes — rests on the arm with every observation behind it. A tie (same order, different symmetry) is reported with both candidates named, for trying in molecular replacement.
**Centering** is accepted when the systematically-absent class is weak relative to the present one by *either* of two floor-independent tests: its mean signed $I/\sigma$ well below the present mean, *or* its rate of individually-significant reflections well below the present class's own significant rate. The second test covers weak and low-energy data, where a positive intensity floor (background and profile leakage) lifts the absent class's mean $I/\sigma$ well above zero and, when the present class is itself weak, carries the plain mean ratio past its bound; a false centering fails both tests, its absent class being as strong as the present one. When several centerings pass, they are ranked by their **net** systematic absences (absent minus violating), not the gross absent count, so a super-centering (e.g. $F$ over a true $C$) whose extra, only-half-populated absent class dilutes the strength ratio does not out-rank the correct lower centering.
- **Twinning check.** A Padilla–Yeates $L$-test ($\langle|L|\rangle$, $\langle L^2\rangle$) and the second moment $\langle I^2\rangle/\langle I\rangle^2$ (taken per resolution shell with noise-only shells skipped and Wilson outliers rejected, so a single strong reflection in a collapsed-mean shell cannot skew it) are written to the merged mmCIF as a twinning diagnostic. Twinning is only flagged in Laue classes where a merohedral twin law can exist; the holohedral high-symmetry classes ($4/mmm$, $6/mmm$, $m\bar{3}m$, and $\bar{3}m$ on a rhombohedral lattice) are exempt, so a low $\langle|L|\rangle$ there is reported as a statistical artefact rather than twinning.
docs/RUGNUX.md
+1 -1
View File
@@ -468,7 +468,7 @@ Scaling and merging:
| `--ice-min-spot-ratio <num>` | The second ice-presence channel: found **spots** on the hexagonal rings over the same q width of ice-free flanks beside them (1 = spots spread evenly). Ice in large crystallites diffracts as discrete spots and leaves the radial profile flat, so `--ice-min-score` alone is blind to it (default: 2.0; 0 disables this channel) |
| `--reject-outliers <num>` | Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for `rot3d`, off otherwise) |
| `--min-image-cc <num>` | Per-image CC limit, percent (default: no limit) |
| `--search-min-zeta <num>` | De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry \|ζ\| reaches this, and keep whichever search found more symmetry (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law. The point group only — the systematic absences always come from the merge of all the observations |
| `--search-min-zeta <num>` | De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry \|ζ\| reaches this, and report both answers (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law. Where the two searches disagree, the merge of all the observations decides — as it always has for the systematic absences |
| `--mosaicity <num>` | Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed |
| `--scaling-iterations <num>` | Scaling iterations with no reference data (default: 3) |
| `-z, --reference-mtz <file>` | Reference MTZ (enables reference-driven scaling) |
rugnux/Rugnux.cpp
+27 -34
View File
@@ -1674,13 +1674,17 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
auto sg_search = SearchSpaceGroup(sm.merged, sg_opts);
// Second opinion from a merge that keeps only well-measured observations (see
// RotationScaleMerge::search_min_zeta), and take whichever search found MORE symmetry.
// The two disagree in one direction only: every crystal on which the Lorentz filter changes
// the answer LOSES an operator, because discarding 40-80% of the observations starves the
// operator correlations - it never invents one. So the higher of the two is the safe pick,
// and it costs nothing on crystals where the filter is irrelevant. Both searches keep their
// vetoes, so a merohedral twin stays refused in each (verified on a real twin at the three
// frame ranges where it over-promotes: refused in both arms every time).
// RotationScaleMerge::search_min_zeta). Where the two disagree the all-observation merge
// decides. The filter earns its place because a reflection that crosses the Ewald sphere
// near-tangentially is measured worst, and a handful of bad measurements can make a real
// operator look like a twin law; but it discards 40-80% of the observations, which starves
// the operator correlations. Both failure directions are real - the filtered arm has been
// seen to lose an operator the full merge confirms, and, at geometries away from the ones
// this comparison was first checked on, to confirm one the full merge refuses. Only one of
// the two has every observation behind it, so the filtered merge reports and the
// all-observation merge decides. Both searches keep their vetoes, so a merohedral twin
// stays refused in each (verified on a real twin at the three frame ranges where it
// over-promotes: refused in both arms every time).
if (rsm && rsm->GetSearchMinZeta() > 0.0 && !sg_search.point_group_hm.empty()) {
const double zeta = rsm->GetSearchMinZeta();
rsm->SetSearchMinZeta(0.0);
@@ -1699,12 +1703,11 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
alt.point_group_hm.empty() ? "?" : alt.point_group_hm, order_of(alt),
sg_search.point_group_hm.empty() ? "?" : sg_search.point_group_hm,
order_of(sg_search));
// The filtered merge is used ONLY to rescue a point group the full merge failed to
// confirm. Everything else - and the screw/centering determination in particular - comes
// from the merge with all the observations, because systematic absences are decided by
// the WEAK reflections and the filter throws most of them away. Preferring the filtered
// arm on a tie cost four crystals their screw axes (P2(1) read as P2) for exactly that
// reason, with the point group and every intensity statistic identical.
// The screw/centering determination in particular comes from the merge with all the
// observations, because systematic absences are decided by the WEAK reflections and the
// filter throws most of them away. Preferring the filtered arm on a tie cost four
// crystals their screw axes (P2(1) read as P2) for exactly that reason, with the point
// group and every intensity statistic identical.
// Checked while both arms are still intact - below, the all-observation arm is adopted
// by value and the two become the same result.
// Which point group, by its symmorphic representative rather than by name - gemmi calls
@@ -1734,29 +1737,19 @@ ProcessResult Rugnux::RunPipeline(RugnuxObserver *observer, bool write_output, b
stats_text << " !! " << msg << "\n\n";
result.warnings.push_back(msg);
}
if (order_of(sg_search) <= order_of(alt)) {
if (order_of(alt) > order_of(sg_search))
logger.Info("Space-group search: all-observation merge supports {} where the "
"Lorentz-filtered one supports {} - taking the higher symmetry",
alt.point_group_hm, sg_search.point_group_hm);
sg_search = alt;
} else {
if (order_of(alt) > order_of(sg_search))
logger.Info("Space-group search: all-observation merge supports {} where the "
"Lorentz-filtered one supports {} - taking the higher symmetry",
alt.point_group_hm, sg_search.point_group_hm);
else if (order_of(alt) < order_of(sg_search))
logger.Info("Space-group search: Lorentz-filtered merge supports {} where the "
"all-observation one supports {} - taking the higher symmetry",
"all-observation one supports {} - keeping the all-observation "
"answer, which has every observation behind it",
sg_search.point_group_hm, alt.point_group_hm);
// Keep the point group the filter rescued, but re-decide the ABSENCES on the merge
// that still has all the observations in it. Screws and centering live in the weak
// reflections, which is precisely what a |zeta| cut throws away - reading them off
// the filtered merge is how four crystals lost their screw axes (P2_1 read as P2).
if (sg_search.point_group_representative.has_value()) {
SearchSpaceGroupOptions abs_opts = sg_opts;
abs_opts.fixed_point_group = *sg_search.point_group_representative;
sg_search = SearchSpaceGroup(sm_all.merged, abs_opts);
}
}
// Either way the all-observation merge is the one carried forward: it is what the
// pre-promotion twinning numbers below are measured on, and the filtered merge exists
// only to answer the point-group question.
sg_search = alt;
// The all-observation merge is the one carried forward: it is what the
// pre-promotion twinning numbers below are measured on, and the filtered merge is the
// second opinion reported above.
sm = std::move(sm_all);
}
rugnux/rugnux_cli.cpp
+1 -1
View File
@@ -151,7 +151,7 @@ void print_usage() {
std::cout << " --mosaicity <num> Diagnostic: fix the scaling mosaicity (deg) instead of the per-image seed" << std::endl;
std::cout << " --prediction-mosaicity <num> Diagnostic: fix the rocking width the PREDICTION window opens to (deg), leaving the partiality on the per-image sigma_M. Prediction and partiality are one number by default, so a sigma_M that moves takes the integrated reflection population with it" << std::endl;
std::cout << " --reject-outliers <num> Per-observation merge outlier rejection, N sigma from the per-reflection median (default: 6 for rot3d, XDS/DIALS-style; 0 = off)" << std::endl;
std::cout << " --search-min-zeta <num> De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry |zeta| reaches this, and keep whichever search found MORE symmetry (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law" << std::endl;
std::cout << " --search-min-zeta <num> De-novo space-group search only: also search a merge of just the observations whose Lorentz geometry |zeta| reaches this, and report both answers (default: 0.85 for rotation, 0 = single search). Reflections crossing the Ewald sphere near-tangentially are measured worst and can make a real symmetry operator look like a twin law; where the two searches disagree, the merge of ALL the observations decides" << std::endl;
std::cout << " --min-image-cc <num> Per-image CC limit in percent (default: no limit)" << std::endl;
std::cout << " --scaling-iterations <num> Number of scaling iterations with no reference data (default: 3)" << std::endl;
std::cout << " -z, --reference-mtz <file> Reference MTZ file" << std::endl;