Remove rugnux work-in-progress documents from Git
This commit is contained in:
@@ -1,123 +0,0 @@
|
||||
<!--
|
||||
SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
SPDX-License-Identifier: GPL-3.0-only
|
||||
-->
|
||||
|
||||
# Rugnux de-novo symmetry / indexing investigation (WIP notes)
|
||||
|
||||
Investigation notes on three de-novo rugnux issues found on hard `/data/rotation_test/` crystals
|
||||
(2026-07-09). The **C2 fix below is implemented but NOT committed — it is kept in a git stash**
|
||||
(see "Restoring the WIP change"). This file is the reference to resume from.
|
||||
|
||||
Test crystals: `EcwtCQ244-B1-1`, `EcwtCQ244-E3` (C2), `Benas_3`, `Benas_7` (F432), plus the ~18-crystal
|
||||
regression battery already in `/data/rotation_test/`.
|
||||
|
||||
---
|
||||
|
||||
## 1. C2 missed de-novo (indexed as P1) — root cause + WIP fix
|
||||
|
||||
**Symptom.** De-novo (no `-S`), the two Ecwt C2 crystals index perfectly (100 %) but are reported as
|
||||
**P1**. Their FFT/LatticeSearch cell is the *primitive* cell of the C2 lattice; the C-centering is
|
||||
never recovered. Forcing `-S 5` gives an excellent C2 merge (E3: 99.7 % complete, ISa ~19), so the
|
||||
data is genuinely C2 — only the *de-novo determination* fails.
|
||||
|
||||
**Where it fails.** In `RotationIndexer::RunIndexing()` the pseudo-symmetry demotion guard
|
||||
(`image_analysis/rotation_indexer/RotationIndexer.cpp`, the `if (!sg && sr.system != Triclinic)`
|
||||
block) refines the metric-symmetry cell and, if it indexes < ½ of a free-triclinic refine, demotes to
|
||||
P1. For a strongly-oblique C2 (β ≈ 128–132°) that constrained refine **diverges**, so a genuine C2 is
|
||||
demoted.
|
||||
|
||||
**Why the constrained refine diverges (verified with 4 parallel analysis agents + a numeric sim).**
|
||||
`XtalOptimizer`'s monoclinic parametrisation (`LatticeToRodriguesLengthsBeta_Mono`) keeps only the
|
||||
lengths and β and **snaps α, γ to exactly 90°**. LatticeSearch accepts a cell whose α, γ are up to 3°
|
||||
off 90°, so the idealisation corrupts the metric, and the refine cannot recover. (Numeric check:
|
||||
exactly-90° monoclinic does not diverge; the collapse is monotonic in the α/γ deviation; the
|
||||
C-centering `NearestAllowedHKL` snap is *not* the cause — MONO/C ≈ MONO/P.) A second, separate failure
|
||||
mode of the *full* refine is a length-doubling drift (c → 2c) because a geometry-only spot-fit cost
|
||||
cannot distinguish a cell from its supercell.
|
||||
|
||||
**The WIP fix (stashed).** In the demotion guard, before demoting: re-run `LatticeSearch` on the
|
||||
refined triclinic primitive; if it is non-triclinic, retry the recovered symmetry with
|
||||
**`XtalOptimizerRotationOnly`** (refines orientation only, keeps the *exact* cell metric — no angle
|
||||
idealisation — and is rotation-norm regularised so it is a stable local move) seeded from the refined
|
||||
triclinic geometry/axis, on `ls_ref.conventional` (the correct centered basis). Standardise β obtuse
|
||||
with `ReorderMonoclinic` (an *acute* β breaks the merge d-spacing → "resolution calculation failed").
|
||||
Accept if `frac_c > 0.6 * frac_tri` (a genuine centered cell indexes ~0.78× its primitive; a false
|
||||
pseudo-symmetric cell indexes ~0.01×, so the two are well separated). Otherwise demote to P1 as before.
|
||||
|
||||
**Validation.** Full 21-crystal baseline-vs-fixed battery diff: **only B1-1 and E3 changed (P1 → C2,
|
||||
both match XDS); all 19 others identical**, including every pseudo-symmetry / difficult case
|
||||
(CQ066/CQ034/EP_cs_02-424 stay P1; EP_cs_01-17/02-10/MyoB stay P2₁; EcwtAL500 was already C2).
|
||||
E3 C2: 73.7 % complete, CC1/2 98.5 %, ISa 16.6. B1-1 C2: ~99 % complete (mid-res), ISa 5.7. Change is
|
||||
`RotationIndexer.cpp` only (~50 lines); `XtalOptimizer` untouched.
|
||||
|
||||
**Open design question (why this is stashed, not committed).** The recovery uses
|
||||
`XtalOptimizerRotationOnly` while the main path uses the full `XtalOptimizer` — an apparent
|
||||
inconsistency. Extensive testing showed the alternatives that would let the *standard* optimizer do the
|
||||
recovery all fail, and clarified *why*:
|
||||
|
||||
- ±15 % per-axis length bounds, ±10° per-axis angle bounds, staged (orientation-pre-refine → full),
|
||||
and even orientation-only through the standard optimizer **all drift to ~0.5 % indexed**.
|
||||
- The cost surface is **ill-conditioned for 6-DOF cell refinement** from a ~59 %-indexed centered-cell
|
||||
start (the gradient points away from the true cell in every direction, so bounds only cap the drift).
|
||||
- Orientation-only through the standard optimizer still drifts because that optimizer's orientation
|
||||
refine is **unregularised**; `XtalOptimizerRotationOnly` works only because of its rotation-norm
|
||||
regulariser.
|
||||
|
||||
So the recovery is really a **re-orientation** step (cell + geometry already known), genuinely
|
||||
different from the main "index from scratch" refine — and the regularised local orientation refiner is
|
||||
the right tool for it. The remaining cleanup ideas to evaluate before committing:
|
||||
|
||||
- Give the **main** full refine a length bound and/or an orientation regulariser, so the whole
|
||||
optimizer does stable *local* improvement and never jumps (e.g. the observed 87 → 131 Å). Then the
|
||||
recovery and main path share the same philosophy. Needs its own battery validation.
|
||||
- Or keep the two-tool split but document it as index-vs-refine (as XDS/DIALS do).
|
||||
|
||||
---
|
||||
|
||||
## 2. Benas_3 / Benas_7 are F432 (a ≈ 70.95 Å), not hexagonal — under-determined point group
|
||||
|
||||
The de-novo cubic a=b=c≈70.2 Å cell rugnux finds is **correct** (XDS's hexagonal 49.9/49.9/121 is the
|
||||
rhombohedral pseudo-setting: 70.2/√2 ≈ 49.6, 70.2·√3 ≈ 121.6). Two issues:
|
||||
|
||||
1. **Spot resolution.** Default spot d_min 1.5 Å → 0 % indexing (noise). `--spot-high-resolution 4.0`
|
||||
→ 62 %. Low-resolution crystals need spot-finding restricted; auto-detecting this would help.
|
||||
2. **Point group F222 instead of F432.** With spots restricted, rugnux gets the cubic metric + the
|
||||
F-centering but adopts **F222** (orthorhombic). SearchSpaceGroup Stage A *does* confirm the cubic
|
||||
operators (axis-permuting, CC 0.84–0.96, all present), but the point-group **selection**
|
||||
(`SearchSpaceGroup.cpp`, `chi2 <= chi2_ref * max_merge_chi2_ratio`) rejects 432 because χ² divides
|
||||
by the sigmas and Benas's weak-data error model is broken (b ≈ 0.55, ISa ≈ 1.8). Forcing `-S 209`
|
||||
(F432) merges well (multiplicity 31–65, completeness 90–100 %, CC1/2 97–100 %), so the data *is*
|
||||
F432.
|
||||
|
||||
**Fix direction:** make the point-group consistency gate **sigma-robust** — use CC / merged-CC1/2
|
||||
under the candidate group (scale/sigma-invariant, already ~1 for Benas) instead of / alongside χ², so
|
||||
genuine high symmetry survives a broken error model without re-admitting pseudo-symmetry. Must be
|
||||
validated against the crystals the χ² gate legitimately protects (CQ066, etc.).
|
||||
|
||||
---
|
||||
|
||||
## 3. Automatic resolution cutoff (CC1/2 logistic, "one shell too far")
|
||||
|
||||
Rugnux currently merges/reports/writes to the most optimistic resolution (integration d_min hardcoded
|
||||
1.0 Å), so a 4 Å crystal gets shells to ~1.5 Å. Desired: pick a cutoff from the **CC1/2 fall-off** fit
|
||||
as a **logistic** in 1/d² (DIALS-style), then keep **one shell past** where it crosses ~0.3 — generous
|
||||
(leaves weak data for ML), not XDS-pessimistic, and it avoids answering "what *is* the resolution".
|
||||
|
||||
CC1/2 (not ⟨I/σ⟩) because it is robust to the broken error model: on Benas ⟨I/σ⟩ < 1 in every shell
|
||||
(an I/σ rule would cut to nothing) while CC1/2 still shows the ~4 Å fall-off. Prototype + experiments
|
||||
in the working scratchpad; lands the physical limit on all test crystals (e.g. lysoC 1.23 Å,
|
||||
Benas ~4.1–4.3 Å) without ever running to 1.5 Å.
|
||||
|
||||
---
|
||||
|
||||
## Restoring the WIP change
|
||||
|
||||
The C2 fix (section 1) is stashed. To resume:
|
||||
|
||||
```
|
||||
git stash list # find the entry with this note
|
||||
git stash apply <stash@{n}> # re-apply RotationIndexer.cpp
|
||||
# build in build_gpu/ (source /opt/rh/gcc-toolset-13/enable first; CUDA on)
|
||||
```
|
||||
Then re-run the battery diff (baseline vs applied) to re-confirm zero regression.
|
||||
@@ -1,88 +0,0 @@
|
||||
<!--
|
||||
SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
SPDX-License-Identifier: GPL-3.0-only
|
||||
-->
|
||||
|
||||
# Rugnux automatic resolution cutoff — design + experiments (WIP)
|
||||
|
||||
Design notes and experiments for an automatic high-resolution cutoff in rugnux scaling/merging
|
||||
(2026-07-09). **Not implemented** — this is the design to build from. Prototype and experiment data
|
||||
live in the working scratchpad (`rescut_logistic.py`, `rescut.py`, run logs).
|
||||
|
||||
## Problem
|
||||
|
||||
Rugnux merges, reports, and writes to the **most optimistic resolution**. There is no
|
||||
resolution-quality cutoff in the merge — the only limit is a manual `--scaling-high-resolution`
|
||||
(default off). The reported high-res edge is set by the prediction/integration d_min, which is
|
||||
**hardcoded to 1.0 Å** (`common/BraggIntegrationSettings.h`, no CLI setter), bounded only by the
|
||||
detector edge. So a crystal that diffracts to 4 Å still gets shells and an MTZ/CIF written to
|
||||
~1–1.5 Å, full of noise. (XDS's `CORRECT.LP` has the same raw behaviour; the difference is that XDS
|
||||
users apply a cutoff downstream.)
|
||||
|
||||
## Design brief
|
||||
|
||||
- Do **not** try to answer "what *is* the resolution" (the ccp4bb debate). Pick a cutoff from the
|
||||
**CC1/2 fall-off**, fit as a **logistic** curve (DIALS-style).
|
||||
- Be **generous, not pessimistic** (XDS-style cuts are too tight): keep **one resolution shell too
|
||||
far**, so maximum-likelihood refinement has a little weak data to work with.
|
||||
- Leave the error model alone.
|
||||
|
||||
## Why CC1/2, not ⟨I/σ⟩
|
||||
|
||||
⟨I/σ⟩ ≥ 1 is a fine limit *when the error model is healthy*, and on the test crystals it reproduces
|
||||
XDS's chosen resolution well (lysoC 1.41 vs 1.43, Ins_I 1.46 vs 1.30, EP_cs_02-10 1.29 vs 1.31). But
|
||||
it fails exactly where a cutoff is most needed: on **Benas** the weak-low-res error model blows up so
|
||||
⟨I/σ⟩ < 1 in **every** shell — an I/σ rule would cut to nothing. **CC1/2 still shows the physical
|
||||
fall-off** there (→ ~4–4.3 Å). CC1/2 is what the brief calls for and is robust to the (untouched)
|
||||
error model.
|
||||
|
||||
## Method
|
||||
|
||||
1. Compute CC1/2 per shell vs `s = 1/d²`. Use **finer bins** than the 10 reported shells for a stable
|
||||
fit (~20–30).
|
||||
2. Fit a logistic `CC½(s) = 1 / (1 + exp(k·(s − s0)))` (low-res plateau ~1, high-res → 0; `s0` =
|
||||
inflection, `k` = steepness). Bounded least-squares (`k > 0`), restricted to the
|
||||
contiguous-from-low-res region so a high-res upward blip can't distort it. (Ceres is already linked;
|
||||
the scratchpad prototype uses a grid+refine fit with no extra dependency.)
|
||||
3. Fall-off limit = `s` where fitted CC½ = **cc_target (default 0.30)**:
|
||||
`s_cut = s0 + ln(1/cc_target − 1) / k`.
|
||||
4. **"One shell too far":** extend by one mean shell width in `s`: `s_final = s_cut + Δs`,
|
||||
`d_final = 1/√s_final`. Clamp to `[d of highest-res reflection, d of lowest-res shell]`.
|
||||
|
||||
## Experiment (logistic + 1 shell, on real rugnux merge tables)
|
||||
|
||||
| crystal | XDS real res | ⟨I/σ⟩≥1 | **logistic + 1 shell** |
|
||||
|----------------|--------------|---------|------------------------|
|
||||
| EcwtCQ244-E3 | ~1.9 | 1.68 | 1.49 |
|
||||
| EcwtCQ244-B1-1 | ~2.4 | 2.36 | 2.10 |
|
||||
| cytC_2 | – | 1.88 | 1.63 |
|
||||
| lysoC_14 | 1.43 | 1.41 | 1.23 |
|
||||
| Ins_I_2 | 1.30 | 1.46 | 1.38 |
|
||||
| EP_cs_02-10 | 1.31 | 1.29 | 1.29 |
|
||||
| **Benas_3** | ~4.5 | none (I/σ<1 all) | **4.30** |
|
||||
| **Benas_7** | ~4.1 | none | **4.10** |
|
||||
|
||||
The logistic sits ~0.1–0.2 Å beyond the strict I/σ limit (the intended generosity), lands at the real
|
||||
limit on the 4 Å Benas where I/σ can't, and never runs to 1.5 Å on a low-res crystal.
|
||||
|
||||
## Implementation plan (in `image_analysis/scale_merge/`)
|
||||
|
||||
- After the final merge computes per-shell CC1/2 (`MergeStatistics`), run the logistic fit + "one
|
||||
shell" extension → `d_cut`. Apply `d_cut` as the high-resolution limit for the **written** MTZ/CIF
|
||||
and the **reported** shell table (filter merged reflections by `d`). Do **not** trim the per-image
|
||||
`_process.h5` (keep full data for re-scaling).
|
||||
- CLI: `--resolution-cutoff cc-logistic|off` (default **cc-logistic**), `--resolution-cc-target <f>`
|
||||
(default 0.30). `--scaling-high-resolution <d>` stays the explicit manual override (wins).
|
||||
- Report the chosen limit + method, e.g. `Auto resolution cutoff: 4.3 A (CC1/2 logistic fall-off,
|
||||
+1 shell; override with --scaling-high-resolution)`.
|
||||
|
||||
## Robustness / edge cases
|
||||
|
||||
- Degenerate fit (too few shells, flat or non-monotone CC1/2): fall back to the full range + warn.
|
||||
- Enforce a low-res floor (never cut into good low-res data).
|
||||
- The fit uses the merge CC1/2 consistently; a v2 could iterate (cut, re-fit), but one pass is enough.
|
||||
|
||||
## Related
|
||||
|
||||
- The broken weak-data error model that makes ⟨I/σ⟩ useless on Benas is also why Benas's cubic point
|
||||
group (F432) is under-determined — see `docs/rugnux_denovo_symmetry_investigation.md`.
|
||||
Reference in New Issue
Block a user