From 4df811bf736d0f0c6eb4bf36d1f606a63c8d6079 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Thu, 9 Jul 2026 16:18:08 +0200 Subject: [PATCH] Remove rugnux work-in-progress documents from Git --- docs/rugnux_denovo_symmetry_investigation.md | 123 ------------------- docs/rugnux_resolution_cutoff_design.md | 88 ------------- 2 files changed, 211 deletions(-) delete mode 100644 docs/rugnux_denovo_symmetry_investigation.md delete mode 100644 docs/rugnux_resolution_cutoff_design.md diff --git a/docs/rugnux_denovo_symmetry_investigation.md b/docs/rugnux_denovo_symmetry_investigation.md deleted file mode 100644 index bc59fe8d..00000000 --- a/docs/rugnux_denovo_symmetry_investigation.md +++ /dev/null @@ -1,123 +0,0 @@ - - -# Rugnux de-novo symmetry / indexing investigation (WIP notes) - -Investigation notes on three de-novo rugnux issues found on hard `/data/rotation_test/` crystals -(2026-07-09). The **C2 fix below is implemented but NOT committed — it is kept in a git stash** -(see "Restoring the WIP change"). This file is the reference to resume from. - -Test crystals: `EcwtCQ244-B1-1`, `EcwtCQ244-E3` (C2), `Benas_3`, `Benas_7` (F432), plus the ~18-crystal -regression battery already in `/data/rotation_test/`. - ---- - -## 1. C2 missed de-novo (indexed as P1) — root cause + WIP fix - -**Symptom.** De-novo (no `-S`), the two Ecwt C2 crystals index perfectly (100 %) but are reported as -**P1**. Their FFT/LatticeSearch cell is the *primitive* cell of the C2 lattice; the C-centering is -never recovered. Forcing `-S 5` gives an excellent C2 merge (E3: 99.7 % complete, ISa ~19), so the -data is genuinely C2 — only the *de-novo determination* fails. - -**Where it fails.** In `RotationIndexer::RunIndexing()` the pseudo-symmetry demotion guard -(`image_analysis/rotation_indexer/RotationIndexer.cpp`, the `if (!sg && sr.system != Triclinic)` -block) refines the metric-symmetry cell and, if it indexes < ½ of a free-triclinic refine, demotes to -P1. For a strongly-oblique C2 (β ≈ 128–132°) that constrained refine **diverges**, so a genuine C2 is -demoted. - -**Why the constrained refine diverges (verified with 4 parallel analysis agents + a numeric sim).** -`XtalOptimizer`'s monoclinic parametrisation (`LatticeToRodriguesLengthsBeta_Mono`) keeps only the -lengths and β and **snaps α, γ to exactly 90°**. LatticeSearch accepts a cell whose α, γ are up to 3° -off 90°, so the idealisation corrupts the metric, and the refine cannot recover. (Numeric check: -exactly-90° monoclinic does not diverge; the collapse is monotonic in the α/γ deviation; the -C-centering `NearestAllowedHKL` snap is *not* the cause — MONO/C ≈ MONO/P.) A second, separate failure -mode of the *full* refine is a length-doubling drift (c → 2c) because a geometry-only spot-fit cost -cannot distinguish a cell from its supercell. - -**The WIP fix (stashed).** In the demotion guard, before demoting: re-run `LatticeSearch` on the -refined triclinic primitive; if it is non-triclinic, retry the recovered symmetry with -**`XtalOptimizerRotationOnly`** (refines orientation only, keeps the *exact* cell metric — no angle -idealisation — and is rotation-norm regularised so it is a stable local move) seeded from the refined -triclinic geometry/axis, on `ls_ref.conventional` (the correct centered basis). Standardise β obtuse -with `ReorderMonoclinic` (an *acute* β breaks the merge d-spacing → "resolution calculation failed"). -Accept if `frac_c > 0.6 * frac_tri` (a genuine centered cell indexes ~0.78× its primitive; a false -pseudo-symmetric cell indexes ~0.01×, so the two are well separated). Otherwise demote to P1 as before. - -**Validation.** Full 21-crystal baseline-vs-fixed battery diff: **only B1-1 and E3 changed (P1 → C2, -both match XDS); all 19 others identical**, including every pseudo-symmetry / difficult case -(CQ066/CQ034/EP_cs_02-424 stay P1; EP_cs_01-17/02-10/MyoB stay P2₁; EcwtAL500 was already C2). -E3 C2: 73.7 % complete, CC1/2 98.5 %, ISa 16.6. B1-1 C2: ~99 % complete (mid-res), ISa 5.7. Change is -`RotationIndexer.cpp` only (~50 lines); `XtalOptimizer` untouched. - -**Open design question (why this is stashed, not committed).** The recovery uses -`XtalOptimizerRotationOnly` while the main path uses the full `XtalOptimizer` — an apparent -inconsistency. Extensive testing showed the alternatives that would let the *standard* optimizer do the -recovery all fail, and clarified *why*: - -- ±15 % per-axis length bounds, ±10° per-axis angle bounds, staged (orientation-pre-refine → full), - and even orientation-only through the standard optimizer **all drift to ~0.5 % indexed**. -- The cost surface is **ill-conditioned for 6-DOF cell refinement** from a ~59 %-indexed centered-cell - start (the gradient points away from the true cell in every direction, so bounds only cap the drift). -- Orientation-only through the standard optimizer still drifts because that optimizer's orientation - refine is **unregularised**; `XtalOptimizerRotationOnly` works only because of its rotation-norm - regulariser. - -So the recovery is really a **re-orientation** step (cell + geometry already known), genuinely -different from the main "index from scratch" refine — and the regularised local orientation refiner is -the right tool for it. The remaining cleanup ideas to evaluate before committing: - -- Give the **main** full refine a length bound and/or an orientation regulariser, so the whole - optimizer does stable *local* improvement and never jumps (e.g. the observed 87 → 131 Å). Then the - recovery and main path share the same philosophy. Needs its own battery validation. -- Or keep the two-tool split but document it as index-vs-refine (as XDS/DIALS do). - ---- - -## 2. Benas_3 / Benas_7 are F432 (a ≈ 70.95 Å), not hexagonal — under-determined point group - -The de-novo cubic a=b=c≈70.2 Å cell rugnux finds is **correct** (XDS's hexagonal 49.9/49.9/121 is the -rhombohedral pseudo-setting: 70.2/√2 ≈ 49.6, 70.2·√3 ≈ 121.6). Two issues: - -1. **Spot resolution.** Default spot d_min 1.5 Å → 0 % indexing (noise). `--spot-high-resolution 4.0` - → 62 %. Low-resolution crystals need spot-finding restricted; auto-detecting this would help. -2. **Point group F222 instead of F432.** With spots restricted, rugnux gets the cubic metric + the - F-centering but adopts **F222** (orthorhombic). SearchSpaceGroup Stage A *does* confirm the cubic - operators (axis-permuting, CC 0.84–0.96, all present), but the point-group **selection** - (`SearchSpaceGroup.cpp`, `chi2 <= chi2_ref * max_merge_chi2_ratio`) rejects 432 because χ² divides - by the sigmas and Benas's weak-data error model is broken (b ≈ 0.55, ISa ≈ 1.8). Forcing `-S 209` - (F432) merges well (multiplicity 31–65, completeness 90–100 %, CC1/2 97–100 %), so the data *is* - F432. - -**Fix direction:** make the point-group consistency gate **sigma-robust** — use CC / merged-CC1/2 -under the candidate group (scale/sigma-invariant, already ~1 for Benas) instead of / alongside χ², so -genuine high symmetry survives a broken error model without re-admitting pseudo-symmetry. Must be -validated against the crystals the χ² gate legitimately protects (CQ066, etc.). - ---- - -## 3. Automatic resolution cutoff (CC1/2 logistic, "one shell too far") - -Rugnux currently merges/reports/writes to the most optimistic resolution (integration d_min hardcoded -1.0 Å), so a 4 Å crystal gets shells to ~1.5 Å. Desired: pick a cutoff from the **CC1/2 fall-off** fit -as a **logistic** in 1/d² (DIALS-style), then keep **one shell past** where it crosses ~0.3 — generous -(leaves weak data for ML), not XDS-pessimistic, and it avoids answering "what *is* the resolution". - -CC1/2 (not ⟨I/σ⟩) because it is robust to the broken error model: on Benas ⟨I/σ⟩ < 1 in every shell -(an I/σ rule would cut to nothing) while CC1/2 still shows the ~4 Å fall-off. Prototype + experiments -in the working scratchpad; lands the physical limit on all test crystals (e.g. lysoC 1.23 Å, -Benas ~4.1–4.3 Å) without ever running to 1.5 Å. - ---- - -## Restoring the WIP change - -The C2 fix (section 1) is stashed. To resume: - -``` -git stash list # find the entry with this note -git stash apply # re-apply RotationIndexer.cpp -# build in build_gpu/ (source /opt/rh/gcc-toolset-13/enable first; CUDA on) -``` -Then re-run the battery diff (baseline vs applied) to re-confirm zero regression. diff --git a/docs/rugnux_resolution_cutoff_design.md b/docs/rugnux_resolution_cutoff_design.md deleted file mode 100644 index e3396305..00000000 --- a/docs/rugnux_resolution_cutoff_design.md +++ /dev/null @@ -1,88 +0,0 @@ - - -# Rugnux automatic resolution cutoff — design + experiments (WIP) - -Design notes and experiments for an automatic high-resolution cutoff in rugnux scaling/merging -(2026-07-09). **Not implemented** — this is the design to build from. Prototype and experiment data -live in the working scratchpad (`rescut_logistic.py`, `rescut.py`, run logs). - -## Problem - -Rugnux merges, reports, and writes to the **most optimistic resolution**. There is no -resolution-quality cutoff in the merge — the only limit is a manual `--scaling-high-resolution` -(default off). The reported high-res edge is set by the prediction/integration d_min, which is -**hardcoded to 1.0 Å** (`common/BraggIntegrationSettings.h`, no CLI setter), bounded only by the -detector edge. So a crystal that diffracts to 4 Å still gets shells and an MTZ/CIF written to -~1–1.5 Å, full of noise. (XDS's `CORRECT.LP` has the same raw behaviour; the difference is that XDS -users apply a cutoff downstream.) - -## Design brief - -- Do **not** try to answer "what *is* the resolution" (the ccp4bb debate). Pick a cutoff from the - **CC1/2 fall-off**, fit as a **logistic** curve (DIALS-style). -- Be **generous, not pessimistic** (XDS-style cuts are too tight): keep **one resolution shell too - far**, so maximum-likelihood refinement has a little weak data to work with. -- Leave the error model alone. - -## Why CC1/2, not ⟨I/σ⟩ - -⟨I/σ⟩ ≥ 1 is a fine limit *when the error model is healthy*, and on the test crystals it reproduces -XDS's chosen resolution well (lysoC 1.41 vs 1.43, Ins_I 1.46 vs 1.30, EP_cs_02-10 1.29 vs 1.31). But -it fails exactly where a cutoff is most needed: on **Benas** the weak-low-res error model blows up so -⟨I/σ⟩ < 1 in **every** shell — an I/σ rule would cut to nothing. **CC1/2 still shows the physical -fall-off** there (→ ~4–4.3 Å). CC1/2 is what the brief calls for and is robust to the (untouched) -error model. - -## Method - -1. Compute CC1/2 per shell vs `s = 1/d²`. Use **finer bins** than the 10 reported shells for a stable - fit (~20–30). -2. Fit a logistic `CC½(s) = 1 / (1 + exp(k·(s − s0)))` (low-res plateau ~1, high-res → 0; `s0` = - inflection, `k` = steepness). Bounded least-squares (`k > 0`), restricted to the - contiguous-from-low-res region so a high-res upward blip can't distort it. (Ceres is already linked; - the scratchpad prototype uses a grid+refine fit with no extra dependency.) -3. Fall-off limit = `s` where fitted CC½ = **cc_target (default 0.30)**: - `s_cut = s0 + ln(1/cc_target − 1) / k`. -4. **"One shell too far":** extend by one mean shell width in `s`: `s_final = s_cut + Δs`, - `d_final = 1/√s_final`. Clamp to `[d of highest-res reflection, d of lowest-res shell]`. - -## Experiment (logistic + 1 shell, on real rugnux merge tables) - -| crystal | XDS real res | ⟨I/σ⟩≥1 | **logistic + 1 shell** | -|----------------|--------------|---------|------------------------| -| EcwtCQ244-E3 | ~1.9 | 1.68 | 1.49 | -| EcwtCQ244-B1-1 | ~2.4 | 2.36 | 2.10 | -| cytC_2 | – | 1.88 | 1.63 | -| lysoC_14 | 1.43 | 1.41 | 1.23 | -| Ins_I_2 | 1.30 | 1.46 | 1.38 | -| EP_cs_02-10 | 1.31 | 1.29 | 1.29 | -| **Benas_3** | ~4.5 | none (I/σ<1 all) | **4.30** | -| **Benas_7** | ~4.1 | none | **4.10** | - -The logistic sits ~0.1–0.2 Å beyond the strict I/σ limit (the intended generosity), lands at the real -limit on the 4 Å Benas where I/σ can't, and never runs to 1.5 Å on a low-res crystal. - -## Implementation plan (in `image_analysis/scale_merge/`) - -- After the final merge computes per-shell CC1/2 (`MergeStatistics`), run the logistic fit + "one - shell" extension → `d_cut`. Apply `d_cut` as the high-resolution limit for the **written** MTZ/CIF - and the **reported** shell table (filter merged reflections by `d`). Do **not** trim the per-image - `_process.h5` (keep full data for re-scaling). -- CLI: `--resolution-cutoff cc-logistic|off` (default **cc-logistic**), `--resolution-cc-target ` - (default 0.30). `--scaling-high-resolution ` stays the explicit manual override (wins). -- Report the chosen limit + method, e.g. `Auto resolution cutoff: 4.3 A (CC1/2 logistic fall-off, - +1 shell; override with --scaling-high-resolution)`. - -## Robustness / edge cases - -- Degenerate fit (too few shells, flat or non-monotone CC1/2): fall back to the full range + warn. -- Enforce a low-res floor (never cut into good low-res data). -- The fit uses the merge CC1/2 consistently; a v2 could iterate (cut, re-fit), but one pass is enough. - -## Related - -- The broken weak-data error model that makes ⟨I/σ⟩ useless on Benas is also why Benas's cubic point - group (F432) is under-determined — see `docs/rugnux_denovo_symmetry_investigation.md`.