v1.0.0-rc.159 #69

Merged

leonarski_f merged 17 commits from french-wilson-model-maps into main 2026-07-13 13:54:03 +02:00

Conversation 0 Commits 17 Files Changed 261

+33936 -217

leonarski_f commented 2026-07-13 12:07:02 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• rugnux: Add --model model.pdb - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
• rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ F/SIGF, mmCIF _refln.F_meas_au, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
• rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a FreeR_flag column its test set is imported instead, letting a whole campaign inherit one shared free set.
• rugnux: A reference MTZ (--reference-mtz) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
• rugnux: --model validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
• rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
• Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
• Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal Pedestal state name is unchanged for back-compatibility).

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).

leonarski_f added 17 commits 2026-07-13 12:07:03 +02:00

gemmi_gph: vendor the model / structure-factor headers into the single gemmi target ... 5addb4483c

Extend the vendored GEMMI subset (v0.7.5) with the atomic-model, structure-factor,
bulk-solvent and map machinery so the whole thing builds as one static `gemmi`
library instead of a separate target:

  - add the model/SF/map sources compiled into `gemmi`:
    pdb, resinfo, polyheur, calculate, eig3, ccp4
  - add the v0.7.5 headers these pull in (model.hpp, dencalc.hpp, sfcalc.hpp,
    solmask.hpp, scaling.hpp, fourier.hpp, grid.hpp, ccp4.hpp, it92.hpp, ...)
    plus third_party/pocketfft (FFT), half, tinydir

Only the low-level string/math/symmetry headers were present before; this makes
the vendored copy a complete, self-consistent gemmi that can read a PDB and do
density / structure-factor / map calculations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: harden R-free flag assignment (Bijvoet-safe, deterministic, stratified) ... 4c95a289b6

The R-free flags were drawn with a per-reflection std::bernoulli_distribution off a
shared std::mt19937(12345). Two problems: in anomalous mode I(+) and I(-) are separate
merged rows, so a Bijvoet pair could be split across the work and free sets (biasing
R-free); and the draw order depended on merge/call order, so it was not reproducible or
parallel-safe. The resolution-shell grouping was cosmetic — it still drew an independent
Bernoulli per row, so it neither tied mates together nor guaranteed a per-shell quota.

New AssignRfreeFlags() (own file RfreeFlags.{h,cpp}) replaces both call sites (stills
MergeOnTheFly, rotation RotationScaleMerge). It:
  - keys each reflection by its Friedel-merged (Laue) ASU index, so all symmetry- and
    Friedel-equivalent reflections (including a Bijvoet pair) share one flag and are never
    split;
  - selects the free set by a deterministic splitmix64 hash of that key (same idiom as the
    CC1/2 half-set split), so it is reproducible run-to-run and order-independent;
  - stratifies by resolution: exactly round(fraction * n) of the distinct reflections in
    each of 20 shells is flagged free.

Verified on an anomalous lyso merge (29536 refl, 5% free): 13548 Bijvoet pairs, 0 split;
free fraction 4.6-5.4% in every resolution shell.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: compute French-Wilson amplitudes at end of merge, write them alongside intensities ... 87af0f419b

Merged reflections carried only intensities; a naive sqrt(max(I,0)) turns every weak or
negative measurement into a biased (or zero) amplitude. Add a French-Wilson step so the
merged reflections themselves carry proper Bayesian amplitudes.

New FrenchWilson.{h,cpp}: ApplyFrenchWilson() fills F and sigmaF on each MergedReflection
with the posterior mean |F| given I and its sigma under the Wilson prior:
  - correct centric vs acentric prior (gemmi is_reflection_centric);
  - epsilon (symmetry-enhancement) multiplicity: the shell mean is <I/eps> and the prior
    mean for a reflection is eps * <I/eps>;
  - numerically stable log-shift integration of the posterior;
  - strong reflections (I > 4 sigma) short-circuit to sqrt(I) where the FW bias is negligible;
  - unusable I/sigma falls back to sqrt(max(I,0)).

It runs as the last step of the merge routine (both MergeOnTheFly and RotationScaleMerge),
so F/sigmaF are part of the merged result and downstream consumers (file writing AND, later,
map calculation) all use the same amplitudes rather than each recomputing FW and risking
divergence. The writers just emit the fields; extended additively:
  - MTZ: F (F) + SIGF (Q) columns;
  - mmCIF: _refln.F_meas_au / _refln.F_meas_sigma_au;
  - HKL text: F sigmaF appended (rfree flag kept in place for back-compat).
Existing intensity columns are unchanged, so current readers keep working while
phenix.refine (etc.) can now refine against the amplitudes.

Verified on lyso (1.3 A): all 173 negative-intensity reflections get F>0, strong reflections
have F/sqrt(I)=1.000, no NaN, and the mmCIF _refln loop is well-formed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add --model for R-free and 2Fo-Fc / Fo-Fc maps from an atomic model ... d6924aa901

New offline capability: `rugnux --model model.pdb` validates the merged intensities
against a PDB model. It computes structure factors from the model on a grid (IT92 x-ray
form factors, FFT), adds an optimized flat bulk solvent (SolventMasker + gemmi Scaling:
k_sol, b_sol, overall k and anisotropic B) and a per-resolution-shell scale, then reports
R-work / R-free and writes 2Fo-Fc and Fo-Fc CCP4 maps plus a map-coefficient MTZ
(FP, FC, PHIC, FWT/PHWT, DELFWT/PHDELWT, FREE). No refinement of the structure itself;
the model is only re-fractionalized into the data cell (a rigid cell adjustment).

The observed amplitudes are the French-Wilson |F| already on the merged reflections
(MergedReflection.F, computed at end of merge), so the R-free/maps use exactly the same
amplitudes as the written reflection file.

ModelValidation.{h,cpp} holds the logic; wired via --model (ProcessConfig.model_path) into
both the full pipeline (Rugnux::Run write site) and the --scale re-merge path. Links the
unified `gemmi` static target for the model / structure-factor / map headers.

Verified on lyso_ref: R-free 0.346, mean 2Fo-Fc at atom centres +2.16 sigma, maps written.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: document French-Wilson amplitudes, R-free flags and model-based maps ... 97e6d99d03

Update CPU_DATA_ANALYSIS.md for the new merge/analysis capabilities:
  - §10.7 R-free test-set flags: Friedel-merged Laue-ASU key (Bijvoet-safe),
    deterministic splitmix64 selection, resolution stratification;
  - §10.8 French-Wilson amplitudes: posterior-mean |F| from I and sigma with the
    Wilson prior (centric/acentric, epsilon multiplicity, strong-reflection
    short-circuit), written as MTZ F/SIGF and mmCIF F_meas_au;
  - §14 model-based validation (rugnux --model): Fcalc, flat bulk solvent + overall
    and per-shell scaling, R-work/R-free vs a model, and 2Fo-Fc / Fo-Fc maps.
Also refresh the scope list and references, and correct the old "intensities only"
note (the output now carries amplitudes too).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: add rc.159 changelog (rugnux --model, French-Wilson amplitudes, R-free flags) ... e7fe203625

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

tests: add Catch2 tests for R-free flag assignment and French-Wilson ... 9f23976e02

RfreeFlagsTest: determinism (pure function of the reflection), ~5% free fraction,
Friedel/Bijvoet pairs never split, symmetry equivalents share a flag, resolution
stratification (free set spans shells), and fraction=0 flags nothing.

FrenchWilsonTest: strong reflections reduce to sqrt(I), weak/negative intensities get
a positive amplitude, amplitudes are always finite and non-negative (incl. centric /
epsilon>1 reflections in P4(3)2(1)2), and unusable sigma falls back to sqrt(max(I,0)).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: add a reindexing-ambiguity resolver ... d00f4303c8

Shared helper to resolve the indexing (alternative-indexing / reindexing) ambiguity that
arises when the lattice symmetry is higher than the crystal Laue symmetry (merohedral /
pseudo-merohedral): a crystal can be validly indexed in several ways related by reindexing
operators, and the right one is chosen by agreement with a reference (reference-MTZ
intensities, or later a model) - so no reference-free ambigator (cosym) is needed.

ReindexAmbiguity.{h,cpp}:
  - ReindexAmbiguityOperators() = gemmi::find_twin_laws() = the coset operators (empty for a
    holohedral crystal, i.e. no ambiguity);
  - ReindexReflections() applies an operator to a copy of the merged reflections;
  - ChooseReindex() picks identity or the twin law that maximizes a caller-supplied score;
  - ReferenceIntensityCC() is a ready scorer (Pearson CC vs a reference, matched by Laue-ASU key).

This does NOT touch the screw-axis / enantiomorph ambiguity (P4_1 vs P4_3), which leaves the
merged intensities unchanged and is adopted from the reference/model rather than scored.

Tests: holohedral -> no operators, merohedral (P4) -> operators present, a misindexed dataset
is recovered by reference CC (score >0.99), a correctly indexed one keeps identity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: fix SG/cell from a reference MTZ, and reindex data into the model's enantiomorph ... e91b520863

Two ways the processing now adopts symmetry from an external reference:

- Reference MTZ (-z) fixes the space group and unit cell, unless -S / -C override them
  (the explicit flag always wins - so -S with the "wrong" enantiomorph is allowed). The cell
  is a soft reference: indexing may still drift within tolerance, as with -C.

- rugnux --model: when the data was merged in the enantiomorph of the model's space group
  (e.g. data P4(1)2(1)2, model P4(3)2(1)2 - the merged intensities cannot tell them apart),
  ValidateAgainstModel reindexes the observed reflections into the model's hand (via gemmi
  change_of_hand_op). This keeps the data setting consistent with the model; it does not change
  the R-factors, which use |F|.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: resolve the indexing ambiguity against a reference MTZ (rotation) ... 465cf1aa5e

A reference MTZ is now allowed for rotation data: it fixes the space group
and cell (on the CLI) and resolves the indexing (merohedral) ambiguity, but
is NOT used to scale - the rotation merge stays self-consistent. Previously
rotation + reference was a hard error.

After the space group is determined, ChooseReindex/ReferenceIntensityCC pick
the reindexing whose merged intensities best correlate with the reference and,
if it is not the identity, the twin-law reindex is baked into the integration
outcome (hkl labels only; the cell is unchanged, the reindex is metric-
preserving), the rotation scale-merge is re-ingested and the data re-merged in
that indexing. No-op for a holohedral crystal (no twin laws), e.g. lysozyme.

Stills resolve the per-image ambiguity in ScaleOnTheFly, not here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: probe indexing ambiguities against the model in validation ... 386f345758

ModelValidation now fits the scaled model to each reindexing of the observed
reflections (identity plus the twin laws from ReindexAmbiguityOperators) and
keeps the one with the lowest R-free. The scale + per-shell + R computation is
factored into a per-candidate lambda so every candidate is scored identically.

No-op for a holohedral crystal (no twin laws), e.g. lysozyme, where the only
candidate is the identity. The enantiomorph/screw ambiguity is still resolved
from the model hand, not probed - |Fcalc| is the same for both hands so R-free
cannot distinguish them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: reference-based space-group/ambiguity resolution and model alignment ... b350c46777

Document, in CPU_DATA_ANALYSIS.md, how a reference MTZ fixes the space group
and cell and resolves the indexing (merohedral) ambiguity by CCref (new
§10.9), and how model validation reindexes the data into the model's
enantiomorph and probes merohedral reindexings by R-free (new §14.5); add the
matching rc.159 changelog entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: rescue a genuine high-symmetry merge from the chi2 gate by its systematic error ... f81830df4f

A weakly-scaled genuine high-symmetry crystal (cubic insulin Ins_I_3) was
demoted I23 -> I222: its cubic 3-fold passes the operator-CC gate, but merging
across it inflates the within-orbit chi2 to ~2x the best subgroup - just past
the self-consistency bound. A merohedral twin (Ins_H_2's R3->R32) lands at the
same chi2 ratio, so the ratio alone cannot separate a genuine-but-poorly-scaled
promotion from a twin.

Add a systematic-error test that does separate them. merge_systematic_b solves
for the extra intensity-proportional error b (sigma^2 + (b I)^2) that a point
group's merge must invoke to reach chi2 = 1. A genuine symmetry step gains
multiplicity without intensity-proportional disagreement, so b barely grows
(Ins_I_3 b x1.6 vs its confirmed subgroup); a twin forces non-equivalent
reflections together, so b balloons (Ins_H_2 x2.2). A candidate whose chi2 is
only just past the ratio bound (max_merge_chi2_rescue) is rescued when its
b, relative to the largest confirmed subgroup (by rotation-set inclusion),
stayed within max_systematic_b_ratio.

Battery: 20/24 space-group match (was 19/24) - Ins_I_3 -> I23, every other
crystal bit-identical including the twins that must stay in lower symmetry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: make model validation batch-consistent for fragment screening ... d891551127

Reworks the --model / French-Wilson / R-free path so a batch of datasets of
one crystal form yields comparable maps and a shared test set, and fixes a
French-Wilson prior bug.

- French-Wilson: fix inverted centric/acentric prior - the call passed
  !centric into the selector, swapping the two Wilson forms for every weak
  reflection. Add a test pinning a centric weak |F| below an acentric one.
- R-free flags: assign by a pure per-hkl hash instead of a per-dataset
  resolution-shell stratification, so every dataset of one crystal form gets
  the same free set (needed for ensemble refinement / PanDDA). Still a pure
  function of the Laue-ASU key, so Bijvoet/symmetry mates never split.
- R-free flags: import a reference MTZ's FreeR_flag column when present
  (CCP4 test=0, auto-complement for a phenix-style test=1), so a whole
  campaign inherits one shared test set. New ApplyReferenceFreeFlags, wired
  post-merge in both the full-analysis and --scale paths.
- Model maps: scale the model with an overall scale, anisotropic B and flat
  bulk solvent only - drop the free-form per-shell K(1/d^2) rescale, which
  reshaped each dataset's radial amplitude profile differently and made a
  batch of maps non-comparable.
- Merohedral indexing: model validation resolves the ambiguity by lowest
  R-free only when there is no reference; with a reference MTZ the indexing
  is already resolved against it (merge stage / stills scaling) and that
  authoritative choice is kept.
- Remove a stale <algorithm> include; update docs and changelog.

Tests: french_wilson, rfree, reference_mtz (new), reindex, hkl_key all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: show calibration progress and relabel Pedestal as Calibration ... 069b00349f

The status-bar pill only filled a progress bar during the Measuring state,
even though the receiver already reports a real progress value during
pedestal/mask calibration. Render the same bar for the Pedestal state
(filled coral to match its pill colour), and relabel the state and its
button as "Calibration"/"CALIBRATE". The internal state value stays
"Pedestal" and the /pedestal endpoint is unchanged for back-compatibility.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: floor the R-free test-set size on small data ... 85fcac09b5

AssignRfreeFlags gains a min_free_reflections floor (default 500): on small
data, where the 5% fraction would give too few test reflections for a stable
R-free (Brunger's ~500-2000 rule), the fraction is lifted toward ~500 free
reflections, capped at 10% so a large test set never steals working data. An
explicit rfree_fraction is still honoured; pass 0 to disable.

For ordinary data (5% already clears the floor) this is inactive and the
fraction stays flat, so the cross-dataset-identical property is preserved; the
floor only lifts the fraction on genuinely small datasets, where per-dataset
R-free stability outweighs cross-dataset identity (a shared reference set keeps
exact identity there). Tests updated to isolate the pure-hash guarantees with
the floor off, plus a new small-data floor + cap test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

VERSION: 1.0.0-rc.159 d34c19f6d3

leonarski_f merged commit dd0bffb283 into main 2026-07-13 13:54:03 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#69