AssignRfreeFlags gains a min_free_reflections floor (default 500): on small
data, where the 5% fraction would give too few test reflections for a stable
R-free (Brunger's ~500-2000 rule), the fraction is lifted toward ~500 free
reflections, capped at 10% so a large test set never steals working data. An
explicit rfree_fraction is still honoured; pass 0 to disable.
For ordinary data (5% already clears the floor) this is inactive and the
fraction stays flat, so the cross-dataset-identical property is preserved; the
floor only lifts the fraction on genuinely small datasets, where per-dataset
R-free stability outweighs cross-dataset identity (a shared reference set keeps
exact identity there). Tests updated to isolate the pure-hash guarantees with
the floor off, plus a new small-data floor + cap test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The status-bar pill only filled a progress bar during the Measuring state,
even though the receiver already reports a real progress value during
pedestal/mask calibration. Render the same bar for the Pedestal state
(filled coral to match its pill colour), and relabel the state and its
button as "Calibration"/"CALIBRATE". The internal state value stays
"Pedestal" and the /pedestal endpoint is unchanged for back-compatibility.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks the --model / French-Wilson / R-free path so a batch of datasets of
one crystal form yields comparable maps and a shared test set, and fixes a
French-Wilson prior bug.
- French-Wilson: fix inverted centric/acentric prior - the call passed
!centric into the selector, swapping the two Wilson forms for every weak
reflection. Add a test pinning a centric weak |F| below an acentric one.
- R-free flags: assign by a pure per-hkl hash instead of a per-dataset
resolution-shell stratification, so every dataset of one crystal form gets
the same free set (needed for ensemble refinement / PanDDA). Still a pure
function of the Laue-ASU key, so Bijvoet/symmetry mates never split.
- R-free flags: import a reference MTZ's FreeR_flag column when present
(CCP4 test=0, auto-complement for a phenix-style test=1), so a whole
campaign inherits one shared test set. New ApplyReferenceFreeFlags, wired
post-merge in both the full-analysis and --scale paths.
- Model maps: scale the model with an overall scale, anisotropic B and flat
bulk solvent only - drop the free-form per-shell K(1/d^2) rescale, which
reshaped each dataset's radial amplitude profile differently and made a
batch of maps non-comparable.
- Merohedral indexing: model validation resolves the ambiguity by lowest
R-free only when there is no reference; with a reference MTZ the indexing
is already resolved against it (merge stage / stills scaling) and that
authoritative choice is kept.
- Remove a stale <algorithm> include; update docs and changelog.
Tests: french_wilson, rfree, reference_mtz (new), reindex, hkl_key all pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A weakly-scaled genuine high-symmetry crystal (cubic insulin Ins_I_3) was
demoted I23 -> I222: its cubic 3-fold passes the operator-CC gate, but merging
across it inflates the within-orbit chi2 to ~2x the best subgroup - just past
the self-consistency bound. A merohedral twin (Ins_H_2's R3->R32) lands at the
same chi2 ratio, so the ratio alone cannot separate a genuine-but-poorly-scaled
promotion from a twin.
Add a systematic-error test that does separate them. merge_systematic_b solves
for the extra intensity-proportional error b (sigma^2 + (b I)^2) that a point
group's merge must invoke to reach chi2 = 1. A genuine symmetry step gains
multiplicity without intensity-proportional disagreement, so b barely grows
(Ins_I_3 b x1.6 vs its confirmed subgroup); a twin forces non-equivalent
reflections together, so b balloons (Ins_H_2 x2.2). A candidate whose chi2 is
only just past the ratio bound (max_merge_chi2_rescue) is rescued when its
b, relative to the largest confirmed subgroup (by rotation-set inclusion),
stayed within max_systematic_b_ratio.
Battery: 20/24 space-group match (was 19/24) - Ins_I_3 -> I23, every other
crystal bit-identical including the twins that must stay in lower symmetry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document, in CPU_DATA_ANALYSIS.md, how a reference MTZ fixes the space group
and cell and resolves the indexing (merohedral) ambiguity by CCref (new
§10.9), and how model validation reindexes the data into the model's
enantiomorph and probes merohedral reindexings by R-free (new §14.5); add the
matching rc.159 changelog entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ModelValidation now fits the scaled model to each reindexing of the observed
reflections (identity plus the twin laws from ReindexAmbiguityOperators) and
keeps the one with the lowest R-free. The scale + per-shell + R computation is
factored into a per-candidate lambda so every candidate is scored identically.
No-op for a holohedral crystal (no twin laws), e.g. lysozyme, where the only
candidate is the identity. The enantiomorph/screw ambiguity is still resolved
from the model hand, not probed - |Fcalc| is the same for both hands so R-free
cannot distinguish them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A reference MTZ is now allowed for rotation data: it fixes the space group
and cell (on the CLI) and resolves the indexing (merohedral) ambiguity, but
is NOT used to scale - the rotation merge stays self-consistent. Previously
rotation + reference was a hard error.
After the space group is determined, ChooseReindex/ReferenceIntensityCC pick
the reindexing whose merged intensities best correlate with the reference and,
if it is not the identity, the twin-law reindex is baked into the integration
outcome (hkl labels only; the cell is unchanged, the reindex is metric-
preserving), the rotation scale-merge is re-ingested and the data re-merged in
that indexing. No-op for a holohedral crystal (no twin laws), e.g. lysozyme.
Stills resolve the per-image ambiguity in ScaleOnTheFly, not here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two ways the processing now adopts symmetry from an external reference:
- Reference MTZ (-z) fixes the space group and unit cell, unless -S / -C override them
(the explicit flag always wins - so -S with the "wrong" enantiomorph is allowed). The cell
is a soft reference: indexing may still drift within tolerance, as with -C.
- rugnux --model: when the data was merged in the enantiomorph of the model's space group
(e.g. data P4(1)2(1)2, model P4(3)2(1)2 - the merged intensities cannot tell them apart),
ValidateAgainstModel reindexes the observed reflections into the model's hand (via gemmi
change_of_hand_op). This keeps the data setting consistent with the model; it does not change
the R-factors, which use |F|.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared helper to resolve the indexing (alternative-indexing / reindexing) ambiguity that
arises when the lattice symmetry is higher than the crystal Laue symmetry (merohedral /
pseudo-merohedral): a crystal can be validly indexed in several ways related by reindexing
operators, and the right one is chosen by agreement with a reference (reference-MTZ
intensities, or later a model) - so no reference-free ambigator (cosym) is needed.
ReindexAmbiguity.{h,cpp}:
- ReindexAmbiguityOperators() = gemmi::find_twin_laws() = the coset operators (empty for a
holohedral crystal, i.e. no ambiguity);
- ReindexReflections() applies an operator to a copy of the merged reflections;
- ChooseReindex() picks identity or the twin law that maximizes a caller-supplied score;
- ReferenceIntensityCC() is a ready scorer (Pearson CC vs a reference, matched by Laue-ASU key).
This does NOT touch the screw-axis / enantiomorph ambiguity (P4_1 vs P4_3), which leaves the
merged intensities unchanged and is adopted from the reference/model rather than scored.
Tests: holohedral -> no operators, merohedral (P4) -> operators present, a misindexed dataset
is recovered by reference CC (score >0.99), a correctly indexed one keeps identity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RfreeFlagsTest: determinism (pure function of the reflection), ~5% free fraction,
Friedel/Bijvoet pairs never split, symmetry equivalents share a flag, resolution
stratification (free set spans shells), and fraction=0 flags nothing.
FrenchWilsonTest: strong reflections reduce to sqrt(I), weak/negative intensities get
a positive amplitude, amplitudes are always finite and non-negative (incl. centric /
epsilon>1 reflections in P4(3)2(1)2), and unusable sigma falls back to sqrt(max(I,0)).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update CPU_DATA_ANALYSIS.md for the new merge/analysis capabilities:
- §10.7 R-free test-set flags: Friedel-merged Laue-ASU key (Bijvoet-safe),
deterministic splitmix64 selection, resolution stratification;
- §10.8 French-Wilson amplitudes: posterior-mean |F| from I and sigma with the
Wilson prior (centric/acentric, epsilon multiplicity, strong-reflection
short-circuit), written as MTZ F/SIGF and mmCIF F_meas_au;
- §14 model-based validation (rugnux --model): Fcalc, flat bulk solvent + overall
and per-shell scaling, R-work/R-free vs a model, and 2Fo-Fc / Fo-Fc maps.
Also refresh the scope list and references, and correct the old "intensities only"
note (the output now carries amplitudes too).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New offline capability: `rugnux --model model.pdb` validates the merged intensities
against a PDB model. It computes structure factors from the model on a grid (IT92 x-ray
form factors, FFT), adds an optimized flat bulk solvent (SolventMasker + gemmi Scaling:
k_sol, b_sol, overall k and anisotropic B) and a per-resolution-shell scale, then reports
R-work / R-free and writes 2Fo-Fc and Fo-Fc CCP4 maps plus a map-coefficient MTZ
(FP, FC, PHIC, FWT/PHWT, DELFWT/PHDELWT, FREE). No refinement of the structure itself;
the model is only re-fractionalized into the data cell (a rigid cell adjustment).
The observed amplitudes are the French-Wilson |F| already on the merged reflections
(MergedReflection.F, computed at end of merge), so the R-free/maps use exactly the same
amplitudes as the written reflection file.
ModelValidation.{h,cpp} holds the logic; wired via --model (ProcessConfig.model_path) into
both the full pipeline (Rugnux::Run write site) and the --scale re-merge path. Links the
unified `gemmi` static target for the model / structure-factor / map headers.
Verified on lyso_ref: R-free 0.346, mean 2Fo-Fc at atom centres +2.16 sigma, maps written.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merged reflections carried only intensities; a naive sqrt(max(I,0)) turns every weak or
negative measurement into a biased (or zero) amplitude. Add a French-Wilson step so the
merged reflections themselves carry proper Bayesian amplitudes.
New FrenchWilson.{h,cpp}: ApplyFrenchWilson() fills F and sigmaF on each MergedReflection
with the posterior mean |F| given I and its sigma under the Wilson prior:
- correct centric vs acentric prior (gemmi is_reflection_centric);
- epsilon (symmetry-enhancement) multiplicity: the shell mean is <I/eps> and the prior
mean for a reflection is eps * <I/eps>;
- numerically stable log-shift integration of the posterior;
- strong reflections (I > 4 sigma) short-circuit to sqrt(I) where the FW bias is negligible;
- unusable I/sigma falls back to sqrt(max(I,0)).
It runs as the last step of the merge routine (both MergeOnTheFly and RotationScaleMerge),
so F/sigmaF are part of the merged result and downstream consumers (file writing AND, later,
map calculation) all use the same amplitudes rather than each recomputing FW and risking
divergence. The writers just emit the fields; extended additively:
- MTZ: F (F) + SIGF (Q) columns;
- mmCIF: _refln.F_meas_au / _refln.F_meas_sigma_au;
- HKL text: F sigmaF appended (rfree flag kept in place for back-compat).
Existing intensity columns are unchanged, so current readers keep working while
phenix.refine (etc.) can now refine against the amplitudes.
Verified on lyso (1.3 A): all 173 negative-intensity reflections get F>0, strong reflections
have F/sqrt(I)=1.000, no NaN, and the mmCIF _refln loop is well-formed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The R-free flags were drawn with a per-reflection std::bernoulli_distribution off a
shared std::mt19937(12345). Two problems: in anomalous mode I(+) and I(-) are separate
merged rows, so a Bijvoet pair could be split across the work and free sets (biasing
R-free); and the draw order depended on merge/call order, so it was not reproducible or
parallel-safe. The resolution-shell grouping was cosmetic — it still drew an independent
Bernoulli per row, so it neither tied mates together nor guaranteed a per-shell quota.
New AssignRfreeFlags() (own file RfreeFlags.{h,cpp}) replaces both call sites (stills
MergeOnTheFly, rotation RotationScaleMerge). It:
- keys each reflection by its Friedel-merged (Laue) ASU index, so all symmetry- and
Friedel-equivalent reflections (including a Bijvoet pair) share one flag and are never
split;
- selects the free set by a deterministic splitmix64 hash of that key (same idiom as the
CC1/2 half-set split), so it is reproducible run-to-run and order-independent;
- stratifies by resolution: exactly round(fraction * n) of the distinct reflections in
each of 20 shells is flagged free.
Verified on an anomalous lyso merge (29536 refl, 5% free): 13548 Bijvoet pairs, 0 split;
free fraction 4.6-5.4% in every resolution shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the vendored GEMMI subset (v0.7.5) with the atomic-model, structure-factor,
bulk-solvent and map machinery so the whole thing builds as one static `gemmi`
library instead of a separate target:
- add the model/SF/map sources compiled into `gemmi`:
pdb, resinfo, polyheur, calculate, eig3, ccp4
- add the v0.7.5 headers these pull in (model.hpp, dencalc.hpp, sfcalc.hpp,
solmask.hpp, scaling.hpp, fourier.hpp, grid.hpp, ccp4.hpp, it92.hpp, ...)
plus third_party/pocketfft (FFT), half, tinydir
Only the low-level string/math/symmetry headers were present before; this makes
the vendored copy a complete, self-consistent gemmi that can read a PDB and do
density / structure-factor / map calculations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
