The asymptote was read only from groups with I/sigma > 20 and required at
least 100 of them, so weak or radiation-damaged data (few strong reflections)
fell back to the higher whole-range b and reported an over-conservative ISa.
Relax the I/sigma threshold in a second tier (>= 50 groups above I/sigma 10)
when the tight one is unmet; the per-group counting-subtracted estimate is
stable down to that threshold. Data with many strong groups is unchanged (it
keeps the tight-threshold asymptote); weak / damaged data now reports the
asymptote its own reflections support instead of falling back.
Battery (32 crystals): no space-group, R_meas or CC1/2 change; strong sets
unchanged; e.g. a room-temperature radiation-damage series recovers ISa 5.4 ->
11.2 (its data already exceeds the reference on CC1/2 and R_meas).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ISa is the I->infinity limit of the signal-to-noise (Diederichs, Acta Cryst.
D66 (2010), 733-740), i.e. the reproducibility of the strongest reflections.
The reported value was taken as 1/b of the whole-intensity-range error-model
fit, whose systematic term b is raised by a mild excess of scatter at
intermediate intensity, so it understated that limit.
Read the asymptote directly instead: for each well-measured reflection group the
counting-subtracted fractional scatter of its symmetry mates estimates the
systematic term, and the robust median over strong groups is the asymptote. It
falls back to the whole-range b when too few strong groups exist (weak or
low-multiplicity data). The reported ISa and the merged-sigma systematic floor
use this value, so merged I/sigma approaches ISa.
The (a, b) fit and the per-observation error model are unchanged, so the merge
weights, anomalous statistics and chi^2 are untouched, and CC1/2 and R_meas
(which do not depend on sigma) are bit-identical. Battery (32 crystals): no
space-group or indexing change, R_meas and CC1/2 unchanged, ISa rises to the
true asymptote of the data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CHANGELOG (rc.160): add the centering rate-gate recovery on weak/low-energy data
and the SigAno anomalous signal-to-noise statistic.
CPU_DATA_ANALYSIS: describe SigAno in the merging-statistics section - its
definition, the standard PDBx mmCIF items, and why a half-set anomalous CC is not
used (complementary halves drive it negative on weak shells) - and note the
floor-independent centering-acceptance test in the space-group-determination notes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compute SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> per report shell and overall
from the full-multiplicity I(+)/I(-) Bijvoet split, store it on
MergeStatisticsShell, and surface it in the console merge-statistics table (new
SigAno column) and the mmCIF via the standard wwPDB PDBx items
_reflns.pdbx_absDiff_over_sigma_anomalous (overall) and
_reflns_shell.pdbx_absDiff_over_sigma_anomalous (per shell) - the dictionary's
home for this quantity, so no jfjoch_ local item is needed (unlike ISa).
SigAno approaches ~0.8 for pure noise and rises above 1 with real anomalous
signal. A half-set anomalous CC is deliberately not used: its two half estimates
are complementary partitions of one observation pool (dI0 + dI1 = 2*dI_full), so
differencing the two Bijvoet hands cancels the large common intensity that keeps
CC1/2 non-negative and, once the anomalous SNR per half drops below 1, drives the
correlation towards -1 rather than 0 - misrepresenting a weak-but-real signal.
Emitted only when an anomalous split was made (has_anom guard), so a non-anomalous
merge keeps its stats block and shell-loop columns byte-for-byte. Export-neutral:
the I(+)/I(-)/IMEAN accumulation is unchanged. On low-energy S-SAD test data SigAno
tracks XDS (13 keV null ~0.89, 6 keV ~1.44, 5 keV ~1.5-1.7) and never goes
spuriously negative.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The de-novo centering gate promoted to a centred lattice only when the mean
signed <I/sigma> of the centering-absent class was <= 0.5x that of the present
class. Weak / low-energy data carry a positive intensity floor (background /
profile leakage) that lifts <I/sigma> of the genuinely-extinct class to ~1.5-2.3
instead of ~0; when the present class is itself weak that additive floor inflates
the ratio past the bound and the true centering is missed. An I-centred cubic
crystal recorded at 5 and 13 keV sat at ratio ~0.57 and stayed primitive P2_1 3,
while the same crystal form at 6 keV (stronger data) correctly merged as I23 - an
energy-dependent space group for one physical crystal.
Add a second, floor-independent acceptance path OR'd with the mean-ratio: the
centering-absent class's strong-reflection rate (violations / absent) versus the
present class's own strong rate. A false centering (absent as strong as present)
fails both arms; a true weak centering passes the rate arm. Battery SG sweep vs
XDS: the low-energy datasets now adopt I23, and no crystal is newly over-centred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The de-novo space-group search's systematic-b veto - which keeps a merohedral
twin in its true lower symmetry - is a RATIO test: reject a point-group promotion
whose merge error-model b balloons past 2x the parent subgroup's. That ratio is
meaningless when the parent b is near zero, on excellent data where symmetry
equivalents already agree almost perfectly. A genuine tetragonal 422 whose 222
parent sits at b=0.008 and 422 at b=0.049 (both tiny, ISa ~20) reads as a 6x
balloon and is wrongly demoted to 222.
Floor the parent b (min_systematic_b_for_veto = 0.05) before forming the ratio,
so the veto only judges a promotion once its b is a non-negligible fraction of I.
A real merohedral twin drives b to ~0.19 regardless of how clean the parent is,
so it is still caught (the insulin R3 + ~20% twin veto still fires, b=0.19 >
0.15). A P41212 58/58/150 crystal at 16 keV goes P212121 -> P41212; full
26-crystal rotation battery: only that crystal changes (its space group now
matches XDS, SG match 22->23/26), every other space group bit-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
De-novo two-pass rotation indexing collapsed on a crystal with a long, finely
spaced axis (a ~150 A c-axis at 16 keV): the unconstrained FFT either collapsed
the long axis to a short sub-multiple or let a denser supercell over-fit the
accumulated cloud - a small global-orientation error throws the many high-order
reflections off along the fine axis, so the true cell scores worst on the raw
cloud, and propagating that inaccurate global orientation to each frame fails.
Add a long-axis rescue that only runs when the standard pass indexes few
validation frames (<50%), so well-indexing crystals are untouched and keep the
fast path: re-run the first pass at a COARSE resolution (low-order reflections
only, where the fine axis stays robust) to recover the true metric, take the
recovered cell with the longest axis directly, then RE-INDEX at full resolution
with that cell as a reference (the -C path) - the reference-cell filter drops the
collapsed/supercell candidates and refines an accurate global lattice.
The scheme feed/index/validate loop is factored into a pick_best() lambda shared
by the standard and constrained passes (the standard pass is behaviour-preserved).
A P41212 58/58/150 test crystal goes 15% -> 99.89% indexed. Full 25-crystal
rotation battery: every other crystal's space group and cell BIT-IDENTICAL, and
the rescue fires only on the failing crystal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NumberTextField quantized every float input to 3 decimals, so the finest
q_spacing reachable through the UI was 0.001 - capping the azimuthal-integration
q-bin count around 200 regardless of the CPU/FPGA backend. Add an optional
`decimals` prop (default 3, unchanged elsewhere) and set it to 5 on the Q
spacing / Low Q / High Q fields, matching the q_spacing minimum of 1e-5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MergeOnTheFly::DeltaCChalfReject assigned its CC1/2 half-sets from a seeded
std::mt19937 drawn in image (call) order, whereas the actual merge, the
rotation merge, the GPU path and the R-free flags all split with the
deterministic HalfForImage(image_id) splitmix64 hash. So the deltaCChalf was
measured on a different half-partition than the reported CC1/2, and was
order-dependent (a reordered outcomes vector gave different halves).
Use HalfForImage(i) - i is the image's stable index, the same image_id AddImage
merges with - so deltaCChalf now reflects the exact CC1/2 split the statistics
report, order-independently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summarise the per-image scale the rotation merge already fits - the CC-to-merge
and mosaicity binned by frame (= dose) across the sweep - and flag a falling
per-image CC (or rising mosaicity) as the radiation-damage / crystal-decay
signature. A data-quality-vs-dose read complementary to the fitted decay
correction; the full per-image table is still written to <prefix>_scaling.txt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A rotation merge now keeps each acentric reflection's Bijvoet split - the
inverse-variance I(+)/I(-) from the same scaled fulls - even when the merge is
Friedel-averaged (the default, no -A). IMEAN stays the primary intensity and is
bit-identical to before; the split is purely additive and scaled
non-anomalously, so a weak anomalous signal is preserved in the output without
having to reprocess with -A. French-Wilson now also fills F(+)/F(-) from the two
hands (one pass, shared Wilson prior).
Writers: the default (Friedel-merged) MTZ and mmCIF now carry IMEAN plus
I(+)/I(-) and F/F(+)/F(-) whenever a reflection has an anomalous split (the
stills path, which computes none, keeps the previous columns). A missing mate or
a centric is emitted as the CCP4 missing-value flag (NaN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rugnux CLI target builds to build*/rugnux/rugnux; the old build*/tools/
rugnux paths are stale leftovers from a previous layout that cmake no longer
updates, so the harness could silently run weeks-old code. Pick the most
recently built binary among the candidates instead of a fixed order.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
De-novo space-group search:
- Determine lattice centering from the strength of the systematically-absent
class (its mean I/sigma vs the present class) instead of a per-reflection
violation count. The count gate was brittle on noisy / obverse-reverse-
twinned data, where enough genuinely-absent reflections cross I/sigma>3 to
trip the 10% bound although the class is several-fold weaker, and wrongly
kept a genuine R-centered lattice primitive. False centerings, whose absent
class is as strong as the present one, stay rejected. Screws keep the count
test (too few axial reflections to average).
- Recover a genuine high-symmetry point group when the merged error model is
badly miscalibrated (weak, low-resolution data whose sigmas are far too
small). The fixed-sigma chi^2 ratio then grows with point-group order for
real symmetry too and wrongly rejects it (a true cubic group reached a
chi^2 ratio ~14); the self-normalising systematic-b test stays valid, so
once chi2_ref shows the ratio is untrustworthy the promotion is confirmed
on the b-test alone. The balloon veto still rejects a twin. This is the gap
left when the earlier log10(chi2_ref) widening was removed for the b-veto.
Also print the adopted space group and unit cell as a clean one-line summary
at the end of the run (de-novo or user-fixed -S), not only in the mmCIF.
Full rotation-test battery: exactly the two intended crystals change (both to
the correct answer, with lower R-meas and higher ISa), every other crystal
unchanged; space-group match 21->22/25.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CHANGELOG: add the Wilson B, decay-CV and space-group b-veto entries; shorten
the verbose modulation / stills-modulation / trimmed-background entries and
remove the dataset identities from them.
CPU_DATA_ANALYSIS: the correction surfaces are now three (add modulation),
on by default, cross-validated on the sigma-independent metric; the background
trimmed mean applies to monochromatic stills as well as rotation (broadband
keeps the sigma-clip); document the dataset-wide Wilson B, the still-partiality
model, and the twin b-veto in the space-group search.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The background trimmed mean is gated on the beam (monochromatic vs broadband),
not on the acquisition mode: a monochromatic still (zero bandwidth) already
gets the trim by default, only broadband (non-zero bandwidth: pink-beam / DMM)
data keep the high-side sigma-clip. The comments, the settings doc and the
--background-trim help wrongly implied the trim was rotation-only / that stills
keep the sigma-clip. Wording only; no behaviour change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The systematic-b test (a genuine symmetry step barely moves the merge error
model's b; a merohedral twin, forcing non-equivalent reflections together,
balloons it) was only ever a RESCUE - it could promote a chi^2-borderline
genuine step but never demote a chi^2-passing one. So a partial twin whose
within-orbit scatter looks self-consistent (merged chi^2 below the ratio
bound) slipped through on chi^2 alone and was over-promoted to the holohedral
group, ballooning b and collapsing the merged ISa.
Make the same test a VETO: a chi^2-passing high-symmetry promotion whose b,
relative to the largest confirmed subgroup, exceeds max_systematic_b_veto is
kept in its true lower symmetry. Calibrated on the rotation-test battery,
where the largest genuine step (a P422 tetragonal) sits at b-ratio ~1.8 and a
merohedral R3->R32 twin at ~2.6 - an empty gap - so 2.0 catches the twin while
never demoting a genuine high-symmetry merge.
Battery space-group match 20->21/25 (an R3 case previously merged as R32 is
now correct), every other crystal unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The point-group self-consistency comments named specific datasets when
illustrating the chi^2-ratio / systematic-b calibration. Describe the
crystallographic situation instead (a merohedral R3->R32 twin, a genuine
cubic step, a tetragonal P41212 case), per the repository's no-sample-names
rule. Comment-only; no behaviour change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RefineDecay's cross-validation still scored the held-out equivalents on a
studentized chi^2, while the absorption/modulation surfaces were switched to a
sigma-independent Rmeas-like agreement (sum|Is-Iref|/sum|Iref|) precisely
because a correction can "pass" a studentized gate by reshaping sigma without
tightening the intensities. The decay slope scales sigma up on late /
high-angle frames, so it is exposed to the same loophole. Score it the same
way.
On the rotation battery this rejects a spurious slope on a mis-indexed
monoclinic (P2_1) case: the studentized gate had accepted a physically absurd
-74 A^2 "decay" that cross-validated yet collapsed the error model (ISa
7.1 -> 1.1) and CC1/2 (to ~2%); the fractional gate scores it a -50% held-out
loss and skips it. A small genuine ~3 A^2 decay on another crystal now
cross-validates and engages, with CC1/2 unchanged. Every other crystal is
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rugnux always writes both MTZ and CIF now (the --scaling-output option was
removed), so passing it made every crystal in the battery fail against the
current binary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a dataset-wide isotropic Wilson B-factor estimate, the analogue of XDS's
"WILSON LINE ... B=" which we did not export. CalcGlobalWilsonB fits ln<I> vs
1/d^2 over the merged reflections (B = -2*slope), skipping the low-resolution
non-linear region (d > 4 A) and shells past the signal limit (<I/sigma> < 1)
so the estimate is insensitive to how far the merged data were carried. It is
diagnostic only - not fed back into scaling - and is written to the mmCIF
(_reflns.B_iso_Wilson_estimate), the printed merge statistics, and the log.
Also harden the per-image Wilson B (CalcWilsonBFactor): accept the fit only
when it is well-correlated and physically plausible (0 < B < 200 A^2), else
leave b_factor unset. A bad frame (an indexing glitch, too few reflections)
otherwise produced a wildly steep Wilson line and a B of several hundred A^2
that polluted the per-image plot; NaN is preferable to garbage.
Diagnostic-only: the merged intensities and every merge statistic are
byte-identical (verified baseline vs modified on the rotation battery).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two new controls in the interactive settings dock:
- Scaling section: a "Detector-plane modulation (stills)" checkbox that toggles
the stills flat-field modulation surface (ScalingSettings::StillsModulation),
alongside the existing decay/absorption "Correction surfaces" checkbox
(rotation modulation is fitted there). Off by default.
- Bragg integration section: a "Background trim" checkbox + fraction that sets
the symmetric trimmed-mean background fraction
(BraggIntegrationSettings::BackgroundTrimFraction; 0 = plain ring mean).
Defaults on at 0.10, matching the integrator default, so the viewer requests
trimming for every dataset (the integration engine still applies its own
per-mode gating).
Viewer-only change; compiles against the current settings API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stills scaling/merging path (ScaleOnTheFly + MergeOnTheFly) had no
correction surfaces, whereas rotation fits decay/absorption/modulation inside
RotationScaleMerge. Serial stills hammer the same detector regions shot after
shot, so a detector-plane (flat-field) systematic is at least as well
determined there as in rotation - it just had no home.
Add MergeOnTheFly::RefineModulation: a 16x16 multiplicative surface over the
predicted detector position, fit against the merged reference and folded into
each reflection's image_scale_corr before the error model and the merge. It
mirrors RotationScaleMerge::ApplyCellSurface - alternating per-cell fit with a
den-weighted geometric-mean gauge (never drifts the overall scale) and
Tikhonov pull to 1 - and is cross-validated the same way: fit the surface on
even images, score the held-out odd equivalents by a sigma-independent
R-meas-like agreement (a fractional metric a sigma-reshaping surface cannot
game), and apply the full-data surface only if the held-out gain clears a
margin. A no-op when the systematic is absent or the data too sparse (< 8 obs
per cell), so it is safe to leave off by default and opt in.
Wired through ScalingSettings::StillsModulation and rugnux --stills-modulation
(default off), in both the full-analysis and --scale paths. On a 17 MP
JUNGFRAU serial set the surface cross-validates with a large held-out gain,
confirming a real detector-plane systematic; whether it improves the merged
data quality is under evaluation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The local Bragg background is the mean over the r2..r3 ring. That mean reads
high because the contaminants that survive the signal-disk mask - neighbour-
spot wings, tails, zingers - are one-sided (positive), so it over-subtracts.
Since a weak intensity is a small difference of large numbers (I = S - nS*b),
a per-pixel background bias is fractionally largest at the resolution edge,
exactly where it hurts most.
Replace the ring mean with a symmetric trimmed mean (sort the ring, drop the
lowest and highest fraction f, average the rest), controlled by a new
BraggIntegrationSettings field and the rugnux `--background-trim <f>` option
(default f=0.10; 0 restores the plain mean). Default on for monochromatic
(rotation) data; broadband (stills) keep their tuned high-side sigma-clip, so
the base engine forces the trim to 0 there. Implemented in both the CPU engine
and the GPU kernel (shared-memory bitonic sort per block, flat-mean fallback
above BKG_TRIM_MAX ring pixels); the two agree.
25-crystal rotation battery (fixed SG/cell): <I/sigma> improved on every
crystal (median +50%), ISa on 20/22, resolution-edge R_meas fell several-fold
(e.g. lyso_ref 1.0 A 108%->43%). Last-shell CC1/2 is rescued where the plain
mean had collapsed to noise (Thau_9 at ~2.0 A 3.8%->64%, ~0.5 A of resolution
regained; cytC_10 0.2%->10%) at a small cost (1-3%) in already-clean shells -
it flattens the CC1/2 fall-off rather than shifting it. Stills unchanged.
Documented in CPU_DATA_ANALYSIS.md section 9.2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflection.completeness (spot-footprint fraction) was only ever set to its
1.0f default and never read anywhere - not in scaling/merge, CBOR, HDF5, or
the viewer. The box-sum integrator already rejects any partial-footprint
reflection outright (n_inner_valid == n_inner), so the intended "down-weight
clipped spots" role is already covered more strictly. Removed to reduce
confusion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rotation scaling gained a third cross-validated correction surface alongside
decay and absorption: a smooth multiplicative factor over the predicted detector
position (px, py) where each full lands. It absorbs detector-response and
geometric flat-field systematics that vary across the detector plane and inflate
R-meas; symmetry-equivalents of one reflection land at different detector
positions as the crystal rotates, over-determining the surface. Because it lives
in the detector frame (not the rotation), the same correction concept applies to
stills.
The surface fit/cross-validate/apply machinery shared with the absorption surface
is factored into ApplyCellSurface; RefineAbsorption and the new RefineModulation
just build their per-full cell assignment (goniometer-frame vs detector-frame)
and call it. The cross-validation now scores a sigma-INDEPENDENT, R-meas-like
fractional agreement of the held-out equivalents rather than a studentized chi^2,
so a surface can no longer pass CV by reshaping sigma without tightening the
intensities - this removes an over-fit regression on mis-indexed data and hardens
the absorption surface too.
On the /data/rotation_test rotation battery (A/B, modulation off vs on): 17 of 22
processed crystals improve R-meas, 0 regress, e.g. lysoC 23.2->16.3%, lyso_2
47.4->28.5%, EcwtAL500 53.2->27.6%, EP_cs_01-17 CC1/2 60->93%; CC1/2 held or
improved everywhere and the anomalous S signal preserved/improved (lyso_ref
SD_MET 4.37->4.46, lysoC 3.07->3.25). On by default with the other surfaces.
Also: --dump-observations now writes the per-full centroid frame and predicted
px/py columns, so the dumped combined fulls are a complete unmerged export.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merged reflections were written in one of mtz/cif/txt selected by
--scaling-output. Write both an MTZ and an mmCIF unconditionally instead - each
has its uses downstream (MTZ for the CCP4/phenix tools, mmCIF for deposition) -
and remove the format selector, the plain-text .hkl writer, and the now-unused
IntensityFormat enum / ScalingSettings::FileFormat plumbing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In anomalous mode the merge keeps the two Friedel mates as separate rows,
which WriteMtzReflections emitted verbatim - two IMEAN rows per Bijvoet pair
that downstream tools had to re-collapse. Pair the mates into one row per
reflection with the standard CCP4 anomalous layout: IMEAN + I(+)/I(-) and the
matching F/F(+)/F(-) amplitudes. A single HKLKeyGenerator(merge_friedel=false)
yields both the shared ASU group key and which mate a row is (.plus); IMEAN/F
are the inverse-variance Friedel mean, centrics/unpaired keep one hand with the
other missing. aimless/mtz2sca/ANODE now read the file directly. The
non-anomalous MTZ path is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
VERSION was bumped to rc.160 but the new items sat under the rc.159
heading, which released (origin/main) does not contain. Add a proper
rc.160 section, move the three misfiled items (supercell fix, -S symbol,
FFTW-on-GPU error) into it, and document the rest of the branch (stills
geometry refinement, -r flex, stills partiality, r=6 box default,
single-pass self-referenced scaling, reference-MTZ-never-a-scale-anchor,
E^2 second-lattice cap, --scale on _process.h5 fix, new spot options,
CBF/TIFF writer removal, negative total_flux, viewer, packaging).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The offline rugnux path never ran CheckDataProcessingSettings (only the
online broker/receiver paths did), so the new --min-pix-per-spot and
--spot-low-resolution knobs were unbounded: --min-pix-per-spot 0 silently
disabled the per-spot filter, and a low-resolution limit finer than the
high-resolution limit made spot finding reject every pixel with no
diagnostic.
Add a parse_number_arg<T> helper (integral or floating, with optional
inclusive bounds) that rejects non-numeric input, trailing garbage, and
out-of-range values, replacing the raw atoi/atof in the spot-finding
options; parse_double_arg/parse_float_arg become thin wrappers over it.
After assembling the settings, validate them with the same
CheckDataProcessingSettings the online receivers use, which also enforces
the cross-field low>=high-resolution constraint.
Also scrubs a residual sample-name comment (the r=6 box-default note) in
this file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CBF/TIFF writer removal left two docs describing them as available
output: JFJOCH_WRITER.md's "Other formats" section and an aside in
HDF5.md. A user following them would issue a now-rejected request. State
that only NXmx HDF5 is written and that the CBF/TIFF enum values are
retained for wire back-compatibility but rejected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rc.160 sample-identity cleanup left protein names, internal dataset
codes, and a measured cell angle in several source comments and one test.
Most sensitive: a measured monoclinic beta tied to an internal code in
Rugnux.cpp and IndexAndRefine.cpp. Rewrite each to describe the
crystallographic situation only (space group / metric relation), keeping
the technical reasoning intact. Comment- and string-only; no behaviour
change. (rugnux_cli.cpp's remaining name is scrubbed in the following
commit, alongside its other change to that file.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
At the twinning-analysis print, detect alternative-indexing (twin-law) operators
for the final cell + space group via ReindexAmbiguityOperators(). When any exist
and no reference was supplied, emit a Warning (also written into the merge stats)
that serial-stills crystals are indexed in a random hand and the ambiguity can
only be broken against a reference (-z / --model). For an obvious merohedral case
(P3/P4/P6...) users expect this, but a PSEUDO-merohedral metric is easy to miss --
e.g. NmHR / 7O8F, where C2 beta=131.8 is pseudo-F-orthorhombic and gemmi reports
the reindex operator h-l,-k,-l. Advisory only; no change to processing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a reference cell is supplied, PostIndexingRefinement::Refine() vetted
candidates on sorted edge lengths only; a cell with the right edges but a wrong
angle (a pseudo-symmetric near-metric, e.g. a monoclinic beta refined to the
wrong value) passed. Add a sorted-angle check against the reference, folding
each angle to its acute complement min(x,180-x) so the obtuse/acute setting
choice is irrelevant (tolerance 10 deg).
Reference-cell path only (no effect de novo, where reference_unit_cell is null):
the /data/rotation_test battery is byte-identical (22/25), and NmHR with the
correct C2 cell is unchanged (9.1%, beta~131.7). A latent guard against
wrong-angle cells slipping through on -C / reference-MTZ runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RotationIndexer's per-candidate XtalOptimizer relied on the struct-default
monoclinic angle bound [60,120], the same latent clamp fixed for the stills
path: a rotation crystal with beta>120 (or <60) would have its cell clamped to
the boundary. Set [30,150] explicitly, matching the indexers.
No change on the /data/rotation_test battery (22/25 SG match, byte-identical) --
no battery crystal has beta outside [60,120]; this is a latent-correctness fix
for future high-beta monoclinic rotation data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stills per-frame XtalOptimizer in IndexAndRefine used the struct-default
monoclinic angle bound [60,120] deg, which clamps a beta outside that window
(e.g. NmHR / 7O8F, C2 beta=131.78) to the boundary and corrupts the per-frame
cell. Both indexers already use [30,150] (FFBIDXIndexer, FFT settings); this
aligns the per-frame refine with them. Rotation path is unaffected
(refine_unit_cell is false there).
Verified on NmHR serial stills: FFBIDX with the correct C2 cell went from
0.01% to 9.10% indexed, recovering the true mC cell (beta~131.7) instead of a
clamped beta=120; 60k-image merge reaches 2.61 A at 99.8% completeness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IndexerThreadPool built its per-worker indexers from the raw requested algorithm
but dispatched the RESOLVED one. Rotation indexing always resolves to the GPU FFT
indexer when a GPU is present, so requesting FFTW (the CPU indexer) on a GPU node
created only the CPU indexer and then failed at dispatch with an opaque "no
indexer available for the resolved indexing algorithm" deep inside a worker.
Validate servability at the pool entry (in the caller's context) and throw an
actionable message instead: FFTW is not available on a GPU node (use FFT/Auto, or
a CPU-only node); FFT/FFBIDX require a GPU. Stills FFTW on a GPU node, which the
algorithm resolution still honours, keeps working.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Datasets may be confidential; sample names and measured unit cells committed to
the repo can leak outside the group working on them. Scrub existing occurrences
and add a "No sample identities in the repository" section to CLAUDE.md
(forbidden: sample/dataset names, internal codes, measured cells tied to a
sample; fine: space group / lattice / twinning descriptors).
- Comments: replace internal dataset codes and protein names with the
crystallographic situation they illustrate (centred vs pseudo-symmetric,
holohedral, cubic, F-cubic/hexagonal, ...).
- Docs: same, in the analysis/writer/stream references and example configs.
- Tests: rename sample-named identifiers, TEST_CASE names, file prefixes and
asserted labels to neutral crystallographic names (e.g. tetragonal_uc);
behaviour unchanged. Reduce the CrystFEL reference PDB to a bare CRYST1 cell
file (cell.pdb) and rename the reference data file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two-pass rotation indexing scored each first-pass scheme by how many
validation frames it indexes, but a spurious axis multiple (2x/3x...)
indexes every frame its true sub-cell does, so the count saturates - both
schemes reach the same frame total - and the tie fell to whichever scheme
ran first. When that first scheme's full-rotation FFT resolves a true
axis only as a 2x/3x harmonic of its length (the fundamental can sit far
below the harmonic in the full-360 cloud), it commits a multiplied cell
and the dataset collapses to P1, even though the other scheme already
found the true cell.
Add an integer-supercell cross-scheme tie-break: when two schemes tie on
frames but their cell volumes differ by a near-integer factor >=2, the
larger is the spurious supercell and the smaller true cell is adopted,
independent of scheme order. The near-integer test distinguishes a real
axis multiplication from a centering coincidence (a rhombohedral H cell
vs its C2 sub-cell is 1.5x and is left alone).
Validated de-novo across the rotation regression battery: the affected
dataset now indexes its correct space group, the override triggers only
where an axis was multiplied, and every other dataset is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
-S now takes either a number ("92") or a Hermann-Mauguin symbol
("P43212"), resolved via gemmi find_spacegroup_by_name. Previously a
non-numeric argument was atoi'd to 0 and tripped the early space-group
validation, aborting the run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fitToViewShorterSideOnce() skips the initial fit-to-view while the viewport has
no real size yet (before the widget is laid out/shown) and relies on a retry
that was never wired: resizeEvent only reset the scene rect, and there is no
showEvent. When the first Redraw() lands before layout settles the view stays
at 1:1, so a small grid-scan plot renders tiny ("zoomed out, not taking the
full picture"); the layout-timing race is why it happened only sometimes.
resizeEvent now retries fitToViewShorterSideOnce() while the initial fit is
still pending (guarded by !initial_fit_done_, so a later user resize never
overrides a manual zoom). It lives in the shared base class, so all image views
get the same robustness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the recently-added stills knobs into jfjoch_viewer - they were reachable
only from the rugnux CLI, and the viewer had silently diverged from the CLI once
--refine-geometry became default-on for stills-with-cell.
- Stills geometry-refinement two-pass (--refine-geometry): a checkbox + frame
count in the reprocessing-job dialog, offered (and defaulted on, matching the
CLI) only for a stills-with-cell run; wired into ProcessConfig and the
"Copy command" generator (joined =N, or =off to reproduce opting out of the
CLI default-on).
- Stills partiality (--still-partiality) checkbox in Bragg integration and the
partiality-uncertainty merge term (--partiality-uncertainty) in Scaling, both
mirrored in the command-line generator.
Rename the "best per-image refinement" mode from "multi" to "flex" everywhere:
GeomRefinementAlgorithmEnum::Flex, CLI -r flex (with "multi" kept as a hidden
back-compat alias), OpenAPI enum "Flex" + regenerated C++/TS clients, viewer
combo, frontend dropdown, and the stored HDF5 string. "multi" collides with
CrystFEL's multi-lattice sense; "flex" = let the pipeline decide. The API now
carries a dedicated Flex value instead of masquerading as BeamCenter.
Also fix a pre-existing bug in Convert(IndexingSettings): OrientationOnly was
missing from the outbound switch and serialized as an invalid API value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delete CBFWriter and the TIFF write path from jfjoch_writer so the writer
only produces the NXmx HDF5 formats. Drop CBF/TIFF from the internal
FileWriterFormat enum and reject them in FileWriterSettings.
The CBF/TIFF values are kept (marked deprecated) in the file_writer_format
OpenAPI enum for back compatibility: incoming requests using them are now
rejected in OpenAPIConvert with a clear "no longer supported" error, and a
stale CBF/TIFF value on the CBOR wire decodes to unset rather than a removed
enumerator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two stills usability fixes.
1. --refine-geometry now defaults ON for stills whenever a reference cell is
available (-C or a reference MTZ) - exactly the case where the geometry
bundle-adjust can act (it anchors on a known cell) and where it lifts weak/
sparse-stills indexing (OCP +42%, KR2 +108% indexed in the target study). It
stays a no-op for rotation (own two-pass) and de-novo stills (no cell yet), so
auto-enabling it only where it does something avoids spurious "skipping"
warnings. --refine-geometry=off opts out; explicit --refine-geometry[=N] forces it.
2. The reference-MTZ auto column selection now falls back to a plain amplitude
column (FP / FOBS / F / FC, squared to an intensity) after F-model and the
intensity (J) columns. A deposition that carries only structure-factor
amplitudes (e.g. KR2's 8cl8, whose only usable column is FP) now seeds CCref
without needing an explicit --reference-column FP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a two-lattice crystal a second lattice deposits intensity on one reciprocal
position but not its symmetry mate, so a contaminated reflection is a one-sided
resolution-normalised-E outlier that poisons the operator I(h)/I(Rh) correlation
in SearchSpaceGroup Stage A. On EP_cs_02-424 under -A this dropped the monoclinic
2-fold CC to 0.326 (below the 0.5 gate) -> the crystal was under-called P1 instead
of P2_1. The existing --reject-outliers cannot see it: it rejects WITHIN a P1 orbit,
but the poison is a BETWEEN-orbit effect (inflates orbit h, not its mate Rh).
Add SearchSpaceGroupOptions::max_e_squared_for_cc (default 9.0, i.e. E>3): after the
resolution-normalised Esq[] is computed, drop the extreme-E tail from the correlation
pairs only (the absence stage keeps the full range - that is where the screw signal
lives). Clean Wilson-distributed data almost never reaches E^2=9 (P ~ 0.01-0.3%), so
it is self-targeting - it trims the overlap tail without touching genuine reflections.
EP_cs_02-424 -A: 2-fold CC 0.326 -> 0.64 -> P2_1 (= XDS). Full 24-crystal rotation
battery (pre/post): only EP_cs_02-424 changed (P1 -> P2_1, SG match 20 -> 21/24); all
23 others bit-identical, including the marginal cubic Ins_I_3 (still I23, carried by
the systematic-b rescue). A Friedel-merge-the-search alternative was tried and rejected
(net-negative: did not fix EP424 and regressed Ins_I_3 -> I222).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two-pass rotation indexer could keep a doubled-axis supercell over its own
primitive sub-cell: when the supercell won candidate (ci) order and the primitive
cleared the indexed-fraction hysteresis by less than the 0.05 "prefer-earlier"
margin, the supercell survived. On pding4_003 full data this gave P222 on a
65.6x131.3x173 cell instead of P422 on 65.6x65.6x173 - the orthorhombic metric
forecloses the 4-fold before the symmetry search ever runs. The FFT peak-finding
is correct (the primitive cell is among the candidates, just out-ranked).
Add a sub-cell override to the selection loop: adopt a later candidate that is a
genuinely smaller cell (> ROT_SUBCELL_VOLUME_RATIO=1.5x smaller volume; a doubling
is 2x) indexing at least as many spots (within ROT_SUBCELL_FRAC_SLACK=0.02). That
is the signature of a spurious doubling - the primitive always indexes >= its
integer multiple, whereas a real superstructure's larger cell indexes MORE (kept
by the existing clearly-more branch) and twins share the cell volume (untouched).
Full 24-crystal rotation battery (pre-fix vs post-fix): pding4_003 P222->P422
(cell halves to 65.6x65.6x173, R_meas 6.6% CC1/2 99.9% ISa 11.9), pding4_001
holds P422, all 22 other crystals bit-identical in space group.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stills reflection recorded at partiality p carries a systematic intensity error ~(dp/p) that is
proportional to <I> and grows as p falls; plain counting sigma misses it, so strong low-p partials
are over-trusted in the merge. Add sigma^2 += (c*<I>*(1-p))^2 in MergeOnTheFly::CorrectedSigma - the
stills-partiality analog of the existing rotation --capture-uncertainty term ((1-captured_fraction)*I
in RotationScaleMerge). Inert when partiality==1 (no --still-partiality), and gated on a real
systematic (error_model_b > 1, i.e. ISa < 1) so it fires on strong/medium stills but auto-skips weak
counting-limited data where it would only over-concentrate the merge and hurt. Opt-in via
--partiality-uncertainty <c> (default 0; ~2.5 recommended with --still-partiality).
Prototyped (Python replica of the merge): lyso8 CC1/2 +1.7, CCref +4.8, R-free -0.023; LOV CCref +2;
harmful on weak OCP (hence the b>1 gate). Full in-binary validation across all serial targets pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Weak/jet serial stills are often geometry-limited: a few-px beam error or a mm-scale detector
distance error (unreachable per-image) fails many frames, but is well-determined jointly from the
strong frames. Mirror the rotation two-pass: an index-only first pass over a spread sample collects
each indexed frame's spots + assigned HKL + orientation; the strongest ~N (default 200) feed one
Ceres bundle adjustment; the refined geometry is applied and the main pass re-indexes + integrates
+ merges every frame from scratch.
GeometryRefiner (reusing the extracted XtalResidual - the RecipToDetector geometry residual pulled
out of XtalOptimizer, behaviour-preserving): one problem with SHARED beam(2)/distance(1)/cell-length(3)
blocks + a PER-FRAME orientation(3) block, robust Cauchy loss, a cell-length regularizer anchoring the
known cell to break the low-resolution distance<->cell-scale degeneracy, DENSE_SCHUR eliminating the
per-frame orientations, and a 3-round HKL-reassignment / tolerance-tightening loop. Tilt is not refined
(gauge-coupled, zero gain). Opt-in via --refine-geometry[=N]; stills only (rotation untouched).
Validated: KR2 7.58% -> 21.85% (matches CrystFEL's 21.5%; a real ~1.4mm distance error + ~3px beam),
OCP 2.92% -> 4.19% (~3px beam). OFF runs are bit-identical to baseline (XtalResidual extraction
non-regressing; rotation lyso_ref de-novo ISa 17.3 unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Indexing-ambiguity detection/resolution is no longer part of scaling: it
is a dedicated per-image step for stills (or a post-full-merge step for
rotation). Pull the stills per-image resolver out of ScaleOnTheFly into a
new ReindexAmbiguityResolver in ReindexAmbiguity.{h,cpp}, alongside the
existing rotation free functions (ChooseReindex / ReferenceIntensityCC).
Both workflows now share the operator generation and the best-op
selection (new file-local PickBestReindex helper); ChooseReindex is
rewired onto it. ScaleOnTheFly is now purely a scaling engine (no
ambiguity_ops, no resolve_ambiguity flag, no ResolveIndexingAmbiguity).
Pure refactor: behaviour is unchanged. IndexAndRefine now holds a
ReindexAmbiguityResolver and calls Resolve() at the same call site.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
