v1.0.0-rc.160 #70

Merged

leonarski_f merged 90 commits from 2607-rc-160 into main 2026-07-19 09:39:29 +02:00

Conversation 0 Commits 90 Files Changed 246

+5275 -5155

leonarski_f commented 2026-07-17 21:36:58 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• rugnux: Add --model model.pdb - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
• rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ F/SIGF, mmCIF _refln.F_meas_au, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
• rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a FreeR_flag column its test set is imported instead, letting a whole campaign inherit one shared free set.
• rugnux: A reference MTZ (--reference-mtz) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
• rugnux: --model validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
• rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
• Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
• Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal Pedestal state name is unchanged for back-compatibility).

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).

leonarski_f added 69 commits 2026-07-17 21:36:58 +02:00

docker: static libdbus + disable Qt glib to shed the viewer's .so tail ... 5baf2a2cac

The jfjoch_viewer runtime pulled a large dynamic .so tail that traces to two
Qt-side integrations, neither of which the viewer actually needs:

- Qt6::DBus -> system libdbus-1.so -> libsystemd (libzstd/liblz4/libcap/
  libgcrypt/libgpg-error) + libselinux (libpcre2). The viewer is a pure
  session-bus client (registers ch.psi.jfjoch_viewer + exports an adaptor for
  single-instance/remote-control), so the daemon-side systemd/selinux features
  are irrelevant. Build a static libdbus from source with those integrations
  disabled and link it into the viewer, the same from-source-static pattern
  already used for OpenSSL. Swap the dbus dev package for expat (dbus's
  configure-time dep; the client libdbus-1 links neither expat nor systemd) so
  nothing pulls the system shared dbus back in.

  Qt discovers dbus through find_package(DBus1) (its FindWrapDBus1.cmake), NOT
  pkg-config -- so point it at our build with -DDBus1_DIR=<prefix>/lib/cmake/
  DBus1 (the CMake package config dbus installs, which imports libdbus-1.a) and
  force QT_FEATURE_dbus_linked=ON so QtDBus links the archive instead of
  dlopen'ing a (now nonexistent) libdbus-1.so.3 at runtime.

- Qt's glib event-dispatcher -> libglib-2.0/libgthread/libpcre. Nothing in the
  viewer (or any lib it links) drives a GLib main context, so switch Qt to its
  own QEventDispatcherUNIX via QT_FEATURE_glib=OFF and drop the glib dev package.

Applied identically across rocky8, rocky9, ubuntu2204, ubuntu2404.

Note: system libcrypto still appears at runtime, but it is dragged in by the
dynamic system krb5/GSSAPI (built against OpenSSL on RHEL/Rocky), which we keep
dynamic for the planned httpd-reverse-proxy Kerberos auth -- not from Qt or curl,
which both statically link the from-source OpenSSL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docker: add parallel image-build + in-container viewer-verify scripts ... 254bd82d52

build_images.sh builds (and optionally pushes) the four build-environment
images as gitea.psi.ch/leonarski_f/jfjoch_<variant>:<TAG>. Runs a JOBS-capped
parallel job pool (default 2; each build internally runs make -j$(nproc), so a
low cap avoids CPU/RAM thrash while still overlapping the network-bound base
pull / install / download phases), streams each build to
docker/build-logs/<variant>-<TAG>.log, prints an OK/FAIL summary, pushes only
the ones that succeeded, and exits non-zero on any failure. Context is the tiny
per-variant dir (no COPY in the Dockerfiles), so the repo is never sent to the
daemon. build-logs/ is already covered by the root .gitignore build*/ rule.

build_in_rocky9.sh builds the viewer (JFJOCH_VIEWER_ONLY) inside a chosen
variant's image and ldd-checks that the dbus/systemd/glib/selinux tail is gone
-- the quick verification for the static-libdbus + glib-off changes. The repo is
mounted but the build tree lives in the container's /tmp, so nothing root-owned
lands in the working copy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docker: build the static libdbus with -fPIC so Qt's PIE tools link it ... b6c2d5848d

The static libdbus linked fine into libQt6DBus.a, but Qt links its D-Bus code
generators (qdbusxml2cpp/qdbuscpp2xml) as position-independent executables
(-fPIE -pie), and the default autotools dbus build produces non-PIC objects:

  ld: libdbus-1.a(...dbus-address.o): relocation R_X86_64_32 against `.rodata'
      can not be used when making a PIE object; recompile with -fPIE

Build libdbus with --with-pic and CFLAGS=-fPIC so its archive objects are
position-independent and link into both the static Qt libs and the PIE tools.
Reproduced (non-PIC -> R_X86_64_32 fail) and fixed (PIC -> PIE whole-archive
link passes) in a rocky8 container before applying.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docker: put the static libdbus on CMAKE_PREFIX_PATH for Qt6::DBus consumers ... d90d498cdb

The Qt build found dbus via -DDBus1_DIR, but a static Qt records the dependency
and every consumer re-runs find_package(DBus1) through Qt's exported config
(Qt6Gui -> Qt6DBus -> find_dependency(WrapDBus1)). Without the dbus prefix on
the consumer's search path that fails:

  Could NOT find WrapDBus1 (missing: DBus1_LIBRARY DBus1_INCLUDE_DIR ...)
  Qt6DBus could not be found because dependency WrapDBus1 could not be found.

Add /opt/dbus-<ver>-static to the image's CMAKE_PREFIX_PATH env so the viewer
build (and the CI .deb/.rpm and tarball builds) resolve it. Verified against the
rocky9 image: find_package(Qt6 ... DBus) fails with the old path and passes with
dbus on the env CMAKE_PREFIX_PATH.

Also make build_in_rocky9.sh self-sufficient on images built before this fix:
it discovers /opt/dbus-*-static/lib/cmake/DBus1 and passes -DDBus1_DIR, so the
viewer verification works without rebuilding the image first.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docker: fix build_in_rocky9.sh shed-check for RHEL9's glib/selinux sources ... 3df9ad631b

The old check flagged glib/selinux/pcre as failures, inheriting a rocky8-host
assumption (glib is Qt-only, selinux is dbus-only) that does not hold on RHEL9:

- libglib is pulled by the system harfbuzz/fontconfig/freetype text stack
  (glib-linked on RHEL9), not by Qt. QT_FEATURE_glib=OFF is honoured (Qt's
  enabled_features has no glib) but cannot drop the font stack's libglib.
- libselinux/libpcre2-8 come from the krb5/GSSAPI stack (libgssapi_krb5 ->
  libselinux) that static curl links for auth -- kept for the reverse-proxy
  Kerberos plan, same bucket as libcrypto.

So hard-fail only on the dbus/systemd tail we actually remove (libdbus/
libsystemd/zstd/lz4/cap/gcrypt/gpg-error), and just report glib/selinux as
expected residuals with the reason. Verified in the rocky9 image: that tail is
gone; the survivors trace to the font stack and krb5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

reader: treat negative total_flux as unknown/absent ... 87778c6500

Serial-stills master files can store /entry/instrument/beam/total_flux as a
-1.0 "unknown flux" sentinel (seen in the OCP dark dataset). DatasetSettings
rejects values below 0, so opening such a file hard-failed with
"Input parameter below min (Total flux)". Reset a negative flux to nullopt so
it is treated as absent instead of aborting the read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ScaleOnTheFly: resolve the stills indexing ambiguity per image vs a reference ... 2d7ddb1b67

Serial stills index each crystal in one of the merohedrally-equivalent hands
at random, so the ambiguity must be broken per image, not globally as the
rotation path (RotationScaleMerge/ChooseReindex) does. When an external
reference MTZ is supplied, IndexAndRefine::ReferenceIntensities now builds the
per-image scaler with resolve_ambiguity=true: for each image it picks the
reindexing (identity or a twin law from ReindexAmbiguityOperators) whose
partiality/Lorentz-corrected intensities correlate best with the reference,
then reindexes that image's reflections before scaling. Off for the self-merge
scaling pass (no trusted reference to break the tie).

Holohedral crystals (no twin law, e.g. lysozyme P43212) are a no-op. Validated
on OCP P3221 (twin law -x,-y,z) with a 7ZSJ reference: CCref 24->48% and
CC1/2 24->64% at 30k images; CC1/2 96.9% / CCref 76% at 150k.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add --min-pix-per-spot CLI option (spot finding) ... 2f3041c44e

The minimum connected strong-pixel count per spot was hardcoded to 2
(SpotFindingSettings::min_pix_per_spot). Expose it on the CLI so serial-stills
data can be run with min-pix 1 (plus a higher --spot-threshold), matching the
CrystFEL peakfinder8 practice that trades noisier spots for a higher indexing
rate. Default unchanged (2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add --spot-low-resolution CLI option (exclude beam halo) ... 097b9a29d1

The spot-finding low-resolution limit was hardcoded at 50 A (r ~36 px), which
admits the direct-beam halo/overload zone (24-50 A) on weakly-diffracting serial
data. On KR2 (I222, 235 A axis, ~3.9 A diffraction) that halo is ~387 strong
pixels/frame vs ~25 real Bragg pixels, swamping the signal ~15:1 (CrystFEL cuts
it with min-res=75 px = 24 A). Expose the limit so it can be tightened, e.g.
--spot-low-resolution 24. Default unchanged (50 A).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: fix --scale on a self-contained _process.h5 (reflections + error model) ... db8ae0452e

Offline --scale re-scales the reflections stored in a _process.h5 without
re-integration, but was broken for a self-contained integrated snapshot:

1. HDF5MetadataSource::ReadReflections fell back per-image to the linked
   source pixel files for every non-indexed frame (a snapshot's master holds
   /entry/reflections only for the sparse indexed images). With the raw
   _data_NNNNNN.h5 absent or not co-located this threw an HDF5 error; when
   present it needlessly reopened multi-GB files thousands of times. Decide
   once whether the master is the authoritative reflection store and, if so,
   never fall back - a missing per-image group just means the frame has none.
   Legacy/VDS acquisitions (no /entry/reflections in the master) still resolve
   reflections lazily to the source data files.

2. The scale-only stills branch never fit the (a,b) error model, so it merged
   with the identity model - far worse intensities than the full pipeline
   (lyso8 CC1/2 76%->21%). Fit RefineErrorModel and honour --reject-outliers,
   mirroring Rugnux.cpp.

Round-trip validated: full merge == integrate(--no-merge) + --scale (identical
error model a=0.793 b=2.287, matching unique reflection counts), and --scale
now reads reflections in ~2.5s with no raw-file access.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: default the stills integration box to r=6 (6,8,12) ... b0202a3aa2

Serial-stills spots span a range of crystal orientations within a single shot, so they
land wider on the detector than the r1=4 monochromatic-rotation default assumes, and the
r1=4 signal box truncates the (bandwidth/mosaicity-broadened) spot wings. The profile-fit
integrator can use a larger box for free - its profile weighting drives far-out background
pixels to ~zero weight, so the estimator variance saturates rather than growing (a plain
box-sum, which sums every pixel at weight 1, degrades). Gated on stills (!rotation_indexing);
rotation stays at 4,6,10 (a larger box there only adds background - measured wash-to-negative
on the XDS rotation battery, with an SG regression); an explicit --integration-radius wins.

Validated: OCP R-free 0.406->0.400, LOV 0.354->0.336; lyso8 1.60->1.57 A; lyso5 (1% DMM)
1.80->1.72 A with +13% reflections at equal CC1/2 - the broadened case benefits most.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add opt-in stills partiality (--still-partiality) ... fb399480e5

Serial stills currently treat every reflection as a full (partiality hardcoded to 1).
Add an opt-in Gaussian excitation-error partiality set at prediction time (CPU + CUDA):
p = exp(-dist_ewald^2 / (2*sigma^2)), sigma^2 = profile_radius^2 + (bandwidth_sigma*|recip_z|)^2,
with sigma = the per-image profile radius (ewald_dist_cutoff/2), so an edge-of-acceptance
reflection keeps p ~ exp(-2). Off by default; the merge weight (~p^2) then down-weights
far-from-Ewald partials instead of trusting them as fulls.

Validated: helps medium/strong stills (LOV R-free 0.336->0.329, lyso8 0.433->0.410, lowers
the systematic error-model b in both) but HURTS weak OCP (dividing by a small, uncertain p
amplifies orientation error -> high-res noise, resolution collapse), so it is left opt-in.
A static forward p explains only ~6-10% of the partiality scatter; the full win needs per-image
post-refinement (future work), for which this is the prediction-side groundwork.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add -r multi geometry refinement (per-image best of three) ... 4a5028d98d

Per-image geometry refinement is a tradeoff that flips per dataset: beam+cell refinement
extends OCP merge quality but DIVERGES on sparse-spot stills (e.g. KR2's ~20 spots around a
232 A axis), pushing good lattices out of tolerance so they fail the acceptance floor.
`-r multi` runs all three (none / orientation / beam_and_lattice) on a copy per image and
keeps whichever indexes the most spots (scorer = fractional-Miller-within-tolerance count,
mirroring AnalyzeIndexing); ties prefer less refinement, to avoid overfitting the sparse list.

Validated: KR2 index 7.08% (-r beam_and_lattice default) -> 10.05% (matches the -r none best),
while OCP R-free stays ~equal to beam_and_lattice. New GeomRefinementAlgorithmEnum::Multi handled
in the CLI/HDF5/command-line echoes; the API convert maps it to beam_and_lattice (offline only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: scale self-referenced stills in a single pass (fix weak-data collapse) ... 9527fa6e2c

The stills self-scaling loop rebuilt the merge reference FROM the just-scaled reflections
every iteration, so on weak data each pass re-fit its own noise and the merged CC1/2
collapsed (PfluDING combine: 0.3% at the default 3 iters). It is provably pointless too:
ScaleOnTheFly::Scale re-solves the per-image G from raw I and never reads the prior
correction, so against a fixed reference passes 2..N are bit-identical - the single G is the
exact one-pass solution and iteration only ever does the harmful reference rebuild (N=1 is
best on every dataset, including strong lyso 82% vs 76% at N=3; leave-one-out and external
anchors were tested and do not beat it). Do one self-reference pass; rotation (RotationScaleMerge)
and the external-reference branch are untouched.

Validated: PfluDING-combined CC1/2 0.3% -> 42.0% by default (pinned 2.5 A); strong data unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: the external reference MTZ is never a scale anchor (stills + rotation) ... 95af137146

An external reference MTZ is a poor per-image scale reference: it is a different crystal/dataset,
so scaling against it injects cross-dataset systematics rather than removing per-image scale error
(measured: reference-scaled R-free 0.410 vs self-scaled 0.388 on OCP; 0.378 vs 0.350 on LOV). Its
real value is fixing the cell/space group, breaking the merohedral indexing ambiguity, reporting
CCref and providing the R-free test set - none of which need it to be a scale anchor.

Make that consistent across both workflows:
- Stills: the per-image pass (ScaleImage) now uses the reference ONLY to break the indexing ambiguity
  (once, for good), not to fit G. Scaling self-references at the post-measurement merge for every run,
  whether or not a reference is given (Rugnux + rugnux_cli --scale).
- Rotation: ScaleImage no longer scales against the reference either (it was dead code - RotationScaleMerge
  recomputes the per-frame scale, so the reference never actually moved the result). Rotation resolves
  the ambiguity globally (ChooseReindex / ReferenceIntensityCC) and self-scales in RotationScaleMerge.
- A reference-as-scale mode, if ever wanted, would be a dedicated later step, not this per-image pass.

Validated: OCP CCref 56.5%->64.6%, R-free 0.400->0.393 (honest 2.36 A cutoff instead of a reference-
propped 1.5 A); LOV CCref 88.3%->90.1%, R-free 0.336->0.331 with more reflections; rotation lyso_ref
with the reference gives ISa 17.8 == de-novo 15.6 (same 15975 reflections, CC1/2 99.8%) - the reference
demonstrably does not touch rotation scaling, only cell/SG + ambiguity + CCref.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: rotation indexing always uses FFT/FFTW, never FFBIDX ... 3f6462b30f

The rotation two-pass accumulates a dense multi-frame reciprocal-space cloud (all first-pass
frames de-rotated into one common frame) and needs a GLOBAL lattice finder - a 3D FFT whose
peaks are the basis vectors. ffbidx is a single-still, known-cell ORIENTATION finder: it assumes
the spots lie on one Ewald shell (one crystal, one orientation) and maximizes a near-integer-hkl
count for a fixed cell. On the volume-filling rotation cloud that objective has no dominant global
maximum, so ffbidx's coarse orientation scan locks onto an orientation valid for only a local wedge
(~18%) -> garbage merge.

GetIndexingAlgorithm resolved Auto->FFBIDX whenever a cell was known, even for rotation - and a
reference MTZ (-z) or -C silently supplies the cell. So a routine rotation run with a reference for
CCref/R-free flipped the indexer to the one solver that cannot handle its input (lyso_ref: ISa 17.8
with FFT -> 0.5 with FFBIDX, no warning). Force rotation to FFT (GPU) / FFTW (CPU) regardless of the
requested algorithm or a known cell.

Validated: lyso_ref -R with -z reference MTZ now indexes via FFT -> ISa 17.8 (was 0.5), 15975 refl.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: move indexing-ambiguity resolution out of ScaleOnTheFly ... 49963ab855

Indexing-ambiguity detection/resolution is no longer part of scaling: it
is a dedicated per-image step for stills (or a post-full-merge step for
rotation). Pull the stills per-image resolver out of ScaleOnTheFly into a
new ReindexAmbiguityResolver in ReindexAmbiguity.{h,cpp}, alongside the
existing rotation free functions (ChooseReindex / ReferenceIntensityCC).

Both workflows now share the operator generation and the best-op
selection (new file-local PickBestReindex helper); ChooseReindex is
rewired onto it. ScaleOnTheFly is now purely a scaling engine (no
ambiguity_ops, no resolve_ambiguity flag, no ResolveIndexingAmbiguity).

Pure refactor: behaviour is unchanged. IndexAndRefine now holds a
ReindexAmbiguityResolver and calls Resolve() at the same call site.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: --refine-geometry stills two-pass (global bundle adjustment + re-index) ... 2f5ed411e2

Weak/jet serial stills are often geometry-limited: a few-px beam error or a mm-scale detector
distance error (unreachable per-image) fails many frames, but is well-determined jointly from the
strong frames. Mirror the rotation two-pass: an index-only first pass over a spread sample collects
each indexed frame's spots + assigned HKL + orientation; the strongest ~N (default 200) feed one
Ceres bundle adjustment; the refined geometry is applied and the main pass re-indexes + integrates
+ merges every frame from scratch.

GeometryRefiner (reusing the extracted XtalResidual - the RecipToDetector geometry residual pulled
out of XtalOptimizer, behaviour-preserving): one problem with SHARED beam(2)/distance(1)/cell-length(3)
blocks + a PER-FRAME orientation(3) block, robust Cauchy loss, a cell-length regularizer anchoring the
known cell to break the low-resolution distance<->cell-scale degeneracy, DENSE_SCHUR eliminating the
per-frame orientations, and a 3-round HKL-reassignment / tolerance-tightening loop. Tilt is not refined
(gauge-coupled, zero gain). Opt-in via --refine-geometry[=N]; stills only (rotation untouched).

Validated: KR2 7.58% -> 21.85% (matches CrystFEL's 21.5%; a real ~1.4mm distance error + ~3px beam),
OCP 2.92% -> 4.19% (~3px beam). OFF runs are bit-identical to baseline (XtalResidual extraction
non-regressing; rotation lyso_ref de-novo ISa 17.3 unchanged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add --partiality-uncertainty stills merge term (partiality-model error) ... 4c5da20eda

A stills reflection recorded at partiality p carries a systematic intensity error ~(dp/p) that is
proportional to <I> and grows as p falls; plain counting sigma misses it, so strong low-p partials
are over-trusted in the merge. Add sigma^2 += (c*<I>*(1-p))^2 in MergeOnTheFly::CorrectedSigma - the
stills-partiality analog of the existing rotation --capture-uncertainty term ((1-captured_fraction)*I
in RotationScaleMerge). Inert when partiality==1 (no --still-partiality), and gated on a real
systematic (error_model_b > 1, i.e. ISa < 1) so it fires on strong/medium stills but auto-skips weak
counting-limited data where it would only over-concentrate the merge and hurt. Opt-in via
--partiality-uncertainty <c> (default 0; ~2.5 recommended with --still-partiality).

Prototyped (Python replica of the merge): lyso8 CC1/2 +1.7, CCref +4.8, R-free -0.023; LOV CCref +2;
harmful on weak OCP (hence the b>1 gate). Full in-binary validation across all serial targets pending.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: de-double spurious rotation supercell in candidate selection ... 4c050b576c

The two-pass rotation indexer could keep a doubled-axis supercell over its own
primitive sub-cell: when the supercell won candidate (ci) order and the primitive
cleared the indexed-fraction hysteresis by less than the 0.05 "prefer-earlier"
margin, the supercell survived. On pding4_003 full data this gave P222 on a
65.6x131.3x173 cell instead of P422 on 65.6x65.6x173 - the orthorhombic metric
forecloses the 4-fold before the symmetry search ever runs. The FFT peak-finding
is correct (the primitive cell is among the candidates, just out-ranked).

Add a sub-cell override to the selection loop: adopt a later candidate that is a
genuinely smaller cell (> ROT_SUBCELL_VOLUME_RATIO=1.5x smaller volume; a doubling
is 2x) indexing at least as many spots (within ROT_SUBCELL_FRAC_SLACK=0.02). That
is the signature of a spurious doubling - the primitive always indexes >= its
integer multiple, whereas a real superstructure's larger cell indexes MORE (kept
by the existing clearly-more branch) and twins share the cell volume (untouched).

Full 24-crystal rotation battery (pre-fix vs post-fix): pding4_003 P222->P422
(cell halves to 65.6x65.6x173, R_meas 6.6% CC1/2 99.9% ISa 11.9), pding4_001
holds P422, all 22 other crystals bit-identical in space group.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: reject 2nd-lattice overlaps in the space-group search (E^2 cap) ... 768a926e0c

On a two-lattice crystal a second lattice deposits intensity on one reciprocal
position but not its symmetry mate, so a contaminated reflection is a one-sided
resolution-normalised-E outlier that poisons the operator I(h)/I(Rh) correlation
in SearchSpaceGroup Stage A. On EP_cs_02-424 under -A this dropped the monoclinic
2-fold CC to 0.326 (below the 0.5 gate) -> the crystal was under-called P1 instead
of P2_1. The existing --reject-outliers cannot see it: it rejects WITHIN a P1 orbit,
but the poison is a BETWEEN-orbit effect (inflates orbit h, not its mate Rh).

Add SearchSpaceGroupOptions::max_e_squared_for_cc (default 9.0, i.e. E>3): after the
resolution-normalised Esq[] is computed, drop the extreme-E tail from the correlation
pairs only (the absence stage keeps the full range - that is where the screw signal
lives). Clean Wilson-distributed data almost never reaches E^2=9 (P ~ 0.01-0.3%), so
it is self-targeting - it trims the overlap tail without touching genuine reflections.

EP_cs_02-424 -A: 2-fold CC 0.326 -> 0.64 -> P2_1 (= XDS). Full 24-crystal rotation
battery (pre/post): only EP_cs_02-424 changed (P1 -> P2_1, SG match 20 -> 21/24); all
23 others bit-identical, including the marginal cubic Ins_I_3 (still I23, carried by
the systematic-b rescue). A Friedel-merge-the-search alternative was tried and rejected
(net-negative: did not fix EP424 and regressed Ins_I_3 -> I222).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: default --refine-geometry on for stills-with-cell; accept F/FP reference columns ... 135765e5b5

Two stills usability fixes.

1. --refine-geometry now defaults ON for stills whenever a reference cell is
   available (-C or a reference MTZ) - exactly the case where the geometry
   bundle-adjust can act (it anchors on a known cell) and where it lifts weak/
   sparse-stills indexing (OCP +42%, KR2 +108% indexed in the target study). It
   stays a no-op for rotation (own two-pass) and de-novo stills (no cell yet), so
   auto-enabling it only where it does something avoids spurious "skipping"
   warnings. --refine-geometry=off opts out; explicit --refine-geometry[=N] forces it.

2. The reference-MTZ auto column selection now falls back to a plain amplitude
   column (FP / FOBS / F / FC, squared to an intensity) after F-model and the
   intensity (J) columns. A deposition that carries only structure-factor
   amplitudes (e.g. KR2's 8cl8, whose only usable column is FP) now seeds CCref
   without needing an explicit --reference-column FP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

writer: remove CBF and TIFF writers; only HDF5 is written ... 4bdf9309dd

Delete CBFWriter and the TIFF write path from jfjoch_writer so the writer
only produces the NXmx HDF5 formats. Drop CBF/TIFF from the internal
FileWriterFormat enum and reject them in FileWriterSettings.

The CBF/TIFF values are kept (marked deprecated) in the file_writer_format
OpenAPI enum for back compatibility: incoming requests using them are now
rejected in OpenAPIConvert with a clear "no longer supported" error, and a
stale CBF/TIFF value on the CBOR wire decodes to unset rather than a removed
enumerator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

viewer: expose stills processing settings; rename geom refinement multi -> flex ... 5d1ddccee8

Wire the recently-added stills knobs into jfjoch_viewer - they were reachable
only from the rugnux CLI, and the viewer had silently diverged from the CLI once
--refine-geometry became default-on for stills-with-cell.

- Stills geometry-refinement two-pass (--refine-geometry): a checkbox + frame
  count in the reprocessing-job dialog, offered (and defaulted on, matching the
  CLI) only for a stills-with-cell run; wired into ProcessConfig and the
  "Copy command" generator (joined =N, or =off to reproduce opting out of the
  CLI default-on).
- Stills partiality (--still-partiality) checkbox in Bragg integration and the
  partiality-uncertainty merge term (--partiality-uncertainty) in Scaling, both
  mirrored in the command-line generator.

Rename the "best per-image refinement" mode from "multi" to "flex" everywhere:
GeomRefinementAlgorithmEnum::Flex, CLI -r flex (with "multi" kept as a hidden
back-compat alias), OpenAPI enum "Flex" + regenerated C++/TS clients, viewer
combo, frontend dropdown, and the stored HDF5 string. "multi" collides with
CrystFEL's multi-lattice sense; "flex" = let the pipeline decide. The API now
carries a dedicated Flex value instead of masquerading as BeamCenter.

Also fix a pre-existing bug in Convert(IndexingSettings): OrientationOnly was
missing from the outbound switch and serialized as an invalid API value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jfjoch_viewer: retry deferred initial fit on resize (grid-scan zoom) ... ebe7e7ec5e

fitToViewShorterSideOnce() skips the initial fit-to-view while the viewport has
no real size yet (before the widget is laid out/shown) and relies on a retry
that was never wired: resizeEvent only reset the scene rect, and there is no
showEvent. When the first Redraw() lands before layout settles the view stays
at 1:1, so a small grid-scan plot renders tiny ("zoomed out, not taking the
full picture"); the layout-timing race is why it happened only sometimes.

resizeEvent now retries fitToViewShorterSideOnce() while the initial fit is
still pending (guarded by !initial_fit_done_, so a later user resize never
overrides a manual zoom). It lives in the shared base class, so all image views
get the same robustness.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: accept a space-group symbol for -S, not just a number ... bd094e585f

-S now takes either a number ("92") or a Hermann-Mauguin symbol
("P43212"), resolved via gemmi find_spacegroup_by_name. Previously a
non-numeric argument was atoi'd to 0 and tripped the early space-group
validation, aborting the run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: fix de-novo rotation indexing adopting a spurious axis-multiple supercell ... 0f3ccda779

Two-pass rotation indexing scored each first-pass scheme by how many
validation frames it indexes, but a spurious axis multiple (2x/3x...)
indexes every frame its true sub-cell does, so the count saturates - both
schemes reach the same frame total - and the tie fell to whichever scheme
ran first. When that first scheme's full-rotation FFT resolves a true
axis only as a 2x/3x harmonic of its length (the fundamental can sit far
below the harmonic in the full-360 cloud), it commits a multiplied cell
and the dataset collapses to P1, even though the other scheme already
found the true cell.

Add an integer-supercell cross-scheme tie-break: when two schemes tie on
frames but their cell volumes differ by a near-integer factor >=2, the
larger is the spurious supercell and the smaller true cell is adopted,
independent of scheme order. The near-integer test distinguishes a real
axis multiplication from a centering coincidence (a rhombohedral H cell
vs its C2 sub-cell is 1.5x and is left alone).

Validated de-novo across the rotation regression battery: the affected
dataset now indexes its correct space group, the override triggers only
where an axis was multiplied, and every other dataset is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Remove sample identities from the repository; document the rule ... abbee2d4dc

Datasets may be confidential; sample names and measured unit cells committed to
the repo can leak outside the group working on them. Scrub existing occurrences
and add a "No sample identities in the repository" section to CLAUDE.md
(forbidden: sample/dataset names, internal codes, measured cells tied to a
sample; fine: space group / lattice / twinning descriptors).

- Comments: replace internal dataset codes and protein names with the
  crystallographic situation they illustrate (centred vs pseudo-symmetric,
  holohedral, cubic, F-cubic/hexagonal, ...).
- Docs: same, in the analysis/writer/stream references and example configs.
- Tests: rename sample-named identifiers, TEST_CASE names, file prefixes and
  asserted labels to neutral crystallographic names (e.g. tetragonal_uc);
  behaviour unchanged. Reduce the CrystFEL reference PDB to a bare CRYST1 cell
  file (cell.pdb) and rename the reference data file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

indexing: clear error when the requested indexer is unavailable on this host ... d8e909a393

IndexerThreadPool built its per-worker indexers from the raw requested algorithm
but dispatched the RESOLVED one. Rotation indexing always resolves to the GPU FFT
indexer when a GPU is present, so requesting FFTW (the CPU indexer) on a GPU node
created only the CPU indexer and then failed at dispatch with an opaque "no
indexer available for the resolved indexing algorithm" deep inside a worker.

Validate servability at the pool entry (in the caller's context) and throw an
actionable message instead: FFTW is not available on a GPU node (use FFT/Auto, or
a CPU-only node); FFT/FFBIDX require a GPU. Stills FFTW on a GPU node, which the
algorithm resolution still honours, keeps working.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

changelog: document the supercell fix, -S symbol, and FFTW-on-GPU error ... 7b2cfb20f8

Also scrub two sample names from existing rc.159 entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

stills: widen per-frame cell-refine angle bound to [30,150] deg ... 643bda3dd3

The stills per-frame XtalOptimizer in IndexAndRefine used the struct-default
monoclinic angle bound [60,120] deg, which clamps a beta outside that window
(e.g. NmHR / 7O8F, C2 beta=131.78) to the boundary and corrupts the per-frame
cell. Both indexers already use [30,150] (FFBIDXIndexer, FFT settings); this
aligns the per-frame refine with them. Rotation path is unaffected
(refine_unit_cell is false there).

Verified on NmHR serial stills: FFBIDX with the correct C2 cell went from
0.01% to 9.10% indexed, recovering the true mC cell (beta~131.7) instead of a
clamped beta=120; 60k-image merge reaches 2.61 A at 99.8% completeness.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rotation: widen candidate-refine angle bound to [30,150] deg ... 0f07b7a499

RotationIndexer's per-candidate XtalOptimizer relied on the struct-default
monoclinic angle bound [60,120], the same latent clamp fixed for the stills
path: a rotation crystal with beta>120 (or <60) would have its cell clamped to
the boundary. Set [30,150] explicitly, matching the indexers.

No change on the /data/rotation_test battery (22/25 SG match, byte-identical) --
no battery crystal has beta outside [60,120]; this is a latent-correctness fix
for future high-beta monoclinic rotation data.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

indexing: check candidate angles against the reference cell in Refine() ... 3e3545770f

When a reference cell is supplied, PostIndexingRefinement::Refine() vetted
candidates on sorted edge lengths only; a cell with the right edges but a wrong
angle (a pseudo-symmetric near-metric, e.g. a monoclinic beta refined to the
wrong value) passed. Add a sorted-angle check against the reference, folding
each angle to its acute complement min(x,180-x) so the obtuse/acute setting
choice is irrelevant (tolerance 10 deg).

Reference-cell path only (no effect de novo, where reference_unit_cell is null):
the /data/rotation_test battery is byte-identical (22/25), and NmHR with the
correct C2 cell is unchanged (9.1%, beta~131.7). A latent guard against
wrong-angle cells slipping through on -C / reference-MTZ runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: warn when the cell/space group has an indexing ambiguity ... b450140816

At the twinning-analysis print, detect alternative-indexing (twin-law) operators
for the final cell + space group via ReindexAmbiguityOperators(). When any exist
and no reference was supplied, emit a Warning (also written into the merge stats)
that serial-stills crystals are indexed in a random hand and the ambiguity can
only be broken against a reference (-z / --model). For an obvious merohedral case
(P3/P4/P6...) users expect this, but a PSEUDO-merohedral metric is easy to miss --
e.g. NmHR / 7O8F, where C2 beta=131.8 is pseudo-F-orthorhombic and gemmi reports
the reindex operator h-l,-k,-l. Advisory only; no change to processing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

VERSION: 1.0.0-rc.160 68272a96e1

Remove residual sample identities missed by the earlier scrub ... d393ec814c

The rc.160 sample-identity cleanup left protein names, internal dataset
codes, and a measured cell angle in several source comments and one test.
Most sensitive: a measured monoclinic beta tied to an internal code in
Rugnux.cpp and IndexAndRefine.cpp. Rewrite each to describe the
crystallographic situation only (space group / metric relation), keeping
the technical reasoning intact. Comment- and string-only; no behaviour
change. (rugnux_cli.cpp's remaining name is scrubbed in the following
commit, alongside its other change to that file.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: stop advertising the removed CBF/TIFF writers ... 8fea7f195a

The CBF/TIFF writer removal left two docs describing them as available
output: JFJOCH_WRITER.md's "Other formats" section and an aside in
HDF5.md. A user following them would issue a now-rejected request. State
that only NXmx HDF5 is written and that the CBF/TIFF enum values are
retained for wire back-compatibility but rejected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rugnux: validate offline spot-finding settings and bound numeric options ... 0e4f91b219

The offline rugnux path never ran CheckDataProcessingSettings (only the
online broker/receiver paths did), so the new --min-pix-per-spot and
--spot-low-resolution knobs were unbounded: --min-pix-per-spot 0 silently
disabled the per-spot filter, and a low-resolution limit finer than the
high-resolution limit made spot finding reject every pixel with no
diagnostic.

Add a parse_number_arg<T> helper (integral or floating, with optional
inclusive bounds) that rejects non-numeric input, trailing garbage, and
out-of-range values, replacing the raw atoi/atof in the spot-finding
options; parse_double_arg/parse_float_arg become thin wrappers over it.
After assembling the settings, validate them with the same
CheckDataProcessingSettings the online receivers use, which also enforces
the cross-field low>=high-resolution constraint.

Also scrubs a residual sample-name comment (the r=6 box-default note) in
this file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

changelog: add the 1.0.0-rc.160 section ... ce15900471

VERSION was bumped to rc.160 but the new items sat under the rc.159
heading, which released (origin/main) does not contain. Add a proper
rc.160 section, move the three misfiled items (supercell fix, -S symbol,
FFTW-on-GPU error) into it, and document the rest of the branch (stills
geometry refinement, -r flex, stills partiality, r=6 box default,
single-pass self-referenced scaling, reference-MTZ-never-a-scale-anchor,
E^2 second-lattice cap, --scale on _process.h5 fix, new spot options,
CBF/TIFF writer removal, negative total_flux, viewer, packaging).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rugnux: write standard CCP4 anomalous MTZ (IMEAN + I(+)/I(-)) ... ce12d69751

In anomalous mode the merge keeps the two Friedel mates as separate rows,
which WriteMtzReflections emitted verbatim - two IMEAN rows per Bijvoet pair
that downstream tools had to re-collapse. Pair the mates into one row per
reflection with the standard CCP4 anomalous layout: IMEAN + I(+)/I(-) and the
matching F/F(+)/F(-) amplitudes. A single HKLKeyGenerator(merge_friedel=false)
yields both the shared ASU group key and which mate a row is (.plus); IMEAN/F
are the inverse-variance Friedel mean, centrics/unpaired keep one hand with the
other missing. aimless/mtz2sca/ANODE now read the file directly. The
non-anomalous MTZ path is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: always write both MTZ and CIF; drop --scaling-output ... eae7ea90dc

The merged reflections were written in one of mtz/cif/txt selected by
--scaling-output. Write both an MTZ and an mmCIF unconditionally instead - each
has its uses downstream (MTZ for the CCP4/phenix tools, mmCIF for deposition) -
and remove the format selector, the plain-text .hkl writer, and the now-unused
IntensityFormat enum / ScalingSettings::FileFormat plumbing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: add detector-plane modulation (flat-field) correction surface ... 818ec8fcbf

Rotation scaling gained a third cross-validated correction surface alongside
decay and absorption: a smooth multiplicative factor over the predicted detector
position (px, py) where each full lands. It absorbs detector-response and
geometric flat-field systematics that vary across the detector plane and inflate
R-meas; symmetry-equivalents of one reflection land at different detector
positions as the crystal rotates, over-determining the surface. Because it lives
in the detector frame (not the rotation), the same correction concept applies to
stills.

The surface fit/cross-validate/apply machinery shared with the absorption surface
is factored into ApplyCellSurface; RefineAbsorption and the new RefineModulation
just build their per-full cell assignment (goniometer-frame vs detector-frame)
and call it. The cross-validation now scores a sigma-INDEPENDENT, R-meas-like
fractional agreement of the held-out equivalents rather than a studentized chi^2,
so a surface can no longer pass CV by reshaping sigma without tightening the
intensities - this removes an over-fit regression on mis-indexed data and hardens
the absorption surface too.

On the /data/rotation_test rotation battery (A/B, modulation off vs on): 17 of 22
processed crystals improve R-meas, 0 regress, e.g. lysoC 23.2->16.3%, lyso_2
47.4->28.5%, EcwtAL500 53.2->27.6%, EP_cs_01-17 CC1/2 60->93%; CC1/2 held or
improved everywhere and the anomalous S signal preserved/improved (lyso_ref
SD_MET 4.37->4.46, lysoC 3.07->3.25). On by default with the other surfaces.

Also: --dump-observations now writes the per-full centroid frame and predicted
px/py columns, so the dumped combined fulls are a complete unmerged export.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

common: remove dead Reflection.completeness field ... 118f0833dc

Reflection.completeness (spot-footprint fraction) was only ever set to its
1.0f default and never read anywhere - not in scaling/merge, CBOR, HDF5, or
the viewer. The box-sum integrator already rejects any partial-footprint
reflection outright (n_inner_valid == n_inner), so the intended "down-weight
clipped spots" role is already covered more strictly. Removed to reduce
confusion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bragg integration: trimmed-mean background, on by default for rotation ... c3e877d5ec

The local Bragg background is the mean over the r2..r3 ring. That mean reads
high because the contaminants that survive the signal-disk mask - neighbour-
spot wings, tails, zingers - are one-sided (positive), so it over-subtracts.
Since a weak intensity is a small difference of large numbers (I = S - nS*b),
a per-pixel background bias is fractionally largest at the resolution edge,
exactly where it hurts most.

Replace the ring mean with a symmetric trimmed mean (sort the ring, drop the
lowest and highest fraction f, average the rest), controlled by a new
BraggIntegrationSettings field and the rugnux `--background-trim <f>` option
(default f=0.10; 0 restores the plain mean). Default on for monochromatic
(rotation) data; broadband (stills) keep their tuned high-side sigma-clip, so
the base engine forces the trim to 0 there. Implemented in both the CPU engine
and the GPU kernel (shared-memory bitonic sort per block, flat-mean fallback
above BKG_TRIM_MAX ring pixels); the two agree.

25-crystal rotation battery (fixed SG/cell): <I/sigma> improved on every
crystal (median +50%), ISa on 20/22, resolution-edge R_meas fell several-fold
(e.g. lyso_ref 1.0 A 108%->43%). Last-shell CC1/2 is rescued where the plain
mean had collapsed to noise (Thau_9 at ~2.0 A 3.8%->64%, ~0.5 A of resolution
regained; cytC_10 0.2%->10%) at a small cost (1-3%) in already-clean shells -
it flattens the CC1/2 fall-off rather than shifting it. Stills unchanged.
Documented in CPU_DATA_ANALYSIS.md section 9.2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

stills: optional detector-plane modulation (flat-field) correction ... 74d7589abe

The stills scaling/merging path (ScaleOnTheFly + MergeOnTheFly) had no
correction surfaces, whereas rotation fits decay/absorption/modulation inside
RotationScaleMerge. Serial stills hammer the same detector regions shot after
shot, so a detector-plane (flat-field) systematic is at least as well
determined there as in rotation - it just had no home.

Add MergeOnTheFly::RefineModulation: a 16x16 multiplicative surface over the
predicted detector position, fit against the merged reference and folded into
each reflection's image_scale_corr before the error model and the merge. It
mirrors RotationScaleMerge::ApplyCellSurface - alternating per-cell fit with a
den-weighted geometric-mean gauge (never drifts the overall scale) and
Tikhonov pull to 1 - and is cross-validated the same way: fit the surface on
even images, score the held-out odd equivalents by a sigma-independent
R-meas-like agreement (a fractional metric a sigma-reshaping surface cannot
game), and apply the full-data surface only if the held-out gain clears a
margin. A no-op when the systematic is absent or the data too sparse (< 8 obs
per cell), so it is safe to leave off by default and opt in.

Wired through ScalingSettings::StillsModulation and rugnux --stills-modulation
(default off), in both the full-analysis and --scale paths. On a 17 MP
JUNGFRAU serial set the surface cross-validates with a large held-out gain,
confirming a real detector-plane systematic; whether it improves the merged
data quality is under evaluation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

viewer: expose detector-plane modulation and background trim in the settings dock ... 3b168a1a46

Two new controls in the interactive settings dock:

- Scaling section: a "Detector-plane modulation (stills)" checkbox that toggles
  the stills flat-field modulation surface (ScalingSettings::StillsModulation),
  alongside the existing decay/absorption "Correction surfaces" checkbox
  (rotation modulation is fitted there). Off by default.

- Bragg integration section: a "Background trim" checkbox + fraction that sets
  the symmetric trimmed-mean background fraction
  (BraggIntegrationSettings::BackgroundTrimFraction; 0 = plain ring mean).
  Defaults on at 0.10, matching the integrator default, so the viewer requests
  trimming for every dataset (the integration engine still applies its own
  per-mode gating).

Viewer-only change; compiles against the current settings API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: dataset Wilson B-factor estimate + robust per-image Wilson B ... 85cf826d2e

Add a dataset-wide isotropic Wilson B-factor estimate, the analogue of XDS's
"WILSON LINE ... B=" which we did not export. CalcGlobalWilsonB fits ln<I> vs
1/d^2 over the merged reflections (B = -2*slope), skipping the low-resolution
non-linear region (d > 4 A) and shells past the signal limit (<I/sigma> < 1)
so the estimate is insensitive to how far the merged data were carried. It is
diagnostic only - not fed back into scaling - and is written to the mmCIF
(_reflns.B_iso_Wilson_estimate), the printed merge statistics, and the log.

Also harden the per-image Wilson B (CalcWilsonBFactor): accept the fit only
when it is well-correlated and physically plausible (0 < B < 200 A^2), else
leave b_factor unset. A bad frame (an indexing glitch, too few reflections)
otherwise produced a wildly steep Wilson line and a B of several hundred A^2
that polluted the per-image plot; NaN is preferable to garbage.

Diagnostic-only: the merged intensities and every merge statistic are
byte-identical (verified baseline vs modified on the rotation battery).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux_vs_xds.py: drop the removed --scaling-output flag ... 8bd0977833

rugnux always writes both MTZ and CIF now (the --scaling-output option was
removed), so passing it made every crystal in the battery fail against the
current binary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: score the decay cross-validation on the sigma-independent metric ... 3bfce24faa

RefineDecay's cross-validation still scored the held-out equivalents on a
studentized chi^2, while the absorption/modulation surfaces were switched to a
sigma-independent Rmeas-like agreement (sum|Is-Iref|/sum|Iref|) precisely
because a correction can "pass" a studentized gate by reshaping sigma without
tightening the intensities. The decay slope scales sigma up on late /
high-angle frames, so it is exposed to the same loophole. Score it the same
way.

On the rotation battery this rejects a spurious slope on a mis-indexed
monoclinic (P2_1) case: the studentized gate had accepted a physically absurd
-74 A^2 "decay" that cross-validated yet collapsed the error model (ISa
7.1 -> 1.1) and CC1/2 (to ~2%); the fractional gate scores it a -50% held-out
loss and skips it. A small genuine ~3 A^2 decay on another crystal now
cross-validates and engages, with CC1/2 unchanged. Every other crystal is
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: remove sample identities from SearchSpaceGroup comments ... e9a913f84b

The point-group self-consistency comments named specific datasets when
illustrating the chi^2-ratio / systematic-b calibration. Describe the
crystallographic situation instead (a merohedral R3->R32 twin, a genuine
cubic step, a tetragonal P41212 case), per the repository's no-sample-names
rule. Comment-only; no behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: veto a b-ballooning merohedral-twin over-promotion (space-group search) ... 7a9d25850c

The systematic-b test (a genuine symmetry step barely moves the merge error
model's b; a merohedral twin, forcing non-equivalent reflections together,
balloons it) was only ever a RESCUE - it could promote a chi^2-borderline
genuine step but never demote a chi^2-passing one. So a partial twin whose
within-orbit scatter looks self-consistent (merged chi^2 below the ratio
bound) slipped through on chi^2 alone and was over-promoted to the holohedral
group, ballooning b and collapsing the merged ISa.

Make the same test a VETO: a chi^2-passing high-symmetry promotion whose b,
relative to the largest confirmed subgroup, exceeds max_systematic_b_veto is
kept in its true lower symmetry. Calibrated on the rotation-test battery,
where the largest genuine step (a P422 tetragonal) sits at b-ratio ~1.8 and a
merohedral R3->R32 twin at ~2.6 - an empty gap - so 2.0 catches the twin while
never demoting a genuine high-symmetry merge.

Battery space-group match 20->21/25 (an R3 case previously merged as R32 is
now correct), every other crystal unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bragg integration: clarify that the trimmed background applies to stills too ... 7d1fb8db77

The background trimmed mean is gated on the beam (monochromatic vs broadband),
not on the acquisition mode: a monochromatic still (zero bandwidth) already
gets the trim by default, only broadband (non-zero bandwidth: pink-beam / DMM)
data keep the high-side sigma-clip. The comments, the settings doc and the
--background-trim help wrongly implied the trim was rotation-only / that stills
keep the sigma-clip. Wording only; no behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: rc.160 changelog (trim + scrub) and CPU_DATA_ANALYSIS updates ... d1f6b382a6

CHANGELOG: add the Wilson B, decay-CV and space-group b-veto entries; shorten
the verbose modulation / stills-modulation / trimmed-background entries and
remove the dataset identities from them.

CPU_DATA_ANALYSIS: the correction surfaces are now three (add modulation),
on by default, cross-validated on the sigma-independent metric; the background
trimmed mean applies to monochromatic stills as well as rotation (broadband
keeps the sigma-clip); document the dataset-wide Wilson B, the still-partiality
model, and the twin b-veto in the space-group search.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: fix de-novo space-group centering and weak-data high symmetry ... 67a18357b5

De-novo space-group search:
- Determine lattice centering from the strength of the systematically-absent
  class (its mean I/sigma vs the present class) instead of a per-reflection
  violation count. The count gate was brittle on noisy / obverse-reverse-
  twinned data, where enough genuinely-absent reflections cross I/sigma>3 to
  trip the 10% bound although the class is several-fold weaker, and wrongly
  kept a genuine R-centered lattice primitive. False centerings, whose absent
  class is as strong as the present one, stay rejected. Screws keep the count
  test (too few axial reflections to average).
- Recover a genuine high-symmetry point group when the merged error model is
  badly miscalibrated (weak, low-resolution data whose sigmas are far too
  small). The fixed-sigma chi^2 ratio then grows with point-group order for
  real symmetry too and wrongly rejects it (a true cubic group reached a
  chi^2 ratio ~14); the self-normalising systematic-b test stays valid, so
  once chi2_ref shows the ratio is untrustworthy the promotion is confirmed
  on the b-test alone. The balloon veto still rejects a twin. This is the gap
  left when the earlier log10(chi2_ref) widening was removed for the b-veto.

Also print the adopted space group and unit cell as a clean one-line summary
at the end of the run (de-novo or user-fixed -S), not only in the mmCIF.

Full rotation-test battery: exactly the two intended crystals change (both to
the correct answer, with lower R-meas and higher ISa), every other crystal
unchanged; space-group match 21->22/25.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux_vs_xds.py: find the current rugnux binary (build*/rugnux/rugnux) ... b945444372

The rugnux CLI target builds to build*/rugnux/rugnux; the old build*/tools/
rugnux paths are stale leftovers from a previous layout that cmake no longer
updates, so the harness could silently run weeks-old code. Pick the most
recently built binary among the candidates instead of a fixed order.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: write I(+)/I(-) and F(+)/F(-) by default (rotation) ... b647cf454c

A rotation merge now keeps each acentric reflection's Bijvoet split - the
inverse-variance I(+)/I(-) from the same scaled fulls - even when the merge is
Friedel-averaged (the default, no -A). IMEAN stays the primary intensity and is
bit-identical to before; the split is purely additive and scaled
non-anomalously, so a weak anomalous signal is preserved in the output without
having to reprocess with -A. French-Wilson now also fills F(+)/F(-) from the two
hands (one pass, shared Wilson prior).

Writers: the default (Friedel-merged) MTZ and mmCIF now carry IMEAN plus
I(+)/I(-) and F/F(+)/F(-) whenever a reflection has an anomalous split (the
stills path, which computes none, keeps the previous columns). A missing mate or
a centric is emitted as the CCP4 missing-value flag (NaN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: radiation-damage report for rotation (per-image CC vs dose) ... 0c33139983

Summarise the per-image scale the rotation merge already fits - the CC-to-merge
and mosaicity binned by frame (= dose) across the sweep - and flag a falling
per-image CC (or rising mosaicity) as the radiation-damage / crystal-decay
signature. A data-quality-vs-dose read complementary to the fitted decay
correction; the full per-image table is still written to <prefix>_scaling.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

merge: deltaCChalf uses the deterministic HalfForImage split, not an RNG ... 65bc40e7c7

MergeOnTheFly::DeltaCChalfReject assigned its CC1/2 half-sets from a seeded
std::mt19937 drawn in image (call) order, whereas the actual merge, the
rotation merge, the GPU path and the R-free flags all split with the
deterministic HalfForImage(image_id) splitmix64 hash. So the deltaCChalf was
measured on a different half-partition than the reported CC1/2, and was
order-dependent (a reordered outcomes vector gave different halves).

Use HalfForImage(i) - i is the image's stable index, the same image_id AddImage
merges with - so deltaCChalf now reflects the exact CC1/2 split the statistics
report, order-independently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: configurable decimal precision, 5 places for azint Q fields ... 799e1e15ec

NumberTextField quantized every float input to 3 decimals, so the finest
q_spacing reachable through the UI was 0.001 - capping the azimuthal-integration
q-bin count around 200 regardless of the CPU/FPGA backend. Add an optional
`decimals` prop (default 3, unchanged elsewhere) and set it to 5 on the Q
spacing / Low Q / High Q fields, matching the q_spacing minimum of 1e-5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rugnux: index long finely-spaced axes (de-novo two-pass long-axis rescue) ... afae1dec2f

De-novo two-pass rotation indexing collapsed on a crystal with a long, finely
spaced axis (a ~150 A c-axis at 16 keV): the unconstrained FFT either collapsed
the long axis to a short sub-multiple or let a denser supercell over-fit the
accumulated cloud - a small global-orientation error throws the many high-order
reflections off along the fine axis, so the true cell scores worst on the raw
cloud, and propagating that inaccurate global orientation to each frame fails.

Add a long-axis rescue that only runs when the standard pass indexes few
validation frames (<50%), so well-indexing crystals are untouched and keep the
fast path: re-run the first pass at a COARSE resolution (low-order reflections
only, where the fine axis stays robust) to recover the true metric, take the
recovered cell with the longest axis directly, then RE-INDEX at full resolution
with that cell as a reference (the -C path) - the reference-cell filter drops the
collapsed/supercell candidates and refines an accurate global lattice.

The scheme feed/index/validate loop is factored into a pick_best() lambda shared
by the standard and constrained passes (the standard pass is behaviour-preserved).
A P41212 58/58/150 test crystal goes 15% -> 99.89% indexed. Full 25-crystal
rotation battery: every other crystal's space group and cell BIT-IDENTICAL, and
the rescue fires only on the failing crystal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: don't veto genuine high symmetry on excellent data (b-veto floor) ... 32c7f9f59f

The de-novo space-group search's systematic-b veto - which keeps a merohedral
twin in its true lower symmetry - is a RATIO test: reject a point-group promotion
whose merge error-model b balloons past 2x the parent subgroup's. That ratio is
meaningless when the parent b is near zero, on excellent data where symmetry
equivalents already agree almost perfectly. A genuine tetragonal 422 whose 222
parent sits at b=0.008 and 422 at b=0.049 (both tiny, ISa ~20) reads as a 6x
balloon and is wrongly demoted to 222.

Floor the parent b (min_systematic_b_for_veto = 0.05) before forming the ratio,
so the veto only judges a promotion once its b is a non-negligible fraction of I.
A real merohedral twin drives b to ~0.19 regardless of how clean the parent is,
so it is still caught (the insulin R3 + ~20% twin veto still fires, b=0.19 >
0.15). A P41212 58/58/150 crystal at 16 keV goes P212121 -> P41212; full
26-crystal rotation battery: only that crystal changes (its space group now
matches XDS, SG match 22->23/26), every other space group bit-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: recover I-centering on weak / low-energy data (centering rate gate) ... 1b5eb80ba4

The de-novo centering gate promoted to a centred lattice only when the mean
signed <I/sigma> of the centering-absent class was <= 0.5x that of the present
class. Weak / low-energy data carry a positive intensity floor (background /
profile leakage) that lifts <I/sigma> of the genuinely-extinct class to ~1.5-2.3
instead of ~0; when the present class is itself weak that additive floor inflates
the ratio past the bound and the true centering is missed. An I-centred cubic
crystal recorded at 5 and 13 keV sat at ratio ~0.57 and stayed primitive P2_1 3,
while the same crystal form at 6 keV (stronger data) correctly merged as I23 - an
energy-dependent space group for one physical crystal.

Add a second, floor-independent acceptance path OR'd with the mean-ratio: the
centering-absent class's strong-reflection rate (violations / absent) versus the
present class's own strong rate. A false centering (absent as strong as present)
fails both arms; a true weak centering passes the rate arm. Battery SG sweep vs
XDS: the low-energy datasets now adopt I23, and no crystal is newly over-centred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: report anomalous signal-to-noise SigAno (stats table + mmCIF) ... 7fe12773de

Compute SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> per report shell and overall
from the full-multiplicity I(+)/I(-) Bijvoet split, store it on
MergeStatisticsShell, and surface it in the console merge-statistics table (new
SigAno column) and the mmCIF via the standard wwPDB PDBx items
_reflns.pdbx_absDiff_over_sigma_anomalous (overall) and
_reflns_shell.pdbx_absDiff_over_sigma_anomalous (per shell) - the dictionary's
home for this quantity, so no jfjoch_ local item is needed (unlike ISa).

SigAno approaches ~0.8 for pure noise and rises above 1 with real anomalous
signal. A half-set anomalous CC is deliberately not used: its two half estimates
are complementary partitions of one observation pool (dI0 + dI1 = 2*dI_full), so
differencing the two Bijvoet hands cancels the large common intensity that keeps
CC1/2 non-negative and, once the anomalous SNR per half drops below 1, drives the
correlation towards -1 rather than 0 - misrepresenting a weak-but-real signal.

Emitted only when an anomalous split was made (has_anom guard), so a non-anomalous
merge keeps its stats block and shell-loop columns byte-for-byte. Export-neutral:
the I(+)/I(-)/IMEAN accumulation is unchanged. On low-energy S-SAD test data SigAno
tracks XDS (13 keV null ~0.89, 6 keV ~1.44, 5 keV ~1.5-1.7) and never goes
spuriously negative.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: document SigAno and the low-energy centering fix ... 174b9585c5

CHANGELOG (rc.160): add the centering rate-gate recovery on weak/low-energy data
and the SigAno anomalous signal-to-noise statistic.

CPU_DATA_ANALYSIS: describe SigAno in the merging-statistics section - its
definition, the standard PDBx mmCIF items, and why a half-set anomalous CC is not
used (complementary halves drive it negative on weak shells) - and note the
floor-independent centering-acceptance test in the space-group-determination notes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: report ISa at its asymptotic definition (strong-reflection limit) ... 88359a57ee

ISa is the I->infinity limit of the signal-to-noise (Diederichs, Acta Cryst.
D66 (2010), 733-740), i.e. the reproducibility of the strongest reflections.
The reported value was taken as 1/b of the whole-intensity-range error-model
fit, whose systematic term b is raised by a mild excess of scatter at
intermediate intensity, so it understated that limit.

Read the asymptote directly instead: for each well-measured reflection group the
counting-subtracted fractional scatter of its symmetry mates estimates the
systematic term, and the robust median over strong groups is the asymptote. It
falls back to the whole-range b when too few strong groups exist (weak or
low-multiplicity data). The reported ISa and the merged-sigma systematic floor
use this value, so merged I/sigma approaches ISa.

The (a, b) fit and the per-observation error model are unchanged, so the merge
weights, anomalous statistics and chi^2 are untouched, and CC1/2 and R_meas
(which do not depend on sigma) are bit-identical. Battery (32 crystals): no
space-group or indexing change, R_meas and CC1/2 unchanged, ISa rises to the
true asymptote of the data.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: recover the asymptotic ISa on weak / radiation-damaged data ... 59e260876a

The asymptote was read only from groups with I/sigma > 20 and required at
least 100 of them, so weak or radiation-damaged data (few strong reflections)
fell back to the higher whole-range b and reported an over-conservative ISa.
Relax the I/sigma threshold in a second tier (>= 50 groups above I/sigma 10)
when the tight one is unmet; the per-group counting-subtracted estimate is
stable down to that threshold. Data with many strong groups is unchanged (it
keeps the tight-threshold asymptote); weak / damaged data now reports the
asymptote its own reflections support instead of falling back.

Battery (32 crystals): no space-group, R_meas or CC1/2 change; strong sets
unchanged; e.g. a room-temperature radiation-damage series recovers ISa 5.4 ->
11.2 (its data already exceeds the reference on CC1/2 and R_meas).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: document the asymptotic ISa reporting and the azint Q-field precision ... 3a23c79ea4

CHANGELOG: describe reporting ISa at its asymptotic definition (read from the
strong equivalents, relaxed threshold on weak/damaged data) and the frontend
azimuthal-integration Q fields gaining 5-decimal precision.
CPU_DATA_ANALYSIS: update the error-model section - ISa is the counting-
subtracted strong-reflection asymptote (and drives the systematic floor), not
1/b of the whole-range fit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: compress the rc.160 changelog entries to one-line summaries ... e209069ccd

The rc.160 entries were multi-sentence explanations; rewrite each as a short
statement of what changed (feature / flag / behaviour), roughly under 200
characters, matching a normal changelog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: identify the producing software in the merged mmCIF and MTZ ... 742a1c1b94

mmCIF: set _software.name to 'Rugnux' (was 'Jungfraujoch') and correct
_software.classification to the CIF-dictionary value 'data reduction'.
MTZ: add the producing software to the header via the title and a HISTORY
record ("From Rugnux <version>, data reduction"), the CCP4 provenance channel,
which was previously empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-17 22:38:32 +02:00

rugnux: rank space groups by net absences, not gross (fix C222 over-centering) ... 7ceffde5a4

Stage B of SearchSpaceGroup chose the largest centering that explained the most
absences and was self-consistent, ranking candidates by the gross observed-absent
count. A false super-centering over-claims: on a C222 crystal, F222 predicts every
C absence (all genuinely weak) PLUS a block of C-present reflections it wrongly calls
absent. Its absent class is then bimodal - many true zeros diluting a strong block -
so its mean <I/sigma>abs (and the violation-rate arm) slip under the centering
strength gate, which cannot see the over-claim in isolation. With the gross-count
ranking F222 then beat the true C222, and the strength gate is scale-invariant here,
so real C-centered data trips it identically (the rotation battery just has no
C-centered crystal on this path; the synthetic unit test does).

Rank instead by the absences GENUINELY explained, absent_observed - absent_violations,
then by fewer violations, then lower space-group number. C222 and F222 net to the same
count (F222's genuine absences ARE C222's), so the fewer-violations tie-break keeps the
honest, less-centred C222. Symmetric: on a real F222 crystal F explains strictly more
weak absences and still wins. Also tighten the "indistinguishable alternatives" grouping
to require the same absent AND violation counts, so a super-centering is never reported
as an equal alternative. No tuned threshold is changed; the ranking is a no-op for any
0-violation winner.

Fixes the SearchSpaceGroup C222 synthetic-space-group test. Rotation battery (32
crystals, de-novo) is space-group-identical to before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 17 commits 2026-07-18 23:38:28 +02:00

integration: carry the box-sum observed centroid through the profile path ... 1911875e1c

Pass A (the box-sum) always computes an intensity-weighted centroid (the
observed spot position), but it was only emitted in BoxSum mode. Rotation
integration runs the profile path, so observed_x/y were left NAN there and no
positional (detector<->reciprocal) residual was possible downstream.

Emit the Pass-A centroid in the profile path too (CPU) and copy d_obs_x/d_obs_y
back in all modes (GPU) - Pass A fills them regardless of integrator mode.
observed_x/y are consumed only by the diagnostic HDF5 reflection table and the
viewer read-back; nothing in scaling/merge reads them, so merged output is
unchanged - the _process.h5 observed_x/y columns simply go from NAN to the real
centroid. This makes the observed position available to post-refinement.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

beam_stop: add ShadowFinder detection class (unwired) ... 9bfecaadf5

Add image_analysis/beam_stop/ShadowFinder.{h,cpp} and SHADOW_FINDER.md: a
self-contained beam-stop shadow detector that accumulates images via AddImage()
and returns a mask from GetMask(), mirroring the shape of DarkMaskAnalysis.

It detects the beam-stop shadow (central disk + holder arm) as an azimuthal
anomaly: an iterated radial-median background baseline, a ratio threshold, a
connectivity-to-beam-centre anchor with module-gap bridging, a central low-res
disk guard capped just inside the innermost reflection, and a reflection guard
that never masks a pixel that recorded real signal.

Not yet wired: not added to image_analysis/CMakeLists.txt and no PixelMask /
Rugnux / viewer changes. SHADOW_FINDER.md documents the algorithm and the
deferred offline (Rugnux bit 9 + viewer user-mask) integration plan.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: record the detector distance and beam centre in the merged mmCIF ... f396385e11

Add _diffrn_detector.jfjoch_{distance_mm,beam_center_x_pxl,beam_center_y_pxl} to
the merged mmCIF header, as feedback of the geometry the data was actually
reduced with. When geometry refinement ran (the rotation two-pass or the stills
global refinement, which update experiment_ before the written pass) these are
the refined values; otherwise they are the nominal header geometry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: --rotation-post-refine two-pass geometry post-refinement (opt-in) ... 1b91823eed

A first pass integrates and post-refines the detector geometry from the observed
diffraction, applies it to experiment_, and a second pass re-indexes and
re-integrates with it - only the second pass is written, so the _process.h5 / mtz
(and the ice-ring flags, which move with the geometry) reflect the refined values.
Default off; enable with --rotation-post-refine.

PostRefineRotationGeometry refines in two SEPARATE cross-validated steps (a joint
fit of everything is fragile - the same lesson as integration, where refining the
profile width and the scale jointly fails but separately works):
  * Step A: cell scale + rotation axis from the observed rocking centroids phi_obs
    (a distance-independent excitation residual, so it pins the absolute cell scale
    that the positional residual leaves degenerate with distance).
  * Step B: detector distance + beam centre from the observed spot positions, with
    the cell fixed at step A.
Each step commits only if it lowers a held-out (deterministic split-half) residual,
else that part of the geometry is left at nominal.

Rugnux::Run wraps RunPipeline(write_output, geometry_prepass); the pre-pass runs
the first scale/merge so RotationScaleMerge fits a frame-order-smoothed mosaicity,
which is captured and fed to the second pass's Bragg prediction (via
IndexAndRefine::SetPredictionMosaicityOverride) rather than re-derived per image.
observed_x/y were plumbed through the profile integrator earlier for the positional
residual. rugnux_vs_xds.py gains --extra-args to sweep the option over the battery.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: two-pass reuses pass-1 space group; floor the fed mosaicity ... c9ed9c3776

The second pass re-searching the space group at the slightly changed geometry
could flip a borderline determination (cubic <-> orthorhombic) and mix the two
passes, collapsing the multiplicity. Fix it: the pre-pass now continues through
the space-group search, so the determined group is already fixed on experiment_
and the second pass reuses it instead of re-searching. The lattice is NOT forced
- the second pass re-indexes at the refined geometry so the cell stays
self-consistent with it (forcing the pre-pass lattice would pin the nominal cell
and undo the refinement). A reindex-derived group (the centred-lattice test) is
in a conventional setting the indexer's primitive frame does not share, so it is
cleared for the second pass to re-search (else its indexing fails outright).

Also floor the mosaicity fed to the second pass's prediction at each frame's own
estimate (max), so a too-narrow smoothed value can never make prediction drop
reflections.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: restrain the gauge-weak beam centre toward the header in geometry refinement ... c140b3567d

Single-axis rotation has a gauge degeneracy: rotating the whole experiment
about the spindle leaves every spot position unchanged, so the beam-centre
component parallel to the spindle is a null / under-determined direction.
Refining it freely lets it wander a few pixels and absorb centroid
systematics into a wrong beam that the co-refined orientation keeps
position-consistent.

Add a soft header prior on that one component in the shared XtalOptimizer:
residual = w*(beam[parallel] - header), with w set so the restraint behaves
like a sigma_px-pixel prior competing with the positional residuals. The
gauge direction has ~zero data sensitivity, so the prior pins it near the
LaB6-monitored header, while a real, well-supported drift can still overcome
it (the beam does drift). One restraint covers all three refinement sites at
once: the noisy accumulated-spot primary, the per-frame filter, and
post-refine.

Validated against independent XDS processing: neutral on well-determined
data (the gauge direction has no effect there), and a real low-resolution
accuracy gain where the free refinement had otherwise wandered off the
header - the merged intensities move back into agreement with XDS while the
internal CC1/2, which is blind to this, barely changes. No space-group
determinations change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: two-pass writes both passes (_01 header geometry, _02 refined) ... 1089b75906

The two-pass geometry post-refinement previously wrote only the second
(refined-geometry) pass. Write both instead, so the header-geometry and
post-refined results can be compared directly and the better one kept:

  <prefix>_01_*  first pass, at the header geometry (the plain single-pass
                 baseline)
  <prefix>_02_*  second pass, at the post-refined geometry

Each is a complete standalone result - its own merged reflections
(.mtz/.cif), per-image _process.h5 (with --write-process-h5), ice-ring
flags and mmCIF detector geometry, all consistent with the geometry that
pass actually used.

Mechanics: the pre-pass now stores the post-refined detector geometry
instead of applying it mid-pass (so the first pass finishes and writes at
the header geometry), and no longer returns early - it continues through the
full write. Run() then applies the stored geometry, lets the second pass
re-search a reindex-derived space group, and routes the two passes to the
_01 / _02 prefixes. The second pass's result is unchanged from before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: two-pass second pass re-indexes de novo, reuses the group for the merge only ... 0ebf90a1ad

The two-pass second pass reused pass-1's space group for both the indexer and
the merge. Reusing it in the INDEXER disables the de-novo pseudo-symmetry
safeguards, so build_sr adopts a wrong cell: a metrically pseudo-centred
crystal doubles into a super-cell (every real reflection plus phantom
half-integer ones - so the merge keeps CC1/2 but halves <I/sigma>), and a huge
oblique cell's constrained monoclinic refine is ill-posed, diverges, and the
re-indexing collapses.

Decouple the two: the second pass now indexes DE NOVO (the group is cleared
before it, safeguards on) so it recovers the true cell, and pass-1's group is
reinstated for the MERGE ONLY, just before RotationScaleMerge, so the symmetry
stays fixed exactly as a user-fixed group would - no re-search, no flip. A
reindex-derived group is still left to re-search.

Neutral for genuine-symmetry data (its de-novo cell already matches the
group); it recovers the pseudo-symmetric cases that previously collapsed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: post-refine commits detector/cell geometry only for a small (< 1 %) move ... 290c30b9be

The rotation geometry post-refine committed the detector distance and cell
scale on any cross-validated improvement within a loose 3 % / 2 % bound. On a
multi-lattice crystal the second lattice's spots bias every cross-validation
fold the same way, so the "it improved" test cannot catch it and a spurious
distance<->cell pair (their radial degeneracy) is committed far off the true
geometry, losing reflections in the re-integration.

Tighten the commit bound to < 1 % (a calibrated header needs < ~0.6 %). The
move SIZE discriminates a genuine header correction from the multi-lattice
failure far better than the fit residual, which genuinely marginal (noisy or
ice-ring) data shares with the multi-lattice case - so a real ~0.3 % correction
is kept while the ~1.7 % multi-lattice one is rejected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: reindex the de-novo primitive cell to the group's conventional setting (two-pass) ... e97a610091

When the two-pass second pass indexes de novo and the pseudo-symmetry guard
keeps a triclinic (primitive) cell - a huge oblique cell whose constrained
centred refine is ill-posed - the reflections are left in the primitive frame
while the reused merge space group expects its conventional setting, so the
merge folds the wrong equivalents (a centred group's half-set CC1/2 collapses).
Reindex the integrated reflections into the group's conventional setting first,
using the same exact-integer primitive->conventional change of basis the
space-group search applies to a demoted result, gated on LatticeSearch
recovering the group's own metric (crystal system + centring).

Fires only for a demoted-triclinic indexing under a higher-symmetry reused
group; every other dataset is bit-identical. Closes the last centred-space-group
case where the second pass's merge was frame-mismatched (CC1/2 72 -> 98).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: per-batch relative-B correction + radiation-damage B monitor ... c5cfdcef7e

Rotation scaling modelled dose-dependent scattering power with a single
global relative-B decay slope only (RefineDecay); per-frame refinement was
scale-only. Add a per-batch relative Debye-Waller B, both as an opt-in
correction and as an always-on measurement.

- FitRelativeBCurve / SolveCurvatureSmoothedB: per rotation-range batch, fit
  the s^2 slope of ln(Iref/Iobs) that the resolution-flat per-frame G leaves,
  regularised by a second-difference (curvature) penalty so a genuine smooth
  relative-B is kept while resolution-correlated noise (which an unconstrained
  per-batch fit would chase into an oscillating curve) is suppressed.

- Correction (--relative-b[=deg], default off): applied after RefineDecay,
  gated by a physical peak-to-peak floor and cross-validated by ASU-group
  parity (a per-batch parameter owns whole frames, so it is scored on held-out
  equivalents, not held-out frames). On a battery re-scaling identical stored
  reflections it is a no-op on well-behaved data and a small gain on
  radiation-damaged rotation data; never a regression.

- Radiation-damage monitor (always on for rotation, report-only): measure the
  relative-B change from the pristine low-dose start of the run to its end
  (a whole-run-mean reference inverts on damaged data) plus the per-batch
  curve, and report them on the command line (in the radiation-damage report
  and the merge statistics) and in the mmCIF (_reflns.jfjoch_radiation_damage_
  relative_B and a per-batch loop). A large magnitude flags radiation damage,
  complementing the existing per-image CC / mosaicity read.

- --scale now reproduces the de-novo rotation merge (it previously omitted
  scale-fulls and smooth-G, silently using a weaker model), and honours
  --relative-b, so an offline re-scale matches the full pipeline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: header line for the per-image _image.dat scaling table ... 6e277668b6

The offline per-image scaling table had bare numeric columns. Prefix it with
a '#' comment header (image_number scale_G bfactor_Ang2 mosaicity_deg
wedge_deg cc_to_merge cc_n) so it is self-describing; numpy.loadtxt / gnuplot
skip the comment line, so existing consumers are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux/viewer: rotation geometry post-refine on by default ... 09e4c1116e

The two-pass rotation geometry post-refine (post-refine detector distance /
beam / cell / rotation axis from the whole sweep, then re-integrate; writes
both passes, _01 header geometry and _02 refined) is robust across the test
battery, so make it the default.

- rugnux CLI: replace the opt-in --rotation-post-refine with an opt-out
  --rotation-no-postrefine; a rotation full-analysis run now post-refines
  unless disabled.
- jfjoch_viewer: add a "Post-refine geometry (rotation, two-pass)" checkbox
  to the analyze-dataset dialog, default on for rotation datasets (a no-op
  for stills), mirroring the existing stills "Refine geometry" toggle; it
  sets ProcessConfig::rotation_postrefine_geometry, which the viewer already
  feeds to the same Rugnux::Run the CLI uses.
- The "Copy command" line renders --rotation-no-postrefine when the box is
  cleared on a rotation dataset.

The ProcessConfig struct default stays off, so the online broker path is
unchanged; only the two user-facing entry points (CLI, viewer) default it on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: two-pass refined pass takes the canonical (unsuffixed) output name ... fcafc61be6

In the rotation two-pass geometry post-refine, the second (refined-geometry)
pass is the result users want, so give it the plain <prefix>_* name and keep
the first (header-geometry) pass as <prefix>_01_*. Previously the refined pass
was <prefix>_02_*, leaving no plain <prefix>_* output; a downstream consumer
that expects the canonical name now finds the refined result there.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: rc.160 changelog + CPU_DATA_ANALYSIS/RUGNUX updates ... 99617c549a

Document the newest rc.160 work: rotation geometry post-refinement (now
default-on), the opt-in per-batch relative-B, the always-on radiation-damage
report, net-absence space-group centering ranking, and software/geometry
provenance in the merged mmCIF. Add a §7.5 for the two-pass post-refinement.
Drop the removed .hkl output from RUGNUX.md and add the new CLI options
(--rotation-no-postrefine, --relative-b, --refine-geometry, --background-trim,
--stills-modulation, --still-partiality, --spot-low-resolution,
--min-pix-per-spot, -S symbol, -r flex).

Add a plain-lysozyme (HEWL) carve-out to the no-sample-identity rule: the
field's universal standard specimen and its reference cell are allowed, while
named user datasets and every other sample stay forbidden.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: code-review fixes across the offline path ... 812ea94a86

- frontend: drop the CBF/TIFF file-format menu items; the broker now rejects
  those formats, so offering them in the picker was a dead, error-producing
  choice (the enum values remain only for back-compat decode).
- RugnuxCommandLine: emit --background-trim (when it differs from the 0.10
  default) and --stills-modulation, so the viewer "copy command line" matches
  what the settings dock actually configured.
- ReindexAmbiguity: multiply the reference-CC intensity by rlp rather than
  dividing by it, matching image_scale_corr everywhere else; drop the now
  unused SafeInv helper.
- rugnux_cli: parse --refine-geometry with the hardened bounded parser instead
  of atoi (reject trailing garbage; bound N so refine_frames*50 can't overflow).
- LoadFCalcFromMtz: drop the generic "first type-F column" catch-all so a
  reference MTZ carrying only map coefficients (FWT/FC_ALL) can't silently seed
  CCref from the wrong quantity.
- WriteReflections: consolidate the duplicated MTZ Friedel/anomalous branches
  into one shared row list + column/row emission (behaviour-preserving).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: two-pass supercell-collapse guard + post-refine fixes ... e6cb627c6f

- Guard the two-pass second pass against a bistable de-novo lattice search
  collapsing onto a spurious supercell after the small post-refine geometry
  move: when pass 2's cell volume balloons past 1.5x pass 1, re-run pass 2 with
  pass-1's correct lattice forced (still integrating at the refined geometry).
  A genuine de-novo demotion to a smaller primitive (pass2 <= pass1) is left
  untouched.
- Keep an empty output prefix empty in the "compute stats, persist nothing"
  mode so the two-pass first pass does not write stray _01_* files.
- Fix the stale ProcessConfig doc-comment to describe the actual output
  (canonical unsuffixed _* second pass, header-geometry _01_* first pass) and
  the --rotation-no-postrefine default-on flag, not the removed _02_*.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-19 00:18:20 +02:00

rugnux: supercell-collapse guard forces pass-1's full indexer result ... 074aca55e8

The two-pass supercell-collapse guard re-ran the second pass with only pass-1's
LATTICE forced (RotationIndexer::ForceLattice), which drops pass-1's refined
per-frame orientation, rotation axis and search metadata (ForceLattice sets a
stub search result and leaves updated_geom_/axis_ unset) - so the re-run
integrated markedly worse than de-novo at the same geometry (measured ~3.3
lower ISa / ~1.5% higher R-meas on the affected dataset).

Capture pass-1's full RotationIndexerResult and force THAT (ForceResult) in the
re-run instead. The second pass then integrates at the refined geometry with
pass-1's refined orientation on the correct cell, matching (slightly beating)
de-novo pass-1 quality - a genuine refined-geometry two-pass result, not a
fallback to the header-geometry pass. Isolation confirmed the geometry move
itself and the mosaicity override were not the cause; the lattice-only force
was. The guard still fires only when the second-pass cell is markedly larger
than pass-1 (a spurious supercell), so all other datasets are untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-19 07:12:00 +02:00

tests: seed spindle-parallel beam at truth in XtalOptimizer_rotation ... 55efd42045

The gauge-weak beam-centre prior (c140b3567) deliberately restrains the
spindle-parallel beam component toward the header instead of refining it
freely. The rotation test seeded that exact component (beam-X, axis along X)
off-truth, so the intended restraint slightly biased the co-refined distance
and cell-c through the distance-cell degeneracy, pushing both just past the
tight thresholds.

Seed beam-X at its true value (the gauge direction is meant to be trusted to
the header); keep beam-Y perturbed to still exercise beam-centre recovery in
the well-constrained direction. Distance/cell now recover within the original
thresholds. Source (the intended prior) untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-19 08:00:10 +02:00

rugnux: also write merged reflections as SHELX HKLF-4 (.hkl) ... a79f05134b

WriteReflections now emits a third format alongside the MTZ and mmCIF: a
SHELX HKLF-4 text file (<prefix>.hkl), the direct input for SHELXC / ANODE /
SHELXD. Fixed FORMAT(3I4,2F8.2), one record per reflection as h k l I sig(I),
ending with the 0 0 0 terminator.

Two things make it a valid substructure-solution input:
- Bijvoet mates are written separately (I(+) at +hkl, I(-) at -hkl) so the
  anomalous differences survive; a reflection with no anomalous split is
  written once as its mean.
- Intensities are put on a common scale so the largest value fits the fixed
  F8.2 field (negative weak intensities would otherwise overflow it). I and
  sigma share the scale, so the anomalous signal is untouched; the absolute
  scale is irrelevant to SHELXC/ANODE, which use only ratios.

The MTZ writer's Bijvoet row-building is factored into a shared BuildMergedRows
helper so the MTZ and HKL outputs agree (MTZ output is byte-equivalent to
before). docs/RUGNUX.md updated to describe the three reflection formats.

Validated end-to-end on lysozyme: SHELXC reads the .hkl and reports real
anomalous signal (<d"/sig> 1.6 at low res), and ANODE locates the Met/Cys S
and a chloride at 6.5-9 sigma from the unmodified file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f merged commit 67dca388bd into main 2026-07-19 09:39:29 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#70