18 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	eae7ea90dc	rugnux: always write both MTZ and CIF; drop --scaling-output ... The merged reflections were written in one of mtz/cif/txt selected by --scaling-output. Write both an MTZ and an mmCIF unconditionally instead - each has its uses downstream (MTZ for the CCP4/phenix tools, mmCIF for deposition) - and remove the format selector, the plain-text .hkl writer, and the now-unused IntensityFormat enum / ScalingSettings::FileFormat plumbing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:45:08 +02:00
leonarski_fandClaude Opus 4.8	0e4f91b219	rugnux: validate offline spot-finding settings and bound numeric options ... The offline rugnux path never ran CheckDataProcessingSettings (only the online broker/receiver paths did), so the new --min-pix-per-spot and --spot-low-resolution knobs were unbounded: --min-pix-per-spot 0 silently disabled the per-spot filter, and a low-resolution limit finer than the high-resolution limit made spot finding reject every pixel with no diagnostic. Add a parse_number_arg<T> helper (integral or floating, with optional inclusive bounds) that rejects non-numeric input, trailing garbage, and out-of-range values, replacing the raw atoi/atof in the spot-finding options; parse_double_arg/parse_float_arg become thin wrappers over it. After assembling the settings, validate them with the same CheckDataProcessingSettings the online receivers use, which also enforces the cross-field low>=high-resolution constraint. Also scrubs a residual sample-name comment (the r=6 box-default note) in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:19:59 +02:00
leonarski_fandClaude Opus 4.8	abbee2d4dc	Remove sample identities from the repository; document the rule ... Datasets may be confidential; sample names and measured unit cells committed to the repo can leak outside the group working on them. Scrub existing occurrences and add a "No sample identities in the repository" section to CLAUDE.md (forbidden: sample/dataset names, internal codes, measured cells tied to a sample; fine: space group / lattice / twinning descriptors). - Comments: replace internal dataset codes and protein names with the crystallographic situation they illustrate (centred vs pseudo-symmetric, holohedral, cubic, F-cubic/hexagonal, ...). - Docs: same, in the analysis/writer/stream references and example configs. - Tests: rename sample-named identifiers, TEST_CASE names, file prefixes and asserted labels to neutral crystallographic names (e.g. tetragonal_uc); behaviour unchanged. Reduce the CrystFEL reference PDB to a bare CRYST1 cell file (cell.pdb) and rename the reference data file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:19:43 +02:00
leonarski_fandClaude Opus 4.8	bd094e585f	rugnux: accept a space-group symbol for -S, not just a number ... -S now takes either a number ("92") or a Hermann-Mauguin symbol ("P43212"), resolved via gemmi find_spacegroup_by_name. Previously a non-numeric argument was atoi'd to 0 and tripped the early space-group validation, aborting the run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 15:23:31 +02:00
leonarski_fandClaude Opus 4.8	5d1ddccee8	viewer: expose stills processing settings; rename geom refinement multi -> flex ... Wire the recently-added stills knobs into jfjoch_viewer - they were reachable only from the rugnux CLI, and the viewer had silently diverged from the CLI once --refine-geometry became default-on for stills-with-cell. - Stills geometry-refinement two-pass (--refine-geometry): a checkbox + frame count in the reprocessing-job dialog, offered (and defaulted on, matching the CLI) only for a stills-with-cell run; wired into ProcessConfig and the "Copy command" generator (joined =N, or =off to reproduce opting out of the CLI default-on). - Stills partiality (--still-partiality) checkbox in Bragg integration and the partiality-uncertainty merge term (--partiality-uncertainty) in Scaling, both mirrored in the command-line generator. Rename the "best per-image refinement" mode from "multi" to "flex" everywhere: GeomRefinementAlgorithmEnum::Flex, CLI -r flex (with "multi" kept as a hidden back-compat alias), OpenAPI enum "Flex" + regenerated C++/TS clients, viewer combo, frontend dropdown, and the stored HDF5 string. "multi" collides with CrystFEL's multi-lattice sense; "flex" = let the pipeline decide. The API now carries a dedicated Flex value instead of masquerading as BeamCenter. Also fix a pre-existing bug in Convert(IndexingSettings): OrientationOnly was missing from the outbound switch and serialized as an invalid API value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 11:28:09 +02:00
leonarski_fandClaude Opus 4.8	135765e5b5	rugnux: default --refine-geometry on for stills-with-cell; accept F/FP reference columns ... Two stills usability fixes. 1. --refine-geometry now defaults ON for stills whenever a reference cell is available (-C or a reference MTZ) - exactly the case where the geometry bundle-adjust can act (it anchors on a known cell) and where it lifts weak/ sparse-stills indexing (OCP +42%, KR2 +108% indexed in the target study). It stays a no-op for rotation (own two-pass) and de-novo stills (no cell yet), so auto-enabling it only where it does something avoids spurious "skipping" warnings. --refine-geometry=off opts out; explicit --refine-geometry[=N] forces it. 2. The reference-MTZ auto column selection now falls back to a plain amplitude column (FP / FOBS / F / FC, squared to an intensity) after F-model and the intensity (J) columns. A deposition that carries only structure-factor amplitudes (e.g. KR2's 8cl8, whose only usable column is FP) now seeds CCref without needing an explicit --reference-column FP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 07:27:25 +02:00
leonarski_fandClaude Opus 4.8	4c5da20eda	rugnux: add --partiality-uncertainty stills merge term (partiality-model error) ... A stills reflection recorded at partiality p carries a systematic intensity error ~(dp/p) that is proportional to <I> and grows as p falls; plain counting sigma misses it, so strong low-p partials are over-trusted in the merge. Add sigma^2 += (c*<I>*(1-p))^2 in MergeOnTheFly::CorrectedSigma - the stills-partiality analog of the existing rotation --capture-uncertainty term ((1-captured_fraction)*I in RotationScaleMerge). Inert when partiality==1 (no --still-partiality), and gated on a real systematic (error_model_b > 1, i.e. ISa < 1) so it fires on strong/medium stills but auto-skips weak counting-limited data where it would only over-concentrate the merge and hurt. Opt-in via --partiality-uncertainty <c> (default 0; ~2.5 recommended with --still-partiality). Prototyped (Python replica of the merge): lyso8 CC1/2 +1.7, CCref +4.8, R-free -0.023; LOV CCref +2; harmful on weak OCP (hence the b>1 gate). Full in-binary validation across all serial targets pending. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 19:52:46 +02:00
leonarski_fandClaude Opus 4.8	2f5ed411e2	rugnux: --refine-geometry stills two-pass (global bundle adjustment + re-index) ... Weak/jet serial stills are often geometry-limited: a few-px beam error or a mm-scale detector distance error (unreachable per-image) fails many frames, but is well-determined jointly from the strong frames. Mirror the rotation two-pass: an index-only first pass over a spread sample collects each indexed frame's spots + assigned HKL + orientation; the strongest ~N (default 200) feed one Ceres bundle adjustment; the refined geometry is applied and the main pass re-indexes + integrates + merges every frame from scratch. GeometryRefiner (reusing the extracted XtalResidual - the RecipToDetector geometry residual pulled out of XtalOptimizer, behaviour-preserving): one problem with SHARED beam(2)/distance(1)/cell-length(3) blocks + a PER-FRAME orientation(3) block, robust Cauchy loss, a cell-length regularizer anchoring the known cell to break the low-resolution distance<->cell-scale degeneracy, DENSE_SCHUR eliminating the per-frame orientations, and a 3-round HKL-reassignment / tolerance-tightening loop. Tilt is not refined (gauge-coupled, zero gain). Opt-in via --refine-geometry[=N]; stills only (rotation untouched). Validated: KR2 7.58% -> 21.85% (matches CrystFEL's 21.5%; a real ~1.4mm distance error + ~3px beam), OCP 2.92% -> 4.19% (~3px beam). OFF runs are bit-identical to baseline (XtalResidual extraction non-regressing; rotation lyso_ref de-novo ISa 17.3 unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 18:17:58 +02:00
leonarski_fandClaude Opus 4.8	95af137146	rugnux: the external reference MTZ is never a scale anchor (stills + rotation) ... An external reference MTZ is a poor per-image scale reference: it is a different crystal/dataset, so scaling against it injects cross-dataset systematics rather than removing per-image scale error (measured: reference-scaled R-free 0.410 vs self-scaled 0.388 on OCP; 0.378 vs 0.350 on LOV). Its real value is fixing the cell/space group, breaking the merohedral indexing ambiguity, reporting CCref and providing the R-free test set - none of which need it to be a scale anchor. Make that consistent across both workflows: - Stills: the per-image pass (ScaleImage) now uses the reference ONLY to break the indexing ambiguity (once, for good), not to fit G. Scaling self-references at the post-measurement merge for every run, whether or not a reference is given (Rugnux + rugnux_cli --scale). - Rotation: ScaleImage no longer scales against the reference either (it was dead code - RotationScaleMerge recomputes the per-frame scale, so the reference never actually moved the result). Rotation resolves the ambiguity globally (ChooseReindex / ReferenceIntensityCC) and self-scales in RotationScaleMerge. - A reference-as-scale mode, if ever wanted, would be a dedicated later step, not this per-image pass. Validated: OCP CCref 56.5%->64.6%, R-free 0.400->0.393 (honest 2.36 A cutoff instead of a reference- propped 1.5 A); LOV CCref 88.3%->90.1%, R-free 0.336->0.331 with more reflections; rotation lyso_ref with the reference gives ISa 17.8 == de-novo 15.6 (same 15975 reflections, CC1/2 99.8%) - the reference demonstrably does not touch rotation scaling, only cell/SG + ambiguity + CCref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 13:01:56 +02:00
leonarski_fandClaude Opus 4.8	9527fa6e2c	rugnux: scale self-referenced stills in a single pass (fix weak-data collapse) ... The stills self-scaling loop rebuilt the merge reference FROM the just-scaled reflections every iteration, so on weak data each pass re-fit its own noise and the merged CC1/2 collapsed (PfluDING combine: 0.3% at the default 3 iters). It is provably pointless too: ScaleOnTheFly::Scale re-solves the per-image G from raw I and never reads the prior correction, so against a fixed reference passes 2..N are bit-identical - the single G is the exact one-pass solution and iteration only ever does the harmful reference rebuild (N=1 is best on every dataset, including strong lyso 82% vs 76% at N=3; leave-one-out and external anchors were tested and do not beat it). Do one self-reference pass; rotation (RotationScaleMerge) and the external-reference branch are untouched. Validated: PfluDING-combined CC1/2 0.3% -> 42.0% by default (pinned 2.5 A); strong data unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:05:08 +02:00
leonarski_fandClaude Opus 4.8	4a5028d98d	rugnux: add -r multi geometry refinement (per-image best of three) ... Per-image geometry refinement is a tradeoff that flips per dataset: beam+cell refinement extends OCP merge quality but DIVERGES on sparse-spot stills (e.g. KR2's ~20 spots around a 232 A axis), pushing good lattices out of tolerance so they fail the acceptance floor. `-r multi` runs all three (none / orientation / beam_and_lattice) on a copy per image and keeps whichever indexes the most spots (scorer = fractional-Miller-within-tolerance count, mirroring AnalyzeIndexing); ties prefer less refinement, to avoid overfitting the sparse list. Validated: KR2 index 7.08% (-r beam_and_lattice default) -> 10.05% (matches the -r none best), while OCP R-free stays ~equal to beam_and_lattice. New GeomRefinementAlgorithmEnum::Multi handled in the CLI/HDF5/command-line echoes; the API convert maps it to beam_and_lattice (offline only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:04:32 +02:00
leonarski_fandClaude Opus 4.8	fb399480e5	rugnux: add opt-in stills partiality (--still-partiality) ... Serial stills currently treat every reflection as a full (partiality hardcoded to 1). Add an opt-in Gaussian excitation-error partiality set at prediction time (CPU + CUDA): p = exp(-dist_ewald^2 / (2*sigma^2)), sigma^2 = profile_radius^2 + (bandwidth_sigma*|recip_z|)^2, with sigma = the per-image profile radius (ewald_dist_cutoff/2), so an edge-of-acceptance reflection keeps p ~ exp(-2). Off by default; the merge weight (~p^2) then down-weights far-from-Ewald partials instead of trusting them as fulls. Validated: helps medium/strong stills (LOV R-free 0.336->0.329, lyso8 0.433->0.410, lowers the systematic error-model b in both) but HURTS weak OCP (dividing by a small, uncertain p amplifies orientation error -> high-res noise, resolution collapse), so it is left opt-in. A static forward p explains only ~6-10% of the partiality scatter; the full win needs per-image post-refinement (future work), for which this is the prediction-side groundwork. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:03:21 +02:00
leonarski_fandClaude Opus 4.8	b0202a3aa2	rugnux: default the stills integration box to r=6 (6,8,12) ... Serial-stills spots span a range of crystal orientations within a single shot, so they land wider on the detector than the r1=4 monochromatic-rotation default assumes, and the r1=4 signal box truncates the (bandwidth/mosaicity-broadened) spot wings. The profile-fit integrator can use a larger box for free - its profile weighting drives far-out background pixels to ~zero weight, so the estimator variance saturates rather than growing (a plain box-sum, which sums every pixel at weight 1, degrades). Gated on stills (!rotation_indexing); rotation stays at 4,6,10 (a larger box there only adds background - measured wash-to-negative on the XDS rotation battery, with an SG regression); an explicit --integration-radius wins. Validated: OCP R-free 0.406->0.400, LOV 0.354->0.336; lyso8 1.60->1.57 A; lyso5 (1% DMM) 1.80->1.72 A with +13% reflections at equal CC1/2 - the broadened case benefits most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:00:26 +02:00
leonarski_fandClaude Opus 4.8	db8ae0452e	rugnux: fix --scale on a self-contained _process.h5 (reflections + error model) ... Offline --scale re-scales the reflections stored in a _process.h5 without re-integration, but was broken for a self-contained integrated snapshot: 1. HDF5MetadataSource::ReadReflections fell back per-image to the linked source pixel files for every non-indexed frame (a snapshot's master holds /entry/reflections only for the sparse indexed images). With the raw _data_NNNNNN.h5 absent or not co-located this threw an HDF5 error; when present it needlessly reopened multi-GB files thousands of times. Decide once whether the master is the authoritative reflection store and, if so, never fall back - a missing per-image group just means the frame has none. Legacy/VDS acquisitions (no /entry/reflections in the master) still resolve reflections lazily to the source data files. 2. The scale-only stills branch never fit the (a,b) error model, so it merged with the identity model - far worse intensities than the full pipeline (lyso8 CC1/2 76%->21%). Fit RefineErrorModel and honour --reject-outliers, mirroring Rugnux.cpp. Round-trip validated: full merge == integrate(--no-merge) + --scale (identical error model a=0.793 b=2.287, matching unique reflection counts), and --scale now reads reflections in ~2.5s with no raw-file access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 10:06:07 +02:00
leonarski_fandClaude Opus 4.8	097b9a29d1	rugnux: add --spot-low-resolution CLI option (exclude beam halo) ... The spot-finding low-resolution limit was hardcoded at 50 A (r ~36 px), which admits the direct-beam halo/overload zone (24-50 A) on weakly-diffracting serial data. On KR2 (I222, 235 A axis, ~3.9 A diffraction) that halo is ~387 strong pixels/frame vs ~25 real Bragg pixels, swamping the signal ~15:1 (CrystFEL cuts it with min-res=75 px = 24 A). Expose the limit so it can be tightened, e.g. --spot-low-resolution 24. Default unchanged (50 A). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 23:14:27 +02:00
leonarski_fandClaude Opus 4.8	2f3041c44e	rugnux: add --min-pix-per-spot CLI option (spot finding) ... The minimum connected strong-pixel count per spot was hardcoded to 2 (SpotFindingSettings::min_pix_per_spot). Expose it on the CLI so serial-stills data can be run with min-pix 1 (plus a higher --spot-threshold), matching the CrystFEL peakfinder8 practice that trades noisier spots for a higher indexing rate. Default unchanged (2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 22:02:49 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00