1187 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	c5cfdcef7e	rugnux: per-batch relative-B correction + radiation-damage B monitor ... Rotation scaling modelled dose-dependent scattering power with a single global relative-B decay slope only (RefineDecay); per-frame refinement was scale-only. Add a per-batch relative Debye-Waller B, both as an opt-in correction and as an always-on measurement. - FitRelativeBCurve / SolveCurvatureSmoothedB: per rotation-range batch, fit the s^2 slope of ln(Iref/Iobs) that the resolution-flat per-frame G leaves, regularised by a second-difference (curvature) penalty so a genuine smooth relative-B is kept while resolution-correlated noise (which an unconstrained per-batch fit would chase into an oscillating curve) is suppressed. - Correction (--relative-b[=deg], default off): applied after RefineDecay, gated by a physical peak-to-peak floor and cross-validated by ASU-group parity (a per-batch parameter owns whole frames, so it is scored on held-out equivalents, not held-out frames). On a battery re-scaling identical stored reflections it is a no-op on well-behaved data and a small gain on radiation-damaged rotation data; never a regression. - Radiation-damage monitor (always on for rotation, report-only): measure the relative-B change from the pristine low-dose start of the run to its end (a whole-run-mean reference inverts on damaged data) plus the per-batch curve, and report them on the command line (in the radiation-damage report and the merge statistics) and in the mmCIF (_reflns.jfjoch_radiation_damage_ relative_B and a per-batch loop). A large magnitude flags radiation damage, complementing the existing per-image CC / mosaicity read. - --scale now reproduces the de-novo rotation merge (it previously omitted scale-fulls and smooth-G, silently using a weaker model), and honours --relative-b, so an offline re-scale matches the full pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 22:08:09 +02:00
leonarski_fandClaude Opus 4.8	e97a610091	rugnux: reindex the de-novo primitive cell to the group's conventional setting (two-pass) ... When the two-pass second pass indexes de novo and the pseudo-symmetry guard keeps a triclinic (primitive) cell - a huge oblique cell whose constrained centred refine is ill-posed - the reflections are left in the primitive frame while the reused merge space group expects its conventional setting, so the merge folds the wrong equivalents (a centred group's half-set CC1/2 collapses). Reindex the integrated reflections into the group's conventional setting first, using the same exact-integer primitive->conventional change of basis the space-group search applies to a demoted result, gated on LatticeSearch recovering the group's own metric (crystal system + centring). Fires only for a demoted-triclinic indexing under a higher-symmetry reused group; every other dataset is bit-identical. Closes the last centred-space-group case where the second pass's merge was frame-mismatched (CC1/2 72 -> 98). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 20:03:47 +02:00
leonarski_fandClaude Opus 4.8	290c30b9be	rugnux: post-refine commits detector/cell geometry only for a small (< 1 %) move ... The rotation geometry post-refine committed the detector distance and cell scale on any cross-validated improvement within a loose 3 % / 2 % bound. On a multi-lattice crystal the second lattice's spots bias every cross-validation fold the same way, so the "it improved" test cannot catch it and a spurious distance<->cell pair (their radial degeneracy) is committed far off the true geometry, losing reflections in the re-integration. Tighten the commit bound to < 1 % (a calibrated header needs < ~0.6 %). The move SIZE discriminates a genuine header correction from the multi-lattice failure far better than the fit residual, which genuinely marginal (noisy or ice-ring) data shares with the multi-lattice case - so a real ~0.3 % correction is kept while the ~1.7 % multi-lattice one is rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 19:41:12 +02:00
leonarski_fandClaude Opus 4.8	0ebf90a1ad	rugnux: two-pass second pass re-indexes de novo, reuses the group for the merge only ... The two-pass second pass reused pass-1's space group for both the indexer and the merge. Reusing it in the INDEXER disables the de-novo pseudo-symmetry safeguards, so build_sr adopts a wrong cell: a metrically pseudo-centred crystal doubles into a super-cell (every real reflection plus phantom half-integer ones - so the merge keeps CC1/2 but halves <I/sigma>), and a huge oblique cell's constrained monoclinic refine is ill-posed, diverges, and the re-indexing collapses. Decouple the two: the second pass now indexes DE NOVO (the group is cleared before it, safeguards on) so it recovers the true cell, and pass-1's group is reinstated for the MERGE ONLY, just before RotationScaleMerge, so the symmetry stays fixed exactly as a user-fixed group would - no re-search, no flip. A reindex-derived group is still left to re-search. Neutral for genuine-symmetry data (its de-novo cell already matches the group); it recovers the pseudo-symmetric cases that previously collapsed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 19:41:12 +02:00
leonarski_fandClaude Opus 4.8	1089b75906	rugnux: two-pass writes both passes (_01 header geometry, _02 refined) ... The two-pass geometry post-refinement previously wrote only the second (refined-geometry) pass. Write both instead, so the header-geometry and post-refined results can be compared directly and the better one kept: <prefix>_01_* first pass, at the header geometry (the plain single-pass baseline) <prefix>_02_* second pass, at the post-refined geometry Each is a complete standalone result - its own merged reflections (.mtz/.cif), per-image _process.h5 (with --write-process-h5), ice-ring flags and mmCIF detector geometry, all consistent with the geometry that pass actually used. Mechanics: the pre-pass now stores the post-refined detector geometry instead of applying it mid-pass (so the first pass finishes and writes at the header geometry), and no longer returns early - it continues through the full write. Run() then applies the stored geometry, lets the second pass re-search a reindex-derived space group, and routes the two passes to the _01 / _02 prefixes. The second pass's result is unchanged from before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 17:46:20 +02:00
leonarski_fandClaude Opus 4.8	c140b3567d	rugnux: restrain the gauge-weak beam centre toward the header in geometry refinement ... Single-axis rotation has a gauge degeneracy: rotating the whole experiment about the spindle leaves every spot position unchanged, so the beam-centre component parallel to the spindle is a null / under-determined direction. Refining it freely lets it wander a few pixels and absorb centroid systematics into a wrong beam that the co-refined orientation keeps position-consistent. Add a soft header prior on that one component in the shared XtalOptimizer: residual = w*(beam[parallel] - header), with w set so the restraint behaves like a sigma_px-pixel prior competing with the positional residuals. The gauge direction has ~zero data sensitivity, so the prior pins it near the LaB6-monitored header, while a real, well-supported drift can still overcome it (the beam does drift). One restraint covers all three refinement sites at once: the noisy accumulated-spot primary, the per-frame filter, and post-refine. Validated against independent XDS processing: neutral on well-determined data (the gauge direction has no effect there), and a real low-resolution accuracy gain where the free refinement had otherwise wandered off the header - the merged intensities move back into agreement with XDS while the internal CC1/2, which is blind to this, barely changes. No space-group determinations change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 17:31:45 +02:00
leonarski_fandClaude Opus 4.8	c9ed9c3776	rugnux: two-pass reuses pass-1 space group; floor the fed mosaicity ... The second pass re-searching the space group at the slightly changed geometry could flip a borderline determination (cubic <-> orthorhombic) and mix the two passes, collapsing the multiplicity. Fix it: the pre-pass now continues through the space-group search, so the determined group is already fixed on experiment_ and the second pass reuses it instead of re-searching. The lattice is NOT forced - the second pass re-indexes at the refined geometry so the cell stays self-consistent with it (forcing the pre-pass lattice would pin the nominal cell and undo the refinement). A reindex-derived group (the centred-lattice test) is in a conventional setting the indexer's primitive frame does not share, so it is cleared for the second pass to re-search (else its indexing fails outright). Also floor the mosaicity fed to the second pass's prediction at each frame's own estimate (max), so a too-narrow smoothed value can never make prediction drop reflections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 13:04:04 +02:00
leonarski_fandClaude Opus 4.8	1b91823eed	rugnux: --rotation-post-refine two-pass geometry post-refinement (opt-in) ... A first pass integrates and post-refines the detector geometry from the observed diffraction, applies it to experiment_, and a second pass re-indexes and re-integrates with it - only the second pass is written, so the _process.h5 / mtz (and the ice-ring flags, which move with the geometry) reflect the refined values. Default off; enable with --rotation-post-refine. PostRefineRotationGeometry refines in two SEPARATE cross-validated steps (a joint fit of everything is fragile - the same lesson as integration, where refining the profile width and the scale jointly fails but separately works): * Step A: cell scale + rotation axis from the observed rocking centroids phi_obs (a distance-independent excitation residual, so it pins the absolute cell scale that the positional residual leaves degenerate with distance). * Step B: detector distance + beam centre from the observed spot positions, with the cell fixed at step A. Each step commits only if it lowers a held-out (deterministic split-half) residual, else that part of the geometry is left at nominal. Rugnux::Run wraps RunPipeline(write_output, geometry_prepass); the pre-pass runs the first scale/merge so RotationScaleMerge fits a frame-order-smoothed mosaicity, which is captured and fed to the second pass's Bragg prediction (via IndexAndRefine::SetPredictionMosaicityOverride) rather than re-derived per image. observed_x/y were plumbed through the profile integrator earlier for the positional residual. rugnux_vs_xds.py gains --extra-args to sweep the option over the battery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 12:19:33 +02:00
leonarski_fandClaude Opus 4.8	f396385e11	rugnux: record the detector distance and beam centre in the merged mmCIF ... Add _diffrn_detector.jfjoch_{distance_mm,beam_center_x_pxl,beam_center_y_pxl} to the merged mmCIF header, as feedback of the geometry the data was actually reduced with. When geometry refinement ran (the rotation two-pass or the stills global refinement, which update experiment_ before the written pass) these are the refined values; otherwise they are the nominal header geometry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 12:19:16 +02:00
leonarski_fandClaude Opus 4.8	9bfecaadf5	beam_stop: add ShadowFinder detection class (unwired) ... Add image_analysis/beam_stop/ShadowFinder.{h,cpp} and SHADOW_FINDER.md: a self-contained beam-stop shadow detector that accumulates images via AddImage() and returns a mask from GetMask(), mirroring the shape of DarkMaskAnalysis. It detects the beam-stop shadow (central disk + holder arm) as an azimuthal anomaly: an iterated radial-median background baseline, a ratio threshold, a connectivity-to-beam-centre anchor with module-gap bridging, a central low-res disk guard capped just inside the innermost reflection, and a reflection guard that never masks a pixel that recorded real signal. Not yet wired: not added to image_analysis/CMakeLists.txt and no PixelMask / Rugnux / viewer changes. SHADOW_FINDER.md documents the algorithm and the deferred offline (Rugnux bit 9 + viewer user-mask) integration plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 12:05:05 +02:00
leonarski_fandClaude Opus 4.8	1911875e1c	integration: carry the box-sum observed centroid through the profile path ... Pass A (the box-sum) always computes an intensity-weighted centroid (the observed spot position), but it was only emitted in BoxSum mode. Rotation integration runs the profile path, so observed_x/y were left NAN there and no positional (detector<->reciprocal) residual was possible downstream. Emit the Pass-A centroid in the profile path too (CPU) and copy d_obs_x/d_obs_y back in all modes (GPU) - Pass A fills them regardless of integrator mode. observed_x/y are consumed only by the diagnostic HDF5 reflection table and the viewer read-back; nothing in scaling/merge reads them, so merged output is unchanged - the _process.h5 observed_x/y columns simply go from NAN to the real centroid. This makes the observed position available to post-refinement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 10:23:17 +02:00
leonarski_fandClaude Opus 4.8	7ceffde5a4	rugnux: rank space groups by net absences, not gross (fix C222 over-centering) ... Stage B of SearchSpaceGroup chose the largest centering that explained the most absences and was self-consistent, ranking candidates by the gross observed-absent count. A false super-centering over-claims: on a C222 crystal, F222 predicts every C absence (all genuinely weak) PLUS a block of C-present reflections it wrongly calls absent. Its absent class is then bimodal - many true zeros diluting a strong block - so its mean <I/sigma>abs (and the violation-rate arm) slip under the centering strength gate, which cannot see the over-claim in isolation. With the gross-count ranking F222 then beat the true C222, and the strength gate is scale-invariant here, so real C-centered data trips it identically (the rotation battery just has no C-centered crystal on this path; the synthetic unit test does). Rank instead by the absences GENUINELY explained, absent_observed - absent_violations, then by fewer violations, then lower space-group number. C222 and F222 net to the same count (F222's genuine absences ARE C222's), so the fewer-violations tie-break keeps the honest, less-centred C222. Symmetric: on a real F222 crystal F explains strictly more weak absences and still wins. Also tighten the "indistinguishable alternatives" grouping to require the same absent AND violation counts, so a super-centering is never reported as an equal alternative. No tuned threshold is changed; the ranking is a no-op for any 0-violation winner. Fixes the SearchSpaceGroup C222 synthetic-space-group test. Rotation battery (32 crystals, de-novo) is space-group-identical to before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 22:35:12 +02:00
leonarski_fandClaude Opus 4.8	742a1c1b94	rugnux: identify the producing software in the merged mmCIF and MTZ ... mmCIF: set _software.name to 'Rugnux' (was 'Jungfraujoch') and correct _software.classification to the CIF-dictionary value 'data reduction'. MTZ: add the producing software to the header via the title and a HISTORY record ("From Rugnux <version>, data reduction"), the CCP4 provenance channel, which was previously empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 21:29:47 +02:00
leonarski_fandClaude Opus 4.8	e209069ccd	docs: compress the rc.160 changelog entries to one-line summaries ... The rc.160 entries were multi-sentence explanations; rewrite each as a short statement of what changed (feature / flag / behaviour), roughly under 200 characters, matching a normal changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 21:02:34 +02:00
leonarski_fandClaude Opus 4.8	3a23c79ea4	docs: document the asymptotic ISa reporting and the azint Q-field precision ... CHANGELOG: describe reporting ISa at its asymptotic definition (read from the strong equivalents, relaxed threshold on weak/damaged data) and the frontend azimuthal-integration Q fields gaining 5-decimal precision. CPU_DATA_ANALYSIS: update the error-model section - ISa is the counting- subtracted strong-reflection asymptote (and drives the systematic floor), not 1/b of the whole-range fit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:58:38 +02:00
leonarski_fandClaude Opus 4.8	59e260876a	rugnux: recover the asymptotic ISa on weak / radiation-damaged data ... The asymptote was read only from groups with I/sigma > 20 and required at least 100 of them, so weak or radiation-damaged data (few strong reflections) fell back to the higher whole-range b and reported an over-conservative ISa. Relax the I/sigma threshold in a second tier (>= 50 groups above I/sigma 10) when the tight one is unmet; the per-group counting-subtracted estimate is stable down to that threshold. Data with many strong groups is unchanged (it keeps the tight-threshold asymptote); weak / damaged data now reports the asymptote its own reflections support instead of falling back. Battery (32 crystals): no space-group, R_meas or CC1/2 change; strong sets unchanged; e.g. a room-temperature radiation-damage series recovers ISa 5.4 -> 11.2 (its data already exceeds the reference on CC1/2 and R_meas). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	88359a57ee	rugnux: report ISa at its asymptotic definition (strong-reflection limit) ... ISa is the I->infinity limit of the signal-to-noise (Diederichs, Acta Cryst. D66 (2010), 733-740), i.e. the reproducibility of the strongest reflections. The reported value was taken as 1/b of the whole-intensity-range error-model fit, whose systematic term b is raised by a mild excess of scatter at intermediate intensity, so it understated that limit. Read the asymptote directly instead: for each well-measured reflection group the counting-subtracted fractional scatter of its symmetry mates estimates the systematic term, and the robust median over strong groups is the asymptote. It falls back to the whole-range b when too few strong groups exist (weak or low-multiplicity data). The reported ISa and the merged-sigma systematic floor use this value, so merged I/sigma approaches ISa. The (a, b) fit and the per-observation error model are unchanged, so the merge weights, anomalous statistics and chi^2 are untouched, and CC1/2 and R_meas (which do not depend on sigma) are bit-identical. Battery (32 crystals): no space-group or indexing change, R_meas and CC1/2 unchanged, ISa rises to the true asymptote of the data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	174b9585c5	docs: document SigAno and the low-energy centering fix ... CHANGELOG (rc.160): add the centering rate-gate recovery on weak/low-energy data and the SigAno anomalous signal-to-noise statistic. CPU_DATA_ANALYSIS: describe SigAno in the merging-statistics section - its definition, the standard PDBx mmCIF items, and why a half-set anomalous CC is not used (complementary halves drive it negative on weak shells) - and note the floor-independent centering-acceptance test in the space-group-determination notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	7fe12773de	rugnux: report anomalous signal-to-noise SigAno (stats table + mmCIF) ... Compute SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> per report shell and overall from the full-multiplicity I(+)/I(-) Bijvoet split, store it on MergeStatisticsShell, and surface it in the console merge-statistics table (new SigAno column) and the mmCIF via the standard wwPDB PDBx items _reflns.pdbx_absDiff_over_sigma_anomalous (overall) and _reflns_shell.pdbx_absDiff_over_sigma_anomalous (per shell) - the dictionary's home for this quantity, so no jfjoch_ local item is needed (unlike ISa). SigAno approaches ~0.8 for pure noise and rises above 1 with real anomalous signal. A half-set anomalous CC is deliberately not used: its two half estimates are complementary partitions of one observation pool (dI0 + dI1 = 2*dI_full), so differencing the two Bijvoet hands cancels the large common intensity that keeps CC1/2 non-negative and, once the anomalous SNR per half drops below 1, drives the correlation towards -1 rather than 0 - misrepresenting a weak-but-real signal. Emitted only when an anomalous split was made (has_anom guard), so a non-anomalous merge keeps its stats block and shell-loop columns byte-for-byte. Export-neutral: the I(+)/I(-)/IMEAN accumulation is unchanged. On low-energy S-SAD test data SigAno tracks XDS (13 keV null ~0.89, 6 keV ~1.44, 5 keV ~1.5-1.7) and never goes spuriously negative. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	1b5eb80ba4	rugnux: recover I-centering on weak / low-energy data (centering rate gate) ... The de-novo centering gate promoted to a centred lattice only when the mean signed <I/sigma> of the centering-absent class was <= 0.5x that of the present class. Weak / low-energy data carry a positive intensity floor (background / profile leakage) that lifts <I/sigma> of the genuinely-extinct class to ~1.5-2.3 instead of ~0; when the present class is itself weak that additive floor inflates the ratio past the bound and the true centering is missed. An I-centred cubic crystal recorded at 5 and 13 keV sat at ratio ~0.57 and stayed primitive P2_1 3, while the same crystal form at 6 keV (stronger data) correctly merged as I23 - an energy-dependent space group for one physical crystal. Add a second, floor-independent acceptance path OR'd with the mean-ratio: the centering-absent class's strong-reflection rate (violations / absent) versus the present class's own strong rate. A false centering (absent as strong as present) fails both arms; a true weak centering passes the rate arm. Battery SG sweep vs XDS: the low-energy datasets now adopt I23, and no crystal is newly over-centred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	32c7f9f59f	rugnux: don't veto genuine high symmetry on excellent data (b-veto floor) ... The de-novo space-group search's systematic-b veto - which keeps a merohedral twin in its true lower symmetry - is a RATIO test: reject a point-group promotion whose merge error-model b balloons past 2x the parent subgroup's. That ratio is meaningless when the parent b is near zero, on excellent data where symmetry equivalents already agree almost perfectly. A genuine tetragonal 422 whose 222 parent sits at b=0.008 and 422 at b=0.049 (both tiny, ISa ~20) reads as a 6x balloon and is wrongly demoted to 222. Floor the parent b (min_systematic_b_for_veto = 0.05) before forming the ratio, so the veto only judges a promotion once its b is a non-negligible fraction of I. A real merohedral twin drives b to ~0.19 regardless of how clean the parent is, so it is still caught (the insulin R3 + ~20% twin veto still fires, b=0.19 > 0.15). A P41212 58/58/150 crystal at 16 keV goes P212121 -> P41212; full 26-crystal rotation battery: only that crystal changes (its space group now matches XDS, SG match 22->23/26), every other space group bit-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	afae1dec2f	rugnux: index long finely-spaced axes (de-novo two-pass long-axis rescue) ... De-novo two-pass rotation indexing collapsed on a crystal with a long, finely spaced axis (a ~150 A c-axis at 16 keV): the unconstrained FFT either collapsed the long axis to a short sub-multiple or let a denser supercell over-fit the accumulated cloud - a small global-orientation error throws the many high-order reflections off along the fine axis, so the true cell scores worst on the raw cloud, and propagating that inaccurate global orientation to each frame fails. Add a long-axis rescue that only runs when the standard pass indexes few validation frames (<50%), so well-indexing crystals are untouched and keep the fast path: re-run the first pass at a COARSE resolution (low-order reflections only, where the fine axis stays robust) to recover the true metric, take the recovered cell with the longest axis directly, then RE-INDEX at full resolution with that cell as a reference (the -C path) - the reference-cell filter drops the collapsed/supercell candidates and refines an accurate global lattice. The scheme feed/index/validate loop is factored into a pick_best() lambda shared by the standard and constrained passes (the standard pass is behaviour-preserved). A P41212 58/58/150 test crystal goes 15% -> 99.89% indexed. Full 25-crystal rotation battery: every other crystal's space group and cell BIT-IDENTICAL, and the rescue fires only on the failing crystal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	799e1e15ec	frontend: configurable decimal precision, 5 places for azint Q fields ... NumberTextField quantized every float input to 3 decimals, so the finest q_spacing reachable through the UI was 0.001 - capping the azimuthal-integration q-bin count around 200 regardless of the CPU/FPGA backend. Add an optional `decimals` prop (default 3, unchanged elsewhere) and set it to 5 on the Q spacing / Low Q / High Q fields, matching the q_spacing minimum of 1e-5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-17 19:28:21 +02:00
leonarski_fandClaude Opus 4.8	65bc40e7c7	merge: deltaCChalf uses the deterministic HalfForImage split, not an RNG ... MergeOnTheFly::DeltaCChalfReject assigned its CC1/2 half-sets from a seeded std::mt19937 drawn in image (call) order, whereas the actual merge, the rotation merge, the GPU path and the R-free flags all split with the deterministic HalfForImage(image_id) splitmix64 hash. So the deltaCChalf was measured on a different half-partition than the reported CC1/2, and was order-dependent (a reordered outcomes vector gave different halves). Use HalfForImage(i) - i is the image's stable index, the same image_id AddImage merges with - so deltaCChalf now reflects the exact CC1/2 split the statistics report, order-independently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 13:19:51 +02:00
leonarski_fandClaude Opus 4.8	0c33139983	rugnux: radiation-damage report for rotation (per-image CC vs dose) ... Summarise the per-image scale the rotation merge already fits - the CC-to-merge and mosaicity binned by frame (= dose) across the sweep - and flag a falling per-image CC (or rising mosaicity) as the radiation-damage / crystal-decay signature. A data-quality-vs-dose read complementary to the fitted decay correction; the full per-image table is still written to <prefix>_scaling.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 11:57:51 +02:00
leonarski_fandClaude Opus 4.8	b647cf454c	rugnux: write I(+)/I(-) and F(+)/F(-) by default (rotation) ... A rotation merge now keeps each acentric reflection's Bijvoet split - the inverse-variance I(+)/I(-) from the same scaled fulls - even when the merge is Friedel-averaged (the default, no -A). IMEAN stays the primary intensity and is bit-identical to before; the split is purely additive and scaled non-anomalously, so a weak anomalous signal is preserved in the output without having to reprocess with -A. French-Wilson now also fills F(+)/F(-) from the two hands (one pass, shared Wilson prior). Writers: the default (Friedel-merged) MTZ and mmCIF now carry IMEAN plus I(+)/I(-) and F/F(+)/F(-) whenever a reflection has an anomalous split (the stills path, which computes none, keeps the previous columns). A missing mate or a centric is emitted as the CCP4 missing-value flag (NaN). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 11:52:31 +02:00
leonarski_fandClaude Opus 4.8	b945444372	rugnux_vs_xds.py: find the current rugnux binary (build*/rugnux/rugnux) ... The rugnux CLI target builds to build*/rugnux/rugnux; the old build*/tools/ rugnux paths are stale leftovers from a previous layout that cmake no longer updates, so the harness could silently run weeks-old code. Pick the most recently built binary among the candidates instead of a fixed order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 10:46:09 +02:00
leonarski_fandClaude Opus 4.8	67a18357b5	rugnux: fix de-novo space-group centering and weak-data high symmetry ... De-novo space-group search: - Determine lattice centering from the strength of the systematically-absent class (its mean I/sigma vs the present class) instead of a per-reflection violation count. The count gate was brittle on noisy / obverse-reverse- twinned data, where enough genuinely-absent reflections cross I/sigma>3 to trip the 10% bound although the class is several-fold weaker, and wrongly kept a genuine R-centered lattice primitive. False centerings, whose absent class is as strong as the present one, stay rejected. Screws keep the count test (too few axial reflections to average). - Recover a genuine high-symmetry point group when the merged error model is badly miscalibrated (weak, low-resolution data whose sigmas are far too small). The fixed-sigma chi^2 ratio then grows with point-group order for real symmetry too and wrongly rejects it (a true cubic group reached a chi^2 ratio ~14); the self-normalising systematic-b test stays valid, so once chi2_ref shows the ratio is untrustworthy the promotion is confirmed on the b-test alone. The balloon veto still rejects a twin. This is the gap left when the earlier log10(chi2_ref) widening was removed for the b-veto. Also print the adopted space group and unit cell as a clean one-line summary at the end of the run (de-novo or user-fixed -S), not only in the mmCIF. Full rotation-test battery: exactly the two intended crystals change (both to the correct answer, with lower R-meas and higher ISa), every other crystal unchanged; space-group match 21->22/25. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 10:46:09 +02:00
leonarski_fandClaude Opus 4.8	d1f6b382a6	docs: rc.160 changelog (trim + scrub) and CPU_DATA_ANALYSIS updates ... CHANGELOG: add the Wilson B, decay-CV and space-group b-veto entries; shorten the verbose modulation / stills-modulation / trimmed-background entries and remove the dataset identities from them. CPU_DATA_ANALYSIS: the correction surfaces are now three (add modulation), on by default, cross-validated on the sigma-independent metric; the background trimmed mean applies to monochromatic stills as well as rotation (broadband keeps the sigma-clip); document the dataset-wide Wilson B, the still-partiality model, and the twin b-veto in the space-group search. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 09:25:39 +02:00
leonarski_fandClaude Opus 4.8	7d1fb8db77	bragg integration: clarify that the trimmed background applies to stills too ... The background trimmed mean is gated on the beam (monochromatic vs broadband), not on the acquisition mode: a monochromatic still (zero bandwidth) already gets the trim by default, only broadband (non-zero bandwidth: pink-beam / DMM) data keep the high-side sigma-clip. The comments, the settings doc and the --background-trim help wrongly implied the trim was rotation-only / that stills keep the sigma-clip. Wording only; no behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 09:23:13 +02:00
leonarski_fandClaude Opus 4.8	7a9d25850c	rugnux: veto a b-ballooning merohedral-twin over-promotion (space-group search) ... The systematic-b test (a genuine symmetry step barely moves the merge error model's b; a merohedral twin, forcing non-equivalent reflections together, balloons it) was only ever a RESCUE - it could promote a chi^2-borderline genuine step but never demote a chi^2-passing one. So a partial twin whose within-orbit scatter looks self-consistent (merged chi^2 below the ratio bound) slipped through on chi^2 alone and was over-promoted to the holohedral group, ballooning b and collapsing the merged ISa. Make the same test a VETO: a chi^2-passing high-symmetry promotion whose b, relative to the largest confirmed subgroup, exceeds max_systematic_b_veto is kept in its true lower symmetry. Calibrated on the rotation-test battery, where the largest genuine step (a P422 tetragonal) sits at b-ratio ~1.8 and a merohedral R3->R32 twin at ~2.6 - an empty gap - so 2.0 catches the twin while never demoting a genuine high-symmetry merge. Battery space-group match 20->21/25 (an R3 case previously merged as R32 is now correct), every other crystal unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 09:23:13 +02:00
leonarski_fandClaude Opus 4.8	e9a913f84b	scale_merge: remove sample identities from SearchSpaceGroup comments ... The point-group self-consistency comments named specific datasets when illustrating the chi^2-ratio / systematic-b calibration. Describe the crystallographic situation instead (a merohedral R3->R32 twin, a genuine cubic step, a tetragonal P41212 case), per the repository's no-sample-names rule. Comment-only; no behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	3bfce24faa	rugnux: score the decay cross-validation on the sigma-independent metric ... RefineDecay's cross-validation still scored the held-out equivalents on a studentized chi^2, while the absorption/modulation surfaces were switched to a sigma-independent Rmeas-like agreement (sum|Is-Iref|/sum|Iref|) precisely because a correction can "pass" a studentized gate by reshaping sigma without tightening the intensities. The decay slope scales sigma up on late / high-angle frames, so it is exposed to the same loophole. Score it the same way. On the rotation battery this rejects a spurious slope on a mis-indexed monoclinic (P2_1) case: the studentized gate had accepted a physically absurd -74 A^2 "decay" that cross-validated yet collapsed the error model (ISa 7.1 -> 1.1) and CC1/2 (to ~2%); the fractional gate scores it a -50% held-out loss and skips it. A small genuine ~3 A^2 decay on another crystal now cross-validates and engages, with CC1/2 unchanged. Every other crystal is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	8bd0977833	rugnux_vs_xds.py: drop the removed --scaling-output flag ... rugnux always writes both MTZ and CIF now (the --scaling-output option was removed), so passing it made every crystal in the battery fail against the current binary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	85cf826d2e	rugnux: dataset Wilson B-factor estimate + robust per-image Wilson B ... Add a dataset-wide isotropic Wilson B-factor estimate, the analogue of XDS's "WILSON LINE ... B=" which we did not export. CalcGlobalWilsonB fits ln<I> vs 1/d^2 over the merged reflections (B = -2*slope), skipping the low-resolution non-linear region (d > 4 A) and shells past the signal limit (<I/sigma> < 1) so the estimate is insensitive to how far the merged data were carried. It is diagnostic only - not fed back into scaling - and is written to the mmCIF (_reflns.B_iso_Wilson_estimate), the printed merge statistics, and the log. Also harden the per-image Wilson B (CalcWilsonBFactor): accept the fit only when it is well-correlated and physically plausible (0 < B < 200 A^2), else leave b_factor unset. A bad frame (an indexing glitch, too few reflections) otherwise produced a wildly steep Wilson line and a B of several hundred A^2 that polluted the per-image plot; NaN is preferable to garbage. Diagnostic-only: the merged intensities and every merge statistic are byte-identical (verified baseline vs modified on the rotation battery). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	3b168a1a46	viewer: expose detector-plane modulation and background trim in the settings dock ... Two new controls in the interactive settings dock: - Scaling section: a "Detector-plane modulation (stills)" checkbox that toggles the stills flat-field modulation surface (ScalingSettings::StillsModulation), alongside the existing decay/absorption "Correction surfaces" checkbox (rotation modulation is fitted there). Off by default. - Bragg integration section: a "Background trim" checkbox + fraction that sets the symmetric trimmed-mean background fraction (BraggIntegrationSettings::BackgroundTrimFraction; 0 = plain ring mean). Defaults on at 0.10, matching the integrator default, so the viewer requests trimming for every dataset (the integration engine still applies its own per-mode gating). Viewer-only change; compiles against the current settings API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 07:48:10 +02:00
leonarski_fandClaude Opus 4.8	74d7589abe	stills: optional detector-plane modulation (flat-field) correction ... The stills scaling/merging path (ScaleOnTheFly + MergeOnTheFly) had no correction surfaces, whereas rotation fits decay/absorption/modulation inside RotationScaleMerge. Serial stills hammer the same detector regions shot after shot, so a detector-plane (flat-field) systematic is at least as well determined there as in rotation - it just had no home. Add MergeOnTheFly::RefineModulation: a 16x16 multiplicative surface over the predicted detector position, fit against the merged reference and folded into each reflection's image_scale_corr before the error model and the merge. It mirrors RotationScaleMerge::ApplyCellSurface - alternating per-cell fit with a den-weighted geometric-mean gauge (never drifts the overall scale) and Tikhonov pull to 1 - and is cross-validated the same way: fit the surface on even images, score the held-out odd equivalents by a sigma-independent R-meas-like agreement (a fractional metric a sigma-reshaping surface cannot game), and apply the full-data surface only if the held-out gain clears a margin. A no-op when the systematic is absent or the data too sparse (< 8 obs per cell), so it is safe to leave off by default and opt in. Wired through ScalingSettings::StillsModulation and rugnux --stills-modulation (default off), in both the full-analysis and --scale paths. On a 17 MP JUNGFRAU serial set the surface cross-validates with a large held-out gain, confirming a real detector-plane systematic; whether it improves the merged data quality is under evaluation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 20:45:44 +02:00
leonarski_fandClaude Opus 4.8	c3e877d5ec	bragg integration: trimmed-mean background, on by default for rotation ... The local Bragg background is the mean over the r2..r3 ring. That mean reads high because the contaminants that survive the signal-disk mask - neighbour- spot wings, tails, zingers - are one-sided (positive), so it over-subtracts. Since a weak intensity is a small difference of large numbers (I = S - nS*b), a per-pixel background bias is fractionally largest at the resolution edge, exactly where it hurts most. Replace the ring mean with a symmetric trimmed mean (sort the ring, drop the lowest and highest fraction f, average the rest), controlled by a new BraggIntegrationSettings field and the rugnux `--background-trim <f>` option (default f=0.10; 0 restores the plain mean). Default on for monochromatic (rotation) data; broadband (stills) keep their tuned high-side sigma-clip, so the base engine forces the trim to 0 there. Implemented in both the CPU engine and the GPU kernel (shared-memory bitonic sort per block, flat-mean fallback above BKG_TRIM_MAX ring pixels); the two agree. 25-crystal rotation battery (fixed SG/cell): <I/sigma> improved on every crystal (median +50%), ISa on 20/22, resolution-edge R_meas fell several-fold (e.g. lyso_ref 1.0 A 108%->43%). Last-shell CC1/2 is rescued where the plain mean had collapsed to noise (Thau_9 at ~2.0 A 3.8%->64%, ~0.5 A of resolution regained; cytC_10 0.2%->10%) at a small cost (1-3%) in already-clean shells - it flattens the CC1/2 fall-off rather than shifting it. Stills unchanged. Documented in CPU_DATA_ANALYSIS.md section 9.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 18:32:57 +02:00
leonarski_fandClaude Opus 4.8	118f0833dc	common: remove dead Reflection.completeness field ... Reflection.completeness (spot-footprint fraction) was only ever set to its 1.0f default and never read anywhere - not in scaling/merge, CBOR, HDF5, or the viewer. The box-sum integrator already rejects any partial-footprint reflection outright (n_inner_valid == n_inner), so the intended "down-weight clipped spots" role is already covered more strictly. Removed to reduce confusion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 18:32:40 +02:00
leonarski_fandClaude Opus 4.8	818ec8fcbf	rugnux: add detector-plane modulation (flat-field) correction surface ... Rotation scaling gained a third cross-validated correction surface alongside decay and absorption: a smooth multiplicative factor over the predicted detector position (px, py) where each full lands. It absorbs detector-response and geometric flat-field systematics that vary across the detector plane and inflate R-meas; symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, over-determining the surface. Because it lives in the detector frame (not the rotation), the same correction concept applies to stills. The surface fit/cross-validate/apply machinery shared with the absorption surface is factored into ApplyCellSurface; RefineAbsorption and the new RefineModulation just build their per-full cell assignment (goniometer-frame vs detector-frame) and call it. The cross-validation now scores a sigma-INDEPENDENT, R-meas-like fractional agreement of the held-out equivalents rather than a studentized chi^2, so a surface can no longer pass CV by reshaping sigma without tightening the intensities - this removes an over-fit regression on mis-indexed data and hardens the absorption surface too. On the /data/rotation_test rotation battery (A/B, modulation off vs on): 17 of 22 processed crystals improve R-meas, 0 regress, e.g. lysoC 23.2->16.3%, lyso_2 47.4->28.5%, EcwtAL500 53.2->27.6%, EP_cs_01-17 CC1/2 60->93%; CC1/2 held or improved everywhere and the anomalous S signal preserved/improved (lyso_ref SD_MET 4.37->4.46, lysoC 3.07->3.25). On by default with the other surfaces. Also: --dump-observations now writes the per-full centroid frame and predicted px/py columns, so the dumped combined fulls are a complete unmerged export. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 13:24:10 +02:00
leonarski_fandClaude Opus 4.8	eae7ea90dc	rugnux: always write both MTZ and CIF; drop --scaling-output ... The merged reflections were written in one of mtz/cif/txt selected by --scaling-output. Write both an MTZ and an mmCIF unconditionally instead - each has its uses downstream (MTZ for the CCP4/phenix tools, mmCIF for deposition) - and remove the format selector, the plain-text .hkl writer, and the now-unused IntensityFormat enum / ScalingSettings::FileFormat plumbing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:45:08 +02:00
leonarski_fandClaude Opus 4.8	ce12d69751	rugnux: write standard CCP4 anomalous MTZ (IMEAN + I(+)/I(-)) ... In anomalous mode the merge keeps the two Friedel mates as separate rows, which WriteMtzReflections emitted verbatim - two IMEAN rows per Bijvoet pair that downstream tools had to re-collapse. Pair the mates into one row per reflection with the standard CCP4 anomalous layout: IMEAN + I(+)/I(-) and the matching F/F(+)/F(-) amplitudes. A single HKLKeyGenerator(merge_friedel=false) yields both the shared ASU group key and which mate a row is (.plus); IMEAN/F are the inverse-variance Friedel mean, centrics/unpaired keep one hand with the other missing. aimless/mtz2sca/ANODE now read the file directly. The non-anomalous MTZ path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:37:56 +02:00
leonarski_fandClaude Opus 4.8	ce15900471	changelog: add the 1.0.0-rc.160 section ... VERSION was bumped to rc.160 but the new items sat under the rc.159 heading, which released (origin/main) does not contain. Add a proper rc.160 section, move the three misfiled items (supercell fix, -S symbol, FFTW-on-GPU error) into it, and document the rest of the branch (stills geometry refinement, -r flex, stills partiality, r=6 box default, single-pass self-referenced scaling, reference-MTZ-never-a-scale-anchor, E^2 second-lattice cap, --scale on _process.h5 fix, new spot options, CBF/TIFF writer removal, negative total_flux, viewer, packaging). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:20:14 +02:00
leonarski_fandClaude Opus 4.8	0e4f91b219	rugnux: validate offline spot-finding settings and bound numeric options ... The offline rugnux path never ran CheckDataProcessingSettings (only the online broker/receiver paths did), so the new --min-pix-per-spot and --spot-low-resolution knobs were unbounded: --min-pix-per-spot 0 silently disabled the per-spot filter, and a low-resolution limit finer than the high-resolution limit made spot finding reject every pixel with no diagnostic. Add a parse_number_arg<T> helper (integral or floating, with optional inclusive bounds) that rejects non-numeric input, trailing garbage, and out-of-range values, replacing the raw atoi/atof in the spot-finding options; parse_double_arg/parse_float_arg become thin wrappers over it. After assembling the settings, validate them with the same CheckDataProcessingSettings the online receivers use, which also enforces the cross-field low>=high-resolution constraint. Also scrubs a residual sample-name comment (the r=6 box-default note) in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:19:59 +02:00
leonarski_fandClaude Opus 4.8	8fea7f195a	docs: stop advertising the removed CBF/TIFF writers ... The CBF/TIFF writer removal left two docs describing them as available output: JFJOCH_WRITER.md's "Other formats" section and an aside in HDF5.md. A user following them would issue a now-rejected request. State that only NXmx HDF5 is written and that the CBF/TIFF enum values are retained for wire back-compatibility but rejected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:19:46 +02:00
leonarski_fandClaude Opus 4.8	d393ec814c	Remove residual sample identities missed by the earlier scrub ... The rc.160 sample-identity cleanup left protein names, internal dataset codes, and a measured cell angle in several source comments and one test. Most sensitive: a measured monoclinic beta tied to an internal code in Rugnux.cpp and IndexAndRefine.cpp. Rewrite each to describe the crystallographic situation only (space group / metric relation), keeping the technical reasoning intact. Comment- and string-only; no behaviour change. (rugnux_cli.cpp's remaining name is scrubbed in the following commit, alongside its other change to that file.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:18:32 +02:00
leonarski_f	68272a96e1	VERSION: 1.0.0-rc.160	2026-07-15 21:41:51 +02:00
leonarski_fandClaude Opus 4.8	b450140816	rugnux: warn when the cell/space group has an indexing ambiguity ... At the twinning-analysis print, detect alternative-indexing (twin-law) operators for the final cell + space group via ReindexAmbiguityOperators(). When any exist and no reference was supplied, emit a Warning (also written into the merge stats) that serial-stills crystals are indexed in a random hand and the ambiguity can only be broken against a reference (-z / --model). For an obvious merohedral case (P3/P4/P6...) users expect this, but a PSEUDO-merohedral metric is easy to miss -- e.g. NmHR / 7O8F, where C2 beta=131.8 is pseudo-F-orthorhombic and gemmi reports the reindex operator h-l,-k,-l. Advisory only; no change to processing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 20:38:31 +02:00
leonarski_fandClaude Opus 4.8	3e3545770f	indexing: check candidate angles against the reference cell in Refine() ... When a reference cell is supplied, PostIndexingRefinement::Refine() vetted candidates on sorted edge lengths only; a cell with the right edges but a wrong angle (a pseudo-symmetric near-metric, e.g. a monoclinic beta refined to the wrong value) passed. Add a sorted-angle check against the reference, folding each angle to its acute complement min(x,180-x) so the obtuse/acute setting choice is irrelevant (tolerance 10 deg). Reference-cell path only (no effect de novo, where reference_unit_cell is null): the /data/rotation_test battery is byte-identical (22/25), and NmHR with the correct C2 cell is unchanged (9.1%, beta~131.7). A latent guard against wrong-angle cells slipping through on -C / reference-MTZ runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:47:29 +02:00
leonarski_fandClaude Opus 4.8	0f07b7a499	rotation: widen candidate-refine angle bound to [30,150] deg ... RotationIndexer's per-candidate XtalOptimizer relied on the struct-default monoclinic angle bound [60,120], the same latent clamp fixed for the stills path: a rotation crystal with beta>120 (or <60) would have its cell clamped to the boundary. Set [30,150] explicitly, matching the indexers. No change on the /data/rotation_test battery (22/25 SG match, byte-identical) -- no battery crystal has beta outside [60,120]; this is a latent-correctness fix for future high-beta monoclinic rotation data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:47:17 +02:00