192 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	c5cfdcef7e	rugnux: per-batch relative-B correction + radiation-damage B monitor ... Rotation scaling modelled dose-dependent scattering power with a single global relative-B decay slope only (RefineDecay); per-frame refinement was scale-only. Add a per-batch relative Debye-Waller B, both as an opt-in correction and as an always-on measurement. - FitRelativeBCurve / SolveCurvatureSmoothedB: per rotation-range batch, fit the s^2 slope of ln(Iref/Iobs) that the resolution-flat per-frame G leaves, regularised by a second-difference (curvature) penalty so a genuine smooth relative-B is kept while resolution-correlated noise (which an unconstrained per-batch fit would chase into an oscillating curve) is suppressed. - Correction (--relative-b[=deg], default off): applied after RefineDecay, gated by a physical peak-to-peak floor and cross-validated by ASU-group parity (a per-batch parameter owns whole frames, so it is scored on held-out equivalents, not held-out frames). On a battery re-scaling identical stored reflections it is a no-op on well-behaved data and a small gain on radiation-damaged rotation data; never a regression. - Radiation-damage monitor (always on for rotation, report-only): measure the relative-B change from the pristine low-dose start of the run to its end (a whole-run-mean reference inverts on damaged data) plus the per-batch curve, and report them on the command line (in the radiation-damage report and the merge statistics) and in the mmCIF (_reflns.jfjoch_radiation_damage_ relative_B and a per-batch loop). A large magnitude flags radiation damage, complementing the existing per-image CC / mosaicity read. - --scale now reproduces the de-novo rotation merge (it previously omitted scale-fulls and smooth-G, silently using a weaker model), and honours --relative-b, so an offline re-scale matches the full pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 22:08:09 +02:00
leonarski_fandClaude Opus 4.8	290c30b9be	rugnux: post-refine commits detector/cell geometry only for a small (< 1 %) move ... The rotation geometry post-refine committed the detector distance and cell scale on any cross-validated improvement within a loose 3 % / 2 % bound. On a multi-lattice crystal the second lattice's spots bias every cross-validation fold the same way, so the "it improved" test cannot catch it and a spurious distance<->cell pair (their radial degeneracy) is committed far off the true geometry, losing reflections in the re-integration. Tighten the commit bound to < 1 % (a calibrated header needs < ~0.6 %). The move SIZE discriminates a genuine header correction from the multi-lattice failure far better than the fit residual, which genuinely marginal (noisy or ice-ring) data shares with the multi-lattice case - so a real ~0.3 % correction is kept while the ~1.7 % multi-lattice one is rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 19:41:12 +02:00
leonarski_fandClaude Opus 4.8	c140b3567d	rugnux: restrain the gauge-weak beam centre toward the header in geometry refinement ... Single-axis rotation has a gauge degeneracy: rotating the whole experiment about the spindle leaves every spot position unchanged, so the beam-centre component parallel to the spindle is a null / under-determined direction. Refining it freely lets it wander a few pixels and absorb centroid systematics into a wrong beam that the co-refined orientation keeps position-consistent. Add a soft header prior on that one component in the shared XtalOptimizer: residual = w*(beam[parallel] - header), with w set so the restraint behaves like a sigma_px-pixel prior competing with the positional residuals. The gauge direction has ~zero data sensitivity, so the prior pins it near the LaB6-monitored header, while a real, well-supported drift can still overcome it (the beam does drift). One restraint covers all three refinement sites at once: the noisy accumulated-spot primary, the per-frame filter, and post-refine. Validated against independent XDS processing: neutral on well-determined data (the gauge direction has no effect there), and a real low-resolution accuracy gain where the free refinement had otherwise wandered off the header - the merged intensities move back into agreement with XDS while the internal CC1/2, which is blind to this, barely changes. No space-group determinations change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 17:31:45 +02:00
leonarski_fandClaude Opus 4.8	c9ed9c3776	rugnux: two-pass reuses pass-1 space group; floor the fed mosaicity ... The second pass re-searching the space group at the slightly changed geometry could flip a borderline determination (cubic <-> orthorhombic) and mix the two passes, collapsing the multiplicity. Fix it: the pre-pass now continues through the space-group search, so the determined group is already fixed on experiment_ and the second pass reuses it instead of re-searching. The lattice is NOT forced - the second pass re-indexes at the refined geometry so the cell stays self-consistent with it (forcing the pre-pass lattice would pin the nominal cell and undo the refinement). A reindex-derived group (the centred-lattice test) is in a conventional setting the indexer's primitive frame does not share, so it is cleared for the second pass to re-search (else its indexing fails outright). Also floor the mosaicity fed to the second pass's prediction at each frame's own estimate (max), so a too-narrow smoothed value can never make prediction drop reflections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 13:04:04 +02:00
leonarski_fandClaude Opus 4.8	1b91823eed	rugnux: --rotation-post-refine two-pass geometry post-refinement (opt-in) ... A first pass integrates and post-refines the detector geometry from the observed diffraction, applies it to experiment_, and a second pass re-indexes and re-integrates with it - only the second pass is written, so the _process.h5 / mtz (and the ice-ring flags, which move with the geometry) reflect the refined values. Default off; enable with --rotation-post-refine. PostRefineRotationGeometry refines in two SEPARATE cross-validated steps (a joint fit of everything is fragile - the same lesson as integration, where refining the profile width and the scale jointly fails but separately works): * Step A: cell scale + rotation axis from the observed rocking centroids phi_obs (a distance-independent excitation residual, so it pins the absolute cell scale that the positional residual leaves degenerate with distance). * Step B: detector distance + beam centre from the observed spot positions, with the cell fixed at step A. Each step commits only if it lowers a held-out (deterministic split-half) residual, else that part of the geometry is left at nominal. Rugnux::Run wraps RunPipeline(write_output, geometry_prepass); the pre-pass runs the first scale/merge so RotationScaleMerge fits a frame-order-smoothed mosaicity, which is captured and fed to the second pass's Bragg prediction (via IndexAndRefine::SetPredictionMosaicityOverride) rather than re-derived per image. observed_x/y were plumbed through the profile integrator earlier for the positional residual. rugnux_vs_xds.py gains --extra-args to sweep the option over the battery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 12:19:33 +02:00
leonarski_fandClaude Opus 4.8	f396385e11	rugnux: record the detector distance and beam centre in the merged mmCIF ... Add _diffrn_detector.jfjoch_{distance_mm,beam_center_x_pxl,beam_center_y_pxl} to the merged mmCIF header, as feedback of the geometry the data was actually reduced with. When geometry refinement ran (the rotation two-pass or the stills global refinement, which update experiment_ before the written pass) these are the refined values; otherwise they are the nominal header geometry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 12:19:16 +02:00
leonarski_fandClaude Opus 4.8	9bfecaadf5	beam_stop: add ShadowFinder detection class (unwired) ... Add image_analysis/beam_stop/ShadowFinder.{h,cpp} and SHADOW_FINDER.md: a self-contained beam-stop shadow detector that accumulates images via AddImage() and returns a mask from GetMask(), mirroring the shape of DarkMaskAnalysis. It detects the beam-stop shadow (central disk + holder arm) as an azimuthal anomaly: an iterated radial-median background baseline, a ratio threshold, a connectivity-to-beam-centre anchor with module-gap bridging, a central low-res disk guard capped just inside the innermost reflection, and a reflection guard that never masks a pixel that recorded real signal. Not yet wired: not added to image_analysis/CMakeLists.txt and no PixelMask / Rugnux / viewer changes. SHADOW_FINDER.md documents the algorithm and the deferred offline (Rugnux bit 9 + viewer user-mask) integration plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 12:05:05 +02:00
leonarski_fandClaude Opus 4.8	1911875e1c	integration: carry the box-sum observed centroid through the profile path ... Pass A (the box-sum) always computes an intensity-weighted centroid (the observed spot position), but it was only emitted in BoxSum mode. Rotation integration runs the profile path, so observed_x/y were left NAN there and no positional (detector<->reciprocal) residual was possible downstream. Emit the Pass-A centroid in the profile path too (CPU) and copy d_obs_x/d_obs_y back in all modes (GPU) - Pass A fills them regardless of integrator mode. observed_x/y are consumed only by the diagnostic HDF5 reflection table and the viewer read-back; nothing in scaling/merge reads them, so merged output is unchanged - the _process.h5 observed_x/y columns simply go from NAN to the real centroid. This makes the observed position available to post-refinement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 10:23:17 +02:00
leonarski_fandClaude Opus 4.8	7ceffde5a4	rugnux: rank space groups by net absences, not gross (fix C222 over-centering) ... Stage B of SearchSpaceGroup chose the largest centering that explained the most absences and was self-consistent, ranking candidates by the gross observed-absent count. A false super-centering over-claims: on a C222 crystal, F222 predicts every C absence (all genuinely weak) PLUS a block of C-present reflections it wrongly calls absent. Its absent class is then bimodal - many true zeros diluting a strong block - so its mean <I/sigma>abs (and the violation-rate arm) slip under the centering strength gate, which cannot see the over-claim in isolation. With the gross-count ranking F222 then beat the true C222, and the strength gate is scale-invariant here, so real C-centered data trips it identically (the rotation battery just has no C-centered crystal on this path; the synthetic unit test does). Rank instead by the absences GENUINELY explained, absent_observed - absent_violations, then by fewer violations, then lower space-group number. C222 and F222 net to the same count (F222's genuine absences ARE C222's), so the fewer-violations tie-break keeps the honest, less-centred C222. Symmetric: on a real F222 crystal F explains strictly more weak absences and still wins. Also tighten the "indistinguishable alternatives" grouping to require the same absent AND violation counts, so a super-centering is never reported as an equal alternative. No tuned threshold is changed; the ranking is a no-op for any 0-violation winner. Fixes the SearchSpaceGroup C222 synthetic-space-group test. Rotation battery (32 crystals, de-novo) is space-group-identical to before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 22:35:12 +02:00
leonarski_fandClaude Opus 4.8	742a1c1b94	rugnux: identify the producing software in the merged mmCIF and MTZ ... mmCIF: set _software.name to 'Rugnux' (was 'Jungfraujoch') and correct _software.classification to the CIF-dictionary value 'data reduction'. MTZ: add the producing software to the header via the title and a HISTORY record ("From Rugnux <version>, data reduction"), the CCP4 provenance channel, which was previously empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 21:29:47 +02:00
leonarski_fandClaude Opus 4.8	59e260876a	rugnux: recover the asymptotic ISa on weak / radiation-damaged data ... The asymptote was read only from groups with I/sigma > 20 and required at least 100 of them, so weak or radiation-damaged data (few strong reflections) fell back to the higher whole-range b and reported an over-conservative ISa. Relax the I/sigma threshold in a second tier (>= 50 groups above I/sigma 10) when the tight one is unmet; the per-group counting-subtracted estimate is stable down to that threshold. Data with many strong groups is unchanged (it keeps the tight-threshold asymptote); weak / damaged data now reports the asymptote its own reflections support instead of falling back. Battery (32 crystals): no space-group, R_meas or CC1/2 change; strong sets unchanged; e.g. a room-temperature radiation-damage series recovers ISa 5.4 -> 11.2 (its data already exceeds the reference on CC1/2 and R_meas). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	88359a57ee	rugnux: report ISa at its asymptotic definition (strong-reflection limit) ... ISa is the I->infinity limit of the signal-to-noise (Diederichs, Acta Cryst. D66 (2010), 733-740), i.e. the reproducibility of the strongest reflections. The reported value was taken as 1/b of the whole-intensity-range error-model fit, whose systematic term b is raised by a mild excess of scatter at intermediate intensity, so it understated that limit. Read the asymptote directly instead: for each well-measured reflection group the counting-subtracted fractional scatter of its symmetry mates estimates the systematic term, and the robust median over strong groups is the asymptote. It falls back to the whole-range b when too few strong groups exist (weak or low-multiplicity data). The reported ISa and the merged-sigma systematic floor use this value, so merged I/sigma approaches ISa. The (a, b) fit and the per-observation error model are unchanged, so the merge weights, anomalous statistics and chi^2 are untouched, and CC1/2 and R_meas (which do not depend on sigma) are bit-identical. Battery (32 crystals): no space-group or indexing change, R_meas and CC1/2 unchanged, ISa rises to the true asymptote of the data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	7fe12773de	rugnux: report anomalous signal-to-noise SigAno (stats table + mmCIF) ... Compute SigAno = <|I(+)-I(-)|>/<sigma(I(+)-I(-))> per report shell and overall from the full-multiplicity I(+)/I(-) Bijvoet split, store it on MergeStatisticsShell, and surface it in the console merge-statistics table (new SigAno column) and the mmCIF via the standard wwPDB PDBx items _reflns.pdbx_absDiff_over_sigma_anomalous (overall) and _reflns_shell.pdbx_absDiff_over_sigma_anomalous (per shell) - the dictionary's home for this quantity, so no jfjoch_ local item is needed (unlike ISa). SigAno approaches ~0.8 for pure noise and rises above 1 with real anomalous signal. A half-set anomalous CC is deliberately not used: its two half estimates are complementary partitions of one observation pool (dI0 + dI1 = 2*dI_full), so differencing the two Bijvoet hands cancels the large common intensity that keeps CC1/2 non-negative and, once the anomalous SNR per half drops below 1, drives the correlation towards -1 rather than 0 - misrepresenting a weak-but-real signal. Emitted only when an anomalous split was made (has_anom guard), so a non-anomalous merge keeps its stats block and shell-loop columns byte-for-byte. Export-neutral: the I(+)/I(-)/IMEAN accumulation is unchanged. On low-energy S-SAD test data SigAno tracks XDS (13 keV null ~0.89, 6 keV ~1.44, 5 keV ~1.5-1.7) and never goes spuriously negative. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	1b5eb80ba4	rugnux: recover I-centering on weak / low-energy data (centering rate gate) ... The de-novo centering gate promoted to a centred lattice only when the mean signed <I/sigma> of the centering-absent class was <= 0.5x that of the present class. Weak / low-energy data carry a positive intensity floor (background / profile leakage) that lifts <I/sigma> of the genuinely-extinct class to ~1.5-2.3 instead of ~0; when the present class is itself weak that additive floor inflates the ratio past the bound and the true centering is missed. An I-centred cubic crystal recorded at 5 and 13 keV sat at ratio ~0.57 and stayed primitive P2_1 3, while the same crystal form at 6 keV (stronger data) correctly merged as I23 - an energy-dependent space group for one physical crystal. Add a second, floor-independent acceptance path OR'd with the mean-ratio: the centering-absent class's strong-reflection rate (violations / absent) versus the present class's own strong rate. A false centering (absent as strong as present) fails both arms; a true weak centering passes the rate arm. Battery SG sweep vs XDS: the low-energy datasets now adopt I23, and no crystal is newly over-centred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	32c7f9f59f	rugnux: don't veto genuine high symmetry on excellent data (b-veto floor) ... The de-novo space-group search's systematic-b veto - which keeps a merohedral twin in its true lower symmetry - is a RATIO test: reject a point-group promotion whose merge error-model b balloons past 2x the parent subgroup's. That ratio is meaningless when the parent b is near zero, on excellent data where symmetry equivalents already agree almost perfectly. A genuine tetragonal 422 whose 222 parent sits at b=0.008 and 422 at b=0.049 (both tiny, ISa ~20) reads as a 6x balloon and is wrongly demoted to 222. Floor the parent b (min_systematic_b_for_veto = 0.05) before forming the ratio, so the veto only judges a promotion once its b is a non-negligible fraction of I. A real merohedral twin drives b to ~0.19 regardless of how clean the parent is, so it is still caught (the insulin R3 + ~20% twin veto still fires, b=0.19 > 0.15). A P41212 58/58/150 crystal at 16 keV goes P212121 -> P41212; full 26-crystal rotation battery: only that crystal changes (its space group now matches XDS, SG match 22->23/26), every other space group bit-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 20:54:03 +02:00
leonarski_fandClaude Opus 4.8	65bc40e7c7	merge: deltaCChalf uses the deterministic HalfForImage split, not an RNG ... MergeOnTheFly::DeltaCChalfReject assigned its CC1/2 half-sets from a seeded std::mt19937 drawn in image (call) order, whereas the actual merge, the rotation merge, the GPU path and the R-free flags all split with the deterministic HalfForImage(image_id) splitmix64 hash. So the deltaCChalf was measured on a different half-partition than the reported CC1/2, and was order-dependent (a reordered outcomes vector gave different halves). Use HalfForImage(i) - i is the image's stable index, the same image_id AddImage merges with - so deltaCChalf now reflects the exact CC1/2 split the statistics report, order-independently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 13:19:51 +02:00
leonarski_fandClaude Opus 4.8	b647cf454c	rugnux: write I(+)/I(-) and F(+)/F(-) by default (rotation) ... A rotation merge now keeps each acentric reflection's Bijvoet split - the inverse-variance I(+)/I(-) from the same scaled fulls - even when the merge is Friedel-averaged (the default, no -A). IMEAN stays the primary intensity and is bit-identical to before; the split is purely additive and scaled non-anomalously, so a weak anomalous signal is preserved in the output without having to reprocess with -A. French-Wilson now also fills F(+)/F(-) from the two hands (one pass, shared Wilson prior). Writers: the default (Friedel-merged) MTZ and mmCIF now carry IMEAN plus I(+)/I(-) and F/F(+)/F(-) whenever a reflection has an anomalous split (the stills path, which computes none, keeps the previous columns). A missing mate or a centric is emitted as the CCP4 missing-value flag (NaN). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 11:52:31 +02:00
leonarski_fandClaude Opus 4.8	67a18357b5	rugnux: fix de-novo space-group centering and weak-data high symmetry ... De-novo space-group search: - Determine lattice centering from the strength of the systematically-absent class (its mean I/sigma vs the present class) instead of a per-reflection violation count. The count gate was brittle on noisy / obverse-reverse- twinned data, where enough genuinely-absent reflections cross I/sigma>3 to trip the 10% bound although the class is several-fold weaker, and wrongly kept a genuine R-centered lattice primitive. False centerings, whose absent class is as strong as the present one, stay rejected. Screws keep the count test (too few axial reflections to average). - Recover a genuine high-symmetry point group when the merged error model is badly miscalibrated (weak, low-resolution data whose sigmas are far too small). The fixed-sigma chi^2 ratio then grows with point-group order for real symmetry too and wrongly rejects it (a true cubic group reached a chi^2 ratio ~14); the self-normalising systematic-b test stays valid, so once chi2_ref shows the ratio is untrustworthy the promotion is confirmed on the b-test alone. The balloon veto still rejects a twin. This is the gap left when the earlier log10(chi2_ref) widening was removed for the b-veto. Also print the adopted space group and unit cell as a clean one-line summary at the end of the run (de-novo or user-fixed -S), not only in the mmCIF. Full rotation-test battery: exactly the two intended crystals change (both to the correct answer, with lower R-meas and higher ISa), every other crystal unchanged; space-group match 21->22/25. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 10:46:09 +02:00
leonarski_fandClaude Opus 4.8	7d1fb8db77	bragg integration: clarify that the trimmed background applies to stills too ... The background trimmed mean is gated on the beam (monochromatic vs broadband), not on the acquisition mode: a monochromatic still (zero bandwidth) already gets the trim by default, only broadband (non-zero bandwidth: pink-beam / DMM) data keep the high-side sigma-clip. The comments, the settings doc and the --background-trim help wrongly implied the trim was rotation-only / that stills keep the sigma-clip. Wording only; no behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 09:23:13 +02:00
leonarski_fandClaude Opus 4.8	7a9d25850c	rugnux: veto a b-ballooning merohedral-twin over-promotion (space-group search) ... The systematic-b test (a genuine symmetry step barely moves the merge error model's b; a merohedral twin, forcing non-equivalent reflections together, balloons it) was only ever a RESCUE - it could promote a chi^2-borderline genuine step but never demote a chi^2-passing one. So a partial twin whose within-orbit scatter looks self-consistent (merged chi^2 below the ratio bound) slipped through on chi^2 alone and was over-promoted to the holohedral group, ballooning b and collapsing the merged ISa. Make the same test a VETO: a chi^2-passing high-symmetry promotion whose b, relative to the largest confirmed subgroup, exceeds max_systematic_b_veto is kept in its true lower symmetry. Calibrated on the rotation-test battery, where the largest genuine step (a P422 tetragonal) sits at b-ratio ~1.8 and a merohedral R3->R32 twin at ~2.6 - an empty gap - so 2.0 catches the twin while never demoting a genuine high-symmetry merge. Battery space-group match 20->21/25 (an R3 case previously merged as R32 is now correct), every other crystal unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 09:23:13 +02:00
leonarski_fandClaude Opus 4.8	e9a913f84b	scale_merge: remove sample identities from SearchSpaceGroup comments ... The point-group self-consistency comments named specific datasets when illustrating the chi^2-ratio / systematic-b calibration. Describe the crystallographic situation instead (a merohedral R3->R32 twin, a genuine cubic step, a tetragonal P41212 case), per the repository's no-sample-names rule. Comment-only; no behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	3bfce24faa	rugnux: score the decay cross-validation on the sigma-independent metric ... RefineDecay's cross-validation still scored the held-out equivalents on a studentized chi^2, while the absorption/modulation surfaces were switched to a sigma-independent Rmeas-like agreement (sum|Is-Iref|/sum|Iref|) precisely because a correction can "pass" a studentized gate by reshaping sigma without tightening the intensities. The decay slope scales sigma up on late / high-angle frames, so it is exposed to the same loophole. Score it the same way. On the rotation battery this rejects a spurious slope on a mis-indexed monoclinic (P2_1) case: the studentized gate had accepted a physically absurd -74 A^2 "decay" that cross-validated yet collapsed the error model (ISa 7.1 -> 1.1) and CC1/2 (to ~2%); the fractional gate scores it a -50% held-out loss and skips it. A small genuine ~3 A^2 decay on another crystal now cross-validates and engages, with CC1/2 unchanged. Every other crystal is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	85cf826d2e	rugnux: dataset Wilson B-factor estimate + robust per-image Wilson B ... Add a dataset-wide isotropic Wilson B-factor estimate, the analogue of XDS's "WILSON LINE ... B=" which we did not export. CalcGlobalWilsonB fits ln<I> vs 1/d^2 over the merged reflections (B = -2*slope), skipping the low-resolution non-linear region (d > 4 A) and shells past the signal limit (<I/sigma> < 1) so the estimate is insensitive to how far the merged data were carried. It is diagnostic only - not fed back into scaling - and is written to the mmCIF (_reflns.B_iso_Wilson_estimate), the printed merge statistics, and the log. Also harden the per-image Wilson B (CalcWilsonBFactor): accept the fit only when it is well-correlated and physically plausible (0 < B < 200 A^2), else leave b_factor unset. A bad frame (an indexing glitch, too few reflections) otherwise produced a wildly steep Wilson line and a B of several hundred A^2 that polluted the per-image plot; NaN is preferable to garbage. Diagnostic-only: the merged intensities and every merge statistic are byte-identical (verified baseline vs modified on the rotation battery). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	74d7589abe	stills: optional detector-plane modulation (flat-field) correction ... The stills scaling/merging path (ScaleOnTheFly + MergeOnTheFly) had no correction surfaces, whereas rotation fits decay/absorption/modulation inside RotationScaleMerge. Serial stills hammer the same detector regions shot after shot, so a detector-plane (flat-field) systematic is at least as well determined there as in rotation - it just had no home. Add MergeOnTheFly::RefineModulation: a 16x16 multiplicative surface over the predicted detector position, fit against the merged reference and folded into each reflection's image_scale_corr before the error model and the merge. It mirrors RotationScaleMerge::ApplyCellSurface - alternating per-cell fit with a den-weighted geometric-mean gauge (never drifts the overall scale) and Tikhonov pull to 1 - and is cross-validated the same way: fit the surface on even images, score the held-out odd equivalents by a sigma-independent R-meas-like agreement (a fractional metric a sigma-reshaping surface cannot game), and apply the full-data surface only if the held-out gain clears a margin. A no-op when the systematic is absent or the data too sparse (< 8 obs per cell), so it is safe to leave off by default and opt in. Wired through ScalingSettings::StillsModulation and rugnux --stills-modulation (default off), in both the full-analysis and --scale paths. On a 17 MP JUNGFRAU serial set the surface cross-validates with a large held-out gain, confirming a real detector-plane systematic; whether it improves the merged data quality is under evaluation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 20:45:44 +02:00
leonarski_fandClaude Opus 4.8	c3e877d5ec	bragg integration: trimmed-mean background, on by default for rotation ... The local Bragg background is the mean over the r2..r3 ring. That mean reads high because the contaminants that survive the signal-disk mask - neighbour- spot wings, tails, zingers - are one-sided (positive), so it over-subtracts. Since a weak intensity is a small difference of large numbers (I = S - nS*b), a per-pixel background bias is fractionally largest at the resolution edge, exactly where it hurts most. Replace the ring mean with a symmetric trimmed mean (sort the ring, drop the lowest and highest fraction f, average the rest), controlled by a new BraggIntegrationSettings field and the rugnux `--background-trim <f>` option (default f=0.10; 0 restores the plain mean). Default on for monochromatic (rotation) data; broadband (stills) keep their tuned high-side sigma-clip, so the base engine forces the trim to 0 there. Implemented in both the CPU engine and the GPU kernel (shared-memory bitonic sort per block, flat-mean fallback above BKG_TRIM_MAX ring pixels); the two agree. 25-crystal rotation battery (fixed SG/cell): <I/sigma> improved on every crystal (median +50%), ISa on 20/22, resolution-edge R_meas fell several-fold (e.g. lyso_ref 1.0 A 108%->43%). Last-shell CC1/2 is rescued where the plain mean had collapsed to noise (Thau_9 at ~2.0 A 3.8%->64%, ~0.5 A of resolution regained; cytC_10 0.2%->10%) at a small cost (1-3%) in already-clean shells - it flattens the CC1/2 fall-off rather than shifting it. Stills unchanged. Documented in CPU_DATA_ANALYSIS.md section 9.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 18:32:57 +02:00
leonarski_fandClaude Opus 4.8	818ec8fcbf	rugnux: add detector-plane modulation (flat-field) correction surface ... Rotation scaling gained a third cross-validated correction surface alongside decay and absorption: a smooth multiplicative factor over the predicted detector position (px, py) where each full lands. It absorbs detector-response and geometric flat-field systematics that vary across the detector plane and inflate R-meas; symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, over-determining the surface. Because it lives in the detector frame (not the rotation), the same correction concept applies to stills. The surface fit/cross-validate/apply machinery shared with the absorption surface is factored into ApplyCellSurface; RefineAbsorption and the new RefineModulation just build their per-full cell assignment (goniometer-frame vs detector-frame) and call it. The cross-validation now scores a sigma-INDEPENDENT, R-meas-like fractional agreement of the held-out equivalents rather than a studentized chi^2, so a surface can no longer pass CV by reshaping sigma without tightening the intensities - this removes an over-fit regression on mis-indexed data and hardens the absorption surface too. On the /data/rotation_test rotation battery (A/B, modulation off vs on): 17 of 22 processed crystals improve R-meas, 0 regress, e.g. lysoC 23.2->16.3%, lyso_2 47.4->28.5%, EcwtAL500 53.2->27.6%, EP_cs_01-17 CC1/2 60->93%; CC1/2 held or improved everywhere and the anomalous S signal preserved/improved (lyso_ref SD_MET 4.37->4.46, lysoC 3.07->3.25). On by default with the other surfaces. Also: --dump-observations now writes the per-full centroid frame and predicted px/py columns, so the dumped combined fulls are a complete unmerged export. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 13:24:10 +02:00
leonarski_fandClaude Opus 4.8	eae7ea90dc	rugnux: always write both MTZ and CIF; drop --scaling-output ... The merged reflections were written in one of mtz/cif/txt selected by --scaling-output. Write both an MTZ and an mmCIF unconditionally instead - each has its uses downstream (MTZ for the CCP4/phenix tools, mmCIF for deposition) - and remove the format selector, the plain-text .hkl writer, and the now-unused IntensityFormat enum / ScalingSettings::FileFormat plumbing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:45:08 +02:00
leonarski_fandClaude Opus 4.8	ce12d69751	rugnux: write standard CCP4 anomalous MTZ (IMEAN + I(+)/I(-)) ... In anomalous mode the merge keeps the two Friedel mates as separate rows, which WriteMtzReflections emitted verbatim - two IMEAN rows per Bijvoet pair that downstream tools had to re-collapse. Pair the mates into one row per reflection with the standard CCP4 anomalous layout: IMEAN + I(+)/I(-) and the matching F/F(+)/F(-) amplitudes. A single HKLKeyGenerator(merge_friedel=false) yields both the shared ASU group key and which mate a row is (.plus); IMEAN/F are the inverse-variance Friedel mean, centrics/unpaired keep one hand with the other missing. aimless/mtz2sca/ANODE now read the file directly. The non-anomalous MTZ path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:37:56 +02:00
leonarski_fandClaude Opus 4.8	d393ec814c	Remove residual sample identities missed by the earlier scrub ... The rc.160 sample-identity cleanup left protein names, internal dataset codes, and a measured cell angle in several source comments and one test. Most sensitive: a measured monoclinic beta tied to an internal code in Rugnux.cpp and IndexAndRefine.cpp. Rewrite each to describe the crystallographic situation only (space group / metric relation), keeping the technical reasoning intact. Comment- and string-only; no behaviour change. (rugnux_cli.cpp's remaining name is scrubbed in the following commit, alongside its other change to that file.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:18:32 +02:00
leonarski_fandClaude Opus 4.8	3e3545770f	indexing: check candidate angles against the reference cell in Refine() ... When a reference cell is supplied, PostIndexingRefinement::Refine() vetted candidates on sorted edge lengths only; a cell with the right edges but a wrong angle (a pseudo-symmetric near-metric, e.g. a monoclinic beta refined to the wrong value) passed. Add a sorted-angle check against the reference, folding each angle to its acute complement min(x,180-x) so the obtuse/acute setting choice is irrelevant (tolerance 10 deg). Reference-cell path only (no effect de novo, where reference_unit_cell is null): the /data/rotation_test battery is byte-identical (22/25), and NmHR with the correct C2 cell is unchanged (9.1%, beta~131.7). A latent guard against wrong-angle cells slipping through on -C / reference-MTZ runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:47:29 +02:00
leonarski_fandClaude Opus 4.8	0f07b7a499	rotation: widen candidate-refine angle bound to [30,150] deg ... RotationIndexer's per-candidate XtalOptimizer relied on the struct-default monoclinic angle bound [60,120], the same latent clamp fixed for the stills path: a rotation crystal with beta>120 (or <60) would have its cell clamped to the boundary. Set [30,150] explicitly, matching the indexers. No change on the /data/rotation_test battery (22/25 SG match, byte-identical) -- no battery crystal has beta outside [60,120]; this is a latent-correctness fix for future high-beta monoclinic rotation data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:47:17 +02:00
leonarski_fandClaude Opus 4.8	643bda3dd3	stills: widen per-frame cell-refine angle bound to [30,150] deg ... The stills per-frame XtalOptimizer in IndexAndRefine used the struct-default monoclinic angle bound [60,120] deg, which clamps a beta outside that window (e.g. NmHR / 7O8F, C2 beta=131.78) to the boundary and corrupts the per-frame cell. Both indexers already use [30,150] (FFBIDXIndexer, FFT settings); this aligns the per-frame refine with them. Rotation path is unaffected (refine_unit_cell is false there). Verified on NmHR serial stills: FFBIDX with the correct C2 cell went from 0.01% to 9.10% indexed, recovering the true mC cell (beta~131.7) instead of a clamped beta=120; 60k-image merge reaches 2.61 A at 99.8% completeness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:13:41 +02:00
leonarski_fandClaude Opus 4.8	d8e909a393	indexing: clear error when the requested indexer is unavailable on this host ... IndexerThreadPool built its per-worker indexers from the raw requested algorithm but dispatched the RESOLVED one. Rotation indexing always resolves to the GPU FFT indexer when a GPU is present, so requesting FFTW (the CPU indexer) on a GPU node created only the CPU indexer and then failed at dispatch with an opaque "no indexer available for the resolved indexing algorithm" deep inside a worker. Validate servability at the pool entry (in the caller's context) and throw an actionable message instead: FFTW is not available on a GPU node (use FFT/Auto, or a CPU-only node); FFT/FFBIDX require a GPU. Stills FFTW on a GPU node, which the algorithm resolution still honours, keeps working. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:19:43 +02:00
leonarski_fandClaude Opus 4.8	abbee2d4dc	Remove sample identities from the repository; document the rule ... Datasets may be confidential; sample names and measured unit cells committed to the repo can leak outside the group working on them. Scrub existing occurrences and add a "No sample identities in the repository" section to CLAUDE.md (forbidden: sample/dataset names, internal codes, measured cells tied to a sample; fine: space group / lattice / twinning descriptors). - Comments: replace internal dataset codes and protein names with the crystallographic situation they illustrate (centred vs pseudo-symmetric, holohedral, cubic, F-cubic/hexagonal, ...). - Docs: same, in the analysis/writer/stream references and example configs. - Tests: rename sample-named identifiers, TEST_CASE names, file prefixes and asserted labels to neutral crystallographic names (e.g. tetragonal_uc); behaviour unchanged. Reduce the CrystFEL reference PDB to a bare CRYST1 cell file (cell.pdb) and rename the reference data file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:19:43 +02:00
leonarski_fandClaude Opus 4.8	5d1ddccee8	viewer: expose stills processing settings; rename geom refinement multi -> flex ... Wire the recently-added stills knobs into jfjoch_viewer - they were reachable only from the rugnux CLI, and the viewer had silently diverged from the CLI once --refine-geometry became default-on for stills-with-cell. - Stills geometry-refinement two-pass (--refine-geometry): a checkbox + frame count in the reprocessing-job dialog, offered (and defaulted on, matching the CLI) only for a stills-with-cell run; wired into ProcessConfig and the "Copy command" generator (joined =N, or =off to reproduce opting out of the CLI default-on). - Stills partiality (--still-partiality) checkbox in Bragg integration and the partiality-uncertainty merge term (--partiality-uncertainty) in Scaling, both mirrored in the command-line generator. Rename the "best per-image refinement" mode from "multi" to "flex" everywhere: GeomRefinementAlgorithmEnum::Flex, CLI -r flex (with "multi" kept as a hidden back-compat alias), OpenAPI enum "Flex" + regenerated C++/TS clients, viewer combo, frontend dropdown, and the stored HDF5 string. "multi" collides with CrystFEL's multi-lattice sense; "flex" = let the pipeline decide. The API now carries a dedicated Flex value instead of masquerading as BeamCenter. Also fix a pre-existing bug in Convert(IndexingSettings): OrientationOnly was missing from the outbound switch and serialized as an invalid API value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 11:28:09 +02:00
leonarski_fandClaude Opus 4.8	135765e5b5	rugnux: default --refine-geometry on for stills-with-cell; accept F/FP reference columns ... Two stills usability fixes. 1. --refine-geometry now defaults ON for stills whenever a reference cell is available (-C or a reference MTZ) - exactly the case where the geometry bundle-adjust can act (it anchors on a known cell) and where it lifts weak/ sparse-stills indexing (OCP +42%, KR2 +108% indexed in the target study). It stays a no-op for rotation (own two-pass) and de-novo stills (no cell yet), so auto-enabling it only where it does something avoids spurious "skipping" warnings. --refine-geometry=off opts out; explicit --refine-geometry[=N] forces it. 2. The reference-MTZ auto column selection now falls back to a plain amplitude column (FP / FOBS / F / FC, squared to an intensity) after F-model and the intensity (J) columns. A deposition that carries only structure-factor amplitudes (e.g. KR2's 8cl8, whose only usable column is FP) now seeds CCref without needing an explicit --reference-column FP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 07:27:25 +02:00
leonarski_fandClaude Opus 4.8	768a926e0c	rugnux: reject 2nd-lattice overlaps in the space-group search (E^2 cap) ... On a two-lattice crystal a second lattice deposits intensity on one reciprocal position but not its symmetry mate, so a contaminated reflection is a one-sided resolution-normalised-E outlier that poisons the operator I(h)/I(Rh) correlation in SearchSpaceGroup Stage A. On EP_cs_02-424 under -A this dropped the monoclinic 2-fold CC to 0.326 (below the 0.5 gate) -> the crystal was under-called P1 instead of P2_1. The existing --reject-outliers cannot see it: it rejects WITHIN a P1 orbit, but the poison is a BETWEEN-orbit effect (inflates orbit h, not its mate Rh). Add SearchSpaceGroupOptions::max_e_squared_for_cc (default 9.0, i.e. E>3): after the resolution-normalised Esq[] is computed, drop the extreme-E tail from the correlation pairs only (the absence stage keeps the full range - that is where the screw signal lives). Clean Wilson-distributed data almost never reaches E^2=9 (P ~ 0.01-0.3%), so it is self-targeting - it trims the overlap tail without touching genuine reflections. EP_cs_02-424 -A: 2-fold CC 0.326 -> 0.64 -> P2_1 (= XDS). Full 24-crystal rotation battery (pre/post): only EP_cs_02-424 changed (P1 -> P2_1, SG match 20 -> 21/24); all 23 others bit-identical, including the marginal cubic Ins_I_3 (still I23, carried by the systematic-b rescue). A Friedel-merge-the-search alternative was tried and rejected (net-negative: did not fix EP424 and regressed Ins_I_3 -> I222). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 23:22:38 +02:00
leonarski_fandClaude Opus 4.8	4c050b576c	rugnux: de-double spurious rotation supercell in candidate selection ... The two-pass rotation indexer could keep a doubled-axis supercell over its own primitive sub-cell: when the supercell won candidate (ci) order and the primitive cleared the indexed-fraction hysteresis by less than the 0.05 "prefer-earlier" margin, the supercell survived. On pding4_003 full data this gave P222 on a 65.6x131.3x173 cell instead of P422 on 65.6x65.6x173 - the orthorhombic metric forecloses the 4-fold before the symmetry search ever runs. The FFT peak-finding is correct (the primitive cell is among the candidates, just out-ranked). Add a sub-cell override to the selection loop: adopt a later candidate that is a genuinely smaller cell (> ROT_SUBCELL_VOLUME_RATIO=1.5x smaller volume; a doubling is 2x) indexing at least as many spots (within ROT_SUBCELL_FRAC_SLACK=0.02). That is the signature of a spurious doubling - the primitive always indexes >= its integer multiple, whereas a real superstructure's larger cell indexes MORE (kept by the existing clearly-more branch) and twins share the cell volume (untouched). Full 24-crystal rotation battery (pre-fix vs post-fix): pding4_003 P222->P422 (cell halves to 65.6x65.6x173, R_meas 6.6% CC1/2 99.9% ISa 11.9), pding4_001 holds P422, all 22 other crystals bit-identical in space group. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 21:53:08 +02:00
leonarski_fandClaude Opus 4.8	4c5da20eda	rugnux: add --partiality-uncertainty stills merge term (partiality-model error) ... A stills reflection recorded at partiality p carries a systematic intensity error ~(dp/p) that is proportional to <I> and grows as p falls; plain counting sigma misses it, so strong low-p partials are over-trusted in the merge. Add sigma^2 += (c*<I>*(1-p))^2 in MergeOnTheFly::CorrectedSigma - the stills-partiality analog of the existing rotation --capture-uncertainty term ((1-captured_fraction)*I in RotationScaleMerge). Inert when partiality==1 (no --still-partiality), and gated on a real systematic (error_model_b > 1, i.e. ISa < 1) so it fires on strong/medium stills but auto-skips weak counting-limited data where it would only over-concentrate the merge and hurt. Opt-in via --partiality-uncertainty <c> (default 0; ~2.5 recommended with --still-partiality). Prototyped (Python replica of the merge): lyso8 CC1/2 +1.7, CCref +4.8, R-free -0.023; LOV CCref +2; harmful on weak OCP (hence the b>1 gate). Full in-binary validation across all serial targets pending. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 19:52:46 +02:00
leonarski_fandClaude Opus 4.8	2f5ed411e2	rugnux: --refine-geometry stills two-pass (global bundle adjustment + re-index) ... Weak/jet serial stills are often geometry-limited: a few-px beam error or a mm-scale detector distance error (unreachable per-image) fails many frames, but is well-determined jointly from the strong frames. Mirror the rotation two-pass: an index-only first pass over a spread sample collects each indexed frame's spots + assigned HKL + orientation; the strongest ~N (default 200) feed one Ceres bundle adjustment; the refined geometry is applied and the main pass re-indexes + integrates + merges every frame from scratch. GeometryRefiner (reusing the extracted XtalResidual - the RecipToDetector geometry residual pulled out of XtalOptimizer, behaviour-preserving): one problem with SHARED beam(2)/distance(1)/cell-length(3) blocks + a PER-FRAME orientation(3) block, robust Cauchy loss, a cell-length regularizer anchoring the known cell to break the low-resolution distance<->cell-scale degeneracy, DENSE_SCHUR eliminating the per-frame orientations, and a 3-round HKL-reassignment / tolerance-tightening loop. Tilt is not refined (gauge-coupled, zero gain). Opt-in via --refine-geometry[=N]; stills only (rotation untouched). Validated: KR2 7.58% -> 21.85% (matches CrystFEL's 21.5%; a real ~1.4mm distance error + ~3px beam), OCP 2.92% -> 4.19% (~3px beam). OFF runs are bit-identical to baseline (XtalResidual extraction non-regressing; rotation lyso_ref de-novo ISa 17.3 unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 18:17:58 +02:00
leonarski_fandClaude Opus 4.8	49963ab855	rugnux: move indexing-ambiguity resolution out of ScaleOnTheFly ... Indexing-ambiguity detection/resolution is no longer part of scaling: it is a dedicated per-image step for stills (or a post-full-merge step for rotation). Pull the stills per-image resolver out of ScaleOnTheFly into a new ReindexAmbiguityResolver in ReindexAmbiguity.{h,cpp}, alongside the existing rotation free functions (ChooseReindex / ReferenceIntensityCC). Both workflows now share the operator generation and the best-op selection (new file-local PickBestReindex helper); ChooseReindex is rewired onto it. ScaleOnTheFly is now purely a scaling engine (no ambiguity_ops, no resolve_ambiguity flag, no ResolveIndexingAmbiguity). Pure refactor: behaviour is unchanged. IndexAndRefine now holds a ReindexAmbiguityResolver and calls Resolve() at the same call site. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 15:35:40 +02:00
leonarski_fandClaude Opus 4.8	95af137146	rugnux: the external reference MTZ is never a scale anchor (stills + rotation) ... An external reference MTZ is a poor per-image scale reference: it is a different crystal/dataset, so scaling against it injects cross-dataset systematics rather than removing per-image scale error (measured: reference-scaled R-free 0.410 vs self-scaled 0.388 on OCP; 0.378 vs 0.350 on LOV). Its real value is fixing the cell/space group, breaking the merohedral indexing ambiguity, reporting CCref and providing the R-free test set - none of which need it to be a scale anchor. Make that consistent across both workflows: - Stills: the per-image pass (ScaleImage) now uses the reference ONLY to break the indexing ambiguity (once, for good), not to fit G. Scaling self-references at the post-measurement merge for every run, whether or not a reference is given (Rugnux + rugnux_cli --scale). - Rotation: ScaleImage no longer scales against the reference either (it was dead code - RotationScaleMerge recomputes the per-frame scale, so the reference never actually moved the result). Rotation resolves the ambiguity globally (ChooseReindex / ReferenceIntensityCC) and self-scales in RotationScaleMerge. - A reference-as-scale mode, if ever wanted, would be a dedicated later step, not this per-image pass. Validated: OCP CCref 56.5%->64.6%, R-free 0.400->0.393 (honest 2.36 A cutoff instead of a reference- propped 1.5 A); LOV CCref 88.3%->90.1%, R-free 0.336->0.331 with more reflections; rotation lyso_ref with the reference gives ISa 17.8 == de-novo 15.6 (same 15975 reflections, CC1/2 99.8%) - the reference demonstrably does not touch rotation scaling, only cell/SG + ambiguity + CCref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 13:01:56 +02:00
leonarski_fandClaude Opus 4.8	4a5028d98d	rugnux: add -r multi geometry refinement (per-image best of three) ... Per-image geometry refinement is a tradeoff that flips per dataset: beam+cell refinement extends OCP merge quality but DIVERGES on sparse-spot stills (e.g. KR2's ~20 spots around a 232 A axis), pushing good lattices out of tolerance so they fail the acceptance floor. `-r multi` runs all three (none / orientation / beam_and_lattice) on a copy per image and keeps whichever indexes the most spots (scorer = fractional-Miller-within-tolerance count, mirroring AnalyzeIndexing); ties prefer less refinement, to avoid overfitting the sparse list. Validated: KR2 index 7.08% (-r beam_and_lattice default) -> 10.05% (matches the -r none best), while OCP R-free stays ~equal to beam_and_lattice. New GeomRefinementAlgorithmEnum::Multi handled in the CLI/HDF5/command-line echoes; the API convert maps it to beam_and_lattice (offline only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:04:32 +02:00
leonarski_fandClaude Opus 4.8	fb399480e5	rugnux: add opt-in stills partiality (--still-partiality) ... Serial stills currently treat every reflection as a full (partiality hardcoded to 1). Add an opt-in Gaussian excitation-error partiality set at prediction time (CPU + CUDA): p = exp(-dist_ewald^2 / (2*sigma^2)), sigma^2 = profile_radius^2 + (bandwidth_sigma*|recip_z|)^2, with sigma = the per-image profile radius (ewald_dist_cutoff/2), so an edge-of-acceptance reflection keeps p ~ exp(-2). Off by default; the merge weight (~p^2) then down-weights far-from-Ewald partials instead of trusting them as fulls. Validated: helps medium/strong stills (LOV R-free 0.336->0.329, lyso8 0.433->0.410, lowers the systematic error-model b in both) but HURTS weak OCP (dividing by a small, uncertain p amplifies orientation error -> high-res noise, resolution collapse), so it is left opt-in. A static forward p explains only ~6-10% of the partiality scatter; the full win needs per-image post-refinement (future work), for which this is the prediction-side groundwork. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:03:21 +02:00
leonarski_fandClaude Opus 4.8	2d7ddb1b67	ScaleOnTheFly: resolve the stills indexing ambiguity per image vs a reference ... Serial stills index each crystal in one of the merohedrally-equivalent hands at random, so the ambiguity must be broken per image, not globally as the rotation path (RotationScaleMerge/ChooseReindex) does. When an external reference MTZ is supplied, IndexAndRefine::ReferenceIntensities now builds the per-image scaler with resolve_ambiguity=true: for each image it picks the reindexing (identity or a twin law from ReindexAmbiguityOperators) whose partiality/Lorentz-corrected intensities correlate best with the reference, then reindexes that image's reflections before scaling. Off for the self-merge scaling pass (no trusted reference to break the tie). Holohedral crystals (no twin law, e.g. lysozyme P43212) are a no-op. Validated on OCP P3221 (twin law -x,-y,z) with a 7ZSJ reference: CCref 24->48% and CC1/2 24->64% at 30k images; CC1/2 96.9% / CCref 76% at 150k. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 19:46:48 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00