1155 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	3bfce24faa	rugnux: score the decay cross-validation on the sigma-independent metric ... RefineDecay's cross-validation still scored the held-out equivalents on a studentized chi^2, while the absorption/modulation surfaces were switched to a sigma-independent Rmeas-like agreement (sum|Is-Iref|/sum|Iref|) precisely because a correction can "pass" a studentized gate by reshaping sigma without tightening the intensities. The decay slope scales sigma up on late / high-angle frames, so it is exposed to the same loophole. Score it the same way. On the rotation battery this rejects a spurious slope on a mis-indexed monoclinic (P2_1) case: the studentized gate had accepted a physically absurd -74 A^2 "decay" that cross-validated yet collapsed the error model (ISa 7.1 -> 1.1) and CC1/2 (to ~2%); the fractional gate scores it a -50% held-out loss and skips it. A small genuine ~3 A^2 decay on another crystal now cross-validates and engages, with CC1/2 unchanged. Every other crystal is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	8bd0977833	rugnux_vs_xds.py: drop the removed --scaling-output flag ... rugnux always writes both MTZ and CIF now (the --scaling-output option was removed), so passing it made every crystal in the battery fail against the current binary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	85cf826d2e	rugnux: dataset Wilson B-factor estimate + robust per-image Wilson B ... Add a dataset-wide isotropic Wilson B-factor estimate, the analogue of XDS's "WILSON LINE ... B=" which we did not export. CalcGlobalWilsonB fits ln<I> vs 1/d^2 over the merged reflections (B = -2*slope), skipping the low-resolution non-linear region (d > 4 A) and shells past the signal limit (<I/sigma> < 1) so the estimate is insensitive to how far the merged data were carried. It is diagnostic only - not fed back into scaling - and is written to the mmCIF (_reflns.B_iso_Wilson_estimate), the printed merge statistics, and the log. Also harden the per-image Wilson B (CalcWilsonBFactor): accept the fit only when it is well-correlated and physically plausible (0 < B < 200 A^2), else leave b_factor unset. A bad frame (an indexing glitch, too few reflections) otherwise produced a wildly steep Wilson line and a B of several hundred A^2 that polluted the per-image plot; NaN is preferable to garbage. Diagnostic-only: the merged intensities and every merge statistic are byte-identical (verified baseline vs modified on the rotation battery). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 08:58:17 +02:00
leonarski_fandClaude Opus 4.8	3b168a1a46	viewer: expose detector-plane modulation and background trim in the settings dock ... Two new controls in the interactive settings dock: - Scaling section: a "Detector-plane modulation (stills)" checkbox that toggles the stills flat-field modulation surface (ScalingSettings::StillsModulation), alongside the existing decay/absorption "Correction surfaces" checkbox (rotation modulation is fitted there). Off by default. - Bragg integration section: a "Background trim" checkbox + fraction that sets the symmetric trimmed-mean background fraction (BraggIntegrationSettings::BackgroundTrimFraction; 0 = plain ring mean). Defaults on at 0.10, matching the integrator default, so the viewer requests trimming for every dataset (the integration engine still applies its own per-mode gating). Viewer-only change; compiles against the current settings API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 07:48:10 +02:00
leonarski_fandClaude Opus 4.8	74d7589abe	stills: optional detector-plane modulation (flat-field) correction ... The stills scaling/merging path (ScaleOnTheFly + MergeOnTheFly) had no correction surfaces, whereas rotation fits decay/absorption/modulation inside RotationScaleMerge. Serial stills hammer the same detector regions shot after shot, so a detector-plane (flat-field) systematic is at least as well determined there as in rotation - it just had no home. Add MergeOnTheFly::RefineModulation: a 16x16 multiplicative surface over the predicted detector position, fit against the merged reference and folded into each reflection's image_scale_corr before the error model and the merge. It mirrors RotationScaleMerge::ApplyCellSurface - alternating per-cell fit with a den-weighted geometric-mean gauge (never drifts the overall scale) and Tikhonov pull to 1 - and is cross-validated the same way: fit the surface on even images, score the held-out odd equivalents by a sigma-independent R-meas-like agreement (a fractional metric a sigma-reshaping surface cannot game), and apply the full-data surface only if the held-out gain clears a margin. A no-op when the systematic is absent or the data too sparse (< 8 obs per cell), so it is safe to leave off by default and opt in. Wired through ScalingSettings::StillsModulation and rugnux --stills-modulation (default off), in both the full-analysis and --scale paths. On a 17 MP JUNGFRAU serial set the surface cross-validates with a large held-out gain, confirming a real detector-plane systematic; whether it improves the merged data quality is under evaluation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 20:45:44 +02:00
leonarski_fandClaude Opus 4.8	c3e877d5ec	bragg integration: trimmed-mean background, on by default for rotation ... The local Bragg background is the mean over the r2..r3 ring. That mean reads high because the contaminants that survive the signal-disk mask - neighbour- spot wings, tails, zingers - are one-sided (positive), so it over-subtracts. Since a weak intensity is a small difference of large numbers (I = S - nS*b), a per-pixel background bias is fractionally largest at the resolution edge, exactly where it hurts most. Replace the ring mean with a symmetric trimmed mean (sort the ring, drop the lowest and highest fraction f, average the rest), controlled by a new BraggIntegrationSettings field and the rugnux `--background-trim <f>` option (default f=0.10; 0 restores the plain mean). Default on for monochromatic (rotation) data; broadband (stills) keep their tuned high-side sigma-clip, so the base engine forces the trim to 0 there. Implemented in both the CPU engine and the GPU kernel (shared-memory bitonic sort per block, flat-mean fallback above BKG_TRIM_MAX ring pixels); the two agree. 25-crystal rotation battery (fixed SG/cell): <I/sigma> improved on every crystal (median +50%), ISa on 20/22, resolution-edge R_meas fell several-fold (e.g. lyso_ref 1.0 A 108%->43%). Last-shell CC1/2 is rescued where the plain mean had collapsed to noise (Thau_9 at ~2.0 A 3.8%->64%, ~0.5 A of resolution regained; cytC_10 0.2%->10%) at a small cost (1-3%) in already-clean shells - it flattens the CC1/2 fall-off rather than shifting it. Stills unchanged. Documented in CPU_DATA_ANALYSIS.md section 9.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 18:32:57 +02:00
leonarski_fandClaude Opus 4.8	118f0833dc	common: remove dead Reflection.completeness field ... Reflection.completeness (spot-footprint fraction) was only ever set to its 1.0f default and never read anywhere - not in scaling/merge, CBOR, HDF5, or the viewer. The box-sum integrator already rejects any partial-footprint reflection outright (n_inner_valid == n_inner), so the intended "down-weight clipped spots" role is already covered more strictly. Removed to reduce confusion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 18:32:40 +02:00
leonarski_fandClaude Opus 4.8	818ec8fcbf	rugnux: add detector-plane modulation (flat-field) correction surface ... Rotation scaling gained a third cross-validated correction surface alongside decay and absorption: a smooth multiplicative factor over the predicted detector position (px, py) where each full lands. It absorbs detector-response and geometric flat-field systematics that vary across the detector plane and inflate R-meas; symmetry-equivalents of one reflection land at different detector positions as the crystal rotates, over-determining the surface. Because it lives in the detector frame (not the rotation), the same correction concept applies to stills. The surface fit/cross-validate/apply machinery shared with the absorption surface is factored into ApplyCellSurface; RefineAbsorption and the new RefineModulation just build their per-full cell assignment (goniometer-frame vs detector-frame) and call it. The cross-validation now scores a sigma-INDEPENDENT, R-meas-like fractional agreement of the held-out equivalents rather than a studentized chi^2, so a surface can no longer pass CV by reshaping sigma without tightening the intensities - this removes an over-fit regression on mis-indexed data and hardens the absorption surface too. On the /data/rotation_test rotation battery (A/B, modulation off vs on): 17 of 22 processed crystals improve R-meas, 0 regress, e.g. lysoC 23.2->16.3%, lyso_2 47.4->28.5%, EcwtAL500 53.2->27.6%, EP_cs_01-17 CC1/2 60->93%; CC1/2 held or improved everywhere and the anomalous S signal preserved/improved (lyso_ref SD_MET 4.37->4.46, lysoC 3.07->3.25). On by default with the other surfaces. Also: --dump-observations now writes the per-full centroid frame and predicted px/py columns, so the dumped combined fulls are a complete unmerged export. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 13:24:10 +02:00
leonarski_fandClaude Opus 4.8	eae7ea90dc	rugnux: always write both MTZ and CIF; drop --scaling-output ... The merged reflections were written in one of mtz/cif/txt selected by --scaling-output. Write both an MTZ and an mmCIF unconditionally instead - each has its uses downstream (MTZ for the CCP4/phenix tools, mmCIF for deposition) - and remove the format selector, the plain-text .hkl writer, and the now-unused IntensityFormat enum / ScalingSettings::FileFormat plumbing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:45:08 +02:00
leonarski_fandClaude Opus 4.8	ce12d69751	rugnux: write standard CCP4 anomalous MTZ (IMEAN + I(+)/I(-)) ... In anomalous mode the merge keeps the two Friedel mates as separate rows, which WriteMtzReflections emitted verbatim - two IMEAN rows per Bijvoet pair that downstream tools had to re-collapse. Pair the mates into one row per reflection with the standard CCP4 anomalous layout: IMEAN + I(+)/I(-) and the matching F/F(+)/F(-) amplitudes. A single HKLKeyGenerator(merge_friedel=false) yields both the shared ASU group key and which mate a row is (.plus); IMEAN/F are the inverse-variance Friedel mean, centrics/unpaired keep one hand with the other missing. aimless/mtz2sca/ANODE now read the file directly. The non-anomalous MTZ path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-16 10:37:56 +02:00
leonarski_fandClaude Opus 4.8	ce15900471	changelog: add the 1.0.0-rc.160 section ... VERSION was bumped to rc.160 but the new items sat under the rc.159 heading, which released (origin/main) does not contain. Add a proper rc.160 section, move the three misfiled items (supercell fix, -S symbol, FFTW-on-GPU error) into it, and document the rest of the branch (stills geometry refinement, -r flex, stills partiality, r=6 box default, single-pass self-referenced scaling, reference-MTZ-never-a-scale-anchor, E^2 second-lattice cap, --scale on _process.h5 fix, new spot options, CBF/TIFF writer removal, negative total_flux, viewer, packaging). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:20:14 +02:00
leonarski_fandClaude Opus 4.8	0e4f91b219	rugnux: validate offline spot-finding settings and bound numeric options ... The offline rugnux path never ran CheckDataProcessingSettings (only the online broker/receiver paths did), so the new --min-pix-per-spot and --spot-low-resolution knobs were unbounded: --min-pix-per-spot 0 silently disabled the per-spot filter, and a low-resolution limit finer than the high-resolution limit made spot finding reject every pixel with no diagnostic. Add a parse_number_arg<T> helper (integral or floating, with optional inclusive bounds) that rejects non-numeric input, trailing garbage, and out-of-range values, replacing the raw atoi/atof in the spot-finding options; parse_double_arg/parse_float_arg become thin wrappers over it. After assembling the settings, validate them with the same CheckDataProcessingSettings the online receivers use, which also enforces the cross-field low>=high-resolution constraint. Also scrubs a residual sample-name comment (the r=6 box-default note) in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:19:59 +02:00
leonarski_fandClaude Opus 4.8	8fea7f195a	docs: stop advertising the removed CBF/TIFF writers ... The CBF/TIFF writer removal left two docs describing them as available output: JFJOCH_WRITER.md's "Other formats" section and an aside in HDF5.md. A user following them would issue a now-rejected request. State that only NXmx HDF5 is written and that the CBF/TIFF enum values are retained for wire back-compatibility but rejected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:19:46 +02:00
leonarski_fandClaude Opus 4.8	d393ec814c	Remove residual sample identities missed by the earlier scrub ... The rc.160 sample-identity cleanup left protein names, internal dataset codes, and a measured cell angle in several source comments and one test. Most sensitive: a measured monoclinic beta tied to an internal code in Rugnux.cpp and IndexAndRefine.cpp. Rewrite each to describe the crystallographic situation only (space group / metric relation), keeping the technical reasoning intact. Comment- and string-only; no behaviour change. (rugnux_cli.cpp's remaining name is scrubbed in the following commit, alongside its other change to that file.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-15 22:18:32 +02:00
leonarski_f	68272a96e1	VERSION: 1.0.0-rc.160	2026-07-15 21:41:51 +02:00
leonarski_fandClaude Opus 4.8	b450140816	rugnux: warn when the cell/space group has an indexing ambiguity ... At the twinning-analysis print, detect alternative-indexing (twin-law) operators for the final cell + space group via ReindexAmbiguityOperators(). When any exist and no reference was supplied, emit a Warning (also written into the merge stats) that serial-stills crystals are indexed in a random hand and the ambiguity can only be broken against a reference (-z / --model). For an obvious merohedral case (P3/P4/P6...) users expect this, but a PSEUDO-merohedral metric is easy to miss -- e.g. NmHR / 7O8F, where C2 beta=131.8 is pseudo-F-orthorhombic and gemmi reports the reindex operator h-l,-k,-l. Advisory only; no change to processing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 20:38:31 +02:00
leonarski_fandClaude Opus 4.8	3e3545770f	indexing: check candidate angles against the reference cell in Refine() ... When a reference cell is supplied, PostIndexingRefinement::Refine() vetted candidates on sorted edge lengths only; a cell with the right edges but a wrong angle (a pseudo-symmetric near-metric, e.g. a monoclinic beta refined to the wrong value) passed. Add a sorted-angle check against the reference, folding each angle to its acute complement min(x,180-x) so the obtuse/acute setting choice is irrelevant (tolerance 10 deg). Reference-cell path only (no effect de novo, where reference_unit_cell is null): the /data/rotation_test battery is byte-identical (22/25), and NmHR with the correct C2 cell is unchanged (9.1%, beta~131.7). A latent guard against wrong-angle cells slipping through on -C / reference-MTZ runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:47:29 +02:00
leonarski_fandClaude Opus 4.8	0f07b7a499	rotation: widen candidate-refine angle bound to [30,150] deg ... RotationIndexer's per-candidate XtalOptimizer relied on the struct-default monoclinic angle bound [60,120], the same latent clamp fixed for the stills path: a rotation crystal with beta>120 (or <60) would have its cell clamped to the boundary. Set [30,150] explicitly, matching the indexers. No change on the /data/rotation_test battery (22/25 SG match, byte-identical) -- no battery crystal has beta outside [60,120]; this is a latent-correctness fix for future high-beta monoclinic rotation data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:47:17 +02:00
leonarski_fandClaude Opus 4.8	643bda3dd3	stills: widen per-frame cell-refine angle bound to [30,150] deg ... The stills per-frame XtalOptimizer in IndexAndRefine used the struct-default monoclinic angle bound [60,120] deg, which clamps a beta outside that window (e.g. NmHR / 7O8F, C2 beta=131.78) to the boundary and corrupts the per-frame cell. Both indexers already use [30,150] (FFBIDXIndexer, FFT settings); this aligns the per-frame refine with them. Rotation path is unaffected (refine_unit_cell is false there). Verified on NmHR serial stills: FFBIDX with the correct C2 cell went from 0.01% to 9.10% indexed, recovering the true mC cell (beta~131.7) instead of a clamped beta=120; 60k-image merge reaches 2.61 A at 99.8% completeness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 19:13:41 +02:00
leonarski_fandClaude Opus 4.8	7b2cfb20f8	changelog: document the supercell fix, -S symbol, and FFTW-on-GPU error ... Also scrub two sample names from existing rc.159 entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:19:43 +02:00
leonarski_fandClaude Opus 4.8	d8e909a393	indexing: clear error when the requested indexer is unavailable on this host ... IndexerThreadPool built its per-worker indexers from the raw requested algorithm but dispatched the RESOLVED one. Rotation indexing always resolves to the GPU FFT indexer when a GPU is present, so requesting FFTW (the CPU indexer) on a GPU node created only the CPU indexer and then failed at dispatch with an opaque "no indexer available for the resolved indexing algorithm" deep inside a worker. Validate servability at the pool entry (in the caller's context) and throw an actionable message instead: FFTW is not available on a GPU node (use FFT/Auto, or a CPU-only node); FFT/FFBIDX require a GPU. Stills FFTW on a GPU node, which the algorithm resolution still honours, keeps working. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:19:43 +02:00
leonarski_fandClaude Opus 4.8	abbee2d4dc	Remove sample identities from the repository; document the rule ... Datasets may be confidential; sample names and measured unit cells committed to the repo can leak outside the group working on them. Scrub existing occurrences and add a "No sample identities in the repository" section to CLAUDE.md (forbidden: sample/dataset names, internal codes, measured cells tied to a sample; fine: space group / lattice / twinning descriptors). - Comments: replace internal dataset codes and protein names with the crystallographic situation they illustrate (centred vs pseudo-symmetric, holohedral, cubic, F-cubic/hexagonal, ...). - Docs: same, in the analysis/writer/stream references and example configs. - Tests: rename sample-named identifiers, TEST_CASE names, file prefixes and asserted labels to neutral crystallographic names (e.g. tetragonal_uc); behaviour unchanged. Reduce the CrystFEL reference PDB to a bare CRYST1 cell file (cell.pdb) and rename the reference data file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 17:19:43 +02:00
leonarski_fandClaude Opus 4.8	0f3ccda779	rugnux: fix de-novo rotation indexing adopting a spurious axis-multiple supercell ... Two-pass rotation indexing scored each first-pass scheme by how many validation frames it indexes, but a spurious axis multiple (2x/3x...) indexes every frame its true sub-cell does, so the count saturates - both schemes reach the same frame total - and the tie fell to whichever scheme ran first. When that first scheme's full-rotation FFT resolves a true axis only as a 2x/3x harmonic of its length (the fundamental can sit far below the harmonic in the full-360 cloud), it commits a multiplied cell and the dataset collapses to P1, even though the other scheme already found the true cell. Add an integer-supercell cross-scheme tie-break: when two schemes tie on frames but their cell volumes differ by a near-integer factor >=2, the larger is the spurious supercell and the smaller true cell is adopted, independent of scheme order. The near-integer test distinguishes a real axis multiplication from a centering coincidence (a rhombohedral H cell vs its C2 sub-cell is 1.5x and is left alone). Validated de-novo across the rotation regression battery: the affected dataset now indexes its correct space group, the override triggers only where an axis was multiplied, and every other dataset is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 16:51:57 +02:00
leonarski_fandClaude Opus 4.8	bd094e585f	rugnux: accept a space-group symbol for -S, not just a number ... -S now takes either a number ("92") or a Hermann-Mauguin symbol ("P43212"), resolved via gemmi find_spacegroup_by_name. Previously a non-numeric argument was atoi'd to 0 and tripped the early space-group validation, aborting the run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 15:23:31 +02:00
leonarski_fandClaude Opus 4.8	ebe7e7ec5e	jfjoch_viewer: retry deferred initial fit on resize (grid-scan zoom) ... fitToViewShorterSideOnce() skips the initial fit-to-view while the viewport has no real size yet (before the widget is laid out/shown) and relies on a retry that was never wired: resizeEvent only reset the scene rect, and there is no showEvent. When the first Redraw() lands before layout settles the view stays at 1:1, so a small grid-scan plot renders tiny ("zoomed out, not taking the full picture"); the layout-timing race is why it happened only sometimes. resizeEvent now retries fitToViewShorterSideOnce() while the initial fit is still pending (guarded by !initial_fit_done_, so a later user resize never overrides a manual zoom). It lives in the shared base class, so all image views get the same robustness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 11:28:09 +02:00
leonarski_fandClaude Opus 4.8	5d1ddccee8	viewer: expose stills processing settings; rename geom refinement multi -> flex ... Wire the recently-added stills knobs into jfjoch_viewer - they were reachable only from the rugnux CLI, and the viewer had silently diverged from the CLI once --refine-geometry became default-on for stills-with-cell. - Stills geometry-refinement two-pass (--refine-geometry): a checkbox + frame count in the reprocessing-job dialog, offered (and defaulted on, matching the CLI) only for a stills-with-cell run; wired into ProcessConfig and the "Copy command" generator (joined =N, or =off to reproduce opting out of the CLI default-on). - Stills partiality (--still-partiality) checkbox in Bragg integration and the partiality-uncertainty merge term (--partiality-uncertainty) in Scaling, both mirrored in the command-line generator. Rename the "best per-image refinement" mode from "multi" to "flex" everywhere: GeomRefinementAlgorithmEnum::Flex, CLI -r flex (with "multi" kept as a hidden back-compat alias), OpenAPI enum "Flex" + regenerated C++/TS clients, viewer combo, frontend dropdown, and the stored HDF5 string. "multi" collides with CrystFEL's multi-lattice sense; "flex" = let the pipeline decide. The API now carries a dedicated Flex value instead of masquerading as BeamCenter. Also fix a pre-existing bug in Convert(IndexingSettings): OrientationOnly was missing from the outbound switch and serialized as an invalid API value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 11:28:09 +02:00
leonarski_fandClaude Opus 4.8	4bdf9309dd	writer: remove CBF and TIFF writers; only HDF5 is written ... Delete CBFWriter and the TIFF write path from jfjoch_writer so the writer only produces the NXmx HDF5 formats. Drop CBF/TIFF from the internal FileWriterFormat enum and reject them in FileWriterSettings. The CBF/TIFF values are kept (marked deprecated) in the file_writer_format OpenAPI enum for back compatibility: incoming requests using them are now rejected in OpenAPIConvert with a clear "no longer supported" error, and a stale CBF/TIFF value on the CBOR wire decodes to unset rather than a removed enumerator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 10:11:39 +02:00
leonarski_fandClaude Opus 4.8	135765e5b5	rugnux: default --refine-geometry on for stills-with-cell; accept F/FP reference columns ... Two stills usability fixes. 1. --refine-geometry now defaults ON for stills whenever a reference cell is available (-C or a reference MTZ) - exactly the case where the geometry bundle-adjust can act (it anchors on a known cell) and where it lifts weak/ sparse-stills indexing (OCP +42%, KR2 +108% indexed in the target study). It stays a no-op for rotation (own two-pass) and de-novo stills (no cell yet), so auto-enabling it only where it does something avoids spurious "skipping" warnings. --refine-geometry=off opts out; explicit --refine-geometry[=N] forces it. 2. The reference-MTZ auto column selection now falls back to a plain amplitude column (FP / FOBS / F / FC, squared to an intensity) after F-model and the intensity (J) columns. A deposition that carries only structure-factor amplitudes (e.g. KR2's 8cl8, whose only usable column is FP) now seeds CCref without needing an explicit --reference-column FP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-15 07:27:25 +02:00
leonarski_fandClaude Opus 4.8	768a926e0c	rugnux: reject 2nd-lattice overlaps in the space-group search (E^2 cap) ... On a two-lattice crystal a second lattice deposits intensity on one reciprocal position but not its symmetry mate, so a contaminated reflection is a one-sided resolution-normalised-E outlier that poisons the operator I(h)/I(Rh) correlation in SearchSpaceGroup Stage A. On EP_cs_02-424 under -A this dropped the monoclinic 2-fold CC to 0.326 (below the 0.5 gate) -> the crystal was under-called P1 instead of P2_1. The existing --reject-outliers cannot see it: it rejects WITHIN a P1 orbit, but the poison is a BETWEEN-orbit effect (inflates orbit h, not its mate Rh). Add SearchSpaceGroupOptions::max_e_squared_for_cc (default 9.0, i.e. E>3): after the resolution-normalised Esq[] is computed, drop the extreme-E tail from the correlation pairs only (the absence stage keeps the full range - that is where the screw signal lives). Clean Wilson-distributed data almost never reaches E^2=9 (P ~ 0.01-0.3%), so it is self-targeting - it trims the overlap tail without touching genuine reflections. EP_cs_02-424 -A: 2-fold CC 0.326 -> 0.64 -> P2_1 (= XDS). Full 24-crystal rotation battery (pre/post): only EP_cs_02-424 changed (P1 -> P2_1, SG match 20 -> 21/24); all 23 others bit-identical, including the marginal cubic Ins_I_3 (still I23, carried by the systematic-b rescue). A Friedel-merge-the-search alternative was tried and rejected (net-negative: did not fix EP424 and regressed Ins_I_3 -> I222). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 23:22:38 +02:00
leonarski_fandClaude Opus 4.8	4c050b576c	rugnux: de-double spurious rotation supercell in candidate selection ... The two-pass rotation indexer could keep a doubled-axis supercell over its own primitive sub-cell: when the supercell won candidate (ci) order and the primitive cleared the indexed-fraction hysteresis by less than the 0.05 "prefer-earlier" margin, the supercell survived. On pding4_003 full data this gave P222 on a 65.6x131.3x173 cell instead of P422 on 65.6x65.6x173 - the orthorhombic metric forecloses the 4-fold before the symmetry search ever runs. The FFT peak-finding is correct (the primitive cell is among the candidates, just out-ranked). Add a sub-cell override to the selection loop: adopt a later candidate that is a genuinely smaller cell (> ROT_SUBCELL_VOLUME_RATIO=1.5x smaller volume; a doubling is 2x) indexing at least as many spots (within ROT_SUBCELL_FRAC_SLACK=0.02). That is the signature of a spurious doubling - the primitive always indexes >= its integer multiple, whereas a real superstructure's larger cell indexes MORE (kept by the existing clearly-more branch) and twins share the cell volume (untouched). Full 24-crystal rotation battery (pre-fix vs post-fix): pding4_003 P222->P422 (cell halves to 65.6x65.6x173, R_meas 6.6% CC1/2 99.9% ISa 11.9), pding4_001 holds P422, all 22 other crystals bit-identical in space group. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 21:53:08 +02:00
leonarski_fandClaude Opus 4.8	4c5da20eda	rugnux: add --partiality-uncertainty stills merge term (partiality-model error) ... A stills reflection recorded at partiality p carries a systematic intensity error ~(dp/p) that is proportional to <I> and grows as p falls; plain counting sigma misses it, so strong low-p partials are over-trusted in the merge. Add sigma^2 += (c*<I>*(1-p))^2 in MergeOnTheFly::CorrectedSigma - the stills-partiality analog of the existing rotation --capture-uncertainty term ((1-captured_fraction)*I in RotationScaleMerge). Inert when partiality==1 (no --still-partiality), and gated on a real systematic (error_model_b > 1, i.e. ISa < 1) so it fires on strong/medium stills but auto-skips weak counting-limited data where it would only over-concentrate the merge and hurt. Opt-in via --partiality-uncertainty <c> (default 0; ~2.5 recommended with --still-partiality). Prototyped (Python replica of the merge): lyso8 CC1/2 +1.7, CCref +4.8, R-free -0.023; LOV CCref +2; harmful on weak OCP (hence the b>1 gate). Full in-binary validation across all serial targets pending. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 19:52:46 +02:00
leonarski_fandClaude Opus 4.8	2f5ed411e2	rugnux: --refine-geometry stills two-pass (global bundle adjustment + re-index) ... Weak/jet serial stills are often geometry-limited: a few-px beam error or a mm-scale detector distance error (unreachable per-image) fails many frames, but is well-determined jointly from the strong frames. Mirror the rotation two-pass: an index-only first pass over a spread sample collects each indexed frame's spots + assigned HKL + orientation; the strongest ~N (default 200) feed one Ceres bundle adjustment; the refined geometry is applied and the main pass re-indexes + integrates + merges every frame from scratch. GeometryRefiner (reusing the extracted XtalResidual - the RecipToDetector geometry residual pulled out of XtalOptimizer, behaviour-preserving): one problem with SHARED beam(2)/distance(1)/cell-length(3) blocks + a PER-FRAME orientation(3) block, robust Cauchy loss, a cell-length regularizer anchoring the known cell to break the low-resolution distance<->cell-scale degeneracy, DENSE_SCHUR eliminating the per-frame orientations, and a 3-round HKL-reassignment / tolerance-tightening loop. Tilt is not refined (gauge-coupled, zero gain). Opt-in via --refine-geometry[=N]; stills only (rotation untouched). Validated: KR2 7.58% -> 21.85% (matches CrystFEL's 21.5%; a real ~1.4mm distance error + ~3px beam), OCP 2.92% -> 4.19% (~3px beam). OFF runs are bit-identical to baseline (XtalResidual extraction non-regressing; rotation lyso_ref de-novo ISa 17.3 unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 18:17:58 +02:00
leonarski_fandClaude Opus 4.8	49963ab855	rugnux: move indexing-ambiguity resolution out of ScaleOnTheFly ... Indexing-ambiguity detection/resolution is no longer part of scaling: it is a dedicated per-image step for stills (or a post-full-merge step for rotation). Pull the stills per-image resolver out of ScaleOnTheFly into a new ReindexAmbiguityResolver in ReindexAmbiguity.{h,cpp}, alongside the existing rotation free functions (ChooseReindex / ReferenceIntensityCC). Both workflows now share the operator generation and the best-op selection (new file-local PickBestReindex helper); ChooseReindex is rewired onto it. ScaleOnTheFly is now purely a scaling engine (no ambiguity_ops, no resolve_ambiguity flag, no ResolveIndexingAmbiguity). Pure refactor: behaviour is unchanged. IndexAndRefine now holds a ReindexAmbiguityResolver and calls Resolve() at the same call site. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 15:35:40 +02:00
leonarski_fandClaude Opus 4.8	3f6462b30f	rugnux: rotation indexing always uses FFT/FFTW, never FFBIDX ... The rotation two-pass accumulates a dense multi-frame reciprocal-space cloud (all first-pass frames de-rotated into one common frame) and needs a GLOBAL lattice finder - a 3D FFT whose peaks are the basis vectors. ffbidx is a single-still, known-cell ORIENTATION finder: it assumes the spots lie on one Ewald shell (one crystal, one orientation) and maximizes a near-integer-hkl count for a fixed cell. On the volume-filling rotation cloud that objective has no dominant global maximum, so ffbidx's coarse orientation scan locks onto an orientation valid for only a local wedge (~18%) -> garbage merge. GetIndexingAlgorithm resolved Auto->FFBIDX whenever a cell was known, even for rotation - and a reference MTZ (-z) or -C silently supplies the cell. So a routine rotation run with a reference for CCref/R-free flipped the indexer to the one solver that cannot handle its input (lyso_ref: ISa 17.8 with FFT -> 0.5 with FFBIDX, no warning). Force rotation to FFT (GPU) / FFTW (CPU) regardless of the requested algorithm or a known cell. Validated: lyso_ref -R with -z reference MTZ now indexes via FFT -> ISa 17.8 (was 0.5), 15975 refl. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 15:18:58 +02:00
leonarski_fandClaude Opus 4.8	95af137146	rugnux: the external reference MTZ is never a scale anchor (stills + rotation) ... An external reference MTZ is a poor per-image scale reference: it is a different crystal/dataset, so scaling against it injects cross-dataset systematics rather than removing per-image scale error (measured: reference-scaled R-free 0.410 vs self-scaled 0.388 on OCP; 0.378 vs 0.350 on LOV). Its real value is fixing the cell/space group, breaking the merohedral indexing ambiguity, reporting CCref and providing the R-free test set - none of which need it to be a scale anchor. Make that consistent across both workflows: - Stills: the per-image pass (ScaleImage) now uses the reference ONLY to break the indexing ambiguity (once, for good), not to fit G. Scaling self-references at the post-measurement merge for every run, whether or not a reference is given (Rugnux + rugnux_cli --scale). - Rotation: ScaleImage no longer scales against the reference either (it was dead code - RotationScaleMerge recomputes the per-frame scale, so the reference never actually moved the result). Rotation resolves the ambiguity globally (ChooseReindex / ReferenceIntensityCC) and self-scales in RotationScaleMerge. - A reference-as-scale mode, if ever wanted, would be a dedicated later step, not this per-image pass. Validated: OCP CCref 56.5%->64.6%, R-free 0.400->0.393 (honest 2.36 A cutoff instead of a reference- propped 1.5 A); LOV CCref 88.3%->90.1%, R-free 0.336->0.331 with more reflections; rotation lyso_ref with the reference gives ISa 17.8 == de-novo 15.6 (same 15975 reflections, CC1/2 99.8%) - the reference demonstrably does not touch rotation scaling, only cell/SG + ambiguity + CCref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 13:01:56 +02:00
leonarski_fandClaude Opus 4.8	9527fa6e2c	rugnux: scale self-referenced stills in a single pass (fix weak-data collapse) ... The stills self-scaling loop rebuilt the merge reference FROM the just-scaled reflections every iteration, so on weak data each pass re-fit its own noise and the merged CC1/2 collapsed (PfluDING combine: 0.3% at the default 3 iters). It is provably pointless too: ScaleOnTheFly::Scale re-solves the per-image G from raw I and never reads the prior correction, so against a fixed reference passes 2..N are bit-identical - the single G is the exact one-pass solution and iteration only ever does the harmful reference rebuild (N=1 is best on every dataset, including strong lyso 82% vs 76% at N=3; leave-one-out and external anchors were tested and do not beat it). Do one self-reference pass; rotation (RotationScaleMerge) and the external-reference branch are untouched. Validated: PfluDING-combined CC1/2 0.3% -> 42.0% by default (pinned 2.5 A); strong data unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:05:08 +02:00
leonarski_fandClaude Opus 4.8	4a5028d98d	rugnux: add -r multi geometry refinement (per-image best of three) ... Per-image geometry refinement is a tradeoff that flips per dataset: beam+cell refinement extends OCP merge quality but DIVERGES on sparse-spot stills (e.g. KR2's ~20 spots around a 232 A axis), pushing good lattices out of tolerance so they fail the acceptance floor. `-r multi` runs all three (none / orientation / beam_and_lattice) on a copy per image and keeps whichever indexes the most spots (scorer = fractional-Miller-within-tolerance count, mirroring AnalyzeIndexing); ties prefer less refinement, to avoid overfitting the sparse list. Validated: KR2 index 7.08% (-r beam_and_lattice default) -> 10.05% (matches the -r none best), while OCP R-free stays ~equal to beam_and_lattice. New GeomRefinementAlgorithmEnum::Multi handled in the CLI/HDF5/command-line echoes; the API convert maps it to beam_and_lattice (offline only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:04:32 +02:00
leonarski_fandClaude Opus 4.8	fb399480e5	rugnux: add opt-in stills partiality (--still-partiality) ... Serial stills currently treat every reflection as a full (partiality hardcoded to 1). Add an opt-in Gaussian excitation-error partiality set at prediction time (CPU + CUDA): p = exp(-dist_ewald^2 / (2*sigma^2)), sigma^2 = profile_radius^2 + (bandwidth_sigma*|recip_z|)^2, with sigma = the per-image profile radius (ewald_dist_cutoff/2), so an edge-of-acceptance reflection keeps p ~ exp(-2). Off by default; the merge weight (~p^2) then down-weights far-from-Ewald partials instead of trusting them as fulls. Validated: helps medium/strong stills (LOV R-free 0.336->0.329, lyso8 0.433->0.410, lowers the systematic error-model b in both) but HURTS weak OCP (dividing by a small, uncertain p amplifies orientation error -> high-res noise, resolution collapse), so it is left opt-in. A static forward p explains only ~6-10% of the partiality scatter; the full win needs per-image post-refinement (future work), for which this is the prediction-side groundwork. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:03:21 +02:00
leonarski_fandClaude Opus 4.8	b0202a3aa2	rugnux: default the stills integration box to r=6 (6,8,12) ... Serial-stills spots span a range of crystal orientations within a single shot, so they land wider on the detector than the r1=4 monochromatic-rotation default assumes, and the r1=4 signal box truncates the (bandwidth/mosaicity-broadened) spot wings. The profile-fit integrator can use a larger box for free - its profile weighting drives far-out background pixels to ~zero weight, so the estimator variance saturates rather than growing (a plain box-sum, which sums every pixel at weight 1, degrades). Gated on stills (!rotation_indexing); rotation stays at 4,6,10 (a larger box there only adds background - measured wash-to-negative on the XDS rotation battery, with an SG regression); an explicit --integration-radius wins. Validated: OCP R-free 0.406->0.400, LOV 0.354->0.336; lyso8 1.60->1.57 A; lyso5 (1% DMM) 1.80->1.72 A with +13% reflections at equal CC1/2 - the broadened case benefits most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 12:00:26 +02:00
leonarski_fandClaude Opus 4.8	db8ae0452e	rugnux: fix --scale on a self-contained _process.h5 (reflections + error model) ... Offline --scale re-scales the reflections stored in a _process.h5 without re-integration, but was broken for a self-contained integrated snapshot: 1. HDF5MetadataSource::ReadReflections fell back per-image to the linked source pixel files for every non-indexed frame (a snapshot's master holds /entry/reflections only for the sparse indexed images). With the raw _data_NNNNNN.h5 absent or not co-located this threw an HDF5 error; when present it needlessly reopened multi-GB files thousands of times. Decide once whether the master is the authoritative reflection store and, if so, never fall back - a missing per-image group just means the frame has none. Legacy/VDS acquisitions (no /entry/reflections in the master) still resolve reflections lazily to the source data files. 2. The scale-only stills branch never fit the (a,b) error model, so it merged with the identity model - far worse intensities than the full pipeline (lyso8 CC1/2 76%->21%). Fit RefineErrorModel and honour --reject-outliers, mirroring Rugnux.cpp. Round-trip validated: full merge == integrate(--no-merge) + --scale (identical error model a=0.793 b=2.287, matching unique reflection counts), and --scale now reads reflections in ~2.5s with no raw-file access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 10:06:07 +02:00
leonarski_fandClaude Opus 4.8	097b9a29d1	rugnux: add --spot-low-resolution CLI option (exclude beam halo) ... The spot-finding low-resolution limit was hardcoded at 50 A (r ~36 px), which admits the direct-beam halo/overload zone (24-50 A) on weakly-diffracting serial data. On KR2 (I222, 235 A axis, ~3.9 A diffraction) that halo is ~387 strong pixels/frame vs ~25 real Bragg pixels, swamping the signal ~15:1 (CrystFEL cuts it with min-res=75 px = 24 A). Expose the limit so it can be tightened, e.g. --spot-low-resolution 24. Default unchanged (50 A). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 23:14:27 +02:00
leonarski_fandClaude Opus 4.8	2f3041c44e	rugnux: add --min-pix-per-spot CLI option (spot finding) ... The minimum connected strong-pixel count per spot was hardcoded to 2 (SpotFindingSettings::min_pix_per_spot). Expose it on the CLI so serial-stills data can be run with min-pix 1 (plus a higher --spot-threshold), matching the CrystFEL peakfinder8 practice that trades noisier spots for a higher indexing rate. Default unchanged (2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 22:02:49 +02:00
leonarski_fandClaude Opus 4.8	2d7ddb1b67	ScaleOnTheFly: resolve the stills indexing ambiguity per image vs a reference ... Serial stills index each crystal in one of the merohedrally-equivalent hands at random, so the ambiguity must be broken per image, not globally as the rotation path (RotationScaleMerge/ChooseReindex) does. When an external reference MTZ is supplied, IndexAndRefine::ReferenceIntensities now builds the per-image scaler with resolve_ambiguity=true: for each image it picks the reindexing (identity or a twin law from ReindexAmbiguityOperators) whose partiality/Lorentz-corrected intensities correlate best with the reference, then reindexes that image's reflections before scaling. Off for the self-merge scaling pass (no trusted reference to break the tie). Holohedral crystals (no twin law, e.g. lysozyme P43212) are a no-op. Validated on OCP P3221 (twin law -x,-y,z) with a 7ZSJ reference: CCref 24->48% and CC1/2 24->64% at 30k images; CC1/2 96.9% / CCref 76% at 150k. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 19:46:48 +02:00
leonarski_fandClaude Opus 4.8	87778c6500	reader: treat negative total_flux as unknown/absent ... Serial-stills master files can store /entry/instrument/beam/total_flux as a -1.0 "unknown flux" sentinel (seen in the OCP dark dataset). DatasetSettings rejects values below 0, so opening such a file hard-failed with "Input parameter below min (Total flux)". Reset a negative flux to nullopt so it is treated as absent instead of aborting the read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 19:46:48 +02:00
leonarski_fandClaude Opus 4.8	3df9ad631b	docker: fix build_in_rocky9.sh shed-check for RHEL9's glib/selinux sources ... The old check flagged glib/selinux/pcre as failures, inheriting a rocky8-host assumption (glib is Qt-only, selinux is dbus-only) that does not hold on RHEL9: - libglib is pulled by the system harfbuzz/fontconfig/freetype text stack (glib-linked on RHEL9), not by Qt. QT_FEATURE_glib=OFF is honoured (Qt's enabled_features has no glib) but cannot drop the font stack's libglib. - libselinux/libpcre2-8 come from the krb5/GSSAPI stack (libgssapi_krb5 -> libselinux) that static curl links for auth -- kept for the reverse-proxy Kerberos plan, same bucket as libcrypto. So hard-fail only on the dbus/systemd tail we actually remove (libdbus/ libsystemd/zstd/lz4/cap/gcrypt/gpg-error), and just report glib/selinux as expected residuals with the reason. Verified in the rocky9 image: that tail is gone; the survivors trace to the font stack and krb5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 19:27:43 +02:00
leonarski_fandClaude Opus 4.8	d90d498cdb	docker: put the static libdbus on CMAKE_PREFIX_PATH for Qt6::DBus consumers ... The Qt build found dbus via -DDBus1_DIR, but a static Qt records the dependency and every consumer re-runs find_package(DBus1) through Qt's exported config (Qt6Gui -> Qt6DBus -> find_dependency(WrapDBus1)). Without the dbus prefix on the consumer's search path that fails: Could NOT find WrapDBus1 (missing: DBus1_LIBRARY DBus1_INCLUDE_DIR ...) Qt6DBus could not be found because dependency WrapDBus1 could not be found. Add /opt/dbus-<ver>-static to the image's CMAKE_PREFIX_PATH env so the viewer build (and the CI .deb/.rpm and tarball builds) resolve it. Verified against the rocky9 image: find_package(Qt6 ... DBus) fails with the old path and passes with dbus on the env CMAKE_PREFIX_PATH. Also make build_in_rocky9.sh self-sufficient on images built before this fix: it discovers /opt/dbus-*-static/lib/cmake/DBus1 and passes -DDBus1_DIR, so the viewer verification works without rebuilding the image first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 19:13:29 +02:00
leonarski_fandClaude Opus 4.8	b6c2d5848d	docker: build the static libdbus with -fPIC so Qt's PIE tools link it ... The static libdbus linked fine into libQt6DBus.a, but Qt links its D-Bus code generators (qdbusxml2cpp/qdbuscpp2xml) as position-independent executables (-fPIE -pie), and the default autotools dbus build produces non-PIC objects: ld: libdbus-1.a(...dbus-address.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIE Build libdbus with --with-pic and CFLAGS=-fPIC so its archive objects are position-independent and link into both the static Qt libs and the PIE tools. Reproduced (non-PIC -> R_X86_64_32 fail) and fixed (PIC -> PIE whole-archive link passes) in a rocky8 container before applying. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 15:52:50 +02:00
leonarski_fandClaude Opus 4.8	254bd82d52	docker: add parallel image-build + in-container viewer-verify scripts ... build_images.sh builds (and optionally pushes) the four build-environment images as gitea.psi.ch/leonarski_f/jfjoch_<variant>:<TAG>. Runs a JOBS-capped parallel job pool (default 2; each build internally runs make -j$(nproc), so a low cap avoids CPU/RAM thrash while still overlapping the network-bound base pull / install / download phases), streams each build to docker/build-logs/<variant>-<TAG>.log, prints an OK/FAIL summary, pushes only the ones that succeeded, and exits non-zero on any failure. Context is the tiny per-variant dir (no COPY in the Dockerfiles), so the repo is never sent to the daemon. build-logs/ is already covered by the root .gitignore build*/ rule. build_in_rocky9.sh builds the viewer (JFJOCH_VIEWER_ONLY) inside a chosen variant's image and ldd-checks that the dbus/systemd/glib/selinux tail is gone -- the quick verification for the static-libdbus + glib-off changes. The repo is mounted but the build tree lives in the container's /tmp, so nothing root-owned lands in the working copy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 15:43:18 +02:00
leonarski_fandClaude Opus 4.8	5baf2a2cac	docker: static libdbus + disable Qt glib to shed the viewer's .so tail ... The jfjoch_viewer runtime pulled a large dynamic .so tail that traces to two Qt-side integrations, neither of which the viewer actually needs: - Qt6::DBus -> system libdbus-1.so -> libsystemd (libzstd/liblz4/libcap/ libgcrypt/libgpg-error) + libselinux (libpcre2). The viewer is a pure session-bus client (registers ch.psi.jfjoch_viewer + exports an adaptor for single-instance/remote-control), so the daemon-side systemd/selinux features are irrelevant. Build a static libdbus from source with those integrations disabled and link it into the viewer, the same from-source-static pattern already used for OpenSSL. Swap the dbus dev package for expat (dbus's configure-time dep; the client libdbus-1 links neither expat nor systemd) so nothing pulls the system shared dbus back in. Qt discovers dbus through find_package(DBus1) (its FindWrapDBus1.cmake), NOT pkg-config -- so point it at our build with -DDBus1_DIR=<prefix>/lib/cmake/ DBus1 (the CMake package config dbus installs, which imports libdbus-1.a) and force QT_FEATURE_dbus_linked=ON so QtDBus links the archive instead of dlopen'ing a (now nonexistent) libdbus-1.so.3 at runtime. - Qt's glib event-dispatcher -> libglib-2.0/libgthread/libpcre. Nothing in the viewer (or any lib it links) drives a GLib main context, so switch Qt to its own QEventDispatcherUNIX via QT_FEATURE_glib=OFF and drop the glib dev package. Applied identically across rocky8, rocky9, ubuntu2204, ubuntu2404. Note: system libcrypto still appears at runtime, but it is dragged in by the dynamic system krb5/GSSAPI (built against OpenSSL on RHEL/Rocky), which we keep dynamic for the planned httpd-reverse-proxy Kerberos auth -- not from Qt or curl, which both statically link the from-source OpenSSL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 15:35:43 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.159	2026-07-13 13:54:03 +02:00