14 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	dfc6703f4a	jfjoch_process: don't double-smooth per-frame G on the reference path ... SmoothImageScaleG rewrites the partials in place (image_scale_corr and image_scale_g). On the no-reference path that is harmless: each scaling pass recomputes G from scratch via ScaleAllImages, so smoothing always runs on freshly-refined G. On the reference path the scaling loop is skipped, so G is computed once and stays; running scale_and_merge twice (P1 then the adopted space group) smoothed the already-smoothed G a second time, compounding into a ~2x wider effective kernel than the configured --smooth-g and biasing the merged intensities. Smooth only on the first pass of the reference path (G is unchanged afterwards, and the smoothed partials persist into the second pass's combine3D). The no-reference path is unchanged. Verified on lyso (600 frames, -P rot3d -z ref.mtz -M): the reference run now logs the smoothing once instead of twice, and the merged MTZ changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 14:24:05 +02:00
leonarski_fandClaude Opus 4.8	35607057d9	CC1/2 merge: assign half-sets by deterministic per-image hash ... MergeOnTheFly::AddImage picked each image's CC1/2 half from a shared mt19937 drawn in call order (and before Mask), so the split depended on iteration/thread order and on how many images were masked. The class is mutex-guarded for concurrent "on-the-fly" use, so any parallel merge would make CC1/2 non-reproducible - a latent race. Assign the half as a splitmix64 hash of the image's stable index instead, computed after Mask. The split is now reproducible run-to-run, independent of AddImage order, parallel-safe, and decoupled from masking. Callers pass the outcome's vector index as the image id. Verified: lyso_ref two-pass -M -P rot3d gives identical CC1/2 across runs (overall 99.6%, P41212); hash split is balanced ~50/50. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 07:25:34 +02:00
leonarski_fandClaude Opus 4.8	a773aaa6a9	WriteReflections: write merging statistics + ISa to the mmCIF output ... The .cif (--scaling-output cif) now carries the per-shell and overall merging statistics in the standard _reflns / _reflns_shell categories (resolution, redundancy, completeness, <I/sigma>, R-rim/R-meas, CC1/2) plus the Diederichs asymptotic I/sigma (ISa) as a free-text _reflns.pdbx_diffrn_ISa item (no standard CIF tag exists). The MergeStatistics and the ISa string are threaded through WriteReflections to the mmCIF writer; jfjoch_process and jfjoch_scale pass them. Values match the text statistics table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 22:29:45 +02:00
leonarski_fandClaude Opus 4.8	a74b8767de	ice rings: add --detect-ice-rings and exclude ice-ring reflections from scaling ... jfjoch_process gains --detect-ice-rings, which (a) activates the existing spot-finder ice flagging (ice spots de-prioritised in indexing) and (b) drops reflections sitting on a hexagonal-ice powder ring from scaling/combine/merge/stats, via a new shared IsOnIceRing() helper over ICE_RING_RES_A using the spot-finder's q half-width. Their integrated intensity is contaminated by the strong, variable ice background, so leaving them in mis-scales the whole frame and inflates the error model. On EP0117 (ice-ring crystal): de-novo space-group determination recovers from P1 to P2 and CC1/2 improves (31->37%). Off by default; a no-op without the flag. This is the first, non-controversial step - the residual gap needs ice-aware background/integration (follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:40:51 +02:00
leonarski_fandClaude Opus 4.8	0783af247c	scaling: express rot3d smooth-g as a rotation range in degrees (XDS DELPHI-like) ... --smooth-g now takes a rotation range in degrees (default 5) instead of a frame count, converted to an odd frame window from the oscillation step inside JFJochProcess. This makes the per-frame scale-G smoothing physical and independent of the frame slicing. (Note: G-smoothing is not the cure for the low-symmetry stochastic collapse - the per-image scale Cauchy loss is - but a degree-based window is the correct parameterization regardless.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:00:53 +02:00
leonarski_fandClaude Opus 4.8	e45a1577d6	jfjoch_process: optional absorption surface for rot3d scaling (--absorption) ... Adds an opt-in smooth absorption correction for rotation scaling. After the rot3d fulls are scaled, --absorption[=num] fits a multiplicative surface A(s1_crystal) - a degree<=4 monomial basis (real spherical harmonics up to l=4, as XDS/DIALS) of the diffracted-beam direction in the crystal/goniometer frame, by ridge-regularized log-linear least-squares of I_scaled/I_ref weighted by (I/sigma)^2, over num iterations (default 3); the surface divides image_scale_corr and the fulls are re-merged. Off by default and a no-op without rot3d. On the test panel (~13 keV, thin crystals) it is metric-neutral - fitted rms(log A) ~3-4%, ISa/CC1/2 unchanged - because absorption is negligible there and the per-frame scale G(phi) already absorbs the angular part. It is kept as a lever for low-energy data (e.g. 6 keV) where absorption becomes significant. Stored as ScalingSettings::absorption_iter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 15:23:01 +02:00
leonarski_fandClaude Opus 4.8	77f1ed2566	jfjoch_process: default scale-fulls for rot3d; skip _process.h5 when merging ... Two offline-processing ergonomics changes. scale-fulls is now ON by default for -P rot3d (it refits the per-frame scale on the combined fulls and lifts ISa substantially, e.g. HEWL rot3d 7.0 -> 16.4). --scale-fulls stays as the explicit opt-in for non-rot3d order; new --no-scale-fulls disables it for rot3d. (scale_fulls is now an optional<bool> defaulting to combine_3d.) Note: on low-completeness data the Unity-reference refit can cost a little CC1/2 (endothiapepsin ~70% complete: -5% in a mid shell); pair with --reject-outliers for the full low-symmetry benefit. When merging (-M), the merged reflections (.mtz/.cif) are the wanted output, so the large per-image _process.h5 is no longer written by default - it routinely ran to hundreds of MB. Pass --write-process-h5 to also emit it. Without merging the _process.h5 is the only output and is always written. Implemented with a ProcessConfig.write_process_h5 flag gating the FileWriter; reflection and image.dat writing are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:50:18 +02:00
leonarski_fandClaude Opus 4.8	99b923e727	scaling: smooth per-frame scale G across frames before the rot3d combine ... ScaleOnTheFly fits each frame's scale G independently with no neighbour coupling, so the few partials of one rocking event are weight-summed in combine3D on inconsistent scales - jitter that never enters the full's counting sigma and instead surfaces as scatter between symmetry mates, inflating the error-model b (low ISa). A centered moving average of log(G) over a small frame window (default 9, on for rot3d) removes it, mirroring XDS's smooth scaling. Complementary to --scale-fulls (which rescales between fulls, after the combine): smoothing fixes within-event scale, scale-fulls fixes between-full. On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa 11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest). Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS. --smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a diagnostic that fixes the scaling mosaicity for sweeps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	02477f1ce4	space group: determine centering from absences; jfjoch_process re-indexes from scratch ... Fixes wrong space-group assignment on cubic insulin (true I2_13, a=77.6). SearchSpaceGroup: drop the lattice-centering gate (and the now-unused lattice_centering option). The indexer often returns the conventional cell, whose geometry hides I/F/C centering - that information lives only in the systematic absences. Stage B now tests every centering of the point group and confirms it from the data, so an indexer-reported 'P' no longer excludes I2_13/I23. (I23 and I2_13 remain indistinguishable by absences and are reported as such.) jfjoch_process: discard any unit cell stored in the input HDF5 by default so the cell is re-determined from scratch. A stale/wrong stored cell otherwise resolves the indexing algorithm to FFBIDX, which trusts that cell and locks onto the wrong lattice (Ins_I_3 went from 2.7% -> 76% indexing). A user-supplied -C cell still applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 16:39:03 +02:00
leonarski_fandClaude Opus 4.8	107bcae0f0	merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2 ... RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned *median* of the squared symmetry-mate deviations, chosen for robustness. But a single deviation squared over its variance is chi-square(1)-distributed, whose median is only 0.4549 of its mean, so the fit was calibrating the variances to 0.4549x their true value: the merged sigmas came out ~1.48x too small and the achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally well-behaved (flat chi^2 across resolution) but globally over-confident, which inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for downstream weighting / French-Wilson. Divide the binned median by the chi-square(1) median (0.4549) to recover an unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the median while targeting reduced chi^2 = 1. Also compute the achieved median reduced chi^2 (same normalization) and report it on the "Error model" line so mis-calibration can no longer drift silently. Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2 2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common constant); only the absolute sigma calibration is corrected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 15:57:42 +02:00
leonarski_fandClaude Opus 4.8	ed98775ba3	rot3d: capture-aware sigma for under-captured fulls + expose per-image scale ... The rot3d combine over-extrapolates fulls reconstructed from only a fraction f of their rocking curve (min_partiality admits f as low as 0.02). Against XDS on HEWL these low-capture fulls are systematically biased HIGH (+15% at f=0.8 to +100% at f=0.3), and the bias - not random scatter - is the strong-reflection floor that hurts anomalous accuracy. --capture-uncertainty <coeff> (default 0 = off, baseline bit-identical) adds a systematic uncertainty ~coeff*(1-f)*I to each full's sigma, so the merge down-weights the over-extrapolated fulls and the error model treats their scatter as expected. Unlike outlier rejection (which trades accuracy for CC1/2), this fixes a real bias, so accuracy improves: at coeff=1.0 the anomalous peak height vs XDS rises CL_CL +16%, SD_MET/SG_CYS +5-6%, ISa 10.7->11.0. Rotation- only (no-op for stills, which never combine). Also expose the per-image scale offline: Combine3D now carries the first-pass per-image scale metadata (G, B, mosaicity, wedge, CC) forward instead of dropping it, and jfjoch_process -M writes <prefix>_image.dat from it (the offline self-scaling result was otherwise unobservable - process.h5's per-image arrays are only filled on the online path). This enabled the XDS DECAY comparison (jfjoch G tracks XDS, r=0.93). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 19:41:20 +02:00
leonarski_fandClaude Opus 4.8	786af96b3b	SearchSpaceGroup: POINTLESS-style rewrite, pipeline integration, twinning test ... Space-group search (image_analysis/scale_merge/SearchSpaceGroup): - Two-stage POINTLESS-style determination. Stage A scores each distinct rotation operator once (was once per candidate space group, ~34x faster on lysozyme: ~26s -> <1s) and picks the largest point group all of whose operators confirm. Stage B picks the maximal space group whose predicted absences are confirmed weak, fixing the prototype's default to the symmorphic group (it returned P422 instead of P4(3)2(1)2). Enantiomorphic / origin-ambiguous pairs (P4(1) vs P4(3), I222 vs I2(1)2(1)2(1)) are reported as indistinguishable. - Constrain candidates to subgroups of the lattice (metric) holohedry and weigh centering only P-vs-metric, fed from rotation indexing's LatticeSearch result. Integration / pipeline: - With no user-fixed space group, predict in P (IndexAndRefine) so the centering-absent reflections are integrated and the search can confirm/deny centering (catching pseudo-centering / a missed superstructure) instead of trusting the metric; a user-fixed group still rejects absences in integration. - JFJochProcess: scale+merge in P1 -> determine the space group -> set it and re-scale+merge in it (statistics then come out in the right symmetry) -> write it to /entry/sample/space_group_number (new EndMessage.space_group_number, preferred by NXmx::Sample). jfjoch_scale no longer searches; it consumes the file's space group (and no longer clobbers it with an empty -S). Twinning (new image_analysis/scale_merge/TwinningAnalysis): Padilla-Yeates L-test (<|L|>, <L^2>; acentric-only, positive intensities so L is bounded) plus a shell-normalised <I^2>/<I>^2 second moment and a twin-fraction estimate. Reported after the final merge in jfjoch_process and jfjoch_scale, and surfaced in the jfjoch_viewer merge-statistics window with a red outline when twinning is suspected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 16:11:28 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00