11 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	a74b8767de	ice rings: add --detect-ice-rings and exclude ice-ring reflections from scaling ... jfjoch_process gains --detect-ice-rings, which (a) activates the existing spot-finder ice flagging (ice spots de-prioritised in indexing) and (b) drops reflections sitting on a hexagonal-ice powder ring from scaling/combine/merge/stats, via a new shared IsOnIceRing() helper over ICE_RING_RES_A using the spot-finder's q half-width. Their integrated intensity is contaminated by the strong, variable ice background, so leaving them in mis-scales the whole frame and inflates the error model. On EP0117 (ice-ring crystal): de-novo space-group determination recovers from P1 to P2 and CC1/2 improves (31->37%). Off by default; a no-op without the flag. This is the first, non-controversial step - the residual gap needs ice-aware background/integration (follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:40:51 +02:00
leonarski_fandClaude Opus 4.8	0783af247c	scaling: express rot3d smooth-g as a rotation range in degrees (XDS DELPHI-like) ... --smooth-g now takes a rotation range in degrees (default 5) instead of a frame count, converted to an odd frame window from the oscillation step inside JFJochProcess. This makes the per-frame scale-G smoothing physical and independent of the frame slicing. (Note: G-smoothing is not the cure for the low-symmetry stochastic collapse - the per-image scale Cauchy loss is - but a degree-based window is the correct parameterization regardless.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:00:53 +02:00
leonarski_fandClaude Opus 4.8	e45a1577d6	jfjoch_process: optional absorption surface for rot3d scaling (--absorption) ... Adds an opt-in smooth absorption correction for rotation scaling. After the rot3d fulls are scaled, --absorption[=num] fits a multiplicative surface A(s1_crystal) - a degree<=4 monomial basis (real spherical harmonics up to l=4, as XDS/DIALS) of the diffracted-beam direction in the crystal/goniometer frame, by ridge-regularized log-linear least-squares of I_scaled/I_ref weighted by (I/sigma)^2, over num iterations (default 3); the surface divides image_scale_corr and the fulls are re-merged. Off by default and a no-op without rot3d. On the test panel (~13 keV, thin crystals) it is metric-neutral - fitted rms(log A) ~3-4%, ISa/CC1/2 unchanged - because absorption is negligible there and the per-frame scale G(phi) already absorbs the angular part. It is kept as a lever for low-energy data (e.g. 6 keV) where absorption becomes significant. Stored as ScalingSettings::absorption_iter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 15:23:01 +02:00
leonarski_fandClaude Opus 4.8	77f1ed2566	jfjoch_process: default scale-fulls for rot3d; skip _process.h5 when merging ... Two offline-processing ergonomics changes. scale-fulls is now ON by default for -P rot3d (it refits the per-frame scale on the combined fulls and lifts ISa substantially, e.g. HEWL rot3d 7.0 -> 16.4). --scale-fulls stays as the explicit opt-in for non-rot3d order; new --no-scale-fulls disables it for rot3d. (scale_fulls is now an optional<bool> defaulting to combine_3d.) Note: on low-completeness data the Unity-reference refit can cost a little CC1/2 (endothiapepsin ~70% complete: -5% in a mid shell); pair with --reject-outliers for the full low-symmetry benefit. When merging (-M), the merged reflections (.mtz/.cif) are the wanted output, so the large per-image _process.h5 is no longer written by default - it routinely ran to hundreds of MB. Pass --write-process-h5 to also emit it. Without merging the _process.h5 is the only output and is always written. Implemented with a ProcessConfig.write_process_h5 flag gating the FileWriter; reflection and image.dat writing are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:50:18 +02:00
leonarski_fandClaude Opus 4.8	99b923e727	scaling: smooth per-frame scale G across frames before the rot3d combine ... ScaleOnTheFly fits each frame's scale G independently with no neighbour coupling, so the few partials of one rocking event are weight-summed in combine3D on inconsistent scales - jitter that never enters the full's counting sigma and instead surfaces as scatter between symmetry mates, inflating the error-model b (low ISa). A centered moving average of log(G) over a small frame window (default 9, on for rot3d) removes it, mirroring XDS's smooth scaling. Complementary to --scale-fulls (which rescales between fulls, after the combine): smoothing fixes within-event scale, scale-fulls fixes between-full. On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa 11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest). Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS. --smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a diagnostic that fixes the scaling mosaicity for sweeps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	02477f1ce4	space group: determine centering from absences; jfjoch_process re-indexes from scratch ... Fixes wrong space-group assignment on cubic insulin (true I2_13, a=77.6). SearchSpaceGroup: drop the lattice-centering gate (and the now-unused lattice_centering option). The indexer often returns the conventional cell, whose geometry hides I/F/C centering - that information lives only in the systematic absences. Stage B now tests every centering of the point group and confirms it from the data, so an indexer-reported 'P' no longer excludes I2_13/I23. (I23 and I2_13 remain indistinguishable by absences and are reported as such.) jfjoch_process: discard any unit cell stored in the input HDF5 by default so the cell is re-determined from scratch. A stale/wrong stored cell otherwise resolves the indexing algorithm to FFBIDX, which trusts that cell and locks onto the wrong lattice (Ins_I_3 went from 2.7% -> 76% indexing). A user-supplied -C cell still applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 16:39:03 +02:00
leonarski_fandClaude Opus 4.8	107bcae0f0	merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2 ... RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned *median* of the squared symmetry-mate deviations, chosen for robustness. But a single deviation squared over its variance is chi-square(1)-distributed, whose median is only 0.4549 of its mean, so the fit was calibrating the variances to 0.4549x their true value: the merged sigmas came out ~1.48x too small and the achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally well-behaved (flat chi^2 across resolution) but globally over-confident, which inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for downstream weighting / French-Wilson. Divide the binned median by the chi-square(1) median (0.4549) to recover an unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the median while targeting reduced chi^2 = 1. Also compute the achieved median reduced chi^2 (same normalization) and report it on the "Error model" line so mis-calibration can no longer drift silently. Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2 2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common constant); only the absolute sigma calibration is corrected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 15:57:42 +02:00
leonarski_fandClaude Opus 4.8	ed98775ba3	rot3d: capture-aware sigma for under-captured fulls + expose per-image scale ... The rot3d combine over-extrapolates fulls reconstructed from only a fraction f of their rocking curve (min_partiality admits f as low as 0.02). Against XDS on HEWL these low-capture fulls are systematically biased HIGH (+15% at f=0.8 to +100% at f=0.3), and the bias - not random scatter - is the strong-reflection floor that hurts anomalous accuracy. --capture-uncertainty <coeff> (default 0 = off, baseline bit-identical) adds a systematic uncertainty ~coeff*(1-f)*I to each full's sigma, so the merge down-weights the over-extrapolated fulls and the error model treats their scatter as expected. Unlike outlier rejection (which trades accuracy for CC1/2), this fixes a real bias, so accuracy improves: at coeff=1.0 the anomalous peak height vs XDS rises CL_CL +16%, SD_MET/SG_CYS +5-6%, ISa 10.7->11.0. Rotation- only (no-op for stills, which never combine). Also expose the per-image scale offline: Combine3D now carries the first-pass per-image scale metadata (G, B, mosaicity, wedge, CC) forward instead of dropping it, and jfjoch_process -M writes <prefix>_image.dat from it (the offline self-scaling result was otherwise unobservable - process.h5's per-image arrays are only filled on the online path). This enabled the XDS DECAY comparison (jfjoch G tracks XDS, r=0.93). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 19:41:20 +02:00
leonarski_fandClaude Opus 4.8	786af96b3b	SearchSpaceGroup: POINTLESS-style rewrite, pipeline integration, twinning test ... Space-group search (image_analysis/scale_merge/SearchSpaceGroup): - Two-stage POINTLESS-style determination. Stage A scores each distinct rotation operator once (was once per candidate space group, ~34x faster on lysozyme: ~26s -> <1s) and picks the largest point group all of whose operators confirm. Stage B picks the maximal space group whose predicted absences are confirmed weak, fixing the prototype's default to the symmorphic group (it returned P422 instead of P4(3)2(1)2). Enantiomorphic / origin-ambiguous pairs (P4(1) vs P4(3), I222 vs I2(1)2(1)2(1)) are reported as indistinguishable. - Constrain candidates to subgroups of the lattice (metric) holohedry and weigh centering only P-vs-metric, fed from rotation indexing's LatticeSearch result. Integration / pipeline: - With no user-fixed space group, predict in P (IndexAndRefine) so the centering-absent reflections are integrated and the search can confirm/deny centering (catching pseudo-centering / a missed superstructure) instead of trusting the metric; a user-fixed group still rejects absences in integration. - JFJochProcess: scale+merge in P1 -> determine the space group -> set it and re-scale+merge in it (statistics then come out in the right symmetry) -> write it to /entry/sample/space_group_number (new EndMessage.space_group_number, preferred by NXmx::Sample). jfjoch_scale no longer searches; it consumes the file's space group (and no longer clobbers it with an empty -S). Twinning (new image_analysis/scale_merge/TwinningAnalysis): Padilla-Yeates L-test (<|L|>, <L^2>; acentric-only, positive intensities so L is bounded) plus a shell-normalised <I^2>/<I>^2 second moment and a twin-fraction estimate. Reported after the final merge in jfjoch_process and jfjoch_scale, and surfaced in the jfjoch_viewer merge-statistics window with a red outline when twinning is suspected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 16:11:28 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00