30 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	ddddfb6ffc	bragg_integration: GPU box + profile-fit integrator (standalone engine) ... Reimplement BraggIntegrate2D (box sum) and ProfileIntegrate2D (Kabsch profile fit) under one roof as a base + CPU + GPU engine, mirroring the AzIntEngine / ROIIntegration pattern. Reads the preprocessed int32 ImagePreprocessorBuffer (masked=INT32_MIN, saturated=INT32_MAX), the same buffer AzIntEngineGPU/ROIIntegrationGPU consume. The CUDA engine runs one block per reflection with shared-memory reductions across six kernels (reset, mask, box-sum, profile learning, profile build, Kabsch fit); the resolution shell is computed inline. The learning/fit hot path is single precision (FP64 is throttled on consumer GPUs; reproduces the double CPU path to ~1e-4). Collapsing the per-frame CUDA API calls into one reset kernel keeps launch-latency overhead low. Standalone for now: NOT wired into IndexAndRefine. See BRAGG_INTEGRATION_ENGINE.md for the design and the binding steps. BraggIntegrationEngineGPUTest checks GPU == CPU across all three modes (box/gaussian/empirical) within numeric tolerance, plus a [bragg_bench] perf sweep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 20:59:45 +02:00
leonarski_fandClaude Opus 4.8	ed9f6ac9eb	integration: remove --reciprocal-profile (proven per-frame dead-end) ... An 11-crystal mosaicity-stratified re-test (/data/rotation_test, off vs on vs a de-contaminated variant, plus a per-frame dump of the fitted widths) showed the dial is net-negative and cannot work in the per-frame paradigm: - The C|q|^2 mosaicity term - the whole point - is unfittable per-frame: the fitted curvature a2 comes out ~0 (often negative) on every crystal, with zero correlation to the XDS mosaicity (0.09..0.42 deg). Strong spots sit at low q where eta^2 q^2 is invisible; the curvature only appears at high q where there are ~0 strong spots. The law degenerates to a straight line. - With a2~0 the high-res width becomes a blind 1/cos^2(2theta) extrapolation, 2-4x wider than per-shell. The per-shell path's high-res "starvation" (flat narrow fallback) is accidentally correct: weak, crowded high-res spots want a narrow aperture, not the true wide spot shape. - The over-wide profile pulls background into weak spots -> R-meas rises, CC1/2 drops in reliable high-multiplicity shells (pding4_001, pding4_003, MyoB, EcwtCQ066). A cap at the widest well-sampled per-shell width recovers the regression, confirming over-widening is the harm. No crystal reliably wins; the apparent overall-CC gains were all in noise shells (mult 2-3, CC<20%). Delete the CLI flag, the BraggIntegrationSettings::reciprocal_profile setting, and the per-frame fit block. Default (per-shell) integration is byte-identical. NEXTGEN_INTEGRATOR.md records the finding as a dead-end for posterity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 11:39:09 +02:00
leonarski_fandClaude Opus 4.8	23d58b6202	integration: robust, well-conditioned per-frame --reciprocal-profile fit ... Fit the reciprocal tangential-width model y(q)=a0+a1*t+a2*t^2 in a centered, standardized variable t=(q-qbar)/qscale instead of the raw {1,q,q^2} monomials: the raw normal matrix went near-singular when the strong spots span a narrow q-range (small cell / sparse still), letting tiny per-frame jitter swing the curvature into a wild over-wide profile. Adds IRLS (Huber) robustness, a ridge on the curvature (sharp-crystal prior), and clamps the applied width to the fitted q-range (no extrapolation). Stays strictly per-frame (no dataset pooling), so it works online and for stills. Neutral on rotation data (cytC high-res CC1/2 win preserved 66.8 vs 65.6%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:01:05 +02:00
leonarski_fandClaude Opus 4.8	3aec235f67	integration: add --reciprocal-profile dial (global reciprocal-space width) ... The per-shell profile width is learned in pixels, so it varies ~4x with resolution (mostly the geometric projection of a near-constant reciprocal-space relrod) and must be binned per shell -> it starves at high resolution / on sparse data. The new --reciprocal-profile flag instead learns ONE global width in reciprocal space, sigma2_q,tan = A + B|q| + C|q|^2: the Jacobian g_tan=cos(2theta) removes the geometric projection, and C|q|^2 is the crystal mosaicity relrod (variance ~(eta|q|)^2). Applied per reflection as sigma2_tan,px = (A + B|q| + C|q|^2)/g_tan^2 (B,C clamped >=0; quadratic->linear->constant fallback). Off by default. On the sharp HEWL test crystal (mosaicity 0.091deg, so C fits to ~0 and it reduces to the validated linear form) it is metric-neutral: ISa 16.2->16.3, anomalous 0.92x unchanged, CCref band 90.0->89.9, CC1/2 a touch lower (per-shell isn't starved at 23k spots/shell, and a global fit is less flexible). So: simpler + more transferable at a small CC1/2 cost, ISa/anomalous held. Its payoff is on MOSAIC crystals (large C|q|^2), where per-shell starves on the wide weak high-res spots and 6 shells are too coarse; both lyso test crystals are sharp, so it ships as a dial to try on mosaic data elsewhere. A separate radial relrod fit was tried and dropped (no gain). See NEXTGEN_INTEGRATOR.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 14:26:08 +02:00
leonarski_fandClaude Opus 4.8	5ba5fe8ad1	integration: centre the profile on the predicted sub-pixel position ... The profile was learned and applied on the integer pixel round(predicted), so a shared profile sits up to 0.5 px off the true spot (and stacking spots with random sub-pixel offsets broadens the learned profile). Build the Gaussian per reflection instead, centred on the predicted sub-pixel offset -- noise-free geometry, unlike the observed centroid, which hurt -- and elongated radially as before. HEWL rotation @1.0A: ISa 15.7->16.2, CCref band 89.9->90.0, CCxds 94.8->95.0 (high-res 1.00A CCref 66.0->66.9); sharp serial stills 1.68A CC1/2 61.6->62.5; anomalous S peak 0.92x XDS (no accuracy traded). De-broadening the learned width by the 1/12 px^2/axis integer-binning floor was tested and rejected (it over-narrows). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 07:34:06 +02:00
leonarski_fandClaude Opus 4.8	26901da42f	integration: de-contaminate the profile width, add a radial parallax ellipse ... The profile-fit width came from the full 13x13 second moment, which runs 3-8x wider than the true spot: neighbour reflections leak into the (unmasked) learning grid -- catastrophic at low res where spots crowd the beam -- and the far corners (lever arm dx^2+dy^2 up to 72) add rectified background noise. Splitting the spot moment into radial vs tangential shows the tangential width is isotropic (mosaicity/divergence) while the radial excess is pure sensor parallax ~tan^2(2th). - Measure the width over the signal disk r1 on the monochromatic path (inherently excludes neighbours, caps the radial tail); keep the generous full-grid width on the broadband/stills path (sparse spots, the centroid-undersampling floor). - Extend the (was bandwidth-only) radial ellipse with an analytic, material-aware parallax term c_par*tan^2(2theta), c_par = Var(z)/pixel^2 from sensor thickness + material + energy (parallax_var_px2; Si and CdTe), plus a fixed weak-spot capture term on the monochromatic path only. HEWL rotation @1.0A: ISa 13.5->15.7, CC1/2 1.12A 91.3->95.9 / 1.05A 83.0->85.2, external CCref band 88.1->89.9, CCxds 93.4->94.8, R-meas 9.4->8.7; low/mid flat. Sharp serial stills gain slightly from parallax; broadband stills neutral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 00:54:59 +02:00
leonarski_fandClaude Opus 4.8	eccd10a0a7	integration: neighbour-mask the profile-fit background ring, widen r3 8->10 ... ProfileIntegrate2D::BoxSum now excludes every predicted reflection's r2 disk from the r2..r3 background annulus (mirroring BraggIntegrate2D), so a neighbour Bragg peak can no longer bias a reflection's background high and over-subtract. With neighbours excluded the annulus can safely widen, so the default r_3 goes 8 -> 10 (more background pixels, lower-variance estimate). Measured (rotation lyso @1.0 A, external CCref/CCxds vs XDS): 1.05 A CCref +1.3 / CCxds +1.3, R-meas -5 pts, low-res R-meas unchanged. Serial (Jet8 @0.0002 bandwidth): 1.68 A CC1/2 +3.9 / CCref +1.1, 1.58 A +4.0 / +1.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:14 +02:00
leonarski_fandClaude Opus 4.8	bbd888dcc3	integration: gate the background-ring sigma-clip to stills (bandwidth>0) ... The 76e88b0f sigma-clip (reject ring pixels above mean+3*sqrt(mean)) de-biases bandwidth-streaked high-resolution stills, but it ran on all data. On rotation (no streaks) it clips legitimate high background pixels and biases the mean low, slightly inflating weak intensities and hurting the anomalous signal. Gate it to bandwidth>0 (stills), matching how the 814dff34 radial-profile change is already gated. Rotation lysozyme (self-scaled, smooth-g, -A): anomalous S-peak 0.84x -> 0.86x of XDS (SD_MET 11.71 -> 11.94, CL_CL 1.28x -> 1.25x), ISa unchanged. Stills (bandwidth set) are byte-identical (clip still applies). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	814dff34cb	integration: radially-elongated profile for bandwidth-streaked high-res stills ... With a finite energy bandwidth each reflection is smeared RADIALLY by sigma_bw = bandwidth_sigma * R_px (R_px = distance from the beam centre, so large at high resolution): high-resolution spots become radial streaks. The isotropic per-shell Gaussian both mis-weights them and clips the streak tail on the fixed profile grid, losing intensity (biased low, noisy). When a bandwidth is set, fit each reflection with a per-reflection Gaussian elongated only along its radial direction - sigma^2_radial = sigma^2_intrinsic + sigma_bw^2, sigma^2_tangential = sigma^2_intrinsic - on a grid grown to hold the streak. Unlike an isotropic widening this adds no tangential background. It only engages where the smear exceeds the intrinsic spot (high resolution); low/mid resolution and monochromatic data (bandwidth 0, e.g. rotation) are untouched. On the HEWL serial-stills jet (with the background sigma-clip) this lifts the overall CC-vs-reference 52 -> 55% and the high-resolution I/sig (1.7 A 0.5 -> 1.4), recovering the 2.0-2.5 A band, with CC1/2 preserved (the per-shot noise the wider region adds is averaged out by the high serial multiplicity). Rotation ISa 19.1 unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 12:01:56 +02:00
leonarski_fandClaude Opus 4.8	76e88b0fca	integration: sigma-clip the background ring to de-bias high-resolution stills ... Weak high-resolution still reflections were systematically over-subtracted: a bandwidth-streaked high-res spot (or a neighbour) leaks into the r2-r3 background annulus and biases its mean high, so the subtracted background is too large and the merged high-resolution intensities go negative (seen as reproducibly negative <I/sig> at 100% completeness and high multiplicity past ~1.9 A). Add one high-outlier sigma-clip pass to the box-sum background (reject ring pixels above mean + 3*sqrt(mean), recompute) so the contamination no longer inflates it. A clean Poisson background is essentially unchanged (~0.1% exceed the cut). On the HEWL serial-stills jet this de-biases the high-res band - <I/sig> 2.03 A 0.9 -> 1.6, 1.79 A -0.1 -> +0.7 - extends the usable resolution ~2.2 -> ~2.0 A and improves overall R-meas 130 -> 124%, with CC1/2 and CC-vs-reference neutral. The rotation crystal is unchanged (ISa 19.1), its clean backgrounds being barely clipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 11:50:31 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	bb9f5c715f	v1.0.0-rc.135 (#44) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4) * jfjoch_viewer: Improve image render performance Reviewed-on: #44 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-16 11:59:59 +02:00
leonarski_f	928789a67c	v1.0.0-rc.130 (#37) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Rotation indexer has two retries if failes * jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot) * jfjoch_broker: Integration calculates background mask based on R2 radius * jfjoch_process: HDF5 files are not saved by default Reviewed-on: #37	2026-03-06 14:38:56 +01:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00
leonarski_f	27496b8207	v1.0.0-rc.122 (#29) ... This is an UNSTABLE release. * jfjoch_broker: Add thresholding to prefer shorter vectors after FFT * jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress) * jfjoch_viewer: Display file opening errors * jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available Reviewed-on: #29 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-16 15:27:40 +01:00
leonarski_f	f44c6520a8	v1.0.0-rc.107 (#12) ... This is an UNSTABLE release. * jfjoch_viewer: Minor polishing of new functionality * jfjoch_broker: User NaN for empty azimuthal bins Reviewed-on: #12 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-24 07:07:32 +01:00
leonarski_f	fb9462f67b	v1.0.0-rc.92	2025-10-23 09:05:11 +02:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00
leonarski_f	14f5051692	1.0.0-rc.89	2025-10-01 22:54:40 +02:00
leonarski_f	99b7dc07f7	1.0.0-rc.87	2025-09-30 20:43:53 +02:00
leonarski_f	5d9d2de4a4	v1.0.0-rc.81	2025-09-21 19:27:51 +02:00
leonarski_f	20cd8dc4ac	v1.0.0-rc.76	2025-09-11 14:37:16 +02:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	fa95858008	v1.0.0-rc.69	2025-08-17 21:21:20 +02:00
leonarski_f	a32cec3404	v1.0.0-rc.67	2025-08-11 11:13:59 +02:00
leonarski_f	0d1b388f4d	v1.0.0-rc.50	2025-06-20 09:52:26 +02:00