Commit Graph
5 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 4.8 99b923e727 scaling: smooth per-frame scale G across frames before the rot3d combine
ScaleOnTheFly fits each frame's scale G independently with no neighbour
coupling, so the few partials of one rocking event are weight-summed in
combine3D on inconsistent scales - jitter that never enters the full's
counting sigma and instead surfaces as scatter between symmetry mates,
inflating the error-model b (low ISa). A centered moving average of
log(G) over a small frame window (default 9, on for rot3d) removes it,
mirroring XDS's smooth scaling. Complementary to --scale-fulls (which
rescales between fulls, after the combine): smoothing fixes within-event
scale, scale-fulls fixes between-full.

On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa
11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest).
Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS.

--smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a
diagnostic that fixes the scaling mosaicity for sweeps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8 ed98775ba3 rot3d: capture-aware sigma for under-captured fulls + expose per-image scale
The rot3d combine over-extrapolates fulls reconstructed from only a fraction f
of their rocking curve (min_partiality admits f as low as 0.02). Against XDS on
HEWL these low-capture fulls are systematically biased HIGH (+15% at f=0.8 to
+100% at f=0.3), and the bias - not random scatter - is the strong-reflection
floor that hurts anomalous accuracy.

--capture-uncertainty <coeff> (default 0 = off, baseline bit-identical) adds a
systematic uncertainty ~coeff*(1-f)*I to each full's sigma, so the merge
down-weights the over-extrapolated fulls and the error model treats their
scatter as expected. Unlike outlier rejection (which trades accuracy for CC1/2),
this fixes a real bias, so accuracy improves: at coeff=1.0 the anomalous peak
height vs XDS rises CL_CL +16%, SD_MET/SG_CYS +5-6%, ISa 10.7->11.0. Rotation-
only (no-op for stills, which never combine).

Also expose the per-image scale offline: Combine3D now carries the first-pass
per-image scale metadata (G, B, mosaicity, wedge, CC) forward instead of
dropping it, and jfjoch_process -M writes <prefix>_image.dat from it (the
offline self-scaling result was otherwise unobservable - process.h5's per-image
arrays are only filled on the online path). This enabled the XDS DECAY
comparison (jfjoch G tracks XDS, r=0.93).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 19:41:20 +02:00
leonarski_f 54c667190f v1.0.0-rc.155 (#65)
Build Packages / Unit tests (push) Successful in 1h26m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m39s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m55s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8) (push) Successful in 12m28s
Build Packages / build:rpm (rocky9) (push) Successful in 13m20s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m43s
Build Packages / DIALS test (push) Successful in 14m21s
Build Packages / XDS test (durin plugin) (push) Successful in 7m48s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m31s
Build Packages / Generate python client (push) Successful in 15s
Build Packages / Build documentation (push) Successful in 53s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
* jfjoch_viewer: Some graphical improvements.
* jfjoch_viewer: Simplify und unify data analysis settings.
* jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.

Reviewed-on: #65
2026-06-25 22:01:48 +02:00
leonarski_f 75e401f0e5 v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

Reviewed-on: #63
2026-06-23 20:29:49 +02:00
leonarski_f fc68a9baed v1.0.0-rc.146 (#56)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m34s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m16s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m49s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m15s
Build Packages / XDS test (durin plugin) (push) Successful in 7m16s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky9) (push) Successful in 10m12s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 47s
Build Packages / DIALS test (push) Successful in 10m18s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 5m46s
Build Packages / build:rpm (rocky8) (push) Successful in 1h41m2s
Build Packages / XDS test (neggia plugin) (push) Successful in 1h59m18s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.

WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

Reviewed-on: #56
2026-05-28 18:48:35 +02:00