Commit Graph
5 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 4.8 35607057d9 CC1/2 merge: assign half-sets by deterministic per-image hash
Build Packages / build:windows:cuda (push) Failing after 4m30s
Build Packages / build:windows:nocuda (push) Successful in 10m0s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m43s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m46s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m53s
Build Packages / build:rpm (rocky8) (push) Successful in 14m54s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 15m10s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m34s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m37s
Build Packages / XDS test (durin plugin) (push) Successful in 8m5s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m50s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (neggia plugin) (push) Successful in 8m44s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m45s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m5s
Build Packages / build:rpm (rocky9) (push) Successful in 13m8s
Build Packages / DIALS test (push) Successful in 12m47s
Build Packages / Unit tests (push) Successful in 1h0m40s
MergeOnTheFly::AddImage picked each image's CC1/2 half from a shared
mt19937 drawn in call order (and before Mask), so the split depended on
iteration/thread order and on how many images were masked. The class is
mutex-guarded for concurrent "on-the-fly" use, so any parallel merge would
make CC1/2 non-reproducible - a latent race.

Assign the half as a splitmix64 hash of the image's stable index instead,
computed after Mask. The split is now reproducible run-to-run, independent
of AddImage order, parallel-safe, and decoupled from masking. Callers pass
the outcome's vector index as the image id.

Verified: lyso_ref two-pass -M -P rot3d gives identical CC1/2 across runs
(overall 99.6%, P41212); hash split is balanced ~50/50.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 07:25:34 +02:00
leonarski_fandClaude Opus 4.8 107bcae0f0 merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m26s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m34s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m59s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m17s
Build Packages / XDS test (durin plugin) (push) Successful in 7m40s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m46s
Build Packages / build:rpm (rocky8) (push) Successful in 12m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m11s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky9) (push) Successful in 13m2s
Build Packages / Build documentation (push) Successful in 42s
Build Packages / DIALS test (push) Successful in 13m38s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m9s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m44s
Build Packages / Unit tests (push) Successful in 57m53s
Build Packages / build:windows:nocuda (push) Successful in 16m26s
Build Packages / build:windows:cuda (push) Successful in 18m26s
RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned
*median* of the squared symmetry-mate deviations, chosen for robustness. But a
single deviation squared over its variance is chi-square(1)-distributed, whose
median is only 0.4549 of its mean, so the fit was calibrating the variances to
0.4549x their true value: the merged sigmas came out ~1.48x too small and the
achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally
well-behaved (flat chi^2 across resolution) but globally over-confident, which
inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for
downstream weighting / French-Wilson.

Divide the binned median by the chi-square(1) median (0.4549) to recover an
unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the
median while targeting reduced chi^2 = 1. Also compute the achieved median
reduced chi^2 (same normalization) and report it on the "Error model" line so
mis-calibration can no longer drift silently.

Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2
2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and
all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common
constant); only the absolute sigma calibration is corrected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:57:42 +02:00
leonarski_f 54c667190f v1.0.0-rc.155 (#65)
Build Packages / Unit tests (push) Successful in 1h26m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m39s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m55s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8) (push) Successful in 12m28s
Build Packages / build:rpm (rocky9) (push) Successful in 13m20s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m43s
Build Packages / DIALS test (push) Successful in 14m21s
Build Packages / XDS test (durin plugin) (push) Successful in 7m48s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m31s
Build Packages / Generate python client (push) Successful in 15s
Build Packages / Build documentation (push) Successful in 53s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
* jfjoch_viewer: Some graphical improvements.
* jfjoch_viewer: Simplify und unify data analysis settings.
* jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.

Reviewed-on: #65
2026-06-25 22:01:48 +02:00
leonarski_f 75e401f0e5 v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

Reviewed-on: #63
2026-06-23 20:29:49 +02:00
leonarski_f fc68a9baed v1.0.0-rc.146 (#56)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m34s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m16s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m49s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m15s
Build Packages / XDS test (durin plugin) (push) Successful in 7m16s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky9) (push) Successful in 10m12s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 47s
Build Packages / DIALS test (push) Successful in 10m18s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 5m46s
Build Packages / build:rpm (rocky8) (push) Successful in 1h41m2s
Build Packages / XDS test (neggia plugin) (push) Successful in 1h59m18s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.

WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

Reviewed-on: #56
2026-05-28 18:48:35 +02:00