7 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Fable 5	6ffb16f142	merge: mask ice rings whose CC1/2 collapses below their resolution neighbours ... An ice-contaminated ring decorrelates its reflections, so its merged half-set CC1/2 falls well below the Bragg trend of its resolution shoulders (e.g. EP_cs_01-17 2.25A ring CC1/2 0.12 vs shoulders 0.89). After the final merge, compute per-hexagonal-ice-ring CC1/2 in the ring band (+/- ice width in q=2pi/d) against the shoulders either side, mask any ring more than 0.05 below its shoulders (reliable shoulder CC1/2 > 0.5, >=20 reflections each) and re-merge without them. Weak/absent rings track their neighbours and stay, so clean crystals are untouched. Data-driven and robust on /data/rotation_test: 16/18 crystals mask nothing and are unchanged (every clean crystal, plus icy-but-fine EP_cs_02-10); EP_cs_01-17 masks 6 rings, CC1/2 6.9% -> 28%; CQ066/pding4_003 mask one marginal ring each. A well-scaled CC1/2 test, replacing the fragile per-image azimuthal cutoffs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:06:39 +02:00
leonarski_fandClaude Fable 5	8c084021e6	ice rings: exclude from scaling, keep in the final merge (recover completeness) ... --detect-ice-rings previously dropped ice-ring reflections outright before scaling, which also removed them from the merge and gutted low/mid-resolution completeness on crystals that merge them fine (e.g. lysoC 80% -> the ice bands show as jagged shell completeness). Instead flag them (new Reflection.on_ice_ring, set in JFJochProcess, carried through the 3D combine) and exclude the flag only where a model is fit: the per-image scale (ScaleOnTheFly::Accept + the per-image CC), and - for the de-novo P1 pass - the space-group search and error model (MergeOnTheFly::ExcludeIceRings, on for for_search). The final in-symmetry merge and its statistics keep them, so completeness is preserved. /data/rotation_test battery vs the previous drop-from-merge behaviour: space group correct on all 18; completeness recovered broadly with CC1/2 and ISa held (cytC_2 82->99.7%, cytC_3 73->99.7%, InsI3 76->99.5%, lysoC 80->99.7%, MyoB 80->99.7%, InsH3 78->99%). Excluding ice from the P1 search merge is what keeps the space group correct: without it InsI3 flipped I23->P1 and EP_cs_01-17 P2->P1. Known limitation: on heavy-ice crystals (EP_cs_01-17) the strong ice is garbage and keeping it in the final merge collapses CC1/2 in the ice shells (91.7->6.9%). Distinguishing strong vs weak/absent rings per crystal needs data-driven, per-ring ice detection (azimuthal radial profile) - the planned next step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:29:37 +02:00
leonarski_fandClaude Opus 4.8	35607057d9	CC1/2 merge: assign half-sets by deterministic per-image hash ... MergeOnTheFly::AddImage picked each image's CC1/2 half from a shared mt19937 drawn in call order (and before Mask), so the split depended on iteration/thread order and on how many images were masked. The class is mutex-guarded for concurrent "on-the-fly" use, so any parallel merge would make CC1/2 non-reproducible - a latent race. Assign the half as a splitmix64 hash of the image's stable index instead, computed after Mask. The split is now reproducible run-to-run, independent of AddImage order, parallel-safe, and decoupled from masking. Callers pass the outcome's vector index as the image id. Verified: lyso_ref two-pass -M -P rot3d gives identical CC1/2 across runs (overall 99.6%, P41212); hash split is balanced ~50/50. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 07:25:34 +02:00
leonarski_fandClaude Opus 4.8	107bcae0f0	merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2 ... RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned *median* of the squared symmetry-mate deviations, chosen for robustness. But a single deviation squared over its variance is chi-square(1)-distributed, whose median is only 0.4549 of its mean, so the fit was calibrating the variances to 0.4549x their true value: the merged sigmas came out ~1.48x too small and the achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally well-behaved (flat chi^2 across resolution) but globally over-confident, which inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for downstream weighting / French-Wilson. Divide the binned median by the chi-square(1) median (0.4549) to recover an unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the median while targeting reduced chi^2 = 1. Also compute the achieved median reduced chi^2 (same normalization) and report it on the "Error model" line so mis-calibration can no longer drift silently. Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2 2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common constant); only the absolute sigma calibration is corrected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 15:57:42 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00