8 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Fable 5	cdbd0bed5d	partiality: fix the rotation-prediction wedge, use the stored value in scaling ... BraggPredictionRot halves settings.wedge_deg to the +/- half-wedge of the partiality erf pair, but IndexAndRefine already passed GetWedge_deg()/2, so the two /2 compounded to a half-wedge of increment/4 - half the correct Kabsch value (increment/2, which ScaleOnTheFly's RotationPartiality already used). Pass the full increment so prediction's partiality matches scaling. With prediction correct, ScaleOnTheFly now uses the stored r.partiality directly (the value the reflection was integrated with) rather than recomputing the erf pair per reflection - recomputing only when scaling overrides the geometry (-w wedge refinement, --mosaicity, or a scaling wedge override). Output-neutral on the /data/rotation_test battery (SG/cell/completeness identical, ISa/CC1/2 within run noise on the stable crystals). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:57:16 +02:00
leonarski_fandClaude Fable 5	8c084021e6	ice rings: exclude from scaling, keep in the final merge (recover completeness) ... --detect-ice-rings previously dropped ice-ring reflections outright before scaling, which also removed them from the merge and gutted low/mid-resolution completeness on crystals that merge them fine (e.g. lysoC 80% -> the ice bands show as jagged shell completeness). Instead flag them (new Reflection.on_ice_ring, set in JFJochProcess, carried through the 3D combine) and exclude the flag only where a model is fit: the per-image scale (ScaleOnTheFly::Accept + the per-image CC), and - for the de-novo P1 pass - the space-group search and error model (MergeOnTheFly::ExcludeIceRings, on for for_search). The final in-symmetry merge and its statistics keep them, so completeness is preserved. /data/rotation_test battery vs the previous drop-from-merge behaviour: space group correct on all 18; completeness recovered broadly with CC1/2 and ISa held (cytC_2 82->99.7%, cytC_3 73->99.7%, InsI3 76->99.5%, lysoC 80->99.7%, MyoB 80->99.7%, InsH3 78->99%). Excluding ice from the P1 search merge is what keeps the space group correct: without it InsI3 flipped I23->P1 and EP_cs_01-17 P2->P1. Known limitation: on heavy-ice crystals (EP_cs_01-17) the strong ice is garbage and keeping it in the final merge collapses CC1/2 in the ice shells (91.7->6.9%). Distinguishing strong vs weak/absent rings per crystal needs data-driven, per-ring ice detection (azimuthal radial profile) - the planned next step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:29:37 +02:00
leonarski_fandClaude Fable 5	1ca5a69054	ScaleOnTheFly: solve the per-image scale by IRLS instead of Ceres ... In the default rotation (rot3d) path only G is refined - B is fixed, mosaicity is pinned and the wedge is not refined - so the predicted intensity G*coeff is linear in G and the robust (Cauchy) per-image scale is a 1-D M-estimate. Solve it directly by iteratively reweighted least squares (a few closed-form weighted ratios) instead of building a Ceres problem per image. Ceres is kept for the cases that are genuinely nonlinear: refining the B-factor or the rotation wedge. Same Cauchy objective as the Ceres path, but ~4x faster at scaling and ~30% faster overall on the /data/rotation_test battery, with space group, cell, ISa, completeness and CC1/2 matching across all 18 crystals (the two that look different, EP_cs_01-17 and EcwtAL500, are run-to-run unstable for both solvers). lyso_ref scaling 25.2->4.3s, cytC_2 15.2->2.6s, battery total 468->316s. Also drop the per-image G/B regularizers (gated by GetScalingRegularize, which nothing enables) and the now-unused RegularizationResidual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 13:55:04 +02:00
leonarski_fandClaude Opus 4.8	8d18c635ff	scaling: robust (Cauchy) loss on the per-image scale fit ... The per-image G/B/mosaicity fit used a plain L2 loss, so a few outlier reflections occasionally dragged a frame into a bad optimum - a stochastic (~15% of runs) per-frame mis-scaling that elevated R-meas and collapsed CC1/2 at low symmetry (the image-level CC1/2 half-split makes the damage look patchy across shells, while the data is genuinely noisier). A Cauchy loss (3 sigma) soft-downweights those outliers without a hard cut: MyoB 0/10 catastrophic (was ~2/10), R-meas stable, and ISa improves on every test crystal (EP0210 9.2->12.4, MyoB 12.5->14.6, lysoC 10.4->11.2, cytC 11.5->12.6), most on low symmetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:00:42 +02:00
leonarski_fandClaude Opus 4.8	99b923e727	scaling: smooth per-frame scale G across frames before the rot3d combine ... ScaleOnTheFly fits each frame's scale G independently with no neighbour coupling, so the few partials of one rocking event are weight-summed in combine3D on inconsistent scales - jitter that never enters the full's counting sigma and instead surfaces as scatter between symmetry mates, inflating the error-model b (low ISa). A centered moving average of log(G) over a small frame window (default 9, on for rot3d) removes it, mirroring XDS's smooth scaling. Complementary to --scale-fulls (which rescales between fulls, after the combine): smoothing fixes within-event scale, scale-fulls fixes between-full. On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa 11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest). Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS. --smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a diagnostic that fixes the scaling mosaicity for sweeps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	82b8a9752c	rotation: fix mosaicity underestimate + default-on capture-uncertainty (ISa 10.7->19.1) ... The rotation per-image mosaicity was ~3x too small (0.045 vs the true 0.13deg), which crippled the partiality model and capped per-observation precision: it predicted reflections on too few frames and over-peaked the rocking partiality, so the rot3d-combined fulls were ~1.7x noisier than XDS's, the integration bottleneck behind the jungfraujoch-vs-XDS ISa gap. Two root causes, both fixed: - CalcMosaicityXDS (Kabsch-2010 MLE) searched each spot's exact-Bragg phi only within +-wedge (the 0.2deg oscillation). Reflections recorded at larger rocking offset - the tail that defines the mosaic width - fell outside and were dropped, truncating the tau distribution so the MLE underestimated ~2x. Widen the search window to wedge+0.8deg; the MLE then converges to the true 0.13deg (and is insensitive to widening further, since it weights by the recorded fraction). - ScaleOnTheFly then re-refined the mosaicity from the intensity residual, which is degenerate with the per-image scale G and collapses it toward its floor. Trust the (now correct) indexing mosaicity and keep it fixed during scaling. With the correct mosaicity, --capture-uncertainty (which down-weights the over-extrapolated under-captured fulls) now pays off strongly, so default it ON (1.0) for the rot3d combine; it stays off for non-rot3d. Together on the HEWL rotation crystal: ISa 10.7 -> 19.1, and anomalous peak height vs XDS goes from 52% to ~78% (CL_CL 1.92x -> 1.29x). This reaches XDS's own published-correction ceiling (DECAY+ABSORP+MODPIX ~= 19.6); the remaining gap to its quoted ISa 28 is the I->inf extrapolation. No effect on the stills path (rotation-only code). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 23:57:04 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00