Commit Graph
4 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 4.8 99b923e727 scaling: smooth per-frame scale G across frames before the rot3d combine
ScaleOnTheFly fits each frame's scale G independently with no neighbour
coupling, so the few partials of one rocking event are weight-summed in
combine3D on inconsistent scales - jitter that never enters the full's
counting sigma and instead surfaces as scatter between symmetry mates,
inflating the error-model b (low ISa). A centered moving average of
log(G) over a small frame window (default 9, on for rot3d) removes it,
mirroring XDS's smooth scaling. Complementary to --scale-fulls (which
rescales between fulls, after the combine): smoothing fixes within-event
scale, scale-fulls fixes between-full.

On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa
11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest).
Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS.

--smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a
diagnostic that fixes the scaling mosaicity for sweeps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8 82b8a9752c rotation: fix mosaicity underestimate + default-on capture-uncertainty (ISa 10.7->19.1)
The rotation per-image mosaicity was ~3x too small (0.045 vs the true 0.13deg),
which crippled the partiality model and capped per-observation precision: it
predicted reflections on too few frames and over-peaked the rocking partiality,
so the rot3d-combined fulls were ~1.7x noisier than XDS's, the integration
bottleneck behind the jungfraujoch-vs-XDS ISa gap.

Two root causes, both fixed:
- CalcMosaicityXDS (Kabsch-2010 MLE) searched each spot's exact-Bragg phi only
  within +-wedge (the 0.2deg oscillation). Reflections recorded at larger rocking
  offset - the tail that defines the mosaic width - fell outside and were dropped,
  truncating the tau distribution so the MLE underestimated ~2x. Widen the search
  window to wedge+0.8deg; the MLE then converges to the true 0.13deg (and is
  insensitive to widening further, since it weights by the recorded fraction).
- ScaleOnTheFly then re-refined the mosaicity from the intensity residual, which
  is degenerate with the per-image scale G and collapses it toward its floor.
  Trust the (now correct) indexing mosaicity and keep it fixed during scaling.

With the correct mosaicity, --capture-uncertainty (which down-weights the
over-extrapolated under-captured fulls) now pays off strongly, so default it ON
(1.0) for the rot3d combine; it stays off for non-rot3d. Together on the HEWL
rotation crystal: ISa 10.7 -> 19.1, and anomalous peak height vs XDS goes from
52% to ~78% (CL_CL 1.92x -> 1.29x). This reaches XDS's own published-correction
ceiling (DECAY+ABSORP+MODPIX ~= 19.6); the remaining gap to its quoted ISa 28 is
the I->inf extrapolation. No effect on the stills path (rotation-only code).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 23:57:04 +02:00
leonarski_f 75e401f0e5 v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

Reviewed-on: #63
2026-06-23 20:29:49 +02:00
leonarski_f fc68a9baed v1.0.0-rc.146 (#56)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m34s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m16s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m49s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m15s
Build Packages / XDS test (durin plugin) (push) Successful in 7m16s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky9) (push) Successful in 10m12s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 47s
Build Packages / DIALS test (push) Successful in 10m18s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 5m46s
Build Packages / build:rpm (rocky8) (push) Successful in 1h41m2s
Build Packages / XDS test (neggia plugin) (push) Successful in 1h59m18s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.

WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

Reviewed-on: #56
2026-05-28 18:48:35 +02:00