On the LysozymeJet5 serial stills the default Gaussian profile-fit
integrator (ProfileIntegrate2D) + reference scaling matched or beat
whole-PixelRefine on every per-shell CC1/2 (overall 95.7% vs 91.9%), ISa
(1.6 vs 1.2) and R-meas (98.5% vs 175%), with CCref a tie -- so PixelRefine
has no remaining advantage. Reference-based per-image scaling is
integrator-agnostic (IndexAndRefine::ReferenceIntensities builds a
ScaleOnTheFly(experiment, reference) applied to any integrator's output),
so the reference-dataset feature (CCref + reference scaling) is kept.
Delete image_analysis/pixel_refinement/, GeomRefinementAlgorithmEnum::
PixelRefine and its gates, BraggIntegrationSettings::ProfileMultiplier
(PixelRefine-only; R1 is shared and kept), and the -r pixelrefine /
--profile-multiplier CLI. The inherited lessons (mean background, de-biased
variance, tight-profile-loses / centroid floor, R-refinement futile) are
folded into NEXTGEN_INTEGRATOR.md.
NOTE: this transiently breaks the viewer build -- the committed viewer
still references the removed enum and ProfileMultiplier. It is fixed in the
next commit (the viewer feature work), held separate while the viewer UI is
being tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite NEXTGEN_INTEGRATOR.md from the early design draft into a full
record: the diagnosis (our intensities are accurate and unbiased per ANODE,
~2x noisier than XDS), the experiment ledger of what worked and the many
dead ends, the reusable tooling, and an honest assessment of the remaining
gap. Fix the stale ProfileIntegrate2D.h header (it is wired and the default
integrator, not a draft).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Masked/error pixels carry the int type minimum and saturated the maximum,
but the lossy codec can nudge them inward by one (masked observed as
INT32_MIN+1 in the decompressed data). The integrators only checked the
exact extremes, so a shifted sentinel in a reflection's background ring was
treated as a valid pixel -> garbage background and intensity for any
reflection whose box clips a module gap. Reject the +/-1 band too (real
calibrated counts never approach the type extremes). Neutral on the
well-centred lyso test set; a correctness fix for gap-clipping reflections.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reference dataset (a): LoadReferenceMtz adds column selection + cell/SG/resolution +
a data-vs-reference consistency check; jfjoch_process/jfjoch_scale gain
--reference-column; the viewer gets a Reference section in the MX settings dock
(worker-owned, independent of the loaded dataset) that flows into reprocessing jobs.
3D combine (-P rot3d): Combine3D weight-sums a reflection's per-frame partials into one
counting-limited full before merging (orthogonal ScalingSettings::combine_3d flag, not a
partiality model), with a de-biased Poisson variance. Crystal 2: ISa 1.7->8.4, R-meas
~67%->18.9%, intensities unchanged (CCref held).
Quality metrics (b): R-meas (Diederichs-Karplus) + redundancy columns in MergeStats; ISa
logged. jfjoch fulls 18.9% vs XDS 4.5% (same ASU/run).
Profile-fit integrator (experimental): ProfileIntegrate2D (--integrator gaussian|empirical)
is a reference-free, rot3d-compatible profile-fit extraction (the decomposed PixelRefine
intensity step). Gaussian: R-meas 18.9->14.6%, ISa ->9.5. Anisotropy/per-region add nothing
(the discriminating info is in the discarded rocking direction). See NEXTGEN_INTEGRATOR.md.
--dump-observations exports the unmerged fulls for XDS comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Rotation indexer has two retries if failes
* jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot)
* jfjoch_broker: Integration calculates background mask based on R2 radius
* jfjoch_process: HDF5 files are not saved by default
Reviewed-on: #37
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_viewer: Minor polishing of new functionality
* jfjoch_broker: User NaN for empty azimuthal bins
Reviewed-on: #12
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 4.8