On the LysozymeJet5 serial stills the default Gaussian profile-fit
integrator (ProfileIntegrate2D) + reference scaling matched or beat
whole-PixelRefine on every per-shell CC1/2 (overall 95.7% vs 91.9%), ISa
(1.6 vs 1.2) and R-meas (98.5% vs 175%), with CCref a tie -- so PixelRefine
has no remaining advantage. Reference-based per-image scaling is
integrator-agnostic (IndexAndRefine::ReferenceIntensities builds a
ScaleOnTheFly(experiment, reference) applied to any integrator's output),
so the reference-dataset feature (CCref + reference scaling) is kept.
Delete image_analysis/pixel_refinement/, GeomRefinementAlgorithmEnum::
PixelRefine and its gates, BraggIntegrationSettings::ProfileMultiplier
(PixelRefine-only; R1 is shared and kept), and the -r pixelrefine /
--profile-multiplier CLI. The inherited lessons (mean background, de-biased
variance, tight-profile-loses / centroid floor, R-refinement futile) are
folded into NEXTGEN_INTEGRATOR.md.
NOTE: this transiently breaks the viewer build -- the committed viewer
still references the removed enum and ProfileMultiplier. It is fixed in the
next commit (the viewer feature work), held separate while the viewer UI is
being tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Profiling showed the joint Ceres solve took ~120 s versus ~3.5 s for the
alternating per-image scaling loop (~35x) for no quality gain (HEWL
anomalous 0.54x vs 0.53x), so it is not worth keeping. Drop
GlobalScale.{h,cpp}, the jfjoch_process --global-scale flag, and
ScalingSettings::GlobalScaling.
While here, in the same scaling/process area: fold scale_fulls into
ScalingSettings (alongside combine_3d) so the CLI and experiment carry it
uniformly, add per-substep [timing] logging to the scaling/merge post-pass
(including the serial MergeAll vs parallel ScaleAllImages split), and carry
structured MergeStatistics + ISa in ProcessResult.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rot3d combine emits fulls with partiality == 1 and image_scale_corr == 1,
so the fulls are only ever scaled as per-frame partials upstream - their
per-frame scale G is fit through the rocking-curve/partiality model
(G*partiality*B*lp*Itrue - Iobs) and so absorbs any model error. XDS/DIALS
instead scale the 3D-integrated fulls directly.
--scale-fulls inserts a second scaling pass on the combined fulls with the
Unity model (G*Itrue - I_full, no partiality term), between Combine3D and
MergeOnTheFly, reusing ScaleOnTheFly on a Unity-configured experiment copy. It
is a pure post-correction (updates the fulls' image_scale_corr 1 -> 1/G, no
re-combine).
HEWL crystal 2, anomalous S-peak height vs XDS: 0.53x -> 0.57x and ISa
9.4 -> 10.5 - improving precision and accuracy together (not the CC1/2-up /
anomalous-down trade-off of outlier rejection).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Refine all per-image scales, per-image mosaicities and the shared per-HKL
true intensities (plus the global wedge) together in one Ceres problem
(GlobalScale.{h,cpp}, GlobalScaleCeres), as an alternative to the existing
alternating merge-then-scale-each-image loop. The result feeds back into each
reflection's image_scale_corr exactly as ScaleOnTheFly does, so the rot3d
combine and MergeOnTheFly run unchanged and every metric stays comparable.
On HEWL crystal 2 the joint fit converges to the same place as the alternating
loop (anomalous S-peak vs XDS 0.53x -> 0.54x, ISa 9.4 both), confirming the
alternating path already reaches the joint optimum. Kept as a validated
alternative and the home for future global corrections.
A shared quadratic absorption surface in detector position was prototyped here
and dropped: it fit large non-physical coefficients (radially degenerate with
the per-HKL resolution structure), lowered the scaling-fit residual but raised
the error-model b (ISa 9.4 -> 7.3) and did not improve anomalous accuracy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reference dataset (a): LoadReferenceMtz adds column selection + cell/SG/resolution +
a data-vs-reference consistency check; jfjoch_process/jfjoch_scale gain
--reference-column; the viewer gets a Reference section in the MX settings dock
(worker-owned, independent of the loaded dataset) that flows into reprocessing jobs.
3D combine (-P rot3d): Combine3D weight-sums a reflection's per-frame partials into one
counting-limited full before merging (orthogonal ScalingSettings::combine_3d flag, not a
partiality model), with a de-biased Poisson variance. Crystal 2: ISa 1.7->8.4, R-meas
~67%->18.9%, intensities unchanged (CCref held).
Quality metrics (b): R-meas (Diederichs-Karplus) + redundancy columns in MergeStats; ISa
logged. jfjoch fulls 18.9% vs XDS 4.5% (same ASU/run).
Profile-fit integrator (experimental): ProfileIntegrate2D (--integrator gaussian|empirical)
is a reference-free, rot3d-compatible profile-fit extraction (the decomposed PixelRefine
intensity step). Gaussian: R-meas 18.9->14.6%, ISa ->9.5. Anisotropy/per-region add nothing
(the discriminating info is in the discarded rocking direction). See NEXTGEN_INTEGRATOR.md.
--dump-observations exports the unmerged fulls for XDS comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
leonarski_f