v1.0.0-rc.155 #65

Merged

leonarski_f merged 17 commits from 2606-viewer-processing into main 2026-06-25 22:01:49 +02:00

Conversation 0 Commits 17 Files Changed 190

+2504 -2729

leonarski_f commented 2026-06-25 20:49:57 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
• jfjoch_viewer: Some graphical improvements.
• jfjoch_viewer: Simplify und unify data analysis settings.
• jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.

leonarski_f added 17 commits 2026-06-25 20:49:57 +02:00

Scaling: reference dataset, 3D combine (-P rot3d), R-meas, profile-fit integrator ... e4b3064254

Reference dataset (a): LoadReferenceMtz adds column selection + cell/SG/resolution +
a data-vs-reference consistency check; jfjoch_process/jfjoch_scale gain
--reference-column; the viewer gets a Reference section in the MX settings dock
(worker-owned, independent of the loaded dataset) that flows into reprocessing jobs.

3D combine (-P rot3d): Combine3D weight-sums a reflection's per-frame partials into one
counting-limited full before merging (orthogonal ScalingSettings::combine_3d flag, not a
partiality model), with a de-biased Poisson variance. Crystal 2: ISa 1.7->8.4, R-meas
~67%->18.9%, intensities unchanged (CCref held).

Quality metrics (b): R-meas (Diederichs-Karplus) + redundancy columns in MergeStats; ISa
logged. jfjoch fulls 18.9% vs XDS 4.5% (same ASU/run).

Profile-fit integrator (experimental): ProfileIntegrate2D (--integrator gaussian|empirical)
is a reference-free, rot3d-compatible profile-fit extraction (the decomposed PixelRefine
intensity step). Gaussian: R-meas 18.9->14.6%, ISa ->9.5. Anisotropy/per-region add nothing
(the discriminating info is in the discarded rocking direction). See NEXTGEN_INTEGRATOR.md.
--dump-observations exports the unmerged fulls for XDS comparison.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Default to the Gaussian profile-fit integrator ... 9ba2401b53

The Gaussian profile-fit (ProfileIntegrate2D) recovers more accurate
intensities than the classical box-sum. On the HEWL rotation set its
ANODE anomalous peak heights at the S/Cl sites rise from 0.46x to 0.53x
of XDS (Met-S 6.0->7.1 sigma, Cys-S 5.6->6.1) and ISa 8.4->9.4, with
R-meas unchanged. Box-sum stays available via --integrator boxsum.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add anomalous-accuracy scoreboard (HEWL ANODE vs XDS) ... 7cc418ed2f

One command to score a jfjoch anomalous merge against XDS on physical
accuracy: feed the -A merge through SHELXC + ANODE (run under qemu-user,
since this WSL2 kernel's vsyscall=none breaks the static SHELX binaries)
and report <d"/sig> per shell and the averaged anomalous peak height at
the model S/Cl sites vs XDS. Unlike R-meas/CC1/2 (precision) it catches
accuracy regressions - e.g. outlier rejection raises CC1/2 but lowers
the anomalous peaks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Reject the +/-1 masked/saturated sentinel band in 2D integration ... 2041ce2bd5

Masked/error pixels carry the int type minimum and saturated the maximum,
but the lossy codec can nudge them inward by one (masked observed as
INT32_MIN+1 in the decompressed data). The integrators only checked the
exact extremes, so a shifted sentinel in a reflection's background ring was
treated as a valid pixel -> garbage background and intensity for any
reflection whose box clips a module gap. Reject the +/-1 band too (real
calibrated counts never approach the type extremes). Neutral on the
well-centred lyso test set; a correctness fix for gap-clipping reflections.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add global (joint) scaling path: jfjoch_process --global-scale ... 50ebf1694e

Refine all per-image scales, per-image mosaicities and the shared per-HKL
true intensities (plus the global wedge) together in one Ceres problem
(GlobalScale.{h,cpp}, GlobalScaleCeres), as an alternative to the existing
alternating merge-then-scale-each-image loop. The result feeds back into each
reflection's image_scale_corr exactly as ScaleOnTheFly does, so the rot3d
combine and MergeOnTheFly run unchanged and every metric stays comparable.

On HEWL crystal 2 the joint fit converges to the same place as the alternating
loop (anomalous S-peak vs XDS 0.53x -> 0.54x, ISa 9.4 both), confirming the
alternating path already reaches the joint optimum. Kept as a validated
alternative and the home for future global corrections.

A shared quadratic absorption surface in detector position was prototyped here
and dropped: it fit large non-physical coefficients (radially degenerate with
the per-HKL resolution structure), lowered the scaling-fit residual but raised
the error-model b (ISa 9.4 -> 7.3) and did not improve anomalous accuracy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add XDS-order scaling of the rot3d fulls: --scale-fulls ... 8d0cd19e48

The rot3d combine emits fulls with partiality == 1 and image_scale_corr == 1,
so the fulls are only ever scaled as per-frame partials upstream - their
per-frame scale G is fit through the rocking-curve/partiality model
(G*partiality*B*lp*Itrue - Iobs) and so absorbs any model error. XDS/DIALS
instead scale the 3D-integrated fulls directly.

--scale-fulls inserts a second scaling pass on the combined fulls with the
Unity model (G*Itrue - I_full, no partiality term), between Combine3D and
MergeOnTheFly, reusing ScaleOnTheFly on a Unity-configured experiment copy. It
is a pure post-correction (updates the fulls' image_scale_corr 1 -> 1/G, no
re-combine).

HEWL crystal 2, anomalous S-peak height vs XDS: 0.53x -> 0.57x and ISa
9.4 -> 10.5 - improving precision and accuracy together (not the CC1/2-up /
anomalous-down trade-off of outlier rejection).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Default two-pass rotation indexing to 100 first-pass images (was 30) ... a37613690e

A larger first-pass image set gives a more stable consensus lattice and so
better per-frame prediction. On HEWL crystal 2 (full 360 deg) this lifts the
anomalous S-peak height vs XDS from 0.57x to 0.61x and ISa from 10.5 to 11.0,
at negligible cost (the first pass is a small fraction of the run). ~100 is the
sweet spot; more (300) does not help further.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the global (joint) scaling path ... c84d581d3e

Profiling showed the joint Ceres solve took ~120 s versus ~3.5 s for the
alternating per-image scaling loop (~35x) for no quality gain (HEWL
anomalous 0.54x vs 0.53x), so it is not worth keeping. Drop
GlobalScale.{h,cpp}, the jfjoch_process --global-scale flag, and
ScalingSettings::GlobalScaling.

While here, in the same scaling/process area: fold scale_fulls into
ScalingSettings (alongside combine_3d) so the CLI and experiment carry it
uniformly, add per-substep [timing] logging to the scaling/merge post-pass
(including the serial MergeAll vs parallel ScaleAllImages split), and carry
structured MergeStatistics + ISa in ProcessResult.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Document the integrator and the XDS-quality investigation ... 6e3c52bf91

Rewrite NEXTGEN_INTEGRATOR.md from the early design draft into a full
record: the diagnosis (our intensities are accurate and unbiased per ANODE,
~2x noisier than XDS), the experiment ledger of what worked and the many
dead ends, the reusable tooling, and an honest assessment of the remaining
gap. Fix the stale ProfileIntegrate2D.h header (it is wired and the default
integrator, not a draft).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Fix anomalous completeness approaching 200% ... 18d9c8ad63

In anomalous mode (Friedel mates kept separate) unique_reflections counts
I+ and I- of each acentric reflection as two unique members, but the
possible-reflection count used the merged ASU, so completeness approached
200%. Count each acentric possible reflection twice (centric once) when not
merging Friedel mates, matching how unique_reflections is counted; verified
overall completeness 99.7% on the HEWL rotation set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the experimental PixelRefine integrator ... 38ea0ec237

On the LysozymeJet5 serial stills the default Gaussian profile-fit
integrator (ProfileIntegrate2D) + reference scaling matched or beat
whole-PixelRefine on every per-shell CC1/2 (overall 95.7% vs 91.9%), ISa
(1.6 vs 1.2) and R-meas (98.5% vs 175%), with CCref a tie -- so PixelRefine
has no remaining advantage. Reference-based per-image scaling is
integrator-agnostic (IndexAndRefine::ReferenceIntensities builds a
ScaleOnTheFly(experiment, reference) applied to any integrator's output),
so the reference-dataset feature (CCref + reference scaling) is kept.

Delete image_analysis/pixel_refinement/, GeomRefinementAlgorithmEnum::
PixelRefine and its gates, BraggIntegrationSettings::ProfileMultiplier
(PixelRefine-only; R1 is shared and kept), and the -r pixelrefine /
--profile-multiplier CLI. The inherited lessons (mean background, de-biased
variance, tight-profile-loses / centroid floor, R-refinement futile) are
folded into NEXTGEN_INTEGRATOR.md.

NOTE: this transiently breaks the viewer build -- the committed viewer
still references the removed enum and ProfileMultiplier. It is fixed in the
next commit (the viewer feature work), held separate while the viewer UI is
being tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Update viewer icons from the new jfjoch.svg logo ... eab1bfab7d

Regenerate jfjoch.png (256-wide, transparent), jfjoch.ico (multi-resolution
16/32/48/64/128/256, square) and jfjoch.icns (16-512) from the new
transparent-background jfjoch.svg, and add the SVG as the source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Consolidate viewer settings into the panel + add the merge-stats window ... 383072ac80

Make the inline settings dock the single home for processing settings and
retire the separate Processing-settings window (and the dock<->window sync):
- "Analyze image" / "Analyze dataset" move to the top of the panel; the
  MX/AzInt toggle decides the dataset-job kind, so the job dialog drops its
  mode combo.
- The panel gains the most-used spot-finding (max spots, high-resolution,
  min pixels), Bragg (Gaussian profile-fit, r1/r2/r3) and scaling (partiality,
  "3D rotation scaling" = rot3d combine + scale-fulls, merge Friedel, refine
  B, resolution limit) handles, a live indexing-algorithm description line,
  and now owns the Bragg/Scaling settings. The now-unused window tab classes
  are deleted.
- Complete the PixelRefine removal on the viewer side (the "Pixel refinement"
  option + profile-multiplier widget), fixing the transient HEAD breakage.

New JFJochMergeStatsWindow: an analysis pop-up for a finished merge (hero
numbers over a per-resolution plot / per-shell table), auto-opened on
completion and reopenable from the processing-jobs dock.

Fixes: disable tear-off dock floating (a floated dock is a dead off-screen
window under WSLg, which has no window manager); version the saved dock
layout so a stale arrangement is discarded instead of restoring a broken one;
keep the Analyze-button icons; right-align and equal-width the stats table;
line-plot / table toggle icons (ToolbarIcons gains linePlot + table).

Add ScalingSettings::HighResolutionLimit_A(optional) so the panel can clear
the resolution limit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Viewer: single "Process as stills" mode + compact, folded panel ... 4462f0998d

Replace the scaling partiality combo + "3D rotation scaling" checkbox with
one "Process as stills" checkbox in the indexing section (enabled only for
rotation datasets, unchecked by default). Unchecked on a rotation dataset
drives the full rotation path (rotation indexing at 60 first-pass images,
Rotation partiality, rot3d combine, scale-fulls); checked treats it as
stills (fixed partiality, per-frame indexing). The Analyze-dataset dialog
drops its rotation options (the panel is the single source) and buildConfig
reads rotation indexing from the experiment.

Fix: the worker's UpdateSpotFindingSettings copied indexing fields one by
one and was dropping RotationIndexing, so the mode never reached jobs.

Also: fold the panel accordions on start except Geometry and Unit cell, and
make the scaling resolution-limit a compact checkbox + field aligned with
the other checkboxes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

TCPImagePuller: add OS-level TCP keep-alive 884e831359

VERSION: 1.0.0-rc.155 0a71d76b0f

CHANGELOG: adjust 1a03e884c4

leonarski_f merged commit 54c667190f into main 2026-06-25 22:01:49 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#65