v1.0.0-rc.153 #63

Merged

leonarski_f merged 229 commits from 2606-pixel-refine into main 2026-06-23 20:29:50 +02:00

Conversation 0 Commits 229 Files Changed 615

+44964 -13457

leonarski_f commented 2026-06-23 19:24:34 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
• jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
• jfjoch_broker: Add ROI calculation in non-FPGA workflow
• jfjoch_broker: Fixes to TCP image pusher
• jfjoch_broker: Remove NUMA bindings
• jfjoch_broker: Improvements to indexing
• jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
• jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
• jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
• jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

leonarski_f added 228 commits 2026-06-23 19:24:34 +02:00

PixelRefine: Work in progress 50f953bcc5

LatticeReduction: Move outside of XtalOptimizer 0d6b278767

PixelRefine: Work in progress 9a991b6614

PixelRefine: Work in progress (Claude) 66a48c4266

PixelRefine: Work in progress (Claude) 6f6098d508

PixelRefine: Add bandwidth contribution 6f31159607

jfjoch_process: First pixel refine integration 155c53acd8

jfjoch_writer: Add incident_wavelength_spread 96edee9b2d

Minor fixes 32e91f7287

PixelRefine: Claude fixed my bugs 698a98be35

jfjoch_viewer: Add pixel refinw and magnifier windows (to be tested) 05711a1077

PixelRefine: Make it faster by doing one cell calculation per shoe-box e8a9b1840d

jfjoch_process: Clarify what happens in scaling when reference data provided 254462b9f2

PIxelRefine: Another iteration e4b66f9cd9

Merge: CC1/2 limit adjustment 5735302691

JFJochMagnifierWindow: Zoom is saved ... it is not optimal (when image is first loaded, than it starts with weird zoom), but can be fixed later 30dcc98f89

JFJochMagnifierWindow: Fixed 8a582b8a90

jfjoch_viewer: fixes to pixel refine feca63f4b9

jfjoch_viewer: fixes to pixel refine window 2d202f1d44

jfjoch_viewer: fix sorting by indexing status 003fea1b1e

jfjoch_viewer: show image CC based on standard pipeline 6af22b6a0c

BraggPrediction: Include X-ray bandwidth 6c85aaba2b

jfjoch_viewer: Better display (to be tested) of pixel refine efe882f4b6

jfjoch_process: Remove postrefinement e051eed033

Bragg integration: option to use azimuthal integration profile e4230bc14e

jfjoch_process: Add option to use azimuthal integration as background for Bragg integration b22d5929a1

Revert "jfjoch_process: Add option to use azimuthal integration as background for Bragg integration" ... 579d36fe71

This reverts commit b22d5929a1.

Revert "Bragg integration: option to use azimuthal integration profile" ... 59281f6330

This reverts commit e4230bc14e.

jfjoch_scale: Remove postrefinement model option 7478c0390f

PixelRefine: Simplify (remove Lorentz correction, remove background from azimuthal integration) bd5fef7f61

XDS plugin: Fix mutex 48d4fb0d0f

PixelRefine: Some improvements d31063ca3f

jfjoch_viewer: missing pixel refine table c8db50ab41

PixelRefine: Improvements to accept more reasonable count of reflections 47dc19dd03

PixelRefine: Results seem to be much better db68c8dc38

PixelRefine: Document on local changes 3cdf26232c

Integration: mean background + global error model (trustworthy sigmas) ... e6a50b45c7

Background estimate: use the mean of the local ring, not the median. For a
right-skewed (Poisson) background the median sits below the mean, so subtracting
it under-subtracts and biases every weak intensity positive; over multiplicity
this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in
both PixelRefine and BraggIntegrate2D (the classical route had the same bug).
<I/sig> now tracks CC honestly and the true resolution limit is visible.

Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the
scatter of symmetry equivalents at the merge level (so both integrators benefit),
and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the
reflection mean (not the per-observation I_i, which biases the weights and
collapses CC); a,b come from a relative-weighted bin regression. Replaces the
earlier per-resolution-shell variant, which was partly masking the background bias.

METHODS.md: document both (Sections 6-7), integrator-agnostic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Error model: harden the fit against pathological inputs (code review) ... c93d381dc8

Addresses code-review findings on RefineErrorModel:
- Floor the 1/dev^2 bin weight relative to the data scale (1e-3 of the median
  bin dev^2), not an absolute 1e-30: a near-zero-scatter bin could otherwise
  acquire a runaway weight and hijack the global (a,b) fit.
- Reject a near-collinear normal-equation system relatively (det > 1e-10*Ass*AII)
  instead of with an absolute threshold that an ill-conditioned fit can pass.
- Reset the model to identity at entry so any early return leaves it inactive
  rather than keeping a stale a/b alongside a freshly-cleared mean map (which
  would make CorrectedSigma fall back to the per-observation I).
- PixelRefine: correct the orient_prior comment - with the sweep on, the LSQ
  anchor is the swept orientation (intended), not the spot-centroid one.

Verified unchanged on the lyso test set (ISa 1.1, CC1/2 90.3%).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: checkpoint before cleanup (factored model + all diagnostic levers) ... 6f2033db00

Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_*
env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel
ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/
ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit
makes Terms 1+2 the model and strips all of this.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: make factored Terms 1+2 the model, remove old wiring ... 100fe7b7e7

PixelRefine is now an intensity-only operation: geometry is fixed (refined
upstream by XtalOptimizer) and the only objective is the factored per-reflection
likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width
R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting
the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills,
1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and
matches the rotation path from the stills path.

Removed:
- the per-pixel ShoeboxResidual loss and PixelResidual cost functor;
- all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R),
  the regularised-orientation LSQ, signal-weighting, and the global sweep;
- Term 3 (per-spot recentring) - a confirmed no-op on both crystals;
- the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the
  PR_* env knobs + stderr dumps in IndexAndRefine;
- the PredictImage/ChiSquaredImage renderers and the entire viewer
  PixelRefine window/table/params + worker bindings + shoebox overlay.

The sweep box-integrator background median became mean (consistency) by virtue
of removing the sweep. METHODS.md rewritten for the current model; findings
recorded in FINDINGS-2026-06.md. Net -2200 lines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... bf6efc7fe9

Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet
(1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots
spill past a radius-3 box), is neutral on the mono crystal and on the classical
integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single
value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to
BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob.

(Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's
tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res,
azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured
2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight
measured width loses to a generous aperture, same lesson as the radius bump.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: R1 multiplier (fix tight-Term2 regression) + env-gated orientation refine ... cfcd4c9e56

Full-jet R-free showed the factored model regressed vs the traditional pipeline
(R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured
profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px
centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix,
XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier
(default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3
(~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free.

Also reinstates the pre-factored per-image orientation refinement (per-pixel
ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation,
re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data
R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the
optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation.

NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier
value and the orientation-refine decision are to be fixed against R-free, then baked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: env-gated orientation + cell-scale sweep (PR_SWEEP) for R-free A/B ... 45ee8c2b40

R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats
traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on
R-free; x9 only buys CC1/2 internal consistency), and per-image orientation
*refinement* is a no-op (0.2618 vs 0.2625).

Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP
(off by default) to test whether the cell-scale degree of freedom - which the
gradient orientation refinement lacks, and which moves the high-res shells radially
- helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for
the gradient path), but that does not certify R-free, so it is left for validation.

NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once
the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Merge: per-observation outlier rejection (env-gated PR_REJECT) ... a47b376dc3

At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO
outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every
merged mean. Add a robust per-observation cut: drop observations whose corrected
intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN.
The error-model sigma already captures the genuine (partiality) scatter, and the
median is a robust centre, so only the tail beyond the real spread is removed -
not good partials. The median is computed in RefineErrorModel (which already pools
the observations per reflection); AddImage applies the cut.

Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the
jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut)
- the cut is vs the data's own median, not the reference, so the gain is real
cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at
low nsigma) are for Filip's full-jet R-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... 6f733d74c2

Promote the per-observation merge outlier rejection from the temporary PR_REJECT
env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0
disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers
CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and
DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 ->
0.2585). Applies to both the PixelRefine and classical merge paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: strip experimental env knobs (orientation, sweep, Lorentz, ML census) ... 102a2a7c81

Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed,
returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper
(r1_multiplier, default 6):
- PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) -
  XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual,
  OrientationRegularizer, PixelObs::weight, the refinement block and its fields.
- PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2
  (per-image overfit). Removes SweepOrientationCell and its fields.
- PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality
  already subsumes it); was already reverted.
- PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5%
  of jet frames are multi-lattice; shelved).

PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged:
crystal 2 / jet / hybrid -R -r pixelrefine all reproduce.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... 8a9d80eb71

(1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real
BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process
--profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the
setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings
outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since
it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PixelRefine: drop crystal-system idealisation, use the indexed cell as-is ... ecdb7048a0

PixelRefine no longer refines the cell, so the symmetry-constrained cell
parameterisation inherited from XtalOptimizer has no manifold to constrain.
The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B
matrix) merely reconstructed the lattice columns it started from, and for
non-triclinic systems it re-idealised the indexed cell (averaging a,b for
tetragonal; a,b,c for cubic; forcing alpha=gamma=90).

Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a
single path: take the three real-space lattice columns (latt.Vec0/1/2())
directly and form the reciprocal node via the general cross-product inverse.
This reproduces every crystal system exactly from the actual cell, is more
faithful (no re-idealisation), and removes the crystal_system field plus two
now-unused includes. PredictedNode de-templated (only ever called with double).
Crystal symmetry still lives where it belongs: indexing (upstream) and merging
(downstream via the space-group HKL key).

A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the
old idealisation was already a no-op): stat tables bit-identical across all
shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% --
the only delta is 4th-digit float-ordering noise in the error-model b
coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free
unchanged. Net -81 lines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

IndexAndRefine: conventionalise de-novo lattice when cell+SG are given ... 40da1aab13

Providing both a unit cell (-C) and space group (-S) silently broke the de-novo
indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the
`sg && GetUnitCell()` branch fed the indexer's raw lattice straight into
symmetry-constrained refinement. FFBIDX returns the lattice already in the
reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive
cell; enforcing the crystal system on its mis-assigned axes rejects every frame.

Fix: for de-novo indexers only, reduce the lattice to the conventional setting
(LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so
it is byte-identical to before (no regression). niggli_class stays unassigned (0)
in this branch - it is a property of the primitive cell incl. centering, which
LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2)
cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred.

Validation (lyso, -C79,79,38 -S96):
  crystal 2 FFT  : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality
  crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical
  jet       FFBIDX: 27.2% (CC1/2 91.9) - byte-identical
(Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no
consensus on the weak serial-still frames, 0% even without -S; to investigate.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

IndexAndRefine: unify cell setting across indexers, keep FFBIDX neutral ... e1e2ca8e49

Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when
cell+SG are given, so every frame reports its cell in ONE consistent setting per
space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same
reflection as different HKLs and cannot be merged).

Use LatticeSearch's conventional cell when its detected symmetry agrees with the
user's space group. On noisy frames LatticeSearch can pick an alternative Bravais
setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell,
[78,78,38]->[110,111,38]) whose system disagrees; there:
  - FFBIDX already returns the reference setting (c-last), consistent with the
    conventional frames, so its raw lattice is safe -> use it. This recovers the
    ~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral.
  - de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order
    (c-first); merging it with the conventional (c-last) frames would corrupt the
    data -> reject the frame instead.

Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice
baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last,
consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis
(~38A) is axis 2 in every merged frame - no axis-order mixing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FFTIndexer: name the histogram axis one_over_d (1/d), not q/Q ... f2f95c44f6

The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d
convention) while the bin width and spot coordinates use the internal 1/d convention -
a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the
same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x
zero-padding of the histogram.

Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an
explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d
is named one_over_d (never q). histogram_size is numerically identical to before, so
behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented
that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi
to a nearby integer shifts indexing ~0.5-1%).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FFTIndexer: pick peaks by prominence above a local background, not raw magnitude ... 929e6e1fa0

The per-direction peak search took argmax|spec|. The projected-spot histogram has a
broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can
exceed the true lattice peaks, so on weak / pink-beam frames every direction reported
a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate
cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors,
the true axes entirely absent.)

Subtract a running-mean background of half-width ~15 A and pick the peak by its
PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp
lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes
win. The prominence is also reported as the magnitude, so FilterFFTResults ranks
directions by real-peak strength rather than envelope. Ported identically to CPU
(prefix-sum window) and GPU (sliding-window in the kernel).

Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX
27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged).
The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes
the envelope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

SpotFindingSettings: Default signal to noise ratio is 4 f878fb9d5d

Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... 545ebdf868

CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial
data. Each image is assigned to one CC1/2 half, so removing it perturbs only that
half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i).
A negative value means dropping the image RAISES CC1/2 (it disagrees with the
consensus). Images whose deltaCChalf is a low-side statistical outlier
(< mean - N*stddev) are skipped when merging. Reference-free.

Two passes over the retained integration outcomes; per-image contributions are
re-derived rather than stored, so memory stays O(unique reflections + images) for
full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off).

Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2
95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS
heterogeneous serial data (some crystals genuinely bad) but slightly trims a
homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative
cut still removes the tail), so it is opt-in. R-free is the real test (user's full
200k). Note: the reported overall N_obs still counts all observations; the exported
merge (and CC1/2) correctly exclude the rejected images.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LoadFCalcFromMtz: fall back to an intensity column when F-model is absent ... 2ba28aea0e

PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a
calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN,
which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and
use it directly as I_ref (F-model is squared, intensities are not).

This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional
integration, or PixelRefine against an external reference) to produce a merge,
then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's
own intensities - free of the reference structure's non-isomorphism bias - and
iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine
runs against it (crystal 2, 55279 reflections, CC1/2 94.3).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

JFJochHDF5Reader: read master indexedLatticeCount (back-compat) ... 6479b91e50

The master-file writer emits /entry/MX/indexedLatticeCount, but the
reader only looked for indexingLatticeCount (the name used by the
per-file MX plugin and data-file read path). Existing master files on
disk therefore returned no indexed-lattice-count vector. Read the
master's indexedLatticeCount first, falling back to indexingLatticeCount,
matching the niggli_class/niggliClass compatibility pattern. The writer
is left unchanged so already-written files keep working.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/CBOR.md: sync field names/types with the serializer ... caff857d8a

Bring the protocol doc in line with CBORStream2Serializer/Deserializer:

- reflection dist_ewald -> rp (dist_ewald is the spot field, not the
  reflection field)
- az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count
- threshold_energy typed as Map(string -> float), not float
- end_date typed as string (it is serialized as a plain text string, with
  no datetime tag, unlike arm_date)
- add missing reflection fields: phi, zeta, image_scale_corr
- add missing spot fields: image, and indexed-only h, k, l, dist_ewald
- add image indexing_lattice_count and end indexed_lattice_count
- drop the duplicated image_scale_factor row in the End table

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: add HDF5.md describing the NeXus layout and JFJoch extensions ... 0381d891bc

New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the
writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout,
NXreflections for integration), the master/data-file layout and the three
NXmx format variants, the NXmx-standard fields that are populated, and every
Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint,
/entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel,
detectorSpecific, calibration, fluorescence, user). Content is derived from
writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked
against the NXmx and NXreflections definitions.

JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to
the new doc; HDF5 is added to the Sphinx toctree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link ... e8d3eb1b08

- Motivation no longer frames the format as serial-only: Jungfraujoch serves
  both rotation and serial MX, and the two-layout rationale (per-image CXI
  spots vs. dataset-wide NXreflections) is presented for both.
- Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an
  integrated-format master whose /entry/data/data is a VDS linking back to the
  original images (all results, no image copies).
- Add a section on the header_appendix / image_appendix (user_data) mechanism:
  how the free-form JSON flows through the start/image CBOR messages and how an
  "hdf5" sub-object is persisted to /entry/user, with an example.
- Link the niggliClass row to International Tables for Crystallography A,
  Table 3.1.3.1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jfjoch_api.yaml: header_appendix hdf5 values go to /entry/user, not /entry/data ... 5143ba0b1f

The description said the "hdf5" sub-object is written under /entry/data, but
the writer (NXmx::UserData) and the example in the same description both use
/entry/user. Correct the path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Documentation: Data analysis description 704e4b4eb3

Fix 638aa379f4

frontend: fix state bugs, slim Plotly bundle, drop CRA leftovers ... 3334e88d3e

- App: make getValues an arrow field so its `this` survives being passed
  as the `update` prop (the post-edit statistics refresh was broken).
- DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual
  so the 1 s statistics poll no longer overwrites a slider mid-edit
  (reference `!=` on a freshly-parsed object was always true → snap-back).
- DataCollection: collapse nested setState in grid-scan callbacks.
- PreviewImage: build an immutable settings copy instead of mutating state
  in place; fix stale comment.
- MultiLinePlotWrapper: drop per-render console.log; render via a custom
  Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only),
  cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip).
- package.json: remove dead react-scripts test/eject scripts and CRA
  eslintConfig, move @redocly/cli to devDependencies.
- Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts,
  setupTests.js).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: add jfjoch_process page, refresh viewer/tools docs, unify CLI naming ... c69b5297d5

- Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its
  options, output files, and the broker/viewer/process distinction; mention
  jfjoch_scale for re-scaling/merging.
- Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env
  vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API.
- Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst.
- jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in
  usage/license/logger.
- jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D
  short flag, make it long-only) and remove dead p/q/i short options.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

build: portability groundwork toward a Windows/MSVC viewer ... 39c808776f

- CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before
  slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's
  bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single
  libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the
  full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process).
- common/NetworkAddressConvert.cpp: guard the network includes for _WIN32
  (winsock2/ws2tcpip vs arpa/inet).
- common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an
  anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) -
  portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

build: guard the librt link to Linux only ... 1056acc3a6

librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a
$<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and
the CUDA block) so -lrt is still passed on Linux - where some distros need it for
clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the
jfjoch_process link line on Linux.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jfjoch_process: spread per-image GPU work across all visible GPUs ... 02fa15c2b9

The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA
device, so all per-image preprocessing/spot-finding/azimuthal integration ran on
GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a
process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op
without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before
building the analysis resources so their CUDA streams/engines land on distinct
devices.

Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU
dispatch with goals and a staged plan (multi-broker GPU isolation via
CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ImageBuffer: replace NUMA interleave with parallel first-touch ... d373ba0490

Allocate the ring buffer with plain malloc and zero it across
hardware_concurrency() threads, so each page is first-touched - and thus
NUMA-placed - by whichever node the scheduler ran the zeroing thread on.
For the random-access buffer this approximates the previous
numa_alloc_interleaved placement, speeds up the one-time fault-in of the
150-200 GB allocation, and drops the libnuma dependency from this file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

acquisition_device: give each device sole ownership of its buffers ... 040cdeacf1

The base AcquisitionDevice no longer allocates or frees frame buffers;
buffer_device is now just a non-owning view of addresses. Each subclass
owns its backing memory and the matching lifecycle:

- PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its
  own destructor (and on ctor failure), symmetric with MapKernelBuffer.
- HLSSimulatedDevice owns plain zeroed heap buffers it points
  buffer_device into, declared before the HLSDevice so they outlive the
  action thread that writes them. The buffers are page-aligned to match
  the real device's kernel DMA buffers - the modelled AXI datamover and
  FPGAIntegrationTest require aligned output buffers.

This drops the NUMA/mmap dance from the simulated path (not
performance-critical) - removing libnuma from acquisition_device - and
replaces the base-class cleanup that had to guess the allocation
strategy with a single clear owner per device.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: refresh NUMA_GPU_REVIEW to match implemented state ... 17732ea774

Several items in the note had landed or were inaccurate. Update it to:
- mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device
  de-NUMA, CUDAWrapper sysfs node lookup;
- add the previously-missing FPGA DMA buffer section - placement is a
  kernel concern (dma_alloc_coherent, device-local), not the userspace
  mbind, which was only the simulator;
- record that libnuma is now down to a single file (NUMAHWPolicy.cpp);
- note that dependency removal (G3) and pinning behaviour (G4) are
  separable axes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove NUMAHWPolicy and the libnuma dependency ... cc925b2668

NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are
placed device-local by the kernel (dma_alloc_coherent), the big RAM ring
buffer is random-access (first-touch handles placement), and GPU work is
already spread across all visible devices. So drop the pinning entirely
and with it libnuma.

- Delete NUMAHWPolicy; the only concern worth keeping - GPU selection -
  is done directly via pin_gpu() (round-robin over visible GPUs) in the
  indexer pool and the Lite analysis threads. CPU-only threads
  (FPGA acquire/pedestal/summation/frame-transform) no longer bind
  anything.
- Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA
  used it.
- numa_policy broker setting is deprecated and ignored (kept in the API
  for backward compatibility; warns once on startup).
- Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake.
- Docs: drop the NUMA dependency, remove the numa_policy config example,
  and document running multiple brokers on disjoint GPUs via
  CUDA_VISIBLE_DEVICES.
- Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

image_analysis: compute ROI statistics in the non-FPGA path ... 58910274bf

MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were
only available on the FPGA path. Add a software ROI engine that mirrors the
FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and
intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so
it can contribute to any subset of up to 16 ROIs.

New image_analysis/roi/ library (JFJochROIIntegration), structured like azint:
a base that precomputes the per-pixel mask and names, a templated CPU engine
(generic over pixel type for a future 16-bit path), and a GPU kernel using
per-block shared-memory atomics for the STXM case (half-detector ROIs).

Masked pixels are excluded entirely; saturated pixels are excluded from the
sums but still count towards the max, matching roi_calc exactly. The engine is
only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already
consume message.roi, so no further changes are needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

TCPStreamPusher: fix zero-copy buffer reuse and make send/END timeouts progress-based ... c0a4801bc4

Three robustness fixes for the writer-facing TCP stream, addressing the
spurious "wrong number of images" / connection failures seen under load.

1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path
   passes a caller-owned buffer with z == nullptr and reuses it for the next
   frame immediately. With MSG_ZEROCOPY the kernel still references that buffer
   after send() returns, so the peer could receive corrupted frames and drop
   the connection mid-stream, truncating the run. Zero-copy is now gated on a
   ZeroCopyReturnValue that keeps the buffer alive until completion.

2. Make the SendAll watchdog measure lack of progress, not total wall-clock.
   The previous absolute deadline tore down a healthy but back-pressured
   connection (slow/starved writer) after a fixed window; the watchdog now
   resets on every byte actually sent, so only a genuinely stuck socket is
   closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR.

3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still
   be draining a backlog of DATA frames when END is sent; each DATA ACK is
   progress, so the timeout only arms once the writer falls silent rather than
   firing while images are legitimately still being drained.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests: add GPU/CPU equivalence test for ROI integration ... 188cbb659d

Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts
every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked
count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks),
negative pixel values (signed weighted-sum path), and an injected saturated
and masked pixel per ROI to cover the "max only" and "fully excluded" branches.

Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present,
mirroring ImageSpotFinderGPUTest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: live detector status bar + dataset-follow sync mode ... 56ddfaef96

Add a status-bar cluster, shown when connected over HTTP, that surfaces
the broker state and live acquisition info:
- broker state box (QProgressBar) with progress drawn as the bar fill,
  polled ~1 Hz on its own timer independent of image sync
- Live/Disconnected connection badge (host:port in tooltip)
- "+N new" badge and effective live-rate (Hz) readout
All widgets are fixed-width and only blanked in place, so the bar never
reflows when things appear/disappear.

Add a third autoload mode (HTTPSyncDataset): manually selecting an image
while following live now freezes the displayed image but keeps the dataset,
plots and image count updating. Surfaced via a tri-state HTTP Sync button
(off / live / data-only).

Also fix GetBrokerStatus() dropping the message field and a couple of
copy-pasted exception strings in JFJochHttpReader.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

broker/gen: gitignore generated cpp-pistache-server output ... b29b870ef2

Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is
hand-written on cpp-httplib, so the generated Pistache server stubs (api/,
impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are
regeneration debris that was previously left untracked and tended to carry a
newer spec version than the committed models. Ignore it so it can't be
committed by accident.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: guard Linux-only constructs for Windows/MSVC builds ... ec1308f4a9

Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux
build: D-Bus stays on by default and the XCB plugin is still used.

- CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on
  Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/
  adaptor + service file only when enabled. Select the platform integration
  plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead
  of hard-coding QXcbIntegrationPlugin.
- JFJochViewerWindow: wrap the adaptor include and D-Bus registration in
  #ifdef JFJOCH_VIEWER_DBUS.
- JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now
  #ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only
  headers are guarded too.
- Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the
  viewer (not defined by MSVC without _USE_MATH_DEFINES).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

plots: expose compression_ratio over the REST/UI plot path ... e1622c3750

The receiver already computes and stores the compression ratio per image
(JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio),
but the type was unreachable from the API: it was missing from ConvertPlotType,
the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and
no UI menu entry existed.

Wire it through all layers:
- broker: add the compression_ratio -> PlotType::CompressionRatio case in
  ConvertPlotType.
- spec: add compression_ratio to the plot_type parameter enum.
- frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what
  regeneration would produce), a "Compression ratio" menu entry, and a Y-axis
  label.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

receiver: count only images the pusher accepted ... 086129f767

images_sent was incremented right after image_pusher.SendImage(*loc), but the
ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it
silently drops the image (releases the slot and returns) when there is no live
connection or the 2 s enqueue deadline expires. So images_sent over-counted on a
broken/slow writer connection and disagreed with the ACK-based GetImagesWritten().

Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted
(enqueued/handed off) and only increment images_sent on success. The slot is
still released on the drop path. The authoritative delivered count remains
GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on
accept, preserving their previous always-counted behaviour.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frame_serialize: use reentrant gmtime in CBOR date decoding ... 52f349b9f1

gmtime() returns a pointer to a shared static buffer, so concurrent
deserialization of CborUnixTime_tTag values was a data race. Decode into a local
struct tm via gmtime_r (gmtime_s on Windows) instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

common: spell the azimuthal d-spacing unit d_A consistently ... 01a97d61fe

PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same
d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic
convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is
left as-is (controlled by the OpenAPI generator's C++ enum casing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

indexing: make the resolved-algorithm invariant explicit in the pool ... 1558dddbb8

IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(),
which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None;
the pool has no policy to resolve Auto itself. The worker handled a stray Auto
with a dead branch and silently produced no result when the resolved algorithm
had no matching indexer built.

- Document on GetIndexingAlgorithm() that it never returns Auto.
- Throw a clear internal error at the pool boundary if Auto ever arrives.
- In the worker, replace the dead Auto branch with a loud failure for any
  resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a
  host without a GPU), instead of returning no result silently.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Clean-up of OpenAPI code d0a1175c65

portability: replace M_PI / std::numbers::pi with a host+device-safe PI ... 5d9e1be814

M_PI is a POSIX <math.h> extension that MSVC does not define without
_USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is
C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several
common/ headers are pulled into .cu device translation units, so std::numbers is
not available there.

Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works
in host code (including MSVC), in CUDA device code, and under C++17/20, and use
it everywhere:
- common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files
- broker/OpenAPIConvert.cpp
- viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent
  convention across the codebase

Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frame_serialize: write CBOR length big-endian without bswap intrinsic ... bee35f7a25

AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length,
which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN.
That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available
on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the
swap and emitted a little-endian length on a little-endian host -> corrupt
frames.

Write the 8 length bytes most-significant-first directly. This is big-endian by
construction on any host, needs no byte-swap intrinsic and no endianness macro,
and is byte-for-byte identical to the previous output on little-endian hosts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

writer: drop unused <sys/stat.h> include from HDF5DataFile ... 57ae145555

Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies
on <filesystem>. Removing the dead POSIX include also avoids a needless
non-portable header for the eventual MSVC build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

mask: accept any-bit-depth TIFF for user mask upload ... 0f5271f14c

PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks
exported by tools like PyFAI (8-bit) failed. Route the upload through the
universal ReadTIFF reader and let PixelMask take a CompressedImage directly:
it validates the 2D shape against the detector's converted/raw layouts,
binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects
float/multi-channel images.

Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over
ReadTIFF, and the now-unused ReadTIFFFromString32 is removed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

internal generator: compare raw image width to RAW_MODULE_COLS ... a7c77f1f08

The raw stacked-module layout check compared the image width (cols) against
RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of
RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Merge upstream d14b8e9e18

Clean-up unused tools b71da6238e

VERSION: 1.0.0-rc.153 eb9b463263

frontend: generate OpenAPI client with @hey-api/openapi-ts ... ceb7d31eac

Replace the unmaintained openapi-typescript-codegen generator with
@hey-api/openapi-ts.

- openapi-ts.config.ts generates src/client from ../broker/jfjoch_api.yaml
  (client-fetch + typescript + sdk plugins; enums as const objects via
  mode: 'javascript', type names preserved)
- npm "openapi" script now runs openapi-ts; drop openapi-typescript-codegen
- migrate all imports from src/openapi to src/client
- DefaultService.foo() -> standalone foo({ body|query, throwOnError })
- OpenAPI.BASE='' -> client.setConfig({ baseUrl: '' }); ButtonWithSnackbar
  uses the relative path directly
- adapt hey-api's inlined enums: broker_status.state/message_severity and
  bit_depth_image/eiger_bit_depth become string/number literal unions;
  file_writer_format NXMX_* -> N_XMX_*
- remove generated src/openapi

Build (tsc + vite) passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

JFJochReceiverFPGA: Fix typo 7ed424a4ca

frontend: convert to function components + TanStack Query ... 237de5a0fc

Replace all class components with function components and move server-state
polling to @tanstack/react-query.

- add @tanstack/react-query; generate query helpers via the hey-api
  @tanstack/react-query plugin (src/client/@tanstack)
- QueryClientProvider + one-time client.setConfig({ baseUrl: '' }) in index.tsx
- App polls statistics with useQuery(getStatisticsOptions, refetchInterval),
  DataProcessingPlot uses getPreviewPlotOptions; manual setInterval polling gone
- memo() on presentational children; with TanStack structural sharing this
  re-renders a child only when its own statistics slice changes
- PreviewImage stays imperative (binary JPEG -> object URL) using useEffect +
  a ref for the object-URL lifecycle
- fix AzIntSettings correction checkboxes that mutated state in place (relied on
  the poll re-render); they now use setState
- drop dead code uncovered during the port (unused upload/deactivate handlers
  and imports in ImageFormatSettings, DetectorSettings, ROI)

Build (tsc + vite) passes; dev server transforms all entry modules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jfjoch_viewer: Fix status_bar, it looks nicer now c000b872bb

frontend: shared SettingsPanel + typed upload with error display ... 59da040e30

Generalize the expert-panel boilerplate (container + title + widgets +
upload button) and surface real upload errors via hey-api.

- SettingsPanel: consistent padded Paper + Typography title (one place to
  size it) + optional Upload button + result snackbar; PanelTitle and
  UploadSnackbarView exported for panels with bespoke layouts (DetectorSettings
  grid, ROI)
- useUpload(mutation): wraps a generated hey-api mutation, shows the server's
  error message on failure (errorMessage handles 400 string / 500 error_message)
- "unsaved changes" dot on panel titles, derived from the existing
  last-downloaded snapshot (dirty = !isEqual(value, lastDownloaded))
- fix PixelMask: TIFF upload was swallowing errors; now shows them
- config panels (FileWriter, ImageFormat, Detector, AzInt, DarkMask, Indexing,
  Instrument, ZeroMQ, ROI, DetectorSelection) upload through the typed SDK
  instead of ButtonWithSnackbar's raw fetch; status/display panels share the
  same shell for consistent spacing
- ButtonWithSnackbar now only powers bodyless action buttons (pedestal/cancel/
  initialize, start/trigger, deactivate)

Build (tsc + vite) passes; dev server transforms all modules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: orange Upload button when panel is dirty ... 5d8e33d52c

Match the unsaved-changes dot: the Upload button uses the warning color
while the panel has unedited changes, primary otherwise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: add zod plugin to hey-api openapi-ts ... 9fc971c1d2

Generate Zod schemas alongside the typed client (src/client/zod.gen.ts),
picking up the OpenAPI field defaults. Enables runtime validation and
default-filling for form payloads.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: drawer-based navigation shell ... 4f9452736a

Replace the stack of toggle-revealed sections with a left navigation
drawer (collapsible, expanded by default) clipped under a persistent
status bar. Dashboard (plots + measurement statistics + spot-finding)
is the default view; every other panel is its own drawer entry, so one
thing shows at a time.

- Group entries under Acquisition / Expert / Reference subheaders;
  ROI folded into On-the-fly processing.
- Pale-indigo drawer with a lime accent on the selected entry.
- Gray out JUNGFRAU calibration / FPGA status when their data is empty.
- API reference and documentation open in-app via iframe.
- Fixed footer pinned to the page bottom.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: add missing React keys in DetectorSelection and ROI ... a76aac65b7

DetectorSelection rendered the info-table rows and the detector-select
menu items without keys; ROI's grid action cells likewise. Add stable
keys to silence the warnings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: add CLAUDE.md with build/test/architecture, code-style and portability guidance ... 88f97fc197

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

common: add azimuthal-angle (phi) sectors to ROIAzimuthal ... 9e57219b31

ROIAzimuthal can now be restricted to an azimuthal-angle sector in
addition to its Q/d range, enabling STXM-style directional ROIs. phi
bounds are optional (both-or-neither); absent means a full 360 ring, so
existing Q-only azimuthal ROIs are unchanged.

- ROIElement::CheckROI gains a phi_deg argument; box/circle ignore it.
- MarkROI gains an optional phi_map; ROIMap builds it from Phi_rad
  alongside the resolution map only when azimuthal ROIs are present.
- phi bounds are normalized to [0,360) and wrap-around sectors are
  supported (phi_min > phi_max).
- ROIConfigAzim carries phi_min/phi_max (kept trivially copyable for the
  union; phi_min == phi_max means full ring).

No phi crosses the wire yet (CBOR/API wiring follows separately).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

api: serialize azimuthal ROI phi sector over OpenAPI and CBOR ... 1d04656dc8

Expose the optional phi_min_deg / phi_max_deg azimuthal-sector bounds
added to ROIAzimuthal through the wire formats. Both are optional, so
older clients and streams (Q-only ROIs) remain valid.

- jfjoch_api.yaml: optional phi_min_deg / phi_max_deg on roi_azimuthal;
  regenerated the cpp-pistache server model and the TS frontend client.
- OpenAPIConvert: map phi both directions (set only when present).
- CBOR start message: emit phi_min/phi_max only for a sector; on decode,
  absent phi means a full ring.
- Documented the new optional CBOR fields; extended the CBOR round-trip
  test with a sector and a full-ring azimuthal ROI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

writer: save ROI bitmap and definitions in the master file ... 03814b1425

Persist regions of interest so an acquisition's ROI layout can be
recovered from the master (_master.h5 / _process.h5):

- /entry/roi/roi_map: uint16 per-pixel bitmask (bit i == ROI i).
- /entry/roi/<name>: logical definition (type + geometry params, incl.
  azimuthal phi sector) plus bit_index tying it to the bitmap plane.

The bitmap rides along in the StartMessage (CBOR, mirroring az_int_map)
and is filled both online (JFJochReceiver::SendStartMessage) and offline
(jfjoch_process). Definitions come from the already-transmitted rois.
Both are dataset-wide metadata, so they are written by the NXmx master
writer (new NXmx::ROI), not the per-image data-file plugin.

Documented the /entry/roi layout in docs/HDF5.md and the master-vs-data
writer convention in CLAUDE.md; added a CBOR roi_map round-trip test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

preview: draw azimuthal ROIs ... 2bd276ae1b

AddROI now renders azimuthal ROIs in addition to box and circle ROIs:
inner/outer arcs at the Q (d) bounds and, for a sector, the two radial
edges at the phi bounds. The per-pixel phi comes from the same
DiffractionGeometry::Phi_rad used to assign ROIs, so the overlay matches
the ROI footprint exactly (including wrap-around sectors). Full-ring
azimuthal ROIs draw as two concentric arcs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

writer/reader: move ROI definitions to /entry/roi_defs and read them back ... c6e7f8aa15

Relocate the ROI definitions and the roi_map bitmap from /entry/roi to a
separate /entry/roi_defs group in the master file. /entry/roi keeps its
original meaning (per-image ROI results only), so an older reader that
iterates /entry/roi is not disturbed by the bitmap/definition entries
(which would otherwise make it try to read /entry/roi/roi_map/max and
fail to open the file at all). Back-compatibility for already-deployed
viewers is the reason for the split.

JFJochHDF5Reader now reads the logical definitions into experiment.ROI()
and the roi_map bitmask (+ a name->bit index) into the dataset, for all
file formats (the master always carries them). Reading both lets a later
viewer either re-derive ROIs with the current geometry (logical) or show
the exact written footprint (bitmap).

Added a write/read round-trip test over VDS and integrated formats.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

writer: link ROI results into the VDS master file ... 6b95600260

In VDS mode the per-image ROI results (max/sum/sum_sq/npixel/x/y) are
written into the data files but were not exposed in the master, so a VDS
master surfaced no ROI statistics. Add virtual datasets under
/entry/roi/<name> in LinkToData_VDS, one group per ROI, mirroring how the
spot-finding and azimuthal-integration arrays are linked. Integrated and
legacy formats are unaffected (the results are already reachable there).

Extended the reader round-trip test to write real ROI results and check
they read back from the master for both VDS and integrated formats.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

common: take ROIAzimuthal phi as plain floats, not optionals ... eadbce51bd

The phi bounds were stored as plain floats internally but passed through
std::optional in the constructor, which obscured the intent. Take them
as float arguments defaulting to 0 instead. The convention is explicit:
phi_min == phi_max means the full ring (all angles), a sector wraps
across 0 when phi_min > phi_max, and otherwise phi_min <= phi_max. The
"both bounds or none" rule belongs at the API/file boundary
(OpenAPIConvert, reader), not in the core class, so it lives there now.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: bundle Sphinx docs locally instead of linking readthedocs ... cee4739221

The Documentation tab pointed at jungfraujoch.readthedocs.io, which lags
behind the installed version. Build the Sphinx docs into frontend/dist/docs
and serve them at /frontend/docs so they always match the package.

- make_doc.sh: accept an output dir (default public), absolutize it, run
  from the script dir, and allow $PYTHON override
- package.json: add "docs" script building into dist/docs
- CMakeLists: run "npm run docs" in the frontend target so CPack packs the
  docs via the existing dist/ install
- App.tsx: point the Documentation DocFrame at /frontend/docs/index.html

Also clean up the Sphinx build (911 -> 0 warnings):
- fix typo myst_heading_anchor -> myst_heading_anchors so cross-doc
  #anchor links actually resolve
- suppress myst.header/myst.xref_missing (auto-generated OpenAPI client
  docs only) and the harmless sphinx_material config.cache note
- drop a dangling "---" transition at the end of CPU_DATA_ANALYSIS.md
- add REPOSITORIES and the generated python_client model pages to toctrees

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: render Changelog inline instead of in a Sphinx iframe ... 6962a61489

The framed Sphinx changelog reused MUI with the same colours, so it read as
a picture-in-picture. Bundle docs/CHANGELOG.md at build time (?raw) and render
it with react-markdown, themed to match the app, for a quick frame-less view.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: link PSI in footer, add Source code and License to References ... 332fd32224

- Footer "Paul Scherrer Institute" now links to https://www.psi.ch
- Reference drawer gains a "Source code" entry linking out to the Gitea
  repository (new nav-item href that opens in a new tab)
- Reference drawer gains a "License" entry rendering the nicely formatted
  docs/LICENSE.md inline (generalises the changelog renderer into MarkdownDoc)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

viewer: draw loaded ROIs on the diffraction image ... 35f6ffdde9

When a file with ROI definitions is opened (read into experiment.ROI()
by the HDF5 reader), the diffraction image now overlays the configured
box and circle ROIs as distinct colour-coded outlines, alongside the
existing resolution rings and spots. This is the first step of the
ROI-map-based multi-ROI canvas: showing the ROIs loaded from a file.

Azimuthal ROIs (wedge rendering) and per-ROI statistics from the bitmap
follow in subsequent steps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: draw azimuthal ROI wedges, with optional fill and labels ... 7d6f79a1da

Azimuthal ROIs now render on the diffraction image as annular sectors
(or full-ring annuli), sampled through DiffractionGeometry::ResPhiToPxl
so the outline follows the ROI footprint, including wrap-around sectors.

Add two side-panel toggles (both default off): a translucent fill for
every ROI (helpful when outline colours clash with the image colour map,
and with many ROIs) and ROI name labels (constant on-screen size). Wired
side panel -> window -> diffraction image like the existing feature
toggles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: fix stray line from origin on full-ring azimuthal ROI ... 33118ebe8b

The full-ring annulus built the inner ring as a separate QPainterPath and
merged it with addPath, which stitched a spurious segment from (0,0) to
the ring. Build both concentric rings as subpaths of one path, each begun
with an explicit moveTo, so there is no connecting segment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: per-ROI statistics on the diffraction image via the CPU ROI engine ... 3996595606

When ROI labels are shown, each ROI's label now also reports its sum,
max and pixel count for the current image. Rather than reimplementing the
accumulation, this reuses the existing ROIIntegrationCPU engine (the
software counterpart of the FPGA roi_calc), built from the experiment and
cached per dataset. A small adapter folds the viewer's gap sentinel
(INT32_MIN+1) onto the engine's masked sentinel (INT32_MIN) so masked and
saturated pixels are handled correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

image_analysis: surgical ROI-only entry points in MXAnalysisWithoutFPGA ... a433399106

Add three entry points so ROI statistics can be (re)computed without a
full re-analysis, in support of interactive ROI editing in the viewer:

- RebuildROI(): recreate the ROI engine after the ROI set changes (the
  CudaStream is now kept as a member so the GPU engine can be rebuilt).
- AnalyzeROIOnly(): decompress + preprocess + ROI, skipping azimuthal
  integration, spot finding and indexing (a new image when re-analysis
  is off).
- RunROIOnly(): rerun only the ROI integration on the already-preprocessed
  image (an interactive ROI move). A full Analyze() still computes ROIs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: worker can update ROIs and recompute them without full re-analysis ... 4e45680e96

The reading worker's experiment is the source of truth for ROIs. Add a
SetROIDefinition slot that updates curr_experiment.ROI(), rebuilds the
analysis ROI engine, mutates the dataset (so the canvas reflects the new
ROIs) and recomputes only the ROIs for the current image via RunROIOnly.
On image load when full re-analysis is off, ROIs are still computed via
AnalyzeROIOnly so the statistics stay current.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: ROI list panel — select, add, rename, delete, per-ROI results ... 2f8d486b51

Add JFJochViewerROIList to the side panel: a list of the dataset's ROIs
with selection, add (box/circle/azimuthal), rename and delete, plus a
statistics readout (sum/mean/std/max/valid/masked/centre-of-mass) for the
selected ROI, taken from the analysis output for the current image. Edits
emit a full ROIDefinition, routed to the worker's SetROIDefinition.

Per-ROI statistics now live in this panel rather than the canvas labels;
the diffraction image's labels show only the ROI name, and the ad-hoc
ROIIntegrationCPU computation there is removed in favour of the analysis
pipeline. The result widget now reports std dev instead of variance.

The single-ROI scratch panel remains for now and will be retired once the
interactive canvas editing replaces it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: ROI panel refinements — persist edits, combobox, compact toggles ... e49a908862

- Edited ROIs now override the file's for every subsequently loaded image
  and survive settings changes, until a new file is opened (previously a
  delete/add was lost on the next image). Tracked via an roi_override_ that
  is re-applied to curr_experiment and to each loaded image's dataset.
- The ROI selector is a compact combobox instead of a list widget.
- "Show labels" and "Translucent fill" share one row to save space.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Harmonize licsense files 0c780a5cbb

viewer: move the selected box/circle ROI on the canvas ... 308265c7b1

Add base-class mouse hooks (roiEditPress/Move/Release) so the diffraction
image can edit the persistent ROI selected in the side-panel combobox.
The selected ROI is highlighted (dashed, thicker), and dragging its
interior moves a box or circle; on release the new geometry is committed
through SetROIDefinition, which recomputes only the ROIs. Resize,
azimuthal handles and delete-key follow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: click to select an ROI, with live move and a hand cursor ... d9dfd36413

Clicking a box/circle on the image now selects it (syncing the side-panel
combobox) and grabs it for moving, with a closed-hand cursor during the
drag. The move recomputes ROI statistics live rather than only on release,
throttled to at most one in-flight recompute so the worker is not flooded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: resize box/circle/azimuthal ROIs and delete with the Delete key ... 00b58cd426

Extend canvas ROI editing beyond move:
- Box: drag corners/edges to resize (handles shown on the selected ROI).
- Circle: drag the perimeter to resize, interior to move.
- Azimuthal: drag the inner/outer arc to change the Q/d range, or a radial
  edge to rotate a phi bound; sampled through the geometry so it tracks
  the conic.
- Delete removes the selected ROI (the view now takes keyboard focus).

All edits go through the same live, throttled SetROIDefinition path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: azimuthal ROI handles, and keep panning inside the ROI ... d413af0e92

Azimuthal ROIs now show discrete grab handles on the selected ROI: the
inner/outer arc (resize Q/d) and, for a sector, the two phi edges
(rotate). Editing is by clicking a handle, with a generous tolerance,
which fixes the previously near-impossible phi-edge grab. Clicking the
(large) interior of an azimuthal ROI now only selects it and lets the
view pan, instead of capturing the gesture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: numeric editing of the selected ROI's bounds ... 1d015145a4

The ROI panel now shows editable numeric fields for the selected ROI,
labelled by type: box min/max X/Y, circle centre/radius, or azimuthal
Q-range and phi sector. Editing a field rebuilds that ROI and commits it
through the same SetROIDefinition path, so all three types can be created
(via +) and dialled in by typing, not only by dragging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: shift/ctrl-drag creates list ROIs; remove the old scratch panel ... 142cb88aa8

The interactive shift-drag (box) / shift+ctrl-drag (circle) now creates a
new persistent ROI in the list instead of feeding the old single-ROI
scratch panel. The base emits a roiScratchDrawn hook on release; the
diffraction image turns the drawn shape into a named ROI committed via
SetROIDefinition.

The old JFJochViewerImageROIStatistics scratch panel and all its wiring
(box/circle configuration, single-ROI result, add/subtract user mask) are
removed from the side panel and window; the ROI list is now the single
source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: download/upload ROIs to the broker ... b2c60dfbd0

Add Download and "Upload to server" buttons to the ROI panel. The HTTP
reader gains GetROIDefinitions (GET /config/roi) and UploadROIDefinitions
(PUT /config/roi), converting between ROIDefinition and the generated
Roi_definitions model (including the optional azimuthal phi sector), the
same shapes OpenAPIConvert uses on the server. Download applies the
fetched ROIs through SetROIDefinition; upload pushes the current ones.
Both are no-ops unless the viewer is connected to a broker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: load user mask from TIFF (replace or add) ... fef9c2f4c3

Two Mask-menu entries read a 16-bit TIFF (via ReadTIFFFromString16) and
apply it as the user mask: "replace" overwrites the current mask, "add"
unions the TIFF into it. The TIFF must match the detector dimensions; any
non-zero pixel masks. Both go through UpdateUserMask_i like the other
mask edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

make_doc.sh: Use default python d4c96d48fc

docs: ROIs/azimuthal apply to both FPGA and DECTRIS workflows ... 52aca9a4a9

Clarify in FPGA_DATA_ANALYSIS.md that the ROI and azimuthal definitions
are also evaluated on CPU via the shared image_analysis library, so they
apply to the DECTRIS SIMPLON (EIGER) path that has no FPGA, not only the
FPGA-accelerated JUNGFRAU/PSI path. Document the three NXmx writer formats
(Legacy/VDS/Integrated) and the two acquisition workflows in CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: remove the dead scratch-ROI widgets and worker methods ... 81b48febcf

With the scratch ROI panel gone, delete the now-orphaned
JFJochViewerImageROIStatistics(_Box/_Circle) widgets and the worker's
unused scratch-ROI plumbing (SetROIBox/SetROICircle, the ROIElement
member, AddROIToUserMask/SubtractROIFromUserMask). ROI-to-mask, if wanted
later, should act on the selected list ROI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: add/subtract the selected ROI to/from the user mask ... abc1d1c079

Restore the ROI-to-mask action on the new list: "Add to mask" and
"Subtract from mask" buttons rasterise the selected ROI into the user
mask (set or clear), through the same UpdateUserMask_i path. The ROI is
evaluated with per-pixel resolution and phi from the geometry, so box,
circle and azimuthal (sector) ROIs all map correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: white, compact, right-aligned ROI numeric fields ... e60b42ea83

The ROI bound editors now have a white background, a capped width and
right-aligned text, which reads better for numeric entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: phi_min/phi_max columns in the azimuthal ROI table ... e9e5ec79cd

Surface the optional azimuthal phi-sector bounds (phi_min_deg/phi_max_deg)
as editable columns, so a sector can be set from the web UI, not only the
viewer. Also fix the Q_max row's d header, which read d_max but is d_min.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontend: auto-fill the missing phi bound for a sector ... a34a033c99

When only one azimuthal phi bound is entered, fill the other so the
sector is well-defined and the server accepts it: a missing phi_min
becomes 0 and a missing phi_max becomes 360. Leaving both empty still
means a full ring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

make_doc.sh: Use python3 4e9505e6bb

make_doc.sh: prefer the newest available python3.x ... 3ce7666738

The bare python3 is often too old for the Sphinx build. Auto-select the
newest python3.13/3.12/3.11/3 found on PATH, still allowing $PYTHON to
override.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Docker: Update 2606 to support python3.12/3.13 1e53db8f64

cmake: drop GCC-only hardcoded flags, bump CUDA to C++20 ... 74c2c808de

Make the build less hostile to MSVC (needed for the viewer port) and
align the CUDA standard with C++.

- Remove the CMAKE_C/CXX_FLAGS_RELEASE overrides: -O3/-DNDEBUG only
  duplicate CMake's own GCC/Clang Release defaults, and the SET()
  clobbered the MSVC defaults. The lone meaningful flag,
  -Wno-deprecated-enum-enum-conversion, merely silenced warnings from
  the vendored Xilinx ap_private.h in the FPGA path; with no -Werror
  it is just log noise, so drop it.
- CMAKE_CUDA_STANDARD 17 -> 20 (+ STANDARD_REQUIRED); CUDA >= 12.8
  already supports C++20.
- Drop the file-scoped -Ofast on JFPedestalCalc.cpp /
  JFConversionFloatingPoint.cpp: it is GCC-only and reaches the MSVC
  viewer via JFJochCommon -> JFCalibration. Those files now build at
  plain -O3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cmake: add JFJOCH_VIEWER_ONLY to build the viewer standalone ... f21c22d453

New option (default OFF) that compiles only jfjoch_viewer and the
libraries it transitively links -- common, image_analysis, writer,
reader, preview, image_puller, frame_serialize, gemmi_gph, compression,
jungfrau, and the OpenAPI model (JFJochAPI) -- skipping the Linux-only
online stack (broker service, receiver, fpga, acquisition_device,
detector_control, image_pusher, xds-plugin, tests, tools) and the
frontend. This is the standalone build path for the cross-platform
viewer.

JFJochAPI is the only piece the viewer needs from broker/, so the broker
service library and executable are gated behind NOT JFJOCH_VIEWER_ONLY.
The full build and JFJOCH_VIEWER_BUILD (viewer on top of the full build)
are unchanged -- that path is the verbatim ELSE() branch.

Verified on Linux: cmake -DJFJOCH_VIEWER_ONLY=ON configures and
`make jfjoch_viewer` links cleanly with the online stack absent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cmake: raise policy floor to 3.5 for FetchContent deps (CMake 4.x) ... 3c6439530d

CMake >= 4.0 (4.3.1 ships bundled with Visual Studio 2026) makes any
cmake_minimum_required(VERSION < 3.5) a fatal error. The fetched libzmq
still declares VERSION 3.0.2, which aborts configuration before our own
CMakeLists gets a say. Set CMAKE_POLICY_VERSION_MINIMUM=3.5 before the
FetchContent_MakeAvailable calls so those subprojects configure; the
variable is simply unused (harmless) on CMake < 3.30.

Verified against CMake 4.3.1: the real libzmq source fails without it and
configures with it, and an in-file SET() propagates into the
add_subdirectory child exactly as a -D cache entry does.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cmake: don't fetch sls/catch2 in a viewer-only build ... 0585fab9bf

Under JFJOCH_VIEWER_ONLY, only the viewer's transitive deps are built, so
sls_detector_package (detector_control/receiver) and catch2 (tests) are
never used -- and sls does not configure under MSVC, which is the whole
point of the viewer-only mode. Skip fetching just those two; zstd, hdf5,
spdlog, libzmq and httplib are still required (JFJochReader links httplib).

Verified: cmake -DJFJOCH_VIEWER_ONLY=ON configures with sls/catch2 absent
and `make jfjoch_viewer` still links.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Update to most recent bitshuffle version fcbe19103d

cmake: fetch libtiff and FFTW instead of relying on system libs ... 0220c6376d

Both are leaf deps (JFJochPreview / JFJochIndexing) that only our own code
looks for, so building them ourselves removes the system-package lottery and
gives a reproducible, statically-linked build on every platform.

- libtiff: FetchContent, library only (jbig/zstd/lzma/jpeg/old-jpeg/tools/
  tests off). The C++ binding (TIFF::CXX / libtiffxx) is packaged
  inconsistently across distros -- missing on Rocky 9 -- and absent on
  Windows, so find_package(TIFF COMPONENTS CXX) was unreliable; that call is
  removed and JFJochPreview links the tiff/tiffxx targets directly. GitHub
  mirror because upstream (gitlab) is unreachable from some restricted hosts.
- FFTW: FetchContent single precision (ENABLE_FLOAT) from the release tarball
  -- the git repo ships no pre-generated codelets (needs OCaml genfft). It's
  now always available, so the CPU FFT indexer is always built and
  JFJOCH_USE_FFTW always defined; the "FFTW disabled" path is gone. Static
  (libfftw3f.a) via the global BUILD_SHARED_LIBS OFF.

Verified on Linux: jfjoch_viewer builds and links libfftw3f.a + libtiff*.a,
all static.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cmake: enable FFTW SSE2/AVX/AVX2 codelets (x86, runtime-dispatched) ... 53645434be

FFTW selects SIMD codelets at runtime via cpuid, so enabling AVX2 does not
require an AVX2 CPU -- it just speeds up the CPU FFT fallback indexer on
capable hardware and falls back to SSE2/scalar otherwise. Arch-guarded to
x86_64/AMD64 (ARM would need ENABLE_NEON). AVX-512 left off on purpose:
it's x86-64-v4 (not our v3 target), the FFT gain over AVX2 is marginal,
older Intel downclocks on it, and it's the riskiest to build under MSVC.

Verified on Linux: SSE2/AVX/AVX2 codelets compile, libfftw3f.a stays a
single static archive, jfjoch_viewer relinks cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cmake: use upstream libtiff (gitlab) at v4.7.1 ... 5a6485da3a

Switch the libtiff FetchContent from the personal GitHub mirror to the
upstream gitlab repo and bump v4.6.0 -> v4.7.1 -- cleaner provenance; the
mirror was only ever a network-reachability workaround.

Verified on Linux: fetches, builds static (libtiff.a/libtiffxx.a), and
jfjoch_viewer links cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Separate JFJochWriter and JFJochStreamWriter c9803ccbfd

CMake: Remove image_puller dependency from the jfjoch_viewer 07ce8b785a

viewer: fix MSVC compatibility in viewer-reachable code ... ebc28744e2

Address portability issues found building jfjoch_viewer with MSVC:

- common/{ADUHistogram,AzimuthalIntegrationProfile}.h and
  image_analysis/spot_finding/StrongPixelSet.h only need DeviceOutput,
  so include fpga/pcie_driver/jfjoch_fpga.h (plain-C, self-contained)
  directly instead of acquisition_device/AcquisitionDevice.h, which
  dragged <unistd.h> into the viewer tree.
- common/time_utc.h: guard gmtime_r/timegm/localtime_r with the MSVC
  equivalents (gmtime_s/_mkgmtime/localtime_s) under _WIN32; drop the
  duplicated includes and add the headers used directly.
- gemmi_gph/gemmi/utf.hpp: vendor the upstream gemmi header; fileutil.hpp
  includes it on Windows for UTF8_to_wchar but it was never vendored.
- writer/HDF5Objects.cpp: ExtractFilename returns path.filename().string()
  (std::filesystem::path has no implicit conversion to std::string on
  Windows, where it is wchar_t-based).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CMake: build StreamWriter only into JFJochStreamWriter ... bcd588b400

StreamWriter.cpp was compiled into both JFJochWriter and JFJochStreamWriter,
so JFJochWriter still transitively pulled in image_puller headers via
StreamWriter.h. Compile it only into JFJochStreamWriter (which already
links JFJochImagePuller); targets needing the streaming writer (jfjoch_writer,
jfjoch_test) link that lib, while JFJochWriter consumers (viewer, tools,
ImagePusher) no longer drag in the puller. Also drop a stray duplicate
HDF5DataFile source listing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ADUHistogram: Fix missing exception header a872b68d6f

CMake: stop libtiff/httplib pulling in webp, sharpyuv, brotli ... b888c53509

libtiff's webp and lerc codec options default ON whenever their library is
found on the build host, dragging in libwebp + libsharpyuv (webp) and libLerc.
We only need DEFLATE (zlib) + the internal LZW codec, so turn both OFF.

httplib likewise auto-enables Brotli content-encoding when system Brotli is
present; gzip/zlib is enough for the broker, so disable it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Fix FFTW build-tree header wiring and two MSVC warnings ... 4a39f2cfcf

FFTW exposes fftw3.h only via $<INSTALL_INTERFACE:include>, so consuming
fftw3f from the build tree (FetchContent) leaves FFTIndexerCPU.h unable to
find the header. A system-installed fftw3.h masks this on Linux, so it only
bit on Windows; add the source-tree api/ dir to fftw3f's interface includes.

Also two viewer-reachable warnings:
- time_utc.h (C4477): %03ld was fed a chrono milliseconds::rep (64-bit), not
  a long; cast the 0-999 value to int and use %03d.
- HDF5Objects.h (C4700): value-initialize the scalar read buffer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

reader: fix Windows path handling in JFJochHDF5Reader ... 025c6394b6

Two spots used POSIX-only path logic that breaks on Windows:

- dataset_name() located the basename with rfind('/'), which misses
  backslash separators, so a C:\...\foo_master.h5 file prefix kept the
  whole directory. Use std::filesystem::path::filename() instead.
- Legacy-format data-file resolution joined paths with
  fmt::format("{}/{}", ...), producing mixed \/ separators and prepending
  the master directory even to absolute link targets. Reuse the existing
  ResolveRelativeToMaster() helper, which joins via std::filesystem and
  leaves absolute targets untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md: record Windows viewer goal of MSVC + CUDA, and unbundled deps ... 1ae2e6869d

The recurring goal is a Windows jfjoch_viewer built with MSVC *and* CUDA ON
(GPU processing is wanted, not a CPU-only fallback); the non-CUDA path is the
macOS fallback. Also note the three deps a self-contained Windows build still
needs but FetchContent does not auto-provide: ZLIB (bundle zlib-ng as
ZLIB::ZLIB), libjpeg-turbo (ExternalProject), and Eigen (header-only, also
needed by Ceres) -- each guarded so the Linux build is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CMake: Ensure static ZLIB (must be before FIND_PACKAGE) 7eb927e82f

jfjoch_process: Missing include <set> 4737eff1a9

CMake: vendor Eigen (5.0.1) and libjpeg-turbo so the build is self-contained ... cd394c5249

Two deps the Linux build picked up from the system were absent on Windows.
Bundle both, unconditionally (matching how libtiff/FFTW are vendored), so the
tree builds without system Eigen/JPEG:

- Eigen 5.0.1 via FetchContent with OVERRIDE_FIND_PACKAGE, so every
  find_package(Eigen3) -- ours, Ceres', and ffbidx' -- resolves to this copy and
  the requested 3.4 version is satisfied by the newer major. ffbidx's bundled
  eigen submodule is disabled (GIT_SUBMODULES "" on fast-indexer) so it no longer
  creates a duplicate Eigen3::Eigen target. Verified on a CUDA build: Ceres 2.3.0,
  the analysis libs, and ffbidx's CUDA code all compile against 5.0.1.

- libjpeg-turbo 3.0.4 via ExternalProject (upstream discourages add_subdirectory);
  built+installed static and imported as JPEG::JPEG, replacing find_package(JPEG)
  in preview/. SIMD is lazy: WITH_SIMD=ON + REQUIRE_SIMD=OFF means NASM-accelerated
  when nasm is present, otherwise a warning and a scalar build (no hard failure).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CMake: fix CI -- export vendored Eigen so Ceres' install(EXPORT) resolves ... d4d2d0d8ad

A clean configure failed at generation:
  install(EXPORT "CeresExport" ...) includes target "ceres" which requires
  target "eigen" that is not in any export set.

Ceres' install(EXPORT CeresExport) is unconditional, and its ceres target
transitively links the eigen target. As a FetchContent subproject Eigen leaves
EIGEN_BUILD_CMAKE_PACKAGE OFF (it defaults to PROJECT_IS_TOP_LEVEL), so its
install(EXPORT Eigen3Targets) never runs and eigen ends up in no installed
export set. Force EIGEN_BUILD_CMAKE_PACKAGE ON so eigen is exported and Ceres'
export validates. Install/export-only change; the build is unaffected.

Verified with a clean configure (fresh CMakeCache): configure + generate now
succeed where they previously errored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: macOS .app / Windows GUI packaging + Qt deploy + CPack generators ... 31630c20a1

Add the platform packaging the viewer was missing so it can ship as a .dmg
(macOS) or installer (Windows):

- viewer: set MACOSX_BUNDLE (a .app bundle on macOS) and WIN32_EXECUTABLE
  (GUI subsystem, no console window) -- both ignored where they don't apply;
  install with BUNDLE/RUNTIME destinations.
- viewer: qt_generate_deploy_app_script + install(SCRIPT) to bundle the Qt
  runtime at install time (windeployqt on Windows, macdeployqt on macOS).
  Gated on Qt >= 6.5, which is where the OUTPUT_SCRIPT signature and the Linux
  deploy path landed; with a static Qt (our Linux builds) it is a no-op.
- CPack: select DragNDrop (.dmg) on macOS and NSIS on Windows, ahead of the
  Linux-only /etc/* probes; DEB/RPM path unchanged.

Verified: configures + generates cleanly on Linux with the viewer enabled
(Qt 6.4.2 -> deploy step correctly skipped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docker: unify static Qt 6.9.1 on all 4 Linux images, drop dead prebuilds ... da5fff71a9

The jungfraujoch CMake now builds HDF5 + libtiff (FetchContent) and
libjpeg-turbo + Eigen (ExternalProject / OVERRIDE_FIND_PACKAGE) itself, so the
per-image pre-builds and system -dev packages for those were dead weight (the
app rebuilt its own copies regardless). Consolidate dependency management into
the one CMakeLists at the cost of longer build time (build env has network).

- Qt: pin static 6.9.1 everywhere. ubuntu2204 was on 6.10.0 and ubuntu2404 used
  the OS Qt (~6.4, dynamic) -- the latter is now a static 6.9.1 build like the
  others, so every image ships a self-contained, identical-Qt viewer.
- Remove the HDF5/libtiff/libjpeg-turbo/Eigen source builds + their system -dev
  packages + the stale /opt/* CMAKE_PREFIX_PATH/PKG_CONFIG_PATH entries.
- nasm: now needed by the project's libjpeg-turbo ExternalProject; added to
  ubuntu2404 (the others already had it).
- Standardize the Qt configure flags across all four (xcb, xcb_xlib=OFF,
  no Wayland -- the qtwayland module was never built, so wayland_client=ON on
  ubuntu2204 was a no-op).

Untested here (no Docker on this host) -- needs a build in each image; the
ubuntu2404 static-Qt dep list in particular should be sanity-checked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docker: drop fftw too -- the project builds FFTW itself ... 8cd46751f9

FFTW is fetched and built by the jungfraujoch CMake (single-precision, from the
release tarball), so the system fftw packages (fftw-static/fftw-devel on Rocky,
libfftw3-dev on Ubuntu) were unused. Remove them from all four images.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CMake: support DGX Spark (Blackwell GB10 sm_121 + aarch64 FFTW NEON) ... 9c9f818ea6

- CUDA: append sm_121 (Blackwell GB10 / DGX Spark) to CMAKE_CUDA_ARCHITECTURES
  when nvcc >= 12.9 knows it. The static list tops out at sm_120 and embeds no
  PTX, so without this a build would fail to launch on Spark ("no kernel image
  available"); guarded so CUDA 12.8 (which predates sm_121) is unaffected.
- FFTW: enable NEON codelets on aarch64 (Grace), mirroring the x86 SSE2/AVX/AVX2
  branch, so single-precision FFT is SIMD-accelerated instead of scalar on ARM.

Both are inert on x86 / CUDA 12.8; verified configure still succeeds there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jfjoch_viewer: Show detected GPUs in the about box a89a4c5beb

image_analysis: force Ceres to build without CUDA ... 5064b5ba47

USE_CUDA is a Ceres cache STRING (default "default" = auto-detect), not an
option(), so a plain SET was shadowed by the cache default and CUDA got
enabled on MSVC. Force it OFF in the cache instead.

Also drop two dead settings: MINIGLOG (Ceres uses abseil logging now, no
longer referenced) and EIGENSPARSE ON (already Ceres' default).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CMake: package the viewer component in JFJOCH_VIEWER_ONLY builds ... 5247acc07e

A viewer-only build builds jfjoch_viewer but the CPack block hard-coded
CPACK_COMPONENTS_ALL to "jfjoch writer" and only appended "viewer" when
JFJOCH_VIEWER_BUILD was set -- which a pure viewer-only build does not set.
cpack then tried to package the empty jfjoch/writer components and failed,
requiring a manual -D CPACK_COMPONENTS_ALL=viewer.

Branch the component list on build mode: viewer-only packages just "viewer";
other modes keep "jfjoch writer" plus driver-dkms/viewer when their flags are
set. Also drops the redundant re-set of the default list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

JFJochReceiverPlots: Move to common/, so it can be linked without bringing JFJochReceiver HW dependencies e6c6536e3a

viewer: Windows NSIS / macOS installer polish + bundle analysis CLIs ... f9396e01e7

NSIS installer was missing several things on Windows:
- License: show the repo GPLv3 LICENSE as the click-through license page
  (CPACK_RESOURCE_FILE_LICENSE).
- Start Menu: no shortcut was created -- add CPACK_PACKAGE_EXECUTABLES for
  jfjoch_viewer, and embed an app icon so the shortcut is not generic. The
  icon is a new multi-size jfjoch.ico (256 PNG + 48/32/16) compiled into the
  .exe via resources/jfjoch.rc; macOS gets the matching jfjoch.icns copied
  into the .app bundle (MACOSX_BUNDLE_ICON_FILE). Both derived from
  resources/jfjoch.png, padded square with transparent bands.
- Naming: install folder, Start Menu group and Add/Remove Programs entry are
  now "Jungfraujoch" instead of "jfjoch <version>".

Also ship the offline analysis CLIs (jfjoch_process/scale/azint/extract_hkl)
in viewer-only / Windows packages: JFJochReceiverPlots now lives in
JFJochCommon, so their stale JFJochReceiver link is dropped, leaving only the
portable libs (reader/image_analysis/writer/common). tools/CMakeLists.txt is
split so those four build everywhere while the hardware tools stay gated
behind NOT JFJOCH_VIEWER_ONLY, and the viewer-only branch now adds tools/.
They install into the "viewer" CPack component.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: drop unused CMAKE_AUTORCC ... e035f7c303

The viewer's only .qrc is compiled explicitly via QT_ADD_RESOURCES into
APP_RESOURCES; no .qrc is added directly as a target source, so AUTORCC never
acted on anything. Remove the dead flag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

licenses: add third-party notices, attribution texts, and viewer license window ... e5034d0a2c

Acknowledge all bundled third-party software and satisfy attribution/notice
requirements, while keeping it maintainable:

- THIRD_PARTY_NOTICES.md: human-readable manifest (component, copyright, SPDX
  license, link) for fetched, vendored, and runtime/SDK dependencies.
- licenses/: verbatim license texts; COLLECT.sh regenerates them from the
  build trees and system SDK locations.
- Bundle the verbatim Qt LGPL-3.0 text and the CUDA Toolkit 12.8 EULA.
- frontend: self-contained npm attribution generator (`npm run licenses` ->
  dist/THIRD_PARTY_LICENSES.txt), wired into the frontend build target.
- Install LICENSE + notices + licenses/ into share/doc/jfjoch for every
  packaged component.
- viewer: Help > "Third-party Licenses" window (QTextBrowser) showing a
  generated, self-contained HTML built from licenses/.
- docs/SOFTWARE.md: drop the stale hand-kept dependency lists; point at the
  manifest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

windows: vendor wingetopt and scope server-only targets out of viewer-only builds ... 1ea5897fca

MSVC/Windows build fixes:

- Force JFJOCH_VIEWER_ONLY ON for Windows and macOS; the broker/receiver/FPGA/
  writer server stack is Linux-only, so those platforms build the viewer subset.
- Vendor an OpenBSD/NetBSD getopt/getopt_long shim (tools/wingetopt/, BSD/ISC)
  as a WIN32-only static lib -- the MSVC CRT has no <getopt.h> -- and link it
  into the four portable CLI tools on Windows.
- jfjoch_extract_hkl.cpp: include <getopt.h> explicitly (it relied on glibc's
  transitive <unistd.h>).
- jfjoch_process.cpp: guard <unistd.h> with #ifndef _WIN32.
- Guard JFJochStreamWriter and jfjoch_writer with IF(NOT JFJOCH_VIEWER_ONLY):
  the former pulls in JFJochImagePuller, the latter uses a fork()/waitpid()
  multi-process design; neither belongs to the portable viewer subset.
- Record wingetopt in THIRD_PARTY_NOTICES.md and licenses/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CMake: provide Eigen externally again; fix cpack NSIS icon escape ... 0c1babd0ca

Revert Eigen from the FetchContent OVERRIDE_FIND_PACKAGE vendoring (cd394c52,
d4d2d0d8) back to an external find_package(Eigen3) dependency, like zlib. The
OVERRIDE_FIND_PACKAGE mechanism makes the Visual-Studio-bundled CMake (4.x
"-msvc") intermittently segfault during configure -- ~1/3 of fresh configures,
and 100% with Ceres CUDA enabled. The fault is in CMake's own FetchContent
variable-stack cleanup, reached when Ceres' find_package(Eigen3) resolves the
override through a nested FetchContent_MakeAvailable. Stock Kitware CMake runs
the identical scripts fine, so it is a bug in the VS-bundled cmake binary;
providing Eigen externally avoids that path entirely and is stable with Ceres
CUDA both on and off (verified 0 crashes; full CUDA build + NSIS installer
succeed under the VS-bundled cmake).

Also fix CPACK_NSIS_INSTALLED_ICON_NAME: the backslash value was written
verbatim into CPackConfig.cmake, where "\j" is an invalid escape that cmake 4.x
(CMP0010 strict) rejects when cpack re-parses it, so cpack failed under the
VS-bundled cmake (older/Kitware cpack only warned). Use a forward slash.

- image_analysis: keep Ceres USE_CUDA OFF (Jungfraujoch does not use Ceres' GPU
  solvers) but drop the now-obsolete "required to avoid the Windows cmake crash"
  rationale from the comment.
- docs: Eigen is a required external dependency again (SOFTWARE.md); the Windows
  viewer build provides zlib + Eigen externally (JFJOCH_VIEWER.md).
- THIRD_PARTY_NOTICES: move Eigen from the fetched-and-linked table to the
  external runtime-libraries table.

Docker images still need Eigen re-added (5.0.1, source build) -- to follow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

windows: bundle cuFFT in installer and tag the CUDA/CPU variant ... ec8a45dc09

cuFFT is the only CUDA component linked dynamically (cudart and the
fast-feedback indexer are static), so the prior Windows installer would
fail to launch the GPU path on a host without a CUDA toolkit. Ship
cufft64_*.dll next to the viewer (CUDA 13 keeps it in bin/x64, earlier
toolkits in bin); the DLL is self-contained, so the installed app needs
only an NVIDIA driver.

Tag the variant where it matters and nowhere else: the installer
filename (-cuda<major> / -cpu) and the Add/Remove Programs entry
("Jungfraujoch (CUDA)" / "(CPU)") advertise it, while the install folder
and Start Menu group stay plain "Jungfraujoch" -- the CUDA build is a
strict superset, so the two variants share a location and replace each
other.

Docs: SOFTWARE.md + JFJOCH_VIEWER.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

reader: split raw-image reading into HDF5ImageLocator + HDF5ImageSource ... 23d27f30c4

Decouple the raw-pixel side of JFJochHDF5Reader from the rest as the first
step toward swappable per-dataset metadata snapshots.

- HDF5ImageLocator: single owner of the legacy/VDS/contiguous layout resolution
  plus a persistent open-file cache, replacing the four duplicated resolvers
  (GetImageLocation, ReadSpots, ReadReflections) and their per-call file caches.
  Also hosts the source-mapping logic (former GetHDF5DataSource body).
- HDF5ImageSource: raw-pixel reading (locator + LoadImageDataset); the part whose
  links to files stay fixed while the metadata master may change.
- JFJochHDF5Reader keeps a thin GetImageLocation/GetRawImage/GetHDF5DataSource that
  delegate to image_source_; the six layout members are gone, parsed into a local
  Layout handed to the source at the end of ReadFile. Cache cleared on Close().

Verified: tests/jfjoch_test [HDF5] (79 cases / 1775 assertions), and
jfjoch_process/azint/extract_hkl/scale relink unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

reader: split metadata reading into HDF5MetadataSource + snapshot map ... 9434878c92

Complete the image/metadata decoupling. JFJochHDF5Reader is now a thin JFJochReader
facade composing one shared HDF5ImageSource (raw pixels) with a name->HDF5MetadataSource
map and an active selection ("snapshots").

- HDF5MetadataSource owns one master file: dataset-level parse (Open, ex-ReadFile),
  per-image metadata (FillPerImage, ex-LoadImage_i), ReadSpots/ReadReflections/
  ReadCalibration, and the master-first helpers. It locates per-image metadata either via
  the shared image source (original file: metadata co-located with pixels) or in its own
  master at the global index (integrated _process.h5 snapshot) - so switching snapshots
  never reloads pixels.
- Facade adds RegisterSnapshot/SetActiveSnapshot/SnapshotNames/ActiveSnapshot; ReadFile
  registers the original as "Original". All existing JFJochReader callers are unchanged.
- The global hdf5_mutex is taken once in the facade; the sources assume it is held.

Adds JFJochReader_Snapshots test (original _master.h5 + a reprocessing _process.h5 over
the same images: pixels stay from the original, metadata follows the active snapshot).

Verified: tests/jfjoch_test [HDF5] (80 cases / 1810 assertions), plus jfjoch_process/
azint/extract_hkl/scale and jfjoch_viewer all build against the refactored reader.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

process: extract reprocessing workflow into shared JFJochProcess library ... 4697f10555

The full processing workflow no longer lives in the CLIs. New process/JFJochProcess
encapsulates it for jfjoch_process, jfjoch_azint and (later) the viewer:

- ProcessMode {AzimuthalIntegration, FullAnalysis}; ProcessConfig carries run control
  (range, threads, output prefix, spot finding, rotation + scaling options, reference
  data) while the DiffractionExperiment carries all algorithm settings.
- Run() executes setup -> optional two-pass rotation pre-pass -> parallel per-image loop
  (std::thread) -> optional scaling/merging post-pass -> NXmxIntegrated _process.h5 that
  links back to the original images. ProcessResult returns stats + merge text.
- Cancel() / std::atomic<bool> (receiver style), checked between images; the CLIs install
  a SIGINT handler that calls it (fixes the previous Ctrl+C gap), the viewer will use the
  same hook. JFJochProcessObserver streams progress / per-image results for a live GUI.

jfjoch_process.cpp and jfjoch_azint.cpp are now thin: argument parsing + experiment
configuration, then JFJochProcess::Run + stats printing. Behaviour and usage messages
are unchanged.

Adds JFJochProcessTest (azimuthal integration round-trip, no-output run, pre-cancel) over
a small generated dataset.

Verified: tests/jfjoch_test [HDF5] (83 cases / 1854 assertions); jfjoch_azint and
jfjoch_process run end-to-end on lyso_test (azint 20 images; full analysis recovers the
lysozyme cell at 25% indexing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests: git-LFS large-dataset harness + jfjoch_process include cleanup ... 39599bc090

- .gitattributes tracks tests/data/*.h5 via git-LFS for big reference datasets.
- tests/TestData.h resolves tests/data files and reports absent / unfetched-LFS-pointer
  so tests can SKIP() instead of failing; tests/data/README.md documents fetching + the
  expected lyso_rotation/lyso_serial datasets.
- JFJochProcessLargeTest: a Catch start-up listener that prints dataset availability, plus
  [large] full-analysis runs (rotation indexing + serial) that SKIP when data is absent.
  Verified against the real 1800-image lyso rotation set (100% indexing, cell 78.2/78.2/37.8)
  and skips cleanly without it.
- jfjoch_process.cpp: drop the ~15 now-unused workflow includes left after the JFJochProcess
  extraction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

process/viewer: command-line generator + JFJochProcessController (GUI engine) ... 800ffecd20

Foundation for making processing a first-class GUI activity:

- process/JFJochProcessCommandLine: reconstruct the equivalent jfjoch_process / jfjoch_azint
  command line from a ProcessConfig + DiffractionExperiment + input path, for handing a job
  off to a cluster. Unit-tested (full + azint).
- viewer/JFJochProcessController: runs one JFJochProcess job off the GUI thread (its own private
  reader; HDF5 access is globally serialized so it is safe next to the interactive reader) and
  reports back via queued Qt signals (started/phaseChanged/progress/finished/failed). Cancel()
  forwards to JFJochProcess::Cancel(). Progress is throttled to ~200 updates per run.

The visible processing UI (jobs window + snapshot switcher + converged settings) builds on this.

Verified: tests/jfjoch_test [process]; jfjoch_viewer links and builds against JFJochProcess.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: Processing jobs window - run jfjoch_process in-GUI + result snapshots ... 5bfa70c4a6

Makes the viewer a processing frontend, not just an image viewer:

- JFJochProcessingJobsWindow (menu "Processing"): a table of processing jobs on the open
  dataset. "New job" configures mode (full/azint), image count, threads, output, and (full)
  rotation indexing + scale/merge, using the viewer's current processing settings. A job can
  be Run locally (off the GUI thread via JFJochProcessController, with live status/progress and
  Cancel) or its jfjoch_process command line copied for a cluster.
- A finished local run is registered as a reader metadata snapshot and becomes the active
  dataset (bottom plots + per-image overlays follow it); "View results" / "Show original" switch
  between runs - so several settings attempts on one dataset can be compared.
- JFJochImageReadingWorker gains GetReprocessingInputs() (one locked read of file + experiment +
  mask + spot-finding settings, so jobs share the interactive settings) and
  RegisterProcessingSnapshot/SetActiveSnapshot slots + snapshotsChanged signal over the reader's
  snapshot API. Files only (not live HTTP).

First-cut / open ends: one job at a time; output lands in the working dir (FileWriter rejects
absolute prefixes); Bragg/scaling settings still use defaults until the converged settings
window lands. Viewer builds and runs (offscreen) cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests: add lyso rotation dataset (1800 images) via git-LFS ... ed8fb5d37c

Real reference dataset for the [large] JFJochProcess_LysoRotation test, stored with git-LFS
(tests/data/*.h5). lyso_rotation_master.h5 is the 1800-image lysozyme rotation series
(its data files keep the names the master links to). The test indexes it at 100% with cell
~78.2/78.2/37.8; it SKIPs when the data is not pulled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests: drop the serial [large] test (serial dataset not shipped) ... c916cf8b2d

Only the ~1800-image rotation dataset is kept in LFS; a separate ~5000-image serial set is too
large to ship, and the serial path can be exercised by running the rotation series in serial
mode if needed. Removes JFJochProcess_LysoSerial and the serial entry from the start-up listener
and README. The rotation [large] test now runs against the committed data and passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: converge processing settings into one tabbed window + Bragg/scaling ... 74f36710e4

One "Processing settings" window with tabs: Spot finding & indexing | Azimuthal | Bragg
integration | Scaling, replacing the two separate settings windows. The spot/index and azimuthal
tabs reuse the existing windows' widgets unchanged (their content is lifted into tabs via
takeCentralWidget), so all their logic/signals keep working; Bragg integration and scaling are new
editable panels (previously not adjustable in the GUI).

JFJochImageReadingWorker gains UpdateBraggIntegrationSettings / UpdateScalingSettings; both persist
as worker state and are re-imported into curr_experiment on file load / dataset update (like the
indexing/azint settings), so they apply to interactive analysis (Bragg) and flow into processing
jobs via GetReprocessingInputs (Bragg + scaling). Scaling only affects the job post-pass, so it is
just stored, not reanalyzed.

Verified: jfjoch_viewer builds and runs (offscreen) with the converged window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: Tests needs LFS to get rotation dataset 95223fc281

viewer: indexing algorithm/refinement radio groups, rotation fix, live job plots ... 940b7e34d4

Processing settings:
- Split the spot/index settings widget into two tabs (Spot finding | Indexing)
  via TakeSpotFindingPage()/TakeIndexingPage().
- Indexing tab now exposes the indexing algorithm (Auto/FFBIDX/FFT/FFTW/None) and
  geometry refinement (None/Orientation/Beam center/Pixel refine) as radio groups
  with short explanations.
- Fix: UpdateSpotFindingSettings dropped algorithm + geometry-refinement when
  copying into indexing_settings (the geom checkbox was a no-op); both now flow
  into jobs via curr_experiment.

Processing jobs window:
- Rotation indexing from the job dialog now also sets RotationIndexing(true) on
  the experiment's IndexingSettings; otherwise IndexAndRefine builds no rotation
  indexer and the two-pass pre-pass throws.
- Status cell shows a QProgressBar with "<done> / <expected>".
- Live results: JFJochProcessController::OnImageProcessed accumulates per-image
  results into a JFJochReaderDataset and emits it throttled (~4 Hz) as liveDataset,
  forwarded to the dataset-info plots so they update while a job runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: dock the processing panel next to the plots; HTTP-mode notice ... 9c417d322f

- Convert JFJochProcessingJobsWindow from a standalone helper window into a
  dockable QWidget panel (toolbar + stacked table/message), placed in the bottom
  dock area and split horizontally to sit in the bottom-right corner next to the
  dataset-info plots. Its show/hide toggle moves to the Window menu via the new
  JFJochViewerMenu::AddDockEntry.
- When connected to a live HTTP stream the panel shows "Dataset re-processing is
  currently available only in File mode" and disables the toolbar, driven by the
  worker's httpConnectionChanged signal.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: fix bottom-dock height, window size, and snapshot-switch lag ... 42219d8c5b

Regressions from docking the processing panel + live plots:
- Default window size is now screen-aware (fits a laptop) instead of a fixed
  1200x1200.
- Give the bottom dock area a guaranteed height (resizeDocks vertical) and a
  minimum height on the dataset-info chart, so the plot and its axis labels stay
  visible next to the processing panel.
- Activating a metadata snapshot (Show original / View results) no longer re-runs
  full analysis: it re-reads the image against the snapshot's stored results with
  reanalysis suppressed, instead of re-indexing on every switch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: show effective indexing algorithm; disable GPU options without CUDA ... b735aec1c4

- The Indexing tab now shows what the selected algorithm resolves to on this
  machine/dataset ("Effective on this system: ..."), mirroring
  DiffractionExperiment::GetIndexingAlgorithm() so Auto is no longer ambiguous
  (GPU present? unit cell known?). Cell-known state is forwarded from the loaded
  dataset via JFJochSettingsWindow::datasetLoaded.
- FFBIDX and FFT (GPU) radio options are disabled on builds without CUDA, where
  only the FFTW CPU indexer exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: overlay one plot line per processing run (with editable names) ... aceff23ce2

The dataset-info plot now overlays every run as a separate named line instead of
replacing the plot when a snapshot is activated:

- Reader gains AllSnapshotDatasets() (every snapshot's dataset, Original first).
- The worker owns the run collection: a stable snapshot id plus an editable
  display label (RunData), emitted as runsChanged(runs, active_id) on file open,
  snapshot register, activate and rename. RenameRun(id, label) updates the legend
  label without touching the reader key (decoupled id vs label).
- The chart view draws the active run as the primary series (markers, hover,
  binning, axes) and the other runs as overlay lines sharing its axes, with a
  legend shown when overlaying (appendSeries factored out for both).
- The dataset-info widget holds the run list + the live run, extracts the selected
  metric from each (ExtractMetric), and unions the available metrics across runs.
- The live run is its own "Live" overlay (lightweight per-tick refresh, no combo
  rebuild); it is cleared on finish so the persisted snapshot takes over.
- The processing jobs table gets a "Started" column and an editable Name column;
  editing a name renames that run's legend label.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: remove/clear runs, stable per-run colours, processing rate + ETA ... 2af822436e

- "Remove result" toolbar action drops a reprocessing run: reader RemoveSnapshot
  (Original is protected; if the removed run was active the view falls back to
  Original), worker RemoveRun, and the table row is removed.
- Opening a new file clears the jobs table (worker fileOpened -> clearJobs) to
  match the reader, which already resets snapshots on ReadFile.
- Plot fixes: each run keeps a stable colour by its position (Original = blue),
  so colours no longer swap when the active run changes; the current-image marker
  is hidden from the legend (was the stray "[Image #N]" entry).
- The status bar shows processing rate (Hz) and estimated remaining time while a
  job runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: give the current-image plot marker a fixed colour ... a412f96f3a

The current-image marker (a scatter series) was picking an arbitrary theme colour
(green), which read as a stray series. Pin it to black so it's an unambiguous
"current image" dot, distinct from the run line colours. Its legend entry is
already hidden.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: looser minimum heights so the bottom docks resize freely ... 7ba097df1c

Lower the dataset-info chart and jobs-table minimum heights (200 -> 80, table 60)
so the bottom dock area can be dragged smaller; the comfortable default size is
still set via resizeDocks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

reader/viewer: snapshot image->original map; subset runs register and overlay ... b9f8c2b675

Foundation for a dataset (snapshot or, later, the main file) being a subset of the
truly collected images:
- JFJochReaderDataset gains source_image_number (image index -> original image
  number; empty = identity).
- HDF5MetadataSource reads /entry/detector/number into that map and an inverse
  (original -> local) map; FillPerImage / ReadSpots translate the requested global
  image to this source's local index via ToLocalIndex (return nothing if the image
  is not covered), so partial snapshots are correct and never read out of bounds.
- RegisterSnapshot now accepts a snapshot with fewer images than the dataset
  (only rejects more), so sub-range / strided reprocessing runs register.
- The dataset-info plot draws each run at its original image numbers (x map from
  source_image_number), so a subset run lands at the right place on the shared
  axis. The live run's map is filled from msg.original_number.

This makes the foundation ready for strided / filtered selections (e.g. reprocess
only images with >N spots) without restricting to a min-max sub-range.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: processing dialog takes start + end image, not a count ... 0f3009069f

Replace the ambiguous "Images" count with explicit "Start image" and "End image"
spinboxes (end 0 = to the last image). buildConfig sets config.start_image /
end_image; the table shows the range ("start-end" / "all") and the progress bar
expects end-start images.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: white-when-active inputs, teal accent, Status column visible ... 03a09c43bc

Styling:
- Entry widgets (line edits, spin boxes, combos, item views) are salmon (the GUI
  default) when inactive and turn white while focused/active.
- A teal accent (#2a9d8f) for selections and the QProgressBar chunk, against the
  salmon background.
- Move the jobs-table Status column (the progress bar) to column 2 so it stays
  visible in the narrow dock instead of scrolling off the right edge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: Sign RPMs on upload 95fe2ce781

viewer: palette theming, Original as a run row, stable plot colours ... 67ad43a2b2

Styling (now via the application palette in main(), not a stylesheet, so it
applies to every widget incl. dialogs):
- Entry fields and item views (line edits, spin boxes, combos, tables) are white
  when enabled and salmon (the GUI default) when disabled - so it's clear where
  you can type. Panels stay salmon (Window role).
- Teal (#2a9d8f) accent via the Highlight role: selections and progress-bar chunks.

Runs list / plots:
- The original file is listed as a first "Original" run (Mode "HDF5"); double-click
  any run row to show it (replaces the "Show original" button). The currently shown
  run is bold (driven by snapshotsChanged).
- Fix the colour "swap" when switching runs: the primary line is matched by the
  active-dataset pointer (not active_id_), so a run keeps its colour even while a
  datasetLoaded/runsChanged pair is mid-flight.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rotation indexer: fix sub-range crash, explicit angle, settings + guards ... c86beeb393

Running rotation indexing on a sub-range (e.g. images 60-120) segfaulted: the
first pass passed the *global* image number to RotationIndexer::ProcessImage,
which indexes v_ (sized to the run's image count) -> out-of-bounds write.

- ProcessImage now takes an explicit mid-exposure angle (optional; falls back to
  the goniometer at the image index), so the indexer no longer assumes its slot
  index equals the goniometer image index. IndexAndRefine supplies it via
  RotationAngle(), matching the angle used for prediction. Added a bounds guard in
  ProcessImage so a bad index can never corrupt memory.
- JFJochProcess feeds the rotation indexer the local ordinal (not the global
  index), and shifts the goniometer (start += start_image*incr, incr *= stride,
  per-image wedge preserved) so local index i maps to the angle of original image
  start+i*stride - fixing rotation angles for the whole sub-range pipeline
  (prediction, refinement, output), not just the indexer.
- Expose "Rotation images" (number used for the first pass) in the job dialog,
  enabled when rotation indexing is on. (Count > available is already clamped by
  select_equally_spaced_image_ordinals.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: reuse jfjoch_hdf5_test via artifact in dials-test ... 94125bd6cb

Build jfjoch_hdf5_test once in a new build-hdf5-test job and hand the binary
to dials-test as an artifact, instead of recompiling it from scratch in every
rocky9 processing job. Uses the christopherhx gitea-{upload,download}-artifact
fork (mirrored under gitea.psi.ch/actions) to avoid the stock v4 action's
GHESNotSupportedError. Probe scope: dials-test only; xds-* still self-build
until the round-trip is confirmed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: salmon group boxes; dataset-info x-axis spans the whole dataset ... 961836837f

- Fusion fills QGroupBox interiors with a flat light colour, so the settings window
  lost its salmon look. A tiny app stylesheet (QGroupBox { background: transparent })
  makes them show the salmon window background again; entry widgets stay white via
  the palette.
- The dataset-info chart now fixes the x-axis to the whole dataset [0, n) (the
  largest run = the original file), so a subset run is drawn at its real image
  positions instead of being stretched to fill the plot. Subsets with binning bin
  by ordinal and place each point at its mapped image number (correct for the
  common contiguous sub-range).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Revert "CI: reuse jfjoch_hdf5_test via artifact in dials-test" ... 9defbffa67

This reverts commit 94125bd6cb.

CMake: Update FFBIDX to support MSVC 5f3126e19e

windows: auto-detect nvcc and drop manual CUDA/zlib flags ... ad5662732d

Locate nvcc via find_program (HINTS CUDA_PATH / /usr/local/cuda) before
CHECK_LANGUAGE(CUDA), so CMAKE_CUDA_COMPILER no longer has to be passed by
hand. CHECK_LANGUAGE only searches PATH, which is missed routinely on Windows
and intermittently on Linux; the CUDA installer always sets CUDA_PATH. An
explicit -DCMAKE_CUDA_COMPILER / $CUDACXX still wins.

Docs: zlib and Eigen are external find_package deps (not auto-fetched);
simplify the Windows configure to just CMAKE_PREFIX_PATH (no CMAKE_CUDA_COMPILER,
no ZLIB_ROOT); fix the Eigen 5.0.1 install recipe to disable BLAS/LAPACK
(they fail MSVC with C1128 /bigobj) and use cmake --install.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

deps: pin external Eigen to 3.4.x across docs and Docker images ... 68f3394f48

Eigen is an external find_package(Eigen3 3.4) dependency. Eigen's
same-major-version rule means a bare 3.4 request only accepts 3.x, so 5.x
cannot be used without changing every requester (jfjoch, Ceres, and the
upstream ffbidx). Standardise on Eigen 3.4.x:

- docs: correct the Windows Eigen install recipe to 3.4.0 and note the
  same-major constraint; SOFTWARE.md now says 3.4.x (not "3.4 or newer").
- docker/{rocky8,rocky9,ubuntu2204,ubuntu2404}: actually install Eigen 3.4.0
  from source to /opt/eigen-3.4 (header-only) and add it to CMAKE_PREFIX_PATH.
  The images previously installed no Eigen at all, relying on the obsolete
  "CMake fetches it" assumption; a rebuild would have failed at configure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

viewer: Phase 0 redesign — slim headers, navy accent, icon nav ... ecc06d9abd

First pass of the viewer redesign (see docs/review/VIEWER_REDESIGN.md), no
structural change:

- TitleLabel: replace the 50px solid #FA7268 section bars with a slim 26px
  navy-bold header + coral accent rule, removing the "venetian blind" stack
  while keeping the salmon identity.
- Palette: switch the accent (QPalette::Highlight) from teal to navy #1F3A5F.
- Image toolbar: replace the unicode arrow glyphs (|<= <= => =>|) with flat
  media icons (first/prev/next/last) + tooltips, and a play icon on Movie.
- Bottom dock: give the per-dataset plot more default height (340 -> 420).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: Phase 1a — dockable shell with saved perspectives ... f5a146d212

Make the layout reconfigurable, the foundation for the redesign:

- The diffraction image becomes the central widget; the right-hand side panel
  is now a dockable "Inspector" (QDockWidget, right area).
- Every dock and toolbar gets a stable objectName so QMainWindow saveState /
  restoreState round-trips. Dataset-info docks are numbered uniquely.
- Persist geometry + dock state to QSettings on close and restore on launch,
  so the user's arrangement resumes.
- New "View" menu with Image / Processing perspectives (show/hide the bottom
  plots + jobs panel) and "Reset layout" (back to the as-built arrangement,
  captured at startup).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: Phase 1b — inline MX/AzInt settings dock ... 58c60e03fe

Surface a surgical subset of processing settings in an always-visible dock
instead of hidden windows:

- New JFJochViewerSettingsDock with an MX / AzInt segmented toggle.
  MX page: geometry (energy, distance, beam X/Y), a new unit-cell +
  space-group editor (no such input existed before; enables known-cell
  ffbidx), spot finding (S/N, photon count, min pixels/spot), indexing
  algorithm and geometry refinement. AzInt page: q range / spacing /
  azimuthal bins plus the existing powder-calibration widget.
- Edits feed straight into the worker (UpdateSpotFindingSettings,
  UpdateAzintSettings, UpdateDataset, FindCenter); fields populate from the
  loaded dataset.
- Add CeO2 and Silicon calibrant presets.
- Dock it left, objectName "settingsDock", show it in the Processing
  perspective (hidden in Image).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: PSI logo + tidy beam-center row ... 9f9e13a4e4

- Show the PSI logo in the menu-bar corner, picking one of four interchangeable
  dot designs at random each launch. Embedded as PNGs (rasterised from the SVG
  sources, kept alongside) so no Qt SVG module dependency is needed.
- Settings dock: put beam center X and Y on a single "Beam center" row to save
  vertical space on laptop screens.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: promote Reanalyze image / Reanalyze dataset hero buttons ... 54526300a9

The two capabilities that set Jungfraujoch apart get prominent, paired navy
buttons at the right of the display toolbar, with purpose-drawn icons (a single
diffraction frame vs a stack of frames):

- "Reanalyze image" re-runs the analysis pipeline on the current image
  (worker Analyze).
- "Reanalyze dataset" opens a new whole-dataset processing job
  (JFJochProcessingJobsWindow::newJob, now public).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: Phase 1c — fold per-image plot into the dataset-info dock ... e7edee6b2e

Remove the duplicated per-image plot from the side panel and host it in the
dataset-info dock instead:

- JFJochViewerDatasetInfo gains a "Per-image" toggle (next to Grid) that swaps
  the stacked view to the existing JFJochViewerSidePanelChart (azimuthal 1D,
  Wilson, I/sigma, spot and fluorescence profiles of the current image). The
  per-dataset metric combo is disabled while in per-image mode.
- The per-image profile follows the displayed image (imageLoaded forwarded).
- Drop the "Image statistics plot" section from JFJochViewerSidePanel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: panel polish — accordions, shared geometry, detector tilt, SG name ... 03ac43da69

Declutter the side and settings panels (review items a–f):

- New CollapsibleSection widget (slim navy header + coral rule + chevron).
- Inspector: Image features / Resolution rings / ROI are now collapsible and
  start folded (ROI auto-expands when ROIs change); drop the Data-analysis and
  Powder-calibration sections (they live in the hero buttons and settings dock).
- Settings dock: move Geometry to a shared section above the MX/AzInt toggle
  (both communities need it); add Detector tilt (PONI rot1/rot2, deg) and
  rename "Beam center" -> "Beam origin" with PONI/XDS tooltips; show the
  space-group number and resolved Hermann–Mauguin symbol on one line; wrap
  sections in accordions, anchored top with a bottom stretch so expanding one
  does not shift the others.
- Toolbar: drop the redundant "Image number" label.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: toolbar rework — icons, prominent scrubber, HTTP states (g–k) ... a7c1560dcb

Complete pass over both toolbars (review items g–k), closing the toolbar topic.

New ToolbarIcons: crisp QPainter vector glyphs (Qt SVG is not a build dep), with
white "on" variants for toggles and a shared flat button style (coral hover,
navy checked).

Navigation toolbar, laid out source-first:
- Open (3.5" diskette) · HTTP sync · Movie (reel-on-top camera, distinct from
  the next/play triangle) ║ the scrub slider, which expands to fill the bar
  (coral track, navy handle) so it is the obvious way to move across the data ║
  precise navigation: first/prev/[number]/next/last, Jump, Sum.
- Open and HTTP-sync reach the file / connect dialogs (JFJochViewerMenu
  openSelected and openHttpSelected, now public).
- HTTP-sync has three states via a status dot (grey disconnected / green live /
  amber frozen); clicking with no live source attached opens the connect dialog.
  Removed the separate "Reanalyze" toggle.

Display toolbar:
- Styled foreground slider (matching), Auto/HDR as styled text toggles.

Hero buttons:
- "Reanalyze image" is now a toggle (worker ReanalyzeImages: run now + keep
  re-analysing on image/settings/processing changes; coral = active).

Processing dock:
- Drop the "New job…" toolbar action (the hero button drives it) and parent the
  job dialog to the main window instead of the dock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: thumbnail hit feed (image strip) ... c62cd99566

A dockable strip of thumbnails for a handful of representative images, with
spots overlaid so a real pattern reads like a constellation; click opens that
image.

- JFJochViewerImageStrip: mode selector (evenly spaced / most spots / indexed,
  computed from the dataset's per-image metrics) + a Spots toggle; a scrollable
  row of thumbnail buttons; click emits imageSelected.
- The reading worker renders thumbnails off-thread and non-disruptively: load
  each image via its reader, downsample by block-maximum (keeps Bragg spots),
  map through the colour-scale LUT, optionally paint spots (coral indexed /
  teal not), and emit thumbnailReady(image_number, QImage). File mode only.
- Docked in the bottom area, shown in the Processing perspective; the colour map
  follows the display toolbar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: hit-feed polish — stacked docks, hidden processing, resolution mode ... 7d24cdba4f

Follow-up to the image strip, addressing the review:

- Composition: stack the plots and the thumbnail strip vertically (plots on top
  with more height, strip below) instead of sharing horizontal space — both
  benefit from width, and the strip needs less height.
- Processing dock is hidden by default and narrower; it reveals itself only when
  a reprocessing job starts (new jobStarted signal), and the Processing
  perspective no longer force-shows it.
- Thumbnail spot overlays now use the same feature (indexed) / spot colours as
  the main viewer, and follow the side-panel colour pickers.
- New strip selection modes "Resolution" and "Background": pick images that span
  that metric's distribution (a quick histogram-representative selection).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: soften heights, stochastic strip selection, stable on live sync ... 164c03f1ce

Address the review on the image strip / hit feed:

- Height constraints: JFJochSimpleChartView used a hard setFixedHeight(300) and,
  being a page in the dataset-info stack, forced the whole plot dock (and thus
  the window) taller than the screen once the strip was stacked below. Make it a
  soft minimum (120). Wrap the settings dock in a QScrollArea so its content can
  scroll instead of forcing window height. Smaller strip thumbnails (96) and
  lower default bottom-dock heights. The window no longer grows past its
  requested size.
- Stochastic selection: representatives are now picked at random within N equal
  bins (over image index, or a metric's sorted order), and a Refresh button
  re-rolls a fresh set — avoiding deterministic-spacing artefacts. "Most spots"
  stays deterministic; "Indexed" becomes spaced-random.
- Live stability: the strip stores dataset updates but only rebuilds on file
  open (worker fileOpened) / mode / Spots / Refresh, so HTTP sync image updates
  no longer trigger constant thumbnail refetching.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: fix thumbnail render crash (worker-thread exception / LUT UB) ... 94b66348c5

The image-strip thumbnail renderer could take the whole viewer down:

- Only the image load was wrapped in try/catch; the rest of RenderThumbnail_i
  (pixel access, LUT mapping, QPainter) ran unguarded on the worker thread, so
  any exception there called std::terminate. Wrap the entire function.
- std::clamp(idx, 0, lutSize - 1) is undefined behaviour when the colour LUT is
  empty (lo > hi). Bail out early if the LUT has no entries.

A click landing while a later thumbnail is still rendering surfaced this.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: keep plot axis labels readable when compressed ... c5457e2e27

When the plot dock was short, Qt Charts dropped the axis labels first, making
the plot hard to read.

- Reclaim Qt Charts' outer graphics-layout padding and trim the inner margins
  (both chart views) so the plot and its labels get the available space.
- Raise the chart minimum height (dataset-info 80 -> 140, per-image 120 -> 140)
  so the dock can't be squeezed below the point where the axis labels fit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: "+ Plot" button + strip fits height with in-bitmap numbers ... 83d006e79b

Plots:
- A "+ Plot" button on the dataset-info panel spawns another plot dock, placed
  beside the previous one (horizontal split) so several metrics can be watched
  side by side. Same path as the Charts menu, now one click away.

Image strip:
- The image number is painted into the thumbnail bitmap (coral badge, top-left)
  instead of a text label under the icon, so no height is lost to it.
- Thumbnails are icon-only and scale to the available dock height (resizeEvent),
  so the strip never needs more room than it has; lowered its minimum.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: per-image plot reuses the dataset-info combo (no second combo) ... 6ead87fcce

In per-image mode the embedded chart brought its own combo box, so the plot
panel showed two. Drop the per-image chart's combo and drive it from the one
dataset-info combo:

- JFJochViewerSidePanelChart loses its combo; it exposes PlotTypes() (the
  available profiles) and a setPlotType(code) slot, driven externally.
- JFJochViewerDatasetInfo fills its single combo with the per-image plot types
  while in per-image mode (per-dataset metrics otherwise) and routes the
  selection to the chart; dataset/runs/live updates leave the per-image combo
  and plot untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: chart top margin + settings dock width bounds ... 9ba8096c0b

- The chart top margin was too tight (2 px), so the highest Y-axis label
  overlapped the top edge; give it 10 px of room (both chart views).
- The settings dock had no width constraints (unlike the inspector), so it
  stretched ugly when widened and would not fold back narrowly. Cap it to
  330–480 px like the inspector.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: settings fields fill the panel width (form growth policy) ... e3bd1a3529

The settings content stopped resizing past ~360 px, leaving dead space when the
dock was widened. Cause: QFormLayout's default field growth policy keeps fields
at their size hint. Set AllNonFixedFieldsGrow on all five forms (geometry, unit
cell, spot finding, indexing, azint) so the fields grow to fill the dock width.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Support providing trimming energies 8eee3dc785

OpenAPI: update 2df1035518

jfjoch_broker: Fix bcf01c28f4

OpenAPI: fix missing trim_energie_eV 066c0dfa90

TCP stream: tolerate writer backpressure via BUSY heartbeats ... 97c3008c39

A slow filesystem could stall the writer's consume pipeline, propagating
TCP backpressure to the pusher. The pusher then treated that backpressure
as a dead peer and force-closed the connection mid-run; the writer
reconnected as a brand-new connection outside the active session, so the
rest of the run was silently dropped and the half-written HDF5 file was
later finalized with holes.

Replace the throughput/progress-based send timeout with a peer-liveness
timeout:

- Add TCPFrameType::BUSY (wire version 2 -> 3).
- TCPImagePuller runs a heartbeat thread that sends BUSY every 250ms on a
  thread independent of the (possibly stalled) write path, so liveness
  keeps flowing during deep stalls. A send_mutex serializes
  ACK/pong/heartbeat writes.
- TCPStreamPusher refreshes last_peer_activity_ns on every inbound frame
  and only declares a connection dead after peer_liveness_timeout (15s) of
  complete silence, tolerating arbitrarily long backpressure while still
  catching a genuinely dead peer (and immediate EPIPE/ECONNRESET).
- Re-key both backpressure waits (the SendAll data path and the post-END
  WaitForEndAck) onto this liveness signal instead of byte-progress /
  DATA-ACK-progress, so a slow final flush at END is tolerated too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

TCP stream: gate SendImage enqueue on writer liveness too ... d156dfa2e8

The broker logs for the dropped runs show the connection torn down ~2s into
a collection (not 3s), via "TCP send failed -> Removed dead connection ->
Accepted (new socket)". That is too early for the SendAll send deadline:
the real gate was the fixed 2-second enqueue deadline in the zerocopy
SendImage path. At the start of a large dataset the writer briefly stalls
draining the socket while it creates the master file and writes the large
START metadata + calibration frames to GPFS; the per-connection queue fills,
and after 2s SendImage marked the connection broken. The writer then
reconnected outside the active session, so the rest of the run was dropped
and the half-written file was finalized at the next START.

Replace the fixed 2s enqueue deadline with the same peer-liveness condition
used on the send path: keep applying backpressure while the writer proves it
is alive (BUSY heartbeats / ACKs refresh last_peer_activity_ns from a thread
independent of the stalled write path), and only declare it dead after the
liveness window of complete silence. A transient startup stall is now ridden
out instead of dropping the run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

broker: fail the run loudly when the writer breaks mid-acquisition ... 145b300fa2

A writer connection that dropped mid-run left a truncated file, yet the
acquisition was still logged as "finished with success": the writer error
returned by ImagePusher::Finalize() (set via transmission_error on a broken
session connection) was captured into receiver_output.writer_err but never
acted upon.

Treat a non-empty writer_err as a failed acquisition: throw so it is reported
as "finished with error" and surfaced to the caller, instead of silently
succeeding with incomplete data. Applies to both the Lite and FPGA workflows,
which share this stop/finalize path. Combined with the liveness changes (which
no longer tear a connection down for a transient stall), this fires only on a
genuine writer break.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/CHANGELOG.md: round out the rc.153 entry ... e121918f55

Add the biggest user-facing changes the entry was missing: the trimming-energies
config change for PSI EIGER (now compulsory), ROI definitions/bitmap in the master
file with azimuthal phi sectors, and the viewer redesign + in-GUI jfjoch_process jobs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-06-23 20:24:43 +02:00

viewer: bound live-monitor memory with frame-in-flight backpressure ... 50b08388d1

The image-reading worker (on its own thread) posted imageLoaded — a full
~40 MB JFJochReaderImage on a 9 Mpix detector — to ~13 GUI consumers over
queued connections, paced only by its own load+analysis time. When the GUI
render (image + azimuthal integration + plots) was the bottleneck, those
queued events piled up without bound and memory blew up.

Funnel every consumer through one window signal: the worker's imageLoaded
crosses the thread boundary exactly once, into JFJochViewerWindow::
OnImageReady, which re-emits imageReady to all GUI consumers synchronously
(same thread) and then acks the worker (imageConsumed). The worker tracks
images_in_flight and AutoLoadTimerExpired stops producing once the cap
(max_images_in_flight = 4) is reached — applied to every auto mode.

Live follow drops intermediate frames (next fetch grabs the newest image);
movie playback paces down to GUI speed without skipping. The existing
adaptive autoload interval is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f merged commit 75e401f0e5 into main 2026-06-23 20:29:50 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#63