Commit Graph
177 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 4.8 ddddfb6ffc bragg_integration: GPU box + profile-fit integrator (standalone engine)
Build Packages / build:windows:nocuda (pull_request) Successful in 14m41s
Build Packages / build:windows:cuda (pull_request) Successful in 16m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (pull_request) Successful in 11m15s
Build Packages / build:rpm (rocky8_nocuda) (pull_request) Successful in 12m46s
Build Packages / build:rpm (ubuntu2204_nocuda) (pull_request) Successful in 12m38s
Build Packages / build:rpm (rocky9_nocuda) (pull_request) Successful in 13m11s
Build Packages / build:rpm (rocky8_sls9) (pull_request) Successful in 12m20s
Build Packages / build:rpm (rocky9_sls9) (pull_request) Successful in 12m22s
Build Packages / build:rpm (ubuntu2404) (pull_request) Successful in 11m7s
Build Packages / build:rpm (ubuntu2204) (pull_request) Successful in 11m55s
Build Packages / build:rpm (rocky8) (pull_request) Successful in 12m56s
Build Packages / Generate python client (pull_request) Successful in 14s
Build Packages / build:rpm (rocky9) (pull_request) Successful in 13m15s
Build Packages / Create release (pull_request) Skipped
Build Packages / Build documentation (pull_request) Successful in 41s
Build Packages / XDS test (durin plugin) (pull_request) Successful in 10m3s
Build Packages / DIALS test (pull_request) Successful in 13m6s
Build Packages / XDS test (neggia plugin) (pull_request) Successful in 6m58s
Build Packages / XDS test (JFJoch plugin) (pull_request) Successful in 7m30s
Build Packages / Unit tests (pull_request) Successful in 58m5s
Build Packages / Unit tests (push) Successful in 1h12m36s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m52s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m35s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 15m29s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m35s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m25s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 16m5s
Build Packages / build:rpm (rocky8) (push) Successful in 15m11s
Build Packages / build:rpm (rocky9) (push) Successful in 13m35s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m59s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m14s
Build Packages / DIALS test (push) Successful in 14m29s
Build Packages / XDS test (durin plugin) (push) Successful in 9m56s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m23s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m3s
Build Packages / Generate python client (push) Successful in 20s
Build Packages / Build documentation (push) Successful in 1m10s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m39s
Build Packages / build:windows:cuda (push) Successful in 18m40s
Reimplement BraggIntegrate2D (box sum) and ProfileIntegrate2D (Kabsch
profile fit) under one roof as a base + CPU + GPU engine, mirroring the
AzIntEngine / ROIIntegration pattern. Reads the preprocessed int32
ImagePreprocessorBuffer (masked=INT32_MIN, saturated=INT32_MAX), the same
buffer AzIntEngineGPU/ROIIntegrationGPU consume.

The CUDA engine runs one block per reflection with shared-memory
reductions across six kernels (reset, mask, box-sum, profile learning,
profile build, Kabsch fit); the resolution shell is computed inline. The
learning/fit hot path is single precision (FP64 is throttled on consumer
GPUs; reproduces the double CPU path to ~1e-4). Collapsing the per-frame
CUDA API calls into one reset kernel keeps launch-latency overhead low.

Standalone for now: NOT wired into IndexAndRefine. See
BRAGG_INTEGRATION_ENGINE.md for the design and the binding steps.
BraggIntegrationEngineGPUTest checks GPU == CPU across all three modes
(box/gaussian/empirical) within numeric tolerance, plus a [bragg_bench]
perf sweep.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 20:59:45 +02:00
leonarski_fandClaude Fable 5 f422988cd2 ice score: per-image ice-ring indicator plumbed through all layers
A single per-image ice_ring_score - the strongest hexagonal-ice ring band/shoulder
intensity ratio from the azimuthal profile (1 = no ice) - computed in the CPU and
FPGA analysis paths and the offline azint worker, then plumbed through every layer:
DataMessage/EndMessage, CBOR (frame_serialize), HDF5 (/entry/MX/iceRingScore),
ScanResult, receiver plots (PlotType::IceRingScore), the OpenAPI spec (plot_type +
scan_result schema, with regenerated broker/gen and frontend client) and
OpenAPIConvert, the reader + Qt viewer, and the React frontend plot. Documented in
docs/CBOR.md, docs/HDF5.md and docs/CPU_DATA_ANALYSIS.md, with the general
"add a per-image quantity" recipe added to CLAUDE.md.

Verified in HDF5: lysoC (weak ice) mean 1.23, EP_cs_01-17 (heavy ice) mean 1.67 /
max 2.23. This is a monitoring quantity - it does not gate scaling (which already
excludes all ice rings) or merging (handled by the CC1/2 ring mask).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 16:37:12 +02:00
leonarski_fandClaude Fable 5 6ffb16f142 merge: mask ice rings whose CC1/2 collapses below their resolution neighbours
An ice-contaminated ring decorrelates its reflections, so its merged half-set
CC1/2 falls well below the Bragg trend of its resolution shoulders (e.g.
EP_cs_01-17 2.25A ring CC1/2 0.12 vs shoulders 0.89). After the final merge,
compute per-hexagonal-ice-ring CC1/2 in the ring band (+/- ice width in q=2pi/d)
against the shoulders either side, mask any ring more than 0.05 below its
shoulders (reliable shoulder CC1/2 > 0.5, >=20 reflections each) and re-merge
without them. Weak/absent rings track their neighbours and stay, so clean
crystals are untouched.

Data-driven and robust on /data/rotation_test: 16/18 crystals mask nothing and
are unchanged (every clean crystal, plus icy-but-fine EP_cs_02-10); EP_cs_01-17
masks 6 rings, CC1/2 6.9% -> 28%; CQ066/pding4_003 mask one marginal ring each.
A well-scaled CC1/2 test, replacing the fragile per-image azimuthal cutoffs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 16:06:39 +02:00
leonarski_fandClaude Fable 5 cdbd0bed5d partiality: fix the rotation-prediction wedge, use the stored value in scaling
BraggPredictionRot halves settings.wedge_deg to the +/- half-wedge of the
partiality erf pair, but IndexAndRefine already passed GetWedge_deg()/2, so the
two /2 compounded to a half-wedge of increment/4 - half the correct Kabsch value
(increment/2, which ScaleOnTheFly's RotationPartiality already used). Pass the
full increment so prediction's partiality matches scaling.

With prediction correct, ScaleOnTheFly now uses the stored r.partiality directly
(the value the reflection was integrated with) rather than recomputing the erf
pair per reflection - recomputing only when scaling overrides the geometry
(-w wedge refinement, --mosaicity, or a scaling wedge override). Output-neutral
on the /data/rotation_test battery (SG/cell/completeness identical, ISa/CC1/2
within run noise on the stable crystals).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 14:57:16 +02:00
leonarski_fandClaude Fable 5 8c084021e6 ice rings: exclude from scaling, keep in the final merge (recover completeness)
--detect-ice-rings previously dropped ice-ring reflections outright before
scaling, which also removed them from the merge and gutted low/mid-resolution
completeness on crystals that merge them fine (e.g. lysoC 80% -> the ice bands
show as jagged shell completeness). Instead flag them (new Reflection.on_ice_ring,
set in JFJochProcess, carried through the 3D combine) and exclude the flag only
where a model is fit: the per-image scale (ScaleOnTheFly::Accept + the per-image
CC), and - for the de-novo P1 pass - the space-group search and error model
(MergeOnTheFly::ExcludeIceRings, on for for_search). The final in-symmetry merge
and its statistics keep them, so completeness is preserved.

/data/rotation_test battery vs the previous drop-from-merge behaviour: space
group correct on all 18; completeness recovered broadly with CC1/2 and ISa held
(cytC_2 82->99.7%, cytC_3 73->99.7%, InsI3 76->99.5%, lysoC 80->99.7%, MyoB
80->99.7%, InsH3 78->99%). Excluding ice from the P1 search merge is what keeps
the space group correct: without it InsI3 flipped I23->P1 and EP_cs_01-17 P2->P1.

Known limitation: on heavy-ice crystals (EP_cs_01-17) the strong ice is garbage
and keeping it in the final merge collapses CC1/2 in the ice shells (91.7->6.9%).
Distinguishing strong vs weak/absent rings per crystal needs data-driven,
per-ring ice detection (azimuthal radial profile) - the planned next step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 14:29:37 +02:00
leonarski_fandClaude Fable 5 1ca5a69054 ScaleOnTheFly: solve the per-image scale by IRLS instead of Ceres
In the default rotation (rot3d) path only G is refined - B is fixed, mosaicity
is pinned and the wedge is not refined - so the predicted intensity G*coeff is
linear in G and the robust (Cauchy) per-image scale is a 1-D M-estimate. Solve
it directly by iteratively reweighted least squares (a few closed-form weighted
ratios) instead of building a Ceres problem per image. Ceres is kept for the
cases that are genuinely nonlinear: refining the B-factor or the rotation wedge.

Same Cauchy objective as the Ceres path, but ~4x faster at scaling and ~30%
faster overall on the /data/rotation_test battery, with space group, cell, ISa,
completeness and CC1/2 matching across all 18 crystals (the two that look
different, EP_cs_01-17 and EcwtAL500, are run-to-run unstable for both solvers).
lyso_ref scaling 25.2->4.3s, cytC_2 15.2->2.6s, battery total 468->316s.

Also drop the per-image G/B regularizers (gated by GetScalingRegularize, which
nothing enables) and the now-unused RegularizationResidual.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 13:55:04 +02:00
leonarski_fandClaude Opus 4.8 ed9f6ac9eb integration: remove --reciprocal-profile (proven per-frame dead-end)
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m21s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 15m0s
Build Packages / build:rpm (rocky8) (push) Successful in 14m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m9s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 15m19s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m28s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m58s
Build Packages / build:windows:nocuda (push) Successful in 16m12s
Build Packages / build:windows:cuda (push) Successful in 18m22s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m52s
Build Packages / XDS test (durin plugin) (push) Successful in 8m8s
Build Packages / Generate python client (push) Successful in 29s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (neggia plugin) (push) Successful in 8m10s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m42s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m25s
Build Packages / build:rpm (rocky9) (push) Successful in 13m12s
Build Packages / DIALS test (push) Successful in 13m48s
Build Packages / Unit tests (push) Successful in 1h16m3s
Build Packages / Unit tests (pull_request) Successful in 56m24s
Build Packages / build:windows:cuda (pull_request) Successful in 11m16s
Build Packages / build:rpm (rocky8_nocuda) (pull_request) Successful in 9m55s
Build Packages / build:rpm (rocky9_nocuda) (pull_request) Successful in 10m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (pull_request) Successful in 11m17s
Build Packages / build:rpm (ubuntu2404_nocuda) (pull_request) Successful in 10m15s
Build Packages / build:rpm (rocky8_sls9) (pull_request) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (pull_request) Successful in 11m57s
Build Packages / build:rpm (rocky8) (pull_request) Successful in 10m48s
Build Packages / build:rpm (rocky9) (pull_request) Successful in 12m13s
Build Packages / build:rpm (ubuntu2204) (pull_request) Successful in 11m22s
Build Packages / build:rpm (ubuntu2404) (pull_request) Successful in 10m34s
Build Packages / DIALS test (pull_request) Successful in 12m24s
Build Packages / XDS test (durin plugin) (pull_request) Successful in 8m45s
Build Packages / XDS test (JFJoch plugin) (pull_request) Successful in 7m23s
Build Packages / XDS test (neggia plugin) (pull_request) Successful in 6m1s
Build Packages / Generate python client (pull_request) Successful in 13s
Build Packages / Build documentation (pull_request) Successful in 47s
Build Packages / Create release (pull_request) Skipped
Build Packages / build:windows:nocuda (pull_request) Successful in 10m4s
An 11-crystal mosaicity-stratified re-test (/data/rotation_test, off vs on
vs a de-contaminated variant, plus a per-frame dump of the fitted widths)
showed the dial is net-negative and cannot work in the per-frame paradigm:

- The C|q|^2 mosaicity term - the whole point - is unfittable per-frame: the
  fitted curvature a2 comes out ~0 (often negative) on every crystal, with zero
  correlation to the XDS mosaicity (0.09..0.42 deg). Strong spots sit at low q
  where eta^2 q^2 is invisible; the curvature only appears at high q where there
  are ~0 strong spots. The law degenerates to a straight line.
- With a2~0 the high-res width becomes a blind 1/cos^2(2theta) extrapolation,
  2-4x wider than per-shell. The per-shell path's high-res "starvation" (flat
  narrow fallback) is accidentally correct: weak, crowded high-res spots want a
  narrow aperture, not the true wide spot shape.
- The over-wide profile pulls background into weak spots -> R-meas rises, CC1/2
  drops in reliable high-multiplicity shells (pding4_001, pding4_003, MyoB,
  EcwtCQ066). A cap at the widest well-sampled per-shell width recovers the
  regression, confirming over-widening is the harm. No crystal reliably wins;
  the apparent overall-CC gains were all in noise shells (mult 2-3, CC<20%).

Delete the CLI flag, the BraggIntegrationSettings::reciprocal_profile setting,
and the per-frame fit block. Default (per-shell) integration is byte-identical.
NEXTGEN_INTEGRATOR.md records the finding as a dead-end for posterity.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 11:39:09 +02:00
leonarski_fandClaude Opus 4.8 00821e32ef rotation indexer: refine candidate cells and pick the best after refinement
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m24s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m32s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m40s
Build Packages / build:rpm (rocky8) (push) Successful in 15m8s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m12s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m25s
Build Packages / build:windows:nocuda (push) Successful in 15m47s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m50s
Build Packages / build:windows:cuda (push) Successful in 17m29s
Build Packages / XDS test (durin plugin) (push) Successful in 8m31s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m3s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (neggia plugin) (push) Successful in 8m31s
Build Packages / Build documentation (push) Successful in 1m6s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m16s
Build Packages / build:rpm (rocky9) (push) Successful in 12m56s
Build Packages / DIALS test (push) Successful in 12m42s
Build Packages / Unit tests (push) Successful in 1h34m10s
De-novo two-pass indexing failed on large-cell / superstructure / modulated
crystals (EcwtAL500 0%, EcwtCQ034 0%) and mis-handled a pseudo-symmetric one
(EcwtCQ066 14%). The common cause: the choice of unit cell was made too early,
on the raw pre-refinement spot fraction, which is an unreliable discriminator
(a correct hexagonal cell indexes only ~13% of the un-refined accumulated
spots, while a wrong larger cell can index more).

Move the decision to after geometry refinement, where it is reliable:

- FFTIndexer now OFFERS widened candidate cells instead of deciding. ReduceResults
  gains a `widen` flag: the standard path (9 shortest vectors) is unchanged; the
  widened path anchors the two short axes and lets the third range over all
  filtered vectors (+dedup) to reach the long axis of an elongated cell.
  FilterFFTResults takes the peak count as a parameter (30 standard, 60 widened).
  RunInternal appends widened candidates only when its standard best indexes
  poorly, so compact crystals are untouched.

- RotationIndexer fully refines the top few candidates and keeps the one that
  indexes the most spots under its own refined geometry (IndexedFraction). Each
  refine is length-bounded (1.2x the found cell) so a free triclinic refine cannot
  drift onto a pseudo-translation / modulation supercell (CQ034's satellites). The
  earlier (primary) candidate is preferred: a later one is adopted only if it
  indexes clearly more and reasonably well in absolute terms, so a twin's noisy
  near-tie cannot displace it. Extra/twin lattices are only searched when the
  chosen cell is the FFT primary (lattice[0]), since MultiLatticeSearch's
  rotations are derived from that primary.

- The pseudo-symmetry guard (de-novo only - a user-fixed space group is always
  honored) is a ratio of refined fractions: refine the primitive as triclinic and
  drop to P only if the constrained cell indexes less than half of it. A false
  promotion indexes badly under its constraint (CQ066 ratio ~0.1) while genuine
  higher symmetry, including R-centred, indexes comparably (Ins_H R3 ratio ~0.7)
  and is kept.

Validated on the full /data/rotation_test battery: AL500 0->89% (C2), CQ034
0->99%, CQ066 14->93% (ISa 7.2->13.7); the other 15 crystals keep their exact
cell, space group, indexing rate and ISa (no regressions).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:17:53 +02:00
leonarski_fandClaude Opus 4.8 35607057d9 CC1/2 merge: assign half-sets by deterministic per-image hash
Build Packages / build:windows:cuda (push) Failing after 4m30s
Build Packages / build:windows:nocuda (push) Successful in 10m0s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m43s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m46s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m53s
Build Packages / build:rpm (rocky8) (push) Successful in 14m54s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 15m10s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m34s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m37s
Build Packages / XDS test (durin plugin) (push) Successful in 8m5s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m50s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (neggia plugin) (push) Successful in 8m44s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m45s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m5s
Build Packages / build:rpm (rocky9) (push) Successful in 13m8s
Build Packages / DIALS test (push) Successful in 12m47s
Build Packages / Unit tests (push) Successful in 1h0m40s
MergeOnTheFly::AddImage picked each image's CC1/2 half from a shared
mt19937 drawn in call order (and before Mask), so the split depended on
iteration/thread order and on how many images were masked. The class is
mutex-guarded for concurrent "on-the-fly" use, so any parallel merge would
make CC1/2 non-reproducible - a latent race.

Assign the half as a splitmix64 hash of the image's stable index instead,
computed after Mask. The split is now reproducible run-to-run, independent
of AddImage order, parallel-safe, and decoupled from masking. Callers pass
the outcome's vector index as the image id.

Verified: lyso_ref two-pass -M -P rot3d gives identical CC1/2 across runs
(overall 99.6%, P41212); hash split is balanced ~50/50.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 07:25:34 +02:00
leonarski_fandClaude Opus 4.8 a773aaa6a9 WriteReflections: write merging statistics + ISa to the mmCIF output
Build Packages / build:windows:cuda (push) Failing after 2m40s
Build Packages / build:windows:nocuda (push) Failing after 2m40s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m47s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m42s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m46s
Build Packages / build:rpm (rocky8) (push) Successful in 14m52s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m10s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m47s
Build Packages / Unit tests (push) Failing after 19m1s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / Build documentation (push) Successful in 1m0s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m54s
Build Packages / XDS test (durin plugin) (push) Successful in 8m7s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m24s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m5s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m4s
Build Packages / build:rpm (rocky9) (push) Successful in 12m58s
Build Packages / DIALS test (push) Successful in 12m32s
The .cif (--scaling-output cif) now carries the per-shell and overall merging
statistics in the standard _reflns / _reflns_shell categories (resolution, redundancy,
completeness, <I/sigma>, R-rim/R-meas, CC1/2) plus the Diederichs asymptotic I/sigma
(ISa) as a free-text _reflns.pdbx_diffrn_ISa item (no standard CIF tag exists). The
MergeStatistics and the ISa string are threaded through WriteReflections to the mmCIF
writer; jfjoch_process and jfjoch_scale pass them. Values match the text statistics table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:29:45 +02:00
leonarski_fandClaude Opus 4.8 636ed1e180 space group: use merge reduced chi^2 (not R-meas) for the consistency check
The R-meas-ratio over-symmetry guard (commit 7ade6d9) missed a near-perfect
pseudo-2-fold: InsH2 (XDS R3) has a false 2-fold at operator CC 0.85 whose merged
intensities still correlate well, so R-meas inflated only ~1.3x and R32 was accepted
(ISa collapsing to 3.9). The reduced chi^2 (within-orbit scatter / sigma^2) is the
right signal - false equivalents disagree by many sigma even when they correlate well.

Measured chi^2 ratio (candidate / best subgroup) across the test set: true point groups
<= 1.63 (a cubic merge of a noisy crystal, InsI3, is the worst real case), false >= 2.26
(InsH2) and 6.0 (InsH3); threshold 2.0 separates them. InsH2 now resolves to R3 (ISa 6.1
from the false R32's 3.9); InsH3/InsI2/InsI3/cytC/lysoC/MyoB/EP0210 unchanged and correct.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 21:54:51 +02:00
leonarski_fandClaude Opus 4.8 fea2790a41 rotation_indexer: detect hexagonal metric de-novo (fixes trigonal cytC indexing)
The hexagonal lattice metric (two equal axes at 60/120 deg, both perpendicular to the
third) is also satisfied by its ortho-hexagonal C-centred supercell, so the
geometry-keyed LatticeSearch lands a metrically-hexagonal lattice on the C-orthorhombic
setting. The -S path already re-expresses it (HexagonalConventional) keyed on the
supplied space group; do the same de-novo, keyed on the metric of the reduced PRIMITIVE
cell (rhombohedral lattices have a rhombohedral primitive cell and are unaffected).

cytC (P3121) now indexes de-novo as hexagonal 83.8/83.8/88.6 gamma=120 and merges in
P3121 (CC1/2 99.7%, ISa 13.1), for both test crystals, instead of the orthohexagonal
C2/P1. lysoC/InsI3/InsH3 unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 21:31:20 +02:00
leonarski_fandClaude Opus 4.8 52248b8f5c docs: design for auto-gated, data-driven ice-ring detection
Records the validated z-score detector (on the azint radial background profile) and
position/width estimation, for a future --detect-ice-rings=auto. Not yet implemented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 21:22:31 +02:00
leonarski_fandClaude Opus 4.8 73d9e1de39 ice rings: widen the exclusion band to 0.03 q (measured ring FWHM)
The hexagonal-ice ring FWHM measured on JUNGFRAU data (azimuthal radial background
profile) is ~0.06 q, so the exclusion half-width should be ~0.03; the previous 0.02
under-covered the strong low-res rings. On EP0117 with --detect-ice-rings this lifts
CC1/2 37->50%, and combined with --reject-outliers 3 (which down-weights the
radiation-damaged late frames) reaches ~94% (XDS 98.5%). Only active with
--detect-ice-rings, so default behaviour is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 21:09:18 +02:00
leonarski_fandClaude Opus 4.8 7ade6d9906 space group: reject over-symmetrised point groups by merge R-meas consistency
Per-operator CC alone cannot separate a real weak symmetry operator from a false
moderate one: a noisy crystal's genuine cubic 2-folds (InsI2 I23, CC ~0.51) score
BELOW a pseudo-symmetric crystal's false 2-fold (InsH3, CC 0.64), so no CC threshold
works. Add a self-consistency test: a candidate point group is accepted only if
merging the intensities under it does not inflate R-meas past max_rmeas_ratio (1.5x)
the most-consistent candidate. A false operator forces non-equivalent reflections
together and blows R-meas up; a real one leaves it flat.

Fixes InsH3: was over-called R32 (ISa collapsed to 2.2 from the forced merge), now
correctly R3/H3 (ISa 10.2, matching XDS). InsI2 stays I23, and lysoC/InsI3/MyoB/
EP0210/cytC/lyso_ref are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:32:03 +02:00
leonarski_fandClaude Opus 4.8 23d58b6202 integration: robust, well-conditioned per-frame --reciprocal-profile fit
Fit the reciprocal tangential-width model y(q)=a0+a1*t+a2*t^2 in a centered,
standardized variable t=(q-qbar)/qscale instead of the raw {1,q,q^2} monomials:
the raw normal matrix went near-singular when the strong spots span a narrow
q-range (small cell / sparse still), letting tiny per-frame jitter swing the
curvature into a wild over-wide profile. Adds IRLS (Huber) robustness, a ridge
on the curvature (sharp-crystal prior), and clamps the applied width to the
fitted q-range (no extrapolation). Stays strictly per-frame (no dataset pooling),
so it works online and for stills. Neutral on rotation data (cytC high-res CC1/2
win preserved 66.8 vs 65.6%).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:01:05 +02:00
leonarski_fandClaude Opus 4.8 8d18c635ff scaling: robust (Cauchy) loss on the per-image scale fit
The per-image G/B/mosaicity fit used a plain L2 loss, so a few outlier
reflections occasionally dragged a frame into a bad optimum - a stochastic
(~15% of runs) per-frame mis-scaling that elevated R-meas and collapsed CC1/2
at low symmetry (the image-level CC1/2 half-split makes the damage look patchy
across shells, while the data is genuinely noisier). A Cauchy loss (3 sigma)
soft-downweights those outliers without a hard cut: MyoB 0/10 catastrophic
(was ~2/10), R-meas stable, and ISa improves on every test crystal
(EP0210 9.2->12.4, MyoB 12.5->14.6, lysoC 10.4->11.2, cytC 11.5->12.6),
most on low symmetry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:00:42 +02:00
leonarski_fandClaude Opus 4.8 21dd99524f rotation_indexer: map FFT primitive cell to the SG conventional setting
When a space group is supplied without a reference cell, de-novo two-pass
rotation indexing fed the FFT's Niggli-reduced primitive cell straight into
XtalOptimizer as if it were the conventional cell. For non-primitive lattices
(centered I/F/R/C, or hexagonal where the primitive pair sits at gamma=60) the
conventional-system model then refined to a wrong minimum and indexed 0% of
frames: cytC (P3121) gave 103.9/103.9/78 instead of 83.7/83.7/88.6, insulin
(I213) 66.7 instead of 77.65, insulin-R3 51/51/36 instead of 81.4/81.4/33.3.

Run LatticeSearch on the FFT primitive cell (it already yields the correct
conventional cell + reindex for I/F/R/C). For the one remaining gap - a
metrically hexagonal lattice that the geometry-keyed search lands on the
ortho-hexagonal C setting - re-express the reduced primitive cell in
conventional hexagonal axes (b -> b - a opens gamma 60 -> 120).

De-novo "-S" now indexes cytC/insu/Ins_H/lyso/MyoB/EP/lyso_ref at 100% with the
correct cell; the "-C -S" path is unchanged. The helper stays in this .cpp
(g++) rather than CrystalLattice.h to avoid recompiling CUDA units, which is
broken under the box's CUDA-13 nvcc; promote it to a method once that is fixed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 14:29:50 +02:00
leonarski_fandClaude Opus 4.8 3aec235f67 integration: add --reciprocal-profile dial (global reciprocal-space width)
Build Packages / Unit tests (push) Successful in 59m53s
Build Packages / build:windows:nocuda (push) Successful in 18m24s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m30s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m33s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m54s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m20s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m2s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m29s
Build Packages / build:rpm (rocky8) (push) Successful in 14m37s
Build Packages / build:rpm (rocky9) (push) Successful in 12m44s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m10s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 12m22s
Build Packages / XDS test (durin plugin) (push) Successful in 7m34s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m41s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m52s
Build Packages / Generate python client (push) Successful in 24s
Build Packages / Build documentation (push) Successful in 1m0s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 16m53s
The per-shell profile width is learned in pixels, so it varies ~4x with resolution
(mostly the geometric projection of a near-constant reciprocal-space relrod) and must
be binned per shell -> it starves at high resolution / on sparse data. The new
--reciprocal-profile flag instead learns ONE global width in reciprocal space,
sigma2_q,tan = A + B|q| + C|q|^2: the Jacobian g_tan=cos(2theta) removes the geometric
projection, and C|q|^2 is the crystal mosaicity relrod (variance ~(eta|q|)^2). Applied
per reflection as sigma2_tan,px = (A + B|q| + C|q|^2)/g_tan^2 (B,C clamped >=0;
quadratic->linear->constant fallback).

Off by default. On the sharp HEWL test crystal (mosaicity 0.091deg, so C fits to ~0 and
it reduces to the validated linear form) it is metric-neutral: ISa 16.2->16.3, anomalous
0.92x unchanged, CCref band 90.0->89.9, CC1/2 a touch lower (per-shell isn't starved at
23k spots/shell, and a global fit is less flexible). So: simpler + more transferable at a
small CC1/2 cost, ISa/anomalous held. Its payoff is on MOSAIC crystals (large C|q|^2),
where per-shell starves on the wide weak high-res spots and 6 shells are too coarse; both
lyso test crystals are sharp, so it ships as a dial to try on mosaic data elsewhere.
A separate radial relrod fit was tried and dropped (no gain). See NEXTGEN_INTEGRATOR.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 14:26:08 +02:00
leonarski_fandClaude Opus 4.8 5ba5fe8ad1 integration: centre the profile on the predicted sub-pixel position
Build Packages / Unit tests (push) Successful in 58m42s
Build Packages / build:windows:cuda (push) Successful in 18m14s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m7s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m4s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m30s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m44s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m25s
Build Packages / build:rpm (rocky8) (push) Successful in 21m46s
Build Packages / build:rpm (rocky9) (push) Successful in 13m20s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m20s
Build Packages / DIALS test (push) Successful in 12m40s
Build Packages / XDS test (durin plugin) (push) Successful in 8m12s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m37s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m23s
Build Packages / Generate python client (push) Successful in 30s
Build Packages / Build documentation (push) Successful in 1m11s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 9m59s
The profile was learned and applied on the integer pixel round(predicted), so a
shared profile sits up to 0.5 px off the true spot (and stacking spots with random
sub-pixel offsets broadens the learned profile). Build the Gaussian per reflection
instead, centred on the predicted sub-pixel offset -- noise-free geometry, unlike the
observed centroid, which hurt -- and elongated radially as before.

HEWL rotation @1.0A: ISa 15.7->16.2, CCref band 89.9->90.0, CCxds 94.8->95.0 (high-res
1.00A CCref 66.0->66.9); sharp serial stills 1.68A CC1/2 61.6->62.5; anomalous S peak
0.92x XDS (no accuracy traded). De-broadening the learned width by the 1/12 px^2/axis
integer-binning floor was tested and rejected (it over-narrows).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 07:34:06 +02:00
leonarski_fandClaude Opus 4.8 26901da42f integration: de-contaminate the profile width, add a radial parallax ellipse
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m30s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m44s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m46s
Build Packages / build:rpm (rocky8) (push) Successful in 15m1s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m20s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m25s
Build Packages / XDS test (durin plugin) (push) Successful in 9m46s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m25s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m44s
Build Packages / Create release (push) Skipped
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky9) (push) Successful in 11m47s
Build Packages / Build documentation (push) Successful in 52s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 13m0s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m27s
Build Packages / DIALS test (push) Successful in 14m10s
Build Packages / Unit tests (push) Successful in 59m3s
Build Packages / build:windows:cuda (push) Successful in 16m54s
Build Packages / build:windows:nocuda (push) Successful in 15m18s
The profile-fit width came from the full 13x13 second moment, which runs 3-8x
wider than the true spot: neighbour reflections leak into the (unmasked) learning
grid -- catastrophic at low res where spots crowd the beam -- and the far corners
(lever arm dx^2+dy^2 up to 72) add rectified background noise. Splitting the spot
moment into radial vs tangential shows the tangential width is isotropic
(mosaicity/divergence) while the radial excess is pure sensor parallax ~tan^2(2th).

- Measure the width over the signal disk r1 on the monochromatic path (inherently
  excludes neighbours, caps the radial tail); keep the generous full-grid width on
  the broadband/stills path (sparse spots, the centroid-undersampling floor).
- Extend the (was bandwidth-only) radial ellipse with an analytic, material-aware
  parallax term c_par*tan^2(2theta), c_par = Var(z)/pixel^2 from sensor thickness +
  material + energy (parallax_var_px2; Si and CdTe), plus a fixed weak-spot capture
  term on the monochromatic path only.

HEWL rotation @1.0A: ISa 13.5->15.7, CC1/2 1.12A 91.3->95.9 / 1.05A 83.0->85.2,
external CCref band 88.1->89.9, CCxds 93.4->94.8, R-meas 9.4->8.7; low/mid flat.
Sharp serial stills gain slightly from parallax; broadband stills neutral.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 00:54:59 +02:00
leonarski_fandClaude Opus 4.8 7a4fadf29b bragg_prediction: reciprocal Lorentz divides by |S||S0|, not S.S0
S.S0 = |S||S0| cos(2theta), so dividing the rocking rate |m2 . (S x S0)| by it
added a spurious 1/cos(2theta) factor (1.0 at low res, 1.8x at 1 A) to the
reciprocal Lorentz, hence to the absolute/Wilson scale. Divide by |S||S0| to get the correct zeta*sin(2theta) (times a constant 1/lambda^2 the overall scale
absorbs). The spurious factor depends on 2theta only, so it cancels within a
resolution shell and between symmetry mates -- CC1/2, CCref and R-meas are
metric-neutral -- but it corrupts the absolute scale and destabilises the
per-image B-factor and cross-resolution error model. CPU and GPU kernels.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8 40f650b9e7 scale_merge: pool the per-frame background across each rocking event in Combine3D
Each partial subtracted its own independently-estimated per-frame background,
so a weak full assembled from N frames accumulated one background-estimation
variance per frame. The true background is flat over the few frames of one
rocking event, so replace each partial's background by the event mean and
correct its intensity by n_bkg*(bkg - pooled), where n_bkg = (sigma^2 - I)/bkg
is the effective background-pixel count -- correct for weak AND strong
reflections (sigma^2/bkg would over-count strong ones and over-correct them).
Single-frame events are a no-op.

Measured (rotation lyso @1.0 A): 1.05 A CC1/2 81.3 -> 83.0, R-meas 81.4 ->
77.2, CCref +0.2, CCxds +0.3; overall R-meas 10.0 -> 9.4%; ISa preserved (13.5).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 22:41:37 +02:00
leonarski_fandClaude Opus 4.8 eccd10a0a7 integration: neighbour-mask the profile-fit background ring, widen r3 8->10
ProfileIntegrate2D::BoxSum now excludes every predicted reflection's r2 disk
from the r2..r3 background annulus (mirroring BraggIntegrate2D), so a neighbour
Bragg peak can no longer bias a reflection's background high and over-subtract.
With neighbours excluded the annulus can safely widen, so the default r_3 goes
8 -> 10 (more background pixels, lower-variance estimate).

Measured (rotation lyso @1.0 A, external CCref/CCxds vs XDS): 1.05 A CCref +1.3
/ CCxds +1.3, R-meas -5 pts, low-res R-meas unchanged. Serial (Jet8 @0.0002
bandwidth): 1.68 A CC1/2 +3.9 / CCref +1.1, 1.58 A +4.0 / +1.6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 22:41:14 +02:00
leonarski_fandClaude Opus 4.8 bbd888dcc3 integration: gate the background-ring sigma-clip to stills (bandwidth>0)
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m41s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 15m4s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 15m5s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m5s
Build Packages / build:rpm (rocky8) (push) Successful in 15m12s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m22s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m58s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m9s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m12s
Build Packages / Generate python client (push) Successful in 21s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 9m17s
Build Packages / Build documentation (push) Successful in 48s
Build Packages / build:rpm (rocky9) (push) Successful in 12m57s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m39s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m20s
Build Packages / DIALS test (push) Successful in 13m15s
Build Packages / build:windows:nocuda (push) Successful in 16m12s
Build Packages / Unit tests (push) Successful in 1h0m33s
Build Packages / build:windows:cuda (push) Successful in 18m12s
The 76e88b0f sigma-clip (reject ring pixels above mean+3*sqrt(mean))
de-biases bandwidth-streaked high-resolution stills, but it ran on all
data. On rotation (no streaks) it clips legitimate high background pixels
and biases the mean low, slightly inflating weak intensities and hurting
the anomalous signal. Gate it to bandwidth>0 (stills), matching how the
814dff34 radial-profile change is already gated.

Rotation lysozyme (self-scaled, smooth-g, -A): anomalous S-peak 0.84x ->
0.86x of XDS (SD_MET 11.71 -> 11.94, CL_CL 1.28x -> 1.25x), ISa unchanged.
Stills (bandwidth set) are byte-identical (clip still applies).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8 99b923e727 scaling: smooth per-frame scale G across frames before the rot3d combine
ScaleOnTheFly fits each frame's scale G independently with no neighbour
coupling, so the few partials of one rocking event are weight-summed in
combine3D on inconsistent scales - jitter that never enters the full's
counting sigma and instead surfaces as scatter between symmetry mates,
inflating the error-model b (low ISa). A centered moving average of
log(G) over a small frame window (default 9, on for rot3d) removes it,
mirroring XDS's smooth scaling. Complementary to --scale-fulls (which
rescales between fulls, after the combine): smoothing fixes within-event
scale, scale-fulls fixes between-full.

On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa
11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest).
Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS.

--smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a
diagnostic that fixes the scaling mosaicity for sweeps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8 02477f1ce4 space group: determine centering from absences; jfjoch_process re-indexes from scratch
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m5s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m29s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m41s
Build Packages / build:rpm (rocky8) (push) Successful in 14m46s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m59s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m13s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m29s
Build Packages / XDS test (durin plugin) (push) Successful in 7m54s
Build Packages / Generate python client (push) Successful in 30s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m12s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (neggia plugin) (push) Successful in 8m23s
Build Packages / Build documentation (push) Successful in 1m8s
Build Packages / build:windows:nocuda (push) Failing after 25m35s
Build Packages / build:windows:cuda (push) Failing after 25m35s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m12s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m2s
Build Packages / build:rpm (rocky9) (push) Successful in 12m45s
Build Packages / DIALS test (push) Successful in 12m52s
Build Packages / Unit tests (push) Successful in 59m5s
Fixes wrong space-group assignment on cubic insulin (true I2_13, a=77.6).

SearchSpaceGroup: drop the lattice-centering gate (and the now-unused
lattice_centering option). The indexer often returns the conventional cell,
whose geometry hides I/F/C centering - that information lives only in the
systematic absences. Stage B now tests every centering of the point group and
confirms it from the data, so an indexer-reported 'P' no longer excludes
I2_13/I23. (I23 and I2_13 remain indistinguishable by absences and are reported
as such.)

jfjoch_process: discard any unit cell stored in the input HDF5 by default so the
cell is re-determined from scratch. A stale/wrong stored cell otherwise resolves
the indexing algorithm to FFBIDX, which trusts that cell and locks onto the wrong
lattice (Ins_I_3 went from 2.7% -> 76% indexing). A user-supplied -C cell still
applies.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 16:39:03 +02:00
leonarski_fandClaude Opus 4.8 107bcae0f0 merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m26s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m34s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m59s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m17s
Build Packages / XDS test (durin plugin) (push) Successful in 7m40s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m46s
Build Packages / build:rpm (rocky8) (push) Successful in 12m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m11s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky9) (push) Successful in 13m2s
Build Packages / Build documentation (push) Successful in 42s
Build Packages / DIALS test (push) Successful in 13m38s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m9s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m44s
Build Packages / Unit tests (push) Successful in 57m53s
Build Packages / build:windows:nocuda (push) Successful in 16m26s
Build Packages / build:windows:cuda (push) Successful in 18m26s
RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned
*median* of the squared symmetry-mate deviations, chosen for robustness. But a
single deviation squared over its variance is chi-square(1)-distributed, whose
median is only 0.4549 of its mean, so the fit was calibrating the variances to
0.4549x their true value: the merged sigmas came out ~1.48x too small and the
achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally
well-behaved (flat chi^2 across resolution) but globally over-confident, which
inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for
downstream weighting / French-Wilson.

Divide the binned median by the chi-square(1) median (0.4549) to recover an
unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the
median while targeting reduced chi^2 = 1. Also compute the achieved median
reduced chi^2 (same normalization) and report it on the "Error model" line so
mis-calibration can no longer drift silently.

Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2
2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and
all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common
constant); only the absolute sigma calibration is corrected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:57:42 +02:00
leonarski_fandClaude Opus 4.8 814dff34cb integration: radially-elongated profile for bandwidth-streaked high-res stills
With a finite energy bandwidth each reflection is smeared RADIALLY by
sigma_bw = bandwidth_sigma * R_px (R_px = distance from the beam centre, so large
at high resolution): high-resolution spots become radial streaks. The isotropic
per-shell Gaussian both mis-weights them and clips the streak tail on the fixed
profile grid, losing intensity (biased low, noisy).

When a bandwidth is set, fit each reflection with a per-reflection Gaussian
elongated only along its radial direction - sigma^2_radial = sigma^2_intrinsic +
sigma_bw^2, sigma^2_tangential = sigma^2_intrinsic - on a grid grown to hold the
streak. Unlike an isotropic widening this adds no tangential background. It only
engages where the smear exceeds the intrinsic spot (high resolution); low/mid
resolution and monochromatic data (bandwidth 0, e.g. rotation) are untouched.

On the HEWL serial-stills jet (with the background sigma-clip) this lifts the
overall CC-vs-reference 52 -> 55% and the high-resolution I/sig (1.7 A 0.5 -> 1.4),
recovering the 2.0-2.5 A band, with CC1/2 preserved (the per-shot noise the wider
region adds is averaged out by the high serial multiplicity). Rotation ISa 19.1
unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 12:01:56 +02:00
leonarski_fandClaude Opus 4.8 76e88b0fca integration: sigma-clip the background ring to de-bias high-resolution stills
Weak high-resolution still reflections were systematically over-subtracted: a
bandwidth-streaked high-res spot (or a neighbour) leaks into the r2-r3 background
annulus and biases its mean high, so the subtracted background is too large and
the merged high-resolution intensities go negative (seen as reproducibly negative
<I/sig> at 100% completeness and high multiplicity past ~1.9 A).

Add one high-outlier sigma-clip pass to the box-sum background (reject ring pixels
above mean + 3*sqrt(mean), recompute) so the contamination no longer inflates it.
A clean Poisson background is essentially unchanged (~0.1% exceed the cut). On the
HEWL serial-stills jet this de-biases the high-res band - <I/sig> 2.03 A 0.9 -> 1.6,
1.79 A -0.1 -> +0.7 - extends the usable resolution ~2.2 -> ~2.0 A and improves
overall R-meas 130 -> 124%, with CC1/2 and CC-vs-reference neutral. The rotation
crystal is unchanged (ISa 19.1), its clean backgrounds being barely clipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 11:50:31 +02:00
leonarski_fandClaude Opus 4.8 7b464e4b3c indexing: deconvolve energy bandwidth from the profile radius
The profile radius (intrinsic excitation-error width = mosaicity + divergence)
was the plain RMS of dist_ewald over indexed spots. With a finite energy
bandwidth that spread is broadened by the bandwidth's radial smear
sigma_bw = bandwidth_sigma*lambda/(2 d^2), which prediction then re-applies per
reflection - so bandwidth was counted twice and the radius was inflated (most at
high resolution, sigma_bw ~ 1/d^2). Subtract the bandwidth variance from the
measured spread so the radius is the intrinsic width. bandwidth = 0
(monochromatic / rotation) is unchanged. Small for narrow bandwidths (~6% of the
variance, ~4% radius on the 1% jet); matters for wide-bandwidth / pink beam.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 10:56:49 +02:00
leonarski_fandClaude Opus 4.8 82b8a9752c rotation: fix mosaicity underestimate + default-on capture-uncertainty (ISa 10.7->19.1)
The rotation per-image mosaicity was ~3x too small (0.045 vs the true 0.13deg),
which crippled the partiality model and capped per-observation precision: it
predicted reflections on too few frames and over-peaked the rocking partiality,
so the rot3d-combined fulls were ~1.7x noisier than XDS's, the integration
bottleneck behind the jungfraujoch-vs-XDS ISa gap.

Two root causes, both fixed:
- CalcMosaicityXDS (Kabsch-2010 MLE) searched each spot's exact-Bragg phi only
  within +-wedge (the 0.2deg oscillation). Reflections recorded at larger rocking
  offset - the tail that defines the mosaic width - fell outside and were dropped,
  truncating the tau distribution so the MLE underestimated ~2x. Widen the search
  window to wedge+0.8deg; the MLE then converges to the true 0.13deg (and is
  insensitive to widening further, since it weights by the recorded fraction).
- ScaleOnTheFly then re-refined the mosaicity from the intensity residual, which
  is degenerate with the per-image scale G and collapses it toward its floor.
  Trust the (now correct) indexing mosaicity and keep it fixed during scaling.

With the correct mosaicity, --capture-uncertainty (which down-weights the
over-extrapolated under-captured fulls) now pays off strongly, so default it ON
(1.0) for the rot3d combine; it stays off for non-rot3d. Together on the HEWL
rotation crystal: ISa 10.7 -> 19.1, and anomalous peak height vs XDS goes from
52% to ~78% (CL_CL 1.92x -> 1.29x). This reaches XDS's own published-correction
ceiling (DECAY+ABSORP+MODPIX ~= 19.6); the remaining gap to its quoted ISa 28 is
the I->inf extrapolation. No effect on the stills path (rotation-only code).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 23:57:04 +02:00
leonarski_fandClaude Opus 4.8 ed98775ba3 rot3d: capture-aware sigma for under-captured fulls + expose per-image scale
The rot3d combine over-extrapolates fulls reconstructed from only a fraction f
of their rocking curve (min_partiality admits f as low as 0.02). Against XDS on
HEWL these low-capture fulls are systematically biased HIGH (+15% at f=0.8 to
+100% at f=0.3), and the bias - not random scatter - is the strong-reflection
floor that hurts anomalous accuracy.

--capture-uncertainty <coeff> (default 0 = off, baseline bit-identical) adds a
systematic uncertainty ~coeff*(1-f)*I to each full's sigma, so the merge
down-weights the over-extrapolated fulls and the error model treats their
scatter as expected. Unlike outlier rejection (which trades accuracy for CC1/2),
this fixes a real bias, so accuracy improves: at coeff=1.0 the anomalous peak
height vs XDS rises CL_CL +16%, SD_MET/SG_CYS +5-6%, ISa 10.7->11.0. Rotation-
only (no-op for stills, which never combine).

Also expose the per-image scale offline: Combine3D now carries the first-pass
per-image scale metadata (G, B, mosaicity, wedge, CC) forward instead of
dropping it, and jfjoch_process -M writes <prefix>_image.dat from it (the
offline self-scaling result was otherwise unobservable - process.h5's per-image
arrays are only filled on the online path). This enabled the XDS DECAY
comparison (jfjoch G tracks XDS, r=0.93).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 19:41:20 +02:00
leonarski_fandClaude Opus 4.8 786af96b3b SearchSpaceGroup: POINTLESS-style rewrite, pipeline integration, twinning test
Space-group search (image_analysis/scale_merge/SearchSpaceGroup):
- Two-stage POINTLESS-style determination. Stage A scores each distinct rotation
  operator once (was once per candidate space group, ~34x faster on lysozyme:
  ~26s -> <1s) and picks the largest point group all of whose operators confirm.
  Stage B picks the maximal space group whose predicted absences are confirmed
  weak, fixing the prototype's default to the symmorphic group (it returned P422
  instead of P4(3)2(1)2). Enantiomorphic / origin-ambiguous pairs (P4(1) vs P4(3),
  I222 vs I2(1)2(1)2(1)) are reported as indistinguishable.
- Constrain candidates to subgroups of the lattice (metric) holohedry and weigh
  centering only P-vs-metric, fed from rotation indexing's LatticeSearch result.

Integration / pipeline:
- With no user-fixed space group, predict in P (IndexAndRefine) so the
  centering-absent reflections are integrated and the search can confirm/deny
  centering (catching pseudo-centering / a missed superstructure) instead of
  trusting the metric; a user-fixed group still rejects absences in integration.
- JFJochProcess: scale+merge in P1 -> determine the space group -> set it and
  re-scale+merge in it (statistics then come out in the right symmetry) -> write
  it to /entry/sample/space_group_number (new EndMessage.space_group_number,
  preferred by NXmx::Sample). jfjoch_scale no longer searches; it consumes the
  file's space group (and no longer clobbers it with an empty -S).

Twinning (new image_analysis/scale_merge/TwinningAnalysis): Padilla-Yeates L-test
(<|L|>, <L^2>; acentric-only, positive intensities so L is bounded) plus a
shell-normalised <I^2>/<I>^2 second moment and a twin-fraction estimate. Reported
after the final merge in jfjoch_process and jfjoch_scale, and surfaced in the
jfjoch_viewer merge-statistics window with a red outline when twinning is suspected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 16:11:28 +02:00
leonarski_f 54c667190f v1.0.0-rc.155 (#65)
Build Packages / Unit tests (push) Successful in 1h26m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m39s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m55s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8) (push) Successful in 12m28s
Build Packages / build:rpm (rocky9) (push) Successful in 13m20s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m43s
Build Packages / DIALS test (push) Successful in 14m21s
Build Packages / XDS test (durin plugin) (push) Successful in 7m48s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m31s
Build Packages / Generate python client (push) Successful in 15s
Build Packages / Build documentation (push) Successful in 53s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
* jfjoch_viewer: Some graphical improvements.
* jfjoch_viewer: Simplify und unify data analysis settings.
* jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.

Reviewed-on: #65
2026-06-25 22:01:48 +02:00
leonarski_f 75e401f0e5 v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

Reviewed-on: #63
2026-06-23 20:29:49 +02:00
leonarski_f 90e804acd7 v1.0.0-rc.150 (#60)
Build Packages / Unit tests (push) Successful in 42m49s
Build Packages / DIALS test (push) Successful in 29m45s
Build Packages / XDS test (durin plugin) (push) Successful in 19m27s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 18m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 13m0s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / Build documentation (push) Successful in 1m25s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m53s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 12m49s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m7s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m9s
Build Packages / build:rpm (rocky8) (push) Successful in 13m24s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m15s
Build Packages / build:rpm (rocky9) (push) Successful in 14m30s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m14s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 9m5s
* jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin.
* jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins.

Reviewed-on: #60
2026-06-15 20:24:15 +02:00
leonarski_f cc3eb8352c v1.0.0-rc.148 (#58)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m28s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m9s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m47s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m58s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m39s
Build Packages / build:rpm (rocky8) (push) Successful in 11m43s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m59s
Build Packages / Generate python client (push) Successful in 35s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m48s
Build Packages / build:rpm (rocky9) (push) Successful in 12m32s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m24s
Build Packages / XDS test (durin plugin) (push) Successful in 7m35s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m50s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m40s
Build Packages / DIALS test (push) Successful in 11m19s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin

Reviewed-on: #58
2026-06-08 08:30:35 +02:00
leonarski_f 75de40f52b v1.0.0-rc.147 (#57)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 7m27s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 8m20s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 7m35s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 5m59s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 7m25s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 8m30s
Build Packages / build:rpm (rocky8) (push) Successful in 7m39s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m16s
Build Packages / build:rpm (rocky9) (push) Successful in 9m35s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m6s
Build Packages / Generate python client (push) Successful in 12s
Build Packages / Build documentation (push) Successful in 31s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 7m6s
Build Packages / DIALS test (push) Successful in 12m3s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m11s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 5m50s
Build Packages / Unit tests (push) Successful in 57m33s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

* jfjoch_viewer: Add reciprocal space viewer
* jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset
* jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress)

Reviewed-on: #57
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-06-02 11:49:24 +02:00
leonarski_f fc68a9baed v1.0.0-rc.146 (#56)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m34s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m16s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m49s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m15s
Build Packages / XDS test (durin plugin) (push) Successful in 7m16s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky9) (push) Successful in 10m12s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 47s
Build Packages / DIALS test (push) Successful in 10m18s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 5m46s
Build Packages / build:rpm (rocky8) (push) Successful in 1h41m2s
Build Packages / XDS test (neggia plugin) (push) Successful in 1h59m18s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.

jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.

WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.

Reviewed-on: #56
2026-05-28 18:48:35 +02:00
leonarski_f 87fde1b32e v1.0.0-rc.142 (#52)
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 16m19s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 16m29s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m51s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m17s
Build Packages / build:rpm (rocky8) (push) Successful in 12m7s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m34s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m44s
Build Packages / build:rpm (rocky9) (push) Successful in 16m4s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m49s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m56s
Build Packages / Generate python client (push) Successful in 1m0s
Build Packages / Unit tests (push) Has been skipped
Build Packages / Create release (push) Has been skipped
Build Packages / Build documentation (push) Successful in 1m43s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m14s
Build Packages / XDS test (durin plugin) (push) Successful in 8m58s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m44s
Build Packages / DIALS test (push) Successful in 13m4s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8
* Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent

Reviewed-on: #52
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-04-30 16:47:53 +02:00
leonarski_f d760b12a18 v1.0.0-rc.141 (#51)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m32s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m19s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 11m49s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m44s
Build Packages / build:rpm (rocky8) (push) Successful in 9m15s
Build Packages / build:rpm (rocky9) (push) Successful in 12m14s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m28s
Build Packages / Generate python client (push) Successful in 21s
Build Packages / XDS test (durin plugin) (push) Successful in 8m58s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m7s
Build Packages / Build documentation (push) Successful in 42s
Build Packages / Create release (push) Has been skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m24s
Build Packages / DIALS test (push) Successful in 14m1s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m26s
Build Packages / Unit tests (push) Failing after 1h20m14s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times
* frontend: Fix selection of FFTW in indexing settings

Reviewed-on: #51
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-04-30 13:04:54 +02:00
leonarski_f 239a441ee6 v1.0.0-rc.140 (#50)
Build Packages / Unit tests (push) Successful in 1h20m34s
Build Packages / build:rpm (rocky8) (push) Successful in 13m32s
Build Packages / Generate python client (push) Successful in 24s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m6s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m32s
Build Packages / XDS test (durin plugin) (push) Successful in 10m49s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 14m8s
Build Packages / DIALS test (push) Successful in 14m57s
Build Packages / Build documentation (push) Successful in 47s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m30s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m40s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m14s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m55s
Build Packages / build:rpm (rocky9) (push) Successful in 14m23s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m48s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m10s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection
* jfjoch_broker: Minor bug fixes for rare conditions

Reviewed-on: #50
2026-04-29 21:40:22 +02:00
leonarski_f 4878318c27 v1.0.0-rc.139 (#49)
Build Packages / Unit tests (push) Successful in 1h17m4s
Build Packages / Generate python client (push) Successful in 1m24s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 12m34s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m37s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 11m11s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 12m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m59s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m4s
Build Packages / build:rpm (rocky8) (push) Successful in 13m12s
Build Packages / XDS test (durin plugin) (push) Successful in 9m57s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m52s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m30s
Build Packages / build:rpm (rocky9) (push) Successful in 13m59s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m24s
Build Packages / Create release (push) Has been skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m54s
Build Packages / DIALS test (push) Successful in 13m14s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets

**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.

Reviewed-on: #49
2026-04-29 09:50:50 +02:00
leonarski_fandtakaba_k 230480e390 v1.0.0-rc.138 (#48)
Build Packages / XDS test (durin plugin) (push) Successful in 8m5s
Build Packages / DIALS test (push) Successful in 12m55s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m41s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 16m13s
Build Packages / build:rpm (rocky8) (push) Successful in 18m17s
Build Packages / build:rpm (rocky9) (push) Successful in 19m14s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m59s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 9m17s
Build Packages / Generate python client (push) Successful in 1m15s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m58s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 14m14s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 11m9s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 18m21s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m14s
Build Packages / Build documentation (push) Successful in 1m43s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m14s
Build Packages / Unit tests (push) Successful in 57m40s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

    jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time
    jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations
    jfjoch_broker: Goniometer axis name is converted to lowercase
    jfjoch_broker: Fix bug, where wrong HTTP error codes were returned
    jfjoch_broker: Improve sigma estimation during merging (K. Takaba)

---------

Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch>
Reviewed-on: #48
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-04-27 19:56:14 +02:00
leonarski_f c981e1b91c v1.0.0-rc.137 (#46)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m7s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m35s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m8s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m24s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m29s
Build Packages / build:rpm (rocky8) (push) Successful in 10m27s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m41s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m1s
Build Packages / Generate python client (push) Successful in 45s
Build Packages / Unit tests (push) Has been skipped
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky9) (push) Successful in 12m48s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m10s
Build Packages / XDS test (durin plugin) (push) Successful in 8m59s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m32s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m39s
Build Packages / DIALS test (push) Successful in 13m13s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: Better track time for each operation in the processing stack
* jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress)
* jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress)
* jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0)

Reviewed-on: #46
2026-04-25 19:59:21 +02:00
leonarski_f c1c170112c v1.0.0-rc.136 (#45)
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 11m17s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m48s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m57s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 15m15s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 15m35s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m29s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m55s
Build Packages / XDS test (durin plugin) (push) Successful in 8m17s
Build Packages / build:rpm (rocky9) (push) Successful in 12m17s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m2s
Build Packages / Generate python client (push) Successful in 30s
Build Packages / Create release (push) Has been skipped
Build Packages / Build documentation (push) Successful in 51s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m13s
Build Packages / DIALS test (push) Successful in 13m19s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m52s
Build Packages / Unit tests (push) Successful in 1h18m25s
Build Packages / build:rpm (rocky8) (push) Successful in 7m2s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress).

Reviewed-on: #45
2026-04-20 11:54:33 +02:00
leonarski_f bb9f5c715f v1.0.0-rc.135 (#44)
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m28s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m47s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m7s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m31s
Build Packages / build:rpm (rocky8) (push) Successful in 12m59s
Build Packages / build:rpm (rocky9) (push) Successful in 14m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m30s
Build Packages / Generate python client (push) Successful in 1m18s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m8s
Build Packages / XDS test (durin plugin) (push) Successful in 9m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m59s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m12s
Build Packages / DIALS test (push) Successful in 11m44s
Build Packages / Unit tests (push) Successful in 1h23m8s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance

Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-04-16 11:59:59 +02:00
leonarski_f 6133da1377 v1.0.0-rc.133
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 11m5s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 11m6s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m47s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m9s
Build Packages / Generate python client (push) Successful in 49s
Build Packages / Unit tests (push) Has been skipped
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m38s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m35s
Build Packages / build:rpm (rocky8) (push) Successful in 9m16s
Build Packages / build:rpm (rocky9) (push) Successful in 10m22s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m20s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 7m52s
Build Packages / Build documentation (push) Failing after 9s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)

Reviewed-on: #41
2026-03-26 20:50:33 +01:00
leonarski_f 166fcdb68f v1.0.0-rc.131 (#39)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 11m20s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m46s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 11m27s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m32s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m57s
Build Packages / build:rpm (rocky8) (push) Successful in 11m54s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m9s
Build Packages / build:rpm (rocky9) (push) Successful in 12m37s
Build Packages / Generate python client (push) Successful in 24s
Build Packages / Create release (push) Has been skipped
Build Packages / Build documentation (push) Successful in 57s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m12s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m4s
Build Packages / Unit tests (push) Successful in 1h17m43s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.

* jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS)
* jfjoch_viewer: Fix calibration (pedestal) images being open flipped
* jfjoch_process: Add space group detection (EXPERIMENTAL)

Reviewed-on: #39
2026-03-07 11:34:04 +01:00