189 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	fccf9b83e7	RotationScaleMerge: drop profiling + env gates, honour forced mosaicity ... Remove the [rsm] per-stage lap timing and the JFJOCH_RSM_NO_GPU / JFJOCH_RSM_CPU_COMBINE env gates now that the GPU-resident path is the validated default (it runs whenever a GPU is present, with the CPU loops as the bit-parity fallback; the diagnostic-dump path still uses the CPU combine). Honour a fixed (forced) mosaicity: SmoothMosaicityAndPartiality now overrides every frame with GetForcedMosaicity() when set, instead of always reading the per-frame integration value - so the caller can route the --mosaicity case through RotationScaleMerge (its partiality recompute makes it a natural fit) rather than a separate path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 11:41:31 +02:00
leonarski_fandClaude Opus 4.8	34b3c3c4e7	RotationScaleMerge: GPU merge + error-model reductions over the resident fulls ... Port the four fulls-walking reductions of MergeAndStats to the GPU, over the fulls group-CSR already resident from scale-fulls: the per-group inv-var mean + leverage- corrected error-model samples, the merge accumulate (inv-var sums + deterministic half-sets, error-model-corrected sigma, with outlier rejection), and R_meas + the per-shell usable count. The host keeps the parts that don't parallelise cleanly or are tiny: the I2-sort + 16-bin (a,b) median fit, the per-group reject median (a per-group median is awkward on the GPU - cheap on the host from the GPU cnt), the merged export, the shells and the gemmi completeness. Only per-group arrays (~55k) + the samples (~n_fulls, for the fit) come back - the fulls are not re-walked on the host. Device HalfForImage (splitmix64) + IceRingIndex mirror the host; the corrected-sigma uses (b*I_for_b)^2 (not b^2*I^2) to match the host rounding; the R_meas usable count requires finite d (the host counts only fulls with a valid shell, and a group's fulls share d, so the shell is assigned per group). Gated on fulls_resident (GPU combine+scale-fulls active); reject is fully supported so it runs for the default rot3d command. merge+stats ~0.49 -> ~0.37s, taking RSM on lyso to ~0.78s (was ~0.91). Validated across the battery: 15/15 deterministic crystals bit-identical to the CPU path (SG / ISa / CC1.2 / completeness / total-obs, and the exact outlier-reject count), only EP_cs_01-24 noise wobbles. The em-sort + a,b fit are the remaining host floor. Non-CUDA build unaffected (use_gpu_merge is always false there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 10:38:36 +02:00
leonarski_fandClaude Opus 4.8	786edf9929	RotationScaleMerge: GPU smooth-G, corr kept resident across the pass ... Apply smooth-G's corr adjustment on the device (a small kernel: corr[i] *= ratio[frame[i]] for flagged frames, double-then-float, matching host SmoothG) so the per-image corr never leaves the GPU: it now stays resident through scaling -> smooth-G -> per-frame CC -> combine, and across the two space-group passes exactly as the old host round-trip did. The host only builds the tiny per-frame ratio (g/g_smooth via the extracted ComputeSmoothGWindow) and refreshes host partials[].corr solely for the CPU-combine path (JFJOCH_RSM_CPU_COMBINE or the diagnostic dump). This drops the post-scale GetCorr and the two SetCorr re-uploads (~3x25MB/pass) plus the 6.3M host corr-adjust loop: scale-partials ~0.21->~0.10s and the smooth+combine region shrinks, taking RSM on lyso to ~0.91s (was ~1.47s with phase-1-only, ~1.71s full-CPU) - under the 1s target for this crystal; merge+stats (~0.49s) is now the dominant chunk. Bit-identical (GPU smooth-G == host SmoothG on the resident corr); validated across the battery (15/15 deterministic crystals bit-identical to CPU across default / CPU-combine / NO_GPU, only EP_cs_01-24 noise wobbles). Non-CUDA build unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 09:49:31 +02:00
leonarski_fandClaude Opus 4.8	87c70195f8	RotationScaleMerge: faster group-hkl via flat group ids + skip Obs stamp ... Profiling showed the per-space-group "group hkl" step (~0.30s/2-pass on lyso) is not gemmi-bound (the ASU keying is ~6ms) but memory-bandwidth-bound: stamping the group id onto, and reading it back from, the `group` field scattered across the 56-byte Obs struct touches the whole ~350MB partials array twice per pass. Precompute the per-obs AcceptReflection finiteness once (immutable) into a flat 1-byte array, then stamp the ASU-group id from rawrun_group + that flat array into a flat group_ids vector for the GPU, and build the group CSR (a stable counting sort, now parallel) from group_ids - all sequential/flat reads. The Obs.group field is written only when a CPU stage will read it (no GPU: scaling/CC/combine otherwise use group_ids / rawrun_group, never partials.group), so the default path skips the strided Obs pass entirely. group hkl ~0.31 -> ~0.20 s/2-pass on lyso. Output is bit-identical (group_ids values and the obs-index-ordered gperm are unchanged), so the merged results are unchanged; validated across the battery (15/15 deterministic crystals bit-identical to the CPU path, only EP_cs_01-24 noise keeps its benign wobble). Non-CUDA build unaffected (need_obs_group is always true there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 09:25:31 +02:00
leonarski_fandClaude Opus 4.8	7d804eb799	RotationScaleMerge: GPU post-smooth group means + per-frame CC ... The single-threaded ReduceGroupMeans over the 6.3M partials (~0.07s/2-pass) and the per-frame diagnostic CC now run on the resident partials on the GPU: after SmoothG, the smoothed corr is uploaded once (and left resident for the combine, dropping the combine's redundant re-upload), then the post-smooth group means (reusing the scaling reduce) and the per-frame Pearson CC (a new one-block-per-frame kernel) run there and only the tiny per-frame cc/cc_n come back. FinalizePerFrameScale is split into ComputePerFrameCC (host reference) + the writeback; the GPU path uses ComputePartialCC. The per-frame CC is diagnostic only (the per-image scaling table), so the tree reduction's ~ulp difference from the CPU is immaterial and it does not touch merged intensities. smooth+CC region ~0.10s GPU vs ~0.15s CPU on lyso. Validated across the battery: 15/15 deterministic crystals run-to-run deterministic and merged output bit-identical to the CPU path (only EP_cs_01-24, unindexable noise, keeps its benign error-model-b wobble). CPU fallbacks (JFJOCH_RSM_CPU_COMBINE / _NO_GPU) unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 08:58:26 +02:00
leonarski_fandClaude Opus 4.8	617184041f	RotationScaleMerge: enable GPU combine + scale-fulls by default ... Steps 1-2 (GPU 3D-combine + resident scale-fulls) are validated bit-parity and run-to-run deterministic against the CPU path across the rotation battery, and cut the combine+scale-fulls region from ~0.46s to ~0.32s on lyso, so make them the default when a GPU is present (consistent with phase-1 partial scaling already being default-on). JFJOCH_RSM_CPU_COMBINE forces the CPU combine/scale-fulls for A/B or debugging; JFJOCH_RSM_NO_GPU still disables the whole GPU path. The only battery crystal whose reported metrics move is EP_cs_01-24 (CC1/2 2%, unindexable noise) whose upstream integration is itself nondeterministic; its merged intensities/CC/completeness are unchanged, only the ill-conditioned error-model b. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 08:35:23 +02:00
leonarski_fandClaude Opus 4.8	ced85bcd9d	RotationScaleMerge: GPU scale-fulls, fulls kept resident (phase 2 step 2) ... Scale the combined fulls (Unity model) on the device so they no longer round-trip between the combine and the merge: after the GPU combine, build the fulls' per-frame and per-ASU-group CSRs on the host from just the small key arrays (f_frame/f_group) with a deterministic counting sort - no GPU stable-sort - then scale in place and download once. The four scaling kernels are reused unchanged except FitPerFrameGKernel, which gains an optional `perm` argument (null for the partials, whose arrays are already frame-contiguous; a frame-grouping permutation for the emit-ordered fulls) so the fulls are scaled without a physical reorder. The Unity model falls out of giving the fulls all-ones partiality/rlp/zeta (coeff = mean), so no other kernel changes and the committed phase-1 partial-scaling path is bit-identical (perm == null -> idx == i). Validated across the rotation battery (JFJOCH_RSM_GPU_COMBINE=1): all 15 deterministic crystals stay run-to-run deterministic and their merged output is bit-identical to the CPU path (SG/ISa/CC1.2/completeness). The lone exception is EP_cs_01-24 (CC1/2 2%, R_meas 379% - unindexable noise): merged intensities/CC/completeness match exactly, but the ill-conditioned 16-bin error-model b fit amplifies the ~1e-7 scale-fulls rounding to ISa 10.6 vs 10.8 - benign, same class as the accepted phase-1 GPU rounding. The 3 upstream-nondeterministic crystals vary as before (GPU-prediction overflow, not this). Scale-fulls drops from ~0.09s to ~0 across the two passes; combine+scale-fulls region ~0.32s GPU vs ~0.46s CPU on lyso. Still opt-in (fulls are downloaded for the host merge; the win grows once the merge/error-model also stay resident). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 07:51:29 +02:00
leonarski_fandClaude Opus 4.8	2c928b27cd	RotationScaleMerge: GPU 3D-combine (CUDA port, phase 2 step 1) ... Port Combine() (partials->fulls) to CUDA, mirroring process_rawrun bit-for-bit: one thread per raw-hkl run splits its usable partials into rocking events (frame gap <= 2), pools background, seeds F, runs 3 de-biased Poisson reweights and the capture-uncertainty term. Emission is deterministic - a count pass, a host exclusive prefix sum for per-run offsets, then an emit pass at those offsets - so fulls come out in raw-run-major/event order, identical to the CPU path; both pass instantiations share the same arithmetic so count == emit exactly. Dmax/Dmin/Fmax reproduce std::max/min NaN semantics (not fmax) for parity. Validated across the 18-crystal rotation battery: all 15 deterministic crystals (P1/P2/C2/H3/I23/P41212/P222/P422) match the CPU combine exactly on SG/ISa/CC1.2/ completeness and run-to-run (fulls count bit-identical); the 3 upstream-nondet crystals vary from GPU-prediction overflow, not the combine. Gated opt-in behind JFJOCH_RSM_GPU_COMBINE (default = CPU combine): combine alone is timing-neutral because the shared 1.2M SortFullsByFrame std::sort dominates and the fulls round-trip adds a copy - it only pays off once the fulls stay resident for scale-fulls + merge. Also add JFJOCH_RSM_NO_GPU master switch to force the CPU fallback (incl. phase-1 scaling) from one binary for A/B parity. SortFullsByFrame extracted from the Combine tail and shared by both paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 07:27:33 +02:00
leonarski_fandClaude Opus 4.8	dd25de461d	RotationScaleMerge: GPU partial-scaling loop (CUDA port, phase 1) ... First stage of moving the rotation scale/merge onto the GPU. The per-frame partial-scaling loop (inverse-variance group-mean reduction -> robust per-frame IRLS G -> corr update, x scaling_iter) now runs in RotationScaleMergeGPU (.cu) when a GPU is present; the CPU loops remain the fallback. The host keeps the one-time raw-hkl sort and the per-space-group gemmi ASU keying, and hands the GPU a group-ordered permutation + CSR so the per-group reduction is a DETERMINISTIC segmented reduction (one thread per group, fixed order, no atomics) - preserving the run-to-run determinism just won on the CPU path (a float atomicAdd reduction would have re-introduced jitter). Reduction is one-thread-per-group (groups average tens of obs, so a block-per-group wastes threads); the IRLS is one block per frame with a deterministic shared-memory reduction. Validated: bit-identical to the CPU path and deterministic run-to-run on lyso/cytC/Ins_H/pding (P41212 ISa 7.8 CC1/2 99.7%, etc.). The scaling kernels are ~7x faster than the CPU compute (~36 ms for 3 iters vs ~0.28 s); end-to-end scale/merge ~2.0 -> ~1.5 s. The remaining gap to the <1 s target is the per-pass host round-trip (corr down/upload for the CPU combine + per-SG group-CSR rebuild); phase 2 keeps the data resident by moving the 3D combine and the merge/error-model onto the GPU too, so nothing round-trips. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 22:26:29 +02:00
leonarski_fandClaude Opus 4.8	29c8ba6112	rotation: deterministic frame-order mosaicity smoothing + partiality recompute ... Prediction applied a mosaicity/profile-radius moving average (RotationParameters) over the last N *processed* frames. Under the parallel per-image loop that window is thread-arrival order, so the smoothed value - and hence which reflections are predicted/integrated - was non-deterministic run-to-run, swinging CC1/2 (and even the space group) on marginal crystals. `-N 1` was deterministic; `-N 32` was not. Fix (as designed with FL): prediction now uses each frame's OWN mosaicity/profile-radius (image-local, deterministic membership - a reflection on the cutoff contributes ~nothing). The smoothing that actually matters is moved into RotationScaleMerge and done in FRAME order (deterministic): per-frame mosaicity is smoothed with the same window as smooth-G, then every partial's partiality is recomputed from it BEFORE the 3D combine. This is the mosaicity analogue of smooth-G: combining a reflection's per-frame partials only tiles the rocking curve correctly (captured fractions summing toward 1) if neighbouring frames share a consistent mosaicity. Battery (18 crystals, /data/rotation_test, 2 runs each): 15/18 now bit-identical run-to-run (the good crystals unchanged - lyso P41212 ISa 7.8 CC1/2 99.7%). The 3 residual crystals (EcwtAL500, EcwtCQ066S, pding4_003 - all large/triclinic cells) still jitter ~0.002%, traced to a SEPARATE, benign cause: the GPU prediction buffer overflow (BraggPredictionRotGPU max_reflections=10000 with a racy atomicAdd/atomicSub) on dense frames - cell/space group stay stable; to be addressed in the GPU prediction/integration rework (naively raising the cap also changes prediction quality, so it is not a one-line bump). Minor label refinements from the recomputed partiality: cytC_2 P321 -> P3121 (now consistent with cytC_3), Ins_I_2/3 report the honest I23/I213 screw-axis ambiguity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 21:58:10 +02:00
leonarski_fandClaude Opus 4.8	b5d9167bf4	RotationScaleMerge: chunked parallelism -> scale/merge phase ~2s ... The per-observation corr update (7.6M items) ran through a work-stealing ParallelFor that does one atomic fetch_add PER item - pure contention for trivial work (measured: update 0.60s vs reduce 0.15s / fit 0.13s in the scale-partials loop). Add ParallelChunks (one contiguous range per worker, no per-item sync) and use it for UpdateCorr, and parallelise the ASU keying (gemmi reduction per distinct raw hkl - HKLKeyGenerator is const, safe to read concurrently) and the group-stamping over disjoint raw-hkl runs. scale-partials 0.90 -> 0.28s, group-hkl 0.20 -> 0.09s, per-pass warm 0.83s, whole scale/merge phase ~3.3 -> ~2.0s. Bit-identical output (same space group, ISa, CC1/2). ParallelChunks is the CPU stand-in for a flat CUDA grid-stride kernel; ParallelFor stays for the heavy, uneven per-frame fits where the atomic amortises and work-stealing balances the load. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 21:31:41 +02:00
leonarski_fandClaude Opus 4.8	29c5410328	rotation scaling: dedicated allocate-once RotationScaleMerge, ~3.4x faster scale/merge ... The rot3d post-pass (scale -> smooth-G -> 3D combine -> scale-fulls -> merge -> error model -> stats) dominated offline wall clock because ScaleOnTheFly + MergeAll + CombineRotationObservations + MergeOnTheFly rebuild a std::map keyed by hkl on every scaling iteration and every merge (7-14 map rebuilds per space-group pass), each re-walking and re-keying millions of per-frame partials. RotationScaleMerge ingests the per-frame partials ONCE into flat vectors and reuses them across both space-group passes: the raw-hkl ordering is sorted a single time at ingest, so the per-pass 3D combine only re-splits events (no sort) and the ASU grouping is one gemmi reduction per distinct raw hkl (~10x fewer) rather than per observation, reusing that order. Every hot step is a flat loop (segmented reduction + per-frame robust IRLS + parallel per-run combine) that also maps directly onto CUDA kernels. CC1/2 and the per-image CC are computed once at the end, not every iteration. It is a distinct path from ScaleOnTheFly, used only for the self-scaling rotation case (Rotation partiality + combine3d, per-image G, no B refinement, no external reference, no absorption surface, no wedge/mosaicity override). Stills, B-factor refinement, reference scaling and the absorption surface stay on the classic path. Numerically equivalent to the classic path (same robust per-frame G, same 3D combine, same XDS-order scale-fulls, same global error model, same merge statistics), validated on the 18-crystal /data/rotation_test battery: 16/18 bit-identical in space group / ISa / CC1/2; the 2 differing crystals are ice-heavy / marginal ones on which the classic path is equally non-deterministic run-to-run (a pre-existing upstream integration race). Scale/merge wall time drops ~3.4x (median 9.8 -> 4.1 s), making higher --scaling-iterations cheap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 21:00:12 +02:00
leonarski_fandClaude Opus 4.8	ddddfb6ffc	bragg_integration: GPU box + profile-fit integrator (standalone engine) ... Reimplement BraggIntegrate2D (box sum) and ProfileIntegrate2D (Kabsch profile fit) under one roof as a base + CPU + GPU engine, mirroring the AzIntEngine / ROIIntegration pattern. Reads the preprocessed int32 ImagePreprocessorBuffer (masked=INT32_MIN, saturated=INT32_MAX), the same buffer AzIntEngineGPU/ROIIntegrationGPU consume. The CUDA engine runs one block per reflection with shared-memory reductions across six kernels (reset, mask, box-sum, profile learning, profile build, Kabsch fit); the resolution shell is computed inline. The learning/fit hot path is single precision (FP64 is throttled on consumer GPUs; reproduces the double CPU path to ~1e-4). Collapsing the per-frame CUDA API calls into one reset kernel keeps launch-latency overhead low. Standalone for now: NOT wired into IndexAndRefine. See BRAGG_INTEGRATION_ENGINE.md for the design and the binding steps. BraggIntegrationEngineGPUTest checks GPU == CPU across all three modes (box/gaussian/empirical) within numeric tolerance, plus a [bragg_bench] perf sweep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 20:59:45 +02:00
leonarski_fandClaude Fable 5	f422988cd2	ice score: per-image ice-ring indicator plumbed through all layers ... A single per-image ice_ring_score - the strongest hexagonal-ice ring band/shoulder intensity ratio from the azimuthal profile (1 = no ice) - computed in the CPU and FPGA analysis paths and the offline azint worker, then plumbed through every layer: DataMessage/EndMessage, CBOR (frame_serialize), HDF5 (/entry/MX/iceRingScore), ScanResult, receiver plots (PlotType::IceRingScore), the OpenAPI spec (plot_type + scan_result schema, with regenerated broker/gen and frontend client) and OpenAPIConvert, the reader + Qt viewer, and the React frontend plot. Documented in docs/CBOR.md, docs/HDF5.md and docs/CPU_DATA_ANALYSIS.md, with the general "add a per-image quantity" recipe added to CLAUDE.md. Verified in HDF5: lysoC (weak ice) mean 1.23, EP_cs_01-17 (heavy ice) mean 1.67 / max 2.23. This is a monitoring quantity - it does not gate scaling (which already excludes all ice rings) or merging (handled by the CC1/2 ring mask). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:37:12 +02:00
leonarski_fandClaude Fable 5	6ffb16f142	merge: mask ice rings whose CC1/2 collapses below their resolution neighbours ... An ice-contaminated ring decorrelates its reflections, so its merged half-set CC1/2 falls well below the Bragg trend of its resolution shoulders (e.g. EP_cs_01-17 2.25A ring CC1/2 0.12 vs shoulders 0.89). After the final merge, compute per-hexagonal-ice-ring CC1/2 in the ring band (+/- ice width in q=2pi/d) against the shoulders either side, mask any ring more than 0.05 below its shoulders (reliable shoulder CC1/2 > 0.5, >=20 reflections each) and re-merge without them. Weak/absent rings track their neighbours and stay, so clean crystals are untouched. Data-driven and robust on /data/rotation_test: 16/18 crystals mask nothing and are unchanged (every clean crystal, plus icy-but-fine EP_cs_02-10); EP_cs_01-17 masks 6 rings, CC1/2 6.9% -> 28%; CQ066/pding4_003 mask one marginal ring each. A well-scaled CC1/2 test, replacing the fragile per-image azimuthal cutoffs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:06:39 +02:00
leonarski_fandClaude Fable 5	cdbd0bed5d	partiality: fix the rotation-prediction wedge, use the stored value in scaling ... BraggPredictionRot halves settings.wedge_deg to the +/- half-wedge of the partiality erf pair, but IndexAndRefine already passed GetWedge_deg()/2, so the two /2 compounded to a half-wedge of increment/4 - half the correct Kabsch value (increment/2, which ScaleOnTheFly's RotationPartiality already used). Pass the full increment so prediction's partiality matches scaling. With prediction correct, ScaleOnTheFly now uses the stored r.partiality directly (the value the reflection was integrated with) rather than recomputing the erf pair per reflection - recomputing only when scaling overrides the geometry (-w wedge refinement, --mosaicity, or a scaling wedge override). Output-neutral on the /data/rotation_test battery (SG/cell/completeness identical, ISa/CC1/2 within run noise on the stable crystals). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:57:16 +02:00
leonarski_fandClaude Fable 5	8c084021e6	ice rings: exclude from scaling, keep in the final merge (recover completeness) ... --detect-ice-rings previously dropped ice-ring reflections outright before scaling, which also removed them from the merge and gutted low/mid-resolution completeness on crystals that merge them fine (e.g. lysoC 80% -> the ice bands show as jagged shell completeness). Instead flag them (new Reflection.on_ice_ring, set in JFJochProcess, carried through the 3D combine) and exclude the flag only where a model is fit: the per-image scale (ScaleOnTheFly::Accept + the per-image CC), and - for the de-novo P1 pass - the space-group search and error model (MergeOnTheFly::ExcludeIceRings, on for for_search). The final in-symmetry merge and its statistics keep them, so completeness is preserved. /data/rotation_test battery vs the previous drop-from-merge behaviour: space group correct on all 18; completeness recovered broadly with CC1/2 and ISa held (cytC_2 82->99.7%, cytC_3 73->99.7%, InsI3 76->99.5%, lysoC 80->99.7%, MyoB 80->99.7%, InsH3 78->99%). Excluding ice from the P1 search merge is what keeps the space group correct: without it InsI3 flipped I23->P1 and EP_cs_01-17 P2->P1. Known limitation: on heavy-ice crystals (EP_cs_01-17) the strong ice is garbage and keeping it in the final merge collapses CC1/2 in the ice shells (91.7->6.9%). Distinguishing strong vs weak/absent rings per crystal needs data-driven, per-ring ice detection (azimuthal radial profile) - the planned next step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:29:37 +02:00
leonarski_fandClaude Fable 5	1ca5a69054	ScaleOnTheFly: solve the per-image scale by IRLS instead of Ceres ... In the default rotation (rot3d) path only G is refined - B is fixed, mosaicity is pinned and the wedge is not refined - so the predicted intensity G*coeff is linear in G and the robust (Cauchy) per-image scale is a 1-D M-estimate. Solve it directly by iteratively reweighted least squares (a few closed-form weighted ratios) instead of building a Ceres problem per image. Ceres is kept for the cases that are genuinely nonlinear: refining the B-factor or the rotation wedge. Same Cauchy objective as the Ceres path, but ~4x faster at scaling and ~30% faster overall on the /data/rotation_test battery, with space group, cell, ISa, completeness and CC1/2 matching across all 18 crystals (the two that look different, EP_cs_01-17 and EcwtAL500, are run-to-run unstable for both solvers). lyso_ref scaling 25.2->4.3s, cytC_2 15.2->2.6s, battery total 468->316s. Also drop the per-image G/B regularizers (gated by GetScalingRegularize, which nothing enables) and the now-unused RegularizationResidual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 13:55:04 +02:00
leonarski_fandClaude Opus 4.8	ed9f6ac9eb	integration: remove --reciprocal-profile (proven per-frame dead-end) ... An 11-crystal mosaicity-stratified re-test (/data/rotation_test, off vs on vs a de-contaminated variant, plus a per-frame dump of the fitted widths) showed the dial is net-negative and cannot work in the per-frame paradigm: - The C|q|^2 mosaicity term - the whole point - is unfittable per-frame: the fitted curvature a2 comes out ~0 (often negative) on every crystal, with zero correlation to the XDS mosaicity (0.09..0.42 deg). Strong spots sit at low q where eta^2 q^2 is invisible; the curvature only appears at high q where there are ~0 strong spots. The law degenerates to a straight line. - With a2~0 the high-res width becomes a blind 1/cos^2(2theta) extrapolation, 2-4x wider than per-shell. The per-shell path's high-res "starvation" (flat narrow fallback) is accidentally correct: weak, crowded high-res spots want a narrow aperture, not the true wide spot shape. - The over-wide profile pulls background into weak spots -> R-meas rises, CC1/2 drops in reliable high-multiplicity shells (pding4_001, pding4_003, MyoB, EcwtCQ066). A cap at the widest well-sampled per-shell width recovers the regression, confirming over-widening is the harm. No crystal reliably wins; the apparent overall-CC gains were all in noise shells (mult 2-3, CC<20%). Delete the CLI flag, the BraggIntegrationSettings::reciprocal_profile setting, and the per-frame fit block. Default (per-shell) integration is byte-identical. NEXTGEN_INTEGRATOR.md records the finding as a dead-end for posterity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 11:39:09 +02:00
leonarski_fandClaude Opus 4.8	00821e32ef	rotation indexer: refine candidate cells and pick the best after refinement ... De-novo two-pass indexing failed on large-cell / superstructure / modulated crystals (EcwtAL500 0%, EcwtCQ034 0%) and mis-handled a pseudo-symmetric one (EcwtCQ066 14%). The common cause: the choice of unit cell was made too early, on the raw pre-refinement spot fraction, which is an unreliable discriminator (a correct hexagonal cell indexes only ~13% of the un-refined accumulated spots, while a wrong larger cell can index more). Move the decision to after geometry refinement, where it is reliable: - FFTIndexer now OFFERS widened candidate cells instead of deciding. ReduceResults gains a `widen` flag: the standard path (9 shortest vectors) is unchanged; the widened path anchors the two short axes and lets the third range over all filtered vectors (+dedup) to reach the long axis of an elongated cell. FilterFFTResults takes the peak count as a parameter (30 standard, 60 widened). RunInternal appends widened candidates only when its standard best indexes poorly, so compact crystals are untouched. - RotationIndexer fully refines the top few candidates and keeps the one that indexes the most spots under its own refined geometry (IndexedFraction). Each refine is length-bounded (1.2x the found cell) so a free triclinic refine cannot drift onto a pseudo-translation / modulation supercell (CQ034's satellites). The earlier (primary) candidate is preferred: a later one is adopted only if it indexes clearly more and reasonably well in absolute terms, so a twin's noisy near-tie cannot displace it. Extra/twin lattices are only searched when the chosen cell is the FFT primary (lattice[0]), since MultiLatticeSearch's rotations are derived from that primary. - The pseudo-symmetry guard (de-novo only - a user-fixed space group is always honored) is a ratio of refined fractions: refine the primitive as triclinic and drop to P only if the constrained cell indexes less than half of it. A false promotion indexes badly under its constraint (CQ066 ratio ~0.1) while genuine higher symmetry, including R-centred, indexes comparably (Ins_H R3 ratio ~0.7) and is kept. Validated on the full /data/rotation_test battery: AL500 0->89% (C2), CQ034 0->99%, CQ066 14->93% (ISa 7.2->13.7); the other 15 crystals keep their exact cell, space group, indexing rate and ISa (no regressions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:17:53 +02:00
leonarski_fandClaude Opus 4.8	35607057d9	CC1/2 merge: assign half-sets by deterministic per-image hash ... MergeOnTheFly::AddImage picked each image's CC1/2 half from a shared mt19937 drawn in call order (and before Mask), so the split depended on iteration/thread order and on how many images were masked. The class is mutex-guarded for concurrent "on-the-fly" use, so any parallel merge would make CC1/2 non-reproducible - a latent race. Assign the half as a splitmix64 hash of the image's stable index instead, computed after Mask. The split is now reproducible run-to-run, independent of AddImage order, parallel-safe, and decoupled from masking. Callers pass the outcome's vector index as the image id. Verified: lyso_ref two-pass -M -P rot3d gives identical CC1/2 across runs (overall 99.6%, P41212); hash split is balanced ~50/50. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 07:25:34 +02:00
leonarski_fandClaude Opus 4.8	a773aaa6a9	WriteReflections: write merging statistics + ISa to the mmCIF output ... The .cif (--scaling-output cif) now carries the per-shell and overall merging statistics in the standard _reflns / _reflns_shell categories (resolution, redundancy, completeness, <I/sigma>, R-rim/R-meas, CC1/2) plus the Diederichs asymptotic I/sigma (ISa) as a free-text _reflns.pdbx_diffrn_ISa item (no standard CIF tag exists). The MergeStatistics and the ISa string are threaded through WriteReflections to the mmCIF writer; jfjoch_process and jfjoch_scale pass them. Values match the text statistics table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 22:29:45 +02:00
leonarski_fandClaude Opus 4.8	636ed1e180	space group: use merge reduced chi^2 (not R-meas) for the consistency check ... The R-meas-ratio over-symmetry guard (commit 7ade6d9) missed a near-perfect pseudo-2-fold: InsH2 (XDS R3) has a false 2-fold at operator CC 0.85 whose merged intensities still correlate well, so R-meas inflated only ~1.3x and R32 was accepted (ISa collapsing to 3.9). The reduced chi^2 (within-orbit scatter / sigma^2) is the right signal - false equivalents disagree by many sigma even when they correlate well. Measured chi^2 ratio (candidate / best subgroup) across the test set: true point groups <= 1.63 (a cubic merge of a noisy crystal, InsI3, is the worst real case), false >= 2.26 (InsH2) and 6.0 (InsH3); threshold 2.0 separates them. InsH2 now resolves to R3 (ISa 6.1 from the false R32's 3.9); InsH3/InsI2/InsI3/cytC/lysoC/MyoB/EP0210 unchanged and correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:54:51 +02:00
leonarski_fandClaude Opus 4.8	fea2790a41	rotation_indexer: detect hexagonal metric de-novo (fixes trigonal cytC indexing) ... The hexagonal lattice metric (two equal axes at 60/120 deg, both perpendicular to the third) is also satisfied by its ortho-hexagonal C-centred supercell, so the geometry-keyed LatticeSearch lands a metrically-hexagonal lattice on the C-orthorhombic setting. The -S path already re-expresses it (HexagonalConventional) keyed on the supplied space group; do the same de-novo, keyed on the metric of the reduced PRIMITIVE cell (rhombohedral lattices have a rhombohedral primitive cell and are unaffected). cytC (P3121) now indexes de-novo as hexagonal 83.8/83.8/88.6 gamma=120 and merges in P3121 (CC1/2 99.7%, ISa 13.1), for both test crystals, instead of the orthohexagonal C2/P1. lysoC/InsI3/InsH3 unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:31:20 +02:00
leonarski_fandClaude Opus 4.8	52248b8f5c	docs: design for auto-gated, data-driven ice-ring detection ... Records the validated z-score detector (on the azint radial background profile) and position/width estimation, for a future --detect-ice-rings=auto. Not yet implemented. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:22:31 +02:00
leonarski_fandClaude Opus 4.8	73d9e1de39	ice rings: widen the exclusion band to 0.03 q (measured ring FWHM) ... The hexagonal-ice ring FWHM measured on JUNGFRAU data (azimuthal radial background profile) is ~0.06 q, so the exclusion half-width should be ~0.03; the previous 0.02 under-covered the strong low-res rings. On EP0117 with --detect-ice-rings this lifts CC1/2 37->50%, and combined with --reject-outliers 3 (which down-weights the radiation-damaged late frames) reaches ~94% (XDS 98.5%). Only active with --detect-ice-rings, so default behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:09:18 +02:00
leonarski_fandClaude Opus 4.8	7ade6d9906	space group: reject over-symmetrised point groups by merge R-meas consistency ... Per-operator CC alone cannot separate a real weak symmetry operator from a false moderate one: a noisy crystal's genuine cubic 2-folds (InsI2 I23, CC ~0.51) score BELOW a pseudo-symmetric crystal's false 2-fold (InsH3, CC 0.64), so no CC threshold works. Add a self-consistency test: a candidate point group is accepted only if merging the intensities under it does not inflate R-meas past max_rmeas_ratio (1.5x) the most-consistent candidate. A false operator forces non-equivalent reflections together and blows R-meas up; a real one leaves it flat. Fixes InsH3: was over-called R32 (ISa collapsed to 2.2 from the forced merge), now correctly R3/H3 (ISa 10.2, matching XDS). InsI2 stays I23, and lysoC/InsI3/MyoB/ EP0210/cytC/lyso_ref are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:32:03 +02:00
leonarski_fandClaude Opus 4.8	23d58b6202	integration: robust, well-conditioned per-frame --reciprocal-profile fit ... Fit the reciprocal tangential-width model y(q)=a0+a1*t+a2*t^2 in a centered, standardized variable t=(q-qbar)/qscale instead of the raw {1,q,q^2} monomials: the raw normal matrix went near-singular when the strong spots span a narrow q-range (small cell / sparse still), letting tiny per-frame jitter swing the curvature into a wild over-wide profile. Adds IRLS (Huber) robustness, a ridge on the curvature (sharp-crystal prior), and clamps the applied width to the fitted q-range (no extrapolation). Stays strictly per-frame (no dataset pooling), so it works online and for stills. Neutral on rotation data (cytC high-res CC1/2 win preserved 66.8 vs 65.6%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:01:05 +02:00
leonarski_fandClaude Opus 4.8	8d18c635ff	scaling: robust (Cauchy) loss on the per-image scale fit ... The per-image G/B/mosaicity fit used a plain L2 loss, so a few outlier reflections occasionally dragged a frame into a bad optimum - a stochastic (~15% of runs) per-frame mis-scaling that elevated R-meas and collapsed CC1/2 at low symmetry (the image-level CC1/2 half-split makes the damage look patchy across shells, while the data is genuinely noisier). A Cauchy loss (3 sigma) soft-downweights those outliers without a hard cut: MyoB 0/10 catastrophic (was ~2/10), R-meas stable, and ISa improves on every test crystal (EP0210 9.2->12.4, MyoB 12.5->14.6, lysoC 10.4->11.2, cytC 11.5->12.6), most on low symmetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:00:42 +02:00
leonarski_fandClaude Opus 4.8	21dd99524f	rotation_indexer: map FFT primitive cell to the SG conventional setting ... When a space group is supplied without a reference cell, de-novo two-pass rotation indexing fed the FFT's Niggli-reduced primitive cell straight into XtalOptimizer as if it were the conventional cell. For non-primitive lattices (centered I/F/R/C, or hexagonal where the primitive pair sits at gamma=60) the conventional-system model then refined to a wrong minimum and indexed 0% of frames: cytC (P3121) gave 103.9/103.9/78 instead of 83.7/83.7/88.6, insulin (I213) 66.7 instead of 77.65, insulin-R3 51/51/36 instead of 81.4/81.4/33.3. Run LatticeSearch on the FFT primitive cell (it already yields the correct conventional cell + reindex for I/F/R/C). For the one remaining gap - a metrically hexagonal lattice that the geometry-keyed search lands on the ortho-hexagonal C setting - re-express the reduced primitive cell in conventional hexagonal axes (b -> b - a opens gamma 60 -> 120). De-novo "-S" now indexes cytC/insu/Ins_H/lyso/MyoB/EP/lyso_ref at 100% with the correct cell; the "-C -S" path is unchanged. The helper stays in this .cpp (g++) rather than CrystalLattice.h to avoid recompiling CUDA units, which is broken under the box's CUDA-13 nvcc; promote it to a method once that is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:29:50 +02:00
leonarski_fandClaude Opus 4.8	3aec235f67	integration: add --reciprocal-profile dial (global reciprocal-space width) ... The per-shell profile width is learned in pixels, so it varies ~4x with resolution (mostly the geometric projection of a near-constant reciprocal-space relrod) and must be binned per shell -> it starves at high resolution / on sparse data. The new --reciprocal-profile flag instead learns ONE global width in reciprocal space, sigma2_q,tan = A + B|q| + C|q|^2: the Jacobian g_tan=cos(2theta) removes the geometric projection, and C|q|^2 is the crystal mosaicity relrod (variance ~(eta|q|)^2). Applied per reflection as sigma2_tan,px = (A + B|q| + C|q|^2)/g_tan^2 (B,C clamped >=0; quadratic->linear->constant fallback). Off by default. On the sharp HEWL test crystal (mosaicity 0.091deg, so C fits to ~0 and it reduces to the validated linear form) it is metric-neutral: ISa 16.2->16.3, anomalous 0.92x unchanged, CCref band 90.0->89.9, CC1/2 a touch lower (per-shell isn't starved at 23k spots/shell, and a global fit is less flexible). So: simpler + more transferable at a small CC1/2 cost, ISa/anomalous held. Its payoff is on MOSAIC crystals (large C|q|^2), where per-shell starves on the wide weak high-res spots and 6 shells are too coarse; both lyso test crystals are sharp, so it ships as a dial to try on mosaic data elsewhere. A separate radial relrod fit was tried and dropped (no gain). See NEXTGEN_INTEGRATOR.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 14:26:08 +02:00
leonarski_fandClaude Opus 4.8	5ba5fe8ad1	integration: centre the profile on the predicted sub-pixel position ... The profile was learned and applied on the integer pixel round(predicted), so a shared profile sits up to 0.5 px off the true spot (and stacking spots with random sub-pixel offsets broadens the learned profile). Build the Gaussian per reflection instead, centred on the predicted sub-pixel offset -- noise-free geometry, unlike the observed centroid, which hurt -- and elongated radially as before. HEWL rotation @1.0A: ISa 15.7->16.2, CCref band 89.9->90.0, CCxds 94.8->95.0 (high-res 1.00A CCref 66.0->66.9); sharp serial stills 1.68A CC1/2 61.6->62.5; anomalous S peak 0.92x XDS (no accuracy traded). De-broadening the learned width by the 1/12 px^2/axis integer-binning floor was tested and rejected (it over-narrows). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 07:34:06 +02:00
leonarski_fandClaude Opus 4.8	26901da42f	integration: de-contaminate the profile width, add a radial parallax ellipse ... The profile-fit width came from the full 13x13 second moment, which runs 3-8x wider than the true spot: neighbour reflections leak into the (unmasked) learning grid -- catastrophic at low res where spots crowd the beam -- and the far corners (lever arm dx^2+dy^2 up to 72) add rectified background noise. Splitting the spot moment into radial vs tangential shows the tangential width is isotropic (mosaicity/divergence) while the radial excess is pure sensor parallax ~tan^2(2th). - Measure the width over the signal disk r1 on the monochromatic path (inherently excludes neighbours, caps the radial tail); keep the generous full-grid width on the broadband/stills path (sparse spots, the centroid-undersampling floor). - Extend the (was bandwidth-only) radial ellipse with an analytic, material-aware parallax term c_par*tan^2(2theta), c_par = Var(z)/pixel^2 from sensor thickness + material + energy (parallax_var_px2; Si and CdTe), plus a fixed weak-spot capture term on the monochromatic path only. HEWL rotation @1.0A: ISa 13.5->15.7, CC1/2 1.12A 91.3->95.9 / 1.05A 83.0->85.2, external CCref band 88.1->89.9, CCxds 93.4->94.8, R-meas 9.4->8.7; low/mid flat. Sharp serial stills gain slightly from parallax; broadband stills neutral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 00:54:59 +02:00
leonarski_fandClaude Opus 4.8	7a4fadf29b	bragg_prediction: reciprocal Lorentz divides by |S||S0|, not S.S0 ... S.S0 = |S||S0| cos(2theta), so dividing the rocking rate |m2 . (S x S0)| by it added a spurious 1/cos(2theta) factor (1.0 at low res, 1.8x at 1 A) to the reciprocal Lorentz, hence to the absolute/Wilson scale. Divide by |S||S0| to get the correct zeta*sin(2theta) (times a constant 1/lambda^2 the overall scale absorbs). The spurious factor depends on 2theta only, so it cancels within a resolution shell and between symmetry mates -- CC1/2, CCref and R-meas are metric-neutral -- but it corrupts the absolute scale and destabilises the per-image B-factor and cross-resolution error model. CPU and GPU kernels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	40f650b9e7	scale_merge: pool the per-frame background across each rocking event in Combine3D ... Each partial subtracted its own independently-estimated per-frame background, so a weak full assembled from N frames accumulated one background-estimation variance per frame. The true background is flat over the few frames of one rocking event, so replace each partial's background by the event mean and correct its intensity by n_bkg*(bkg - pooled), where n_bkg = (sigma^2 - I)/bkg is the effective background-pixel count -- correct for weak AND strong reflections (sigma^2/bkg would over-count strong ones and over-correct them). Single-frame events are a no-op. Measured (rotation lyso @1.0 A): 1.05 A CC1/2 81.3 -> 83.0, R-meas 81.4 -> 77.2, CCref +0.2, CCxds +0.3; overall R-meas 10.0 -> 9.4%; ISa preserved (13.5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:37 +02:00
leonarski_fandClaude Opus 4.8	eccd10a0a7	integration: neighbour-mask the profile-fit background ring, widen r3 8->10 ... ProfileIntegrate2D::BoxSum now excludes every predicted reflection's r2 disk from the r2..r3 background annulus (mirroring BraggIntegrate2D), so a neighbour Bragg peak can no longer bias a reflection's background high and over-subtract. With neighbours excluded the annulus can safely widen, so the default r_3 goes 8 -> 10 (more background pixels, lower-variance estimate). Measured (rotation lyso @1.0 A, external CCref/CCxds vs XDS): 1.05 A CCref +1.3 / CCxds +1.3, R-meas -5 pts, low-res R-meas unchanged. Serial (Jet8 @0.0002 bandwidth): 1.68 A CC1/2 +3.9 / CCref +1.1, 1.58 A +4.0 / +1.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:14 +02:00
leonarski_fandClaude Opus 4.8	bbd888dcc3	integration: gate the background-ring sigma-clip to stills (bandwidth>0) ... The 76e88b0f sigma-clip (reject ring pixels above mean+3*sqrt(mean)) de-biases bandwidth-streaked high-resolution stills, but it ran on all data. On rotation (no streaks) it clips legitimate high background pixels and biases the mean low, slightly inflating weak intensities and hurting the anomalous signal. Gate it to bandwidth>0 (stills), matching how the 814dff34 radial-profile change is already gated. Rotation lysozyme (self-scaled, smooth-g, -A): anomalous S-peak 0.84x -> 0.86x of XDS (SD_MET 11.71 -> 11.94, CL_CL 1.28x -> 1.25x), ISa unchanged. Stills (bandwidth set) are byte-identical (clip still applies). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	99b923e727	scaling: smooth per-frame scale G across frames before the rot3d combine ... ScaleOnTheFly fits each frame's scale G independently with no neighbour coupling, so the few partials of one rocking event are weight-summed in combine3D on inconsistent scales - jitter that never enters the full's counting sigma and instead surfaces as scatter between symmetry mates, inflating the error-model b (low ISa). A centered moving average of log(G) over a small frame window (default 9, on for rot3d) removes it, mirroring XDS's smooth scaling. Complementary to --scale-fulls (which rescales between fulls, after the combine): smoothing fixes within-event scale, scale-fulls fixes between-full. On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa 11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest). Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS. --smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a diagnostic that fixes the scaling mosaicity for sweeps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	02477f1ce4	space group: determine centering from absences; jfjoch_process re-indexes from scratch ... Fixes wrong space-group assignment on cubic insulin (true I2_13, a=77.6). SearchSpaceGroup: drop the lattice-centering gate (and the now-unused lattice_centering option). The indexer often returns the conventional cell, whose geometry hides I/F/C centering - that information lives only in the systematic absences. Stage B now tests every centering of the point group and confirms it from the data, so an indexer-reported 'P' no longer excludes I2_13/I23. (I23 and I2_13 remain indistinguishable by absences and are reported as such.) jfjoch_process: discard any unit cell stored in the input HDF5 by default so the cell is re-determined from scratch. A stale/wrong stored cell otherwise resolves the indexing algorithm to FFBIDX, which trusts that cell and locks onto the wrong lattice (Ins_I_3 went from 2.7% -> 76% indexing). A user-supplied -C cell still applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 16:39:03 +02:00
leonarski_fandClaude Opus 4.8	107bcae0f0	merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2 ... RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned *median* of the squared symmetry-mate deviations, chosen for robustness. But a single deviation squared over its variance is chi-square(1)-distributed, whose median is only 0.4549 of its mean, so the fit was calibrating the variances to 0.4549x their true value: the merged sigmas came out ~1.48x too small and the achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally well-behaved (flat chi^2 across resolution) but globally over-confident, which inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for downstream weighting / French-Wilson. Divide the binned median by the chi-square(1) median (0.4549) to recover an unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the median while targeting reduced chi^2 = 1. Also compute the achieved median reduced chi^2 (same normalization) and report it on the "Error model" line so mis-calibration can no longer drift silently. Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2 2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common constant); only the absolute sigma calibration is corrected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 15:57:42 +02:00
leonarski_fandClaude Opus 4.8	814dff34cb	integration: radially-elongated profile for bandwidth-streaked high-res stills ... With a finite energy bandwidth each reflection is smeared RADIALLY by sigma_bw = bandwidth_sigma * R_px (R_px = distance from the beam centre, so large at high resolution): high-resolution spots become radial streaks. The isotropic per-shell Gaussian both mis-weights them and clips the streak tail on the fixed profile grid, losing intensity (biased low, noisy). When a bandwidth is set, fit each reflection with a per-reflection Gaussian elongated only along its radial direction - sigma^2_radial = sigma^2_intrinsic + sigma_bw^2, sigma^2_tangential = sigma^2_intrinsic - on a grid grown to hold the streak. Unlike an isotropic widening this adds no tangential background. It only engages where the smear exceeds the intrinsic spot (high resolution); low/mid resolution and monochromatic data (bandwidth 0, e.g. rotation) are untouched. On the HEWL serial-stills jet (with the background sigma-clip) this lifts the overall CC-vs-reference 52 -> 55% and the high-resolution I/sig (1.7 A 0.5 -> 1.4), recovering the 2.0-2.5 A band, with CC1/2 preserved (the per-shot noise the wider region adds is averaged out by the high serial multiplicity). Rotation ISa 19.1 unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 12:01:56 +02:00
leonarski_fandClaude Opus 4.8	76e88b0fca	integration: sigma-clip the background ring to de-bias high-resolution stills ... Weak high-resolution still reflections were systematically over-subtracted: a bandwidth-streaked high-res spot (or a neighbour) leaks into the r2-r3 background annulus and biases its mean high, so the subtracted background is too large and the merged high-resolution intensities go negative (seen as reproducibly negative <I/sig> at 100% completeness and high multiplicity past ~1.9 A). Add one high-outlier sigma-clip pass to the box-sum background (reject ring pixels above mean + 3*sqrt(mean), recompute) so the contamination no longer inflates it. A clean Poisson background is essentially unchanged (~0.1% exceed the cut). On the HEWL serial-stills jet this de-biases the high-res band - <I/sig> 2.03 A 0.9 -> 1.6, 1.79 A -0.1 -> +0.7 - extends the usable resolution ~2.2 -> ~2.0 A and improves overall R-meas 130 -> 124%, with CC1/2 and CC-vs-reference neutral. The rotation crystal is unchanged (ISa 19.1), its clean backgrounds being barely clipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 11:50:31 +02:00
leonarski_fandClaude Opus 4.8	7b464e4b3c	indexing: deconvolve energy bandwidth from the profile radius ... The profile radius (intrinsic excitation-error width = mosaicity + divergence) was the plain RMS of dist_ewald over indexed spots. With a finite energy bandwidth that spread is broadened by the bandwidth's radial smear sigma_bw = bandwidth_sigma*lambda/(2 d^2), which prediction then re-applies per reflection - so bandwidth was counted twice and the radius was inflated (most at high resolution, sigma_bw ~ 1/d^2). Subtract the bandwidth variance from the measured spread so the radius is the intrinsic width. bandwidth = 0 (monochromatic / rotation) is unchanged. Small for narrow bandwidths (~6% of the variance, ~4% radius on the 1% jet); matters for wide-bandwidth / pink beam. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 10:56:49 +02:00
leonarski_fandClaude Opus 4.8	82b8a9752c	rotation: fix mosaicity underestimate + default-on capture-uncertainty (ISa 10.7->19.1) ... The rotation per-image mosaicity was ~3x too small (0.045 vs the true 0.13deg), which crippled the partiality model and capped per-observation precision: it predicted reflections on too few frames and over-peaked the rocking partiality, so the rot3d-combined fulls were ~1.7x noisier than XDS's, the integration bottleneck behind the jungfraujoch-vs-XDS ISa gap. Two root causes, both fixed: - CalcMosaicityXDS (Kabsch-2010 MLE) searched each spot's exact-Bragg phi only within +-wedge (the 0.2deg oscillation). Reflections recorded at larger rocking offset - the tail that defines the mosaic width - fell outside and were dropped, truncating the tau distribution so the MLE underestimated ~2x. Widen the search window to wedge+0.8deg; the MLE then converges to the true 0.13deg (and is insensitive to widening further, since it weights by the recorded fraction). - ScaleOnTheFly then re-refined the mosaicity from the intensity residual, which is degenerate with the per-image scale G and collapses it toward its floor. Trust the (now correct) indexing mosaicity and keep it fixed during scaling. With the correct mosaicity, --capture-uncertainty (which down-weights the over-extrapolated under-captured fulls) now pays off strongly, so default it ON (1.0) for the rot3d combine; it stays off for non-rot3d. Together on the HEWL rotation crystal: ISa 10.7 -> 19.1, and anomalous peak height vs XDS goes from 52% to ~78% (CL_CL 1.92x -> 1.29x). This reaches XDS's own published-correction ceiling (DECAY+ABSORP+MODPIX ~= 19.6); the remaining gap to its quoted ISa 28 is the I->inf extrapolation. No effect on the stills path (rotation-only code). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 23:57:04 +02:00
leonarski_fandClaude Opus 4.8	ed98775ba3	rot3d: capture-aware sigma for under-captured fulls + expose per-image scale ... The rot3d combine over-extrapolates fulls reconstructed from only a fraction f of their rocking curve (min_partiality admits f as low as 0.02). Against XDS on HEWL these low-capture fulls are systematically biased HIGH (+15% at f=0.8 to +100% at f=0.3), and the bias - not random scatter - is the strong-reflection floor that hurts anomalous accuracy. --capture-uncertainty <coeff> (default 0 = off, baseline bit-identical) adds a systematic uncertainty ~coeff*(1-f)*I to each full's sigma, so the merge down-weights the over-extrapolated fulls and the error model treats their scatter as expected. Unlike outlier rejection (which trades accuracy for CC1/2), this fixes a real bias, so accuracy improves: at coeff=1.0 the anomalous peak height vs XDS rises CL_CL +16%, SD_MET/SG_CYS +5-6%, ISa 10.7->11.0. Rotation- only (no-op for stills, which never combine). Also expose the per-image scale offline: Combine3D now carries the first-pass per-image scale metadata (G, B, mosaicity, wedge, CC) forward instead of dropping it, and jfjoch_process -M writes <prefix>_image.dat from it (the offline self-scaling result was otherwise unobservable - process.h5's per-image arrays are only filled on the online path). This enabled the XDS DECAY comparison (jfjoch G tracks XDS, r=0.93). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 19:41:20 +02:00
leonarski_fandClaude Opus 4.8	786af96b3b	SearchSpaceGroup: POINTLESS-style rewrite, pipeline integration, twinning test ... Space-group search (image_analysis/scale_merge/SearchSpaceGroup): - Two-stage POINTLESS-style determination. Stage A scores each distinct rotation operator once (was once per candidate space group, ~34x faster on lysozyme: ~26s -> <1s) and picks the largest point group all of whose operators confirm. Stage B picks the maximal space group whose predicted absences are confirmed weak, fixing the prototype's default to the symmorphic group (it returned P422 instead of P4(3)2(1)2). Enantiomorphic / origin-ambiguous pairs (P4(1) vs P4(3), I222 vs I2(1)2(1)2(1)) are reported as indistinguishable. - Constrain candidates to subgroups of the lattice (metric) holohedry and weigh centering only P-vs-metric, fed from rotation indexing's LatticeSearch result. Integration / pipeline: - With no user-fixed space group, predict in P (IndexAndRefine) so the centering-absent reflections are integrated and the search can confirm/deny centering (catching pseudo-centering / a missed superstructure) instead of trusting the metric; a user-fixed group still rejects absences in integration. - JFJochProcess: scale+merge in P1 -> determine the space group -> set it and re-scale+merge in it (statistics then come out in the right symmetry) -> write it to /entry/sample/space_group_number (new EndMessage.space_group_number, preferred by NXmx::Sample). jfjoch_scale no longer searches; it consumes the file's space group (and no longer clobbers it with an empty -S). Twinning (new image_analysis/scale_merge/TwinningAnalysis): Padilla-Yeates L-test (<|L|>, <L^2>; acentric-only, positive intensities so L is bounded) plus a shell-normalised <I^2>/<I>^2 second moment and a twin-fraction estimate. Reported after the final merge in jfjoch_process and jfjoch_scale, and surfaced in the jfjoch_viewer merge-statistics window with a red outline when twinning is suspected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 16:11:28 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60	2026-06-15 20:24:15 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00