1179 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	ddddfb6ffc	bragg_integration: GPU box + profile-fit integrator (standalone engine) ... Reimplement BraggIntegrate2D (box sum) and ProfileIntegrate2D (Kabsch profile fit) under one roof as a base + CPU + GPU engine, mirroring the AzIntEngine / ROIIntegration pattern. Reads the preprocessed int32 ImagePreprocessorBuffer (masked=INT32_MIN, saturated=INT32_MAX), the same buffer AzIntEngineGPU/ROIIntegrationGPU consume. The CUDA engine runs one block per reflection with shared-memory reductions across six kernels (reset, mask, box-sum, profile learning, profile build, Kabsch fit); the resolution shell is computed inline. The learning/fit hot path is single precision (FP64 is throttled on consumer GPUs; reproduces the double CPU path to ~1e-4). Collapsing the per-frame CUDA API calls into one reset kernel keeps launch-latency overhead low. Standalone for now: NOT wired into IndexAndRefine. See BRAGG_INTEGRATION_ENGINE.md for the design and the binding steps. BraggIntegrationEngineGPUTest checks GPU == CPU across all three modes (box/gaussian/empirical) within numeric tolerance, plus a [bragg_bench] perf sweep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 20:59:45 +02:00
leonarski_fandClaude Fable 5	347228d008	jfjoch_process: azimuthal-integration CLI + default 0.01 1/A q-spacing ... Add -q/--azim-q-spacing, --azim-min-q, --azim-max-q, --azim-phi-bins (mirroring jfjoch_azint) so offline processing can set the radial binning, applied before the azint mapping is built. Set the AzimuthalIntegrationSettings default spacing to 0.01 1/A (was 0.05): the coarse default barely resolved the narrow ice rings, diluting the ice-ring score. Finer binning sharpens it a lot with no effect on processing - EP_cs_01-17 ice score 4.6->7.3 (max 11->23), clean cytC stays ~1.0, and space group / cell / ISa / completeness are unchanged (cytC, InsI3, MyoB, pding4_001 verified full-image). Documented in JFJOCH_PROCESS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 17:24:05 +02:00
leonarski_fandClaude Fable 5	5737d2e1d0	ice score: measure against a smooth whole-profile background, not shoulders ... The score's baseline was two adjacent shoulder bins with a bin-overlap bug - the ring's edge bins were counted in both the ring and the shoulder, since GetMeanValueOfBins is inclusive. At the typical (coarse) azint binning (dq ~ 0.05 in q, wider than the 0.03 ring half-width) a shoulder is only ~1 bin, so the ratio was noisy and poorly separated. Replace it with the ring intensity over a smooth whole-profile background: a running median of the non-ice bins, interpolated under each ring. Clean crystals now sit at ~1.0 and ice separates far more cleanly on /data/rotation_test: cytC 1.06->1.03, lysoC 1.23->2.77, EP_cs_01-17 1.67->4.51 (max 11.4). A z-score / abnormality probability was tried but is uninformative here - with many photons any real ice ring is highly significant, so the useful discriminator is the ice magnitude (this ratio), noted in CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 17:05:39 +02:00
leonarski_fandClaude Fable 5	65c7eb6366	docs: complete the per-image-quantity recipe with the HDF5 read-back path ... Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:43:02 +02:00
leonarski_fandClaude Fable 5	0399d1799d	ice score: read back from stored HDF5 (viewer) + write to the NXmx master ... The per-image ice_ring_score was written to the data files (HDF5DataFilePluginMX) but the master-file write and the stored-file read path were missing, so opening a processed HDF5 (e.g. in jfjoch_viewer) would not surface it. Mirror bkg_estimate: write /entry/MX/iceRingScore in the NXmx master (HDF5NXmx, from EndMessage), and read it back in HDF5MetadataSource into the dataset (master + data-file reads) and per image into the message. Verified the write; the read is a byte-for-byte mirror of the working bkgEstimate round-trip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:42:17 +02:00
leonarski_fandClaude Fable 5	f422988cd2	ice score: per-image ice-ring indicator plumbed through all layers ... A single per-image ice_ring_score - the strongest hexagonal-ice ring band/shoulder intensity ratio from the azimuthal profile (1 = no ice) - computed in the CPU and FPGA analysis paths and the offline azint worker, then plumbed through every layer: DataMessage/EndMessage, CBOR (frame_serialize), HDF5 (/entry/MX/iceRingScore), ScanResult, receiver plots (PlotType::IceRingScore), the OpenAPI spec (plot_type + scan_result schema, with regenerated broker/gen and frontend client) and OpenAPIConvert, the reader + Qt viewer, and the React frontend plot. Documented in docs/CBOR.md, docs/HDF5.md and docs/CPU_DATA_ANALYSIS.md, with the general "add a per-image quantity" recipe added to CLAUDE.md. Verified in HDF5: lysoC (weak ice) mean 1.23, EP_cs_01-17 (heavy ice) mean 1.67 / max 2.23. This is a monitoring quantity - it does not gate scaling (which already excludes all ice rings) or merging (handled by the CC1/2 ring mask). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:37:12 +02:00
leonarski_fandClaude Fable 5	6ffb16f142	merge: mask ice rings whose CC1/2 collapses below their resolution neighbours ... An ice-contaminated ring decorrelates its reflections, so its merged half-set CC1/2 falls well below the Bragg trend of its resolution shoulders (e.g. EP_cs_01-17 2.25A ring CC1/2 0.12 vs shoulders 0.89). After the final merge, compute per-hexagonal-ice-ring CC1/2 in the ring band (+/- ice width in q=2pi/d) against the shoulders either side, mask any ring more than 0.05 below its shoulders (reliable shoulder CC1/2 > 0.5, >=20 reflections each) and re-merge without them. Weak/absent rings track their neighbours and stay, so clean crystals are untouched. Data-driven and robust on /data/rotation_test: 16/18 crystals mask nothing and are unchanged (every clean crystal, plus icy-but-fine EP_cs_02-10); EP_cs_01-17 masks 6 rings, CC1/2 6.9% -> 28%; CQ066/pding4_003 mask one marginal ring each. A well-scaled CC1/2 test, replacing the fragile per-image azimuthal cutoffs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 16:06:39 +02:00
leonarski_fandClaude Fable 5	cdbd0bed5d	partiality: fix the rotation-prediction wedge, use the stored value in scaling ... BraggPredictionRot halves settings.wedge_deg to the +/- half-wedge of the partiality erf pair, but IndexAndRefine already passed GetWedge_deg()/2, so the two /2 compounded to a half-wedge of increment/4 - half the correct Kabsch value (increment/2, which ScaleOnTheFly's RotationPartiality already used). Pass the full increment so prediction's partiality matches scaling. With prediction correct, ScaleOnTheFly now uses the stored r.partiality directly (the value the reflection was integrated with) rather than recomputing the erf pair per reflection - recomputing only when scaling overrides the geometry (-w wedge refinement, --mosaicity, or a scaling wedge override). Output-neutral on the /data/rotation_test battery (SG/cell/completeness identical, ISa/CC1/2 within run noise on the stable crystals). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:57:16 +02:00
leonarski_fandClaude Fable 5	8c084021e6	ice rings: exclude from scaling, keep in the final merge (recover completeness) ... --detect-ice-rings previously dropped ice-ring reflections outright before scaling, which also removed them from the merge and gutted low/mid-resolution completeness on crystals that merge them fine (e.g. lysoC 80% -> the ice bands show as jagged shell completeness). Instead flag them (new Reflection.on_ice_ring, set in JFJochProcess, carried through the 3D combine) and exclude the flag only where a model is fit: the per-image scale (ScaleOnTheFly::Accept + the per-image CC), and - for the de-novo P1 pass - the space-group search and error model (MergeOnTheFly::ExcludeIceRings, on for for_search). The final in-symmetry merge and its statistics keep them, so completeness is preserved. /data/rotation_test battery vs the previous drop-from-merge behaviour: space group correct on all 18; completeness recovered broadly with CC1/2 and ISa held (cytC_2 82->99.7%, cytC_3 73->99.7%, InsI3 76->99.5%, lysoC 80->99.7%, MyoB 80->99.7%, InsH3 78->99%). Excluding ice from the P1 search merge is what keeps the space group correct: without it InsI3 flipped I23->P1 and EP_cs_01-17 P2->P1. Known limitation: on heavy-ice crystals (EP_cs_01-17) the strong ice is garbage and keeping it in the final merge collapses CC1/2 in the ice shells (91.7->6.9%). Distinguishing strong vs weak/absent rings per crystal needs data-driven, per-ring ice detection (azimuthal radial profile) - the planned next step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:29:37 +02:00
leonarski_fandClaude Fable 5	1ca5a69054	ScaleOnTheFly: solve the per-image scale by IRLS instead of Ceres ... In the default rotation (rot3d) path only G is refined - B is fixed, mosaicity is pinned and the wedge is not refined - so the predicted intensity G*coeff is linear in G and the robust (Cauchy) per-image scale is a 1-D M-estimate. Solve it directly by iteratively reweighted least squares (a few closed-form weighted ratios) instead of building a Ceres problem per image. Ceres is kept for the cases that are genuinely nonlinear: refining the B-factor or the rotation wedge. Same Cauchy objective as the Ceres path, but ~4x faster at scaling and ~30% faster overall on the /data/rotation_test battery, with space group, cell, ISa, completeness and CC1/2 matching across all 18 crystals (the two that look different, EP_cs_01-17 and EcwtAL500, are run-to-run unstable for both solvers). lyso_ref scaling 25.2->4.3s, cytC_2 15.2->2.6s, battery total 468->316s. Also drop the per-image G/B regularizers (gated by GetScalingRegularize, which nothing enables) and the now-unused RegularizationResidual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 13:55:04 +02:00
leonarski_fandClaude Fable 5	5c9175688a	jfjoch_process: let --detect-ice-rings=on|off override the dataset ... The master file's detectorSpecific/detect_ice_rings key is read back by HDF5MetadataSource onto the experiment, so a dataset collected with ice-ring detection on was reprocessed with exclusion forced on (dropping ~9% of reflections, gutting low/mid-res completeness on clean crystals) and jfjoch_process had no way to turn it off. Make --detect-ice-rings an optional<bool> taking =on|off (bare = on, back-compatible) and apply it after the dataset value is copied in, so an explicit CLI choice wins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 13:18:09 +02:00
leonarski_fandClaude Opus 4.8	ed9f6ac9eb	integration: remove --reciprocal-profile (proven per-frame dead-end) ... An 11-crystal mosaicity-stratified re-test (/data/rotation_test, off vs on vs a de-contaminated variant, plus a per-frame dump of the fitted widths) showed the dial is net-negative and cannot work in the per-frame paradigm: - The C|q|^2 mosaicity term - the whole point - is unfittable per-frame: the fitted curvature a2 comes out ~0 (often negative) on every crystal, with zero correlation to the XDS mosaicity (0.09..0.42 deg). Strong spots sit at low q where eta^2 q^2 is invisible; the curvature only appears at high q where there are ~0 strong spots. The law degenerates to a straight line. - With a2~0 the high-res width becomes a blind 1/cos^2(2theta) extrapolation, 2-4x wider than per-shell. The per-shell path's high-res "starvation" (flat narrow fallback) is accidentally correct: weak, crowded high-res spots want a narrow aperture, not the true wide spot shape. - The over-wide profile pulls background into weak spots -> R-meas rises, CC1/2 drops in reliable high-multiplicity shells (pding4_001, pding4_003, MyoB, EcwtCQ066). A cap at the widest well-sampled per-shell width recovers the regression, confirming over-widening is the harm. No crystal reliably wins; the apparent overall-CC gains were all in noise shells (mult 2-3, CC<20%). Delete the CLI flag, the BraggIntegrationSettings::reciprocal_profile setting, and the per-frame fit block. Default (per-shell) integration is byte-identical. NEXTGEN_INTEGRATOR.md records the finding as a dead-end for posterity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 11:39:09 +02:00
leonarski_fandClaude Fable 5	7e3ad84a5b	jfjoch_process/scale: default to mmCIF, auto-detect rotation, rot3d partiality ... - Default reflection format is now mmCIF (ScalingSettings.h, the single source of truth). jfjoch_process and jfjoch_scale no longer hard-code a local MTZ default; they override only when --scaling-output is given. jfjoch_viewer inherits this. - A dataset with a rotation goniometer axis is processed as rotation data (two-pass indexing) by default; add --process-as-stills to force per-frame stills. -R still tunes the first-pass image count. - rot3d is the default partiality model for rotation processing (fixed for stills) when no explicit -P is given. - Update docs/JFJOCH_PROCESS.md: new de-novo rotation/stills examples, corrected -R/-P/--scaling-output defaults, --process-as-stills, and a real Integration table. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 08:34:50 +02:00
leonarski_fandClaude Opus 4.8	4b393082d4	ci: drop requests dependency and use PowerShell for the Windows release upload ... The Windows viewer runner has Python but not the 'requests' package, and does not necessarily have bash. So: - rewrite gitea_upload_file.py to use only the Python stdlib (urllib), which works with a bare interpreter on both the Linux package runners and Windows; also drop the file's unused create_release() (gitea_create_release.py owns that); - run the Windows 'Upload installer to release' step in PowerShell (always present) instead of bash, globbing the NSIS .exe with Get-ChildItem. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:51:54 +02:00
leonarski_fandClaude Opus 4.8	ae1ea41151	ci: matrix the Windows build and upload installers to the release on tag ... The two Windows jobs (build-windows / build-windows-nocuda) differed only in -DJFJOCH_USE_CUDA=ON/OFF, so collapse them into one matrixed job (variant cuda/nocuda), mirroring the build-rpm matrix. On a tag, upload the NSIS installer (jfjoch-<version>-win64-{cuda<major>|cpu}.exe, named in CMakeLists.txt) to the release via gitea_upload_file.py, the same helper the RPM/DEB nocuda variants use to attach viewer/writer artifacts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:47:01 +02:00
leonarski_fandClaude Opus 4.8	48a7a67283	docs: publish third-party notices as a documentation page ... The THIRD_PARTY_NOTICES.md manifest lived only at the repo root and was referenced from docs/SOFTWARE.md via a ../ link that escapes the Sphinx source tree, so it never rendered in the published docs and was absent from the navigation. Add docs/THIRD_PARTY_NOTICES.md to the General toctree and fix the SOFTWARE.md link. The docs page is generated from the canonical root file by update_version.sh (like the python-client docs): licenses/*.txt links point at the repository, and the project-license links point at the in-docs LICENSE / FPGA_LICENSE pages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:42:22 +02:00
leonarski_f	ec904bc21d	Remove Gitlab remains	2026-07-01 21:36:36 +02:00
leonarski_f	91cae946a9	VERSION: 1.0.0-rc.156	2026-07-01 21:33:37 +02:00
leonarski_fandClaude Opus 4.8	0f327e4550	docs: add rc.156 changelog entry ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:32:51 +02:00
leonarski_fandClaude Opus 4.8	fa121f0079	jfjoch_viewer: show "Multiple lattices detected" in image statistics ... Add a row after the Indexing solution that flags images with more than one indexed lattice (indexing_lattice_count > 1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:30:10 +02:00
leonarski_fandClaude Opus 4.8	00821e32ef	rotation indexer: refine candidate cells and pick the best after refinement ... De-novo two-pass indexing failed on large-cell / superstructure / modulated crystals (EcwtAL500 0%, EcwtCQ034 0%) and mis-handled a pseudo-symmetric one (EcwtCQ066 14%). The common cause: the choice of unit cell was made too early, on the raw pre-refinement spot fraction, which is an unreliable discriminator (a correct hexagonal cell indexes only ~13% of the un-refined accumulated spots, while a wrong larger cell can index more). Move the decision to after geometry refinement, where it is reliable: - FFTIndexer now OFFERS widened candidate cells instead of deciding. ReduceResults gains a `widen` flag: the standard path (9 shortest vectors) is unchanged; the widened path anchors the two short axes and lets the third range over all filtered vectors (+dedup) to reach the long axis of an elongated cell. FilterFFTResults takes the peak count as a parameter (30 standard, 60 widened). RunInternal appends widened candidates only when its standard best indexes poorly, so compact crystals are untouched. - RotationIndexer fully refines the top few candidates and keeps the one that indexes the most spots under its own refined geometry (IndexedFraction). Each refine is length-bounded (1.2x the found cell) so a free triclinic refine cannot drift onto a pseudo-translation / modulation supercell (CQ034's satellites). The earlier (primary) candidate is preferred: a later one is adopted only if it indexes clearly more and reasonably well in absolute terms, so a twin's noisy near-tie cannot displace it. Extra/twin lattices are only searched when the chosen cell is the FFT primary (lattice[0]), since MultiLatticeSearch's rotations are derived from that primary. - The pseudo-symmetry guard (de-novo only - a user-fixed space group is always honored) is a ratio of refined fractions: refine the primitive as triclinic and drop to P only if the constrained cell indexes less than half of it. A false promotion indexes badly under its constraint (CQ066 ratio ~0.1) while genuine higher symmetry, including R-centred, indexes comparably (Ins_H R3 ratio ~0.7) and is kept. Validated on the full /data/rotation_test battery: AL500 0->89% (C2), CQ034 0->99%, CQ066 14->93% (ISa 7.2->13.7); the other 15 crystals keep their exact cell, space group, indexing rate and ISa (no regressions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 21:17:53 +02:00
leonarski_fandClaude Opus 4.8	dfc6703f4a	jfjoch_process: don't double-smooth per-frame G on the reference path ... SmoothImageScaleG rewrites the partials in place (image_scale_corr and image_scale_g). On the no-reference path that is harmless: each scaling pass recomputes G from scratch via ScaleAllImages, so smoothing always runs on freshly-refined G. On the reference path the scaling loop is skipped, so G is computed once and stays; running scale_and_merge twice (P1 then the adopted space group) smoothed the already-smoothed G a second time, compounding into a ~2x wider effective kernel than the configured --smooth-g and biasing the merged intensities. Smooth only on the first pass of the reference path (G is unchanged afterwards, and the smoothed partials persist into the second pass's combine3D). The no-reference path is unchanged. Verified on lyso (600 frames, -P rot3d -z ref.mtz -M): the reference run now logs the smoothing once instead of twice, and the merged MTZ changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 14:24:05 +02:00
leonarski_f	ab2176960f	OpenAPI: Fix smargon descriptions	2026-07-01 14:12:09 +02:00
leonarski_fandClaude Opus 4.8	c67ea2fa42	receiver: guard size_t underflow in compression buffer sizing ... When per-image CBOR metadata comes within 32 bytes of the buffer slot size, slot_size - (metadata_size + 32) wrapped around (size_t), passing a huge output_size to CompressImage. That defeated the buffer-too-small guard and let the compressor write the full image past the end of the slot, corrupting adjacent memory; AppendImage then threw a plain JFJochException that aborted the whole collection after the fact. Detect metadata_size + 32 >= slot_size explicitly and throw CompressionBufferTooSmallException, so the existing catch drops just this frame gracefully - the case the change was meant to handle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 14:05:20 +02:00
leonarski_fandClaude Opus 4.8	b7613c559c	smargon: write chi/phi goniometer position into NXmx transformations ... Add an optional Smargon static positioner (chi/phi angles + rotation axes) that is reconstructed into the NXmx sample transformation chain. Chi/phi are appended at the innermost end of the chain (closest to the sample) for both the goniometer and grid-scan branches, with axes defaulting to chi {0,0,1} and phi = omega default {1,0,0}. - SmargonPosition gains chi_axis/phi_axis (common/JFJochMessages.h) - OpenAPI: optional phi_axis/chi_axis arrays; clients regenerated - OpenAPIConvert wires Dataset_settings.smargon -> DatasetSettings - CBOR serializer/deserializer round-trip the axes - tests: CBORSerialize_Start_Smargon Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 11:45:08 +02:00
leonarski_fandClaude Opus 4.8	35607057d9	CC1/2 merge: assign half-sets by deterministic per-image hash ... MergeOnTheFly::AddImage picked each image's CC1/2 half from a shared mt19937 drawn in call order (and before Mask), so the split depended on iteration/thread order and on how many images were masked. The class is mutex-guarded for concurrent "on-the-fly" use, so any parallel merge would make CC1/2 non-reproducible - a latent race. Assign the half as a splitmix64 hash of the image's stable index instead, computed after Mask. The split is now reproducible run-to-run, independent of AddImage order, parallel-safe, and decoupled from masking. Callers pass the outcome's vector index as the image id. Verified: lyso_ref two-pass -M -P rot3d gives identical CC1/2 across runs (overall 99.6%, P41212); hash split is balanced ~50/50. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 07:25:34 +02:00
leonarski_fandClaude Opus 4.8	a773aaa6a9	WriteReflections: write merging statistics + ISa to the mmCIF output ... The .cif (--scaling-output cif) now carries the per-shell and overall merging statistics in the standard _reflns / _reflns_shell categories (resolution, redundancy, completeness, <I/sigma>, R-rim/R-meas, CC1/2) plus the Diederichs asymptotic I/sigma (ISa) as a free-text _reflns.pdbx_diffrn_ISa item (no standard CIF tag exists). The MergeStatistics and the ISa string are threaded through WriteReflections to the mmCIF writer; jfjoch_process and jfjoch_scale pass them. Values match the text statistics table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 22:29:45 +02:00
leonarski_fandClaude Opus 4.8	ff91ca5e3a	jfjoch_process: default --reject-outliers 6 for rot3d (XDS-like) ... Per-observation merge outlier rejection at 6 sigma is neutral-or-better across every test crystal (CC1/2 up or flat: lysoC 99.7->99.9, MyoB 98.9->99.6, EP0210 97.6->98.8, cytC 99.7->100.0, InsI2 99.2->99.7; R-meas slightly lower everywhere; ISa unchanged), and it is the lever for radiation-damaged / ice data (EP0117 reaches CC1/2 ~95% with it). Make it the rot3d default, like scale-fulls and smooth-g; --reject-outliers 0 disables it. Off for the non-rot3d partiality models, which were not benchmarked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 22:04:36 +02:00
leonarski_fandClaude Opus 4.8	636ed1e180	space group: use merge reduced chi^2 (not R-meas) for the consistency check ... The R-meas-ratio over-symmetry guard (commit 7ade6d9) missed a near-perfect pseudo-2-fold: InsH2 (XDS R3) has a false 2-fold at operator CC 0.85 whose merged intensities still correlate well, so R-meas inflated only ~1.3x and R32 was accepted (ISa collapsing to 3.9). The reduced chi^2 (within-orbit scatter / sigma^2) is the right signal - false equivalents disagree by many sigma even when they correlate well. Measured chi^2 ratio (candidate / best subgroup) across the test set: true point groups <= 1.63 (a cubic merge of a noisy crystal, InsI3, is the worst real case), false >= 2.26 (InsH2) and 6.0 (InsH3); threshold 2.0 separates them. InsH2 now resolves to R3 (ISa 6.1 from the false R32's 3.9); InsH3/InsI2/InsI3/cytC/lysoC/MyoB/EP0210 unchanged and correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:54:51 +02:00
leonarski_fandClaude Opus 4.8	fea2790a41	rotation_indexer: detect hexagonal metric de-novo (fixes trigonal cytC indexing) ... The hexagonal lattice metric (two equal axes at 60/120 deg, both perpendicular to the third) is also satisfied by its ortho-hexagonal C-centred supercell, so the geometry-keyed LatticeSearch lands a metrically-hexagonal lattice on the C-orthorhombic setting. The -S path already re-expresses it (HexagonalConventional) keyed on the supplied space group; do the same de-novo, keyed on the metric of the reduced PRIMITIVE cell (rhombohedral lattices have a rhombohedral primitive cell and are unaffected). cytC (P3121) now indexes de-novo as hexagonal 83.8/83.8/88.6 gamma=120 and merges in P3121 (CC1/2 99.7%, ISa 13.1), for both test crystals, instead of the orthohexagonal C2/P1. lysoC/InsI3/InsH3 unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:31:20 +02:00
leonarski_fandClaude Opus 4.8	52248b8f5c	docs: design for auto-gated, data-driven ice-ring detection ... Records the validated z-score detector (on the azint radial background profile) and position/width estimation, for a future --detect-ice-rings=auto. Not yet implemented. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:22:31 +02:00
leonarski_fandClaude Opus 4.8	f3c51b0266	jfjoch_process: --min-image-cc is in percent, not a fraction (fix crash) ... --min-image-cc <num> is documented in percent and the per-image CC is a fraction in [0,1], but the value was passed straight to MinCCForImage (which requires [0,1]), so the documented usage (e.g. --min-image-cc 30) threw an uncaught JFJochException and aborted. Divide the percent argument by 100. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:09:18 +02:00
leonarski_fandClaude Opus 4.8	73d9e1de39	ice rings: widen the exclusion band to 0.03 q (measured ring FWHM) ... The hexagonal-ice ring FWHM measured on JUNGFRAU data (azimuthal radial background profile) is ~0.06 q, so the exclusion half-width should be ~0.03; the previous 0.02 under-covered the strong low-res rings. On EP0117 with --detect-ice-rings this lifts CC1/2 37->50%, and combined with --reject-outliers 3 (which down-weights the radiation-damaged late frames) reaches ~94% (XDS 98.5%). Only active with --detect-ice-rings, so default behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:09:18 +02:00
leonarski_fandClaude Opus 4.8	a74b8767de	ice rings: add --detect-ice-rings and exclude ice-ring reflections from scaling ... jfjoch_process gains --detect-ice-rings, which (a) activates the existing spot-finder ice flagging (ice spots de-prioritised in indexing) and (b) drops reflections sitting on a hexagonal-ice powder ring from scaling/combine/merge/stats, via a new shared IsOnIceRing() helper over ICE_RING_RES_A using the spot-finder's q half-width. Their integrated intensity is contaminated by the strong, variable ice background, so leaving them in mis-scales the whole frame and inflates the error model. On EP0117 (ice-ring crystal): de-novo space-group determination recovers from P1 to P2 and CC1/2 improves (31->37%). Off by default; a no-op without the flag. This is the first, non-controversial step - the residual gap needs ice-aware background/integration (follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:40:51 +02:00
leonarski_fandClaude Opus 4.8	7ade6d9906	space group: reject over-symmetrised point groups by merge R-meas consistency ... Per-operator CC alone cannot separate a real weak symmetry operator from a false moderate one: a noisy crystal's genuine cubic 2-folds (InsI2 I23, CC ~0.51) score BELOW a pseudo-symmetric crystal's false 2-fold (InsH3, CC 0.64), so no CC threshold works. Add a self-consistency test: a candidate point group is accepted only if merging the intensities under it does not inflate R-meas past max_rmeas_ratio (1.5x) the most-consistent candidate. A false operator forces non-equivalent reflections together and blows R-meas up; a real one leaves it flat. Fixes InsH3: was over-called R32 (ISa collapsed to 2.2 from the forced merge), now correctly R3/H3 (ISa 10.2, matching XDS). InsI2 stays I23, and lysoC/InsI3/MyoB/ EP0210/cytC/lyso_ref are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:32:03 +02:00
leonarski_fandClaude Opus 4.8	23d58b6202	integration: robust, well-conditioned per-frame --reciprocal-profile fit ... Fit the reciprocal tangential-width model y(q)=a0+a1*t+a2*t^2 in a centered, standardized variable t=(q-qbar)/qscale instead of the raw {1,q,q^2} monomials: the raw normal matrix went near-singular when the strong spots span a narrow q-range (small cell / sparse still), letting tiny per-frame jitter swing the curvature into a wild over-wide profile. Adds IRLS (Huber) robustness, a ridge on the curvature (sharp-crystal prior), and clamps the applied width to the fitted q-range (no extrapolation). Stays strictly per-frame (no dataset pooling), so it works online and for stills. Neutral on rotation data (cytC high-res CC1/2 win preserved 66.8 vs 65.6%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:01:05 +02:00
leonarski_fandClaude Opus 4.8	0783af247c	scaling: express rot3d smooth-g as a rotation range in degrees (XDS DELPHI-like) ... --smooth-g now takes a rotation range in degrees (default 5) instead of a frame count, converted to an odd frame window from the oscillation step inside JFJochProcess. This makes the per-frame scale-G smoothing physical and independent of the frame slicing. (Note: G-smoothing is not the cure for the low-symmetry stochastic collapse - the per-image scale Cauchy loss is - but a degree-based window is the correct parameterization regardless.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:00:53 +02:00
leonarski_fandClaude Opus 4.8	8d18c635ff	scaling: robust (Cauchy) loss on the per-image scale fit ... The per-image G/B/mosaicity fit used a plain L2 loss, so a few outlier reflections occasionally dragged a frame into a bad optimum - a stochastic (~15% of runs) per-frame mis-scaling that elevated R-meas and collapsed CC1/2 at low symmetry (the image-level CC1/2 half-split makes the damage look patchy across shells, while the data is genuinely noisier). A Cauchy loss (3 sigma) soft-downweights those outliers without a hard cut: MyoB 0/10 catastrophic (was ~2/10), R-meas stable, and ISa improves on every test crystal (EP0210 9.2->12.4, MyoB 12.5->14.6, lysoC 10.4->11.2, cytC 11.5->12.6), most on low symmetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 20:00:42 +02:00
leonarski_fandClaude Opus 4.8	e45a1577d6	jfjoch_process: optional absorption surface for rot3d scaling (--absorption) ... Adds an opt-in smooth absorption correction for rotation scaling. After the rot3d fulls are scaled, --absorption[=num] fits a multiplicative surface A(s1_crystal) - a degree<=4 monomial basis (real spherical harmonics up to l=4, as XDS/DIALS) of the diffracted-beam direction in the crystal/goniometer frame, by ridge-regularized log-linear least-squares of I_scaled/I_ref weighted by (I/sigma)^2, over num iterations (default 3); the surface divides image_scale_corr and the fulls are re-merged. Off by default and a no-op without rot3d. On the test panel (~13 keV, thin crystals) it is metric-neutral - fitted rms(log A) ~3-4%, ISa/CC1/2 unchanged - because absorption is negligible there and the per-frame scale G(phi) already absorbs the angular part. It is kept as a lever for low-energy data (e.g. 6 keV) where absorption becomes significant. Stored as ScalingSettings::absorption_iter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 15:23:01 +02:00
leonarski_fandClaude Opus 4.8	77f1ed2566	jfjoch_process: default scale-fulls for rot3d; skip _process.h5 when merging ... Two offline-processing ergonomics changes. scale-fulls is now ON by default for -P rot3d (it refits the per-frame scale on the combined fulls and lifts ISa substantially, e.g. HEWL rot3d 7.0 -> 16.4). --scale-fulls stays as the explicit opt-in for non-rot3d order; new --no-scale-fulls disables it for rot3d. (scale_fulls is now an optional<bool> defaulting to combine_3d.) Note: on low-completeness data the Unity-reference refit can cost a little CC1/2 (endothiapepsin ~70% complete: -5% in a mid shell); pair with --reject-outliers for the full low-symmetry benefit. When merging (-M), the merged reflections (.mtz/.cif) are the wanted output, so the large per-image _process.h5 is no longer written by default - it routinely ran to hundreds of MB. Pass --write-process-h5 to also emit it. Without merging the _process.h5 is the only output and is always written. Implemented with a ProcessConfig.write_process_h5 flag gating the FileWriter; reflection and image.dat writing are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:50:18 +02:00
leonarski_fandClaude Opus 4.8	21dd99524f	rotation_indexer: map FFT primitive cell to the SG conventional setting ... When a space group is supplied without a reference cell, de-novo two-pass rotation indexing fed the FFT's Niggli-reduced primitive cell straight into XtalOptimizer as if it were the conventional cell. For non-primitive lattices (centered I/F/R/C, or hexagonal where the primitive pair sits at gamma=60) the conventional-system model then refined to a wrong minimum and indexed 0% of frames: cytC (P3121) gave 103.9/103.9/78 instead of 83.7/83.7/88.6, insulin (I213) 66.7 instead of 77.65, insulin-R3 51/51/36 instead of 81.4/81.4/33.3. Run LatticeSearch on the FFT primitive cell (it already yields the correct conventional cell + reindex for I/F/R/C). For the one remaining gap - a metrically hexagonal lattice that the geometry-keyed search lands on the ortho-hexagonal C setting - re-express the reduced primitive cell in conventional hexagonal axes (b -> b - a opens gamma 60 -> 120). De-novo "-S" now indexes cytC/insu/Ins_H/lyso/MyoB/EP/lyso_ref at 100% with the correct cell; the "-C -S" path is unchanged. The helper stays in this .cpp (g++) rather than CrystalLattice.h to avoid recompiling CUDA units, which is broken under the box's CUDA-13 nvcc; promote it to a method once that is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:29:50 +02:00
leonarski_fandClaude Opus 4.8	3aec235f67	integration: add --reciprocal-profile dial (global reciprocal-space width) ... The per-shell profile width is learned in pixels, so it varies ~4x with resolution (mostly the geometric projection of a near-constant reciprocal-space relrod) and must be binned per shell -> it starves at high resolution / on sparse data. The new --reciprocal-profile flag instead learns ONE global width in reciprocal space, sigma2_q,tan = A + B|q| + C|q|^2: the Jacobian g_tan=cos(2theta) removes the geometric projection, and C|q|^2 is the crystal mosaicity relrod (variance ~(eta|q|)^2). Applied per reflection as sigma2_tan,px = (A + B|q| + C|q|^2)/g_tan^2 (B,C clamped >=0; quadratic->linear->constant fallback). Off by default. On the sharp HEWL test crystal (mosaicity 0.091deg, so C fits to ~0 and it reduces to the validated linear form) it is metric-neutral: ISa 16.2->16.3, anomalous 0.92x unchanged, CCref band 90.0->89.9, CC1/2 a touch lower (per-shell isn't starved at 23k spots/shell, and a global fit is less flexible). So: simpler + more transferable at a small CC1/2 cost, ISa/anomalous held. Its payoff is on MOSAIC crystals (large C|q|^2), where per-shell starves on the wide weak high-res spots and 6 shells are too coarse; both lyso test crystals are sharp, so it ships as a dial to try on mosaic data elsewhere. A separate radial relrod fit was tried and dropped (no gain). See NEXTGEN_INTEGRATOR.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 14:26:08 +02:00
leonarski_fandClaude Opus 4.8	5ba5fe8ad1	integration: centre the profile on the predicted sub-pixel position ... The profile was learned and applied on the integer pixel round(predicted), so a shared profile sits up to 0.5 px off the true spot (and stacking spots with random sub-pixel offsets broadens the learned profile). Build the Gaussian per reflection instead, centred on the predicted sub-pixel offset -- noise-free geometry, unlike the observed centroid, which hurt -- and elongated radially as before. HEWL rotation @1.0A: ISa 15.7->16.2, CCref band 89.9->90.0, CCxds 94.8->95.0 (high-res 1.00A CCref 66.0->66.9); sharp serial stills 1.68A CC1/2 61.6->62.5; anomalous S peak 0.92x XDS (no accuracy traded). De-broadening the learned width by the 1/12 px^2/axis integer-binning floor was tested and rejected (it over-narrows). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 07:34:06 +02:00
leonarski_fandClaude Opus 4.8	26901da42f	integration: de-contaminate the profile width, add a radial parallax ellipse ... The profile-fit width came from the full 13x13 second moment, which runs 3-8x wider than the true spot: neighbour reflections leak into the (unmasked) learning grid -- catastrophic at low res where spots crowd the beam -- and the far corners (lever arm dx^2+dy^2 up to 72) add rectified background noise. Splitting the spot moment into radial vs tangential shows the tangential width is isotropic (mosaicity/divergence) while the radial excess is pure sensor parallax ~tan^2(2th). - Measure the width over the signal disk r1 on the monochromatic path (inherently excludes neighbours, caps the radial tail); keep the generous full-grid width on the broadband/stills path (sparse spots, the centroid-undersampling floor). - Extend the (was bandwidth-only) radial ellipse with an analytic, material-aware parallax term c_par*tan^2(2theta), c_par = Var(z)/pixel^2 from sensor thickness + material + energy (parallax_var_px2; Si and CdTe), plus a fixed weak-spot capture term on the monochromatic path only. HEWL rotation @1.0A: ISa 13.5->15.7, CC1/2 1.12A 91.3->95.9 / 1.05A 83.0->85.2, external CCref band 88.1->89.9, CCxds 93.4->94.8, R-meas 9.4->8.7; low/mid flat. Sharp serial stills gain slightly from parallax; broadband stills neutral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 00:54:59 +02:00
leonarski_fandClaude Opus 4.8	b51036ad30	broker: don't clobber sensor thickness/material with the API default ... JFJochBrokerParser set SensorThickness_um / SensorMaterial unconditionally from the request's Detector model, but that model defaults them to 320 um / Si with IsSet=false. So any start request that didn't explicitly carry the sensor overwrote the detector-reported value (DECTRIS SIMPLON read) or the detector-specific default with 320 um -- the "PILATUS4 ends up 320 um" symptom. Guard both with the IsSet flag, mirroring highVoltageV just above. The receiver -> FillMessage -> CBOR -> writer chain was already correct; the value was simply wrong at the source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	6b989b4213	reader: read sensor thickness/material from the HDF5 metadata ... HDF5MetadataSource ignored /entry/instrument/detector/sensor_thickness and sensor_material and left the DetectorSetup default (320 um, Si). Read them when present (NXmx stores thickness in metres), so reprocessing honours the recorded sensor -- which the parallax/absorption model needs. (The acquisition path still records the 320 um default because DetDECTRIS sets no per-model thickness; that is a separate, acquisition-side fix.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	7a4fadf29b	bragg_prediction: reciprocal Lorentz divides by |S||S0|, not S.S0 ... S.S0 = |S||S0| cos(2theta), so dividing the rocking rate |m2 . (S x S0)| by it added a spurious 1/cos(2theta) factor (1.0 at low res, 1.8x at 1 A) to the reciprocal Lorentz, hence to the absolute/Wilson scale. Divide by |S||S0| to get the correct zeta*sin(2theta) (times a constant 1/lambda^2 the overall scale absorbs). The spurious factor depends on 2theta only, so it cancels within a resolution shell and between symmetry mates -- CC1/2, CCref and R-meas are metric-neutral -- but it corrupts the absolute scale and destabilises the per-image B-factor and cross-resolution error model. CPU and GPU kernels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	40f650b9e7	scale_merge: pool the per-frame background across each rocking event in Combine3D ... Each partial subtracted its own independently-estimated per-frame background, so a weak full assembled from N frames accumulated one background-estimation variance per frame. The true background is flat over the few frames of one rocking event, so replace each partial's background by the event mean and correct its intensity by n_bkg*(bkg - pooled), where n_bkg = (sigma^2 - I)/bkg is the effective background-pixel count -- correct for weak AND strong reflections (sigma^2/bkg would over-count strong ones and over-correct them). Single-frame events are a no-op. Measured (rotation lyso @1.0 A): 1.05 A CC1/2 81.3 -> 83.0, R-meas 81.4 -> 77.2, CCref +0.2, CCxds +0.3; overall R-meas 10.0 -> 9.4%; ISa preserved (13.5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:37 +02:00
leonarski_fandClaude Opus 4.8	eccd10a0a7	integration: neighbour-mask the profile-fit background ring, widen r3 8->10 ... ProfileIntegrate2D::BoxSum now excludes every predicted reflection's r2 disk from the r2..r3 background annulus (mirroring BraggIntegrate2D), so a neighbour Bragg peak can no longer bias a reflection's background high and over-subtract. With neighbours excluded the annulus can safely widen, so the default r_3 goes 8 -> 10 (more background pixels, lower-variance estimate). Measured (rotation lyso @1.0 A, external CCref/CCxds vs XDS): 1.05 A CCref +1.3 / CCxds +1.3, R-meas -5 pts, low-res R-meas unchanged. Serial (Jet8 @0.0002 bandwidth): 1.68 A CC1/2 +3.9 / CCref +1.1, 1.58 A +4.0 / +1.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:14 +02:00
leonarski_fandClaude Opus 4.8	bbd888dcc3	integration: gate the background-ring sigma-clip to stills (bandwidth>0) ... The 76e88b0f sigma-clip (reject ring pixels above mean+3*sqrt(mean)) de-biases bandwidth-streaked high-resolution stills, but it ran on all data. On rotation (no streaks) it clips legitimate high background pixels and biases the mean low, slightly inflating weak intensities and hurting the anomalous signal. Gate it to bandwidth>0 (stills), matching how the 814dff34 radial-profile change is already gated. Rotation lysozyme (self-scaled, smooth-g, -A): anomalous S-peak 0.84x -> 0.86x of XDS (SD_MET 11.71 -> 11.94, CL_CL 1.28x -> 1.25x), ISa unchanged. Stills (bandwidth set) are byte-identical (clip still applies). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00