196 Commits

Author	SHA1	Message	Date
leonarski_f	5f3126e19e	CMake: Update FFBIDX to support MSVC	2026-06-22 17:51:57 +02:00
leonarski_f	c86beeb393	rotation indexer: fix sub-range crash, explicit angle, settings + guards ... Running rotation indexing on a sub-range (e.g. images 60-120) segfaulted: the first pass passed the *global* image number to RotationIndexer::ProcessImage, which indexes v_ (sized to the run's image count) -> out-of-bounds write. - ProcessImage now takes an explicit mid-exposure angle (optional; falls back to the goniometer at the image index), so the indexer no longer assumes its slot index equals the goniometer image index. IndexAndRefine supplies it via RotationAngle(), matching the angle used for prediction. Added a bounds guard in ProcessImage so a bad index can never corrupt memory. - JFJochProcess feeds the rotation indexer the local ordinal (not the global index), and shifts the goniometer (start += start_image*incr, incr *= stride, per-image wedge preserved) so local index i maps to the angle of original image start+i*stride - fixing rotation angles for the whole sub-range pipeline (prediction, refinement, output), not just the indexer. - Expose "Rotation images" (number used for the first pass) in the job dialog, enabled when rotation indexing is on. (Count > available is already clamped by select_equally_spaced_image_ordinals.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 15:59:21 +02:00
leonarski_f	0c1babd0ca	CMake: provide Eigen externally again; fix cpack NSIS icon escape ... Revert Eigen from the FetchContent OVERRIDE_FIND_PACKAGE vendoring (cd394c52, d4d2d0d8) back to an external find_package(Eigen3) dependency, like zlib. The OVERRIDE_FIND_PACKAGE mechanism makes the Visual-Studio-bundled CMake (4.x "-msvc") intermittently segfault during configure -- ~1/3 of fresh configures, and 100% with Ceres CUDA enabled. The fault is in CMake's own FetchContent variable-stack cleanup, reached when Ceres' find_package(Eigen3) resolves the override through a nested FetchContent_MakeAvailable. Stock Kitware CMake runs the identical scripts fine, so it is a bug in the VS-bundled cmake binary; providing Eigen externally avoids that path entirely and is stable with Ceres CUDA both on and off (verified 0 crashes; full CUDA build + NSIS installer succeed under the VS-bundled cmake). Also fix CPACK_NSIS_INSTALLED_ICON_NAME: the backslash value was written verbatim into CPackConfig.cmake, where "\j" is an invalid escape that cmake 4.x (CMP0010 strict) rejects when cpack re-parses it, so cpack failed under the VS-bundled cmake (older/Kitware cpack only warned). Use a forward slash. - image_analysis: keep Ceres USE_CUDA OFF (Jungfraujoch does not use Ceres' GPU solvers) but drop the now-obsolete "required to avoid the Windows cmake crash" rationale from the comment. - docs: Eigen is a required external dependency again (SOFTWARE.md); the Windows viewer build provides zlib + Eigen externally (JFJOCH_VIEWER.md). - THIRD_PARTY_NOTICES: move Eigen from the fetched-and-linked table to the external runtime-libraries table. Docker images still need Eigen re-added (5.0.1, source build) -- to follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-20 22:33:12 +02:00
leonarski_f	5064b5ba47	image_analysis: force Ceres to build without CUDA ... USE_CUDA is a Ceres cache STRING (default "default" = auto-detect), not an option(), so a plain SET was shadowed by the cache default and CUDA got enabled on MSVC. Force it OFF in the cache instead. Also drop two dead settings: MINIGLOG (Ceres uses abseil logging now, no longer referenced) and EIGENSPARSE ON (already Ceres' default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 16:08:38 +02:00
leonarski_f	cd394c5249	CMake: vendor Eigen (5.0.1) and libjpeg-turbo so the build is self-contained ... Two deps the Linux build picked up from the system were absent on Windows. Bundle both, unconditionally (matching how libtiff/FFTW are vendored), so the tree builds without system Eigen/JPEG: - Eigen 5.0.1 via FetchContent with OVERRIDE_FIND_PACKAGE, so every find_package(Eigen3) -- ours, Ceres', and ffbidx' -- resolves to this copy and the requested 3.4 version is satisfied by the newer major. ffbidx's bundled eigen submodule is disabled (GIT_SUBMODULES "" on fast-indexer) so it no longer creates a duplicate Eigen3::Eigen target. Verified on a CUDA build: Ceres 2.3.0, the analysis libs, and ffbidx's CUDA code all compile against 5.0.1. - libjpeg-turbo 3.0.4 via ExternalProject (upstream discourages add_subdirectory); built+installed static and imported as JPEG::JPEG, replacing find_package(JPEG) in preview/. SIMD is lazy: WITH_SIMD=ON + REQUIRE_SIMD=OFF means NASM-accelerated when nasm is present, otherwise a warning and a scalar build (no hard failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 07:55:56 +02:00
leonarski_f	ebc28744e2	viewer: fix MSVC compatibility in viewer-reachable code ... Address portability issues found building jfjoch_viewer with MSVC: - common/{ADUHistogram,AzimuthalIntegrationProfile}.h and image_analysis/spot_finding/StrongPixelSet.h only need DeviceOutput, so include fpga/pcie_driver/jfjoch_fpga.h (plain-C, self-contained) directly instead of acquisition_device/AcquisitionDevice.h, which dragged <unistd.h> into the viewer tree. - common/time_utc.h: guard gmtime_r/timegm/localtime_r with the MSVC equivalents (gmtime_s/_mkgmtime/localtime_s) under _WIN32; drop the duplicated includes and add the headers used directly. - gemmi_gph/gemmi/utf.hpp: vendor the upstream gemmi header; fileutil.hpp includes it on Windows for UTF8_to_wchar but it was never vendored. - writer/HDF5Objects.cpp: ExtractFilename returns path.filename().string() (std::filesystem::path has no implicit conversion to std::string on Windows, where it is wchar_t-based). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 23:15:27 +02:00
leonarski_f	0220c6376d	cmake: fetch libtiff and FFTW instead of relying on system libs ... Both are leaf deps (JFJochPreview / JFJochIndexing) that only our own code looks for, so building them ourselves removes the system-package lottery and gives a reproducible, statically-linked build on every platform. - libtiff: FetchContent, library only (jbig/zstd/lzma/jpeg/old-jpeg/tools/ tests off). The C++ binding (TIFF::CXX / libtiffxx) is packaged inconsistently across distros -- missing on Rocky 9 -- and absent on Windows, so find_package(TIFF COMPONENTS CXX) was unreliable; that call is removed and JFJochPreview links the tiff/tiffxx targets directly. GitHub mirror because upstream (gitlab) is unreachable from some restricted hosts. - FFTW: FetchContent single precision (ENABLE_FLOAT) from the release tarball -- the git repo ships no pre-generated codelets (needs OCaml genfft). It's now always available, so the CPU FFT indexer is always built and JFJOCH_USE_FFTW always defined; the "FFTW disabled" path is gone. Static (libfftw3f.a) via the global BUILD_SHARED_LIBS OFF. Verified on Linux: jfjoch_viewer builds and links libfftw3f.a + libtiff*.a, all static. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 22:16:05 +02:00
leonarski_f	a433399106	image_analysis: surgical ROI-only entry points in MXAnalysisWithoutFPGA ... Add three entry points so ROI statistics can be (re)computed without a full re-analysis, in support of interactive ROI editing in the viewer: - RebuildROI(): recreate the ROI engine after the ROI set changes (the CudaStream is now kept as a member so the GPU engine can be rebuilt). - AnalyzeROIOnly(): decompress + preprocess + ROI, skipping azimuthal integration, spot finding and indexing (a new image when re-analysis is off). - RunROIOnly(): rerun only the ROI integration on the already-preprocessed image (an interactive ROI move). A full Analyze() still computes ROIs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	d14b8e9e18	Merge upstream	2026-06-18 18:52:22 +02:00
leonarski_f	5d9e1be814	portability: replace M_PI / std::numbers::pi with a host+device-safe PI ... M_PI is a POSIX <math.h> extension that MSVC does not define without _USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several common/ headers are pulled into .cu device translation units, so std::numbers is not available there. Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works in host code (including MSVC), in CUDA device code, and under C++17/20, and use it everywhere: - common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files - broker/OpenAPIConvert.cpp - viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent convention across the codebase Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:54:01 +02:00
leonarski_f	1558dddbb8	indexing: make the resolved-algorithm invariant explicit in the pool ... IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(), which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None; the pool has no policy to resolve Auto itself. The worker handled a stray Auto with a dead branch and silently produced no result when the resolved algorithm had no matching indexer built. - Document on GetIndexingAlgorithm() that it never returns Auto. - Throw a clear internal error at the pool boundary if Auto ever arrives. - In the worker, replace the dead Auto branch with a loud failure for any resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a host without a GPU), instead of returning no result silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_f	58910274bf	image_analysis: compute ROI statistics in the non-FPGA path ... MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were only available on the FPGA path. Add a software ROI engine that mirrors the FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so it can contribute to any subset of up to 16 ROIs. New image_analysis/roi/ library (JFJochROIIntegration), structured like azint: a base that precomputes the per-pixel mask and names, a templated CPU engine (generic over pixel type for a future 16-bit path), and a GPU kernel using per-block shared-memory atomics for the STXM case (half-detector ROIs). Masked pixels are excluded entirely; saturated pixels are excluded from the sums but still count towards the max, matching roi_calc exactly. The engine is only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already consume message.roi, so no further changes are needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 21:15:54 +02:00
leonarski_f	cc925b2668	Remove NUMAHWPolicy and the libnuma dependency ... NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are placed device-local by the kernel (dma_alloc_coherent), the big RAM ring buffer is random-access (first-touch handles placement), and GPU work is already spread across all visible devices. So drop the pinning entirely and with it libnuma. - Delete NUMAHWPolicy; the only concern worth keeping - GPU selection - is done directly via pin_gpu() (round-robin over visible GPUs) in the indexer pool and the Lite analysis threads. CPU-only threads (FPGA acquire/pedestal/summation/frame-transform) no longer bind anything. - Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA used it. - numa_policy broker setting is deprecated and ignored (kept in the API for backward compatibility; warns once on startup). - Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake. - Docs: drop the NUMA dependency, remove the numa_policy config example, and document running multiple brokers on disjoint GPUs via CUDA_VISIBLE_DEVICES. - Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 20:25:36 +02:00
leonarski_f	2ba28aea0e	LoadFCalcFromMtz: fall back to an intensity column when F-model is absent ... PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN, which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and use it directly as I_ref (F-model is squared, intensities are not). This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional integration, or PixelRefine against an external reference) to produce a merge, then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's own intensities - free of the reference structure's non-isomorphism bias - and iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine runs against it (crystal 2, 55279 reflections, CC1/2 94.3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 14:19:28 +02:00
leonarski_f	545ebdf868	Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial data. Each image is assigned to one CC1/2 half, so removing it perturbs only that half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i). A negative value means dropping the image RAISES CC1/2 (it disagrees with the consensus). Images whose deltaCChalf is a low-side statistical outlier (< mean - N*stddev) are skipped when merging. Reference-free. Two passes over the retained integration outcomes; per-image contributions are re-derived rather than stored, so memory stays O(unique reflections + images) for full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off). Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2 95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS heterogeneous serial data (some crystals genuinely bad) but slightly trims a homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative cut still removes the tail), so it is opt-in. R-free is the real test (user's full 200k). Note: the reported overall N_obs still counts all observations; the exported merge (and CC1/2) correctly exclude the rejected images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 12:32:52 +02:00
leonarski_f	f878fb9d5d	SpotFindingSettings: Default signal to noise ratio is 4	2026-06-16 08:53:05 +02:00
leonarski_f	929e6e1fa0	FFTIndexer: pick peaks by prominence above a local background, not raw magnitude ... The per-direction peak search took argmax|spec|. The projected-spot histogram has a broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can exceed the true lattice peaks, so on weak / pink-beam frames every direction reported a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors, the true axes entirely absent.) Subtract a running-mean background of half-width ~15 A and pick the peak by its PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes win. The prominence is also reported as the magnitude, so FilterFFTResults ranks directions by real-peak strength rather than envelope. Ported identically to CPU (prefix-sum window) and GPU (sliding-window in the kernel). Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX 27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged). The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes the envelope). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:25:06 +02:00
leonarski_f	f2f95c44f6	FFTIndexer: name the histogram axis one_over_d (1/d), not q/Q ... The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d convention) while the bin width and spot coordinates use the internal 1/d convention - a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x zero-padding of the histogram. Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d is named one_over_d (never q). histogram_size is numerically identical to before, so behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi to a nearby integer shifts indexing ~0.5-1%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:24:59 +02:00
leonarski_f	e1e2ca8e49	IndexAndRefine: unify cell setting across indexers, keep FFBIDX neutral ... Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when cell+SG are given, so every frame reports its cell in ONE consistent setting per space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same reflection as different HKLs and cannot be merged). Use LatticeSearch's conventional cell when its detected symmetry agrees with the user's space group. On noisy frames LatticeSearch can pick an alternative Bravais setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell, [78,78,38]->[110,111,38]) whose system disagrees; there: - FFBIDX already returns the reference setting (c-last), consistent with the conventional frames, so its raw lattice is safe -> use it. This recovers the ~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral. - de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order (c-first); merging it with the conventional (c-last) frames would corrupt the data -> reject the frame instead. Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last, consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis (~38A) is axis 2 in every merged frame - no axis-order mixing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:50:56 +02:00
leonarski_f	40da1aab13	IndexAndRefine: conventionalise de-novo lattice when cell+SG are given ... Providing both a unit cell (-C) and space group (-S) silently broke the de-novo indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the `sg && GetUnitCell()` branch fed the indexer's raw lattice straight into symmetry-constrained refinement. FFBIDX returns the lattice already in the reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive cell; enforcing the crystal system on its mis-assigned axes rejects every frame. Fix: for de-novo indexers only, reduce the lattice to the conventional setting (LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so it is byte-identical to before (no regression). niggli_class stays unassigned (0) in this branch - it is a property of the primitive cell incl. centering, which LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2) cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred. Validation (lyso, -C79,79,38 -S96): crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical (Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no consensus on the weak serial-still frames, 0% even without -S; to investigate.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:35:03 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60	2026-06-15 20:24:15 +02:00
leonarski_f	ecdb7048a0	PixelRefine: drop crystal-system idealisation, use the indexed cell as-is ... PixelRefine no longer refines the cell, so the symmetry-constrained cell parameterisation inherited from XtalOptimizer has no manifold to constrain. The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B matrix) merely reconstructed the lattice columns it started from, and for non-triclinic systems it re-idealised the indexed cell (averaging a,b for tetragonal; a,b,c for cubic; forcing alpha=gamma=90). Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a single path: take the three real-space lattice columns (latt.Vec0/1/2()) directly and form the reciprocal node via the general cross-product inverse. This reproduces every crystal system exactly from the actual cell, is more faithful (no re-idealisation), and removes the crystal_system field plus two now-unused includes. PredictedNode de-templated (only ever called with double). Crystal symmetry still lives where it belongs: indexing (upstream) and merging (downstream via the space-group HKL key). A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the old idealisation was already a no-op): stat tables bit-identical across all shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% -- the only delta is 4th-digit float-ordering noise in the error-model b coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free unchanged. Net -81 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 14:11:26 +02:00
leonarski_f	8a9d80eb71	PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... (1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process --profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:48:11 +02:00
leonarski_f	102a2a7c81	PixelRefine: strip experimental env knobs (orientation, sweep, Lorentz, ML census) ... Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed, returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper (r1_multiplier, default 6): - PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) - XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual, OrientationRegularizer, PixelObs::weight, the refinement block and its fields. - PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2 (per-image overfit). Removes SweepOrientationCell and its fields. - PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality already subsumes it); was already reverted. - PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5% of jet frames are multi-lattice; shelved). PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged: crystal 2 / jet / hybrid -R -r pixelrefine all reproduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:34:54 +02:00
leonarski_f	6f733d74c2	Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... Promote the per-observation merge outlier rejection from the temporary PR_REJECT env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0 disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 -> 0.2585). Applies to both the PixelRefine and classical merge paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 12:42:25 +02:00
leonarski_f	a47b376dc3	Merge: per-observation outlier rejection (env-gated PR_REJECT) ... At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every merged mean. Add a robust per-observation cut: drop observations whose corrected intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN. The error-model sigma already captures the genuine (partiality) scatter, and the median is a robust centre, so only the tail beyond the real spread is removed - not good partials. The median is computed in RefineErrorModel (which already pools the observations per reflection); AddImage applies the cut. Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut) - the cut is vs the data's own median, not the reference, so the gain is real cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at low nsigma) are for Filip's full-jet R-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:56:53 +02:00
leonarski_f	45ee8c2b40	PixelRefine: env-gated orientation + cell-scale sweep (PR_SWEEP) for R-free A/B ... R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on R-free; x9 only buys CC1/2 internal consistency), and per-image orientation *refinement* is a no-op (0.2618 vs 0.2625). Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP (off by default) to test whether the cell-scale degree of freedom - which the gradient orientation refinement lacks, and which moves the high-res shells radially - helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for the gradient path), but that does not certify R-free, so it is left for validation. NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 19:59:50 +02:00
leonarski_f	cfcd4c9e56	PixelRefine: R1 multiplier (fix tight-Term2 regression) + env-gated orientation refine ... Full-jet R-free showed the factored model regressed vs the traditional pipeline (R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix, XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier (default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3 (~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free. Also reinstates the pre-factored per-image orientation refinement (per-pixel ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation, re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation. NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier value and the orientation-refine decision are to be fixed against R-free, then baked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 09:18:09 +02:00
leonarski_f	bf6efc7fe9	Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet (1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots spill past a radius-3 box), is neutral on the mono crystal and on the classical integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob. (Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res, azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured 2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight measured width loses to a generous aperture, same lesson as the radius bump.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 23:27:36 +02:00
leonarski_f	100fe7b7e7	PixelRefine: make factored Terms 1+2 the model, remove old wiring ... PixelRefine is now an intensity-only operation: geometry is fixed (refined upstream by XtalOptimizer) and the only objective is the factored per-reflection likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills, 1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and matches the rotation path from the stills path. Removed: - the per-pixel ShoeboxResidual loss and PixelResidual cost functor; - all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R), the regularised-orientation LSQ, signal-weighting, and the global sweep; - Term 3 (per-spot recentring) - a confirmed no-op on both crystals; - the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the PR_* env knobs + stderr dumps in IndexAndRefine; - the PredictImage/ChiSquaredImage renderers and the entire viewer PixelRefine window/table/params + worker bindings + shoebox overlay. The sweep box-integrator background median became mean (consistency) by virtue of removing the sweep. METHODS.md rewritten for the current model; findings recorded in FINDINGS-2026-06.md. Net -2200 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 22:02:18 +02:00
leonarski_f	6f2033db00	PixelRefine: checkpoint before cleanup (factored model + all diagnostic levers) ... Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_* env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/ ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit makes Terms 1+2 the model and strips all of this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 21:34:33 +02:00
leonarski_f	c93d381dc8	Error model: harden the fit against pathological inputs (code review) ... Addresses code-review findings on RefineErrorModel: - Floor the 1/dev^2 bin weight relative to the data scale (1e-3 of the median bin dev^2), not an absolute 1e-30: a near-zero-scatter bin could otherwise acquire a runaway weight and hijack the global (a,b) fit. - Reject a near-collinear normal-equation system relatively (det > 1e-10*Ass*AII) instead of with an absolute threshold that an ill-conditioned fit can pass. - Reset the model to identity at entry so any early return leaves it inactive rather than keeping a stale a/b alongside a freshly-cleared mean map (which would make CorrectedSigma fall back to the per-observation I). - PixelRefine: correct the orient_prior comment - with the sweep on, the LSQ anchor is the swept orientation (intended), not the spot-centroid one. Verified unchanged on the lyso test set (ISa 1.1, CC1/2 90.3%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:53:02 +02:00
leonarski_f	e6a50b45c7	Integration: mean background + global error model (trustworthy sigmas) ... Background estimate: use the mean of the local ring, not the median. For a right-skewed (Poisson) background the median sits below the mean, so subtracting it under-subtracts and biases every weak intensity positive; over multiplicity this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in both PixelRefine and BraggIntegrate2D (the classical route had the same bug). <I/sig> now tracks CC honestly and the true resolution limit is visible. Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the scatter of symmetry equivalents at the merge level (so both integrators benefit), and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the reflection mean (not the per-observation I_i, which biases the weights and collapses CC); a,b come from a relative-weighted bin regression. Replaces the earlier per-resolution-shell variant, which was partly masking the background bias. METHODS.md: document both (Sections 6-7), integrator-agnostic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:39:32 +02:00
leonarski_f	3cdf26232c	PixelRefine: Document on local changes	2026-06-12 17:29:28 +02:00
leonarski_f	db68c8dc38	PixelRefine: Results seem to be much better	2026-06-12 17:28:18 +02:00
leonarski_f	47dc19dd03	PixelRefine: Improvements to accept more reasonable count of reflections	2026-06-12 10:12:23 +02:00
leonarski_f	d31063ca3f	PixelRefine: Some improvements	2026-06-11 21:25:05 +02:00
leonarski_f	bd5fef7f61	PixelRefine: Simplify (remove Lorentz correction, remove background from azimuthal integration)	2026-06-10 18:37:46 +02:00
leonarski_f	59281f6330	Revert "Bragg integration: option to use azimuthal integration profile" ... This reverts commit e4230bc14e.	2026-06-10 16:36:37 +02:00
leonarski_f	e4230bc14e	Bragg integration: option to use azimuthal integration profile	2026-06-10 14:41:36 +02:00
leonarski_f	e051eed033	jfjoch_process: Remove postrefinement	2026-06-09 19:46:39 +02:00
leonarski_f	efe882f4b6	jfjoch_viewer: Better display (to be tested) of pixel refine	2026-06-09 16:28:17 +02:00
leonarski_f	6c85aaba2b	BraggPrediction: Include X-ray bandwidth	2026-06-09 15:01:44 +02:00
leonarski_f	5735302691	Merge: CC1/2 limit adjustment	2026-06-09 11:06:16 +02:00
leonarski_f	e4b66f9cd9	PIxelRefine: Another iteration	2026-06-09 08:04:27 +02:00
leonarski_f	e8a9b1840d	PixelRefine: Make it faster by doing one cell calculation per shoe-box	2026-06-08 22:45:22 +02:00
leonarski_f	05711a1077	jfjoch_viewer: Add pixel refinw and magnifier windows (to be tested)	2026-06-08 21:22:19 +02:00
leonarski_f	698a98be35	PixelRefine: Claude fixed my bugs	2026-06-08 20:06:53 +02:00
leonarski_f	32e91f7287	Minor fixes	2026-06-08 15:56:20 +02:00
leonarski_f	155c53acd8	jfjoch_process: First pixel refine integration	2026-06-08 15:37:27 +02:00