1164 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	040cdeacf1	acquisition_device: give each device sole ownership of its buffers ... The base AcquisitionDevice no longer allocates or frees frame buffers; buffer_device is now just a non-owning view of addresses. Each subclass owns its backing memory and the matching lifecycle: - PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its own destructor (and on ctor failure), symmetric with MapKernelBuffer. - HLSSimulatedDevice owns plain zeroed heap buffers it points buffer_device into, declared before the HLSDevice so they outlive the action thread that writes them. The buffers are page-aligned to match the real device's kernel DMA buffers - the modelled AXI datamover and FPGAIntegrationTest require aligned output buffers. This drops the NUMA/mmap dance from the simulated path (not performance-critical) - removing libnuma from acquisition_device - and replaces the base-class cleanup that had to guess the allocation strategy with a single clear owner per device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:18:32 +02:00
leonarski_fandClaude Opus 4.8	d373ba0490	ImageBuffer: replace NUMA interleave with parallel first-touch ... Allocate the ring buffer with plain malloc and zero it across hardware_concurrency() threads, so each page is first-touched - and thus NUMA-placed - by whichever node the scheduler ran the zeroing thread on. For the random-access buffer this approximates the previous numa_alloc_interleaved placement, speeds up the one-time fault-in of the 150-200 GB allocation, and drops the libnuma dependency from this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 17:03:13 +02:00
leonarski_fandClaude Opus 4.8	02fa15c2b9	jfjoch_process: spread per-image GPU work across all visible GPUs ... The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA device, so all per-image preprocessing/spot-finding/azimuthal integration ran on GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before building the analysis resources so their CUDA streams/engines land on distinct devices. Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU dispatch with goals and a staged plan (multi-broker GPU isolation via CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:29:52 +02:00
leonarski_fandClaude Opus 4.8	1056acc3a6	build: guard the librt link to Linux only ... librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a $<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and the CUDA block) so -lrt is still passed on Linux - where some distros need it for clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the jfjoch_process link line on Linux. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:40:09 +02:00
leonarski_fandClaude Opus 4.8	39c808776f	build: portability groundwork toward a Windows/MSVC viewer ... - CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process). - common/NetworkAddressConvert.cpp: guard the network includes for _WIN32 (winsock2/ws2tcpip vs arpa/inet). - common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) - portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:37:21 +02:00
leonarski_fandClaude Opus 4.8	c69b5297d5	docs: add jfjoch_process page, refresh viewer/tools docs, unify CLI naming ... - Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its options, output files, and the broker/viewer/process distinction; mention jfjoch_scale for re-scaling/merging. - Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API. - Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst. - jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in usage/license/logger. - jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D short flag, make it long-only) and remove dead p/q/i short options. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 07:38:36 +02:00
leonarski_fandClaude Opus 4.8	3334e88d3e	frontend: fix state bugs, slim Plotly bundle, drop CRA leftovers ... - App: make getValues an arrow field so its `this` survives being passed as the `update` prop (the post-edit statistics refresh was broken). - DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual so the 1 s statistics poll no longer overwrites a slider mid-edit (reference `!=` on a freshly-parsed object was always true → snap-back). - DataCollection: collapse nested setState in grid-scan callbacks. - PreviewImage: build an immutable settings copy instead of mutating state in place; fix stale comment. - MultiLinePlotWrapper: drop per-render console.log; render via a custom Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only), cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip). - package.json: remove dead react-scripts test/eject scripts and CRA eslintConfig, move @redocly/cli to devDependencies. - Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts, setupTests.js). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:51:45 +02:00
leonarski_f	638aa379f4	Fix	2026-06-16 20:28:25 +02:00
leonarski_f	704e4b4eb3	Documentation: Data analysis description	2026-06-16 20:24:01 +02:00
leonarski_fandClaude Opus 4.8	5143ba0b1f	jfjoch_api.yaml: header_appendix hdf5 values go to /entry/user, not /entry/data ... The description said the "hdf5" sub-object is written under /entry/data, but the writer (NXmx::UserData) and the example in the same description both use /entry/user. Correct the path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:00:40 +02:00
leonarski_fandClaude Opus 4.8	e8d3eb1b08	docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link ... - Motivation no longer frames the format as serial-only: Jungfraujoch serves both rotation and serial MX, and the two-layout rationale (per-image CXI spots vs. dataset-wide NXreflections) is presented for both. - Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an integrated-format master whose /entry/data/data is a VDS linking back to the original images (all results, no image copies). - Add a section on the header_appendix / image_appendix (user_data) mechanism: how the free-form JSON flows through the start/image CBOR messages and how an "hdf5" sub-object is persisted to /entry/user, with an example. - Link the niggliClass row to International Tables for Crystallography A, Table 3.1.3.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:47:36 +02:00
leonarski_fandClaude Opus 4.8	0381d891bc	docs: add HDF5.md describing the NeXus layout and JFJoch extensions ... New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout, NXreflections for integration), the master/data-file layout and the three NXmx format variants, the NXmx-standard fields that are populated, and every Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint, /entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel, detectorSpecific, calibration, fluorescence, user). Content is derived from writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked against the NXmx and NXreflections definitions. JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to the new doc; HDF5 is added to the Sphinx toctree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:36:44 +02:00
leonarski_fandClaude Opus 4.8	caff857d8a	docs/CBOR.md: sync field names/types with the serializer ... Bring the protocol doc in line with CBORStream2Serializer/Deserializer: - reflection dist_ewald -> rp (dist_ewald is the spot field, not the reflection field) - az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count - threshold_energy typed as Map(string -> float), not float - end_date typed as string (it is serialized as a plain text string, with no datetime tag, unlike arm_date) - add missing reflection fields: phi, zeta, image_scale_corr - add missing spot fields: image, and indexed-only h, k, l, dist_ewald - add image indexing_lattice_count and end indexed_lattice_count - drop the duplicated image_scale_factor row in the End table Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:11:37 +02:00
leonarski_fandClaude Opus 4.8	6479b91e50	JFJochHDF5Reader: read master indexedLatticeCount (back-compat) ... The master-file writer emits /entry/MX/indexedLatticeCount, but the reader only looked for indexingLatticeCount (the name used by the per-file MX plugin and data-file read path). Existing master files on disk therefore returned no indexed-lattice-count vector. Read the master's indexedLatticeCount first, falling back to indexingLatticeCount, matching the niggli_class/niggliClass compatibility pattern. The writer is left unchanged so already-written files keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:03:34 +02:00
leonarski_fandClaude Opus 4.8	2ba28aea0e	LoadFCalcFromMtz: fall back to an intensity column when F-model is absent ... PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN, which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and use it directly as I_ref (F-model is squared, intensities are not). This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional integration, or PixelRefine against an external reference) to produce a merge, then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's own intensities - free of the reference structure's non-isomorphism bias - and iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine runs against it (crystal 2, 55279 reflections, CC1/2 94.3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 14:19:28 +02:00
leonarski_fandClaude Opus 4.8	545ebdf868	Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial data. Each image is assigned to one CC1/2 half, so removing it perturbs only that half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i). A negative value means dropping the image RAISES CC1/2 (it disagrees with the consensus). Images whose deltaCChalf is a low-side statistical outlier (< mean - N*stddev) are skipped when merging. Reference-free. Two passes over the retained integration outcomes; per-image contributions are re-derived rather than stored, so memory stays O(unique reflections + images) for full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off). Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2 95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS heterogeneous serial data (some crystals genuinely bad) but slightly trims a homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative cut still removes the tail), so it is opt-in. R-free is the real test (user's full 200k). Note: the reported overall N_obs still counts all observations; the exported merge (and CC1/2) correctly exclude the rejected images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 12:32:52 +02:00
leonarski_f	f878fb9d5d	SpotFindingSettings: Default signal to noise ratio is 4	2026-06-16 08:53:05 +02:00
leonarski_fandClaude Opus 4.8	929e6e1fa0	FFTIndexer: pick peaks by prominence above a local background, not raw magnitude ... The per-direction peak search took argmax|spec|. The projected-spot histogram has a broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can exceed the true lattice peaks, so on weak / pink-beam frames every direction reported a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors, the true axes entirely absent.) Subtract a running-mean background of half-width ~15 A and pick the peak by its PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes win. The prominence is also reported as the magnitude, so FilterFFTResults ranks directions by real-peak strength rather than envelope. Ported identically to CPU (prefix-sum window) and GPU (sliding-window in the kernel). Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX 27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged). The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes the envelope). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:25:06 +02:00
leonarski_fandClaude Opus 4.8	f2f95c44f6	FFTIndexer: name the histogram axis one_over_d (1/d), not q/Q ... The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d convention) while the bin width and spot coordinates use the internal 1/d convention - a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x zero-padding of the histogram. Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d is named one_over_d (never q). histogram_size is numerically identical to before, so behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi to a nearby integer shifts indexing ~0.5-1%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:24:59 +02:00
leonarski_fandClaude Opus 4.8	e1e2ca8e49	IndexAndRefine: unify cell setting across indexers, keep FFBIDX neutral ... Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when cell+SG are given, so every frame reports its cell in ONE consistent setting per space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same reflection as different HKLs and cannot be merged). Use LatticeSearch's conventional cell when its detected symmetry agrees with the user's space group. On noisy frames LatticeSearch can pick an alternative Bravais setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell, [78,78,38]->[110,111,38]) whose system disagrees; there: - FFBIDX already returns the reference setting (c-last), consistent with the conventional frames, so its raw lattice is safe -> use it. This recovers the ~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral. - de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order (c-first); merging it with the conventional (c-last) frames would corrupt the data -> reject the frame instead. Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last, consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis (~38A) is axis 2 in every merged frame - no axis-order mixing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:50:56 +02:00
leonarski_fandClaude Opus 4.8	40da1aab13	IndexAndRefine: conventionalise de-novo lattice when cell+SG are given ... Providing both a unit cell (-C) and space group (-S) silently broke the de-novo indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the `sg && GetUnitCell()` branch fed the indexer's raw lattice straight into symmetry-constrained refinement. FFBIDX returns the lattice already in the reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive cell; enforcing the crystal system on its mis-assigned axes rejects every frame. Fix: for de-novo indexers only, reduce the lattice to the conventional setting (LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so it is byte-identical to before (no regression). niggli_class stays unassigned (0) in this branch - it is a property of the primitive cell incl. centering, which LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2) cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred. Validation (lyso, -C79,79,38 -S96): crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical (Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no consensus on the weak serial-still frames, 0% even without -S; to investigate.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:35:03 +02:00
leonarski_fandClaude Opus 4.8	ecdb7048a0	PixelRefine: drop crystal-system idealisation, use the indexed cell as-is ... PixelRefine no longer refines the cell, so the symmetry-constrained cell parameterisation inherited from XtalOptimizer has no manifold to constrain. The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B matrix) merely reconstructed the lattice columns it started from, and for non-triclinic systems it re-idealised the indexed cell (averaging a,b for tetragonal; a,b,c for cubic; forcing alpha=gamma=90). Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a single path: take the three real-space lattice columns (latt.Vec0/1/2()) directly and form the reciprocal node via the general cross-product inverse. This reproduces every crystal system exactly from the actual cell, is more faithful (no re-idealisation), and removes the crystal_system field plus two now-unused includes. PredictedNode de-templated (only ever called with double). Crystal symmetry still lives where it belongs: indexing (upstream) and merging (downstream via the space-group HKL key). A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the old idealisation was already a no-op): stat tables bit-identical across all shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% -- the only delta is 4th-digit float-ordering noise in the error-model b coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free unchanged. Net -81 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 14:11:26 +02:00
leonarski_fandClaude Opus 4.8	8a9d80eb71	PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... (1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process --profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:48:11 +02:00
leonarski_fandClaude Opus 4.8	102a2a7c81	PixelRefine: strip experimental env knobs (orientation, sweep, Lorentz, ML census) ... Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed, returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper (r1_multiplier, default 6): - PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) - XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual, OrientationRegularizer, PixelObs::weight, the refinement block and its fields. - PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2 (per-image overfit). Removes SweepOrientationCell and its fields. - PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality already subsumes it); was already reverted. - PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5% of jet frames are multi-lattice; shelved). PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged: crystal 2 / jet / hybrid -R -r pixelrefine all reproduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:34:54 +02:00
leonarski_fandClaude Opus 4.8	6f733d74c2	Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... Promote the per-observation merge outlier rejection from the temporary PR_REJECT env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0 disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 -> 0.2585). Applies to both the PixelRefine and classical merge paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 12:42:25 +02:00
leonarski_fandClaude Opus 4.8	a47b376dc3	Merge: per-observation outlier rejection (env-gated PR_REJECT) ... At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every merged mean. Add a robust per-observation cut: drop observations whose corrected intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN. The error-model sigma already captures the genuine (partiality) scatter, and the median is a robust centre, so only the tail beyond the real spread is removed - not good partials. The median is computed in RefineErrorModel (which already pools the observations per reflection); AddImage applies the cut. Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut) - the cut is vs the data's own median, not the reference, so the gain is real cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at low nsigma) are for Filip's full-jet R-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:56:53 +02:00
leonarski_fandClaude Opus 4.8	45ee8c2b40	PixelRefine: env-gated orientation + cell-scale sweep (PR_SWEEP) for R-free A/B ... R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on R-free; x9 only buys CC1/2 internal consistency), and per-image orientation *refinement* is a no-op (0.2618 vs 0.2625). Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP (off by default) to test whether the cell-scale degree of freedom - which the gradient orientation refinement lacks, and which moves the high-res shells radially - helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for the gradient path), but that does not certify R-free, so it is left for validation. NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 19:59:50 +02:00
leonarski_fandClaude Opus 4.8	cfcd4c9e56	PixelRefine: R1 multiplier (fix tight-Term2 regression) + env-gated orientation refine ... Full-jet R-free showed the factored model regressed vs the traditional pipeline (R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix, XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier (default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3 (~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free. Also reinstates the pre-factored per-image orientation refinement (per-pixel ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation, re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation. NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier value and the orientation-refine decision are to be fixed against R-free, then baked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 09:18:09 +02:00
leonarski_fandClaude Opus 4.8	bf6efc7fe9	Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet (1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots spill past a radius-3 box), is neutral on the mono crystal and on the classical integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob. (Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res, azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured 2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight measured width loses to a generous aperture, same lesson as the radius bump.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 23:27:36 +02:00
leonarski_fandClaude Opus 4.8	100fe7b7e7	PixelRefine: make factored Terms 1+2 the model, remove old wiring ... PixelRefine is now an intensity-only operation: geometry is fixed (refined upstream by XtalOptimizer) and the only objective is the factored per-reflection likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills, 1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and matches the rotation path from the stills path. Removed: - the per-pixel ShoeboxResidual loss and PixelResidual cost functor; - all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R), the regularised-orientation LSQ, signal-weighting, and the global sweep; - Term 3 (per-spot recentring) - a confirmed no-op on both crystals; - the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the PR_* env knobs + stderr dumps in IndexAndRefine; - the PredictImage/ChiSquaredImage renderers and the entire viewer PixelRefine window/table/params + worker bindings + shoebox overlay. The sweep box-integrator background median became mean (consistency) by virtue of removing the sweep. METHODS.md rewritten for the current model; findings recorded in FINDINGS-2026-06.md. Net -2200 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 22:02:18 +02:00
leonarski_fandClaude Opus 4.8	6f2033db00	PixelRefine: checkpoint before cleanup (factored model + all diagnostic levers) ... Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_* env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/ ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit makes Terms 1+2 the model and strips all of this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 21:34:33 +02:00
leonarski_fandClaude Opus 4.8	c93d381dc8	Error model: harden the fit against pathological inputs (code review) ... Addresses code-review findings on RefineErrorModel: - Floor the 1/dev^2 bin weight relative to the data scale (1e-3 of the median bin dev^2), not an absolute 1e-30: a near-zero-scatter bin could otherwise acquire a runaway weight and hijack the global (a,b) fit. - Reject a near-collinear normal-equation system relatively (det > 1e-10*Ass*AII) instead of with an absolute threshold that an ill-conditioned fit can pass. - Reset the model to identity at entry so any early return leaves it inactive rather than keeping a stale a/b alongside a freshly-cleared mean map (which would make CorrectedSigma fall back to the per-observation I). - PixelRefine: correct the orient_prior comment - with the sweep on, the LSQ anchor is the swept orientation (intended), not the spot-centroid one. Verified unchanged on the lyso test set (ISa 1.1, CC1/2 90.3%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:53:02 +02:00
leonarski_fandClaude Opus 4.8	e6a50b45c7	Integration: mean background + global error model (trustworthy sigmas) ... Background estimate: use the mean of the local ring, not the median. For a right-skewed (Poisson) background the median sits below the mean, so subtracting it under-subtracts and biases every weak intensity positive; over multiplicity this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in both PixelRefine and BraggIntegrate2D (the classical route had the same bug). <I/sig> now tracks CC honestly and the true resolution limit is visible. Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the scatter of symmetry equivalents at the merge level (so both integrators benefit), and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the reflection mean (not the per-observation I_i, which biases the weights and collapses CC); a,b come from a relative-weighted bin regression. Replaces the earlier per-resolution-shell variant, which was partly masking the background bias. METHODS.md: document both (Sections 6-7), integrator-agnostic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:39:32 +02:00
leonarski_f	3cdf26232c	PixelRefine: Document on local changes	2026-06-12 17:29:28 +02:00
leonarski_f	db68c8dc38	PixelRefine: Results seem to be much better	2026-06-12 17:28:18 +02:00
leonarski_f	47dc19dd03	PixelRefine: Improvements to accept more reasonable count of reflections	2026-06-12 10:12:23 +02:00
leonarski_f	c8db50ab41	jfjoch_viewer: missing pixel refine table	2026-06-12 07:47:28 +02:00
leonarski_f	d31063ca3f	PixelRefine: Some improvements	2026-06-11 21:25:05 +02:00
leonarski_f	48d4fb0d0f	XDS plugin: Fix mutex	2026-06-11 18:35:02 +02:00
leonarski_f	bd5fef7f61	PixelRefine: Simplify (remove Lorentz correction, remove background from azimuthal integration)	2026-06-10 18:37:46 +02:00
leonarski_f	7478c0390f	jfjoch_scale: Remove postrefinement model option	2026-06-10 16:37:15 +02:00
leonarski_f	59281f6330	Revert "Bragg integration: option to use azimuthal integration profile" ... This reverts commit e4230bc14e.	2026-06-10 16:36:37 +02:00
leonarski_f	579d36fe71	Revert "jfjoch_process: Add option to use azimuthal integration as background for Bragg integration" ... This reverts commit b22d5929a1.	2026-06-10 16:36:37 +02:00
leonarski_f	b22d5929a1	jfjoch_process: Add option to use azimuthal integration as background for Bragg integration	2026-06-10 14:52:58 +02:00
leonarski_f	e4230bc14e	Bragg integration: option to use azimuthal integration profile	2026-06-10 14:41:36 +02:00
leonarski_f	e051eed033	jfjoch_process: Remove postrefinement	2026-06-09 19:46:39 +02:00
leonarski_f	efe882f4b6	jfjoch_viewer: Better display (to be tested) of pixel refine	2026-06-09 16:28:17 +02:00
leonarski_f	6c85aaba2b	BraggPrediction: Include X-ray bandwidth	2026-06-09 15:01:44 +02:00
leonarski_f	6af22b6a0c	jfjoch_viewer: show image CC based on standard pipeline	2026-06-09 13:28:56 +02:00
leonarski_f	003fea1b1e	jfjoch_viewer: fix sorting by indexing status	2026-06-09 13:28:35 +02:00