36 Commits

Author	SHA1	Message	Date
leonarski_f	39599bc090	tests: git-LFS large-dataset harness + jfjoch_process include cleanup ... - .gitattributes tracks tests/data/*.h5 via git-LFS for big reference datasets. - tests/TestData.h resolves tests/data files and reports absent / unfetched-LFS-pointer so tests can SKIP() instead of failing; tests/data/README.md documents fetching + the expected lyso_rotation/lyso_serial datasets. - JFJochProcessLargeTest: a Catch start-up listener that prints dataset availability, plus [large] full-analysis runs (rotation indexing + serial) that SKIP when data is absent. Verified against the real 1800-image lyso rotation set (100% indexing, cell 78.2/78.2/37.8) and skips cleanly without it. - jfjoch_process.cpp: drop the ~15 now-unused workflow includes left after the JFJochProcess extraction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 14:16:34 +02:00
leonarski_f	4697f10555	process: extract reprocessing workflow into shared JFJochProcess library ... The full processing workflow no longer lives in the CLIs. New process/JFJochProcess encapsulates it for jfjoch_process, jfjoch_azint and (later) the viewer: - ProcessMode {AzimuthalIntegration, FullAnalysis}; ProcessConfig carries run control (range, threads, output prefix, spot finding, rotation + scaling options, reference data) while the DiffractionExperiment carries all algorithm settings. - Run() executes setup -> optional two-pass rotation pre-pass -> parallel per-image loop (std::thread) -> optional scaling/merging post-pass -> NXmxIntegrated _process.h5 that links back to the original images. ProcessResult returns stats + merge text. - Cancel() / std::atomic<bool> (receiver style), checked between images; the CLIs install a SIGINT handler that calls it (fixes the previous Ctrl+C gap), the viewer will use the same hook. JFJochProcessObserver streams progress / per-image results for a live GUI. jfjoch_process.cpp and jfjoch_azint.cpp are now thin: argument parsing + experiment configuration, then JFJochProcess::Run + stats printing. Behaviour and usage messages are unchanged. Adds JFJochProcessTest (azimuthal integration round-trip, no-output run, pre-cancel) over a small generated dataset. Verified: tests/jfjoch_test [HDF5] (83 cases / 1854 assertions); jfjoch_azint and jfjoch_process run end-to-end on lyso_test (azint 20 images; full analysis recovers the lysozyme cell at 25% indexing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 10:36:18 +02:00
leonarski_f	1ea5897fca	windows: vendor wingetopt and scope server-only targets out of viewer-only builds ... MSVC/Windows build fixes: - Force JFJOCH_VIEWER_ONLY ON for Windows and macOS; the broker/receiver/FPGA/ writer server stack is Linux-only, so those platforms build the viewer subset. - Vendor an OpenBSD/NetBSD getopt/getopt_long shim (tools/wingetopt/, BSD/ISC) as a WIN32-only static lib -- the MSVC CRT has no <getopt.h> -- and link it into the four portable CLI tools on Windows. - jfjoch_extract_hkl.cpp: include <getopt.h> explicitly (it relied on glibc's transitive <unistd.h>). - jfjoch_process.cpp: guard <unistd.h> with #ifndef _WIN32. - Guard JFJochStreamWriter and jfjoch_writer with IF(NOT JFJOCH_VIEWER_ONLY): the former pulls in JFJochImagePuller, the latter uses a fork()/waitpid() multi-process design; neither belongs to the portable viewer subset. - Record wingetopt in THIRD_PARTY_NOTICES.md and licenses/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 18:50:20 +02:00
leonarski_f	e6c6536e3a	JFJochReceiverPlots: Move to common/, so it can be linked without bringing JFJochReceiver HW dependencies	2026-06-20 16:26:11 +02:00
leonarski_f	4737eff1a9	jfjoch_process: Missing include <set>	2026-06-20 07:44:50 +02:00
leonarski_f	03814b1425	writer: save ROI bitmap and definitions in the master file ... Persist regions of interest so an acquisition's ROI layout can be recovered from the master (_master.h5 / _process.h5): - /entry/roi/roi_map: uint16 per-pixel bitmask (bit i == ROI i). - /entry/roi/<name>: logical definition (type + geometry params, incl. azimuthal phi sector) plus bit_index tying it to the bitmap plane. The bitmap rides along in the StartMessage (CBOR, mirroring az_int_map) and is filled both online (JFJochReceiver::SendStartMessage) and offline (jfjoch_process). Definitions come from the already-transmitted rois. Both are dataset-wide metadata, so they are written by the NXmx master writer (new NXmx::ROI), not the per-image data-file plugin. Documented the /entry/roi layout in docs/HDF5.md and the master-vs-data writer convention in CLAUDE.md; added a CBOR roi_map round-trip test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 09:41:26 +02:00
leonarski_f	02fa15c2b9	jfjoch_process: spread per-image GPU work across all visible GPUs ... The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA device, so all per-image preprocessing/spot-finding/azimuthal integration ran on GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before building the analysis resources so their CUDA streams/engines land on distinct devices. Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU dispatch with goals and a staged plan (multi-broker GPU isolation via CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:29:52 +02:00
leonarski_f	c69b5297d5	docs: add jfjoch_process page, refresh viewer/tools docs, unify CLI naming ... - Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its options, output files, and the broker/viewer/process distinction; mention jfjoch_scale for re-scaling/merging. - Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API. - Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst. - jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in usage/license/logger. - jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D short flag, make it long-only) and remove dead p/q/i short options. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 07:38:36 +02:00
leonarski_f	545ebdf868	Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial data. Each image is assigned to one CC1/2 half, so removing it perturbs only that half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i). A negative value means dropping the image RAISES CC1/2 (it disagrees with the consensus). Images whose deltaCChalf is a low-side statistical outlier (< mean - N*stddev) are skipped when merging. Reference-free. Two passes over the retained integration outcomes; per-image contributions are re-derived rather than stored, so memory stays O(unique reflections + images) for full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off). Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2 95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS heterogeneous serial data (some crystals genuinely bad) but slightly trims a homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative cut still removes the tail), so it is opt-in. R-free is the real test (user's full 200k). Note: the reported overall N_obs still counts all observations; the exported merge (and CC1/2) correctly exclude the rejected images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 12:32:52 +02:00
leonarski_f	8a9d80eb71	PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... (1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process --profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:48:11 +02:00
leonarski_f	6f733d74c2	Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... Promote the per-observation merge outlier rejection from the temporary PR_REJECT env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0 disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 -> 0.2585). Applies to both the PixelRefine and classical merge paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 12:42:25 +02:00
leonarski_f	a47b376dc3	Merge: per-observation outlier rejection (env-gated PR_REJECT) ... At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every merged mean. Add a robust per-observation cut: drop observations whose corrected intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN. The error-model sigma already captures the genuine (partiality) scatter, and the median is a robust centre, so only the tail beyond the real spread is removed - not good partials. The median is computed in RefineErrorModel (which already pools the observations per reflection); AddImage applies the cut. Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut) - the cut is vs the data's own median, not the reference, so the gain is real cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at low nsigma) are for Filip's full-jet R-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:56:53 +02:00
leonarski_f	bf6efc7fe9	Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet (1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots spill past a radius-3 box), is neutral on the mono crystal and on the classical integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob. (Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res, azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured 2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight measured width loses to a generous aperture, same lesson as the radius bump.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 23:27:36 +02:00
leonarski_f	e6a50b45c7	Integration: mean background + global error model (trustworthy sigmas) ... Background estimate: use the mean of the local ring, not the median. For a right-skewed (Poisson) background the median sits below the mean, so subtracting it under-subtracts and biases every weak intensity positive; over multiplicity this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in both PixelRefine and BraggIntegrate2D (the classical route had the same bug). <I/sig> now tracks CC honestly and the true resolution limit is visible. Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the scatter of symmetry equivalents at the merge level (so both integrators benefit), and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the reflection mean (not the per-observation I_i, which biases the weights and collapses CC); a,b come from a relative-weighted bin regression. Replaces the earlier per-resolution-shell variant, which was partly masking the background bias. METHODS.md: document both (Sections 6-7), integrator-agnostic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:39:32 +02:00
leonarski_f	db68c8dc38	PixelRefine: Results seem to be much better	2026-06-12 17:28:18 +02:00
leonarski_f	579d36fe71	Revert "jfjoch_process: Add option to use azimuthal integration as background for Bragg integration" ... This reverts commit b22d5929a1.	2026-06-10 16:36:37 +02:00
leonarski_f	b22d5929a1	jfjoch_process: Add option to use azimuthal integration as background for Bragg integration	2026-06-10 14:52:58 +02:00
leonarski_f	e051eed033	jfjoch_process: Remove postrefinement	2026-06-09 19:46:39 +02:00
leonarski_f	254462b9f2	jfjoch_process: Clarify what happens in scaling when reference data provided	2026-06-09 07:28:13 +02:00
leonarski_f	155c53acd8	jfjoch_process: First pixel refine integration	2026-06-08 15:37:27 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	87fde1b32e	v1.0.0-rc.142 (#52) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8 * Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent Reviewed-on: #52 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 16:47:53 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00
leonarski_f	239a441ee6	v1.0.0-rc.140 (#50) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection * jfjoch_broker: Minor bug fixes for rare conditions Reviewed-on: #50	2026-04-29 21:40:22 +02:00
leonarski_f	230480e390	v1.0.0-rc.138 (#48) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations jfjoch_broker: Goniometer axis name is converted to lowercase jfjoch_broker: Fix bug, where wrong HTTP error codes were returned jfjoch_broker: Improve sigma estimation during merging (K. Takaba) --------- Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch> Reviewed-on: #48 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-27 19:56:14 +02:00
leonarski_f	c981e1b91c	v1.0.0-rc.137 (#46) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Better track time for each operation in the processing stack * jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress) * jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress) * jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0) Reviewed-on: #46	2026-04-25 19:59:21 +02:00
leonarski_f	6133da1377	v1.0.0-rc.133 ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Use httplib for HTTP server instead of Pistache * jfjoch_broker: Drop OpenSSL support * jfjoch_broker: Base work for multi-lattice support in the future * Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib) Reviewed-on: #41	2026-03-26 20:50:33 +01:00
leonarski_f	166fcdb68f	v1.0.0-rc.131 (#39) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS) * jfjoch_viewer: Fix calibration (pedestal) images being open flipped * jfjoch_process: Add space group detection (EXPERIMENTAL) Reviewed-on: #39	2026-03-07 11:34:04 +01:00
leonarski_f	928789a67c	v1.0.0-rc.130 (#37) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Rotation indexer has two retries if failes * jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot) * jfjoch_broker: Integration calculates background mask based on R2 radius * jfjoch_process: HDF5 files are not saved by default Reviewed-on: #37	2026-03-06 14:38:56 +01:00
leonarski_f	3036b1d37f	v1.0.0-rc.128 (#35) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Handle properly reuse of image buffer locations * jfjoch_broker: Fix bug in counting idle slots * jfjoch_broker: Force obtuse angle for monoclinic cells * jfjoch_process: Change scaling refinement tolerance Reviewed-on: #35 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-03 22:24:44 +01:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	32b681e591	v1.0.0-rc.126 (#33) ... This is an UNSTABLE release. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug for monoclinic space groups being wrongly refined when beta is much different from 90 deg. Reviewed-on: #33 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-19 07:01:38 +01:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00