Two deps the Linux build picked up from the system were absent on Windows.
Bundle both, unconditionally (matching how libtiff/FFTW are vendored), so the
tree builds without system Eigen/JPEG:
- Eigen 5.0.1 via FetchContent with OVERRIDE_FIND_PACKAGE, so every
find_package(Eigen3) -- ours, Ceres', and ffbidx' -- resolves to this copy and
the requested 3.4 version is satisfied by the newer major. ffbidx's bundled
eigen submodule is disabled (GIT_SUBMODULES "" on fast-indexer) so it no longer
creates a duplicate Eigen3::Eigen target. Verified on a CUDA build: Ceres 2.3.0,
the analysis libs, and ffbidx's CUDA code all compile against 5.0.1.
- libjpeg-turbo 3.0.4 via ExternalProject (upstream discourages add_subdirectory);
built+installed static and imported as JPEG::JPEG, replacing find_package(JPEG)
in preview/. SIMD is lazy: WITH_SIMD=ON + REQUIRE_SIMD=OFF means NASM-accelerated
when nasm is present, otherwise a warning and a scalar build (no hard failure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both are leaf deps (JFJochPreview / JFJochIndexing) that only our own code
looks for, so building them ourselves removes the system-package lottery and
gives a reproducible, statically-linked build on every platform.
- libtiff: FetchContent, library only (jbig/zstd/lzma/jpeg/old-jpeg/tools/
tests off). The C++ binding (TIFF::CXX / libtiffxx) is packaged
inconsistently across distros -- missing on Rocky 9 -- and absent on
Windows, so find_package(TIFF COMPONENTS CXX) was unreliable; that call is
removed and JFJochPreview links the tiff/tiffxx targets directly. GitHub
mirror because upstream (gitlab) is unreachable from some restricted hosts.
- FFTW: FetchContent single precision (ENABLE_FLOAT) from the release tarball
-- the git repo ships no pre-generated codelets (needs OCaml genfft). It's
now always available, so the CPU FFT indexer is always built and
JFJOCH_USE_FFTW always defined; the "FFTW disabled" path is gone. Static
(libfftw3f.a) via the global BUILD_SHARED_LIBS OFF.
Verified on Linux: jfjoch_viewer builds and links libfftw3f.a + libtiff*.a,
all static.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M_PI is a POSIX <math.h> extension that MSVC does not define without
_USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is
C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several
common/ headers are pulled into .cu device translation units, so std::numbers is
not available there.
Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works
in host code (including MSVC), in CUDA device code, and under C++17/20, and use
it everywhere:
- common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files
- broker/OpenAPIConvert.cpp
- viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent
convention across the codebase
Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(),
which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None;
the pool has no policy to resolve Auto itself. The worker handled a stray Auto
with a dead branch and silently produced no result when the resolved algorithm
had no matching indexer built.
- Document on GetIndexingAlgorithm() that it never returns Auto.
- Throw a clear internal error at the pool boundary if Auto ever arrives.
- In the worker, replace the dead Auto branch with a loud failure for any
resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a
host without a GPU), instead of returning no result silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are
placed device-local by the kernel (dma_alloc_coherent), the big RAM ring
buffer is random-access (first-touch handles placement), and GPU work is
already spread across all visible devices. So drop the pinning entirely
and with it libnuma.
- Delete NUMAHWPolicy; the only concern worth keeping - GPU selection -
is done directly via pin_gpu() (round-robin over visible GPUs) in the
indexer pool and the Lite analysis threads. CPU-only threads
(FPGA acquire/pedestal/summation/frame-transform) no longer bind
anything.
- Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA
used it.
- numa_policy broker setting is deprecated and ignored (kept in the API
for backward compatibility; warns once on startup).
- Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake.
- Docs: drop the NUMA dependency, remove the numa_policy config example,
and document running multiple brokers on disjoint GPUs via
CUDA_VISIBLE_DEVICES.
- Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-direction peak search took argmax|spec|. The projected-spot histogram has a
broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can
exceed the true lattice peaks, so on weak / pink-beam frames every direction reported
a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate
cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors,
the true axes entirely absent.)
Subtract a running-mean background of half-width ~15 A and pick the peak by its
PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp
lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes
win. The prominence is also reported as the magnitude, so FilterFFTResults ranks
directions by real-peak strength rather than envelope. Ported identically to CPU
(prefix-sum window) and GPU (sliding-window in the kernel).
Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX
27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged).
The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes
the envelope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d
convention) while the bin width and spot coordinates use the internal 1/d convention -
a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the
same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x
zero-padding of the histogram.
Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an
explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d
is named one_over_d (never q). histogram_size is numerically identical to before, so
behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented
that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi
to a nearby integer shifts indexing ~0.5-1%).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8
* Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent
Reviewed-on: #52
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection
* jfjoch_broker: Minor bug fixes for rare conditions
Reviewed-on: #50
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress).
Reviewed-on: #45
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.11 instead).
* jfjoch_broker: Experimental rotation (3D) indexing
* jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values
Reviewed-on: #18
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_viewer: Minor polishing of new functionality
* jfjoch_broker: User NaN for empty azimuthal bins
Reviewed-on: #12
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_f
