Generalize the expert-panel boilerplate (container + title + widgets +
upload button) and surface real upload errors via hey-api.
- SettingsPanel: consistent padded Paper + Typography title (one place to
size it) + optional Upload button + result snackbar; PanelTitle and
UploadSnackbarView exported for panels with bespoke layouts (DetectorSettings
grid, ROI)
- useUpload(mutation): wraps a generated hey-api mutation, shows the server's
error message on failure (errorMessage handles 400 string / 500 error_message)
- "unsaved changes" dot on panel titles, derived from the existing
last-downloaded snapshot (dirty = !isEqual(value, lastDownloaded))
- fix PixelMask: TIFF upload was swallowing errors; now shows them
- config panels (FileWriter, ImageFormat, Detector, AzInt, DarkMask, Indexing,
Instrument, ZeroMQ, ROI, DetectorSelection) upload through the typed SDK
instead of ButtonWithSnackbar's raw fetch; status/display panels share the
same shell for consistent spacing
- ButtonWithSnackbar now only powers bodyless action buttons (pedestal/cancel/
initialize, start/trigger, deactivate)
Build (tsc + vite) passes; dev server transforms all modules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace all class components with function components and move server-state
polling to @tanstack/react-query.
- add @tanstack/react-query; generate query helpers via the hey-api
@tanstack/react-query plugin (src/client/@tanstack)
- QueryClientProvider + one-time client.setConfig({ baseUrl: '' }) in index.tsx
- App polls statistics with useQuery(getStatisticsOptions, refetchInterval),
DataProcessingPlot uses getPreviewPlotOptions; manual setInterval polling gone
- memo() on presentational children; with TanStack structural sharing this
re-renders a child only when its own statistics slice changes
- PreviewImage stays imperative (binary JPEG -> object URL) using useEffect +
a ref for the object-URL lifecycle
- fix AzIntSettings correction checkboxes that mutated state in place (relied on
the poll re-render); they now use setState
- drop dead code uncovered during the port (unused upload/deactivate handlers
and imports in ImageFormatSettings, DetectorSettings, ROI)
Build (tsc + vite) passes; dev server transforms all entry modules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The raw stacked-module layout check compared the image width (cols) against
RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of
RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks
exported by tools like PyFAI (8-bit) failed. Route the upload through the
universal ReadTIFF reader and let PixelMask take a CompressedImage directly:
it validates the 2D shape against the detector's converted/raw layouts,
binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects
float/multi-channel images.
Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over
ReadTIFF, and the now-unused ReadTIFFFromString32 is removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies
on <filesystem>. Removing the dead POSIX include also avoids a needless
non-portable header for the eventual MSVC build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length,
which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN.
That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available
on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the
swap and emitted a little-endian length on a little-endian host -> corrupt
frames.
Write the 8 length bytes most-significant-first directly. This is big-endian by
construction on any host, needs no byte-swap intrinsic and no endianness macro,
and is byte-for-byte identical to the previous output on little-endian hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M_PI is a POSIX <math.h> extension that MSVC does not define without
_USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is
C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several
common/ headers are pulled into .cu device translation units, so std::numbers is
not available there.
Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works
in host code (including MSVC), in CUDA device code, and under C++17/20, and use
it everywhere:
- common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files
- broker/OpenAPIConvert.cpp
- viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent
convention across the codebase
Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(),
which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None;
the pool has no policy to resolve Auto itself. The worker handled a stray Auto
with a dead branch and silently produced no result when the resolved algorithm
had no matching indexer built.
- Document on GetIndexingAlgorithm() that it never returns Auto.
- Throw a clear internal error at the pool boundary if Auto ever arrives.
- In the worker, replace the dead Auto branch with a loud failure for any
resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a
host without a GPU), instead of returning no result silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same
d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic
convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is
left as-is (controlled by the OpenAPI generator's C++ enum casing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gmtime() returns a pointer to a shared static buffer, so concurrent
deserialization of CborUnixTime_tTag values was a data race. Decode into a local
struct tm via gmtime_r (gmtime_s on Windows) instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
images_sent was incremented right after image_pusher.SendImage(*loc), but the
ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it
silently drops the image (releases the slot and returns) when there is no live
connection or the 2 s enqueue deadline expires. So images_sent over-counted on a
broken/slow writer connection and disagreed with the ACK-based GetImagesWritten().
Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted
(enqueued/handed off) and only increment images_sent on success. The slot is
still released on the drop path. The authoritative delivered count remains
GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on
accept, preserving their previous always-counted behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The receiver already computes and stores the compression ratio per image
(JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio),
but the type was unreachable from the API: it was missing from ConvertPlotType,
the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and
no UI menu entry existed.
Wire it through all layers:
- broker: add the compression_ratio -> PlotType::CompressionRatio case in
ConvertPlotType.
- spec: add compression_ratio to the plot_type parameter enum.
- frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what
regeneration would produce), a "Compression ratio" menu entry, and a Y-axis
label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux
build: D-Bus stays on by default and the XCB plugin is still used.
- CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on
Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/
adaptor + service file only when enabled. Select the platform integration
plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead
of hard-coding QXcbIntegrationPlugin.
- JFJochViewerWindow: wrap the adaptor include and D-Bus registration in
#ifdef JFJOCH_VIEWER_DBUS.
- JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now
#ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only
headers are guarded too.
- Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the
viewer (not defined by MSVC without _USE_MATH_DEFINES).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is
hand-written on cpp-httplib, so the generated Pistache server stubs (api/,
impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are
regeneration debris that was previously left untracked and tended to carry a
newer spec version than the committed models. Ignore it so it can't be
committed by accident.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a status-bar cluster, shown when connected over HTTP, that surfaces
the broker state and live acquisition info:
- broker state box (QProgressBar) with progress drawn as the bar fill,
polled ~1 Hz on its own timer independent of image sync
- Live/Disconnected connection badge (host:port in tooltip)
- "+N new" badge and effective live-rate (Hz) readout
All widgets are fixed-width and only blanked in place, so the bar never
reflows when things appear/disappear.
Add a third autoload mode (HTTPSyncDataset): manually selecting an image
while following live now freezes the displayed image but keeps the dataset,
plots and image count updating. Surfaced via a tri-state HTTP Sync button
(off / live / data-only).
Also fix GetBrokerStatus() dropping the message field and a couple of
copy-pasted exception strings in JFJochHttpReader.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts
every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked
count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks),
negative pixel values (signed weighted-sum path), and an injected saturated
and masked pixel per ROI to cover the "max only" and "fully excluded" branches.
Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present,
mirroring ImageSpotFinderGPUTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three robustness fixes for the writer-facing TCP stream, addressing the
spurious "wrong number of images" / connection failures seen under load.
1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path
passes a caller-owned buffer with z == nullptr and reuses it for the next
frame immediately. With MSG_ZEROCOPY the kernel still references that buffer
after send() returns, so the peer could receive corrupted frames and drop
the connection mid-stream, truncating the run. Zero-copy is now gated on a
ZeroCopyReturnValue that keeps the buffer alive until completion.
2. Make the SendAll watchdog measure lack of progress, not total wall-clock.
The previous absolute deadline tore down a healthy but back-pressured
connection (slow/starved writer) after a fixed window; the watchdog now
resets on every byte actually sent, so only a genuinely stuck socket is
closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR.
3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still
be draining a backlog of DATA frames when END is sent; each DATA ACK is
progress, so the timeout only arms once the writer falls silent rather than
firing while images are legitimately still being drained.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were
only available on the FPGA path. Add a software ROI engine that mirrors the
FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and
intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so
it can contribute to any subset of up to 16 ROIs.
New image_analysis/roi/ library (JFJochROIIntegration), structured like azint:
a base that precomputes the per-pixel mask and names, a templated CPU engine
(generic over pixel type for a future 16-bit path), and a GPU kernel using
per-block shared-memory atomics for the STXM case (half-detector ROIs).
Masked pixels are excluded entirely; saturated pixels are excluded from the
sums but still count towards the max, matching roi_calc exactly. The engine is
only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already
consume message.roi, so no further changes are needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are
placed device-local by the kernel (dma_alloc_coherent), the big RAM ring
buffer is random-access (first-touch handles placement), and GPU work is
already spread across all visible devices. So drop the pinning entirely
and with it libnuma.
- Delete NUMAHWPolicy; the only concern worth keeping - GPU selection -
is done directly via pin_gpu() (round-robin over visible GPUs) in the
indexer pool and the Lite analysis threads. CPU-only threads
(FPGA acquire/pedestal/summation/frame-transform) no longer bind
anything.
- Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA
used it.
- numa_policy broker setting is deprecated and ignored (kept in the API
for backward compatibility; warns once on startup).
- Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake.
- Docs: drop the NUMA dependency, remove the numa_policy config example,
and document running multiple brokers on disjoint GPUs via
CUDA_VISIBLE_DEVICES.
- Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Several items in the note had landed or were inaccurate. Update it to:
- mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device
de-NUMA, CUDAWrapper sysfs node lookup;
- add the previously-missing FPGA DMA buffer section - placement is a
kernel concern (dma_alloc_coherent, device-local), not the userspace
mbind, which was only the simulator;
- record that libnuma is now down to a single file (NUMAHWPolicy.cpp);
- note that dependency removal (G3) and pinning behaviour (G4) are
separable axes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The base AcquisitionDevice no longer allocates or frees frame buffers;
buffer_device is now just a non-owning view of addresses. Each subclass
owns its backing memory and the matching lifecycle:
- PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its
own destructor (and on ctor failure), symmetric with MapKernelBuffer.
- HLSSimulatedDevice owns plain zeroed heap buffers it points
buffer_device into, declared before the HLSDevice so they outlive the
action thread that writes them. The buffers are page-aligned to match
the real device's kernel DMA buffers - the modelled AXI datamover and
FPGAIntegrationTest require aligned output buffers.
This drops the NUMA/mmap dance from the simulated path (not
performance-critical) - removing libnuma from acquisition_device - and
replaces the base-class cleanup that had to guess the allocation
strategy with a single clear owner per device.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allocate the ring buffer with plain malloc and zero it across
hardware_concurrency() threads, so each page is first-touched - and thus
NUMA-placed - by whichever node the scheduler ran the zeroing thread on.
For the random-access buffer this approximates the previous
numa_alloc_interleaved placement, speeds up the one-time fault-in of the
150-200 GB allocation, and drops the libnuma dependency from this file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA
device, so all per-image preprocessing/spot-finding/azimuthal integration ran on
GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a
process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op
without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before
building the analysis resources so their CUDA streams/engines land on distinct
devices.
Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU
dispatch with goals and a staged plan (multi-broker GPU isolation via
CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a
$<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and
the CUDA block) so -lrt is still passed on Linux - where some distros need it for
clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the
jfjoch_process link line on Linux.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before
slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's
bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single
libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the
full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process).
- common/NetworkAddressConvert.cpp: guard the network includes for _WIN32
(winsock2/ws2tcpip vs arpa/inet).
- common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an
anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) -
portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its
options, output files, and the broker/viewer/process distinction; mention
jfjoch_scale for re-scaling/merging.
- Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env
vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API.
- Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst.
- jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in
usage/license/logger.
- jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D
short flag, make it long-only) and remove dead p/q/i short options.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- App: make getValues an arrow field so its `this` survives being passed
as the `update` prop (the post-edit statistics refresh was broken).
- DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual
so the 1 s statistics poll no longer overwrites a slider mid-edit
(reference `!=` on a freshly-parsed object was always true → snap-back).
- DataCollection: collapse nested setState in grid-scan callbacks.
- PreviewImage: build an immutable settings copy instead of mutating state
in place; fix stale comment.
- MultiLinePlotWrapper: drop per-render console.log; render via a custom
Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only),
cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip).
- package.json: remove dead react-scripts test/eject scripts and CRA
eslintConfig, move @redocly/cli to devDependencies.
- Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts,
setupTests.js).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The description said the "hdf5" sub-object is written under /entry/data, but
the writer (NXmx::UserData) and the example in the same description both use
/entry/user. Correct the path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Motivation no longer frames the format as serial-only: Jungfraujoch serves
both rotation and serial MX, and the two-layout rationale (per-image CXI
spots vs. dataset-wide NXreflections) is presented for both.
- Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an
integrated-format master whose /entry/data/data is a VDS linking back to the
original images (all results, no image copies).
- Add a section on the header_appendix / image_appendix (user_data) mechanism:
how the free-form JSON flows through the start/image CBOR messages and how an
"hdf5" sub-object is persisted to /entry/user, with an example.
- Link the niggliClass row to International Tables for Crystallography A,
Table 3.1.3.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the
writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout,
NXreflections for integration), the master/data-file layout and the three
NXmx format variants, the NXmx-standard fields that are populated, and every
Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint,
/entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel,
detectorSpecific, calibration, fluorescence, user). Content is derived from
writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked
against the NXmx and NXreflections definitions.
JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to
the new doc; HDF5 is added to the Sphinx toctree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the protocol doc in line with CBORStream2Serializer/Deserializer:
- reflection dist_ewald -> rp (dist_ewald is the spot field, not the
reflection field)
- az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count
- threshold_energy typed as Map(string -> float), not float
- end_date typed as string (it is serialized as a plain text string, with
no datetime tag, unlike arm_date)
- add missing reflection fields: phi, zeta, image_scale_corr
- add missing spot fields: image, and indexed-only h, k, l, dist_ewald
- add image indexing_lattice_count and end indexed_lattice_count
- drop the duplicated image_scale_factor row in the End table
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The master-file writer emits /entry/MX/indexedLatticeCount, but the
reader only looked for indexingLatticeCount (the name used by the
per-file MX plugin and data-file read path). Existing master files on
disk therefore returned no indexed-lattice-count vector. Read the
master's indexedLatticeCount first, falling back to indexingLatticeCount,
matching the niggli_class/niggliClass compatibility pattern. The writer
is left unchanged so already-written files keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a
calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN,
which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and
use it directly as I_ref (F-model is squared, intensities are not).
This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional
integration, or PixelRefine against an external reference) to produce a merge,
then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's
own intensities - free of the reference structure's non-isomorphism bias - and
iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine
runs against it (crystal 2, 55279 reflections, CC1/2 94.3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname
Reviewed-on: #61
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial
data. Each image is assigned to one CC1/2 half, so removing it perturbs only that
half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i).
A negative value means dropping the image RAISES CC1/2 (it disagrees with the
consensus). Images whose deltaCChalf is a low-side statistical outlier
(< mean - N*stddev) are skipped when merging. Reference-free.
Two passes over the retained integration outcomes; per-image contributions are
re-derived rather than stored, so memory stays O(unique reflections + images) for
full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off).
Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2
95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS
heterogeneous serial data (some crystals genuinely bad) but slightly trims a
homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative
cut still removes the tail), so it is opt-in. R-free is the real test (user's full
200k). Note: the reported overall N_obs still counts all observations; the exported
merge (and CC1/2) correctly exclude the rejected images.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-direction peak search took argmax|spec|. The projected-spot histogram has a
broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can
exceed the true lattice peaks, so on weak / pink-beam frames every direction reported
a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate
cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors,
the true axes entirely absent.)
Subtract a running-mean background of half-width ~15 A and pick the peak by its
PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp
lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes
win. The prominence is also reported as the magnitude, so FilterFFTResults ranks
directions by real-peak strength rather than envelope. Ported identically to CPU
(prefix-sum window) and GPU (sliding-window in the kernel).
Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX
27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged).
The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes
the envelope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d
convention) while the bin width and spot coordinates use the internal 1/d convention -
a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the
same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x
zero-padding of the histogram.
Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an
explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d
is named one_over_d (never q). histogram_size is numerically identical to before, so
behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented
that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi
to a nearby integer shifts indexing ~0.5-1%).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when
cell+SG are given, so every frame reports its cell in ONE consistent setting per
space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same
reflection as different HKLs and cannot be merged).
Use LatticeSearch's conventional cell when its detected symmetry agrees with the
user's space group. On noisy frames LatticeSearch can pick an alternative Bravais
setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell,
[78,78,38]->[110,111,38]) whose system disagrees; there:
- FFBIDX already returns the reference setting (c-last), consistent with the
conventional frames, so its raw lattice is safe -> use it. This recovers the
~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral.
- de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order
(c-first); merging it with the conventional (c-last) frames would corrupt the
data -> reject the frame instead.
Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice
baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last,
consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis
(~38A) is axis 2 in every merged frame - no axis-order mixing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Providing both a unit cell (-C) and space group (-S) silently broke the de-novo
indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the
`sg && GetUnitCell()` branch fed the indexer's raw lattice straight into
symmetry-constrained refinement. FFBIDX returns the lattice already in the
reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive
cell; enforcing the crystal system on its mis-assigned axes rejects every frame.
Fix: for de-novo indexers only, reduce the lattice to the conventional setting
(LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so
it is byte-identical to before (no regression). niggli_class stays unassigned (0)
in this branch - it is a property of the primitive cell incl. centering, which
LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2)
cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred.
Validation (lyso, -C79,79,38 -S96):
crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality
crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical
jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical
(Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no
consensus on the weak serial-still frames, 0% even without -S; to investigate.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
