Generate Zod schemas alongside the typed client (src/client/zod.gen.ts),
picking up the OpenAPI field defaults. Enables runtime validation and
default-filling for form payloads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the unsaved-changes dot: the Upload button uses the warning color
while the panel has unedited changes, primary otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalize the expert-panel boilerplate (container + title + widgets +
upload button) and surface real upload errors via hey-api.
- SettingsPanel: consistent padded Paper + Typography title (one place to
size it) + optional Upload button + result snackbar; PanelTitle and
UploadSnackbarView exported for panels with bespoke layouts (DetectorSettings
grid, ROI)
- useUpload(mutation): wraps a generated hey-api mutation, shows the server's
error message on failure (errorMessage handles 400 string / 500 error_message)
- "unsaved changes" dot on panel titles, derived from the existing
last-downloaded snapshot (dirty = !isEqual(value, lastDownloaded))
- fix PixelMask: TIFF upload was swallowing errors; now shows them
- config panels (FileWriter, ImageFormat, Detector, AzInt, DarkMask, Indexing,
Instrument, ZeroMQ, ROI, DetectorSelection) upload through the typed SDK
instead of ButtonWithSnackbar's raw fetch; status/display panels share the
same shell for consistent spacing
- ButtonWithSnackbar now only powers bodyless action buttons (pedestal/cancel/
initialize, start/trigger, deactivate)
Build (tsc + vite) passes; dev server transforms all modules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace all class components with function components and move server-state
polling to @tanstack/react-query.
- add @tanstack/react-query; generate query helpers via the hey-api
@tanstack/react-query plugin (src/client/@tanstack)
- QueryClientProvider + one-time client.setConfig({ baseUrl: '' }) in index.tsx
- App polls statistics with useQuery(getStatisticsOptions, refetchInterval),
DataProcessingPlot uses getPreviewPlotOptions; manual setInterval polling gone
- memo() on presentational children; with TanStack structural sharing this
re-renders a child only when its own statistics slice changes
- PreviewImage stays imperative (binary JPEG -> object URL) using useEffect +
a ref for the object-URL lifecycle
- fix AzIntSettings correction checkboxes that mutated state in place (relied on
the poll re-render); they now use setState
- drop dead code uncovered during the port (unused upload/deactivate handlers
and imports in ImageFormatSettings, DetectorSettings, ROI)
Build (tsc + vite) passes; dev server transforms all entry modules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The raw stacked-module layout check compared the image width (cols) against
RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of
RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks
exported by tools like PyFAI (8-bit) failed. Route the upload through the
universal ReadTIFF reader and let PixelMask take a CompressedImage directly:
it validates the 2D shape against the detector's converted/raw layouts,
binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects
float/multi-channel images.
Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over
ReadTIFF, and the now-unused ReadTIFFFromString32 is removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies
on <filesystem>. Removing the dead POSIX include also avoids a needless
non-portable header for the eventual MSVC build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length,
which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN.
That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available
on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the
swap and emitted a little-endian length on a little-endian host -> corrupt
frames.
Write the 8 length bytes most-significant-first directly. This is big-endian by
construction on any host, needs no byte-swap intrinsic and no endianness macro,
and is byte-for-byte identical to the previous output on little-endian hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M_PI is a POSIX <math.h> extension that MSVC does not define without
_USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is
C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several
common/ headers are pulled into .cu device translation units, so std::numbers is
not available there.
Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works
in host code (including MSVC), in CUDA device code, and under C++17/20, and use
it everywhere:
- common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files
- broker/OpenAPIConvert.cpp
- viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent
convention across the codebase
Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(),
which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None;
the pool has no policy to resolve Auto itself. The worker handled a stray Auto
with a dead branch and silently produced no result when the resolved algorithm
had no matching indexer built.
- Document on GetIndexingAlgorithm() that it never returns Auto.
- Throw a clear internal error at the pool boundary if Auto ever arrives.
- In the worker, replace the dead Auto branch with a loud failure for any
resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a
host without a GPU), instead of returning no result silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same
d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic
convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is
left as-is (controlled by the OpenAPI generator's C++ enum casing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gmtime() returns a pointer to a shared static buffer, so concurrent
deserialization of CborUnixTime_tTag values was a data race. Decode into a local
struct tm via gmtime_r (gmtime_s on Windows) instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
images_sent was incremented right after image_pusher.SendImage(*loc), but the
ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it
silently drops the image (releases the slot and returns) when there is no live
connection or the 2 s enqueue deadline expires. So images_sent over-counted on a
broken/slow writer connection and disagreed with the ACK-based GetImagesWritten().
Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted
(enqueued/handed off) and only increment images_sent on success. The slot is
still released on the drop path. The authoritative delivered count remains
GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on
accept, preserving their previous always-counted behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The receiver already computes and stores the compression ratio per image
(JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio),
but the type was unreachable from the API: it was missing from ConvertPlotType,
the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and
no UI menu entry existed.
Wire it through all layers:
- broker: add the compression_ratio -> PlotType::CompressionRatio case in
ConvertPlotType.
- spec: add compression_ratio to the plot_type parameter enum.
- frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what
regeneration would produce), a "Compression ratio" menu entry, and a Y-axis
label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux
build: D-Bus stays on by default and the XCB plugin is still used.
- CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on
Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/
adaptor + service file only when enabled. Select the platform integration
plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead
of hard-coding QXcbIntegrationPlugin.
- JFJochViewerWindow: wrap the adaptor include and D-Bus registration in
#ifdef JFJOCH_VIEWER_DBUS.
- JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now
#ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only
headers are guarded too.
- Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the
viewer (not defined by MSVC without _USE_MATH_DEFINES).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is
hand-written on cpp-httplib, so the generated Pistache server stubs (api/,
impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are
regeneration debris that was previously left untracked and tended to carry a
newer spec version than the committed models. Ignore it so it can't be
committed by accident.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a status-bar cluster, shown when connected over HTTP, that surfaces
the broker state and live acquisition info:
- broker state box (QProgressBar) with progress drawn as the bar fill,
polled ~1 Hz on its own timer independent of image sync
- Live/Disconnected connection badge (host:port in tooltip)
- "+N new" badge and effective live-rate (Hz) readout
All widgets are fixed-width and only blanked in place, so the bar never
reflows when things appear/disappear.
Add a third autoload mode (HTTPSyncDataset): manually selecting an image
while following live now freezes the displayed image but keeps the dataset,
plots and image count updating. Surfaced via a tri-state HTTP Sync button
(off / live / data-only).
Also fix GetBrokerStatus() dropping the message field and a couple of
copy-pasted exception strings in JFJochHttpReader.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts
every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked
count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks),
negative pixel values (signed weighted-sum path), and an injected saturated
and masked pixel per ROI to cover the "max only" and "fully excluded" branches.
Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present,
mirroring ImageSpotFinderGPUTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three robustness fixes for the writer-facing TCP stream, addressing the
spurious "wrong number of images" / connection failures seen under load.
1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path
passes a caller-owned buffer with z == nullptr and reuses it for the next
frame immediately. With MSG_ZEROCOPY the kernel still references that buffer
after send() returns, so the peer could receive corrupted frames and drop
the connection mid-stream, truncating the run. Zero-copy is now gated on a
ZeroCopyReturnValue that keeps the buffer alive until completion.
2. Make the SendAll watchdog measure lack of progress, not total wall-clock.
The previous absolute deadline tore down a healthy but back-pressured
connection (slow/starved writer) after a fixed window; the watchdog now
resets on every byte actually sent, so only a genuinely stuck socket is
closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR.
3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still
be draining a backlog of DATA frames when END is sent; each DATA ACK is
progress, so the timeout only arms once the writer falls silent rather than
firing while images are legitimately still being drained.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were
only available on the FPGA path. Add a software ROI engine that mirrors the
FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and
intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so
it can contribute to any subset of up to 16 ROIs.
New image_analysis/roi/ library (JFJochROIIntegration), structured like azint:
a base that precomputes the per-pixel mask and names, a templated CPU engine
(generic over pixel type for a future 16-bit path), and a GPU kernel using
per-block shared-memory atomics for the STXM case (half-detector ROIs).
Masked pixels are excluded entirely; saturated pixels are excluded from the
sums but still count towards the max, matching roi_calc exactly. The engine is
only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already
consume message.roi, so no further changes are needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are
placed device-local by the kernel (dma_alloc_coherent), the big RAM ring
buffer is random-access (first-touch handles placement), and GPU work is
already spread across all visible devices. So drop the pinning entirely
and with it libnuma.
- Delete NUMAHWPolicy; the only concern worth keeping - GPU selection -
is done directly via pin_gpu() (round-robin over visible GPUs) in the
indexer pool and the Lite analysis threads. CPU-only threads
(FPGA acquire/pedestal/summation/frame-transform) no longer bind
anything.
- Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA
used it.
- numa_policy broker setting is deprecated and ignored (kept in the API
for backward compatibility; warns once on startup).
- Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake.
- Docs: drop the NUMA dependency, remove the numa_policy config example,
and document running multiple brokers on disjoint GPUs via
CUDA_VISIBLE_DEVICES.
- Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Several items in the note had landed or were inaccurate. Update it to:
- mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device
de-NUMA, CUDAWrapper sysfs node lookup;
- add the previously-missing FPGA DMA buffer section - placement is a
kernel concern (dma_alloc_coherent, device-local), not the userspace
mbind, which was only the simulator;
- record that libnuma is now down to a single file (NUMAHWPolicy.cpp);
- note that dependency removal (G3) and pinning behaviour (G4) are
separable axes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The base AcquisitionDevice no longer allocates or frees frame buffers;
buffer_device is now just a non-owning view of addresses. Each subclass
owns its backing memory and the matching lifecycle:
- PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its
own destructor (and on ctor failure), symmetric with MapKernelBuffer.
- HLSSimulatedDevice owns plain zeroed heap buffers it points
buffer_device into, declared before the HLSDevice so they outlive the
action thread that writes them. The buffers are page-aligned to match
the real device's kernel DMA buffers - the modelled AXI datamover and
FPGAIntegrationTest require aligned output buffers.
This drops the NUMA/mmap dance from the simulated path (not
performance-critical) - removing libnuma from acquisition_device - and
replaces the base-class cleanup that had to guess the allocation
strategy with a single clear owner per device.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allocate the ring buffer with plain malloc and zero it across
hardware_concurrency() threads, so each page is first-touched - and thus
NUMA-placed - by whichever node the scheduler ran the zeroing thread on.
For the random-access buffer this approximates the previous
numa_alloc_interleaved placement, speeds up the one-time fault-in of the
150-200 GB allocation, and drops the libnuma dependency from this file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA
device, so all per-image preprocessing/spot-finding/azimuthal integration ran on
GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a
process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op
without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before
building the analysis resources so their CUDA streams/engines land on distinct
devices.
Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU
dispatch with goals and a staged plan (multi-broker GPU isolation via
CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a
$<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and
the CUDA block) so -lrt is still passed on Linux - where some distros need it for
clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the
jfjoch_process link line on Linux.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before
slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's
bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single
libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the
full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process).
- common/NetworkAddressConvert.cpp: guard the network includes for _WIN32
(winsock2/ws2tcpip vs arpa/inet).
- common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an
anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) -
portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its
options, output files, and the broker/viewer/process distinction; mention
jfjoch_scale for re-scaling/merging.
- Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env
vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API.
- Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst.
- jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in
usage/license/logger.
- jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D
short flag, make it long-only) and remove dead p/q/i short options.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- App: make getValues an arrow field so its `this` survives being passed
as the `update` prop (the post-edit statistics refresh was broken).
- DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual
so the 1 s statistics poll no longer overwrites a slider mid-edit
(reference `!=` on a freshly-parsed object was always true → snap-back).
- DataCollection: collapse nested setState in grid-scan callbacks.
- PreviewImage: build an immutable settings copy instead of mutating state
in place; fix stale comment.
- MultiLinePlotWrapper: drop per-render console.log; render via a custom
Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only),
cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip).
- package.json: remove dead react-scripts test/eject scripts and CRA
eslintConfig, move @redocly/cli to devDependencies.
- Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts,
setupTests.js).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The description said the "hdf5" sub-object is written under /entry/data, but
the writer (NXmx::UserData) and the example in the same description both use
/entry/user. Correct the path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Motivation no longer frames the format as serial-only: Jungfraujoch serves
both rotation and serial MX, and the two-layout rationale (per-image CXI
spots vs. dataset-wide NXreflections) is presented for both.
- Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an
integrated-format master whose /entry/data/data is a VDS linking back to the
original images (all results, no image copies).
- Add a section on the header_appendix / image_appendix (user_data) mechanism:
how the free-form JSON flows through the start/image CBOR messages and how an
"hdf5" sub-object is persisted to /entry/user, with an example.
- Link the niggliClass row to International Tables for Crystallography A,
Table 3.1.3.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the
writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout,
NXreflections for integration), the master/data-file layout and the three
NXmx format variants, the NXmx-standard fields that are populated, and every
Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint,
/entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel,
detectorSpecific, calibration, fluorescence, user). Content is derived from
writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked
against the NXmx and NXreflections definitions.
JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to
the new doc; HDF5 is added to the Sphinx toctree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the protocol doc in line with CBORStream2Serializer/Deserializer:
- reflection dist_ewald -> rp (dist_ewald is the spot field, not the
reflection field)
- az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count
- threshold_energy typed as Map(string -> float), not float
- end_date typed as string (it is serialized as a plain text string, with
no datetime tag, unlike arm_date)
- add missing reflection fields: phi, zeta, image_scale_corr
- add missing spot fields: image, and indexed-only h, k, l, dist_ewald
- add image indexing_lattice_count and end indexed_lattice_count
- drop the duplicated image_scale_factor row in the End table
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The master-file writer emits /entry/MX/indexedLatticeCount, but the
reader only looked for indexingLatticeCount (the name used by the
per-file MX plugin and data-file read path). Existing master files on
disk therefore returned no indexed-lattice-count vector. Read the
master's indexedLatticeCount first, falling back to indexingLatticeCount,
matching the niggli_class/niggliClass compatibility pattern. The writer
is left unchanged so already-written files keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a
calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN,
which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and
use it directly as I_ref (F-model is squared, intensities are not).
This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional
integration, or PixelRefine against an external reference) to produce a merge,
then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's
own intensities - free of the reference structure's non-isomorphism bias - and
iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine
runs against it (crystal 2, 55279 reflections, CC1/2 94.3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname
Reviewed-on: #61
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial
data. Each image is assigned to one CC1/2 half, so removing it perturbs only that
half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i).
A negative value means dropping the image RAISES CC1/2 (it disagrees with the
consensus). Images whose deltaCChalf is a low-side statistical outlier
(< mean - N*stddev) are skipped when merging. Reference-free.
Two passes over the retained integration outcomes; per-image contributions are
re-derived rather than stored, so memory stays O(unique reflections + images) for
full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off).
Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2
95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS
heterogeneous serial data (some crystals genuinely bad) but slightly trims a
homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative
cut still removes the tail), so it is opt-in. R-free is the real test (user's full
200k). Note: the reported overall N_obs still counts all observations; the exported
merge (and CC1/2) correctly exclude the rejected images.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-direction peak search took argmax|spec|. The projected-spot histogram has a
broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can
exceed the true lattice peaks, so on weak / pink-beam frames every direction reported
a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate
cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors,
the true axes entirely absent.)
Subtract a running-mean background of half-width ~15 A and pick the peak by its
PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp
lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes
win. The prominence is also reported as the magnitude, so FilterFFTResults ranks
directions by real-peak strength rather than envelope. Ported identically to CPU
(prefix-sum window) and GPU (sliding-window in the kernel).
Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX
27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged).
The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes
the envelope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d
convention) while the bin width and spot coordinates use the internal 1/d convention -
a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the
same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x
zero-padding of the histogram.
Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an
explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d
is named one_over_d (never q). histogram_size is numerically identical to before, so
behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented
that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi
to a nearby integer shifts indexing ~0.5-1%).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
