1183 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	a7c77f1f08	internal generator: compare raw image width to RAW_MODULE_COLS ... The raw stacked-module layout check compared the image width (cols) against RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 16:09:05 +02:00
leonarski_fandClaude Opus 4.8	0f5271f14c	mask: accept any-bit-depth TIFF for user mask upload ... PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks exported by tools like PyFAI (8-bit) failed. Route the upload through the universal ReadTIFF reader and let PixelMask take a CompressedImage directly: it validates the 2D shape against the detector's converted/raw layouts, binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects float/multi-channel images. Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over ReadTIFF, and the now-unused ReadTIFFFromString32 is removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 16:05:04 +02:00
leonarski_fandClaude Opus 4.8	57ae145555	writer: drop unused <sys/stat.h> include from HDF5DataFile ... Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies on <filesystem>. Removing the dead POSIX include also avoids a needless non-portable header for the eventual MSVC build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 15:02:39 +02:00
leonarski_fandClaude Opus 4.8	bee35f7a25	frame_serialize: write CBOR length big-endian without bswap intrinsic ... AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length, which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN. That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the swap and emitted a little-endian length on a little-endian host -> corrupt frames. Write the 8 length bytes most-significant-first directly. This is big-endian by construction on any host, needs no byte-swap intrinsic and no endianness macro, and is byte-for-byte identical to the previous output on little-endian hosts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 15:02:39 +02:00
leonarski_fandClaude Opus 4.8	5d9e1be814	portability: replace M_PI / std::numbers::pi with a host+device-safe PI ... M_PI is a POSIX <math.h> extension that MSVC does not define without _USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several common/ headers are pulled into .cu device translation units, so std::numbers is not available there. Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works in host code (including MSVC), in CUDA device code, and under C++17/20, and use it everywhere: - common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files - broker/OpenAPIConvert.cpp - viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent convention across the codebase Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:54:01 +02:00
leonarski_f	d0a1175c65	Clean-up of OpenAPI code	2026-06-18 14:29:53 +02:00
leonarski_fandClaude Opus 4.8	1558dddbb8	indexing: make the resolved-algorithm invariant explicit in the pool ... IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(), which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None; the pool has no policy to resolve Auto itself. The worker handled a stray Auto with a dead branch and silently produced no result when the resolved algorithm had no matching indexer built. - Document on GetIndexingAlgorithm() that it never returns Auto. - Throw a clear internal error at the pool boundary if Auto ever arrives. - In the worker, replace the dead Auto branch with a loud failure for any resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a host without a GPU), instead of returning no result silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_fandClaude Opus 4.8	01a97d61fe	common: spell the azimuthal d-spacing unit d_A consistently ... PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is left as-is (controlled by the OpenAPI generator's C++ enum casing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_fandClaude Opus 4.8	52f349b9f1	frame_serialize: use reentrant gmtime in CBOR date decoding ... gmtime() returns a pointer to a shared static buffer, so concurrent deserialization of CborUnixTime_tTag values was a data race. Decode into a local struct tm via gmtime_r (gmtime_s on Windows) instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_fandClaude Opus 4.8	086129f767	receiver: count only images the pusher accepted ... images_sent was incremented right after image_pusher.SendImage(*loc), but the ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it silently drops the image (releases the slot and returns) when there is no live connection or the 2 s enqueue deadline expires. So images_sent over-counted on a broken/slow writer connection and disagreed with the ACK-based GetImagesWritten(). Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted (enqueued/handed off) and only increment images_sent on success. The slot is still released on the drop path. The authoritative delivered count remains GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on accept, preserving their previous always-counted behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:24:48 +02:00
leonarski_fandClaude Opus 4.8	e1622c3750	plots: expose compression_ratio over the REST/UI plot path ... The receiver already computes and stores the compression ratio per image (JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio), but the type was unreachable from the API: it was missing from ConvertPlotType, the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and no UI menu entry existed. Wire it through all layers: - broker: add the compression_ratio -> PlotType::CompressionRatio case in ConvertPlotType. - spec: add compression_ratio to the plot_type parameter enum. - frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what regeneration would produce), a "Compression ratio" menu entry, and a Y-axis label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:52:41 +02:00
leonarski_fandClaude Opus 4.8	ec1308f4a9	viewer: guard Linux-only constructs for Windows/MSVC builds ... Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux build: D-Bus stays on by default and the XCB plugin is still used. - CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/ adaptor + service file only when enabled. Select the platform integration plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead of hard-coding QXcbIntegrationPlugin. - JFJochViewerWindow: wrap the adaptor include and D-Bus registration in #ifdef JFJOCH_VIEWER_DBUS. - JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now #ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only headers are guarded too. - Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the viewer (not defined by MSVC without _USE_MATH_DEFINES). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:50:49 +02:00
leonarski_fandClaude Opus 4.8	b29b870ef2	broker/gen: gitignore generated cpp-pistache-server output ... Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is hand-written on cpp-httplib, so the generated Pistache server stubs (api/, impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are regeneration debris that was previously left untracked and tended to carry a newer spec version than the committed models. Ignore it so it can't be committed by accident. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:45:04 +02:00
leonarski_fandClaude Opus 4.8	56ddfaef96	viewer: live detector status bar + dataset-follow sync mode ... Add a status-bar cluster, shown when connected over HTTP, that surfaces the broker state and live acquisition info: - broker state box (QProgressBar) with progress drawn as the bar fill, polled ~1 Hz on its own timer independent of image sync - Live/Disconnected connection badge (host:port in tooltip) - "+N new" badge and effective live-rate (Hz) readout All widgets are fixed-width and only blanked in place, so the bar never reflows when things appear/disappear. Add a third autoload mode (HTTPSyncDataset): manually selecting an image while following live now freezes the displayed image but keeps the dataset, plots and image count updating. Surfaced via a tri-state HTTP Sync button (off / live / data-only). Also fix GetBrokerStatus() dropping the message field and a couple of copy-pasted exception strings in JFJochHttpReader. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 08:13:07 +02:00
leonarski_fandClaude Opus 4.8	188cbb659d	tests: add GPU/CPU equivalence test for ROI integration ... Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks), negative pixel values (signed weighted-sum path), and an injected saturated and masked pixel per ROI to cover the "max only" and "fully excluded" branches. Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present, mirroring ImageSpotFinderGPUTest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 22:28:42 +02:00
leonarski_fandClaude Opus 4.8	c0a4801bc4	TCPStreamPusher: fix zero-copy buffer reuse and make send/END timeouts progress-based ... Three robustness fixes for the writer-facing TCP stream, addressing the spurious "wrong number of images" / connection failures seen under load. 1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path passes a caller-owned buffer with z == nullptr and reuses it for the next frame immediately. With MSG_ZEROCOPY the kernel still references that buffer after send() returns, so the peer could receive corrupted frames and drop the connection mid-stream, truncating the run. Zero-copy is now gated on a ZeroCopyReturnValue that keeps the buffer alive until completion. 2. Make the SendAll watchdog measure lack of progress, not total wall-clock. The previous absolute deadline tore down a healthy but back-pressured connection (slow/starved writer) after a fixed window; the watchdog now resets on every byte actually sent, so only a genuinely stuck socket is closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR. 3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still be draining a backlog of DATA frames when END is sent; each DATA ACK is progress, so the timeout only arms once the writer falls silent rather than firing while images are legitimately still being drained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 22:02:09 +02:00
leonarski_fandClaude Opus 4.8	58910274bf	image_analysis: compute ROI statistics in the non-FPGA path ... MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were only available on the FPGA path. Add a software ROI engine that mirrors the FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so it can contribute to any subset of up to 16 ROIs. New image_analysis/roi/ library (JFJochROIIntegration), structured like azint: a base that precomputes the per-pixel mask and names, a templated CPU engine (generic over pixel type for a future 16-bit path), and a GPU kernel using per-block shared-memory atomics for the STXM case (half-detector ROIs). Masked pixels are excluded entirely; saturated pixels are excluded from the sums but still count towards the max, matching roi_calc exactly. The engine is only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already consume message.roi, so no further changes are needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 21:15:54 +02:00
leonarski_fandClaude Opus 4.8	cc925b2668	Remove NUMAHWPolicy and the libnuma dependency ... NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are placed device-local by the kernel (dma_alloc_coherent), the big RAM ring buffer is random-access (first-touch handles placement), and GPU work is already spread across all visible devices. So drop the pinning entirely and with it libnuma. - Delete NUMAHWPolicy; the only concern worth keeping - GPU selection - is done directly via pin_gpu() (round-robin over visible GPUs) in the indexer pool and the Lite analysis threads. CPU-only threads (FPGA acquire/pedestal/summation/frame-transform) no longer bind anything. - Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA used it. - numa_policy broker setting is deprecated and ignored (kept in the API for backward compatibility; warns once on startup). - Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake. - Docs: drop the NUMA dependency, remove the numa_policy config example, and document running multiple brokers on disjoint GPUs via CUDA_VISIBLE_DEVICES. - Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 20:25:36 +02:00
leonarski_fandClaude Opus 4.8	17732ea774	docs: refresh NUMA_GPU_REVIEW to match implemented state ... Several items in the note had landed or were inaccurate. Update it to: - mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device de-NUMA, CUDAWrapper sysfs node lookup; - add the previously-missing FPGA DMA buffer section - placement is a kernel concern (dma_alloc_coherent, device-local), not the userspace mbind, which was only the simulator; - record that libnuma is now down to a single file (NUMAHWPolicy.cpp); - note that dependency removal (G3) and pinning behaviour (G4) are separable axes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:22:00 +02:00
leonarski_fandClaude Opus 4.8	040cdeacf1	acquisition_device: give each device sole ownership of its buffers ... The base AcquisitionDevice no longer allocates or frees frame buffers; buffer_device is now just a non-owning view of addresses. Each subclass owns its backing memory and the matching lifecycle: - PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its own destructor (and on ctor failure), symmetric with MapKernelBuffer. - HLSSimulatedDevice owns plain zeroed heap buffers it points buffer_device into, declared before the HLSDevice so they outlive the action thread that writes them. The buffers are page-aligned to match the real device's kernel DMA buffers - the modelled AXI datamover and FPGAIntegrationTest require aligned output buffers. This drops the NUMA/mmap dance from the simulated path (not performance-critical) - removing libnuma from acquisition_device - and replaces the base-class cleanup that had to guess the allocation strategy with a single clear owner per device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:18:32 +02:00
leonarski_fandClaude Opus 4.8	d373ba0490	ImageBuffer: replace NUMA interleave with parallel first-touch ... Allocate the ring buffer with plain malloc and zero it across hardware_concurrency() threads, so each page is first-touched - and thus NUMA-placed - by whichever node the scheduler ran the zeroing thread on. For the random-access buffer this approximates the previous numa_alloc_interleaved placement, speeds up the one-time fault-in of the 150-200 GB allocation, and drops the libnuma dependency from this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 17:03:13 +02:00
leonarski_fandClaude Opus 4.8	02fa15c2b9	jfjoch_process: spread per-image GPU work across all visible GPUs ... The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA device, so all per-image preprocessing/spot-finding/azimuthal integration ran on GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before building the analysis resources so their CUDA streams/engines land on distinct devices. Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU dispatch with goals and a staged plan (multi-broker GPU isolation via CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:29:52 +02:00
leonarski_fandClaude Opus 4.8	1056acc3a6	build: guard the librt link to Linux only ... librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a $<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and the CUDA block) so -lrt is still passed on Linux - where some distros need it for clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the jfjoch_process link line on Linux. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:40:09 +02:00
leonarski_fandClaude Opus 4.8	39c808776f	build: portability groundwork toward a Windows/MSVC viewer ... - CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process). - common/NetworkAddressConvert.cpp: guard the network includes for _WIN32 (winsock2/ws2tcpip vs arpa/inet). - common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) - portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:37:21 +02:00
leonarski_fandClaude Opus 4.8	c69b5297d5	docs: add jfjoch_process page, refresh viewer/tools docs, unify CLI naming ... - Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its options, output files, and the broker/viewer/process distinction; mention jfjoch_scale for re-scaling/merging. - Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API. - Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst. - jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in usage/license/logger. - jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D short flag, make it long-only) and remove dead p/q/i short options. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 07:38:36 +02:00
leonarski_fandClaude Opus 4.8	3334e88d3e	frontend: fix state bugs, slim Plotly bundle, drop CRA leftovers ... - App: make getValues an arrow field so its `this` survives being passed as the `update` prop (the post-edit statistics refresh was broken). - DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual so the 1 s statistics poll no longer overwrites a slider mid-edit (reference `!=` on a freshly-parsed object was always true → snap-back). - DataCollection: collapse nested setState in grid-scan callbacks. - PreviewImage: build an immutable settings copy instead of mutating state in place; fix stale comment. - MultiLinePlotWrapper: drop per-render console.log; render via a custom Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only), cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip). - package.json: remove dead react-scripts test/eject scripts and CRA eslintConfig, move @redocly/cli to devDependencies. - Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts, setupTests.js). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:51:45 +02:00
leonarski_f	638aa379f4	Fix	2026-06-16 20:28:25 +02:00
leonarski_f	704e4b4eb3	Documentation: Data analysis description	2026-06-16 20:24:01 +02:00
leonarski_fandClaude Opus 4.8	5143ba0b1f	jfjoch_api.yaml: header_appendix hdf5 values go to /entry/user, not /entry/data ... The description said the "hdf5" sub-object is written under /entry/data, but the writer (NXmx::UserData) and the example in the same description both use /entry/user. Correct the path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:00:40 +02:00
leonarski_fandClaude Opus 4.8	e8d3eb1b08	docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link ... - Motivation no longer frames the format as serial-only: Jungfraujoch serves both rotation and serial MX, and the two-layout rationale (per-image CXI spots vs. dataset-wide NXreflections) is presented for both. - Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an integrated-format master whose /entry/data/data is a VDS linking back to the original images (all results, no image copies). - Add a section on the header_appendix / image_appendix (user_data) mechanism: how the free-form JSON flows through the start/image CBOR messages and how an "hdf5" sub-object is persisted to /entry/user, with an example. - Link the niggliClass row to International Tables for Crystallography A, Table 3.1.3.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:47:36 +02:00
leonarski_fandClaude Opus 4.8	0381d891bc	docs: add HDF5.md describing the NeXus layout and JFJoch extensions ... New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout, NXreflections for integration), the master/data-file layout and the three NXmx format variants, the NXmx-standard fields that are populated, and every Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint, /entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel, detectorSpecific, calibration, fluorescence, user). Content is derived from writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked against the NXmx and NXreflections definitions. JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to the new doc; HDF5 is added to the Sphinx toctree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:36:44 +02:00
leonarski_fandClaude Opus 4.8	caff857d8a	docs/CBOR.md: sync field names/types with the serializer ... Bring the protocol doc in line with CBORStream2Serializer/Deserializer: - reflection dist_ewald -> rp (dist_ewald is the spot field, not the reflection field) - az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count - threshold_energy typed as Map(string -> float), not float - end_date typed as string (it is serialized as a plain text string, with no datetime tag, unlike arm_date) - add missing reflection fields: phi, zeta, image_scale_corr - add missing spot fields: image, and indexed-only h, k, l, dist_ewald - add image indexing_lattice_count and end indexed_lattice_count - drop the duplicated image_scale_factor row in the End table Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:11:37 +02:00
leonarski_fandClaude Opus 4.8	6479b91e50	JFJochHDF5Reader: read master indexedLatticeCount (back-compat) ... The master-file writer emits /entry/MX/indexedLatticeCount, but the reader only looked for indexingLatticeCount (the name used by the per-file MX plugin and data-file read path). Existing master files on disk therefore returned no indexed-lattice-count vector. Read the master's indexedLatticeCount first, falling back to indexingLatticeCount, matching the niggli_class/niggliClass compatibility pattern. The writer is left unchanged so already-written files keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:03:34 +02:00
leonarski_fandClaude Opus 4.8	2ba28aea0e	LoadFCalcFromMtz: fall back to an intensity column when F-model is absent ... PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN, which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and use it directly as I_ref (F-model is squared, intensities are not). This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional integration, or PixelRefine against an external reference) to produce a merge, then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's own intensities - free of the reference structure's non-isomorphism bias - and iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine runs against it (crystal 2, 55279 reflections, CC1/2 94.3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 14:19:28 +02:00
leonarski_fandClaude Opus 4.8	545ebdf868	Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial data. Each image is assigned to one CC1/2 half, so removing it perturbs only that half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i). A negative value means dropping the image RAISES CC1/2 (it disagrees with the consensus). Images whose deltaCChalf is a low-side statistical outlier (< mean - N*stddev) are skipped when merging. Reference-free. Two passes over the retained integration outcomes; per-image contributions are re-derived rather than stored, so memory stays O(unique reflections + images) for full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off). Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2 95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS heterogeneous serial data (some crystals genuinely bad) but slightly trims a homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative cut still removes the tail), so it is opt-in. R-free is the real test (user's full 200k). Note: the reported overall N_obs still counts all observations; the exported merge (and CC1/2) correctly exclude the rejected images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 12:32:52 +02:00
leonarski_f	f878fb9d5d	SpotFindingSettings: Default signal to noise ratio is 4	2026-06-16 08:53:05 +02:00
leonarski_fandClaude Opus 4.8	929e6e1fa0	FFTIndexer: pick peaks by prominence above a local background, not raw magnitude ... The per-direction peak search took argmax|spec|. The projected-spot histogram has a broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can exceed the true lattice peaks, so on weak / pink-beam frames every direction reported a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors, the true axes entirely absent.) Subtract a running-mean background of half-width ~15 A and pick the peak by its PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes win. The prominence is also reported as the magnitude, so FilterFFTResults ranks directions by real-peak strength rather than envelope. Ported identically to CPU (prefix-sum window) and GPU (sliding-window in the kernel). Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX 27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged). The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes the envelope). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:25:06 +02:00
leonarski_fandClaude Opus 4.8	f2f95c44f6	FFTIndexer: name the histogram axis one_over_d (1/d), not q/Q ... The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d convention) while the bin width and spot coordinates use the internal 1/d convention - a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x zero-padding of the histogram. Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d is named one_over_d (never q). histogram_size is numerically identical to before, so behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi to a nearby integer shifts indexing ~0.5-1%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:24:59 +02:00
leonarski_fandClaude Opus 4.8	e1e2ca8e49	IndexAndRefine: unify cell setting across indexers, keep FFBIDX neutral ... Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when cell+SG are given, so every frame reports its cell in ONE consistent setting per space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same reflection as different HKLs and cannot be merged). Use LatticeSearch's conventional cell when its detected symmetry agrees with the user's space group. On noisy frames LatticeSearch can pick an alternative Bravais setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell, [78,78,38]->[110,111,38]) whose system disagrees; there: - FFBIDX already returns the reference setting (c-last), consistent with the conventional frames, so its raw lattice is safe -> use it. This recovers the ~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral. - de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order (c-first); merging it with the conventional (c-last) frames would corrupt the data -> reject the frame instead. Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last, consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis (~38A) is axis 2 in every merged frame - no axis-order mixing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:50:56 +02:00
leonarski_fandClaude Opus 4.8	40da1aab13	IndexAndRefine: conventionalise de-novo lattice when cell+SG are given ... Providing both a unit cell (-C) and space group (-S) silently broke the de-novo indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the `sg && GetUnitCell()` branch fed the indexer's raw lattice straight into symmetry-constrained refinement. FFBIDX returns the lattice already in the reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive cell; enforcing the crystal system on its mis-assigned axes rejects every frame. Fix: for de-novo indexers only, reduce the lattice to the conventional setting (LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so it is byte-identical to before (no regression). niggli_class stays unassigned (0) in this branch - it is a property of the primitive cell incl. centering, which LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2) cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred. Validation (lyso, -C79,79,38 -S96): crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical (Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no consensus on the weak serial-still frames, 0% even without -S; to investigate.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:35:03 +02:00
leonarski_fandClaude Opus 4.8	ecdb7048a0	PixelRefine: drop crystal-system idealisation, use the indexed cell as-is ... PixelRefine no longer refines the cell, so the symmetry-constrained cell parameterisation inherited from XtalOptimizer has no manifold to constrain. The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B matrix) merely reconstructed the lattice columns it started from, and for non-triclinic systems it re-idealised the indexed cell (averaging a,b for tetragonal; a,b,c for cubic; forcing alpha=gamma=90). Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a single path: take the three real-space lattice columns (latt.Vec0/1/2()) directly and form the reciprocal node via the general cross-product inverse. This reproduces every crystal system exactly from the actual cell, is more faithful (no re-idealisation), and removes the crystal_system field plus two now-unused includes. PredictedNode de-templated (only ever called with double). Crystal symmetry still lives where it belongs: indexing (upstream) and merging (downstream via the space-group HKL key). A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the old idealisation was already a no-op): stat tables bit-identical across all shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% -- the only delta is 4th-digit float-ordering noise in the error-model b coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free unchanged. Net -81 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 14:11:26 +02:00
leonarski_fandClaude Opus 4.8	8a9d80eb71	PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... (1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process --profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:48:11 +02:00
leonarski_fandClaude Opus 4.8	102a2a7c81	PixelRefine: strip experimental env knobs (orientation, sweep, Lorentz, ML census) ... Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed, returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper (r1_multiplier, default 6): - PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) - XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual, OrientationRegularizer, PixelObs::weight, the refinement block and its fields. - PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2 (per-image overfit). Removes SweepOrientationCell and its fields. - PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality already subsumes it); was already reverted. - PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5% of jet frames are multi-lattice; shelved). PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged: crystal 2 / jet / hybrid -R -r pixelrefine all reproduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:34:54 +02:00
leonarski_fandClaude Opus 4.8	6f733d74c2	Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... Promote the per-observation merge outlier rejection from the temporary PR_REJECT env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0 disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 -> 0.2585). Applies to both the PixelRefine and classical merge paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 12:42:25 +02:00
leonarski_fandClaude Opus 4.8	a47b376dc3	Merge: per-observation outlier rejection (env-gated PR_REJECT) ... At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every merged mean. Add a robust per-observation cut: drop observations whose corrected intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN. The error-model sigma already captures the genuine (partiality) scatter, and the median is a robust centre, so only the tail beyond the real spread is removed - not good partials. The median is computed in RefineErrorModel (which already pools the observations per reflection); AddImage applies the cut. Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut) - the cut is vs the data's own median, not the reference, so the gain is real cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at low nsigma) are for Filip's full-jet R-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:56:53 +02:00
leonarski_fandClaude Opus 4.8	45ee8c2b40	PixelRefine: env-gated orientation + cell-scale sweep (PR_SWEEP) for R-free A/B ... R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on R-free; x9 only buys CC1/2 internal consistency), and per-image orientation *refinement* is a no-op (0.2618 vs 0.2625). Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP (off by default) to test whether the cell-scale degree of freedom - which the gradient orientation refinement lacks, and which moves the high-res shells radially - helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for the gradient path), but that does not certify R-free, so it is left for validation. NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 19:59:50 +02:00
leonarski_fandClaude Opus 4.8	cfcd4c9e56	PixelRefine: R1 multiplier (fix tight-Term2 regression) + env-gated orientation refine ... Full-jet R-free showed the factored model regressed vs the traditional pipeline (R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix, XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier (default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3 (~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free. Also reinstates the pre-factored per-image orientation refinement (per-pixel ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation, re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation. NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier value and the orientation-refine decision are to be fixed against R-free, then baked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 09:18:09 +02:00
leonarski_fandClaude Opus 4.8	bf6efc7fe9	Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet (1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots spill past a radius-3 box), is neutral on the mono crystal and on the classical integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob. (Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res, azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured 2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight measured width loses to a generous aperture, same lesson as the radius bump.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 23:27:36 +02:00
leonarski_fandClaude Opus 4.8	100fe7b7e7	PixelRefine: make factored Terms 1+2 the model, remove old wiring ... PixelRefine is now an intensity-only operation: geometry is fixed (refined upstream by XtalOptimizer) and the only objective is the factored per-reflection likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills, 1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and matches the rotation path from the stills path. Removed: - the per-pixel ShoeboxResidual loss and PixelResidual cost functor; - all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R), the regularised-orientation LSQ, signal-weighting, and the global sweep; - Term 3 (per-spot recentring) - a confirmed no-op on both crystals; - the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the PR_* env knobs + stderr dumps in IndexAndRefine; - the PredictImage/ChiSquaredImage renderers and the entire viewer PixelRefine window/table/params + worker bindings + shoebox overlay. The sweep box-integrator background median became mean (consistency) by virtue of removing the sweep. METHODS.md rewritten for the current model; findings recorded in FINDINGS-2026-06.md. Net -2200 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 22:02:18 +02:00
leonarski_fandClaude Opus 4.8	6f2033db00	PixelRefine: checkpoint before cleanup (factored model + all diagnostic levers) ... Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_* env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/ ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit makes Terms 1+2 the model and strips all of this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 21:34:33 +02:00