1193 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	237de5a0fc	frontend: convert to function components + TanStack Query ... Replace all class components with function components and move server-state polling to @tanstack/react-query. - add @tanstack/react-query; generate query helpers via the hey-api @tanstack/react-query plugin (src/client/@tanstack) - QueryClientProvider + one-time client.setConfig({ baseUrl: '' }) in index.tsx - App polls statistics with useQuery(getStatisticsOptions, refetchInterval), DataProcessingPlot uses getPreviewPlotOptions; manual setInterval polling gone - memo() on presentational children; with TanStack structural sharing this re-renders a child only when its own statistics slice changes - PreviewImage stays imperative (binary JPEG -> object URL) using useEffect + a ref for the object-URL lifecycle - fix AzIntSettings correction checkboxes that mutated state in place (relied on the poll re-render); they now use setState - drop dead code uncovered during the port (unused upload/deactivate handlers and imports in ImageFormatSettings, DetectorSettings, ROI) Build (tsc + vite) passes; dev server transforms all entry modules. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 21:32:36 +02:00
leonarski_f	7ed424a4ca	JFJochReceiverFPGA: Fix typo	2026-06-18 20:48:19 +02:00
leonarski_fandClaude Opus 4.8	ceb7d31eac	frontend: generate OpenAPI client with @hey-api/openapi-ts ... Replace the unmaintained openapi-typescript-codegen generator with @hey-api/openapi-ts. - openapi-ts.config.ts generates src/client from ../broker/jfjoch_api.yaml (client-fetch + typescript + sdk plugins; enums as const objects via mode: 'javascript', type names preserved) - npm "openapi" script now runs openapi-ts; drop openapi-typescript-codegen - migrate all imports from src/openapi to src/client - DefaultService.foo() -> standalone foo({ body|query, throwOnError }) - OpenAPI.BASE='' -> client.setConfig({ baseUrl: '' }); ButtonWithSnackbar uses the relative path directly - adapt hey-api's inlined enums: broker_status.state/message_severity and bit_depth_image/eiger_bit_depth become string/number literal unions; file_writer_format NXMX_* -> N_XMX_* - remove generated src/openapi Build (tsc + vite) passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 20:30:15 +02:00
leonarski_f	eb9b463263	VERSION: 1.0.0-rc.153	2026-06-18 19:26:15 +02:00
leonarski_f	b71da6238e	Clean-up unused tools	2026-06-18 18:54:31 +02:00
leonarski_f	d14b8e9e18	Merge upstream	2026-06-18 18:52:22 +02:00
leonarski_fandClaude Opus 4.8	a7c77f1f08	internal generator: compare raw image width to RAW_MODULE_COLS ... The raw stacked-module layout check compared the image width (cols) against RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 16:09:05 +02:00
leonarski_fandClaude Opus 4.8	0f5271f14c	mask: accept any-bit-depth TIFF for user mask upload ... PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks exported by tools like PyFAI (8-bit) failed. Route the upload through the universal ReadTIFF reader and let PixelMask take a CompressedImage directly: it validates the 2D shape against the detector's converted/raw layouts, binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects float/multi-channel images. Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over ReadTIFF, and the now-unused ReadTIFFFromString32 is removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 16:05:04 +02:00
leonarski_fandClaude Opus 4.8	57ae145555	writer: drop unused <sys/stat.h> include from HDF5DataFile ... Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies on <filesystem>. Removing the dead POSIX include also avoids a needless non-portable header for the eventual MSVC build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 15:02:39 +02:00
leonarski_fandClaude Opus 4.8	bee35f7a25	frame_serialize: write CBOR length big-endian without bswap intrinsic ... AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length, which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN. That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the swap and emitted a little-endian length on a little-endian host -> corrupt frames. Write the 8 length bytes most-significant-first directly. This is big-endian by construction on any host, needs no byte-swap intrinsic and no endianness macro, and is byte-for-byte identical to the previous output on little-endian hosts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 15:02:39 +02:00
leonarski_fandClaude Opus 4.8	5d9e1be814	portability: replace M_PI / std::numbers::pi with a host+device-safe PI ... M_PI is a POSIX <math.h> extension that MSVC does not define without _USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several common/ headers are pulled into .cu device translation units, so std::numbers is not available there. Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works in host code (including MSVC), in CUDA device code, and under C++17/20, and use it everywhere: - common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files - broker/OpenAPIConvert.cpp - viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent convention across the codebase Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:54:01 +02:00
leonarski_f	d0a1175c65	Clean-up of OpenAPI code	2026-06-18 14:29:53 +02:00
leonarski_fandClaude Opus 4.8	1558dddbb8	indexing: make the resolved-algorithm invariant explicit in the pool ... IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(), which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None; the pool has no policy to resolve Auto itself. The worker handled a stray Auto with a dead branch and silently produced no result when the resolved algorithm had no matching indexer built. - Document on GetIndexingAlgorithm() that it never returns Auto. - Throw a clear internal error at the pool boundary if Auto ever arrives. - In the worker, replace the dead Auto branch with a loud failure for any resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a host without a GPU), instead of returning no result silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_fandClaude Opus 4.8	01a97d61fe	common: spell the azimuthal d-spacing unit d_A consistently ... PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is left as-is (controlled by the OpenAPI generator's C++ enum casing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_fandClaude Opus 4.8	52f349b9f1	frame_serialize: use reentrant gmtime in CBOR date decoding ... gmtime() returns a pointer to a shared static buffer, so concurrent deserialization of CborUnixTime_tTag values was a data race. Decode into a local struct tm via gmtime_r (gmtime_s on Windows) instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_fandClaude Opus 4.8	086129f767	receiver: count only images the pusher accepted ... images_sent was incremented right after image_pusher.SendImage(*loc), but the ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it silently drops the image (releases the slot and returns) when there is no live connection or the 2 s enqueue deadline expires. So images_sent over-counted on a broken/slow writer connection and disagreed with the ACK-based GetImagesWritten(). Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted (enqueued/handed off) and only increment images_sent on success. The slot is still released on the drop path. The authoritative delivered count remains GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on accept, preserving their previous always-counted behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:24:48 +02:00
leonarski_fandClaude Opus 4.8	e1622c3750	plots: expose compression_ratio over the REST/UI plot path ... The receiver already computes and stores the compression ratio per image (JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio), but the type was unreachable from the API: it was missing from ConvertPlotType, the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and no UI menu entry existed. Wire it through all layers: - broker: add the compression_ratio -> PlotType::CompressionRatio case in ConvertPlotType. - spec: add compression_ratio to the plot_type parameter enum. - frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what regeneration would produce), a "Compression ratio" menu entry, and a Y-axis label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:52:41 +02:00
leonarski_fandClaude Opus 4.8	ec1308f4a9	viewer: guard Linux-only constructs for Windows/MSVC builds ... Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux build: D-Bus stays on by default and the XCB plugin is still used. - CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/ adaptor + service file only when enabled. Select the platform integration plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead of hard-coding QXcbIntegrationPlugin. - JFJochViewerWindow: wrap the adaptor include and D-Bus registration in #ifdef JFJOCH_VIEWER_DBUS. - JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now #ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only headers are guarded too. - Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the viewer (not defined by MSVC without _USE_MATH_DEFINES). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:50:49 +02:00
leonarski_fandClaude Opus 4.8	b29b870ef2	broker/gen: gitignore generated cpp-pistache-server output ... Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is hand-written on cpp-httplib, so the generated Pistache server stubs (api/, impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are regeneration debris that was previously left untracked and tended to carry a newer spec version than the committed models. Ignore it so it can't be committed by accident. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:45:04 +02:00
leonarski_fandClaude Opus 4.8	56ddfaef96	viewer: live detector status bar + dataset-follow sync mode ... Add a status-bar cluster, shown when connected over HTTP, that surfaces the broker state and live acquisition info: - broker state box (QProgressBar) with progress drawn as the bar fill, polled ~1 Hz on its own timer independent of image sync - Live/Disconnected connection badge (host:port in tooltip) - "+N new" badge and effective live-rate (Hz) readout All widgets are fixed-width and only blanked in place, so the bar never reflows when things appear/disappear. Add a third autoload mode (HTTPSyncDataset): manually selecting an image while following live now freezes the displayed image but keeps the dataset, plots and image count updating. Surfaced via a tri-state HTTP Sync button (off / live / data-only). Also fix GetBrokerStatus() dropping the message field and a couple of copy-pasted exception strings in JFJochHttpReader. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 08:13:07 +02:00
leonarski_fandClaude Opus 4.8	188cbb659d	tests: add GPU/CPU equivalence test for ROI integration ... Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks), negative pixel values (signed weighted-sum path), and an injected saturated and masked pixel per ROI to cover the "max only" and "fully excluded" branches. Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present, mirroring ImageSpotFinderGPUTest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 22:28:42 +02:00
leonarski_fandClaude Opus 4.8	c0a4801bc4	TCPStreamPusher: fix zero-copy buffer reuse and make send/END timeouts progress-based ... Three robustness fixes for the writer-facing TCP stream, addressing the spurious "wrong number of images" / connection failures seen under load. 1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path passes a caller-owned buffer with z == nullptr and reuses it for the next frame immediately. With MSG_ZEROCOPY the kernel still references that buffer after send() returns, so the peer could receive corrupted frames and drop the connection mid-stream, truncating the run. Zero-copy is now gated on a ZeroCopyReturnValue that keeps the buffer alive until completion. 2. Make the SendAll watchdog measure lack of progress, not total wall-clock. The previous absolute deadline tore down a healthy but back-pressured connection (slow/starved writer) after a fixed window; the watchdog now resets on every byte actually sent, so only a genuinely stuck socket is closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR. 3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still be draining a backlog of DATA frames when END is sent; each DATA ACK is progress, so the timeout only arms once the writer falls silent rather than firing while images are legitimately still being drained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 22:02:09 +02:00
leonarski_fandClaude Opus 4.8	58910274bf	image_analysis: compute ROI statistics in the non-FPGA path ... MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were only available on the FPGA path. Add a software ROI engine that mirrors the FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so it can contribute to any subset of up to 16 ROIs. New image_analysis/roi/ library (JFJochROIIntegration), structured like azint: a base that precomputes the per-pixel mask and names, a templated CPU engine (generic over pixel type for a future 16-bit path), and a GPU kernel using per-block shared-memory atomics for the STXM case (half-detector ROIs). Masked pixels are excluded entirely; saturated pixels are excluded from the sums but still count towards the max, matching roi_calc exactly. The engine is only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already consume message.roi, so no further changes are needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 21:15:54 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62 1.0.0-rc.152	2026-06-17 20:36:24 +02:00
leonarski_fandClaude Opus 4.8	cc925b2668	Remove NUMAHWPolicy and the libnuma dependency ... NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are placed device-local by the kernel (dma_alloc_coherent), the big RAM ring buffer is random-access (first-touch handles placement), and GPU work is already spread across all visible devices. So drop the pinning entirely and with it libnuma. - Delete NUMAHWPolicy; the only concern worth keeping - GPU selection - is done directly via pin_gpu() (round-robin over visible GPUs) in the indexer pool and the Lite analysis threads. CPU-only threads (FPGA acquire/pedestal/summation/frame-transform) no longer bind anything. - Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA used it. - numa_policy broker setting is deprecated and ignored (kept in the API for backward compatibility; warns once on startup). - Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake. - Docs: drop the NUMA dependency, remove the numa_policy config example, and document running multiple brokers on disjoint GPUs via CUDA_VISIBLE_DEVICES. - Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 20:25:36 +02:00
leonarski_fandClaude Opus 4.8	17732ea774	docs: refresh NUMA_GPU_REVIEW to match implemented state ... Several items in the note had landed or were inaccurate. Update it to: - mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device de-NUMA, CUDAWrapper sysfs node lookup; - add the previously-missing FPGA DMA buffer section - placement is a kernel concern (dma_alloc_coherent, device-local), not the userspace mbind, which was only the simulator; - record that libnuma is now down to a single file (NUMAHWPolicy.cpp); - note that dependency removal (G3) and pinning behaviour (G4) are separable axes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:22:00 +02:00
leonarski_fandClaude Opus 4.8	040cdeacf1	acquisition_device: give each device sole ownership of its buffers ... The base AcquisitionDevice no longer allocates or frees frame buffers; buffer_device is now just a non-owning view of addresses. Each subclass owns its backing memory and the matching lifecycle: - PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its own destructor (and on ctor failure), symmetric with MapKernelBuffer. - HLSSimulatedDevice owns plain zeroed heap buffers it points buffer_device into, declared before the HLSDevice so they outlive the action thread that writes them. The buffers are page-aligned to match the real device's kernel DMA buffers - the modelled AXI datamover and FPGAIntegrationTest require aligned output buffers. This drops the NUMA/mmap dance from the simulated path (not performance-critical) - removing libnuma from acquisition_device - and replaces the base-class cleanup that had to guess the allocation strategy with a single clear owner per device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:18:32 +02:00
leonarski_fandClaude Opus 4.8	d373ba0490	ImageBuffer: replace NUMA interleave with parallel first-touch ... Allocate the ring buffer with plain malloc and zero it across hardware_concurrency() threads, so each page is first-touched - and thus NUMA-placed - by whichever node the scheduler ran the zeroing thread on. For the random-access buffer this approximates the previous numa_alloc_interleaved placement, speeds up the one-time fault-in of the 150-200 GB allocation, and drops the libnuma dependency from this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 17:03:13 +02:00
leonarski_fandClaude Opus 4.8	02fa15c2b9	jfjoch_process: spread per-image GPU work across all visible GPUs ... The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA device, so all per-image preprocessing/spot-finding/azimuthal integration ran on GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before building the analysis resources so their CUDA streams/engines land on distinct devices. Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU dispatch with goals and a staged plan (multi-broker GPU isolation via CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:29:52 +02:00
leonarski_fandClaude Opus 4.8	1056acc3a6	build: guard the librt link to Linux only ... librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a $<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and the CUDA block) so -lrt is still passed on Linux - where some distros need it for clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the jfjoch_process link line on Linux. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:40:09 +02:00
leonarski_fandClaude Opus 4.8	39c808776f	build: portability groundwork toward a Windows/MSVC viewer ... - CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process). - common/NetworkAddressConvert.cpp: guard the network includes for _WIN32 (winsock2/ws2tcpip vs arpa/inet). - common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) - portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:37:21 +02:00
leonarski_fandClaude Opus 4.8	c69b5297d5	docs: add jfjoch_process page, refresh viewer/tools docs, unify CLI naming ... - Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its options, output files, and the broker/viewer/process distinction; mention jfjoch_scale for re-scaling/merging. - Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API. - Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst. - jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in usage/license/logger. - jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D short flag, make it long-only) and remove dead p/q/i short options. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 07:38:36 +02:00
leonarski_fandClaude Opus 4.8	3334e88d3e	frontend: fix state bugs, slim Plotly bundle, drop CRA leftovers ... - App: make getValues an arrow field so its `this` survives being passed as the `update` prop (the post-edit statistics refresh was broken). - DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual so the 1 s statistics poll no longer overwrites a slider mid-edit (reference `!=` on a freshly-parsed object was always true → snap-back). - DataCollection: collapse nested setState in grid-scan callbacks. - PreviewImage: build an immutable settings copy instead of mutating state in place; fix stale comment. - MultiLinePlotWrapper: drop per-render console.log; render via a custom Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only), cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip). - package.json: remove dead react-scripts test/eject scripts and CRA eslintConfig, move @redocly/cli to devDependencies. - Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts, setupTests.js). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:51:45 +02:00
leonarski_f	638aa379f4	Fix	2026-06-16 20:28:25 +02:00
leonarski_f	704e4b4eb3	Documentation: Data analysis description	2026-06-16 20:24:01 +02:00
leonarski_fandClaude Opus 4.8	5143ba0b1f	jfjoch_api.yaml: header_appendix hdf5 values go to /entry/user, not /entry/data ... The description said the "hdf5" sub-object is written under /entry/data, but the writer (NXmx::UserData) and the example in the same description both use /entry/user. Correct the path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:00:40 +02:00
leonarski_fandClaude Opus 4.8	e8d3eb1b08	docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link ... - Motivation no longer frames the format as serial-only: Jungfraujoch serves both rotation and serial MX, and the two-layout rationale (per-image CXI spots vs. dataset-wide NXreflections) is presented for both. - Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an integrated-format master whose /entry/data/data is a VDS linking back to the original images (all results, no image copies). - Add a section on the header_appendix / image_appendix (user_data) mechanism: how the free-form JSON flows through the start/image CBOR messages and how an "hdf5" sub-object is persisted to /entry/user, with an example. - Link the niggliClass row to International Tables for Crystallography A, Table 3.1.3.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:47:36 +02:00
leonarski_fandClaude Opus 4.8	0381d891bc	docs: add HDF5.md describing the NeXus layout and JFJoch extensions ... New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout, NXreflections for integration), the master/data-file layout and the three NXmx format variants, the NXmx-standard fields that are populated, and every Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint, /entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel, detectorSpecific, calibration, fluorescence, user). Content is derived from writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked against the NXmx and NXreflections definitions. JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to the new doc; HDF5 is added to the Sphinx toctree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:36:44 +02:00
leonarski_fandClaude Opus 4.8	caff857d8a	docs/CBOR.md: sync field names/types with the serializer ... Bring the protocol doc in line with CBORStream2Serializer/Deserializer: - reflection dist_ewald -> rp (dist_ewald is the spot field, not the reflection field) - az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count - threshold_energy typed as Map(string -> float), not float - end_date typed as string (it is serialized as a plain text string, with no datetime tag, unlike arm_date) - add missing reflection fields: phi, zeta, image_scale_corr - add missing spot fields: image, and indexed-only h, k, l, dist_ewald - add image indexing_lattice_count and end indexed_lattice_count - drop the duplicated image_scale_factor row in the End table Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:11:37 +02:00
leonarski_fandClaude Opus 4.8	6479b91e50	JFJochHDF5Reader: read master indexedLatticeCount (back-compat) ... The master-file writer emits /entry/MX/indexedLatticeCount, but the reader only looked for indexingLatticeCount (the name used by the per-file MX plugin and data-file read path). Existing master files on disk therefore returned no indexed-lattice-count vector. Read the master's indexedLatticeCount first, falling back to indexingLatticeCount, matching the niggli_class/niggliClass compatibility pattern. The writer is left unchanged so already-written files keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:03:34 +02:00
leonarski_fandClaude Opus 4.8	2ba28aea0e	LoadFCalcFromMtz: fall back to an intensity column when F-model is absent ... PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN, which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and use it directly as I_ref (F-model is squared, intensities are not). This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional integration, or PixelRefine against an external reference) to produce a merge, then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's own intensities - free of the reference structure's non-isomorphism bias - and iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine runs against it (crystal 2, 55279 reflections, CC1/2 94.3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 14:19:28 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.151	2026-06-16 14:13:29 +02:00
leonarski_fandClaude Opus 4.8	545ebdf868	Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial data. Each image is assigned to one CC1/2 half, so removing it perturbs only that half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i). A negative value means dropping the image RAISES CC1/2 (it disagrees with the consensus). Images whose deltaCChalf is a low-side statistical outlier (< mean - N*stddev) are skipped when merging. Reference-free. Two passes over the retained integration outcomes; per-image contributions are re-derived rather than stored, so memory stays O(unique reflections + images) for full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off). Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2 95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS heterogeneous serial data (some crystals genuinely bad) but slightly trims a homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative cut still removes the tail), so it is opt-in. R-free is the real test (user's full 200k). Note: the reported overall N_obs still counts all observations; the exported merge (and CC1/2) correctly exclude the rejected images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 12:32:52 +02:00
leonarski_f	f878fb9d5d	SpotFindingSettings: Default signal to noise ratio is 4	2026-06-16 08:53:05 +02:00
leonarski_fandClaude Opus 4.8	929e6e1fa0	FFTIndexer: pick peaks by prominence above a local background, not raw magnitude ... The per-direction peak search took argmax|spec|. The projected-spot histogram has a broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can exceed the true lattice peaks, so on weak / pink-beam frames every direction reported a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors, the true axes entirely absent.) Subtract a running-mean background of half-width ~15 A and pick the peak by its PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes win. The prominence is also reported as the magnitude, so FilterFFTResults ranks directions by real-peak strength rather than envelope. Ported identically to CPU (prefix-sum window) and GPU (sliding-window in the kernel). Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX 27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged). The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes the envelope). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:25:06 +02:00
leonarski_fandClaude Opus 4.8	f2f95c44f6	FFTIndexer: name the histogram axis one_over_d (1/d), not q/Q ... The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d convention) while the bin width and spot coordinates use the internal 1/d convention - a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x zero-padding of the histogram. Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d is named one_over_d (never q). histogram_size is numerically identical to before, so behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi to a nearby integer shifts indexing ~0.5-1%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:24:59 +02:00
leonarski_fandClaude Opus 4.8	e1e2ca8e49	IndexAndRefine: unify cell setting across indexers, keep FFBIDX neutral ... Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when cell+SG are given, so every frame reports its cell in ONE consistent setting per space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same reflection as different HKLs and cannot be merged). Use LatticeSearch's conventional cell when its detected symmetry agrees with the user's space group. On noisy frames LatticeSearch can pick an alternative Bravais setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell, [78,78,38]->[110,111,38]) whose system disagrees; there: - FFBIDX already returns the reference setting (c-last), consistent with the conventional frames, so its raw lattice is safe -> use it. This recovers the ~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral. - de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order (c-first); merging it with the conventional (c-last) frames would corrupt the data -> reject the frame instead. Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last, consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis (~38A) is axis 2 in every merged frame - no axis-order mixing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:50:56 +02:00
leonarski_fandClaude Opus 4.8	40da1aab13	IndexAndRefine: conventionalise de-novo lattice when cell+SG are given ... Providing both a unit cell (-C) and space group (-S) silently broke the de-novo indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the `sg && GetUnitCell()` branch fed the indexer's raw lattice straight into symmetry-constrained refinement. FFBIDX returns the lattice already in the reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive cell; enforcing the crystal system on its mis-assigned axes rejects every frame. Fix: for de-novo indexers only, reduce the lattice to the conventional setting (LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so it is byte-identical to before (no regression). niggli_class stays unassigned (0) in this branch - it is a property of the primitive cell incl. centering, which LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2) cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred. Validation (lyso, -C79,79,38 -S96): crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical (Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no consensus on the weak serial-still frames, 0% even without -S; to investigate.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:35:03 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60 1.0.0-rc.150	2026-06-15 20:24:15 +02:00
leonarski_fandClaude Opus 4.8	ecdb7048a0	PixelRefine: drop crystal-system idealisation, use the indexed cell as-is ... PixelRefine no longer refines the cell, so the symmetry-constrained cell parameterisation inherited from XtalOptimizer has no manifold to constrain. The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B matrix) merely reconstructed the lattice columns it started from, and for non-triclinic systems it re-idealised the indexed cell (averaging a,b for tetragonal; a,b,c for cubic; forcing alpha=gamma=90). Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a single path: take the three real-space lattice columns (latt.Vec0/1/2()) directly and form the reciprocal node via the general cross-product inverse. This reproduces every crystal system exactly from the actual cell, is more faithful (no re-idealisation), and removes the crystal_system field plus two now-unused includes. PredictedNode de-templated (only ever called with double). Crystal symmetry still lives where it belongs: indexing (upstream) and merging (downstream via the space-group HKL key). A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the old idealisation was already a no-op): stat tables bit-identical across all shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% -- the only delta is 4th-digit float-ordering noise in the error-model b coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free unchanged. Net -81 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 14:11:26 +02:00