v1.0.0-rc.153 #63

Merged

leonarski_f merged 229 commits from 2606-pixel-refine into main 2026-06-23 20:29:50 +02:00

Conversation 0 Commits 229 Files Changed 615

+44964 -13457

229 Commits

Author	SHA1	Message	Date
leonarski_f	50b08388d1	viewer: bound live-monitor memory with frame-in-flight backpressure ... The image-reading worker (on its own thread) posted imageLoaded — a full ~40 MB JFJochReaderImage on a 9 Mpix detector — to ~13 GUI consumers over queued connections, paced only by its own load+analysis time. When the GUI render (image + azimuthal integration + plots) was the bottleneck, those queued events piled up without bound and memory blew up. Funnel every consumer through one window signal: the worker's imageLoaded crosses the thread boundary exactly once, into JFJochViewerWindow:: OnImageReady, which re-emits imageReady to all GUI consumers synchronously (same thread) and then acks the worker (imageConsumed). The worker tracks images_in_flight and AutoLoadTimerExpired stops producing once the cap (max_images_in_flight = 4) is reached — applied to every auto mode. Live follow drops intermediate frames (next fetch grabs the newest image); movie playback paces down to GUI speed without skipping. The existing adaptive autoload interval is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 20:15:51 +02:00
leonarski_f	e121918f55	docs/CHANGELOG.md: round out the rc.153 entry ... Add the biggest user-facing changes the entry was missing: the trimming-energies config change for PSI EIGER (now compulsory), ROI definitions/bitmap in the master file with azimuthal phi sectors, and the viewer redesign + in-GUI jfjoch_process jobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 19:03:40 +02:00
leonarski_f	145b300fa2	broker: fail the run loudly when the writer breaks mid-acquisition ... A writer connection that dropped mid-run left a truncated file, yet the acquisition was still logged as "finished with success": the writer error returned by ImagePusher::Finalize() (set via transmission_error on a broken session connection) was captured into receiver_output.writer_err but never acted upon. Treat a non-empty writer_err as a failed acquisition: throw so it is reported as "finished with error" and surfaced to the caller, instead of silently succeeding with incomplete data. Applies to both the Lite and FPGA workflows, which share this stop/finalize path. Combined with the liveness changes (which no longer tear a connection down for a transient stall), this fires only on a genuine writer break. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 18:27:04 +02:00
leonarski_f	d156dfa2e8	TCP stream: gate SendImage enqueue on writer liveness too ... The broker logs for the dropped runs show the connection torn down ~2s into a collection (not 3s), via "TCP send failed -> Removed dead connection -> Accepted (new socket)". That is too early for the SendAll send deadline: the real gate was the fixed 2-second enqueue deadline in the zerocopy SendImage path. At the start of a large dataset the writer briefly stalls draining the socket while it creates the master file and writes the large START metadata + calibration frames to GPFS; the per-connection queue fills, and after 2s SendImage marked the connection broken. The writer then reconnected outside the active session, so the rest of the run was dropped and the half-written file was finalized at the next START. Replace the fixed 2s enqueue deadline with the same peer-liveness condition used on the send path: keep applying backpressure while the writer proves it is alive (BUSY heartbeats / ACKs refresh last_peer_activity_ns from a thread independent of the stalled write path), and only declare it dead after the liveness window of complete silence. A transient startup stall is now ridden out instead of dropping the run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 18:09:37 +02:00
leonarski_f	97c3008c39	TCP stream: tolerate writer backpressure via BUSY heartbeats ... A slow filesystem could stall the writer's consume pipeline, propagating TCP backpressure to the pusher. The pusher then treated that backpressure as a dead peer and force-closed the connection mid-run; the writer reconnected as a brand-new connection outside the active session, so the rest of the run was silently dropped and the half-written HDF5 file was later finalized with holes. Replace the throughput/progress-based send timeout with a peer-liveness timeout: - Add TCPFrameType::BUSY (wire version 2 -> 3). - TCPImagePuller runs a heartbeat thread that sends BUSY every 250ms on a thread independent of the (possibly stalled) write path, so liveness keeps flowing during deep stalls. A send_mutex serializes ACK/pong/heartbeat writes. - TCPStreamPusher refreshes last_peer_activity_ns on every inbound frame and only declares a connection dead after peer_liveness_timeout (15s) of complete silence, tolerating arbitrarily long backpressure while still catching a genuinely dead peer (and immediate EPIPE/ECONNRESET). - Re-key both backpressure waits (the SendAll data path and the post-END WaitForEndAck) onto this liveness signal instead of byte-progress / DATA-ACK-progress, so a slow final flush at END is tolerated too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:55:59 +02:00
leonarski_f	066c0dfa90	OpenAPI: fix missing trim_energie_eV	2026-06-23 16:52:16 +02:00
leonarski_f	bcf01c28f4	jfjoch_broker: Fix	2026-06-23 16:33:28 +02:00
leonarski_f	2df1035518	OpenAPI: update	2026-06-23 16:31:29 +02:00
leonarski_f	8eee3dc785	Support providing trimming energies	2026-06-23 16:26:15 +02:00
leonarski_f	e3bd1a3529	viewer: settings fields fill the panel width (form growth policy) ... The settings content stopped resizing past ~360 px, leaving dead space when the dock was widened. Cause: QFormLayout's default field growth policy keeps fields at their size hint. Set AllNonFixedFieldsGrow on all five forms (geometry, unit cell, spot finding, indexing, azint) so the fields grow to fill the dock width. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	9ba8096c0b	viewer: chart top margin + settings dock width bounds ... - The chart top margin was too tight (2 px), so the highest Y-axis label overlapped the top edge; give it 10 px of room (both chart views). - The settings dock had no width constraints (unlike the inspector), so it stretched ugly when widened and would not fold back narrowly. Cap it to 330–480 px like the inspector. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	6ead87fcce	viewer: per-image plot reuses the dataset-info combo (no second combo) ... In per-image mode the embedded chart brought its own combo box, so the plot panel showed two. Drop the per-image chart's combo and drive it from the one dataset-info combo: - JFJochViewerSidePanelChart loses its combo; it exposes PlotTypes() (the available profiles) and a setPlotType(code) slot, driven externally. - JFJochViewerDatasetInfo fills its single combo with the per-image plot types while in per-image mode (per-dataset metrics otherwise) and routes the selection to the chart; dataset/runs/live updates leave the per-image combo and plot untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	83d006e79b	viewer: "+ Plot" button + strip fits height with in-bitmap numbers ... Plots: - A "+ Plot" button on the dataset-info panel spawns another plot dock, placed beside the previous one (horizontal split) so several metrics can be watched side by side. Same path as the Charts menu, now one click away. Image strip: - The image number is painted into the thumbnail bitmap (coral badge, top-left) instead of a text label under the icon, so no height is lost to it. - Thumbnails are icon-only and scale to the available dock height (resizeEvent), so the strip never needs more room than it has; lowered its minimum. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	c5457e2e27	viewer: keep plot axis labels readable when compressed ... When the plot dock was short, Qt Charts dropped the axis labels first, making the plot hard to read. - Reclaim Qt Charts' outer graphics-layout padding and trim the inner margins (both chart views) so the plot and its labels get the available space. - Raise the chart minimum height (dataset-info 80 -> 140, per-image 120 -> 140) so the dock can't be squeezed below the point where the axis labels fit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	94b66348c5	viewer: fix thumbnail render crash (worker-thread exception / LUT UB) ... The image-strip thumbnail renderer could take the whole viewer down: - Only the image load was wrapped in try/catch; the rest of RenderThumbnail_i (pixel access, LUT mapping, QPainter) ran unguarded on the worker thread, so any exception there called std::terminate. Wrap the entire function. - std::clamp(idx, 0, lutSize - 1) is undefined behaviour when the colour LUT is empty (lo > hi). Bail out early if the LUT has no entries. A click landing while a later thumbnail is still rendering surfaced this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	164c03f1ce	viewer: soften heights, stochastic strip selection, stable on live sync ... Address the review on the image strip / hit feed: - Height constraints: JFJochSimpleChartView used a hard setFixedHeight(300) and, being a page in the dataset-info stack, forced the whole plot dock (and thus the window) taller than the screen once the strip was stacked below. Make it a soft minimum (120). Wrap the settings dock in a QScrollArea so its content can scroll instead of forcing window height. Smaller strip thumbnails (96) and lower default bottom-dock heights. The window no longer grows past its requested size. - Stochastic selection: representatives are now picked at random within N equal bins (over image index, or a metric's sorted order), and a Refresh button re-rolls a fresh set — avoiding deterministic-spacing artefacts. "Most spots" stays deterministic; "Indexed" becomes spaced-random. - Live stability: the strip stores dataset updates but only rebuilds on file open (worker fileOpened) / mode / Spots / Refresh, so HTTP sync image updates no longer trigger constant thumbnail refetching. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	7d24cdba4f	viewer: hit-feed polish — stacked docks, hidden processing, resolution mode ... Follow-up to the image strip, addressing the review: - Composition: stack the plots and the thumbnail strip vertically (plots on top with more height, strip below) instead of sharing horizontal space — both benefit from width, and the strip needs less height. - Processing dock is hidden by default and narrower; it reveals itself only when a reprocessing job starts (new jobStarted signal), and the Processing perspective no longer force-shows it. - Thumbnail spot overlays now use the same feature (indexed) / spot colours as the main viewer, and follow the side-panel colour pickers. - New strip selection modes "Resolution" and "Background": pick images that span that metric's distribution (a quick histogram-representative selection). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	c62cd99566	viewer: thumbnail hit feed (image strip) ... A dockable strip of thumbnails for a handful of representative images, with spots overlaid so a real pattern reads like a constellation; click opens that image. - JFJochViewerImageStrip: mode selector (evenly spaced / most spots / indexed, computed from the dataset's per-image metrics) + a Spots toggle; a scrollable row of thumbnail buttons; click emits imageSelected. - The reading worker renders thumbnails off-thread and non-disruptively: load each image via its reader, downsample by block-maximum (keeps Bragg spots), map through the colour-scale LUT, optionally paint spots (coral indexed / teal not), and emit thumbnailReady(image_number, QImage). File mode only. - Docked in the bottom area, shown in the Processing perspective; the colour map follows the display toolbar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	a7c1560dcb	viewer: toolbar rework — icons, prominent scrubber, HTTP states (g–k) ... Complete pass over both toolbars (review items g–k), closing the toolbar topic. New ToolbarIcons: crisp QPainter vector glyphs (Qt SVG is not a build dep), with white "on" variants for toggles and a shared flat button style (coral hover, navy checked). Navigation toolbar, laid out source-first: - Open (3.5" diskette) · HTTP sync · Movie (reel-on-top camera, distinct from the next/play triangle) ║ the scrub slider, which expands to fill the bar (coral track, navy handle) so it is the obvious way to move across the data ║ precise navigation: first/prev/[number]/next/last, Jump, Sum. - Open and HTTP-sync reach the file / connect dialogs (JFJochViewerMenu openSelected and openHttpSelected, now public). - HTTP-sync has three states via a status dot (grey disconnected / green live / amber frozen); clicking with no live source attached opens the connect dialog. Removed the separate "Reanalyze" toggle. Display toolbar: - Styled foreground slider (matching), Auto/HDR as styled text toggles. Hero buttons: - "Reanalyze image" is now a toggle (worker ReanalyzeImages: run now + keep re-analysing on image/settings/processing changes; coral = active). Processing dock: - Drop the "New job…" toolbar action (the hero button drives it) and parent the job dialog to the main window instead of the dock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	03ac43da69	viewer: panel polish — accordions, shared geometry, detector tilt, SG name ... Declutter the side and settings panels (review items a–f): - New CollapsibleSection widget (slim navy header + coral rule + chevron). - Inspector: Image features / Resolution rings / ROI are now collapsible and start folded (ROI auto-expands when ROIs change); drop the Data-analysis and Powder-calibration sections (they live in the hero buttons and settings dock). - Settings dock: move Geometry to a shared section above the MX/AzInt toggle (both communities need it); add Detector tilt (PONI rot1/rot2, deg) and rename "Beam center" -> "Beam origin" with PONI/XDS tooltips; show the space-group number and resolved Hermann–Mauguin symbol on one line; wrap sections in accordions, anchored top with a bottom stretch so expanding one does not shift the others. - Toolbar: drop the redundant "Image number" label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	e7edee6b2e	viewer: Phase 1c — fold per-image plot into the dataset-info dock ... Remove the duplicated per-image plot from the side panel and host it in the dataset-info dock instead: - JFJochViewerDatasetInfo gains a "Per-image" toggle (next to Grid) that swaps the stacked view to the existing JFJochViewerSidePanelChart (azimuthal 1D, Wilson, I/sigma, spot and fluorescence profiles of the current image). The per-dataset metric combo is disabled while in per-image mode. - The per-image profile follows the displayed image (imageLoaded forwarded). - Drop the "Image statistics plot" section from JFJochViewerSidePanel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	54526300a9	viewer: promote Reanalyze image / Reanalyze dataset hero buttons ... The two capabilities that set Jungfraujoch apart get prominent, paired navy buttons at the right of the display toolbar, with purpose-drawn icons (a single diffraction frame vs a stack of frames): - "Reanalyze image" re-runs the analysis pipeline on the current image (worker Analyze). - "Reanalyze dataset" opens a new whole-dataset processing job (JFJochProcessingJobsWindow::newJob, now public). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	9f9e13a4e4	viewer: PSI logo + tidy beam-center row ... - Show the PSI logo in the menu-bar corner, picking one of four interchangeable dot designs at random each launch. Embedded as PNGs (rasterised from the SVG sources, kept alongside) so no Qt SVG module dependency is needed. - Settings dock: put beam center X and Y on a single "Beam center" row to save vertical space on laptop screens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	58c60e03fe	viewer: Phase 1b — inline MX/AzInt settings dock ... Surface a surgical subset of processing settings in an always-visible dock instead of hidden windows: - New JFJochViewerSettingsDock with an MX / AzInt segmented toggle. MX page: geometry (energy, distance, beam X/Y), a new unit-cell + space-group editor (no such input existed before; enables known-cell ffbidx), spot finding (S/N, photon count, min pixels/spot), indexing algorithm and geometry refinement. AzInt page: q range / spacing / azimuthal bins plus the existing powder-calibration widget. - Edits feed straight into the worker (UpdateSpotFindingSettings, UpdateAzintSettings, UpdateDataset, FindCenter); fields populate from the loaded dataset. - Add CeO2 and Silicon calibrant presets. - Dock it left, objectName "settingsDock", show it in the Processing perspective (hidden in Image). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	f5a146d212	viewer: Phase 1a — dockable shell with saved perspectives ... Make the layout reconfigurable, the foundation for the redesign: - The diffraction image becomes the central widget; the right-hand side panel is now a dockable "Inspector" (QDockWidget, right area). - Every dock and toolbar gets a stable objectName so QMainWindow saveState / restoreState round-trips. Dataset-info docks are numbered uniquely. - Persist geometry + dock state to QSettings on close and restore on launch, so the user's arrangement resumes. - New "View" menu with Image / Processing perspectives (show/hide the bottom plots + jobs panel) and "Reset layout" (back to the as-built arrangement, captured at startup). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	ecc06d9abd	viewer: Phase 0 redesign — slim headers, navy accent, icon nav ... First pass of the viewer redesign (see docs/review/VIEWER_REDESIGN.md), no structural change: - TitleLabel: replace the 50px solid #FA7268 section bars with a slim 26px navy-bold header + coral accent rule, removing the "venetian blind" stack while keeping the salmon identity. - Palette: switch the accent (QPalette::Highlight) from teal to navy #1F3A5F. - Image toolbar: replace the unicode arrow glyphs (|<= <= => =>|) with flat media icons (first/prev/next/last) + tooltips, and a play icon on Movie. - Bottom dock: give the per-dataset plot more default height (340 -> 420). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 16:26:15 +02:00
leonarski_f	68f3394f48	deps: pin external Eigen to 3.4.x across docs and Docker images ... Eigen is an external find_package(Eigen3 3.4) dependency. Eigen's same-major-version rule means a bare 3.4 request only accepts 3.x, so 5.x cannot be used without changing every requester (jfjoch, Ceres, and the upstream ffbidx). Standardise on Eigen 3.4.x: - docs: correct the Windows Eigen install recipe to 3.4.0 and note the same-major constraint; SOFTWARE.md now says 3.4.x (not "3.4 or newer"). - docker/{rocky8,rocky9,ubuntu2204,ubuntu2404}: actually install Eigen 3.4.0 from source to /opt/eigen-3.4 (header-only) and add it to CMAKE_PREFIX_PATH. The images previously installed no Eigen at all, relying on the obsolete "CMake fetches it" assumption; a rebuild would have failed at configure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-22 20:53:42 +02:00
leonarski_f	ad5662732d	windows: auto-detect nvcc and drop manual CUDA/zlib flags ... Locate nvcc via find_program (HINTS CUDA_PATH / /usr/local/cuda) before CHECK_LANGUAGE(CUDA), so CMAKE_CUDA_COMPILER no longer has to be passed by hand. CHECK_LANGUAGE only searches PATH, which is missed routinely on Windows and intermittently on Linux; the CUDA installer always sets CUDA_PATH. An explicit -DCMAKE_CUDA_COMPILER / $CUDACXX still wins. Docs: zlib and Eigen are external find_package deps (not auto-fetched); simplify the Windows configure to just CMAKE_PREFIX_PATH (no CMAKE_CUDA_COMPILER, no ZLIB_ROOT); fix the Eigen 5.0.1 install recipe to disable BLAS/LAPACK (they fail MSVC with C1128 /bigobj) and use cmake --install. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-22 19:47:01 +02:00
leonarski_f	5f3126e19e	CMake: Update FFBIDX to support MSVC	2026-06-22 17:51:57 +02:00
leonarski_f	9defbffa67	Revert "CI: reuse jfjoch_hdf5_test via artifact in dials-test" ... This reverts commit 94125bd6cb.	2026-06-22 17:49:38 +02:00
leonarski_f	961836837f	viewer: salmon group boxes; dataset-info x-axis spans the whole dataset ... - Fusion fills QGroupBox interiors with a flat light colour, so the settings window lost its salmon look. A tiny app stylesheet (QGroupBox { background: transparent }) makes them show the salmon window background again; entry widgets stay white via the palette. - The dataset-info chart now fixes the x-axis to the whole dataset [0, n) (the largest run = the original file), so a subset run is drawn at its real image positions instead of being stretched to fill the plot. Subsets with binning bin by ordinal and place each point at its mapped image number (correct for the common contiguous sub-range). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 16:56:18 +02:00
leonarski_f	94125bd6cb	CI: reuse jfjoch_hdf5_test via artifact in dials-test ... Build jfjoch_hdf5_test once in a new build-hdf5-test job and hand the binary to dials-test as an artifact, instead of recompiling it from scratch in every rocky9 processing job. Uses the christopherhx gitea-{upload,download}-artifact fork (mirrored under gitea.psi.ch/actions) to avoid the stock v4 action's GHESNotSupportedError. Probe scope: dials-test only; xds-* still self-build until the round-trip is confirmed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 16:08:48 +02:00
leonarski_f	c86beeb393	rotation indexer: fix sub-range crash, explicit angle, settings + guards ... Running rotation indexing on a sub-range (e.g. images 60-120) segfaulted: the first pass passed the *global* image number to RotationIndexer::ProcessImage, which indexes v_ (sized to the run's image count) -> out-of-bounds write. - ProcessImage now takes an explicit mid-exposure angle (optional; falls back to the goniometer at the image index), so the indexer no longer assumes its slot index equals the goniometer image index. IndexAndRefine supplies it via RotationAngle(), matching the angle used for prediction. Added a bounds guard in ProcessImage so a bad index can never corrupt memory. - JFJochProcess feeds the rotation indexer the local ordinal (not the global index), and shifts the goniometer (start += start_image*incr, incr *= stride, per-image wedge preserved) so local index i maps to the angle of original image start+i*stride - fixing rotation angles for the whole sub-range pipeline (prediction, refinement, output), not just the indexer. - Expose "Rotation images" (number used for the first pass) in the job dialog, enabled when rotation indexing is on. (Count > available is already clamped by select_equally_spaced_image_ordinals.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 15:59:21 +02:00
leonarski_f	67ad43a2b2	viewer: palette theming, Original as a run row, stable plot colours ... Styling (now via the application palette in main(), not a stylesheet, so it applies to every widget incl. dialogs): - Entry fields and item views (line edits, spin boxes, combos, tables) are white when enabled and salmon (the GUI default) when disabled - so it's clear where you can type. Panels stay salmon (Window role). - Teal (#2a9d8f) accent via the Highlight role: selections and progress-bar chunks. Runs list / plots: - The original file is listed as a first "Original" run (Mode "HDF5"); double-click any run row to show it (replaces the "Show original" button). The currently shown run is bold (driven by snapshotsChanged). - Fix the colour "swap" when switching runs: the primary line is matched by the active-dataset pointer (not active_id_), so a run keeps its colour even while a datasetLoaded/runsChanged pair is mid-flight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 15:34:06 +02:00
leonarski_f	95fe2ce781	CI: Sign RPMs on upload	2026-06-22 15:22:21 +02:00
leonarski_f	03a09c43bc	viewer: white-when-active inputs, teal accent, Status column visible ... Styling: - Entry widgets (line edits, spin boxes, combos, item views) are salmon (the GUI default) when inactive and turn white while focused/active. - A teal accent (#2a9d8f) for selections and the QProgressBar chunk, against the salmon background. - Move the jobs-table Status column (the progress bar) to column 2 so it stays visible in the narrow dock instead of scrolling off the right edge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 15:10:36 +02:00
leonarski_f	0f3009069f	viewer: processing dialog takes start + end image, not a count ... Replace the ambiguous "Images" count with explicit "Start image" and "End image" spinboxes (end 0 = to the last image). buildConfig sets config.start_image / end_image; the table shows the range ("start-end" / "all") and the progress bar expects end-start images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 15:01:25 +02:00
leonarski_f	b9f8c2b675	reader/viewer: snapshot image->original map; subset runs register and overlay ... Foundation for a dataset (snapshot or, later, the main file) being a subset of the truly collected images: - JFJochReaderDataset gains source_image_number (image index -> original image number; empty = identity). - HDF5MetadataSource reads /entry/detector/number into that map and an inverse (original -> local) map; FillPerImage / ReadSpots translate the requested global image to this source's local index via ToLocalIndex (return nothing if the image is not covered), so partial snapshots are correct and never read out of bounds. - RegisterSnapshot now accepts a snapshot with fewer images than the dataset (only rejects more), so sub-range / strided reprocessing runs register. - The dataset-info plot draws each run at its original image numbers (x map from source_image_number), so a subset run lands at the right place on the shared axis. The live run's map is filled from msg.original_number. This makes the foundation ready for strided / filtered selections (e.g. reprocess only images with >N spots) without restricting to a min-max sub-range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 14:59:23 +02:00
leonarski_f	7ba097df1c	viewer: looser minimum heights so the bottom docks resize freely ... Lower the dataset-info chart and jobs-table minimum heights (200 -> 80, table 60) so the bottom dock area can be dragged smaller; the comfortable default size is still set via resizeDocks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 12:24:47 +02:00
leonarski_f	a412f96f3a	viewer: give the current-image plot marker a fixed colour ... The current-image marker (a scatter series) was picking an arbitrary theme colour (green), which read as a stray series. Pin it to black so it's an unambiguous "current image" dot, distinct from the run line colours. Its legend entry is already hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 12:13:39 +02:00
leonarski_f	2af822436e	viewer: remove/clear runs, stable per-run colours, processing rate + ETA ... - "Remove result" toolbar action drops a reprocessing run: reader RemoveSnapshot (Original is protected; if the removed run was active the view falls back to Original), worker RemoveRun, and the table row is removed. - Opening a new file clears the jobs table (worker fileOpened -> clearJobs) to match the reader, which already resets snapshots on ReadFile. - Plot fixes: each run keeps a stable colour by its position (Original = blue), so colours no longer swap when the active run changes; the current-image marker is hidden from the legend (was the stray "[Image #N]" entry). - The status bar shows processing rate (Hz) and estimated remaining time while a job runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 12:00:15 +02:00
leonarski_f	aceff23ce2	viewer: overlay one plot line per processing run (with editable names) ... The dataset-info plot now overlays every run as a separate named line instead of replacing the plot when a snapshot is activated: - Reader gains AllSnapshotDatasets() (every snapshot's dataset, Original first). - The worker owns the run collection: a stable snapshot id plus an editable display label (RunData), emitted as runsChanged(runs, active_id) on file open, snapshot register, activate and rename. RenameRun(id, label) updates the legend label without touching the reader key (decoupled id vs label). - The chart view draws the active run as the primary series (markers, hover, binning, axes) and the other runs as overlay lines sharing its axes, with a legend shown when overlaying (appendSeries factored out for both). - The dataset-info widget holds the run list + the live run, extracts the selected metric from each (ExtractMetric), and unions the available metrics across runs. - The live run is its own "Live" overlay (lightweight per-tick refresh, no combo rebuild); it is cleared on finish so the persisted snapshot takes over. - The processing jobs table gets a "Started" column and an editable Name column; editing a name renames that run's legend label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 11:28:47 +02:00
leonarski_f	b735aec1c4	viewer: show effective indexing algorithm; disable GPU options without CUDA ... - The Indexing tab now shows what the selected algorithm resolves to on this machine/dataset ("Effective on this system: ..."), mirroring DiffractionExperiment::GetIndexingAlgorithm() so Auto is no longer ambiguous (GPU present? unit cell known?). Cell-known state is forwarded from the loaded dataset via JFJochSettingsWindow::datasetLoaded. - FFBIDX and FFT (GPU) radio options are disabled on builds without CUDA, where only the FFTW CPU indexer exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 11:09:21 +02:00
leonarski_f	42219d8c5b	viewer: fix bottom-dock height, window size, and snapshot-switch lag ... Regressions from docking the processing panel + live plots: - Default window size is now screen-aware (fits a laptop) instead of a fixed 1200x1200. - Give the bottom dock area a guaranteed height (resizeDocks vertical) and a minimum height on the dataset-info chart, so the plot and its axis labels stay visible next to the processing panel. - Activating a metadata snapshot (Show original / View results) no longer re-runs full analysis: it re-reads the image against the snapshot's stored results with reanalysis suppressed, instead of re-indexing on every switch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 09:58:51 +02:00
leonarski_f	9c417d322f	viewer: dock the processing panel next to the plots; HTTP-mode notice ... - Convert JFJochProcessingJobsWindow from a standalone helper window into a dockable QWidget panel (toolbar + stacked table/message), placed in the bottom dock area and split horizontally to sit in the bottom-right corner next to the dataset-info plots. Its show/hide toggle moves to the Window menu via the new JFJochViewerMenu::AddDockEntry. - When connected to a live HTTP stream the panel shows "Dataset re-processing is currently available only in File mode" and disables the toolbar, driven by the worker's httpConnectionChanged signal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 08:35:56 +02:00
leonarski_f	940b7e34d4	viewer: indexing algorithm/refinement radio groups, rotation fix, live job plots ... Processing settings: - Split the spot/index settings widget into two tabs (Spot finding | Indexing) via TakeSpotFindingPage()/TakeIndexingPage(). - Indexing tab now exposes the indexing algorithm (Auto/FFBIDX/FFT/FFTW/None) and geometry refinement (None/Orientation/Beam center/Pixel refine) as radio groups with short explanations. - Fix: UpdateSpotFindingSettings dropped algorithm + geometry-refinement when copying into indexing_settings (the geom checkbox was a no-op); both now flow into jobs via curr_experiment. Processing jobs window: - Rotation indexing from the job dialog now also sets RotationIndexing(true) on the experiment's IndexingSettings; otherwise IndexAndRefine builds no rotation indexer and the two-pass pre-pass throws. - Status cell shows a QProgressBar with "<done> / <expected>". - Live results: JFJochProcessController::OnImageProcessed accumulates per-image results into a JFJochReaderDataset and emits it throttled (~4 Hz) as liveDataset, forwarded to the dataset-info plots so they update while a job runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-22 08:30:49 +02:00
leonarski_f	95223fc281	CI: Tests needs LFS to get rotation dataset	2026-06-22 07:17:31 +02:00
leonarski_f	74f36710e4	viewer: converge processing settings into one tabbed window + Bragg/scaling ... One "Processing settings" window with tabs: Spot finding & indexing | Azimuthal | Bragg integration | Scaling, replacing the two separate settings windows. The spot/index and azimuthal tabs reuse the existing windows' widgets unchanged (their content is lifted into tabs via takeCentralWidget), so all their logic/signals keep working; Bragg integration and scaling are new editable panels (previously not adjustable in the GUI). JFJochImageReadingWorker gains UpdateBraggIntegrationSettings / UpdateScalingSettings; both persist as worker state and are re-imported into curr_experiment on file load / dataset update (like the indexing/azint settings), so they apply to interactive analysis (Bragg) and flow into processing jobs via GetReprocessingInputs (Bragg + scaling). Scaling only affects the job post-pass, so it is just stored, not reanalyzed. Verified: jfjoch_viewer builds and runs (offscreen) with the converged window. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 17:09:20 +02:00
leonarski_f	c916cf8b2d	tests: drop the serial [large] test (serial dataset not shipped) ... Only the ~1800-image rotation dataset is kept in LFS; a separate ~5000-image serial set is too large to ship, and the serial path can be exercised by running the rotation series in serial mode if needed. Removes JFJochProcess_LysoSerial and the serial entry from the start-up listener and README. The rotation [large] test now runs against the committed data and passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 16:46:56 +02:00
leonarski_f	ed8fb5d37c	tests: add lyso rotation dataset (1800 images) via git-LFS ... Real reference dataset for the [large] JFJochProcess_LysoRotation test, stored with git-LFS (tests/data/*.h5). lyso_rotation_master.h5 is the 1800-image lysozyme rotation series (its data files keep the names the master links to). The test indexes it at 100% with cell ~78.2/78.2/37.8; it SKIPs when the data is not pulled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 16:05:25 +02:00
leonarski_f	5bfa70c4a6	viewer: Processing jobs window - run jfjoch_process in-GUI + result snapshots ... Makes the viewer a processing frontend, not just an image viewer: - JFJochProcessingJobsWindow (menu "Processing"): a table of processing jobs on the open dataset. "New job" configures mode (full/azint), image count, threads, output, and (full) rotation indexing + scale/merge, using the viewer's current processing settings. A job can be Run locally (off the GUI thread via JFJochProcessController, with live status/progress and Cancel) or its jfjoch_process command line copied for a cluster. - A finished local run is registered as a reader metadata snapshot and becomes the active dataset (bottom plots + per-image overlays follow it); "View results" / "Show original" switch between runs - so several settings attempts on one dataset can be compared. - JFJochImageReadingWorker gains GetReprocessingInputs() (one locked read of file + experiment + mask + spot-finding settings, so jobs share the interactive settings) and RegisterProcessingSnapshot/SetActiveSnapshot slots + snapshotsChanged signal over the reader's snapshot API. Files only (not live HTTP). First-cut / open ends: one job at a time; output lands in the working dir (FileWriter rejects absolute prefixes); Bragg/scaling settings still use defaults until the converged settings window lands. Viewer builds and runs (offscreen) cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 14:38:17 +02:00
leonarski_f	800ffecd20	process/viewer: command-line generator + JFJochProcessController (GUI engine) ... Foundation for making processing a first-class GUI activity: - process/JFJochProcessCommandLine: reconstruct the equivalent jfjoch_process / jfjoch_azint command line from a ProcessConfig + DiffractionExperiment + input path, for handing a job off to a cluster. Unit-tested (full + azint). - viewer/JFJochProcessController: runs one JFJochProcess job off the GUI thread (its own private reader; HDF5 access is globally serialized so it is safe next to the interactive reader) and reports back via queued Qt signals (started/phaseChanged/progress/finished/failed). Cancel() forwards to JFJochProcess::Cancel(). Progress is throttled to ~200 updates per run. The visible processing UI (jobs window + snapshot switcher + converged settings) builds on this. Verified: tests/jfjoch_test [process]; jfjoch_viewer links and builds against JFJochProcess. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 14:26:48 +02:00
leonarski_f	39599bc090	tests: git-LFS large-dataset harness + jfjoch_process include cleanup ... - .gitattributes tracks tests/data/*.h5 via git-LFS for big reference datasets. - tests/TestData.h resolves tests/data files and reports absent / unfetched-LFS-pointer so tests can SKIP() instead of failing; tests/data/README.md documents fetching + the expected lyso_rotation/lyso_serial datasets. - JFJochProcessLargeTest: a Catch start-up listener that prints dataset availability, plus [large] full-analysis runs (rotation indexing + serial) that SKIP when data is absent. Verified against the real 1800-image lyso rotation set (100% indexing, cell 78.2/78.2/37.8) and skips cleanly without it. - jfjoch_process.cpp: drop the ~15 now-unused workflow includes left after the JFJochProcess extraction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 14:16:34 +02:00
leonarski_f	4697f10555	process: extract reprocessing workflow into shared JFJochProcess library ... The full processing workflow no longer lives in the CLIs. New process/JFJochProcess encapsulates it for jfjoch_process, jfjoch_azint and (later) the viewer: - ProcessMode {AzimuthalIntegration, FullAnalysis}; ProcessConfig carries run control (range, threads, output prefix, spot finding, rotation + scaling options, reference data) while the DiffractionExperiment carries all algorithm settings. - Run() executes setup -> optional two-pass rotation pre-pass -> parallel per-image loop (std::thread) -> optional scaling/merging post-pass -> NXmxIntegrated _process.h5 that links back to the original images. ProcessResult returns stats + merge text. - Cancel() / std::atomic<bool> (receiver style), checked between images; the CLIs install a SIGINT handler that calls it (fixes the previous Ctrl+C gap), the viewer will use the same hook. JFJochProcessObserver streams progress / per-image results for a live GUI. jfjoch_process.cpp and jfjoch_azint.cpp are now thin: argument parsing + experiment configuration, then JFJochProcess::Run + stats printing. Behaviour and usage messages are unchanged. Adds JFJochProcessTest (azimuthal integration round-trip, no-output run, pre-cancel) over a small generated dataset. Verified: tests/jfjoch_test [HDF5] (83 cases / 1854 assertions); jfjoch_azint and jfjoch_process run end-to-end on lyso_test (azint 20 images; full analysis recovers the lysozyme cell at 25% indexing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 10:36:18 +02:00
leonarski_f	9434878c92	reader: split metadata reading into HDF5MetadataSource + snapshot map ... Complete the image/metadata decoupling. JFJochHDF5Reader is now a thin JFJochReader facade composing one shared HDF5ImageSource (raw pixels) with a name->HDF5MetadataSource map and an active selection ("snapshots"). - HDF5MetadataSource owns one master file: dataset-level parse (Open, ex-ReadFile), per-image metadata (FillPerImage, ex-LoadImage_i), ReadSpots/ReadReflections/ ReadCalibration, and the master-first helpers. It locates per-image metadata either via the shared image source (original file: metadata co-located with pixels) or in its own master at the global index (integrated _process.h5 snapshot) - so switching snapshots never reloads pixels. - Facade adds RegisterSnapshot/SetActiveSnapshot/SnapshotNames/ActiveSnapshot; ReadFile registers the original as "Original". All existing JFJochReader callers are unchanged. - The global hdf5_mutex is taken once in the facade; the sources assume it is held. Adds JFJochReader_Snapshots test (original _master.h5 + a reprocessing _process.h5 over the same images: pixels stay from the original, metadata follows the active snapshot). Verified: tests/jfjoch_test [HDF5] (80 cases / 1810 assertions), plus jfjoch_process/ azint/extract_hkl/scale and jfjoch_viewer all build against the refactored reader. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 10:15:09 +02:00
leonarski_f	23d27f30c4	reader: split raw-image reading into HDF5ImageLocator + HDF5ImageSource ... Decouple the raw-pixel side of JFJochHDF5Reader from the rest as the first step toward swappable per-dataset metadata snapshots. - HDF5ImageLocator: single owner of the legacy/VDS/contiguous layout resolution plus a persistent open-file cache, replacing the four duplicated resolvers (GetImageLocation, ReadSpots, ReadReflections) and their per-call file caches. Also hosts the source-mapping logic (former GetHDF5DataSource body). - HDF5ImageSource: raw-pixel reading (locator + LoadImageDataset); the part whose links to files stay fixed while the metadata master may change. - JFJochHDF5Reader keeps a thin GetImageLocation/GetRawImage/GetHDF5DataSource that delegate to image_source_; the six layout members are gone, parsed into a local Layout handed to the source at the end of ReadFile. Cache cleared on Close(). Verified: tests/jfjoch_test [HDF5] (79 cases / 1775 assertions), and jfjoch_process/azint/extract_hkl/scale relink unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-21 10:15:09 +02:00
leonarski_f	ec8a45dc09	windows: bundle cuFFT in installer and tag the CUDA/CPU variant ... cuFFT is the only CUDA component linked dynamically (cudart and the fast-feedback indexer are static), so the prior Windows installer would fail to launch the GPU path on a host without a CUDA toolkit. Ship cufft64_*.dll next to the viewer (CUDA 13 keeps it in bin/x64, earlier toolkits in bin); the DLL is self-contained, so the installed app needs only an NVIDIA driver. Tag the variant where it matters and nowhere else: the installer filename (-cuda<major> / -cpu) and the Add/Remove Programs entry ("Jungfraujoch (CUDA)" / "(CPU)") advertise it, while the install folder and Start Menu group stay plain "Jungfraujoch" -- the CUDA build is a strict superset, so the two variants share a location and replace each other. Docs: SOFTWARE.md + JFJOCH_VIEWER.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-21 09:17:52 +02:00
leonarski_f	0c1babd0ca	CMake: provide Eigen externally again; fix cpack NSIS icon escape ... Revert Eigen from the FetchContent OVERRIDE_FIND_PACKAGE vendoring (cd394c52, d4d2d0d8) back to an external find_package(Eigen3) dependency, like zlib. The OVERRIDE_FIND_PACKAGE mechanism makes the Visual-Studio-bundled CMake (4.x "-msvc") intermittently segfault during configure -- ~1/3 of fresh configures, and 100% with Ceres CUDA enabled. The fault is in CMake's own FetchContent variable-stack cleanup, reached when Ceres' find_package(Eigen3) resolves the override through a nested FetchContent_MakeAvailable. Stock Kitware CMake runs the identical scripts fine, so it is a bug in the VS-bundled cmake binary; providing Eigen externally avoids that path entirely and is stable with Ceres CUDA both on and off (verified 0 crashes; full CUDA build + NSIS installer succeed under the VS-bundled cmake). Also fix CPACK_NSIS_INSTALLED_ICON_NAME: the backslash value was written verbatim into CPackConfig.cmake, where "\j" is an invalid escape that cmake 4.x (CMP0010 strict) rejects when cpack re-parses it, so cpack failed under the VS-bundled cmake (older/Kitware cpack only warned). Use a forward slash. - image_analysis: keep Ceres USE_CUDA OFF (Jungfraujoch does not use Ceres' GPU solvers) but drop the now-obsolete "required to avoid the Windows cmake crash" rationale from the comment. - docs: Eigen is a required external dependency again (SOFTWARE.md); the Windows viewer build provides zlib + Eigen externally (JFJOCH_VIEWER.md). - THIRD_PARTY_NOTICES: move Eigen from the fetched-and-linked table to the external runtime-libraries table. Docker images still need Eigen re-added (5.0.1, source build) -- to follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-20 22:33:12 +02:00
leonarski_f	1ea5897fca	windows: vendor wingetopt and scope server-only targets out of viewer-only builds ... MSVC/Windows build fixes: - Force JFJOCH_VIEWER_ONLY ON for Windows and macOS; the broker/receiver/FPGA/ writer server stack is Linux-only, so those platforms build the viewer subset. - Vendor an OpenBSD/NetBSD getopt/getopt_long shim (tools/wingetopt/, BSD/ISC) as a WIN32-only static lib -- the MSVC CRT has no <getopt.h> -- and link it into the four portable CLI tools on Windows. - jfjoch_extract_hkl.cpp: include <getopt.h> explicitly (it relied on glibc's transitive <unistd.h>). - jfjoch_process.cpp: guard <unistd.h> with #ifndef _WIN32. - Guard JFJochStreamWriter and jfjoch_writer with IF(NOT JFJOCH_VIEWER_ONLY): the former pulls in JFJochImagePuller, the latter uses a fork()/waitpid() multi-process design; neither belongs to the portable viewer subset. - Record wingetopt in THIRD_PARTY_NOTICES.md and licenses/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 18:50:20 +02:00
leonarski_f	e5034d0a2c	licenses: add third-party notices, attribution texts, and viewer license window ... Acknowledge all bundled third-party software and satisfy attribution/notice requirements, while keeping it maintainable: - THIRD_PARTY_NOTICES.md: human-readable manifest (component, copyright, SPDX license, link) for fetched, vendored, and runtime/SDK dependencies. - licenses/: verbatim license texts; COLLECT.sh regenerates them from the build trees and system SDK locations. - Bundle the verbatim Qt LGPL-3.0 text and the CUDA Toolkit 12.8 EULA. - frontend: self-contained npm attribution generator (`npm run licenses` -> dist/THIRD_PARTY_LICENSES.txt), wired into the frontend build target. - Install LICENSE + notices + licenses/ into share/doc/jfjoch for every packaged component. - viewer: Help > "Third-party Licenses" window (QTextBrowser) showing a generated, self-contained HTML built from licenses/. - docs/SOFTWARE.md: drop the stale hand-kept dependency lists; point at the manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 18:35:50 +02:00
leonarski_f	e035f7c303	viewer: drop unused CMAKE_AUTORCC ... The viewer's only .qrc is compiled explicitly via QT_ADD_RESOURCES into APP_RESOURCES; no .qrc is added directly as a target source, so AUTORCC never acted on anything. Remove the dead flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 17:22:48 +02:00
leonarski_f	f9396e01e7	viewer: Windows NSIS / macOS installer polish + bundle analysis CLIs ... NSIS installer was missing several things on Windows: - License: show the repo GPLv3 LICENSE as the click-through license page (CPACK_RESOURCE_FILE_LICENSE). - Start Menu: no shortcut was created -- add CPACK_PACKAGE_EXECUTABLES for jfjoch_viewer, and embed an app icon so the shortcut is not generic. The icon is a new multi-size jfjoch.ico (256 PNG + 48/32/16) compiled into the .exe via resources/jfjoch.rc; macOS gets the matching jfjoch.icns copied into the .app bundle (MACOSX_BUNDLE_ICON_FILE). Both derived from resources/jfjoch.png, padded square with transparent bands. - Naming: install folder, Start Menu group and Add/Remove Programs entry are now "Jungfraujoch" instead of "jfjoch <version>". Also ship the offline analysis CLIs (jfjoch_process/scale/azint/extract_hkl) in viewer-only / Windows packages: JFJochReceiverPlots now lives in JFJochCommon, so their stale JFJochReceiver link is dropped, leaving only the portable libs (reader/image_analysis/writer/common). tools/CMakeLists.txt is split so those four build everywhere while the hardware tools stay gated behind NOT JFJOCH_VIEWER_ONLY, and the viewer-only branch now adds tools/. They install into the "viewer" CPack component. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 16:40:46 +02:00
leonarski_f	e6c6536e3a	JFJochReceiverPlots: Move to common/, so it can be linked without bringing JFJochReceiver HW dependencies	2026-06-20 16:26:11 +02:00
leonarski_f	5247acc07e	CMake: package the viewer component in JFJOCH_VIEWER_ONLY builds ... A viewer-only build builds jfjoch_viewer but the CPack block hard-coded CPACK_COMPONENTS_ALL to "jfjoch writer" and only appended "viewer" when JFJOCH_VIEWER_BUILD was set -- which a pure viewer-only build does not set. cpack then tried to package the empty jfjoch/writer components and failed, requiring a manual -D CPACK_COMPONENTS_ALL=viewer. Branch the component list on build mode: viewer-only packages just "viewer"; other modes keep "jfjoch writer" plus driver-dkms/viewer when their flags are set. Also drops the redundant re-set of the default list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 16:13:24 +02:00
leonarski_f	5064b5ba47	image_analysis: force Ceres to build without CUDA ... USE_CUDA is a Ceres cache STRING (default "default" = auto-detect), not an option(), so a plain SET was shadowed by the cache default and CUDA got enabled on MSVC. Force it OFF in the cache instead. Also drop two dead settings: MINIGLOG (Ceres uses abseil logging now, no longer referenced) and EIGENSPARSE ON (already Ceres' default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 16:08:38 +02:00
leonarski_f	a89a4c5beb	jfjoch_viewer: Show detected GPUs in the about box	2026-06-20 16:08:13 +02:00
leonarski_f	9c9f818ea6	CMake: support DGX Spark (Blackwell GB10 sm_121 + aarch64 FFTW NEON) ... - CUDA: append sm_121 (Blackwell GB10 / DGX Spark) to CMAKE_CUDA_ARCHITECTURES when nvcc >= 12.9 knows it. The static list tops out at sm_120 and embeds no PTX, so without this a build would fail to launch on Spark ("no kernel image available"); guarded so CUDA 12.8 (which predates sm_121) is unaffected. - FFTW: enable NEON codelets on aarch64 (Grace), mirroring the x86 SSE2/AVX/AVX2 branch, so single-precision FFT is SIMD-accelerated instead of scalar on ARM. Both are inert on x86 / CUDA 12.8; verified configure still succeeds there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 15:42:00 +02:00
leonarski_f	8cd46751f9	docker: drop fftw too -- the project builds FFTW itself ... FFTW is fetched and built by the jungfraujoch CMake (single-precision, from the release tarball), so the system fftw packages (fftw-static/fftw-devel on Rocky, libfftw3-dev on Ubuntu) were unused. Remove them from all four images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 15:32:05 +02:00
leonarski_f	da5fff71a9	docker: unify static Qt 6.9.1 on all 4 Linux images, drop dead prebuilds ... The jungfraujoch CMake now builds HDF5 + libtiff (FetchContent) and libjpeg-turbo + Eigen (ExternalProject / OVERRIDE_FIND_PACKAGE) itself, so the per-image pre-builds and system -dev packages for those were dead weight (the app rebuilt its own copies regardless). Consolidate dependency management into the one CMakeLists at the cost of longer build time (build env has network). - Qt: pin static 6.9.1 everywhere. ubuntu2204 was on 6.10.0 and ubuntu2404 used the OS Qt (~6.4, dynamic) -- the latter is now a static 6.9.1 build like the others, so every image ships a self-contained, identical-Qt viewer. - Remove the HDF5/libtiff/libjpeg-turbo/Eigen source builds + their system -dev packages + the stale /opt/* CMAKE_PREFIX_PATH/PKG_CONFIG_PATH entries. - nasm: now needed by the project's libjpeg-turbo ExternalProject; added to ubuntu2404 (the others already had it). - Standardize the Qt configure flags across all four (xcb, xcb_xlib=OFF, no Wayland -- the qtwayland module was never built, so wayland_client=ON on ubuntu2204 was a no-op). Untested here (no Docker on this host) -- needs a build in each image; the ubuntu2404 static-Qt dep list in particular should be sanity-checked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 15:28:29 +02:00
leonarski_f	31630c20a1	viewer: macOS .app / Windows GUI packaging + Qt deploy + CPack generators ... Add the platform packaging the viewer was missing so it can ship as a .dmg (macOS) or installer (Windows): - viewer: set MACOSX_BUNDLE (a .app bundle on macOS) and WIN32_EXECUTABLE (GUI subsystem, no console window) -- both ignored where they don't apply; install with BUNDLE/RUNTIME destinations. - viewer: qt_generate_deploy_app_script + install(SCRIPT) to bundle the Qt runtime at install time (windeployqt on Windows, macdeployqt on macOS). Gated on Qt >= 6.5, which is where the OUTPUT_SCRIPT signature and the Linux deploy path landed; with a static Qt (our Linux builds) it is a no-op. - CPack: select DragNDrop (.dmg) on macOS and NSIS on Windows, ahead of the Linux-only /etc/* probes; DEB/RPM path unchanged. Verified: configures + generates cleanly on Linux with the viewer enabled (Qt 6.4.2 -> deploy step correctly skipped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 15:05:40 +02:00
leonarski_f	d4d2d0d8ad	CMake: fix CI -- export vendored Eigen so Ceres' install(EXPORT) resolves ... A clean configure failed at generation: install(EXPORT "CeresExport" ...) includes target "ceres" which requires target "eigen" that is not in any export set. Ceres' install(EXPORT CeresExport) is unconditional, and its ceres target transitively links the eigen target. As a FetchContent subproject Eigen leaves EIGEN_BUILD_CMAKE_PACKAGE OFF (it defaults to PROJECT_IS_TOP_LEVEL), so its install(EXPORT Eigen3Targets) never runs and eigen ends up in no installed export set. Force EIGEN_BUILD_CMAKE_PACKAGE ON so eigen is exported and Ceres' export validates. Install/export-only change; the build is unaffected. Verified with a clean configure (fresh CMakeCache): configure + generate now succeed where they previously errored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 13:38:46 +02:00
leonarski_f	cd394c5249	CMake: vendor Eigen (5.0.1) and libjpeg-turbo so the build is self-contained ... Two deps the Linux build picked up from the system were absent on Windows. Bundle both, unconditionally (matching how libtiff/FFTW are vendored), so the tree builds without system Eigen/JPEG: - Eigen 5.0.1 via FetchContent with OVERRIDE_FIND_PACKAGE, so every find_package(Eigen3) -- ours, Ceres', and ffbidx' -- resolves to this copy and the requested 3.4 version is satisfied by the newer major. ffbidx's bundled eigen submodule is disabled (GIT_SUBMODULES "" on fast-indexer) so it no longer creates a duplicate Eigen3::Eigen target. Verified on a CUDA build: Ceres 2.3.0, the analysis libs, and ffbidx's CUDA code all compile against 5.0.1. - libjpeg-turbo 3.0.4 via ExternalProject (upstream discourages add_subdirectory); built+installed static and imported as JPEG::JPEG, replacing find_package(JPEG) in preview/. SIMD is lazy: WITH_SIMD=ON + REQUIRE_SIMD=OFF means NASM-accelerated when nasm is present, otherwise a warning and a scalar build (no hard failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 07:55:56 +02:00
leonarski_f	4737eff1a9	jfjoch_process: Missing include <set>	2026-06-20 07:44:50 +02:00
leonarski_f	7eb927e82f	CMake: Ensure static ZLIB (must be before FIND_PACKAGE)	2026-06-20 07:40:56 +02:00
leonarski_f	1ae2e6869d	CLAUDE.md: record Windows viewer goal of MSVC + CUDA, and unbundled deps ... The recurring goal is a Windows jfjoch_viewer built with MSVC *and* CUDA ON (GPU processing is wanted, not a CPU-only fallback); the non-CUDA path is the macOS fallback. Also note the three deps a self-contained Windows build still needs but FetchContent does not auto-provide: ZLIB (bundle zlib-ng as ZLIB::ZLIB), libjpeg-turbo (ExternalProject), and Eigen (header-only, also needed by Ceres) -- each guarded so the Linux build is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 07:38:06 +02:00
leonarski_f	025c6394b6	reader: fix Windows path handling in JFJochHDF5Reader ... Two spots used POSIX-only path logic that breaks on Windows: - dataset_name() located the basename with rfind('/'), which misses backslash separators, so a C:\...\foo_master.h5 file prefix kept the whole directory. Use std::filesystem::path::filename() instead. - Legacy-format data-file resolution joined paths with fmt::format("{}/{}", ...), producing mixed \/ separators and prepending the master directory even to absolute link targets. Reuse the existing ResolveRelativeToMaster() helper, which joins via std::filesystem and leaves absolute targets untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 07:23:07 +02:00
leonarski_f	4a39f2cfcf	Fix FFTW build-tree header wiring and two MSVC warnings ... FFTW exposes fftw3.h only via $<INSTALL_INTERFACE:include>, so consuming fftw3f from the build tree (FetchContent) leaves FFTIndexerCPU.h unable to find the header. A system-installed fftw3.h masks this on Linux, so it only bit on Windows; add the source-tree api/ dir to fftw3f's interface includes. Also two viewer-reachable warnings: - time_utc.h (C4477): %03ld was fed a chrono milliseconds::rep (64-bit), not a long; cast the 0-999 value to int and use %03d. - HDF5Objects.h (C4700): value-initialize the scalar read buffer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 07:15:47 +02:00
leonarski_f	b888c53509	CMake: stop libtiff/httplib pulling in webp, sharpyuv, brotli ... libtiff's webp and lerc codec options default ON whenever their library is found on the build host, dragging in libwebp + libsharpyuv (webp) and libLerc. We only need DEFLATE (zlib) + the internal LZW codec, so turn both OFF. httplib likewise auto-enables Brotli content-encoding when system Brotli is present; gzip/zlib is enough for the broker, so disable it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 07:10:04 +02:00
leonarski_f	a872b68d6f	ADUHistogram: Fix missing exception header	2026-06-20 06:14:15 +02:00
leonarski_f	bcd588b400	CMake: build StreamWriter only into JFJochStreamWriter ... StreamWriter.cpp was compiled into both JFJochWriter and JFJochStreamWriter, so JFJochWriter still transitively pulled in image_puller headers via StreamWriter.h. Compile it only into JFJochStreamWriter (which already links JFJochImagePuller); targets needing the streaming writer (jfjoch_writer, jfjoch_test) link that lib, while JFJochWriter consumers (viewer, tools, ImagePusher) no longer drag in the puller. Also drop a stray duplicate HDF5DataFile source listing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 23:15:49 +02:00
leonarski_f	ebc28744e2	viewer: fix MSVC compatibility in viewer-reachable code ... Address portability issues found building jfjoch_viewer with MSVC: - common/{ADUHistogram,AzimuthalIntegrationProfile}.h and image_analysis/spot_finding/StrongPixelSet.h only need DeviceOutput, so include fpga/pcie_driver/jfjoch_fpga.h (plain-C, self-contained) directly instead of acquisition_device/AcquisitionDevice.h, which dragged <unistd.h> into the viewer tree. - common/time_utc.h: guard gmtime_r/timegm/localtime_r with the MSVC equivalents (gmtime_s/_mkgmtime/localtime_s) under _WIN32; drop the duplicated includes and add the headers used directly. - gemmi_gph/gemmi/utf.hpp: vendor the upstream gemmi header; fileutil.hpp includes it on Windows for UTF8_to_wchar but it was never vendored. - writer/HDF5Objects.cpp: ExtractFilename returns path.filename().string() (std::filesystem::path has no implicit conversion to std::string on Windows, where it is wchar_t-based). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 23:15:27 +02:00
leonarski_f	07ce8b785a	CMake: Remove image_puller dependency from the jfjoch_viewer	2026-06-19 23:03:07 +02:00
leonarski_f	c9803ccbfd	Separate JFJochWriter and JFJochStreamWriter	2026-06-19 22:56:11 +02:00
leonarski_f	5a6485da3a	cmake: use upstream libtiff (gitlab) at v4.7.1 ... Switch the libtiff FetchContent from the personal GitHub mirror to the upstream gitlab repo and bump v4.6.0 -> v4.7.1 -- cleaner provenance; the mirror was only ever a network-reachability workaround. Verified on Linux: fetches, builds static (libtiff.a/libtiffxx.a), and jfjoch_viewer links cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 22:47:36 +02:00
leonarski_f	53645434be	cmake: enable FFTW SSE2/AVX/AVX2 codelets (x86, runtime-dispatched) ... FFTW selects SIMD codelets at runtime via cpuid, so enabling AVX2 does not require an AVX2 CPU -- it just speeds up the CPU FFT fallback indexer on capable hardware and falls back to SSE2/scalar otherwise. Arch-guarded to x86_64/AMD64 (ARM would need ENABLE_NEON). AVX-512 left off on purpose: it's x86-64-v4 (not our v3 target), the FFT gain over AVX2 is marginal, older Intel downclocks on it, and it's the riskiest to build under MSVC. Verified on Linux: SSE2/AVX/AVX2 codelets compile, libfftw3f.a stays a single static archive, jfjoch_viewer relinks cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 22:37:09 +02:00
leonarski_f	0220c6376d	cmake: fetch libtiff and FFTW instead of relying on system libs ... Both are leaf deps (JFJochPreview / JFJochIndexing) that only our own code looks for, so building them ourselves removes the system-package lottery and gives a reproducible, statically-linked build on every platform. - libtiff: FetchContent, library only (jbig/zstd/lzma/jpeg/old-jpeg/tools/ tests off). The C++ binding (TIFF::CXX / libtiffxx) is packaged inconsistently across distros -- missing on Rocky 9 -- and absent on Windows, so find_package(TIFF COMPONENTS CXX) was unreliable; that call is removed and JFJochPreview links the tiff/tiffxx targets directly. GitHub mirror because upstream (gitlab) is unreachable from some restricted hosts. - FFTW: FetchContent single precision (ENABLE_FLOAT) from the release tarball -- the git repo ships no pre-generated codelets (needs OCaml genfft). It's now always available, so the CPU FFT indexer is always built and JFJOCH_USE_FFTW always defined; the "FFTW disabled" path is gone. Static (libfftw3f.a) via the global BUILD_SHARED_LIBS OFF. Verified on Linux: jfjoch_viewer builds and links libfftw3f.a + libtiff*.a, all static. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 22:16:05 +02:00
leonarski_f	fcbe19103d	Update to most recent bitshuffle version	2026-06-19 22:10:06 +02:00
leonarski_f	0585fab9bf	cmake: don't fetch sls/catch2 in a viewer-only build ... Under JFJOCH_VIEWER_ONLY, only the viewer's transitive deps are built, so sls_detector_package (detector_control/receiver) and catch2 (tests) are never used -- and sls does not configure under MSVC, which is the whole point of the viewer-only mode. Skip fetching just those two; zstd, hdf5, spdlog, libzmq and httplib are still required (JFJochReader links httplib). Verified: cmake -DJFJOCH_VIEWER_ONLY=ON configures with sls/catch2 absent and `make jfjoch_viewer` still links. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 21:02:28 +02:00
leonarski_f	3c6439530d	cmake: raise policy floor to 3.5 for FetchContent deps (CMake 4.x) ... CMake >= 4.0 (4.3.1 ships bundled with Visual Studio 2026) makes any cmake_minimum_required(VERSION < 3.5) a fatal error. The fetched libzmq still declares VERSION 3.0.2, which aborts configuration before our own CMakeLists gets a say. Set CMAKE_POLICY_VERSION_MINIMUM=3.5 before the FetchContent_MakeAvailable calls so those subprojects configure; the variable is simply unused (harmless) on CMake < 3.30. Verified against CMake 4.3.1: the real libzmq source fails without it and configures with it, and an in-file SET() propagates into the add_subdirectory child exactly as a -D cache entry does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 21:02:28 +02:00
leonarski_f	f21c22d453	cmake: add JFJOCH_VIEWER_ONLY to build the viewer standalone ... New option (default OFF) that compiles only jfjoch_viewer and the libraries it transitively links -- common, image_analysis, writer, reader, preview, image_puller, frame_serialize, gemmi_gph, compression, jungfrau, and the OpenAPI model (JFJochAPI) -- skipping the Linux-only online stack (broker service, receiver, fpga, acquisition_device, detector_control, image_pusher, xds-plugin, tests, tools) and the frontend. This is the standalone build path for the cross-platform viewer. JFJochAPI is the only piece the viewer needs from broker/, so the broker service library and executable are gated behind NOT JFJOCH_VIEWER_ONLY. The full build and JFJOCH_VIEWER_BUILD (viewer on top of the full build) are unchanged -- that path is the verbatim ELSE() branch. Verified on Linux: cmake -DJFJOCH_VIEWER_ONLY=ON configures and `make jfjoch_viewer` links cleanly with the online stack absent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 21:02:28 +02:00
leonarski_f	74c2c808de	cmake: drop GCC-only hardcoded flags, bump CUDA to C++20 ... Make the build less hostile to MSVC (needed for the viewer port) and align the CUDA standard with C++. - Remove the CMAKE_C/CXX_FLAGS_RELEASE overrides: -O3/-DNDEBUG only duplicate CMake's own GCC/Clang Release defaults, and the SET() clobbered the MSVC defaults. The lone meaningful flag, -Wno-deprecated-enum-enum-conversion, merely silenced warnings from the vendored Xilinx ap_private.h in the FPGA path; with no -Werror it is just log noise, so drop it. - CMAKE_CUDA_STANDARD 17 -> 20 (+ STANDARD_REQUIRED); CUDA >= 12.8 already supports C++20. - Drop the file-scoped -Ofast on JFPedestalCalc.cpp / JFConversionFloatingPoint.cpp: it is GCC-only and reaches the MSVC viewer via JFJochCommon -> JFCalibration. Those files now build at plain -O3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 21:02:28 +02:00
leonarski_f	1e53db8f64	Docker: Update 2606 to support python3.12/3.13	2026-06-19 20:56:13 +02:00
leonarski_f	3ce7666738	make_doc.sh: prefer the newest available python3.x ... The bare python3 is often too old for the Sphinx build. Auto-select the newest python3.13/3.12/3.11/3 found on PATH, still allowing $PYTHON to override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 16:37:32 +02:00
leonarski_f	4e9505e6bb	make_doc.sh: Use python3	2026-06-19 16:06:49 +02:00
leonarski_f	a34a033c99	frontend: auto-fill the missing phi bound for a sector ... When only one azimuthal phi bound is entered, fill the other so the sector is well-defined and the server accepts it: a missing phi_min becomes 0 and a missing phi_max becomes 360. Leaving both empty still means a full ring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 16:05:15 +02:00
leonarski_f	e9e5ec79cd	frontend: phi_min/phi_max columns in the azimuthal ROI table ... Surface the optional azimuthal phi-sector bounds (phi_min_deg/phi_max_deg) as editable columns, so a sector can be set from the web UI, not only the viewer. Also fix the Q_max row's d header, which read d_max but is d_min. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 16:01:13 +02:00
leonarski_f	e60b42ea83	viewer: white, compact, right-aligned ROI numeric fields ... The ROI bound editors now have a white background, a capped width and right-aligned text, which reads better for numeric entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 16:01:13 +02:00
leonarski_f	abc1d1c079	viewer: add/subtract the selected ROI to/from the user mask ... Restore the ROI-to-mask action on the new list: "Add to mask" and "Subtract from mask" buttons rasterise the selected ROI into the user mask (set or clear), through the same UpdateUserMask_i path. The ROI is evaluated with per-pixel resolution and phi from the geometry, so box, circle and azimuthal (sector) ROIs all map correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 15:08:59 +02:00
leonarski_f	81b48febcf	viewer: remove the dead scratch-ROI widgets and worker methods ... With the scratch ROI panel gone, delete the now-orphaned JFJochViewerImageROIStatistics(_Box/_Circle) widgets and the worker's unused scratch-ROI plumbing (SetROIBox/SetROICircle, the ROIElement member, AddROIToUserMask/SubtractROIFromUserMask). ROI-to-mask, if wanted later, should act on the selected list ROI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:58:33 +02:00
leonarski_f	52aca9a4a9	docs: ROIs/azimuthal apply to both FPGA and DECTRIS workflows ... Clarify in FPGA_DATA_ANALYSIS.md that the ROI and azimuthal definitions are also evaluated on CPU via the shared image_analysis library, so they apply to the DECTRIS SIMPLON (EIGER) path that has no FPGA, not only the FPGA-accelerated JUNGFRAU/PSI path. Document the three NXmx writer formats (Legacy/VDS/Integrated) and the two acquisition workflows in CLAUDE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:55:01 +02:00
leonarski_f	d4c96d48fc	make_doc.sh: Use default python	2026-06-19 14:52:40 +02:00
leonarski_f	fef9c2f4c3	viewer: load user mask from TIFF (replace or add) ... Two Mask-menu entries read a 16-bit TIFF (via ReadTIFFFromString16) and apply it as the user mask: "replace" overwrites the current mask, "add" unions the TIFF into it. The TIFF must match the detector dimensions; any non-zero pixel masks. Both go through UpdateUserMask_i like the other mask edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:52:21 +02:00
leonarski_f	b2c60dfbd0	viewer: download/upload ROIs to the broker ... Add Download and "Upload to server" buttons to the ROI panel. The HTTP reader gains GetROIDefinitions (GET /config/roi) and UploadROIDefinitions (PUT /config/roi), converting between ROIDefinition and the generated Roi_definitions model (including the optional azimuthal phi sector), the same shapes OpenAPIConvert uses on the server. Download applies the fetched ROIs through SetROIDefinition; upload pushes the current ones. Both are no-ops unless the viewer is connected to a broker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:47:10 +02:00
leonarski_f	142cb88aa8	viewer: shift/ctrl-drag creates list ROIs; remove the old scratch panel ... The interactive shift-drag (box) / shift+ctrl-drag (circle) now creates a new persistent ROI in the list instead of feeding the old single-ROI scratch panel. The base emits a roiScratchDrawn hook on release; the diffraction image turns the drawn shape into a named ROI committed via SetROIDefinition. The old JFJochViewerImageROIStatistics scratch panel and all its wiring (box/circle configuration, single-ROI result, add/subtract user mask) are removed from the side panel and window; the ROI list is now the single source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:31:23 +02:00
leonarski_f	1d015145a4	viewer: numeric editing of the selected ROI's bounds ... The ROI panel now shows editable numeric fields for the selected ROI, labelled by type: box min/max X/Y, circle centre/radius, or azimuthal Q-range and phi sector. Editing a field rebuilds that ROI and commits it through the same SetROIDefinition path, so all three types can be created (via +) and dialled in by typing, not only by dragging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:24:57 +02:00
leonarski_f	d413af0e92	viewer: azimuthal ROI handles, and keep panning inside the ROI ... Azimuthal ROIs now show discrete grab handles on the selected ROI: the inner/outer arc (resize Q/d) and, for a sector, the two phi edges (rotate). Editing is by clicking a handle, with a generous tolerance, which fixes the previously near-impossible phi-edge grab. Clicking the (large) interior of an azimuthal ROI now only selects it and lets the view pan, instead of capturing the gesture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:21:23 +02:00
leonarski_f	00b58cd426	viewer: resize box/circle/azimuthal ROIs and delete with the Delete key ... Extend canvas ROI editing beyond move: - Box: drag corners/edges to resize (handles shown on the selected ROI). - Circle: drag the perimeter to resize, interior to move. - Azimuthal: drag the inner/outer arc to change the Q/d range, or a radial edge to rotate a phi bound; sampled through the geometry so it tracks the conic. - Delete removes the selected ROI (the view now takes keyboard focus). All edits go through the same live, throttled SetROIDefinition path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 14:07:16 +02:00
leonarski_f	d9dfd36413	viewer: click to select an ROI, with live move and a hand cursor ... Clicking a box/circle on the image now selects it (syncing the side-panel combobox) and grabs it for moving, with a closed-hand cursor during the drag. The move recomputes ROI statistics live rather than only on release, throttled to at most one in-flight recompute so the worker is not flooded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	308265c7b1	viewer: move the selected box/circle ROI on the canvas ... Add base-class mouse hooks (roiEditPress/Move/Release) so the diffraction image can edit the persistent ROI selected in the side-panel combobox. The selected ROI is highlighted (dashed, thicker), and dragging its interior moves a box or circle; on release the new geometry is committed through SetROIDefinition, which recomputes only the ROIs. Resize, azimuthal handles and delete-key follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	0c780a5cbb	Harmonize licsense files	2026-06-19 13:56:09 +02:00
leonarski_f	e49a908862	viewer: ROI panel refinements — persist edits, combobox, compact toggles ... - Edited ROIs now override the file's for every subsequently loaded image and survive settings changes, until a new file is opened (previously a delete/add was lost on the next image). Tracked via an roi_override_ that is re-applied to curr_experiment and to each loaded image's dataset. - The ROI selector is a compact combobox instead of a list widget. - "Show labels" and "Translucent fill" share one row to save space. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	2f8d486b51	viewer: ROI list panel — select, add, rename, delete, per-ROI results ... Add JFJochViewerROIList to the side panel: a list of the dataset's ROIs with selection, add (box/circle/azimuthal), rename and delete, plus a statistics readout (sum/mean/std/max/valid/masked/centre-of-mass) for the selected ROI, taken from the analysis output for the current image. Edits emit a full ROIDefinition, routed to the worker's SetROIDefinition. Per-ROI statistics now live in this panel rather than the canvas labels; the diffraction image's labels show only the ROI name, and the ad-hoc ROIIntegrationCPU computation there is removed in favour of the analysis pipeline. The result widget now reports std dev instead of variance. The single-ROI scratch panel remains for now and will be retired once the interactive canvas editing replaces it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	4e45680e96	viewer: worker can update ROIs and recompute them without full re-analysis ... The reading worker's experiment is the source of truth for ROIs. Add a SetROIDefinition slot that updates curr_experiment.ROI(), rebuilds the analysis ROI engine, mutates the dataset (so the canvas reflects the new ROIs) and recomputes only the ROIs for the current image via RunROIOnly. On image load when full re-analysis is off, ROIs are still computed via AnalyzeROIOnly so the statistics stay current. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	a433399106	image_analysis: surgical ROI-only entry points in MXAnalysisWithoutFPGA ... Add three entry points so ROI statistics can be (re)computed without a full re-analysis, in support of interactive ROI editing in the viewer: - RebuildROI(): recreate the ROI engine after the ROI set changes (the CudaStream is now kept as a member so the GPU engine can be rebuilt). - AnalyzeROIOnly(): decompress + preprocess + ROI, skipping azimuthal integration, spot finding and indexing (a new image when re-analysis is off). - RunROIOnly(): rerun only the ROI integration on the already-preprocessed image (an interactive ROI move). A full Analyze() still computes ROIs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	3996595606	viewer: per-ROI statistics on the diffraction image via the CPU ROI engine ... When ROI labels are shown, each ROI's label now also reports its sum, max and pixel count for the current image. Rather than reimplementing the accumulation, this reuses the existing ROIIntegrationCPU engine (the software counterpart of the FPGA roi_calc), built from the experiment and cached per dataset. A small adapter folds the viewer's gap sentinel (INT32_MIN+1) onto the engine's masked sentinel (INT32_MIN) so masked and saturated pixels are handled correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	33118ebe8b	viewer: fix stray line from origin on full-ring azimuthal ROI ... The full-ring annulus built the inner ring as a separate QPainterPath and merged it with addPath, which stitched a spurious segment from (0,0) to the ring. Build both concentric rings as subpaths of one path, each begun with an explicit moveTo, so there is no connecting segment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	7d6f79a1da	viewer: draw azimuthal ROI wedges, with optional fill and labels ... Azimuthal ROIs now render on the diffraction image as annular sectors (or full-ring annuli), sampled through DiffractionGeometry::ResPhiToPxl so the outline follows the ROI footprint, including wrap-around sectors. Add two side-panel toggles (both default off): a translucent fill for every ROI (helpful when outline colours clash with the image colour map, and with many ROIs) and ROI name labels (constant on-screen size). Wired side panel -> window -> diffraction image like the existing feature toggles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	35f6ffdde9	viewer: draw loaded ROIs on the diffraction image ... When a file with ROI definitions is opened (read into experiment.ROI() by the HDF5 reader), the diffraction image now overlays the configured box and circle ROIs as distinct colour-coded outlines, alongside the existing resolution rings and spots. This is the first step of the ROI-map-based multi-ROI canvas: showing the ROIs loaded from a file. Azimuthal ROIs (wedge rendering) and per-ROI statistics from the bitmap follow in subsequent steps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 13:56:09 +02:00
leonarski_f	332fd32224	frontend: link PSI in footer, add Source code and License to References ... - Footer "Paul Scherrer Institute" now links to https://www.psi.ch - Reference drawer gains a "Source code" entry linking out to the Gitea repository (new nav-item href that opens in a new tab) - Reference drawer gains a "License" entry rendering the nicely formatted docs/LICENSE.md inline (generalises the changelog renderer into MarkdownDoc) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-19 13:23:51 +02:00
leonarski_f	6962a61489	frontend: render Changelog inline instead of in a Sphinx iframe ... The framed Sphinx changelog reused MUI with the same colours, so it read as a picture-in-picture. Bundle docs/CHANGELOG.md at build time (?raw) and render it with react-markdown, themed to match the app, for a quick frame-less view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-19 13:21:19 +02:00
leonarski_f	cee4739221	frontend: bundle Sphinx docs locally instead of linking readthedocs ... The Documentation tab pointed at jungfraujoch.readthedocs.io, which lags behind the installed version. Build the Sphinx docs into frontend/dist/docs and serve them at /frontend/docs so they always match the package. - make_doc.sh: accept an output dir (default public), absolutize it, run from the script dir, and allow $PYTHON override - package.json: add "docs" script building into dist/docs - CMakeLists: run "npm run docs" in the frontend target so CPack packs the docs via the existing dist/ install - App.tsx: point the Documentation DocFrame at /frontend/docs/index.html Also clean up the Sphinx build (911 -> 0 warnings): - fix typo myst_heading_anchor -> myst_heading_anchors so cross-doc #anchor links actually resolve - suppress myst.header/myst.xref_missing (auto-generated OpenAPI client docs only) and the harmless sphinx_material config.cache note - drop a dangling "---" transition at the end of CPU_DATA_ANALYSIS.md - add REPOSITORIES and the generated python_client model pages to toctrees Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-19 12:44:00 +02:00
leonarski_f	eadbce51bd	common: take ROIAzimuthal phi as plain floats, not optionals ... The phi bounds were stored as plain floats internally but passed through std::optional in the constructor, which obscured the intent. Take them as float arguments defaulting to 0 instead. The convention is explicit: phi_min == phi_max means the full ring (all angles), a sector wraps across 0 when phi_min > phi_max, and otherwise phi_min <= phi_max. The "both bounds or none" rule belongs at the API/file boundary (OpenAPIConvert, reader), not in the core class, so it lives there now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 11:03:40 +02:00
leonarski_f	6b95600260	writer: link ROI results into the VDS master file ... In VDS mode the per-image ROI results (max/sum/sum_sq/npixel/x/y) are written into the data files but were not exposed in the master, so a VDS master surfaced no ROI statistics. Add virtual datasets under /entry/roi/<name> in LinkToData_VDS, one group per ROI, mirroring how the spot-finding and azimuthal-integration arrays are linked. Integrated and legacy formats are unaffected (the results are already reachable there). Extended the reader round-trip test to write real ROI results and check they read back from the master for both VDS and integrated formats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 10:38:24 +02:00
leonarski_f	c6e7f8aa15	writer/reader: move ROI definitions to /entry/roi_defs and read them back ... Relocate the ROI definitions and the roi_map bitmap from /entry/roi to a separate /entry/roi_defs group in the master file. /entry/roi keeps its original meaning (per-image ROI results only), so an older reader that iterates /entry/roi is not disturbed by the bitmap/definition entries (which would otherwise make it try to read /entry/roi/roi_map/max and fail to open the file at all). Back-compatibility for already-deployed viewers is the reason for the split. JFJochHDF5Reader now reads the logical definitions into experiment.ROI() and the roi_map bitmask (+ a name->bit index) into the dataset, for all file formats (the master always carries them). Reading both lets a later viewer either re-derive ROIs with the current geometry (logical) or show the exact written footprint (bitmap). Added a write/read round-trip test over VDS and integrated formats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 10:35:18 +02:00
leonarski_f	2bd276ae1b	preview: draw azimuthal ROIs ... AddROI now renders azimuthal ROIs in addition to box and circle ROIs: inner/outer arcs at the Q (d) bounds and, for a sector, the two radial edges at the phi bounds. The per-pixel phi comes from the same DiffractionGeometry::Phi_rad used to assign ROIs, so the overlay matches the ROI footprint exactly (including wrap-around sectors). Full-ring azimuthal ROIs draw as two concentric arcs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 09:44:57 +02:00
leonarski_f	03814b1425	writer: save ROI bitmap and definitions in the master file ... Persist regions of interest so an acquisition's ROI layout can be recovered from the master (_master.h5 / _process.h5): - /entry/roi/roi_map: uint16 per-pixel bitmask (bit i == ROI i). - /entry/roi/<name>: logical definition (type + geometry params, incl. azimuthal phi sector) plus bit_index tying it to the bitmap plane. The bitmap rides along in the StartMessage (CBOR, mirroring az_int_map) and is filled both online (JFJochReceiver::SendStartMessage) and offline (jfjoch_process). Definitions come from the already-transmitted rois. Both are dataset-wide metadata, so they are written by the NXmx master writer (new NXmx::ROI), not the per-image data-file plugin. Documented the /entry/roi layout in docs/HDF5.md and the master-vs-data writer convention in CLAUDE.md; added a CBOR roi_map round-trip test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 09:41:26 +02:00
leonarski_f	1d04656dc8	api: serialize azimuthal ROI phi sector over OpenAPI and CBOR ... Expose the optional phi_min_deg / phi_max_deg azimuthal-sector bounds added to ROIAzimuthal through the wire formats. Both are optional, so older clients and streams (Q-only ROIs) remain valid. - jfjoch_api.yaml: optional phi_min_deg / phi_max_deg on roi_azimuthal; regenerated the cpp-pistache server model and the TS frontend client. - OpenAPIConvert: map phi both directions (set only when present). - CBOR start message: emit phi_min/phi_max only for a sector; on decode, absent phi means a full ring. - Documented the new optional CBOR fields; extended the CBOR round-trip test with a sector and a full-ring azimuthal ROI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 09:28:25 +02:00
leonarski_f	9e57219b31	common: add azimuthal-angle (phi) sectors to ROIAzimuthal ... ROIAzimuthal can now be restricted to an azimuthal-angle sector in addition to its Q/d range, enabling STXM-style directional ROIs. phi bounds are optional (both-or-neither); absent means a full 360 ring, so existing Q-only azimuthal ROIs are unchanged. - ROIElement::CheckROI gains a phi_deg argument; box/circle ignore it. - MarkROI gains an optional phi_map; ROIMap builds it from Phi_rad alongside the resolution map only when azimuthal ROIs are present. - phi bounds are normalized to [0,360) and wrap-around sectors are supported (phi_min > phi_max). - ROIConfigAzim carries phi_min/phi_max (kept trivially copyable for the union; phi_min == phi_max means full ring). No phi crosses the wire yet (CBOR/API wiring follows separately). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 09:21:39 +02:00
leonarski_f	88f97fc197	docs: add CLAUDE.md with build/test/architecture, code-style and portability guidance ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 08:11:07 +02:00
leonarski_f	a76aac65b7	frontend: add missing React keys in DetectorSelection and ROI ... DetectorSelection rendered the info-table rows and the detector-select menu items without keys; ROI's grid action cells likewise. Add stable keys to silence the warnings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 08:01:03 +02:00
leonarski_f	4f9452736a	frontend: drawer-based navigation shell ... Replace the stack of toggle-revealed sections with a left navigation drawer (collapsible, expanded by default) clipped under a persistent status bar. Dashboard (plots + measurement statistics + spot-finding) is the default view; every other panel is its own drawer entry, so one thing shows at a time. - Group entries under Acquisition / Expert / Reference subheaders; ROI folded into On-the-fly processing. - Pale-indigo drawer with a lime accent on the selected entry. - Gray out JUNGFRAU calibration / FPGA status when their data is empty. - API reference and documentation open in-app via iframe. - Fixed footer pinned to the page bottom. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 07:57:57 +02:00
leonarski_f	9fc971c1d2	frontend: add zod plugin to hey-api openapi-ts ... Generate Zod schemas alongside the typed client (src/client/zod.gen.ts), picking up the OpenAPI field defaults. Enables runtime validation and default-filling for form payloads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 07:24:52 +02:00
leonarski_f	5d8e33d52c	frontend: orange Upload button when panel is dirty ... Match the unsaved-changes dot: the Upload button uses the warning color while the panel has unedited changes, primary otherwise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 22:14:10 +02:00
leonarski_f	59da040e30	frontend: shared SettingsPanel + typed upload with error display ... Generalize the expert-panel boilerplate (container + title + widgets + upload button) and surface real upload errors via hey-api. - SettingsPanel: consistent padded Paper + Typography title (one place to size it) + optional Upload button + result snackbar; PanelTitle and UploadSnackbarView exported for panels with bespoke layouts (DetectorSettings grid, ROI) - useUpload(mutation): wraps a generated hey-api mutation, shows the server's error message on failure (errorMessage handles 400 string / 500 error_message) - "unsaved changes" dot on panel titles, derived from the existing last-downloaded snapshot (dirty = !isEqual(value, lastDownloaded)) - fix PixelMask: TIFF upload was swallowing errors; now shows them - config panels (FileWriter, ImageFormat, Detector, AzInt, DarkMask, Indexing, Instrument, ZeroMQ, ROI, DetectorSelection) upload through the typed SDK instead of ButtonWithSnackbar's raw fetch; status/display panels share the same shell for consistent spacing - ButtonWithSnackbar now only powers bodyless action buttons (pedestal/cancel/ initialize, start/trigger, deactivate) Build (tsc + vite) passes; dev server transforms all modules. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 22:04:55 +02:00
leonarski_f	c000b872bb	jfjoch_viewer: Fix status_bar, it looks nicer now	2026-06-18 21:59:10 +02:00
leonarski_f	237de5a0fc	frontend: convert to function components + TanStack Query ... Replace all class components with function components and move server-state polling to @tanstack/react-query. - add @tanstack/react-query; generate query helpers via the hey-api @tanstack/react-query plugin (src/client/@tanstack) - QueryClientProvider + one-time client.setConfig({ baseUrl: '' }) in index.tsx - App polls statistics with useQuery(getStatisticsOptions, refetchInterval), DataProcessingPlot uses getPreviewPlotOptions; manual setInterval polling gone - memo() on presentational children; with TanStack structural sharing this re-renders a child only when its own statistics slice changes - PreviewImage stays imperative (binary JPEG -> object URL) using useEffect + a ref for the object-URL lifecycle - fix AzIntSettings correction checkboxes that mutated state in place (relied on the poll re-render); they now use setState - drop dead code uncovered during the port (unused upload/deactivate handlers and imports in ImageFormatSettings, DetectorSettings, ROI) Build (tsc + vite) passes; dev server transforms all entry modules. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 21:32:36 +02:00
leonarski_f	7ed424a4ca	JFJochReceiverFPGA: Fix typo	2026-06-18 20:48:19 +02:00
leonarski_f	ceb7d31eac	frontend: generate OpenAPI client with @hey-api/openapi-ts ... Replace the unmaintained openapi-typescript-codegen generator with @hey-api/openapi-ts. - openapi-ts.config.ts generates src/client from ../broker/jfjoch_api.yaml (client-fetch + typescript + sdk plugins; enums as const objects via mode: 'javascript', type names preserved) - npm "openapi" script now runs openapi-ts; drop openapi-typescript-codegen - migrate all imports from src/openapi to src/client - DefaultService.foo() -> standalone foo({ body|query, throwOnError }) - OpenAPI.BASE='' -> client.setConfig({ baseUrl: '' }); ButtonWithSnackbar uses the relative path directly - adapt hey-api's inlined enums: broker_status.state/message_severity and bit_depth_image/eiger_bit_depth become string/number literal unions; file_writer_format NXMX_* -> N_XMX_* - remove generated src/openapi Build (tsc + vite) passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 20:30:15 +02:00
leonarski_f	eb9b463263	VERSION: 1.0.0-rc.153	2026-06-18 19:26:15 +02:00
leonarski_f	b71da6238e	Clean-up unused tools	2026-06-18 18:54:31 +02:00
leonarski_f	d14b8e9e18	Merge upstream	2026-06-18 18:52:22 +02:00
leonarski_f	a7c77f1f08	internal generator: compare raw image width to RAW_MODULE_COLS ... The raw stacked-module layout check compared the image width (cols) against RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 16:09:05 +02:00
leonarski_f	0f5271f14c	mask: accept any-bit-depth TIFF for user mask upload ... PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks exported by tools like PyFAI (8-bit) failed. Route the upload through the universal ReadTIFF reader and let PixelMask take a CompressedImage directly: it validates the 2D shape against the detector's converted/raw layouts, binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects float/multi-channel images. Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over ReadTIFF, and the now-unused ReadTIFFFromString32 is removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 16:05:04 +02:00
leonarski_f	57ae145555	writer: drop unused <sys/stat.h> include from HDF5DataFile ... Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies on <filesystem>. Removing the dead POSIX include also avoids a needless non-portable header for the eventual MSVC build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 15:02:39 +02:00
leonarski_f	bee35f7a25	frame_serialize: write CBOR length big-endian without bswap intrinsic ... AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length, which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN. That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the swap and emitted a little-endian length on a little-endian host -> corrupt frames. Write the 8 length bytes most-significant-first directly. This is big-endian by construction on any host, needs no byte-swap intrinsic and no endianness macro, and is byte-for-byte identical to the previous output on little-endian hosts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 15:02:39 +02:00
leonarski_f	5d9e1be814	portability: replace M_PI / std::numbers::pi with a host+device-safe PI ... M_PI is a POSIX <math.h> extension that MSVC does not define without _USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several common/ headers are pulled into .cu device translation units, so std::numbers is not available there. Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works in host code (including MSVC), in CUDA device code, and under C++17/20, and use it everywhere: - common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files - broker/OpenAPIConvert.cpp - viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent convention across the codebase Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:54:01 +02:00
leonarski_f	d0a1175c65	Clean-up of OpenAPI code	2026-06-18 14:29:53 +02:00
leonarski_f	1558dddbb8	indexing: make the resolved-algorithm invariant explicit in the pool ... IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(), which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None; the pool has no policy to resolve Auto itself. The worker handled a stray Auto with a dead branch and silently produced no result when the resolved algorithm had no matching indexer built. - Document on GetIndexingAlgorithm() that it never returns Auto. - Throw a clear internal error at the pool boundary if Auto ever arrives. - In the worker, replace the dead Auto branch with a loud failure for any resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a host without a GPU), instead of returning no result silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_f	01a97d61fe	common: spell the azimuthal d-spacing unit d_A consistently ... PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is left as-is (controlled by the OpenAPI generator's C++ enum casing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_f	52f349b9f1	frame_serialize: use reentrant gmtime in CBOR date decoding ... gmtime() returns a pointer to a shared static buffer, so concurrent deserialization of CborUnixTime_tTag values was a data race. Decode into a local struct tm via gmtime_r (gmtime_s on Windows) instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:25:30 +02:00
leonarski_f	086129f767	receiver: count only images the pusher accepted ... images_sent was incremented right after image_pusher.SendImage(*loc), but the ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it silently drops the image (releases the slot and returns) when there is no live connection or the 2 s enqueue deadline expires. So images_sent over-counted on a broken/slow writer connection and disagreed with the ACK-based GetImagesWritten(). Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted (enqueued/handed off) and only increment images_sent on success. The slot is still released on the drop path. The authoritative delivered count remains GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on accept, preserving their previous always-counted behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 14:24:48 +02:00
leonarski_f	e1622c3750	plots: expose compression_ratio over the REST/UI plot path ... The receiver already computes and stores the compression ratio per image (JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio), but the type was unreachable from the API: it was missing from ConvertPlotType, the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and no UI menu entry existed. Wire it through all layers: - broker: add the compression_ratio -> PlotType::CompressionRatio case in ConvertPlotType. - spec: add compression_ratio to the plot_type parameter enum. - frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what regeneration would produce), a "Compression ratio" menu entry, and a Y-axis label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:52:41 +02:00
leonarski_f	ec1308f4a9	viewer: guard Linux-only constructs for Windows/MSVC builds ... Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux build: D-Bus stays on by default and the XCB plugin is still used. - CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/ adaptor + service file only when enabled. Select the platform integration plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead of hard-coding QXcbIntegrationPlugin. - JFJochViewerWindow: wrap the adaptor include and D-Bus registration in #ifdef JFJOCH_VIEWER_DBUS. - JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now #ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only headers are guarded too. - Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the viewer (not defined by MSVC without _USE_MATH_DEFINES). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:50:49 +02:00
leonarski_f	b29b870ef2	broker/gen: gitignore generated cpp-pistache-server output ... Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is hand-written on cpp-httplib, so the generated Pistache server stubs (api/, impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are regeneration debris that was previously left untracked and tended to carry a newer spec version than the committed models. Ignore it so it can't be committed by accident. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 13:45:04 +02:00
leonarski_f	56ddfaef96	viewer: live detector status bar + dataset-follow sync mode ... Add a status-bar cluster, shown when connected over HTTP, that surfaces the broker state and live acquisition info: - broker state box (QProgressBar) with progress drawn as the bar fill, polled ~1 Hz on its own timer independent of image sync - Live/Disconnected connection badge (host:port in tooltip) - "+N new" badge and effective live-rate (Hz) readout All widgets are fixed-width and only blanked in place, so the bar never reflows when things appear/disappear. Add a third autoload mode (HTTPSyncDataset): manually selecting an image while following live now freezes the displayed image but keeps the dataset, plots and image count updating. Surfaced via a tri-state HTTP Sync button (off / live / data-only). Also fix GetBrokerStatus() dropping the message field and a couple of copy-pasted exception strings in JFJochHttpReader. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-18 08:13:07 +02:00
leonarski_f	188cbb659d	tests: add GPU/CPU equivalence test for ROI integration ... Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks), negative pixel values (signed weighted-sum path), and an injected saturated and masked pixel per ROI to cover the "max only" and "fully excluded" branches. Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present, mirroring ImageSpotFinderGPUTest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 22:28:42 +02:00
leonarski_f	c0a4801bc4	TCPStreamPusher: fix zero-copy buffer reuse and make send/END timeouts progress-based ... Three robustness fixes for the writer-facing TCP stream, addressing the spurious "wrong number of images" / connection failures seen under load. 1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path passes a caller-owned buffer with z == nullptr and reuses it for the next frame immediately. With MSG_ZEROCOPY the kernel still references that buffer after send() returns, so the peer could receive corrupted frames and drop the connection mid-stream, truncating the run. Zero-copy is now gated on a ZeroCopyReturnValue that keeps the buffer alive until completion. 2. Make the SendAll watchdog measure lack of progress, not total wall-clock. The previous absolute deadline tore down a healthy but back-pressured connection (slow/starved writer) after a fixed window; the watchdog now resets on every byte actually sent, so only a genuinely stuck socket is closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR. 3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still be draining a backlog of DATA frames when END is sent; each DATA ACK is progress, so the timeout only arms once the writer falls silent rather than firing while images are legitimately still being drained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 22:02:09 +02:00
leonarski_f	58910274bf	image_analysis: compute ROI statistics in the non-FPGA path ... MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were only available on the FPGA path. Add a software ROI engine that mirrors the FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so it can contribute to any subset of up to 16 ROIs. New image_analysis/roi/ library (JFJochROIIntegration), structured like azint: a base that precomputes the per-pixel mask and names, a templated CPU engine (generic over pixel type for a future 16-bit path), and a GPU kernel using per-block shared-memory atomics for the STXM case (half-detector ROIs). Masked pixels are excluded entirely; saturated pixels are excluded from the sums but still count towards the max, matching roi_calc exactly. The engine is only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already consume message.roi, so no further changes are needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 21:15:54 +02:00
leonarski_f	cc925b2668	Remove NUMAHWPolicy and the libnuma dependency ... NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are placed device-local by the kernel (dma_alloc_coherent), the big RAM ring buffer is random-access (first-touch handles placement), and GPU work is already spread across all visible devices. So drop the pinning entirely and with it libnuma. - Delete NUMAHWPolicy; the only concern worth keeping - GPU selection - is done directly via pin_gpu() (round-robin over visible GPUs) in the indexer pool and the Lite analysis threads. CPU-only threads (FPGA acquire/pedestal/summation/frame-transform) no longer bind anything. - Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA used it. - numa_policy broker setting is deprecated and ignored (kept in the API for backward compatibility; warns once on startup). - Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake. - Docs: drop the NUMA dependency, remove the numa_policy config example, and document running multiple brokers on disjoint GPUs via CUDA_VISIBLE_DEVICES. - Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 20:25:36 +02:00
leonarski_f	17732ea774	docs: refresh NUMA_GPU_REVIEW to match implemented state ... Several items in the note had landed or were inaccurate. Update it to: - mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device de-NUMA, CUDAWrapper sysfs node lookup; - add the previously-missing FPGA DMA buffer section - placement is a kernel concern (dma_alloc_coherent, device-local), not the userspace mbind, which was only the simulator; - record that libnuma is now down to a single file (NUMAHWPolicy.cpp); - note that dependency removal (G3) and pinning behaviour (G4) are separable axes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:22:00 +02:00
leonarski_f	040cdeacf1	acquisition_device: give each device sole ownership of its buffers ... The base AcquisitionDevice no longer allocates or frees frame buffers; buffer_device is now just a non-owning view of addresses. Each subclass owns its backing memory and the matching lifecycle: - PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its own destructor (and on ctor failure), symmetric with MapKernelBuffer. - HLSSimulatedDevice owns plain zeroed heap buffers it points buffer_device into, declared before the HLSDevice so they outlive the action thread that writes them. The buffers are page-aligned to match the real device's kernel DMA buffers - the modelled AXI datamover and FPGAIntegrationTest require aligned output buffers. This drops the NUMA/mmap dance from the simulated path (not performance-critical) - removing libnuma from acquisition_device - and replaces the base-class cleanup that had to guess the allocation strategy with a single clear owner per device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 19:18:32 +02:00
leonarski_f	d373ba0490	ImageBuffer: replace NUMA interleave with parallel first-touch ... Allocate the ring buffer with plain malloc and zero it across hardware_concurrency() threads, so each page is first-touched - and thus NUMA-placed - by whichever node the scheduler ran the zeroing thread on. For the random-access buffer this approximates the previous numa_alloc_interleaved placement, speeds up the one-time fault-in of the 150-200 GB allocation, and drops the libnuma dependency from this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 17:03:13 +02:00
leonarski_f	02fa15c2b9	jfjoch_process: spread per-image GPU work across all visible GPUs ... The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA device, so all per-image preprocessing/spot-finding/azimuthal integration ran on GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before building the analysis resources so their CUDA streams/engines land on distinct devices. Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU dispatch with goals and a staged plan (multi-broker GPU isolation via CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:29:52 +02:00
leonarski_f	1056acc3a6	build: guard the librt link to Linux only ... librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a $<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and the CUDA block) so -lrt is still passed on Linux - where some distros need it for clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the jfjoch_process link line on Linux. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:40:09 +02:00
leonarski_f	39c808776f	build: portability groundwork toward a Windows/MSVC viewer ... - CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process). - common/NetworkAddressConvert.cpp: guard the network includes for _WIN32 (winsock2/ws2tcpip vs arpa/inet). - common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) - portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 08:37:21 +02:00
leonarski_f	c69b5297d5	docs: add jfjoch_process page, refresh viewer/tools docs, unify CLI naming ... - Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its options, output files, and the broker/viewer/process distinction; mention jfjoch_scale for re-scaling/merging. - Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API. - Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst. - jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in usage/license/logger. - jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D short flag, make it long-only) and remove dead p/q/i short options. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 07:38:36 +02:00
leonarski_f	3334e88d3e	frontend: fix state bugs, slim Plotly bundle, drop CRA leftovers ... - App: make getValues an arrow field so its `this` survives being passed as the `update` prop (the post-edit statistics refresh was broken). - DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual so the 1 s statistics poll no longer overwrites a slider mid-edit (reference `!=` on a freshly-parsed object was always true → snap-back). - DataCollection: collapse nested setState in grid-scan callbacks. - PreviewImage: build an immutable settings copy instead of mutating state in place; fix stale comment. - MultiLinePlotWrapper: drop per-render console.log; render via a custom Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only), cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip). - package.json: remove dead react-scripts test/eject scripts and CRA eslintConfig, move @redocly/cli to devDependencies. - Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts, setupTests.js). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:51:45 +02:00
leonarski_f	638aa379f4	Fix	2026-06-16 20:28:25 +02:00
leonarski_f	704e4b4eb3	Documentation: Data analysis description	2026-06-16 20:24:01 +02:00
leonarski_f	5143ba0b1f	jfjoch_api.yaml: header_appendix hdf5 values go to /entry/user, not /entry/data ... The description said the "hdf5" sub-object is written under /entry/data, but the writer (NXmx::UserData) and the example in the same description both use /entry/user. Correct the path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:00:40 +02:00
leonarski_f	e8d3eb1b08	docs/HDF5.md: generalize to rotation+serial, add process.h5, user_data, ITC link ... - Motivation no longer frames the format as serial-only: Jungfraujoch serves both rotation and serial MX, and the two-layout rationale (per-image CXI spots vs. dataset-wide NXreflections) is presented for both. - Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an integrated-format master whose /entry/data/data is a VDS linking back to the original images (all results, no image copies). - Add a section on the header_appendix / image_appendix (user_data) mechanism: how the free-form JSON flows through the start/image CBOR messages and how an "hdf5" sub-object is persisted to /entry/user, with an example. - Link the niggliClass row to International Tables for Crystallography A, Table 3.1.3.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:47:36 +02:00
leonarski_f	0381d891bc	docs: add HDF5.md describing the NeXus layout and JFJoch extensions ... New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout, NXreflections for integration), the master/data-file layout and the three NXmx format variants, the NXmx-standard fields that are populated, and every Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint, /entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel, detectorSpecific, calibration, fluorescence, user). Content is derived from writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked against the NXmx and NXreflections definitions. JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to the new doc; HDF5 is added to the Sphinx toctree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:36:44 +02:00
leonarski_f	caff857d8a	docs/CBOR.md: sync field names/types with the serializer ... Bring the protocol doc in line with CBORStream2Serializer/Deserializer: - reflection dist_ewald -> rp (dist_ewald is the spot field, not the reflection field) - az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count - threshold_energy typed as Map(string -> float), not float - end_date typed as string (it is serialized as a plain text string, with no datetime tag, unlike arm_date) - add missing reflection fields: phi, zeta, image_scale_corr - add missing spot fields: image, and indexed-only h, k, l, dist_ewald - add image indexing_lattice_count and end indexed_lattice_count - drop the duplicated image_scale_factor row in the End table Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:11:37 +02:00
leonarski_f	6479b91e50	JFJochHDF5Reader: read master indexedLatticeCount (back-compat) ... The master-file writer emits /entry/MX/indexedLatticeCount, but the reader only looked for indexingLatticeCount (the name used by the per-file MX plugin and data-file read path). Existing master files on disk therefore returned no indexed-lattice-count vector. Read the master's indexedLatticeCount first, falling back to indexingLatticeCount, matching the niggli_class/niggliClass compatibility pattern. The writer is left unchanged so already-written files keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 19:03:34 +02:00
leonarski_f	2ba28aea0e	LoadFCalcFromMtz: fall back to an intensity column when F-model is absent ... PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN, which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and use it directly as I_ref (F-model is squared, intensities are not). This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional integration, or PixelRefine against an external reference) to produce a merge, then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's own intensities - free of the reference structure's non-isomorphism bias - and iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine runs against it (crystal 2, 55279 reflections, CC1/2 94.3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 14:19:28 +02:00
leonarski_f	545ebdf868	Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf) ... CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial data. Each image is assigned to one CC1/2 half, so removing it perturbs only that half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i). A negative value means dropping the image RAISES CC1/2 (it disagrees with the consensus). Images whose deltaCChalf is a low-side statistical outlier (< mean - N*stddev) are skipped when merging. Reference-free. Two passes over the retained integration outcomes; per-image contributions are re-derived rather than stored, so memory stays O(unique reflections + images) for full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off). Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2 95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS heterogeneous serial data (some crystals genuinely bad) but slightly trims a homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative cut still removes the tail), so it is opt-in. R-free is the real test (user's full 200k). Note: the reported overall N_obs still counts all observations; the exported merge (and CC1/2) correctly exclude the rejected images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 12:32:52 +02:00
leonarski_f	f878fb9d5d	SpotFindingSettings: Default signal to noise ratio is 4	2026-06-16 08:53:05 +02:00
leonarski_f	929e6e1fa0	FFTIndexer: pick peaks by prominence above a local background, not raw magnitude ... The per-direction peak search took argmax|spec|. The projected-spot histogram has a broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can exceed the true lattice peaks, so on weak / pink-beam frames every direction reported a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors, the true axes entirely absent.) Subtract a running-mean background of half-width ~15 A and pick the peak by its PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes win. The prominence is also reported as the magnitude, so FilterFFTResults ranks directions by real-peak strength rather than envelope. Ported identically to CPU (prefix-sum window) and GPU (sliding-window in the kernel). Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX 27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged). The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes the envelope). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:25:06 +02:00
leonarski_f	f2f95c44f6	FFTIndexer: name the histogram axis one_over_d (1/d), not q/Q ... The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d convention) while the bin width and spot coordinates use the internal 1/d convention - a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x zero-padding of the histogram. Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d is named one_over_d (never q). histogram_size is numerically identical to before, so behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi to a nearby integer shifts indexing ~0.5-1%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 07:24:59 +02:00
leonarski_f	e1e2ca8e49	IndexAndRefine: unify cell setting across indexers, keep FFBIDX neutral ... Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when cell+SG are given, so every frame reports its cell in ONE consistent setting per space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same reflection as different HKLs and cannot be merged). Use LatticeSearch's conventional cell when its detected symmetry agrees with the user's space group. On noisy frames LatticeSearch can pick an alternative Bravais setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell, [78,78,38]->[110,111,38]) whose system disagrees; there: - FFBIDX already returns the reference setting (c-last), consistent with the conventional frames, so its raw lattice is safe -> use it. This recovers the ~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral. - de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order (c-first); merging it with the conventional (c-last) frames would corrupt the data -> reject the frame instead. Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last, consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis (~38A) is axis 2 in every merged frame - no axis-order mixing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:50:56 +02:00
leonarski_f	40da1aab13	IndexAndRefine: conventionalise de-novo lattice when cell+SG are given ... Providing both a unit cell (-C) and space group (-S) silently broke the de-novo indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the `sg && GetUnitCell()` branch fed the indexer's raw lattice straight into symmetry-constrained refinement. FFBIDX returns the lattice already in the reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive cell; enforcing the crystal system on its mis-assigned axes rejects every frame. Fix: for de-novo indexers only, reduce the lattice to the conventional setting (LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so it is byte-identical to before (no regression). niggli_class stays unassigned (0) in this branch - it is a property of the primitive cell incl. centering, which LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2) cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred. Validation (lyso, -C79,79,38 -S96): crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical (Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no consensus on the weak serial-still frames, 0% even without -S; to investigate.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:35:03 +02:00
leonarski_f	ecdb7048a0	PixelRefine: drop crystal-system idealisation, use the indexed cell as-is ... PixelRefine no longer refines the cell, so the symmetry-constrained cell parameterisation inherited from XtalOptimizer has no manifold to constrain. The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B matrix) merely reconstructed the lattice columns it started from, and for non-triclinic systems it re-idealised the indexed cell (averaging a,b for tetragonal; a,b,c for cubic; forcing alpha=gamma=90). Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a single path: take the three real-space lattice columns (latt.Vec0/1/2()) directly and form the reciprocal node via the general cross-product inverse. This reproduces every crystal system exactly from the actual cell, is more faithful (no re-idealisation), and removes the crystal_system field plus two now-unused includes. PredictedNode de-templated (only ever called with double). Crystal symmetry still lives where it belongs: indexing (upstream) and merging (downstream via the space-group HKL key). A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the old idealisation was already a no-op): stat tables bit-identical across all shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% -- the only delta is 4th-digit float-ordering noise in the error-model b coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free unchanged. Net -81 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 14:11:26 +02:00
leonarski_f	8a9d80eb71	PixelRefine: profile multiplier as --profile-multiplier flag; outlier rejection default off ... (1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process --profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:48:11 +02:00
leonarski_f	102a2a7c81	PixelRefine: strip experimental env knobs (orientation, sweep, Lorentz, ML census) ... Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed, returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper (r1_multiplier, default 6): - PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) - XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual, OrientationRegularizer, PixelObs::weight, the refinement block and its fields. - PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2 (per-image overfit). Removes SweepOrientationCell and its fields. - PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality already subsumes it); was already reverted. - PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5% of jet frames are multi-lattice; shelved). PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged: crystal 2 / jet / hybrid -R -r pixelrefine all reproduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 13:34:54 +02:00
leonarski_f	6f733d74c2	Merge: make outlier rejection a ScalingSettings parameter + CLI flag, default 6 ... Promote the per-observation merge outlier rejection from the temporary PR_REJECT env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0 disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 -> 0.2585). Applies to both the PixelRefine and classical merge paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 12:42:25 +02:00
leonarski_f	a47b376dc3	Merge: per-observation outlier rejection (env-gated PR_REJECT) ... At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every merged mean. Add a robust per-observation cut: drop observations whose corrected intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN. The error-model sigma already captures the genuine (partiality) scatter, and the median is a robust centre, so only the tail beyond the real spread is removed - not good partials. The median is computed in RefineErrorModel (which already pools the observations per reflection); AddImage applies the cut. Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut) - the cut is vs the data's own median, not the reference, so the gain is real cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at low nsigma) are for Filip's full-jet R-free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:56:53 +02:00
leonarski_f	45ee8c2b40	PixelRefine: env-gated orientation + cell-scale sweep (PR_SWEEP) for R-free A/B ... R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on R-free; x9 only buys CC1/2 internal consistency), and per-image orientation *refinement* is a no-op (0.2618 vs 0.2625). Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP (off by default) to test whether the cell-scale degree of freedom - which the gradient orientation refinement lacks, and which moves the high-res shells radially - helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for the gradient path), but that does not certify R-free, so it is left for validation. NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 19:59:50 +02:00
leonarski_f	cfcd4c9e56	PixelRefine: R1 multiplier (fix tight-Term2 regression) + env-gated orientation refine ... Full-jet R-free showed the factored model regressed vs the traditional pipeline (R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix, XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier (default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3 (~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free. Also reinstates the pre-factored per-image orientation refinement (per-pixel ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation, re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation. NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier value and the orientation-refine decision are to be fixed against R-free, then baked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 09:18:09 +02:00
leonarski_f	bf6efc7fe9	Integration radius: default r1=4, CLI flag, PixelRefine shares the knob ... Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet (1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots spill past a radius-3 box), is neutral on the mono crystal and on the classical integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob. (Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res, azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured 2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight measured width loses to a generous aperture, same lesson as the radius bump.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 23:27:36 +02:00
leonarski_f	100fe7b7e7	PixelRefine: make factored Terms 1+2 the model, remove old wiring ... PixelRefine is now an intensity-only operation: geometry is fixed (refined upstream by XtalOptimizer) and the only objective is the factored per-reflection likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills, 1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and matches the rotation path from the stills path. Removed: - the per-pixel ShoeboxResidual loss and PixelResidual cost functor; - all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R), the regularised-orientation LSQ, signal-weighting, and the global sweep; - Term 3 (per-spot recentring) - a confirmed no-op on both crystals; - the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the PR_* env knobs + stderr dumps in IndexAndRefine; - the PredictImage/ChiSquaredImage renderers and the entire viewer PixelRefine window/table/params + worker bindings + shoebox overlay. The sweep box-integrator background median became mean (consistency) by virtue of removing the sweep. METHODS.md rewritten for the current model; findings recorded in FINDINGS-2026-06.md. Net -2200 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 22:02:18 +02:00
leonarski_f	6f2033db00	PixelRefine: checkpoint before cleanup (factored model + all diagnostic levers) ... Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_* env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/ ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit makes Terms 1+2 the model and strips all of this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-13 21:34:33 +02:00
leonarski_f	c93d381dc8	Error model: harden the fit against pathological inputs (code review) ... Addresses code-review findings on RefineErrorModel: - Floor the 1/dev^2 bin weight relative to the data scale (1e-3 of the median bin dev^2), not an absolute 1e-30: a near-zero-scatter bin could otherwise acquire a runaway weight and hijack the global (a,b) fit. - Reject a near-collinear normal-equation system relatively (det > 1e-10*Ass*AII) instead of with an absolute threshold that an ill-conditioned fit can pass. - Reset the model to identity at entry so any early return leaves it inactive rather than keeping a stale a/b alongside a freshly-cleared mean map (which would make CorrectedSigma fall back to the per-observation I). - PixelRefine: correct the orient_prior comment - with the sweep on, the LSQ anchor is the swept orientation (intended), not the spot-centroid one. Verified unchanged on the lyso test set (ISa 1.1, CC1/2 90.3%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:53:02 +02:00
leonarski_f	e6a50b45c7	Integration: mean background + global error model (trustworthy sigmas) ... Background estimate: use the mean of the local ring, not the median. For a right-skewed (Poisson) background the median sits below the mean, so subtracting it under-subtracts and biases every weak intensity positive; over multiplicity this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in both PixelRefine and BraggIntegrate2D (the classical route had the same bug). <I/sig> now tracks CC honestly and the true resolution limit is visible. Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the scatter of symmetry equivalents at the merge level (so both integrators benefit), and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the reflection mean (not the per-observation I_i, which biases the weights and collapses CC); a,b come from a relative-weighted bin regression. Replaces the earlier per-resolution-shell variant, which was partly masking the background bias. METHODS.md: document both (Sections 6-7), integrator-agnostic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 18:39:32 +02:00
leonarski_f	3cdf26232c	PixelRefine: Document on local changes	2026-06-12 17:29:28 +02:00
leonarski_f	db68c8dc38	PixelRefine: Results seem to be much better	2026-06-12 17:28:18 +02:00
leonarski_f	47dc19dd03	PixelRefine: Improvements to accept more reasonable count of reflections	2026-06-12 10:12:23 +02:00
leonarski_f	c8db50ab41	jfjoch_viewer: missing pixel refine table	2026-06-12 07:47:28 +02:00
leonarski_f	d31063ca3f	PixelRefine: Some improvements	2026-06-11 21:25:05 +02:00
leonarski_f	48d4fb0d0f	XDS plugin: Fix mutex	2026-06-11 18:35:02 +02:00
leonarski_f	bd5fef7f61	PixelRefine: Simplify (remove Lorentz correction, remove background from azimuthal integration)	2026-06-10 18:37:46 +02:00
leonarski_f	7478c0390f	jfjoch_scale: Remove postrefinement model option	2026-06-10 16:37:15 +02:00
leonarski_f	59281f6330	Revert "Bragg integration: option to use azimuthal integration profile" ... This reverts commit e4230bc14e.	2026-06-10 16:36:37 +02:00
leonarski_f	579d36fe71	Revert "jfjoch_process: Add option to use azimuthal integration as background for Bragg integration" ... This reverts commit b22d5929a1.	2026-06-10 16:36:37 +02:00
leonarski_f	b22d5929a1	jfjoch_process: Add option to use azimuthal integration as background for Bragg integration	2026-06-10 14:52:58 +02:00
leonarski_f	e4230bc14e	Bragg integration: option to use azimuthal integration profile	2026-06-10 14:41:36 +02:00
leonarski_f	e051eed033	jfjoch_process: Remove postrefinement	2026-06-09 19:46:39 +02:00
leonarski_f	efe882f4b6	jfjoch_viewer: Better display (to be tested) of pixel refine	2026-06-09 16:28:17 +02:00
leonarski_f	6c85aaba2b	BraggPrediction: Include X-ray bandwidth	2026-06-09 15:01:44 +02:00
leonarski_f	6af22b6a0c	jfjoch_viewer: show image CC based on standard pipeline	2026-06-09 13:28:56 +02:00
leonarski_f	003fea1b1e	jfjoch_viewer: fix sorting by indexing status	2026-06-09 13:28:35 +02:00
leonarski_f	2d202f1d44	jfjoch_viewer: fixes to pixel refine window	2026-06-09 13:20:46 +02:00
leonarski_f	feca63f4b9	jfjoch_viewer: fixes to pixel refine	2026-06-09 12:35:06 +02:00
leonarski_f	8a582b8a90	JFJochMagnifierWindow: Fixed	2026-06-09 12:09:40 +02:00
leonarski_f	30dcc98f89	JFJochMagnifierWindow: Zoom is saved ... it is not optimal (when image is first loaded, than it starts with weird zoom), but can be fixed later	2026-06-09 12:03:45 +02:00
leonarski_f	5735302691	Merge: CC1/2 limit adjustment	2026-06-09 11:06:16 +02:00
leonarski_f	e4b66f9cd9	PIxelRefine: Another iteration	2026-06-09 08:04:27 +02:00
leonarski_f	254462b9f2	jfjoch_process: Clarify what happens in scaling when reference data provided	2026-06-09 07:28:13 +02:00
leonarski_f	e8a9b1840d	PixelRefine: Make it faster by doing one cell calculation per shoe-box	2026-06-08 22:45:22 +02:00
leonarski_f	05711a1077	jfjoch_viewer: Add pixel refinw and magnifier windows (to be tested)	2026-06-08 21:22:19 +02:00
leonarski_f	698a98be35	PixelRefine: Claude fixed my bugs	2026-06-08 20:06:53 +02:00
leonarski_f	32e91f7287	Minor fixes	2026-06-08 15:56:20 +02:00
leonarski_f	96edee9b2d	jfjoch_writer: Add incident_wavelength_spread	2026-06-08 15:47:12 +02:00
leonarski_f	155c53acd8	jfjoch_process: First pixel refine integration	2026-06-08 15:37:27 +02:00
leonarski_f	6f31159607	PixelRefine: Add bandwidth contribution	2026-06-08 13:59:01 +02:00
leonarski_f	6f6098d508	PixelRefine: Work in progress (Claude)	2026-06-08 13:44:39 +02:00
leonarski_f	66a48c4266	PixelRefine: Work in progress (Claude)	2026-06-08 12:43:10 +02:00
leonarski_f	9a991b6614	PixelRefine: Work in progress	2026-06-08 11:59:48 +02:00
leonarski_f	0d6b278767	LatticeReduction: Move outside of XtalOptimizer	2026-06-08 11:11:35 +02:00
leonarski_f	50f953bcc5	PixelRefine: Work in progress	2026-06-08 09:41:58 +02:00