The image-reading worker (on its own thread) posted imageLoaded — a full
~40 MB JFJochReaderImage on a 9 Mpix detector — to ~13 GUI consumers over
queued connections, paced only by its own load+analysis time. When the GUI
render (image + azimuthal integration + plots) was the bottleneck, those
queued events piled up without bound and memory blew up.
Funnel every consumer through one window signal: the worker's imageLoaded
crosses the thread boundary exactly once, into JFJochViewerWindow::
OnImageReady, which re-emits imageReady to all GUI consumers synchronously
(same thread) and then acks the worker (imageConsumed). The worker tracks
images_in_flight and AutoLoadTimerExpired stops producing once the cap
(max_images_in_flight = 4) is reached — applied to every auto mode.
Live follow drops intermediate frames (next fetch grabs the newest image);
movie playback paces down to GUI speed without skipping. The existing
adaptive autoload interval is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the biggest user-facing changes the entry was missing: the trimming-energies
config change for PSI EIGER (now compulsory), ROI definitions/bitmap in the master
file with azimuthal phi sectors, and the viewer redesign + in-GUI jfjoch_process jobs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A writer connection that dropped mid-run left a truncated file, yet the
acquisition was still logged as "finished with success": the writer error
returned by ImagePusher::Finalize() (set via transmission_error on a broken
session connection) was captured into receiver_output.writer_err but never
acted upon.
Treat a non-empty writer_err as a failed acquisition: throw so it is reported
as "finished with error" and surfaced to the caller, instead of silently
succeeding with incomplete data. Applies to both the Lite and FPGA workflows,
which share this stop/finalize path. Combined with the liveness changes (which
no longer tear a connection down for a transient stall), this fires only on a
genuine writer break.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The broker logs for the dropped runs show the connection torn down ~2s into
a collection (not 3s), via "TCP send failed -> Removed dead connection ->
Accepted (new socket)". That is too early for the SendAll send deadline:
the real gate was the fixed 2-second enqueue deadline in the zerocopy
SendImage path. At the start of a large dataset the writer briefly stalls
draining the socket while it creates the master file and writes the large
START metadata + calibration frames to GPFS; the per-connection queue fills,
and after 2s SendImage marked the connection broken. The writer then
reconnected outside the active session, so the rest of the run was dropped
and the half-written file was finalized at the next START.
Replace the fixed 2s enqueue deadline with the same peer-liveness condition
used on the send path: keep applying backpressure while the writer proves it
is alive (BUSY heartbeats / ACKs refresh last_peer_activity_ns from a thread
independent of the stalled write path), and only declare it dead after the
liveness window of complete silence. A transient startup stall is now ridden
out instead of dropping the run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A slow filesystem could stall the writer's consume pipeline, propagating
TCP backpressure to the pusher. The pusher then treated that backpressure
as a dead peer and force-closed the connection mid-run; the writer
reconnected as a brand-new connection outside the active session, so the
rest of the run was silently dropped and the half-written HDF5 file was
later finalized with holes.
Replace the throughput/progress-based send timeout with a peer-liveness
timeout:
- Add TCPFrameType::BUSY (wire version 2 -> 3).
- TCPImagePuller runs a heartbeat thread that sends BUSY every 250ms on a
thread independent of the (possibly stalled) write path, so liveness
keeps flowing during deep stalls. A send_mutex serializes
ACK/pong/heartbeat writes.
- TCPStreamPusher refreshes last_peer_activity_ns on every inbound frame
and only declares a connection dead after peer_liveness_timeout (15s) of
complete silence, tolerating arbitrarily long backpressure while still
catching a genuinely dead peer (and immediate EPIPE/ECONNRESET).
- Re-key both backpressure waits (the SendAll data path and the post-END
WaitForEndAck) onto this liveness signal instead of byte-progress /
DATA-ACK-progress, so a slow final flush at END is tolerated too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The settings content stopped resizing past ~360 px, leaving dead space when the
dock was widened. Cause: QFormLayout's default field growth policy keeps fields
at their size hint. Set AllNonFixedFieldsGrow on all five forms (geometry, unit
cell, spot finding, indexing, azint) so the fields grow to fill the dock width.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The chart top margin was too tight (2 px), so the highest Y-axis label
overlapped the top edge; give it 10 px of room (both chart views).
- The settings dock had no width constraints (unlike the inspector), so it
stretched ugly when widened and would not fold back narrowly. Cap it to
330–480 px like the inspector.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In per-image mode the embedded chart brought its own combo box, so the plot
panel showed two. Drop the per-image chart's combo and drive it from the one
dataset-info combo:
- JFJochViewerSidePanelChart loses its combo; it exposes PlotTypes() (the
available profiles) and a setPlotType(code) slot, driven externally.
- JFJochViewerDatasetInfo fills its single combo with the per-image plot types
while in per-image mode (per-dataset metrics otherwise) and routes the
selection to the chart; dataset/runs/live updates leave the per-image combo
and plot untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plots:
- A "+ Plot" button on the dataset-info panel spawns another plot dock, placed
beside the previous one (horizontal split) so several metrics can be watched
side by side. Same path as the Charts menu, now one click away.
Image strip:
- The image number is painted into the thumbnail bitmap (coral badge, top-left)
instead of a text label under the icon, so no height is lost to it.
- Thumbnails are icon-only and scale to the available dock height (resizeEvent),
so the strip never needs more room than it has; lowered its minimum.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the plot dock was short, Qt Charts dropped the axis labels first, making
the plot hard to read.
- Reclaim Qt Charts' outer graphics-layout padding and trim the inner margins
(both chart views) so the plot and its labels get the available space.
- Raise the chart minimum height (dataset-info 80 -> 140, per-image 120 -> 140)
so the dock can't be squeezed below the point where the axis labels fit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The image-strip thumbnail renderer could take the whole viewer down:
- Only the image load was wrapped in try/catch; the rest of RenderThumbnail_i
(pixel access, LUT mapping, QPainter) ran unguarded on the worker thread, so
any exception there called std::terminate. Wrap the entire function.
- std::clamp(idx, 0, lutSize - 1) is undefined behaviour when the colour LUT is
empty (lo > hi). Bail out early if the LUT has no entries.
A click landing while a later thumbnail is still rendering surfaced this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address the review on the image strip / hit feed:
- Height constraints: JFJochSimpleChartView used a hard setFixedHeight(300) and,
being a page in the dataset-info stack, forced the whole plot dock (and thus
the window) taller than the screen once the strip was stacked below. Make it a
soft minimum (120). Wrap the settings dock in a QScrollArea so its content can
scroll instead of forcing window height. Smaller strip thumbnails (96) and
lower default bottom-dock heights. The window no longer grows past its
requested size.
- Stochastic selection: representatives are now picked at random within N equal
bins (over image index, or a metric's sorted order), and a Refresh button
re-rolls a fresh set — avoiding deterministic-spacing artefacts. "Most spots"
stays deterministic; "Indexed" becomes spaced-random.
- Live stability: the strip stores dataset updates but only rebuilds on file
open (worker fileOpened) / mode / Spots / Refresh, so HTTP sync image updates
no longer trigger constant thumbnail refetching.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the image strip, addressing the review:
- Composition: stack the plots and the thumbnail strip vertically (plots on top
with more height, strip below) instead of sharing horizontal space — both
benefit from width, and the strip needs less height.
- Processing dock is hidden by default and narrower; it reveals itself only when
a reprocessing job starts (new jobStarted signal), and the Processing
perspective no longer force-shows it.
- Thumbnail spot overlays now use the same feature (indexed) / spot colours as
the main viewer, and follow the side-panel colour pickers.
- New strip selection modes "Resolution" and "Background": pick images that span
that metric's distribution (a quick histogram-representative selection).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A dockable strip of thumbnails for a handful of representative images, with
spots overlaid so a real pattern reads like a constellation; click opens that
image.
- JFJochViewerImageStrip: mode selector (evenly spaced / most spots / indexed,
computed from the dataset's per-image metrics) + a Spots toggle; a scrollable
row of thumbnail buttons; click emits imageSelected.
- The reading worker renders thumbnails off-thread and non-disruptively: load
each image via its reader, downsample by block-maximum (keeps Bragg spots),
map through the colour-scale LUT, optionally paint spots (coral indexed /
teal not), and emit thumbnailReady(image_number, QImage). File mode only.
- Docked in the bottom area, shown in the Processing perspective; the colour map
follows the display toolbar.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete pass over both toolbars (review items g–k), closing the toolbar topic.
New ToolbarIcons: crisp QPainter vector glyphs (Qt SVG is not a build dep), with
white "on" variants for toggles and a shared flat button style (coral hover,
navy checked).
Navigation toolbar, laid out source-first:
- Open (3.5" diskette) · HTTP sync · Movie (reel-on-top camera, distinct from
the next/play triangle) ║ the scrub slider, which expands to fill the bar
(coral track, navy handle) so it is the obvious way to move across the data ║
precise navigation: first/prev/[number]/next/last, Jump, Sum.
- Open and HTTP-sync reach the file / connect dialogs (JFJochViewerMenu
openSelected and openHttpSelected, now public).
- HTTP-sync has three states via a status dot (grey disconnected / green live /
amber frozen); clicking with no live source attached opens the connect dialog.
Removed the separate "Reanalyze" toggle.
Display toolbar:
- Styled foreground slider (matching), Auto/HDR as styled text toggles.
Hero buttons:
- "Reanalyze image" is now a toggle (worker ReanalyzeImages: run now + keep
re-analysing on image/settings/processing changes; coral = active).
Processing dock:
- Drop the "New job…" toolbar action (the hero button drives it) and parent the
job dialog to the main window instead of the dock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Declutter the side and settings panels (review items a–f):
- New CollapsibleSection widget (slim navy header + coral rule + chevron).
- Inspector: Image features / Resolution rings / ROI are now collapsible and
start folded (ROI auto-expands when ROIs change); drop the Data-analysis and
Powder-calibration sections (they live in the hero buttons and settings dock).
- Settings dock: move Geometry to a shared section above the MX/AzInt toggle
(both communities need it); add Detector tilt (PONI rot1/rot2, deg) and
rename "Beam center" -> "Beam origin" with PONI/XDS tooltips; show the
space-group number and resolved Hermann–Mauguin symbol on one line; wrap
sections in accordions, anchored top with a bottom stretch so expanding one
does not shift the others.
- Toolbar: drop the redundant "Image number" label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the duplicated per-image plot from the side panel and host it in the
dataset-info dock instead:
- JFJochViewerDatasetInfo gains a "Per-image" toggle (next to Grid) that swaps
the stacked view to the existing JFJochViewerSidePanelChart (azimuthal 1D,
Wilson, I/sigma, spot and fluorescence profiles of the current image). The
per-dataset metric combo is disabled while in per-image mode.
- The per-image profile follows the displayed image (imageLoaded forwarded).
- Drop the "Image statistics plot" section from JFJochViewerSidePanel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two capabilities that set Jungfraujoch apart get prominent, paired navy
buttons at the right of the display toolbar, with purpose-drawn icons (a single
diffraction frame vs a stack of frames):
- "Reanalyze image" re-runs the analysis pipeline on the current image
(worker Analyze).
- "Reanalyze dataset" opens a new whole-dataset processing job
(JFJochProcessingJobsWindow::newJob, now public).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Show the PSI logo in the menu-bar corner, picking one of four interchangeable
dot designs at random each launch. Embedded as PNGs (rasterised from the SVG
sources, kept alongside) so no Qt SVG module dependency is needed.
- Settings dock: put beam center X and Y on a single "Beam center" row to save
vertical space on laptop screens.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surface a surgical subset of processing settings in an always-visible dock
instead of hidden windows:
- New JFJochViewerSettingsDock with an MX / AzInt segmented toggle.
MX page: geometry (energy, distance, beam X/Y), a new unit-cell +
space-group editor (no such input existed before; enables known-cell
ffbidx), spot finding (S/N, photon count, min pixels/spot), indexing
algorithm and geometry refinement. AzInt page: q range / spacing /
azimuthal bins plus the existing powder-calibration widget.
- Edits feed straight into the worker (UpdateSpotFindingSettings,
UpdateAzintSettings, UpdateDataset, FindCenter); fields populate from the
loaded dataset.
- Add CeO2 and Silicon calibrant presets.
- Dock it left, objectName "settingsDock", show it in the Processing
perspective (hidden in Image).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the layout reconfigurable, the foundation for the redesign:
- The diffraction image becomes the central widget; the right-hand side panel
is now a dockable "Inspector" (QDockWidget, right area).
- Every dock and toolbar gets a stable objectName so QMainWindow saveState /
restoreState round-trips. Dataset-info docks are numbered uniquely.
- Persist geometry + dock state to QSettings on close and restore on launch,
so the user's arrangement resumes.
- New "View" menu with Image / Processing perspectives (show/hide the bottom
plots + jobs panel) and "Reset layout" (back to the as-built arrangement,
captured at startup).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First pass of the viewer redesign (see docs/review/VIEWER_REDESIGN.md), no
structural change:
- TitleLabel: replace the 50px solid #FA7268 section bars with a slim 26px
navy-bold header + coral accent rule, removing the "venetian blind" stack
while keeping the salmon identity.
- Palette: switch the accent (QPalette::Highlight) from teal to navy #1F3A5F.
- Image toolbar: replace the unicode arrow glyphs (|<= <= => =>|) with flat
media icons (first/prev/next/last) + tooltips, and a play icon on Movie.
- Bottom dock: give the per-dataset plot more default height (340 -> 420).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eigen is an external find_package(Eigen3 3.4) dependency. Eigen's
same-major-version rule means a bare 3.4 request only accepts 3.x, so 5.x
cannot be used without changing every requester (jfjoch, Ceres, and the
upstream ffbidx). Standardise on Eigen 3.4.x:
- docs: correct the Windows Eigen install recipe to 3.4.0 and note the
same-major constraint; SOFTWARE.md now says 3.4.x (not "3.4 or newer").
- docker/{rocky8,rocky9,ubuntu2204,ubuntu2404}: actually install Eigen 3.4.0
from source to /opt/eigen-3.4 (header-only) and add it to CMAKE_PREFIX_PATH.
The images previously installed no Eigen at all, relying on the obsolete
"CMake fetches it" assumption; a rebuild would have failed at configure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Locate nvcc via find_program (HINTS CUDA_PATH / /usr/local/cuda) before
CHECK_LANGUAGE(CUDA), so CMAKE_CUDA_COMPILER no longer has to be passed by
hand. CHECK_LANGUAGE only searches PATH, which is missed routinely on Windows
and intermittently on Linux; the CUDA installer always sets CUDA_PATH. An
explicit -DCMAKE_CUDA_COMPILER / $CUDACXX still wins.
Docs: zlib and Eigen are external find_package deps (not auto-fetched);
simplify the Windows configure to just CMAKE_PREFIX_PATH (no CMAKE_CUDA_COMPILER,
no ZLIB_ROOT); fix the Eigen 5.0.1 install recipe to disable BLAS/LAPACK
(they fail MSVC with C1128 /bigobj) and use cmake --install.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fusion fills QGroupBox interiors with a flat light colour, so the settings window
lost its salmon look. A tiny app stylesheet (QGroupBox { background: transparent })
makes them show the salmon window background again; entry widgets stay white via
the palette.
- The dataset-info chart now fixes the x-axis to the whole dataset [0, n) (the
largest run = the original file), so a subset run is drawn at its real image
positions instead of being stretched to fill the plot. Subsets with binning bin
by ordinal and place each point at its mapped image number (correct for the
common contiguous sub-range).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build jfjoch_hdf5_test once in a new build-hdf5-test job and hand the binary
to dials-test as an artifact, instead of recompiling it from scratch in every
rocky9 processing job. Uses the christopherhx gitea-{upload,download}-artifact
fork (mirrored under gitea.psi.ch/actions) to avoid the stock v4 action's
GHESNotSupportedError. Probe scope: dials-test only; xds-* still self-build
until the round-trip is confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Running rotation indexing on a sub-range (e.g. images 60-120) segfaulted: the
first pass passed the *global* image number to RotationIndexer::ProcessImage,
which indexes v_ (sized to the run's image count) -> out-of-bounds write.
- ProcessImage now takes an explicit mid-exposure angle (optional; falls back to
the goniometer at the image index), so the indexer no longer assumes its slot
index equals the goniometer image index. IndexAndRefine supplies it via
RotationAngle(), matching the angle used for prediction. Added a bounds guard in
ProcessImage so a bad index can never corrupt memory.
- JFJochProcess feeds the rotation indexer the local ordinal (not the global
index), and shifts the goniometer (start += start_image*incr, incr *= stride,
per-image wedge preserved) so local index i maps to the angle of original image
start+i*stride - fixing rotation angles for the whole sub-range pipeline
(prediction, refinement, output), not just the indexer.
- Expose "Rotation images" (number used for the first pass) in the job dialog,
enabled when rotation indexing is on. (Count > available is already clamped by
select_equally_spaced_image_ordinals.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Styling (now via the application palette in main(), not a stylesheet, so it
applies to every widget incl. dialogs):
- Entry fields and item views (line edits, spin boxes, combos, tables) are white
when enabled and salmon (the GUI default) when disabled - so it's clear where
you can type. Panels stay salmon (Window role).
- Teal (#2a9d8f) accent via the Highlight role: selections and progress-bar chunks.
Runs list / plots:
- The original file is listed as a first "Original" run (Mode "HDF5"); double-click
any run row to show it (replaces the "Show original" button). The currently shown
run is bold (driven by snapshotsChanged).
- Fix the colour "swap" when switching runs: the primary line is matched by the
active-dataset pointer (not active_id_), so a run keeps its colour even while a
datasetLoaded/runsChanged pair is mid-flight.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Styling:
- Entry widgets (line edits, spin boxes, combos, item views) are salmon (the GUI
default) when inactive and turn white while focused/active.
- A teal accent (#2a9d8f) for selections and the QProgressBar chunk, against the
salmon background.
- Move the jobs-table Status column (the progress bar) to column 2 so it stays
visible in the narrow dock instead of scrolling off the right edge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the ambiguous "Images" count with explicit "Start image" and "End image"
spinboxes (end 0 = to the last image). buildConfig sets config.start_image /
end_image; the table shows the range ("start-end" / "all") and the progress bar
expects end-start images.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for a dataset (snapshot or, later, the main file) being a subset of the
truly collected images:
- JFJochReaderDataset gains source_image_number (image index -> original image
number; empty = identity).
- HDF5MetadataSource reads /entry/detector/number into that map and an inverse
(original -> local) map; FillPerImage / ReadSpots translate the requested global
image to this source's local index via ToLocalIndex (return nothing if the image
is not covered), so partial snapshots are correct and never read out of bounds.
- RegisterSnapshot now accepts a snapshot with fewer images than the dataset
(only rejects more), so sub-range / strided reprocessing runs register.
- The dataset-info plot draws each run at its original image numbers (x map from
source_image_number), so a subset run lands at the right place on the shared
axis. The live run's map is filled from msg.original_number.
This makes the foundation ready for strided / filtered selections (e.g. reprocess
only images with >N spots) without restricting to a min-max sub-range.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lower the dataset-info chart and jobs-table minimum heights (200 -> 80, table 60)
so the bottom dock area can be dragged smaller; the comfortable default size is
still set via resizeDocks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The current-image marker (a scatter series) was picking an arbitrary theme colour
(green), which read as a stray series. Pin it to black so it's an unambiguous
"current image" dot, distinct from the run line colours. Its legend entry is
already hidden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- "Remove result" toolbar action drops a reprocessing run: reader RemoveSnapshot
(Original is protected; if the removed run was active the view falls back to
Original), worker RemoveRun, and the table row is removed.
- Opening a new file clears the jobs table (worker fileOpened -> clearJobs) to
match the reader, which already resets snapshots on ReadFile.
- Plot fixes: each run keeps a stable colour by its position (Original = blue),
so colours no longer swap when the active run changes; the current-image marker
is hidden from the legend (was the stray "[Image #N]" entry).
- The status bar shows processing rate (Hz) and estimated remaining time while a
job runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dataset-info plot now overlays every run as a separate named line instead of
replacing the plot when a snapshot is activated:
- Reader gains AllSnapshotDatasets() (every snapshot's dataset, Original first).
- The worker owns the run collection: a stable snapshot id plus an editable
display label (RunData), emitted as runsChanged(runs, active_id) on file open,
snapshot register, activate and rename. RenameRun(id, label) updates the legend
label without touching the reader key (decoupled id vs label).
- The chart view draws the active run as the primary series (markers, hover,
binning, axes) and the other runs as overlay lines sharing its axes, with a
legend shown when overlaying (appendSeries factored out for both).
- The dataset-info widget holds the run list + the live run, extracts the selected
metric from each (ExtractMetric), and unions the available metrics across runs.
- The live run is its own "Live" overlay (lightweight per-tick refresh, no combo
rebuild); it is cleared on finish so the persisted snapshot takes over.
- The processing jobs table gets a "Started" column and an editable Name column;
editing a name renames that run's legend label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The Indexing tab now shows what the selected algorithm resolves to on this
machine/dataset ("Effective on this system: ..."), mirroring
DiffractionExperiment::GetIndexingAlgorithm() so Auto is no longer ambiguous
(GPU present? unit cell known?). Cell-known state is forwarded from the loaded
dataset via JFJochSettingsWindow::datasetLoaded.
- FFBIDX and FFT (GPU) radio options are disabled on builds without CUDA, where
only the FFTW CPU indexer exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regressions from docking the processing panel + live plots:
- Default window size is now screen-aware (fits a laptop) instead of a fixed
1200x1200.
- Give the bottom dock area a guaranteed height (resizeDocks vertical) and a
minimum height on the dataset-info chart, so the plot and its axis labels stay
visible next to the processing panel.
- Activating a metadata snapshot (Show original / View results) no longer re-runs
full analysis: it re-reads the image against the snapshot's stored results with
reanalysis suppressed, instead of re-indexing on every switch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Convert JFJochProcessingJobsWindow from a standalone helper window into a
dockable QWidget panel (toolbar + stacked table/message), placed in the bottom
dock area and split horizontally to sit in the bottom-right corner next to the
dataset-info plots. Its show/hide toggle moves to the Window menu via the new
JFJochViewerMenu::AddDockEntry.
- When connected to a live HTTP stream the panel shows "Dataset re-processing is
currently available only in File mode" and disables the toolbar, driven by the
worker's httpConnectionChanged signal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Processing settings:
- Split the spot/index settings widget into two tabs (Spot finding | Indexing)
via TakeSpotFindingPage()/TakeIndexingPage().
- Indexing tab now exposes the indexing algorithm (Auto/FFBIDX/FFT/FFTW/None) and
geometry refinement (None/Orientation/Beam center/Pixel refine) as radio groups
with short explanations.
- Fix: UpdateSpotFindingSettings dropped algorithm + geometry-refinement when
copying into indexing_settings (the geom checkbox was a no-op); both now flow
into jobs via curr_experiment.
Processing jobs window:
- Rotation indexing from the job dialog now also sets RotationIndexing(true) on
the experiment's IndexingSettings; otherwise IndexAndRefine builds no rotation
indexer and the two-pass pre-pass throws.
- Status cell shows a QProgressBar with "<done> / <expected>".
- Live results: JFJochProcessController::OnImageProcessed accumulates per-image
results into a JFJochReaderDataset and emits it throttled (~4 Hz) as liveDataset,
forwarded to the dataset-info plots so they update while a job runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One "Processing settings" window with tabs: Spot finding & indexing | Azimuthal | Bragg
integration | Scaling, replacing the two separate settings windows. The spot/index and azimuthal
tabs reuse the existing windows' widgets unchanged (their content is lifted into tabs via
takeCentralWidget), so all their logic/signals keep working; Bragg integration and scaling are new
editable panels (previously not adjustable in the GUI).
JFJochImageReadingWorker gains UpdateBraggIntegrationSettings / UpdateScalingSettings; both persist
as worker state and are re-imported into curr_experiment on file load / dataset update (like the
indexing/azint settings), so they apply to interactive analysis (Bragg) and flow into processing
jobs via GetReprocessingInputs (Bragg + scaling). Scaling only affects the job post-pass, so it is
just stored, not reanalyzed.
Verified: jfjoch_viewer builds and runs (offscreen) with the converged window.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only the ~1800-image rotation dataset is kept in LFS; a separate ~5000-image serial set is too
large to ship, and the serial path can be exercised by running the rotation series in serial
mode if needed. Removes JFJochProcess_LysoSerial and the serial entry from the start-up listener
and README. The rotation [large] test now runs against the committed data and passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real reference dataset for the [large] JFJochProcess_LysoRotation test, stored with git-LFS
(tests/data/*.h5). lyso_rotation_master.h5 is the 1800-image lysozyme rotation series
(its data files keep the names the master links to). The test indexes it at 100% with cell
~78.2/78.2/37.8; it SKIPs when the data is not pulled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes the viewer a processing frontend, not just an image viewer:
- JFJochProcessingJobsWindow (menu "Processing"): a table of processing jobs on the open
dataset. "New job" configures mode (full/azint), image count, threads, output, and (full)
rotation indexing + scale/merge, using the viewer's current processing settings. A job can
be Run locally (off the GUI thread via JFJochProcessController, with live status/progress and
Cancel) or its jfjoch_process command line copied for a cluster.
- A finished local run is registered as a reader metadata snapshot and becomes the active
dataset (bottom plots + per-image overlays follow it); "View results" / "Show original" switch
between runs - so several settings attempts on one dataset can be compared.
- JFJochImageReadingWorker gains GetReprocessingInputs() (one locked read of file + experiment +
mask + spot-finding settings, so jobs share the interactive settings) and
RegisterProcessingSnapshot/SetActiveSnapshot slots + snapshotsChanged signal over the reader's
snapshot API. Files only (not live HTTP).
First-cut / open ends: one job at a time; output lands in the working dir (FileWriter rejects
absolute prefixes); Bragg/scaling settings still use defaults until the converged settings
window lands. Viewer builds and runs (offscreen) cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for making processing a first-class GUI activity:
- process/JFJochProcessCommandLine: reconstruct the equivalent jfjoch_process / jfjoch_azint
command line from a ProcessConfig + DiffractionExperiment + input path, for handing a job
off to a cluster. Unit-tested (full + azint).
- viewer/JFJochProcessController: runs one JFJochProcess job off the GUI thread (its own private
reader; HDF5 access is globally serialized so it is safe next to the interactive reader) and
reports back via queued Qt signals (started/phaseChanged/progress/finished/failed). Cancel()
forwards to JFJochProcess::Cancel(). Progress is throttled to ~200 updates per run.
The visible processing UI (jobs window + snapshot switcher + converged settings) builds on this.
Verified: tests/jfjoch_test [process]; jfjoch_viewer links and builds against JFJochProcess.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- .gitattributes tracks tests/data/*.h5 via git-LFS for big reference datasets.
- tests/TestData.h resolves tests/data files and reports absent / unfetched-LFS-pointer
so tests can SKIP() instead of failing; tests/data/README.md documents fetching + the
expected lyso_rotation/lyso_serial datasets.
- JFJochProcessLargeTest: a Catch start-up listener that prints dataset availability, plus
[large] full-analysis runs (rotation indexing + serial) that SKIP when data is absent.
Verified against the real 1800-image lyso rotation set (100% indexing, cell 78.2/78.2/37.8)
and skips cleanly without it.
- jfjoch_process.cpp: drop the ~15 now-unused workflow includes left after the JFJochProcess
extraction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The full processing workflow no longer lives in the CLIs. New process/JFJochProcess
encapsulates it for jfjoch_process, jfjoch_azint and (later) the viewer:
- ProcessMode {AzimuthalIntegration, FullAnalysis}; ProcessConfig carries run control
(range, threads, output prefix, spot finding, rotation + scaling options, reference
data) while the DiffractionExperiment carries all algorithm settings.
- Run() executes setup -> optional two-pass rotation pre-pass -> parallel per-image loop
(std::thread) -> optional scaling/merging post-pass -> NXmxIntegrated _process.h5 that
links back to the original images. ProcessResult returns stats + merge text.
- Cancel() / std::atomic<bool> (receiver style), checked between images; the CLIs install
a SIGINT handler that calls it (fixes the previous Ctrl+C gap), the viewer will use the
same hook. JFJochProcessObserver streams progress / per-image results for a live GUI.
jfjoch_process.cpp and jfjoch_azint.cpp are now thin: argument parsing + experiment
configuration, then JFJochProcess::Run + stats printing. Behaviour and usage messages
are unchanged.
Adds JFJochProcessTest (azimuthal integration round-trip, no-output run, pre-cancel) over
a small generated dataset.
Verified: tests/jfjoch_test [HDF5] (83 cases / 1854 assertions); jfjoch_azint and
jfjoch_process run end-to-end on lyso_test (azint 20 images; full analysis recovers the
lysozyme cell at 25% indexing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete the image/metadata decoupling. JFJochHDF5Reader is now a thin JFJochReader
facade composing one shared HDF5ImageSource (raw pixels) with a name->HDF5MetadataSource
map and an active selection ("snapshots").
- HDF5MetadataSource owns one master file: dataset-level parse (Open, ex-ReadFile),
per-image metadata (FillPerImage, ex-LoadImage_i), ReadSpots/ReadReflections/
ReadCalibration, and the master-first helpers. It locates per-image metadata either via
the shared image source (original file: metadata co-located with pixels) or in its own
master at the global index (integrated _process.h5 snapshot) - so switching snapshots
never reloads pixels.
- Facade adds RegisterSnapshot/SetActiveSnapshot/SnapshotNames/ActiveSnapshot; ReadFile
registers the original as "Original". All existing JFJochReader callers are unchanged.
- The global hdf5_mutex is taken once in the facade; the sources assume it is held.
Adds JFJochReader_Snapshots test (original _master.h5 + a reprocessing _process.h5 over
the same images: pixels stay from the original, metadata follows the active snapshot).
Verified: tests/jfjoch_test [HDF5] (80 cases / 1810 assertions), plus jfjoch_process/
azint/extract_hkl/scale and jfjoch_viewer all build against the refactored reader.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Decouple the raw-pixel side of JFJochHDF5Reader from the rest as the first
step toward swappable per-dataset metadata snapshots.
- HDF5ImageLocator: single owner of the legacy/VDS/contiguous layout resolution
plus a persistent open-file cache, replacing the four duplicated resolvers
(GetImageLocation, ReadSpots, ReadReflections) and their per-call file caches.
Also hosts the source-mapping logic (former GetHDF5DataSource body).
- HDF5ImageSource: raw-pixel reading (locator + LoadImageDataset); the part whose
links to files stay fixed while the metadata master may change.
- JFJochHDF5Reader keeps a thin GetImageLocation/GetRawImage/GetHDF5DataSource that
delegate to image_source_; the six layout members are gone, parsed into a local
Layout handed to the source at the end of ReadFile. Cache cleared on Close().
Verified: tests/jfjoch_test [HDF5] (79 cases / 1775 assertions), and
jfjoch_process/azint/extract_hkl/scale relink unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cuFFT is the only CUDA component linked dynamically (cudart and the
fast-feedback indexer are static), so the prior Windows installer would
fail to launch the GPU path on a host without a CUDA toolkit. Ship
cufft64_*.dll next to the viewer (CUDA 13 keeps it in bin/x64, earlier
toolkits in bin); the DLL is self-contained, so the installed app needs
only an NVIDIA driver.
Tag the variant where it matters and nowhere else: the installer
filename (-cuda<major> / -cpu) and the Add/Remove Programs entry
("Jungfraujoch (CUDA)" / "(CPU)") advertise it, while the install folder
and Start Menu group stay plain "Jungfraujoch" -- the CUDA build is a
strict superset, so the two variants share a location and replace each
other.
Docs: SOFTWARE.md + JFJOCH_VIEWER.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revert Eigen from the FetchContent OVERRIDE_FIND_PACKAGE vendoring (cd394c52,
d4d2d0d8) back to an external find_package(Eigen3) dependency, like zlib. The
OVERRIDE_FIND_PACKAGE mechanism makes the Visual-Studio-bundled CMake (4.x
"-msvc") intermittently segfault during configure -- ~1/3 of fresh configures,
and 100% with Ceres CUDA enabled. The fault is in CMake's own FetchContent
variable-stack cleanup, reached when Ceres' find_package(Eigen3) resolves the
override through a nested FetchContent_MakeAvailable. Stock Kitware CMake runs
the identical scripts fine, so it is a bug in the VS-bundled cmake binary;
providing Eigen externally avoids that path entirely and is stable with Ceres
CUDA both on and off (verified 0 crashes; full CUDA build + NSIS installer
succeed under the VS-bundled cmake).
Also fix CPACK_NSIS_INSTALLED_ICON_NAME: the backslash value was written
verbatim into CPackConfig.cmake, where "\j" is an invalid escape that cmake 4.x
(CMP0010 strict) rejects when cpack re-parses it, so cpack failed under the
VS-bundled cmake (older/Kitware cpack only warned). Use a forward slash.
- image_analysis: keep Ceres USE_CUDA OFF (Jungfraujoch does not use Ceres' GPU
solvers) but drop the now-obsolete "required to avoid the Windows cmake crash"
rationale from the comment.
- docs: Eigen is a required external dependency again (SOFTWARE.md); the Windows
viewer build provides zlib + Eigen externally (JFJOCH_VIEWER.md).
- THIRD_PARTY_NOTICES: move Eigen from the fetched-and-linked table to the
external runtime-libraries table.
Docker images still need Eigen re-added (5.0.1, source build) -- to follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MSVC/Windows build fixes:
- Force JFJOCH_VIEWER_ONLY ON for Windows and macOS; the broker/receiver/FPGA/
writer server stack is Linux-only, so those platforms build the viewer subset.
- Vendor an OpenBSD/NetBSD getopt/getopt_long shim (tools/wingetopt/, BSD/ISC)
as a WIN32-only static lib -- the MSVC CRT has no <getopt.h> -- and link it
into the four portable CLI tools on Windows.
- jfjoch_extract_hkl.cpp: include <getopt.h> explicitly (it relied on glibc's
transitive <unistd.h>).
- jfjoch_process.cpp: guard <unistd.h> with #ifndef _WIN32.
- Guard JFJochStreamWriter and jfjoch_writer with IF(NOT JFJOCH_VIEWER_ONLY):
the former pulls in JFJochImagePuller, the latter uses a fork()/waitpid()
multi-process design; neither belongs to the portable viewer subset.
- Record wingetopt in THIRD_PARTY_NOTICES.md and licenses/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Acknowledge all bundled third-party software and satisfy attribution/notice
requirements, while keeping it maintainable:
- THIRD_PARTY_NOTICES.md: human-readable manifest (component, copyright, SPDX
license, link) for fetched, vendored, and runtime/SDK dependencies.
- licenses/: verbatim license texts; COLLECT.sh regenerates them from the
build trees and system SDK locations.
- Bundle the verbatim Qt LGPL-3.0 text and the CUDA Toolkit 12.8 EULA.
- frontend: self-contained npm attribution generator (`npm run licenses` ->
dist/THIRD_PARTY_LICENSES.txt), wired into the frontend build target.
- Install LICENSE + notices + licenses/ into share/doc/jfjoch for every
packaged component.
- viewer: Help > "Third-party Licenses" window (QTextBrowser) showing a
generated, self-contained HTML built from licenses/.
- docs/SOFTWARE.md: drop the stale hand-kept dependency lists; point at the
manifest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The viewer's only .qrc is compiled explicitly via QT_ADD_RESOURCES into
APP_RESOURCES; no .qrc is added directly as a target source, so AUTORCC never
acted on anything. Remove the dead flag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NSIS installer was missing several things on Windows:
- License: show the repo GPLv3 LICENSE as the click-through license page
(CPACK_RESOURCE_FILE_LICENSE).
- Start Menu: no shortcut was created -- add CPACK_PACKAGE_EXECUTABLES for
jfjoch_viewer, and embed an app icon so the shortcut is not generic. The
icon is a new multi-size jfjoch.ico (256 PNG + 48/32/16) compiled into the
.exe via resources/jfjoch.rc; macOS gets the matching jfjoch.icns copied
into the .app bundle (MACOSX_BUNDLE_ICON_FILE). Both derived from
resources/jfjoch.png, padded square with transparent bands.
- Naming: install folder, Start Menu group and Add/Remove Programs entry are
now "Jungfraujoch" instead of "jfjoch <version>".
Also ship the offline analysis CLIs (jfjoch_process/scale/azint/extract_hkl)
in viewer-only / Windows packages: JFJochReceiverPlots now lives in
JFJochCommon, so their stale JFJochReceiver link is dropped, leaving only the
portable libs (reader/image_analysis/writer/common). tools/CMakeLists.txt is
split so those four build everywhere while the hardware tools stay gated
behind NOT JFJOCH_VIEWER_ONLY, and the viewer-only branch now adds tools/.
They install into the "viewer" CPack component.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A viewer-only build builds jfjoch_viewer but the CPack block hard-coded
CPACK_COMPONENTS_ALL to "jfjoch writer" and only appended "viewer" when
JFJOCH_VIEWER_BUILD was set -- which a pure viewer-only build does not set.
cpack then tried to package the empty jfjoch/writer components and failed,
requiring a manual -D CPACK_COMPONENTS_ALL=viewer.
Branch the component list on build mode: viewer-only packages just "viewer";
other modes keep "jfjoch writer" plus driver-dkms/viewer when their flags are
set. Also drops the redundant re-set of the default list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
USE_CUDA is a Ceres cache STRING (default "default" = auto-detect), not an
option(), so a plain SET was shadowed by the cache default and CUDA got
enabled on MSVC. Force it OFF in the cache instead.
Also drop two dead settings: MINIGLOG (Ceres uses abseil logging now, no
longer referenced) and EIGENSPARSE ON (already Ceres' default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CUDA: append sm_121 (Blackwell GB10 / DGX Spark) to CMAKE_CUDA_ARCHITECTURES
when nvcc >= 12.9 knows it. The static list tops out at sm_120 and embeds no
PTX, so without this a build would fail to launch on Spark ("no kernel image
available"); guarded so CUDA 12.8 (which predates sm_121) is unaffected.
- FFTW: enable NEON codelets on aarch64 (Grace), mirroring the x86 SSE2/AVX/AVX2
branch, so single-precision FFT is SIMD-accelerated instead of scalar on ARM.
Both are inert on x86 / CUDA 12.8; verified configure still succeeds there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FFTW is fetched and built by the jungfraujoch CMake (single-precision, from the
release tarball), so the system fftw packages (fftw-static/fftw-devel on Rocky,
libfftw3-dev on Ubuntu) were unused. Remove them from all four images.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The jungfraujoch CMake now builds HDF5 + libtiff (FetchContent) and
libjpeg-turbo + Eigen (ExternalProject / OVERRIDE_FIND_PACKAGE) itself, so the
per-image pre-builds and system -dev packages for those were dead weight (the
app rebuilt its own copies regardless). Consolidate dependency management into
the one CMakeLists at the cost of longer build time (build env has network).
- Qt: pin static 6.9.1 everywhere. ubuntu2204 was on 6.10.0 and ubuntu2404 used
the OS Qt (~6.4, dynamic) -- the latter is now a static 6.9.1 build like the
others, so every image ships a self-contained, identical-Qt viewer.
- Remove the HDF5/libtiff/libjpeg-turbo/Eigen source builds + their system -dev
packages + the stale /opt/* CMAKE_PREFIX_PATH/PKG_CONFIG_PATH entries.
- nasm: now needed by the project's libjpeg-turbo ExternalProject; added to
ubuntu2404 (the others already had it).
- Standardize the Qt configure flags across all four (xcb, xcb_xlib=OFF,
no Wayland -- the qtwayland module was never built, so wayland_client=ON on
ubuntu2204 was a no-op).
Untested here (no Docker on this host) -- needs a build in each image; the
ubuntu2404 static-Qt dep list in particular should be sanity-checked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the platform packaging the viewer was missing so it can ship as a .dmg
(macOS) or installer (Windows):
- viewer: set MACOSX_BUNDLE (a .app bundle on macOS) and WIN32_EXECUTABLE
(GUI subsystem, no console window) -- both ignored where they don't apply;
install with BUNDLE/RUNTIME destinations.
- viewer: qt_generate_deploy_app_script + install(SCRIPT) to bundle the Qt
runtime at install time (windeployqt on Windows, macdeployqt on macOS).
Gated on Qt >= 6.5, which is where the OUTPUT_SCRIPT signature and the Linux
deploy path landed; with a static Qt (our Linux builds) it is a no-op.
- CPack: select DragNDrop (.dmg) on macOS and NSIS on Windows, ahead of the
Linux-only /etc/* probes; DEB/RPM path unchanged.
Verified: configures + generates cleanly on Linux with the viewer enabled
(Qt 6.4.2 -> deploy step correctly skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A clean configure failed at generation:
install(EXPORT "CeresExport" ...) includes target "ceres" which requires
target "eigen" that is not in any export set.
Ceres' install(EXPORT CeresExport) is unconditional, and its ceres target
transitively links the eigen target. As a FetchContent subproject Eigen leaves
EIGEN_BUILD_CMAKE_PACKAGE OFF (it defaults to PROJECT_IS_TOP_LEVEL), so its
install(EXPORT Eigen3Targets) never runs and eigen ends up in no installed
export set. Force EIGEN_BUILD_CMAKE_PACKAGE ON so eigen is exported and Ceres'
export validates. Install/export-only change; the build is unaffected.
Verified with a clean configure (fresh CMakeCache): configure + generate now
succeed where they previously errored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two deps the Linux build picked up from the system were absent on Windows.
Bundle both, unconditionally (matching how libtiff/FFTW are vendored), so the
tree builds without system Eigen/JPEG:
- Eigen 5.0.1 via FetchContent with OVERRIDE_FIND_PACKAGE, so every
find_package(Eigen3) -- ours, Ceres', and ffbidx' -- resolves to this copy and
the requested 3.4 version is satisfied by the newer major. ffbidx's bundled
eigen submodule is disabled (GIT_SUBMODULES "" on fast-indexer) so it no longer
creates a duplicate Eigen3::Eigen target. Verified on a CUDA build: Ceres 2.3.0,
the analysis libs, and ffbidx's CUDA code all compile against 5.0.1.
- libjpeg-turbo 3.0.4 via ExternalProject (upstream discourages add_subdirectory);
built+installed static and imported as JPEG::JPEG, replacing find_package(JPEG)
in preview/. SIMD is lazy: WITH_SIMD=ON + REQUIRE_SIMD=OFF means NASM-accelerated
when nasm is present, otherwise a warning and a scalar build (no hard failure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The recurring goal is a Windows jfjoch_viewer built with MSVC *and* CUDA ON
(GPU processing is wanted, not a CPU-only fallback); the non-CUDA path is the
macOS fallback. Also note the three deps a self-contained Windows build still
needs but FetchContent does not auto-provide: ZLIB (bundle zlib-ng as
ZLIB::ZLIB), libjpeg-turbo (ExternalProject), and Eigen (header-only, also
needed by Ceres) -- each guarded so the Linux build is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two spots used POSIX-only path logic that breaks on Windows:
- dataset_name() located the basename with rfind('/'), which misses
backslash separators, so a C:\...\foo_master.h5 file prefix kept the
whole directory. Use std::filesystem::path::filename() instead.
- Legacy-format data-file resolution joined paths with
fmt::format("{}/{}", ...), producing mixed \/ separators and prepending
the master directory even to absolute link targets. Reuse the existing
ResolveRelativeToMaster() helper, which joins via std::filesystem and
leaves absolute targets untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FFTW exposes fftw3.h only via $<INSTALL_INTERFACE:include>, so consuming
fftw3f from the build tree (FetchContent) leaves FFTIndexerCPU.h unable to
find the header. A system-installed fftw3.h masks this on Linux, so it only
bit on Windows; add the source-tree api/ dir to fftw3f's interface includes.
Also two viewer-reachable warnings:
- time_utc.h (C4477): %03ld was fed a chrono milliseconds::rep (64-bit), not
a long; cast the 0-999 value to int and use %03d.
- HDF5Objects.h (C4700): value-initialize the scalar read buffer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
libtiff's webp and lerc codec options default ON whenever their library is
found on the build host, dragging in libwebp + libsharpyuv (webp) and libLerc.
We only need DEFLATE (zlib) + the internal LZW codec, so turn both OFF.
httplib likewise auto-enables Brotli content-encoding when system Brotli is
present; gzip/zlib is enough for the broker, so disable it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
StreamWriter.cpp was compiled into both JFJochWriter and JFJochStreamWriter,
so JFJochWriter still transitively pulled in image_puller headers via
StreamWriter.h. Compile it only into JFJochStreamWriter (which already
links JFJochImagePuller); targets needing the streaming writer (jfjoch_writer,
jfjoch_test) link that lib, while JFJochWriter consumers (viewer, tools,
ImagePusher) no longer drag in the puller. Also drop a stray duplicate
HDF5DataFile source listing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address portability issues found building jfjoch_viewer with MSVC:
- common/{ADUHistogram,AzimuthalIntegrationProfile}.h and
image_analysis/spot_finding/StrongPixelSet.h only need DeviceOutput,
so include fpga/pcie_driver/jfjoch_fpga.h (plain-C, self-contained)
directly instead of acquisition_device/AcquisitionDevice.h, which
dragged <unistd.h> into the viewer tree.
- common/time_utc.h: guard gmtime_r/timegm/localtime_r with the MSVC
equivalents (gmtime_s/_mkgmtime/localtime_s) under _WIN32; drop the
duplicated includes and add the headers used directly.
- gemmi_gph/gemmi/utf.hpp: vendor the upstream gemmi header; fileutil.hpp
includes it on Windows for UTF8_to_wchar but it was never vendored.
- writer/HDF5Objects.cpp: ExtractFilename returns path.filename().string()
(std::filesystem::path has no implicit conversion to std::string on
Windows, where it is wchar_t-based).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch the libtiff FetchContent from the personal GitHub mirror to the
upstream gitlab repo and bump v4.6.0 -> v4.7.1 -- cleaner provenance; the
mirror was only ever a network-reachability workaround.
Verified on Linux: fetches, builds static (libtiff.a/libtiffxx.a), and
jfjoch_viewer links cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FFTW selects SIMD codelets at runtime via cpuid, so enabling AVX2 does not
require an AVX2 CPU -- it just speeds up the CPU FFT fallback indexer on
capable hardware and falls back to SSE2/scalar otherwise. Arch-guarded to
x86_64/AMD64 (ARM would need ENABLE_NEON). AVX-512 left off on purpose:
it's x86-64-v4 (not our v3 target), the FFT gain over AVX2 is marginal,
older Intel downclocks on it, and it's the riskiest to build under MSVC.
Verified on Linux: SSE2/AVX/AVX2 codelets compile, libfftw3f.a stays a
single static archive, jfjoch_viewer relinks cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both are leaf deps (JFJochPreview / JFJochIndexing) that only our own code
looks for, so building them ourselves removes the system-package lottery and
gives a reproducible, statically-linked build on every platform.
- libtiff: FetchContent, library only (jbig/zstd/lzma/jpeg/old-jpeg/tools/
tests off). The C++ binding (TIFF::CXX / libtiffxx) is packaged
inconsistently across distros -- missing on Rocky 9 -- and absent on
Windows, so find_package(TIFF COMPONENTS CXX) was unreliable; that call is
removed and JFJochPreview links the tiff/tiffxx targets directly. GitHub
mirror because upstream (gitlab) is unreachable from some restricted hosts.
- FFTW: FetchContent single precision (ENABLE_FLOAT) from the release tarball
-- the git repo ships no pre-generated codelets (needs OCaml genfft). It's
now always available, so the CPU FFT indexer is always built and
JFJOCH_USE_FFTW always defined; the "FFTW disabled" path is gone. Static
(libfftw3f.a) via the global BUILD_SHARED_LIBS OFF.
Verified on Linux: jfjoch_viewer builds and links libfftw3f.a + libtiff*.a,
all static.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Under JFJOCH_VIEWER_ONLY, only the viewer's transitive deps are built, so
sls_detector_package (detector_control/receiver) and catch2 (tests) are
never used -- and sls does not configure under MSVC, which is the whole
point of the viewer-only mode. Skip fetching just those two; zstd, hdf5,
spdlog, libzmq and httplib are still required (JFJochReader links httplib).
Verified: cmake -DJFJOCH_VIEWER_ONLY=ON configures with sls/catch2 absent
and `make jfjoch_viewer` still links.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CMake >= 4.0 (4.3.1 ships bundled with Visual Studio 2026) makes any
cmake_minimum_required(VERSION < 3.5) a fatal error. The fetched libzmq
still declares VERSION 3.0.2, which aborts configuration before our own
CMakeLists gets a say. Set CMAKE_POLICY_VERSION_MINIMUM=3.5 before the
FetchContent_MakeAvailable calls so those subprojects configure; the
variable is simply unused (harmless) on CMake < 3.30.
Verified against CMake 4.3.1: the real libzmq source fails without it and
configures with it, and an in-file SET() propagates into the
add_subdirectory child exactly as a -D cache entry does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New option (default OFF) that compiles only jfjoch_viewer and the
libraries it transitively links -- common, image_analysis, writer,
reader, preview, image_puller, frame_serialize, gemmi_gph, compression,
jungfrau, and the OpenAPI model (JFJochAPI) -- skipping the Linux-only
online stack (broker service, receiver, fpga, acquisition_device,
detector_control, image_pusher, xds-plugin, tests, tools) and the
frontend. This is the standalone build path for the cross-platform
viewer.
JFJochAPI is the only piece the viewer needs from broker/, so the broker
service library and executable are gated behind NOT JFJOCH_VIEWER_ONLY.
The full build and JFJOCH_VIEWER_BUILD (viewer on top of the full build)
are unchanged -- that path is the verbatim ELSE() branch.
Verified on Linux: cmake -DJFJOCH_VIEWER_ONLY=ON configures and
`make jfjoch_viewer` links cleanly with the online stack absent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the build less hostile to MSVC (needed for the viewer port) and
align the CUDA standard with C++.
- Remove the CMAKE_C/CXX_FLAGS_RELEASE overrides: -O3/-DNDEBUG only
duplicate CMake's own GCC/Clang Release defaults, and the SET()
clobbered the MSVC defaults. The lone meaningful flag,
-Wno-deprecated-enum-enum-conversion, merely silenced warnings from
the vendored Xilinx ap_private.h in the FPGA path; with no -Werror
it is just log noise, so drop it.
- CMAKE_CUDA_STANDARD 17 -> 20 (+ STANDARD_REQUIRED); CUDA >= 12.8
already supports C++20.
- Drop the file-scoped -Ofast on JFPedestalCalc.cpp /
JFConversionFloatingPoint.cpp: it is GCC-only and reaches the MSVC
viewer via JFJochCommon -> JFCalibration. Those files now build at
plain -O3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bare python3 is often too old for the Sphinx build. Auto-select the
newest python3.13/3.12/3.11/3 found on PATH, still allowing $PYTHON to
override.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When only one azimuthal phi bound is entered, fill the other so the
sector is well-defined and the server accepts it: a missing phi_min
becomes 0 and a missing phi_max becomes 360. Leaving both empty still
means a full ring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surface the optional azimuthal phi-sector bounds (phi_min_deg/phi_max_deg)
as editable columns, so a sector can be set from the web UI, not only the
viewer. Also fix the Q_max row's d header, which read d_max but is d_min.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ROI bound editors now have a white background, a capped width and
right-aligned text, which reads better for numeric entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restore the ROI-to-mask action on the new list: "Add to mask" and
"Subtract from mask" buttons rasterise the selected ROI into the user
mask (set or clear), through the same UpdateUserMask_i path. The ROI is
evaluated with per-pixel resolution and phi from the geometry, so box,
circle and azimuthal (sector) ROIs all map correctly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the scratch ROI panel gone, delete the now-orphaned
JFJochViewerImageROIStatistics(_Box/_Circle) widgets and the worker's
unused scratch-ROI plumbing (SetROIBox/SetROICircle, the ROIElement
member, AddROIToUserMask/SubtractROIFromUserMask). ROI-to-mask, if wanted
later, should act on the selected list ROI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clarify in FPGA_DATA_ANALYSIS.md that the ROI and azimuthal definitions
are also evaluated on CPU via the shared image_analysis library, so they
apply to the DECTRIS SIMPLON (EIGER) path that has no FPGA, not only the
FPGA-accelerated JUNGFRAU/PSI path. Document the three NXmx writer formats
(Legacy/VDS/Integrated) and the two acquisition workflows in CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two Mask-menu entries read a 16-bit TIFF (via ReadTIFFFromString16) and
apply it as the user mask: "replace" overwrites the current mask, "add"
unions the TIFF into it. The TIFF must match the detector dimensions; any
non-zero pixel masks. Both go through UpdateUserMask_i like the other
mask edits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add Download and "Upload to server" buttons to the ROI panel. The HTTP
reader gains GetROIDefinitions (GET /config/roi) and UploadROIDefinitions
(PUT /config/roi), converting between ROIDefinition and the generated
Roi_definitions model (including the optional azimuthal phi sector), the
same shapes OpenAPIConvert uses on the server. Download applies the
fetched ROIs through SetROIDefinition; upload pushes the current ones.
Both are no-ops unless the viewer is connected to a broker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The interactive shift-drag (box) / shift+ctrl-drag (circle) now creates a
new persistent ROI in the list instead of feeding the old single-ROI
scratch panel. The base emits a roiScratchDrawn hook on release; the
diffraction image turns the drawn shape into a named ROI committed via
SetROIDefinition.
The old JFJochViewerImageROIStatistics scratch panel and all its wiring
(box/circle configuration, single-ROI result, add/subtract user mask) are
removed from the side panel and window; the ROI list is now the single
source.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ROI panel now shows editable numeric fields for the selected ROI,
labelled by type: box min/max X/Y, circle centre/radius, or azimuthal
Q-range and phi sector. Editing a field rebuilds that ROI and commits it
through the same SetROIDefinition path, so all three types can be created
(via +) and dialled in by typing, not only by dragging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Azimuthal ROIs now show discrete grab handles on the selected ROI: the
inner/outer arc (resize Q/d) and, for a sector, the two phi edges
(rotate). Editing is by clicking a handle, with a generous tolerance,
which fixes the previously near-impossible phi-edge grab. Clicking the
(large) interior of an azimuthal ROI now only selects it and lets the
view pan, instead of capturing the gesture.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend canvas ROI editing beyond move:
- Box: drag corners/edges to resize (handles shown on the selected ROI).
- Circle: drag the perimeter to resize, interior to move.
- Azimuthal: drag the inner/outer arc to change the Q/d range, or a radial
edge to rotate a phi bound; sampled through the geometry so it tracks
the conic.
- Delete removes the selected ROI (the view now takes keyboard focus).
All edits go through the same live, throttled SetROIDefinition path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clicking a box/circle on the image now selects it (syncing the side-panel
combobox) and grabs it for moving, with a closed-hand cursor during the
drag. The move recomputes ROI statistics live rather than only on release,
throttled to at most one in-flight recompute so the worker is not flooded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add base-class mouse hooks (roiEditPress/Move/Release) so the diffraction
image can edit the persistent ROI selected in the side-panel combobox.
The selected ROI is highlighted (dashed, thicker), and dragging its
interior moves a box or circle; on release the new geometry is committed
through SetROIDefinition, which recomputes only the ROIs. Resize,
azimuthal handles and delete-key follow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Edited ROIs now override the file's for every subsequently loaded image
and survive settings changes, until a new file is opened (previously a
delete/add was lost on the next image). Tracked via an roi_override_ that
is re-applied to curr_experiment and to each loaded image's dataset.
- The ROI selector is a compact combobox instead of a list widget.
- "Show labels" and "Translucent fill" share one row to save space.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add JFJochViewerROIList to the side panel: a list of the dataset's ROIs
with selection, add (box/circle/azimuthal), rename and delete, plus a
statistics readout (sum/mean/std/max/valid/masked/centre-of-mass) for the
selected ROI, taken from the analysis output for the current image. Edits
emit a full ROIDefinition, routed to the worker's SetROIDefinition.
Per-ROI statistics now live in this panel rather than the canvas labels;
the diffraction image's labels show only the ROI name, and the ad-hoc
ROIIntegrationCPU computation there is removed in favour of the analysis
pipeline. The result widget now reports std dev instead of variance.
The single-ROI scratch panel remains for now and will be retired once the
interactive canvas editing replaces it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reading worker's experiment is the source of truth for ROIs. Add a
SetROIDefinition slot that updates curr_experiment.ROI(), rebuilds the
analysis ROI engine, mutates the dataset (so the canvas reflects the new
ROIs) and recomputes only the ROIs for the current image via RunROIOnly.
On image load when full re-analysis is off, ROIs are still computed via
AnalyzeROIOnly so the statistics stay current.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add three entry points so ROI statistics can be (re)computed without a
full re-analysis, in support of interactive ROI editing in the viewer:
- RebuildROI(): recreate the ROI engine after the ROI set changes (the
CudaStream is now kept as a member so the GPU engine can be rebuilt).
- AnalyzeROIOnly(): decompress + preprocess + ROI, skipping azimuthal
integration, spot finding and indexing (a new image when re-analysis
is off).
- RunROIOnly(): rerun only the ROI integration on the already-preprocessed
image (an interactive ROI move). A full Analyze() still computes ROIs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When ROI labels are shown, each ROI's label now also reports its sum,
max and pixel count for the current image. Rather than reimplementing the
accumulation, this reuses the existing ROIIntegrationCPU engine (the
software counterpart of the FPGA roi_calc), built from the experiment and
cached per dataset. A small adapter folds the viewer's gap sentinel
(INT32_MIN+1) onto the engine's masked sentinel (INT32_MIN) so masked and
saturated pixels are handled correctly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The full-ring annulus built the inner ring as a separate QPainterPath and
merged it with addPath, which stitched a spurious segment from (0,0) to
the ring. Build both concentric rings as subpaths of one path, each begun
with an explicit moveTo, so there is no connecting segment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Azimuthal ROIs now render on the diffraction image as annular sectors
(or full-ring annuli), sampled through DiffractionGeometry::ResPhiToPxl
so the outline follows the ROI footprint, including wrap-around sectors.
Add two side-panel toggles (both default off): a translucent fill for
every ROI (helpful when outline colours clash with the image colour map,
and with many ROIs) and ROI name labels (constant on-screen size). Wired
side panel -> window -> diffraction image like the existing feature
toggles.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a file with ROI definitions is opened (read into experiment.ROI()
by the HDF5 reader), the diffraction image now overlays the configured
box and circle ROIs as distinct colour-coded outlines, alongside the
existing resolution rings and spots. This is the first step of the
ROI-map-based multi-ROI canvas: showing the ROIs loaded from a file.
Azimuthal ROIs (wedge rendering) and per-ROI statistics from the bitmap
follow in subsequent steps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Footer "Paul Scherrer Institute" now links to https://www.psi.ch
- Reference drawer gains a "Source code" entry linking out to the Gitea
repository (new nav-item href that opens in a new tab)
- Reference drawer gains a "License" entry rendering the nicely formatted
docs/LICENSE.md inline (generalises the changelog renderer into MarkdownDoc)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The framed Sphinx changelog reused MUI with the same colours, so it read as
a picture-in-picture. Bundle docs/CHANGELOG.md at build time (?raw) and render
it with react-markdown, themed to match the app, for a quick frame-less view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Documentation tab pointed at jungfraujoch.readthedocs.io, which lags
behind the installed version. Build the Sphinx docs into frontend/dist/docs
and serve them at /frontend/docs so they always match the package.
- make_doc.sh: accept an output dir (default public), absolutize it, run
from the script dir, and allow $PYTHON override
- package.json: add "docs" script building into dist/docs
- CMakeLists: run "npm run docs" in the frontend target so CPack packs the
docs via the existing dist/ install
- App.tsx: point the Documentation DocFrame at /frontend/docs/index.html
Also clean up the Sphinx build (911 -> 0 warnings):
- fix typo myst_heading_anchor -> myst_heading_anchors so cross-doc
#anchor links actually resolve
- suppress myst.header/myst.xref_missing (auto-generated OpenAPI client
docs only) and the harmless sphinx_material config.cache note
- drop a dangling "---" transition at the end of CPU_DATA_ANALYSIS.md
- add REPOSITORIES and the generated python_client model pages to toctrees
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phi bounds were stored as plain floats internally but passed through
std::optional in the constructor, which obscured the intent. Take them
as float arguments defaulting to 0 instead. The convention is explicit:
phi_min == phi_max means the full ring (all angles), a sector wraps
across 0 when phi_min > phi_max, and otherwise phi_min <= phi_max. The
"both bounds or none" rule belongs at the API/file boundary
(OpenAPIConvert, reader), not in the core class, so it lives there now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In VDS mode the per-image ROI results (max/sum/sum_sq/npixel/x/y) are
written into the data files but were not exposed in the master, so a VDS
master surfaced no ROI statistics. Add virtual datasets under
/entry/roi/<name> in LinkToData_VDS, one group per ROI, mirroring how the
spot-finding and azimuthal-integration arrays are linked. Integrated and
legacy formats are unaffected (the results are already reachable there).
Extended the reader round-trip test to write real ROI results and check
they read back from the master for both VDS and integrated formats.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Relocate the ROI definitions and the roi_map bitmap from /entry/roi to a
separate /entry/roi_defs group in the master file. /entry/roi keeps its
original meaning (per-image ROI results only), so an older reader that
iterates /entry/roi is not disturbed by the bitmap/definition entries
(which would otherwise make it try to read /entry/roi/roi_map/max and
fail to open the file at all). Back-compatibility for already-deployed
viewers is the reason for the split.
JFJochHDF5Reader now reads the logical definitions into experiment.ROI()
and the roi_map bitmask (+ a name->bit index) into the dataset, for all
file formats (the master always carries them). Reading both lets a later
viewer either re-derive ROIs with the current geometry (logical) or show
the exact written footprint (bitmap).
Added a write/read round-trip test over VDS and integrated formats.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AddROI now renders azimuthal ROIs in addition to box and circle ROIs:
inner/outer arcs at the Q (d) bounds and, for a sector, the two radial
edges at the phi bounds. The per-pixel phi comes from the same
DiffractionGeometry::Phi_rad used to assign ROIs, so the overlay matches
the ROI footprint exactly (including wrap-around sectors). Full-ring
azimuthal ROIs draw as two concentric arcs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist regions of interest so an acquisition's ROI layout can be
recovered from the master (_master.h5 / _process.h5):
- /entry/roi/roi_map: uint16 per-pixel bitmask (bit i == ROI i).
- /entry/roi/<name>: logical definition (type + geometry params, incl.
azimuthal phi sector) plus bit_index tying it to the bitmap plane.
The bitmap rides along in the StartMessage (CBOR, mirroring az_int_map)
and is filled both online (JFJochReceiver::SendStartMessage) and offline
(jfjoch_process). Definitions come from the already-transmitted rois.
Both are dataset-wide metadata, so they are written by the NXmx master
writer (new NXmx::ROI), not the per-image data-file plugin.
Documented the /entry/roi layout in docs/HDF5.md and the master-vs-data
writer convention in CLAUDE.md; added a CBOR roi_map round-trip test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose the optional phi_min_deg / phi_max_deg azimuthal-sector bounds
added to ROIAzimuthal through the wire formats. Both are optional, so
older clients and streams (Q-only ROIs) remain valid.
- jfjoch_api.yaml: optional phi_min_deg / phi_max_deg on roi_azimuthal;
regenerated the cpp-pistache server model and the TS frontend client.
- OpenAPIConvert: map phi both directions (set only when present).
- CBOR start message: emit phi_min/phi_max only for a sector; on decode,
absent phi means a full ring.
- Documented the new optional CBOR fields; extended the CBOR round-trip
test with a sector and a full-ring azimuthal ROI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ROIAzimuthal can now be restricted to an azimuthal-angle sector in
addition to its Q/d range, enabling STXM-style directional ROIs. phi
bounds are optional (both-or-neither); absent means a full 360 ring, so
existing Q-only azimuthal ROIs are unchanged.
- ROIElement::CheckROI gains a phi_deg argument; box/circle ignore it.
- MarkROI gains an optional phi_map; ROIMap builds it from Phi_rad
alongside the resolution map only when azimuthal ROIs are present.
- phi bounds are normalized to [0,360) and wrap-around sectors are
supported (phi_min > phi_max).
- ROIConfigAzim carries phi_min/phi_max (kept trivially copyable for the
union; phi_min == phi_max means full ring).
No phi crosses the wire yet (CBOR/API wiring follows separately).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DetectorSelection rendered the info-table rows and the detector-select
menu items without keys; ROI's grid action cells likewise. Add stable
keys to silence the warnings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the stack of toggle-revealed sections with a left navigation
drawer (collapsible, expanded by default) clipped under a persistent
status bar. Dashboard (plots + measurement statistics + spot-finding)
is the default view; every other panel is its own drawer entry, so one
thing shows at a time.
- Group entries under Acquisition / Expert / Reference subheaders;
ROI folded into On-the-fly processing.
- Pale-indigo drawer with a lime accent on the selected entry.
- Gray out JUNGFRAU calibration / FPGA status when their data is empty.
- API reference and documentation open in-app via iframe.
- Fixed footer pinned to the page bottom.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generate Zod schemas alongside the typed client (src/client/zod.gen.ts),
picking up the OpenAPI field defaults. Enables runtime validation and
default-filling for form payloads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the unsaved-changes dot: the Upload button uses the warning color
while the panel has unedited changes, primary otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalize the expert-panel boilerplate (container + title + widgets +
upload button) and surface real upload errors via hey-api.
- SettingsPanel: consistent padded Paper + Typography title (one place to
size it) + optional Upload button + result snackbar; PanelTitle and
UploadSnackbarView exported for panels with bespoke layouts (DetectorSettings
grid, ROI)
- useUpload(mutation): wraps a generated hey-api mutation, shows the server's
error message on failure (errorMessage handles 400 string / 500 error_message)
- "unsaved changes" dot on panel titles, derived from the existing
last-downloaded snapshot (dirty = !isEqual(value, lastDownloaded))
- fix PixelMask: TIFF upload was swallowing errors; now shows them
- config panels (FileWriter, ImageFormat, Detector, AzInt, DarkMask, Indexing,
Instrument, ZeroMQ, ROI, DetectorSelection) upload through the typed SDK
instead of ButtonWithSnackbar's raw fetch; status/display panels share the
same shell for consistent spacing
- ButtonWithSnackbar now only powers bodyless action buttons (pedestal/cancel/
initialize, start/trigger, deactivate)
Build (tsc + vite) passes; dev server transforms all modules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace all class components with function components and move server-state
polling to @tanstack/react-query.
- add @tanstack/react-query; generate query helpers via the hey-api
@tanstack/react-query plugin (src/client/@tanstack)
- QueryClientProvider + one-time client.setConfig({ baseUrl: '' }) in index.tsx
- App polls statistics with useQuery(getStatisticsOptions, refetchInterval),
DataProcessingPlot uses getPreviewPlotOptions; manual setInterval polling gone
- memo() on presentational children; with TanStack structural sharing this
re-renders a child only when its own statistics slice changes
- PreviewImage stays imperative (binary JPEG -> object URL) using useEffect +
a ref for the object-URL lifecycle
- fix AzIntSettings correction checkboxes that mutated state in place (relied on
the poll re-render); they now use setState
- drop dead code uncovered during the port (unused upload/deactivate handlers
and imports in ImageFormatSettings, DetectorSettings, ROI)
Build (tsc + vite) passes; dev server transforms all entry modules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The raw stacked-module layout check compared the image width (cols) against
RAW_MODULE_SIZE (the per-module pixel count, 524288) instead of
RAW_MODULE_COLS (1024), so a raw-geometry TIFF could never be accepted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PUT /config/user_mask.tiff only accepted 32-bit unsigned TIFF, so masks
exported by tools like PyFAI (8-bit) failed. Route the upload through the
universal ReadTIFF reader and let PixelMask take a CompressedImage directly:
it validates the 2D shape against the detector's converted/raw layouts,
binarizes any 8/16/32-bit integer image (non-zero == masked), and rejects
float/multi-channel images.
Also dedupe the TIFF readers: ReadTIFFFromString16 is now a thin wrapper over
ReadTIFF, and the now-unused ReadTIFFFromString32 is removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nothing in HDF5DataFile.cpp uses sys/stat.h (no stat/mkdir/S_* etc.); it relies
on <filesystem>. Removing the dead POSIX include also avoids a needless
non-portable header for the eventual MSVC build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AppendImage wrote the CBOR byte-string length (major type 2, 8-byte length,
which is defined big-endian) via __builtin_bswap64 under #ifdef LITTLE_ENDIAN.
That is broken two ways off GCC/Clang+glibc: __builtin_bswap64 is not available
on MSVC, and LITTLE_ENDIAN is not defined there, so the #else path skipped the
swap and emitted a little-endian length on a little-endian host -> corrupt
frames.
Write the 8 length bytes most-significant-first directly. This is big-endian by
construction on any host, needs no byte-swap intrinsic and no endianness macro,
and is byte-for-byte identical to the previous output on little-endian hosts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
M_PI is a POSIX <math.h> extension that MSVC does not define without
_USE_MATH_DEFINES. std::numbers::pi (introduced in the viewer guard commit) is
C++20, but CUDA here is compiled as C++17 (CMAKE_CUDA_STANDARD 17) and several
common/ headers are pulled into .cu device translation units, so std::numbers is
not available there.
Add common/JFJochMath.h with a dependency-free `constexpr double PI` that works
in host code (including MSVC), in CUDA device code, and under C++17/20, and use
it everywhere:
- common/ and image_analysis/ (incl. CUDA .cu): 78 M_PI occurrences, 22 files
- broker/OpenAPIConvert.cpp
- viewer/: the 5 files that used std::numbers::pi now use PI, for one consistent
convention across the codebase
Verified to build: JFJochImageAnalysis (incl. CUDA), jfjoch_viewer, JFJochBroker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IndexerThreadPool dispatches on DiffractionExperiment::GetIndexingAlgorithm(),
which already resolves Auto to a concrete algorithm (FFTW/FFT/FFBIDX) or None;
the pool has no policy to resolve Auto itself. The worker handled a stray Auto
with a dead branch and silently produced no result when the resolved algorithm
had no matching indexer built.
- Document on GetIndexingAlgorithm() that it never returns Auto.
- Throw a clear internal error at the pool boundary if Auto ever arrives.
- In the worker, replace the dead Auto branch with a loud failure for any
resolved algorithm that has no matching indexer (e.g. a GPU algorithm on a
host without a GPU), instead of returning no result silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PlotAzintUnit used D_A while MultiLinePlotUnits used d_A for the same
d-spacing-in-angstrom unit. Align PlotAzintUnit to the lowercase crystallographic
convention already used by MultiLinePlotUnits. Generated Plot_unit_x::D_A is
left as-is (controlled by the OpenAPI generator's C++ enum casing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gmtime() returns a pointer to a shared static buffer, so concurrent
deserialization of CborUnixTime_tTag values was a data race. Decode into a local
struct tm via gmtime_r (gmtime_s on Windows) instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
images_sent was incremented right after image_pusher.SendImage(*loc), but the
ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it
silently drops the image (releases the slot and returns) when there is no live
connection or the 2 s enqueue deadline expires. So images_sent over-counted on a
broken/slow writer connection and disagreed with the ACK-based GetImagesWritten().
Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted
(enqueued/handed off) and only increment images_sent on success. The slot is
still released on the drop path. The authoritative delivered count remains
GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on
accept, preserving their previous always-counted behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The receiver already computes and stores the compression ratio per image
(JFJochReceiverPlots Add/GetPlots/GetPlotRaw all handle PlotType::CompressionRatio),
but the type was unreachable from the API: it was missing from ConvertPlotType,
the OpenAPI plot_type enum, and the TS client, so requesting it returned 500 and
no UI menu entry existed.
Wire it through all layers:
- broker: add the compression_ratio -> PlotType::CompressionRatio case in
ConvertPlotType.
- spec: add compression_ratio to the plot_type parameter enum.
- frontend: add COMPRESSION_RATIO to the generated plot_type enum (matches what
regeneration would produce), a "Compression ratio" menu entry, and a Y-axis
label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step toward building jfjoch_viewer on Windows/MSVC. No change to the Linux
build: D-Bus stays on by default and the XCB plugin is still used.
- CMake: make Qt6::DBus optional via JFJOCH_VIEWER_DBUS (ON on Linux, OFF on
Windows/macOS where Qt6::DBus does not exist); compile/link/install the dbus/
adaptor + service file only when enabled. Select the platform integration
plugin per-OS (QXcb on Linux, QWindows on Windows, QCocoa on macOS) instead
of hard-coding QXcbIntegrationPlugin.
- JFJochViewerWindow: wrap the adaptor include and D-Bus registration in
#ifdef JFJOCH_VIEWER_DBUS.
- JFJochImageReadingWorker: the POSIX open()/fstat()/NFS-errno preflight is now
#ifndef _WIN32, with a portable QFileInfo fallback elsewhere; POSIX-only
headers are guarded too.
- Replace the POSIX M_PI/M_PI_2 extension with C++20 std::numbers::pi across the
viewer (not defined by MSVC without _USE_MATH_DEFINES).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only gen/model/ is compiled (into JFJochAPI) and tracked; the REST server is
hand-written on cpp-httplib, so the generated Pistache server stubs (api/,
impl/, main-api-server.cpp, CMakeLists.txt, README.md, .openapi-generator*) are
regeneration debris that was previously left untracked and tended to carry a
newer spec version than the committed models. Ignore it so it can't be
committed by accident.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a status-bar cluster, shown when connected over HTTP, that surfaces
the broker state and live acquisition info:
- broker state box (QProgressBar) with progress drawn as the bar fill,
polled ~1 Hz on its own timer independent of image sync
- Live/Disconnected connection badge (host:port in tooltip)
- "+N new" badge and effective live-rate (Hz) readout
All widgets are fixed-width and only blanked in place, so the bar never
reflows when things appear/disappear.
Add a third autoload mode (HTTPSyncDataset): manually selecting an image
while following live now freezes the displayed image but keeps the dataset,
plots and image count updating. Surfaced via a tri-state HTTP Sync button
(off / live / data-only).
Also fix GetBrokerStatus() dropping the message field and a couple of
copy-pasted exception strings in JFJochHttpReader.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runs ROIIntegrationGPU and ROIIntegrationCPU on identical input and asserts
every per-ROI field (sum, sum_square, max, pixels, weighted centre, masked
count) matches bit-for-bit. Uses overlapping ROI boxes (multi-bit masks),
negative pixel values (signed weighted-sum path), and an injected saturated
and masked pixel per ROI to cover the "max only" and "fully excluded" branches.
Guarded by JFJOCH_USE_CUDA and skips with a warning when no CUDA GPU is present,
mirroring ImageSpotFinderGPUTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three robustness fixes for the writer-facing TCP stream, addressing the
spurious "wrong number of images" / connection failures seen under load.
1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path
passes a caller-owned buffer with z == nullptr and reuses it for the next
frame immediately. With MSG_ZEROCOPY the kernel still references that buffer
after send() returns, so the peer could receive corrupted frames and drop
the connection mid-stream, truncating the run. Zero-copy is now gated on a
ZeroCopyReturnValue that keeps the buffer alive until completion.
2. Make the SendAll watchdog measure lack of progress, not total wall-clock.
The previous absolute deadline tore down a healthy but back-pressured
connection (slow/starved writer) after a fixed window; the watchdog now
resets on every byte actually sent, so only a genuinely stuck socket is
closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR.
3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still
be draining a backlog of DATA frames when END is sent; each DATA ACK is
progress, so the timeout only arms once the writer falls silent rather than
firing while images are legitimately still being drained.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MXAnalysisWithoutFPGA never filled DataMessage.roi, so ROI integrals were
only available on the FPGA path. Add a software ROI engine that mirrors the
FPGA roi_calc kernel: per-ROI sum, sum of squares, good-pixel count, max and
intensity-weighted centre of mass, with each pixel carrying a 16-bit mask so
it can contribute to any subset of up to 16 ROIs.
New image_analysis/roi/ library (JFJochROIIntegration), structured like azint:
a base that precomputes the per-pixel mask and names, a templated CPU engine
(generic over pixel type for a future 16-bit path), and a GPU kernel using
per-block shared-memory atomics for the STXM case (half-detector ROIs).
Masked pixels are excluded entirely; saturated pixels are excluded from the
sums but still count towards the max, matching roi_calc exactly. The engine is
only constructed when at least one ROI is defined. Downstream CBOR/HDF5 already
consume message.roi, so no further changes are needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NUMA CPU/memory pinning is no longer worthwhile: the FPGA DMA buffers are
placed device-local by the kernel (dma_alloc_coherent), the big RAM ring
buffer is random-access (first-touch handles placement), and GPU work is
already spread across all visible devices. So drop the pinning entirely
and with it libnuma.
- Delete NUMAHWPolicy; the only concern worth keeping - GPU selection -
is done directly via pin_gpu() (round-robin over visible GPUs) in the
indexer pool and the Lite analysis threads. CPU-only threads
(FPGA acquire/pedestal/summation/frame-transform) no longer bind
anything.
- Drop get_gpu_numa_node() (sysfs lookup) - only SelectGPUAndItsNUMA
used it.
- numa_policy broker setting is deprecated and ignored (kept in the API
for backward compatibility; warns once on startup).
- Remove NUMA_LIBRARY / numa.h / numaif.h detection from CMake.
- Docs: drop the NUMA dependency, remove the numa_policy config example,
and document running multiple brokers on disjoint GPUs via
CUDA_VISIBLE_DEVICES.
- Remove NUMA_GPU_REVIEW.md (the planning note; this work is now done).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Several items in the note had landed or were inaccurate. Update it to:
- mark DONE: G2 pin_gpu, ImageBuffer first-touch, acquisition_device
de-NUMA, CUDAWrapper sysfs node lookup;
- add the previously-missing FPGA DMA buffer section - placement is a
kernel concern (dma_alloc_coherent, device-local), not the userspace
mbind, which was only the simulator;
- record that libnuma is now down to a single file (NUMAHWPolicy.cpp);
- note that dependency removal (G3) and pinning behaviour (G4) are
separable axes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The base AcquisitionDevice no longer allocates or frees frame buffers;
buffer_device is now just a non-owning view of addresses. Each subclass
owns its backing memory and the matching lifecycle:
- PCIExpressDevice mmap's the kernel DMA buffers and munmap's them in its
own destructor (and on ctor failure), symmetric with MapKernelBuffer.
- HLSSimulatedDevice owns plain zeroed heap buffers it points
buffer_device into, declared before the HLSDevice so they outlive the
action thread that writes them. The buffers are page-aligned to match
the real device's kernel DMA buffers - the modelled AXI datamover and
FPGAIntegrationTest require aligned output buffers.
This drops the NUMA/mmap dance from the simulated path (not
performance-critical) - removing libnuma from acquisition_device - and
replaces the base-class cleanup that had to guess the allocation
strategy with a single clear owner per device.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allocate the ring buffer with plain malloc and zero it across
hardware_concurrency() threads, so each page is first-touched - and thus
NUMA-placed - by whichever node the scheduler ran the zeroing thread on.
For the random-access buffer this approximates the previous
numa_alloc_interleaved placement, speeds up the one-time fault-in of the
150-200 GB allocation, and drops the libnuma dependency from this file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The offline worker threads built MXAnalysisWithoutFPGA without selecting a CUDA
device, so all per-image preprocessing/spot-finding/azimuthal integration ran on
GPU 0 (only the indexer pool was distributed). Add pin_gpu() to CUDAWrapper - a
process-wide round-robin counter (counter++ % get_gpu_count(), no thread id, no-op
without a GPU, honours CUDA_VISIBLE_DEVICES) - and call it once per worker before
building the analysis resources so their CUDA streams/engines land on distinct
devices.
Also add NUMA_GPU_REVIEW.md: a working note mapping ImageBuffer/NUMAHWPolicy/GPU
dispatch with goals and a staged plan (multi-broker GPU isolation via
CUDA_VISIBLE_DEVICES, dropping libnuma, reassessing NUMA pinning for the FPGA path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
librt is Linux/glibc-only and does not exist on macOS/Windows. Link it via a
$<$<PLATFORM_ID:Linux>:rt> generator expression (both the main JFJochCommon link and
the CUDA block) so -lrt is still passed on Linux - where some distros need it for
clock_gettime/timer_*/pthreads - and dropped elsewhere. Verified -lrt is still on the
jfjoch_process link line on Linux.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CMakeLists.txt: fetch libzmq at the top level (zeromq/libzmq v4.3.5) before
slsDetectorPackage, so this project controls the ZeroMQ version instead of sls's
bundled archive. sls reuses it via its if(NOT libzmq_POPULATED) guard, so a single
libzmq-static target is built (no duplicate-target/double-symbol clash). Verified the
full Linux build still links (JFJochZMQ -> JFJochReceiver -> jfjoch_process).
- common/NetworkAddressConvert.cpp: guard the network includes for _WIN32
(winsock2/ws2tcpip vs arpa/inet).
- common/ImageBuffer.cpp: use std::malloc/std::free for the non-NUMA path instead of an
anonymous mmap (the mapping had no huge-page/mbind flags, so it was equivalent) -
portable and removes the POSIX-only dependency. The NUMA-interleave path is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add docs/JFJOCH_PROCESS.md describing the offline analysis tool, its
options, output files, and the broker/viewer/process distinction; mention
jfjoch_scale for re-scaling/merging.
- Rewrite docs/JFJOCH_VIEWER.md for consistency: functionality, HTTP env
vars (JUNGFRAUJOCH_HTTP_HOST/PORT), command line, and the real D-Bus API.
- Refresh docs/TOOLS.md to the current set of tools; add both pages to index.rst.
- jfjoch_process: fix stale self-name (jfjoch_analysis -> jfjoch_process) in
usage/license/logger.
- jfjoch_scale: unify --scaling-high-resolution with jfjoch_process (drop -D
short flag, make it long-only) and remove dead p/q/i short options.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- App: make getValues an arrow field so its `this` survives being passed
as the `update` prop (the post-edit statistics refresh was broken).
- DataProcessingSettings: guard prop sync with last_downloaded_s + _.isEqual
so the 1 s statistics poll no longer overwrites a slider mid-edit
(reference `!=` on a freshly-parsed object was always true → snap-back).
- DataCollection: collapse nested setState in grid-scan callbacks.
- PreviewImage: build an immutable settings copy instead of mutating state
in place; fix stale comment.
- MultiLinePlotWrapper: drop per-render console.log; render via a custom
Plot built on plotly.js-cartesian-dist-min (scatter+heatmap only),
cutting the main bundle 6.0->2.6 MB (1.8->0.8 MB gzip).
- package.json: remove dead react-scripts test/eject scripts and CRA
eslintConfig, move @redocly/cli to devDependencies.
- Delete unused CRA scaffolding (serviceWorker.js, react-app-env.d.ts,
setupTests.js).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The description said the "hdf5" sub-object is written under /entry/data, but
the writer (NXmx::UserData) and the example in the same description both use
/entry/user. Correct the path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Motivation no longer frames the format as serial-only: Jungfraujoch serves
both rotation and serial MX, and the two-layout rationale (per-image CXI
spots vs. dataset-wide NXreflections) is presented for both.
- Document the <prefix>_process.h5 reprocessing output of jfjoch_process: an
integrated-format master whose /entry/data/data is a VDS linking back to the
original images (all results, no image copies).
- Add a section on the header_appendix / image_appendix (user_data) mechanism:
how the free-form JSON flows through the start/image CBOR messages and how an
"hdf5" sub-object is persisted to /entry/user, with an example.
- Link the niggliClass row to International Tables for Crystallography A,
Table 3.1.3.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New docs/HDF5.md documents the on-disk HDF5/NeXus format produced by the
writer: a FAIR/derived-metadata rationale (CXI-style per-image spot layout,
NXreflections for integration), the master/data-file layout and the three
NXmx format variants, the NXmx-standard fields that are populated, and every
Jungfraujoch extension group (/entry/MX, /entry/reflections, /entry/azint,
/entry/roi, /entry/image, /entry/profiling, /entry/detector, /entry/xfel,
detectorSpecific, calibration, fluorescence, user). Content is derived from
writer/HDF5NXmx.cpp and writer/HDF5DataFilePlugin*.cpp and cross-checked
against the NXmx and NXreflections definitions.
JFJOCH_WRITER.md's stale, partial structure table is replaced by a pointer to
the new doc; HDF5 is added to the Sphinx toctree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the protocol doc in line with CBORStream2Serializer/Deserializer:
- reflection dist_ewald -> rp (dist_ewald is the spot field, not the
reflection field)
- az_int_std -> az_int_profile_std, az_int_count -> az_int_profile_count
- threshold_energy typed as Map(string -> float), not float
- end_date typed as string (it is serialized as a plain text string, with
no datetime tag, unlike arm_date)
- add missing reflection fields: phi, zeta, image_scale_corr
- add missing spot fields: image, and indexed-only h, k, l, dist_ewald
- add image indexing_lattice_count and end indexed_lattice_count
- drop the duplicated image_scale_factor row in the End table
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The master-file writer emits /entry/MX/indexedLatticeCount, but the
reader only looked for indexingLatticeCount (the name used by the
per-file MX plugin and data-file read path). Existing master files on
disk therefore returned no indexed-lattice-count vector. Read the
master's indexedLatticeCount first, falling back to indexingLatticeCount,
matching the niggli_class/niggliClass compatibility pattern. The writer
is left unchanged so already-written files keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PixelRefine's reference can now be a merged/observed-intensity MTZ, not only a
calculated F-model. If F-model is missing, read a mean-intensity column (IMEAN,
which a jfjoch merge writes; also I/IOBS/Iobs/I-obs, or any 'J'-type column) and
use it directly as I_ref (F-model is squared, intensities are not).
This enables a SELF-SEEDED / EM-style workflow: run a first pass (traditional
integration, or PixelRefine against an external reference) to produce a merge,
then re-run PixelRefine with `-z <that merge>.mtz` so it scales against the data's
own intensities - free of the reference structure's non-isomorphism bias - and
iterate. Verified: a jfjoch IMEAN merge loads as the reference and PixelRefine
runs against it (crystal 2, 55279 reflections, CC1/2 94.3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial
data. Each image is assigned to one CC1/2 half, so removing it perturbs only that
half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i).
A negative value means dropping the image RAISES CC1/2 (it disagrees with the
consensus). Images whose deltaCChalf is a low-side statistical outlier
(< mean - N*stddev) are skipped when merging. Reference-free.
Two passes over the retained integration outcomes; per-image contributions are
re-derived rather than stored, so memory stays O(unique reflections + images) for
full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off).
Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2
95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS
heterogeneous serial data (some crystals genuinely bad) but slightly trims a
homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative
cut still removes the tail), so it is opt-in. R-free is the real test (user's full
200k). Note: the reported overall N_obs still counts all observations; the exported
merge (and CC1/2) correctly exclude the rejected images.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-direction peak search took argmax|spec|. The projected-spot histogram has a
broad low-frequency ENVELOPE (spots cluster near the origin) whose magnitude can
exceed the true lattice peaks, so on weak / pink-beam frames every direction reported
a short envelope vector (~10 A) and the real 38-79 A axes never surfaced -> 0 candidate
cells -> 0% indexing. (Diagnosed on the lyso jet: the FFT returned only 10-15 A vectors,
the true axes entirely absent.)
Subtract a running-mean background of half-width ~15 A and pick the peak by its
PROMINENCE (mag - background) instead. The smooth envelope cancels to ~0 while sharp
lattice peaks - fundamentals and harmonics alike - keep their height, so the real axes
win. The prominence is also reported as the magnitude, so FilterFFTResults ranks
directions by real-peak strength rather than envelope. Ported identically to CPU
(prefix-sum window) and GPU (sliding-window in the kernel).
Validation (lyso, de-novo): jet FFT 0% -> 20.5% (CPU and GPU identical; vs FFBIDX
27%); crystal 2 95.3% -> 95.5% (no regression, CC1/2 95.8 / CCref 92.7 unchanged).
The ~15 A window is the validated optimum (wider over-smooths, narrower under-removes
the envelope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FFT histogram extent was written as maxQ = 2*pi/HighRes (the powder Q = 2*pi/d
convention) while the bin width and spot coordinates use the internal 1/d convention -
a unit conflation. It is benign: len_coeff = 2*max_length/histogram_size carries the
same factor, so recovered cell lengths are correct; the 2*pi only acts as a ~6.28x
zero-padding of the histogram.
Rewrite it transparently: one_over_d_max = 1/HighRes (1/d), with the 2*pi kept as an
explicitly-named oversampling factor. In this codebase Q always denotes 2*pi/d, so 1/d
is named one_over_d (never q). histogram_size is numerically identical to before, so
behaviour is unchanged (FFT de-novo crystal 2 95.44%, bit-identical merge). Documented
that the exact padding amount is load-bearing at the marginal-frame level (rounding 2*pi
to a nearby integer shifts indexing ~0.5-1%).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to 40da1aab. Conventionalise the indexed lattice for BOTH indexers when
cell+SG are given, so every frame reports its cell in ONE consistent setting per
space group (mixed axis orders like [78,78,38] vs [38,78,78] index the same
reflection as different HKLs and cannot be merged).
Use LatticeSearch's conventional cell when its detected symmetry agrees with the
user's space group. On noisy frames LatticeSearch can pick an alternative Bravais
setting (e.g. the sqrt2 C-centred description of a primitive tetragonal cell,
[78,78,38]->[110,111,38]) whose system disagrees; there:
- FFBIDX already returns the reference setting (c-last), consistent with the
conventional frames, so its raw lattice is safe -> use it. This recovers the
~3% of frames the unconditional version lost: FFBIDX is byte-for-byte neutral.
- de-novo (FFT/FFTW) return a Niggli-primitive cell with a DIFFERENT axis order
(c-first); merging it with the conventional (c-last) frames would corrupt the
data -> reject the frame instead.
Validation (lyso): FFBIDX c2 71.44% / jet 27.15% (both unchanged vs raw-lattice
baseline, CC1/2 94.6 / 91.9); FFT c2 +C+S 92.72% (CC1/2 95.8, all c-last,
consistent); FFT c2 de-novo 95.44% (unchanged). Verified per-frame that the c-axis
(~38A) is axis 2 in every merged frame - no axis-order mixing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Providing both a unit cell (-C) and space group (-S) silently broke the de-novo
indexers (FFT/FFTW) -> 0% indexing, while each flag alone worked. Root cause: the
`sg && GetUnitCell()` branch fed the indexer's raw lattice straight into
symmetry-constrained refinement. FFBIDX returns the lattice already in the
reference setting, but FFT/FFTW return an arbitrarily-oriented Niggli-primitive
cell; enforcing the crystal system on its mis-assigned axes rejects every frame.
Fix: for de-novo indexers only, reduce the lattice to the conventional setting
(LatticeSearch) before refinement. FFBIDX keeps using its raw lattice as-is, so
it is byte-identical to before (no regression). niggli_class stays unassigned (0)
in this branch - it is a property of the primitive cell incl. centering, which
LatticeSearch cannot recover from a user-supplied (possibly centered, e.g. C2)
cell. A proper primitive-cell indexing path (CrystFEL-style) is deferred.
Validation (lyso, -C79,79,38 -S96):
crystal 2 FFT : 0% -> 94.9% (CC1/2 95.8, CCref 92.7) = de-novo quality
crystal 2 FFBIDX: 71.4% (CC1/2 94.6) - byte-identical
jet FFBIDX: 27.2% (CC1/2 91.9) - byte-identical
(Jet FFT stays 0% - that is a separate, still-open issue: de-novo finds no
consensus on the weak serial-still frames, 0% even without -S; to investigate.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PixelRefine no longer refines the cell, so the symmetry-constrained cell
parameterisation inherited from XtalOptimizer has no manifold to constrain.
The decompose->rebuild round-trip (Gram-Schmidt orientation + symmetry B
matrix) merely reconstructed the lattice columns it started from, and for
non-triclinic systems it re-idealised the indexed cell (averaging a,b for
tetragonal; a,b,c for cubic; forcing alpha=gamma=90).
Replace both six-way switches (PredictedNode and BuildParameterBlocks) with a
single path: take the three real-space lattice columns (latt.Vec0/1/2())
directly and form the reciprocal node via the general cross-product inverse.
This reproduces every crystal system exactly from the actual cell, is more
faithful (no re-idealisation), and removes the crystal_system field plus two
now-unused includes. PredictedNode de-templated (only ever called with double).
Crystal symmetry still lives where it belongs: indexing (upstream) and merging
(downstream via the space-group HKL key).
A/B (both lyso P4_3 2_1 2 crystals, refined under tetragonal constraint, so the
old idealisation was already a no-op): stat tables bit-identical across all
shells -- crystal 2 CC1/2 94.6% / CCref 92.5%, jet CC1/2 91.9% / CCref 55.8% --
the only delta is 4th-digit float-ordering noise in the error-model b
coefficient (0.8143->0.8141, same ISa). Same merged intensities => R-free
unchanged. Net -81 lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(1) Promote the Term-2 profile-width multiplier from the PR_RMULT env knob to a real
BraggIntegrationSettings::profile_multiplier (default 6) + jfjoch_process
--profile-multiplier flag. Removes the last PR_* env knob; IndexAndRefine reads the
setting. (2) Flip the merge outlier rejection default to OFF (ScalingSettings
outlier_reject_nsigma 6 -> 0); it stays available via --reject-outliers <nsigma>, since
it helps the jet R-free but is dataset-dependent (drops crystal-2 CCref).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the env-gated experiments that A/B'd to dead-ends or are no longer needed,
returning PixelRefine to the clean factored Terms 1+2 plus the one validated keeper
(r1_multiplier, default 6):
- PR_ORIENT (per-image orientation refinement): R-free no-op (0.2618 vs 0.2625) -
XtalOptimizer's orientation is already optimal. Removes ShoeboxResidual,
OrientationRegularizer, PixelObs::weight, the refinement block and its fields.
- PR_SWEEP (orientation + cell-scale sweep): R-free no-op, degraded high-res CC1/2
(per-image overfit). Removes SweepOrientationCell and its fields.
- PR_LORENTZ (rotation Lorentz/zeta): hurt both directions (the factored partiality
already subsumes it); was already reverted.
- PR_MLCENSUS (multi-lattice census in AnalyzeIndexing): served its purpose (~3-5%
of jet frames are multi-lattice; shelved).
PR_RMULT (the validated Term-2 multiplier knob) is kept. Defaults unchanged:
crystal 2 / jet / hybrid -R -r pixelrefine all reproduce.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Promote the per-observation merge outlier rejection from the temporary PR_REJECT
env knob to a real setting: ScalingSettings::OutlierRejectNsigma (default 6, <=0
disables), driven into MergeOnTheFly via the constructor, with a --reject-outliers
CLI flag in jfjoch_process. Default-on at 6 sigma matches XDS (MISFITS/REJECT) and
DIALS (normalised-deviation test); it validated on the full-jet R-free (0.2625 ->
0.2585). Applies to both the PixelRefine and classical merge paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
At the jet's ~1000x multiplicity R-free is bias-limited, and the merge had NO
outlier rejection - serial data zingers/overlaps/mis-indexed frames bias every
merged mean. Add a robust per-observation cut: drop observations whose corrected
intensity lies > reject_nsigma error-model sigmas from the reflection's MEDIAN.
The error-model sigma already captures the genuine (partiality) scatter, and the
median is a robust centre, so only the tail beyond the real spread is removed -
not good partials. The median is computed in RefineErrorModel (which already pools
the observations per reflection); AddImage applies the cut.
Env-gated via PR_REJECT=<nsigma> (off by default); logs the count removed. On the
jet (CC proxy) it lifts CCref +8 (nsigma 6, 0.6% cut) to +11 (nsigma 3, 7.4% cut)
- the cut is vs the data's own median, not the reference, so the gain is real
cleaner means. R-free validation + the nsigma sweet spot (over-rejection risk at
low nsigma) are for Filip's full-jet R-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
R-free validation (full jet, 1.5 A) confirmed the r1_multiplier fix: PR x6 beats
traditional (R-free 0.2625 vs 0.2802), the multiplier optimum is ~6 (x6==x9 on
R-free; x9 only buys CC1/2 internal consistency), and per-image orientation
*refinement* is a no-op (0.2618 vs 0.2625).
Re-adds the reference-driven orientation + uniform cell-scale SWEEP behind PR_SWEEP
(off by default) to test whether the cell-scale degree of freedom - which the
gradient orientation refinement lacks, and which moves the high-res shells radially
- helps R-free at 1.5 A on serial data. CCref is a near-no-op on the jet (as for
the gradient path), but that does not certify R-free, so it is left for validation.
NOTE: PR_RMULT / PR_ORIENT / PR_SWEEP remain temporary diagnostic env knobs; once
the sweep R-free is in, bake r1_multiplier=6, drop the no-op paths, strip the knobs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-jet R-free showed the factored model regressed vs the traditional pipeline
(R-free 0.34 vs 0.26, CCref 58 vs 76), traced to Term 2 using the raw measured
profile width: it is too tight (~0.002 A^-1), so the template sits off the ~0.4 px
centroid-floor scatter and underestimates the intensity (box-sum is immune). Fix,
XDS-style (integrate over ~6 sigma): multiply the measured R1 by r1_multiplier
(default 6) before use. On the jet at 1.3 A this recovers CCref 55.2 -> 59.3
(~ traditional's 60.5); crystal 2 -0.7. Tunable via env PR_RMULT for R-free.
Also reinstates the pre-factored per-image orientation refinement (per-pixel
ShoeboxResidual against the shoebox, regularised to the spot-centroid orientation,
re-predict), behind env PR_ORIENT (off by default), to A/B its effect on serial-data
R-free. On the jet CCref it is a no-op (XtalOptimizer orientation already at the
optimum), but CCref is a weak R-free proxy here, so it is left for R-free validation.
NOTE: PR_RMULT / PR_ORIENT are temporary diagnostic env knobs; the final multiplier
value and the orientation-refine decision are to be fixed against R-free, then baked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bumped the default signal-box radius to r1=4 (r2=6, r3=8): on the lysozyme jet
(1% DMM) it lifts CCref 50.1->52.2% and CC1/2 90.8->92.5% (its broadened spots
spill past a radius-3 box), is neutral on the mono crystal and on the classical
integrator. Added 'jfjoch_process --integration-radius <r1|r1,r2,r3>' (a single
value derives r2=r1+2, r3=r1+4) and wired PixelRefine's shoebox radius to
BraggIntegrationSettings::GetR1() so it shares the classical integrator's knob.
(Explored but rejected: an elliptical/anisotropic Term-2 profile. The jet's
tangential spots are mildly anisotropic - axis ratio 1.15->1.44 low->high res,
azimuthal mosaic, separate from the radial DMM bandwidth - but using the measured
2x2 covariance as the profile DEGRADED the jet, CC1/2 92.5->83.5: the tight
measured width loses to a generous aperture, same lesson as the radius bump.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PixelRefine is now an intensity-only operation: geometry is fixed (refined
upstream by XtalOptimizer) and the only objective is the factored per-reflection
likelihood (FACTORED_MODEL.md Terms 1+2) - measured per-resolution profile width
R1 plus one Fisher-weighted intensity/scaling residual per reflection, fitting
the per-image scale G and B. Validated on crystal 2 (fixed_master.h5 as stills,
1.7 A): CC1/2 84-92%, CCref 77-92%, flat - reproduces the env-flag prototype and
matches the rotation path from the stills path.
Removed:
- the per-pixel ShoeboxResidual loss and PixelResidual cost functor;
- all in-PixelRefine geometry refinement (orientation/cell/beam/distance/R),
the regularised-orientation LSQ, signal-weighting, and the global sweep;
- Term 3 (per-spot recentring) - a confirmed no-op on both crystals;
- the diagnostic scaffolding (covariance, centroid, adaptive_R1) and the
PR_* env knobs + stderr dumps in IndexAndRefine;
- the PredictImage/ChiSquaredImage renderers and the entire viewer
PixelRefine window/table/params + worker bindings + shoebox overlay.
The sweep box-integrator background median became mean (consistency) by virtue
of removing the sweep. METHODS.md rewritten for the current model; findings
recorded in FINDINGS-2026-06.md. Net -2200 lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Snapshot of the messy state: factored likelihood Terms 1+2+3 behind PR_*
env flags (PR_INTENSITY/PR_SHAPE/PR_RECENTER) alongside the old per-pixel
ShoeboxResidual, plus diagnostic scaffolding (PR_R0/R1/COV/FIX_R0/FIX_R/
ADAPT_R1/CENTROID/RECENTER) and the FACTORED_MODEL.md spec. Next commit
makes Terms 1+2 the model and strips all of this.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses code-review findings on RefineErrorModel:
- Floor the 1/dev^2 bin weight relative to the data scale (1e-3 of the median
bin dev^2), not an absolute 1e-30: a near-zero-scatter bin could otherwise
acquire a runaway weight and hijack the global (a,b) fit.
- Reject a near-collinear normal-equation system relatively (det > 1e-10*Ass*AII)
instead of with an absolute threshold that an ill-conditioned fit can pass.
- Reset the model to identity at entry so any early return leaves it inactive
rather than keeping a stale a/b alongside a freshly-cleared mean map (which
would make CorrectedSigma fall back to the per-observation I).
- PixelRefine: correct the orient_prior comment - with the sweep on, the LSQ
anchor is the swept orientation (intended), not the spot-centroid one.
Verified unchanged on the lyso test set (ISa 1.1, CC1/2 90.3%).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Background estimate: use the mean of the local ring, not the median. For a
right-skewed (Poisson) background the median sits below the mean, so subtracting
it under-subtracts and biases every weak intensity positive; over multiplicity
this becomes fake <I/sig> of a few in no-signal high-resolution shells. Fixed in
both PixelRefine and BraggIntegrate2D (the classical route had the same bug).
<I/sig> now tracks CC honestly and the true resolution limit is visible.
Error model: fit a global a, b (XDS form sigma'^2 = a*sigma^2 + (b*I)^2) from the
scatter of symmetry equivalents at the merge level (so both integrators benefit),
and print it with ISa = 1/b in jfjoch_process. The (b*I)^2 term uses the
reflection mean (not the per-observation I_i, which biases the weights and
collapses CC); a,b come from a relative-weighted bin regression. Replaces the
earlier per-resolution-shell variant, which was partly masking the background bias.
METHODS.md: document both (Sections 6-7), integrator-agnostic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
