31 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	5fe8a967cd	grid scan: the five lanes become one, with a single home for the crystal type and its settings ... Integration of the per-image detection scores, the analysis mode, the grid-scan crystal search, its rugnux entry point and the viewer display. GridScanCrystal/GridScanResult had two definitions - a placeholder in common/ and the real one in image_analysis/ - which is a redefinition in any translation unit reaching both, and tests/RasterReportTest.cpp reaches both. Unified into common/GridScanResult.h, beside ScanResult where the data type belongs, leaving the algorithm in image_analysis/. Same reason UnitCell lives in common while the indexers do not. GridScanAnalysisSettings is now the only home for the search parameters, replacing the loose GridScanAnalysisParameters struct the raster lane carried while the class did not yet exist. Three values changed with the move: - decisive_single_cell_score 0.9 -> 0.6. 0.9 drops a real two-cell crystal peaking at 0.751 and costs a loop on the labelled corpus. 0.6 is the middle of a measured gap: over 67 rasters no water raster peaks above 0.15 and no ice raster above 0.50, while the weakest confirmed-protein raster peaks at 0.67. - max_crystals is std::optional, unset meaning no cap. 0 as a sentinel for "unlimited" reads as "find nothing", which is the opposite of what it did. - grow_score_threshold was missing from the class entirely. The viewer reads protein_score, ice_score and the crystal list from the reader rather than a local stub, and asks the broker for ice_ring_ratio rather than the retired ice_ring_score spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:59:04 +02:00
leonarski_f	de3fdf9534	Merge branch 'worktree-agent-ae76aa0c66b33a4db' into grid-scan-integration	2026-09-08 07:55:31 +02:00
leonarski_fandClaude Opus 5	0f00b76a9a	image analysis: the ice score takes the pipeline's own band width, and the ice quantities are named for what kind of number they are ... Two things, both about telling one ice quantity from another. The ice score's spot channel had its own band half-width of 0.02 A^-1 while the spot finder marks ice rings at 0.03 (ice_ring_width_Q_recipA). The 0.02 was justified by a 5 pp specificity gain measured on the PYTHON PROTOTYPE, which used a fitted beam centre and a mask-derived coverage table; the shipped port, which takes the geometry's centre and the azimuthal profile's own live pixel count, does not reproduce it. Measured over the corpus by truth class rather than by directory label, at 0.02 vs 0.03 on the combined score: ice loops 62.13/62.19%, _icy protein 89.03/89.79%, _clean protein 16.51/16.31%, water 17.19/20.03%. The widths are indistinguishable except on water, where one of the four loops is independently known to carry a full hexagonal pattern. So the width is now a parameter and the pipeline's own value is passed in - one band width, not two. The 0.012 tolerance in the radial channel is NOT a second band width, and is renamed CENTRE_SMEAR_Q to say so: it is how far either side the channel looks for the bin a mis-set beam centre moved the ring to. The rest is naming. Three kinds of number were all called score, or built from things called count, and a reader could not tell from the name whether 1 meant "none" or "certain" - which are opposite. The convention, now stated in docs/CPU_DATA_ANALYSIS.md: *_score is bounded [0,1] and 1 is certainty, *_ratio is unbounded and 1 is nothing, *_count is a count. The C++ identifiers for the ice ring ratio follow it (ice_ring_score -> ice_ring_ratio, GetIceRingScore -> GetIceRingRatio, PlotType::IceRingScore -> IceRingRatio), and the local in the scaling gate that shadowed the new ice_score while meaning the ring ratio is renamed with them. Nothing outside the source moved: the CBOR keys ice_ring_score and ice_ring_score_mean, the datasets /entry/MX/iceRingScore and iceRingScoreMean, the ice_ring_score plot type and the --ice-min-score flag are all unchanged, and were checked to be after the rename. Renaming those changes stored files, the stream format, the REST API and a CLI flag, and is a separate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:19:03 +02:00
leonarski_f	294eabdb4b	Merge branch 'rc167' into worktree-agent-ae76aa0c66b33a4db ... # Conflicts: # viewer/JFJochViewerDatasetInfo.cpp	2026-09-08 00:00:55 +02:00
leonarski_fandClaude Opus 5	cd31747c99	viewer: TEMPORARY STUB - the grid-scan composite reads protein, ice and the crystal list from a local placeholder ... REVERT THIS COMMIT AT INTEGRATION. protein_score, ice_score and the grid-scan crystal list are not on JFJochReaderDataset yet; they arrive through reader/ with a separate change. Until then GridScanDataStub.h defines GridScanCrystal and three accessors that return nothing, so the widget and the dataset-info panel are already written against the real shape of the data and compile today. Reverting this commit leaves exactly the two includes and the CMake entry to replace: take GridScanCrystal from the reader, and the three accessors become ds.protein_score, ds.ice_score and ds.grid_scan_result at their call sites in JFJochViewerDatasetInfo.cpp. Nothing else changes - both data paths fill the same JFJochReaderDataset, so HTTP and HDF5 light up together. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:00:13 +02:00
leonarski_fandClaude Opus 5	0c6167b7c5	viewer: a grid scan shows protein and ice over the shape of the loop, with a frame around each crystal ... Two scalar maps side by side never answered the question a raster is taken to answer. The grid-scan view gains a composite: the normalised background gives the loop its shape in grey, ice paints it cyan, protein orange, and where both are present the two mix in proportion. Protein and ice go in as they come - they are saturating scores already in [0,1], and stretching one would turn "no protein anywhere" into "protein everywhere". Only the background is normalised, with its range floored, so a scan of pure air stays flat instead of having its noise stretched into a convincing loop. Each crystal found in the raster gets a frame, drawn oriented: centred on the crystal, sized along its own principal axes and rotated by its angle. An axis-aligned box would hide the case the axes exist to expose - a 100 x 10 um needle at 45 degrees reads as a compact blob. The extent is in micrometres and goes through both step sizes separately, which a real 20 x 16 um raster needs. A blob within 1.2 of round is drawn as a circle, since its angle means nothing. The single-scalar path is untouched; the composite is one more entry in the metric combo, offered first for a grid scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 23:59:56 +02:00
leonarski_fandClaude Opus 5	26a82ddc72	spindle: the per-image severity travels the whole data path, and absence travels with it ... The score was computed and then thrown away - carried on the per-image message but transported nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image /entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot and scan_result schemas, and the viewer and frontend plot menus. Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody could look, and the second must engage the recovery protocol while the first must not. The round-trip tests pin all three states through CBOR and through a written-and-reopened file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00
leonarski_f	e2b240356c	v1.0.0-rc.121 (#28) ... This is an UNSTABLE release. * jfjoch_broker: Report changes in the image buffer, so viewer doesn't reload constantly * jfjoch_viewer: Improve performance of loading images * jfjoch_viewer: Auto-throttle image loading in HTTP-sync / movie modes * jfjoch_viewer: Auto-foreground calculated with histogram * jfjoch_viewer: Fix rare segmentation fault Reviewed-on: #28 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-12 21:24:20 +01:00
leonarski_f	d1a4c19ef3	v1.0.0-rc.117 (#24) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_viewer: Add ROI results to the dataset info plots * jfjoch_writer: Remove HTTP interface, as it is not needed/used at the moment Reviewed-on: #24 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-05 22:03:07 +01:00
leonarski_f	c8bbf8a186	v1.0.0-rc.115 (#22) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Default spot finding settings can be configured via config JSON * jfjoch_viewer: FFT analysis of data in the dataset plot Reviewed-on: #22 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-04 11:56:14 +01:00
leonarski_f	31a357fa57	v1.0.0-rc.113 (#19) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Improve handling of rotation indexing * jfjoch_broker: More information saved in CBOR end message (WIP) * jfjoch_writer: Save rotation indexing lattice parameters and Niggli class * jfjoch_viewer: Remove (for now) primitive cell information Reviewed-on: #19 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-02 09:29:22 +01:00
leonarski_f	224cc8b89c	v1.0.0-rc.110 (#16) ... This is an UNSTABLE release. * jfjoch_broker: Add auto-contrast option for preview images * Frontend: Add logo image * jfjoch_viewer: Add logo image * jfjoch_viewer: For image chart allow to set min value to zero * jfjoch_viewer: For resolution estimation plots, visualization uses 1/d^2 as measure * jfjoch_viewer: Add 3D unit cell visualization (experimental/WIP/not really there) * Documentation: Add logo image Reviewed-on: #16 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-28 12:47:35 +01:00
leonarski_f	ddcc24fba8	v1.0.0-rc.106 (#11) ... This is an UNSTABLE release. * jfjoch_viewer: Allow for multiple dataset info plots * jfjoch_viewer: Highlight current element in grid Reviewed-on: #11 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-21 11:03:29 +01:00
leonarski_f	a6b5cb5629	v1.0.0-rc.105 (#10) ... This is an UNSTABLE release. * jfjoch_viewer: Clean-up widgets slightly * jfjoch_viewer: Limit right panel to 600 pixels * jfjoch_viewer: Parse crystal symmetry type * jfjoch_viewer: Grid scan view takes color map and can be fit to zoom Reviewed-on: #10 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-20 09:46:25 +01:00
leonarski_f	95acf3aba3	v1.0.0-rc.104 (#9) ... This is an UNSTABLE release. jfjoch_writer: Fix and improve the way grid scan geometry is saved (non-NXmx extension makes it way easier) jfjoch_viewer: Display grid scan results in 2D (work in progress) jfjoch_viewer: Improve auto-scaling on start of images (work in progress) jfjoch_viewer: Add B-factor and resolution estimate to the dataset info plots Reviewed-on: #9 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-19 17:28:10 +01:00
leonarski_f	808691251e	v1.0.0-rc.101 (#6) ... This is an UNSTABLE release. * jfjoch_viewer: Auto load is better handling change of states * jfjoch_viewer: Fix DBus registration * jfjoch_viewer: Handle charts better with vertical lines on hover and status bar update * jfjoch_viewer: Calculate ROI in a more efficient way Reviewed-on: #6 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-16 12:35:00 +01:00
leonarski_f	5bb8898a0e	v1.0.0-rc.85	2025-09-24 22:52:31 +02:00
leonarski_f	caf2a9de4f	v1.0.0-rc.79	2025-09-15 21:38:58 +02:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	fa95858008	v1.0.0-rc.69	2025-08-17 21:21:20 +02:00
leonarski_f	06c5b9cf7f	1.0.0-rc.65	2025-07-16 20:19:48 +02:00
leonarski_f	b245967df3	v1.0.0-rc.38	2025-05-12 14:17:24 +02:00
leonarski_f	040cf08386	v1.0.0-rc.36	2025-05-05 19:32:22 +02:00
leonarski_f	ddf4c75645	v1.0.0-rc.31	2025-03-02 13:15:28 +01:00