17 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	a95aca382c	Remove dead code left behind by recent changes ... None of this has a reader: - ScalingSettings::scaling_regularize and its setter/getter - ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read), with the timing that only fed the latter - JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the comment claiming the retry uses it - the retry keys off initial_fit_done_ - JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include in a file that uses none - an unused gemmi::Op accumulator in the spindle-angle helper - <random> in Merge.{h,cpp}, from before the half-set split became a hash - an orphaned comment describing the Ceres B-factor residual deleted in 014e43a4c, and two trailing comments that had collided on one line Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:29:42 +02:00
leonarski_fandClaude Opus 5	dc5721ad8a	Viewer: repaint the resolution readout when the view scrolls ... The "d = ... A" readout moved from a scene item flagged ItemIgnoresTransformations to a fixed viewport position painted in drawForeground, which is what made a hover update dirty a small rect instead of the whole viewport. But QGraphicsView pans by blitting the viewport: the painted text is shifted along with the image and left there, and the pending update for its old position is translated away too, so dragging the image smears ghost copies of the readout across the corner. Dirty the old and the new rect when the view scrolls - still a couple of hundred pixels, not the viewport. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:22:27 +02:00
leonarski_fandClaude Opus 5	16bf3408f0	Address code-review findings; make detection limits detector-driven ... One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 5	84a1538495	Viewer: drop the in-view ROI accumulation, the worker already does it ... The statistics shown for a drawn ROI do not come from the view at all. Drawing one promotes it to a named ROI, roiGeometryEdited goes to the reading worker, and the worker's per-image results arrive in ImageData().roi, which is what the Inspector's ROI section displays. So accumulateROI/CalcROI/roiCalculated were a second implementation of the same thing whose output nothing read -- and the worker's version is the better one: it handles the mask and it persists per image. Remove them. The view now owns only the ROI's geometry and gestures, which is all the worker needs from it. This corrects the previous commit's claim that nothing surfaces ROI statistics: the Inspector does, via the worker. Verified by drawing a box over the beam centre: Sum 65453, Max 1634, Mean 0.468, centre of mass (786.3, 843.4) against a beam centre of (764, 850). Note the numbers appear from the next analysed frame onward, since the worker attaches them at analysis time and the displayed frame was analysed before the ROI existed -- that behaviour is unchanged here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 21:42:30 +02:00
leonarski_fandClaude Opus 5	57f9e42382	Viewer: the region of interest belongs to the diffraction view alone ... Drawing an ROI only means something where there are detector counts to accumulate. Gate the gesture on a virtual AllowROI(), true only for JFJochDiffractionImage: shift-drag, the resize handles, the hover cursor and the "Clear ROI" context entry now do nothing in the azimuthal, grid-scan and calibration views, which cannot report anything about a box anyway. The statistics move out of the base class into the diffraction view and read the int32 image directly, so no float copy of the detector image is built for them either. With the labels already converted, image_fp is now untouched by the diffraction view, and the lazy EnsurePixelValues machinery it needed is gone. image_fp stays as the base's representation for the views whose data really is float: the azimuthal profile, the grid-scan 1/sigma^2 map, and the calibration viewer's eight source types. Removed with it: the ROI readouts in the calibration and 2D azimuthal windows, which were the only two consumers of roiCalculated -- the diffraction view emitted it and nothing listened. Nothing surfaces ROI statistics now; the pixel-mask case wants rectangles counting excluded pixels and deserves its own design. JFJochViewerROIResult is still used by the side-panel ROI list, so the widget stays. Verified in the GUI: shift-drag in the diffraction view still draws the box, turns it into a named ROI and runs the statistics; fit-view panel remains pixel-identical to the pre-series baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 21:32:47 +02:00
leonarski_fandClaude Opus 5	27615a8a1d	Viewer: label pixels from the int32 image, and paint them instead of building items ... Two changes to the per-pixel value labels, which appear above 30x zoom. They were up to 5000 QGraphicsSimpleTextItems created and destroyed on every overlay rebuild - so on every pan step while zoomed in. Paint them in drawForeground() instead: no item churn, no scene invalidation, and the text is laid out in viewport pixels so it is a constant readable size rather than a scene-space font scaled by 0.2. Same approach as the magnifier's labels. The value text becomes a virtual, PixelLabel(). The base still formats from image_fp, which is what the genuinely float-valued views hold (azimuthal profile, grid-scan 1/sigma^2, the calibration viewer's eight source types). JFJochDiffractionImage overrides it to read the int32 image directly: counts are exact integers, so routing them through float32 is a detour that also cannot represent summed values above 2^24 exactly. Verified at 38 wheel clicks over a module edge: identical values and gap/contrast handling to the previous float path, now centred in each pixel. Fit-view panel still pixel-identical to the pre-series baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 21:27:17 +02:00
leonarski_fandClaude Opus 5	4892c57119	Viewer: paint the resolution readout in drawForeground, not as a scene item ... The hovered "d = ... A" readout was a QGraphicsTextItem flagged ItemIgnoresTransformations, repositioned on every mouse motion. Qt cannot compute a tight dirty rect for an item that ignores the view transform, so it marks the entire viewport dirty whenever such an item moves or changes text -- and this one moved constantly. Paint it in drawForeground() in viewport pixels instead, and repaint only the union of its old and new rectangles. That also removes the item lifetime special-casing: it was deliberately kept out of overlay_items_, had to be nulled by hand after scene()->clear(), and carried comments in three places warning about the dangling pointer. This does not reduce raw X11 traffic -- there every repaint uploads the whole window whatever the damage -- but it cuts the work per hover, and it does matter under a compressing remote protocol (VNC/NX/xpra), which encodes only the region that actually changed. Verified against the previous build: same text, colour and position. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 20:13:04 +02:00
leonarski_fandClaude Opus 5	501ce1ba3d	Viewer: rate-limit hover feedback to ~15 Hz ... The status bar, the resolution readout and the magnifier were all regenerated on every single mouse motion event. Each regeneration repaints, and on a remote X session a repaint uploads the whole window regardless of how little changed, so the pointer merely crossing the image saturates the link: measured with a counting relay in front of the X server, 50 motions over the image cost 273 MB, and a build with the hover work removed cost 18 KB. Rate-limit it. Two details matter: - The limit is applied inline, not from a timer. Running the update inside the mouse event keeps its damage in the same repaint as anything else that event triggers (a pan). A first attempt deferred the work to a timer instead, which split one repaint into two and made panning measurably worse. - The catch-up that reports the final position is debounced, not queued per skipped motion, so it fires once after the pointer stops rather than repeatedly mid-gesture. mouseHover() now takes the scene position and modifiers instead of the event, which also removes the identical mapToScene() from all four implementations. Hover traffic over 3 repeats: 173 MB mean -> 140 MB, and the run-to-run spread drops from +-14% to +-2%. The harness tops out near 30 motions/s, barely above the 15 Hz limit; a real mouse reports far faster, where the cap does more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 20:12:36 +02:00
leonarski_fandClaude Opus 5	3eccc58961	Viewer: colour the diffraction image straight from int32 ... image_fp is the base class's one pixel representation, and it earns that for three of the four image widgets: the azimuthal image is already float, the grid scan holds computed 1/sigma^2 floats, and the calibration viewer accepts eight source types from uint8 to float64. The diffraction image is the odd one out -- its source is a large int32 buffer -- and it is the one paying: a full int32 -> float pass plus a second resident copy of the image, on every frame. Split the mapping from the source. PixelColorMap holds the precomputed LUT constants and does value -> colour; a virtual ColorRow() picks the pixels out of whatever buffer the subclass has. Both paths now go through the same Apply(), so only the gap/bad/saturated dispatch differs, and it lines up exactly with the encoding LoadImageInternal used: GAP_PXL_VALUE -> NAN -> gap ERROR_PXL_VALUE -> -INF -> bad SATURATED_PXL_VALUE -> +INF -> saturated The base class still needs real pixel values for ROI statistics and per-pixel labels, so image_fp is filled on demand instead of per frame -- and only when something reads it: a non-empty scratch ROI, or labels above 30x zoom. Neither happens while simply looking at frames, and nothing else routinely sets roiBox (the named ROIs are computed in the reading worker, not here). 18.1 Mpx: 10.5 -> 5.9 ms per frame and 72 MB less resident. 4.5 Mpx: 1.8 -> 1.0 ms and 18 MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 19:12:55 +02:00
leonarski_fandClaude Opus 5	0b926af5af	Viewer: cache the traced resolution-ring contours ... DrawResolutionRings traced every ring point by point on each overlay rebuild: 361 ResPhiToPxl calls per ring, so about 4000 geometry evaluations per rebuild with the 11 ice rings shown -- and a rebuild happens on every pan step. The contours depend only on the ring list and the geometry, neither of which changes while the view moves, so keep them. The cache is keyed on the ring list (which RingMode::Auto recomputes from the visible area, so it still re-traces when it should) and cleared in loadImage for a possibly-new geometry. Labels are still placed per rebuild: they depend on the visible rect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 19:04:52 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00
leonarski_f	36d0507758	v1.0.0-rc.119 (#26) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Add binary export of data analysis plots over OpenAPI * jfjoch_broker: Minor fixes to HTTP error handling * jfjoch_viewer: Prefer binary plots over JSON plots * jfjoch_viewer: Change foreground with F button + wheel * jfjoch_viewer: Change way how angles are displayed * jfjoch_viewer: Display resolution of the mouse cursor in top left corner Reviewed-on: #26 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-08 19:53:35 +01:00
leonarski_f	224cc8b89c	v1.0.0-rc.110 (#16) ... This is an UNSTABLE release. * jfjoch_broker: Add auto-contrast option for preview images * Frontend: Add logo image * jfjoch_viewer: Add logo image * jfjoch_viewer: For image chart allow to set min value to zero * jfjoch_viewer: For resolution estimation plots, visualization uses 1/d^2 as measure * jfjoch_viewer: Add 3D unit cell visualization (experimental/WIP/not really there) * Documentation: Add logo image Reviewed-on: #16 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-28 12:47:35 +01:00
leonarski_f	d53e93023b	v1.0.0-rc.108 (#13) ... This is an UNSTABLE release. * jfjoch_viewer: Fix bug when resolution estimation/B-Factor/Profile radius were not set (NaN) * jfjoch_viewer: Show spots is off by default, resolution ring mode is enabled by default * jfjoch_viewer: Fit to window of image is now default when size of the grid changes Reviewed-on: #13 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-24 12:27:54 +01:00
leonarski_f	95acf3aba3	v1.0.0-rc.104 (#9) ... This is an UNSTABLE release. jfjoch_writer: Fix and improve the way grid scan geometry is saved (non-NXmx extension makes it way easier) jfjoch_viewer: Display grid scan results in 2D (work in progress) jfjoch_viewer: Improve auto-scaling on start of images (work in progress) jfjoch_viewer: Add B-factor and resolution estimate to the dataset info plots Reviewed-on: #9 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-19 17:28:10 +01:00