feat(vision): add lightweight VisionTuning GUI widget, extend testing howto

VisionTuning (csaxs_bec/bec_widgets/widgets/vision_tuning/) is a small BECWidget
for tuning a VisionInterface interactively: pick a device/camera, flip through
raw/grayscale/edges/clahe, set/clear an ROI, and watch the live diagnostic_image
preview -- modeled on z_ConsoleButtonsWidget's "few knobs" simplicity rather than
the full OMNY_XRayEye alignment GUI. Registered via `bw-generate-cli --target
csaxs_bec` (regenerates client.py/designer_plugins.py, the actual mechanism
gui.test.new("VisionTuning", ...) resolves against).

VISION_PIPELINE_TESTING_HOWTO.md gains a section on testing against the real
flomni sample-transfer h5 captures now in reference_images/: loading a capture's
single `reference_image` dataset (via set_image(), since load_image_file() only
reads cv2-loadable files), picking a reference vs. current frame by hand, setting
an ROI, and visually checking the edge/contour detection before trusting a score
-- deliberately a manual workflow, not an automated score sweep, since same-named
capture steps (get/put) can come from different physical scenes.

reference_images/README.md corrected to document both OMNY's per-camera-pair h5
layout and flomni's single-dataset layout, matching what's actually in the
directory.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FT8PkvfdQUBKc5X4pHrxB6

csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md

@@ -188,6 +188,89 @@ a human would notice. Useful for sanity-checking `edge_low_threshold`/`edge_high
 (`vision_toolkit.detect_edges()`'s defaults, `50`/`150`) against real lighting/background, which the
 synthetic square can't tell you anything about.
 
+## Real flomni sample-transfer captures: reference vs. current, ROI, and visual edge-detection
+
+`csaxs_bec/devices/vision/reference_images/` (gitignored, see its own `README.md`) is where a local
+set of real captures from flomni's `save_reference_image()` can be dropped for exactly this kind of
+manual testing -- one HDF5 file per confirm gate, e.g.
+`sample_transfer_reference_image_get_1_20260701_102105.h5`. Unlike OMNY's own format (one file per
+camera *pair*, keyed by camera name -- see `_otransfer_save_reference_images()`'s docstring), a
+flomni capture is one file with a single `reference_image` dataset:
+
+```python
+import h5py
+
+with h5py.File("csaxs_bec/devices/vision/reference_images/sample_transfer_reference_image_get_1_20260701_102105.h5", "r") as f:
+    image = f["reference_image"][()]  # RGB ndarray, e.g. (1920, 1080, 3) uint8
+```
+
+`load_image_file()` can't read this (`cv2.imread`-only, no HDF5 support) -- load the array yourself
+as above and hand it to `DummyVisionCamera` directly with `set_image()`, which takes an ndarray from
+anywhere:
+
+```python
+dev.vision_cam_a.set_image(image)
+```
+
+**This step needs a person, not a score sweep.** A filename like `get_1`/`get_2`/`get_3` only means
+"captured at the same confirm-gate step name" -- it does *not* mean "same physical scene". OMNY's
+`get` gate fires both when picking a pin up from the sample stage and when picking one up from
+storage, and storage has multiple slots -- two `get`-named captures can legitimately show completely
+different backgrounds. Don't assume same-named files form a matched set; look at them (or diff their
+overlays) before deciding which one is a valid "reference" for which "current" frame. This is a
+judgment call about what the two images actually show, not something a blind pairwise score sweep
+over all combinations can decide correctly on its own.
+
+The manual loop, in a running BEC session:
+
+```python
+# 1. Pick one capture as the reference, load and save it under a tag.
+with h5py.File(".../sample_transfer_reference_image_get_1_....h5", "r") as f:
+    dev.vision_cam_a.set_image(f["reference_image"][()])
+dev.vision_test_monitor.save_reference("manual_check")
+
+# 2. Load a different capture as the current frame and score it.
+with h5py.File(".../sample_transfer_reference_image_get_2_....h5", "r") as f:
+    dev.vision_cam_a.set_image(f["reference_image"][()])
+dev.vision_test_monitor.score("manual_check")
+```
+
+**Define an ROI before trusting the score.** These are real photos of a cluttered, reflective metal
+background -- `vision_toolkit.largest_contour()` picks the single largest edge blob in the *whole*
+frame, which on a busy background can lock onto a glare boundary or a scratch instead of the actual
+gripper/pin. Crop to the region that actually contains the object before scoring, and check the crop
+visually first (see below) rather than guessing coordinates blind:
+
+```python
+dev.vision_test_interface.set_roi((300, 700, 600, 550), camera_name="vision_cam_a")  # x, y, w, h
+dev.vision_test_monitor.score("manual_check")  # re-score with the ROI applied
+dev.vision_test_interface.set_roi(None, camera_name="vision_cam_a")  # clear it again
+```
+
+**Look at the edge detection and contour, don't just trust the number.** The fastest loop is calling
+the Layer 0 toolkit functions directly on the loaded array (no BEC round-trip needed) and writing
+the result to a file to view:
+
+```python
+from csaxs_bec.devices.vision import vision_toolkit as vt
+import cv2
+
+crop = vt.crop_to_roi(image, (300, 700, 600, 550))       # same ROI as above, or none
+edges = vt.detect_edges(crop, low_threshold=50, high_threshold=150)
+contour = vt.largest_contour(edges)
+overlay = vt.draw_contour_overlay(crop, contour)          # contour drawn in red on the crop
+
+cv2.imwrite("/tmp/check_crop.png", cv2.cvtColor(crop, cv2.COLOR_RGB2BGR))
+cv2.imwrite("/tmp/check_edges.png", edges)
+cv2.imwrite("/tmp/check_overlay.png", cv2.cvtColor(overlay, cv2.COLOR_RGB2BGR))
+```
+
+Open `check_overlay.png` and confirm the red contour actually traces the gripper/pin, not something
+else in the background, before trusting any score computed from it -- if it doesn't, adjust the ROI
+and/or `edge_low_threshold`/`edge_high_threshold` and try again. This is also the way to see the
+`diagnostic_image` overlay's equivalent without needing the GUI dock that doesn't exist yet (see
+below).
+
 ## What this does and doesn't prove
 
 Proves: the Layer 0/1/2 code paths work correctly end to end (registration, contour scoring,

csaxs_bec/bec_widgets/widgets/client.py

@@ -19,6 +19,7 @@ _Widgets = {
     "OMNY_XRayEye": "OMNY_XRayEye",
     "SAXSWidget": "SAXSWidget",
     "SlitControl": "SlitControl",
+    "VisionTuning": "VisionTuning",
     "z_ConsoleButtonsWidget": "z_ConsoleButtonsWidget",
 }
 
@@ -48,21 +49,16 @@ class DataViewer(RPCBase):
 
 
 class OMNY_SampleStorage(RPCBase):
-    """View and correct the FlOMNI sample-storage records."""
+    """View and correct the sample-storage records for whichever setup (FlOMNI"""
 
     _IMPORT_MODULE = "csaxs_bec.bec_widgets.widgets.sample_storage.sample_storage"
 
     @rpc_call
     def refresh(self) -> "None":
         """
-        Re-read all slots (one bulk device round-trip) and update only the
-        cells whose state actually changed.
-
-        Repainting every cell on every 2 s poll — even when nothing changed,
-        which is almost always the case for a sample magazine — was needless
-        work on the GUI thread. We diff against the last-seen state and only
-        call set_state() on cells that differ, so a steady-state poll does no
-        UI work at all.
+        Re-read all slots (one bulk device round-trip) and update only
+        the cells whose state actually changed. Dispatches to the active
+        setup's own refresh method.
         """
 
 
@@ -431,6 +427,12 @@ class SlitControl(RPCBase):
         """
 
 
+class VisionTuning(RPCBase):
+    """Pick a ``VisionInterface`` device/camera, tune its processing mode and ROI, watch the"""
+
+    _IMPORT_MODULE = "csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning"
+
+
 class XRayEye2DControl(RPCBase):
     _IMPORT_MODULE = "csaxs_bec.bec_widgets.widgets.xray_eye.x_ray_eye"
 

csaxs_bec/bec_widgets/widgets/designer_plugins.py

@@ -14,6 +14,7 @@ designer_plugins = {
     "OMNY_XRayEye": ("csaxs_bec.bec_widgets.widgets.xray_eye.x_ray_eye", "OMNY_XRayEye"),
     "SAXSWidget": ("csaxs_bec.bec_widgets.widgets.saxs_widget.saxs_widget", "SAXSWidget"),
     "SlitControl": ("csaxs_bec.bec_widgets.widgets.slit_control.slit_control", "SlitControl"),
+    "VisionTuning": ("csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning", "VisionTuning"),
     "z_ConsoleButtonsWidget": (
         "csaxs_bec.bec_widgets.widgets.console_buttons.console_buttons",
         "z_ConsoleButtonsWidget",
@@ -27,5 +28,6 @@ widget_icons = {
     "OMNY_XRayEye": "widgets",
     "SAXSWidget": "table_chart",
     "SlitControl": "widgets",
+    "VisionTuning": "widgets",
     "z_ConsoleButtonsWidget": "widgets",
 }

csaxs_bec/bec_widgets/widgets/vision_tuning/__init__.py

@@ -0,0 +1,3 @@
+from csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning import VisionTuning
+
+__all__ = ["VisionTuning"]

csaxs_bec/bec_widgets/widgets/vision_tuning/register_vision_tuning.py

@@ -0,0 +1,15 @@
+def main():  # pragma: no cover
+    from qtpy import PYSIDE6
+
+    if not PYSIDE6:
+        print("PYSIDE6 is not available in the environment. Cannot patch designer.")
+        return
+    from PySide6.QtDesigner import QPyDesignerCustomWidgetCollection
+
+    from csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning_plugin import VisionTuningPlugin
+
+    QPyDesignerCustomWidgetCollection.addCustomWidget(VisionTuningPlugin())
+
+
+if __name__ == "__main__":  # pragma: no cover
+    main()

csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.py

@@ -0,0 +1,236 @@
+"""A lightweight tuning widget for :class:`~csaxs_bec.devices.vision.vision_interface.VisionInterface`
+(Layer 1 of ``AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md``).
+
+Deliberately small -- modeled on ``z_ConsoleButtonsWidget``'s "yesno" simplicity, not on the full
+``OMNY_XRayEye`` alignment GUI: pick a ``VisionInterface`` device and one of its cameras, flip
+through processing modes, set/clear an ROI, and watch the live ``diagnostic_image`` result. No
+``SampleVisionMonitor`` (scoring) controls here -- that is a different device/concern and would
+roughly double the knob count; a follow-up panel can add it later if wanted.
+"""
+
+from __future__ import annotations
+
+import typing
+
+from bec_lib import bec_logger
+from bec_widgets import BECWidget, SafeSlot
+from bec_widgets.widgets.plots.image.image import Image
+from qtpy.QtCore import QTimer
+from qtpy.QtWidgets import (
+    QCheckBox,
+    QComboBox,
+    QFormLayout,
+    QHBoxLayout,
+    QLabel,
+    QPushButton,
+    QSpinBox,
+    QVBoxLayout,
+    QWidget,
+)
+
+from csaxs_bec.devices.vision.vision_interface import VisionMode
+
+logger = bec_logger.logger
+
+# The exact dotted deviceClass path VisionInterface instances are configured with -- see
+# device_configs/simulated_omny/vision_test.yaml's vision_test_interface entry.
+_VISION_INTERFACE_CLASS = "csaxs_bec.devices.vision.vision_interface.VisionInterface"
+
+# VisionMode is the public Literal type VisionInterface.set_mode()/_VALID_MODES are built from --
+# read the mode list from it rather than importing the private _VALID_MODES tuple directly or
+# duplicating the mode strings here.
+_MODES: tuple[str, ...] = typing.get_args(VisionMode)
+
+_LIVE_POLL_MS = 500
+
+
+class VisionTuning(BECWidget, QWidget):
+    """Pick a ``VisionInterface`` device/camera, tune its processing mode and ROI, watch the
+    live processed preview.
+
+    Every control here is a thin wrapper around an already-existing device RPC method
+    (``set_mode``/``get_mode``, ``set_roi``/``get_roi``, ``camera_names``, ``refresh``) -- this
+    widget does no image processing of its own, and calls those methods no more often than the
+    operator actually changes something (mode/ROI/device/camera state is cached locally after the
+    first read on selection, not re-queried from the device on every refresh tick) -- consistent
+    with ``VisionInterface``'s own ``get_last_image()``/``refresh()`` never triggering a live
+    camera acquisition themselves, only ever returning/republishing whatever the camera last had.
+    """
+
+    PLUGIN = True
+    USER_ACCESS: list[str] = []
+
+    def __init__(self, parent=None, **kwargs):
+        super().__init__(parent=parent, **kwargs)
+        self.get_bec_shortcuts()
+        self._current_camera_name: str | None = None
+        self._live_timer = QTimer(self)
+        self._live_timer.setInterval(_LIVE_POLL_MS)
+        self._live_timer.timeout.connect(self._on_refresh_clicked)
+        self._build_ui()
+        self._populate_devices()
+
+    # -- UI construction --------------------------------------------------------------------
+
+    def _build_ui(self):
+        root = QVBoxLayout(self)
+
+        form = QFormLayout()
+        self.device_combo = QComboBox(parent=self)
+        self.camera_combo = QComboBox(parent=self)
+        self.mode_combo = QComboBox(parent=self)
+        self.mode_combo.addItems(list(_MODES))
+        form.addRow("Device", self.device_combo)
+        form.addRow("Camera", self.camera_combo)
+        form.addRow("Mode", self.mode_combo)
+        root.addLayout(form)
+
+        roi_row = QHBoxLayout()
+        self.roi_x = QSpinBox(parent=self)
+        self.roi_y = QSpinBox(parent=self)
+        self.roi_w = QSpinBox(parent=self)
+        self.roi_h = QSpinBox(parent=self)
+        for spin_box in (self.roi_x, self.roi_y, self.roi_w, self.roi_h):
+            spin_box.setRange(0, 100_000)
+        self.roi_w.setValue(0)
+        self.roi_h.setValue(0)
+        for label_text, spin_box in (
+            ("x", self.roi_x),
+            ("y", self.roi_y),
+            ("w", self.roi_w),
+            ("h", self.roi_h),
+        ):
+            roi_row.addWidget(QLabel(label_text, parent=self))
+            roi_row.addWidget(spin_box)
+        self.apply_roi_button = QPushButton("Apply ROI", parent=self)
+        self.clear_roi_button = QPushButton("Clear ROI", parent=self)
+        roi_row.addWidget(self.apply_roi_button)
+        roi_row.addWidget(self.clear_roi_button)
+        root.addLayout(roi_row)
+
+        control_row = QHBoxLayout()
+        self.refresh_button = QPushButton("Refresh", parent=self)
+        self.live_checkbox = QCheckBox("Live", parent=self)
+        control_row.addWidget(self.refresh_button)
+        control_row.addWidget(self.live_checkbox)
+        control_row.addStretch()
+        root.addLayout(control_row)
+
+        self.image = Image(parent=self)
+        root.addWidget(self.image, stretch=1)
+
+        self.device_combo.currentTextChanged.connect(self._on_device_changed)
+        self.camera_combo.currentTextChanged.connect(self._on_camera_changed)
+        self.mode_combo.currentTextChanged.connect(self._on_mode_changed)
+        self.apply_roi_button.clicked.connect(self._on_apply_roi)
+        self.clear_roi_button.clicked.connect(self._on_clear_roi)
+        self.refresh_button.clicked.connect(self._on_refresh_clicked)
+        self.live_checkbox.toggled.connect(self._on_live_toggled)
+
+    def _populate_devices(self):
+        """List every configured ``VisionInterface`` instance, once, at widget construction --
+        static-at-open, not a live-refreshing list (matches this widget's own "lightweight, few
+        knobs" scope; re-open the widget after a config change instead)."""
+        names = sorted(
+            name
+            for name, device in self.dev.items()
+            if device._config.get("deviceClass") == _VISION_INTERFACE_CLASS
+        )
+        self.device_combo.blockSignals(True)
+        self.device_combo.clear()
+        self.device_combo.addItems(names)
+        self.device_combo.blockSignals(False)
+        if names:
+            self._on_device_changed(names[0])
+
+    def _current_interface(self):
+        name = self.device_combo.currentText()
+        if not name:
+            return None
+        return getattr(self.dev, name, None)
+
+    # -- handlers -----------------------------------------------------------------------------
+
+    @SafeSlot(str)
+    def _on_device_changed(self, device_name: str):
+        interface = self._current_interface()
+        camera_names = interface.camera_names if interface is not None else []
+        self.camera_combo.blockSignals(True)
+        self.camera_combo.clear()
+        self.camera_combo.addItems(camera_names)
+        self.camera_combo.blockSignals(False)
+        if device_name:
+            self.image.image(device=device_name, signal="diagnostic_image")
+        if camera_names:
+            self._on_camera_changed(camera_names[0])
+        else:
+            self._current_camera_name = None
+
+    @SafeSlot(str)
+    def _on_camera_changed(self, camera_name: str):
+        self._current_camera_name = camera_name or None
+        interface = self._current_interface()
+        if interface is None or not camera_name:
+            return
+
+        mode = interface.get_mode()
+        self.mode_combo.blockSignals(True)
+        self.mode_combo.setCurrentText(mode)
+        self.mode_combo.blockSignals(False)
+
+        roi = interface.get_roi(camera_name)
+        for spin_box, value in zip(
+            (self.roi_x, self.roi_y, self.roi_w, self.roi_h), roi if roi is not None else (0, 0, 0, 0)
+        ):
+            spin_box.blockSignals(True)
+            spin_box.setValue(value)
+            spin_box.blockSignals(False)
+
+        self._on_refresh_clicked()
+
+    @SafeSlot(str)
+    def _on_mode_changed(self, mode: str):
+        interface = self._current_interface()
+        if interface is None or not mode:
+            return
+        interface.set_mode(mode)
+        self._on_refresh_clicked()
+
+    @SafeSlot()
+    def _on_apply_roi(self):
+        interface = self._current_interface()
+        if interface is None or not self._current_camera_name:
+            return
+        roi = (self.roi_x.value(), self.roi_y.value(), self.roi_w.value(), self.roi_h.value())
+        interface.set_roi(roi, camera_name=self._current_camera_name)
+        self._on_refresh_clicked()
+
+    @SafeSlot()
+    def _on_clear_roi(self):
+        interface = self._current_interface()
+        if interface is None or not self._current_camera_name:
+            return
+        interface.set_roi(None, camera_name=self._current_camera_name)
+        self._on_refresh_clicked()
+
+    @SafeSlot()
+    def _on_refresh_clicked(self):
+        """Pull+process+publish once. Never triggers a live camera acquisition -- ``refresh()``
+        only republishes whichever frame the camera last had (see ``get_last_image()``'s contract
+        on ``DummyVisionCamera``/``IDSCamera``/``AlliedVisionAravisCamera``), so calling this
+        repeatedly (e.g. from the Live timer below) is just re-publishing a cached read, not
+        polling hardware for a new one."""
+        interface = self._current_interface()
+        if interface is None or not self._current_camera_name:
+            return
+        try:
+            interface.refresh(self._current_camera_name)
+        except Exception:
+            logger.exception("VisionTuning: refresh failed")
+
+    @SafeSlot(bool)
+    def _on_live_toggled(self, checked: bool):
+        if checked:
+            self._live_timer.start()
+        else:
+            self._live_timer.stop()

csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.pyproject

@@ -0,0 +1 @@
+{'files': ['vision_tuning.py']}

csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning_plugin.py

@@ -0,0 +1,57 @@
+# Copyright (C) 2022 The Qt Company Ltd.
+# SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause
+
+from qtpy.QtDesigner import QDesignerCustomWidgetInterface
+from qtpy.QtWidgets import QWidget
+
+from bec_widgets.utils.bec_designer import designer_material_icon
+from csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning import VisionTuning
+
+DOM_XML = """
+<ui language='c++'>
+    <widget class='VisionTuning' name='vision_tuning'>
+    </widget>
+</ui>
+"""
+
+
+class VisionTuningPlugin(QDesignerCustomWidgetInterface):  # pragma: no cover
+    def __init__(self):
+        super().__init__()
+        self._form_editor = None
+
+    def createWidget(self, parent):
+        if parent is None:
+            return QWidget()
+        t = VisionTuning(parent)
+        return t
+
+    def domXml(self):
+        return DOM_XML
+
+    def group(self):
+        return ""
+
+    def icon(self):
+        return designer_material_icon(VisionTuning.ICON_NAME)
+
+    def includeFile(self):
+        return "vision_tuning"
+
+    def initialize(self, form_editor):
+        self._form_editor = form_editor
+
+    def isContainer(self):
+        return False
+
+    def isInitialized(self):
+        return self._form_editor is not None
+
+    def name(self):
+        return "VisionTuning"
+
+    def toolTip(self):
+        return ""
+
+    def whatsThis(self):
+        return self.toolTip()

csaxs_bec/devices/vision/reference_images/README.md

@@ -3,18 +3,24 @@
 This directory is the conventional local drop-in location for real-camera reference-image
 captures used to test the vision pipeline (`AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md`) --
 e.g. OMNY's `_otransfer_save_reference_images()` output
-(`sample_transfer_reference_image_<step>_<timestamp>.h5`, one HDF5 file per confirm gate, one
-dataset per camera in the active pair).
+(`omny_sample_transfer_reference_image_<step>_<timestamp>.h5`, one HDF5 file per confirm gate, one
+dataset per camera in the active pair, keyed by camera name) or flomni's
+`save_reference_image()` output (`sample_transfer_reference_image_<step>_<timestamp>.h5`, one file
+per confirm gate, a single `reference_image` dataset -- flomni only ever shows one camera). Same
+naming convention, different HDF5 layout inside -- check which format a given file actually is
+(`h5py.File(path).keys()`) before assuming one or the other.
 
 **Not committed to git.** This repo's `.gitignore` excludes `**.h5` ("file writer data"), and a
 real capture set is tens of megabytes -- exactly the kind of binary data that rule is for. Only
 this `README.md` is tracked, so the directory (and the convention) exists in git even though its
 actual contents don't.
 
-To populate it locally: drop `.h5` capture files here directly (from a real OMNY session, or
+To populate it locally: drop `.h5` capture files here directly (from a real OMNY/flomni session, or
 transferred another way -- e.g. as a packaged archive from a remote session). See
 `../../bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md` for how to use
-real images (there, `DummyVisionCamera.load_image_file()`) against the vision pipeline; loading
-straight from this HDF5 multi-camera-pair format specifically isn't wired up yet as of this
-writing (`load_image_file()` reads a single plain image file via `cv2.imread`, not this format) --
-check the plan doc's implementation-status section for the current state before assuming otherwise.
+real images against the vision pipeline. `DummyVisionCamera.load_image_file()` only reads a plain
+image file via `cv2.imread` (jpg/png/...), not either HDF5 layout above -- loading straight from
+HDF5 isn't wired up as a one-liner yet, so read the dataset yourself with `h5py` and hand the array
+to `set_image()` instead (see the howto's "Real flomni sample-transfer captures" section for the
+exact steps, including picking a reference vs. current frame, setting an ROI, and visually
+inspecting the edge detection before trusting a score).