From c9626d6007a0409d86eabeda66ae5754c1bfebc8 Mon Sep 17 00:00:00 2001 From: Mirko Holler Date: Mon, 7 Sep 2026 05:31:21 -0300 Subject: [PATCH] feat(vision): add lightweight VisionTuning GUI widget, extend testing howto VisionTuning (csaxs_bec/bec_widgets/widgets/vision_tuning/) is a small BECWidget for tuning a VisionInterface interactively: pick a device/camera, flip through raw/grayscale/edges/clahe, set/clear an ROI, and watch the live diagnostic_image preview -- modeled on z_ConsoleButtonsWidget's "few knobs" simplicity rather than the full OMNY_XRayEye alignment GUI. Registered via `bw-generate-cli --target csaxs_bec` (regenerates client.py/designer_plugins.py, the actual mechanism gui.test.new("VisionTuning", ...) resolves against). VISION_PIPELINE_TESTING_HOWTO.md gains a section on testing against the real flomni sample-transfer h5 captures now in reference_images/: loading a capture's single `reference_image` dataset (via set_image(), since load_image_file() only reads cv2-loadable files), picking a reference vs. current frame by hand, setting an ROI, and visually checking the edge/contour detection before trusting a score -- deliberately a manual workflow, not an automated score sweep, since same-named capture steps (get/put) can come from different physical scenes. reference_images/README.md corrected to document both OMNY's per-camera-pair h5 layout and flomni's single-dataset layout, matching what's actually in the directory. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01FT8PkvfdQUBKc5X4pHrxB6 --- .../AI_docs/VISION_PIPELINE_TESTING_HOWTO.md | 83 ++++++ csaxs_bec/bec_widgets/widgets/client.py | 20 +- .../bec_widgets/widgets/designer_plugins.py | 2 + .../widgets/vision_tuning/__init__.py | 3 + .../vision_tuning/register_vision_tuning.py | 15 ++ .../widgets/vision_tuning/vision_tuning.py | 236 ++++++++++++++++++ .../vision_tuning/vision_tuning.pyproject | 1 + .../vision_tuning/vision_tuning_plugin.py | 57 +++++ .../devices/vision/reference_images/README.md | 20 +- 9 files changed, 421 insertions(+), 16 deletions(-) create mode 100644 csaxs_bec/bec_widgets/widgets/vision_tuning/__init__.py create mode 100644 csaxs_bec/bec_widgets/widgets/vision_tuning/register_vision_tuning.py create mode 100644 csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.py create mode 100644 csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.pyproject create mode 100644 csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning_plugin.py diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md index 05ad2c62..75bb5ec9 100644 --- a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md @@ -188,6 +188,89 @@ a human would notice. Useful for sanity-checking `edge_low_threshold`/`edge_high (`vision_toolkit.detect_edges()`'s defaults, `50`/`150`) against real lighting/background, which the synthetic square can't tell you anything about. +## Real flomni sample-transfer captures: reference vs. current, ROI, and visual edge-detection + +`csaxs_bec/devices/vision/reference_images/` (gitignored, see its own `README.md`) is where a local +set of real captures from flomni's `save_reference_image()` can be dropped for exactly this kind of +manual testing -- one HDF5 file per confirm gate, e.g. +`sample_transfer_reference_image_get_1_20260701_102105.h5`. Unlike OMNY's own format (one file per +camera *pair*, keyed by camera name -- see `_otransfer_save_reference_images()`'s docstring), a +flomni capture is one file with a single `reference_image` dataset: + +```python +import h5py + +with h5py.File("csaxs_bec/devices/vision/reference_images/sample_transfer_reference_image_get_1_20260701_102105.h5", "r") as f: + image = f["reference_image"][()] # RGB ndarray, e.g. (1920, 1080, 3) uint8 +``` + +`load_image_file()` can't read this (`cv2.imread`-only, no HDF5 support) -- load the array yourself +as above and hand it to `DummyVisionCamera` directly with `set_image()`, which takes an ndarray from +anywhere: + +```python +dev.vision_cam_a.set_image(image) +``` + +**This step needs a person, not a score sweep.** A filename like `get_1`/`get_2`/`get_3` only means +"captured at the same confirm-gate step name" -- it does *not* mean "same physical scene". OMNY's +`get` gate fires both when picking a pin up from the sample stage and when picking one up from +storage, and storage has multiple slots -- two `get`-named captures can legitimately show completely +different backgrounds. Don't assume same-named files form a matched set; look at them (or diff their +overlays) before deciding which one is a valid "reference" for which "current" frame. This is a +judgment call about what the two images actually show, not something a blind pairwise score sweep +over all combinations can decide correctly on its own. + +The manual loop, in a running BEC session: + +```python +# 1. Pick one capture as the reference, load and save it under a tag. +with h5py.File(".../sample_transfer_reference_image_get_1_....h5", "r") as f: + dev.vision_cam_a.set_image(f["reference_image"][()]) +dev.vision_test_monitor.save_reference("manual_check") + +# 2. Load a different capture as the current frame and score it. +with h5py.File(".../sample_transfer_reference_image_get_2_....h5", "r") as f: + dev.vision_cam_a.set_image(f["reference_image"][()]) +dev.vision_test_monitor.score("manual_check") +``` + +**Define an ROI before trusting the score.** These are real photos of a cluttered, reflective metal +background -- `vision_toolkit.largest_contour()` picks the single largest edge blob in the *whole* +frame, which on a busy background can lock onto a glare boundary or a scratch instead of the actual +gripper/pin. Crop to the region that actually contains the object before scoring, and check the crop +visually first (see below) rather than guessing coordinates blind: + +```python +dev.vision_test_interface.set_roi((300, 700, 600, 550), camera_name="vision_cam_a") # x, y, w, h +dev.vision_test_monitor.score("manual_check") # re-score with the ROI applied +dev.vision_test_interface.set_roi(None, camera_name="vision_cam_a") # clear it again +``` + +**Look at the edge detection and contour, don't just trust the number.** The fastest loop is calling +the Layer 0 toolkit functions directly on the loaded array (no BEC round-trip needed) and writing +the result to a file to view: + +```python +from csaxs_bec.devices.vision import vision_toolkit as vt +import cv2 + +crop = vt.crop_to_roi(image, (300, 700, 600, 550)) # same ROI as above, or none +edges = vt.detect_edges(crop, low_threshold=50, high_threshold=150) +contour = vt.largest_contour(edges) +overlay = vt.draw_contour_overlay(crop, contour) # contour drawn in red on the crop + +cv2.imwrite("/tmp/check_crop.png", cv2.cvtColor(crop, cv2.COLOR_RGB2BGR)) +cv2.imwrite("/tmp/check_edges.png", edges) +cv2.imwrite("/tmp/check_overlay.png", cv2.cvtColor(overlay, cv2.COLOR_RGB2BGR)) +``` + +Open `check_overlay.png` and confirm the red contour actually traces the gripper/pin, not something +else in the background, before trusting any score computed from it -- if it doesn't, adjust the ROI +and/or `edge_low_threshold`/`edge_high_threshold` and try again. This is also the way to see the +`diagnostic_image` overlay's equivalent without needing the GUI dock that doesn't exist yet (see +below). + ## What this does and doesn't prove Proves: the Layer 0/1/2 code paths work correctly end to end (registration, contour scoring, diff --git a/csaxs_bec/bec_widgets/widgets/client.py b/csaxs_bec/bec_widgets/widgets/client.py index abe03e3b..44e2f459 100644 --- a/csaxs_bec/bec_widgets/widgets/client.py +++ b/csaxs_bec/bec_widgets/widgets/client.py @@ -19,6 +19,7 @@ _Widgets = { "OMNY_XRayEye": "OMNY_XRayEye", "SAXSWidget": "SAXSWidget", "SlitControl": "SlitControl", + "VisionTuning": "VisionTuning", "z_ConsoleButtonsWidget": "z_ConsoleButtonsWidget", } @@ -48,21 +49,16 @@ class DataViewer(RPCBase): class OMNY_SampleStorage(RPCBase): - """View and correct the FlOMNI sample-storage records.""" + """View and correct the sample-storage records for whichever setup (FlOMNI""" _IMPORT_MODULE = "csaxs_bec.bec_widgets.widgets.sample_storage.sample_storage" @rpc_call def refresh(self) -> "None": """ - Re-read all slots (one bulk device round-trip) and update only the - cells whose state actually changed. - - Repainting every cell on every 2 s poll — even when nothing changed, - which is almost always the case for a sample magazine — was needless - work on the GUI thread. We diff against the last-seen state and only - call set_state() on cells that differ, so a steady-state poll does no - UI work at all. + Re-read all slots (one bulk device round-trip) and update only + the cells whose state actually changed. Dispatches to the active + setup's own refresh method. """ @@ -431,6 +427,12 @@ class SlitControl(RPCBase): """ +class VisionTuning(RPCBase): + """Pick a ``VisionInterface`` device/camera, tune its processing mode and ROI, watch the""" + + _IMPORT_MODULE = "csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning" + + class XRayEye2DControl(RPCBase): _IMPORT_MODULE = "csaxs_bec.bec_widgets.widgets.xray_eye.x_ray_eye" diff --git a/csaxs_bec/bec_widgets/widgets/designer_plugins.py b/csaxs_bec/bec_widgets/widgets/designer_plugins.py index ad5b6c7c..9c9ca1ac 100644 --- a/csaxs_bec/bec_widgets/widgets/designer_plugins.py +++ b/csaxs_bec/bec_widgets/widgets/designer_plugins.py @@ -14,6 +14,7 @@ designer_plugins = { "OMNY_XRayEye": ("csaxs_bec.bec_widgets.widgets.xray_eye.x_ray_eye", "OMNY_XRayEye"), "SAXSWidget": ("csaxs_bec.bec_widgets.widgets.saxs_widget.saxs_widget", "SAXSWidget"), "SlitControl": ("csaxs_bec.bec_widgets.widgets.slit_control.slit_control", "SlitControl"), + "VisionTuning": ("csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning", "VisionTuning"), "z_ConsoleButtonsWidget": ( "csaxs_bec.bec_widgets.widgets.console_buttons.console_buttons", "z_ConsoleButtonsWidget", @@ -27,5 +28,6 @@ widget_icons = { "OMNY_XRayEye": "widgets", "SAXSWidget": "table_chart", "SlitControl": "widgets", + "VisionTuning": "widgets", "z_ConsoleButtonsWidget": "widgets", } diff --git a/csaxs_bec/bec_widgets/widgets/vision_tuning/__init__.py b/csaxs_bec/bec_widgets/widgets/vision_tuning/__init__.py new file mode 100644 index 00000000..4cddc7ce --- /dev/null +++ b/csaxs_bec/bec_widgets/widgets/vision_tuning/__init__.py @@ -0,0 +1,3 @@ +from csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning import VisionTuning + +__all__ = ["VisionTuning"] diff --git a/csaxs_bec/bec_widgets/widgets/vision_tuning/register_vision_tuning.py b/csaxs_bec/bec_widgets/widgets/vision_tuning/register_vision_tuning.py new file mode 100644 index 00000000..18859fdd --- /dev/null +++ b/csaxs_bec/bec_widgets/widgets/vision_tuning/register_vision_tuning.py @@ -0,0 +1,15 @@ +def main(): # pragma: no cover + from qtpy import PYSIDE6 + + if not PYSIDE6: + print("PYSIDE6 is not available in the environment. Cannot patch designer.") + return + from PySide6.QtDesigner import QPyDesignerCustomWidgetCollection + + from csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning_plugin import VisionTuningPlugin + + QPyDesignerCustomWidgetCollection.addCustomWidget(VisionTuningPlugin()) + + +if __name__ == "__main__": # pragma: no cover + main() diff --git a/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.py b/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.py new file mode 100644 index 00000000..11dbe76a --- /dev/null +++ b/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.py @@ -0,0 +1,236 @@ +"""A lightweight tuning widget for :class:`~csaxs_bec.devices.vision.vision_interface.VisionInterface` +(Layer 1 of ``AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md``). + +Deliberately small -- modeled on ``z_ConsoleButtonsWidget``'s "yesno" simplicity, not on the full +``OMNY_XRayEye`` alignment GUI: pick a ``VisionInterface`` device and one of its cameras, flip +through processing modes, set/clear an ROI, and watch the live ``diagnostic_image`` result. No +``SampleVisionMonitor`` (scoring) controls here -- that is a different device/concern and would +roughly double the knob count; a follow-up panel can add it later if wanted. +""" + +from __future__ import annotations + +import typing + +from bec_lib import bec_logger +from bec_widgets import BECWidget, SafeSlot +from bec_widgets.widgets.plots.image.image import Image +from qtpy.QtCore import QTimer +from qtpy.QtWidgets import ( + QCheckBox, + QComboBox, + QFormLayout, + QHBoxLayout, + QLabel, + QPushButton, + QSpinBox, + QVBoxLayout, + QWidget, +) + +from csaxs_bec.devices.vision.vision_interface import VisionMode + +logger = bec_logger.logger + +# The exact dotted deviceClass path VisionInterface instances are configured with -- see +# device_configs/simulated_omny/vision_test.yaml's vision_test_interface entry. +_VISION_INTERFACE_CLASS = "csaxs_bec.devices.vision.vision_interface.VisionInterface" + +# VisionMode is the public Literal type VisionInterface.set_mode()/_VALID_MODES are built from -- +# read the mode list from it rather than importing the private _VALID_MODES tuple directly or +# duplicating the mode strings here. +_MODES: tuple[str, ...] = typing.get_args(VisionMode) + +_LIVE_POLL_MS = 500 + + +class VisionTuning(BECWidget, QWidget): + """Pick a ``VisionInterface`` device/camera, tune its processing mode and ROI, watch the + live processed preview. + + Every control here is a thin wrapper around an already-existing device RPC method + (``set_mode``/``get_mode``, ``set_roi``/``get_roi``, ``camera_names``, ``refresh``) -- this + widget does no image processing of its own, and calls those methods no more often than the + operator actually changes something (mode/ROI/device/camera state is cached locally after the + first read on selection, not re-queried from the device on every refresh tick) -- consistent + with ``VisionInterface``'s own ``get_last_image()``/``refresh()`` never triggering a live + camera acquisition themselves, only ever returning/republishing whatever the camera last had. + """ + + PLUGIN = True + USER_ACCESS: list[str] = [] + + def __init__(self, parent=None, **kwargs): + super().__init__(parent=parent, **kwargs) + self.get_bec_shortcuts() + self._current_camera_name: str | None = None + self._live_timer = QTimer(self) + self._live_timer.setInterval(_LIVE_POLL_MS) + self._live_timer.timeout.connect(self._on_refresh_clicked) + self._build_ui() + self._populate_devices() + + # -- UI construction -------------------------------------------------------------------- + + def _build_ui(self): + root = QVBoxLayout(self) + + form = QFormLayout() + self.device_combo = QComboBox(parent=self) + self.camera_combo = QComboBox(parent=self) + self.mode_combo = QComboBox(parent=self) + self.mode_combo.addItems(list(_MODES)) + form.addRow("Device", self.device_combo) + form.addRow("Camera", self.camera_combo) + form.addRow("Mode", self.mode_combo) + root.addLayout(form) + + roi_row = QHBoxLayout() + self.roi_x = QSpinBox(parent=self) + self.roi_y = QSpinBox(parent=self) + self.roi_w = QSpinBox(parent=self) + self.roi_h = QSpinBox(parent=self) + for spin_box in (self.roi_x, self.roi_y, self.roi_w, self.roi_h): + spin_box.setRange(0, 100_000) + self.roi_w.setValue(0) + self.roi_h.setValue(0) + for label_text, spin_box in ( + ("x", self.roi_x), + ("y", self.roi_y), + ("w", self.roi_w), + ("h", self.roi_h), + ): + roi_row.addWidget(QLabel(label_text, parent=self)) + roi_row.addWidget(spin_box) + self.apply_roi_button = QPushButton("Apply ROI", parent=self) + self.clear_roi_button = QPushButton("Clear ROI", parent=self) + roi_row.addWidget(self.apply_roi_button) + roi_row.addWidget(self.clear_roi_button) + root.addLayout(roi_row) + + control_row = QHBoxLayout() + self.refresh_button = QPushButton("Refresh", parent=self) + self.live_checkbox = QCheckBox("Live", parent=self) + control_row.addWidget(self.refresh_button) + control_row.addWidget(self.live_checkbox) + control_row.addStretch() + root.addLayout(control_row) + + self.image = Image(parent=self) + root.addWidget(self.image, stretch=1) + + self.device_combo.currentTextChanged.connect(self._on_device_changed) + self.camera_combo.currentTextChanged.connect(self._on_camera_changed) + self.mode_combo.currentTextChanged.connect(self._on_mode_changed) + self.apply_roi_button.clicked.connect(self._on_apply_roi) + self.clear_roi_button.clicked.connect(self._on_clear_roi) + self.refresh_button.clicked.connect(self._on_refresh_clicked) + self.live_checkbox.toggled.connect(self._on_live_toggled) + + def _populate_devices(self): + """List every configured ``VisionInterface`` instance, once, at widget construction -- + static-at-open, not a live-refreshing list (matches this widget's own "lightweight, few + knobs" scope; re-open the widget after a config change instead).""" + names = sorted( + name + for name, device in self.dev.items() + if device._config.get("deviceClass") == _VISION_INTERFACE_CLASS + ) + self.device_combo.blockSignals(True) + self.device_combo.clear() + self.device_combo.addItems(names) + self.device_combo.blockSignals(False) + if names: + self._on_device_changed(names[0]) + + def _current_interface(self): + name = self.device_combo.currentText() + if not name: + return None + return getattr(self.dev, name, None) + + # -- handlers ----------------------------------------------------------------------------- + + @SafeSlot(str) + def _on_device_changed(self, device_name: str): + interface = self._current_interface() + camera_names = interface.camera_names if interface is not None else [] + self.camera_combo.blockSignals(True) + self.camera_combo.clear() + self.camera_combo.addItems(camera_names) + self.camera_combo.blockSignals(False) + if device_name: + self.image.image(device=device_name, signal="diagnostic_image") + if camera_names: + self._on_camera_changed(camera_names[0]) + else: + self._current_camera_name = None + + @SafeSlot(str) + def _on_camera_changed(self, camera_name: str): + self._current_camera_name = camera_name or None + interface = self._current_interface() + if interface is None or not camera_name: + return + + mode = interface.get_mode() + self.mode_combo.blockSignals(True) + self.mode_combo.setCurrentText(mode) + self.mode_combo.blockSignals(False) + + roi = interface.get_roi(camera_name) + for spin_box, value in zip( + (self.roi_x, self.roi_y, self.roi_w, self.roi_h), roi if roi is not None else (0, 0, 0, 0) + ): + spin_box.blockSignals(True) + spin_box.setValue(value) + spin_box.blockSignals(False) + + self._on_refresh_clicked() + + @SafeSlot(str) + def _on_mode_changed(self, mode: str): + interface = self._current_interface() + if interface is None or not mode: + return + interface.set_mode(mode) + self._on_refresh_clicked() + + @SafeSlot() + def _on_apply_roi(self): + interface = self._current_interface() + if interface is None or not self._current_camera_name: + return + roi = (self.roi_x.value(), self.roi_y.value(), self.roi_w.value(), self.roi_h.value()) + interface.set_roi(roi, camera_name=self._current_camera_name) + self._on_refresh_clicked() + + @SafeSlot() + def _on_clear_roi(self): + interface = self._current_interface() + if interface is None or not self._current_camera_name: + return + interface.set_roi(None, camera_name=self._current_camera_name) + self._on_refresh_clicked() + + @SafeSlot() + def _on_refresh_clicked(self): + """Pull+process+publish once. Never triggers a live camera acquisition -- ``refresh()`` + only republishes whichever frame the camera last had (see ``get_last_image()``'s contract + on ``DummyVisionCamera``/``IDSCamera``/``AlliedVisionAravisCamera``), so calling this + repeatedly (e.g. from the Live timer below) is just re-publishing a cached read, not + polling hardware for a new one.""" + interface = self._current_interface() + if interface is None or not self._current_camera_name: + return + try: + interface.refresh(self._current_camera_name) + except Exception: + logger.exception("VisionTuning: refresh failed") + + @SafeSlot(bool) + def _on_live_toggled(self, checked: bool): + if checked: + self._live_timer.start() + else: + self._live_timer.stop() diff --git a/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.pyproject b/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.pyproject new file mode 100644 index 00000000..ba09e272 --- /dev/null +++ b/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning.pyproject @@ -0,0 +1 @@ +{'files': ['vision_tuning.py']} \ No newline at end of file diff --git a/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning_plugin.py b/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning_plugin.py new file mode 100644 index 00000000..cffa63ef --- /dev/null +++ b/csaxs_bec/bec_widgets/widgets/vision_tuning/vision_tuning_plugin.py @@ -0,0 +1,57 @@ +# Copyright (C) 2022 The Qt Company Ltd. +# SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause + +from qtpy.QtDesigner import QDesignerCustomWidgetInterface +from qtpy.QtWidgets import QWidget + +from bec_widgets.utils.bec_designer import designer_material_icon +from csaxs_bec.bec_widgets.widgets.vision_tuning.vision_tuning import VisionTuning + +DOM_XML = """ + + + + +""" + + +class VisionTuningPlugin(QDesignerCustomWidgetInterface): # pragma: no cover + def __init__(self): + super().__init__() + self._form_editor = None + + def createWidget(self, parent): + if parent is None: + return QWidget() + t = VisionTuning(parent) + return t + + def domXml(self): + return DOM_XML + + def group(self): + return "" + + def icon(self): + return designer_material_icon(VisionTuning.ICON_NAME) + + def includeFile(self): + return "vision_tuning" + + def initialize(self, form_editor): + self._form_editor = form_editor + + def isContainer(self): + return False + + def isInitialized(self): + return self._form_editor is not None + + def name(self): + return "VisionTuning" + + def toolTip(self): + return "" + + def whatsThis(self): + return self.toolTip() diff --git a/csaxs_bec/devices/vision/reference_images/README.md b/csaxs_bec/devices/vision/reference_images/README.md index 21f2fd4b..e7752c8e 100644 --- a/csaxs_bec/devices/vision/reference_images/README.md +++ b/csaxs_bec/devices/vision/reference_images/README.md @@ -3,18 +3,24 @@ This directory is the conventional local drop-in location for real-camera reference-image captures used to test the vision pipeline (`AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md`) -- e.g. OMNY's `_otransfer_save_reference_images()` output -(`sample_transfer_reference_image__.h5`, one HDF5 file per confirm gate, one -dataset per camera in the active pair). +(`omny_sample_transfer_reference_image__.h5`, one HDF5 file per confirm gate, one +dataset per camera in the active pair, keyed by camera name) or flomni's +`save_reference_image()` output (`sample_transfer_reference_image__.h5`, one file +per confirm gate, a single `reference_image` dataset -- flomni only ever shows one camera). Same +naming convention, different HDF5 layout inside -- check which format a given file actually is +(`h5py.File(path).keys()`) before assuming one or the other. **Not committed to git.** This repo's `.gitignore` excludes `**.h5` ("file writer data"), and a real capture set is tens of megabytes -- exactly the kind of binary data that rule is for. Only this `README.md` is tracked, so the directory (and the convention) exists in git even though its actual contents don't. -To populate it locally: drop `.h5` capture files here directly (from a real OMNY session, or +To populate it locally: drop `.h5` capture files here directly (from a real OMNY/flomni session, or transferred another way -- e.g. as a packaged archive from a remote session). See `../../bec_ipython_client/plugins/omny/AI_docs/VISION_PIPELINE_TESTING_HOWTO.md` for how to use -real images (there, `DummyVisionCamera.load_image_file()`) against the vision pipeline; loading -straight from this HDF5 multi-camera-pair format specifically isn't wired up yet as of this -writing (`load_image_file()` reads a single plain image file via `cv2.imread`, not this format) -- -check the plan doc's implementation-status section for the current state before assuming otherwise. +real images against the vision pipeline. `DummyVisionCamera.load_image_file()` only reads a plain +image file via `cv2.imread` (jpg/png/...), not either HDF5 layout above -- loading straight from +HDF5 isn't wired up as a one-liner yet, so read the dataset yourself with `h5py` and hand the array +to `set_image()` instead (see the howto's "Real flomni sample-transfer captures" section for the +exact steps, including picking a reference vs. current frame, setting an ROI, and visually +inspecting the edge detection before trusting a score).