From 3241e880cee15fe954e72c4349f5e63ddc6225f2 Mon Sep 17 00:00:00 2001 From: x01dc Date: Sun, 6 Sep 2026 12:57:48 +0200 Subject: [PATCH] docs(omny): design plan for machine-vision auto-confirm on sample transfer Explores classical CV (registration + edge/SSIM/contour matching against multiple reference images) to reduce the manual y/n confirms in otransfer_get_sample()/otransfer_put_sample(), reusing the existing reference-image capture and the push_smear_preview() GUI-overlay pattern already proven for LamNI's rotation-center calibration. Exploration only; implementation to follow on a separate branch. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_018Mupip8x7Hean6E3FFvpbj --- .../MACHINE_VISION_AUTOCONFIRM_PLAN.md | 147 ++++++++++++++++++ .../plugins/omny/AI_docs/OPEN_ISSUES.md | 6 +- 2 files changed, 151 insertions(+), 2 deletions(-) create mode 100644 csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md new file mode 100644 index 00000000..a150b4ff --- /dev/null +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/MACHINE_VISION_AUTOCONFIRM_PLAN.md @@ -0,0 +1,147 @@ +# OMNY sample-transfer machine-vision auto-confirm — design plan (2026-09-06) + +Exploration only — **no code in this pass**. Written on `update-omny` so a future session/branch +can pick this up without re-deriving it. Companion docs: `OPEN_ISSUES.md` (the live-testing effort +this plan grows out of — see its "full gripper sample-transfer chain... untested" item) and +`SAMPLE_STORAGE_TEST_CHECKLIST.md`. + +## Motivation + +Mirko's starting question: is there a Python package for comparing a live gripper/pin camera image +against a set of reference images, robust to exposure changes and small camera-alignment drift, to +decide "is the pin sitting in the gripper correctly" — classic machine-vision, not deep learning. +Applied to BEC: could this let OMNY's sample-transfer flow (`otransfer_get_sample()` / +`otransfer_put_sample()`) auto-approve routine steps instead of stopping for a human `y/n` at every +single one, while still catching real problems (pin damage, missed grip, wrong shuttle state)? + +This doc maps that idea onto OMNY's actual confirm gates and camera/GUI plumbing, and proposes a +staged plan. **Goal for this session: write this plan down. Implementation is a separate, later +branch.** + +## Where today's human confirmations actually live + +All of these are in `omny_sample_transfer_mixin.py` and go through +`otransfer_confirm_dialog()` (line 76 — GUI `ConsoleButtonsWidget` yes/no if the camera console is +already open, else a blocking CLI `yesno()`): + +1. **`_otransfer_confirm()`** (line 848) — the main per-step handshake during an actual mount/get, + driven by the Galil controller's `confirm`/`mntprgs` variables (`XQ#GRPUT`/`XQ#GRGET` macros). + Loops until the controller reports `mntprgs == 0`; each iteration is one manual "All OK?" + gate. For a *get*, the first iteration additionally hard-checks the top-pin position against a + fixed threshold (`-25.4`, line 866) and asks an extra, more alarming confirm if it's exceeded — + this is a **physical safety check**, not a generic step confirm, and should stay independent of + any vision system. +2. **`otransfer_get_sample()`** (line 934) — "confirm there is no sample in the gripper, it would + be dropped" before starting at all. +3. **`otransfer_get_sample()`** (line 971) / **`otransfer_put_sample()`** (line 812) — "Ready to + start the [un]mount process now?" right before triggering `XQ#GRGET`/`XQ#GRPUT`. +4. **`otransfer_put_sample()`** (line 836) — "Pick another sample from this shuttle? Keep shield + open?" after a successful put. +5. **`_otransfer_shuttle_align()`** (line 189) — one-off "shall I try to move the aligner to the + correct start position" recovery prompt, only on a specific bad-limit-switch state. Edge case, + not a routine per-transfer gate — out of scope for auto-confirm. +6. **`otransfer_park_loadlock_slot()`** (line 523) — CLI-only `yesno()` by design (operator is + physically at the loadlock, not watching the GUI camera console) — also out of scope here. + +Gates 1–4 are the routine, repeated-every-transfer confirmations this plan targets. Every one of +them already calls **`_otransfer_save_reference_images()`** (line 885, landed in `a7429b2`) right +at the confirm point, which dumps the currently-active camera pair's `get_last_image()` frames to +an HDF5 file under `~/data/raw/logs/omny_sample_transfer_images/`, named by step + timestamp. **This +is already the reference-image corpus this plan would compare against** — it exists purely as a +post-hoc audit log today, with no comparison logic reading it back. + +## Existing infrastructure to build on (no new plumbing needed for a prototype) + +- **`IDSCamera.get_last_image()`** (`csaxs_bec/devices/ids_cameras/ids_camera.py:234`) — already + how `_otransfer_save_reference_images()` gets frames; same call works for a live comparison. +- **`OMNYGuiTools._omnycam_active_pair`** (`gui_tools.py:88-105`) — tracks which camera pair + (`cam200`+`cam203` parking/shuttle, or `cam201`+`cam202` sample stage) is currently shown, exactly + the info needed to pick the matching reference subset. +- **`IDSCamera.push_preview_image()` / `push_smear_preview()`** (`ids_camera.py:240,269`) — a + device-server-side hook to publish a processed/composite array back through the same + `device_preview` channel the GUI already subscribes to, without disturbing the live acquisition + feed. **This exact mechanism is already proven for a similar purpose**: LamNI's rotation-center + calibration (`LamNI/x_ray_eye_align.py:737-797`, see `[[project_lamni_rotation_center_calibration]]`-era + work) pushes a growing composite image via `push_smear_preview()`, and + `bec_widgets/widgets/xray_eye/x_ray_eye.py` (`smear_preview_toggle`, lines 386-873) already has a + working UI toggle to switch an `Image` dock between the live feed and this diagnostic channel. + **A vision-diff overlay for OMNY sample transfer should reuse this pattern** — push an + aligned-reference/diff/contour-overlay composite via `push_smear_preview()` on whichever of + cam200/201/202/203 is active, and add the same toggle affordance to OMNY's camera docks + (`gui_tools.py:127-152`) — rather than inventing a third dock or a new preview channel. +- **`opencv-python`** is already a project dependency (`pyproject.toml:27`); `scikit-image` is not + yet (needed only if SSIM specifically is wanted — see below). + +## Proposed CV pipeline (classical, no deep learning) + +Per the original discussion this grew out of: classical machine vision is enough for a +well-constrained "is the pin geometry right" question; no training data, no model to keep in sync +with hardware changes. + +1. **Registration first.** Camera mounts can drift slightly, so raw pixel comparison isn't + meaningful until the current frame is aligned to the reference. `cv2.findTransformECC` for + small translation/rotation (the expected case — a bolted-down camera); an ORB/SIFT + + `cv2.findHomography` fallback if perspective drift turns out to be larger than ECC handles. +2. **Exposure-robust representation.** Compare edges/gradients (Sobel/Canny) or CLAHE-normalized + images, not raw intensity — illumination changes should not look like a defect. +3. **Similarity/anomaly scoring**, two complementary signals: + - **Contour/silhouette match** (`cv2.findContours` + `cv2.matchShapes`) on the extracted + pin/gripper silhouette — the most directly relevant signal for "is the pin sitting right", + and close to fully lighting-invariant. + - **SSIM** (`skimage.metrics.structural_similarity`) on the registered, normalized pair as a + general structural-anomaly fallback, in case a defect doesn't show up as a clean contour + change. (Pulls in `scikit-image` as a new dependency — only if this signal proves worth its + weight; contour matching alone may be sufficient given how constrained the geometry is.) +4. **Multi-reference scoring.** Score against *every* reference image for the given + `(pin_position, step_name, camera)` combination and take the best (minimum-distance) match, not + a single fixed reference — natural sample-to-sample and lighting variation shouldn't inflate the + anomaly score just because one particular reference happened to be a bad match. +5. **Escalate only if needed.** If classical contour/SSIM scoring proves insufficiently robust to + subtle real defects, `anomalib` (e.g. PatchCore) is the documented fallback — but per the + original assessment this is very likely overkill for a fixed pin/gripper geometry, and shouldn't + be built preemptively. + +## What "getting rid of human confirmations" should actually mean — staged rollout + +Physical hardware, real samples, real damage risk (see the top-pin threshold in gate 1 above) means +this should not be an all-or-nothing flag flip. Proposed phases for the later implementation branch: + +- **Phase 0 — instrument, don't automate.** Compute and log the vision score at every existing gate + (1–4 above); still always show the human the same confirm prompt, but now alongside the score and + a diff/overlay visualization (via the `push_smear_preview()` pattern above). Purpose: build a + labeled dataset of real operator pass/fail decisions vs. score, across real lighting and sample + variety, before trusting any threshold. No behavior change to the transfer flow itself. +- **Phase 1 — assist, never silently reject.** A comfortably-high score can auto-answer "yes" at a + gate (with the decision logged and visibly flagged as auto-confirmed, and an easy override); a + low score never fails silently or auto-aborts — it escalates to the same confirm prompt, now with + the diff visualization made prominent, so the human's attention goes exactly where the anomaly + is. This mirrors how the existing top-pin-position check already works (a hard numeric threshold + that *escalates* the confirm text, never silently proceeds or silently aborts). +- **Phase 2 — skip the prompt on high-confidence routine steps.** Only once Phase 0/1 data shows + the scoring is reliable across real operational conditions, let specific gates (candidates: gate + 2's "no sample in gripper" check, gate 4's "pick another sample" prompt — the more purely + informational ones) skip the prompt outright on a high score, behind an explicit opt-in/kill + switch. The physical safety checks (top-pin threshold, `_otransfer_check_sensor_connected()`) + stay hard-coded and independent of the vision system regardless of phase — vision augments the + judgment calls, it doesn't replace the interlocks. + +## Open questions / decisions needed before implementation starts + +- **Reference corpus curation.** Today's saved images are a timestamped append-only run log, not a + curated "golden" set — every run adds a new file, nothing marks one as trustworthy for + comparison. Need either a promotion step (operator marks a saved image as a reference) or a + separate deliberately-curated directory, distinct from the audit-log files + `_otransfer_save_reference_images()` already writes. +- **Per-camera-pair segmentation.** Parking/shuttle view (cam200+cam203) and sample-stage view + (cam201+cam202) have different backgrounds/lighting; silhouette extraction likely needs tuning + per pair, not one global set of CV parameters. +- **Threshold calibration.** Can't be picked a priori — needs the Phase 0 logged-score dataset + across real sessions before any auto-confirm threshold is chosen. +- **Where scores/results live.** Likely alongside the existing HDF5 reference-image files (same + per-step file, additional dataset for score + registration transform), rather than a new storage + mechanism. + +## Non-goals for this session + +No dependencies added, no code written, no new GUI elements built. This document is the handoff for +the follow-up branch. diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/OPEN_ISSUES.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/OPEN_ISSUES.md index 802c382d..ad400426 100644 --- a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/OPEN_ISSUES.md +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/OPEN_ISSUES.md @@ -3,9 +3,11 @@ Tracks the live-testing effort for bringing OMNY up to flomni's/BEC's current standards on branch `update-omny`: bugs found (fixed and unfixed) and what's still untested. Written so a future session can pick this up without re-deriving it. Companion docs: -`../../flomni/AI_docs/SIMULATED_ENDSTATIONS.md` (the sim architecture) and +`../../flomni/AI_docs/SIMULATED_ENDSTATIONS.md` (the sim architecture), `../../flomni/AI_docs/FLOMNI_TO_LAMNI_COMPARISON.md` (the flomni→lamni migration this OMNY work -is meant to mirror). +is meant to mirror), and `MACHINE_VISION_AUTOCONFIRM_PLAN.md` (design-only exploration of using +classical machine vision on the sample-transfer camera feeds to reduce/replace the manual confirm +gates in `otransfer_get_sample()`/`otransfer_put_sample()` — not started, separate future branch). ## Current status