docs(omny): design plan for machine-vision auto-confirm on sample transfer
CI for csaxs_bec / test (push) Successful in 1m58s

Explores classical CV (registration + edge/SSIM/contour matching against
multiple reference images) to reduce the manual y/n confirms in
otransfer_get_sample()/otransfer_put_sample(), reusing the existing
reference-image capture and the push_smear_preview() GUI-overlay pattern
already proven for LamNI's rotation-center calibration. Exploration only;
implementation to follow on a separate branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Mupip8x7Hean6E3FFvpbj
This commit is contained in:
x01dcandClaude Sonnet 5 committed 2026-09-06 12:57:48 +02:00
1 parent 415d55f599
commit 3241e880ce
2 files changed
+151 -2

No files matched your search

@@ -0,0 +1,147 @@
# OMNY sample-transfer machine-vision auto-confirm — design plan (2026-09-06)
Exploration only — **no code in this pass**. Written on `update-omny` so a future session/branch
can pick this up without re-deriving it. Companion docs: `OPEN_ISSUES.md` (the live-testing effort
this plan grows out of — see its "full gripper sample-transfer chain... untested" item) and
`SAMPLE_STORAGE_TEST_CHECKLIST.md`.
## Motivation
Mirko's starting question: is there a Python package for comparing a live gripper/pin camera image
against a set of reference images, robust to exposure changes and small camera-alignment drift, to
decide "is the pin sitting in the gripper correctly" — classic machine-vision, not deep learning.
Applied to BEC: could this let OMNY's sample-transfer flow (`otransfer_get_sample()` /
`otransfer_put_sample()`) auto-approve routine steps instead of stopping for a human `y/n` at every
single one, while still catching real problems (pin damage, missed grip, wrong shuttle state)?
This doc maps that idea onto OMNY's actual confirm gates and camera/GUI plumbing, and proposes a
staged plan. **Goal for this session: write this plan down. Implementation is a separate, later
branch.**
## Where today's human confirmations actually live
All of these are in `omny_sample_transfer_mixin.py` and go through
`otransfer_confirm_dialog()` (line 76 — GUI `ConsoleButtonsWidget` yes/no if the camera console is
already open, else a blocking CLI `yesno()`):
1. **`_otransfer_confirm()`** (line 848) — the main per-step handshake during an actual mount/get,
driven by the Galil controller's `confirm`/`mntprgs` variables (`XQ#GRPUT`/`XQ#GRGET` macros).
Loops until the controller reports `mntprgs == 0`; each iteration is one manual "All OK?"
gate. For a *get*, the first iteration additionally hard-checks the top-pin position against a
fixed threshold (`-25.4`, line 866) and asks an extra, more alarming confirm if it's exceeded —
this is a **physical safety check**, not a generic step confirm, and should stay independent of
any vision system.
2. **`otransfer_get_sample()`** (line 934) — "confirm there is no sample in the gripper, it would
be dropped" before starting at all.
3. **`otransfer_get_sample()`** (line 971) / **`otransfer_put_sample()`** (line 812) — "Ready to
start the [un]mount process now?" right before triggering `XQ#GRGET`/`XQ#GRPUT`.
4. **`otransfer_put_sample()`** (line 836) — "Pick another sample from this shuttle? Keep shield
open?" after a successful put.
5. **`_otransfer_shuttle_align()`** (line 189) — one-off "shall I try to move the aligner to the
correct start position" recovery prompt, only on a specific bad-limit-switch state. Edge case,
not a routine per-transfer gate — out of scope for auto-confirm.
6. **`otransfer_park_loadlock_slot()`** (line 523) — CLI-only `yesno()` by design (operator is
physically at the loadlock, not watching the GUI camera console) — also out of scope here.
Gates 1–4 are the routine, repeated-every-transfer confirmations this plan targets. Every one of
them already calls **`_otransfer_save_reference_images()`** (line 885, landed in `a7429b2`) right
at the confirm point, which dumps the currently-active camera pair's `get_last_image()` frames to
an HDF5 file under `~/data/raw/logs/omny_sample_transfer_images/`, named by step + timestamp. **This
is already the reference-image corpus this plan would compare against** — it exists purely as a
post-hoc audit log today, with no comparison logic reading it back.
## Existing infrastructure to build on (no new plumbing needed for a prototype)
- **`IDSCamera.get_last_image()`** (`csaxs_bec/devices/ids_cameras/ids_camera.py:234`) — already
how `_otransfer_save_reference_images()` gets frames; same call works for a live comparison.
- **`OMNYGuiTools._omnycam_active_pair`** (`gui_tools.py:88-105`) — tracks which camera pair
(`cam200`+`cam203` parking/shuttle, or `cam201`+`cam202` sample stage) is currently shown, exactly
the info needed to pick the matching reference subset.
- **`IDSCamera.push_preview_image()` / `push_smear_preview()`** (`ids_camera.py:240,269`) — a
device-server-side hook to publish a processed/composite array back through the same
`device_preview` channel the GUI already subscribes to, without disturbing the live acquisition
feed. **This exact mechanism is already proven for a similar purpose**: LamNI's rotation-center
calibration (`LamNI/x_ray_eye_align.py:737-797`, see `[[project_lamni_rotation_center_calibration]]`-era
work) pushes a growing composite image via `push_smear_preview()`, and
`bec_widgets/widgets/xray_eye/x_ray_eye.py` (`smear_preview_toggle`, lines 386-873) already has a
working UI toggle to switch an `Image` dock between the live feed and this diagnostic channel.
**A vision-diff overlay for OMNY sample transfer should reuse this pattern** — push an
aligned-reference/diff/contour-overlay composite via `push_smear_preview()` on whichever of
cam200/201/202/203 is active, and add the same toggle affordance to OMNY's camera docks
(`gui_tools.py:127-152`) — rather than inventing a third dock or a new preview channel.
- **`opencv-python`** is already a project dependency (`pyproject.toml:27`); `scikit-image` is not
yet (needed only if SSIM specifically is wanted — see below).
## Proposed CV pipeline (classical, no deep learning)
Per the original discussion this grew out of: classical machine vision is enough for a
well-constrained "is the pin geometry right" question; no training data, no model to keep in sync
with hardware changes.
1. **Registration first.** Camera mounts can drift slightly, so raw pixel comparison isn't
meaningful until the current frame is aligned to the reference. `cv2.findTransformECC` for
small translation/rotation (the expected case — a bolted-down camera); an ORB/SIFT +
`cv2.findHomography` fallback if perspective drift turns out to be larger than ECC handles.
2. **Exposure-robust representation.** Compare edges/gradients (Sobel/Canny) or CLAHE-normalized
images, not raw intensity — illumination changes should not look like a defect.
3. **Similarity/anomaly scoring**, two complementary signals:
- **Contour/silhouette match** (`cv2.findContours` + `cv2.matchShapes`) on the extracted
pin/gripper silhouette — the most directly relevant signal for "is the pin sitting right",
and close to fully lighting-invariant.
- **SSIM** (`skimage.metrics.structural_similarity`) on the registered, normalized pair as a
general structural-anomaly fallback, in case a defect doesn't show up as a clean contour
change. (Pulls in `scikit-image` as a new dependency — only if this signal proves worth its
weight; contour matching alone may be sufficient given how constrained the geometry is.)
4. **Multi-reference scoring.** Score against *every* reference image for the given
`(pin_position, step_name, camera)` combination and take the best (minimum-distance) match, not
a single fixed reference — natural sample-to-sample and lighting variation shouldn't inflate the
anomaly score just because one particular reference happened to be a bad match.
5. **Escalate only if needed.** If classical contour/SSIM scoring proves insufficiently robust to
subtle real defects, `anomalib` (e.g. PatchCore) is the documented fallback — but per the
original assessment this is very likely overkill for a fixed pin/gripper geometry, and shouldn't
be built preemptively.
## What "getting rid of human confirmations" should actually mean — staged rollout
Physical hardware, real samples, real damage risk (see the top-pin threshold in gate 1 above) means
this should not be an all-or-nothing flag flip. Proposed phases for the later implementation branch:
- **Phase 0 — instrument, don't automate.** Compute and log the vision score at every existing gate
(1–4 above); still always show the human the same confirm prompt, but now alongside the score and
a diff/overlay visualization (via the `push_smear_preview()` pattern above). Purpose: build a
labeled dataset of real operator pass/fail decisions vs. score, across real lighting and sample
variety, before trusting any threshold. No behavior change to the transfer flow itself.
- **Phase 1 — assist, never silently reject.** A comfortably-high score can auto-answer "yes" at a
gate (with the decision logged and visibly flagged as auto-confirmed, and an easy override); a
low score never fails silently or auto-aborts — it escalates to the same confirm prompt, now with
the diff visualization made prominent, so the human's attention goes exactly where the anomaly
is. This mirrors how the existing top-pin-position check already works (a hard numeric threshold
that *escalates* the confirm text, never silently proceeds or silently aborts).
- **Phase 2 — skip the prompt on high-confidence routine steps.** Only once Phase 0/1 data shows
the scoring is reliable across real operational conditions, let specific gates (candidates: gate
2's "no sample in gripper" check, gate 4's "pick another sample" prompt — the more purely
informational ones) skip the prompt outright on a high score, behind an explicit opt-in/kill
switch. The physical safety checks (top-pin threshold, `_otransfer_check_sensor_connected()`)
stay hard-coded and independent of the vision system regardless of phase — vision augments the
judgment calls, it doesn't replace the interlocks.
## Open questions / decisions needed before implementation starts
- **Reference corpus curation.** Today's saved images are a timestamped append-only run log, not a
curated "golden" set — every run adds a new file, nothing marks one as trustworthy for
comparison. Need either a promotion step (operator marks a saved image as a reference) or a
separate deliberately-curated directory, distinct from the audit-log files
`_otransfer_save_reference_images()` already writes.
- **Per-camera-pair segmentation.** Parking/shuttle view (cam200+cam203) and sample-stage view
(cam201+cam202) have different backgrounds/lighting; silhouette extraction likely needs tuning
per pair, not one global set of CV parameters.
- **Threshold calibration.** Can't be picked a priori — needs the Phase 0 logged-score dataset
across real sessions before any auto-confirm threshold is chosen.
- **Where scores/results live.** Likely alongside the existing HDF5 reference-image files (same
per-step file, additional dataset for score + registration transform), rather than a new storage
mechanism.
## Non-goals for this session
No dependencies added, no code written, no new GUI elements built. This document is the handoff for
the follow-up branch.
@@ -3,9 +3,11 @@
Tracks the live-testing effort for bringing OMNY up to flomni's/BEC's current standards on
branch `update-omny`: bugs found (fixed and unfixed) and what's still untested. Written so a
future session can pick this up without re-deriving it. Companion docs:
`../../flomni/AI_docs/SIMULATED_ENDSTATIONS.md` (the sim architecture) and
`../../flomni/AI_docs/SIMULATED_ENDSTATIONS.md` (the sim architecture),
`../../flomni/AI_docs/FLOMNI_TO_LAMNI_COMPARISON.md` (the flomni→lamni migration this OMNY work
is meant to mirror).
is meant to mirror), and `MACHINE_VISION_AUTOCONFIRM_PLAN.md` (design-only exploration of using
classical machine vision on the sample-transfer camera feeds to reduce/replace the manual confirm
gates in `otransfer_get_sample()`/`otransfer_put_sample()` — not started, separate future branch).
## Current status