docs(omny): design plan for machine-vision auto-confirm on sample transfer
CI for csaxs_bec / test (push) Successful in 1m58s
CI for csaxs_bec / test (push) Successful in 1m58s
Explores classical CV (registration + edge/SSIM/contour matching against multiple reference images) to reduce the manual y/n confirms in otransfer_get_sample()/otransfer_put_sample(), reusing the existing reference-image capture and the push_smear_preview() GUI-overlay pattern already proven for LamNI's rotation-center calibration. Exploration only; implementation to follow on a separate branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Mupip8x7Hean6E3FFvpbj
This commit is contained in:
1 parent
415d55f599
commit
3241e880ce
2 files changed
+151
-2
No files matched your search
@@ -0,0 +1,147 @@
|
||||
# OMNY sample-transfer machine-vision auto-confirm — design plan (2026-09-06)
|
||||
|
||||
Exploration only — **no code in this pass**. Written on `update-omny` so a future session/branch
|
||||
can pick this up without re-deriving it. Companion docs: `OPEN_ISSUES.md` (the live-testing effort
|
||||
this plan grows out of — see its "full gripper sample-transfer chain... untested" item) and
|
||||
`SAMPLE_STORAGE_TEST_CHECKLIST.md`.
|
||||
|
||||
## Motivation
|
||||
|
||||
Mirko's starting question: is there a Python package for comparing a live gripper/pin camera image
|
||||
against a set of reference images, robust to exposure changes and small camera-alignment drift, to
|
||||
decide "is the pin sitting in the gripper correctly" — classic machine-vision, not deep learning.
|
||||
Applied to BEC: could this let OMNY's sample-transfer flow (`otransfer_get_sample()` /
|
||||
`otransfer_put_sample()`) auto-approve routine steps instead of stopping for a human `y/n` at every
|
||||
single one, while still catching real problems (pin damage, missed grip, wrong shuttle state)?
|
||||
|
||||
This doc maps that idea onto OMNY's actual confirm gates and camera/GUI plumbing, and proposes a
|
||||
staged plan. **Goal for this session: write this plan down. Implementation is a separate, later
|
||||
branch.**
|
||||
|
||||
## Where today's human confirmations actually live
|
||||
|
||||
All of these are in `omny_sample_transfer_mixin.py` and go through
|
||||
`otransfer_confirm_dialog()` (line 76 — GUI `ConsoleButtonsWidget` yes/no if the camera console is
|
||||
already open, else a blocking CLI `yesno()`):
|
||||
|
||||
1. **`_otransfer_confirm()`** (line 848) — the main per-step handshake during an actual mount/get,
|
||||
driven by the Galil controller's `confirm`/`mntprgs` variables (`XQ#GRPUT`/`XQ#GRGET` macros).
|
||||
Loops until the controller reports `mntprgs == 0`; each iteration is one manual "All OK?"
|
||||
gate. For a *get*, the first iteration additionally hard-checks the top-pin position against a
|
||||
fixed threshold (`-25.4`, line 866) and asks an extra, more alarming confirm if it's exceeded —
|
||||
this is a **physical safety check**, not a generic step confirm, and should stay independent of
|
||||
any vision system.
|
||||
2. **`otransfer_get_sample()`** (line 934) — "confirm there is no sample in the gripper, it would
|
||||
be dropped" before starting at all.
|
||||
3. **`otransfer_get_sample()`** (line 971) / **`otransfer_put_sample()`** (line 812) — "Ready to
|
||||
start the [un]mount process now?" right before triggering `XQ#GRGET`/`XQ#GRPUT`.
|
||||
4. **`otransfer_put_sample()`** (line 836) — "Pick another sample from this shuttle? Keep shield
|
||||
open?" after a successful put.
|
||||
5. **`_otransfer_shuttle_align()`** (line 189) — one-off "shall I try to move the aligner to the
|
||||
correct start position" recovery prompt, only on a specific bad-limit-switch state. Edge case,
|
||||
not a routine per-transfer gate — out of scope for auto-confirm.
|
||||
6. **`otransfer_park_loadlock_slot()`** (line 523) — CLI-only `yesno()` by design (operator is
|
||||
physically at the loadlock, not watching the GUI camera console) — also out of scope here.
|
||||
|
||||
Gates 1–4 are the routine, repeated-every-transfer confirmations this plan targets. Every one of
|
||||
them already calls **`_otransfer_save_reference_images()`** (line 885, landed in `a7429b2`) right
|
||||
at the confirm point, which dumps the currently-active camera pair's `get_last_image()` frames to
|
||||
an HDF5 file under `~/data/raw/logs/omny_sample_transfer_images/`, named by step + timestamp. **This
|
||||
is already the reference-image corpus this plan would compare against** — it exists purely as a
|
||||
post-hoc audit log today, with no comparison logic reading it back.
|
||||
|
||||
## Existing infrastructure to build on (no new plumbing needed for a prototype)
|
||||
|
||||
- **`IDSCamera.get_last_image()`** (`csaxs_bec/devices/ids_cameras/ids_camera.py:234`) — already
|
||||
how `_otransfer_save_reference_images()` gets frames; same call works for a live comparison.
|
||||
- **`OMNYGuiTools._omnycam_active_pair`** (`gui_tools.py:88-105`) — tracks which camera pair
|
||||
(`cam200`+`cam203` parking/shuttle, or `cam201`+`cam202` sample stage) is currently shown, exactly
|
||||
the info needed to pick the matching reference subset.
|
||||
- **`IDSCamera.push_preview_image()` / `push_smear_preview()`** (`ids_camera.py:240,269`) — a
|
||||
device-server-side hook to publish a processed/composite array back through the same
|
||||
`device_preview` channel the GUI already subscribes to, without disturbing the live acquisition
|
||||
feed. **This exact mechanism is already proven for a similar purpose**: LamNI's rotation-center
|
||||
calibration (`LamNI/x_ray_eye_align.py:737-797`, see `[[project_lamni_rotation_center_calibration]]`-era
|
||||
work) pushes a growing composite image via `push_smear_preview()`, and
|
||||
`bec_widgets/widgets/xray_eye/x_ray_eye.py` (`smear_preview_toggle`, lines 386-873) already has a
|
||||
working UI toggle to switch an `Image` dock between the live feed and this diagnostic channel.
|
||||
**A vision-diff overlay for OMNY sample transfer should reuse this pattern** — push an
|
||||
aligned-reference/diff/contour-overlay composite via `push_smear_preview()` on whichever of
|
||||
cam200/201/202/203 is active, and add the same toggle affordance to OMNY's camera docks
|
||||
(`gui_tools.py:127-152`) — rather than inventing a third dock or a new preview channel.
|
||||
- **`opencv-python`** is already a project dependency (`pyproject.toml:27`); `scikit-image` is not
|
||||
yet (needed only if SSIM specifically is wanted — see below).
|
||||
|
||||
## Proposed CV pipeline (classical, no deep learning)
|
||||
|
||||
Per the original discussion this grew out of: classical machine vision is enough for a
|
||||
well-constrained "is the pin geometry right" question; no training data, no model to keep in sync
|
||||
with hardware changes.
|
||||
|
||||
1. **Registration first.** Camera mounts can drift slightly, so raw pixel comparison isn't
|
||||
meaningful until the current frame is aligned to the reference. `cv2.findTransformECC` for
|
||||
small translation/rotation (the expected case — a bolted-down camera); an ORB/SIFT +
|
||||
`cv2.findHomography` fallback if perspective drift turns out to be larger than ECC handles.
|
||||
2. **Exposure-robust representation.** Compare edges/gradients (Sobel/Canny) or CLAHE-normalized
|
||||
images, not raw intensity — illumination changes should not look like a defect.
|
||||
3. **Similarity/anomaly scoring**, two complementary signals:
|
||||
- **Contour/silhouette match** (`cv2.findContours` + `cv2.matchShapes`) on the extracted
|
||||
pin/gripper silhouette — the most directly relevant signal for "is the pin sitting right",
|
||||
and close to fully lighting-invariant.
|
||||
- **SSIM** (`skimage.metrics.structural_similarity`) on the registered, normalized pair as a
|
||||
general structural-anomaly fallback, in case a defect doesn't show up as a clean contour
|
||||
change. (Pulls in `scikit-image` as a new dependency — only if this signal proves worth its
|
||||
weight; contour matching alone may be sufficient given how constrained the geometry is.)
|
||||
4. **Multi-reference scoring.** Score against *every* reference image for the given
|
||||
`(pin_position, step_name, camera)` combination and take the best (minimum-distance) match, not
|
||||
a single fixed reference — natural sample-to-sample and lighting variation shouldn't inflate the
|
||||
anomaly score just because one particular reference happened to be a bad match.
|
||||
5. **Escalate only if needed.** If classical contour/SSIM scoring proves insufficiently robust to
|
||||
subtle real defects, `anomalib` (e.g. PatchCore) is the documented fallback — but per the
|
||||
original assessment this is very likely overkill for a fixed pin/gripper geometry, and shouldn't
|
||||
be built preemptively.
|
||||
|
||||
## What "getting rid of human confirmations" should actually mean — staged rollout
|
||||
|
||||
Physical hardware, real samples, real damage risk (see the top-pin threshold in gate 1 above) means
|
||||
this should not be an all-or-nothing flag flip. Proposed phases for the later implementation branch:
|
||||
|
||||
- **Phase 0 — instrument, don't automate.** Compute and log the vision score at every existing gate
|
||||
(1–4 above); still always show the human the same confirm prompt, but now alongside the score and
|
||||
a diff/overlay visualization (via the `push_smear_preview()` pattern above). Purpose: build a
|
||||
labeled dataset of real operator pass/fail decisions vs. score, across real lighting and sample
|
||||
variety, before trusting any threshold. No behavior change to the transfer flow itself.
|
||||
- **Phase 1 — assist, never silently reject.** A comfortably-high score can auto-answer "yes" at a
|
||||
gate (with the decision logged and visibly flagged as auto-confirmed, and an easy override); a
|
||||
low score never fails silently or auto-aborts — it escalates to the same confirm prompt, now with
|
||||
the diff visualization made prominent, so the human's attention goes exactly where the anomaly
|
||||
is. This mirrors how the existing top-pin-position check already works (a hard numeric threshold
|
||||
that *escalates* the confirm text, never silently proceeds or silently aborts).
|
||||
- **Phase 2 — skip the prompt on high-confidence routine steps.** Only once Phase 0/1 data shows
|
||||
the scoring is reliable across real operational conditions, let specific gates (candidates: gate
|
||||
2's "no sample in gripper" check, gate 4's "pick another sample" prompt — the more purely
|
||||
informational ones) skip the prompt outright on a high score, behind an explicit opt-in/kill
|
||||
switch. The physical safety checks (top-pin threshold, `_otransfer_check_sensor_connected()`)
|
||||
stay hard-coded and independent of the vision system regardless of phase — vision augments the
|
||||
judgment calls, it doesn't replace the interlocks.
|
||||
|
||||
## Open questions / decisions needed before implementation starts
|
||||
|
||||
- **Reference corpus curation.** Today's saved images are a timestamped append-only run log, not a
|
||||
curated "golden" set — every run adds a new file, nothing marks one as trustworthy for
|
||||
comparison. Need either a promotion step (operator marks a saved image as a reference) or a
|
||||
separate deliberately-curated directory, distinct from the audit-log files
|
||||
`_otransfer_save_reference_images()` already writes.
|
||||
- **Per-camera-pair segmentation.** Parking/shuttle view (cam200+cam203) and sample-stage view
|
||||
(cam201+cam202) have different backgrounds/lighting; silhouette extraction likely needs tuning
|
||||
per pair, not one global set of CV parameters.
|
||||
- **Threshold calibration.** Can't be picked a priori — needs the Phase 0 logged-score dataset
|
||||
across real sessions before any auto-confirm threshold is chosen.
|
||||
- **Where scores/results live.** Likely alongside the existing HDF5 reference-image files (same
|
||||
per-step file, additional dataset for score + registration transform), rather than a new storage
|
||||
mechanism.
|
||||
|
||||
## Non-goals for this session
|
||||
|
||||
No dependencies added, no code written, no new GUI elements built. This document is the handoff for
|
||||
the follow-up branch.
|
||||
@@ -3,9 +3,11 @@
|
||||
Tracks the live-testing effort for bringing OMNY up to flomni's/BEC's current standards on
|
||||
branch `update-omny`: bugs found (fixed and unfixed) and what's still untested. Written so a
|
||||
future session can pick this up without re-deriving it. Companion docs:
|
||||
`../../flomni/AI_docs/SIMULATED_ENDSTATIONS.md` (the sim architecture) and
|
||||
`../../flomni/AI_docs/SIMULATED_ENDSTATIONS.md` (the sim architecture),
|
||||
`../../flomni/AI_docs/FLOMNI_TO_LAMNI_COMPARISON.md` (the flomni→lamni migration this OMNY work
|
||||
is meant to mirror).
|
||||
is meant to mirror), and `MACHINE_VISION_AUTOCONFIRM_PLAN.md` (design-only exploration of using
|
||||
classical machine vision on the sample-transfer camera feeds to reduce/replace the manual confirm
|
||||
gates in `otransfer_get_sample()`/`otransfer_put_sample()` — not started, separate future branch).
|
||||
|
||||
## Current status
|
||||
|
||||
|
||||
Reference in new issue
Block a user