docs(vision): plan gripper-relative localization as the real fix for background variance

Edge-threshold tuning (previous commit) helps but is still fighting a background
that isn't static -- flomni's camera moves with the gripper, and OMNY has
several sample/pick positions with shuttles that may or may not be loaded, so
there's no single reference background to diff against either.

Reframing (Mirko, this session): the gripper's shape is the one constant across
every position/state -- find it first, then locate/assess the pin relative to
it. This plan records three real experiments run against the flomni captures
already in reference_images/ (largest-area contour, unfiltered Hu-moment search
over all contours, and a blur+area-filtered version) -- none cleanly located the
gripper yet, and documents why (the gripper is bright/reflective with little
internal gradient for Canny to find, so the "reference contour" extracted this
session was actually a marker mark, not the gripper's own silhouette) -- plus
concrete next steps: brightness-based segmentation instead of edge detection,
a deliberately-curated gripper template, a gripper-relative pin-search ROI, and
re-evaluating reference-frame diffing as a post-localization step.

No production code changed in this pass -- exploration only, matching
MACHINE_VISION_AUTOCONFIRM_PLAN.md's own convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQpiwJcJZJKGo6Pzozfsi1

csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md

@@ -0,0 +1,112 @@
+# Gripper-relative localization for machine-vision confirms — exploration plan (2026-09-07)
+
+Exploration only -- no production code changed in this pass (a few throwaway interactive scripts
+were run against real captures, not committed). Grows out of `MACHINE_VISION_AUTOCONFIRM_PLAN.md`'s
+Phase 0 and this session's edge-detection tuning work (`vision_toolkit.detect_edges()`'s new
+`blur_ksize`, `VisionInterface.set_edge_params()`, the `VisionTuning` GUI widget) -- written so a
+future session can pick this up without re-deriving it.
+
+## Why edge-threshold tuning alone isn't the fix
+
+A real photo of the gripper/sample gap against a CNC-machined metal background showed dense
+machining scratches registering as Canny edges as strongly as the actual object boundary, at any
+threshold (verified on `sample_transfer_reference_image_get_1_...h5`: `low=50/high=150` finds
+~203k edge pixels, `low=150/high=250` still ~76k, both dominated by texture). Blurring before Canny
+helps a lot (down to ~5k at `blur_ksize=9`, ~1.8k at `blur_ksize=15` -- see `VisionInterface`'s new
+`set_edge_params()`), but that's still fighting the background rather than avoiding the fight.
+
+Background/reference-frame differencing (diff the current frame against the saved per-tag
+reference, after `register_to_reference()`) was discussed as a stronger alternative -- but Mirko
+pointed out a real constraint that rules out a *single* background model: in flomni the camera
+moves together with the gripper, so the surrounding view is never static; in OMNY there are several
+sample/pick positions, and shuttles that may or may not be loaded, changing what's visible around
+the gripper depending on state. There is no one "empty" background to subtract.
+
+## The reframing: find the gripper, not "the edge"
+
+Mirko's framing (this session): the gripper's own shape *is* constant across every position/state,
+unlike the background. So instead of trying to make edge detection robust to an unpredictable
+background, locate the one thing that's always the same shape -- the gripper -- and use its found
+location as an anchor for everything else (pin alignment/presence relative to it). If gripper
+localization works reliably, the same primitive works for flomni and OMNY both -- OMNY doesn't need
+a separate solution, per Mirko's own framing ("if that works, we are also ready for omny").
+
+## What was tried this session (real captures, not synthetic)
+
+Using `sample_transfer_reference_image_get_1_...h5` (already in `reference_images/`):
+
+1. **Largest-area contour (today's `vision_toolkit.largest_contour()`)**: picks a contour centered
+   around (103, 500) on the full frame -- nowhere near the gripper, which is visually centered
+   around (600, 975). Confirms the existing failure mode directly.
+2. **Hu-moment shape match (`contour_similarity_score()`) against a hand-picked "gripper" reference
+   contour, searched over every contour in the full frame (`cv2.RETR_LIST`), no area filter**: 8429
+   candidate contours at default noisy thresholds; the best-scoring match (lowest Hu-moment
+   distance) was nowhere near the gripper either -- tiny noise fragments can coincidentally match a
+   reference's *shape* since Hu moments are scale-invariant by construction (a speck the same shape
+   as the reference, just 100x smaller, scores identically to the real thing).
+3. **Same idea + blur (`blur_ksize=9`) + stricter thresholds (150/250) + an area filter (candidate
+   area within 0.3x-3x of the reference's area)**: cut candidates from 8429 to 93 -- meaningfully
+   better -- but the best match still landed roughly 250-300px off from the true gripper center.
+   Not a clean win yet.
+
+**Diagnosis (not yet resolved)**: the "reference gripper contour" used above was itself just
+`largest_contour()` inside a hand-picked ROI at whatever thresholds happened to be set -- and at
+those settings, the contour actually found inside that ROI was tiny (area=102), consistent with an
+earlier finding this session that a small dark clip/marker mark, not the gripper's own silhouette,
+tends to be the strongest *edge* feature in that region. The gripper itself is bright, reflective,
+mostly-uniform metal -- it likely has very little internal gradient for Canny to find at all. Canny/
+Hu-moment contour matching may simply not be the right primitive for finding the gripper's outline
+specifically, as distinct from finding a real pin/sample against it (a lower-contrast object Canny
+is already reasonably suited to elsewhere in this codebase).
+
+## Recommended next steps (not yet implemented)
+
+1. **Try brightness-based segmentation for the gripper silhouette instead of edge detection.** A
+   bright, reflective metal gripper against a darker background is a better fit for a threshold/
+   segmentation approach (e.g. Otsu threshold on the grayscale/CLAHE-normalized image +
+   `cv2.RETR_EXTERNAL` contour of the largest bright blob) than gradient-based Canny, which depends
+   on *internal* texture/gradient the gripper mostly doesn't have. Layer 0 (`vision_toolkit.py`)
+   has no segmentation-by-intensity function yet -- a small new addition alongside
+   `detect_edges()`, e.g. `segment_by_intensity()`.
+2. **Curate a real gripper reference/template deliberately**, not by auto-extracting
+   `largest_contour()` from an arbitrary ROI at whatever thresholds happen to be set at test time --
+   e.g. crop tightly to just the gripper from a clean, manually-confirmed capture, and save that
+   contour (or a small library of them, across the couple of orientations/scales the gripper
+   actually appears at) as a proper, versioned template.
+3. **Once gripper localization is reliable, define the pin-search ROI relative to the found gripper
+   location** (e.g. "N pixels beyond the gripper's leading edge/centroid") instead of the fixed,
+   hand-tuned ROI (`(300, 700, 600, 550)`) used throughout this session's testing -- this is what
+   actually generalizes across OMNY's several sample/storage positions, since the ROI would move
+   with wherever the gripper is actually found in that frame instead of being one static rectangle.
+4. **Feature-based matching (ORB/SIFT + homography) as a fallback/alternative to contour shape
+   matching**, if brightness segmentation doesn't pan out either -- more robust to partial occlusion
+   (a loaded shuttle blocking part of the view, per Mirko's point) than a single-contour shape
+   match, at the cost of being a heavier, less classical-simple technique than the rest of this
+   toolkit; worth trying only if steps 1-2 don't get far enough on their own.
+5. **Re-evaluate whether per-tag reference-frame diffing (`register_to_reference()` +
+   `cv2.absdiff()`, discussed earlier this session) is better used *after* gripper localization** --
+   e.g. once the gripper's found and roughly aligned, diffing the pin-region against the
+   corresponding region of the tag's saved reference might be a simpler, more robust presence/
+   alignment check than another round of contour extraction. Worth prototyping once 1-3 land --
+   this is what unlocks pin-alignment-relative-to-gripper, the actual Phase-0 goal, not gripper
+   localization for its own sake.
+
+## Applicability to OMNY
+
+Per Mirko: OMNY has multiple sample/pick positions, and shuttles that may or may not be loaded --
+meaning the background around the gripper is genuinely different depending on state, not just "the
+same background under different lighting." A working gripper-localization primitive (steps 1-3
+above) needs no OMNY-specific logic to handle this -- it searches for the gripper wherever it
+actually is in the frame, rather than assuming a fixed expected background -- so this is the one
+piece of Phase 0 work that, if it works for flomni, should carry over to OMNY without a second
+implementation effort, consistent with `vision_toolkit.py`'s existing endstation-agnostic design
+(its own module docstring already frames it as shared by "any device or client-side code," OMNY and
+a possible future x-ray-eye-alignment consumer alike).
+
+## What this doesn't touch
+
+`VisionInterface`'s edge-threshold/blur tuning (this session's other change) and the `VisionTuning`
+widget stay as-is and remain useful regardless of this direction -- gripper localization would sit
+*above* them (a new Layer 0 function plus whatever Layer 1/2 wiring ends up calling it), not replace
+the existing edge-detection knobs, which are still the right tool for tuning pin/edge detection once
+a gripper-relative ROI exists to run them inside of.