diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md new file mode 100644 index 00000000..03bc455f --- /dev/null +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md @@ -0,0 +1,112 @@ +# Gripper-relative localization for machine-vision confirms — exploration plan (2026-09-07) + +Exploration only -- no production code changed in this pass (a few throwaway interactive scripts +were run against real captures, not committed). Grows out of `MACHINE_VISION_AUTOCONFIRM_PLAN.md`'s +Phase 0 and this session's edge-detection tuning work (`vision_toolkit.detect_edges()`'s new +`blur_ksize`, `VisionInterface.set_edge_params()`, the `VisionTuning` GUI widget) -- written so a +future session can pick this up without re-deriving it. + +## Why edge-threshold tuning alone isn't the fix + +A real photo of the gripper/sample gap against a CNC-machined metal background showed dense +machining scratches registering as Canny edges as strongly as the actual object boundary, at any +threshold (verified on `sample_transfer_reference_image_get_1_...h5`: `low=50/high=150` finds +~203k edge pixels, `low=150/high=250` still ~76k, both dominated by texture). Blurring before Canny +helps a lot (down to ~5k at `blur_ksize=9`, ~1.8k at `blur_ksize=15` -- see `VisionInterface`'s new +`set_edge_params()`), but that's still fighting the background rather than avoiding the fight. + +Background/reference-frame differencing (diff the current frame against the saved per-tag +reference, after `register_to_reference()`) was discussed as a stronger alternative -- but Mirko +pointed out a real constraint that rules out a *single* background model: in flomni the camera +moves together with the gripper, so the surrounding view is never static; in OMNY there are several +sample/pick positions, and shuttles that may or may not be loaded, changing what's visible around +the gripper depending on state. There is no one "empty" background to subtract. + +## The reframing: find the gripper, not "the edge" + +Mirko's framing (this session): the gripper's own shape *is* constant across every position/state, +unlike the background. So instead of trying to make edge detection robust to an unpredictable +background, locate the one thing that's always the same shape -- the gripper -- and use its found +location as an anchor for everything else (pin alignment/presence relative to it). If gripper +localization works reliably, the same primitive works for flomni and OMNY both -- OMNY doesn't need +a separate solution, per Mirko's own framing ("if that works, we are also ready for omny"). + +## What was tried this session (real captures, not synthetic) + +Using `sample_transfer_reference_image_get_1_...h5` (already in `reference_images/`): + +1. **Largest-area contour (today's `vision_toolkit.largest_contour()`)**: picks a contour centered + around (103, 500) on the full frame -- nowhere near the gripper, which is visually centered + around (600, 975). Confirms the existing failure mode directly. +2. **Hu-moment shape match (`contour_similarity_score()`) against a hand-picked "gripper" reference + contour, searched over every contour in the full frame (`cv2.RETR_LIST`), no area filter**: 8429 + candidate contours at default noisy thresholds; the best-scoring match (lowest Hu-moment + distance) was nowhere near the gripper either -- tiny noise fragments can coincidentally match a + reference's *shape* since Hu moments are scale-invariant by construction (a speck the same shape + as the reference, just 100x smaller, scores identically to the real thing). +3. **Same idea + blur (`blur_ksize=9`) + stricter thresholds (150/250) + an area filter (candidate + area within 0.3x-3x of the reference's area)**: cut candidates from 8429 to 93 -- meaningfully + better -- but the best match still landed roughly 250-300px off from the true gripper center. + Not a clean win yet. + +**Diagnosis (not yet resolved)**: the "reference gripper contour" used above was itself just +`largest_contour()` inside a hand-picked ROI at whatever thresholds happened to be set -- and at +those settings, the contour actually found inside that ROI was tiny (area=102), consistent with an +earlier finding this session that a small dark clip/marker mark, not the gripper's own silhouette, +tends to be the strongest *edge* feature in that region. The gripper itself is bright, reflective, +mostly-uniform metal -- it likely has very little internal gradient for Canny to find at all. Canny/ +Hu-moment contour matching may simply not be the right primitive for finding the gripper's outline +specifically, as distinct from finding a real pin/sample against it (a lower-contrast object Canny +is already reasonably suited to elsewhere in this codebase). + +## Recommended next steps (not yet implemented) + +1. **Try brightness-based segmentation for the gripper silhouette instead of edge detection.** A + bright, reflective metal gripper against a darker background is a better fit for a threshold/ + segmentation approach (e.g. Otsu threshold on the grayscale/CLAHE-normalized image + + `cv2.RETR_EXTERNAL` contour of the largest bright blob) than gradient-based Canny, which depends + on *internal* texture/gradient the gripper mostly doesn't have. Layer 0 (`vision_toolkit.py`) + has no segmentation-by-intensity function yet -- a small new addition alongside + `detect_edges()`, e.g. `segment_by_intensity()`. +2. **Curate a real gripper reference/template deliberately**, not by auto-extracting + `largest_contour()` from an arbitrary ROI at whatever thresholds happen to be set at test time -- + e.g. crop tightly to just the gripper from a clean, manually-confirmed capture, and save that + contour (or a small library of them, across the couple of orientations/scales the gripper + actually appears at) as a proper, versioned template. +3. **Once gripper localization is reliable, define the pin-search ROI relative to the found gripper + location** (e.g. "N pixels beyond the gripper's leading edge/centroid") instead of the fixed, + hand-tuned ROI (`(300, 700, 600, 550)`) used throughout this session's testing -- this is what + actually generalizes across OMNY's several sample/storage positions, since the ROI would move + with wherever the gripper is actually found in that frame instead of being one static rectangle. +4. **Feature-based matching (ORB/SIFT + homography) as a fallback/alternative to contour shape + matching**, if brightness segmentation doesn't pan out either -- more robust to partial occlusion + (a loaded shuttle blocking part of the view, per Mirko's point) than a single-contour shape + match, at the cost of being a heavier, less classical-simple technique than the rest of this + toolkit; worth trying only if steps 1-2 don't get far enough on their own. +5. **Re-evaluate whether per-tag reference-frame diffing (`register_to_reference()` + + `cv2.absdiff()`, discussed earlier this session) is better used *after* gripper localization** -- + e.g. once the gripper's found and roughly aligned, diffing the pin-region against the + corresponding region of the tag's saved reference might be a simpler, more robust presence/ + alignment check than another round of contour extraction. Worth prototyping once 1-3 land -- + this is what unlocks pin-alignment-relative-to-gripper, the actual Phase-0 goal, not gripper + localization for its own sake. + +## Applicability to OMNY + +Per Mirko: OMNY has multiple sample/pick positions, and shuttles that may or may not be loaded -- +meaning the background around the gripper is genuinely different depending on state, not just "the +same background under different lighting." A working gripper-localization primitive (steps 1-3 +above) needs no OMNY-specific logic to handle this -- it searches for the gripper wherever it +actually is in the frame, rather than assuming a fixed expected background -- so this is the one +piece of Phase 0 work that, if it works for flomni, should carry over to OMNY without a second +implementation effort, consistent with `vision_toolkit.py`'s existing endstation-agnostic design +(its own module docstring already frames it as shared by "any device or client-side code," OMNY and +a possible future x-ray-eye-alignment consumer alike). + +## What this doesn't touch + +`VisionInterface`'s edge-threshold/blur tuning (this session's other change) and the `VisionTuning` +widget stay as-is and remain useful regardless of this direction -- gripper localization would sit +*above* them (a new Layer 0 function plus whatever Layer 1/2 wiring ends up calling it), not replace +the existing edge-detection knobs, which are still the right tool for tuning pin/edge detection once +a gripper-relative ROI exists to run them inside of.