diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md index 51489df5..9559ff2e 100644 --- a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md @@ -1,10 +1,44 @@ -# Gripper-relative localization for machine-vision confirms — exploration plan (2026-09-07) +# Gripper-relative localization for machine-vision confirms — exploration plan (2026-09-07, updated 2026-09-08) -Exploration only -- no production code changed in this pass (a few throwaway interactive scripts -were run against real captures, not committed). Grows out of `MACHINE_VISION_AUTOCONFIRM_PLAN.md`'s -Phase 0 and this session's edge-detection tuning work (`vision_toolkit.detect_edges()`'s new -`blur_ksize`, `VisionInterface.set_edge_params()`, the `VisionTuning` GUI widget) -- written so a -future session can pick this up without re-deriving it. +Grew out of `MACHINE_VISION_AUTOCONFIRM_PLAN.md`'s Phase 0 and (originally) a same-session +edge-detection tuning pass (`vision_toolkit.detect_edges()`'s `blur_ksize`, +`VisionInterface.set_edge_params()`, the `VisionTuning` GUI widget -- all committed, unrelated to +gripper localization itself, still useful as-is). Unlike when this doc started, real production code +has since landed here too (`find_dark_holes()`, `detect_aruco_markers()`, the `VisionTuning` +`"holes"` mode) -- read on for what's real/tested vs. still open. + +## Status as of 2026-09-08 -- start here in a fresh session + +**Current direction: fiducial markers (ArUco), not trying to recognize the gripper itself.** +Two prior approaches were tried and both hit real, understood dead ends against real photos (not +guessed) -- see the dated updates below for the full evidence trail: +- Edge detection (Canny, any threshold/blur) never finds a contour anywhere near the gripper's + actual size -- it has almost no internal gradient for Canny to find at all (reflective metal). +- The gripper's jaw *notch* segments cleanly as a `find_dark_holes()` "hole" -- but only when it + physically exists: with the pin fully seated, the notch/gap simply isn't there. Not a detection + bug, a real state-dependence that rules out the notch as the sole anchor. + +**What's real and committed right now**: +- `vision_toolkit.find_dark_holes(image, area_range)` -- Otsu + inverted `RETR_CCOMP` hole search, + tested, works well *when the notch exists*. +- `vision_toolkit.detect_aruco_markers(image, dictionary_name)` -- generic ArUco detection, tested + against synthetic markers only (no real marked gripper photo exists yet). +- `VisionInterface`/`VisionTuning` both have a live `"holes"` mode to watch `find_dark_holes()` + results interactively (Device → Camera → Mode=`holes` → set area min/max → Apply). +- `csaxs_bec/devices/vision/fiducial_markers/` -- 8 printable ArUco tags (`DICT_4X4_50`, ids 0-7), + ids 0/1 assigned (flomni gripper left/right jaw), 2-7 free for OMNY's other views. See that + folder's own `README.md`. + +**What's next, concretely, once physical tags exist** (nothing below this point can proceed without +a real photo of the gripper with tags actually mounted -- that's on Mirko, not blocked on code): +1. Confirm `detect_aruco_markers()` finds the mounted tags on a real capture (lighting/print quality + could differ from the synthetic round-trip already verified). +2. Define the pin-search region in tag-relative coordinates instead of a fixed frame ROI. +3. Re-run `find_dark_holes()`/`detect_edges()` for the pin check itself, now inside that small, + reliably-anchored region instead of an unconstrained, shifting frame. +4. **Open, not yet decided**: OMNY's vacuum-compatible fabrication method (laser engraving vs. + Kapton vs. something else) -- needs sign-off from whoever manages vacuum-material compatibility + at cSAXS, not a call to make in a coding session. ## Why edge-threshold tuning alone isn't the fix @@ -215,16 +249,37 @@ environment's OpenCV 5.0 build), so a scratch essentially never false-triggers o returns the 4 corner points directly -- from a known physical tag size that's position, in-plane rotation, *and* scale in one step, with no custom geometry code to get right. -**Chosen spec**: `DICT_4X4_50`, marker id `0` for the left jaw, id `1` for the right jaw -- printed -as large as each jaw's flat outer face allows (bigger is more robust to glare/low local contrast, -don't minimize it), on matte (not glossy) material, mounted with a bit of white quiet-zone margin -left around the black border (don't cut flush to the marker edge), kept flat (no wrinkling/doming, -which would corrupt the corner-based pose estimate). Two tags, not one: redundancy if one is -occluded from some angle, and the distance between them becomes a free signal for jaw open/closed -state later. Generated marker images (`DICT_4X4_50`, ids 0/1, with a 15%-of-side white margin) were -produced and print-verified this session (round-tripped through `cv2.aruco` detection correctly, -including after aggressive downscale+blur simulating a small tag seen from a distance) -- see -`detect_aruco_markers()` below for how to regenerate them. +**Chosen spec**: `DICT_4X4_50`, marker id `0` for the left jaw, id `1` for the right jaw (confirmed, +not left/right symmetric duplicates of the same id -- distinct ids make each detection +self-identifying regardless of where it lands in frame, rather than relying on position alone to +tell them apart) -- printed as large as each jaw's flat outer face allows (bigger is more robust to +glare/low local contrast, don't minimize it; mounting doesn't need to cover the whole jaw, any +consistently-visible flat spot works), on matte (not glossy) material, mounted with a bit of white +quiet-zone margin left around the black border (don't cut flush to the marker edge), kept flat (no +wrinkling/doming, which would corrupt the corner-based pose estimate). + +**Printable marker files: `csaxs_bec/devices/vision/fiducial_markers/` (committed -- small, +deterministic, reproducible, not real capture data, unlike `reference_images/`).** Eight markers +generated (ids 0-7, `DICT_4X4_50`, 780x780px, 15%-of-side white margin), not just the two currently +assigned -- OMNY has several camera views/sites of the same physical gripper (parking/shuttle pair +vs. sample-stage pair, per `VisionInterface`'s own docstring, possibly others), and a single tag per +jaw face may not be visible from every angle it's viewed from there. Ids 2-7 are generated and +available, not yet assigned -- see that folder's own `README.md` for the regeneration recipe and +the current id-assignment table; update it, don't let it drift out of sync with this doc, once +OMNY's actual mounting positions are decided. All eight round-trip-verified through +`vision_toolkit.detect_aruco_markers()` this session, including after aggressive downscale (to +60x60) + blur simulating a small tag seen from a distance. + +**Material -- flomni vs. OMNY differ, and this is a real open decision, not yet made.** flomni is +not a vacuum environment (per the captures already in `reference_images/`) -- plain paper + glue, as +originally planned, is fine there. **OMNY is vacuum**: paper, most glues, and most tapes outgas and +are not vacuum-safe as-is. Two candidate fabrication methods, neither committed to yet: (a) +laser-engrave/mark the pattern directly onto the metal jaw -- no adhesive, nothing to outgas, +permanent (an established technique, "Direct Part Marking"); (b) a known low-outgassing material +like Kapton (polyimide) film/tape, already familiar at synchrotron beamlines, if engraving isn't +practical. **Whoever manages vacuum-material compatibility at cSAXS needs to confirm before +committing** -- this is a real vacuum-safety call, outside what a software session should decide on +its own. **New Layer 0 primitive, already implemented and tested (not waiting on the physical markers)**: `vision_toolkit.detect_aruco_markers(image, dictionary_name="DICT_4X4_50") -> list[ArucoDetection]` diff --git a/csaxs_bec/devices/vision/fiducial_markers/README.md b/csaxs_bec/devices/vision/fiducial_markers/README.md new file mode 100644 index 00000000..7069a258 --- /dev/null +++ b/csaxs_bec/devices/vision/fiducial_markers/README.md @@ -0,0 +1,60 @@ +# Gripper fiducial markers (ArUco, DICT_4X4_50) + +Printable ArUco markers for the gripper-localization approach in +`../../bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md` -- see that doc for +the full reasoning (why fiducials over trying to recognize the gripper itself, why ArUco over +hand-drawn dots). Detected client-side by `vision_toolkit.detect_aruco_markers()`. + +## What's here + +`aruco_4x4_50_id0.png` .. `aruco_4x4_50_id7.png` -- eight markers, `DICT_4X4_50`, each 780x780 px +(a 600px marker + a 90px/15% white quiet-zone margin on every side). Fully reproducible from the +dictionary name + id -- these files are a convenience (so they don't need regenerating/reprinting +from scratch each time), not the source of truth: + +```python +import cv2 +import cv2.aruco as aruco + +dictionary = aruco.getPredefinedDictionary(aruco.DICT_4X4_50) +marker = aruco.generateImageMarker(dictionary, marker_id, 600) # marker_id: 0-7 here +bordered = cv2.copyMakeBorder(marker, 90, 90, 90, 90, cv2.BORDER_CONSTANT, value=255) +cv2.imwrite(f"aruco_4x4_50_id{marker_id}.png", bordered) +``` + +Round-trip-verified this session: `vision_toolkit.detect_aruco_markers()` correctly decodes each +file back to its own id, including after aggressive downscale (to 60x60) + Gaussian blur simulating +a small tag seen from a distance. + +## Current id assignment + +- **id 0** -- flomni gripper, left jaw. +- **id 1** -- flomni gripper, right jaw. +- **ids 2-7** -- generated and available, not yet assigned. OMNY has several camera views/sites of + the same physical gripper (per Mirko: parking/shuttle pair vs. sample-stage pair, others + possibly), and a single tag per jaw face may not be visible from every angle the gripper is + viewed from there -- extra ids are here so additional tags (different faces/orientations of the + same gripper, or a second gripper) can be assigned without regenerating anything. Update this + list once OMNY's actual mounting positions are decided -- don't let it go stale. + +## Printing + +Print at whatever physical size the target flat mounting face allows -- bigger is more robust to +glare/low local contrast, don't minimize it. These PNGs are high enough resolution (780px) to print +sharply from a few mm up to several cm; set the *physical* print size explicitly (e.g. insert into a +word processor doc, set the image object's size in mm, print at 100% scale) rather than "fit to +page", which won't give a controlled physical size. + +Leave the white quiet-zone margin intact when trimming -- don't cut flush to the marker's black +border, detection relies on that contrast margin. Mount flat (no wrinkling/doming), which would +distort the corner positions the pose/scale estimate is computed from. + +**Material -- flomni vs. OMNY differ.** flomni is not a vacuum environment (per the captures in +`../reference_images/`): plain paper + glue, as originally planned, is fine. **OMNY is vacuum** -- +paper, most glues, and most tapes outgas and are not vacuum-safe. Options, not yet decided: +laser-engraving/marking the pattern directly onto the metal jaw (no adhesive, nothing to outgas, +permanent -- an established technique, usually called Direct Part Marking); or a known +low-outgassing material like Kapton (polyimide) film/tape, already familiar at synchrotron +beamlines, if engraving isn't practical. **Confirm with whoever manages vacuum-material +compatibility at cSAXS before committing to a fabrication method for OMNY** -- this is a real +vacuum-safety call, not a software one. diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id0.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id0.png new file mode 100644 index 00000000..98ad3ba5 Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id0.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id1.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id1.png new file mode 100644 index 00000000..9e1601ec Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id1.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id2.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id2.png new file mode 100644 index 00000000..41305c81 Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id2.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id3.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id3.png new file mode 100644 index 00000000..5a0c2563 Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id3.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id4.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id4.png new file mode 100644 index 00000000..69fe0c09 Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id4.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id5.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id5.png new file mode 100644 index 00000000..61c2bf0d Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id5.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id6.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id6.png new file mode 100644 index 00000000..745ca398 Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id6.png differ diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id7.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id7.png new file mode 100644 index 00000000..484f6a86 Binary files /dev/null and b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id7.png differ