From 0a2f783e5ca3cd4a0c31ba2067df8c1d2a7a6412 Mon Sep 17 00:00:00 2001 From: Mirko Holler Date: Tue, 8 Sep 2026 11:19:08 -0300 Subject: [PATCH] docs(vision): 8 printable ArUco tags, vacuum-material note, session handoff summary Generates and commits the actual printable marker files (DICT_4X4_50, ids 0-7, 780x780px with a 15% white quiet-zone margin) into csaxs_bec/devices/vision/fiducial_markers/, with their own README documenting the regeneration recipe and current id assignment (0/1 = flomni gripper left/ right jaw, confirmed distinct rather than the same id on both sides; 2-7 generated but unassigned, since OMNY has multiple camera views/sites of the same gripper and a single tag per jaw face may not be visible from every angle). All eight round-trip-verified through detect_aruco_markers() before committing, including after aggressive downscale+blur. Records the open, not-yet-decided question for OMNY specifically: it's a vacuum environment, so flomni's plain paper+glue plan doesn't carry over -- laser engraving vs. Kapton vs. another vacuum-safe method needs sign-off from whoever manages vacuum-material compatibility at cSAXS, not a software-session call. Also adds a "Status as of 2026-09-08 -- start here" section at the top of GRIPPER_LOCALIZATION_PLAN.md, since the user is closing this session and picking the work back up later -- summarizes what's real/committed vs. still open without needing to read the full chronological log first. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_015fDXnPuNvm3qUsUb4V4Gfo --- .../omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md | 87 ++++++++++++++---- .../devices/vision/fiducial_markers/README.md | 60 ++++++++++++ .../fiducial_markers/aruco_4x4_50_id0.png | Bin 0 -> 6039 bytes .../fiducial_markers/aruco_4x4_50_id1.png | Bin 0 -> 5716 bytes .../fiducial_markers/aruco_4x4_50_id2.png | Bin 0 -> 5814 bytes .../fiducial_markers/aruco_4x4_50_id3.png | Bin 0 -> 6017 bytes .../fiducial_markers/aruco_4x4_50_id4.png | Bin 0 -> 6039 bytes .../fiducial_markers/aruco_4x4_50_id5.png | Bin 0 -> 6039 bytes .../fiducial_markers/aruco_4x4_50_id6.png | Bin 0 -> 5814 bytes .../fiducial_markers/aruco_4x4_50_id7.png | Bin 0 -> 5582 bytes 10 files changed, 131 insertions(+), 16 deletions(-) create mode 100644 csaxs_bec/devices/vision/fiducial_markers/README.md create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id0.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id1.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id2.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id3.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id4.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id5.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id6.png create mode 100644 csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id7.png diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md index 51489df5..9559ff2e 100644 --- a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md @@ -1,10 +1,44 @@ -# Gripper-relative localization for machine-vision confirms — exploration plan (2026-09-07) +# Gripper-relative localization for machine-vision confirms — exploration plan (2026-09-07, updated 2026-09-08) -Exploration only -- no production code changed in this pass (a few throwaway interactive scripts -were run against real captures, not committed). Grows out of `MACHINE_VISION_AUTOCONFIRM_PLAN.md`'s -Phase 0 and this session's edge-detection tuning work (`vision_toolkit.detect_edges()`'s new -`blur_ksize`, `VisionInterface.set_edge_params()`, the `VisionTuning` GUI widget) -- written so a -future session can pick this up without re-deriving it. +Grew out of `MACHINE_VISION_AUTOCONFIRM_PLAN.md`'s Phase 0 and (originally) a same-session +edge-detection tuning pass (`vision_toolkit.detect_edges()`'s `blur_ksize`, +`VisionInterface.set_edge_params()`, the `VisionTuning` GUI widget -- all committed, unrelated to +gripper localization itself, still useful as-is). Unlike when this doc started, real production code +has since landed here too (`find_dark_holes()`, `detect_aruco_markers()`, the `VisionTuning` +`"holes"` mode) -- read on for what's real/tested vs. still open. + +## Status as of 2026-09-08 -- start here in a fresh session + +**Current direction: fiducial markers (ArUco), not trying to recognize the gripper itself.** +Two prior approaches were tried and both hit real, understood dead ends against real photos (not +guessed) -- see the dated updates below for the full evidence trail: +- Edge detection (Canny, any threshold/blur) never finds a contour anywhere near the gripper's + actual size -- it has almost no internal gradient for Canny to find at all (reflective metal). +- The gripper's jaw *notch* segments cleanly as a `find_dark_holes()` "hole" -- but only when it + physically exists: with the pin fully seated, the notch/gap simply isn't there. Not a detection + bug, a real state-dependence that rules out the notch as the sole anchor. + +**What's real and committed right now**: +- `vision_toolkit.find_dark_holes(image, area_range)` -- Otsu + inverted `RETR_CCOMP` hole search, + tested, works well *when the notch exists*. +- `vision_toolkit.detect_aruco_markers(image, dictionary_name)` -- generic ArUco detection, tested + against synthetic markers only (no real marked gripper photo exists yet). +- `VisionInterface`/`VisionTuning` both have a live `"holes"` mode to watch `find_dark_holes()` + results interactively (Device → Camera → Mode=`holes` → set area min/max → Apply). +- `csaxs_bec/devices/vision/fiducial_markers/` -- 8 printable ArUco tags (`DICT_4X4_50`, ids 0-7), + ids 0/1 assigned (flomni gripper left/right jaw), 2-7 free for OMNY's other views. See that + folder's own `README.md`. + +**What's next, concretely, once physical tags exist** (nothing below this point can proceed without +a real photo of the gripper with tags actually mounted -- that's on Mirko, not blocked on code): +1. Confirm `detect_aruco_markers()` finds the mounted tags on a real capture (lighting/print quality + could differ from the synthetic round-trip already verified). +2. Define the pin-search region in tag-relative coordinates instead of a fixed frame ROI. +3. Re-run `find_dark_holes()`/`detect_edges()` for the pin check itself, now inside that small, + reliably-anchored region instead of an unconstrained, shifting frame. +4. **Open, not yet decided**: OMNY's vacuum-compatible fabrication method (laser engraving vs. + Kapton vs. something else) -- needs sign-off from whoever manages vacuum-material compatibility + at cSAXS, not a call to make in a coding session. ## Why edge-threshold tuning alone isn't the fix @@ -215,16 +249,37 @@ environment's OpenCV 5.0 build), so a scratch essentially never false-triggers o returns the 4 corner points directly -- from a known physical tag size that's position, in-plane rotation, *and* scale in one step, with no custom geometry code to get right. -**Chosen spec**: `DICT_4X4_50`, marker id `0` for the left jaw, id `1` for the right jaw -- printed -as large as each jaw's flat outer face allows (bigger is more robust to glare/low local contrast, -don't minimize it), on matte (not glossy) material, mounted with a bit of white quiet-zone margin -left around the black border (don't cut flush to the marker edge), kept flat (no wrinkling/doming, -which would corrupt the corner-based pose estimate). Two tags, not one: redundancy if one is -occluded from some angle, and the distance between them becomes a free signal for jaw open/closed -state later. Generated marker images (`DICT_4X4_50`, ids 0/1, with a 15%-of-side white margin) were -produced and print-verified this session (round-tripped through `cv2.aruco` detection correctly, -including after aggressive downscale+blur simulating a small tag seen from a distance) -- see -`detect_aruco_markers()` below for how to regenerate them. +**Chosen spec**: `DICT_4X4_50`, marker id `0` for the left jaw, id `1` for the right jaw (confirmed, +not left/right symmetric duplicates of the same id -- distinct ids make each detection +self-identifying regardless of where it lands in frame, rather than relying on position alone to +tell them apart) -- printed as large as each jaw's flat outer face allows (bigger is more robust to +glare/low local contrast, don't minimize it; mounting doesn't need to cover the whole jaw, any +consistently-visible flat spot works), on matte (not glossy) material, mounted with a bit of white +quiet-zone margin left around the black border (don't cut flush to the marker edge), kept flat (no +wrinkling/doming, which would corrupt the corner-based pose estimate). + +**Printable marker files: `csaxs_bec/devices/vision/fiducial_markers/` (committed -- small, +deterministic, reproducible, not real capture data, unlike `reference_images/`).** Eight markers +generated (ids 0-7, `DICT_4X4_50`, 780x780px, 15%-of-side white margin), not just the two currently +assigned -- OMNY has several camera views/sites of the same physical gripper (parking/shuttle pair +vs. sample-stage pair, per `VisionInterface`'s own docstring, possibly others), and a single tag per +jaw face may not be visible from every angle it's viewed from there. Ids 2-7 are generated and +available, not yet assigned -- see that folder's own `README.md` for the regeneration recipe and +the current id-assignment table; update it, don't let it drift out of sync with this doc, once +OMNY's actual mounting positions are decided. All eight round-trip-verified through +`vision_toolkit.detect_aruco_markers()` this session, including after aggressive downscale (to +60x60) + blur simulating a small tag seen from a distance. + +**Material -- flomni vs. OMNY differ, and this is a real open decision, not yet made.** flomni is +not a vacuum environment (per the captures already in `reference_images/`) -- plain paper + glue, as +originally planned, is fine there. **OMNY is vacuum**: paper, most glues, and most tapes outgas and +are not vacuum-safe as-is. Two candidate fabrication methods, neither committed to yet: (a) +laser-engrave/mark the pattern directly onto the metal jaw -- no adhesive, nothing to outgas, +permanent (an established technique, "Direct Part Marking"); (b) a known low-outgassing material +like Kapton (polyimide) film/tape, already familiar at synchrotron beamlines, if engraving isn't +practical. **Whoever manages vacuum-material compatibility at cSAXS needs to confirm before +committing** -- this is a real vacuum-safety call, outside what a software session should decide on +its own. **New Layer 0 primitive, already implemented and tested (not waiting on the physical markers)**: `vision_toolkit.detect_aruco_markers(image, dictionary_name="DICT_4X4_50") -> list[ArucoDetection]` diff --git a/csaxs_bec/devices/vision/fiducial_markers/README.md b/csaxs_bec/devices/vision/fiducial_markers/README.md new file mode 100644 index 00000000..7069a258 --- /dev/null +++ b/csaxs_bec/devices/vision/fiducial_markers/README.md @@ -0,0 +1,60 @@ +# Gripper fiducial markers (ArUco, DICT_4X4_50) + +Printable ArUco markers for the gripper-localization approach in +`../../bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md` -- see that doc for +the full reasoning (why fiducials over trying to recognize the gripper itself, why ArUco over +hand-drawn dots). Detected client-side by `vision_toolkit.detect_aruco_markers()`. + +## What's here + +`aruco_4x4_50_id0.png` .. `aruco_4x4_50_id7.png` -- eight markers, `DICT_4X4_50`, each 780x780 px +(a 600px marker + a 90px/15% white quiet-zone margin on every side). Fully reproducible from the +dictionary name + id -- these files are a convenience (so they don't need regenerating/reprinting +from scratch each time), not the source of truth: + +```python +import cv2 +import cv2.aruco as aruco + +dictionary = aruco.getPredefinedDictionary(aruco.DICT_4X4_50) +marker = aruco.generateImageMarker(dictionary, marker_id, 600) # marker_id: 0-7 here +bordered = cv2.copyMakeBorder(marker, 90, 90, 90, 90, cv2.BORDER_CONSTANT, value=255) +cv2.imwrite(f"aruco_4x4_50_id{marker_id}.png", bordered) +``` + +Round-trip-verified this session: `vision_toolkit.detect_aruco_markers()` correctly decodes each +file back to its own id, including after aggressive downscale (to 60x60) + Gaussian blur simulating +a small tag seen from a distance. + +## Current id assignment + +- **id 0** -- flomni gripper, left jaw. +- **id 1** -- flomni gripper, right jaw. +- **ids 2-7** -- generated and available, not yet assigned. OMNY has several camera views/sites of + the same physical gripper (per Mirko: parking/shuttle pair vs. sample-stage pair, others + possibly), and a single tag per jaw face may not be visible from every angle the gripper is + viewed from there -- extra ids are here so additional tags (different faces/orientations of the + same gripper, or a second gripper) can be assigned without regenerating anything. Update this + list once OMNY's actual mounting positions are decided -- don't let it go stale. + +## Printing + +Print at whatever physical size the target flat mounting face allows -- bigger is more robust to +glare/low local contrast, don't minimize it. These PNGs are high enough resolution (780px) to print +sharply from a few mm up to several cm; set the *physical* print size explicitly (e.g. insert into a +word processor doc, set the image object's size in mm, print at 100% scale) rather than "fit to +page", which won't give a controlled physical size. + +Leave the white quiet-zone margin intact when trimming -- don't cut flush to the marker's black +border, detection relies on that contrast margin. Mount flat (no wrinkling/doming), which would +distort the corner positions the pose/scale estimate is computed from. + +**Material -- flomni vs. OMNY differ.** flomni is not a vacuum environment (per the captures in +`../reference_images/`): plain paper + glue, as originally planned, is fine. **OMNY is vacuum** -- +paper, most glues, and most tapes outgas and are not vacuum-safe. Options, not yet decided: +laser-engraving/marking the pattern directly onto the metal jaw (no adhesive, nothing to outgas, +permanent -- an established technique, usually called Direct Part Marking); or a known +low-outgassing material like Kapton (polyimide) film/tape, already familiar at synchrotron +beamlines, if engraving isn't practical. **Confirm with whoever manages vacuum-material +compatibility at cSAXS before committing to a fabrication method for OMNY** -- this is a real +vacuum-safety call, not a software one. diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id0.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id0.png new file mode 100644 index 0000000000000000000000000000000000000000..98ad3ba52d4843c3b1cf2268d9baa7a4cf236310 GIT binary patch literal 6039 zcmeAS@N?(olHy`uVBq!ia0y~yVCDf~4xj+TZISxr3=HCNo-U3d6^w5WZscWl6ky&M z_}}$(0lWAg74C_e#-FDf8P|_eWCX!9gU@@DJU6z?d@t(VrFy1jeMFE>;SLFNwl=gmf>Hj#zdk~bs z9%%1-pl!|)4@`JX>wh$@zavlsN|2+9M?+&YZH(p=Qi}p}k@ydyKm;sg?gZ2v3IqtQXLff_R$9S0_N4BS4rbWfI>dT1(W#NE@?&t;uc GLK6U7QFcE7 literal 0 HcmV?d00001 diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id1.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id1.png new file mode 100644 index 0000000000000000000000000000000000000000..9e1601ec9d748d51c89a715b0d04db03866482f3 GIT binary patch literal 5716 zcmeAS@N?(olHy`uVBq!ia0y~yVCDf~4xj+TZISxr3=Cq@o-U3d6^w5WHgYmM3NUXh z{O@XA!*4CX=dEe{*>mPh|5Imbg3_K%HZq<+W#-Q%XU?dnrrC{3VF`^c)iZarr-^va z*cq;)InAJWeMFE>;^b)V%Qcqxne{?7 zprUP`(SMa6pwg`-^`GYlPysh;&}bNqrXwn3kspVoyYwM>>!|p>fEs9~-1njF5jP|g zerz=tiHBsmF8w_Ve?W6<%|qU!;*i|=h&x|nA0%6g##cChWQAnoQKNz7+-QJ+%J0z> z0;=Ffvkb7}29-9zVgy{{14|oFm_SkpIFW#w2BN^y23U-Mn*zYn22_lIs(w&u11zDy zZ3bXz11zDyZ3JLx11h0FZGuswL5-%-00FfsM^ngXmI0R`qaESVjxdco!lNUZBn{qv ZW?mW|k)_@%@*Ol>?CI*~vd$@?2>_dlvq%5{ literal 0 HcmV?d00001 diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id2.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id2.png new file mode 100644 index 0000000000000000000000000000000000000000..41305c81894bcca90b66c97c22feca1f44fb397f GIT binary patch literal 5814 zcmeAS@N?(olHy`uVBq!ia0y~yVCDf~4xj+TZISxr3=Cqmo-U3d6^w5WZscWl6ky&M z_}}$(0lWAg74C_e#-FDf8P|_eWCX!9gU@@DJU6z?d@t(VrFy1jeMFE>;ivk z#sfP+P3u98RdH~8byV?aXpE+f(VPNouaZ<0j1H=djwub)klN@lDY-+$_Fv>8vTCJF Qt3U(Np00i_>zopr077Fx-T(jq literal 0 HcmV?d00001 diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id3.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id3.png new file mode 100644 index 0000000000000000000000000000000000000000..5a0c2563c56555d23b27c4d7816503007f13ea6c GIT binary patch literal 6017 zcmeAS@N?(olHy`uVBq!ia0y~yVCDf~4xj+TZISxr3=HBPo-U3d6^w5WZscWl6ky&M z_}}&P4?dYYD%=w_jX!(Noauk+OifVQv&lxr^QX-Gx#Y|l_0%-GQ7J5;(WQE3Pq>ce zG=tA^8<#{JI#YdgQWB@}dr|LEiP0dTO=9@brGF=&<{&85J<#4);QWy_9+>o+*8e!f zdsMszl>YWT;x-qF{{Tvaql!mEV>E4y=9JN*V6F05)S1|AhkEF~CMl z>(6jd0~*+bc@_`Ks=x-!jCxQDW>nE=5R9gRfyfAX9Ofcmdq)|-EI1knqk%y0>@YgE nG&-tA$MM9`QD2e;(lyE$ w4TRB@Fc7(5bkt~c&}^W_3`fU-$sGf?U&FO0%R=$;EYOI%r>mdKI;Vst05{lgjsO4v literal 0 HcmV?d00001 diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id5.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id5.png new file mode 100644 index 0000000000000000000000000000000000000000..61c2bf0d393b29cd893c5685ae20aa26f2aec22d GIT binary patch literal 6039 zcmeAS@N?(olHy`uVBq!ia0y~yVCDf~4xj+TZISxr3=HCNo-U3d6^w5WZscWl6ky&M z_}}$(0lWAg74C_e#-FDf8P|_eWCX!9gU@@DJU6z?d@t(VrFy1jeMFE>;}o|EBdn4)Gop2j|mK#iOAynl?sr3b3{! zttcpRo)HeJ==Y5>Mgw6qB@9q5_#g_c=uhz;42MG7#t8C(ng3(c<(Lu9;8Z#Un2PSt6-2R5;o~$G7@r;N_#)L^zQ`J90Vn> z2ip4zoIkS00~21;`X7gQkBZlT66C1j(a;!88>2ae2Eu4c7>Hc( zc@?N02j;<1#%Lgnri1~>1*6s9Xd7U(AvCbLV02JrbWCZWhSWxfNy!}|wy)BR$ntpE R#S0pc_H^}gS?83{1OTa%+k^lB literal 0 HcmV?d00001 diff --git a/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id7.png b/csaxs_bec/devices/vision/fiducial_markers/aruco_4x4_50_id7.png new file mode 100644 index 0000000000000000000000000000000000000000..484f6a8664454c8574f843f1c398a8e3709020fc GIT binary patch literal 5582 zcmeAS@N?(olHy`uVBq!ia0y~yVCDf~4xj+TZISxr3=E=EJzX3_Dj44$Jjlsn$iTdD z=D+T58)cse9v0M*Ona6*V`hJ1nqAw>pAw@StYNUwHSHa@ZkyQ5o%&lOqEepiyO5MI zaprHQ?oo-+AfZiS_;pA+#jjP&x%*42`H4APz{K~GJAZf)eGv4{S4o7aqN!RQr(EXgMf>jv6!?Mx*J7;wV3(0Hu1b#s>?H(x^97qEH+ zwqA?1zMPZ>W#CbRM#E?{9Z{Y|Mu#j$2W#j!el