From 639dfdae318d56988901cec7adab812a6e59be4e Mon Sep 17 00:00:00 2001 From: Mirko Holler Date: Tue, 8 Sep 2026 09:53:34 -0300 Subject: [PATCH] docs(vision): record real annotated-template tests -- brightness segmentation fails too Extracted real left-jaw/right-jaw/pin templates from Mirko's hand-annotated photo (reference_images/Screenshot ... .png + Slide1.png, both local-only like the other reference_images/ captures, not committed) and tested Otsu brightness segmentation against them -- the recommendation from the previous plan update. Result: also doesn't cleanly work, but now for a specific, understood reason rather than a guess. Canny never finds a contour anywhere near the jaws' real area at any tuning (confirms rather than assumes edge detection is the wrong primitive here). Otsu segmentation does separate the two jaw wedges and the notch between them sharply -- but the pin's own backlit glow bridges the base of both jaws into one blob, and the jaw tops fade into equally-bright background with no boundary. Excluding near-saturated pixels didn't break the bridge either. Recorded three concrete next avenues: color- rather than grayscale-based segmentation (the glow may differ in hue/saturation from reflective metal), explicitly masking the glow column (its position is predictable), and Hough-line-based wedge/notch detection instead of blob-based localization -- the last one also matches what Mirko's own annotation actually is: open boundary curves, not closed part outlines, because that's genuinely all that's cleanly visible. No production code changed -- exploration only. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_015fDXnPuNvm3qUsUb4V4Gfo --- .../omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md index 03bc455f..67585c16 100644 --- a/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md +++ b/csaxs_bec/bec_ipython_client/plugins/omny/AI_docs/GRIPPER_LOCALIZATION_PLAN.md @@ -103,6 +103,61 @@ implementation effort, consistent with `vision_toolkit.py`'s existing endstation (its own module docstring already frames it as shared by "any device or client-side code," OMNY and a possible future x-ray-eye-alignment consumer alike). +## Update 2026-09-08: real annotated ground truth, and brightness segmentation's own failure mode + +Mirko provided a real photo (closed gripper, pin loaded, only the pin's upper collar visible above +the receiving piece) with the true parts hand-traced: a left-jaw boundary (red), a right-jaw +boundary (orange), and the pin/collar (green) -- saved as +`reference_images/Screenshot 2026-09-08 at 09.42.44.png` (the photo) and +`reference_images/Slide1.png` (the annotated version, same frame scaled down ~3.73x with white +padding). This is real ground truth, not a guessed template -- worth using directly rather than +auto-deriving a template from noisy data again (recommendation 2 above, now actually done). + +**Extracted templates from the hand-drawn lines** (color-threshold each line, dilate ~9px into a +ribbon so `cv2.findContours` gets a closed shape, map Slide1's coordinates back to the full-res +photo via the measured scale/offset): `left_jaw` (area 31353, centroid (752,186)), `right_jaw` (area +31334, centroid (1076,186) -- near mirror-symmetric with left_jaw, as expected), `pin` (area 16874, +centroid (897,405), below and between both jaws). These numbers are real and reusable -- worth +keeping as an actual template file next time, not re-deriving them again from the PNGs. + +**Recommendation 1 (brightness/Otsu segmentation) was tried against these real templates and also +doesn't cleanly work, for a specific, now-understood reason.** Canny at every threshold/blur +combination tried only ever found contours up to ~5500px² in this frame -- an order of magnitude +below the ~17-31k template areas -- confirming (not just suspecting) that Canny never traces the +gripper's actual outline at all here, at any tuning. Otsu brightness thresholding does find the +jaws as bright regions, and visually looks promising (dumped to `/tmp/roi_binary2.png` during this +session, not committed) -- **the two jaw wedges and the small dark notch between them are sharply, +correctly separated from each other** in the binary image. But a single connected blob of ~150-217k +px² still results, because **the pin's own backlit glow forms a bright vertical column that bridges +the base of both jaws together**, and the top of both jaws also fades into equally-bright +out-of-focus background with no boundary between them. Excluding near-saturated (>250) pixels as a +separate "glare" class didn't break this bridge either -- the connecting glow isn't uniformly +saturated to 255 throughout. + +**So the real obstacle isn't "edges vs. brightness" as a technique choice -- it's that the pin's own +illumination (presumably deliberate backlighting for a different purpose) visually connects +everything bright in the frame into one blob, defeating both approaches' assumption that the +gripper is a boundary-enclosed region.** Next things worth trying, in rough order of effort: + +1. **Work in color, not grayscale, before segmenting.** `to_grayscale()` was applied immediately in + every attempt above -- the pin's glow may have a different color temperature (warmer/more + saturated yellow, per the raw photo) than the gripper's own metal reflection, in which case a + color-based mask (e.g. HSV saturation/hue, not just intensity) might separate "light source + glow" from "reflective metal surface" where plain brightness cannot. +2. **Explicitly model and mask out the glow column.** Its rough position (a vertical band directly + below the notch, between the jaws) is predictable across captures -- subtracting or excluding a + known glow-shaped region before segmenting might be enough to let the jaws separate on their own. +3. **Geometric/line-based detection instead of blob-based.** The Otsu binary shows each jaw's visible + boundary is fundamentally two near-straight converging edges (a wedge/V shape) -- `cv2.HoughLinesP` + to find line segments, then searching for a *pair* of lines whose angle/convergence matches the + known gripper geometry, sidesteps the "closed blob" assumption entirely (this also matches what + Mirko's own hand-drawn annotation actually is -- open boundary curves, not closed part outlines; + the annotator only traced what was genuinely visible as a clean edge, same constraint the + algorithm is hitting). +4. Steps 2-5 from the original recommendation list (deliberate template curation, gripper-relative + pin ROI, feature matching, post-localization reference diffing) are unaffected by this update and + still apply once *some* localization technique above actually works. + ## What this doesn't touch `VisionInterface`'s edge-threshold/blur tuning (this session's other change) and the `VisionTuning`