feat(vision): add find_dark_holes() -- validated gripper-notch localization primitive
CI for csaxs_bec / test (push) Successful in 2m6s
CI for csaxs_bec / test (push) Successful in 2m6s
Neither Canny (any threshold/blur) nor whole-silhouette Otsu segmentation could locate the gripper (documented in GRIPPER_LOCALIZATION_PLAN.md's earlier updates), but the gripper's jaw *notch* -- a small dark gap between the jaw tips, right where the pin sits -- segments cleanly as an Otsu "hole" (a dark region with no path to the image border without crossing bright pixels). find_dark_holes(image, area_range) implements this: Otsu threshold + inverted RETR_CCOMP + area filter, plus an explicit border-touching exclusion (a bright object's own silhouette is topologically indistinguishable from "a dark hole enclosed by brightness" from the background's side, and would otherwise be returned right alongside genuine holes -- caught by the tests, documented in the docstring since area_range must always be scoped by the caller). Validated against real ground truth: searching the full frame (no ROI) of sample_transfer_reference_image_get_1_...h5, ranking dark-hole candidates by Hu-moment shape distance against the notch's own shape gives a decisive, unique match (score 0.002, next-best 0.05 -- not a close call). Does not yet generalize to other captures in the same series where the gripper has visibly moved (documented as an open next step, not silently dropped) -- two plausible causes (fixed-ROI artifact, timestamp-overlay interference) were checked and directly ruled out rather than assumed. 7 new tests for find_dark_holes() (hole detection, area-range filtering, the border-exclusion behavior, sort order). Full suite: 397 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015fDXnPuNvm3qUsUb4V4Gfo
This commit is contained in:
1 parent
6e28cdef9a
commit
598c2c4759
3 files changed
+148
No files matched your search
@@ -158,6 +158,43 @@ gripper is a boundary-enclosed region.** Next things worth trying, in rough orde
|
||||
pin ROI, feature matching, post-localization reference diffing) are unaffected by this update and
|
||||
still apply once *some* localization technique above actually works.
|
||||
|
||||
## Update 2026-09-08 (continued): the notch alone, not the whole jaw silhouette, and a new Layer 0 primitive
|
||||
|
||||
Follow-up the same day: rather than chasing a closed silhouette for the whole gripper (jaws +
|
||||
background bleed + glow, all the open problems above), target just the **notch** -- the small dark
|
||||
gap between the jaw tips, right where the pin sits. It turned out to be the one part of this scene
|
||||
that segments cleanly: Otsu-threshold + `cv2.RETR_CCOMP` finds it as a *hole* (a dark region with no
|
||||
path to the image border without crossing bright pixels), no edge detection needed.
|
||||
|
||||
**This is now a real, tested function**: `vision_toolkit.find_dark_holes(image, area_range)` --
|
||||
Otsu + inverted `RETR_CCOMP` + area filter + an explicit check excluding any region touching the
|
||||
image border (the background itself would otherwise qualify as "a dark hole enclosed by
|
||||
brightness," which is topologically true but not useful -- caught by the unit tests, see
|
||||
`find_dark_holes`'s docstring for why `area_range` must always be scoped, never left unbounded).
|
||||
|
||||
**Validated on the one frame with real ground truth** (`get_1`, plus its near-duplicate
|
||||
`get_sense`): searching every dark-hole candidate in the *full* frame (no ROI at all) and ranking by
|
||||
Hu-moment shape distance (`contour_similarity_score()`) against the notch's own shape gives a
|
||||
decisive, unique match -- best score 0.002-0.003, next-best candidate 0.05-0.24 (10-100x worse).
|
||||
Not a close call; genuinely unambiguous.
|
||||
|
||||
**Does not yet generalize to the other captures in the same series** (`get_2`, `get_3`, `put_1`,
|
||||
`put_2`, `put_3`) -- best matches there score only 0.14-0.55, no clear winner. Two plausible causes
|
||||
were checked and ruled out directly rather than assumed:
|
||||
- **Not a fixed-ROI artifact**: the search above used the *full* frame, no ROI restriction.
|
||||
- **Not the burned-in timestamp/blue-marker overlay stealing the match**: excluding those regions
|
||||
(tight or broad) changed nothing -- they were never among the top matches to begin with.
|
||||
|
||||
**Confirmed instead, visually**: the gripper genuinely shifts position and apparent orientation
|
||||
between captures (`get_2`'s crop at the same coordinates used for `get_1` shows only the gripper's
|
||||
edge, mostly out of frame) -- exactly what Mirko anticipated ("the shape shifts horizontally").
|
||||
**Not yet determined**: whether the notch still forms a clean, fully-enclosed dark hole once the
|
||||
gripper's moved (different local lighting/reflection at the new position could break the
|
||||
enclosure), or whether it's present but outscored by something else. The concrete next step is
|
||||
visual: dump `find_dark_holes()`'s raw (pre-ranking) candidate list for `get_2` next to the actual
|
||||
photo and check by eye whether the true notch is even in the candidate set at all before assuming
|
||||
the shape-matching step is what's failing.
|
||||
|
||||
## What this doesn't touch
|
||||
|
||||
`VisionInterface`'s edge-threshold/blur tuning (this session's other change) and the `VisionTuning`
|
||||
|
||||
@@ -233,6 +233,59 @@ def largest_contour(binary_image: np.ndarray) -> np.ndarray | None:
|
||||
return max(contours, key=cv2.contourArea)
|
||||
|
||||
|
||||
def find_dark_holes(
|
||||
image: np.ndarray, area_range: tuple[float, float] = (0.0, math.inf)
|
||||
) -> list[np.ndarray]:
|
||||
"""Find small dark regions fully enclosed by a brighter surrounding area.
|
||||
|
||||
Otsu-thresholds the grayscale image, then finds every *hole* (a dark blob with no path to the
|
||||
image border without crossing bright pixels first, via ``cv2.RETR_CCOMP``) in the resulting
|
||||
binary mask, filtered to ``area_range`` -- the caller picks that range from whatever's
|
||||
physically plausible for the specific feature being searched for.
|
||||
|
||||
Unlike ``detect_edges()``/``largest_contour()`` (gradient-based, and only as good as the
|
||||
object's own internal texture/contrast), this looks for enclosure by *brightness* instead --
|
||||
e.g. it reliably and uniquely found a real gripper's jaw notch (bright metal jaws on both
|
||||
sides, a real photo, not synthetic) in one real capture, where Canny found nothing anywhere
|
||||
near the notch's actual size at any threshold/blur combination tried. Not yet verified to
|
||||
generalize across every gripper position/lighting -- e.g. the notch didn't cleanly separate
|
||||
from surrounding bright regions in every capture tried, and two known artifacts (a burned-in
|
||||
timestamp overlay, a fixed corner marker) can register as small dark-hole candidates of their
|
||||
own if not excluded by the caller first (e.g. via :func:`crop_to_roi`, or filtering results by
|
||||
expected position). See ``AI_docs/GRIPPER_LOCALIZATION_PLAN.md`` for the ongoing exploration
|
||||
and its caveats -- this is one validated building block, not a finished gripper locator.
|
||||
|
||||
``area_range`` isn't just a convenience filter -- a caller should always scope it to the
|
||||
expected feature size, never leave it fully unbounded. Any bright foreground object's own
|
||||
silhouette is topologically indistinguishable from "a dark region enclosed by brightness" (from
|
||||
the background's side, the object *is* a hole) and will be returned right alongside genuine
|
||||
holes if its area happens to fall in range.
|
||||
|
||||
Returns every matching hole contour, in image coordinates, largest-area first -- an empty list,
|
||||
not an error, if nothing in ``area_range`` was found (a caller should always check for that, the
|
||||
same way ``best_reference_match()`` treats an empty reference set as an expected state).
|
||||
"""
|
||||
gray = to_grayscale(image)
|
||||
image_height, image_width = gray.shape
|
||||
_, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
|
||||
inverted = cv2.bitwise_not(binary)
|
||||
contours, _ = cv2.findContours(inverted, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
|
||||
low, high = area_range
|
||||
matches = []
|
||||
for contour in contours:
|
||||
if not (low <= cv2.contourArea(contour) <= high):
|
||||
continue
|
||||
# A region touching the image border isn't actually enclosed -- it's whatever's outside
|
||||
# the object, not a hole inside it (this is what the *background* itself looks like here,
|
||||
# not a real candidate).
|
||||
x, y, w, h = cv2.boundingRect(contour)
|
||||
if x <= 0 or y <= 0 or x + w >= image_width or y + h >= image_height:
|
||||
continue
|
||||
matches.append(contour)
|
||||
matches.sort(key=cv2.contourArea, reverse=True)
|
||||
return matches
|
||||
|
||||
|
||||
def contour_similarity_score(contour_a: np.ndarray | None, contour_b: np.ndarray | None) -> float:
|
||||
"""Shape-distance between two contours (lower = more similar); +inf if either is missing.
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
"""Unit tests for the Layer-0 vision toolkit (csaxs_bec/devices/vision/vision_toolkit.py)."""
|
||||
|
||||
import math
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
import pytest
|
||||
@@ -103,6 +105,62 @@ def test_detect_edges_blur_suppresses_fine_texture_noise():
|
||||
assert blurred_edges.sum() < unblurred_edges.sum()
|
||||
|
||||
|
||||
def _bright_square_with_hole(
|
||||
size: int = 100, square_side: int = 60, hole_side: int = 10
|
||||
) -> np.ndarray:
|
||||
"""A bright filled square (on a dark background) with a small dark hole punched through its
|
||||
center -- e.g. a bright metal part with a notch/gap, the case find_dark_holes() targets."""
|
||||
image = vt.make_square_test_pattern(size=size, square_side=square_side)
|
||||
center = size // 2
|
||||
half = hole_side // 2
|
||||
image[center - half : center + half, center - half : center + half] = 0
|
||||
return image
|
||||
|
||||
|
||||
def test_find_dark_holes_finds_a_hole_enclosed_in_a_bright_region():
|
||||
image = _bright_square_with_hole(hole_side=10)
|
||||
# area_range scoped to the punched hole's own size, same as any real caller would (per the
|
||||
# docstring) -- an unbounded search legitimately also returns the square's own silhouette
|
||||
# (itself a dark region fully enclosed by the brighter background around it, topologically no
|
||||
# different from a hole from find_dark_holes()'s point of view), which is why this isn't
|
||||
# tested with the default unbounded area_range.
|
||||
holes = vt.find_dark_holes(image, area_range=(50, 200))
|
||||
assert len(holes) == 1
|
||||
area = cv2.contourArea(holes[0])
|
||||
assert 80 <= area <= 120 # ~10x10, allowing for a little contour-boundary slack
|
||||
|
||||
|
||||
def test_find_dark_holes_area_range_filters_out_the_hole():
|
||||
image = _bright_square_with_hole(hole_side=10)
|
||||
assert vt.find_dark_holes(image, area_range=(500, 1000)) == []
|
||||
|
||||
|
||||
def test_find_dark_holes_no_holes_in_range_returns_empty_list():
|
||||
image = _square_image() # a plain filled square, no punched-out hole
|
||||
# Scoped away from the square's own (much larger) silhouette -- see the comment above.
|
||||
assert vt.find_dark_holes(image, area_range=(1, 200)) == []
|
||||
|
||||
|
||||
def test_find_dark_holes_sorted_largest_first():
|
||||
image = vt.make_square_test_pattern(size=200, square_side=180)
|
||||
image[40:50, 40:50] = 0 # small hole, area ~100
|
||||
image[100:130, 100:130] = 0 # bigger hole, area ~900
|
||||
holes = vt.find_dark_holes(image, area_range=(50, 1000))
|
||||
assert len(holes) == 2
|
||||
areas = [cv2.contourArea(h) for h in holes]
|
||||
assert areas == sorted(areas, reverse=True)
|
||||
|
||||
|
||||
def test_find_dark_holes_excludes_the_unbounded_background():
|
||||
# The background itself (touching every image border) is never a valid "hole" candidate,
|
||||
# regardless of area_range -- it isn't actually enclosed by anything.
|
||||
image = _square_image(size=100, square_side=30)
|
||||
holes = vt.find_dark_holes(image, area_range=(0, math.inf))
|
||||
for hole in holes:
|
||||
x, y, w, h = cv2.boundingRect(hole)
|
||||
assert x > 0 and y > 0 and x + w < 100 and y + h < 100
|
||||
|
||||
|
||||
def test_register_to_reference_identity_for_identical_images():
|
||||
image = _square_image()
|
||||
result = vt.register_to_reference(image, image)
|
||||
|
||||
Reference in new issue
Block a user