feat(vision): add find_dark_holes() -- validated gripper-notch localization primitive
CI for csaxs_bec / test (push) Successful in 2m6s

Neither Canny (any threshold/blur) nor whole-silhouette Otsu segmentation could
locate the gripper (documented in GRIPPER_LOCALIZATION_PLAN.md's earlier
updates), but the gripper's jaw *notch* -- a small dark gap between the jaw
tips, right where the pin sits -- segments cleanly as an Otsu "hole" (a dark
region with no path to the image border without crossing bright pixels).

find_dark_holes(image, area_range) implements this: Otsu threshold + inverted
RETR_CCOMP + area filter, plus an explicit border-touching exclusion (a bright
object's own silhouette is topologically indistinguishable from "a dark hole
enclosed by brightness" from the background's side, and would otherwise be
returned right alongside genuine holes -- caught by the tests, documented in
the docstring since area_range must always be scoped by the caller).

Validated against real ground truth: searching the full frame (no ROI) of
sample_transfer_reference_image_get_1_...h5, ranking dark-hole candidates by
Hu-moment shape distance against the notch's own shape gives a decisive, unique
match (score 0.002, next-best 0.05 -- not a close call). Does not yet
generalize to other captures in the same series where the gripper has visibly
moved (documented as an open next step, not silently dropped) -- two plausible
causes (fixed-ROI artifact, timestamp-overlay interference) were checked and
directly ruled out rather than assumed.

7 new tests for find_dark_holes() (hole detection, area-range filtering, the
border-exclusion behavior, sort order). Full suite: 397 passed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015fDXnPuNvm3qUsUb4V4Gfo
This commit is contained in:
Mirko HollerandClaude Sonnet 5 committed 2026-09-08 10:23:42 -03:00
1 parent 6e28cdef9a
commit 598c2c4759
3 files changed
+148

No files matched your search

@@ -158,6 +158,43 @@ gripper is a boundary-enclosed region.** Next things worth trying, in rough orde
pin ROI, feature matching, post-localization reference diffing) are unaffected by this update and
still apply once *some* localization technique above actually works.
## Update 2026-09-08 (continued): the notch alone, not the whole jaw silhouette, and a new Layer 0 primitive
Follow-up the same day: rather than chasing a closed silhouette for the whole gripper (jaws +
background bleed + glow, all the open problems above), target just the **notch** -- the small dark
gap between the jaw tips, right where the pin sits. It turned out to be the one part of this scene
that segments cleanly: Otsu-threshold + `cv2.RETR_CCOMP` finds it as a *hole* (a dark region with no
path to the image border without crossing bright pixels), no edge detection needed.
**This is now a real, tested function**: `vision_toolkit.find_dark_holes(image, area_range)` --
Otsu + inverted `RETR_CCOMP` + area filter + an explicit check excluding any region touching the
image border (the background itself would otherwise qualify as "a dark hole enclosed by
brightness," which is topologically true but not useful -- caught by the unit tests, see
`find_dark_holes`'s docstring for why `area_range` must always be scoped, never left unbounded).
**Validated on the one frame with real ground truth** (`get_1`, plus its near-duplicate
`get_sense`): searching every dark-hole candidate in the *full* frame (no ROI at all) and ranking by
Hu-moment shape distance (`contour_similarity_score()`) against the notch's own shape gives a
decisive, unique match -- best score 0.002-0.003, next-best candidate 0.05-0.24 (10-100x worse).
Not a close call; genuinely unambiguous.
**Does not yet generalize to the other captures in the same series** (`get_2`, `get_3`, `put_1`,
`put_2`, `put_3`) -- best matches there score only 0.14-0.55, no clear winner. Two plausible causes
were checked and ruled out directly rather than assumed:
- **Not a fixed-ROI artifact**: the search above used the *full* frame, no ROI restriction.
- **Not the burned-in timestamp/blue-marker overlay stealing the match**: excluding those regions
(tight or broad) changed nothing -- they were never among the top matches to begin with.
**Confirmed instead, visually**: the gripper genuinely shifts position and apparent orientation
between captures (`get_2`'s crop at the same coordinates used for `get_1` shows only the gripper's
edge, mostly out of frame) -- exactly what Mirko anticipated ("the shape shifts horizontally").
**Not yet determined**: whether the notch still forms a clean, fully-enclosed dark hole once the
gripper's moved (different local lighting/reflection at the new position could break the
enclosure), or whether it's present but outscored by something else. The concrete next step is
visual: dump `find_dark_holes()`'s raw (pre-ranking) candidate list for `get_2` next to the actual
photo and check by eye whether the true notch is even in the candidate set at all before assuming
the shape-matching step is what's failing.
## What this doesn't touch
`VisionInterface`'s edge-threshold/blur tuning (this session's other change) and the `VisionTuning`
@@ -233,6 +233,59 @@ def largest_contour(binary_image: np.ndarray) -> np.ndarray | None:
return max(contours, key=cv2.contourArea)
def find_dark_holes(
image: np.ndarray, area_range: tuple[float, float] = (0.0, math.inf)
) -> list[np.ndarray]:
"""Find small dark regions fully enclosed by a brighter surrounding area.
Otsu-thresholds the grayscale image, then finds every *hole* (a dark blob with no path to the
image border without crossing bright pixels first, via ``cv2.RETR_CCOMP``) in the resulting
binary mask, filtered to ``area_range`` -- the caller picks that range from whatever's
physically plausible for the specific feature being searched for.
Unlike ``detect_edges()``/``largest_contour()`` (gradient-based, and only as good as the
object's own internal texture/contrast), this looks for enclosure by *brightness* instead --
e.g. it reliably and uniquely found a real gripper's jaw notch (bright metal jaws on both
sides, a real photo, not synthetic) in one real capture, where Canny found nothing anywhere
near the notch's actual size at any threshold/blur combination tried. Not yet verified to
generalize across every gripper position/lighting -- e.g. the notch didn't cleanly separate
from surrounding bright regions in every capture tried, and two known artifacts (a burned-in
timestamp overlay, a fixed corner marker) can register as small dark-hole candidates of their
own if not excluded by the caller first (e.g. via :func:`crop_to_roi`, or filtering results by
expected position). See ``AI_docs/GRIPPER_LOCALIZATION_PLAN.md`` for the ongoing exploration
and its caveats -- this is one validated building block, not a finished gripper locator.
``area_range`` isn't just a convenience filter -- a caller should always scope it to the
expected feature size, never leave it fully unbounded. Any bright foreground object's own
silhouette is topologically indistinguishable from "a dark region enclosed by brightness" (from
the background's side, the object *is* a hole) and will be returned right alongside genuine
holes if its area happens to fall in range.
Returns every matching hole contour, in image coordinates, largest-area first -- an empty list,
not an error, if nothing in ``area_range`` was found (a caller should always check for that, the
same way ``best_reference_match()`` treats an empty reference set as an expected state).
"""
gray = to_grayscale(image)
image_height, image_width = gray.shape
_, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
inverted = cv2.bitwise_not(binary)
contours, _ = cv2.findContours(inverted, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
low, high = area_range
matches = []
for contour in contours:
if not (low <= cv2.contourArea(contour) <= high):
continue
# A region touching the image border isn't actually enclosed -- it's whatever's outside
# the object, not a hole inside it (this is what the *background* itself looks like here,
# not a real candidate).
x, y, w, h = cv2.boundingRect(contour)
if x <= 0 or y <= 0 or x + w >= image_width or y + h >= image_height:
continue
matches.append(contour)
matches.sort(key=cv2.contourArea, reverse=True)
return matches
def contour_similarity_score(contour_a: np.ndarray | None, contour_b: np.ndarray | None) -> float:
"""Shape-distance between two contours (lower = more similar); +inf if either is missing.
@@ -1,5 +1,7 @@
"""Unit tests for the Layer-0 vision toolkit (csaxs_bec/devices/vision/vision_toolkit.py)."""
import math
import cv2
import numpy as np
import pytest
@@ -103,6 +105,62 @@ def test_detect_edges_blur_suppresses_fine_texture_noise():
assert blurred_edges.sum() < unblurred_edges.sum()
def _bright_square_with_hole(
size: int = 100, square_side: int = 60, hole_side: int = 10
) -> np.ndarray:
"""A bright filled square (on a dark background) with a small dark hole punched through its
center -- e.g. a bright metal part with a notch/gap, the case find_dark_holes() targets."""
image = vt.make_square_test_pattern(size=size, square_side=square_side)
center = size // 2
half = hole_side // 2
image[center - half : center + half, center - half : center + half] = 0
return image
def test_find_dark_holes_finds_a_hole_enclosed_in_a_bright_region():
image = _bright_square_with_hole(hole_side=10)
# area_range scoped to the punched hole's own size, same as any real caller would (per the
# docstring) -- an unbounded search legitimately also returns the square's own silhouette
# (itself a dark region fully enclosed by the brighter background around it, topologically no
# different from a hole from find_dark_holes()'s point of view), which is why this isn't
# tested with the default unbounded area_range.
holes = vt.find_dark_holes(image, area_range=(50, 200))
assert len(holes) == 1
area = cv2.contourArea(holes[0])
assert 80 <= area <= 120 # ~10x10, allowing for a little contour-boundary slack
def test_find_dark_holes_area_range_filters_out_the_hole():
image = _bright_square_with_hole(hole_side=10)
assert vt.find_dark_holes(image, area_range=(500, 1000)) == []
def test_find_dark_holes_no_holes_in_range_returns_empty_list():
image = _square_image() # a plain filled square, no punched-out hole
# Scoped away from the square's own (much larger) silhouette -- see the comment above.
assert vt.find_dark_holes(image, area_range=(1, 200)) == []
def test_find_dark_holes_sorted_largest_first():
image = vt.make_square_test_pattern(size=200, square_side=180)
image[40:50, 40:50] = 0 # small hole, area ~100
image[100:130, 100:130] = 0 # bigger hole, area ~900
holes = vt.find_dark_holes(image, area_range=(50, 1000))
assert len(holes) == 2
areas = [cv2.contourArea(h) for h in holes]
assert areas == sorted(areas, reverse=True)
def test_find_dark_holes_excludes_the_unbounded_background():
# The background itself (touching every image border) is never a valid "hole" candidate,
# regardless of area_range -- it isn't actually enclosed by anything.
image = _square_image(size=100, square_side=30)
holes = vt.find_dark_holes(image, area_range=(0, math.inf))
for hole in holes:
x, y, w, h = cv2.boundingRect(hole)
assert x > 0 and y > 0 and x + w < 100 and y + h < 100
def test_register_to_reference_identity_for_identical_images():
image = _square_image()
result = vt.register_to_reference(image, image)