bec_widgets/HEATMAP_DATA_API_AUDIT.md

# Heatmap DataAPI draft — audit notes (`draft/heatmap_data_api`)

*Date: 2026-07-28. Audited head: `820afb20` ("wip"). Companion library audit with the full
54-finding registry: `bec_data_api/DATA_API_AUDIT.md` (widget findings appear there with the
same evidence and status fields). Fix commit on this branch: `08200a05`. Nothing was pushed.*

**Reading guide.** The sections up to "Known remaining gaps" audit the **original draft**
(`820afb20`) and keep their line references to that code. Everything from
"Status after the widget migration" onwards records what was built afterwards: the draft's
approach was validated, then rewritten onto the DataAPI v2 contract and generalized to all six
plotting widgets. The bec-side counterpart is `bec_data_api/DATA_API_AUDIT.md` §4.

This draft was assessed as an **integration probe** for the bec-side DataAPI, per its known
"rough" status. Verdict: the shape is right — a live buffered subscription feeding
`update_plot` through a Qt signal is exactly the pattern the DataAPI should serve — but the
draft had one blocking defect that came from the library (mid-scan `add_device`
misalignment froze the plot; fixed on the bec side, commit `d89e9ae9` there) and several
integration defects of its own, four of which are fixed here.

## What was wrong, and what changed in `08200a05`

1. **Setup order could permanently freeze the plot (library defect, widget trigger).**
   `_setup_data_api_subscription` adds x, y, z one by one after `set_callback`; on a scan with
   N recorded points the old library emitted N x-only bundles and then permanently paired
   x point N+k with y/z point k, so the widget's length check skipped every update for the rest
   of the scan. Proven by probe (`x (7) / y (2) / z (2), widget length check passes: False`);
   the probe passes against the fixed library (all series length 7, zero misaligned pairs).
   No widget change needed — but note the new library semantic: adding a device to an active
   bundle re-emits the aligned history, and buffered mode restarts its accumulation.

2. **Legacy fallback was gated on subscription existence, not health.** A live rebind can
   fail per device (e.g. z neither monitored nor async in the new scan — the ValueError is
   swallowed inside the scan-status dispatch); x/y stayed subscribed, z stayed dead, and both
   the DataAPI path (z=None → skip) and the legacy path (disabled because `_data_subscription
   is not None`) produced nothing: a permanently blank heatmap. Fixed with
   `DataSubscription.unbound_devices` (bec commit `f6cd80fa`) + `_data_api_feed_healthy()`:
   the legacy path stays active whenever any device is unbound.
   Test: `test_heatmap_unhealthy_subscription_keeps_legacy_updates`.

3. **A failed setup leaked a half-configured live subscription.** `self._data_subscription`
   was assigned only after all `add_device` calls, so the `except` branch's cleanup closed
   nothing; the orphan (strongly holding `data_api_update.emit`) would rebind on every scan and
   emit forever. Fixed by assigning immediately after `create_subscription`.
   Test: `test_heatmap_failed_subscription_setup_closes_partial_subscription`.

4. **No rate limit on the DataAPI feed.** The legacy path is throttled by a 5 Hz
   `pg.SignalProxy`, but `data_api_update` was connected straight to `update_plot`: one full
   O(N) buffer walk + grid rebuild per point on the GUI thread (measured library-side: 60.4 s
   of buffer-walking alone for a 36000-point scan). Fixed by routing the feed through its own
   5 Hz SignalProxy delivering only the newest payload — lossless, because buffered payloads
   carry the full state.

5. **Stale payloads could redraw over new content.** The buffered branch ignored the payload's
   `scan_id` (and shadowed its `metadata` parameter), so an emission queued on the Qt event
   loop before a scan switch or a history-view entry redrew over the new scan/history. Fixed:
   buffered payloads are dropped when the subscription is gone, a history view is active, or
   `metadata["scan_id"]` mismatches; the shadowing local was renamed.
   Test: `test_heatmap_update_plot_drops_stale_data_api_payload`.

## Known remaining gaps (as of the draft; all closed since — see the next section)

These were consequences of DataAPI scope, recorded in the library audit's recommendations:

- **The legacy fetch machinery must stay** until the DataAPI covers history scans and
  device-readback fallbacks; property-change refreshes (`sync_signal_update`) still perform the
  full legacy fetch even while the feed is active, and both writers share `_grid_index`.
- **Array-valued async z fragments** pass the length check but crash the scalar grid assignment
  (`ValueError: setting an array element with a sequence`, swallowed by SafeSlot) — the buffered
  contract needs an explicit "scalar per aligned point" guarantee or a defined flattening.
- **End-of-scan settle updates are a no-op** on both paths: the `QTimer.singleShot(...,
  self.update_plot)` fallback is skipped by `@SafeSlot(verify_sender=True)` (sender is None),
  and the DataAPI path has no scan-end reconciliation. Pre-existing on main; call it with
  `_override_slot_params={"verify_sender": False}` and add an end-of-scan handler when the
  DataAPI grows a scan-closed notification.
- `_setup_data_api_subscription` is only reached via `plot()`; a widget restored from config
  without a `plot()` call never uses the DataAPI.
- The widget tests still fake the DataAPI; one integration-style test driving a real
  `BECLiveDataPlugin` emission into `update_plot` would tie the payload shapes together
  (the library-side alignment tests currently cover that contract).

## Status after the widget migration (2026-07-29)

The gaps above were the reason this draft could not be shipped as-is. All of them were closed by
the v2 redesign plus the widget migration; the draft's own code no longer exists.

| Gap from the section above | Resolution |
| --- | --- |
| Legacy fetch machinery must stay (no history/readback coverage) | `HistoryDataPlugin` and `DeviceStreamPlugin` (bec `a8448b05`, `63b87bf3`) cover terminal scans and scan-less device streams, so the heatmap port (`e9fe0518`) drives **live and history through the same `_on_data_update`**. At the time of the migration the legacy pull survived as an explicitly health-gated fallback (`_data_api_feed_healthy`); it was **removed entirely afterwards** together with the last legacy data paths of all plot widgets — feed health is surfaced through `bridge.healthy`, and there is deliberately no second data-access path. |
| Array-valued async z fragments crash grid rendering | The columnar contract makes the shape explicit per source (`SourceData.values` + `metadata["async_update_type"]`), and `update.aligned()` guarantees equal-length columns, so the widget no longer infers structure from a buffered list. |
| End-of-scan settle updates are a no-op | Deleted along with the `QTimer.singleShot` hacks in every port. The backend performs a final flush at scan end and re-routes the same live subscription to the authoritative file when the scan-history entry appears (`a8448b05`), which is the real end-of-scan reconciliation the hack was approximating. |
| `_setup_data_api_subscription` only reachable via `plot()` | Closed afterwards: a widget restored from a saved configuration now starts the data feed itself (`_start_data_feed`), without requiring a `plot()` call; while idle it binds to the latest finished scan and switches to live-follow when a scan starts. |
| Widget tests fake the DataAPI; payload shapes untied | Partly closed: the bec side now has real-connector tests (`TestDeviceStreamsThroughRealConnector`) and real-message history fixtures. Widget suites still fake the bridge deliberately (they test rendering, not transport). The real-HDF5 gap is closed: the widget history fixtures write genuine HDF5 files (`create_history_file`, h5py) and the heatmap/waveform history tests read them through the history plugin end to end. |

### What the widgets look like now

All six plotting widgets — Heatmap (`e9fe0518`), ScatterWaveform (`47851dab`), Waveform
(`e0abc7f3`), Image (`235d5614`), MultiWaveform and MotorMap (`2288aa95`) — consume
`bec_widgets/utils/qt_data_subscription.py`:

```python
self._data_bridge = QtDataSubscription(
    self.client, sources=[(dev, entry), ...], scan="live",  # or a scan_id, or None for streams
    parent=self, min_emit_interval=self.update_interval_s,  # driven by the update_rate property
)
self._data_bridge.updated.connect(self._on_data_update)   # SubscriptionUpdate on the Qt thread
```

Deleted across those ports: three verbatim copies of the live/history fetch fork
(`_fetch_scan_data_and_access`), four scan-lifecycle handlers with their
`QTimer.singleShot(100/300)` settle hacks, two hand-rolled async reconstructors
(`on_async_readback`, `adjust_image_buffer`), three history resolvers' data paths, the per-widget
`pg.SignalProxy` **data** throttles (rate limiting is backend-side; presentation proxies stay),
and every direct dispatcher registration for `device_async_signal` / `device_preview` /
`device_monitor_1d` / `device_readback`.

Two things the migration got wrong and had to correct — both worth knowing before porting a
seventh widget:

- **Removing functionality is not the same as deferring it.** The Waveform port deleted the
  large-dataset guard and recorded it as a follow-up; it is critical functionality. It is back
  (bec `af11c666` + widgets `f57949ae`) with the original names, dialog and RPC surface, now
  gated backend-side and loading off-thread.
- **A fake bridge does not prove the transport works.** Three live-only failures (queued Qt
  delivery, pydantic `_StoredDataInfo`, pubsub `MessageObject` payloads) all passed the widget
  suites. See DATA_API_AUDIT.md §4.2.

## What a correct integration looks like (for the next widget)

Create one `QtDataSubscription` per widget configuration, declaring **all** sources at once
(the backend partitions them into correlation groups, so mixing monitored, async and standalone
sources in one call is fine); render from `update.aligned()` and `update.axis(...)` rather than
re-deriving lengths or x data; use `scan="live"` for the running scan, a scan id for history and
`scan=None` for device streams (with `max_points`); gate any remaining fallback on
`bridge.healthy`; and close the bridge in `cleanup()`. The signal bridge, Qt-thread marshalling,
stale-scan filtering, rate limiting and the size gate are all in the wrapper and the backend —
none of it should be re-implemented per widget.

## Post-migration addendum (2026-08-13)

Landed after the sections above were written; recorded here so this document matches the code:

- **Legacy sweep**: the health-gated legacy fallbacks were removed from all six widgets — the
  Data API is the only data path (see the corrected rows above).
- **`update_rate` on `PlotBase`** (1-100 Hz, per-widget defaults, RPC/Designer-visible): wired
  into every bridge through the backend's runtime-adjustable `min_emit_interval`.
- **Incremental async rendering** matching main's update rates, with a throughput benchmark
  (`tests/unit_tests/benchmarks/test_data_api_widget_throughput.py`).
- **Waveform large datasets**: min/max envelope decimation above one million points with
  zoom-window re-rendering down to raw samples, and a labelled progress bar below the plot fed
  by the backend's chunked-read `progress_callback` via `bridge.progress`.
- **Columns may be numpy arrays** (history reads); the widgets' source checks are
  length-based, never truthiness-based — regression-tested with numpy-backed fixtures.