Real incident, 2026-09-06: SLAAR21-LMOT-M521/M524:MOTOR_1.RBV (ordinary
delay-stage motor readbacks) got demoted mid-scan after a single 5s sweep
measured ~14 Hz during that step's move/settle burst. Demotion is
persisted to disk, so a channel that is idle except during moves stayed
wrongly stuck on the slow, direct-CA-get path afterwards.
_sweep_once now requires AUTO_MONITOR_MAX_RATE to be exceeded for
AUTO_MONITOR_SUSTAINED_SWEEPS consecutive sweeps (default 3, ~15s) before
demoting, tracked per-channel in _fast_streak and reset the moment a sweep
sees the channel back at a normal rate - a burst confined to one scan step
no longer counts as evidence of a genuinely fast channel. Also adds
forget_channel(pvname) to undo a single already-mismeasured entry without
wiping the whole learned list via clear_fast_channels().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root-cause investigation of a real scan abort (p19734 run 1146,
TimeoutError in Daq.get_pulse_id):
- ca_tuning._sweep_loop() never called epics.ca.use_initial_context()
before demoting channels (pv.auto_monitor = False / remove_callback,
which calls ca_clear_subscription()) - the same CA-context class of
bug this codebase already fixed once for real (see
namespace_store._ca_thread, parallel_init). Fixed, with tests.
- Daq.get_pulse_id() defaulted its wait budget to self.timeout, which
doubles as the broker HTTP request timeout (2s) - a transient ~3.4s
CA gap had no room to resolve. Split into a dedicated
pulse_id_timeout (5s default) so the two can be tuned independently.
- get_pulse_id()'s TimeoutError was a bare "timeout hit" with no
evidence of why. Added ca_tuning.describe_channel() (factored out of
report_none_read's existing connected/history/CA-cache logic) and
wired it into both TimeoutError sites, so a future occurrence
self-diagnoses instead of requiring log archaeology.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lemke_handClaude Sonnet 5