resolve_lazy() only unwraps a Proxy (checks for __resolved__/
__wrapped__); handed a bare NamespaceComponent instead - which this
property built directly rather than receiving as a replace_Namespace
Components-processed constructor argument, unlike pgroup/checker/
run_table - it just returned that same unresolved instance unchanged.
Every real call site (start_scan_monitoring/end_scan_monitoring) got a
NamespaceComponent with no start_monitoring/stop_monitoring of its own,
raising AttributeError on every scan - caught and silently turned into
"status server: start monitoring failed ... falling back" by
_status_server_failed, so the server-side path this session's status-
server work was meant to exercise has never actually run from a real
scan. Fix: call .get() ourselves before resolve_lazy().
tests/test_daq_status_server.py's FakeStatusServerComponent stood in
directly for the already-resolved object and had no .get() of its own,
relying on the old code never calling it - added one (returns self).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A 24h soak test (repeated start/stop cycles against the real bernina
namespace, ~11,500 channels/recording) showed the status server's RSS
climbing steadily and never coming back down, even though every
recording was correctly stopped and dropped. Root cause: per-channel
buffers stored values as plain Python lists of boxed floats - millions
of individually heap-allocated objects per recording. CPython's
small-object allocator frees these correctly but does not reliably
hand the underlying memory back to the OS, so RSS ratchets up by
roughly one recording's worth of data every cycle.
_append now stores a channel's values in an array.array('d') where
possible - raw C doubles, allocated/freed via malloc/free directly, so
the memory actually returns to the OS. Falls back to a plain list, in
place, the first time a channel's value isn't array.array-compatible
(enum/string PVs, or a waveform value size that slipped past
max_value_elements) - checked via isinstance, so a normal scalar
channel never pays for it and a non-numeric one only pays it once, not
on every update. stop() still hands back plain lists as before (every
existing caller - tests, the /recording/stop JSON response,
write_monitor_recording - already expects that; array.array fails a
bare `== [...]` comparison), so this is purely an internal
representation change during a recording's own lifetime, not a
contract change.
Isolated a competing hypothesis (the async attach loop's per-recording
ThreadPoolExecutor+CA-context-attach) and ruled it out first: 60 cycles
of attach/detach against 5 real PVs showed flat RSS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A full-namespace, unfiltered 10-minute recording is 570-790 MB, mostly
from a handful of waveform/digitizer channels - too big for a small
local /tmp and unbounded over a long --repeat-for-hours run. Adds a
--max-value-elements passthrough to drop large values at the source,
and --keep-last to prune older run<NNNN> output directories so total
disk usage stays bounded regardless of how long the loop runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Repeats back-to-back start/stop recording cycles (one per simulated
scan) for a total wall-clock budget instead of one long continuous
recording, closer to what a real sequence of scans looks like. A single
iteration's failure is logged and skipped rather than ending the whole
soak test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every classic PSI/GFA elog post (via the ElogsMultiplexer's elog_gfa
backend) was failing with a confusing LogbookInvalidMessageID, traced
live: getDefaultElogInstance() defaults the elog username to the
current OS user (whoami), but the gopass-stored password
(bernina/elog/elog-password) belongs to the shared gac-bernina
account -- so a real password was always paired with the wrong
username, and the server's login-page response was never recognized
as an auth failure because the installed elog package's `type=password`
check doesn't match this server's quoted HTML. Pin user="gac-bernina"
explicitly for elog_gfa; this classic elog is expected to be
deprecated soon in favor of scilog, so not worth a deeper fix.
Also fixes an unrelated bug found along the way in
Assembly.status_to_elog: self.alias.get_full_name was missing its
call parens, so every status-attachment filename embedded a raw bound
method repr (with a memory address) instead of the assembly's name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that _launch_live_parse_cache points the cache writer at the
writable res/ tree (previous commit), try it by default when starting
an eco session from this personal checkout, rather than requiring
bernina.daq.live_parse_cache = True to be set manually every session.
Still a no-op against any escape-fel predating daq_cache_writer, and
still live-toggleable. Not yet the production default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
escape-fel's daq_cache_writer subprocess was finding scan_info_rel.json
via its default raw/-first search order and failing every poll with
OSError: [Errno 30] Read-only file system, since raw/ is read-only for
this account (found via a real live_parse_cache test on
saresb-vcons-03, run 80). Now always pass --metadata-file pointing at
the writable res/run_data/daq/run<NNNN>/aux/scan_info_rel.json copy
(same convention _write_status_locally already uses), and
--broker-address-aux so the child uploads its cache file into the
run's raw/aux/ afterward -- the same copy_user_files call append_aux
makes for every other DAQ-client-written aux file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via a real scan on saresb-vcons-03 (pgroup p22891, run 79):
status.json/scan_info_rel.json/aliases.json/namespace_monitor.ixp.h5
all correctly landed in run0079/aux/ (the real run directory - p22891's
run0001..run0079 have all been zero-padded since 2025), but
scan_monitor.pkl landed in a separate, spurious run79/aux/ instead -
missing the :04d format spec every other write site in this file uses
(copy_scan_info_to_raw, append_start_status_to_scan,
append_status_to_scan_and_store all correctly zero-pad). The write
itself "succeeded" since ensure_dir/open_group_writable don't validate
against a real run directory, just create whatever path they're given
- so this was silent, and (per matching unpadded run8/run9 directories
dated Nov 2025) not new today.
Regression test drives end_scan_monitors directly (Daq.__new__, mocked
ensure_dir/open_group_writable/append_aux, no real filesystem access) -
confirmed it fails against the pre-fix code and passes against the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by running a real scan against a real bs stream on
saresb-vcons-03 (a test eco.status_server + a real BsStreamCounter),
then reading last_arrays back afterward, mirroring exactly what a user
would do interactively - arr._scan_parameter raised deep inside h5py
("invalid identifier type to function").
_store_arrays() copied CounterValue.store_arrays's reload-then-close
pattern verbatim ("re-open read-back handles so last_arrays keeps
working after the write file closes"), but escape.Array's own dataset
wrapper (ArrayH5Dataset) is lazy for at least some attributes even
past load_from_result_file (default lazy_loading=False notwithstanding)
- so the *reloaded* handles this assigned into self.last_arrays were
themselves tied to a second h5py.File that gets closed two lines later,
just as unusable as what it was replacing.
There was never a need to reload in the first place: the arrays
already in self.last_arrays (from _build_arrays) are plain in-memory,
numpy-backed Array objects with no file dependency at all. Dropped the
reload entirely; verified against the real recorded data (real scan
parameter values, correct shape) both immediately and via a fresh
DataSet.load_from_result_file() in a separate process.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live, against the real test server on saresb-cons-05: a fresh
full-namespace recording only attached to 18 of 13366 channels, yet the
report showed thousands of channels already "with data" - a real epics.PV
is not hashable (no __hash__), so it can never be a dict key, weak or
otherwise. _auto_monitor_refs[self.pv]/.get(self.pv) raised `TypeError:
unhashable type: 'PV'` on every real PV - but only *after*
pv.add_callback() had already registered the actual callback (which was
already collecting real data). RecordingSession's per-channel
`except Exception` then counted the TypeError as an ordinary attach
failure, and since the exception meant self._monitors[name] was never set
either, stop() never reached that callback - a leaked, permanently-running
subscription per channel, for nearly every channel.
The unit tests didn't catch this because the fake PV they use is a plain
object, hashable by identity by default - nothing about it exercised the
real constraint. Fixed by keying _auto_monitor_refs on id(self.pv) instead
(a plain dict, not weak: safe here because every CallbackEpics using a
given PV holds a live strong reference to it for as long as its entry
exists, and the entry is removed the moment the count reaches zero, so an
id() already in the dict is never a stale/reused one) - the actual pyepics
PV was never the problem, WeakKeyDictionary's hashability requirement was.
Also made FakePV explicitly unhashable (__hash__ = None) so the test suite
exercises this constraint for real going forward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Optional, off by default: launches escape-fel's upcoming live parse-cache
warmer (`python -m escape.swissfel.live_reduce --mode daq-cache`) as a
genuine child process once per scan, right where copy_scan_info_to_raw
already knows scan_info_rel.json's path - see escape-fel's own
"Integrating live parse-cache warming into the DAQ client" doc for what the
subprocess itself does (polls for newly-complete steps, calls
load_dataset_from_scan with checknstore_parsing_result="same_directory" to
warm the cache next to scan_info_rel.json, exits once the run goes quiet).
Feature-gated on the actual implementing attribute
(escape.swissfel.live_reduce.daq_cache_writer), not just module presence,
so turning this on ahead of the escape-fel release that adds it is a
harmless no-op - checked at launch time, not at Daq construction, so it
starts working the moment escape-fel is upgraded without needing the flag
re-toggled. At most one child per scan (copy_scan_info_to_raw fires once
per step plus once at scan end, not just once) - reuses a still-running
one, relaunches only if the previous one already exited. A plain
subprocess.Popen of *this* process on purpose, per the doc's own warning:
the child needs to inherit whatever account this Daq/session runs as,
since a different account (even in the same pgroup) can fail to write the
cache file into aux/ outright.
Also: bernina.status_server now points at ECO_STATUS_SERVER's URL when one
is set (the same env var Daq already reads), instead of always the
production default - lets a session point both Daq and the inspectable
status_server object at a test server (e.g.
ECO_STATUS_SERVER=http://saresb-vcons-03:8092) without touching production.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Multiple RecordingSessions could already start concurrently (start_recording
only refuses reusing the same recording_id), and each already gets its own
independent copy of every update - pyepics' PV.add_callback() natively
supports several simultaneous callbacks per PV. The one real bug this
exposed: CallbackEpics.start()/stop() used to snapshot pv.auto_monitor at
start and blindly restore it at stop, correct for exactly one CallbackEpics
per PV, wrong for two - whichever stopped first could clobber the
auto_monitor a still-running sibling needs (another overlapping recording,
or the status server's own permanent baseline monitor, which is already a
second CallbackEpics on top of any single recording today). Fixed with a
plain per-PV reference count in eco.epics_utils.utilities_epics: only the
first attacher changes auto_monitor, only the last detacher restores it -
stop() order no longer matters.
Also: a recording that starts while another is already running now
backfills its own buffer for any shared channel from whatever the earlier
one already collected (RecordingSession._seed_buffer_from_siblings) - a
one-time list copy at start(), truncated to the new session's own
max_points_per_channel, not a live share. So the later recording's data for
the overlap window comes from the earlier one instead of starting empty.
Ending the earlier recording (stop + backgrounded write/upload, already a
daemon thread, not blocking) leaves the later one's own monitors/buffers
completely untouched - each session only ever manages its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plain .h5 doesn't distinguish this file's format (one escape.ArrayTimestamps
per channel, see NAMESPACE_MONITOR_FORMAT.md) from any other HDF5 file in
aux/. filename is an opaque string end to end (capture_recording ->
/recording/capture -> write_monitor_recording just does directory /
filename, h5py doesn't care about the name) so this is a pure rename, no
behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the only way to stop using an already-configured status
server was setting the private _status_server = None directly - which
also throws away the configured URL, so re-enabling meant remembering
and restoring it by hand.
use_running_status_server (default True, matching current behavior)
is checked first thing in the status_client property - the single
choke point every status-server-backed callback in this class already
goes through (append_start_status_to_scan, start_scan_monitoring,
copy_aliases_to_scan, use_status_server(), ...), so gating it there
covers all of them for free. _status_server itself is left untouched,
so toggling this back to True brings back the same client, same
base_url, with nothing to re-supply.
Live-toggleable on an already-running session, no restart needed:
bernina.daq.use_running_status_server = False
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
start_scan_monitoring/end_scan_monitoring were unconditionally wired
into callbacks_start_scan/_end_scan with no way to turn them off short
of editing the callback lists directly. Needed now because
RecordingSession.start() attaches a CA monitor to every monitorable
channel in a single-threaded, unthreaded loop (namespace_store.py:1108),
and start_recording()'s HTTP call only gets the general
status_server_timeout (10s default) rather than the much larger
status_server_snapshot_timeout (180s) the analogous whole-namespace
snapshot/capture call gets -- reliably too slow at bernina's namespace
size, producing a "start monitoring failed ... falling back to the
local namespace mechanism" warning on every scan, easily mistaken for
the whole status-server integration being broken (it isn't -- that's
an unrelated, independently-succeeding call).
scan_monitoring defaults to True (unchanged behavior); set
bernina.daq.scan_monitoring = False on an already-running session for
an immediate, no-restart opt-out, or pass scan_monitoring=False at
construction. Checked first in both methods, before touching
status_client at all, so disabling it can't itself produce a warning.
Proper fix (making /recording/start async like /status/capture already
is) tracked separately, not done here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the remaining gap after the previous commit's start-time guard:
restart's non-systemd branch used to be an unconditional
"cmd_stop; cmd_start -b" - if nothing was running locally, cmd_stop was
a silent no-op and cmd_start went ahead and started a brand-new local
instance regardless of whether the real one was already up elsewhere
(exactly how the cons-03/04/05 crosstalk happened - cmd_start's own new
guard already covers a plain `start`, but `restart` reached the same
code with different framing and needed its own check before getting
there, not just relying on the guard to reject it after the fact).
New cmd_restart(): local pid found -> stop+start in place (unchanged
behavior for the normal case); none found but $URL is healthy
elsewhere -> respawn that one instead (self-re-exec over HTTP, not a
new local process); nothing running anywhere -> start fresh here, same
as before. managed_by_systemd path unchanged.
Verified by stubbing server_pid/remote_duplicate_running/cmd_stop/
cmd_start/cmd_respawn and confirming cmd_restart dispatches to exactly
the right one for each of the three cases - not against any real host.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Direct fix for a real incident (2026-09-16): three independent
eco.status_server instances ended up running simultaneously across
saresb-cons-03/04/05, each holding its own silently-drifting in-memory
namespace, discovered only by manually sweeping every console host.
cmd_start now checks remote_duplicate_running() before starting: if
$URL (almost always the site default from the env file, e.g.
saresb-cons-04) resolves to a *different* host than the one `start` is
running on, and that host's /health reports ready, refuse to start a
second one here. Deliberately does NOT block starting elsewhere when
$URL's host is actually down/unreachable - a genuine "the real one is
gone, stand up a replacement" case is not the mistake this guards
against. No shared lock file: the config directory isn't group-writable
across every account that might run this (same class of problem
eco_cnf_bernina/memory/ once hit), and $URL/health is already the right
source of truth, so there is no extra state to keep in sync or clean up
after a crash. -f/--force skips the check for the rare legitimate case.
Verified against local fake HTTP servers (different-host+healthy ->
refused; same-host -> allowed without even checking; different-host+
unreachable -> allowed) - never against any of the three real hosts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
restart/stop/start only ever manage a process on the local host
(pgrep/systemctl are inherently local) -- pointing $URL at a server
running elsewhere, as a different account, does nothing useful there:
it just finds no local process and starts a brand-new one right here.
respawn instead POSTs /admin/restart to $URL over HTTP, same as the
GUI's "Restart process" button (client.reinit(mode="restart") ->
client.restart()) -- the server re-execs *itself* (os.execv, same
PID/user/host it already runs as), so this correctly reaches the real
process regardless of where the CLI is invoked from. -w polls /health
afterward until it reports ready with a new instance_id, mirroring
StatusServerClient.restart(wait=True)'s own timing.
Verified against a local fake HTTP server (no-wait, --wait, and the
connection-refused path) -- not against the real one, deliberately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scans.acquiring_scan.* (daq_run_number, initial_values, ...) previously
only ever reached the status server via push_status(), which is
job-read-only (GET /status/job/<id>?include_status=1) -- never merged
into the actual status.json written to disk (store.snapshot() is
purely Alias.get_all()-driven, structurally blind to anything without
a CA channel, which is everything under a scan/Daq object).
Daq._client_status_for_scan(scan) now builds {"status": {...},
"status_channels": {...}, "status_times": {...}} from scan.get_status()
and self.get_status() (both already exactly this shape), keys prefixed
"scans."/"daq.", and hands it to _server_status_for_scan ->
_capture_on_server -> StatusServerClient.capture(client_status=...).
Daq's own get_status() is currently empty (nothing appended to its own
status_collection yet) -- included anyway so anything added there later
flows through with no further wiring.
Server-side, POST /status/capture merges client_status into its own
snapshot (status/status_channels/status_times, additively -- never
replacing the server's own alias-driven entries) before
write_status_snapshot, so it lands in the actual file. Timing is
already correct without reordering anything:
count_run_number_up_and_attach_to_scan (which sets scan.daq_run_number)
already runs before append_start_status_to_scan, both scan.get_status()
sources are populated in StepScan.__init__ itself, and the merge point
is inside the very same capture request already made -- no second
round trip.
Best-effort throughout (client_status=None falls back to exactly the
prior server-only behavior; a failure building it is caught and
printed, never raised).
Verified end-to-end against the real Flask app (two new tests: the
merge landing in the written status.json, and omitting client_status
leaving it unchanged) plus the full existing status-server/daq-client
suites (188 passed) -- one pre-existing test's exact-kwargs assertion
updated for the new always-present client_status=None fallback value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives BsStreamCounter the other thing it was missing compared to the
older EPICS-monitor CounterValue (which already auto-creates and
stores arrays every scan): a scan's per-channel bins used to live only
in memory (last_value(s), .grid(), the raw _channels), gone the moment
the next scan tore them down, with nothing written to disk.
_build_arrays() converts self._last_channels into real escape.Array
objects via Stream.to_array() (escape-fel, previous commit), with the
parameter axis relabeled to the scan's actual adjustable values
(scan.scan_info["scan_values"], same source
eco.acquisition.counters.parameter_from_scan uses) instead of
to_array()'s own default -- this counter's internal Scan is keyed on a
synthetic step index (see _StepIndexSource), not anything physical.
Falls back to the synthetic step_index for a channel whose bin count
doesn't match the scan's step count (a step that timed out with zero
samples never created a bin) rather than mislabeling steps.
_store_arrays() writes them to a fresh .esc.h5 via
DataSet.create_with_new_result_file/.append/.store(), mirroring
CounterValue.store_arrays's own round-trip. Both run from
_on_scan_end, gated by a new store=True constructor flag (storage_dir=
"auto" picks the directory, same contract as CounterValue's own);
any failure is printed, never raised, so it can't fail the scan.
Verified directly against a synthetic bs stream (escape.stream's
TestStream/LocalEventHandler, no real beamline) and added two
regression tests: the parameter/step_lengths values survive a full
store+reload round-trip, and store=False writes nothing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StepScan.scan_all()'s end-of-scan confirmation (inputimeout.inputimeout()
for return_at_end="timeout", plain input() for "question") blocked the
calling thread -- normally the interactive session's own thread, which
also owns any open live-plot window's GUI event loop -- for the whole
wait, freezing every such window for up to the 10s timeout after every
single scan. Same class of bug as the per-step live-plot freeze fixed
earlier via Acquisition's on_tick.
Added _inputimeout_nonblocking()/_input_nonblocking(): run the actual
read in a background thread and poll it with short joins from the
calling thread instead of one blocking call, flushing every open
matplotlib figure's GUI events between polls. Verified directly
(mocked inputimeout to control timing): both the timeout and
answer-arrives paths return the same result as before, with GUI
flushes actually happening during the wait rather than only after it.
Also passes BsStreamCounter's own name as label= into
Stream.plot_med() (escape-fel) so its live plot's title/y-axis show
e.g. "mon_opt.intensity" instead of the raw bs channel it subscribes
to ("SAROP21-PBPS133:INTENSITY").
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per user feedback on the ipymagic Qt-modal `...` fallback:
a) The picker window now has two tabs -- Components (the namespace tree,
as before) and a new Commands tab (eco.logs recent/frequent commands),
which previously only existed in the terminal Tab-completion overlay,
not the Qt fallback.
b) A pick no longer executes the current cell at all: _AstTransformer.
visit empties the cell's AST body and hands the expanded code to the
next prompt via IPython's set_next_input (the same mechanism %edit/
%load use) -- terminal and notebook/qtconsole alike. Verified live that
the cell body never runs (no assignment happens) and the next prompt is
pre-filled with the correct, directly re-executable expression.
c) In the picker, a single click only highlights a row now (tree, or a
Recent/Bookmark combo choice) -- it never fires a pick by itself. Only
Enter/Return, a double-click, or one shared Go button (below both tabs,
confirms whichever tab is active) commits a selection.
ComponentSelectorQt keeps its old click-to-select-immediately behavior
by default (require_confirm=False) since select_component/scan_
launcher_qt.py's existing picker dialog already depends on it --
confirm-gating is opt-in, used only by the new PickerQtWindow.
d) Dropped the "Value" tree column (and _format_value): get_current_value()
per row is a real, non-free call (EPICS or whatever) and the result is
stale the instant it's drawn anyway.
pick_component_modal now returns (kind, text, obj) instead of (path, obj)
-- kind is "component" (text relative to root, via _full_path as before)
or "command" (text used verbatim, never run through _full_path).
Added tests/test_component_selector_qt.py for the new confirm-gating/
CommandSelectorQt behavior, and extended tests/test_ipymagic.py for the
3-tuple contract and the cancel-and-set-next-input behavior. Verified live
via Xvfb: tabs render, a plain click doesn't pick, Enter/double-click and
the shared Go button do, and the full ipymagic path (pick -> next input,
no execution) works end to end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the hand-rolled Fit-only toolbar button with one call to
escape.plot_utilities.attach_escape_buttons(before_click=stop_animation),
so the plain EPICS-monitor live plot (CounterValue) gets the same Fit,
Peak, and Peak params (find_peak center/FWHM overlay) tools the
bs-stream live plots already have, instead of missing peak analysis
entirely. before_click stops the FuncAnimation before a button draws on
the axes -- otherwise the animation's own per-frame ax.cla() would wipe
whatever the button just drew.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per user feedback: terminal-ness alone shouldn't skip the Qt fallback --
only an actually unavailable display should. Replaces _in_real_terminal()
with _has_display() (QApplication.instance() already up, or DISPLAY/
WAYLAND_DISPLAY set), checked before ever constructing a QApplication --
confirmed live that doing so with no display is an uncatchable native
abort (SIGABRT), not a Python exception, so it has to be avoided rather
than caught.
Verified live with a synthetic root (bernina's own startup is currently
broken by an unrelated in-progress edit to eco/elements/adj_obj.py from
another session, so this bypasses that): no-display prints the hint and
leaves '...' in place without crashing; DISPLAY set via Xvfb opens the
real Qt picker and completes the pick as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous fix called flush_events() inside BsStreamCounter's
_wait_for_n_new() -- but that loop runs on a background PropagatingThread
whenever it's reached through acquire() (Acquisition is built with
hold=False there, which starts the thread immediately), not on the
scan's calling/main thread. Pumping a Qt event loop from any thread but
the one that owns it is unsafe and doesn't actually get the live plot
repainted; meanwhile the *main* thread -- the one actually blocked, and
the one whose GUI event loop needs servicing -- was sitting in a plain
Acquisition.wait() -> thread.join(), untouched.
Fixed by giving Acquisition an optional on_tick callback: wait() now
polls the acquisition thread in short joins instead of one blocking
join(), calling on_tick() after each -- on the *calling* thread, which
is where it belongs. BsStreamCounter.acquire() passes its
_flush_live_plot as on_tick.
This exposed a real, independent bug in PropagatingThread.join(): it
unconditionally read self.exc/self.ret after joining, which only exist
once the thread's run() has actually completed -- a timed join that
returns before the thread finishes raised AttributeError. Fixed by
checking is_alive() first and returning None (mirroring a plain
Thread.join(timeout=...) not having anything to report yet) instead of
touching attributes that aren't there yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1. startup_inline.py's ipymagic.start(eval(scope)) passed the bare package
*module* (import eco.bernina as bernina gives a plain module, no
.name, no status_collection) instead of the real eco.utilities.config.
Namespace instance every device is actually registered onto. Confirmed
live: every picked path fell back to a literal, undefined "root."
prefix (NameError on use), and the tree/completions looked sparse and
near-random -- build_tree() on a module has no children at all, so
what did show up was only leftover Recent/Recommended entries from
unrelated earlier use, not an actual namespace walk. Fixed by using
eco.{scope}.namespace instead, with a warned fallback to the old
behavior for a scope that doesn't follow that convention.
2. The Qt-modal fallback (a `...` that reaches Enter unexpanded) was
popping a Qt window even from a plain terminal session -- surprising,
and against the point of the Tab-completion overlay being the intended
terminal-native path. Now skipped in a real terminal (detected via
isinstance(get_ipython(), TerminalInteractiveShell)), printing a hint
toward Tab instead; unchanged in a notebook/qtconsole/script, where
there's no equivalent terminal-native alternative.
Both verified against the real facility entry-point chain (run
startup_inline_new.py -l -s bernina): Tab-completion now returns real,
correctly root-prefixed deep paths instead of scattered leftovers, and a
bare `...` at Enter in the terminal now prints the hint and raises a
plain TypeError on use instead of a confusing NameError on "root".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the terminal-native picker: typing the trigger and pressing Tab pops
IPython's own completion menu (same widget, works in a real terminal or a
notebook -- ipykernel's completion goes through the same IPCompleter
matcher pipeline) instead of the Qt modal. Two triggers:
"..." component mode: Recent, Recommended (FrequencyCounts-ranked),
then a live substring-filterable walk of the namespace tree.
"...h" command mode (mnemonic: history): Recent/Recommended prior
*commands* (eco.logs.recent_commands/frequent_commands) -- no
namespace section, nothing to browse there.
Selecting an entry only ever inserts text at the cursor -- it never
executes, in either mode -- so the rest of the line can still be composed/
edited before submitting. This composes with the existing Qt-modal `...`
mechanism rather than conflicting with it: accepting a completion replaces
`...` in the buffer before the line is ever submitted, so if you don't use
Tab and just hit Enter, the AST-transform fallback still applies to
whatever literal `...` is left.
Real bug found while building this and wiring it through a live
IPCompleter (not just unit tests): eco.widgets.component_selector's
ComponentNode.name (and so pick_component_modal's return value) is
relative to `root` (confirmed against Alias.get_full_name(base=...), which
stops at base) -- not a standalone, globally-evaluable expression the way
this module had assumed everywhere. Both the Qt-modal path (_EllipsisPicker)
and this new completer were splicing that relative fragment in directly,
silently dropping the root's own name from the printed/inserted code.
Fixed with _full_path(root, relative_path), which reattaches the name
`root` is actually bound to (its own `.name`, matching this codebase's
`import eco.bernina as bernina` convention) -- verified against a real
IPCompleter.complete() call, not just the mocked unit tests, which is
exactly what surfaced the bug in the first place (a hand-fed "already
correct" path in the earlier Qt smoke test had been masking it).
Also fixed the smoke-test script that had encoded the same wrong
assumption, for anyone re-running it later.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The command-history data sources for eco.ipymagic's command-picker mode
("...h") -- merged across every kernel-log session on disk (same source
eco.logs.widget()'s timeline already reads), deduplicated by keeping each
distinct command's most-recent occurrence (recent_commands) or ranking by
use count with a recency tie-break (frequent_commands), rather than
letting a command run every few minutes crowd out everything else.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RecentComponents is pure move-to-front recency; "Recommended" needs a
different signal -- something used constantly but not in the last hour
should still surface even after newer one-off components have pushed it
out of Recent. Same AdjustableFS-persisted-json, per-user pattern as
RecentComponents (mirrors it deliberately rather than sharing a base
class -- the value shapes and update semantics genuinely differ: a
deduped list you move to front, vs. a dict of counts you increment).
Wired into the same touch_from_write() chokepoint RecentComponents
already uses (every real adjustable write, via Changer), so Recommended
gets populated from real usage automatically, same as Recent -- no
separate hook needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A scan runs as one long synchronous call on the calling thread -- the
same thread an interactive Qt/ipympl session's GUI event loop normally
runs on. Nothing services that event loop while a scan is in progress,
so a live counter plot (escape.stream.Stream.plot_med) sat unpainted
(often literally black -- the canvas never got its first real paint
event) for the whole scan, and its redraw QTimer never fired either;
both only caught up once the scan call returned.
_wait_for_n_new() already loops in ~50ms slices waiting for new data,
so flush the plot figure's pending GUI events there every iteration --
costs nothing extra and lets the plot actually update live during a
running scan instead of only at the very end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
eco/startup_inline.py is what eco_cli.py/scripts/eco-dev and (via
startup_inline_new.py's shim) the real /sf/bernina/bin wrappers all run,
so it's the right place for this to be on by default rather than opt-in
-- a plain library import (from eco import bernina) still has to ask for
it explicitly. Root comes from `scope`, the same object already bound by
this file's own `import eco.{scope} as {scope}` a few lines up.
Verified live through the actual wrapper chain (run startup_inline_new.py
-l -s bernina): ipymagic installed on the real shell with bernina as root.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
from eco import bernina
from eco import ipymagic
ipymagic.start(bernina)
In [1]: (1 / ...).plot()
(1 / bernina.mono.energy).plot()
<plot appears>
In [2]: cen = (left := ...) - (right := ...)
cen = (left := bernina.mono.energy) - (right := bernina.attenuator.transmission)
A bare `...` (the builtin Ellipsis literal -- not an IPython special form,
so this works anywhere IPython's own ast_transformers hook runs: terminal,
notebook/JupyterLab, qtconsole) is replaced, before execution, by the
picked component's real dotted path -- the expanded code is printed, then
run, so what actually executed stays visible and re-typeable. Multiple
`...` in one statement resolve left to right. A `...` used as a numpy
subscript (`arr[...]`) is left alone. Reuses eco.widgets.component_selector
for the actual tree/fold-state/Recent-list machinery -- nothing new there.
New eco.widgets.component_selector_qt.pick_component_modal(): the existing
ComponentSelectorQtWindow is built for embedding/async display (start()
hands off to IPython's own Qt pumping, or run() blocks the whole process
forever); an AST transform needs an actual answer before it can even
return control to IPython, so this drives the same window with its own
nested QEventLoop that quits the instant a pick (or a close) happens.
Also fixed a real bug found while testing this live: IPython's
ast_transformers calls `.visit(node)` on each registered entry (expects
an ast.NodeTransformer-shaped object) -- a bare function registered there
throws 'function' object has no attribute 'visit' once, and IPython
*silently* unregisters it with only a warning, no visible error. Verified
end-to-end against a real Qt window on a real X11 display (Xvfb) plus a
real IPython shell: spawned the picker, drove a real selection via
QTimer, confirmed the printed/executed code and result were both correct.
Deliberately namespace-tree-only for now -- picking from prior *commands*
(eco.logs' kernel history) is a planned second mode, not built here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For "from eco import bernina" with no startup script involved at all --
the same per-session JSONL activity log every interactive entry point
already gets (eco.widgets.kernel_registry.install_shell_logger), exposed
as one explicit opt-in call instead of only happening as a side effect of
going through a startup script:
from eco import bernina
from eco import globallogger
globallogger.start()
Deliberately opt-in, not automatic on `import eco` -- a plain library
import (a short script, a notebook cell) shouldn't get a
~/.eco/kernel_logs/*.jsonl file appearing on disk just from importing.
Defaults the session label to the caller's resolved identity (eco.
elements.access.current_identity) rather than a generic tag, so two
people on a shared account still end up with distinguishable logs -- the
same "global user based" reasoning eco.elements.recent already applies to
the per-user Recent-components list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lemke_h
