Repeats back-to-back start/stop recording cycles (one per simulated
scan) for a total wall-clock budget instead of one long continuous
recording, closer to what a real sequence of scans looks like. A single
iteration's failure is logged and skipped rather than ending the whole
soak test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every classic PSI/GFA elog post (via the ElogsMultiplexer's elog_gfa
backend) was failing with a confusing LogbookInvalidMessageID, traced
live: getDefaultElogInstance() defaults the elog username to the
current OS user (whoami), but the gopass-stored password
(bernina/elog/elog-password) belongs to the shared gac-bernina
account -- so a real password was always paired with the wrong
username, and the server's login-page response was never recognized
as an auth failure because the installed elog package's `type=password`
check doesn't match this server's quoted HTML. Pin user="gac-bernina"
explicitly for elog_gfa; this classic elog is expected to be
deprecated soon in favor of scilog, so not worth a deeper fix.
Also fixes an unrelated bug found along the way in
Assembly.status_to_elog: self.alias.get_full_name was missing its
call parens, so every status-attachment filename embedded a raw bound
method repr (with a memory address) instead of the assembly's name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that _launch_live_parse_cache points the cache writer at the
writable res/ tree (previous commit), try it by default when starting
an eco session from this personal checkout, rather than requiring
bernina.daq.live_parse_cache = True to be set manually every session.
Still a no-op against any escape-fel predating daq_cache_writer, and
still live-toggleable. Not yet the production default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
escape-fel's daq_cache_writer subprocess was finding scan_info_rel.json
via its default raw/-first search order and failing every poll with
OSError: [Errno 30] Read-only file system, since raw/ is read-only for
this account (found via a real live_parse_cache test on
saresb-vcons-03, run 80). Now always pass --metadata-file pointing at
the writable res/run_data/daq/run<NNNN>/aux/scan_info_rel.json copy
(same convention _write_status_locally already uses), and
--broker-address-aux so the child uploads its cache file into the
run's raw/aux/ afterward -- the same copy_user_files call append_aux
makes for every other DAQ-client-written aux file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via a real scan on saresb-vcons-03 (pgroup p22891, run 79):
status.json/scan_info_rel.json/aliases.json/namespace_monitor.ixp.h5
all correctly landed in run0079/aux/ (the real run directory - p22891's
run0001..run0079 have all been zero-padded since 2025), but
scan_monitor.pkl landed in a separate, spurious run79/aux/ instead -
missing the :04d format spec every other write site in this file uses
(copy_scan_info_to_raw, append_start_status_to_scan,
append_status_to_scan_and_store all correctly zero-pad). The write
itself "succeeded" since ensure_dir/open_group_writable don't validate
against a real run directory, just create whatever path they're given
- so this was silent, and (per matching unpadded run8/run9 directories
dated Nov 2025) not new today.
Regression test drives end_scan_monitors directly (Daq.__new__, mocked
ensure_dir/open_group_writable/append_aux, no real filesystem access) -
confirmed it fails against the pre-fix code and passes against the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by running a real scan against a real bs stream on
saresb-vcons-03 (a test eco.status_server + a real BsStreamCounter),
then reading last_arrays back afterward, mirroring exactly what a user
would do interactively - arr._scan_parameter raised deep inside h5py
("invalid identifier type to function").
_store_arrays() copied CounterValue.store_arrays's reload-then-close
pattern verbatim ("re-open read-back handles so last_arrays keeps
working after the write file closes"), but escape.Array's own dataset
wrapper (ArrayH5Dataset) is lazy for at least some attributes even
past load_from_result_file (default lazy_loading=False notwithstanding)
- so the *reloaded* handles this assigned into self.last_arrays were
themselves tied to a second h5py.File that gets closed two lines later,
just as unusable as what it was replacing.
There was never a need to reload in the first place: the arrays
already in self.last_arrays (from _build_arrays) are plain in-memory,
numpy-backed Array objects with no file dependency at all. Dropped the
reload entirely; verified against the real recorded data (real scan
parameter values, correct shape) both immediately and via a fresh
DataSet.load_from_result_file() in a separate process.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live, against the real test server on saresb-cons-05: a fresh
full-namespace recording only attached to 18 of 13366 channels, yet the
report showed thousands of channels already "with data" - a real epics.PV
is not hashable (no __hash__), so it can never be a dict key, weak or
otherwise. _auto_monitor_refs[self.pv]/.get(self.pv) raised `TypeError:
unhashable type: 'PV'` on every real PV - but only *after*
pv.add_callback() had already registered the actual callback (which was
already collecting real data). RecordingSession's per-channel
`except Exception` then counted the TypeError as an ordinary attach
failure, and since the exception meant self._monitors[name] was never set
either, stop() never reached that callback - a leaked, permanently-running
subscription per channel, for nearly every channel.
The unit tests didn't catch this because the fake PV they use is a plain
object, hashable by identity by default - nothing about it exercised the
real constraint. Fixed by keying _auto_monitor_refs on id(self.pv) instead
(a plain dict, not weak: safe here because every CallbackEpics using a
given PV holds a live strong reference to it for as long as its entry
exists, and the entry is removed the moment the count reaches zero, so an
id() already in the dict is never a stale/reused one) - the actual pyepics
PV was never the problem, WeakKeyDictionary's hashability requirement was.
Also made FakePV explicitly unhashable (__hash__ = None) so the test suite
exercises this constraint for real going forward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Optional, off by default: launches escape-fel's upcoming live parse-cache
warmer (`python -m escape.swissfel.live_reduce --mode daq-cache`) as a
genuine child process once per scan, right where copy_scan_info_to_raw
already knows scan_info_rel.json's path - see escape-fel's own
"Integrating live parse-cache warming into the DAQ client" doc for what the
subprocess itself does (polls for newly-complete steps, calls
load_dataset_from_scan with checknstore_parsing_result="same_directory" to
warm the cache next to scan_info_rel.json, exits once the run goes quiet).
Feature-gated on the actual implementing attribute
(escape.swissfel.live_reduce.daq_cache_writer), not just module presence,
so turning this on ahead of the escape-fel release that adds it is a
harmless no-op - checked at launch time, not at Daq construction, so it
starts working the moment escape-fel is upgraded without needing the flag
re-toggled. At most one child per scan (copy_scan_info_to_raw fires once
per step plus once at scan end, not just once) - reuses a still-running
one, relaunches only if the previous one already exited. A plain
subprocess.Popen of *this* process on purpose, per the doc's own warning:
the child needs to inherit whatever account this Daq/session runs as,
since a different account (even in the same pgroup) can fail to write the
cache file into aux/ outright.
Also: bernina.status_server now points at ECO_STATUS_SERVER's URL when one
is set (the same env var Daq already reads), instead of always the
production default - lets a session point both Daq and the inspectable
status_server object at a test server (e.g.
ECO_STATUS_SERVER=http://saresb-vcons-03:8092) without touching production.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Multiple RecordingSessions could already start concurrently (start_recording
only refuses reusing the same recording_id), and each already gets its own
independent copy of every update - pyepics' PV.add_callback() natively
supports several simultaneous callbacks per PV. The one real bug this
exposed: CallbackEpics.start()/stop() used to snapshot pv.auto_monitor at
start and blindly restore it at stop, correct for exactly one CallbackEpics
per PV, wrong for two - whichever stopped first could clobber the
auto_monitor a still-running sibling needs (another overlapping recording,
or the status server's own permanent baseline monitor, which is already a
second CallbackEpics on top of any single recording today). Fixed with a
plain per-PV reference count in eco.epics_utils.utilities_epics: only the
first attacher changes auto_monitor, only the last detacher restores it -
stop() order no longer matters.
Also: a recording that starts while another is already running now
backfills its own buffer for any shared channel from whatever the earlier
one already collected (RecordingSession._seed_buffer_from_siblings) - a
one-time list copy at start(), truncated to the new session's own
max_points_per_channel, not a live share. So the later recording's data for
the overlap window comes from the earlier one instead of starting empty.
Ending the earlier recording (stop + backgrounded write/upload, already a
daemon thread, not blocking) leaves the later one's own monitors/buffers
completely untouched - each session only ever manages its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plain .h5 doesn't distinguish this file's format (one escape.ArrayTimestamps
per channel, see NAMESPACE_MONITOR_FORMAT.md) from any other HDF5 file in
aux/. filename is an opaque string end to end (capture_recording ->
/recording/capture -> write_monitor_recording just does directory /
filename, h5py doesn't care about the name) so this is a pure rename, no
behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the only way to stop using an already-configured status
server was setting the private _status_server = None directly - which
also throws away the configured URL, so re-enabling meant remembering
and restoring it by hand.
use_running_status_server (default True, matching current behavior)
is checked first thing in the status_client property - the single
choke point every status-server-backed callback in this class already
goes through (append_start_status_to_scan, start_scan_monitoring,
copy_aliases_to_scan, use_status_server(), ...), so gating it there
covers all of them for free. _status_server itself is left untouched,
so toggling this back to True brings back the same client, same
base_url, with nothing to re-supply.
Live-toggleable on an already-running session, no restart needed:
bernina.daq.use_running_status_server = False
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
start_scan_monitoring/end_scan_monitoring were unconditionally wired
into callbacks_start_scan/_end_scan with no way to turn them off short
of editing the callback lists directly. Needed now because
RecordingSession.start() attaches a CA monitor to every monitorable
channel in a single-threaded, unthreaded loop (namespace_store.py:1108),
and start_recording()'s HTTP call only gets the general
status_server_timeout (10s default) rather than the much larger
status_server_snapshot_timeout (180s) the analogous whole-namespace
snapshot/capture call gets -- reliably too slow at bernina's namespace
size, producing a "start monitoring failed ... falling back to the
local namespace mechanism" warning on every scan, easily mistaken for
the whole status-server integration being broken (it isn't -- that's
an unrelated, independently-succeeding call).
scan_monitoring defaults to True (unchanged behavior); set
bernina.daq.scan_monitoring = False on an already-running session for
an immediate, no-restart opt-out, or pass scan_monitoring=False at
construction. Checked first in both methods, before touching
status_client at all, so disabling it can't itself produce a warning.
Proper fix (making /recording/start async like /status/capture already
is) tracked separately, not done here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the remaining gap after the previous commit's start-time guard:
restart's non-systemd branch used to be an unconditional
"cmd_stop; cmd_start -b" - if nothing was running locally, cmd_stop was
a silent no-op and cmd_start went ahead and started a brand-new local
instance regardless of whether the real one was already up elsewhere
(exactly how the cons-03/04/05 crosstalk happened - cmd_start's own new
guard already covers a plain `start`, but `restart` reached the same
code with different framing and needed its own check before getting
there, not just relying on the guard to reject it after the fact).
New cmd_restart(): local pid found -> stop+start in place (unchanged
behavior for the normal case); none found but $URL is healthy
elsewhere -> respawn that one instead (self-re-exec over HTTP, not a
new local process); nothing running anywhere -> start fresh here, same
as before. managed_by_systemd path unchanged.
Verified by stubbing server_pid/remote_duplicate_running/cmd_stop/
cmd_start/cmd_respawn and confirming cmd_restart dispatches to exactly
the right one for each of the three cases - not against any real host.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Direct fix for a real incident (2026-09-16): three independent
eco.status_server instances ended up running simultaneously across
saresb-cons-03/04/05, each holding its own silently-drifting in-memory
namespace, discovered only by manually sweeping every console host.
cmd_start now checks remote_duplicate_running() before starting: if
$URL (almost always the site default from the env file, e.g.
saresb-cons-04) resolves to a *different* host than the one `start` is
running on, and that host's /health reports ready, refuse to start a
second one here. Deliberately does NOT block starting elsewhere when
$URL's host is actually down/unreachable - a genuine "the real one is
gone, stand up a replacement" case is not the mistake this guards
against. No shared lock file: the config directory isn't group-writable
across every account that might run this (same class of problem
eco_cnf_bernina/memory/ once hit), and $URL/health is already the right
source of truth, so there is no extra state to keep in sync or clean up
after a crash. -f/--force skips the check for the rare legitimate case.
Verified against local fake HTTP servers (different-host+healthy ->
refused; same-host -> allowed without even checking; different-host+
unreachable -> allowed) - never against any of the three real hosts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
restart/stop/start only ever manage a process on the local host
(pgrep/systemctl are inherently local) -- pointing $URL at a server
running elsewhere, as a different account, does nothing useful there:
it just finds no local process and starts a brand-new one right here.
respawn instead POSTs /admin/restart to $URL over HTTP, same as the
GUI's "Restart process" button (client.reinit(mode="restart") ->
client.restart()) -- the server re-execs *itself* (os.execv, same
PID/user/host it already runs as), so this correctly reaches the real
process regardless of where the CLI is invoked from. -w polls /health
afterward until it reports ready with a new instance_id, mirroring
StatusServerClient.restart(wait=True)'s own timing.
Verified against a local fake HTTP server (no-wait, --wait, and the
connection-refused path) -- not against the real one, deliberately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scans.acquiring_scan.* (daq_run_number, initial_values, ...) previously
only ever reached the status server via push_status(), which is
job-read-only (GET /status/job/<id>?include_status=1) -- never merged
into the actual status.json written to disk (store.snapshot() is
purely Alias.get_all()-driven, structurally blind to anything without
a CA channel, which is everything under a scan/Daq object).
Daq._client_status_for_scan(scan) now builds {"status": {...},
"status_channels": {...}, "status_times": {...}} from scan.get_status()
and self.get_status() (both already exactly this shape), keys prefixed
"scans."/"daq.", and hands it to _server_status_for_scan ->
_capture_on_server -> StatusServerClient.capture(client_status=...).
Daq's own get_status() is currently empty (nothing appended to its own
status_collection yet) -- included anyway so anything added there later
flows through with no further wiring.
Server-side, POST /status/capture merges client_status into its own
snapshot (status/status_channels/status_times, additively -- never
replacing the server's own alias-driven entries) before
write_status_snapshot, so it lands in the actual file. Timing is
already correct without reordering anything:
count_run_number_up_and_attach_to_scan (which sets scan.daq_run_number)
already runs before append_start_status_to_scan, both scan.get_status()
sources are populated in StepScan.__init__ itself, and the merge point
is inside the very same capture request already made -- no second
round trip.
Best-effort throughout (client_status=None falls back to exactly the
prior server-only behavior; a failure building it is caught and
printed, never raised).
Verified end-to-end against the real Flask app (two new tests: the
merge landing in the written status.json, and omitting client_status
leaving it unchanged) plus the full existing status-server/daq-client
suites (188 passed) -- one pre-existing test's exact-kwargs assertion
updated for the new always-present client_status=None fallback value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives BsStreamCounter the other thing it was missing compared to the
older EPICS-monitor CounterValue (which already auto-creates and
stores arrays every scan): a scan's per-channel bins used to live only
in memory (last_value(s), .grid(), the raw _channels), gone the moment
the next scan tore them down, with nothing written to disk.
_build_arrays() converts self._last_channels into real escape.Array
objects via Stream.to_array() (escape-fel, previous commit), with the
parameter axis relabeled to the scan's actual adjustable values
(scan.scan_info["scan_values"], same source
eco.acquisition.counters.parameter_from_scan uses) instead of
to_array()'s own default -- this counter's internal Scan is keyed on a
synthetic step index (see _StepIndexSource), not anything physical.
Falls back to the synthetic step_index for a channel whose bin count
doesn't match the scan's step count (a step that timed out with zero
samples never created a bin) rather than mislabeling steps.
_store_arrays() writes them to a fresh .esc.h5 via
DataSet.create_with_new_result_file/.append/.store(), mirroring
CounterValue.store_arrays's own round-trip. Both run from
_on_scan_end, gated by a new store=True constructor flag (storage_dir=
"auto" picks the directory, same contract as CounterValue's own);
any failure is printed, never raised, so it can't fail the scan.
Verified directly against a synthetic bs stream (escape.stream's
TestStream/LocalEventHandler, no real beamline) and added two
regression tests: the parameter/step_lengths values survive a full
store+reload round-trip, and store=False writes nothing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StepScan.scan_all()'s end-of-scan confirmation (inputimeout.inputimeout()
for return_at_end="timeout", plain input() for "question") blocked the
calling thread -- normally the interactive session's own thread, which
also owns any open live-plot window's GUI event loop -- for the whole
wait, freezing every such window for up to the 10s timeout after every
single scan. Same class of bug as the per-step live-plot freeze fixed
earlier via Acquisition's on_tick.
Added _inputimeout_nonblocking()/_input_nonblocking(): run the actual
read in a background thread and poll it with short joins from the
calling thread instead of one blocking call, flushing every open
matplotlib figure's GUI events between polls. Verified directly
(mocked inputimeout to control timing): both the timeout and
answer-arrives paths return the same result as before, with GUI
flushes actually happening during the wait rather than only after it.
Also passes BsStreamCounter's own name as label= into
Stream.plot_med() (escape-fel) so its live plot's title/y-axis show
e.g. "mon_opt.intensity" instead of the raw bs channel it subscribes
to ("SAROP21-PBPS133:INTENSITY").
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per user feedback on the ipymagic Qt-modal `...` fallback:
a) The picker window now has two tabs -- Components (the namespace tree,
as before) and a new Commands tab (eco.logs recent/frequent commands),
which previously only existed in the terminal Tab-completion overlay,
not the Qt fallback.
b) A pick no longer executes the current cell at all: _AstTransformer.
visit empties the cell's AST body and hands the expanded code to the
next prompt via IPython's set_next_input (the same mechanism %edit/
%load use) -- terminal and notebook/qtconsole alike. Verified live that
the cell body never runs (no assignment happens) and the next prompt is
pre-filled with the correct, directly re-executable expression.
c) In the picker, a single click only highlights a row now (tree, or a
Recent/Bookmark combo choice) -- it never fires a pick by itself. Only
Enter/Return, a double-click, or one shared Go button (below both tabs,
confirms whichever tab is active) commits a selection.
ComponentSelectorQt keeps its old click-to-select-immediately behavior
by default (require_confirm=False) since select_component/scan_
launcher_qt.py's existing picker dialog already depends on it --
confirm-gating is opt-in, used only by the new PickerQtWindow.
d) Dropped the "Value" tree column (and _format_value): get_current_value()
per row is a real, non-free call (EPICS or whatever) and the result is
stale the instant it's drawn anyway.
pick_component_modal now returns (kind, text, obj) instead of (path, obj)
-- kind is "component" (text relative to root, via _full_path as before)
or "command" (text used verbatim, never run through _full_path).
Added tests/test_component_selector_qt.py for the new confirm-gating/
CommandSelectorQt behavior, and extended tests/test_ipymagic.py for the
3-tuple contract and the cancel-and-set-next-input behavior. Verified live
via Xvfb: tabs render, a plain click doesn't pick, Enter/double-click and
the shared Go button do, and the full ipymagic path (pick -> next input,
no execution) works end to end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the hand-rolled Fit-only toolbar button with one call to
escape.plot_utilities.attach_escape_buttons(before_click=stop_animation),
so the plain EPICS-monitor live plot (CounterValue) gets the same Fit,
Peak, and Peak params (find_peak center/FWHM overlay) tools the
bs-stream live plots already have, instead of missing peak analysis
entirely. before_click stops the FuncAnimation before a button draws on
the axes -- otherwise the animation's own per-frame ax.cla() would wipe
whatever the button just drew.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per user feedback: terminal-ness alone shouldn't skip the Qt fallback --
only an actually unavailable display should. Replaces _in_real_terminal()
with _has_display() (QApplication.instance() already up, or DISPLAY/
WAYLAND_DISPLAY set), checked before ever constructing a QApplication --
confirmed live that doing so with no display is an uncatchable native
abort (SIGABRT), not a Python exception, so it has to be avoided rather
than caught.
Verified live with a synthetic root (bernina's own startup is currently
broken by an unrelated in-progress edit to eco/elements/adj_obj.py from
another session, so this bypasses that): no-display prints the hint and
leaves '...' in place without crashing; DISPLAY set via Xvfb opens the
real Qt picker and completes the pick as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous fix called flush_events() inside BsStreamCounter's
_wait_for_n_new() -- but that loop runs on a background PropagatingThread
whenever it's reached through acquire() (Acquisition is built with
hold=False there, which starts the thread immediately), not on the
scan's calling/main thread. Pumping a Qt event loop from any thread but
the one that owns it is unsafe and doesn't actually get the live plot
repainted; meanwhile the *main* thread -- the one actually blocked, and
the one whose GUI event loop needs servicing -- was sitting in a plain
Acquisition.wait() -> thread.join(), untouched.
Fixed by giving Acquisition an optional on_tick callback: wait() now
polls the acquisition thread in short joins instead of one blocking
join(), calling on_tick() after each -- on the *calling* thread, which
is where it belongs. BsStreamCounter.acquire() passes its
_flush_live_plot as on_tick.
This exposed a real, independent bug in PropagatingThread.join(): it
unconditionally read self.exc/self.ret after joining, which only exist
once the thread's run() has actually completed -- a timed join that
returns before the thread finishes raised AttributeError. Fixed by
checking is_alive() first and returning None (mirroring a plain
Thread.join(timeout=...) not having anything to report yet) instead of
touching attributes that aren't there yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1. startup_inline.py's ipymagic.start(eval(scope)) passed the bare package
*module* (import eco.bernina as bernina gives a plain module, no
.name, no status_collection) instead of the real eco.utilities.config.
Namespace instance every device is actually registered onto. Confirmed
live: every picked path fell back to a literal, undefined "root."
prefix (NameError on use), and the tree/completions looked sparse and
near-random -- build_tree() on a module has no children at all, so
what did show up was only leftover Recent/Recommended entries from
unrelated earlier use, not an actual namespace walk. Fixed by using
eco.{scope}.namespace instead, with a warned fallback to the old
behavior for a scope that doesn't follow that convention.
2. The Qt-modal fallback (a `...` that reaches Enter unexpanded) was
popping a Qt window even from a plain terminal session -- surprising,
and against the point of the Tab-completion overlay being the intended
terminal-native path. Now skipped in a real terminal (detected via
isinstance(get_ipython(), TerminalInteractiveShell)), printing a hint
toward Tab instead; unchanged in a notebook/qtconsole/script, where
there's no equivalent terminal-native alternative.
Both verified against the real facility entry-point chain (run
startup_inline_new.py -l -s bernina): Tab-completion now returns real,
correctly root-prefixed deep paths instead of scattered leftovers, and a
bare `...` at Enter in the terminal now prints the hint and raises a
plain TypeError on use instead of a confusing NameError on "root".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the terminal-native picker: typing the trigger and pressing Tab pops
IPython's own completion menu (same widget, works in a real terminal or a
notebook -- ipykernel's completion goes through the same IPCompleter
matcher pipeline) instead of the Qt modal. Two triggers:
"..." component mode: Recent, Recommended (FrequencyCounts-ranked),
then a live substring-filterable walk of the namespace tree.
"...h" command mode (mnemonic: history): Recent/Recommended prior
*commands* (eco.logs.recent_commands/frequent_commands) -- no
namespace section, nothing to browse there.
Selecting an entry only ever inserts text at the cursor -- it never
executes, in either mode -- so the rest of the line can still be composed/
edited before submitting. This composes with the existing Qt-modal `...`
mechanism rather than conflicting with it: accepting a completion replaces
`...` in the buffer before the line is ever submitted, so if you don't use
Tab and just hit Enter, the AST-transform fallback still applies to
whatever literal `...` is left.
Real bug found while building this and wiring it through a live
IPCompleter (not just unit tests): eco.widgets.component_selector's
ComponentNode.name (and so pick_component_modal's return value) is
relative to `root` (confirmed against Alias.get_full_name(base=...), which
stops at base) -- not a standalone, globally-evaluable expression the way
this module had assumed everywhere. Both the Qt-modal path (_EllipsisPicker)
and this new completer were splicing that relative fragment in directly,
silently dropping the root's own name from the printed/inserted code.
Fixed with _full_path(root, relative_path), which reattaches the name
`root` is actually bound to (its own `.name`, matching this codebase's
`import eco.bernina as bernina` convention) -- verified against a real
IPCompleter.complete() call, not just the mocked unit tests, which is
exactly what surfaced the bug in the first place (a hand-fed "already
correct" path in the earlier Qt smoke test had been masking it).
Also fixed the smoke-test script that had encoded the same wrong
assumption, for anyone re-running it later.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The command-history data sources for eco.ipymagic's command-picker mode
("...h") -- merged across every kernel-log session on disk (same source
eco.logs.widget()'s timeline already reads), deduplicated by keeping each
distinct command's most-recent occurrence (recent_commands) or ranking by
use count with a recency tie-break (frequent_commands), rather than
letting a command run every few minutes crowd out everything else.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RecentComponents is pure move-to-front recency; "Recommended" needs a
different signal -- something used constantly but not in the last hour
should still surface even after newer one-off components have pushed it
out of Recent. Same AdjustableFS-persisted-json, per-user pattern as
RecentComponents (mirrors it deliberately rather than sharing a base
class -- the value shapes and update semantics genuinely differ: a
deduped list you move to front, vs. a dict of counts you increment).
Wired into the same touch_from_write() chokepoint RecentComponents
already uses (every real adjustable write, via Changer), so Recommended
gets populated from real usage automatically, same as Recent -- no
separate hook needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A scan runs as one long synchronous call on the calling thread -- the
same thread an interactive Qt/ipympl session's GUI event loop normally
runs on. Nothing services that event loop while a scan is in progress,
so a live counter plot (escape.stream.Stream.plot_med) sat unpainted
(often literally black -- the canvas never got its first real paint
event) for the whole scan, and its redraw QTimer never fired either;
both only caught up once the scan call returned.
_wait_for_n_new() already loops in ~50ms slices waiting for new data,
so flush the plot figure's pending GUI events there every iteration --
costs nothing extra and lets the plot actually update live during a
running scan instead of only at the very end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
eco/startup_inline.py is what eco_cli.py/scripts/eco-dev and (via
startup_inline_new.py's shim) the real /sf/bernina/bin wrappers all run,
so it's the right place for this to be on by default rather than opt-in
-- a plain library import (from eco import bernina) still has to ask for
it explicitly. Root comes from `scope`, the same object already bound by
this file's own `import eco.{scope} as {scope}` a few lines up.
Verified live through the actual wrapper chain (run startup_inline_new.py
-l -s bernina): ipymagic installed on the real shell with bernina as root.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
from eco import bernina
from eco import ipymagic
ipymagic.start(bernina)
In [1]: (1 / ...).plot()
(1 / bernina.mono.energy).plot()
<plot appears>
In [2]: cen = (left := ...) - (right := ...)
cen = (left := bernina.mono.energy) - (right := bernina.attenuator.transmission)
A bare `...` (the builtin Ellipsis literal -- not an IPython special form,
so this works anywhere IPython's own ast_transformers hook runs: terminal,
notebook/JupyterLab, qtconsole) is replaced, before execution, by the
picked component's real dotted path -- the expanded code is printed, then
run, so what actually executed stays visible and re-typeable. Multiple
`...` in one statement resolve left to right. A `...` used as a numpy
subscript (`arr[...]`) is left alone. Reuses eco.widgets.component_selector
for the actual tree/fold-state/Recent-list machinery -- nothing new there.
New eco.widgets.component_selector_qt.pick_component_modal(): the existing
ComponentSelectorQtWindow is built for embedding/async display (start()
hands off to IPython's own Qt pumping, or run() blocks the whole process
forever); an AST transform needs an actual answer before it can even
return control to IPython, so this drives the same window with its own
nested QEventLoop that quits the instant a pick (or a close) happens.
Also fixed a real bug found while testing this live: IPython's
ast_transformers calls `.visit(node)` on each registered entry (expects
an ast.NodeTransformer-shaped object) -- a bare function registered there
throws 'function' object has no attribute 'visit' once, and IPython
*silently* unregisters it with only a warning, no visible error. Verified
end-to-end against a real Qt window on a real X11 display (Xvfb) plus a
real IPython shell: spawned the picker, drove a real selection via
QTimer, confirmed the printed/executed code and result were both correct.
Deliberately namespace-tree-only for now -- picking from prior *commands*
(eco.logs' kernel history) is a planned second mode, not built here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For "from eco import bernina" with no startup script involved at all --
the same per-session JSONL activity log every interactive entry point
already gets (eco.widgets.kernel_registry.install_shell_logger), exposed
as one explicit opt-in call instead of only happening as a side effect of
going through a startup script:
from eco import bernina
from eco import globallogger
globallogger.start()
Deliberately opt-in, not automatic on `import eco` -- a plain library
import (a short script, a notebook cell) shouldn't get a
~/.eco/kernel_logs/*.jsonl file appearing on disk just from importing.
Defaults the session label to the caller's resolved identity (eco.
elements.access.current_identity) rather than a generic tag, so two
people on a shared account still end up with distinguishable logs -- the
same "global user based" reasoning eco.elements.recent already applies to
the per-user Recent-components list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There were two full copies of the interactive startup script: eco/
startup_inline.py (the package copy, run via `eco_cli.py`'s `run -m eco.
startup_inline`, used by both the eco console-script and scripts/eco-dev)
and a second copy at the repo root, execed by startup_inline_new.py -- the
back-compat shim the real /sf/bernina/bin/{eco,eco_new,eco-in,eco_from}
wrappers hardcode by absolute path (root-owned, not editable here).
The two had already drifted: the root copy was silently missing the
kernel_registry.install_shell_logger() call the package copy had gained,
so every session started the normal way at a console (the actual daily
entry point) was never logged at all, despite eco.logs' viewer assuming
every console session has a log. Repointed the shim to exec eco/
startup_inline.py directly instead of the sibling root copy, and deleted
the now-fully-redundant root copy -- one real script left to maintain, so
this class of drift can't recur.
Verified live: ran a real ipython session through startup_inline_new.py
exactly as the /sf/bernina/bin wrappers do (`run startup_inline_new.py -l
-s bernina`), confirmed a fresh ~/.eco/kernel_logs/*.jsonl appeared with
the probe command actually logged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
install_stream_scans() monkey-patches escape.stream.Stream with the
same lazy .scans property bs_scannable already gives DetectorBsStream
(factored the shared body into _make_scans_property()) -- so any
Stream instance anywhere in eco, built through DetectorBsStream,
directly via escape.stream.Stream(...), or produced by Stream
arithmetic (a / b, .element(0), ...), can be scanned with
some_stream.scans.ascan(...)/.meshscan(...) immediately, no wrapping
needed. Called once, idempotently, from
detectors_psi._ensure_bs_event_worker() -- the earliest reliable
"this session touches bs streams" signal, since essentially every real
Bernina device constructs a DetectorBsStream well before any
interactive command -- and from this module's own import.
Also fixes the actual reason a 1-D BsStreamCounter scan never showed a
live plot: it never had any plotting wired in at all (CounterValue, the
older EPICS-monitor-based counter, always did). BsStreamCounter now
has live_plot=True by default: for a single-channel, non-grid counter,
_on_scan_start() calls plot_med() on the same already-accumulating
per-step Stream it uses internally (escape.stream.Stream.plot_med(),
median + percentile bands + peak overlay) -- costs nothing beyond
drawing, since the data is already flowing either way. _on_scan_end()
does one final replot() then stops the redraw timer (not
accumulate(False) -- this counter never tears down its channel
subscription mid-session, see _build_bins()'s docstring), so the
scan's true final result stays visible.
Also fixed a small bug this surfaced: _StepIndexSource (the synthetic
Scan parameter BsStreamCounter's binning is built on) was missing a
.unit attribute that Plot.plot()'s axis auto-labeling assumes any scan
parameter has.
Verified against a real scalar bs channel (SAROP21-PBPS133:INTENSITY)
with a real DummyAdjustable-driven ascan: both a bare
escape.stream.Stream never touched by any eco wrapper class, and the
existing DetectorBsTest.scans path, now build a real live Plot
automatically.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Investigated the "camera panels don't dock nicely" report live: spawned
real demo viewers under a real X11 display, docked them into an actual
EcoDesktopApp via the production embed path, and screenshotted several
container sizes. The image itself always scaled correctly (Fit zoom mode)
-- the actual cost was the always-open Settings dock (up to 400px), which
in a multi-camera grid routinely ate most or all of a squeezed camera's
own share of the width, leaving the image a sliver and pushing toolbar
rows into overflow.
Defaulting Settings to collapsed only when embed=True (standalone windows
keep the previous open-by-default behavior, where the ~780px this was
tuned for is actually available) reclaims that space immediately. Verified
live: re-ran the same two-camera-grid scenarios after the change -- both
cameras now get a proper, usable image area instead of one being reduced
to a sliver.
Remaining, out of scope for this fix: toolbar rows can still overflow in
a narrow dock (QToolBar's own overflow chevron, already flagged elsewhere
in this file as unreliable) -- a follow-up would need icon-only toolbar
buttons or an explicit "More" menu, which needs real icons added for
several text-only actions first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
self._step_sources is now a list (one _StepIndexSource per grid
dimension, sized from scan.grid_specs in _on_scan_start -- 1 for an
ordinary linear scan, matching the previous behavior exactly), and
_advance_step() sets each dimension's value from
grid_specs["index_plan"][step_index] instead of the flat step_index
when the scan has one. Scan() already keys bins on the tuple of however
many parameters it's given (see escape.stream.Grid, added earlier),
so this is the whole mechanism N-D binning needed -- the existing
linear step_index still indexes into DataManager._data unchanged,
since a real StepScan/meshscan visits each grid cell exactly once, in
index_plan order, so scan._values[step_index] == index_plan[step_index]
holds the same way the plain 1-D case always relied on.
New BsStreamCounter.grid(channel=None, shape=None, positions=None,
dimension_names=None) wraps one channel as a live escape.stream.Grid.
Works both while a grid scan is still running (reads self._channels)
and after it finishes (reads a new self._last_channels/_last_scan/
_last_grid_specs snapshot, taken in _on_scan_end() right before
_build_bins() replaces the just-finished scan's bins with fresh, empty
ones for standalone use) -- confirmed this snapshot is actually needed:
without it, grid() after a completed meshscan saw nothing, even though
last_value(s) (an already-extracted plain value, not a bin reference)
reported the final step fine.
Verified against real dispatcher channels with real DummyAdjustable-
driven scans: a 5-step ascan against JF01T03V01:roi_intensities;
a 3x3 meshscan against SAROP21-PBPS133:INTENSITY end to end through
.grid().mean(plot=True) (a real live heatmap, no NaNs); and a 3x3
meshscan against JF01T03V01:roi_intensities itself (array-valued, 2
ROIs) surfacing a real, separate escape.stream.Grid bug fixed alongside
this (see the escape-fel commit) rather than silently producing an
all-NaN grid.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Remove det_jf: now that det_diff was fixed to JF01T03V01 (previous
commit), det_jf was a second device pointing at the same physical
Jungfrau, otherwise unreferenced anywhere.
- Add bernina.dummy (DummyMexicanHat): x, y (DummyAdjustable) + imex
(DetectorGet reading a live 2D Mexican-hat signal of their current
values, (1-r^2)*exp(-r^2/2), r=sqrt(x^2+y^2) -- stays within [-1, 1]
for any x, y, small Gaussian noise by default). No real hardware;
for testing grid-scan / live 2D counter-grid machinery.
- Fix Scans.meshscan(..., repetitions=N>1): grid_specs["index_plan"] is
built once, for a single pass over the grid, before StepScan.__init__
repeats `values`/`pulses_per_step` by `repetitions` -- so
do_next_step()'s gridspecs["index_plan"][self.next_step] indexed past
the end (IndexError) the moment the scan entered its second
repetition. Now repeats index_plan the same way.
- Add scripts/eco-escape-test: like eco-dev, but also prepends a sibling
escape-fel checkout to PYTHONPATH (assumed to live next to this eco
checkout), for testing local checkouts of both together instead of
whatever's installed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bernina_config.json (the live, persisted config_bernina store) has both
xrd_config.jfs and robot_config.jf_id/jf_name saying det_diff is
JF01T03V01, but bernina.py had it hardcoded to JF07T32V01. Confirmed
JF01T03V01:roi_intensities is currently live on the dispatcher while
JF07T32V01's is not -- the physical detector assignment moved and this
hardcoded id never got updated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
EmbeddedProcessWindow's body stayed completely blank while the spawned
viewer subprocess was still starting up (eco's own first import alone is
~10-15s) -- indistinguishable from a hang. Now shows a note explaining
what's happening, plus how to skip the wait (separate_process=False) and
what that costs (the viewer then freezes during any blocking call in this
session). Cleared automatically by whichever of embed()/
show_floating_note()/show_error() resolves next.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CameraBasler/CameraPCO._default_dock_in was True, so the plain
default-widget path (e.g. starting eco and opening a camera viewer) always
docked into a shared EcoDesktopApp workbench, auto-creating one if none
was open -- surprising for anyone not asking for that. Reverted to None
(standalone, matching Assembly's own base default); widget(dock_in=True)
still opts in explicitly and still auto-creates the workbench container
if none exists yet, unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test_channel_lists.py: test_compare_reports_missing_and_exceeding asserted
set(result) == set(CHANNEL_LIST_CHANNELTYPES) (expecting channels_BS/
channels_BSCAM present) right next to asserting those same two keys are
*not* in result -- self-contradictory. compare_channel_lists() only ever
reports a list name that was actually present in `recorded`, matching the
module's own docstring; fixed the assertion to match.
test_cameras_swissfel.py: test_camera_basler_viewer_separate_process_passes_
name_and_cam_class called CameraBasler._widget_viewer as an unbound method
against a bare fake `self` that didn't carry _CAM_CLASS_PATH, so
getattr(self, "_CAM_CLASS_PATH", None) read None instead of the real
class's value. Added _CAM_CLASS_PATH to the fake to match what a real
CameraBasler instance would have.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lemke_handClaude Sonnet 5