Three follow-ups to the status_server namespace component and
recording feature:
1. StatusServer's recording settings are now AdjustableFS, not
AdjustableMemory - each a small JSON file under the same shared
eco_cnf_bernina/configuration tree channels_JF/config_JFs already
use. The point: a setting changed in one process is visible to any
other process reading the same filesystem (another session, or the
status server itself) with no REST call or other new communication
- confirmed with a test constructing two independent StatusServer
instances against the same config_dir.
2. RecordingSession.size_report(): per-channel projected bandwidth -
rate (this channel's own point count over elapsed time) times a
sample value's size (element count * dtype itemsize, "frequency *
data shape * bitdepth") - to judge storage/bandwidth cost from a
short trial recording before committing to a long one. Works on a
still-running recording. Exposed via GET /recording/<id>?size=1
(optional &size_top_n=N) and StatusServerClient.recording(...,
size=True).
3. failed_required for recordings: a channel that fails to attach and
belongs to a required namespace component (namespace.required_names())
is now reported separately in a recording's report() - the same
"everyone else failing is expected, this failing is not" distinction
/health's failed_required already makes for init failures. New
client-side warn_recording_failed_required (red, but does not
suggest reinit() the way the health one does - a demoted/
disconnected channel isn't fixed by rebuilding the namespace),
wired into Daq.start_scan_monitoring.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two-tier fix for a namespace recording's "channel with zero updates
is simply absent from the file" gap:
1. RecordingSession.start() now seeds a channel with its current value
(add_current_value=True) when that's actually free: pyepics'
PV.get_with_metadata() returns the cached value with no CA traffic
when auto_monitor is already True and the channel is connected -
true for the large majority of channels under ca_tuning.make_pv()'s
default policy. A demoted "fast" channel or a disconnected one
would make this a real blocking get, so those are left unseeded, as
before. New n_seeded counter reports how many got the free seed.
2. For the rest, RecordingSession.backfill_from_status() lets a status
snapshot that was already being taken for another reason (a scan's
own status_run_start/status_run_end capture) opportunistically fill
in one value for any channel still at zero points - never a trigger
of new CA traffic on its own. NamespaceMonitorStore gains
pgroup/run_number on start_recording() (purely for this lookup) and
backfill_running_recordings(), wired into both /status/capture and
/status/snapshot right after their own snapshot() call. New
n_backfilled counter.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
start_scan_monitoring/end_scan_monitoring are now in
callbacks_start_scan/callbacks_end_scan - server-only (no local
fallback, unchanged from when these were written), so a scan without a
status server configured simply gets no aux/namespace_monitor.h5, the
same as it already gets no server-backed aliases.json/status.json.
Placement matters: start_scan_monitoring runs after
count_run_number_up_and_attach_to_scan (needs scan.daq_run_number) and
append_start_status_to_scan (shares its per-scan server-in-use
decision); end_scan_monitoring runs before copy_scan_info_to_raw,
which is what actually writes the "monitors" scan parameter into
scan_info_rel.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
end_scan_monitoring() now writes to aux/namespace_monitor.h5 (instead
of the general-purpose recording API's default monitors.esc.h5) and
registers it via scan.set_scan_parameter("monitors",
"aux/namespace_monitor.h5") right after dispatching the capture job -
the same "register the path now, the async job finishes later"
pattern copy_aliases_to_scan/append_start_status_to_scan already use
for "aliases"/"status", so copy_scan_info_to_raw's scan_info_rel.json
carries it the same way.
Added eco/status_server/NAMESPACE_MONITOR_FORMAT.md: a handoff spec
for escape_fel describing the file format (one escape.ArrayTimestamps
per channel, keyed by the same dotted alias space as status.json/
aliases.json), how to load it (escape.DataSet(results_file=...)
already dict2structure()s it correctly for free - found by reading
escape_fel 0.2.7's actual source, not guessed), how it relates to and
should replace the existing pickle-based "monitor data hack" in
escape/swissfel/parse.py, and the merge-order needed for a co-loaded
bs/detector dataset to supersede the CA-monitor one for the same
component. Not yet exercised against a real load - flagged as such.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
POST /recording/capture: stops a recording, writes it, and uploads it
to the run - all in the background, mirroring /status/capture and
/aliases/capture. Stopping (detaching callbacks) happens synchronously
before responding, so the recording is unambiguously over by the time
the caller gets a job id; only the write (scales with points recorded)
and the broker upload are backgrounded. Load-tested against the live
server first (start/get_status/stop concurrently): snapshot latency is
unaffected by a concurrent recording (~11s either way), ~38-44% of one
core combined, no thread/memory leak across repeated start/stop.
Daq gains start_scan_monitoring()/end_scan_monitoring() - start a
namespace-wide recording keyed to the run, and capture+upload it -
built to be called at a scan's start/end but deliberately NOT added to
callbacks_start_scan/callbacks_end_scan yet. No local fallback: unlike
status/aliases, a local recording would need this session's own
namespace holding a live monitor per channel for the whole scan, which
is exactly the per-session cost the status server exists to avoid.
Also fixes a latent cross-feature bug found while adding this: GET
/status/job/<id>?include_status=1 merged in any pending
scans.acquiring_scan.* push (from the just-merged push_status
mechanism) by defaulting the job's "key" to "status_run_start" when
absent - but aliases-capture jobs (and now recording-capture jobs)
carry pgroup/run_number for their own reporting without ever setting
"key". Reading such a job with include_status=1 could have silently
stolen a push meant for the real status_run_start job of the same run.
Fixed by requiring an explicit "key" on the job before merging - only
genuine status/capture jobs set one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scans.acquiring_scan.* (description, scan_command, initial_values, ...) can
never reach the status server's own snapshot: it's built from DetectorMemory
objects with no CA channel, and the server's channel registry is driven
entirely by Alias.get_all(), which only returns aliases that have one. The
run table's "Custom table" sheet already has header cells expecting these
exact column names, so they've been silently blank.
Add a small push mechanism instead of another metadata.*-style workaround:
Daq now hands the values it already resolves (the same ones metadata.* is
built from) to the status server via a new POST /status/push, which merges
them into the matching /status/capture job's status the next time it is
read with include_status=True - the natural point, since that's exactly
when daq_client asks for the run's status to fill the run table.
NamespaceMonitorStore.push_status/pop_pushed_status hold the pushed values
scoped by (pgroup, run_number, key), consumed once and dropped after 5
minutes if nothing ever reads them, so a push whose scan aborts can't
attach itself to a later, unrelated run.
The server already holds a fully-initialized namespace, so its
namespace.alias.get_all() is complete and free (pure in-memory tree
walk, no CA traffic). The client side's own copy_aliases_to_scan
called self.namespace.alias.get_all() unconditionally, including in
status-server mode - against a namespace that mode deliberately never
initializes, so aux/aliases.json was silently missing everything the
local session had not happened to touch.
- namespace_server.py: GET /aliases (synchronous, for direct
inspection) and POST /aliases/capture (async: compute, write
aliases.json, upload to the run - mirrors /status/capture, shares
its job-id space via GET /status/job/<id>).
- storage.py: write_aliases_file(), same group-writable-tree handling
as write_status_snapshot().
- client.py: StatusServerClient.aliases() / .capture_aliases().
- daq_client.py: copy_aliases_to_scan now asks the same per-scan
status-server decision append_start_status_to_scan already makes
and caches, delegating to the server when it's in use and falling
back to the local path (unchanged) otherwise.
Also fixes a latent bug found along the way: Daq.__init__ accepted an
`instrument` kwarg but never stored it as self.instrument, so every
f"/sf/{self.instrument or 'bernina'}/..." path in
_write_status_locally()/write_status() raised AttributeError on any
real Daq - masked because the only test reaching that path tolerates
"any exception" as "fell back correctly".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Henrik Lemke
lemke_handClaude Sonnet 5