Notes what was actually exercised against the live sim (registration/
list/unregister, per-job param snapshot including the reset-to-None
case, the hook firing during a real tomo_queue_execute() run, and the
unregistered-hook incomplete/resume path) and the scan_repeat retry
bug found and fixed along the way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by testing the new at_each_angle_hook feature against the live
sim: _tomo_scan_at_angle is wrapped in @scan_repeat(max_repeats=10,
default=True), which retries *any* exception, including a FlomniError
raised for an unregistered hook -- a condition that will never
succeed by retrying. The operator saw 10 silent retries before a
generic "TooManyScanRestarts", with the actual, actionable error
message buried in __cause__ instead of reaching
tomo_queue_execute()'s printed failure message. Adds an exc_handler
that skips the retry for any FlomniError (a definite, non-transient
failure by convention throughout this file), letting the real message
surface immediately. General fix, not narrowly scoped to hooks --
every other FlomniError this method could raise gets the same benefit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a "Custom behavior at each projection angle" subsection to the
Tomography section of the flOMNI user manual (docs/user/ptychography/flomni.md):
what a hook function looks like, a polarizer-modulation example, and
how to register/activate/deactivate/list/unregister one, with an
explicit warning about resetting at_each_angle_hook before queuing a
normal follow-up job. Also links it from the quick-start queueing
section. AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md gets a new section
3.4 recording the design decision (session-only registry, not a
hardcoded one; a new _TOMO_SCAN_PARAM_NAMES entry, not a new job kind;
the GUI param-snapshot leak found and fixed while wiring this up; no
GUI widget yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TomoParamsWidget/TomoQueueDialog mirror flomni.py's snapshotted-param
list by hand (QUEUE_PARAM_NAMES/DEFAULTS) and it hadn't picked up the
new at_each_angle_hook parameter. Left as-is, a GUI-added job's params
dict would have no at_each_angle_hook key at all, so the executor's
per-job setattr loop would never touch it -- silently leaving whatever
hook the *previous* queued job had active in place for a job that
never asked for one. Adds the key to QUEUE_PARAM_NAMES/DEFAULTS (fixes
"Add current params to queue", which reads live global vars) and has
add_edited_to_queue() explicitly pass the live value through (there's
no editable widget for it yet, so it can't come from the form fields).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flomni._at_each_angle() already had an escape hatch for custom
per-projection behaviour (a flomni_at_each_angle function checked via
builtins), but it's invisible to the tomo queue -- not a snapshotted
parameter, so a queued job had no way to select it. Adds a
session-only registry (register_at_each_angle_hook/unregister/list)
populated from the IPython session, plus a new snapshotted
at_each_angle_hook parameter so a tomo-queue job can carry its own
hook selection through add/execute/resume like any other tomo param.
Curated means the queue only ever stores a hook *name*, never code --
_at_each_angle() raises a clear error rather than silently falling
back to default behaviour if a job's named hook isn't registered in
the session running the queue (hooks don't survive a kernel restart).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates plan section 6.4 and the GUI test checklist (item 5, new test
step 23a) for narrowing sort-mode/tomo_queue_move() reordering to
pending-only, with running and incomplete both acting as a floor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sort mode and tomo_queue_move() let a pending job be dragged/moved
visually ahead of an incomplete one. Harmless to actual run order --
tomo_queue_execute()'s pick-next loop always resumes the first
running/incomplete job it finds regardless of list position -- but
confusing to read, since list order looked like run order and wasn't.
Narrows the GUI's _MOVABLE_STATUSES to ("pending",) and the CLI's
floor check to the highest index among all running/incomplete jobs,
not just the running one. Found while testing the reorder feature
against a real interrupted scan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feye_out() unconditionally moved dev.fttrx1 out of the way, raising
an unhandled DeviceConfigError on any session (e.g. sim) where that
device isn't configured, after already having closed the shutter and
moved other optics. Now checks "fttrx1" in dev first (same pattern
already used for rtx elsewhere in this file) and, if absent, asks via
OMNYTools.yesno() whether to continue without moving it instead of
crashing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates plan section 6.1 (marks the two-write-paths rule as actually
implemented now) and the GUI test checklist (section 1 item 8, and
flags section 3 item 4 as needing re-testing since its old passing
result was against the since-replaced soft-warn Submit behaviour).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the only way to queue a new tomo parameter set was to
Submit it live first (writing the running scan's global vars) or use
the separate queue dialog's "Add current params to queue", which
snapshots the live vars and silently ignores any unsubmitted edit --
so there was no way to prepare a new parameter set in the GUI while a
scan was running without perturbing it.
Edit mode now shows an "Add to queue" button that packages the
current (possibly unsubmitted) form fields as a new pending tomo job,
writing only tomo_queue -- never the live param vars -- so it's always
allowed. Submit becomes a real hard block while the beamline is busy
instead of a Yes/Cancel confirmation, now that there's somewhere else
to put an in-progress edit (TOMO_QUEUE_COMMAND_JOBS_PLAN.md section
6.1's two-write-paths rule, finally enforced as written).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tomo_scan() called bec.messaging.scilog directly with no guard, so a
session without scilog enabled raised RuntimeError from inside
tomo_queue_execute() -- marking the job "incomplete" and pausing the
whole queue over a logging side effect unrelated to whether the
tomogram succeeded. Adds _scilog_write(), mirroring the enabled-check
already used for the PDF report entry, and routes the three unguarded
call sites (tomo_scan, scilog_last_ptycho_scans, the alignment-scan
entry) through it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates plan section 6.7 and the GUI test checklist (section 1 item 7,
new test step 27) for the Label/Details stretch columns and the
dropped Added-at column.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
All 9 columns previously shared equal Stretch width, squeezing the two
free-text columns (Label, command-step Details) down to the same width
as short fixed-content ones like Type or Exp (s). Short columns now
auto-fit their content; Label and Details stretch to fill what's left
and left-align instead of centering. Also drops the Added-at column --
that value is still available via the row tooltip added earlier.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously start() did nothing at all when the BEC account didn't
match the system user (e.g. a dev/sim session run under a different
account) -- now it starts a status page on a separate fallback port
(default 8081) for local debugging, while still skipping everything
that assumes the active account is the "real" one for this host: the
cross-session singleton lock, remote upload, and session.htpasswd
changes. Not yet tested against a live/sim session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates the command-jobs plan (status line, section 6.4's
implementation note, new section 6.7, commit breakdown) and the GUI
test checklist (section 1's what-was-built list, a new section 6a
with test steps 19-26, and the corrected known-limitations entry) to
reflect tomo_queue_move and the sort-mode/tooltip GUI work. Neither
has been validated against a live session yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a checkable "Sort queue..." mode: other queue-mutating buttons
disable, the table goes single-select, and Move up/down swap the
selected row with an adjacent one -- but only between two
pending/incomplete rows, so a running or done neighbour blocks the
swap (gets the running-job floor and done-row pinning from
TOMO_QUEUE_COMMAND_JOBS_PLAN.md sections 6.3/6.4 without drag-and-drop
math). Also gives every row a full-detail tooltip (label, status,
steps/params, added-at) so hovering surfaces what the Label/Details
columns truncate at typical dialog widths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lets a queued job be moved to a new position via the CLI, refusing to
touch a running/done job and refusing any move that would displace the
currently running job (a hard floor, per the mid-run-reorder design in
AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md section 6.4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TOMO_QUEUE_GUI_TESTING.md is the manual verification plan for the new
command-job builder: baseline regression checks, the builder itself,
the status-keyed delete/clear guards, and an end-to-end mixed-queue
run, plus this iteration's known gaps (no mid-run reorder, one device
per move step, no live limit validation). Nobody has clicked through
the actual widget against a live session yet - it's built and
smoke-tested at the logic level only, not GUI-validated.
Section 8 covers continuing this work after leaving the current coding
session: a Claude Project with AI_docs/ attached as project knowledge
is for planning/discussion, a Claude Code session attached to this
repo is what's actually needed to keep editing tomo_params.py - and a
ready-to-paste prompt for starting that next session.
Also updates TOMO_QUEUE_COMMAND_JOBS_PLAN.md's status line: Step 3 has
a first iteration built, not the "not started" it said before.
First iteration of Step 3 (TOMO_QUEUE_COMMAND_JOBS_PLAN.md section 6)
on top of the existing TomoParamsWidget/TomoQueueDialog baseline:
- CommandJobBuilderDialog: a structured step-by-step form (action
dropdown, per-action fields) built entirely from the live
tomo_queue_actions global var, so it tracks flomni.py's action
registry and device allow-list with no hardcoded mirror. Produces a
job dict identical to what tomo_queue_add_command() writes.
- TomoQueueDialog's queue table gained a "Details" column and renders
command jobs (CMD + step summary + idempotency marker) the same way
flomni.tomo_queue_show() does.
- "Add current params to queue" now stamps kind="tomo" and a fresh id
on the job, matching the CLI's own self-describing format.
- Delete/clear now refuse when a "running" job is involved instead of
silently allowing it (plan section 6.3).
Mid-run drag-reorder of queue jobs (plan section 6.4), multi-device
single move steps, and live device-limit validation are deliberately
not in this iteration - see TOMO_QUEUE_GUI_TESTING.md.
Generalizes the patterns and gotchas learned building the flOMNI
tomo-queue e2e suite (main-thread-only live-update machinery, the
builtins/reload import-order bootstrap, neutralizing real side
effects, the demo-config trap in pytest_bec_e2e's shipped fixtures,
the omny_e2e_tests/ folder convention, and the Redis
cross-test-contamination fixture bug) into a standalone reference for
adding e2e coverage to any plugin in this repo, not just flomni.
Intended to be added to a Claude Project alongside this repo so a
coding assistant finds it and follows these patterns instead of
re-deriving them from scratch.
Adds a "Command jobs" reference section covering
tomo_queue_add_command(), the move/optimize_idgap action registry, the
mokev/idgap device allow-list, tomo_queue_show() output format, and
the idempotent-vs-prompt crash-resume behaviour, plus a pointer to it
from the quick-start queueing section.
TOMO_QUEUE_COMMAND_JOBS_PLAN.md and TOMO_QUEUE_TESTING.md were written
while command jobs (Step 2) were still in progress; update both now
that it's implemented and sim-tested (16/16 e2e tests passing,
including the 6 new command-job ones - see TOMO_QUEUE_TESTING.md
section 4D for what each covers), and record the Redis test-state
contamination gotcha found while testing it.
Also fixes the harness-script path references in SIMULATED_ENDSTATIONS.md
and the two scripts' own docstrings, stale after tests/e2e and
tests/sim_*_harness.py moved to omny_e2e_tests/ (in the previous
commit) - kept out of tests/ specifically so gitea's CI, which runs
pytest against ./csaxs_bec/tests/ unconditionally, doesn't sweep up
tests that need a live sim + Redis.
Covers add-time validation (unknown action, disallowed device,
out-of-range/wrong-type params, unknown kwarg - each rejected without
landing in the queue), the execution-time allow-list re-check as the
real safety boundary, the tomo_queue_actions global var matching the
live registry, tomo_queue_show()/the webpage generator surviving a
command job, legacy kind-less jobs still running as tomograms
alongside a real command-job device move, and the non-idempotent
resume prompt (distinct from the executor's own start-confirmation
prompt).
Also fixes flomni_sim: it only snapshotted/restored whatever was
already in the tomo_queue/tomo_progress global vars, so a stale entry
left behind by an earlier crashed run or manual debugging session
against the same shared sim Redis would contaminate every test that
assumed it started from an empty queue. Now resets both to empty at
setup, in addition to snapshotting/restoring around the test.
Type-tags queue jobs with a "kind" field (back-compat: missing kind
defaults to "tomo"), and adds "command" jobs that reconfigure the
beamline (currently: absolute moves on an allow-listed device, plus an
optimize_idgap stub) between tomograms instead of running a scan.
Command jobs are added via tomo_queue_add_command() against a curated
named-action registry (_TOMO_QUEUE_ACTIONS) - not arbitrary code - so
every queued action is auditable and safe to persist across a kernel
restart. Validation is two-layer: schema checks at add time, and each
action re-checks its own arguments against live hardware at execution
time, which is the actual safety boundary. Crash-resume for a command
job re-runs the whole step sequence from the top if every step is
idempotent, or prompts the operator otherwise.
flomni_webpage_generator.py is guarded against command jobs, which
have no "params" key.
See AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md for the full design.
Sections A-C of TOMO_QUEUE_TESTING.md's checklist, run against a live sim
session: params-restored-per-job, legacy queue migration, empty/all-done
no-ops, start_index semantics, exception/SIGKILL crash-resume, resume-
before-fresh ordering, and concurrent queue edits from a second client.
All 10 tests pass together with no cross-test interference.
The flomni_sim fixture and _bootstrap.py factor out what it takes to
construct a live Flomni against the sim (builtins/reload bootstrap,
side-effect neutralization, RT-feedback/fsamx setup) for reuse by both
in-process tests and the SIGKILL-based subprocess tests.
Also adds the AI_docs/ handoff docs (testing checklist results, command-
jobs plan with the action-registry decisions) that this work updated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The constant lists live scan properties snapshotted/restored by the tomo
queue, not queue state itself -- the old name was misleading. Confirmed
with Mirko. Updates the matching comment in tomo_params.py, which mirrors
this tuple as QUEUE_PARAM_NAMES (unchanged name, own mirror constant).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XrayEyeAlign.gui is a property delegating to lamni.xeyegui and has no
setter. Mock lamni.xeyegui in the update_frame tests; drop the redundant
alignment_images reset, which __init__ already handles.
Drop test_save_frame (method no longer exists). Rewrite test_update_frame
against dev.fsh/dev.cam_xeye instead of the removed LamNI.alignment module.
Patch the module-level umv in test_tomo_rotate rather than builtins.
LamNI.__init__ calls set_client(), which unconditionally accessed
client.gui. This broke test_x_ray_eye_align.py under ClientMock, which
has no gui attribute. Use getattr with a None default; behaviour on a
real BEC client is unchanged.
- Enable DAP fit parameter forwarding (self.gui.set_dap_params_forwarding),
never called before -- without it the dap_params_update signal was never
connected, so no fit data could ever reach fit_params_x/y regardless of
how long anything waited. Disabled again once the fit is read (or on
abort, via the existing KeyboardInterrupt cleanup in lamni.py).
- Replace the fixed 5s sleep before reading the DAP fit with
_wait_for_dap_fit(), which polls fit_params_x/y for the real converged
shape (up to 30s) instead of guessing a duration.
- Fix write_output()'s offset reference: was using one of the 8 angle
points as its own zero-reference (first via an off-by-one at k=1, then
at k=2), which forces that point's offset to exactly 0 by construction
and biases the fit. Now references alignment_values[0] (the FZP centre,
the only angle-independent point in the procedure), matching both
FlOMNI's own reference point and the original SPEC macro
(l_xrayeyealign.mac) exactly, including the asymmetric x/y sign
convention.
- pixel_calibration is now a property reading the cam_xeye device's
pixel_calibration user parameter, matching FlOMNI's camera-sourced
calibration, with the old hardcoded constant kept only as a fallback
(PIXEL_CALIBRATION_DEFAULT) -- by design, to match FlOMNI's own pattern.
- lamni_optics_mixin.py: lfzp_in() and loptics_out() now print before/after
re-establishing interferometer feedback, previously silent.
- device_configs: add the missing userParameter.pixel_calibration to
cam_xeye in ptycho_lamni.yaml and simulated_omny/simulated_lamni.yaml
(present on the FlOMNI side already; this is what the pixel_calibration
property above actually reads at runtime).
- Add lamnigui_stop_gui(), matching flomnigui_stop_gui()
- lamnigui_show_xeyealign(): check/enable cam_xeye live mode on every
call, not just when the xeyegui widget is first created -- matches
flomnigui_show_xeyealign()
- lamnigui_idle(): refresh the splash text on every call rather than
only at first creation -- matches flomnigui_idle()
Replace the plain self.progress dict with a global-var-backed
_ProgressProxy (same shape and key as FlOMNI's), so progress survives
a client restart. Adds heartbeat + idle-time-gap detection in
_tomo_scan_at_angle, ETA computation in _print_progress, and
tomo_start_time/reset handling in tomo_scan(). Adds
tomo_progress_reset() and _format_duration() for parity.
gui_tools.py's progress display already read these fields; no GUI
change needed.
feat(lamni): persist alignment lookup-table corrections
corr_pos_x/corr_pos_y/corr_angle (+ _2 iteration) become
global-var-backed properties (lamni_corr_pos_x, lamni_corr_pos_y,
lamni_corr_angle, ..._2) instead of plain instance lists that reset
on every client restart. No default correction file exists for
LamNI, so reset_correction()/reset_correction_2() just clear to
empty.
feat(lamni): add frames_per_trigger (burst mode) support
LamNIFermatScan already inherits from AsyncFlyScanBase and already
accepts/passes through frames_per_trigger -- only the client-side
plumbing was missing. Adds a validated, global-var-backed
frames_per_trigger property, threads it into
tomo_scan_projection()'s lamni_fermat_scan() call, and exposes it in
tomo_parameters(). Also folds it into the idle-time cadence estimate
alongside stitch tile count.
feat(lamni): add crosshair and structured HDF5 output to X-ray eye alignment
Set the alignment crosshair at the FZP centre (step k=0, raw pixel
coords), hidden on normal completion and on Ctrl-C abort.
update_frame() now actually captures and stores each frame
(previously not stored at all); roi_pixel_data is now collected at
every submit. write_output() additionally writes a timestamped HDF5
file (alignment_values, alignment_images, roi_pixel_data,
alignment_fit) alongside the existing plain-text archival file.
fix(lamni): stop resetting correction state on every XrayEyeAlign instantiation
A fresh XrayEyeAlign is constructed on every call to both
xrayeye_alignment_start() and xrayeye_update_frame(); resetting
corr_pos_x/y/angle and tomo_fit_xray_eye in __init__ meant the
latter -- meant as a lightweight frame check -- silently wiped valid
persisted correction data every time. Reset moved to the start of
align() itself.
fix(lamni): use dev.fsh instead of dev.omnyfsh for the alignment shutter
The shared XRayEye widget monitors and controls dev.fsh; the
alignment script was opening/closing a different device, which
would desync the GUI's shutter toggle from the actual hardware
state.
fix(lamni): correct stale self.align reference in MagLamNI.rotate_slowly
self.align no longer exists as an attribute now that
LamNIAlignmentMixin is mixed directly into LamNI. Use
self.tomo_fovx_offset/self.tomo_fovy_offset directly.
Protocol-level simulation of the flOMNI hardware injected via socket_cls:
real device/controller classes run unchanged against state machines
implementing fgalil.dmc, galil_micos_upr.dmc, the Smaract MCS protocol
and the Orchestra CommunicationServer. Cameras reuse the real device
classes with a synthetic frame source. Includes simulated_flomni device
config (boots referenced at 'in' positions) and an offline harness
covering moves, referencing, rt feedback/tracker, flyer scan readout
and the gripper transfer routine.
appel_candx01dc
