Generalizes the patterns and gotchas learned building the flOMNI
tomo-queue e2e suite (main-thread-only live-update machinery, the
builtins/reload import-order bootstrap, neutralizing real side
effects, the demo-config trap in pytest_bec_e2e's shipped fixtures,
the omny_e2e_tests/ folder convention, and the Redis
cross-test-contamination fixture bug) into a standalone reference for
adding e2e coverage to any plugin in this repo, not just flomni.
Intended to be added to a Claude Project alongside this repo so a
coding assistant finds it and follows these patterns instead of
re-deriving them from scratch.
Adds a "Command jobs" reference section covering
tomo_queue_add_command(), the move/optimize_idgap action registry, the
mokev/idgap device allow-list, tomo_queue_show() output format, and
the idempotent-vs-prompt crash-resume behaviour, plus a pointer to it
from the quick-start queueing section.
TOMO_QUEUE_COMMAND_JOBS_PLAN.md and TOMO_QUEUE_TESTING.md were written
while command jobs (Step 2) were still in progress; update both now
that it's implemented and sim-tested (16/16 e2e tests passing,
including the 6 new command-job ones - see TOMO_QUEUE_TESTING.md
section 4D for what each covers), and record the Redis test-state
contamination gotcha found while testing it.
Also fixes the harness-script path references in SIMULATED_ENDSTATIONS.md
and the two scripts' own docstrings, stale after tests/e2e and
tests/sim_*_harness.py moved to omny_e2e_tests/ (in the previous
commit) - kept out of tests/ specifically so gitea's CI, which runs
pytest against ./csaxs_bec/tests/ unconditionally, doesn't sweep up
tests that need a live sim + Redis.
Covers add-time validation (unknown action, disallowed device,
out-of-range/wrong-type params, unknown kwarg - each rejected without
landing in the queue), the execution-time allow-list re-check as the
real safety boundary, the tomo_queue_actions global var matching the
live registry, tomo_queue_show()/the webpage generator surviving a
command job, legacy kind-less jobs still running as tomograms
alongside a real command-job device move, and the non-idempotent
resume prompt (distinct from the executor's own start-confirmation
prompt).
Also fixes flomni_sim: it only snapshotted/restored whatever was
already in the tomo_queue/tomo_progress global vars, so a stale entry
left behind by an earlier crashed run or manual debugging session
against the same shared sim Redis would contaminate every test that
assumed it started from an empty queue. Now resets both to empty at
setup, in addition to snapshotting/restoring around the test.
Type-tags queue jobs with a "kind" field (back-compat: missing kind
defaults to "tomo"), and adds "command" jobs that reconfigure the
beamline (currently: absolute moves on an allow-listed device, plus an
optimize_idgap stub) between tomograms instead of running a scan.
Command jobs are added via tomo_queue_add_command() against a curated
named-action registry (_TOMO_QUEUE_ACTIONS) - not arbitrary code - so
every queued action is auditable and safe to persist across a kernel
restart. Validation is two-layer: schema checks at add time, and each
action re-checks its own arguments against live hardware at execution
time, which is the actual safety boundary. Crash-resume for a command
job re-runs the whole step sequence from the top if every step is
idempotent, or prompts the operator otherwise.
flomni_webpage_generator.py is guarded against command jobs, which
have no "params" key.
See AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md for the full design.
Sections A-C of TOMO_QUEUE_TESTING.md's checklist, run against a live sim
session: params-restored-per-job, legacy queue migration, empty/all-done
no-ops, start_index semantics, exception/SIGKILL crash-resume, resume-
before-fresh ordering, and concurrent queue edits from a second client.
All 10 tests pass together with no cross-test interference.
The flomni_sim fixture and _bootstrap.py factor out what it takes to
construct a live Flomni against the sim (builtins/reload bootstrap,
side-effect neutralization, RT-feedback/fsamx setup) for reuse by both
in-process tests and the SIGKILL-based subprocess tests.
Also adds the AI_docs/ handoff docs (testing checklist results, command-
jobs plan with the action-registry decisions) that this work updated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The constant lists live scan properties snapshotted/restored by the tomo
queue, not queue state itself -- the old name was misleading. Confirmed
with Mirko. Updates the matching comment in tomo_params.py, which mirrors
this tuple as QUEUE_PARAM_NAMES (unchanged name, own mirror constant).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XrayEyeAlign.gui is a property delegating to lamni.xeyegui and has no
setter. Mock lamni.xeyegui in the update_frame tests; drop the redundant
alignment_images reset, which __init__ already handles.
Drop test_save_frame (method no longer exists). Rewrite test_update_frame
against dev.fsh/dev.cam_xeye instead of the removed LamNI.alignment module.
Patch the module-level umv in test_tomo_rotate rather than builtins.
LamNI.__init__ calls set_client(), which unconditionally accessed
client.gui. This broke test_x_ray_eye_align.py under ClientMock, which
has no gui attribute. Use getattr with a None default; behaviour on a
real BEC client is unchanged.
- Enable DAP fit parameter forwarding (self.gui.set_dap_params_forwarding),
never called before -- without it the dap_params_update signal was never
connected, so no fit data could ever reach fit_params_x/y regardless of
how long anything waited. Disabled again once the fit is read (or on
abort, via the existing KeyboardInterrupt cleanup in lamni.py).
- Replace the fixed 5s sleep before reading the DAP fit with
_wait_for_dap_fit(), which polls fit_params_x/y for the real converged
shape (up to 30s) instead of guessing a duration.
- Fix write_output()'s offset reference: was using one of the 8 angle
points as its own zero-reference (first via an off-by-one at k=1, then
at k=2), which forces that point's offset to exactly 0 by construction
and biases the fit. Now references alignment_values[0] (the FZP centre,
the only angle-independent point in the procedure), matching both
FlOMNI's own reference point and the original SPEC macro
(l_xrayeyealign.mac) exactly, including the asymmetric x/y sign
convention.
- pixel_calibration is now a property reading the cam_xeye device's
pixel_calibration user parameter, matching FlOMNI's camera-sourced
calibration, with the old hardcoded constant kept only as a fallback
(PIXEL_CALIBRATION_DEFAULT) -- by design, to match FlOMNI's own pattern.
- lamni_optics_mixin.py: lfzp_in() and loptics_out() now print before/after
re-establishing interferometer feedback, previously silent.
- device_configs: add the missing userParameter.pixel_calibration to
cam_xeye in ptycho_lamni.yaml and simulated_omny/simulated_lamni.yaml
(present on the FlOMNI side already; this is what the pixel_calibration
property above actually reads at runtime).
- Add lamnigui_stop_gui(), matching flomnigui_stop_gui()
- lamnigui_show_xeyealign(): check/enable cam_xeye live mode on every
call, not just when the xeyegui widget is first created -- matches
flomnigui_show_xeyealign()
- lamnigui_idle(): refresh the splash text on every call rather than
only at first creation -- matches flomnigui_idle()
Replace the plain self.progress dict with a global-var-backed
_ProgressProxy (same shape and key as FlOMNI's), so progress survives
a client restart. Adds heartbeat + idle-time-gap detection in
_tomo_scan_at_angle, ETA computation in _print_progress, and
tomo_start_time/reset handling in tomo_scan(). Adds
tomo_progress_reset() and _format_duration() for parity.
gui_tools.py's progress display already read these fields; no GUI
change needed.
feat(lamni): persist alignment lookup-table corrections
corr_pos_x/corr_pos_y/corr_angle (+ _2 iteration) become
global-var-backed properties (lamni_corr_pos_x, lamni_corr_pos_y,
lamni_corr_angle, ..._2) instead of plain instance lists that reset
on every client restart. No default correction file exists for
LamNI, so reset_correction()/reset_correction_2() just clear to
empty.
feat(lamni): add frames_per_trigger (burst mode) support
LamNIFermatScan already inherits from AsyncFlyScanBase and already
accepts/passes through frames_per_trigger -- only the client-side
plumbing was missing. Adds a validated, global-var-backed
frames_per_trigger property, threads it into
tomo_scan_projection()'s lamni_fermat_scan() call, and exposes it in
tomo_parameters(). Also folds it into the idle-time cadence estimate
alongside stitch tile count.
feat(lamni): add crosshair and structured HDF5 output to X-ray eye alignment
Set the alignment crosshair at the FZP centre (step k=0, raw pixel
coords), hidden on normal completion and on Ctrl-C abort.
update_frame() now actually captures and stores each frame
(previously not stored at all); roi_pixel_data is now collected at
every submit. write_output() additionally writes a timestamped HDF5
file (alignment_values, alignment_images, roi_pixel_data,
alignment_fit) alongside the existing plain-text archival file.
fix(lamni): stop resetting correction state on every XrayEyeAlign instantiation
A fresh XrayEyeAlign is constructed on every call to both
xrayeye_alignment_start() and xrayeye_update_frame(); resetting
corr_pos_x/y/angle and tomo_fit_xray_eye in __init__ meant the
latter -- meant as a lightweight frame check -- silently wiped valid
persisted correction data every time. Reset moved to the start of
align() itself.
fix(lamni): use dev.fsh instead of dev.omnyfsh for the alignment shutter
The shared XRayEye widget monitors and controls dev.fsh; the
alignment script was opening/closing a different device, which
would desync the GUI's shutter toggle from the actual hardware
state.
fix(lamni): correct stale self.align reference in MagLamNI.rotate_slowly
self.align no longer exists as an attribute now that
LamNIAlignmentMixin is mixed directly into LamNI. Use
self.tomo_fovx_offset/self.tomo_fovy_offset directly.
Protocol-level simulation of the flOMNI hardware injected via socket_cls:
real device/controller classes run unchanged against state machines
implementing fgalil.dmc, galil_micos_upr.dmc, the Smaract MCS protocol
and the Orchestra CommunicationServer. Cameras reuse the real device
classes with a synthetic frame source. Includes simulated_flomni device
config (boots referenced at 'in' positions) and an offline harness
covering moves, referencing, rt feedback/tracker, flyer scan readout
and the gripper transfer routine.
Replace 11 identical global-var-backed getter/setter property pairs with a
single _GlobalVarParam descriptor. Behavior is unchanged: same keys, defaults,
and None-handling; stitch_x/stitch_y retain int enforcement via the descriptor's
type_ guard (typeguard.check_type). Net -57 lines with no functional change.
xrayeye_update_frame() now checks all configured beamline states and, if any
are not valid, warns the user (the frame may be empty when the beam is down)
and asks before taking a frame. The automated alignment path is unaffected.
ftransfer_sample_change(): clarify the remove-only prompt to "sample currently
in the sample stage", and include tray slot 20 in the empty-slot scan
(range(1, 20) -> range(1, 21)) in both the remove and mount branches, matching
check_position_is_valid's 0-20 range.
When the eye is already out, optics in, and feedback running, skip the
eye-out/optics-in transition to avoid an interferometer reset. Enables
repeated alignment scans and interleaving alignment into a tomogram
without leaving the current measurement position.
feedback_disable/enable_with_reset cycle re-zeros the interferometers
and moves off the current sample position. Only run it when the FZP
actually needs to move; add force_feedback_reset kwarg and _ffzp_is_in
helper.
X-Ray Eye alignment GUI + script now share a single, device-configured
pixel calibration and the GUI shows live ROI dimensions.
GUI (x_ray_eye.py):
- Add read-only "ROI size" field below the message box, showing the active
ROI's x/y extent, updated live via each ROI's sigRegionChanged. Rectangles
report width/height; circular ROIs read raw state["size"] so an ellipse
state would be reflected (currently aspect-locked upstream).
- Add "Calibration" field showing microns/pixel, or "uncalibrated (pixels)"
when the camera parameter is absent.
- Resolve mm/pixel from dev.cam_xeye.user_parameter["pixel_calibration"];
fall back to raw pixels (scale 1.0, unit "px") if unavailable.
- Enlarge the message field from 60 to 90 px height.
Alignment script (x_ray_eye_align.py):
- Replace the hard-coded PIXEL_CALIBRATION constant with a pixel_calibration
property reading the same cam_xeye user parameter; fall back to
PIXEL_CALIBRATION_DEFAULT if unavailable.
- Fold the recurring factor of 2 into the calibration: xval_x_* come straight
from raw ROI pixels on submit, so the true scale is 0.05/113 mm/pixel.
Default is now 0.05/113 and the three /2 factors (FZP center x, height
centering, FOV) are removed. Computed physical values are unchanged.
Device config (cam_xeye_pixel_calibration.yaml):
- Add pixel_calibration userParameter (0.05/113 mm per raw ROI pixel) to
cam_xeye. Merge the userParameter block into the existing device entry.
scilog_last_scans now puts the user comment at the top of the entry and
drops the flOMNI logo attachment. number_of_scans accepts 0 to send a
comment-only entry. When fewer projection records exist than requested
(e.g. non-ptycho scans, which write no timing record), the available ones
are used and a note reports how many of the requested count were found.
At BEC session start, compare bec.active_account against the new global
var defaults_applied_for_account. On a genuine account change, interactively
offer to reset all tomo scan parameters to their defaults for the new
experiment; a client restart within the same account is a silent no-op.
Add _set_default_tomo_params() writing all 20 scan parameters via the
property setters.
process looks like a BEC client), plus a delayed (500 ms) backup
stop_devices request, defaulting to stop-all so the widget doubles as
a generic emergency stop when opened standalone from the launcher menu
Mirko Holler
appel_candx01dc
