diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/SIMULATED_ENDSTATIONS.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/SIMULATED_ENDSTATIONS.md index 412a90f..9d0ae74 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/SIMULATED_ENDSTATIONS.md +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/SIMULATED_ENDSTATIONS.md @@ -63,9 +63,10 @@ Protocol fidelity is grounded in the actual hardware-side sources: `fgalil.dmc`, `simulated_omny.yaml` (generated), `simulated_bl_endstation.yaml` (pre-existing) - `bin/generate_simulated_configs.py` — regenerates the LamNI/OMNY configs from `ptycho_lamni.yaml`/`ptycho_omny.yaml` (rerun after real-config changes) -- `tests/sim_flomni_harness.py`, `tests/sim_lamni_omny_harness.py` — offline harnesses - (no BEC deployment needed): 39 + 26 checks covering moves, referencing, feedback, - tracker, scans through the flyers, gripper transfers, storage, cameras +- `omny_e2e_tests/sim_flomni_harness.py`, `omny_e2e_tests/sim_lamni_omny_harness.py` — + offline harnesses (no BEC deployment needed): 39 + 26 checks covering moves, + referencing, feedback, tracker, scans through the flyers, gripper transfers, + storage, cameras ## Real-code fixes made along the way (separate commits) @@ -151,4 +152,4 @@ harness cannot see. `sim_stppermm`, `sim_encpermm`, `sim_referenced`, `sim_analog_inputs`, `sim_digital_inputs/outputs`, `sim_variables`, `sim_limit_low/high_active`, `sim_point_dwell_s`, `sim_noise_std`, `sim_samples`. -- Harnesses run standalone: `python tests/sim_flomni_harness.py` (repo on PYTHONPATH). +- Harnesses run standalone: `python omny_e2e_tests/sim_flomni_harness.py` (repo on PYTHONPATH). diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md index 745120c..1c19f99 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md @@ -1,16 +1,23 @@ # Plan: Command Jobs in the FlOMNI Tomo Queue -**Status:** design agreed, **and the §10 open questions are now resolved with Mirko** -(one small detail — `optimize_idgap`'s peak-detection signal — deferred to the -implementation session). **Step 1 (executor rework) is implemented and its sim -checklist has passed** (`TOMO_QUEUE_TESTING.md` §4, sections A–C). Steps 2 and 3 -below are not yet written — this is now unblocked to start. Implement in a dedicated -session — this touches the CLI (`flomni.py`), the GUI widget (`tomo_params.py` / -`TomoQueueDialog`), and the web page generator (`flomni_webpage_generator.py`). +**Status:** **Step 2 is implemented** in `flomni.py` (schema type-tagging, action +registry, `move` + `optimize_idgap` (stub) actions, `tomo_queue_add_command`, +`tomo_queue_show`/`tomo_queue_execute` branching, `_run_command_job` resume policy, +`tomo_queue_actions` global-var publishing) plus a small additive fix in +`flomni_webpage_generator.py` so a command job renders its step summary instead of a +blank params table. Sim-tested: `omny_e2e_tests/test_tomo_queue_command_jobs.py`, +6/6 passing (see `TOMO_QUEUE_TESTING.md` §4D for what each test covers); real device +movement was exercised against `ftray` (sim-testable stand-in), not `mokev`/`idgap` +(real front-end only — see §10). **Step 3 (the `TomoQueueDialog` GUI work) is not +started** — a separate, larger PyQt change. Step 1 (executor rework) is implemented +and its sim checklist has passed (`TOMO_QUEUE_TESTING.md` §4, sections A–C). **Prerequisite (met):** the checklist in `TOMO_QUEUE_TESTING.md` has passed against the simulated flOMNI. That document is the companion to this one: it covers *what -Step 1 changed and how to test it*; this one covers *what comes next*. +Step 1 changed and how to test it*; this one covers *what comes next*. The e2e test +suite for both steps lives in `omny_e2e_tests/` at the repo root (moved out of +`tests/e2e/` so it doesn't get swept into gitea's generic `tests/` CI discovery — +these tests need a live sim + Redis, not just `pytest`). > **This document supersedes the earlier version of the same name.** The earlier > draft assumed the old index-based executor, single-action command jobs, and left diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_TESTING.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_TESTING.md index d47b9b4..ac1ebee 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_TESTING.md +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_TESTING.md @@ -9,9 +9,18 @@ step — command jobs — which is **not** implemented). C of the checklist have now passed against the live sim** (10/10 tests, run together with no cross-test interference — see §4). **All of §6's open questions are now resolved with Mirko** (registry/allow-list in `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` §10; -the `_TOMO_SCAN_PARAM_NAMES` rename is done) — **Step 2 is unblocked to start.** Only -**D (real instrument)**, needing beam time, remains open. Steps 2 and 3 are designed -but not written. +the `_TOMO_SCAN_PARAM_NAMES` rename is done). **Step 2 (command jobs) is now +implemented and sim-tested too** — §4A gained a fourth checklist section, D, covering +`omny_e2e_tests/test_tomo_queue_command_jobs.py` (6/6 passed). §5's description of +Step 2 is otherwise still accurate as the as-built design. Only **§4's section E (real +instrument)**, needing beam time, and **Step 3 (the GUI)** remain open — Step 3 is +designed (§5) but not started. + +**Test folder moved:** the e2e suite (and the two sim-harness scripts that bootstrap +it) now live in `omny_e2e_tests/` at the repo root, not `tests/e2e/` — kept out of +`tests/` specifically so it doesn't get swept up by gitea's generic `tests/` +discovery/CI run (these tests need a live sim + Redis, not just `pytest`). Run them +directly: `pytest omny_e2e_tests/`. --- @@ -124,7 +133,7 @@ config with BEC's demo devices**. Write our own fixture that reuses the plumbing loads `device_configs/simulated_omny/simulated_flomni.yaml` explicitly or assumes the running session already has it. -### The fixture: `tests/e2e/_bootstrap.py` + `tests/e2e/conftest.py` → `flomni_sim` — BUILT +### The fixture: `omny_e2e_tests/_bootstrap.py` + `omny_e2e_tests/conftest.py` → `flomni_sim` — BUILT This is the reusable piece — worth more than the checklist itself. It's built and factored into two files so section B/C tests (which need a real, separate OS process @@ -199,7 +208,7 @@ interruptible mid-scan for the crash-resume tests (§4, B5). ## 4. THE CHECKLIST -### A. Sim, no crash needed — ✅ PASSED (`tests/e2e/test_tomo_queue.py`) +### A. Sim, no crash needed — ✅ PASSED (`omny_e2e_tests/test_tomo_queue.py`) 1. **✅ Params actually restored per job.** `test_params_restored_per_job`. Two jobs with distinct `tomo_angle_stepsize` (89.9 → N=2, 59.9 → N=3); a background thread @@ -220,7 +229,7 @@ interruptible mid-scan for the crash-resume tests (§4, B5). ### B. Sim, crash / interrupt paths — ✅ PASSED 5. **✅ Real crash-resume.** `test_real_crash_resume` - (`tests/e2e/test_tomo_queue_crash_subprocess.py`). `tomo_queue_execute()` run as a + (`omny_e2e_tests/test_tomo_queue_crash_subprocess.py`). `tomo_queue_execute()` run as a real OS subprocess (required — see §3), `SIGKILL`ed mid-job1. Confirmed via the subprocess's own log: killed at `subtomo 3, angle 45.0`; the resume subprocess printed `Resuming tomo scan at subtomo 3, angle 45.000 deg` and re-acquired that @@ -228,14 +237,14 @@ interruptible mid-scan for the crash-resume tests (§4, B5). after the resume subprocess starts, then comparing the full snapshot) before the rest of the queue completed. Assertion design agreed beforehand. 6. **✅ Exception mid-scan → pause.** `test_exception_mid_scan_pauses_queue` - (`tests/e2e/test_tomo_queue_crash.py`). `tomo_scan` patched to raise on its first + (`omny_e2e_tests/test_tomo_queue_crash.py`). `tomo_scan` patched to raise on its first call only. Job → `incomplete`, exception re-raised out of `tomo_queue_execute()`, later job stayed `pending`; re-running resumed the failed job first. 7. **✅ Resume-before-fresh, explicitly.** `test_resume_before_fresh_explicit` (same file). An `incomplete` job below a fresh `pending` one in list order ran first regardless of position (tracked via an `update_by_id` call-order spy). -### C. Concurrency — the path the GUI will use — ✅ PASSED (`tests/e2e/test_tomo_queue_concurrency.py`) +### C. Concurrency — the path the GUI will use — ✅ PASSED (`omny_e2e_tests/test_tomo_queue_concurrency.py`) 8. **✅ Edit the queue from a second client while one runs.** `test_concurrent_queue_edits_from_second_client`. Session A = subprocess executing @@ -250,23 +259,74 @@ interruptible mid-scan for the crash-resume tests (§4, B5). just returns `False`), job1's scan finished on its own terms, job2 still ran and was marked `done` correctly. -### D. Real instrument (not sim) — NOT DONE, needs Mirko / beam time +### D. Command jobs (Step 2) — ✅ PASSED (`omny_e2e_tests/test_tomo_queue_command_jobs.py`) -10. One queued tomogram end-to-end on the real instrument, to confirm the sim's +10. **✅ Add-time validation rejects bad steps, and rejects them without landing + anything in the queue.** `test_add_command_validation_errors`. Unknown action, + a device not on the `move` allow-list, an out-of-range/wrong-type + `optimize_idgap.search_range`, and an unknown kwarg are each rejected with the + expected `ValueError`; afterward the queue is confirmed still empty. +11. **✅ Execution-time allow-list re-check is the real safety boundary.** + `test_move_execution_time_allowlist_recheck`. A job with a disallowed device + appended directly (bypassing `tomo_queue_add_command`) is refused by + `_queue_action_move()` itself at run time and correctly left `incomplete`, not + silently marked `done`. +12. **✅ `tomo_queue_actions` global var matches the live registry.** + `test_tomo_queue_actions_global_var_matches_registry`. Confirms the published + var (for the future GUI, §7 of the plan doc) isn't a stale/hand-maintained copy. +13. **✅ `tomo_queue_show()` and the web page generator survive a command job.** + `test_show_and_webpage_survive_command_job`. A command job (no `params` key) + renders its step summary + idempotency marker in both, instead of `KeyError`ing. +14. **✅ Back-compat + real dispatch, combined.** + `test_legacy_job_no_kind_defaults_to_tomo_and_command_moves_ftray`. A `kind`-less + legacy queue entry still runs as a tomogram; in the same mixed-queue run, a + command job actually moves a device (`ftray`, the sim-testable stand-in for + `mokev`/`idgap` — see the plan doc §3.1) via the real `move` action path. +15. **✅ Non-idempotent resume prompts exactly once, and respects "no".** + `test_non_idempotent_command_resume_prompts`. A command job flagged + `idempotent=False`, left `incomplete`, triggers the resume prompt (distinct from + the executor's own top-level "OK to start?" prompt) on the next + `tomo_queue_execute()`; answering "no" leaves the device unmoved and marks the + job `done` without re-running it. + +### E. Real instrument (not sim) — NOT DONE, needs Mirko / beam time + +16. One queued tomogram end-to-end on the real instrument, to confirm the sim's timing/progress semantics match reality closely enough that resume lands on the right angle. +17. `move` on `mokev`/`idgap` end-to-end — neither device is in the flomni sim + config (see the plan doc §3.1), so Step 2's `move` action itself has only been + sim-exercised via `ftray`, never against the real devices it's actually for. -**Gate:** A–C passed on the sim (10/10 tests, run together, no cross-test -interference — ~18 min). **Step 1 is verified enough to build Steps 2/3 on.** D and -§6 remain open before Step 2 work should actually start. +**Gate:** A–D passed on the sim (16/16 tests total). **Steps 1 and 2 are verified +enough to build Step 3 (the GUI) on.** E remains open, and blocks calling `move` on +`mokev`/`idgap` production-ready. + +### A gotcha found testing Step 2 (not a Step-2 bug) + +**The sim's shared Redis is not reset between separate `pytest` invocations, and the +`flomni_sim` fixture only snapshots/restores whatever's already there — it does not +reset to empty.** A prior run that errored out before reaching its `finally` block +(or manual poking at the live sim) can leave a stale job stuck mid-queue — e.g. at +`status: "running"`, or written in a pre-`kind`-field format — that then contaminates +every subsequent test run that assumes it starts from an empty queue: expect +spurious failures like "queue not empty after a rejected add" or `KeyError: 'kind'` +on an old-format entry that isn't actually related to the test that hit it. Fixed by +having `flomni_sim` reset `tomo_queue`/`tomo_progress` to empty at setup (still +snapshotting/restoring around that, so a real operator session on the same sim isn't +disturbed) — see `conftest.py`. If this resurfaces, check +`redis-cli get "user/vars/tomo_queue"` for leftover state before assuming it's a +real regression. --- -## 5. Steps 2 & 3 — designed, NOT implemented +## 5. Steps 2 & 3 — Step 2 implemented and sim-tested (§4D), Step 3 still just designed -Full design for command jobs is in `TOMO_QUEUE_COMMAND_JOBS_PLAN.md`. Decisions -settled since that document was written — **it is not fully up to date; this section -supersedes it where they conflict**: +Full design for command jobs is in `TOMO_QUEUE_COMMAND_JOBS_PLAN.md`, which is now +up to date with what actually shipped in Step 2 (both documents describe the same +as-built behaviour; this section is kept as a second description for convenience, +not because they disagree). Step 3 (the GUI, below) is unstarted — this section is +still just its design. ### Command jobs (Step 2) @@ -329,12 +389,10 @@ Agreed interaction model: fine), and it gives false confidence. - Execution stays **CLI-only**; the GUI server process can't run the blocking scan loop. -### Webpage generator (`flomni_webpage_generator.py`) +### Webpage generator (`flomni_webpage_generator.py`) — DONE -Not yet touched. When command jobs land, any code doing `job["params"][...]` will -`KeyError` on a command job (no `params` key) — guard with -`job.get("kind", "tomo") == "tomo"`. Also re-check projection/ETA counting so command -jobs don't skew tomogram-based estimates. +Guarded against command jobs (`job.get("kind", "tomo") == "tomo"` before touching +`job["params"]`), verified by §4D13 (`test_show_and_webpage_survive_command_job`). --- diff --git a/omny_e2e_tests/sim_flomni_harness.py b/omny_e2e_tests/sim_flomni_harness.py index 4c15c88..92c7a58 100644 --- a/omny_e2e_tests/sim_flomni_harness.py +++ b/omny_e2e_tests/sim_flomni_harness.py @@ -14,7 +14,7 @@ paths used by the flomni IPython-client scripts: 7. Gripper sample transfer routine (#MNTMODE, #GRGET incl. confirm handshake) 8. Simulated cameras (frame generation) -Run with the repo root on PYTHONPATH: python tests/sim_flomni_harness.py +Run with the repo root on PYTHONPATH: python omny_e2e_tests/sim_flomni_harness.py """ from __future__ import annotations diff --git a/omny_e2e_tests/sim_lamni_omny_harness.py b/omny_e2e_tests/sim_lamni_omny_harness.py index 870418e..b388d5a 100644 --- a/omny_e2e_tests/sim_lamni_omny_harness.py +++ b/omny_e2e_tests/sim_lamni_omny_harness.py @@ -4,7 +4,7 @@ Offline harness for the simulated LamNI and OMNY hardware stacks. Exercises the real device classes against the protocol simulations without a BEC deployment, covering the paths used by the LamNI/OMNY IPython-client scripts. -Run with the repo root on PYTHONPATH: python tests/sim_lamni_omny_harness.py +Run with the repo root on PYTHONPATH: python omny_e2e_tests/sim_lamni_omny_harness.py """ from __future__ import annotations