@@ -9,9 +9,18 @@ step — command jobs — which is **not** implemented).
C of the checklist have now passed against the live sim** (10/10 tests, run together
with no cross-test interference — see §4). **All of §6's open questions are now
resolved with Mirko** (registry/allow-list in `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` §10;
the `_TOMO_SCAN_PARAM_NAMES` rename is done) — **Step 2 is unblocked to start. ** Only
**D (real instrument) ** , needing beam time, remains open. Steps 2 and 3 are designed
but not written.
the `_TOMO_SCAN_PARAM_NAMES` rename is done). **Step 2 (command jobs) is now
implemented and sim-tested too** — §4A gained a fourth checklist section, D, covering
`omny_e2e_tests/test_tomo_queue_command_jobs.py` (6/6 passed). §5's description of
Step 2 is otherwise still accurate as the as-built design. Only * * §4's section E (real
instrument)**, needing beam time, and **Step 3 (the GUI) ** remain open — Step 3 is
designed (§5) but not started.
**Test folder moved: ** the e2e suite (and the two sim-harness scripts that bootstrap
it) now live in `omny_e2e_tests/` at the repo root, not `tests/e2e/` — kept out of
`tests/` specifically so it doesn't get swept up by gitea's generic `tests/`
discovery/CI run (these tests need a live sim + Redis, not just `pytest` ). Run them
directly: `pytest omny_e2e_tests/` .
---
@@ -124,7 +133,7 @@ config with BEC's demo devices**. Write our own fixture that reuses the plumbing
loads `device_configs/simulated_omny/simulated_flomni.yaml` explicitly or assumes the
running session already has it.
### The fixture: `tests/e2e/ _bootstrap.py` + `tests/e2e/ conftest.py` → `flomni_sim` — BUILT
### The fixture: `omny_e2e_ tests/_bootstrap.py` + `omny_e2e_ tests/conftest.py` → `flomni_sim` — BUILT
This is the reusable piece — worth more than the checklist itself. It's built and
factored into two files so section B/C tests (which need a real, separate OS process
@@ -199,7 +208,7 @@ interruptible mid-scan for the crash-resume tests (§4, B5).
## 4. THE CHECKLIST
### A. Sim, no crash needed — ✅ PASSED (` tests/e2e/ test_tomo_queue.py`)
### A. Sim, no crash needed — ✅ PASSED (` omny_e2e_ tests/test_tomo_queue.py`)
1. **✅ Params actually restored per job.** ` test_params_restored_per_job`. Two jobs
with distinct ` tomo_angle_stepsize` (89.9 → N=2, 59.9 → N=3); a background thread
@@ -220,7 +229,7 @@ interruptible mid-scan for the crash-resume tests (§4, B5).
### B. Sim, crash / interrupt paths — ✅ PASSED
5. **✅ Real crash-resume.** ` test_real_crash_resume`
(` tests/e2e/ test_tomo_queue_crash_subprocess.py`). ` tomo_queue_execute()` run as a
(` omny_e2e_ tests/test_tomo_queue_crash_subprocess.py`). ` tomo_queue_execute()` run as a
real OS subprocess (required — see §3), ` SIGKILL`ed mid-job1. Confirmed via the
subprocess's own log: killed at ` subtomo 3, angle 45.0`; the resume subprocess
printed ` Resuming tomo scan at subtomo 3, angle 45.000 deg` and re-acquired that
@@ -228,14 +237,14 @@ interruptible mid-scan for the crash-resume tests (§4, B5).
after the resume subprocess starts, then comparing the full snapshot) before the
rest of the queue completed. Assertion design agreed beforehand.
6. **✅ Exception mid-scan → pause.** ` test_exception_mid_scan_pauses_queue`
(` tests/e2e/ test_tomo_queue_crash.py`). ` tomo_scan` patched to raise on its first
(` omny_e2e_ tests/test_tomo_queue_crash.py`). ` tomo_scan` patched to raise on its first
call only. Job → ` incomplete`, exception re-raised out of ` tomo_queue_execute()`,
later job stayed ` pending`; re-running resumed the failed job first.
7. **✅ Resume-before-fresh, explicitly.** ` test_resume_before_fresh_explicit`
(same file). An ` incomplete` job below a fresh ` pending` one in list order ran
first regardless of position (tracked via an ` update_by_id` call-order spy).
### C. Concurrency — the path the GUI will use — ✅ PASSED (` tests/e2e/ test_tomo_queue_concurrency.py`)
### C. Concurrency — the path the GUI will use — ✅ PASSED (` omny_e2e_ tests/test_tomo_queue_concurrency.py`)
8. **✅ Edit the queue from a second client while one runs.**
` test_concurrent_queue_edits_from_second_client`. Session A = subprocess executing
@@ -250,23 +259,74 @@ interruptible mid-scan for the crash-resume tests (§4, B5).
just returns ` False`), job1's scan finished on its own terms, job2 still ran and
was marked ` done` correctly.
### D. Real instrument (not sim) — NOT DONE, needs Mirko / beam time
### D. Command jobs (Step 2) — ✅ PASSED (` omny_e2e_tests/test_tomo_queue_command_jobs.py `)
10. One queued tomogram end-to-end on the real instrument, to confirm the sim's
10. **✅ Add-time validation rejects bad steps, and rejects them without landing
anything in the queue.** ` test_add_command_validation_errors`. Unknown action,
a device not on the ` move` allow-list, an out-of-range/wrong-type
` optimize_idgap.search_range`, and an unknown kwarg are each rejected with the
expected ` ValueError`; afterward the queue is confirmed still empty.
11. **✅ Execution-time allow-list re-check is the real safety boundary.**
` test_move_execution_time_allowlist_recheck`. A job with a disallowed device
appended directly (bypassing ` tomo_queue_add_command`) is refused by
` _queue_action_move()` itself at run time and correctly left ` incomplete`, not
silently marked ` done`.
12. **✅ ` tomo_queue_actions` global var matches the live registry.**
` test_tomo_queue_actions_global_var_matches_registry`. Confirms the published
var (for the future GUI, §7 of the plan doc) isn't a stale/hand-maintained copy.
13. **✅ ` tomo_queue_show()` and the web page generator survive a command job.**
` test_show_and_webpage_survive_command_job`. A command job (no ` params` key)
renders its step summary + idempotency marker in both, instead of ` KeyError`ing.
14. **✅ Back-compat + real dispatch, combined.**
` test_legacy_job_no_kind_defaults_to_tomo_and_command_moves_ftray`. A ` kind`-less
legacy queue entry still runs as a tomogram; in the same mixed-queue run, a
command job actually moves a device (` ftray`, the sim-testable stand-in for
` mokev`/` idgap` — see the plan doc §3.1) via the real ` move` action path.
15. **✅ Non-idempotent resume prompts exactly once, and respects "no".**
` test_non_idempotent_command_resume_prompts`. A command job flagged
` idempotent=False`, left ` incomplete`, triggers the resume prompt (distinct from
the executor's own top-level "OK to start?" prompt) on the next
` tomo_queue_execute()`; answering "no" leaves the device unmoved and marks the
job ` done` without re-running it.
### E. Real instrument (not sim) — NOT DONE, needs Mirko / beam time
16. One queued tomogram end-to-end on the real instrument, to confirm the sim's
timing/progress semantics match reality closely enough that resume lands on the
right angle.
17. ` move` on ` mokev`/` idgap` end-to-end — neither device is in the flomni sim
config (see the plan doc §3.1), so Step 2's ` move` action itself has only been
sim-exercised via ` ftray`, never against the real devices it's actually for.
**Gate:** A– C passed on the sim (10 /10 tests, run together, no cross-test
interference — ~18 min). **Step 1 is verified enough to build Steps 2/3 on.** D and
§6 remain open before Step 2 work should actually start .
**Gate:** A– D passed on the sim (16 /16 tests total). **Steps 1 and 2 are verified
enough to build Step 3 (the GUI) on.** E remains open, and blocks calling ` move` on
` mokev `/` idgap ` production-ready .
### A gotcha found testing Step 2 (not a Step-2 bug)
**The sim's shared Redis is not reset between separate ` pytest` invocations, and the
` flomni_sim` fixture only snapshots/restores whatever's already there — it does not
reset to empty.** A prior run that errored out before reaching its ` finally` block
(or manual poking at the live sim) can leave a stale job stuck mid-queue — e.g. at
` status: "running"`, or written in a pre-` kind`-field format — that then contaminates
every subsequent test run that assumes it starts from an empty queue: expect
spurious failures like "queue not empty after a rejected add" or ` KeyError: 'kind'`
on an old-format entry that isn't actually related to the test that hit it. Fixed by
having ` flomni_sim` reset ` tomo_queue`/` tomo_progress` to empty at setup (still
snapshotting/restoring around that, so a real operator session on the same sim isn't
disturbed) — see ` conftest.py`. If this resurfaces, check
` redis-cli get "user/vars/tomo_queue"` for leftover state before assuming it's a
real regression.
---
## 5. Steps 2 & 3 — designed, NOT implement ed
## 5. Steps 2 & 3 — Step 2 implemented and sim-tested (§4D), Step 3 still just design ed
Full design for command jobs is in ` TOMO_QUEUE_COMMAND_JOBS_PLAN.md`. Decisions
settled since that document was written — **it is not fully up to date; this section
supersedes it where they conflict**:
Full design for command jobs is in ` TOMO_QUEUE_COMMAND_JOBS_PLAN.md`, which is now
up to date with what actually shipped in Step 2 (both documents describe the same
as-built behaviour; this section is kept as a second description for convenience,
not because they disagree). Step 3 (the GUI, below) is unstarted — this section is
still just its design.
### Command jobs (Step 2)
@@ -329,12 +389,10 @@ Agreed interaction model:
fine), and it gives false confidence.
- Execution stays **CLI-only**; the GUI server process can't run the blocking scan loop.
### Webpage generator (` flomni_webpage_generator.py`)
### Webpage generator (` flomni_webpage_generator.py`) — DONE
Not yet touched. When command jobs la nd, any code doing ` job["params"][...] ` will
` KeyError ` on a command job (no ` params ` key) — guard with
` job.get("kind", "tomo") == "tomo"`. Also re-check projection/ETA counting so command
jobs don't skew tomogram-based estimates.
Guarded against command jobs (` job.get("ki nd" , "tomo") == "tomo" ` before touching
` job["params"] `), verified by §4D13 (` test_show_and_webpage_survive_ command_ job`).
---
Mirko Holler