From d8b7737cc743ac66ac0cbcb2aea6f43ea855f0e1 Mon Sep 17 00:00:00 2001 From: x01dc Date: Tue, 14 Jul 2026 13:03:24 +0200 Subject: [PATCH] docs(flomni): document the declined-scan and stale-running-job fixes User manual: notes that declining the fermat-scan confirmation now fails the job instead of silently skipping a projection, plus how to recover a job stuck at "running" (tomo_queue_delete has no status guard; update_by_id to fix the status directly; the GUI's staleness override). AI_docs: plan section 3.4 records the second real incident (declined confirmation silently returning) and its fix; section 6.3 records the stale-"running" GUI guard fix. GUI test checklist gets section 6e (test steps 48-52) and section 1 item 12, status line updated to flag 6c/6d/6e as not yet clicked through. Co-Authored-By: Claude Sonnet 5 --- .../AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md | 34 ++++++++++++ .../flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md | 55 +++++++++++++++++-- docs/user/ptychography/flomni.md | 21 +++++-- 3 files changed, 101 insertions(+), 9 deletions(-) diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md index 7adf640..5bf05a8 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md @@ -355,6 +355,24 @@ is sorted; the webpage generator's `current_params` payload includes environment, no `pytest-qt`) or the rendered webpage HTML/JS — see `TOMO_QUEUE_GUI_TESTING.md` for the manual checklist covering both. +**Second real incident, found at the beamline: a declined confirmation silently did +nothing.** A hook called `tomo_scan_projection(angle)` without `_internal=True` while +`single_point_instead_of_fermat_scan` was on. That combination hits +`tomo_scan_projection()`'s own "Run a fermat scan anyway?" confirmation (unrelated to +the hook mechanism itself, but only reachable in practice through a hook or a direct +CLI call) — at *every* projection angle, since the hook runs once per angle. +Declining used to just print "Aborted." and `return`, with no exception -- so nothing +signaled the executor that a projection was skipped. The eventual `Ctrl-C` (a +`KeyboardInterrupt`, not caught by `tomo_queue_execute()`'s `except Exception`) left +the job stuck at `running` forever, which is the same failure mode the row-lock note +in §6.3 now documents. Fixed at the source: declining now raises `FlomniError` +instead of returning silently, so — combined with the `scan_repeat` `exc_handler` fix +above — a decline correctly fails the job immediately and marks it `incomplete` +rather than either silently dropping data or leaving an untraceable stuck state. +Sim-verified: a direct declined call raises with a clear message; through the queue, +via a hook missing `_internal=True`, the job ends up `incomplete`, not stuck +`running` or silently `done`. + --- ## 4. New / changed CLI methods (`flomni.py`) @@ -503,6 +521,22 @@ marked `running` while a queue run marks exactly one — so per-row status is th - `pending` / `incomplete`: freely draggable and deletable. - `done`: pinned (harmless to move, but it makes the list confusing to read). +**Real incident: `running` isn't always the live item.** The rule above assumes +*something* transitions a `running` job onward -- either the scan finishes, or an +exception marks it `incomplete`. Neither happens if the process dies hard enough to +skip both (a `KeyboardInterrupt`, which `tomo_queue_execute()`'s `except Exception` +does not catch since `KeyboardInterrupt` isn't an `Exception`; a kernel restart). The +job is then stuck at `running` forever with no live process behind it, and the +GUI's delete/clear guard -- keyed purely on that status string -- blocked it +unconditionally, with no escape from the GUI at all (the CLI's `tomo_queue_delete()` +has no such guard and already worked, but that isn't discoverable from the GUI). +Fixed by checking `tomo_progress`'s heartbeat (same signal/window as +`TomoParamsWidget._is_tomo_running()`, 120 s): a `running` job with a stale or +missing heartbeat is very likely orphaned, so delete/clear offer an explicit +"looks stale -- proceed anyway?" confirmation instead of refusing outright. A +genuinely fresh heartbeat still blocks as before -- this narrows the guard's blind +spot, it doesn't remove the guard. + ### 6.4 Reorder is allowed **mid-run** It takes effect at the next job boundary: the executor finishes the running job, marks diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md index 10f36ed..998d697 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md @@ -12,10 +12,12 @@ gaps (all fixed, see section 6c below): a confusing duplicate row-number column, `tomo_parameters()`/the queue table/scilog/the PDF report/the progress-ring label not showing an active hook at all, and a clarification that `unregister_at_each_angle_hook()` doesn't clear `at_each_angle_hook` -(confirmed expected, now documented). **Sections 6c and 6d are new and not -yet clicked through** — 6c is the fixes above, 6d is the new "Load into -editor" feature (reuse a queued job's settings as a new edit). Companion -docs: +(confirmed expected, now documented). **Sections 6c, 6d, and 6e are new and +not yet clicked through** — 6c is the fixes above, 6d is the new "Load into +editor" feature (reuse a queued job's settings as a new edit), 6e is a +declined-scan/stuck-queue incident found live at the beamline and its fix +(CLI-side sim-verified; the GUI delete/clear override itself is not). +Companion docs: `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` section 6 (the design this implements), `TOMO_QUEUE_TESTING.md` (Steps 1 & 2, already sim-validated). @@ -131,6 +133,16 @@ support on top of it: they already do. Confirms before discarding an in-progress edit. Disabled outside a valid single-tomo-job selection and during sort mode. See section 6d below — not yet click-tested. +12. **Declined-scan failure + stale "running" recovery.** Two fixes for a + real stuck-queue incident: `tomo_scan_projection()` now raises instead + of silently returning when the "run a fermat scan anyway?" confirmation + is declined, so a queue job fails cleanly (`incomplete`) instead of + silently dropping a projection or (after an eventual Ctrl-C) getting + stuck at `running` with no live process. And delete/clear on a + `running` row now check `tomo_progress`'s heartbeat: stale or missing + offers a "proceed anyway?" override instead of refusing forever; a + fresh heartbeat still blocks outright. See section 6e below — not yet + click-tested. --- @@ -404,6 +416,41 @@ GUI bug and reporting gaps for the hook feature. 47. Confirm "Load into editor" is also disabled while **Sort queue…** mode is active, consistent with the other queue-mutating buttons. +## 6e. Test plan — new: declined-scan failure + stale "running" recovery + +Found at the beamline (thanks again, Mirko) — a hook without `_internal=True` +plus single-point mode got stuck asking for confirmation at every angle, and +answering "no" (repeatedly) then Ctrl-C'ing left a job stuck at `running` +forever with the GUI unable to delete it. + +48. Set `flomni.single_point_instead_of_fermat_scan = True`, then call + `flomni.tomo_scan_projection(10.0)` directly from the CLI. Answer **no** + to "Run a fermat scan anyway?". Confirm it now **raises** (a traceback, + not just "Aborted." with a silent return). +49. Register a hook that calls `tomo_scan_projection(angle)` **without** + `_internal=True`, activate it, queue a job with single-point mode on, + and run the queue. When asked to confirm the fermat scan, answer **no**. + Confirm the job ends up `incomplete` (not stuck `running`, not silently + `done` with missing projections) and `tomo_queue_execute()` prints a + clear "did not complete" message naming the actual problem. +50. Hand-simulate the stuck-`running` scenario: pick a `pending` job's id + (`flomni._tomo_queue_proxy.as_list()`) and + `flomni._tomo_queue_proxy.update_by_id(job_id, status="running")` + directly (standing in for "the process died mid-scan"), with no scan + actually active (`tomo_progress`'s heartbeat absent or >120s old). + In the GUI, select that row and click **Delete selected**. Confirm you + get the **new** "looks stale — proceed anyway?" dialog (not the old + unconditional refusal), and that clicking **Yes** actually deletes it + after the normal delete confirmation. Repeat for **Clear all**. +51. Same setup, but with `tomo_progress`'s heartbeat fresh (<120s old) — + e.g. start a real short scan running so the heartbeat updates. Confirm + delete/clear on the `running` row still refuse outright, no staleness + override offered — a genuinely active job must still be unaffected. +52. Cross-check from the CLI: `flomni.tomo_queue_delete(index)` should + always work on a stuck `running` row regardless of GUI state (no + status guard there at all) — useful as a fallback even without the + GUI open. + ## 7. Known limitations of this first iteration (by design, not bugs) Worth knowing before you file something as broken: diff --git a/docs/user/ptychography/flomni.md b/docs/user/ptychography/flomni.md index e3457b2..7d89934 100644 --- a/docs/user/ptychography/flomni.md +++ b/docs/user/ptychography/flomni.md @@ -348,6 +348,8 @@ Several tomo parameter sets can be queued and run sequentially on the same sampl The queue is persisted (it survives a BEC client restart). Each job's status is one of `pending`, `running`, `incomplete`, or `done`. A job that did not run to completion (an exception was caught, or the BEC client itself crashed mid-scan) is automatically resumed - rather than restarted - the next time `flomni.tomo_queue_execute()` is called. +If the process running the queue dies hard enough that it never gets to write a status update at all (a kernel restart, a Ctrl-C during a blocking prompt) a job can be left stuck showing `running` with nothing actually running. `flomni.tomo_queue_delete(index)` has no status guard and works on a stuck row regardless; you can also fix the status directly with `flomni._tomo_queue_proxy.update_by_id(job_id, status="incomplete")` (to make it resumable) or `status="done"` (to skip it). The GUI's Delete/Clear normally refuse to touch a `running` row, but detect a stale one (no recent scan heartbeat) and offer to proceed anyway with an explicit confirmation instead of blocking forever. + Example: ``` flomni.tomo_parameters() # set up parameter set #1 @@ -482,11 +484,20 @@ projection; `tomo_acquire_at_angle(angle)` always runs a single-point acquisitio Use whichever matches how `tomo_parameters()` has this job configured (`single_point_instead_of_fermat_scan`) — calling the wrong one for the job's mode produces the wrong kind of data, and calling `tomo_scan_projection()` directly while -single-point mode is on prints a warning and asks for confirmation, which will hang -an unattended queue run. A reminder of which one to use fires the moment you switch -single-point mode on (`flomni.single_point_instead_of_fermat_scan = True`), and again -any time you check `flomni.tomo_parameters()` while it's on; the params panel (GUI) -shows the same reminder as a visible note whenever "Single-point scan" is checked. +single-point mode is on prints a warning and asks for confirmation. **Declining that +confirmation now fails the job** (raises, rather than silently skipping that +projection) — inside a custom hook or the normal tomo flow this correctly marks the +queue job `incomplete` and pauses the queue, instead of quietly acquiring fewer +projections than expected with no trace, or (if you get asked at every angle and +eventually give up and hit Ctrl-C) leaving a job stuck at status `running` forever +with nothing actually running. If you hit this, use `tomo_acquire_at_angle(angle)` +in your hook instead, or fix the queue entry via `flomni.tomo_queue_delete(index)` +(no status guard, works even on a stuck `running` row) or by marking it done/ +incomplete directly: `flomni._tomo_queue_proxy.update_by_id(job_id, status=...)`. +A reminder of which one to use fires the moment you switch single-point mode on +(`flomni.single_point_instead_of_fermat_scan = True`), and again any time you check +`flomni.tomo_parameters()` while it's on; the params panel (GUI) shows the same +reminder as a visible note whenever "Single-point scan" is checked. **GUI:** the tomo parameters panel has an "At-each-angle hook" dropdown, listing whatever is currently registered from the CLI (`register_at_each_angle_hook()`) —