docs(flomni): document the declined-scan and stale-running-job fixes

User manual: notes that declining the fermat-scan confirmation now
fails the job instead of silently skipping a projection, plus how to
recover a job stuck at "running" (tomo_queue_delete has no status
guard; update_by_id to fix the status directly; the GUI's staleness
override).

AI_docs: plan section 3.4 records the second real incident (declined
confirmation silently returning) and its fix; section 6.3 records the
stale-"running" GUI guard fix. GUI test checklist gets section 6e
(test steps 48-52) and section 1 item 12, status line updated to flag
6c/6d/6e as not yet clicked through.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md

@@ -355,6 +355,24 @@ is sorted; the webpage generator's `current_params` payload includes
 environment, no `pytest-qt`) or the rendered webpage HTML/JS — see
 `TOMO_QUEUE_GUI_TESTING.md` for the manual checklist covering both.
 
+**Second real incident, found at the beamline: a declined confirmation silently did
+nothing.** A hook called `tomo_scan_projection(angle)` without `_internal=True` while
+`single_point_instead_of_fermat_scan` was on. That combination hits
+`tomo_scan_projection()`'s own "Run a fermat scan anyway?" confirmation (unrelated to
+the hook mechanism itself, but only reachable in practice through a hook or a direct
+CLI call) — at *every* projection angle, since the hook runs once per angle.
+Declining used to just print "Aborted." and `return`, with no exception -- so nothing
+signaled the executor that a projection was skipped. The eventual `Ctrl-C` (a
+`KeyboardInterrupt`, not caught by `tomo_queue_execute()`'s `except Exception`) left
+the job stuck at `running` forever, which is the same failure mode the row-lock note
+in §6.3 now documents. Fixed at the source: declining now raises `FlomniError`
+instead of returning silently, so — combined with the `scan_repeat` `exc_handler` fix
+above — a decline correctly fails the job immediately and marks it `incomplete`
+rather than either silently dropping data or leaving an untraceable stuck state.
+Sim-verified: a direct declined call raises with a clear message; through the queue,
+via a hook missing `_internal=True`, the job ends up `incomplete`, not stuck
+`running` or silently `done`.
+
 ---
 
 ## 4. New / changed CLI methods (`flomni.py`)
@@ -503,6 +521,22 @@ marked `running` while a queue run marks exactly one — so per-row status is th
 - `pending` / `incomplete`: freely draggable and deletable.
 - `done`: pinned (harmless to move, but it makes the list confusing to read).
 
+**Real incident: `running` isn't always the live item.** The rule above assumes
+*something* transitions a `running` job onward -- either the scan finishes, or an
+exception marks it `incomplete`. Neither happens if the process dies hard enough to
+skip both (a `KeyboardInterrupt`, which `tomo_queue_execute()`'s `except Exception`
+does not catch since `KeyboardInterrupt` isn't an `Exception`; a kernel restart). The
+job is then stuck at `running` forever with no live process behind it, and the
+GUI's delete/clear guard -- keyed purely on that status string -- blocked it
+unconditionally, with no escape from the GUI at all (the CLI's `tomo_queue_delete()`
+has no such guard and already worked, but that isn't discoverable from the GUI).
+Fixed by checking `tomo_progress`'s heartbeat (same signal/window as
+`TomoParamsWidget._is_tomo_running()`, 120 s): a `running` job with a stale or
+missing heartbeat is very likely orphaned, so delete/clear offer an explicit
+"looks stale -- proceed anyway?" confirmation instead of refusing outright. A
+genuinely fresh heartbeat still blocks as before -- this narrows the guard's blind
+spot, it doesn't remove the guard.
+
 ### 6.4 Reorder is allowed **mid-run**
 
 It takes effect at the next job boundary: the executor finishes the running job, marks

csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md

@@ -12,10 +12,12 @@ gaps (all fixed, see section 6c below): a confusing duplicate row-number
 column, `tomo_parameters()`/the queue table/scilog/the PDF report/the
 progress-ring label not showing an active hook at all, and a clarification
 that `unregister_at_each_angle_hook()` doesn't clear `at_each_angle_hook`
-(confirmed expected, now documented). **Sections 6c and 6d are new and not
-yet clicked through** — 6c is the fixes above, 6d is the new "Load into
-editor" feature (reuse a queued job's settings as a new edit). Companion
-docs:
+(confirmed expected, now documented). **Sections 6c, 6d, and 6e are new and
+not yet clicked through** — 6c is the fixes above, 6d is the new "Load into
+editor" feature (reuse a queued job's settings as a new edit), 6e is a
+declined-scan/stuck-queue incident found live at the beamline and its fix
+(CLI-side sim-verified; the GUI delete/clear override itself is not).
+Companion docs:
 `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` section 6 (the design this implements),
 `TOMO_QUEUE_TESTING.md` (Steps 1 & 2, already sim-validated).
 
@@ -131,6 +133,16 @@ support on top of it:
     they already do. Confirms before discarding an in-progress edit.
     Disabled outside a valid single-tomo-job selection and during sort
     mode. See section 6d below — not yet click-tested.
+12. **Declined-scan failure + stale "running" recovery.** Two fixes for a
+    real stuck-queue incident: `tomo_scan_projection()` now raises instead
+    of silently returning when the "run a fermat scan anyway?" confirmation
+    is declined, so a queue job fails cleanly (`incomplete`) instead of
+    silently dropping a projection or (after an eventual Ctrl-C) getting
+    stuck at `running` with no live process. And delete/clear on a
+    `running` row now check `tomo_progress`'s heartbeat: stale or missing
+    offers a "proceed anyway?" override instead of refusing forever; a
+    fresh heartbeat still blocks outright. See section 6e below — not yet
+    click-tested.
 
 ---
 
@@ -404,6 +416,41 @@ GUI bug and reporting gaps for the hook feature.
 47. Confirm "Load into editor" is also disabled while **Sort queue…** mode
     is active, consistent with the other queue-mutating buttons.
 
+## 6e. Test plan — new: declined-scan failure + stale "running" recovery
+
+Found at the beamline (thanks again, Mirko) — a hook without `_internal=True`
+plus single-point mode got stuck asking for confirmation at every angle, and
+answering "no" (repeatedly) then Ctrl-C'ing left a job stuck at `running`
+forever with the GUI unable to delete it.
+
+48. Set `flomni.single_point_instead_of_fermat_scan = True`, then call
+    `flomni.tomo_scan_projection(10.0)` directly from the CLI. Answer **no**
+    to "Run a fermat scan anyway?". Confirm it now **raises** (a traceback,
+    not just "Aborted." with a silent return).
+49. Register a hook that calls `tomo_scan_projection(angle)` **without**
+    `_internal=True`, activate it, queue a job with single-point mode on,
+    and run the queue. When asked to confirm the fermat scan, answer **no**.
+    Confirm the job ends up `incomplete` (not stuck `running`, not silently
+    `done` with missing projections) and `tomo_queue_execute()` prints a
+    clear "did not complete" message naming the actual problem.
+50. Hand-simulate the stuck-`running` scenario: pick a `pending` job's id
+    (`flomni._tomo_queue_proxy.as_list()`) and
+    `flomni._tomo_queue_proxy.update_by_id(job_id, status="running")`
+    directly (standing in for "the process died mid-scan"), with no scan
+    actually active (`tomo_progress`'s heartbeat absent or >120s old).
+    In the GUI, select that row and click **Delete selected**. Confirm you
+    get the **new** "looks stale — proceed anyway?" dialog (not the old
+    unconditional refusal), and that clicking **Yes** actually deletes it
+    after the normal delete confirmation. Repeat for **Clear all**.
+51. Same setup, but with `tomo_progress`'s heartbeat fresh (<120s old) —
+    e.g. start a real short scan running so the heartbeat updates. Confirm
+    delete/clear on the `running` row still refuse outright, no staleness
+    override offered — a genuinely active job must still be unaffected.
+52. Cross-check from the CLI: `flomni.tomo_queue_delete(index)` should
+    always work on a stuck `running` row regardless of GUI state (no
+    status guard there at all) — useful as a fallback even without the
+    GUI open.
+
 ## 7. Known limitations of this first iteration (by design, not bugs)
 
 Worth knowing before you file something as broken:

docs/user/ptychography/flomni.md

@@ -348,6 +348,8 @@ Several tomo parameter sets can be queued and run sequentially on the same sampl
 
 The queue is persisted (it survives a BEC client restart). Each job's status is one of `pending`, `running`, `incomplete`, or `done`. A job that did not run to completion (an exception was caught, or the BEC client itself crashed mid-scan) is automatically resumed - rather than restarted - the next time `flomni.tomo_queue_execute()` is called.
 
+If the process running the queue dies hard enough that it never gets to write a status update at all (a kernel restart, a Ctrl-C during a blocking prompt) a job can be left stuck showing `running` with nothing actually running. `flomni.tomo_queue_delete(index)` has no status guard and works on a stuck row regardless; you can also fix the status directly with `flomni._tomo_queue_proxy.update_by_id(job_id, status="incomplete")` (to make it resumable) or `status="done"` (to skip it). The GUI's Delete/Clear normally refuse to touch a `running` row, but detect a stale one (no recent scan heartbeat) and offer to proceed anyway with an explicit confirmation instead of blocking forever.
+
 Example:
 ```
 flomni.tomo_parameters()                  # set up parameter set #1
@@ -482,11 +484,20 @@ projection; `tomo_acquire_at_angle(angle)` always runs a single-point acquisitio
 Use whichever matches how `tomo_parameters()` has this job configured
 (`single_point_instead_of_fermat_scan`) — calling the wrong one for the job's mode
 produces the wrong kind of data, and calling `tomo_scan_projection()` directly while
-single-point mode is on prints a warning and asks for confirmation, which will hang
-an unattended queue run. A reminder of which one to use fires the moment you switch
-single-point mode on (`flomni.single_point_instead_of_fermat_scan = True`), and again
-any time you check `flomni.tomo_parameters()` while it's on; the params panel (GUI)
-shows the same reminder as a visible note whenever "Single-point scan" is checked.
+single-point mode is on prints a warning and asks for confirmation. **Declining that
+confirmation now fails the job** (raises, rather than silently skipping that
+projection) — inside a custom hook or the normal tomo flow this correctly marks the
+queue job `incomplete` and pauses the queue, instead of quietly acquiring fewer
+projections than expected with no trace, or (if you get asked at every angle and
+eventually give up and hit Ctrl-C) leaving a job stuck at status `running` forever
+with nothing actually running. If you hit this, use `tomo_acquire_at_angle(angle)`
+in your hook instead, or fix the queue entry via `flomni.tomo_queue_delete(index)`
+(no status guard, works even on a stuck `running` row) or by marking it done/
+incomplete directly: `flomni._tomo_queue_proxy.update_by_id(job_id, status=...)`.
+A reminder of which one to use fires the moment you switch single-point mode on
+(`flomni.single_point_instead_of_fermat_scan = True`), and again any time you check
+`flomni.tomo_parameters()` while it's on; the params panel (GUI) shows the same
+reminder as a visible note whenever "Single-point scan" is checked.
 
 **GUI:** the tomo parameters panel has an "At-each-angle hook" dropdown, listing
 whatever is currently registered from the CLI (`register_at_each_angle_hook()`) —