docs(flomni): add plugin-agnostic e2e testing guide

Generalizes the patterns and gotchas learned building the flOMNI
tomo-queue e2e suite (main-thread-only live-update machinery, the
builtins/reload import-order bootstrap, neutralizing real side
effects, the demo-config trap in pytest_bec_e2e's shipped fixtures,
the omny_e2e_tests/ folder convention, and the Redis
cross-test-contamination fixture bug) into a standalone reference for
adding e2e coverage to any plugin in this repo, not just flomni.

Intended to be added to a Claude Project alongside this repo so a
coding assistant finds it and follows these patterns instead of
re-deriving them from scratch.

csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/E2E_TESTING_GUIDE.md

@@ -0,0 +1,282 @@
+# How E2E Tests Work in csaxs_bec
+
+**Purpose of this document:** a plugin-agnostic guide to writing and running
+end-to-end tests against a live simulated BEC deployment in this repo. It
+generalizes the patterns built for the flOMNI tomo-queue e2e suite
+(`omny_e2e_tests/`) so the same approach can be reused for LamNI, OMNY, or any
+other plugin — without re-deriving the traps below from scratch. If you are
+an AI coding assistant asked to add or extend e2e tests in this repo, read
+this file first.
+
+Companion docs, for concrete worked examples:
+- `SIMULATED_ENDSTATIONS.md` — what the simulated device layer actually is
+  (protocol-level socket injection) and how to bring a sim session up.
+- `TOMO_QUEUE_TESTING.md` — the fullest worked example: a real e2e checklist
+  built and passed against the flOMNI sim, including every gotcha in this
+  document as it was actually hit.
+
+---
+
+## 1. What "e2e" means here, and when you need it
+
+A **unit test** exercises code with everything below it mocked or stubbed. An
+**e2e test** in this repo means: a real `BECIPythonClient`, a real running
+`ScanServer`/`ScanBundler`/`DeviceServer`/`SciHub`, real Redis, and real
+plugin/device Python code (`Flomni`, `dev.*`, ...) — running against the
+**simulated** device layer (`csaxs_bec/devices/sim/*`, see
+`SIMULATED_ENDSTATIONS.md`) instead of real hardware. Nothing about the
+control-flow code under test is mocked; only the wire protocol at the very
+bottom is simulated.
+
+Write an e2e test (instead of, or in addition to, a unit test) when:
+- the thing you need to prove is that a **global-var-backed** persistence
+  mechanism actually round-trips through real Redis (a mock client's
+  `get_global_var`/`set_global_var` proves nothing about serialization).
+  See "Redis-backed jobs..." below for a case this bit us in practice, and note this is different from
+  Redis contamination *between test runs*, which is §4 below.
+- you need to prove a **live scan/move actually happened** — e.g. that a
+  queued job's parameters were really restored onto the live property a
+  scan reads from, not just that the right method was called with the right
+  arguments.
+- you're testing **crash-resume** or **concurrency** — behavior that only
+  exists because state lives in a shared external store (Redis) that
+  survives process death, and multiple independent clients can touch it at
+  once. A mock can't simulate "the process was SIGKILLed."
+- you're testing something that depends on **real device motion timing**
+  (progress reporting, ETA estimates, ordering of events during a move).
+
+Prove the logic with a fast mock/unit test first if you can (see
+`TOMO_QUEUE_TESTING.md` §2 for an example of extracting real source via AST
+into a mock harness) — it's much cheaper to iterate on. The e2e test is what
+makes it *actually* verified, not just internally consistent.
+
+---
+
+## 2. Folder convention: `omny_e2e_tests/`, not `tests/`
+
+E2e tests for this repo live in a **separate top-level folder**,
+`omny_e2e_tests/`, not under `tests/`. This is deliberate: gitea's CI
+(`.gitea/workflows/ci.yml`) runs `pytest ... ./csaxs_bec/tests/`
+unconditionally on every push/PR, but these tests need a live sim (a running
+BEC deployment + Redis), not just a Python environment. Keeping them
+physically outside `tests/` means:
+- they never get silently swept into (and slow down, or fail in) the
+  standard CI run,
+- there's no confusion about whether a given `tests/...` path is a fast unit
+  test or a slow e2e test that assumes a live sim is already up.
+
+Run them explicitly: `pytest omny_e2e_tests/` (or a specific file/test within
+it), against a sim session you've already started — see
+`SIMULATED_ENDSTATIONS.md` for how to bring one up.
+
+If you're adding e2e coverage for a new plugin, put it under
+`omny_e2e_tests/` too (a subfolder per plugin is fine once there's more than
+one), not under `tests/`.
+
+---
+
+## 3. The trap: don't use the shipped demo-config client fixtures
+
+BEC ships a pytest plugin, **`pytest_bec_e2e`** (registered `pytest11` entry
+point, declared as a `dev` extra in `pyproject.toml`), providing a **live**
+`BECClient` fixture stack: real server, real Redis, real global vars. Without
+`--start-servers` it **attaches to an already-running server** — exactly the
+sim session you started by hand.
+
+**Do not use `bec_client_lib_with_demo_config` / `bec_ipython_client_with_demo_config`
+as they ship.** Both call `bec.config.load_demo_config(force=True)` internally.
+Against a running sim session, this **overwrites the loaded simulated device
+config** (e.g. `simulated_flomni.yaml`) with BEC's own generic demo devices —
+silently breaking every test that assumes your plugin's real devices exist.
+
+Instead, write your own fixture that reuses the lower-level plumbing those
+fixtures are built on (`bec_servers`, `bec_redis_fixture`,
+`bec_services_config_file_path`) but either loads your own device config
+explicitly, or — simpler, and what `omny_e2e_tests/` does — assumes the
+already-running sim session has it loaded, and never calls
+`load_demo_config()`/`update_session_with_file()` at all.
+
+---
+
+## 4. The fixture pattern: a `build_<plugin>()` bootstrap
+
+The reusable core, worth more than any individual test: a small set of files
+that build a real, working plugin instance (`Flomni`, or your plugin's
+equivalent) pointed at the running sim, with every real-instrument/test-only
+side effect handled **once**, in one place.
+
+- **`_bootstrap.py`** — `build_<plugin>(services_config_path) -> (bec, obj)`:
+  connects a `BECIPythonClient` (no demo-config load, see §3), does the
+  builtins/reload bootstrap (§5.2), neutralizes side effects (§5.3), and
+  performs any one-time real-instrument setup your plugin needs before it's
+  usable (§5.4). Also exposes a `shutdown(bec)` helper.
+- **`conftest.py`** — the actual pytest fixture (e.g. `flomni_sim`). Calls
+  `build_<plugin>()`, and snapshots + **resets to a known-empty state** +
+  restores any shared global-var state around each test (§6 — do not skip
+  the reset step).
+- **`_run_<x>_subprocess.py`** — a standalone script, *not* a pytest file,
+  that calls `build_<plugin>()` and then does the one blocking/dangerous
+  thing your crash/concurrency tests need to kill or race (e.g.
+  `flomni.tomo_queue_execute()`). Needed because of the main-thread
+  constraint in §5.1 — see that section for why this can't just be a Python
+  thread inside the test process.
+- **`_<x>_helpers.py`** — everything else shared across test files: fast
+  test-data presets (so a real operation finishes in seconds, not minutes —
+  §5.5), a `spawn_..._subprocess()` wrapper around `subprocess.Popen` for the
+  script above, a `wait_until(predicate, timeout)` poller, and any
+  background-thread "sample state while something blocking runs in the
+  foreground" helper (§5.1).
+
+Every future e2e effort for the *same* plugin should extend these four files,
+not duplicate their logic. A new plugin gets its own set, following the same
+shape.
+
+---
+
+## 5. Gotchas (hard-won, apply generally — not bugs, just non-obvious)
+
+These came from getting a real flOMNI scan to actually run end-to-end; none
+of them are specific to tomo-queue, and any new e2e suite in this repo will
+likely hit the same ones.
+
+### 5.1 Main-thread-only live-update machinery
+
+`BECIPythonClient`'s live-update machinery (anything that goes through
+`ipython_live_updates.process_request` — which includes ordinary device
+moves like `scans.umv(...)`) installs a `SIGINT` handler **per request**.
+Python only allows `signal.signal()` from a process's real main thread.
+
+Consequence: **the actual scan/move/queue-execute call can never run on a
+background Python thread.** Two different escapes, depending on what you
+need:
+- To *observe* live state (progress, a global var) while a blocking call
+  runs in the foreground on the main thread: sample on a background thread
+  instead (see `ProgressSampler` in `omny_e2e_tests/_queue_helpers.py`).
+- To *kill* a running call (crash-resume tests) or run *two* independent
+  sessions concurrently (concurrency tests): spawn a real OS subprocess (its
+  own real main thread) via the `_run_<x>_subprocess.py` script, not a
+  thread. `subprocess.Popen` + `SIGKILL`/`terminate()`, not
+  `threading.Thread` + some cooperative stop flag.
+
+### 5.2 The builtins/reload bootstrap is order-sensitive
+
+Plugin modules commonly have module-level helpers (e.g. `flomni.py`'s
+`def umv(*args): return scans.umv(...)`) that resolve globals like `scans`
+from **the module's own globals**, populated only by an
+`if builtins.__dict__.get("bec") is not None:` block that runs once, at
+**import time**. If the module is imported (even transitively, by an
+unrelated earlier import) before `builtins.bec` is set, that block never
+runs, and the helper `NameError`s the first time it's actually called —
+possibly minutes into a test, on the call that matters.
+
+Fix, in this order: set `builtins.__dict__["bec"]` (and `dev`, `scans`, ...)
+**first**, then `importlib.reload()` the plugin module so the
+import-time block re-runs against the now-populated builtins. This is needed
+even in a fresh subprocess — the module may already have been imported once,
+transitively, before your bootstrap code got to run.
+
+### 5.3 Real side effects must be neutralized for tests
+
+`Flomni.__init__` (and likely your plugin's `__init__`) has real,
+unconditional side effects that are wrong in a test process: starting a
+local HTTP server and background threads that POST to a real external URL
+(webpage generator), sending a real SciLog summary at the end of every scan
+regardless of whether SciLog is configured/reachable, blocking on an
+operator confirmation prompt (`yesno`) that would otherwise hang a headless
+test run forever.
+
+Patch these **once**, as plain class-attribute assignment in
+`neutralize_side_effects()` inside `_bootstrap.py` — not with pytest's
+`monkeypatch` per test, since every caller (the in-process fixture, the
+subprocess script) wants the same patches for the life of the whole process,
+and there's nothing to revert. Find these by actually running an
+unpatched instantiation once and see what tries to reach the network in the
+test log, wraps every scan in `try/except RuntimeError` — don't just guess.
+
+### 5.4 One-time real-instrument setup you must satisfy before anything works
+
+Some plugins need a real hardware-adjacent precondition satisfied before
+their normal operations will run at all — e.g. flOMNI's RT interferometer
+feedback loop must be enabled, and the sample stage moved near its "in"
+position, before a scan can start; a second call must check whether that's
+already true (the check itself, or the setup call, may be expensive — e.g.
+re-zeroing interferometers — so don't redo it if a previous session in the
+same test run already did). This kind of thing generally isn't visible from
+reading the plugin's own primary API — it's usually folded into some other
+higher-level "alignment" or "init" flow that tests otherwise skip. Find it by
+tracing what a manual/real operator session does before their first scan,
+not by trial and error against test failures.
+
+### 5.5 Make the real operation fast enough to actually test
+
+A real tomogram or scan can take many minutes at production settings — too
+slow to run repeatedly in a test suite, especially for crash tests that need
+to interrupt it mid-flight without the whole suite taking forever. Build a
+"fast" parameter preset (single point per position instead of a full raster,
+short per-point dwell/counting time, small field of view, coarse angular
+stepping, ...) that still exercises the *real* code path, just with a much
+smaller N — see `SHORT_SCAN_PARAMS`/`FAST_STEPSIZE` in
+`omny_e2e_tests/_queue_helpers.py` for a concrete example (a full 8-subtomo
+tomogram in roughly a minute, still long enough to interrupt mid-scan for
+crash-resume tests).
+
+---
+
+## 6. Redis state is not reset between separate `pytest` invocations
+
+**This is the single most likely cause of a mysterious, hard-to-reproduce
+e2e test failure in this repo, and it is easy to build a fixture that looks
+correct but doesn't actually protect against it.**
+
+The simulated BEC deployment's Redis instance is a **long-lived shared
+resource** — it persists across separate `pytest` invocations, across a
+crashed test run that never reached its fixture's `finally` block, and
+across manual poking at the same sim session (e.g. someone debugging by
+hand against the same Redis). A fixture that only **snapshots the current
+value and restores it afterward** — which is the right pattern for not
+disturbing a real operator's concurrent session — does **not** protect
+against inheriting contamination that was already there *before* your test
+started.
+
+Concretely: this bit the tomo-queue command-job e2e suite. A prior debugging
+session had left a stale job stuck at `status: "running"` in the
+`tomo_queue` global var. Every subsequent test that assumed it started from
+an empty queue failed in confusing, seemingly-unrelated ways — "queue not
+empty after a rejected add", `KeyError: 'kind'` on an old-format entry that
+had nothing to do with the test that hit it. The fix
+(`omny_e2e_tests/conftest.py`) is to **explicitly reset the relevant global
+var(s) to a known-empty value at fixture setup**, in addition to (not
+instead of) snapshotting and restoring around the test:
+
+```python
+queue_snapshot = bec.get_global_var(_QUEUE_VAR)
+bec.set_global_var(_QUEUE_VAR, [])   # <- the part that's easy to forget
+try:
+    yield flomni
+finally:
+    bec.set_global_var(_QUEUE_VAR, queue_snapshot)
+```
+
+**Any new e2e fixture that reads/writes a global var must reset it to a
+known value at setup, not just restore whatever was there afterward.** If
+you hit a failure that doesn't make sense given the test's own logic, check
+for leftover state before assuming it's a real regression:
+`redis-cli get "user/vars/<the_global_var_key>"`.
+
+---
+
+## 7. Checklist for adding a new e2e test file
+
+1. Does a `build_<plugin>()`-style bootstrap already exist for this plugin?
+   If not, build one following §4, working through §5's gotchas as you hit
+   them (you will).
+2. Does the fixture reset every global var it touches to a known-empty value
+   at setup (§6), not just snapshot/restore?
+3. If the test needs to kill a running operation or run two sessions
+   concurrently, does it use a real subprocess (§5.1), not a thread?
+4. Is there a "fast" parameter preset so the test doesn't take minutes to
+   run (§5.5)?
+5. Does the test file live under `omny_e2e_tests/` (§2), not `tests/`?
+6. Run it against a live sim session before calling it done — see
+   `SIMULATED_ENDSTATIONS.md` for bringing one up. Nothing here is verified
+   until it has actually run against the sim (or real instrument).