diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/E2E_TESTING_GUIDE.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/E2E_TESTING_GUIDE.md new file mode 100644 index 0000000..7abbda2 --- /dev/null +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/E2E_TESTING_GUIDE.md @@ -0,0 +1,282 @@ +# How E2E Tests Work in csaxs_bec + +**Purpose of this document:** a plugin-agnostic guide to writing and running +end-to-end tests against a live simulated BEC deployment in this repo. It +generalizes the patterns built for the flOMNI tomo-queue e2e suite +(`omny_e2e_tests/`) so the same approach can be reused for LamNI, OMNY, or any +other plugin — without re-deriving the traps below from scratch. If you are +an AI coding assistant asked to add or extend e2e tests in this repo, read +this file first. + +Companion docs, for concrete worked examples: +- `SIMULATED_ENDSTATIONS.md` — what the simulated device layer actually is + (protocol-level socket injection) and how to bring a sim session up. +- `TOMO_QUEUE_TESTING.md` — the fullest worked example: a real e2e checklist + built and passed against the flOMNI sim, including every gotcha in this + document as it was actually hit. + +--- + +## 1. What "e2e" means here, and when you need it + +A **unit test** exercises code with everything below it mocked or stubbed. An +**e2e test** in this repo means: a real `BECIPythonClient`, a real running +`ScanServer`/`ScanBundler`/`DeviceServer`/`SciHub`, real Redis, and real +plugin/device Python code (`Flomni`, `dev.*`, ...) — running against the +**simulated** device layer (`csaxs_bec/devices/sim/*`, see +`SIMULATED_ENDSTATIONS.md`) instead of real hardware. Nothing about the +control-flow code under test is mocked; only the wire protocol at the very +bottom is simulated. + +Write an e2e test (instead of, or in addition to, a unit test) when: +- the thing you need to prove is that a **global-var-backed** persistence + mechanism actually round-trips through real Redis (a mock client's + `get_global_var`/`set_global_var` proves nothing about serialization). + See "Redis-backed jobs..." below for a case this bit us in practice, and note this is different from + Redis contamination *between test runs*, which is §4 below. +- you need to prove a **live scan/move actually happened** — e.g. that a + queued job's parameters were really restored onto the live property a + scan reads from, not just that the right method was called with the right + arguments. +- you're testing **crash-resume** or **concurrency** — behavior that only + exists because state lives in a shared external store (Redis) that + survives process death, and multiple independent clients can touch it at + once. A mock can't simulate "the process was SIGKILLed." +- you're testing something that depends on **real device motion timing** + (progress reporting, ETA estimates, ordering of events during a move). + +Prove the logic with a fast mock/unit test first if you can (see +`TOMO_QUEUE_TESTING.md` §2 for an example of extracting real source via AST +into a mock harness) — it's much cheaper to iterate on. The e2e test is what +makes it *actually* verified, not just internally consistent. + +--- + +## 2. Folder convention: `omny_e2e_tests/`, not `tests/` + +E2e tests for this repo live in a **separate top-level folder**, +`omny_e2e_tests/`, not under `tests/`. This is deliberate: gitea's CI +(`.gitea/workflows/ci.yml`) runs `pytest ... ./csaxs_bec/tests/` +unconditionally on every push/PR, but these tests need a live sim (a running +BEC deployment + Redis), not just a Python environment. Keeping them +physically outside `tests/` means: +- they never get silently swept into (and slow down, or fail in) the + standard CI run, +- there's no confusion about whether a given `tests/...` path is a fast unit + test or a slow e2e test that assumes a live sim is already up. + +Run them explicitly: `pytest omny_e2e_tests/` (or a specific file/test within +it), against a sim session you've already started — see +`SIMULATED_ENDSTATIONS.md` for how to bring one up. + +If you're adding e2e coverage for a new plugin, put it under +`omny_e2e_tests/` too (a subfolder per plugin is fine once there's more than +one), not under `tests/`. + +--- + +## 3. The trap: don't use the shipped demo-config client fixtures + +BEC ships a pytest plugin, **`pytest_bec_e2e`** (registered `pytest11` entry +point, declared as a `dev` extra in `pyproject.toml`), providing a **live** +`BECClient` fixture stack: real server, real Redis, real global vars. Without +`--start-servers` it **attaches to an already-running server** — exactly the +sim session you started by hand. + +**Do not use `bec_client_lib_with_demo_config` / `bec_ipython_client_with_demo_config` +as they ship.** Both call `bec.config.load_demo_config(force=True)` internally. +Against a running sim session, this **overwrites the loaded simulated device +config** (e.g. `simulated_flomni.yaml`) with BEC's own generic demo devices — +silently breaking every test that assumes your plugin's real devices exist. + +Instead, write your own fixture that reuses the lower-level plumbing those +fixtures are built on (`bec_servers`, `bec_redis_fixture`, +`bec_services_config_file_path`) but either loads your own device config +explicitly, or — simpler, and what `omny_e2e_tests/` does — assumes the +already-running sim session has it loaded, and never calls +`load_demo_config()`/`update_session_with_file()` at all. + +--- + +## 4. The fixture pattern: a `build_()` bootstrap + +The reusable core, worth more than any individual test: a small set of files +that build a real, working plugin instance (`Flomni`, or your plugin's +equivalent) pointed at the running sim, with every real-instrument/test-only +side effect handled **once**, in one place. + +- **`_bootstrap.py`** — `build_(services_config_path) -> (bec, obj)`: + connects a `BECIPythonClient` (no demo-config load, see §3), does the + builtins/reload bootstrap (§5.2), neutralizes side effects (§5.3), and + performs any one-time real-instrument setup your plugin needs before it's + usable (§5.4). Also exposes a `shutdown(bec)` helper. +- **`conftest.py`** — the actual pytest fixture (e.g. `flomni_sim`). Calls + `build_()`, and snapshots + **resets to a known-empty state** + + restores any shared global-var state around each test (§6 — do not skip + the reset step). +- **`_run__subprocess.py`** — a standalone script, *not* a pytest file, + that calls `build_()` and then does the one blocking/dangerous + thing your crash/concurrency tests need to kill or race (e.g. + `flomni.tomo_queue_execute()`). Needed because of the main-thread + constraint in §5.1 — see that section for why this can't just be a Python + thread inside the test process. +- **`__helpers.py`** — everything else shared across test files: fast + test-data presets (so a real operation finishes in seconds, not minutes — + §5.5), a `spawn_..._subprocess()` wrapper around `subprocess.Popen` for the + script above, a `wait_until(predicate, timeout)` poller, and any + background-thread "sample state while something blocking runs in the + foreground" helper (§5.1). + +Every future e2e effort for the *same* plugin should extend these four files, +not duplicate their logic. A new plugin gets its own set, following the same +shape. + +--- + +## 5. Gotchas (hard-won, apply generally — not bugs, just non-obvious) + +These came from getting a real flOMNI scan to actually run end-to-end; none +of them are specific to tomo-queue, and any new e2e suite in this repo will +likely hit the same ones. + +### 5.1 Main-thread-only live-update machinery + +`BECIPythonClient`'s live-update machinery (anything that goes through +`ipython_live_updates.process_request` — which includes ordinary device +moves like `scans.umv(...)`) installs a `SIGINT` handler **per request**. +Python only allows `signal.signal()` from a process's real main thread. + +Consequence: **the actual scan/move/queue-execute call can never run on a +background Python thread.** Two different escapes, depending on what you +need: +- To *observe* live state (progress, a global var) while a blocking call + runs in the foreground on the main thread: sample on a background thread + instead (see `ProgressSampler` in `omny_e2e_tests/_queue_helpers.py`). +- To *kill* a running call (crash-resume tests) or run *two* independent + sessions concurrently (concurrency tests): spawn a real OS subprocess (its + own real main thread) via the `_run__subprocess.py` script, not a + thread. `subprocess.Popen` + `SIGKILL`/`terminate()`, not + `threading.Thread` + some cooperative stop flag. + +### 5.2 The builtins/reload bootstrap is order-sensitive + +Plugin modules commonly have module-level helpers (e.g. `flomni.py`'s +`def umv(*args): return scans.umv(...)`) that resolve globals like `scans` +from **the module's own globals**, populated only by an +`if builtins.__dict__.get("bec") is not None:` block that runs once, at +**import time**. If the module is imported (even transitively, by an +unrelated earlier import) before `builtins.bec` is set, that block never +runs, and the helper `NameError`s the first time it's actually called — +possibly minutes into a test, on the call that matters. + +Fix, in this order: set `builtins.__dict__["bec"]` (and `dev`, `scans`, ...) +**first**, then `importlib.reload()` the plugin module so the +import-time block re-runs against the now-populated builtins. This is needed +even in a fresh subprocess — the module may already have been imported once, +transitively, before your bootstrap code got to run. + +### 5.3 Real side effects must be neutralized for tests + +`Flomni.__init__` (and likely your plugin's `__init__`) has real, +unconditional side effects that are wrong in a test process: starting a +local HTTP server and background threads that POST to a real external URL +(webpage generator), sending a real SciLog summary at the end of every scan +regardless of whether SciLog is configured/reachable, blocking on an +operator confirmation prompt (`yesno`) that would otherwise hang a headless +test run forever. + +Patch these **once**, as plain class-attribute assignment in +`neutralize_side_effects()` inside `_bootstrap.py` — not with pytest's +`monkeypatch` per test, since every caller (the in-process fixture, the +subprocess script) wants the same patches for the life of the whole process, +and there's nothing to revert. Find these by actually running an +unpatched instantiation once and see what tries to reach the network in the +test log, wraps every scan in `try/except RuntimeError` — don't just guess. + +### 5.4 One-time real-instrument setup you must satisfy before anything works + +Some plugins need a real hardware-adjacent precondition satisfied before +their normal operations will run at all — e.g. flOMNI's RT interferometer +feedback loop must be enabled, and the sample stage moved near its "in" +position, before a scan can start; a second call must check whether that's +already true (the check itself, or the setup call, may be expensive — e.g. +re-zeroing interferometers — so don't redo it if a previous session in the +same test run already did). This kind of thing generally isn't visible from +reading the plugin's own primary API — it's usually folded into some other +higher-level "alignment" or "init" flow that tests otherwise skip. Find it by +tracing what a manual/real operator session does before their first scan, +not by trial and error against test failures. + +### 5.5 Make the real operation fast enough to actually test + +A real tomogram or scan can take many minutes at production settings — too +slow to run repeatedly in a test suite, especially for crash tests that need +to interrupt it mid-flight without the whole suite taking forever. Build a +"fast" parameter preset (single point per position instead of a full raster, +short per-point dwell/counting time, small field of view, coarse angular +stepping, ...) that still exercises the *real* code path, just with a much +smaller N — see `SHORT_SCAN_PARAMS`/`FAST_STEPSIZE` in +`omny_e2e_tests/_queue_helpers.py` for a concrete example (a full 8-subtomo +tomogram in roughly a minute, still long enough to interrupt mid-scan for +crash-resume tests). + +--- + +## 6. Redis state is not reset between separate `pytest` invocations + +**This is the single most likely cause of a mysterious, hard-to-reproduce +e2e test failure in this repo, and it is easy to build a fixture that looks +correct but doesn't actually protect against it.** + +The simulated BEC deployment's Redis instance is a **long-lived shared +resource** — it persists across separate `pytest` invocations, across a +crashed test run that never reached its fixture's `finally` block, and +across manual poking at the same sim session (e.g. someone debugging by +hand against the same Redis). A fixture that only **snapshots the current +value and restores it afterward** — which is the right pattern for not +disturbing a real operator's concurrent session — does **not** protect +against inheriting contamination that was already there *before* your test +started. + +Concretely: this bit the tomo-queue command-job e2e suite. A prior debugging +session had left a stale job stuck at `status: "running"` in the +`tomo_queue` global var. Every subsequent test that assumed it started from +an empty queue failed in confusing, seemingly-unrelated ways — "queue not +empty after a rejected add", `KeyError: 'kind'` on an old-format entry that +had nothing to do with the test that hit it. The fix +(`omny_e2e_tests/conftest.py`) is to **explicitly reset the relevant global +var(s) to a known-empty value at fixture setup**, in addition to (not +instead of) snapshotting and restoring around the test: + +```python +queue_snapshot = bec.get_global_var(_QUEUE_VAR) +bec.set_global_var(_QUEUE_VAR, []) # <- the part that's easy to forget +try: + yield flomni +finally: + bec.set_global_var(_QUEUE_VAR, queue_snapshot) +``` + +**Any new e2e fixture that reads/writes a global var must reset it to a +known value at setup, not just restore whatever was there afterward.** If +you hit a failure that doesn't make sense given the test's own logic, check +for leftover state before assuming it's a real regression: +`redis-cli get "user/vars/"`. + +--- + +## 7. Checklist for adding a new e2e test file + +1. Does a `build_()`-style bootstrap already exist for this plugin? + If not, build one following §4, working through §5's gotchas as you hit + them (you will). +2. Does the fixture reset every global var it touches to a known-empty value + at setup (§6), not just snapshot/restore? +3. If the test needs to kill a running operation or run two sessions + concurrently, does it use a real subprocess (§5.1), not a thread? +4. Is there a "fast" parameter preset so the test doesn't take minutes to + run (§5.5)? +5. Does the test file live under `omny_e2e_tests/` (§2), not `tests/`? +6. Run it against a live sim session before calling it done — see + `SIMULATED_ENDSTATIONS.md` for bringing one up. Nothing here is verified + until it has actually run against the sim (or real instrument).