csaxs_bec/docs/SIMULATED_ENDSTATIONS.md

# Simulated Endstations for cSAXS BEC (flOMNI / LamNI / OMNY)

Status: implemented and validated (2026-07). flOMNI extensively live-tested against a real
BEC deployment; LamNI/OMNY validated by offline harness, live smoke test pending.

## Purpose

Full software development and testing of the flOMNI/LamNI/OMNY control code without
hardware and without a running instrument. The real ophyd device classes, controllers,
scan plugins, GUI widgets, and IPython-client scripts run **unchanged**; only the wire
protocols are simulated. Swapping a device config is the only difference between the
simulated and the real system.

## Architecture: protocol-level socket injection

All three endstations funnel every hardware interaction through socket-based controllers
(`ophyd_devices.utils.controller.Controller`), which accept a `socket_cls` constructor
parameter. The simulation injects `SocketIO`-compatible classes that implement the wire
protocols as in-process state machines:

- Commands are dispatched synchronously inside `put()`; replies go to a queue drained by
  `receive()`. Commands that are reply-less on the real hardware must not enqueue replies
  (the main desync risk of naive mocks) — reply behavior follows the real server sources.
- Simulation state is shared per (host, port) via `SimStateRegistry`, mirroring the
  controller singleton. Device subclasses seed axis parameters at construction.
- The sim device classes are thin subclasses of the real ones whose only jobs are
  (a) defaulting `socket_cls` and (b) seeding sim state from `sim_*` deviceConfig keys.
- Cross-device choreography (e.g. flOMNI `feedback_enable_with_reset` touching fsamroy,
  fsamx and the ftrackz Galil) runs through the real Python code against each
  controller's own sim socket — no logic is duplicated.

Protocol fidelity is grounded in the actual hardware-side sources: `fgalil.dmc`,
`galil_micos_upr.dmc` (FUPR), `lamni.dmc`, `controller1.dmc` (OMNY), flOMNI
`CommunicationServer.cpp`, `LAMNI_server.cpp`, `OMNY_server.cpp`.

## File inventory (all under csaxs_bec unless noted)

- `devices/sim/sim_socket.py` — `SimSocketBase` + `SimStateRegistry`
- `devices/sim/sim_galil.py` — shared Galil DMC state machine (motion at constant
  velocity, threads, referencing `#FES/#FRM/#REFAX`, gripper transfer routines
  `#GRGET/#GRPUT` with `confirm` handshake and `atconfir` autoconfirm, `#Tracker`,
  `#MNTMODE/#POSMODE`, `#STOP`, `AB`/`ST`, direct axis commands `PA/PR/BG/SH/MO/AC/DC/SP/LD`,
  encoder axes `TP`/`encpermm`, dialect variables via `sim_variables`) plus
  `SimFlomniGalilMotor`, `SimFuprGalilMotor`
- `devices/sim/sim_smaract.py` — Smaract MCS protocol + `SimSmaractMotor` (serves
  flOMNI fosa*, LamNI losa*, OMNY ocs*/oshield)
- `devices/sim/sim_rt_flomni.py` — flOMNI Orchestra server (feedback l0/l1/l2/l3,
  tracker T/Ts, scans s/sc/sd/sr, 10-field sample rows, paced by `sim_point_dwell_s`)
- `devices/sim/sim_lamni.py` — LamNI Galil dialect (swver 28, encoder axes 0–5,
  air-bearing interlocks) + LAMNI server (J0/J1/J5/J6, J2/J4/J7, As/Ar cap sensors,
  Ss/Sr sampler, V velocity, 21-field rows)
- `devices/sim/sim_omny.py` — OMNY Galil dialect (swver 134, mntax=4, signed mntprgs,
  tempab/timeab/IgNoFol, grstat/relpos/toppin, `#GRSTAT/#CRESET/#SAFETY`, NETIO power
  switch stubbed always-on) + OMNY server dialect (J2 six fields, `y`→1 semantics,
  19-field rows) + patched-EPICS `SimOMNYSampleStorage/Dewar/Temperatures`
- `devices/sim/sim_cameras.py` — `SimWebcamViewer` / `SimIDSCamera`: real camera classes
  with synthetic frame source (crosshair + gaussian blob); optional per-frame noise via
  `sim_noise_std` (default 0 = static, cheap over remote desktop)
- `devices/sim/sim_sample_storage.py` — `SimFlomniSampleStorage` (patched EPICS,
  seedable via `sim_samples`, default slots 0–3 populated)
- `device_configs/simulated_omny/` — **all** simulated configs live here:
  `simulated_flomni.yaml` (hand-maintained), `simulated_lamni.yaml` and
  `simulated_omny.yaml` (generated), `simulated_bl_endstation.yaml` (pre-existing)
- `bin/generate_simulated_configs.py` — regenerates the LamNI/OMNY configs from
  `ptycho_lamni.yaml`/`ptycho_omny.yaml` (rerun after real-config changes)
- `tests/sim_flomni_harness.py`, `tests/sim_lamni_omny_harness.py` — offline harnesses
  (no BEC deployment needed): 39 + 26 checks covering moves, referencing, feedback,
  tracker, scans through the flyers, gripper transfers, storage, cameras

## Real-code fixes made along the way (separate commits)

- `ddg_1.py`: `bec_server.scan_server...ScanInfo` import moved under `TYPE_CHECKING`
  (typing-only runtime dependency broke against the redeployed BEC layout).
- `flomni.py`: all four scilog sends wrapped in try/except RuntimeError — accounts
  without scilog no longer crash `tomo_scan`/alignment/timing reports.

## Delivery policy

Tarballs contain ONLY simulation-package files (`devices/sim/*` except
`simulated_beamline_devices.py`, `device_configs/simulated_omny/*`, generator, harnesses,
this doc) plus the single agreed device-layer fix (`ddg_1.py`). Fixes to user-maintained files (`flomni.py`, ...) are delivered as diffs
only — never as full files, since the working copies evolve during testing.

## BEC integration lessons (important for future sim devices)

1. **Explicit signatures required**: the device server passes only deviceConfig keys that
   appear as named parameters in `inspect.signature(dev_cls)`, and injects
   `device_manager` only if named. `*args/**kwargs`-only subclasses silently lose their
   entire config.
2. **msgpack strict_map_key**: no integer keys anywhere in deviceConfig — quote channel
   numbers (`"1": -1.6`).
3. Hidden second transports exist outside the socket seam (OMNY NETIO power-switch HTTP,
   camera MJPEG/pyueye) and need their own stubs.

## Deliberate simulation semantics ("working hardware", no error injection)

- Axes boot **referenced** (real init zeroes at limits; set `sim_referenced: false` to
  exercise init procedures) at positions from `in` user params, clamped into limits.
- `allaxref`/`axisref[n]` are genuine writable DMC variables (forcing `allaxref=0` makes
  moves refuse, as on hardware); recomputed by referencing routines.
- Constant-velocity motion with per-axis speeds from the DMC `#INIT` tables; RT scans
  advance at `sim_point_dwell_s` per position (default 20 ms) so ETA/timing logic is
  meaningfully testable.
- Sampled scan rows carry targets + ~2 nm gaussian noise.
- LamNI: the interferometer readback is coupled to the coarse stages — with feedback off,
  measured (x, y) = lamni_from_stage_coordinates(lsamx-center, lsamy-center)*1000 um
  (geometry constants imported from LamNIFermatScan). This makes
  lamni_new_scan_center_interferometer converge. Coupling is configured via
  `sim_coarse_coupling` on rtx/rty/rt_positions (generator fills it from the lsamx/lsamy
  config). lsamx/lsamy boot at their 'center' user parameters; speeds follow the
  lamni.dmc axspeed table.
- A LamNI fermat scan re-enables feedback itself (`feedback_enable_without_reset` in the
  scan plugin) — starting a scan with user-disabled feedback therefore works, on the
  real system as well as in the sim; it only fails on real hardware when feedback cannot
  engage (air on, interferometer error), an error mode deliberately not simulated.
- Thread 0 is dispatch-only, as on the real controller: `#NEWPAR` keeps it active for
  ~50 ms while the motion itself is reported via the firmware flags (`MG_BG`, motor on).
  The trailing thread-0 wait in the (unmodified) setpoint code therefore returns quickly
  and move progress is published, matching real hardware. An earlier sim iteration
  wrongly modeled thread 0 as busy for the whole move and "fixed" the device code to
  match — that change was reverted; the device files are untouched.
- Referencing (`#FES/#FRM/#REFAX`) applies its outcome synchronously (limit switch,
  position 0) while only the apparent motion is time-based, so status polls never race
  the result.
- Transfer confirm handshakes are real (GUI + gripper-camera flow works); set
  `atconfir=1` on the controller for unattended runs.
- `#GRSTAT` always reports success; NETIO always on; unknown Galil commands reply `:`
  and log `[sim galil] unhandled` warnings — grep the device-server log after a session
  to find fidelity gaps worth adding (full DMC command-reference emulation was
  deliberately not pursued).
- LamNI `sd` reply uses the detector-trigger wording; `sd<cycles>` internal-timing
  variant not modeled. `sr` replies are integer-formatted per LAMNI_server.

## Live-test status (flOMNI, testing account, redeployed BEC)

Passed: sample storage + GUI, moves incl. live progress bars and Ctrl-C, rotation→RT
angle coupling, feedback enable/disable/status, tracker, fermat scans incl. progress and
abort, X-ray eye GUI with camera frames, storage bookkeeping. Pending: full transfer
chain end-to-end, tomo_scan (unblocked by the scilog fix), tomo queue executor
(crash-resume checklist), then queue-reordering implementation. LamNI/OMNY: harness-
validated; a short live smoke test (config load, a move, one scan, one OMNY transfer)
recommended before relying on them, since flOMNI testing surfaced integration issues the
harness cannot see.

## Operational notes

- Deploy: extract the tarball in the repo root (overwrites sim files + the three fixed
  real files). Remove stale `device_configs/simulated_flomni/` and `simulated_lamni/`
  folders from earlier iterations if present.
- Sim tuning knobs (deviceConfig): `sim_initial_position`, `sim_velocity`,
  `sim_stppermm`, `sim_encpermm`, `sim_referenced`, `sim_analog_inputs`,
  `sim_digital_inputs/outputs`, `sim_variables`, `sim_limit_low/high_active`,
  `sim_point_dwell_s`, `sim_noise_std`, `sim_samples`.
- Harnesses run standalone: `python tests/sim_flomni_harness.py` (repo on PYTHONPATH).