# PandABox (`omny_panda`) burst acquisition: two operation modes > **Status: Mode B confirmed working end-to-end against real PandA hardware > (2026-09-15, omny-panda.psi.ch)** — correct per-channel raw values, masked > `gate_detector_active`, and `frame_counter`-based completion all verified, > then speed-tested up to 20 kHz with no reported issues (though see the > caveat under "Saving / restoring layouts" — the saved `omny_freerun.ini` > layout itself was captured at 10 kHz, not 20 kHz). `ptycho_flomni.yaml` > currently has `omny_panda_continuous` (Mode B) active — note the PandA's > Design was switched back to `OMNY` on 2026-09-15 solely to capture the > `omny.ini` layout file below, so double-check which Design is actually > loaded on the physical PandA before running a scan against either config. ## Context flomni burst scans acquire `frames_per_trigger` sub-frames per scan point. Historically, the PandABox (`omny_panda`, device class `PandaBoxOMNY` in `csaxs_bec/devices/panda_box/panda_box_omny.py`) reports one PCAP-aggregated statistic (Mean/Min/Max/Sum/Diff) per gate, discarding the time structure within a gate. This document describes the two supported operation modes — the existing per-gate aggregate mode, and a new continuous free-running raw mode — and how the PandA hardware and `ptycho_flomni.yaml` device config must be set up for each. Mode selection is a single software flag (`raw_stream_mode` in the device config); it does **not** by itself reconfigure the PandA hardware. The PandA layout must independently match the intended mode (see "Open question" at the end) before that mode can be used. ## Mode comparison | | Mode A: legacy (default) | Mode B: free-running (new) | |---|---|---| | Selected by | `raw_stream_mode` unset / `false` | `raw_stream_mode: true` | | PCAP capture fields | `Mean`/`Min`/`Max`/`Sum`/`Diff` per gate | `Value` (raw instantaneous sample) + gate bit + frame counter | | Gate/arm timing | Pulsed per point / sub-frame (rate = scan rate × `frames_per_trigger`) | PCAP held open continuously for the burst window, free-running at `CLOCK1`'s configured rate (started at **100 Hz** for initial testing on 2026-09-15; will be ramped up as testing progresses) | | `_acquisition_group` (BEC async metadata) | `fly` / `monitored` / `burst`, chosen from `scan_type`/`frames_per_trigger` | `free_running` | | Message publishing | One `.data.put()` per PandA network frame, unchanged | Coalesced client-side: flushed every `raw_stream_flush_row_count` rows or `raw_stream_flush_interval` seconds, whichever comes first (mitigates Redis/HDF5 message-rate risk — see the acquisition plan's data-path notes) | | Completion check (`on_complete`) | Exact count: waits for `*PCAP.CAPTURED?` to reach `num_points * frames_per_trigger` | Exact count: waits for `COUNTER1.OUT?` (`frame_counter`) to reach `num_points * frames_per_trigger`. `*PCAP.CAPTURED?` can't be used here — `CLOCK1` drives `PCAP.gate`/`PCAP.trig` continuously (see "Continuous gate/arm" below), so the captured count never naturally stops increasing while armed; `COUNTER1` instead counts completed detector exposures directly (it's clocked off the detector-trigger's falling edge), so it reaches the expected total exactly when the last exposure finishes | | Data completeness verification | Implicit in the exact-count completion check | Same mechanism as Mode A: `frame_counter` (`COUNTER1`) is both the completion signal and the post-hoc completeness record — a gap in an otherwise-monotonic sequence in the HDF5 data means dropped rows even though the expected total was reached | Both modes are implemented in the same `PandaBoxOMNY` class; `raw_stream_mode` defaults to `false` so any device config that doesn't set it keeps today's exact behavior unchanged. ## Physical wiring (Mode B only) The detector-trigger signal is patched into **TTL input 1** (`TTLIN1`) on the PandA front panel. That same `TTLIN1.VAL` signal feeds both downstream consumers (see "Block configuration" below): it is the source counted by `COUNTER1` (the frame counter) and the source captured into the gate bit. ## Block configuration ### Mode A (legacy) — reference baseline The blocks already configured and aliased today, for reference / so this layout can be restored unambiguously (see `ptycho_flomni.yaml`): - `FMC_IN.VAL1`/`FMC_IN.VAL2` (capacitive sensor analog inputs) — `Min`, `Max`, `Mean` capture fields. - `INENC1`-`INENC4.VAL` (position-capture encoder inputs) — `Min`, `Max`, `Mean` capture fields. - `PCAP.GATE_DURATION` — `Value`. - PCAP gate is pulsed once per point / sub-frame by the existing external trigger electronics (unchanged by this work). ### Mode B (new) — free-running raw capture - **Gate-bit input**: the detector-trigger signal, wired into `TTLIN1` (see "Physical wiring" above). `TTLIN1.VAL` is captured into **`PCAP.BITS0`, bit offset 0** — confirmed 2026-09-15 against the omny-panda hardware via: ``` TTLIN1.VAL.CAPTURE_WORD? -> OK =PCAP.BITS0 TTLIN1.VAL.OFFSET? -> OK =0 ``` `PCAP.BITS0` is a shared 32-bit capture word — other captured `bit_out` signals (several TTLIN/LVDSIN/encoder bits are also flagged for capture on this layout) may occupy other bit positions of the same word, so the raw `PCAP.BITS0.Value` is **not** itself a clean 0/1. `PandaBoxOMNY` masks it down to bit 0 in `raw_stream_mode` (see `_GATE_BIT_OFFSET` in `panda_box_omny.py`) before publishing it as `gate_detector_active` — treat that masking as required, not optional, unless it's confirmed nothing else shares `BITS0`. Re-run the query above (read-only, safe against live hardware) if the PandA layout is ever rebuilt, since bit assignment is fixed by the FPGA build and not guaranteed stable across rebuilds. - **Frame counter**: `COUNTER1`, triggered off `TTLIN1.VAL`'s **falling edge** (see layout below) — i.e. it increments once per *completed* detector exposure, not once per PCAP capture tick as originally guessed here. Captured with `CAPTURE=Value`; this is both the data-completeness signal and the completion-detection signal (`frame_counter` in `signal_alias`, see below, and "Completion check" in the mode-comparison table above) — once the last expected exposure's trigger pulse has fallen, `COUNTER1.OUT` reaches `num_points * frames_per_trigger` exactly. - **Raw analog/position capture**: `FMC_IN.VAL1`, `FMC_IN.VAL2` and all four `INENC1`-`4.VAL` are switched from aggregate capture to `CAPTURE=Value` on this layout (confirmed 2026-09-15) — i.e. every analog/encoder channel goes raw together, not a subset. This means the current PandA layout cannot simultaneously produce Mode A's aggregate (Min/Max/Mean) data — see "Open question" below. - **Continuous gate/arm**: `CLOCK1`, enabled by the constant `ONE`, drives `PCAP`'s `gate` and `trig` inputs continuously (instead of the external per-point trigger used in Mode A), so PCAP free-runs at `CLOCK1`'s configured rate. Testing started at **100 Hz** on 2026-09-15 and will be ramped up; there is no PandA-side layout change needed to change the rate, only `CLOCK1`'s period. ``` TTLIN1 (TTL input 1, detector trigger) ├──▶ COUNTER1.trig, falling edge (Up/Down pulse counter) ──▶ COUNTER1.OUT = frame_counter (CAPTURE=Value) │ = completion signal too: reaches num_points * frames_per_trigger when the last exposure finishes └──▶ captured into PCAP.BITS0 bit 0 ──▶ masked in software = gate_detector_active CLOCK1 (enabled by constant ONE) ──▶ PCAP.gate, PCAP.trig (continuous arm/capture) ``` ### Possible variant: gate continuous capture by exposure-active (proposed, not implemented) Discussed 2026-09-19, not yet implemented or tested against hardware. Today `PCAP.gate` and `PCAP.trig` are both driven by `CLOCK1.OUT`, so PCAP free-runs continuously at `CLOCK1`'s rate regardless of whether the detector is actually exposing -- including idle/dead time between exposures. The proposal is to split the two roles so they're driven by different sources: ``` CLOCK1 (enabled by constant ONE) ──▶ PCAP.trig (sample-rate clock, unchanged) TTLIN1.VAL (detector-trigger / exposure-active) ──▶ PCAP.gate (only capture while exposing) ``` `PCAP.trig` keeps ticking at `CLOCK1`'s configured rate exactly as today; `PCAP.gate` switches from `CLOCK1.OUT` to `TTLIN1.VAL`, so a `CLOCK1` tick only produces a captured row while the detector is actively exposing -- idle-time ticks are simply dropped, not written. This should be a pure PandA-layout change -- no `ptycho_flomni.yaml` or `panda_box_omny.py` change would be needed, since it doesn't change which BEC signal names exist or what they mean: - `TTLIN1.VAL` is already proven usable as a `PCAP.gate` source -- it's what Mode A already uses (`PCAP.GATE`/`PCAP.TRIG` = `TTLIN1.VAL` there). - `frame_counter` (`COUNTER1`, clocked directly off `TTLIN1.VAL`'s falling edge, not off `PCAP`) is unaffected -- still counts every completed exposure and still works as the completion/grouping signal exactly as today. - `gate_detector_active` (`PCAP.BITS0` bit 0, also `TTLIN1.VAL`) would become trivially always `1` on every captured row, since a row is only ever captured while that signal is high -- harmless, since `frame_counter` already does the actual per-exposure grouping, not the gate bit. Motivation: at 20 kHz continuous (current, ungated behavior), the free-running raw stream produces roughly 67-86 MB/minute (~4-5 GB/hour) of raw payload across the 10 captured fields (see `panda_box_free_running_data_schema.md` for the field list; the range reflects uncertainty over whether the four `INENC` channels report as `int32` or `float64`). Gating capture to only actual exposure windows would cut that in proportion to the detector's duty cycle -- e.g. roughly 10x less at a 10% duty cycle -- with no BEC-side software change required. If this is implemented: update this section (and the "Continuous gate/arm" wiring diagram above) to confirmed status, record which `.ini` layout captures it, and re-verify the `PCAP.BITS0` bit-0 assignment per the caveat under "Gate-bit input" above (bit assignment is fixed by the FPGA build, not guaranteed stable across a layout rebuild). ## Saving / restoring layouts Use `ophyd_devices/devices/panda_box/utility_scripts.py` to save the current PandA configuration to a `.ini` layout file and reload it later, so a configuration survives a PandA power cycle. Both layouts are saved in this repo under `csaxs_bec/devices/panda_box/layouts/` — `omny.ini` (Mode A) and `omny_freerun.ini` (Mode B), captured 2026-09-15 directly from the PandA design named `OMNY`/`omny_freerun` respectively (switch the PandA's active Design first, then save): ``` python ophyd_devices/devices/panda_box/utility_scripts.py \ --host omny-panda.psi.ch --save-layout csaxs_bec/devices/panda_box/layouts/omny.ini python ophyd_devices/devices/panda_box/utility_scripts.py \ --host omny-panda.psi.ch --save-layout csaxs_bec/devices/panda_box/layouts/omny_freerun.ini ``` To restore either, swap `--save-layout` for `--load-layout` with the same file. Diffing the two confirms the mode-specific differences documented above: `PCAP.GATE`/`PCAP.TRIG` are `TTLIN1.VAL` in Mode A vs `CLOCK1.OUT` in Mode B; `CLOCK1.ENABLE` is `ZERO` (off) in Mode A vs `ONE` in Mode B; `INENC1`-`4.VAL.CAPTURE`/`FMC_IN.VAL1/VAL2.CAPTURE` are `Min Max Mean` in Mode A vs `Value` in Mode B; `PCAP.BITS0.CAPTURE`/`COUNTER1.OUT.CAPTURE` are `No` in Mode A vs `Value` in Mode B. `omny_freerun.ini` was captured with `CLOCK1.PERIOD=0.0001` (100 µs → 10 kHz) — note this doesn't match the 20 kHz reported as tested successfully the same day, so it may not be the exact layout state the 20 kHz test ran at; worth double-checking before relying on this file as "the" validated high-rate layout. ## Signal alias mapping The physical PandA block feeding each BEC signal name, kept here and in `csaxs_bec/device_configs/ptycho_flomni.yaml` (`omny_panda.deviceConfig.signal_alias` for Mode A, `omny_panda_continuous.deviceConfig.signal_alias` for Mode B) in sync as a single source of truth: | PandA block | Mode | BEC signal name | |---|---|---| | `FMC_IN.VAL1.Min`/`.Max`/`.Mean` | A | `cap_voltage_fzp_y_min`/`_max`/`_mean` | | `FMC_IN.VAL2.Min`/`.Max`/`.Mean` | A | `cap_voltage_fzp_x_min`/`_max`/`_mean` | | `INENC1.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_fzp_y_min`/`_max`/`_mean` | | `INENC2.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_fzp_x_min`/`_max`/`_mean` | | `INENC3.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_rotz_min`/`_max`/`_mean` | | `INENC4.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_rotx_min`/`_max`/`_mean` | | `PCAP.GATE_DURATION.Value` | A, B | `pcap_gate_duration_value` | | `FMC_IN.VAL1/VAL2.Value`, `INENC1`-`4.VAL.Value` | B | `cap_voltage_fzp_y`/`_x`, `interf_st_fzp_y`/`_x`/`_rotz`/`_rotx` | | `PCAP.BITS0.Value` (bit 0 = `TTLIN1.VAL`, confirmed 2026-09-15; masked in software, see above) | B | `gate_detector_active` | | `COUNTER1.OUT.Value` (confirmed 2026-09-15, triggered off `TTLIN1.VAL`'s falling edge) | B | `frame_counter` | The BEC-side data schema produced by Mode B (what ends up in the HDF5 async dataset) is documented separately in `docs/developer/panda_box_free_running_data_schema.md`. ## Known data-quality caveat: stale `frame_counter` carryover (found + fixed 2026-09-21) Diagnosed against a real scan (`S06313`): grouping raw `interf_st_fzp_x`/`_y` samples by `frame_counter` (restricted to `gate_detector_active == 1`) and averaging per group reproduces `rt_positions`'s trusted `average_x_st_fzp`/`average_y_st_fzp` to R² > 0.9999 once a single constant per-axis offset is removed -- an expected reference-zero difference between the PandA/`INENC` counter and RT's own internal tracking, irrelevant to reconstruction since it's the same constant for every point in a scan. This also confirms the grouping/filtering method itself: naively treating every raw row as its own position (without grouping by `frame_counter` and filtering to `gate_detector_active == 1`) does not reproduce the true positions and cannot be fixed by any scale/offset correction, since most raw rows are samples taken between real exposures, not at them (see "Continuous gate/arm" above). One point out of 472 was a clear outlier under that method: its `frame_counter` group had 38 extra samples (1036 vs. the usual ~998) and its recomputed position was off by ~116 nm in Y, against ~5 nm typical. Root cause: PandA's `COUNTER1.OUT` (the block behind `frame_counter`) is not cleared by `Arm()`. `COUNTER1.ENABLE=PCAP.ACTIVE` only gates *counting*, not the persisted output value, and `COUNTER1.SET` -- the block's reset-trigger input -- is wired to the constant `ZERO` in the saved layout (see `layouts/omny_freerun.ini`), i.e. never pulsed on arm, unlike `ENABLE` which *is* wired to a live source (`PCAP.ACTIVE`). So the first raw samples of a new acquisition can briefly carry a stale `frame_counter` value left over from the previous acquisition's tail -- observed directly in `S06313`'s raw data: 36 samples reading a stale `167` before dropping to `0` and counting correctly from there. Harmless on its own, except that stale value collided with a real point index later in the *same* scan (point 167), so grouping by `frame_counter` silently mixed ~38 stale rows into that one point's average. Fix, implemented in `panda_box_omny.py` (software-only, no PandA layout change needed): every time PCAP is armed -- including `on_pre_scan` retries, since each `Arm()` reopens the same stale-value window -- `PandaBoxOMNY` now drops any raw rows captured before it observes `frame_counter == 0` for the first time in that acquisition (`_drop_stale_frame_counter_rows`, called from `_receive_frame_data`), and drops any leftover buffered rows from an abandoned previous attempt at the same point (`_reset_raw_stream_state`, called from `on_stage` and from `_try_arm` right before every `Arm()`). Every signal's row list is truncated at the same index to keep channels aligned. Verified against `S06313`: dropping the (37, in that scan) leading stale rows before grouping brings point 167's residual from ~116 nm down to ~0.1 nm, in line with the rest of the scan. A more thorough fix would rewire `COUNTER1.SET` on the PandA layout itself (from the constant `ZERO` to `PCAP.ACTIVE`'s rising edge, matching how `ENABLE` is already wired) so `OUT` resets at the source on every arm, rather than relying on the client to detect and discard the stale prefix. Not implemented here since it requires a PandA layout change to be tested against live hardware; the software-side drop is a correct and sufficient fix in the meantime, and remains a reasonable defense-in-depth even if the layout is fixed later. If this is revisited: re-verify against a range of scans (especially ones with `frame_counter` staying below a few hundred, where a stale-value collision like point 167's is most likely) before removing the software-side check. This also means the "gap in an otherwise-monotonic sequence" completeness check described in the mode-comparison table's "Data completeness verification" row needs one caveat: a single backward step at the very start of the raw sequence, from the previous acquisition's terminal value down to `0`, is this expected (now-filtered) artifact, not a sign of dropped rows. ## Conditional arming (`arm_scan_allowlist`, added 2026-09-22) Generic BEC staging (`stage_all_devices`/`pre_scan_all_devices` in `bec_server.scan_server.scans.scan_actions`) has no concept of "this device only matters for scan X" -- every `enabled: true` device gets `stage()`/ `pre_scan()`/`unstage()` called for *every* scan that runs (`grid_scan`, `line_scan`, alignment moves, ...), not just the ones that actually consume its data. `deviceTags` (e.g. `ptycho_flomni`) are never consulted by that staging code -- they're purely a client-side grouping convenience. So without any further gating, `omny_panda_continuous` was being `Arm()`ed and disarmed for every single scan run during a session, even though only `flomni_fermat_scan` reads its data. That's unnecessary hardware arm/disarm cycling, and it directly compounds the stale-`frame_counter` bug above: more arm cycles mean more windows where a mid-scan abort can leave a stale counter value for the *next* acquisition to inherit (see `AI_docs/PANDA_POSITION_VALIDATION.md` sec 7 for a real incident traced to exactly this). `PandaBoxOMNY` now takes an `arm_scan_allowlist: list[str] | None` constructor kwarg (`None` by default -- arm for every scan, unchanged behavior). When set, `on_stage()` computes `_should_arm_panda` from `scan_parameters.scan_name` against that list; `on_pre_scan()` skips sending `Arm()` entirely (resolves immediately) when it's `False`, and `on_complete()` skips the wait-for-frame-count poll loop the same way. `on_stage()`'s own readout-thread/TCP-connection startup is untouched by this -- it doesn't touch PandA's hardware register state (`COUNTER1`), unlike `Arm()`, so it stays out of scope. `ptycho_flomni.yaml`'s `omny_panda_continuous` sets `arm_scan_allowlist: [flomni_fermat_scan]`. ## Switching between modes `omny_panda` (Mode A) and `omny_panda_continuous` (Mode B) need different PandA hardware layouts and can't run simultaneously (see "Open question" below), so exactly one should be active in `ptycho_flomni.yaml` at a time, matching whichever layout is currently loaded on the PandA — **comment out the other one's block entirely**, don't rely on `enabled: false` alone. That's a correction from earlier in this project: `enabled: false` on a device BEC has never connected to is an inert placeholder — `connect_device` in `bec_server/device_server/devices/devicemanager.py` is only called when `enabled` — so the theory was that both entries could stay uncommented and switching would just be flipping the flag. In practice (2026-09-15), disabling `omny_panda_continuous` this way was **not** sufficient once it had already been connected in the running session: both devices ended up active simultaneously against the same physical PandA. A device that's already connected doesn't appear to get disconnected just because a later config reload marks it disabled. Commenting the block out (so BEC never even constructs the device object) is the reliable way to guarantee it's gone; if in doubt, confirm the other device has actually dropped its connection (e.g. via a device server restart) rather than trusting `enabled: false` alone. Confirmed end-to-end 2026-09-15: Mode B tested working (correct per-channel values, masked `gate_detector_active`, `frame_counter`-based completion), then switched back to Mode A (commented out) for A/B comparison testing. ## Open question, now partially resolved Can both modes coexist under a single static PandA layout — i.e. is mode switching purely the `raw_stream_mode` software flag, with the PandA always capturing both the aggregate and raw fields simultaneously — or do Mode A and Mode B require two distinct saved layouts that must be loaded before a scan depending on the desired mode? As currently wired for testing (2026-09-15), the answer looks like **two distinct layouts are required**: `FMC_IN.VAL1/VAL2` and `INENC1`-`4.VAL` are each set to `CAPTURE=Value` (single selection), not simultaneously `CAPTURE=Value,Min,Max,Mean` — so this layout cannot also produce Mode A's aggregate statistics at the same time. This has not yet been deliberately tested against the alternative (whether the PandA capture UI actually supports selecting `Value` together with `Min`/`Max`/`Mean` on the same field) — if that turns out to work, a single layout could serve both modes. Once Mode B testing is far enough along to save a named layout (see "Saving / restoring layouts" below), settle this explicitly and record the answer plus, if two layouts are needed, the load procedure to switch between them.