csaxs_bec/docs/developer/panda_box_free_running_setup.md

PandABox (omny_panda) burst acquisition: two operation modes

Status: Mode B confirmed working end-to-end against real PandA hardware (2026-09-15, omny-panda.psi.ch) — correct per-channel raw values, masked gate_detector_active, and frame_counter-based completion all verified, then speed-tested up to 20 kHz with no reported issues (though see the caveat under "Saving / restoring layouts" — the saved omny_freerun.ini layout itself was captured at 10 kHz, not 20 kHz). ptycho_flomni.yaml currently has omny_panda_continuous (Mode B) active — note the PandA's Design was switched back to OMNY on 2026-09-15 solely to capture the omny.ini layout file below, so double-check which Design is actually loaded on the physical PandA before running a scan against either config.

Context

flomni burst scans acquire frames_per_trigger sub-frames per scan point. Historically, the PandABox (omny_panda, device class PandaBoxOMNY in csaxs_bec/devices/panda_box/panda_box_omny.py) reports one PCAP-aggregated statistic (Mean/Min/Max/Sum/Diff) per gate, discarding the time structure within a gate. This document describes the two supported operation modes — the existing per-gate aggregate mode, and a new continuous free-running raw mode — and how the PandA hardware and ptycho_flomni.yaml device config must be set up for each.

Mode selection is a single software flag (raw_stream_mode in the device config); it does not by itself reconfigure the PandA hardware. The PandA layout must independently match the intended mode (see "Open question" at the end) before that mode can be used.

Mode comparison

	Mode A: legacy (default)	Mode B: free-running (new)
Selected by	raw_stream_mode unset / false	raw_stream_mode: true
PCAP capture fields	Mean/Min/Max/Sum/Diff per gate	Value (raw instantaneous sample) + gate bit + frame counter
Gate/arm timing	Pulsed per point / sub-frame (rate = scan rate × frames_per_trigger)	PCAP held open continuously for the burst window, free-running at CLOCK1's configured rate (started at 100 Hz for initial testing on 2026-09-15; will be ramped up as testing progresses)
_acquisition_group (BEC async metadata)	fly / monitored / burst, chosen from scan_type/frames_per_trigger	free_running
Message publishing	One .data.put() per PandA network frame, unchanged	Coalesced client-side: flushed every raw_stream_flush_row_count rows or raw_stream_flush_interval seconds, whichever comes first (mitigates Redis/HDF5 message-rate risk — see the acquisition plan's data-path notes)
Completion check (on_complete)	Exact count: waits for *PCAP.CAPTURED? to reach num_points * frames_per_trigger	Exact count: waits for COUNTER1.OUT? (frame_counter) to reach num_points * frames_per_trigger. *PCAP.CAPTURED? can't be used here — CLOCK1 drives PCAP.gate/PCAP.trig continuously (see "Continuous gate/arm" below), so the captured count never naturally stops increasing while armed; COUNTER1 instead counts completed detector exposures directly (it's clocked off the detector-trigger's falling edge), so it reaches the expected total exactly when the last exposure finishes
Data completeness verification	Implicit in the exact-count completion check	Same mechanism as Mode A: frame_counter (COUNTER1) is both the completion signal and the post-hoc completeness record — a gap in an otherwise-monotonic sequence in the HDF5 data means dropped rows even though the expected total was reached

Both modes are implemented in the same PandaBoxOMNY class; raw_stream_mode defaults to false so any device config that doesn't set it keeps today's exact behavior unchanged.

Physical wiring (Mode B only)

The detector-trigger signal is patched into TTL input 1 (TTLIN1) on the PandA front panel. That same TTLIN1.VAL signal feeds both downstream consumers (see "Block configuration" below): it is the source counted by COUNTER1 (the frame counter) and the source captured into the gate bit.

Block configuration

Mode A (legacy) — reference baseline

The blocks already configured and aliased today, for reference / so this layout can be restored unambiguously (see ptycho_flomni.yaml):

• FMC_IN.VAL1/FMC_IN.VAL2 (capacitive sensor analog inputs) — Min, Max, Mean capture fields.
• INENC1-INENC4.VAL (position-capture encoder inputs) — Min, Max, Mean capture fields.
• PCAP.GATE_DURATION — Value.
• PCAP gate is pulsed once per point / sub-frame by the existing external trigger electronics (unchanged by this work).

Mode B (new) — free-running raw capture

• Gate-bit input: the detector-trigger signal, wired into TTLIN1 (see "Physical wiring" above). TTLIN1.VAL is captured into PCAP.BITS0, bit offset 0 — confirmed 2026-09-15 against the omny-panda hardware via:

  TTLIN1.VAL.CAPTURE_WORD?   -> OK =PCAP.BITS0
  TTLIN1.VAL.OFFSET?         -> OK =0
  

  PCAP.BITS0 is a shared 32-bit capture word — other captured bit_out signals (several TTLIN/LVDSIN/encoder bits are also flagged for capture on this layout) may occupy other bit positions of the same word, so the raw PCAP.BITS0.Value is not itself a clean 0/1. PandaBoxOMNY masks it down to bit 0 in raw_stream_mode (see _GATE_BIT_OFFSET in panda_box_omny.py) before publishing it as gate_detector_active — treat that masking as required, not optional, unless it's confirmed nothing else shares BITS0. Re-run the query above (read-only, safe against live hardware) if the PandA layout is ever rebuilt, since bit assignment is fixed by the FPGA build and not guaranteed stable across rebuilds.

• Frame counter: COUNTER1, triggered off TTLIN1.VAL's falling edge (see layout below) — i.e. it increments once per completed detector exposure, not once per PCAP capture tick as originally guessed here. Captured with CAPTURE=Value; this is both the data-completeness signal and the completion-detection signal (frame_counter in signal_alias, see below, and "Completion check" in the mode-comparison table above) — once the last expected exposure's trigger pulse has fallen, COUNTER1.OUT reaches num_points * frames_per_trigger exactly.

• Raw analog/position capture: FMC_IN.VAL1, FMC_IN.VAL2 and all four INENC1-4.VAL are switched from aggregate capture to CAPTURE=Value on this layout (confirmed 2026-09-15) — i.e. every analog/encoder channel goes raw together, not a subset. This means the current PandA layout cannot simultaneously produce Mode A's aggregate (Min/Max/Mean) data — see "Open question" below.

• Continuous gate/arm: CLOCK1, enabled by the constant ONE, drives PCAP's gate and trig inputs continuously (instead of the external per-point trigger used in Mode A), so PCAP free-runs at CLOCK1's configured rate. Testing started at 100 Hz on 2026-09-15 and will be ramped up; there is no PandA-side layout change needed to change the rate, only CLOCK1's period.

  TTLIN1 (TTL input 1, detector trigger)
    ├──▶ COUNTER1.trig, falling edge  (Up/Down pulse counter) ──▶ COUNTER1.OUT  = frame_counter (CAPTURE=Value)
    │      = completion signal too: reaches num_points * frames_per_trigger when the last exposure finishes
    └──▶ captured into PCAP.BITS0 bit 0          ──▶ masked in software = gate_detector_active
  
  CLOCK1 (enabled by constant ONE) ──▶ PCAP.gate, PCAP.trig   (continuous arm/capture)
  

Possible variant: gate continuous capture by exposure-active (proposed, not implemented)

Discussed 2026-09-19, not yet implemented or tested against hardware. Today PCAP.gate and PCAP.trig are both driven by CLOCK1.OUT, so PCAP free-runs continuously at CLOCK1's rate regardless of whether the detector is actually exposing -- including idle/dead time between exposures. The proposal is to split the two roles so they're driven by different sources:

CLOCK1 (enabled by constant ONE)              ──▶ PCAP.trig   (sample-rate clock, unchanged)
TTLIN1.VAL (detector-trigger / exposure-active) ──▶ PCAP.gate   (only capture while exposing)

PCAP.trig keeps ticking at CLOCK1's configured rate exactly as today; PCAP.gate switches from CLOCK1.OUT to TTLIN1.VAL, so a CLOCK1 tick only produces a captured row while the detector is actively exposing -- idle-time ticks are simply dropped, not written.

This should be a pure PandA-layout change -- no ptycho_flomni.yaml or panda_box_omny.py change would be needed, since it doesn't change which BEC signal names exist or what they mean:

• TTLIN1.VAL is already proven usable as a PCAP.gate source -- it's what Mode A already uses (PCAP.GATE/PCAP.TRIG = TTLIN1.VAL there).
• frame_counter (COUNTER1, clocked directly off TTLIN1.VAL's falling edge, not off PCAP) is unaffected -- still counts every completed exposure and still works as the completion/grouping signal exactly as today.
• gate_detector_active (PCAP.BITS0 bit 0, also TTLIN1.VAL) would become trivially always 1 on every captured row, since a row is only ever captured while that signal is high -- harmless, since frame_counter already does the actual per-exposure grouping, not the gate bit.

Motivation: at 20 kHz continuous (current, ungated behavior), the free-running raw stream produces roughly 67-86 MB/minute (~4-5 GB/hour) of raw payload across the 10 captured fields (see panda_box_free_running_data_schema.md for the field list; the range reflects uncertainty over whether the four INENC channels report as int32 or float64). Gating capture to only actual exposure windows would cut that in proportion to the detector's duty cycle -- e.g. roughly 10x less at a 10% duty cycle -- with no BEC-side software change required.

If this is implemented: update this section (and the "Continuous gate/arm" wiring diagram above) to confirmed status, record which .ini layout captures it, and re-verify the PCAP.BITS0 bit-0 assignment per the caveat under "Gate-bit input" above (bit assignment is fixed by the FPGA build, not guaranteed stable across a layout rebuild).

Saving / restoring layouts

Use ophyd_devices/devices/panda_box/utility_scripts.py to save the current PandA configuration to a .ini layout file and reload it later, so a configuration survives a PandA power cycle. Both layouts are saved in this repo under csaxs_bec/devices/panda_box/layouts/ — omny.ini (Mode A) and omny_freerun.ini (Mode B), captured 2026-09-15 directly from the PandA design named OMNY/omny_freerun respectively (switch the PandA's active Design first, then save):

python ophyd_devices/devices/panda_box/utility_scripts.py \
    --host omny-panda.psi.ch --save-layout csaxs_bec/devices/panda_box/layouts/omny.ini

python ophyd_devices/devices/panda_box/utility_scripts.py \
    --host omny-panda.psi.ch --save-layout csaxs_bec/devices/panda_box/layouts/omny_freerun.ini

To restore either, swap --save-layout for --load-layout with the same file. Diffing the two confirms the mode-specific differences documented above: PCAP.GATE/PCAP.TRIG are TTLIN1.VAL in Mode A vs CLOCK1.OUT in Mode B; CLOCK1.ENABLE is ZERO (off) in Mode A vs ONE in Mode B; INENC1-4.VAL.CAPTURE/FMC_IN.VAL1/VAL2.CAPTURE are Min Max Mean in Mode A vs Value in Mode B; PCAP.BITS0.CAPTURE/COUNTER1.OUT.CAPTURE are No in Mode A vs Value in Mode B.

omny_freerun.ini was captured with CLOCK1.PERIOD=0.0001 (100 µs → 10 kHz) — note this doesn't match the 20 kHz reported as tested successfully the same day, so it may not be the exact layout state the 20 kHz test ran at; worth double-checking before relying on this file as "the" validated high-rate layout.

Signal alias mapping

The physical PandA block feeding each BEC signal name, kept here and in csaxs_bec/device_configs/ptycho_flomni.yaml (omny_panda.deviceConfig.signal_alias for Mode A, omny_panda_continuous.deviceConfig.signal_alias for Mode B) in sync as a single source of truth:

PandA block	Mode	BEC signal name
FMC_IN.VAL1.Min/.Max/.Mean	A	cap_voltage_fzp_y_min/_max/_mean
FMC_IN.VAL2.Min/.Max/.Mean	A	cap_voltage_fzp_x_min/_max/_mean
INENC1.VAL.Min/.Max/.Mean	A	interf_st_fzp_y_min/_max/_mean
INENC2.VAL.Min/.Max/.Mean	A	interf_st_fzp_x_min/_max/_mean
INENC3.VAL.Min/.Max/.Mean	A	interf_st_rotz_min/_max/_mean
INENC4.VAL.Min/.Max/.Mean	A	interf_st_rotx_min/_max/_mean
PCAP.GATE_DURATION.Value	A, B	pcap_gate_duration_value
FMC_IN.VAL1/VAL2.Value, INENC1-4.VAL.Value	B	cap_voltage_fzp_y/_x, interf_st_fzp_y/_x/_rotz/_rotx
PCAP.BITS0.Value (bit 0 = TTLIN1.VAL, confirmed 2026-09-15; masked in software, see above)	B	gate_detector_active
COUNTER1.OUT.Value (confirmed 2026-09-15, triggered off TTLIN1.VAL's falling edge)	B	frame_counter

The BEC-side data schema produced by Mode B (what ends up in the HDF5 async dataset) is documented separately in docs/developer/panda_box_free_running_data_schema.md.

Known data-quality caveat: stale frame_counter carryover (found + fixed 2026-09-21)

Diagnosed against a real scan (S06313): grouping raw interf_st_fzp_x/_y samples by frame_counter (restricted to gate_detector_active == 1) and averaging per group reproduces rt_positions's trusted average_x_st_fzp/average_y_st_fzp to R² > 0.9999 once a single constant per-axis offset is removed -- an expected reference-zero difference between the PandA/INENC counter and RT's own internal tracking, irrelevant to reconstruction since it's the same constant for every point in a scan. This also confirms the grouping/filtering method itself: naively treating every raw row as its own position (without grouping by frame_counter and filtering to gate_detector_active == 1) does not reproduce the true positions and cannot be fixed by any scale/offset correction, since most raw rows are samples taken between real exposures, not at them (see "Continuous gate/arm" above).

One point out of 472 was a clear outlier under that method: its frame_counter group had 38 extra samples (1036 vs. the usual ~998) and its recomputed position was off by ~116 nm in Y, against ~5 nm typical.

Root cause: PandA's COUNTER1.OUT (the block behind frame_counter) is not cleared by Arm(). COUNTER1.ENABLE=PCAP.ACTIVE only gates counting, not the persisted output value, and COUNTER1.SET -- the block's reset-trigger input -- is wired to the constant ZERO in the saved layout (see layouts/omny_freerun.ini), i.e. never pulsed on arm, unlike ENABLE which is wired to a live source (PCAP.ACTIVE). So the first raw samples of a new acquisition can briefly carry a stale frame_counter value left over from the previous acquisition's tail -- observed directly in S06313's raw data: 36 samples reading a stale 167 before dropping to 0 and counting correctly from there. Harmless on its own, except that stale value collided with a real point index later in the same scan (point 167), so grouping by frame_counter silently mixed ~38 stale rows into that one point's average.

Fix, implemented in panda_box_omny.py (software-only, no PandA layout change needed): every time PCAP is armed -- including on_pre_scan retries, since each Arm() reopens the same stale-value window -- PandaBoxOMNY now drops any raw rows captured before it observes frame_counter == 0 for the first time in that acquisition (_drop_stale_frame_counter_rows, called from _receive_frame_data), and drops any leftover buffered rows from an abandoned previous attempt at the same point (_reset_raw_stream_state, called from on_stage and from _try_arm right before every Arm()). Every signal's row list is truncated at the same index to keep channels aligned. Verified against S06313: dropping the (37, in that scan) leading stale rows before grouping brings point 167's residual from ~116 nm down to ~0.1 nm, in line with the rest of the scan.

A more thorough fix would rewire COUNTER1.SET on the PandA layout itself (from the constant ZERO to PCAP.ACTIVE's rising edge, matching how ENABLE is already wired) so OUT resets at the source on every arm, rather than relying on the client to detect and discard the stale prefix. Not implemented here since it requires a PandA layout change to be tested against live hardware; the software-side drop is a correct and sufficient fix in the meantime, and remains a reasonable defense-in-depth even if the layout is fixed later. If this is revisited: re-verify against a range of scans (especially ones with frame_counter staying below a few hundred, where a stale-value collision like point 167's is most likely) before removing the software-side check.

This also means the "gap in an otherwise-monotonic sequence" completeness check described in the mode-comparison table's "Data completeness verification" row needs one caveat: a single backward step at the very start of the raw sequence, from the previous acquisition's terminal value down to 0, is this expected (now-filtered) artifact, not a sign of dropped rows.

Conditional arming (arm_scan_allowlist, added 2026-09-22)

Generic BEC staging (stage_all_devices/pre_scan_all_devices in bec_server.scan_server.scans.scan_actions) has no concept of "this device only matters for scan X" -- every enabled: true device gets stage()/ pre_scan()/unstage() called for every scan that runs (grid_scan, line_scan, alignment moves, ...), not just the ones that actually consume its data. deviceTags (e.g. ptycho_flomni) are never consulted by that staging code -- they're purely a client-side grouping convenience. So without any further gating, omny_panda_continuous was being Arm()ed and disarmed for every single scan run during a session, even though only flomni_fermat_scan reads its data. That's unnecessary hardware arm/disarm cycling, and it directly compounds the stale-frame_counter bug above: more arm cycles mean more windows where a mid-scan abort can leave a stale counter value for the next acquisition to inherit (see AI_docs/PANDA_POSITION_VALIDATION.md sec 7 for a real incident traced to exactly this).

PandaBoxOMNY now takes an arm_scan_allowlist: list[str] | None constructor kwarg (None by default -- arm for every scan, unchanged behavior). When set, on_stage() computes _should_arm_panda from scan_parameters.scan_name against that list; on_pre_scan() skips sending Arm() entirely (resolves immediately) when it's False, and on_complete() skips the wait-for-frame-count poll loop the same way. on_stage()'s own readout-thread/TCP-connection startup is untouched by this -- it doesn't touch PandA's hardware register state (COUNTER1), unlike Arm(), so it stays out of scope. ptycho_flomni.yaml's omny_panda_continuous sets arm_scan_allowlist: [flomni_fermat_scan].

Switching between modes

omny_panda (Mode A) and omny_panda_continuous (Mode B) need different PandA hardware layouts and can't run simultaneously (see "Open question" below), so exactly one should be active in ptycho_flomni.yaml at a time, matching whichever layout is currently loaded on the PandA — comment out the other one's block entirely, don't rely on enabled: false alone.

That's a correction from earlier in this project: enabled: false on a device BEC has never connected to is an inert placeholder — connect_device in bec_server/device_server/devices/devicemanager.py is only called when enabled — so the theory was that both entries could stay uncommented and switching would just be flipping the flag. In practice (2026-09-15), disabling omny_panda_continuous this way was not sufficient once it had already been connected in the running session: both devices ended up active simultaneously against the same physical PandA. A device that's already connected doesn't appear to get disconnected just because a later config reload marks it disabled. Commenting the block out (so BEC never even constructs the device object) is the reliable way to guarantee it's gone; if in doubt, confirm the other device has actually dropped its connection (e.g. via a device server restart) rather than trusting enabled: false alone.

Confirmed end-to-end 2026-09-15: Mode B tested working (correct per-channel values, masked gate_detector_active, frame_counter-based completion), then switched back to Mode A (commented out) for A/B comparison testing.

Open question, now partially resolved

Can both modes coexist under a single static PandA layout — i.e. is mode switching purely the raw_stream_mode software flag, with the PandA always capturing both the aggregate and raw fields simultaneously — or do Mode A and Mode B require two distinct saved layouts that must be loaded before a scan depending on the desired mode?

As currently wired for testing (2026-09-15), the answer looks like two distinct layouts are required: FMC_IN.VAL1/VAL2 and INENC1-4.VAL are each set to CAPTURE=Value (single selection), not simultaneously CAPTURE=Value,Min,Max,Mean — so this layout cannot also produce Mode A's aggregate statistics at the same time. This has not yet been deliberately tested against the alternative (whether the PandA capture UI actually supports selecting Value together with Min/Max/Mean on the same field) — if that turns out to work, a single layout could serve both modes. Once Mode B testing is far enough along to save a named layout (see "Saving / restoring layouts" below), settle this explicitly and record the answer plus, if two layouts are needed, the load procedure to switch between them.