csaxs_bec/docs/developer/panda_box_free_running_data_schema.md

PandABox free-running (raw_stream_mode) raw data schema

Status: schema for the acquisition side only. This document defines what raw data omny_panda writes to its HDF5 async dataset when raw_stream_mode: true (see docs/developer/panda_box_free_running_setup.md for the hardware/mode setup, confirmed against real PandA hardware 2026-09-15). Reconstructing the previous averaged/stdev-per-position view from this raw data (binning by the gate_detector_active transitions or timestamps) is a separate, not-yet-planned analysis task and is intentionally out of scope here.

Where this data lands

omny_panda is configured with readoutPriority: async (csaxs_bec/device_configs/ptycho_flomni.yaml). Its data signal is a DynamicSignal (ophyd_devices/devices/panda_box/panda_box.py) that BEC's async file writer appends to an HDF5 dataset per scan, under this device's async data group. In raw_stream_mode, each write corresponds to one coalesced flush (see the setup doc's mode-comparison table), not one raw PandA network frame — the row-level data below is still preserved exactly, just batched differently for publishing efficiency.

Per-row fields

Each row corresponds to one PCAP capture tick (one raw sample at the free-running capture rate — CLOCK1's configured rate, started at 100 Hz for initial testing on 2026-09-15, see the setup doc for the current rate):

Field (BEC signal name)	Source PandA block	Meaning
cap_voltage_fzp_y/cap_voltage_fzp_x etc. (raw analog/encoder value)	FMC_IN.VAL1.Value / FMC_IN.VAL2.Value / INENC1-4.VAL.Value	The instantaneous raw sample at this capture tick, replacing the Mode A aggregate (Mean/Min/Max). All six channels (both FMC analog inputs and all four encoders) are switched to raw capture together on this layout — not a subset.
gate_detector_active	PCAP.BITS0.Value, bit 0 (TTLIN1.VAL), masked in software — see setup doc	The detector-trigger signal, sampled at this same tick: 1 while a detector trigger is active, 0 otherwise. PCAP.BITS0 is a shared 32-bit word; PandaBoxOMNY masks it down to bit 0 before this reaches the HDF5 file (see _GATE_BIT_OFFSET in panda_box_omny.py) — the raw word itself is not a clean 0/1. This is what a future reconstruction step would use to delimit which raw samples belong to which detector-gate window.
frame_counter	COUNTER1.OUT.Value, triggered off TTLIN1.VAL's falling edge (i.e. increments once per completed detector exposure, not once per PCAP capture tick as originally guessed here)	Monotonically incrementing hardware counter. Doubles as the completion signal for on_complete (PandaBoxOMNY._on_complete_free_running, panda_box_omny.py) — waits for it to reach num_points * frames_per_trigger, since *PCAP.CAPTURED? free-runs continuously and never naturally stops while armed. Also the post-hoc data-completeness record: a gap-free, strictly increasing sequence means no detector exposures were missed between the PandA and the HDF5 file; a gap indicates lost data (e.g. from the Redis stream's MAXLEN trimming if the file writer fell behind — see the acquisition plan's data-path notes). Because it counts completed exposures rather than capture ticks, it verifies detector-frame completeness directly rather than raw-sample completeness.
PCAP.TS_TRIG.Value (existing)	PCAP.TS_TRIG	Absolute/relative timestamp already captured today; usable for time-alignment against other detectors' data during future reconstruction.
PCAP.GATE_DURATION.Value (existing, aliased as pcap_gate_duration_value)	PCAP.GATE_DURATION	Existing field, retained; less meaningful once the gate is held open continuously for the whole burst window rather than pulsed per point (see setup doc, Mode B gate/arm row) — re-evaluate whether this still needs to be captured now that real hardware testing has started.

All rows across a burst window share the same set of fields; there is no per-point boundary marker in this raw stream other than what can be derived from gate_detector_active transitions and/or absolute timestamps — no explicit "scan point index" field is captured by the PandA itself.

What this does not include

• No on-the-fly or post-scan computation of Mean/Min/Max/stdev from the raw samples — the raw stream is the complete record; deriving the previous per-point statistics from it is a separate future task.
• No changes to how Mode A (legacy) data is written — this schema only applies when raw_stream_mode: true.