feat(flomni): add continuous free-running raw-sample mode to PandaBoxOMNY
CI for csaxs_bec / test (push) Successful in 2m14s

Adds an opt-in raw_stream_mode to omny_panda alongside the existing
per-gate averaged burst acquisition, so a burst window can instead be
captured as continuous raw samples plus a detector-gate bit and a
monotonic frame counter for completeness checking. Client-side
coalescing bounds the message rate to the BEC message bus regardless of
PandA's own network-frame batching, and completion uses a settle-based
check since an exact expected sample count isn't predictable for
continuous capture. Defaults to off, preserving today's behavior
exactly; hardware wiring/layout and the raw data schema are documented
under docs/developer/ pending PandA hardware access to configure and
validate against.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
x01dcandClaude Sonnet 5 authored and x12sa committed 2026-09-15 13:44:07 +02:00
1 parent 3f719c8119
commit 82ecab4c21
7 files changed
+551 -32

No files matched your search

@@ -0,0 +1,47 @@
# PandABox free-running (`raw_stream_mode`) raw data schema
> **Status: schema for the acquisition side only.** This document defines
> what raw data `omny_panda` writes to its HDF5 async dataset when
> `raw_stream_mode: true` (see `docs/developer/panda_box_free_running_setup.md`
> for the hardware/mode setup). Reconstructing the previous
> averaged/stdev-per-position view from this raw data (binning by the
> `gate_detector_active` transitions or timestamps) is a separate,
> not-yet-planned analysis task and is intentionally out of scope here.
## Where this data lands
`omny_panda` is configured with `readoutPriority: async`
(`csaxs_bec/device_configs/ptycho_flomni.yaml`). Its `data` signal is a
`DynamicSignal` (`ophyd_devices/devices/panda_box/panda_box.py`) that BEC's
async file writer appends to an HDF5 dataset per scan, under this device's
async data group. In `raw_stream_mode`, each write corresponds to one
coalesced flush (see the setup doc's mode-comparison table), not one raw
PandA network frame — the row-level data below is still preserved exactly,
just batched differently for publishing efficiency.
## Per-row fields
Each row corresponds to one PCAP capture tick (one raw sample at the
free-running capture rate, e.g. ~20 kHz — see the setup doc for the actual
configured rate):
| Field (BEC signal name) | Source PandA block | Meaning |
|---|---|---|
| *(raw analog/position value(s), name TBD per setup doc)* | e.g. `FMC_IN.VAL1.Value` / `INENC1.VAL.Value` | The instantaneous raw sample at this capture tick, replacing the Mode A aggregate (Mean/Min/Max) for whichever channel(s) are switched to raw capture. |
| `gate_detector_active` | `PCAP.BITS0.Value` (block TBD) | The other detectors' gate signal, sampled at this same tick: `1` while the other detectors are actively gated/acquiring, `0` otherwise. This is what a future reconstruction step would use to delimit which raw samples belong to which detector-gate window. |
| `frame_counter` | `COUNTER1.OUT.Value` (block TBD) | Monotonically incrementing hardware counter, one increment per capture tick. Used to verify data completeness after acquisition: a gap-free, strictly increasing sequence means no samples were dropped between the PandA and the HDF5 file; a gap indicates lost data (e.g. from the Redis stream's `MAXLEN` trimming if the file writer fell behind — see the acquisition plan's data-path notes). |
| `PCAP.TS_TRIG.Value` (existing) | `PCAP.TS_TRIG` | Absolute/relative timestamp already captured today; usable for time-alignment against other detectors' data during future reconstruction. |
| `PCAP.GATE_DURATION.Value` (existing, aliased as `pcap_gate_duration_value`) | `PCAP.GATE_DURATION` | Existing field, retained; less meaningful once the gate is held open continuously for the whole burst window rather than pulsed per point (see setup doc, Mode B gate/arm row) — re-evaluate whether this still needs to be captured once real hardware is available. |
All rows across a burst window share the same set of fields; there is no
per-point boundary marker in this raw stream other than what can be derived
from `gate_detector_active` transitions and/or absolute timestamps — no
explicit "scan point index" field is captured by the PandA itself.
## What this does *not* include
- No on-the-fly or post-scan computation of Mean/Min/Max/stdev from the raw
samples — the raw stream is the complete record; deriving the previous
per-point statistics from it is a separate future task.
- No changes to how Mode A (legacy) data is written — this schema only
applies when `raw_stream_mode: true`.
@@ -0,0 +1,122 @@
# PandABox (`omny_panda`) burst acquisition: two operation modes
> **Status: design + software implemented, hardware not yet configured or
> tested.** The PandA layout changes and target sample rate below are
> placeholders (marked TODO) until real PandA hardware is available to
> configure and measure against. This document should be updated with actual
> values once that happens, per the flomni burst-acquisition redesign plan.
## Context
flomni burst scans acquire `frames_per_trigger` sub-frames per scan point.
Historically, the PandABox (`omny_panda`, device class `PandaBoxOMNY` in
`csaxs_bec/devices/panda_box/panda_box_omny.py`) reports one PCAP-aggregated
statistic (Mean/Min/Max/Sum/Diff) per gate, discarding the time structure
within a gate. This document describes the two supported operation modes —
the existing per-gate aggregate mode, and a new continuous free-running raw
mode — and how the PandA hardware and `ptycho_flomni.yaml` device config must
be set up for each.
Mode selection is a single software flag (`raw_stream_mode` in the device
config); it does **not** by itself reconfigure the PandA hardware. The PandA
layout must independently match the intended mode (see "Open question" at
the end) before that mode can be used.
## Mode comparison
| | Mode A: legacy (default) | Mode B: free-running (new) |
|---|---|---|
| Selected by | `raw_stream_mode` unset / `false` | `raw_stream_mode: true` |
| PCAP capture fields | `Mean`/`Min`/`Max`/`Sum`/`Diff` per gate | `Value` (raw instantaneous sample) + gate bit + frame counter |
| Gate/arm timing | Pulsed per point / sub-frame (rate = scan rate × `frames_per_trigger`) | PCAP held open continuously for the burst window, free-running at a high internal sample rate (target: **TODO**, e.g. ~20 kHz) |
| `_acquisition_group` (BEC async metadata) | `fly` / `monitored` / `burst`, chosen from `scan_type`/`frames_per_trigger` | `free_running` |
| Message publishing | One `.data.put()` per PandA network frame, unchanged | Coalesced client-side: flushed every `raw_stream_flush_row_count` rows or `raw_stream_flush_interval` seconds, whichever comes first (mitigates Redis/HDF5 message-rate risk — see the acquisition plan's data-path notes) |
| Completion check (`on_complete`) | Exact count: waits for `*PCAP.CAPTURED?` to reach `num_points * frames_per_trigger` | Settle-based: waits for `*PCAP.CAPTURED?` to stop changing for `_settle_timeout_in_s`, since an exact expected sample count can't be predicted client-side for continuous capture |
| Data completeness verification | Implicit in the exact-count completion check | The `frame_counter` signal (Mode B only) — a monotonically incrementing hardware counter — is the record to check for gaps/drops after the fact |
Both modes are implemented in the same `PandaBoxOMNY` class; `raw_stream_mode`
defaults to `false` so any device config that doesn't set it keeps today's
exact behavior unchanged.
## Physical wiring (Mode B only)
**TODO**: which PandA digital input connector/pin the other detectors' gate
TTL signal is patched into. Fill in once the cabling is done.
## Block configuration
### Mode A (legacy) — reference baseline
The blocks already configured and aliased today, for reference / so this
layout can be restored unambiguously (see `ptycho_flomni.yaml`):
- `FMC_IN.VAL1`/`FMC_IN.VAL2` (capacitive sensor analog inputs) — `Min`,
`Max`, `Mean` capture fields.
- `INENC1`-`INENC4.VAL` (position-capture encoder inputs) — `Min`, `Max`,
`Mean` capture fields.
- `PCAP.GATE_DURATION` — `Value`.
- PCAP gate is pulsed once per point / sub-frame by the existing external
trigger electronics (unchanged by this work).
### Mode B (new) — free-running raw capture
- **Gate-bit input**: the other detectors' gate signal (wired above) →
**TODO: which `PCAP.BITSx` block and bit position** — set
`CAPTURE=Value` so every captured row records the instantaneous gate
state (0/1).
- **Frame counter**: dedicate **TODO: which `COUNTERx.OUT` block** (any of
`COUNTER1`-`COUNTER8` not already in use) to free-run, incrementing once
per PCAP capture tick, with `CAPTURE=Value`. This is the
data-completeness signal (`frame_counter` in `signal_alias`, see below).
- **Raw analog/position capture**: switch **TODO: which of
`FMC_IN.VAL1/VAL2`/`INENC1`-`4.VAL`** from aggregate capture to
`CAPTURE=Value`, so the raw time series is recorded instead of (or in
addition to) the per-gate statistics.
- **Continuous gate/arm**: instead of pulsing PCAP per point, keep the gate
open for the whole burst window (armed once per point-group / scan) so
PCAP free-runs at the target sample rate — **TODO: confirm target rate**
(design target: ~20 kHz) and how the continuous gate is generated (e.g. a
PandA-internal clock/pulse-generator block feeding PCAP's gate input,
rather than the external per-point trigger used in Mode A).
## Saving / restoring layouts
Use `ophyd_devices/devices/panda_box/utility_scripts.py` to save the current
PandA configuration to a `.ini` layout file and reload it later, so a
configuration survives a PandA power cycle. **TODO**: once Mode B's block
configuration is finalized against real hardware, save it as a named layout
(e.g. `flomni_panda_free_running.ini`) alongside the existing Mode A layout,
and record the exact save/load commands used here.
## Signal alias mapping
The physical PandA block feeding each BEC signal name, kept here and in
`csaxs_bec/device_configs/ptycho_flomni.yaml` (`omny_panda.deviceConfig.signal_alias`)
in sync as a single source of truth:
| PandA block | Mode | BEC signal name |
|---|---|---|
| `FMC_IN.VAL1.Min`/`.Max`/`.Mean` | A | `cap_voltage_fzp_y_min`/`_max`/`_mean` |
| `FMC_IN.VAL2.Min`/`.Max`/`.Mean` | A | `cap_voltage_fzp_x_min`/`_max`/`_mean` |
| `INENC1.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_fzp_y_min`/`_max`/`_mean` |
| `INENC2.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_fzp_x_min`/`_max`/`_mean` |
| `INENC3.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_rotz_min`/`_max`/`_mean` |
| `INENC4.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_rotx_min`/`_max`/`_mean` |
| `PCAP.GATE_DURATION.Value` | A | `pcap_gate_duration_value` |
| `PCAP.BITS0.Value` (**TODO: confirm bit block**) | B | `gate_detector_active` |
| `COUNTER1.OUT.Value` (**TODO: confirm counter block**) | B | `frame_counter` |
The BEC-side data schema produced by Mode B (what ends up in the HDF5 async
dataset) is documented separately in
`docs/developer/panda_box_free_running_data_schema.md`.
## Open question to resolve once hardware is available
Can both modes coexist under a single static PandA layout — i.e. is mode
switching purely the `raw_stream_mode` software flag, with the PandA always
capturing both the aggregate and raw fields simultaneously — or do Mode A
and Mode B require two distinct saved layouts that must be loaded before a
scan depending on the desired mode? If two layouts are required,
`raw_stream_mode` alone is necessary but not sufficient to switch modes, and
the operator must also load the matching PandA layout beforehand — update
this document with the answer and, if applicable, the load procedure.