feat(flomni): add continuous free-running raw-sample mode to PandaBoxOMNY
CI for csaxs_bec / test (push) Successful in 2m14s
CI for csaxs_bec / test (push) Successful in 2m14s
Adds an opt-in raw_stream_mode to omny_panda alongside the existing per-gate averaged burst acquisition, so a burst window can instead be captured as continuous raw samples plus a detector-gate bit and a monotonic frame counter for completeness checking. Client-side coalescing bounds the message rate to the BEC message bus regardless of PandA's own network-frame batching, and completion uses a settle-based check since an exact expected sample count isn't predictable for continuous capture. Defaults to off, preserving today's behavior exactly; hardware wiring/layout and the raw data schema are documented under docs/developer/ pending PandA hardware access to configure and validate against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3f719c8119
commit
82ecab4c21
7 files changed
+551
-32
No files matched your search
@@ -0,0 +1,47 @@
|
||||
# PandABox free-running (`raw_stream_mode`) raw data schema
|
||||
|
||||
> **Status: schema for the acquisition side only.** This document defines
|
||||
> what raw data `omny_panda` writes to its HDF5 async dataset when
|
||||
> `raw_stream_mode: true` (see `docs/developer/panda_box_free_running_setup.md`
|
||||
> for the hardware/mode setup). Reconstructing the previous
|
||||
> averaged/stdev-per-position view from this raw data (binning by the
|
||||
> `gate_detector_active` transitions or timestamps) is a separate,
|
||||
> not-yet-planned analysis task and is intentionally out of scope here.
|
||||
|
||||
## Where this data lands
|
||||
|
||||
`omny_panda` is configured with `readoutPriority: async`
|
||||
(`csaxs_bec/device_configs/ptycho_flomni.yaml`). Its `data` signal is a
|
||||
`DynamicSignal` (`ophyd_devices/devices/panda_box/panda_box.py`) that BEC's
|
||||
async file writer appends to an HDF5 dataset per scan, under this device's
|
||||
async data group. In `raw_stream_mode`, each write corresponds to one
|
||||
coalesced flush (see the setup doc's mode-comparison table), not one raw
|
||||
PandA network frame — the row-level data below is still preserved exactly,
|
||||
just batched differently for publishing efficiency.
|
||||
|
||||
## Per-row fields
|
||||
|
||||
Each row corresponds to one PCAP capture tick (one raw sample at the
|
||||
free-running capture rate, e.g. ~20 kHz — see the setup doc for the actual
|
||||
configured rate):
|
||||
|
||||
| Field (BEC signal name) | Source PandA block | Meaning |
|
||||
|---|---|---|
|
||||
| *(raw analog/position value(s), name TBD per setup doc)* | e.g. `FMC_IN.VAL1.Value` / `INENC1.VAL.Value` | The instantaneous raw sample at this capture tick, replacing the Mode A aggregate (Mean/Min/Max) for whichever channel(s) are switched to raw capture. |
|
||||
| `gate_detector_active` | `PCAP.BITS0.Value` (block TBD) | The other detectors' gate signal, sampled at this same tick: `1` while the other detectors are actively gated/acquiring, `0` otherwise. This is what a future reconstruction step would use to delimit which raw samples belong to which detector-gate window. |
|
||||
| `frame_counter` | `COUNTER1.OUT.Value` (block TBD) | Monotonically incrementing hardware counter, one increment per capture tick. Used to verify data completeness after acquisition: a gap-free, strictly increasing sequence means no samples were dropped between the PandA and the HDF5 file; a gap indicates lost data (e.g. from the Redis stream's `MAXLEN` trimming if the file writer fell behind — see the acquisition plan's data-path notes). |
|
||||
| `PCAP.TS_TRIG.Value` (existing) | `PCAP.TS_TRIG` | Absolute/relative timestamp already captured today; usable for time-alignment against other detectors' data during future reconstruction. |
|
||||
| `PCAP.GATE_DURATION.Value` (existing, aliased as `pcap_gate_duration_value`) | `PCAP.GATE_DURATION` | Existing field, retained; less meaningful once the gate is held open continuously for the whole burst window rather than pulsed per point (see setup doc, Mode B gate/arm row) — re-evaluate whether this still needs to be captured once real hardware is available. |
|
||||
|
||||
All rows across a burst window share the same set of fields; there is no
|
||||
per-point boundary marker in this raw stream other than what can be derived
|
||||
from `gate_detector_active` transitions and/or absolute timestamps — no
|
||||
explicit "scan point index" field is captured by the PandA itself.
|
||||
|
||||
## What this does *not* include
|
||||
|
||||
- No on-the-fly or post-scan computation of Mean/Min/Max/stdev from the raw
|
||||
samples — the raw stream is the complete record; deriving the previous
|
||||
per-point statistics from it is a separate future task.
|
||||
- No changes to how Mode A (legacy) data is written — this schema only
|
||||
applies when `raw_stream_mode: true`.
|
||||
@@ -0,0 +1,122 @@
|
||||
# PandABox (`omny_panda`) burst acquisition: two operation modes
|
||||
|
||||
> **Status: design + software implemented, hardware not yet configured or
|
||||
> tested.** The PandA layout changes and target sample rate below are
|
||||
> placeholders (marked TODO) until real PandA hardware is available to
|
||||
> configure and measure against. This document should be updated with actual
|
||||
> values once that happens, per the flomni burst-acquisition redesign plan.
|
||||
|
||||
## Context
|
||||
|
||||
flomni burst scans acquire `frames_per_trigger` sub-frames per scan point.
|
||||
Historically, the PandABox (`omny_panda`, device class `PandaBoxOMNY` in
|
||||
`csaxs_bec/devices/panda_box/panda_box_omny.py`) reports one PCAP-aggregated
|
||||
statistic (Mean/Min/Max/Sum/Diff) per gate, discarding the time structure
|
||||
within a gate. This document describes the two supported operation modes —
|
||||
the existing per-gate aggregate mode, and a new continuous free-running raw
|
||||
mode — and how the PandA hardware and `ptycho_flomni.yaml` device config must
|
||||
be set up for each.
|
||||
|
||||
Mode selection is a single software flag (`raw_stream_mode` in the device
|
||||
config); it does **not** by itself reconfigure the PandA hardware. The PandA
|
||||
layout must independently match the intended mode (see "Open question" at
|
||||
the end) before that mode can be used.
|
||||
|
||||
## Mode comparison
|
||||
|
||||
| | Mode A: legacy (default) | Mode B: free-running (new) |
|
||||
|---|---|---|
|
||||
| Selected by | `raw_stream_mode` unset / `false` | `raw_stream_mode: true` |
|
||||
| PCAP capture fields | `Mean`/`Min`/`Max`/`Sum`/`Diff` per gate | `Value` (raw instantaneous sample) + gate bit + frame counter |
|
||||
| Gate/arm timing | Pulsed per point / sub-frame (rate = scan rate × `frames_per_trigger`) | PCAP held open continuously for the burst window, free-running at a high internal sample rate (target: **TODO**, e.g. ~20 kHz) |
|
||||
| `_acquisition_group` (BEC async metadata) | `fly` / `monitored` / `burst`, chosen from `scan_type`/`frames_per_trigger` | `free_running` |
|
||||
| Message publishing | One `.data.put()` per PandA network frame, unchanged | Coalesced client-side: flushed every `raw_stream_flush_row_count` rows or `raw_stream_flush_interval` seconds, whichever comes first (mitigates Redis/HDF5 message-rate risk — see the acquisition plan's data-path notes) |
|
||||
| Completion check (`on_complete`) | Exact count: waits for `*PCAP.CAPTURED?` to reach `num_points * frames_per_trigger` | Settle-based: waits for `*PCAP.CAPTURED?` to stop changing for `_settle_timeout_in_s`, since an exact expected sample count can't be predicted client-side for continuous capture |
|
||||
| Data completeness verification | Implicit in the exact-count completion check | The `frame_counter` signal (Mode B only) — a monotonically incrementing hardware counter — is the record to check for gaps/drops after the fact |
|
||||
|
||||
Both modes are implemented in the same `PandaBoxOMNY` class; `raw_stream_mode`
|
||||
defaults to `false` so any device config that doesn't set it keeps today's
|
||||
exact behavior unchanged.
|
||||
|
||||
## Physical wiring (Mode B only)
|
||||
|
||||
**TODO**: which PandA digital input connector/pin the other detectors' gate
|
||||
TTL signal is patched into. Fill in once the cabling is done.
|
||||
|
||||
## Block configuration
|
||||
|
||||
### Mode A (legacy) — reference baseline
|
||||
|
||||
The blocks already configured and aliased today, for reference / so this
|
||||
layout can be restored unambiguously (see `ptycho_flomni.yaml`):
|
||||
|
||||
- `FMC_IN.VAL1`/`FMC_IN.VAL2` (capacitive sensor analog inputs) — `Min`,
|
||||
`Max`, `Mean` capture fields.
|
||||
- `INENC1`-`INENC4.VAL` (position-capture encoder inputs) — `Min`, `Max`,
|
||||
`Mean` capture fields.
|
||||
- `PCAP.GATE_DURATION` — `Value`.
|
||||
- PCAP gate is pulsed once per point / sub-frame by the existing external
|
||||
trigger electronics (unchanged by this work).
|
||||
|
||||
### Mode B (new) — free-running raw capture
|
||||
|
||||
- **Gate-bit input**: the other detectors' gate signal (wired above) →
|
||||
**TODO: which `PCAP.BITSx` block and bit position** — set
|
||||
`CAPTURE=Value` so every captured row records the instantaneous gate
|
||||
state (0/1).
|
||||
- **Frame counter**: dedicate **TODO: which `COUNTERx.OUT` block** (any of
|
||||
`COUNTER1`-`COUNTER8` not already in use) to free-run, incrementing once
|
||||
per PCAP capture tick, with `CAPTURE=Value`. This is the
|
||||
data-completeness signal (`frame_counter` in `signal_alias`, see below).
|
||||
- **Raw analog/position capture**: switch **TODO: which of
|
||||
`FMC_IN.VAL1/VAL2`/`INENC1`-`4.VAL`** from aggregate capture to
|
||||
`CAPTURE=Value`, so the raw time series is recorded instead of (or in
|
||||
addition to) the per-gate statistics.
|
||||
- **Continuous gate/arm**: instead of pulsing PCAP per point, keep the gate
|
||||
open for the whole burst window (armed once per point-group / scan) so
|
||||
PCAP free-runs at the target sample rate — **TODO: confirm target rate**
|
||||
(design target: ~20 kHz) and how the continuous gate is generated (e.g. a
|
||||
PandA-internal clock/pulse-generator block feeding PCAP's gate input,
|
||||
rather than the external per-point trigger used in Mode A).
|
||||
|
||||
## Saving / restoring layouts
|
||||
|
||||
Use `ophyd_devices/devices/panda_box/utility_scripts.py` to save the current
|
||||
PandA configuration to a `.ini` layout file and reload it later, so a
|
||||
configuration survives a PandA power cycle. **TODO**: once Mode B's block
|
||||
configuration is finalized against real hardware, save it as a named layout
|
||||
(e.g. `flomni_panda_free_running.ini`) alongside the existing Mode A layout,
|
||||
and record the exact save/load commands used here.
|
||||
|
||||
## Signal alias mapping
|
||||
|
||||
The physical PandA block feeding each BEC signal name, kept here and in
|
||||
`csaxs_bec/device_configs/ptycho_flomni.yaml` (`omny_panda.deviceConfig.signal_alias`)
|
||||
in sync as a single source of truth:
|
||||
|
||||
| PandA block | Mode | BEC signal name |
|
||||
|---|---|---|
|
||||
| `FMC_IN.VAL1.Min`/`.Max`/`.Mean` | A | `cap_voltage_fzp_y_min`/`_max`/`_mean` |
|
||||
| `FMC_IN.VAL2.Min`/`.Max`/`.Mean` | A | `cap_voltage_fzp_x_min`/`_max`/`_mean` |
|
||||
| `INENC1.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_fzp_y_min`/`_max`/`_mean` |
|
||||
| `INENC2.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_fzp_x_min`/`_max`/`_mean` |
|
||||
| `INENC3.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_rotz_min`/`_max`/`_mean` |
|
||||
| `INENC4.VAL.Min`/`.Max`/`.Mean` | A | `interf_st_rotx_min`/`_max`/`_mean` |
|
||||
| `PCAP.GATE_DURATION.Value` | A | `pcap_gate_duration_value` |
|
||||
| `PCAP.BITS0.Value` (**TODO: confirm bit block**) | B | `gate_detector_active` |
|
||||
| `COUNTER1.OUT.Value` (**TODO: confirm counter block**) | B | `frame_counter` |
|
||||
|
||||
The BEC-side data schema produced by Mode B (what ends up in the HDF5 async
|
||||
dataset) is documented separately in
|
||||
`docs/developer/panda_box_free_running_data_schema.md`.
|
||||
|
||||
## Open question to resolve once hardware is available
|
||||
|
||||
Can both modes coexist under a single static PandA layout — i.e. is mode
|
||||
switching purely the `raw_stream_mode` software flag, with the PandA always
|
||||
capturing both the aggregate and raw fields simultaneously — or do Mode A
|
||||
and Mode B require two distinct saved layouts that must be loaded before a
|
||||
scan depending on the desired mode? If two layouts are required,
|
||||
`raw_stream_mode` alone is necessary but not sufficient to switch modes, and
|
||||
the operator must also load the matching PandA layout beforehand — update
|
||||
this document with the answer and, if applicable, the load procedure.
|
||||
Reference in new issue
Block a user