docs(flomni): document PandA Mode B recovered-position validation method

Write-up for continuing this testing in a future session without needing to
re-derive it: the required gate_detector_active/frame_counter grouping
(and why naive per-row comparison can't be fixed by any scale/offset
correction), the exact interferometer + capacitive-sensor conversion
formula, the stale-frame_counter carryover bug and its fix (3eb6e03), and
the validation method (R²/residuals against rt_positions' trusted
average_x_st_fzp/_y). Also records the two agreed next steps: burst
acquisition testing and reduced data mode (the latter still undefined,
to be scoped in that session).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NPZ9gzN2rt3wgiMnTKyrug

csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/PANDA_POSITION_VALIDATION.md

@@ -0,0 +1,186 @@
+# Validating PandaBox (`omny_panda_continuous`) recovered positions against RT
+
+**Purpose of this document:** how to check whether positions recovered from
+the PandaBox's Mode B (free-running raw stream, `raw_stream_mode: true`)
+capture actually match the true stage positions, using the RT controller's
+own trusted per-point averages as ground truth. Read this before continuing
+that testing in a new session — it captures the full methodology, the exact
+conversion formula, and a bug that was found and fixed this way, so none of
+it needs re-deriving from scratch.
+
+Companion docs:
+- `docs/developer/panda_box_free_running_setup.md` — the PandA layout/mode
+  design doc. Has the mode comparison, signal alias mapping, and (as of
+  2026-09-21) the "Known data-quality caveat: stale `frame_counter`
+  carryover" section this investigation produced.
+- `docs/developer/panda_box_free_running_data_schema.md` — the HDF5 field
+  list for the Mode B raw stream.
+- `csaxs_bec/devices/panda_box/panda_box_omny.py` — the device. See
+  `_drop_stale_frame_counter_rows`/`_reset_raw_stream_state` for the fix
+  described below.
+
+---
+
+## 1. The problem this validates
+
+Mode B captures raw samples continuously at `CLOCK1`'s rate for the whole
+burst window — **not** one sample per scan point. Only a subset of rows
+coincide with an actual detector exposure (`gate_detector_active == 1`), and
+even those need to be grouped by which point they belong to
+(`frame_counter`). Treating every raw row as its own recovered position
+(without grouping/filtering) produces a scatter that roughly traces the
+scan's envelope but does not correspond point-for-point to the true
+positions — and critically, **no scale/offset correction can fix that**,
+because it's a many-to-one correspondence problem, not a coordinate error.
+This was confirmed visually (see the git history around 2026-09-20/21 for the
+red/blue scatter plot that started this investigation) before the grouping
+method below was worked out.
+
+## 2. Ground truth: `rt_positions`
+
+The RT controller logs its own per-point values, already correctly averaged
+and known-good (used successfully in reconstruction):
+
+- `rt_positions_data_target_x`/`_y` — commanded (programmed) position.
+- `rt_positions_data_average_x_st_fzp`/`_y` — **the trusted measured
+  position**, in microns. Compare recovered PandA positions against these,
+  not against `target_x`/`_y`.
+- `rt_positions_data_stdev_x_st_fzp`/`_y` — RT's own per-point stdev, useful
+  as a sanity scale for how much residual noise to expect (order 0.005 µm in
+  the validated scan).
+
+One row per scan point (472 points in the validated scan, `S06313`).
+
+## 3. Recovering positions from the Mode B raw stream
+
+### 3.1 Raw fields (all same length, one entry per raw sample)
+
+Under `entry/collection/devices/omny_panda_continuous/` in the master HDF5:
+`omny_panda_continuous_data_<field>/value`, where `<field>` is one of
+`interf_st_fzp_x`, `interf_st_fzp_y`, `cap_voltage_fzp_x`, `cap_voltage_fzp_y`,
+`frame_counter`, `gate_detector_active` (plus `interf_st_rotx`/`_rotz`,
+`pcap_gate_duration_value`, not needed for x/y position recovery).
+
+### 3.2 Required grouping/filtering — do this before anything else
+
+1. Filter to rows where `gate_detector_active == 1` (real exposure, not
+   inter-point flight time).
+2. Group the remaining rows by `frame_counter` (one group per scan point;
+   `frame_counter` ranges `0..num_points-1` for a single-frame-per-point
+   scan — see §6 for the burst case, still untested).
+3. Average `interf_st_fzp_x`/`_y` and `cap_voltage_fzp_x`/`_y` within each
+   group.
+
+Skipping this step is the #1 way to get a result that looks roughly
+right-shaped but doesn't actually match — see §1.
+
+### 3.3 Stale `frame_counter` carryover — check this if a session predates the fix
+
+PandA's `COUNTER1.OUT` is not cleared by `Arm()`, so the first raw samples of
+an acquisition can briefly carry a stale `frame_counter` value left over from
+the *previous* acquisition's tail. If that stale value collides with a real
+point index later in the same scan, it contaminates that point's group.
+`panda_box_omny.py` fixes this client-side as of commit `3eb6e03`
+(2026-09-21) — it drops rows before the first observed `frame_counter == 0`
+on every arm. **Any HDF5 file captured before that fix was live on the real
+device** (i.e. anything from before 2026-09-21, `S06313` included) still has
+the stale rows in it and needs the same drop applied manually before
+grouping:
+
+```python
+reset_idx = np.argmax(frame_counter_raw == 0.0)  # first occurrence
+frame_counter_raw = frame_counter_raw[reset_idx:]
+# ...apply the same [reset_idx:] slice to every other raw field, in lockstep
+```
+
+For anything captured after the fix, this is unnecessary (already handled
+device-side) — but it's cheap and harmless to keep doing as a defense-in-depth
+check; if it ever finds `reset_idx > 0` again on a fixed device, that's a
+regression worth investigating.
+
+### 3.4 Conversion formula (from Mirko, confirmed 2026-09-21)
+
+```python
+factor_x = -0.000309078
+factor_y = 0.000154539
+
+interfx = grouped_interf_st_fzp_x * factor_x   # microns
+interfy = grouped_interf_st_fzp_y * factor_y   # microns
+
+capmuy = -(grouped_cap_voltage_fzp_y) * 5.0 / 20.0
+capmux = (grouped_cap_voltage_fzp_x) * 5.0 / 20.0
+
+CapX_FZP_microns = 0.9539 * capmux + 0.0009008 * capmuy
+CapY_FZP_microns = -0.023140 * capmux + 1.0120 * capmuy
+
+pos_x_um = interfx - CapX_FZP_microns   # microns -- compare directly against average_x_st_fzp
+pos_y_um = interfy - CapY_FZP_microns   # microns -- compare directly against average_y_st_fzp
+```
+
+**Units trap:** the original formula this was transcribed from ends with
+`pos_x = (interfx - CapX_FZP_microns) * 1e-6` to get meters, for whatever
+consumed it downstream. `rt_positions_data_average_x_st_fzp`/`_y` are in
+**microns**, so compare against `pos_x_um`/`pos_y_um` *before* that final
+`* 1e-6` — don't convert one side to meters and leave the other in microns.
+
+`factor_x`/`factor_y` were independently re-derived by least-squares fit
+against `average_x_st_fzp`/`_y` from `S06313` and matched Mirko's values to 4
+significant figures (fit: `-3.089108e-04`/`1.545243e-04` vs. given
+`-0.000309078`/`0.000154539`) — good cross-check if a future factor is ever
+in doubt.
+
+## 4. Validating a match
+
+```python
+def r2(pred, true):
+    ss_res = np.sum((true - pred) ** 2)
+    ss_tot = np.sum((true - true.mean()) ** 2)
+    return 1 - ss_res / ss_tot
+```
+
+Expected result on a correctly grouped, correctly converted scan (from
+`S06313`, post-fix):
+- A single near-constant offset per axis between `pos_x_um`/`pos_y_um` and
+  `average_x_st_fzp`/`_y` (validated: X ≈ −1.26 µm, Y ≈ +0.99 µm) — this is
+  expected (RT's internal reference zero vs. the PandA/`INENC` counter's own
+  free-running zero don't coincide) and irrelevant to reconstruction, since
+  it's the same constant for every point in one scan. Subtract the mean
+  residual per axis before judging fit quality.
+- After removing that offset: R² > 0.9999 on both axes, residual std ~0.25–
+  0.3 nm, max residual under ~1 nm across all points (post-fix; pre-fix, one
+  point was off by ~116 nm — see §3.3).
+
+If R² is poor and doesn't clean up after removing a constant offset (i.e. a
+best-fit affine still doesn't work — this is the actual symptom that flagged
+the original bug, see §1), suspect the grouping/filtering step (§3.2) before
+suspecting the conversion factors.
+
+## 5. Practical notes
+
+- Use `/opt/bec_deployments/production/bec_venv/bin/python3` for any
+  h5py-based analysis — the system `python3` doesn't have `h5py` installed.
+- `S06313`'s files were at `/tmp/S06313/` (`S06313_master.h5` has both
+  `omny_panda_continuous` and `rt_positions`; `S06313_eiger_*` are detector
+  data, not needed for this). `/tmp` is not persistent — get a fresh scan's
+  files from whoever's driving the beamtime if this path is gone.
+- Relevant H5 group prefix:
+  `entry/collection/devices/<device>/<device>_data_<field>/value` (and
+  `.../timestamp`, much shorter — one entry per buffer flush, not per row,
+  see `_flush_raw_stream_buffer_locked` in `panda_box_omny.py`).
+
+## 6. Next steps (as of 2026-09-21, not yet done)
+
+1. **Burst acquisition**: test the case where the number of PandA frames
+   equals the number of burst sub-frames (`frames_per_trigger > 1`), and RT
+   still logs exactly one `average_x_st_fzp`/`_y` per *point* (not per
+   sub-frame). This changes the grouping in §3.2 — open question going in:
+   does `frame_counter` increment once per sub-frame trigger (meaning
+   `frames_per_trigger` distinct `frame_counter` values need to be
+   aggregated per RT point) or once per point? Check
+   `panda_box_free_running_setup.md`'s completion-check description
+   (`COUNTER1` "increments on the detector-trigger's falling edge") against
+   how many falling edges occur per point when `frames_per_trigger > 1`
+   before assuming either way.
+2. **Reduced data mode**: not yet defined in this doc — get the actual
+   definition/scope from Mirko at the start of that session rather than
+   assuming; don't extend this validation methodology to it until then.