diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/PANDA_POSITION_VALIDATION.md b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/PANDA_POSITION_VALIDATION.md index 2e6094e6..a223f483 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/PANDA_POSITION_VALIDATION.md +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/PANDA_POSITION_VALIDATION.md @@ -157,8 +157,11 @@ suspecting the conversion factors. ## 5. Practical notes -- Use `/opt/bec_deployments/production/bec_venv/bin/python3` for any - h5py-based analysis — the system `python3` doesn't have `h5py` installed. +- Use a BEC production venv `python3` for any h5py-based analysis — the + system `python3` doesn't have `h5py` installed. Path is host-dependent: + `/opt/bec_deployments/production/bec_venv/bin/python3` on some hosts, + `/sls/x12sa/config/bec/production/bec_venv/bin/python3` on others (e.g. + `x12sa-bec-001`) — check both if one is missing. - `S06313`'s files were at `/tmp/S06313/` (`S06313_master.h5` has both `omny_panda_continuous` and `rt_positions`; `S06313_eiger_*` are detector data, not needed for this). `/tmp` is not persistent — get a fresh scan's @@ -184,3 +187,85 @@ suspecting the conversion factors. 2. **Reduced data mode**: not yet defined in this doc — get the actual definition/scope from Mirko at the start of that session rather than assuming; don't extend this validation methodology to it until then. + +## 7. Post-fix confirmation against live scans (2026-09-22) + +Checked two scans taken after `3eb6e03` landed (both at `/tmp/S/` on +`x12sa-bec-001`, files from 2026-09-22 ~12:45–12:58): + +- **`S00386`** (174 points, ~348k raw rows): `frame_counter` starts at `0` on + the very first raw row and has zero backward steps across the whole raw + stream — no leading stale-value prefix at all, unlike pre-fix `S06313`. + Full grouping+conversion validation against `rt_positions`: R² = + 0.99999997 (X) / 0.99999998 (Y) after removing the expected constant + per-axis offset (−1.23 µm X, +1.08 µm Y, consistent with §4's `S06313` + values); residual std 0.30 nm (X) / 0.25 nm (Y); worst single-point + residual 1.02 nm (X) / 0.51 nm (Y) — no outlier anywhere near the ~116 nm + the pre-fix bug produced. Clean confirmation of the fix. +- **`S00388`** (785 points, ~1.4M raw rows): one backward step, at raw row + 36 (`frame_counter` reads `28` for rows 0–36, then drops to `0` from row + 37 on) — structurally the same leading-stale-prefix signature the fix is + meant to strip, still present in data captured after the fix. However, + every one of those 37 stale rows has `gate_detector_active == 0`, so + they'd never enter a point's average regardless (§3.2 already filters to + `gate_detector_active == 1` before grouping) — this scan happened not to + be exposed to the bug's actual failure mode. Full validation still comes + out clean: R² = 0.99999999 (X) / 0.99999998 (Y), residual std 0.27 nm (X) + / 0.26 nm (Y), max residual 0.85 nm (X) / 0.56 nm (Y). + +**Root-cause investigation (updated 2026-09-22, still not fully resolved):** + +Ruled out first: the leading `frame_counter` column can't be missing from +`S00388`'s first post-arm chunk the way an earlier version of this note +speculated. `pandablocks.connections.DataConnection`'s captured field set is +fixed once per TCP data connection (established once per `on_stage`, before +any `FrameData` is produced), so every `FrameData` chunk within one +acquisition carries the same columns — the "layout doesn't capture +`frame_counter`" fallback in `_drop_stale_frame_counter_rows` +(`counter_entry is None`) isn't reachable here. + +What actually happened, confirmed from `ScanServer.log`/`DeviceServer.log` +for 2026-09-22: the scan immediately preceding `S00388` — scan **#387** +(786 points, `eb7cfd11-3610-4511-94a0-75f49b403e62`) — was opened at +`12:50:10` and force-**aborted** by the user only ~7 seconds later +(`12:50:16`–`17`, `action: 'abort'` in the scan queue). At `exp_time=0.1`s/point +that's on the right order to land around point ~20–30, consistent with the +leftover `frame_counter == 28` that then showed up at the start of `#388`'s +raw stream (`#388` itself opened cleanly at `12:50:39`, no arm retries +logged — matches the "one `on_stage` per scan number" mapping confirmed +against the log). This is exactly the hardware mechanism `3eb6e03`'s commit +message describes (`COUNTER1.OUT` not cleared by `Arm()`) — just triggered +by an **abort mid-scan** rather than a normal scan-to-scan transition, which +is what `S06313` (the original bug report) and `S00386` (clean, §7 above) +both were. + +Why the client-side filter didn't strip it is not fully pinned down. +`PandaBoxOMNY.stop()` (inherited, not overridden) does not call +`_reset_raw_stream_state()` — only `on_stage()` (once per scan) and +`_try_arm()` (right before every `Arm()`) do, and both ran normally for +`#388` per the logs. So by the code's own logic the state should have been +freshly reset well before `#388`'s `Arm()` went out. One structural +suspect, not yet confirmed as the actual cause: `_raw_stream_buffer`/ +`_raw_stream_buffer_row_count`/`_raw_stream_last_flush` are all mutated only +under `_raw_stream_buffer_lock`, but `_raw_stream_seen_reset` is set outside +that lock in `_reset_raw_stream_state()` (after the `with` block) and read +outside the lock at the top of `_drop_stale_frame_counter_rows()` — the one +piece of this acquisition's reset state that isn't synchronized the same +way as everything else it's reset alongside. Whether that gap is actually +reachable in a plain abort → restage sequence (as opposed to needing an +`on_pre_scan` retry, which `#388` didn't have) isn't shown by the logs +available here; would need instrumentation (e.g. temporarily logging +`_raw_stream_seen_reset` transitions) against a reproduced abort-then-rescan +to confirm. + +Practically: harmless in both scans checked so far, since the stale rows' +`gate_detector_active` was `0` throughout — they never entered a point's +average regardless of the row-level filter. But an abort mid-scan is a +routine, user-triggered event (unlike an `on_pre_scan` retry), so this path +is worth closing rather than relying on the gate happening to be low during +the stale window. If revisited: consider (a) locking `_raw_stream_seen_reset` +reads/writes the same as the buffer fields, and (b) also calling +`_reset_raw_stream_state()` from `on_stop`/`stop()`, not only `on_stage`/ +`_try_arm`, so an aborted acquisition can't leave stale state for the next +one to inherit. Re-check against a scan where the stale run's rows show +`gate_detector_active == 1` before considering this fully closed. diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/flomni.py b/csaxs_bec/bec_ipython_client/plugins/flomni/flomni.py index 4f0f3c79..60205aed 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/flomni.py +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/flomni.py @@ -386,10 +386,10 @@ class FlomniSampleTransferMixin: def ensure_fheater_up(self): axis_id = dev.fheater._config["deviceConfig"].get("axis_Id") axis_id_numeric = self.axis_id_to_numeric(axis_id) - low, high = dev.fheater.controller.get_motor_limit_switch(axis_id) + _low, high = dev.fheater.controller.get_motor_limit_switch(axis_id) if high: raise FlomniError("fheater in high limit. How did we get here?? Aborting.") - if not low: + if not self._fheater_is_up(): self.ensure_osa_back() if dev.fheater.readback.get() < -0.2: umv(dev.fheater, -0.2) @@ -481,6 +481,8 @@ class FlomniSampleTransferMixin: self.feye_in() print("Moving X-ray optics out.") self.foptics_out() + if dev.fheater.user_parameter.get("enabled"): + self.move_fheater_down() self.xrayeye_update_frame() def laser_tracker_show_all(self): @@ -718,6 +720,9 @@ class FlomniSampleTransferMixin: umv(dev.fsamroy, 0) + self.feye_in() + self.foptics_out() + self.feedback_disable() self.ensure_fheater_up() @@ -737,7 +742,6 @@ class FlomniSampleTransferMixin: self.laser_tracker_off() time.sleep(0.05) - self.foptics_out() umv(dev.fsamroy, 90) diff --git a/csaxs_bec/bec_ipython_client/plugins/flomni/flomni_optics_mixin.py b/csaxs_bec/bec_ipython_client/plugins/flomni/flomni_optics_mixin.py index 5fabf040..2853a47d 100644 --- a/csaxs_bec/bec_ipython_client/plugins/flomni/flomni_optics_mixin.py +++ b/csaxs_bec/bec_ipython_client/plugins/flomni/flomni_optics_mixin.py @@ -23,37 +23,48 @@ class FlomniOpticsMixin: foptz_val = dev.foptz.readback.get() return -fosaz_target + (33 - foptz_val) + def _fheater_is_up(self) -> bool: + axis_id = dev.fheater._config["deviceConfig"].get("axis_Id") + low, _high = dev.fheater.controller.get_motor_limit_switch(axis_id) + return bool(low) + + def _fheater_is_down(self) -> bool: + fheater_in = dev.fheater.user_parameter.get("in") + if fheater_in is None: + return False + return np.isclose(dev.fheater.readback.get(), fheater_in, atol=0.1) + def feye_out(self): dev.fsh.fshclose() if self.csaxs is not None: self.csaxs.fil_trans(1, print_only=False) - sl1xs_before = self._sl1xs_before_eye - sl1ys_before = self._sl1ys_before_eye - if sl1xs_before is not None and sl1ys_before is not None: - # if the slit was manually re-adjusted while the eye was in, - # offer to keep the current size instead of restoring the - # pre-eye value - current_sl1xs = dev.sl1xs.user_readback.get() - current_sl1ys = dev.sl1ys.user_readback.get() - expected_sl1xs = sl1xs_before + 0.3 - expected_sl1ys = sl1ys_before + 0.3 - tol = 0.01 # mm (10 microns) - manually_adjusted = not np.isclose( - current_sl1xs, expected_sl1xs, atol=tol - ) or not np.isclose(current_sl1ys, expected_sl1ys, atol=tol) + # sl1xs_before = self._sl1xs_before_eye + # sl1ys_before = self._sl1ys_before_eye + # if sl1xs_before is not None and sl1ys_before is not None: + # # if the slit was manually re-adjusted while the eye was in, + # # offer to keep the current size instead of restoring the + # # pre-eye value + # current_sl1xs = dev.sl1xs.user_readback.get() + # current_sl1ys = dev.sl1ys.user_readback.get() + # expected_sl1xs = sl1xs_before + 0.3 + # expected_sl1ys = sl1ys_before + 0.3 + # tol = 0.01 # mm (10 microns) + # manually_adjusted = not np.isclose( + # current_sl1xs, expected_sl1xs, atol=tol + # ) or not np.isclose(current_sl1ys, expected_sl1ys, atol=tol) - if manually_adjusted and self.OMNYTools.yesno( - "sl1xs/sl1ys appear to have been adjusted manually while the " - "eye was in. Keep the current slit size instead of " - "restoring the pre-eye value?" - ): - print("Keeping current slit size, not restoring pre-eye value.") - else: - umv(dev.sl1xs, sl1xs_before, dev.sl1ys, sl1ys_before) - self._sl1xs_before_eye = None - self._sl1ys_before_eye = None + # if manually_adjusted and self.OMNYTools.yesno( + # "sl1xs/sl1ys appear to have been adjusted manually while the " + # "eye was in. Keep the current slit size instead of " + # "restoring the pre-eye value?" + # ): + # print("Keeping current slit size, not restoring pre-eye value.") + # else: + # umv(dev.sl1xs, sl1xs_before, dev.sl1ys, sl1ys_before) + # self._sl1xs_before_eye = None + # self._sl1ys_before_eye = None self.foptics_in() self.flomnigui_show_xeyealign() @@ -104,21 +115,22 @@ class FlomniOpticsMixin: current_feyex = dev.feyex.readback.get() current_feyey = dev.feyey.readback.get() - # check if both are close enough (within 0.01) - if np.isclose(current_feyex, feyex_in, atol=0.01) and np.isclose(current_feyey, feyey_in, atol=0.01): - # both already in position → do nothing - pass - else: - if "sl1xs" in dev and "sl1ys" in dev: - self._sl1xs_before_eye = dev.sl1xs.user_readback.get() - self._sl1ys_before_eye = dev.sl1ys.user_readback.get() - scans.umv(dev.sl1xs, 0.3, dev.sl1ys, 0.3, relative=True) + # # check if both are close enough (within 0.01) + # if np.isclose(current_feyex, feyex_in, atol=0.01) and np.isclose(current_feyey, feyey_in, atol=0.01): + # # both already in position → do nothing + # pass + # else: + # if "sl1xs" in dev and "sl1ys" in dev: + # self._sl1xs_before_eye = dev.sl1xs.user_readback.get() + # self._sl1ys_before_eye = dev.sl1ys.user_readback.get() + # scans.umv(dev.sl1xs, 0.3, dev.sl1ys, 0.3, relative=True) - if self.csaxs is not None: - self.csaxs.fil_trans(0.05, print_only=False) + # if self.csaxs is not None: + # self.csaxs.fil_trans(0.05, print_only=False) - # move both axes to the desired "in" positions - umv(dev.feyex, feyex_in, dev.feyey, feyey_in) + # # move both axes to the desired "in" positions + # umv(dev.feyex, feyex_in, dev.feyey, feyey_in) + umv(dev.feyex, feyex_in, dev.feyey, feyey_in) def _ffzp_in(self): foptx_in = self._get_user_param_safe("foptx", "in") @@ -206,13 +218,6 @@ class FlomniOpticsMixin: dev.rtx.controller.feedback_enable_with_reset() def fosa_in(self): - # TODO(commissioning): the OSA travels inside the heater's (fheater) - # envelope, but this function never checks fheater position before - # driving fosaz. fosaz must only move while fheater is fully "up" or - # fully "down" -- an intermediate heater position here risks a - # collision. Add a heater-state check (see ensure_osa_back()/ - # ensure_fheater_up() in flomni.py for the pattern) once fheater's - # up/down limits are commissioned. # 6.2 keV, 170 um FZP # umv(dev.losax, -1.4450000, dev.losay, -0.1800) # umv(dev.losaz, -1) @@ -244,6 +249,13 @@ class FlomniOpticsMixin: ) if need_move_osa: + if not (self._fheater_is_up() or self._fheater_is_down()): + raise FlomniOpticsError( + "Refusing to move OSA to its IN position: fheater is neither " + "at its 'up' (retracted) limit nor at its 'down' (heating) " + "position. An intermediate heater position risks a collision " + "with the OSA. Move the heater fully up or fully down first." + ) remaining = self._osa_remaining_space(fosaz_in) if remaining <= 0: foptz_val = dev.foptz.readback.get() @@ -307,7 +319,13 @@ class FlomniOpticsMixin: def fosa_out(self): - self.ensure_fheater_up() + if not (self._fheater_is_up() or self._fheater_is_down()): + raise FlomniOpticsError( + "Refusing to move OSA to its OUT position: fheater is neither " + "at its 'up' (retracted) limit nor at its 'down' (heating) " + "position. An intermediate heater position risks a collision " + "with the OSA. Move the heater fully up or fully down first." + ) curtain_is_triggered = dev.foptz.controller.fosaz_light_curtain_is_triggered() if not curtain_is_triggered: fosaz_out = self._get_user_param_safe("fosaz", "out") diff --git a/csaxs_bec/device_configs/ptycho_flomni.yaml b/csaxs_bec/device_configs/ptycho_flomni.yaml index 036b72a1..649e045c 100644 --- a/csaxs_bec/device_configs/ptycho_flomni.yaml +++ b/csaxs_bec/device_configs/ptycho_flomni.yaml @@ -46,13 +46,7 @@ feyey: in: -10.09 deviceTags: - ptycho_flomni -# TODO(commissioning): needs userParameter (e.g. in/up, down) -- move_fheater_down() -# in flomni.py currently reads user_parameter.get("in") and gets None. -# KNOWN GAP: fosa_in()/foptics_in() (flomni_optics_mixin.py) never check fheater -# position before driving fosaz. The OSA travels inside the heater's envelope, so -# fosaz must only move while fheater is fully at its "up" or fully at its "down" -# limit -- an intermediate heater position during fosa_in() risks a collision. -# Verify/add that check once real up/down limits are known from hardware. + fheater: description: Heater Y deviceClass: csaxs_bec.devices.omny.galil.fgalil_ophyd.FlomniGalilMotor @@ -69,6 +63,10 @@ fheater: readOnly: false readoutPriority: baseline connectionTimeout: 20 + userParameter: + #in: -0.5 + in: -11.5 + enabled: 1 deviceTags: - ptycho_flomni @@ -447,7 +445,7 @@ fosaz: #out: 6 #170 micron, 60 nm, 7.9 kev, foptz 16.9, probe size 7.5 mu in: 14.1 - out: 6 + out: 4 # micron, 30 nm, 7.9 kev, very close to the sample. make sure foptz is 32.02 or smaller //abe's fzp's # in: 0.5 # out: -5 @@ -703,30 +701,30 @@ calculated_signal: # deviceTags: # - ptycho_flomni -# omny_panda_continuous: -# readoutPriority: async -# deviceClass: csaxs_bec.devices.panda_box.panda_box_omny.PandaBoxOMNY -# deviceConfig: -# host: omny-panda.psi.ch -# raw_stream_mode: true -# signal_alias: -# FMC_IN.VAL1.Value: cap_voltage_fzp_y -# FMC_IN.VAL2.Value: cap_voltage_fzp_x -# INENC1.VAL.Value: interf_st_fzp_y -# INENC2.VAL.Value: interf_st_fzp_x -# INENC3.VAL.Value: interf_st_rotz -# INENC4.VAL.Value: interf_st_rotx -# PCAP.GATE_DURATION.Value: pcap_gate_duration_value -# # TTLIN1 (detector trigger) feeds both blocks below - confirmed 2026-09-15 against the -# # omny-panda hardware, see docs/developer/panda_box_free_running_setup.md. PCAP.BITS0 -# # is a shared 32-bit word; PandaBoxOMNY masks it to bit 0 (TTLIN1.VAL) before -# # publishing gate_detector_active. COUNTER1 counts TTLIN1's falling edge, i.e. one -# # increment per completed exposure; it also doubles as the on_complete signal. Confirmed -# # working end-to-end 2026-09-15; now under speed testing beyond the initial 100 Hz. -# PCAP.BITS0.Value: gate_detector_active -# COUNTER1.OUT.Value: frame_counter -# enabled: true -# readOnly: false -# softwareTrigger: false -# deviceTags: -# - ptycho_flomni +omny_panda_continuous: + readoutPriority: async + deviceClass: csaxs_bec.devices.panda_box.panda_box_omny.PandaBoxOMNY + deviceConfig: + host: omny-panda.psi.ch + raw_stream_mode: true + signal_alias: + FMC_IN.VAL1.Value: cap_voltage_fzp_y + FMC_IN.VAL2.Value: cap_voltage_fzp_x + INENC1.VAL.Value: interf_st_fzp_y + INENC2.VAL.Value: interf_st_fzp_x + INENC3.VAL.Value: interf_st_rotz + INENC4.VAL.Value: interf_st_rotx + PCAP.GATE_DURATION.Value: pcap_gate_duration_value + # TTLIN1 (detector trigger) feeds both blocks below - confirmed 2026-09-15 against the + # omny-panda hardware, see docs/developer/panda_box_free_running_setup.md. PCAP.BITS0 + # is a shared 32-bit word; PandaBoxOMNY masks it to bit 0 (TTLIN1.VAL) before + # publishing gate_detector_active. COUNTER1 counts TTLIN1's falling edge, i.e. one + # increment per completed exposure; it also doubles as the on_complete signal. Confirmed + # working end-to-end 2026-09-15; now under speed testing beyond the initial 100 Hz. + PCAP.BITS0.Value: gate_detector_active + COUNTER1.OUT.Value: frame_counter + enabled: true + readOnly: false + softwareTrigger: false + deviceTags: + - ptycho_flomni diff --git a/csaxs_bec/device_configs/simulated_omny/simulated_flomni.yaml b/csaxs_bec/device_configs/simulated_omny/simulated_flomni.yaml index 97dd2f98..a4eaa26e 100644 --- a/csaxs_bec/device_configs/simulated_omny/simulated_flomni.yaml +++ b/csaxs_bec/device_configs/simulated_omny/simulated_flomni.yaml @@ -104,6 +104,9 @@ fheater: readOnly: false readoutPriority: baseline connectionTimeout: 20 + userParameter: + in: -0.5 + enabled: 1 deviceTags: - simulated_flomni