GUI/DAQ: hutch-safety gating, error pop-ups, and automation pause-and-wait

Fix the GUI pop-up path and add personnel-safety-system (PSS) gating so
door-open / beam-down / shutter-closed conditions are surfaced and acted on.

- exception pop-ups: connect the previously-orphaned http_error signal;
  failed user operations now raise a modal dialog, background/polling errors
  a non-modal banner.
- PSS device (devices/pss_state.py) reading EH1-PSYS PROHIBITED-STATE /
  ALARM-STATE; new critical DoorSafetyError + DOOR_SAFETY_ERROR code.
- mounting service blocks mount/unmount when the hutch is not prohibited or
  an alarm is active; /status now publishes pss_prohibited / pss_alarm.
- GUI blocks manual mount/unmount and the automation Run button immediately
  (pop-up) on door-open, and shows a warning banner while an alarm is active.
- centralise per-action precondition checks (ring current, safety shutter,
  hutch door) into one combined "continue?" dialog with a session-global
  "don't ask again for 1 hour" snooze, applied to all data-collection buttons.
- live automation pauses and auto-resumes on bad conditions (beam, shutter,
  door, robot) with continue-now / stop overrides, gated by a default-on
  "Pause on bad conditions" checkbox replacing the dead CHECK_ENABLED constant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
appleb_m
2026-06-24 15:30:08 +02:00
co-authored by Claude Opus 4.8
parent a8855bfbba
commit 47e05dc28c
14 changed files with 523 additions and 46 deletions
+32
View File
@@ -1,4 +1,36 @@
# Changelog
## 0.3.2 (unreleased)
- **Authentication & Sessions**:
- Migrated DAQ server URLs to HTTPS with per-beamline configurable certificate paths (cacert) and mTLS; GUI now validates certificates and handles self-signed certs / SSL errors in `DAQWorker` and `SSEClient`.
- Reworked authentication around Kerberos/SPNEGO token negotiation (via `curl`) behind an Apache/HTTPS proxy; removed the direct `gssapi` dependency and simplified proxy-header handling.
- GUI session management: track active GUIs in Redis, show the current baton holder, let staff force-close erroneous sessions, and auto-close idle GUIs after a grace period.
- **Operations refactor (major)**:
- Standardised every multi-step procedure under `aare/daq/operations/<name>/` as a `*Service` with constructor-injected context (`deps`, `runtime` providers, `services` protocols, `settings`); added the `rotation` operation and `screenshot`/common helpers.
- Major refactor of the mounting service, plus loop-centering, face-detection and raster services, with runtime-state providers/protocols and updated unit tests.
- **Exception model (major refactor)**:
- Reworked the exception hierarchy into `AareException` with a load-bearing `critical` flag, routing roots (`AutomationError`/`AareUserError`/`AareAuthError`), empty device-family bases for `isinstance` grouping, and centralised `AareErrorCode`s served through FastAPI handlers; the GUI reads `body.critical` to pause/halt automation.
- Added `StateTransitionFailed`, `MaintenanceStateException`, `BECCommunicationError`, `BeamlineBusyTimeoutException`, `AutoRasterSampleSkipped` and improved `TransformationInvalidException` handling; moved excessive logging out of the exception classes themselves.
- **DAQ/Server**:
- Converted the in-use EPICS devices to BEC (ring current, energy, front/back light, zoom, transmission, fast shutter, detector Z), and cached BEC device handles as variables for a large speed-up (~6 µs vs ~200 ms per read).
- Store detector metadata and TELL events in Redis so `/status` polling stays fast; moved detector-Z soft-limit checks into the device with a `detector_limit_modifier` to avoid hitting limits.
- Added a TELL state handler that derives activity/phase from the TELL SSE stream and surfaces the current robot action in the GUI BUSY label, status bar and recovery panel.
- Critical detector faults now halt automation (and surface during manual collection); added blower-on for every mount, anneal in/out, `change_energy` and `mono_pitch_scan` controls.
- Scaling/skip handling for over-large raster grids (Jungfrau buffer limit), with a longer busy timeout and busy-timeout exceptions.
- **GUI**:
- Added portrait mode, a compact automation view with mode-switching buttons, a Redis-backed automation progress panel, a beamline-state panel, a shared busy-overlay widget, and a splash screen / app banner / icon.
- Added a local-contact panel that restarts backend device connections (JFJoch, TELL, Aerotech, Smargon), reinitialises BEC macros/devices and opens beamline web helpers.
- Performance: raster grid manager now renders a single bitmap to remove lag on large grids; added a Database/User-values toggle to data-collection settings.
- **Common, Devices & Config**:
- Moved all hardware URLs (GUI, JFJoch, Aerotech, BEC, Smargon, TELL) into per-beamline YAML read via `cfg_get`; added `lens_magnification`, `maximum_flux`, `detector_distance_minimum`, `detector_limit_modifier`, beamline-specific transmission PVs and cryojet temperature PVs; removed unused EPICS devices.
- **Workflows**:
- Added `maintenance2sa`/`maintenance2se` transitions, a safe detector-Z position check, explicit per-transition error messages, and switched to `ABR_POS`/`ABR_OMEGA` mount positions; removed the unused `ws` mode.
- **Hutch safety & error pop-ups** (in progress):
- Re-wired the GUI pop-up path: failed user operations (mount/unmount) now raise a modal dialog (e.g. *Mounting Failed: No Pin in Gripper*) and background/polling errors a non-modal banner — previously the `http_error` signal was emitted but never connected, so these were silently dropped.
- Added a `PssState` device reading the hutch personnel-safety system (`…-EH1-PSYS:PROHIBITED-STATE` / `:ALARM-STATE`); mounts are blocked (critical `DoorSafetyError`, automation queue paused) when the hutch is not in the prohibited state or a safety alarm is active, and an active alarm also raises a non-modal warning banner via `/status`.
- `/status` now publishes `pss_prohibited`/`pss_alarm`; the GUI blocks manual mount/unmount and the automation Run button immediately (with a pop-up) when the hutch door is open, instead of letting the robot fail to move server-side.
- Centralised the per-action precondition checks (ring current, safety shutter, hutch door) into one combined "continue?" dialog used by all data-collection buttons (Evaluate grid, X-ray centering, Run screening/rotation/fluorescence, simple rotation), with a session-global "Don't ask me again for 1 hour" snooze.
- Live automation now pauses and auto-resumes when the beam, experiment shutter, hutch door or robot (TELL) are not ready (with Continue-now / Stop overrides), gated by a "Pause on bad conditions" checkbox (default on) that replaces the dead `CHECK_ENABLED` constant.
## 0.3.1 (2026-04-29)
- **DAQ/Server**:
- Improved baton exchange logic with automatic timeout and Redis-backed state.
+1
View File
@@ -77,6 +77,7 @@ class AareErrorCode(StrEnum):
TRANSFORMATION_INVALID_EXCEPTION = "TRANSFORMATION_INVALID_EXCEPTION"
MAGNET_POSITION_SENSOR_ERORR = "MAGNET_POSITION_SENSOR_ERORR" # NOTE: class name "Erorr" has a typo; preserved
SMART_MAGNET_FAULT_EXCEPTION = "SMART_MAGNET_FAULT_EXCEPTION"
DOOR_SAFETY_ERROR = "DOOR_SAFETY_ERROR"
# User errors
MANUAL_MOUNT_EXCEPTION = "MANUAL_MOUNT_EXCEPTION"
+19
View File
@@ -185,6 +185,25 @@ class SmartMagnetFaultException(AutomationError):
return self.message
class DoorSafetyError(AutomationError):
"""Hutch personnel-safety system does not permit robot motion.
Raised before a mount/unmount when ``…-EH1-PSYS:PROHIBITED-STATE`` is not in
the prohibited state (door safety could not be activated, so the robot will
not move) or when ``…-EH1-PSYS:ALARM-STATE`` reports an active alarm. Always
critical so automation halts and the GUI shows a pop-up.
"""
critical: ClassVar[bool] = True
def __init__(self, message: str = "Door safety could not be activated", *, critical: bool | None = None):
super().__init__(message, critical=critical)
self.message = message
def __str__(self) -> str:
return self.message
class TellCommandWhileBusyException(TellException):
"""Custom exception for trying to move Tell when it is busy"""
+7
View File
@@ -553,6 +553,13 @@ class BeamlineStatus(BaseModel):
commissioning_mode: bool
dtz_min: float
dtz_max: float
# Hutch personnel-safety system state. ``pss_prohibited`` is True when the
# hutch is interlocked so the robot may move (PROHIBITED-STATE); the GUI
# blocks a mount when it is False. ``pss_alarm`` is True when ALARM-STATE
# != 0 (warning). Defaults keep older payloads/constructors valid and avoid
# the GUI false-blocking when an old server omits the field.
pss_prohibited: bool = True
pss_alarm: bool = False
class SessionStatus(BaseModel):
+4
View File
@@ -3185,6 +3185,8 @@ class AareDAQ:
cryojet_temp = self.__devs.cryojet_temp
shutter_open = self.__devs.shutter
exp_shutter_open = self.__devs.exp_shutter.state()
pss_prohibited = self.__devs.pss.is_prohibited()
pss_alarm = self.__devs.pss.alarm_active()
flux = self.__devs.full_flux
samcam_settings = self.__devs.samcam_settings
bl = self.__bl
@@ -3223,6 +3225,8 @@ class AareDAQ:
commissioning_mode=commisioning_mode,
dtz_min=dtz_min,
dtz_max=dtz_max,
pss_prohibited=pss_prohibited,
pss_alarm=pss_alarm,
)
except Exception as e:
+3
View File
@@ -19,6 +19,7 @@ from aare.devices.area_detector import epicsAD, AutoEnum
from aare.devices.enum_pv import EnumPV
from aare.devices.experimental_hutch_shutter import ExperimentalHutchShutter
from aare.devices.my_motor import MyMotor
from aare.devices.pss_state import PssState
from aare.devices.set_get_pv import SetGetPV, PredefinedPV
from aare.devices.tell_client import make_tell_client
@@ -39,6 +40,8 @@ class BeamlineDevices:
logger.debug("initialising BEC worker done")
self.__smargon = smargon.Smargon(beamline)
self.exp_shutter = ExperimentalHutchShutter(beamline=self._beamline)
# Personnel Safety System: gates whether the robot is allowed to move.
self.pss = PssState(beamline=self._beamline)
#faster to define the dtz object here than in functions and then use
self.__dtz = self.bec_worker.dev.det_z
@@ -2,6 +2,7 @@ import time
from aare.common.exception_handler import (
CriticalTellException,
DoorSafetyError,
MountingFailed,
TellCommunicationError,
UnmountingFailed,
@@ -114,7 +115,29 @@ class MountingService:
self.logger.error(message)
raise CriticalTellException(f"Critical error in TELL mount: {message}")
def _check_hutch_safety(self) -> None:
"""Block robot motion unless the hutch PSS permits it.
The robot only moves when the hutch is in the *prohibited* state; if it
is not, the door safety could not be activated. A non-zero alarm state
is also blocked. Always raised critical so automation halts and the GUI
shows a pop-up.
"""
pss = self.ctx.deps.devs.pss
if not pss.is_prohibited():
self.logger.error(
"Hutch is not in prohibited state; robot motion not permitted"
)
raise DoorSafetyError(
"Door safety could not be activated: close the hutch doors and "
"complete the safety search before mounting."
)
if pss.alarm_active():
self.logger.error("Hutch safety alarm is active; blocking robot motion")
raise DoorSafetyError("Hutch safety alarm is active. Mounting blocked.")
def _prepare_mount_hardware(self) -> None:
self._check_hutch_safety()
self.ctx.deps.devs.smargon_move_home()
self.ctx.deps.devs.aerotech_pos = self.ctx.settings.mount_position
self._magnet_position_sensor_check(timeout=360.0)
@@ -161,6 +184,8 @@ class MountingService:
previous_sample=unmounted_sample,
did_unmount_previous=unmounted_sample is not None,
)
except DoorSafetyError:
raise
except Exception as e:
raise UnmountingFailed(f"Failed to unmount: {e}") from e
+67
View File
@@ -0,0 +1,67 @@
"""Personnel Safety System (PSS) state for the experimental hutch.
The sample-changer robot may only move when the hutch is in the *prohibited*
state (doors closed/searched, no personnel access). When
``…-EH1-PSYS:PROHIBITED-STATE`` reads ``0`` the robot will not move, so a
mount/unmount must be blocked. A non-zero ``…-EH1-PSYS:ALARM-STATE`` signals a
safety alarm.
This mirrors the simple PV-wrapper pattern used by
:class:`aare.devices.experimental_hutch_shutter.ExperimentalHutchShutter`
(same ``EH1-PSYS`` PV family). In ``SIMULATED`` the device reports a safe state
(prohibited, no alarm) so the simulated mount path runs without hardware.
"""
from aare.common.beamline import MXBeamline
from aare.common.logger_config import setup_logger
from epics import PV
logger = setup_logger("aareDAQ")
class PssState:
def __init__(self, beamline: MXBeamline):
self._simulated = beamline == MXBeamline.SIMULATED
BEAMLINE = beamline.value.upper()
if self._simulated:
self.__prohibited = None
self.__alarm = None
else:
self.__prohibited = PV(f"{BEAMLINE}-EH1-PSYS:PROHIBITED-STATE")
self.__alarm = PV(f"{BEAMLINE}-EH1-PSYS:ALARM-STATE")
def is_prohibited(self) -> bool:
"""True when the hutch is interlocked so the robot is allowed to move.
``PROHIBITED-STATE`` reports ``"PROHIBITED"`` (robot may move) or ``0``
(robot will not move). A disconnected/unknown PV is treated as *not*
prohibited, so the mount is blocked fail-safe.
"""
if self._simulated:
return True
value = self.__prohibited.get()
if value is None:
logger.warning(
"PROHIBITED-STATE PV returned no value; treating hutch as not prohibited"
)
return False
if isinstance(value, str):
prohibited = value.strip().upper() == "PROHIBITED"
else:
prohibited = value != 0
if not prohibited:
# Log the raw value when we block, so the PV semantics can be
# confirmed against the hardware without spamming the happy path.
logger.info(f"Hutch not in prohibited state; PROHIBITED-STATE raw value={value!r}")
return prohibited
def alarm_active(self) -> bool:
"""True when ``ALARM-STATE`` is non-zero (a safety alarm is active)."""
if self._simulated:
return False
value = self.__alarm.get()
if value is None:
return False
if isinstance(value, str):
return value.strip().upper() not in ("", "0", "OK", "NONE", "NO ALARM")
return value != 0
+141 -6
View File
@@ -78,6 +78,7 @@ from aare.gui.widgets.no_wheel_scroll_area import NoWheelScrollArea
from aare.gui.widgets.status_bar import StatusBar
from aare.gui.widgets.video_image import VideoGraphicsView
from aare.gui.widgets.busy_overlay import build_busy_overlay_style
from aare.gui.widgets.message_box import precondition_check
logger = setup_logger("aareGUI")
@@ -685,10 +686,12 @@ class MainWindow(QMainWindow):
self.job_list_panel.unmount.connect(self.daq.unmount)
self.job_list_panel.park_and_dry.connect(self.daq.park_and_dry)
self.tell_samples.mount.connect(self.daq.mount)
self.tell_samples.unmount.connect(self.daq.unmount)
self.ref_tools_panel.mount.connect(self.daq.mount)
self.ref_tools_panel.unmount.connect(self.daq.unmount)
# Gate manual mounts/unmounts on the hutch PSS state so the user gets an
# immediate pop-up instead of the robot failing to move server-side.
self.tell_samples.mount.connect(self._on_manual_mount_requested)
self.tell_samples.unmount.connect(self._on_manual_unmount_requested)
self.ref_tools_panel.mount.connect(self._on_manual_mount_requested)
self.ref_tools_panel.unmount.connect(self._on_manual_unmount_requested)
self.data_collection.raster.grid_size_updated.connect(self.raster.update_grid_size)
self.data_collection.raster.exp_time_updated.connect(self.raster.update_exposure_time)
@@ -701,7 +704,7 @@ class MainWindow(QMainWindow):
self.raster.grid_scan.connect(self.daq.raster_scan)
self.raster.grid_scan_auto.connect(self.daq.raster_scan_auto)
self.data_collection.screening.rotation_scan.connect(self.daq.standard_scan)
self.data_collection.simple.rotation_scan.connect(self.daq.standard_scan)
self.data_collection.simple.rotation_scan.connect(self._on_simple_rotation_requested)
self.data_collection.simple.parameters_changed.connect(self.daq.smart_params)
self.raster.grid_scan_size_changed.connect(self.data_collection.raster.grid_scan_size_change)
@@ -780,6 +783,11 @@ class MainWindow(QMainWindow):
self.daq.update.connect(self.update_daq_status)
self.daq.sample_missing.connect(self.show_sample_missing_dialog)
# Generic/background HTTP errors -> non-modal runtime alert; failed
# user operations (mount/unmount/...) -> modal pop-up.
self.daq.http_error.connect(self._on_http_error)
self.daq.operation_failed.connect(self._on_operation_failed)
self.daq.pss_alarm_changed.connect(self._on_pss_alarm_changed)
self.daq.standard_scan_completed.connect(self.rotation.scan_completed)
self.daq.raster_scan_completed.connect(self.raster.grid_scan_completed)
@@ -796,7 +804,7 @@ class MainWindow(QMainWindow):
self.status_bar.get_all_pgroups.connect(self.daq.get_all_pgroups)
self.daq.staff_pgroups_loaded.connect(self.status_bar.staff_pgroups_loaded)
self.data_collection.fluo.fluo_scan.connect(self.daq.fluorimeter_spectrum)
self.data_collection.fluo.fluo_scan.connect(self._on_fluo_scan_requested)
self.daq.fluorimeter_spectrum_update.connect(self.fluor_panel.update_plot)
self.daq.fluorimeter_spectrum_update.connect(lambda: self.fluor_panel_dock.setVisible(True))
@@ -1632,6 +1640,133 @@ class MainWindow(QMainWindow):
)
@Slot(str)
@Slot(str)
def _on_http_error(self, message: str) -> None:
"""Surface generic/background HTTP errors as a non-modal runtime alert.
These come from polling, resync and other background requests; a modal
pop-up would be too intrusive, so they go to the runtime dock/banner.
"""
self._portrait_alert_primary(message, True)
@Slot(bool)
def _on_pss_alarm_changed(self, active: bool) -> None:
"""Non-modal warning banner for the hutch PSS alarm (ALARM-STATE != 0).
Edge-triggered from /status: shows a sticky warning while active and a
brief confirmation when it clears.
"""
if active:
self._show_runtime_notification(
title="Hutch safety alarm",
message=(
"The hutch personnel-safety system reports an active alarm "
"(ALARM-STATE != 0). Mounting is blocked until it clears — "
"check the hutch / call your local contact."
),
level="warning",
sticky=True,
)
else:
self._show_runtime_notification(
title="Hutch safety alarm cleared",
message="The hutch safety alarm has cleared.",
level="success",
sticky=False,
auto_clear_ms=4000,
)
@Slot(str, str, bool)
def _on_operation_failed(self, title: str, message: str, critical: bool) -> None:
"""Modal pop-up for a failed user-triggered operation (mount, unmount, ...).
Critical failures (e.g. door safety could not be activated) also pause
the automation queue defensively, so we never keep dispatching samples.
"""
logger.error(f"Operation failed [{title}] (critical={critical}): {message}")
if critical:
try:
if self.job_list_panel is not None and self.job_list_panel.is_running():
self.job_list_panel.pause_automation()
except Exception as e:
logger.error(f"Failed to pause automation after operation failure: {e}")
try:
show = QMessageBox.critical if critical else QMessageBox.warning
show(self, title, message)
except Exception as e:
logger.error(f"Failed to show operation failure popup: {e}")
def _hutch_blocks_mount(self) -> str | None:
"""Reason the hutch PSS currently blocks a mount, or None if OK.
Uses the latest ``/status``; if no status has arrived yet we defer to
the authoritative server-side gate rather than guess.
"""
bl = getattr(self._latest_daq_status, "bl", None)
if bl is None:
return None
if getattr(bl, "pss_prohibited", True) is False:
return (
"Door safety could not be activated: close the hutch doors and "
"complete the safety search before mounting."
)
if getattr(bl, "pss_alarm", False):
return "The hutch safety alarm is active. Mounting is blocked until it clears."
return None
def _on_manual_mount_requested(self, sample, reference: bool = False) -> None:
"""Pre-check the hutch before sending a manual mount to the server.
Gives the user an immediate pop-up if the door is open / alarm active,
instead of the robot silently failing to move server-side.
"""
reason = self._hutch_blocks_mount()
if reason is not None:
logger.warning(f"Manual mount blocked by hutch PSS: {reason}")
try:
QMessageBox.critical(self, "Mounting Failed", reason)
except Exception as e:
logger.error(f"Failed to show mount-blocked popup: {e}")
return
self.daq.mount(sample, reference)
def _on_manual_unmount_requested(self) -> None:
"""Block a manual unmount if the hutch isn't ready (robot can't move)."""
reason = self._hutch_blocks_mount()
if reason is not None:
logger.warning(f"Manual unmount blocked by hutch PSS: {reason}")
try:
QMessageBox.critical(self, "Unmounting Failed", reason)
except Exception as e:
logger.error(f"Failed to show unmount-blocked popup: {e}")
return
self.daq.unmount()
def _precondition_ok(self) -> bool:
"""Run the shared ring-current / shutter / door 'continue?' check.
Uses the latest ``/status``; defers (returns True) if no status yet.
"""
bl = getattr(self._latest_daq_status, "bl", None)
if bl is None:
return True
return precondition_check(
self,
ring_current=getattr(bl, "ring_current_mA", None),
shutter_open=getattr(bl, "exp_shutter_open", None),
door_prohibited=getattr(bl, "pss_prohibited", None),
)
def _on_fluo_scan_requested(self, params) -> None:
if self._precondition_ok():
self.daq.fluorimeter_spectrum(params)
def _on_simple_rotation_requested(self, request) -> None:
if self._precondition_ok():
self.daq.standard_scan(request)
def _on_manual_collection_critical_failure(self, message: str) -> None:
logger.critical(f"Manual collection critical detector failure: {message}")
+51 -28
View File
@@ -5,8 +5,7 @@ from PySide6.QtGui import QKeySequence, QShortcut
from aare.common.models import SampleShortInfoList, SampleShortInfo, BeamlineStateEnum, SessionsStateEnum
from aare.gui.models.sample_queue_model import SampleQueueSpreadsheet
from aare.gui.widgets.message_box import ring_current_low_check, experiment_hutch_shutter_check, LOW_CURRENT_THRESHOLD, \
ring_current_auto_check
from aare.gui.widgets.message_box import LOW_CURRENT_THRESHOLD, conditions_auto_check
from aare.gui.widgets.title_label import TitleLabel
from aare.common.models import DAQStatusModel
@@ -14,8 +13,6 @@ from aare.common.logger_config import setup_logger
logger = setup_logger("aareGUI")
CHECK_ENABLED = False
class SampleQueuePanel(QFrame):
auto_scan = Signal(SampleShortInfo)
unmount = Signal()
@@ -37,6 +34,8 @@ class SampleQueuePanel(QFrame):
self._step_through = False
self.ring_current = None
self._experiment_shutter_state = None
self._door_prohibited = None
self._tell_connected = True
self.loop_restart_requested = None
self.__recovery_timer = QTimer(self)
@@ -86,10 +85,19 @@ class SampleQueuePanel(QFrame):
self.park_and_dry_when_cleared = QCheckBox("Park and dry when automation finishes", self)
self.park_and_dry_when_cleared.setChecked(True)
self.pause_on_conditions_cb = QCheckBox("Pause on bad conditions (beam/shutter/door/robot)", self)
self.pause_on_conditions_cb.setChecked(True)
self.pause_on_conditions_cb.setToolTip(
"When on, automation refuses to start and pauses between samples if the beam, "
"experiment shutter, hutch doors or robot are not ready, auto-resuming when they "
"recover. Uncheck for testing to run straight through."
)
button_layout.addWidget(self.play_button)
button_layout.addWidget(self.remove_button)
button_layout.addWidget(self.clear_button)
button_layout.addWidget(self.park_and_dry_when_cleared)
button_layout.addWidget(self.pause_on_conditions_cb)
layout.addLayout(button_layout)
self._emit_samples_in_queue_changed()
@@ -179,15 +187,25 @@ class SampleQueuePanel(QFrame):
self._emit_samples_in_queue_changed()
def ring_current_check(self):
if not ring_current_low_check(self, self.ring_current):
self.table_model.set_running(False)
self.__set_to_pause = True
self.__pause = True
self.play_button.setText("▶ Run")
self._emit_samples_in_queue_changed()
return False
return True
def _checks_enabled(self) -> bool:
return self.pause_on_conditions_cb.isChecked()
def _bad_conditions(self) -> list[str]:
"""Beamline conditions that currently block/should pause automation."""
problems: list[str] = []
if self.ring_current is None or self.ring_current < LOW_CURRENT_THRESHOLD:
rc = "unknown" if self.ring_current is None else f"{round(self.ring_current, 2)} mA"
problems.append(f"beam (ring current {rc})")
if not self._experiment_shutter_state:
problems.append("experiment shutter closed")
if self._door_prohibited is False:
problems.append("hutch doors open / not searched")
if not self._tell_connected:
problems.append("robot (TELL) not connected")
return problems
def _conditions_ok(self) -> bool:
return len(self._bad_conditions()) == 0
def resume_automation(self):
if self.__warning_msg_box:
@@ -250,7 +268,8 @@ class SampleQueuePanel(QFrame):
def run(self):
self.__recovery_timer.stop()
if CHECK_ENABLED:
checks_enabled = self._checks_enabled()
if checks_enabled:
if self.__warning_msg_box:
self.__warning_msg_box.done(0)
self.__warning_msg_box = None
@@ -303,14 +322,17 @@ class SampleQueuePanel(QFrame):
return
if len(self.table_model.samples) > 0:
if CHECK_ENABLED:
if not self.ring_current_check():
logger.debug("low ring current, skipping")
if checks_enabled:
bad = self._bad_conditions()
if bad:
logger.warning(f"Cannot start automation; beamline not ready: {bad}")
self.show_error_dialog(
title="Beamline not ready",
msg="Cannot start automation:\n- " + "\n- ".join(bad),
info="Fix the above, or untick 'Pause on bad conditions' for testing.",
)
return
elif not experiment_hutch_shutter_check(parent=self, shutter_state=self._experiment_shutter_state):
logger.debug("experiment shutter closed, user chose to skip")
self.table_model.set_running(True)
self.__set_to_pause = False
self.__pause = False
@@ -346,6 +368,8 @@ class SampleQueuePanel(QFrame):
def update_daq_status(self, s: DAQStatusModel):
self.ring_current = s.bl.ring_current_mA
self._experiment_shutter_state = s.bl.exp_shutter_open
self._door_prohibited = getattr(s.bl, "pss_prohibited", None)
self._tell_connected = getattr(s, "tell_connected", True)
self.__beamline_state = s.state
self.__busy = s.busy
self.__baton_holder = s.session.session
@@ -364,21 +388,20 @@ class SampleQueuePanel(QFrame):
if self._current_db_id is not None and db_id != self._current_db_id:
return
if CHECK_ENABLED and (self.ring_current is None or (self.ring_current < LOW_CURRENT_THRESHOLD)):
if self._checks_enabled() and not self._conditions_ok():
self.pause_automation(set_id_to_None=False)
bad = self._bad_conditions()
logger.warning(f"Automation paused; beamline not ready: {bad}")
msg = "Automation paused — beamline not ready:\n- " + "\n- ".join(bad)
def current_ok() -> bool:
return (self.ring_current is not None) and (self.ring_current >= LOW_CURRENT_THRESHOLD)
if ring_current_auto_check(self, self.ring_current, current_ok):
logger.debug("Ring current recovered or user chose to continue,"
"resuming automation")
if conditions_auto_check(self, msg, self._conditions_ok):
logger.debug("Conditions recovered or user chose to continue; resuming automation")
self.table_model.set_running(True)
self.__pause = False
self.play_button.setText("⏸ Pause")
self._emit_samples_in_queue_changed()
else:
logger.debug("No beam, or user quit, ending automation")
logger.debug("Conditions not met or user stopped; ending automation")
self.unmount.emit()
self._emit_samples_in_queue_changed()
return
+10 -6
View File
@@ -13,7 +13,7 @@ from PySide6.QtWidgets import (
from aare.common.diffraction_geometry import DiffractionGeometry
from aare.common.logger_config import setup_logger
from aare.common.models import DAQStatusModel, SessionsStateEnum
from aare.gui.widgets.message_box import ring_current_low_check, experiment_hutch_shutter_check
from aare.gui.widgets.message_box import precondition_check
from aare.gui.widgets.number_line_edit import DbOverrideLineEdit
logger = setup_logger("aareGUI")
@@ -46,6 +46,7 @@ class ScanSettingsPanel(QWidget):
self._target_dose = None
self._ring_current = None
self._experiment_shutter_state = None
self._door_prohibited = None
# Fields whose value can come from the database or be overridden by the
# user; the panel-level toggle drives them all together.
@@ -163,6 +164,7 @@ class ScanSettingsPanel(QWidget):
self.__diffraction = s.diffraction
self._ring_current = s.bl.ring_current_mA
self._experiment_shutter_state = s.bl.exp_shutter_open
self._door_prohibited = getattr(s.bl, "pss_prohibited", None)
can_edit = (not s.busy) and (s.session.session in (SessionsStateEnum.OwnedByYou, SessionsStateEnum.PendingElseToYou))
self._can_edit_params = can_edit
# Lock/unlock the override fields
@@ -320,11 +322,13 @@ class ScanSettingsPanel(QWidget):
return []
def check_before_run (self, scan_kind: str):
if not ring_current_low_check(self, self._ring_current):
logger.warning("Low ring current, User chose to not to continue scan")
return False
if not experiment_hutch_shutter_check(self, self._experiment_shutter_state):
logger.warning("experiment shutter closed, user chose to not to continue scan")
if not precondition_check(
self,
ring_current=self._ring_current,
shutter_open=self._experiment_shutter_state,
door_prohibited=self._door_prohibited,
):
logger.warning("Beamline not ready; user chose not to continue scan")
return False
p = self.parent()
if hasattr(p, "file_path_panel"):
+30 -1
View File
@@ -1,6 +1,7 @@
import copy
import os
import random
import re
import time
import json
import logging
@@ -82,6 +83,12 @@ class DAQWorker(QObject):
detector_error = Signal(str, bool) # (message, is_error)
auth_error = Signal()
sample_missing = Signal(str)
# Failure of a user-triggered operation POST (mount, unmount, ...). Carries
# (title, message, critical) and is surfaced as a modal pop-up by the GUI.
operation_failed = Signal(str, str, bool)
# Hutch PSS alarm (ALARM-STATE != 0) became active/inactive. Edge-triggered
# from /status; surfaced as a non-modal warning banner.
pss_alarm_changed = Signal(bool)
automated_scan_done = Signal(int, bool, str) # sample ID, success
run_number_incremented = Signal()
@@ -157,6 +164,9 @@ class DAQWorker(QObject):
self._last_smargon_error: str | None = None
self._last_aerotech_connected: bool | None = None
self._last_aerotech_error: str | None = None
# Edge-trigger the hutch PSS alarm banner; start False so a clear hutch
# at startup doesn't emit a spurious "cleared" notification.
self._last_pss_alarm: bool = False
self._server_connected: bool | None = None
self._last_server_error: str | None = None
@@ -434,6 +444,11 @@ class DAQWorker(QObject):
parsed_response = DAQStatusModel.model_validate_json(response_data)
self.update.emit(parsed_response)
pss_alarm = bool(getattr(getattr(parsed_response, "bl", None), "pss_alarm", False))
if pss_alarm != self._last_pss_alarm:
self._last_pss_alarm = pss_alarm
self.pss_alarm_changed.emit(pss_alarm)
self._last_status_can_read = True
self._restart_blocked_sse_streams_if_access_restored()
@@ -621,10 +636,24 @@ class DAQWorker(QObject):
self.sample_missing.emit(error_info.message)
else:
logger.error(f"{error_info.message}")
self.http_error.emit(error_info.message)
title = self._operation_error_title(error_info.exception_class)
self.operation_failed.emit(title, error_info.message, error_info.critical)
reply.deleteLater()
@staticmethod
def _operation_error_title(exception_class: str | None) -> str:
"""Human-friendly dialog title from an exception class name.
``MountingFailed`` -> ``"Mounting Failed"``; falls back to
``"Operation Failed"`` when the class is unknown.
"""
if not exception_class:
return "Operation Failed"
spaced = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", exception_class)
spaced = spaced.replace("Exception", "Error").strip()
return spaced or "Operation Failed"
def _handle_sample_resync_response(self, reply: QNetworkReply):
try:
response_data = self.handle_response(reply)
+83 -2
View File
@@ -1,10 +1,80 @@
import time
from PySide6.QtCore import QTimer, QEventLoop
from PySide6.QtWidgets import QMessageBox
from PySide6.QtWidgets import QMessageBox, QCheckBox
from aare.common.logger_config import setup_logger
logger = setup_logger("aareGUI")
LOW_CURRENT_THRESHOLD = 100.0
SNOOZE_SECONDS = 3600.0 # "Don't ask again for 1 hour"
class _PreconditionSnooze:
"""Session-global 'don't ask me again for 1 hour' for precondition prompts.
In-memory only, so it auto-clears when the GUI restarts.
"""
def __init__(self):
self._until = 0.0
def active(self) -> bool:
return time.monotonic() < self._until
def snooze(self) -> None:
self._until = time.monotonic() + SNOOZE_SECONDS
logger.info("Precondition prompts snoozed for 1 hour")
def clear(self) -> None:
self._until = 0.0
# Shared across all panels for the session.
precondition_snooze = _PreconditionSnooze()
def precondition_problems(ring_current, shutter_open, door_prohibited) -> list[str]:
"""List the beamline conditions that are currently not satisfied."""
problems: list[str] = []
if ring_current is None:
problems.append("Ring current is unknown.")
elif ring_current < LOW_CURRENT_THRESHOLD:
problems.append(f"Ring current is low: {round(ring_current, 2)} mA.")
if not shutter_open:
problems.append("Experiment safety shutter is closed.")
if door_prohibited is False:
problems.append("Hutch is not in the prohibited state (doors open / not searched).")
return problems
def precondition_check(parent, *, ring_current, shutter_open, door_prohibited) -> bool:
"""Combined ring current / safety shutter / hutch door pre-action check.
Shows ONE 'Do you wish to continue?' dialog listing every failing
condition, with a 'Don't ask me again for 1 hour' checkbox. Returns True to
proceed. Honours the session-global snooze and never blocks outright.
"""
if precondition_snooze.active():
return True
problems = precondition_problems(ring_current, shutter_open, door_prohibited)
if not problems:
return True
box = QMessageBox(parent)
box.setIcon(QMessageBox.Icon.Warning)
box.setWindowTitle("Beamline not ready")
box.setText("\n".join(problems) + "\n\nDo you wish to continue?")
box.setStandardButtons(QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No)
box.setDefaultButton(QMessageBox.StandardButton.No)
snooze_cb = QCheckBox("Don't ask me again for 1 hour")
box.setCheckBox(snooze_cb)
proceed = box.exec() == QMessageBox.StandardButton.Yes
if proceed and snooze_cb.isChecked():
precondition_snooze.snooze()
return proceed
def reply_box(parent, title: str = "Warning", msg: str = "Warning." ):
return QMessageBox.question(
@@ -82,7 +152,18 @@ def experiment_hutch_shutter_check(parent, shutter_state) -> bool:
def ring_current_auto_check(parent, ring_current, check_func) -> bool:
msg = "Ring current: unknown." if ring_current is None else f"Ring current is low {round(ring_current, 2)} mA."
box = timer_box(parent, title="Ring current too low", msg=msg, condition_func=check_func)
return conditions_auto_check(parent, msg, check_func, title="Ring current too low")
def conditions_auto_check(parent, msg: str, check_func, title: str = "Beamline not ready") -> bool:
"""Pause-and-wait dialog that auto-resumes when ``check_func()`` becomes True.
Used to hold automation between samples until the beamline recovers. The
box auto-accepts (resume) as soon as the condition is satisfied; the user
can also click 'Yes' to continue now (override) or 'No' to stop automation.
Returns True to resume, False to stop.
"""
box = timer_box(parent, title=title, msg=msg, condition_func=check_func)
loop = QEventLoop()
def finish(_=None):
if loop.isRunning():
@@ -2,7 +2,7 @@ import types
import sys
from aare.common.coordinate import AerotechCoordinate, Coordinate
from aare.common.exception_handler import CriticalTellException, MountingFailed
from aare.common.exception_handler import CriticalTellException, DoorSafetyError, MountingFailed
from aare.common.models import DewarAddress, SampleShortInfo
from aare.devices.tell_client import TellEventValueEnum
from aare.daq.operations.mounting.models import MountingContext, MountingResult, MountingDependencies, MountingSettings
@@ -35,7 +35,7 @@ def _make_sample(sample_id: int, name: str) -> SampleShortInfo:
)
def _make_context(previous_sample=None):
def _make_context(previous_sample=None, *, prohibited=True, alarm=False):
tell = types.SimpleNamespace(
mount=lambda **kwargs: TellEventValueEnum.SUCCESS,
unmount=lambda **kwargs: None,
@@ -51,6 +51,10 @@ def _make_context(previous_sample=None):
aerotech_pos=None,
tell=tell,
magnet_position_sensor=types.SimpleNamespace(value=0),
pss=types.SimpleNamespace(
is_prohibited=lambda: prohibited,
alarm_active=lambda: alarm,
),
)
streak = {"count": 0}
@@ -225,4 +229,47 @@ def test_execute_mount_critical_tell_error_does_not_increment_failure_streak(moc
assert result.success is False
assert isinstance(result.error, CriticalTellException)
assert ctx.deps.cfg.get_mount_failure_streak() == 0
assert ctx.deps.cfg.get_mount_failure_streak() == 0
def test_mount_blocked_when_not_prohibited(mock_logger):
"""Doors open / hutch not in prohibited state -> critical DoorSafetyError,
and it is not counted as a mount-failure streak."""
target_sample = _make_sample(2, "new")
ctx = _make_context(prohibited=False)
service = MountingService(context=ctx, logger=mock_logger)
result = service.execute(target=target_sample)
assert result.success is False
assert isinstance(result.error, DoorSafetyError)
assert result.error.critical is True
# Door safety is a pre-flight gate, not a mount attempt: streak stays clear.
assert ctx.deps.cfg.get_mount_failure_streak() == 0
def test_mount_blocked_when_alarm_active(mock_logger):
target_sample = _make_sample(3, "new")
ctx = _make_context(alarm=True)
service = MountingService(context=ctx, logger=mock_logger)
result = service.execute(target=target_sample)
assert result.success is False
assert isinstance(result.error, DoorSafetyError)
assert result.error.critical is True
def test_unmount_blocked_when_not_prohibited_keeps_door_error(mock_logger):
"""Unmount path must not swallow DoorSafetyError into UnmountingFailed."""
previous_sample = _make_sample(1, "old")
ctx = _make_context(previous_sample=previous_sample, prohibited=False)
service = MountingService(context=ctx, logger=mock_logger)
result = service.execute(target=None)
assert result.success is False
assert isinstance(result.error, DoorSafetyError)
assert result.error.critical is True