AareDAQ/CHANGELOG.md

Changelog

0.3.2 (unreleased)

• Authentication & Sessions:
  • Migrated DAQ server URLs to HTTPS with per-beamline configurable certificate paths (cacert) and mTLS; GUI now validates certificates and handles self-signed certs / SSL errors in DAQWorker and SSEClient.
  • Reworked authentication around Kerberos/SPNEGO token negotiation (via curl) behind an Apache/HTTPS proxy; removed the direct gssapi dependency and simplified proxy-header handling.
  • GUI session management: track active GUIs in Redis, show the current baton holder, let staff force-close erroneous sessions, and auto-close idle GUIs after a grace period.
• Operations refactor (major):
  • Standardised every multi-step procedure under aare/daq/operations/<name>/ as a *Service with constructor-injected context (deps, runtime providers, services protocols, settings); added the rotation operation and screenshot/common helpers.
  • Major refactor of the mounting service, plus loop-centering, face-detection and raster services, with runtime-state providers/protocols and updated unit tests.
• Exception model (major refactor):
  • Reworked the exception hierarchy into AareException with a load-bearing critical flag, routing roots (AutomationError/AareUserError/AareAuthError), empty device-family bases for isinstance grouping, and centralised AareErrorCodes served through FastAPI handlers; the GUI reads body.critical to pause/halt automation.
  • Added StateTransitionFailed, MaintenanceStateException, BECCommunicationError, BeamlineBusyTimeoutException, AutoRasterSampleSkipped and improved TransformationInvalidException handling; moved excessive logging out of the exception classes themselves.
• DAQ/Server:
  • Converted the in-use EPICS devices to BEC (ring current, energy, front/back light, zoom, transmission, fast shutter, detector Z), and cached BEC device handles as variables for a large speed-up (~6 µs vs ~200 ms per read).
  • Store detector metadata and TELL events in Redis so /status polling stays fast; moved detector-Z soft-limit checks into the device with a detector_limit_modifier to avoid hitting limits.
  • Added a TELL state handler that derives activity/phase from the TELL SSE stream and surfaces the current robot action in the GUI BUSY label, status bar and recovery panel.
  • Critical detector faults now halt automation (and surface during manual collection); added blower-on for every mount, anneal in/out, change_energy and mono_pitch_scan controls.
  • Scaling/skip handling for over-large raster grids (Jungfrau buffer limit), with a longer busy timeout and busy-timeout exceptions.
• GUI:
  • Added portrait mode, a compact automation view with mode-switching buttons, a Redis-backed automation progress panel, a beamline-state panel, a shared busy-overlay widget, and a splash screen / app banner / icon.
  • Added a local-contact panel that restarts backend device connections (JFJoch, TELL, Aerotech, Smargon), reinitialises BEC macros/devices and opens beamline web helpers.
  • Performance: raster grid manager now renders a single bitmap to remove lag on large grids; added a Database/User-values toggle to data-collection settings.
• Common, Devices & Config:
  • Moved all hardware URLs (GUI, JFJoch, Aerotech, BEC, Smargon, TELL) into per-beamline YAML read via cfg_get; added lens_magnification, maximum_flux, detector_distance_minimum, detector_limit_modifier, beamline-specific transmission PVs and cryojet temperature PVs; removed unused EPICS devices.
• Workflows:
  • Added maintenance2sa/maintenance2se transitions, a safe detector-Z position check, explicit per-transition error messages, and switched to ABR_POS/ABR_OMEGA mount positions; removed the unused ws mode.
• Hutch safety & error pop-ups (in progress):
  • Re-wired the GUI pop-up path: failed user operations (mount/unmount) now raise a modal dialog (e.g. Mounting Failed: No Pin in Gripper) and background/polling errors a non-modal banner — previously the http_error signal was emitted but never connected, so these were silently dropped. Blocking-but-benign errors (e.g. "Beamline is busy" from acting mid-operation) are surfaced quietly in the log/status bar instead of a modal.
  • Added a PssState device reading the hutch personnel-safety system (…-EH1-PSYS:PROHIBITED-STATE / :ALARM-STATE); mounts are blocked (critical DoorSafetyError, automation queue paused) when the hutch is not in the prohibited state or a safety alarm is active, and an active alarm also raises a non-modal warning banner via /status.
  • /status now publishes pss_prohibited/pss_alarm; the GUI blocks manual mount/unmount and the automation Run button immediately (with a pop-up) when the hutch door is open, instead of letting the robot fail to move server-side.
  • Centralised the per-action precondition checks (ring current, safety shutter, hutch door) into one combined "continue?" dialog used by all data-collection buttons (Evaluate grid, X-ray centering, Run screening/rotation/fluorescence, simple rotation), with a session-global "Don't ask me again for 1 hour" snooze.
  • Live automation now pauses and auto-resumes when the beam, experiment shutter, hutch door or robot (TELL) are not ready (with Continue-now / Stop overrides), gated by a "Pause on bad conditions" checkbox (default on) that replaces the dead CHECK_ENABLED constant.
  • Restored "No Pin in Gripper" detection: a mount that returns on the intermediate TELL "Busy" state event now verifies the robot's get_mounted_sample() after the command completes, so an empty-gripper mount surfaces MountingFailed (and the GUI pop-up) instead of silently reporting success.

0.3.1 (2026-04-29)

• DAQ/Server:
  • Improved baton exchange logic with automatic timeout and Redis-backed state.
• GUI:
  • Updated about dialog with correct version and copyright.
  • Improved layout persistence and state management.

0.3.0 (2026-04-24)

• Merge three repos into one, remove Ultralytics as dependency
• Testing & CI/CD:
  • Implemented comprehensive pytest infrastructure for unit and integration tests.
  • Integrated pytest-cov and pytest-mock into the testing workflow with 100% coverage on core modules like coordinate and exception_handler.
  • Added automated testing stage to Gitea workflow (.gitea/workflows/action.yaml) ensuring tests pass before build and documentation updates.
  • Added pytest-qt support for GUI testing.
  • Added tests for aerotech, models, autofocus, automation, beamline, diffraction_geometry, enum_pv, face_detection, jfjoch_client, mlbox, my_motor, spreadsheet_updater, and workflows.
• DAQ/Server:
  • Added lifespan function for one-time initialization of core objects (devices, daq, config) on startup, reducing conflicting Redis and device connections.
  • Big Aerotech upgrade: transitioned to Aerotech API with ported coordinate functions.
  • Re-enabled and updated automation workflow 2.0 with new backend endpoints.
  • Added park_and_dry functionality for repeated mount failures.
  • Workflow assertions to ensure scintillator is down when possible.
  • Improved aerotech movement and position calculations.
  • Implementation of camera auto-exposure and zoom-dependent settings.
  • Added SAS-TT school demo temporary raster function.
  • Enhanced status polling with robust error handling for Tell, Smargon, and Aerotech communication.
  • ml_box updated with beamline-specific URLs, model class types, and new YOLO26 engine.
  • Added separate authentication pathway for staff-only beamline recovery operations.
  • Improved database message limits for omega (to 2 decimal places).
  • Added screenshot to database from GUI; face detection sequence and beamline recovery updates.
  • Throttled UserRightsException logging to reduce log noise.
  • Implemented persistent automation progress tracking using Redis across multiple workers.
  • Added automation progress panel to the GUI for real-time monitoring.
  • Improved raster and rotation scan error handling and centralized database message handling.
  • Updated server_exception_handler.py and error_codes.py to cover all custom exceptions.
  • Added auto_exposure endpoint and improved autofocus logic.
  • Fixed multiple bugs in raster scan positioning and state transitions.
  • Refactored MLBOX and LOOP_CENTERING types; loop centering now avoids moving if only non-crystal classes (pin, ice, needle) are found.
  • Split LOOP_CENTERING from main DAQ script for better maintainability.
• Baton & Sessions:
  • Major rework of the baton exchange procedure and session management.
  • Added NOT_BATON_HOLDER error code and specialized handling for baton requests.
  • Integrated baton status messages into the camera image display and status bar.
  • Improved session state transitions with new Pending states and P-group selection logic.
• GUI:
  • Added ui_state_manager to handle QSettings for persistent UI state.
  • Major rework of predict_subscriber and prediction stream updates; corrected default ZMQ stream for ML box.
  • Added FPS counter and sharpness monitoring for sample camera.
  • Improved camera thread performance and authentication failure handling.
  • Added X10SA specific camera and prediction configurations.
  • New display_status for user-facing connection/error messages.
  • Integrated dev_help panel with logs and error code search.
  • Added framework for building GUI tutorials with first draft of X10SA tutorial.
  • Added alert banner for critical disconnection issues (server/Tell/Aerotech).
  • New screenshot button and controls in sample camera GUI.
  • Added SSE support for face detection panel updates during automation.
  • Integrated sample position resync in recovery panel to reduce Tell polling.
  • Fixed matplotlib warnings in face detection panel and improved layout persistence.
  • Added beamline_state_panel with station widgets for improved state management and visibility.
  • Performance optimizations: throttled raster grid drawing to 25Hz and improved dense heat map rendering.
  • Added automation progress panel and wired it to the persistent Redis backend.
  • Improved prediction subscriber performance and target stability visualization.
  • Added shortcuts and improved user feedback in the status bar using BeamlineStateEnum display names.
  • Improved prediction subscriber and metrics panel performance.
• TELL & Infrastructure:
  • Added mTLS certificate authentication for aaredb, tellupdater, and spreadsheetupdater.
  • Improved SSE client resilience in tellupdater with retry mechanisms and enhanced error logging.
  • Major refactor: removed redundant code and added simulated client.
  • Converted wait functions to use inbuilt pshell functionality.
  • Added error handling and proxy support for connection failures.
  • Fixed event names for mounting, and added missing events for gripper detection and motion sync.
  • Refactored SSE connection handling for simplicity and robustness.
  • Added control for pin blower and annealing (in/out) to TellClient.
  • Improved TELL HTTP communication handling to prevent swallowed errors.
  • Added magnet position sensor state checks and exception handling.
  • Improved Gitea workflows with better dependency management and testing steps.
• Common & Devices:
  • Added BEC worker and Aerotech controller initialization in devices.py.
  • Added magnet position sensor functions and centralized transmission settings.
  • Added centralized error_codes and refactored exception handling to use FastAPI handlers.
  • Expanded device support with new PVs (beamstop, scintillator, collimator, cryojet, etc.).
  • Cleaned up mx_lib wait functions and my_motor class; added clean_filename helper.
  • Improved Smargon connection exception handling.
  • Updated ABR_POS_HOME configuration and added comments for cryo positions.
  • Added image upload comments and improved authentication error messaging.
  • Centralized custom exceptions in aare.common.exception_handler.
  • Added display_name to BeamlineStateEnum for consistent UI strings.
  • Refactored SampleShortInfo handling across DAQ and database modules.
  • Multiple bug fixes in ML box prediction logic and coordinate transformations.
  • JFJOCH wrapper: Added retries for image retrieval and new wait_till_running method.

0.2.69

• check beamline state before running raster, rotation and automation
• lowered polling rate of daq status to improve GUI performance
• robot drys then parks after repeated mount failures

0.2.68:

• added check box to numberlineedits to switch between user input and spreadsheet
• re-enabled MLprediction
• fixed bug in gui where manual sample couldnt be created when run nubmer was negative.
• Combined beamline view now split vertically not horizontally
• New error handling in DAQ for robot mounting errors
• Error box pop ups in GUI when sample gripper detection error occurs
• Automation queue pauses if sample gripper detection error occurs multiple times to allow robot dry, if continues stops and wiats for user interverntion
• added ingest_scan to send rotation and screening result to db

0.2.67:

• added run number to sampleshortinfo
• ingest raster result, including beam center to database
• aerotech goes to mount postion for manual mount
• logger config correctly expands file path for logger location.
• smart_rotiation_panel fix - but will be mvoed to DAQ in future update
• Fluoresence scan can be saved to CSV with right click
• Wait for ready before sending Tell to dry

0.2.66:

• moved rastergridingestmodel and centreofmassmodel from models to raster_grid.py
• after raster load closest image to new centre