Feat/flomni queue with commands #259
@@ -0,0 +1,450 @@
|
||||
# BEC – Beamline Experiment Control: System Overview
|
||||
|
||||
> **Erstellt:** April 2026
|
||||
> **Zweck:** Referenzdokument für Claude (AI-Assistent) zur Orientierung im BEC-Ökosystem am cSAXS-Beamline (X12SA, PSI).
|
||||
> **Nicht entwickelt von Nutzer:** Der Nutzer entwickelt *BEC-Begleit-Software* für cSAXS und die 3D-Ptychographie-Setups (LamNI, OMNY), nicht den BEC-Kern selbst.
|
||||
|
||||
---
|
||||
|
||||
## 1. Was ist BEC?
|
||||
|
||||
**BEC** steht für **Beamline and Experiment Control**. Es ist ein modulares, microservice-basiertes System zur Steuerung von Experimenten an Großforschungsanlagen (Synchrotrons, Neutronenquellen). Entwickelt am PSI (Paul Scherrer Institut), Schweiz.
|
||||
|
||||
- **Lizenz:** BSD-3-Clause
|
||||
- **Sprache:** Python (>99%)
|
||||
- **Python-Versionen:** 3.11, 3.12, 3.13
|
||||
- **Aktuelle Version:** ~3.117.x (bec-Kern), ~3.5.x (bec_widgets), ~1.36.x (ophyd_devices)
|
||||
- **Kommunikations-Backbone:** Redis (Pub/Sub + Streams)
|
||||
- **Dokumentation:** https://bec.readthedocs.io / https://beamline-experiment-control.readthedocs.io
|
||||
---
|
||||
|
||||
## 2. Architektur-Überblick
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ BEC Ecosystem │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌─────────────────┐ │
|
||||
│ │ bec_ipython_ │ │ bec_widgets │ ← GUIs (PySide6) │
|
||||
│ │ client (CLI) │ │ (Qt/PySide6) │ │
|
||||
│ └────────┬────────┘ └────────┬────────┘ │
|
||||
│ │ │ │
|
||||
│ └──────────┬───────────┘ │
|
||||
│ │ bec_lib (gemeinsame Bibliothek) │
|
||||
│ │ Redis Connector / Messages / Endpoints │
|
||||
│ │ │
|
||||
│ ┌──────────▼───────────────────────────┐ │
|
||||
│ │ REDIS │ │
|
||||
│ │ (Pub/Sub + Streams, single source │ │
|
||||
│ │ of truth für alle Services) │ │
|
||||
│ └──────────┬───────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼──────────────────┐ │
|
||||
│ │ bec_server │ │
|
||||
│ │ ┌─────────────────────┐ │ │
|
||||
│ │ │ device_server │ │ ← ophyd / EPICS │
|
||||
│ │ │ scan_server │ │ │
|
||||
│ │ │ scan_bundler │ │ │
|
||||
│ │ │ file_writer │ │ │
|
||||
│ │ │ data_processing │ │ ← DAP Plugins │
|
||||
│ │ │ scihub │ │ ← Logbook, Atlas │
|
||||
│ │ └─────────────────────┘ │ │
|
||||
│ └──────────┬──────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼──────────────────┐ │
|
||||
│ │ Hardware / EPICS IOCs │ │
|
||||
│ │ (Motoren, Detektoren, ...) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Beamline-spezifische Plugins (z.B. csaxs_bec):
|
||||
- Erweitern device_server, scan_server, file_writer, bec_widgets
|
||||
- Laden über BEC Plugin-System (pyproject.toml entry points)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Kern-Repositories
|
||||
|
||||
Alle unter: https://gitea.psi.ch/bec (Mirrors von https://github.com/bec-project)
|
||||
|
||||
| Repository | Beschreibung | Wichtig für |
|
||||
|---|---|---|
|
||||
| **bec** | Haupt-Monorepo | Alles |
|
||||
| **bec_widgets** | Qt/PySide6 GUI-Toolkit | GUI-Entwicklung |
|
||||
| **ophyd_devices** | Geräteklassen (ophyd-Erweiterung) | Device-Entwicklung |
|
||||
| **csaxs_bec** | cSAXS-spezifische Plugins | cSAXS / LamNI / OMNY |
|
||||
|
||||
---
|
||||
|
||||
## 4. Repository: `bec` (Monorepo)
|
||||
|
||||
**URL:** https://gitea.psi.ch/bec/bec
|
||||
**GitHub:** https://github.com/bec-project/bec
|
||||
**Struktur:**
|
||||
|
||||
```
|
||||
bec/
|
||||
├── bec_lib/ ← Gemeinsame Python-Bibliothek
|
||||
│ └── bec_lib/
|
||||
│ ├── client.py ← BECClient Hauptklasse
|
||||
│ ├── device.py ← Device-Klassen (Wrapper um ophyd)
|
||||
│ ├── devicemanager.py ← DeviceManager (dev.samx, etc.)
|
||||
│ ├── endpoints.py ← Redis Message Endpoints (alle Topics)
|
||||
│ ├── messages.py ← Message-Typen (Pydantic-ähnlich)
|
||||
│ ├── redis_connector.py ← Redis-Verbindung (pub/sub, streams)
|
||||
│ ├── connector.py ← Abstrakte Connector-Klasse
|
||||
│ ├── scan_items.py ← Scan-Objekte
|
||||
│ ├── queue_items.py ← Scan-Queue-Management
|
||||
│ ├── callback_handler.py← Callback-System
|
||||
│ ├── alarm_handler.py ← Alarm-Handling
|
||||
│ ├── bec_service.py ← Basis-Klasse für alle BEC-Services
|
||||
│ ├── bec_errors.py ← Fehlerklassen
|
||||
│ ├── logger.py ← Logging-System
|
||||
│ ├── file_utils.py ← Datei-Hilfsfunktionen
|
||||
│ ├── live_scan_data.py ← Live-Daten während Scan
|
||||
│ ├── dap_plugins.py ← Data Analysis Plugins
|
||||
│ ├── dap_plugin_objects.py
|
||||
│ ├── lmfit_serializer.py← lmfit-Integration (Kurvenanpassung)
|
||||
│ ├── metadata_schema.py ← Scan-Metadata Schema
|
||||
│ ├── plugin_helper.py ← Plugin-Hilfsfunktionen
|
||||
│ ├── logbook_connector.py← Scilog/Logbook
|
||||
│ ├── acl_login.py ← Zugangskontrolle
|
||||
│ ├── bl_state_manager.py← Beamline State
|
||||
│ ├── codecs.py ← Serialisierung (msgpack)
|
||||
│ ├── numpy_encoder.py ← NumPy-Serialisierung
|
||||
│ ├── configs/ ← Default-Konfigurationen
|
||||
│ ├── procedures/ ← Scan-Prozeduren
|
||||
│ └── utils/ ← Hilfsfunktionen
|
||||
│
|
||||
├── bec_server/ ← Server-Seite (alle Microservices)
|
||||
│ └── bec_server/
|
||||
│ ├── device_server/ ← Kommuniziert mit Hardware via ophyd/EPICS
|
||||
│ ├── scan_server/ ← Führt Scans aus, verwaltet Queue
|
||||
│ ├── scan_bundler/ ← Bündelt Scan-Daten (Device + Scan Events)
|
||||
│ ├── file_writer/ ← Schreibt HDF5/NeXus-Dateien
|
||||
│ ├── file_writer_plugins/← Plugin-System für Dateiformat
|
||||
│ ├── data_processing/ ← Online-Datenanalyse (DAP)
|
||||
│ ├── procedures/ ← Server-seitige Prozeduren
|
||||
│ ├── scihub/ ← Anbindung externe Dienste (Atlas, Scilog)
|
||||
│ └── bec_server_utils/ ← CLI-Argument-Parsing
|
||||
│
|
||||
├── bec_ipython_client/ ← IPython CLI Interface
|
||||
│ └── bec_ipython_client/
|
||||
│ ├── bec_startup.py ← Startup-Script (lädt beim `bec`-Command)
|
||||
│ ├── main.py ← Einstiegspunkt
|
||||
│ ├── bec_magics.py ← IPython Magic Commands (%bec, etc.)
|
||||
│ ├── beamline_mixin.py ← Beamline-spezifische Mixins
|
||||
│ ├── callbacks/ ← CLI-Callbacks (Progress, Printing)
|
||||
│ ├── high_level_interfaces/ ← Höhere API (scans, dev, etc.)
|
||||
│ └── plugins/ ← Plugin-Verzeichnis
|
||||
│
|
||||
├── pytest_bec_e2e/ ← End-to-End Tests
|
||||
├── docs/ ← Sphinx-Dokumentation
|
||||
├── bin/ ← Start-Scripts
|
||||
└── bec_config_template.yaml ← Beispiel-Konfiguration
|
||||
```
|
||||
|
||||
### Wichtige Konzepte in bec_lib
|
||||
|
||||
**Redis Endpoints** (`endpoints.py`): Alle Redis-Topics sind hier zentral definiert.
|
||||
Beispiele:
|
||||
- `MessageEndpoints.device_readback(device_name)` → live Gerätewert
|
||||
- `MessageEndpoints.scan_segment` → Scan-Datenpunkte
|
||||
- `MessageEndpoints.scan_status` → Scan-Status
|
||||
**BECClient** (`client.py`): Haupt-Einstiegspunkt für alle Clients:
|
||||
```python
|
||||
from bec_lib.client import BECClient
|
||||
bec = BECClient()
|
||||
bec.start()
|
||||
scans = bec.scans
|
||||
dev = bec.device_manager.devices
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Repository: `ophyd_devices`
|
||||
|
||||
**URL:** https://gitea.psi.ch/bec/ophyd_devices
|
||||
**GitHub:** https://github.com/bec-project/ophyd_devices
|
||||
**Beschreibung:** Erweiterung von [ophyd](https://blueskyproject.io/ophyd/) für Geräte, die nicht durch Standard-EPICS-ophyd abgedeckt sind.
|
||||
|
||||
```
|
||||
ophyd_devices/
|
||||
├── ophyd_devices/
|
||||
│ ├── devices/ ← Konkrete Geräteklassen (EpicsMotor-Subklassen, etc.)
|
||||
│ ├── interfaces/ ← Abstrakte Interface-Klassen (SimplePositioner, etc.)
|
||||
│ ├── sim/ ← Simulierte Geräte für Tests/Demos
|
||||
│ ├── configs/ ← Beispiel-Gerätekonfigurationen (YAML)
|
||||
│ ├── utils/ ← Hilfsfunktionen (Static Device Tests, etc.)
|
||||
│ └── tests/ ← Unit-Tests
|
||||
└── tests/ ← Integrations-Tests
|
||||
```
|
||||
|
||||
**Zentrale Konzepte:**
|
||||
- Geräte erben von ophyd (`Device`, `EpicsMotor`, `EpicsSignal`, etc.)
|
||||
- BEC-spezifische Erweiterungen: Callbacks für Redis, BEC-Signale
|
||||
- `interfaces/` enthält abstrakte Klassen wie `SimplePositioner`
|
||||
- Simulierte Geräte in `sim/` für Offline-Entwicklung
|
||||
---
|
||||
|
||||
## 6. Repository: `bec_widgets`
|
||||
|
||||
**URL:** https://gitea.psi.ch/bec/bec_widgets
|
||||
**GitHub:** https://github.com/bec-project/bec_widgets
|
||||
**Framework:** PySide6 (Qt6)
|
||||
|
||||
```
|
||||
bec_widgets/
|
||||
├── bec_widgets/
|
||||
│ ├── widgets/
|
||||
│ │ ├── plots/ ← Waveform, MultiWaveform, Image/Heatmap
|
||||
│ │ ├── control/ ← PositionerBox, TweakWidget, ScanControl
|
||||
│ │ ├── containers/ ← BECDockArea (Haupt-Layout-Container)
|
||||
│ │ ├── dap/ ← Data Analysis Plugin Widgets (lmfit Dialog)
|
||||
│ │ ├── editors/ ← Code/Config-Editoren
|
||||
│ │ ├── progress/ ← Scan-Fortschrittsanzeige
|
||||
│ │ ├── services/ ← Service-Status-Widgets
|
||||
│ │ ├── utility/ ← Diverse Hilfs-Widgets
|
||||
│ │ └── games/ ← Easter Egg 🎮
|
||||
│ ├── cli/ ← RPC-CLI Bindings (auto-generiert via bw-generate-cli)
|
||||
│ ├── applications/ ← Fertige Anwendungen (BEC Launcher, Designer, Terminal)
|
||||
│ ├── utils/ ← Plugin-Template-Generator, Hilfsfunktionen
|
||||
│ ├── examples/ ← Beispiel-Widgets
|
||||
│ └── assets/ ← Icons, Bilder
|
||||
├── tests/
|
||||
└── docs/
|
||||
```
|
||||
|
||||
**Kern-Konzepte:**
|
||||
- `BECWidget` (Basisklasse): Alle Widgets erben davon. Bietet BEC/Redis-Verbindung, RPC.
|
||||
- `BECDispatcher`: Vermittelt Redis-Events an Qt-Slots.
|
||||
- `USER_ACCESS` Liste: Definiert, welche Methoden über RPC erreichbar sind.
|
||||
- `SafeSlot`: Thread-sicherer Qt-Slot Decorator.
|
||||
- `BECDockArea`: Haupt-GUI-Container mit Drag-and-Drop-Docking.
|
||||
**Widget erstellen (Minimalbeispiel):**
|
||||
```python
|
||||
from bec_widgets import BECWidget, SafeSlot
|
||||
from bec_lib.endpoints import MessageEndpoints
|
||||
|
||||
class MyWidget(BECWidget, QWidget):
|
||||
USER_ACCESS = ["my_method"]
|
||||
|
||||
def __init__(self, parent=None, **kwargs):
|
||||
super().__init__(parent=parent, **kwargs)
|
||||
self.bec_dispatcher.connect_slot(
|
||||
self.on_data,
|
||||
MessageEndpoints.device_readback("samx")
|
||||
)
|
||||
|
||||
@SafeSlot(dict, dict)
|
||||
def on_data(self, data, meta):
|
||||
...
|
||||
```
|
||||
|
||||
**CLI starten:** `bec` → öffnet IPython + BECDockArea GUI
|
||||
**Designer:** `bec-designer` → Qt Designer mit BEC-Widgets
|
||||
**RPC generieren:** `bw-generate-cli --target <plugin-repo>`
|
||||
|
||||
---
|
||||
|
||||
## 7. Repository: `csaxs_bec`
|
||||
|
||||
**URL:** https://gitea.psi.ch/bec/csaxs_bec
|
||||
**Beschreibung:** cSAXS-spezifische Plugins und Konfigurationen für BEC (X12SA Beamline, PSI)
|
||||
|
||||
```
|
||||
csaxs_bec/
|
||||
├── csaxs_bec/
|
||||
│ ├── devices/ ← cSAXS-Hardware-Geräteklassen
|
||||
│ │ ├── epics/ ← Standard EPICS Geräte (MCS-Karte, etc.)
|
||||
│ │ ├── ids_cameras/ ← IDS-Kameras (Alignment-Kameras)
|
||||
│ │ ├── jungfraujoch/ ← JungFrauJoch (JFJ) Detektor-Integration
|
||||
│ │ ├── npoint/ ← nPoint Piezo-Positionierer (Nano-Scans)
|
||||
│ │ ├── omny/ ← OMNY Tomographie-Stage
|
||||
│ │ ├── panda_box/ ← PandaBox Timing/Trigger-System
|
||||
│ │ ├── pseudo_devices/ ← Pseudo-Geräte (berechnete Positionen)
|
||||
│ │ ├── sls_devices/ ← SLS-spezifische Geräte (Synchrotron Light Source)
|
||||
│ │ ├── smaract/ ← SmarAct Piezo-Positionierer
|
||||
│ │ ├── tests_utils/ ← Test-Hilfsfunktionen (patch_dual_pvs)
|
||||
│ │ └── device_list.md ← Dokumentation aller Geräte
|
||||
│ │
|
||||
│ ├── scans/ ← cSAXS-spezifische Scan-Plugins
|
||||
│ │ ├── flomni_fermat_scan.py ← FlOMNI Fermat-Spiral Scan (Ptychographie)
|
||||
│ │ ├── omny_fermat_scan.py ← OMNY Fermat-Spiral Scan
|
||||
│ │ ├── LamNIFermatScan.py ← LamNI Fermat-Scan (3D Ptychographie!)
|
||||
│ │ ├── jungfrau_joch_scan.py ← JFJ-spezifischer Scan
|
||||
│ │ ├── sgalil_grid.py ← SGalil Grid-Scan (Galil-Controller)
|
||||
│ │ ├── owis_grid.py ← OWIS Grid-Scan
|
||||
│ │ ├── scan_plugin_template.py← Vorlage für neue Scan-Plugins
|
||||
│ │ └── metadata_schema/ ← cSAXS Metadata-Schema
|
||||
│ │
|
||||
│ ├── bec_widgets/ ← cSAXS-spezifische GUI-Widgets
|
||||
│ │ ├── widgets/
|
||||
│ │ │ ├── xray_eye/ ← X-Ray Eye Widget (Kamera-Alignment-Tool)
|
||||
│ │ │ └── client.py ← RPC-Client für cSAXS-Widgets
|
||||
│ │ └── auto_updates/ ← Automatische GUI-Updates
|
||||
│ │
|
||||
│ ├── bec_ipython_client/ ← cSAXS CLI-Erweiterungen
|
||||
│ │ └── ... ← Web-Generator, Auth-Integration
|
||||
│ │
|
||||
│ ├── device_configs/ ← YAML-Gerätekonfigurationen
|
||||
│ │ └── ... ← z.B. bec_device_config_sastt.yaml
|
||||
│ │
|
||||
│ ├── file_writer/ ← cSAXS-spezifischer File Writer Plugin
|
||||
│ ├── dap_services/ ← Online Datenanalyse (DAP) Services
|
||||
│ ├── macros/ ← BEC-Makros
|
||||
│ ├── services/ ← Zusatz-Services
|
||||
│ └── deployments/ ← Deployment-Konfigurationen
|
||||
│
|
||||
├── bin/ ← Start-Scripts
|
||||
├── docs/ ← Dokumentation
|
||||
└── tests/ ← Tests
|
||||
```
|
||||
|
||||
### Hardware am cSAXS (X12SA)
|
||||
|
||||
| Gerät | Klasse/Modul | Typ |
|
||||
|---|---|---|
|
||||
| Eiger | EPICS IOC | Röntgen-Flächendetektor |
|
||||
| Pilatus 300K | EPICS IOC | Röntgen-Flächendetektor |
|
||||
| FalconX1 | EPICS IOC | Fluoreszenz-Detektor |
|
||||
| JungFrauJoch (JFJ) | `devices/jungfraujoch/` | Röntgen-Detektor (High-rate) |
|
||||
| OMNY | `devices/omny/` | Tomographie-Drehtisch |
|
||||
| nPoint | `devices/npoint/` | Piezo-Positionierer (Nano) |
|
||||
| SmarAct | `devices/smaract/` | Piezo-Positionierer |
|
||||
| PandaBox | `devices/panda_box/` | Timing/Trigger-System |
|
||||
| MCS-Karte | `devices/epics/` | Multi-Channel Scaler |
|
||||
| IDS-Kameras | `devices/ids_cameras/` | Sichtbare-Licht-Kameras |
|
||||
| SGalil | EPICS/Config | Galil-Motorcontroller |
|
||||
| OWIS | Config | Motorcontroller |
|
||||
| Delay Generator (DDG) | EPICS IOC | Pulsgenerator (Timing) |
|
||||
|
||||
### Scans am cSAXS
|
||||
|
||||
| Scan | Datei | Verwendung |
|
||||
|---|---|---|
|
||||
| `sgalil_grid` | `sgalil_grid.py` | Standard-Raster-Scan (Galil-Controller) |
|
||||
| `omny_fermat_scan` | `omny_fermat_scan.py` | OMNY Fermat-Spiral (Ptychographie) |
|
||||
| `flomni_fermat_scan` | `flomni_fermat_scan.py` | FlOMNI Fermat-Spiral |
|
||||
| `LamNIFermatScan` | `LamNIFermatScan.py` | **3D Ptychographie (LamNI)** |
|
||||
| `jungfrau_joch_scan` | `jungfrau_joch_scan.py` | JFJ-Burst-Akquisition |
|
||||
| `owis_grid` | `owis_grid.py` | OWIS-Raster-Scan |
|
||||
|
||||
---
|
||||
|
||||
## 8. Beamline-Setup: cSAXS (X12SA)
|
||||
|
||||
**Beamline:** X12SA, Paul Scherrer Institut (PSI), Schweiz
|
||||
**Synchrotron:** SLS (Swiss Light Source)
|
||||
|
||||
### 3D Ptychographie / LamNI
|
||||
|
||||
**LamNI** = *Laminar Nano-Imaging* – das 3D Ptychographie-Setup am cSAXS.
|
||||
Relevante Komponenten:
|
||||
- **Scan:** `LamNIFermatScan` (Fermat-Spiral, optimiert für Ptychographie)
|
||||
- **Positionierung:** nPoint Piezo-Stages (Nanometer-Präzision) + übergeordnete Motoren
|
||||
- **Detektor:** Eiger oder ähnlich (kohärentes Streumuster)
|
||||
- **Timing:** PandaBox oder Delay-Generator
|
||||
### OMNY
|
||||
|
||||
OMNY = Tomographie-System mit Drehtisch.
|
||||
- **Scan:** `omny_fermat_scan`
|
||||
- **Gerät:** `devices/omny/`
|
||||
- Verwendet einen dedizierten "Flyer" für kontinuierliche Akquisition
|
||||
### Startup am cSAXS
|
||||
|
||||
```bash
|
||||
# BEC-Server starten (auf pc15543)
|
||||
cd ~/Data10/software
|
||||
source bec_venv/bin/activate
|
||||
bec-server start
|
||||
tmux attach -t bec
|
||||
|
||||
# BEC-Client starten
|
||||
bec
|
||||
|
||||
# Gerätekonfiguration laden
|
||||
bec.config.update_session_with_file('~/Data10/software/csaxs-bec/bec_plugins/configs/bec_device_config_sastt.yaml')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. BEC Plugin-System
|
||||
|
||||
BEC nutzt Python Entry Points für Plugins. Ein Plugin-Paket (z.B. `csaxs_bec`) registriert seine Komponenten in `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[project.entry-points."bec.plugins"]
|
||||
csaxs_bec = "csaxs_bec"
|
||||
|
||||
[project.entry-points."bec.scan_plugins"]
|
||||
csaxs_scans = "csaxs_bec.scans"
|
||||
|
||||
[project.entry-points."bec.file_writer_plugins"]
|
||||
csaxs_writer = "csaxs_bec.file_writer"
|
||||
```
|
||||
|
||||
BEC lädt automatisch alle registrierten Plugins beim Start.
|
||||
|
||||
---
|
||||
|
||||
## 10. Kommunikation & Datenfluss
|
||||
|
||||
```
|
||||
Hardware → EPICS IOC → ophyd (device_server) → Redis
|
||||
↓
|
||||
Client (bec_lib) ← Redis ← scan_bundler ← scan_server
|
||||
↓
|
||||
file_writer → HDF5/NeXus Dateien
|
||||
↓
|
||||
data_processing (DAP) → Online-Analyse
|
||||
```
|
||||
|
||||
**Wichtige Redis Message-Typen** (in `bec_lib/messages.py`):
|
||||
- `DeviceMessage` – Gerätewerte
|
||||
- `ScanMessage` – Scan-Informationen
|
||||
- `BECStatus` – Service-Status
|
||||
- `AlarmMessage` – Alarme/Fehler
|
||||
- `LogMessage` – Log-Einträge
|
||||
- `ScilogMessage` – Logbuch-Einträge
|
||||
---
|
||||
|
||||
## 11. Entwicklungs-Workflow
|
||||
|
||||
### Code-Standards
|
||||
- **Formatter:** Black (`--line-length=100 --skip-magic-trailing-comma`)
|
||||
- **Import-Sortierung:** isort (`--profile=black`)
|
||||
- **Linter:** Pylint
|
||||
- **Tests:** pytest
|
||||
- **Commit-Format:** Conventional Commits (`feat:`, `fix:`, `refactor:`, etc.)
|
||||
### Typisches Widget-Entwicklungs-Muster (csaxs_bec)
|
||||
|
||||
1. Widget-Klasse in `csaxs_bec/bec_widgets/widgets/<name>/` erstellen
|
||||
2. Von `BECWidget` + Qt-Widget erben
|
||||
3. `USER_ACCESS` für RPC definieren
|
||||
4. Auf Redis-Endpoints subscriben via `bec_dispatcher.connect_slot()`
|
||||
5. RPC-Client generieren: `bw-generate-cli --target csaxs_bec`
|
||||
6. Qt-Designer-Plugin wird automatisch registriert
|
||||
### Typisches Scan-Entwicklungs-Muster
|
||||
|
||||
1. Scan-Klasse in `csaxs_bec/scans/` erstellen
|
||||
2. Von BEC-Basis-Scan-Klasse erben
|
||||
3. In `pyproject.toml` registrieren
|
||||
4. Zugriff im Client: `scans.mein_scan(...)`
|
||||
---
|
||||
|
||||
## 12. Nützliche Links
|
||||
|
||||
| Ressource | URL |
|
||||
|---|---|
|
||||
| BEC Gitea | https://gitea.psi.ch/bec |
|
||||
| BEC GitHub | https://github.com/bec-project |
|
||||
| BEC Docs | https://bec.readthedocs.io |
|
||||
| BEC Widgets Docs | https://bec-widgets.readthedocs.io |
|
||||
| ophyd Docs | https://blueskyproject.io/ophyd/ |
|
||||
| cSAXS BEC Repo | https://gitea.psi.ch/bec/csaxs_bec |
|
||||
|
||||
---
|
||||
|
||||
*Dieses Dokument sollte dem Projekt hinzugefügt werden und dient als primäre Referenz für Claude in zukünftigen Gesprächen.*
|
||||
@@ -0,0 +1,282 @@
|
||||
# How E2E Tests Work in csaxs_bec
|
||||
|
||||
**Purpose of this document:** a plugin-agnostic guide to writing and running
|
||||
end-to-end tests against a live simulated BEC deployment in this repo. It
|
||||
generalizes the patterns built for the flOMNI tomo-queue e2e suite
|
||||
(`omny_e2e_tests/`) so the same approach can be reused for LamNI, OMNY, or any
|
||||
other plugin — without re-deriving the traps below from scratch. If you are
|
||||
an AI coding assistant asked to add or extend e2e tests in this repo, read
|
||||
this file first.
|
||||
|
||||
Companion docs, for concrete worked examples:
|
||||
- `SIMULATED_ENDSTATIONS.md` — what the simulated device layer actually is
|
||||
(protocol-level socket injection) and how to bring a sim session up.
|
||||
- `TOMO_QUEUE_TESTING.md` — the fullest worked example: a real e2e checklist
|
||||
built and passed against the flOMNI sim, including every gotcha in this
|
||||
document as it was actually hit.
|
||||
|
||||
---
|
||||
|
||||
## 1. What "e2e" means here, and when you need it
|
||||
|
||||
A **unit test** exercises code with everything below it mocked or stubbed. An
|
||||
**e2e test** in this repo means: a real `BECIPythonClient`, a real running
|
||||
`ScanServer`/`ScanBundler`/`DeviceServer`/`SciHub`, real Redis, and real
|
||||
plugin/device Python code (`Flomni`, `dev.*`, ...) — running against the
|
||||
**simulated** device layer (`csaxs_bec/devices/sim/*`, see
|
||||
`SIMULATED_ENDSTATIONS.md`) instead of real hardware. Nothing about the
|
||||
control-flow code under test is mocked; only the wire protocol at the very
|
||||
bottom is simulated.
|
||||
|
||||
Write an e2e test (instead of, or in addition to, a unit test) when:
|
||||
- the thing you need to prove is that a **global-var-backed** persistence
|
||||
mechanism actually round-trips through real Redis (a mock client's
|
||||
`get_global_var`/`set_global_var` proves nothing about serialization).
|
||||
See "Redis-backed jobs..." below for a case this bit us in practice, and note this is different from
|
||||
Redis contamination *between test runs*, which is §4 below.
|
||||
- you need to prove a **live scan/move actually happened** — e.g. that a
|
||||
queued job's parameters were really restored onto the live property a
|
||||
scan reads from, not just that the right method was called with the right
|
||||
arguments.
|
||||
- you're testing **crash-resume** or **concurrency** — behavior that only
|
||||
exists because state lives in a shared external store (Redis) that
|
||||
survives process death, and multiple independent clients can touch it at
|
||||
once. A mock can't simulate "the process was SIGKILLed."
|
||||
- you're testing something that depends on **real device motion timing**
|
||||
(progress reporting, ETA estimates, ordering of events during a move).
|
||||
|
||||
Prove the logic with a fast mock/unit test first if you can (see
|
||||
`TOMO_QUEUE_TESTING.md` §2 for an example of extracting real source via AST
|
||||
into a mock harness) — it's much cheaper to iterate on. The e2e test is what
|
||||
makes it *actually* verified, not just internally consistent.
|
||||
|
||||
---
|
||||
|
||||
## 2. Folder convention: `omny_e2e_tests/`, not `tests/`
|
||||
|
||||
E2e tests for this repo live in a **separate top-level folder**,
|
||||
`omny_e2e_tests/`, not under `tests/`. This is deliberate: gitea's CI
|
||||
(`.gitea/workflows/ci.yml`) runs `pytest ... ./csaxs_bec/tests/`
|
||||
unconditionally on every push/PR, but these tests need a live sim (a running
|
||||
BEC deployment + Redis), not just a Python environment. Keeping them
|
||||
physically outside `tests/` means:
|
||||
- they never get silently swept into (and slow down, or fail in) the
|
||||
standard CI run,
|
||||
- there's no confusion about whether a given `tests/...` path is a fast unit
|
||||
test or a slow e2e test that assumes a live sim is already up.
|
||||
|
||||
Run them explicitly: `pytest omny_e2e_tests/` (or a specific file/test within
|
||||
it), against a sim session you've already started — see
|
||||
`SIMULATED_ENDSTATIONS.md` for how to bring one up.
|
||||
|
||||
If you're adding e2e coverage for a new plugin, put it under
|
||||
`omny_e2e_tests/` too (a subfolder per plugin is fine once there's more than
|
||||
one), not under `tests/`.
|
||||
|
||||
---
|
||||
|
||||
## 3. The trap: don't use the shipped demo-config client fixtures
|
||||
|
||||
BEC ships a pytest plugin, **`pytest_bec_e2e`** (registered `pytest11` entry
|
||||
point, declared as a `dev` extra in `pyproject.toml`), providing a **live**
|
||||
`BECClient` fixture stack: real server, real Redis, real global vars. Without
|
||||
`--start-servers` it **attaches to an already-running server** — exactly the
|
||||
sim session you started by hand.
|
||||
|
||||
**Do not use `bec_client_lib_with_demo_config` / `bec_ipython_client_with_demo_config`
|
||||
as they ship.** Both call `bec.config.load_demo_config(force=True)` internally.
|
||||
Against a running sim session, this **overwrites the loaded simulated device
|
||||
config** (e.g. `simulated_flomni.yaml`) with BEC's own generic demo devices —
|
||||
silently breaking every test that assumes your plugin's real devices exist.
|
||||
|
||||
Instead, write your own fixture that reuses the lower-level plumbing those
|
||||
fixtures are built on (`bec_servers`, `bec_redis_fixture`,
|
||||
`bec_services_config_file_path`) but either loads your own device config
|
||||
explicitly, or — simpler, and what `omny_e2e_tests/` does — assumes the
|
||||
already-running sim session has it loaded, and never calls
|
||||
`load_demo_config()`/`update_session_with_file()` at all.
|
||||
|
||||
---
|
||||
|
||||
## 4. The fixture pattern: a `build_<plugin>()` bootstrap
|
||||
|
||||
The reusable core, worth more than any individual test: a small set of files
|
||||
that build a real, working plugin instance (`Flomni`, or your plugin's
|
||||
equivalent) pointed at the running sim, with every real-instrument/test-only
|
||||
side effect handled **once**, in one place.
|
||||
|
||||
- **`_bootstrap.py`** — `build_<plugin>(services_config_path) -> (bec, obj)`:
|
||||
connects a `BECIPythonClient` (no demo-config load, see §3), does the
|
||||
builtins/reload bootstrap (§5.2), neutralizes side effects (§5.3), and
|
||||
performs any one-time real-instrument setup your plugin needs before it's
|
||||
usable (§5.4). Also exposes a `shutdown(bec)` helper.
|
||||
- **`conftest.py`** — the actual pytest fixture (e.g. `flomni_sim`). Calls
|
||||
`build_<plugin>()`, and snapshots + **resets to a known-empty state** +
|
||||
restores any shared global-var state around each test (§6 — do not skip
|
||||
the reset step).
|
||||
- **`_run_<x>_subprocess.py`** — a standalone script, *not* a pytest file,
|
||||
that calls `build_<plugin>()` and then does the one blocking/dangerous
|
||||
thing your crash/concurrency tests need to kill or race (e.g.
|
||||
`flomni.tomo_queue_execute()`). Needed because of the main-thread
|
||||
constraint in §5.1 — see that section for why this can't just be a Python
|
||||
thread inside the test process.
|
||||
- **`_<x>_helpers.py`** — everything else shared across test files: fast
|
||||
test-data presets (so a real operation finishes in seconds, not minutes —
|
||||
§5.5), a `spawn_..._subprocess()` wrapper around `subprocess.Popen` for the
|
||||
script above, a `wait_until(predicate, timeout)` poller, and any
|
||||
background-thread "sample state while something blocking runs in the
|
||||
foreground" helper (§5.1).
|
||||
|
||||
Every future e2e effort for the *same* plugin should extend these four files,
|
||||
not duplicate their logic. A new plugin gets its own set, following the same
|
||||
shape.
|
||||
|
||||
---
|
||||
|
||||
## 5. Gotchas (hard-won, apply generally — not bugs, just non-obvious)
|
||||
|
||||
These came from getting a real flOMNI scan to actually run end-to-end; none
|
||||
of them are specific to tomo-queue, and any new e2e suite in this repo will
|
||||
likely hit the same ones.
|
||||
|
||||
### 5.1 Main-thread-only live-update machinery
|
||||
|
||||
`BECIPythonClient`'s live-update machinery (anything that goes through
|
||||
`ipython_live_updates.process_request` — which includes ordinary device
|
||||
moves like `scans.umv(...)`) installs a `SIGINT` handler **per request**.
|
||||
Python only allows `signal.signal()` from a process's real main thread.
|
||||
|
||||
Consequence: **the actual scan/move/queue-execute call can never run on a
|
||||
background Python thread.** Two different escapes, depending on what you
|
||||
need:
|
||||
- To *observe* live state (progress, a global var) while a blocking call
|
||||
runs in the foreground on the main thread: sample on a background thread
|
||||
instead (see `ProgressSampler` in `omny_e2e_tests/_queue_helpers.py`).
|
||||
- To *kill* a running call (crash-resume tests) or run *two* independent
|
||||
sessions concurrently (concurrency tests): spawn a real OS subprocess (its
|
||||
own real main thread) via the `_run_<x>_subprocess.py` script, not a
|
||||
thread. `subprocess.Popen` + `SIGKILL`/`terminate()`, not
|
||||
`threading.Thread` + some cooperative stop flag.
|
||||
|
||||
### 5.2 The builtins/reload bootstrap is order-sensitive
|
||||
|
||||
Plugin modules commonly have module-level helpers (e.g. `flomni.py`'s
|
||||
`def umv(*args): return scans.umv(...)`) that resolve globals like `scans`
|
||||
from **the module's own globals**, populated only by an
|
||||
`if builtins.__dict__.get("bec") is not None:` block that runs once, at
|
||||
**import time**. If the module is imported (even transitively, by an
|
||||
unrelated earlier import) before `builtins.bec` is set, that block never
|
||||
runs, and the helper `NameError`s the first time it's actually called —
|
||||
possibly minutes into a test, on the call that matters.
|
||||
|
||||
Fix, in this order: set `builtins.__dict__["bec"]` (and `dev`, `scans`, ...)
|
||||
**first**, then `importlib.reload()` the plugin module so the
|
||||
import-time block re-runs against the now-populated builtins. This is needed
|
||||
even in a fresh subprocess — the module may already have been imported once,
|
||||
transitively, before your bootstrap code got to run.
|
||||
|
||||
### 5.3 Real side effects must be neutralized for tests
|
||||
|
||||
`Flomni.__init__` (and likely your plugin's `__init__`) has real,
|
||||
unconditional side effects that are wrong in a test process: starting a
|
||||
local HTTP server and background threads that POST to a real external URL
|
||||
(webpage generator), sending a real SciLog summary at the end of every scan
|
||||
regardless of whether SciLog is configured/reachable, blocking on an
|
||||
operator confirmation prompt (`yesno`) that would otherwise hang a headless
|
||||
test run forever.
|
||||
|
||||
Patch these **once**, as plain class-attribute assignment in
|
||||
`neutralize_side_effects()` inside `_bootstrap.py` — not with pytest's
|
||||
`monkeypatch` per test, since every caller (the in-process fixture, the
|
||||
subprocess script) wants the same patches for the life of the whole process,
|
||||
and there's nothing to revert. Find these by actually running an
|
||||
unpatched instantiation once and see what tries to reach the network in the
|
||||
test log, wraps every scan in `try/except RuntimeError` — don't just guess.
|
||||
|
||||
### 5.4 One-time real-instrument setup you must satisfy before anything works
|
||||
|
||||
Some plugins need a real hardware-adjacent precondition satisfied before
|
||||
their normal operations will run at all — e.g. flOMNI's RT interferometer
|
||||
feedback loop must be enabled, and the sample stage moved near its "in"
|
||||
position, before a scan can start; a second call must check whether that's
|
||||
already true (the check itself, or the setup call, may be expensive — e.g.
|
||||
re-zeroing interferometers — so don't redo it if a previous session in the
|
||||
same test run already did). This kind of thing generally isn't visible from
|
||||
reading the plugin's own primary API — it's usually folded into some other
|
||||
higher-level "alignment" or "init" flow that tests otherwise skip. Find it by
|
||||
tracing what a manual/real operator session does before their first scan,
|
||||
not by trial and error against test failures.
|
||||
|
||||
### 5.5 Make the real operation fast enough to actually test
|
||||
|
||||
A real tomogram or scan can take many minutes at production settings — too
|
||||
slow to run repeatedly in a test suite, especially for crash tests that need
|
||||
to interrupt it mid-flight without the whole suite taking forever. Build a
|
||||
"fast" parameter preset (single point per position instead of a full raster,
|
||||
short per-point dwell/counting time, small field of view, coarse angular
|
||||
stepping, ...) that still exercises the *real* code path, just with a much
|
||||
smaller N — see `SHORT_SCAN_PARAMS`/`FAST_STEPSIZE` in
|
||||
`omny_e2e_tests/_queue_helpers.py` for a concrete example (a full 8-subtomo
|
||||
tomogram in roughly a minute, still long enough to interrupt mid-scan for
|
||||
crash-resume tests).
|
||||
|
||||
---
|
||||
|
||||
## 6. Redis state is not reset between separate `pytest` invocations
|
||||
|
||||
**This is the single most likely cause of a mysterious, hard-to-reproduce
|
||||
e2e test failure in this repo, and it is easy to build a fixture that looks
|
||||
correct but doesn't actually protect against it.**
|
||||
|
||||
The simulated BEC deployment's Redis instance is a **long-lived shared
|
||||
resource** — it persists across separate `pytest` invocations, across a
|
||||
crashed test run that never reached its fixture's `finally` block, and
|
||||
across manual poking at the same sim session (e.g. someone debugging by
|
||||
hand against the same Redis). A fixture that only **snapshots the current
|
||||
value and restores it afterward** — which is the right pattern for not
|
||||
disturbing a real operator's concurrent session — does **not** protect
|
||||
against inheriting contamination that was already there *before* your test
|
||||
started.
|
||||
|
||||
Concretely: this bit the tomo-queue command-job e2e suite. A prior debugging
|
||||
session had left a stale job stuck at `status: "running"` in the
|
||||
`tomo_queue` global var. Every subsequent test that assumed it started from
|
||||
an empty queue failed in confusing, seemingly-unrelated ways — "queue not
|
||||
empty after a rejected add", `KeyError: 'kind'` on an old-format entry that
|
||||
had nothing to do with the test that hit it. The fix
|
||||
(`omny_e2e_tests/conftest.py`) is to **explicitly reset the relevant global
|
||||
var(s) to a known-empty value at fixture setup**, in addition to (not
|
||||
instead of) snapshotting and restoring around the test:
|
||||
|
||||
```python
|
||||
queue_snapshot = bec.get_global_var(_QUEUE_VAR)
|
||||
bec.set_global_var(_QUEUE_VAR, []) # <- the part that's easy to forget
|
||||
try:
|
||||
yield flomni
|
||||
finally:
|
||||
bec.set_global_var(_QUEUE_VAR, queue_snapshot)
|
||||
```
|
||||
|
||||
**Any new e2e fixture that reads/writes a global var must reset it to a
|
||||
known value at setup, not just restore whatever was there afterward.** If
|
||||
you hit a failure that doesn't make sense given the test's own logic, check
|
||||
for leftover state before assuming it's a real regression:
|
||||
`redis-cli get "user/vars/<the_global_var_key>"`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Checklist for adding a new e2e test file
|
||||
|
||||
1. Does a `build_<plugin>()`-style bootstrap already exist for this plugin?
|
||||
If not, build one following §4, working through §5's gotchas as you hit
|
||||
them (you will).
|
||||
2. Does the fixture reset every global var it touches to a known-empty value
|
||||
at setup (§6), not just snapshot/restore?
|
||||
3. If the test needs to kill a running operation or run two sessions
|
||||
concurrently, does it use a real subprocess (§5.1), not a thread?
|
||||
4. Is there a "fast" parameter preset so the test doesn't take minutes to
|
||||
run (§5.5)?
|
||||
5. Does the test file live under `omny_e2e_tests/` (§2), not `tests/`?
|
||||
6. Run it against a live sim session before calling it done — see
|
||||
`SIMULATED_ENDSTATIONS.md` for bringing one up. Nothing here is verified
|
||||
until it has actually run against the sim (or real instrument).
|
||||
+607
@@ -0,0 +1,607 @@
|
||||
# cSAXS Setups: LamNI, OMNY, FlOMNI – Vollständige technische Referenz
|
||||
|
||||
> **Quellen:** Quellcode `csaxs_bec` + vier peer-reviewed Publikationen:
|
||||
> - Holler et al., *J. Synchrotron Rad.* **27**, 730–736 (2020) → LamNI
|
||||
> - Holler et al., *Rev. Sci. Instrum.* **89**, 043706 (2018) → OMNY
|
||||
> - Holler et al., *Rev. Sci. Instrum.* **83**, 073703 (2012) → FlOMNI (RT-Vorgänger)
|
||||
> - Holler & Raabe, *Opt. Eng.* **54**, 054101 (2015) → Tracking-Interferometer
|
||||
|
||||
---
|
||||
|
||||
## ❗ WICHTIGE KORREKTUR: Was ist LamNI wirklich?
|
||||
|
||||
**LamNI = Laminographic Nano-Imaging** (NICHT "Laminar Nano-Imaging"!)
|
||||
|
||||
LamNI ist ein Instrument für **ptychographische Röntgen-Laminographie** (PyXL).
|
||||
Laminographie unterscheidet sich grundlegend von Tomographie:
|
||||
|
||||
| | Tomographie (OMNY) | Laminographie (LamNI) |
|
||||
|---|---|---|
|
||||
| Rotationsachse | ⊥ zum Röntgenstrahl (90°) | Geneigt zum Strahl (≠ 90°) |
|
||||
| Probenform ideal | Zylindrisch (Pfeiler/Pillar) | Planar (flache Chips, Schichten) |
|
||||
| FOV-Winkelbereich | 0°–180° (vollständig) | 0°–360° (voller Umlauf nötig) |
|
||||
| Fehlende Information | Keine (vollständig) | Missing cone (statt Wedge) |
|
||||
| Präparation | Destruktiv (FIB-Pillar nötig) | Minimal (großflächige Proben!) |
|
||||
| Anwendung bei PSI | OMNY | LamNI |
|
||||
|
||||
**Laminographiewinkel bei LamNI:** 61° (zwischen Rotationsachse und Röntgenstrahl)
|
||||
Erzeugt durch: Erst 15° Kippen um x-Achse, dann 60° um y-Achse.
|
||||
|
||||
**Das erklärt den Code direkt:**
|
||||
```python
|
||||
MOVEMENT_SCALE_X = np.sin(np.radians(15)) * np.cos(np.radians(30))
|
||||
MOVEMENT_SCALE_Y = np.cos(np.radians(15))
|
||||
```
|
||||
Die 15° entsprechen der Stage-Verkippung in x. Die Koordinatentransformation ist notwendig, weil der Piezo-Scanner in der geneigten Stage-Ebene bewegt, die Interferometer aber im Laborrahmen messen.
|
||||
|
||||
---
|
||||
|
||||
## 1. LamNI – Laminographic Nano-Imaging
|
||||
|
||||
### Physikalisches Prinzip
|
||||
|
||||
Kohärente Röntgenstrahlung (~6–10 keV) wird durch eine Fresnel-Zonenplatte (FZP) fokussiert. Die Probe wird in einem Fermat-Spiralmuster abgetastet (Ptychographie). Für die 3D-Information wird die Probe um die geneigte Achse (61° zur Strahlrichtung) rotiert. Das erlaubt die Abbildung **flacher, großflächiger Proben** (ICs, Schichten) ohne destruktive Pillar-Präparation.
|
||||
|
||||
**Anwendungen:**
|
||||
- Integrierte Schaltkreise (IC, 16-nm-Technologie-Knoten demonstriert: 18.9 nm Auflösung)
|
||||
- Magnetische Strukturen (zeitaufgelöste Laminographie, 50 nm / 70 ps)
|
||||
- Großflächige Materialien in 3D
|
||||
### Hardware-Komponenten (aus Paper 2020)
|
||||
|
||||
```
|
||||
LamNI Mechanik (110 × 130 cm Grundplatte, 1 Tonne Gewicht)
|
||||
│
|
||||
├── Röntgenoptik (upstream)
|
||||
│ ├── FZP – auf xyz-Stages (Steinmeyer Mechatronik, Schrittmotoren)
|
||||
│ ├── CS (Central Stop) – auf xy-Stages (SmarAct GmbH)
|
||||
│ ├── OSA (Order Sorting Aperture) – auf xyz-Stages (SmarAct GmbH)
|
||||
│ ├── Flachspiegel (x und y) für Interferometer am FZP-Halter
|
||||
│ └── Helium-Flugrohr (reduziert Luftstreuung)
|
||||
│
|
||||
└── Probenstage (gestapelt)
|
||||
├── Grobe Translationsstages (Steinmeyer Mechatronik, Schrittmotoren)
|
||||
│ └── Für Stitching und Region-of-Interest-Auswahl
|
||||
├── Großer Rotationstisch (air-bearing, 365°, Radius 175 mm)
|
||||
│ ├── Renishaw Tonic Dual-Encoder-System
|
||||
│ ├── Luftlager → nach Rotation abgelassen → mechanisch geklemmt → steifer
|
||||
│ └── Schrittmotor-Antrieb (Steinmeyer Mechatronik)
|
||||
├── Piezo-Scanner (Npoint NPXY100-216)
|
||||
│ ├── 2 Translationsachsen (x, y in Stage-Ebene)
|
||||
│ ├── 100 µm Stellbereich je Achse
|
||||
│ ├── Interne kapazitive Sensoren (nur für Vorpositionierung)
|
||||
│ └── Closed-loop zum exteroceptiven Interferometer (Hauptregelung!)
|
||||
└── Anti-Rotations-Stage (Kugellager, PASSIV – kein Antrieb!)
|
||||
├── Trägt die Interferometer-Spiegel
|
||||
├── Dreht sich NICHT mit dem Probentisch (hält Spiegel senkrecht)
|
||||
├── Passiver Antrieb: Hebelarm + zwei Hardstops am Rotationstisch
|
||||
├── Pneumatische Bremse: Fixiert Spiegel während Scan
|
||||
└── Kapazitive Sensoren (A: Abstand zu Hardstops, B: Winkelmotion)
|
||||
```
|
||||
|
||||
### Device-Namen in BEC (LamNI)
|
||||
|
||||
| BEC-Device | Hardware | Einheit | Anmerkung |
|
||||
|---|---|---|---|
|
||||
| `rtx` | RT-Controller X (Piezo-Scanner Npoint X-Achse) | µm | Interferometer-geregelt |
|
||||
| `rty` | RT-Controller Y (Piezo-Scanner Npoint Y-Achse) | µm | Interferometer-geregelt |
|
||||
| `lsamx` | Koarse-Motor X (Steinmeyer) | mm | Read-only bei aktivem Feedback |
|
||||
| `lsamy` | Koarse-Motor Y (Steinmeyer) | mm | Read-only bei aktivem Feedback |
|
||||
| `lsamrot` | Großer Rotationstisch (air-bearing) | Grad | Laminographie-Winkel |
|
||||
| `loptx` | Optik-Motor X | mm | FZP/OSA-Ausrichtung |
|
||||
| `lopty` | Optik-Motor Y | mm | |
|
||||
| `loptz` | Optik-Motor Z | mm | |
|
||||
|
||||
### Interferometer-System (LamNI)
|
||||
|
||||
- **Typ:** Doppeldurchgang-Flachspiegel (KEIN Tracking-Interferometer!)
|
||||
Grund: 12 mm × 12 mm Scanbereich zu groß für Tracking mit Kugelspiegeln
|
||||
- **Laser:** Zygo Inc., 6 mm Strahldurchmesser
|
||||
- **Gemessen:** Relative Position FZP ↔ Probe (x und y, exteroceptiv)
|
||||
- **Anti-Rotations-Stage** hält Spiegel senkrecht zum Interferometerstrahl während Rotation
|
||||
- **Positionsstabilität:** 1.1 nm Standardabweichung (x und y)
|
||||
- **Positioniergenauigkeit:** 4.4 nm (H), 4.1 nm (V) std (ptychographisch verifiziert)
|
||||
- **Schrittantwort:** < 40 ms für 1.5 µm Schritt
|
||||
### LamNI Geometrie im Detail
|
||||
|
||||
```
|
||||
Koordinatensystem (Labor):
|
||||
z = Röntgenstrahl-Richtung
|
||||
x = horizontal
|
||||
y = vertikal (zeigt nach oben)
|
||||
|
||||
Stage-Verkippung:
|
||||
1. 15° Kippung um x-Achse
|
||||
2. 60° Kippung um y-Achse
|
||||
→ Laminographie-Winkel (Strahl ↔ Rotationsachse) = 61°
|
||||
|
||||
Piezo-Scan-Ebene:
|
||||
= Probenebene (geneigt gegenüber Labor-Koordinaten)
|
||||
|
||||
Interferometer-Messung:
|
||||
= In Laborkoordinaten x und y
|
||||
|
||||
→ Koordinatentransformation notwendig:
|
||||
(Code: lamni_to_stage_coordinates / lamni_from_stage_coordinates)
|
||||
|
||||
Rotation im Code (Winkelkorrektur):
|
||||
alpha = (angle - 300 + 30.538) / 180 * pi
|
||||
→ Die 300° und 30.538° sind mechanische Offsets des Setups
|
||||
```
|
||||
|
||||
### Anti-Rotations-Mechanismus (LamNI-spezifisch)
|
||||
|
||||
```
|
||||
Problem: Interferometer-Spiegel müssen senkrecht zum Strahl bleiben,
|
||||
aber Probe dreht sich mit dem Rotationstisch!
|
||||
|
||||
Lösung: Anti-Rotations-Stage auf dem Piezo-Scanner
|
||||
1. Großer Rotationstisch dreht (z.B. 45°)
|
||||
2. Hebelarm der kleinen Stage berührt Hardstop an der Basis des großen Tisches
|
||||
3. Kleine Stage dreht sich passiv um -45°
|
||||
4. Ergebnis: Spiegel bleiben in Laborkoordinaten ausgerichtet ✓
|
||||
|
||||
Sicherheit:
|
||||
- Kapazitive Sensoren (A): Abstand Hebel ↔ Hardstop überwacht
|
||||
- Kapazitive Sensoren (B): Winkelmotion der kleinen Stage
|
||||
- Pneumatische Bremse: Fixiert Spiegel während Scan
|
||||
- Auto-Entklemmen wenn Hebel zu nah an Hardstop (Sicherheitsmechanismus)
|
||||
|
||||
Prozedur bei Rotationswechsel:
|
||||
1. Piezo auf interne kapazitive Sensoren zentrieren
|
||||
2. Interferometer-Feedback ausschalten
|
||||
3. Bremse der kleinen Stage lösen
|
||||
4. Luftlager einschalten
|
||||
5. Großen Tisch zum Zielwinkel drehen (Encoder-Feedback)
|
||||
6. Kleine Stage klemmen
|
||||
7. Feinausrichtung der kleinen Stage (kap. Sensoren A → Servo-Regelung)
|
||||
8. Luftlager ausschalten (erhöht Steifigkeit)
|
||||
9. Letzten Encoder-Wert speichern (tatsächlicher Winkel für Rekonstruktion)
|
||||
```
|
||||
|
||||
### Scan-Aufruf und Parameter (LamNI)
|
||||
|
||||
```python
|
||||
# Standard LamNI Fermat-Scan
|
||||
scans.lamni_fermat_scan(
|
||||
fov_size=[20], # FOV in Piezo-Ebene [µm], max ~80 µm
|
||||
# [x] = quadratisch, [x,y] = rechteckig
|
||||
step=0.5, # Schrittweite [µm]
|
||||
exp_time=0.1, # Belichtungszeit [s]
|
||||
angle=0, # Laminographie-Rotationswinkel [Grad]
|
||||
center_x=0.02, # Scan-Zentrum X bei 0° [mm] – WIRD rotiert!
|
||||
center_y=0, # Scan-Zentrum Y [mm]
|
||||
shift_x=0, # Zusatzshift X [mm] – wird NICHT rotiert
|
||||
shift_y=0, # Zusatzshift Y [mm]
|
||||
stitch_x=0, # Stitch-Versatz X [µm]
|
||||
stitch_y=0, # Stitch-Versatz Y [µm]
|
||||
fov_circular=0, # Kreisförmiges FOV [µm] (zusätzliches Cropping)
|
||||
stitch_overlap=1, # Stitch-Überlapp [µm]
|
||||
scan_type="fly", # "fly" (HW-getriggert) oder "step"
|
||||
frames_per_trigger=1,
|
||||
)
|
||||
|
||||
# Zu neuem Scan-Zentrum fahren (rotations-kompensiert):
|
||||
scans.lamni_move_to_scan_center(shift_x_mm, shift_y_mm, angle_deg)
|
||||
```
|
||||
|
||||
### LamNI Leistungsdaten
|
||||
|
||||
| Parameter | Wert |
|
||||
|---|---|
|
||||
| Scan-Bereich | 12 mm × 12 mm (über Stitching) |
|
||||
| Piezo-Bereich | 100 µm × 100 µm (Npoint NPXY100-216) |
|
||||
| Positionsstabilität | 1.1 nm std |
|
||||
| Schrittantwort | < 40 ms für 1.5 µm |
|
||||
| Positioniergenauigkeit | 4 nm std (ptychographisch) |
|
||||
| Beste 3D-Auflösung | 18.9 nm (IC 16-nm-Technologie) |
|
||||
| Gewicht | 1 Tonne |
|
||||
| Grundplatte | 110 × 130 cm |
|
||||
|
||||
---
|
||||
|
||||
## 2. OMNY – tOMography Nano crYo Stage
|
||||
|
||||
### Physikalisches Prinzip
|
||||
|
||||
OMNY = kryogene Ptychographie-Tomographie. Die Rotationsachse ist senkrecht zum Röntgenstrahl (klassische Tomographie, 90°). Kryogene Bedingungen (90 K mit LN₂, 10 K mit LHe) schützen strahlungsempfindliche Proben.
|
||||
|
||||
**Betrieb bei cSAXS seit:** Juni 2015
|
||||
**Vakuum:** Ultra-Hochvakuum (10⁻⁷–10⁻⁸ mbar)
|
||||
**Probengeometrie:** Zylindrische Pfeiler (OMNY Pins)
|
||||
|
||||
### Hardware-Komponenten (aus Paper 2018)
|
||||
|
||||
```
|
||||
OMNY Mechanik (Vakuumkammer Ø 110 cm, 4 Tonnen Gewicht)
|
||||
Mineral-Cast-Block: 110 × 110 × 52 cm³
|
||||
│
|
||||
├── Röntgenoptik
|
||||
│ ├── FZP – auf 2D Piezo-Stage (nPoint, Inc., NPXY100D)
|
||||
│ │ └── Controller: NPoint LC.403
|
||||
│ ├── CS (Central Stop) – auf xy-Stage (SmarAct SLC-2430-S-UHV)
|
||||
│ └── OSA – auf xyz-Stage; Cryo 2 gekühlt (90 K)
|
||||
│ └── Spiegel-Mounts: SmarAct STT-25.4-UHV-TI (motorisiert)
|
||||
│
|
||||
├── Probenstage
|
||||
│ ├── 4 Schrittmotor-Stages (xyz + Rotation um y)
|
||||
│ │ ├── Linear: ±5 mm Range (Steinmeyer Mechatronik, Dresden)
|
||||
│ │ ├── Rotation: 365°, Renishaw Tonic UHV Encoder
|
||||
│ │ └── Controller: 3× Galil DMC-4080-D4140
|
||||
│ ├── Delta-Scanner (Tripod-Geometrie, PSI-Eigenentwicklung, Maag et al.)
|
||||
│ │ ├── 3 translatorische DOF, 450 µm Gesamtbereich
|
||||
│ │ ├── Schrittantwort: < 30 ms für ~2 µm Schritte
|
||||
│ │ ├── Lokale Metrologie: attocube Faserinterferometer (nanometrisch!)
|
||||
│ │ ├── Piezo aufgeteilt in 4 unabhängige Stapel:
|
||||
│ │ │ ├── High-Current Stapel: VF-500 Verstärker (1A) – schnelle Bewegung
|
||||
│ │ │ └── Low-Noise Stapel: SVR-150 – Feinregelung (< 10 nm Rauschen)
|
||||
│ │ └── Rotationsmatrix-Korrektur (Delta-Geometrie ≠ xyz des Labors)
|
||||
│ └── Referenzspiegel / Probenhalter (Al, diamantgedreht, Au-beschichtet)
|
||||
│ ├── LT Ultra GmbH (Herstellung)
|
||||
│ ├── Vertikal: Flachfläche für Doppelpass-Interferometer
|
||||
│ ├── Horizontal: Kugelform für Tracking-Interferometer
|
||||
│ ├── Oberflächenfehler < 300 nm, Rauigkeit ~1 nm
|
||||
│ └── Sample direkt montiert (minimiert Dead-Path!)
|
||||
│
|
||||
├── Kryosystem
|
||||
│ ├── Cryo 1: Helitran LT3B (LN₂ oder LHe, offener Kreislauf, leise!)
|
||||
│ │ └── O₂-freies Kupfergeflecht (514 mm) → Referenzspiegel
|
||||
│ │ └── 4 × 16 mm² + 10 × 23 mm² (gestuft für Kraftreduktion)
|
||||
│ ├── Cryo 2: LN₂-Rohrsystem → OSA (90K), Greifer, Parkstation, Therm. Schirm (108K)
|
||||
│ │ └── Thermischer Schirm: SmarAct SLC-1750-S-UHV (kann weggefahren werden)
|
||||
│ └── Aktive Temperaturregelung: Pt100 (Heraeus) + Heizwicklungen (Nichrom)
|
||||
│ └── Stabilität bei LN₂: 90.000 K ± 3.3 mK (über Tage!)
|
||||
│
|
||||
├── Interferometer-System (Zygo Inc.)
|
||||
│ ├── Laser: Modell 7714, 3 mm Strahldurchmesser
|
||||
│ ├── Karten: Modell 4004 + 2400 (VME-Bus)
|
||||
│ ├── VME64-to-PCI Adapter: Abaco Systems Modell 810
|
||||
│ ├── 5 gemessene Größen:
|
||||
│ │ ├── xy: Probe ↔ OSA (Tracking-IF für x, Flach für y)
|
||||
│ │ ├── xy: OSA ↔ FZP (Doppeldurchgang-Flachspiegel)
|
||||
│ │ └── Rotationswinkel um z des Probentisches
|
||||
│ └── Tracking-Interferometer-Stage:
|
||||
│ ├── y-Piezo: Dynamic Structures ZSA-400-PSI (400 µm, 200 Hz)
|
||||
│ ├── z-Stepper + z-Piezo (50 µm): folgt Kugelspiegel-Wobble
|
||||
│ └── Bremse: Piezoelektrisch (erhöht Steifigkeit der y-Stage)
|
||||
│
|
||||
├── Elektronik (National Instruments)
|
||||
│ ├── NI 6259: Analog In/Out (2 Kanäle)
|
||||
│ ├── NI 6733: Analog Out (8 Kanäle, Delta-Scanner + FZP)
|
||||
│ └── NI 6602: Counter/Encoder (attocube-Quadratur-Encoder)
|
||||
│
|
||||
└── Load-Lock (Leica VCT100-basiert, modifiziert)
|
||||
├── Turbomolekularpumpe (nur Befüllung/Transfer)
|
||||
├── Bis 6 OMNY Pins pro Shuttle
|
||||
└── Probengreifer für kryogenen Transfer
|
||||
```
|
||||
|
||||
### OMNY Real-Time-Kontrollsystem (Fig. 7 im Paper)
|
||||
|
||||
```
|
||||
Host-PC: Ubuntu Linux + RTAI 4.0 (Echtzeit-Kernel)
|
||||
Schleifenfrequenz: 2.5 kHz
|
||||
Frühere Kontrolle: SPEC (Certified Scientific Software) → jetzt: BEC!
|
||||
|
||||
Vollständiger Signalfluss:
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ 2.5 kHz Loop │
|
||||
│ │
|
||||
│ Zygo-IF ──→ Slew-Rate-Limiter ──→ PID Sample xy │
|
||||
│ │ │ │ │
|
||||
│ │ Trajektorie-Gen. Rotationsmatrix │
|
||||
│ │ │ │
|
||||
│ │ PID Delta xy ──→ VF-500 │
|
||||
│ │ │ SVR-150 │
|
||||
│ Attocube ──→ Delta PID ──→ Range Extension │
|
||||
│ │ │
|
||||
│ Zygo FZP ──→ PID FZP xy ──→ NPoint LC.403 │
|
||||
│ │
|
||||
│ PSD Tracking ──→ PID Tracking yz ──→ ZSA-400-PSI Piezo │
|
||||
│ │
|
||||
│ Position-Sampler: Mean + Std jeder Achse während Belichtung│
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
Ethernet → SPEC/BEC (asynchron, nicht Echtzeit)
|
||||
EPICS soft-IOC: Probenpositionen, Temperaturen, Vakuum-Status
|
||||
```
|
||||
|
||||
### OMNY Mikroskop (X-Ray Eye)
|
||||
|
||||
Das interne Mikroskop (X-Ray Eye) hat 3 Modi:
|
||||
|
||||
1. **Weitwinkel** (Übersicht): Edmund Optics Objektiv + IDS UI-524xCP-C
|
||||
2. **Röntgen-Ast**: Szintillator (LuAG:Ce, 0.1 mm) → Mitutoyo 20x → IDS UI-548xCP-M
|
||||
- Optische Auflösung: 1 µm an der Szintillatorebene
|
||||
3. **Sichtbar-Ast**: Leica Plan APO + Leica Z16 APO → IDS UI-548xCP-M
|
||||
- Auflösung: 2.5 µm bei 400 nm, 66.4 mm vor Szintillator
|
||||
Das X-Ray Eye ist die Basis des **`xray_eye`** BEC-Widgets in `csaxs_bec/bec_widgets/widgets/xray_eye/`!
|
||||
|
||||
### Device-Namen in BEC (OMNY)
|
||||
|
||||
| BEC-Device | Hardware | Beschreibung |
|
||||
|---|---|---|
|
||||
| `rtx` | Delta-Scanner X / RT-Controller | Interferometer-geregelt, Flyer-Start |
|
||||
| `rty` | Delta-Scanner Y | |
|
||||
| `rtz` | Delta-Scanner Z | Z-Richtung (Defokussierung) |
|
||||
| `osamroy` | Rotationstisch | Tomographie-Winkel [Grad], 365° |
|
||||
| `osamx` | Koarse-Motor X | Proben-Grobpositionierung |
|
||||
| `rt_omny` | Sync-Flyer-Device | Überwacht Scan-Fortschritt |
|
||||
|
||||
### OMNY Alignment-Prozedur (aus Paper)
|
||||
|
||||
1. **Grob-Alignment**: X-Ray Eye Kamera + GUI (früher LabView, jetzt BEC)
|
||||
- 5 Rotationswinkel gleichmäßig verteilt von 0° bis 180°
|
||||
2. **Fein-Alignment**: 5 Ptychographie-Scans, sinusförmige Kurvenanpassung
|
||||
3. **Affine-Korrektur**: Einmalig pro Run (Flipping-Experiment + FSC-Optimierung)
|
||||
### OMNY Leistungsdaten
|
||||
|
||||
| Parameter | Wert |
|
||||
|---|---|
|
||||
| Probe-Stabilität | < 10 nm std (in-position während Belichtung) |
|
||||
| Positioniergenauigkeit | 8.1 nm (H), 4.7 nm (V) std residues |
|
||||
| Beste 2D-Auflösung | 8.5 nm (FRC, 1-bit Threshold) |
|
||||
| Beste 3D-Auflösung (IC) | 14.6 nm (Nature 2017) |
|
||||
| Beste 3D-Auflösung (biologisch, hochstreuend) | 27 nm (Käferflügelschuppe) |
|
||||
| Biologisch schwach streuend | 111 nm (Chlamydomonas in OMNY, LN₂) |
|
||||
| Probengröße max | ~50.000 µm³ Volumen |
|
||||
| Röntgenenergie | 6.2 keV typisch |
|
||||
| Detektor-Abstand | 7.33 m (Pilatus 2M) |
|
||||
| Temperaturstabilität | 90.000 K ± 3.3 mK (über Tage) |
|
||||
| Experimentlaufzeit | 5–6 Wochen kryogen (ohne Aufwärmen) |
|
||||
|
||||
---
|
||||
|
||||
## 3. FlOMNI – Fly-OMNY (Raumtemperatur, Fly-Scan)
|
||||
|
||||
### Was ist FlOMNI?
|
||||
|
||||
**FlOMNI** ist der **Raumtemperatur**-Nachfolger des ursprünglichen RT-Instruments (Holler 2012). Es ist als **Async Fly-Scan** (`AsyncFlyScanBase`) konzipiert: die Probe bewegt sich kontinuierlich, während die Detektoren laufend aufnehmen. Das 2012er Paper beschreibt den direkten Vorgänger.
|
||||
|
||||
**Grundprinzip des Closed-Loop-Schemas (aus 2012er Paper):**
|
||||
```
|
||||
1. Probe zu Zielposition bewegen (nur interne kapazitive Sensoren)
|
||||
2. Interferometer-Feedback an FZP-Piezo EINSCHALTEN
|
||||
3. Belichtung: FZP kompensiert verbleibende Drift (< 50 nm Amplitude)
|
||||
4. Interferometer-Mittelwert → Input für Ptychographie-Rekonstruktion
|
||||
5. Interferometer-Feedback AUSSCHALTEN, FZP-Piezo zurückzentrieren
|
||||
6. Nächste Position
|
||||
```
|
||||
|
||||
Bei FlOMNI ist dies für kontinuierliche Bewegung optimiert (Fly-Scan).
|
||||
|
||||
### FlOMNI Besonderheiten
|
||||
|
||||
| Merkmal | Wert/Beschreibung |
|
||||
|---|---|
|
||||
| Betrieb | Raumtemperatur, Atmosphäre |
|
||||
| Scan-Modus | Async Fly-Scan |
|
||||
| BEC-Basisklasse | `AsyncFlyScanBase` |
|
||||
| Tracker | **Laser-Tracker** (aktive Strahlverfolgung) |
|
||||
| Trigger | DDG1 (Delay-Generator 1), `EXT_RISING_EDGE` |
|
||||
| Cleanup | DDG1 zurück auf `SINGLE_SHOT` nach Scan |
|
||||
| Scan-Umkehr | Jeder 2. Scan reversed (Redis global var) |
|
||||
| FOV X max | 200 µm |
|
||||
| FOV Y max | 100 µm |
|
||||
| Z-Shift max | ±100 µm |
|
||||
| Probenmagazin | `flomni_sample_storage` – automatischer Probenwechsel! |
|
||||
| Rotation | `fsamroy` (FlOMNI Rotationsmotor) |
|
||||
| Flyer | `rt_positions` (AsyncFlyScanBase Monitor) |
|
||||
| Temp-Monitoring | `flomni_temp_and_humidity.py` |
|
||||
|
||||
### Laser-Tracker (FlOMNI-spezifisch)
|
||||
|
||||
Im Gegensatz zu OMNY (klassisches Tracking-Interferometer) hat FlOMNI einen Laser-Tracker:
|
||||
|
||||
```python
|
||||
# Im Scan-Code (flomni_fermat_scan.py):
|
||||
yield from self.stubs.send_rpc_and_wait("rtx", "controller.laser_tracker_on")
|
||||
tracker_signal = yield from self.stubs.send_rpc_and_wait(
|
||||
"rtx", "controller.laser_tracker_check_signalstrength"
|
||||
)
|
||||
# Ergebnis: "ok", "low", oder "toolow"
|
||||
|
||||
if tracker_signal == "low":
|
||||
# Alarm (WARNING), Scan kann weiterlaufen, Realignment empfohlen
|
||||
elif tracker_signal == "toolow":
|
||||
raise ScanAbortion("Laser-Tracker-Signal zu schwach – Realignment nötig!")
|
||||
|
||||
# Nach Positionierung zum Scan-Zentrum:
|
||||
yield from self.stubs.send_rpc_and_wait(
|
||||
"rtx", "controller.move_samx_to_scan_region", self.cenx
|
||||
)
|
||||
```
|
||||
|
||||
### Device-Namen in BEC (FlOMNI)
|
||||
|
||||
| BEC-Device | Beschreibung |
|
||||
|---|---|
|
||||
| `rtx` | RT-Controller X + Laser-Tracker-Interface |
|
||||
| `rty` | RT-Controller Y |
|
||||
| `rtz` | RT-Controller Z (Defokussierung) |
|
||||
| `fsamroy` | FlOMNI Rotationsmotor [Grad] |
|
||||
| `ddg1` | Delay-Generator 1 (Trigger, EPICS) |
|
||||
| `rt_positions` | Async-Flyer-Device |
|
||||
|
||||
---
|
||||
|
||||
## 4. Tracking-Interferometer – Gemeinsame Schlüsseltechnologie
|
||||
|
||||
Das **Error-Motion-Compensating Tracking Interferometer** (Holler & Raabe, Opt. Eng. 2015, Patent WO 2012079875 A1) ist Grundlage für OMNY (und den RT-Vorgänger/FlOMNI):
|
||||
|
||||
### Das Problem bei rotierenden Proben
|
||||
|
||||
Bei Standard-Interferometern mit Kugelspiegel muss der Kugelspiegel präzise auf der Rotationsachse zentriert sein. Das ist in der Praxis nicht möglich → Kugelspiegel wackelt → Strahl trifft nicht mehr auf Kugelmitte → Signal verloren.
|
||||
|
||||
**Lösung 1 (einfach):** Interferometer auf x/y-Stage, PSD misst Strahlposition → Tracking
|
||||
**Problem:** Tracking-Stage selbst hat Fehlermotion (Kugeellager: ~200 µrad Winkelfehler → ~3 µm Messfehler)
|
||||
|
||||
### Die optische Kompensation (Holler & Raabe 2015)
|
||||
|
||||
```
|
||||
Aufbau (vier polarisierende Strahlteiler PBS1-PBS4, zwei externe Referenzspiegel R1, R2):
|
||||
|
||||
Referenzstrahl: f1,f2 → PBS1 → R1 → PBS1 → PBS2 → R2 → PBS2 → HWP → PBS4
|
||||
Messstrahl: f1,f2 → PBS1 → PBS3 → Kugelspiegel → PBS3 → R2 → PBS4
|
||||
|
||||
Optische Wege:
|
||||
Referenz: 2·(a + b) [a = PBS1 zu R1, b = PBS2 zu R2]
|
||||
Messung: 2·(c + d) [c = PBS1 zu Sphäre, d = PBS3 zu R2]
|
||||
|
||||
OPD = 2·(a+b) - 2·(c+d)
|
||||
|
||||
Wenn Tracking-Stage sich um Δ bewegt (Translation):
|
||||
a,c kürzen sich um Δ; b,d verlängern sich um Δ → OPD unverändert ✓
|
||||
|
||||
Wenn Tracking-Stage sich dreht (Rotation um Zentrum):
|
||||
a,d kürzen sich; b,c verlängern sich → OPD unverändert ✓
|
||||
|
||||
R1 sitzt bei FZP-Position → direkte Differenzmessung Probe ↔ FZP!
|
||||
```
|
||||
|
||||
**PSD misst Kugelposition in x/y:**
|
||||
→ Closed-Loop zum Tracking → Kugel muss NICHT zentriert sein auf Rotationsachse!
|
||||
|
||||
**Ergebnis:**
|
||||
- Ohne Kompensation: ~3 µm Restfehler (Kugeellager-Winkelfehler)
|
||||
- Mit Kompensation: ~15 nm Restfehler (Faktor 200 besser!)
|
||||
- In OMNY verwendet: Positionsstabilität < 10 nm std
|
||||
### Warum LamNI kein Tracking-Interferometer braucht
|
||||
|
||||
Der LamNI-Scan ist 12 mm × 12 mm groß. Das überschreitet die praktikable Reichweite des Tracking-Systems mit Kugelspiegeln (begrenzt durch Tiefenschärfe des fokussierten Laserstrahls). Daher: Flachspiegel + Anti-Rotations-Stage als alternative Lösung.
|
||||
|
||||
---
|
||||
|
||||
## 5. nPoint Piezo-Controller – Zwei verschiedene Rollen!
|
||||
|
||||
| | Bei OMNY | Bei LamNI |
|
||||
|---|---|---|
|
||||
| **Modell** | NPXY100D | NPXY100-216 |
|
||||
| **Steuert** | **FZP-Position** (Röntgenoptik!) | **Probe** (Piezo-Scanner) |
|
||||
| **Controller** | NPoint LC.403 | im RT-Controller integriert |
|
||||
| **Bereich** | klein (Optik-Stabilisierung) | 100 µm × 100 µm |
|
||||
| **Zweck** | FZP in closed-loop zur Interferometermessung | Proben-Scanning für Ptychographie |
|
||||
|
||||
**BEC-Driver:** `csaxs_bec/devices/npoint/npoint.py`
|
||||
- Binary-Protokoll (LC.400/LC.403 Controller)
|
||||
- `0xA0 [addr] 0x55` = Lesen; `0xA2 [addr] [data] 0x55` = Schreiben
|
||||
- Positionsbereich: 0–100% → intern 0–1048574 Integerwert
|
||||
- 3 Achsen pro Controller
|
||||
---
|
||||
|
||||
## 6. RT-Controller (LamNI-spezifisch) – Socket-Protokoll
|
||||
|
||||
Der `RtLamniController` kommuniziert über TCP/IP Socket (ASCII-Kommandos):
|
||||
|
||||
| Kommando | Bedeutung |
|
||||
|---|---|
|
||||
| `J0` | Interferometer-Feedback deaktivieren |
|
||||
| `J4` | Aktuelle Interferometerposition lesen (x, y, ...) |
|
||||
| `J5` | Feedback aktivieren (ohne Reset der Zielposition) |
|
||||
| `J6` | Feedback + Winkelinterferometer-Reset deaktivieren |
|
||||
| `J2` | Feedback-Status + Interferometer-Signalstärke (SSI 0, SSI 1) |
|
||||
| `J3` | SSI-Mittelung triggern (vor J2 aufrufen!) |
|
||||
| `J7` | Winkel-Interferometer: Status, Position, Signalstärke |
|
||||
| `sc` | Scan stoppen + Trajektorie löschen |
|
||||
| `sd` | Punkt-für-Punkt-Scan starten |
|
||||
| `sr` | Scan-Status: (Mode, geplante Pos., aktuelle Pos.) |
|
||||
| `o` | Ist Achse am Ziel? (bool) |
|
||||
| `s{x:.5f},{y:.5f},0` | Position zum Scan-Buffer hinzufügen |
|
||||
| `a{angle_rad}` | Rotationswinkel setzen (rad, mit Mechanik-Offset) |
|
||||
| `V{um_per_s}` | Achsengeschwindigkeit setzen |
|
||||
| `V0` | Maximale Geschwindigkeit |
|
||||
| `As` / `Ar` | Analog-Sampling starten/lesen |
|
||||
| `Ss` / `Sr` | Positions-Sampling starten/lesen |
|
||||
| `pa0,{x}` | Zielposition Achse 0 setzen (für Feedback-Enable) |
|
||||
| `pa1,{y}` | Zielposition Achse 1 setzen |
|
||||
|
||||
**Scan-Status-Modi:**
|
||||
- Mode 0: Direkte Positionierung (kein Scan aktiv)
|
||||
- Mode 2: RT Point-Scan läuft
|
||||
- Mode 3: RT Point-Scan startet
|
||||
**Interferometer-Achsen** (aus `show_signal_strength_interferometer`):
|
||||
- Achse 0: "ST FZP horizontal" (Zonenplatte, horizontal)
|
||||
- Achse 1: "ST FZP vertical" (Zonenplatte, vertikal)
|
||||
- Achse 2: Winkel-Interferometer (Rotation)
|
||||
---
|
||||
|
||||
## 7. Galil Motor-Controller (OMNY)
|
||||
|
||||
**Typ:** Galil DMC-4080-D4140 (3 Einheiten für 19 Stages)
|
||||
**Encoder:** Renishaw Tonic UHV
|
||||
|
||||
Funktionen:
|
||||
- Closed-loop zum Encoder
|
||||
- Following-Error-Überwachung (Schritt- vs. Encoderposition)
|
||||
- Motortemperatur-Überwachung
|
||||
- Automatische Reichweitenerweiterung der Tracking-Piezo-Stages
|
||||
BEC-Treiber: `csaxs_bec/devices/omny/galil/`
|
||||
|
||||
---
|
||||
|
||||
## 8. Detektoren und Röntgenstrahl am cSAXS
|
||||
|
||||
### Röntgenstrahl
|
||||
- **Undulator:** In-Vakuum, 19 mm Periode
|
||||
- **Vertikale Quellgröße:** 20 µm
|
||||
- **Sekundärquelle:** Spalt 20 µm breit, 12.1 m downstream vom Undulator
|
||||
- **Monochromator:** Si(111) Doppelkristall (fixed-exit) + Fused-Silica-Spiegel
|
||||
- **Typische Energie:** 6.2 keV (λ = 0.2 nm)
|
||||
### Detektoren
|
||||
|
||||
| Detektor | Pixelgröße | Verwendung | Abstand (typisch) |
|
||||
|---|---|---|---|
|
||||
| Pilatus 2M | 172 µm | OMNY, LamNI | 7.33 m |
|
||||
| Pilatus 300K | 172 µm | cSAXS Standard | variabel |
|
||||
| Eiger 1.5M | 75 µm | Moderne OMNY/FlOMNI | variabel |
|
||||
| JungFrauJoch (JFJ) | – | Hochrate | variabel |
|
||||
| IDS UI-524xCP-C | – | OMNY Weitwinkel-Kamera | intern |
|
||||
| IDS UI-548xCP-M | – | OMNY Röntgen-Auge + Sichtbar | intern |
|
||||
|
||||
---
|
||||
|
||||
## 9. Gesamtvergleich der drei Setups
|
||||
|
||||
| Eigenschaft | LamNI | OMNY | FlOMNI |
|
||||
|---|---|---|---|
|
||||
| Methode | Ptych. Laminographie | Ptych. Tomographie (kryogen) | Ptych. Tomographie (RT) |
|
||||
| Rotationsachse | 61° geneigt | 90° (senkrecht) | 90° (senkrecht) |
|
||||
| Temperatur | Raumtemperatur | 90 K (LN₂) / 10 K (LHe) | Raumtemperatur |
|
||||
| Vakuum | Nein | UHV (10⁻⁸ mbar) | Nein |
|
||||
| Scan-Modus | Step (HW-getriggert) | Sync Fly-Scan | Async Fly-Scan |
|
||||
| BEC-Basisklasse | `ScanBase` | `SyncFlyScanBase` | `AsyncFlyScanBase` |
|
||||
| FOV (Scanning) | 12 mm × 12 mm (Stitch) | ~50 µm × 50 µm | 200 × 100 µm |
|
||||
| Piezo (Probe) | Npoint NPXY100-216 | Delta-Scanner (PSI) | RT-Scanner |
|
||||
| Piezo (FZP) | xyz-Stages (Steinm.) | Npoint NPXY100D + LC.403 | – |
|
||||
| Interferometer | Flachspiegel + Anti-Rotation | Tracking (Kugelspiegel) | Laser-Tracker |
|
||||
| Positionsstabilität | 1.1 nm | < 10 nm | < 10 nm |
|
||||
| Rotation BEC | `lsamrot` | `osamroy` | `fsamroy` |
|
||||
| Flyer-Device | – | `rt_omny` | `rt_positions` |
|
||||
| Gewicht | 1 Tonne | 4 Tonnen | ähnlich |
|
||||
| Beste Auflösung | 18.9 nm 3D | 8.5 nm 2D / 14.6 nm 3D | ~15 nm |
|
||||
| Probenform | Flach/großflächig | Zylindrisch (OMNY Pins) | Zylindrisch |
|
||||
| Probenmagazin | Nein | Load-Lock (6 Pins/Shuttle) | `flomni_sample_storage` |
|
||||
| Scan-Min. Punkte | 20 (ScanAbortion sonst) | 20 | 20 |
|
||||
|
||||
---
|
||||
|
||||
## 10. Kritische Erkenntnisse für Softwareentwicklung
|
||||
|
||||
1. **LamNI ist LAMINOGRAPHIE** (61° geneigt), nicht Tomographie – beeinflusst alle Positionsberechnungen
|
||||
2. **Koordinatentransformation** im Code physikalisch begründet durch Stage-Verkippung (15°/60°)
|
||||
3. **`rtx`/`rty`** = Piezo-Scanner in Interferometer-Koordinaten (µm), nicht Stage-Koordinaten
|
||||
4. **`lsamx`/`lsamy`** = Koarse-Motoren (mm), werden auf read-only gesetzt bei aktivem Feedback
|
||||
5. **nPoint** steuert bei OMNY die **FZP** (nicht Probe!), bei LamNI die **Probe**
|
||||
6. **OMNY** hat einen eigenen Real-Time-Kernel (RTAI 4.0, 2.5 kHz) – kommuniziert über Ethernet
|
||||
7. **Anti-Rotations-Stage** ist mechanische Besonderheit von LamNI (passiv, Hebelarm)
|
||||
8. **Tracking-Interferometer** (Holler & Raabe 2015) ist Schlüsseltechnologie für OMNY/FlOMNI
|
||||
9. **FlOMNI** hat Laser-Tracker (nicht klassisches Tracking-Interferometer)
|
||||
10. **DDG1** muss bei FlOMNI vor und nach Scan umgeschaltet werden (EXT_RISING_EDGE / SINGLE_SHOT)
|
||||
11. **Trajektorie-Umkehr** (FlOMNI + OMNY) wird in Redis persistent gespeichert
|
||||
12. Die Drift-Korrektur in LamNI läuft in bis zu **2 Iterationen**, Abbruch bei Drift > 150 µm
|
||||
13. **Delta-Scanner (OMNY)**: Tripod-Geometrie → Rotationsmatrix notwendig für xyz-Kontrolle
|
||||
14. **Attocube-Faserinterferometer**: Lokale Metrologie im Delta-Scanner (für schnelle PID-Schleife)
|
||||
15. **OMNY Affine-Korrektur**: Einmalig pro Experimentrun bestimmt (Flipping-Experiment + FSC)
|
||||
16. **OMNY Tracking-Stage y-Piezo** (ZSA-400-PSI, 400 µm, 200 Hz): ähnliche Eigenschaften wie Delta-Scanner – daher geeignet für kontinuierliche Bewegung während Ptychographie
|
||||
---
|
||||
|
||||
*Erstellt April 2026. Quellen: Quellcode csaxs_bec + 4 peer-reviewed Publikationen (Holler et al. 2012, 2015, 2018, 2020).*
|
||||
+5
-4
@@ -63,9 +63,10 @@ Protocol fidelity is grounded in the actual hardware-side sources: `fgalil.dmc`,
|
||||
`simulated_omny.yaml` (generated), `simulated_bl_endstation.yaml` (pre-existing)
|
||||
- `bin/generate_simulated_configs.py` — regenerates the LamNI/OMNY configs from
|
||||
`ptycho_lamni.yaml`/`ptycho_omny.yaml` (rerun after real-config changes)
|
||||
- `tests/sim_flomni_harness.py`, `tests/sim_lamni_omny_harness.py` — offline harnesses
|
||||
(no BEC deployment needed): 39 + 26 checks covering moves, referencing, feedback,
|
||||
tracker, scans through the flyers, gripper transfers, storage, cameras
|
||||
- `omny_e2e_tests/sim_flomni_harness.py`, `omny_e2e_tests/sim_lamni_omny_harness.py` —
|
||||
offline harnesses (no BEC deployment needed): 39 + 26 checks covering moves,
|
||||
referencing, feedback, tracker, scans through the flyers, gripper transfers,
|
||||
storage, cameras
|
||||
|
||||
## Real-code fixes made along the way (separate commits)
|
||||
|
||||
@@ -151,4 +152,4 @@ harness cannot see.
|
||||
`sim_stppermm`, `sim_encpermm`, `sim_referenced`, `sim_analog_inputs`,
|
||||
`sim_digital_inputs/outputs`, `sim_variables`, `sim_limit_low/high_active`,
|
||||
`sim_point_dwell_s`, `sim_noise_std`, `sim_samples`.
|
||||
- Harnesses run standalone: `python tests/sim_flomni_harness.py` (repo on PYTHONPATH).
|
||||
- Harnesses run standalone: `python omny_e2e_tests/sim_flomni_harness.py` (repo on PYTHONPATH).
|
||||
@@ -0,0 +1,752 @@
|
||||
# Plan: Command Jobs in the FlOMNI Tomo Queue
|
||||
|
||||
**Status: Steps 1–3 are implemented and fully validated**, both by sim and by a
|
||||
live click-through of the entire GUI checklist. **Step 2** (`flomni.py`: schema
|
||||
type-tagging, action registry, `move` + `optimize_idgap` (stub) actions,
|
||||
`tomo_queue_add_command`, `tomo_queue_show`/`tomo_queue_execute` branching,
|
||||
`_run_command_job` resume policy, `tomo_queue_actions` global-var publishing, plus
|
||||
the matching `flomni_webpage_generator.py` fix) is sim-tested:
|
||||
`omny_e2e_tests/test_tomo_queue_command_jobs.py`, 6/6 passing (see
|
||||
`TOMO_QUEUE_TESTING.md` §4D); real device movement exercised against `ftray`
|
||||
(sim-testable stand-in), not `mokev`/`idgap` (real front-end only — see §10). Since
|
||||
then: a CLI-level `tomo_queue_move(index, new_index)` alongside
|
||||
`tomo_queue_add`/`_delete`/`_clear`, enforcing the same running-job floor as §6.4;
|
||||
and a curated, session-only `at_each_angle_hook` registry (§3.4) with a GUI dropdown
|
||||
and webpage display, sim-validated (CLI side) and click-tested (GUI side, both green
|
||||
— see `TOMO_QUEUE_GUI_TESTING.md` sections 6b–6f). **Step 3** (the `TomoQueueDialog`
|
||||
GUI work) — the command-job builder dialog (§6.5), command-job-aware queue
|
||||
rendering, the status-keyed delete/clear guards from §6.3 (since extended with a
|
||||
staleness-aware override, see below), a "Sort queue…" mode for mid-run reorder of
|
||||
pending rows (§6.4 — up/down buttons, not drag-and-drop, see the note under §6.4 for
|
||||
why), full-detail row tooltips, "Load into editor" (§6.8), and the
|
||||
live-vs-unsaved-edit warnings on both "Add to queue" surfaces (§6.1) — **has been
|
||||
fully click-tested against a live session** (`TOMO_QUEUE_GUI_TESTING.md`, items
|
||||
1–55, all passing). Two real incidents were found and fixed along the way: a job
|
||||
stuck at `running` forever with no live process behind it (§6.3's staleness note,
|
||||
§3.4's declined-confirmation note), and the two-window live-vs-unsaved-edit
|
||||
confusion (§6.1). Step 1 (executor rework) is implemented and its sim checklist has
|
||||
passed (`TOMO_QUEUE_TESTING.md` §4, sections A–C). Remaining known gaps (scope cuts,
|
||||
not bugs) are tracked in `TOMO_QUEUE_GUI_TESTING.md` §7.
|
||||
|
||||
**Prerequisite (met):** the checklist in `TOMO_QUEUE_TESTING.md` has passed against
|
||||
the simulated flOMNI. That document is the companion to this one: it covers *what
|
||||
Step 1 changed and how to test it*; this one covers *what comes next*. The e2e test
|
||||
suite for both steps lives in `omny_e2e_tests/` at the repo root (moved out of
|
||||
`tests/e2e/` so it doesn't get swept into gitea's generic `tests/` CI discovery —
|
||||
these tests need a live sim + Redis, not just `pytest`).
|
||||
|
||||
> **This document supersedes the earlier version of the same name.** The earlier
|
||||
> draft assumed the old index-based executor, single-action command jobs, and left
|
||||
> the registry-distribution question open. All three have since been decided.
|
||||
|
||||
**Goal:** Allow the tomo queue to hold, in addition to tomogram jobs, "command" jobs
|
||||
that reconfigure the beamline between tomograms — move a device, change energy, move
|
||||
optics in/out, optimize a parameter — so one queue can express *"tomogram A, then
|
||||
reconfigure, then tomogram B"*, unattended.
|
||||
|
||||
---
|
||||
|
||||
## 0. Chosen model, and what was rejected
|
||||
|
||||
**Chosen: a named-action registry.** Command jobs name actions from a curated
|
||||
allow-list, plus keyword arguments. Everything stays JSON-serializable, auditable in
|
||||
`client.get_global_var("tomo_queue")`, and safe to persist across kernel restarts.
|
||||
|
||||
**Rejected: storing Python source strings and `exec`-ing them later.** Not auditable;
|
||||
fails at run time (hours into an overnight run) rather than at add time; breaks the
|
||||
crash-resume model. If the registry ever can't express something, adding an entry is a
|
||||
small, reviewed change — preferable to a permanent arbitrary-code channel on a live
|
||||
beamline.
|
||||
|
||||
**Also rejected: a code editor in the GUI with a syntax check.** Syntax-valid is nearly
|
||||
worthless as a safety check — `umv(dev.fsamx, -1e9)` parses perfectly and drives the
|
||||
stage into a hard stop. A syntax check gives the operator *false confidence*. The
|
||||
structured builder (§4) is both safer and friendlier: there is nothing to type wrong.
|
||||
|
||||
---
|
||||
|
||||
## 1. Current state
|
||||
|
||||
### Step 1 — DONE (executor rework)
|
||||
|
||||
`tomo_queue_execute()` is now an **id-based pick-next loop**: it re-reads the queue
|
||||
from its global var at the start of every job and addresses jobs by a stable `id`
|
||||
(not by list index). `_TomoQueueProxy` gained `update_by_id()` and `ensure_ids()`;
|
||||
`tomo_queue_add()` stamps a uuid on each job.
|
||||
|
||||
This is what makes it safe to **append, delete a pending job, and reorder the pending
|
||||
tail while the queue runs** — the foundation the GUI work (§5) depends on. It also
|
||||
introduced **resume-before-fresh** ordering: any `incomplete`/`running` job runs before
|
||||
any fresh `pending` one, protecting the single shared `progress` global var.
|
||||
|
||||
See `TOMO_QUEUE_TESTING.md` §1 for the full rationale and the invariants to preserve.
|
||||
|
||||
### Job dict schema today
|
||||
|
||||
```python
|
||||
{
|
||||
"id": str, # uuid4 hex; stable across reorder/edit
|
||||
"label": str,
|
||||
"params": {<_TOMO_QUEUE_PARAM_NAMES>: value, ...},
|
||||
"status": "pending" | "running" | "incomplete" | "done",
|
||||
"added_at": ISO datetime string,
|
||||
}
|
||||
```
|
||||
|
||||
Every job today is *implicitly* a tomogram. Adding command jobs = making the job
|
||||
**type-tagged** and branching in the executor.
|
||||
|
||||
### The GUI as it stands
|
||||
|
||||
`TomoQueueDialog` (in `tomo_params.py`) reads/writes the same `tomo_queue` global var
|
||||
directly, renders one row per job, colour-codes by status, and polls every 2 s.
|
||||
Execution is CLI-only (the GUI server process can't run the blocking scan loop).
|
||||
|
||||
---
|
||||
|
||||
## 2. Schema: type-tagged jobs
|
||||
|
||||
Add a `"kind"` field. **Backward compatible:** a job with no `kind` is treated as
|
||||
`"tomo"`, so existing persisted queues and existing GUI code keep working.
|
||||
|
||||
### Tomo job (unchanged except the tag)
|
||||
```python
|
||||
{
|
||||
"kind": "tomo", # NEW; absent => "tomo"
|
||||
"id": str,
|
||||
"label": str,
|
||||
"params": {...},
|
||||
"status": ...,
|
||||
"added_at": ...,
|
||||
}
|
||||
```
|
||||
|
||||
### Command job (new) — a **list of steps**, not a single action
|
||||
```python
|
||||
{
|
||||
"kind": "command",
|
||||
"id": str,
|
||||
"label": str,
|
||||
"steps": [ # ordered; run in sequence within the one job
|
||||
{"action": "move", "kwargs": {"positions": {"mokev": 6.2}}},
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 0.5}},
|
||||
],
|
||||
"idempotent": bool, # true iff EVERY step is idempotent
|
||||
"status": ...,
|
||||
"added_at": ...,
|
||||
}
|
||||
```
|
||||
(`foptics_out`/`foptics_in`-style steps aren't in the initial registry — see §3 —
|
||||
this example only uses the two actions Step 2 actually ships with.)
|
||||
|
||||
**Why a list.** "Between tomogram A and B: change energy, re-peak idgap" is one
|
||||
*intent* and should be one queue row — it reads correctly, and it is atomic to reason
|
||||
about. Composing it from three separate queue rows is worse on both counts.
|
||||
|
||||
Notes:
|
||||
- `kwargs` must be JSON-serializable. No device objects, no callables — actions resolve
|
||||
device names to `dev.*` themselves at execution time.
|
||||
- Steps within a job are **independent**: none of them consumes another's runtime
|
||||
result. Anything result-dependent belongs *inside* a single action (§3.2).
|
||||
|
||||
---
|
||||
|
||||
## 3. The action registry
|
||||
|
||||
A class-level dict on `Flomni` mapping an action name to a spec. Callables are existing
|
||||
`Flomni`/mixin methods or small reviewed wrappers, so the queue can only ever invoke
|
||||
reviewed beamline operations.
|
||||
|
||||
**Each registry entry has four things:**
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `func` | name of the bound method to call |
|
||||
| `idempotent` | default: safe to re-run after a crash? |
|
||||
| `help` | short operator-facing description |
|
||||
| `params` | **schema** of the action's arguments (possibly empty) |
|
||||
|
||||
**Decided with Mirko: the initial registry is deliberately minimal — `move` and
|
||||
`optimize_idgap` only.** No `foptics_in`/`foptics_out`/`feye_in`/`feye_out`/shutter
|
||||
action for now, even though the first three already exist as callable methods —
|
||||
Step 2 ships with move-on-allow-listed-devices plus one compound action; further
|
||||
one-click actions are "dedicated scripts" to be written and added later as their own
|
||||
reviewed changes, not part of this rollout. No dedicated energy-change wrapper either
|
||||
— energy changes go through `move` on `mokev` (see §3.1), not a special action.
|
||||
|
||||
```python
|
||||
_TOMO_QUEUE_ACTIONS = {
|
||||
"move": {
|
||||
"func": "_queue_action_move",
|
||||
"idempotent": True, # absolute moves re-run harmlessly
|
||||
"help": "Move device(s) to absolute position(s).",
|
||||
"params": {
|
||||
"positions": {"type": "device_positions"}, # {device: target}, allow-listed
|
||||
},
|
||||
},
|
||||
# compound actions — see §3.2
|
||||
"optimize_idgap": {
|
||||
"func": "optimize_idgap",
|
||||
"idempotent": True,
|
||||
"help": "Scan idgap over a search range and move to the peak.",
|
||||
"params": {
|
||||
"search_range": {
|
||||
"type": "float", "default": 0.5, "min": 0.0, "max": 2.0,
|
||||
"unit": "mm", "label": "Search range",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Still open (confirm before/while implementing `optimize_idgap`):** which
|
||||
signal/device the scan reads to locate "the peak" — deferred by Mirko to the
|
||||
implementation session, not resolved yet. The `search_range` default/bounds above
|
||||
are confirmed as "roughly right" placeholders; unit (`mm`) not explicitly
|
||||
reconfirmed but not contradicted either.
|
||||
|
||||
### 3.1 Curated device allow-list (NOT "any device in `dev`")
|
||||
|
||||
The `move` action may only touch devices on an explicit, reviewed allow-list — the
|
||||
things you actually reconfigure between tomograms. "Any device in `dev`" is the
|
||||
difference between *a reconfiguration tool* and *a remote control for the whole
|
||||
endstation*.
|
||||
|
||||
**Decided with Mirko: `mokev` (energy) and `idgap` (undulator gap) — nothing else
|
||||
yet.** Neither device is in the flomni sim config (`simulated_flomni.yaml`); both
|
||||
live in `bl_frontend.yaml` (shared beamline front-end devices), so `move` on either
|
||||
is only exercisable against the real front end, not the flomni sim used for
|
||||
`TOMO_QUEUE_TESTING.md`.
|
||||
|
||||
```python
|
||||
_TOMO_QUEUE_MOVE_DEVICES = {
|
||||
"mokev": {"label": "Energy", "unit": "keV"},
|
||||
"idgap": {"label": "Undulator gap", "unit": "mm"}, # unit unconfirmed, see above
|
||||
}
|
||||
|
||||
def _queue_action_move(self, positions: dict):
|
||||
"""positions: {device_name: absolute_target, ...}. Absolute umv only."""
|
||||
args = []
|
||||
for name, target in positions.items():
|
||||
if name not in self._TOMO_QUEUE_MOVE_DEVICES:
|
||||
raise ValueError(f"Queue move: device '{name}' is not queue-movable.")
|
||||
if name not in dev:
|
||||
raise ValueError(f"Queue move: unknown device '{name}'.")
|
||||
args += [dev[name], target]
|
||||
umv(*args)
|
||||
```
|
||||
|
||||
The GUI's device dropdown is built from this allow-list, so an operator cannot even
|
||||
select a device that isn't sanctioned.
|
||||
|
||||
### 3.2 Compound actions, and where result-dependent logic lives
|
||||
|
||||
**Rule: anything where step B needs step A's *runtime result* is ONE named action, not
|
||||
several queue steps.**
|
||||
|
||||
The motivating case is "scan idgap, move to peak". That is not three primitives — it is
|
||||
one compound operation with an internal result. It becomes a single registry entry
|
||||
(`optimize_idgap`) whose method does the scan, finds the peak, and moves there. The
|
||||
cleverness lives in reviewed Python; the queue only ever holds the *intent*:
|
||||
|
||||
```python
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 0.5}}
|
||||
```
|
||||
|
||||
Why this matters: the queue then **never has to carry runtime values between steps**, so
|
||||
`kwargs` stays plain JSON and crash-resume stays honest. Trying to express "move to
|
||||
wherever the peak turned out to be" as separate queue steps is exactly what would force
|
||||
a code channel.
|
||||
|
||||
**Compound actions may take parameters** (`search_range` above) — declared in the same
|
||||
`params` schema as any other action. Parameters are *scalar inputs the operator sets at
|
||||
add time*: numbers, bools, a choice from a fixed list, a device from the allow-list. The
|
||||
schema is deliberately **not** expressive enough to encode logic, expressions, or
|
||||
references to other jobs' results. That limitation is the point — a parameter is a value
|
||||
you can validate at add time.
|
||||
|
||||
### 3.3 Validation is two-layer
|
||||
|
||||
- **Schema validation at add time** (GUI + `tomo_queue_add_command`): types, min/max,
|
||||
device on the allow-list, OK disabled until valid. This is for *ergonomics* — catch
|
||||
the typo now, not at 2am.
|
||||
- **Method validation at execution time**: each action re-checks its own arguments
|
||||
against live hardware. This is the *safety boundary*, because the schema and the
|
||||
hardware can disagree (limits changed; a value arrived via the CLI bypassing the
|
||||
form). A refusal here trips the executor's `incomplete`-and-pause path, which is the
|
||||
correct outcome.
|
||||
|
||||
### 3.4 A separate, session-only registry: `at_each_angle_hook`
|
||||
|
||||
A different escape hatch predates the command-job registry: `Flomni._at_each_angle()`
|
||||
already checks for a `flomni_at_each_angle` function in `builtins.__dict__` and, if
|
||||
present, calls it instead of the default per-projection acquisition — the motivating
|
||||
case being something like "record a projection, move a polarizer in, record again,
|
||||
move it back out." That hook is arbitrary code, which is fine for a one-off
|
||||
interactive run, but it isn't a snapshotted parameter, so a **queued** tomo job has no
|
||||
way to say "use this hook for this job." Decided with Mirko: add it as a curated
|
||||
registry, same spirit as `_TOMO_QUEUE_ACTIONS` but distinct from it —
|
||||
|
||||
- **Curated here means "queue stores a name, never code," not "hardcoded in
|
||||
flomni.py."** Hooks are registered from the IPython session at runtime via
|
||||
`Flomni.register_at_each_angle_hook(name, func)` into a plain instance dict
|
||||
(`self._at_each_angle_hooks`), *not* published as a class-level registry the way
|
||||
`_TOMO_QUEUE_ACTIONS` is. This keeps the "any hook a job references is a Python
|
||||
callable that was deliberately named and registered, never raw code sitting in the
|
||||
queue" invariant, while still letting operators write ad hoc hooks per experiment
|
||||
(unlike `move`/`optimize_idgap`, these aren't pre-reviewed methods shipped in
|
||||
`flomni.py`).
|
||||
- **`at_each_angle_hook` is a new entry in `_TOMO_SCAN_PARAM_NAMES`** (a
|
||||
`_GlobalVarParam(None)`, defaulting to no hook) rather than a new job `kind` or
|
||||
queue mechanism — it's just another snapshotted tomo parameter, so
|
||||
`tomo_queue_add()`/`tomo_queue_execute()` restore it per job for free, same as
|
||||
`fovx` or `tomo_countingtime`.
|
||||
- **Session-only, and that's surfaced loudly.** Hooks live in memory, not a global
|
||||
var (functions aren't JSON-serializable), so they don't survive a kernel restart.
|
||||
`_at_each_angle()` raises a clear `FlomniError` — not a silent fallback to default
|
||||
behaviour — if a job's `at_each_angle_hook` name isn't registered in the session
|
||||
currently running the queue.
|
||||
- **Found and fixed a real param-snapshot leak in the GUI while wiring this up.**
|
||||
`tomo_params.py`'s `QUEUE_PARAM_NAMES`/`DEFAULTS` mirror `_TOMO_SCAN_PARAM_NAMES`
|
||||
by hand (the same drift risk `tomo_queue_actions` was built to avoid for command
|
||||
jobs — see section 7). Before adding `at_each_angle_hook` to that mirror, a
|
||||
GUI-added job's `params` dict would have no `at_each_angle_hook` key at all, so
|
||||
the executor's `for name, value in job["params"].items(): setattr(...)` loop would
|
||||
never touch it — silently leaving whatever hook the *previous* queued job had
|
||||
active in place for a job that never asked for one. Fixed by adding the key to
|
||||
`QUEUE_PARAM_NAMES`/`DEFAULTS`, and by having `add_edited_to_queue()` explicitly
|
||||
pass through the live value (there's no editable widget for it yet).
|
||||
- **GUI widget added.** `Flomni._publish_at_each_angle_hooks()` republishes
|
||||
registered names as `tomo_at_each_angle_hooks` on every register/unregister call
|
||||
(reset to empty in `__init__`, since hooks are session-only), mirroring
|
||||
`tomo_queue_actions`'s no-hardcoded-mirror approach (section 7). `TomoParamsWidget`
|
||||
reads it into a "At-each-angle hook" dropdown, rebuilt on every populate so it
|
||||
tracks CLI-side register/unregister calls live. The currently-set value is always
|
||||
shown, even if it isn't in the published list (a different or since-restarted
|
||||
session registered it), labeled "(not registered in this session)" instead of
|
||||
silently vanishing. Special-cased in `_populate_fields()`/`_read_fields()` like
|
||||
`tomo_type` — the generic `QComboBox` handling assumes integer combo data, which
|
||||
doesn't fit a hook-name string or `None`.
|
||||
- **Webpage display added.** `TQ_PARAM_DISPLAY`/`_CURRENT_PARAM_KEYS` are both
|
||||
hardcoded param-key mirrors (queue job detail, and the live "Current measurement"
|
||||
section respectively) — `at_each_angle_hook` didn't crash either path when absent,
|
||||
but also didn't show up. Added to `_CURRENT_PARAM_KEYS` (Python, gated on the value
|
||||
being set) and rendered as a conditional row in `buildParamRows()` (JS) — left out
|
||||
of `TQ_PARAM_DISPLAY` deliberately, so a normal job doesn't grow a "-" row nobody
|
||||
needs; it only appears when a hook is actually active.
|
||||
- See `docs/user/ptychography/flomni.md`'s Tomography section for the full
|
||||
operator-facing writeup (what a hook function looks like, how to register/activate/
|
||||
unregister/list one).
|
||||
|
||||
**Sim-validated** (against the live simulated flOMNI, not just mocked): registration/
|
||||
list/unregister bookkeeping; the params snapshot carrying `at_each_angle_hook` per
|
||||
job (and correctly resetting to `None` for the next one); the hook actually firing
|
||||
during a real `tomo_queue_execute()` run; an unregistered hook's job ending up
|
||||
`incomplete` and resuming cleanly once the hook is re-registered. Found and fixed one
|
||||
real bug in the process, unrelated to the hook logic itself: `_tomo_scan_at_angle` is
|
||||
wrapped in `@scan_repeat(max_repeats=10, default=True)`, which retried the new
|
||||
`FlomniError` (unregistered hook) 10 times before giving up with a generic
|
||||
`TooManyScanRestarts` — the actual error message never reached the operator. Fixed
|
||||
with an `exc_handler` that skips the retry for any `FlomniError` (see the
|
||||
`_retry_unless_flomni_error` docstring in `flomni.py`); this benefits every other
|
||||
`FlomniError` the method can raise, not just the hook-registry one.
|
||||
|
||||
Also sim-validated (a follow-up pass, adding the GUI/webpage wiring): the
|
||||
`tomo_at_each_angle_hooks` global var updates correctly on register/unregister and
|
||||
is sorted; the webpage generator's `current_params` payload includes
|
||||
`at_each_angle_hook` when set and omits it (rather than sending `null`) when not.
|
||||
**Not** sim/click-tested: the actual `QComboBox` in `TomoParamsWidget` (headless
|
||||
environment, no `pytest-qt`) or the rendered webpage HTML/JS — see
|
||||
`TOMO_QUEUE_GUI_TESTING.md` for the manual checklist covering both.
|
||||
|
||||
**Second real incident, found at the beamline: a declined confirmation silently did
|
||||
nothing.** A hook called `tomo_scan_projection(angle)` without `_internal=True` while
|
||||
`single_point_instead_of_fermat_scan` was on. That combination hits
|
||||
`tomo_scan_projection()`'s own "Run a fermat scan anyway?" confirmation (unrelated to
|
||||
the hook mechanism itself, but only reachable in practice through a hook or a direct
|
||||
CLI call) — at *every* projection angle, since the hook runs once per angle.
|
||||
Declining used to just print "Aborted." and `return`, with no exception -- so nothing
|
||||
signaled the executor that a projection was skipped. The eventual `Ctrl-C` (a
|
||||
`KeyboardInterrupt`, not caught by `tomo_queue_execute()`'s `except Exception`) left
|
||||
the job stuck at `running` forever, which is the same failure mode the row-lock note
|
||||
in §6.3 now documents. Fixed at the source: declining now raises `FlomniError`
|
||||
instead of returning silently, so — combined with the `scan_repeat` `exc_handler` fix
|
||||
above — a decline correctly fails the job immediately and marks it `incomplete`
|
||||
rather than either silently dropping data or leaving an untraceable stuck state.
|
||||
Sim-verified: a direct declined call raises with a clear message; through the queue,
|
||||
via a hook missing `_internal=True`, the job ends up `incomplete`, not stuck
|
||||
`running` or silently `done`.
|
||||
|
||||
---
|
||||
|
||||
## 4. New / changed CLI methods (`flomni.py`)
|
||||
|
||||
### `tomo_queue_add_command(steps, label=None, idempotent=None)`
|
||||
|
||||
Validate every step's `action` against `_TOMO_QUEUE_ACTIONS`; validate its `kwargs`
|
||||
against the action's `params` schema (including the device allow-list for `move`);
|
||||
confirm JSON-serializability (round-trip `json.dumps`); resolve `idempotent`
|
||||
(explicit arg > **all steps idempotent**); append a `kind="command"` job with a fresh
|
||||
uuid. Return the new index. Mirror the print style of `tomo_queue_add`.
|
||||
|
||||
Convenience for the common single-step case is fine (`steps` may accept a single dict),
|
||||
but the stored schema is always a list.
|
||||
|
||||
```python
|
||||
flomni.tomo_queue_add_command(
|
||||
[{"action": "move", "kwargs": {"positions": {"mokev": 6.2}}},
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 0.5}}],
|
||||
label="reconfigure to 6.2 keV",
|
||||
)
|
||||
```
|
||||
|
||||
### `tomo_queue_add(...)` — unchanged behaviour
|
||||
Keep as-is, but write `"kind": "tomo"` explicitly going forward (new jobs
|
||||
self-describing; old ones without the field still default to tomo on read).
|
||||
|
||||
### `tomo_queue_show()` — render both kinds
|
||||
Branch on `kind`. Command rows: show the step sequence compactly + an idempotency
|
||||
marker. Keep column alignment so the two kinds read as one list.
|
||||
|
||||
```
|
||||
[2] pending reconfigure 6.2keV CMD move{mokev:6.2} > optimize_idgap{search_range:0.5} [idem]
|
||||
```
|
||||
|
||||
### `tomo_queue_execute()` — branch on kind
|
||||
Inside the existing **pick-next loop** (do not reintroduce an index loop), after
|
||||
selecting the job and determining `resume_job`:
|
||||
|
||||
```python
|
||||
kind = job.get("kind", "tomo")
|
||||
self._tomo_queue_proxy.update_by_id(job_id, status="running")
|
||||
try:
|
||||
if kind == "command":
|
||||
self._run_command_job(job, resume_job)
|
||||
else: # "tomo"
|
||||
for name, value in job["params"].items():
|
||||
setattr(self, name, value)
|
||||
if resume_job:
|
||||
self.tomo_scan_resume()
|
||||
else:
|
||||
self.tomo_scan()
|
||||
except Exception as exc:
|
||||
self._tomo_queue_proxy.update_by_id(job_id, status="incomplete")
|
||||
... # existing pause/hint/raise
|
||||
self._tomo_queue_proxy.update_by_id(job_id, status="done")
|
||||
```
|
||||
|
||||
Note the tomo branch reads `job["params"]` — the command branch must never touch it
|
||||
(command jobs have no `params` key).
|
||||
|
||||
### `_run_command_job(job, resume_job)`
|
||||
Apply the resume policy (§5), then run each step in order: resolve the action spec,
|
||||
`getattr(self, spec["func"])`, call with `**step["kwargs"]`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Crash-resume semantics
|
||||
|
||||
Tomo jobs resume mid-scan via `tomo_scan_resume()` because progress lives in global
|
||||
vars. Command jobs have no such state. Policy:
|
||||
|
||||
**No per-step resume cursor.** Deliberately: tracking which step of a multi-step job
|
||||
completed means more persisted state and more edge cases than this warrants.
|
||||
|
||||
**The job-level `idempotent` flag decides:**
|
||||
|
||||
- `idempotent == True` (all steps idempotent — absolute moves, in/out, a scan-to-peak):
|
||||
silently **re-run the whole step sequence from the top**. Re-running an absolute move
|
||||
or an in-out is harmless.
|
||||
- `idempotent == False` (any relative/stateful step): **do not silently re-run.** Stop
|
||||
and prompt the operator (`OMNYTools.yesno`): *"Command job '<label>' may have
|
||||
partially run before the crash. Re-run it? (y/n)"*. On "no", mark it `done` and
|
||||
continue; on "yes", re-run from the top.
|
||||
|
||||
Edge case handled transparently: a command job that *completed* but crashed before its
|
||||
status hit `done` (stuck at `running`). Idempotent → re-runs harmlessly. Non-idempotent
|
||||
→ asks, which is the safe default.
|
||||
|
||||
**Write actions to be idempotent wherever possible.** Anything relative or stateful
|
||||
(e.g. `bec.queue.next_dataset_number += 1`, feedback toggles) must be marked
|
||||
`idempotent=False` if ever added.
|
||||
|
||||
---
|
||||
|
||||
## 6. GUI (`tomo_params.py` / `TomoQueueDialog`)
|
||||
|
||||
The agreed interaction model. Its whole purpose: **the GUI stays usable while a
|
||||
tomogram is running** — you can queue more work without touching the running scan.
|
||||
A first iteration against this model is now built — see `TOMO_QUEUE_GUI_TESTING.md`
|
||||
for what's actually implemented vs. still open per subsection below.
|
||||
|
||||
### 6.1 The load-bearing rule: two separate write paths
|
||||
|
||||
| control | writes | allowed while a scan runs? |
|
||||
|---|---|---|
|
||||
| **Done** | the **live param global vars** | **NO — must be blocked** |
|
||||
| **Add to queue** (new) | **only** the `tomo_queue` var | **yes, always** |
|
||||
|
||||
This separation is the entire reason the GUI can be used during a run. The scan
|
||||
**re-reads the live param global vars on every projection** (`fovx`, `fovy`,
|
||||
`tomo_shellstep`, `tomo_countingtime`, `frames_per_trigger`, `stitch_x/y`,
|
||||
`corridor_size`, …), so writing them mid-scan *perturbs the running acquisition*.
|
||||
"Add to queue" must never touch them.
|
||||
|
||||
**"Add to queue" cannot call the CLI `tomo_queue_add()`** — that snapshots the *live*
|
||||
vars and would ignore the operator's unapplied form edits. The dialog must assemble the
|
||||
job dict itself: edited form fields, plus current live values for the params the form
|
||||
doesn't expose, so the snapshot is still a complete `_TOMO_QUEUE_PARAM_NAMES` set.
|
||||
|
||||
Gate "Done" on the same busy signal `TomoParamsWidget` already computes (tomo heartbeat
|
||||
OR BEC scan-queue status).
|
||||
|
||||
**Implemented as designed, eventually.** The first GUI iteration shipped "Done" (named
|
||||
`Submit` in the actual widget) as a soft warn — a Yes/Cancel confirmation dialog while
|
||||
busy, not a block — because there was no "Add to queue" path yet for an in-progress
|
||||
edit, and blocking Submit outright with nowhere else to put those edits would have been
|
||||
a pure regression. A later pass added `TomoParamsWidget.add_edited_to_queue()` (a new
|
||||
"Add to queue" button next to Submit, visible only in edit mode) that assembles the job
|
||||
dict from the current form fields — exactly per this section's "must assemble the job
|
||||
dict itself" rule — and only then did `submit_params()` become the intended hard block.
|
||||
|
||||
**The same live-vs-unsaved-edit confusion has two doors, not one.** Found at the
|
||||
beamline: `TomoQueueDialog.add_to_queue()` (the "Add current params to queue" button
|
||||
in the *separate* ☰ Queue control… window) *also* reads live vars via
|
||||
`self._load_params()`, same as `tomo_queue_add()` always did — it was never updated to
|
||||
know the params panel might have an unsubmitted edit open. An operator who edits a
|
||||
value, doesn't click the panel's own "Add to queue", and instead reaches for the queue
|
||||
window's button gets the stale pre-edit values queued with no indication anything's
|
||||
off — the exact trap this section's "must assemble the job dict itself" rule was
|
||||
written to avoid, just reachable from a second window that rule didn't originally
|
||||
cover. Fixed with a warning (not a block, unlike Submit): `TomoQueueDialog.add_to_queue()`
|
||||
now checks `self.parent()._edit_mode` and, if set, confirms before proceeding, naming
|
||||
the panel's own "Add to queue" as the correct button. A warning rather than a hard
|
||||
block because queuing the live params while an unrelated edit happens to be open
|
||||
elsewhere is still legitimate — unlike Submit's case, there's no live-scan-perturbation
|
||||
risk here, just a wording/expectation mismatch worth flagging rather than forbidding.
|
||||
|
||||
### 6.2 No in-place row editing
|
||||
|
||||
Editing a queued job = **delete it, add a new one, reorder into place**. One
|
||||
construction path for jobs; no snapshot-vs-live ambiguity for existing rows.
|
||||
|
||||
### 6.3 Row locks keyed on `status` (not on the global busy signal)
|
||||
|
||||
Use the *row's own* status, because a bare `flomni.tomo_scan()` leaves no queue row
|
||||
marked `running` while a queue run marks exactly one — so per-row status is the precise
|
||||
"this is the live item" signal.
|
||||
|
||||
- `running`: **not draggable, not deletable.** Deletable only after the measurement is
|
||||
stopped (it is then `incomplete`).
|
||||
- `pending` / `incomplete`: freely draggable and deletable.
|
||||
- `done`: pinned (harmless to move, but it makes the list confusing to read).
|
||||
|
||||
**Real incident: `running` isn't always the live item.** The rule above assumes
|
||||
*something* transitions a `running` job onward -- either the scan finishes, or an
|
||||
exception marks it `incomplete`. Neither happens if the process dies hard enough to
|
||||
skip both (a `KeyboardInterrupt`, which `tomo_queue_execute()`'s `except Exception`
|
||||
does not catch since `KeyboardInterrupt` isn't an `Exception`; a kernel restart). The
|
||||
job is then stuck at `running` forever with no live process behind it, and the
|
||||
GUI's delete/clear guard -- keyed purely on that status string -- blocked it
|
||||
unconditionally, with no escape from the GUI at all (the CLI's `tomo_queue_delete()`
|
||||
has no such guard and already worked, but that isn't discoverable from the GUI).
|
||||
Fixed by checking `tomo_progress`'s heartbeat (same signal/window as
|
||||
`TomoParamsWidget._is_tomo_running()`, 120 s): a `running` job with a stale or
|
||||
missing heartbeat is very likely orphaned, so delete/clear offer an explicit
|
||||
"looks stale -- proceed anyway?" confirmation instead of refusing outright. A
|
||||
genuinely fresh heartbeat still blocks as before -- this narrows the guard's blind
|
||||
spot, it doesn't remove the guard.
|
||||
|
||||
### 6.4 Reorder is allowed **mid-run**
|
||||
|
||||
It takes effect at the next job boundary: the executor finishes the running job, marks
|
||||
it `done`, re-reads the queue, and picks the first `pending` in the *current* order. No
|
||||
interruption, no index math.
|
||||
|
||||
**Reject any drop that would move a row above the running row.** The running row is a
|
||||
hard floor for drops — otherwise the list would read as if a pending job runs "before"
|
||||
the thing already in progress.
|
||||
|
||||
**Implemented as a "Sort queue…" toggle mode with up/down buttons, not free-form
|
||||
drag-and-drop.** A checkable "Sort queue…" button puts the dialog into a focused
|
||||
reorder mode: other queue-mutating buttons (add/add command/delete/clear) disable,
|
||||
the table becomes single-select, and "▲ Move up" / "▼ Move down" swap the selected
|
||||
row with its neighbour. A swap is only ever between two `pending` rows — a `running`,
|
||||
`incomplete`, or `done` neighbour blocks the swap outright, which gets the
|
||||
running/incomplete floor and the done-row pinning (§6.3) for free without separate
|
||||
index-range bookkeeping. Reaching a position more than one row away takes multiple
|
||||
clicks, unlike drag-and-drop, but sidesteps implementing "reject any drop above the
|
||||
running row" as live drag-event math. Writes go straight to the `tomo_queue` global
|
||||
var, same pattern as add/delete/clear (§6.1's two-write-paths rule doesn't apply here
|
||||
— reordering never touches the live param global vars). The CLI got the equivalent
|
||||
`tomo_queue_move(index, new_index)`, so both surfaces can reorder the pending tail.
|
||||
|
||||
**`incomplete` is a floor too, not just `running`.** The first cut treated
|
||||
`incomplete` as movable alongside `pending`, reasoning that resume-before-fresh
|
||||
already protects the real execution order regardless of list position. In practice
|
||||
that let an operator drag a `pending` job visually ahead of an `incomplete` one —
|
||||
functionally harmless (the `incomplete` job still resumes first), but confusing to
|
||||
read, since list order looked like it meant run order and didn't. Fixed by narrowing
|
||||
`_MOVABLE_STATUSES` to `("pending",)` in the GUI and the equivalent floor check in
|
||||
`tomo_queue_move()` (blocks any `new_index` at or below the highest index among all
|
||||
`running`/`incomplete` jobs, not just the running one).
|
||||
|
||||
### 6.5 The command-job builder — a structured form, not a text box
|
||||
|
||||
An **"Add command…"** control opens a sub-dialog:
|
||||
|
||||
- an ordered **step list** (add / remove / reorder steps before confirming),
|
||||
- per step: an **action dropdown** populated from the registry, then **only the fields
|
||||
that action declares** in its `params` schema — for `move`, a device dropdown (from
|
||||
the allow-list) + numeric target; for `optimize_idgap`, a `search_range` field
|
||||
pre-filled with its default; for parameterless actions, nothing,
|
||||
- numeric fields validated live (schema `min`/`max`, and the device's real
|
||||
`dev.<name>.limits` where applicable) — **OK disabled until every step is valid**,
|
||||
- a label field and an `idempotent` checkbox pre-filled from the steps' registry
|
||||
defaults.
|
||||
|
||||
On confirm, append a `kind="command"` job in exactly the format the CLI writes, so CLI
|
||||
and GUI stay interchangeable.
|
||||
|
||||
### 6.6 Unchanged
|
||||
|
||||
Execution stays **CLI-only** (`flomni.tomo_queue_execute()`); the dialog keeps its
|
||||
"execute from CLI" hint. The 2 s poll timer and `closeEvent`/`showEvent` logic are
|
||||
untouched. Keep the existing `fsamroy` setup guard — command actions are
|
||||
FlOMNI-specific; don't expose command-add in non-FlOMNI sessions.
|
||||
|
||||
### 6.7 Row detail tooltips and column sizing
|
||||
|
||||
The table originally gave all 9 columns equal (`Stretch`) width, which squeezed
|
||||
Label and Details — the two free-text columns — down to the same width as narrow
|
||||
fixed-content columns like "Type" or "Exp (s)". Fixed this two ways:
|
||||
|
||||
- **Per-column resize modes.** Short columns (`#`, Status, Type, Projections,
|
||||
Exp (s), Step (µm)) use `ResizeToContents`; Label and Details use `Stretch`, so
|
||||
they split whatever width the short columns don't need. Left-aligned instead of
|
||||
centered now that they're wide free-text columns, not short fixed-width ones.
|
||||
- **Dropped the "Added at" column** entirely — one less column competing for
|
||||
space, and it's rarely the thing an operator needs to scan at a glance.
|
||||
|
||||
Neither eliminates truncation for a very long command summary or label, so every
|
||||
cell in a row still carries a tooltip with the full job detail — label, status,
|
||||
added-at, and either the complete params dump (tomo job) or the numbered step list
|
||||
plus idempotent flag (command job). Hovering any cell in the row shows it.
|
||||
|
||||
### 6.8 Reusing a queued job's settings — "Load into editor"
|
||||
|
||||
Motivating case: a tomogram from the queue finished, and later the operator wants
|
||||
to rerun something similar. Select a single **tomo** job (command jobs have no
|
||||
`params`, so they're excluded from selection eligibility) in the queue dialog and
|
||||
click **"Load into editor"** to enter edit mode on the params panel, populated from
|
||||
that job's saved snapshot instead of the live backend.
|
||||
|
||||
**Deliberately not a third write path.** This only calls `_populate_fields()` with
|
||||
the job's `params` instead of the live values `enter_edit_mode()` would normally
|
||||
load — nothing is written anywhere by loading. From there, the operator uses the
|
||||
same two paths every other edit already uses: Submit (writes live, hard-blocked
|
||||
while busy, §6.1) or "Add to queue" (queue-only, always allowed). No new safety
|
||||
gating needed because none of the existing gating is bypassed.
|
||||
|
||||
**Guards against silently discarding an in-progress edit.** If the panel is already
|
||||
in edit mode when "Load into editor" is clicked, a confirmation dialog fires first
|
||||
(naming the job being loaded, if it has a label) — a stray click on a queue row
|
||||
would otherwise throw away unsaved changes with no warning.
|
||||
|
||||
Implementation: `enter_edit_mode()` was refactored to funnel through a new private
|
||||
`_enter_edit_mode_with(params)` (the actual field-populate/enable/button-swap
|
||||
logic), so `load_params_into_editor()` is a thin wrapper adding only the
|
||||
params-source and confirm-before-discard behavior. `TomoQueueDialog` gates the
|
||||
button's enabled state on "exactly one row selected, and it's a tomo job, and not
|
||||
in sort mode" — re-evaluated on selection change and on every table refresh (a
|
||||
selected row's underlying job can change identity if the queue was edited from
|
||||
elsewhere while it stayed selected).
|
||||
|
||||
---
|
||||
|
||||
## 7. Registry distribution: publish via global var
|
||||
|
||||
On `Flomni.__init__`, write a serializable description of `_TOMO_QUEUE_ACTIONS` +
|
||||
`_TOMO_QUEUE_MOVE_DEVICES` (names, params schemas, idempotent defaults, help) to a
|
||||
`tomo_queue_actions` global var. The GUI reads it and builds its dropdowns and forms
|
||||
from it.
|
||||
|
||||
**Decided: this, not a static mirror.** A hardcoded mirror in the widget would drift —
|
||||
exactly the problem `QUEUE_PARAM_NAMES` already has between `flomni.py` and
|
||||
`tomo_params.py`. With the global var, a new action is a registry entry plus the method;
|
||||
the GUI grows the right fields with **zero widget changes**.
|
||||
|
||||
---
|
||||
|
||||
## 8. Web page generator (`flomni_webpage_generator.py`)
|
||||
|
||||
Confirm exactly what it renders for the queue/progress when implementing. It must not
|
||||
break on a job with `kind="command"` and **no `params` key**:
|
||||
|
||||
- Any code doing `job["params"][...]` will `KeyError` — guard with
|
||||
`job.get("kind", "tomo") == "tomo"` before touching `params`.
|
||||
- Add a sensible rendering for command jobs (steps summary + status), consistent with
|
||||
how tomo jobs are shown.
|
||||
- Re-check any "projections remaining" / ETA computation so command jobs (which have no
|
||||
projections) don't skew tomogram-based estimates.
|
||||
|
||||
---
|
||||
|
||||
## 9. Validation & delivery
|
||||
|
||||
Per house rules (see `TOMO_QUEUE_TESTING.md` §0): `ast.parse` + no **new** `black`
|
||||
hunks (`--line-length=100 --skip-magic-trailing-comma`) on every edited file; minimal
|
||||
diffs; Conventional Commits; **sim validation is the gate**.
|
||||
|
||||
Behavioural checks (mock first, then sim):
|
||||
|
||||
- mixed queue (tomo, command, tomo) runs in order; actions dispatched with correct kwargs
|
||||
- a job dict **without** a `kind` field still runs as a tomogram (back-compat)
|
||||
- multi-step command job runs its steps in order
|
||||
- idempotent resume re-runs from the top; non-idempotent resume prompts
|
||||
- `tomo_queue_show`, the GUI dialog, and the web generator all survive a command job
|
||||
with no `params` key
|
||||
- a `move` to a device **not** on the allow-list is refused at add time *and* at
|
||||
execution time
|
||||
|
||||
### Suggested commit breakdown
|
||||
1. `feat(flomni): type-tag tomo queue jobs (kind field, back-compat default)`
|
||||
2. `feat(flomni): add command-job action registry, device allow-list and _run_command_job`
|
||||
3. `feat(flomni): tomo_queue_add_command + show/execute branching`
|
||||
4. `feat(flomni): publish action registry via tomo_queue_actions global var`
|
||||
5. `feat(bec_widgets): add-to-queue button and command-job builder in TomoQueueDialog`
|
||||
6. `feat(bec_widgets): status-keyed row locks in TomoQueueDialog` (reordering split
|
||||
out to #8 — shipped later than the locks)
|
||||
7. `fix(flomni): guard webpage generator against command jobs without params`
|
||||
8. `feat(flomni): add tomo_queue_move for reordering pending/incomplete jobs`
|
||||
9. `feat(bec_widgets): sort-mode reorder buttons and full-detail row tooltips in
|
||||
TomoQueueDialog`
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions — resolved with Mirko, one detail still TBD
|
||||
|
||||
**Resolved (this round):**
|
||||
|
||||
1. **The final action set**: `move` + `optimize_idgap` only, deliberately minimal.
|
||||
No `foptics_in`/`foptics_out`/`feye_in`/`feye_out`/shutter in Step 2, despite the
|
||||
first three already existing as callable methods — explicitly deferred to future
|
||||
"dedicated scripts" not yet written. No energy-change wrapper — energy changes go
|
||||
through `move` on `mokev`.
|
||||
2. **The curated device allow-list** for `move`: `mokev` and `idgap`. Both live in
|
||||
`bl_frontend.yaml`, not the flomni sim config — `move` on either is only
|
||||
exercisable against the real front end.
|
||||
3. `optimize_idgap`'s `search_range` default/bounds (0.5mm default, 0–2mm) confirmed
|
||||
as roughly right placeholders.
|
||||
4. `move` kwargs shape: nested `{"positions": {name: target}}`, as specced.
|
||||
|
||||
**Still open — deferred by Mirko to the implementation session, not blocking on it
|
||||
now:**
|
||||
|
||||
- What signal/device `optimize_idgap` should scan/read to locate "the peak".
|
||||
- `idgap`'s unit (`mm` assumed in §3.1, not explicitly reconfirmed).
|
||||
|
||||
Resolved since the earlier draft: registry distribution (global var, §7);
|
||||
non-idempotent resume behaviour (prompt, §5); multi-step jobs (yes, §2); ordering
|
||||
constraints (jobs are independent and run in order; the executor's resume-before-fresh
|
||||
rule handles the one real hazard).
|
||||
@@ -0,0 +1,575 @@
|
||||
# Tomo Queue GUI — Step 3, First Iteration: What to Test, and How to Continue
|
||||
|
||||
**Read this first if you're picking this work back up**, whether that's you
|
||||
(Mirko) at the beamline, or a fresh Claude Code session with no memory of
|
||||
building it. Section 8 explains exactly how to hand this off to a new
|
||||
session/chat.
|
||||
|
||||
**Status: fully validated.** Every item in sections 3 through 6f (1–55) has
|
||||
been clicked through against a live session by Mirko and passes — the
|
||||
complete GUI checklist for this iteration is green. Along the way, real
|
||||
testing surfaced and got fixed: the old soft-warn Submit (now a hard
|
||||
block, item 4/8); a confusing duplicate row-number column, hook visibility
|
||||
gaps across `tomo_parameters()`/the queue table/scilog/the PDF
|
||||
report/the progress ring (section 6c, items 37–41); a stuck-`running`
|
||||
job with no live process behind it, caused by a declined confirmation
|
||||
that used to silently return instead of raising (fixed at the source,
|
||||
plus a staleness-aware delete/clear override — section 6e, items
|
||||
48–52, confirmed against an actual stuck job found live); and a second
|
||||
live-vs-unsaved-edit confusion reachable from the separate Queue
|
||||
control window, not just the params panel (section 6f, items 53–55).
|
||||
No open items remain from this checklist. If new GUI/queue work starts,
|
||||
extend this document rather than starting a new one — the fixture/setup
|
||||
sections (1, 2, 8) still apply. Companion docs:
|
||||
`TOMO_QUEUE_COMMAND_JOBS_PLAN.md` section 6 (the design this implements),
|
||||
`TOMO_QUEUE_TESTING.md` (Steps 1 & 2, already sim-validated).
|
||||
|
||||
---
|
||||
|
||||
## 1. What was built
|
||||
|
||||
All changes are in `csaxs_bec/bec_widgets/widgets/tomo_params/tomo_params.py`.
|
||||
The pre-existing `TomoParamsWidget`/`TomoQueueDialog` baseline (param editing,
|
||||
busy banner, basic queue add/delete/clear, 2 s polling) was already there
|
||||
before this iteration — see `TOMO_QUEUE_COMMAND_JOBS_PLAN.md`'s "The GUI as
|
||||
it stands" for what that baseline covers. This iteration adds command-job
|
||||
support on top of it:
|
||||
|
||||
1. **`CommandJobBuilderDialog`** (new class) — the structured builder from
|
||||
plan section 6.5. Opened via a new **"Add command…"** button in
|
||||
`TomoQueueDialog`. Reads the live `tomo_queue_actions` global var (the
|
||||
registry `Flomni.__init__` publishes) so its action dropdown and
|
||||
per-action fields always match whatever's actually in `flomni.py` — no
|
||||
hardcoded mirror. Lets you add steps in order (action + its own fields
|
||||
only — a device dropdown and target for `move`, a bounded spinbox for
|
||||
`optimize_idgap`'s `search_range`), remove a step, reorder steps within
|
||||
the job (move up/down), and set a label + idempotent flag (defaults to
|
||||
"every step idempotent" ANDed, same as the CLI, until you touch the
|
||||
checkbox yourself). On confirm it produces a job dict byte-for-byte
|
||||
compatible with what `flomni.tomo_queue_add_command()` writes.
|
||||
2. **Command-job-aware queue table.** `TomoQueueDialog`'s table gained a
|
||||
"Details" column; a command job shows `CMD` in the Type column and its
|
||||
step summary + idempotency marker in Details (e.g.
|
||||
`move{'positions': {'mokev': 6.2}} > optimize_idgap{'search_range': 0.5} [idem]`
|
||||
— literally the same string `flomni.tomo_queue_show()` prints).
|
||||
3. **Self-describing tomo jobs.** "Add current params to queue" now stamps
|
||||
`"kind": "tomo"` and a fresh `"id"` on the job it writes, matching what
|
||||
the CLI's `tomo_queue_add()` has done since Step 1/2 (previously the GUI
|
||||
wrote jobs with neither field — harmless, since the executor treats a
|
||||
kind-less job as `"tomo"` and heals missing ids on the next
|
||||
`tomo_queue_execute()`, but no longer necessary).
|
||||
4. **Status-keyed delete/clear guards** (plan section 6.3). Deleting a
|
||||
selection that includes a `"running"` job, or clearing the queue while
|
||||
any job is `"running"`, now refuses with an explanatory message instead
|
||||
of silently doing it.
|
||||
5. **"Sort queue…" mode** (plan section 6.4) — a checkable button that
|
||||
enters a focused reorder mode: add/add-command/delete/clear disable, the
|
||||
table becomes single-select, and "▲ Move up" / "▼ Move down" swap the
|
||||
selected row with its neighbour. A swap only ever happens between two
|
||||
`pending` rows — a `running`, `incomplete`, or `done` neighbour blocks
|
||||
it, which is what keeps the running/incomplete job as a floor and done
|
||||
jobs pinned. (`incomplete` was movable in an earlier pass; narrowed to a
|
||||
floor after testing showed a pending job could be dragged visually ahead
|
||||
of an incomplete one — harmless to actual run order, since
|
||||
resume-before-fresh always resumes it first regardless of position, but
|
||||
confusing to read.) Not drag-and-drop (see section 7's old note on why
|
||||
that was deferred) — this is the up/down-button alternative. The CLI got
|
||||
the same capability via `flomni.tomo_queue_move(index, new_index)`.
|
||||
6. **Full-detail row tooltips.** Every cell in a queue row now carries a
|
||||
tooltip with the complete job detail — label, status, added-at, and
|
||||
either the full params dump (tomo job) or the numbered step list +
|
||||
idempotent flag (command job) — so hovering the row surfaces what the
|
||||
Label/Details columns truncate at typical dialog widths.
|
||||
7. **Column sizing.** Label and Details now stretch to fill available width
|
||||
(left-aligned) instead of sharing equal `Stretch` width with narrow
|
||||
columns like Type/Exp (s)/Step (µm), which now auto-fit their content
|
||||
instead. The "Added at" column was dropped — that value is still in the
|
||||
row tooltip (item 6).
|
||||
8. **"Add to queue" while editing, and a real Submit block.** Edit mode now
|
||||
shows an **"Add to queue"** button next to Submit. It packages the
|
||||
current (possibly unsubmitted) form fields as a new `pending` tomo job
|
||||
and writes only `tomo_queue` — never the live param vars — so it's
|
||||
always allowed, scan running or not (plan section 6.1). `Submit` is now
|
||||
a real hard block while the beamline is busy instead of a Yes/Cancel
|
||||
confirmation dialog, since there's finally somewhere else to put an
|
||||
in-progress edit. `flomni.py` also got a small, unrelated robustness fix
|
||||
in the same pass: `tomo_scan()`/`scilog_last_ptycho_scans()`/the
|
||||
alignment-scan scilog entry now go through a new `_scilog_write()`
|
||||
helper that tolerates scilog being disabled instead of raising
|
||||
`RuntimeError` and pausing the whole queue over a logging side effect
|
||||
(mirrors the existing enabled-check already used for the PDF report
|
||||
entry).
|
||||
9. **"At-each-angle hook" dropdown.** A new combo box in the params panel,
|
||||
sourced from the live `tomo_at_each_angle_hooks` global var (published
|
||||
by `flomni.register_at_each_angle_hook()`/`unregister_at_each_angle_hook()`
|
||||
— no hardcoded mirror). Rebuilt every time the panel populates, so hooks
|
||||
registered/unregistered from the CLI while the dialog is open show up
|
||||
without reopening it. The currently-set hook always appears in the list
|
||||
even if it isn't currently registered anywhere (labeled "(not registered
|
||||
in this session)"), so it's never silently hidden. Selecting one and
|
||||
clicking Submit or "Add to queue" writes/snapshots it exactly like any
|
||||
other tomo parameter. The status webpage also picked up a matching
|
||||
display: a queued job's active hook (if any) shows in its expanded
|
||||
detail, and the live "Current measurement" section shows it too when a
|
||||
scan with a hook active is running outside the queue. CLI-side
|
||||
(registry, param snapshot, execution, crash-resume) sim-validated
|
||||
against a live session — see `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` section
|
||||
3.4. The dropdown and webpage rendering themselves have **not** been
|
||||
click-tested — see section 6b below.
|
||||
10. **Fixes from real 6b testing.** The queue table no longer shows two
|
||||
different-based row indices (Qt's own 1-based gutter is hidden; only
|
||||
the dialog's 0-based `#` column, matching CLI indices, remains). A tomo
|
||||
job's Details column now shows `hook: <name>` (or `... (not
|
||||
registered)`) when one is set — previously always blank for tomo jobs.
|
||||
`tomo_parameters()`, the end-of-scan scilog entry, `write_pdf_report()`
|
||||
(both the PDF file and the scilog entry it sends), and
|
||||
`flomnigui_show_progress()`'s ring label all now show the active hook's
|
||||
name, flagged if it isn't currently registered — the scilog entry and
|
||||
PDF report additionally include the hook's actual source code when it
|
||||
is still registered, so what a tomogram did is part of the permanent
|
||||
record. Click-tested (section 6c, items 37–41) and passing.
|
||||
11. **"Load into editor."** Select a single tomo job in the queue dialog
|
||||
(command jobs are excluded — no `params`) and click **"Load into
|
||||
editor"** to enter edit mode on the params panel populated from that
|
||||
job's saved settings instead of live values. Loading never writes
|
||||
anything itself — Submit/"Add to queue" from there work exactly as
|
||||
they already do. Confirms before discarding an in-progress edit.
|
||||
Disabled outside a valid single-tomo-job selection and during sort
|
||||
mode. Click-tested (section 6d, items 42–47) and passing.
|
||||
12. **Declined-scan failure + stale "running" recovery.** Two fixes for a
|
||||
real stuck-queue incident: `tomo_scan_projection()` now raises instead
|
||||
of silently returning when the "run a fermat scan anyway?" confirmation
|
||||
is declined, so a queue job fails cleanly (`incomplete`) instead of
|
||||
silently dropping a projection or (after an eventual Ctrl-C) getting
|
||||
stuck at `running` with no live process. And delete/clear on a
|
||||
`running` row now check `tomo_progress`'s heartbeat: stale or missing
|
||||
offers a "proceed anyway?" override instead of refusing forever; a
|
||||
fresh heartbeat still blocks outright. Click-tested (section 6e) and
|
||||
passing — the staleness override fixed an actual stuck job found live.
|
||||
13. **Warn before "Add current params to queue" mid-edit.** That button
|
||||
(in the ☰ Queue control… window) reads live global vars, not the
|
||||
params panel's in-progress edit -- a second place the same
|
||||
live-vs-unsaved-edit confusion as item 8 could bite, since it's a
|
||||
different window from the panel's own (correct) "Add to queue"
|
||||
button. Now warns and names the correct button before proceeding if
|
||||
the panel is mid-edit; no warning if it isn't. Click-tested (section
|
||||
6f, items 53–55) and passing.
|
||||
|
||||
---
|
||||
|
||||
## 2. How to open the widget
|
||||
|
||||
This is a `BECWidget` plugin (`PLUGIN = True`), so it's added to a BEC GUI
|
||||
layout the same way any other BEC widget is — via the widget picker in the
|
||||
running BEC GUI application, in a FlOMNI session (`bec --session flomni`).
|
||||
It only renders its real UI if `fsamroy` is present in the session (the
|
||||
FlOMNI discriminator); otherwise it shows the "setup not detected" message.
|
||||
|
||||
If you don't already have a FlOMNI GUI session up, see
|
||||
`SIMULATED_ENDSTATIONS.md` for bringing up the simulated flOMNI, or start the
|
||||
real session per `docs/user/ptychography/flomni.md`'s "How to setup flOMNI
|
||||
(software)" section.
|
||||
|
||||
---
|
||||
|
||||
## 3. Test plan — baseline regression (should already work; confirm nothing broke)
|
||||
|
||||
These exercise the pre-existing widget, not new code, but touching the same
|
||||
file is exactly how a regression sneaks in.
|
||||
|
||||
1. Open the widget. Confirm the params panel populates from the live global
|
||||
vars (matches what `flomni.tomo_parameters()` prints in a parallel CLI).
|
||||
2. Click **Edit**, change a couple of values (e.g. `tomo_countingtime`,
|
||||
`fovx`), click **Submit**. Confirm the values landed
|
||||
(`flomni.tomo_countingtime` in the CLI reflects the new value).
|
||||
3. Click **Edit** again, change something, click **Cancel**. Confirm the
|
||||
field reverts to the last-submitted value, not your edit.
|
||||
4. Start a real (or fast/sim) tomo scan from the CLI
|
||||
(`flomni.tomo_scan()`), then open the widget while it's running. Confirm
|
||||
the busy banner appears with a reason and now says Submit is blocked.
|
||||
Click **Edit**, change a value, click **Submit** — confirm it's refused
|
||||
outright (a plain warning dialog, no Yes/Cancel choice) and nothing gets
|
||||
written (`flomni.<param>` in the CLI still shows the old value). Then
|
||||
click **"Add to queue"** instead — confirm it prompts for a label, adds a
|
||||
new `pending` tomo job with your edited values (check via
|
||||
`flomni.tomo_queue_show()` or the queue dialog), and does **not** change
|
||||
any live param (`flomni.<param>` still shows the pre-edit value).
|
||||
5. Click **☰ Queue control…**. Confirm the dialog opens and the table
|
||||
reflects the current `tomo_queue` (including anything you added earlier
|
||||
via CLI in prior testing sessions).
|
||||
|
||||
## 4. Test plan — new: command-job builder
|
||||
|
||||
6. Click **"Add command…"**. Confirm the dialog opens with an **action
|
||||
dropdown showing `move` and `optimize_idgap`** (or whatever's currently
|
||||
in `flomni.py`'s `_TOMO_QUEUE_ACTIONS` — if you've added a third action
|
||||
since this was written, it should appear automatically with no GUI
|
||||
change needed; that's the point of reading the live registry).
|
||||
7. With `move` selected, confirm the field area shows a **device dropdown**
|
||||
(should list `mokev`, `idgap`, `ftray` per the current allow-list) and a
|
||||
**numeric target field**. Pick a device, set a target, click **"Add
|
||||
step"**. Confirm a line appears in the step list above, formatted like
|
||||
`move{'positions': {'mokev': 6.2}}`.
|
||||
8. Switch the action dropdown to `optimize_idgap`. Confirm the field area
|
||||
now shows a single **search_range** spinbox pre-filled with `0.5`,
|
||||
range-limited (try typing outside 0–2, confirm the widget clamps it
|
||||
rather than accepting it). Click **"Add step"**.
|
||||
9. Confirm the **idempotent checkbox auto-checked itself** after both adds
|
||||
(both current actions default to idempotent=True). Uncheck it by hand,
|
||||
then add another step — confirm it **stays unchecked** (your override
|
||||
sticks; it doesn't get silently recomputed back to checked). Leave it
|
||||
checked/matching your intent before continuing.
|
||||
10. With two steps in the list, select the second and click **"Move up"**.
|
||||
Confirm the list order swaps. Click **"Remove step"** on one entry,
|
||||
confirm it's gone from the list.
|
||||
11. Type a label, click **OK**. Confirm the dialog closes and a new row
|
||||
appears in the queue table with **Type = `CMD`** and a **Details**
|
||||
column showing the step summary + `[idem]`/`[NOT idem]` marker matching
|
||||
what you set.
|
||||
12. Cross-check from the CLI: `flomni.tomo_queue_show()` should print that
|
||||
exact same job, in the exact same `CMD ... [idem]` format. `flomni._tomo_queue_proxy.as_list()[-1]`
|
||||
should show `"kind": "command"`, a `steps` list matching what you built,
|
||||
and no `"params"` key.
|
||||
|
||||
## 5. Test plan — new: status-keyed guards
|
||||
|
||||
13. Start a queue running from the CLI (`flomni.tomo_queue_execute()`) with
|
||||
at least one job that takes a little while (or use the
|
||||
`omny_e2e_tests/_queue_helpers.py`-style fast params if you're doing
|
||||
this against the sim and want it quick — see `TOMO_QUEUE_TESTING.md`
|
||||
section 3's "making scans short enough to test").
|
||||
14. While a job shows **status = running** in the GUI table, select that row
|
||||
and click **"Delete selected"**. Confirm you get a warning dialog
|
||||
refusing the delete, not a confirmation prompt that would actually do it.
|
||||
15. With the same running job, click **"Clear all"**. Confirm you get a
|
||||
warning refusing to clear, rather than wiping the queue out from under
|
||||
the executing job.
|
||||
16. Let the queue finish (or stop it / let it go `incomplete`), then confirm
|
||||
you *can* now delete/clear normally.
|
||||
|
||||
## 6. Test plan — mixed queue, end to end
|
||||
|
||||
17. Build a small realistic sequence via the GUI only: a tomo job ("Add
|
||||
current params to queue"), then a command job (e.g. a `move` on
|
||||
`ftray` if you're on the sim — `mokev`/`idgap` only exist on the real
|
||||
front-end, see plan section 3.1), then another tomo job.
|
||||
18. From the CLI, run `flomni.tomo_queue_execute()`. Confirm: the tomo job
|
||||
runs, the command job actually moves the device (check
|
||||
`dev.<device>.readback.get()` before/after), and the second tomo job
|
||||
runs after — all while the GUI table (polling every 2 s) shows each
|
||||
job's status flip from `pending` → `running` → `done` in order, without
|
||||
needing to close/reopen the dialog.
|
||||
|
||||
## 6a. Test plan — new: sort mode and row tooltips
|
||||
|
||||
19. Queue at least three jobs (mix of tomo/command is fine). Click
|
||||
**"Sort queue…"**. Confirm: add/add-command/delete/clear all grey out,
|
||||
the table only lets you select one row at a time, and "▲ Move up" /
|
||||
"▼ Move down" appear (disabled until you select a row).
|
||||
20. Select a middle `pending` row. Confirm both move buttons enable (unless
|
||||
it's first/last among movable rows). Click **"▼ Move down"** a couple
|
||||
of times, confirm the row visibly moves down the table and selection
|
||||
follows it. Click **"▲ Move up"** to move it back.
|
||||
21. Start the queue running from the CLI (`flomni.tomo_queue_execute()`)
|
||||
with a job that takes a little while. While it shows `running`, open
|
||||
sort mode and select it. Confirm both move buttons are **disabled** —
|
||||
you can't move the running row at all.
|
||||
22. Still in sort mode with the queue running, select the `pending` row
|
||||
directly after the running one and click **"▲ Move up"**. Confirm
|
||||
nothing happens (button should already be disabled once you'd hit the
|
||||
running row) — the running row must never be displaced.
|
||||
23. If any job in the queue is `done`, select a `pending` row adjacent to
|
||||
it and confirm the move button on that side is disabled — done rows
|
||||
are pinned, not swapped past.
|
||||
23a. Get a job to `incomplete` (stop a running scan, or let one raise) and
|
||||
leave it in the queue. In sort mode, select it and confirm both move
|
||||
buttons are disabled (it can't be moved itself). Select the `pending`
|
||||
row directly after it and confirm "Move up" is disabled too — a
|
||||
pending job must never be draggable ahead of an `incomplete` one, even
|
||||
though `tomo_queue_execute()` would still resume the incomplete job
|
||||
first regardless (this was a real gap found during testing: the first
|
||||
pass let `incomplete` swap freely with `pending`).
|
||||
24. Click **"Sort queue…"** again to leave sort mode. Confirm the other
|
||||
buttons re-enable and multi-row selection (for delete) works again.
|
||||
25. Cross-check from the CLI: with the queue back in a known order, run
|
||||
`flomni.tomo_queue_move(<idx>, <new_idx>)` on two pending jobs and
|
||||
confirm the GUI table picks up the new order on its next 2 s poll.
|
||||
Then try moving a `done` or `running` job — confirm it raises
|
||||
`ValueError` instead of silently reordering.
|
||||
26. Hover the mouse over a **Label** cell whose text is cut off by column
|
||||
width. Confirm a tooltip appears with the full label, status, added-at
|
||||
timestamp, and (for a command job) the numbered step list + idempotent
|
||||
flag, or (for a tomo job) the full parameter dump. Confirm hovering any
|
||||
other cell in the same row shows the identical tooltip.
|
||||
27. Check column widths generally: `#`, Status, Type, Projections, Exp (s),
|
||||
and Step (µm) should size themselves to their (short) content; Label
|
||||
and Details should take up the rest of the dialog width and left-align
|
||||
their text, instead of every column being an equal fixed share. Confirm
|
||||
there's no "Added at" column at all (that data is now tooltip-only, see
|
||||
item 26). Resize the dialog wider/narrower and confirm Label/Details
|
||||
grow/shrink while the short columns stay put.
|
||||
|
||||
## 6b. Test plan — new: at_each_angle hook dropdown + webpage display
|
||||
|
||||
28. Open the widget. Confirm the params panel has an **"At-each-angle
|
||||
hook"** field showing **"None (default)"**, disabled (greyed out)
|
||||
outside edit mode, same as every other param.
|
||||
29. From the CLI (in a parallel session), register a hook:
|
||||
```python
|
||||
def noop_hook(flomni, angle):
|
||||
flomni.tomo_acquire_at_angle(angle)
|
||||
flomni.register_at_each_angle_hook("test_hook", noop_hook)
|
||||
```
|
||||
Within ~2s (the poll interval), without touching the GUI, confirm the
|
||||
dropdown would show `test_hook` as a selectable option next time you
|
||||
click **Edit** (it rebuilds on populate, which only happens outside
|
||||
edit mode — open Edit to see it).
|
||||
30. Click **Edit**, select `test_hook` in the dropdown, click **Submit**
|
||||
(beamline must be idle for this one — Submit is a hard block while
|
||||
busy, see item 4). Confirm it's accepted and
|
||||
`flomni.at_each_angle_hook` reads `"test_hook"` from the CLI.
|
||||
Set it back to **"None (default)"** and Submit again to reset.
|
||||
31. Click **Edit**, select `test_hook` again, but this time click **"Add to
|
||||
queue"** instead of Submit. Confirm: a new job appears in the queue
|
||||
dialog, `flomni.tomo_queue_show()` / `flomni._tomo_queue_proxy.as_list()[-1]["params"]["at_each_angle_hook"]`
|
||||
is `"test_hook"`, and — important — the **live**
|
||||
`flomni.at_each_angle_hook` was *not* changed (still whatever it was
|
||||
before you opened Edit), confirming Add-to-queue never touches live
|
||||
params.
|
||||
32. From the CLI, `flomni.unregister_at_each_angle_hook("test_hook")` while
|
||||
`flomni.at_each_angle_hook` (or a queued job) still references it.
|
||||
Click **Edit** again in the GUI. Confirm the dropdown still shows
|
||||
`test_hook` (doesn't silently disappear) but labeled **"test_hook (not
|
||||
registered in this session)"**.
|
||||
33. Run the queue with a `test_hook`-using job active (re-register the hook
|
||||
first). While it's `running`, check the **status webpage** — the
|
||||
queue's expanded job detail should show an **"At-each-angle hook"**
|
||||
row with the hook name; a plain job (no hook) should show no such row
|
||||
at all (not even a blank one).
|
||||
34. While that job is running, also check the webpage's **"Current
|
||||
measurement"** panel (outside the queue card) — it should show the
|
||||
same "At-each-angle hook" row, since it reads live params, not just
|
||||
the queue.
|
||||
35. Set `flomni.at_each_angle_hook = None` and run a plain job. Confirm the
|
||||
webpage shows no "At-each-angle hook" row anywhere once nothing has it
|
||||
set — this feature should be invisible on the page unless actually in
|
||||
use.
|
||||
36. Click **Edit**. Confirm the yellow-ish note under "Random shift max"
|
||||
("use `tomo_acquire_at_angle`, not `tomo_scan_projection`...") is
|
||||
**hidden** while "Single-point scan" is unchecked. Check the
|
||||
**"Single-point scan"** checkbox — confirm the note **appears**
|
||||
immediately (no need to Submit first). Uncheck it — confirm it
|
||||
disappears again. Cross-check the CLI side: `flomni.single_point_instead_of_fermat_scan = True`
|
||||
should print the same reminder immediately (not just when you next
|
||||
call `flomni.tomo_parameters()`), and setting it to `True` again right
|
||||
after should stay silent (only the off→on transition prints).
|
||||
|
||||
## 6c. Test plan — new: row-number gutter, Details-column hook, report/progress display
|
||||
|
||||
Found during real testing of section 6b (thanks, Mirko) — a mix of a small
|
||||
GUI bug and reporting gaps for the hook feature.
|
||||
|
||||
37. Open **☰ Queue control…** with a few jobs queued. Confirm there is
|
||||
**exactly one** index column — the dialog's own `#` (0-based, matching
|
||||
`tomo_queue_show()`/`tomo_queue_delete()`/`tomo_queue_move()` indices).
|
||||
Qt's own 1-based row-number gutter that used to sit to its left should
|
||||
be gone.
|
||||
38. Queue a tomo job with an active, currently-registered
|
||||
`at_each_angle_hook`. Confirm its **Details** column shows
|
||||
`hook: <name>`. Then unregister that hook
|
||||
(`flomni.unregister_at_each_angle_hook(...)`) and refresh — confirm
|
||||
Details now shows `hook: <name> (not registered)` instead of silently
|
||||
reverting to blank.
|
||||
39. `flomni.at_each_angle_hook = "some_hook"` with `some_hook` **not**
|
||||
registered. Confirm `flomni.tomo_parameters()` prints
|
||||
`At-each-angle hook = some_hook (NOT registered in this session)`, not
|
||||
just the bare name.
|
||||
40. With a hook registered and active, run `flomni.flomnigui_show_progress()`
|
||||
(or trigger it via a queued tomo job) and confirm the ring's center
|
||||
label includes a `Hook: <name>` line. Confirm the line disappears once
|
||||
`at_each_angle_hook` is `None`.
|
||||
41. Run a short tomo scan with a registered, active hook. Confirm:
|
||||
- the end-of-scan scilog entry includes an "At-each-angle hook: name"
|
||||
line, and — since the hook is still registered — a
|
||||
"At-each-angle hook source ('name'): ..." block with the actual
|
||||
function source underneath;
|
||||
- `flomni.write_pdf_report()` shows the hook name as a report line in
|
||||
both the generated PDF (`~/data/raw/documentation/tomo_scan_ID_<id>.pdf`)
|
||||
and the scilog entry it separately sends, and appends the hook's
|
||||
source code to both as well.
|
||||
Then repeat with the hook unregistered before the scan/report runs —
|
||||
confirm the name still shows (flagged "NOT registered"), but no source
|
||||
code block appears anywhere.
|
||||
|
||||
## 6d. Test plan — new: "Load into editor"
|
||||
|
||||
42. Queue a `done` tomo job (run one, or add one via CLI with `status`
|
||||
hand-edited to `done`). Select it in the queue dialog and confirm
|
||||
**"Load into editor"** is enabled. Select a command job (or multiple
|
||||
rows, or nothing) and confirm it's **disabled** in each case.
|
||||
43. Click **"Load into editor"** with nothing else in progress. Confirm the
|
||||
params panel enters edit mode (Edit button hides, Submit/"Add to
|
||||
queue"/Cancel appear) with every field populated from the *job's*
|
||||
saved values, not the live ones — pick a job whose settings clearly
|
||||
differ from current live params to make this obvious.
|
||||
44. With the panel now in edit mode from step 43, change a value, then
|
||||
click **"Load into editor"** again on a different (or the same) job.
|
||||
Confirm a **confirmation dialog** appears naming the job before
|
||||
discarding your edit; Cancel leaves your edit untouched, Yes reloads
|
||||
from the newly selected job's settings.
|
||||
45. After loading, click **Cancel** — confirm fields revert to the *live*
|
||||
values (not back to the loaded job's values) since Cancel always
|
||||
restores from the backend.
|
||||
46. After loading, click **"Add to queue"** — confirm a new job is added
|
||||
to the queue with the loaded (and any further tweaked) settings, and
|
||||
that live params are untouched. Then repeat ending with **Submit**
|
||||
instead (beamline idle) — confirm live params update to match.
|
||||
47. Confirm "Load into editor" is also disabled while **Sort queue…** mode
|
||||
is active, consistent with the other queue-mutating buttons.
|
||||
|
||||
## 6e. Test plan — new: declined-scan failure + stale "running" recovery
|
||||
|
||||
Found at the beamline (thanks again, Mirko) — a hook without `_internal=True`
|
||||
plus single-point mode got stuck asking for confirmation at every angle, and
|
||||
answering "no" (repeatedly) then Ctrl-C'ing left a job stuck at `running`
|
||||
forever with the GUI unable to delete it.
|
||||
|
||||
48. Set `flomni.single_point_instead_of_fermat_scan = True`, then call
|
||||
`flomni.tomo_scan_projection(10.0)` directly from the CLI. Answer **no**
|
||||
to "Run a fermat scan anyway?". Confirm it now **raises** (a traceback,
|
||||
not just "Aborted." with a silent return).
|
||||
49. Register a hook that calls `tomo_scan_projection(angle)` **without**
|
||||
`_internal=True`, activate it, queue a job with single-point mode on,
|
||||
and run the queue. When asked to confirm the fermat scan, answer **no**.
|
||||
Confirm the job ends up `incomplete` (not stuck `running`, not silently
|
||||
`done` with missing projections) and `tomo_queue_execute()` prints a
|
||||
clear "did not complete" message naming the actual problem.
|
||||
50. Hand-simulate the stuck-`running` scenario: pick a `pending` job's id
|
||||
(`flomni._tomo_queue_proxy.as_list()`) and
|
||||
`flomni._tomo_queue_proxy.update_by_id(job_id, status="running")`
|
||||
directly (standing in for "the process died mid-scan"), with no scan
|
||||
actually active (`tomo_progress`'s heartbeat absent or >120s old).
|
||||
In the GUI, select that row and click **Delete selected**. Confirm you
|
||||
get the **new** "looks stale — proceed anyway?" dialog (not the old
|
||||
unconditional refusal), and that clicking **Yes** actually deletes it
|
||||
after the normal delete confirmation. Repeat for **Clear all**.
|
||||
51. Same setup, but with `tomo_progress`'s heartbeat fresh (<120s old) —
|
||||
e.g. start a real short scan running so the heartbeat updates. Confirm
|
||||
delete/clear on the `running` row still refuse outright, no staleness
|
||||
override offered — a genuinely active job must still be unaffected.
|
||||
52. Cross-check from the CLI: `flomni.tomo_queue_delete(index)` should
|
||||
always work on a stuck `running` row regardless of GUI state (no
|
||||
status guard there at all) — useful as a fallback even without the
|
||||
GUI open.
|
||||
|
||||
## 6f. Test plan — new: warn before "Add current params to queue" mid-edit
|
||||
|
||||
Found during 6e testing — a second confusion between the two "add to queue"
|
||||
buttons living on two different windows.
|
||||
|
||||
53. Click **Edit** in the params panel, change a value, but do **not**
|
||||
click Submit or the panel's own "Add to queue" yet. Open **☰ Queue
|
||||
control…** and click **"Add current params to queue"** (the button in
|
||||
that window, not the panel). Confirm a warning dialog appears naming
|
||||
the live-vs-unsaved-edit mismatch and pointing at the panel's own "Add
|
||||
to queue" button, before anything is queued.
|
||||
54. From that dialog, click **Cancel** — confirm nothing was added to the
|
||||
queue. Click the button again and this time confirm **Yes** — confirm
|
||||
it queues the **live** (pre-edit) values, not your in-progress edit,
|
||||
exactly as the warning said it would.
|
||||
55. With the params panel **not** in edit mode, click "Add current params
|
||||
to queue" — confirm it queues normally with **no** warning (only
|
||||
fires when an edit is actually in progress).
|
||||
|
||||
## 7. Known limitations of this first iteration (by design, not bugs)
|
||||
|
||||
Worth knowing before you file something as broken:
|
||||
|
||||
- **Reorder is up/down buttons in a "Sort queue…" mode, not drag-and-drop**
|
||||
(plan section 6.4). Moving a row more than one position takes several
|
||||
clicks — click-and-drag across the table isn't implemented. This was a
|
||||
deliberate scope cut for the first iteration: the "reject any drop above
|
||||
the running row" logic the plan calls for is nontrivial as live
|
||||
drag-event math, but trivial as "an adjacent swap refuses a
|
||||
running/done neighbour" (which the button implementation uses), so this
|
||||
gets the same running-row/done-row invariants without the drag machinery.
|
||||
Revisit if button-clicking to reorder a long queue proves too slow in
|
||||
practice.
|
||||
- **One device per `move` step in the builder**, even though the underlying
|
||||
schema (`positions: {device: target, ...}`) technically supports moving
|
||||
several devices in one step. The CLI can still do this
|
||||
(`tomo_queue_add_command` with a `positions` dict of more than one
|
||||
device); the GUI builder just doesn't expose it yet. Add extra `move`
|
||||
steps instead if you need to move more than one device.
|
||||
- **No live device-limit validation.** The plan (section 6.5) mentions
|
||||
validating against `dev.<name>.limits` in addition to the schema's
|
||||
min/max; this iteration only enforces the schema bounds (via the spinbox
|
||||
range), not real hardware limits. The execution-time re-check in
|
||||
`_queue_action_move()` is still the actual safety boundary regardless
|
||||
(two-layer validation, plan section 3.3), so this is a UX gap, not a
|
||||
safety gap.
|
||||
- **Resolved: `Submit` is now a hard block while the beamline is busy**,
|
||||
matching plan section 6.1 literally instead of the earlier soft-warn
|
||||
confirmation dialog. The gap that made the soft warn tempting in the first
|
||||
place -- wanting to prepare a parameter set while a scan runs -- is now
|
||||
covered by a new **"Add to queue"** button that appears next to Submit in
|
||||
edit mode: it packages the current (possibly unsubmitted) form fields as
|
||||
a new `pending` tomo-queue job and writes only to `tomo_queue`, never the
|
||||
live param vars, so it's always allowed. See test step 4 in section 3.
|
||||
- **No automated widget tests.** `pytest-qt` isn't a dependency of this
|
||||
repo, so there's no `qtbot`-driven test suite here — the smoke test that
|
||||
validated the builder's logic (dynamic fields, step ordering, idempotent
|
||||
default, job-dict shape) was a one-off script against a fake client, not
|
||||
a committed test. If this GUI work continues, consider whether adding
|
||||
`pytest-qt` is worth it.
|
||||
|
||||
---
|
||||
|
||||
## 8. Continuing this work in a fresh Claude session
|
||||
|
||||
You'll be leaving this coding session at some point. Two different things
|
||||
you might want a fresh Claude session to do, and they need different setups:
|
||||
|
||||
### To keep writing/fixing GUI code
|
||||
|
||||
You need a **Claude Code session with access to this repository** — a new
|
||||
terminal/CLI session, the desktop app, or a browser-based Claude Code
|
||||
session, pointed at this same checkout (or a fresh `git clone` of this
|
||||
branch after you push). A plain chat on claude.ai, without a connected
|
||||
coding environment, **cannot edit files in this repo** — it can only discuss
|
||||
and produce text/snippets for you to paste in yourself.
|
||||
|
||||
When you start that fresh session, just point it at this file and the plan
|
||||
doc — it doesn't need you to re-explain any of the above:
|
||||
|
||||
> Read `csaxs_bec/bec_ipython_client/plugins/flomni/AI_docs/TOMO_QUEUE_GUI_TESTING.md`
|
||||
> and `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` section 6, then continue the Step 3
|
||||
> GUI work in `tomo_params.py` — pick up from section 7's known
|
||||
> limitations, starting with [whichever one you want tackled next].
|
||||
|
||||
### To plan/discuss without editing code (a Claude Project)
|
||||
|
||||
This is what a **Claude Project** on claude.ai is good for: attach the
|
||||
`AI_docs/` files (this one, `TOMO_QUEUE_COMMAND_JOBS_PLAN.md`,
|
||||
`TOMO_QUEUE_TESTING.md`, `E2E_TESTING_GUIDE.md`, `SIMULATED_ENDSTATIONS.md`)
|
||||
as project knowledge, and a chat inside that project will have the full
|
||||
design/status context without you re-explaining it every time — useful for
|
||||
sketching out the next iteration's design, reviewing test results, or
|
||||
drafting the instruction you then hand to an actual Claude Code session (the
|
||||
E2E_TESTING_GUIDE.md was written for exactly this handoff pattern already;
|
||||
this document and the plan doc work the same way for the GUI).
|
||||
|
||||
**The concrete loop that actually moves code forward:** discuss/plan in the
|
||||
Project chat if useful, then paste the resulting instruction into a real
|
||||
Claude Code session attached to this repo to make the edit, then come back
|
||||
here (or to a fresh Code session) to test it against a live GUI — a Project
|
||||
chat by itself never touches `tomo_params.py`.
|
||||
|
||||
If you run through sections 3–6 above yourself before switching sessions,
|
||||
paste your findings (what passed, what didn't, exact error text if
|
||||
something broke) into whichever session picks this up next — "I clicked
|
||||
through it and it didn't work" is much less useful than "step 8: typing 2.5
|
||||
into search_range with the sim's flomni_sim fixture up, the widget raised
|
||||
`AttributeError: ...`". Update this file's section 1 status line once
|
||||
you've actually run through the checklist, so the next session doesn't
|
||||
re-read "not yet validated" after you've validated it.
|
||||
@@ -0,0 +1,410 @@
|
||||
# Tomo Queue — Executor Rework: Testing & Continuity
|
||||
|
||||
**Read this first.** It is the handoff for the tomo-queue work: what changed, what
|
||||
must be tested against the simulated flOMNI, and what is deliberately *not* built
|
||||
yet. Companion to `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` (the design for the *next*
|
||||
step — command jobs — which is **not** implemented).
|
||||
|
||||
**Status:** Step 1 (executor rework) implemented in `flomni.py`. **Sections A, B, and
|
||||
C of the checklist have now passed against the live sim** (10/10 tests, run together
|
||||
with no cross-test interference — see §4). **All of §6's open questions are now
|
||||
resolved with Mirko** (registry/allow-list in `TOMO_QUEUE_COMMAND_JOBS_PLAN.md` §10;
|
||||
the `_TOMO_SCAN_PARAM_NAMES` rename is done). **Step 2 (command jobs) is now
|
||||
implemented and sim-tested too** — §4A gained a fourth checklist section, D, covering
|
||||
`omny_e2e_tests/test_tomo_queue_command_jobs.py` (6/6 passed). §5's description of
|
||||
Step 2 is otherwise still accurate as the as-built design. Only **§4's section E (real
|
||||
instrument)**, needing beam time, and **Step 3 (the GUI)** remain open — Step 3 is
|
||||
designed (§5) but not started.
|
||||
|
||||
**Test folder moved:** the e2e suite (and the two sim-harness scripts that bootstrap
|
||||
it) now live in `omny_e2e_tests/` at the repo root, not `tests/e2e/` — kept out of
|
||||
`tests/` specifically so it doesn't get swept up by gitea's generic `tests/`
|
||||
discovery/CI run (these tests need a live sim + Redis, not just `pytest`). Run them
|
||||
directly: `pytest omny_e2e_tests/`.
|
||||
|
||||
---
|
||||
|
||||
## 0. House rules (apply to every change here)
|
||||
|
||||
- **Minimal diffs.** No incidental Black reformats of unrelated lines. `flomni.py`
|
||||
has pre-existing formatting drift (5 `black` hunks that predate this work) —
|
||||
**leave it alone**; do not "fix" it in the same commit.
|
||||
- Validate every edited `.py` with `ast.parse` and
|
||||
`black --check --line-length=100 --skip-magic-trailing-comma`. The correct check
|
||||
is *"no NEW black hunks vs. the unmodified file"*, not "black is clean".
|
||||
- Targeted `str_replace`-style edits, not file rewrites.
|
||||
- Conventional Commits.
|
||||
- **Sim/hardware validation is the final gate.** Unit tests and mock harnesses come
|
||||
first, but nothing is "verified" until it has run against the simulated (and
|
||||
ultimately real) instrument.
|
||||
|
||||
---
|
||||
|
||||
## 1. What changed (Step 1 — already in `flomni.py`)
|
||||
|
||||
The tomo queue executor was reworked from a **fixed-index loop** to an
|
||||
**id-based pick-next loop**. Diff is +100/−17 across 5 hunks:
|
||||
|
||||
1. `import uuid`.
|
||||
2. `_TomoQueueProxy` gained:
|
||||
- `update_by_id(job_id, **kwargs) -> bool` — write status by stable id, not list
|
||||
index. Returns `False` if the id is gone (job deleted concurrently).
|
||||
- `ensure_ids() -> list` — assign a uuid to any job lacking one, persist. Heals
|
||||
legacy queues written before the `id` field existed.
|
||||
3. `tomo_queue_add()` now stamps `"id": uuid.uuid4().hex` on each new job.
|
||||
4. `tomo_queue_execute()` rewritten: re-reads the queue from its global var at the
|
||||
start of **every** job (no up-front snapshot) and addresses jobs by `id`.
|
||||
|
||||
### Why (the motivation — don't undo this)
|
||||
|
||||
The old loop bound `for i in range(start_index, len(jobs))` against a snapshot taken
|
||||
once, then wrote status back **by index** via `update(i, ...)`, which reloaded the
|
||||
*live* var. If the queue was edited while running (GUI, or a second CLI session),
|
||||
indices drifted: status writes landed on the **wrong job**, or `update()` raised
|
||||
`IndexError`. Appended jobs were also never picked up by the running loop.
|
||||
|
||||
The pick-next loop makes it safe to **append, delete a pending job, and reorder the
|
||||
pending tail while the queue is running**. Those edits take effect at the next job
|
||||
boundary.
|
||||
|
||||
### Two deliberate behaviours (these are not bugs)
|
||||
|
||||
- **Resume-before-fresh.** If any job is `incomplete`/`running`, it runs **before**
|
||||
any fresh `pending` job, regardless of list order. Tomo progress lives in ONE
|
||||
shared `progress` global var — starting a fresh `tomo_scan()` ahead of an
|
||||
interrupted one would overwrite the very state `tomo_scan_resume()` needs. The old
|
||||
strictly-in-order loop made this impossible; allowing reorder reintroduces the
|
||||
risk, hence the explicit guard.
|
||||
- **`start_index` semantics changed.** It is now "position in the *current* order
|
||||
below which fresh `pending` jobs are ignored", applied at each pick. Resumable jobs
|
||||
**ignore** `start_index` (a crashed job must be recovered wherever it sits).
|
||||
Default `0` behaves as before.
|
||||
|
||||
### Invariant to preserve
|
||||
|
||||
**At most one job may ever be `running`/`incomplete` at a time**, because `progress`
|
||||
is a single global var. Nothing in append/delete/reorder can create a second one
|
||||
(only execution sets those statuses). Do not add a "mark incomplete manually"
|
||||
affordance without revisiting this.
|
||||
|
||||
### What Step 1 does NOT do
|
||||
|
||||
No `kind` field, no command jobs, no GUI changes, no webpage-generator changes.
|
||||
A queue of plain tomo jobs behaves exactly as before, plus the ids.
|
||||
|
||||
---
|
||||
|
||||
## 2. Offline validation already done
|
||||
|
||||
`ast.parse` clean; **no new black hunks** vs. the unmodified file. A mock-client
|
||||
harness (extracting the *real* `_TomoQueueProxy` + `tomo_queue_execute` source via
|
||||
AST, not a reimplementation) passes 10 scenarios: in-order run; done-jobs skipped;
|
||||
id-less legacy jobs healed; resume-before-fresh; `running` treated as resumable;
|
||||
reorder / delete / append mid-run; failure → `incomplete` + re-raise + correct
|
||||
resume on re-run; `update_by_id` found/missing.
|
||||
|
||||
**This proves the logic, not the integration.** The mock stubs `tomo_scan()`, so it
|
||||
cannot prove that params are really restored onto the live global vars the scan
|
||||
reads, nor that `tomo_scan_resume()` recovers real mid-scan `progress` state. That
|
||||
is what the sim is for.
|
||||
|
||||
---
|
||||
|
||||
## 3. THE TEST HARNESS — use `pytest_bec_e2e`, and mind the trap
|
||||
|
||||
BEC already ships **`pytest_bec_e2e`** (registered `pytest11` entry point, in the
|
||||
`bec` repo). It provides a **live** `BECClient` — real server, real Redis, real
|
||||
global vars. Without `--start-servers` it **attaches to an already-running server**,
|
||||
which is exactly the sim session. Added to `csaxs_bec`'s `dev` extras
|
||||
(`pyproject.toml`) as `pytest-bec-e2e`.
|
||||
|
||||
### ⚠️ TRAP — do not use `bec_client_lib` as-is
|
||||
|
||||
Both shipped client fixtures (`bec_client_lib_with_demo_config`,
|
||||
`bec_ipython_client_with_demo_config`) call:
|
||||
|
||||
```python
|
||||
bec.config.load_demo_config(force=True)
|
||||
```
|
||||
|
||||
Against a running sim session this **overwrites the loaded simulated-flOMNI device
|
||||
config with BEC's demo devices**. Write our own fixture that reuses the plumbing
|
||||
(`bec_servers`, `bec_redis_fixture`, `bec_services_config_file_path`) but either
|
||||
loads `device_configs/simulated_omny/simulated_flomni.yaml` explicitly or assumes the
|
||||
running session already has it.
|
||||
|
||||
### The fixture: `omny_e2e_tests/_bootstrap.py` + `omny_e2e_tests/conftest.py` → `flomni_sim` — BUILT
|
||||
|
||||
This is the reusable piece — worth more than the checklist itself. It's built and
|
||||
factored into two files so section B/C tests (which need a real, separate OS process
|
||||
to kill or run concurrently — see below) can build the same `Flomni` outside pytest:
|
||||
|
||||
- `_bootstrap.py`: `build_flomni(services_config_path) -> (bec, flomni)` — connects
|
||||
(no demo-config load), does the builtins/reload bootstrap (see gotcha below),
|
||||
neutralizes side effects, and does the one-time RT-feedback/fsamx setup (see
|
||||
gotchas below). Also `neutralize_side_effects()` and `shutdown(bec)`.
|
||||
- `conftest.py`: the `flomni_sim` pytest fixture, which calls `build_flomni()` and
|
||||
snapshots/restores the `tomo_queue` / `tomo_progress` global vars around each test.
|
||||
- `_run_queue_subprocess.py`: a standalone script (`python _run_queue_subprocess.py
|
||||
<services_config_path> [start_index]`) that calls `build_flomni()` +
|
||||
`tomo_queue_execute()`. Used by every section B/C test as the thing that gets
|
||||
`SIGKILL`ed or run concurrently with a second client.
|
||||
- `_queue_helpers.py`: `SHORT_SCAN_PARAMS` / `FAST_STEPSIZE` / `add_short_job()` for
|
||||
building fast test tomograms; `ProgressSampler` (see thread gotcha below);
|
||||
`spawn_queue_subprocess()`; `wait_until()`.
|
||||
|
||||
Every future piece of work (command jobs, the GUI queue dialog, LamNI parity) can
|
||||
reuse all four.
|
||||
|
||||
### Gotchas found getting a real scan to run (not obvious from reading `flomni.py`)
|
||||
|
||||
None of these are Step-1 bugs — they're pre-existing `tomo_scan()`/environment
|
||||
dependencies that just hadn't been exercised end-to-end on this dev machine before.
|
||||
All are handled once, in `_bootstrap.py`:
|
||||
|
||||
- **`BECIPythonClient`'s live-update machinery requires the main thread.**
|
||||
`scans.umv(...)` (and anything else that goes through
|
||||
`ipython_live_updates.process_request`) installs a `SIGINT` handler per request,
|
||||
which Python only allows from a process's real main thread. So `tomo_queue_execute()`
|
||||
itself can **never** run on a background Python thread — for section A's
|
||||
"observe progress while it runs" this means *sampling* runs on a background thread
|
||||
while execution stays on the main thread (`ProgressSampler`); for section B/C's
|
||||
"kill it" / "run two of these at once" this means each session is a real OS
|
||||
subprocess, not a thread.
|
||||
- **The builtins/reload bootstrap is order-sensitive.** `flomni.py`'s module-level
|
||||
`def umv(*args): return scans.umv(...)` resolves `scans` from the *module's own
|
||||
globals*, populated only by the `if builtins.__dict__.get("bec") is not None:`
|
||||
block at import time. If `flomni.py` gets imported before `builtins.bec` is set
|
||||
(likely, transitively), that block never runs and `umv` `NameError`s at call time.
|
||||
Fix: set `builtins.bec/dev/scans` first, then `importlib.reload()` the module.
|
||||
- **`Flomni.__init__` has real side effects that must be neutralized for tests:**
|
||||
`FlomniWebpageGenerator.start()` opens a local port and POSTs to a real external
|
||||
URL; `tomo_scan()` unconditionally sends a SciLog summary at the end of every scan
|
||||
(not configured on this dev machine) — both patched to no-ops.
|
||||
- **RT feedback + `fsamx` positioning is a real one-time hardware-setup step,**
|
||||
normally folded into x-ray eye alignment (which tests skip): `fsamx` must be near
|
||||
its configured "in" position before `feedback_enable_with_reset()` will run, and
|
||||
`feedback_enable_with_reset()` itself locks `fsamx` read-only as its own last step
|
||||
(the RT controller's PID correction owns fine positioning after that) — so a second
|
||||
call must check `feedback_is_running()` first, or a plain `umv(fsamx, ...)` raises
|
||||
`DisabledDeviceError`. Also expensive (re-zeros the interferometers), so skip it
|
||||
entirely once feedback is already running from a previous session.
|
||||
- **Local output directories aren't created automatically:** `tomo_scan()` writes to
|
||||
`~/data/raw/{documentation,logs,webpage,analysis/online/ptycho}` unconditionally
|
||||
(PDF report, scan-number log, ...) — these didn't exist on this dev machine and had
|
||||
to be created once (`mkdir -p`), not part of the repo.
|
||||
|
||||
### Making scans short enough to test
|
||||
|
||||
`sim_point_dwell_s` defaults to 20 ms. `SHORT_SCAN_PARAMS` in `_queue_helpers.py`
|
||||
sets `single_point_instead_of_fermat_scan=True` (one acquisition per angle instead of
|
||||
a full Fermat grid — `tomo_acquire_at_angle()`, not `tomo_scan_projection()`), small
|
||||
`fovx`/`fovy`, large `tomo_shellstep`, and a small `tomo_countingtime`. Combined with
|
||||
a coarse `tomo_angle_stepsize` (`FAST_STEPSIZE = 170.0` → 1 projection/subtomo, 8
|
||||
total) a full tomogram runs in roughly a minute — long enough to still be
|
||||
interruptible mid-scan for the crash-resume tests (§4, B5).
|
||||
|
||||
---
|
||||
|
||||
## 4. THE CHECKLIST
|
||||
|
||||
### A. Sim, no crash needed — ✅ PASSED (`omny_e2e_tests/test_tomo_queue.py`)
|
||||
|
||||
1. **✅ Params actually restored per job.** `test_params_restored_per_job`. Two jobs
|
||||
with distinct `tomo_angle_stepsize` (89.9 → N=2, 59.9 → N=3); a background thread
|
||||
polls the shared `tomo_progress` global var (execution itself must stay on the
|
||||
main thread — see the threading gotcha in §3) while each job runs and confirms
|
||||
`progress["subtomo_total_projections"]`, computed live by `sub_tomo_scan()` from
|
||||
the *live* global var, matches each job's own value in the right order. This is
|
||||
the "did the SCAN actually use them" observable, agreed before running it.
|
||||
2. **✅ Legacy queue migration.** `test_legacy_queue_migration`. Two id-less jobs
|
||||
written via `set_global_var("tomo_queue", [...])` ran cleanly; afterward every job
|
||||
carried a unique healed `id`.
|
||||
3. **✅ Empty / all-done queue.** `test_empty_and_all_done_queue_are_noops`. Both
|
||||
paths print their no-op message, never call `OMNYTools.yesno`, never scan.
|
||||
4. **✅ `start_index`.** `test_start_index_semantics`. `start_index=1` skipped a
|
||||
pending job 0; marking it `incomplete` and re-running with the same `start_index`
|
||||
ran it anyway (resumable jobs ignore `start_index`).
|
||||
|
||||
### B. Sim, crash / interrupt paths — ✅ PASSED
|
||||
|
||||
5. **✅ Real crash-resume.** `test_real_crash_resume`
|
||||
(`omny_e2e_tests/test_tomo_queue_crash_subprocess.py`). `tomo_queue_execute()` run as a
|
||||
real OS subprocess (required — see §3), `SIGKILL`ed mid-job1. Confirmed via the
|
||||
subprocess's own log: killed at `subtomo 3, angle 45.0`; the resume subprocess
|
||||
printed `Resuming tomo scan at subtomo 3, angle 45.000 deg` and re-acquired that
|
||||
*exact* subtomo/angle (checked by polling `tomo_progress` for a new `heartbeat`
|
||||
after the resume subprocess starts, then comparing the full snapshot) before the
|
||||
rest of the queue completed. Assertion design agreed beforehand.
|
||||
6. **✅ Exception mid-scan → pause.** `test_exception_mid_scan_pauses_queue`
|
||||
(`omny_e2e_tests/test_tomo_queue_crash.py`). `tomo_scan` patched to raise on its first
|
||||
call only. Job → `incomplete`, exception re-raised out of `tomo_queue_execute()`,
|
||||
later job stayed `pending`; re-running resumed the failed job first.
|
||||
7. **✅ Resume-before-fresh, explicitly.** `test_resume_before_fresh_explicit`
|
||||
(same file). An `incomplete` job below a fresh `pending` one in list order ran
|
||||
first regardless of position (tracked via an `update_by_id` call-order spy).
|
||||
|
||||
### C. Concurrency — the path the GUI will use — ✅ PASSED (`omny_e2e_tests/test_tomo_queue_concurrency.py`)
|
||||
|
||||
8. **✅ Edit the queue from a second client while one runs.**
|
||||
`test_concurrent_queue_edits_from_second_client`. Session A = subprocess executing
|
||||
4 jobs; session B = the pytest process's own client, mid-job1: reordered the
|
||||
pending tail (job4 ahead of job2), deleted job3, appended job5 — all via the same
|
||||
redis-backed `tomo_queue` global var, no live scan-param vars touched. Session A
|
||||
picked up every change at the next job boundary: final run order was job1, job4,
|
||||
job2, job5, with job3 never appearing.
|
||||
9. **✅ Delete the *running* job mid-run.** `test_delete_running_job_mid_run`.
|
||||
Deleted job1 (index 0, `status="running"`) out from under the executing
|
||||
subprocess. Degraded gracefully: no crash (`update_by_id` on the vanished id
|
||||
just returns `False`), job1's scan finished on its own terms, job2 still ran and
|
||||
was marked `done` correctly.
|
||||
|
||||
### D. Command jobs (Step 2) — ✅ PASSED (`omny_e2e_tests/test_tomo_queue_command_jobs.py`)
|
||||
|
||||
10. **✅ Add-time validation rejects bad steps, and rejects them without landing
|
||||
anything in the queue.** `test_add_command_validation_errors`. Unknown action,
|
||||
a device not on the `move` allow-list, an out-of-range/wrong-type
|
||||
`optimize_idgap.search_range`, and an unknown kwarg are each rejected with the
|
||||
expected `ValueError`; afterward the queue is confirmed still empty.
|
||||
11. **✅ Execution-time allow-list re-check is the real safety boundary.**
|
||||
`test_move_execution_time_allowlist_recheck`. A job with a disallowed device
|
||||
appended directly (bypassing `tomo_queue_add_command`) is refused by
|
||||
`_queue_action_move()` itself at run time and correctly left `incomplete`, not
|
||||
silently marked `done`.
|
||||
12. **✅ `tomo_queue_actions` global var matches the live registry.**
|
||||
`test_tomo_queue_actions_global_var_matches_registry`. Confirms the published
|
||||
var (for the future GUI, §7 of the plan doc) isn't a stale/hand-maintained copy.
|
||||
13. **✅ `tomo_queue_show()` and the web page generator survive a command job.**
|
||||
`test_show_and_webpage_survive_command_job`. A command job (no `params` key)
|
||||
renders its step summary + idempotency marker in both, instead of `KeyError`ing.
|
||||
14. **✅ Back-compat + real dispatch, combined.**
|
||||
`test_legacy_job_no_kind_defaults_to_tomo_and_command_moves_ftray`. A `kind`-less
|
||||
legacy queue entry still runs as a tomogram; in the same mixed-queue run, a
|
||||
command job actually moves a device (`ftray`, the sim-testable stand-in for
|
||||
`mokev`/`idgap` — see the plan doc §3.1) via the real `move` action path.
|
||||
15. **✅ Non-idempotent resume prompts exactly once, and respects "no".**
|
||||
`test_non_idempotent_command_resume_prompts`. A command job flagged
|
||||
`idempotent=False`, left `incomplete`, triggers the resume prompt (distinct from
|
||||
the executor's own top-level "OK to start?" prompt) on the next
|
||||
`tomo_queue_execute()`; answering "no" leaves the device unmoved and marks the
|
||||
job `done` without re-running it.
|
||||
|
||||
### E. Real instrument (not sim) — NOT DONE, needs Mirko / beam time
|
||||
|
||||
16. One queued tomogram end-to-end on the real instrument, to confirm the sim's
|
||||
timing/progress semantics match reality closely enough that resume lands on the
|
||||
right angle.
|
||||
17. `move` on `mokev`/`idgap` end-to-end — neither device is in the flomni sim
|
||||
config (see the plan doc §3.1), so Step 2's `move` action itself has only been
|
||||
sim-exercised via `ftray`, never against the real devices it's actually for.
|
||||
|
||||
**Gate:** A–D passed on the sim (16/16 tests total). **Steps 1 and 2 are verified
|
||||
enough to build Step 3 (the GUI) on.** E remains open, and blocks calling `move` on
|
||||
`mokev`/`idgap` production-ready.
|
||||
|
||||
### A gotcha found testing Step 2 (not a Step-2 bug)
|
||||
|
||||
**The sim's shared Redis is not reset between separate `pytest` invocations, and the
|
||||
`flomni_sim` fixture only snapshots/restores whatever's already there — it does not
|
||||
reset to empty.** A prior run that errored out before reaching its `finally` block
|
||||
(or manual poking at the live sim) can leave a stale job stuck mid-queue — e.g. at
|
||||
`status: "running"`, or written in a pre-`kind`-field format — that then contaminates
|
||||
every subsequent test run that assumes it starts from an empty queue: expect
|
||||
spurious failures like "queue not empty after a rejected add" or `KeyError: 'kind'`
|
||||
on an old-format entry that isn't actually related to the test that hit it. Fixed by
|
||||
having `flomni_sim` reset `tomo_queue`/`tomo_progress` to empty at setup (still
|
||||
snapshotting/restoring around that, so a real operator session on the same sim isn't
|
||||
disturbed) — see `conftest.py`. If this resurfaces, check
|
||||
`redis-cli get "user/vars/tomo_queue"` for leftover state before assuming it's a
|
||||
real regression.
|
||||
|
||||
---
|
||||
|
||||
## 5. Steps 2 & 3 — Step 2 implemented and sim-tested (§4D), Step 3 still just designed
|
||||
|
||||
Full design for command jobs is in `TOMO_QUEUE_COMMAND_JOBS_PLAN.md`, which is now
|
||||
up to date with what actually shipped in Step 2 (both documents describe the same
|
||||
as-built behaviour; this section is kept as a second description for convenience,
|
||||
not because they disagree). Step 3 (the GUI, below) is unstarted — this section is
|
||||
still just its design.
|
||||
|
||||
### Command jobs (Step 2)
|
||||
|
||||
- **Named-action registry.** No `exec` of stored code strings — rejected as
|
||||
unauditable, fails at run time, breaks crash-resume.
|
||||
- **Multi-step jobs.** A command job holds a **list** of steps, not a single action,
|
||||
so "optics out → change energy → optics in" is ONE queue row:
|
||||
```python
|
||||
{"kind": "command", "label": "...", "steps": [{"action": ..., "kwargs": {...}}, ...],
|
||||
"idempotent": bool, "status": ..., "id": ..., "added_at": ...}
|
||||
```
|
||||
- **No per-step resume cursor.** On resume: if all steps are idempotent, re-run the
|
||||
whole sequence from the top (re-doing an absolute move is harmless). If any step is
|
||||
non-idempotent, the job is flagged non-idempotent and resume **prompts** the
|
||||
operator (`OMNYTools.yesno`) rather than silently re-running.
|
||||
- **Curated device allow-list** for the generic `move` action. NOT "any device in
|
||||
`dev`". This is the difference between a reconfiguration tool and a remote control
|
||||
for the whole endstation.
|
||||
- **Registry entry = 4 things:** bound method name, `idempotent` default, help string,
|
||||
and a **params schema** (possibly empty).
|
||||
- **Result-dependent logic lives INSIDE a named action, never across queue steps.**
|
||||
E.g. "scan idgap, move to peak" is ONE action (`optimize_idgap`) whose method does
|
||||
the scan, finds the peak, and moves — *not* three queue steps. The queue therefore
|
||||
never has to carry runtime values between steps; `kwargs` stays plain JSON. A
|
||||
compound action may take parameters (e.g. `search_range`), declared in its params
|
||||
schema like any other.
|
||||
- **Publish the registry via a global var** (`tomo_queue_actions`) on `Flomni.__init__`
|
||||
so the GUI builds its dropdowns/forms from it — no hardcoded mirror to drift (the
|
||||
`QUEUE_PARAM_NAMES` mirror between `flomni.py` and `tomo_params.py` already has this
|
||||
problem).
|
||||
- **Back-compat:** a job with no `kind` is treated as `"tomo"`.
|
||||
|
||||
### GUI (Step 3) — `tomo_params.py` / `TomoQueueDialog`
|
||||
|
||||
Agreed interaction model:
|
||||
|
||||
- **Two separate write paths.** "Done" writes the **live param global vars** → must be
|
||||
**blocked while a scan is running** (the scan re-reads those vars per projection, so
|
||||
editing them mid-run perturbs the running acquisition). "Add to queue" writes **only
|
||||
the `tomo_queue` var** → always allowed, even mid-run. *This separation is the whole
|
||||
reason the GUI is usable during a run — do not let "Add to queue" touch live vars.*
|
||||
- "Add to queue" **cannot** call the CLI `tomo_queue_add()` — that snapshots the
|
||||
**live** vars and would ignore the operator's unapplied form edits. The dialog must
|
||||
assemble the job dict itself (edited fields + current live values for params the form
|
||||
doesn't expose) so the snapshot is complete.
|
||||
- **No in-place row editing.** Editing a queued job = delete it + add a new one +
|
||||
reorder into place. Keeps one construction path for jobs.
|
||||
- **Row locks keyed on `status`, not on a global busy signal:**
|
||||
- `running` row: not draggable, not deletable. Deletable only after stop (then it is
|
||||
`incomplete`).
|
||||
- `pending` / `incomplete`: freely draggable and deletable.
|
||||
- `done`: pinned (harmless to move, but confusing to read).
|
||||
- **Reorder is allowed mid-run**, and takes effect at the next job boundary. Reject any
|
||||
drop that would move a row **above** the running row — the running row is a hard floor
|
||||
for drops, so the visual order stays honest about what runs next.
|
||||
- A **command-job builder pop-up** (structured form, not a text editor): action
|
||||
dropdown from the registry, fields from the action's params schema, live validation
|
||||
against real device limits, OK disabled until valid. **No syntax-checked code box** —
|
||||
syntax-valid is nearly worthless as a safety check (`umv(dev.fsamx, -1e9)` parses
|
||||
fine), and it gives false confidence.
|
||||
- Execution stays **CLI-only**; the GUI server process can't run the blocking scan loop.
|
||||
|
||||
### Webpage generator (`flomni_webpage_generator.py`) — DONE
|
||||
|
||||
Guarded against command jobs (`job.get("kind", "tomo") == "tomo"` before touching
|
||||
`job["params"]`), verified by §4D13 (`test_show_and_webpage_survive_command_job`).
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions for Mirko — all resolved
|
||||
|
||||
- **The final action set and device allow-list**, see `TOMO_QUEUE_COMMAND_JOBS_PLAN.md`
|
||||
§10 — `move` + `optimize_idgap` only, `_TOMO_QUEUE_MOVE_DEVICES = {mokev, idgap}`.
|
||||
One detail (`optimize_idgap`'s peak-detection signal) deferred to the
|
||||
implementation session, not blocking.
|
||||
- **`_TOMO_QUEUE_PARAM_NAMES` renamed to `_TOMO_SCAN_PARAM_NAMES`** (it reads live
|
||||
properties, not queue state — the old name was misleading). Done in `flomni.py`;
|
||||
the `tomo_params.py` comment referencing it by name was updated too (the
|
||||
`QUEUE_PARAM_NAMES` mirror constant itself keeps its own name, unchanged).
|
||||
|
||||
Nothing left blocking Step 2 except D (real instrument, needs beam time).
|
||||
@@ -0,0 +1,239 @@
|
||||
# flOMNI / cSAXS status monitoring — development handoff
|
||||
|
||||
Consolidated context for continuing development in Claude Code. Supersedes nothing;
|
||||
read alongside `project_briefing_update5.md` and `multi_setup_webpage_architecture.md`
|
||||
for the deepest detail on the last session.
|
||||
|
||||
---
|
||||
|
||||
## 1. What this project is
|
||||
|
||||
A modern Python-based experiment status monitoring system for the **flOMNI**
|
||||
ptychographic tomography instrument on the **cSAXS beamline (X12SA)** at the
|
||||
**Swiss Light Source (SLS), PSI**. It replaces a legacy bash/spec-based approach.
|
||||
|
||||
Purpose: give colleagues a live view of experiment status while monitoring
|
||||
remotely, via a public-facing authenticated web page with audio alerting.
|
||||
|
||||
Owner: Mirko (beamline scientist/engineer, PSI).
|
||||
|
||||
### Repos
|
||||
- `gitea.psi.ch/OMNY/BEC` — Python / BEC side
|
||||
- `gitea.psi.ch/OMNY/Webserver` — web server files
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────┐ ┌────────────────────────────┐
|
||||
│ PSI internal network (trusted)│ HTTPS │ omny.online (IONOS) │
|
||||
│ │ POST │ lighttpd + PHP │
|
||||
│ BEC iPython session ├────────►│ - auth_gate.php / login.php│
|
||||
│ flomni_webpage_generator.py │ │ - status.html / status.json│
|
||||
│ (polls BEC every ~15s) │ │ - IP-whitelisted upload │
|
||||
└──────────────────────────────┘ └────────────────────────────┘
|
||||
▲
|
||||
fallback: Raspberry Pi "tamipi"
|
||||
(FritzBox DynDNS)
|
||||
```
|
||||
|
||||
### Producer — `flomni_webpage_generator.py`
|
||||
- Background service inside a **BEC iPython session**.
|
||||
- Polls BEC client objects every **~15 s**, generates `status.html` + `status.json`,
|
||||
uploads via **HTTP POST** to `omny.online`.
|
||||
- Plugin architecture:
|
||||
- `WebpageGeneratorBase` → `FlomniWebpageGenerator` (and stub `LamniWebpageGenerator`)
|
||||
- Factory `make_webpage_generator()` selects the plugin by BEC session name.
|
||||
- **`stop()` / `start()` cycle is required after any edit** to regenerate and
|
||||
re-upload `status.html`.
|
||||
|
||||
### Web server
|
||||
- **lighttpd** on IONOS Webhosting Plus (`omny.online`); Raspberry Pi 3B
|
||||
("tamipi") behind FritzBox DynDNS as fallback.
|
||||
- **PHP auth**: HMAC-SHA256 signed cookies. Two htpasswd files:
|
||||
- `users.htpasswd` — permanent beamline staff
|
||||
- `session.htpasswd` — the currently active rotating e-account (e.g. `p23092`)
|
||||
- **Upload endpoint** IP-whitelisted to the PSI subnet `129.129.122.0/24`.
|
||||
- **SSL**: Let's Encrypt via **webroot** authenticator. (Standalone previously
|
||||
caused *silent* renewal failures — do not go back to it.)
|
||||
- **Cookie generation-binding**: a monotonic `session.gen` counter invalidates old
|
||||
session cookies after an e-account rotation. Fixed staff users get sentinel
|
||||
`-1`, exempting them from the generation check.
|
||||
|
||||
### Monitoring
|
||||
- Raspberry Pi crontab `*/5 * * * *` runs `omny_upload_monitor.sh`, checking
|
||||
`status.json` age against a 300 s threshold.
|
||||
|
||||
---
|
||||
|
||||
## 3. BEC internals (hard-won, do not re-derive)
|
||||
|
||||
- **Scan activity**: `primary.info[0].active_request_block is not None`.
|
||||
Do **not** use the queue `status` field — it is always `'RUNNING'` while BEC is alive.
|
||||
- **Beamline states and scan interlock are independent.**
|
||||
`bec.builtin_actors.scan_interlock.states_watched` returns **accepted** (not blocking) statuses.
|
||||
- **`blocked` status** uses `primary.locks`.
|
||||
- **A block interrupts the running scan**, clearing `active_request_block`
|
||||
(→ `queue_has_active_scan = False`). So `_derive_status()` must return `blocked` on
|
||||
`beamline_blocking OR queue_locks` **independent of active scan**; only a
|
||||
fresh-heartbeat `scanning` outranks it.
|
||||
- `beamline_blocking = (enabled is True) AND any(state mismatched)` — gated on
|
||||
`enabled` so a disabled/unknown interlock never raises BLOCKED.
|
||||
- **Tomo projection count**:
|
||||
- type 1: `floor(range / stepsize) * 8` — **truncation, not rounding**; 8 sub-tomograms.
|
||||
- types 2/3: `golden_max_number_of_projections`.
|
||||
- Now generalised in JS as `calcProjections()` with kinds `equally_spaced_grid`
|
||||
and `golden_capped`, driven by the `TOMO_TYPES` data blob.
|
||||
|
||||
---
|
||||
|
||||
## 4. Multi-setup design (current)
|
||||
|
||||
Capability flags live as **class attributes** on `WebpageGeneratorBase`
|
||||
(not constructor args — so startup code is unchanged):
|
||||
|
||||
- `HAS_TOMO_QUEUE: bool` — gates the `tomo_queue` global-var read in `_cycle()`
|
||||
and the "Tomo queue" card in `_render_html()`.
|
||||
- `TOMO_TYPES: dict` — declarative config, embedded as JSON into the page; drives
|
||||
`calcProjections()`.
|
||||
|
||||
Defaults reproduce flomni behavior exactly. BEC's own primary scan queue
|
||||
(`queue_status` / `queue_locks`) and beamline states are **common — not gated**.
|
||||
|
||||
**Principle**: setup-specific behavior = class-level capability flag / declarative
|
||||
config with a flomni-preserving default. Never a hardcoded per-setup branch in JS.
|
||||
|
||||
**Principle**: when one concept appears in two places (e.g. "blocking" on the pill
|
||||
*and* the card; params in the queue *and* the instrument pane), derive both from a
|
||||
single identically-defined signal or shared helper (`buildParamRows()`) so they
|
||||
cannot drift.
|
||||
|
||||
---
|
||||
|
||||
## 5. Current sound map (flomni)
|
||||
|
||||
| Transition | Sound |
|
||||
|---|---|
|
||||
| `scanning → idle` (normal finish) | rising success arpeggio (C5-E5-G5), repeats every 30 s until Confirm |
|
||||
| `scanning → running` / other non-blocked stop | falling `warningChime`, 30 s until Confirm |
|
||||
| `scanning → blocked` | **silent**; Blocked LED pulses; `blockedChime` only after 60 min, then hourly, auto-clears |
|
||||
| feed stale / 3 failed fetches | high triple pip, 30 s until Confirm feed |
|
||||
| Enable/test button | 880-1100-880 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Known-good implementation gotchas
|
||||
|
||||
- **iOS Web Audio**: must use a **synchronous** silent 1-sample `BufferSource`
|
||||
started directly in the gesture handler, plus fire-and-forget `ctx.resume()` with
|
||||
~80 ms `setTimeout` offsets. Promise `.then()` chains **fail** outside WebKit's
|
||||
gesture scope.
|
||||
- **f-string escaping**: all JS braces must be doubled (`{{`, `}}`) in Python
|
||||
f-string HTML generation.
|
||||
- **lighttpd**: negative-lookahead regex for pass-through rules is unreliable — use
|
||||
explicit pass-through entries. `%0` in redirect rules evaluates empty if the
|
||||
parent regex didn't match — use a hardcoded hostname + `$0`.
|
||||
- **`handle_error` placement**: belongs on the **server class** (`BaseServer`
|
||||
subclass), not on the request handler — overriding it on the handler has no effect
|
||||
for this error path.
|
||||
- **Mobile viewport**: `body { min-height: 100dvh }` (with `100vh` fallback); plain
|
||||
`100vh` on iOS leaves empty scroll room below the page.
|
||||
- **File discipline**: editing stale base files has been a recurring, costly error.
|
||||
**Always verify feature presence with `grep -c` before editing any file.**
|
||||
|
||||
---
|
||||
|
||||
## 7. Security model
|
||||
|
||||
- PSI internal network is **trusted**. The primary threat is the **inbound path**
|
||||
from the public web server back into the internal network.
|
||||
- Key risks and mitigations:
|
||||
- HTTP redirect-following in `requests.post()` → SSRF. Mitigated with
|
||||
`allow_redirects=False`.
|
||||
- DNS rebinding on the DynDNS hostname when `verify=False`.
|
||||
- Unbounded response-body reading.
|
||||
- `session_query.php` / `set_password.php` form the **one** feedback loop where the
|
||||
trusted side acts on public-server responses — treat with care.
|
||||
- **Principle**: implement the safest correct solution, not a minimal patch.
|
||||
|
||||
---
|
||||
|
||||
## 8. Open items
|
||||
|
||||
### Verify on next flomni restart
|
||||
- **Blocked fix**: on the next real block, confirm the pill flips to BLOCKED and
|
||||
the LED pulses (this was the reported bug).
|
||||
- **`_CURRENT_PARAM_KEYS`**: 7 names are proven (`fovx`, `fovy`, `stitch_x`,
|
||||
`stitch_y`, `tomo_shellstep`, `tomo_countingtime`, `tomo_angle_stepsize`).
|
||||
5 are **assumptions** to verify as global vars — `tomo_type`, `tomo_angle_range`,
|
||||
`frames_per_trigger`, `single_point_instead_of_fermat_scan`,
|
||||
`ptycho_reconstruct_foldername`. Any that isn't a real global var is silently
|
||||
skipped (no row); remap in `_CURRENT_PARAM_KEYS` if a name differs.
|
||||
- **Projections parity**: the queue-card Projections number for the next job should
|
||||
be identical to before (`floor(range/step) * n_subtomos`).
|
||||
- **Let's Encrypt auto-renewal**: confirm it is functioning (~90 days after the last
|
||||
manual renewal).
|
||||
|
||||
### Producer-side, not yet implemented (from briefing 4)
|
||||
- `estimated_finish_time` in `flomni.py`
|
||||
- `tomo_start_scan_number` in `flomni.py`
|
||||
- `set_web_password()` shortcut on the `Flomni` class in `flomni.py`
|
||||
|
||||
### EPS / machine status page (in progress)
|
||||
- **Starting point**: `X_X12SA_BLStatus.ui` — flat, macro-free caQtDM beamline status
|
||||
panel covering undulator gap, EPS permit, vacuum, temperatures, monochromator /
|
||||
mirror / slit positions, BPMs, filter states.
|
||||
- **Decided architecture**:
|
||||
- **Separate daemon**, decoupled from the main webpage generator.
|
||||
- **CA monitors preferred over polling.**
|
||||
- **Two-tier update cadence**: fast (shutters, EPS-OK, ring current) / slow
|
||||
(vacuum, temperature).
|
||||
- **Card/table layout first**; SVG synoptic deferred.
|
||||
- **Blocked on**: ~110 unresolved PVs from `caInclude` sub-templates
|
||||
(`EPS_Temp.ui`, `EPS_Gauge.ui`, `EPS_IonPump.ui`, `EPS_Valve.ui`, `EPS_BST.ui`, …)
|
||||
— need the template files.
|
||||
- **Next deliverables requested**: `index.html`, the JS, and the JSON schema files,
|
||||
so the PV manifest and `eps_status.json` schema can be built.
|
||||
- **Consider**: the PSI **EPICS archiver REST interface** to sidestep CA firewall
|
||||
concerns.
|
||||
|
||||
### Deferred
|
||||
- **LamNI**: fill in `LamniWebpageGenerator` — real `tomo_type` key values in
|
||||
`TOMO_TYPES`, device paths / `_collect_setup_data()`, `LamNI.png`. Add a
|
||||
session-name branch to `make_webpage_generator()`.
|
||||
- **Omny**: new subclass; `TOMO_TYPES` with a public 2-subtomo type plus a hidden
|
||||
8-subtomo type restricted to staff. The staff-only visibility mechanism is
|
||||
**still to be designed** (likely a role/visibility field filtered out before the
|
||||
config reaches a standard user's page). Add to the factory.
|
||||
- **SVG synoptic view** for the EPS page.
|
||||
- **Edge case (noted only)**: `scanning → blocked → idle` directly (block clears
|
||||
exactly as the scan finishes) yields no success tone, since the transition into
|
||||
idle came from `blocked`. Normal flow clears back to `scanning` first, so the
|
||||
success tone fires on the real finish. Revisit only if BEC ever exposes an
|
||||
explicit scan-aborted signal.
|
||||
|
||||
---
|
||||
|
||||
## 9. Tools & environment
|
||||
|
||||
- **Languages**: Python (BEC iPython env), PHP (lighttpd), JavaScript (frontend), Bash
|
||||
- **Python libs**: `requests` (upload), `Pillow` (thumbnails), `http.server`
|
||||
(local server, port 8080)
|
||||
- **PHP**: `password_hash()` / `password_verify()` (bcrypt)
|
||||
- **Beamline systems**: EPICS (PVs), caQtDM (`.ui` panels), BEC client
|
||||
(`bec.queue`, `bec.get_global_var`, `bec.builtin_actors.scan_interlock`)
|
||||
- **Infra**: lighttpd, certbot / Let's Encrypt (webroot), IONOS Webhosting Plus,
|
||||
Raspberry Pi 3B, PSI Gitea
|
||||
|
||||
---
|
||||
|
||||
## 10. Working style
|
||||
|
||||
- Discuss architecture and confirm technical details **before** implementing code.
|
||||
- Live testing happens on the actual beamline system; results are reported back
|
||||
iteratively.
|
||||
- Write a project briefing `.md` after major sessions to enable seamless continuation.
|
||||
- Prefer clean, non-redundant UI; be decisive about **removing** elements rather than
|
||||
replacing them with alternatives.
|
||||
- Decoupled daemon architecture for new monitoring components.
|
||||
@@ -1,10 +1,12 @@
|
||||
import builtins
|
||||
import datetime
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import h5py
|
||||
@@ -1619,7 +1621,7 @@ class _TomoQueueProxy:
|
||||
Each entry is a plain dict: ``{"label": ..., "params": {...}, "status":
|
||||
..., "added_at": ...}``, where ``params`` holds a snapshot of all
|
||||
global-var-backed tomo scan parameters (see
|
||||
``Flomni._TOMO_QUEUE_PARAM_NAMES``). Stored as a single list under
|
||||
``Flomni._TOMO_SCAN_PARAM_NAMES``). Stored as a single list under
|
||||
``tomo_queue`` so the queue is visible from any BEC client session via
|
||||
``client.get_global_var("tomo_queue")`` and survives a kernel restart.
|
||||
"""
|
||||
@@ -1659,6 +1661,53 @@ class _TomoQueueProxy:
|
||||
jobs[index].update(kwargs)
|
||||
self._save(jobs)
|
||||
|
||||
def update_by_id(self, job_id: str, **kwargs) -> bool:
|
||||
"""Update the job whose ``id`` matches ``job_id``.
|
||||
|
||||
Returns True if a matching job was found and updated, False otherwise
|
||||
(e.g. the job was deleted concurrently). Addressing a job by its stable
|
||||
id rather than its list index makes status writes safe against queue
|
||||
edits made while the queue runs: a reorder, or a delete of some *other*
|
||||
job, can no longer misdirect a status write to the wrong entry.
|
||||
"""
|
||||
jobs = self._load()
|
||||
for job in jobs:
|
||||
if job.get("id") == job_id:
|
||||
job.update(kwargs)
|
||||
self._save(jobs)
|
||||
return True
|
||||
return False
|
||||
|
||||
def move(self, index: int, new_index: int) -> None:
|
||||
"""Move the job at ``index`` to ``new_index``, shifting the rest.
|
||||
|
||||
Pure list mechanics -- no status checks. Callers (e.g.
|
||||
``Flomni.tomo_queue_move``) are responsible for deciding whether a
|
||||
given move is allowed (running-job floor, movable statuses, ...).
|
||||
"""
|
||||
jobs = self._load()
|
||||
job = jobs.pop(index)
|
||||
jobs.insert(new_index, job)
|
||||
self._save(jobs)
|
||||
|
||||
def ensure_ids(self) -> list:
|
||||
"""Assign a stable id to any job that lacks one and persist the result.
|
||||
|
||||
Back-compat shim for queues written before jobs carried an ``id`` field
|
||||
(and for any job added by older code). Called once at the top of
|
||||
tomo_queue_execute() so the pick-next executor can address every job by
|
||||
id. Returns the (possibly healed) job list.
|
||||
"""
|
||||
jobs = self._load()
|
||||
changed = False
|
||||
for job in jobs:
|
||||
if not job.get("id"):
|
||||
job["id"] = uuid.uuid4().hex
|
||||
changed = True
|
||||
if changed:
|
||||
self._save(jobs)
|
||||
return jobs
|
||||
|
||||
def clear(self) -> None:
|
||||
self._save([])
|
||||
|
||||
@@ -1731,6 +1780,11 @@ class Flomni(
|
||||
self.special_angle_tolerance = 20
|
||||
self._current_special_angles = []
|
||||
self._beam_is_okay = True
|
||||
# Session-only (not global-var-backed): the hooks themselves are
|
||||
# Python callables, which can't be persisted the way params/queue
|
||||
# jobs are. Only the *name* selecting one is ever snapshotted (see
|
||||
# at_each_angle_hook / register_at_each_angle_hook()).
|
||||
self._at_each_angle_hooks: dict = {}
|
||||
self._progress_proxy = _ProgressProxy(self.client)
|
||||
# Deliberately NOT calling reset() here: this dict is persisted via a
|
||||
# BEC global var specifically so it survives a kernel restart (e.g.
|
||||
@@ -1742,6 +1796,13 @@ class Flomni(
|
||||
# below); use tomo_progress_reset() if you want to explicitly clear
|
||||
# stale progress without starting a new scan.
|
||||
self._tomo_queue_proxy = _TomoQueueProxy(self.client)
|
||||
# Published so a future GUI never needs a hardcoded mirror of the
|
||||
# action registry -- see _describe_tomo_queue_actions().
|
||||
self.client.set_global_var("tomo_queue_actions", self._describe_tomo_queue_actions())
|
||||
# Reset to empty on every Flomni() -- hooks are session-only (see
|
||||
# self._at_each_angle_hooks above), so a stale list published by a
|
||||
# since-ended session must not linger and look registered.
|
||||
self._publish_at_each_angle_hooks()
|
||||
from csaxs_bec.bec_ipython_client.plugins.flomni.flomni_webpage_generator import (
|
||||
FlomniWebpageGenerator,
|
||||
)
|
||||
@@ -1956,6 +2017,12 @@ class Flomni(
|
||||
|
||||
manual_shift_y = _GlobalVarParam(0.0)
|
||||
|
||||
# Name of a registered at_each_angle hook (see register_at_each_angle_hook()),
|
||||
# or None for the default per-projection behaviour. Snapshotted like any
|
||||
# other tomo scan parameter, so a tomo-queue job can carry its own hook
|
||||
# selection -- see _at_each_angle().
|
||||
at_each_angle_hook = _GlobalVarParam(None)
|
||||
|
||||
@property
|
||||
def single_point_random_shift_max(self):
|
||||
"""Maximum absolute random shift [um] applied independently in x and y
|
||||
@@ -2118,6 +2185,18 @@ class Flomni(
|
||||
|
||||
@single_point_instead_of_fermat_scan.setter
|
||||
def single_point_instead_of_fermat_scan(self, val: bool):
|
||||
# Fire the reminder right at the moment it's switched on, not just
|
||||
# when tomo_parameters() happens to be called afterward. Gated on
|
||||
# the off->on transition (reading the property itself for the old
|
||||
# value, before the write below) so restoring the same True value
|
||||
# across consecutive queued jobs doesn't reprint it every time.
|
||||
if val and not self.single_point_instead_of_fermat_scan:
|
||||
print(
|
||||
"\x1b[93mNote: with single-point mode on, acquire a single projection "
|
||||
"(including inside a custom at_each_angle hook) with "
|
||||
"tomo_acquire_at_angle(angle), not tomo_scan_projection(angle) -- the "
|
||||
"latter always runs a full Fermat scan regardless of this setting.\x1b[0m"
|
||||
)
|
||||
self.client.set_global_var("single_point_instead_of_fermat_scan", val)
|
||||
|
||||
@property
|
||||
@@ -2200,9 +2279,7 @@ class Flomni(
|
||||
f"Alignment scan numbers: {scan_list_str}\n"
|
||||
)
|
||||
print(scilog_content)
|
||||
bec.messaging.scilog.new().add_text(scilog_content.replace("\n", "<br>")).add_tags(
|
||||
"alignmentscan"
|
||||
).send()
|
||||
self._scilog_write(scilog_content, "alignmentscan")
|
||||
|
||||
def write_alignment_scan_numbers(self, first_scan):
|
||||
import os
|
||||
@@ -2362,7 +2439,20 @@ class Flomni(
|
||||
# finally do the scan at this angle
|
||||
self._tomo_scan_at_angle(angle, subtomo_number)
|
||||
|
||||
@scan_repeat(max_repeats=10, default=True)
|
||||
@staticmethod
|
||||
def _retry_unless_flomni_error(exc: Exception, attempt: int) -> bool:
|
||||
"""scan_repeat() exc_handler: retry any exception except FlomniError.
|
||||
|
||||
FlomniError marks a definite, non-transient failure (bad config,
|
||||
unmet precondition, ...) -- e.g. _at_each_angle() raises it when a
|
||||
job's at_each_angle_hook name isn't registered in this session.
|
||||
Retrying that up to max_repeats times just burns 10 attempts before
|
||||
surfacing a much less informative TooManyScanRestarts, instead of
|
||||
the actual, actionable error message immediately.
|
||||
"""
|
||||
return not isinstance(exc, FlomniError)
|
||||
|
||||
@scan_repeat(max_repeats=10, default=True, exc_handler=_retry_unless_flomni_error)
|
||||
def _tomo_scan_at_angle(self, angle, subtomo_number):
|
||||
|
||||
if 0 <= angle < self.tomo_angle_range + 0.05:
|
||||
@@ -2675,6 +2765,9 @@ class Flomni(
|
||||
f"Measurement ID: {self.tomo_id}",
|
||||
f"Sample: {self.sample_name}",
|
||||
]
|
||||
hook_description = self._describe_active_hook()
|
||||
if hook_description:
|
||||
timing_lines.append(f"At-each-angle hook: {hook_description}")
|
||||
if elapsed_s is not None:
|
||||
timing_lines.append(f"Total measurement time: {self._format_duration(elapsed_s)}")
|
||||
timing_lines.append(
|
||||
@@ -2686,9 +2779,12 @@ class Flomni(
|
||||
for line in timing_lines[3:]:
|
||||
print(line)
|
||||
timing_content = "\n".join(timing_lines)
|
||||
bec.messaging.scilog.new().add_text(timing_content.replace("\n", "<br>")).add_tags(
|
||||
"tomoscan"
|
||||
).send()
|
||||
hook_source = self._active_hook_source()
|
||||
if hook_source:
|
||||
timing_content += (
|
||||
f"\n\nAt-each-angle hook source ('{self.at_each_angle_hook}'):\n{hook_source}"
|
||||
)
|
||||
self._scilog_write(timing_content, "tomoscan")
|
||||
|
||||
def tomo_scan_resume(self) -> None:
|
||||
"""Resume a tomo_scan() that crashed or was interrupted, picking up
|
||||
@@ -2751,6 +2847,29 @@ class Flomni(
|
||||
return f"{m}m {s:02d}s"
|
||||
return f"{s}s"
|
||||
|
||||
def _scilog_write(self, text: str, tags: str | list) -> bool:
|
||||
"""Write a text entry to scilog, tolerating scilog being disabled or
|
||||
unreachable. This is documentation of a scan, not part of the scan
|
||||
itself -- it must never fail a caller (in particular a tomo_queue
|
||||
job, which would otherwise be marked "incomplete" and pause the
|
||||
whole queue over a logging side effect that has nothing to do with
|
||||
whether the tomogram actually succeeded).
|
||||
|
||||
Returns True if the entry was sent, False if it was skipped/failed
|
||||
(already logged as a warning either way -- callers that don't care
|
||||
about the outcome can ignore the return value).
|
||||
"""
|
||||
scilog = getattr(bec.messaging, "scilog", None)
|
||||
if scilog is None or not getattr(scilog, "_enabled", False):
|
||||
logger.warning("scilog is not enabled; skipping scilog entry.")
|
||||
return False
|
||||
try:
|
||||
scilog.new().add_text(text.replace("\n", "<br>")).add_tags(tags).send()
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.warning(f"Failed to write to scilog: {exc}")
|
||||
return False
|
||||
|
||||
def _print_progress(self):
|
||||
# --- compute and store estimated remaining time -----------------------
|
||||
start_str = self.progress.get("tomo_start_time")
|
||||
@@ -2801,6 +2920,19 @@ class Flomni(
|
||||
)
|
||||
|
||||
def _at_each_angle(self, angle: float) -> None:
|
||||
hook_name = self.at_each_angle_hook
|
||||
if hook_name:
|
||||
hook = self._at_each_angle_hooks.get(hook_name)
|
||||
if hook is None:
|
||||
raise FlomniError(
|
||||
f"at_each_angle_hook '{hook_name}' is not registered in this "
|
||||
"session. Hooks are session-only and do not survive a kernel "
|
||||
f"restart -- call register_at_each_angle_hook({hook_name!r}, "
|
||||
"<func>) again, then re-run tomo_queue_execute() to resume."
|
||||
)
|
||||
hook(self, angle)
|
||||
return
|
||||
|
||||
if "flomni_at_each_angle" in builtins.__dict__:
|
||||
# pylint: disable=undefined-variable
|
||||
flomni_at_each_angle(self, angle)
|
||||
@@ -2812,6 +2944,109 @@ class Flomni(
|
||||
|
||||
self.tomo_scan_projection(angle, _internal=True)
|
||||
|
||||
def register_at_each_angle_hook(self, name: str, func) -> None:
|
||||
"""Register a per-projection hook function under ``name``, so a
|
||||
tomo-queue job can select it by name via the ``at_each_angle_hook``
|
||||
parameter and have it run automatically -- unlike the older
|
||||
``flomni_at_each_angle`` builtins-based hook, which the queue can't
|
||||
see (it isn't a snapshotted parameter, so a queued job has no way
|
||||
to record "use this hook").
|
||||
|
||||
``func`` is called as ``func(self, angle)`` at every projection
|
||||
angle, exactly like the interactive ``flomni_at_each_angle`` hook --
|
||||
write it the same way, just give it a name and register it here
|
||||
instead of assigning it to the ``flomni_at_each_angle`` builtin.
|
||||
|
||||
Hooks are **session-only**: they live in this ``Flomni`` instance's
|
||||
memory, not in a global var, so they do NOT survive a kernel
|
||||
restart. A queued job that references a hook by name raises a clear
|
||||
error at execution time if the name isn't registered in the session
|
||||
running the queue -- re-run this registration call, then resume
|
||||
``tomo_queue_execute()``.
|
||||
|
||||
Example -- record a projection, insert a polarizer, record again::
|
||||
|
||||
def polarizer_modulation(flomni, angle):
|
||||
flomni.tomo_scan_projection(angle, _internal=True)
|
||||
umv(dev.polarizer, "in")
|
||||
flomni.tomo_scan_projection(angle, _internal=True)
|
||||
umv(dev.polarizer, "out")
|
||||
|
||||
flomni.register_at_each_angle_hook("polarizer_mod", polarizer_modulation)
|
||||
flomni.at_each_angle_hook = "polarizer_mod"
|
||||
flomni.tomo_queue_add(label="polarizer run")
|
||||
flomni.at_each_angle_hook = None # back to default for the next job
|
||||
"""
|
||||
if not callable(func):
|
||||
raise TypeError(f"register_at_each_angle_hook: {func!r} is not callable.")
|
||||
self._at_each_angle_hooks[name] = func
|
||||
self._publish_at_each_angle_hooks()
|
||||
print(f"Registered at_each_angle hook '{name}'.")
|
||||
|
||||
def unregister_at_each_angle_hook(self, name: str) -> None:
|
||||
"""Remove a previously registered at_each_angle hook by name."""
|
||||
if self._at_each_angle_hooks.pop(name, None) is None:
|
||||
print(f"No at_each_angle hook named '{name}' is registered.")
|
||||
else:
|
||||
self._publish_at_each_angle_hooks()
|
||||
print(f"Unregistered at_each_angle hook '{name}'.")
|
||||
|
||||
def _publish_at_each_angle_hooks(self) -> None:
|
||||
"""Publish the names of all currently registered at_each_angle
|
||||
hooks as the ``tomo_at_each_angle_hooks`` global var (a plain list
|
||||
of strings) -- so a GUI (or any other client) can build a dropdown
|
||||
from it directly, no hardcoded mirror, same reasoning as
|
||||
``tomo_queue_actions`` for the command-job registry (see
|
||||
_describe_tomo_queue_actions()). Unlike that registry this list
|
||||
changes at runtime, whenever register_/unregister_at_each_angle_hook()
|
||||
is called -- not just once at __init__.
|
||||
"""
|
||||
self.client.set_global_var("tomo_at_each_angle_hooks", sorted(self._at_each_angle_hooks))
|
||||
|
||||
def list_at_each_angle_hooks(self) -> list:
|
||||
"""Print and return the names of all at_each_angle hooks registered
|
||||
in this session. The active one (``self.at_each_angle_hook``, if
|
||||
any) is marked."""
|
||||
names = sorted(self._at_each_angle_hooks)
|
||||
if not names:
|
||||
print("No at_each_angle hooks registered.")
|
||||
else:
|
||||
for name in names:
|
||||
marker = " (active)" if name == self.at_each_angle_hook else ""
|
||||
print(f" {name}{marker}")
|
||||
return names
|
||||
|
||||
def _describe_active_hook(self) -> str | None:
|
||||
"""Human-readable description of ``self.at_each_angle_hook`` for
|
||||
display in tomo_parameters()/scilog/the PDF report: the name, or
|
||||
the name flagged as unregistered if it doesn't (or no longer)
|
||||
resolve in this session -- e.g. unregistered after the job that
|
||||
used it was added, or set by a session that has since restarted.
|
||||
Returns None if no hook is set at all.
|
||||
"""
|
||||
hook = self.at_each_angle_hook
|
||||
if not hook:
|
||||
return None
|
||||
if hook in self._at_each_angle_hooks:
|
||||
return hook
|
||||
return f"{hook} (NOT registered in this session)"
|
||||
|
||||
def _active_hook_source(self) -> str | None:
|
||||
"""Source code of the currently active at_each_angle hook, for
|
||||
inclusion in the scilog/PDF record -- so what a tomogram actually
|
||||
did is part of the permanent log, not just the hook's name.
|
||||
Returns None if there's no active hook, it isn't registered in
|
||||
this session, or its source can't be retrieved (e.g. it wasn't
|
||||
defined via a normal `def`/interactive cell).
|
||||
"""
|
||||
hook_func = self._at_each_angle_hooks.get(self.at_each_angle_hook)
|
||||
if hook_func is None:
|
||||
return None
|
||||
try:
|
||||
return inspect.getsource(hook_func)
|
||||
except (OSError, TypeError):
|
||||
return None
|
||||
|
||||
def _golden(self, ii, howmany_sorted, maxangle, reverse=False):
|
||||
"""returns the iis golden ratio angle of sorted bunches of howmany_sorted and its subtomo number"""
|
||||
golden = []
|
||||
@@ -2957,7 +3192,7 @@ class Flomni(
|
||||
"""Write one per-tomogram timing record at the end of a tomo_scan().
|
||||
|
||||
Captures a full snapshot of the scan parameters (reusing
|
||||
_TOMO_QUEUE_PARAM_NAMES -- the single source of truth for "every
|
||||
_TOMO_SCAN_PARAM_NAMES -- the single source of truth for "every
|
||||
parameter that affects how the scan runs") together with the
|
||||
tomogram-level wall-clock timing already tracked in self.progress:
|
||||
total elapsed, the accumulated idle time detected from inter-
|
||||
@@ -2986,7 +3221,7 @@ class Flomni(
|
||||
"active_s": active_s,
|
||||
"total_projections": self.progress.get("total_projections"),
|
||||
"tomo_type_label": self.progress.get("tomo_type"),
|
||||
"params": {name: getattr(self, name) for name in self._TOMO_QUEUE_PARAM_NAMES},
|
||||
"params": {name: getattr(self, name) for name in self._TOMO_SCAN_PARAM_NAMES},
|
||||
}
|
||||
self._append_timing_record(self._TOMOGRAM_TIMING_LOG, record)
|
||||
|
||||
@@ -3087,10 +3322,7 @@ class Flomni(
|
||||
content = "\n".join(lines)
|
||||
print(content)
|
||||
|
||||
bec = builtins.__dict__.get("bec")
|
||||
bec.messaging.scilog.new().add_text(content.replace("\n", "<br>")).add_tags(
|
||||
"tomoscan"
|
||||
).send()
|
||||
self._scilog_write(content, "tomoscan")
|
||||
|
||||
def _write_tomo_scan_number(self, scan_number: int, angle: float, subtomo_number: int) -> None:
|
||||
tomo_scan_numbers_file = os.path.expanduser("~/data/raw/logs/tomography_scannumbers.txt")
|
||||
@@ -3121,8 +3353,28 @@ class Flomni(
|
||||
" acquisition at this angle use tomo_acquire_at_angle(angle) instead.\x1b[0m"
|
||||
)
|
||||
if not self.OMNYTools.yesno("Run a fermat scan anyway?", "n"):
|
||||
print("Aborted. Use tomo_acquire_at_angle(angle) for a single-point acquisition.")
|
||||
return
|
||||
# Raise, don't silently return: this runs inside
|
||||
# tomo_scan()'s per-angle loop (directly, or via a custom
|
||||
# at_each_angle hook) as often as not, and a bare `return`
|
||||
# here means the projection that was supposed to happen
|
||||
# simply never did -- no exception, no incomplete status,
|
||||
# no trace that data is now missing from the tomogram. A
|
||||
# real incident: a hook that called this without
|
||||
# _internal=True got asked at every angle, declining just
|
||||
# skipped each one silently, and the eventual Ctrl-C (which
|
||||
# tomo_queue_execute()'s `except Exception` does NOT catch,
|
||||
# since KeyboardInterrupt isn't an Exception) left the job
|
||||
# stuck at status "running" forever with no process behind
|
||||
# it. Raising here means a decline now correctly fails the
|
||||
# job, marks it "incomplete", and pauses the queue --
|
||||
# exactly the crash-resume contract tomo_queue_execute()
|
||||
# already documents.
|
||||
raise FlomniError(
|
||||
"tomo_scan_projection: declined to run a Fermat scan while "
|
||||
"single_point_instead_of_fermat_scan is on. Use "
|
||||
"tomo_acquire_at_angle(angle) for a single-point acquisition, "
|
||||
"or pass _internal=True to force the Fermat scan without asking."
|
||||
)
|
||||
|
||||
dev.rtx.controller.laser_tracker_check_signalstrength()
|
||||
|
||||
@@ -3320,6 +3572,14 @@ class Flomni(
|
||||
print(f"Single point instead of fermat = {self.single_point_instead_of_fermat_scan}")
|
||||
if self.single_point_instead_of_fermat_scan:
|
||||
print(f"Single point random shift max <um> = {self.single_point_random_shift_max}")
|
||||
print(
|
||||
"\x1b[93mNote: with single-point mode on, acquire a single projection "
|
||||
"(including inside a custom at_each_angle hook) with "
|
||||
"tomo_acquire_at_angle(angle), not tomo_scan_projection(angle) -- the "
|
||||
"latter always runs a full Fermat scan regardless of this setting.\x1b[0m"
|
||||
)
|
||||
if self.at_each_angle_hook:
|
||||
print(f"At-each-angle hook = {self._describe_active_hook()}")
|
||||
print("")
|
||||
|
||||
if self.tomo_type == 1:
|
||||
@@ -3510,10 +3770,11 @@ class Flomni(
|
||||
|
||||
# Ordered set of all global-var-backed tomo scan parameters: exactly the
|
||||
# settings shown by tomo_parameters(), plus manual_shift_y/
|
||||
# tomo_stitch_overlap/corridor_size which also affect the scan but are
|
||||
# only set directly as properties. This is the full "parameter set" that
|
||||
# tomo_queue_add()/tomo_queue_execute() snapshot and restore.
|
||||
_TOMO_QUEUE_PARAM_NAMES = (
|
||||
# tomo_stitch_overlap/corridor_size/at_each_angle_hook which also affect
|
||||
# the scan but are only set directly as properties. This is the full
|
||||
# "parameter set" that tomo_queue_add()/tomo_queue_execute() snapshot
|
||||
# and restore.
|
||||
_TOMO_SCAN_PARAM_NAMES = (
|
||||
"tomo_countingtime",
|
||||
"tomo_shellstep",
|
||||
"fovx",
|
||||
@@ -3534,8 +3795,58 @@ class Flomni(
|
||||
"golden_projections_at_0_deg_for_damage_estimation",
|
||||
"zero_deg_reference_at_each_subtomo",
|
||||
"corridor_size",
|
||||
"at_each_angle_hook",
|
||||
)
|
||||
|
||||
# Curated allow-list for the "move" queue action -- NOT "any device in
|
||||
# dev". Only devices actually reconfigured between tomograms belong here;
|
||||
# this is the difference between a reconfiguration tool and a remote
|
||||
# control for the whole endstation. mokev/idgap are real front-end
|
||||
# devices (bl_frontend.yaml, not the flomni sim config); ftray (sample
|
||||
# transfer tray, harmless to move, untouched by tomo-scan code) is
|
||||
# included so the move action has a sim-testable target.
|
||||
_TOMO_QUEUE_MOVE_DEVICES = {
|
||||
"mokev": {"label": "Energy", "unit": "keV"},
|
||||
"idgap": {"label": "Undulator gap", "unit": "mm"},
|
||||
"ftray": {"label": "Sample transfer tray (sim-testable)", "unit": "mm"},
|
||||
}
|
||||
|
||||
# Named-action registry for tomo-queue command jobs. Each entry maps a
|
||||
# name to a bound Flomni method (arbitrary, reviewed Python -- a
|
||||
# "script") plus enough metadata to validate and render it generically:
|
||||
# `func` is the bound method name, `idempotent` is the default safe-to-
|
||||
# rerun-after-a-crash flag, `help` is operator-facing, `params` is a
|
||||
# schema of the method's kwargs (possibly empty). Adding a new script is
|
||||
# exactly: write a new bound method + one new entry here -- nothing else
|
||||
# (tomo_queue_add_command/_show/_run_command_job, the tomo_queue_actions
|
||||
# global var) needs to change. Deliberately minimal for now: no
|
||||
# foptics_in/out, feye_in/out, or shutter actions yet, even though the
|
||||
# first two already exist as callable methods -- those are future
|
||||
# "dedicated scripts" to add later as their own reviewed changes.
|
||||
_TOMO_QUEUE_ACTIONS = {
|
||||
"move": {
|
||||
"func": "_queue_action_move",
|
||||
"idempotent": True, # absolute moves re-run harmlessly
|
||||
"help": "Move device(s) to absolute position(s).",
|
||||
"params": {"positions": {"type": "device_positions"}},
|
||||
},
|
||||
"optimize_idgap": {
|
||||
"func": "optimize_idgap",
|
||||
"idempotent": True,
|
||||
"help": "Scan idgap over a search range and move to the peak. STUB: no-op.",
|
||||
"params": {
|
||||
"search_range": {
|
||||
"type": "float",
|
||||
"default": 0.5,
|
||||
"min": 0.0,
|
||||
"max": 2.0,
|
||||
"unit": "mm",
|
||||
"label": "Search range",
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def tomo_queue_add(self, label: str = None) -> int:
|
||||
"""Snapshot the currently set tomo parameters and append them as a
|
||||
new job to the tomo queue (persisted, survives a kernel restart).
|
||||
@@ -3556,9 +3867,11 @@ class Flomni(
|
||||
Returns:
|
||||
The index of the newly added job.
|
||||
"""
|
||||
params = {name: getattr(self, name) for name in self._TOMO_QUEUE_PARAM_NAMES}
|
||||
params = {name: getattr(self, name) for name in self._TOMO_SCAN_PARAM_NAMES}
|
||||
index = len(self._tomo_queue_proxy)
|
||||
job = {
|
||||
"kind": "tomo",
|
||||
"id": uuid.uuid4().hex,
|
||||
"label": label or f"job_{index + 1}",
|
||||
"params": params,
|
||||
"status": "pending",
|
||||
@@ -3568,6 +3881,164 @@ class Flomni(
|
||||
print(f"Added tomo queue job #{index} ({job['label']}).")
|
||||
return index
|
||||
|
||||
def tomo_queue_add_command(self, steps, label: str = None, idempotent: bool = None) -> int:
|
||||
"""Append a "command" job -- an ordered list of named-registry actions
|
||||
that reconfigure the beamline (move a device, ...) -- to the tomo
|
||||
queue.
|
||||
|
||||
Unlike tomo_queue_add(), this does not snapshot the current tomo scan
|
||||
parameters; a command job has no "params" key at all.
|
||||
|
||||
Typical usage::
|
||||
|
||||
flomni.tomo_queue_add_command(
|
||||
[{"action": "move", "kwargs": {"positions": {"mokev": 6.2}}},
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 0.5}}],
|
||||
label="reconfigure to 6.2 keV",
|
||||
)
|
||||
|
||||
Args:
|
||||
steps: A single {"action": ..., "kwargs": {...}} dict, or a list
|
||||
of them, run in sequence within this one job.
|
||||
label: Optional name for the job, shown by tomo_queue_show().
|
||||
Defaults to "job_<n>".
|
||||
idempotent: Overrides the default of "every step's registry
|
||||
idempotent flag, ANDed together". Set this explicitly if the
|
||||
registry defaults don't capture whether this particular
|
||||
sequence is safe to blindly re-run after a crash.
|
||||
|
||||
Returns:
|
||||
The index of the newly added job.
|
||||
"""
|
||||
if isinstance(steps, dict):
|
||||
steps = [steps]
|
||||
if not steps:
|
||||
raise ValueError("tomo_queue_add_command: at least one step is required.")
|
||||
|
||||
resolved_steps = []
|
||||
all_idempotent = True
|
||||
for step in steps:
|
||||
action_name = step.get("action")
|
||||
kwargs = step.get("kwargs", {})
|
||||
self._validate_action_kwargs(action_name, kwargs)
|
||||
all_idempotent = all_idempotent and self._TOMO_QUEUE_ACTIONS[action_name]["idempotent"]
|
||||
resolved_steps.append({"action": action_name, "kwargs": kwargs})
|
||||
|
||||
try:
|
||||
json.dumps(resolved_steps)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError(f"Command job steps must be JSON-serializable: {exc}") from exc
|
||||
|
||||
index = len(self._tomo_queue_proxy)
|
||||
job = {
|
||||
"kind": "command",
|
||||
"id": uuid.uuid4().hex,
|
||||
"label": label or f"job_{index + 1}",
|
||||
"steps": resolved_steps,
|
||||
"idempotent": all_idempotent if idempotent is None else idempotent,
|
||||
"status": "pending",
|
||||
"added_at": datetime.datetime.now().isoformat(),
|
||||
}
|
||||
self._tomo_queue_proxy.append(job)
|
||||
print(f"Added command queue job #{index} ({job['label']}).")
|
||||
return index
|
||||
|
||||
def _validate_action_kwargs(self, action_name: str, kwargs: dict) -> None:
|
||||
"""Add-time schema validation for a command-job step's kwargs.
|
||||
|
||||
Ergonomic, not the safety boundary: catches a typo or an out-of-range
|
||||
value now instead of at 2am. Each action method re-validates its own
|
||||
arguments against live hardware at execution time -- that's the
|
||||
actual safety boundary, since the schema and the hardware can
|
||||
disagree (limits changed; a value arrived via the CLI, bypassing
|
||||
this).
|
||||
"""
|
||||
spec = self._TOMO_QUEUE_ACTIONS.get(action_name)
|
||||
if spec is None:
|
||||
raise ValueError(f"Unknown queue action '{action_name}'.")
|
||||
schema = spec["params"]
|
||||
for key in kwargs:
|
||||
if key not in schema:
|
||||
raise ValueError(f"Action '{action_name}' has no parameter '{key}'.")
|
||||
for name, field in schema.items():
|
||||
if name not in kwargs:
|
||||
continue
|
||||
value = kwargs[name]
|
||||
field_type = field.get("type")
|
||||
if field_type == "float":
|
||||
if not isinstance(value, (int, float)) or isinstance(value, bool):
|
||||
raise ValueError(f"{action_name}.{name} must be a number, got {value!r}.")
|
||||
lo, hi = field.get("min"), field.get("max")
|
||||
if lo is not None and value < lo:
|
||||
raise ValueError(f"{action_name}.{name}={value} is below the minimum {lo}.")
|
||||
if hi is not None and value > hi:
|
||||
raise ValueError(f"{action_name}.{name}={value} is above the maximum {hi}.")
|
||||
elif field_type == "device_positions":
|
||||
if not isinstance(value, dict) or not value:
|
||||
raise ValueError(
|
||||
f"{action_name}.{name} must be a non-empty {{device: target}} dict."
|
||||
)
|
||||
for device_name in value:
|
||||
if device_name not in self._TOMO_QUEUE_MOVE_DEVICES:
|
||||
raise ValueError(
|
||||
f"Queue move: device '{device_name}' is not queue-movable."
|
||||
)
|
||||
|
||||
def _queue_action_move(self, positions: dict) -> None:
|
||||
"""positions: {device_name: absolute_target, ...}. Absolute umv only.
|
||||
|
||||
Re-checks the allow-list at execution time (not just add time, see
|
||||
_validate_action_kwargs) -- the actual safety boundary, since a value
|
||||
could have arrived via the CLI, bypassing tomo_queue_add_command's
|
||||
validation.
|
||||
"""
|
||||
args = []
|
||||
for name, target in positions.items():
|
||||
if name not in self._TOMO_QUEUE_MOVE_DEVICES:
|
||||
raise ValueError(f"Queue move: device '{name}' is not queue-movable.")
|
||||
if name not in dev:
|
||||
raise ValueError(f"Queue move: unknown device '{name}'.")
|
||||
args += [dev[name], target]
|
||||
umv(*args)
|
||||
|
||||
def optimize_idgap(self, search_range: float = 0.5) -> None:
|
||||
"""Scan idgap over +/- search_range around its current position and
|
||||
move to the peak of <signal not yet decided>.
|
||||
|
||||
STUB -- not implemented. The peak-detection signal/device is not yet
|
||||
decided (see AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md section 10).
|
||||
This is the ONLY place that needs to change to make this a real
|
||||
action: the registry entry, resume policy, and (eventually) GUI
|
||||
wiring already treat it as a normal idempotent compound action, so
|
||||
dropping in the real scan-and-move-to-peak logic here (or swapping
|
||||
which device/signal it reads) is a self-contained change.
|
||||
"""
|
||||
print(
|
||||
f"optimize_idgap(search_range={search_range}): not yet implemented -- "
|
||||
"peak-detection signal not yet decided. No-op."
|
||||
)
|
||||
|
||||
def _describe_tomo_queue_actions(self) -> dict:
|
||||
"""Serializable description of the action registry + move allow-list.
|
||||
|
||||
Published as the ``tomo_queue_actions`` global var (see __init__) so
|
||||
the GUI (or any other client) can build its dropdowns/forms from it
|
||||
directly -- no hardcoded mirror to drift, which is exactly the
|
||||
problem the QUEUE_PARAM_NAMES mirror between this file and
|
||||
tomo_params.py already has.
|
||||
"""
|
||||
return {
|
||||
"actions": {
|
||||
name: {
|
||||
"idempotent": spec["idempotent"],
|
||||
"help": spec["help"],
|
||||
"params": spec["params"],
|
||||
}
|
||||
for name, spec in self._TOMO_QUEUE_ACTIONS.items()
|
||||
},
|
||||
"move_devices": dict(self._TOMO_QUEUE_MOVE_DEVICES),
|
||||
}
|
||||
|
||||
def tomo_queue_delete(self, *indices: int) -> None:
|
||||
"""Delete one or more jobs from the tomo queue by index.
|
||||
|
||||
@@ -3584,6 +4055,48 @@ class Flomni(
|
||||
job = self._tomo_queue_proxy.pop(index)
|
||||
print(f"Deleted tomo queue job #{index} ({job['label']}).")
|
||||
|
||||
def tomo_queue_move(self, index: int, new_index: int) -> None:
|
||||
"""Move a pending tomo queue job to a new position.
|
||||
|
||||
Only ``pending`` jobs can be reordered -- a ``running``,
|
||||
``incomplete``, or ``done`` job stays put. ``incomplete`` is a floor
|
||||
exactly like ``running``: tomo_queue_execute()'s pick-next loop
|
||||
always resumes whichever of the two it finds first, regardless of
|
||||
list order, so a pending job moved "ahead" of one in the list would
|
||||
visually suggest a run order that isn't real. A move is also
|
||||
refused if it would place the job at or above the position of any
|
||||
currently ``running``/``incomplete`` job. Takes effect at the next
|
||||
job boundary if the queue is executing (see tomo_queue_execute()'s
|
||||
pick-next model).
|
||||
|
||||
Args:
|
||||
index: Current index of the job to move.
|
||||
new_index: Index to move it to (list-insert semantics: the rest
|
||||
of the queue shifts to make room).
|
||||
"""
|
||||
jobs = self._tomo_queue_proxy.as_list()
|
||||
if not 0 <= index < len(jobs):
|
||||
raise ValueError(f"tomo_queue_move: index {index} out of range.")
|
||||
if not 0 <= new_index < len(jobs):
|
||||
raise ValueError(f"tomo_queue_move: new_index {new_index} out of range.")
|
||||
job = jobs[index]
|
||||
if job.get("status", "pending") != "pending":
|
||||
raise ValueError(
|
||||
f"tomo_queue_move: job #{index} is '{job.get('status')}' -- only "
|
||||
"pending jobs can be reordered."
|
||||
)
|
||||
floor_indices = [
|
||||
i for i, j in enumerate(jobs) if j.get("status") in ("running", "incomplete")
|
||||
]
|
||||
if floor_indices and new_index <= max(floor_indices):
|
||||
raise ValueError(
|
||||
f"tomo_queue_move: can't move to position {new_index} -- job "
|
||||
f"#{max(floor_indices)} is running/incomplete and is a hard floor "
|
||||
"for reordering."
|
||||
)
|
||||
self._tomo_queue_proxy.move(index, new_index)
|
||||
print(f"Moved tomo queue job #{index} -> #{new_index} ({job['label']}).")
|
||||
|
||||
def tomo_queue_clear(self) -> None:
|
||||
"""Empty the tomo queue."""
|
||||
self._tomo_queue_proxy.clear()
|
||||
@@ -3596,6 +4109,14 @@ class Flomni(
|
||||
print("Tomo queue is empty.")
|
||||
return jobs
|
||||
for i, job in enumerate(jobs):
|
||||
if job.get("kind", "tomo") == "command":
|
||||
steps_str = " > ".join(
|
||||
f"{step['action']}{step['kwargs']}" if step["kwargs"] else step["action"]
|
||||
for step in job["steps"]
|
||||
)
|
||||
idem = "idem" if job.get("idempotent") else "NOT idem"
|
||||
print(f"[{i}] {job['status']:>10s} {job['label']:<20s} CMD {steps_str} [{idem}]")
|
||||
continue
|
||||
p = job["params"]
|
||||
print(
|
||||
f"[{i}] {job['status']:>10s} {job['label']:<20s} "
|
||||
@@ -3613,9 +4134,29 @@ class Flomni(
|
||||
``tomo_scan()`` -- or, for a job that didn't run to completion last
|
||||
time, ``tomo_scan_resume()``, so it picks back up mid-scan instead
|
||||
of restarting from subtomo 1 / angle 0. Jobs already marked "done"
|
||||
are skipped on the next call, so simply calling tomo_queue_execute()
|
||||
again resumes from where it stopped (e.g. after fixing a hardware
|
||||
issue).
|
||||
are skipped, so simply calling tomo_queue_execute() again resumes
|
||||
from where it stopped (e.g. after fixing a hardware issue).
|
||||
|
||||
Pick-next execution model
|
||||
-------------------------
|
||||
The queue is re-read from its global var at the start of every job
|
||||
(not snapshotted once up front), and each job is addressed by its
|
||||
stable ``id``, not by its list index. This is what makes it safe to
|
||||
edit the queue *while it runs*: appending a job, deleting a pending
|
||||
job, or reordering the pending tail all take effect at the next job
|
||||
boundary, and a reorder/delete can never misdirect a status write to
|
||||
the wrong entry (an index-based loop could). Legacy jobs written
|
||||
before the ``id`` field existed are healed once up front by
|
||||
ensure_ids().
|
||||
|
||||
Resume-before-fresh ordering: if any job is in a resumable state, it
|
||||
is always run before any fresh "pending" job, regardless of list
|
||||
order. This preserves the single-resumable-job invariant -- tomo
|
||||
progress lives in one shared ``progress`` global var, so starting a
|
||||
fresh tomo_scan() before resuming an interrupted one would overwrite
|
||||
the very state tomo_scan_resume() needs. (Under the old strictly
|
||||
in-order loop this couldn't happen; allowing reorder makes it
|
||||
possible, hence the explicit guard.)
|
||||
|
||||
A job is considered not-yet-complete (and so gets resumed rather
|
||||
than restarted) if its status is "incomplete" (a Python exception
|
||||
@@ -3632,54 +4173,136 @@ class Flomni(
|
||||
yourself to recover from a crash (bypassing the queue), that scan
|
||||
is now actually finished even though the queue still has the job
|
||||
marked "incomplete" or "running" -- mark it done yourself before
|
||||
calling this again, or it will be re-run from scratch:
|
||||
flomni._tomo_queue_proxy.update(job_index, status="done")
|
||||
calling this again, or it will be re-run from scratch. Address the
|
||||
job by its id (indices can shift if the queue is reordered):
|
||||
flomni._tomo_queue_proxy.update_by_id(job_id, status="done")
|
||||
|
||||
Command jobs (job["kind"] == "command", added by
|
||||
tomo_queue_add_command()) are dispatched to _run_command_job()
|
||||
instead of tomo_scan()/tomo_scan_resume() -- see its docstring for
|
||||
their own resume policy. A job with no "kind" key is treated as
|
||||
"tomo" (back-compat for queues persisted before command jobs
|
||||
existed).
|
||||
|
||||
Args:
|
||||
start_index: Queue index to start from. Defaults to 0, but jobs
|
||||
already marked "done" are skipped automatically either way.
|
||||
start_index: Position in the *current* queue order below which
|
||||
fresh "pending" jobs are ignored. Defaults to 0. Resumable
|
||||
jobs are always run regardless of start_index (a crashed
|
||||
job must be recovered wherever it sits). Jobs already marked
|
||||
"done" are skipped automatically either way. Note that with
|
||||
reordering active this is applied to the queue's current
|
||||
order at each pick.
|
||||
"""
|
||||
self._tomo_queue_proxy.ensure_ids()
|
||||
jobs = self._tomo_queue_proxy.as_list()
|
||||
if not jobs:
|
||||
print("Tomo queue is empty.")
|
||||
return
|
||||
|
||||
runnable = [
|
||||
job
|
||||
for idx, job in enumerate(jobs)
|
||||
if job["status"] != "done"
|
||||
and (job["status"] in ("incomplete", "running") or idx >= start_index)
|
||||
]
|
||||
if not runnable:
|
||||
print("No pending tomo queue jobs to run.")
|
||||
return
|
||||
|
||||
if not self.OMNYTools.yesno(
|
||||
f"Starting automatic execution of {len(jobs) - start_index} queued tomo scan(s) on"
|
||||
f"Starting automatic execution of {len(runnable)} queued tomo scan(s) on"
|
||||
f" sample '{self.sample_name}'. OK?",
|
||||
"y",
|
||||
):
|
||||
print("Aborted.")
|
||||
return
|
||||
|
||||
for i in range(start_index, len(jobs)):
|
||||
job = jobs[i]
|
||||
if job["status"] == "done":
|
||||
continue
|
||||
while True:
|
||||
jobs = self._tomo_queue_proxy.as_list()
|
||||
# Resume any in-flight/interrupted job first, before starting any
|
||||
# fresh one, so the shared progress var is never clobbered. Search
|
||||
# the whole list (not the start_index slice): a crashed job must be
|
||||
# recovered wherever it sits.
|
||||
job = next((j for j in jobs if j["status"] in ("incomplete", "running")), None)
|
||||
if job is None:
|
||||
job = next(
|
||||
(
|
||||
j
|
||||
for idx, j in enumerate(jobs)
|
||||
if idx >= start_index and j["status"] == "pending"
|
||||
),
|
||||
None,
|
||||
)
|
||||
if job is None:
|
||||
break
|
||||
|
||||
job_id = job["id"]
|
||||
label = job["label"]
|
||||
kind = job.get("kind", "tomo")
|
||||
resume_job = job["status"] in ("incomplete", "running")
|
||||
|
||||
print(f"\n=== Tomo queue job {i + 1}/{len(jobs)}: {job['label']} ===")
|
||||
for name, value in job["params"].items():
|
||||
setattr(self, name, value)
|
||||
print(f"\n=== Tomo queue job: {label} ===")
|
||||
|
||||
self._tomo_queue_proxy.update(i, status="running")
|
||||
self._tomo_queue_proxy.update_by_id(job_id, status="running")
|
||||
try:
|
||||
if resume_job:
|
||||
self.tomo_scan_resume()
|
||||
if kind == "command":
|
||||
self._run_command_job(job, resume_job)
|
||||
else:
|
||||
self.tomo_scan()
|
||||
for name, value in job["params"].items():
|
||||
setattr(self, name, value)
|
||||
if resume_job:
|
||||
self.tomo_scan_resume()
|
||||
else:
|
||||
self.tomo_scan()
|
||||
except Exception as exc:
|
||||
self._tomo_queue_proxy.update(i, status="incomplete")
|
||||
print(f"Tomo queue job {i} ({job['label']}) did not complete: {exc}")
|
||||
self._tomo_queue_proxy.update_by_id(job_id, status="incomplete")
|
||||
print(f"Tomo queue job '{label}' did not complete: {exc}")
|
||||
print(
|
||||
"Queue paused. Fix the issue and call tomo_queue_execute() "
|
||||
"again to resume from this job."
|
||||
)
|
||||
raise
|
||||
self._tomo_queue_proxy.update(i, status="done")
|
||||
self._tomo_queue_proxy.update_by_id(job_id, status="done")
|
||||
|
||||
print("\nTomo queue finished -- all jobs done.")
|
||||
|
||||
def _run_command_job(self, job: dict, resume_job: bool) -> None:
|
||||
"""Run (or resume) a "command" job's steps in order.
|
||||
|
||||
No per-step resume cursor -- deliberately: tracking which step of a
|
||||
multi-step job completed means more persisted state and more edge
|
||||
cases than this warrants. Instead the job-level "idempotent" flag
|
||||
decides:
|
||||
|
||||
- idempotent: silently re-run every step from the top. Harmless --
|
||||
absolute moves and in/out actions re-run cleanly.
|
||||
- not idempotent: prompt the operator rather than silently re-run,
|
||||
which could be actively wrong for a relative/stateful step. "no"
|
||||
leaves the steps un-run; the caller (tomo_queue_execute) still
|
||||
marks the job "done" either way, matching the same edge case tomo
|
||||
jobs have -- a job that actually finished but crashed before its
|
||||
status hit "done".
|
||||
"""
|
||||
label = job["label"]
|
||||
if resume_job:
|
||||
if job.get("idempotent"):
|
||||
print(
|
||||
f"Command job '{label}' may have partially run before the crash "
|
||||
"-- idempotent, re-running from the top."
|
||||
)
|
||||
else:
|
||||
if not self.OMNYTools.yesno(
|
||||
f"Command job '{label}' may have partially run before the crash "
|
||||
"(not every step is idempotent). Re-run it from the top?",
|
||||
"n",
|
||||
):
|
||||
print(f"Skipping '{label}' -- it will be marked done without re-running.")
|
||||
return
|
||||
for step in job["steps"]:
|
||||
spec = self._TOMO_QUEUE_ACTIONS[step["action"]]
|
||||
method = getattr(self, spec["func"])
|
||||
method(**step["kwargs"])
|
||||
|
||||
def rt_off(self):
|
||||
dev.rtx.enabled = False
|
||||
dev.rty.enabled = False
|
||||
@@ -3732,13 +4355,21 @@ class Flomni(
|
||||
f"{'Number of individual sub-tomograms:':<{padding}}{8:>{padding}}\n",
|
||||
f"{'Angular step within sub-tomogram:':<{padding}}{self.tomo_angle_stepsize:>{padding}.2f}\n",
|
||||
]
|
||||
hook_description = self._describe_active_hook()
|
||||
if hook_description:
|
||||
content.append(f"{'At-each-angle hook:':<{padding}}{hook_description:>{padding}}\n")
|
||||
content = "".join(content)
|
||||
hook_source = self._active_hook_source()
|
||||
user_target = os.path.expanduser(
|
||||
f"~/data/raw/documentation/tomo_scan_ID_{self.tomo_id}.pdf"
|
||||
)
|
||||
with PDFWriter(user_target) as file:
|
||||
file.write(header)
|
||||
file.write(content)
|
||||
if hook_source:
|
||||
file.write(
|
||||
f"\nAt-each-angle hook source ('{self.at_each_angle_hook}'):\n{hook_source}"
|
||||
)
|
||||
# subprocess.run(
|
||||
# "xterm /work/sls/spec/local/XOMNY/bin/upload/upload_last_pon.sh &", shell=True
|
||||
# )
|
||||
@@ -3766,8 +4397,13 @@ class Flomni(
|
||||
logger.warning("SciLog is not enabled; skipping PDF report entry.")
|
||||
return
|
||||
|
||||
scilog_text = content
|
||||
if hook_source:
|
||||
scilog_text += (
|
||||
f"\n\nAt-each-angle hook source ('{self.at_each_angle_hook}'):\n{hook_source}"
|
||||
)
|
||||
scilog.new().add_attachment(logo_file, width=200).add_text(
|
||||
content.replace("\n", "<br>")
|
||||
scilog_text.replace("\n", "<br>")
|
||||
).add_tags("tomoscan").send()
|
||||
|
||||
|
||||
|
||||
@@ -30,8 +30,17 @@ class FlomniOpticsMixin:
|
||||
|
||||
# move rotation stage to zero to avoid problems with wires
|
||||
umv(dev.fsamroy, 0)
|
||||
fttrx_in = self._get_user_param_safe("feyex", "fttrx_in")
|
||||
umv(dev.fttrx1, fttrx_in)
|
||||
|
||||
if "fttrx1" not in dev:
|
||||
if not self.OMNYTools.yesno(
|
||||
"Device 'fttrx1' does not exist in the current config, so it "
|
||||
"can't be moved out of the way. Continue anyway?"
|
||||
):
|
||||
print("Aborting feye_out(): fttrx1 not moved.")
|
||||
return
|
||||
else:
|
||||
fttrx_in = self._get_user_param_safe("feyex", "fttrx_in")
|
||||
umv(dev.fttrx1, fttrx_in)
|
||||
|
||||
def feye_in(self):
|
||||
bec.queue.next_dataset_number += 1
|
||||
|
||||
@@ -60,6 +60,16 @@ At _launch(), the generator queries session_query.php to find which account was
|
||||
previously active. If the account has changed the session.htpasswd is cleared
|
||||
immediately (old user loses access) and uploaded. The new user calls
|
||||
set_web_password() to set their own password and gain access.
|
||||
|
||||
Account-mismatch fallback (dev/sim sessions):
|
||||
----------------------------------------------
|
||||
If bec_client.active_account doesn't match the OS login (_check_account_match)
|
||||
-- typically a leftover session under the wrong experiment account, or a dev/sim
|
||||
session run under a personal account -- start() no longer does nothing. It
|
||||
starts a *local-only* status page on a separate port (default 8081, see
|
||||
mismatch_local_port) for local debugging: no singleton lock (so it can never
|
||||
block the real session's takeover), no remote upload, no session.htpasswd
|
||||
changes. Look for "[local-only: account mismatch]" in the startup log / status().
|
||||
"""
|
||||
|
||||
import datetime
|
||||
@@ -92,6 +102,12 @@ _LOCK_STALE_AFTER_S = 35 # just over 2 missed cycles (2×15s+5s margin)
|
||||
_CYCLE_INTERVAL_S = 15
|
||||
_TOMO_HEARTBEAT_STALE_S = 90
|
||||
|
||||
# Fallback port used when the BEC account doesn't match the system user (see
|
||||
# _check_account_match / start()). Deliberately different from the default
|
||||
# local_port so a local-only fallback session never collides with a
|
||||
# legitimately-started one on the same host.
|
||||
_MISMATCH_LOCAL_PORT = 8081
|
||||
|
||||
# Path where online ptycho reconstructions are written by the analysis pipeline.
|
||||
# Subfolders follow the pattern dset_NNNNN and contain S#####_*_phase.png etc.
|
||||
_PTYCHO_ONLINE_DIR = "~/data/raw/analysis/online/ptycho"
|
||||
@@ -619,6 +635,7 @@ class WebpageGeneratorBase:
|
||||
verbosity: int = VERBOSITY_NORMAL,
|
||||
upload_url: str = None,
|
||||
local_port: int = 8080,
|
||||
mismatch_local_port: int = _MISMATCH_LOCAL_PORT,
|
||||
):
|
||||
self._bec = bec_client
|
||||
self._output_dir = Path(output_dir).expanduser().resolve()
|
||||
@@ -626,7 +643,9 @@ class WebpageGeneratorBase:
|
||||
self._verbosity = verbosity
|
||||
self._uploader = HttpUploader(upload_url) if upload_url else None
|
||||
self._local_port = local_port
|
||||
self._mismatch_local_port = mismatch_local_port
|
||||
self._local_server = None # created fresh each _launch()
|
||||
self._local_only = False # True while running the account-mismatch fallback
|
||||
|
||||
# Derive companion URLs from upload_url
|
||||
if upload_url is not None:
|
||||
@@ -658,9 +677,16 @@ class WebpageGeneratorBase:
|
||||
Progress messages during the wait are printed at verbosity >= 2 only.
|
||||
"""
|
||||
if not _check_account_match(self._bec):
|
||||
if self._thread is not None and self._thread.is_alive():
|
||||
self._log(VERBOSITY_NORMAL, "WebpageGenerator already running in this session.")
|
||||
return
|
||||
self._log(VERBOSITY_NORMAL,
|
||||
"WebpageGenerator: BEC account does not match system user. "
|
||||
"Not starting.", level="warning")
|
||||
f"Starting a local-only status page on port "
|
||||
f"{self._mismatch_local_port} instead (no singleton lock, no "
|
||||
"remote upload, no session-auth changes) — likely a dev/sim "
|
||||
"session running under a different account.", level="warning")
|
||||
self._launch(local_only=True)
|
||||
return
|
||||
|
||||
if self._thread is not None and self._thread.is_alive():
|
||||
@@ -674,9 +700,19 @@ class WebpageGeneratorBase:
|
||||
|
||||
self._launch()
|
||||
|
||||
def _launch(self) -> None:
|
||||
def _launch(self, local_only: bool = False) -> None:
|
||||
"""Actually start the worker thread and write static files. Called by
|
||||
start() on the normal path, or by the lock-watcher thread after takeover."""
|
||||
start() on the normal path, or by the lock-watcher thread after takeover.
|
||||
|
||||
local_only=True is the account-mismatch fallback (see start()): serves
|
||||
the status page on a separate port for local debugging only, and skips
|
||||
everything that assumes the active BEC account is the "real" one for
|
||||
this host — the singleton lock (a mismatched account shouldn't contend
|
||||
for it against the legitimate session) and remote upload / session-auth
|
||||
(writing session.htpasswd or uploading under the wrong account could
|
||||
step on the real experiment's web page).
|
||||
"""
|
||||
self._local_only = local_only
|
||||
self._output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Copy logo once at startup — HTML is also written once here,
|
||||
@@ -686,15 +722,18 @@ class WebpageGeneratorBase:
|
||||
_render_html(_PHONE_NUMBERS, self.HAS_TOMO_QUEUE, self.TOMO_TYPES)
|
||||
)
|
||||
|
||||
# Check whether the active BEC account matches the session user on the
|
||||
# remote server. If not, clear session.htpasswd so the old user loses
|
||||
# web access immediately. The new user then calls set_web_password().
|
||||
self._check_session_user()
|
||||
if not local_only:
|
||||
# Check whether the active BEC account matches the session user on
|
||||
# the remote server. If not, clear session.htpasswd so the old
|
||||
# user loses web access immediately. The new user then calls
|
||||
# set_web_password().
|
||||
self._check_session_user()
|
||||
|
||||
# Start local HTTP server (always on; a fresh instance per _launch).
|
||||
# Start local HTTP server (always on; a fresh instance per _launch()).
|
||||
port = self._mismatch_local_port if local_only else self._local_port
|
||||
if self._local_server is not None and self._local_server.is_alive():
|
||||
self._local_server.stop()
|
||||
self._local_server = LocalHttpServer(self._output_dir, self._local_port)
|
||||
self._local_server = LocalHttpServer(self._output_dir, port)
|
||||
try:
|
||||
self._local_server.start()
|
||||
local_url_msg = f" local={self._local_server.url}"
|
||||
@@ -703,7 +742,7 @@ class WebpageGeneratorBase:
|
||||
self._log(VERBOSITY_NORMAL, str(exc), level="warning")
|
||||
|
||||
# Upload static files (html + logo) once at startup
|
||||
if self._uploader is not None:
|
||||
if not local_only and self._uploader is not None:
|
||||
self._uploader.upload_dir_async(self._output_dir)
|
||||
|
||||
self._stop_event.clear()
|
||||
@@ -711,10 +750,12 @@ class WebpageGeneratorBase:
|
||||
target=self._run, name="WebpageGenerator", daemon=True
|
||||
)
|
||||
self._thread.start()
|
||||
mode_note = " [local-only: account mismatch]" if local_only else ""
|
||||
self._log(VERBOSITY_NORMAL,
|
||||
f"WebpageGenerator started owner={self._owner_id} "
|
||||
f"WebpageGenerator started{mode_note} owner={self._owner_id} "
|
||||
f"output={self._output_dir} interval={self._cycle_interval}s"
|
||||
+ (f" upload={self._uploader._url}" if self._uploader else " upload=disabled")
|
||||
+ (f" upload={self._uploader._url}"
|
||||
if (self._uploader and not local_only) else " upload=disabled")
|
||||
+ f"\n ➜ Status page:{local_url_msg}")
|
||||
|
||||
def stop(self) -> None:
|
||||
@@ -729,6 +770,7 @@ class WebpageGeneratorBase:
|
||||
self._local_server.stop()
|
||||
self._local_server = None
|
||||
self._release_lock()
|
||||
self._local_only = False
|
||||
self._log(VERBOSITY_NORMAL, "WebpageGenerator stopped.")
|
||||
|
||||
@property
|
||||
@@ -752,7 +794,8 @@ class WebpageGeneratorBase:
|
||||
active_account = "unknown"
|
||||
print(
|
||||
f"WebpageGenerator\n"
|
||||
f" This session running : {running}\n"
|
||||
f" This session running : {running}"
|
||||
+ (" [local-only: account mismatch]" if self._local_only else "") + "\n"
|
||||
f" Lock owner : {lock.get('owner_id', 'none')}\n"
|
||||
f" Lock heartbeat : {lock.get('heartbeat', 'never')}\n"
|
||||
f" Output dir : {self._output_dir}\n"
|
||||
@@ -980,10 +1023,14 @@ class WebpageGeneratorBase:
|
||||
except Exception as exc:
|
||||
self._log(VERBOSITY_NORMAL,
|
||||
f"WebpageGenerator cycle error: {exc}", level="warning")
|
||||
try:
|
||||
self._write_lock()
|
||||
except Exception:
|
||||
pass
|
||||
if not self._local_only:
|
||||
# The singleton lock coordinates real sessions for this
|
||||
# account; a mismatched-account fallback session must never
|
||||
# claim it (see _launch()'s local_only docstring).
|
||||
try:
|
||||
self._write_lock()
|
||||
except Exception:
|
||||
pass
|
||||
self._stop_event.wait(max(0.0, self._cycle_interval - (_epoch() - t0)))
|
||||
|
||||
def _cycle(self) -> None:
|
||||
@@ -1166,7 +1213,9 @@ class WebpageGeneratorBase:
|
||||
# If the server is unreachable the cycle continues normally; failures
|
||||
# are logged as warnings. If an upload is still in progress from the
|
||||
# previous cycle the new request is dropped (logged at DEBUG level).
|
||||
if self._uploader is not None:
|
||||
# Never upload from the account-mismatch fallback (local_only) — the
|
||||
# active account isn't the one the remote page's auth is set up for.
|
||||
if self._uploader is not None and not self._local_only:
|
||||
new_scan = ptycho.get("scan_id")
|
||||
if new_scan and new_scan != prev_ptycho_scan:
|
||||
# Scan changed: ask server to delete old S*_*.png/jpg first,
|
||||
@@ -1436,6 +1485,7 @@ class FlomniWebpageGenerator(WebpageGeneratorBase):
|
||||
"frames_per_trigger",
|
||||
"single_point_instead_of_fermat_scan",
|
||||
"ptycho_reconstruct_foldername",
|
||||
"at_each_angle_hook",
|
||||
]
|
||||
|
||||
# label -> dotpath under device_manager.devices
|
||||
@@ -2816,6 +2866,12 @@ function buildParamRows(params){{
|
||||
if(nproj!=null) rows+='<tr><td>Projections</td><td>'+nproj+'</td></tr>';
|
||||
}}
|
||||
}});
|
||||
// Only shown when actually set -- not in TQ_PARAM_DISPLAY since every
|
||||
// normal job would otherwise show a "-" row nobody needs to see; this one
|
||||
// is worth surfacing precisely because it's not the default.
|
||||
if(params.at_each_angle_hook){{
|
||||
rows+='<tr><td>At-each-angle hook</td><td>'+esc(fmtTqParam('at_each_angle_hook',params.at_each_angle_hook))+'</td></tr>';
|
||||
}}
|
||||
return rows;
|
||||
}}
|
||||
|
||||
@@ -2845,6 +2901,7 @@ function renderTomoQueue(jobs){{
|
||||
jobs.forEach((job,i)=>{{
|
||||
const status=job.status||'pending';
|
||||
const label=job.label||('Job '+(i+1));
|
||||
const isCommand=job.kind==='command';
|
||||
const params=job.params||{{}};
|
||||
const key=(job.added_at||'')+'|'+label;
|
||||
presentKeys.add(key);
|
||||
@@ -2855,7 +2912,13 @@ function renderTomoQueue(jobs){{
|
||||
shouldOpen=true;
|
||||
autoOpened.add(key);
|
||||
}}
|
||||
const paramRows=buildParamRows(params);
|
||||
// Command jobs have no "params" key -- render their step sequence
|
||||
// instead of an (otherwise blank) params table.
|
||||
const paramRows=isCommand?'':buildParamRows(params);
|
||||
const stepsSummary=isCommand
|
||||
?(job.steps||[]).map(s=>s.action+(s.kwargs&&Object.keys(s.kwargs).length?JSON.stringify(s.kwargs):'')).join(' → ')
|
||||
:'';
|
||||
const idemBadge=isCommand?(job.idempotent?' [idem]':' [NOT idem]'):'';
|
||||
const addedAt=job.added_at
|
||||
?'Added '+new Date(job.added_at).toLocaleString([],{{month:'short',day:'numeric',hour:'2-digit',minute:'2-digit'}})
|
||||
:'';
|
||||
@@ -2866,7 +2929,9 @@ function renderTomoQueue(jobs){{
|
||||
+'<span class="tq-badge tq-'+esc(status)+'">'+esc(status)+'</span>'
|
||||
+'</summary>'
|
||||
+'<div class="tq-details">'
|
||||
+(paramRows?'<table class="tq-params">'+paramRows+'</table>':'')
|
||||
+(isCommand
|
||||
?'<div class="tq-steps">CMD '+esc(stepsSummary)+esc(idemBadge)+'</div>'
|
||||
:(paramRows?'<table class="tq-params">'+paramRows+'</table>':''))
|
||||
+(addedAt?'<div class="tq-added">'+esc(addedAt)+'</div>':'')
|
||||
+'</div>'
|
||||
+'</details>';
|
||||
|
||||
@@ -325,6 +325,12 @@ class flomniGuiTools:
|
||||
f" Est. remaining: {eta_display}\n"
|
||||
f" Est. finish: {finish_display}"
|
||||
)
|
||||
# self._describe_active_hook() comes from Flomni itself (this is
|
||||
# a mixin, always combined with it) -- shown only when a hook is
|
||||
# actually active, same as tomo_parameters()'s CLI note.
|
||||
hook_description = self._describe_active_hook()
|
||||
if hook_description:
|
||||
text += f"\n Hook: {hook_description}"
|
||||
self.progressbar.set_center_label(text)
|
||||
except Exception as exc:
|
||||
logger.warning(f"flOMNI progress GUI update skipped: {exc}")
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -180,7 +180,7 @@ fsamroy:
|
||||
- 365
|
||||
port: 8084
|
||||
sign: -1
|
||||
sim_velocity: 25
|
||||
sim_velocity: 250 # 10x hardware
|
||||
sim_initial_position: 0
|
||||
enabled: true
|
||||
onFailure: buffer
|
||||
|
||||
@@ -86,7 +86,9 @@ Several parameter sets can be queued and run one after another automatically, e.
|
||||
1. `flomni.tomo_queue_show()` review the queue before starting.
|
||||
1. `flomni.tomo_queue_execute()` run all queued scans in sequence.
|
||||
|
||||
If a queued scan is interrupted, the next call to `flomni.tomo_queue_execute()` resumes it automatically rather than starting over, then continues with the remaining entries. See [Tomography](user.ptychography.flomni.tomography) below for the full command reference.
|
||||
If a queued scan is interrupted, the next call to `flomni.tomo_queue_execute()` resumes it automatically rather than starting over, then continues with the remaining entries.
|
||||
|
||||
The same queue can also hold **command jobs** that reconfigure the beamline between scans (e.g. change energy, then re-peak the undulator gap) instead of running a tomogram, so "scan A, reconfigure, scan B" runs unattended as one queue. A queued tomogram can also run custom code at each projection angle (e.g. moving a polarizer in and out between two exposures) via a registered **at_each_angle hook**. See [Tomography](user.ptychography.flomni.tomography) below for the full command reference, including command jobs and hooks.
|
||||
|
||||
#### If something went wrong…
|
||||
|
||||
@@ -346,6 +348,8 @@ Several tomo parameter sets can be queued and run sequentially on the same sampl
|
||||
|
||||
The queue is persisted (it survives a BEC client restart). Each job's status is one of `pending`, `running`, `incomplete`, or `done`. A job that did not run to completion (an exception was caught, or the BEC client itself crashed mid-scan) is automatically resumed - rather than restarted - the next time `flomni.tomo_queue_execute()` is called.
|
||||
|
||||
If the process running the queue dies hard enough that it never gets to write a status update at all (a kernel restart, a Ctrl-C during a blocking prompt) a job can be left stuck showing `running` with nothing actually running. `flomni.tomo_queue_delete(index)` has no status guard and works on a stuck row regardless; you can also fix the status directly with `flomni._tomo_queue_proxy.update_by_id(job_id, status="incomplete")` (to make it resumable) or `status="done"` (to skip it). The GUI's Delete/Clear normally refuse to touch a `running` row, but detect a stale one (no recent scan heartbeat) and offer to proceed anyway with an explicit confirmation instead of blocking forever.
|
||||
|
||||
Example:
|
||||
```
|
||||
flomni.tomo_parameters() # set up parameter set #1
|
||||
@@ -356,6 +360,178 @@ flomni.tomo_queue_show()
|
||||
flomni.tomo_queue_execute() # runs both, in order, on this sample
|
||||
```
|
||||
|
||||
**GUI: reusing an earlier job's settings.** In the ☰ Queue control… dialog, select a
|
||||
single tomo job (not a command job — those have no scan parameters) and click
|
||||
**"Load into editor"** to pull its saved settings into the params panel's editor,
|
||||
as if you'd just clicked Edit. Nothing is written yet — review or tweak the fields,
|
||||
then Submit (writes live, blocked while the beamline is busy) or "Add to queue" (always
|
||||
allowed) as usual. If you already had an edit in progress, it asks before discarding it.
|
||||
|
||||
**GUI: two different "Add to queue" buttons.** The params panel's own **"Add to
|
||||
queue"** (visible in edit mode) queues whatever you've typed, unsubmitted. The ☰
|
||||
Queue control… window's **"Add current params to queue"** is a different button on a
|
||||
different window — it always queues the *live* parameters, regardless of any edit
|
||||
open in the panel. If you have an unsaved edit open and click the queue window's
|
||||
button instead of the panel's, it warns and names the correct one before proceeding,
|
||||
since it would otherwise queue your last-submitted values, not what you just typed.
|
||||
|
||||
#### Command jobs — reconfiguring the beamline between scans
|
||||
|
||||
In addition to tomogram jobs, the same queue can hold **command jobs**: an ordered
|
||||
list of beamline reconfiguration steps (move a device, ...) that run instead of a
|
||||
scan. This is what lets one queue express *"tomogram A, then reconfigure, then
|
||||
tomogram B"* unattended, e.g. change energy and re-peak the undulator gap between
|
||||
two tomograms on the same sample.
|
||||
|
||||
`flomni.tomo_queue_add_command(steps, label=None, idempotent=None)`
|
||||
|
||||
- `steps`: a single `{"action": ..., "kwargs": {...}}` dict, or a list of them run in
|
||||
sequence within that one job.
|
||||
- `label`: optional name shown by `tomo_queue_show()`, same as for `tomo_queue_add()`.
|
||||
- `idempotent`: normally inferred (safe to blindly re-run after a crash only if every
|
||||
step is); override explicitly if needed.
|
||||
|
||||
Only actions from a fixed, reviewed registry can be queued — not arbitrary code:
|
||||
|
||||
| action | does | parameters |
|
||||
| --- | --- | --- |
|
||||
| `move` | Move device(s) to absolute position(s). Only devices on the allow-list below can be targeted. | `positions`: `{device: target}`, e.g. `{"mokev": 6.2}` |
|
||||
| `optimize_idgap` | Scan the undulator gap over a range and move to the peak. **Not yet implemented (no-op stub).** | `search_range`: mm, default 0.5, range 0–2 |
|
||||
|
||||
Devices allowed for `move`: `mokev` (energy, keV), `idgap` (undulator gap, mm). A
|
||||
`move` naming any other device is rejected, both when the job is added and again when
|
||||
it actually runs.
|
||||
|
||||
Example — change energy, then re-peak idgap, before the next tomogram:
|
||||
```
|
||||
flomni.tomo_queue_add_command(
|
||||
[{"action": "move", "kwargs": {"positions": {"mokev": 6.2}}},
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 0.5}}],
|
||||
label="reconfigure to 6.2 keV",
|
||||
)
|
||||
```
|
||||
|
||||
`tomo_queue_show()` lists command jobs alongside tomogram jobs, e.g.:
|
||||
```
|
||||
[2] pending reconfigure to 6.2 keV CMD move{'positions': {'mokev': 6.2}} > optimize_idgap{'search_range': 0.5} [idem]
|
||||
```
|
||||
|
||||
If a command job is interrupted by a crash, there is no per-step resume point — the
|
||||
whole job is either safe to redo from the top or it isn't:
|
||||
- If every step is idempotent (the usual case — an absolute move is harmless to
|
||||
repeat), `flomni.tomo_queue_execute()` silently re-runs the whole job from the top.
|
||||
- If any step is not idempotent, you are asked whether to re-run the job from the top
|
||||
or mark it done as-is.
|
||||
|
||||
#### Custom behavior at each projection angle
|
||||
|
||||
By default, every projection in a tomo scan is a single ptychography scan at that
|
||||
angle. For cases that need something more — e.g. record a projection, move a
|
||||
polarizer in, record again, move it back out, at every angle — write a small Python
|
||||
function and register it as an **at_each_angle hook**. Once registered, it can be
|
||||
selected per tomo-queue job, so the whole modulated tomogram runs unattended along
|
||||
with any other queued jobs.
|
||||
|
||||
A hook is a function `func(flomni, angle)`, called once per projection angle instead
|
||||
of the normal acquisition. Acquire the actual projection(s) inside it the same way
|
||||
`tomo_scan()` would on its own — `tomo_scan_projection(angle)` for a normal
|
||||
(Fermat-scan) measurement, or `tomo_acquire_at_angle(angle)` if `tomo_parameters()`
|
||||
has single-point mode enabled for this job (see the warning below for why it matters
|
||||
which one you call):
|
||||
```python
|
||||
def polarizer_modulation(flomni, angle):
|
||||
flomni.tomo_scan_projection(angle)
|
||||
umv(dev.polarizer, "in")
|
||||
flomni.tomo_scan_projection(angle)
|
||||
umv(dev.polarizer, "out")
|
||||
```
|
||||
|
||||
| command | explanation |
|
||||
| --- | --- |
|
||||
| `flomni.register_at_each_angle_hook("name", func)` | Load a hook function under `name`. |
|
||||
| `flomni.at_each_angle_hook = "name"` | Activate it — the next tomo-queue job added will use it. |
|
||||
| `flomni.at_each_angle_hook = None` | Deactivate it — back to normal projections for the next job. |
|
||||
| `flomni.list_at_each_angle_hooks()` | Print the names of all currently registered hooks (the active one is marked). |
|
||||
| `flomni.unregister_at_each_angle_hook("name")` | Remove a hook by name. |
|
||||
|
||||
Example — queue a polarizer-modulated tomogram, then a normal one:
|
||||
```python
|
||||
flomni.register_at_each_angle_hook("polarizer_mod", polarizer_modulation)
|
||||
|
||||
flomni.tomo_parameters() # set up the scan parameters as usual
|
||||
flomni.at_each_angle_hook = "polarizer_mod" # activate the hook
|
||||
flomni.tomo_queue_add("polarizer run")
|
||||
|
||||
flomni.at_each_angle_hook = None # back to normal -- do not skip this!
|
||||
flomni.tomo_queue_add("plain follow-up scan")
|
||||
|
||||
flomni.tomo_queue_execute() # runs both, hook active only for the first
|
||||
```
|
||||
**Do not forget to reset `flomni.at_each_angle_hook = None`** before adding a job
|
||||
that should run normally — each queued job remembers whatever `at_each_angle_hook`
|
||||
was set to at the moment it was added (like every other tomo parameter), so a job
|
||||
added right after a hook-using one without resetting it first will silently run with
|
||||
that hook still active.
|
||||
|
||||
Registered hooks are **session-only** — they live in memory and do not survive a
|
||||
kernel restart. If `tomo_queue_execute()` reaches a job whose `at_each_angle_hook`
|
||||
isn't registered in the current session (e.g. after restarting BEC), it stops with a
|
||||
clear error rather than silently running a plain scan; re-run
|
||||
`register_at_each_angle_hook()` for that hook, then call `tomo_queue_execute()`
|
||||
again to resume.
|
||||
|
||||
**Editing a hook after registering it:** `register_at_each_angle_hook()` stores
|
||||
whichever function object you pass it at that moment — it is not a live link to the
|
||||
function's name. If you edit the function's source and re-run the `def` in your
|
||||
session but do **not** call `register_at_each_angle_hook()` again, the *old* version
|
||||
keeps running. Always re-register after an edit.
|
||||
|
||||
**`tomo_scan_projection()` vs `tomo_acquire_at_angle()`:** these are not
|
||||
interchangeable. `tomo_scan_projection(angle)` always runs a full Fermat-scan
|
||||
projection; `tomo_acquire_at_angle(angle)` always runs a single-point acquisition.
|
||||
Use whichever matches how `tomo_parameters()` has this job configured
|
||||
(`single_point_instead_of_fermat_scan`) — calling the wrong one for the job's mode
|
||||
produces the wrong kind of data, and calling `tomo_scan_projection()` directly while
|
||||
single-point mode is on prints a warning and asks for confirmation. **Declining that
|
||||
confirmation now fails the job** (raises, rather than silently skipping that
|
||||
projection) — inside a custom hook or the normal tomo flow this correctly marks the
|
||||
queue job `incomplete` and pauses the queue, instead of quietly acquiring fewer
|
||||
projections than expected with no trace, or (if you get asked at every angle and
|
||||
eventually give up and hit Ctrl-C) leaving a job stuck at status `running` forever
|
||||
with nothing actually running. If you hit this, use `tomo_acquire_at_angle(angle)`
|
||||
in your hook instead, or fix the queue entry via `flomni.tomo_queue_delete(index)`
|
||||
(no status guard, works even on a stuck `running` row) or by marking it done/
|
||||
incomplete directly: `flomni._tomo_queue_proxy.update_by_id(job_id, status=...)`.
|
||||
A reminder of which one to use fires the moment you switch single-point mode on
|
||||
(`flomni.single_point_instead_of_fermat_scan = True`), and again any time you check
|
||||
`flomni.tomo_parameters()` while it's on; the params panel (GUI) shows the same
|
||||
reminder as a visible note whenever "Single-point scan" is checked.
|
||||
|
||||
**GUI:** the tomo parameters panel has an "At-each-angle hook" dropdown, listing
|
||||
whatever is currently registered from the CLI (`register_at_each_angle_hook()`) —
|
||||
the GUI process can't register hooks itself, only select one by name for the job
|
||||
you're about to submit or add to the queue. Pick "None (default)" for a normal scan.
|
||||
If a job's stored hook isn't currently registered anywhere, it still shows up in the
|
||||
dropdown labeled "(not registered in this session)" so it's never silently hidden.
|
||||
A queued job's active hook (if any) also appears on the status webpage (queue detail
|
||||
and "Current measurement"), the ☰ Queue control… table's Details column, and the
|
||||
`flomnigui_show_progress()` ring's center label.
|
||||
|
||||
**Unregistering does not clear `at_each_angle_hook`.** `unregister_at_each_angle_hook()`
|
||||
only removes the function from this session's registry — if `at_each_angle_hook` (or a
|
||||
queued job) still names it, that reference is untouched; it just becomes an
|
||||
unregistered name, flagged as such wherever it's shown (CLI, GUI dropdown, webpage,
|
||||
Details column). Set `flomni.at_each_angle_hook = None` (or the GUI dropdown's "None
|
||||
(default)") explicitly if you want to actually clear the selection, not just remove
|
||||
the hook it points to.
|
||||
|
||||
**Where the hook shows up in the permanent record:** `flomni.tomo_parameters()`,
|
||||
the end-of-scan scilog entry, and `write_pdf_report()` all show the active hook's
|
||||
name (flagged if unregistered) — and, when it's still registered, its full **source
|
||||
code** too (scilog entry and PDF report only, not the console print), so what a
|
||||
tomogram actually did is part of the log even if the hook itself is edited or
|
||||
removed later.
|
||||
|
||||
### Sample storage and transfer
|
||||
|
||||
[See short version](user.ptychography.flomni.transfer)
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Shared bootstrap for constructing a live Flomni against the running sim.
|
||||
|
||||
Used both by conftest.py (in-process, section A) and by
|
||||
_run_queue_subprocess.py (spawned as a separate OS process, section B/C) --
|
||||
pulled out here so both paths for "build a real Flomni pointed at the sim"
|
||||
share one definition of what real-instrument setup and side-effect
|
||||
neutralization is needed. See TOMO_QUEUE_TESTING.md section 3.
|
||||
|
||||
Deliberately does NOT call bec.config.load_demo_config()/
|
||||
update_session_with_file(): assumes the running sim session already has
|
||||
device_configs/simulated_omny/simulated_flomni.yaml loaded.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import builtins
|
||||
import importlib
|
||||
|
||||
from bec_ipython_client import BECIPythonClient
|
||||
from bec_lib.messaging_services import SciLogMessagingService
|
||||
from bec_lib.redis_connector import RedisConnector
|
||||
from bec_lib.service_config import ServiceConfig
|
||||
|
||||
from csaxs_bec.bec_ipython_client.plugins.flomni import flomni as flomni_module
|
||||
from csaxs_bec.bec_ipython_client.plugins.flomni.flomni_webpage_generator import (
|
||||
FlomniWebpageGenerator,
|
||||
)
|
||||
from csaxs_bec.bec_ipython_client.plugins.omny.omny_general_tools import OMNYTools
|
||||
|
||||
|
||||
class _NoOpSciLogMessage:
|
||||
"""Chainable no-op standing in for a real SciLog message object."""
|
||||
|
||||
def add_text(self, *a, **k):
|
||||
return self
|
||||
|
||||
def add_tags(self, *a, **k):
|
||||
return self
|
||||
|
||||
def add_attachment(self, *a, **k):
|
||||
return self
|
||||
|
||||
def send(self, *a, **k):
|
||||
return self
|
||||
|
||||
|
||||
def neutralize_side_effects() -> None:
|
||||
"""Patch away operator-facing/networked side effects of Flomni().
|
||||
|
||||
Plain class-attribute assignment, not pytest's monkeypatch: every
|
||||
caller (in-process fixture, subprocess script) wants the same patches
|
||||
for the life of the process, so there's nothing to revert.
|
||||
|
||||
- OMNYTools.yesno -> always answers yes, so tomo_queue_execute()'s
|
||||
confirmation prompt doesn't hang a headless run.
|
||||
- FlomniWebpageGenerator.start -> no-op. Flomni.__init__ normally starts
|
||||
this unconditionally, which opens a local HTTP server and spawns
|
||||
background threads that POST to a real external URL.
|
||||
- SciLogMessagingService.new -> no-op. tomo_scan() unconditionally sends
|
||||
a SciLog summary at the end of every scan; SciLog isn't
|
||||
configured/reachable in this dev environment.
|
||||
"""
|
||||
OMNYTools.yesno = lambda self, *a, **k: True
|
||||
FlomniWebpageGenerator.start = lambda self: None
|
||||
SciLogMessagingService.new = lambda self, *a, **k: _NoOpSciLogMessage()
|
||||
|
||||
|
||||
def build_flomni(services_config_path):
|
||||
"""Connect to the already-running sim and return (bec, flomni)."""
|
||||
config = ServiceConfig(services_config_path)
|
||||
bec = BECIPythonClient(config, RedisConnector, forced=True, wait_for_server=True)
|
||||
bec.start()
|
||||
|
||||
# flomni.py's module-level `umv` resolves `scans` from the module's own
|
||||
# globals, populated only by the `if builtins.__dict__.get("bec") is not
|
||||
# None:` block at import time. Set builtins first, then reload so it
|
||||
# re-runs (needed even on a fresh process: importing flomni_module above
|
||||
# ran that block once already, before bec existed).
|
||||
builtins.__dict__["bec"] = bec
|
||||
builtins.__dict__["dev"] = bec.device_manager.devices
|
||||
builtins.__dict__["scans"] = bec.scans
|
||||
importlib.reload(flomni_module)
|
||||
builtins.__dict__["umv"] = flomni_module.umv
|
||||
|
||||
neutralize_side_effects()
|
||||
|
||||
flomni = flomni_module.Flomni(bec)
|
||||
|
||||
# RT interferometer feedback + fsamx-in-position setup, mirroring
|
||||
# Flomni._align_setup(). Only intervene when actually needed: fsamx is
|
||||
# normally left read_only (locked) by a *previous*
|
||||
# feedback_enable_with_reset() call (its own last step, so the RT
|
||||
# controller's PID correction owns fine positioning going forward), and
|
||||
# a move request against a read_only device raises DisabledDeviceError.
|
||||
# feedback_enable_with_reset() itself is expensive (re-zeros the
|
||||
# interferometers) -- skip it entirely if feedback is already running,
|
||||
# which it normally is after the first caller in a test session sets it
|
||||
# up (device-server state persists across client (re)connections).
|
||||
rtx = bec.device_manager.devices.rtx
|
||||
if not rtx.controller.feedback_is_running():
|
||||
fsamx = bec.device_manager.devices.fsamx
|
||||
fsamx_in = fsamx.user_parameter.get("in")
|
||||
if abs(fsamx.readback.get() - fsamx_in) > 0.3:
|
||||
fsamx.read_only = False
|
||||
flomni_module.umv(fsamx, fsamx_in)
|
||||
flomni.feedback_enable_with_reset()
|
||||
|
||||
return bec, flomni
|
||||
|
||||
|
||||
def shutdown(bec) -> None:
|
||||
bec.shutdown()
|
||||
bec._client._reset_singleton()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Shared helpers for driving tomo_queue_execute() against the live sim."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
_SUBPROCESS_SCRIPT = Path(__file__).with_name("_run_queue_subprocess.py")
|
||||
|
||||
SHORT_SCAN_PARAMS = dict(
|
||||
# Single acquisition per angle instead of a full Fermat grid -- makes a
|
||||
# queued "tomogram" fast enough to run in a test.
|
||||
single_point_instead_of_fermat_scan=True,
|
||||
tomo_countingtime=0.05,
|
||||
frames_per_trigger=1,
|
||||
fovx=2,
|
||||
fovy=2,
|
||||
tomo_shellstep=1,
|
||||
tomo_angle_range=180,
|
||||
zero_deg_reference_at_each_subtomo=False,
|
||||
)
|
||||
|
||||
# N = int(tomo_angle_range / tomo_angle_stepsize) = 1 -> 8 projections total,
|
||||
# the fastest a real tomogram can be for tests that don't care about the
|
||||
# per-job N itself, just that a job runs cleanly.
|
||||
FAST_STEPSIZE = 170.0
|
||||
|
||||
|
||||
def short_params(flomni, tomo_angle_stepsize: float = FAST_STEPSIZE) -> dict:
|
||||
for name, value in SHORT_SCAN_PARAMS.items():
|
||||
setattr(flomni, name, value)
|
||||
flomni.tomo_angle_stepsize = tomo_angle_stepsize
|
||||
return {name: getattr(flomni, name) for name in flomni._TOMO_SCAN_PARAM_NAMES}
|
||||
|
||||
|
||||
def add_short_job(flomni, label: str, tomo_angle_stepsize: float = FAST_STEPSIZE) -> int:
|
||||
short_params(flomni, tomo_angle_stepsize)
|
||||
return flomni.tomo_queue_add(label)
|
||||
|
||||
|
||||
class ProgressSampler:
|
||||
"""Samples ``flomni.progress[key]`` on a background thread while the
|
||||
caller runs something blocking (e.g. ``tomo_queue_execute()``) in the
|
||||
foreground.
|
||||
|
||||
``tomo_queue_execute()`` itself must stay on the main thread:
|
||||
BECIPythonClient's live-update machinery installs a SIGINT handler per
|
||||
scan request (``ipython_live_updates.process_request``), and Python only
|
||||
allows ``signal.signal()`` from the main thread of the main interpreter
|
||||
-- calling it from a worker thread raises ``ValueError``. So it's the
|
||||
*sampling*, not the execution, that runs in the background here.
|
||||
"""
|
||||
|
||||
def __init__(self, flomni, key: str = "subtomo_total_projections", interval: float = 0.05):
|
||||
self._flomni = flomni
|
||||
self._key = key
|
||||
self._interval = interval
|
||||
self._observations: list = []
|
||||
self._stop = threading.Event()
|
||||
self._thread = threading.Thread(target=self._run, daemon=True)
|
||||
|
||||
def _run(self):
|
||||
while not self._stop.is_set():
|
||||
self._observations.append(self._flomni.progress.get(self._key))
|
||||
time.sleep(self._interval)
|
||||
|
||||
def __enter__(self) -> "ProgressSampler":
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info) -> None:
|
||||
self._stop.set()
|
||||
self._thread.join(timeout=5)
|
||||
|
||||
@property
|
||||
def observations(self) -> list:
|
||||
return list(self._observations)
|
||||
|
||||
def first_index_of(self, value) -> int | None:
|
||||
try:
|
||||
return self._observations.index(value)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def spawn_queue_subprocess(
|
||||
services_config_path, log_path, start_index: int = 0
|
||||
) -> subprocess.Popen:
|
||||
"""Launch _run_queue_subprocess.py as a real OS process against the same
|
||||
sim, so it can be SIGKILLed -- or run concurrently with a second client
|
||||
editing the queue -- the way a real crashed/second kernel would be.
|
||||
|
||||
Output goes to ``log_path``, not a PIPE: a killed/long-running subprocess
|
||||
whose stdout pipe nobody drains can deadlock once the OS pipe buffer
|
||||
fills (tomo_queue_execute() prints a lot -- progress bars per move).
|
||||
"""
|
||||
with open(log_path, "w") as log_file:
|
||||
# Popen dup()s the fd for the child; safe to close our copy right
|
||||
# after spawning instead of leaking it for the subprocess's lifetime.
|
||||
return subprocess.Popen(
|
||||
[sys.executable, str(_SUBPROCESS_SCRIPT), str(services_config_path), str(start_index)],
|
||||
stdout=log_file,
|
||||
stderr=subprocess.STDOUT,
|
||||
)
|
||||
|
||||
|
||||
def wait_until(predicate, timeout: float, interval: float = 0.1):
|
||||
"""Poll ``predicate()`` until it returns a truthy value or ``timeout``
|
||||
elapses. Returns the truthy value, or raises TimeoutError.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
result = predicate()
|
||||
if result:
|
||||
return result
|
||||
time.sleep(interval)
|
||||
raise TimeoutError(f"condition not met within {timeout}s")
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Standalone script: connect to the running sim and call
|
||||
flomni.tomo_queue_execute(). Run as a real OS subprocess (its own main
|
||||
thread) rather than a Python thread inside the test process: BECIPythonClient
|
||||
's live-update machinery installs a SIGINT handler per scan request, which
|
||||
Python only allows from a process's real main thread (see
|
||||
_queue_helpers.ProgressSampler for the same constraint hit from the thread
|
||||
side). Section B/C tests spawn this and SIGKILL it (or edit the queue
|
||||
concurrently from a second client) to simulate a crashed kernel / a second
|
||||
operator session.
|
||||
|
||||
Usage: python _run_queue_subprocess.py <services_config_path> [start_index]
|
||||
Exit code is 0 on a clean run, 1 if tomo_queue_execute() raised.
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
from _bootstrap import build_flomni, shutdown
|
||||
|
||||
|
||||
def main() -> int:
|
||||
services_config_path = sys.argv[1]
|
||||
start_index = int(sys.argv[2]) if len(sys.argv) > 2 else 0
|
||||
|
||||
bec, flomni = build_flomni(services_config_path)
|
||||
try:
|
||||
flomni.tomo_queue_execute(start_index=start_index)
|
||||
except Exception as exc: # noqa: BLE001 - reported via exit code, not re-raised
|
||||
print(f"_run_queue_subprocess: tomo_queue_execute() raised: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
finally:
|
||||
shutdown(bec)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Fixtures for end-to-end tomo-queue tests against a running flOMNI sim.
|
||||
|
||||
Deliberately does NOT use the shipped ``bec_client_lib_with_demo_config`` /
|
||||
``bec_ipython_client_with_demo_config`` fixtures from pytest_bec_e2e: both
|
||||
call ``bec.config.load_demo_config(force=True)``, which against a running sim
|
||||
session overwrites the loaded simulated-flOMNI device config with BEC's demo
|
||||
devices. See TOMO_QUEUE_TESTING.md section 3.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from _bootstrap import build_flomni, shutdown
|
||||
|
||||
_QUEUE_VAR = "tomo_queue"
|
||||
_PROGRESS_VAR = "tomo_progress"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def flomni_sim(bec_redis_fixture, bec_services_config_file_path, bec_servers):
|
||||
"""A live ``Flomni`` instance attached to the already-running sim session.
|
||||
|
||||
See ``_bootstrap.build_flomni()`` for what connecting/constructing this
|
||||
involves and why. The ``tomo_queue`` / ``tomo_progress`` global vars are
|
||||
snapshotted before the test and restored after, so tests don't leak
|
||||
queue/progress state into each other or into a real operator's session
|
||||
on the same sim. The queue is also reset to empty at setup: it lives in
|
||||
the shared sim Redis, so a prior crashed run or manual debugging session
|
||||
can leave stale entries behind that would otherwise contaminate every
|
||||
test that assumes it starts from an empty queue.
|
||||
"""
|
||||
bec, flomni = build_flomni(bec_services_config_file_path)
|
||||
|
||||
queue_snapshot = bec.get_global_var(_QUEUE_VAR)
|
||||
progress_snapshot = bec.get_global_var(_PROGRESS_VAR)
|
||||
bec.set_global_var(_QUEUE_VAR, [])
|
||||
bec.set_global_var(_PROGRESS_VAR, None)
|
||||
|
||||
try:
|
||||
yield flomni
|
||||
finally:
|
||||
bec.set_global_var(_QUEUE_VAR, queue_snapshot)
|
||||
bec.set_global_var(_PROGRESS_VAR, progress_snapshot)
|
||||
shutdown(bec)
|
||||
@@ -14,7 +14,7 @@ paths used by the flomni IPython-client scripts:
|
||||
7. Gripper sample transfer routine (#MNTMODE, #GRGET incl. confirm handshake)
|
||||
8. Simulated cameras (frame generation)
|
||||
|
||||
Run with the repo root on PYTHONPATH: python tests/sim_flomni_harness.py
|
||||
Run with the repo root on PYTHONPATH: python omny_e2e_tests/sim_flomni_harness.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -4,7 +4,7 @@ Offline harness for the simulated LamNI and OMNY hardware stacks.
|
||||
Exercises the real device classes against the protocol simulations without a BEC
|
||||
deployment, covering the paths used by the LamNI/OMNY IPython-client scripts.
|
||||
|
||||
Run with the repo root on PYTHONPATH: python tests/sim_lamni_omny_harness.py
|
||||
Run with the repo root on PYTHONPATH: python omny_e2e_tests/sim_lamni_omny_harness.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -0,0 +1,8 @@
|
||||
"""Smoke test for the flomni_sim fixture -- construct it, touch no scan."""
|
||||
|
||||
|
||||
def test_flomni_sim_constructs_and_reads_params(flomni_sim):
|
||||
flomni = flomni_sim
|
||||
print("sample_name:", flomni.sample_name)
|
||||
print("tomo_countingtime:", flomni.tomo_countingtime)
|
||||
flomni.tomo_queue_show()
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Checklist section A -- sim, no crash needed. See TOMO_QUEUE_TESTING.md."""
|
||||
|
||||
import datetime
|
||||
|
||||
from _queue_helpers import ProgressSampler
|
||||
from _queue_helpers import add_short_job as _add_short_job
|
||||
from _queue_helpers import short_params as _short_params
|
||||
|
||||
|
||||
def test_params_restored_per_job(flomni_sim):
|
||||
"""A1: each job's snapshotted params, not just whatever the global var
|
||||
last held, actually reach the running scan.
|
||||
|
||||
job1 and job2 differ only in tomo_angle_stepsize. sub_tomo_scan() derives
|
||||
progress["subtomo_total_projections"] = int(tomo_angle_range /
|
||||
tomo_angle_stepsize) live, at scan time, from the *live* global var --
|
||||
so if the queue's per-job setattr() restore didn't really work, both
|
||||
jobs would show whatever stepsize happened to be live last (i.e. job2's,
|
||||
since it was set most recently), not each job's own value.
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
|
||||
job1_stepsize = 89.9
|
||||
job2_stepsize = 59.9
|
||||
expected_n = {"job1": int(180 / job1_stepsize), "job2": int(180 / job2_stepsize)}
|
||||
assert expected_n["job1"] != expected_n["job2"], "test setup must pick distinguishable N"
|
||||
|
||||
_add_short_job(flomni, "job1", job1_stepsize)
|
||||
_add_short_job(flomni, "job2", job2_stepsize)
|
||||
|
||||
# Reset first: leftover progress state from a previous run could
|
||||
# otherwise coincidentally satisfy the "value observed" check below
|
||||
# before job1 has actually started.
|
||||
flomni.progress.reset()
|
||||
|
||||
with ProgressSampler(flomni) as sampler:
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
observed = sampler.observations
|
||||
job1_idx = sampler.first_index_of(expected_n["job1"])
|
||||
job2_idx = sampler.first_index_of(expected_n["job2"])
|
||||
|
||||
assert job1_idx is not None, (
|
||||
f"job1 (stepsize={job1_stepsize}) should have shown "
|
||||
f"subtomo_total_projections={expected_n['job1']} at some point; observed {observed}"
|
||||
)
|
||||
assert job2_idx is not None, (
|
||||
f"job2 (stepsize={job2_stepsize}) should have shown "
|
||||
f"subtomo_total_projections={expected_n['job2']} at some point; observed {observed}"
|
||||
)
|
||||
assert job1_idx < job2_idx, (
|
||||
"job1's N should be observed before job2's -- jobs ran out of order "
|
||||
f"(job1 first at {job1_idx}, job2 first at {job2_idx})"
|
||||
)
|
||||
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
assert [j["status"] for j in jobs] == ["done", "done"]
|
||||
|
||||
|
||||
def test_empty_and_all_done_queue_are_noops(flomni_sim, monkeypatch, capsys):
|
||||
"""A3: tomo_queue_execute() on an empty queue, and on a queue whose jobs
|
||||
are all "done", must print a no-op message, never prompt, and never
|
||||
trigger a scan.
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
|
||||
prompt_calls = []
|
||||
from csaxs_bec.bec_ipython_client.plugins.omny.omny_general_tools import OMNYTools
|
||||
|
||||
monkeypatch.setattr(
|
||||
OMNYTools, "yesno", lambda self, *a, **k: prompt_calls.append((a, k)) or True
|
||||
)
|
||||
|
||||
# -- empty queue --
|
||||
flomni._tomo_queue_proxy.clear()
|
||||
capsys.readouterr() # discard anything buffered so far
|
||||
flomni.tomo_queue_execute()
|
||||
out = capsys.readouterr().out
|
||||
assert "Tomo queue is empty." in out
|
||||
assert prompt_calls == []
|
||||
|
||||
# -- all-done queue --
|
||||
_add_short_job(flomni, "already_done")
|
||||
job_id = flomni._tomo_queue_proxy.as_list()[0]["id"]
|
||||
flomni._tomo_queue_proxy.update_by_id(job_id, status="done")
|
||||
capsys.readouterr()
|
||||
flomni.tomo_queue_execute()
|
||||
out = capsys.readouterr().out
|
||||
assert "No pending tomo queue jobs to run." in out
|
||||
assert prompt_calls == []
|
||||
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
assert len(jobs) == 1 and jobs[0]["status"] == "done" # untouched by the no-op call
|
||||
|
||||
|
||||
def test_start_index_semantics(flomni_sim):
|
||||
"""A4: start_index is "position below which fresh pending jobs are
|
||||
ignored" -- it does NOT apply to resumable (incomplete/running) jobs,
|
||||
which always run regardless of position.
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
|
||||
_add_short_job(flomni, "job0")
|
||||
_add_short_job(flomni, "job1")
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
job0_id, job1_id = jobs[0]["id"], jobs[1]["id"]
|
||||
|
||||
flomni.tomo_queue_execute(start_index=1)
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert jobs[job0_id]["status"] == "pending", "job0 should have been skipped by start_index"
|
||||
assert jobs[job1_id]["status"] == "done", "job1 should have run"
|
||||
|
||||
# Mark job0 "incomplete" (as if it had crashed) and drop progress so its
|
||||
# resume is a clean no-op ("No tomo scan in progress to resume"). This
|
||||
# test is about *selection* (does start_index wrongly skip a resumable
|
||||
# job?), not resume accuracy -- that's section B.
|
||||
flomni._tomo_queue_proxy.update_by_id(job0_id, status="incomplete")
|
||||
flomni.progress.reset()
|
||||
|
||||
flomni.tomo_queue_execute(start_index=1)
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert (
|
||||
jobs[job0_id]["status"] == "done"
|
||||
), "job0 is resumable (incomplete) and must run despite start_index=1"
|
||||
assert jobs[job1_id]["status"] == "done", "job1 (already done) should not have been re-run"
|
||||
|
||||
|
||||
def test_legacy_queue_migration(flomni_sim):
|
||||
"""A2: a queue written before jobs carried an ``id`` field (e.g. by an
|
||||
older client version) must run cleanly, and ensure_ids() must heal every
|
||||
job with a unique id as a side effect of tomo_queue_execute().
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
bec = flomni.client
|
||||
|
||||
legacy_jobs = []
|
||||
for label in ("legacy1", "legacy2"):
|
||||
params = _short_params(flomni)
|
||||
legacy_jobs.append(
|
||||
{
|
||||
# deliberately no "id" key -- pre-Step-1 job shape
|
||||
"label": label,
|
||||
"params": params,
|
||||
"status": "pending",
|
||||
"added_at": datetime.datetime.now().isoformat(),
|
||||
}
|
||||
)
|
||||
bec.set_global_var("tomo_queue", legacy_jobs)
|
||||
|
||||
flomni.tomo_queue_show()
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
assert [j["status"] for j in jobs] == ["done", "done"]
|
||||
ids = [j.get("id") for j in jobs]
|
||||
assert all(ids), f"every job should have been healed with an id, got {ids}"
|
||||
assert len(set(ids)) == len(ids), f"healed ids should be unique, got {ids}"
|
||||
@@ -0,0 +1,196 @@
|
||||
"""Step 2 -- command jobs. See AI_docs/TOMO_QUEUE_COMMAND_JOBS_PLAN.md.
|
||||
|
||||
mokev/idgap (the real curated move targets) aren't in the flomni sim config
|
||||
(simulated_flomni.yaml) -- only on the real front end (bl_frontend.yaml), so
|
||||
the real device-moving path here is exercised against `ftray` (sample
|
||||
transfer tray), added to _TOMO_QUEUE_MOVE_DEVICES specifically for this
|
||||
sim-testability, per TOMO_QUEUE_TESTING.md.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
from _queue_helpers import FAST_STEPSIZE, add_short_job
|
||||
|
||||
FTRAY_A = -50.0
|
||||
FTRAY_B = -10.0
|
||||
|
||||
|
||||
def test_add_command_validation_errors(flomni_sim):
|
||||
flomni = flomni_sim
|
||||
|
||||
with pytest.raises(ValueError, match="Unknown queue action"):
|
||||
flomni.tomo_queue_add_command({"action": "not_a_real_action", "kwargs": {}})
|
||||
|
||||
with pytest.raises(ValueError, match="not queue-movable"):
|
||||
flomni.tomo_queue_add_command({"action": "move", "kwargs": {"positions": {"fsamx": 0.0}}})
|
||||
|
||||
with pytest.raises(ValueError, match="above the maximum"):
|
||||
flomni.tomo_queue_add_command(
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 99.0}}
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="must be a number"):
|
||||
flomni.tomo_queue_add_command(
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": "wide"}}
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="has no parameter"):
|
||||
flomni.tomo_queue_add_command({"action": "move", "kwargs": {"bogus_kwarg": 1}})
|
||||
|
||||
# None of the rejected adds should have landed in the queue.
|
||||
assert flomni._tomo_queue_proxy.as_list() == []
|
||||
|
||||
|
||||
def test_move_execution_time_allowlist_recheck(flomni_sim):
|
||||
"""Validation is two-layer (doc §3.3): tomo_queue_add_command() already
|
||||
refuses a disallowed device at add time (see
|
||||
test_add_command_validation_errors), but _queue_action_move() must
|
||||
independently refuse it again at execution time -- the actual safety
|
||||
boundary, since a job could reach the queue by some other path than the
|
||||
validated CLI call (bypassed here by appending the raw dict directly, the
|
||||
same way test_legacy_queue_migration simulates a pre-existing queue
|
||||
entry).
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
job = {
|
||||
"kind": "command",
|
||||
"id": uuid.uuid4().hex,
|
||||
"label": "sneaked past add-time validation",
|
||||
"steps": [{"action": "move", "kwargs": {"positions": {"fsamx": 0.0}}}],
|
||||
"idempotent": True,
|
||||
"status": "pending",
|
||||
"added_at": datetime.datetime.now().isoformat(),
|
||||
}
|
||||
flomni._tomo_queue_proxy.append(job)
|
||||
|
||||
with pytest.raises(ValueError, match="not queue-movable"):
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
assert flomni._tomo_queue_proxy.as_list()[0]["status"] == "incomplete"
|
||||
|
||||
|
||||
def test_tomo_queue_actions_global_var_matches_registry(flomni_sim):
|
||||
flomni = flomni_sim
|
||||
published = flomni.client.get_global_var("tomo_queue_actions")
|
||||
assert set(published["actions"]) == {"move", "optimize_idgap"}
|
||||
assert published["move_devices"]["ftray"]["unit"] == "mm"
|
||||
assert published["actions"]["move"]["idempotent"] is True
|
||||
assert published["actions"]["optimize_idgap"]["params"]["search_range"]["default"] == 0.5
|
||||
|
||||
|
||||
def test_show_and_webpage_survive_command_job(flomni_sim, capsys):
|
||||
flomni = flomni_sim
|
||||
flomni.tomo_queue_add_command(
|
||||
{"action": "move", "kwargs": {"positions": {"ftray": FTRAY_A}}},
|
||||
label="move ftray",
|
||||
idempotent=True,
|
||||
)
|
||||
capsys.readouterr()
|
||||
jobs = flomni.tomo_queue_show()
|
||||
out = capsys.readouterr().out
|
||||
assert "CMD move{'positions': {'ftray': -50.0}}" in out
|
||||
assert "[idem]" in out
|
||||
assert jobs[0]["kind"] == "command"
|
||||
|
||||
from csaxs_bec.bec_ipython_client.plugins.flomni.flomni_webpage_generator import _render_html
|
||||
|
||||
html = _render_html(phone_numbers=[], has_tomo_queue=True, tomo_types={1: "x"})
|
||||
assert "isCommand" in html # the rendering branch exists; full JS isn't executed here
|
||||
|
||||
|
||||
def test_legacy_job_no_kind_defaults_to_tomo_and_command_moves_ftray(flomni_sim):
|
||||
"""B-style back-compat + the real command-job execution path, combined
|
||||
into one mixed-queue run to keep the (only) real scan in this file to
|
||||
one job.
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
bec = flomni.client
|
||||
|
||||
# Start from a known position, not whatever a previous test run happened
|
||||
# to leave ftray at -- ftray's position is real device-server state that
|
||||
# persists across separate pytest invocations, unlike the tomo_queue/
|
||||
# tomo_progress global vars the flomni_sim fixture snapshots/restores.
|
||||
flomni._queue_action_move({"ftray": FTRAY_B})
|
||||
ftray_before = bec.device_manager.devices.ftray.readback.get()
|
||||
assert ftray_before == pytest.approx(FTRAY_B)
|
||||
|
||||
add_short_job(flomni, "legacy_tomo", FAST_STEPSIZE)
|
||||
# Simulate a queue entry written before the "kind" field existed.
|
||||
jobs = bec.get_global_var("tomo_queue")
|
||||
del jobs[0]["kind"]
|
||||
bec.set_global_var("tomo_queue", jobs)
|
||||
|
||||
flomni.tomo_queue_add_command(
|
||||
{"action": "move", "kwargs": {"positions": {"ftray": FTRAY_A}}}, label="move ftray"
|
||||
)
|
||||
flomni.tomo_queue_add_command(
|
||||
{"action": "optimize_idgap", "kwargs": {"search_range": 0.5}}, label="idgap stub"
|
||||
)
|
||||
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
assert [j["status"] for j in jobs] == ["done", "done", "done"]
|
||||
# Still no "kind" key -- proves it ran (and got marked done) via the
|
||||
# back-compat default, not that execute() silently added one.
|
||||
assert "kind" not in jobs[0]
|
||||
assert "params" in jobs[0]
|
||||
assert jobs[1]["kind"] == "command"
|
||||
|
||||
ftray_after = bec.device_manager.devices.ftray.readback.get()
|
||||
assert ftray_after == pytest.approx(FTRAY_A)
|
||||
assert ftray_after != pytest.approx(ftray_before)
|
||||
|
||||
|
||||
def test_non_idempotent_command_resume_prompts(flomni_sim, monkeypatch):
|
||||
flomni = flomni_sim
|
||||
bec = flomni.client
|
||||
ftray = bec.device_manager.devices.ftray
|
||||
|
||||
# Use the same real move path the "move" action itself uses, rather than
|
||||
# the device's underlying ophyd object directly (ftray's BEC wrapper is
|
||||
# a Positioner here, unlike fsamx's DeviceBase -- no .obj to reach for).
|
||||
flomni._queue_action_move({"ftray": FTRAY_B})
|
||||
start = ftray.readback.get()
|
||||
assert start == pytest.approx(FTRAY_B)
|
||||
|
||||
index = flomni.tomo_queue_add_command(
|
||||
{"action": "move", "kwargs": {"positions": {"ftray": FTRAY_A}}},
|
||||
label="non-idempotent move",
|
||||
idempotent=False,
|
||||
)
|
||||
job_id = flomni._tomo_queue_proxy.as_list()[index]["id"]
|
||||
flomni._tomo_queue_proxy.update_by_id(job_id, status="incomplete")
|
||||
|
||||
prompts = []
|
||||
from csaxs_bec.bec_ipython_client.plugins.omny.omny_general_tools import OMNYTools
|
||||
|
||||
def _yesno(resume_answer):
|
||||
# tomo_queue_execute()'s own top-level "OK to start?" confirmation
|
||||
# goes through the same OMNYTools.yesno -- always let that one
|
||||
# through, and only record/answer the *resume* prompt from
|
||||
# _run_command_job, which is what this test is actually about.
|
||||
def _fn(self, message, *a, **k):
|
||||
if message.startswith("Starting automatic execution"):
|
||||
return True
|
||||
prompts.append(message)
|
||||
return resume_answer
|
||||
|
||||
return _fn
|
||||
|
||||
# "no" -> must not re-run: ftray stays put, job still ends up "done".
|
||||
monkeypatch.setattr(OMNYTools, "yesno", _yesno(False))
|
||||
flomni.tomo_queue_execute()
|
||||
assert len(prompts) == 1
|
||||
assert ftray.readback.get() == pytest.approx(start)
|
||||
assert flomni._tomo_queue_proxy.as_list()[0]["status"] == "done"
|
||||
|
||||
# Re-arm as incomplete and try again with "yes" -> must re-run.
|
||||
flomni._tomo_queue_proxy.update_by_id(job_id, status="incomplete")
|
||||
monkeypatch.setattr(OMNYTools, "yesno", _yesno(True))
|
||||
flomni.tomo_queue_execute()
|
||||
assert len(prompts) == 2
|
||||
assert ftray.readback.get() == pytest.approx(FTRAY_A)
|
||||
assert flomni._tomo_queue_proxy.as_list()[0]["status"] == "done"
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Checklist section C -- concurrency, the path the GUI will use. See TOMO_QUEUE_TESTING.md.
|
||||
|
||||
Session A (executing the queue) is always a subprocess -- tomo_queue_execute()
|
||||
must run on a process's real main thread (see _queue_helpers.ProgressSampler).
|
||||
Session B (editing concurrently) is the pytest process's own flomni_sim
|
||||
connection: a second, independent client hitting the same redis-backed
|
||||
tomo_queue global var, exactly like a second operator's kernel or the future
|
||||
GUI dialog would.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import uuid
|
||||
|
||||
from _queue_helpers import add_short_job, spawn_queue_subprocess, wait_until
|
||||
|
||||
|
||||
def test_concurrent_queue_edits_from_second_client(
|
||||
flomni_sim, bec_services_config_file_path, tmp_path
|
||||
):
|
||||
"""C8: while session A executes the queue, session B reorders the
|
||||
pending tail, deletes a pending job, and appends a new one, all mid-job1.
|
||||
Session A must pick up every change at the next job boundary and write
|
||||
statuses to the right jobs -- the real-redis version of what the offline
|
||||
mock harness proved in memory (section 2 of the doc).
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
bec = flomni.client
|
||||
|
||||
add_short_job(flomni, "job1")
|
||||
add_short_job(flomni, "job2")
|
||||
add_short_job(flomni, "job3")
|
||||
add_short_job(flomni, "job4")
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
ids = {j["label"]: j["id"] for j in jobs}
|
||||
|
||||
proc = spawn_queue_subprocess(bec_services_config_file_path, tmp_path / "sessionA.log")
|
||||
try:
|
||||
wait_until(lambda: flomni._tomo_queue_proxy.as_list()[0]["status"] == "running", timeout=30)
|
||||
|
||||
# -- session B's edits, all mid-job1: reorder the pending tail
|
||||
# (job4 ahead of job2), delete job3, append job5. Deliberately does
|
||||
# NOT touch any live tomo_* param global var here (only the
|
||||
# tomo_queue list itself) -- job1's own live scan in session A is
|
||||
# reading those same params every projection; touching them mid-run
|
||||
# would perturb the *running* job, which is a different hazard than
|
||||
# the one this test is about (see TOMO_QUEUE_COMMAND_JOBS_PLAN's
|
||||
# "Done must be blocked while running" rule for the GUI).
|
||||
current = {j["id"]: j for j in bec.get_global_var("tomo_queue")}
|
||||
reordered = [
|
||||
current[ids["job1"]], # the running job, left in place
|
||||
current[ids["job4"]],
|
||||
current[ids["job2"]],
|
||||
]
|
||||
bec.set_global_var("tomo_queue", reordered)
|
||||
|
||||
job5 = dict(current[ids["job1"]]) # clone an existing job's params snapshot
|
||||
job5.update(
|
||||
id=uuid.uuid4().hex,
|
||||
label="job5",
|
||||
status="pending",
|
||||
added_at=datetime.datetime.now().isoformat(),
|
||||
)
|
||||
flomni._tomo_queue_proxy.append(job5)
|
||||
|
||||
returncode = proc.wait(timeout=420) # 4 real jobs x ~8 projections each
|
||||
finally:
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.wait(timeout=10)
|
||||
|
||||
log_text = (tmp_path / "sessionA.log").read_text()
|
||||
assert returncode == 0, f"session A failed (see log):\n{log_text[-4000:]}"
|
||||
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
labels_in_order = [j["label"] for j in jobs]
|
||||
statuses = {j["label"]: j["status"] for j in jobs}
|
||||
|
||||
assert "job3" not in labels_in_order, "job3 was deleted mid-run and must not reappear"
|
||||
assert labels_in_order == [
|
||||
"job1",
|
||||
"job4",
|
||||
"job2",
|
||||
"job5",
|
||||
], f"queue should reflect the reorder+append made mid-run, got {labels_in_order}"
|
||||
assert statuses == {"job1": "done", "job4": "done", "job2": "done", "job5": "done"}
|
||||
|
||||
|
||||
def test_delete_running_job_mid_run(flomni_sim, bec_services_config_file_path, tmp_path):
|
||||
"""C9: deleting the *running* job mid-run (unprotected at CLI level --
|
||||
the GUI will block it, per section 5 of the doc) must degrade
|
||||
gracefully: update_by_id() returns False, job1's scan finishes on its
|
||||
own terms, nothing crashes, and no status gets written to the wrong job.
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
|
||||
add_short_job(flomni, "job1")
|
||||
add_short_job(flomni, "job2")
|
||||
|
||||
proc = spawn_queue_subprocess(bec_services_config_file_path, tmp_path / "sessionA.log")
|
||||
try:
|
||||
wait_until(lambda: flomni._tomo_queue_proxy.as_list()[0]["status"] == "running", timeout=30)
|
||||
|
||||
# Delete the running job (index 0) out from under session A.
|
||||
flomni.tomo_queue_delete(0)
|
||||
|
||||
returncode = proc.wait(timeout=180)
|
||||
finally:
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.wait(timeout=10)
|
||||
|
||||
log_text = (tmp_path / "sessionA.log").read_text()
|
||||
assert (
|
||||
returncode == 0
|
||||
), f"session A should degrade gracefully, not crash (see log):\n{log_text[-4000:]}"
|
||||
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
labels = [j["label"] for j in jobs]
|
||||
assert labels == ["job2"], f"deleted job1 must not reappear, got {labels}"
|
||||
assert jobs[0]["status"] == "done", "job2 should still have run after job1's scan finished"
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Checklist section B -- sim, crash/interrupt paths. See TOMO_QUEUE_TESTING.md.
|
||||
|
||||
B5 (real SIGKILL crash-resume) lives in test_tomo_queue_crash_subprocess.py --
|
||||
it needs a real OS process to kill, unlike B6/B7 here.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from _queue_helpers import add_short_job
|
||||
|
||||
|
||||
def test_exception_mid_scan_pauses_queue(flomni_sim, monkeypatch):
|
||||
"""B6: tomo_scan() raising mid-job must leave that job "incomplete",
|
||||
re-raise out of tomo_queue_execute(), and leave later jobs untouched
|
||||
("pending"). Re-running afterward must resume the failed job first,
|
||||
before the still-pending one.
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
|
||||
add_short_job(flomni, "job1")
|
||||
add_short_job(flomni, "job2")
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
job1_id, job2_id = jobs[0]["id"], jobs[1]["id"]
|
||||
|
||||
call_count = {"n": 0}
|
||||
original_tomo_scan = type(flomni).tomo_scan
|
||||
|
||||
def _raise_once(self, *a, **k):
|
||||
call_count["n"] += 1
|
||||
if call_count["n"] == 1:
|
||||
raise RuntimeError("simulated mid-scan failure")
|
||||
return original_tomo_scan(self, *a, **k)
|
||||
|
||||
monkeypatch.setattr(type(flomni), "tomo_scan", _raise_once)
|
||||
|
||||
with pytest.raises(RuntimeError, match="simulated mid-scan failure"):
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert jobs[job1_id]["status"] == "incomplete", "failed job must be marked incomplete"
|
||||
assert jobs[job2_id]["status"] == "pending", "later job must be untouched by the failure"
|
||||
|
||||
# tomo_scan() raised before setting progress["tomo_start_time"], so the
|
||||
# resume attempt below is itself a clean no-op ("No tomo scan in
|
||||
# progress to resume") -- this test is about *selection order*
|
||||
# (resume-before-fresh), not resume accuracy (that's B5).
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert jobs[job1_id]["status"] == "done", "the previously-failed job must have been retried"
|
||||
assert jobs[job2_id]["status"] == "done", "the pending job must have run after it"
|
||||
assert call_count["n"] == 2, "tomo_scan should have been called exactly once per attempt"
|
||||
|
||||
|
||||
def test_resume_before_fresh_explicit(flomni_sim, monkeypatch):
|
||||
"""B7: an "incomplete" job below a fresh "pending" job in list order
|
||||
must still run first, regardless of position (new pick-next behavior;
|
||||
the old strictly-in-order loop couldn't reorder around this at all).
|
||||
"""
|
||||
flomni = flomni_sim
|
||||
|
||||
add_short_job(flomni, "fresh_pending")
|
||||
add_short_job(flomni, "crashed_earlier")
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
fresh_id, crashed_id = jobs[0]["id"], jobs[1]["id"]
|
||||
|
||||
# crashed_earlier sits BELOW fresh_pending in list order, but is
|
||||
# resumable -- it must be picked first regardless.
|
||||
flomni._tomo_queue_proxy.update_by_id(crashed_id, status="incomplete")
|
||||
flomni.progress.reset() # so its "resume" is a clean, fast no-op
|
||||
|
||||
run_order = []
|
||||
original_update_by_id = type(flomni._tomo_queue_proxy).update_by_id
|
||||
|
||||
def _track(self, job_id, **kwargs):
|
||||
if kwargs.get("status") == "running":
|
||||
run_order.append(job_id)
|
||||
return original_update_by_id(self, job_id, **kwargs)
|
||||
|
||||
monkeypatch.setattr(type(flomni._tomo_queue_proxy), "update_by_id", _track)
|
||||
flomni.tomo_queue_execute()
|
||||
|
||||
assert run_order == [
|
||||
crashed_id,
|
||||
fresh_id,
|
||||
], f"crashed_earlier (resumable) should run before fresh_pending, got order {run_order}"
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert jobs[fresh_id]["status"] == "done"
|
||||
assert jobs[crashed_id]["status"] == "done"
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Checklist section B5 -- real crash-resume via SIGKILL. See TOMO_QUEUE_TESTING.md.
|
||||
|
||||
Needs a real OS process to kill: a SIGKILLed process never reaches
|
||||
tomo_queue_execute()'s except-block, so the running job's status is left
|
||||
stuck at "running" -- exactly the "kernel crashed mid-scan" scenario the doc
|
||||
calls for, and not reproducible by raising an exception in-process (that's
|
||||
B6, in test_tomo_queue_crash.py).
|
||||
"""
|
||||
|
||||
import signal
|
||||
|
||||
from _queue_helpers import add_short_job, spawn_queue_subprocess, wait_until
|
||||
|
||||
|
||||
def test_real_crash_resume(flomni_sim, bec_services_config_file_path, tmp_path):
|
||||
flomni = flomni_sim
|
||||
|
||||
add_short_job(flomni, "job1")
|
||||
add_short_job(flomni, "job2")
|
||||
jobs = flomni._tomo_queue_proxy.as_list()
|
||||
job1_id, job2_id = jobs[0]["id"], jobs[1]["id"]
|
||||
|
||||
flomni.progress.reset()
|
||||
|
||||
proc = spawn_queue_subprocess(bec_services_config_file_path, tmp_path / "run1.log")
|
||||
try:
|
||||
# Wait until job1 is genuinely mid-scan (subtomo 3 of 8, at
|
||||
# FAST_STEPSIZE's N=1-projection-per-subtomo) before killing -- not
|
||||
# the very first projection, so this is unambiguously a *resume*,
|
||||
# not indistinguishable from a fresh start.
|
||||
wait_until(lambda: (flomni.progress.get("subtomo") or 0) >= 3, timeout=90)
|
||||
|
||||
# One atomic snapshot -- reading subtomo/angle/heartbeat as three
|
||||
# separate round-trips could catch values from different instants.
|
||||
before = flomni.progress.as_dict()
|
||||
|
||||
proc.send_signal(signal.SIGKILL)
|
||||
proc.wait(timeout=10)
|
||||
finally:
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.wait(timeout=10)
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert jobs[job1_id]["status"] == "running", (
|
||||
"a SIGKILLed job never reaches tomo_queue_execute()'s except-block, so "
|
||||
f"it should be stuck at 'running', not {jobs[job1_id]['status']!r}"
|
||||
)
|
||||
assert jobs[job2_id]["status"] == "pending", "job2 must be untouched by job1's crash"
|
||||
|
||||
# Resume: a second, fresh subprocess (a stand-in for "operator restarts
|
||||
# their kernel") should pick job1 up mid-scan, not restart it.
|
||||
proc2 = spawn_queue_subprocess(bec_services_config_file_path, tmp_path / "run2.log")
|
||||
try:
|
||||
after = wait_until(
|
||||
lambda: _snapshot_if_new_heartbeat(flomni, before["heartbeat"]), timeout=60
|
||||
)
|
||||
returncode = proc2.wait(timeout=180)
|
||||
finally:
|
||||
if proc2.poll() is None:
|
||||
proc2.kill()
|
||||
proc2.wait(timeout=10)
|
||||
|
||||
log_text = (tmp_path / "run2.log").read_text()
|
||||
assert returncode == 0, f"resume subprocess failed (see log):\n{log_text[-4000:]}"
|
||||
|
||||
assert after["subtomo"] == before["subtomo"], (
|
||||
f"resume should re-acquire the exact in-flight subtomo {before['subtomo']}, "
|
||||
f"got {after['subtomo']}"
|
||||
)
|
||||
assert after["angle"] == before["angle"], (
|
||||
f"resume should re-acquire the exact in-flight angle {before['angle']}, "
|
||||
f"got {after['angle']}"
|
||||
)
|
||||
|
||||
jobs = {j["id"]: j for j in flomni._tomo_queue_proxy.as_list()}
|
||||
assert jobs[job1_id]["status"] == "done", "job1 should have completed after resuming"
|
||||
assert jobs[job2_id]["status"] == "done", "job2 should have run after job1 finished"
|
||||
|
||||
|
||||
def _snapshot_if_new_heartbeat(flomni, heartbeat_before):
|
||||
"""Return a full progress snapshot the first time ``heartbeat`` differs
|
||||
from its pre-kill value (i.e. a fresh `_tomo_scan_at_angle` attempt has
|
||||
started), else None so wait_until() keeps polling.
|
||||
"""
|
||||
snapshot = flomni.progress.as_dict()
|
||||
if snapshot.get("heartbeat") != heartbeat_before:
|
||||
return snapshot
|
||||
return None
|
||||
@@ -38,6 +38,7 @@ dev = [
|
||||
"pylint",
|
||||
"pytest",
|
||||
"pytest-random-order",
|
||||
"pytest-bec-e2e",
|
||||
"ophyd_devices",
|
||||
"bec_server",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user