menzelandClaude Opus 4.8
513252a60c
Guard the recovery notification on msg_count > 0, so a brief excursion that never produced an alarm message (e.g. recovered before any send, or all sends failed) does not emit a lone RESOLVED. Recovery is still logged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
132 lines
5.0 KiB
Markdown
132 lines
5.0 KiB
Markdown
## EPICS2SMS
|
|
|
|
Monitor one or more EPICS PVs and send an SMS alert via PSI mail infrastructure.
|
|
|
|
This tool **does not use local `/usr/sbin/sendmail`**. It sends mail via **direct SMTP**
|
|
to `smtp.psi.ch` to ensure the message is handled as a registered
|
|
system/application sender.
|
|
|
|
### Files
|
|
|
|
- `epics2sms.sh`
|
|
Runs one or more independent **monitors**, each polling a PV, keeping a
|
|
moving average, and sending SMS/mail when the average crosses a directional
|
|
threshold.
|
|
|
|
- `smtp_send.py`
|
|
SMTP transport helper (no authentication). Sets both header `From:` and
|
|
envelope sender (`MAIL FROM`) to the configured sender address.
|
|
|
|
### Monitors
|
|
|
|
Each monitor is one pipe-separated entry in the `MONITORS` array:
|
|
|
|
```
|
|
pv | dir | threshold | poll_s | alarm_poll_s | n_avg | n_msg | msg_delay_s | text
|
|
```
|
|
|
|
| Field | Meaning |
|
|
|----------------|---------------------------------------------------------------------|
|
|
| `pv` | EPICS PV name |
|
|
| `dir` | alarm direction: `lt` (avg `<` threshold) or `gt` (avg `>` threshold) |
|
|
| `threshold` | numeric threshold |
|
|
| `poll_s` | seconds between polls while in normal state |
|
|
| `alarm_poll_s` | seconds between polls while in alarm (how often the average is refreshed and recovery is checked) |
|
|
| `n_avg` | samples averaged before an alarm can fire ("N Mittel") |
|
|
| `n_msg` | max messages per alarm episode, then silent ("N Messages") |
|
|
| `msg_delay_s` | minimum seconds between two alarm messages; polling keeps running faster, but messages are throttled to this spacing (the first message of an episode goes out immediately) |
|
|
| `text` | message text, also the start of the SMS body |
|
|
|
|
Multiple monitors may watch the **same** PV with different thresholds and
|
|
timing; each runs as its own background loop with independent state.
|
|
|
|
Example (current config):
|
|
|
|
```
|
|
MONITORS=(
|
|
"X12SA-OP-CC:L1Level_MON|lt|55|60|60|4|10|300|X12SA-OP-CC:L1Level_MON <55%"
|
|
"X12SA-OP-CC:L1Level_MON|gt|75|10|10|4|10|300|X12SA-OP-CC:L1Level_MON >75%"
|
|
)
|
|
```
|
|
|
|
On startup, the script sends **one brief notification** to all recipients
|
|
listing the monitors it is watching (PV, threshold, timing) so you can confirm
|
|
it is running. This is best-effort: if it fails to send, monitoring still
|
|
starts.
|
|
|
|
Behaviour of one monitor:
|
|
|
|
- polls the PV every `poll_s` seconds while normal, every `alarm_poll_s` while
|
|
in alarm
|
|
- keeps a moving average of the last `n_avg` samples; an alarm can only fire
|
|
once the window is full, so a single spurious reading cannot trigger it
|
|
- while the average is in alarm, sends the first message immediately, then up
|
|
to `n_msg` messages total, spaced at least `msg_delay_s` apart, then stays
|
|
silent
|
|
- once the average returns to normal, it resets the message counter and — if
|
|
at least one alarm message was sent this episode — sends **one** recovery
|
|
("RESOLVED") message
|
|
- unreadable / non-numeric reads are logged and skipped (they do not enter the
|
|
average and never trigger an alarm)
|
|
|
|
### Why SMTP instead of sendmail?
|
|
|
|
On beamline console hosts, local `sendmail` may be routed through internal relays
|
|
or firewall rules and can lead to **silent drops** for mail-to-SMS gateways.
|
|
Direct SMTP to `smtp.psi.ch` makes the transport path explicit and reproducible.
|
|
|
|
### Requirements
|
|
|
|
- EPICS CA tools available on the host (`caget`).
|
|
- Python 3 available (`python3`).
|
|
- Network access from the host to `smtp.psi.ch:25`.
|
|
|
|
### Configuration
|
|
|
|
Edit `epics2sms.sh`:
|
|
|
|
- `MONITORS`
|
|
One entry per monitor (see **Monitors** above).
|
|
- `RECIPIENTS`
|
|
Applied to every monitor. Include the SMS gateway address, e.g.:
|
|
- `0041797554007@sms.switch.ch`
|
|
- `079XXXXXXX@mail2sms.swisscom.com`
|
|
- `FROM_ADDR`
|
|
Must be a **registered PSI system/application sender**, e.g. `cSAXS@psi.ch`.
|
|
- `CA_TIMEOUT`
|
|
Channel Access timeout (`caget -w`) in seconds. The `caget` default (~1 s) can
|
|
be too short for the first connection to a PV, which then looks like
|
|
"not connected"; keep this well below the shortest poll interval.
|
|
|
|
`FROM_ADDR` is critical: messages from host/user addresses like
|
|
`e23080@x12sa-cons-02.psi.ch` may be filtered and never reach the SMS gateway.
|
|
|
|
### Quick test
|
|
|
|
Send a normal email:
|
|
|
|
```
|
|
python3 smtp_send.py --to andreas.menzel@psi.ch --from-addr cSAXS@psi.ch "Test $(date)"
|
|
```
|
|
|
|
Send an SMS (via mail2sms):
|
|
|
|
```
|
|
python3 smtp_send.py --to 079XXXXXXX@mail2sms.swisscom.com --from-addr cSAXS@psi.ch "SMS test $(date)"
|
|
```
|
|
|
|
### Run the monitor
|
|
|
|
```
|
|
./epics2sms.sh
|
|
```
|
|
|
|
All monitors run in one process; `Ctrl-C` (or `SIGTERM`) stops them together.
|
|
|
|
### Notes / Troubleshooting
|
|
|
|
- If no SMS arrives but SMTP returns success, ask IT to check Exchange/relay
|
|
transport rules for the registered sender and the mail2sms gateway domain.
|
|
- Use `smtp_send.py` directly for transport debugging (it should return non-zero
|
|
on errors).
|