6.4 KiB
Pipeline server debugging
Logs from the pipeline server are now available on kibana.
Example query strings are:
- "message": “name:<PIPELINE_NAME>" (for pipeline logs)
- "message": "camera:<CAMERA_NAME>" (for camera logs)
- "host.hostname":"sf-daqsync-XX.psi.ch" (for server logs)
- “systemd.unit":”XXX.service" (for service logs)
https://kibana.psi.ch/s/camserver/app/logs
To check the metrics of the cluster (CPU, Memory, network):
https://kibana.psi.ch/s/camserver/app/metrics/inventory
Or a specific server directly:
https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch
To monitor service availability and automatic alarm reporting (by email):
https://kibana.psi.ch/s/camserver/app/uptime
https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open
Dashboards to display individual pipeline metrics.
E.g. this one for PSSS:
https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))
Dashboards to stream individual pipeline metrics
| PBPS Device | Kibana Log Stream |
|---|---|
| SARFE10-PBPS053 | SARFE10-PBPS053_proc |
| SAROP11-PBPS110 | SAROP11-PBPS110_proc |
| SAROP11-PBPS122 | SAROP11-PBPS122_proc |
| SAROP21-PBPS103 | SAROP21-PBPS103_proc |
| SAROP21-PBPS133 | SAROP21-PBPS133_proc |
| SARES21-PBPS141 | SARES21-PBPS141_proc |
| SAROP31-PBPS113 | SAROP31-PBPS113_proc |
| SAROP31-PBPS149 | SAROP31-PBPS149_proc |
- SARFE10-PSSS059_psss
Note the stream may take around 30 seconds to load