34 lines
1.9 KiB
Markdown
34 lines
1.9 KiB
Markdown
## Pipeline server debugging
|
|
Logs from the pipeline server are now available on kibana.
|
|
|
|
Example query strings are:
|
|
- "message": “name:<PIPELINE_NAME>" (for pipeline logs)
|
|
- "message": "camera:<CAMERA_NAME>" (for camera logs)
|
|
- "host.hostname":"sf-daqsync-XX.psi.ch" (for server logs)
|
|
- “systemd.unit":”XXX.service" (for service logs)
|
|
|
|
https://kibana.psi.ch/s/camserver/app/logs
|
|
|
|
To check the metrics of the cluster (CPU. Memory, network):
|
|
https://kibana.psi.ch/s/camserver/app/metrics/inventory
|
|
|
|
Or a specific server directly:
|
|
https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch
|
|
|
|
To monitor service availability and automatic alarm reporting (by email):
|
|
https://kibana.psi.ch/s/camserver/app/uptime
|
|
https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open
|
|
|
|
Dashboards to display individual pipeline metrics.
|
|
E.g this one for PSSS:
|
|
https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))
|
|
|
|
Dashboards to stream individual pipeline metrics.
|
|
E.g this one for SAROP21-PBPS103_proc:
|
|
https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n)&logPosition=(end:now,position:!n,start:now-1d,streamLive:!t)&logFilter=(language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS103_proc%22%27)
|
|
|
|
**Note the stream may take around 30 seconds to load**
|
|
|
|
## Cam Server git
|
|
|
|
https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py |