photon-diagnostics/wiki
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update Pipeline server

Browse Source
This commit is contained in:
arrell_c 2025-01-27 12:44:47 +01:00
parent 96d6839c46
commit d325547bc4
1 changed files with 26 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
Pipeline-server.md
View File

@ -2,33 +2,42 @@
Logs from the pipeline server are now available on kibana.
Example query strings are:
- "message": “name:<PIPELINE_NAME>" (for pipeline logs)
- "message": "camera:<CAMERA_NAME>" (for camera logs)
- "host.hostname":"sf-daqsync-XX.psi.ch" (for server logs)
- "message": “name:<PIPELINE_NAME>" (for pipeline logs)
- "message": "camera:<CAMERA_NAME>" (for camera logs)
- "host.hostname":"sf-daqsync-XX.psi.ch" (for server logs)
- “systemd.unit":”XXX.service" (for service logs)
https://kibana.psi.ch/s/camserver/app/logs
[https://kibana.psi.ch/s/camserver/app/logs](https://kibana.psi.ch/s/camserver/app/logs)
To check the metrics of the cluster (CPU. Memory, network):
https://kibana.psi.ch/s/camserver/app/metrics/inventory
To check the metrics of the cluster (CPU, Memory, network):
[https://kibana.psi.ch/s/camserver/app/metrics/inventory](https://kibana.psi.ch/s/camserver/app/metrics/inventory)
Or a specific server directly:
https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch
[https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch](https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch)
To monitor service availability and automatic alarm reporting (by email):
https://kibana.psi.ch/s/camserver/app/uptime
https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open
[https://kibana.psi.ch/s/camserver/app/uptime](https://kibana.psi.ch/s/camserver/app/uptime)
[https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open](https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open)
Dashboards to display individual pipeline metrics.
E.g this one for PSSS:
https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))
Dashboards to display individual pipeline metrics.
E.g. this one for PSSS:
[https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))](https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now)))
Dashboards to stream individual pipeline metrics.
- [SAROP21-PBPS103_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n)&logPosition=(end:now,position:!n,start:now-1d,streamLive:!t)&logFilter=(language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS103_proc%22%27))
### Dashboards to stream individual pipeline metrics
- [SARFE10-PSSS059_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n)&logPosition=(end:now,position:(tiebreaker:545,time:1734080766159),start:now-1d,streamLive:!t)&logFilter=(language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss%22%27))
| **PBPS Device** | **Kibana Log Stream** |
|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| SARFE10-PBPS053 | [SARFE10-PBPS053_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PBPS053_proc%22%27%29) |
| SAROP11-PBPS110 | [SAROP11-PBPS110_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS110_proc%22%27%29) |
| SAROP11-PBPS122 | [SAROP11-PBPS122_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS122_proc%22%27%29) |
| SAROP21-PBPS103 | [SAROP21-PBPS103_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS103_proc%22%27%29) |
| SAROP21-PBPS133 | [SAROP21-PBPS133_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS133_proc%22%27%29) |
| SARES21-PBPS141 | [SARES21-PBPS141_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARES21-PBPS141_proc%22%27%29) |
| SAROP31-PBPS113 | [SAROP31-PBPS113_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP31-PBPS113_proc%22%27%29) |
| SAROP31-PBPS149 | [SAROP31-PBPS149_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP31-PBPS149_proc%22%27%29) |
- [SARFE10-PSSS059_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n)&logPosition=(end:now,position:(tiebreaker:545,time:1734080766159),start:now-1d,streamLive:!t)&logFilter=(language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss%22%27))
**Note the stream may take around 30 seconds to load**
## Cam Server git
https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py
[https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py](https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py)