61 lines
7.4 KiB
Markdown
61 lines
7.4 KiB
Markdown
## Pipeline server debugging
|
|
Logs from the pipeline server are now available on Kibana.
|
|
|
|
Example query strings are:
|
|
- `"message": "name:<PIPELINE_NAME>"` (for pipeline logs)
|
|
- `"message": "camera:<CAMERA_NAME>"` (for camera logs)
|
|
- `"host.hostname":"sf-daqsync-XX.psi.ch"` (for server logs)
|
|
- `"systemd.unit":"XXX.service"` (for service logs)
|
|
|
|
[Kibana Log Stream](https://kibana.psi.ch/s/camserver/app/logs)
|
|
|
|
To check the metrics of the cluster (CPU, Memory, network):
|
|
[Kibana Metrics](https://kibana.psi.ch/s/camserver/app/metrics/inventory)
|
|
|
|
Or a specific server directly:
|
|
[Kibana Server Metrics](https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch)
|
|
|
|
To monitor service availability and automatic alarm reporting (by email):
|
|
[Service Uptime](https://kibana.psi.ch/s/camserver/app/uptime)
|
|
[Open Alerts](https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open)
|
|
|
|
Dashboards to display individual pipeline metrics.
|
|
E.g., this one for PSSS:
|
|
[PSSS Dashboard](https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=%28filters:!%28%29,refreshInterval:%28pause:!t,value:0%29,time:%28from:now-1h,to:now%29%29)
|
|
|
|
---
|
|
|
|
### **Dashboards to stream individual pipeline metrics**
|
|
|
|
#### **PBPS Pipelines**
|
|
| **PBPS Device** | **Kibana Log Stream** |
|
|
|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
| SARFE10-PBPS053 | [SARFE10-PBPS053_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PBPS053_proc%22%27%29) |
|
|
| SAROP11-PBPS110 | [SAROP11-PBPS110_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS110_proc%22%27%29) |
|
|
| SAROP11-PBPS122 | [SAROP11-PBPS122_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS122_proc%22%27%29) |
|
|
| SAROP21-PBPS103 | [SAROP21-PBPS103_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS103_proc%22%27%29) |
|
|
| SAROP21-PBPS133 | [SAROP21-PBPS133_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS133_proc%22%27%29) |
|
|
| SARES21-PBPS141 | [SARES21-PBPS141_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARES21-PBPS141_proc%22%27%29) |
|
|
| SAROP31-PBPS113 | [SAROP31-PBPS113_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP31-PBPS113_proc%22%27%29) |
|
|
| SAROP31-PBPS149 | [SAROP31-PBPS149_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP31-PBPS149_proc%22%27%29) |
|
|
|
|
---
|
|
|
|
#### **Spectrometer Pipelines**
|
|
| **Spectrometer Device** | **Kibana Log Stream** |
|
|
|------------------------|----------------------|
|
|
| SATOP31-PMOS132-2D_pmos | [SATOP31-PMOS132-2D_pmos](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SATOP31-PMOS132-2D_pmos%22%27%29) |
|
|
| SATOP21-PMOS127-2D_pmos | [SATOP21-PMOS127-2D_pmos](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SATOP21-PMOS127-2D_pmos%22%27%29) |
|
|
| SATOP11-PSAS079_psas | [SATOP11-PSAS079_psas](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SATOP11-PSAS079_psas%22%27%29) |
|
|
| SARFE10-PSSS059_psss | [SARFE10-PSSS059_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss%22%27%29) |
|
|
| SARFE10-PSSS059-LB_psss | [SARFE10-PSSS059-LB_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059-LB_psss%22%27%29) |
|
|
| SARFE10-PSSS059_psss_avg | [SARFE10-PSSS059_psss_avg](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss_avg%22%27%29) |
|
|
| SATOP31-PSRD132_pmos | [SATOP31-PSRD132_pmos](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language%3Akuery%2Cquery%3A%27%22message%22%3A%20%22name%3ASATOP31-PSRD132_pmos%22%27%29) |
|
|
| SATOP31-PSRD175_pmos | [SATOP31-PSRD175_pmos](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language%3Akuery%2Cquery%3A%27%22message%22%3A%20%22name%3ASATOP31-PSRD175_pmos%22%27%29) |
|
|
|
|
---
|
|
|
|
## Cam Server Git Repository
|
|
For pipeline functions and code, see the PSI Cam Server repository:
|
|
[Cam Server GitHub](https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py)
|