diff --git a/Pipeline-server.md b/Pipeline-server.md index 536f4fb..548bb09 100644 --- a/Pipeline-server.md +++ b/Pipeline-server.md @@ -1,43 +1,53 @@ ## Pipeline server debugging -Logs from the pipeline server are now available on kibana. +Logs from the pipeline server are now available on Kibana. Example query strings are: -- "message": “name:" (for pipeline logs) -- "message": "camera:" (for camera logs) -- "host.hostname":"sf-daqsync-XX.psi.ch" (for server logs) -- “systemd.unit":”XXX.service" (for service logs) +- `"message": "name:"` (for pipeline logs) +- `"message": "camera:"` (for camera logs) +- `"host.hostname":"sf-daqsync-XX.psi.ch"` (for server logs) +- `"systemd.unit":"XXX.service"` (for service logs) -[https://kibana.psi.ch/s/camserver/app/logs](https://kibana.psi.ch/s/camserver/app/logs) +[Kibana Log Stream](https://kibana.psi.ch/s/camserver/app/logs) To check the metrics of the cluster (CPU, Memory, network): -[https://kibana.psi.ch/s/camserver/app/metrics/inventory](https://kibana.psi.ch/s/camserver/app/metrics/inventory) - +[Kibana Metrics](https://kibana.psi.ch/s/camserver/app/metrics/inventory) + Or a specific server directly: -[https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch](https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch) - +[Kibana Server Metrics](https://kibana.psi.ch/s/camserver/app/metrics/detail/host/sf-daqsync-03.psi.ch) + To monitor service availability and automatic alarm reporting (by email): -[https://kibana.psi.ch/s/camserver/app/uptime](https://kibana.psi.ch/s/camserver/app/uptime) -[https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open](https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open) - +[Service Uptime](https://kibana.psi.ch/s/camserver/app/uptime) +[Open Alerts](https://kibana.psi.ch/s/camserver/app/observability/alerts?workflowStatus=open) + Dashboards to display individual pipeline metrics. -E.g. this one for PSSS: -[https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))](https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))) +E.g., this one for PSSS: +[PSSS Dashboard](https://kibana.psi.ch/s/camserver/app/dashboards#/view/058f4b00-87a3-11ee-8a7f-6175eba7c3d6?_g=%28filters:!%28%29,refreshInterval:%28pause:!t,value:0%29,time:%28from:now-1h,to:now%29%29) -### Dashboards to stream individual pipeline metrics +--- -| **PBPS Device** | **Kibana Log Stream** | -|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| SARFE10-PBPS053 | [SARFE10-PBPS053_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PBPS053_proc%22%27%29) | -| SAROP11-PBPS110 | [SAROP11-PBPS110_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS110_proc%22%27%29) | -| SAROP11-PBPS122 | [SAROP11-PBPS122_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS122_proc%22%27%29) | -| SAROP21-PBPS103 | [SAROP21-PBPS103_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS103_proc%22%27%29) | -| SAROP21-PBPS133 | [SAROP21-PBPS133_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP21-PBPS133_proc%22%27%29) | -| SARES21-PBPS141 | [SARES21-PBPS141_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARES21-PBPS141_proc%22%27%29) | -| SAROP31-PBPS113 | [SAROP31-PBPS113_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP31-PBPS113_proc%22%27%29) | -| SAROP31-PBPS149 | [SAROP31-PBPS149_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=%28flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n%29&logPosition=%28end:now,position:!n,start:now-1d,streamLive:!t%29&logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP31-PBPS149_proc%22%27%29) | +### **Dashboards to stream individual pipeline metrics** -- [SARFE10-PSSS059_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n)&logPosition=(end:now,position:(tiebreaker:545,time:1734080766159),start:now-1d,streamLive:!t)&logFilter=(language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss%22%27)) -**Note the stream may take around 30 seconds to load** +#### **PBPS Pipelines** +| **PBPS Device** | **Kibana Log Stream** | +|----------------|----------------------| +| SARFE10-PBPS053 | [SARFE10-PBPS053_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PBPS053_proc%22%27%29) | +| SAROP11-PBPS110 | [SAROP11-PBPS110_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS110_proc%22%27%29) | +| SAROP11-PBPS122 | [SAROP11-PBPS122_proc](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SAROP11-PBPS122_proc%22%27%29) | -## Cam Server git -[https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py](https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py) +--- + +#### **Spectrometer Pipelines** +| **Spectrometer Device** | **Kibana Log Stream** | +|------------------------|----------------------| +| SATOP31-PMOS132-2D_pmos | [SATOP31-PMOS132-2D_pmos](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SATOP31-PMOS132-2D_pmos%22%27%29) | +| SATOP21-PMOS127-2D_pmos | [SATOP21-PMOS127-2D_pmos](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SATOP21-PMOS127-2D_pmos%22%27%29) | +| SATOP11-PSAS079_psas | [SATOP11-PSAS079_psas](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SATOP11-PSAS079_psas%22%27%29) | +| SARFE10-PSSS059_psss | [SARFE10-PSSS059_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss%22%27%29) | +| SARFE10-PSSS059-LB_psss | [SARFE10-PSSS059-LB_psss](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059-LB_psss%22%27%29) | +| SARFE10-PSSS059_psss_avg | [SARFE10-PSSS059_psss_avg](https://kibana.psi.ch/s/camserver/app/logs/stream?logFilter=%28language:kuery,query:%27%22message%22:%20%22name:SARFE10-PSSS059_psss_avg%22%27%29) | + +--- + +## Cam Server Git Repository +For pipeline functions and code, see the PSI Cam Server repository: +[Cam Server GitHub](https://github.com/paulscherrerinstitute/cam_server/blob/master/cam_server/pipeline/data_processing/functions.py)