sf_databuffer/operation-tools

Overview

This document provides instructions on how to install and to operate the data acquisition (daq) environment.

The DataBuffer currently consists of 13 nodes located in the server room next to the control room in WBGB. The nodes are located in the SwissFEL network and are named sf-daqbuf-21 up to sf-daqbuf-33.

Each node of the cluster currently runs 2 independent services:

• daq-dispatcher-node.service - used to dispatch and record data (filestorage-cluster)
• daq-query-node.service - used to query data

The machines are installed and managed via ansible scripts and playbooks. To be able to run these scripts password-less ssh needs to be setup from the machine running these scripts.

Prerequisites

Note: This shouldn't be needed if working on a standard GFA machine like sflca!

To be able to execute the steps and commands outlined in this document following prerequisites need to be met:

• Ansible needs to be installed on your machine

• Password-less ssh needs to be setup from your machine

  • Use ssh-copy-id to copy your public key to the servers

    ssh-copy-id your-user@sf-daqbuf-xx.psi.ch
    

  • Have a tunneling directive in your ~/.ssh/config like this:

    Host sfbastion
      Hostname sf-gw.psi.ch
      ControlMaster auto
      ControlPath ~/.ssh/mux-%r@%h:%p
      ControlPersist 8h
    
    Host sf-data-api* sf-dispatcher-api* sf-daq*
      Hostname %h
      ProxyJump username@sfbastion
    

• Sudo rights are needed on databuffer/... servers

• Clone the sf_databuffer repository and switch to the operation-tools folder

To verify that password-less login and sudo rights are in place on all servers of the inventory run:

ansible-playbook check-access.yml

Checks

Checks on the cluster can be performed via ansible ad-hoc commands:

ansible databuffer -m shell -a 'uptime'

# Maintenance

## Restart Procedures
Following are restart procedures meant for emergency interventions on the SwissFEL DAQ system 

__Before issuing a restart check the which endstations are attended via one of the following operations panel and check whether a restart is ok with all attended endstations!__

bash -c 'caqtdm -noMsg -stylefile sfop.qss -macro chargelim=15 S_OP_overview.ui' bash -c 'caqtdm -noMsg -stylefile sfop.qss -macro chargelim=15 S_OP_overview_for_photonics.ui' bash -c 'caqtdm -noMsg -stylefile sfop.qss S_OP_Messages_all_stations_small.ui'

Inform the sfOperations@psi.ch mailing list before the restart!

### Restart DataBuffer

This is the procedure to follow to restart the DataBuffer in an emergency.

After checking whether the restart is really necessary do this:

- login to sf-lca.psi.ch (_sflca is cluster in the machine network !!!!_)
- clone the databuffer repository (if you haven't yet), change to the operation-tools directory and/or pull the latest changes

```bash
git clone https://gitea.psi.ch/archiver_config/sf_databuffer.git
cd sf_databuffer/operation-tools
# and/or
git pull

• call the restart script

ansible-playbook restart-all.yml

• After waiting ~5min restart the recording again:

cd ..
./bufferutils upload

The required time to wait until restarting the recording somehow depends on the amount of cleanup work the DataBuffer has to do after a crash/screw up. The more bad the crash/screw up was wait for a longer time. In most cases 5 min should be enough.