Puppetmaster At PSI
===================

Procedure
---------

Manual Check Whether Puppetmaster Is Running
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Contemporary, the `puppetmasterd` process often crashes and automatic
restart fails. Thus it has to be started manually again. Therefore login
to `pxeserv01`, check whether the process is still running and if not,
restart `puppetmasterd`.

Check whether the `puppetmasterd` is running::

    [root@pxeserv01 ~]# ps ax | grep puppet
      549 pts/0    S+     0:00 grep puppet
     6641 ?    S      7:53 /bin/bash ./test-and-restart-puppetmaster.sh

It's not running.

Check the logfile. The logfile is situated on AFS::

    # tail /afs/psi.ch/service/linux/puppet/var/log/puppetmaster.log
    ...
    Mon Jan 26 09:22:27 +0100 2009 Puppet (notice): Compiled configuration for slsnedi1.psi.ch in 0.09 seconds
    Mon Jan 26 09:24:55 +0100 2009 Puppet (notice): Caught TERM; shutting down
    Mon Jan 26 09:24:55 +0100 2009 Puppet (notice): Shutting down
    Mon Jan 26 09:24:55 +0100 2009 Puppet (err): Could not remove PID file /var/run/puppet/puppetmasterd.pid
    Mon Jan 26 09:25:01 +0100 2009 Puppet (err): Could not create PID file: /var/run/puppet/puppetmasterd.pid


In this case remove `/var/run/puppet/puppetmasterd.pid` and restart the server::

    # rm /var/run/puppet/puppetmasterd.pid
    # /etc/init.d/puppetmaster restart


Check again whether the `puppetmasterd` is running::

    # [root@pxeserv01 ~]# ps ax | grep puppet
      549 pts/0    S+     0:00 grep puppet
     6641 ?    S      7:53 /bin/bash ./test-and-restart-puppetmaster.sh
    31599 ?    Ssl    0:03 /usr/bin/ruby /usr/sbin/puppetmasterd \
                    --logdest=/afs/psi.ch/service/linux/puppet/var/log/puppetmaster.log


Finally, test it on a client::

    # [root@pc7377 ~]# psi-puppet
    ...
    puppetd was running successfully


Automatic Check Whether Puppetmaster Is Running
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On `pxeserv01` the script `/root/test-and-restart-puppetmaster.sh`
continuously observes the `puppetmasterd` and should restart it when
it's hanging.

The script is started in the following way::

    # [root@pxeserv01 etc]# nohup /root/test-and-restart-puppetmaster.sh &


The output is written to:

- `$HOME/nohup.out`

  The logfile.

- `/dev/shm/puppetwatch.1`

- `/dev/shm/puppetwatch.2`