Puppet Trouble-shooting in TWiki
================================

Introduction
------------

This is the location were to post puppet problems and solutions at AIT
and GFA.

Please use the following format according the first entry below:

- HEADING2: Error (HOSTNAME): "BEGIN OF ERROR MESSAGE ..."
  *(The HOSTNAME in parentheses is optional, if the problem might affect any host.)*

- HEADING3: Context, a short description in what situation the error appears.

- HEADING3: Solution, if known, provide a possible solution to fix the problem. 


Puppet Server Errors
--------------------

Error (pxeserv01): "Starting puppetmaster: Could not prepare for execution ..."
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Context
.......

The puppetmaster daemon on pxeserv01 received an unattended SIGTERM
and stopped.  Maybe there is a self signed certificate which expired
or something.  Anyway, restarting failed showing the following error::

    # [root@pxeserv01 puppet]# /etc/init.d/puppetmaster start
    Starting puppetmaster: Could not prepare for execution: Retrieved certificate does not \
    match private key; please remove certificate from server and regenerate it with the current key


Solution
........

Create a new certificate for the server.

**IMPORTANT NOTE**: Recreating a new certificate for the server showed
that the clients which still have the old certificate of the server
can not connect anymore to it.  Thus you need to be sure you have a
way to update the certificate of the server on the clients before
applying this.  Eventually, a better solution might be to just sign
the old server certificate with `puppetca`.  It was not tested whether
this works, too.

Remove /var/puppet/ssl/ on the server::
	
	# rm -rf /var/puppet/ssl/


Then try to start puppetmaster, what failed again, but recreated the
ssl subdirectory and its content::

	# /etc/init.d/puppetmaster start


Eventually, sign the certificate for pxeserv01 itself::

	# puppetca --sign pxeserv01.psi.ch
	# /etc/init.d/puppetmaster start


Via our management console the certificates on the clients had to be
deleted, in order they can retrieve the new server certificate::

	# for i in host1 host2 ... 
	# do
	#        ssh -o ConnectTimeout=1 root@$i rm -rf /var/puppet/ssl/
	# done


Puppet Client Errors
--------------------

Error: "Could not request certificate: ..."
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Context
.......

Running puppetd on any client might fail showing the following error::

	# puppetd --test
	err: Could not request certificate: Retrieved certificate does not match private key;
	please remove certificate from server and regenerate it with the current key
	Exiting; failed to retrieve certificate and watiforcert is disabled


Solution
........

The problem is that for what reason ever the client certificate is not
valid, i.e. it does not match its copy stored on the puppet server.

Usually this happens when a host is reinstalled keeping its original
hostname.  During the initial puppetd run, the client requests a new
certificate from the server, which denies, because there is already a
certificate on the server for that hostname.

To fix this you have to delete the old certificate on client and
server.

On the server::

	# puppetca --clean HOSTNAME.psi.ch
	HOSTNAME.psi.ch
	notice: Removing file Puppet::SSL::Certificate HOSTNAME.psi.ch at 
	'/var/puppet/ssl/ca/signed/mpc1273.psi.ch.pem'

If this notice does not appear, the certificate was not deleted.

On the client::

	# rm -rf /var/puppet/ssl/

Then, run puppetd again::

	# puppetd --test


Error: "notice: Run of Puppet configuration client already in progress; skipping ..."
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Context
.......

Running puppetd on any client might fail showing the following error,
even `ps ax | grep puppetd` does not show a running puppet process::

	# puppetd --test
	notice: Run of Puppet configuration client already in progress; skipping


Solution
........

There is still an old lock file `/var/puppet/state/puppetdlock`.
Remove the lock file and run puppetd again::

	# rm /var/puppet/state/puppetdlock
