Installation `psi-puppet2`
==========================

References
----------

- https://wiki.intranet.psi.ch/AIT/PuppetServer2009

- http://reductivelabs.com/trac/puppet/wiki/PluginsInModules


Introduction
------------

This document describes the Installation of the new puppet server
psi-puppet2, which replaces the puppet server running on pxeserv01.

The psi-puppet2 server is an interim solution for the pxeserv01
replacement for the following reasons:

- The puppet service on pxeserv01 is very unstable.

- The performance on pxeserv01 is very bad, what makes work almost impossible.

- The psi-puppet1 server is not ready for production yet.

The new system will be setup as a 32 bit SL5 server with the latest
puppet server package and dependencies installed.

The puppet server configuration is the same as the one on pxeserv01.

**Note**: Don't confuse psi-puppet2 with psi-puppet1, which not only
is thought as a replacement of the puppet server on pxeserv01 but also
comes with a reimplementation of the whole puppet configuration,
whereas psi-puppet2 takes over the puppet configuration from
pxeserv01.


Description of the Basic Server Setup
-------------------------------------

- Hardware

  Dell Power Edge 1750

- Operating System

  SL51 32 bit Server

- Network configuration

  Static IP for Production Server: 129.129.190.160/24
  Hostname: psi-puppet2.psi.ch

- Required RPMS

  - puppet-server (http://download.fedora.redhat.com/pub/epel/5/i386/repoview/)
  - augeas-libs
  - facter
  - puppet (http://download.fedora.redhat.com/pub/epel/5/i386/repoview/)
  - ruby
  - ruby-augeas
  - ruby-libs
  - ruby-shadow
  - ruby-irb (required for reading help)
  - ruby-rdoc (required for reading help)


Procedure
---------

Setup The Puppet Server
~~~~~~~~~~~~~~~~~~~~~~~

Install The Puppet Server Software
..................................

Use the yum repositories `/etc/yum.repos.d/epeli386.repo` and
`/etc/yum.repos.d/epelx86_64.repo` to download and install the latest
puppet packages::

    [epeli386]
    name=epel i386
    baseurl=http://download.fedora.redhat.com/pub/epel/5/i386/
    enabled=0


Install the puppet-server package. Dependencies will be resolved::

    # yum --enablerepo=epeli386 install puppet-server

    ...
    Dependencies Resolved

    =============================================================================
     Package         Arch       Version      Repository    Size
    =============================================================================
    Installing:
     puppet-server       noarch     0.24.8-1.el5.1   epel           26 k
    Installing for dependencies:
     augeas-libs         i386       0.5.1-1.el5      epel          196 k
     facter          noarch     1.5.5-1.el5      epel           54 k
     puppet          noarch     0.24.8-1.el5.1   epel          542 k
     ruby            i386       1.8.5-5.el5_2.6  sl5update     279 k
     ruby-augeas         i386       0.2.0-1.el5      epel           17 k
     ruby-libs           i386       1.8.5-5.el5_2.6  sl5update     1.6 M
     ruby-shadow         i386       1.4.1-7.el5      epel          9.5 k

    Transaction Summary
    =============================================================================
    Install      8 Package(s)
    Update       0 Package(s)
    Remove       0 Package(s)
    ...


For later use, download the required RPMS and copy them to our yum
repository, see section `Put Puppet Related RPMS To Our Yum
Repository` below.


Configure The Puppet Server
...........................

Use the puppet server configuration of pxeserv01 on psi-puppet2.

The configuration files of the puppet server, directory
`/etc/puppet/`, are located on AFS, see section `Mount AFS Volumes`.

The log is on the local disk in `/var/log/puppet`.  To set the logfile
edit the line `PUPPETMASTER_OPTS` in
`/etc/rc.d/init.d/puppetmaster`. For testing also the debug option
`-d` is enabled::

    PUPPETMASTER_OPTS="-v -d -l /var/log/puppet/puppetmaster.log"

Enable puppetmaster in runlevels 3, 4 and 5::

    # chkconfig --levels 345 puppetmaster on


Mount AFS Volumes on Puppet Server
..................................

The puppet server configuration files as well as the puppet manifests
for clients are located on AFS.

AFS is already mounted as `/afs` in this default SL5 server installation::

    # mount
    ...
    AFS on /afs type afs (rw)

Now, we want to remount `/afs/psi.ch/service/linux/puppet/etc/puppet`
on `/etc/puppet`.  Therefor the mount option `bind` is used, which
facilitates to remount parts of already mounted filesystems on an
alternative location in the file hierarchy.

As shown below we do the remount in `/etc/rc.local`, which is executed
after all the other init scripts::

    #!/bin/sh

    touch /var/lock/subsys/local

    # Puppet
    mount -o bind /afs/psi.ch/service/linux/puppet/etc/puppet /etc/puppet

    # Restart Services depending on afs mounts
    /etc/init.d/puppetmaster restart

Before the `rc.local` script can be applied the proper AFS permissions
have to be set.

Check the AFS permissions::

    # [gasser_m@pc7377 ~]
    # fs la /afs/psi.ch/service/linux/

    Access list for /afs/psi.ch/service/linux/ is
    Normal rights:
      svc_linux:tools l
      svc_linux:puppet l
      svc_linux:readonly rl
      svc_linux:pxe l
      svc_linux rlidwka


Create a new AFS group for the puppet server::

    # pts creategr svc_linux:puppet_hosts -owner svc_linux
    group svc_linux:puppet_hosts has id -10851

Add the IP of psi-puppet2 to this group::

    # pts adduser 129.129.190.160 svc_linux:puppet_hosts

Set the AFS permissions::

    # fs sa /afs/psi.ch/service/linux/  svc_linux:puppet_hosts l
    # fs sa /afs/psi.ch/service/linux/puppet/ svc_linux:puppet_hosts rl
    # fs sa /afs/psi.ch/service/linux/puppet/etc svc_linux:puppet_hosts rl
    # fs sa /afs/psi.ch/service/linux/puppet/var svc_linux:puppet_hosts rl
    # fs sa /afs/psi.ch/service/linux/puppet/etc/puppet svc_linux:puppet_hosts rl

Recursively set the same permissions to all subdirectories underneath
`/afs/psi.ch/service/linux/puppet/etc/puppet`::

    # cd /afs/psi.ch/service/linux/puppet/etc/puppet
    # find -noleaf -type d -exec fs sa {} svc_linux:puppet_hosts rl \;

Then restart the AFS service::

    # service afs restart

Finally, run the `rc.local` script or just reboot psi-puppet2 to see
whether everything comes up.


Allow User Login Access To psi-puppet2
......................................

To enable remote login via ssh and execution of root commands via
sudo, the following files have to be edited.

- `/etc/passwd`

- `/etc/shadow`

- `/etc/security/ssh.allow`

- `/etc/group`

- `/etc/hosts.allow`

- `/etc/sudoers`


passwd::

    gasser_m:!:2374:840:Gasser Marc:/afs/psi.ch/user/g/gasser_m:/bin/bash
    kapeller:!:3804:9102:Rene Kapeller:/afs/psi.ch/user/k/kapeller:/bin/bash
    billich:!:3830:840:Heinrich Billich:/afs/psi.ch/user/b/billich:/bin/bash
    markushin:!:3883:840:Valeri Markushin:/afs/psi.ch/user/m/markushin:/bin/bash

shadow::

    gasser_m:NP:::::::
    kapeller:NP:::::::
    billich:NP:::::::
    markushin:NP:::::::

group::

    ait::840
    sls::9102

ssh.allow::

    kapeller
    billich
    markushin
    gasser_m

hosts.allow::

    sshd: ... pc7377 pc7377.psi.ch gfalc05 gfalc05.psi.ch pc4568 pc4568.psi.ch


Configuring Puppet Reporting
............................

There are a number of different report processors available on the
master. The default report, store, simply stores the report file on
the disk.

By default, each client is configured not to report back to the
master. It has to be enabled either by the report option in
`puppet.conf` or using `--report` on the command line.

`/etc/puppet/puppet.conf`::

    [puppetd]
        report = true 


Command line::

    # puppetd --report


Store Report Processor
,,,,,,,,,,,,,,,,,,,,,,

Enable the store reports by using the `reports` configuration option
in the puppemasterd section of the `puppet.conf` file on the master.

`/etc/puppet/puppet.conf`::

    [puppetmasterd]
        reports = store

The default reports directory is $vardir/reports.

Rrdgraph Report Processor
,,,,,,,,,,,,,,,,,,,,,,,,,

To enable the `rrdgraph` reports, `rrdtool` and `rrdtool-ruby`
packages have to be installed.

The packages are available from the the `psi-beta` repository, they
originate from the repository shown below.

`/etc/yum.repos.d/epeli386.repo`::

    [epeli386]
    name=epel i386
    baseurl=http://download.fedora.redhat.com/pub/epel/5/i386/
    enabled=0

    # yum install rrdtool rrdtool-ruby

Then, configure `puppet.conf` by adding the lines shown below in the
corresponding section.  Here store and rrdgraph are enabled.

`/etc/puppet/puppet.conf`::

    [puppetmasterd]
        reports = store,rrdgraph
        rrddir = $vardir/rrd
        rrdinterval = $runinterval
        rrdgraph = true

Install the Ganglia Monitor Daemon
..................................

Install `ganglia-gmond` and add the configuration file shown below::

    # yum install ganglia-gmond

Use the same `/etc/gmond.conf` as on psi-puppet1::

    # /etc/init.d/gmond start

For the ganglia server configuration ask Valeri Markushin.

See puppet at [[http://129.129.190.27/ganglia/][http://129.129.190.27/ganglia/]].

Setup The Puppet Client 
~~~~~~~~~~~~~~~~~~~~~~~~

Either install or update the puppet package on a client::

    # yum --enablerepo=psi-beta install puppet

    # yum --enablerepo=psi-beta update puppet

Test The Puppet Clients
~~~~~~~~~~~~~~~~~~~~~~~

To find out which versions of puppet servers and clients are
compatible to each other and in which order they should be updated,
the following tests are performed.


Test the Old Client With The New Server
.......................................

Set the new server `psi-puppet2.psi.ch` in `/etc/puppet/puppet.conf`,
remove the certificate and run puppetd::

    # [root@vsl5132de-ut]
    # cd /var/puppet/
    # rm -rf ssl/
    # puppetd --test

Ok.


Test the New Client With The New Server
.......................................

Update the puppet client.  Set the new server `psi-puppet2.psi.ch` in
`/etc/puppet/puppet.conf`, remove the certificate and run puppetd::

    # [root@vsl5132de-ut]
    # yum --enablerepo=psi-beta update puppet

    # cd /var/puppet/
    # rm -rf ssl/
    # puppetd --test

Ok.


Test the New Client With The Old Server
.......................................

Update the puppet client. Then set the old server `pxeserv01.psi.ch`
in `/etc/puppet/puppet.conf`, remove the certificate and run puppetd::

    # [root@vsl5132de-ut]
    # yum --enablerepo=psi-beta update puppet

    ...
    =============================================================================
     Package         Arch       Version      Repository    Size
    =============================================================================
    Updating:
     puppet          noarch     0.24.8-1.el5.1   psi-beta      542 k
    Installing for dependencies:
     augeas-libs         i386       0.5.1-1.el5      psi-beta      196 k
     ruby-augeas         i386       0.2.0-1.el5      psi-beta       17 k
     ruby-shadow         i386       1.4.1-7.el5      psi-beta      9.5 k
    Updating for dependencies:
     facter          noarch     1.5.5-1.el5      psi-beta       54 k

    Transaction Summary
    =============================================================================
    Install      3 Package(s)
    Update       2 Package(s)
    Remove       0 Package(s)
    ...

    # cd /var/puppet/
    # rm -rf ssl/
    # puppetd --test

Not ok. A lot of error messages appear. It seems the new client is not
compatible to the old server!

Conclusion
..........

First update the server, then the client.

So, when you have an old client addressing the new server for tests,
and finally you wanna change it back to the old server, this should
work by just modifying `/etc/puppet/puppet.conf` on the client.

However, once you updated the client you can not go back to the old
server, unless you downgrade the client --- with yum this means
removing the new client and reinstalling the old one.

Put Puppet Related RPMS To Our Yum Repository
---------------------------------------------

To have a consistent puppet installation on our hosts we put them to
our repository.

Use yumdowloader, package yum-utils, to download puppet related RPMS::

    # root@psi-puppet2
    # yum install yum-utils

    # yumdownloader --enablerepo=epeli386 puppet-server augeas-libs facter \
    #       puppet ruby-augeas ruby-shadow

    # yumdownloader --enablerepo=epelx86_64  augeas-libs ruby-augeas ruby-shadow


**Note**: There seem to be only i386 and noarch versions of the
 required RPMS.

Before they are linked into the latest `testing` and `current`
repositories they should be tested. So copy them to the `psi-beta`
repository first::

    # scp *rpm gasser_m@tux50:/afs/psi.ch/software/linux/dist/scientific/51/beta

    # gasser_m@tux50
    # cd /afs/psi.ch/software/linux/dist/scientific/51/beta
    # createrepo .

To test the puppet clients see section `Test The Puppet Clients`.

If the tests passed successfully copy them to the `others` repository
and create symbolic links to `testing` and `current` to make them
available::

    # [root@psi-puppet2]
    # scp *rpm gasser_m@tux50:/afs/psi.ch/software/linux/dist/scientific/51/others/all

    # gasser_m@tux50
    # cd /afs/psi.ch/software/linux/dist/scientific/51/others/all
    # createrepo .


Update Puppet
-------------

Keep the following order.

- Update the puppet server.

- Update the puppet client.


Update The Puppet Server
~~~~~~~~~~~~~~~~~~~~~~~~

At the time of writing the latest Puppet version 0.25.1 was only
available at http://tmz.fedorapeople.org/repo/puppet/epel/5/i386/.

Download the required packages to SL51 psi-beta repository first::

    # wget http://tmz.fedorapeople.org/repo/puppet/epel/5/i386/puppet-server-0.25.1-0.3.el5.noarch.rpm
    # wget http://tmz.fedorapeople.org/repo/puppet/epel/5/i386/puppet-0.25.1-0.3.el5.noarch.rpm

Then login to a test server and stop the puppetmaster daemon::

    # /etc/init.d/puppetmaster stop

Because the /etc/puppet/ is on AFS, root has no write permissions.
So, first umount the etc/puppet from AFS, then run `yum update` using
the psi-beta repository, and remount etc/puppet::

    # umount /etc/puppet
    # yum --enablerepo=psi-beta update puppet-server
    # mount -o bind /afs/psi.ch/service/linux/puppet/etc/puppet /etc/puppet

Set the following options in /etc/init.d/puppetmaster::

    PUPPETMASTER_OPTS="-v -d -l /var/log/puppet/puppetmaster.log"

Finally, restart the service and test it with a client::

    # /etc/init.d/puppetmaster start

Login to a client and run puppetd::

    # puppetd --test

Update The Puppet Client
~~~~~~~~~~~~~~~~~~~~~~~~

Basically, the 0.24.x clients should be compatible to the 0.25.x
server. Though, there are some changes between the versions, that
might cause troubles. Thus, the puppet clients should be updated, too.

On a SL51 client run::

    # yum --enablerepo=psi-beta update puppet

On a SL54 client run::

    # yum update puppet

As soon as the client is updated, its `puppet.conf` needs some
modifications, because of the "factsync" option which is deprecated
and replaced by "pluginsync" in the 0.25.x versions.

`/etc/puppet/puppet.conf` on a 0.25.x client: "factsync" is replaced
by "pluginsync", factpath is set (not clear whether the latter is
necessary)::

    [main]
        vardir = /var/puppet
        logdir = /var/log/puppet
        rundir = /var/run/puppet
        ssldir = $vardir/ssl
        pluginsync = true
        factpath = $vardir/lib/facter
        environment = DesktopSL5Unstable

    [puppetd]
        report = true
        classfile = $vardir/classes.txt
        localconfig = $vardir/localconfig
        server = psi-puppet1.psi.ch


Further the directory structure on the server for placing facts
changed. The new structure if using modules and environments is
illustrated below taking the environment "DesktopSL5Unstable" as an
example:

The modulepath for DesktopSL5Unstable is::

    modulepath = /var/puppet/environments/DesktopSL5Unstable/modules

A stub module called "custom" has to be created in the "$modulepath"
subdirectory to keep the files::

    $modulepath/custom/
            `-- lib/
                |-- facter/
                |   `-- sysconfig_psi_desktop.rb
                `-- puppet/
                |-- provider/
                `-- type/

This subdirectory tree under custom is implicitly searched by the
puppet server.

**Notes**:

If you have both, the new and the old variant to keep facts, option
"pluginsync" enabled, only the new script location will be considered.

If you run only the old variant to keep facts::

    $modulepath/facts/files/somescript.rb

with option "factsync" enabled, they will be loaded giving out a
warning "... use pluginsync instead of factsync ..."

For more information see the
http://reductivelabs.com/trac/puppet/wiki/PluginsInModules.
