Load Balancer `llclb1`
======================

References
----------

- http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html

- http://www.linuxvirtualserver.org/


Introduction
------------

This document describes the setup of `llclb1.psi.ch`, the Linux Login
Cluster LoadBalancer for the ssh service on llc5 and llc6.

The load balancing is implemented by means of the ipvsadm utility.

The forwarding method is LVS_DR, direct routing (see below).

Terms and Abbreviations
~~~~~~~~~~~~~~~~~~~~~~~

- LVS (Linux Virtual Server) 

  The Linux Virtual Server is a scalable server built on a cluster of
  real servers, with the load balancer (director) running on the Linux
  operating system (LVS = director + realservers).  The architecture
  of the server cluster is fully transparent to end users, and the
  users interact as if it were a single server.

- IPVS, ip_vs

  The code that patches the linux kernel on the director.

- Director (Load Balancer)

  The node that runs the ipvs code. Clients connect to the
  director. The director forwards packets to the realservers. The
  director is nothing but an IP router with special rules that make
  the LVS work.

- Realservers (Servers)

  The hosts that have the services. The realservers handle the
  requests from the clients.

- Client

  The host or user level process that connects to the VIP on the
  director.

- Forwarding method

  Currently LVS-NAT, LVS-DR, LVS-Tun. The director is a router with
  somewhat different rules for forwarding packets than a normal
  router.  The forwarding method determines how the director sends
  packets from the client to the realservers.

- Scheduling

  The algorithm the director uses to select a realserver to service a
  new connection request from a client (ipvsadm and schedulers).

- VIP

  Virtual IP, the IP on the director that the client connects to.

- DIP

  Director IP, the IP on the director in the network.

- RIP

  Realserver IP, the IP on the realserver.


General setup of an LVS
~~~~~~~~~~~~~~~~~~~~~~~

The following figure illustrates a general layout of a network with an
LVS::

                 _________        _________       _________
                |         |      |         |     |         |
                | CLIENT 1|      | CLIENT 2|     | CLIENT N|
                |_________|      |_________|     |_________|
                    |               |              |
                    --------------------------------
                                    |
                                 ___|_____
                                |         |
                                | GATEWAY |
                                |_________|
                                    |
          Linux Virtual Server      |
       ...............................................................
       .                            |                                .
       .                        ___VIP____                           .
       .                       |          |                          .
       .                       | DIRECTOR |                          .
       .                       |__________|                          .
       .                           DIP                               .
       .                            |                                .
       .         ------------------------------------------          .
       .         |                  |                     |          .
       .         |                  |                     |          .
       .   _____RIP1_____      ____RIP2______      _____RIPN_____    .
       .  |              |    |              |    |              |   .
       .  | REALSERVER 1 |    | REALSERVER 2 |    | REALSERVER N |   .
       .  |              |    |              |    |              |   .
       .  |______________|    |______________|    |______________|   .
       .                                                             .
       ...............................................................   



One or Two NICs on the Director
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you have one NIC on the director, the VIP and the RIP are on the
same physical network interface, where at least one virtual NIC was
added, to hold one of the IP addresses.

If you have two NICs you can assign one to the VIP network and the
second to the RIP network.

Here we have one NIC on the director.


Requirements For LVS-DR
-----------------------

For a reasonable LVS-DR setup the following is required:

- One or more clients which are on networks different from the VIP and
  DIP network.

- The realservers must be on the same network as the director (the
  realservers and director can arp each other).

- One director host with at least one network interface.

- One, better two static IPs for the director (one for the VIP, and
  one for the DIP).

- Two realservers.

- One static IP for each realserver.


**Note to the Number of IPs on the director**:

Depending on the service that is routed through the director, it might
be useful to have two different IPs for the VIP and the DIP.

E.g.: If you load balance an ssh service and you assign one IP to your
NIC, which acts as VIP and DIP at the same time you can not reach your
director anymore via ssh, because all ssh requests are routed through
to one of the realservers.

Installation Procedure
----------------------

Director Installation
~~~~~~~~~~~~~~~~~~~~~

An SL54 server installation was performed on llclb1.

The director configuration shown below is implemented by means of
puppet modules on puppet server psi-puppet1.


Realserver Installation
~~~~~~~~~~~~~~~~~~~~~~~

Contemporary, the realservers llc5 and llc6 are SL46 Desktop Enhanced
systems.


One NIC LVS-DR Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The director processes only the client-to-server half of a connection
in the virtual server via direct routing, and the response packets can
follow separate network routes to the clients. This can greatly
increase the scalability of virtual server.

Compared to the virtual server via IP tunneling approach, this
approach doesn't have tunneling overhead (In fact, this overhead is
minimal in most situations), but requires that one of the load
balancer's interfaces and the real server's interfaces must be in the
same physical segment.

The following figure illustrates the setup of the LVS-DR with llclb1
(director) having one NIC. The whole LVS is on the same network::


                 _________        _________       _________
                |         |      |         |     |         |
                | CLIENT 1|      | CLIENT 2|     | CLIENT N|
                |_________|      |_________|     |_________|
                    |               |              |
                    --------------------------------
                                    |
                                 ___|_____
                                |         |
                                | GATEWAY | IP=129.129.190.1
                                |_________|
                                    |
          LVS (for service ssh)     |
       .....................................................................
       .                            |                                      .
       .              __________    |                                      .
       .             |  llc     |   |   VIP=129.129.190.54 (eth0:1)        .
       .             |          |   |                                      .
       .             | DIRECTOR |---|                                      .
       .             |          |   |                                      .
       .             |  llclb1  |   |   DIP=129.129.190.53 (eth0)          .
       .             |__________|   |                                      .
       .                            |                                      .
       .                            |                                      .
       .         ------------------------------------                      .
       .         |                                  |                      .
       .         |                                  |                      .
       .  RIP1=129.129.193.175                  RIP2=129.129.193.176       .
       .   ______________                        ______________            .
       .  |              |                      |              |           .
       .  | REALSERVER 1 |                      | REALSERVER 2 |           .
       .  |     llc5     |                      |     llc6     |           .
       .  |______________|                      |______________|           .
       .                                                                   .   
       .....................................................................   


Network Configuration
.....................

Configure the LVS network according the scheme shown above.  Static
IPs for llclb1, llc, llc5 and llc6 have to be assigned.


Director Configuration
......................

Configure the static device eth0 and restart the network.

`/etc/sysconfig/network-scripts/ifcfg-eth0`::

    DEVICE=eth0
    BOOTPROTO=none
    HWADDR=00:14:5E:6B:13:3E
    ONBOOT=yes
    IPADDR=129.129.190.53
    NETMASK=255.255.255.0
    GATEWAY=129.129.193.1
    TYPE=Ethernet
    USERCTL=no
    IPV6INIT=no
    PEERDNS=yes


`/etc/sysconfig/network`::

    NETWORKING=yes
    NETWORKING_IPV6=no
    HOSTNAME=llclb1



Install the package ipvsadm::

    # yum install ipvsadm

Setup the LVS for ssh on the director using the following script:
Note: It might be better to add it to the init scripts.

`/etc/setup-LVS-DR-director.conf`::

    #!/bin/bash
    #---------------mini-rc.lvs_dr-director------------------------

    ###
    ### Network configuration
    ###

    VIP1=129.129.190.54

    RIP1=129.129.190.175
    RIP2=129.129.190.176


    # Set ip_forward OFF for lvs-dr director (1 on, 0 off)
    # (there is no forwarding in the conventional sense for LVS-DR)
    cat       /proc/sys/net/ipv4/ip_forward
    echo "0" >/proc/sys/net/ipv4/ip_forward

    # Director is not gw for realservers: leave icmp redirects on
    echo 'setting icmp redirects (1 on, 0 off) '
    echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects
    cat       /proc/sys/net/ipv4/conf/all/send_redirects
    echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects
    cat       /proc/sys/net/ipv4/conf/default/send_redirects
    echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects
    cat       /proc/sys/net/ipv4/conf/eth0/send_redirects

    # Add ethernet device and routing for VIP $VIP1
    /sbin/ifconfig eth0:1 $VIP1 broadcast $VIP1 netmask 255.255.255.255
    /sbin/route add -host $VIP1 dev eth0:1
    # Listing ifconfig info for VIP $VIP1
    /sbin/ifconfig eth0:1

    # Check VIP $VIP1 is reachable from self (director)
    /bin/ping -c 1 $VIP1
    # Listing routing info for VIP $VIP1
    /bin/netstat -rn

    ###
    ### Setup_ipvsadm_table
    ###

    # Clear ipvsadm table
    /sbin/ipvsadm -C

    # Installing LVS services with ipvsadm
    # Add ssh to VIP with round robin scheduling
    /sbin/ipvsadm -A -t ${VIP1}:ssh -s rr

    # Forward ssh to realserver using direct routing with weight 1
    /sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP1 -g -w 1
    # Check realserver reachable from director
    ping -c 1 $RIP1

    # Forward ssh to realserver using direct routing with weight 1
    /sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP2 -g -w 1
    # Check realserver reachable from director
    ping -c 1 $RIP2

    # Set tcp timeout to 72 hours while leaving
    # tcpfin and udp timeouts unchanged.
    /sbin/ipvsadm --set 259200 0 0

    # List timeout values
    /sbin/ipvsadm -L --timeout

    # Displaying ipvsadm settings
    /sbin/ipvsadm

    # Not installing a default gw for LVS_TYPE vs-dr

    ###
    ### Delete an LVS entry
    ###
    #
    # Example: remove/delete ssh forwarding to RIP2
    #
    #       /sbin/ipvsadm -d -t ${VIP1}:ssh -r RIP2
    #
    #

    #---------------mini-rc.lvs_dr-director------------------------


Realserver Configuration
........................

The realserver shall send responses not to the VIP of the load
balancer, rather to the client directly. This requires the iptables
rule below.

Settings of realserver 2 for instance:

`/etc/sysconfig/network-scripts/ifcfg-eth0`::

    DEVICE=eth0
    BOOTPROTO=none
    HWADDR=00:06:5B:8C:3C:8E
    ONBOOT=yes
    TYPE=Ethernet
    DHCP_HOSTNAME=llc6
    PEERDNS=yes
    IPADDR=129.129.190.176
    NETMASK=255.255.255.0
    GATEWAY=129.129.190.1
    USERCTL=no
    IPV6INIT=no


`/etc/sysconfig/network`::

    NETWORKING=yes
    HOSTNAME=llc6


`/etc/sysconfig/lvs`::

    # LVS configuration file for LLC and LLCX
    VIP=129.129.190.54


`/etc/init.d/lvs`::

    #! /bin/sh
    #
    # chkconfig: 345 90 10
    # description: Startscript to initialize this machine as an lvs real server.

    # Get network configuration
    . /etc/sysconfig/network
    # Get functions
    . /etc/rc.d/init.d/functions
    # Get VIP from the LVS configuration file
    . /etc/sysconfig/lvs

    # Check that networking is up
    if [ ${NETWORKING} = "no" ] ; then
            exit 0
    fi

    RETVAL=0

    # See how we were called.
    case "$1" in
      start)
            # Add rule
            echo "Starting load balancing mechanism with NAT iptables "
            /sbin/iptables -t nat -A PREROUTING -d $VIP  -j REDIRECT
            ;;
      stop)
            # Delete rule
            echo "Stopping load balancing mechanism with NAT iptables "
            /sbin/iptables -t nat -D PREROUTING -d $VIP  -j REDIRECT
            ;;
      *)
            echo "Usage: $0 {start|stop}"
            exit 1
            ;;
    esac

    exit $RETVAL


Update Procedure
----------------

Director Update
~~~~~~~~~~~~~~~

Login to llclb1 as root and run yum update::

    # yum clean all
    # yum update

Then reboot the director::

    # reboot

After rebooting no lvs rules are set by default::

    [root@llclb1 ~]# ipvsadm -L
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

As soon as the realservers are updated and rebooted, too (see next
section), run the lvs setup script to initialize the lvs rules for the
ssh loadbalancing::

    # sh /etc/setup-LVS-DR-director.conf


    0
    setting icmp redirects (1 on, 0 off)
    1
    1
    1
    SIOCADDRT: File exists
    eth0:1    Link encap:Ethernet  HWaddr 00:14:5E:6B:13:3E
              inet addr:129.129.190.54  Bcast:129.129.190.54  Mask:255.255.255.255
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              Interrupt:169 Memory:d8300000-d8310000

    PING 129.129.190.54 (129.129.190.54) 56(84) bytes of data.
    64 bytes from 129.129.190.54: icmp_seq=1 ttl=64 time=0.053 ms

    --- 129.129.190.54 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    129.129.190.54  0.0.0.0         255.255.255.255 UH        0 0          0 eth0
    129.129.190.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    0.0.0.0         129.129.190.1   0.0.0.0         UG        0 0          0 eth0
    PING 129.129.190.175 (129.129.190.175) 56(84) bytes of data.
    64 bytes from 129.129.190.175: icmp_seq=1 ttl=64 time=2.13 ms

    --- 129.129.190.175 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 2.139/2.139/2.139/0.000 ms
    PING 129.129.190.176 (129.129.190.176) 56(84) bytes of data.
    64 bytes from 129.129.190.176: icmp_seq=1 ttl=64 time=0.172 ms

    --- 129.129.190.176 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.172/0.172/0.172/0.000 ms
    Timeout (tcp tcpfin udp): 259200 120 300
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  llc.psi.ch:ssh rr
      -> llc6.psi.ch:ssh              Route   1      1          1
      -> llc5.psi.ch:ssh              Route   1      0          0


Realserver Update
~~~~~~~~~~~~~~~~~

The realservers should be updated automatically as they are standard
SL desktop hosts.  Login as root to the corresponding realserver,
e.g. llc5, and verify that the update was performed correctly, if not
fix it first.

Then reboot the realserver. The iptables rule for the direct routing
are initialized automatically by the init script /etc/init.d/lvs::

    # reboot

Eventually, test the ssh connection from any client to llc::

    # [anyuser@anyhost] ssh llc

