Load Balancer `llclb1` ====================== References ---------- - http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html - http://www.linuxvirtualserver.org/ Introduction ------------ This document describes the setup of `llclb1.psi.ch`, the Linux Login Cluster LoadBalancer for the ssh service on llc5 and llc6. The load balancing is implemented by means of the ipvsadm utility. The forwarding method is LVS_DR, direct routing (see below). Terms and Abbreviations ~~~~~~~~~~~~~~~~~~~~~~~ - LVS (Linux Virtual Server) The Linux Virtual Server is a scalable server built on a cluster of real servers, with the load balancer (director) running on the Linux operating system (LVS = director + realservers). The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single server. - IPVS, ip_vs The code that patches the linux kernel on the director. - Director (Load Balancer) The node that runs the ipvs code. Clients connect to the director. The director forwards packets to the realservers. The director is nothing but an IP router with special rules that make the LVS work. - Realservers (Servers) The hosts that have the services. The realservers handle the requests from the clients. - Client The host or user level process that connects to the VIP on the director. - Forwarding method Currently LVS-NAT, LVS-DR, LVS-Tun. The director is a router with somewhat different rules for forwarding packets than a normal router. The forwarding method determines how the director sends packets from the client to the realservers. - Scheduling The algorithm the director uses to select a realserver to service a new connection request from a client (ipvsadm and schedulers). - VIP Virtual IP, the IP on the director that the client connects to. - DIP Director IP, the IP on the director in the network. - RIP Realserver IP, the IP on the realserver. General setup of an LVS ~~~~~~~~~~~~~~~~~~~~~~~ The following figure illustrates a general layout of a network with an LVS:: _________ _________ _________ | | | | | | | CLIENT 1| | CLIENT 2| | CLIENT N| |_________| |_________| |_________| | | | -------------------------------- | ___|_____ | | | GATEWAY | |_________| | Linux Virtual Server | ............................................................... . | . . ___VIP____ . . | | . . | DIRECTOR | . . |__________| . . DIP . . | . . ------------------------------------------ . . | | | . . | | | . . _____RIP1_____ ____RIP2______ _____RIPN_____ . . | | | | | | . . | REALSERVER 1 | | REALSERVER 2 | | REALSERVER N | . . | | | | | | . . |______________| |______________| |______________| . . . ............................................................... One or Two NICs on the Director ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you have one NIC on the director, the VIP and the RIP are on the same physical network interface, where at least one virtual NIC was added, to hold one of the IP addresses. If you have two NICs you can assign one to the VIP network and the second to the RIP network. Here we have one NIC on the director. Requirements For LVS-DR ----------------------- For a reasonable LVS-DR setup the following is required: - One or more clients which are on networks different from the VIP and DIP network. - The realservers must be on the same network as the director (the realservers and director can arp each other). - One director host with at least one network interface. - One, better two static IPs for the director (one for the VIP, and one for the DIP). - Two realservers. - One static IP for each realserver. **Note to the Number of IPs on the director**: Depending on the service that is routed through the director, it might be useful to have two different IPs for the VIP and the DIP. E.g.: If you load balance an ssh service and you assign one IP to your NIC, which acts as VIP and DIP at the same time you can not reach your director anymore via ssh, because all ssh requests are routed through to one of the realservers. Installation Procedure ---------------------- Director Installation ~~~~~~~~~~~~~~~~~~~~~ An SL54 server installation was performed on llclb1. The director configuration shown below is implemented by means of puppet modules on puppet server psi-puppet1. Realserver Installation ~~~~~~~~~~~~~~~~~~~~~~~ Contemporary, the realservers llc5 and llc6 are SL46 Desktop Enhanced systems. One NIC LVS-DR Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The director processes only the client-to-server half of a connection in the virtual server via direct routing, and the response packets can follow separate network routes to the clients. This can greatly increase the scalability of virtual server. Compared to the virtual server via IP tunneling approach, this approach doesn't have tunneling overhead (In fact, this overhead is minimal in most situations), but requires that one of the load balancer's interfaces and the real server's interfaces must be in the same physical segment. The following figure illustrates the setup of the LVS-DR with llclb1 (director) having one NIC. The whole LVS is on the same network:: _________ _________ _________ | | | | | | | CLIENT 1| | CLIENT 2| | CLIENT N| |_________| |_________| |_________| | | | -------------------------------- | ___|_____ | | | GATEWAY | IP=129.129.190.1 |_________| | LVS (for service ssh) | ..................................................................... . | . . __________ | . . | llc | | VIP=129.129.190.54 (eth0:1) . . | | | . . | DIRECTOR |---| . . | | | . . | llclb1 | | DIP=129.129.190.53 (eth0) . . |__________| | . . | . . | . . ------------------------------------ . . | | . . | | . . RIP1=129.129.193.175 RIP2=129.129.193.176 . . ______________ ______________ . . | | | | . . | REALSERVER 1 | | REALSERVER 2 | . . | llc5 | | llc6 | . . |______________| |______________| . . . ..................................................................... Network Configuration ..................... Configure the LVS network according the scheme shown above. Static IPs for llclb1, llc, llc5 and llc6 have to be assigned. Director Configuration ...................... Configure the static device eth0 and restart the network. `/etc/sysconfig/network-scripts/ifcfg-eth0`:: DEVICE=eth0 BOOTPROTO=none HWADDR=00:14:5E:6B:13:3E ONBOOT=yes IPADDR=129.129.190.53 NETMASK=255.255.255.0 GATEWAY=129.129.193.1 TYPE=Ethernet USERCTL=no IPV6INIT=no PEERDNS=yes `/etc/sysconfig/network`:: NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=llclb1 Install the package ipvsadm:: # yum install ipvsadm Setup the LVS for ssh on the director using the following script: Note: It might be better to add it to the init scripts. `/etc/setup-LVS-DR-director.conf`:: #!/bin/bash #---------------mini-rc.lvs_dr-director------------------------ ### ### Network configuration ### VIP1=129.129.190.54 RIP1=129.129.190.175 RIP2=129.129.190.176 # Set ip_forward OFF for lvs-dr director (1 on, 0 off) # (there is no forwarding in the conventional sense for LVS-DR) cat /proc/sys/net/ipv4/ip_forward echo "0" >/proc/sys/net/ipv4/ip_forward # Director is not gw for realservers: leave icmp redirects on echo 'setting icmp redirects (1 on, 0 off) ' echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects cat /proc/sys/net/ipv4/conf/all/send_redirects echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects cat /proc/sys/net/ipv4/conf/default/send_redirects echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects cat /proc/sys/net/ipv4/conf/eth0/send_redirects # Add ethernet device and routing for VIP $VIP1 /sbin/ifconfig eth0:1 $VIP1 broadcast $VIP1 netmask 255.255.255.255 /sbin/route add -host $VIP1 dev eth0:1 # Listing ifconfig info for VIP $VIP1 /sbin/ifconfig eth0:1 # Check VIP $VIP1 is reachable from self (director) /bin/ping -c 1 $VIP1 # Listing routing info for VIP $VIP1 /bin/netstat -rn ### ### Setup_ipvsadm_table ### # Clear ipvsadm table /sbin/ipvsadm -C # Installing LVS services with ipvsadm # Add ssh to VIP with round robin scheduling /sbin/ipvsadm -A -t ${VIP1}:ssh -s rr # Forward ssh to realserver using direct routing with weight 1 /sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP1 -g -w 1 # Check realserver reachable from director ping -c 1 $RIP1 # Forward ssh to realserver using direct routing with weight 1 /sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP2 -g -w 1 # Check realserver reachable from director ping -c 1 $RIP2 # Set tcp timeout to 72 hours while leaving # tcpfin and udp timeouts unchanged. /sbin/ipvsadm --set 259200 0 0 # List timeout values /sbin/ipvsadm -L --timeout # Displaying ipvsadm settings /sbin/ipvsadm # Not installing a default gw for LVS_TYPE vs-dr ### ### Delete an LVS entry ### # # Example: remove/delete ssh forwarding to RIP2 # # /sbin/ipvsadm -d -t ${VIP1}:ssh -r RIP2 # # #---------------mini-rc.lvs_dr-director------------------------ Realserver Configuration ........................ The realserver shall send responses not to the VIP of the load balancer, rather to the client directly. This requires the iptables rule below. Settings of realserver 2 for instance: `/etc/sysconfig/network-scripts/ifcfg-eth0`:: DEVICE=eth0 BOOTPROTO=none HWADDR=00:06:5B:8C:3C:8E ONBOOT=yes TYPE=Ethernet DHCP_HOSTNAME=llc6 PEERDNS=yes IPADDR=129.129.190.176 NETMASK=255.255.255.0 GATEWAY=129.129.190.1 USERCTL=no IPV6INIT=no `/etc/sysconfig/network`:: NETWORKING=yes HOSTNAME=llc6 `/etc/sysconfig/lvs`:: # LVS configuration file for LLC and LLCX VIP=129.129.190.54 `/etc/init.d/lvs`:: #! /bin/sh # # chkconfig: 345 90 10 # description: Startscript to initialize this machine as an lvs real server. # Get network configuration . /etc/sysconfig/network # Get functions . /etc/rc.d/init.d/functions # Get VIP from the LVS configuration file . /etc/sysconfig/lvs # Check that networking is up if [ ${NETWORKING} = "no" ] ; then exit 0 fi RETVAL=0 # See how we were called. case "$1" in start) # Add rule echo "Starting load balancing mechanism with NAT iptables " /sbin/iptables -t nat -A PREROUTING -d $VIP -j REDIRECT ;; stop) # Delete rule echo "Stopping load balancing mechanism with NAT iptables " /sbin/iptables -t nat -D PREROUTING -d $VIP -j REDIRECT ;; *) echo "Usage: $0 {start|stop}" exit 1 ;; esac exit $RETVAL Update Procedure ---------------- Director Update ~~~~~~~~~~~~~~~ Login to llclb1 as root and run yum update:: # yum clean all # yum update Then reboot the director:: # reboot After rebooting no lvs rules are set by default:: [root@llclb1 ~]# ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn As soon as the realservers are updated and rebooted, too (see next section), run the lvs setup script to initialize the lvs rules for the ssh loadbalancing:: # sh /etc/setup-LVS-DR-director.conf 0 setting icmp redirects (1 on, 0 off) 1 1 1 SIOCADDRT: File exists eth0:1 Link encap:Ethernet HWaddr 00:14:5E:6B:13:3E inet addr:129.129.190.54 Bcast:129.129.190.54 Mask:255.255.255.255 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:169 Memory:d8300000-d8310000 PING 129.129.190.54 (129.129.190.54) 56(84) bytes of data. 64 bytes from 129.129.190.54: icmp_seq=1 ttl=64 time=0.053 ms --- 129.129.190.54 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 129.129.190.54 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 129.129.190.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 129.129.190.1 0.0.0.0 UG 0 0 0 eth0 PING 129.129.190.175 (129.129.190.175) 56(84) bytes of data. 64 bytes from 129.129.190.175: icmp_seq=1 ttl=64 time=2.13 ms --- 129.129.190.175 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 2.139/2.139/2.139/0.000 ms PING 129.129.190.176 (129.129.190.176) 56(84) bytes of data. 64 bytes from 129.129.190.176: icmp_seq=1 ttl=64 time=0.172 ms --- 129.129.190.176 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.172/0.172/0.172/0.000 ms Timeout (tcp tcpfin udp): 259200 120 300 IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP llc.psi.ch:ssh rr -> llc6.psi.ch:ssh Route 1 1 1 -> llc5.psi.ch:ssh Route 1 0 0 Realserver Update ~~~~~~~~~~~~~~~~~ The realservers should be updated automatically as they are standard SL desktop hosts. Login as root to the corresponding realserver, e.g. llc5, and verify that the update was performed correctly, if not fix it first. Then reboot the realserver. The iptables rule for the direct routing are initialized automatically by the init script /etc/init.d/lvs:: # reboot Eventually, test the ssh connection from any client to llc:: # [anyuser@anyhost] ssh llc