17 KiB
Load Balancer llclb1
References
- http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html
- http://www.linuxvirtualserver.org/
Introduction
This document describes the setup of llclb1.psi.ch, the Linux Login Cluster LoadBalancer for the ssh service on llc5 and llc6.
The load balancing is implemented by means of the ipvsadm utility.
The forwarding method is LVS_DR, direct routing (see below).
Terms and Abbreviations
LVS (Linux Virtual Server)
The Linux Virtual Server is a scalable server built on a cluster of real servers, with the load balancer (director) running on the Linux operating system (LVS = director + realservers). The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single server.
IPVS, ip_vs
The code that patches the linux kernel on the director.
Director (Load Balancer)
The node that runs the ipvs code. Clients connect to the director. The director forwards packets to the realservers. The director is nothing but an IP router with special rules that make the LVS work.
Realservers (Servers)
The hosts that have the services. The realservers handle the requests from the clients.
Client
The host or user level process that connects to the VIP on the director.
Forwarding method
Currently LVS-NAT, LVS-DR, LVS-Tun. The director is a router with somewhat different rules for forwarding packets than a normal router. The forwarding method determines how the director sends packets from the client to the realservers.
Scheduling
The algorithm the director uses to select a realserver to service a new connection request from a client (ipvsadm and schedulers).
VIP
Virtual IP, the IP on the director that the client connects to.
DIP
Director IP, the IP on the director in the network.
RIP
Realserver IP, the IP on the realserver.
General setup of an LVS
The following figure illustrates a general layout of a network with an LVS:
_________ _________ _________
| | | | | |
| CLIENT 1| | CLIENT 2| | CLIENT N|
|_________| |_________| |_________|
| | |
--------------------------------
|
___|_____
| |
| GATEWAY |
|_________|
|
Linux Virtual Server |
...............................................................
. | .
. ___VIP____ .
. | | .
. | DIRECTOR | .
. |__________| .
. DIP .
. | .
. ------------------------------------------ .
. | | | .
. | | | .
. _____RIP1_____ ____RIP2______ _____RIPN_____ .
. | | | | | | .
. | REALSERVER 1 | | REALSERVER 2 | | REALSERVER N | .
. | | | | | | .
. |______________| |______________| |______________| .
. .
............................................................... One or Two NICs on the Director
If you have one NIC on the director, the VIP and the RIP are on the same physical network interface, where at least one virtual NIC was added, to hold one of the IP addresses.
If you have two NICs you can assign one to the VIP network and the second to the RIP network.
Here we have one NIC on the director.
Requirements For LVS-DR
For a reasonable LVS-DR setup the following is required:
- One or more clients which are on networks different from the VIP and DIP network.
- The realservers must be on the same network as the director (the realservers and director can arp each other).
- One director host with at least one network interface.
- One, better two static IPs for the director (one for the VIP, and one for the DIP).
- Two realservers.
- One static IP for each realserver.
Note to the Number of IPs on the director:
Depending on the service that is routed through the director, it might be useful to have two different IPs for the VIP and the DIP.
E.g.: If you load balance an ssh service and you assign one IP to your NIC, which acts as VIP and DIP at the same time you can not reach your director anymore via ssh, because all ssh requests are routed through to one of the realservers.
Installation Procedure
Director Installation
An SL54 server installation was performed on llclb1.
The director configuration shown below is implemented by means of puppet modules on puppet server psi-puppet1.
Realserver Installation
Contemporary, the realservers llc5 and llc6 are SL46 Desktop Enhanced systems.
One NIC LVS-DR Configuration
The director processes only the client-to-server half of a connection in the virtual server via direct routing, and the response packets can follow separate network routes to the clients. This can greatly increase the scalability of virtual server.
Compared to the virtual server via IP tunneling approach, this approach doesn't have tunneling overhead (In fact, this overhead is minimal in most situations), but requires that one of the load balancer's interfaces and the real server's interfaces must be in the same physical segment.
The following figure illustrates the setup of the LVS-DR with llclb1 (director) having one NIC. The whole LVS is on the same network:
_________ _________ _________
| | | | | |
| CLIENT 1| | CLIENT 2| | CLIENT N|
|_________| |_________| |_________|
| | |
--------------------------------
|
___|_____
| |
| GATEWAY | IP=129.129.190.1
|_________|
|
LVS (for service ssh) |
.....................................................................
. | .
. __________ | .
. | llc | | VIP=129.129.190.54 (eth0:1) .
. | | | .
. | DIRECTOR |---| .
. | | | .
. | llclb1 | | DIP=129.129.190.53 (eth0) .
. |__________| | .
. | .
. | .
. ------------------------------------ .
. | | .
. | | .
. RIP1=129.129.193.175 RIP2=129.129.193.176 .
. ______________ ______________ .
. | | | | .
. | REALSERVER 1 | | REALSERVER 2 | .
. | llc5 | | llc6 | .
. |______________| |______________| .
. .
..................................................................... Network Configuration
Configure the LVS network according the scheme shown above. Static IPs for llclb1, llc, llc5 and llc6 have to be assigned.
Director Configuration
Configure the static device eth0 and restart the network.
`/etc/sysconfig/network-scripts/ifcfg-eth0`:
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:14:5E:6B:13:3E
ONBOOT=yes
IPADDR=129.129.190.53
NETMASK=255.255.255.0
GATEWAY=129.129.193.1
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes`/etc/sysconfig/network`:
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=llclb1Install the package ipvsadm:
# yum install ipvsadmSetup the LVS for ssh on the director using the following script: Note: It might be better to add it to the init scripts.
`/etc/setup-LVS-DR-director.conf`:
#!/bin/bash
#---------------mini-rc.lvs_dr-director------------------------
###
### Network configuration
###
VIP1=129.129.190.54
RIP1=129.129.190.175
RIP2=129.129.190.176
# Set ip_forward OFF for lvs-dr director (1 on, 0 off)
# (there is no forwarding in the conventional sense for LVS-DR)
cat /proc/sys/net/ipv4/ip_forward
echo "0" >/proc/sys/net/ipv4/ip_forward
# Director is not gw for realservers: leave icmp redirects on
echo 'setting icmp redirects (1 on, 0 off) '
echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects
cat /proc/sys/net/ipv4/conf/all/send_redirects
echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects
cat /proc/sys/net/ipv4/conf/default/send_redirects
echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects
cat /proc/sys/net/ipv4/conf/eth0/send_redirects
# Add ethernet device and routing for VIP $VIP1
/sbin/ifconfig eth0:1 $VIP1 broadcast $VIP1 netmask 255.255.255.255
/sbin/route add -host $VIP1 dev eth0:1
# Listing ifconfig info for VIP $VIP1
/sbin/ifconfig eth0:1
# Check VIP $VIP1 is reachable from self (director)
/bin/ping -c 1 $VIP1
# Listing routing info for VIP $VIP1
/bin/netstat -rn
###
### Setup_ipvsadm_table
###
# Clear ipvsadm table
/sbin/ipvsadm -C
# Installing LVS services with ipvsadm
# Add ssh to VIP with round robin scheduling
/sbin/ipvsadm -A -t ${VIP1}:ssh -s rr
# Forward ssh to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP1 -g -w 1
# Check realserver reachable from director
ping -c 1 $RIP1
# Forward ssh to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP2 -g -w 1
# Check realserver reachable from director
ping -c 1 $RIP2
# Set tcp timeout to 72 hours while leaving
# tcpfin and udp timeouts unchanged.
/sbin/ipvsadm --set 259200 0 0
# List timeout values
/sbin/ipvsadm -L --timeout
# Displaying ipvsadm settings
/sbin/ipvsadm
# Not installing a default gw for LVS_TYPE vs-dr
###
### Delete an LVS entry
###
#
# Example: remove/delete ssh forwarding to RIP2
#
# /sbin/ipvsadm -d -t ${VIP1}:ssh -r RIP2
#
#
#---------------mini-rc.lvs_dr-director------------------------Realserver Configuration
The realserver shall send responses not to the VIP of the load balancer, rather to the client directly. This requires the iptables rule below.
Settings of realserver 2 for instance:
`/etc/sysconfig/network-scripts/ifcfg-eth0`:
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:06:5B:8C:3C:8E
ONBOOT=yes
TYPE=Ethernet
DHCP_HOSTNAME=llc6
PEERDNS=yes
IPADDR=129.129.190.176
NETMASK=255.255.255.0
GATEWAY=129.129.190.1
USERCTL=no
IPV6INIT=no`/etc/sysconfig/network`:
NETWORKING=yes
HOSTNAME=llc6`/etc/sysconfig/lvs`:
# LVS configuration file for LLC and LLCX
VIP=129.129.190.54`/etc/init.d/lvs`:
#! /bin/sh
#
# chkconfig: 345 90 10
# description: Startscript to initialize this machine as an lvs real server.
# Get network configuration
. /etc/sysconfig/network
# Get functions
. /etc/rc.d/init.d/functions
# Get VIP from the LVS configuration file
. /etc/sysconfig/lvs
# Check that networking is up
if [ ${NETWORKING} = "no" ] ; then
exit 0
fi
RETVAL=0
# See how we were called.
case "$1" in
start)
# Add rule
echo "Starting load balancing mechanism with NAT iptables "
/sbin/iptables -t nat -A PREROUTING -d $VIP -j REDIRECT
;;
stop)
# Delete rule
echo "Stopping load balancing mechanism with NAT iptables "
/sbin/iptables -t nat -D PREROUTING -d $VIP -j REDIRECT
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
;;
esac
exit $RETVALUpdate Procedure
Director Update
Login to llclb1 as root and run yum update:
# yum clean all
# yum updateThen reboot the director:
# rebootAfter rebooting no lvs rules are set by default:
[root@llclb1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConnAs soon as the realservers are updated and rebooted, too (see next section), run the lvs setup script to initialize the lvs rules for the ssh loadbalancing:
# sh /etc/setup-LVS-DR-director.conf
0
setting icmp redirects (1 on, 0 off)
1
1
1
SIOCADDRT: File exists
eth0:1 Link encap:Ethernet HWaddr 00:14:5E:6B:13:3E
inet addr:129.129.190.54 Bcast:129.129.190.54 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Memory:d8300000-d8310000
PING 129.129.190.54 (129.129.190.54) 56(84) bytes of data.
64 bytes from 129.129.190.54: icmp_seq=1 ttl=64 time=0.053 ms
--- 129.129.190.54 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
129.129.190.54 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
129.129.190.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 129.129.190.1 0.0.0.0 UG 0 0 0 eth0
PING 129.129.190.175 (129.129.190.175) 56(84) bytes of data.
64 bytes from 129.129.190.175: icmp_seq=1 ttl=64 time=2.13 ms
--- 129.129.190.175 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.139/2.139/2.139/0.000 ms
PING 129.129.190.176 (129.129.190.176) 56(84) bytes of data.
64 bytes from 129.129.190.176: icmp_seq=1 ttl=64 time=0.172 ms
--- 129.129.190.176 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.172/0.172/0.172/0.000 ms
Timeout (tcp tcpfin udp): 259200 120 300
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP llc.psi.ch:ssh rr
-> llc6.psi.ch:ssh Route 1 1 1
-> llc5.psi.ch:ssh Route 1 0 0Realserver Update
The realservers should be updated automatically as they are standard SL desktop hosts. Login as root to the corresponding realserver, e.g. llc5, and verify that the update was performed correctly, if not fix it first.
Then reboot the realserver. The iptables rule for the direct routing are initialized automatically by the init script /etc/init.d/lvs:
# rebootEventually, test the ssh connection from any client to llc:
# [anyuser@anyhost] ssh llc