534 lines
17 KiB
ReStructuredText
534 lines
17 KiB
ReStructuredText
Load Balancer `llclb1`
|
|
======================
|
|
|
|
References
|
|
----------
|
|
|
|
- http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html
|
|
|
|
- http://www.linuxvirtualserver.org/
|
|
|
|
|
|
Introduction
|
|
------------
|
|
|
|
This document describes the setup of `llclb1.psi.ch`, the Linux Login
|
|
Cluster LoadBalancer for the ssh service on llc5 and llc6.
|
|
|
|
The load balancing is implemented by means of the ipvsadm utility.
|
|
|
|
The forwarding method is LVS_DR, direct routing (see below).
|
|
|
|
Terms and Abbreviations
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
- LVS (Linux Virtual Server)
|
|
|
|
The Linux Virtual Server is a scalable server built on a cluster of
|
|
real servers, with the load balancer (director) running on the Linux
|
|
operating system (LVS = director + realservers). The architecture
|
|
of the server cluster is fully transparent to end users, and the
|
|
users interact as if it were a single server.
|
|
|
|
- IPVS, ip_vs
|
|
|
|
The code that patches the linux kernel on the director.
|
|
|
|
- Director (Load Balancer)
|
|
|
|
The node that runs the ipvs code. Clients connect to the
|
|
director. The director forwards packets to the realservers. The
|
|
director is nothing but an IP router with special rules that make
|
|
the LVS work.
|
|
|
|
- Realservers (Servers)
|
|
|
|
The hosts that have the services. The realservers handle the
|
|
requests from the clients.
|
|
|
|
- Client
|
|
|
|
The host or user level process that connects to the VIP on the
|
|
director.
|
|
|
|
- Forwarding method
|
|
|
|
Currently LVS-NAT, LVS-DR, LVS-Tun. The director is a router with
|
|
somewhat different rules for forwarding packets than a normal
|
|
router. The forwarding method determines how the director sends
|
|
packets from the client to the realservers.
|
|
|
|
- Scheduling
|
|
|
|
The algorithm the director uses to select a realserver to service a
|
|
new connection request from a client (ipvsadm and schedulers).
|
|
|
|
- VIP
|
|
|
|
Virtual IP, the IP on the director that the client connects to.
|
|
|
|
- DIP
|
|
|
|
Director IP, the IP on the director in the network.
|
|
|
|
- RIP
|
|
|
|
Realserver IP, the IP on the realserver.
|
|
|
|
|
|
General setup of an LVS
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The following figure illustrates a general layout of a network with an
|
|
LVS::
|
|
|
|
_________ _________ _________
|
|
| | | | | |
|
|
| CLIENT 1| | CLIENT 2| | CLIENT N|
|
|
|_________| |_________| |_________|
|
|
| | |
|
|
--------------------------------
|
|
|
|
|
___|_____
|
|
| |
|
|
| GATEWAY |
|
|
|_________|
|
|
|
|
|
Linux Virtual Server |
|
|
...............................................................
|
|
. | .
|
|
. ___VIP____ .
|
|
. | | .
|
|
. | DIRECTOR | .
|
|
. |__________| .
|
|
. DIP .
|
|
. | .
|
|
. ------------------------------------------ .
|
|
. | | | .
|
|
. | | | .
|
|
. _____RIP1_____ ____RIP2______ _____RIPN_____ .
|
|
. | | | | | | .
|
|
. | REALSERVER 1 | | REALSERVER 2 | | REALSERVER N | .
|
|
. | | | | | | .
|
|
. |______________| |______________| |______________| .
|
|
. .
|
|
...............................................................
|
|
|
|
|
|
|
|
One or Two NICs on the Director
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If you have one NIC on the director, the VIP and the RIP are on the
|
|
same physical network interface, where at least one virtual NIC was
|
|
added, to hold one of the IP addresses.
|
|
|
|
If you have two NICs you can assign one to the VIP network and the
|
|
second to the RIP network.
|
|
|
|
Here we have one NIC on the director.
|
|
|
|
|
|
Requirements For LVS-DR
|
|
-----------------------
|
|
|
|
For a reasonable LVS-DR setup the following is required:
|
|
|
|
- One or more clients which are on networks different from the VIP and
|
|
DIP network.
|
|
|
|
- The realservers must be on the same network as the director (the
|
|
realservers and director can arp each other).
|
|
|
|
- One director host with at least one network interface.
|
|
|
|
- One, better two static IPs for the director (one for the VIP, and
|
|
one for the DIP).
|
|
|
|
- Two realservers.
|
|
|
|
- One static IP for each realserver.
|
|
|
|
|
|
**Note to the Number of IPs on the director**:
|
|
|
|
Depending on the service that is routed through the director, it might
|
|
be useful to have two different IPs for the VIP and the DIP.
|
|
|
|
E.g.: If you load balance an ssh service and you assign one IP to your
|
|
NIC, which acts as VIP and DIP at the same time you can not reach your
|
|
director anymore via ssh, because all ssh requests are routed through
|
|
to one of the realservers.
|
|
|
|
Installation Procedure
|
|
----------------------
|
|
|
|
Director Installation
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
An SL54 server installation was performed on llclb1.
|
|
|
|
The director configuration shown below is implemented by means of
|
|
puppet modules on puppet server psi-puppet1.
|
|
|
|
|
|
Realserver Installation
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Contemporary, the realservers llc5 and llc6 are SL46 Desktop Enhanced
|
|
systems.
|
|
|
|
|
|
One NIC LVS-DR Configuration
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The director processes only the client-to-server half of a connection
|
|
in the virtual server via direct routing, and the response packets can
|
|
follow separate network routes to the clients. This can greatly
|
|
increase the scalability of virtual server.
|
|
|
|
Compared to the virtual server via IP tunneling approach, this
|
|
approach doesn't have tunneling overhead (In fact, this overhead is
|
|
minimal in most situations), but requires that one of the load
|
|
balancer's interfaces and the real server's interfaces must be in the
|
|
same physical segment.
|
|
|
|
The following figure illustrates the setup of the LVS-DR with llclb1
|
|
(director) having one NIC. The whole LVS is on the same network::
|
|
|
|
|
|
_________ _________ _________
|
|
| | | | | |
|
|
| CLIENT 1| | CLIENT 2| | CLIENT N|
|
|
|_________| |_________| |_________|
|
|
| | |
|
|
--------------------------------
|
|
|
|
|
___|_____
|
|
| |
|
|
| GATEWAY | IP=129.129.190.1
|
|
|_________|
|
|
|
|
|
LVS (for service ssh) |
|
|
.....................................................................
|
|
. | .
|
|
. __________ | .
|
|
. | llc | | VIP=129.129.190.54 (eth0:1) .
|
|
. | | | .
|
|
. | DIRECTOR |---| .
|
|
. | | | .
|
|
. | llclb1 | | DIP=129.129.190.53 (eth0) .
|
|
. |__________| | .
|
|
. | .
|
|
. | .
|
|
. ------------------------------------ .
|
|
. | | .
|
|
. | | .
|
|
. RIP1=129.129.193.175 RIP2=129.129.193.176 .
|
|
. ______________ ______________ .
|
|
. | | | | .
|
|
. | REALSERVER 1 | | REALSERVER 2 | .
|
|
. | llc5 | | llc6 | .
|
|
. |______________| |______________| .
|
|
. .
|
|
.....................................................................
|
|
|
|
|
|
Network Configuration
|
|
.....................
|
|
|
|
Configure the LVS network according the scheme shown above. Static
|
|
IPs for llclb1, llc, llc5 and llc6 have to be assigned.
|
|
|
|
|
|
Director Configuration
|
|
......................
|
|
|
|
Configure the static device eth0 and restart the network.
|
|
|
|
`/etc/sysconfig/network-scripts/ifcfg-eth0`::
|
|
|
|
DEVICE=eth0
|
|
BOOTPROTO=none
|
|
HWADDR=00:14:5E:6B:13:3E
|
|
ONBOOT=yes
|
|
IPADDR=129.129.190.53
|
|
NETMASK=255.255.255.0
|
|
GATEWAY=129.129.193.1
|
|
TYPE=Ethernet
|
|
USERCTL=no
|
|
IPV6INIT=no
|
|
PEERDNS=yes
|
|
|
|
|
|
`/etc/sysconfig/network`::
|
|
|
|
NETWORKING=yes
|
|
NETWORKING_IPV6=no
|
|
HOSTNAME=llclb1
|
|
|
|
|
|
|
|
Install the package ipvsadm::
|
|
|
|
# yum install ipvsadm
|
|
|
|
Setup the LVS for ssh on the director using the following script:
|
|
Note: It might be better to add it to the init scripts.
|
|
|
|
`/etc/setup-LVS-DR-director.conf`::
|
|
|
|
#!/bin/bash
|
|
#---------------mini-rc.lvs_dr-director------------------------
|
|
|
|
###
|
|
### Network configuration
|
|
###
|
|
|
|
VIP1=129.129.190.54
|
|
|
|
RIP1=129.129.190.175
|
|
RIP2=129.129.190.176
|
|
|
|
|
|
# Set ip_forward OFF for lvs-dr director (1 on, 0 off)
|
|
# (there is no forwarding in the conventional sense for LVS-DR)
|
|
cat /proc/sys/net/ipv4/ip_forward
|
|
echo "0" >/proc/sys/net/ipv4/ip_forward
|
|
|
|
# Director is not gw for realservers: leave icmp redirects on
|
|
echo 'setting icmp redirects (1 on, 0 off) '
|
|
echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects
|
|
cat /proc/sys/net/ipv4/conf/all/send_redirects
|
|
echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects
|
|
cat /proc/sys/net/ipv4/conf/default/send_redirects
|
|
echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects
|
|
cat /proc/sys/net/ipv4/conf/eth0/send_redirects
|
|
|
|
# Add ethernet device and routing for VIP $VIP1
|
|
/sbin/ifconfig eth0:1 $VIP1 broadcast $VIP1 netmask 255.255.255.255
|
|
/sbin/route add -host $VIP1 dev eth0:1
|
|
# Listing ifconfig info for VIP $VIP1
|
|
/sbin/ifconfig eth0:1
|
|
|
|
# Check VIP $VIP1 is reachable from self (director)
|
|
/bin/ping -c 1 $VIP1
|
|
# Listing routing info for VIP $VIP1
|
|
/bin/netstat -rn
|
|
|
|
###
|
|
### Setup_ipvsadm_table
|
|
###
|
|
|
|
# Clear ipvsadm table
|
|
/sbin/ipvsadm -C
|
|
|
|
# Installing LVS services with ipvsadm
|
|
# Add ssh to VIP with round robin scheduling
|
|
/sbin/ipvsadm -A -t ${VIP1}:ssh -s rr
|
|
|
|
# Forward ssh to realserver using direct routing with weight 1
|
|
/sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP1 -g -w 1
|
|
# Check realserver reachable from director
|
|
ping -c 1 $RIP1
|
|
|
|
# Forward ssh to realserver using direct routing with weight 1
|
|
/sbin/ipvsadm -a -t ${VIP1}:ssh -r $RIP2 -g -w 1
|
|
# Check realserver reachable from director
|
|
ping -c 1 $RIP2
|
|
|
|
# Set tcp timeout to 72 hours while leaving
|
|
# tcpfin and udp timeouts unchanged.
|
|
/sbin/ipvsadm --set 259200 0 0
|
|
|
|
# List timeout values
|
|
/sbin/ipvsadm -L --timeout
|
|
|
|
# Displaying ipvsadm settings
|
|
/sbin/ipvsadm
|
|
|
|
# Not installing a default gw for LVS_TYPE vs-dr
|
|
|
|
###
|
|
### Delete an LVS entry
|
|
###
|
|
#
|
|
# Example: remove/delete ssh forwarding to RIP2
|
|
#
|
|
# /sbin/ipvsadm -d -t ${VIP1}:ssh -r RIP2
|
|
#
|
|
#
|
|
|
|
#---------------mini-rc.lvs_dr-director------------------------
|
|
|
|
|
|
Realserver Configuration
|
|
........................
|
|
|
|
The realserver shall send responses not to the VIP of the load
|
|
balancer, rather to the client directly. This requires the iptables
|
|
rule below.
|
|
|
|
Settings of realserver 2 for instance:
|
|
|
|
`/etc/sysconfig/network-scripts/ifcfg-eth0`::
|
|
|
|
DEVICE=eth0
|
|
BOOTPROTO=none
|
|
HWADDR=00:06:5B:8C:3C:8E
|
|
ONBOOT=yes
|
|
TYPE=Ethernet
|
|
DHCP_HOSTNAME=llc6
|
|
PEERDNS=yes
|
|
IPADDR=129.129.190.176
|
|
NETMASK=255.255.255.0
|
|
GATEWAY=129.129.190.1
|
|
USERCTL=no
|
|
IPV6INIT=no
|
|
|
|
|
|
`/etc/sysconfig/network`::
|
|
|
|
NETWORKING=yes
|
|
HOSTNAME=llc6
|
|
|
|
|
|
`/etc/sysconfig/lvs`::
|
|
|
|
# LVS configuration file for LLC and LLCX
|
|
VIP=129.129.190.54
|
|
|
|
|
|
`/etc/init.d/lvs`::
|
|
|
|
#! /bin/sh
|
|
#
|
|
# chkconfig: 345 90 10
|
|
# description: Startscript to initialize this machine as an lvs real server.
|
|
|
|
# Get network configuration
|
|
. /etc/sysconfig/network
|
|
# Get functions
|
|
. /etc/rc.d/init.d/functions
|
|
# Get VIP from the LVS configuration file
|
|
. /etc/sysconfig/lvs
|
|
|
|
# Check that networking is up
|
|
if [ ${NETWORKING} = "no" ] ; then
|
|
exit 0
|
|
fi
|
|
|
|
RETVAL=0
|
|
|
|
# See how we were called.
|
|
case "$1" in
|
|
start)
|
|
# Add rule
|
|
echo "Starting load balancing mechanism with NAT iptables "
|
|
/sbin/iptables -t nat -A PREROUTING -d $VIP -j REDIRECT
|
|
;;
|
|
stop)
|
|
# Delete rule
|
|
echo "Stopping load balancing mechanism with NAT iptables "
|
|
/sbin/iptables -t nat -D PREROUTING -d $VIP -j REDIRECT
|
|
;;
|
|
*)
|
|
echo "Usage: $0 {start|stop}"
|
|
exit 1
|
|
;;
|
|
esac
|
|
|
|
exit $RETVAL
|
|
|
|
|
|
Update Procedure
|
|
----------------
|
|
|
|
Director Update
|
|
~~~~~~~~~~~~~~~
|
|
|
|
Login to llclb1 as root and run yum update::
|
|
|
|
# yum clean all
|
|
# yum update
|
|
|
|
Then reboot the director::
|
|
|
|
# reboot
|
|
|
|
After rebooting no lvs rules are set by default::
|
|
|
|
[root@llclb1 ~]# ipvsadm -L
|
|
IP Virtual Server version 1.2.1 (size=4096)
|
|
Prot LocalAddress:Port Scheduler Flags
|
|
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
|
|
|
|
As soon as the realservers are updated and rebooted, too (see next
|
|
section), run the lvs setup script to initialize the lvs rules for the
|
|
ssh loadbalancing::
|
|
|
|
# sh /etc/setup-LVS-DR-director.conf
|
|
|
|
|
|
0
|
|
setting icmp redirects (1 on, 0 off)
|
|
1
|
|
1
|
|
1
|
|
SIOCADDRT: File exists
|
|
eth0:1 Link encap:Ethernet HWaddr 00:14:5E:6B:13:3E
|
|
inet addr:129.129.190.54 Bcast:129.129.190.54 Mask:255.255.255.255
|
|
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
|
|
Interrupt:169 Memory:d8300000-d8310000
|
|
|
|
PING 129.129.190.54 (129.129.190.54) 56(84) bytes of data.
|
|
64 bytes from 129.129.190.54: icmp_seq=1 ttl=64 time=0.053 ms
|
|
|
|
--- 129.129.190.54 ping statistics ---
|
|
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
|
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
|
|
Kernel IP routing table
|
|
Destination Gateway Genmask Flags MSS Window irtt Iface
|
|
129.129.190.54 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
|
|
129.129.190.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
|
|
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
|
|
0.0.0.0 129.129.190.1 0.0.0.0 UG 0 0 0 eth0
|
|
PING 129.129.190.175 (129.129.190.175) 56(84) bytes of data.
|
|
64 bytes from 129.129.190.175: icmp_seq=1 ttl=64 time=2.13 ms
|
|
|
|
--- 129.129.190.175 ping statistics ---
|
|
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
|
rtt min/avg/max/mdev = 2.139/2.139/2.139/0.000 ms
|
|
PING 129.129.190.176 (129.129.190.176) 56(84) bytes of data.
|
|
64 bytes from 129.129.190.176: icmp_seq=1 ttl=64 time=0.172 ms
|
|
|
|
--- 129.129.190.176 ping statistics ---
|
|
1 packets transmitted, 1 received, 0% packet loss, time 0ms
|
|
rtt min/avg/max/mdev = 0.172/0.172/0.172/0.000 ms
|
|
Timeout (tcp tcpfin udp): 259200 120 300
|
|
IP Virtual Server version 1.2.1 (size=4096)
|
|
Prot LocalAddress:Port Scheduler Flags
|
|
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
|
|
TCP llc.psi.ch:ssh rr
|
|
-> llc6.psi.ch:ssh Route 1 1 1
|
|
-> llc5.psi.ch:ssh Route 1 0 0
|
|
|
|
|
|
Realserver Update
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
The realservers should be updated automatically as they are standard
|
|
SL desktop hosts. Login as root to the corresponding realserver,
|
|
e.g. llc5, and verify that the update was performed correctly, if not
|
|
fix it first.
|
|
|
|
Then reboot the realserver. The iptables rule for the direct routing
|
|
are initialized automatically by the init script /etc/init.d/lvs::
|
|
|
|
# reboot
|
|
|
|
Eventually, test the ssh connection from any client to llc::
|
|
|
|
# [anyuser@anyhost] ssh llc
|
|
|