NOTE: This is a sub-page off the main redundant cloud hosting configuration guide. If you've arrived at this page first, I recommend reviewing the parent page first.
End goal of this step
When you've completed this portion of the guide, you should have the following:
- two failover load balancers running ldirectord, heartbeat and LVS-TUN
- IP-IP tunneling configured on the web nodes
Getting started
For this step, you'll need a virtual IP address that can float between both of your load balancers and your web nodes. You'll need this configured for two reasons: your load balancers will receive traffic on that IP address and your web nodes will reply using that IP address as a source. Heartbeat will determine which load balancer will take control of the virtual IP at one time and the web nodes will be configured so they won't steal the IP from the load balancers with an ARP reply.
Configuring the load balancers
You'll need to set up heartbeat just like you did for the database servers in the first step of this guide. All of the configuration should be the same with the exception of IP addresses and hostnames being different. Obviously, you'll also need a new /etc/heartbeat/haresources this time around:
lb1.mydomain.com 11.22.33.44/24/eth0 ldirectord::/etc/ha.d/ldirectord.cf
Be sure to replace 11.22.33.44 with your virtual IP address that will be shared amongst your servers. Don't start heartbeat yet - we still need to configure ldirectord. Start by installing a couple of packages:
yum -y install ipvsadm ldirectord
The ipvsadm package gives you the tools you need to manage and monitor your IPVS configuration. Luckily, ldirectord makes the configuration and monitoring a piece of cake, so we won't need ipvsadm for any configuration quite yet. Let's make a simple ldirectord configuration file on both nodes:
cat /etc/heartbeat/ldirectord.cf
checktimeout=10
checkinterval=2
autoreload=yes
logfile="local0"
quiescent=yes
virtual=11.22.33.44:80
real=10.1.100.40:80 ipip
real=10.1.100.50:80 ipip
service=http
request="healthcheck.html"
receive="OK"
scheduler=rr
protocol=tcp
checktype=negotiateThe configuration file above sets up the following:
- ldirectord will do a
GET /healthcheck.htmlcall to each web node (and expect to see the text "OK") every two seconds and allow them 10 seconds to reply before dropping them from the balancing table - if you make ldirectord configuration file changes, they'll be picked up immediately without a restart of the daemon
- the load balancers will use a round robin algorithm to choose a web node
- traffic will come in on the virtual IP (11.22.33.44) and be redirected via IP-IP tunneling to the two real servers via their private interfaces
If you start heartbeat on the first node, it should bring ldirectord up automatically. You can check your IPVS configuration by running ipvsadm -L. For example, my production load balancer looks like this:
# ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP dfw-vip.mhtx.net:http rr -> web2.mhtx.net:http Tunnel 1 46 93 -> web1.mhtx.net:http Tunnel 1 39 92
Since you haven't configured your web nodes yet, your columns should show zeroes across the board. Don't worry - we'll have that fixed shortly.
Setting up the web nodes
We'll need a web server as well as the arptables tools on our web nodes:
yum -y install httpd arptables_jf
Setting up the health checks is quick process:
echo "OK" > /var/www/html/healthcheck.html /etc/init.d/httpd start chkconfig httpd on curl localhost/healthcheck.html
If you test your healthcheck page, you should see something like this:
[root@web1 ~]# curl localhost/healthcheck.html OK
For the IP-IP tunneling to terminate properly on your web nodes, you'll need to set up a tunnel interface:
# cat /etc/sysconfig/network-scripts/ifcfg-tunl0: DEVICE=tunl0 IPADDR=11.22.33.44 NETMASK=255.255.255.255 BROADCAST=11.22.33.44 ONBOOT=yes
Be sure to replace the 11.22.33.44 with your virtual IP address. We'll need a simple static route for this interface to work properly:
# cat /etc/sysconfig/network-scripts/route-tunl0: 11.22.33.44 dev tunl0
Again, be sure to use your virtual IP address in this file. We need to set some kernel parameters for the new tunnel interface in /sbin/ifup-local (thanks to Jason Gill for pointing this out):
#!/bin/sh
if [ $1 = "tunl0" ]; then
echo 1 > /proc/sys/net/ipv4/conf/tunl0/forwarding
echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
fiMake it executable:
chmod +x /sbin/ifup-local
We now need to ensure that the web nodes don't accidentally send out an ARP reply on the network. This would cause them to steal the IP from the load balancers and it would essentially knock your configuration offline. I do this with arptables:
arptables -A IN -j DROP -d 11.22.33.44 arptables -A OUT -j DROP -d 11.22.33.44 /etc/init.d/arptables_jf save chkconfig arptables_jf on
There are some elegant ways to do this within your interface configuration files or via adjusting values in /proc, but I've found this method to be both simple and reliable. You should now be able to bring up the new interface safely:
ifup tunl0
Testing
To test out this configuration, ensure that heartbeat is running on both load balancers and that ipvsadm on the load balancers shows that both of your web nodes are responding. You should be able to pull the healthcheck page through your load balancers with curl:
$ curl http://11.22.33.44/healthcheck.html OK
If that works, try adding the following line to /etc/httpd/conf/httpd.conf on each web node:
Header add Node "web1"
You'll want to adjust the name of each node in the configuration file so that you can tell which is which. Reload your apache configuration on each web node and then try again:
$ curl -i http://11.22.33.44/healthcheck.html 2>/dev/null | grep ^Node Node: web1 $ curl -i http://11.22.33.44/healthcheck.html 2>/dev/null | grep ^Node Node: web2 $ curl -i http://11.22.33.44/healthcheck.html 2>/dev/null | grep ^Node Node: web1
As you can see, the load balancers are choosing each web node on a round robin algorithm. You may also want to run tcpdump on the load balancers to ensure that the traffic is coming through the load balancer and then out via the web nodes.
Try running the following on the load balancer while stopping apache on one of the web nodes:
watch --interval=1 'ipvsadm -L'
You should see the weight column switch from one to zero when a web node isn't responding properly.
Conclusion
You should now have two redundant load balancers that are constantly monitoring the availability of your web nodes. Also, your web nodes are configured to accept the encapsulated packets from the load balancers via the private interface and then respond to the traffic via their own public interfaces.
