LVS

Playing with LVS – so that I don’t have to connect to individual Cassandra server.

What I planned for LVS:

  • 192.168.1.99 will be VIP
  • f5 (192.168.1.205) will be LVS director … you are right, its name is f5 ;)
  • f1 (192.168.1.101) to f4 (192.168.1.104) will be real server
  • will do DR mode (I think it is called single-B on most L4 switches …)

The configuration is actually pretty simple, as long as you get it right, on LVS director (f5):

  • Configure VIP to eth0:0 as an alias, netmask should be 255.255.255.255, broadcast is VIP itself
  • Add following rules with ipvsadm (yea … you need to install this package)
    -A -t 192.168.1.99:9160 -s wlc
    -a -t 192.168.1.99:9160 -r 192.168.1.101:9160 -g -w 1
    -a -t 192.168.1.99:9160 -r 192.168.1.102:9160 -g -w 1
    -a -t 192.168.1.99:9160 -r 192.168.1.103:9160 -g -w 1
    -a -t 192.168.1.99:9160 -r 192.168.1.104:9160 -g -w 1
  • Start LVS (after restart network to make sure VIP is effective) by “ipvsadm –start-daemon master”
  • If you want to stop LVS you can do “ipvsadm –stop-daemon master”

Now let’s turn to real servers – all real servers (f1~f4) are doing the same:

  • Add these lines to /etc/sysctl.conf and then do a “sysctl -p”
    net.ipv4.conf.dummy0.arp_ignore = 1
    net.ipv4.conf.dummy0.arp_announce = 2
    net.ipv4.conf.all.arp_ignore = 1
    net.ipv4.conf.all.arp_announce = 2
  • Configure VIP to dummy0 as an alias, netmask should be 255.255.255.255, broadcast is VIP itself
  • Change ThriftAddress in storage-conf.xml for Cassandra, to 0.0.0.0 so that Thrift serves on all interfaces
  • Remember to restart Cassandra so to make new configuration takes effect

Now, launch your favorite client, connecting to 192.168.1.99:9160, you should get everything as if connect to individual server.

Tags: , ,

categories Triviality | Hang | datetime 2009-11-25 00:46

4 Comments

  • By Hang, 2009-11-25 @ 00:51

    I didn’t put command/configuration for adding alias to eth0/lo because Ubuntu and Fedora are far from “similar”, there are a lot web pages talking about this so I don’t have to copy them here, just to mention that you need to configure the IP and netmask only, nothing else.

  • By Hang, 2009-11-25 @ 11:55

    I found some problems with my setup above – the real servers are unable to close the connection so the TCP connections are getting more and more, need to figure this out.

  • By Hang, 2009-11-25 @ 15:36

    It seems the problem comes from iptables, everything works fine after disabling it, however, I still need to figure out what is needed for LVS as I cannot leave iptables off …

  • By Hang, 2009-11-25 @ 15:55

    Here is the “fix, changing this line in /etc/sysconfig/iptables:

    -A INPUT -m state –state NEW -m tcp -p tcp –dport 9160 -j ACCEPT

    to:

    -A INPUT -m tcp -p tcp –dport 9160 -j ACCEPT

    solved the problem. I still need to read some more to know why.

Other Links to this Post

RSS feed for comments on this post. TrackBack URI

Leave a comment

WordPress Themes