Nov 252009
 

Playing with LVS – so that I don’t have to connect to individual Cassandra server.

What I planned for LVS:

  • 192.168.1.99 will be VIP
  • f5 (192.168.1.205) will be LVS director … you are right, its name is f5 ๐Ÿ˜‰
  • f1 (192.168.1.101) to f4 (192.168.1.104) will be real server
  • will do DR mode (I think it is called single-B on most L4 switches …)

The configuration is actually pretty simple, as long as you get it right, on LVS director (f5):

  • Configure VIP to eth0:0 as an alias, netmask should be 255.255.255.255, broadcast is VIP itself
  • Add following rules with ipvsadm (yea … you need to install this package)
    -A -t 192.168.1.99:9160 -s wlc
    -a -t 192.168.1.99:9160 -r 192.168.1.101:9160 -g -w 1
    -a -t 192.168.1.99:9160 -r 192.168.1.102:9160 -g -w 1
    -a -t 192.168.1.99:9160 -r 192.168.1.103:9160 -g -w 1
    -a -t 192.168.1.99:9160 -r 192.168.1.104:9160 -g -w 1
  • Start LVS (after restart network to make sure VIP is effective) by “ipvsadm –start-daemon master”
  • If you want to stop LVS you can do “ipvsadm –stop-daemon master”

Now let’s turn to real servers – all real servers (f1~f4) are doing the same:

  • Add these lines to /etc/sysctl.conf and then do a “sysctl -p”
    net.ipv4.conf.dummy0.arp_ignore = 1
    net.ipv4.conf.dummy0.arp_announce = 2
    net.ipv4.conf.all.arp_ignore = 1
    net.ipv4.conf.all.arp_announce = 2
  • Configure VIP to dummy0 as an alias, netmask should be 255.255.255.255, broadcast is VIP itself
  • Change ThriftAddress in storage-conf.xml for Cassandra, to 0.0.0.0 so that Thrift serves on all interfaces
  • Remember to restart Cassandra so to make new configuration takes effect

Now, launch your favorite client, connecting to 192.168.1.99:9160, you should get everything as if connect to individual server.

  4 Responses to “LVS”

  1. I didn’t put command/configuration for adding alias to eth0/lo because Ubuntu and Fedora are far from “similar”, there are a lot web pages talking about this so I don’t have to copy them here, just to mention that you need to configure the IP and netmask only, nothing else.

  2. I found some problems with my setup above – the real servers are unable to close the connection so the TCP connections are getting more and more, need to figure this out.

  3. It seems the problem comes from iptables, everything works fine after disabling it, however, I still need to figure out what is needed for LVS as I cannot leave iptables off …

  4. Here is the “fix, changing this line in /etc/sysconfig/iptables:

    -A INPUT -m state –state NEW -m tcp -p tcp –dport 9160 -j ACCEPT

    to:

    -A INPUT -m tcp -p tcp –dport 9160 -j ACCEPT

    solved the problem. I still need to read some more to know why.

Sorry, the comment form is closed at this time.