May 252015
 

Just to remind myself in case I get lost again:

  • Install Oracle JRE 7, at least for now JRE 8 is not recommended
    • Editย /etc/apt/sources.list.d/webupd8team-java.list with these lines:
      deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main               
      deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main
    • Import GPG
      sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
    • Install
      sudo apt-get update
      sudo apt-get install oracle-java7-installer
  • Install Cassandra
    • Edit /etc/apt/sources.list.d/cassandra.source.list with these lines
      deb http://debian.datastax.com/community 2.1 main
      deb-src http://debian.datastax.com/community 2.1 main
    • Import keys
      curl -L http://debian.datastax.com/debian/repo_key | sudo apt-key add -
    • Install
      sudo apt-get update
      sudo apt-get install cassandra
Jan 222010
 

I “unfollowed” all Linux distros as they didn’t give me much useful information, I’m still following CouchDB, Cassandra, and MongoDB, so far their tweets are kind of helpful.

Also I subscribed to Cassandra’s user mailing list, lots of interesting topics there.

Jan 192010
 

I setup a testing environment on couple of company boxes to see how Cassandra performs with real machines (real here means powerful enough to be a data node), here are details of the environment:

  • Two client nodes, one server nodes, all are RHEL 4.x. I use two clients nodes as I found that during the performance test, single client machine is unable to generate enough load
  • All three machines are 8 cores/16G memory (well, memory is not a big deal for my tests)
  • Running Cassandray 0.5.0 RC3 (built from svn last night)
  • Client is using Python

Here is the graph for simple request (single key lookup):

It seems the result is pretty encouraging – query per second of the server is growing almost linearly, at about 5,000 QPS, over CPU utilization is still under 40% (25% user, 12% sys), I cannot get more client boxes to test, but if it goes this way, and let’s make 80% is threshold of CPU utilization, then this kind of box can handle 10K QPS, roughly, with latency at around 3ms.

Note that CPU utilization, QPS per client, and latency is not quite clear as the overall QPS is too high, but you can get some ideas from next graph …

Here is the graph for application (login, which will do one user lookup, and then 10~100 user lookups, each lookup is to get one buddy’s information):

The result is kind of worrying me, since the CPU utilization is 70% already (45% user and 25% sys), it seems 200 QPS is what the cluster can provide. However, thinking of the login operation is doing way too many table lookups (average 55 lookups per login), so just matches the simple lookup we discussed above (10K QPS per box), while latency is at around 80ms.

Actually, 20% sys is pretty bad, means the kernel is busy switching (I didn’t check vmstat during that time, but this is a reasonable guess), but again, this may be reasonable since the machine is handling 16 active clients who are sending bunch of requests, while it has only 8 physical cores so context switching is unavoidable.

Since everything’s linear, I can assume 4 cores boxes can offer 5,000 QPS with reasonable latency. I will do some similar tests with MySQL and memcached, and I will do similar test with multiple data nodes as well, since I got impression that multiple data nodes is far slower than single node (inter-node communication?).

Jan 182010
 

Actually this apply to adding new nodes, or removing existing nodes:

  • Add the new node in, make sure AutoBootstrap is set to true so it will step into bootstrap node, for removing old nodes, just shut it down
  • For all existing nodes current in ring, do the calculation to get different ranges which used for token, the formula is 2^127/nodes.
  • Now, on each node, run “nodeprobe move token-for-this-node”, you can run only one node at a time as data will be moved around
  • After all nodes finish the move, do “nodeprobe cleanup” to remove useless entries

The serving should not be affected during the operation though performance may be affected.

The “loadbalance” does not make things perfect, though Cassandra guys mentioned it should be good enough.

Jan 182010
 

It seems Cassandra creates a big pile of threads for different tasks, I didn’t step into details, but I’m pretty sure it has more than 40 threads with default setting on a 2-nodes cluster. So multi core may not be a concern, as all these threads can run on different cores to fully utilize the CPU resource.

However, my tests show something really worry me – multi-node cluster performs worse than single node (due to inter-node communication I believe), and multi-cores deployment is slower than single-core deployment (this is something I don’t quite understand, may be because of L1/L2 cache?).

I need some hardware to test it as well, as VM is not that good for this kind of test. Then I suddenly recall I still have some 8-core/16G boxes in company sitting idle there, I can use one or two to do the test for sure ๐Ÿ˜‰

Jan 132010
 

I have some questions about cassandra that haven’t got answer yet, I’m writing them done now to make sure I won’t forget them in the future, all these questions are pretty critical for operation:

  • How does cassandra utilize multi-core? Does it have multiple threads internally, handling different requests in different threads? If the answer is no then it will be pretty ugly since I have to run multiple cassandra instances on one machine
  • Is cassandra (maybe Java) capable to handle large memory? Make it clear, can it fully utilize 64-bit machine’s memory (8G/16G/64G)? Again if the answer is no I have to run multiple instances per machine.
  • I know cassandra can replicate data from one colo to another colo, but what I understand is that they are virtually same cluster – is it possible to make two colos both have full data set, and for client request it only return data from local node?
  • Is it possible to stream updates to cassandra to another source? What I want is capturing a live data set in another data store (such as RDBMS) for other purpose, so prefer a plug-in type of implementation so that I can grab updates and send to different downstream.

Will post more once there is anything jump into my mind, and will post answers (separated blog) if I hear of any.