Adding new nodes to Cassandra cluster

 Triviality  1 Response »
Jan 182010
 

Actually this apply to adding new nodes, or removing existing nodes:

  • Add the new node in, make sure AutoBootstrap is set to true so it will step into bootstrap node, for removing old nodes, just shut it down
  • For all existing nodes current in ring, do the calculation to get different ranges which used for token, the formula is 2^127/nodes.
  • Now, on each node, run “nodeprobe move token-for-this-node”, you can run only one node at a time as data will be moved around
  • After all nodes finish the move, do “nodeprobe cleanup” to remove useless entries

The serving should not be affected during the operation though performance may be affected.

The “loadbalance” does not make things perfect, though Cassandra guys mentioned it should be good enough.

Cassandra and multi-core

 Triviality  3 Responses »
Jan 182010
 

It seems Cassandra creates a big pile of threads for different tasks, I didn’t step into details, but I’m pretty sure it has more than 40 threads with default setting on a 2-nodes cluster. So multi core may not be a concern, as all these threads can run on different cores to fully utilize the CPU resource.

However, my tests show something really worry me – multi-node cluster performs worse than single node (due to inter-node communication I believe), and multi-cores deployment is slower than single-core deployment (this is something I don’t quite understand, may be because of L1/L2 cache?).

I need some hardware to test it as well, as VM is not that good for this kind of test. Then I suddenly recall I still have some 8-core/16G boxes in company sitting idle there, I can use one or two to do the test for sure ๐Ÿ˜‰