Performance of VMs

 Triviality  3 Responses »
Nov 182009
 

I used to run 4 VMs on my dual core machine but actually was running nothing seriousely. A while back I read an article from Ubuntu about its cloud setup, mentioning you should not run VMs more than number of cores you have, I was laughing at the article at that time, since it sounds ridiclous – as long as CPU can handle the load, we should run as many VMs as possible and – the bottleneck seems coming freom memory to me.

Obviously I was wrong. Once I started testing Casasandra, I got lots of timeouts from 4 VMs, I checked memory, there was no problem at all, i.e. every VM’s memory is in physical memory. I was also doubt Ubuntu is not doing as good as Fedore, obviously Iwas wrong either, then I recalled the article, shut down 2 VMs and … you know, things work like a chime.

There could be saomew thesis telling why CPU is the bottleneck, but I don’t have time to dig it out, the most important lesson I took from this case is, don’t take anythingn in granted, respect preofessional advices

Changes …

 Triviality  No Responses »
Nov 172009
 

Here is the deal – I decided to drop PHP and moving to Python, so that I can spend less time on dealing with less-supported PHP (in this nosql wave), I’ve removed PHP from all dev/test environments and wish I won’t come back later.

Actually I’ve made PHP works, but I’m just not feeling well as not many people are using PHP and it seems hard to seek help whenever needed. Also, seems setting up Python with Apache (through wsgi) is not that difficult. It could be a good chance to lear Python as well, though I did some PyS60 a while ago (for Jabber on E90).

BTW, I’ve upgraded all Fedora instances (3 of them) to Fedora 12, so far so good.

Moving to Ubuntu 9.10

 Triviality  No Responses »
Oct 302009
 

I just upgraded following machines to ubuntu 9.10:

  • debian 5
  • ubuntu LTS 8.04
  • ubuntu 9.04

so now for all debian-like distro I’m running pure Ubuntu.

Fedora will release 12 in mid-November, I will “upgrade” both centos and fedora 11, and after that I will keep running both Ubuntu and Fedora, as my Linux platform, 50-50.

 Posted by at 02:04  Tagged with:

Stick with Fedora and Ubuntu

 Triviality  No Responses »
Oct 252009
 

I’m building NoSQL stuffs that are pretty much from source (instead of well packaged), so I have some restriction on libraries version, such as libevent, Berkeley DB, etc.

Obviously CentOS is a bad choice for this as its goal to maintain stability, similar to Ubuntu LTS version. Surprisingly, Debian is not that up-to-date in repo (for example, db4.6 only, not 4.7).

So I think I should stick with Fedora and Ubuntu (non-LTS version) which seems keeping cutting edge stuffs. I will also check other distro – I guess as long as they can maintain 6-months release cycle, they will be “up-to-date”.

I will get rid of CentOS and Debian for now, and if I cannot find any other distro meet the requirement I will make all my testing nodes running Fedora and Ubuntu, half-half.

NoSQL – start with consistent hashing

 Triviality  2 Responses »
Oct 232009
 

Most NoSQL solutions are kind of caching, with persistent data store, with or without replication support. One of the key issue in production environment is using consistent hashing to avoid cache failure.

I talked to a friend days ago about memcached deployment problem, he asked question about what to do with adding new memcached node to expand capacity, to avoid loading bunch of data from database to cache nodes. I told him I don’t have any experience, but if I encounter this problem, I will try to restart memcached client machines one by one, to use new configuration, so to avoid put massive load to database, also I will think about changing hashing function of memcached client, try to maximize entries that can keep partition unchanged.

It turned out my second idea is correct (I should have read all those articles before talking to him :P). There are couple of articles discussing about this issue, and the good start point, of course, is wikipedia.

I tried libketama, seems pretty good in term of retention rate. I did some tests that could be (sort of) real world use case. Say, we have 4 weak (512M) nodes and want to replace them with all new nodes with double capacity (1G), I’m going to add new nodes to the cluster one by one, and then remove old nodes one by one, and here are what I got:

cluster capacity capacity
changed		 key moved
4x512M 2G 0% 0%
4x512M
1x1G		 3G 50% 40%
4x512M
2x1G		 4G 33% 30%
4x512M
3x1G		 5G 25% 25%
4x512M
4x1G		 6G 20% 20%
3x512M
4x1G		 5.5G 8% 12%
2x512M
4x1G		 5G 9% 13%
1x512M
4x1G		 4.5G 10% 18%
4x1G 4G 11% 19%

relatively, percentage of keys got moved to other partitions is close to capacity changes, which means it is close to the best number.

And key distribution is pretty even (capacity/utilization, node #1~#4 are 512M, #5~38 are 1G):

node #1 node #2 node #3 node #4 node #5 node #6 node #7 node #8
25.0%

25.6%

 25.0%

21.7%

 25.0%

24.7%

 25.0%

28.0%

 – – – –
16.7%

16.9%

 16.7%

15.2%

 16.7%

19.0%

 16.7%

17.7%

 33.3%

31.1%

 – – –
12.5%

13.5%

 12.5%

10.8%

 12.5%

13.7%

 12.5%

12.7%

 25.0%

24.5%

 25.0%

24.8%

 – –
10.0%

10.9%

 10.0%

9.4%

 10.0%

11.0%

 10.0%

8.3%

 20.0%

19.6%

 20.0%

20.0%

 20.0%

20.9%

 –
8.3%

8.9%

 8.3%

8.3%

 8.3%

8.1%

 8.3%

7.0%

 16.7%

16.7%

 16.7%

17.1%

 16.7%

17.9%

 16.7%

16.1%
– 9.1%

9.0%

 9.1%

9.6%

 9.1%

8.2%

 18.2%

17.5%

 18.2%

18.3%

 18.2%

19.8%

 18.2%

17.6%
– – 10.0%

9.7%

 10.0%

8.9%

 20.0%

20.3%

 20.0%

20.5%

 20.0%

21.9%

 20.0%

18.6%
– – – 11.1%

9.2%

 22.2%

22.3%

 22.2%

22.2%

 22.2%

25.2%

 22.2%

21.1%
– – – – 25.0%

24.2%

 25.0%

24.5%

 25.0%

27.2%

 25.0%

24.1%

I still need to try out fnv to see if it has better distribution and/or less key shakiness, from the article above it was said at least it has better performance.

NoSQL – something I’m thinking of

 Triviality  1 Response »
Oct 212009
 

I came across this article today, which talks about non-RDBMS solution for data storage (or a fancy name, data persistent layer).

It’s a great start point – I’ve been thinking about non-RDBMS solution for a while, but other than LDAP I don’t have anything else in my mind (weird enough, seems LDAP is not in the list, maybe it’s just too old to be “new tech”).

I will try out those projects mentioned in the article, and will post some data here if I get anything.

Test progress

 Triviality  No Responses »
Sep 252009
 

There was not much progress with performance test as mentioned here.

I first did test with LDAP vs. memcached, then found there was not much difference between those two, which is suspicious, so I started running test to compare pure database and memcached solution, which ideally should show lot of difference, but I failed.

I still haven’t figured out what was the real cause, it seems VM is not that good to do this kind of test – thinking of limited resource (CPU, mem, network, disk, etc), all VMs will compete each other. Again, this is my guess and I will try to see what would be the best solution to get performance done in VM environment.

Even if I failed, it shows that VM will not be good for performance testing, nice try ๐Ÿ™‚ .

ufw on Debian

 Triviality  7 Responses »
Sep 032009
 

Debian does not come with a good (I mean, easy to use :D) firewall, and to me ufw on Ubuntu is pretty cool so searched around, found this link:

http://blog.ropetin.com/?p=94

and copy some words here in case someday that URL becomes inaccessible:

# sudo apt-get install python
# wget http://people.ubuntu.com/~jamie/ufw/ufw-0.25.tar.gz
# tar -xzvf ufw-0.25.tar.gz
# cd ufw-0.25
# sudo python ./setup.py install

Once installed it needs to be configured, which really depends on your requirements.ย  A basic setup that allows SSH access, while blocking most other access can be configured by;

# sudo ufw logging on
# sudo ufw default deny
# sudo ufw allow proto tcp from any to any port 22
# sudo ufw enable

VMWare ESXi and VMWare vSphere

 Triviality  2 Responses »
Aug 312009
 

At the first glance VMWare ESXi is the right way to go for VM stuffs, but later on once I found out vSphere is actually not free, I got hit, after I reimaged the Vista box and turned it to a dedicated VM box.

After going through various sources, I decided to stay with ESXi for a while, because:

  1. It will be free to me for 60 days (well, another 58 days indeed)
  2. Xen is not that easy to be setup as I can see, need to learn more
  3. VMWare Go just announced today and I will take a try later on to see how it works

So far, running 6 VMs on a dual-core/3G mem machine is acceptable, I’m going to launch more tests on this box and will remove my VMs on Virtual box soon.

finished mysql replication

 Triviality  No Responses »
Aug 282009
 

Now I have 5 Linux VMs up and running with mysql replication configured and tested – this is a single master setup.

I did some research around and found that dual/multiple masters is still not a good solution thinking of the potential conflicts. It seems auto increment will not be a big issue as if end user does not have direct mysql access, then they will not be able to change auto_increment_increment and auto_increment_offset. However, it is still possible that two or more masters get updates for the same record at same or close time, which will cause a tricky question – what should be the last status of the record? It’s not totally rely on when the client (of mysql) triggered the update, it also depends on how fast the replication goes.

Some people/application may not care about this too much – such as user update profile information, if it’s user him/herself edit the record from two browsers and hit submit at almost the same time, it’s up to him/her to take the risk. However, this is extrem dangerous if this is a transaction system, that something to do with money, so again, it’s up to developer to decide using dual/multiple masters or not. If the application itself can guarantee different clients won’t write to same record at the same or close time (or better be never), such as data are geo-related, then multiple master is still a good solution.

Anyway, since dual/multiple masters won’t give me any benifit (no performance gain, no improvement on reliability), so I will leave the test to the future. For now, I will just leave the replication there (well, will add some monitoring stuffs) and go with mysql proxy test.

Also

 Older Entries  Newer Entries