Relationship is the bottleneck

Testing a prototype that uses Cassandra as the back end storage, the simple application is doing user authentication stuffs, it logs in and then get user’s profile and then show details on the web page.

I hit performance problem with buddy related operation – every user may have 0~20 buddies, I want to show each buddy’s last login time on the result page, and actually I’ve retrieve everything for those buddies. The most direct implementation as I did first, is using user object to get data back, obviously this is not good as for every user object, client needs to access Cassandra cluster to get data back, the TCP round trip would be a pain.

Then I added something to user object’s constructor, which load all buddies info in one shot (Cassandra’s multiget_slice API), things are getting better but this doesn’t seems reasonable to me as for most time, we don’t need buddy info (such as authentication), and getting buddies info back is just a waste of time.

So I added a new method to the user class, called load_buddies, this will load buddies info on-demand. This makes authentication pretty fast, but still keep the ability of loading buddies info in batch mode.

After all these the performance is … still not good, my test case is one login failure every ten requests, and for successfully logged user, I should buddy id and last access time, and also change the user’s last login time. The performance, with my current setting, the worst response time is about a second, while 90% request were done in less than 600ms.

There must be something can be tuned, though VM could be the reason of slowness. I will check following stuffs:

• Apache HTTPd configuration, it seems prefork is performing better than worker, there may be more can be tuned include both HTTPd and wsgi
• Python class optimization, I will review the implementation of user class, as I don’t want to make user class too complicated to be used
• Cassandra performance, actually this is what I’m worrying about, as during the tests, Cassandra boxes’ CPU utilization is about 80% – 70% on user, 10% on sys, roughly, it could be the bottleneck

Without the buddy operation everything’s fine – the worst response time is about 600 ms while 90% requests are below 400ms. Relationship is a pain, it’s the bottleneck, but in this social era, there is no web application can live without relationship …

BTW, my testing environment:

• Test client running on PowerBook, using ab – I will check if there is anything else can be useful
• Servers are all running on same physical box controlled by proxmox, this includes a web server, a LVS director (to load balance Cassandra nodes), and 3 Cassandra nodes
• The server box uses Ethernet, PowerBook is on wireless. I don’t think there is any issue for this as connect time is pretty low.

Moving the PowerBook back as a … PowerBook

Since proxmox gives me great VM performance, I’m moving PowerBook (was running Fedora) back to be a PowerBoox, running OSX 10.5.

It will be a client box still, but it will not be a dedicated dev related box.

How to change uid on OS X Leopard

Seems the PowerBook is the last Unix system that does not have the right uid for myself, but obviously changing uid is totally different from Linux/FreeBSD.

Here is the source and it tells using dscl, and found dscl can do almost all user/group management, and could be more, but I don’t need more .

Hello world is always easy

Finished HelloWorld with Objective-C on Mac using Xcode, everything seems fine, though I know hello world is always simple and easy.

It seems to me Objective-C is something similar to smalltalk, but I’m not quite sure as I’m not familiar with smalltalk either. Anyway the syntax is sort of weird to me, as it does not like a procedure/function oriented language, but more like a functional one.

I will keep play with Xcode for sometime during this long weekend, and back to my PPCam (Ada on Windows using GNAT) next week – again, it is not just as simple as hello world, get to spend more time to get things done.

Which application type is suitable for Mac?

I don’t quite understand application types under Mac (OS X), I remember I stopped trying out XCode just because I cannot determine which one to use, and I also made wrong assumption that Java is the way to go as I can share (some) source codes cross platforms.

Now I got a book talking about how to build Mac applications with XCode, and it clearly stated:

For Carbon:

The Carbon template defines a pure Carbon application. Carbon is the C interfaces to Aqua and the legacy support for applications written using the APIs in Mac OS 9 and earlier. Use this template if you are porting an application from OS 9 to OS X, or you want to create native applications using only C or C++. The main.c file includes a skeletal application that loads the main.nib file, creates the application’s window, and starts the main event loop. See http://developer.apple.com/carbon/ for more information about Carbon applications.

For Java/Cocoa:

Mac OS X includes an extensive bridge between the Java programming language and the Objective-C APIs defined in the Cocoa framework. These Java proxy objects make it possible to use most of the Cocoa and Foundation frameworks as though they had been written in Java. If you need to create a native Cocoa application and prefer, or require, the Java programming language, create a Cocoa-Java application. The template sets up a Cocoa application whose native executable is the Cocoa-Java stub. This stub application creates a Java runtime, that loads and starts the Java classes which constitute the application. Read more about mixing Java and Cocoa at http://developer.apple.com/documentation/Cocoa/Java-date.html. Apple has recently announced that it will no longer be updating in the Cocoa-Java bridge beyond Mac OS X 10.3, so the future of this technology is murky.

For Cocoa:

Cocoa is the preferred application development technology for Mac OS X. It is based on the Objective-C programming language. The Cocoa framework is the Objective-C interfaces to Aqua and most other Apple technologies. If OS X has a “native” development environment, Objective-C and Cocoa are it. A few technologies are only available using Cocoa interfaces. The Cocoa template creates a minimal Cocoa application. The only functional part of the project is the main.m file which does nothing but create and start the generic NSApplication object. It does provide generic handling of the About This Program menu item, but it does not include a Credits.rtf file.

From three paragraphs above, it is clearly Java is NOT the way to go, it will be dropped by Apple sometime in the future, also Carbon is for legacy applications.

So, the right way is indeed doing Cocoa application with Objective-C, I get to read around to see if I can get some tests done.

I’ve upgraded my PowerBook months back with latest OS X (Leopard), plus makek it 2G memory, so that I’ve been able to install XCode, however, the machine had been idle since the upgrade, I should do something so not to waste my money (I have enough time now, not using my time is just a waste).

LogMeIn.com

Got this site from co-worker, pretty good. Now I need to test the speed of connecting from China, and once it works I will install it on my parents’ computer so that I can help checking their computer problems once there is any.

Better than remote desktop or VNC, as it does not need port forwarding nor firewall holes.