{"id":810,"date":"2009-12-08T12:10:38","date_gmt":"2009-12-08T19:10:38","guid":{"rendered":"http:\/\/xiehang.com\/blog\/?p=810"},"modified":"2009-12-08T12:10:38","modified_gmt":"2009-12-08T19:10:38","slug":"relationship-is-the-bottleneck","status":"publish","type":"post","link":"https:\/\/xiehang.com\/blog\/2009\/12\/08\/relationship-is-the-bottleneck\/","title":{"rendered":"Relationship is the bottleneck"},"content":{"rendered":"<p>Testing a prototype that uses Cassandra as the back end storage, the simple application is doing user authentication stuffs, it logs in and then get user&#8217;s profile and then show details on the web page.<\/p>\n<p>I hit performance problem with buddy related operation &#8211; every user may have 0~20 buddies, I want to show each buddy&#8217;s last login time on the result page, and actually I&#8217;ve retrieve everything for those buddies. The most direct implementation as I did first, is using user object to get data back, obviously this is not good as for every user object, client needs to access Cassandra cluster to get data back, the TCP round trip would be a pain.<\/p>\n<p>Then I added something to user object&#8217;s constructor, which load all buddies info in one shot (Cassandra&#8217;s multiget_slice API), things are getting better but this doesn&#8217;t seems reasonable to me as for most time, we don&#8217;t need buddy info (such as authentication), and getting buddies info back is just a waste of time.<\/p>\n<p>So I added a new method to the user class, called load_buddies, this will load buddies info on-demand. This makes authentication pretty fast, but still keep the ability of loading buddies info in batch mode.<\/p>\n<p>After all these the performance is &#8230; still not good, my test case is one login failure every ten requests, and for successfully logged user, I should buddy id and last access time, and also change the user&#8217;s last login time. The performance, with my current setting, the worst response time is about a second, while 90% request were done in less than 600ms.<\/p>\n<p>There must be something can be tuned, though VM could be the reason of slowness. I will check following stuffs:<\/p>\n<ul>\n<li>Apache HTTPd configuration, it seems prefork is performing better than worker, there may be more can be tuned include both HTTPd and wsgi<\/li>\n<li>Python class optimization, I will review the implementation of user class, as I don&#8217;t want to make user class too complicated to be used<\/li>\n<li>Cassandra performance, actually this is what I&#8217;m worrying about, as during the tests, Cassandra boxes&#8217; CPU utilization is about 80% &#8211; 70% on user, 10% on sys, roughly, it could be the bottleneck<\/li>\n<\/ul>\n<p>Without the buddy operation everything&#8217;s fine &#8211; the worst response time is about 600 ms while 90% requests are below 400ms. Relationship is a pain, it&#8217;s the bottleneck, but in this social era, there is no web application can live without relationship &#8230;<\/p>\n<p>BTW, my testing environment:<\/p>\n<ul>\n<li>Test client running on PowerBook, using ab &#8211; I will check if there is anything else can be useful<\/li>\n<li>Servers are all running on same physical box controlled by proxmox, this includes a web server, a LVS director (to load balance Cassandra nodes), and 3 Cassandra nodes<\/li>\n<li>The server box uses Ethernet, PowerBook is on wireless. I don&#8217;t think there is any issue for this as connect time is pretty low.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Testing a prototype that uses Cassandra as the back end storage, the simple application is doing user authentication stuffs, it logs in and then get user&#8217;s profile and then show details on the web page. I hit performance problem with buddy related operation &#8211; every user may have 0~20 buddies, I want to show each <a href='https:\/\/xiehang.com\/blog\/2009\/12\/08\/relationship-is-the-bottleneck\/' class='excerpt-more'>[&#8230;]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[47,13,133,16,27,28],"_links":{"self":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/810"}],"collection":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/comments?post=810"}],"version-history":[{"count":2,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/810\/revisions"}],"predecessor-version":[{"id":812,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/810\/revisions\/812"}],"wp:attachment":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/media?parent=810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/categories?post=810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/tags?post=810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}