{"id":1147,"date":"2011-11-17T08:46:09","date_gmt":"2011-11-17T15:46:09","guid":{"rendered":"http:\/\/xiehang.com\/blog\/?p=1147"},"modified":"2011-11-17T08:46:09","modified_gmt":"2011-11-17T15:46:09","slug":"playing-with-hive","status":"publish","type":"post","link":"https:\/\/xiehang.com\/blog\/2011\/11\/17\/playing-with-hive\/","title":{"rendered":"Playing with Hive"},"content":{"rendered":"<p>I got 700K lines of apache log files from a friend&#8217;s web server and imported them to the testing Hadoop instance running on my MacBoox, following the instructions listed <a href=\"https:\/\/cwiki.apache.org\/confluence\/display\/Hive\/GettingStarted\">here<\/a> I successfully run some analysis.<\/p>\n<p>Note, the last section in the article talking about the <\/i>Apache Weblog Data<\/i> doesn&#8217;t seem to be correct &#8211; it lacks of some space (&#8221; &#8220;) after ^ and it gave me quite some headache since I&#8217;m not familiar with Java regular expressions. Luckily <a href=\"https:\/\/issues.apache.org\/jira\/browse\/HIVE-662\">Hive issue 662<\/a> mentioned in the article gave me the correct regex to get things done.<\/p>\n<p>It seems I can only learn to play with Hive\/Hadoop cos Hadoop running on MacBook is still a single node installation which is &#8230; <b>SLOW<\/b>, but so far I&#8217;m fine with it as I don&#8217;t have high volume of data to be processed. As a reference, getting top accessed IPs (which I used to figure out potential abusers) took 83 seconds. The HSQL is simple, something like &#8220;select host, count(*) cc from apachelog group by host order by cc desc limit 10;&#8221;.<\/p>\n<p>Hadoop is a richmen&#8217;s game, seems it only improve the performance whenever you have lots of nodes as it can well distributed tasks.<\/p>\n<p>BTW, <i>Hadoop: The Definitive Guide<\/i> is a good book \ud83d\ude42 .<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I got 700K lines of apache log files from a friend&#8217;s web server and imported them to the testing Hadoop instance running on my MacBoox, following the instructions listed here I successfully run some analysis. Note, the last section in the article talking about the Apache Weblog Data doesn&#8217;t seem to be correct &#8211; it <a href='https:\/\/xiehang.com\/blog\/2011\/11\/17\/playing-with-hive\/' class='excerpt-more'>[&#8230;]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[82,230,240,231,239],"_links":{"self":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/1147"}],"collection":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/comments?post=1147"}],"version-history":[{"count":1,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/1147\/revisions"}],"predecessor-version":[{"id":1148,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/1147\/revisions\/1148"}],"wp:attachment":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/media?parent=1147"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/categories?post=1147"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/tags?post=1147"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}