Jul 192013

I was assigned to a Web project which is to present data analysis result to users. Original data came from Web log, plus some extra information, then headed to Hive, then populated statistics files after scientist’s analysis There are several interesting topics: geo graph, rendering another web page, and metrics graph, roughly speaking, I have no idea of any of these at the beginning of the project.

I think the best decision I made is to use GD based solution, actually that could be the only solution I can think about, I decided to use PHP for Web, plus Perl for batch processing, this actually seems not to be quite right as I’m migrating everything to PHP now as there is not much “real” batch processing, and everything could be done in shell. Also I decided to use server-side DOM model (read: PHP DOM) so not to slow down the project by my poor JS skills, actually my PHP skill is not that good but JS is definitely *poor*. Continue reading »

Jul 172013

Haven’t dig into legal issues yet, but for my hobby project I got everything regarding map from here:


After getting shp files from the web site, I use pyshp module:


To extract data to plain text format so that other programs can read it directly. There is an old version of pyshp comes with Ubuntu, but it’s sufficient to me.