{"id":1321,"date":"2012-04-24T22:36:17","date_gmt":"2012-04-25T05:36:17","guid":{"rendered":"http:\/\/xiehang.com\/blog\/?p=1321"},"modified":"2014-01-28T11:10:45","modified_gmt":"2014-01-28T18:10:45","slug":"spiders-good-and-bad","status":"publish","type":"post","link":"https:\/\/xiehang.com\/blog\/2012\/04\/24\/spiders-good-and-bad\/","title":{"rendered":"Spiders &#8211; good and bad"},"content":{"rendered":"<p>While I was migrating my web sites from AWS US east to US west, it&#8217;s interesting to see behavior of those spiders, good and bad.<\/p>\n<p>I set TTL of my domain to 30 minutes, and after 12 hours some spiders are still NOT going to my new sites:<!--more--><\/p>\n<ul>\n<li>Baidu &#8211; Mozilla\/5.0 (compatible; Baiduspider\/2.0; +http:\/\/www.baidu.com\/search\/spider.html)<\/li>\n<li>Netease &#8211; Mozilla\/5.0 (compatible;YoudaoFeedFetcher\/1.0;http:\/\/www.youdao.com\/help\/reader\/faq\/topic006\/;1 subscribers;)<\/li>\n<li>Yandex &#8211; Mozilla\/5.0 (compatible; YandexBot\/3.0; +http:\/\/yandex.com\/bots)<\/li>\n<li>Tencent &#8211; Sosospider+(+http:\/\/help.soso.com\/webspider.htm)<\/li>\n<\/ul>\n<p>and among them Baidu and Netease were doing really agressive crawling for whatever reason &#8230; am I update my blog too frequently? \ud83d\ude00<\/p>\n<p>And here is the good list:<\/p>\n<ul>\n<li>Google &#8211; Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)<\/li>\n<li>Google &#8211; Feedfetcher-Google; (+http:\/\/www.google.com\/feedfetcher.html; 1 subscribers; feed-id=13667814432325378539)<\/li>\n<li>Bing &#8211; Mozilla\/5.0 (compatible; bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm)<\/li>\n<li>What are these???\n<ul>\n<li>Mozilla\/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http:\/\/spinn3r.com\/robot) Gecko\/2010040121 Firefox\/3.0.19<\/li>\n<li>Mozilla\/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http:\/\/www.entireweb.com\/about\/search_tech\/speedy_spider\/)<\/li>\n<li>Mozilla\/5.0 (compatible; Ezooms\/1.0; ezooms.bot@gmail.com)<\/li>\n<li>BlogSearch\/2 +http:\/\/www.icerocket.com\/<\/li>\n<li>magpie-crawler\/1.1 (U; Linux amd64; en-GB; +http:\/\/www.brandwatch.net)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Weird thing is that I didn&#8217;t see Sohu and Yahoo over there, maybe Yahoo had stopped crawling (replace homegrown technique with Microsoft&#8217;s Bing?). Also, it seems Tencent has deployment issue as it&#8217;s crawling both the new and old site, seems like its crawlers are not in sync.<\/p>\n<p>Will watch for another day then will shutdown the old box.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>While I was migrating my web sites from AWS US east to US west, it&#8217;s interesting to see behavior of those spiders, good and bad. I set TTL of my domain to 30 minutes, and after 12 hours some spiders are still NOT going to my new sites:<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[328,329,330],"_links":{"self":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/1321"}],"collection":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/comments?post=1321"}],"version-history":[{"count":3,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/1321\/revisions"}],"predecessor-version":[{"id":1659,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/posts\/1321\/revisions\/1659"}],"wp:attachment":[{"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/media?parent=1321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/categories?post=1321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xiehang.com\/blog\/wp-json\/wp\/v2\/tags?post=1321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}