Archive for the ‘SEO’ Category

Did You Get The IP Address Of That Spider?

I need not emphasize the importance of tracking the Search Engine spiders crawling record of your websites, need I? Whether you want to check the zeal and regularity of Search Engines in visiting your website, want to find out which specific pages were crawled, want to verify the obedience of crawlers towards your robots.txt file or you simply want to filter spider activity from your web metrics reports, you need to keep track of the various spiders and bots sent away to do the Search Engine’s dirty work. Provided here is a list of the IP addresses of the majority of Search Engine crawlers, for which I can’t take credit, which solely belongs to Dan Kramer of IPLists, whose efforts have made the world of Web 2.0 a better place. If you would like to do more than thank Dan for his efforts, you could buy him a drink.

Google IP List

Yahoo IP List

AltaVista IP List

Lycos IP List

Miscellaneous IP List

Non-SE Bots IP List

Tell the world about this post:
[Ask] [del.icio.us] [Digg] [Facebook] [Google] [LinkedIn] [Mister Wong] [Reddit] [Squidoo] [StumbleUpon] [Twitter] [Yahoo!] [Email] More »
Powered by Bookmarkify™

9 Oct 2009
Tags: , , , , ,
Posted in SEO | No Comments »