WebmasterWorld Back In Google, Tabke Blogs for Bots

A few weeks ago, the web’s most popular webmaster and SEO forum, Webmasterworld.com, made an abrupt and complete exit from Google’s search index after owner Brett Tabke implemented a set of measures to keep robots and spiders from capturing the site’s content. One of these measures included a ban on all bots in the site’s robots.txt file. While Yahoo and MSN retained many of the site’s pages in their search index, Google offers webmasters a “removal” feature. A webmaster can exclude individual pages, sections of a site, or even all site content in the website’s robots.txt file, and then tell Google to check the robots.txt file and remove excluded content from its index. This is a useful feature if a webmaster discovers that Google has indexed confidential content, or if a site reorganization has left Google with incorrect pages in its index. Google’s security for this procedure is simple: only the owner of a site can edit its robots.txt file; there is no further test to see if the individual requesting the removal is, in fact, from the site itself. While Tabke had no intention of immediately purging all Webmasterworld.com pages from Google, as soon as the completely exclusionary robots.txt file was in place, various other individuals submitted removal requests. Google dutifully cleared its index of all the pages in a day or so.

Now, Webmasterworld.com pages are reappearing in Google. At the moment, Google is up to more than 300K pages indexed, and the site once again has a Google PageRank of 7..

In another amusing turn, Tabke has turned the Webmasterworld’s robots.txt file into a blog by putting in comment entries. Undoubtedly, in the last few weeks this has been the most-viewed robots.txt file ever, and adding commentary will keep this string going. Tabke also links to a script file that provides simple user agent-based delivery of variations on the robots.txt file.

I’m delighted to see WebmasterWorld back in Google. I can’t count the times when I have searched Google to solve a technical problem, only to end up being directed back to a thread at WMW. This is an experience no doubt shared daily by thousands of searchers, many of whom haven’t visited the site before. Every community needs a flow of new members, and what better source than topical searches? Welcome back!

Tweet This Post

Add this post to: del.icio.us - Digg it - Stumble it - Furl - Yahoo MyWeb No Comments so far
Leave a comment



Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

Name (required)

E-mail (required)

URI

Your Comment

(required)