Google improves its Robots.txt tools

Friday, August 24th, 2007 | Google, SEO, Search Industry

We’ve improved Webmaster Central’s robots.txt analysis tool to recognize sitemap declarations and relative urls. Earlier versions weren’t aware of sitemaps at all, and understood only absolute URLs; anything else was reported as Syntax not understood. The improved version now tells you whether your sitemap’s URL and scope are valid. You can also test against relative URLs with a lot less typing.

Reporting is better, too. You’ll now be told of multiple problems per line if they exist, unlike earlier versions which only reported the first problem encountered. And we’ve made other general improvements to analysis and validation.

Imagine that you’re responsible for the domain www.example.com and you want search engines to index everything on your site, except for your /images folder. You also want to make sure your sitemap gets noticed, so you save the following as your robots.txt file:

disalow images

user-agent: *
Disallow:

sitemap: http://www.example.com/sitemap.xml

You visit Webmaster Central to test your site against the robots.txtanalysis tool using these two test URLs:

Internet Marketing & SEO

Internet Marketing & Search Engine Optimization - IMSEO.ORG