Internet Marketing & SEO

Yahoo! Search Support for X-Robots-Tag Directive to Simplify Control

Wednesday, December 12th, 2007 | SEO, Search Industry, Yahoo

Today we’re announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we’re extending our support of page level exclusion tags — NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW — to provide additional control for archiving and summarization of ANY file type. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. ), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Additionally, webmasters no longer need access to html templates in order to express exclusions for html files. To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:

X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results.
Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.

X-Robots-Tag: NOARCHIVE — If you don’t want to display cache link in the search results page.

X-Robots-Tag: NOSNIPPET — If you don’t want to display summary in the search results page.

X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.

Along with this change, we’ll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process.

We’re at SES in Chicago and WebmasterWorld’s PubCon in Las Vegas, participating in a few different panels this week. Please find us if you have any questions or suggestions or drop us your feedback here.

Sharad Verma
Yahoo! Search

from:Yahoo! Search Blog

1 Comment to Yahoo! Search Support for X-Robots-Tag Directive to Simplify Control

Leave a comment