The X-Robots-Tag : Control Those Robots, Without a Meta, the Google Way

July 19th, 2007

Hot on the heels of my previous post “Google Takes the Robots Meta Tag Further“ comes news that Google has now added the ability to manipulate robots without a Robots Meta Tag or a robots.txt file. How? and what use can this possibly serve?

How? Is through the headers of a web page, and why? is because you may want to prevent Google from indexing documents such as PDF, XLS and DOC files served directly from your website.

To use the new X-Robots-Tag directive all you need to do is create a HTTP header in your script that serves the document. In PHP this would be like so:

<?php header(’X-Robots-Tag: unavailable_after: 25-Dec-2007 13:00:00 GMT’) ?>

The X-Robots-Tag also appears to support all the same commands as the standard robots meta tag (noIndex,noFollow,etc).

Entry Filed under: Search Engines, News and Views