The X-Robots-Tag : Control Those Robots, Without a Meta, the Google Way

Hot on the heels of my previous post “Google Takes the Robots Meta Tag Further“ comes news that Google has now added the ability to manipulate robots without a Robots Meta Tag or a robots.txt file. How? and what use can this possibly serve?

How? Is through the headers of a web page, and why? is because you may want to prevent Google from indexing documents such as PDF, XLS and DOC files served directly from your website.

To use the new X-Robots-Tag directive all you need to do is create a HTTP header in your script that serves the document. In PHP this would be like so:

<?php header(’X-Robots-Tag: unavailable_after: 25-Dec-2007 13:00:00 GMT’) ?>

The X-Robots-Tag also appears to support all the same commands as the standard robots meta tag (noIndex,noFollow,etc).

Add comment July 19th, 2007