The Robots Meta Tag Explained

March 7th, 2007

The meta robots tag is a standard created over a decade ago, initially, to allow website authors to prevent page indexing by search engine spiders. Over the years this tag has evolved into a far more flexible meta tool.

Todays Robots Meta Tag Commands

• ALL - Sets all directives on. ALL = INDEX,FOLLOW
• NONE - Sets all directives off. NONE=NOINDEX,NOFOLLOW
• INDEX - An indexing robot should index the page.
• FOLLOW - A robot is to follow links on the page.
• NOINDEX - An indexing robot should not index the page.
• NOFOLLOW - Prevents an indexing robot from following any links on the page.
• NOARCHIVE - Prevents a cached copy of this page from being available in the search results.
• NOSNIPPET - Google only command that prevents a description from appearing below the page in the search results. Also enforces NOARCHIVE.
• NOODP - Stops the Open Directory Project description of the page from being used in the description in the search results and instead forces the search listing to use the DESCRIPTION meta data of the indexed page.
• NOYDIR - This tag works in a similar way to the NOODP tag and ensures that Yahoo will not use the title and description from your Yahoo Directory listing but instead is forced to display the meta TITLE and DESCRIPTION of the indexed page.

These commands can be issued using the robots meta tag, placed anywhere inside the head of your individual web page. Eg: <meta name=”ROBOTS” content=”NOINDEX”>

Multiple commands can be enforced by adding a comma seperated list to the content field of the robots meta tag. Eg: <meta name=”ROBOTS” content=”NOINDEX,NOFOLLOW,NOODP”>

The commands in the content fields MUST be comma seperated for your robots tag to be valid according to Google. These tags are also NOT case-sensitive.

Making the distinction between robots

All of the commands above can be targetted globally by using the command ROBOTS in the name field of the meta tag, but what if you want to tell each search engine something different?

You need to have a robots tag for each particular search engine you want to target. Each tag must identify the robot by name and list the command or commands pertinent to that particular robot. Eg:

<meta name=”GOOGLEBOT” content=”NOODP”>
<meta name=”MSNBOT” content=”NOFOLLOW”>
<meta name=”SLURP” content=”NOARCHIVE, NOYDIR”>

But what if I wanted to tell all the search engines one thing and just one search engine another thing?

You can have multiple tags in your page that target ROBOTS globally and individual robots by name. Google, Microsoft and Yahoo say that thier robots will honor both of these declarations. Eg:

<meta name=”ROBOTS” content=”NOARCHIVE”>
<meta name=”GOOGLEBOT” content=”NOODP”>

Entry Filed under: SEO Industry, Search Engines