Posts filed under 'Search Engines'
Hot on the heels of my previous post “Google Takes the Robots Meta Tag Further“ comes news that Google has now added the ability to manipulate robots without a Robots Meta Tag or a robots.txt file. How? and what use can this possibly serve?
How? Is through the headers of a web page, and why? is because you may want to prevent Google from indexing documents such as PDF, XLS and DOC files served directly from your website.
To use the new X-Robots-Tag directive all you need to do is create a HTTP header in your script that serves the document. In PHP this would be like so:
<?php header(’X-Robots-Tag: unavailable_after: 25-Dec-2007 13:00:00 GMT’) ?>
The X-Robots-Tag also appears to support all the same commands as the standard robots meta tag (noIndex,noFollow,etc).
July 19th, 2007
Today is … Google Searchology day. What is Google Searchology day you may well ask. Google Searchology day is a day devoted for Google to discuss its future plans regarding search and search innovations. The main feature is an invitation only press conference which will be webcast at 9:30 PT (US Pacific Time).
The speaker list looks very impressive, with Marissa Mayer, VP of Search Products & User Experience; Udi Manber, VP of Engineering; and Craig Silverstein, Technology Director all presenting.
To read the full Google press release, click here
May 16th, 2007
I am constantly impressed by the state of the Search Engine Marketing industry in the UK, in particular the amount of research and ‘noise’ provided by the industry itself. In particular I was impressed with a recent report by Dylan Thwaites (and probably Jon Myers had a lot to do with it - with reference to the whitepaper itself referencing the author Jon Myers). That aside, a very insightful and interesting read,
The whitepaper itself focussed on recent research by the IAB/PWC which stated that 2.5% of all advertising spend in the US is spent on Search, compared against 11.4% in the UK. Dylan went onto state that it was his opinion that the UK online marketing industry is ‘the most advanced in the world’.
“The US market has roughly four times the volume of searches as the UK market and in bid economics should command higher prices for highly competitive phrases. It does not and this indicates a less competitive and less developed market state”.
Not mincing his word you may agree !!!!
The report concluded:
1. The search engines pay commission in the UK and this has had a firestarter effect on search penetration
2. The US market has a much higher proportion of in house search marketers, who to be less innovative and do not operate search as effectively as search agencies, slowing the flow of funds from other media
3. Many US search agencies reacted slowly to the paid search opportunity due to organic inertia
4. The UK market is more mature and more competitive than the US market and hence more intensely innovative
5. Cost per click prices are marginally higher in the UK, but there is no evidence to show that the engines are abusing their market dominance
Whilst I don’t necessarily agree with all the point raised in the report, points 1-3 within the above conclusions would seem to be on the ball from recent research we have conducted, however point 4 is slightly more contentious. In my opinion there are only a handful of agencies in the UK that can handle the large scale projects for the larger organisations, BigMouthMedia, Latitude, a couple of the WPP organisations to name a few. Whilst I don’t argue with his analysis of the recent MSN trials in France vs UK, whether this is comparable to the US market is debatable IMO
However on the whole, the report is well worth a read, and can be accessed by clicking here
May 4th, 2007
Been browsing round the results today, and noticed a couple of changes not noticed before.
1) It was only a matter of time before they included it, following the development of Google Checkout, however right next to the-direct-group.co.uk result was a banner for Google Checkout.
However just because they can I have to say, they shouldn’t. One of the best things about Google is the fact that it has always been impartial without lending weighting one way or another (apart from factors such as Bid Price/Quality Score etc), however this really is a break from the norm, and one that I wholeheartedly disagree with (no doubt this will lead to a rath of people redeveloping their e-commerce platforms to incorporate Google Checkout)
2) Google joins the Web2.0 Personalised Search club. Whilst doing a site:example.com on google noticed the ability to ‘note this’ on the results.
No doubt more to come from Google in the coming months
April 24th, 2007
Google has just annoumced a raft of new services to the Google webmaster tools, particularly tackling requests for page removal, directory or subdirectory removal or indeed complete site removal. Obviously many of these improvements tackle issues which may arise after your site has been indexed, however this is still a significant improvement in terms of functionality.
For sites that you’ve verified ownership for in your webmaster tools account, you’ll now see a new option under the Diagnostic tab called URL Removals. To get started, simply click the URL Removals link, then New Removal Request. Choose the option that matches the type of removal you’d like from the following options:
- Individual URL’s, web pages or images or other files
- A directory or sub directory on your site
- Your entire site
- Cached copy of a search result
Individual URLs, web pages, images or other files
Choose this option if you’d like to remove a URL or image. In order for the URL to be eligible for removal, one of the following must be true:
Obviously if all of the above do not apply, your page will only be reindexed again… common sense really
Removing a directory
Choose this option if you’d like to remove a directory or sub directory. As with the deletion of individual files, the directories should not be indexable (ie advised by your robots.txt file that they are not to be indexed.)
To remove a folder simply provide the path to the folder/directory you wish to remove ie:
For instance, if you request removal of the following:
http://www.example.co.uk/testfolder
type testfolder after the path provided in the Webmaster Tools interface.
Removal of an entire site
Choose this option only if you want to remove an entire site from the Google index. This option will remove all subdirectories and files.
To use this option, you must block the site using a robots.txt file.
Note:
Do not use this option to remove the non-preferred version of your site’s URLs from being indexed.
If you want all of your URLs indexed using the www version, specify the version you want indexed using the Preferred domain tool in the Webmaster Tools system(and do a 301 redirect to the preferred version, if possible).
Other changes introduced to ther Webmaster interface include:
- Advise of changing of page content
- Checking status of removal requests
- Advise to reinclude content
- Requesting removal of content you do not own
All in all this provides both Search engine marketing specialists and webmasters with a very useful set of tools in order to administer the indexation of their site more effectively.
To read the full google blogpost click here
Other reading:
Search Engine Land - Google introduces improved content removal toolsÂ
Managing Google results
April 18th, 2007
According to a recent report by E-Consultancy and search marketing firm, Neutralize (*\*), UK companies are seeing the benefits of paid search engine optimisation or SEO.
The report went on to state, that nearly two thirds of those surveyed had seen increases in excess of 300%, with fourty percent of those experiencing increases of over 500%. Unsuprisingly this has had a knock on effect in terms of increased confidence in UK SEO, and UK search engine marketing in general.
Other key findings in the report were
- 87% of search engine marketers said their organisations paid to advertise on Google, compared to 45% for Yahoo, 33% for MSN, 11% for MIVA and 6% for Ask.
- More than half of respondents (56%) said that their companies spend in excess of £10000 on paid search annually. A quarter of respondents (25%) reported spending more than £100000 annually.
- 62% plan to greatly increase investment in Search Engine Optimisation (SEO) for natural/organic listings.
- On average, respondants spent circa 32% of their marketing budget on online marketing.
April 16th, 2007
Google has recently announced some interesting new features to sitemaps.org. As well as now being available in 18 languages, including Japanese and German, but they have also made it easier to advise search engine robots of sitemap files.
By adding a line as follows:
Sitemap: http://www.e-gain.co.uk/sitemap.xml (including full URL ie http:// etc
to your robots.txt file, any search engine supporting the sitemap protocol will be able to retrieve the sitemap and process accordingly, including Google, Yahoo, MSN and ASK.
To read further please visit
April 12th, 2007
Google confirmed, Wednesday 4th Apr 2007, that the machines and workers at the 520-acre Berkeley County center site, 20 miles northwest of Charleston, South Carolina will be part of $600 million investment in the company’s worldwide network to handle Internet traffic.
Google hopes to have the first servers on line by year’s end, although the initial investment includes infrastructure that would allow the company to quickly expand if needed. The entire center should be operational within 18 months, said Rhett Weiss, the company’s senior team leader for global infrastructure.
“The company chose the site because it had water to cool the banks of computers as well as fiber optic cable connections, Weiss said. “The fiber optic connection is sort of the modern version of what Charleston traditionally was as a port city, The fiber optic network is sort of the import-export aspect of the project. That’s how the information is coming in and out.”
Andrew Johnson, who handles Google’s East Coast hardware operations, said the company does not reveal the size of the buildings or number of servers for competitive reasons.
In January, Google announced a similar investment in Lenoir, North Carolina and is also is looking at investing in another server farm near Blythewood on the outskirts of Columbia.
April 6th, 2007
The meta robots tag is a standard created over a decade ago, initially, to allow website authors to prevent page indexing by search engine spiders. Over the years this tag has evolved into a far more flexible meta tool.
Todays Robots Meta Tag Commands
- ALL - Sets all directives on. ALL = INDEX,FOLLOW
- NONE - Sets all directives off. NONE=NOINDEX,NOFOLLOW
- INDEX - An indexing robot should index the page.
- FOLLOW - A robot is to follow links on the page.
- NOINDEX - An indexing robot should not index the page.
- NOFOLLOW - Prevents an indexing robot from following any links on the page.
- NOARCHIVE - Prevents a cached copy of this page from being available in the search results.
- NOSNIPPET - Google only command that prevents a description from appearing below the page in the search results. Also enforces NOARCHIVE.
- NOODP - Stops the Open Directory Project description of the page from being used in the description in the search results and instead forces the search listing to use the DESCRIPTION meta data of the indexed page.
- NOYDIR - This tag works in a similar way to the NOODP tag and ensures that Yahoo will not use the title and description from your Yahoo Directory listing but instead is forced to display the meta TITLE and DESCRIPTION of the indexed page.
These commands can be issued using the robots meta tag, placed anywhere inside the head of your individual web page. Eg: <meta name=”ROBOTS” content=”NOINDEX”>
Multiple commands can be enforced by adding a comma seperated list to the content field of the robots meta tag. Eg: <meta name=”ROBOTS” content=”NOINDEX,NOFOLLOW,NOODP”>
The commands in the content fields MUST be comma seperated for your robots tag to be valid according to Google. These tags are also NOT case-sensitive.
Making the distinction between robots
All of the commands above can be targetted globally by using the command ROBOTS in the name field of the meta tag, but what if you want to tell each search engine something different?
You need to have a robots tag for each particular search engine you want to target. Each tag must identify the robot by name and list the command or commands pertinent to that particular robot. Eg:
<meta name=”GOOGLEBOT” content=”NOODP”>
<meta name=”MSNBOT” content=”NOFOLLOW”>
<meta name=”SLURP” content=”NOARCHIVE, NOYDIR”>
But what if I wanted to tell all the search engines one thing and just one search engine another thing?
You can have multiple tags in your page that target ROBOTS globally and individual robots by name. Google, Microsoft and Yahoo say that thier robots will honor both of these declarations. Eg:
<meta name=”ROBOTS” content=”NOARCHIVE”>
<meta name=”GOOGLEBOT” content=”NOODP”>
March 7th, 2007
I have read a lot recently regarding personalisation in particular from Google, and much has been debated within the SEO community itself on various forums such as SEW, Threadwatch and Cre8asite as to the ongoing effects of introducing personalisation.
In general, whilst the delivery mechanism of such a drastic change in focus will remain the overiding factor in determining whether such a change works to Google’s benefit (and delivers us consumers the results we expect), the effects of such change will provide us Search Engine Marketeers with new challenges.
However in my opinion such a change can only be good for the industry. Gone are going to be the days where results were based on the number of “number 1 terms” and campaigns will be judged by what they physically deliver to a client in terms of ROI, bringing it inline with other forms of advertising both off and online.
This should in turn make us search engine marketeers/engineers start thinking like more traditional marketeers, and rely on a variety of marketing techniques in order to ensure ROI on the websites. In my opinion not a bad thing, and I doubt that other forms of advertising will ignore those that can deliver such results. That and the use of PPC will probably increase in popularity as advertisers endeavour to ensure constant visibility in key areas.
March 6th, 2007
Next Posts
Previous Posts
