Rand, en este video, presenta varios aspectos a considerar al lanzar un blog corporativo. Me ha parecido un video interesante. Os lo recomiento !
SEOmoz Whiteboard Friday - Corporate Blogging Tips from Scott Willoughby on Vimeo.
Os adjunto algunos links publicados recientemente en el blog de “Webmaster Central”.
Duplicate Content:
http://googlewebmastercentral.blogspot.com/2008/09/demystifying-duplicate-content-penalty.html
Creating a site:
http://www.google.com/support/webmasters/bin/topic.py?topic=8522
Deleting or blocking content via meta tags:
http://www.google.com/support/webmasters/bin/answer.py?answer=61050
Es interesante visualizar también el video:
No es nada espectacular… obviamente, pero me ha hecho ilusión.
Después de bastante tiempo… al final se ha decidido 
l único problema es que la selección no ha sido de lo mas inteligente. Tendré que ayudarle un poco mediante las webmaster Tools.
No es muy normal ver que productos de Google sufran problemas, caídas o similares.
Pero Google tampoco es perfecto, y hoy… ha estado sirviendo analytics sin estilos
Este snapshot queda ya aquí para la posteridad
Una herramienta indispensable !! Google recientemente ha anunciado la puesta en producción de una nueva herramienta llamada “Google Inside Search”.
La verdad es que hay que aplaudir a la gente de Google. Parecido a google trends, google ahora ofrece la posibilidad de analizar términos de búsqueda de manera detallada, por localidades y/o regiones y en cualquier parte del mundo.
Sinceramente, de obligada consulta en el momento de definir cualquier tipo de estratégia ya sea SEO o SEM.
En general, en España, no prestamos demasiada importáncia al posicionamiento en Yahoo o MSN, puesto que la cuota de mercado que estos buscadores tienen no es demasiado relevante.
En otros paises, esto no es tan claro.. y Yahoo por ejemplo tiene mas importancia.
En estos casos, una buena estratégia para posicionarse también en Yahoo es muy importante.
Os adjunto 3 recursos de obligada lectura si os encontrais en esta situación:
http://www.beanstalk-inc.com/articles/seo/seo-for-yahoo.htm
http://www.webpronews.com/expertarticles/2006/10/11/yahoo-seo-techniques
http://www.seobook.com/relevancy/
Hola -
Matt Cutts ha anunciado una actualización de pr inminente, y la expiración de algunas penalizaciones aplicadas en el pasado. Vamos a ver que cambios produce en sus índices, y sobre todo… que cambios en los posicionamientos genera.
Si teneis algún portal afectado por alguna penalización… estad atentos
Os adjunto el comentario original:
New Toolbar PageRanks coming
Hey folks, I wanted to let you know that new toolbar PageRank values should become visible over the next few days. I’m expecting that also in the next few days that we’ll be expiring some older penalties on websites.
Como ya debereis saber, google ha modificado sus algoritmos de crawling para entender (o intentar entender el todopoderoso flash que tantos dolores de cabeza traen.
Os adjunto un articulo de Vanesa Fox hablando de algunas recomendaciones. A mi modo de ver… es de obligada lectura !!
A couple of weeks ago, Adobe announced that it was working with Google and Yahoo! on making Flash content easier to index in search engines. Google said it was using the search-engine specific Flash player that Adobe had made available (Yahoo!’s integration is still in the works). While I think it’s great and absolutely vital that search engines continue to evolve beyond strictly text (to ensure they are providing the best possible experience for their users), I don’t think this announcement means that all the Flash content on the web will now suddenly start ranking in search results and I don’t think that Flash developers can stop thinking about search engine optimization.
How search engines work
It all goes back to how search engines work. At least for now (even with all of the advancements in the last year around universal search), the foundations of the major search engines are based on text. The web began with primarily text-only pages and the search engine algorithms were built on that idea. When people started searching for information, they searched with words. We’re used to asking for things in words, after all, and since words were what the web was made up of, the questions and answers matched up quite well. Search engines are a bit of a middleman (middlemachine?) between a searcher’s textual questions and a web site’s textual answers.Searching continues to be text based
Sure, you might imagine other types of exchanges. I might want to upload a picture of a person and ask for all the other pictures on the web of that person. Or I might want to search through the audio of a song for a particular lyric. All of those types of searches and more are coming (and some have been tried, with varying degrees of success), but at least for now, those applications are not how the three major search engines work and not how most people search.Over time, search engines have experimented with different elements on pages beyond simply the text itself to better understand what those pages are about. Although since these experiments are built on a text-based foundation, the experiments have also still mostly focused on text. For instance, search engines found that the text that’s in the title may be a strong indicator of the focus on the page. The textual caption under and image is likely describing that image.
How Flash fits in with text-based search engines
Now, consider Flash. Most Flash pages contain little text. Those that do could often just as easily display that text outside of the Flash components (which would make it easier for those on screen readers and mobile phones, for instance, to view the content).With this latest innovation in crawling Flash, Google can more easily access the text in Flash, but they still can’t process it quite as well as it can HTML text because they aren’t extracting any meta data about that text. As I mentioned earlier, search engines are now storing all kinds of meta data based on the structure of the text in HTML, like if it’s in a title tag, or an H1 and so on. So Flash-based text has that disadvantage.
Provide a separate URL for each piece of Flash content
Another consideration is how the Flash application itself is constructed. This new Flash player that Adobe is making available to Google and Yahoo! helps the search engines in that it enables them to access content it never could before. The crawlers can interact with the Flash application as a user would and crawl deeper into the application to get to text that may be four or five levels deep. On first glance, this may seem similar to search engine crawlers following links within HTML sites, but it can actually be quite different.HTML pages (generally) have unique URLs for each page. Flash applications can be constructed that way, but can also be constructed so that as you go deeper into the application, the URL doesn’t change. This can be problematic for lots of usability reasons that have nothing to do with search. For instance, the back button in the browser doesn’t work. Users can’t easily email, Digg, or otherwise share a particular section of the Flash application easily. Bookmarking only works for the beginning of the Flash app.
As you might imagine, it also causes problems in search. Sure, the search engine crawlers may now be able to get to some of that content several levels in, but they have to index all of the text under a single URL. (Also note that they likely won’t index all of the application in this case; they will execute only a certain number of interactions.)
Say information about your latest product line is available once you choose “products” from the home page, then “new” from the products page, then “coming soon” from the new page. If the URL of the application doesn’t change for each interaction, then search engines will have to index the content from the home page, products page, new page, and coming soon page all under a single URL. When a searcher looks for your latest product line, that URL may appear in the results. But once the searcher clicks over, they aren’t brought to your coming soon page, they see your home page, and may have no idea where to go from there. If you ensure your Flash app uses a different URL for each page, then the searcher can be brought directly to the page that has the right content, which should greatly improve conversion rates and lower bounce rates.
But if you take the announcement that Google can now index Flash at face value, without looking deeper, you may not realize this, and think that your single-URL Flash application is now perfectly positioned for search.
Taking back the tour
Want an example of how the statement “Google can now index Flash” isn’t the whole story?I’ve been watching the Tour de France. It’s playing on the Versus network for the first time this year. I’d never heard of the Versus network before (since it seems to mostly show ultimate fighting cage matches, this may be because I’m not its target audience; not to mention that I wasn’t the target audience for the network under its previous name, OLN, as I think it mostly played shows about people fishing then), and the network is looking to capitalize on this potential new audience.
Versus is spending a lot of money on its Tour de France campaign “Take Back the Tour”. It has put together flashy commercials and an equally flashy website.
Versus probably would like to be found when people search for [tour de france]. The Tour de France page on the main versus.com domain shows up in the search results, but the Take Back The Tour site that they spent so money money on? Nowhere to be found.
Well, they’re spending all the money on commercials and print ads, so maybe people have been searching for [take back the tour] as well. The site does rank #1 for that query on both Google and Live (although it’s down at #8 on Yahoo!). For all three engines, even those who do the search because they saw an ad might not be sure if the takebackthetour.com listing is really the official site based on how the listing looks in the search results.
You can see that at this point, Google doesn’t see any content on the site and in fact, notes on the cached page that [take back the tour] appears only in links pointing to the page. Since it can’t extract any text, it has no way of knowing that the site is about the Tour de France.
Google still doesn’t Flash executed via JavaScript
So. What’s the problem? Google crawls Flash now and all should be well. I see at least two problems. The first is fundamental. The Flash executes via JavaScript. Google noted in their blog post that:“Googlebot does not execute some types of JavaScript. So if your web page loads a Flash file via JavaScript, Google may not be aware of that Flash file, in which case it will not be indexed.”
They did update the post later to say that:
“For our July 1st launch, we didn’t enable Flash indexing for Flash files embedded via SWFObject. We’re now rolling out an update that enables support for common JavaScript techniques for embedding Flash, including SWFObject and SWFObject2.”
Will this update help the Take Back the Tour site? Maybe not.
Can Google find any words to index?
Another big obstacle to the crawl of this site is that even if Google could get to the Flash, it would find few words to index. Nearly all of the text on the site is contained in images. The first thing you see when you go to the site is lots of words, but the only ones that seem to be text, rather than part of the image, are in the link “join the movement”.So, once Google can access the Flash, it will be able to crawl and index those words. This design is a theme throughout the site. Links like “back” are text. Nearly everything else is in images.
Let’s pretend for a moment that they changed the Flash file so that the text wasn’t contained in images (and that the JavaScript problem didn’t exist). Would this help indexing? Yes and no.
No separate URLs can lead to a poor experience for searchers
Each time you click a link in the Flash file, you are taken to another page, but the URL doesn’t change. It stays at takebackthetour.com no matter how you navigate. That means that any text Google does pick up will be indexed under that one URL.By clicking about three levels deep, I can find TV spots about the tour. If the site designers added some text about those TV spots, using the language of their customers, then searchers looking for [tour de france video] or something similar might see the takebackthetour.com site come up in their search results. But when they clicked through to the site, they wouldn’t see the TV spots. They would see the Flash splash page. And they would have to figure out how to navigate through the site to find the video section. Chances are that many searchers would scan the initial page that came up, not see what they were looking for and go back to the search results to find another site.
Little change for viral success
This makes for a poor user experience from search, but consider also that the creators of this campaign obviously are hoping it goes viral. If you want a site to go viral, you have to make it easily shareable. Sure, people may love the rant section or the video section or the contest, but no URL of any of these sections exists for those people to email, Digg, Twitter, Stumble, or otherwise share. A viral campaign that requires every person who shares the content to say, “go to this URL, then click ‘join the movement’, then click ‘how will you take back the tour’ is over before it even begins.And what about accessibility? And those on the go? I watched the first night of the tour at a friend’s house. What if I had seen the commercial, wanted to check it out, and pulled up the site on my Windows Mobile Smartphone? I would have had this awesome experience:
It’s not even an accurate error message, since the first problem is that I don’t have JavaScript support.
Be smart about Flash
Clearly, a few problems still exist with Flash websites. My view is this:
- It’s important for web technology providers to think about things like accessibility and search engine optimization or those who implement those technologies will turn to other solutions. To this end, Adobe should be commended for continuing to evolve their offerings to better serve the needs of their users.
- Search engines have to continue to evolve beyond HTML as their primary goal is to provide the best possible results for searchers. They can’t rely on site owners across the web understanding what technologies are better for search. Google is clearly working on “organizing all the world’s information”, not just all the information well optimized for search engines, and this latest Flash development is an important part of that evolution.
- If you operate a business online, search is an important acquisition channel. Don’t leave such an important avenue for gaining new customers in the hands of others. Ensure that you are making it as easy as possible for search engines to find your content.
- Flash may very well be a great technology for your site, but implement it wisely.
Gracias a “Lost Remote”, he encontrado este post de un compañero que encontró un interesante experimento de Google en crowdsourced SERP personalización, lo que ellos llaman “Modificar Resultados de la búsqueda”.
Desde la sección de preguntas frecuentes:
Esta característica le permite influir en su experiencia de búsqueda de añadir, mover y eliminar los resultados de la búsqueda. Cuando usted busca para las mismas palabras clave de nuevo, mientras usted está conectado a su cuenta de Google, podrá continuar para ver los cambios. Si más adelante desea volver a sus cambios, puede deshacer cualquier modificación que hayas realizado.
Nota: Esta es una característica experimental que sirve a una selección aleatoria de los participantes y pueden estar disponibles sólo para unas pocas semanas.
Interesante…
Os adjunto un contenido muy bueno, extraido directamente de la fuente original.
De recomendable lectura !!
Controlling what content is blocked from being found in search engines is crucial for many websites. Fortunately, the major search engines and other well-behaved robots observe the Robots Exclusion Protocol (REP), which has evolved organically since the early 1990’s to provide a set of controls over what parts of a web site search engines robots can crawl and index.
Article Sections:
The Robots Exclusion Protocol provides controls that can be applied at the site level (robots.txt), at the page level (META tag, or X-Robots-Tag), or at the HTML element level to control both the crawl of your site and the way it’s listed in the search engine results pages (SERPs). Below is a table listing the common scenarios, directives, and which search engines support them.
| Use Case | Robots.txt | META/ X-Robots-Tag | Other | Supported By |
| Allow access to your content | Allow | FOLLOW INDEX |
Google Yahoo Microsoft |
|
| Disallow access to your content | Disallow | NOINDEX NOFOLLOW |
Google Yahoo Microsoft |
|
| Disallow access to index images on the page | NOIMAGEINDEX | |||
| Disallow the display of a cached version of your content in the SERP | NOARCHIVE | Google Yahoo Microsoft |
||
| Disallow the creation of a description for this content in the SERP | NOSNIPPET | Google Yahoo Microsoft |
||
| Disallow the translation of your content into other languages | NOTRANSLATE | |||
| Do not follow or give weight to links within this content | NOFOLLOW | a href attribute: rel=NOFOLLOW |
Google Yahoo Microsoft |
|
| Do not use the Open Directory Project (ODP) to create descriptions for your content in the SERP | NOODP | Google Yahoo Microsoft |
||
| Do not use the Yahoo Directory to create descriptions for your content in the SERP | NOYDIR | Yahoo | ||
| Do not index this specific element within an HTML page | class=robots-nocontent | Yahoo | ||
| Stop indexing this content after a specific date | UNAVAILABLE_AFTER | |||
| Specify a sitemap file or a sitemap index file | Sitemap | Google Yahoo Microsoft |
||
| Specify how frequently a crawler may access your website | Crawl-Delay | Google WMT | Yahoo Microsoft |
|
| Authenticate the identity of the crawler | Reverse DNS Lookup | Google Yahoo Microsoft |
||
| Request removal of your content from the engine’s index | Google WMT Yahoo SE Microsoft WMT |
Google Yahoo Microsoft |
One of the first steps in managing the robots is knowing what type of content should be public vs. private. Start with the assumption that by default, everything is public, then explicitly identify the items that are private.
If you want search engines to access all the content on your site, you don’t need a robots.txt file at all. When a search engine tries to access the robots.txt file on your site and the server can’t return one (ideally by returning a 404 HTTP status code), the search engine treats this the same as a robots.txt file that allows access to everything.
Every website and every business has a different set of needs, so there’s no blanket rule for what to make private, but some common elements may apply.
REP is flexible and can be implemented a number of ways. This flexibility lets you easily specify some policies for your entire site (or subdomain) and then enhance them more granularly at the page or link level as needed.
Site wide directives are stored in a robots.txt file, which must be located in the root directory of each domain or sub-domain. (e.g. http://janeandrobot.com/robots.txt). Robots.txt files located in subfolders are ignored.
A robots.txt file is a UTF-8 encoded file that contains entries that consist of a user-agent line (that tells the search engine robot if the entry is directed at it) and one or more directives that specify content that the search engine robot is blocked from crawling or indexing. A simple robots.txt file is shown below.
User-agent: *
Disallow: /private
user-agent: - Specifies which robots the entry applies to.
* to specify that this entry applies to all search engine robots.user-agent: * (rather than in addition to those entries).The major search engines have multiple robots that crawl the web for different types of content (such as images or mobile). They generally begin all robots with the same name so that if you block the major robot, all robots for that search engine are blocked as well. However, if you want to block only the more specific robot, you can block it directly and still allow web crawl access.
Disallow: - Specifies what content is blocked
/)./. For instance, Disallow: /images blocks access to /images/, /images/image1.jpg, and /images10.You can specify other rules for search engine robots in addition to the standard instructions that block access to content as noted in other robot instructions.
Some things to note about robots.txt implementation:
Disallow: /images would block http://www.example.com/images but not http://www.example.com/Images.Block all robots - Useful when your site is in pre-launch development and isn’t ready for search traffic.
# This keeps out all well-behaved robots.
# Disallow: * is not valid.
User-agent: *
Disallow: /
Keep out all bots by default - Blocks all pages except those specified. Not recommended as is difficult to maintain and diagnose.
# Stay out unless otherwise stated
User-agent: *
Disallow: /
Allow: /Public/
Allow: /articles/
Allow: /images/
Block specific content - The most common usage of robots.txt.
# Block access to the images folder
User-agent: *Disallow: /images/
Allow specific content - Block a folder, but allow access to selected pages in that folder.
# Block everything in the images folder
# Except allow images/image1.jpg
User-agent: *
Disallow: /images/
Allow: /images/image1.jpg
Allow specific robot - Block a class of robots (for instance, Googlebot), but allow a specific bot in that class (for instance, Googlebot-Mobile).
# Block Googlebot access
# Allow Googlebot-Mobile access
User-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
The major engines support two types of pattern matching.
Block access to URLs that contain a set of characters - Use the asterisk (*) to specify a wildcard.
# Block access to all URLs that include an ampersand
User-agent: *
Disallow: /*&
This directive would block search engines from crawling http://www.example.com/page1.asp?id=5&sessionid=xyz.
Block access to URLs that end with a set of characters - Use the dollar sign ($) to specify end of line.
# Block access to all URLs that end in .cgi
User-agent: *
Disallow: /*.cgi$
This directive would block search engines from crawling http://www.example.com/script1.cgi but not from crawling http://www.example.com/script1.cgi?value=1.
Selectively allow access to a URL that matches a blocked pattern - Use the Allow directive in conjunction with pattern matching for more complex implementations.
# Block access to URLs that contain ?
# Allow access to URLs that end in ?
User-agent: *
Disallow: /*?
Allow: /*?$
That directive blocks all URLs that contain ? except those that end in ?. In this example, the default version of the page will be indexable:
http://www.example.com/productlisting.aspx?Variations of the page will be blocked:
http://www.example.com/productlisting.aspx?nav=pricehttp://www.example.com/productlisting.aspx?sort=alphaSpecify a Sitemap or Sitemap index file - If you’d like to provide search engines with a comprehensive list of your best URLs, you can provide one or more Sitemap autodiscovery directives. Note, user-agent does not apply to this directive so you cannot use this to specify a Sitemap to some but not all search engines.
# Please take my sitemap and index everything!
Sitemap: http://janeandrobot.com/sitemap.axd
Reduce the crawling load - This only works with Microsoft and Yahoo. For Google you’ll need to specify a slower crawling speed through their Webmaster Tools. Be careful when implementing this because if you slow down the crawl too much, robots won’t be able to get to all of your site and you may lose pages from the index.
# MSNBot, please wait 5 seconds in between visits
User-agent: msnbot
Crawl-delay: 5
# Yahoo's Slurp, please wait 12 seconds in between visits
User-agent: slurp
Crawl-delay: 12
The REP page-level directives allow you to refine the site wide policies on a page-by-page basis
Placing a meta tag on the page - Place the meta tag in the head tag. Each directive should be comma delimited inside the tag. E.g. <meta name=”ROBOTS” content=”Directive1, Directive 2>.
<html>
<head>
<title>Your title here</title>
<meta name="ROBOTS" content="NOINDEX">
</head>
<body>Your page here</body>
</html>
Targeting a specific search engine - Within the meta tag you can specify which search engine you would like to target, or you can target them all.
<!-- Applies to All Robots -->
<meta name="ROBOTS" content="NOINDEX">
<!-- ONLY GoogleBot -->
<meta name="Googlebot" content="NOINDEX">
<!-- ONLY Slurp (Yahoo) -->
<meta name="Slurp" content="NOINDEX">
<!-- ONLY MSNBot (Microsoft) -->
<meta name="MSNBot" content="NOINDEX">
Control how your listings - there are a set of options you can use to determine how your site will show up on the SERP. You can exert some control over how the description is created, and remove the “Cached page” link.
<!-- Do not show a description for this page -->
<meta name="ROBOTS" content="NOSNIPPET">
<!-- Do not use http://dmoz.org to create a description -->
<meta name="ROBOTS" content="NOODP">
<!-- Do not present a cached version of the document in a search result -->
<meta name="ROBOTS" content="NOARCHIVE">
Using other directives - Other meta robots directives are shown below.
<!-- Do not trust links on this page, could be user generated content (UCG) -->
<meta name="ROBOTS" content="NOFOLLOW">
<!-- Do not index this page -->
<meta name="ROBOTS" content="NOINDEX">
<!-- Do not index any images on this page (will still index the if they are linked
elsewhere) Better to use Robots.txt if you really want them safe.
This is a Google Only tag. -->
<meta name="GOOGLEBOT" content="NOIMAGEINDEX">
<!-- Do not translate this page into other languages-->
<meta name="ROBOTS" content="NOTRANSLATE">
<!-- NOT RECOMMENDED, there really isn't much point in using these -->
<meta name="ROBOTS" content="FOLLOW">
<meta name="ROBOTS" content="UNAVAILABLE_AFTER">
Allows developers to specify page-level REP directives for non text/html content types like PDF, DOC, PPT, or dynamically generated images.
Using the X-Robots-Tag - to use the X-Robots-Tag, simply add it to your header as shown below. To specify multiple directives you can either comma delimit them, or add them as separate header items.
HTTP/1.x 200 OK
Cache-Control: private
Content-Length: 2199552
Content-Type: application/octet-stream
Server: Microsoft-IIS/7.0
content-disposition: inline; filename=01 - The truth about SEO.ppt
X-Robots-Tag: noindex, nosnippet
X-Powered-By: ASP.NET
Date: Sun, 01 Jun 2008 19:25:47 GMT
The X-Robots-Tag directive supports most of the same directives as the meta tag. The only limitation with this method over the meta tag implementation is that there is no way to target a specific robot - though that probably isn’t a big deal for most use cases.
You can further refine your site level and page level directives within several content tags.
Each anchor tag (link) can be modified to tell search engines that you do not trust where this URL is pointing to. This is typically used for links within user generated content (UCG) like wikis, blog comments, reviews and other community sites.
<a href="#" rel="NOFOLLOW">My Hyperlink</a>
Also, in Yahoo Search you can specify which <div> elements on a page you would not like indexed using the class=robots-nocontent attribute. However, we don’t highly recommend using this tag because it is not supported in any other engine, making it not super-useful.
<div class="robots-nocontent">
No content for you! (or at least Yahoo!)
</div>
While implementing the REP is generally straight-forward, there are a few common mistakes.
user-agent: *) and also declare a section for Googlebot (user-agent: Googlebot), Google will disregard all sections in the robots.txt file except the Googlebot section. This could potentially leave you exposing much more content to Google that you might have thought.
# This keeps out all well-behaved robots
User-agent: *
Disallow: /
# This looks like it is giving Google access to only this directory, but since it is a
# GoogleBot specific section, Google will disregard the previous section
# and access the whole site.
User-agent: Googlebot
Allow: /Content_For_Google/
NOFOLLOW at either the page or the link level, it is still possible for the links from the page to be indexed because the search engine may have found a reference to them from another source. Another note, using rel="NOFOLLOW" within your anchor text is still perceived as a recommendation by the search engines, not a command.
To ensure that content is not indexed, either use the Disallow directive at the site level, or use NOINDEX at the page level.
FOLLOW and INDEX directives as they will not be taken into account by the search engines. It sounds silly but I’ve seen a few sites that have implemented these on every page and every link.
Another directive that is not recommended is the NOCACHE directive. This was created by Microsoft, and is synonymous with NOARCHIVE. While they will most likely always continue to support the directive, it is better to use NOARCHIVE so it will work on all the search engines.
As you’re implementing your REP design, you should test it both before you deploy it and after. The easiest way to test this is to use the robots validator in either Google or Microsoft’s Webmaster Tools. These tools are generally good enough test beds for most folks, however advanced developers (or paranoid ones with critical business requirements) will want to definitively know what the robots are doing, not simply rely on what the robots say they are doing. These folks will want to look at their tools as well look at their server logs.
In addition to using validation tools, reporting tools from the search engines on what they couldn’t acces, and looking at logs data to see what the search engine robots are crawling, you should check the search engine results to see if any pages you are intending to block are being indexed. If they are, use the methods described in this section to ensure you are blocking them correctly and use the search engine tools to request that the pages be removed.
When Blocked Content Appears to be Indexed - If search engines are blocked from crawling pages, they may still index the URL if the robot finds a link to that URL on a page that isn’t blocked. The listing may display the URL only, such as shown below.
Or, it may include a title and in some instances, a description. This makes it appear as though the search engine robot is disregarding the directive that blocks access to the page, but the search engine is in fact obeying the directive not to crawl the page and is using anchor text from the link to that page and descriptive details from either the page that contains the link or a source such as the Open Directory Project.
For more details, see:
Search Engine Tools For Validation - Both Google and Microsoft provide some tools as part of their Webmaster Centers to help you verify if you’ve configured your REP the way you expect. Let’s start with Google’s tools:
The first thing you should check are the list of URLs that Google has seen from your website and not indexed due to the REP. Note you can also download the list and filter, sort, and have-your-way-with-it in Excel.
The next step is to use their interactive robots.txt tool to analyze your rules and test specific URLs for blockage. When you pull up the tool they already should have it pre-populated with the robots.txt file they have on file from the last time they crawled. You can input a list of URLs you’d like to check below, select the user-agent you’d like to check against and the tool will tell you if they are blocked or not. You can also use the tool to test changes to your robots.txt file to see how Google would interpret things.
Microsoft has a similar tool in their Webmaster Center that will validate a robots.txt file against the standard that MSNBot supports. To use the tool, simply log in copy & paste your robots.txt file into the top field and select Validate. A list of all detectable issues are displayed in the bottom box.
More Accurate Views of Robot Access Through Your Logs - If you have a specific business need to ensure that the robots are following your rules, (or you’re just paranoid) then you should not simply rely on the tools they provide to test compliance. You’re going to need to go straight to the horse’s mouth and analyze your web server logs to see exactly what they are doing. There is no one easy tool for doing this, you’ll likely have to use an existing tool like one of these (Microsoft HTTP Log Parser) or write your own. It isn’t difficult, it will simply take some time to implement. A useful reference for this is a list of all the robot user agents, and more complete list of bots from Google, and Microsoft.
Verifying Robot Identity - Another thing you’ll likely want to consider in this endeavor is to validate that the robots are who they actually say they are. Google, Yahoo and Microsoft all support Reverse DNS authentication of their robots. The process is pretty simple and described here by Google, Yahoo and Microsoft, essentially you simply find out what range their robot’s DNS is hosted in, and use that in your tool. This way, if the address changes (which it will), you don’t need to update your code.
Should you find any issues, where one of the robots are not minding the REP, or are misbehaving in some other way, you can always communicate directly with each engine through one of their forums:
If you find that you haven’t implemented the techniques described here correctly and private content from your site is indexed, each of the major search engines has methods available for requesting that it be removed. For more information, see:
