Blogger allows you to specify custom header tags for pages of your blog. This article tries to explain the meaning of these tags. We start by explaining what robots are and then proceed to header tags and robot header tags.

Robots

Robots are computer programs which are programmed to access web pages for a variety of purposes. An example of a robot is the Google crawler which fetches pages from the internet, analyses content and stores them in its database for use in Google Search. Website owners can communicate with robots using some tags or mark-up which can be understood by robots.

Header tags

When a web page is requested from a server, a number of request and response headers are associated with it. Each of these headers has a different purpose. For example, a response header can be used to indicate that the requested page can be cached for certain time duration.

Robot header tags are those headers which are meant for consumption by robots.

Setting robot header tags in Blogger

Blogger provides an interface to set robot header tags. It can be accessed at Settings -> Search Preferences.

Under ‘Crawlers and Indexing’, there is an option named ‘Custom robots header tags’ which is disabled by default. On clicking ‘Edit’, you will be asked if you want to enable custom robots header tags. Click ‘Yes’ and you will be presented with a number of tags for different types of pages – ‘Home Page’, ‘Archive and Search Pages’ and ‘Default for Posts and Pages’. Select the options you want and click ‘Save Changes’.

First, we will look at Blogger’s classification of pages and then at the various robot tags.

Classification of pages by Blogger

As far as robot header tags are concerned, all pages on a blogger blog are classified into three categories:

  • Home Page – This category includes a single page, the home page of your blog.
  • Archive and Search pages – Archive pages are those which list multiple blog posts (or snippets) on a single web page. These are the page accessed by clicking the ‘Older’and ‘Newer’ links at the bottom of the home page and all pages reached by clicking these links, or the pages accessed by clicking the Year or Month links in the blog archives.
    Search pages are those which open on searching in a blogger blog. These pages also show multiple blog posts on a single page.
  • Pages and Posts – These are the blog post pages or standalone blogger pages. These pages show only a single blog post and the associated comments.

Robots header tags

The various robot header tags allowed are all, noindex, nofollow, none, noarchive, nosnippet, noodp, notranslate, noimageindex and unavailable_after. Each of these is explained below.

  • all – No restrictions are placed on the robots. They are allowed to crawl, index and serve content. This is the default value.
  • noindex – The page shouldn’t be indexed i.e. it shouldn’t be shown in search results.
  • nofollow – All links on the page containing this directive will be treated as nofollow. The page will be indexed but the linked pages will not be crawled. Note that if there is another web page somewhere on the internet having links to those pages marked as nofollow by you, then those pages will be crawled. The nofollow directive simply states that ‘Do not index the linked pages because I have linked to them. But, if someone else has linked to them and has not marked them as nofollow, then you are free to index them’.
  • none – none is the combination of noindex and nofollow.
  • noarchive -  Google caches certain web pages and shows a link to the cached pages in search results. If noarchive tag is specified, then the cached link will not be shown in search results.
  • nosnippet – Along with links to webpages, Google search results also shows a text snippet to give the searcher an idea of the contents of that page. If nosnippet is set, then text snippets will not be shown for that page in search results.
  • noodp – The Open Directory Project (dmoz.org) is a human created directory of websites and webpages. The human editors also add descriptions for these webpages. Sometimes, Google uses these descriptions (and other information) to improve the text snippets or other information shown in search results. If the noodp tag is set, Google will not use metadata from ODP.
  • notranslate – If the language of the user is different from the language of the webpage, Google shows a link in its search results for translating that page. If notranslate is set, then such links won’t be shown.
  • noimageindex – If set, the images on that page will not be indexed. This is similar to nofollow tag. If the same image is present on some other page, then that image may be indexed.
  • unavailable_after – The webpage will not be shown in search results after the specified date and time.

How robot tags are specified

The robot tags are specified as a HTTP header with the key being ‘X-Robots-Tag’. The various tags separated by commas form the value of that header. For example, if noodp, nosnippet and notranslate are set, then the header will be

X-Robots-Tag:noodp,nosnippet,notranslate

Live Experiment

To appreciate this article in a better way, we will now see how you can use Chrome to see the robot tags.

Open your Chrome browser and press F12. This will open the Inspector window at the bottom. Select the Network tab. This tab shows all resource requests made by you. Open your blog’s home page in the same Chrome tab. The request will be shown in the Network tab (The name of that network request will be the URL of your blog. See the image below where the blog’s URL is www.alightheartedtalk.com).

Clicking on it will show the details corresponding to that request. You can see the Request headers and the Response headers. The Response headers doesn’t contain any header named ‘X-Robots-Tag’ because robots tags are not yet activated.

Now, set some robots tags for your home page using the method described in the ‘Setting robot header tags in Blogger’ section of this blog post and reload your blog’s home page. (You may have to press Ctrl+F5 to force a hard refresh so that the webpage won’t be served from your browser cache)  Now, you will see an ‘X-Robots-Tag’ corresponding to the tags selected by you.

Should you play with robot header tags?

Incorrect use of robot tags can affect the performance of your site in search results. These options should be used only if you are completely sure of what you are doing. Generally, a normal blogger wouldn’t be requiring these options.