Blogger allows you to specify custom header tags for pages of your blog. This article tries to explain the meaning of these tags. We start by explaining what robots are and then proceed to header tags and robot header tags.
Robots are computer programs which are programmed to access web pages for a variety of purposes. An example of a robot is the Google crawler which fetches pages from the internet, analyses content and stores them in its database for use in Google Search. Website owners can communicate with robots using some tags or mark-up which can be understood by robots.
When a web page is requested from a server, a number of request and response headers are associated with it. Each of these headers has a different purpose. For example, a response header can be used to indicate that the requested page can be cached for certain time duration.
Robot header tags are those headers which are meant for consumption by robots.
Setting robot header tags in Blogger
Blogger provides an interface to set robot header tags. It can be accessed at Settings -> Search Preferences.
Under ‘Crawlers and Indexing’, there is an option named ‘Custom robots header tags’ which is disabled by default. On clicking ‘Edit’, you will be asked if you want to enable custom robots header tags. Click ‘Yes’ and you will be presented with a number of tags for different types of pages – ‘Home Page’, ‘Archive and Search Pages’ and ‘Default for Posts and Pages’. Select the options you want and click ‘Save Changes’.
First, we will look at Blogger’s classification of pages and then at the various robot tags.
Classification of pages by Blogger
As far as robot header tags are concerned, all pages on a blogger blog are classified into three categories:
Robots header tags
The various robot header tags allowed are all, noindex, nofollow, none, noarchive, nosnippet, noodp, notranslate, noimageindex and unavailable_after. Each of these is explained below.
How robot tags are specified
The robot tags are specified as a HTTP header with the key being ‘X-Robots-Tag’. The various tags separated by commas form the value of that header. For example, if noodp, nosnippet and notranslate are set, then the header will be
To appreciate this article in a better way, we will now see how you can use Chrome to see the robot tags.
Open your Chrome browser and press F12. This will open the Inspector window at the bottom. Select the Network tab. This tab shows all resource requests made by you. Open your blog’s home page in the same Chrome tab. The request will be shown in the Network tab (The name of that network request will be the URL of your blog. See the image below where the blog’s URL is www.alightheartedtalk.com).
Clicking on it will show the details corresponding to that request. You can see the Request headers and the Response headers. The Response headers doesn’t contain any header named ‘X-Robots-Tag’ because robots tags are not yet activated.
Now, set some robots tags for your home page using the method described in the ‘Setting robot header tags in Blogger’ section of this blog post and reload your blog’s home page. (You may have to press Ctrl+F5 to force a hard refresh so that the webpage won’t be served from your browser cache) Now, you will see an ‘X-Robots-Tag’ corresponding to the tags selected by you.
Should you play with robot header tags?
Incorrect use of robot tags can affect the performance of your site in search results. These options should be used only if you are completely sure of what you are doing. Generally, a normal blogger wouldn’t be requiring these options.
Leave a Reply