What is Robots.txt File and How to Set it up?

Hi Again! As I promised to write all the basic SEO tips on Stoogles.com, I have started the series Summarized SEO world and now I am writing its 8^th part which is about Robots.txt. Hope you are enjoying the basic SEO tips here. If you have suggestions then do let me know. I love to hear from you.

In case you have missed the first 7 parts of this series, I am sharing the links of them here so that you will not have to face any hassle to find them. Check them out here:-

1. Site Analysis
2. Competitor Analysis
3. Keyword Research
4. URL Optimization
5. Writing Title and meta description
6. Content Optimization
7. Image Optimization

Introduction to Robots.txt

Robots.txt is nothing but a simple text file through which you can control the behaviour of search engine robots either to crawl and index your website/any particular directory or not.

By default every website allows the Search engines robots but if you want to restricts the robots either to not to crawl any certain directory, file or the full site then you will need the robots.txt file in which you have to write instructions for search engine bots.

Why should you use robots.txt file on your site?

There are times when you are developing your site online and dealing with so many test pages, files etc. Those might be crawled by Search engines bots and as you know at the time of development and design those pages might not have good content, and if search engines crawl and index them then it will not good for your site’s SEO. There might be so many garbage from your site indexed by Search engines already which you will have to de-index later. That will be cumbersome to do. Hence, at the time of development robots.txt can be a great help through which you can restrict the search engine robots.
Secondly, when your site is running fine and you might have some private folders or anything which you don’t want to show up in search engines then Robots.txt plays an important role here. You can disallow them from crawling and indexing certain files and folders.

How to Write Robots.txt file

There is very simple rule to write the instructions in a robots.txt file. Lets have a look on the simplest robots. Txt:-

User-Agent:*
Disallow: /

Here The very first line “User-Agent” defines the agent’s or bot’s name such as Googlebot. T
The wildcard “*” defines all the robots.
The second line says “Disallow” where you define the particular directory or any particular directory which you dnt want to get crawled. Here “/” says that don’t crawl the whole site.

If you want to hide only 1 directory such as “private” from all the robots then :-

User-Agent:*
Disallow: /private

If you want to hide this directory only from Googlebot then:-

User-Agent: Googlebot
Disallow: /private

If you want to disallow certain page in a directory then you may write as:-

User-Agent: Googlebot
Disallow: /private/abc.html

In this way you may write a robots.txt file for your website or blog but there are so many generator out there online which can help you create this easily.

Here are names of few Google Bots for your reference:

• Googlebot
• Googlebot-Mobile
• Googlebot-Image
• Mediapartners-Google
• Adsbot-Google

Where to upload your robots.txt file?

After writing the file save it as robots.txt on your system and then upload it to the root directory of your website. The url should form as yoursite.com/robots.txt

I know this post is not a new thing but I hope it might help many newbies to understand the importance of robots.txt. If you like the post, I would like you to share this on your social profiles. Thank your very much for giving time to read this post.