If you’ve ever wondered:
“what is a robots.txt file?”
“Do you need a robots.txt file?”
“How do I Optimize WordPress Robots.txt File for SEO?”
“what is the Importance robots.txt ?”
Recently one of our readers asked us if they need to have a robots.txt file and what importance does it have on SEO. So we have created a step-by-step guide that will show you EXACTLY how to optimize robots.txt file for seo.
So what is Robots.txt?
Robots.txt is your anti-sitemap and instead of telling Google about your blog contents you want the bots to index, the Robots.txt file stops them and tells them what pages of your site that you don’t want to index, and show on search results.
Robots.txt plays a major role in SEO and search engine ranking but if you configure the robots.txt file in a wrong way your presence from search engines may completely be wiped.
SEO is a vast in itself and it has lot of elements in it and robots.txt is one of them.
Here is the breakdown of the contents:
- Do You Need a Robots.txt File?
- Where is the Robots.txt file in my website?
- How to Create a Robots.txt file for my website?
- How to Use Robots.txt file?
- Optimizing Your Robots.txt File for SEO
- Adding Your XML Sitemap to Robots.txt File
- What Does an Ideal Robots.txt File Should Look Like?
- Is My Content Affected By New Robots.txt File?
Do You Need a Robots.txt File?
Robots.txt file puts a stop on the search bots to index your website’s particular areas. This doesn’t mean it is a bad thing. Actually an absence of robots.txt file will not stop search engines from crawling and indexing your website.
Robots.txt file protects your confidential files, admin folders and other files and folders that you don’t want to show up on search results and also tells them which file to crawl as Robots.txt file is the first file that is crawled by bots.
If you submit your XML sitemaps on search engines, then the bots know where to look for your contents and crawl it unless you have specified it in Webmasters Tools.
We highly recommend you to create a robots.txt file immediately if you dont have any.
Where is the Robots.txt file in my website?
Robots.txt file is usually found in your site’s root folder, if its not there then you need to create one.
In WordPress it resides in the root directory.
SmartActiveBlogger also runs on WordPress and out robots.txt file is located in the root directory – https://www.smartactiveblogger.com/robots.txt
How to Create a Robots.txt file for my website?
You can create robots using your cPanel, Hosting File Manager or with an FTP Client. We will use FTP Client to create the Robots.txt file. The process is same for cPanel, you just need to Login to your account.
First we need to connect the FTP Client with our WordPress directory, or login to our cPanel and navigate to File Manager.
Use a notepad and create an ordinary text file and name it robots.txt. Next simply upload it to your site’s root directory. If you already have a robots.txt file then you don’t need to create a new one.
How to Use Robots.txt file?
If this is your first time to robots.txt then it may feel little scary, but don’t worry. We have created this guide for you keeping in mind that you are completely new to robots.txt
The format of robots.txt is pretty simple, the text “user-agent” is actually the name of the bot that you are trying to instruct.For example, Googlebot or Bingbot. You can use asterisk * to instruct all bots.
The next line follows with “Allow or Disallow” command, so the search bots know either it should follow the mentioned directory or not.
If set to Allow, the search bots will crawl and index the ones and if Disallow, it will ignore and won’t index the directory and its contents.
See how a sample robots.txt file looks like
In this sample robots.txt, the * means all bots are instructed the below commands, and Allow: /wp-contents/uploads/ means the contents inside the upload folder will be index by the bot.
In the next two lines, we have disallowed 1 directory and 1 file.
Optimizing Your Robots.txt File for SEO
Optimizing your robots.txt for seo is quite simple, but you should follow the guidelines of google webmasters. Google advises users not to use robots.txt file to hide low quality contents on your website like category, date and other archive pages.
WordPress plugins that you use to add meta tags and descriptions for your contents can be used to manage and optimize the seo by adding nofollow and noindex to archive pages.
Other files are login page, registration pages already have the nofollow and noindex tags, so you don’t need to add them in your robots.txt.
It is recommended to you to add disallow to the followings:
Disallow: /go/ [This is the outbound link on our blog]
Why this files are disallowed?
This files can be used by a hacker or someone who is running a malicious query to locate websites that are running a specific version, only disallowing will protect you as it won’t show up on search results.
Plugins folder is recommended to disallow as someone can find sites having a particular plugin which are vulnerable and can be exploited.
Adding Your XML Sitemap to Robots.txt File
If you are using any plugins for creating XML Sitemaps then those sitemaps will not be added to your robots.txt. You should add your sitemap URL in your robots.txt, by doing this you make sure that the search bots can find all the urls of your blog that it needs to index.
Adding your XML Sitemap in robots.txt can help your site get index if for any reason you forgot to add your sitemap in your webmasters tools.
What Does an Ideal Robots.txt File Should Look Like?
For every website the robots.txt file differs.
Here is another example of a robots.txt file, this time it is the one we use here on SmartActiveBlogger:
That’s it for your robots.txt
Is My Content Affected By New Robots.txt File?
Well after doing all those changes you should atleast know if your contents were affected or not. And to check that we have a fantastic tool in Google webmasters known as Fetch as Google located at Crawl > Fetch as Google.
In Fetch as Google tab, add any link of your post and test it by clicking fetch and render. The tool will let you know about the bot type and the status.
You can also check the effect of robots.txt by visiting the robots.txt tester in the webmasters which is located under Crawl > Robots.txt Tester. You just need to add any link from your blog that you want to test if the bot has access to it or not, just click the test button and you will find the status.
That’s all. We hope this article helped you learn how to optimize your WordPress robots.txt file for SEO. You may also want to see our guide on 15 Best Ways To Increase Google Crawl Rate Of Your Website that you should definitely read.
Do you have experience with any of these? Would you add others to the list? Share your thoughts in the comment section below.