Have you optimized your WordPress Robots.txt file for SEO?

If you haven’t, you are ignoring an important aspect of SEO. Robots.txt file plays a significant role on your site’s SEO.

You are lucky that WordPress automatically creates a Robots.txt file for you. Having this file is half of the battle. You have to make sure that Robots.txt file is optimized to get the full benefits.

Robots.txt file tells search engine bots what pages to crawl and what pages to avoid. In this post, I will show you how to edit and optimize Robots.txt file in WordPress.

What is Robots.txt File?

Let’s start with basic.

Robots.txt file is a text file which instructs search engine bots how to crawl and index a site. Whenever any search engine bots come to your site, it reads the robots.txt file and follow the instructions. By using this file, you can specify bots which part of your site to crawl and which part to avoid. However, the absence of robots.txt will not stop search engine bots to crawl and index your site.

Editing & Understanding Robots.txt in WordPress

I’ve already said that every WordPress site has a default robots.txt file in root directory. You can check your robots.txt by going to http://yourdomain.com/robots.txt. For example, you can check our robots.txt file from here: https://roadtoblogging.com/robots.txt

If you don’t have a robots.txt file, you’ll have to create one. It’s very easy to do. Just create a text file in your computer and save it as robots.txt and upload it to your root directory. You can upload it via FTP Manager or cPanel File Manager.

Now let’s see how to edit your robots.txt file.

You can edit your robots.txt file by using FTP Manager or cPanel File Manager. But it’s time-consuming and a bit difficult.

The best way to edit Robots.txt file is, using a plugin. There are several WordPress robots.txt plugins out there. I prefer Yoast SEO. This is the best SEO plugin for WordPress. I’ve already shared how to set up Yoast SEO.

Yoast SEO allows you to modify the robots.txt file from your WordPress admin area. However, if you don’t want to use Yoast plugin, you can use other plugins like WP Robots Txt.

Once you’ve installed and activated Yoast SEO plugin, go to WordPress Admin Panel > SEO > Tools.

Then click on “File editor”.

Then you need to click on “Create robots.txt file”.

Then you will get the Robots.txt file editor. You can configure your robots.txt file from here.

Before editing the file, you need to understand the commands of the file. There are three commands mainly.

• User-agent – Defines the name of the search engine bots like Googlebot or Bingbot. You can use an asterisk (*) to refer to all search engine bots.
• Disallow – Instructs search engines not to crawl and index some parts of your site.
• Allow – Instructs search engines to crawl and index which parts you want to index.

Here’s a sample of Robots.txt file.

[code]

User-agent: *
Disallow: /wp-admin/
Allow: /

[/code]

This robots.txt file instructs all search engine bots to crawl the site. In the 2nd line, it tells search engine bots not to crawl the /wp-admin/ part. In 3rd line, it instructs search engine bots to crawl and index whole website.

Configuring & Optimizing Robots.txt File for SEO

A simple misconfigure in Robots.txt file can completely deindex your site from search engines. For example, if you use the command “Disallow: /” in Robots.txt file, your site will be deindexed from search engines. So you need to be careful while configuring.

Another important thing is optimization of Robots.txt file for SEO. Before going to the best practices of Robots.txt SEO, I’d like to warn you about some bad practices.

• Don’t use Robots.txt file to hide low-quality contents. The best practice is to use noindex and nofollow meta tag. You can do this by using Yoast SEO plugin.
• Don’t use Robots.txt file to stop search engines to index your Categories, Tags, Archives, Author pages, etc. You can add nofollow and noindex meta tags to those pages by using Yoast SEO plugin.
• Don’t use Robots.txt file to handle duplicate content. There are other ways.

Now let’s see how you can make Robots.txt file SEO friendly.

1. At first, you need to determine which parts of your site you don’t want search engine bots to crawl. I prefer disallowing /wp-admin/, /wp-content/plugins/, /readme.html, /trackback/.
2. Adding “Allow: /” derivatives on Robots.txt file is not so important as bots will crawl your site anyway. But you can use it for the particular bot.
3. Adding sitemaps to Robots.txt file is also a good practice. Read: How to create Sitemap

Here’s an example of ideal Robots.txt file for WordPress.

[code]

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Disallow: /readme.html
Disallow: /trackback/
Disallow: /go/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/

Sitemap: https://roadtoblogging.com/post-sitemap.xml
Sitemap: https://roadtoblogging.com/page-sitemap.xml

[/code]

You can check RTB Robots.txt file here: https://roadtoblogging.com/robots.txt

Testing Robots.txt File in Google Webmaster Tools

After updating your Robots.txt file, you have to test the Robots.txt file to check if any content is impacted by update.

You can use Google Search Console to check if there is any “Error” or “Warning” for your Robots.txt file. Just login to Google Search Console and select the site. Then go to Crawl > robots.txt Tester and click on “Submit” button.

A box will be popped up. Just click on “Submit” button.

Then reload the page and check if the file is updated. It might take some time to update the Robots.txt file.

If it hasn’t updated yet, you can enter your Robots.txt file code into the box to check if there are any errors or warnings. It will show the errors and warnings there.

If you notice any errors or warnings in the robots.txt file, you have to fix it by editing the robots.txt file.

Final Thoughts

I hope this post helped you to optimize your WordPress robots.txt file. If you have any confusion regarding, feel free to ask us via comments.

However, if you want to make your WordPress blog SEO friendly, you can read our post on How to Setup WordPress Yoast SEO Plugin.

If you find this post helpful, please help me by sharing this post on Facebook, Twitter or Google+.