How to Edit and Optimize the WordPress Robots.txt File for SEO

The robots.txt file is a short text file at the root of your site that tells search engine crawlers which parts of the site they may fetch. WordPress generates a sensible default, and for most blogs the right robots.txt is only a few lines long. The risk runs the other way: one wrong line can block Google from your CSS, your images, or the whole site. This guide explains what the file does, how to edit it in WordPress with Rank Math, Yoast, or by hand, what an optimized file looks like in 2026, and how to test it. It also covers the new question of whether to block AI crawlers.

What Robots.txt Does (and Does Not Do)

When a crawler such as Googlebot arrives, it reads yourdomain.com/robots.txt first and follows its rules about which URLs to crawl. Three things to understand before editing:

• It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other sites link to it; Google just cannot read it. To keep a page out of search results, use a noindex meta tag (Rank Math and Yoast set this per page) and leave it crawlable.
• It is advisory. Well-behaved bots obey it; scrapers ignore it. It is not a security tool.
• Blocking resources hurts rankings. Google renders pages like a browser. If robots.txt blocks /wp-content/ or /wp-includes/, Google cannot load your CSS and JavaScript and may judge the page broken or not mobile-friendly.

The Commands

• User-agent: which crawler the rules apply to. * means all of them.
• Disallow: a path the crawler should not fetch.
• Allow: an exception inside a disallowed path.
• Sitemap: the full URL of your XML sitemap. You can list several.

WordPress’s virtual default looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/wp-sitemap.xml

How to Edit Robots.txt in WordPress

With Rank Math

Go to Rank Math > General Settings > Edit robots.txt. Edit the text and save. Rank Math serves the file dynamically, so this works even if no physical file exists. If a physical robots.txt file is present in your root folder, the editor is disabled until you delete that file.

With Yoast SEO

Go to Yoast SEO > Tools > File editor. If no file exists, click Create robots.txt file; otherwise edit and save.

By hand

Create a plain text file named robots.txt and upload it to the root of your site (the folder containing wp-config.php, usually public_html) with cPanel File Manager or FileZilla. A physical file overrides the WordPress virtual one.

An Optimized Robots.txt for WordPress in 2026

Less is more. This is what we recommend for a typical blog:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /go/

Sitemap: https://yourdomain.com/sitemap_index.xml

• /wp-admin/ keeps crawlers out of the dashboard; admin-ajax.php is allowed because themes and plugins call it on the front end.
• /?s= and /search/ stop crawlers wasting time on internal search result pages, which are thin and infinite.
• /go/ (or whatever prefix your affiliate link plugin uses) keeps cloaked affiliate redirects out of the crawl.
• Sitemap points to the index your SEO plugin generates: /sitemap_index.xml for Rank Math and Yoast, /wp-sitemap.xml for core WordPress. See how to create a sitemap.

What not to add, even though older guides (including the first version of this post) suggested it:

• Disallow: /wp-content/plugins/ or /wp-includes/: blocks CSS and JavaScript Google needs to render pages.
• Disallow: /wp-content/uploads/: removes your images from Google Images.
• Blocking categories, tags, author pages, or feeds: use noindex in your SEO plugin instead if you want them out of search.
• Disallow: /: deindexes the entire site over time. The one line that must never be there.

Should You Block AI Crawlers?

Since 2023 the AI companies publish crawler names you can block: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s AI training, separate from Googlebot), CCBot (Common Crawl), PerplexityBot, and others. Blocking them stops your content being used for model training by bots that honor robots.txt, but it can also remove you from AI answers and search features that now send real traffic. Many publishers block training crawlers and allow search-related ones. If you decide to block, add separate groups:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Do not block Googlebot, Bingbot, or the AI search crawlers (OAI-SearchBot, Claude-SearchBot) if you want to appear in those search products.

Testing Your Robots.txt

Google removed the old robots.txt Tester. Now open Google Search Console > Settings > Crawling > robots.txt to see the version Google last fetched, any errors, and request a recrawl. To check whether a specific URL is blocked, paste it into the URL Inspection tool; a blocked page shows “Blocked by robots.txt” under crawl. The Bing Webmaster Tools robots.txt tester still works for syntax checks.

Frequently Asked Questions

Where is the robots.txt file in WordPress?

At yourdomain.com/robots.txt. If no physical file exists, WordPress generates one on the fly, and SEO plugins let you edit that virtual file from the dashboard.

Does robots.txt improve SEO?

A correct file avoids problems (blocked resources, wasted crawl budget) rather than adding ranking power. For a small blog the main benefit is pointing crawlers to your sitemap and keeping them out of search results pages.

How do I stop Google indexing a page?

Add a noindex tag to that page in Rank Math or Yoast (Advanced tab) and leave it crawlable. Blocking it in robots.txt can keep it in the index as a URL-only listing.

How long does Google take to notice robots.txt changes?

Google refetches robots.txt roughly once a day. You can request a recrawl in Search Console under Settings > robots.txt.