Robots.txt Syntax, Setup, and How to Test It Before You Break Something

Your robots.txt file is a plain-text rulebook that tells crawlers which parts of your site they can request. And that’s why, if you get one rule wrong, you can accidentally block your entire site from Google. 

This guide covers the syntax you actually need, how to build the file correctly, and how to test it before it goes live, not after.

Now here’s a quick test you should do before we get into the technicalities of the robots.txt file:

Here’s how it will look on a website browser: 

What Robots.txt Actually Controls

Robots.txt controls crawling, not indexing. This distinction trips up more people than any other part of this file. Now, note that a robots.txt file has unique elements that you use to give special instructions to crawlers.

Starting with the “Disallow” element, it tells a crawler not to request a URL. It does not remove the URL from Google’s search results if Google already indexed it or found it through another website or page. 

But suppose you want Google to stop showing a page in search results; a noindex meta tag or HTTP header is the appropriate tag to add to that page. That does the job instead of using a robots.txt rule. 

The point is that blocking a page in robots.txt while also trying to “noindex” it backfires. And that’s because Google can’t see the noindex tag if it’s not allowed to crawl the page in the first place.

Read our guide on indexation issues to learn how to keep pages out of Google’s search results. 

Robots.txt Syntax, Line by Line

A robots.txt file uses a few simple rules called directives. Each one tells search engines what they can or cannot crawl. These directives include the following:

User-agent

The User-agent directive tells the rule which search engine crawler it applies to. You can target one crawler or all crawlers with an asterisk (*).

Note: 

* applies the rule to every search engine crawler.

Googlebot applies the rule only to Google’s crawler.

Bingbot applies the rule only to Bing’s crawler.

Disallow

The Disallow directive blocks search engines from crawling a folder or page. If you leave the value empty, search engines can crawl everything. 

Note:

The first line blocks the /admin/ folder.

The second line blocks the /cart/ folder.

The third line blocks nothing and allows search engines to crawl your entire website.

Allow

The Allow directive lets you allow a file or subfolder inside a folder that you blocked. You can use it to block a folder while still allowing search engines to crawl only a file or subfolder. 

In this example, search engines cannot crawl the /wp-admin/ folder, but they can still crawl the admin-ajax.php file.

Wildcards: * and $

Wildcards help you create more flexible rules.

• The asterisk (*) matches any group of characters.
• The dollar sign ($) tells the rule to match only the end of a URL.

Explanation: The first rule blocks any URL that contains a sort parameter. And the second rule only blocks PDF files. It does not block URLs that simply contain “.pdf” as part of a longer address.

Sitemap

The Sitemap directive tells search engines where to find your XML sitemap. You can place this line anywhere in the file.

This is one of the most useful lines in a robots.txt file because it helps search engines find new pages faster. 

You can see our XML sitemap guide if you haven’t set one up yet.

Crawl-delay

The Crawl-delay directive tells some search engines to wait a few seconds before making another request to your website. 

Bing and some other search engines follow this rule. Google ignores it. If you want to slow down Googlebot, change the crawl rate in Google Search Console instead.

Comments

A comment starts with a #. Search engines ignore everything after it. Use comments to leave notes for anyone who edits the file later.

In this example, the comment explains that the next rule blocks the website’s internal search results pages. 

Build or Edit Your Robots.txt File

To build or edit a robots.txt file, you need to create a plain text file with the right rules and place it in your website’s root directory. But if your website uses a CMS, you can also edit the file with the platform’s built-in editor. After that, you can add the user-agents and the Allow or Disallow rules you need before uploading or saving the file. 

You can build or edit a robots.txt file in different ways. The right method depends on your website setup. For example, you may use WordPress, Shopify, or another CMS. You may manage your own server or custom website. Or, you may have multiple subdomains. Based on your setup, follow the right path below to build or edit your robots.txt file. 

Path A: You Use WordPress, Shopify, or Another CMS

First, check if your website already has a robots.txt file by visiting this URL on your web browser:

yoursite.com/robots.txt

Most CMS platforms create a default robots.txt file for you, like the example shown above. If you use WordPress, you may not need to create the file yourself. Some SEO plugins, such as Rank Math, let you edit the file inside WordPress. If you use Yoast SEO Free, you will need to edit the file with your hosting provider’s File Manager, an FTP program, or another trusted file management tool. 

But if you’re using Shopify, you can change specific rules with a robots.txt.liquid template instead of replacing the entire file. This is because the platform automatically creates a default robots.txt file. 

Path B: You Manage Your Own Server or Custom Website

Following this path, you would need to create a plain-text file with the name robots.txt. And in doing this, ensure you use only lowercase letters.

You can then put the file in your website’s main folder (root directory). Once done correctly, web users should be able to open it at: yoursite.com/robots.txt.

As a rule, you should not put the file inside another folder. More importantly, you should save the file as a plain text file with UTF-8 encoding, not as an HTML file or a rich text document.

Another thing to note when creating this file is that you should write only one Disallow or Allow rule per line. Also, do not add more than one path to the same line.

Path C: You Use Multiple Subdomains

If you have multiple subdomains, note that each subdomain needs its own robots.txt file.

Here’s a typical example, although fictitious:

• example.com/robots.txt
• blog.example.com/robots.txt

This is essential because each file controls only its own subdomain.

And that’s why you should check every subdomain separately. Typically, a correct robots.txt file on your main website does not control your blog, staging site, or any other subdomain.

Test Your robots.txt File Before You Use It

Google no longer has the old robots.txt Tester tool in Search Console. But you can still test your file safely by following these steps.

First, write your changes in a plain text editor. Do not edit the live robots.txt file on your website first.

Next, validate your file using a robots.txt parser that adheres to the official specification. You can use Google’s open-source robots.txt library (used in Google Search) or another trusted validator. This helps you find and fix mistakes before you publish the file.

And if you are worried about one page, use the URL Inspection tool in Search Console. It shows whether the URL is allowed or blocked by Googlebot, based on your live robots.txt file.

When everything looks correct, upload the robots.txt file to your site’s root (so it’s reachable at https://yourdomain.com/robots.txt).

Finally, open https://yourdomain.com/robots.txt in a private or incognito browser window to ensure the file matches the one you uploaded and is publicly accessible. 

Note: you can learn more about how to write and submit a robots.txt based on Google’s specification. 

Verifying Google Has Picked Up Your Created Robots.txt file

The first step is to go to Settings on your Search Console and open the robots.txt report.

Then confirm the report shows your current file and the correct last-crawled date, as you can see in the image.

And if you need Google to see an urgent change faster, use the request recrawl option in that same report.

Finally, you can check the Page Indexing report a few days later for any new “blocked by robots.txt” entries you didn’t expect.

5-Minute Robots.txt Check

You can check your live robots.txt file today, even if you haven’t changed it in a long time. Here are actionable steps you can take to do this:

☐ Open yoursite.com/robots.txt.  The goal is to ensure it opens as plain text and shows no errors.

☐ Check that it does not say “Disallow: /” for all user-agents. Looking at this is important because this setting blocks search engines from your whole website.

☐ Ensure your CSS and JavaScript folders are not blocked. And that’s because search engines need these files to see your pages the right way.

☐ Check that the Sitemap line points to the correct sitemap that still works.

☐ Test three important pages, like your homepage, a product or feature page, and your pricing page. This is to ensure that none of them are blocked by a Disallow rule.

☐ Look for uppercase and lowercase mistakes. For example, /Admin/ is different from /admin/ because these paths are case-sensitive.

If you find any problems, you must fix them before you talk about crawl budget or indexing. And that’s because a broken robots.txt file can hurt both.

Common Mistakes with robots.txt and How to Fix Them

Mistake	Fix
Blocking /wp-content/ or /assets/ entirely, which hides CSS and JS from Google	Allow crawlers to fetch stylesheets and scripts so pages render correctly
Using robots.txt to try to remove a page from search results	Use a noindex tag on the page itself, and make sure the page is crawlable so Google can see that tag
A leftover “Disallow: /” pushed from staging to production	Treat this as an urgent fix; check it every time you deploy from staging
Relying on Crawl-delay to control Googlebot	Use the crawl rate settings in Search Console instead, since Google ignores this directive
Placing the file in a subfolder instead of the domain root	Move it to yoursite.com/robots.txt exactly, or it will not be recognized
Assuming one robots.txt covers all subdomains	Create and check a separate file for each subdomain you run

How This Fits Into Your Wider Technical SEO

Robots.txt works alongside your sitemap and your overall crawl budget. In practice, a clean file helps crawlers spend their time on pages that matter. But that is only one input among several. 

You can see our SaaS technical SEO guide for the full picture on how these pieces fit together.

Frequently Asked Questions about Robots.txt

Does robots.txt stop a page from being indexed?

Not reliably. And this is because while it stops crawling, a blocked page can still appear in search results, sometimes with no description, if other sites link to it. This is why you should use a “noindex” tag for pages you need out of search results entirely.

Can I have more than one robots.txt file?

No. You can only have one robots.txt file for each website or host. Put it in the main folder (root directory) of that website. And note if you have subdomains, each one needs its own robots.txt file.

Will a syntax error break the whole file?

Usually crawlers skip a malformed line rather than the whole file, but behavior varies by crawler. Again, this is why it’s crucial to test before deploying rather than relying on that.

Do I need to block admin or login pages?

It’s common practice, but remember it only stops crawling, not access. The point is that security for those pages should come from authentication, not robots.txt.

How often should I check my robots.txt file?

After every migration, replatform, or staging-to-production deploy. Those are the moments this file most often breaks.

Your Action Checklist for This Week