Every website has visitors that never buy anything, never fill out a form and never read a page the way a person does. Search engine crawlers, monitoring services, SEO tools, archive projects and a growing number of AI-related bots request pages around the clock. Some help customers find you. Others collect content for purposes you may or may not support. And some ignore the rules entirely.

The simple text file that sits at the root of most sites, robots.txt, is how you tell well-behaved crawlers what you want. Yet many business owners have never opened theirs. This guide explains what robots.txt can and cannot do, how it works with page-level controls, and how to decide which bots you are letting in.

What robots.txt is

Robots.txt is a plain text file placed at the top level of a website, such as example.com/robots.txt. It contains rules that tell crawlers which paths they may or may not request. In a March 7, 2025 post, Robots Refresher: robots.txt — a flexible way to control how machines explore your website, Google noted that robots.txt has been in active use for over 30 years and is broadly supported by crawler operators, including tools for site owners, services and search engines.

The same post explained that you can leave the file empty, or not have one at all, if your whole site may be crawled. Otherwise, you add rules. A rule group starts with a user-agent line naming the crawler, followed by allow or disallow lines listing paths. The example Google gave for keeping all bots away from a cart page is just two lines: a user-agent line with an asterisk, meaning every crawler, followed by a disallow line for the /cart path. Even a beginner can read and edit rules like these easily.

The protocol was formalized as an internet standard in September 2022. RFC 9309, the Robots Exclusion Protocol, published by the Internet Engineering Task Force, specifies how crawlers should read and apply these rules.

What robots.txt cannot do

The most important line in RFC 9309 may be one of the shortest: these rules are not a form of access authorization. Robots.txt is a request. Reputable crawlers honor it, but nothing technically prevents a bot from ignoring it.

The standard’s security considerations go further. They state that the protocol is not a substitute for valid content security measures, and that listing paths in robots.txt exposes them publicly, making those paths discoverable. In other words, adding a line like “disallow: /private-client-files/” tells the whole world that folder exists.

If content must stay private, protect it with authentication, such as a login, and appropriate server permissions. Use robots.txt to manage crawling of public content, not to hide sensitive material.

How crawlers match rules

Understanding a few details from RFC 9309 helps avoid surprises:

  • Crawlers identify themselves with a product token. The standard says crawlers set their own name to find relevant groups in the file, and that this token should appear in the User-Agent header the crawler sends.
  • The most specific match wins. When allow and disallow rules both match a URL, the crawler must use the most specific match, which the standard defines as the one with the most octets, in practice the longest matching rule. If an allow rule and a disallow rule are equivalent, the allow rule should be used.
  • Matching is case sensitive. The standard says matching should be case sensitive, so /Photos and /photos are different paths.
  • Caching is limited. Crawlers may cache robots.txt, but should not use a cached version for more than 24 hours unless the file is unreachable.
  • Size limits exist. Crawlers should impose a parsing limit, and that limit must be at least 500 kibibytes, so extremely long files risk having later rules ignored.

When your robots.txt file is unavailable

Server responses for the file itself matter more than many owners realize. RFC 9309 describes two situations:

  1. Unavailable. If the server returns a status code in the 400 range, such as “not found,” the crawler may access any resources on the server. A missing file effectively means no restrictions.
  2. Unreachable. If the server returns errors in the 500 range or the request fails because of network problems, the crawler must assume complete disallow. The standard notes that if the file stays unreachable for a reasonably long period, for example 30 days, crawlers may treat it as unavailable or continue to use a cached copy.

This means a misconfigured server or firewall that returns errors for robots.txt can stop well-behaved crawlers from visiting the site at all, including search engines. Checking that the file returns a normal response is a basic but valuable test.

Hands in turquoise sleeves draw a yellow highlighter line across a blank sheet on a red desk.

Choosing which bots to welcome

Robots.txt lets you write rules for specific crawlers. Google’s March 2025 post gave several examples, including telling two named bots to stay out of a search path, telling a bot to crawl a blog but not its drafts, and blocking one crawler from everything except the home page while allowing others full access. That last example used a fictional “aicorp-trainer-bot” to show how a site owner could limit a training crawler without affecting search engines.

Before writing rules, decide what each type of visitor is worth to your business:

  • Search engine crawlers help customers find you. Blocking them usually means pages disappear from search results.
  • AI-related crawlers may collect content for training models or for answering questions. Some site owners welcome the exposure; others prefer to limit it. Many operators publish the names their crawlers use, so you can address them individually.
  • SEO and monitoring tools may be useful if you use them, or simply add load if you do not.
  • Unknown or abusive bots often ignore robots.txt. Handling them requires server-level tools such as rate limiting or firewall rules.

There is no single right answer. A local service business that depends on being found may choose a very open policy. A publisher of original research may be more selective. Write the decision down, so future changes are deliberate rather than accidental. Revisit it when your business model changes, when you publish a new type of content, or when a crawler you have not seen before starts showing up regularly in your logs.

Page-level controls: the other half of the protocol

Robots.txt controls crawling, but it does not directly control what appears in search results. For that, there are page-level controls. In a March 14, 2025 post, Robots Refresher: page-level granularity, Google explained that robots meta tags date back to 1996, and that the X-Robots-Tag HTTP header was later added so the same controls could apply to non-HTML files such as PDFs and images.

These controls can tell crawlers not to index a page, not to follow its links or not to show a snippet. They can address all crawlers or a specific one. The post noted that when several valid directives apply to one crawler, the most restrictive valid directives apply, so a crawler-specific rule and a general rule are effectively combined.

One detail trips up many sites. The post explained that these instructions are sent together with a URL, so crawlers can only take them into account if they are not disallowed from crawling the URL through robots.txt. If you block a page in robots.txt and also add a noindex tag, the crawler may never see the tag.

The post suggested a practical division: use robots.txt for broader controls that cover large parts of a website, and page-level controls for individual pages. Some actions are only possible with one mechanism. Stopping the crawling of endless search result pages requires robots.txt, while preventing a snippet from being shown requires page-level controls.

Editing and testing your file safely

Google’s March 7 post pointed out that most content management systems offer a built-in way to change robots.txt, sometimes with checkboxes or a simple form, and that you can start a line with # to leave a comment explaining why a rule exists. It also noted that testing tools built by the web community can confirm the file works as intended.

A safe editing routine looks like this:

  1. Save a copy of the current file before changing anything.
  2. Add a comment above each new rule explaining its purpose and the date.
  3. Test the new rules against important URLs, such as the home page, service pages and product pages.
  4. Confirm the file loads with a normal response, not an error.
  5. Watch search traffic and server logs for a few weeks after the change.

Our web development team can review crawler rules, server responses and page-level controls together, so changes do not accidentally block important pages.

A smiling man in a mustard shirt looks at his phone on a teal sofa beside a red cushion.

Look at who is actually visiting

Rules are only half the picture. Server logs show which bots request pages, how often and which paths they visit. Even a simple monthly review can reveal crawlers you did not know about, bots hammering a search page or tools requesting old URLs that no longer exist.

Compare what you see with your written policy. If a crawler you intended to block keeps appearing, check whether it honors robots.txt and whether its name in your rules matches the token it actually uses. If unknown bots cause heavy load, consider server-level protections rather than more robots.txt lines.

Crawlers also follow the structure you build. Clear navigation and logical URLs help them find what matters, which is one reason good UI and UX design benefits machines as well as people.

A fictional example

Consider a fictional architecture firm whose robots.txt had not been touched since the site launched. A review found three problems. The file disallowed a folder named for confidential client drawings, advertising its existence. A rule meant to block a staging area also matched the public portfolio because of a missing trailing slash. And a firewall setting occasionally returned server errors for the robots.txt file itself.

The firm moved the client drawings behind a login, corrected the portfolio rule, fixed the firewall and decided, after discussion, to limit one AI training crawler while allowing search engines and other crawlers. Each rule now has a comment explaining why it exists, and the office manager reviews the server log summary each month.

A robots.txt review checklist

  • The file loads at the site root with a normal response.
  • No private paths are listed; private content requires a login.
  • Each rule has a comment explaining its purpose.
  • Rules are tested against key URLs.
  • Pages with noindex are not also blocked in robots.txt.
  • Decisions about search, AI and tool crawlers are written down.
  • Server logs are reviewed regularly for unexpected bots.
  • Heavy or abusive traffic is handled at the server level.

Bots will keep visiting whether or not you have a plan. Make sure the pages they find are worth reading, too; our copywriting team helps businesses create content that serves both visitors and the crawlers that bring them.