What is robots.txt? Definition, syntax and examples

How does robots.txt work?

Robots.txt works as a set of crawl rules that a well-behaved crawler downloads and reads before it requests any other URL on the host. Google describes its main purpose as preventing your server from being overloaded with requests. Compliance is voluntary: Googlebot, Bingbot and other reputable crawlers follow it, but scrapers may not, so it is not a security tool.

Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it, usually without a description. To keep a page out of the index, use noindex and let Google crawl the page so it can see the tag.

Robots.txt syntax and rules

A robots.txt file consists of groups that start with a user-agent line followed by allow and disallow rules. Google supports 4 fields: user-agent, allow, disallow and sitemap. A simple example as plain text:

• User-agent: *
• Disallow: /cart/
• Disallow: /*?sort=
• Allow: /cart/help/
• Sitemap: https://example.com/sitemap.xml

The layers below summarize how Google interprets the file.

Location and scope

The file must sit at the root of the host, for example https://example.com/robots.txt. It applies only to that exact protocol, host and port, so shop.example.com and example.com need separate files.

Wildcards and precedence

The asterisk matches any sequence of characters and the dollar sign marks the end of a URL. Paths are case-sensitive. When rules conflict, Google applies the most specific rule (the longest matching path), and on a tie the least restrictive one wins.

HTTP status codes

A 4xx response (except 429) means no restrictions at all. A 5xx response makes Google pause crawling for 12 hours, then use the last cached copy for up to 30 days. Google follows up to 5 redirect hops to reach the file.

Unsupported rules

Google ignores crawl-delay, nofollow and noindex in robots.txt. It stopped honoring noindex there on 1 September 2019, noting that such rules were contradicted by other rules in all but 0.001% of files. Bing, by contrast, supports crawl-delay as a relative throttle value.

Robots.txt in SEO and AI search

In SEO, robots.txt is a tool for spending crawl budget on pages that matter: blocking internal search results, endless filter combinations and cart pages on large e-commerce sites. It should never block CSS or JavaScript that Google needs to render the page, because that changes how Google understands the content.

Robots.txt is also where you control AI crawlers. Google offers the Google-Extended token, which decides whether Google may use your content to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google states that it does not affect inclusion or ranking in Google Search. Other AI crawlers identify themselves with their own user-agent names, so you can allow or block each one in a separate group.

Robots.txt on multilingual and multi-country sites

On international sites, robots.txt mistakes can silently remove an entire market from Google. Each ccTLD and each subdomain (de.example.com, cz.example.com) needs its own file, and the rules should match across versions. Google’s guidance on locale-adaptive pages says robots rules should be identical for all locales. Blocking /en/ during a staging phase and forgetting to remove the rule is a mistake I have seen more than once after a language launch.

Blocking hreflang alternates is equally harmful. If Google cannot crawl the German version, it cannot confirm the return link in your hreflang set, and the annotations for that pair are lost. Check robots.txt for every host in the Search Console robots.txt report after each deployment.

Robots.txt review is part of every technical SEO project I run, and of my international SEO audits for sites with several domains. Pair it with a clean XML sitemap.