In this article
Before a well-behaved crawler visits any page on your site, it first asks for one file: /robots.txt. The file lists which parts of the site it may crawl and which it should skip. It is simple, and that is the danger: a single wrong line can hide an entire website from search.
The shape of the file
The file is plain text, built from groups. Each group starts with the crawler it addresses, followed by the rules for it.
| Directive | Meaning | Example |
|---|---|---|
User-agent | Which crawler the group applies to. * means all. | User-agent: Googlebot |
Disallow | A path the crawler should not fetch | Disallow: /admin/ |
Allow | A path that is allowed, used to carve out exceptions | Allow: /admin/help/ |
Sitemap | Full address of your XML sitemap, outside any group | Sitemap: https://example.com/sitemap.xml |
Paths are case-sensitive and start with a slash. /Admin/ and /admin/ are different.
A complete example
- 1User-agent: *The rules below apply to every crawler that has no group of its own.
- 2Disallow: /admin/Do not crawl anything under /admin/.
- 3Disallow: /cart/Skip the shopping cart. It has no value in search.
- 4Allow: /admin/help/But the help pages inside /admin/ may be crawled.
- 5Sitemap: https://example.com/sitemap.xmlTell crawlers where the full list of pages is.
A crawler follows the most specific matching group. A named group such as Googlebot replaces the * group for that crawler. It does not add to it.
How crawlers choose between rules
Rules can overlap. Google and most major crawlers pick the longest matching path, because it is the most specific. If an Allow and a Disallow match with equal length, the less restrictive one, Allow, wins.
| URL | Matching rules | Result |
|---|---|---|
/admin/login | Disallow: /admin/ | Blocked |
/admin/help/faq | Disallow: /admin/ and Allow: /admin/help/ (longer) | Allowed |
/cart/ | Disallow: /cart/ | Blocked |
/blog/post-1 | No rule matches | Allowed (the default) |
Anything not mentioned is allowed. You only list what you want kept out.
Wildcards and the end marker
Two special characters work in Google, Bing and most other major crawlers. Both can save you from writing hundreds of lines.
Disallow: /*?sort= blocks any URL containing ?sort=.Disallow: /*.pdf$ blocks addresses that end in .pdf./shop also matches /shop-sale. /shop/ matches only inside the folder.Test a new pattern before publishing. A star in the wrong place can match far more than you meant.
The mistakes that hurt most
- An empty value means no restrictions
- Every page may be crawled
- Often used as an explicit 'allow all'
- The slash means the whole site
- Crawlers stop fetching all pages
- Pages can drop out of search over time
This often happens when a staging file is copied to the live site. Check robots.txt after every launch.
- Staging rules on the live site. A leftover
Disallow: /from a test server is the classic cause of a sudden loss of traffic. - Blocking CSS and JavaScript. If a crawler cannot load your styles and scripts, it cannot see the page as visitors do.
- Using robots.txt to hide pages. The file is public. Listing
/secret-offers/advertises exactly where it is. - Wrong location. It must be at the root of each host.
blog.example.comneeds its own file. - Blocking a page you want removed. If a page is blocked, the crawler never sees a noindex tag on it.
robots.txt against noindex
These two tools solve different problems and are often confused.
| Goal | Use | Why |
|---|---|---|
| Save crawl effort on low-value URLs | Disallow in robots.txt | The crawler does not fetch the page at all |
| Keep a page out of search results | noindex meta tag or header | The crawler must be able to fetch the page to see it |
| Protect private data | Passwords or access rules on the server | robots.txt is a request, not a lock |
| Remove an old page quickly | Removal tool in Search Console, then noindex or 404 | Blocking alone can leave the address listed |
Never combine Disallow and noindex on the same page and expect noindex to work.
Limits and details worth knowing
- Size. Google reads about the first 500 KiB of the file and ignores the rest.
- Encoding. Save as plain UTF-8 text with the name in lowercase:
robots.txt. - Status code. A 404 for the file means 'no restrictions'. A server error (5xx) can make Google pause crawling the site.
- Crawl-delay. Some crawlers honour it. Google ignores it.
- Caching. Crawlers cache the file, often for up to a day, so changes may take a while to be noticed.
- AI crawlers. Many publishers now add groups for specific AI user-agents. Each one needs its own
User-agentline.
Before you publish
- Open the file in a browser at yoursite.com/robots.txt to confirm it loads.
- Search for `Disallow: /` with nothing after the slash, and be sure it is meant.
- Test your important URLs in the robots.txt report or tester in Search Console.
- Add the Sitemap line with the full https address.
- Recheck after every redesign or move to a new server.
Questions people ask
What is robots.txt?
It is a text file at the root of a website that tells crawlers which paths they may or may not fetch. It uses the directives User-agent, Disallow, Allow and Sitemap.
Does Disallow remove a page from Google?
Not reliably. It stops crawling, but the address can still be listed if other pages link to it. To keep a page out of results, use a noindex tag on a page that is not blocked.
What does Disallow: / do?
It blocks the entire site for the crawlers in that group. A Disallow line with no value blocks nothing.
Where should robots.txt be placed?
At the root of the host, for example https://example.com/robots.txt. Files in subfolders are not read, and each subdomain needs its own.
Can I use wildcards in robots.txt?
Yes. The star matches any sequence of characters and the dollar sign marks the end of a URL. For example, Disallow: /*.pdf$ blocks addresses ending in .pdf.
Is robots.txt a security measure?
No. It is a public request that only cooperative crawlers follow. Protect private pages with a login or server access rules.