# robots.txt explained: Allow, Disallow, wildcards and the mistakes that block your site

> One small text file can tell search engines to crawl your whole site or none of it. This guide covers the four directives that matter, how crawlers pick between conflicting rules, why robots.txt is not a security tool, and the single-character slip that has taken sites out of search.

Published 2026-10-09 by Toolcloak. 4 minute read. Original: https://toolcloak.pages.dev/articles/robots-txt-explained-allow-disallow-wildcards-and-the-mistakes-that-block-your-site
Related free tool: [Robots.txt Generator](https://toolcloak.pages.dev/tools/robots-txt-generator)

Before a well-behaved crawler visits any page on your site, it first asks for one file: **/robots.txt**. The file lists which parts of the site it may crawl and which it should skip. It is simple, and that is the danger: a single wrong line can hide an entire website from search.

> **In short**
> - The file must live at the **root** of the host: `https://example.com/robots.txt`. A copy in a subfolder is ignored.
> - Four directives do almost everything: **User-agent**, **Disallow**, **Allow** and **Sitemap**.
> - `Disallow: /` blocks the **whole site**. `Disallow:` with nothing after it blocks nothing.
> - robots.txt controls **crawling**, not indexing, and it is **not security**. Anyone can read it, and a blocked URL can still appear in results if other sites link to it.

## The shape of the file

The file is plain text, built from groups. Each group starts with the crawler it addresses, followed by the rules for it.

| Directive | Meaning | Example |
| --- | --- | --- |
| `User-agent` | Which crawler the group applies to. `*` means all. | `User-agent: Googlebot` |
| `Disallow` | A path the crawler should not fetch | `Disallow: /admin/` |
| `Allow` | A path that is allowed, used to carve out exceptions | `Allow: /admin/help/` |
| `Sitemap` | Full address of your XML sitemap, outside any group | `Sitemap: https://example.com/sitemap.xml` |

*Paths are case-sensitive and start with a slash. `/Admin/` and `/admin/` are different.*

Try it: [Robots.txt Generator](https://toolcloak.pages.dev/tools/robots-txt-generator). Choose crawlers, add allow and disallow paths and your sitemap, then copy the finished file.

## A complete example

**A typical small-site file, read line by line**

1. **User-agent: *.** The rules below apply to every crawler that has no group of its own.
2. **Disallow: /admin/.** Do not crawl anything under /admin/.
3. **Disallow: /cart/.** Skip the shopping cart. It has no value in search.
4. **Allow: /admin/help/.** But the help pages inside /admin/ may be crawled.
5. **Sitemap: https://example.com/sitemap.xml.** Tell crawlers where the full list of pages is.

*A crawler follows the most specific matching group. A named group such as Googlebot replaces the * group for that crawler. It does not add to it.*

## How crawlers choose between rules

Rules can overlap. Google and most major crawlers pick the **longest matching path**, because it is the most specific. If an Allow and a Disallow match with equal length, the less restrictive one, Allow, wins.

| URL | Matching rules | Result |
| --- | --- | --- |
| `/admin/login` | `Disallow: /admin/` | Blocked |
| `/admin/help/faq` | `Disallow: /admin/` and `Allow: /admin/help/` (longer) | Allowed |
| `/cart/` | `Disallow: /cart/` | Blocked |
| `/blog/post-1` | No rule matches | Allowed (the default) |

*Anything not mentioned is allowed. You only list what you want kept out.*

## Wildcards and the end marker

Two special characters work in Google, Bing and most other major crawlers. Both can save you from writing hundreds of lines.

**Pattern characters**

| Box | What it means |
| --- | --- |
| * (star) | Matches any run of characters. `Disallow: /*?sort=` blocks any URL containing `?sort=`. |
| $ (end) | Matches the end of the URL. `Disallow: /*.pdf$` blocks addresses that end in .pdf. |
| Trailing slash | `/shop` also matches `/shop-sale`. `/shop/` matches only inside the folder. |
| No regex | Only these two characters are special. Full regular expressions do not work. |

*Test a new pattern before publishing. A star in the wrong place can match far more than you meant.*

## The mistakes that hurt most

**One character apart**

| Blocks nothing | Blocks everything |
| --- | --- |
| Disallow: | Disallow: / |
| An empty value means no restrictions | The slash means the whole site |
| Every page may be crawled | Crawlers stop fetching all pages |
| Often used as an explicit 'allow all' | Pages can drop out of search over time |

*This often happens when a staging file is copied to the live site. Check robots.txt after every launch.*

1. **Staging rules on the live site.** A leftover `Disallow: /` from a test server is the classic cause of a sudden loss of traffic.
2. **Blocking CSS and JavaScript.** If a crawler cannot load your styles and scripts, it cannot see the page as visitors do.
3. **Using robots.txt to hide pages.** The file is public. Listing `/secret-offers/` advertises exactly where it is.
4. **Wrong location.** It must be at the root of each host. `blog.example.com` needs its own file.
5. **Blocking a page you want removed.** If a page is blocked, the crawler never sees a noindex tag on it.

## robots.txt against noindex

These two tools solve different problems and are often confused.

| Goal | Use | Why |
| --- | --- | --- |
| Save crawl effort on low-value URLs | `Disallow` in robots.txt | The crawler does not fetch the page at all |
| Keep a page out of search results | `noindex` meta tag or header | The crawler must be able to fetch the page to see it |
| Protect private data | Passwords or access rules on the server | robots.txt is a request, not a lock |
| Remove an old page quickly | Removal tool in Search Console, then noindex or 404 | Blocking alone can leave the address listed |

*Never combine Disallow and noindex on the same page and expect noindex to work.*

> **It is not security:** Well-behaved crawlers obey robots.txt. Malicious bots ignore it and may use it as a map. Anything private belongs behind a login.

## Limits and details worth knowing

- **Size.** Google reads about the first 500 KiB of the file and ignores the rest.
- **Encoding.** Save as plain UTF-8 text with the name in lowercase: `robots.txt`.
- **Status code.** A 404 for the file means 'no restrictions'. A server error (5xx) can make Google pause crawling the site.
- **Crawl-delay.** Some crawlers honour it. Google ignores it.
- **Caching.** Crawlers cache the file, often for up to a day, so changes may take a while to be noticed.
- **AI crawlers.** Many publishers now add groups for specific AI user-agents. Each one needs its own `User-agent` line.

## Before you publish

1. **Open the file in a browser** at yoursite.com/robots.txt to confirm it loads.
2. **Search for `Disallow: /`** with nothing after the slash, and be sure it is meant.
3. **Test your important URLs** in the robots.txt report or tester in Search Console.
4. **Add the Sitemap line** with the full https address.
5. **Recheck after every redesign or move** to a new server.

## Frequently asked questions

### What is robots.txt?

It is a text file at the root of a website that tells crawlers which paths they may or may not fetch. It uses the directives User-agent, Disallow, Allow and Sitemap.

### Does Disallow remove a page from Google?

Not reliably. It stops crawling, but the address can still be listed if other pages link to it. To keep a page out of results, use a noindex tag on a page that is not blocked.

### What does Disallow: / do?

It blocks the entire site for the crawlers in that group. A Disallow line with no value blocks nothing.

### Where should robots.txt be placed?

At the root of the host, for example https://example.com/robots.txt. Files in subfolders are not read, and each subdomain needs its own.

### Can I use wildcards in robots.txt?

Yes. The star matches any sequence of characters and the dollar sign marks the end of a URL. For example, Disallow: /*.pdf$ blocks addresses ending in .pdf.

### Is robots.txt a security measure?

No. It is a public request that only cooperative crawlers follow. Protect private pages with a login or server access rules.