Robots.txt Generator
Developer ToolsCreate a robots.txt file for your site
FreePrivateInstant
Similar tools you may like
History
All history →Robots.txt Generator — at a glance
Tell search engines which parts of your website they may visit, with a ready-to-upload robots.txt file.
Blocked: /admin/
Sitemap URL
Run
User-agent: *
Disallow: /admin/
Sitemap: https://…
How to use it — step by step
No input needed — choose the crawler, blocked folders and your sitemap link.
1
Enter your values
Fill in the boxes. Sensible example values are already filled in for you.
2
Generate robots.txt
Press the button. The answer is worked out instantly on your device.
3
Read or copy the result
See the big answer and the extra details, then press Copy to keep them.
Pro tips
Small things that make the result better.
✦Upload the file to the root of your site: yoursite.com/robots.txt.
✦Never use robots.txt to hide private data — it is public.
✦Always include your sitemap URL to help Google find pages.
Quick facts
3crawler choices
Sitemapline included
Validstandard syntax
Freeno signup
Frequently Asked Questions
Where do I put robots.txt?+
In the main folder of your domain so it opens at https://yoursite.com/robots.txt.
Does Disallow remove a page from Google?+
Not always. It only stops crawling. Use a noindex tag to keep a page out of search results.
Is my data safe?+
Yes. Everything runs inside your own browser. Your text, files and images are never uploaded to any server.
More Free Tools
Guides you might like
All guides →Sitemap XML Generatorsitemap.xml explained: what to include, the 50,000 URL limit and HTML sitemapsChmod Calculatorchmod permissions explained: what 755, 644 and 777 actually meanNumber Base ConverterBinary, hex and decimal: counting in bases 2, 10 and 16HTTP Status Code LookupHTTP status codes: the five families, and the dozen you will actually meet