Robots.txt Generator
Build a valid robots.txt with allow/disallow rules and a sitemap line.
Runs entirely in your browser — your input never leaves your deviceAbout this tool
A robots.txtfile tells search-engine crawlers which paths they may or may not request. This generator builds a syntactically valid file from simple inputs — user-agent, allow/disallow paths, an optional crawl-delay, and your sitemap URL — and previews it live.
Use the presets to allow or block all crawlers instantly, then refine. Place the result at the root of your site, e.g. https://yoursite.com/robots.txt.
Why use it
Valid output
Builds correct syntax from simple allow/disallow inputs.
Presets
Allow-all or block-all in one click, then refine.
Sitemap line
Add your sitemap URL to aid discovery.
Common use cases
- Create a robots.txt for a new site
- Block crawlers from private paths
- Add a crawl-delay
- Point search engines at your sitemap
Frequently asked questions
Where do I put robots.txt?
At the root of your domain — it must be reachable at https://yoursite.com/robots.txt. Crawlers only look there.
Does Disallow hide a page from Google?
No. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of the index, use a noindex meta tag instead.
Should I add my sitemap here?
Yes — adding a Sitemap: line helps search engines discover all your URLs. You can include the absolute sitemap URL in this generator.
Is an empty Disallow valid?
Yes. 'Disallow:' with nothing after it means 'allow everything' for that user-agent, which is the standard way to permit full crawling.