Robots.txt Generator
Form-based robots.txt builder with download.
Robots.txt Generator builds a clean robots.txt from form options—user-agent rules, Allow/Disallow paths, optional crawl-delay, host, and sitemap—locally in your browser with copy and download.
How it works
Enter a user-agent, list allow and disallow paths (one per line), and optionally add sitemap, crawl-delay, and host directives. The output updates live so you can copy or download robots.txt.
Benefits
- Form-driven robots.txt without memorizing directive order
- Copy and download the finished file
- Optional sitemap and crawl-delay fields
- Runs fully client-side
Use cases
- Draft production robots.txt for a marketing site
- Block staging environments from polite crawlers
- Add a Sitemap line pointing at your XML index
- Experiment with path rules before deploying
Best practices
- Keep robots.txt at the site root over HTTPS
- Do not rely on Disallow for secrecy
- Point Sitemap to a valid absolute URL
- Test with your search engine’s robots testing tools after deploy
Common mistakes
- Blocking CSS/JS needed for rendering
- Using robots.txt instead of noindex for thin pages you still crawl
- Assuming every bot honors Crawl-delay
- Leaving staging open with User-agent: * and empty Disallow
Privacy
Robots.txt Generator runs locally. wowTools does not receive your paths or sitemap URL unless you enable on-device history.
Frequently asked questions
- What does the Robots.txt Generator create?
- It builds a robots.txt from user-agent, Allow/Disallow paths, optional Crawl-delay, Host, and Sitemap lines—ready to copy or download.
- Does robots.txt guarantee pages stay private?
- No. robots.txt is a crawler courtesy file. Sensitive content needs authentication. Disallow does not encrypt or hide URLs from users.
- Is Crawl-delay widely supported?
- Crawl-delay is non-standard. Some crawlers honor it; Google generally ignores it. Prefer crawl controls in Search Console when available.
- Can I block indexing with robots.txt alone?
- Disallow can reduce crawling, but use noindex meta/headers when you need stronger indexation control for URLs that may still be discovered.
- Does this tool upload my rules?
- No. Generation runs entirely in your browser. Download the file and deploy it to your site root yourself.
Related tools
Sitemap XML Generator
seo
Build basic sitemap XML from a URL list.
Canonical URL Checker
seo
Client-side canonical URL heuristics (no crawl).
Meta Tag Preview
seo
SERP-style meta title and description preview.
Keyword Density Analyzer
seo
Local text stats and keyword density (no snake oil).
Open Graph Preview
seo
OG card layout mocks and meta tag export.
Slug Generator
seo
URL slugify with diacritic stripping, locally.