Does robots.txt guarantee pages stay private?

No. robots.txt is a crawler courtesy file. Sensitive content needs authentication. Disallow does not encrypt or hide URLs from users.

Is Crawl-delay widely supported?

Crawl-delay is non-standard. Some crawlers honor it; Google generally ignores it. Prefer crawl controls in Search Console when available.

Can I block indexing with robots.txt alone?

Disallow can reduce crawling, but use noindex meta/headers when you need stronger indexation control for URLs that may still be discovered.

Does this tool upload my rules?

No. Generation runs entirely in your browser. Download the file and deploy it to your site root yourself.

Robots.txt Generator

Form-based robots.txt builder with download.

Rules

User-agent

Disallow paths (one per line)

Allow paths (one per line)

Sitemap URL

Crawl-delay (optional)

Host (optional)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

File ready

robots.txt ready

Robots.txt Generator builds a clean robots.txt from form options—user-agent rules, Allow/Disallow paths, optional crawl-delay, host, and sitemap—locally in your browser with copy and download.

How it works

Enter a user-agent, list allow and disallow paths (one per line), and optionally add sitemap, crawl-delay, and host directives. The output updates live so you can copy or download robots.txt.

Benefits

Form-driven robots.txt without memorizing directive order
Copy and download the finished file
Optional sitemap and crawl-delay fields
Runs fully client-side

Use cases

Draft production robots.txt for a marketing site
Block staging environments from polite crawlers
Add a Sitemap line pointing at your XML index
Experiment with path rules before deploying

Best practices

Keep robots.txt at the site root over HTTPS
Do not rely on Disallow for secrecy
Point Sitemap to a valid absolute URL
Test with your search engine’s robots testing tools after deploy

Common mistakes

Blocking CSS/JS needed for rendering
Using robots.txt instead of noindex for thin pages you still crawl
Assuming every bot honors Crawl-delay
Leaving staging open with User-agent: * and empty Disallow

Privacy

Robots.txt Generator runs locally. ToolHub does not receive your paths or sitemap URL unless you enable on-device history.

Frequently asked questions

What does the Robots.txt Generator create?: It builds a robots.txt from user-agent, Allow/Disallow paths, optional Crawl-delay, Host, and Sitemap lines—ready to copy or download.
Does robots.txt guarantee pages stay private?: No. robots.txt is a crawler courtesy file. Sensitive content needs authentication. Disallow does not encrypt or hide URLs from users.
Is Crawl-delay widely supported?: Crawl-delay is non-standard. Some crawlers honor it; Google generally ignores it. Prefer crawl controls in Search Console when available.
Can I block indexing with robots.txt alone?: Disallow can reduce crawling, but use noindex meta/headers when you need stronger indexation control for URLs that may still be discovered.
Does this tool upload my rules?: No. Generation runs entirely in your browser. Download the file and deploy it to your site root yourself.

Sitemap XML Generator

seo

Build basic sitemap XML from a URL list.

sitemap
xml
seo
urlset

Canonical URL Checker

seo

Client-side canonical URL heuristics (no crawl).

canonical
seo
url
duplicate

Meta Tag Preview

seo

SERP-style meta title and description preview.

meta
seo
serp
title

Keyword Density Analyzer

seo

Local text stats and keyword density (no snake oil).

keyword
density
seo
text

Open Graph Preview

seo

OG card layout mocks and meta tag export.

open-graph
og
seo
social

Slug Generator

seo

URL slugify with diacritic stripping, locally.

slug
url
seo
permalink