Robots.txt Generator

📖 What this does

robots.txt is a plain text file at the root of a site that tells crawlers which directories they may enter and which to leave alone. The syntax is small, but writing it by hand leads to two familiar mistakes: a path missing its leading slash, which silently disables the rule, and a pile of rules that contradict each other, so a directory you meant to allow is actually caught by a more specific disallow.

This tool turns rules into rows you can add and remove, then generates the file content grouped by User-agent. At the bottom it does two more things: it lists any rules whose format looks suspicious, and it gives you a pair of inputs to simulate whether a given crawler would be allowed or blocked on a given path.

The verdict follows what mainstream crawlers actually do: among several rules for the same crawler, the longest, most specific path wins, not the one written first. That is the opposite of what many people assume, which is exactly why the test box is there.

✨ Features

• Add and remove rules visually, each with a crawler name, an allow or disallow action, and a path
• Five presets: allow all, block all, block admin only, WordPress default, and block AI crawlers
• Output grouped by User-agent automatically, with repeated crawlers merged into one block
• Supports Crawl-delay, Host and any number of Sitemap lines
• Crawl verdict test: enter a crawler name and a path to see allow or block under longest-match resolution
• Live checks on path format and site-wide blocking risk, with each issue listed separately

🚀 How to use

1. Start from whichever preset is closest to what you need, then edit from there
2. Add your own rows with the add button; use * as the crawler name to apply a rule to every crawler
3. Fill in Crawl-delay, Host and Sitemap addresses if you need them
4. Type a crawler name and a concrete path into the test box and confirm the verdict matches what you expect
5. Download the file and upload it to the root of your domain

⚠️ Keep in mind

• This is not access control: robots.txt is a courtesy agreement. Scrapers that ignore it can still read the site, so anything genuinely private needs authentication behind it
• It only works at the root: the file is read from the domain root and nowhere else. The same file in a subdirectory is ignored
• Blocked pages can still be indexed: disallow stops crawling, not indexing. If other sites link to a blocked URL it may still surface in results, so pair it with noindex
• Case and trailing slashes matter: /Admin and /admin are two different rules, and whether a path ends with a slash changes what matches
• The AI crawler list moves: the names in the preset are the ones common today. New crawlers appear regularly, so revisit the list now and then

Two smaller things that catch people out. A blank Disallow is not the same as a disallow of slash: an empty value means nothing is excluded, which is how the allow-everything preset works, whereas a single slash excludes the entire site. And crawlers read the file once and cache it for a while, so after uploading a corrected version the change will not take effect instantly in the wild - expect a delay of up to a day before the new rules are honoured everywhere.