← All tools

Generator

Robots.txt Generator

Create a clean robots.txt file for common SEO setups, then adjust it before uploading to your site root.

Runs instantly. No account required.

How to use the robots.txt generator

  1. Enter your site URL, like https://example.com.
  2. List any paths to block, one per line, like /admin or /cart.
  3. Run the tool and copy the file.
  4. Save it as robots.txt and upload it to the root of your site.

If you leave the paths empty, the file allows everything, which is the right choice for most small sites.

Example robots.txt

User-agent: *
Disallow: /admin
Disallow: /cart
Sitemap: https://example.com/sitemap.xml

What to block (and what not to)

Usually fine to block: admin areas, cart and checkout pages, internal search results and staging copies of your site.

Don’t block: your CSS and JavaScript files, images you want in image search, or any page you want to rank. Google needs to load your page fully to understand it.

robots.txt and AI search

Many AI tools send their own crawlers, such as GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot. If you want your site cited in AI answers, make sure your robots.txt doesn’t block them by accident. If you want to opt out, add a separate group for each one.

Frequently asked questions

What is a robots.txt file?

It is a plain text file at the root of your site, like example.com/robots.txt. It tells search engine crawlers which parts of your site they may visit.

Does robots.txt keep a page out of Google?

Not reliably. A blocked page can still show up in results if other sites link to it. To keep a page out of search, use a "noindex" tag and let crawlers see it, or put it behind a login.

Where do I upload robots.txt?

It must be at the top level of your domain, such as https://example.com/robots.txt. A file in a subfolder is ignored.

Should I list my sitemap in robots.txt?

Yes. A Sitemap line helps search engines find your sitemap even if you never submit it in their tools. This generator adds it for you.

Can I block AI crawlers like GPTBot?

Yes. Add a group with "User-agent: GPTBot" and "Disallow: /". Keep in mind that blocking AI crawlers can also keep your site out of AI answers and citations.

What happens if I make a mistake?

A single "Disallow: /" under "User-agent: *" blocks your whole site from all crawlers. Always double-check before you upload, and test the file in Google Search Console.