Robots.txt Generator

What is a robots.txt file?

A robots.txt file is a plain text file at the root of your website (for example example.com/robots.txt). It tells search engine crawlers such as Googlebot and Bingbot which parts of your site they may crawl. A clear robots.txt saves crawl budget and keeps bots away from admin areas, search result pages and other low-value URLs.

How to create robots.txt with this generator

1. Choose whether all robots are allowed or disallowed by default.
2. List any folders you want to block, one per line, such as /wp-admin/ or /checkout/.
3. Optionally block individual bots, including AI crawlers like GPTBot and CCBot, or SEO crawlers like AhrefsBot.
4. Add the full URL of your XML sitemap.
5. Click Download and upload the file to the root folder of your domain.

Robots.txt rules explained

• User-agent: * – the rules below apply to every crawler.
• Disallow: /folder/ – do not crawl anything inside this folder.
• Disallow: (empty) – nothing is blocked.
• Crawl-delay: 10 – asks some bots (not Google) to wait between requests.
• Sitemap: – points crawlers to your sitemap so new pages are found faster.

Common mistakes to avoid

Never block your CSS or JavaScript folders, because Google needs them to render your pages. Remember that robots.txt stops crawling but not indexing: to keep a page out of Google, use a noindex robots meta tag instead, which you can create with our meta tag generator. If you do not have a sitemap yet, build one with the XML sitemap generator.

Example robots.txt for common websites

WordPress site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=

Sitemap: https://example.com/sitemap_index.xml

Online shop: also block cart, checkout and account pages, plus filter URLs that create endless combinations, for example Disallow: /cart/, Disallow: /checkout/ and Disallow: /*?filter=.

Wildcards and pattern matching

• * matches any sequence of characters: Disallow: /*?sessionid= blocks every URL containing that parameter.
• $ marks the end of a URL: Disallow: /*.pdf$ blocks PDF files.
• Rules are case-sensitive: /Admin/ and /admin/ are different paths.
• The most specific (longest) matching rule wins, so Allow: /wp-admin/admin-ajax.php overrides Disallow: /wp-admin/.

Blocking AI crawlers

Many site owners now choose whether AI companies may use their content for training. Well-known AI user agents include GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended (Google’s AI training control) and others. Blocking Google-Extended does not affect your normal Google Search rankings. Tick the bots you want to block and the generator adds the rules for you.

How to test your robots.txt

1. Upload the file and open yourdomain.com/robots.txt in a browser to confirm it is live.
2. In Google Search Console, open Settings → robots.txt to see the version Google fetched and any errors.
3. Use the URL Inspection tool on an important page to confirm it is “Crawl allowed”.

robots.txt, noindex and canonical: which to use?

Use robots.txt to save crawl budget on low-value areas. Use a noindex meta tag to keep a page out of search results (the page must stay crawlable so Google can see the tag). Use a canonical tag when two URLs show the same content and you want one to rank. Mixing these up is one of the most common technical SEO mistakes.

Frequently asked questions