Immortal SEO Logo
Back to ToolsSEO Tools

Robots.txt Generator

Create a properly formatted robots.txt file to control how search engines crawl your website.

What is a robots.txt file?

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests.

While robots.txt can be used to keep a web page out of Google, it's not a reliable method for preventing indexing. To keep a page out of Google, use more comprehensive methods like password protection or a noindex directive.

Important Notes:

  • Your robots.txt file should be placed at the root of your website (e.g., https://example.com/robots.txt).
  • Different crawlers might interpret the rules differently, so be specific in your directives.
  • For more advanced control, consider using meta robots tags or the X-Robots-Tag HTTP header.

Robots.txt Generator

Quick Templates:

User-Agent Rules:

Allow:

No allow directives

Disallow:

No disallow directives

Sitemaps:

Crawl Delay (Optional):

seconds

Note: Crawl-delay is not supported by all search engines.

Host Directive (Optional):

Note: Host directive is primarily used by Yandex.

Generated Code

# Your robots.txt will appear here
Download robots.txt

Next Steps:

  1. Download the generated robots.txt file
  2. Upload it to the root directory of your website
  3. Verify it works by visiting yourdomain.com/robots.txt
  4. Test your robots.txt using Google Search Console
Back to All Tools

Robots.txt Best Practices

Do:

  • Use robots.txt to prevent crawler overload and exclude non-public parts of your site
  • Include sitemap URL(s) in your robots.txt file
  • Be specific with your directives and user-agents
  • Test your robots.txt file using Google Search Console

Don't:

  • Rely solely on robots.txt for content security or preventing indexing
  • Block CSS and JavaScript files that help search engines render your pages
  • Use complex regular expressions that might confuse crawlers
  • Block important pages that should be indexed and discoverable

Need Help With Technical SEO?

Our technical SEO experts can help optimize your website's structure for better crawling, indexing, and ranking.

Learn More