All tools (159)

Robots.txt Generator

Create a valid robots.txt with allow and disallow rules, crawl-delay and a sitemap line — free, in your browser.

Report a problem

How to use

  1. Pick a preset, or choose Custom and list the paths you want to block under Disallow (one per line).
  2. Optionally add a crawl-delay and your sitemap URL so search engines can find all your pages.
  3. Copy the result or download robots.txt, then upload it to the root of your website.

FAQ

Where do I put the robots.txt file?

In the root of your domain, so it’s reachable at https://yoursite.com/robots.txt. Search engines look for it there before crawling.

Does Disallow hide a page from Google completely?

No. Disallow stops crawling, but a blocked URL can still appear in results if it’s linked elsewhere. To keep a page out of the index, use a noindex meta tag (see our Meta Tag Generator) instead of, or as well as, robots.txt.

Is anything uploaded?

No. The file is built entirely in your browser from what you enter. Nothing is sent to a server, so it’s private and works offline.