Crawler rules
About this robots.txt generator
A robots.txt file gives instructions to search-engine crawlers before they visit a site. It must be a plain text file at the root address, for example https://example.com/robots.txt. The rules can allow the whole site, stop crawlers from visiting selected paths or request that they avoid the whole site.
These rules are requests for well-behaved crawlers, not access control. The file is public, some bots may ignore it, and a blocked URL can still appear in search results without its page text. Never use robots.txt to protect passwords, customer data, admin pages or other private content.
The Sitemap line tells crawlers where the XML sitemap is. It does not replace submitting the sitemap in a search-engine account, but it gives crawlers another reliable way to find it.
What the lines mean
Paths are case-sensitive on many servers. Check the finished file carefully before publishing it.
| Line | Meaning |
|---|---|
| User-agent: * | The following rules apply to every crawler. |
| Allow: / | Crawlers may visit the whole site. |
| Disallow: /path/ | Ask crawlers not to visit this path and the URLs below it. |
| Sitemap: | Give the full public address of the XML sitemap. |
How to create a robots.txt file
- Choose crawler access: Allow the site, list paths to block, or block the entire site only when that is truly intended.
- Add the sitemap: Use the complete public URL of sitemap.xml. Review every generated rule.
- Publish at the root: Download or copy the file and place it at /robots.txt on the same host.
Frequently asked questions
Can robots.txt hide private information?
No. It is public and only asks crawlers not to visit. Protect private pages with authentication and correct server permissions.
Does Disallow remove a page from Google?
Not reliably. A blocked URL can still be listed if other pages link to it. To remove a page, let the crawler visit it and return a noindex directive, or use the search engine removal tools when appropriate.
Where must robots.txt be uploaded?
At the root of the exact host and protocol it controls, such as https://example.com/robots.txt. A file in /folder/robots.txt does not control the whole site.
Should I add Crawl-delay?
Usually not. Google does not support Crawl-delay in robots.txt, and other crawlers interpret it differently. Use the search engine settings or server controls if crawling causes load.