How to create a robots.txt file
- Start from a preset, then edit the rules.
- Add your sitemap addresses.
- Copy or download robots.txt and upload it to your site's top folder.
About this tool
Build a robots.txt file without remembering the syntax. Choose which crawlers a rule applies to, list the paths to block or allow, add your sitemap addresses and, if you want, ask AI training crawlers to stay out. Mistakes such as a path with no leading slash or a sitemap that is not a full address are pointed out. Then download the file.
Common questions
Does robots.txt hide a page?
No. It asks well-behaved crawlers not to visit. Anyone can still open the page, and blocked pages can still appear in search if other sites link to them.
Where does the file go?
In the top folder of your site, so it opens at your-site/robots.txt. A file in a subfolder is ignored.
Will blocking AI crawlers hurt my search ranking?
No. The AI crawlers listed are separate from Googlebot and Bingbot. Only crawlers that choose to obey robots.txt will comply.
Is it free to use?
Yes. There's no sign-up, and you can use it as often as you like.
Is my text uploaded anywhere?
No. This tool runs inside your browser, so what you enter stays on your device.
Do I need to install anything?
No. It works in your browser on a computer or a phone.