Robots.txt Generator

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

How to make Robot By Using Google Robots File Generator?

Robots txt file is easy to make but people who aren’t aware of how to, they need to follow the following instructions to save time.

1. When you have landed on the page of New robots txt generator, you will see a couple of options, not all options are mandatory, but you need to choose carefully. The first row contains, default values for all robots and if you want to keep a crawl-delay. Leave them as they are if you don’t want to change them as shown in the below image:
2. The second row is about sitemap, make sure you have one and don’t forget to mention it in the robot’s txt file.
3. After this, you can choose from a couple of options for search engines if you want search engines bots to crawl or not, the second block is for images if you're going to allow their indexation the third column is for the mobile version of the website.
4. The last option is for disallowing, where you will restrict the crawlers from indexing the areas of the page. Make sure to add the forward slash before filling the field with the address of the directory or page.