Free Robots.txt Generator

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Free Robots.txt Generator helps you to create robots.txt file.  You can specify which files/URLs are accessible to the search engine crawler's bots every time they visit your site.

What is Robots.txt file?

The robots.txt file is a plain text file that adheres to the Robots Exclusion Standard. Robot.txt contain several rules about allowed and refused URL for search engine bot visits. 

Robots.txt is located in the root directory of your website. So, the robots.txt file for the https://seocheckfree.com is located at https://seocheckfree.com/robots.txt.

How To Use Robots.txt Generator?

Follow these steps to free generate the best Robots.txt:

1. The default setting is allowed for all bots to crawl your site. You can choose refused if you need most of your links disallowed for search engines.
2. Specify the crawl delay from 0 to 120 seconds. It is the time interval the crawler bot crawls each URL. You can set slow interval if you feel bots visits has a bad impact on your server loads.
3. Robots.txt Generator needs your blog/website sitemap URL. Your sitemap XML is usually located on the root of your domain example.com/sitemap.xls. If you haven't created it, we provide XML Sitemap Generator to do it easily.
4. Specify permission of particular search engine robots.
5. If you have some private directories on your website, specify them om Restricted Directories field.
6. Create your Robots.txt by hit the button. You can choose to Create and Save as Robot.txt if you want it automatically downloaded to your computer.
7. Now, Create 'robots.txt' file at your root directory. Copy the above text and paste it into the text file.

Robots.txt important for SEO

SEO includes crawling, indexing and ranking. The correct robots.txt settings strongly support the first two aspects. Robots.txt ensures that the URL we want to be featured should be crawlable by search engine bots and indexed to get the best ranking.

In addition, robots.txt can prevent the appearance of URLs that will be categorized as duplicate content and are not required for indexing to appear on search results pages.

Robots.txt example

Robots.txt
User-agent: *
Disallow: /search
Allow: /search/howsearchworks
Disallow: /admin
Disallow: /user

Robots.txt tester

Google crawler bots respond to robots.txt every time they visit the website. To verify whether your robots.txt file is set correctly, you can use robot.txt tester from Google Webmaster.

Robots.txt disadvantages:

While it has many advantages, a robots.txt file also has some limitations.

Robots.txt cannot prevent access to specific directories.

For those of you who use the disallowed crawling command on some directories because they have secret files or content, robots.txt is good at preventing bots from accessing them but not effective enough to get visitors to access those directories.

Disallow all; there is still a possibility that it can still be crawled and indexed.

Robots.txt files prevent Google from finding or indexing content, but if a disallowed URL is linked from elsewhere on the web, we may still find and index it. Google search results will continue to show the page's URL and potentially other publicly available information like anchor text from links pointing to the page. Use the noindex meta tag, the response header, or removing the page altogether are effective ways to keep your URL from appearing in Google search results.