Robots.txt file is a very important file if you want to have a good ranking on search engines, many websites don't offer this file. A Robots.txt file is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc. It defines which paths are off limits for spiders to visit. This is useful if you want to hide some personal information or some secret files.

What is Robots.txt
The Robots Exclusion Protocol (REP) is a group of web standards that regulate Web robot behavior and search engine indexing. In simple language, we can say Robots.txt file is a special text file that is always located in your Web server's root directory which contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

Example Robots.txt Format

Allow indexing of everything
User-agent: *
Disallow:

Disallow indexing of everything
User-agent: *
Disallow: /

Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder
User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html