The robots.txt file is used to instruct search engine robots about
what pages on your website should be crawled and consequently indexed.
Most websites have files and folders that are not relevant for search
engines (like images or admin files) therefore creating a robots.txt
file can actually improve your website indexation.
A robots.txt is a simple text file that can be created with Notepad.
If you are using WordPress a sample robots.txt file would be:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
“User-agent: *” means that all the search bots (from Google, Yahoo,
MSN and so on) should use those instructions to crawl your website.
Unless your website is complex you will not need to set different
instructions for different spiders.
“Disallow: /wp-” will make sure that the search engines will not
crawl the WordPress files. This line will exclude all files and foldes
starting with “wp-” from the indexation, avoiding duplicated content
and admin files.
If you are not using WordPress just substitute the Disallow lines
with files or folders on your website that should not be crawled, for
instance:
User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/
After you created the robots.txt file just upload it to your root directory and you are done!