go – the opposite of a sitemap. This is the first thing search engines look for when they index your site and is an essential component o
f a well-organized and professionally created website
Steps To follow:
Open Notepad
Write the first line. All robots.txt files start with this line:
User-Agent: *
Decide what areas you do not want to have indexed.
* If you want all parts of your site to be indexed, add this line:
Disallow:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Disallow: /images/
Disallow: /private/
Save the file. Use the name “robots.txt”
Upload to the main directory of your website.
Note:
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
source:wikihow.com
For More details visit: http://www.robotstxt.org/
f a well-organized and professionally created website
Steps To follow:
Open Notepad
Write the first line. All robots.txt files start with this line:
User-Agent: *
Decide what areas you do not want to have indexed.
* If you want all parts of your site to be indexed, add this line:
Disallow:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Disallow: /images/
Disallow: /private/
Save the file. Use the name “robots.txt”
Upload to the main directory of your website.
Note:
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
source:wikihow.com
For More details visit: http://www.robotstxt.org/
