Here you go:
1.On the Webmaster Tools Home page, click the site you want.
2.Under Site configuration, click Crawler access.
3.Click the Generate robots.txt tab.
4.Choose your default robot access. We recommend that you allow all robots, and use the next step to exclude any specific bots you don't want accessing your site. This will help prevent problems with accidentally blocking crucial crawlers from your site.
5.Specify any additional rules. For example, to block Googlebot from all files and directories on your site:1.In the Action list, select Disallow.
2.In the Robot list, click Googlebot.
3.In the Files or directories box, type /.
4.Click Add. The code for your robots.txt file will be automatically generated.
6.Save your robots.txt file by downloading the file or copying the contents to a text file and saving as robots.txt. Save the file to the highest-level directory of your site. The robots.txt file must reside in the root of the domain and must be named "robots.txt". A robots.txt file located in a subdirectory isn't valid, as bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location, but http://www.example.com/mysite/robots.txt is not.
1.On the Webmaster Tools Home page, click the site you want.
2.Under Site configuration, click Crawler access.
3.Click the Generate robots.txt tab.
4.Choose your default robot access. We recommend that you allow all robots, and use the next step to exclude any specific bots you don't want accessing your site. This will help prevent problems with accidentally blocking crucial crawlers from your site.
5.Specify any additional rules. For example, to block Googlebot from all files and directories on your site:1.In the Action list, select Disallow.
2.In the Robot list, click Googlebot.
3.In the Files or directories box, type /.
4.Click Add. The code for your robots.txt file will be automatically generated.
6.Save your robots.txt file by downloading the file or copying the contents to a text file and saving as robots.txt. Save the file to the highest-level directory of your site. The robots.txt file must reside in the root of the domain and must be named "robots.txt". A robots.txt file located in a subdirectory isn't valid, as bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location, but http://www.example.com/mysite/robots.txt is not.
No comments:
Post a Comment