Friday, 3 April 2015

What will the following robots.txt file do? User-agent:Googlebot Disallow:/*? User-agent:Scooter Disallow:

a. It will allow Google to crawl any of the dynamically generated pages. It will also allow thealtavista scooter     bot to access every page
b. It will disallow Google from crawling any of the dynamically generated pages. It will also disallow the             altavista scooter bot from accessing any page
c. It will disallow Google from crawling any of the dynamically generated pages. It will allow the altavista           scooter bot to access every page
d. None of the above

No comments:

Post a Comment