Feb 14th, 2008, 17:07
|
 |
SuperMember
|
|
Join Date: Dec 2007
Location: London
Age: 35
Posts: 288
Thanks: 6
Thanked 0 Times in 0 Posts
|
|
|
Re: Adivse on robots.txt
Quote:
|
To only allow Google (out of all the 'honest' bots), swap the two statements you have, so that you first declare all user-agents disallowed, then overwrite the permissions for Google.
|
like this?
- HTML: Select all
# robots.txt for http://www.mysite.com
User-agent: *
Disallow:/
User-agent: Google
Disallow:
* refers to all ?
Last edited by Oak; Feb 14th, 2008 at 17:45.
|