Adivse on robots.txt

Oak · Feb 14th, 2008, 05:40

Hi,

Please give advise about adding a robots.txt file to your website directory and search engines.

Thanks

unitedcraig · Feb 14th, 2008, 08:44

i think this will help you

http://www.robotstxt.org/

craig

Oak · Feb 14th, 2008, 16:16

Yes I have been on that page but I am still not sure. What is best practice, for allowing and disallowing?

Will this do?

Quote:

# robots.txt for http://www.mysite.com

User-agent: Google
Disallow:

User-agent: *
Disallow: /

Thanks

spinal007 · Feb 14th, 2008, 16:25

The above will stop every search engine from indexing your entire website.
It will work if that's what you want.

Oak · Feb 14th, 2008, 16:35

Oh.. I thought that would stop all search engines except for Google. Please can you recommend best procedure. I mean obviously I don't want spam or hackers, what should I do?

aso186 · Feb 14th, 2008, 16:59

If you're trying to protect against spam-bots, robots.txt isn't going to help you, since most evils will ignore it anyway

To only allow Google (out of all the 'honest' bots), swap the two statements you have, so that you first declare all user-agents disallowed, then overwrite the permissions for Google.

Oak · Feb 14th, 2008, 17:07

Quote:

To only allow Google (out of all the 'honest' bots), swap the two statements you have, so that you first declare all user-agents disallowed, then overwrite the permissions for Google.

like this?

HTML: Select all

# robots.txt for http://www.mysite.com

User-agent:  *
Disallow:/

User-agent: Google
Disallow:

* refers to all ?

aso186 · Feb 14th, 2008, 18:06

Yeah. But you might want to wait for someone else to confirm this (I'm not 100% sure).

EDIT: Just checked it with Google Webmaster tools. All's good except you need to use Googlebot instead of just 'Google'.

Oak · Feb 14th, 2008, 18:12

hmm

the code bellow was taken from http://www.robotstxt.org/robotstxt.html

HTML: Select all

[b]To allow a single robot[/b]

  User-agent: Google
Disallow:

User-agent: *
Disallow: /

aso186 · Feb 14th, 2008, 18:18

Using Google's robots.txt checker, it confirms that Google is not registered by the Googlebot.

You must use Googlebot in order for it to work. Otherwise, it doesn't matter which order the statements are.

Oak · Feb 14th, 2008, 18:20

Ok, you understand it better than I do!

Oak · Feb 15th, 2008, 09:22

That's done now. Cheers!

Are there any other bots that should be allowed?

spinal007 · Feb 15th, 2008, 11:20

Ok, my 2 cents:
robots.txt is a convention. robots.txt is not a way of protecting your website, it's just a way of telling bots how you'd like them to behave. but there's no guarantee that they'll obey it.

If you want to stop bad bots, use server-side scripting. check the user agent, respond with 404 if you don't like it. but even then, some bad bots will identify themselves as a browser, so you'll never know....

andyf · Feb 15th, 2008, 20:06

It's very difficult to classify bad/good bots, some bots just crawl your site to steal your content or emai ID or some for helping you for indexing your content for SE's.
You have to study your log file for deciding who are helpful bots and accordingly should are allowed.
Not all bots obey robots.txt file

Oak · Feb 16th, 2008, 12:09

Cheers, it's becoming more clear!

Quote:

If you want to stop bad bots, use server-side scripting. check the user agent, respond with 404 if you don't like it. but even then, some bad bots will identify themselves as a browser, so you'll never know....

Ok well I would not know how to set this up .

Quote:

You have to study your log file for deciding who are helpful bots and accordingly should are allowed.
Not all bots obey robots.txt file

Log file?

spinal007 · Feb 16th, 2008, 12:26

My point is: what you are trying to do requires a lot of work if you want to make it reliable - otherwise it's just a waste of time on something that doesn't actually work. And, in most cases, you cannot stop anyone from crawling your pages. Unless you password protect them.

But why do you want to do this anyway? What's the issue?

Oak · Feb 16th, 2008, 15:12

Ok, I take your meaning.

I thought that you were meant to specify search engines that you want to search your site, so I did that for Google bot, but that's it. So no other reason than that.

spinal007 · Feb 16th, 2008, 15:23

So, to sum up:
The purpose of robots.txt is to ask a bot not to index your site.
You don't need this if you want a bot to index your site.
And if you didn't, this wouldn't work anyway.

Case closed!