? IS "meta name="robots" content="?" necessary in pages ?

Love2Java · Jul 16th, 2007, 12:51

I've been working on my own site project for some time, and now I'm making fixes and updates, working out bugs, etc., with hopes of testing it live in the near future...

I have a question about the <meta name="robots" content="?"> tag, of which I know there are a few options for the content part, is it necessary to specify this in my web pages?
What exactly do these robots do?
Are there certain types of sites that specifying content="none" would be of benefit?
Besides analyzing a page, can they get into the java classes/servlets I've created for use on my site?
Can they retrieve personal info if there's code in my pages that actually do so?
Do I have reason to be concerned about these robots and should I specify the content in the tag as "none"?

Any input is greatly appreciated,
Thank You,
-Love2Java

1840dsgn · Jul 16th, 2007, 12:55

Quote:

What exactly do these robots do?

As far as I'm aware. Robots are the spiders sent out by search engines like Google, yahoo etc.

By changing the robots meta you can tell them not to index certain pages of your site.

This can also be done using a robots.txt file, which is the more common way of doing it.

For example on my server I've got my site, and also my CMS. My robots file tells the search engine that i don't want it to index my CMS files, this way they won't appear in Google.

Love2Java · Jul 17th, 2007, 13:26

Thank you for explaining this for me

The majority of my pages are JSP's (Java Server Pages)...Java is my core programming language. Therefore, I have a lot of Java Servlets that my JSP pages work with...should I assume that these robots are capable of getting into my class files if they are used in my pages?

I would like to know how "far" these bots are capable of going when digging around in web pages for indexing.

-Love2Java

imagn · Aug 6th, 2007, 23:21

They're really only looking for "live" text and images to both index and rank your respective pages.

You should use both the meta tag and robots.txt file

<meta name="robots" content="index, follow" />

http://www.gnc-web-creations.com/cre...tstxt_file.htm

Love2Java · Aug 8th, 2007, 13:12

Quote:

Originally Posted by imagn

What does "index, follow" mean?

karinne · Aug 8th, 2007, 13:28

A little google searching got this

HTML Author's Guide to the Robots META tag.

Quote:

The content of the Robots META tag contains directives separated by commas. The currently defined directives are [NO]INDEX and [NO]FOLLOW. The INDEX directive specifies if an indexing robot should index the page. The FOLLOW directive specifies if a robot is to follow links on the page. The defaults are INDEX and FOLLOW. The values ALL and NONE set all directives on or off: ALL=INDEX,FOLLOW and NONE=NOINDEX,NOFOLLOW.

Love2Java · Aug 8th, 2007, 13:48

Thank you for the explanation karinne, much appreciated