Stop Google putting a website into the search engine

I hope this is the right place to post this, but how do i stop google putting a website into google? it used cpanel. just wondering how to go about doing this

Hmm I dont really understand why you wouldnt... If im not mistaken there isnt anyway to bypass it to my knowledge but i could be wrong

Create a robots.txt file at the root of your site, and copy and paste this snippet;

Code: Select all

User-agent: Google
Disallow: /

This instructs the Google agent to ignore all content from your root downwards (i.e. everything), but will not affect other spiders and SE's.

However, if your site is already in Google's index, it will take a while to propagate the changes. You may even have to user Google's Webmaster Tools to instruct Google to manually remove your site from it's index.

only a .txt file? thought it would have been a .html or something like that. no my website is not in the search engine yet, the the hosting still needs 24 hours, what do you mean by the root? i have no idea where to put it lol >.<

Oh i forgot to say, i would like it not to appear in any search engine lol, sorry.

In which case, use;

Code: Select all

User-agent: *
Disallow: /

The asterisk (*) is a 'wildcard' that applies to all user agents.

The robots.txt file is a web standard for instructing spiders which areas of your site you wish to be included / excluded in their crawl (more information).

Note that all the major players (Google, Yahoo, MSN etc.) obey the robots.txt convention, but there are spiders out there that do not (namely insignificant spiders and spambots).
The root is the 'top level' of your website (i.e. www.example.com, not www.example.com/folder/)

As Aso has said, a robots.txt file is a file that all search engines request before spidering any content - it tells them what they CANT have on your site....

The content of the file Aso has posted above basically tells all robots to go away as they cant have anything.

Upload the file to the root of your website so it is accessible via yourdomain.com/robots.txt

__________________
Click the 'Thanks!' button if this post has helped you

Rob - Webforumz Founder

But im waiting for the hosting to go through now, but the second it gets up and ready google is going to be swarming it with spiders and stuff. or will it not do that until i have content on it? lol. I have never used a cpanel before so ya know

cPanel's got nothing to do with Google, so don't worry!

cPanel is simply one of many popular systems for managing common (and complex) server-related tasks.

Google can find out about your site pretty much no matter what, but so long as that robots.txt file is hosted in your main website folder, that'll be the first thing Google checks. And as soon as it realises it's not wanted, it'll leave!

Plus until your hosting is set up, Google won't be able to index anything anyway!

The robots.txt file is like the 'Keep of the Grass' sign at the local park. The kids/robots get to the edge/domain and see/read the sign/file and are nice good kids/robots and obey the sign/file.

__________________
Jack Franklin - Webforumz Moderator
(x)HTML | CSS | PHP | MySQL | JQuery (Javascript)
Contact: My Blog | Twitter | Delicious
Want Lessons? PM me.
If you think I've helped, please press the 'Thanks' Button.

a little off topic but as its he has a solution...

why? lol
i thought everyone was tryin to be found :S
sorry, i just wondered what this would be useful for?

cheers

Mick, sometimes during the development phases of a site when you would NOT want people arriving at the site then it's useful - personally, I always password protect the site but others use this among other methods to keep people and bots out.

__________________
Click the 'Thanks!' button if this post has helped you

Rob - Webforumz Founder

Actually, while we're on the subject - if you block Google (and other bots) with a robots.txt file, I'm guessing it (they) will still keep coming back every now and then to check if it's still wanted or not?

ah thats quite a nifty trick then, didnt think of that.

cheers

You would think so, logically.

__________________
Jack Franklin - Webforumz Moderator
(x)HTML | CSS | PHP | MySQL | JQuery (Javascript)
Contact: My Blog | Twitter | Delicious
Want Lessons? PM me.
If you think I've helped, please press the 'Thanks' Button.

I reckon that they will come back but not so often, however I don't know this for fact.

Tryed that password thing but failed

So this robots file has no script tags or anything? just that text?

Yup. That's it.

How did you try password protection? If you're running Apache (90% chance you are), you can use htaccess