How to block Googlebot from checking a certain directory of my website?

Epifrin · Mar 26th, 2008, 10:07

I plan to make a new database in a new directory of my website (mydomain.com/testing/) & I wouldn't want Google or any other search bots to index it as it has little or no relevance to do with the main purpose of the website. How can I do so? Any help is appreciated.

aso186 · Mar 26th, 2008, 10:32

Create a text file named robots.txt and add the following;

Code: Select all

User-agent: *
Disallow: /testing/

Place this file in the root folder of your server (www. yoursite .com).

welshstew · Mar 26th, 2008, 11:46

Remember that the robots.txt file will be browsable by the curious user / inquisitive little buggers, so they will still be able to see what directories you are trying to hide unless you change the access permissions on the file itself.

www.natwest.co.uk/robots.txt
http://www.bbc.co.uk/robots.txt