robots and hiding url's

welshstew · Aug 28th, 2007, 11:32

Hi,

I was wondering if anyone had any knowledge about how to hide urls from search engines and the curious.

I have a number of pages in development which I don't want to display quite yet. The obvious way is to implement a robots.txt file. However, this is easily viewable and the 'curious' user would be able to go straight to the url. i.e. www.natwest.co.uk/robots.txt brings up the folders that they want to hide.

the other option is to use robots meta data to hide the page. however, does this also hide the folder url?
i.e. if implemented it will hide www.webaddress.com/folder/default.asp but will it also hide www.webaddress.com/folder/

So I have probably confused you all, but in summery, I want to hide both the page and its associated file path. (www.webaddress.com/folder/default.asp and www.webaddress.com/folder/) and want to know if the meta tag for robots will do this

Any suggestions would be appreciated.

Thanks,

Stew

alexgeek · Sep 2nd, 2007, 02:15

this should help
http://www.robotstxt.org/wc/meta-user.html

just put this on the pages you don't want indexed
<meta name="robots" content="noindex">

HostingDeals · Sep 2nd, 2007, 04:19

Not all engines obey that Alex. I recommend using robots.txt for this. Wiki explains the basics of using it. http://en.wikipedia.org/wiki/Robots.txt

alexgeek · Sep 2nd, 2007, 10:56

ah okay,
i'm no SEO specialist sorry.
that's rob

Daniel · Sep 2nd, 2007, 11:19

Yeah as HostingPlus says the best way is by using robots.txt

welshstew · Sep 2nd, 2007, 12:33

Yeah, the problem is that a robots.txt file is browsable to the end user. i.e. http://www.bbc.co.uk/robots.txt

Which means that any inquisitive user can just have a quick look at one file to find anything that I don't want them to see (yet !).

Will the robots meta data hide the folder path (www.mysite.co/folder1/) as well as the direct url (www.mysite.co/folder1/default.asp) if I only put one page into that folder?

Daniel · Sep 2nd, 2007, 12:42

what about CHMOD?

welshstew · Sep 2nd, 2007, 14:08

Quote:

Originally Posted by DanDoughty

what about CHMOD?

bless you, nasty cold you have.....

not sure what that is but a bit of googling should so me the way.

thanks for the pointer Dan

welshstew · Sep 2nd, 2007, 14:12

Thanks Dan, read through some stuff and it looks like I'll need to use cacls (change access control list) as I use a windows environment.

Looks like it might be a bit of overkill though, so will probably stick with the robots meta and hope it keeps most people out.

Thanks for all your help.

Stew

Daniel · Sep 2nd, 2007, 14:24

CHMOD - Its the permissions - like 777, 000 etc.

alexgeek · Sep 2nd, 2007, 14:48

windows server doesn't use it

you just right click the file and change the permissions that way

Daniel · Sep 2nd, 2007, 15:34

Still CHMOD

HostingDeals · Sep 2nd, 2007, 16:52

Ok, it sounds like you might be trying to hide a sales page or download page or something. To ensure google is blocked, I'd suggest a password prompt that you'll give to your paying customers.

Log into your control panel and look for a authentication option.

Google won't be able to index it, you won't need robots.txt and your job is done.

Quote:

Originally Posted by welshstew

Yeah, the problem is that a robots.txt file is browsable to the end user. i.e. http://www.bbc.co.uk/robots.txt

Which means that any inquisitive user can just have a quick look at one file to find anything that I don't want them to see (yet !).

Will the robots meta data hide the folder path (www.mysite.co/folder1/) as well as the direct url (www.mysite.co/folder1/default.asp) if I only put one page into that folder?