chmod 750 crawlable?

Blake121 · Jan 8th, 2008, 23:39

If I chmoded a directory on my server to 750 (Forbidden) would a search engine be able to crawl it?

Also even if they can't, would they index the directory since they have found it?

Or would I have to use a robots.txt file?

alexgeek · Jan 9th, 2008, 06:05

Use a robots.txt, although a search engine would only crawl it if there's a link to it.

aso186 · Jan 9th, 2008, 15:15

Like Alex says, the spider could only find it if there was a link to it.

Spiders do not crawl your server, they simply crawl what they are served. In other words, if you can't access it in a browser, then neither can the spider.

To play safe, pop this in a robots.txt file at the root of your server

Code: Select all

User-Agent: *
Disallow: /directory/

The asterisk is a 'wildcard' for all spider agents. You can specify certain agents for certain rules. Check out this guide on robots.txt or google robots.txt.

Blake121 · Jan 9th, 2008, 16:57

Yeah, yeah. I know about robots.txt, Just I didn't want to use one.

It's quite obvious to find directory's you're not supposed to be in by looking up the robots.txt file.