This is a discussion on "How to Stop Robots" within the Web Page Design section. This forum, and the thread "How to Stop Robots are both part of the Design Your Website category.
|
|
|
|
|
 |
||
How to Stop Robots
|
||
| Notices |
 |
|
|
LinkBack | Thread Tools |
Jan 25th, 2006, 18:13
|
|||
|
|||
How to Stop Robots
Hi, just a quick question,
Is there a way to stop robots from scanning parts of a website? 
|
|
|
|
Jan 25th, 2006, 19:11
|
||||
|
||||
|
Re: How to Stop Robots
open up notepad and put the following into it:
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /private/ change the directory paths to the one(s) you want to block, then save the file as robots.txt Stick the file in the top-level of your site and FTP it up. Last edited by herkalees; Jan 26th, 2006 at 12:31.
|
|
Jan 26th, 2006, 08:44
|
|||
|
|||
|
Re: How to Stop Robots
I thought you should use Robots.txt, interesting that you can put it in .htaccess
|
|
Jan 26th, 2006, 12:31
|
||||
|
||||
|
Re: How to Stop Robots
Whoa, wait a minute. I was supposed to write "put all this in robots.txt". Why did I write ".htaccess"? I'm losing my mind. I'll edit my above post.
|
|
| Tags |
| stop, robots |
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Robots.txt | simonb | Search Engine Optimization (SEO) | 9 | Mar 28th, 2008 11:43 |
| robots.txt | alexgeek | Search Engine Optimization (SEO) | 2 | Oct 22nd, 2007 22:11 |
| robots | alexgeek | Search Engine Optimization (SEO) | 8 | Sep 28th, 2007 15:43 |
| Security vs robots | cbisson | Website Planning | 11 | Jun 28th, 2007 12:51 |
