This is a discussion on "Robot Questions" within the Search Engine Optimization (SEO) section. This forum, and the thread "Robot Questions are both part of the Promote Your Website category.
|
|
|
|
|
 |
||
Robot Questions
|
||
| Notices |
 |
|
|
LinkBack | Thread Tools |
|
#1
Oct 23rd, 2007, 20:09
|
|||
|
|||
|
Robot Questions
I have been reading up on a robots.txt file. I am finding most answers on what it does and how to create one. There is one question that I cannot seem to find an answer to. Do I need one if I have a sitemap.xml file? Do I need one or the other or both? Is there a difference? I already have exclusions cited in my sitemap file. TIA.
       
|
|
|
|
#2
Oct 23rd, 2007, 20:13
|
||||
|
||||
|
Re: Robot Questions
A sitemap is a list of pages on your ste.
Robots.txt is used to stop robots indexing certain pages/directories.
Last Blog Entry: 3D Chess in your browser! (Mar 14th, 2008)
|
|
#3
Oct 23rd, 2007, 20:23
|
|||
|
|||
|
Re: Robot Questions
I exclude the URLs that I don't want to be indexed and then created the sitemap which is then uploaded to Google. Would Googlebot still crawl those URLs since I don't have the robots file? I should make the robots file for other bots then?
|
|
#4
Oct 23rd, 2007, 20:26
|
||||
|
||||
|
Re: Robot Questions
Robots will still find you if you dont have a robots.txt file...
__________________
Marc Staff Manager - Webforumz.com Want to be a moderator? PM me.
|
|
#5
Oct 23rd, 2007, 20:26
|
||||
|
||||
|
Re: Robot Questions
Yes, you should put all the files you don't want indexed in a robots.txt file.
If it's a secret file, password protect it!
Last Blog Entry: 3D Chess in your browser! (Mar 14th, 2008)
|
|
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| A few questions | Travis R | Web Page Design | 6 | Oct 6th, 2006 19:06 |
| Questions!!! | craig | Webforumz Suggestions and Feedback | 27 | Dec 8th, 2005 10:59 |
