This is a discussion on "Robot Questions" within the Search Engine Optimization (SEO) section. This forum, and the thread "Robot Questions are both part of the Promote Your Website category.
|
|
|
|
|
 |
||
Robot Questions
|
||
| Notices |
 |
|
|
LinkBack | Thread Tools |
Oct 23rd, 2007, 20:09
|
|||
|
|||
|
Robot Questions
I have been reading up on a robots.txt file. I am finding most answers on what it does and how to create one. There is one question that I cannot seem to find an answer to. Do I need one if I have a sitemap.xml file? Do I need one or the other or both? Is there a difference? I already have exclusions cited in my sitemap file. TIA.
|
|
|
|
Oct 23rd, 2007, 20:13
|
||||
|
||||
|
Re: Robot Questions
A sitemap is a list of pages on your ste.
Robots.txt is used to stop robots indexing certain pages/directories.
Last Blog Entry: 3D Chess in your browser! (Mar 14th, 2008)
|
|
Oct 23rd, 2007, 20:23
|
|||
|
|||
|
Re: Robot Questions
I exclude the URLs that I don't want to be indexed and then created the sitemap which is then uploaded to Google. Would Googlebot still crawl those URLs since I don't have the robots file? I should make the robots file for other bots then?
|
|
Oct 23rd, 2007, 20:26
|
||||
|
||||
|
Re: Robot Questions
Robots will still find you if you dont have a robots.txt file...
|
|
Oct 23rd, 2007, 20:26
|
||||
|
||||
|
Re: Robot Questions
Yes, you should put all the files you don't want indexed in a robots.txt file.
If it's a secret file, password protect it!
Last Blog Entry: 3D Chess in your browser! (Mar 14th, 2008)
|
|
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| hi and questions | linn | Introduce Yourself | 9 | Jan 4th, 2007 23:55 |
| A few questions | Travis R | Web Page Design | 6 | Oct 6th, 2006 19:06 |
| Questions!!! | craig | Webforumz Suggestions and Feedback | 27 | Dec 8th, 2005 10:59 |
| What questions do you ask? | gSmith | Introduce Yourself | 3 | Oct 1st, 2005 08:12 |
