Web Design and Development Forums

Robot Questions

This is a discussion on "Robot Questions" within the Search Engines section. This forum, and the thread "Robot Questions are both part of the Search Engines and SEO category.

Old Oct 23rd, 2007, 20:09   #1 (permalink)
Junior Member
 
Join Date: May 2007
Location: Michigan
Age: 27
Posts: 23
Robot Questions
I have been reading up on a robots.txt file. I am finding most answers on what it does and how to create one. There is one question that I cannot seem to find an answer to. Do I need one if I have a sitemap.xml file? Do I need one or the other or both? Is there a difference? I already have exclusions cited in my sitemap file. TIA.
__________________
Character - It doesn't matter what you are...just as long as you be a good one.
prizm is offline  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Old Oct 23rd, 2007, 20:13   #2 (permalink)
Administrator
 
alexgeek's Avatar
 
Join Date: Jul 2007
Location: Webforumz 24/7
Age: 15
Posts: 4,102
Blog Entries: 9
Send a message via MSN to alexgeek
Re: Robot Questions
A sitemap is a list of pages on your ste.
Robots.txt is used to stop robots indexing certain pages/directories.
__________________
Languages: PHP, mySQL (queries), C#, (X)html, CSS, JS.
alexgeek is offline  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Old Oct 23rd, 2007, 20:23   #3 (permalink)
Junior Member
 
Join Date: May 2007
Location: Michigan
Age: 27
Posts: 23
Re: Robot Questions
I exclude the URLs that I don't want to be indexed and then created the sitemap which is then uploaded to Google. Would Googlebot still crawl those URLs since I don't have the robots file? I should make the robots file for other bots then?
__________________
Character - It doesn't matter what you are...just as long as you be a good one.
prizm is offline  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Old Oct 23rd, 2007, 20:26   #4 (permalink)
 
Join Date: Apr 2007
Location: Scotland, UK
Age: 15
Posts: 1,750
Send a message via MSN to Marc Send a message via Skype™ to Marc
Re: Robot Questions
Robots will still find you if you dont have a robots.txt file...
__________________
I'm back!!!!
Marc is offline  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Old Oct 23rd, 2007, 20:26   #5 (permalink)
Administrator
 
alexgeek's Avatar
 
Join Date: Jul 2007
Location: Webforumz 24/7
Age: 15
Posts: 4,102
Blog Entries: 9
Send a message via MSN to alexgeek
Re: Robot Questions
Yes, you should put all the files you don't want indexed in a robots.txt file.
If it's a secret file, password protect it!
__________________
Languages: PHP, mySQL (queries), C#, (X)html, CSS, JS.
alexgeek is offline  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Old Oct 23rd, 2007, 20:29   #6 (permalink)
Junior Member
 
Join Date: May 2007
Location: Michigan
Age: 27
Posts: 23
Re: Robot Questions
Excellent - Thank you!
__________________
Character - It doesn't matter what you are...just as long as you be a good one.
prizm is offline  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Spurl this Post!Reddit! Wong this Post!
Reply With Quote
Reply

« Local Business Listing Merging | Search engine help »
Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
three questions alexgeek Graphics Forum 2 Sep 13th, 2007 06:39
1 questions Voodoochilli Webforumz Cafe 11 Jul 26th, 2007 14:17
What questions do you ask? gSmith Introduce Yourself 3 Oct 1st, 2005 08:12
2 questions... benbacardi ASP Forum 5 May 6th, 2005 21:25



Latest Updates

All Points SEO Security Advisory - CHECK YOUR SITE NOW!

Creative Coding :: February 2008
Webforumz is sponsored by: WESH UK Web Hosting
All times are GMT. The time now is 14:59.

Sleep Study Scoring :: Free Bet :: Website Templates :: Online Betting :: Bookmakers :: Funny Quotes :: Internet Recruitment Software :: Microsoft CRM Experts :: Online Casino :: Decorated Christmas Trees :: Midwife Forums :: Football Betting :: Ecommerce Software :: Web Hosting :: Football Stats :: Dry Cleaning Collection :: xtreme wales - extreme clothing :: Apuestas :: Sharepoint Consultants :: Website Optimisation :: Office Clearance London :: Sharepoint Experts :: Sports Betting :: Casino :: Website Templates :: Web Design Development India :: Online Gambling

Powered by: vBulletin Version 3.7, Copyright ©2000 - 2008, Jelsoft Enterprises Limited.
© 2003-2008 Webforumz.com : All Rights Reserved
Search Engine Friendly URLs by vBSEO 3.2.0 RC6


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59