Getting Google to ignore pages not in sitemap?

nate2099 · Jan 31st, 2008, 02:56

Hi,

Is there any way to do what the title suggests?

I have recently majorly changed a site but left it on the same domain. Instead of having a page named profile.php?member=username I now have profile.htm?member=userid, of course I now have hundreds of 404's on google. Can I get rid of them without having to go through them 1 at a time with googles remove url tool?

Thanks,

Nathan.

saltedm8 · Jan 31st, 2008, 09:17

you can use robots.txt to disallow indexing of certain pages, here is an example of how it is done

# robots.txt
User-agent: *
Disallow: /myfolder/mypage.html
Disallow: /cgi-bin/
Sitemap: http://www.yoursite.com/sitemap.gz

this will generate one for you http://www.mcanerin.com/EN/search-engine/robots-txt.asp

if you dont know anything about robots.txt, search on google to find out exacelly what it is for

nate2099 · Jan 31st, 2008, 09:28

Hi,

Thanks for your reply. I know all about robots, I was just hoping there might be a way to actually remove the url's from google so they don't show up in search results. I think I might just have to create a 404 page and link to the homepage or something..
Thanks again.

saltedm8 · Jan 31st, 2008, 10:20

no, dont do a 404, you are just telling the search engines that the page no longer exists, try a htaccess re-direct ( 301 "moved permanently" ) - that way you will not loose visitors

follow this to create it http://www.isitebuild.com/301-redirect.htm

you could also sort the robots.txt to disallow indexing, that way in a few months you can remove the lot

artisites · Apr 8th, 2008, 15:30

when you create a google sitemap, does it produce these links as seen on google

nate2099 · Apr 8th, 2008, 21:47

No. That is done automatically for sites that google deems worthy.. ie lots of traffic!! (as far as I understand anyway)

artisites · Apr 8th, 2008, 21:57

so how would you code your site to allow for this - are they just top level anchor links?

i was sure i'd seen sites with low traffic have those results too...

saltedm8 · Apr 9th, 2008, 01:08

as said above, they are high traffic results that google has chosen to layout that way, I am sure that if a site has enough traffic google will do that to it anyway regardless of coding, - does not have to be a big website either as far as i know, but since part of seo is content based, i can only guess that 99% of them are large sites with lots of content - like webforumz