Mediapartners-Google/2.1

Smokie · Feb 26th, 2004, 09:25

Rob,

Whats the mediapartners thing all about?

Googlebot/2.1 (+http://www.googlebot.com/bot.html)
64.68.82.204

Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)
64.68.87.55

Rob · Feb 26th, 2004, 11:33

Hi Smokie....

You have basically just hit upon the two different google spiders:-
Googlebot/2.1 (+http://www.googlebot.com/bot.html) is the normal google spider that crawls
the web for content to display in Search Engine Result Pages (SERPS)

Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html) is the google
adsense / adwords crawler bot. This bot will only crawl pages that have google adsense
adverts on them. Generally the bot will come along once it detects an advert being
served from a page the adsense bot has not already crawled... it generally happens within
one to three mins of someone hitting a new page with adverts on... once it has the page crawled,
it can determine what adverts to display on that page. It will re-crawl from time to time to make sure the pages content has not changed.... if it has changed, it may be that it needs to adjust the type of adverts served on that page.

There is a big difference between these bots.... the first only gathers for the google search engine....
the later only gathers info to aid in presenting contextual advertising. Having the mediapartners-Google bot crawl a page does NOT help you get listed in search results... as far as I know the two bot's do not share any database, or communicate in any way.

Contrary to some peoples beliefs, having google adverts on a site does not speed up the crawling process, and has no effect on search engine results what-so-ever.

PS: As a test, put a new post in the testing forums... and then view that post. Then go to the active users page (link is in the 'active users' section on home page) ... within a very small space of time(1 > 3 mins), you will see the adsense crawler come by.

Hope this helps mate.

Smokie · Feb 26th, 2004, 11:37

Thanks Rob, but I dont have any google adsense ads on my site, why is this bot visiting me?

Rob · Feb 26th, 2004, 12:01

Ok... after some research involving a unique URL at smokie's site, we've pinned down this phenomenon.

This is caused by Opera.

Unregistered versions of opera contain advertising. One of the user preferences is for opera to serve up relevant adverts based on what you are browsing.... and yes, you've guessed it... these adverts come from google.

When you hit a page in opera that google adsense doesn't already know about, then along comes Mr Adsense crawler.

Problem solved.

Smokie · Feb 26th, 2004, 12:04

Very interesting, thanks Rob!

Rob · Feb 26th, 2004, 12:09

A thought just occured to me with what I said above:-
<blockquote id="quote"><font size="1" face="geneva, verdana, arial" id="quote">quote:<hr height="1" noshade id="quote">as far as I know the two bot's do not share any database, or communicate in any way.<hr height="1" noshade id="quote"></blockquote id="quote"></font id="quote">

It would make total sense for the adsense crawler to pass information to the google search engine crawler.
Google can then get an edge on competitors by making sure it has within it's indexes sites which users are surfing (in opera).

Whether I am right or not, I don't know... but I'll investigate.

Smokie.... can you add some real content to that unique URL? add a title, meta stuff a H1 and some normal text.

Include in the text some weird non-sensicle word so we can try and find it in the SERPS later. Also... do not link to that page from anywhere. This will make a good experiment.

Rob · Feb 26th, 2004, 12:48

We've kicked off this experiment to see if google passes url's between crawlers...
We'll keep everyone posted.

Webforumz Staff · Feb 26th, 2004, 22:40

Lol, gotta love it. This will be pretty interesting!

Smokie · Jul 19th, 2004, 16:11

Rob, just to finish this off - The page we created which is not linked to by any other page has never been visited by Google.

I guess you knew that was gonna happen anyway!

Cheers
Smokie