Jul
15
2007

Using AdSense To Speed Up Search Engine Crawling & Indexing

Will putting AdSense on my site speed up the time in which Google finds and indexes my pages?

Share
Email

Since this is a new site and not in Google yet, it would be a good time to test this. AdSense ads on it as I do with most of my sites. When Google first started AdSense I had been in the middle of starting up some other sites which I put AdSense on within the first week of launching. One week from that point I found I had been indexed on Google. I will be testing this theory again but now with RobMalon.com. I will report my findings in a later post but for now, some research.

Media Bot & Google Bots

There are two bots that Google uses which we can identify by user agent from our server logs. MediaBot and a GoogleBot each with their respective purpose.

Don’t let this confuse you with the fact that there are tons of Google bots each with their own unique purpose that crawl the web all the time. One might look at your meta tags, another your H1′s, and others your inbound/outbound links etc.

An announcement by Google’s Matt Cutts at the Boston PubCon in 2006 verified MediaBot pages can also be used for Google’s main index:

“Pages with AdSense will not be indexed more frequently. It’s literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we’d retrieve the page from the crawl cache. But there’s no boost at all in rankings if you’re in AdSense or Google News. You don’t get any more pages crawled either.”

So no boost to rankings, which is to be expected. But it sounds like they use a universal cache. Which makes sense if you count the number of people using multiple Google services that all require the same data. You would NEVER want to go to that much work to log the same data three or four times and blow more processing power recording and managing it all.

AdSense Needs Page Cache

It stands the reason that if you put AdSense on your site you’ll get indexed sooner. AdSense Ads show targeted content based on your page’s attributes. Essentially the same attributes you would need to determine a rank in a search engine index.

So, the two bots are separate. The systems are different and probably don’t interact with each other as Matt stated. However, the cache storage is likely being pulled from the same data bin and created in the same place no matter which one gets to your site first. If it isn’t then the data possibly needs to be stored in a very different way for optimization purposes within Google AdSense . Any loss on having to send out another bot is marginal in comparison, thus a separate cache location along with different bots for the purpose.

Just because Google has your cache in one system, doesnt mean the data points exist in the other application to be included. Unless some sort of flag is being switched to notify the other system to take a look sometime soon.

At the end of the day, if you already have data, why not use it. Google is looking to be acurate and comprehensive, so I doubt they would skimp on data they already had laying around.

Rankings increase, no. But I imagine it does help, in the way of being a squeaky wheel. “Me me me me me”. Assuming the cache is shared, this is a positive effect. You are making yourself a bigger customer to Google if you’re using multiple products of theirs. Like most businesses, they are probably catering to that.

Finding The Cache Time

Once your page is indexed, no matter how it got there, you’ll be able to access their snapshot of it by clicking on the “cache” in the search results.

Google Cache Link

Google Cache Link

Then take a look at the header of the page, which will look something like this.

If you really want to be nosy you can go into your access.log on your server and look for that exact time. It would probably show you the IP and agent string.

Unfortunately I am updating this post from years before and no longer have access to the 2007 data. So if anyone ever runs this test again, it would be one more thing to check on. FYI, in 2007 the Cache data did not appear in the image I’ve provided above. You had to put a date/timestamp generator in your footer to track the exact time like this.

Update 11-18-10: I’ve had AdSense disabled on this site for a few months and have noticed it takes much longer to get an updated cache (or any cache at all) into Google searches. I’ll add them back in at some point and repost about this.

Respond: Leave A Comment | Trackback URL

Entrupeners, Subscribe for the lastest tools, tips, and tutorials.


Leave a Reply

Custom Theme by Rob Malon | Content & Design © 2010 - RobMalon.Com - Chicago, Illinois