I’ve written about the metrics companies quite a bit recently (see So ComScore Understands the Page View Problem? and Page Views Are Already Dead!. And comScore is getting a beating again. Today Techcrunch reports on new conflicting data between the two metrics companies Hitwise and comScore:
[Hitwise's] data is saying that this week, for the first time, Google Blogsearch surpassed Technorati in total visits.
(…) Comscore tells a much different story, and one that makes little sense given the facts.
As mentioned before, comScore skews towards “regular” consumers (rather than more advanced users/geeks). So Technorati has less geeky users? Hard to believe…
And Hitwise, what’s up with that 0.0023% vs. 0.0025% of total internet traffic? There must be a better way to represent these numbers! You know, one that looks less ridiculous.
None of the metrics companies have made a public statement about the controversy surrounding the page view (PV) metric yet. However Fred Wilson’s comment indicates for the first time that they understand the problem and are working on a solution. Fred’s blog A VC is well read and I had no clue that he’s on the board of comScore.
He also stresses the fact that PVs should not be confused with number of ads seen:
Ajax and other more modern web technologies allow for new ads to be displayed without a page reload. Ad views can grow even as page views decline.
And he brings up another point that I haven’t really considered yet. More and more pages are integrating widgets and third party content. How should those be measured? Who should get credit for the page view? In fact comScore is probably tracking this as a full PV of the target server today if the request is triggered client-side and the response has content-type text/html. See Comscore’s Google Numbers Are Too Low for some technical details.
He wraps up with a promise:
The bottom line is a page view isn’t a page view anymore. It’s a lot more and a lot less. And we are going to come up with new measurement terms in 2007 that recognize this fact.
Google’s Adam Cutts was also joining the discussion a few days ago with his post Page view metrics? Bah, humbug!. Thanks for defending Yahoo! btw:
I want to come to Yahoo’s defense about something. A recent spate of reports says that Yahoo has been surpassed by various companies in terms of page views. Why is that relatively bogus? Because of Yahoo’s switch to AJAX for its mail.
More: Page Views Are Already Dead!
Update 12/28/06: Two more links related to Ajax and PVs: AJAX Affects Page Views (on O’Reilly Radar) and MySpace Passes Yahoo In Page Views But Not Audience (on search engine land)
Update 2 12/28/06: Fred Wilson defended comScore back in late October 06 noting that he’s an investor and board member (I must have missed that). He also points to the bias due to the panel approach that I also mentioned in Comscore’s Google Numbers Are Too Low.
Update 3 12/28/06: If comScore understands that the PV metric is obsolete and has to be replaced with something more meaningful, why is it still making the headline and first paragraph in their press release? To their credit, they’re including a comment about Ajax (but only further down):
Further, Yahoo’s increased integration of AJAX technology may have had a dampening effect on page views, as the technology enables real-time site updates without the need to refresh a page.
The Ajaxian is asking: Death of Page View Metrics? based on Steve Rubel’s prediction that the metric has four years to live.
The page view metric (PV) is already dead! Not everyone got the memo though…
Page Views (PVs) have never been a great metric (better than hits though). It is easy to architect a site to generate more PVs. Mike Davidson explains how MySpace is
abusing this. And it’s also easy to game the system for example with a hidden iframe.
In fact, a higher number of PVs indicates low usability. If more PVs was better than there would be no Ajax! Remember how online maps used to work? High PV count. Compare this to how they work now with only one PV.
Steve Rubel mentions several Google products but I believe Yahoo! Mail* could actually spearhead the change. Take the largest email provider with its huge number of PVs and see the numbers drop as more and more users migrate to the beta. I believe this should be visible in the October/November numbers – see ComScore. We will see a declining number of PVs while the number of users is growing.
This is not the first time this topic comes up. evhead, Zawondy and others wrote about it before. I think the geeks understand the problem. However, it takes more time for advertisers, press and Wall Street to understand that the game has changed. I believe the transition will happen in two stages:
- Page view metric is declared dead after losing significance over time
- A new metric is introduced as unique users and time spent are not sufficient to measure engagement
It will be a slow transition to a better metric. I sure hope it’s not going to take four years though!
I’m surprised by the reaction (or lack thereof) of the metrics companies, mainly ComScore and NetRatings. Their entire business model is based on the page view metric (unique users and time spent are closely related). They’re only slowly starting to understand and acknowledge the problem.
The only usable metric today is unique users/visitors. But even that metric has to be taken with a grain of salt. (Don’t get me started on the accuracy of their current metrics…)
* Disclaimer: I work for Yahoo! Mail. The thoughts above are obviously mine.
A New York Times article compares the market position of some Google products to its competitors [via Techcrunch] and notes that they’re often trailing the competition by quite a bit. Check out this graph and judge for yourself. The numbers are based on Comscore Media Metrix.
While some bloggers are defending Google furiously, others are joining the bashing. As usual the truth is probably somewhere in between.
Two things are important that I haven’t really seen in any of the comments:
Trends matter more than current numbers. Several products are fairly new. What really matters is if they’re still growing and how fast. Being number 10 today is ok if you’re number 1 tomorrow. Comparing Comscore numbers over a period of time should give some indication here.
Comscore’s methodology is biased
Google has a clear disadvantage when it comes to how Comscore calculates its numbers (except for search). Comscore’s numbers are based on a panel. A piece of software is installed on the participants’ computer. It monitors every URL visited and sends a report back to Comscore. The total numbers are achieved by extrapolating the actual numbers.
I’m convinced that advanced Internet users are underrepresented in this panel. Or how do you like the idea of having a HTTP proxy that monitors all your traffic. Now this is exactly the crowd that uses Google’s services more than the average user. Therefore I expect these numbers to be biased in favor of the more main stream competitors. By how much is hard to tell though.
Related reading: Google Backlash – Round 2
Update 12/28/06: Fred Wilson, a board member of comScore, explains the bias:
Comscore is a “consumer panel”. It measures mainstream web users. It is not a “leading edge” panel and it will almost certainly undercount “geek” services like Delicious and Digg.