Okay this is a problem that probably means I am anal but it bugs me. I’ve been monitoring this site since March 2004 with SiteMeter. It’s a nice little service, particularly since I don’t have to pay for it. It’s fun to look at SiteMeter and see what people are looking at on my blog. It’s nice to know what interests them, what doesn’t, and how many pages they looked at while they are at my blog. For a few snippets of HTML in my MovableType templates I can get this information fairly easily. So can you. Look for the little SiteMeter icon at the bottom of my pages.
Lately though my traffic, as measured by SiteMeter has been down. Way down. These things happen. The gods of Google can give and they can take. About 80% of my traffic comes from keyword searches, usually using Google. Perhaps last year when I was heavy into writing political blog entries my site was more likely to get noticed. But my political entries are fewer and further between. Why? Mostly because others say it much better than I do. It is hard to come up with a unique perspective or new political thoughts these days. Also it’s easier to strike out if you make political predictions. I’ve made some great calls and I’ve made some lousy calls. But s’okay. No one gets them all. And I suspect I am doing better than most political prognosticators.
Still it’s reasonable to wonder what’s going on. Based on Sitemeter my blog has been nearly flushed down the toilet. A month or so ago according to SiteMeter I was averaging two hundred or more page views per day. Lately I’ve gone down to as low as thirty page views per day. Perhaps with summer coming on people have other distractions. Or it could be that SiteMeter is doing a lousy job of monitoring my web site.
I am virtually certain it is the latter. Because my domain is hosted I have access to the actual web server logs. My web host offers two packages that summarize my web log information for me, Webalizer and Awstats. Awstats is the better package so when I want to use it that’s the one I use.
Here is my latest SiteMeter monthly chart showing visits and page views over the last month:
And for the record here are my statistics from Webalizer since June 1st. Note that both Webalizer and Awstats seem to aggregate the statistics only once a day. That makes sense since it is probably pretty CPU intensive. It is one of the reasons that the SiteMeter solution sounded like a better solution. Its reports are all generated dynamically from a database.
Finally here is my Awstats report for June to date:
So what’s going on? Let’s take a few same dates and compare statistics. On June 1st according to SiteMeter I had about 165 page views. Webalizer reports 976 page views, Awstats shows 926. It could be that I was serving pages that were not metered. But every blog page, with the exception of certain comment pages, is SiteMetered. One thing I realized digging into it is that a log of people read my blog through newsreaders. You can’t add SiteMeter HTML code to an XML document, which is basically what a syndicated newsfeed file is. So SiteMeter never sees those requests. About 15% of my page views are to my index.rdf newsfeed page. My Atom newsfeed gets about 5% of my total page views. So for sure SiteMeter is not seeing 20% of the people visiting my web site.
It may also be that search engines spend a lot of time trolling my pages inflating my page view count. I have 1153 “hits” from search engines for the month to date out of a total of 5374 page views. I suspect Awstats and Webalizer are smart enough to subtract them from my page view counts. But if not about 21% of my page views are search engines trolling content, not actual humans reading my content.
You can read the statistics. On June 6th SiteMeter said I had around 165 page views. Webalizer said 398. Awstats said 354. On June 12th Sitemeter said about 60, Webalizer 343 and Awstats said 277.
Awstats and Webalizer are reading from the same source: my Apache web server log. It is curious that they come up with different numbers. You would expect them to be identical. I suspect that they each measure successful visits and page views differently. It is also possible that each calculates on different time zones. Webalizer though routinely shows more visits and page views than Awstats. Perhaps it includes unsuccessful page views it is totals.
But overall, either because not all my pages have SiteMeter code on them, or because SiteMeter is not capturing all the information (or some combination of both), SiteMeter underreported my actual page views by 82% on June 1st, 53% on June 6th and 78% on June 12th.
Here’s what I think is going on. Granted SiteMeter can only record pages with SiteMeter HTML on it. But this is probably no more than 25% of the pages being served. Given this it is likely that SiteMeter is not recording huge numbers of visits and page views that I am getting. Why? Well, embedded in my HTML for a blog entry is a URL that traverses the Internet back to sitemeter.com. Your browser essentially “pings” a SiteMeter server reporting the page being served on my site, and other information. It could be the Internet is busy and the “ping” doesn’t go through from the client computer. It could be the client computer has some software that prohibits these “pings” from getting out. (This is unlikely.) But it’s most likely that most of these “pings” actually reach the SiteMeter computer but it is not recording them. Why? Probably it is because SiteMeter can’t keep up with them all. It seems that everyone and their grandmother is SiteMetering their web pages these days.
Conclusion: SiteMeter’s numbers vastly underreport your actual visits and page views. Since most of us are getting it for free we shouldn’t be surprised. I know enough now at least not to be foolish enough to pay them money for advanced features. I will keep SiteMetering my site for the present. But now that I’ve delved into the matter I am confident that its numbers are pretty meaningless. I think this is a problem with any hosted (external) statistics service. Your web server log, if you can access it, is going to be accurate. Trust your server log and your web host’s statistics program and treat SiteMeter with a ton of salt.