The downside of search engines

The Thinker by Rodin

It’s no secret that my blog’s hits are way down compared to five or ten years ago. Trying to figure out why this is has been hard, and hasn’t been aided by my general apathy. There are days when my hits are in the single digits.

I recently saw a one-day spike of 97 page views, which turned out to be one person skimming lots of my pages. With 2038 posts and content going back to December 2002, you would think that I’d be getting a lot of traffic because I have a ton of content. But I’m not. Most of my stuff is being ignored because search engines don’t see it as relevant anymore.

At least that’s what I’ve learned in my latest research into the issue. Crazily, this site would probably get a lot more hits if it were a lot smaller. What would be left would be mostly my most accessed pages, i.e. the “relevant” ones. Merely keeping the old blog posts around, or posts that are very dated or rarely read, is apparently keeping many people from finding my site in the first place.

It wasn’t always that way, but over the years search engines (and Google in particular) have been concentrating ruthlessly on relevancy. If they don’t see your content as relevant, then for all practical purposes it will sit forever at the bottom of their search index. So if you are on the web because you enjoy encountering serendipitous content, well, a search engine can’t help you. In fact, it will get in your way. Even worse, there’s no easy way to discover serendipitous or random content. In their holy and obsessive quest to highlight only relevant content, I think search engines actually become less useful.

The exact process search engines use to determine relevancy is not known, but the general outlines are understood. If your content is newsy, i.e. topical, i.e. recent, it will rank higher. This is because in most cases people are searching for answers to a problem, so what’s current is more likely to be relevant than older stuff. The more sites that link to your content, the more relevant they think it is. It’s the 21st century equivalent of being popular by popular people saying they like you. Curiously, the algorithms search engines use are smart enough to figure out fake popularity. If you have a campaign to convince other sites to link to your site if you link to their sites, search engines will notice this stuff, and assume you are trying to game the system.

Yet Search Engine Optimization (SEO) is all about gaming the system, i.e. convincing search engines that your content is relevant. The SEO industry is huge, with lots of shysters out there offering to boost your site’s search engine ranking. While the broad outlines of getting better search ranking are pretty well known, no one can say for certain if a given strategy will work. Moreover, search engines are constantly tweaking their algorithms on the quest for even more relevant search results.

The result of all this is that site owners become like hamsters on a wheel, engaging in what is often a fruitless effort to boost their search ranking. Of course, organizations and companies that can afford to do so can achieve some success. It’s like lobbyists buying political influence in Washington: your site too can be more widely read, if you bring the right amount of bread or hire people with enough expertise in the SEO field. Or you can pay sites like Google directly to highlight your site using their Adwords program.

And there are plenty of people making careers out of this stuff. They pore over their daily Google Analytics reports. They attend SEO conferences. They watch YouTube videos on line on how to boost their search rankings. The tradeoff though is either time or money, and since how you spend your time effectively is money, it’s largely the same thing. You can probably boost your site’s traffic if you are willing to spend the time on it, or hire people to spend the time on it on your behalf.

So in trying to be helpful, search engines are creating an unequal playing field, providing increasingly tailored search results and giving a bias to those with time and money. It gets ridiculous sometimes. Google learns so much about you from your Google account and previous searches that it returns results that are perhaps too skewed toward your biases, making you miss important stuff like perhaps actual fact-based news. I wrote about this recently when I compared Google’s search to DuckDuckGo’s search. I found DuckDuckGo’s search results a whole lot better than Google’s because I was outside of Google’s filter bubble.

Even DuckDuckGo though is still on a quest for relevant search results. It has no serendipitous search capabilities either. Perhaps some search engine will find some niche market in returning serendipitous content. Then maybe my search rankings and site traffic will go back to the good old days.

One of the steps to returning to those good old days for me though might be to move the old and unsearched content to a new domain, say occams-razor-archive.info, or just purge it never to be seen again. I’d have to be careful not to actually link to a new archive site though, because that too could lower my site’s search rankings. And I’d have to disguise this blog into something it is not, and spend a lot of time tweaking it so that search engines will pay more attention to it. Probably the best way to increase my page hits would be to start a completely new blog, and fine tune it with punchy short paragraphs, post of no more than 500 words (so viewers don’t lose interest), always using an active voice and creating post titles that are carefully researched to get high interest.

I doubt I will do that. That takes a lot of energy I don’t have. Frankly, I sort of resent the search engine system we are forced to use. I disagree with the way it doesn’t rank the breadth of my content as important, when it should. There’s nothing I can really do about it though other than join this game, which doesn’t interest me. So this blog is likely to rank lower and seem less relevant in the years ahead.

But you can help me fight back. If you like what you read, then bookmark my site and come back to it regularly. Add it to your feed. Tell your friends, “Hey, I found this neat little site on the web!” Use the handy form on the sidebar (or off the menu if using a mobile device) to get emails when I make a new post. Show Google that you value longer and serendipitous content like my site. Maybe in time they will learn that in chasing the holy grail of relevancy, they are effectively hiding a lot of hidden gems across the web.

DuckDuckGo is a better search engine

The Thinker by Rodin

Google pretty much owns the search engine market, but why? Perhaps it was because they were the first to do it well. In the old days of search engines when we were forced to use sites like Lycos and AltaVista, finding useful stuff on the web was excruciating, requiring you to go through many pages of results (and usually wait … these were back in the connect-by-modem days). Google figured out how to turn relevance into an algorithm. Basically, the more sites that link to a page, the more relevant it is.

There’s more to a search engine than that, of course, but that was the big innovation. And not surprisingly, once that was figured out other search engines figured out how to do this too. For most search engine queries, you will get a set of similar results.

But with Google search, you get more, but in this case more maybe less. Specifically, what you get is what Facebook figured out, probably after seeing what Google was doing. Google will watch your behavior closely and give you more of what it thinks you like. To do that of course, it had to learn a whole lot more about you. And most of us are happy to comply, since most of us are logged into Google accounts. Even if we are not, given that we leave cookies in our browsers, not to mention IP addresses, so Google can usually figure out it’s you. With every search engine query and use of its services it learns more about you. Google probably knows you better than your spouse does.

The downside is that, like Facebook, you end up in a filter bubble. Google knows if you swing left and so your queries are unlikely to show links that swing right, at least not on the first page. In short, Google and other search engines undercut its own quest to provide relevant search results by providing you links to stuff you are more likely to click on. They are relevant as long as you want results that reflect your biases.

It’s a profitable strategy for Google. It just watches you behind the scenes. Its powerful algorithms give you more and more reasons to invest time with Google and its many services. I plead guilty because like almost everyone I have a Google account. I use GMail extensively. I long ago stopped using an email client. I do all my email using GMail’s web interface. I do turn off the marketing (thanks, Google!) and, gosh, with all that space and it’s amazing search interface I can find pretty much anything in my email in a few seconds spanning more than a decade of use. Since I have lots of clients, it’s quite a value to do things like easily figure out what their issue was six month ago. I’m willing to pay to use GMail, if I have to.

But I don’t have to use Google search. Since discovering DuckDuckGo, I rarely use it for search anymore. That’s because DuckDuckGo is kind of retro in a way: it provides more relevant results by getting you out of search personalization. I’m getting more relevant results with it. Moreover, DuckDuckGo doesn’t track your usage. It doesn’t know who you are. It doesn’t follow you around with ads. It just gives you highly relevant search results with incredible speed. I can hardly press the enter key before I get a page of results.

My result is now a lot of highly relevant content I wasn’t seeing before, mainly because it wasn’t on the first page of results. Google was figuring I didn’t want to see this other stuff, but I do. Actually, it was feeding me links that matched my biases, hoping I would stick around. With DuckDuckGo though, searching is becoming more useful again. Moreover, I am learning stuff by reading sites I probably wouldn’t have otherwise. I’m becoming more informed, getting actual news and perspective. Real news after all tells you objective truth, not selective truth. I’m moving out of my own filter bubble and into a wider world. I am gleaning actual insight. It’s neat and kind of humbling. But it shouldn’t be. Rather, the question should be why did Google and other search engines put us in this bubble in the first place? Of course: they were chasing mammon, putting their best interests ahead of yours

To use DuckDuckGo regularly, you have to change your default search engine. It’s not hard to do; it’s just something we don’t think about because for most of us we can’t imagine there is something out there better than Google. You’ve been sold a bill of goods that isn’t quite true.

Try it for a week and tell me I’m wrong. I’ve been using it for a few months now. I can’t see ever going back.

For more reasons of the virtues of this search engines and some of Google’s slightly evil side, watch this video: