Spam Solutions for phpBB and MovableType

The Thinker by Rodin

I was pleased to discover two real spam solutions for phpBB and MovableType recently.

phpBB is open source forum software. As you might expect it is written in the PHP programming language, which is installed by default on virtually every UNIX or Linux based web server. I run a message board using phpBB as well as earn some spare change installing and writing modifications to this popular software. However, spam has been a real problem lately for phpBB message boards. Spammers have created software that automatically creates and registers phony users for phpBB message boards. Their software is clever enough to defeat the Visual Confirmation modification, which is now integrated into phpBB. (This modification shows a word embedded in an image that you have to enter into the registration form in order to register.) Once “registered” these spam robots sometimes post spam as topics on the message board. They always place in the “Interests” and “Home Page” fields or the Member List pointers to spam sites.

My workarounds to date have had limited success. That is until I found the Anti-bot Question Modification. This is a clever solution. It requires, as part of the registration process, that the user answer a question that only a human could answer. Since I have installed it, I have had zero spam registrations. (I used to get dozens a week.) One small problem is that the modification was written in German. The English translation is workable, however. Therefore, if you have spam and a phpBB forum then installing this modification should be a no-brainer. In the event that the spam robots learn how to defeat the standard questions, simply create your own. You can also change the name of the registration form variable that collects the answer to the question easily through the Administrator Control Panel, further adding complexity which will drive away spam robots.

MovableType is the software I use to run this blog. With MovableType, the problem has been comment spam. The solution I found is mt-keystrokes. It uses Javascript to infer that a human entered information into a comment field. When a user types information into the comment text field, it triggers a Javascript event. This in turn causes the value of a hidden field posted with the form to change. This plug-in then has to check for the correct value in this field. If it has not changed, it assumes the form was submitted by a robot and is consequently spam. Otherwise, it assumes a human entered the comment. So far, it has worked flawlessly. As a result, my Junk Comments folder has been gloriously empty. There is no reason to sift through it looking for that one comment that might be legitimate. However, I was unable to get it to work correctly unless I used the form variable they provided. Consequently, this solution may be a temporary balm.

Now if only I could permanently banish email spam from my life. I have found a combination of solutions, but nothing that guarantees me that I will not miss a legitimate message or two. I strongly suspect the whole email architecture of the Internet will have to change before that problem is solved.

Update from the Spam Wars

The Thinker by Rodin

You would think that if I were getting five comments per hour that would be a sign that I was running a successful blog. Alas, of the 178 comments I averaged in the last 36 hours, only one of them was a legitimate comment. The rest were spam masquerading as comments.

Fortunately, my Movable Type blogging software does a good job of filtering out these obnoxious spam comments. They are moved into a Junk Comments area. I can periodically delete them manually, or I can simply forget about them. I have Movable Type configured to automatically delete junk comments after one week. Occasionally I do scan them to see if there is a legitimate comment in there among all the obvious spam. If you sent in a comment that I did not post, please accept my apologies. It probably was inadvertently interpreted to be a junk comment. I simply do not have the patience to review a hundred or more comments a day to find the one comment in a thousand that is not junk.

At least blog comment spam is easier to deal with than the blog Trackback spam. It was a nice idea until the spammers discovered they could create phony trackbacks, which, instead of taking users to actual blog entries, took them to spam sites instead. I was averaging about 200 or more bogus trackbacks a day. After a couple years, with this abuse getting worse every day, I said enough. I turned off the trackback feature. Those who want to see my trackbacks can use a feature on Technorati.

The spam comments I get are the usual crap, but they tend to be topical. Texas Holdem Poker spammers keep sending me spam comments. Others are selling weight loss drugs like Phentermine. There is a whole subset of spam from those pretending to sell anti-impotence drugs. Others figure I or my readers must be perverts. I am not sure why my blog has been targeted for those who might be interested in black gay sex, but somehow I doubt those who are interested in it would be coming here. “Britney” leaves many comments, along with “Bill”, “Alena” and “Dave”. “Britney” just keeps writing me, often many times a day, sometimes within seconds of her last comment. “Not much is happening in my life right now”, Britney tells me pretty much every day. She is apparently she is obsessed with anal sex, so much so that she simply must include links to anal sex sites in her comments. I think she needs therapy, or a least a high colonic.

There are solutions to my comment spam problem. I could require anyone who leaves a comment to be first be authenticated. Movable Type offers just such a service called TypeKey. The problem is, of course, most commenters do not want to go through the hassle of the authentication process. Even if they do it, it is still a hassle to use it when leaving comments. Therefore, like most bloggers I turn off authentication and do my best to find the legitimate comments that come in.

I also run a phpBB forum (actually two of them). Here too, the spammers have been busy, since phpBB is the most popular open source forum software out there. Having to repeat numbers or letters embedded in an image in the registration no longer fazes them. They have written software that is able to decipher the symbols in the image. The spammer’s automated scripts then enter the numbers or letters into the registration form. phpBB allows a user to specify their web site and a signature block. Spammers will of course link to their spam sites in these fields.

I have tried a number of tactics to deal with these spam users on my forums. First, I turned on the feature that required administrator approval to add a new user. This turned out to be overly burdensome on me. I was constantly getting emails asking me to approve or disapprove new users. So I went back to requiring that new users reply to an email in order to complete registration. This would be fine except that phpBB by default shows unapproved users in the directory. And that suffices for spammers because it gives them the opportunity to have their email addresses and web sites made available to anyone who might be trolling through a forum’s user directory.

One tactic I have employed is this phpBB modification that won’t allow new users to enter website or signature information in their profile until they complete registration. Still, spammers are clever. They invent user names that describe what they are up to, like “freeringtoness” in the hopes that you will send them email or a private message. The Russians apparently have too much time on their hands. It appears that the majority of spam registrations come from them. If a user has an email address is from a .ru domain, watch out.

I have since developed two more coping tactics. First, I made a small modification to the user list function. It now shows only those users who have posted messages. Second, I wrote a little program that removes forum users who registered more than a week ago but never bothered to post a message or complete registration. I figure that if it bothers them, they can reregister later. I then set up a cron job to run this program automatically once a week. Spam problem solved, sort of. Occasionally a spammer will actually complete registration and post a message, and then I must manually delete their posts and delete their account. Right now, this is not hassle. Spammers are, if nothing else, persistent, so I figure eventually I will get dozens of these a day.

On the junk mail side, my strategy has not changed. My ISP (cox.net) offers server side junk mail removal, which I have enabled. Unfortunately, it does not catch all the spam. Still, it keeps my email box from being overwhelmed with spam. Therefore, I also use ChoiceMail whitelist software. (There is a free and somewhat crippled version of ChoiceMail that works for one POP account. The full featured version costs $39.95.) Those who make it through my ISP’s junk mail filter will still have to go through a challenge/response system, unless they know the magic words to put on the subject line. (Hint: if you want to send me an email, look at the right column of this page.) This works fine although I still scan my ChoiceMail Junk Box and Unknown Senders list periodically. Occasionally, there is something I want to read or know about.

If the past is any guide, I will have to maintain vigilance and continually refine my strategies for coping with spam. Perhaps I should try the strategy AOL was promoting: demand payment to have spam arrive in my inbox. I just hope my ISP does not start trying this strategy too.

Spammers must die

The Thinker by Rodin

We all hate spammers. There is truly nothing good that you can say about them. They allegedly constitute a form of human life, but if this is true then it is only on the sub species level.

Most people, no matter how evil, have some form of conscience underneath it all. At the very least when they do something wrong they feel guilty about it. Not so spammers. They are shameless. Give them an inch and they don’t just take a mile. They take a light year. They are human cockroaches. They will do anything and everything they can think of to connect you with unwanted advertising. There is no tactic off limit. In fact they have no limits whatsoever. The end justifies the means.

Fortunately my ISP now provides a server based spam filter. It seems to work reasonably well and captures perhaps 95% of the spam. But even so there is a lot of spam that still manages to get through. Since I use ChoiceMail any unsolicited email that gets through the server spam filter gets an automatic challenge email from ChoiceMail. It requires that the emailer to go to their website and fill out a special form for me to receive their email. New emailers have to enter a number or phrase embedded into a graphic on the web site, and provide a written justification on why I should read their unsolicited content. Those who don’t respond end up on my blacklist. Even if they respond I still have the option to reject them manually.

I find it educational to go through my spam occasionally and see what new tactics spammers are using. Lately I’ve been getting emails with excerpts from famous novels. Of course there is at least one embedded linked image that will take me to their site. I guess this is one way to get me to read Stephen King. The hope is that the content will seem legitimate and thus pass through most spam filters. But this is yet one more example, if it were needed, that spammers are soulless scum. Of course they have no qualms about using copyrighted works of others simply to send spam.

As email program spam filters get better with strategies like Bayesian algorithms of course spammers will keep trying cleverer solutions to let the spam through. No doubt you’ve seen some of these. One tactic: create an authentic looking, almost snooty looking email address. In my spam box is an email purportedly from AtlantaBallet.com. For some reason the Atlanta Ballet wants to sell me Bextra. Umm, no, I don’t think so. Spammers may be ingenious at getting the spam through, but they must have oatmeal for brains in the common sense department. If I were in the market for Bextra I certainly wouldn’t buy it from some shady dealer pretending to be the Atlanta Ballet.

Words are also getting subtly mistyped to pass through spam filters. Viagra becomes V1agra. Copy becomes C0py. Affordable becomes Aff0rdable. Do they really think I am going to buy anything from someone who cannot even spell? I don’t think so! And what’s with these ridiculous email addresses? Do they really think I will open up emails from gjfmdillwmywmkj@aol.com and Rxelx@manonthemoon.com?

And can someone please terminate these ridiculous Nigerian email scams? Goodness, they were old ten years ago! Every conceivable variation has been tried. There is no one left in the world with an email account that has not received a hundred copies of these. Maybe they snared some naïve people during the first six months, but today even imbeciles know to trash this stuff. And yet it keeps coming and coming.

What really incenses me though are those spammers who use my good name and email address to pass off their spam. Of course my friends are likely to assume the email is from me because it has my name and email address on it. So it sails right through their spam filter because I am in their address book. But now my friends have to treat my email address with suspicion. Perhaps they get 100 emails a day from me that are spam. Perhaps out of frustration they have added me to their blacklist.

If spam were limited just to email then perhaps it would be endurable. But email is yesterday’s spam frontier. Spammers’ tactics are getting increasingly ruthless and non-discriminating. For example, in this blog I routinely average 1-5 fake “comments” a day. Needless to say like all spam this spam is programmed. A computer has sniffed my site, determined that I have a Movable Type weblog, found the CGI program that processes comments (even though I renamed it) and sends a canned HTTP request masquerading as legitimate comments. Fortunately I review all comments before they are published, but I still need to remove them manually. And that means to some extent I still must read them.

But now even blog comment spam is insufficient. The latest twist is to create bogus blog trackback entries. Movable Type is not yet programmed hold trackbacks in a queue for approval. So anyone who looks at a trackback entry before I have a chance to remove it is directed to a spammer’s website.

(Yes, I know about Movable Type plug-ins like MT-Blacklist. It’s of some help, but no silver bullet.)

The response from our legislatures has been anemic. The Can-Spam Act has done nothing of the sort. The government gives lip service to tracking down and prosecuting spammers. In reality there is not much they can do. Spammers can and do move so quickly that law enforcement cannot keep up with them.

I cannot see any short-term solution to this problem. Signing all email with digital certificates could potentially help solve the problem. However a valid digital certificate is easy to acquire. With the right software you can create your own. And just because the email is legitimate doesn’t necessarily mean it is something I want to read. Eventually we will need some newer approach that does away with the drawbacks inherent in our twenty year old SMTP email protocol. Blogs have been suggested as one way to circumvent the problem. Instead of sending email people could leave public or private comments on your blog. But as I have discovered that is a simple magnet for spam too.

Sadly I see no solution on the horizon other than a brand new SMTP-less email architecture. Otherwise it may be that the convenience of email will no longer be worth its hassle. Using snail mail may be time consuming and costly but at least advertisers have to pay for the privilege of putting their fliers in my mailbox. Perhaps some sort of new system where those who send you unsolicited email must pay a fee when you read the email it is the way it will eventually have to be.

One thing is for sure: if the exponential growth of spam on the internet keeps increasing at its current rate eventually there will be no bandwidth left for more prosaic usages like surfing the web. Our whole Internet-based infrastructure could be rendered obsolete by soulless spammers. The good news is that spam would die. The bad news is that electronic commerce as we know it would be gone. So let’s hope a new email system that fixes these defects is embraced before it is too late.

(I’m betting this entry will get its share of comment spam.)

Spam: Absolutely Not!

The Thinker by Rodin

Every time I think I have exorcised spam from my life, spammers become yet a little cleverer. The latest twist: I am getting 2-3 “spam” comments a day to this weblog. Sometimes the comments include links to porn sites or just your run of the mill scams, sometimes there is an innocuous message with a link to a “homepage” which, of course, is a spam site. Today’s little outrage took me to a preteen sex site. Oh sure, I can’t wait to learn more about that. And my penis size suits me just fine, thank you very much.

I have a zero tolerance policy for spam. I simply won’t put up with it. I did for years because I had no choice. I looked at server-based solutions that would require a one-time authentication from someone unknown (not on my “white list” to use the terminology) who wanted to send me unsolicited email: such solutions typically require the user to type in an encrypted number or word embedded in an image in order to get the mail through. It’s a great idea except, of course, my ISP doesn’t offer it: I’ve got cox.net. Finally I stumbled on a PC based “white list” solution called ChoiceMail One that does the same thing. Essentially it creates a mailbox on my PC between my real mailbox, and only people on my white list get through. The rest have to go through the challenge and response system.

Yes, it was a pain for about six weeks. I had to go through my email and manually add lots of addresses, cutting and pasting from a text editor. (I use Eudora. It would not have been a problem had I used Outlook.) Then I constantly checked the spam trap to let those people in I forgot to add. There are lots of them you don’t think about: banks, very old friends, web sites you use a lot. But after six weeks I seem to get over the hump. I check the spam trap about once a week now, which is about how often ChoiceMail One shreds the stuff.

But spamming web logs … this hits a new low even for the spam industry, which has values lower than a ten-dollar whore. First of all I cannot figure out why they bother. Do they think this is DailyKOS? No, I don’t get a whole lot of comments, which his fine. I’d like my web log to be more popular but my self-esteem doesn’t depend on it. Most of you reading this will never bother to read my comments. I usually will since I have the software set up to send me an email when a comment is posted.

The Moveable Type software that runs this blog clearly wasn’t designed for this sort of attack. All I can do is ban IP addresses and that gets to be very time consuming.

But I won’t put up with my web log being spammed too. I figured there had to be a way around it and it seems like someone created a solution very recently. I went to Hotscripts.com and searched on “spam” and sure enough there was a free solution by a very nice fellow who put together a site called JunkEater.com just to protect web logs and guest books. I tried the solution and it works like a charm!

Is it perfect? No. But spammers are lazy. They have computers run canned scripts to post this spam on their behalf. They won’t actually be any humans sitting down and reading my web log and going through the steps manually. So it’s unlikely a computer will be able to read the image with the embedded number in it, and add it to the comment form for my weblog.

I now wait anxiously for the next form of attack from the spam community. I know they are planning their next moves. But I, or someone else, will find a technology that will foil the bastards.

Thank you very much, JunkEater.com for an elegant solution. All I had to do was register at their site, fill out a few forms and change the comment form on this site and I was done. I’ll be glad to give them some money occasionally to support this free site; we need to encourage people like this to give their best.

Enjoy what I hope will be my spam free web log.