Content Filtering Exposed

Content Filtering Exposed

by Adam C. Engst ace@tidbits.com

Geoff's article "Email Filtering: Killing the Killer App" struck some chords. We received a number of reprint requests (including one to post to the Investigative Journalists and Editors mailing list) and were contacted by other publications (including the New York Times) about the subject. David Strom ran with the topic in his Web Informant column, and I was a guest on David Lawrence's Online Tonight radio show on Wednesday to talk about it (there's a streaming version available if you'd like to listen to the conversation).

It's certainly gratifying to see that we can raise awareness of problems like this to such an extent, but the real story is that we're coming up on a critical tipping point for email. The mushrooming volume of spam has caused the value and utility of email to drop significantly for many people already, and the way overzealous server-side content filtering makes email unreliable stands only to worsen the very problem it's attempting to fix.

Spam seeks to fill your mailbox, and poorly targeted content filtering, in attempting to prevent the spam from passing through your mail server, can block many of the messages you want to receive. Both hurt the utility of email. Making the problem even worse is that many people (such as Mac.com users - see the discussion Dan Frakes started on MacInTouch on the topic) don't even realize that such content filtering is taking place, so they may never realize that legitimate messages are going missing. <http://www.macintouch.com/applemailfiltering.html>

Collateral Spammage 2002
There may be no way to determine what percentage of mail servers have some sort of content filtering in place, but I think this is a good occasion to reprise our poll from two years ago asking how much spam you receive each week. When I went back to check that poll's results, I was shocked to see that the answer at the highest end of the range was "more than 71 spams per week." I'm receiving almost that many per day now! That's 25 percent of my mail! So please, visit our home page and tell us how many spams you're receiving per week these days so we can all see how much worse the problem has gotten over the last two years.

Clarifications and Effects
Although many people instantly understood what the effect of an overzealous server-side content filter could be, it wasn't entirely clear to all. First off, I want to make it clear that our concerns with content filtering in no way mean that we're in favor of spam. As far as we're concerned, spam is the scourge of the Internet, and we've devoted far more time, energy, and money in fighting spam than almost any small business. Similarly, the fact that we're opposed to erroneous content filtering doesn't mean we're opposed to spam filtering in general, whether it's performed at the server or in users' email programs. There are many ways of blocking spam and rampant PC worms at the server that don't rely on arbitrary content filtering. We employ server-side filters ourselves, but we take pains to minimize the likelihood that our filtering will cause problems for legitimate senders, and whenever we find that it has done so, we work to help address the problem.

Second, since Geoff focussed on the effect that server-side content filtering was having on our attempts to deliver TidBITS issues to our subscribers, many people didn't put it together that this sort of content filtering applies to _all_ email messages, not just those coming from mailing lists or email publications like TidBITS. The size of our mailing list means we notice the problem sooner and suffer more than individuals will, but email messages sent from individual to individual cannot escape the effects of poorly written server-side content filters. The lost mail may not be a big deal, or it might be exceedingly important personal news or critical business communication. Neither the sender nor the recipient have any way of knowing. To paraphrase John Donne, never send to know for whom the content filters toll; they toll for thee.

Third and finally, a common theme among the messages we received was that losing some legitimate messages was worth the reduction in spam thanks to content filtering. Obviously, I can't argue with individual situations - it may be that your mail is sufficiently unimportant that you don't care if some never arrives. More generally, though, I feel that attitude is a tremendously slippery slope. Spammers are parasites who will kill their host, but treating the disease with content filtering is almost certain to have the same effect. Just as we don't automatically treat infections with amputation, neither should we automatically treat spam with server-side content filters.

Overall Practicalities
Last week, we suggested a few ways you could get TidBITS even if your mail server refused to accept an issue; this week, let me suggest a few ways we can work together to address the problem of bad content filtering.

Contact your ISP or network administrator and ask them point blank if they are performing content filtering on your email, being clear to distinguish from general spam filtering. If so, see if they understand the consequences of those actions. Most likely will, but may consider the loss of some legitimate mail an acceptable trade-off. If they persist in that belief, ask if there's any way the content filtering can be turned off, at least for your account if not every account. I doubt most will respond to a single person complaining, but if you can make the case (often a business case) to other users affected by the content filters, the groundswell might be sufficient to get content filtering removed. As an alternative approach, you might suggest they modify the system so messages caught by content filters are merely quarantined, rather than being deleted, so users at least have the opportunity to recover important messages that ran afoul of the content filters. (The downside of the quarantine approach is that it makes checking email more difficult for the user, thus potentially _increasing_ the cost of dealing with spam.)

If all else fails, I would encourage you to find another ISP or mail host that does not perform content filtering (or at least lets you control what happens to matched messages). Be sure to convey your reasons for switching ISPs to the customer service department at the old ISP so they understand how the lack of reasonable filtering policies negatively affects their business too. Obviously, if you're dealing with your company's network administrator, there's no way to switch, but it will probably be easier to make a business case to management about the effect of legitimate mail being deleted.

Once you've established that all the messages that should reach you are coming through, the next step is to manage them effectively on your machine. TidBITS Talk participants have contributed a number of suggestions for how they manage their spam, and for those of you who are Macworld subscribers, check the August issue (not yet on the Web) for my article on stopping spam. <http://db.tidbits.com/getbits.acgi?tlkthrd=1684>

The core problem is, of course, spam itself, and the Internet community will have to come together to address spam at a fundamental level. There have been numerous proposals, ranging from legislation (probably necessary at some level, but flawed in its geographic scope and enforcement provisions) to modifications to the Simple Mail Transfer Protocol (SMTP) that delivers every message to its intended recipient. Other efforts focus on plugging the economic loophole that spam exploits; ensuring that spam doesn't pay would certainly take a bite out of the spam load. Most likely, we'll need a combination of approaches, and the urgency of developing them increases daily.