C/R and "Spam Index" Conversation Roundup

I wanted to pull together some of the conversations that have been flying around recently about challenge/response spam filtering and this "spam index" idea. As is often the case, quite a bit of the value is in the conversation, in addition to the original posts, hence this roundup...

Anonymous:

As the holder of a domain name frequently forged into the From: or Reply-To: fields of spam, I can testify for certain that it doesn't work. In fact, whenever I receive a challenge to one of those forged addresses, I make sure to reply to it to make sure the spam gets through. Petty, perhaps, but I'm not being paid to filter C/R users' spam, so I'll pass it through.

Dean Harding:

I'll admit I was a bit suspicious that if challenge/response was such a panacea why were there not more people using it? My point was not that people should start using challenge/response, though, it was more to just point out that many people are still not happy with their spam filtering.

Len Dressler:

[Richi,] you're really kind of a dork ... It appears you have some sort of agenda of your own, fairly skewed towards blacklist and the like, which from an IT managers perspective, is a joke.

Richi:

Len, you're entitled to your opinion, and I will defend your right to express it to the best of my ability. Fact is, state of the art spam filters catch 95-99% of spam, with a vanishingly-small false positive rate. Such spam filters use a combination of techniques ... I see no evidence that a single approach—such as IP blacklisting—is viable.

Anonymous:

I was interested in learning of Peter's methodology ... I attempted to register on his web site in order to download a copy of his report. I'm still waiting for a response, who knows maybe his acceptance e-mail was justifiably intercepted by my spam filter.

Sandman:

If its my inbox, it is a communication tool for me, and I own the right to ask people to verify they are who they say they are.

Don Marti:

I see lots of “I just started using C-R, it’s great” posts, but no “I’ve been using C-R for years and it’s great” posts. C-R is something that you try and give up on. Or, in my case, watch other people try and give up on.

Anonymous:

Effective spam control is possible. It doesn't require cumbersome and work-flow disruptive band-aid solutions like C/R ... What's needed and has been proven to be most effective is a human feedback component. Several of the best anti-spam products available today include this as part of their toolset.

This is not to say that you need a solution where YOU have to be the human in the loop. The best vendors in the space do that for you and push new rules out to their customers every 10 mins or so.

Devil's Advocate:

Asking various people "how happy" they are with their present anti-spam product has absolutely no bearing on the effectiveness of those products ... if you ask if a C/R user sees less spam, you're going to get a "yes". But, what if you ask all the innocent 3rd parties that receive the challenges (which the C/R user doesn't see)? ... All C/R succeeds in doing is displacing the original spam volume in favour of its own variety of spam ... [and] shows a blatant disrepect for the health of the Internet.

Anonymous:

Nonsense - I am no expert, just a user, but every fact you make is wrong.

Richi:

In my spamtrap archive, I have several samples of inappropriate challenges from every C/R system known to me. Just in the past month, I've got challenge-spam from: [long list deleted]
...
Still don't believe that C/R systems send spam to innocent 3rd parties?

Peter Brockmann:

Your last post proves precisely the point. Users don't care and shouldn't have to care about what falls into YOUR inbox, only what falls into THEIRS.

Richi:

So users don't care that they're sending spam, as long as they don't get any?
...
Increasingly, the main issue with C/R isn't that it annoys innocent 3rd parties -- it's that the backscatter hits spamtraps, causing legitimate challenges to go undelivered. Hence, the false positive rate of C/R is actually surprisingly high.

Ask a C/R user about this though, and they'll often be blissfully unaware. It's hard to know when one is missing a legitimate unsolicited message from someone you don't know.

David Merrill:

For recipients, challenge-response and sender verification methods are good, but their use can get your domain blacklisted. Why? Because each incoming message, spam or not, generates an outgoing message, and spammers can (and do) use those in denial-of-service attacks.

Justin Mason:

Focussing the debate on the “user’s inbox” ignores the overall picture, including everyone else’s mailbox, which is where C/R fails.

But my favourite comment has to be from Al Iverson, on the membership-only list, SPAM-L (Al kindly gave me his permission to be quoted here):

C/R is trapped in this eternal September of newbie solution developers who think they're the bee's knees because they figured out how to implement a "new" version of C/R (which is usually exactly the same as every other one). Then they act like a kicked puppy when we don't jump for joy over how awesome it is to see...yet another implementation of C/R.

Eternal September of newbie solution developers? Priceless!

Who is Peter Brockmann?

So, according to one Peter Brockmann, challenge/response (C/R) spam filtering is a wonderful thing, and beats all other anti-spam techniques into a cocked hat.

Huh? What? How did he come to that conclusion?

I've beaten the "C/R filters are a terrible idea" meme to death, as have many others, so I'm not going to repeat all that here. If you're new to the arguments, take a stroll through these posts (perhaps you should work from the bottom up).

But I was about to write about Peter's methodology. However, it would have been an identical post to the one Justin Mason wrote -- he beat me to the punch. So here are Justin's money quotes:

The “Spam Index” is a proprietary measurement of spam filtering, created by Brockmann and Company. A lower “Spam Index” score is better, apparently, so C/R wins!
...
However — there’s a fundamental flaw with that “Spam Index” measurement, though; it’s designed to make C/R look good ... The “Spam Index” therefore considers a false negative as about as important as a false positive. However, in real terms, if a user’s legit mail is lost by a spam filter, that’s a much bigger failure than letting some more spam through. When measuring filters, you have to consider false positives as much more serious!
...
[And] the situations where C/R fails are ignored. Is it any wonder C/R wins when the criteria are skewed to make that happen?

I too took a close look at his methodology. It is really, really, horribly biased in favour of C/R. Unbelievably so. By orders of magnitude, arguably.

The idea is that one can come up with a neat "score" for the performance of a spam filter -- of course, the exact composition and weighting of such a score can sway the results in any direction one chooses.

Statistics aside, asking C/R users if they're happy isn't the be-all and end-all of anti-spam research. C/R users may indeed be happy -- happily unaware that their spam filter is sending spam by replying to innocent third parties who's addresses have been forged by spammers.

(As an aside, I note with amusement that Peter mis-categorizes Commtouch and IronPort as DNSBLs -- which he calls "RBLs", so perhaps Trend Micro should whine at him about trademark infringement.)

So what's going on here? I first came across Peter earlier this month, when I noticed some rather odd edits to the Wikipedia page about Challenge-response spam filtering made by one Pjbrockmann. The edits did rather deviate from Wikipedia's prized "neutral point of view" (NPOV). I also noticed a sneaky link back to his site from the page: naughty-naughty (as a great philosopher once said).

So, let's check out brockmann.com. The About page says, "Brockmann is a Wikipedia contributor." Well, golly, so he is. (Perhaps I should add that to my puff piece too.) His Wikipedia contributions extend to being dinged twice in April and June for spam and non-NPOV (the more recent issue noted above would make it three). Not so great.

Justin alleges that Peter has a relationship with Sendio. I don't know about that, but I do see he also mentions SpamArrest as an example of C/R. But does this (presumed) relationship stop him being objective? As Steve Hunt says, it, "Depends on what you mean by objective":

We are all mere mortals, and my own personal preferences will be very clear in the posts. Actually, my personal preferences and biases pay the bills ... Does that make me less than objective? I don't think so, but use your own judgment ... I commonly won’t expose which vendors I’ve helped because – frankly – it’s none of your business. It doesn’t change my ability to speak frankly and truthfully, and you might look at the list of companies and assume some bias that really doesn’t exist.

I like how Steve puts this, but I differ from Steve and Peter in that my personal preference is to maintain a list of clients in public (it's not a complete list, mainly for reasons of confidentiality -- e.g., when I've worked on expert witness contracts). So I guess you might look at that and, "Assume some bias that really doesn’t exist."

But, as an independent adviser/analyst/consultant, I also hope that you'll find that what I have to say is actually true.