FAQ: Suffering Backscatter

Dear Richi, I have about 20-30 returned emails from some entity/person who is somehow using my domain to send out bulk email. How is that even possible?

Sadly, it's trivial for a spammer to forge your address. It's not your Web host's fault.

Some badly configured email servers auto-reply to spam. That's what you're seeing.

If you want to complain to anyone, complain to the people running the servers who are auto-replying to you. Here's a template complaint I've used before...

Hello. You are sending spam to me by bouncing spam to an unrelated person. I did not send the spam to your server: spammers forge the message sender. Hence, your reply goes to an innocent third party.

Perhaps you sent an unsolicited bounce because your mail server is incorrectly configured. Please don't do that. You should *reject* during the SMTP conversation, not *bounce* after accepting the spam message. It is not necessary for your MTA to send a non-delivery DSN -- you should reject at the point of SMTP RCPT with a 553 error or equivalent.

Or perhaps you're auto-replying to spam. Presumably you filter spam before delivering inbound email. In which case, this reply shows that spam is getting through those filters.

It's bad practice to accept a message for a non-existent user. If you accept and then bounce, you're sending spam. For more information, please see http://www.spamcop.net/fom-serve/cache/329.html

If this was an isolated error, there's no need to be concerned that you will be blacklisted as a spam source. It usually takes several complaints to illustrate a pattern of email abuse.

However, I urge you to correctly configure your mail servers.

More info at an old post of mine: I Got 25,000 Spam Messages in Two Days!

BoxSentry Ditches Challenge/Response; Fights False Positives

Update Apr 25 6.30am UTC: fix name of product (thanks, Meng)

Singapore-based BoxSentry has historically been known as a challenge/response spam filter vendor. Readers will probably be aware that I'm no fan of C/R.

As time goes by, BoxSentry has gradually de-emphasized C/R, but until recently it was still sending challenges for a small but significant proportion of the spam it received -- and hence was sending unsolicited "replies" to people who had never sent email to the BoxSentry user.

Manish Goel, BoxSentry's CEO, confirmed to me that his company no longer uses C/R. That's great news for Internet users. Well done, Manish; I know that I and others have been thorns in your side for a while about this; I appreciate your good humour in our occasional, heated debates!


Manish also brought other news. While beefing up their technology base -- in part to compensate for the loss of the C/R layer -- the company has developed new techniques to better identify false positives.

BoxSentry has wrapped the new techniques in a product it's calling LogiQ. The idea is that it can run alongside a traditional spam filter and automatically retrieve any false positives it finds.

As an illustration, Manish offered a "typical" example: over the test period, a deployed spam filter from one of the well-known vendors delivered 11,500 legitimate messages, but LogicQ found an additional 680 false positives in the filter's quarantine. That's a roughly average false positive rate, in my experience. Not the exactly state-of-the-art, but pretty representative of deployed spam filters. It might equate to one false positive every week per user.

Manish says that 100% of the false positives identified with these new techniques really are false positives -- although they may not catch all of them.

A bold claim; I'm looking forward to digging into the details of the techniques under NDA...

C/R and "Spam Index" Conversation Roundup

I wanted to pull together some of the conversations that have been flying around recently about challenge/response spam filtering and this "spam index" idea. As is often the case, quite a bit of the value is in the conversation, in addition to the original posts, hence this roundup...

Anonymous:

As the holder of a domain name frequently forged into the From: or Reply-To: fields of spam, I can testify for certain that it doesn't work. In fact, whenever I receive a challenge to one of those forged addresses, I make sure to reply to it to make sure the spam gets through. Petty, perhaps, but I'm not being paid to filter C/R users' spam, so I'll pass it through.

Dean Harding:

I'll admit I was a bit suspicious that if challenge/response was such a panacea why were there not more people using it? My point was not that people should start using challenge/response, though, it was more to just point out that many people are still not happy with their spam filtering.

Len Dressler:

[Richi,] you're really kind of a dork ... It appears you have some sort of agenda of your own, fairly skewed towards blacklist and the like, which from an IT managers perspective, is a joke.

Richi:

Len, you're entitled to your opinion, and I will defend your right to express it to the best of my ability. Fact is, state of the art spam filters catch 95-99% of spam, with a vanishingly-small false positive rate. Such spam filters use a combination of techniques ... I see no evidence that a single approach—such as IP blacklisting—is viable.

Anonymous:

I was interested in learning of Peter's methodology ... I attempted to register on his web site in order to download a copy of his report. I'm still waiting for a response, who knows maybe his acceptance e-mail was justifiably intercepted by my spam filter.

Sandman:

If its my inbox, it is a communication tool for me, and I own the right to ask people to verify they are who they say they are.

Don Marti:

I see lots of “I just started using C-R, it’s great” posts, but no “I’ve been using C-R for years and it’s great” posts. C-R is something that you try and give up on. Or, in my case, watch other people try and give up on.

Anonymous:

Effective spam control is possible. It doesn't require cumbersome and work-flow disruptive band-aid solutions like C/R ... What's needed and has been proven to be most effective is a human feedback component. Several of the best anti-spam products available today include this as part of their toolset.

This is not to say that you need a solution where YOU have to be the human in the loop. The best vendors in the space do that for you and push new rules out to their customers every 10 mins or so.

Devil's Advocate:

Asking various people "how happy" they are with their present anti-spam product has absolutely no bearing on the effectiveness of those products ... if you ask if a C/R user sees less spam, you're going to get a "yes". But, what if you ask all the innocent 3rd parties that receive the challenges (which the C/R user doesn't see)? ... All C/R succeeds in doing is displacing the original spam volume in favour of its own variety of spam ... [and] shows a blatant disrepect for the health of the Internet.

Anonymous:

Nonsense - I am no expert, just a user, but every fact you make is wrong.

Richi:

In my spamtrap archive, I have several samples of inappropriate challenges from every C/R system known to me. Just in the past month, I've got challenge-spam from: [long list deleted]
...
Still don't believe that C/R systems send spam to innocent 3rd parties?

Peter Brockmann:

Your last post proves precisely the point. Users don't care and shouldn't have to care about what falls into YOUR inbox, only what falls into THEIRS.

Richi:

So users don't care that they're sending spam, as long as they don't get any?
...
Increasingly, the main issue with C/R isn't that it annoys innocent 3rd parties -- it's that the backscatter hits spamtraps, causing legitimate challenges to go undelivered. Hence, the false positive rate of C/R is actually surprisingly high.

Ask a C/R user about this though, and they'll often be blissfully unaware. It's hard to know when one is missing a legitimate unsolicited message from someone you don't know.

David Merrill:

For recipients, challenge-response and sender verification methods are good, but their use can get your domain blacklisted. Why? Because each incoming message, spam or not, generates an outgoing message, and spammers can (and do) use those in denial-of-service attacks.

Justin Mason:

Focussing the debate on the “user’s inbox” ignores the overall picture, including everyone else’s mailbox, which is where C/R fails.

But my favourite comment has to be from Al Iverson, on the membership-only list, SPAM-L (Al kindly gave me his permission to be quoted here):

C/R is trapped in this eternal September of newbie solution developers who think they're the bee's knees because they figured out how to implement a "new" version of C/R (which is usually exactly the same as every other one). Then they act like a kicked puppy when we don't jump for joy over how awesome it is to see...yet another implementation of C/R.

Eternal September of newbie solution developers? Priceless!

Who is Peter Brockmann?

So, according to one Peter Brockmann, challenge/response (C/R) spam filtering is a wonderful thing, and beats all other anti-spam techniques into a cocked hat.

Huh? What? How did he come to that conclusion?

I've beaten the "C/R filters are a terrible idea" meme to death, as have many others, so I'm not going to repeat all that here. If you're new to the arguments, take a stroll through these posts (perhaps you should work from the bottom up).

But I was about to write about Peter's methodology. However, it would have been an identical post to the one Justin Mason wrote -- he beat me to the punch. So here are Justin's money quotes:

The “Spam Index” is a proprietary measurement of spam filtering, created by Brockmann and Company. A lower “Spam Index” score is better, apparently, so C/R wins!
...
However — there’s a fundamental flaw with that “Spam Index” measurement, though; it’s designed to make C/R look good ... The “Spam Index” therefore considers a false negative as about as important as a false positive. However, in real terms, if a user’s legit mail is lost by a spam filter, that’s a much bigger failure than letting some more spam through. When measuring filters, you have to consider false positives as much more serious!
...
[And] the situations where C/R fails are ignored. Is it any wonder C/R wins when the criteria are skewed to make that happen?

I too took a close look at his methodology. It is really, really, horribly biased in favour of C/R. Unbelievably so. By orders of magnitude, arguably.

The idea is that one can come up with a neat "score" for the performance of a spam filter -- of course, the exact composition and weighting of such a score can sway the results in any direction one chooses.

Statistics aside, asking C/R users if they're happy isn't the be-all and end-all of anti-spam research. C/R users may indeed be happy -- happily unaware that their spam filter is sending spam by replying to innocent third parties who's addresses have been forged by spammers.

(As an aside, I note with amusement that Peter mis-categorizes Commtouch and IronPort as DNSBLs -- which he calls "RBLs", so perhaps Trend Micro should whine at him about trademark infringement.)

So what's going on here? I first came across Peter earlier this month, when I noticed some rather odd edits to the Wikipedia page about Challenge-response spam filtering made by one Pjbrockmann. The edits did rather deviate from Wikipedia's prized "neutral point of view" (NPOV). I also noticed a sneaky link back to his site from the page: naughty-naughty (as a great philosopher once said).

So, let's check out brockmann.com. The About page says, "Brockmann is a Wikipedia contributor." Well, golly, so he is. (Perhaps I should add that to my puff piece too.) His Wikipedia contributions extend to being dinged twice in April and June for spam and non-NPOV (the more recent issue noted above would make it three). Not so great.

Justin alleges that Peter has a relationship with Sendio. I don't know about that, but I do see he also mentions SpamArrest as an example of C/R. But does this (presumed) relationship stop him being objective? As Steve Hunt says, it, "Depends on what you mean by objective":

We are all mere mortals, and my own personal preferences will be very clear in the posts. Actually, my personal preferences and biases pay the bills ... Does that make me less than objective? I don't think so, but use your own judgment ... I commonly won’t expose which vendors I’ve helped because – frankly – it’s none of your business. It doesn’t change my ability to speak frankly and truthfully, and you might look at the list of companies and assume some bias that really doesn’t exist.

I like how Steve puts this, but I differ from Steve and Peter in that my personal preference is to maintain a list of clients in public (it's not a complete list, mainly for reasons of confidentiality -- e.g., when I've worked on expert witness contracts). So I guess you might look at that and, "Assume some bias that really doesn’t exist."

But, as an independent adviser/analyst/consultant, I also hope that you'll find that what I have to say is actually true.

Sender Authentication Doesn't Fix Challenge/Response

Happy new year. Sorry that the first post of January is about challenge/response (again), but surprisingly few people seem to get it.

There's this idea floating around that challenge/response filters are OK if they check SPF, SenderID, or DomainKeys -- only challenging messages that pass those checks.

Twaddle. This idea that SPF or SIDF or DKIM can tell you whether a message is forged is naive.

Firstly, implementation on the sender side is spotty. If there's no SPF record or DKIM header to check, you're back to square one.

Secondly, don't forget that most spam is sent by virus-infected computers (corralled into a botnet). There's nothing to stop virus writers from sending spam that passes an SPF/PRA/DK check at the receiving end.

Another Challenge/Response Datapoint

Sorry to harp on about challenge/response, but on the topic of C/R causing many false positives, I just noticed this post on The Admin Zone:

I HATE challenge-response spam blocking with a passion. All the time, I get Earthlink members signing up on my message board, but not putting the domain name in their whitelist. When vBulletin sends out a validation email, the following bounces back into my mailbox ... As a matter of principle, the mods and I NEVER respond to email challenges; we NEVER "click the link below" to be added to a whitelist.

If an existing user starts using challenge-response spamblocking, forget to put my domain in their whitelist, subscribe to threads, and as a result fill my mailbox with challenges, they're suspended for a week. Behind spam, it is my number two pet peeve.

Boxbe: Another C/R Spamhaus

Some buzz today about Boxbe -- a service that promises to forward unsolicited email only from those willing to pay a fee for your attention. I signed up to take a look, and was frankly horrified by what I found.

Boxbe is a front for another of these awful challenge/response setups. Look at the reply I got to a test message:

Subject: Held: testing

The message you sent to richi@boxbe.com regarding "testing" is being held undelivered because he or she has not pre-approved your email address [redacted] for access.

To deliver your message, you can:

* Take a short test (a simple test by following the link below
[link redacted]

* Pay a small fee (USD $0.15) which
Boxbe will share with the richi@boxbe.com. This is intended
for advertisers. To pay, click on the link below:
[link redacted]

Sigh. In case you've not heard the mantra already:

1. Challenge/response causes spam (because spammers forge the sender)
   
2. So if you use C/R, you're a spammer
3. Filtering your spam is not my job
   
4. If everyone used it, email wouldn't work!

Prediction: if Boxbe gets popular, spammers will start sending to it, which will cause backscatter complaints, which will cause blacklisting of Boxbe's servers.

Here's why backscatter is bad, and here's more about the stupid idea that is challenge/response. But don't just take my word for it.

Other Boxbe coverage at Wired, GigaOM, Download Squad.

"Challenge/response filters have more Achilles' heels than they have feet"

I am such a media whore. That was your humble blogger, quoted in an InformationWeek article:

Spam Filtering Floods Innocent Inboxes
Do challenge/response spam filtering systems create more problems than they solve? One analyst argues against them.
By Thomas Claburn

Two weeks ago, Ferris Research messaging analyst Richi Jennings awoke to find his e-mail inbox filling with spam at a rate of about a message per second. Over the course of two days, a spammer using a bot net -- a collection of PCs that have been subverted through security exploits to send spam -- sent an estimated 10 million messages that purported to come from several of Jennings's e-mail addresses.

That resulted in more than 25,000 bounce messages, from ISPs that return spam to the supposed sender (rather than deleting it) and from challenge/response filters that reply to spam with a note asking the listed sender to answer a challenge question before the initial message gets delivered.
...
Despite the fact the Symantec's Brightmail service did "an impressively good job" in blocking most of the bounced e-mails, Jennings nonetheless had to deal with hundreds of unwanted messages.
...
"Over the last year or two, I've spoken to countless challenge/response filter vendors and they all have their own excuse about why their solution is completely different, and really, yes, they agree this is a problem with badly written challenge/response spam filters, but their spam filter would never do anything so stupid and broken," says Jennings. "And of course I'm looking at an example from just about every one of those vendors that I got two weeks ago."
...
Tal Golan, CTO, president, and founder of Sendio, maker of a challenge/response e-mail appliance used by more than 150 enterprise consumers, disagrees strongly with Jennings's assertion that challenge-based filtering has problems. "Without question, the benefit to the whole community at large drastically outweighs that FUD [fear, uncertainty, and doubt] that's out there in the marketplace that somehow challenge/response makes the problem worse," he says. "The real issue is that filters don't work. From our perspective, challenge/response is the only solution. This whole concept of backscatter is just not true. Very, very rarely do spammers forge the e-mail addresses of legitimate companies anymore."

[Read the full article]

I Got 25,000 Spam Messages in Two Days!

Late last week, some idiot spammer decided it would be a neat trick to send a metric boatload of spam messages in my name (see also Joe Job). I estimate that in the space of 48 hours, his botnet spewed a million messages that appeared to come from one of my domains.

Unsurprisingly, a small percentage of those messages bounced. Guess where the bounces ended up? In my email. All 25,000 of them...

What can we learn from this?

1. Symantec's Brightmail spam filter is really good. OK, I kinda knew this already, but the Brightmail filters that sit in front of my mail service did a near-perfect job of sifting out the bounces from the real email.
   
   
2. Way too many email servers are badly broken, to the extent that they bounce email to unknown addresses, instead of rejecting it. Some of this is down to configurations that accept everything at the perimeter and only later decide the mailbox doesn't exist, but mostly it just seems to be broken software. (If you run a mail system that does this, for the love of all that's holy please fix it.)
   
   
3. Way too many ISP abuse desks seem to think (2) is perfectly acceptable behavior.
   
   
4. Way too many sites allow their users to auto-reply to email willy-nilly. Don't these people have spam filters? Amusingly, some do, as can be seen from the SpamAssassin-like headers added to the bounced spam, yet even though the message scores higher than the spam cutoff, they're still kindly letting me know that they're out of the office.
   
   
5. Way too many ISP abuse desks seem to think (4) is perfectly acceptable behavior, too.
   
   
6. Challenge/Response spam filters are a royal scourge. (See blog posts passim). It's not my job to filter your spam for you.
   
   
7. SpamCop is still an excellent resource.
   

Some spammer probably thinks he's been jolly clever and put one over an "anti". However, the state of the art in spam filtering is just too good.

PC World's Steve Bass Repents?

Last week, I wrote about how PC World's Steve Bass was promoting those evil, evil challenge/response spam bouncing products. I pointed out in my blog post and also in private email to Steve that these things can get their users blacklisted, because misdirected challenges are as bad as the spam itself.

Today, Steve has a new post up, calling me a "Polite ... self-proclaimed spam expert." Errr, well, those who know me may not agree with the first bit. And I'm not sure the second bit is quite my choice of words, but my clients seem to think so. Never mind. Onwards...

Fortunately, Steve has first-hand experience of the problem:

I get a half-dozen or so of these misguided challenge/response e-mails every day

Unfortunately, Steve links to a Wikipedia explanation of something with a similar name but which is nothing to do with spam. Presumably he meant to link to Challenge-response spam filtering. Oopsy.

In fact, reading his explanation of C/R, I'm not sure he actually understands the problem. See if you agree:

You can set some programs to bounce messages back to spammers and make them think your address is no longer working. Quite often a message from a challenge/response system will get treated as spam and bounced back with the rest of the junk e-mail. And quite often these messages float around the Net when someone using challenge/response also has a computer virus.
...
The spamming part comes into play when the person sending the e-mail receives a reply from the challenge/response program, challenging the sender to prove he or she isn't a spambot.

Well I'd have put it a bit differently. How about this:

Q:You can set some programs to reply to spammers; great idea, right?
A: No, because the replies hardly ever go to spammers -- spammers forge the message's sender. So they don't work.

Q: But it's only spam and we don't care about those messages, so it's OK... right?
A: No, because the forged senders are often real email addresses, with real people at the end of them. So you're causing unwanted email to be sent to them.

In other words, Challenge/Response makes you a spammer.

Update: Steve posted more on this topic. Steve's right on when he says:

Challenge/response ... doesn't work. I'll give you an example. A PC World reader sends me an e-mail and I take a couple of minutes to respond. Then I get an e-mail challenging me, asking me to take an extra step -- click here, go to a Web site, or maybe stand in the corner and whistle a show tune.

Nope, not me, Pal. I've already been a good Netizen and responded to the reader's e-mail; and I'm not about to spend more time on this. If the person sending me the e-mail had a spark or two, they'd have added me to their whitelist before sending me a message. So I watched how I responded to getting a challenge e-mail, figured everyone else would do the same thing, and decided not to bother with it.

And if you're looking for the debate between me and Jeff Hendrickson, it's right here.

PC World Offers Dangerous Spam Advice

Meet Steve Bass. Steve blogs at pcworld.com. Watch Steve blog. Blog, Steve, blog. Steve just blogged a bunch of spam filtering resources. Unfortunately, his list is heavy on the challenge/response FUSSP meme. Ooops!

For the record, Choicemail's "unknown-sender registration" and the "bounce" features of MailSnoop and MailWasher are really terrible ideas. (Don't forget that the "sender" of spam is almost always forged.)

I do wish consumer-focused journalists like Steve wouldn't promote these features -- he'll get his readers blacklisted, causing their email not to go through.

Update: Steve has responded. (If you're looking for the debate between me and Jeff Hendrickson, click here to read the latest discussion and follow the link at the end.)

For more background, see:

• Spamcop's backscatter page
• Why Challenge/Response Is Bad
• Here We Go Again
• Another Anti-Spam Tool To Avoid

Another anti-spam tool to avoid

Some company called hendrickson software components is touting a new spam filter called Em@ilCRX.

Guess what? It...

...uses an automated challenge response system, and reverse DNS validation to stop spam from making it into your email inbox.

Oh brother. All together now, say it with me:

1. Challenge/response causes spam
   
2. If you use it, you're a spammer
3. If everyone used it, email wouldn't work!

This topic previously covered here and here.

Tags: spam.

Here we go again

So... I was trying to make heads or tails of this press release this morning:

“The [spam] filtering approach was designed to handle junk mail for people receiving between zero and sufficient numbers to cause a nuisance. The real issue now is for people in the flood category, where filtering is not viable.”

Huh? What are they talking about? Let's read on...

Figures vary for the volume of unwanted mail. ... Using 80% means that four out of five mails for users [who get 30 or more spam messages per day] need to be marked, filtered, re-directed, quarantined and possibly archived.

Aside from the dodgy mathematics (there's no direct correlation between the number of legitimate and spam messages you receive), what is the point of all this?

High volumes are starting to strain the filtering approach because the filter has to take action on each mail it determines to be unwanted. This strains computing resources and also obliges recipients to take some action. Because the mail may have come from a source that has sent mail before, the receiver cannot ignore it.

Uh-oh, I have a bad feeling about this...

The alternative to filtering lies in the challenge-response method of dealing with spam, as used by the NMS’s Australian-developed TotalBlock solution.

Bingo! Yes, dear reader, it's our old "friend" challenge/response again. You may recall my previous post on this subject.

So, to summarize, if spam is a "nuisance" to you, why not turn it around and be a nuisance to legitimate senders who want to communicate with you, and be a nuisance to the poor people who are getting their email addresses used as forged spam senders? Yeah, and let's "oblige senders to take some action" instead. That's reasonable. Sheesh.

When will you people figure it out? In nice, simple language:

1. Challenge/response causes spam
   
2. If you use it, you're a spammer
3. If everyone used it, email wouldn't work!

Tags: TotalBlock, spam.

Why Challenge/Response is bad

Challenge/response (C/R) is disliked by users and legitimate bulk mailers alike. Unfortunately, anti-spam technologists who should know better keep re-inventing it.

The most recent example that I've come across is SquareAnswer. Whenever I hear about a new anti-spam vendor with "secret," "revolutionary," "patent-pending" technology, that suffer "zero false positives," I roll my eyes and prepare for yet another C/R product.

What is it? Briefly, if a C/R recipient is sent email "from" a sender that it's never heard of before, it auto-replies with a challenge. Until the sender has satisfactorily responded to the challenge, their mail doesn't get through to the recipient's inbox.

Although possibly useful in some environments, it's basically a terrible idea. It's generally worse than today's state of the art spam filters, which use techniques such as Bayesian filtering, heuristics, and "out of band" connection data analysis. Here's why...

1. Users hate receiving challenges; especially if their email address has been forged by a spammer and they've never even heard of the person it came from, let alone emailed them. A significant number of people just don't respond to challenges, which means that the false positive problem is worse than with conventional filtering.
2. Legitimate mailers hate it because they can't deal with the flood of challenges when they send out newsletters. Again, the false positive (or "deliverability") problem is worse. Much worse, in this case.
3. C/R shifts the cost of spam from recipients to the senders of legitimate mail. How dare you make me prove that I am who I say I am? I've already published an unambiguous SPF record that says that my IP address is permitted to send email from my domain; what more do you want? We won't win the war against spam until the costs are shifted to the spammers.
4. Users who employ C/R are seen by some as spammers in their own right. It's part of the phenomenon known as "backscatter." Imagine if your email address was used by spammers to forge the "sender" of their pill-pushing messages. You would expect to receive many non-delivery reports from mailboxes that no longer exist, "we don't want your spam" bounces from badly-configures spam filters, and challenges from people running C/R systems. How is this better than the spam we're trying to kill?
5. If you run a C/R system, you are likely to be blacklisted for spamming, and your ISP will receive abuse complaints about you. You may even lose your connectivity as a penalty for violating your ISP's Terms Of Service or Acceptable Use Policy.

Vendors: enough with the C/R reinvention already!

Users and IT managers: don't buy it. There are much better ways to filter spam without the problems that C/R will cause you.

Categories: spam, SquareAnswer, challenge/response.