CampaignFoundations.com

I run a GNU Mailman list for the International Committee of the Green Party of the US. This morning someone forwarded an essay from blackagendareport.com, questioning the motives and veracity of the "Save Darfur" movement. Several African nations are mentioned, and large numbers of dollars spelled out. It triggered three of Spamassassin's ADVANCE_FEE rules, totaling 7.9. My threshhold for adding the ***SPAM*** indication to the subject line is 8.3. The additional 0.4 points came from the HTML-only and "no real name" tests.

The writer makes a good argument that "Save Darfur" is not what it seems. Spammers are making it hard for his message to get out.

More subscribers to cagreens.org, greens.org, and gp-us.org mailing lists use Yahoo Mail than use any other email service provider.

Our subscribers post to over a hundred of these mailing lists through Yahoo Mail's web site. Those postings usually contain a few hundred characters of message that the subscriber actually wrote, and tens of thousands of characters of Yahoo advertising and formatting. The actual message is often less than 5% of the total bulk. And sometimes these postings come from those online newspaper articles that have an "email this page" button. Those can be 99% cruft.

That is, when you send Yahoo Mail, you're sending tons of cruft. Hotmail and MSN are even worse.

Many other subscribers to these lists use smaller, independent Internet services. Now that email is over 95% spam, these companies can't afford to develop and maintain their own spam defenses. So they outsource it. These companies' customers are consumers. They want it cheap, and they don't mind if some legitimate messages are blocked along with the spam. So these outsourced spam defenses are getting *very* aggressive.

Now, when the outsourced spam defense sees a message filled with tons of cruft, the spam score goes way up. But if it came from Yahoo, they take some points off, because they know all Yahoo mail is full of tons of cruft.

But when you post through a gp-us.org list from Yahoo, your message isn't coming from Yahoo any more. It's coming from gp-us.org, so the special cruft discount for Yahoo senders doesn't apply. Some of this stuff is now spammy-looking enough that the outsourced defenses are rejecting it as spam. I'm seeing people getting unsubscribed from lists because their independent email companies are rejecting too much list traffic.

Yahoo Mail is becoming incompatible with the public email system.

Someone told me recently they think it's intentional. Yahoo wants to use its large market share to hurt other email providers. If that's true, it means the break-up of the public email system into proprietary corporate islands. But I think it's just a coincidence of accidents. Nvever ascribe to malice what can more easily be explained by...

If you really want to do something about phishes, don't bother reporting them to your ISP. Millions of people already hit that "this is spam" button. And it's only useful if you do it within a few minutes of your ISP's receiving it.

Instead, open the message source and find the URL of the fake bank site. We call that the payload URL. The whole point of the spam is to get you to go there with a web browser. The fake bank site was created by a dangerous gang of criminals. Don't forget that. Do not visit the URL with a web browser. It probably contains malware and will attack your PC.

Sometimes these are hosted on spammer-friendly ISPs in Eastern Europe or China. Do not report those. But most fake bank or credit union or Paypal sites are hosted on servers that the criminal broke into.

With a little common sense, you can safely figure out where the fake bank site is hosted. Look at the payload URL in the spam message source. You can spot it among the decoys and images because it's the one with "click here" or the domain name of the real bank.

If the payload URL is something like http://www.podunk-realtor.com/images/.hideme/bankofamerica.com/, you can be pretty sure it's a break-in, and the Podunk Realtor and his ISP or web design firm have no idea they are supporting large scale felony fraud. If it's more like http://www.cheaphosting.com/~someguy/bankofamerica.com/, it's probably a shared hosting account at a giant web hosting company, purchased with a stolen credit card. If it's http://www.paypaI.com/ (a subtle misspelling), the hosting company and the Registrar are probably in on it, and there's no point in reporting those. Some web hosting places are so careless that they might as well be in on it.

If it's in eastern Europe or China, leave it to the professionals. Stop here. You do not want to provoke the Russian mafia.

Copy the domain name of the fake bank site out of the message source and trace it (with "tcptraceroute" on unix or "tracert" on MSWindows, or use Samspade.org) to its hosting.

The output of the traceroute starts at your PC (or at Samspade) and shows the route to the fake site. The last item is the compromised server. It may have a domain name that belongs to the ISP, or to one of his customers. With a little practice you'll know which is which at a glance. Look up the IP address in Whois. ISPs generally own their own IP addresses. Even if they don't, it gives you a pretty good clue as to what company owns or controls that end of the route. In the case I called Podunk Realtor, it is probably safe to look at the front page of the Realtor's site with a browser, and find some contact info that way. Look up the domain name on the server, in Whois. That will give you more contact info, of the Podunk Realtor or his ISP or both.

Report the break-in to the owner of the server and his ISP. Out of the hundreds of thousands who received that phish run, you may be the only person to report it properly. Do not include a copy of the spam. That will probably prevent the victim from receiving it. Just tell them the IP address of the compromised server, and the URL you found in the spam.

You can look up contact addresses of a well-run ISP at www.abuse.net. Or just send to abuse@example.net where, of
course, example.net is the ISP's domain name. (If that bounces, report the ignorant ISP to RFC-Ignorant.org.)

Do that with one phish a week and you're doing more than all the times you ever hit "this is spam."

When we started seeing SMTP callbacks, aka "Sender Address Verification," several members of news.admin.net-abuse.email, including myself, said it was a Bad Idea.

It's trivially easy to get around SAV. The spammer just puts known deliverable addresses in his envelope-sender. Of course he needs to use thousands or millions of those in each spam run, to evade statistical filters. I'm surprised it took until now for them to figure that out. And because they have to be deliverable, it takes a higher quality list for the fake senders than for the spam recipients.

Meanwhile, there are still a lot of "anti-spam appliances" and other broken SMTP servers that accept and return messages to bad addresses, rather than refusing them. But you can't return spam once you've accepted it into your queue. You don't have an address for the spammer, and he isn't interested anyway. So the returned spam messages become a new form of spam known as "backscatter." Until recently, the Barracuda appliance in its default configuration sent backscatter. They've fixed that. Qmail-1.03 sends backscatter. There are patches for that. One popular Qmail backscatter patch is called "chkuser."

Two unforseen consequences combine for another harm. 1. SAV is becoming popular. 2. Backscatter. The backscatter used to go to the same poor quality address lists the spammers send to. So most of it never got delivered; it stuck in the Barracuda appliance or Qmail queue. But now it's getting delivered, adding to the spam load and degrading the statistical filtering results.

Not only are spammers destroying the public email system, but misguided Final Ultimate Solutions (FUSSPs) are damaging it too.

Big news, Robert Soloway busted for wire fraud, credit card fraud, CAN-SPAM violations, and using a bot-net.

Big deal. The prosecutor isn't even asking for prison time. That's the strongest signal yet that the US Government doesn't actually regard spamming and creating and running bot-nets as criminal behavior. It shows how successful US spammers have been at positioning themselves as persecuted entrepreneurs, not as criminal gangs. You can thank the Direct Marketing Association, the American Civil Liberties Union, and corrupt, stupid congresscritters like Zoe Lofgren for that.

And you can thank the US "news media" for spiking the story as systematically as they have blacked out anything in the Project Censored Yearbook.

Creating and using a bot-net is one of the most destructive computer crimes. A single bot-net operation can cause hundreds of millions of dollars of economic loss to consumers and businesses. Imagine if some criminal invented an automatic way to break into a hundred thousand people's cars and misuse them. Now imagine the DMA and the ACLU said that's okay, it's free speech!

Why is that so hard to understand? Because computers are "technical" and cars aren't?

My systems try to reject as much incoming spam as possible by its origin, so we don't have to spend time analyzing it all. There are basically three ways:

1. The sender's IP address, or the range it's in, is known to send nothing but spam.
2. The sender's software makes some protocol mistake during the SMTP session that legitimate senders don't make.
3. The sender identifies itself with a name that isn't defined, or its IP Address (numerical network location) doesn't have a name, or has a name that isn't defiined.

Now and then someone complains they can't send email to my users. Sometimes it's due to a mistake I made spelling out one of the "spammy" IP Address ranges in my block list. Sometimes the person is sending from a spammy place and I decide to override or remove the block for that particular place.

This week we had a new one. The sender's IP Address doesn't really have a name: its name is only an alias (a "CNAME") of something else. Of course I put an exception in my Postfix setup for that particular sender. (Chico.com.)

But tracking down the problem revealed an interesting cause. The Domain Name System was invented in the early 1980s by some geniuses (Mockapetris, Barr, Postel, etc) and it works in a way that is described in some "RFC" documents from the Internet Engineering Task Force. These RFCs achieve "standards track" status after years of discussion and testing by more geniuses. RFCs level the playing field, by saying exactly how Internet software programs must talk to each other. That way anybody can write Internet software and expect it to work pretty well with what's already out there.

The RFCs say an Internet Protocol Address should be given a name and that name should be published in a "Pointer Resource Record" ("PTR Record" for short) in the DNS. The geniuses reserved a special domain for those, in-addr,arpa. I suppose that means something like "inverse addresses on the ARPANET." Then they say the name in the PTR Record must be defined by an Address Record.

Postfix lets you reject email from senders who don't have those two things (a PTR Record and a corresponding Address Record) going for them. The test is called "smtpd_client_restrictions = reject_unknown_client" and it's really productive. There are tens of millions of unnamed cable modems and DSL lines, full of trojaned Microsoft boxes, sending spam, and we reject it all.

The standard book about the DNS, DNS and BIND by Albitz and Liu, mentions the requirement that the PTR name must have a real Address Record. (They also mention that a workaround exists in the Internet Systems Consortium's domain name resolver subroutines, for PTRs that have an alias instead of a real name.) Cisco's book says "PTRs use official names not aliases." IBM's tutorial for setting up DNS on its unix (AIX) servers says "the name in the PTR record should have an actual Address record."

But Microsoft's Knowlege Base says go ahead and use CNAME Aliases for the names in your PTR records! They're telling people to break the rule that helps us reject spam efficiently. Why? Well, Microsoft has an attitude about the Internet standards. They don't like anything that levels the playing field. So their software intentionally misoperates in subtle ways. That way, if you're in an all-Microsoft shop, your stuff will work, inside your shop, and you'll think those weirdos out there who use software from anybody else are using broken software. They're counting on the all-Microsoft users to not know or care about the standards. This attitude and behavior was identified in Microsoft internal memos as a strategy. It's called "embrace, extend, and extinguish."

Some jackass has been signing one o f my users up to Yahoogroups lists. He sends six or eight attached files from each, and moves on. It's been going on for hours. I blocked the email address he's hitting. He keeps hitting it anyway. Cluttering up the log but not getting through.

The server that connects to mine has a hostname in rDNS like this: n33a.bullet.sp1.yahoo.com. The IP address is 209.131.38.214 and its neighbors. ARIN says that's Yahoo.

I forwarded a few of them to abuse at yahoo.com. Abuse says "After careful evaluation, we have determined that this email message did not originate from the Yahoo! Mail system. The "yahoo.com" address associated with the email does not exist. It appears the true sender of this message forged the header information to give the impression that it came from Yahoo! Mail."

Dumber than a box of rocks.

This morning's "Quality Meds at Clearance Price" spam came from a trojaned consumer box on "broadband" in Malaysia. It had a bogus EHLO/HELO name. Either of those would have gotten it blocked, except it was addressed to Postmaster. You're not supposed to block spam to that RFC2142 address. (I'm getting tired of that rule.) Spammer must be pretty confident he's complaint-proof.

The spammer just gives his domain name, a throwaway at Register.com. They tell me these are usually paid for with stolen credit cards. He spells the domain name with spaces around the dot, to avoid triggering Spamassassin's "URL seen in spam" rule. The contact info in the registration is clearly bogus: 666 devils rd, lucifer, miami, +1.3056669990. Yeah, sure, lots of real people at that 666 exchange. By the time Register.com (Verisign still owns them?) takes it down, he'll have moved on.

The spammer's web server is hosted at Mzima Networks. A large colocation provider with data centers in the US, Honk Kong, Tokyo, and four cities in western Europe. A colocation provider rents you rack space in his data center for your server, which you connect to his network. Usually he reallocates you some IP addresses. If you're big enough you bring your own. Mzima has 21 entries on the Spamhaus block list. Mostly bunches of sixteen IP addresses. Most belonging to well known. chronic, "career" spammers. This one turns out to be "iMedia Networks." The 512 IP addresses are reallocated from Mzima to an " SBC Telecom Consulting, Inc." It's been there nine months.

I called Mzima. They told me that their customers can spam all they want, as long as they do it on someone else's network, and I should complain to the cable company in Malaysia. As long as the spam came from a bot-net, it's none of Mzima's business. Of course, well run networks won't accept email from an IP address assigned to a criminal like iMedia Networks anyway. He just sells his pills through them.

Mzima claims to be "connecting to multiple Tier-1 carriers and numerous private peers." But whenever I trace route to their spammer havens the route goes through Internet backbone carrier Level3. Of course Level3 doesn't give a damn about the criminal selling his fake pills through their network. They know the government isn't going to bother them, and Mzima pays them well.

Spammers exist because of the knowing, willful negligence of companies like Mzima Networks and Level3 Communications.

What you can do: Ask your ISP to "null route" the pill spammer's IP address range, 72.37.186/23. They're not expecting that. They're expecting you to complain about the bot-net pill spam, but they think you're too stupid to figure out that the spammer's web hosting matters more. Tell them you'd prefer that they not carry the pill spammer's traffic. Not just his email, which comes from everywhere, but his Web server and his bot-net controller too. Nobody's going to miss any legitimate traffic from there, because there isn't any. This happens, occasionally, to the very worst of the worst spammers. It renders their IP addresses fairly worthless, and they have to buy a new allocation from Mzima. Which leaves Mzima stuck with 512 IP addresses that nobody wants.

Of course, if we get the kind of "net neutrality" Moveon.org has been pushing for, such shunning becomes illegal. Under today's "free trade" agreements, the boycotts that forced the end of Apartheid in South Africa would be illegal. Think about it. Do you really want a "free trade" Internet? You can bet Level3 and Mzima do. And the spammers would just love it.

My server refused Michael's email. He's running a server at home on a phone company DSL line with a static IP address. "What's this about? Can I not send you mail because I have SBC DSL?"

Not exactly. It's because he's sending directly from a residential SBC DSL line with a "generic" name in reverse DNS.
$ host -t ptr 68.124.123.45 #(not his real address)
45.123.124.68.in-addr.arpa domain name pointer adsl-68-124-123-45.dsl.pltn13.pacbell.net.
Legitimate email senders just about always have a pointer record in the in-addr.arpa domain. The name in it suggests it's actually supposed to be sending mail. Picking a couple at random out of this morning's email:
$ host 204.13.164.18
18.164.13.204.in-addr.arpa domain name pointer mx1.riseup.net.
$ host 66.159.220.136
136.220.159.66.in-addr.arpa domain name pointer amybiehl.greens.org.
Take a look at amybiehl's network neighbors.
$ whi -v 66.159.220 132 141
132.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-132.dslextreme.com.
133.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-133.dslextreme.com.
134.220.159.66.in-addr.arpa domain name pointer alexsoo.net.
135.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-135.dslextreme.com.
136.220.159.66.in-addr.arpa domain name pointer amybiehl.greens.org.
137.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-137.dslextreme.com.
138.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-138.dslextreme.com.
139.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-139.dslextreme.com.
140.220.159.66.in-addr.arpa domain name pointer netblock-66-159-220-140.dslextreme.com.

It's pretty easy to see which are the servers and which are just generic residential lines. Michael's email was refused because his server's pointer resource record name adsl-68-124-123-45.dsl.pltn13.pacbell.net matched the regular expression/^(adsl|ppp)-.*.(dsl|dialup)\..*\.pacbell\.net$/which stops an amazing amount of spam without any content analysis. SBC/Ameritech/Snet/AT&T/Pacbell generics are in the top twenty spam sources world wide. They claim to be "rolling out" port 25 blocking, but there are glaciers that move faster. I'll whitelist your PTR name, but I doubt many admins would bother. If you're gonna run a server in the middle of a block that's 99.99% Microsoft-DSL-residential spam zombies you're eventually gonna have to ask SBC for a non-generic name in rDNS. Or send through the SMTP relays SBC provides.

Your MTA probably has a routing table where you can relay out to certain domains and send directly by default. I can show you how to do it in Qmail and Postfix. I've noticed about a dozen domains that block amybiehl, and I route to those through my server off of Speakeasy.

You'll find this kind of preemptive blocking will only get more common. I'm kind of surprised you haven't hit it already. Some pretty large networks have been using this technique for years. DSL Extreme charged me $20 to install a custom PTR name. Sonic and Speakeasy did it for free.

The "see greens.org/delist" in my server's rejection message was supposed to lead you to my whitelist request form. Someone submits that form about once a month. (Not counting the crackers who shove junk into it every day. They're looking for leaky forms they can exploit to send spam.)

A poster on Techrepublic boasted that his workstation security suite (for MS-Windows) "blocks" spam.

AVG Internet Security does a lot of good things. I recommend it to my customers who still use Windoze. We prefer it to Symantec or McAfee. But it doesn't block spam. Nor do its competitors.

If you're using the typical consumer setup where you download your email via POP3 from your ISP's mailbox server, your workstation doesn't see the spam until it's already been delivered.

AVG Internet Security and its competitors filter spam. That is, they analyze and sort it. One good optimization you can do with POP3 is pull all the message headers, analyze them, and delete the obvious spam from the mailbox before downloading the whole messages. I'd be surprised if they don't do that, at least as an option. But that's not available if you want to download all the spam into a local spam "folder" to look for false positives.

Only your email service provider can block spam. That's because blocking happens before the SMTP server (receiving system) has accepted the message. The SMTP server has to consider the source while the wanna-be SMTP client (sender) is waiting to connect, or analyze the message on the fly while the client is waiting for a response.

There are two significant differences between blocking and filtering.

1. Blocked spam from spamware just disappears. (Spamware is the specialized software criminal spammers use for sending. Most of it is installed on PCs the criminals have broken into, using trojans or rootkits or the like. A lot of it just connects and blasts away, without paying any attention to the responses from the SMTP server.) But blocked spam from a "legitimate" sender piles up in the sender's outbound queue or gets returned. That gives the sender feedback that he's sending unwanted mail and/or the address is bad. In the case of a legitimate sender exploited by a criminal spammer, it gives him feedback that his security is compromised. Filtered spam appears to the sender to have been delivered. The "legitimate" spammers (the minority who try to comply with CAN-SPAM et. al.) are deprived of the chance to clean bad addresses off their lists. Those criminal spammers who pay any attention to SMTP responses at all are told your address is deliverable, which makes it more valuable to sell to other spamemrs. In the end, filtering is a way of (partially) automating the process of "just hitting delete." It adds to the overall problem.
2. Blocked spam doesn't cost the non-recipient anything to store or download. And blocking before body content analysis is a whole lot cheaper than filtering.

This morning I got a spam from a hobby server on DSLextreme.com. I'm a DSL Extreme customer. Their customer service is great, and they don't tolerate spam. In fact, the last time I reported one of these to them, they misinterpreted the report and blocked me. Shoot first, ask questions later.

So today I copied the spam sample to my server at Explosive.net and sent the report from there. Explosive is really great, too. But their Internet Protocol Address (IPA) space is on Speakeasy.net, and Speakeasy's just as mean to spammers as DSL Extreme is. So I'm still taking a chance.

That's what it's come to. The well-run retail ISPs are few and far between. You don't want to be anywhere else. But the well-run ISPs are on such a hair-trigger you have to think twice about sending legitimate email that could be mistaken for spam. Argh.

Meanwhile, I'm shopping for a low-cost, well run virtual private server (VPS) in squeaky-clean IPA space. My users want to host video clips and I can't do it from the colocation at Explosive. Speakeasy and DSLExtreme don't offer VPS. I considered GPLhost but they're on PCCW, which doesn't seem to handle abuse complaints competently. Drop me a line if you've got any ideas. Charlie Lima Sierra at Truffula dot Sierra Juliet dot Charlie Alpha dot Uniform Sierra. Geez, think they'll harvest that?

I've seen estimates that less than one in a million spams results in a well-directed complaint.

Almost every week I see bad advice about where to report incoming spam.

Never reply to a spam message. The reply address is probably bogus, and if it's real, you just made your address more valuable to other spammers. You can't mailbomb them. You can't exhaust their web servers with repeated requests, either.

Don't report spam if you're not computer literate enough to save a spam into a plain text file and look at the headers. That means the lines in the message headers that begin with the word "Received:." If the files you save don't have those, don't bother. If you do not know the difference between a plain text file and an MS-Word document with the font set to Courier, don't bother. But if you can include the message in-line, not as an attachment, without destroying the headers or adding word processor crap, go for it. Report it to:

Your email service provider. That's nice. Sometimes it helps "educate" or "train" their filters. AOL and Yahoo! Mail do that. It does approximately nothing to the spammer. Your ISP is probably not going to contact his ISPs.

Spamcop. That's nice too. It helps ISPs who subscribe to Spamcop's block list block more spam from the same source. Don't bother if the spam is more than an hour old. Unfortunately Spamcop also offers a "personal" software product that's supposed to analyze the spam and help you generate a report. But it's not very accurate, and a lot of ISPs, maybe most, ignore those reports.

The FTC. You can forward spam with complete headers to spam@uce.gov. They keep statistics.

The SEC. You can forward stock spam to enforcement@sec.gov. They bust some criminals sometimes. If it's "image" spam, put the stock symbol that's being promoted in your subject line.

news.admin.net-abuse.sightings. That's a Usenet newsgroup for posting spam samples. People use it to research spam patterns. If you can't post the contents of a plain text file, in-line, to a newsgroup, don't bother.

The owner of the exploited equipment. Almost all spam is sent through computers the spammers don't own. Spammers break into servers through leaky Web applications. Or they steal or guess weak passwords. They break into PCs in people's homes, on DSL or Cable, through "virus" infected email and malware infected Web pages.

Look at the Received: header line where your service provider receives the message from someplace that's not your service provider. (If you can't read, don't bother.) Maybe it's a cable company you've heard of. Look up that company's abuse reporting address. There's a service for doing that, at abuse.net. You can query abuse.net with your whois program (e.g., whois -h whois.abuse.net hotmail.com), or use its web site. The DSL or cable company will (sometimes) contact the owner of the compromised computer.

The giveaway for those DSL or cable senders is a so-called "generic address." I'll pick two examples from today's incoming spam. The hostname wsip-70-183-84-39.dl.dl.cox.net is generic. It's got numbers in it that are the same as its IP address. The hostname mercury1.networknoc.com is not generic. If it's not a generic name, it's not one of those home machines. It's either web hosting or a small business. You can figure out who the ISP is with your traceroute or tcptraceroute program. You'll never figure out who the owners of the individual cable/DSL zombies are. But their ISPs know. You can try calling the owners of the exploited web servers themselves. But if you can't talk about the spam they're sending authoritatively, they'll think you're just harassing them or trying to sell something. It's easier to just send a spam sample to the abuse address at their ISP.

Sometimes spammers break into other people's computers to host their name servers or Web servers or both. Never go to the URL in a spam message. If you use MS-Outlook [Express] or Thunderbird don't even open your email with image display enabled. But you can trace the name in the URL. And you can trace the name servers. They're named in the domain's Whois entry or you can look them up with your host or nslookup or dig commands. If the servers are on cable/DSL or at hosting places in western Europe, Australia, or North America, report them. Elsewhere, it's probably not worth the trouble. If it's in China, South Korea, Russia, or Bulgaria, sad to say, don't bother. As far as I know, all ISPs in those countries are spammer-friendly. You can look up the ISP at Spamhaus.org if you're not sure. There is no point in reporting spam to a spammer-friendly ISP.

"Free" email providers. A certain type of spammer prefers to use throwaway accounts at Yahoo Mail, Hotmail, Excite.com, etc. Those are the "advance fee fraud" or "Nigeria 419" scammers. If they're fresh, report these to the abuse address at the email company. If you received it more than 24 hours ago, don't bother. Notice that the "Reply to" address is hardly ever the same as the "From" in these things. Sometimes the Reply to address is repeated in the message body. Those are the ones that are worth reporting. The From address had already been discarded by the time you saw the spam.
Notice that abuse@hotmail.com does not work and never has. Hotmail (Microsoft) thinks the rules of the Internet don't apply to them, and their special abuse address is report_spam@hotmail.com. Also, the abuse address for Yahoo Mail is always abuse@yahoo.com, even for the country domains like yahoo.co.uk.

That's all. If you're savvy enough to find other assets in the spammer's network, you already knew all this stuff and didn't have to read this far. Experts go after the credit card processors. Uber-experts sometimes take legal action. But this is not work for amateurs. Remember spammers are criminals. Spam is international organized crime. You don't want to provoke these people if you don't know what you're doing.

I reported a spam to an email admin at an Indian reservation. He replied with a nice thank you note. His technician has been trying to stop the spam coming out of their MSFT system for a few days, with no success. I offer general advice:

Hi Justin, thanks.

I hope you won't mind some unsolicited general advice about the problem.

You're using a Microsoft system for your exposed email server. That's going to be an ongoing headache. Believe it or not, and despite everything you have read in the trade press and heard from Microsoft's sales force, their operating system is not designed to be exposed (on a "routable" address) directly to the Internet.

The customers Microsoft listens to, that they design their system for, are the Fortune 500 corporations. Consumers, small business, and distributors like Dell and Gateway, are taken for granted, because they have been taught they "have no choice." Their (our) needs are not considered in Microsoft's design decisions. Fortune 500 corporations do not expose Microsoft systems to the Internet. They hide them behind layers of protection: proxy servers, firewalls, "policy servers," and other equipment.

You would be wise to start thinking about placing some non-Microsoft system between your exposed address and your internal Microsoft email system, to relay email in and out, and be a firewall. PCs are very cheap now. You can put a PC running FreeBSD or GNU+Linux between the Internet and your private network for less than you spend on "anti-virus" junk for a few MSFT machines. The PC you retired because it wasn't fast enough to run Windows XP very well will usually do. You can stop the "virus" email and 90% of the incoming spam with it, as well as the criminals who compromised your current system.

It takes a bigger PC to run today's comprehensive spam and virus filters, but even a serious compute engine only costs a few hundred bucks these days, and all the software you need to do it is truly free and trustworthy.

A painless and risk-free first step down this road is to try a couple of "Live Linux" CDs. These let you temporarily run a fully functional computer system on your current PC, directly off the CD, without disturbing your current software installation and without installing anything. I recommend Knoppix.net, but Ubuntulinux.org is more popular. If you have an older, smaller PC, you might try damnsmalllinux.org instead.

--
Best wishes,
Cameron in San José
http://greens.org/cls/

After resisting for years, I've taken the second step down the slippery slope of content filtering. My first lines of spam defense will continue to be source blocking and SMTP mistake-catching. But that only gets you so far. The criminals who break into legitimate web hosts get through. The only way to get them is analyze the messages.

The first step was Postfix' header_checks and body_checks. They stop some of the most obvious stuff. But Postfix warns you not to get carried away, and you can't combine different checks. "If it says it's from Paypal but it wasn't sent from their IP space" is too complex.

The second step is a big one. We set up a special local server, Amavis-new, that Postfix can consult as it decides whether to accept a message. This evaluation has to happen fast, while the sender (client) is waiting for the receiver's (server's) decision. Once you accept the message into your delivery queue, it's too late to refuse it. You can't return it, because once it's yours you don't really know where it came from. The client is long-gone, and the "From:" address in spam is always a lie.

Amavis-new's biggest module is Spamassassin, a collection of thousands of little "tests" that can be intricately selected, combined, and scored. Amavis-new considers Spamassassin's opinion of the message and advises Postfix to refuse the spammiest. It leaves marks on the messages it accepts, so that the final recipients can sort them as they're delivered. A very cool contraption. Each part is carefully and independently maintained.

It's software for professionals; the "documentation" is great reference material but scant tutorial. And there are lots of ways to put the pieces together. The maintainers of each piece have rather little to say about all those ways. They're responsible for their respective pieces, but you're responsible for your contraption. I have the O'Reilly Spamassassin book and the No Starch Postfix book (they're both pretty good) and I still had to ask for help. Someone on the Debian-ISPs list sent me exactly the clue I needed, immediately. Somewhere in Amavis-new's documentation they tell you that amavisd will only mark up messages destined for "local" recipients. That's what the @local_domains_maps variable is about. It's in the sample config file.

@local_domains_maps list of lookup tables are used in deciding whether a
recipient is local or not, or in other words, if the message is outgoing
or not. This affects inserting spam-related headers for local recipients,
limiting recipient virus notifications (if enabled) to local recipients,
in deciding if address extension may be appended, and in SQL lookups
for non-fqdn addresses. Set it up correctly if you need features
that rely on this setting (or just leave empty otherwise).

Well that clears everything up. Spamassassin itself is distributed through the amazing Comprehensive Perl Archive Network. Perl just gets it for you. Even though Perl is nearly as efficient doing complicated things as you would be in a lower-level compiled language, there are a lot of tests and Spamassassin is big and slow. You can run Postfix on any old PC, but you need a modern CPU and lots of RAM to run this contraption. I'm going through disabling the tests that duplicate things I already did in Postfix.

It comes up all the time. "We're losing this escalating battle of blocking and filtering and reporting abuse. So why don't we just change the public SMTP email system (insert technological wonder fix here) so it's less vulnerable. I'm a genius! I've invented the Final Ultimate Solution to the Spam Problem (FUSSP)!"

Technological wonder fixes include: centralized filtering plants like Postini, Sender Policy Framework, postage via (insert your pet micropayment scheme here), certify senders at some central authority, Challenge-Response systems, block by default and whitelist by default, and more.

Each of these techno-fixes has its own faults, which have been well described elswhere. But they share one common problem: if you somehow magically manage to impose (insert aforementioned techno-fix here) across the whole Internet, it's not the public SMTP email system any more. So what you are really proposing is to replace the public email system with some other system.

All revolutions have the same problem. First you smash the state. Then your replacement state is supposed to take over. But the instant the state is smashed, there's a power vacuum, and a race with no rules begins. While your replacement state is fiddling around with tedious processes like elections and confirmations and adopting a constitution, a bunch of thugs is establishing an unethical dictatorship. It's faster. First brute to the top claims the flag, no matter who he had to kill to get there. If the public email system falls, its replacement will be worse. Here's why.

In prehistoric times, before, say, 1994, the Internet was governed cooperatively, by consensus, by bodies like the Internet Engineering Task Force (IETF). New services were developed, or at least adopted, in the open. Standards were evaluated by their merits. A simple rulebook, the set of IETF Requests For Comment, said how everything would work together. SMTP email is RFCs 2821 and 2822. They sort of depend on rules of responsibility like RFC 2142 (it says your postmaster@ and abuse@ addresses are supposed to work...) among others. There was never any RFC Police, it was simply known that if your software didn't conform it wouldn't work well with other people's software, and if you had abusers on your network everybody would wall you off in their firewalls and you'd lose your connectivity, and that was enough. The public email system was developed under this system of merit-based consensus.

Creating the Internet may be the biggest project in human history done under consensus governance and functional Anarchy. Anarchy with a capital A doesn't mean chaos, it means there's so much personal responsibility that you don't need a government. Nobody in charge. No cops, none needed.

Then a bunch of marketroids took over. They emerged from pods which arrived from outer space or Wall Street or someplace, an invading army of high maintenance parasites. Moneymen. They brought with them the unethical concept of intentionally violating the RFCs to obtain some kind of competitive advantage. Microsoft (stock symbol MSFT) announced it was going to "embrace and extend the Internet!" and published a bunch of software that doesn't play well with everybody else's, on purpose, to begin to force computer users and developers to choose between universal interoperability and the way that MSFT could control.

At about the same time, a tiny handful of Internet "entrepreneurs" decided the rules that held the network together didn't apply to them, and they were going to let their customers develop email spam as a new kind of advertising medium. (Which makes as much sense as going into business sticking advertisements on other people's store windows and billboards, and garage doors, and trees...) Net99, later known as AGIS, was the first to be really public about it. They said consensus governance was "a throwback to the sixties" and the people who used it were "neckbeard geeks." They went under, but the idea caught on with the marketroids, who were still trying to figure out whether they were going to "turn the Internet into" a new kind of shopping mall or a new kind of television. Anything but a new kind of public library or college.

The days of friendly consensus were over. Netscape and MSFT introduced conflicting "extensions" to HTML, the language of Web pages. Yahoo and AOL each introduced instant messaging that didn't talk to the other guy's system. Real Networks got away with introducing a trade secret way to stream audio, killing off the far more economical and efficient and open system of multicasting, and the MBONE network that had used it for years. Any replacement "email" system will go the same way. Competing systems that don't talk to each other. At least not very well.

Will we use MSFT's micropayment scheme, or Yahoo's, or Ebay's, or Google's? Will email software have to know how to use all four? What if MSFT's system doesn't work with the other three but they ship it in Vista Service Pack 1? I can answer that: MSFT owns and controls the new "email" system.

At the same time we lost the ability to deploy new open services, we pretty much lost the ability to deploy major changes to the services already in use. You can break the system we have into pieces, but there is no way to push a significant change in how things work all the way out to the edges. Most people administering email sytems today have never heard of the IETF and wouldn't read an RFC to save their businesses. They just do whatever the salesman or the tech support voice tells them to so they can go back to their "real" job.

So it turns out we only have two choices, fight to save the system we have, or let the bad guys destroy it while the marketroids sit back and laugh.

This was on the business page in the Los Angeles Times. "An Azusa man who defrauded users of Time Warner Inc.'s America Online unit by sending e-mails requesting credit data became the first defendant found guilty by a jury under a 2003 federal law barring Internet spam." (Link.)
The so-called "CAN-SPAM Act" (aka You Can Spam Now) doesn't actually ban spam, it legalizes it. It specifies a few things spammers must do to be "legitimate." No fake headers, valid "remove" mechanism, physical contact info. Almost all spammers ignore this law, as they ignore other laws. (In 1997 the Direct Marketing Association, fronted by an utterly clueless ACLU, killed the only reasonable spam law ever written in the US Congress.)

According to the story, this was a one-man phishing operation and the guy took in about a million dollars.

Four years to the first conviction. Sentencing in June. My prediction: the criminal positions himself as a "businessman" and gets a pat on the wrist. Fine and probation, and he doesn't even give up all the ill-gotten gains. At worst he goes to one of those "gentlemen's" minimum security places for a month or two.

Well, congratulations to AOL for pushing this through. We know the FBI doesn't move on these cases unless someone does most of the work for them.

I got a fake Bank of America message yesterday, sent from a compromised MS-Windoze computer on Surewest DSL. Left a message on the owner's Web site and he called me.

He was pretty angry, but it wasn't at me in particular. He's trying to run email service for three hundred customers on that computer, using some commercial mail-server-in-a-box product. He'd already fielded four trouble calls on it that day, and that was a typical day. It's keeping him from running his web design business. He didn't know it was spamming. Surewest hadn't called him, of course, despite my report to their RFC2142 abuse address.

The email server wasn't coping with the spam load, and it was just "falling over." He's not using a DNSBL, but trying to filter the raw stream. Sorry, buddy, a 3 GHz P4 can't keep up with that any more. You have to block some fraction of it first.

He was absolutely sure he couldn't possibly have a spam bot or an intruder. Because he spends $hundreds/year on antivirus software (and the one he uses has a better reputation than Symantec or McAfee), checkups by Web sites, and specialized software that's supposed to monitor his outbound traffic for spam content. Evidently these measures don't work so well.

He was absolutely sure the spammers were "spoofing his IP address." That's so hard to do that even though the big spam gangs know how to do it, they don't bother. An explanation of why that's true would have gone high over his head so I didn't try.

It took a while but I got him to look in his server's outbound queue, and there were hundreds of Bank of America and Amazon phishes waiting to go out. Not all receivers are ready on the first try. He was puzzled. You're only looking at the leftovers, buddy, most of it's been sent. That was what it took to convince him something was happening on his machine that he didn't know about. The spambot was generating phish messages and letting his commercial email software queue it and send it. Phishers seem to like doing that way Exploiting a legitimate server doesn't get you blocked so much.

I explained that no antivirus can defend you from the zero day threat, that is an attack so new your antivirus doesn't detect it yet. It was the first time he'd heard of that.

We talked about possible solutions but he wasn't confident he could do any of them without disrupting his business. He'd been considering outsourcing the email operation and I think this convinced him. Spammers drove a small businessman out of the field, after nearly ruining his business and his life. Now there are fewer choices of email providers for the rest of us. Tell that to some bozo who thinks spam doesn't hurt anybody and you should "just hit delete."

A comment posted to my callbacks article suggests Google may have resorted to SMTP callbacks. AKA Sender Address Verification. Not only that, but their callback sender identifies itself as mx.google.com, and that name doesn't resolve in DNS. If it's true, everybody running an email server faces a choice.

1. We can figure out how to configure our servers to accept the bad HELO name from Google. But we can't just give mx.google.com a pass, because then the spammers would all use that name.
2. We can let Google's callbacks fail. Then Gmail users will refuse our email and they won't know why.

Maybe it's just a transient failure, and Google will publish an Address resource record in DNS for mx.google.com and fix it.

Spamming is a lot like other types of industrial pollution. It happens because "legitimate" Internet companies make a calculated business decision that they can get away with tolerating it to some degree. Some tolerate a lot, others hardly any. They know there won't be any law enforcement. The main consequence of hosting spammers is some of their IP addresses will get listed in databases like the Spamhaus Block List. (There won't even be adverse publicity: the rare newspaper or trade journal story never holds the "legitimate" ISPs responsible.) Some just want to save the money an abuse desk staff would cost. Or they've laid off the technical staff that would have been able to block the outbound "port 25" route the bot-nets send through. Others are attracted to the high rates the big spammers are willing to pay. The twenty-something MBAs and uninformed, timid lawyers who influence these decisions can find plenty of justifications for tolerating spammers on their networks. Just as they can justify dumping toxic waste overseas, stinking feedlots, clearcut runoff, and any other pollution whose source is at all obscure.

In the early days of the crisis, spammers simply paid more for the same service than law-abiding customers would. It was understood the premium was a fee for the ISP to ignore some level of complaints. A few ISPs (AT&T and Paetec...) got caught putting this agreement in writing; we call that a pink contract. Pink is the color of Hormel's SPAM.

For all I know there are still pink contracts in Asia and eastern Europe, but I haven't heard a pink contract allegation against a North American ISP in years. Here in North America, the spammer simply buys much more service than he is actually going to use. He rents a whole rack in the data center to hold just one or two servers, or he orders a $5000/month T-3 connection when a $600/month T-1 or $100/month SDSL connection would easily handle the traffic he is going to generate. (He's not going to send spam through his own link, he's just going to host the target Web sites and control his bot-nets through it.) Salespeople for Internet services beyond the consumer retail level, it seems, work on commission. So the overspending by the spammer gives him an advocate inside the ISP who will fight hard to keep him connected despite the complaints, and despite the crimes he is committing which the ISP is an accessory to.

Spammers are a natural response to the ecological niche that tolerance creates. It's no more their "fault" than a dirty kitchen is the fault of the cockroaches that thrive there. The rationalization we hear from the hosting companies is almost always simple buck passing. It's always somebody else's fault, their "hands are tied," you're complaining to the wrong people, yadda yadda yadda.

Here's the most common reaction I get from my friends to my concerns about the spam crisis.

I don't get much spam. Postini/Yahoo/Earthlink (etc) filters it into a spam box for me and I never even look at it.

You may not have seen it, but you still PAID FOR IT. Every message your ISP received for filtering cost network bandwidth. Bandwidth costs money. Every message your ISP stored because there's too much volume to filter in real time cost storage space. Disk drives may be really cheap but managing them and backing them up and powering them and cooling them isn't. Filtering one 50 KB message for spam and attachment viruses takes several tenths of a second on a 2GHz CPU that burns 80 watts. That's as much energy as sending you a typical web page, and it happens many times more often. With billions of spam messages per day, spam is consuming significant amounts of fossil fuel. One ISP told me spam filtering consumes more electricity than everything else in his data center. Now that spam is 97% of email, and the average spam message size is over 10KB, the cost of receiving and storing and filtering all that junk that you "never see" is the biggest component of the cost of your Internet service. You don't see it in your inbox, you see it on your monthly bill and you'll see it in anthropogenic climate change.

But there's a bigger problem that filtering doesn't solve. The volume of spam has been doubling in less than a year. It could double ten more times. There are enough vulnerable Microsoft PCs for spammers to take over. Spam would be more than 99.9% of email. But you could not stand to pay two hundred times as much for Internet service. Filtering will hide the problem from consumers until most of the Internet email system has already collapsed. It will prevent us from doing anything effective to stop spam and save email.

That's the real harm filtering does. Hiding the problem prevents you from fixing it.