On 11 November 2008 McColo, a Californian server hosting company, was disconnected from the Internet. This took the controllers for 6 major botnets offline. It has been widely reported that email spam volumes were markedly reduced for some time thereafter. But did disconnecting McColo only get rid of “easy to block” spam?
In a paper presented this week at the Sixth Conference on Email and Antispam (CEAS) I examined email traffic data for for the incoming email to a UK ISP to see what effect the disconnection had.
The first thing to note is the amount of spam — which was high (and highly variable from day to day) — reduced sharply. The figure shows the amount of spam each day in weeks 1 to 7. McColo was disconnected late on the Tuesday of week 6.
However, several heuristics cause spam to be rejected before an (expensive to run) content filtering stage: email from sites in the SpamHaus Policy Blocklist (PBL) is discarded; greylisting is used; and the ISP is picky about SMTP protocol compliance. Before McColo’s disconnection, between 32% and 56% of the spam did not need to be assessed by the content filters; thereafter a more consistent 43% or so was detected before the content filter stage. The second figure shows this, the key message being that although spam was less, a lot of the huge variation from day to day was email that didn’t need to be assessed by the content filters.
The ISP applies some special rules for subsets of customers. The next graph (note the different vertical axis) show how much email could be discarded because it was sent to non-existent addresses. On some days this mechanism immediately discards 900,000 emails. However, once McColo has been shut down, a mere 50,000 or so emails a day are blocked.
Spam to a few customers is rejected if the sender IP address is on a blacklist. This mechanism was being tweaked during the study period, so data is only valid during the first half of November. Nevertheless, the last figure shows a clear effect when McColo is shut down. The decrease is proportionately more than the overall drop in spam, meaning that blacklists ceased to be as useful in the immediate aftermath of the McColo closure.
The disconnection of McColo was obviously a Good Thing, because of the substantial, albeit temporary, reduction in spam. However, particular types of detection mechanism ceased to be as effective. Headlines of 60+% spam reduction only tell one part of a complex story.
@ Richard,
Sorry but I’m going to sound picky again 8(
I know the dates involved are very much in your head but that is not the case for the first time reader.
When you say a graph shows 7 weeks and the pivitol event happens on Tue of week 6 I had to go back to the top of the article to find 11th Nov.
Further down you say only the first half of Nov is valid but it is by no means clear what you mean and the exact days etc.
Finally you have a bar chart which is numbered 1 to 16 on the bottom but it is difficult to see how this relates to the rest of the article.
I’m assuming it relates to the days in the first half of Nov?
@Clive
I would recommend reading the paper… which has a few more words in it, and therefore should be clearer.
Further down you say only the first half of Nov is valid but it is by no means clear what you mean and the exact days etc.
Finally you have a bar chart which is numbered 1 to 16 on the bottom but it is difficult to see how this relates to the rest of the article.
The bar chart numbers are the ordinals of the days of November, it is the “last figure” referred to in the text, and you can see how the effect of this blocking rule decreases on the 12th Nov, and becomes far less effective from the 13th onwards.