How does the Onlinespamfilter filter learn? Part 2: The structure of email

Spammers constantly change their approach to outsmart spam filters. So a spam filter can’t stand still either. Last time we explained how spam filters learn by analysing word combinations in emails. Besides looking at the words, spam filters also look at the structure of an email. How do they learn from that?
Penalty points for spam
Spam filters check the structure of an email against thousands of ‘rules’. Does the email contain a form? Is there a plain-text version available alongside the formatted HTML version? How long has the domain name the email was sent from existed? How many letters does that domain name have? You can hardly think of anything so crazy that there isn’t a rule for it.
And every one of those rules carries points. Or rather: penalty points. Because the higher the number of points, the more it leans towards spam. If an email’s final score goes over a hard limit of, say, 10 points, then according to this method we are unmistakably dealing with spam.
Now, one rule weighs more heavily than another. For example, a possibly not entirely sound analysis once concluded that a domain name of twelve letters often points to spam. Because this isn’t a very strong indication of spam, this rule only adds a few tenths of a point. A stronger indication of spam, for example when the body and header of an email contain more than 78 characters per line, earns more points. Conversely, points are also deducted for rules that are associated with normal email traffic.
New rules of the game
This anti-spam tool, called SpamAssassin, is open source and anyone can download it. The system isn’t very smart, though: it doesn’t learn from what it comes across, while the world of spam is of course constantly on the move. That’s why Onlinespamfilter regularly adjusts the ‘rules of the game’. Based on new knowledge, we give some rules more weight and others less.
Sometimes we adjust the scoring by hand. For example, we lower the points for rules that fire on ‘no text version present’ and ‘domain name does not exist’ when a customer uses a printer that sends scanned documents by email. Such devices tend to ignore a few rules of etiquette, such as including text and setting a domain name that exists. Only if all sorts of other things are wrong with such an email as well will it reach the ‘definitely spam’ score.
Learning automatically
But the spam filter also comes up with suggestions of its own. It compares the outcome of the scoring with that of other scanning methods, such as the statistical word analysis we discussed last time. If the filter gets it wrong a few times, it will suggest charging more or fewer ‘spam points’ for certain rules.
That process isn’t fully automatic yet, by the way. Although the filter sometimes comes up with sound suggestions, it misses the mark completely in at least half of the cases. For example, 99% of all email coming from China is spam, but at Onlinespamfilter we know from experience that a score based too heavily on geography can lead to email being wrongly marked as spam. However smart the filter is, when it comes to recognising spam, we humans still have the final say!
