How does the Onlinespamfilter filter learn? Part 1: Statistical word analysis

A magnifying glass over a text with highlighted words

Spammers constantly change their approach to outsmart spam filters. So a spam filter cannot stand still. The Onlinespamfilter spam filter learns in no fewer than four different ways. In the coming weeks we will look at each of those ways. In part 1 of this series: the statistical analysis of words in email messages.

Suspicious word combinations

This method centres on word combinations. One word is often not enough to judge whether something is spam. Take a spam-sensitive word like viagra. If the same message also contains words like ‘order’ or ‘payment’, then that email is definitely relevant to a medicines wholesaler. What you don’t want as a business are emails in which, besides ‘viagra’, words like ‘manhood’ or ‘add size in the bedroom’ also appear. If such words are close together in the text, for example with at most six words between them, the spam filter raises the alarm.

Smart probability calculation

When the spam filter comes across a particular word combination, it then applies a smart way of calculating probability to it, based on Bayesian statistics. The filter not only calculates the probability that it is spam, but also gives a confidence level: how sure is the filter that its calculation is right? It bases that confidence on previous experience. Put together, this gives an overall probability that predicts whether a word combination it has found points to spam or not.

Suppose an email comes in and the filter indicates that the probability that it is spam is 90%. But the word combination in question has occurred so rarely that the filter is only 30% sure of it. With that much doubt, we assume the email is innocent and it is simply let through.

Fast learning curve

But how does the spam filter actually learn, so that it can say with more and more certainty whether a word combination points to spam or not? First of all, you give the filter some basic knowledge. In the beginning, for example, we spent days building a database. We took thousands of emails from our own mail servers and those of our customers, and divided them into two categories: one with clearly innocent emails, the other with definite spam.
Next, you have to train the spam filter. As soon as it wrongly flags a message as spam, it gets a corrective tap on the wrist, as it were. The faster you correct mistakes, the faster the filter picks up the new lesson. A single tap is then enough to keep it from making the same mistake again. Our spam filters learn lightning fast this way; within just a few weeks they were predicting 99.99% of cases correctly.

Individual training

On top of the foundation that Onlinespamfilter provides, you can ‘train’ your spam filter further as you see fit. Each customer lays their own ‘layer’ over the database of suspicious word combinations, so the spam filter behaves slightly differently for you than for another customer. For example, we saw that a pharmaceutical wholesaler does want to receive certain emails about viagra. And a mass-mailed email from China offering radiators or wheel rims can be very interesting for a company that trades in car parts.

What we learn from the spam filter

Our spam filters are so smart that these days we regularly learn something from them ourselves. An email gets stopped and we think: surely this isn’t spam? Then we look at the statistics, and it turns out that the email really was sent in the hundreds at once. What a spam filter spots immediately, a human sometimes only sees – by reading very carefully – in the second or third paragraph.
Curious about the other ways spam filters learn? Read about them in our next blog posts!