How does the Onlinespamfilter filter learn? Part 3: Big data

A silhouette in front of a wall full of glowing data points

Spammers constantly change their approach to outsmart spam filters. So a spam filter has to keep up with the times. In the previous parts we explained that spam filters learn from word combinations and the structure of email, but they keep track of much more data than that. Think of images and URLs in the email, or the behaviour of servers and IP addresses. Collecting and analysing data about the context of emails enables spam filters to spot new connections at lightning speed.

Which data is stored?

For every email that passes through our spam filters, we store data in real time. Which country does the email come from? What’s in the subject line? What are the dimensions of the attached image? There are many other contextual details that can be relevant when detecting spam. The tricky part is that you don’t know in advance what is going to deviate, and so what you need to look out for. That’s why spam filters simply collect all the data they find in one big catch-all bin.

How is the data stored?

This catch-all bin with large amounts of unstructured information (‘big data’) is a so-called NoSQL database. The data in this database is not queried and combined on a relational basis, but according to a key-value strategy. With a cleverly built data structure, it is then possible to make new connections in real time.

How is the data analysed?

It would of course be great if this ‘big database’ spotted patterns entirely by itself and recognised spam before we humans could suspect anything. For now that’s still a thing of the future, but by entering the right queries we get a long way. For example, we ask the database questions like: how often does the same little file come by, how often do we see the same URL, or the same IP number within a short time? Then the statistics do the work. If a striking number of emails suddenly come by from a particular region in Australia, or images with exactly the same dimensions, that’s probably no coincidence and we’re dealing with spam.

How instructive is big data?

Collecting and analysing big data is an excellent way to quickly detect large spam waves, in which hundreds of identical (or very similar) emails are suddenly fired off per second. These are new developments that move too fast for the other filtering methods. If the ‘big data’ points to such a virus attack, the spam filter teaches the other methods about it.

Is everything kept?

Of course, by no means all of the data collected is relevant. And old data certainly isn’t; spam that was sent three days ago has long since been recognised and stopped by today (six months ago is ancient history in the spam business!). So for each type of data we set a maximum retention period, after which the spam filter may ‘forget’ that data. For example, it may clear out data about a suspicious sender after two hours, while we hold on to data about IP addresses a little longer.

Spam that is sent at most a few times a day over a period of several weeks won’t be caught by this ‘big data method’. Fortunately, the other methods come to the rescue there. Because if we kept storing all the data, it would become a very expensive business!