SMTP Traffic Filtering

Cloud based SMTP filtering for spam and virus signatures detection

Comendo came to us looking to improve the reliability, efficiency and accuracy of their spam filter integrated with the MTA (Mail Transfer Agent) from MessageSystems. The original filter did not have good test coverage and did not meet performance requirements. We also created an analytics toolkit to gather and disect relevant data.

During our long-term work relations we also developed a virus filtering service that allowed to identify new virus signatures in spam traffic.

When

2013

Our Role

Development,
DevOps

Client

Comendo, J2 Global

Tech

C++, Boost, Golang, Redis, CentOS, Puppet

STMP spam filtering

The client offers a comprehensive email service to their corporate and govt customers serving over 10k domains. The solution is based on the MTA from MessageSystems — Ecelerity.

To counter spam, the client uses its own anti-spam daemon which scans an SMTP flow reaching a processing rate of over 50 messages per second. The original daemon implementation didn’t meet performance requirements.
 
We took over the codebase and created a scalable, semi-self learning spam filter compatible with any MTA. New content based filtering algorithms have been added.

The Technology Stack

C++

In the MTA, filter
client & daemon

Boost Libraries

To accelerate development speed & optimise computation

MySQL

In the central database to store email signatures

Golang

Data handling, analytics and self-learning features

Perl

Communication between the database and 3rd party software

Integration with MTA

In later releases, we made improvements to scalability, architecture, the central database and compatibility with 3rd party software. We added several sophisticated spam-filtering algorithms, an advanced admin toolset and detailed statistics. We also improved compatibility with other daemons such as a virus filter.

Today the spam filter has a processing rate of over 50 messages per second and is fully scalable. The system can update itself dynamically without interrupting the flow of emails.

A successful foundation for future improvement

We delivered 4 key outcomes:

Performance

The system can handle high peaks with a wider stack of content based algorithms. Its CPU and memory consumption has been reduced

Maintanability

The newly created daemon has been test-driven developed which allowed to introduce changes to the filtering algorithms quicker

Self-learning

The system learns from earlier spam-attacks, with a growing central signature database.

Integration

Puppet was introduced to ease off the deployment process

Work with us

We’d love to discuss your project

Send me a Non-Disclosure Agreement (NDA)
Back
Next
Next
Thanks, we’ll be in touch soon
Oops! Something went wrong while submitting the form