Putting spam in the slow lane

HP Labs researchers have developed an experimental system that can reduce delays to legitimate e-mail caused by an ever-mounting volume of spam and junk messages.

If you have ever waited, frustrated, for an important e-mail message while endless spam and junk pour into your inbox, then HP Labs researchers could have the answer.

It’s an experimental system that can reduce delays to legitimate e-mail – delays caused by an ever-mounting volume of spam and junk messages – by giving good mail priority over bad.

A prototype of the system is now running in a live trial on one of the mail servers at HP Laboratories in Bristol, UK, where it was developed.

The system is deployed upstream of conventional mail-server spam filters. It rapidly separates out messages from suspicious IP addresses to reduce backlogs during spam surges and then places those messages in a different queue.

The system could be used to complement current spam-filtering technology, speeding the delivery of legitimate e-mail messages by sending good e-mail through the filters and into the network first. Suspected spam would go into a lower-priority queue.

“It puts junk e-mail into a slow lane before it reaches the anti-spam filters, allowing the good mail to go into the filtering process without delay,” says Miranda Mowbray, one of the researchers who developed the technology.

As anyone with an e-mail address knows, the quantity of junk mail is soaring. Some 64 percent of all e-mail is spam – and this figure is expected to rise, according to Brightmail, a maker of anti-spam tools. The increasing number of spam surges and virus attacks tie up mail servers and can delay e-mail delivery for hours, including messages that could be of critical business importance.

Part of the problem is that, upon arrival at the e-mail gateway, incoming messages are put in a queue so that the content can be scanned for viruses and spam before it is passed on to the recipient.

During spam surges, the scanning process can create a performance bottleneck in the system and legitimate mail is delayed waiting for junk mail to be filtered out.

The HP Labs team has devised an approach that gives delivery priority to mail it categorises as good over mail it tags as junk.

<b>Separating wheat from chaff</b>

The HP Labs prioritisation technology classifies a server as good if fewer than half the e-mails received from that IP address were junk – spam, virus or undeliverable. The classification is based on statistics from SpamAssassin, an anti-spam application, Sophos, the anti-virus scanner, plus the results of the first delivery attempts.

This classification can be used to speed up the delivery of non-junk e-mail by creating two queues. The queue of e-mail from good servers is given priority access to the existing content scanner. This ensures that there are virtually no delays to e-mail in this queue, even if the server is heavily loaded with junk mail.

During a recent virus attack, delays of 2.7 hours were experienced to all mail processed by a large commercial mail server. If the prioritisation technology had been in operation, the delay would have been only 22 seconds for the good queue, but four hours for the junk queue. The results come from performance data from real mail that was passed through the experimental system.

At the moment the system predicts good messages with 74 percent accuracy and junk messages with 95 percent accuracy. This is sufficient to select suspect messages to be delayed in favour of legitimate e-mails before they are scanned for content.

“The power of this classification method is speed. We know almost immediately if a message should be tagged as good mail or junk,” says researcher Dan Twining. “This immediacy, along with the system’s relative accuracy and its lightweight implementation, allows us to optimise all other anti-spam techniques.”

Researchers ran a three-month live trial of the technology using the sendmail mail transfer agent in a large-scale e-mail system. The team has now developed a Postfix-based version of its prioritisation technology. The Postfix mail transfer agent is more commonly used by large companies and enterprises.

On the web