Abstract

To date, statistical spam filters are probably the most heavily studied, and most widely adopted technology for detecting junk emails. However, among other disadvantages, they fail to detect spam that cannot be predicated by machine learning algorithms on which they are based. Neither they identify spam that is sent in an image format. In addition, these filters need to be regularly trained, particularly when false positive occurs. Signature-based collaborative spam detection (SCSD) seems to provide a promising solution addressing all these problems. What is in particular attractive is that it can provide a reasonalbe solution to detect unforeseeable new spam, which intuitively appears to be mission impossible. In this talk, I will discuss reesarch issues in SCSD , and report our enhancements to two representative systems, Razor and DCC . One key problem addressed by us is that SCSD approaches usually rely on huge databases of email signatures (i.e., checksums), demanding lots of resource in signature lookup as well as signature database storage, transmission and merging. In our enhancements, signature lookups can be performed in O(1), i.e. constant time, independent of the number of signatures in the database. Space-efficient representation can reduce signature dababase size by a factor of 25.6 or more for Razor-style systems before any data compression algorithm is applied. A simple but efficient algorithm for merging different signature databases is also supported. If time allows, some ongoing work and open problems will also be discussed.