Comparing dbacl with other classifiers
The key to comparing dbacl(1) with other
classifiers is the mailcross(1) testsuite command.
Simply put, this command allows you to compare the error rates of several
classifiers on a common set of training documents. The rates you obtain
are of course only estimates, and likely to vary somewhat depending on the
actual sample emails you use. Thus it is possible for one classifier to
perform better than another with one set of documents, while performing worse
with a different set.
Unfortunately, there does not exist a truly representative set of email documents
for everyone on the planet. Moreover, one person's email characteristics vary slowly over time. Consequently, it makes little sense to compare the
performance of different classifiers on different sets of documents. Instead,
the task of choosing the best classifier for yourself can only be done reliably by referencing your own email, and by comparing classifiers on exactly the same emails.
The mailcross(1) testsuite must be given a set
of categories, with sample emails from each category in mbox format.
After selecting all the classifiers to be compared, it remains only to leave
the script running over night. The summary is usually inspected the next morning.
The method used to estimate classification errors is a standard cross validation.
The training emails are split into a number of roughly equal sized subsets, all of which, except for one, are used for learning. The remaining subset, which wasn't learned, is predicted. Finally, the percentage of errors is calculated for each category by averaging results over all possible choices of the prediction subset.
Note that this is neither the only way to estimate prediction errors, nor
even accepted as a good way by all academics. However, it's independent of
the classifier, widely used around the world, and easy to program.
You can cross validate as many categories as you like, provided the classifiers all support multiple categories. For example, you could compare dbacl(1) and ifile on many categories.
However, most email junk filters can only cope with two categories, representing junk mail and regular mail. When comparing the performance of these classifiers, such as bogofilter for example, the mailcross(1) testsuite is hard coded to function with two categories named spam and notspam. You must use these category names, or the results will not make sense.