Better Bayesian Filtering

January 2003 (This article was given as a talk at the 2003 Spam Conference.

2003 年 1 月（本文是在 2003 年垃圾邮件会议上作为演讲发表的。

It describes the work I've done to improve the performance of the algorithm described in A Plan for Spam , and what I plan to do in the future.) The first discovery I'd like to present here is an algorithm for lazy evaluation of research papers.

它描述了我为提高垃圾邮件计划中描述的算法的性能所做的工作，以及我将来计划做的事情。）我想在这里展示的第一个发现是一种用于研究论文惰性评估的算法。

Just write whatever you want and don't cite any previous work, and indignant readers will send you references to all the papers you

只要写任何你想写的东西，不要引用任何以前的作品，愤怒的读者就会向你发送你所写的所有论文的参考文献。

should have cited.

应该引用的。

I discovered this algorithm after ``A Plan for Spam'' [1] was on Slashdot.

我在 Slashdot 上发表《垃圾邮件计划》[1] 后发现了这个算法。

Spam filtering is a subset of text classification, which is a well established field, but the first papers about Bayesian spam filtering per se seem to have been two given at the same conference in 1998, one by Pantel and Lin [2], and another by a group from Microsoft Research [3].

垃圾邮件过滤是文本分类的一个子集，这是一个成熟的领域，但关于贝叶斯垃圾邮件过滤本身的第一篇论文似乎是在 1998 年的同一会议上发表的两篇论文，一篇由 Pantel 和 Lin [2] 发表，另一篇由 Microsoft Research 的一个小组发表 [3]。

When I heard about this work I was a bit surprised.

当我听说这项工作时，我有点惊讶。

If

如果

people had been onto Bayesian filtering four years ago, why wasn't everyone using it?

人们四年前就开始使用贝叶斯过滤，为什么不是每个人都使用它？

When I read the papers I found out why.

当我阅读报纸时，我发现了原因。

Pantel and Lin's filter was the more effective of the two, but it only caught 92% of spam, with 1.16% false positives.

Pantel 和 Lin 的过滤器是两者中更有效的，但它只捕获了 92% 的垃圾邮件，误报率为 1.16%。

When I tried writing a Bayesian spam filter, it caught 99.5% of spam with less than .03% false positives [4].

当我尝试编写贝叶斯垃圾邮件过滤器时，它捕获了 99.5% 的垃圾邮件，误报率低于 0.03% [4]。

It's always alarming when two people trying the same experiment get widely divergent results.

当两个人尝试同一个实验却得到截然不同的结果时，总是令人震惊。

It's especially alarming here because those two sets of numbers might yield opposite conclusions.

这里尤其令人担忧，因为这两组数字可能会得出相反的结论。

Different users have different requirements, but I think for many people a filtering rate of 92% with 1.16% false positives means that filtering is not an acceptable solution, whereas 99.5% with less than .03% false positives means that it is.

不同的用户有不同的要求，但我认为对于很多人来说，过滤率为 92% 且误报率为 1.16% 意味着过滤不是可接受的解决方案，而 99.5% 且误报率低于 0.03% 则意味着过滤是可接受的解决方案。

So why did we get such different numbers?

那么为什么我们得到如此不同的数字呢？

I haven't tried to reproduce Pantel and Lin's results, but

我没有尝试重现潘特尔和林的结果，但是

from reading the paper I see five things that probably account for the difference.

通过阅读这篇论文，我发现五件事可能是造成这种差异的原因。

One is simply that they trained their filter on very little data: 160 spam and 466 nonspam mails.

其中之一很简单，他们只用很少的数据来训练过滤器：160 封垃圾邮件和 466 封非垃圾邮件。

Filter performance should still be climbing with data sets that small.

对于这么小的数据集，过滤器性能应该仍然会上升。

So their numbers may not even be an accurate measure of the performance of their algorithm, let alone of Bayesian spam filtering in general.

因此，他们的数字甚至可能无法准确衡量其算法的性能，更不用说一般的贝叶斯垃圾邮件过滤了。

But I think the most important difference is probably

但我认为最重要的区别可能是

that they ignored message headers.

他们忽略了邮件标题。

To anyone who has worked on spam filters, this will seem a perverse decision.

对于任何从事垃圾邮件过滤器工作的人来说，这似乎是一个不正当的决定。

And yet in the very first filters I tried writing, I ignored the headers too.

然而，在我尝试编写的第一个过滤器中，我也忽略了标题。

Why?

为什么？

Because I wanted to keep the problem neat.

因为我想让问题保持整洁。

I didn't know much about mail headers then, and they seemed to me full of random stuff.

那时我对邮件标题了解不多，在我看来它们充满了随机的东西。

There is a lesson here for filter writers: don't ignore data.

这里给过滤器编写者一个教训：不要忽略数据。

You'd think this lesson would

你可能认为这节课会

be too obvious to mention, but I've had to learn it several times.

太明显了，无法提及，但我不得不学习好几次。

Third, Pantel and Lin stemmed the tokens, meaning they reduced e.g. both ``mailing'' and ``mailed'' to the root ``mail''.

第三，Pantel 和 Lin 阻止了代币，这意味着他们减少了例如“mailing”和“mailed”都发送到根“mail”。

They may have felt they were forced to do this by the small size of their corpus, but if so this is a kind of premature optimization.

他们可能觉得由于语料库规模太小而被迫这样做，但如果是这样的话，这就是一种过早的优化。

Fourth, they calculated probabilities differently.

第四，他们计算概率的方式不同。

They used all the tokens, whereas I only

他们用了所有的代币，而我只用了

use the 15 most significant.

使用 15 个最重要的。

If you use all the tokens you'll tend to miss longer spams, the type where someone tells you their life story up to the point where they got rich from some multilevel marketing scheme.

如果您使用所有代币，您可能会错过较长的垃圾邮件，即有人向您讲述他们的生活故事，直到他们通过某种多层次营销计划致富的程度。

And such an algorithm would be easy for spammers to spoof: just add a big chunk of random text to counterbalance the spam terms.

这样的算法很容易被垃圾邮件发送者欺骗：只需添加一大块随机文本即可抵消垃圾邮件术语。

Finally, they didn't bias against false positives.

最后，他们并没有对误报产生偏见。

I think

我认为

any spam filtering algorithm ought to have a convenient knob you can twist to decrease the false positive rate at the expense of the filtering rate.

任何垃圾邮件过滤算法都应该有一个方便的旋钮，您可以旋转它来降低误报率，但会牺牲过滤率。

I do this by counting the occurrences of tokens in the nonspam corpus double.

我通过计算非垃圾邮件语料库中标记的出现次数来做到这一点。

I don't think it's a good idea to treat spam filtering as a straight text classification problem.

我认为将垃圾邮件过滤视为直接的文本分类问题不是一个好主意。

You can use text classification techniques, but solutions can and should reflect the fact that the text is email, and spam

您可以使用文本分类技术，但解决方案可以而且应该反映文本是电子邮件和垃圾邮件的事实

in particular.

特别是。

Email is not just text; it has structure.

电子邮件不仅仅是文本；它有结构。

Spam filtering is not just classification, because false positives are so much worse than false negatives that you should treat them as a different kind of error.

垃圾邮件过滤不仅仅是分类，因为误报比漏报严重得多，因此您应该将它们视为不同类型的错误。

And the source of error is not just random variation, but a live human spammer working actively to defeat your filter.

错误的来源不仅仅是随机变化，而是活生生的垃圾邮件发送者积极地工作以击败您的过滤器。

Tokens Another project I heard about after the Slashdot article was Bill Yerazunis'

代币 在 Slashdot 文章之后我听说的另一个项目是 Bill Yerazunis 的项目

CRM114 [5].

CRM114 [5]。

This is the counterexample to the design principle I just mentioned.

这就是我刚才提到的设计原则的反例。

It's a straight text classifier, but such a stunningly effective one that it manages to filter spam almost perfectly without even knowing that's what it's doing.

它是一个直接的文本分类器，但却非常有效，以至于它几乎可以完美地过滤垃圾邮件，甚至在不知道它正在做什么的情况下。

Once I understood how CRM114 worked, it seemed inevitable that I would eventually have to move from filtering based on single words to an approach like this.

一旦我了解了 CRM114 的工作原理，我似乎不可避免地最终不得不从基于单个单词的过滤转变为这样的方法。

But first, I thought,

但首先，我想，

I'll see how far I can get with single words.

我会看看我能用一个词走多远。

And the answer is, surprisingly far.

答案是令人惊讶的遥远。

Mostly I've been working on smarter tokenization.

我主要致力于更智能的标记化。

On current spam, I've been able to achieve filtering rates that approach CRM114's.

对于当前的垃圾邮件，我已经能够达到接近 CRM114 的过滤率。

These techniques are mostly orthogonal to Bill's; an optimal solution might incorporate both.

这些技术大多与比尔的技术正交。最佳解决方案可能会将两者结合起来。

``A Plan for Spam'' uses a very simple definition of a token.

“垃圾邮件计划”使用了一个非常简单的标记定义。

Letters, digits, dashes, apostrophes,

字母、数字、破折号、撇号、

and dollar signs are constituent characters, and everything else is a token separator.

和美元符号是组成字符，其他所有内容都是标记分隔符。

I also ignored case.

我也忽略了大小写。

Now I have a more complicated definition of a token: Case is preserved.

现在我对令牌有一个更复杂的定义：保留大小写。

Exclamation points are constituent characters.

感叹号是组成字符。

Periods and commas are constituents if they occur between two digits.

如果句号和逗号出现在两个数字之间，则它们是组成部分。

This lets me get ip addresses and prices intact.

这让我可以完整地获取 IP 地址和价格。

A price range like $20-25 yields two tokens, $20 and $25.

像 20-25 美元这样的价格范围会产生两种代币：20 美元和 25 美元。

Tokens that occur within the To, From, Subject, and Return-Path lines, or within urls, get marked accordingly.

出现在“收件人”、“发件人”、“主题”和“返回路径”行中或 url 中的标记会得到相应的标记。

E.g. ``foo'' in the Subject line becomes ``Subject*foo''.

例如。主题行中的“foo”变为“Subject*foo”。

(The asterisk could be any character you don't allow as a constituent.) Such measures increase the filter's vocabulary, which makes it more discriminating.

（星号可以是您不允许作为组成部分的任何字符。）此类措施会增加过滤器的词汇量，从而使其更具辨别力。

For example, in the current filter, ``free'' in the Subject line has a spam probability of 98%, whereas the same token

例如，在当前过滤器中，主题行中的“free”的垃圾邮件概率为 98%，而相同的标记

in the body has a spam probability of only 65%.

体内存在垃圾邮件的概率只有65%。

Here are some of the current probabilities [6]: Subject*FREE 0.9999 free!!

以下是当前的一些概率 [6]： 主题*免费 0.9999 免费！

0.9999 To*free 0.9998 Subject*free 0.9782 free!

0.9999 目标*免费 0.9998 主题*免费 0.9782 免费！

0.9199 Free 0.9198 Url*free 0.9091 FREE 0.8747 From*free 0.7636 free 0.6546

0.9199 免费 0.9198 网址*免费 0.9091 免费 0.8747 来自*免费 0.7636 免费 0.6546