A Plan for Spam
<img src="http://www.virtumundo.com/images/spacer.gif" height=15 width=1> Like to build things?
<img src="http://www.virtumundo.com/images/spacer.gif" height=15 width=1> 喜欢建造东西吗？
Try Hacker News .
尝试一下黑客新闻。
</td > August 2002 (This article describes the spam-filtering techniques used in the spamproof web-based mail reader we built to exercise Arc .
</td> 2002 年 8 月（本文介绍了我们为运用 Arc 而构建的基于 Web 的防垃圾邮件阅读器中使用的垃圾邮件过滤技术。
An improved algorithm is described in Better Bayesian Filtering .) I think it's possible to stop spam, and that content-based filters are the way to do it.
更好的贝叶斯过滤中描述了一种改进的算法。）我认为阻止垃圾邮件是可能的，并且基于内容的过滤器就是实现这一点的方法。
The Achilles heel of the spammers is their message.
垃圾邮件发送者的致命弱点是他们的信息。
They can circumvent any other barrier you set up.
他们可以绕过您设置的任何其他障碍。
They have so far, at least.
至少到目前为止他们是这样的。
But they have to deliver their message, whatever it is.
但无论是什么，他们都必须传达自己的信息。
If we can write software that recognizes their messages, there is no way they can get around that.
如果我们可以编写能够识别他们的消息的软件，他们就无法绕过这个问题。
_ _ _ To the recipient, spam is easily recognizable.
_ _ _ 对于收件人来说，垃圾邮件很容易识别。
If you hired someone to read your mail and discard the spam, they would
如果您雇用某人来阅读您的邮件并丢弃垃圾邮件，他们会
have little trouble doing it.
做起来没有什么困难。
How much do we have to do, short of AI, to automate this process?
如果没有人工智能，我们还需要做多少工作来实现这一过程的自动化？
I think we will be able to solve the problem with fairly simple algorithms.
我认为我们将能够用相当简单的算法来解决这个问题。
In fact, I've found that you can filter present-day spam acceptably well using nothing more than a Bayesian combination of the spam probabilities of individual words.
事实上，我发现仅使用单个单词的垃圾邮件概率的贝叶斯组合就可以很好地过滤当今的垃圾邮件。
Using a slightly tweaked (as described below) Bayesian filter, we now miss less than 5 per 1000 spams, with 0 false positives.
使用稍微调整过的（如下所述）贝叶斯过滤器，我们现在每 1000 封垃圾邮件中漏掉的垃圾邮件少于 5 封，误报率为 0。
The statistical approach is not usually the first one people try when they write spam filters.
统计方法通常不是人们在编写垃圾邮件过滤器时首先尝试的方法。
Most hackers' first instinct is to try to write software that recognizes individual properties of spam.
大多数黑客的第一反应是尝试编写能够识别垃圾邮件个别属性的软件。
You look at spams and you think, the gall of these guys to try sending me mail that begins "Dear Friend" or has a subject line that's all uppercase and ends in eight exclamation points.
当您查看垃圾邮件时，您会想，这些家伙竟然试图向我发送以“亲爱的朋友”开头或主题行全部大写并以八个感叹号结尾的邮件，真是太厚颜无耻了。
I can filter out that stuff with about one line of code.
我可以用大约一行代码过滤掉这些东西。
And so you do, and in the beginning it works.
你就这么做了，一开始它是有效的。
A few simple rules will take a big bite out of your incoming spam.
一些简单的规则将大大减少您收到的垃圾邮件。
Merely looking for the word "click" will catch 79.7% of the emails in my spam corpus, with only 1.2% false positives.
仅查找“点击”一词即可捕获我的垃圾邮件语料库中 79.7% 的电子邮件，误报率仅为 1.2%。
I spent about six months writing software that looked for individual spam features before I tried the statistical approach.
在尝试统计方法之前，我花了大约六个月的时间编写软件来查找各个垃圾邮件的特征。
What I found was that recognizing that last few
我发现认识到最后几个
percent of spams got very hard, and that as I made the filters stricter I got more false positives.
垃圾邮件的百分比变得非常困难，并且当我将过滤器设置得更严格时，我得到了更多的误报。
False positives are innocent emails that get mistakenly identified as spams.
误报是指被错误地识别为垃圾邮件的无辜电子邮件。
For most users, missing legitimate email is an order of magnitude worse than receiving spam, so a filter that yields false positives is like an acne cure that carries a risk of death to the patient.
对于大多数用户来说，丢失合法电子邮件比收到垃圾邮件要严重一个数量级，因此产生误报的过滤器就像治疗痤疮一样，会给患者带来死亡风险。
The more spam a user gets, the less
用户收到的垃圾邮件越多，垃圾邮件就越少
likely he'll be to notice one innocent mail sitting in his spam folder.
他很可能会注意到他的垃圾邮件文件夹里有一封无辜的邮件。
And strangely enough, the better your spam filters get, the more dangerous false positives become, because when the filters are really good, users will be more likely to ignore everything they catch.
奇怪的是，你的垃圾邮件过滤器越好，误报就越危险，因为当过滤器真的很好时，用户将更有可能忽略他们捕获的所有内容。
I don't know why I avoided trying the statistical approach for so long.
我不知道为什么我这么长时间都避免尝试统计方法。
I think it was because I got addicted to trying to identify spam features myself, as if I were playing
我认为这是因为我沉迷于尝试自己识别垃圾邮件功能，就像我在玩游戏一样
some kind of competitive game with the spammers.
与垃圾邮件发送者进行某种竞争性游戏。
(Nonhackers don't often realize this, but most hackers are very competitive.) When I did try statistical analysis, I found immediately that it was much cleverer than I had been.
（非黑客通常不会意识到这一点，但大多数黑客都非常有竞争力。）当我尝试统计分析时，我立即发现它比我以前聪明得多。
It discovered, of course, that terms like "virtumundo" and "teens" were good indicators of spam.
当然，它发现“virtumundo”和“teens”等术语是垃圾邮件的良好指标。
But it also discovered that "per" and "FL" and "ff0000" are good indicators of spam.
但它还发现“per”、“FL”和“ff0000”是垃圾邮件的良好指标。
In fact, "ff0000" (html for bright red)
事实上，“ff0000”（html 表示亮红色）
turns out to be as good an indicator of spam as any pornographic term.
事实证明，它与任何色情术语一样是垃圾邮件的良好指标。
_ _ _ Here's a sketch of how I do statistical filtering.
_ _ _ 这是我如何进行统计过滤的草图。
I start with one corpus of spam and one of nonspam mail.
我从一组垃圾邮件和一组非垃圾邮件开始。
At the moment each one has about 4000 messages in it.
目前，每条消息中大约有 4000 条消息。
I scan the entire text, including headers and embedded html and javascript, of each message in each corpus.
我扫描每个语料库中每条消息的整个文本，包括标题以及嵌入的 html 和 javascript。
I currently consider alphanumeric characters, dashes, apostrophes, and dollar signs to be part of tokens,
我目前认为字母数字字符、破折号、撇号和美元符号是标记的一部分，
and everything else to be a token separator.
以及其他所有内容作为令牌分隔符。
(There is probably room for improvement here.) I ignore tokens that are all digits, and I also ignore html comments, not even considering them as token separators.
（这里可能还有改进的空间。）我忽略了全是数字的标记，我也忽略了 html 注释，甚至不将它们视为标记分隔符。
I count the number of times each token (ignoring case, currently) occurs in each corpus.
我计算每个标记（当前忽略大小写）在每个语料库中出现的次数。
At this stage I end up with two large hash tables, one for each corpus, mapping tokens to number of occurrences.
在这一阶段，我最终得到两个大型哈希表，每个语料库一个，将标记映射到出现次数。
Next I create a third hash table, this time mapping each token to the probability that an email containing it is a spam, which I calculate as follows [1]: (let ((g (* 2 (or (gethash word good) 0))) (b (or (gethash word bad) 0))) (unless (< (+ g b) 5) (max .01 (min .99 (float (/ (min 1 (/ b nbad)) (+ (min 1 (/ g ngood)) (min 1 (/ b nbad)))))))))
接下来，我创建第三个哈希表，这次将每个标记映射到包含它的电子邮件是垃圾邮件的概率，我计算如下 [1]： (let ((g (* 2 (or (gethash word good) 0))) (b (or (gethash word bad) 0))) (unless (< (+ g b) 5) (max .01 (min .99 (float (/ (min 1 (/ b nbad)))) (+ (最小值 1 (/ g n好)) (最小值 1 (/ b n坏)))))))))