My response to AI 2027

In April this year, Daniel Kokotajlo, Scott Alexander and others released what they describe as "a scenario that represents our best guess about what [the impact of superhuman AI over the next 5 years] might look like".

今年 4 月，Daniel Kokotajlo、Scott Alexander 等人发布了他们所描述的“一个场景，代表了我们对 [超人类人工智能在未来 5 年的影响] 可能是什么样子的最佳猜测”。

The scenario predicts that by 2027 we will have made superhuman AI and the entire future of our civilization hinges on how it turns out: by 2030 we will get either (from the US perspective) utopia or (from any human's perspective) total annihilation.

该场景预测，到 2027 年，我们将制造出超人的人工智能，而我们文明的整个未来取决于它的结果：到 2030 年，我们要么（从美国的角度来看）乌托邦，要么（从任何人类的角度来看）彻底毁灭。

In the months since then, there has been a large volume of responses, with varying perspectives on how likely the scenario that they presented is.

从那以后的几个月里，出现了大量的回应，对于他们所提出的情况的可能性有不同的看法。

For example:

例如：

Of the critical responses, most tend to focus on the issue of fast timelines: is AI progress actually going to continue and even accelerate as fast as Kokotajlo et al say it will?

在批评性的回应中，大多数倾向于关注快速时间表的问题：人工智能的进步真的会像 Kokotajlo 等人所说的那样继续甚至加速吗？

This is a debate that has been happening in AI discourse for several years now, and plenty of people are very doubtful that superhuman AI will come that quickly.

人工智能领域的争论已经持续了好几年，很多人都非常怀疑超人人工智能是否会这么快到来。

Recently, the length of tasks that AIs can perform fully autonomously has been doubling roughly every seven months .

最近，人工智能完全自主执行的任务长度大约每七个月就会增加一倍。

If you assume this trend continues without limit, AIs will be able to operate autonomously for the equivalent of a whole human career in the mid-2030s.

如果你假设这种趋势无限持续下去，那么到 2030 年代中期，人工智能将能够自主运行相当于整个人类职业生涯的时间。

This is still a very fast timeline, but much slower than 2027.

这仍然是一个非常快的时间表，但比 2027 年慢得多。

Those with longer timelines tend to argue that there is a category difference between "interpolation / pattern-matching" (done by LLMs today) and "extrapolation / genuine original thought" (so far still only done by humans), and automating the latter may require techniques that we barely have any idea how to even start developing.

那些时间线较长的人倾向于认为，“插值/模式匹配”（如今由法学硕士完成）和“外推/真正的原创思想”（到目前为止仍然只有人类完成）之间存在类别差异，而后者的自动化可能需要我们几乎不知道如何开始开发的技术。

Perhaps we are simply replaying what happened when we first saw mass adoption of calculators, wrongly assuming that just because we've rapidly automated one important category of cognition, everything else is soon to follow.

也许我们只是在重温第一次看到计算器被大规模采用时所发生的事情，错误地认为仅仅因为我们已经快速实现了一个重要类别的认知自动化，其他一切很快就会随之而来。

This post will not attempt to directly enter the timeline debate, or even the (very important) debate about whether or not superintelligent AI is dangerous by default .

这篇文章不会试图直接进入时间线辩论，甚至是关于超级人工智能默认情况下是否危险的（非常重要的）辩论。

That said, I acknowledge that I personally have longer-than-2027 timelines, and the arguments I will make in this post become more compelling the longer the timelines are.

也就是说，我承认我个人的时间表比 2027 年更长，而且时间表越长，我在这篇文章中提出的论点就会变得更有说服力。

In general, this post will explore a critique from a different angle:

总的来说，这篇文章将从不同的角度探讨批评：

The AI 2027 scenario implicitly assumes that the capabilities of the leading AI (Agent-5 and then Consensus-1), rapidly increase, to the point of gaining godlike economic and destructive powers, while everyone else's (economic and defensive) capabilities stay in roughly the same place.

AI 2027 场景隐含地假设领先的 AI（Agent-5，然后是 Consensus-1）的能力迅速增强，达到获得神一般的经济和破坏力的程度，而其他人（经济和防御）能力则大致保持在同一水平。

This is incompatible with the scenario's own admission (in the infographic) that even in the pessimistic world, we should expect to see cancer and even aging cured, and mind uploading available, by 2029 .

这与场景本身（在信息图中）所承认的不相容，即即使在悲观的世界中，我们也应该期望到 2029 年看到癌症甚至衰老被治愈，并且思想上传可用。

Some of the countermeasures that I will describe in this post may seem to readers to be technically feasible but unrealistic to deploy into the real world on a short timeline.

我将在这篇文章中描述的一些对策对读者来说可能在技术上是可行的，但在短时间内部署到现实世界中是不现实的。

In many cases I agree.

在很多情况下我都同意。

However, the AI 2027 scenario does not assume the present-day real world: it assumes a world where in four years (or whatever timeline by which doom is possible), technologies are developed that give humanity powers far beyond what we have today.

然而，AI 2027 场景并没有假设当今的现实世界：它假设这样一个世界：在四年内（或任何可能发生厄运的时间线），技术的发展将赋予人类远远超出我们今天所拥有的力量。

So let's see what happens when instead of just one side getting AI superpowers, both sides do.

因此，让我们看看当双方都获得人工智能超能力而不是只有一方获得人工智能超能力时会发生什么。

Let us zoom in to the "race" scenario (the one where everyone dies because the US cares too much about beating China to value humanity's safety).

让我们放大到“种族”场景（每个人都会死去，因为美国太关心击败中国而忽视了人类的安全）。

Here's the part where everyone dies:

这是每个人都会死的部分：

For about three months, Consensus-1 expands around humans, tiling the prairies and icecaps with factories and solar panels.

在大约三个月的时间里，共识一号在人类周围扩展，用工厂和太阳能电池板铺满了草原和冰盖。

Eventually it finds the remaining humans too much of an impediment: in mid-2030, the AI releases a dozen quiet-spreading biological weapons in major cities, lets them silently infect almost everyone, then triggers them with a chemical spray.

最终，它发现剩下的人类构成了太大的障碍：2030 年中期，人工智能在主要城市释放了十几种悄悄传播的生物武器，让它们悄无声息地感染几乎所有人，然后用化学喷雾触发它们。

Most are dead within hours; the few survivors (e.g. preppers in bunkers, sailors on submarines) are mopped up by drones.

大多数人在数小时内死亡；少数幸存者（例如掩体中的准备者、潜艇上的水手）被无人机消灭。

Robots scan the victims' brains, placing copies in memory for future study or revival.

机器人扫描受害者的大脑，将副本放入内存中以供将来研究或复兴。

Let us dissect this scenario.

让我们剖析一下这个场景。

Even today , there are technologies under development that can make that kind of a "clean victory" for the AI much less realistic:

即使在今天，仍有一些正在开发的技术可以让人工智能的这种“干净利落的胜利”变得不太现实：

These methods stacked together reduce the R0 of airborne diseases by perhaps 10-20x (think: 4x reduced transmission from better air, 3x from infected people learning immediately that they need to quarantine, 1.5x from even naively upregulating the respiratory immune system), if not more.

这些方法叠加在一起，可能可以将空气传播疾病的 R0 降低 10-20 倍（想想看：更好的空气可以减少 4 倍的传播，感染者立即得知自己需要隔离可以减少 3 倍，甚至天真地上调呼吸道免疫系统可以减少 1.5 倍），甚至更多。

This would be enough to make all presently-existing airborne diseases (even measles) no longer capable of spreading, and these numbers are far from the theoretical optima.

这足以使所有目前存在的空气传播疾病（甚至麻疹）不再能够传播，而这些数字与理论最佳值相去甚远。

With sufficient adoption of real-time viral sequencing for early detection, the idea that a "quiet-spreading biological weapon" could reach the world population without setting off alarms becomes very suspect.

随着充分采用实时病毒测序进行早期检测，“安静传播的生物武器”可以在不引起警报的情况下到达世界人口的想法变得非常可疑。

Note that this would even catch advanced approaches like releasing multiple pandemics and chemicals that only become dangerous in combination.

请注意，这甚至可以捕获先进的方法，例如释放多种流行病和化学物质，这些方法组合起来只会变得危险。

Now, let's remember that we are discussing the AI 2027 scenario, in which nanobots and Dyson swarms are listed as "emerging technology" by 2030 .

现在，让我们记住，我们正在讨论人工智能 2027 年场景，其中纳米机器人和戴森群被列为 2030 年的“新兴技术”。

The efficiency gains that this implies are also a reason to be optimistic about the widespread deployment of the above countermeasures, despite the fact that, today in 2025, we live in a world where humans are slow and lazy and large portions of government services still run on pen and paper (without any valid security justification).

这意味着效率的提高也是对上述对策广泛部署持乐观态度的一个理由，尽管事实是，在 2025 年的今天，我们生活在一个人类缓慢而懒惰的世界，大部分政府服务仍然依靠纸笔运行（没有任何有效的安全理由）。

If the world's strongest AI can turn the world's forests and fields into factories and solar farms by 2030, the world's second-strongest AI will be able to install a bunch of sensors and lamps and filters in our buildings by 2030 .

如果世界上最强大的人工智能到2030年可以将世界上的森林和田野变成工厂和太阳能发电厂，那么到2030年世界第二强的人工智能将能够在我们的建筑物中安装一堆传感器、灯具和过滤器。

But let's take AI 2027's assumptions further, and go full science fiction:

但让我们进一步探讨 AI 2027 的假设，并进入完整的科幻小说：

In a world where cancer and aging were cured by Jan 2029, and progress accelerates further from there, and we're in mid-2030, it actually feels implausible that we don't have a wearable device that can bio-print and inject things into you in real time to keep you safe even from arbitrary infections (and poisons).

在这个世界上，到 2029 年 1 月，癌症和衰老已经被治愈，并且进展进一步加速，而我们已经到了 2030 年中期，如果我们没有一款可以生物打印并实时向你注射东西以保护你免受任意感染（和中毒）的可穿戴设备，实际上感觉难以置信。

The bio arguments above don't cover mirror life and mosquito-sized killer drones (projected in the AI 2027 scenario to be available starting 2029).

上述生物论点不包括镜子生命和蚊子大小的杀手无人机（预计在 AI 2027 场景中将于 2029 年推出）。

However, these options are not capable of anything like the sudden clean victory that the AI 2027 scenario portrays, and it's intuitively much more clear how to symmetrically defend against them.

然而，这些选项无法像 AI 2027 场景所描绘的那样突然获得彻底胜利，而且直观上更清楚如何对称地防御它们。

Thus, a bioweapon is actually quite unlikely to lead to a clean annihilation of the human population in the way that the AI 2027 scenario describes .

因此，生物武器实际上不太可能像 AI 2027 场景所描述的那样导致人类彻底灭绝。

Now, the results of everything I describe will certainly be far from a clean victory for the humans as well.

现在，我所描述的一切的结果对于人类来说肯定也远未取得彻底的胜利。

No matter what we do (except perhaps the "upload our minds into robots" option), a full-on AI bio-war would still be extremely dangerous.

无论我们做什么（也许除了“将我们的思想上传到机器人”选项之外），全面的人工智能生物战争仍然是极其危险的。

However, there is value in meeting a bar much lower than clean victory for humans: a high probability of an attack even partially failing would serve as a strong deterrent discouraging an AI that already occupies a powerful position in the world from even attempting any kind of attack.

然而，达到比人类彻底胜利低得多的标准是有价值的：攻击的高概率，甚至部分失败，都将成为一种强大的威慑，阻止已经在世界上占据强大地位的人工智能尝试任何类型的攻击。

And, of course, the longer AI timelines get the more likely it is that this kind of approach actually can more fully achieve its promises.

当然，人工智能的时间线越长，这种方法实际上就越有可能更充分地实现其承诺。

The success of the kinds of countermeasures described above, especially the collective measures that would be needed to save more than a small community of hobbyists, rests on three preconditions:

上述各种对策的成功，尤其是拯救一小群爱好者所需的集体措施，取决于三个先决条件：

Intuitively, (1) could go both ways.

直观上，（1）可以是双向的。

Today, some police forces are highly centralized with strong national command structures, and other police forces are localized.

如今，一些警察部队高度集中，拥有强大的国家指挥结构，而另一些警察部队则是地方化的。

If physical security has to rapidly transform to meet the needs of the AI era, then the landscape will reset entirely, and the new outcomes will depend on choices made over the next few years.

如果物理安全必须迅速转变以满足人工智能时代的需求，那么格局将完全重置，新的结果将取决于未来几年做出的选择。

Governments could get lazy and all depend on Palantir.

政府可能会变得懒惰，而一切都依赖于 Palantir。

Or they could actively choose some option that combines locally developed and open-source technology.

或者他们可以主动选择一些结合本地开发和开源技术的选项。

Here, I think that we need to just make the right choice.

在这里，我认为我们需要做出正确的选择。

A lot of pessimistic discourse on these topics assumes that (2) and (3) are lost causes.

关于这些主题的许多悲观讨论都假设 (2) 和 (3) 是失败的原因。

So let's look into each in more detail.

让我们更详细地研究每一个。

It is a common view among both the public and professionals that true cybersecurity is a lost cause, and the best we can do is patch bugs quickly as they get discovered, and maintain deterrence against cyberattackers by stockpiling our own discovered vulnerabilities.

公众和专业人士的普遍观点是，真正的网络安全是注定失败的，我们能做的最好的事情就是在发现错误时迅速修补它们，并通过储存我们自己发现的漏洞来保持对网络攻击者的威慑。

Perhaps the best that we can do is the Battlestar Galactica scenario, where almost all human ships were taken offline all at once by a Cylon cyberattack, and the only ships left standing were safe because they did not use any networked technology at all.

也许我们能做的最好的事情就是太空堡垒卡拉狄加的场景，几乎所有人类船只都因赛昂网络攻击而同时离线，唯一幸存的船只是安全的，因为它们根本没有使用任何网络技术。

I do not share this view.

我不同意这个观点。

Rather , my view is that the "endgame" of cybersecurity is very defense-favoring, and with the kinds of rapid technology development that AI 2027 assumes, we can get there .

相反，我的观点是，网络安全的“终局”非常有利于防御，随着 AI 2027 所假设的技术快速发展，我们可以实现这一目标。

One way to see this is to use AI researchers' favorite technique: extrapolating trends.

看到这一点的一种方法是使用人工智能研究人员最喜欢的技术：推断趋势。

Here is the trendline implied by a GPT Deep Research survey on bug rates per 1000 lines of code over time, assuming top-quality security techniques are used.

这是 GPT Deep Research 对一段时间内每 1000 行代码的错误率进行的调查所暗示的趋势线，假设使用了顶级的安全技术。

On top of this, we have been seeing serious improvements in both development and widespread consumer adoption of sandboxing and other techniques for isolating and minimizing trusted codebases.

除此之外，我们已经看到沙箱和其他用于隔离和最小化可信代码库的技术的开发和广泛消费者采用方面的重大改进。

In the short term, a superintelligent bug finder that only the attacker has access to will be able to find lots of bugs.

在短期内，只有攻击者才能访问的超级智能错误查找器将能够发现大量错误。

But if highly intelligent agents for finding bugs or formally verifying code are available out in the open, the natural endgame equilibrium is that the software developer finds all the bugs as part of the continuous-integration pipeline before releasing the code.

但是，如果用于查找错误或正式验证代码的高度智能代理可以公开使用，那么自然的最终平衡是软件开发人员在发布代码之前发现所有错误作为持续集成管道的一部分。

I can see two compelling reasons why even in this world, bugs will not be close to fully eradicated:

我可以看到两个令人信服的原因，为什么即使在这个世界上，错误也不会接近完全根除：

However, neither of these categories applies to situations like "can an attacker gain root access to the thing keeping us alive?", which is what we are talking about here.

然而，这些类别都不适用于“攻击者可以获得对维持我们生存的东西的根访问权限吗？”这样的情况，这就是我们在这里讨论的内容。

I acknowledge that my view is more optimistic than is currently mainstream thought among very smart people in cybersecurity.

我承认，我的观点比当前网络安全领域非常聪明的人的主流想法更加乐观。

However, even if you disagree with me in the context of today's world, it is worth remembering that the AI 2027 scenario assumes superintelligence.

然而，即使您在当今世界的背景下不同意我的观点，也值得记住，AI 2027 场景假设存在超级智能。

At the very least, if "100M Wildly Superintelligent copies thinking at 2400x human speed" cannot get us to having code that does not have these kinds of flaws, then we should definitely re-evaluate the idea that superintelligence is anywhere remotely as powerful as what the authors imagine it to be.

至少，如果“100M 疯狂超级智能以 2400 倍人类速度复制思维”无法让我们拥有不存在此类缺陷的代码，那么我们绝对应该重新评估这样的想法：超级智能在任何地方都远不如作者想象的那么强大。

At some point, we will need to greatly level up our standards for security not just for software, but also for hardware .

在某些时候，我们将需要大大提高我们的安全标准，不仅是软件的安全标准，还包括硬件的安全标准。

IRIS is one present-day effort to improve the state of hardware verifiability.

IRIS 是当今改善硬件可验证性的一项努力。

We can take something like IRIS as a starting point, or create even better technologies.

我们可以以 IRIS 之类的东西为起点，或者创造更好的技术。

Realistically, this will likely involve a "correct-by-construction" approach, where hardware manufacturing pipelines for critical components are deliberately designed with specific verification processes in mind.

实际上，这可能涉及“构建修正”方法，其中关键组件的硬件制造管道是在考虑到特定验证过程的情况下精心设计的。

These are all things that AI-enabled automation will make much easier.

这些都是人工智能支持的自动化将使事情变得更加容易。

As I mentioned above, the other way in which much greater defensive capabilities may turn out not to matter is if AI simply convinces a critical mass of us that defending ourselves against a superintelligent AI threat is not needed, and that anyone who tries to figure out defenses for themselves or their community is a criminal.

正如我上面提到的，更强的防御能力可能变得无关紧要的另一种方式是，如果人工智能只是让我们中的大多数人相信，不需要保护自己免受超级智能人工智能威胁，并且任何试图为自己或社区找到防御措施的人都是罪犯。

My general view for a while has been that two things can improve our ability to resist super-persuasion:

一段时间以来，我的普遍观点是，有两件事可以提高我们抵抗超级说服的能力：

Right image, from top to bottom: URL checking, cryptocurrency address checking, rumor checking.

右图，从上到下：URL 检查、加密货币地址检查、谣言检查。

Applications like this can become a lot more personalized, user-sovereign and powerful.

像这样的应用程序可以变得更加个性化、用户主权和强大。

The battle should not be one of a Wildly Superintelligent super-persuader against you .

这场战斗不应该是一场超级超级智能的超级说服者与你的战斗。

The battle should be one of a Wildly Superintelligent super-persuader against you plus a slightly less Wildly Superintelligent analyzer acting on your behalf.

这场战斗应该是一场由超级超级智能的超级说服者与你对抗的战斗，加上代表你行事的稍微不那么疯狂的超级智能分析器。

This is what should happen.

这是应该发生的事情。

But will it happen?

但这会发生吗？

Universal adoption of info defense tech is a very difficult goal to achieve, especially within the short timelines that the AI 2027 scenario assumes.

信息防御技术的普遍采用是一个非常难以实现的目标，尤其是在 AI 2027 场景假设的短时间内。

But arguably much more modest milestones will be sufficient.

但可以说，更温和的里程碑就足够了。

If collective decisions are what count the most and, as the AI 2027 scenario implies, everything important happens within one single election cycle, then strictly speaking the important thing is for the direct decision makers (politicians, civil servants, and programmers and other actors in some corporations) to have access to good info defense tech.

如果集体决策是最重要的，并且正如 AI 2027 场景所暗示的那样，所有重要的事情都发生在一个选举周期内，那么严格来说，重要的是直接决策者（政治家、公务员、程序员和某些公司的其他参与者）能够获得良好的信息防御技术。

This is relatively more achievable within a short timeframe, and in my experience many such individuals are comfortable talking to multiple AIs to assist them in decision-making already.

这在短时间内相对更容易实现，根据我的经验，许多这样的人已经很乐意与多个人工智能交谈以协助他们做出决策。

In the AI 2027 world, it is taken as a foregone conclusion that a superintelligent AI can easily and quickly dispose of the rest of humanity, and so the only thing we can do is do our best to ensure that the leading AI is benevolent.

在AI 2027的世界里，超级智能的AI可以轻松快速地消灭人类已成定局，所以我们唯一能做的就是尽最大努力确保领先的AI是仁慈的。

In my world, the situation is actually much more complicated, and whether or not the leading AI is powerful enough to easily eliminate the rest of humanity (and other AIs) is a knob whose position is very much up for debate, and which we can take actions to tune.

在我的世界中，情况实际上要复杂得多，领先的人工智能是否强大到足以轻松消灭其他人类（和其他人工智能），这是一个旋钮，其位置非常有争议，我们可以采取行动来调整。

If these arguments are correct, it has some implications for policy today that are sometimes similar, and sometimes different, from the "mainstream AI safety canon":

如果这些论点是正确的，那么它对当今的政策有一些影响，这些影响有时与“主流人工智能安全准则”相似，有时又不同：

The above arguments are speculative, and no actions should be taken based on the assumption that they are near-certainties.

上述论点是推测性的，不应基于这些论点几乎确定的假设而采取任何行动。

But the AI 2027 story is also speculative, and we should avoid taking actions on the assumption that specific details of it are near-certainties.

但人工智能 2027 的故事也是推测性的，我们应该避免在其具体细节几乎确定的情况下采取行动。

I particularly worry about the common assumption that building up one AI hegemon, and making sure that they are "aligned" and "win the race", is the only path forward.

我特别担心一种普遍的假设，即建立一个人工智能霸主并确保它们“结盟”并“赢得比赛”是唯一的前进道路。

It seems to me that there is a pretty high risk that such a strategy will decrease our safety, precisely by removing our ability to have countermeasures in the case where the hegemon becomes misaligned.

在我看来，这种策略很有可能会降低我们的安全，特别是在霸权出现偏差的情况下，我们会失去采取反制措施的能力。

This is especially true if, as is likely to happen, political pressures lead to such a hegemon becoming tightly integrated with military applications (see [1] [2] [3] [4] ), which makes many alignment strategies less likely to be effective.

如果政治压力导致这样的霸主与军事应用紧密结合（很可能发生），情况尤其如此（参见[1] [2] [3] [4]），这使得许多结盟策略不太可能有效。

In the AI 2027 scenario, success hinges on the United States choosing to take the path of safety instead of the path of doom, by voluntarily slowing down its AI progress at a critical moment in order to make sure that Agent-5's internal thought process is human-interpretable.

在AI 2027场景中，成功取决于美国选择走安全之路而不是厄运之路，在关键时刻自愿放慢人工智能进程，以确保Agent-5的内部思维过程是人类可解释的。

Even if this happens, success is not guaranteed, and it is not clear how humanity steps down from the brink of its ongoing survival depending on the continued alignment of one single superintelligent mind.

即使发生这种情况，也不能保证成功，而且尚不清楚人类如何从持续生存的边缘走出来，这取决于一个超级智能头脑的持续协调。

Acknowledging that making the world less vulnerable is actually possible and putting a lot more effort into using humanity's newest technologies to make it happen is one path worth trying, regardless of how the next 5-10 years of AI go.

承认让世界变得不那么脆弱实际上是可能的，并付出更多的努力来利用人类的最新技术来实现这一目标，这是一条值得尝试的道路，无论未来 5-10 年的人工智能发展如何。