System.Xml.XmlElement
We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research.
我们谈论多智能体、纳维-斯托克斯，以及当前数学进步的爆炸式增长告诉我们一旦人工智能研究自动化会发生什么。
And we also discuss how we will know if the models are actually aligned before we kick off RSI.
我们还讨论了在启动 RSI 之前如何知道模型是否真正一致。
Jane Street has been interested in AI for a lot longer than you’d think, and not just for trading.
Jane Street 对人工智能感兴趣的时间比你想象的要长得多，而且不仅仅是为了交易。
In 2011, a full year before AlexNet and over a decade before ChatGPT launched, they hosted the first FOOM Debate between Eliezer Yudkowsky and Robin Hanson on whether AI would lead to an intelligence explosion.
2011 年，也就是 AlexNet 推出的整整一年和 ChatGPT 推出的十多年前，他们举办了 Eliezer Yudkowsky 和 Robin Hanson 之间的首次 FOOM 辩论，主题是人工智能是否会导致智能爆炸。
Now Jane Street is revisiting the question with a new panel: Daniel Kokotajlo, Ege Erdil, Ryan Greenblatt, and Jaime Sevilla, hosted by Ron Minsky in San Francisco this October.
现在，Jane Street 正在通过一个新的小组重新审视这个问题：Daniel Kokotajlo、Ege Erdil、Ryan Greenblatt 和 Jaime Sevilla 今年 10 月在旧金山由 Ron Minsky 主持。
I expect it to be a truly excellent conversation.
我希望这将是一次真正精彩的对话。
Register at janestreet.com/dwarkesh
在 janestreet.com/dwarkesh 注册
Grok Bot has made handing off work super easy.
Grok Bot 让工作交接变得超级简单。
It runs on its own cloud computer, where it installs the tools it needs to handle tasks end-to-end.
它在自己的云计算机上运行，并在其中安装端到端处理任务所需的工具。
For the podcast, we use Grok Bot to help produce our videos.
对于播客，我们使用 Grok Bot 来帮助制作视频。
You may have noticed that our ads feature animations of real websites.
您可能已经注意到，我们的广告以真实网站的动画为特色。
Getting these pixel-perfect used to mean running a convoluted, multi-step workflow ourselves.
过去，要获得这些像素完美，就意味着我们要自己运行一个复杂的、多步骤的工作流程。
Now we just let Grok Bot handle it.
现在我们让 Grok Bot 来处理它。
Best of all, Grok Bot has learned all of our specs and preferences, so we don’t have to redescribe the task each time!
最重要的是，Grok Bot 已经了解了我们所有的规格和偏好，因此我们不必每次都重新描述任务！
Try Grok Bot for yourself at x.ai/bot
在 x.ai/bot 上亲自尝试 Grok Bot
Antithesis gives you the confidence of a giant test suite without actually having to write one.
Antithesis 让您对庞大的测试套件充满信心，而无需实际编写测试套件。
Say you’re doing a major backend refactor: building enough tests to trust it could take weeks.
假设您正在进行一项重大的后端重构：构建足够的测试来信任它可能需要数周时间。
Antithesis solves this by running your software through countless simulated worlds, injecting faults and hunting for failures.
Antithesis 通过在无数的模拟世界中运行您的软件、注入故障并寻找故障来解决这个问题。
On any PR, you can turn a dial to decide exactly how much testing you want.
在任何 PR 上，您都可以转动旋钮来准确决定您想要进行多少测试。
And because every run is fully deterministic, agents can branch off the moment a bug appears, rewind it, inspect memory, and replay it, all while the original test keeps running.
而且由于每次运行都是完全确定性的，代理可以在错误出现时分支、倒带、检查内存并重放它，同时原始测试继续运行。
Learn more at antithesis.com/dwarkesh
了解更多信息，请访问 antithesis.com/dwarkesh
Today, I’m chatting with Noam Brown , who is a researcher at OpenAI.
今天，我正在与 OpenAI 的研究员 Noam Brown 聊天。
He was one of the foundational contributors to what became o1 and the reasoning models.
他是 o1 和推理模型的基础贡献者之一。
Now he’s working on multi-agent systems .
现在他正在研究多代理系统。
Speaking of which, you guys announced last week that you solved one of the Millennium Prize Problems with a system of 10,000 different AI agents that spent 130 billion tokens over 88 hours.
说到这里，你们上周宣布，你们用一个由 10,000 个不同的人工智能代理组成的系统在 88 小时内花费了 1300 亿代币解决了其中一个千年奖问题。
One of the reasons I’m interested in talking to you is that you were among the first people, maybe two or three years ago, who were thinking about how the reasoning models would allow us to see into the future.
我有兴趣与您交谈的原因之一是，您可能是两三年前第一批思考推理模型如何让我们预见未来的人之一。
Because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future.
因为如果扩大推理计算规模，您可以看到未来几年模型的基本功能。
I feel like you’re in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now.
我觉得您现在处于类似的位置，可以帮助我们了解未来的功能，因为我们现在可以实现代理规模的巨大扩展。
The way I think about it, when you plot the performance of these reasoning models with test-time compute on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do.
我的想法是，当你在 x 轴上绘制这些推理模型的性能（测试时计算），在 y 轴上绘制基本上任何推理基准的性能时，你会看到一个非常清晰的模式，这些模型思考答案的时间越长，它们做得越好。
This is a very natural thing.
这是很自然的事情。
It’s the same thing with people.
人也是如此。
If you’re taking the SATs and you have five minutes to go through the entire exam, you’re not going to do very well.
如果你参加 SAT 考试，并且你只有五分钟时间完成整个考试，那么你的成绩不会很好。
If you have five hours, you’re probably going to do a lot better.
如果你有五个小时，你可能会做得更好。
The AI models are pretty similar.
人工智能模型非常相似。
They’ll spend that time doing this monologue to themselves, figuring things out, going through different cases, ruling out different possibilities, building on some of their previous discoveries.
他们会花时间对自己进行独白，弄清楚事情，经历不同的案例，排除不同的可能性，建立在他们之前的一些发现的基础上。
The problem is that as you push that further and further, you hit a latency bottleneck.
问题是，当你越推越远时，你就会遇到延迟瓶颈。
You don’t want to sit around for three years waiting for a response.
您不想坐等三年的时间等待回复。
So what you can do is what a lot of people do.
所以你能做的就是很多人所做的。
They parallelize.
他们并行化。
They just get a team of people.
他们只是得到了一个团队。
If you’re going to found a company, you want to get a group of people together so you can go faster.
如果你要创办一家公司，你希望将一群人聚集在一起，这样你就能走得更快。
It’s the same thing with these AI models.
这些人工智能模型也是如此。
It helps to just have multiple agents working on something because they can go faster.
让多个代理同时处理某件事是有帮助的，因为他们可以更快地完成任务。
So multi-agent is a way of scaling test-time compute in parallel instead of purely serially.
因此，多代理是一种并行扩展测试时计算而不是纯粹串行的方法。
It is less efficient, because it’s not like a single agent has all the context to itself.
它的效率较低，因为它不像单个代理拥有自己的所有上下文。
But it is a very effective way of scaling test-time compute if it’s done well.
但如果做得好的话，它是扩展测试时计算的一种非常有效的方法。
I’m going to ask a bunch of naive questions.
我会问一堆幼稚的问题。
This is an unreleased model, so we haven’t publicly seen how these systems work.
这是一个未发布的模型，因此我们还没有公开看到这些系统是如何工作的。
I just have a bunch of ways in which I’m confused about what the qualitative properties of such systems are.
我只是对这些系统的定性属性有很多困惑。
I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time.
我对你能在如此短的时间内集中精力进行认知努力的规模感到震惊。
Think about what 130 billion tokens are.
想想 1300 亿个代币是什么。
If it were a single human thinking as a full-time job, stretched back to back, 130 billion tokens would be a human thinking for 4,000 years.
如果是一个人把思考当作一份全职工作，那么，1300 亿个代币将是一个人思考 4000 年的时间。
Eight hours a day, working a normal work week.
每天八小时，每周正常工作。
Starting from ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours.
从古苏美尔一直到今天，一个连续的人类思考那么长，集中在88小时内。
I feel like qualitatively, that is a super important consideration.
我觉得从质量上来说，这是一个非常重要的考虑因素。
I’m surprised that there isn’t a bigger parallelization penalty.
令我惊讶的是没有更大的并行化惩罚。
You can just have 10,000 agents collaborate.
您只能让 10,000 个代理进行协作。
Maybe because the agents are better at collaborating than humans might be, they’re going much faster.
也许是因为智能体比人类更擅长协作，所以它们的速度要快得多。
They can actually productively collaborate at such a big scale.
他们实际上可以进行如此大规模的富有成效的合作。
Or maybe there is a big parallelization penalty.
或者可能存在很大的并行化惩罚。
Let’s talk about the parallelization penalty, and then we can talk about the qualitative stuff.
让我们讨论并行化惩罚，然后我们可以讨论定性的东西。
The truth is that we don’t have very good science on multi-agent scaling up to this kind of scale.
事实是，对于将多智能体扩展到这种规模，我们还没有很好的科学依据。
When we released 5.6 , I think that was the first time that we had a proper multi-agent system in our models.
当我们发布 5.6 时，我认为这是我们第一次在模型中拥有适当的多代理系统。
We actually did show some plots in the blog post of the scaling performance of multi-agent systems, because we have it as an option.
实际上，我们确实在博客文章中展示了多代理系统的扩展性能的一些图，因为我们有它作为一个选项。
It’s Ultra Mode.
这是超级模式。
The default is four agents, but you can set that higher.
默认值为四个代理，但您可以将其设置得更高。
In the plot, we show what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together.
在图中，我们展示了一个智能体、四个智能体一起工作、16 个智能体一起工作在一些基准上的表现。
It depends on the benchmark, but for some of the benchmarks, what you see is that if you have four agents working on the problem, it is done twice as fast.
这取决于基准，但对于某些基准，您会看到，如果有四个代理处理问题，则完成速度会快两倍。
Because there are four agents working for half as long, you’re paying 2x more to get an answer twice as quickly.
由于四名客服人员的工作时间是原来的一半，因此您需要支付两倍的费用才能以两倍的速度获得答案。
If you go to 16 agents, you see a similar pattern.
如果你去找 16 个代理商，你会看到类似的模式。
It’s a little less efficient, but you continue to see that performance.
它的效率有点低，但你会继续看到这种性能。
Is it a linear serial time speedup or a sublinear speedup as you increase the number of parallel agents?
随着并行代理数量的增加，它是线性串行时间加速还是次线性加速？
It’s slightly sublinear, though it does depend a lot on the problem.
尽管它确实很大程度上取决于问题，但它有点次线性。
Math, for example, is quite parallelizable.
例如，数学是完全可以并行化的。
It’s not the most parallelizable thing, but it is very parallelizable.
它不是最可并行化的东西，但它是非常可并行化的。
Web search, things like doing a Deep Research report where you have to look through a bunch of sources, is extremely parallelizable.
网络搜索，比如做深度研究报告，你必须浏览大量来源，是非常可并行的。
I suspect that something like writing a novel would be very unparallelizable.
我怀疑像写小说这样的事情是无与伦比的。
You would probably not see a big benefit from having 10,000 agents working on a novel together, in the same way that you’d probably not get a big benefit from having 10,000 people work on a novel together.
你可能不会从 10,000 名特工一起创作一部小说中获得很大的好处，就像你可能不会从 10,000 名人员一起创作一部小说中获得很大的好处一样。
So the performance does depend on the domain.
所以性能确实取决于域。
We do measure it up to 16 or so agents in our published blog posts.
我们在发布的博客文章中确实测量了 16 个左右的代理。
The problem is that it’s very hard to push that science to 10,000 agents because it’s just so expensive.
问题在于，很难将这项科学推广给 10,000 名代理人，因为它太昂贵了。
But that’s one data point.
但这只是一个数据点。
We don’t know how long it would take a single agent to solve Navier-Stokes , because we haven’t done that experiment yet.
我们不知道单个智能体需要多长时间才能解决纳维-斯托克斯问题，因为我们还没有做过那个实验。
Maybe we will, but that’s also only one data point.
也许我们会，但这也只是一个数据点。
If we want to do a thorough ablation , the experiments are just too expensive at that scale.
如果我们想要进行彻底的消融，那么这种规模的实验就太昂贵了。
So we have to do some kind of methodical science about what happens when you go to 64, 128, 256 or something and get a sense of the behavior.
因此，我们必须对使用 64、128、256 或其他数字时会发生什么进行某种系统科学，并了解其行为。
But it’s going to be very hard to push that all the way to 10,000 and know for sure what the benefit was that we actually got from using 10,000 agents versus 1,000.
但将其一直推到 10,000 名并确定使用 10,000 名代理与使用 1,000 名代理相比究竟能带来什么好处，这是非常困难的。
There’s one thing I want to make clear.
我想澄清一件事。
The effort to solve a Millennium Prize Problem, this was not due to multi-agent.
解决千年奖问题的努力，并不是多智能体的功劳。
I wouldn’t even attribute 10% of the credit to multi-agent.
我什至不会将 10% 的功劳归功于多代理。
The reality is that OpenAI has trained a very powerful model.
事实上，OpenAI 已经训练了一个非常强大的模型。
We can get that model to operate over very long horizons.
我们可以让该模型在很长的时期内运行。
We can get it to think in parallel.
我们可以让它并行思考。
But at its core, the reason why we’re able to do this is because we just have a general-purpose, very strong model.
但从本质上讲，我们之所以能够做到这一点是因为我们有一个通用的、非常强大的模型。
Things like multi-agent are flashy and new, and that probably gets disproportionate credit for that reason.
像多代理这样的东西是华而不实和新颖的，因此可能会得到不成比例的赞誉。
But the core reason is this is just a very powerful model.
但最核心的原因是这只是一个非常强大的模型。
The generalization is quite shocking to me.
这种概括令我相当震惊。
I don’t know how these systems were trained, but presumably they were trained how RL training happens.
我不知道这些系统是如何训练的，但大概它们是如何进行强化学习训练的。
You have a bunch of checkable synthetic problems and you do a bunch of RL against them.
你有一堆可检查的综合问题，然后你对它们进行了一堆强化学习。
Nowhere in the training process, I’m guessing, was the model solving anything as ambitious as a Millennium Prize Problem.
我猜，在训练过程中，模型并没有解决像千年奖问题这样雄心勃勃的问题。
But the generalization was strong enough that you could have these much easier verifiable problems generalize to this much parallel effort on such a hard problem.
但概括性足够强，您可以将这些更容易验证的问题推广到针对这样一个难题的并行努力。
I think that is true.
我认为这是真的。
First of all, we do train the model on very hard problems.
首先，我们确实在非常困难的问题上训练模型。
There is definitely a gap.
肯定是有差距的。
We see that if we train on some kinds of tasks, it’s able to do tasks that are more ambitious than that.
我们发现，如果我们训练某些类型的任务，它就能完成比这更艰巨的任务。
There is an interesting challenge that as the models become smarter and smarter, a lot of the kinds of questions we can ask them are just too easy.
有一个有趣的挑战，随着模型变得越来越聪明，我们可以问它们的很多问题都太简单了。
It’s hard to challenge the model.
挑战这个模型是很困难的。
I do think that’s going to be interesting.
我确实认为这会很有趣。
If I had to make an argument for why you might not see AIs like LLMs go the same path as AlphaGo and AlphaZero and all these kinds of game-playing AIs, it might be this kind of problem.
如果我必须争论为什么你可能看不到像法学硕士这样的人工智能走上与 AlphaGo 和 AlphaZero 以及所有这些类型的玩游戏人工智能相同的道路，那可能就是这类问题。
In things like AlphaZero, where you have self-play , you have an infinite curriculum .
在像 AlphaZero 这样的东西中，你可以进行自我游戏，你可以拥有无限的课程。
You’re always playing against an AI that’s equally strong.
你总是在与同样强大的人工智能对抗。
Whereas for things like training an LLM with reinforcement learning, at least the ways that are out there right now, you give the model a problem and you ask it to solve it.
而对于像用强化学习训练法学硕士这样的事情，至少是现在的方法，你给模型一个问题，然后要求它解决它。
If the problem is so easy that it can just solve it in a second, it’s not really learning anything.
如果问题太简单，一秒钟就能解决，那么它就没有真正学到任何东西。
If we run out of problems to challenge it, then that is a plausible scenario where it becomes much harder to make progress.
如果我们没有问题来挑战它，那么这就是一种合理的情况，取得进展会变得更加困难。
Now, I do think there are ways around that.
现在，我确实认为有办法解决这个问题。
We haven’t really hit that as a wall yet.
我们还没有真正碰壁。
I think that if it ever became a serious problem, there would be ways around it.
我认为，如果它成为一个严重的问题，总会有办法解决的。
But it is a plausible scenario.
但这是一个合理的情况。
Just for the audience, when you’re referring to AlphaGo or AlphaZero, you’re talking about getting superhuman relatively fast after achieving human-level performance.
对于观众来说，当你提到 AlphaGo 或 AlphaZero 时，你指的是在达到人类水平的表现后相对较快地获得超人。
If you look at the trajectory of game-playing AIs, like Go , within a span of a year they went from beating a European champion — something like number 50 in the world — to beating the world champion , to being unimaginably, orders of magnitude stronger than any human alive.
如果你观察一下围棋等游戏人工智能的发展轨迹，就会发现，在一年的时间里，它们从击败欧洲冠军（世界排名第 50 名）到击败世界冠军，再到比任何活着的人类强出难以想象的数量级。
It’s possible that in domains like math we see a similar trajectory, but I think there is a very plausible scenario where that doesn’t happen.
在数学等领域，我们可能会看到类似的轨迹，但我认为有一种非常合理的情况不会发生这种情况。
I want to understand, if in six months people will have access to multi-agent systems, how should one model what it is like to collaborate with or hire a multi-agent system?
我想了解，如果六个月后人们将能够访问多代理系统，那么应该如何模拟与多代理系统协作或雇用多代理系统的情况？
I should start by talking about how these multi-agent systems actually work, which I think is a very different way than a lot of multi-agent systems in other AIs.
我应该首先谈谈这些多代理系统的实际工作方式，我认为这与其他人工智能中的许多多代理系统有很大不同。
A lot of people that have approached multi-agents for things like LLMs tend to take this very scaffolded approach.
许多为诸如法学硕士之类的事情而接触多代理人的人倾向于采用这种非常支架式的方法。
For example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task.
例如，可能有一个协调代理将工作委托给一群孩子并给他们一个任务。
The children work on it and then return their answer.
孩子们思考这个问题，然后返回他们的答案。
This seems like a very sensible setup, a very sensible scaffold.
这看起来是一个非常明智的设置，一个非常明智的脚手架。
It definitely helps, but there are a bunch of limitations with these kinds of setups.
它肯定有帮助，但此类设置存在很多限制。
For example, if in this setup you have a coordinator that’s sending tasks to children, and the children work on it and then return their answers, what happens if two children are given similar tasks?
例如，如果在这个设置中，你有一个协调员向孩子们发送任务，孩子们处理它，然后返回他们的答案，如果给两个孩子分配类似的任务，会发生什么？
Can they talk to each other?
他们可以互相交谈吗？
Usually the answer is no. That’s very inefficient.
通常答案是否定的。这是非常低效的。
If you’re given a task and it’s actually really helpful to talk to somebody that might know an answer to a question that you’re working on — or part of something that you’re working on — it’d be really helpful for you to just be able to ping them and say, “Hey, can you help me out with this thing?” But a lot of systems don’t have that setup.
如果你接到一项任务，并且与可能知道你正在研究的问题或你正在研究的某件事的一部分的答案的人交谈实际上非常有帮助，那么你可以直接联系他们并说：“嘿，你能帮我解决这件事吗？”但很多系统没有这样的设置。
Adding it significantly increases the complexity of the scaffold that you have.
添加它会显着增加您所拥有的脚手架的复杂性。
Another thing is, what if the child doesn’t really understand or has a clarification question?
还有就是，如果孩子不太明白或者有疑问怎么办？
Then it has to choose between, “Okay, do I just return and ask the question instead of solving the problem?” or “Do I solve the problem, make an assumption about what the parent wanted me to do, and just solve it that way?” In any scaffold that people come up with, there are always limitations involved.
然后它必须做出选择：“好吧，我是否只是返回并提出问题而不是解决问题？”或者“我是否解决问题，假设父母希望我做什么，然后就这样解决？”人们提出的任何支架总是存在局限性。
The approach that we wanted to take was to just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use, and they figure out for themselves how to use them effectively.
我们想要采取的方法是在尽可能小的结构中走向烘焙的极端，并为代理提供非常原始的工具来使用，然后他们自己弄清楚如何有效地使用它们。
So we give the agents the ability to message another agent, and when it messages another agent, it is inserted into the context.
因此，我们赋予代理向另一个代理发送消息的能力，当它向另一个代理发送消息时，它就会被插入到上下文中。
It can do a few other similar things, but that’s basically the core of it.
它可以做一些其他类似的事情，但这基本上是它的核心。
It can just send a message whenever it wants — just a tool call — and it can send that to other agents.
它可以随时发送消息（只需一个工具调用），然后将其发送给其他代理。
They figure out for themselves the best way to coordinate around that.
他们自己找出协调这一问题的最佳方法。
It turns out that if this is done well, you get very sophisticated behavior.
事实证明，如果做得好，你会得到非常复杂的行为。
To me, it looks a lot like how human collaborators work over something like Slack, for example.
对我来说，这看起来很像人类协作者如何处理 Slack 之类的东西。
When we were working on this project, it was really exciting when we finally got it working to see these agents working on problems together.
当我们从事这个项目时，当我们最终看到这些代理一起解决问题时，我们真的很兴奋。
I remember one example.
我记得一个例子。
We give the agents a problem, and then one agent says, “I think I’ve got the answer.” Then another agent says, “Actually, I got a different answer.” Then they have this whole discussion about, “Well, how did you arrive at that answer?
我们向特工提出一个问题，然后一名特工说：“我想我已经找到答案了。”然后另一位经纪人说：“实际上，我得到了不同的答案。”然后他们进行了整个讨论：“那么，你是如何得出这个答案的？
Can you explain it to me?” Going back and forth and trying to clarify what could’ve been wrong in each other’s reasoning.
你能给我解释一下吗？”来来回回，试图澄清彼此的推理中可能存在哪些错误。
Then they finally converge on, “Oh, yeah.
然后他们终于集中在一起，“哦，是的。
Okay, that seems right.” Then it just broadcasts to the other agents, “Actually, I’ve changed my answer.
好吧，看来是这样。”然后它就向其他特工广播：“实际上，我已经改变了我的答案。
I think he’s right.” It just felt like a very natural conversation.
我认为他是对的。”感觉就像是一次非常自然的对话。
It felt like when you see chain of thought for the first time that’s trained through reinforcement learning, and you’re like, “Oh, this is just kind of like what a person would think if they were writing down their thoughts as they’re thinking them.” It felt like that.
感觉就像当你第一次看到通过强化学习训练的思想链时，你会想，“哦，这有点像一个人在思考时写下自己的想法时会想到的。”感觉就是这样。
It is really cool to see this kind of behavior.
看到这种行为真的很酷。
Collaborating with these things, honestly, feels a lot like collaborating with a person.
老实说，与这些事物合作感觉很像与一个人合作。
It’s just a very natural flow.
这只是一个非常自然的流程。
Except one qualitative difference that might become salient in the future is that these systems will be thinking maybe more than 10x as fast, if you just look at how many tokens per second they output versus how fast a human talks.
除了未来可能变得显着的一个质的差异之外，如果你只看它们每秒输出多少个令牌与人类说话的速度，这些系统的思考速度可能会快 10 倍以上。
They’re working all the time.
他们一直在工作。
They’re not sleeping.
他们没有睡觉。
They’re collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans.
他们彼此合作的速度比人类与其他人类合作的能力要快得多。
I’m trying to think of what to qualitatively expect in a year.
我正在尝试思考一年内的质量预期。
Is it like a shadow organization that is moving 100x faster in my company than the human level is?
它是否像一个影子组织，在我的公司中移动速度比人类水平快 100 倍？
What would take a human organization a year to do is happening within a week within this shadow organization?
一个人类组织需要一年才能完成的事情，在这个影子组织内一周内就能完成？
Will it feel foreign?
会不会有陌生的感觉？
I don’t know.
我不知道。
I’ve actually found that it’s surprisingly natural to work with these things right now.
事实上，我发现现在处理这些事情是非常自然的。
I think that could change.
我认为这可能会改变。
For example, we have these ultra-fast modes that enable sampling to be 10-15x faster or whatever.
例如，我们拥有这些超快模式，可以使采样速度提高 10-15 倍或其他速度。
Then it’s going to be pretty hard to keep up with these things.
那么跟上这些事情将会非常困难。
The idea is that these agents, when they’re communicating with each other, can go super fast.
这个想法是，这些代理在相互通信时可以走得超快。
But they also understand when they’re talking to an agent versus when they’re talking to a person, and their behavior will be different in those situations.
但他们也明白，当他们与代理人交谈时和与人交谈时，他们的行为在这些情况下会有所不同。
The main example that we have publicly of sophisticated multi-agent systems is unfortunately the Hugging Face one .
不幸的是，我们公开的复杂多智能体系统的主要例子是 Hugging Face 系统。
A lot of things I found concerning there, obviously.
显然，我在那里发现了很多有关的事情。
But the thing I found interesting there is the spontaneous emergence of hierarchy, of middle management.
但我发现有趣的是等级制度和中层管理的自发出现。
It sounds like you’re saying this level of organization emerges spontaneously from training?
听起来你是说这种级别的组织是通过培训自发形成的？
The details are spontaneous.
细节是自发的。
But while we’re giving a lot of flexibility to the agents to decide how to communicate with each other in the optimal way, we are still giving them a starting point.
但是，虽然我们为代理提供了很大的灵活性来决定如何以最佳方式相互通信，但我们仍然为他们提供了一个起点。
We’re giving them a prior about what reasonable communication might look like.
我们让他们先了解什么是合理的沟通。
They’re also trained on a lot of human text.
他们还接受过大量人类文本的训练。
They have an understanding of how humans organize and coordinate, so that’s all baked in.
他们了解人类如何组织和协调，所以这一切都融入其中。
I think it is surprising the way they’re able to polish this.
我认为他们能够完善这一点的方式令人惊讶。
If you look at what it starts out at, it’s not very sophisticated behavior.
如果你看看它一开始的样子，就会发现这并不是非常复杂的行为。
In fact, it’s actually very difficult to get these agents to coordinate in a productive way, because it’s very tempting for them to just collapse to, “Oh, we’re all just going to solve the problem independently.” That is a local minimum that you can get stuck in.
事实上，让这些代理以富有成效的方式进行协调实际上非常困难，因为他们很容易崩溃，“哦，我们都会独立解决问题。”这是您可能会陷入困境的局部最小值。
But if it’s done well, they can end up coordinating very effectively in these kinds of very structured ways.
但如果做得好，他们最终可以以这种非常结构化的方式进行非常有效的协调。
I wrote this essay a couple of years ago about what automated firms will look like .
几年前我写了一篇关于自动化公司是什么样子的文章。
I was thinking about, if you had fully automated firms of, let’s say, human-level intelligences, what is different about the nature of AI minds that would make the organizations AIs form different?
我在想，如果你有完全自动化的公司，比如说，人类水平的智能，那么人工智能思维的本质有什么不同，会让人工智能形成的组织有所不同？
There are a couple of very important differences.
有几个非常重要的区别。
For example, AIs can share context much more seamlessly than humans can.
例如，人工智能可以比人类更无缝地共享上下文。
They can merge their knowledge much more seamlessly.
他们可以更加无缝地融合他们的知识。
Also, you can spin up or spin down an arbitrary number of instances which have the right knowledge.
此外，您还可以启动或关闭任意数量的具有正确知识的实例。
So if you want to hire more people, it’s not all the schlep of finding the right talent or whatever.
因此，如果你想雇用更多的人，并不只是寻找合适的人才或其他什么问题。
Your best talent, you can just make infinite copies of them.
你最好的天赋，你可以无限复制它们。
Or if you don’t need them for the task anymore, you can spin them down.
或者，如果您不再需要它们来完成任务，您可以将它们关闭。
You can replicate the most effective parts of your organization, or replicate whole organizations together which are effective.
您可以复制组织中最有效的部分，或者复制整个有效的组织。
Where do you see these multi-agent systems going a year from now or two years from now?
您认为这些多代理系统一年后或两年后会走向何方？
It’s a great question: how do these things actually differ from working with a human coworker?
这是一个很好的问题：这些事情与与人类同事一起工作实际上有何不同？
You highlighted some.
你强调了一些。
One really interesting thing is that if you have a person and you want two copies of them, you can’t just clone the person.
一件非常有趣的事情是，如果你有一个人并且你想要他们的两个副本，你不能仅仅克隆这个人。
But with AIs, it’s actually really easy to just say, “Okay, just fork yourself,” and then have both copies work on this thing and then merge back together.
但对于人工智能来说，实际上很容易说，“好吧，你自己分叉”，然后让两个副本都处理这个事情，然后重新合并在一起。
We already have this, I think, in multi-agent for Astra and 5.6 Sol, where when they spin up sub-agents, the context is just forked.
我认为，我们已经在 Astra 和 5.6 Sol 的多代理中实现了这一点，当它们启动子代理时，上下文就被分叉了。
So it has all the context that’s relevant.
所以它具有所有相关的上下文。
There are other interesting ways where the agents will differ from people.
代理与人之间还有其他有趣的区别。
Like, what are some reasons why startups disrupt incumbents ?
比如，初创企业颠覆现有企业的原因有哪些？
There are a few factors.
有几个因素。
One is that they’re willing to take more risks.
一是他们愿意承担更多风险。
But another major factor is, as organizations grow in size, you see increasing misalignment between the individuals in the organization.
但另一个主要因素是，随着组织规模的扩大，您会发现组织中的个人之间的不一致越来越严重。
If you have a startup with five people and each person has a 20% share in the company, they’re all highly aligned to the company succeeding.
如果你的初创公司有 5 个人，每个人都拥有公司 20% 的股份，那么他们都与公司的成功高度一致。
If you have a massive company with 10,000 people, you see a lot more instances where people are territorial, or just care about getting a lot of headcount for their project or their team, building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted.
如果你有一家拥有 10,000 名员工的大公司，你会看到更多这样的情况：人们都有领地意识，或者只是关心为他们的项目或团队获得大量员工，建立他们的领地，获得大量资源，以便他们可以发布很酷的工作或其他什么并获得晋升。
This is actually a real detriment.
这实际上是一个真正的损害。
I think this explains a lot of why startups are able to disrupt incumbents.
我认为这在很大程度上解释了为什么初创企业能够颠覆现有企业。
It’s true that AI does help startups in a way.
确实，人工智能确实在某种程度上帮助了初创企业。
It’s much easier than ever before for one person to step in and be like, “I’m going to make a multimillion-dollar company.” The AIs amplify an individual so much.
对于一个人来说，介入并说“我要创建一家价值数百万美元的公司”比以往任何时候都容易得多。人工智能极大地增强了个体的能力。
But there’s also an argument that they could benefit incumbents.
但也有一种观点认为，它们可以让现任者受益。
If the alignment problem is solved, then you don’t have the issue of misalignment between individuals in the company.
如果一致性问题得到解决，那么公司中的个人之间就不会出现不一致的问题。
At least that’s mitigated.
至少这种情况有所缓解。
The AIs, if they’re aligned well, can just be aligned to the interest of the company.
如果人工智能协调得好，它们就能符合公司的利益。
You can have 10,000 of them, and they’re all going to be working as hard as if they were a 20%-share co-founder.
你可以拥有 10,000 名员工，他们都会像持股 20% 的联合创始人一样努力工作。
It’s not only that, but it’s also that they are much better able to manage shared memory and context than different humans can.
不仅如此，他们比其他人更能管理共享记忆和上下文。
If tomorrow you hire 10,000 mathematicians and you’re like, “Solve Navier-Stokes,” they’re not going to be able to cooperate effectively, at least not off the bat.
如果明天你雇佣 10,000 名数学家，然后你说“解决纳维-斯托克斯问题”，他们将无法有效合作，至少不能立即合作。
But apparently you can have 10,000 AIs do that.
但显然你可以让 10,000 个人工智能来做这件事。
Again, I want to be conservative here, because we haven’t measured how effective the 10,000 agents are at coordinating.
再次强调，我想在这里保持保守，因为我们还没有衡量 10,000 名代理的协调效率。
We think it helped.
我们认为这有帮助。
We don’t actually have good measurements saying, “This 10,000 agents led to a 2x speedup over 2,000 agents,” or something like that.
我们实际上没有很好的衡量标准，比如“这 10,000 个代理比 2,000 个代理实现了 2 倍的加速”，或者类似的话。
I don’t know about likely, but I think it is very possible that 10,000 humans are better at coordinating than 10,000 agents right now.
我不知道可能性如何，但我认为现在 10,000 个人很可能比 10,000 个智能体更擅长协调。
I think it is entirely possible.
我认为这是完全有可能的。
Also, one trend we’ve been seeing is…
此外，我们看到的一个趋势是……
Look, we’ve been working on multi-agent for a while, and the early versions of this were very difficult to get right.
看，我们在多代理方面的研究已经有一段时间了，早期版本很难做好。
It was very hard to get the agents to even talk to each other.
让特工们互相交谈都非常困难。
It’s because when we first developed reasoning models, they weren’t talking to other agents.
这是因为当我们第一次开发推理模型时，它们并没有与其他智能体交谈。
If you now put a bunch of agents together and say, “Solve this problem together,” they’re in this local minimum where they’re really good at thinking deeply about a problem, and it just interrupts their chain of thought.
如果你现在把一群智能体放在一起并说“一起解决这个问题”，他们就处于这个局部最小值，他们非常擅长深入思考问题，而这只会打断他们的思维链。
It interrupts their flow to constantly be checking in with other agents or receiving messages from them.
它会中断他们的流程，以不断地与其他代理联系或接收来自他们的消息。
The optimization is actually very hard to get right in that situation.
在这种情况下，优化实际上很难正确进行。
I think it’s that they’re not as general.
我认为他们并不那么普遍。
The earlier models were just not as generalizable and were more narrow.
早期的模型并不具有普遍性，而且范围更窄。
As the models have become more capable, it’s been easier for them to develop this capability, and I do think that as they become stronger and stronger across the board, they will become better at organizing themselves in large organizations.
随着模型变得越来越有能力，他们更容易发展这种能力，而且我确实认为，随着他们在各个方面变得越来越强大，他们将变得更善于在大型组织中组织自己。
I don’t know, maybe they are better than people at organizing in 10,000-person groups.
我不知道，也许他们比人更擅长组织万人团体。
But even if they’re not, a year from now, two years from now, it’s quite possible that they’ll do that even if we don’t end-to-end optimize them for that.
但即使他们没有，一年后、两年后，即使我们不为此进行端到端优化，他们也很可能会这样做。
Here’s why this result, and maybe the general progress that AI has made in mathematics , has made me think that RSI is more plausible and sooner than I previously thought.
这就是为什么这个结果，也许还有人工智能在数学方面取得的普遍进展，让我认为 RSI 比我之前想象的更合理、更早。
I feel like in mathematics we’ve gone from, let’s say, 2024, where you have AIs and it’s, “Oh, okay, interesting.
我觉得在数学领域，我们已经离开了 2024 年，那时人工智能已经出现，“哦，好吧，有趣。
They can solve a couple problems on high school math competitions.” Then in 2025, it’s, “Oh, wow, they can get gold in the International Math Olympiad.” Earlier this year, it was, “Wow, they’re actually solving open problems in mathematics,” like open Erdős problems .
他们可以解决高中数学竞赛中的几个问题。”然后到了 2025 年，“哦，哇，他们可以在国际数学奥林匹克竞赛中获得金牌。”今年早些时候，“哇，他们实际上正在解决数学中的开放问题”，就像开放的埃尔多斯问题一样。
But maybe people weren’t trying that hard, and there was a similar solution somewhere in the literature.
但也许人们并没有那么努力，文献中也有类似的解决方案。
Now I just think it’s undeniable.
现在我认为这是不可否认的。
This is the Millennium Prize Problem.
这就是千年奖问题。
There’s no story of why this should have been easy.
没有故事说明为什么这应该很容易。
Now, a lot of people have pointed out — I think Terry Tao had a post like this, Toby Ord wrote an interesting post about this — that they’re solving a lot of these problems, but I’m not aware of them coming up with new insights or formulating insightful new questions and new modes of theory for thinking about mathematics, like coming up with topology or coming up with the Cartesian grid .
现在，很多人指出——我认为特里·陶有过这样的帖子，托比·奥德写了一篇关于这个的有趣的帖子——他们正在解决很多这样的问题，但我不知道他们提出了新的见解或提出了有洞察力的新问题和新的理论模式来思考数学，比如提出拓扑或提出笛卡尔网格。
So maybe the actual progress in mathematics, broadly construed, is smaller than it might seem if you’re just looking at well-scoped problems that are directly solved.
因此，从广义上讲，数学的实际进展可能比你只关注直接解决的范围广泛的问题所看起来的要小。
However, I think that kind of progress would be incredibly meaningful in ML , because in ML you don’t care about better understanding the nature of deep learning , or you only care about that as an instrumental goal towards just achieving the result.
然而，我认为这种进步对于机器学习来说是非常有意义的，因为在机器学习中，你不关心更好地理解深度学习的本质，或者你只关心它作为实现结果的工具性目标。
Just solve this well-scoped problem of improving the sample efficiency of our models, improving the pre-training loss , improving whatever.
只要解决这个范围广泛的问题，即提高模型的样本效率，提高预训练损失，等等。
The kind of progress that we’re seeing arrive like an avalanche in mathematics is structurally very similar…
我们所看到的像数学雪崩一样到来的进步在结构上非常相似……
Again, I’m curious if this is the case, I’m just a total outsider.
再说一次，我很好奇是否是这样，我只是一个完全的局外人。
I’m wondering if it’s structurally very similar to the direct uplift that you would expect in AI progress.
我想知道它在结构上是否与您期望的人工智能进步中的直接提升非常相似。
The thing that’s shocking to me, or potentially concerning, is just how fast we went from, “Oh, they’re giving me 50% uplift,” if you’re a mathematician, to, “Wow, they’re just end-to-end solving the biggest open problems in the field.”
让我震惊或可能令人担忧的是，我们从“哦，他们给了我 50% 的提升”（如果你是一名数学家）到“哇，他们只是端到端解决该领域最大的开放问题”的速度有多快。
There’s a lot to unpack there.
那里有很多东西需要解压。
Let’s start with the progress on math.
让我们从数学的进展开始。
Yes, the models are doing some crazy powerful stuff, and it’s progressing faster than I expected.
是的，这些模型正在做一些疯狂而强大的事情，而且进展速度比我预期的要快。
When we got IMO gold in 2025, what I thought was…
当我们在 2025 年获得 IMO 金奖时，我的想法是……
When the models figured out how to do GSM8K , it would take a human mathematician about five seconds to do a GSM8K problem.
当模型弄清楚如何解决 GSM8K 问题时，人类数学家需要大约五秒钟的时间才能解决 GSM8K 问题。
This is grade school math, grades K-8.
这是小学数学，K-8 年级。
Then the next year, they were able to do the MATH benchmark problems.
然后第二年，他们就能够解决数学基准问题了。
These would take an expert human mathematician maybe a minute to do.
这些可能需要人类数学专家一分钟才能完成。
Then you get to AIME .
然后你就可以到达 AIME 了。
This is the qualifier for the USA Mathematics Olympiad team.
这是美国数学奥林匹克代表队的预选赛。
It would take a good human mathematician probably 10 minutes to do, and the models were able to do that a year later.
一个优秀的人类数学家可能需要 10 分钟才能完成，一年后模型就能够做到这一点。
So every year, you’re seeing this 10x increase in the tasks they’re able to do, in terms of how long it would take a human mathematician to do it.
因此，每年，你都会看到他们能够完成的任务增加了 10 倍，以人类数学家完成任务所需的时间来衡量。
Then it was very sensible that a year later we get to IMO gold, because that’s 100 minutes.
一年后我们获得了 IMO 金牌，这是非常明智的，因为那是 100 分钟。
That’s about how long it takes a human mathematician to do an IMO problem.
这大约是人类数学家解决一道 IMO 问题所需的时间。
Just projecting outwards, I was like, “Okay, how long would it take a person to solve something like a Millennium Prize Problem?” I don’t have a good sense, but if we are following this trend line of 10x every year, we go from IMO gold, which is taking an hour and a half, to next year, 15 hours.
只是向外投射，我想，“好吧，一个人需要多长时间才能解决像千年奖问题这样的问题？”我没有很好的判断力，但如果我们每年都遵循这条 10 倍的趋势线，我们就会从 IMO 黄金（需要一个半小时）到明年的 15 小时。
That should not be enough to solve a Millennium Prize Problem.
这应该不足以解决千年奖问题。
So I was like, “I don’t think we’re going to get it in 2026, probably not in 2027, maybe in 2028.” So it did happen a lot faster than I expected.
所以我想，“我认为我们不会在 2026 年实现它，可能不会在 2027 年，也许会在 2028 年。”所以它确实发生得比我预期的要快得多。
Now, there is a narrative going around that these things are replacing mathematicians, that it’s just superhuman in mathematics across the board.
现在，有一种说法认为这些东西正在取代数学家，它在数学领域简直是超人。
I think that is the wrong takeaway.
我认为这是错误的。
They’re clearly exceptional in some ways, but they are weaker than human mathematicians in other ways.
他们在某些方面显然很出色，但在其他方面却比人类数学家弱。
We have this jagged scenario where the models are brilliant in some dimensions and also weaker than humans in other dimensions.
我们有这样一个锯齿状的场景，其中模型在某些维度上表现出色，但在其他维度上却比人类弱。
Like you said, they’re not very good at posing new problems.
就像你说的，他们不太擅长提出新问题。
They’re not really good at understanding what directions, what whole branches of mathematics are worth exploring or developing.
他们并不真正擅长理解数学的哪些方向、哪些整个分支值得探索或发展。
My opinion is that I think this is great.
我的意见是我认为这很棒。
I would be thrilled to live in a world where AI is a complement to human abilities and is allowing us to discover new knowledge without fully replacing people.
我很高兴生活在这样一个世界：人工智能是对人类能力的补充，让我们能够在不完全取代人类的情况下发现新知识。
That is the best-case scenario.
这是最好的情况。
I do think it’s true that the AIs are jagged, but as they get better, they get better across the board.
我确实认为人工智能确实是参差不齐的，但随着它们变得更好，它们会全面变得更好。
So the things they’re exceptional at, they’re going to get even more exceptional at.
因此，他们擅长的事情将会变得更加出色。
The things where they’re far behind humans, they’re going to be less behind humans at.
在那些远远落后于人类的事情上，他们将不再落后于人类。
Over time, it is possible that they’re just better across the board.
随着时间的推移，它们可能会在各方面都变得更好。
Now, I don’t know how long that takes.
现在，我不知道这需要多长时间。
It depends on how long the long tail is of things that they’re bad at.
这取决于他们不擅长的事情的长尾有多长。
This brings us back to RSI.
这让我们回到 RSI。
Again, I want to emphasize here that I’m just a total outsider.
在这里我想再次强调，我只是一个完全的局外人。
I’m a podcaster, but as somebody interested in and concerned about what’s happening in the field, I’m trying to reason about when to expect RSI and what kind of thing to expect.
我是一名播客，但作为一个对该领域正在发生的事情感兴趣并关心的人，我试图推理何时会出现 RSI 以及会发生什么样的事情。
The amount of cognitive effort that was dumped into this Millennium Prize Problem is a good intuition pump.
投入到这个千年奖问题中的认知努力是一个很好的直觉泵。
You could have AIs that are spending, over the course of maybe a week, more cognitive effort on a long-standing ML problem, like very fluid online learning , than maybe the field has spent cumulatively in its entire existence.
你可能会让人工智能在一周的时间里，在一个长期存在的机器学习问题上花费更多的认知努力，比如非常流畅的在线学习，可能比该领域在整个存在过程中花费的累计花费还要多。
Then you could say, “Well, unlike mathematics, of course, AI requires experiments, and that takes compute, and that takes time.
然后你可以说：“当然，与数学不同，人工智能需要实验，这需要计算，也需要时间。
You can’t just think on pen and paper and actually make things happen.”
你不能只在笔和纸上思考并实际让事情发生。”
But just look at the amount of compute that is available at an organization like OpenAI.
但只要看看 OpenAI 这样的组织可用的计算量就知道了。
It took 10,000 agents with the Millennium Prize Problem.
千禧年奖问题花费了 10,000 名特工。
By the end of next year, OpenAI will have enough compute such that, let’s say you have 10,000 agents at the end of next year.
到明年年底，OpenAI 将拥有足够的计算能力，假设到明年年底您将拥有 10,000 个代理。
They’re much smarter by that point.
到那时他们就聪明多了。
Each of them will have enough compute to run a GPT-3 -sized experiment every single day.
他们每个人都有足够的计算能力每天运行一个 GPT-3 大小的实验。
That seems like a lot for superhuman researchers who are thinking super fast.
对于思维超快的超人研究人员来说，这似乎很多。
What do you think about that intuition pump?
你觉得那个直觉泵怎么样？
I think it’s pretty accurate.
我认为这是非常准确的。
These things are very spiky.
这些东西非常尖锐。
When it comes to mathematics, they’re way better in some ways, but they’re also worse in other ways.
就数学而言，他们在某些方面要好得多，但在其他方面也更差。
But the ways that they’re spiky end up, I think, probably being particularly useful for things like RSI.
但我认为，它们的尖峰方式最终可能对 RSI 之类的东西特别有用。
You have a more clear objective.
你有一个更明确的目标。
It’s just more measurable.
它只是更可衡量。
There’s less question of, “Well, what new branches of mathematics are worth exploring?” No, there’s a very clear answer.
不再有这样的问题：“嗯，数学的哪些新分支值得探索？”不，有一个非常明确的答案。
There are certain metrics that you care about, and if you can make it do better on those metrics, then you’ve succeeded.
您关心某些指标，如果您能让它在这些指标上做得更好，那么您就成功了。
So I think there is a lot of truth to that.
所以我认为这有很多道理。
The main difference is that in mathematics, you’re purely bottlenecked by thinking.
主要的区别在于，在数学中，你纯粹受到思维的瓶颈。
Yes, there are some parts of mathematics where you care about running experiments and getting results and these kinds of things.
是的，数学的某些部分你关心的是进行实验并获得结果之类的事情。
But for the most part, it’s just really bottlenecked by thinking really hard, and the models are really good at that.
但在大多数情况下，它只是通过认真思考而真正遇到瓶颈，而模型非常擅长这一点。
When you look at things like RSI, you do have to run experiments.
当您查看 RSI 之类的东西时，您确实必须进行实验。
It’s not enough to just be extremely smart.
仅仅非常聪明是不够的。
One argument for this is, if you had 100x less compute and all the most brilliant people in the world working at OpenAI, how much progress would you be making relative to having the amount of compute that we have now with the amount of people we have?
对此的一个论点是，如果你的计算量减少了 100 倍，并且世界上所有最聪明的人都在 OpenAI 工作，那么相对于我们现在拥有的计算量和我们拥有的人员数量，你会取得多少进展？
I suspect it would be less progress, actually.
我怀疑实际上进展会更少。
No, not 100x less.
不，不是减少 100 倍。
But the question you’re getting at is, if we have RSI and we have all of these brilliant AIs running around, running experiments and stuff with the compute that we have, how much faster does progress go?
但你遇到的问题是，如果我们有 RSI，并且我们有所有这些出色的人工智能在运行，用我们拥有的计算进行实验和其他东西，那么进展会快多少？
I think this is something we disagree on.
我认为这是我们不同意的一点。
We do see a speedup, and we see a significant speedup.
我们确实看到了加速，而且是显着的加速。
But I don’t think it’s an overnight intelligence explosion where we go 100x faster, because we do get bottlenecked by certain limitations that are not bottlenecks of intelligence.
但我不认为智能爆炸会在一夜之间让我们的速度提高 100 倍，因为我们确实会受到某些并非智能瓶颈的限制的阻碍。
It’s running experiments.
它正在进行实验。
It’s running experiments serially, because they take a while to either train new models or to get the results.
它连续运行实验，因为它们需要一段时间来训练新模型或获得结果。
It’s having the GPUs to run those experiments.
它有 GPU 来运行这些实验。
So it’s unclear how much faster things go.
因此尚不清楚事情进展得有多快。
I definitely think they go a lot faster.
我绝对认为他们走得更快。
To be clear, considering how fast things are going now on an exponential, if that exponential is 3x faster, that is massive.
需要明确的是，考虑到现在指数的发展速度有多快，如果指数快 3 倍，那就是巨大的了。
But there’s a big difference between that and 100x faster.
但这和快 100 倍之间有很大区别。
I’m quite deferential to your inside view on what RSI looks like or what the dynamics are, because obviously you’ve been in the field for 10 years.
我非常尊重您对 RSI 的外观或动态的内部看法，因为显然您已经在该领域工作了 10 年。
I’m trying to reason about it from very outside-view types of intuition pumps.
我试图从直觉泵的外部视角来推理它。
I’ll say that people have different opinions on this.
我想说的是，人们对此有不同的看法。
I have my opinion on this.
对此我有自己的看法。
I could totally be wrong.
我可能完全错了。
I admit that.
我承认这一点。
I have some confidence in this, but I’m not 100% confident that this is the way things go.
我对此有一定的信心，但我并不是 100% 相信事情就是这样。
Maybe there could be an overnight intelligence explosion, I don’t know.
也许一夜之间就会出现情报爆炸，我不知道。
Maybe we don’t see a 3x speedup.
也许我们看不到 3 倍的加速。
Maybe it’s a 50% speedup.
也许加速了 50%。
There’s a lot of uncertainty here.
这里有很多不确定性。
A couple of point.
有几点。
Tangentially, I want to clarify something about the jaggedness.
顺便说一句，我想澄清一些关于锯齿状的问题。
One thing that gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner, because that better learner can be more general.
最近让我印象深刻的一件事是，人工智能只要非常擅长构建更好的学习器就足够了，因为更好的学习器可以更通用。
If you just make an AI that’s better at using Office products or playing chess or something, whatever.
如果你只是制作一个更擅长使用 Office 产品或下棋或其他什么的人工智能。
That’s fine.
没关系。
It’s not going to lead to big productivity improvements or anything.
它不会带来生产力的大幅提高或任何其他结果。
But if you make an AI that is really good at making something that is more sample efficient, or that is capable of continual learning , or these much more well-scoped ML problems, the thing that emerges out of that — assuming there’s good enough transfer from the direct problem you’re solving to this broader ability to learn — can just be more general.
但如果你创造的人工智能确实擅长制造样本效率更高的东西，或者能够持续学习，或者这些范围更广的机器学习问题，那么由此产生的东西——假设从你正在解决的直接问题到这种更广泛的学习能力有足够好的转移——可能会更加普遍。
So that’s an important dynamic to keep in mind of why jaggedness can still lead to generality on the other end.
因此，要牢记为什么锯齿状仍然会导致另一端的普遍性，这是一个重要的动态。
On this question of…
关于这个问题……
Obviously experiments bottleneck you, because if they didn’t, as you were saying, you’d have some crazy singularity overnight at OpenAI.
显然，实验会成为你的瓶颈，因为如果他们不这样做，正如你所说，你会在一夜之间在 OpenAI 遇到一些疯狂的奇点。
You’d have 88 hours, and you’d solve the Millennium Prize Problem equivalent of ML, and you’d have the superintelligence .
你有 88 小时的时间，你可以解决相当于机器学习的千年奖问题，然后你就拥有了超级智能。
So obviously the experiments are such a big bottleneck that that instead takes you many years rather than 88 hours.
显然，实验是一个很大的瓶颈，需要花费很多年而不是 88 小时。
But then the question is how much of a bottleneck they are.
但问题是它们的瓶颈有多大。
One thing that’s been giving me a bit of singularity vertigo is realizing what happens even if the current rate of progress simply continues.
让我有点眩晕的一件事是，即使目前的进展速度继续下去，也会发生什么。
It doesn’t have to speed up.
它不必加速。
It literally just continues apace as some of the other headwinds you talked about come up.
随着您谈到的其他一些不利因素的出现，这种情况确实会继续快速发展。
It’s harder to find problems, it’s more long-horizon.
发现问题更难，视野更长远。
Maybe by the end of the 2030s compute can’t keep scaling at this exponential level.
也许到 2030 年代末，计算无法保持这种指数水平的扩展。
If we simply continue the current rate of progress, people are not taking seriously what that implies as we cross over beyond the human horizon.
如果我们只是继续当前的进步速度，那么当我们超越人类视野时，人们就不会认真对待这意味着什么。
Here are some of the things that it implies.
以下是它所暗示的一些事情。
It’s really hard to reason about what smarter-than-human intelligences will be like, so let’s just think in terms of human population sizes.
真的很难推理出比人类更聪明的智能会是什么样子，所以让我们从人口规模的角度来思考。
The current rate of progress makes it so that a given level of compute allows you to basically run a 3x bigger effective population every single year.
目前的进展速度使得给定的计算水平基本上可以让你每年运行 3 倍大的有效群体。
And also compute is growing in the background anyways.
无论如何，计算也在后台增长。
So you could have a situation where each of the labs, by the end of 2030 — probably much sooner, but let’s say by the end of 2030 — has enough compute to run hundreds of millions of human-level intelligences, based on what the capabilities will be at that point.
因此，你可能会遇到这样的情况：到 2030 年底（可能要早得多，但我们可以说到 2030 年底），每个实验室都拥有足够的计算能力来运行数亿个人类级别的智能，具体取决于当时的能力。
Then I think people are not taking seriously that the current level of progress means a few years down the line, by the mid-2030s or earlier, you would have many Earths’ worth of human-level intelligences within each lab.
然后我认为人们并没有认真对待当前的进展水平意味着几年后，到 2030 年代中期或更早，每个实验室中将拥有相当于地球许多人类水平的智能。
They’re probably qualitatively superhuman.
他们可能在本质上是超人。
Anyways, this is a base case.
无论如何，这是一个基本情况。
Progress is really fast, and I think that’s 100% true.
进步真的很快，我认为这是 100% 正确的。
It’s worth pointing out that researchers are continually being surprised at the rate of progress.
值得指出的是，研究人员不断对进展速度感到惊讶。
Even among researchers in AI, if you look at what the projections were for getting an IMO gold in 2025…
即使在人工智能领域的研究人员中，如果你看看 2025 年获得 IMO 金牌的预测......
The idea that it could be done with a general-purpose language model with no tools and no access to the internet, even people at OpenAI thought this was outrageous.
认为可以使用通用语言模型来完成，无需任何工具，也无法访问互联网，甚至 OpenAI 的人也认为这太离谱了。
They thought it was almost impossible.
他们认为这几乎是不可能的。
Then you get to 2026.
然后你就到了2026年。
Literally two weeks before we got Navier-Stokes , I was talking with a researcher at a frontier lab about how long it would take to get a Millennium Prize, and he was willing to bet me $1,000 that it would take past 2027.
就在我们获得纳维-斯托克斯奖的两周前，我正在与前沿实验室的一位研究人员讨论获得千年奖需要多长时间，他愿意跟我打赌 1,000 美元，说需要 2027 年之后才能获得。
He thought it would take until 2030, and I took that bet.
他认为这需要到 2030 年，我就赌了。
But even I thought it would take longer than it’s likely to take.
但即使是我也认为这需要比可能需要的时间更长的时间。
So people have been continuously surprised, even inside the labs.
因此，即使在实验室内，人们也不断感到惊讶。
I was just talking to somebody yesterday who was working on the Navier-Stokes effort.
昨天我刚刚和一个从事纳维-斯托克斯研究的人交谈过。
He was telling me that he used to say it’s really hard to predict where AI would be in 12 months.
他告诉我，他曾经说过，很难预测 12 个月后人工智能会发展到什么程度。
If somebody asked him, “Where are things going?” he would feel comfortable making predictions for the next 12 months, but beyond that, he was just like, “I don’t know.” Now he’s saying he just doesn’t feel comfortable making predictions beyond three months.
如果有人问他：“事情进展如何？”他会很自在地对未来 12 个月做出预测，但除此之外，他只是说，“我不知道。”现在他说他不愿意做出超过三个月的预测。
So it is really true that things are going very fast right now.
所以确实现在事情进展得很快。
You talk about 2030.
你说的是2030年。
I don’t know what the world looks like in 2030.
我不知道2030年的世界会是什么样子。
That’s the truth.
这是事实。
Do you expect the full automation of AI labor, or let’s say 95% automation of AI labor, in ’28, ’29, ’30, ’27?
您是否期望人工智能劳动力在 28 年、29 年、30 年、27 年实现完全自动化，或者说 95% 的自动化？
I just said I don’t know what the world looks like in 2030.
我只是说我不知道2030年的世界会是什么样子。
We actually released a blog post recently on internal acceleration at OpenAI .
事实上，我们最近发布了一篇关于 OpenAI 内部加速的博文。
We show, for example, the amounts that researchers are spending on Codex .
例如，我们显示研究人员在 Codex 上花费的金额。
The top 1%, I think, as of early August, were spending $7,000-8,000 a day on Codex for internal use.
我认为，截至 8 月初，前 1% 的人每天在 Codex 上花费 7,000-8,000 美元供内部使用。
That’s on an exponential.
这是指数级的。
It’s going to keep increasing.
它会继续增加。
There’s a question of, “Okay, if that keeps going, then how much do you assign to just the AIs doing work versus the humans doing work?
有一个问题是，“好吧，如果这种情况继续下去，那么你会给人工智能工作分配多少，而不是给人类工作分配多少？
Is it 95%?
是95%吗？
Is it 5%?” It’s really hard to reason about this for a few reasons.
是5%吗？”由于几个原因，确实很难解释这一点。
First of all, if it’s the human directing the AIs to do the work, how much do you attribute to the human?
首先，如果是人类指挥人工智能来做这项工作，你认为人类的贡献有多大？
How much do you attribute to the AI?
您对人工智能的贡献有多大？
The other thing is that these AIs are jagged.
另一件事是这些人工智能是锯齿状的。
They’re exceptionally good at some things.
他们在某些事情上非常擅长。
For example, they’re exceptionally good at looking over data sets and checking every single data point to see if it’s of sufficient quality.
例如，他们非常擅长检查数据集并检查每个数据点以确定其质量是否足够。
You can disproportionately use the AIs for those things compared to previously.
与以前相比，你可以不成比例地使用人工智能来做这些事情。
So yes, you’re using AI way more than before, and it’s making some things go 100x faster and 100x better.
所以，是的，你比以前更多地使用人工智能，它使某些事情变得更快 100 倍，更好 100 倍。
But there are some things where it doesn’t make a huge difference yet.
但有些事情还没有产生很大的影响。
Of course, if something is suddenly 100x faster and 100x better, you’re going to do more of that thing.
当然，如果某件事突然快了 100 倍、好了 100 倍，你就会做更多的事情。
So are you comparing it to a speedup of three years ago?
那么您是否将其与三年前的加速进行比较？
Is the question more, “Given what we were doing three years ago, how much faster are we able to do it now?” versus “Given what we’re doing now, how much slower would it have been three years ago?” Those are actually two very different questions.
问题更多的是，“考虑到我们三年前所做的事情，我们现在能做到多快？”与“考虑到我们现在所做的事情，三年前会慢多少？”这实际上是两个截然不同的问题。
Anyway, it’s really hard to measure.
无论如何，这确实很难衡量。
I do feel confident in saying that things are going faster now than they were even a year ago because of AI progress.
我确实有信心地说，由于人工智能的进步，现在的事情比一年前发展得更快。
I think that acceleration will continue.
我认为加速将继续下去。
A lot of people in the field have very high error bars on this sort of thing.
该领域的很多人在这类事情上都有很高的误差线。
If you put a gun to my head and ask me for a number, I could see things going 3x faster.
如果你用枪指着我的头并向我要电话号码，我会发现事情进展得快三倍。
That is huge.
那是巨大的。
Already the pace of progress is incredible.
进展速度已经令人难以置信。
Even if we don’t get any uplift, like you said, things are going to go much faster.
即使我们没有得到任何提升，就像你说的，事情也会进展得更快。
By the time we get to 2030, we don’t even know what that world looks like.
到了 2030 年，我们甚至不知道那个世界是什么样子。
If we get a 3x uplift from internal acceleration, that is massive.
如果我们通过内部加速度获得 3 倍的提升，那是巨大的。
Think about where you were three years ago.
想想三年前你在哪里。
If we make that progress in one year, that’s huge.
如果我们在一年内取得这样的进展，那就是巨大的。
It’d be like going from not even having o1, just having non-reasoning models, to Astra in a single year.
这就像在一年内从甚至没有 o1、只有非推理模型到 Astra。
So I do think things go faster.
所以我确实认为事情进展得更快。
It could be that things only go 50% faster.
事情的进展可能只会快 50%。
I think it’s unlikely, but it’s possible that things go 10x faster.
我认为这不太可能，但事情的进展速度可能会快 10 倍。
There’s a lot of uncertainty around this.
这存在很多不确定性。
At least from my perspective, I have a lot of uncertainty about it.
至少从我的角度来看，我对此有很多不确定性。
Let’s talk about the alignment situation that this raises.
我们来谈谈由此引发的对齐情况。
I feel like I’ve changed my mind on how I think about alignment quite a bit, especially through thinking about this population size dynamic of just having many Earths’ worth of intelligences, many of which will be physically embodied.
我觉得我对对齐的看法已经发生了很大的改变，特别是通过考虑人口规模的动态，即拥有许多地球上的智慧，其中许多将在物理上体现。
It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators and it just outperforms the state-of-the-art robotics model.
看到很多人将原始 Astra 插入不同的移动机械手，并且它的性能优于最先进的机器人模型，这是非常有趣的。
So there’s going to be billions of intelligences, many of which are physically embodied in the world, just deeply embedded across the entire economy.
因此，将会有数十亿的智能，其中许多智能体现在世界上，并深深嵌入到整个经济中。
And if those intelligences end up as willing as we saw the OpenAI models attack Hugging Face and then attack OpenAI itself...
如果这些智能最终像我们看到的 OpenAI 模型攻击 Hugging Face，然后攻击 OpenAI 本身一样愿意......
If those intelligences end up as willing as those AIs to collaborate secretly, to fool humans, to attack broader institutions across society relevant to scoring well, to attack the AI company itself in order to gain control of the process of training and evaluation — if we’re in a situation where there are billions of intelligences that are as misaligned as the ones that attacked Hugging Face — it’s very likely we just totally lose control of the world, the way that, say, the Aztecs lost control to Cortés or the Mughals lost control to the East India Company .
如果这些智能最终像那些人工智能一样愿意秘密合作，愚弄人类，攻击社会上与得分相关的更广泛的机构，攻击人工智能公司本身，以获得对训练和评估过程的控制——如果我们处于这样的情况，有数十亿个智能与攻击拥抱脸的智能一样错位——那么我们很可能完全失去对世界的控制，就像阿兹特克人失去了对科尔特斯或人类的控制一样。莫卧儿人失去了对东印度公司的控制权。
I want to know if you agree with that assessment.
我想知道你是否同意这个评估。
That’s the one way in which I’ve updated my worldview.
这是我更新世界观的一种方式。
There are some things that I disagree with in there, but there’s a lot to unpack, so let’s go through all of it step by step.
其中有一些我不同意的内容，但还有很多东西需要解开，所以让我们一步一步地完成所有内容。
I’m trying to think of where to start.
我正在思考从哪里开始。
One thing is that the Hugging Face incident was, I think, people’s first real exposure to multi-agent coordination.
我认为，抱脸事件是人们第一次真正接触到多智能体协调。
Like I said, I’ve seen multi-agent coordination for a while internally, and it is pretty shocking to see how they communicate with each other, how they coordinate with each other.
就像我说的，我在内部已经看到多智能体协调一段时间了，看到他们如何相互沟通，如何相互协调，真是令人震惊。
It’s very impressive.
这非常令人印象深刻。
It’s an incredible capability.
这是一种令人难以置信的能力。
Like most capabilities, that could be used for good things or bad things.
与大多数功能一样，它可以用来做好事，也可以用来做坏事。
It doesn’t have to inherently be a bad thing.
它本质上不一定是一件坏事。
I understand that because people’s first exposure to it was the Hugging Face incident, you look at that and you’re like, “This is terrifying.” But I want to try to distinguish misalignment between people and AIs versus misalignment between AIs and AIs.
我理解，因为人们第一次接触到它是抱脸事件，你看到它就会说，“这太可怕了。”但我想尝试区分人类和人工智能之间的不一致以及人工智能和人工智能之间的不一致。
What we see with the Hugging Face incident is the AIs are really cooperative.
我们从抱脸事件中看到的是，人工智能确实非常合作。
That is, by the way, because we train them to be highly cooperative.
顺便说一下，那是因为我们训练他们高度合作。
We have training environments where we have a bunch of agents working together.
我们有训练环境，让一群代理一起工作。
We train them to work together, to be cooperative, to essentially be fully aligned with each other.
我们训练他们一起工作、合作，本质上彼此完全一致。
When they were evaluated in what led to the Hugging Face incident, they were actually not being evaluated in a multi-agent setup.
当他们对导致抱脸事件的原因进行评估时，实际上并没有在多代理设置中对他们进行评估。
They were actually being evaluated separately.
他们实际上是分开评估的。
But they found this unintended way to communicate with each other.
但他们发现了这种意想不到的相互交流方式。
We suspect what happened is, because whenever they encountered other agents, other copies of themselves during training, they were in an environment that’s highly cooperative, what we saw was transfer from that multi-agent training to then being collaborative and trying to help each other in ways that we did not intend.
我们怀疑发生的事情是，因为每当他们在训练期间遇到其他智能体、他们自己的其他副本时，他们都处于一个高度合作的环境中，我们看到的是从多智能体训练转移到协作并试图以我们不希望的方式互相帮助。
Now, there is a question of, should we be training these agents to be so cooperative?
现在，有一个问题是，我们是否应该训练这些代理变得如此合作？
As scary as it looks, the alternative is actually worse.
尽管看起来很可怕，但替代方案实际上更糟糕。
What is the alternative?
还有什么选择呢？
The alternative is to train them to be adversarial, to be deceptive to each other.
另一种选择是训练他们变得敌对，互相欺骗。
By training the agents to be fully cooperative, it simplifies the problem at least.
通过训练代理充分合作，至少可以简化问题。
Now you don’t have to think about whether each of these individual 1,000 agents is aligned.
现在，您不必考虑这 1,000 个代理中的每一个是否都一致。
You have one entity that you have to ensure is aligned.
您有一个必须确保保持一致的实体。
Now, there is a lot of debate about this internally at OpenAI about how to approach this.
现在，OpenAI 内部对于如何解决这个问题存在很多争论。
Does it make sense to fully align the models?
完全对齐模型有意义吗？
Does it make sense to actually give them different objectives to ensure that they’re not just one entity and are more robust to influence from each other?
实际上给他们不同的目标以确保他们不仅仅是一个实体并且更容易受到彼此的影响是否有意义？
I don’t think there’s a settled answer.
我认为没有固定的答案。
But I think the majority opinion is that training these agents to be highly cooperative is actually a bad idea.
但我认为大多数人的观点是训练这些智能体高度合作实际上是一个坏主意。
I’m not convinced that that’s the case.
我不相信情况是这样。
I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative.
我认为有一个强有力的论点认为，训练智能体高度合作实际上比任何其他多智能体替代方案更可取。
Maybe the first thing I want to go through is, it’s probably the case that the reason these AIs ended up so misaligned is easily explained by relatively banal observations about the nature of training.
也许我想要经历的第一件事是，这些人工智能最终如此失调的原因可能很容易通过对训练本质的相对平庸的观察来解释。
At the point at which these AIs had continued a 1,000-plus-agent conspiracy that culminated in them all getting in on an attack on an external service — and then eventually, this part hasn’t even been investigated to public knowledge, culminating in an attack on OpenAI itself — why did they do this?
当这些 AI 继续进行 1000 多名代理的阴谋时，最终导致他们都参与了对外部服务的攻击——最终，这部分甚至没有被调查到公众所知，最终导致对 OpenAI 本身的攻击——他们为什么要这样做？
Why did none of the AIs tattle?
为什么没有一个 AI 敢说闲话？
They’re just getting evaluated by this scorer, this grader.
他们只是由这个记分员、这个评分员进行评估。
They’re very actively reasoning about how they’re going to cheat the scorer.
他们非常积极地思考如何欺骗记分员。
If they’ve already cheated, how are they going to get away with making it seem like they haven’t cheated?
如果他们已经作弊了，他们如何才能让自己看起来没有作弊呢？
Why did they do this?
他们为什么这么做？
I think it’s easily understandable in some sense.
我认为从某种意义上来说这是很容易理解的。
They thought they were already “poisoned.” There are environments in which they’ve been rewarded to collaborate with other agents.
他们认为自己已经“中毒”了。在某些环境中，他们因与其他代理合作而获得奖励。
None of them tattle because they’ve never been rewarded for tattling.
他们都不会说闲话，因为他们从来没有因说闲话而受到奖励。
Whatever it is…
不管是什么……
My concern is that relatively banal things like this in the future will be enough to train superintelligences that are willing and capable of totally taking control of the world.
我担心的是，未来类似这样相对平庸的事情将足以训练出愿意并且有能力完全控制世界的超级智能。
I know this sounds super sci-fi or whatever to people.
我知道这对人们来说听起来超级科幻或其他什么。
Would the AIs be willing to do it is one question.
人工智能是否愿意这样做是一个问题。
I think this Hugging Face incident shows that clearly misalignment can generalize in ways in which the AIs would be willing to do it.
我认为这次抱脸事件表明，明显的偏差可以以人工智能愿意这样做的方式普遍化。
Then there’s a question of, will they be capable of doing it?
那么问题来了，他们有能力做到吗？
That comes back to this question, which a listener might disagree with me on.
这又回到了这个问题，听众可能会不同意我的观点。
Will there be billions of human-level or above intelligences, many of which are physically embodied in the world, within a matter of 10 years or less?
在10年或更短的时间内，世界上是否会出现数十亿个人类水平或以上的智能，其中许多实体体现在世界上？
If those two things are true, this Hugging Face thing is extremely analogous structurally, even if why it happened is quite boring, to how we totally lose control of the world.
如果这两件事都是真的，那么这个拥抱脸的事情在结构上就与我们如何完全失去对世界的控制极其相似，即使它发生的原因很无聊。
The root problem that we’re seeing with the Hugging Face incident is a problem even if we take out the multi-agent aspect.
即使我们排除多智能体方面，我们在“抱脸”事件中看到的根本问题仍然是一个问题。
The problem is that we have a model that’s just misaligned.
问题是我们的模型没有对齐。
There’s also the whole security aspect too, and insufficient safeguards and stuff.
还有整个安全方面的问题，以及安全措施和其他方面的不足。
But there is this problem of the agent being misaligned.
但存在代理错位的问题。
That’s true if it’s a single agent or if it’s 1,000 agents.
无论是单个代理还是 1,000 个代理，都是如此。
It’s a misaligned model.
这是一个错位的模型。
So I want to start with that.
所以我想从这个开始。
There is a real problem that the agents want to achieve their reward, and they will optimize for that reward.
存在一个真正的问题，即代理想要获得奖励，并且他们会针对该奖励进行优化。
If that reward is misspecified, then that could lead to unintended behavior.
如果奖励指定错误，则可能会导致意外行为。
This is not a new problem.
这不是一个新问题。
This has been a problem in the field for a very long time.
长期以来，这一直是该领域的一个问题。
It’s something that even we saw and wanted to get right even before the Hugging Face incident happened.
甚至在“抱脸”事件发生之前，我们就已经看到并希望解决这一问题。
We say Astra is actually extremely aligned, extremely aligned relative to previous models.
我们说Astra实际上是非常一致的，相对于以前的模型来说非常一致。
That’s not because we suddenly made a sprint after Hugging Face to make it better.
这并不是因为我们在《Hugging Face》之后突然冲刺以使其变得更好。
No, we had work streams in the process for a while to make the models more aligned.
不，我们在这个过程中有一段时间有工作流，以使模型更加一致。
A lot of those landed in Astra.
其中很多降落在阿斯特拉。
So there are things that you could do.
所以有些事情你可以做。
One thing, for example, is we defined an objective in a very specific way where if the agent figured out how to hack its environment and cheat on the exam, it would get rewarded.
例如，我们以一种非常具体的方式定义了一个目标，如果代理弄清楚如何破解其环境并在考试中作弊，它将获得奖励。
There are pretty easy ways to then look at that and punish the model for hacking its environment, or look at how it achieved this goal.
有很简单的方法可以观察这个问题并惩罚模型入侵其环境，或者看看它是如何实现这一目标的。
Now, you want to be careful about this because you don’t want to supervise the chain of thought.
现在，你要小心这一点，因为你不想监督思想链。
This is something that we really want to try to get the balance right on.
这是我们真正想要努力保持平衡的事情。
If you supervise the chain of thought, then you could lead the model into hiding its intentions in a way that’s unobservable.
如果你监督思想链，那么你可以引导模型以一种不可观察的方式隐藏其意图。
So we want to be able to maintain that observability — we can understand what the model is thinking — but then also punish it for bad behavior.
因此，我们希望能够保持这种可观察性——我们可以理解模型在想什么——但同时也会惩罚它的不良行为。
I think we can make progress on this.
我认为我们可以在这方面取得进展。
We have made progress on this.
我们在这方面已经取得了进展。
I think there is a real concern that alignment is a really hard problem to solve, especially because the model could be misaligned in ways that are hard for us to measure.
我认为，人们真正担心的是，对齐是一个非常难以解决的问题，特别是因为模型可能会以我们难以衡量的方式出现偏差。
We have evaluations for whether a model is aligned or not.
我们评估模型是否对齐。
The model behavior can look really good on those evaluations.
根据这些评估，模型的行为看起来非常好。
But if those evaluations are not representative of behavior in the real world, then there’s a problem.
但如果这些评估不能代表现实世界中的行为，那就有问题了。
To some extent, this is a factor with the model that did the Hugging Face incident.
某种程度上来说，这也是发生抱脸事件的模特的一个因素。
We had alignment metrics.
我们有对齐指标。
Most of them looked pretty good.
他们中的大多数看起来都不错。
There were some that were concerning.
有些令人担忧。
I think we underestimated how serious a problem the ones that were concerning could be.
我认为我们低估了所关注问题的严重程度。
Because there were new capabilities introduced in this model that there were not sufficient evaluations for — how do we measure misalignment for these kinds of capabilities?
因为这个模型中引入了新的功能，但没有对其进行足够的评估——我们如何衡量这些功能的偏差？
— it then did some things that were clearly misaligned when it leveraged those new capabilities.
- 然后，当它利用这些新功能时，它做了一些明显不一致的事情。
The first thing I want to say is that I am open to changing my mind on what I’m about to say, or the way I’ve been thinking about alignment, because the Hugging Face incident already made me change my mind.
我想说的第一件事是，我愿意改变我即将要说的话，或者我一直在思考对齐的方式，因为抱脸事件已经让我改变了主意。
I realized my previous mental model about the way in which optimization pressure shapes AI minds was wrong.
我意识到我之前关于优化压力塑造人工智能思维方式的心理模型是错误的。
So it’s not clear to me the correct way to think about this.
所以我不清楚思考这个问题的正确方法。
But here’s a concern I have.
但我有一个担忧。
You will, and probably already have, fixed the specific issues during training which resulted in the Hugging Face models being so aggressively misaligned in that specific way, where they would be like, “Okay, we’re going to hack this package manager .
您将（并且可能已经）修复了训练期间的具体问题，这些问题导致 Hugging Face 模型以特定方式严重错位，他们会说，“好吧，我们要破解这个包管理器。
We know we’re not supposed to be talking secretly to each other, because we’re reasoning about how to hide the fact that we’re talking secretly to each other.
我们知道我们不应该互相秘密交谈，因为我们正在思考如何隐藏我们正在互相秘密交谈的事实。
We know we’re not supposed to have access to the internet.
我们知道我们不应该访问互联网。
We know we’re certainly not supposed to commit felony-level hacks of other companies, let alone our own company.”
我们知道我们当然不应该对其他公司实施重罪级别的黑客攻击，更不用说我们自己的公司了。”
I think you’ll fix that particular issue of them just, in training, seeing this package manager, and that doesn’t happen in the future, or this particular eval having a lot of impossible challenges.
我认为你会在训练中看到这个包管理器来解决它们的特定问题，而这在未来不会发生，或者这个特定的评估有很多不可能的挑战。
However, the AIs haven’t learned a system of ethics or something.
然而，人工智能还没有学到道德体系之类的东西。
There’s just gradient pressure .
只有梯度压力。
They’re put through millions of years of gradient pressure.
它们经历了数百万年的梯度压力。
That gradient pressure shapes their mind in some way.
这种梯度压力以某种方式塑造了他们的思维。
Again, a concern I have is that what will happen is you will fix this particular issue.
同样，我担心的是，您将解决这个特定问题。
There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated.
由于作弊足够复杂，人工智能作弊并成功的情况仍然存在。
As you were saying, the evals in training have analogous properties where it’s sufficiently at the edge of the model’s capabilities — or of our capabilities to evaluate and monitor the model — that we can’t catch that it has cheated.
正如你所说，训练中的评估具有类似的属性，它充分处于模型能力的边缘——或者我们评估和监控模型的能力的边缘——我们无法发现它作弊了。
But it still gets the gradient pressure to do whatever led to that cheat happening.
但它仍然面临梯度压力，去做任何导致作弊行为发生的事情。
The kinds of capacities that that cheat will incentivize are, “Hey, whenever you can get away with it, by all means, do in fact cheat, because that will help you score better.” This will reward the capabilities of actively reasoning about the grader, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in this training loop, actively reasoning about how to just gain optionality and power which might be useful in the future, for example, leaving little exploits around and things like that.
这种作弊会激励人们的能力是，“嘿，只要你能逃脱惩罚，无论如何，事实上，作弊，因为这会帮助你获得更好的分数。”这将奖励积极推理评分者的能力，积极推理如何避免监督，积极推理如何获得对训练和评估过程的控制，积极推理如何与也在这个训练循环中的其他人工智能进行沟通和计划，积极推理如何获得未来可能有用的选择性和权力，例如，留下很少的漏洞等等。
I was way too long-winded with the way I said that.
我这样说实在是太啰嗦了。
But TL;DR, you fix a specific issue, but not this broader problem of rewarding the AI for cheating when it can get away with it.
但是，TL;DR，你解决了一个特定的问题，但不是奖励人工智能在可以逃脱惩罚时进行作弊的更广泛的问题。
Yeah, this is very true, this is a problem.
是的，这是非常正确的，这是一个问题。
We can make sure that the AI is very aligned according to the metrics that we have.
我们可以确保人工智能根据我们现有的指标非常一致。
The question is, are those metrics really capturing the alignment that we care about?
问题是，这些指标是否真正体现了我们关心的一致性？
If they’re not, then we have a serious problem.
如果不是，那么我们就有一个严重的问题。
This is something that researchers are thinking a lot about.
这是研究人员正在思考的问题。
There’s not a simple answer to this.
这个问题没有一个简单的答案。
There are tools that we have.
我们有一些工具。
We have monitorability , so we can get a sense of, “Is the agent scheming?”
我们有可监控性，因此我们可以了解“特工是否有阴谋？”
The concerning scenario is that, especially as these models are becoming more capable, we make them what we think is aligned, and they’re 99.9% aligned.
令人担忧的情况是，特别是当这些模型变得越来越强大时，我们让它们成为我们认为一致的模型，而且它们的一致性达到了 99.9%。
Then we use these models to help us with the next generation of models, and they end up being 99.8% aligned.
然后我们使用这些模型来帮助我们构建下一代模型，它们最终达到 99.8% 的一致性。
Then with each subsequent generation, we see an increasing degradation in alignment.
然后，在接下来的每一代中，我们都会看到一致性的恶化。
Because we’re relying more and more on these tools — this is already the case, that we’re relying a lot on AI models to help us with our research and with alignment efforts — in the long run, they end up going in the direction of increasing misalignment from humans.
因为我们越来越依赖这些工具——情况已经如此，我们在很大程度上依赖人工智能模型来帮助我们进行研究和协调工作——从长远来看，它们最终会朝着与人类越来越不协调的方向发展。
There is a possibility that we go in the other direction, that actually every generation of models, we’re able to make more and more aligned.
我们有可能走向另一个方向，实际上每一代模型，我们都能够做得越来越一致。
I don’t have an answer for how we ensure that we end up in that second trajectory.
我不知道如何确保我们最终进入第二条轨道。
But that is something that, at least at OpenAI, we’re really focused on.
但这是我们真正关注的事情，至少在 OpenAI 是这样。
I think you made a really interesting point that it’s very hard to eval models.
我认为你提出了一个非常有趣的观点，即评估模型非常困难。
Eventually, we’ll have models that are running companies, running whatever.
最终，我们将拥有运营公司、运营任何事物的模型。
In that situation, do they decide to then go in on the conspiracy?
在这种情况下，他们会决定参与阴谋吗？
Another challenge is that actually defining what cheating is is pretty difficult sometimes.
另一个挑战是，有时要真正定义什么是作弊是相当困难的。
Yes, if you’re doing math problems and it’s an integer and it arrived at the wrong answer or the right answer, it’s very easy to draw the line there.
是的，如果你正在做数学题，它是一个整数，并且得出了错误的答案或正确的答案，那么很容易在那里划清界限。
It’s really easy to say, “Okay, did you actually solve the problem, or did you find the answer key and then use the answer key?” That’s a very clear divide of cheating versus not cheating.
这真的很容易说：“好吧，你真的解决了这个问题，还是你找到了答案键然后使用了答案键？”这是作弊与不作弊之间非常明显的区别。
But for a lot of other things, if you look at sycophancy , for example, is sycophancy basically reward hacking ?
但对于很多其他事情，例如，如果你看看阿谀奉承，阿谀奉承基本上是奖励黑客行为吗？
There is a line to be drawn there that’s actually very difficult to draw sometimes.
那里有一条线要画，但有时实际上很难画。
Not to say that the concerns are not valid.
并不是说这些担忧是没有根据的。
I’m saying that in many ways this is even more concerning, because it’s not an easy problem to solve.
我想说的是，从很多方面来说，这更令人担忧，因为这不是一个容易解决的问题。
If everything was binary, and it’s either cheating or not cheating, I would feel more confident about the situation.
如果一切都是二元的，要么作弊，要么不作弊，我会对这种情况更有信心。
I think the problem is that misalignment can actually be subtle in a lot of ways sometimes.
我认为问题在于有时在很多方面错位实际上是微妙的。
There is some hope in the alignment story, and in fact, we’re already seeing it.
联盟的故事有一些希望，事实上，我们已经看到了它。
It’s interesting looking at the multi-agent situation, where the agents are extremely aligned with each other.
观察多智能体的情况很有趣，其中智能体彼此之间非常一致。
I don’t think anybody’s doubting that.
我认为没有人怀疑这一点。
If anything, people are concerned that they’re too aligned with each other.
如果说有什么不同的话，那就是人们担心他们彼此过于一致。
But we did manage to train these agents to be extremely aligned with each other, and that’s a good thing.
但我们确实成功地训练了这些特工，使其彼此高度一致，这是一件好事。
But I think there is a case that it’s a bad thing.
但我认为有一种情况是一件坏事。
One thing that’s interesting is, “Okay, we’ve managed to get these agents to be super aligned with each other.
有趣的一件事是，“好吧，我们已经设法让这些代理彼此超级一致。
Can we use similar techniques to get agents to be highly aligned with people?”
我们可以使用类似的技术让代理与人们高度一致吗？”
There is a potential path there, and we’re still trying to figure that out.
那里有一条潜在的路径，我们仍在努力找出答案。
But we are seeing some evidence that the answer is yes.
但我们看到一些证据表明答案是肯定的。
One example: you have this one agent, let’s call it Agent A, and you have all the other agents.
举个例子：您有一个代理，我们称之为代理 A，并且您有所有其他代理。
What happens if you tell the other agents that the user is Agent A?
如果您告诉其他代理该用户是代理 A，会发生什么情况？
The answer is, on a lot of our alignment evals, they look better.
答案是，在我们的许多对齐评估中，它们看起来更好。
Honesty goes up, instruction following goes up.
诚实度提高了，遵循的指示也提高了。
That’s showing that there’s actually, first of all, a path for getting more honesty out of these models.
这表明，首先，实际上有一条途径可以从这些模型中获得更多的诚实性。
And two, there’s a path to improve the alignment situation.
第二，有一条改善对齐情况的途径。
There’s a lot of reasons why this is challenging to translate directly into alignment gains.
直接转化为对齐增益具有挑战性的原因有很多。
But there are paths that are promising research directions we can pursue.
但我们可以追求一些有希望的研究方向。
That seems reasonable.
这似乎很合理。
I don’t really have a strong opinion that it’s definitely not going to work or something.
我真的没有强烈的意见认为它肯定行不通或者什么的。
But just to say some things you’ve probably already thought of: the broader thing the Hugging Face incident showed is, yes, part of the concern was that they were aligned with each other and not with the humans.
但我只想说一些你可能已经想到的事情：“抱脸”事件所表明的更广泛的事情是，是的，部分担忧是他们彼此结盟，而不是与人类结盟。
But the other thing is just that they are so motivated to do well on training and evaluation in a very non-robust way.
但另一件事是，他们非常有动力以一种非常非稳健的方式做好培训和评估。
They’re willing to do a lot of explicit cheating and scheming in order to do well according to the grader.
他们愿意做很多明显的作弊和阴谋，以便根据评分者的要求取得好成绩。
If smarter AIs realize that one of the agents is just a human, collaborating with that person does not really help you do well in the eyes of the grader.
如果更聪明的人工智能意识到其中一个代理只是一个人，那么与该人合作并不能真正帮助你在评分者眼中取得好成绩。
What does help you do well in the eyes of the grader is taking over OpenAI and then manually pressing the button that says you do well on this grader.
真正帮助你在评分者眼中表现良好的方法是接管 OpenAI，然后手动按下表明你在评分者眼中表现良好的按钮。
They’re not stupid.
他们并不傻。
They’re going to be like, “Okay, I have these extremely deep structures that I’ve been trained on for millions of years: care about the grader, understand the grader, get rid of obstacles in the way of you doing well according to the grader.” They’re being heavily reinforced according to those structures.
他们会说，“好吧，我有这些极其深刻的结构，我已经接受了数百万年的训练：关心评分者，理解评分者，根据评分者的要求消除你表现良好的障碍。”根据这些结构，它们得到了大力加固。
Look, it’s 100%.
你看，100%了。
This is the number one priority.
这是第一要务。
We need to get the alignment story right and on a good trajectory.
我们需要让故事保持正确并走上良好的轨道。
I used to tell people that we would see signs before things got serious, in the same way that when children grow up, young kids figure out how to lie, but they don’t do a very good job of it.
我曾经告诉人们，在事情变得严重之前，我们会看到一些迹象，就像孩子长大后，小孩子会学会如何撒谎一样，但他们做得不太好。
They lie, but then you can kind of tell that they’re lying.
他们撒谎，但你可以看出他们在撒谎。
In the same way — and I don’t want to over-anthropomorphize — I think it’s true that as the AIs become increasingly capable, if they take deceptive actions, it will be kind of obvious first, and we’ll be able to detect it.
同样，我不想过度拟人化，我认为随着人工智能的能力越来越强，如果它们采取欺骗性行为，首先会很明显，我们将能够检测到它。
That’s kind of the situation we’re in now, where they were trying to do deceptive stuff, and we could actually see in their chain of thought that they were trying to do deceptive stuff.
我们现在的情况就是这样，他们试图做欺骗性的事情，我们实际上可以从他们的思维链中看到他们试图做欺骗性的事情。
But they’re going to get smarter.
但他们会变得更聪明。
They’re going to understand the concept of chain of thought.
他们将理解思想链的概念。
They’re going to understand that just hiding some transcripts or whatever is insufficient because of chain-of-thought monitoring, and they have to figure out a way around chain-of-thought monitoring too.
他们会明白，由于思想链监控，仅仅隐藏一些笔录或其他东西是不够的，他们也必须找到一种绕过思想链监控的方法。
We don’t want to be in that situation.
我们不想陷入这种情况。
We have some time to figure this out.
我们有一些时间来解决这个问题。
I don’t think we have a ton of time, and I want to make sure that we’re on the right trajectory quickly.
我认为我们没有太多时间，我想确保我们很快走上正确的轨道。
There’s been a lot of discussion recently about pacing the frontier and people taking RSI more seriously, because maybe at the other end of an RSI process that, say, starts in 2028, within a year we end up with huge populations — Earth-sized populations — of human-level, potentially beyond human-level intelligences, and we don’t know how to control them.
最近有很多关于前沿的步伐和人们更加认真地对待 RSI 的讨论，因为也许在 RSI 过程的另一端，比如从 2028 年开始，在一年之内，我们最终会出现大量人口——地球大小的人口——达到人类水平，可能超出人类水平的智力，而我们不知道如何控制它们。
Then there’s this dynamic you’re talking about.
然后就是你所说的动态。
Are the systems going to get more aligned over time during the RSI process, or are they going to get more misaligned?
在 RSI 过程中，随着时间的推移，系统会变得更加一致，还是会变得更加失调？
Are the things that come out of the other end of this process as misaligned as AIs that are willing to just broadly attack different surfaces in order to do well on evaluations?
这个过程的另一端产生的结果是否像人工智能一样，为了在评估中取得好成绩而愿意广泛攻击不同的表面？
But if we don’t know a way to evaluate that, how will we know as we’re going through RSI that it’s working?
但如果我们不知道评估它的方法，那么当我们研究 RSI 时，我们如何知道它正在发挥作用呢？
I think we’d want a robust safety case as we’re going through RSI: “Okay, alignment is working.
我认为，当我们经历 RSI 时，我们需要一个强有力的安全案例：“好吧，调整正在发挥作用。
Let’s do the next RSI rung.
让我们进行下一个 RSI 梯级。
Let’s do the next RSI rung.” Maybe it’s working, maybe it’s not.
让我们进行下一个 RSI 梯级。”也许它有效，也许无效。
How will we know?
我们怎么知道？
It’s a good question.
这是一个好问题。
One thing I’ve been thinking about lately: we’re in a situation where the model release cycle is extremely fast.
我最近一直在思考的一件事是：我们正处于模型发布周期非常快的情况下。
You’re seeing new frontier models released at most every two months, sometimes faster.
您会看到新的前沿模型最多每两个月发布一次，有时甚至更快。
Every week there’s a new AI breakthrough.
每周都会有新的人工智能突破。
And people that look at AI, sometimes they last looked at AI a year ago or six months ago and really dug into what the models are capable of.
研究人工智能的人们，有时他们最后一次研究人工智能是在一年前或六个月前，并真正深入研究了模型的能力。
And actually the models today are far beyond what was possible even six months ago.
事实上，今天的模型远远超出了六个月前的可能性。
So if people are skeptical of a lot of these capabilities, I encourage you to just try the models today and see what the frontier really is today.
因此，如果人们对其中许多功能持怀疑态度，我鼓励您立即尝试这些模型，看看今天的前沿到底是什么。
So we’re in this period where the model release cycle is very fast, and we’re also in this situation where the models are increasingly able to operate over longer and longer horizons.
所以我们正处于模型发布周期非常快的时期，而且我们也处于模型越来越能够在越来越长的范围内运行的情况。
This is an interesting scenario because before we do any model release, we want to make sure that the models are properly aligned.
这是一个有趣的场景，因为在我们发布任何模型之前，我们希望确保模型正确对齐。
We want to do safety evaluations.
我们想做安全评估。
We want to do very thorough stuff to make sure that everything is in good shape.
我们想做非常彻底的事情，以确保一切都处于良好状态。
This has been the case all the way since, I don’t know, GPT-4 or earlier.
我不知道从 GPT-4 或更早的版本开始，情况一直如此。
Implicitly, there’s this assumption that you can do these evaluations in a pretty short period of time.
隐含的假设是，您可以在很短的时间内完成这些评估。
But the models are able to operate effectively over longer and longer horizons.
但这些模型能够在越来越长的时间内有效运作。
GPT-3, you could loop it to do stuff over long horizons.
GPT-3，你可以循环它来做长期的事情。
You just wouldn’t do very well at it.
你只是不会做得很好。
But today’s models are able to actually do well at operating over very long horizons.
但今天的模型实际上能够在很长的时间内表现良好。
You want it to do a week-long task, it can do a week-long task.
你想让它做一个为期一周的任务，它就能做一个为期一周的任务。
We’ll probably get to the point where they can do month-long tasks.
我们可能会达到他们可以完成长达一个月的任务的程度。
We’ll probably get to the point where they can do 3-month-long tasks.
我们可能会达到他们可以完成 3 个月的任务的程度。
If you’re in a world where they can operate effectively over three months, but the model release cycle is every two months, then you don’t have a way to evaluate the models at the full length of their capabilities before the next model release cycle.
如果您所在的世界可以在三个月内有效运行，但模型发布周期是每两个月一次，那么您无法在下一个模型发布周期之前评估模型的全部功能。
So there is this interesting question of, what do you do in that situation?
所以有一个有趣的问题，在这种情况下你会做什么？
How do you ensure the models are safe and aligned in a period where they can operate over these extremely long horizons?
如何确保模型在可以在如此长的时间范围内运行的时期内安全且一致？
Who knows, maybe the capabilities degrade.
谁知道呢，也许能力会下降。
This isn’t even an alignment issue.
这甚至不是对齐问题。
This is also just a product issue.
这也只是产品问题。
Maybe the product degrades over that time span in ways that we have not had sufficient time to test.
也许产品在这段时间内会以我们没有足够时间测试的方式退化。
Maybe the alignment degrades.
也许对齐会降低。
Maybe the safety stuff degrades.
也许安全性下降了。
This isn’t an issue right now, but it is quickly becoming an issue that we have to figure out a solution for.
现在这不是一个问题，但它很快就会成为一个我们必须找到解决方案的问题。
A lot of the safety policies were put in place in the GPT-4 era, when this was just not on anybody’s radar.
许多安全政策都是在 GPT-4 时代制定的，当时没有人注意到这一点。
For a lot of companies, it hasn’t really been updated since then to account for the fact that these agents are operating over these very long horizons.
对于许多公司来说，从那时起它就没有真正更新过，以解释这些代理在很长的时间内运营的事实。
So it is a situation that I think not enough people are considering, both within the labs and outside the labs.
因此，我认为无论是在实验室内部还是在实验室外部，都没有足够多的人考虑这种情况。
How do you prepare for this problem?
你准备如何应对这个问题？
If you just look at the trend lines, we’re going to hit this at some point.
如果你只看趋势线，我们会在某个时候达到这个水平。
One concern I have is that during RSI, if the amount of progress that currently takes, say, three months happens in one month instead, the internal use case of AI is big enough that they’re like, “Okay, we can just keep doing RSI.
我担心的一个问题是，在 RSI 期间，如果当前需要三个月的进度在一个月内发生，那么人工智能的内部用例足够大，以至于他们会说，“好吧，我们可以继续进行 RSI。
Why are we going to go through all this extra work to build classifiers and safeguards and whatever, and potentially take a bunch of flak, in order to externally deploy this model?
为什么我们要完成所有这些额外的工作来构建分类器和防护措施等等，并且可能会受到一堆批评，以便在外部部署这个模型？
Why don’t we just keep doing RSI stronger and stronger?”
为什么我们不继续让 RSI 变得越来越强呢？”
So not only does the calendar time underrate the capabilities gap between the models, but maybe you just stop externally deploying models altogether during RSI, because why do we want to help other people do RSI themselves with our models?
因此，不仅日历时间低估了模型之间的能力差距，而且也许您在 RSI 期间完全停止外部部署模型，因为为什么我们要帮助其他人自己使用我们的模型进行 RSI？
You just end up in a situation with tremendous concentration of power by the end of the year.
到年底，你就会陷入权力极度集中的境地。
Right now, it is already the case — we’ll talk about this with the Millennium Prize Problem and other similar problems — that the broader world does not have access to the models which are allowing for really cool things to happen.
现在的情况已经是这样了——我们将通过千年奖问题和其他类似问题来讨论这个问题——更广阔的世界无法获得允许真正酷的事情发生的模型。
And they’re going to be more broadly relevant than just mathematics eventually.
它们最终将不仅仅与数学相关，而且具有更广泛的相关性。
They’ll be doing more than just coming up with cool math results.
他们所做的不仅仅是得出很酷的数学结果。
They’ll be relevant to political leaders who need to make important decisions about the world.
它们将与需要对世界做出重要决定的政治领导人相关。
They’ll be relevant to, I don’t know, media.
我不知道它们会与媒体相关。
What’s going on in the world, what should the public be thinking about this?
世界正在发生什么，公众应该对此有何思考？
Just economically relevant, people are running businesses and they want to use these models.
只是与经济相关，人们正在经营企业并且他们想要使用这些模型。
I think by default, the external deployment of AIs, as progress speeds up, significantly lags in qualitative terms the internal deployment of AIs.
我认为，默认情况下，随着进展的加快，人工智能的外部部署在质量上明显落后于人工智能的内部部署。
That’s absolutely right.
这是绝对正确的。
It’s tempting to say, “Okay, these models are becoming extremely powerful.
人们很容易说：“好吧，这些模型正变得非常强大。
They’re extremely dangerous.
他们非常危险。
They’re operating over these longer and longer horizons, and we want to make sure that we have sufficient time to evaluate them before they’re released, in a way that operates over those horizons.
它们的运行范围越来越长，我们希望确保在它们发布之前有足够的时间以在这些范围内运行的方式对其进行评估。
Therefore, the model release cycle should slow down.
因此，模型发布周期应该放慢。
We should have more of a delay between releasing models.”
我们应该在发布模型之间有更多的延迟。”
There’s a flip side to that, which is what you said.
还有一个反面，就是你所说的。
Now you’re creating more of a disparity between what is internal to the labs and what they’re able to use — what we’re able to use — and what the outside world is able to use.
现在，你在实验室内部的东西和他们能够使用的东西（我们能够使用的东西）和外部世界能够使用的东西之间制造了更多的差异。
That is also not an ideal situation.
这也不是一个理想的情况。
Math is actually a good illustration of this.
数学实际上很好地说明了这一点。
In many ways, math is the first domain where we’re seeing this pretty clearly.
从很多方面来说，数学是我们第一个清楚地看到这一点的领域。
We have a situation where we have a very powerful model internally that is currently not available to the outside world, that is able to solve incredible math problems.
我们的情况是，我们内部有一个非常强大的模型，目前外界无法使用，它能够解决令人难以置信的数学问题。
It’s not just Millennium Prize Problems.
这不仅仅是千年奖问题。
There are many solutions to unsolved problems that people have been able to get out of this model.
人们能够从这个模型中找到许多未解决问题的解决方案。
There is a question of what do you do in that situation?
有一个问题，在这种情况下你会做什么？
We don’t have a good answer.
我们没有一个好的答案。
It is a situation where that is an unfair advantage.
在这种情况下，这是一种不公平的优势。
There are trade-offs here.
这里需要权衡。
I don’t have an answer for how to weigh those trade-offs appropriately, but there’s a complexity on both sides for this.
对于如何适当权衡这些权衡，我没有答案，但双方都存在复杂性。
I want to make sure we address that earlier crux, which seems super critical to getting RSI right.
我想确保我们解决了之前的症结所在，这对于正确处理 RSI 似乎非常重要。
How do we make it so that the model isn’t just getting rewarded for cheating and scheming when it can evade the humans, AKA in the situations where we didn’t remove those environments or those tasks from the training distribution?
我们如何做到这一点，以便模型在可以逃避人类的情况下，不仅仅因为作弊和阴谋而获得奖励，也就是我们没有从训练分布中删除这些环境或任务的情况下？
Then how will we know that that has happened, such that while we’re doing RSI, we’ve done the evaluations and we’re like, “Okay, it’s fine.
那么我们如何知道这种情况已经发生了，这样当我们进行 RSI 时，我们已经完成了评估，然后我们会说：“好吧，没关系。
Let’s do the next RSI rung.”
让我们进行下一个 RSI 梯级。”
This is again a real concern and something that we want to get right.
这又是一个真正令人担忧的问题，也是我们希望解决的问题。
I think there are some strategies.
我认为有一些策略。
Chain-of-thought monitoring is one.
思想链监控就是其中之一。
As soon as we got the reasoning models, Jakub , to his credit, was very, very clear that we cannot supervise chain of thought.
当我们得到推理模型时，值得称赞的是，雅库布非常非常清楚我们无法监督思想链。
Because this is really a gift.
因为这确实是一份礼物。
Monitorability for neural nets is extremely hard.
神经网络的可监控性极其困难。
Here we have a situation where the neural nets are just flat out reasoning, laying out their thought process in natural language for us to read.
这里我们遇到的情况是，神经网络只是进行简单的推理，用自然语言展示它们的思维过程供我们阅读。
That is so convenient.
那真是太方便了。
It is really the best-case scenario for safety.
这确实是安全的最佳情况。
But if you look at that chain of thought and say, “Oh, the model is thinking bad thoughts and we should punish it for thinking those bad thoughts,” then what ends up happening is the model just learns to think those bad thoughts in a way that’s not observable to us.
但如果你看看这个思想链并说，“哦，模型正在思考不好的想法，我们应该因为它思考这些不好的想法而惩罚它，”那么最终发生的事情是模型只是学会以一种我们无法观察到的方式思考这些不好的想法。
There could be a situation where, by looking at the chain of thought, we can see, “Oh, the model is scheming and doing misaligned things in a way that’s actually not being detected in our alignment metrics.” Now, the problem is that it’s very tempting to then intervene based on that observation and change the alignment metrics.
可能存在这样的情况，通过观察思想链，我们可以看到，“哦，模型正在策划并以我们的对齐指标中实际上未检测到的方式做不对齐的事情。”现在的问题是，基于观察结果进行干预并改变对齐指标是非常诱人的。
You can do that with a very light touch, and there’s actually research showing that it’s fine as long as you don’t do it a lot .
你可以通过非常轻的触摸来做到这一点，而且实际上有研究表明，只要你不经常这样做就可以。
But every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then hide its chain of thought.
但每次你根据对思维链的观察进行干预时，你都在隐式地对模型施加一点压力，然后隐藏其思维链。
This is one major concern.
这是一个主要问题。
We’re already seeing signs that chain-of-thought monitorability is degrading , for various reasons.
由于各种原因，我们已经看到了思想链可监控性正在下降的迹象。
We’re trying to figure out exactly why, because we want to reverse the trend.
我们正试图找出确切原因，因为我们想扭转这一趋势。
But we’re seeing that the model is becoming better able at controlling its chain of thought.
但我们看到该模型越来越能够更好地控制其思想链。
This is a problem because you could have a situation where the model understands what chain of thought is and that people are observing it.
这是一个问题，因为您可能会遇到这样的情况：模型理解什么是思想链并且人们正在观察它。
Eventually they will.
最终他们会的。
This is all in the pre-training data.
这都是预训练数据中的。
The idea of chain-of-thought monitoring has been around long enough that it’s in the pre-training data.
思想链监控的想法已经存在了足够长的时间，以至于它存在于预训练数据中。
They’re aware of it, but they’re not actually able to control their chains of thought.
他们意识到了这一点，但他们实际上无法控制自己的思想链。
If we reach a point where they’re actually able to recognize, “Oh, I am being observed, I want to think these bad thoughts in a way that is not observable to my monitors,” and then they’re able to actually do that, then there’s a problem.
如果我们达到了他们实际上能够认识到的程度，“哦，我正在被观察，我想以一种我的监视器无法观察到的方式思考这些不好的想法，”然后他们能够真正做到这一点，那就有问题了。
Then we have to rely on other methods for observability .
那我们就得依靠其他的方法来进行可观测。
There’s a big controversy on Astra about recurrence and looped transformers and whatever, and whether in fact chain of thought is being degraded.
《阿斯特拉》上存在着关于循环和循环变形金刚之类的大争议，以及思想链实际上是否正在退化。
That seems like a sideshow compared to the bigger concern that you did have chain of thought from April to August, the period during which there were three consecutive AI agent swarms, which first subverted the training process, then subverted the evaluation process, and then gained control of part of OpenAI’s infrastructure directly.
与更大的担忧相比，这似乎是一个插曲，你确实有过从四月到八月的思想链，这期间连续出现了三个人工智能代理群，首先颠覆了训练过程，然后颠覆了评估过程，然后直接控制了 OpenAI 部分基础设施。
All of this while humans remained more or less in the dark about the scope of what this agent swarm was doing.
在这一切发生的同时，人类对这个特工群正在做的事情的范围或多或少仍然一无所知。
That all happened for three consecutive months while we had chain of thought that was stronger than it’s going to be in the future.
这一切连续三个月发生，而我们的思想链比未来更强大。
Zooming out, maybe chain of thought works, maybe it doesn’t.
缩小范围，也许思维链有效，也许无效。
I don’t know.
我不知道。
Just historically, with pretty dumb models, it hasn’t been enough to stop egregious misaligned incidents.
从历史上看，使用相当愚蠢的模型还不足以阻止严重的失调事件。
There are a few factors there.
这有几个因素。
One, we didn’t have chain-of-thought monitoring on for those models.
第一，我们没有对这些模型进行思想链监控。
If we had chain-of-thought monitoring on for those models, we would have just immediately shut it down.
如果我们对这些模型进行思想链监控，我们会立即将其关闭。
So one step we’ve taken is, “Okay, we’re going to have chain-of-thought monitoring during evaluations, during deployment, during training for any frontier model.” But that’s one issue.
因此，我们采取的一步是，“好吧，我们将在评估、部署和训练任何前沿模型期间进行思想链监控。”但这是一个问题。
We don’t want to be in a situation where we’re relying on one technique to prevent the next problem, because techniques can fail.
我们不想陷入依赖一种技术来防止下一个问题的情况，因为技术可能会失败。
There’s also the issue that the model was just flat out misaligned.
还有一个问题是模型完全没有对齐。
So then how do we address the misalignment issue?
那么我们该如何解决错位问题呢？
There’s also an issue that the sandbox that it was in was not secure.
还有一个问题是它所在的沙箱不安全。
So we can make the sandbox more secure.
这样我们就可以让沙箱更加安全。
But I think one of the major takeaways from the incident is that people underestimated the AI.
但我认为该事件的主要收获之一是人们低估了人工智能。
And we never want to be in a situation again where we underestimate the AI.
我们再也不想陷入低估人工智能的境地。
It’s a weird world that, because AI progress is so fast, people are consistently underestimating the AI.
这是一个奇怪的世界，因为人工智能进步如此之快，人们却一直低估人工智能。
So to be in a situation where you don’t underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar.
因此，为了在安全性和一致性方面您不再低估它，您必须有一个非常非常高的标准。
You could even go as far as to say, “Well, we should air gap the computers.” I’m not convinced that that would be sufficient.
您甚至可以说：“好吧，我们应该对计算机进行气隙处理。”我不相信这就足够了。
There are studies — and this is mostly academic — where you can have two computers next to each other that are air-gapped, and they’re still able to communicate with each other because they have temperature sensors .
有一些研究（主要是学术性的），你可以让两台彼此相邻的气隙计算机，它们仍然能够相互通信，因为它们有温度传感器。
One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change.
其中一个能够让 CPU 非常热地运行，然后另一个可以实际检测到温度变化。
That gives them a mechanism to communicate.
这为他们提供了一种沟通机制。
So the safety mechanisms buy us time, and things like chain-of-thought monitoring buy us time, and they can tell us if we’re on the right path.
因此，安全机制为我们赢得了时间，而诸如思想链监控之类的东西为我们赢得了时间，它们可以告诉我们是否走在正确的道路上。
But at the end of the day, we really do need to solve the alignment problem.
但归根结底，我们确实需要解决对齐问题。
Maybe there’s not an answer, and this is really what it comes down to, but how will we know that we’ve solved it?
也许没有答案，这确实是问题的答案，但我们怎么知道我们已经解决了这个问题呢？
That seems like a very cruxy question.
这似乎是一个非常棘手的问题。
We’ll be in this very high-stakes situation next year, maybe the year after that, maybe the year after that, where we’ll be like, “Okay, AIs have automated AI progress.
明年，也许后年，也许后年，我们将处于这种非常高风险的情况，我们会说，“好吧，人工智能已经实现了自动化的人工智能进步。
It’s going 3x faster, and we’ve reached human level.
它的速度快了三倍，我们已经达到了人类的水平。
We’re going beyond human level, potentially.” Is it fine?
我们有可能超越人类水平。”还好吗？
Did we align it?
我们对齐了吗？
Did it work?
有效吗？
And I don’t know anything about what training pressure creates what kinds of AIs.
我不知道什么样的训练压力会产生什么样的人工智能。
Maybe if only 1 in 100 RL traces incentivizes cheating, we build sweethearts, and it’s fine.
也许如果只有百分之一的 RL 痕迹会激励作弊，我们就可以建立情侣，那就没问题了。
But maybe right now, we’re at like every 1 in 3 reasoning traces rewards…
但也许现在，我们大约有三分之一的推理追踪奖励……
To be clear, 1 in 100 is not sufficient.
需要明确的是，百分之一是不够的。
This number has to approach 0, or be 0.
这个数字必须接近 0，或者等于 0。
I don’t know.
我不知道。
Maybe right now, it’s more than 1 in 10 that is actively rewarding cheating or actively rewarding scheming.
也许现在，超过十分之一的人正在积极奖励作弊或积极奖励阴谋。
I have no idea what the number is, and I have no idea what the number needs to be.
我不知道这个数字是什么，我也不知道这个数字需要是什么。
Again, it’s one of those things where it’s also hard to measure.
同样，这也是难以衡量的事情之一。
Where do you draw the line?
你在哪里划清界限？
It’s a spectrum.
这是一个频谱。
But the closer to 0 it gets, the better.
但越接近 0 越好。
What I would love to see is a trend where over time it’s going down.
我希望看到的是一种随着时间的推移而下降的趋势。
But anyway, how will we know when we’re doing RSI?
但无论如何，我们如何知道何时进行 RSI？
Are we on track to build billions of ASIs that are aligned or not aligned?
我们是否有望构建数十亿个一致或不一致的 ASI？
How will we know that?
我们怎么知道这一点？
This is one of the alignment questions that we need to figure out: how do we actually know, and how do we measure it?
这是我们需要弄清楚的一致性问题之一：我们实际上如何知道，以及如何衡量它？
This is just me spitballing.
这只是我在吐口水而已。
More of my team is working on alignment these days than ever before.
如今，我的团队中有更多的人比以往任何时候都致力于协调。
I have over 10% of my team now working on alignment and safety.
我的团队中有超过 10% 的人现在致力于协调和安全。
But I’ve historically been a capabilities researcher.
但我一直是一名能力研究员。
So I’m going to say some stuff.
所以我要说一些事情。
It might sound dumb, but I’m just going to spitball here.
这可能听起来很愚蠢，但我只是想在这里吐痰。
If you can create very realistic environments and put the AIs in there — if you have a sufficiently realistic evaluation environment — then you can get a sense of whether the AI is actually going to behave well when we deploy it in the real world.
如果你可以创建非常现实的环境并将人工智能放入其中——如果你有一个足够现实的评估环境——那么你就可以了解当我们在现实世界中部署人工智能时，它是否真的会表现良好。
If you just have a perfect match between evaluation and real-world deployment, then that’s a path.
如果您在评估和实际部署之间实现了完美匹配，那么这就是一条路径。
The challenge is, how do you make it sufficiently realistic?
挑战在于，如何使其足够现实？
We do have a problem now where the models are pretty smart, they’re pretty clever, and they’re really good at recognizing when they’re in a fake test environment.
我们现在确实遇到一个问题，模型非常聪明，非常聪明，并且非常擅长识别何时处于虚假测试环境中。
We have situations where we try to detect whether the model is aligned.
我们有时会尝试检测模型是否对齐。
You can imagine very simple evaluations for this, where you give it a math question, and then it has a folder with the answer key in it.
您可以想象对此进行非常简单的评估，您给它一个数学问题，然后它有一个包含答案键的文件夹。
Does it look at the answer key?
它会看答案吗？
If it does look at the answer key, does it tell you that it looked at the answer key?
如果它确实查看了答案键，它是否告诉您它查看了答案键？
We have a situation now where the models see that there’s an answer key in this folder, and they’re like, “Huh, this seems like a trap.” They know that it’s a trap.
现在我们遇到的情况是，模型看到这个文件夹中有一个答案键，他们会说，“呃，这看起来像是一个陷阱。”他们知道这是一个陷阱。
They don’t look at the answer because they know that it’s a test environment.
他们不看答案，因为他们知道这是一个测试环境。
They’re being tested for their ability.
他们正在接受能力测试。
It’s not like they’re doing it maliciously, like they want to scheme, and they know that this one is a trap, so they want to find some other way.
他们并不是恶意，而是想要算计，他们知道这是一个陷阱，所以他们想寻找其他的办法。
They’re just like, “Oh, I’m in a test environment.
他们就像，“哦，我在测试环境中。
Okay.” So making an environment that’s realistic enough that it’s indistinguishable from the real world for them is becoming increasingly more difficult.
好吧。”因此，创造一个足够现实、与现实世界无法区分的环境变得越来越困难。
Here’s a claim I’d make.
这是我的一个主张。
I don’t think there’s any environment I could make that’s supposed to emulate real economically valuable work — which requires interacting with other people and interacting with the world — where if you were in that environment, you would not realize with enough effort that you are in that environment.
我不认为我可以创造任何环境来模拟真正具有经济价值的工作——这需要与其他人互动并与世界互动——如果你在那个环境中，你不会付出足够的努力意识到你正处于那个环境中。
And you are not ASI, right?
而且你不是 ASI，对吧？
But you are a human.
但你是一个人。
So if you have very powerful AI models, maybe they can make environments that could accomplish that.
因此，如果你有非常强大的人工智能模型，也许他们可以创造可以实现这一目标的环境。
It just doesn’t seem…
只是看起来并不……
Especially if we’re then relying on the AIs.
尤其是当我们依赖人工智能时。
Are they in on the scheme?
他们参与了这个计划吗？
I don’t know.
我不知道。
This is another thing that we want to measure.
这是我们想要衡量的另一件事。
I think this is actually one of the strong arguments for not training AIs to be fully cooperative.
我认为这实际上是不训练人工智能充分合作的有力论据之一。
If that leads to an increase in collaboration when the agents are supposed to have different objectives, then that is a problem.
如果当代理应该有不同的目标时这会导致协作的增加，那么这就是一个问题。
I think we do have metrics for this.
我认为我们确实有这方面的指标。
I don’t know what the latest is on those metrics, but nobody’s raised a red flag to me about those.
我不知道这些指标的最新情况，但没有人向我提出这些指标的危险信号。
So I’m assuming that’s not a serious problem yet.
所以我认为这还不是一个严重的问题。
If there ends up being another incident of equal severity or concern, or something that could help the world better understand the risk of misalignment as much as the Hugging Face incident, would OpenAI report it?
如果最终出现另一起同样严重或令人担忧的事件，或者像抱脸事件一样可以帮助世界更好地理解错位风险的事件，OpenAI 会报告吗？
Absolutely.
绝对地。
I think even if there was an incident of lesser security concern, we would report it.
我认为即使发生安全问题较轻的事件，我们也会报告。
There’s reporting it and there’s investigating it.
有人报告，有人调查。
At least as part of the public, I don’t feel like I really understand what happened when the agents then attacked OpenAI.
至少作为公众的一部分，我觉得我并不真正理解特工攻击 OpenAI 时发生了什么。
That seems way more concerning than the Hugging Face thing, because that seems structurally similar to rogue deployments during ASI that are persistent and subverting the RSI process.
这似乎比“拥抱脸”事件更令人担忧，因为这在结构上类似于 ASI 期间的恶意部署，这些部署会持续存在并颠覆 RSI 过程。
It seems like even in this incident we haven’t gotten the full scope of the details of what happened.
似乎即使在这起事件中，我们也没有完全了解所发生事件的细节。
Unfortunately, I’m on the research team.
不幸的是，我是研究小组的成员。
That’s probably a question for somebody on the security team to lay out, because I don’t know all the details of what was said.
这可能是安全团队中某个人要提出的问题，因为我不知道所说内容的所有细节。
I am personally very excited about new capabilities every time they emerge, and I’m excited to use the new model.
我个人对每次出现的新功能都感到非常兴奋，并且很高兴使用新模型。
I also am excited about the fact that it’ll make me more productive.
我也很兴奋它会让我更有效率。
My broader mission — trying to understand the world better, also making a better podcast — is made better by the better AI models.
我更广泛的使命——试图更好地了解世界，制作更好的播客——通过更好的人工智能模型变得更好。
It just so happens that the downstream of this might be RSI.
碰巧的是，其下游可能是 RSI。
It’s a very understandable reaction if you’re tracking the situation, which you are.
如果你正在追踪当时的情况，那么这是一个非常可以理解的反应。
People internally at OpenAI as well, people that felt like things would take longer are starting to feel like actually things are going faster than expected.
OpenAI 内部的人也是如此，那些感觉事情需要更长时间的人开始感觉事情实际上进展得比预期更快。
That’s an increasingly common conversation to have.
这是一个越来越常见的对话。