System.Xml.XmlElement

I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.

我和我认识的一些在开放公司工作的最有洞察力的人工智能研究人员聚在一起，因为我想了解前沿实际发生的事情以及接下来会发生什么的细节。

Antithesis helps you trust your code.

对立可以帮助您信任您的代码。

As agents generate more and more of your software, the bottleneck shifts from your engineers actually writing code to verifying it.

随着代理生成越来越多的软件，瓶颈从工程师实际编写代码转移到验证代码。

Antithesis does that testing for you.

Antithesis 会为你做这个测试。

Ron Minsky, who co-leads Jane Street’s tech group, told me that Antithesis was able to help his team shake out bugs in software that had already undergone heavy review.

Jane Street 技术团队的联合负责人罗恩·明斯基 (Ron Minsky) 告诉我，Antithesis 能够帮助他的团队消除已经经过严格审查的软件中的错误。

If you want to see how it fits into your development process, go to antithesis.com/dwarkesh

如果您想了解它如何适合您的开发流程，请访问 antithesis.com/dwarkesh

Grok Bot has been a great way to hand off tasks.

Grok Bot 是一种传递任务的好方法。

My team uses it as a producer: whenever my editor posts a rough cut of an interview in Slack, Grok Bot opens the transcript on its own computer, matches my notes to the exact moments they refer to, and uses a file of my preferences to suggest edits.

我的团队将它用作制作人：每当我的编辑在 Slack 中发布采访的粗略剪辑时，Grok Bot 就会在自己的计算机上打开记录，将我的笔记与他们引用的确切时刻进行匹配，并使用我偏好的文件来建议编辑。

Then it sends me its top clip candidates so I can review everything from my phone, which saves my editors from sorting through hours of footage.

然后它会向我发送最热门的候选剪辑，以便我可以通过手机查看所有内容，这使我的剪辑师无需整理几个小时的素材。

Try Grok Bot for yourself at x.ai/bot

在 x.ai/bot 上亲自尝试 Grok Bot

Jane Street just launched its most ambitious competition yet: design a protocol-emulator ASIC.

Jane Street 刚刚发起了迄今为止最雄心勃勃的竞赛：设计协议仿真器 ASIC。

Basically, if you have a chip you want to test outside of a live system, you should be able to connect it to your design and have it simulate realistic traffic.

基本上，如果您想要在实时系统之外测试芯片，您应该能够将其连接到您的设计并让它模拟真实的流量。

Jane Street wants general-purpose, reprogrammable designs that can work across multiple protocols and remain useful as new ones emerge.

Jane Street 希望通用的、可重新编程的设计能够跨多种协议工作，并且在新协议出现时仍然有用。

The most novel submissions will actually get taped out, and the winners will receive a physical copy!

最新颖的提交内容将被录制下来，获奖者将收到实体副本！

The competition is open until January 18, 2027, and teams are encouraged.

比赛开放至2027年1月18日，鼓励各队参赛。

To get started download the template code at janestreet.com/dwarkesh

要开始下载模板代码，请访问 janestreet.com/dwarkesh

Today, I’m chatting with three of my AI researcher friends from whom I learn a lot every time we talk.

今天，我正在和我的三位人工智能研究员朋友聊天，每次谈话我都能从他们身上学到很多东西。

They also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record.

他们也恰好在有点开放的实验室和公司，所以你们实际上可以记录下来说一些事情。

I’m joined by Beren Millidge , who is the CTO of Zyphra , which is developing open source models.

Beren Millidge 也加入了我的行列，他是 Zyphra 的 CTO，该公司正在开发开源模型。

John Schulman is the chief scientist at Thinking Machines , previously a co-founder of OpenAI, and led the RLHF work that led to ChatGPT.

John Schulman 是 Thinking Machines 的首席科学家，之前是 OpenAI 的联合创始人，并领导了 RLHF 工作，最终催生了 ChatGPT。

And Charlie O’Neill is head of model training at Baseten .

Charlie O’Neill 是 Baseten 模型培训的负责人。

The first question I have: If we’re in 2036 and we don’t have billions of crazy superintelligence s running around that have radically transformed the world, what is the most likely reason that doesn’t end up being the case?

我的第一个问题是：如果到了 2036 年，我们还没有数十亿疯狂的超级智能从根本上改变世界，那么最有可能的原因是什么？

Other than exogenous political shocks, or there’s a war, or they ban AI or something.

除了外来的政治冲击，或者发生战争，或者他们禁止人工智能之类的。

What is the most likely technical reason that 2036 isn’t a crazy alien superintelligence world?

2036 年不是一个疯狂的外星超级智能世界，最有可能的技术原因是什么？

There’s been a classic thing, almost like Moravec’s paradox , where we think of the AI as, “If it can do this, it’s going to be amazing.” If it can solve these hard maths problems, if it can win at chess, blah, blah, blah…

有一个经典的事情，几乎就像莫拉维克悖论一样，我们认为人工智能是，“如果它能做到这一点，那就太棒了。”如果它能解决这些困难的数学问题，如果它能在国际象棋中获胜，等等，等等，等等……

Then it solves these things, and it’s not that impactful.

然后就解决了这些事情，也没有那么大的影响力。

Obviously, it’s somewhat impactful, but not everything.

显然，它有一定影响，但不是全部。

If somehow that continues, and there’s never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at everything that people put into a benchmark or put into an environment.

如果这种情况持续下去，并且永远不会出现真正的泛化火花，我认为这可能会导致人工智能在人们放入基准或环境中的所有事情上都表现得非常出色。

But there’s still some persistent sim-to-real gap which is somehow blocking everything.

但仍然存在一些持续存在的模拟与真实差距，这在某种程度上阻碍了一切。

I think this is unlikely.

我认为这不太可能。

We do actually see this kind of generalization even from RL in practice already.

实际上，我们已经在实践中从强化学习中看到了这种泛化。

But if it is just ridiculously hard to generalize meta-learning, plus we don’t solve continual learning and it’s just super hard and impossible…

但如果泛化元学习极其困难，再加上我们不解决持续学习问题，那就超级困难且不可能……

This would be my default scenario in that case.

在这种情况下，这将是我的默认场景。

I agree with that.

我同意这一点。

Humans have a lot of advantages over models now.

现在人类比模型有很多优势。

Each time a new model comes out, it’ll catch up in some of these areas.

每次新车型问世，它都会在其中一些领域迎头赶上。

But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment, or the models can’t check themselves well enough.

但最终你会遇到模型较弱的地方和判断力较差的地方，或者模型无法很好地自我检查的地方。

There’s this cycle that keeps repeating where a new model comes out and people are blown away and they’re like, “This is it.

这种循环不断重复，新车型问世，人们大吃一惊，他们会说：“就是它了。”

This is AGI.” But then they use it a bit, and it starts to feel dumb after a month or so.

这就是通用人工智能。”但当他们使用了一点之后，大约一个月后就开始感觉很愚蠢。

That cycle just might keep going.

这个循环可能会继续下去。

It’s hard to predict how many times it’s going to repeat.

很难预测它会重复多少次。

Right now, you don’t get explosive growth in capabilities because you still get bottlenecked enough when you’re trying to do research and engineering.

现在，你的能力还没有得到爆炸性的增长，因为当你尝试进行研究和工程时，你仍然遇到足够的瓶颈。

Even if the model can write way more code than a person, it doesn’t make you 100X more productive.

即使模型可以编写比人多得多的代码，它也不会让你的工作效率提高 100 倍。

So maybe there are just more of these cycles than we would expect.

因此，这些周期的数量可能比我们预期的要多。

For me, it’s a question of how far off the global optimum of “a learner you could have on a chip” is from the transformer + RL, basically the current recipe.

对我来说，问题在于“芯片上的学习器”的全局最优值与变压器+强化学习（基本上是当前的配方）相差多远。

People imagine that once you have an agent which is better than all humans at AI research, even if it’s 0.1% better than all humans, then the fact that you can run hundreds of thousands, if not millions, of these in parallel — and you can run them much faster as chips speed up — is going to outweigh every other bottleneck.

人们想象，一旦你拥有一个在人工智能研究中比所有人类都更好的代理，即使它比所有人类好 0.1%，那么你可以并行运行数十万甚至数百万个代理，而且随着芯片加速，你可以更快地运行它们，这一事实将超越所有其他瓶颈。

You’re eventually going to hit this very fast takeoff with regards to self-improvement .

在自我提升方面，你最终将会实现非常快速的起飞。

I could imagine that if we continue along the trajectory that we’re currently on with that paradigm, where it’s basically self-attention, RL, scaling up RL environments …

我可以想象，如果我们继续沿着目前的范式发展，基本上就是自我关注、强化学习、扩大强化学习环境……

Think about what happened with Moore’s law .

想想摩尔定律发生了什么。

We had this very nice straight line and that held for a really, really long time.

我们有一条非常漂亮的直线，并且保持了非常非常长的时间。

But there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going .

但为了保持比例定律的继续，必须发生许多离散的不连续性和创新。

The same thing has happened with LLMs .

同样的事情也发生在法学硕士身上。

We had this pre-training scaling law , and then that was hitting diminishing returns.

我们有这个预训练缩放定律，然后就出现了收益递减。

Then we came up with RL and solved that, and then we got this new diminishing returns curve to hit that made it keep looking like a straight line going up.

然后我们提出了强化学习并解决了这个问题，然后我们得到了这条新的收益递减曲线，让它看起来一直像一条上升的直线。

So if it requires another one of those discontinuities to solve, I’m not sure that the current method of training LLMs with these RL environments, even RSI -targeted RL environments, would be able to discover that discontinuity.

因此，如果需要解决另一个不连续性，我不确定当前使用这些 RL 环境（甚至是以 RSI 为目标的 RL 环境）训练 LLM 的方法是否能够发现该不连续性。

If not, we’re probably going to hit this asymptotic curve.

如果不是，我们可能会达到这条渐近曲线。

But do you think the discontinuity will be harder than anything that’s come since 2012?

但您认为这种不连续性会比 2012 年以来出现的任何情况都更严重吗？

If we had the answer to that, we’d kind of have the ability to implement it.

如果我们找到了答案，我们就有能力实施它。

But maybe we should distinguish between a discontinuity which adds to the current paradigm, which is cumulative — there’s something beyond the RL that we have to discover, and maybe they’re capable of connecting the dots in that straight line — or, again, how far off the global optimum are we?

但也许我们应该区分当前范式的不连续性，这是累积性的——我们必须发现强化学习之外的东西，也许它们能够将直线上的点连接起来——或者，再说一遍，我们离全局最优值还有多远？

Do we have to go back and throw out gradient descent and neural nets in general?

我们是否必须回去并抛弃一般的梯度下降和神经网络？

I don’t think, if you continue to scale up the current paradigm, an LLM, no matter how many LLMs you’re running, is necessarily capable of discovering that if it’s too far away.

我不认为，如果你继续扩大当前的范式，一个法学硕士，无论你正在运行多少个法学硕士，都不一定能够发现它是否太遥远了。

The only hope really is if deep learning just can’t get us to an AI which can at least dominate human research and human development, including the human ability to come up with new paradigms and so forth.

唯一的希望是，如果深度学习无法让我们获得至少能够主导人类研究和人类发展的人工智能，包括人类提出新范式等的能力。

Or, I don’t know, maybe humans would also never have discovered the next learning architecture.

或者，我不知道，也许人类也永远不会发现下一个学习架构。

But to the extent humans could have discovered it eventually…

但就人类最终可能发现它而言……

But it just seems like…

但它看起来就像……

If you just look at the progress that’s happened since 2012 till now, and you just continue that on —I know it’s just been powered by huge amounts of compute scaling and so forth— it would be weird if it just didn’t get to the point where it could dominate humans, at least in R&D, especially over the next few years.

如果你只看自 2012 年至今所发生的进展，然后继续下去——我知道它只是由大量的计算扩展等提供动力——如果它没有达到可以主宰人类的地步，至少在研发方面，尤其是在接下来的几年里，那会很奇怪。

Ryan Greenblatt was on the podcast recently.

瑞安·格林布拉特最近参加了播客。

He made this point that I’d be curious to get your thoughts on.

他提出了这一点，我很想了解你的想法。

You could imagine, as AIs get more and more capable, that they’re capable of making progress on simulations which incentivize getting better at not only AI R&D, but at science generally.

你可以想象，随着人工智能的能力越来越强，它们能够在模拟方面取得进展，这不仅会激励人工智能研发，而且会在整个科学领域取得进步。

This is a thing that all the labs are targeting and many startups are targeting.

这是所有实验室和许多初创公司都瞄准的目标。

Another intuition pump is if you look at the Elo score of chess bots since the ’80s.

另一个直觉泵是如果你看看 80 年代以来国际象棋机器人的 Elo 分数。

There’s a very linear increase in Elo over time.

随着时间的推移，Elo 呈线性增长。

But there’s this huge discontinuity as they cross the human range, from human experts always winning against AIs to human experts never winning against AIs, as this linear increase in Elo happens.

但当它们跨越人类范围时，就存在巨大的不连续性，从人类专家总是战胜人工智能到人类专家从未战胜人工智能，因为 Elo 发生了这种线性增长。

I agree with your point that so far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world.

我同意你的观点，即到目前为止，人工智能能力对于世界的最终经济影响来说还没有那么重要。

But that is because they’re slowly rising in Elo relative to humans.

但那是因为他们的 Elo 相对于人类正在慢慢上升。

I agree it would be very surprising.

我同意这会非常令人惊讶。

The only way for this to not happen is if, as you said, it somehow asymptotes just before.

避免这种情况发生的唯一方法是，正如您所说，它之前以某种方式渐近线。

Because we’re already pretty close, in my opinion, to where we’ll start crossing the human Elo score.

因为在我看来，我们已经非常接近我们将开始跨越人类 Elo 分数的地方。

So we’ll need to asymptote before that.

所以我们需要在此之前渐近。

That’s the only way — in this scenario you pose where somehow we’re sitting here in 2035 and everything is normal — for this to happen, I think.

我认为，这是发生这种情况的唯一方法——在这个场景中，你以某种方式摆出我们在 2035 年坐在这里，一切都很正常的方式。

The only other way is there’s some dramatic regulation on AI.

唯一的另一种方法是对人工智能进行一些严格的监管。

This is what I see as the most likely way for this scenario to happen, actually, rather than a technical thing.

实际上，我认为这是最有可能发生这种情况的方式，而不是技术问题。

I think there’s different kinds of research.

我认为有不同类型的研究。

There’s research in the autoresearch style where the objective is already specified very cleanly and you’re optimizing that objective.

有一些自动研究风格的研究，其中的目标已经非常明确地指定，并且您正在优化该目标。

I think everyone is picturing that if we continue along this path of making pre-training loss go down and making our environments have the reward on them go up, that’s going to lead to improvement.

我认为每个人都在想象，如果我们继续沿着这条道路继续前进，减少训练前的损失，并使我们的环境获得回报，那么这将带来进步。

But maybe what Ryan is talking about is this much more open-ended type of science which is required for paradigm shifts, where we can’t specify the objective, and the AIs are definitely not able to specify that objective either.

但也许瑞安所说的是这种更加开放的科学类型，这是范式转变所必需的，我们无法指定目标，人工智能也绝对无法指定该目标。

We have to be really, really careful about how we specify objectives for any of these things.

我们必须非常非常小心地指定这些事情的目标。

Maybe your point is that the nature of the breakthroughs that have happened since 2012 is that we have found…

也许你的观点是，自 2012 年以来发生的突破的本质是我们发现……

In 2012, people weren’t saying…

2012 年，人们并没有说……

I’m assuming, I don’t know, you guys were there.

我假设，我不知道，你们在那里。

Or at least John, you were there.

或者至少约翰，你在那里。

But I was not.

但我不是。

Actually, John, I’m curious for your wisdom of the ages, or wisdom of being in the trenches way back when.

事实上，约翰，我很好奇你的历代智慧，或者说是很久以前在战壕里的智慧。

Presumably, a big breakthrough was realizing that next token prediction is the…

据推测，一个重大突破是意识到下一个代币预测是……

You wouldn’t have thought that the nanoGPT speed run is the thing to be optimizing for in 2014.

您不会想到 nanoGPT 速度运行是 2014 年要优化的事情。

But now that we have come to this new paradigm, you would think to do a speed run on that and have AIs get really good at that.

但现在我们已经来到了这个新的范式，你可能会想在这个范式上进行快速运行，并让人工智能真正擅长于此。

But maybe there’s a next inner loop to optimize that the AIs wouldn’t anticipate.

但也许还有人工智能无法预料的下一个内部循环需要优化。

There’s an outer loop of revenue or something that eventually should be strong, but it’s a very slow outer loop.

有一个收入的外循环或者最终应该很强的东西，但这是一个非常缓慢的外循环。

In fact, I remember in the early OpenAI days having the intuition that just minimizing log loss wasn’t going to get you to intelligence.

事实上，我记得在 OpenAI 早期，我的直觉是，仅仅最大限度地减少对数丢失并不会让你获得智能。

Because the important bits are accounting for such a small fraction of the loss that it was going to be overwhelmed by noise.

因为重要的比特只占损失的一小部分，以至于它会被噪声淹没。

So just training a language model on next token prediction wasn’t going to learn the interesting things you want it to learn.

因此，仅仅训练语言模型进行下一个标记预测并不能学到你想要它学习的有趣的东西。

We needed to craft better objectives that would put more emphasis on the important things.

我们需要制定更好的目标，更加重视重要的事情。

You can make all sorts of arguments for this.

你可以为此提出各种各样的论据。

You could say, “Oh, humans probably don’t learn how to model everything in our environment.

你可以说，“哦，人类可能不会学习如何对我们环境中的一切进行建模。

Most people can’t create a photorealistic reproduction of some kind of scene they’ve looked at.

大多数人无法对他们所看到的某种场景进行逼真的再现。

So we must need a better objective.” But then it turned out that it just worked anyway.

所以我们必须需要一个更好的目标。”但后来事实证明它无论如何都有效。

As you were pointing out, the inner loop, even in current AI research, of post-training benchmarks or whatever, doesn’t necessarily translate into what users like.

正如你所指出的，即使在当前的人工智能研究中，训练后基准或其他什么的内部循环也不一定会转化为用户喜欢的东西。

Oh, yeah.

哦，是的。

The whole field relies a lot on generalization and it’s very hard to predict when you’re going to get generalization, or when you’re going to get some kind of out-of-distribution generalization .

整个领域在很大程度上依赖于泛化，并且很难预测何时会获得泛化，或者何时会获得某种分布外的泛化。

We know that if you train on the task you care about, you’re going to do better.

我们知道，如果您针对自己关心的任务进行训练，您会做得更好。

But the most important advances are often types of generalization that we have no right to expect.

但最重要的进步往往是我们无权期望的概括类型。

For example, from just pre-training on this very naive next-token-prediction objective to various tasks of interest that require understanding of the input in some deep way, or learning some skill from pre-training that’s very rare and not heavily represented.

例如，从仅对这个非常幼稚的下一个令牌预测目标进行预训练，到需要以某种深入方式理解输入的各种感兴趣的任务，或者从预训练中学习一些非常罕见且代表性不足的技能。

Then also generalization from these verifiable tasks to less verifiable ones, this is also a type of generalization that there’s no reason a priori to expect.

然后还从这些可验证的任务到不可验证的任务的概括，这也是一种没有理由先验地期望的概括。

This is an interesting question, because one intuition pump you could have for why you would see some sort of singularity very rapidly — without even scaling up the inputs to AI progress that are not just AI labor — is that before every single 7-figure experiment you run, you spend an equivalent amount of compute on AI labor.

这是一个有趣的问题，因为你可能有一个直觉泵，解释为什么你会非常快地看到某种奇点——甚至没有扩大对人工智能进步的投入，而不仅仅是人工智能劳动力——是在你运行每一个 7 位数的实验之前，你在人工智能劳动力上花费了等量的计算量。

So you just have automated versions of you guys spending a century thinking about what is the optimal experiment to run, doing small-scale ablations , developing literally a century’s worth of theory, going back even before deep learning.

所以你们只是拥有自动化版本的你们，花了一个世纪的时间思考什么是最好的实验，进行小规模的消融，发展了一个世纪的理论，甚至可以追溯到深度学习之前。

Before you decide what experiment to run, you’re doing extremely optimal setting up of the experiment.

在决定运行什么实验之前，您正在对实验进行极其优化的设置。

Then you do a century of thinking after the experiment is over, where you’re analyzing what happened and what the next experiment to run is.

然后，在实验结束后，你会进行一个世纪的思考，分析发生了什么以及下一个要进行的实验是什么。

If you think hard enough, you probably could have expected some of these things beforehand.

如果你认真思考，你可能会事先预料到其中一些事情。

There is probably some very clever way to do a small-scale experiment that’ll let you build the theory that then will generalize to the large-scale experiment.

可能有一些非常聪明的方法来进行小规模实验，让你建立理论，然后将其推广到大规模实验。

So I would expect that we’re nowhere near the ceiling of how well you can do research.

所以我预计我们还远未达到研究的上限。

I would imagine a future where AI is doing a lot of analysis and theory building, spending a comparable amount of compute to the amount that you’re spending on the experiments themselves, doing various kinds of analysis and building a theory around what we’ve seen so far.

我想象未来人工智能会进行大量的分析和理论构建，花费与实验本身相当的计算量，进行各种分析并围绕我们迄今为止所看到的建立理论。

I think there are really concrete examples of this when the objective is well specified.

我认为，当目标明确时，确实有具体的例子。

All thinking can do is update your posterior based on the bits that you’ve gotten since you formed your prior.

思维所能做的就是根据自形成先验以来所获得的信息来更新后验。

You can’t gain any new bits from just thinking.

仅靠思考你无法获得任何新的东西。

But when the objective is well specified and there is this data sitting around, I imagine there will be this big speed-up in the current paradigm we’re in.

但是，当目标明确并且有这些数据时，我想我们当前所处的范式将会有很大的加速。

A good example of this is if you got an AI to think about the Kaplan scaling laws .

一个很好的例子是，如果你有一个人工智能来思考卡普兰缩放定律。

An AI at this point would have noticed, “Oh, they’ve just taken these intermediate checkpoints and didn’t account for the annealing , and so this is wrong.” That would have been caught years earlier.

此时人工智能会注意到，“哦，他们只是采取了这些中间检查点，并没有考虑退火，所以这是错误的。”那几年前就已经被抓到了。

We would have cut off a year or two of progress just from that observation from an AI.

仅凭人工智能的观察，我们就会中断一两年的进展。

Again, once the objective is well specified, which is lower pre-training loss or whatever, there are many, many good examples where if you just thought about it a bit more, you would have been able to cut down significantly on things that you’ve done.

再说一遍，一旦明确了目标，即降低预训练损失或其他什么，就会有很多很多很好的例子，如果你多思考一下，你就能够显着减少你所做的事情。

So muP , and how learning rate scales with model size, and realizing that model width is important in that as well.

所以 muP ，以及学习率如何随模型大小缩放，并认识到模型宽度也很重要。

I feel like you can really back out a lot of these things and cut off a lot of low-hanging fruit.

我觉得你真的可以取消很多这样的事情，并砍掉很多容易实现的目标。

I would imagine a 10x speed-up if our thing is just, “Maximize the objective we’re currently on.”

如果我们的目标只是“最大化我们当前的目标”，我可以想象速度会提高 10 倍。

But I don’t see how that generalizes at all to coming up with the right objective in the first place.

但我完全不明白这如何概括为首先提出正确的目标。

Just thinking doesn’t necessarily buy you the right objective in the first place.

仅仅思考并不一定能让你从一开始就找到正确的目标。

I think this is really the key question for any kind of very rapid RSI from current AIs.

我认为这确实是当前人工智能任何快速 RSI 的关键问题。

How well can AIs generalize to learning their own objectives?

人工智能能够在多大程度上概括并学习自己的目标？

To have any kind of self-propelling automated loop, we need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time.

为了拥有任何类型的自我推进的自动化循环，我们需要人工智能提出目标，优化它们，找出目标，提出一个新的目标，并让它在很长一段时间内不会偏离轨道。

To come back to Moravec’s paradox, there might be a case of Moravec’s paradox where we think this kind of autonomy and being self-encapsulated — so we can think of what we should do ourselves and then go do it and have this loop — is super easy because we always do this.

回到莫拉维克悖论，可能有一种莫拉维克悖论的情况，我们认为这种自治和自我封装——这样我们就可以自己思考应该做什么，然后去做并进行这样的循环——是非常容易的，因为我们总是这样做。

Obviously, evolution needs to create creatures that can survive by themselves for long periods of time.

显然，进化需要创造出能够长时间独立生存的生物。

And this just might be something that for some reason is really hard for the AI, in the same way that locomotion stuff is really hard but math is super easy despite being super hard for us.

由于某种原因，这可能对人工智能来说确实很难，就像运动的东西确实很难，但数学却非常简单，尽管对我们来说非常困难。

Yeah, exactly.

是的，完全正确。

This is another possibility, but I agree, there’s no obvious evidence for this.

这是另一种可能性，但我同意，没有明显的证据证明这一点。

In fact, the fact that our agents are now super persistent and it’s quite easy to do this is kind of evidence against this.

事实上，我们的特工现在非常执着，而且很容易做到这一点，这一事实就是反对这一点的证据。

But this would potentially be one of the reasons why we just don’t get this immediate takeoff, if this is hard.

但这可能是我们无法立即起飞的原因之一，如果这很难的话。

If you look back from 2012 till now — or maybe from when you started doing your research till now — what part of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, seems like the thing that would be the last thing humans would have to do before AI totally automates AI R&D?

如果你回顾从 2012 年到现在，或者从你开始做研究到现在，从那时起发生的所有创新，包括纯粹的工程创新，包括纯粹的概念创新，有哪一部分似乎是在人工智能完全自动化人工智能研发之前人类要做的最后一件事？

Probably just iteratively asking the right questions.

可能只是反复提出正确的问题。

If you can get the AI to do any experiment, you still need to decide what experiments to do.

如果你可以让人工智能做任何实验，你仍然需要决定做哪些实验。

Right now I think AIs are not very good at this compared to coding the experiment.

目前，我认为与编写实验代码相比，人工智能在这方面还不太擅长。

Whenever we talk about research, they propose a bunch of miscellaneous things which are very, very tiny steps.

每当我们谈论研究时，他们都会提出一堆杂项，这些都是非常非常小的步骤。

Or even going from DeepMind ’s approach of, “We’re going to solve intelligence by learning to play games at a superhuman level,” to one random researcher like Radford being, “I’m going to try and just predict the next token of a very wide swath of data”…

或者甚至从 DeepMind 的方法“我们将通过学习超人水平的游戏来解决智力问题”，到像 Radford 这样的随机研究人员，“我将尝试预测大量数据中的下一个标记”……

Even once Radford had discovered that, it took a while before people decided to scale it up, because we had to come up with the idea of scaling laws and the fact that you could very reliably predict these things.

即使雷德福发现了这一点，人们也花了一段时间才决定扩大规模，因为我们必须提出缩放定律的想法，以及你可以非常可靠地预测这些事情的事实。

I would say that the last job for humans, or the role for humans that’ll last the longest, is defining the objective and deciding what we actually want.

我想说，人类最后的工作，或者人类持续时间最长的角色，就是定义目标并决定我们真正想要什么。

In that vein, something like deciding how the AI assistants should behave, or what it means to be helpful, or what the objective is when we’re doing RL from human feedback, is one such thing.

在这种情况下，诸如决定人工智能助手应该如何表现，或者提供帮助意味着什么，或者当我们根据人类反馈进行强化学习时的目标是什么，都是这样的事情。

Then later, defining constitutions and model specs is another one.

然后，定义宪法和模型规格是另一回事。

Even if the AIs can do all the technical work, we’ll still have to do a lot of that and decide what we actually want.

即使人工智能可以完成所有的技术工作，我们仍然需要做很多工作并决定我们真正想要什么。

Alignment is sort of the answer.

对齐是某种答案。

But alignment itself can be decomposed into specification of the objective, or figuring out what the right objective should be, and then actually achieving or optimizing the objective you’ve defined.

但一致性本身可以分解为目标的规范，或者弄清楚正确的目标应该是什么，然后实际实现或优化您定义的目标。

I think the first one is not going to go away anytime soon.

我认为第一个不会很快消失。

If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave.

如果我考虑一个培训后团队以及为什么团队中需要很多人，那只是因为有很多不同的领域你必须弄清楚模型应该如何表现。

It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.

将整个事情自动化是非常困难的，因为必须有人考虑模型在这个领域应该如何表现。

What is the story for why there isn’t huge consolidation in model providers?

模型提供商没有大规模整合的原因是什么？

There are just so many things that point to centralization here.

这里有很多事情都表明集中化。

If you step back over the course of years, is there something that is going to prevent that?

如果你退后几年，有什么可以阻止这种情况发生吗？

I think distillation is the main thing that fights against the centralizing force.

我认为蒸馏是对抗集权的主要手段。

Basically anything that can be learned through RL can be distilled very easily, because it’s a small number of bits.

基本上，任何可以通过强化学习学到的东西都可以很容易地提炼出来，因为它的位数很少。

It’s something that you can learn from a small amount of data.

您可以从少量数据中了解到这一点。

If you can get trajectories from the model that show a behavior, you can easily distill it.

如果您可以从模型中获取显示行为的轨迹，则可以轻松地提取它。

I think distillation is one of the things that fights centralization.

我认为蒸馏是对抗中心化的事情之一。

There is also a possibility that there’ll be company-specific models, that it’ll be possible to learn from deployment and have a company continually improving its own model.

还有一种可能性是存在特定于公司的模型，可以从部署中学习并让公司不断改进自己的模型。

Such a system could be provided by the current oligopoly of model providers or some other currently smaller company.

这样的系统可以由当前寡头垄断的模型提供商或其他一些当前较小的公司提供。

But I think that’ll change the game a bit.

但我认为这会稍微改变游戏规则。

I also want to point out that continual learning, honestly, doesn’t stop distillation.

我还想指出，老实说，持续学习并不能阻止蒸馏。

Even if your model is improving every day, people could be distilling it every day.

即使你的模型每天都在改进，人们也可能每天都在提炼它。

The loops could just operate at the same pace.

循环可以以相同的速度运行。

That makes sense.

这是有道理的。

So copying model behavior…

所以复制模型行为......

I guess you need to know yourself what the right distribution to prompt is in order to get the relevant model behavior?

我想您需要知道提示的正确分布是什么才能获得相关的模型行为？

Oh, yeah.

哦，是的。

For just distilling with supervised learning , the prompt distribution is extremely important.

对于监督学习的蒸馏来说，即时分布极其重要。

It’s very non-trivial to distill a model, even if you have full access to it and have the chain of thought and everything.

提炼一个模型是非常重要的，即使你可以完全访问它并且拥有思想链和一切。

It’s non-trivial to distill all of the useful capabilities from it, because you need to prompt the model with something.

从中提取所有有用的功能并不简单，因为您需要用一些东西来提示模型。

You need to prompt it with realistic prompts.

你需要用现实的提示来提示它。

You need to have a really wide distribution of realistic prompts.

您需要广泛分布真实的提示。

One thing that’s been coming out recently is that some of the Chinese companies are probably using these router services which are designed to allow people in China to use the US frontier models, which would otherwise be blocked in China.

最近出现的一件事是，一些中国公司可能正在使用这些路由器服务，这些服务旨在允许中国人使用美国边境模型，否则这些模型在中国将被阻止。

There are all these router or proxy services that allow people in China to use these models, mostly for coding.

所有这些路由器或代理服务都允许中国人使用这些模型，主要用于编码。

And these router services are collecting and selling some of the data.

这些路由器服务正在收集和出售一些数据。

This is a very useful data set for distillation because it gives you the perfect prompt distribution.

这是一个对于蒸馏非常有用的数据集，因为它为您提供了完美的提示分布。

I think this is one of those things where AIs help a lot.

我认为这是人工智能能提供很大帮助的事情之一。

If you actually look at the frontier pipelines, or the Chinese models that they’ve actually put in their papers, they get seed prompts from somewhere, which is some combination of humans and this kind of data.

如果你真的看看前沿管道，或者他们实际上放入论文中的中国模型，他们会从某个地方得到种子提示，这是人类和此类数据的某种组合。

Then they synthesize a vast coverage from those seed prompts using their existing models or the other frontier models.

然后，他们使用现有模型或其他前沿模型从这些种子提示中综合出广泛的覆盖范围。

You can automate an awful lot of this prompt distribution gathering and environment creation.

您可以自动完成大量的即时分发收集和环境创建。

Humans need to provide increasingly fewer bits as the models get better.

随着模型变得更好，人类需要提供越来越少的比特。

But it still seems you’re bottlenecked by having a service which has users going through it.

但您似乎仍然因拥有一项需要用户使用的服务而遇到瓶颈。

Not necessarily.

未必。

That’s obviously very helpful, but theoretically, you can just think about what users want.

这显然非常有帮助，但从理论上讲，你可以只考虑用户想要什么。

But the whole point is that the user says, “Make me an application like this.

但重点是用户说：“给我一个这样的应用程序。

Oh, that didn’t work.

哦，那没用。

I actually want you to make this new feature.

我实际上希望你开发这个新功能。

But actually, let’s step back and do this other thing.” Capturing that whole trace is the thing.

但实际上，让我们退后一步，做另一件事。”捕捉整个痕迹就是关键。

Or to the extent you could have done that anyway, then you just have RSI.

或者在一定程度上你无论如何都可以做到这一点，那么你就拥有 RSI。

Ultimately, if you have this fully automated loop, that is basically RSI.

最终，如果你有这个完全自动化的循环，那基本上就是 RSI。

The AI is deciding the data, it’s deciding the training.

人工智能决定数据，决定训练。

That is the loop.

这就是循环。

But it depends how much human information you need.

但这取决于您需要多少人类信息。

At some point, if you’re just like, “I want traces that look like this,” you prompt that to the model.

在某些时候，如果您只是想，“我想要看起来像这样的痕迹”，您会向模型提示这一点。

The model will be able to come up with a pretty good approximation.

该模型将能够得出一个相当好的近似值。

But what if you want to do, “Make me a really good politician,” and then it has to anticipate de novo how a discussion in the Senate halls would go or something?

但是，如果你想做“让我成为一名真正优秀的政治家”，然后它必须从头预测参议院大厅的讨论将如何进行或其他什么？

I just feel like there are going to be a lot of things which are—

我只是觉得将会有很多事情——

Ironically, this is actually easier for the distillers than the frontier labs.

讽刺的是，这对于酿酒师来说实际上比前沿实验室更容易。

The distiller’s just like, “I want a good politician.” They go to the frontier model.

酿酒师就像是，“我想要一个好的政治家。”他们走向前沿模型。

The frontier model already knows how to be a good politician, so it just generates those traces.

边境模型已经知道如何成为一名优秀的政治家，因此它只是生成这些痕迹。

Whereas if you actually want to build the first model that does this, you have to actually somehow get data on what politicians do every day and build that.

然而，如果你真的想构建第一个模型来做到这一点，你实际上必须以某种方式获取政治家每天所做的事情的数据并构建它。

It’s actually much easier to say, “I want something like this,” and then get the AI to produce a billion variations, than to actually create the thing like this to begin with.

实际上，说“我想要这样的东西”，然后让人工智能产生十亿种变化，比一开始就实际创建这样的东西要容易得多。

I think you can actually make a really concrete prediction based off this observation that the Chinese labs have this router data.

我认为你实际上可以根据中国实验室拥有此路由器数据的观察做出非常具体的预测。

The thing that started this originally was I was saying, “Isn’t it weird how Sonnet 5 and Opus 5 are almost objectively worse models than GLM-5.3 and Kimi K3 , even though they’ve had access to not only distillation but logit distillation from Mythos ?” The counter was that the prompt distribution really, really matters.

最初开始的事情是我说的，“Sonnet 5 和 Opus 5 在客观上几乎是比 GLM-5.3 和 Kimi K3 更糟糕的模型，这不是很奇怪吗？尽管它们不仅可以使用 Mythos 的蒸馏，还可以使用 Logit 蒸馏？”反驳是，及时分发确实非常重要。

You need to see what users are doing so that you can distill these behaviors and things in.

您需要了解用户在做什么，以便您可以提取这些行为和事物。

I think the prediction from this is that the frontier labs don’t necessarily have much of an advantage, if at all, in RL environments now.

我认为由此得出的预测是，前沿实验室现在在强化学习环境中不一定有太大优势（如果有的话）。

Yes, user distribution matters for general behavior and so on, but the best measure of a capability is the very, very hard RL environments you’ve made at the frontier.

是的，用户分布对于一般行为等很重要，但衡量能力的最佳标准是你在前沿创建的非常非常困难的 RL 环境。

If you have access to those RL environments as Anthropic, and you have access to logit distillation, and you’ve still made a worse model, then maybe—

如果你可以作为 Anthropic 访问这些 RL 环境，并且可以访问 Logit 蒸馏，并且你仍然制作了一个更糟糕的模型，那么也许 -

Then real-world deployment matters more than the environment.

那么现实世界的部署比环境更重要。

That’s really interesting.

这真的很有趣。

But they had to incentivize those capabilities in the first place in Fable , or the frontier model.

但他们必须首先在《神鬼寓言》或前沿模型中激励这些能力。

So it’s weird that they can’t incentivize them again with a smaller model or something.

所以奇怪的是他们不能用更小的模型或其他东西再次激励他们。

Maybe we’re just in this weird uncanny valley where trying to copy that frontier model too much, the student-teacher gap, whatever it is, is just too large.

也许我们正处于这个奇怪的恐怖谷，过度地试图复制前沿模型，学生与教师之间的差距，无论是什么，都太大了。

People have made this point with Opus.

人们已经用 Opus 表达了这一点。

The difference between Opus 4.6 and Opus 5 is that Opus 5 really feels like it’s got this AI-as-a-judge checking every possible thing it’s done.

Opus 4.6 和 Opus 5 之间的区别在于，Opus 5 确实让人感觉它有一个人工智能法官来检查它所做的每一件可能的事情。

That’s why it uses so many tokens.

这就是它使用如此多代币的原因。

It tries to think about all these things, but it doesn’t necessarily have the big model smell of Fable to know when to stop doing that, or when’s a good path to go down.

它试图思考所有这些事情，但它不一定具有《神鬼寓言》的大模型味道来知道何时停止这样做，或者何时是走下去的好路。

I would offer a slightly different hypothesis.

我会提供一个稍微不同的假设。

I would say there are a couple of different axes for the environments you can create.

我想说，您可以创建几个不同的环境轴。

One of them is difficulty and the other is realism.

一是难度，二是现实性。

It’s comparatively easy to create a lot of difficult environments that involve doing a much more complicated task or doing something that requires a lot more cleverness.

创建许多困难的环境相对容易，其中涉及执行更复杂的任务或执行需要更多聪明的事情。

You could say this is the benchmaxxing distribution, because a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task that’s easy to verify.

你可以说这是 benchmaxxing 分布，因为许多最著名的基准测试只涉及执行一些非常困难的类似谜题的任务，很容易验证。

Then there’s the realism axis, where you want the model to be good in the realistic coding agent setting where there’s multiple back-and-forths with the human and there’s multiple objectives.

然后是现实主义轴，您希望模型在现实编码代理设置中表现良好，其中与人类进行多次来回操作并且有多个目标。

The labs who are crafting the model behavior for the first time need to push in both directions.

首次构建模型行为的实验室需要双向推动。

To get good model behavior, you need to really push on the realism axis and have rubrics or some kind of human feedback that’s informing the reward function you use there.

为了获得良好的模型行为，您需要真正推动现实主义轴，并有规则或某种人类反馈来通知您在那里使用的奖励函数。

But if you try to do distillation naively, you end up just matching the teacher on the benchmaxxing distribution.

但如果你天真地尝试进行蒸馏，你最终只会在 benchmaxxing 分布上与老师匹配。

If you don’t have enough of the environments that really exercise the capabilities in these trickier realistic settings, then you’re not going to get those into your student model.

如果您没有足够的环境来真正锻炼这些更棘手的现实环境中的能力，那么您就不会将这些纳入您的学生模型中。

I think maybe one thing that’s happening is the big models generalize better from the tricky narrow tasks to these more realistic tasks.

我认为也许正在发生的一件事是，大模型可以更好地从棘手的狭窄任务推广到这些更现实的任务。

If you have a really good realistic prompt distribution for distillation, you can match the big model really well.

如果你有一个非常好的现实的蒸馏提示分布，你可以很好地匹配大模型。

But if you only have this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks, but you do worse on this broader distribution.

但是，如果您只有这种易于验证的任务分布，那么您可以在所有基准上匹配大模型，但在这种更广泛的分布上表现更差。

That might even explain something about the smaller Anthropic models, like Sonnet 5, though it’s hard to predict exactly what they’re doing to post-train those models.

这甚至可以解释一些较小的人择模型，如 Sonnet 5，尽管很难准确预测他们在训练这些模型后会做什么。

It could also be that they’re always changing their post-training stack, and they just got a few things wrong in some of these models.

也可能是他们总是在改变训练后的堆栈，而他们只是在其中一些模型中出现了一些错误。

I don’t know…

我不知道……

they turned something up too high and created some quirks that people really don’t like.

他们把某些东西调得太高，并创造了一些人们真正不喜欢的怪癖。

It’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.

训练后很容易以某种方式搞砸，而这些方式不会出现在基准测试中。

Just one other very basic point is that the frontier AI labs buy all their data from big data companies.

另一个非常基本的一点是，前沿人工智能实验室的所有数据都是从大数据公司购买的。

The Chinese can also just buy the same data from data companies .

中国人也可以从数据公司购买相同的数据。

And they are.

他们确实如此。

Exactly.

确切地。

There’s a lot of people being annoyed about this, but if they have exactly the same data and they can buy that, they can also distill.

很多人对此感到恼火，但如果他们拥有完全相同的数据并且可以购买，他们也可以进行提炼。

It means it’s quite easy to keep up, really.

这意味着真的很容易跟上。

The other question I had is how the first models that are capable of automating AI R&D will actually be trained.

我的另一个问题是如何训练第一个能够自动化人工智能研发的模型。

There’s a toy version, which is this thing that Ryan was talking about.

有一个玩具版本，这就是 Ryan 所说的东西。

You just have GPT-8 try to build GPT-3 size models that are really good at inner loop type challenges: beating video games that require continual learning, or just getting to a certain loss with the least amount of compute, et cetera.

你只需让 GPT-8 尝试构建真正擅长应对内循环类型挑战的 GPT-3 大小模型：击败需要持续学习的视频游戏，或者只是用最少的计算量达到一定的损失，等等。

But John, I think you had an interesting point that maybe that’s not the way it actually will happen in practice.

但是约翰，我认为你有一个有趣的观点，也许这并不是实际发生的方式。

So I’m curious, by the point at which you have AIs that are actually capable of automating AI R&D, how are they probably trained?

所以我很好奇，当你拥有真正能够实现人工智能研发自动化的人工智能时，它们可能是如何训练的？

We’ll probably do some combination of learning from human feedback to absorb the researchers’ taste, and just creating a lot of practice environments which involve doing multi-step research projects.

我们可能会结合从人类反馈中学习来吸收研究人员的品味，并创建大量涉及进行多步骤研究项目的实践环境。

People will in practice do some combination of those two things and, each iteration, patch whatever seems to be most broken in the last iteration.

在实践中，人们会将这两件事结合起来，并且在每次迭代中修补上一次迭代中似乎最严重的问题。

Researchers will be using the AIs a lot and will notice that they have some consistent weaknesses.

研究人员将大量使用人工智能，并会注意到它们有一些一贯的弱点。

Those things will either be patched by collecting human feedback or creating environments.

这些问题将通过收集人类反馈或创建环境来修补。

Maybe a useful way to think about this is how much of the lineage we roll back and then let self-play from there.

也许思考这个问题的一个有用方法是我们回滚多少血统，然后从那里开始自我发挥。

In the limit, you’re picturing just giving them a GPU and maybe neural nets or something and saying, “Okay, figure out how to train a model to do these particular tasks.” The way it currently works is we go up to the very edge of the lineage and say, “Okay, here are the bugs Anthropic has found in their training stack in the last few months.

在极限情况下，你会想象只给他们一个 GPU，也许还有神经网络之类的东西，然后说：“好吧，弄清楚如何训练模型来完成这些特定任务。”目前的工作方式是我们走到谱系的最边缘并说：“好吧，这是 Anthropic 在过去几个月中在他们的训练堆栈中发现的错误。

We’ll turn those into environments.” You need to train and get better on the frontier.

我们将把它们变成环境。”你需要在边境训练并变得更好。

So you obviously lock in all the previous history of the lineage.

所以你显然锁定了该谱系的所有先前历史。

But you could imagine a world in which you roll back to before GRPO or something.

但你可以想象一个你回滚到 GRPO 或其他东西之前的世界。

Then you have environments which try to get it to discover the best form to RL models on, and then maybe you roll further and further back…

然后你的环境会尝试让它发现 RL 模型的最佳形式，然后也许你会越来越回滚……

But I think we will still be so compute bottlenecked that people will just keep staying at the frontier and essentially diffing the bugs and whatever improvements they found since the last model version, turning those into training environments.

但我认为我们仍然会遇到计算瓶颈，人们只会继续停留在前沿，从本质上区分错误以及自上一个模型版本以来发现的任何改进，将它们转变为训练环境。

Which is also really good for having non-stale, new data between model generations.

这对于在模型代之间拥有非陈旧的新数据也非常有用。

Again, this is basically continual learning within the AI lab, of distilling the last three months of AI research progress through environments and RLHF-type stuff back into the model itself.

同样，这基本上是人工智能实验室内的持续学习，通过环境和 RLHF 类型的东西将过去三个月的人工智能研究进展提炼回模型本身。

And it is distilling.

它正在蒸馏。

That’s maybe why some of us feel like it’s asymptotic.

这也许就是为什么我们中的一些人感觉它是渐近的。

You’re always just trying to get the last three months of progress.

你总是只是想取得过去三个月的进步。

That progress is being contributed to by AIs, of course, but it also still has humans in the loop.

当然，这一进步是由人工智能做出的贡献，但人类也参与其中。

It feels like you’re just constantly inching closer and closer to what the human researchers are finding and capable of doing.

感觉就像你在不断地接近人类研究人员的发现和能力。

The one thing I will say, though, is obviously if you’re just distilling on trajectories, you can never go above it.

不过，我要说的一件事是，显然如果你只是在轨迹上提炼，你永远无法超越它。

But environments can go quite a far way above what a human can do.

但环境的影响远远超出人类的能力。

It’s very easy to design an environment that no human can solve, but the AI can obviously still try and solve it.

设计一个人类无法解决的环境很容易，但人工智能显然仍然可以尝试解决它。

That would be the path to go ahead of just what the human AI research is.

这将是超越人类人工智能研究的道路。

I feel like in AI research especially, it’s very easy to define goals.

我觉得特别是在人工智能研究中，定义目标非常容易。

You could say the loss needs to be 1.3 or something, and no human can get that now.

你可以说损失需要是 1.3 左右，但现在没有人能做到这一点。

But that’s an extremely measurable, verifiable task.

但这是一项极其可衡量、可验证的任务。

If the AI gets that, then great.

如果人工智能能做到这一点，那就太好了。

Or I don’t know, building a 100 million parameter model that beats Minecraft.

或者我不知道，构建一个击败 Minecraft 的 1 亿参数模型。

That’s maybe too easy, but beats a much more complicated game or something.

这可能太简单了，但胜过更复杂的游戏或其他游戏。

Isn’t it crazy that 100 million parameter models beat Minecraft ?

1亿参数模型打败Minecraft是不是很疯狂？

We’re calling that too easy?

我们说这太容易了？

Imagine if you said that five years ago.

想象一下，如果你五年前这么说。

I would say a lot of research is not exactly like that, though, where it’s hill climbing on a well-defined goal.

不过，我想说的是，很多研究并不完全是这样，它是在朝着明确的目标爬山。

It’s more like, here’s an intuition we have about some way models should be better.

更像是，这是我们对模型应该更好的某些方式的直觉。

We also have some idea for an algorithm that seems to go a little bit in this direction.

我们也有一些算法的想法，似乎有点朝着这个方向发展。

So let’s come up with a task that is designed to show signs of life on this approach, and see if we get those signs of life.

因此，让我们提出一个旨在通过这种方法显示生命迹象的任务，看看我们是否得到了这些生命迹象。

If we do, we can make successively more realistic versions of the task.

如果我们这样做，我们就可以连续制作该任务的更现实的版本。

It’s a lot more guided by intuition.

它更多地是由直觉引导的。

The inner loop is to test for that intuition rather than the test itself leading to the insight.

内循环是测试直觉，而不是测试本身带来洞察力。

Right.

正确的。

You’re not directly optimizing for the eventual objective you care about or the practical production objective.

您并不是直接针对您关心的最终目标或实际生产目标进行优化。

You’re relaxing your objective a little bit.

你稍微放松了你的目标。

You’re saying, “Let’s relax on the realism axis a little bit and find some methods that actually work, and then try to get back to realism later after the method matures a little bit.”

你是说，“让我们在现实主义轴上放松一点，找到一些真正有效的方法，然后在方法成熟一点后尝试回到现实主义。”

There’s also research that’s more oriented towards explaining things and developing a theory.

还有一些研究更倾向于解释事物和发展理论。

Often we don’t have mathematical theories in machine learning that are that predictive.

通常我们在机器学习中没有那么具有预测性的数学理论。

But we have a lot of more informal theories for what’s going on.

但对于正在发生的事情，我们有很多更非正式的理论。

Presumably the models will be trained on some combination of all of these tasks.

据推测，模型将接受所有这些任务的某种组合的训练。

Some will be very easily verifiable, some will be LLM-as-a-judge or just ask the human, “Does this look reasonable?” The hope would be that these would all generalize to these much harder, more vague, fuzzy kinds of tasks.

有些很容易验证，有些将是法学硕士作为法官，或者只是问人类，“这看起来合理吗？”希望这些都能够推广到这些更困难、更模糊、模糊的任务。

It probably will to some extent.

可能会在某种程度上。

Whether it generalizes enough that the loop can become self-sealing without humans being in the loop at all is unclear.

目前还不清楚它是否足够普遍，以至于环路可以在没有人类参与的情况下自我密封。

Maybe taking a step back.

也许退一步。

Here’s what it seems to me the plan for AI research going forward is.

在我看来，人工智能研究的未来计划是这样的。

You tell me if you think it’s going to work or if you agree with this characterization.

你告诉我你是否认为它会起作用或者你是否同意这种描述。

The bet is that we will scale up RLVR training across millions of diverse environments, across hundreds of different kinds of domains.

我们的赌注是，我们将在数百万个不同环境、数百个不同领域中扩大 RLVR 训练规模。

What will emerge at the other end is an agent which has learned these basic skills — or less than basic skills — around being persistent, being able to triage information and context, eventually having end-to-end optimization of working with other agents and things like that.

另一端将会出现的是一个代理，它已经学习了这些基本技能（或少于基本技能），围绕着坚持不懈，能够对信息和上下文进行分类，最终实现与其他代理合作的端到端优化以及类似的事情。

Such an agent will be very sample efficient within the context —you’ve done research on how you scale up in-context learning to make it arbitrarily long, but you keep scaling it up.

这样的代理在上下文中将非常高效——你已经研究了如何扩展上下文学习以使其任意长，但你不断扩展它。

And what comes out the other end will be something that basically functions like a drop-in remote worker over the course of a week or a month.

另一端的功能基本上就像远程工作人员在一周或一个月的时间内发挥的作用。

First of all, do you agree that that is the bet the labs are making?

首先，您是否同意这是实验室的赌注？

And second, is that enough?

其次，这就足够了吗？

Basically learning how to learn within these simulacra within a data center, and then getting deployed into the real world, but not actually learning from real-world deployment…

基本上学习如何在数据中心内的这些拟像中学习，然后部署到现实世界中，但实际上并没有从现实世界的部署中学习......

only learning these meta skills from the simulated environments in the data center.

仅从数据中心的模拟环境中学习这些元技能。

I think it’s now hard to separate out how much of the labs’ effort is going towards direct RSI versus making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run.

我认为现在很难区分实验室有多少努力是用于直接 RSI，还是制作一般智能的模型，他们可以继续部署这些模型来收集收入，为下一次大型培训运行提供资金。

For the latter, yes, that’s probably just the bet they’re making.

对于后者，是的，这可能只是他们所做的赌注。

It’s very clear, the pattern of where these environments are going over the last few years.

过去几年这些环境的发展模式非​​常清楚。

Anthropic’s lineage of environments is a very clear example of this.

人类的环境谱系就是一个非常明显的例子。

First, we just focus on coding and we’re going to get really, really good at that.

首先，我们只专注于编码，并且我们会非常非常擅长这一点。

Then, from the task horizon that we’ve got from coding — which is probably the lowest-hanging fruit in terms of data available on the internet to create environments, and their own internal stuff that they can turn into environments — then we’re going to generalize.

然后，从我们从编码中获得的任务范围——就互联网上可用于创建环境的数据而言，这可能是最容易实现的成果，以及他们可以将其转化为环境的内部东西——然后我们将进行概括。

We’re going to go up to finance next, and literally just so much Excel data and all that sort of stuff in the RL training.

接下来我们将进入金融领域，实际上就是 RL 训练中的大量 Excel 数据和所有此类内容。

Then it’s PowerPoints.

然后是 PowerPoint。

It’s this long tail of the working economy.

这就是工作经济的长尾。

That seemed to work really well.

这看起来效果非常好。

A lot of the other labs, even the open source labs, have now realized that that was the correct bet to make.

许多其他实验室，甚至开源实验室，现在都意识到这是正确的选择。

But what is the implication from that?

但这意味着什么呢？

When I had Dario on the podcast, the thing I asked him was, if you truly expect models which will be human-like in their ability to learn on the job, why would you try to bake in all these skills of working with PowerPoint or something?

当我在播客上邀请 Dario 时，我问他的是，如果你真的期望模型在工作中学习的能力与人类相似，那么你为什么要尝试融入所有这些使用 PowerPoint 或其他东西的技能呢？

Wouldn’t you just expect the model to be able to pick that up while it’s deployed?

您难道不希望模型能够在部署时识别它吗？

There’s multiple different explanations.

有多种不同的解释。

One is just that we expect models to get there soon, but they’re not there yet, so why not amortize these skills into the model training?

一是我们期望模型很快就能实现这一目标，但它们还没有实现，那么为什么不将这些技能分摊到模型训练中呢？

Another is that we’re not concentrated on making it really good at widely deployed work.

另一个原因是我们并没有专注于让它真正擅长广泛部署的工作。

We just want it really good at RSI.

我们只是希望它在 RSI 方面表现出色。

This is just a way for us to get revenue so that we can pour it back into a model that is actually really good at doing RSI development.

这只是我们获得收入的一种方式，以便我们可以将其重新投入到实际上非常擅长 RSI 开发的模型中。

Then once the singularity happens, the thing that comes out the other end will be really good at all the things which seem like bottlenecks to the current generation of models.

一旦奇点发生，另一端出现的东西将非常擅长解决当前一代模型看似瓶颈的所有事情。

John, I don’t know if you have takes on how one should construe why there is so much task-specific knowledge in these models if the path is this kind of generalization.

约翰，我不知道你是否认为如果路径是这种概括，人们应该如何解释为什么这些模型中有这么多特定于任务的知识。

If the models were good enough at learning in context, then in theory, you wouldn’t need to train them on finance.

如果模型足够擅长在上下文中学习，那么从理论上讲，你不需要对它们进行金融方面的培训。

They would just be able to read all the books on the fly and figure out how to do everything in the appropriate jurisdiction.

他们将能够即时阅读所有书籍并弄清楚如何在适当的管辖范围内做所有事情。

You could argue that you need to do a lot of this domain-specific training just to make them more efficient.

您可能会说，您需要进行大量特定领域的培训，只是为了提高它们的效率。

Even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights , so the model would be more efficient at runtime.

即使他们足够聪明，可以即时解决这个问题，您仍然可能需要进行大量强化学习，并将所有这些直觉融入到权重中，这样模型在运行时会更加高效。

In practice, it does seem like model providers are going domain by domain and trying to strengthen the models in the highest value domains.

在实践中，模型提供商似乎确实在逐个领域进行研究，并试图加强最高价值领域的模型。

I’d say that that’s one of the answers to why the models have gotten so much better.

我想说，这就是为什么模型变得如此出色的答案之一。

It’s just because the model providers have covered a lot of the high-value domains and the most common types of skills.

只是因为模型提供者涵盖了很多高价值领域和最常见的技能类型。

Another thing is just that it’s not that expensive to do both at the same time.

另一件事是，同时做这两件事并不那么昂贵。

The models are massive.

模型很大。

They can easily afford, in terms of their parameters, to learn everything.

就他们的参数而言，他们可以轻松地负担得起学习一切的费用。

There is likely some transfer.

可能有一些转移。

Even if finance is not specific, the information is important for RSI.

即使财务信息不具体，这些信息对于 RSI 也很重要。

Just the general meta-learning of how to figure out what’s important, how to have taste, how to do long-horizon work is potentially generalizable.

只是关于如何弄清楚什么是重要的、如何有品味、如何进行长期工作的一般元学习就可能具有普遍性。

There’s not that much RSI data in the world as well.

世界上也没有那么多 RSI 数据。

It’s hard to generate and requires a lot of effort.

生成起来很困难，需要付出很大的努力。

So if you can amortize in this other data, you get some transfer from it.

因此，如果您可以摊销其他数据，您就会从中获得一些转移。

You already have masses of compute and masses of parameter space, so why not do that as well as, obviously, the direct commercial intent of selling a model?

您已经拥有大量计算和大量参数空间，那么为什么不这样做以及销售模型的直接商业意图呢？

I’ll add that there’s one question about whether this current paradigm of doing sim-to-real will be the dominant one forever.

我要补充一点，有一个问题是，当前这种从模拟到真实的范式是否会永远占据主导地位。

You look at what the real-world tasks are like.

您看看现实世界的任务是什么样的。

Then you try to create a bunch of environments that can be simulated in the data center, and you can do RL on them.

然后你尝试创建一堆可以在数据中心模拟的环境，你可以对它们进行强化学习。

Obviously, this has been very successful.

显然，这是非常成功的。

But it has a lot of weaknesses, because a lot of things are just hard to simulate, especially if they involve interacting with a bunch of humans in real time.

但它有很多弱点，因为很多事情都很难模拟，特别是当它们涉及与一群人实时交互时。

So there’s some question about whether sim-to-real will be the dominant framework forever.

因此，存在一些关于“模拟到真实”是否会永远成为主导框架的问题。

I think sim-to-real has to be the dominant framework while sample efficiency is low, because right now you need thousands and thousands of interactions with the humans.

我认为模拟到真实必须成为主导框架，而样本效率很低，因为现在你需要与人类进行成千上万次交互。

No human is going to sit there and be in the loop of RL training.

没有人会坐在那里参与强化学习训练。

So we have to simulate that now to get the samples you need.

所以我们现在必须进行模拟以获得您需要的样本。

But obviously, if sample efficiency improves a lot, you’d expect learning from deployment to become a much bigger part of it.

但显然，如果样本效率提高很多，您会期望从部署中学习将成为其中更重要的一部分。

Though there are also other things you could do.

尽管您还可以做其他事情。

You can learn off-policy , so you can take all the traces, and even without resimulating everything, you can potentially learn something from them.

您可以进行off-policy学习，这样您就可以获取所有痕迹，即使不重新模拟所有内容，您也可以从中学到一些东西。

I want to ask more about this, because it’s weird that you have 50% of compute that’s spent on inference that is not directly helping the model become better.

我想问更多关于这个问题的问题，因为奇怪的是，你有 50% 的计算花费在推理上，而这并不能直接帮助模型变得更好。

One of the key advantages you’d expect digital minds to eventually have is that, unlike a human who gets to have 50 years of real-world experience, a model will get to experience, through all its instances, millions of years of deployment across all kinds of economically relevant work in the economy.

您期望数字思维最终拥有的关键优势之一是，与拥有 50 年现实世界经验的人类不同，模型将在其所有实例中经历数百万年的经济中各种与经济相关的工作的部署。

Right now, that data is just not, in a meaningful sense, helping the model get better.

目前，从有意义的意义上来说，这些数据并不能帮助模型变得更好。

It seems so obvious that eventually models should be able to learn from this data.

显然，模型最终应该能够从这些数据中学习。

Once they do, you would have something that almost feels like a widely deployed intelligence explosion, because the model is assimilating so much information across all these deployed instances.

一旦他们这样做，你就会感觉几乎像是广泛部署的情报爆炸，因为该模型正在吸收所有这些部署实例中的大量信息。

When do you expect this kind of hive-mind, crazy shit to start happening?

你预计这种蜂巢思维、疯狂的事情什么时候会开始发生？

I think broadly, at a very basic level, this is already happening…

我认为从广义上讲，在非常基本的层面上，这种情况已经发生了……

just in the next generation of models.

就在下一代模型中。

Right now, you can obviously take your deployment data and put this in the pre-train or the mid-train of future models, especially if you do some kind of filtering or some kind of judgment or annotation or synthesization of that.

现在，您显然可以将部署数据放入未来模型的预训练或训练中，特别是如果您进行某种过滤或某种判断、注释或综合。

I think it explains quite a bit.

我认为这已经解释了很多。

I don’t know whether the labs do this, because theoretically, they claim not to train on people’s data.

我不知道实验室是否这样做，因为从理论上讲，他们声称不会用人的数据进行训练。

But the Chinese 100% do.

但中国人100%这样做。

They definitely get this advantage.

他们肯定会得到这个优势。

This is basically what distillation is.

这基本上就是蒸馏。

They take the models, they get some fraction of their deployment data by pinging the model, and then they train their next generation of models on it.

他们获取模型，通过 ping 模型获取部分部署数据，然后在其上训练下一代模型。

They can certainly do it on their own models as well.

他们当然也可以在自己的模型上做到这一点。

There’s no reason not to whatsoever.

没有理由不这样做。

I completely agree with this.

我完全同意这一点。

If you zoom out far enough, this is definitely happening.

如果你把镜头拉得足够远，这肯定会发生。

What we’re all picturing, the holy grail of continual learning, is this very organic, live loop of an individual model getting an experience and live-updating on the spot and learning from that.

我们所想象的持续学习的圣杯，是一个非常有机的、实时的循环，单个模型获得经验并现场实时更新并从中学习。

A lot of things break when you zoom into that level of granularity.

当你放大到这个粒度级别时，很多事情都会崩溃。

But the big labs are doing this.

但大型实验室正在这样做。

The closed models are doing this.

封闭模型正在这样做。

There are also early signs of life of people using open-source models doing this at a much faster cadence.

也有早期迹象表明，人们使用开源模型以更快的节奏做到这一点。

A good example is probably Composer .

Composer 可能是一个很好的例子。

Harvey ’s doing the same thing with legal agents.

哈维正在与法律代理人做同样的事情。

You have some sort of model, and you are getting very specific environments from the data that you have for that particular task, and things that users are complaining about, and all the feedback that you’re somehow extracting from your specific deployments.

您有某种模型，并且您可以从该特定任务的数据中获得非常具体的环境、用户抱怨的事情以及您以某种方式从特定部署中提取的所有反馈。

A lot of these companies have the advantage over the big labs in that they can use this data really, really well.

这些公司中的许多公司比大型实验室具有优势，因为他们可以非常非常好地使用这些数据。

Then they will create environments.

然后他们会创造环境。

They will do a big post-train of Kimi K3.

他们将在《Kimi K3》之后进行大型后期训练。

They will go deploy it.

他们将去部署它。

They might do some online learning as well, like Composer did online…

他们也可能会进行一些在线学习，就像 Composer 在网上所做的那样……

basically REINFORCE for a long time.

基本上长期强化。

There’s still a human in the loop.

循环中仍然有人。

There’s still a human saying, “Okay, these are the signals we care about.

仍然有人说：“好吧，这些是我们关心的信号。

Here’s how we’re going to create environments from the data that we have.” It’s still a longer cadence than maybe the one that you’re thinking of, but it really is happening.

以下是我们如何利用现有数据创建环境的方法。”它的节奏可能比你想象的要长，但它确实正在发生。

Eventually that loop will become faster and faster.

最终这个循环会变得越来越快。

The Composer thing is interesting because this is where, in Cursor , people press Tab or they don’t press Tab on the next completion that the model suggests.

Composer 的事情很有趣，因为在 Cursor 中，人们按 Tab 键，或者在模型建议的下一个完成时不按 Tab 键。

Based on that, every single day, Composer gets better at predicting the next—

基于此，Composer 每天都能更好地预测下一个——

That was the old Tab model.

那是旧的 Tab 模型。

They actually did the same thing not just for the Tab model, but for the actual generative model.

他们实际上不仅对 Tab 模型做了同样的事情，而且对实际的生成模型也做了同样的事情。

It’s hard because when you do online reinforcement learning , you don’t have groups.

这很难，因为当你进行在线强化学习时，你没有小组。

You just have one user saying one thing, and then you get one rollout.

您只需让一位用户说一件事，然后您就可以进行一次部署。

So you have a big variance-reduction problem.

所以你有一个很大的方差减少问题。

Cursor’s fuzzy answer to this was, “We have very good heuristics which are able to estimate how much better than average this response was, or how much worse than average this response was.” Then they would do this big REINFORCE update.

Cursor 对此的模糊回答是，“我们有非常好的启发式方法，能够估计这个响应比平均水平好多少，或者比平均水平差多少。”然后他们会进行这个重大的强化更新。

Their solution to whether it got worse or not was that if it improved on CursorBench , they would deploy the new model every five hours.

他们对于情况是否变得更糟的解决方案是，如果在 CursorBench 上有所改善，他们将每五个小时部署一次新模型。

If it didn’t, they would throw that version out.

如果没有，他们就会丢弃该版本。

I think your biggest problem is actually just not knowing what the reward function should be for natural data.

我认为你最大的问题实际上只是不知道自然数据的奖励函数应该是什么。

If you use some kind of superficial signal, like did they accept the edit, that might get reward-hacked in some way.

如果您使用某种肤浅的信号，例如他们是否接受编辑，则可能会以某种方式受到奖励黑客攻击。

But isn’t this a bigger issue with the sim-to-real thing, where the longer-horizon tasks get, the harder they are to simulate within a data center?

但这对于模拟真实事物来说不是一个更大的问题吗？任务范围越长，在数据中心内模拟它们就越困难？

It seems to me that even in coding, we’re already getting to the point where there’s not some year-long coding task that doesn’t eventually require you to talk to a client or interact with the company or interact with users.

在我看来，即使在编码方面，我们也已经达到了这样的程度：没有一些长达一年的编码任务最终不需要你与客户交谈或与公司互动或与用户互动。

If you think about the gamut of things we would want AI to be capable of, eventually superintelligence should be able to run a business, or start a new business and make it profitable, or have a profitable day trading in the markets, or win a court case.

如果你考虑一下我们希望人工智能能够完成的所有事情，最终超级智能应该能够经营一家企业，或者开办一项新业务并使其盈利，或者在市场上进行有利可图的日内交易，或者赢得一场法庭诉讼。

These are all things which are very hard to simulate in a data center.

这些都是很难在数据中心模拟的事情。

An inherent part of the learning there is interacting with the real world.

在那里学习的一个固有部分是与现实世界的互动。

Maybe they learn how to get better at these things from the transfer between sim-to-real.

也许他们从模拟到真实的转换中学习如何在这些事情上做得更好。

But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them.

但或者，也许你确实需要从这些类型的交互中更新权重，以便更好地掌握它们。

If that is the case — if transfer isn’t strong enough and you do need weight updates — then the fact that the models are quite sample-inefficient is maybe a deeper problem.

如果是这种情况——如果转移不够强大并且你确实需要权重更新——那么模型的样本效率相当低这一事实可能是一个更深层次的问题。

The reason I’m curious about this is that by default, I don’t see how you don’t get some kind of crazy recursive self-improvement within the next 10 years.

我对此感到好奇的原因是，默认情况下，我不明白你如何在未来 10 年内不会获得某种疯狂的递归自我完善。

But the one reason why that might not happen is that in terms of the sample efficiency of weight updates, models just seem way far behind humans.

但这种情况可能不会发生的原因之一是，就权重更新的样本效率而言，模型似乎远远落后于人类。

They’re plausibly a millionfold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from cold start to finishing training.

就人类从出生到成年看到的数据量与模型从冷启动到完成训练看到的数据量而言，它们似乎落后人类一百万倍。

This is all to say, first of all, is there going to be good transfer between simulations and the extremely long-horizon, really complicated real shit that we want the AI to do in the real world?

这就是说，首先，模拟和我们希望人工智能在现实世界中做的极其遥远、非常复杂的真实事情之间是否会有良好的转移？

And if not, does that really mean that the lack of sample efficiency in these models comes to bite us?

如果不是，这是否真的意味着这些模型中样本效率的缺乏会困扰我们？

Maybe the way I’d break down the two types of tasks — the ones in which models get good and the ones where models will still continue to struggle — is whether the task is cumulative, or whether you have this non-stationary distribution where you have to keep learning and relitigating a bunch of stuff.

也许我分解两种类型的任务的方式——模型变得很好的任务和模型仍然会继续挣扎的任务——是任务是否是累积的，或者是否有这种非平稳分布，你必须不断学习和重新处理一堆东西。

An example of a cumulative task might be RSI.

RSI 就是累积任务的一个例子。

It’s theoretically possible to have a less-than-a-million-token Python file which from scratch trains a model that is capable of recursive self-improvement.

理论上，有一个少于一百万个令牌的 Python 文件可以从头开始训练一个能够递归自我改进的模型。

Every discovery that you make is a line in the sand that you hold.

你的每一个发现都是你手中的一条线。

If it’s true that for RSI we don’t need to discover a new attention variant or whatever, then once you’ve discovered attention , and once you’ve discovered mixture of experts , and once you discover GRPO , you just add that to the training stack and that’s there.

如果对于 RSI 来说，我们确实不需要发现新的注意力变体或其他什么，那么一旦你发现了注意力，一旦你发现了专家的混合，一旦你发现了 GRPO，你只需将其添加到训练堆栈中即可。

A good example of this is 5.6 Sol training, 5.6 Terra, or whichever one OpenAI told us it trained.

一个很好的例子是 5.6 Sol 训练、5.6 Terra 或 OpenAI 告诉我们它训练过的任何一个。

It didn’t have to go back and discover attention.

它不必回去发现注意力。

It basically would have called a bunch of scripts, like pre-training.sh and post-training.sh, and just done that.

它基本上会调用一堆脚本，例如 pre-training.sh 和 post-training.sh，然后就完成了。

That’s an example of a cumulative task.

这是累积任务的一个例子。

I think the real world — and the reason people are thinking so much about continual learning — is not really a cumulative task.

我认为现实世界——以及人们如此多地思考持续学习的原因——并不是真正的累积任务。

Imagine in a law firm, you have an agent acting as a legal associate.

想象一下，在一家律师事务所，您有一名代理人担任法律助理。

That’s a very non-stationary distribution.

这是一个非常不稳定的分布。

You have to be able to fit in your context all the relationships between all the important people at that company, which are also changing all the time.

您必须能够将公司所有重要人员之间的所有关系融入到您的环境中，而这些关系也在不断变化。

You have all these implicit ways about how things are done, where to find information, et cetera.

关于如何完成工作、在哪里查找信息等等，您有所有这些隐含的方式。

That’s not as clean an example of a cumulative task as RSI is.

这并不是 RSI 那样清晰的累积任务示例。

I think there will be this breakdown between tasks.

我认为任务之间会有这种细分。

But if the labs realize that — and they do believe that RSI is cumulative in the sense that we don’t need to go back and discover some brand-new architecture or whatever — then maybe more and more effort and compute gets focused on that versus the other tasks.

但如果实验室意识到这一点——而且他们确实相信 RSI 是累积性的，因为我们不需要回去发现一些全新的架构或其他什么——那么也许越来越多的努力和计算会集中在这一点上，而不是其他任务上。

I would say today’s models are weaker than humans in a lot of different ways.

我想说，今天的模型在很多方面都比人类弱。

Some of them might have to do with sample efficiency in a certain regime.

其中一些可能与特定情况下的样本效率有关。

In some regimes, models are very sample-efficient, like learning in context.

在某些情况下，模型的样本效率非常高，就像在上下文中学习一样。

But then there might be some medium-length regime where they’re less sample-efficient, because humans can do some kind of weight update more efficiently than models.

但可能存在一些中等长度的机制，其中样本效率较低，因为人类可以比模型更有效地进行某种权重更新。

I think being less sample-efficient in certain regimes might be one of the sources of weakness.

我认为在某些制度下样本效率较低可能是弱点的根源之一。

But I think there are other sources of weakness that are completely different from that.

但我认为还有其他与此完全不同的弱点来源。

For example, having lower diversity of thought than humans, or being bad at certain kinds of long-horizon judgments.

例如，思想的多样性低于人类，或者不擅长某些长期判断。

I think a lot of what people call taste is something about behavior that works in the long run, and that people have realized works in the long run.

我认为人们所谓的品味很多都是关于长期有效的行为，人们已经意识到从长远来看有效。

Not everything, but some aspect of taste.

不是一切，而是品味的某些方面。

Especially for something like software engineering, I think a lot of taste is “What are the systems that are going to be maintainable and work well in the long run of this project?”

特别是对于像软件工程这样的东西，我认为很多人的品味是“从这个项目的长期运行来看，哪些系统是可维护的并且运行良好的？”

There are a variety of weaknesses of models which limit RSI along with other things.

模型存在多种弱点，这些弱点会限制 RSI 以及其他因素。

Some of them are related to sample efficiency, and some of them aren’t.

其中一些与样本效率有关，有些则无关。

Maybe an interesting thought experiment is this.

也许这是一个有趣的思想实验。

Let’s say you were able to give a model a context window of a trillion tokens, or whatever you would have needed to fit in your experience prior to, let’s say, RLHF.

假设您能够为模型提供一万亿代币的上下文窗口，或者您在 RLHF 之前需要适应您的体验的任何内容。

It’s got all that experience in the context window, and it has the same sample efficiency and in-context learning ability as it does at a million tokens.

它在上下文窗口中获得了所有经验，并且具有与一百万个令牌相同的样本效率和上下文学习能力。

Do you think taste is then solved?

你觉得味道就解决了吗？

Would it be able to make the same judgments that you did?

它能做出和你一样的判断吗？

Or is there something fundamentally missing, apart from just a longer context window with the same sample efficiency?

或者除了具有相同采样效率的更长上下文窗口之外，是否存在根本性缺失的东西？

It would have to be trained to learn from that context.

它必须经过培训才能从这种环境中学习。

Either it would have to be trained to learn the right update to make from that context, or it would have to generalize.

要么必须对其进行训练，以学习从该上下文中进行正确的更新，要么必须进行概括。

So you don’t think you can just dump it all in, your whole life, your research experience?

所以你不认为你可以把你的一生、你的研究经历全部扔进去吗？

You still need the data to train it on long context.

您仍然需要数据来在长上下文中训练它。

Even if you could theoretically get a trillion context, you would need a trillion lengths of data to train it.

即使理论上您可以获得一万亿个上下文，您也需要一万亿长度的数据来训练它。

Right now you have 10k context you can’t just dump in a million.

现在你有一万个上下文，你不能直接转储一百万个。

In theory, I think, yes.

从理论上讲，我认为是的。

This really just comes down to the question of how meta-learnable taste is from shorter-horizon episodes.

这实际上归结为一个问题：如何从较短的视野中元学习品味。

I feel like there’s no obvious reason it’s super long, because humans somehow developed taste without having many long episodes.

我觉得没有明显的理由让它超长，因为人类在没有很多长剧集的情况下以某种方式发展了品味。

We don’t live to be 10,000.

我们活不到一万岁。

We develop pretty quickly.

我们发展得很快。

If you think about even a PhD, the difference between a first-year PhD student and a final-year student or postdoc, that’s five years maybe.

如果你考虑一下博士学位，第一年的博士生和最后一年的学生或博士后之间的区别，可能就是五年。

They’ve only done maybe 10-30 research projects in total.

他们总共只完成了 10-30 个研究项目。

But somehow they develop taste quite quickly from a relatively short succession of small things.

但不知何故，他们从相对较短的连续小事中很快就培养出了品味。

Theoretically, it’s possible to develop it like that.

理论上是可以这样开发的。

The AI obviously will have vastly more experience in which to develop taste, to meta-learn it.

人工智能显然将拥有更多的经验来培养品味，进行元学习。

Then the question is how well that generalizes to really long-horizon things, which I think is really unsolved at this point.

接下来的问题是，这对长期事物的推广效果如何，我认为目前这个问题还没有解决。

We don’t know.

我们不知道。

Going back to this question, eventually there should be a regime where AIs are learning a ton from each individual instance of deployment that they have.

回到这个问题，最终应该有一个机制，让人工智能从它们拥有的每个单独的部署实例中学习大量内容。

Currently you could say there’s a fuzzy meta process by which models do improve from deployment.

目前，您可以说存在一个模糊的元过程，模型可以通过该过程从部署中进行改进。

But I feel like it’s a very weak feedback loop.

但我觉得这是一个非常薄弱的​​反馈循环。

Do you see this on the horizon, where there’s this hive mind kind of learning that’s very rapid, and if so, how exactly does it happen?

你是否在地平线上看到了这种情况，这种蜂巢思维的学习速度非常快，如果是这样，它到底是如何发生的？

I would say that whether we get a hive mind that learns from all of its deployment experience is in a big part about incentives, rather than being a technical question.

我想说的是，我们是否拥有一个能够从所有部署经验中学习的蜂巢思维，很大程度上取决于激励措施，而不是一个技术问题。

Companies aren’t going to want to have the model provider learn from all of their deployment, because that might just reduce the advantage of their business.

公司不会希望模型提供商从他们的所有部署中学习，因为这可能只会降低他们的业务优势。

I think the economics of this will pressure, not necessarily weight updates to one big common shared model, but modules that get subbed in.

我认为这样做的经济性会带来压力，不一定是对一个大型通用共享模型的权重更新，而是对替换的模块的权重。

A very obvious example of this is a LoRA , but it might be something else.

一个非常明显的例子是 LoRA，但也可能是其他东西。

There’s been a lot of work to try and fit an arbitrary context length into a fixed size.

我们做了很多工作来尝试将任意上下文长度调整为固定大小。

This is all the linear attention stuff.

这就是所有线性注意力的东西。

And cartridges , which are essentially KV caches trained to be very, very compressed to fit in a lot of information.

还有卡带，它本质上是 KV 缓存，被训练得非常非常压缩以适应大量信息。

That’s another example of something that companies may be willing to sign up for, if that gets subbed into the model and it’s not actually changing the base underlying model itself.

这是公司可能愿意签署的另一个例子，如果它被嵌入到模型中并且实际上并没有改变基本的底层模型本身。

There are many different versions of learning from your data in real time.

从数据中实时学习有许多不同的版本。

The latter ones are not really helping the big labs, because they are just these modules.

后者并不能真正帮助大型实验室，因为它们只是这些模块。

But I think the economic pressure will force the labs to go down that path first before they can embark on this...

但我认为经济压力将迫使实验室先走这条路，然后才能开始……

Which economic pressure, though?

但有哪些经济压力呢？

I feel like even if you have a bunch of cartridges or LoRAs or whatnot, you can still just take all these traces and dump them into the pre-training of your next generation of models.

我觉得即使你有一堆卡带或 LoRA 之类的东西，你仍然可以获取所有这些痕迹并将它们转储到下一代模型的预训练中。

Yes.

是的。

It may be a more indirect form of learning that the big labs are getting.

这可能是大型实验室正在获得的一种更为间接的学习形式。

That’s obviously still really valuable to them.

这显然对他们来说仍然非常有价值。

But I can’t imagine a world in which we start off with, “We’re going to just directly train this one big model on all the exact data that we’re getting.”

但我无法想象一个世界，我们一开始就说：“我们将根据我们获得的所有精确数据直接训练这个大模型。”

No, I think it will definitely go through stages, because this is assuming there’s one discontinuous event where suddenly we fix weight updates continuously.

不，我认为它肯定会经历几个阶段，因为这是假设有一个不连续的事件，我们突然连续修复权重更新。

In practice, I think it’s much more likely to be that the cartridges and stuff allow you to specialize in deployments.

在实践中，我认为更有可能的是，墨盒和其他东西可以让您专门进行部署。

Then you generate traces, you put that in your model, and three months later you come out with a model which is better at this stuff.

然后你生成痕迹，将其放入你的模型中，三个月后你就会得到一个在这方面做得更好的模型。

You specialize it again, you consolidate it again, and then eventually we’ll just make this leap faster and faster.

你再次专业化它，你再次巩固它，然后最终我们会让这一飞跃变得越来越快。

Instead of releasing a model every three months, now it’s every week, and then every day, and then every hour, at which point we’ve basically solved it.

不再是每三个月发布一个模型，现在是每周，然后是每天，然后是每小时，到那时我们就基本上解决了。

I think this is a good point as well, because you asked how far off the current paradigm we are from being able to do this.

我认为这也是一个很好的观点，因为你问我们距离当前的范例还有多远才能做到这一点。

We’ve done a bit of research into this, and people have done a lot of research.

我们对此做了一些研究，人们也做了很多研究。

At a really large scale, when you wash out enough noise and you have large enough batches, this outer-loop process of putting data into mid-training and creating our own environments does work in some sort of continual learning regime.

在非常大的规模上，当你消除足够的噪音并且拥有足够大的批次时，这种将数据放入训练中期并创建我们自己的环境的外循环过程确实在某种持续学习机制中起作用。

But the problem is, when you zoom in close enough at a micro level — I’ve got one model and I’m trying to update it again for a law firm or something, and I’m trying to do that very continuously with a relatively small amount of data — all the methods kind of break down a bit.

但问题是，当你在微观层面上放大得足够近时——我有一个模型，我试图为一家律师事务所或其他机构再次更新它，并且我试图用相对少量的数据连续地做到这一点——所有的方法都会有点崩溃。

If I SFT the model on just successful traces, off-policy or on-policy, eventually in the very iterative regime, when you’re doing hundreds of these micro-updates, you see catastrophic forgetting .

如果我仅在成功的跟踪上对模型进行 SFT，无论是离策略还是在策略，最终在非常迭代的状态下，当你进行数百次这样的微更新时，你会看到灾难性的遗忘。

You see forgetting of previous information learned on top of the base model that was much earlier on, and you see degradation of general capabilities.

您会看到在更早的基础模型之上学到的先前信息被遗忘，并且您会看到一般功能的退化。

On-policy distillation seems to push this horizon out a little bit, but it still eventually succumbs to the same thing.

在政策蒸馏似乎将这一视野推开了一点，但它最终仍然屈服于同样的事情。

RL is good at getting capabilities in, but it’s not as good at getting knowledge in, this very explicit knowledge of, “Ah, okay, this person does this at this law firm, and this is a very specific process we find.” You have to pour in a lot of compute to create the right environments to get the knowledge in with RL.

强化学习擅长获取能力，但不擅长获取知识，这种非常明确的知识，“啊，好吧，这个人在这家律师事务所做这件事，这是我们发现的一个非常具体的过程。”您必须投入大量计算来创建合适的环境来获取 RL 知识。

Do you think the fundamental issue here — why you get worse at these other skills or there’s forgetting — is fundamentally an issue of capacity or an issue of techniques?

你认为这里的根本问题——为什么你在这些其他技能上变得更差或者出现遗忘——从根本上来说是能力问题还是技术问题？

A little bit of both.

两者都有一点。

I think SFT and even on-policy distillation can be way too destructive.

我认为 SFT 甚至同策略蒸馏都可能具有太大的破坏性。

The reason RL is so nice is because it changes a very, very small amount about the model.

强化学习之所以如此出色，是因为它对模型的改变非常非常小。

There’s a lot of evidence for why this is the case.

有很多证据可以证明为什么会出现这种情况。

It just tweaks it in this very, very small loss valley to get it into the right point.

它只是在这个非常非常小的损失谷中进行调整，使其进入正确的点。

But that also then limits what you can do with RL, how much you can actually change the model.

但这也限制了你可以用强化学习做什么，你可以实际改变模型多少。

So you’re saying the reason this isn’t a winner-take-all, potentially, is that it is just very hard to distill that much information into the base model?

所以你是说这不是赢家通吃的原因，可能是因为很难将这么多信息提炼到基本模型中？

It’s easy to distill it into a different base model.

很容易将其提炼成不同的基本模型。

This is where I think it’s mostly technique.

我认为这主要是技术的地方。

It’s definitely not that there isn’t capacity.

绝对不是没有能力。

If you had some model with all this data, and you take literally the same-size model and pre-train it from scratch with all of the stuff in mid-training, it will be better.

如果您有一个包含所有这些数据的模型，并且您采用相同大小的模型并在训练中期使用所有内容从头开始对其进行预训练，那么效果会更好。

I think that’s a lot of what’s happening today.

我认为今天发生的很多事情都是如此。

There’s very much a bottleneck that stops us from just keeping training the same model forever, versus just getting all the data from the old model and training a new model from scratch.

有一个很大的瓶颈阻止我们永远训练同一个模型，而不是从旧模型中获取所有数据并从头开始训练新模型。

This is exactly as Charlie was saying: some combination of plasticity and catastrophic forgetting.

这正如查理所说：可塑性和灾难性遗忘的某种结合。

If you just naively train on non-stationary data, because you’re adding new data as you go, this is messing with the data distribution, so the old stuff is just forgotten.

如果你只是天真地训练非平稳数据，因为你在添加新数据时，这会扰乱数据分布，所以旧的东西就会被遗忘。

We don’t really have good methods to stop that from happening.

我们确实没有好的方法来阻止这种情况的发生。

So maybe in the limit you’re just bottlenecked by retraining the model from scratch with all this new information.

因此，也许在极限情况下，您只是通过使用所有这些新信息从头开始重新训练模型而遇到了瓶颈。

Yes, which of course is very expensive.

是的，这当然非常昂贵。

Training a model from scratch is expensive.

从头开始训练模型的成本很高。

Not necessarily.

未必。

Maybe eventually, if you have continual learning, you never train a new model.

也许最终，如果你不断学习，你就永远不会训练新模型。

You just have a model and it keeps learning and expanding.

你只有一个模型，它会不断学习和扩展。

That’s the question.

这就是问题所在。

I think we have pushed back how much from scratch we need to do.

我认为我们已经推迟了需要从头开始做的事情。

It is definitely possible now to take the pre-trained base and do very good mid-training on top of that, kind of continuously, plus some RL from different checkpoints that are later on in the training.

现在绝对有可能采用预训练的基础，并在此基础上进行非常好的中期训练，连续进行，再加上训练后期不同检查点的一些强化学习。

That’s looking more like continual learning, but it’s certainly not the case of taking the most recent model, applying a couple of very small updates, and iteratively never losing anything.

这看起来更像是持续学习，但肯定不是采用最新模型、应用一些非常小的更新、迭代地永不丢失任何内容的情况。

Sorry, but I’m a bit confused, because isn’t this literally what happens during training?

抱歉，但我有点困惑，因为这不是训练期间发生的事情吗？

During post-training or something, you have a model that’s already gone through so much training, and then you distill some fork that’s been further RL’d.

在训练后或其他过程中，你有一个已经经过如此多训练的模型，然后你提炼出一些经过进一步强化学习的分叉。

Isn’t that literally what happens?

这不是字面上发生的事情吗？

But it’s still at a large enough scale, I think, that you’re washing out a lot of the noise, and you’re not just focused on one distribution, which, as Beren said, is the issue.

但我认为，它的规模仍然足够大，你可以消除很多噪音，而且你不仅仅关注一种发行版，正如贝伦所说，这就是问题所在。

If you’re just focusing on one task—

如果你只专注于一项任务——

But in the eventual regime you’d be doing…

但在最终的制度中你会做……

There are billions of deployed instances.

有数十亿个已部署实例。

You’re learning from all of them at once, so hopefully there’s some washing out of noise from that.

你同时向他们所有人学习，所以希望能从中消除一些噪音。

As Charlie was saying, you can definitely do continual mid-training for a long time, and you can roll back to a checkpoint and give it new mid-training data.

正如查理所说，你绝对可以长时间进行持续的中期训练，并且可以回滚到某个检查点并为其提供新的中期训练数据。

But at the same time, you can’t do this indefinitely.

但与此同时，你不能无限期地这样做。

If you just keep continually training the same base forever, it asymptotes at some point.

如果你永远不断地训练同一个基础，它会在某个时刻渐近。

You can’t just learn new stuff in that base.

你不能只是在那个基地学习新东西。

This is why people end up training new bases.

这就是人们最终训练新基地的原因。

Otherwise you would just keep mid-training the same base forever.

否则你就会在训练中永远保持同一个基础。

Let’s talk a bit about data now.

现在我们来谈谈数据。

I’m generally interested in this question of how much of AI progress is just explained by data progress.

我通常对这个问题感兴趣：人工智能的进步有多少是由数据进步来解释的。

That doesn’t mean it will necessarily be hard to automate, but that’s a separate question.

这并不意味着它一定很难自动化，但这是一个单独的问题。

Is there some data distribution which, if you trained current architectures on it, would result in a superintelligence that totally dominates human experts across every single field?

是否存在某种数据分布，如果您在其上训练当前的体系结构，将导致超级智能在每个领域完全统治人类专家？

Are we talking about pre-training plus post-training data, environments as well?

我们是否也在谈论预训练和训练后的数据、环境？

I think the existence of this is obvious.

我认为这种现象的存在是显而易见的。

It’s just whether we can create the right environment to get there.

只是我们能否创造合适的环境来实现这一目标。

In the trivial case, we could just train it to output the Python file which trains the actual superintelligence.

在简单的情况下，我们可以训练它输出训练实际超级智能的 Python 文件。

Just have that memorized in the weights.

只需将其记住在权重中即可。

Yes, there’s probably a ladder of RL environments that is possible to construct such that you would get an AI researcher which is at least as good as a human researcher.

是的，可能有一个 RL 环境的阶梯可以构建，这样你就会得到一个至少与人类研究人员一样好的人工智能研究人员。

But the effort to climb each successive rung grows kind of exponentially.

但攀登每一个连续阶梯的努力呈指数级增长。

Those are the two things you have to trade off against as to how fast we’re going to hit that final rung where it’s better.

这是你必须权衡的两件事，以决定我们以多快的速度达到更好的最后一级。

I think that’s fairly clear.

我认为这相当清楚。

We’re still relatively early in RL environment creation.

我们在 RL 环境创建方面还处于相对早期的阶段。

There are a lot of asymmetries that we exploit in order to create good environments.

为了创造良好的环境，我们利用了许多不对称性。

One of the asymmetries which we’ve talked about before is that there are environments where it’s easier to go backwards than forwards.

我们之前讨论过的不对称之一是，在某些环境中后退比前进更容易。

What I mean by that is, it’s very easy to define this complex data-generating process, and this is the latent variable you keep hidden from the model.

我的意思是，定义这个复杂的数据生成过程非常容易，这是您对模型隐藏的潜在变量。

You can generate arbitrarily complex environments, and the model has to do a lot of irreducible token spend and irreducible work to figure out what that data-generating process was.

您可以生成任意复杂的环境，并且模型必须执行大量不可简化的代币支出和不可简化的工作才能弄清楚数据生成过程是什么。

There are asymmetries in terms of injecting information from the real world.

从现实世界注入信息存在不对称性。

Anthropic finds a bug through tens of thousands of humans and LLMs combined, and turns that into a very, very neat environment which a single LLM could theoretically find within a few million tokens.

Anthropic 通过数万名人类和法学硕士的结合发现了一个错误，并将其转变为一个非常非常整洁的环境，理论上单个法学硕士可以在几百万代币内找到该环境。

There are all these asymmetries which we’re cherry-picking, and we’re counting on this kind of task-horizon generalization.

我们正在挑选所有这些不对称性，并且我们指望这种任务范围的概括。

But I think it’s just going to hit diminishing returns at some point, diminishing returns in how hard it is to create those environments in the first place, coming up with them, because you can’t necessarily just have these processes where it’s easier to go backwards than forwards.

但我认为，在某些时候，它只会遇到收益递减的问题，收益递减在于首先创建这些环境、提出它们的难度，因为你不一定能在这些过程中后退比前进更容易。

You actually have to sit down and construct something that looks like a long enough time horizon with humans, and it’s going to be a really complex task to create.

实际上，你必须坐下来，与人类一起构建一个看起来足够长的时间范围的东西，这将是一项非常复杂的任务。

Then there are also going to be the compute and time bottlenecks for the agent to actually do those tasks.

然后，代理实际执行这些任务也会遇到计算和时间瓶颈。

I think you’re just going to start seeing this curve flatten out.

我想你将会开始看到这条曲线变平。

I saw something about how someone fine-tuned the Talkie model , which is only trained on data up to 1930, on modern coding agent data.

我看到有人如何对 Talkie 模型进行微调，该模型仅在 1930 年之前的数据和现代编码代理数据上进行训练。

It did better than Claude 3 Opus on SWE-bench .

它在 SWE-bench 上的表现比 Claude 3 Opus 更好。

So this model that has no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and behave better as a coding agent than this much larger pre-trained model, which is pretty crazy.

因此，这个对代码一无所知的模型可以在适量的数据上进行微调，并且作为编码代理的表现比这个更大的预训练模型更好，这是非常疯狂的。

It kind of shows you that once you have an example of the right expert behavior, it’s actually surprisingly easy to copy that into a relatively weak model.

它向你表明，一旦你有了正确的专家行为的例子，将其复制到相对较弱的模型中实际上是非常容易的。

But a counterexample to that is a paper recently where they trained a model up to fifth-grade maths, and also primary-school English and stuff, so it was a decent language model.

但一个反例是最近的一篇论文，他们训练了一个模型，直到五年级数学，还有小学英语等等，所以这是一个不错的语言模型。

They tried to RL it to do late high school and college maths.

他们试图通过强化学习来完成高中和大学的数学题。

The gap was just too large.

差距太大了。

They couldn’t get it to climb at all.

他们根本无法让它爬上去。

But if you did successive rungs of year 7 maths and then year 8 maths…

但如果你连续完成了 7 年级数学，然后是 8 年级数学……

and so on, you could obviously climb to year 12.

依此类推，你显然可以升到12年级。

Again, it’s just what is the distance between the rungs on those ladders, and how hard is it to create?

再说一遍，问题在于这些梯子上的横档之间的距离是多少，以及创建起来有多难？

This just comes back to the RL signal problem.

这又回到了 RL 信号问题。

RL is not very good at exploring right now.

RL 目前不太擅长探索。

If the model can’t get it in 128 rollouts, it’s very unlikely to get signal to progress.

如果模型在 128 次部署中无法获得成功，那么就不太可能获得进展信号。

This is why in RL we need curricula , whereas in pre-training we don’t, because that’s not a problem for pre-training at all.

这就是为什么在强化学习中我们需要课程，而在预训练中则不需要，因为这对于预训练来说根本不是问题。

Again, pre-training data is different to post-training data.

同样，训练前数据与训练后数据不同。

I imagine as we continue on, humans will be involved less and less, but that doesn’t change the fact that you’re bottlenecked on how much signal you can extract from the real world.

我想，随着我们继续前进，人类的参与将会越来越少，但这并不能改变这样一个事实：你在从现实世界中提取多少信号方面遇到了瓶颈。

There’s a lot of signal in the world, and that’s true.

世界上有很多信号，这是事实。

There’s people doing spreadsheet tasks, there’s people doing legal tasks and all this sort of stuff.

有人做电子表格任务，有人做法律任务等等。

But at the capability frontier of where the models are at now, how many bits in the world are actually really relevant to improving the model’s capabilities?

但在模型目前所处的能力前沿，世界上有多少位实际上与提高模型的能力真正相关？

How many new maths problems are being solved that are just beyond the reach or grasp of the current models?

有多少新的数学问题正在被解决，而这些问题超出了当前模型的范围或掌握？

How many new coding problems are being created or solved that are beyond the reach of the current models?

有多少新的编码问题正在被创建或解决，而这些问题超出了当前模型的范围？

I think that’s why the diminishing returns kick in, because even the world as a whole is not giving you the bits that are useful for tipping you into the next basin of capability.

我认为这就是为什么收益递减的原因，因为即使整个世界也没有给你有用的信息，让你进入下一个能力盆地。

I totally agree with this.

我完全同意这一点。

It’s really a question of where the signal is coming from.

这实际上是一个信号从哪里来的问题。

In pre-training, the signal is already in Common Crawl .

在预训练中，信号已经处于 Common Crawl 中。

For the tasks that you care about in pre-training, the problem is not getting signal at all.

对于您在预训练中关心的任务，问题在于根本没有获得信号。

It’s filtering out all the noise that exists.

它过滤掉所有存在的噪音。

That’s quite an automatable process.

这是一个相当自动化的过程。

But as the models get better, as we enter mid-training and post-training, the signal just doesn’t exist anywhere in the original data we have.

但随着模型变得更好，当我们进入训练中期和训练后时，信号在我们拥有的原始数据中不存在。

No amount of filtering will get this.

再多的过滤也无法得到这个。

There’s no hidden proof of a Millennium Prize problem sitting in Common Crawl that we can just filter until we see it.

Common Crawl 中没有隐藏的千禧年奖问题的证据，我们可以过滤直到看到它。

At that point, you have to get bits some other way, either from humans directly, asking them to write out their reasoning, or by creating environments where humans decide what environment should be created and what the objectives of these environments are, or some kind of training on the human data that exists in deployment.

到那时，你必须以其他方式获取信息，要么直接从人类那里，要求他们写出他们的推理，要么通过创建环境，让人类决定应该创建什么环境以及这些环境的目标是什么，或者对部署中存在的人类数据进行某种培训。

You have to get the bits from somewhere.

你必须从某个地方获取这些数据。

There’s a question of how much of the progress in pre-training is being driven by data.

问题是预训练的进展有多少是由数据驱动的。

I did this investigation with Jerry Han , who’s a student at Princeton, where we trained all the recipes from 2019 till now pairwise with all the data sets from 2019 to now.

我和普林斯顿大学的学生 Jerry Han 一起进行了这项调查，我们将 2019 年至今的所有食谱与 2019 年至今的所有数据集配对训练。

You’re training GPT-2 on the newest data set, like Ultra-FineWeb .

您正在最新的数据集（例如 Ultra-FineWeb）上训练 GPT-2。

You train Delphi , which is the newest open source training recipe, on the Pile or some old data set.

您可以在 Pile 或一些旧数据集上训练 Delphi，这是最新的开源训练方法。

You do the whole grid.

你做整个网格。

You see, getting to some level of capabilities, how much less compute does it take, across this grid?

您会看到，达到某种级别的功能，在整个网格中需要的计算量会减少多少？

You see that the data seems to explain something like a 12.0x compute efficiency gain, but the architecture improvements explain something like a 3.7x compute efficiency gain, at a very small scale.

您会看到，数据似乎解释了诸如 12.0 倍计算效率增益之类的事情，但架构改进解释了诸如 3.7 倍计算效率增益之类的事情，而且规模非常小。

To the extent that that is true at large scale — that most of the pre-training compute efficiency gains are coming from better data — how much can that continue?

如果大规模情况下确实如此——大部分预训练计算效率的提高都来自更好的数据——这种情况能持续多久？

Can you keep filtering data more and more and building more and more synthetic data ?

您能否继续过滤越来越多的数据并构建越来越多的综合数据？

Do you have a sense of how much this kind of pre-training progress can continue?

你是否感觉到这样的预训练进度还能持续到什么程度？

My prior is that, again, the low-hanging fruit is somewhat exhausted.

我的前提是，唾手可得的成果已经有些枯竭了。

We got the internet as this big block, and it’s not like the internet is necessarily growing at the same rate.

我们把互联网当作一个大块，但互联网并不一定以同样的速度增长。

All the useful stuff on the internet isn’t growing at the same rate.

互联网上所有有用的东西并不是以同样的速度增长的。

We’ve probably got a bunch of 0.1% loss drops to go, but definitely not as many as have currently occurred.

我们可能还会出现大量 0.1% 的损失，但绝对不会像目前发生的那么多。

But it’s also really interesting that you find this cumulative 33x improvement across both.

但也很有趣的是，您发现两者的累计改进达到了 33 倍。

I think it was Epoch or someone who estimated 3x a year since 2019, which would imply something like 3 7 , over 2,000X improvement.

我认为是 Epoch 或某人估计自 2019 年以来每年增长 3 倍，这意味着类似 3 7 ，超过 2,000 倍的改进。

So where’s that missing 100x or whatever coming from?

那么缺失的 100 倍或其他什么来自哪里呢？

That probably gives you a good signal of how much of this is post-training.

这可能会给你一个很好的信号，让你知道其中有多少是训练后的。

I think the explanation has to be that a lot of the compute efficiency gains are scale dependent, and we’re starting at extremely small scale.

我认为解释必须是，许多计算效率的提高都依赖于规模，而我们是从非常小的规模开始的。

That raises a question of whether the data compute efficiency gains or the algorithmic compute efficiency gains have more scale dependence.

这就提出了一个问题：数据计算效率增益或算法计算效率增益是否具有更大的规模依赖性。

I don’t know if you have a prior on that.

我不知道你是否有这方面的先验。

We just didn’t have enough compute to investigate that question.

我们只是没有足够的计算能力来研究这个问题。

Just naively, theoretically, the scale dependence of the architecture is fairly well known, and you can fit a straight line to it.

天真地讲，从理论上讲，架构的尺度依赖性是众所周知的，您可以为其拟合一条直线。

Whereas I would have no idea how to do that for combining pre-training plus post-training data and mid-training data.

然而我不知道如何将训练前数据、训练后数据和训练中期数据结合起来。

Funnily enough, I feel like data is actually more important with scale.

有趣的是，我觉得数据实际上随着规模变得更重要。

I feel like architectures are kind of a one-time thing.

我觉得架构是一次性的东西。

Saying just an X% efficiency gain is kind of misleading, because what an architecture does is let you reach a qualitatively new regime which you couldn’t reach with the old architecture.

仅仅说 X% 的效率增益有点误导，因为架构的作用是让您达到一个新的质量状态，这是旧架构无法达到的。

Within that regime, obviously the data is the primary thing determining it.

在这种制度下，显然数据是决定它的首要因素。

But if we didn’t have even GQA , if we were doing full attention all day, it would be ridiculously expensive to do a million context.

但如果我们连 GQA 都没有，如果我们整天全神贯注，那么做一百万个上下文将会非常昂贵。

Because of that, we could never use the data which is actually at a million context, so we couldn’t get these capabilities.

因此，我们永远无法使用实际在一百万个上下文中的数据，因此我们无法获得这些功能。

Even though if you just do a naive “how much does this do at 2K context”, where the architecture isn’t unlocking anything, then the data will look much more important than in some sense it is.

即使你只是天真地想“这在 2K 环境下能起到多大作用”，而架构没有解锁任何东西，那么数据看起来会比在某种意义上重要得多。

It’s unclear to me that these things are really just multiplicative gains in this way.

我不清楚这些东西是否真的只是以这种方式获得乘数收益。

On scale dependence, I think a lot of the mid-training and post-training data we have now actually gets better with scale, because a lot of it — the very long context horizon environment stuff — really requires big models to be able to make use of it.

关于规模依赖性，我认为我们现在拥有的许多训练中期和训练后数据实际上随着规模的扩大而变得更好，因为其中很多——非常长的上下文范围环境的东西——确实需要大模型才能利用它。

If you try and train your 100 million parameter model on SWE-bench traces, it’s not going to get anywhere.

如果您尝试在 SWE 基准轨迹上训练您的 1 亿参数模型，它不会有任何进展。

It’s not going to show you the same kind of improvement that you would get if you train an actual sensible size model on it.

它不会向您展示在其上训练实际合理尺寸模型时所获得的相同改进。

It’s hard as well now because so many of the architecture changes — you look at Kimi, for instance, or DeepSeek — they’re doing these architectural modifications not just with dropping the pre-training loss in mind, but with how the models are going to be used in the real world.

现在也很困难，因为很多架构发生了变化——例如 Kimi 或 DeepSeek——他们在进行这些架构修改时不仅考虑到了训练前损失，而且还考虑到了模型将如何在现实世界中使用。

The inference efficiency, having some form of compressed attention in the DeepSeek models, is not necessarily geared around a fundamental trade-off improvement.

DeepSeek 模型中具有某种形式的压缩注意力的推理效率不一定围绕根本性的权衡改进。

It’s just, “Okay, we’re considering how the models are going to be used.”

只是，“好吧，我们正在考虑如何使用这些模型。”

One question I’m curious about, to understand the future, is how parameter scaling will go as we’re getting into more of an RL-heavy regime.

为了了解未来，我很好奇的一个问题是，当我们进入更多强化强化的状态时，参数缩放将如何进行。

You can look at open source architectures and see how fast parameters have been scaling.

您可以查看开源架构并了解参数扩展的速度。

Maybe it’s roughly 2x every year for frontier open source models.

对于前沿开源模型来说，每年大约是 2 倍。

To the extent that even frontier closed source models have 100B or 200B active parameters , do you think that keeps 2x-ing year over year?

即使是前沿的闭源模型也有 100B 或 200B 的有效参数，您认为这会逐年保持 2 倍增长吗？

Or, now that we’re in an RL regime where you also want to conserve compute on rollouts…

或者，现在我们处于 RL 机制中，您还希望在部署时节省计算量……

Also, maybe there is a threshold effect where you have enough capacity and at that point increasing parameters arbitrarily doesn’t matter as much.

另外，也许存在阈值效应，即您有足够的容量，此时任意增加参数就不再那么重要了。

Do you guys have a sense of, in 2030, how many active parameters a frontier model will have?

你们是否知道，到 2030 年，前沿模型将有多少个活跃参数？

I think for the next few years, because we are so focused on doing longer and longer horizon rollouts for RL, where inference efficiency matters a lot, it feels like the models aren’t necessarily saturated on their ability to do that.

我认为在接下来的几年里，因为我们如此专注于为 RL 进行越来越长的推广，其中推理效率非常重要，所以感觉模型的能力不一定达到饱和。

The bottleneck is still the environments.

瓶颈仍然是环境。

So we might see a little bit of a plateau.

所以我们可能会看到一些平台期。

I have a feeling that Mythos and the GPT models are much smaller than the 10 trillion parameter range that people are talking about.

我有一种感觉，Mythos 和 GPT 模型比人们谈论的 10 万亿参数范围要小得多。

Even just naively comparing them to open source models, you can probably back out that conclusion.

即使只是天真地将它们与开源模型进行比较，您也可能会推翻这个结论。

Probably for the next few years, I wouldn’t imagine a huge growth in the number of parameters.

可能在接下来的几年里，我不会想象参数数量会有巨大的增长。

But again, there’s so many different things to trade off here.

但同样，这里有很多不同的东西需要权衡。

You decide the size of your model based on how much pre-training data you have, and then the difficulty of the RL environments that you’ve got to train on.

您可以根据拥有的预训练数据量以及需要训练的 RL 环境的难度来决定模型的大小。

You ideally want to get to the optimal point where you can get a decent pass@1 or something on the hardest environments you have.

理想情况下，您希望达到最佳点，在最困难的环境中获得不错的 pass@1 或其他东西。

It wouldn’t make sense to make a bigger model pass there, because then you’re just paying much more inference than you need to.

让更大的模型通过那里是没有意义的，因为那样你只是付出了比你需要的更多的推理。

So a lot of it depends on how quickly Mercor and the in-house teams can scale up the complexity of the RL environments they’re training on.

因此，这在很大程度上取决于 Mercor 和内部团队能够以多快的速度扩大他们正在训练的 RL 环境的复杂性。

I would expect the models to keep getting bigger just because people are scaling up compute and the GPUs are getting bigger.

我预计模型会不断变得更大，因为人们正在扩大计算规模，GPU 也变得越来越大。

But exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways.

但它们到底变大了多少，在某种程度上取决于缩放定律，但并不明显。

One thing is that I think data efficiency is going to be a bigger driver than compute efficiency of the exact architectures people use, now that we’re getting to the regime where we’re running low on high quality pre-training data.

有一件事是，我认为数据效率将比人们使用的确切架构的计算效率成为更大的驱动因素，现在我们已经进入了高质量预训练数据不足的状态。

That might affect how sparse you want to make the model.

这可能会影响您想要制作模型的稀疏程度。

I also think we don’t understand sparsity that well.

我还认为我们不太了解稀疏性。

Parameters are a different resource than active parameters.

参数是与活动参数不同的资源。

Sparsity has definitely increased a bit, but it’s not clear that it’s going to keep increasing without bound.

稀疏性确实增加了一点，但尚不清楚它是否会无限制地继续增加。

There might be some kind of sweet spot.

可能有某种甜蜜点。

There’s an argument that sparsity should make data efficiency worse, because you might have to learn the same thing on multiple experts, though that’s debatable.

有一种观点认为，稀疏性会使数据效率变得更糟，因为你可能必须向多个专家学习相同的东西，尽管这是有争议的。

I don’t think we have a good enough theory of scaling laws that we really understand why sparsity is helping, how much it’ll help, and if that’ll plateau at some point at a certain level of sparsity.

我认为我们没有足够好的缩放定律理论，无法真正理解稀疏性为何有帮助、有多大帮助，以及在一定程度的稀疏性下是否会在某个时刻达到稳定状态。

Sorry, can you spell out exactly what the implication of data efficiency would be on parameters?

抱歉，您能准确说明数据效率对参数的影响吗？

It sounds like you’d say there should be less sparsity, but what are the other implications on parameter scaling?

听起来你会说应该减少稀疏性，但是对参数缩放的其他影响是什么？

Just that with the scaling law, you’re not trying to optimize compute efficiency.

只是根据缩放定律，您并不是在尝试优化计算效率。

You have all your choices you can make on the architecture.

您可以对架构做出所有选择。

Each of these gives you a different scaling law.

其中每一个都给出了不同的缩放定律。

Traditionally, you would look at some kind of envelope based on compute.

传统上，您会查看某种基于计算的信封。

You would look at performance versus compute and take the envelope of the best models.

您将比较性能与计算，并采用最佳模型的范围。

But if we’re making that decision based on data — we’re assuming we can spend a lot of compute, so data is on our x-axis instead of compute — then we just get a different set of optima, or a different set of models that are on that frontier.

但是，如果我们根据数据做出决策——我们假设我们可以花费大量计算，因此数据位于我们的 x 轴而不是计算上——那么我们只会得到一组不同的最优值，或者一组位于该前沿的不同模型。

I also don’t think that we’ve necessarily doubled the size of the models every year for the last few years.

我也不认为在过去几年里我们每年都必须将模型的尺寸增加一倍。

People have been training 1 trillion parameter models for at least a few years.

人们至少几年来一直在训练 1 万亿个参数模型。

There was even an open source one called Falcon .

甚至还有一个名为 Falcon 的开源软件。

Liam from Periodic Labs , I think, posted yesterday on Twitter about how an early experiment was training a 1 trillion parameter model that was very, very sparse.

我想，来自Periodic Labs 的 Liam 昨天在 Twitter 上发布了有关早期实验如何训练一个非常非常稀疏的 1 万亿参数模型的信息。

It was very, very good at knowledge but terrible at reasoning because it was so sparse.

它在知识方面非常非常擅长，但在推理方面却很糟糕，因为它太稀疏了。

It feels like we’ve been playing in this 100 billion up to 2 trillion parameter range for at least a little bit.

感觉我们已经在这个 1000 亿到 2 万亿的参数范围中玩了至少一点时间了。

It certainly hasn’t been this nice linear increase.

这当然不是这么好的线性增长。

I feel like there’s two things.

我感觉有两件事。

As Charlie was saying, inference efficiency is super important for RL rollouts.

正如 Charlie 所说，推理效率对于 RL 的推出非常重要。

This will really push down active parameters quite a lot.

这确实会大大降低活动参数。

I think the total parameters really depends a lot on the hardware as well.

我认为总参数实际上也很大程度上取决于硬件。

You really need very high memory bandwidth and VRAM size to actually be able to serve multi-trillion parameter models.

您确实需要非常高的内存带宽和 VRAM 大小才能真正能够服务数万亿参数模型。

Right now, people are still using a lot of H100s and stuff.

现在，人们仍在使用大量 H100 之类的东西。

As everyone moves to GBs and then Vera Rubins , we’ll get more of the ability to scale and actually serve and do large RL inference at larger scales.

随着每个人都转向 GB，然后转向 Vera Rubins，我们将获得更多的扩展能力，并在更大规模上实际服务和进行大型 RL 推理。

The data question I think is interesting, because naively, larger models are much more sample efficient in the actual data points.

我认为数据问题很有趣，因为天真地说，较大的模型在实际数据点中的样本效率更高。

Even if you’re not saturating the model, it’s still better to go bigger, because larger models generalize better and get to a better loss for the same amount of data.

即使您没有使模型饱和，最好还是增大模型，因为较大的模型具有更好的泛化能力，并且对于相同数量的数据可以获得更好的损失。

Right now I think we have a lot of data, and that’s not the constraint.

现在我认为我们有很多数据，但这不是限制。

Compute is.

计算是.

So we’re having smaller models which are very inference efficient.

所以我们有更小的模型，推理效率非常高。

But if compute is no longer the bottleneck, it might come back to larger models which are undersaturated, but have this generalization ability because they’re much larger.

但如果计算不再是瓶颈，它可能会回到未饱和的更大模型，但具有这种泛化能力，因为它们更大。

If you just look at the basic Chinchilla scaling law and you just maximize out parameters, it actually decreases the amount of data you need to get to the same loss very little.

如果你只看基本的 Chinchilla 缩放定律，并且最大化参数，它实际上会减少达到相同损失所需的数据量。

If you go to infinity on parameters, the amount of data you need I think goes down less than 10X, just because of the nature of the power law.

如果你的参数达到无穷大，我认为你需要的数据量会减少不到 10 倍，这只是因为幂律的性质。

But we’re now on the way-too-much-data side of the Chinchilla laws.

但我们现在正处于龙猫法律的数据过多的一方。

Right now we over-train models according to Chinchilla.

根据 Chinchilla 的说法，现在我们对模型进行了过度训练。

So we could easily get back to a point where, as we’re running out of data, we move back to the Chinchilla optimal point, or even a bit on the under-training model side.

因此，我们可以很容易地回到一个点，当我们用完数据时，我们会回到 Chinchilla 最佳点，甚至回到训练不足的模型方面。

But surely, even with these new chips that come online, we’re just going to be so compute bottlenecked for the next few years that that won’t necessarily be the case.

但可以肯定的是，即使有了这些新芯片上线，我们在未来几年内仍将面临计算瓶颈，因此情况不一定如此。

This depends on the ratio you have of training and inference compute, really.

这实际上取决于您的训练和推理计算的比例。

If you’re super bottlenecked on data, not on compute, you should go bigger.

如果你在数据上遇到超级瓶颈，而不是在计算上，那么你应该做得更大。

If you’re super bottlenecked on compute, you should always go smaller.

如果你在计算上遇到超级瓶颈，你应该总是缩小规模。

You can also use computer-generated synthetic data, so it’s one of these very hard things to predict.

您还可以使用计算机生成的合成数据，因此这是非常难以预测的事情之一。

I think part of the reason it took people so long to figure out the scaling laws in the first place was that if you don’t get all these things right, then you don’t get such a clean relationship.

我认为人们花了这么长时间才弄清楚比例定律的部分原因是，如果你没有把所有这些事情都做好，那么你就不会得到如此干净的关系。

The beautiful straight lines on graphs hide a lot of complexity in how you have to make sure to scale every hyperparameter the right way, or parameterize your optimizer in a way that scales and where you don’t have to change your hyperparameters as you change the model size.

图表上美丽的直线隐藏了很多复杂性，您必须确保以正确的方式缩放每个超参数，或者以可缩放的方式参数化优化器，并且在更改模型大小时无需更改超参数。

Bugs have their own clean scaling laws as well.

错误也有自己的清晰的缩放法则。

Like with Kaplan forgetting the cosine annealing thing, or even just not considering embedding parameters , I think.

我认为，就像卡普兰忘记了余弦退火的事情，或者甚至只是不考虑嵌入参数一样。

That messed up the estimate at smaller models because embedding parameters are a decent size of the model.

这扰乱了较小模型的估计，因为嵌入参数是模型的相当大的尺寸。

A bit on RL.

有点关于RL。

A year ago, a lot of people were making this argument that RL will not be super successful at scaling for models.

一年前，很多人都认为强化学习在模型扩展方面不会非常成功。

John, you wrote a research paper where you were pointing out that models learn one bit per episode when you RL.

约翰，你写了一篇研究论文，指出模型在强化学习时每集学习一位。

They learn, “Did I get the answer right or did I get it wrong?” Then I wrote some blog posts earlier this year where I was like, “It’s even worse than that,” because when the pass rate is low and the model is very unlikely to get the answer right, it learns almost nothing at all from an RL episode.

他们了解到，“我的答案是对还是错？”然后我在今年早些时候写了一些博客文章，我当时想，“这甚至比这更糟糕”，因为当通过率很低并且模型不太可能得到正确的答案时，它几乎从 RL 事件中学习不到任何东西。

But I look at the models today, and they seem pretty smart.

但我今天看看这些模型，它们看起来非常聪明。

It seems to be the result of scaling up RL.

这似乎是强化强化学习规模扩大的结果。

Beren, you had a post a few weeks ago where you were trying to explain what’s going on.

贝伦，几周前你曾发过一篇文章，试图解释正在发生的事情。

Why has RL been more successful than one would have naively thought?

为什么强化学习比人们天真的想象的更成功？

I think the success of RL comes down to a bunch of different things.

我认为强化学习的成功取决于很多不同的因素。

First, what is slightly underestimated is the mid-training.

首先，稍微被低估的是中期训练。

An awful lot of what we see as successes of RL actually comes from very, very good mid-training data, which is where we’re essentially doing pre-training but on synthetic reasoning data and the kind of environments that get the model warm-started for RL.

我们所看到的强化学习的大部分成功实际上都来自非常非常好的训练中期数据，这就是我们本质上进行预训练的地方，但是基于合成推理数据以及让模型为强化学习热启动的环境。

This takes the model almost 80% of the way to the final RL checkpoint often.

这通常会使模型完成几乎 80% 的路程，直至最终的强化学习检查点。

Then what RL does on top of that is essentially tweaking the policy.

那么强化学习在此基础上所做的本质上就是调整策略。

This is one of the reasons why it doesn’t need as many bits as you would naively think.

这就是为什么它不需要像你天真地想象的那么多位的原因之一。

It doesn’t have to learn all of these behaviors from scratch.

它不必从头开始学习所有这些行为。

It needs just a few bits from these episodes, which you do get.

它只需要这些剧集中的一些片段，你确实得到了。

The other thing that I point out in my blog is that these bits are extremely high signal compared to regular pre-training, which is why you need RL at all versus just SFT-ing on successful reasoning traces.

我在博客中指出的另一件事是，与常规预训练相比，这些位的信号非常高，这就是为什么您需要 RL，而不是在成功的推理轨迹上仅进行 SFT。

There’s two things.

有两件事。

Yes, one, it’s exactly the bits about how to get the answer right.

是的，第一，这正是关于如何获得正确答案的部分。

But this is not exactly how you think of it, because in SFT, you have a trace.

但这并不完全是你想的那样，因为在SFT中，你有痕迹。

You have, say, a bunch of math reasoning and then the answer at the end.

比如说，你有一堆数学推理，最后就有答案。

The bit is still there.

那个位还在那里。

You still SFT on the answer token.

您仍然对答案标记进行 SFT。

What’s important is that the objective ignores all the other bits.

重要的是目标忽略了所有其他部分。

In SFT, you have to try and match the exact reasoning tokens that the model produces.

在 SFT 中，您必须尝试匹配模型生成的确切推理标记。

You’re essentially getting too many bits about the exact way this other model you’re training on reasons.

您实际上对您正在训练的另一个模型的确切方式了解了太多的原因。

For RL, you only get the one bit.

对于强化学习，你只能得到一位。

That means that signal is not drowned out in the noise of all the other bits the model has.

这意味着信号不会被模型所有其他位的噪声淹没。

It’s really a super dramatic increase in the signal-to-noise ratio during training, which is why RL is so dramatically efficient in terms of steps.

在训练过程中，信噪比确实得到了极大的提高，这就是强化学习在步骤方面如此高效的原因。

There’s been so much debate about what RL does to the model versus mid-training or SFT or whatever.

关于 RL 对模型的作用与训练中期或 SFT 等的作用有很多争论。

Everyone talks about how pass@1 will go up, but pass@256 will go down.

大家都在说pass@1怎么会上去，而pass@256却会下来。

Very rare correct reasoning traces will be down-weighted and outweighed by a gradient signal from easier reasoning traces.

非常罕见的正确推理轨迹将被降低权重，并被来自更容易推理轨迹的梯度信号所取代。

I think the simple way to view RL now is that if you have a large enough amount of compute to sample a large enough group size — such that your probability of getting a bunch of correct answers is past some not insignificant probability — then it will be up-weighted.

我认为现在看待强化学习的简单方法是，如果你有足够多的计算量来对足够大的组进行采样——这样你得到一堆正确答案的概率就超过了一些不小的概率——那么它的权重就会增加。

To Beren’s point, mid-training and more pre-training — the pass@1, the starting point for RL — scales in a log number of pre-training tokens.

就 Beren 的观点而言，训练中期和更多预训练（pass@1，强化学习的起点）以对数数量的预训练标记进行缩放。

Can I ask some very basic questions?

我可以问一些非常基本的问题吗？

That answer makes sense, and maybe there’s empirical research which shows that this is what’s happening.

这个答案是有道理的，也许有实证研究表明这就是正在发生的事情。

But then I just look at the models themselves…

但后来我只看模型本身......

I don’t know what’s happened.

我不知道发生了什么事。

Maybe you can give me a sense of what is the basis of the AI progress over the last year.

也许你可以让我了解一下去年人工智能进展的基础是什么。

Maybe it’s just up-weighting the policies which were going to do the correct thinking anyways.

也许这只是增加了无论如何都会进行正确思考的政策的权重。

But it just seems like qualitatively, the models have gotten so much more capable.

但从质量上看，这些模型的能力似乎变得更强了。

Maybe there’s no inherent contradiction there.

也许这并没有什么内在的矛盾。

But how do we square the relatively small impact this take would imply that RL would have with the actual qualitative capabilities the models seem to be gaining?

但是，我们如何平衡强化学习所带来的相对较小的影响与模型似乎获得的实际定性能力呢？

One thing I want to point out here is that it doesn’t necessarily imply that RL has a small effect.

我想在这里指出的一件事是，这并不一定意味着 RL 的影响很小。

Even if you have a few bits and you only change the parameters a small amount, the actual impact on function space — the input-to-output mapping the model learns — can still be super dramatic.

即使您只有几个位并且只更改了少量参数，对函数空间（模型学习的输入到输出映射）的实际影响仍然可能非常巨大。

Even one bit can change your function space a lot.

即使是一点点也可以极大地改变你的功能空间。

It can rule out half the hypothesis space, which is huge.

它可以排除一半的假设空间，这个空间是巨大的。

I don’t think it’s necessarily the case that small amounts of bits, small amounts of RL, once you’re starting from a really good point, means that you don’t have dramatic impacts in behavior.

我认为，一旦你从一个非常好的起点开始，少量的比特、少量的强化学习并不一定意味着你不会对行为产生巨大的影响。

At least…

至少……

not necessarily.

不一定。

I think it comes down to two things.

我认为这可以归结为两件事。

The first thing is that everyone was hoping that RL would generalize this reasoning across all these different domains.

首先，每个人都希望强化学习能够将这种推理推广到所有这些不同的领域。

I don’t think we necessarily got this horizontal generalization.

我认为我们不一定得到这种横向概括。

Just training on math doesn’t necessarily make you the greatest coder.

仅进行数学训练并不一定能让你成为最伟大的程序员。

You do have to do RL on code environments.

你确实必须在代码环境中进行强化学习。

I think what we did get, though, is horizon generalization.

不过，我认为我们确实得到的是地平线概括。

The models just learned how to use more tokens for longer and still make progress on some sort of task.

这些模型刚刚学会了如何在更长时间内使用更多代币，并且仍然在某种任务上取得进展。

You can train on environments where they get longer and longer and then put them into a completely new environment.

您可以在它们变得越来越长的环境中进行训练，然后将它们放入一个全新的环境中。

Yes, they may not have generalized the reasoning patterns which allow them to do well in that environment, but they’ve at least generalized the ability to continue on that task for longer, which is correlated with success.

是的，他们可能没有概括出使他们在那种环境中表现出色的推理模式，但他们至少概括了更长时间地继续执行该任务的能力，这与成功相关。

There was a paper called EdgeBench which showed that the rate at which models can work for longer is doubling every three months.

有一篇名为 EdgeBench 的论文表明，模型可以工作更长时间的速度每三个月翻一番。

That’s clear evidence of generalization.

这是普遍化的明显证据。

The final way to think about it is, in pre-training, there’s this idea of quanta.

最后一种思考方式是，在预训练中，存在量子的概念。

You have this very smooth pre-training loss curve.

你有这个非常平滑的训练前损失曲线。

When you look at what’s happening in the model, the model is learning all these very discrete tasks, and there’s all these emergent points where there’s a phase transition.

当您查看模型中发生的情况时，模型正在学习所有这些非常离散的任务，并且存在所有这些存在相变的紧急点。

It didn’t have induction heads , now it has induction heads.

以前没有感应头，现在有了感应头。

There’s tens of thousands, millions, probably hundreds of millions of these things.

这些东西有数万、数百万、甚至数亿。

You average them all together and you get this very smooth loss curve.

将它们全部平均，就得到了非常平滑的损失曲线。

To an extent, a similar thing is happening for RL.

在某种程度上，强化学习也发生了类似的事情。

There is this very slow outer loop, as Beren mentioned.

正如贝伦提到的，有一个非常慢的外循环。

We will train a model and then RL it, and then in the next model iteration of training, we will dump a bunch of these synthetic reasoning traces into the mid-training data.

我们将训练一个模型，然后对其进行强化学习，然后在训练的下一个模型迭代中，我们将把一堆这些合成推理轨迹转储到训练中期数据中。

We’re kind of hitting all these quanta for all these different tasks, and on an individual task level, it may look like a phase transition.

我们正在为所有这些不同的任务达到所有这些量子，并且在单个任务级别上，它可能看起来像一个阶段过渡。

You’re suddenly going from a 0.5% pass rate to a 90% pass rate on a particular finance task or Excel task or whatever.

在某项特定的财务任务或 Excel 任务或其他任务上，您的通过率突然从 0.5% 上升到 90%。

But you average all these things together, plus the horizon generalization, and you kind of go, “Wow, we’ve got qualitatively better models.”

但是你把所有这些东西加在一起平均，再加上视野概括，你就会说，“哇，我们有了质量更好的模型。”

I think a lot of this as well is just…

我认为其中很多也只是……

RL does generalize a bit.

强化学习确实具有一定的概括性。

You get some transfer between math and code, or puzzles and math and this kind of stuff.

你可以在数学和代码、谜题和数学之类的东西之间进行一些转换。

Also, the sheer amount of environments people are targeting is just vastly greater.

此外，人们所针对的环境数量要大得多。

Before, when you tried to do some task which you do in your daily life, two years ago, the labs wouldn’t really care about this.

两年前，当你尝试做一些日常生活中要做的任务时，实验室不会真正关心这个。

They wouldn’t train the model for it.

他们不会为此训练模型。

Now it’s just so much broader.

现在范围更广了。

They have a lot of environments targeting this specific thing.

他们有很多针对这个特定事物的环境。

Earlier in the conversation we were talking about RL in the context of causing this entropy collapse , or just concentrating probability on solutions the base model had already done, and causing relatively sparse updates in the policy.

在谈话的早些时候，我们在导致这种熵崩溃的背景下讨论了强化学习，或者只是将概率集中在基本模型已经完成的解决方案上，并导致策略中相对稀疏的更新。

But I think there’s also another story about RL, which is going back to the Atari games and then AlphaGo coming up with move 37 , the super creative move.

但我认为还有另一个关于 RL 的故事，这可以追溯到 Atari 游戏，然后 AlphaGo 提出了第 37 步，这是一个超级有创意的棋步。

Because it was never initialized on human data, it can think in ways that humans are not even thinking and come up with extremely creative solutions.

因为它从未根据人类数据进行初始化，所以它可以以人类甚至没有思考过的方式进行思考，并提出极具创造性的解决方案。

Do you have a sense of when we should expect, or if we should expect, RL on LLMs to result in things like move 37, extreme creativity even beyond human creativity, because there’s just de novo initialization of intelligence?

你是否知道我们什么时候应该期望，或者我们是否应该期望，法学硕士的强化学习会产生像第 37 步这样的结果，甚至超越人类创造力的极端创造力，因为这只是智力的从头初始化？

A couple of things here.

这里有几件事。

First off, I think that AlphaGo is using MCTS , which obviously does more exploration and stuff than regular policy gradients .

首先，我认为 AlphaGo 使用的是 MCTS，它显然比常规策略梯度做了更多的探索和工作。

But I also think that RL doesn’t necessarily reduce the creativity.

但我也认为强化学习并不一定会降低创造力。

This is obviously qualitative, but if we look at the OpenAI-Hugging Face incident , these models were coming up with multiple zero-days at a time to break out of the sandbox .

这显然是定性的，但如果我们看看 OpenAI-Hugging Face 事件，这些模型一次会想出多个零日漏洞来突破沙箱。

This is clearly some level of move 37 creativity already, which we just get from the general generalization properties of the LLMs.

这显然已经是第 37 步创造力的某种程度了，这是我们从法学硕士的一般泛化属性中得到的。

It’s definitely not the case that RL is totally destroying entropy, especially on long horizons.

强化学习绝对不会完全摧毁熵，尤其是从长远来看。

One thing that people call creativity is just solving hard search problems.

人们称之为创造力的一件事就是解决困难的搜索问题。

Move 37 is obviously an example of that, or writing some kind of poem that satisfies a ton of different constraints.

第 37 步显然就是一个例子，或者说写某种满足大量不同约束的诗歌。

That’s something AI is obviously going to be extremely good at, if trained for it.

如果经过训练，人工智能显然会非常擅长这一点。

Then there’s another way in which the diversity of the models’ outputs is a lot lower after RL, and they develop these tics.

然后还有另一种方式，其中模型输出的多样性在强化学习之后要低很多，并且它们会产生这些抽动。

Even though the models seem like they’re good at writing, when you do some kind of distributional analysis, you find that they’re reusing certain themes all the time and they’re using the same character names all the time.

尽管这些模型看起来很擅长写作，但当你进行某种分布分析时，你会发现他们一直在重复使用某些主题，并且一直使用相同的角色名称。

You’re not getting the same kind of diversity that you get from human authors.

你无法获得与人类作者相同的多样性。

You’re getting one really good style.

你会得到一种非常好的风格。

So I think that kind of diversity has definitely been cut down by RL a lot.

所以我认为这种多样性肯定被强化学习大大削弱了。

In fact, since we were talking about distillation earlier, one thing that’s happening is that so many people are distilling, mostly from Claude, that all the open-weight model s write the same way as Claude and have the same tics.

事实上，自从我们之前讨论过蒸馏以来，正在发生的一件事是，很多人都在蒸馏，主要是从 Claude 那里进行的，所有开放权重模型的编写方式都与 Claude 相同，并且具有相同的抽动。

This seems kind of concerning to me, that we’re having this monoculture emerge.

这对我来说似乎有点令人担忧，我们正在出现这种单一文化。

Again, I don’t think this is fundamental to RL as a method, though.

不过，我再次强调，这并不是强化学习方法的基础。

The same with distillation.

与蒸馏相同。

Even with distillation, you’re just training on the data.

即使使用蒸馏，您也只是在数据上进行训练。

Just because your data is not super broad, that doesn’t mean the training method itself is somehow wrong.

仅仅因为你的数据不是非常广泛，并不意味着训练方法本身就有问题。

It’s a problem with the data.

这是数据的问题。

I think a lot of the RL entropy collapse, for instance, is basically due to exploitation of fairly simple verifiers when you don’t have a huge diversity of environments.

例如，我认为很多 RL 熵崩溃基本上是由于在没有巨大多样性的环境时利用相当简单的验证器造成的。

The writing, for instance, is presumably graded by some judge.

例如，写作可能是由某个评委评分的。

The judge has some specific tics, and the model is learning to reward hack the judge, and that’s why it collapses.

法官有一些特定的问题，模型正在学习奖励破解法官，这就是它崩溃的原因。

But this is really a problem with the judge.

但这确实是法官的问题。

It’s not a problem with RL in general.

一般来说，这对于 RL 来说不是问题。

Okay, super rapid-fire predictions about the future.

好吧，关于未来的超级快速预测。

I want timelines on the following couple of questions.

我想要以下几个问题的时间表。

By when do we have models which…

到什么时候我们才会有这样的模型……

Here’s what it feels like to a user.

这就是用户的感受。

You basically hire it as a drop-in remote worker for all kinds of white-collar work?

您基本上雇用它作为各种白领工作的临时远程工作者？

Not just coding, but video editing, law, paralegal, et cetera.

不仅仅是编码，还有视频编辑、法律、律师助理等等。

It’s literally an actual remote worker, with full computer use, with literally a month of seamless learning and operation, executing on complex projects that require interacting with other people, et cetera.

它实际上是一个真正的远程工作者，可以充分使用计算机，可以进行一个月的无缝学习和操作，执行需要与其他人交互的复杂项目，等等。

Everything a human worker could do over a month.

人类工人在一个月内可以完成的所有事情。

If you mandate it to use a browser or whatever — rather than the firm setting up the information to be programmatically accessible — maybe a couple of years.

如果你要求它使用浏览器或其他什么东西——而不是公司将信息设置为可通过编程方式访问——也许需要几年的时间。

But if it’s not browser-based — it can send Slack messages, it can do all this stuff — I’d still probably say around a year.

但如果它不是基于浏览器的——它可以发送 Slack 消息，它可以做所有这些事情——我仍然可能会说大约一年。

I would say maybe three years for the full generality.

我想说，为了全面的普遍性，也许需要三年。

But to Charlie’s point, we will end up with a lot of people making their organizations easier for the AIs to use, and so you get 80-90% of the way there before that.

但就查理的观点而言，我们最终会得到很多人让他们的组织更易于人工智能使用，所以在此之前你已经完成了 80-90% 的工作。

I think there’s going to be a long tail of miscellaneous stuff which some human can do, which will take the models quite a while to do.

我认为会有一些人类可以做的各种事情的长尾，这将需要模型相当长的时间才能完成。

I think this really comes down to a question of how quickly we can solve this kind of online learning, and whether we can get 80-90% of the way there with compaction and writing files to yourself and stuff.

我认为这实际上归结为一个问题，即我们能够以多快的速度解决这种在线学习，以及我们是否可以通过压缩和向自己写入文件等方式获得 80-90% 的完成率。

That’s my big uncertainty.

这是我最大的不确定性。

I really don’t know.

我真的不知道。

An example of something that it wouldn’t be good at is if I have to yell at someone to get something at work, or really push someone to get something done.

它不擅长的一个例子是，如果我必须对某人大喊大叫才能在工作中完成某件事，或者真的推动某人完成某件事。

The model just isn’t going to do that.

该模型不会这样做。

It’s going to be too nice.

那就太好了。

I’d say there’s a wide variation in quality of human remote workers.

我想说，远程工作者的素质存在很大差异。

If you try to hire someone off of Upwork to do a software engineering project, there’s going to be a huge variation.

如果你尝试从 Upwork 雇佣某人来完成软件工程项目，将会出现巨大的差异。

It’s often quite hard to get them to do a good job or pay attention to all the feedback you’re giving.

通常很难让他们做好工作或关注你提供的所有反馈。

I would guess that in some cases, the pre-AI version of this was worse than what you can get now from existing AI.

我猜想，在某些情况下，人工智能之前的版本比你现在从现有人工智能中得到的还要糟糕。

I think it might end up being a little complicated, because to some extent we already have this for some not-so-high-quality work.

我认为这最终可能会有点复杂，因为在某种程度上我们已经为一些不太高质量的工作提供了这个。

But then obviously we’re not matching human level in certain higher-quality forms of work.

但显然我们在某些更高质量的工作形式上还没有达到人类水平。

But I basically agree with Charlie and Beren that maybe we’ll have some version of this in a year or so that’s okay.

但我基本上同意查理和贝伦的观点，也许我们会在一年左右的时间内推出这个版本，没关系。

We’ll have that form factor, and it’ll be able to do some things really well, some things not so well, and things will be improving from there.

我们将拥有这种外形尺寸，它将能够很好地完成某些事情，某些事情则不太好，并且事情将从那里得到改善。

We shift the goalposts based on the very long tail all the time.

我们一直根据长尾来改变球门柱。

I feel like you’ve used this example before of doing your taxes or something.

我觉得你之前在报税或做其他事情之前用过这个例子。

This year, I literally just told Codex to go get everything I needed and send it to the accountant.

今年，我实际上只是告诉 Codex 去获取我需要的一切并将其发送给会计师。

There was this massive list of stuff it had to use computers to click through and download.

有这么一大堆东西，必须使用计算机才能点击并下载。

It did it.

它做到了。

It was perfect.

太完美了。

A lot of this stuff it can already do.

很多这样的事情它已经可以做到。

Okay: give you 10x total productivity uplift.

好的：总体生产力提升 10 倍。

Basically, if it takes you a year to make a breakthrough now, you make a breakthrough every month.

基本上，如果你现在需要一年的时间才能取得突破，那么你每个月都会取得突破。

I think I would just refuse to give you a scalar on this.

我想我会拒绝给你一个标量。

We might already be past that in some types of work.

在某些类型的工作中，我们可能已经超越了这一点。

Let’s say you’re trying to do certain types of math, and—

假设您正在尝试做某些类型的数学，并且 -

Oh, sorry.

哦，抱歉。

But for you as AI researchers trying to advance the state of AI research.

但对于那些试图推动人工智能研究进步的人工智能研究人员来说。

How much are AI researchers sped up or uplifted?

人工智能研究人员的速度或提升有多大？

I’m realizing you probably have a very different definition of a fully general remote worker.

我意识到您可能对完全通用的远程工作者有非常不同的定义。

I could have specified that earlier.

我本可以早点指出这一点。

I’m picturing normal white-collar work over the period of a month.

我正在想象一个月内正常白领的工作。

I think it starts to diverge a little bit past two months.

我认为过去两个月开始出现了一些分歧。

A very competent white-collar worker, but not necessarily a super creative researcher.

一个非常有能力的白领，但不一定是一个超级有创造力的研究员。

I can kind of see that, actually, because right now it’s already definitely more than 10X for coding stuff.

实际上，我可以看到这一点，因为现在编码内容的速度肯定已经超过 10 倍。

So if it can do even one or two loops of experimental feedback, that would actually be massive already.

因此，如果它甚至可以进行一两个循环的实验反馈，那实际上已经是巨大的了。

So 10x uplift of AI researchers within two years.

因此，人工智能研究人员的数量在两年内增加了 10 倍。

If you plug that into a very naive model of AI progress and how much is coming from AI researchers, and there’s a 10x increase in their productivity, you have a radically accelerated pace of AI progress starting two years from now.

如果你将其代入一个非常幼稚的人工智能进展模型以及人工智能研究人员的贡献量，并且他们的生产力提高了 10 倍，那么从现在起两年后，人工智能的进展速度就会急剧加快。

I think this will mean that AI progress doesn’t get bottlenecked on AI researchers’ ability to run small experiments.

我认为这将意味着人工智能的进步不会因为人工智能研究人员进行小型实验的能力而受到瓶颈。

It gets bottlenecked on other things.

它会在其他事情上遇到瓶颈。

Of course.

当然。

But it just happens 10x faster, which is a huge deal.

但它的速度却快了 10 倍，这是一件大事。

That also helps the next thing, which gives you a 100x speedup, happen sooner, et cetera.

这也有助于接下来的事情，让你的速度提高 100 倍，发生得更快，等等。

My capacity to absorb information and make the Bayesian optimal decision on the next experiment.

我吸收信息并在下一个实验中做出贝叶斯最优决策的能力。

I’m assuming that you can delegate some of this to the AI.

我假设你可以将其中一些工作委托给人工智能。

The AI is becoming decent at deciding.

人工智能的决策能力正在变得越来越出色。

It’s run this experiment, it’s got this result, it runs the next experiment.

它运行了这个实验，得到了这个结果，它运行了下一个实验。

If it can run two or three experiments in a row without crashing, then that is actually a big uplift.

如果它可以连续运行两到三个实验而不崩溃，那么这实际上是一个很大的提升。

Okay, final question.

好的，最后一个问题。

An AI which dominates top human experts across every single field of work that can be done over a computer.

人工智能在可以通过计算机完成的每个工作领域中都统治着顶级人类专家。

So not only AI research, but all cognitive work.

因此，不仅是人工智能研究，还有所有认知工作。

Not just short-horizon work, but literally, if it takes three years or something, the AI will still do better than humans.

不仅仅是短期工作，而且从字面上看，如果需要三年或更长的时间，人工智能仍然会比人类做得更好。

AI is obviously getting more attention.

人工智能显然正在受到更多关注。

It’s one of the harder things, but a lot of energy is being put into it.

这是比较困难的事情之一，但我们投入了大量精力。

It’s also not one of the hardest things for AI, because it involves a lot of code and math, which models are really good at.

对于人工智能来说，这也不是最难的事情之一，因为它涉及大量的代码和数学，而这些都是模型真正擅长的。

For things that involve 3D and spatial stuff and physical stuff, I think that will take a little longer.

对于涉及 3D 和空间内容以及物理内容的事情，我认为这将需要更长的时间。

If it’s mechanical engineering or something, and it’s not getting the most attention right now, that might take a little longer.

如果是机械工程之类的，而且现在还没有得到最多的关注，那可能需要更长的时间。

But it also does include fields where there is relatively little data because of the nature of the field, and it has to learn that data on the fly.

但由于该领域的性质，它也确实包括数据相对较少的领域，并且它必须动态学习这些数据。

For example, it has to become superhuman at being an engineer at TSMC or something.

例如，它必须成为超人，成为台积电的工程师什么的。

So you would have to assume that you can give the AI the same onboarding material.

因此，你必须假设你可以为人工智能提供相同的入门材料。

Then something has to be solved about longer-horizon learning.

然后必须解决有关长期学习的问题。

Yeah, I think so.

是的，我想是的。

I think there are so many things in the world where, even if you have some sort of memory system external to the model, and even if context length grows a little bit, there are just fundamentally things where, even if you could research the information or write notes yourself, you’d need more than a million-token context window today.

我认为世界上有很多事情，即使你在模型之外有某种记忆系统，即使上下文长度增加了一点，但从根本上讲，即使你可以自己研究信息或写笔记，你今天也需要超过一百万个令牌的上下文窗口。

I kind of agree on the 5-year range, at least for the stuff that labs are focusing on.

我有点同意 5 年的范围，至少对于实验室关注的东西来说是这样。

But I think there’s going to be a long tail of stuff which the AI could theoretically go out and learn about, but no one has bothered to do it and the compute hasn’t been allocated to that.

但我认为，理论上人工智能可以出去学习很多东西，但没有人费心去做，计算也没有分配给它。

So that might take longer for literally every single human expert.

因此，对于每个人类专家来说，这可能需要更长的时间。

Sorry, but by this I also included the ability to learn a new domain as fast as a human.

抱歉，但我还包括了像人类一样快速学习新领域的能力。

I think that’s not necessarily necessary, because the AI will have vastly greater experience than any human.

我认为这不一定是必要的，因为人工智能将比任何人类拥有更多的经验。

Thanks so much for doing this, guys.

非常感谢你们这样做，伙计们。

I feel like this was a great format for getting different experts to disagree and debate and discuss things together.

我觉得这是一种很好的形式，可以让不同的专家提出不同意见，一起辩论和讨论问题。

It was very productive.

这是非常富有成效的。