2026 年 AI 现状：大模型、编程、扩展定律、中国、智能体、GPU、AGI
State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI
纵览 2026 年 AI 格局，从大模型与扩展定律到编程、智能体、GPU 算力、中国竞争与 AGI。
2025 年，Anthropic 在法庭上输了，要赔作者十五亿美元。花钱买书扫描那部分法院认了，用种子下载那部分没认。这笔钱在硅谷的讨论里几乎没留下什么痕迹，可它是十五亿美元。
这件事说明的东西比金额本身更重要：训练数据是这场竞赛里最贵、最不透明、最容易打起来的资产。第 490 期 Lex Fridman 播客里，Nathan Lambert 把话说得更直：
如果你加入一个前沿实验室，想产生影响，最好的做法就是找到更好的新数据。那些花哨的算法突破，比如想出怎么缩放强化学习，是科学家眼里最性感的想法。有团队做成了，可更多的贡献其实是“我把数据做得更好了”，或者“我把基础设施做得更好了，让我团队里每个人跑实验快了 5%”。（Nathan Lambert）
这期节目 2026 年 1 月底录制，嘉宾除了 Nathan，还有 Sebastian Raschka。前者是 Allen Institute for AI 的后训练负责人，写过 RLHF 领域的标准参考书；后者写过《从零构建大语言模型》和《从零构建推理模型》。三个人坐下来聊了一整年 AI 到底发生了什么，接下来一年可能会发生什么。
聊到最后能拎出一条主线：过去一年最大的变化不在模型架构上，在训练方法和数据质量上；模型之间的能力差距在收窄，差异化竞争正从模型本身转向产品形态和工程效率；2026 年的看点不在更大的模型，在推理成本、上下文管理和智能体系统的成熟度。
DeepSeek 丢了王冠，接班的还是中国公司
2025 年 1 月，DeepSeek R1 发布，整个行业震了一下。接近甚至达到当时最前沿的水平，据说用的算力少得多、成本低得多。一年过去，Nathan 的判断是 DeepSeek 正在失去它在中国开源模型界头把交椅的位置，接班的却不是美国公司。
现在有一大堆中国科技公司在发布非常强的前沿开源权重模型，DeepSeek 正在失去它作为中国最杰出开源模型制造者的王冠，Z.ai 的 GLM、MiniMax、月之暗面的 Kimi K2 Thinking，尤其最近几个月，光芒更盛。（Nathan Lambert）
DeepSeek 没变差。Sebastian 补了一个关键视角：其他公司在用 DeepSeek 的思路做自己的模型，架构几乎一样，只是训练得更晚，所以看起来更好。这种跳跃式竞争的结果就是，最新的模型往往就是当下最好的模型，领先者并没有做错什么。
有个被忽略的商业逻辑。Nathan 经常收到一些中国公司的主动接洽，对方很清楚一件事：美国的大科技公司和 IT 企业出于安全顾虑，不会为中国的 API 付费。开源权重模型于是成了进入美国 AI 支出市场的通道。企业下载权重、自己部署、自己微调，数据不出内网，安全顾虑绕开了，影响力照样打出去了。
许可证的友好程度也在起作用。Llama 和 Gemma 的许可证带用户数上限之类的附加条款，超过一定规模要向 Meta 或 Google 报告财务状况。中国这批模型用的是几乎没有附加限制的开源许可证。想做商业产品的团队，这个差别是实打实的。
两家中国开源公司已经开始走上市流程。MiniMax 和 Z.ai 都递了 IPO 材料，在西方做大量的市场推广，想要西方用户的注意力。DeepSeek 反而一直沉默，它由一家叫 High-Flyer 的对冲基金孵化，外界不清楚它到底拿模型干什么、在不在意这些事。同样是开源，背后的激励结构完全不同。
2025 年谁赢了，2026 年谁会赢
Lex 问得很直接。Nathan 的回答绕了一下，落脚点很明确。
2025 年的消费者聊天机器人赛道上，动量在 Gemini 这边，可它是从一个很低的起点追上来的。Bard 时代的失败不用回避，Google 顶着组织层面的混乱把 Gemini 3 做出来，这件事本身值得给 credit。Nathan 说自己不太敢押 Gemini 赢过 ChatGPT，OpenAI 作为在位者的优势太大，而且 OpenAI 看起来乱，却非常擅长把事情做成。
GPT-5 收到的评价很分裂，其中有个被低估的设计：high-line 功能做了个路由器，大部分用户不再像以前那样烧 GPU 成本。这是省钱的做法，也是让产品跑得更稳的做法。Nathan 说，很难把“我个人喜欢什么模型”和“什么能在普通用户那里产生差异化”分开来看。
2026 年，Nathan 的判断是：Gemini 会继续缩小和 ChatGPT 的差距，Google 的规模优势和把研究与产品分离的能力是结构性优势；Anthropic 会在软件和企业侧继续成功，这条路他们已经反复走过；基础设施这块 Google 的 TPU 给了成本优势，英伟达芯片的利润率太高，Google 能从芯片到数据中心自己搭一整套，不用付这笔溢价。
要赌一个全新的范式，Nathan 还是押 OpenAI。Deep Research、Sora、o1 这些定义性的东西都从 OpenAI 出来。他们的研究部门一次又一次证明了自己能把新想法变成产品。所以 2026 年大概率是关于规模，关于把模型里那些低垂的果实摘下来。
还有一个有意思的观察：三个人聊了二十分钟各自怎么在日常工具之间切换，谁都没提中国模型。Nathan 的解释很简单，美国模型目前就是更好，他愿意为那点边际智能付钱。中国模型在 OpenRouter、Perplexity 上都能跑，用起来图个新鲜，但不会回头。
我们不在这儿讨论中国模型，是因为美国模型目前更好，我们就用它。我试过那些开源模型，“挺好玩，但我不会回去用”。（Nathan Lambert）
他还提了一个技术层面的差异：从服务方式看，中国模型的每个副本用的 GPU 更少，代价是速度更慢、某些错误模式不一样。对用户来说，速度和智能同时占优才有意义。这个差距今年会不会反转，是这条赛道最值得盯的事。
架构没怎么变，变的是别的东西
这是整期节目里信息密度最高的一段。Sebastian 花了不少时间解释一个看着反直觉的事实：从 GPT-2 到 gpt-oss-120b，底层架构的变化远没有外界想象的那么大。
变化主要是几个模块的替换：多头注意力换成分组查询注意力，LayerNorm 换成 RMSNorm，激活函数从 sigmoid 一类换成 ReLU 一类，加上混合专家层。混合专家（Mixture of Experts，MoE）的逻辑是把原来那个昂贵的全连接层拆成多个专家网络，每个 token 只激活其中一小部分。有 256 个专家，路由器根据输入决定用哪几个。模型参数量可以做得很大，每次前向传播消耗的算力却不会同比例增长。
如果你熟悉深度神经网络，这就像把 sigmoid 换成 ReLU，并没有从根本上改变网络，只是一个小调整。基本上就是这样，它并没有本质上的不同，还是同一个架构。你可以通过添加这些改动，从一种架构转换到另一种。（Sebastian Raschka）
那进步到底发生在哪里？答案分三层。
第一层是训练阶段的分化。GPT-2 时代只有预训练，现在有预训练、中期训练、后训练。ChatGPT 本质上是一个 GPT-3，架构和 GPT-2 一样，新东西是监督微调加人类反馈强化学习。能力解锁发生在算法层面，不在架构层面。
第二层是系统工程。Nathan 提到英伟达发布会上的 FP8、FP4 这些精度优化，本质是让同样的硬件塞进更多计算、更快地做实验。训练代码库的变化是巨大的，架构看起来一模一样。拿同样的 GPU 训练 gpt-oss-20b，实际耗时可能远少于当年训练 GPT-2 的时间。这些系统层面的进步不直接给模型新能力，它让研究人员能更快试错、更快找到更好的配置。
第三层是替代架构的探索。文本扩散模型、Mamba 这类状态空间模型都在尝试，都有取舍。目前最前沿的性能还是来自自回归 Transformer，替代方案更多在成本敏感的场景里找空间。
缩放定律：四个旋钮都还转，性价比变了
Nathan 先给了个技术定义：缩放定律是计算量加数据量和留出集预测准确率之间的幂律关系。这个关系在预训练里已经验证了十几个数量级。后来 o1 引入了推理时缩放，让模型在回答前生成更多 token 来思考。再后来人们发现强化学习训练也能缩放，对数增加训练算力，线性提升评估表现。
现在是四个旋钮：预训练规模、强化学习训练时长、推理时计算量、上下文长度跟数据质量。问题不在哪个死了，在钱该往哪儿花。
Sebastian 算了一笔账。预训练是固定成本，训练完了能力就永远在那儿。推理时缩放按查询付费，用户越多花得越多。如果你的模型半年后就会被替换掉，花五千万美元多训练两个月可能不划算，不如把钱花在推理时缩放上。反过来，ChatGPT 有海量用户，每次查询都多烧推理算力会要命，所以 GPT-5 用了个更小的模型加路由器。
Nathan 补了成本结构里最关键的一点：训练成本相对于服务成本其实很低。DeepSeek 那个著名的五百万美元预训练成本按云市场价格算的。OLMo 3 的论文第 2.4 节详细写了集群租用、工程问题、多随机种子的总成本，大约两百万美元。可服务几亿用户的经常性成本是几十亿美元级别的算力。一千块 GPU 一天就是十万美元，这些公司可能上百万块 GPU 在跑。模型做大之后，服务成本的压力会倒逼团队把模型做小、把效率做高。
这些模型，很多人花一到一千万美元就能训一个出来，可用它服务几百万用户的经常性成本，是真的几十亿美元级别的算力。（Nathan Lambert）
关于“预训练已死”的说法，Nathan 认为那只是兴奋点转移了，实际发生的事情并非如此。他用 OLMo 3 的发布节奏举例：十一月二十号的截止日期，强化学习跑了五天；十二月又跑了一次，让 RL 多跑三周半，模型明显变好，就发布了。这种决策在训练模型时经常发生，你不能无限期留着不发布。研究人员的新改进要收进来，要重做预训练，要做后训练，要给用户，要做安全测试。整个循环被这些现实约束推着往前走。
Sebastian 的补充更干脆：在算力无限的理想世界里，四个旋钮全开。现实里算力有限，你要决定这一笔钱花在把模型做大，还是花在把知识解锁得更彻底。今年 Claude 4.5 的表现就说明了一件事，把大量预算砸进预训练未必是最好的选择，同样的钱花在推理时缩放或者后训练上可能回报更高。
数据才是真正的护城河
预训练数据集的规模以万亿 token 计。小模型五到十万亿，Qwen 有文档记录到五十万亿，据说闭源实验室能到一百万亿。真正用来训练模型的只是这个漏斗里很小的一部分。
合成数据的定义比很多人想的宽。DeepSeek 和 AI2 都做过 OCR 方向的论文，用光学字符识别从海量 PDF 里提取文本，这被归类为预训练的合成数据。ChatGPT 现在能给出很好的答案，拿这些最好的答案去训练，也是合成数据。这和早期 ChatGPT 幻觉数据完全不是一回事。
数据配比的方法论已经相当科学化。从 Common Crawl 里采样不同来源的小样本，比如 GitHub、Stack Exchange、Reddit、维基百科，训练小模型，测评估表现，做线性回归，得到最优配比。评估集一变，最优配比就变了。OLMo 3 做的大量工作就是加入推理相关的新数据源，让模型在数学和代码上更强，然后重新跑这套配比流程。
如果你改变评估方式，你的数据集就会大幅改变。OLMo 3 的很多工作就是加入新的推理数据源，让模型在数学和代码上更强，然后做这个混合流程，它会给你答案。（Nathan Lambert）
版权这块，2025 年还有一层复杂性被大多数人略过了。即使你买了电子书，用它训练模型也处在灰色地带。大公司已经开始找私有企业谈专有数据授权，私有企业越来越保护自己的数据，因为那是未来几年的护城河。制药、法律、金融这些行业，迟早会从前沿实验室挖人，用自己的私有数据训练内部模型。这些数据你根本拿不到，临床试验数据大部分时候不对外开放。从这个角度看，预训练缩放远没有死，只是战场从通用模型转向了垂直领域。
RLVR 解锁了什么
可验证奖励强化学习（Reinforcement Learning with Verifiable Rewards，RLVR）是 2025 年最重要的训练方法突破。Nathan 的团队在 DeepSeek 之前就在 Tulu 3 的工作里提出了这个术语，真正的训练突破是 DeepSeek 做的。
原理不复杂：让模型生成答案，用可验证的方式判断对错，正确率就是强化学习的奖励。数学题有标准答案，代码题能跑测试用例，这些领域天然适合。传统 RLHF 用的是人类偏好训练出来的奖励模型，问题域一下子窄了很多。换成可验证奖励之后，优化可以扩展到更大的规模。
Sebastian 给了一个亲手做过的例子：拿 Qwen 3 基础模型在 MATH-500 上跑 RLVR，基础模型准确率大约 15%，只跑了 50 步，几分钟，准确率从 15% 跳到 50%。
这个例子背后有争议。Nathan 指出，今年有两篇论文讨论 Qwen 的数据污染问题，具体表现是：如果你把数学题的单词保留但改掉数字，Qwen 会不加工具地给出一个高精度小数答案。这意味着它在某个时刻见过几乎一模一样的问题，并且用工具算出了高精度结果。一个不带工具的语言模型不可能天然产生这种输出。所以从 15% 到 50% 的提升，到底有多少是真正学会了新知识，有多少只是把预训练里已有的东西激活出来，研究社区还没有定论。
即使只是激活，这件事本身也很漂亮。你只给模型问题和答案，不约束中间过程，模型会自己发展出逐步推导的行为，会在发现自己算错的时候说“啊，我搞错了，让我重来”。DeepSeek R1 论文里把这个叫做 aha moment。Nathan 的补充是，预训练阶段模型已经看过整个互联网，包括数学课的转录文本里有人说“你试试这个，哦我搞砸了”。RLVR 做的是放大这类行为，因为它们在让模型思考更久、检查自己工作的时候非常有用。
我认为这些 aha moment 有点假，因为预训练阶段你实际上已经看过了整个互联网。你肯定见过有人在解释自己的工作，甚至是口头的，比如数学讲座的转录：“你试试这个，哦我搞砸了。”强化学习，这套 RLVR，非常擅长放大这些行为，因为它们在让模型思考更久、检查自己工作的时候非常有用。（Nathan Lambert）
RLVR 和推理时缩放是一个完美的组合。DeepSeek R1 论文展示了训练越久，模型回答越长，用的 token 越多。简单任务上变贵了，复杂任务上的准确率提升了。这两个东西互相强化，一起改变了模型的使用体验。
后训练的完整链条
Nathan 把后训练拆成三段讲。
中期训练提供模型学习技能所需的基础。o1 这类推理模型能出现，前提是中期训练阶段有大量精心策划的推理轨迹数据，就是模型把问题拆成中间步骤、一步步求解的文本。没有这些数据，后训练阶段的强化学习就没有可放大的行为。
RLVR 让模型在难题上反复试错，把大量算力投入到 trial-and-error 学习里。这里有个关键机制叫 GRPO，奖励基于同一个问题的多个答案之间的相对好坏来分配。所有答案都一样，就没有信号。所以实验室在找更难的题，这就是为什么你会听到科学领域、更难的软件工程问题，那些领域里模型很难全对，才有训练信号。
RLHF 是最后收尾，让模型更好用。组织方式、语气、风格这些人类反馈能教会的东西，是 ChatGPT 当年让人觉得神奇的根源。Nathan 说风格、格式和解题方法在训练时其实紧密相连，RLHF 仍然可以让模型数学更好，可验证领域是更直接的路径。
算力分配上，Nathan 提到 Grok 4 透露过预训练和后训练用的算力差不多。两者的硬件特性完全不同。预训练是计算密集型的，所有 GPU 互相通信，效率极高。强化学习是内存密集型的，要生成很长的序列，注意力机制在长序列下内存是二次增长。从时间维度看，RL 训练的天数正在接近预训练，同时用的 GPU 数量少一些。实验室里有条经验法则，预训练不要超过一个月，因为大规模训练失败是常态，十万块 GPU 里几乎肯定至少有一块是坏的。你规划了两个月的大集群第五十天挂了，机会成本太大。GPT-4 那种训练三个月、所有人都吃惊它居然成了的 YOLO 式做法，现在没人想再来一次。
从零构建是最好的学习方式
Sebastian 的两本书核心方法论就一句话：自己动手写代码。图的错误你可能看不出来，概念解释你可能理解偏，代码能跑就是对的，不能跑就是错的。这是最精确的验证方式。
他推荐的学习路径很具体：先用 GPT-2 这样的小模型在一张 GPU 上跑通，理解注意力机制、预训练、监督微调各自在干什么。然后去 Hugging Face 上看目标模型的 config 文件，一百行左右的人类可读配置里写清楚了用了多少层、什么注意力机制、什么位置编码。从 GPT-2 出发，把这些组件一个个加上去，加载预训练权重，看输出能不能和 Transformers 库的参考实现对齐。对齐就等于架构写对了，等于给自己发了一个验证奖励。
最难的地方在于找到你要理解的那一部分，这就像大海捞针。但好在它有一个能跑的完整实现，你可以反向工作。（Sebastian Raschka）
他自己复现 OLMo 3 的时候，卡在 RoPE 位置编码的 YaRN 扩展上，那里有自定义的缩放逻辑，一开始怎么都对不上。在一整天的挣扎里，你反而真正理解了那个东西在干什么。最后用单元测试对上了参考实现，那种确认感是看论文得不到的。
这里有个更深的讨论。Lex 提到一份对 791 名十年以上经验的专业开发者的调查，结果显示资深开发者比初级开发者更倾向于让 AI 生成超过一半的交付代码。直觉上你会觉得初级开发者更需要 AI，数据反过来了。一种解释是资深开发者更擅长审查 AI 生成的代码，所以更信任它。另一种解释是 AI 还解决不了初级开发者面对的那些问题。
Sebastian 的担忧是：你从来不自己动手，怎么成为专家？数学课本上直接看答案也能学到东西，先自己做一遍、卡住、再去看答案，你会把解法放进自己的心智框架里。LLM 永远在旁边，你还愿不愿意经历那个挣扎的过程？挣扎不舒服，不经历又永远到不了那个能用 LLM 做杠杆的专家位置。
我认为存在一个黄金平衡点，也许诀窍是你每天安排专门的两小时离线时间用来学习，一天中其余时间使用 LLM。我认为人们仍然要投资自己，这很重要，不能把所有事都交给 LLM。（Sebastian Raschka）
职业选择：学术界、开源实验室还是前沿实验室
Nathan 把这几个选项的梯度讲得很直白：越闭源，钱越多，你能拿到的 credit 越少。学术界能清楚地建立作品集，博士生基本不拿钱，能走这条路的人往往有比较稳定的安全网。前沿实验室给的钱是改变人生的量级，OpenAI 平均每位员工每年股票收入超过一百万美元，可你在里面是一台机器上的齿轮，发表越来越少。
他带学生的时候会跟学生说：如果你拿到的是 OpenAI 的 offer，离开博士项目我觉得可以理解；如果是个可能归零的随机创业公司，别去。
有个观察很有意思。Nathan 说他的教授朋友平均来说比在前沿实验室工作的朋友更快乐。原因不在工作量，教授要写基金、要教书、要做研究，三份工作合一。原因在有学生、有持续的指导关系、有一个跟人有关的使命。在事情变化很快、很混乱的时代，这种东西对人的滋养很强。
9/9/6 的工作文化被拿出来讨论，早九点到晚九点、一周六天，七十二小时。Sebastian 说这几乎翻转了他对学术界的认知，以前当教授是三份工作合一的压力，现在前沿实验室的强度让教授相比之下显得没那么夸张。Nathan 提到《苹果在中国》那本书里写的供应链团队，有专门的“拯救婚姻”项目代号，同事看到某个人需要回家陪家人，会发红色警报让这个人周末必须回去。书里还提到有人因为这种工作强度去世。这些人没有被强迫，他们对产品太有热情，自己进入了那种状态。Sebastian 说自己也有过，做出过背部和颈部的毛病，没人逼他，是他自己想干。
文本扩散、工具调用、持续学习
文本扩散模型的逻辑从图像生成搬过来。图像那边用扩散过程逐步去噪，效果超过了 GAN。文本是离散的，直觉上不好做去噪，BERT 那种并行填空的思路给了启发：从一段随机文本出发，多轮迭代逐步精炼。好处是一次可以生成多个 token，理论上更快。Google 已经发布了 Gemini Diffusion，声称在多数基准上同等质量下生成速度快很多。
Nathan 听到的实际用例是代码创业公司。用户说“改一下这个”，返回的是一大段代码 diff，不需要太多外部上下文，用扩散模型能很快生成很长的 diff。用自回归模型要几分钟，用户等不了，每一秒都在流失用户。工具调用这块扩散模型不好做，自回归的链条被外部工具打断之后，扩散的并行生成没法自然衔接。Nathan 原本以为不同类型模型会更快地分化到不同用途，实际比预想慢。
工具调用本身是 2025 年一个重要的解锁。gpt-oss 是第一个真正带着工具使用意识训练的开源权重模型，生态还没完全准备好。幻觉问题最好的解法之一就是别让模型硬记，数学用计算器、事实查搜索。Sebastian 说这没法完全解决幻觉，模型得知道什么时候该调工具，互联网上的信息也不总是对的。方向是对的。
还有一篇 Sebastian 很喜欢的论文，递归语言模型。做长上下文任务时把整段内容拆成子任务，让模型决定哪些子任务合理，递归调用自己去解决，每个子任务可以调工具去查资料，最后汇总。这个思路的价值在不改进模型本身，改进的是模型被使用的方式。
持续学习是目前最模糊的地带。Nathan 给了个判断标准：语言模型能解决很多任务，可 AI 社区一个关键里程碑是 AI 能替代任何远程工作者，接收信息、解决数字任务。限制在语言模型不会像员工那样从反馈里学习。你雇一个编辑，他可能犯错，你告诉他，他不会再犯。语言模型没有这种快速修改自己、快速学习的能力。
反方观点是：也许不需要改权重，只需要给更多上下文。你可以写很长的文档，说“这是我所有的博客文章，我喜欢这种写法，我的声音基于这个”。大部分人不会给模型提供这些。智能体模型才刚刚开始。取舍就在这里：通过持续学习更新权重让模型快速学习，还是给更多上下文、靠模型本身足够聪明来制造“学会了”的表象。
Sebastian 的看法更务实：权重更新这件事已经有不同版本在做了。GPT-5 到 5.1 到 5.2 就是快速策划的更新，社区反馈了做不到的事情，更新权重，发布下一个模型。RLVR 是更细粒度的版本，跑一遍就更新一次。问题在不能对每个人单独做，成本太高。这件事只有在设备端、成本由消费者承担的时候才可行，就像苹果把模型放在手机上让它从使用中学习。
机器人、世界模型和 AGI 时间线
Sebastian 提到 Meta 有一篇叫 Coder World Models 的论文，把世界模型的概念用到 LLM 上：不只检查最终答案对不对，还检查中间变量对不对。模型在学一个代码环境。这个方向很合理，只是贵。它让整个过程更精细，不只建模结果，还建模过程。
他回忆起读研时参加 CASP 蛋白质结构预测竞赛的经历。AlphaFold 第一版显式建模了物理相互作用、分子物理、不可能的角度这些东西。到第二代，这些被去掉了，直接暴力缩放。LLM 现在正处在暴力缩放有效的阶段，Sebastian 觉得某个时间点把物理约束那套东西找回来可能是有意义的。CASP 那种“benchmark 存在但没人知道答案，要等实验揭晓”的机制，LLM 领域也需要一个。
机器人这块 Nathan 的态度很分裂。他对消费级家用学习型机器人非常悲观，对自动驾驶非常乐观，对仓储自动化这类为机器人重新设计过的环境也非常乐观。Amazon 建了全新的配送中心，从一开始就为机器人设计，没按人类习惯建好再塞机器人进去。这条路径合理得多，重复性任务、人类能做但不想做、环境可以重新设计。可它比大部分人预测的花更长时间。
Lex 补了一个很少被提及的点：安全。LLM 里的失败模式是有趣的谈资，在机器人领域、在人们家里、在数百万分钟和数十亿次交互里，你几乎不被允许失败。具身系统放到真实世界里，你要解决的问题比单纯思考通用机器人学习问题多得多。
AGI 和 ASI 的定义没有人真正达成一致。Nathan 说有个还算合理的锚点是“能复现大部分数字经济工作”，远程工作者是个不错的例子。这个定义他不喜欢。今天语言模型虽然强大，远不是即插即用的远程工作者。有些事比远程工作难得多，做出一个你事先无法设想的科学发现，或者读取所有医疗记录，找到没人知道的疾病关联，发现某种常见药能治疗某种罕见癌症。这些是超级智能级别的问题。
AI 2027 报告把目标定在超级人类程序员，然后超级 AI 研究员，再是超级智能 AI 研究员，最后是完整 ASI。最初的预测 2027 到 2028 年，后来推后三到四年到 2031 年。Nathan 的预测比 2031 年还要靠后。他所属的阵营认为 AI 是锯齿状的，有些事极其擅长，有些事非常糟糕。当模型接近自动化软件工程师的时候，它在传统 ML 系统和前端上很强，在分布式 ML 上很差，因为大规模分布式学习的训练数据很少。这种锯齿会持续存在。人类有创造力，会利用模型的超强能力去填补模型的弱点，快速推进。最好的 AI 研究员就是能启用这种超能力的人。
大科技公司花一千亿美元的速度，会比我们得到一个自动化 AI 研究员、引发 AI 研究奇点的速度快得多。（Nathan Lambert）
广告、并购和 Llama 的黄昏
Nathan 预测广告会来。现在 LLM 便宜是因为大量补贴，最终会通过广告变现。Google 最有可能先做成这件事，它已经有广告供应侧。如果它想清楚怎么把 Gemini 应用的需求转化成有用的广告，随时可以开闸。现在没人做的原因很现实：竞争对手都没做，先做的会因为新闻头条损失声誉。只要有人开了头，飞轮一旦转起来，十年后广告收入能支撑更好的研发、更好的模型，这个逻辑是 YouTube 主导视频市场的逻辑。Netflix 害怕 YouTube，因为 YouTube 的广告模式让它能持续投入，Netflix 靠订阅费。
并购方面，Dario 不会卖 Anthropic，这是 Nathan 的判断。整合压力已经出现：Groq 估值两百亿美元，Scale AI 接近三百亿。有些交易结构对硅谷生态是有害的，那种授权式收购，没做到全员被带上，只拿走顶尖人才。你加入一家创业公司，即使不太成功，公司也可能被低价收购，你的期权能兑现。授权式收购把这个生态的底层逻辑抽掉了。
Llama 的故事被拿出来当案例。Nathan 不觉得会有开源权重的 Llama 5。Sebastian 的分析是：Llama 1、2、3 是开源权重模型的先驱，社区很爱。问题出在上层管理者看到 Llama 在社区里受欢迎，想把开源当成更大的声量工具。Llama 4 那些超大模型是为了刷榜，可 Llama 的目标从来不该是刷榜击败 ChatGPT。目标是让人们能用、能信任、能修改、能理解。这包括有小模型，不必是最好的模型。这些模型因为用了偏好数据专门调过，在 benchmark 上表现比实际好，小模型又没做，没人跑得动大模型，整个事情就变得很奇怪。Nathan 的补充更直接：在内部分歧和错位的激励下崩溃了，研究人员想做出最好的模型，中间有一层管理层想证明自己做了这些事。
还有个被低估的细节。社区对 Llama 4 的猛烈批评，可能反过来推了 Meta 一把。公司本来指望正面报道，结果是负面报道，心想“我们试着做了件好事，给你们一个开源模型，你们这么骂”，于是就有了“那我们重新评估一下和开源的关系”。X 上的舆论生态能把社区带偏，Groq 4.1 和 Grok Code Fast 也是类似的例子，Reddit 和 X 上没人夸，用的人却不少。
ATOM 项目和开源模型的美国缺口
Nathan 花了五个月做政策方面的工作，推一个叫 ATOM 的项目，American Truly Open Models。起因是 2025 年 7 月，市面上有四五个 DeepSeek 级别的中国开源权重模型，美国这边是零。
他的论证逻辑很清晰：开源模型是 AI 研究的引擎，人们做研究是从这些模型开始的，拥有它们很重要。美国应该在造最好的模型，这样最好的研究发生在美国，美国公司能从“AI 研究发生在这里”这件事上获取价值。现在的局面是，网站上的图表全是 Qwen、Qwen、Qwen，这些来自中国公司的优秀模型正在美国和国际上积累影响力。美国在 AI 上的支出远超中国。造一个比闭源前沿实验室落后半代到一代的开源模型，成本大约一亿美元。很多钱，跟这些公司的体量比不算什么。需要一个中心化的力量把愿意做这件事的人聚起来。
反馈是积极的。NSF 给了 AI2 四年一亿美元的资助，这是 NSF 历史上最大的计算机科学 grant。英伟达 2025 年动作更大，Nemotron 3 发布的模型和部分数据都更开放，英伟达这个体量的公司很少有愿意这么做的。Reflection AI 二十亿美元的融资明确说要造美国开源模型。白宫的 AI 行动计划里有专门一节叫“鼓励开源和开放权重 AI”，定义这类模型并论证它们对创新和创业的独特价值。
Nathan 说他本来可以让这个故事更病毒式传播，讲中国 AI 和威权国家结合、变成 ASI、统治世界、所以我们需要自己的美国模型。他刻意讲创新和科学，那个结局更现实，也是他更愿意去实现的世界。
关于曼哈顿计划式的中央化和保密，Sebastian 和 Nathan 都认为不可能。知识一旦存在就 containment 不住。训练这些模型的成本，一亿到一亿美元，全世界有大量的人能够达到。这些模型会在世界各地被训练出来。要阻止这件事，需要对互联网做一次彻底的重构，建一道美国的大防火墙，而防火墙是已知不太好用的。
硬件、伟人和一百年后
英伟达的护城河在 CUDA 生态，二十年积累下来的兼容性。Sebastian 读研时做生物物理模拟、分子动力学，实验室里就有一块 Tesla GPU 专门做计算。那时候没人知道这是为 AI 准备的。有了 LLM 之后，复制一个 CUDA 容易多了，当初花十五年是因为难。训练和推理的算力分离正在发生，Groq 被收购就是为了这个，Vera Rubin 架构里有专门为 pre-fill 设计的芯片，不需要高带宽内存，per flop 的拥有成本低很多。只要 AI 进展的速度快，英伟达的平台最灵活，人们会想要这个。如果停滞了，定制芯片就有更多时间去做。
关于伟人史观，Nathan 的观点是：这些人让事情发生得更早、更快。科学上很多伟大科学家都承认自己在正确的时间出现在正确的地点，最终别人也会有这个想法。Jensen 让 GPU 革命 manifest 得更快、更聚焦。ChatGPT 最终会发生，这种建设最终会发生，不会这么快。另一条时间线里，深度学习可能再等二十年，或者又一个寒冬来了，因为 GPU 不在。Sebastian 说这就像投资 ETF 和个股，个股波动更大，最终会到同一个地方，区别在聚焦。热情和专注让事情提前发生。
一百年后回头看，Sebastian 认为被记住的是“计算”这个大伞。被记住的并非 AI、CUDA、GPU 的具体细节，是计算本身。这可能是一个摩尔定律式的讨论。Nathan 想的是连接性，互联网和算力能不能合并。连接人这件事非常基础，你想找到世界上某个领域最好的人，那个人在世界某个地方，信息能流动到那里是根本性的。AI 也依赖这个。他之前说“一个中心模型”的梦想正在死去，正在演化的是人们为不同任务拥有多个 agent，已经在发生了，不同任务用不同 Claude。这被描述为数据中心里的多个 AGI，每个管理一些东西，互相通信。这极其依赖网络和信息自由流动，网络，尤其 GPU 之间的网络，本身就是算力缩放的一部分。
一百年后的人怎么和设备交互。Sebastian 说很难想象还是手机和笔记本电脑，会不会是植入设备。Lex 说必须是脑机接口，除非人类和现实交互的方式彻底改变。Nathan 觉得人们还是会随身带一块物理的计算砖，人需要一个私密的接口。可能不像 iPhone，可能用得少很多，你还是会带点什么。他不确定人脑能不能处理非视觉方式灌进来的信息，比如你给计算机一个信号，然后你就知道了你的收件箱，这事人脑能不能承受。人一百年不会变。Agency 和社区是人真正想要的东西，你亲近的人、能和他们一起做事、能给自己的生活赋予意义。这些东西不会在我们可以讨论的时间尺度上被人类生物学改变。
Lex 补了一段必须被记住的话：社会转型过程中大量工作消失，每一个失去的工作都是一个正在受苦的人。落到每个具体的人身上，那是真实的个人悲剧。我们讨论经济学、讨论 GDP 会好、讨论新工作会被创造出来的时候，不能忘记这一点。Nathan 说他希望社会在 AI 垃圾内容里淹得足够深，然后突然醒悟，受不了了，物理世界的东西重新获得溢价。就像艺术，数字复制品可以完美，你站在博物馆看真迹，你知道那是人手做出来的，那种欣赏是不一样的。
关于人类和机器的战争，Sebastian 说他不担心 AI 接管，原因在意识。人类决定自己想做什么，AI 在目前的实现里，你告诉它做什么它就做什么。代理权还在你手里，它没有拿走你的代理权，它变成了一个工具。它比其他工具更自动、更强大，仍然是你在掌控。
如果要在 2026 年做点什么
这期节目信息量很大，落到行动上，对读者真正有用的东西集中在几个方向。下面这几条可以作为自己的行动清单。
- 自己从零写一遍 GPT-2，然后复现一个当红开源模型的架构。单张 GPU 能跑通就行。看 config 文件，列组件，从 GPT-2 出发一个个加上去，加载预训练权重，用输出对齐参考实现作为验证。这个过程的摩擦本身就是学习。
- 把每天两小时设成离线学习时间，其余时间才用 LLM。挣扎不是敌人，卡住才是真正在学的证据。把一切外包出去，两年后你会发现自己再也够不着专家用 LLM 做杠杆的那个位置。
- 如果做研究，先做窄，再做深。读三五篇论文就能站到某个细分领域的前沿，然后写一封认真的邮件给作者。匿名 X 账号和 ML 圈的距离比想象中近得多。没算力的人做 eval 是最优解：找到 Claude 明显失败的地方，如果你的 eval 被下一代模型写进博客，职业火箭就起飞了。
- 如果做产品，关注文本扩散在长 diff 生成上的应用，以及工具调用在开源生态里的缺口。前者已经被代码创业公司用来解决用户等待时间问题，后者是开源模型落后闭源模型一个身位的地方，谁补上都可能吃到红利。
- 如果做企业决策，考虑用中国开源权重模型做内部部署。许可证干净，可以在自己的硬件上跑，数据不出内网。美国大公司因为安全顾虑不会买中国 API，下载权重自己部署是另一回事。这正在成为很多美国创业公司的默认选项。
- 如果你是资深开发者，别只把 AI 当加速器，把它当教你新东西的合作者。调查数据里资深开发者比初级开发者更依赖 AI 生成交付代码，真正的杠杆在于用 AI 拓展你原本够不着的领域，用它做你已经会做的事价值有限。
- 如果你在硅谷，走出去。读历史书，去别的地方看看。Twitter 和 Substack 不是整个世界。泡泡有用，待在泡泡里太久会失去判断现实的能力。一百年后回头看，被记住的是计算本身。计算最终服务于什么，取决于此刻还在泡泡外面看着这一切的人。
The following is a conversation all about the state of the art in artificial intelligence, including some of the exciting technical breakthroughs and developments in AI that happened over the past year, and some of the interesting things we think might happen this upcoming year.
以下是有关人工智能最新技术的对话，包括过去一年中人工智能领域发生的一些令人兴奋的技术突破和发展，以及我们认为明年可能会发生的一些有趣的事情。
00:00 At times, it does get super technical, but we do try to make sure that it remains accessible to folks outside the field without ever dumbing it down.
有时，它确实变得非常技术性，但我们确实努力确保它仍然可供领域外的人们使用，而不会降低它的水平。
It is a great honor and pleasure to be able to do this kind of episode with two of my favorite people in the AI community, Sebastian Raschka and Nathan Lambert.
能够与人工智能社区中我最喜欢的两个人 Sebastian Raschka 和 Nathan Lambert 一起制作这样的剧集，我感到非常荣幸和高兴。
They are both widely respected machine learning researchers and engineers who also happen to be great communicators, educators, writers, and X posters.
他们都是广受尊敬的机器学习研究人员和工程师，同时也是伟大的传播者、教育家、作家和 X 海报。
00:51 Sebastian is the author of two books I highly recommend for beginners and experts alike.
塞巴斯蒂安是我强烈推荐给初学者和专家的两本书的作者。
First is Build a Large Language Model from Scratch, and Build a Reasoning Model from Scratch.
首先是从头开始构建大型语言模型，然后从头开始构建推理模型。
I truly believe in the machine learning and computer science world, the best way to learn and understand something is to build it yourself from scratch.
我坚信在机器学习和计算机科学世界中，学习和理解某些东西的最好方法就是从头开始自己构建它。
Nathan is the post-training lead at the Allen Institute for AI, and author of the definitive book on reinforcement learning from human feedback.
Nathan 是艾伦人工智能研究所的培训后负责人，也是关于从人类反馈中进行强化学习的权威书籍的作者。
Both of them have great X accounts, great Substacks.
他们俩都有很棒的 X 账户和很棒的子堆栈。
Sebastian has courses on YouTube, Nathan has a podcast.
Sebastian 在 YouTube 上有课程，Nathan 有播客。
And everyone should absolutely follow all of those.
每个人都应该绝对遵循所有这些。
This is the Lex Fridman podcast.
这是莱克斯·弗里德曼播客。
01:40 To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, get feedback, and so on.
为了支持它，请在描述中查看我们的赞助商，您还可以在其中找到联系我、提出问题、获取反馈等的链接。
And now, dear friends, here’s Sebastian Raschka and Nathan Lambert.
现在，亲爱的朋友们，这是塞巴斯蒂安·拉斯卡和内森·兰伯特。
01:57 So I think one useful lens to look at all this through is the so-called DeepSeek moment.
因此，我认为观察这一切的一个有用的镜头就是所谓的 DeepSeek 时刻。
This happened about a year ago in January 2025, when the open weight Chinese company DeepSeek released DeepSeek R1 that I think it’s fair to say surprised everyone with near or at state-of-the-art performance, with allegedly much less compute for much cheaper.
这发生在大约一年前的 2025 年 1 月，当时开放权重的中国公司 DeepSeek 发布了 DeepSeek R1，我认为可以公平地说，它以接近或达到最先进的性能让所有人感到惊讶，据称计算量要少得多，而且价格要便宜得多。
And from then to today, the AI competition has gotten insane, both on the research level and the product level.
从那时到今天，人工智能的竞争无论是在研究层面还是在产品层面都变得疯狂。
It’s just been accelerating.
它只是一直在加速。
02:32 Let’s discuss all of this today, and maybe let’s start with some spicy questions if we can.
今天让我们讨论所有这些，如果可以的话，也许我们可以从一些尖锐的问题开始。
Who’s winning at the international level?
谁在国际层面获胜？
Would you say it’s the set of companies in China or the set of companies in the United States?
你说是中国的公司集合还是美国的公司集合？
Sebastian, Nathan, it’s good to see you guys.
塞巴斯蒂安，内森，很高兴见到你们。
So Sebastian, who do you think is winning?
那么塞巴斯蒂安，你认为谁会获胜？
So winning is a very broad term.
所以获胜是一个非常广泛的术语。
02:53 I would say you mentioned the DeepSeek moment, and I do think DeepSeek is definitely winning the hearts of the people who work on open weight models because they share these as open models.
我想说你提到了 DeepSeek 时刻，我确实认为 DeepSeek 绝对赢得了开放权重模型研究人员的心，因为他们将这些模型作为开放模型进行共享。
Winning, I think, has multiple timescales to it.
我认为，胜利有多个时间尺度。
We have today, we have next year, we have in ten years.
我们有今天，我们有明年，我们有十年。
One thing I know for sure is that I don’t think nowadays, in 2026, that there will be any company having access to a technology that no other company has access to.
我确信的一件事是，我认为现在到 2026 年，不会有任何公司能够获得其他公司无法获得的技术。
And that is mainly because researchers are frequently changing jobs, changing labs.
这主要是因为研究人员经常更换工作、更换实验室。
They rotate.
他们旋转。
So I don’t think there will be a clear winner in terms of technology access.
因此，我认为在技术获取方面不会有明显的赢家。
03:37 However, I do think the differentiating factor will be budget and hardware constraints.
然而，我确实认为差异化因素将是预算和硬件限制。
I don’t think the ideas will be proprietary, but rather the resources that are needed to implement them.
我不认为这些想法是专有的，而是实现它们所需的资源。
And so I don’t currently see a winner-takes-all scenario.
因此，我目前没有看到赢家通吃的情况。
I can’t see that at the moment.
我现在看不到。
Nathan, what do you think?
内森，你觉得怎么样？
You see the labs put different energy into what they’re trying to do.
你会看到实验室在他们想做的事情上投入了不同的精力。
04:00 To demarcate the point in time when we’re recording this, the hype over Anthropic’s Claude Opus 4.5 model has been absolutely insane.
为了界定我们记录这一点的时间点，对 Anthropic 的 Claude Opus 4.5 模型的炒作绝对是疯狂的。
I’ve used it and built stuff in the last few weeks, and it’s almost gotten to the point where it feels like a bit of a meme in terms of the hype.
在过去的几周里，我已经使用了它并构建了一些东西，就炒作而言，它几乎已经达到了一种模因的程度。
It’s kind of funny because this is very organic, and then if we go back a few months ago, Gemini 3 from Google got released, and it seemed like the marketing and wow factor of that release was super high.
这很有趣，因为这是非常有机的，然后如果我们回到几个月前，Google 发布了 Gemini 3，而且该版本的营销和令人惊叹的因素似乎非常高。
But then at the end of November, Claude Opus 4.5 was released and the hype has been growing, while Gemini 3 was before this.
但随后在 11 月底，Claude Opus 4.5 发布，炒作不断升温，而 Gemini 3 则更早。
04:44 And it kind of feels like people don’t really talk about it as much, even though when it came out, everybody was like, this is Gemini’s moment to retake Google’s structural advantages in AI.
感觉人们并没有真正谈论它，尽管当它问世时，每个人都在想，这是双子座重新夺回谷歌在人工智能领域结构性优势的时刻。
Gemini 3 is a fantastic model, and I still use it.
Gemini 3 是一款很棒的型号，我仍然在使用它。
It’s just that differentiation is lower.
只是差异化较低。
I agree with what you’re saying, Sebastian, that the idea space is very fluid, but culturally Anthropic is known for betting very hard on code, and this Claude Code thing is working out for them right now.
塞巴斯蒂安，我同意你所说的，即想法空间非常流动，但从文化上来说，人类以在代码上下注而闻名，而克劳德代码的事情现在正在为他们带来好处。
So I think that even if the ideas flow pretty freely, so much of this is bottlenecked by human effort and the culture of organizations, where Anthropic seems to at least be presenting as the least chaotic.
所以我认为，即使想法非常自由地流动，其中很大一部分受到人类努力和组织文化的瓶颈，而人类似乎至少表现得最不混乱。
05:23 It’s a bit of an advantage if they can keep doing that for a while.
如果他们能继续这样做一段时间，这会是一个优势。
But on the other side of things, there’s a lot of ominous technology from China where there are way more labs than DeepSeek.
但另一方面，中国有很多不祥的技术，中国的实验室比 DeepSeek 多得多。
DeepSeek kicked off a movement within China similar to how ChatGPT kicked off a movement in the US where everything had a chatbot.
DeepSeek 在中国掀起了一场运动，类似于 ChatGPT 在美国掀起了一场运动，在美国，一切都有聊天机器人。
There are now tons of tech companies in China that are releasing very strong frontier open weight models, to the point where I would say that DeepSeek is kind of losing its crown as the preeminent open model maker in China, and the likes of Z.ai with their GLM models, MiniMax’s models, and Kimi K2 Thinking from Moonshot, especially in the last few months, have shone more brightly.
现在中国有大量的科技公司正在发布非常强大的前沿开放权重模型，以至于我想说 DeepSeek 正在失去其作为中国杰出开放模型制造商的桂冠，而像 Z.ai 的 GLM 模型、MiniMax 模型和 Moonshot 的 Kimi K2 Thinking 等公司，尤其是在过去的几个月里，表现得更加耀眼。
06:04 The new DeepSeek models are still very strong, but that could be looked back on as a big narrative point where in 2025 DeepSeek came and provided this platform for way more Chinese companies that are releasing these fantastic models to have this new type of operation.
新的 DeepSeek 模型仍然非常强大，但这可以看作是一个重要的叙事点，2025 年 DeepSeek 的出现为更多中国公司提供了这个平台，这些公司正在发布这些出色的模型来进行这种新型运营。
These models from these Chinese companies are open weight, and depending on this trajectory, the business models that these American companies are doing could be at risk.
这些中国公司的这些模式是开放的，根据这种轨迹，这些美国公司正在做的商业模式可能面临风险。
But currently, a lot of people are paying for AI software in the US, and historically in China and other parts of the world, people don’t pay a lot for software.
但目前，在美国，很多人都在为人工智能软件付费，而从历史上看，在中国和世界其他地区，人们并没有为软件支付太多费用。
So some of these models like DeepSeek have the love of the people because they are open weight.
因此，像 DeepSeek 这样的一些模型因其开放权重而受到人们的喜爱。
06:37 How long do you think the Chinese companies keep releasing open weight models?
您认为中国公司持续发布开放重量模型多久了？
I would say for a few years.
我想说几年。
06:47 I think that, like in the US, there’s not a clear business model for it.
我认为，就像在美国一样，没有明确的商业模式。
I have been writing about open models for a while, and these Chinese companies have realized it.
我写开放模式已经有一段时间了，这些中国公司已经意识到了这一点。
I get inbound from some of them.
我从其中一些人那里入境。
They’re smart and realize the same constraints, which is that a lot of top US tech companies and other IT companies won’t pay for an API subscription to Chinese companies for security concerns.
他们很聪明，也意识到了同样的限制，即出于安全考虑，许多美国顶级科技公司和其他 IT 公司不会向中国公司支付 API 订阅费用。
This has been a long-standing habit in tech, and the people at these companies then see open weight models as an ability to influence and take part in a huge growing AI expenditure market in the US.
这是科技领域长期存在的习惯，这些公司的人员将开放权重模型视为影响和参与美国不断增长的巨大人工智能支出市场的能力。
They’re very realistic about this, and it’s working for them.
他们对此非常现实，而且这对他们很有用。
07:24 And I think the government will see that that is building a lot of influence internationally in terms of uptake of the technology, so there’s going to be a lot of incentives to keep it going.
我认为政府会看到这在技术的采用方面正在国际上产生很大的影响力，因此将会有很多激励措施来保持它的发展。
But building these models and doing the research is very expensive, so at some point, I expect consolidation.
但构建这些模型并进行研究非常昂贵，因此在某些时候，我预计会进行整合。
But I don’t expect that to be a story of 2026; there will be more open model builders throughout 2026 than there were in 2025.
但我不认为这会成为 2026 年的故事。到 2026 年，开放模型构建器将会比 2025 年更多。
And a lot of the notable ones will be in China.
许多著名的将在中国。
You were going to say something?
你本来想说点什么吗？
Yes.
是的。
07:51 You mentioned DeepSeek losing its crown.
您提到 DeepSeek 失去了王冠。
I do think to some extent, yes, but we also have to consider that they are still slightly ahead.
我确实认为在某种程度上是的，但我们也必须考虑到他们仍然稍微领先。
It’s not that DeepSeek got worse, it’s just like the other ones are using the ideas from DeepSeek.
并不是 DeepSeek 变得更糟，只是其他人都在使用 DeepSeek 的想法。
For example, you mentioned Kimi, same architecture, they’re training it.
例如，你提到 Kimi，同样的架构，他们正在训练它。
And then again, we have this leapfrogging where they might be at some point in time a bit better because they have the more recent model.
再说一次，我们有这种跨越式的发展，他们可能在某个时间点做得更好一些，因为他们有更新的型号。
I think this comes back to the fact that there won’t be a clear winner.
我认为这又回到了一个事实：不会有明显的赢家。
One person releases something, the other one comes in, and the most recent model is probably always the best model.
一个人发布了一些东西，另一个人进来了，最新的模型可能总是最好的模型。
Yeah.
是的。
08:30 We’ll also see the Chinese companies have different incentives.
我们还将看到中国公司有不同的激励措施。
DeepSeek is very secretive, whereas some of these startups are like the MiniMaxes and Z.ais of the world.
DeepSeek 非常神秘，而其中一些初创公司就像世界上的 MiniMaxes 和 Z.ais 一样。
Those two literally have filed IPO paperwork, and they’re trying to get Western mindshare and do a lot of outreach there.
这两家公司实际上已经提交了 IPO 文件，他们正试图获得西方的关注并在那里进行大量的宣传。
So I don’t know if these incentives will change the model development, because DeepSeek famously is built by a hedge fund, Highflyer Capital, and we don’t know exactly what they use the models for or if they care about this.
所以我不知道这些激励措施是否会改变模型的开发，因为众所周知，DeepSeek 是由对冲基金 Highflyer Capital 建立的，我们不知道他们到底用这些模型做什么，也不知道他们是否关心这个。
They’re secretive in terms of communication, but they’re not secretive in terms of the technical reports that describe how their models work.
他们在沟通方面是保密的，但在描述其模型如何工作的技术报告方面却并不保密。
08:59 They’re still open on that front.
他们在这方面仍然开放。
And we should also say on the Claude Opus 4.5 hype, there’s the layer of something being the darling of the X echo chamber, the Twitter echo chamber, and the actual amount of people that are using the model.
我们还应该说，关于 Claude Opus 4.5 的炒作，有一层是 X 回声室、Twitter 回声室的宠儿，以及使用该模型的实际人数。
I think it’s probably fair to say that ChatGPT and Gemini are focused on the broad user base that just wants to solve problems in their daily lives, and that user base is gigantic.
我认为可以公平地说，ChatGPT 和 Gemini 专注于只想解决日常生活问题的广大用户群，而且用户群是巨大的。
So the hype about the coding may not be representative of the actual use.
因此，有关编码的炒作可能并不代表实际使用情况。
I would say also a lot of the usage patterns are name recognition and brand, but also almost muscle memory, where ChatGPT has been around for a long time.
我想说，很多使用模式都是名称识别和品牌，但也几乎是肌肉记忆，ChatGPT 已经存在很长时间了。
09:38 People just got used to using it, and it’s almost like a flywheel where they recommend it to other users.
人们刚刚习惯了使用它，它几乎就像一个飞轮，他们将其推荐给其他用户。
One interesting point is also the customization of LLMs.
一个有趣的一点是法学硕士的定制。
For example, ChatGPT has a memory feature.
例如，ChatGPT 具有记忆功能。
So you may have a subscription and you use it for personal stuff, but I don’t know if you want to use that same thing at work because there is a boundary between private and work.
所以你可能有订阅，并且将其用于个人事务，但我不知道你是否想在工作中使用同样的东西，因为私人和工作之间存在界限。
If you’re working at a company, they might not allow that or you may not want that.
如果您在公司工作，他们可能不允许这样做，或者您可能不希望这样做。
10:16 And I think that’s also an interesting point where you might have multiple subscriptions.
我认为这也是一个有趣的点，你可能有多个订阅。
One is just clean code; it has nothing of your personal images or hobby projects in there.
一是干净的代码；二是干净的代码。它里面没有任何您的个人图像或爱好项目。
It’s just for work.
这只是为了工作。
And then the other one is your personal thing.
然后另一件事是你个人的事情。
I think the future involves multiple models for different use cases.
我认为未来涉及不同用例的多种模型。
It doesn’t mean you only have to have one.
这并不意味着您只需拥有一个。
What model do you think won 2025, and what model do you think is going to win ’26?
您认为哪种型号会赢得 2025 年的胜利，哪种型号会赢得 26 年的胜利？
I think in the context of consumer chatbots, the question is: are you willing to bet on Gemini over ChatGPT?
我认为在消费者聊天机器人的背景下，问题是：你愿意把赌注押在 Gemini 上而不是 ChatGPT 上吗？
10:43 Which I would say in my gut feels like a bit of a risky bet because OpenAI has been the incumbent and there are so many benefits to that in tech.
我想说，我的直觉感觉这是一个有点冒险的赌注，因为 OpenAI 一直是老牌企业，而且在技术领域有很多好处。
I think the momentum in 2025 was on Gemini’s side, but they were starting from such a low point.
我认为2025年的势头是在双子座这边，但他们是从如此低的起点开始的。
RIP Bard and those earlier attempts.
安息吧巴德和那些早期的尝试。
I think huge credit to them for powering through the organizational chaos to make that happen.
我认为他们在组织混乱中克服困难、实现这一目标的巨大功劳。
But also it’s hard to bet against OpenAI because they always come off as so chaotic, but they’re very good at landing things.
但也很难对 OpenAI 下赌注，因为它们总是显得如此混乱，但它们非常擅长落地。
11:26 Personally, I have very mixed reviews of GPT-5, but it must have saved them so much money with the high-line feature being a router where most users are no longer charging their GPU costs as much.
就我个人而言，我对 GPT-5 的评价褒贬不一，但它一定为他们节省了很多钱，因为它的高线功能是路由器，大多数用户不再收取那么多的 GPU 成本。
So I think it’s very hard to dissociate the things that I like out of models versus the things that are actually going to be a general public differentiator.
所以我认为很难将我喜欢的模型与那些实际上将成为公众差异化因素的东西区分开来。
What do you think about 2026?
你对2026年有何看法？
11:50 Who’s going to win?
谁会赢？
I’ll say something, even though it’s risky.
我会说一些话，尽管这有风险。
11:52 I think Gemini will continue to make progress on ChatGPT.
我认为Gemini会在ChatGPT上继续取得进展。
Google has the scale when both of these are operating at such extreme scales, and Google has the ability to separate research and product a bit better, whereas you hear so much about OpenAI being chaotic operationally and chasing the high-impact thing, which is a very startup culture.
当这两者都以如此极端的规模运营时，谷歌就有了规模，而且谷歌有能力更好地将研究和产品分开，而你听到很多关于 OpenAI 运营混乱并追求高影响力的事情，这是一种非常初创的文化。
Then on the software and enterprise side, I think Anthropic will have continued success as they’ve again and again been set up for that.
然后在软件和企业方面，我认为 Anthropic 将继续取得成功，因为他们一次又一次地为此做好了准备。
Obviously Google Cloud has a lot of offerings, but I think this Gemini name brand is important for them to build.
显然，谷歌云有很多产品，但我认为这个 Gemini 品牌对他们来说很重要。
12:28 Google Cloud will continue to do well, but that’s a more complex thing to explain in the ecosystem because that’s competing with the likes of Azure and AWS rather than on the model provider side.
谷歌云将继续表现出色，但这在生态系统中解释起来是一件更复杂的事情，因为它是与 Azure 和 AWS 等公司竞争，而不是在模型提供商方面竞争。
So in infrastructure, you think TPUs give them an advantage?
那么在基础设施方面，您认为 TPU 给他们带来了优势吗？
Largely because the margin on NVIDIA chips is insane and Google can develop everything from top to bottom to fit their stack and not have to pay this margin, and they’ve had a head start in building data centers.
很大程度上是因为 NVIDIA 芯片的利润非常高，而谷歌可以从上到下开发所有东西来适应他们的堆栈，而不必支付这个利润，而且他们在建设数据中心方面已经处于领先地位。
12:45 So all of these things that have both high lead times and very hard margins on high costs, Google has a kind of historical advantage there.
因此，所有这些产品的交付时间都很高，而且成本很高，利润率也很高，谷歌在这方面拥有一种历史优势。
And if there’s going to be a new paradigm, it’s most likely to come from OpenAI.
如果要出现一种新范式，它最有可能来自 OpenAI。
Their research division again and again has shown this ability to land a new research idea or a product.
他们的研究部门一次又一次地展示了这种提出新研究想法或产品的能力。
Like Deep Research, Sora, o1 thinking models—all these definitional things have come from OpenAI, and that’s got to be one of their top traits as an organization.
像 Deep Research、Sora、o1 思维模型一样，所有这些定义性的东西都来自 OpenAI，这一定是他们作为一个组织的首要特征之一。
13:28 So it’s kind of hard to bet against that, but I think a lot of this year will be about scale and optimizing what could be described as low-hanging fruit in models.
所以很难反对这一点，但我认为今年的大部分内容将是关于规模和优化模型中那些容易实现的成果。
And clearly there’s a trade-off between intelligence and speed.
显然，智能和速度之间需要权衡。
13:37 This is what GPT-5 was trying to solve behind the scenes.
这就是 GPT-5 试图在幕后解决的问题。
It’s like, do people actually want intelligence, the broad public, or do they want speed?
就像，人们真的想要情报、广大公众，还是想要速度？
I think it’s a nice variety actually, or the option to have a toggle there.
我认为实际上这是一个很好的品种，或者可以在那里进行切换的选项。
13:52 For my personal usage, most of the time when I look something up, I use ChatGPT to ask a quick question and get the information I wanted fast.
就我个人的使用而言，大多数时候当我查找内容时，我会使用 ChatGPT 来快速提问并快速获取我想要的信息。
For most daily tasks, I use the quick model.
对于大多数日常任务，我使用快速模型。
Nowadays, I think the auto mode is pretty good where you don’t have to specifically say “thinking” or “non-thinking.” Then again, I also sometimes want the pro mode.
现在我觉得自动模式挺好的，不用特意说“想”或者“不想”。话又说回来，我有时也想要专业模式。
Very often, when I have something written, I put it into ChatGPT and say, “Hey, do a very thorough check.
很多时候，当我写了一些东西时，我会将其放入 ChatGPT 中并说：“嘿，进行一次非常彻底的检查。
Are all my references correct?
我所有的参考资料都正确吗？
Are all my thoughts correct?
我所有的想法都是正确的吗？
Did I make any formatting mistakes?
我犯了任何格式错误吗？
Are the figure numbers wrong?” or something like that.
数字有错吗？”或类似的东西。
And I don’t need that right away.
我不需要立即这样做。
14:33 I can finish my stuff, maybe have dinner, let it run, come back and go through it.
我可以完成我的事情，也许吃晚饭，让它运行，然后回来完成它。
This is where I think it’s important to have this option.
这就是我认为有这个选择很重要的地方。
I would go crazy if for each query I had to wait 30 minutes, or even 10 minutes.
如果每个查询都需要等待 30 分钟，甚至 10 分钟，我会发疯的。
That’s me.
这就是我。
14:46 I’m sitting over here losing my mind that you use the router and the non-thinking model.
我坐在这里失去了理智，你使用了路由器和非思考模型。
I’m like, “How do you live with that?”
我想，“你怎么忍受这个？”
14:55 That’s like my reaction.
这就像我的反应。
I’ve been heavily on ChatGPT for a while.
我已经大量使用 ChatGPT 一段时间了。
I never touched GPT-5 non-thinking.
我从来没有接触过GPT-5非思考。
I find it just…
我发现它只是……
its tone and then its propensity for errors.
它的语气，然后是它的错误倾向。
It just has a higher likelihood of errors.
它只是出错的可能性更高。
Some of this is from back when OpenAI released o3, which was the first model to do this Deep Research and find many sources and integrate them for you.
其中一些来自 OpenAI 发布 o3 时，它是第一个进行深度研究并找到许多来源并为您集成它们的模型。
So I became habituated with that.
所以我就习惯了。
I will only use GPT-5.2 thinking or pro when I’m finding any sort of information query for work, whether that’s a paper or some code reference.
当我寻找任何类型的工作信息查询时，无论是论文还是代码参考，我只会使用 GPT-5.2 Thinking 或 Pro。
I will regularly have five pro queries going simultaneously, each looking for one specific paper or feedback on an equation.
我会定期同时进行五个专业查询，每个查询都寻找一篇特定的论文或有关方程式的反馈。
I have a fun example where I just needed the answer as fast as possible for this podcast before I was going on the trip.
我有一个有趣的例子，在我出发之前，我只需要尽快得到这个播客的答案。
15:38 I have a local GPU running at home and I wanted to run a long RL experiment.
我家里有一个本地 GPU，我想运行一个长时间的 RL 实验。
Usually I unplug things because if you’re not at home, you don’t want to have things plugged in, and I accidentally unplugged the GPU.
通常我会拔掉插头，因为如果你不在家，你就不想插上插头，而我不小心拔掉了 GPU 的插头。
My wife was already in the car and it was like, “Oh dang.” Basically, I wanted a Bash script as fast as possible that runs my different experiments and the evaluation.
我的妻子已经上了车，当时我就想，“天啊。”基本上，我想要一个尽可能快的 Bash 脚本来运行我的不同实验和评估。
I know how to use the Bash terminal, but in that moment I just needed the command in 10 seconds.
我知道如何使用 Bash 终端，但在那一刻我只需要 10 秒内的命令。
This is a hilarious situation but yeah, so what did you use?
这是一个搞笑的情况，但是，是的，那么你用了什么？
So I did the non-thinking fastest model.
所以我做了非思考最快的模型。
16:21 It gave me the Bash command.
它给了我 Bash 命令。
I wanted to chain different scripts to each other and route this to a log file with the `tee` command.
我想将不同的脚本相互链接，并使用“tee”命令将其路由到日志文件。
Off the top of my head, I was just in a hurry; I could have thought about it myself.
我突然意识到，我只是很着急而已。我自己也可以考虑一下。
By the way, I don’t know if there’s a representative case: wife waiting in the car, you have to run, unplug the GPU, you have to generate a Bash script.
对了，不知道有没有一个代表性的案例：老婆在车里等，你要跑，拔掉GPU，你要生成Bash脚本。
16:37 This sounds like a movie…
这听起来像电影……
…Mission Impossible.
……不可能的任务。
I use Gemini for that.
我为此使用双子座。
16:46 I use thinking for all the information stuff and then Gemini for fast things or stuff that I could sometimes Google.
我用思维来思考所有的信息，然后用双子座来思考快速的事物或我有时可以谷歌搜索的事物。
It’s good at explaining things and I trust that it has this background of knowledge and it’s simple.
它善于解释事物，我相信它有这个知识背景，而且很简单。
And the Gemini app has gotten a lot better.
Gemini 应用程序已经变得更好了。
17:01 It’s good for those sorts of things.
这对这类事情很有好处。
And then for code and any sort of philosophical discussion, I use Claude Opus 4.5, also always with extended thinking.
然后，对于代码和任何形式的哲学讨论，我使用 Claude Opus 4.5，也始终具有扩展思维。
Extended thinking and inference-time scaling is just a way to make the models marginally smarter.
扩展思维和推理时间缩放只是使模型稍微智能的一种方法。
I will always edge on that side when the progress is very high because you don’t know when that’ll unlock a new use case.
当进度非常高时，我总是会站在这一边，因为你不知道什么时候会解锁新的用例。
And then I sometimes use Grok for real-time information or finding something on AI Twitter that I knew I saw and I need to dig up.
然后，我有时会使用 Grok 获取实时信息，或者在 AI Twitter 上查找我知道我看到并且需要挖掘的内容。
Although when Grok 4 came out, the Grok 4 Heavy—which was their pro variant—was actually very good and I was pretty impressed with it, and then I just kind of lost track of it with muscle memory from having the ChatGPT app open.
尽管当 Grok 4 推出时，Grok 4 Heavy（他们的专业版本）实际上非常好，并且给我留下了深刻的印象，然后我只是因为打开 ChatGPT 应用程序而失去了肌肉记忆。
So I use many different things.
所以我使用很多不同的东西。
Yeah.
是的。
17:45 I actually do use Grok 4 Heavy for debugging.
实际上我确实使用 Grok 4 Heavy 进行调试。
For hardcore debugging that the other ones can’t solve, I find that it’s the best at.
对于其他解决不了的硬核调试，我发现它是最擅长的。
And it’s interesting because you say ChatGPT is the best interface.
这很有趣，因为你说 ChatGPT 是最好的界面。
For me, for that same reason—but this could be just momentum— Gemini is the better interface for me.
对我来说，出于同样的原因（但这可能只是动力），Gemini 对我来说是更好的界面。
I think because I fell in love with their needle-in-the-haystack capabilities.
我想是因为我爱上了他们大海捞针的能力。
If I ever put in something that has a lot of context but I’m looking for very specific information to make sure it tracks all of it, I find Gemini has been the best.
如果我放入了一些有很多上下文的东西，但我正在寻找非常具体的信息以确保它跟踪所有内容，我发现 Gemini 是最好的。
So it’s funny with some of these models, if they win your heart over—
所以，如果其中一些模型赢得了你的心，那就很有趣了——
18:28 …for one particular feature on a particular day, for that particular query or prompt, you’re like, “This model’s better.” And so you’ll just stick with it for a bit until it does something really dumb.
...对于特定日期的一项特定功能，对于特定查询或提示，您会说，“这个模型更好。”所以你会坚持一段时间，直到它做出一些非常愚蠢的事情。
There’s like a threshold effect.
这就像阈值效应。
Some smart thing happens and then you fall in love with it, and then it does some dumb thing and you’re like, “You know what?
一些聪明的事情发生了，然后你爱上了它，然后它做了一些愚蠢的事情，你就像，“你知道吗？
I’m gonna switch and try Claude or ChatGPT.” And all that kind of stuff.
我要换用 Claude 或 ChatGPT。”诸如此类的事情。
This is exactly it.
就是这样。
18:51 You use it until it breaks, until you have a problem, and then you change the LLM.
你使用它直到它坏掉，直到你遇到问题，然后你改变LLM。
I think it’s the same way we use anything, like our favorite text editor, operating system, or browser.
我认为这就像我们使用任何东西的方式一样，比如我们最喜欢的文本编辑器、操作系统或浏览器。
I mean, there are so many browser options: Safari, Firefox, Chrome.
我的意思是，有很多浏览器选项：Safari、Firefox、Chrome。
They’re relatively similar, but then there are edge cases, maybe extensions you want to use, and then you switch.
它们相对相似，但也有一些边缘情况，也许你想使用扩展，然后你就切换了。
But I don’t think anyone types the same thing into different browsers and compares them.
但我认为没有人会在不同的浏览器中输入相同的内容并进行比较。
You only do that when the website doesn’t render or if something breaks.
仅当网站无法呈现或出现问题时才执行此操作。
So that’s a good point.
所以这是一个很好的观点。
You use it until it breaks, and then you explore other options.
你使用它直到它坏掉，然后你探索其他选择。
On the long context thing, I was also a Gemini user for this, but the GPT-5.2 release blog had crazy long context scores, where a lot of people were like, “Did they just figure out some algorithmic change?” It went from like 30% to like 70% or something in this minor model update.
就长上下文而言，我也是 Gemini 用户，但 GPT-5.2 发布博客有疯狂的长上下文分数，很多人都在想，“他们只是想出了一些算法更改吗？”在这个小模型更新中，它从 30% 左右上升到 70% 左右。
19:28 So it’s also very hard to keep track of all of these things, but now I look more favorably at GPT-5.2’s long context.
因此，跟踪所有这些事情也非常困难，但现在我更看好 GPT-5.2 的长上下文。
So it’s just kind of like a never-ending battle to actually get to testing this.
因此，实际测试这一点就像一场永无休止的战斗。
Well, it’s interesting that none of us talked about the Chinese models from a user perspective.
嗯，有趣的是，我们没有人从用户的角度谈论中国模式。
19:57 What does that say?
那说什么？
Does that mean the Chinese models are not as good, or does that mean we’re just very biased and US-focused?
这是否意味着中国模式没有那么好，或者这是否意味着我们只是非常有偏见并且以美国为中心？
I do think that’s currently the discrepancy between the model and the platform.
我确实认为这就是目前模型和平台之间的差异。
20:11 I think the open models are more known for the open weights, not their platform yet.
我认为开放模型因开放权重而闻名，而不是其平台。
There are also a lot of companies that are willing to sell you open-model inference at a very low cost.
还有很多公司愿意以非常低的成本向你出售开放模型推理。
20:21 I think, like OpenRouter, it’s easy to look at multi-model things.
我认为，就像 OpenRouter 一样，查看多模型事物很容易。
You can run DeepSeek on Perplexity.
您可以在 Perplexity 上运行 DeepSeek。
I think all of us sitting here are like, “We use OpenAI GPT-5 Pro consistently.” We’re all willing to pay for the marginal—
我认为坐在这里的所有人都会说，“我们始终如一地使用 OpenAI GPT-5 Pro。”我们都愿意为边际付出代价——
20:39 …intelligence gain.
……情报增益。
And these models from the US are better in terms of the outputs.
美国的这些模型在输出方面更好。
I think the question is, will they stay better for this year and for years going forward?
我认为问题是，他们今年和未来几年会保持更好的状态吗？
But so long as they’re better, I’m going to pay for them.
但只要他们更好，我就会付钱给他们。
I think there’s also analysis that shows that the way the Chinese models are served—which you could argue is due to export controls or not—is that they use fewer GPUs per replica, which makes them slower and leads to different errors.
我认为还有分析表明，中国模型的服务方式（你可能会说是否是由于出口管制）是它们每个副本使用的 GPU 较少，这使得它们速度较慢并导致不同的错误。
It’s about speed and intelligence.
这是关于速度和智力的。
21:09 If these things are in your favor as a user, I think in the US a lot of users will go for this.
如果这些东西对你作为用户有利，我认为在美国很多用户都会这么做。
I think that is one thing that will spur these Chinese companies to want to compete in other ways, whether it’s free or substantially lower costs, or it’ll breed creativity in terms of offerings, which is good for the ecosystem.
我认为这将刺激这些中国公司以其他方式竞争，无论是免费还是大幅降低成本，或者它将在产品方面培育创造力，这对生态系统有利。
But I just think the simple thing is the US models are currently better, and we use them.
但我认为简单的事情是美国的模式目前更好，我们使用它们。
I tried these other open models, and I’m like, “Fun, but I’m not gonna…
我尝试了其他开放模型，我想，“有趣，但我不会……
I don’t go back to it.”
我不会再回去了。”
We didn’t really mention programming.
我们并没有真正提到编程。
21:38 That’s another use case that a lot of people deeply care about.
这是很多人非常关心的另一个用例。
I use basically half-and-half Cursor and Claude Code, because I find them to be fundamentally different experiences and both useful.
我基本上使用游标和克劳德代码各占一半，因为我发现它们是完全不同的体验并且都很有用。
You program quite a bit— …so what do you use?
您经常编程——……那么您使用什么？
What’s the current vibe?
目前的氛围如何？
So, I use the Codeium plugin for VS Code.
因此，我使用 VS Code 的 Codeium 插件。
21:59 You know, it’s very convenient.
你知道，这非常方便。
It’s just a plugin, and then it’s a chat interface that has access to your repository.
它只是一个插件，然后它是一个可以访问您的存储库的聊天界面。
I know that Claude Code is a bit different.
我知道克劳德·代码有点不同。
It is a bit more agentic.
它更具有代理性。
It touches more things; it does the whole project for you.
它触及的东西更多；它会为您完成整个项目。
I’m not quite there yet where I’m comfortable with that because maybe I’m a control freak, but I still like to see what’s going on.
我还没有完全适应这一点，因为也许我是一个控制狂，但我仍然想看看发生了什么。
Codeium is the sweet spot for me right now where it is helping me, but it is not taking over completely.
Codeium 现在对我来说是最有利的地方，它对我有帮助，但它并没有完全取代我。
I should mention, one of the reasons I do use Claude Code is to build the skill of programming with English.
我应该提到，我使用 Claude Code 的原因之一是培养英语编程技能。
22:29 I mean, the experience is fundamentally different.
我的意思是，体验是根本不同的。
As opposed to micromanaging the details of the generation and looking at the diff—which you can in Cursor if that’s the IDE you use—you are understanding the code deeply as you progress, versus just thinking in this design space and guiding it at a macro level.
与对生成的细节进行微观管理并查看差异（如果您使用的是 Cursor，则可以在 Cursor 中查看差异）相反，您是在不断进步的过程中深入理解代码，而不是仅仅在这个设计空间中思考并在宏观层面上进行指导。
I think that is another way of thinking about the programming process.
我认为这是思考编程过程的另一种方式。
Also, Claude Code just seems to be a better utilization of Claude Opus 4.5.
此外，Claude Code 似乎是对 Claude Opus 4.5 的更好利用。
It’s a good side-by-side for people to do.
这对人们来说是一件好事。
23:18 You can have Claude Code open, you can have Cursor open, you can have VS Code open, and you can select the same models on all of them— …and ask questions, and it’s very interesting.
你可以打开 Claude Code，你可以打开 Cursor，你可以打开 VS Code，你可以在所有这些上选择相同的模型 - ......并提出问题，这非常有趣。
Claude Code is way better in that domain.
克劳德·代码在这个领域做得更好。
It’s remarkable.
这很了不起。
All right, we should say that both of you are legit on multiple fronts: researchers, programmers, educators, and on the book front, too.
好吧，我们应该说你们俩在多个方面都是合法的：研究人员、程序员、教育工作者，在书籍方面也是如此。
23:32 Nathan, at some point soon, hopefully has an RLHF book coming out.
内森（Nathan）希望很快就能出版一本 RLHF 书。
It’s available for preorder, and there’s a full digital preprint.
它可以预订，并且有完整的数字预印本。
23:50 I’m just making it pretty and better organized for the physical thing, which is a lot of why I do it—it’s fun to create things that you think are excellent in physical form when so much of our life is digital.
我只是为了让物理事物变得更漂亮、更好地组织起来，这就是我这样做的很多原因——当我们的生活大部分都是数字化的时候，创造你认为在物理形式上非常出色的东西是很有趣的。
I should say, going to Perplexity here, Sebastian Raschka is a machine learning researcher and author known for several influential books.
我应该说，说到 Perplexity，Sebastian Raschka 是一位机器学习研究员和作家，以多本有影响力的书籍而闻名。
24:05 A couple that I wanted to mention—and a book I highly recommend—is Build a Large Language Model From Scratch, and the new one, Build a Reasoning Model From Scratch.
我想提的两本书（也是我强烈推荐的一本书）是《从头开始构建大型语言模型》，以及新书《从头开始构建推理模型》。
I’m really excited about that.
我对此感到非常兴奋。
Building stuff from scratch is one of the most powerful ways of learning.
从头开始构建东西是最有效的学习方式之一。
Honestly, building an LLM from scratch is a lot of fun and a lot to learn.
老实说，从头开始攻读法学硕士非常有趣，而且需要学习很多东西。
24:27 Like you said, it’s probably the best way to learn how something really works, because you can look at figures, but figures can have mistakes.
就像你说的，这可能是了解某些东西如何真正运作的最好方法，因为你可以看数字，但数字可能有错误。
You can look at conceptual explanations, but you might misunderstand them.
你可以看看概念性的解释，但你可能会误解它们。
But if there is code and the code works, you know it’s correct.
但如果有代码并且代码有效，你就知道它是正确的。
There’s no misunderstanding; it’s precise.
没有误会；这是精确的。
Otherwise, it wouldn’t work.
否则，就行不通。
I think that’s the beauty behind coding.
我认为这就是编码背后的美妙之处。
It doesn’t lie.
它没有说谎。
It’s math, basically.
基本上，这是数学。
Even with math, you can have mistakes in a book you would never notice because you aren’t running the math while reading, so you can’t verify it.
即使是数学，你也可能在一本你永远不会注意到的书中出现错误，因为你在阅读时没有运行数学，所以你无法验证它。
And with code, what’s nice is you can verify it.
有了代码，好处是你可以验证它。
Yeah, I agree with you about the Build a Large Language Model From Scratch book.
是的，我同意你关于《从头开始构建大型语言模型》一书的看法。
25:09 It’s nice to tune out everything else, the internet and so on, and just focus on the book.
抛开其他一切、互联网等，只专注于书本是件好事。
But, you know, compared to history books, it’s just less lonely somehow.
但是，你知道，与历史书相比，它只是不那么孤独。
It’s really more fun.
真的更好玩。
For example, on the programming front, I think it’s genuinely more fun to program with an LLM.
例如，在编程方面，我认为与法学硕士一起编程确实更有趣。
And I think it’s genuinely more fun to read with an LLM.
我认为与法学硕士一起阅读确实更有趣。
But you’re right.
但你是对的。
This distraction should be minimized.
应尽量减少这种干扰。
So you use the LLM to basically enrich the experience, maybe add more context.
因此，您使用法学硕士基本上可以丰富经验，也许可以添加更多背景。
Maybe I just…
也许我只是……
the rate of ‘aha’ moments for me on a small scale is really high with LLMs.
对于我来说，法学硕士的“顿悟”时刻在小范围内的发生率非常高。
100%.
100%。
25:54 I also want to correct myself: I’m not suggesting not to use LLMs.
我还想纠正自己：我并不是建议不要使用法学硕士。
I suggest doing it in multiple passes.
我建议分多次进行。
Like, one pass just offline, focus mode, and then after that…
就像，一次通过离线模式，专注模式，然后……
I mean, I also take notes, but I try to resist the urge to immediately look things up.
我的意思是，我也做笔记，但我努力克制住立即查资料的冲动。
I do a second pass.
我进行第二遍。
For me, it’s just more structured this way and I get less…
对我来说，这样更有条理，我得到的更少……
I mean, sometimes things are answered in the chapter, but also it just helps to let it sink in and think about it.
我的意思是，有时事情会在本章中得到解答，但它也有助于让人们深入思考并思考它。
Other people have different preferences.
其他人有不同的偏好。
I would highly recommend using LLMs when reading books.
我强烈建议在读书时使用法学硕士。
For me, it’s just not the first thing to do; it’s the second pass.
对我来说，这不是第一件事；而是我要做的第一件事。这是第二关。
By way of recommendation, I do the opposite.
作为推荐，我做了相反的事情。
26:29 I like to use the LLM at the beginning— …to lay out the full context of what is this world that I’m now stepping into.
我喜欢在一开始就使用法学硕士——……来展示我现在正在步入的这个世界的完整背景。
But I try to avoid clicking out of the LLM into the world of Twitter and blogs because then you’re down this rabbit hole.
但我尽量避免离开法学硕士进入推特和博客的世界，因为那样你就会陷入这个兔子洞。
You’re reading somebody’s opinion, there’s a flame war about a particular topic, and all of a sudden you’re now in the realm of the internet and Reddit and so on.
你正在阅读某人的观点，关于某个特定主题存在激烈的争论，突然之间你就进入了互联网和 Reddit 等领域。
But if you’re purely letting the LLM give you the context of why this matters, what are the big picture ideas…
但如果你纯粹是让法学硕士告诉你为什么这很重要，那么大局想法是什么……
sometimes books themselves are good at doing that, but not always.
有时书籍本身就擅长做到这一点，但并非总是如此。
This is why I like the ChatGPT app, because it gives the AI a home in your computer where you can focus on it, rather than just being another tab in my mess of internet options.
这就是我喜欢 ChatGPT 应用程序的原因，因为它为人工智能在您的计算机中提供了一个家，您可以在那里专注于它，而不仅仅是我混乱的互联网选项中的另一个选项卡。
27:12 And I think Claude Code in particular does a good job of making that a joy, where it seems very engaging as a product design to be an interface that your AI will then go out into the world.
我认为克劳德·科德（Claude Code）在让这成为一种乐趣方面做得尤其出色，作为一种产品设计，它似乎非常吸引人，成为你的人工智能将走向世界的界面。
There’s something very intangible between it and Codex; it just feels warm and engaging, whereas Codex from OpenAI can often be as good but it just feels a little bit rough around the edges.
它和 Codex 之间有一些非常无形的东西；它给人的感觉很温暖、很吸引人，而 OpenAI 的 Codex 通常也很好，但感觉边缘有点粗糙。
27:45 Whereas Claude Code makes it fun to build things, particularly from scratch where you trust that it’ll make something.
而 Claude Code 让构建事物变得有趣，尤其是从头开始，你相信它会创造出一些东西。
Obviously this is good for websites and refreshing tooling, which I use it for, or data analysis.
显然，这对于网站和刷新工具（我用它来进行数据分析）很有好处。
On my blog, we scrape Hugging Face so we keep the download numbers for every dataset and model over time now.
在我的博客上，我们抓取了 Hugging Face，因此我们现在可以保留每个数据集和模型的下载数量。
Claude was just like, “Yeah, I’ve made use of that data, no problem.” And I was like, “That would’ve taken me days.” And then I have enough situational awareness to be like, “Okay, these trends obviously make sense,” and you can check things.
克劳德说：“是的，我已经使用了这些数据，没问题。”我当时想，“那会花我好几天的时间。”然后我有足够的态势感知能力，“好吧，这些趋势显然是有道理的”，你可以检查一下。
But that’s just a wonderful interface where you can have an intermediary and not have to do the awful low-level work that you would have to do to maintain different web projects.
但这只是一个很棒的界面，您可以在其中拥有一个中介，而不必做维护不同 Web 项目所需的可怕的低级工作。
All right.
好的。
28:29 So we just talked about a bunch of the closed-weight models.
所以我们刚刚讨论了一些封闭重量模型。
Let’s talk about the open ones.
我们来谈谈开放的。
Tell me about the landscape of open LLM models.
请告诉我开放式法学硕士模式的概况。
Which are interesting ones?
哪些是有趣的？
Which stand out to you and why?
哪一个对您来说很突出，为什么？
We already mentioned DeepSeek.
我们已经提到过 DeepSeek。
Do you wanna see how many we can name off the top of our head?
你想看看我们能一下子说出多少个名字吗？
Yeah, yeah.
是啊是啊。
28:47 Without looking at notes.
不看笔记。
DeepSeek, Kimi, MiniMax, Z.ai, Antlang.
DeepSeek、Kimi、MiniMax、Z.ai、Antlang。
28:48 We’re just going Chinese.
我们只是去中国化。
Let’s throw in Mistral AI, Gemma— …gpt-oss, the open source model by OpenAI.
让我们引入 Mistral AI、Gemma——……gpt-oss，OpenAI 的开源模型。
28:57 Actually, NVIDIA had a really cool one, Nemotron 3.
事实上，NVIDIA 有一款非常酷的产品，Nemotron 3。
There’s a lot of stuff, especially at the end of the year.
尤其是年底，事情很多。
Qwen might be the one—
Qwen 可能就是那个——
Oh, yeah.
哦，是的。
29:12 Qwen was the obvious name I was gonna say.
Qwen 是我要说的显而易见的名字。
I was trying to get through…
我试图渡过难关……
you can get at least 10 Chinese and at least 10 Western.
你可以得到至少10个中餐和至少10个西餐。
I mean, OpenAI released their first open model—
我的意思是，OpenAI 发布了他们的第一个开放模型——
A long time ago.
很久以前了。
…since GPT-2.
…自 GPT-2 以来。
29:22 When I was writing about OpenAI’s open model release, people were like, “Don’t forget about GPT-2,” which I thought was really funny because it’s just such a different time.
当我撰写有关 OpenAI 开放模型发布的文章时，人们会说，“不要忘记 GPT-2”，我认为这真的很有趣，因为这是一个如此不同的时代。
But gpt-oss is actually a very strong model and does some things that the other models don’t do very well.
但 gpt-oss 实际上是一个非常强大的模型，并且做了一些其他模型做得不太好的事情。
Selfishly, I’ll promote a bunch of Western companies; both in the US and Europe have these fully open models.
自私地说，我会推广一批西方公司；美国和欧洲都有这些完全开放的模式。
I work at the Allen Institute for AI where we’ve been building OLMo, which releases data and code and all of this.
我在艾伦人工智能研究所工作，我们在那里构建了 OLMo，它发布数据和代码以及所有这些。
And now we have actual competition for people that are trying to release everything so that other people can train these models.
现在，我们对那些试图发布所有内容以便其他人可以训练这些模型的人进行了实际的竞争。
29:57 So there’s the Institute for Foundation Models/LM360, which has had their K2 models of various types.
基础模型研究所/LM360 拥有各种类型的 K2 模型。
Apertus is a Swiss research consortium.
Apertus 是一家瑞士研究联盟。
Hugging Face has SmolLM, which is very popular.
Hugging Face 有 SmolLM，非常受欢迎。
And NVIDIA’s Nemotron has started releasing data as well.
NVIDIA 的 Nemotron 也开始发布数据。
And then Stanford’s Marine Community Project, which is kind of making it so there’s a pipeline for people to open a GitHub issue and implement a new idea and then have it run in a stable language modeling stack.
然后是斯坦福大学的海洋社区项目，该项目有点让人们可以打开 GitHub 问题并实现新想法，然后让它在稳定的语言建模堆栈中运行。
So this space, that list was way smaller in 2024-
所以到 2024 年，这个空间、这个列表会小得多——
30:31 …
……
so I think it was just AI2.
所以我认为这只是AI2。
So that’s a great thing for more people to get involved and to understand language models, which doesn’t really have a Chinese company that is an analog.
因此，对于更多人参与并理解语言模型来说，这是一件好事，而中国还没有一家公司可以与之类似。
While I’m talking, I’ll say that the Chinese open language models tend to be much bigger and that gives them this higher peak performance as MoEs, whereas a lot of these things that we like a lot, whether it was Gemma or Nemotron, have tended to be smaller models from the US, which is starting to change.
在我说话的同时，我想说的是，中国的开放语言模型往往更大，这使它们作为 MoE 具有更高的峰值性能，而我们非常喜欢的很多东西，无论是 Gemma 还是 Nemotron，都往往是来自美国的较小模型，这种情况正在开始改变。
Mistral Large 3 came out, which was a giant MoE model, very similar to DeepSeek architecture in December.
Mistral Large 3问世，这是一个巨型MoE模型，与12月的DeepSeek架构非常相似。
And then a startup, Reka AI, and both Nemotron have…
然后是一家初创公司 Reka AI，两家 Nemotron 都拥有……
Nemotron and NVIDIA have teased MoE models way bigger than 100 billion parameters-
Nemotron 和 NVIDIA 已经调侃了超过 1000 亿个参数的 MoE 模型 -
31:16 …
……
in the 400 billion parameter range coming in this Q1 2026 timeline.
2026 年第一季度的时间线将包含 4000 亿个参数范围。
So I think this kind of balance is set to change this year in terms of what people are using the Chinese versus US open models for, which I’m personally going to be very excited to watch.
因此，我认为今年这种平衡将在人们使用中国与美国开放模式的用途方面发生变化，我个人会非常兴奋地看到这一点。
First of all, huge props for being able to name so many of these.
首先，能够说出这么多的名字是一个巨大的支持。
31:32 Did you actually name LLaMA?
你真的命名为 LLaMA 吗？
No.
不。
I feel like …
我觉得……
RIP.
安息吧。
This was not on purpose.
这不是故意的。
RIP LLaMA.
安息吧骆驼。
31:43 All right.
好的。
Can you mention what are some interesting models that stand out?
您能提到哪些有趣且脱颖而出的模型吗？
You mentioned Qwen 3 is obviously a standout.
你提到Qwen 3显然是一个佼佼者。
So I would say the year’s almost book-ended by DeepSeek-V3 and DeepSeek R1.
所以我想说，DeepSeek-V3 和 DeepSeek R1 几乎已经结束了这一年。
31:51 And then on the other hand, in December, DeepSeek-V3.2.
另一方面，12 月，DeepSeek-V3.2。
Because what I like about those is they always have an interesting architecture tweak- …
因为我喜欢它们的原因是它们总是有有趣的架构调整——……
that others don’t have.
其他人没有的。
But otherwise, if you want to go with the familiar but really good performance, Qwen 3 and, like Nathan said, also gpt-oss.
但除此之外，如果您想使用熟悉但真正出色的性能，Qwen 3 以及像 Nathan 所说的 gpt-oss。
And I think with gpt-oss, what’s interesting about it is it’s kind of the first open-weight model that was really trained with tool use in mind, which I do think is a bit of a paradigm shift where the ecosystem was not quite ready for it.
我认为 gpt-oss 的有趣之处在于，它是第一个真正考虑到工具使用而进行训练的开放权重模型，我确实认为这是一种范式转变，生态系统还没有完全准备好。
So with tool use, I mean that the LLM is able to do a web search or call a Python interpreter.
因此，对于工具的使用，我的意思是法学硕士能够进行网络搜索或调用Python解释器。
32:33 And I do think it’s a standout because it’s a huge unlock.
我确实认为它很出色，因为它是一个巨大的解锁。
One of the most common complaints about LLMs is, for example, hallucinations, right?
例如，对法学硕士最常见的抱怨之一是幻觉，对吧？
And so, in my opinion, one of the best ways to solve hallucinations is to not try to always remember information or make things up.
因此，在我看来，解决幻觉的最佳方法之一就是不要试图总是记住信息或编造事情。
For math, why not use a calculator app or Python?
对于数学，为什么不使用计算器应用程序或 Python？
32:54 If I ask the LLM, “Who won the soccer World Cup in 1998?” instead of just trying to memorize, it could go do a search.
如果我问法学硕士，“谁赢得了 1998 年足球世界杯？”它不仅可以尝试记忆，还可以进行搜索。
I think mostly it’s usually still a Google search.
我认为大多数情况下仍然是谷歌搜索。
So ChatGPT and gpt-oss, they would do a tool call to Google, maybe find the FIFA website, and find that it was France.
所以 ChatGPT 和 gpt-oss，他们会对 Google 进行工具调用，也许找到 FIFA 网站，然后发现是法国。
It would get you that information reliably instead of just trying to memorize it.
它会给你可靠的信息，而不仅仅是试图记住它。
So I think it’s a huge unlock which right now is not fully utilized yet by the open-weight ecosystem.
所以我认为这是一个巨大的解锁，目前开放权重生态系统尚未充分利用。
A lot of people don’t use tool call modes because I think it’s a trust thing.
很多人不使用工具调用模式，因为我认为这是一个信任的事情。
You don’t want to run this on your computer where it has access to tools and could wipe your hard drive, so you want to containerize that.
您不想在计算机上运行它，因为它可以访问工具并可能擦除您的硬盘驱动器，因此您希望将其容器化。
But I do think that is a really important step for the upcoming years to have this ability.
但我确实认为，对于未来几年拥有这种能力来说，这是非常重要的一步。
So a few quick things.
所以一些快速的事情。
33:44 First of all, thank you for defining what you mean by tool use.
首先，感谢您定义工具使用的含义。
I think that’s a great thing to do in general for the concepts we’re talking about, even things as sort of well-established as MOEs.
我认为对于我们正在讨论的概念来说，这是一件很棒的事情，即使是像 MOE 这样成熟的事情。
You have to say that means mixture of experts, and you kind of have to build up an intuition for people about what that means, how it’s actually utilized, what are the different flavors.
你必须说这意味着专家的混合，你必须让人们对这意味着什么、它的实际使用方式以及不同的口味有什么建立直觉。
So what does it mean that there’s just such an explosion of open models?
那么，开放模型的爆炸式增长意味着什么呢？
What’s your intuition?
你的直觉是什么？
If you’re releasing an open model, you want people to use it, is the first and foremost thing.
如果你要发布一个开放模型，你希望人们使用它，这是首要的事情。
34:13 And then after that comes things like transparency and trust.
然后是透明度和信任之类的事情。
I think when you look at China, the biggest reason is that they want people around the world to use these models, and I think a lot of people will not.
我认为当你看看中国时，最大的原因是他们希望世界各地的人们使用这些模型，但我认为很多人不会。
If you look outside of the US, a lot of people will not pay for software, but they might have computing resources where you can put a model on it and run it.
如果你看看美国以外的地区，很多人不会为软件付费，但他们可能拥有计算资源，你可以在其中放置模型并运行它。
I think there can also be data that you don’t want to send to the cloud.
我认为也可能存在您不想发送到云端的数据。
So the number one thing is getting people to use models, use AI, or use your AI that might not be able to do it without having access to the model.
因此，第一件事是让人们使用模型、使用人工智能，或者使用你的人工智能，而如果没有模型的访问权限，他们可能无法做到这一点。
I guess we should state explicitly, so we’ve been talking about these Chinese models and open weight models.
我想我们应该明确说明，所以我们一直在谈论这些中国模型和开放权重模型。
34:46 Oftentimes, the way they’re run is locally.
通常，它们的运行方式是本地的。
So it’s not like you’re sending your data to China or to whoever developed the model in Silicon Valley.
因此，这并不是说您要将数据发送到中国或硅谷开发该模型的任何人。
A lot of American startups make money by hosting these models from China and selling them.
许多美国初创公司通过托管和销售来自中国的这些模型来赚钱。
35:04 It’s called selling tokens, which means somebody will call the model to do some piece of work.
这称为销售代币，这意味着有人会调用该模型来完成某些工作。
I think the other reason is for US companies like OpenAI.
我认为另一个原因是像 OpenAI 这样的美国公司。
OpenAI is so GPU deprived; they’re at the limits of the GPUs.
OpenAI 的 GPU 匮乏；它们已达到 GPU 的极限。
Whenever they make a release, they’re always talking about how their GPUs are hurting.
每当他们发布版本时，他们总是谈论他们的 GPU 如何受到伤害。
And I think in one of these gpt-oss-120b release sessions, Sam Altman said, “Oh, we’re releasing this because we can use your GPUs.
我认为在一次 gpt-oss-120b 发布会议上，Sam Altman 说道：“哦，我们发布这个是因为我们可以使用你们的 GPU。
We don’t have to use our GPUs and OpenAI can still get distribution out of this,” which is another very real thing, because it doesn’t cost them anything.
我们不必使用 GPU，OpenAI 仍然可以从中获得分发，”这是另一件非常真实的事情，因为这不需要他们花费任何费用。
And for the user, I think also, I mean, there are users who just use the model locally how they would use ChatGPT.
对于用户来说，我也认为，有些用户只是在本地使用该模型，就像他们使用 ChatGPT 一样。
35:43 But also for companies, I think it’s a huge unlock to have these models because you can customize them, you can train them, you can add more data post-training, like specialize them into, let’s say, law, medical models, whatever you have.
但对于公司来说，我认为拥有这些模型是一个巨大的解锁，因为你可以定制它们，你可以训练它们，你可以在训练后添加更多数据，比如将它们专门用于法律、医疗模型等任何你拥有的模型。
And you mentioned Llama; the appeal of the open weight models from China is that the licenses are even friendlier.
你提到了骆驼；来自中国的开放重量模型的吸引力在于许可证更加友好。
I think they are just unrestricted open source licenses, whereas if we use something like Llama or Gemma, there are some strings attached.
我认为它们只是不受限制的开源许可证，而如果我们使用 Llama 或 Gemma 之类的东西，就会附加一些条件。
I think it’s like an upper limit in terms of how many users you have.
我认为这就像你拥有的用户数量的上限。
36:21 And then if you exceed so many million users, you have to report your financial situation to, let’s say, Meta or something like that.
然后，如果你的用户超过了数百万，你就必须向 Meta 或类似的机构报告你的财务状况。
And I think while it is a free model, there are strings attached, and people like things where strings are not attached.
我认为虽然它是一个免费模型，但有附加条件，人们喜欢不附加条件的东西。
So I think that’s also one of the reasons besides performance why the open weight models from China are so popular, because you can just use them.
所以我认为除了性能之外，这也是来自中国的开放重量模型如此受欢迎的原因之一，因为你可以直接使用它们。
There’s no catch in that sense.
从这个意义上来说，没有什么问题。
The ecosystem has gotten better on that front, but mostly downstream of these new providers providing such open licenses.
生态系统在这方面已经变得更好，但主要是这些提供此类开放许可证的新提供商的下游。
36:46 That was funny when you pulled up Perplexity and said, “Kimi K2 Thinking hosted in the US.” Which is an exact example of what we’re talking about where people are sensitive to this.
当你打开 Perplexity 并说“Kimi K2 Thinking 在美国主办”时，这很有趣。这正是我们正在谈论的人们对此敏感的例子。
Kimi K2 Thinking is a model that is very popular.
Kimi K2 Thinking是一款非常受欢迎的机型。
People say that has very good creative writing and also in doing some software things.
人们说他有很好的创意写作，也擅长做一些软件方面的事情。
So it’s just these little quirks that people pick up on with different models that they like.
所以人们只是在他们喜欢的不同型号上发现了这些小怪癖。
What are some interesting ideas that some of these models have explored that you can speak to, like that are particularly interesting to you?
这些模型中的一些模型探索了哪些有趣的想法，您可以谈论这些想法，比如您特别感兴趣的想法？
Maybe we can go chronologically.
也许我们可以按时间顺序来。
37:21 I mean, there was, of course, DeepSeek R1 that came out in January of 2025.
我的意思是，当然还有 2025 年 1 月推出的 DeepSeek R1。
However, this was based on DeepSeek-V3, which came out the year before in December 2024.
然而，这是基于 DeepSeek-V3 的，该版本于前一年的 2024 年 12 月发布。
There are multiple things on the architecture side.
架构方面有很多事情。
What is fascinating is you can still—I mean, that’s what I do with my from-scratch coding projects—you can still start with GPT-2, and you can add things to that model to make it into this other model.
令人着迷的是，您仍然可以——我的意思是，这就是我从头开始的编码项目所做的——您仍然可以从 GPT-2 开始，并且可以向该模型添加内容以使其成为另一个模型。
So it’s all still kind of like the same lineage.
所以这一切仍然有点像同一个血统。
There is a very close relationship between those.
它们之间有着非常密切的关系。
But top of my head, DeepSeek, what was unique there is the Mixture of Experts.
但在我看来，DeepSeek 的独特之处在于专家的混合。
I mean, they were not inventing Mixture of Experts.
我的意思是，他们并没有发明混合专家。
38:00 We can maybe talk a bit more about what Mixture of Experts means.
我们也许可以更多地讨论专家混合的含义。
But just to list these things first before we dive into detail: Mixture of Experts, but then they also had multi-head latent attention, which is a tweak to the attention mechanism.
但在我们深入细节之前先列出这些东西：专家的混合，但他们也有多头潜在注意力，这是对注意力机制的调整。
This was, I would say in 2025, the main distinguishing factor between these open weight models: different tweaks to make inference or KV cache size more economical.
我想说，到 2025 年，这是这些开放权重模型之间的主要区别因素：进行不同的调整，使推理或 KV 缓存大小更加经济。
We can also define KV cache in a few moments.
稍后我们还可以定义 KV 缓存。
But it makes it more economical to have long context, to shrink the KV cache size.
但它使得拥有长上下文、缩小 KV 缓存大小变得更经济。
So what are tweaks that we can do?
那么我们可以做哪些调整呢？
Most of them focused on the attention mechanism.
其中大部分都集中在注意力机制上。
There is multi-head latent attention in DeepSeek; there is group query attention, which is still very popular.
DeepSeek中存在多头潜在注意力；还有群组查询注意力，目前还是很流行的。
38:44 It’s not invented by any of those models; it goes back a few years.
它不是由这些模型中的任何一个发明的；这要追溯到几年前。
But that would be the other option.
但这将是另一种选择。
Sliding window attention, I think OLMo 3 uses it if I remember correctly.
滑动窗口注意，如果我没记错的话，我认为OLMo 3使用了它。
So there are these different tweaks that make the models different.
因此，这些不同的调整使模型有所不同。
Otherwise, I put them all together in an article once where I just compared them; they are surprisingly similar.
除此之外，我曾经将它们全部放在一篇文章中，只是对它们进行了比较；它们惊人地相似。
It’s just different numbers in terms of how many repetitions of the transformer block you have in the center and just little knobs that people tune.
只是中心变压器块的重复次数和人们调节的小旋钮方面的数字不同。
But what’s so nice about it is it works no matter what.
但它的好处是无论如何它都能工作。
You can tweak things, you can move the normalization layers around to get some performance gains.
您可以进行调整，可以移动标准化层以获得一些性能提升。
39:23 And OLMo is always very good in ablation studies, showing what it actually does to the model if you move something around.
OLMo 在消融研究方面总是非常出色，可以显示如果您移动某些物体，它实际上会对模型产生什么影响。
Ablation studies: does it make it better or worse?
消融研究：它会使情况变得更好还是更糟？
But there are so many ways you can implement a transformer and make it still work.
但是有很多方法可以实现变压器并使其仍然工作。
The big ideas that are still prevalent are Mixture of Experts, multi-head latent attention, sliding window attention, and group query attention.
仍然流行的重要思想是专家混合、多头潜在注意力、滑动窗口注意力和组查询注意力。
And then at the end of the year, we saw a focus on making the attention mechanism scale linearly with inference token prediction.
然后在年底，我们看到重点是使注意力机制与推理令牌预测线性扩展。
So there was Qwen3-neXt, for example, which added a gated delta net.
例如，Qwen3-neXt 添加了门控 Delta 网络。
It’s inspired by state space models, where you have a fixed state that you keep updating.
它的灵感来自于状态空间模型，其中你有一个不断更新的固定状态。
But it makes essentially this attention cheaper, or it replaces attention with a cheaper operation.
但它本质上使这种注意力变得更便宜，或者用更便宜的操作取代了注意力。
And it may be useful to step back and talk about transformer architecture in general.
退一步讨论变压器的总体架构可能会很有用。
Yeah, so maybe we should start with GPT-2 architecture, the transformer that was derived from the “Attention Is All You Need” paper.
是的，所以也许我们应该从 GPT-2 架构开始，这个转换器源自“Attention Is All You Need”论文。
40:21 So the “Attention Is All You Need” paper had a transformer architecture that had two parts: an encoder and a decoder.
因此，“Attention Is All You Need”论文有一个 Transformer 架构，该架构由两部分组成：编码器和解码器。
And GPT went with just focusing in on the decoder part.
GPT 只专注于解码器部分。
It is essentially still a neural network and it has this attention mechanism inside.
它本质上仍然是一个神经网络，内部有这种注意力机制。
And you predict one token at a time.
你一次预测一个标记。
You pass it through an embedding layer.
您将其传递给嵌入层。
There’s the transformer block.
这是变压器块。
The transformer block has attention modules and a fully connected layer.
变压器块具有注意力模块和全连接层。
And there are some normalization layers in between.
中间有一些标准化层。
But it’s essentially neural network layers with this attention mechanism.
但它本质上是具有这种注意力机制的神经网络层。
So coming from GPT-2 when we move on to gpt-oss-120b, there is, for example, the Mixture of Experts layer.
因此，当我们从 GPT-2 转向 gpt-oss-120b 时，就会出现专家混合层等。
It’s not invented by GPT-OSS; it’s a few years old.
它不是 GPT-OSS 发明的；它已经有几年了。
41:04 But it is essentially a tweak to make the model larger without consuming more compute in each forward pass.
但这本质上是一种调整，目的是使模型更大，而不会在每次前向传递中消耗更多计算。
So there is this fully connected layer, and if listeners are familiar with multi-layer perceptrons, you can think of a mini multi-layer perceptron, a fully connected neural network layer inside the transformer.
所以就有了这个全连接层，如果听众熟悉多层感知器，你可以想到一个迷你多层感知器，变压器内部的全连接神经网络层。
And it’s very expensive because it’s fully connected.
而且它非常昂贵，因为它是完全连接的。
If you have a thousand inputs and a thousand outputs, that’s like a million connections.
如果你有一千个输入和一千个输出，那就相当于一百万个连接。
And it’s a very expensive part in this transformer.
这是这个变压器中非常昂贵的部件。
And the idea is to kind of expand that into multiple feedforward networks.
这个想法是将其扩展到多个前馈网络。
So instead of having one, let’s say you have 256, but you don’t use all of them at the same time.
因此，假设您有 256 个，而不是一个，但您不会同时使用所有这些。
41:49 So you now have a router that says, “Okay, based on this input token, it would be useful to use this fully connected network.” And in that context, it’s called an expert.
现在你有了一个路由器，它会说：“好吧，根据这个输入令牌，使用这个完全连接的网络将会很有用。”在这种情况下，它被称为专家。
So a Mixture of Experts means you have multiple experts.
因此，专家混合意味着您拥有多名专家。
And depending on what your input is—let’s say it’s more math-heavy—it would use different experts compared to, let’s say, translating input text from English to Spanish.
根据您的输入内容（假设它更注重数学），与将输入文本从英语翻译成西班牙语相比，它会使用不同的专家。
It would maybe consult different experts.
它可能会咨询不同的专家。
It’s not as clear-cut to say, “Okay, this is only an expert for math and this for Spanish.” It’s a bit more fuzzy.
并不是那么明确地说，“好吧，这只是数学专家，这是西班牙语专家。”有点模糊。
But the idea is essentially that you pack more knowledge into the network, but not all the knowledge is used all the time.
但其本质是把更多的知识打包到网络中，但并不是所有的知识都会一直被使用。
42:27 That would be very wasteful.
那会非常浪费。
So yeah, kind of like during the token generation, you are more selective.
所以是的，有点像在代币生成过程中，你更有选择性。
There’s a router that selects which tokens should go to which expert.
有一个路由器可以选择哪些代币应该发送给哪个专家。
It adds more complexity.
它增加了更多的复杂性。
It’s harder to train.
训练起来比较困难。
There’s a lot that can go wrong, like collapse and everything.
有很多事情可能会出错，比如崩溃之类的。
So I think that’s why OLMo 3 still uses dense…
所以我认为这就是 OLMo 3 仍然使用密集的原因......
I mean, you have, I think, OLMo models with Mixture of Experts, but dense models, where dense means…
我的意思是，我认为，您拥有具有专家混合的 OLMo 模型，但是密集模型，其中密集意味着……
So also, it’s jargon.
所以，这也是行话。
There’s a distinction between dense and sparse.
密集和稀疏之间有区别。
So Mixture of Experts is considered sparse because we have a lot of experts, but only a few of them are active.
因此，专家混合被认为是稀疏的，因为我们有很多专家，但其中只有少数是活跃的。
And then dense would be the opposite, where you only have, like, one fully connected module, and it’s always utilized.
而密集则相反，你只有一个完全连接的模块，并且它总是被利用。
So maybe this is a good place to also talk about KV cache.
所以也许这是一个讨论 KV 缓存的好地方。
43:08 But actually, before that, even zooming out, fundamentally, how many new ideas have been implemented from GPT-2 to today?
但实际上，在此之前，即使缩小，从根本上来说，从 GPT-2 到今天，有多少新的想法得以实现？
Like, how different really are these architectures?
比如，这些架构到底有多大不同？
Picture like the Mixture of Experts.
想象一下专家的混合体。
43:25 The attention mechanism in gpt-oss-120b, that would be the Group Query Attention mechanism.
gpt-oss-120b 中的注意力机制，即 Group Query Attention 机制。
So it’s a slight tweak from multi-head attention to Group Query Attention, so that we have two.
所以这是从多头注意力到组查询注意力的轻微调整，所以我们有两个。
I think they replaced LayerNorm by RMSNorm, but it’s just like a different normalization there and not a big change.
我认为他们用 RMSNorm 取代了 LayerNorm，但这只是一个不同的标准化，而不是一个大的变化。
It’s just like a tweak.
这就像一个调整。
The nonlinear activation function—for people familiar with deep neural networks, I mean, it’s the same as changing sigmoid with ReLU.
非线性激活函数——对于熟悉深度神经网络的人来说，我的意思是，它与用 ReLU 改变 sigmoid 是一样的。
It’s not changing the network fundamentally.
它并没有从根本上改变网络。
It’s just like a tweak.
这就像一个调整。
And that’s about it, I would say.
我想说，就是这样。
It’s not really fundamentally that different.
本质上并没有那么不同。
It’s still the same architecture.
它仍然是相同的架构。
So you can convert one from one…
所以你可以从一个转换...
You can go from one into the other by just adding these changes, basically.
基本上，您只需添加这些更改就可以从一种变为另一种。
It fundamentally is still the same architecture.
它基本上仍然是相同的架构。
Mm-hmm.
嗯嗯。
44:12 Yep.
是的。
So for example, you mentioned my book earlier.
例如，您之前提到过我的书。
That’s a GPT-2 model in the book because it’s simple and it’s very small, so 124 million parameters approximately.
这是书中的一个 GPT-2 模型，因为它很简单，而且非常小，大约有 1.24 亿个参数。
But in the bonus materials, I do have OLMo from scratch, Gemini 3 from scratch, and other types of from-scratch models.
但在奖励材料中，我确实有从头开始的 OLMo、从头开始的 Gemini 3 以及其他类型的从头开始的模型。
And I always start with my GPT-2 model and just, you know, add different components and you get from one to the other.
我总是从我的 GPT-2 模型开始，然后添加不同的组件，然后就可以从一个模型过渡到另一个模型。
It’s kind of like a lineage in a sense.
从某种意义上说，这有点像血统。
Yeah.
是的。
Can you build up an intuition for people?
你能为人们建立直觉吗？
44:37 Because sort of when you zoom out and look at it, there’s so much rapid advancement in the AI world, and at the same time, fundamentally the architectures have not changed.
因为当你缩小并观察它时，人工智能世界有如此快速的进步，而与此同时，从根本上说，架构并没有改变。
So where is all the turbulence, the turmoil of the advancement happening?
那么，所有的动荡、进步的动荡都在哪里呢？
Where are the gains to be had?
收益在哪里？
So there are the different stages where you develop or train the network.
因此，开发或训练网络有不同的阶段。
45:01 You have pre-training.
你有预训练。
Now back in the day, it was just pre-training with GPT-2.
现在回到过去，这只是 GPT-2 的预训练。
Now you have pre-training, mid-training, and post-training.
现在你有训练前、训练中和训练后。
So I think right now we are in the post-training focus stage.
所以我认为现在我们正处于培训后的重点阶段。
I mean, pre-training still gives you advantages if you scale it up to better, higher quality data.
我的意思是，如果你将其扩展到更好、更高质量的数据，预训练仍然会给你带来优势。
But then we have capability unlocks that were not there with GPT-2, for example.
但是，例如，我们可以解锁 GPT-2 所没有的功能。
ChatGPT is basically a GPT-3 model.
ChatGPT 基本上是一个 GPT-3 模型。
And GPT-3 is the same as GPT-2 in terms of architecture.
并且GPT-3在架构上与GPT-2相同。
What was new was adding the supervised fine-tuning and the Reinforcement Learning with Human Feedback.
新功能是添加监督微调和带有人类反馈的强化学习。
So, it’s more on the algorithmic side rather than the architecture.
因此，更多的是算法方面而不是架构方面。
I would say that the systems also change a lot.
我想说系统也发生了很大变化。
45:44 I think if you listen to NVIDIA’s announcements, they talk about things like, “You now do FP8, you can now do FP4.” And what is happening is these labs are figuring out how to utilize more compute to put into one model, which lets them train faster and lets them put more data in.
我想如果你听过 NVIDIA 的公告，他们会说“你现在可以做 FP8，你现在可以做 FP4”。目前正在发生的事情是，这些实验室正在研究如何利用更多的计算能力将更多的计算放入一个模型中，这可以让他们训练得更快，并让他们输入更多的数据。
And then you can find better configurations faster by doing this.
然后您可以通过这样做更快地找到更好的配置。
So you can look at the tokens per second per GPU as a metric that you look at when you’re doing large-scale training.
因此，您可以将每个 GPU 每秒的令牌数作为进行大规模训练时查看的指标。
And you can go from, like, 10K to 13K by turning on FP8 training, which means you’re using less memory per parameter in the model.
通过打开 FP8 训练，您可以从 10K 增加到 13K，这意味着模型中每个参数使用的内存更少。
And by saving less information, you do less communication and you can train faster.
通过保存更少的信息，您可以减少沟通，并且可以更快地训练。
46:24 So all of these system things underpin way faster experimentation on data and algorithms.
因此，所有这些系统事物都支持更快的数据和算法实验。
It’s this kind of loop that keeps going where it’s kinda hard to describe when you look at the architecture and they’re exactly the same.
正是这种循环一直持续下去，当你查看架构时，很难描述它们是完全相同的。
But the code base used to train these models is gonna be vastly different— …and you could probably…
但用于训练这些模型的代码库将会有很大不同——……你可能……
the GPUs are different, but you probably train gpt-oss-20b way faster in wall clock time than GPT-2— …was trained at the time.
GPU 是不同的，但你训练 gpt-oss-20b 的时间可能比 GPT-2 快得多——……当时训练的。
Yeah.
是的。
46:54 Like you said, they had, for example, in the Mixture of Experts, this NVIDIA FP4 optimization where you get more throughput.
就像您所说的，例如，他们在 Mixture of Experts 中进行了 NVIDIA FP4 优化，可以让您获得更多吞吐量。
But I do think for the speed, this is true, but it doesn’t give the model new capabilities in a sense.
但我确实认为就速度而言，这是事实，但它并没有在某种意义上赋予模型新的功能。
It’s just: how much can we make the computation coarser without suffering in terms of model performance degradation?
只是：在不降低模型性能的情况下，我们可以将计算粗化到什么程度？
But I do think there are alternatives popping up to the transformer.
但我确实认为变压器有替代方案。
There are text diffusion models, a completely different paradigm.
有文本传播模型，这是一种完全不同的范式。
And although text diffusion models might use transformer architectures, it’s not an autoregressive transformer.
尽管文本扩散模型可能使用变压器架构，但它不是自回归变压器。
And also Mamba models; it’s a State Space Model.
还有曼巴模型；这是一个状态空间模型。
47:34 But they do have trade-offs, and what’s true is there’s nothing that has replaced the autoregressive transformer as the state-of-the-art model.
但它们确实需要权衡，而且事实是，没有任何东西可以取代自回归变压器作为最先进的模型。
So, for state-of-the-art, you would still go with that thing, but there are now alternatives for the cheaper end—alternatives that are kind of making compromises, but it’s not just one architecture anymore.
因此，对于最先进的技术，你仍然会选择那种东西，但现在有更便宜的替代方案——一些妥协的替代方案，但它不再只是一种架构。
There are little ones coming up.
有小孩子来了。
But if we talk about the state-of-the-art, it’s pretty much still the transformer architecture, autoregressive, derived from GPT-2 essentially.
但如果我们谈论最先进的技术，它基本上仍然是 Transformer 架构，自回归，本质上源自 GPT-2。
I guess the big question here is—we talked quite a bit here on the architecture behind the pre-training—are the scaling laws holding strong across pre-training, post-training, inference, context size, data, and synthetic data?
我想这里最大的问题是——我们在这里谈论了很多关于预训练背后的架构——缩放法则在训练前、训练后、推理、上下文大小、数据和合成数据中是否仍然有效？
I’d like to start with the technical definition of a scaling law-
我想从缩放定律的技术定义开始——
48:23 …which kind of informs all of this.
……这说明了这一切。
The scaling law is the power law relationship between…
比例定律是…之间的幂律关系
You can think of the x-axis—what you are scaling—as a combination of compute and data, which are kind of similar, and then the y-axis is like the held-out prediction accuracy over our next tokens.
您可以将 x 轴（您正在缩放的内容）视为计算和数据的组合，这有点相似，然后 y 轴就像我们下一个标记的保留预测精度。
We talked about models being autoregressive.
我们讨论了自回归模型。
It’s like if you keep a set of text that the model has not seen, how accurate will it get when you train?
就像如果你保留一组模型没有见过的文本，那么当你训练时它的准确度有多高？
And the idea of scaling laws came when people figured out that that was a very predictable relationship.
当人们发现这是一种非常可预测的关系时，缩放定律的想法就出现了。
I think that technical term is continuing, and then the question is, what do users get out of it?
我认为这个技术术语还在继续，那么问题是，用户从中得到什么？
And then there are more types of scaling, where OpenAI’s o1 was famous for introducing inference-time scaling.
还有更多类型的扩展，其中 OpenAI 的 o1 以引入推理时间扩展而闻名。
49:07 And I think less famously for also showing that you can scale reinforcement learning training and get kind of this log x-axis and then a linear increase in performance on the y-axis.
我认为不太出名的是还表明您可以扩展强化学习训练并获得这种对数 x 轴，然后在 y 轴上线性提高性能。
So there are kind of these three axes now where the traditional scaling laws are talked about for pre-training—which is how big your model is and how big your dataset is—and then scaling reinforcement learning, which is like how long can you do this trial and error learning that we’ll talk about.
因此，现在在这三个轴上讨论了预训练的传统缩放法则（即模型有多大和数据集有多大），然后缩放强化学习，这就像我们将要讨论的试错学习可以进行多长时间。
We’ll define more of this, and then this inference-time compute, which is just letting the model generate more tokens on a specific problem.
我们将定义更多的内容，然后是推理时间计算，这只是让模型在特定问题上生成更多标记。
49:37 So I’m kind of bullish; they’re all really still working, but the low-hanging fruit has mostly been taken, especially in the last year on Reinforcement Learning with Verifiable Rewards, which is this RLVR, and then inference-time scaling.
所以我有点看好；它们确实仍在发挥作用，但唾手可得的成果大部分已经被实现，特别是去年在具有可验证奖励的强化学习方面，即 RLVR，然后是推理时间扩展。
That’s why these models feel so different to use, where previously you would get that first token immediately.
这就是为什么这些模型使用起来感觉如此不同，以前你会立即获得第一个令牌。
And now they’ll go off for seconds, minutes, or even hours generating these hidden thoughts before giving you the first word of your answer.
现在，他们会花几秒钟、几分钟甚至几个小时产生这些隐藏的想法，然后才给你答案的第一个词。
And that’s all about this inference-time scaling, which is such a wonderful kind of step function in terms of how the models change abilities.
这就是推理时间缩放的全部内容，就模型如何改变能力而言，这是一种非常棒的阶跃函数。
They enabled this tool use stuff and enabled this much better software engineering that we were talking about.
他们启用了这个工具，并启用了我们正在讨论的更好的软件工程。
50:17 And this is, when we say enabled, almost entirely downstream of the fact that this Reinforcement Learning with Verifiable Rewards training just let the models pick up these skills very easily.
当我们说启用时，这几乎完全是事实的下游，这种带有可验证奖励的强化学习训练只是让模型非常容易地掌握这些技能。
So if you look at the reasoning process when the models are generating a lot of tokens, what it’ll often be doing is: it tries a tool, it looks at what it gets back, it tries another API, it sees what it gets back and if it solves the problem.
因此，如果你观察模型生成大量令牌时的推理过程，它通常会做的是：它尝试一个工具，查看它返回的内容，它尝试另一个 API，它查看它返回的内容以及是否解决了问题。
The models, when you’re training them, very quickly learn to do this.
当你训练模型时，它们很快就会学会这样做。
50:46 And then at the end of the day, that gives this kind of general foundation where the model can use CLI commands very nicely in your repo, handle Git for you, move things around, organize things, or search to find more information—which, if we were sitting in these chairs a year ago, is something that we didn’t really think of the models doing.
最后，这提供了这种通用基础，模型可以在您的存储库中很好地使用 CLI 命令、为您处理 Git、移动事物、组织事物或搜索以查找更多信息——如果我们一年前坐在这些椅子上，我们并没有真正想到模型会做这些事情。
So this is just something that has happened this year and has totally transformed how we think of using AI, which I think is very magical.
所以这只是今年发生的事情，彻底改变了我们对使用人工智能的看法，我认为这非常神奇。
It’s such an interesting evolution and unlocks so much value.
这是一个非常有趣的演变，并释放了如此多的价值。
But it’s not clear what the next avenue will be in terms of unlocking stuff like this.
但目前尚不清楚解锁此类内容的下一个途径是什么。
51:23 I think that there’s—we’ll get to continual learning later, but there’s a lot of buzz around certain areas of AI, but no one knows when the next step function will really come.
我认为，我们稍后会持续学习，但围绕人工智能的某些领域有很多讨论，但没有人知道下一步功能何时真正到来。
So you’ve actually said quite a lot of things there, and said profound things quickly.
所以你实际上讲了很多东西，而且讲得很快。
51:31 It would be nice to unpack them a little bit.
如果能稍微拆开它们的包装就好了。
You say you’re bullish basically on every version of scaling.
你说你基本上看好每个版本的扩展。
So can we just start at the beginning?
那么我们可以从头开始吗？
Pre-training: are we implying that the low-hanging fruit on pre-training scaling has been picked?
预训练：我们是否意味着预训练扩展上容易实现的目标已经被摘取了？
Has pre-training hit a plateau, or are you still bullish on even pre-training?
预训练是否已达到稳定水平，或者您仍然看好预训练？
Pre-training has gotten extremely expensive.
预训练变得极其昂贵。
52:01 I think to scale up pre-training, it’s also implying that you’re going to serve a very large model to the users.
我认为扩大预训练规模也意味着你将为用户提供一个非常大的模型。
So I think that it’s been loosely established the likes of GPT-4 and similar models were around one trillion parameters at the biggest size.
因此，我认为 GPT-4 之类的类似模型已经松散地确定了最大尺寸约为一万亿个参数。
There’s a lot of rumors that they’ve actually gotten smaller as training has gotten more efficient.
有很多传言称，随着训练变得更加高效，它们实际上变得更小了。
You want to make the model smaller because then your costs of serving go down proportionately.
您希望缩小模型，因为这样您的服务成本就会相应下降。
The cost of training these models is really low relative to the cost of serving them to hundreds of millions of users.
相对于为数亿用户提供服务的成本而言，训练这些模型的成本确实很低。
I think DeepSeek had this famous number of about five million dollars for pre-training at cloud market rates.
我认为 DeepSeek 拥有大约 500 万美元的著名数字，用于按云市场价格进行预训练。
52:40 In the OLMo 3 paper, section 2.4, we just detailed how long we had the GPU clusters sitting around for training—which includes engineering issues, multiple seeds—and it was about two million dollars to rent the cluster to deal with all the problems and headaches of training a model.
在 OLMo 3 论文第 2.4 节中，我们刚刚详细说明了 GPU 集群用于训练的时间（包括工程问题、多个种子），租用集群来处理训练模型的所有问题和难题大约需要 200 万美元。
So these models are…
所以这些模型是……
a lot of people could get one to 10 million dollars to train a model, but the recurring costs of serving millions of users is really billions of dollars of compute.
很多人可以获得一到一千万美元来训练模型，但为数百万用户提供服务的经常性成本实际上是数十亿美元的计算成本。
A thousand GPU rental you can pay 100 grand a day for.
一千个 GPU 的租金你每天可以支付 100 英镑。
And these companies could have millions of GPUs.
这些公司可能拥有数百万个 GPU。
You can look at how much these things cost to sit around.
你可以看看这些东西坐在那里要花多少钱。
53:19 So that’s kind of a big thing, and then it’s like, if scaling is actually giving you a better model, is it going to be financially worth it?
所以这是一件大事，然后就好像，如果扩展实际上给了你一个更好的模型，那么它在经济上值得吗？
And I think we’ll slowly push it out as AI solves more compelling tasks—like the likes of Claude Opus 4.5 making Claude Code just work for things.
我认为，随着人工智能解决更引人注目的任务，我们会慢慢推出它——比如 Claude Opus 4.5 等，让 Claude Code 能够为事情工作。
I launched this project called the ATOM project, which is American Truly Open Models, in July, and that was like a true vibe-coded website.
我在 7 月份启动了这个名为 ATOM 项目的项目，即 American Truly Open Models，这就像一个真正的氛围编码网站。
I have a job to make plots and stuff.
我的工作是制作情节之类的。
Then I came back to refresh it in the last few weeks and Claude Opus 4.5, versus whatever model was available at the time, just crushed all the issues that it had from building in June and July.
然后我在过去几周回来更新它，Claude Opus 4.5 与当时可用的任何模型相比，刚刚解决了 6 月和 7 月构建时遇到的所有问题。
It might be a bigger model.
它可能是一个更大的模型。
There’s a lot of things that go into this, but there’s still progress coming.
这涉及很多事情，但仍有进展。
So what you’re speaking to is the nuance of the y-axis of the scaling laws—that the way it’s experienced versus on a benchmark, the actual intelligence might be different.
所以你所说的是缩放定律 y 轴的细微差别，即它的体验方式与基准相比，实际的智能可能会有所不同。
54:04 But still, your intuition about pre-training: if you scale the size of compute, will the models get better?
但是，您对预训练的直觉仍然是：如果扩大计算规模，模型会变得更好吗？
Not whether it’s financially viable, but just from the law aspect of it, do you think the models will get smarter?
不是在经济上是否可行，而是从法律方面来看，你认为这些模型会变得更聪明吗？
Yeah.
是的。
54:28 And I think that there’s…
我认为有……
And this sometimes comes off as almost disillusioned from leadership at AI companies saying this, but they’re like, “It’s held for 13 orders of magnitude of compute; why would it ever end?” So I think fundamentally it is pretty unlikely to stop.
有时，人工智能公司的领导层说这句话时几乎是幻灭了，但他们会说，“它可以维持 13 个数量级的计算；为什么它会结束？”所以我认为从根本上来说它不太可能停止。
It’s just like eventually we’re not even going to be able to test the bigger scales because of all the problems that come with more compute.
就像最终我们甚至无法测试更大的规模一样，因为更多计算带来的所有问题。
I think that there’s a lot of talk on how 2026 is a year when very large NVIDIA Blackwell compute clusters—like gigawatt-scale facilities—are coming online.
我认为，关于 2026 年非常大的 NVIDIA Blackwell 计算集群（如千兆瓦级设施）上线的一年有很多讨论。
And these were all contracts for power and data centers that were signed and sought out in ’22 and 2023, before or right after ChatGPT.
这些都是在 ChatGPT 之前或之后于 22 年和 2023 年签署和寻求的电力和数据中心合同。
55:13 So it took this two-to-three-year lead time to build these bigger clusters to train the models, while there’s obviously immense interest in building even more data centers than that.
因此，我们花了两到三年的时间来构建这些更大的集群来训练模型，而显然人们对构建更多的数据中心有着巨大的兴趣。
So that is kind of the crux that people are saying: these new clusters are coming.
这就是人们所说的症结所在：这些新的集群即将到来。
The labs are going to have more compute for training.
实验室将拥有更多的计算资源用于培训。
They’re going to utilize this, but it’s not a given.
他们会利用这一点，但这并不是理所当然的。
I’ve seen so much progress that I expect it, and I expect a little bit bigger models.
我已经看到了如此多的进步，我期待它，我期待更大的模型。
I would say it’s more like we’ll see a $2,000 subscription this year; we’ve already seen $200 subscriptions.
我想说今年我们会看到 2,000 美元的订阅费；我们已经看到了 200 美元的订阅。
It’s like that could 10x again, and these are the kind of things that could come—and they’re all downstream of a bigger model that offers just a little bit more of a cutting edge.
就像可以再增长 10 倍一样，这些都是可能发生的事情，而且它们都是更大模型的下游，该模型提供了更多的尖端技术。
So, it’s reported that xAI is going to hit that one-gigawatt scale early ’26, and a full two gigawatts by year end.
因此，据报道，xAI 将在 26 年初达到 1 吉瓦的规模，到年底将达到整整 2 吉瓦。
55:53 How do you think they’ll utilize that in the context of scaling laws?
您认为他们会如何在缩放定律的背景下利用它？
Is a lot of that inference?
是不是有很多这样的推论？
Is a lot of that training?
这样的训练很多吗？
It ends up being all of the above.
最终变成了以上所有内容。
56:12 I think that all of your decisions when you’re training a model come back to pre-training.
我认为训练模型时的所有决定都会回到训练前。
So if you’re going to scale RL on a model, you still need to decide on your architecture that enables this.
因此，如果您要在模型上扩展强化学习，您仍然需要决定支持此目的的架构。
We were talking about other architectures and using different types of attention.
我们正在讨论其他架构并使用不同类型的注意力。
We’re also talking about Mixture of Experts models.
我们还在讨论专家混合模型。
The sparse nature of MoE models makes it much more efficient to do generation, which becomes a big part of post-training, and you need to have your architecture ready so that you can actually scale up this compute.
MoE 模型的稀疏性质使其生成更加高效，这成为训练后的重要组成部分，并且您需要准备好架构，以便能够实际扩展此计算。
I still think most of the compute is going in at pre-training.
我仍然认为大部分计算都是在预训练时进行的。
Because you can still make a model better, you still want to go and revisit this.
因为你仍然可以让模型变得更好，所以你仍然想重新审视这个。
56:53 You still want the best base model that you can.
您仍然想要尽可能最好的基础模型。
And in a few years that’ll saturate and the RL compute will just go longer.
几年后，这个数字就会饱和，强化学习的计算时间将会变得更长。
Are there people who disagree with you that say basically pre-training is dead?
是否有人不同意你的观点，认为预训练基本上已经死了？
57:00 That it’s all about scaling inference, scaling post-training, scaling context, continual learning, and scaling synthetic data?
这一切都是关于扩展推理、扩展训练后、扩展上下文、持续学习和扩展合成数据？
People vibe that way and describe it in that way, but I think it’s not the practice that is happening.
人们有那样的感觉并以那样的方式描述它，但我认为这并不是正在发生的做法。
It’s just the general vibe of people saying this thing is dead—
人们普遍认为这东西已经死了——
The excitement is elsewhere.
兴奋在别处。
57:21 So the low-hanging fruit— …in RL is elsewhere.
因此，强化学习中唾手可得的成果——……在其他地方。
For example, we released our model in November.
例如，我们在 11 月份发布了我们的模型。
Every company has deadlines.
每个公司都有最后期限。
Our deadline was like November 20th, and for that, our run was five days, which compared to 2024 is a very long time to just be doing post-training on a model of about 30 billion parameters.
我们的截止日期是 11 月 20 日，为此，我们的运行时间为 5 天，与 2024 年相比，对于大约 300 亿个参数的模型进行后期训练来说，这是一个非常长的时间。
It’s not a big model.
这不是一个大模型。
And then in December, we had another release, which was just letting the RL run for another three and a half weeks, and the model got notably better, so we released it.
然后在 12 月，我们发布了另一个版本，只是让 RL 再运行三周半，模型明显变得更好，所以我们发布了它。
And that’s a big amount of time to just allocate to something that is going to be your peak— …for the year.
这需要大量的时间来分配给那些将成为你今年巅峰的事情。
So it’s like—
所以就像——
The reasoning is—
理由是——
There’s these types of decisions that happen when they’re training a model where they just can’t leave it forever.
当他们训练一个模型时，会做出这些类型的决定，但他们不能永远离开它。
57:58 You have to keep pulling in the improvements you have from your researchers.
你必须不断吸收研究人员的改进成果。
So you redo pre-training, you’ll do this post-training for a month, but then you need to give it to your users.
所以你重新进行预训练，你将进行一个月的后训练，但随后你需要将其提供给你的用户。
You need to do safety testing.
您需要进行安全测试。
I think there’s a lot in place that reinforces this cycle of just keep updating the models.
我认为有很多地方可以强化这个不断更新模型的循环。
There’s things to improve.
有一些事情需要改进。
You get a new compute cluster that lets you do something maybe more stably or faster.
您将获得一个新的计算集群，可以让您更稳定或更快速地执行某些操作。
You hear a lot about Blackwell having rollout issues, where at AI2 most of the models we’re pre-training are on like 1,000 to 2,000 GPUs.
您经常听说 Blackwell 存在推出问题，在 AI2，我们预训练的大多数模型都在 1,000 到 2,000 个 GPU 上。
58:36 But when you’re pre-training on 10,000 or 100,000 GPUs, you hit very different failures.
但是，当您在 10,000 个或 100,000 个 GPU 上进行预训练时，您会遇到截然不同的失败。
GPUs are known to break in weird ways, and doing a 100,000 GPU run is like…
众所周知，GPU 会以奇怪的方式出现故障，进行 100,000 次 GPU 运行就像……
you’re pretty much guaranteed to always have at least one GPU that is down.
几乎可以保证至少有一个 GPU 处于关闭状态。
And you need to have your training code handle that redundancy, which is just a very different problem.
你需要让你的训练代码处理这种冗余，这是一个非常不同的问题。
Whereas what we’re doing like, “Oh, I’m playing with post-training on DJI Spark,” or people learning ML, what they’re battling to train these biggest models is just like— …mass distributed scale, and it’s very different.
而我们正在做的事情就像是，“哦，我正在 DJI Spark 上进行后期训练”，或者人们正在学习 ML，而他们正在努力训练这些最大的模型，就像——……大规模分布式规模，这是非常不同的。
But that’s somewhat different than…
但这与……有些不同。
that’s a systems problem—
这是一个系统问题——
59:11 …in order to enable the scaling laws, especially at pre-training.
…为了启用缩放法则，特别是在预训练时。
You need all of these GPUs at once.
您同时需要所有这些 GPU。
When we shift to reinforcement learning, it actually lends itself to heterogeneous compute because you have many copies of the model.
当我们转向强化学习时，它实际上适合异构计算，因为你有许多模型副本。
To do a primer for language model reinforcement learning, what you’re doing is you have two sets of GPUs.
要了解语言模型强化学习的入门知识，您需要拥有两组 GPU。
One you can call the actor and one you call the learner.
一种你可以称为演员，另一种你可以称为学习者。
The learner is where your actual reinforcement learning updates happen.
学习器是实际强化学习更新发生的地方。
These are traditionally policy gradient algorithms.
这些是传统的策略梯度算法。
Proximal Policy Optimization, PPO, and Group Relative Policy Optimization, GRPO, are the two popular classes.
近端策略优化 (PPO) 和组相对策略优化 (GRPO) 是两个流行的类别。
59:50 On the other side, you’re going to have actors which are generating completions, and these completions are the things that you’re going to grade.
另一方面，演员将生成完成内容，而这些完成内容就是您要评分的内容。
Reinforcement learning is all about optimizing reward.
强化学习就是为了优化奖励。
In practice, you can have a lot of different actors in different parts of the world doing different types of problems, and then you send it back to this highly networked compute cluster to do this actual learning, where you take the gradients and you need to have a tightly meshed network where you can do different types of parallelism and spread out your model for efficient training.
在实践中，您可以让世界不同地区的许多不同参与者解决不同类型的问题，然后将其发送回这个高度网络化的计算集群来进行实际学习，您可以在其中获取梯度，并且需要有一个紧密的网状网络，您可以在其中执行不同类型的并行性并展开模型以进行有效的训练。
Every different type of training and serving has these considerations you need to scale.
每种不同类型的培训和服务都有您需要扩展的考虑因素。
1:00:27 We talked about pre-training, we talked about RL, and then inference time scaling is: how do you serve a model that’s thinking for an hour to 100 million users?
我们讨论了预训练，我们讨论了 RL，然后推理时间缩放是：如何为 1 亿用户思考一小时的模型提供服务？
I don’t really know about that, but I know that’s a hard problem.
我真的不知道，但我知道这是一个难题。
In order to give people this intelligence, there’s all these systems problems, and we need more compute and you need more stable compute to do it.
为了给人们这种智能，存在所有这些系统问题，我们需要更多的计算，并且需要更稳定的计算来完成它。
But you’re bullish on all of these kinds of scaling is what I’m hearing.
但我听到的是，你们对所有这些类型的扩张都持乐观态度。
1:00:46 On the inference, on the reasoning, even on the pre-training?
关于推理，关于推理，甚至关于预训练？
Yeah, so that’s a big can of worms, but there are basically two knobs: training and inference scaling, where you can get gains.
是的，这是一大堆蠕虫，但基本上有两个旋钮：训练和推理缩放，你可以从中获得收益。
1:00:54 In a world where we had infinite compute resources, you’d want to do all of them.
在一个拥有无限计算资源的世界中，您会想要完成所有这些任务。
You have training, you have inference scaling, and training is like a hierarchy: pre-training, mid-training, and post-training.
你有训练，你有推理扩展，训练就像一个层次结构：训练前、训练中期和训练后。
Changing the model size, more training data, training a bigger model—it gives you more knowledge.
改变模型大小、更多训练数据、训练更大的模型——它会给你更多的知识。
Then the model is a better base model, or what we still call a foundation model, and it unlocks capabilities.
那么该模型就是一个更好的基础模型，或者我们仍然称之为基础模型，它可以释放功能。
But you don’t necessarily have the model be able to solve your most complex tasks—
但你的模型不一定能够解决你最复杂的任务——
1:01:34 …tasks during pre-training or after pre-training.
…预训练期间或预训练之后的任务。
You still have these other unlock phases, mid-training or post-training with RL, that unlocks capabilities that the model has in terms of knowledge from the pre-training.
您仍然有这些其他解锁阶段，即强化学习的训练中期或训练后，可以解锁模型在预训练知识方面所具有的功能。
And I think, sure, if you do more pre-training, you get a better base model that you can unlock later.
我认为，当然，如果你进行更多的预训练，你就会得到一个更好的基础模型，以后可以解锁。
But like Nathan said, it just becomes too expensive.
但正如内森所说，它变得太贵了。
We don’t have infinite compute, so you have to decide: do I want to spend that compute more on making the model larger?
我们没有无限的计算，所以你必须决定：我是否想花更多的计算来使模型更大？
It’s a trade-off.
这是一种权衡。
In an ideal world, you want to do all of them.
在理想的世界中，您想要完成所有这些。
And I think in that sense, scaling is still pretty much alive.
我认为从这个意义上说，规模化仍然存在。
1:02:08 You would still get a better model, but like we saw with Claude 4.5, it’s just not worth it.
你仍然会得到一个更好的模型，但就像我们在 Claude 4.5 中看到的那样，这是不值得的。
I mean, because you can unlock more performance with other techniques at that moment, especially if you look at inference scaling.
我的意思是，因为此时您可以使用其他技术释放更多性能，特别是如果您考虑推理扩展。
That’s one of the biggest gains this year with o1, where it took a smaller model further than pre-training a larger model like Claude 4.5.
这是 o1 今年最大的收获之一，它比预训练 Claude 4.5 这样的较大模型更进一步地使用了较小的模型。
So, I wouldn’t say pre-training scaling is dead; it’s just that there are other more attractive ways to scale right now.
所以，我不会说预训练扩展已经死了。只是现在还有其他更有吸引力的扩展方式。
But at some point, you will still want to make some progress on the pre-training.
但在某些时候，您仍然希望在预训练上取得一些进展。
The thing to consider is where you want to spend your money.
需要考虑的是你想把钱花在哪里。
1:02:47 If you spend it more on pre-training, it’s a fixed cost.
如果你在预训练上花费更多，那就是固定成本。
You train the model, and then it has this capability forever.
你训练模型，然后它就永远拥有这种能力。
You can always use it.
您可以随时使用它。
With inference scaling, you don’t spend money during training; you spend money later per query, and then it’s about the math.
通过推理扩展，您在训练期间无需花钱；你稍后每次查询都要花钱，然后就是数学问题了。
How long is my model going to be on the market if I replace it in half a year?
如果我半年更换的话，我的型号还能上市多久？
Maybe it’s not worth spending 5 million, 10 million, or 100 million dollars on training it longer.
也许花 500 万、1000 万或 1 亿美元来训练它更长时间是不值得的。
Maybe I will just do more inference scaling and get the performance from there.
也许我会做更多的推理扩展并从中获得性能。
It maybe costs me 2 million in terms of user queries.
就用户查询而言，我可能花费了 200 万。
It becomes a question of how many users you have and doing the math.
这变成了你有多少用户并进行数学计算的问题。
I think that’s also where it’s interesting, where ChatGPT is in a position.
我认为这也是有趣的地方，ChatGPT 所处的位置。
1:03:27 I think they have a lot of users where they need to go a bit cheaper, where they have that GPT-5 model that is a bit smaller.
我认为他们有很多用户需要更便宜的价格，他们有更小的 GPT-5 模型。
For other companies, their customers have other trade-offs.
对于其他公司来说，他们的客户还有其他权衡。
For example, there were the math problems or the Math Olympiad where they had a proprietary model, and I’m pretty sure it’s just a model that has been fine-tuned a little bit more, but most of it was inference scaling to achieve peak performance in certain tasks where you don’t need that all the time.
例如，在数学问题或数学奥林匹克竞赛中，他们有一个专有模型，我很确定这只是一个经过微调的模型，但其中大部分是推理扩展，以在某些任务中实现峰值性能，而您并不总是需要它。
But yeah, long story short, I do think pre-training, mid-training, post-training, and inference scaling are all still things you want to do.
但是，长话短说，我确实认为训练前、训练中、训练后和推理扩展仍然是你想做的事情。
At the moment, this year, it’s finding the right ratio that gives you the best bang for the buck, basically.
目前，今年，基本上，它正在寻找合适的比率，为您带来最大的收益。
I think this might be a good place to define pre-training, mid-training, and post-training.
我认为这可能是定义训练前、训练中和训练后的好地方。
So, pre-training is the classic training one next token prediction at a time.
因此，预训练是一次训练下一个标记预测的经典方法。
1:04:18 You have a big corpus of data.
您拥有大量数据。
Nathan probably also has very interesting insights there because of OLMo 3.
由于 OLMo 3，Nathan 可能也有非常有趣的见解。
A big portion of the paper focuses on the right data mix.
本文的很大一部分重点关注正确的数据组合。
So, pre-training is essentially just training across entropy loss, training on next token prediction on a vast corpus of internet data, books, papers and so forth.
因此，预训练本质上只是熵损失的训练，在大量互联网数据、书籍、论文等的语料库上训练下一个标记预测。
It has changed a little bit over the years in the sense people used to throw in everything they can.
这些年来，情况发生了一些变化，人们过去常常竭尽全力地投入一切。
Now, it’s not just raw data.
现在，这不仅仅是原始数据。
It’s also synthetic data where people rephrase certain things.
它也是人们重新表述某些事物的合成数据。
So synthetic data doesn’t necessarily mean purely AI-made-up data.
因此，合成数据并不一定意味着纯粹是人工智能编造的数据。
1:04:58 It’s also taking something from a Wikipedia article and then rephrasing it as a Q&A question or summarizing it, rewarding it, and making better data that way.
它还从维基百科文章中获取一些内容，然后将其重新表述为问答问题或对其进行总结、奖励，并以这种方式生成更好的数据。
I think of it like with humans.
我认为这就像人类一样。
If someone reads a book compared to a messy—no offense, but like—Reddit post or something like that.
如果有人把一本书比作一个混乱的——无意冒犯，但就像——Reddit 帖子或类似的东西。
I do think you learn—no offense, but I think—
我确实认为你学到了——无意冒犯，但我认为——
There’s going to be a post about this, Sebastian.
将会有一篇关于此的文章，塞巴斯蒂安。
Some Reddit data is very coveted and excellent for training.
一些 Reddit 数据非常令人垂涎，非常适合训练。
1:05:28 You just have to filter it.
你只需要过滤它。
And I think that’s the idea.
我认为这就是这个想法。
1:05:33 I think it’s like if someone took that and rephrases it in a, let’s say, more concise and structured way— I think it’s higher quality data that gets the LLM maybe the same—you get the same LLM out of it at the end, but it gets there faster.
我认为这就像如果有人接受了这一点，并以更简洁和结构化的方式重新表述——我认为获得法学硕士的质量更高的数据可能是相同的——你最终会得到相同的法学硕士，但它到达的速度更快。
It trains faster because if the grammar and the punctuation are correct, it already learns the correct way versus getting information from a messy way and then learning later how to correct that.
它训练得更快，因为如果语法和标点符号正确，它就已经学会了正确的方式，而不是从混乱的方式获取信息，然后再学习如何纠正。
So, I think that is how pre-training evolved and why scaling still works; it’s not just about the amount of data, it’s also the tricks to make that data better for you.
所以，我认为这就是预训练的演变方式以及扩展仍然有效的原因；这不仅仅是数据量的问题，还包括让数据更好地为您服务的技巧。
And then mid-training is…
然后训练中期是……
I mean, it used to be called pre-training.
我的意思是，它曾经被称为预训练。
1:06:21 I think it’s called mid-training because it was awkward to have pre-training and post-training but nothing in the middle, right?
我认为之所以称为中期训练，是因为有训练前和训练后但中间没有任何内容很尴尬，对吗？
It sounds a bit weird.
听起来有点奇怪。
You have pre-training and post-training, but what’s the actual training?
你有预训练和后训练，但实际的训练是什么？
So, the mid-training is usually similar to pre-training, but it’s a bit more specialized.
所以，中期训练通常与预训练类似，但更专业一些。
It’s the same algorithm, but what you do is you focus, for example, on long context documents.
这是相同的算法，但您所做的是专注于例如长上下文文档。
The reason you don’t do that during pre-training is because you don’t have that many long context documents.
您在预训练期间不这样做的原因是因为您没有那么多长上下文文档。
We have a specific phase.
我们有一个特定的阶段。
And one problem of LLMs is still that it’s a neural network; it has the problem of catastrophic forgetting.
法学硕士的一个问题仍然是它是一个神经网络；它存在灾难性遗忘的问题。
1:06:56 So, you teach it something, it forgets other things.
所以，你教它一些东西，它会忘记其他东西。
It’s not 100% forgetting, but there’s no free lunch.
虽然不是百分百忘记，但天下没有免费的午餐。
It’s also the same with humans.
人类也同样如此。
If you ask me some math I learned 10 years ago, I wouldn’t know; I would have to look at it again.
如果你问我十年前学过的数学，我不会知道；我得再看一遍。
Nathan was actually saying that he’s consuming so much content that there’s a catastrophic forgetting issue.
内森实际上是说他消耗了太多的内容，以至于出现了灾难性的遗忘问题。
Yeah, I’m trying to learn so much about AI, and it’s like when I was learning about pre-training parallelism, I’m like, “I lost something and I don’t know what it was.”
是的，我正在努力学习很多关于人工智能的知识，就像当我学习预训练并行性时，我会想，“我失去了一些东西，但我不知道它是什么。”
I don’t want to anthropomorphize LLMs, but I think it’s the same in terms of how humans learn.
我不想将法学硕士拟人化，但我认为就人类的学习方式而言，这是一样的。
1:07:22 Quantity is not always better because it’s about being selective.
数量并不总是更好，因为它是有选择性的。
Mid-training is being selective in terms of quality content at the end, so the last thing the LLM has seen is the quality stuff.
中期培训是在最后对高质量内容进行选择性，因此法学硕士最后看到的就是高质量的内容。
And then post-training is all the fine-tuning: supervised fine-tuning, DPO, RLVR with human feedback and so forth.
然后训练后就是所有的微调：监督微调、DPO、带有人类反馈的 RLVR 等等。
So, the refinement stages.
所以，细化阶段。
And it’s also interesting, the cost thing, right?
这也很有趣，成本问题，对吧？
Pre-training, you spend a lot of money on that right now.
预训练，你现在在这上面花了很多钱。
RL a bit less.
RL少了一点。
RL, you don’t really teach it knowledge; it’s more like unlocking the knowledge.
RL，你并没有真正教它知识；这更像是解锁知识。
1:08:03 It’s more like skill learning, like how to solve problems with the knowledge that it has from pre-training.
它更像是技能学习，比如如何利用预训练中获得的知识解决问题。
There are actually three papers this year, or last year, 2025, on RL for pre-training.
今年，或者去年，2025 年，实际上已经有三篇关于 RL 预训练的论文。
But I don’t think anyone does that in production.
但我认为没有人在生产中这样做。
Toy, toy examples for now.
玩具，现在的玩具示例。
Toy examples, right.
玩具示例，对。
1:08:18 Но to generalize, RL post-training is more like the skill unlock, where pre-training is like soaking up the knowledge essentially.
概括地说，强化学习后训练更像是技能解锁，而预训练本质上就像吸收知识。
A few things that could be helpful for people.
一些可能对人们有帮助的事情。
1:08:26 A lot of people think of synthetic data as being bad for training models.
许多人认为合成数据不利于训练模型。
You mentioned that DeepSeek got an OCR—Optical Character Recognition—paper.
您提到 DeepSeek 获得了一篇 OCR（光学字符识别）论文。
A lot of labs did; AI2 had one, others had multiple.
很多实验室都这么做了； AI2 有一个，其他有多个。
And the reason each of these labs has these is because there’s vast amounts of PDFs and other digital documents on the web in formats that aren’t encoded with text easily.
这些实验室之所以拥有这些，是因为网络上存在大量难以用文本编码的格式的 PDF 和其他数字文档。
So you use these, like DeepSeek OCR or what we called OLMo OCR, to extract what can be trillions of tokens of candidate data.
因此，您可以使用 DeepSeek OCR 或我们所说的 OLMo OCR 之类的工具来提取可能数万亿个标记的候选数据。
Pre-training dataset size is on the order of trillions; it’s measured in trillions of tokens.
预训练数据集规模为万亿量级；它以数万亿代币来衡量。
1:09:10 Smaller models from researchers can be something like 5 to 10 trillion.
研究人员的较小模型可能约为 5 到 10 万亿。
Qwen is documented going up to like 50 trillion, and there’s rumors that these closed labs can go to 100 trillion tokens.
据记录，Qwen 的代币数量高达 50 万亿美元，有传言称这些封闭实验室的代币数量可达 100 万亿美元。
Getting this potential data is a very big funnel, and the data you actually train the model on is a small percentage of this.
获取这些潜在数据是一个非常大的漏斗，而您实际训练模型的数据只占其中的一小部分。
This character recognition data would be described as synthetic data for pre-training in a lab.
该字符识别数据将被描述为用于实验室预训练的合成数据。
And then there’s also the fact that ChatGPT now gives wonderful answers, and you can train on those best answers; that’s synthetic data.
还有一个事实是，ChatGPT 现在可以提供精彩的答案，您可以根据这些最佳答案进行训练；那是合成数据。
It’s very different than the early ChatGPT hallucinations data.
这与早期的 ChatGPT 幻觉数据有很大不同。
One interesting question is, if I recall correctly, OLMo 3 was trained with less data than specifically some other open-weight models, maybe even OLMo 2.
一个有趣的问题是，如果我没记错的话，OLMo 3 的训练数据比其他一些开放权重模型（甚至可能是 OLMo 2）要少。
1:09:48 But you still got better performance, and that might be one of the examples of how the data helped.
但你仍然获得了更好的性能，这可能是数据提供帮助的例子之一。
It’s mostly down to data quality.
这主要取决于数据质量。
1:10:01 I think if we had more compute, we would train for longer.
我认为如果我们有更多的计算能力，我们就能训练更长时间。
I think we’d ultimately see that as something we would want to do.
我认为我们最终会认为这是我们想做的事情。
Especially with big models, you need more compute because big models can absorb more from data, and you get more benefit out of this.
特别是对于大型模型，您需要更多的计算，因为大型模型可以从数据中吸收更多，并且您可以从中获得更多好处。
It’s like one of those logarithmic graphs—a small model will level off sooner if you’re measuring tons of tokens, and bigger models need more.
它就像那些对数图之一——如果你测量大量的代币，小模型会更快趋于平稳，而更大的模型需要更多。
But mostly, we aren’t training that big of models right now at AI2, and getting the highest quality data we can is the natural starting point.
但大多数情况下，我们现在在 AI2 上并没有训练那么大的模型，获得尽可能高质量的数据是自然的起点。
Is there something to be said about the topic of data quality?
关于数据质量这个话题有什么要说的吗？
1:10:38 Is there some low-hanging fruit there still where the quality could be improved?
是否还有一些容易实现的目标可以提高质量？
It’s like turning the crank.
就像转动曲柄一样。
1:10:46 So I think historically, in the open, there’s been a canonical best pre-training dataset that has moved around between who has the most recent one or the best recent effort.
所以我认为，从历史上看，在公开的情况下，存在一个规范的最佳预训练数据集，该数据集在最近的数据集或最近最好的数据集之间移动。
Like AI2’s Dolma was very early with the first OLMo and Hugging Face had FineWeb.
就像 AI2 的 Dolma 很早就推出了第一个 OLMo，而 Hugging Face 则推出了 FineWeb。
And there’s the DCLM project, which stands for Data Comp Language Model.
还有 DCLM 项目，它代表数据计算语言模型。
There’s been Data Comp for other machine learning projects, and they had a very strong dataset.
其他机器学习项目也有 Data Comp，他们拥有非常强大的数据集。
A lot of it is the internet becoming fairly closed off, so we have Common Crawl, which I think is hundreds of trillions of tokens, and you filter it.
其中很大一部分是互联网变得相当封闭，所以我们有 Common Crawl，我认为这是数百万亿的代币，你可以过滤它。
1:11:21 And it looks like a lot of scientific work where you’re training classifiers and making decisions based on how you prune down this dataset into the highest quality stuff and the stuff that suits your tasks.
它看起来像很多科学工作，你在训练分类器并根据如何将数据集修剪成最高质量的东西和适合你的任务的东西来做出决策。
Previously, language models were tested a lot more on knowledge and conversational things, but now they’re expected to do math and code.
以前，语言模型在知识和对话事物上进行了更多测试，但现在人们期望它们能够进行数学和编码。
To train a reasoning model, you need to remix your whole dataset.
要训练推理模型，您需要重新混合整个数据集。
And there are actually some wonderful scientific methods here where you can take your gigantic dataset and sample a lot of really tiny things from different sources, like GitHub, Stack Exchange, Reddit, or Wikipedia.
实际上，这里有一些很棒的科学方法，您可以使用庞大的数据集并从不同来源（例如 GitHub、Stack Exchange、Reddit 或 Wikipedia）采样许多非常微小的东西。
1:11:56 You can sample small things from them, train small models on each of these mixes, and measure their performance on your evaluations.
您可以从中采样一些小东西，在每种混合上训练小模型，并根据您的评估来衡量它们的性能。
And you can just do basic linear regression, and it’s like, “Here’s your optimal dataset.” But if your evaluations change, your dataset changes a lot.
你可以只进行基本的线性回归，就像“这是你的最佳数据集”。但如果你的评估发生变化，你的数据集就会发生很大变化。
So a lot of OLMo 3 was adding new sources for reasoning to be better at math and code, and then you do this mixing procedure and it gives you the answer.
因此，许多 OLMo 3 都添加了新的推理源，以便更好地擅长数学和代码，然后您执行此混合过程，它会给您答案。
I think a lot of that’s happened at labs this year; there are new hot things, whether it’s coding environments or web navigation, and you just need to bring in new data and change your whole pre-training so that your post-training can work better.
我认为今年实验室里发生了很多这样的事情；有新的热门事物，无论是编码环境还是网络导航，你只需要引入新数据并改变你的整个预训练，以便你的训练后能够更好地发挥作用。
And that’s like the constant re-evolution and the re-determining of what they care about for their models.
这就像不断的重新进化和重新确定他们关心的模型的内容。
Are there fun anecdotes of what sources of data are particularly high quality that we wouldn’t expect?
是否有一些有趣的轶事表明哪些数据源的质量特别高，超出了我们的预期？
1:12:35 You mentioned Reddit sometimes can be a source.
你提到 Reddit 有时可以成为一个来源。
Reddit was very useful.
Reddit 非常有用。
1:12:45 I think that PDFs are definitely one.
我认为 PDF 绝对是其中之一。
Oh, especially arXiv.
哦，尤其是 arXiv。
Yeah, AI2 has run Semantic Scholar for a long time, which is a competitor to Google Scholar with a lot more features.
是的，AI2 已经运行 Semantic Scholar 很长时间了，它是 Google Scholar 的竞争对手，具有更多功能。
1:12:52 To do this, AI2 has found and scraped a lot of PDFs for openly accessible papers that might not be behind the closed paid garden of a certain publisher—truly open scientific PDFs.
为此，AI2 发现并抓取了大量可公开访问的论文 PDF，这些论文可能不在某个出版商的封闭付费花园后面——真正开放的科学 PDF。
If you sit on all of these and process them, you can get value out of it.
如果你坐在所有这些上并处理它们，你就可以从中获得价值。
I think a lot of that style of work has been done by the frontier labs much earlier.
我认为前沿实验室很早就已经完成了很多这种风格的工作。
You need to have a pretty skilled researcher that understands how things change models, and they bring it in and clean it; it’s a lot of labor.
你需要有一个非常熟练的研究人员，了解事物如何改变模型，并将其引入并清理；这是很多劳动。
1:13:34 I think at a lot of frontier labs, when they scale researchers, a lot more goes into data.
我认为在很多前沿实验室，当他们扩大研究人员规模时，会更多地投入到数据中。
If you join a frontier lab and you want to have impact, the best way to do it is just find new data that’s better.
如果您加入前沿实验室并且希望产生影响，最好的方法就是找到更好的新数据。
The fancy, glamorous algorithmic things, like figuring out how to make o1, is like the sexiest thought for a scientist.
奇特、迷人的算法事物，比如弄清楚如何制造 o1，就像是科学家最性感的想法。
It’s like, “Oh, I figured out how to scale RL.” There’s a group that did that, but I think most of the contributions are-
就像，“哦，我想出了如何扩展强化学习。”有一个小组做到了这一点，但我认为大部分贡献是-
On the dataset.
在数据集上。
…
……
1:13:58 “I’m gonna make the data better,” or, “I’m gonna make the infrastructure better so that everybody on my team can run experiments 5% faster.”
“我要让数据变得更好，”或者，“我要让基础设施变得更好，这样我团队中的每个人都可以将实验速度提高 5%。”
At the same time, I think it’s also one of the closest guarded secrets—what your training data is—for legal reasons.
同时，我认为出于法律原因，这也是最保守的秘密之一——你的训练数据是什么。
1:14:04 And so there’s also a lot of work that goes into hiding what your training data was, essentially, trying to get the model to not give away the sources because of those legal reasons.
因此，还需要做很多工作来隐藏训练数据，本质上是试图让模型不会因为这些法律原因而泄露数据源。
The other thing, to be complete, is that some people are trying to train on only licensed data, whereas Common Crawl is a scrape of the whole internet.
另一件事，完整地说，是有些人试图仅使用许可数据进行训练，而 Common Crawl 是整个互联网的一部分。
1:14:19 If I host multiple websites, I’m happy to have them train language models, but I’m not explicitly licensing what governs it.
如果我托管多个网站，我很高兴让它们训练语言模型，但我没有明确许可管理它的内容。
Therefore, Common Crawl is largely unlicensed, which means your consent really hasn’t been provided for how to use the data.
因此，Common Crawl 基本上是未经许可的，这意味着实际上尚未就如何使用数据提供您的同意。
There’s another idea where you can train language models only on data that has been licensed explicitly so that the kind of governing contract is provided.
还有另一种想法，您可以仅根据已明确许可的数据来训练语言模型，以便提供某种管理合同。
I’m not sure if Apertus is the copyright thing or the license thing.
我不确定 Apertus 是版权问题还是许可问题。
I know that the reason they did it was for an EU compliance thing, where they wanted to make sure that their model fit one of those checks.
我知道他们这样做的原因是为了遵守欧盟规定，他们想确保他们的模型符合其中一项检查。
Mm-hmm.
嗯嗯。
1:15:01 And on that note, there’s also the distinction between the licensing.
在这一点上，许可之间也存在区别。
Some people, like you said, just purchase the license.
有些人，就像你说的，只是购买许可证。
Let’s say they buy an Amazon Kindle book or a Manning book, and then use that in the training data; that is a gray zone because you paid for the content and you might want to train on it.
假设他们购买了一本 Amazon Kindle 书或一本 Manning 书，然后在训练数据中使用它；这是一个灰色地带，因为您为内容付费，并且您可能想对其进行培训。
But then there are also restrictions where even that shouldn’t be allowed.
但也存在一些限制，甚至不允许这样做。
That is where it gets a bit fuzzy.
这就是它变得有点模糊的地方。
1:15:28 And I think that is still a hot topic right now.
我认为这仍然是现在的热门话题。
Big companies like OpenAI approached private companies for their proprietary data, and private companies are becoming more and more protective of their data because they know, “Okay, this is going to be my moat in a few years.” And I do think that’s the interesting question.
像 OpenAI 这样的大公司会向私营公司寻求其专有数据，而私营公司也越来越保护自己的数据，因为他们知道，“好吧，几年后这将成为我的护城河。”我确实认为这是一个有趣的问题。
If LLMs become more commoditized, and a lot of people learn about LLMs, there will be a lot more people able to train them.
如果法学硕士变得更加商品化，并且很多人了解法学硕士，就会有更多的人能够培训他们。
Of course, there are infrastructure challenges.
当然，还有基础设施方面的挑战。
1:16:00 But if you think of big industries like pharmaceuticals, law, or finance, I do think they at some point will hire people from other frontier labs to build their in-house models on their proprietary data, which will be another unlock with pre-training that is currently not there.
但如果你想到像制药、法律或金融这样的大行业，我确实认为他们在某个时候会雇佣其他前沿实验室的人员在他们的专有数据上构建他们的内部模型，这将是目前尚不存在的预训练的另一个解锁。
Because even if you wanted to, you can’t get that data—you can’t get access to clinical trials most of the time and these types of things.
因为即使你想，你也无法获得这些数据——大多数时候你无法获得临床试验和此类信息。
So I do think scaling in that sense might still be pretty much alive if you look at domain-specific applications, because right now we are just looking at general-purpose LLMs like ChatGPT, Anthropic, and so forth.
因此，我确实认为，如果你关注特定领域的应用程序，那么这种意义上的扩展可能仍然存在，因为现在我们只关注通用的法学硕士，如 ChatGPT、Anthropic 等。
They are just general purpose.
它们只是通用目的。
They’re not even scratching the surface of what an LLM can do if it is really specifically trained and designed for a specific task.
如果法学硕士确实是针对特定任务经过专门培训和设计的，那么他们甚至没有触及法学硕士可以做什么的表面。
I think on the data thing, this is one of the things where, like, this happened in 2025 and we totally forget it: Anthropic lost in court and owed $1.5 billion to authors.
我认为就数据而言，这是发生在 2025 年的事情之一，我们完全忘记了：Anthropic 在法庭上败诉，欠作者 15 亿美元。
1:16:47 Anthropic, I think, bought thousands of books and scanned them and was cleared legally for that because they bought the books, and that is going through the system.
我认为，人性化购买了数千本书籍并对其进行了扫描，并获得了合法许可，因为他们购买了这些书籍，并且正在通过系统。
And then on the other side, they also torrented some books, and I think this torrenting was the path where the court said that they were then culpable to pay these billions of dollars to authors, which is just such a mind-boggling lawsuit that kind of just came and went.
另一方面，他们也下载了一些书籍，我认为这种下载是法院说他们有责任向作者支付数十亿美元的途径，这就是一场令人难以置信的诉讼，就这样来了又去。
Like, that is so much money- …
比如说，这么多钱——……
from the VC ecosystem.
来自风险投资生态系统。
These are court cases that will define the future of human civilization because it’s clear that data drives a lot of this, and there’s this very complicated human tension.
这些法庭案件将定义人类文明的未来，因为很明显，数据在很大程度上推动了这一切，而且人类之间存在着非常复杂的紧张关系。
1:17:22 I mean, you can empathize.
我的意思是，你可以感同身受。
You’re both authors.
你们都是作者。
And there’s some degree to which, I mean, you put your heart and soul and your sweat and tears into the writing that you do.
我的意思是，在某种程度上，你把你的心和灵魂、你的汗水和泪水投入到你所做的写作中。
It feels a little bit like theft for somebody to train on your data without giving you credit.
有人在不给你信用的情况下训练你的数据，感觉有点像盗窃。
And there are, like Nathan said, also two layers to it.
正如内森所说，它也有两层。
1:17:49 Someone might buy the book and then train on it, which could be argued fair or not fair, but then there are the straight-up companies who use pirated books where they’re not even compensating the author.
有人可能会购买这本书，然后对其进行培训，这可能会被认为公平或不公平，但也有一些直接的公司使用盗版书籍，甚至不向作者提供补偿。
That is, I think, where people got a bit angry about it specifically, I would say.
我认为，这就是人们对此感到有点愤怒的地方。
Yeah, but there has to be some kind of compensation scheme.
是的，但必须有某种补偿计划。
1:18:06 This is like moving towards something like Spotify streaming did originally for music.
这就像 Spotify 流媒体最初为音乐所做的那样。
You know, what does that compensation look like?
你知道补偿是什么样的吗？
You have to define those kinds of models.
您必须定义这些类型的模型。
You have to think through all of that.
你必须把这一切都想清楚。
One other thing I think people are generally curious about, I’d love to get your thoughts: as LLMs are used more and more, if you look at even arXiv or GitHub, more and more of the data is generated by LLMs.
我认为人们普遍好奇的另一件事是，我很想听听您的想法：随着 LLM 的使用越来越多，如果您查看 arXiv 或 GitHub，就会发现越来越多的数据是由 LLM 生成的。
What do you do in that kind of world?
在那样的世界里你会做什么？
How big of a problem is that?
这是一个多大的问题？
The largest problem is the infrastructure and systems, but from an AI point of view, it’s kind of inevitable.
最大的问题是基础设施和系统，但从人工智能的角度来看，这是不可避免的。
So it’s basically LLM-generated data that’s curated by humans essentially, right?
所以它基本上是法学硕士生成的数据，本质上是由人类管理的，对吗？
Yes, and I think that a lot of open source contributors are legitimately burning out.
是的，我认为许多开源贡献者正在合理地精疲力尽。
1:18:49 If you have a popular open source repo, somebody’s like, “Oh, I want to do open source AI.
如果你有一个流行的开源存储库，有人会说，“哦，我想做开源人工智能。
It’s good for my career,” and they just vibe code something and throw it in.
这对我的职业生涯有好处，”他们只是编码一些东西然后把它扔进去。
You might get more of this than I do.
你可能比我得到更多。
Yeah, so I actually have a case study here.
是的，所以我实际上在这里有一个案例研究。
1:19:05 I have a repository called mlxtend that I developed as a student, around 10 or 15 years ago, and it is a reasonably popular library still for certain algorithms, especially frequent data mining stuff.
我有一个名为 mlxtend 的存储库，是我在大约 10 或 15 年前作为学生开发的，对于某些算法（尤其是频繁的数据挖掘内容）来说，它仍然是一个相当流行的库。
There were recently two or three people who submitted a lot of PRs in a very short amount of time.
最近有两三个人在很短的时间内提交了大量的 PR。
I do think LLMs have been involved in submitting these PRs.
我确实认为法学硕士参与了这些 PR 的提交。
Me, as the maintainer, there are two things.
作为维护者，我有两件事。
First, I’m a bit overwhelmed; I don’t have time to read through it because, especially since it’s an older library, that is not a priority for me.
首先，我有点不知所措；我没有时间通读它，因为，特别是因为它是一个较旧的图书馆，这对我来说不是优先考虑的事情。
At the same time, I kind of also appreciate it because I think something people forget is it’s not just using the LLM.
同时，我也有点欣赏它，因为我认为人们忘记了这不仅仅是使用法学硕士。
1:19:46 There’s still a human layer that verifies something, and that is in a sense also how data is labeled, right?
仍然有一个人类层来验证某些东西，从某种意义上说，这也是数据的标记方式，对吗？
One of the most expensive things is getting labeled data for RLHF (Reinforcement Learning from Human Feedback) phases.
最昂贵的事情之一是获取 RLHF（人类反馈强化学习）阶段的标记数据。
This is kind of like that, where it goes through phases and then you actually get higher quality data out of it.
这有点像那样，它会经历几个阶段，然后你实际上会从中获得更高质量的数据。
So I don’t mind it, in a sense.
所以从某种意义上说，我不介意。
It can feel overwhelming, but I do think there is also value in it.
它可能会让人感到不知所措，但我确实认为它也有其价值。
It feels like there’s a fundamental difference between raw LLM-generated data and LLM-generated data with a human in the loop that does some kind of verification, even if that verification is a small percentage- …
感觉原始LLM生成的数据和LLM生成的数据之间存在根本区别，其中有人在循环中进行某种验证，即使该验证只占很小的比例......
1:20:11 of the lines of code.
代码行数。
I think this goes with anything where people think, “Oh, yeah.
我认为这与人们认为“哦，是的。
1:20:25 I can just use an LLM to learn about XYZ,” which is true.
我可以通过法学硕士来了解 XYZ，”这是事实。
You can, but there might be a person who is an expert who might have used an LLM to write specific code.
你可以，但可能有人是专家，他可能使用法学硕士来编写特定的代码。
There is this human work that went into it to make it nice and throwing out the not-so-nice parts to pre-digest it for you, and that saves you time.
人类的工作是让它变得更好，并扔掉不太好的部分来为你预先消化它，这可以节省你的时间。
And I think that’s the value-add where you have someone filtering things or even using the LLMs correctly.
我认为这就是有人过滤事物甚至正确使用法学硕士的增值之处。
I think this is still labor that you get for free.
我认为这仍然是你免费获得的劳动力。
For example, when you read a Substack article.
例如，当您阅读 Substack 文章时。
1:21:05 I could maybe ask an LLM to give me opinions on that, but I wouldn’t even know what to ask.
我也许可以请法学硕士给我对此的意见，但我什至不知道该问什么。
And I think there is still value in reading that article compared to me going to the LLM because you are the expert.
我认为与我去攻读法学硕士相比，阅读那篇文章仍然有价值，因为你是专家。
You select what knowledge is actually spot on and should be included, and you give me this executive summary.
您选择实际正确且应包含的知识，然后给我这份执行摘要。
This is a huge value-add because now I don’t have to waste three to five hours to go through this myself and maybe get some incorrect information.
这是一个巨大的附加值，因为现在我不必浪费三到五个小时自己完成这个过程，并且可能会得到一些不正确的信息。
And so I think that’s also where the future still is for writers, even though there are LLMs that can save you time.
所以我认为这也是作家的未来，尽管有法学硕士可以节省你的时间。
It’s kind of fascinating to actually watch—and I’m sure you guys do this, but for me to look at the difference between a summary and the original content.
实际观看起来很有趣——我相信你们都会这样做，但对我来说，要看看摘要和原始内容之间的区别。
1:21:43 Even if it’s a page-long summary of page-long content, it’s interesting to see how the LLM-based summary takes the edge off.
即使它是一页长内容的摘要，看看基于法学硕士的摘要如何发挥优势也是很有趣的。
What is the signal it removes from the thing?
它从物体上移除的信号是什么？
The voice is what I talk about a lot.
声音是我经常谈论的话题。
Voice?
嗓音？
1:22:09 Well, voice…
嗯，声音……
I would love to hear what you mean by voice, that’s really powerful, but sometimes there’s like literally insights.
我很想听听你所说的声音是什么意思，这真的很强大，但有时就像字面上的见解。
Like in removing an insight, you’re actually fundamentally changing the meaning of the thing.
就像删除洞察力一样，您实际上从根本上改变了事物的含义。
So I’m continuously disappointed by how bad LLMs are at really getting to the core insights, which is what a great summary does.
因此，我一直对法学硕士在真正获得核心见解方面的糟糕程度感到失望，而这正是优秀总结的作用。
Yet even if you use extensive, extremely elaborate prompts where I’m really trying to dig for the insights, it’s still not quite there which…
然而，即使你使用广泛、极其复杂的提示，我真的想挖掘见解，但它仍然不完全在那里……
I mean, that’s a whole deep philosophical question about what is human knowledge and wisdom and what does it mean to be insightful.
我的意思是，这是一个深刻的哲学问题，关于什么是人类知识和智慧，以及洞察力意味着什么。
But when you talk about the voice, what do you mean?
但是当你谈论声音时，你的意思是什么？
So when I write, I think a lot of what I’m trying to do is take what you think as a researcher, which is very raw.
所以当我写作时，我认为我想做的很多事情都是按照你作为一名研究人员的想法，这是非常原始的。
1:22:52 A researcher is trying to encapsulate an idea at the frontier of their understanding, and they’re trying to put what is a feeling into words.
研究人员试图将一个想法概括在他们理解的前沿，他们试图将一种感觉转化为语言。
And I think that in my writing, I try to do this, which makes it come across as raw but also high-information in a way that some people will get and some won’t.
我认为在我的写作中，我尝试这样做，这使得它给人一种原始但又富含信息的感觉，有些人会得到，有些人则不会。
And that’s kind of the nature of research.
这就是研究的本质。
And I think this is something that language models don’t do well.
我认为这是语言模型做得不好的地方。
Particularly, they’re all trained with this reinforcement learning from human feedback which is designed to take feedback from a lot of people and, in a way, average how the model behaves from this.
特别是，他们都接受了来自人类反馈的强化学习的训练，该学习旨在获取很多人的反馈，并在某种程度上平均模型的行为方式。
1:23:30 And I think that it’s going to be hard for a model to be very incisive when there’s that sort of filter in it.
我认为，当模型中有这种过滤器时，它很难变得非常深入。
This is a wonderful fundamental problem for researchers in RLHF: this provides so much utility in making the models better, but also the problem formulation has this knot in it that you can’t get past.
对于 RLHF 的研究人员来说，这是一个很棒的基本问题：这为使模型变得更好提供了很多实用性，但问题的表述也有一个无法克服的结。
These language models don’t have this prior in their deep expression that they’re trying to get at.
这些语言模型在其想要表达的深层表达中并没有先验知识。
I don’t think it’s impossible to do.
我不认为这是不可能做到的。
I think there are stories of models that really shock people.
我认为有些模特的故事确实让人震惊。
Like, I would love to have tried Bing Sydney—did that have more voice?
比如，我很想尝试一下 Bing Sydney——它有更多的声音吗？
Because it would so often go off the rails on people and affect…
因为它经常会让人们出轨并影响……
1:24:13 And what is historically, obviously, a scary way—like telling a reporter to leave his wife—is a crazy model to potentially put in general adoption.
显然，从历史上看，一种可怕的方式——比如告诉记者离开他的妻子——是一种可能被普遍采用的疯狂模式。
But that’s kind of the trade-off: is this RLHF process, in some ways, adding limitations?
但这是一种权衡：这个 RLHF 过程是否在某些方面增加了限制？
That’s a terrifying place to be as one of these frontier labs and companies because millions of people are using them.
作为这些前沿实验室和公司之一，这是一个可怕的地方，因为数百万人正在使用它们。
There was a lot of backlash last year with GPT-4o getting removed.
去年 GPT-4o 被删除引起了很多强烈反对。
1:24:35 I’ve personally never used the model, but I’ve talked to people at OpenAI who get emails from users that might be detecting subtle differences in the deployments in the middle of the night.
我个人从未使用过该模型，但我与 OpenAI 的人员交谈过，他们收到用户发来的电子邮件，这些电子邮件可能会在半夜检测到部署中的细微差异。
And they email them and say, “My friend is different.” They find these employees’ emails and send them things because they are so attached to what is a set of model weights and a configuration that is deployed to the users.
他们给他们发电子邮件说：“我的朋友与众不同。”他们找到这些员工的电子邮件并向他们发送内容，因为他们非常重视一组模型权重和部署给用户的配置。
We see this with TikTok.
我们在 TikTok 上看到了这一点。
I don’t use TikTok, but supposedly, in five minutes, the algorithm gets you.
我不使用 TikTok，但据说，五分钟内，算法就能识别你。
It’s locked in.
它被锁定了。
And those are language models doing recommendations.
这些是做推荐的语言模型。
1:25:15 Like, I think there are ways that you can do this with a language model where, within five minutes of chatting with it, the model just gets you.
就像，我认为你可以通过语言模型来做到这一点，在与它聊天的五分钟内，模型就能让你明白。
And that is something that people aren’t really ready for.
人们还没有真正做好准备。
I think that—don’t give that to kids.
我认为——不要把它给孩子们。
Don’t give that to kids- at least until we know what’s happening.
不要把这种东西给孩子们——至少在我们知道发生了什么之前。
But there’s also going to be this mechanism…
但也会有这个机制……
1:25:30 What’s going to happen with these LLMs as they’re used more and more…
随着这些法学硕士的使用越来越多，会发生什么……
Unfortunately, the nature of the human condition is such that people commit suicide.
不幸的是，人类状况的本质就是人们自杀。
And what journalists will do is report extensively on the people who commit suicide, and they will very likely link it to the LLMs because they have that data about the conversations.
记者要做的就是广泛报道自杀者，他们很可能会将其与法学硕士联系起来，因为他们拥有有关对话的数据。
If you’re really struggling, if you’re depressed, if you’re thinking about suicide, you’re going probably to talk to LLMs about it.
如果你真的很挣扎，如果你很沮丧，如果你正在考虑自杀，你可能会和法学硕士谈论这件事。
And so what journalists will do is say, “The suicide was committed because of the LLM.” And that’s going to lead to the companies, because of legal issues and so on, more and more taking the edge off of the LLM.
所以记者会说，“自杀是因为法学硕士。”由于法律问题等原因，这将导致公司越来越多地削弱法学硕士的优势。
1:26:13 So it’s going to be as generic as possible.
所以它将尽可能通用。
It’s so difficult to operate in this space because, of course, you don’t want an LLM to cause harm to humans at that level, but also, this is the nature of the human experience—to have a rich conversation, a fulfilling conversation, one that challenges you and from which you grow.
在这个领域开展工作非常困难，因为当然，你不希望法学硕士对那个级别的人类造成伤害，而且，这也是人类经历的本质——进行丰富的对话，一次令人满意的对话，一次挑战你并从中成长的对话。
You need that edge.
你需要那种优势。
And that’s something extremely difficult for AI researchers on the RLHF front to actually have to solve because you’re actually dealing with the human condition.
对于 RLHF 前沿的人工智能研究人员来说，这是极其困难的事情，因为你实际上是在处理人类的状况。
A lot of researchers at these companies are so well-motivated.
这些公司的许多研究人员都非常积极主动。
1:26:47 Anthropic and OpenAI are culturally so wanting to do good for the world through this.
Anthropic 和 OpenAI 在文化上都希望通过这一点为世界做好事。
And it’s such a…
这是这样一个……
I’m like, “Ooh, I don’t want to work on this,” because, on the one hand, a lot of people see AI as a health ally, as somebody they can talk to about their health confidentially, but then it bleeds all the way into talking about mental health.
我想，“哦，我不想从事这个工作”，因为，一方面，很多人将人工智能视为健康盟友，作为他们可以秘密谈论自己健康的人，但随后它就一路延伸到谈论心理健康。
It’s heartbreaking that this might be the thing where somebody goes over the edge, but other people might be saved.
令人心碎的是，这可能是某人超出极限的事情，但其他人可能会得救。
And there’s things that as a researcher training models, it’s like, I don’t want to train image generation models and release them openly because I don’t want to enable somebody to have a tool on their laptop that can harm other people.
作为训练模型的研究人员，我不想训练图像生成模型并公开发布它们，因为我不想让某人在笔记本电脑上拥有可能伤害其他人的工具。
1:27:34 I don’t have the infrastructure in my company to do that safely.
我的公司没有足够的基础设施来安全地做到这一点。
There are a lot of areas like this where it needs people who will approach it with complexity and the conviction that it’s just such a hard problem.
有很多这样的领域，需要人们以复杂的态度来处理它，并坚信这是一个难题。
But also, we as a society and as users of these technologies need to make sure that we’re having the complicated conversation about it versus just fearmongering that big tech is causing harm to humans or stealing your data.
而且，作为一个社会和这些技术的用户，我们需要确保我们对此进行复杂的对话，而不是仅仅散布大型技术正在对人类造成伤害或窃取您的数据的恐慌。
1:27:47 It’s more complicated than that.
事情比那更复杂。
And you’re right, there’s a very large number of people inside these companies, many of whom you know and many of whom I know, that deeply care about helping people.
你是对的，这些公司里有很多人，其中很多你认识，我也认识，他们非常关心帮助别人。
They are considering the full human experience of people from across the world, not just Silicon Valley—what their needs are and what that means.
他们正在考虑来自世界各地的人们的完整人类体验，而不仅仅是硅谷——他们的需求是什么以及这意味着什么。
It’s really difficult to design this one system that is able to help all these different kinds of people across different age groups, cultures, and mental conditions.
设计一个能够帮助不同年龄层、文化和精神状况的不同类型的人的系统确实很困难。
I wish that the timing of AI was different regarding the relationship of big tech to the average person.
我希望就大型科技与普通人的关系而言，人工智能出现的时机有所不同。
1:28:31 Big tech’s reputation is so low, and because AI is so expensive, it’s inevitably going to be a big tech thing.
大型科技公司的声誉如此之低，而且由于人工智能如此昂贵，它不可避免地会成为一个大型科技公司。
It takes so many resources, and people say the US is, quote-unquote, “betting the economy on AI” with this build-out.
它需要如此多的资源，人们说美国通过这种扩建“将经济押在人工智能上”。
To have these be intertwined at the same time makes for such a hard communication environment.
这些因素同时交织在一起，造成了如此艰难的沟通环境。
It would be good for me to go talk to more people in the world who hate big tech and see AI as a continuation of that.
对我来说，与世界上更多讨厌大型科技并将人工智能视为其延续的人交谈是件好事。
One of the things you actually recommend, one of the antidotes that you talk about, is to find agency in this whole system, as opposed to sitting back in a powerless way and consuming the AI slop as it rapidly takes over the internet.
你实际上推荐的一件事，你谈论的解药之一，就是在整个系统中找到代理，而不是无能为力地袖手旁观，在人工智能迅速占领互联网时消耗它。
1:29:02 Find agency by using AI to build things—build apps, build…
通过使用人工智能来构建事物来寻找代理——构建应用程序、构建……
One, that actually helps you build intuition, but two, it’s empowering because you can understand how it works and what the weaknesses are.
一，这实际上可以帮助你建立直觉，但二，它可以赋予你力量，因为你可以理解它是如何工作的以及它的弱点是什么。
It gives your voice power to say, “This is bad use of the technology, and this is good use of technology.” You’re more plugged into the system then, so you can understand it better and steer it better as a consumer.
它让你有权利说：“这是对技术的错误使用，这是对技术的良好使用。”然后，您会更加融入系统，因此您可以作为消费者更好地理解它并更好地引导它。
I think that’s a good point you brought up about agency.
我认为你提出的关于代理的观点很好。
1:29:48 Instead of ignoring it and saying, “Okay, I’m not going to use it,” I think it’s probably long-term healthier to say, “Okay, it’s out there.
我认为，与其忽视它并说“好吧，我不会使用它”，不如说“好吧，它就在那里。”
I can’t put it back.” It’s like the internet and computers when they first came out.
我无法把它放回去。”就像互联网和计算机刚出现时一样。
How do I make the best use of it, and how does it help me up-level myself?
我如何充分利用它，它如何帮助我提升自己？
The one thing I worry about here, though, is if you just fully use it for something you love to do, the thing you love to do is no longer there.
不过，我在这里担心的一件事是，如果你只是将它完全用于你喜欢做的事情，那么你喜欢做的事情就不再存在了。
That could potentially lead to burnout.
这可能会导致倦怠。
For example, if I use an LLM to do all my coding for me, now there’s no coding; I’m just managing something that is coding for me.
例如，如果我使用法学硕士为我完成所有编码工作，那么现在就不需要编码了；我只是在管理一些为我编码的东西。
1:30:24 Two years later, let’s say, if I just do that eight hours a day—having something code for me—do I still feel fulfilled?
两年后，假设我每天只工作八小时——为自己编写一些代码——我还会感到满足吗？
Is this hurting me in terms of being excited about my job and what I’m doing?
这是否会伤害我对我的工作和正在做的事情的兴奋感？
Am I still proud to build something?
我仍然为建造一些东西而感到自豪吗？
On that topic of enjoyment, it’s quite interesting.
说到享受这个话题，还蛮有趣的。
1:30:43 We should just throw this in there, that there’s this recent survey of about 791 professional developers—professional meaning 10-plus years of experience.
我们应该把这个放在里面，最近有一项针对大约 791 名专业开发人员的调查——专业意味着 10 多年的经验。
That’s a long time.
那是很长一段时间了。
1:30:55 As a junior developer?
作为初级开发人员？
Yeah, in this day and age.
是的，在这个时代。
1:31:01 The results are surprising on many fronts.
结果在很多方面都令人惊讶。
They break it down by junior and senior developers, and it shows that both groups use AI-generated code in the code they ship.
他们将其按初级和高级开发人员进行了细分，结果表明这两个小组在他们发布的代码中都使用了人工智能生成的代码。
This is not just for fun or learning; this is code they ship.
这不仅仅是为了娱乐或学习；这是他们发布的代码。
Most of them use it for around 50% or more.
他们中的大多数人使用它的比例约为 50% 或更多。
What’s interesting is that for the category where over 50% of the shipped code is AI-generated, senior developers are much more likely to do so.
有趣的是，对于超过 50% 的交付代码是 AI 生成的类别，高级开发人员更有可能这样做。
But you don’t want AI to take away the thing you love.
但你不希望人工智能夺走你所爱的东西。
I think this speaks to my experience.
我想这符合我的经历。
These particular results show that about 80% of people find it either somewhat more enjoyable or significantly more enjoyable to use AI as part of their work.
这些具体结果表明，大约 80% 的人认为使用人工智能作为工作的一部分更加令人愉快或明显更加愉快。
I think it depends on the task.
我认为这取决于任务。
1:31:59 From my personal usage, for example, I have a website where I sometimes tweak things.
例如，从我个人的使用情况来看，我有一个网站，有时我会在其中进行一些调整。
I personally don’t enjoy this, so if the AI can help me implement something on my website, I’m all for it.
我个人不喜欢这个，所以如果人工智能可以帮助我在我的网站上实现一些东西，我会全力以赴。
It’s great.
这很棒。
But at the same time, when I solve a complex problem—if there’s a bug, and I hunt this bug and find it—it’s the best feeling in the world.
但与此同时，当我解决一个复杂的问题时——如果有一个错误，我寻找这个错误并找到它——这是世界上最好的感觉。
You get so much joy.
你会得到很多快乐。
But now, if you don’t even think about the bug and just go directly to the LLM, you never have that kind of feeling, right?
但现在，如果你连bug都不去想，直接去读LLM，你就永远不会有那种感觉了，对吧？
1:32:38 But then there could be a middle ground where you try it yourself, you can’t find it, you use the LLM, and then you don’t get frustrated because it helps you move on to something that you enjoy.
但是，可能会有一个中间立场，你自己尝试一下，找不到它，你使用法学硕士，然后你就不会感到沮丧，因为它可以帮助你继续从事你喜欢的事情。
Looking at these statistics, what is not factored in is that it’s averaging over all different scenarios.
查看这些统计数据，没有考虑到它是所有不同场景的平均值。
We don’t know if it’s for the core task or for something mundane that people would not have enjoyed otherwise.
我们不知道这是为了核心任务还是为了人们不会喜欢的平凡事物。
In a sense, AI is really great for doing mundane things that take a lot of work.
从某种意义上说，人工智能确实非常适合做需要大量工作的平凡事情。
1:33:09 For example, my wife has a podcast for book club discussions, and she was transferring the show notes from Spotify to YouTube, and the links somehow broke.
例如，我的妻子有一个用于读书俱乐部讨论的播客，她将节目笔记从 Spotify 传输到 YouTube，但链接不知何故中断了。
She had some episodes with 100 links or something, and it would have been really painful to go in there and fix each link manually.
她有一些剧集有 100 个链接之类的，进入那里手动修复每个链接真的很痛苦。
So I suggested, “Hey, let’s try ChatGPT.” We copied the text into ChatGPT, and it fixed them.
所以我建议，“嘿，我们试试 ChatGPT。”我们将文本复制到 ChatGPT 中，它修复了这些问题。
Instead of two hours going from link to link, it made that work seamless.
从一个链接到另一个链接不再需要花费两个小时，而是使工作变得无缝。
I think everyone has a use case where AI is useful for something like that—something that would be really boring and mundane.
我认为每个人都有一个用例，人工智能可以用于类似的事情——一些非常无聊和平凡的事情。
For me personally, since we’re talking about coding, a lot of the enjoyment comes from the cursor side—Claude Code side—where I have a pair programmer.
就我个人而言，由于我们正在谈论编码，所以很多乐趣都来自于光标一侧 - 克劳德代码一侧 - 我有一个结对程序员。
1:33:51 It’s less lonely.
这样就不那么孤独了。
You made debugging sound like this great joy.
你让调试听起来很有趣。
No, I would say debugging is like a drink of water after you’ve been going through a desert for— —for days.
不，我想说调试就像你在沙漠里呆了几天之后喝了一杯水。
You skip the whole desert part where you’re suffering.
你跳过了你正在受苦的整个沙漠部分。
Sometimes it’s nice to have a friend who can’t really find the bug, but can give you some intuition about the code, and together you go through the desert and find that drink of water.
有时，有一个朋友虽然不能真正找到错误，但可以给你一些关于代码的直觉，然后和你一起穿越沙漠，找到那杯水，这真是太好了。
For me, maybe it speaks to the loneliness of the programming experience.
对我来说，也许这说明了编程经历的孤独感。
That is a source of joy.
这是快乐的源泉。
It’s maybe also related to delayed gratification.
这可能也与延迟满足有关。
1:34:48 I’m a person who even as a kid liked the idea of Christmas presents better than actually getting them.
我是一个从小就喜欢圣诞礼物而不是真正收到圣诞礼物的人。
I would look forward to the day, but then it’s over and I’m disappointed.
我本来很期待这一天的到来，但当一切结束后，我感到很失望。
Maybe it’s like food—it tastes better when you’re really hungry.
也许这就像食物一样——当你真的饿了的时候味道会更好。
With debugging, it’s not always great; it’s often frustrating, but if you can solve it, then it’s great.
调试并不总是那么好；这常常令人沮丧，但如果你能解决它，那就太好了。
But there’s also a Goldilocks zone where if it’s too hard, then you’re wasting your time.
但也有一个“金发姑娘区”，如果它太难，那么你就是在浪费时间。
I think another challenge, though, is: how will people learn?
不过，我认为另一个挑战是：人们如何学习？
1:35:33 The chart we looked at showed that more senior developers are shipping AI-generated code than the junior ones.
我们查看的图表显示，与初级开发人员相比，更多的高级开发人员正在交付人工智能生成的代码。
I think it’s interesting because intuitively you would think it’s the junior developers because they don’t know how to do the thing yet.
我认为这很有趣，因为直觉上你会认为是初级开发人员，因为他们还不知道如何做这件事。
It could mean the AI is not good enough yet to solve those tasks, but it could also mean experts are more effective at using it—they know how to review the code and they trust it more.
这可能意味着人工智能还不够好，无法解决这些任务，但这也可能意味着专家在使用它方面更加有效——他们知道如何审查代码并且更加信任它。
One issue in society in the future will be: how do you become an expert if you never try to do the thing yourself?
未来社会的一个问题是：如果你不尝试自己做一件事，如何成为专家？
1:36:12 I learned by trying things myself.
我通过自己尝试来学习。
With math textbooks, if you look at the solutions, you learn something, but you learn better if you try first and then appreciate the solution because you know how to put it into your mental framework.
对于数学教科书，如果你查看解决方案，你会学到一些东西，但如果你先尝试然后欣赏解决方案，你会学得更好，因为你知道如何将其放入你的心理框架中。
If LLMs are here all the time, would you actually go through the length of struggling?
如果LLM一直在这里，你真的会经历漫长的挣扎吗？
Would you be willing to struggle?
你愿意奋斗吗？
Struggle is not nice, but if you use the LLM to do everything, at some point you will never really take the next step and you won’t get that unlock that you get as an expert using an LLM.
奋斗并不好，但如果你用法学硕士做所有的事情，在某些时候你将永远不会真正迈出下一步，你也无法获得使用法学硕士作为专家所获得的解锁。
1:36:53 So, I think there’s a Goldilocks sweet spot where maybe the trick is you make dedicated offline time where you study two hours a day, and the rest of the day you use LLMs.
所以，我认为有一个“金发姑娘”的最佳点，也许诀窍是你专门安排线下时间，每天学习两个小时，剩下的时间则使用法学硕士。
I think it’s important for people to still invest in themselves, in my opinion, and not just LLM everything.
我认为，在我看来，人们仍然要对自己进行投资，而不仅仅是法学硕士的一切，这一点很重要。
Yeah, there is a sense that we, together as a civilization, each individually have to find that Goldilocks zone.
是的，有一种感觉，我们作为一个文明，每个人都必须找到那个金发姑娘区。
1:37:10 And in the programming context as developers.
作为开发人员在编程环境中。
Now, we’ve had this fascinating conversation that started with pre-training and mid-training.
现在，我们从训练前和训练中开始进行了这场有趣的对话。
Let’s get to post-training.
让我们开始后期培训。
There’s a lot of fun stuff in post-training.
训练后有很多有趣的事情。
So, what are some of the interesting ideas in post-training?
那么，后期培训中有哪些有趣的想法呢？
The biggest one from 2025 is learning this reinforcement learning with verifiable rewards, RLVR.
2025 年最大的挑战是学习这种具有可验证奖励的强化学习，RLVR。
1:37:31 You can scale up the training there, which means doing a lot of this kind of iterative generate-grade loop, and that lets the models learn both interesting behaviors on the tool use and software side.
您可以在那里扩大训练规模，这意味着进行大量这种迭代生成级循环，这可以让模型学习工具使用和软件方面的有趣行为。
This could be searching, running commands on their own and seeing the outputs, and then also that training enables this inference-time scaling very nicely.
这可以是搜索、自行运行命令并查看输出，然后训练也可以很好地实现这种推理时间缩放。
It just turned out that this paradigm was very nicely linked, where this kind of RL training enables inference-time scaling.
事实证明，这种范式之间的联系非常紧密，这种 RL 训练可以实现推理时间缩放。
But inference-time scaling could have been found in different ways.
但推理时间的缩放可以通过不同的方式找到。
So, it was kind of this perfect storm where the models change a lot, and the way that they’re trained is a major factor in doing so.
所以，这是一场完美的风暴，模型发生了很大的变化，而它们的训练方式是其中的一个主要因素。
1:38:15 And this has changed how people approach post-training dramatically.
这极大地改变了人们接受培训后的方式。
Can you describe RLVR, popularized by DeepSeek R1?
您能描述一下由 DeepSeek R1 推广的 RLVR 吗？
1:38:20 Can you describe how it works?
你能描述一下它是如何工作的吗？
Yeah.
是的。
1:38:25 Fun fact, I was on the team that came up with the term RLVR, which is from our Tulu 3 work before DeepSeek.
有趣的是，我所在的团队提出了 RLVR 这个术语，它来自 DeepSeek 之前我们的 Tulu 3 工作。
We don’t take a lot of credit for being the people to popularize the scaling RL, but as much fun as academics get, as an aside, is the ability to name and influence—
我们并没有因为是推广强化强化学习的人而获得太多赞誉，但顺便说一句，与学术界一样有趣的是命名和影响的能力——
1:38:43 —the discourse, because the closed labs can only say so much.
——话语，因为封闭的实验室只能说这么多。
One of the things you can do as an academic is, while you might not have the compute to train the model, you can frame things in a way that ends up being…
作为一名学者，你可以做的一件事是，虽然你可能没有计算能力来训练模型，但你可以以一种最终成为……的方式构建事物。
I describe it as like a community can come together around this RLVR term, which is very fun.
我将其描述为一个社区可以围绕这个 RLVR 术语聚集在一起，这非常有趣。
And then DeepSeek are the people that did the training breakthrough, which is, they scaled the reinforcement learning.
DeepSeek 是在训练方面取得突破的人，也就是说，他们扩展了强化学习。
They have the model generate answers and then grade the completion if it was right, and then that accuracy is your reward for reinforcement learning.
他们让模型生成答案，然后对完成情况进行评分（如果答案正确），然后这种准确性就是对强化学习的奖励。
So reinforcement learning is classically an agent that acts in an environment, and the environment gives it a state and a reward back, and you try to maximize this reward.
因此，强化学习通常是一个在环境中行动的代理，环境给它一个状态和一个奖励，而你试图最大化这个奖励。
1:39:26 In the case of language models, the reward is normally accuracy on a set of verifiable tasks, whether it’s math problems or coding tasks.
就语言模型而言，奖励通常是一组可验证任务的准确性，无论是数学问题还是编码任务。
And it starts to get blurry with things like factual domains.
诸如事实领域之类的事情开始变得模糊。
That is also, in some ways, verifiable or constraints on your instruction, like ‘respond only with words that start with A.’ All of these things are verifiable in some way.
在某些方面，这也是对你的指示的可验证或限制，例如“仅用以 A 开头的单词进行响应”。所有这些事情都可以以某种方式验证。
The core idea is you find a lot more of these problems that are verifiable and you let the model try it many times while taking these RL gradient updates.
核心思想是，你会发现更多可验证的问题，并让模型在进行 RL 梯度更新时多次尝试。
The infrastructure evolved from reinforcement learning from human feedback, RLHF, where in that era, the score they were trying to optimize was a learned reward model of aggregate human preferences.
基础设施是从人类反馈的强化学习（RLHF）演变而来的，在那个时代，他们试图优化的分数是人类总体偏好的学习奖励模型。
1:40:13 So you kind of changed the problem domains and that let the optimization go on to much bigger scales, which kind of kickstarted a major change in what the models can do and how people use them.
所以你改变了问题领域，让优化进行到更大的范围，这引发了模型的功能和人们如何使用它们的重大变化。
What kind of domains is RLVR amenable to?
RLVR 适用于哪些领域？
Math and code are the famous ones, and then there’s a lot of work kind of on what is called the rubrics, which is related to a word people might have heard, LLM-as-a-judge.
数学和代码是最有名的，然后还有很多关于所谓的标准的工作，这与人们可能听说过的一个词有关，法学硕士作为法官。
1:40:28 For each problem, I’ll have a set of problems in my training dataset.
对于每个问题，我的训练数据集中都会有一组问题。
I will then have another language model and ask it, “What would a good answer to this problem look like?” And then you could try the problem a bunch of times over and over again and assign a score based on this rubric.
然后我将有另一种语言模型并询问它：“这个问题的一个好的答案是什么样的？”然后你可以一遍又一遍地尝试这个问题，并根据这个评分标准分配一个分数。
So that’s not necessarily verifiable like a math and code domain, but this rubrics idea and other scientific problems where it might be a little bit more vague is where a lot of the attention is.
因此，这不一定像数学和代码领域那样可验证，但这种标题概念和其他可能有点模糊的科学问题却引起了很多关注。
They’re trying to push this set of methods into these more open-ended domains so the models can learn a lot more.
他们试图将这组方法推入这些更加开放的领域，以便模型可以学到更多东西。
I think that’s called reinforcement learning with AI feedback, right?
我认为这就是所谓的带有人工智能反馈的强化学习，对吗？
That’s the older term from it that was coined in Anthropic’s Constitutional AI paper.
这是 Anthropic 的宪法人工智能论文中创造的旧术语。
1:41:14 So a lot of these things come in cycles.
所以很多事情都是循环出现的。
Also, just one step back for the RLVR.
另外，对于 RLVR 来说，只需退一步。
1:41:21 I think the interesting, beautiful thing here is that you ask the LLM a math question, you know the correct answer, and you let the LLM figure it out, but how it does it is…
我认为这里有趣、美妙的事情是，你问法学硕士一个数学问题，你知道正确的答案，然后你让法学硕士弄清楚，但它是如何做到的……
I mean, you don’t really constrain it much.
我的意思是，你并没有真正限制它太多。
There are some constraints you can add, like ‘use the same language’ or ‘don’t switch between Spanish and English.’ But let’s say you’re pretty much hands-off.
您可以添加一些限制，例如“使用相同的语言”或“不要在西班牙语和英语之间切换”。但假设您几乎不干涉。
1:41:44 You only give the question and the answer, and then the LLM has the task to arrive at the right answer.
你只需给出问题和答案，然后法学硕士的任务就是得出正确的答案。
But the beautiful thing here is what happens in practice: the LLM will do a step-by-step description, like how a student or a mathematician would derive the solution.
但这里的美妙之处在于实践中发生的事情：法学硕士将进行一步一步的描述，就像学生或数学家如何得出解决方案一样。
It will use those steps and that helps the model to improve its own accuracy.
它将使用这些步骤，帮助模型提高其自身的准确性。
And then, like you said, the inference scaling.
然后，就像你说的，推理缩放。
Inference scaling loosely means spending more compute while using the LLM during inference, and here the inference scaling is that the model would use more tokens.
松散的推理扩展意味着在推理过程中使用 LLM 时会花费更多的计算，这里的推理扩展是模型将使用更多的令牌。
In the DeepSeek R1 paper, they showed the longer they train the model, the longer the responses are.
在 DeepSeek R1 论文中，他们表明训练模型的时间越长，响应就越长。
1:42:28 They grow over time.
它们随着时间的推移而成长。
They use more tokens, so it becomes more expensive for simple tasks, but these explanations help the model with accuracy.
他们使用更多的令牌，因此对于简单的任务来说变得更加昂贵，但这些解释有助于模型的准确性。
There are also a lot of papers showing what the model explains does not necessarily have to be correct, or maybe it’s even unrelated to the answer, but for some reason, it still helps the model—the fact that it is explaining.
还有很多论文表明模型解释的内容不一定是正确的，甚至可能与答案无关，但出于某种原因，它仍然对模型有帮助——它正在解释的事实。
And I think it’s also—again, I don’t want to anthropomorphize these LLMs—but it’s kind of like how we humans operate, right?
我认为这也是——再说一次，我不想将这些法学硕士拟人化——但这有点像我们人类的运作方式，对吧？
If there’s a complex math problem in a math class, you usually have a note paper and you do it step by step.
如果数学课上有一个复杂的数学问题，你通常会有一张便条纸，然后一步步做。
You cross things out.
你把事情划掉。
1:43:03 And the model also self-corrects, and that was, I think, the aha moment in the DeepSeek R1 paper.
而且该模型还会自我修正，我认为这就是 DeepSeek R1 论文中的顿悟时刻。
They called it the ‘aha moment’ because the model itself recognized it made a mistake and then said, “Ah, I did something wrong, let me try again.” I think that’s just so cool that this falls out of just giving it the correct answer and having it figure out how to do it—that it kind of does, in a sense, what a human would do.
他们称之为“顿悟时刻”，因为模型本身意识到它犯了一个错误，然后说：“啊，我做错了，让我再试一次。”我认为这太酷了，只要给它正确的答案并让它弄清楚如何去做，从某种意义上说，它就可以做人类会做的事情。
Although LLMs don’t think like humans, it’s a kind of interesting coincidence.
虽然法学硕士不像人类那样思考，但这确实是一种有趣的巧合。
And the nice side effect is it’s great for us humans to see these steps.
好的副作用是，看到这些步骤对我们人类来说是件好事。
It builds trust, and we can learn or double-check things.
它建立了信任，我们可以学习或仔细检查事情。
There’s a lot in here.
这里有很多东西。
1:43:40 I think- There’s been a lot of debate this year on if the language models—I think these aha moments are kind of fake because in pre-training, you essentially have seen the whole internet.
我认为——今年关于语言模型是否存在很多争论——我认为这些顿悟时刻有点假，因为在预训练中，你基本上已经看到了整个互联网。
So you have definitely seen people explaining their work, even verbally, like a transcript of a math lecture: “You try this, oh, I messed this up.” And what reinforcement learning—this RLVR—is very good at doing, is amplifying— —these behaviors, because they’re very useful in enabling the model to think longer and to check its work.
所以你肯定见过人们解释他们的工作，甚至是口头解释，就像数学讲座的笔录：“你试试这个，哦，我搞砸了。”强化学习（即 RLVR）非常擅长做的事情就是放大这些行为，因为它们对于使模型能够思考更长时间并检查其工作非常有用。
I agree that it is very beautiful that this training kind of…
我同意这种训练非常美妙……
the model learns to amplify this in a way that is just so useful at the final answers being better.
该模型学会以一种非常有用的方式放大这一点，从而使最终答案变得更好。
I can give you also a hands-on example.
我还可以给你一个实际的例子。
1:44:16 I was training the Qwen 3 base model with RLVR on MATH-500.
我正在 MATH-500 上使用 RLVR 训练 Qwen 3 基础模型。
The base model had an accuracy of about 15%.
基本模型的准确度约为 15%。
Just 50 steps, like in a few minutes with RLVR, the model went from 15% to 50% accuracy.
只需 50 个步骤，就像 RLVR 只需要几分钟，模型的准确率就从 15% 提高到 50%。
And you can’t tell me it’s learning anything fundamentally about math in—
你不能告诉我它正在学习任何关于数学的基本知识——
The Qwen example is weird because there’s been two papers this year, one of which I was on, that talks about data contamination in Qwen— —and specifically that they train on a lot of this special mid-training phase that we— —can chime in on for a minute because it’s weird— —because they train on problems that are almost identical to MATH.
Qwen 的例子很奇怪，因为今年有两篇论文，其中一篇是我参与的，讨论了 Qwen 中的数据污染——特别是他们在很多特殊的中间训练阶段进行训练，我们——可以插话一分钟，因为这很奇怪——因为他们训练的问题几乎与数学相同。
Exactly.
确切地。
1:44:53 And so you can see that basically the RL is not teaching the model any new knowledge about math.
所以你可以看到，基本上强化学习并没有向模型传授任何关于数学的新知识。
You can’t do that in 50 steps.
你不可能只用 50 个步骤就能做到这一点。
So the knowledge is already there in the pre-training; you’re just unlocking it.
所以知识在预训练中就已经有了；你只是解锁它。
I still disagree with the premise because there’s a lot of weird complexities that you can’t prove.
我仍然不同意这个前提，因为有很多奇怪的复杂性你无法证明。
1:45:03 One of the things that points to weirdness is that if you take the Qwen 3 so-called base model—you could Google “math dataset Hugging Face” and take a problem—if you put it into Qwen 3 base…
奇怪的事情之一是，如果你采用 Qwen 3 所谓的基础模型——你可以谷歌“数学数据集 Hugging Face”并解决一个问题——如果你把它放入 Qwen 3 基础......
all these math problems have words, so it would be like, “Alice has five apples and gives three to whoever,” and there are these word problems.
所有这些数学问题都有文字，所以就像“爱丽丝有五个苹果，给任何人三个”，并且有这些文字问题。
With these Qwen-based models, why people are suspicious of them is if you change the numbers but keep the words— —Qwen will produce, without tools, a very high accuracy decimal representation—
对于这些基于 Qwen 的模型，人们之所以怀疑它们，是因为如果你改变数字但保留文字——Qwen 将在没有工具的情况下产生非常高精度的十进制表示——
1:45:43 —of the answer, which means at some point it was shown problems that were almost identical to the test set, and it was using tools to get a very high precision answer.
- 答案，这意味着在某些时候它会显示与测试集几乎相同的问题，并且它正在使用工具来获得非常高精度的答案。
But a language model without tools will never actually have this.
但没有工具的语言模型永远不会真正拥有这个。
So it’s been this big debate in the research community: how much of these reinforcement learning papers that are training on Qwen and measuring specifically on this math benchmark—where there’s been multiple papers talking about contamination—how much can you believe them?
因此，研究界存在着一场大争论：这些强化学习论文中有多少是在 Qwen 上进行训练并专门针对这个数学基准进行测量的（有多篇论文讨论了污染），你能相信它们多少？
I think this is what caused the reputation of RLVR being about formatting, because you can get these gains so quickly and therefore it must already be in the model.
我认为这就是 RLVR 因格式化而享有盛誉的原因，因为你可以如此快速地获得这些收益，因此它一定已经存在于模型中。
But there’s a lot of complexity here.
但这里有很多复杂性。
It’s not really like controlled experimentation— —so we don’t really know.
这并不像受控实验——所以我们真的不知道。
But if it weren’t true, I would say distillation wouldn’t work, right?
但如果不是真的，我会说蒸馏不起作用，对吗？
1:46:26 Distillation can work to some extent, but the biggest problem—and I’m researching this contamination—is we don’t know what’s in the data.
蒸馏在某种程度上可以发挥作用，但最大的问题——我正在研究这种污染——是我们不知道数据中有什么。
Unless you have a new dataset, it is really impossible.
除非你有新的数据集，否则这真的是不可能的。
Even something simpler like MMLU, which is a multiple-choice benchmark—if you just change the format slightly, like using a dot instead of a parenthesis, the model accuracy will vastly differ.
即使是像 MMLU 这样简单的东西，它是一个多项选择基准，如果你只是稍微改变格式，比如使用点而不是括号，模型的准确性将会有很大的不同。
I think that that could be like a model issue rather than a general issue.
我认为这可能是一个模型问题，而不是一个普遍问题。
It’s not even malicious by the developers of the LLM, like, “Hey, we want to cheat at that benchmark.” It’s just it has seen something at some point.
法学硕士的开发者甚至没有恶意，比如，“嘿，我们想在那个基准上作弊。”只是它在某个时刻看到了一些东西。
1:47:09 I think the only fair way to evaluate an LLM is to have a new benchmark that is after the cutoff date when the model was deployed.
我认为评估法学硕士的唯一公平方法是在模型部署的截止日期之后制定一个新的基准。
Can we lay out what would be the recipe of all the things that go into post-training?
我们能否列出培训后所有事情的秘诀是什么？
1:47:22 And you mentioned RLVR was a really exciting, effective thing.
您提到 RLVR 是一件非常令人兴奋且有效的事情。
Maybe we should elaborate.
也许我们应该详细说明。
RLHF still has a really important component to play.
RLHF 仍然有一个非常重要的组成部分需要发挥。
What kind of other ideas are there on post-training?
对于后期培训还有哪些其他的想法？
I think you can take this in order.
我想你可以按顺序处理这个问题。
1:47:40 You could view it as what made o1, which is this first reasoning model, possible.
您可以将其视为使 o1（第一个推理模型）成为可能的原因。
You’re going to have similar interventions where you start with mid-training.
从训练中期开始，您将进行类似的干预。
The thing that is rumored to enable o1 and similar models is really careful data curation where you’re providing a broad set of what is called reasoning traces.
据传，支持 o1 和类似模型的实际上是仔细的数据管理，其中提供了广泛的所谓推理轨迹。
This is just the model generating words in a forward process that reflects breaking down a problem into intermediate steps and trying to solve them.
这只是在正向过程中生成单词的模型，反映了将问题分解为中间步骤并尝试解决它们。
So at mid-training, you need to have data similar to this so that when you move into post-training, primarily with these verifiable rewards, it can learn.
因此，在训练中期，您需要拥有与此类似的数据，以便当您进入训练后（主要是通过这些可验证的奖励）时，它可以学习。
1:48:27 And then what is happening today is you’re figuring out which problems to give the model, how long you can train it for, and how much inference you can enable the model to use when solving these verifiable problems.
然后今天发生的事情是你要弄清楚要给模型提供哪些问题，你可以训练它多长时间，以及在解决这些可验证的问题时你可以让模型使用多少推理。
As models get better, certain problems are no longer useful; the model will solve them 100% of the time, and therefore there’s very little signal.
随着模型变得更好，某些问题不再有用；该模型将 100% 解决这些问题，因此信号非常少。
If we look at the GRPO equation, this one is famous for this because essentially the reward given to the agent is based on how good a given action—a completion—is relative to the other answers to that same problem.
如果我们看一下 GRPO 方程，就会发现这个方程因此而闻名，因为本质上给予代理的奖励是基于给定动作（完成）相对于同一问题的其他答案的好坏程度。
So if all the problems get the same answer, there’s no signal in these types of algorithms.
因此，如果所有问题都得到相同的答案，那么这些类型的算法就没有信号。
1:49:09 So what they’re doing is finding harder problems, which is why you hear about things like scientific domains, which are so hard to get anything right in.
所以他们正在做的是寻找更难的问题，这就是为什么你会听到科学领域之类的东西，这些领域很难得到任何正确的东西。
If you have a lab or something, it just generates so many tokens, or much harder software problems.
如果你有一个实验室或其他什么东西，它只会生成很多代币，或者更难的软件问题。
The frontier models are all pushing into these harder domains where they can train on more problems and the model will learn more skills at once.
前沿模型都在进军这些更难的领域，在那里它们可以训练更多的问题，并且模型将立即学习更多的技能。
The RLHF link to this is that RLHF has been, and still is, the finishing touch on the models, where it makes them more useful by improving the organization, style, or tone.
RLHF 与此的联系是，RLHF 过去是、现在仍然是模型的点睛之笔，它通过改进组织、风格或语气使模型变得更加有用。
1:49:42 There are different things that resonate with different audiences.
有不同的东西能引起不同的观众的共鸣。
Some people like a really quirky model, and RLHF could be good at enabling that personality, and some people hate the markdown bulleted list thing that the models do, but it’s actually really good for quickly parsing information.
有些人喜欢非常古怪的模型，而 RLHF 可能擅长实现这种个性，而有些人讨厌模型所做的 Markdown 项目符号列表，但它实际上非常适合快速解析信息。
This human feedback stage is really great for putting this into the model at the end of the day.
这个人类反馈阶段非常适合在一天结束时将其放入模型中。
It’s what made ChatGPT so magical for people.
这就是 ChatGPT 对人们如此神奇的原因。
And that use has actually remained fairly stable.
而且这种使用实际上保持相当稳定。
This formatting can also help the models get better at math problems, for example.
例如，这种格式还可以帮助模型更好地解决数学问题。
1:50:17 The border between style and formatting and the method that you use to answer a problem are actually very closely linked when you’re training these models.
当您训练这些模型时，样式和格式之间的界限以及您用来回答问题的方法实际上是紧密相连的。
RLHF can still make a model better at math, but these verifiable domains are a much more direct process for doing this because it makes more sense with the problem formulation.
RLHF 仍然可以使模型在数学方面变得更好，但这些可验证的领域是执行此操作的更直接的过程，因为它对问题表述更有意义。
To summarize: mid-training gives the model the skills it needs to learn; RL with verifiable rewards lets the model try many times, putting a lot of compute into trial-and-error learning across hard problems; and then RLHF finishes the model, making it easy to use and rounding it out.
总结一下：训练中期为模型提供了需要学习的技能；具有可验证奖励的强化学习可以让模型进行多次尝试，将大量计算投入到针对难题的试错学习中；然后RLHF完成模型，使其易于使用并完善。
Can you comment on the amount of compute required for RL VR?
您能否评论一下 RL VR 所需的计算量？
It’s only gone up and up.
它只是不断上升。
1:51:06 I think Grok 4 was famous for saying they use a similar amount of compute for pre-training and post-training.
我认为 Grok 4 因在预训练和训练后使用相似的计算量而闻名。
Back to the scaling discussion, they involve very different hardware for scaling.
回到扩展讨论，它们涉及非常不同的扩展硬件。
Pre-training is very compute-bound, which is like the FLOPS discussion: how many matrix multiplications can you get through in one time.
预训练非常受计算限制，这就像 FLOPS 讨论：一次可以完成多少次矩阵乘法。
Because in RL you’re generating these answers and trying the model in real-world environments, it ends up being much more memory-bound.
因为在强化学习中，你要生成这些答案并在现实环境中尝试该模型，所以它最终会更加受内存限制。
You’re generating long sequences, and the attention mechanisms have a behavior where you get a quadratic increase in memory as you get to longer sequences.
你正在生成长序列，注意力机制有一种行为，当你生成更长的序列时，你的记忆会呈二次方增加。
So the compute becomes very different.
所以计算变得非常不同。
1:51:44 In pre-training, we would talk about a model—if we go back to the Biden administration executive order—it’s like 10 to the 25th FLOPS to train a model.
在预训练中，我们会谈论一个模型——如果我们回到拜登政府的行政命令——训练一个模型就像 10 到 25 次 FLOPS。
If you’re using FLOPS in post-training, it’s a lot weirder because the reality is just how many hours you are allocating how many GPUs for.
如果你在后期训练中使用 FLOPS，那就更奇怪了，因为现实就是你分配了多少个 GPU 几个小时。
In terms of time, the RL compute is getting much closer because you just can’t put it all into one system.
就时间而言，强化学习计算越来越接近，因为你无法将所有内容都放入一个系统中。
Pre-training is so computationally dense where all the GPUs are talking to each other and it’s extremely efficient, whereas RL has all these moving parts and it can take a long time to generate a sequence of a hundred thousand tokens.
预训练的计算量非常大，所有 GPU 都在相互通信，而且效率极高，而 RL 拥有所有这些移动部件，可能需要很长时间才能生成包含 10 万个标记的序列。
1:52:17 If you think about Gemini 3 Pro taking an hour, what if your training run has to sample for an hour?
如果您认为 Gemini 3 Pro 需要一个小时，那么如果您的训练运行必须采样一个小时怎么办？
You have to make sure that’s handled efficiently.
您必须确保有效处理。
So in GPU hours or wall-clock hours, the RL runs are probably approaching the same number of days as pre-training, but they probably aren’t using as many GPUs at the same time.
因此，在 GPU 时间或挂钟时间中，强化学习运行的天数可能接近与预训练相同的天数，但它们可能不会同时使用那么多 GPU。
There are rules of thumb in labs where you don’t want your pre-training runs to last more than a month because they fail catastrophically.
实验室有一些经验法则，您不希望预训练持续超过一个月，因为它们会发生灾难性的失败。
If you are planning a huge cluster to be held for two months and then it fails on day 50, the opportunity costs are just so big.
如果您计划将一个大型集群保留两个月，然后在第 50 天失败，那么机会成本就非常大了。
1:52:54 People don’t want to put all their eggs in one basket.
人们不想把所有鸡蛋放在一个篮子里。
GPT-4 was like the ultimate YOLO run, and nobody ever wanted to do it before where it took three months to train and everybody was shocked that it worked.
GPT-4 就像终极的 YOLO 跑步，以前没有人愿意这样做，需要三个月的时间来训练，每个人都对它的效果感到震惊。
I think people are a little bit more cautious and incremental now.
我认为人们现在更加谨慎和渐进。
So RLVR is more unlimited in how much you can train or still get benefit, whereas RLHF, because it’s preference tuning, reaches a certain point where it doesn’t really make sense to spend more budget on it.
因此，RLVR 在您可以训练多少或仍然获得收益方面更加不受限制，而 RLHF，因为它是偏好调整，达到了某个点，在它上面花费更多预算实际上没有意义。
1:53:07 To take a step back with preference tuning: there are multiple people that can give multiple explanations for the same thing and they can both be correct, but at some point, you learn a certain style and it doesn’t make sense to iterate on it.
退后一步进行偏好调整：有多个人可以对同一件事给出多种解释，而且他们都可能是正确的，但在某些时候，你学习了某种风格，迭代它是没有意义的。
My favorite example is if relatives ask me what laptop they should buy.
我最喜欢的例子是，如果亲戚问我他们应该买什么笔记本电脑。
I give them an explanation or ask about their use case, and they might prioritize battery life and storage.
我给他们解释或询问他们的用例，他们可能会优先考虑电池寿命和存储。
1:53:46 Other people, like us, would prioritize RAM and compute.
其他人，比如我们，会优先考虑 RAM 和计算。
Both answers are correct, but different people require different answers.
两个答案都是正确的，但不同的人需要不同的答案。
With preference tuning, you are trying to average somehow; you are asking the data labelers to give you the preferred answer and then you train on that.
通过偏好调整，您试图以某种方式进行平均；你要求数据标记者给你首选的答案，然后你以此为基础进行训练。
But at some point, you learn that average preferred answer, and there’s no reason to keep training longer on it because it’s just a style.
但在某些时候，你会学到平均首选答案，并且没有理由继续训练更长时间，因为它只是一种风格。
With RLVR, you let the model solve more and more complex, difficult problems.
通过 RLVR，您可以让模型解决越来越复杂、困难的问题。
So I think it makes more sense to allocate more budget long-term to RLVR.
因此，我认为长期向 RLVR 分配更多预算更有意义。
1:54:27 Right now, we are in an RLVR 1.0 phase where it’s still that simple thing where we have a question and answer, but we don’t do anything with the stuff in between.
现在，我们处于 RLVR 1.0 阶段，仍然是一个简单的事情，我们有一个问题和答案，但我们不会对中间的东西做任何事情。
There were multiple research papers, by Google for example, on process reward models that also give scores for the explanation—how correct is the explanation?
例如，谷歌发表了多篇关于过程奖励模型的研究论文，这些论文也为解释给出了分数——解释的正确性如何？
I think that will be the next thing, let’s say RLVR 2.0 for this year, focusing on the steps between question and answer and how to leverage that information to improve the explanation and accuracy.
我认为这将是下一件事，比如说今年的 RLVR 2.0，重点关注问答之间的步骤以及如何利用这些信息来提高解释和准确性。
That’s one angle.
这是一个角度。
And there was a DeepSeek-V3.2 paper where they also had interesting inference scaling.
还有一篇 DeepSeek-V3.2 论文，其中也有有趣的推理缩放。
1:55:11 Well, first they had developed models that grade themselves as a separate model.
嗯，首先他们开发了模型，将自己作为一个单独的模型进行评分。
I think that will be one aspect.
我认为这将是一方面。
And the other, like Nathan mentioned, will be RLVR branching into other domains.
另一个，就像 Nathan 提到的，RLVR 将分支到其他领域。
The place where people are excited is value functions— —which is pretty similar.
人们兴奋的地方是价值函数——这非常相似。
1:55:23 Process reward models assign how good something is to each intermediate step in a reasoning process, whereas value functions apply value to every token the language model generates.
过程奖励模型将事物的好坏分配给推理过程中的每个中间步骤，而价值函数将价值应用于语言模型生成的每个标记。
Both of these have been largely unproven in the language modeling and reasoning model era.
这两者在语言建模和推理模型时代基本上都未经证实。
People are more optimistic about value functions for whatever reason now.
无论出于何种原因，现在人们对价值函数更加乐观。
I think process reward models were tried a lot more in the pre-o1 era, and a lot of people had headaches with them.
我认为过程奖励模型在 o1 时代之前被尝试过很多次，很多人对此感到头疼。
Value models have a very deep history in reinforcement learning.
价值模型在强化学习领域有着悠久的历史。
1:56:06 They’re one of the first things that were core to deep reinforcement learning existing—training value models.
它们是现有深度强化学习的核心核心之一——训练价值模型。
So right now the literature shows people are excited about trying value models, but there’s very little proof in it.
因此，目前的文献表明人们对尝试价值模型感到兴奋，但缺乏证据。
And there are negative examples in trying to scale up process reward models.
在尝试扩大流程奖励模型方面也存在一些负面例子。
1:56:22 These things don’t always hold in the future.
这些事情并不总是在未来成立。
To summarize the scaling: you don’t want to do too much RLHF because of how the signal scales.
总结一下缩放：由于信号缩放的方式，您不想做太多 RLHF。
People have worked on RLHF for years, especially after ChatGPT, but the first release of a reasoning model trained with RLVR, OpenAI’s o1, had a scaling plot where if you increase the training compute logarithmically, you get a linear increase in evaluations.
人们已经在 RLHF 上工作了很多年，尤其是在 ChatGPT 之后，但是使用 RLVR 训练的推理模型的第一个版本（OpenAI 的 o1）有一个缩放图，如果您以对数方式增加训练计算，您会得到评估的线性增加。
This has been reproduced multiple times; I think DeepSeek had a plot like this.
此内容已被转载多次；我认为 DeepSeek 有这样的情节。
But there’s no scaling law for RLHF where if you log-increase the compute, you get linear performance.
但 RLHF 没有缩放定律，如果对数增加计算量，就会获得线性性能。
1:57:02 In fact, the seminal scaling paper for RLHF is about scaling loss for reward model over-optimization.
事实上，RLHF 的开创性缩放论文是关于奖励模型过度优化的缩放损失。
That’s a big line to draw with RLVR and the methods we have now; they will follow this scaling paradigm where you can let the best runs go for an extra 10x and you get performance, but you can’t do this with RLHF.
这是 RLVR 和我们现在拥有的方法所划定的一条大线；他们将遵循这种扩展范例，您可以让最佳运行额外增加 10 倍，从而获得性能，但 RLHF 无法做到这一点。
That is going to be field-defining.
这将是领域定义。
To do the best RLHF you might not need the extra 10 or 100x compute, but to do the best RLVR you do.
要实现最佳 RLHF，您可能不需要额外的 10 倍或 100 倍计算，但要实现最佳 RLVR，则需要。
There’s a seminal paper from a Meta internship called “The Art of Scaling Reinforcement Learning with Language Models.”
Meta 实习有一篇开创性的论文，名为“使用语言模型扩展强化学习的艺术”。
1:57:47 Their framework is called ScaleRL.
他们的框架称为 ScaleRL。
Their incremental experiment was like 10,000 V100 hours, which is thousands or tens of thousands of dollars per experiment, and they do a lot of them.
他们的增量实验大约是 10,000 V100 小时，每个实验要花费数千或数万美元，而且他们做了很多次实验。
This cost is not accessible to the average academic, which creates a hard equilibrium when trying to figure out how to learn from each community.
这种成本是普通学者无法承受的，这在试图弄清楚如何向每个社区学习时产生了一种硬平衡。
I was wondering if we could take a bit of a tangent and talk about education and learning.
我想知道我们是否可以岔开话题，谈谈教育和学习。
1:58:11 If you’re somebody listening to this who’s a smart person interested in programming and interested in AI, I presume building something from scratch is a good beginning.
如果你是一个对编程感兴趣并且对人工智能感兴趣的聪明人，我认为从头开始构建一些东西是一个好的开始。
Can you just take me through what you would recommend people do?
您能告诉我您建议人们做什么吗？
I would personally start, like you said, by implementing a simple model from scratch that you can run on your computer.
正如您所说，我个人会从头开始实现一个可以在计算机上运行的简单模型。
1:58:32 The goal of building a model from scratch is not to have something you use every day for your personal projects.
从头开始构建模型的目的不是为了拥有每天用于个人项目的东西。
It’s not going to be your personal assistant replacing an existing open-weight model or ChatGPT.
它不会成为您的私人助理来取代现有的开放重量模型或 ChatGPT。
It’s to see exactly what goes into the LLM, what exactly comes out of the LLM, and how pre-training works on your own computer.
这是为了了解法学硕士的具体内容、法学硕士的具体结果以及预培训如何在您自己的计算机上进行。
And then you learn about pre-training, supervised fine-tuning, and the attention mechanism.
然后你了解预训练、监督微调和注意力机制。
1:59:03 You get a solid understanding of how things work, but at some point you will reach a limit because smaller models can only do so much.
您对事物的工作原理有了深入的了解，但在某些时候您会达到极限，因为较小的模型只能做这么多。
The problem with learning about LLMs at scale is that it’s exponentially more complex to make a larger model because it’s not just that the model becomes larger.
大规模学习法学硕士的问题在于，制作更大的模型会变得更加复杂，因为不仅仅是模型变得更大。
You have to think about sharding your parameters across multiple GPUs.
您必须考虑将参数分片到多个 GPU 上。
Even for the KV cache, there are multiple ways you can implement it.
即使对于 KV 缓存，也有多种实现方式。
One is just to understand how it works, like a cache you grow step-by-step by concatenating lists, but then that wouldn’t be optimal on GPUs.
一是只是了解它是如何工作的，就像通过连接列表逐步增长的缓存一样，但这在 GPU 上并不是最佳的。
You would pre-allocate a tensor and then fill it in.
您可以预先分配一个张量，然后填充它。
But that adds another 20 or 30 lines of code.
但这又增加了 20 或 30 行代码。
1:59:45 And for each thing, you add so much code.
对于每件事，你都添加了很多代码。
I think the trick with the book is basically to understand how the LLM works.
我认为这本书的诀窍基本上是理解法学硕士是如何运作的。
It’s not going to be your production-level LLM, but once you have that, you can understand the production-level LLM.
它不会是你的生产级法学硕士，但一旦你拥有了它，你就可以理解生产级法学硕士。
So you’re trying to always build an LLM that’s going to fit on one GPU?
所以你总是试图建立一个适合一个 GPU 的法学硕士？
Yes.
是的。
2:00:00 Most of the examples I have fit on one GPU.
我的大多数示例都适合在一个 GPU 上。
I have some bonus materials on some MoE models; one or two of them may require multiple GPUs, but the goal is to have it on one GPU.
我有一些 MoE 模型的奖励材料；其中一两个可能需要多个 GPU，但目标是在一个 GPU 上实现。
And the beautiful thing is also you can self-verify.
最美妙的是你还可以自我验证。
It’s almost like RLVR.
这几乎就像 RLVR 一样。
When you code these from scratch, you can take an existing model from the Hugging Face Transformers library.
当您从头开始编写这些代码时，您可以从 Hugging Face Transformers 库中获取现有模型。
The Hugging Face Transformers library is great, but if you want to learn about LLMs, I think that’s not the best place to start because the code is so complex.
Hugging Face Transformers 库很棒，但如果您想了解 LLM，我认为这不是最好的起点，因为代码非常复杂。
It has to fit so many use cases and some people use it in production.
它必须适应如此多的用例，并且有些人在生产中使用它。
It has to be really sophisticated, so it’s intertwined and hard; it’s not linear to read.
它必须非常复杂，所以它是相互交织且困难的；阅读起来不是线性的。
It started as a fine-tuning library, and then it grew to be the standard representation of every model architecture and the way it is loaded.
它最初是一个微调库，然后发展成为每个模型架构及其加载方式的标准表示。
2:00:39 Hugging Face is the default place to get a model, and Transformers is the software that enables it— —so people can easily load a model— —and do something basic with it.
Hugging Face 是获取模型的默认位置，而 Transformers 是支持该模型的软件——因此人们可以轻松加载模型——并用它做一些基本的事情。
And all frontier labs that have open-weight models have a Hugging Face Transformers version of it, from DeepSeek to gpt-oss.
所有拥有开放权重模型的前沿实验室都有 Hugging Face Transformers 版本，从 DeepSeek 到 gpt-oss。
2:00:56 That’s the canonical way that you can load them.
这是加载它们的规范方式。
But again, even the Transformers library is not used in production for inference.
但同样，即使 Transformers 库也没有在生产中用于推理。
People use SGLang or vLLM, and it adds another layer of complexity.
人们使用 SGLang 或 vLLM，这又增加了一层复杂性。
We should say that the Transformers library has something like 400 models.
应该说 Transformers 库大约有 400 个模型。
So it’s the one library that tries to implement a lot of LLMs, and so you have a huge codebase.
因此，它是一个尝试实现大量 LLM 的库，因此您拥有庞大的代码库。
2:01:19 It’s massive.
这是巨大的。
It’s—I don’t know, maybe millions— —hundreds of thousands of lines of code.
这是——我不知道，也许有数百万——数十万行代码。
Understanding the part that you want to understand is like finding the needle in the haystack.
理解你想理解的部分就像大海捞针一样。
But what’s beautiful about it is you have a working implementation, so you can work backwards from it.
但它的美妙之处在于你有一个有效的实现，所以你可以从它开始向后工作。
What I would recommend doing is if I want to understand, for example, how OLMo 3 is implemented, I would look at the weights in the model hub and the config file.
我建议做的是，如果我想了解 OLMo 3 是如何实现的，我会查看模型中心和配置文件中的权重。
You can see, “Oh, they used so many layers.
你可以看到，“哦，他们用了很多层。
They use group query attention.” Then you see all the components in a human-readable 100-line config file.
他们使用群体查询注意力。”然后您会看到人类可读的 100 行配置文件中的所有组件。
And then you start with your GPT-2 model and add these things.
然后从 GPT-2 模型开始并添加这些内容。
2:02:06 The cool thing here is you can then load the pre-trained weights and see if they work in your model.
这里最酷的事情是您可以加载预先训练的权重并查看它们是否适用于您的模型。
You want to match the same output that you get with a Transformers model, and then you can use that basically as a verifiable reward to make your architecture correct.
您希望匹配使用 Transformers 模型获得的相同输出，然后您可以将其基本上用作可验证的奖励，以使您的架构正确。
Sometimes it takes me a day.
有时我需要一天的时间。
With OLMo 3, the challenge was RoPE for the position embeddings; they had a YaRN extension and there was some custom scaling there.
对于 OLMo 3，挑战在于位置嵌入的 RoPE；他们有一个 YaRN 扩展，并且有一些自定义缩放。
I couldn’t quite match it at first, but in this struggle you kind of understand things.
一开始我还不太适应，但在这场斗争中你会明白一些事情。
At the end, you know you have it correct because you can unit test it against the reference implementation.
最后，您知道它是正确的，因为您可以针对参考实现对其进行单元测试。
I think that’s one of the best ways to learn.
我认为这是最好的学习方式之一。
Basically, you reverse-engineer something.
基本上，你对某些东西进行逆向工程。
I think that is something everyone interested in getting into AI today should do, and that’s why I liked your book.
我认为这是今天每个有兴趣进入人工智能领域的人都应该做的事情，这就是我喜欢你的书的原因。
2:02:51 I came to language models from the RL and robotics field, so I never had taken the time to just learn all the fundamentals.
我是从强化学习和机器人领域接触到语言模型的，所以我从来没有花时间去学习所有的基础知识。
The Transformer architecture is as fundamental today as deep learning was in the past, and people need to learn it.
今天的 Transformer 架构就像过去的深度学习一样基础，人们需要学习它。
I think where a lot of people get overwhelmed is how to apply this to have an impact or find a career path.
我认为很多人不知所措的地方是如何应用它来产生影响或找到职业道路。
2:03:23 AI language models make this fundamental stuff so accessible, and people with motivation will learn it.
人工智能语言模型使这些基础知识变得如此容易理解，有动力的人会学习它。
Then it’s like, “How do I get the cycles on goal to contribute to research?” I’m actually fairly optimistic because the field moves so fast that a lot of times the best people don’t fully solve a problem because there’s a bigger, lower-hanging fruit to solve, so they move on.
然后就像，“我如何让周期达到目标，为研究做出贡献？”我实际上相当乐观，因为这个领域发展得如此之快，以至于很多时候最优秀的人都无法完全解决问题，因为有一个更大、更容易实现的目标需要解决，所以他们会继续前进。
In my RLHF book, I try to take post-training techniques and describe how they influence the model.
在我的 RLHF 书中，我尝试采用训练后技术并描述它们如何影响模型。
It’s remarkable how many things people just stop studying.
值得注意的是，有很多东西人们就停止了研究。
2:04:06 I think people trying to go narrow after doing the fundamentals is good.
我认为人们在完成基础知识后尝试缩小范围是好的。
Reading relevant papers and being engaged in the ecosystem—you actually…
阅读相关论文并参与生态系统——你实际上……
The proximity that random people have online to leading researchers is incredible.
随机的人在网上与领先研究人员的接近程度令人难以置信。
The anonymous accounts on X in ML are very popular, and no one knows who all these people are.
ML 中 X 上的匿名帐户非常流行，没有人知道这些人是谁。
It could just be random people who study this stuff deeply.
深入研究这些东西的人可能只是随机的。
Especially with AI tools to help you keep digging into things you don’t understand, it’s very useful.
尤其是用AI工具来帮助你不断挖掘你不理解的东西，这是非常有用的。
There are research areas that might only have three papers you need to read, and then one of the authors will probably email you back.
有些研究领域可能只需要阅读三篇论文，然后其中一位作者可能会给您回复电子邮件。
2:04:45 But you have to put in a lot of effort into these emails to show you understand the field.
但你必须在这些电子邮件中投入大量精力才能表明你了解该领域。
It would take a newcomer weeks of work to truly grasp a very narrow area, but going narrow after the fundamentals is very useful.
新手需要花费几周的时间才能真正掌握一个非常狭窄的领域，但是在基础知识之后缩小范围是非常有用的。
I became very interested in character training—how you make a model funny, sarcastic, or serious, and what you do to the data to achieve this.
我对角色训练非常感兴趣——如何让模型变得有趣、讽刺或严肃，以及如何处理数据来实现这一目标。
A student at Oxford reached out to me and said, “Hey, I’m interested in this,” and I advised him.
牛津大学的一名学生联系我说：“嘿，我对此很感兴趣，”我向他提出了建议。
Now that paper exists.
现在那篇论文已经存在了。
There were maybe only two or three people in the world very interested in that specific topic.
世界上可能只有两三个人对这个特定主题非常感兴趣。
2:05:25 He’s a PhD student, which gives you an advantage, but for me, that was a topic where I was waiting for someone to say, “Hey, I have time to spend cycles on this.” I’m sure there are a lot more narrow things where you’re just like, “It doesn’t make sense that there was no answer to this.” There’s so much information coming in that people feel they can’t grab onto anything, but if you actually stick to one area, I think there are a lot of interesting things to learn.
他是一名博士生，这给了你一个优势，但对我来说，这是一个我在等待有人说的话题，“嘿，我有时间花很多时间在这个上面。”我确信还有很多更狭隘的事情，你会说，“这个问题没有答案是没有意义的。”信息太多，人们感觉自己无法抓住任何东西，但如果你真正专注于一个领域，我认为有很多有趣的东西需要学习。
Yeah, I think you can’t try to do it all because it would be very overwhelming and you would burn out.
是的，我认为你不能尝试全部完成，因为这会让人不知所措，而且你会精疲力尽。
2:05:48 For example, I haven’t kept up with computer vision in a long time; I’ve just focused on LLMs.
比如，我已经很久没有跟上计算机视觉的步伐了；我只专注于法学硕士。
But coming back to your book, I think it’s a really great resource and a good bang for the buck if you want to learn about RLHF.
但回到你的书，我认为如果你想了解 RLHF，这是一个非常好的资源，而且物有所值。
I wouldn’t just go out there and read raw RLHF papers because you would be spending two years—
我不会只是去那里阅读原始的 RLHF 论文，因为你会花两年的时间——
—and some of them contradict each other.
——其中一些是相互矛盾的。
2:06:10 I’ve just edited the book, and there’s no chapter where I had to say, “X papers say one thing and Y papers say another, and we’ll see what comes out to be true.”
我刚刚编辑了这本书，没有一章需要我说：“X 篇论文说的是一件事，Y 篇论文说的是另一件事，我们拭目以待。”
What are some of the ideas we might have missed in the bigger picture of post-training?
在培训后的大局中，我们可能错过了哪些想法？
2:06:21 To go through the table of contents: first, you did the problem setup, training overview, what are preferences, preference data and the optimization tools, reward modeling, regularization, instruction tuning, rejection sampling, reinforcement learning.
浏览目录：首先，您进行了问题设置、训练概述、偏好是什么、偏好数据和优化工具、奖励建模、正则化、指令调整、拒绝采样、强化学习。
Then constitutional AI and AI feedback, reasoning and inference-time scaling, tool use and function calling, synthetic data and distillation, evaluation, and then the open questions section: over-optimization, style and information, product UX, character and post-training.
然后是人工智能和人工智能反馈、推理和推理时间缩放、工具使用和函数调用、合成数据和蒸馏、评估，然后是开放问题部分：过度优化、风格和信息、产品用户体验、角色和后期培训。
What are some ideas worth mentioning that connect both the educational component and the research component?
有哪些值得一提的连接教育部分和研究部分的想法？
You mentioned the character training, which is pretty interesting.
你提到了性格训练，这很有趣。
Character training is interesting because there’s so little out there, but we talked about how people engage with these models.
角色训练很有趣，因为那里的东西太少了，但我们讨论了人们如何与这些模型互动。
2:07:08 We feel good using them because they’re positive, but that can go too far; it can be too positive.
我们使用它们感觉很好，因为它们是积极的，但这可能太过分了；它可能过于积极。
It’s essentially how you change your data or decision-making to make it exactly what you want.
从本质上讲，这就是您如何更改数据或决策以使其完全符合您的要求。
OpenAI has this thing called a “model spec,” which is essentially their internal guideline for what they want the model to do, and they publish this to developers.
OpenAI 有一个叫做“模型规范”的东西，这本质上是他们希望模型做什么的内部指南，并将其发布给开发人员。
So you can know what is a failure of OpenAI’s training—where they have the intention but haven’t met it yet—versus what is something they actually wanted to do that you just don’t like.
因此，您可以知道 OpenAI 培训的失败是什么（他们有意图但尚未实现），以及什么是他们实际上想做而您只是不喜欢的事情。
2:07:46 That transparency is very nice, but all the methods for curating these documents and how easy it is to follow them is not very well known.
这种透明度非常好，但整理这些文档的所有方法以及遵循它们的难易程度还不是很为人所知。
I think the way the book is designed is that the reinforcement learning chapter is obviously what people want because everybody hears about it with RLVR, and it’s the same algorithms and the same math, but you can use it in very different documents.
我认为这本书的设计方式是，强化学习章节显然是人们想要的，因为每个人都通过 RLVR 听说过它，它是相同的算法和相同的数学，但你可以在非常不同的文档中使用它。
I think the core of RLHF is how messy preferences are.
我认为RLHF的核心是偏好有多混乱。
It’s essentially a rehash of a paper I wrote years ago, but this is the chapter that tells you why RLHF is never fully solvable, because the way that RL is set up assumes that preferences can be quantified and reduced to single values.
这本质上是我几年前写的一篇论文的翻版，但这一章告诉你为什么 RLHF 永远无法完全解决，因为 RL 的设置方式假设偏好可以量化并简化为单个值。
2:08:33 I think it relates in the economics literature to the Von Neumann-Morgenstern utility theorem.
我认为它在经济学文献中与冯·诺依曼-摩根斯坦效用定理有关。
That is the chapter where all of that philosophical, economic, and psychological context tells you what gets compressed when doing RLHF.
在这一章中，所有哲学、经济和心理学背景都会告诉您在进行 RLHF 时哪些内容会被压缩。
Later in the book, you use this RL map to make the number go up.
在本书的后面部分，您将使用这个 RL 映射来使数字上升。
I think that’s why it’ll be very rewarding for people to do research on, because quantifying preferences is something humans have designed the problem around to make them studyable.
我认为这就是为什么人们进行研究会非常有价值，因为量化偏好是人类设计问题以使它们变得可研究的东西。
But there are fundamental debates; for example, in a language model response, you have different things you care about, whether it’s accuracy or style.
但存在一些根本性的争论；例如，在语言模型响应中，您关心的事物有不同，无论是准确性还是风格。
2:09:13 When you’re collecting the data, they all get compressed into, “I like this more than another.” There’s a lot of research in other areas of the world that goes into how you should actually do this.
当你收集数据时，它们都会被压缩成“我比另一个更喜欢这个”。世界其他地区有很多研究探讨如何实际做到这一点。
I think social choice theory is the subfield of economics around how you should aggregate preferences.
我认为社会选择理论是经济学的一个子领域，围绕如何汇总偏好。
I went to a workshop that published a white paper on how you can think about using social choice theory for RLHF.
我参加了一个研讨会，该研讨会发表了一份白皮书，介绍如何考虑将社会选择理论用于 RLHF。
I want people who get excited about the math to stumble into this broader context.
我希望那些对数学感兴趣的人能够偶然进入这个更广泛的背景。
I also keep a list of all the tech reports of reasoning models that I like.
我还保留了我喜欢的推理模型的所有技术报告的列表。
In Chapter 14, where there’s a short summary of RLVR, there’s a gigantic table where I list every single reasoning model that I like.
第 14 章对 RLVR 进行了简短总结，并在一张巨大的表格中列出了我喜欢的每一个推理模型。
I think in education, a lot of it needs to be, at this point, what I like—
我认为在教育领域，目前很多内容都需要是我喜欢的——
2:10:08 —because language models are so good at the math.
——因为语言模型非常擅长数学。
For example, the famous paper on Direct Preference Optimization, which is a much simpler way of solving the problem than RL—the derivations in the appendix skip steps of math.
例如，关于直接偏好优化的著名论文，它是一种比强化学习简单得多的解决问题的方法——附录中的推导跳过了数学步骤。
I tried for this book to redo the derivations and I was like, “What the heck is this log trick that they use?” But when doing it with language models, they just say, “This is the log trick.” I don’t know if I like that the math is so commoditized.
我试图在这本书中重做推导，我想，“他们使用的这个日志技巧到底是什么？”但当使用语言模型时，他们只是说，“这是日志技巧。”我不知道我是否喜欢数学如此商品化。
I think some of the struggle in reading this appendix— —and following the math is good for learning.
我认为阅读本附录的一些困难以及遵循数学对于学习是有好处的。
Yeah, we’re returning to this often on the topic of education.
是的，我们经常回到教育这个话题。
2:10:43 You both have brought up the word “struggle” quite a bit.
你们都多次提到“斗争”这个词。
There is value in that.
这是有价值的。
If you’re not struggling as part of this process, you’re not fully following the proper process for learning, I suppose.
我想，如果你在这个过程中没有遇到困难，那么你就没有完全遵循正确的学习过程。
Some of the providers are starting to work on models for education designed to not give…
一些提供者开始研究旨在不提供……的教育模式。
2:11:02 actually, I haven’t used them, but I would guess they’re designed to not give all the information at once— —and make people work for it.
事实上，我没有使用过它们，但我猜它们的设计初衷是不会一次性提供所有信息，并让人们为之工作。
I think you could train models to do this and it would be a wonderful contribution.
我认为你可以训练模型来做到这一点，这将是一个很棒的贡献。
In the book, you had to reevaluate every decision— Which is such a great example.
在书中，你必须重新评估每一个决定——这是一个很好的例子。
I think there’s a chance we work on it at AI2, which I think would be so fun.
我认为我们有机会在 AI2 进行这方面的工作，我认为这会很有趣。
It makes sense.
这是有道理的。
2:11:26 I did something like that the other day for video games.
前几天我为电子游戏做了类似的事情。
In my spare time, I like video games with puzzles, like Zelda and Metroid.
在业余时间，我喜欢带有谜题的电子游戏，例如《塞尔达》和《银河战士》。
There’s this new game where I got really stuck.
有一款新游戏让我陷入了困境。
I didn’t want to struggle for two days, so I used an LLM.
我不想挣扎两天，所以我用了LLM。
But I told it, “Please don’t add any spoilers.
但我告诉它：“请不要添加任何剧透。
I’m at this point; what do I have to do next?” You can do the same thing for math where you say, “I’m at this point and I’m getting stuck.
我此时此刻；接下来我该怎么办？”你可以对数学做同样的事情，你会说：“我现在已经陷入困境了。
Don’t give me the full solution, but what is something I could try?” You kind of carefully probe it.
不要给我完整的解决方案，但我可以尝试什么？”你仔细地探究一下。
2:12:02 But the problem is that it requires discipline.
但问题是它需要纪律。
A lot of people enjoy math, but there are also a lot of people who need to do it for their homework, and then it’s just a shortcut.
很多人喜欢数学，但也有很多人需要做作业，而这只是一条捷径。
We can develop an educational LLM, but the other LLMs are still there, and there’s still a temptation to use them.
我们可以开发教育法学硕士，但其他法学硕士仍然存在，并且仍然存在使用它们的诱惑。
I think a lot of people, especially in college, understand the stuff they’re passionate about— —they’re self-aware about it, and they understand it shouldn’t be easy.
我认为很多人，尤其是在大学里的人，都了解他们所热衷的东西——他们对此有自我意识，并且他们明白这不应该是一件容易的事。
2:12:20 Like, I think we just have to develop a good taste— —talk about research taste, like school taste about stuff that you should be struggling on— —and stuff you shouldn’t be struggling on.
就像，我认为我们只需要培养一种良好的品味——谈论研究品味，就像学校对你应该为之奋斗的东西的品味——以及你不应该为之奋斗的东西。
Which is tricky to know, because sometimes you don’t have good long-term vision about what would be actually useful to you in your career.
这很难知道，因为有时你对什么对你的职业生涯真正有用没有良好的长期愿景。
But you have to develop that taste, yeah.
但你必须培养这种品味，是的。
I was talking to maybe my fiance or friends about this, and it’s like there’s this brief 10-year window where all of the homework and all the exams could be digital.
我可能正在和我的未婚夫或朋友谈论这件事，就像有一个短暂的 10 年窗口，所有作业和所有考试都可以数字化。
2:12:51 But before that, everybody had to do all the exams in bluebooks because there was no other way.
但在此之前，每个人都必须完成蓝皮书上的所有考试，因为没有其他办法。
And now after AI, everybody’s going to need to be in bluebooks and oral exams because everybody could cheat so easily.
现在，在人工智能之后，每个人都需要参加蓝皮书和口语考试，因为每个人都可以很容易地作弊。
It’s like this brief generation that had a different education system where everything could be digital, but you still couldn’t cheat.
这就像这一代人拥有不同的教育体系，一切都可以数字化，但你仍然不能作弊。
And now it’s just going back.
现在它又要回去了。
It’s just very funny.
这真是太有趣了。
You mention character training.
你提到性格训练。
2:13:20 Just zooming out on a more general topic: for that topic, how much compute was required?
只是缩小一个更普遍的主题：对于该主题，需要多少计算？
And in general, to contribute as a researcher, are there places where not too much compute is required where you can actually contribute as an individual researcher?
一般来说，作为一名研究人员做出贡献，是否有一些地方不需要太多计算，而您实际上可以作为个人研究人员做出贡献？
For the character training thing, I think this research is built on fine-tuning about seven billion parameter models with LoRA, which is like a…
对于角色训练的事情，我认为这项研究是建立在用 LoRA 微调大约 70 亿个参数模型的基础上的，这就像……
2:13:39 Essentially, you’re only fine-tuning a small subset of the weights of the model.
本质上，您只需微调模型权重的一小部分。
I don’t know exactly how many GPU hours that would take.
我不知道具体需要多少 GPU 小时。
But it’s doable.
但这是可行的。
Not doable for every academic.
并非每个学者都可行。
2:13:55 So the situation for some academics is so dire that the only work you can do is doing inference where you have closed models or open models, and you get completions from them and you can look at them and understand the models.
因此，对于某些学者来说，情况非常严峻，以至于你唯一能做的工作就是在封闭模型或开放模型中进行推理，你可以从它们那里得到完成结果，你可以查看它们并理解模型。
And that’s very well-suited to evaluation, where you want to be the best at creating representative problems that the models fail on or show certain abilities, which I think that you can break through with.
这非常适合评估，你希望最擅长创建模型失败的代表性问题或表现出某些能力，我认为你可以突破这一点。
I think that the top-end goal for a researcher working on evaluation, if you want to have career momentum, is that the frontier labs pick up your evaluation.
我认为，如果你想拥有职业发展动力，从事评估工作的研究人员的最高目标是让前沿实验室接受你的评估。
So you don’t need to have every project do this.
所以你不需要让每个项目都这样做。
2:14:33 But if you go from a small university with no compute and you figure out something that Claude struggles with and then the next Claude model has it in the blog post, there’s your career rocket ship.
但是，如果您来自一所没有计算能力的小型大学，并且您发现了克劳德正在努力解决的问题，然后下一个克劳德模型将其出现在博客文章中，那就是您的职业火箭飞船。
I think that that’s hard but if you want to scope the maximum possible impact with minimum compute, it’s something like that—which is just get very narrow, and it takes learning of where the models are going.
我认为这很难，但如果你想用最少的计算来确定最大可能的影响范围，那就是这样的——范围变得非常狭窄，并且需要了解模型的发展方向。
So you need to build a tool that tests where Claude 4.5 will fail.
因此，您需要构建一个工具来测试 Claude 4.5 会失败的地方。
If I’m going to start a research project, I need to think where the models in eight months are going to be struggling.
如果我要开始一个研究项目，我需要考虑八个月后模型将在哪些方面陷入困境。
But what about developing totally novel ideas?
但是开发全新的想法又如何呢？
This is a trade-off.
这是一个权衡。
2:15:08 I think that if you’re doing a PhD, you could also be like, “It’s too risky to work in language models.
我认为，如果你正在攻读博士学位，你也可能会说，“在语言模型中工作风险太大。
I’m going way longer term,” which is like— —what is the thing that’s going to define language model development in 10 years?
我的目标是更长远的发展”，这就像——10 年后定义语言模型开发的是什么？
Which I think that I end up being a person that’s pretty practical.
我认为我最终会成为一个非常实际的人。
I mean, I went to my PhD where it was like, “I got into Berkeley.
我的意思是，当我攻读博士学位时，感觉就像是，“我进入了伯克利。
Worst case, I get a master’s, and then I go work in tech.” And so I’m very practical about it.
最坏的情况是，我获得硕士学位，然后去科技行业工作。”所以我对此非常务实。
The life afforded to people to work at these AI companies, the amount of…
在这些人工智能公司工作的人们所享受的生活、……的数量
OpenAI’s average compensation is over a million dollars in stock a year per employee.
OpenAI 的平均薪酬为每位员工每年超过 100 万美元的股票。
For any normal person in the US, getting into this AI lab is transformative for your life.
对于美国的任何一个普通人来说，进入这个人工智能实验室都会改变你的生活。
So I’m pretty practical about it—
所以我对此很实际——
2:15:50 —there’s still a lot of upward mobility working in language models if you’re focused.
——如果你专注的话，语言模型中仍然有很多向上流动的机会。
And looking at the outcomes, look at these jobs.
看看结果，看看这些工作。
But from a research perspective, for transformative impact and these academic awards, being the next Yann LeCun comes from not caring about language model development very much.
但从研究的角度来看，就变革性影响和这些学术奖项而言，成为下一个 Yann LeCun 源于不太关心语言模型开发。
It’s a big financial sacrifice in that case.
在这种情况下，这是一个巨大的经济牺牲。
So I get to work with some awesome students, and they’re like, “Should I go work at an AI lab?” And I’m like, “You’re getting a PhD at a top school.
所以我开始和一些很棒的学生一起工作，他们会问，“我应该去人工智能实验室工作吗？”我说，“你在一所顶尖学校获得博士学位。
2:16:09 Are you going to leave to go to a lab?” If you go work at a top lab, I don’t blame you.
你要离开去实验室吗？”如果你去顶级实验室工作，我不会责怪你。
Don’t go work at some random startup that might go to zero.
不要随意去某个可能为零的初创公司工作。
But if you’re going to OpenAI, I think it could be worth leaving a PhD for.
但如果你要去 OpenAI，我认为放弃博士学位可能值得。
Let’s more rigorously think through this.
让我们更仔细地思考一下这个问题。
2:16:30 Where would you give a recommendation for people to do a research contribution?
您会在哪里推荐人们做出研究贡献？
The options are academia—get a PhD, spend five years publishing, though compute resources are constrained.
学术界的选择是——获得博士学位，花五年时间发表论文，尽管计算资源有限。
There are research labs that are more focused on open-weight models, so working there.
有些研究实验室更专注于开放权重模型，因此在那里工作。
Or closed frontier labs.
或者关闭前沿实验室。
OpenAI, Anthropic, xAI, and so on.
OpenAI、Anthropic、xAI 等。
The two gradients are: the more closed, the more money you tend to get, but also you get less credit.
这两个梯度是：越封闭，你往往获得的钱就越多，但你获得的信用也就越少。
2:17:04 In terms of building a portfolio of things that you’ve done, it’s very clear what you have done as an academic.
在建立你所做过的事情的组合方面，你作为一名学者所做的事情是非常清楚的。
Versus if you are going to trade this fairly reasonable progression for being a cog in the machine, which could also be very fun.
相反，如果你打算用这个相当合理的进展来换取成为机器中的一个齿轮，这也可能非常有趣。
I think they’re very different career paths.
我认为他们的职业道路非常不同。
But the opportunity cost for being a researcher is very high because PhD students are paid essentially nothing.
但成为一名研究人员的机会成本非常高，因为博士生基本上没有任何报酬。
I think it ends up rewarding people that have a fairly stable safety net and they realize they can operate in the long term, doing very interesting work and getting a very interesting job.
我认为这最终会奖励那些拥有相当稳定的安全网的人，他们意识到自己可以长期经营，做非常有趣的工作并找到一份非常有趣的工作。
2:17:50 So it is a privileged position to be like, “I’m going to see out my PhD and figure it out after because I want to do this.” And at the same time, the academic ecosystem is getting bombarded by funding getting cut and stuff.
因此，“我将完成我的博士学位并在之后解决这个问题，因为我想这样做，”这是一个特权地位。与此同时，学术生态系统正受到资金削减等问题的轰炸。
There are just so many different trade-offs where I understand plenty of people that are like, “I don’t enjoy it.
有很多不同的权衡，我知道很多人会说，“我不喜欢它。
I can’t deal with this funding search.
我无法处理这种资金搜索。
My grant got cut for no reason by the government,” or, “I don’t know what’s going to happen.” So I think there’s a lot of uncertainty and trade-offs that, in my opinion, favor just taking the well-paying job with meaningful impact.
我的补助金被政府无缘无故地削减了，”或者，“我不知道会发生什么。”因此，我认为存在很多不确定性和权衡，在我看来，有利于只接受具有有意义影响的高薪工作。
It’s not like you’re getting paid to sit around at OpenAI.
这并不是说你在 OpenAI 工作就能获得报酬。
You’re building the cutting edge of things that are— changing millions of people’s relationship to tech.
您正在打造前沿事物，改变数百万人与科技的关系。
But publication-wise, they’re being more secretive, increasingly so.
但在出版方面，他们变得更加保密，而且越来越保密。
2:18:34 So you’re publishing less and less.
所以你发表的文章越来越少。
You are having a positive impact at scale, but you’re a cog in the machine.
你正在产生大规模的积极影响，但你只是机器中的一个齿轮。
I think, honestly, it hasn’t changed that much.
我认为，老实说，它并没有改变那么多。
2:18:47 I have been in academia; I’m not in academia anymore.
我曾在学术界工作过；我已经不在学术界了。
At the same time, I wouldn’t want to miss my time in academia.
同时，我也不想错过在学术界的时光。
But what I wanted to say before I get to that part, I think it hasn’t changed that much.
但在讲到这一部分之前我想说的是，我认为它并没有太大改变。
I was using AI or machine learning methods for applications in computational biology with collaborators, and a lot of people went from academia directly to Google.
我与合作者一起使用人工智能或机器学习方法在计算生物学中应用，很多人直接从学术界转到谷歌。
I think it’s the same thing.
我认为这是同一件事。
Back then, professors were sad that their students went into industry because they couldn’t carry on their legacy in that sense.
当时，教授们对他们的学生进入工业界感到悲伤，因为他们无法在这个意义上继承他们的遗产。
I think it’s the same thing.
我认为这是同一件事。
It hasn’t changed that much.
它并没有改变那么多。
The only thing that has changed is the scale.
唯一改变的是规模。
2:19:32 But, you know, cool stuff was always developed in industry that was closed.
但是，你知道，很酷的东西总是在封闭的行业中开发出来的。
You couldn’t talk about it.
你不能谈论它。
I think the difference now is your preference.
我认为现在的区别在于你的偏好。
Do you like to talk about your work and publish, or are you more in a closed lab?
你喜欢谈论你的工作并发表文章，还是更喜欢在封闭的实验室里？
That’s one difference—the compensation, of course.
这是一个区别——当然是补偿。
But it’s always been like that.
但一直都是这样。
So it really depends on where you feel comfortable.
所以这实际上取决于你在哪里感觉舒服。
And also, nothing is forever.
而且，没有什么是永远的。
The only thing right now is there’s a third option, which is starting a startup.
现在唯一的办法就是第三种选择，那就是创业。
There are a lot of people doing startups.
有很多人在创业。
Very risky move, but it’s a high-risk, high-reward type of situation, whereas joining an industry lab is pretty safe and offers upward mobility.
这是一个非常危险的举动，但这是一种高风险、高回报的情况，而加入行业实验室是相当安全的，并且提供向上的流动性。
2:20:16 Honestly, I think once you have been at an industry lab, it will be easier to find future jobs.
老实说，我认为一旦你进入了行业实验室，未来找到工作就会更容易。
But then again, it’s like, how much do you enjoy the team and working on proprietary things versus how do you like the publishing work?
但话又说回来，你对团队和专有事物的工作有多喜欢，而你对出版工作的喜欢程度如何？
I mean, publishing is stressful.
我的意思是，出版是有压力的。
Acceptance rates at conferences can be arbitrary and very frustrating, but it’s also high reward.
会议的接受率可能是任意的并且非常令人沮丧，但这也是很高的回报。
If you have a paper published, you feel good because your name is on there.
如果你发表了一篇论文，你会感觉很好，因为上面有你的名字。
You have a high accomplishment.
你有很高的成就。
I feel like my friends who are professors seem on average happier than my friends who work at— a frontier lab, to be totally honest.
老实说，我觉得我的教授朋友平均比在前沿实验室工作的朋友更快乐。
2:20:48 Because there’s just a grounding and— the frontier labs definitely do this 9/9/6— which essentially is shorthand for work all the time.
因为只有一个基础——前沿实验室肯定会做这个 9/9/6——这本质上是一直工作的简写。
Can you describe 9/9/6 as a culture?
您能将 9/9/6 描述为一种文化吗？
2:21:03 I believe you could say it was invented in China and adopted in Silicon Valley.
我相信你可以说它是在中国发明并在硅谷采用的。
What’s 9/9/6?
9/9/6 是什么？
It’s 9:00 AM to 9:00 PM—
现在是上午 9:00 到晚上 9:00 —
six days a week.
每周六天。
Six days a week.
每周六天。
2:21:15 What is that, 72 hours?
那是什么，72小时？
Is this basically the standard in AI companies in Silicon Valley?
这基本上是硅谷AI公司的标准吗？
More and more this kind of grind mindset.
这种磨练的心态越来越多。
Yeah, I mean, maybe not exactly like that, but I think there is a trend towards it.
是的，我的意思是，也许不完全是这样，但我认为有这样的趋势。
2:21:26 And it’s interesting—I think it almost flipped because when I was in academia, I felt like that because as a professor, you had to write grants, you had to teach, and you had to do your research.
这很有趣——我认为它几乎翻转了，因为当我在学术界时，我有这样的感觉，因为作为一名教授，你必须写资助，你必须教学，你必须做你的研究。
It’s like three jobs in one, and it is more than a full-time job if you want to be successful.
这就像三份工作合二为一，如果你想成功，这不仅仅是一份全职工作。
And I feel like now, like Nathan just said, the professors in comparison to a lab have even less pressure or workload than at a frontier lab because—
我觉得现在，就像内森刚才说的那样，与实验室相比，教授们的压力或工作量甚至比前沿实验室还要小，因为——
I think they work a lot.
我认为他们工作很多。
2:21:57 They’re just so fulfilled.
他们真是太满足了。
By working with students— …and having a constant runway of mentorship and a mission that is very people-oriented.
通过与学生合作——……并拥有持续的指导和以人为本的使命。
I think in an era when things are moving very fast and are very chaotic, it’s very rewarding to people.
我认为在一个事物发展非常快而且非常混乱的时代，这对人们来说是非常有价值的。
Yeah, and I think at a startup, there’s this pressure.
是的，我认为在初创公司，存在这种压力。
2:22:11 You have to make it.
你必须做到。
It is really important that people put in the time, but it is really hard because you have to deliver constantly.
人们投入时间确实很重要，但这真的很困难，因为你必须不断地交付。
I’ve been at a startup.
我曾在一家初创公司工作过。
I had a good time, but I don’t know if I could do it forever.
我过得很开心，但我不知道我是否能永远这样。
It’s an interesting pace and it’s exactly like we talked about in the beginning.
这是一个有趣的节奏，就像我们一开始谈到的那样。
These models are leapfrogging each other, and they are just constantly trying to take the next step compared to their competitors.
这些模型正在相互超越，并且与竞争对手相比，它们只是不断尝试迈出下一步。
It’s just ruthless right now.
现在简直就是无情。
I think this leapfrogging nature and having multiple players is actually an underrated driver of language modeling progress where competition is so deeply ingrained.
我认为这种跨越式的性质和拥有多个参与者实际上是语言建模进步的一个被低估的驱动力，因为竞争是如此根深蒂固。
2:22:42 These companies have intentionally created very strong cultures.
这些公司有意创造了非常强大的文化。
For example, Anthropic is known to be culturally deeply committed and organized.
例如，Anthropic 以其在文化上的坚定承诺和组织而闻名。
We hear so little from them, and everybody at Anthropic seems very aligned.
我们很少听到他们的消息，但 Anthropic 的每个人似乎都非常一致。
Being in a culture that is super tight and having this competitive dynamic is a thing that’s going to make you work hard and create things that are better.
生活在一个非常紧张的文化中，并且拥有这种竞争动力，会让你努力工作并创造出更好的东西。
2:23:20 But that comes at the cost of human capital.
但这是以人力资本为代价的。
You can only do this for so long, and people are definitely burning out.
你只能这样做这么长时间，人们肯定会精疲力尽。
I wrote a post on burnout as I’ve tread in and out of this myself, especially trying to be a manager while doing full-mode training.
我写了一篇关于职业倦怠的文章，因为我自己也曾经历过这种情况，尤其是在进行全模式培训时试图成为一名经理。
It’s a crazy job.
这是一项疯狂的工作。
In the book Apple in China, Patrick McGee talked about how hard the Apple engineers worked to set up the supply chains in China.
在《苹果在中国》一书中，帕特里克·麦吉谈到了苹果工程师在中国建立供应链的努力。
He mentioned they had “saving marriage” programs, and he said in a podcast that people died from this level of working hard.
他提到他们有“拯救婚姻”计划，他在播客中说，人们会因为这种程度的努力工作而死亡。
It’s a perfect environment for creating progress based on human expense.
这是一个以人力成本为基础创造进步的完美环境。
The human expense is the 996 that we started this with, where people do really grind.
人力成本是我们开始时的 996，人们确实很辛苦。
I also read this book.
我也读过这本书。
2:24:08 I think they had a code word for if someone had to go home to spend time with their family to save the marriage.
我认为他们有一个暗号，表示如果有人必须回家与家人共度时光以挽救婚姻。
Then the colleagues said, “Okay, this is red alert for this situation.
然后同事们说：“好吧，这是针对这种情况的红色警报。
We have to let that person go home this weekend.” But at the same time, I don’t think they were forced to work.
这个周末我们必须让那个人回家。”但与此同时，我不认为他们是被迫工作的。
They were so passionate about the product that you get into that mindset.
他们对产品充满热情，以至于你会陷入这种心态。
I had that sometimes as an academic, and as an independent person.
作为一名学者和一个独立的人，我有时会遇到这种情况。
I overwork, and it’s unhealthy.
我工作过度，这不健康。
I had back issues and neck issues because I did not take the breaks that I should have.
我有背部和颈部问题，因为我没有得到应有的休息。
But it’s not because anyone forced me; it’s because I wanted to work because it’s exciting stuff.
但这并不是因为有人强迫我；而是因为有人强迫我。这是因为我想工作，因为它是令人兴奋的事情。
That’s what OpenAI and Anthropic are like.
OpenAI 和 Anthropic 就是这样的。
2:24:46 They want to do this work.
他们想做这项工作。
Yeah, but there’s also a feeling of fervor that’s building, especially in Silicon Valley, aligned with the scaling laws idea.
是的，但也有一种与规模法则理念相一致的热情正在兴起，尤其是在硅谷。
2:24:49 There’s this hype where the world will be transformed in a scale of weeks and you want to be at the center of it.
有这样一种炒作，世界将在几周内发生转变，而你想成为其中的中心。
I have the great fortune of having conversations with a wide variety of human beings, and I get to see all these bubbles and echo chambers across the world.
我很幸运能够与各种各样的人进行对话，并且我可以看到世界各地的所有这些气泡和回声室。
It’s fascinating to see how we humans form them.
看看我们人类如何形成它们是很有趣的。
I think it’s fair to say that Silicon Valley is a kind of echo chamber, a kind of silo and bubble.
我认为可以公平地说，硅谷是一种回音室、一种筒仓和泡沫。
I think bubbles are actually really useful and effective.
我认为气泡实际上非常有用且有效。
It’s not necessarily a negative thing because you can be ultra-productive.
这不一定是一件坏事，因为你可以非常高效。
2:25:34 It could be the Steve Jobs reality distortion field, because you just convince each other the breakthroughs are imminent, and by convincing each other of that, you make the breakthroughs imminent.
这可能是史蒂夫·乔布斯的现实扭曲场，因为你只要让对方相信突破即将到来，通过让对方相信这一点，你就能让突破迫在眉睫。
Bryne Hobart wrote a book classifying bubbles.
布莱恩·霍巴特写了一本对气泡进行分类的书。
2:25:48 One of them is financial bubbles, which involve speculation and are bad, and the other is effectively for build-outs, because it pushes people to build.
其中之一是金融泡沫，它涉及投机，是不好的；另一个是有效的扩建，因为它推动人们建造。
I do think AI is in this, but I worry about it transitioning to a financial bubble.
我确实认为人工智能参与其中，但我担心它会转变为金融泡沫。
Yeah, but also in the space of ideas, that bubble creates a reality distortion field.
是的，但在思想空间中，泡沫创造了现实扭曲场。
2:26:05 That means you are deviating from reality, and if you go too far while also working 996, you might miss some fundamental aspects of the human experience.
这意味着你偏离了现实，如果你在工作996的同时走得太远，你可能会错过人类经验的一些基本方面。
This is a common problem in Silicon Valley.
这是硅谷的一个普遍问题。
It’s a very specific geographic area.
这是一个非常特殊的地理区域。
You might not understand the Midwest perspective or the experience of all the other different humans in the United States and across the world.
您可能不了解中西部的观点或美国和世界各地所有其他不同人类的经历。
You speak a certain way to each other and convince each other of a certain thing, and that can get you into real trouble.
你们以某种方式向对方说话，并让对方相信某件事，这可能会让你们陷入真正的麻烦。
2:26:47 Whether AI is a big success and becomes a powerful technology or it’s not, in either trajectory you can get yourself into trouble.
无论人工智能是否取得巨大成功并成为一项强大的技术，无论在哪一条轨迹上，你都可能会给自己带来麻烦。
So you have to consider all of that.
所以你必须考虑所有这些。
Here you are, a young person trying to decide what you want to do with your life.
你是一个年轻人，正在努力决定自己的人生目标。
The thing that is…
事情是……
2:27:02 I don’t even really understand this, but the SF AI memes have gotten to the point where the “permanent underclass” was one of them.
我什至不太明白这一点，但科幻人工智能模因已经到了“永久下层阶级”就是其中之一的地步。
This was the idea that the last six months of 2025 was the only time to build durable value in an AI startup or model.
人们认为 2025 年的最后六个月是人工智能初创公司或模型建立持久价值的唯一时机。
Otherwise, all the value will be captured by existing companies and you will therefore be poor.
否则，所有的价值都会被现有的公司所获取，你就会变得贫穷。
That’s an example of the SF thing that goes so far.
这是迄今为止科幻小说的一个例子。
I still think for young people who are really passionate about having an impact in AI, being physically in SF is the most likely place where you’re going to do this.
我仍然认为，对于那些真正热衷于对人工智能产生影响的年轻人来说，在旧金山是最有可能做到这一点的地方。
But it has trade-offs.
但它也有其弊端。
I think SF is an incredible place, but there is a bit of a bubble.
我认为旧金山是一个令人难以置信的地方，但有一点泡沫。
2:27:41 And if you go into that bubble, which is extremely valuable, just get out also.
如果你进入了那个非常有价值的泡沫，那就也退出吧。
Read history books, read literature, and visit other places in the world.
阅读历史书、阅读文学作品、参观世界其他地方。
Twitter and Substack are not the entire world.
Twitter 和 Substack 并不是整个世界。
I think I would say, one of the people I worked with is moving to SF, and I need to get him a copy of Season of the Witch.
我想我会说，与我一起工作的一个人要搬到旧金山，我需要给他一份《女巫季节》的副本。
2:28:01 It’s a history of SF from 1960 to 1985 that goes through the hippie revolution, the culture emerging in the city, the HIV/AIDS crisis, and other things.
这是一部从 1960 年到 1985 年的科幻史，经历了嬉皮革命、城市文化兴起、艾滋病危机等。
That is so recent, with so much turmoil and hurt, but also love in SF.
那是最近的事情，有那么多的动荡和伤害，但也有旧金山的爱。
No one knows about this.
没有人知道这件事。
It’s a great book, Season of the Witch; I recommend it.
这是一本很棒的书，《女巫的季节》；我推荐它。
A bunch of my SF friends who do get out recommended it to me.
我的一群出去玩的旧金山朋友向我推荐了它。
I lived there and I didn’t appreciate this context, and it’s just so recent.
我住在那里，我并不欣赏这种环境，而且它只是最近的事。
Yeah.
是的。
2:28:46 Okay, let’s…
好吧，让我们……
we talked a lot about many things, certainly about what was exciting last year.
我们谈论了很多事情，当然还有去年令人兴奋的事情。
But this year, one of the things you guys mentioned that’s exciting is the scaling of text diffusion models and just a different exploration of text diffusion.
但今年，你们提到的令人兴奋的事情之一是文本传播模型的扩展以及对文本传播的不同探索。
Can you talk about what that is and what possibilities it holds?
您能谈谈这是什么以及它有哪些可能性吗？
So, different kinds of approaches than the current LMs?
那么，与当前的 LM 相比，有哪些不同的方法呢？
Yeah, so we talked a lot about the transformer architecture and the autoregressive transformer architecture specifically, like GPT.
是的，所以我们具体讨论了 Transformer 架构和自回归 Transformer 架构，例如 GPT。
2:29:13 And it doesn’t mean no one else is working on anything else.
这并不意味着没有其他人在做其他事情。
People are always on the lookout for the next big thing, because I think it would be almost stupid not to.
人们总是在寻找下一件大事，因为我认为不这样做几乎是愚蠢的。
Sure, right now the transformer architecture is the thing and it works best, but it’s always a good idea to not put all your eggs into one basket.
当然，现在变压器架构是最流行的，而且效果最好，但不要把所有鸡蛋都放在一个篮子里总是一个好主意。
People are developing alternatives to the autoregressive transformer.
人们正在开发自回归变压器的替代品。
One of them would be, for example, text diffusion models.
例如，其中之一是文本传播模型。
2:29:49 And listeners may know diffusion models from image generation, like Stable Diffusion popularized it.
听众可能会从图像生成中了解扩散模型，例如稳定扩散（Stable Diffusion）使其普及。
Back then, people used GANs, Generative Adversarial Networks.
当时，人们使用 GAN（生成对抗网络）。
And then there was this diffusion process where you iteratively de-noise an image, and that resulted in really good quality images over time.
然后是一个扩散过程，您可以迭代地对图像进行去噪，随着时间的推移，这会产生质量非常好的图像。
Other companies build their own diffusion models.
其他公司建立了自己的扩散模型。
And now people are like, “Okay, can we try this also for text?” It doesn’t make intuitive sense yet because it feels like it’s not something continuous like a pixel that we can differentiate.
现在人们会说，“好吧，我们可以在文本上也尝试一下吗？”它还没有直观意义，因为感觉它不是像像素那样连续的东西，我们可以区分。
It’s discrete text, so how do we implement that de-noising process?
它是离散文本，那么我们如何实现去噪过程呢？
2:30:25 But it’s kind of similar to the BERT models by Google.
但它有点类似于谷歌的 BERT 模型。
When you go back to the original transformer, there were the encoder and the decoder.
当你回到原来的变压器时，有编码器和解码器。
The decoder is what we are using right now in GPT and so forth.
解码器就是我们现在在 GPT 等中使用的解码器。
The encoder is more like a parallel technique where you have multiple tokens that you fill in in parallel.
编码器更像是一种并行技术，其中您有多个并行填充的标记。
GPT models do autoregressive completion one token at a time.
GPT 模型一次执行一个标记的自回归完成。
In BERT models, you have a sentence that has gaps—you mask them out—and then one iteration is filling in those gaps.
在 BERT 模型中，你有一个有间隙的句子——你将它们屏蔽掉——然后一次迭代会填补这些间隙。
2:31:02 And text diffusion is kind of like that, where you are starting with some random text, and then you are filling in the missing parts or refining them iteratively over multiple iterations.
文本扩散有点像这样，您从一些随机文本开始，然后填充缺失的部分或通过多次迭代迭代地完善它们。
The cool thing here is that this can do multiple tokens at the same time, so it has the promise of being more efficient.
这里很酷的一点是，它可以同时处理多个令牌，因此有望提高效率。
Now, the trade-off is, of course, how good is the quality?
当然，现在的权衡是质量有多好？
It might be faster, but the more de-noising steps you do, the better the text becomes.
它可能会更快，但是您执行的降噪步骤越多，文本就会变得越好。
People are trying to see if that is a valid alternative to the autoregressive model in terms of giving you the same quality for less compute.
人们试图看看这是否是自回归模型的有效替代方案，能够以更少的计算量提供相同的质量。
2:31:46 Right now, there are papers that suggest if you want to get the same quality, you have to crank up the de-noising steps and then you end up spending the same compute you would spend on an autoregressive model.
目前，有论文表明，如果你想获得相同的质量，你必须加快去噪步骤，然后你最终会花费与自回归模型相同的计算量。
The other downside is that while it’s parallel, some tasks are not.
另一个缺点是，虽然它是并行的，但有些任务不是并行的。
For reasoning tasks or tool use where you have to ask a code interpreter to give you an intermediate result, it is kind of tricky with diffusion models.
对于推理任务或工具使用，你必须要求代码解释器给你一个中间结果，扩散模型有点棘手。
So there are some hybrids.
所以有一些混合体。
But the main idea is how can we parallelize it.
但主要思想是我们如何并行化它。
It’s an interesting avenue.
这是一条有趣的途径。
I think right now there are mostly research models out there, like LaMDA and some other ones.
我认为现在主要有研究模型，比如 LaMDA 和其他一些模型。
2:32:24 I saw some by startups, some deployed models, but there is no big diffusion model at scale yet on the level of Gemini or ChatGPT.
我看到了一些初创公司的模型，一些已部署的模型，但目前还没有像 Gemini 或 ChatGPT 那样的大规模扩散模型。
But there was an announcement by Google where they said they are launching Gemini Diffusion, and they put it into context of their Nano 2 model.
但谷歌宣布，他们正在推出 Gemini Diffusion，并将其置于 Nano 2 模型的背景下。
They said for the same quality on most benchmarks, we can generate things much faster.
他们表示，对于大多数基准测试的相同质量，我们可以更快地生成东西。
I don’t think the text diffusion model is going to replace autoregressive LLMs, but it will be something for quick, cheap, at-scale tasks.
我不认为文本扩散模型会取代自回归法学硕士，但它将适合快速、廉价、大规模的任务。
Maybe the free tier in the future will be something like that.
也许未来的免费套餐也会是这样的。
I think there are a couple of examples where it’s actually started to be used.
我认为有几个实际开始使用它的例子。
2:33:04 To paint an example of why this is so much better: when a model like GPT-5 takes time to respond, it’s generating one token at a time.
举一个例子来说明为什么这要好得多：当像 GPT-5 这样的模型需要时间响应时，它一次生成一个令牌。
This diffusion idea is essentially generating all of those tokens in the completion in one batch, which is why it could be way faster.
这种扩散的想法本质上是在一批完成中生成所有这些代币，这就是为什么它可以更快。
2:33:27 The startups I’m hearing are code startups where you have a codebase and somebody is effectively vibe coding.
我听到的初创公司都是代码初创公司，你有一个代码库，有人在有效地进行氛围编码。
They say, “Make this change,” and a code diff is essentially a huge reply from the model.
他们说，“进行此更改”，代码差异本质上是模型的巨大回复。
It doesn’t have to have that much external context, and you can get it really fast by using these diffusion models.
它不必有那么多的外部背景，并且您可以通过使用这些扩散模型快速获得它。
They use text diffusion to generate really long diffs because doing it with an autoregressive model would take minutes, and that time causes a lot of churn for a user-facing product.
他们使用文本扩散来生成非常长的差异，因为使用自回归模型执行此操作需要几分钟，而这段时间会导致面向用户的产品产生大量流失。
Every second, you lose users.
每一秒，你都会失去用户。
So I think that it’s going to be this thing where it’s going to-
所以我认为这将会是——
2:34:02 -grow and have some applications, but I actually thought that different types of models were going to be used for different things sooner than they have been.
-增长并拥有一些应用程序，但我实际上认为不同类型的模型将比以前更快地用于不同的事情。
I think the tool use point is the one that’s stopping them from being most general purpose because, with something like Claude Code or ChatGPT with search, the autoregressive chain is interrupted with an external tool, and I don’t know how to do that with the diffusion setup.
我认为工具的使用点是阻止它们成为最通用用途的地方，因为使用 Claude Code 或 ChatGPT 等带有搜索功能的东西，自回归链会被外部工具打断，而我不知道如何通过扩散设置来做到这一点。
So what’s the future of tool use this year and in the coming years?
那么今年和未来几年工具使用的未来是什么？
2:34:28 Do you think there’s going to be a lot of developments there, and how that’s integrated into the entire stack?
您认为那里会有很多发展吗？如何将其集成到整个堆栈中？
I do think right now it’s mostly on the proprietary LLM side, but we will see more of that in open-source tooling.
我确实认为现在主要是在专有的法学硕士方面，但我们会在开源工具中看到更多这样的情况。
2:34:37 It is a huge unlock because then you can really outsource certain tasks from just memorization to actual computation—you know, instead of having the LLM memorize what is 23 plus 5, just use a calculator.
这是一个巨大的解锁，因为这样你就可以真正将某些任务从记忆外包到实际计算——你知道，法学硕士不需要记住 23 加 5，只需使用计算器即可。
So do you think that can help solve hallucinations?
那么你认为这可以帮助解决幻觉吗？
Not solve it, but reduce it.
不是解决它，而是减少它。
2:35:01 Still, the LLM needs to know when to ask for a tool call.
尽管如此，法学硕士仍需要知道何时请求工具调用。
And second, it doesn’t mean the internet is always correct.
其次，这并不意味着互联网总是正确的。
You can do a web search for who won the World Cup in 1998, but it still needs to find the right website and get the right information.
您可以在网络上搜索 1998 年世界杯冠军是谁，但仍然需要找到正确的网站并获取正确的信息。
You can still go to the incorrect website and get incorrect information.
您仍然可以访问不正确的网站并获取不正确的信息。
I don’t think it will fully solve it, but it is improving.
我不认为它会完全解决这个问题，但它正在改善。
There was another cool paper earlier this year—I think it was December 31st, so not technically 2026, but close—on the recursive language model.
今年早些时候还有另一篇很酷的论文——我认为那是 12 月 31 日，所以技术上不是 2026 年，但也很接近——关于递归语言模型。
2:35:43 That’s a cool idea to take this even a bit further.
这是一个很酷的想法，可以更进一步。
Nathan, you mentioned earlier it’s harder to do cool research in academia because of the compute budget.
Nathan，您之前提到过，由于计算预算的原因，在学术界进行出色的研究变得更加困难。
If I recall correctly, they did everything with GPT-5, so they didn’t even use local models.
如果我没记错的话，他们所做的一切都是 GPT-5，所以他们甚至没有使用本地模型。
But the idea is, for a long-context task, instead of having the LLM solve all of it in one shot or in a chain, you break it down into sub-tasks.
但我们的想法是，对于一个长上下文任务，不是让法学硕士一次性或一连串地解决所有问题，而是将其分解为子任务。
You have the LLM decide what is a good sub-task and then recursively call an LLM to solve that.
你让 LLM 决定什么是好的子任务，然后递归调用 LLM 来解决这个问题。
2:36:16 And then adding tools—you know, each sub-task maybe goes to the web and gathers information, and then you pull it all together at the end.
然后添加工具——你知道，每个子任务可能会访问网络并收集信息，然后最后将它们整合在一起。
I think there’s going to be a lot of unlock using things like that where you don’t necessarily improve the LLM itself, you improve how the LLM is used and what it can use.
我认为使用类似的东西会有很多解锁，你不一定要改进LLM本身，你可以改进LLM的使用方式和它可以使用的内容。
One downside right now with tool use is you have to give the LLM permission to use tools.
目前使用工具的一个缺点是您必须授予法学硕士使用工具的权限。
That will take some trust, especially if you want to unlock things like having an LLM answer emails for you, or just sort them.
这需要一些信任，特别是如果你想解锁一些东西，比如让法学硕士为你回复电子邮件，或者只是对它们进行排序。
I don’t know if I would today give an LLM access to my emails, right?
我不知道今天是否允许法学硕士访问我的电子邮件，对吗？
I mean, this is a huge risk.
我的意思是，这是一个巨大的风险。
I think there’s one last point on the tool use thing.
我认为关于工具使用的问题还有最后一点。
2:37:03 You hinted at this, and we’ve both come at this in our own ways: open versus closed models use tools in very different ways.
您暗示了这一点，我们都以自己的方式解决了这一问题：开放模型与封闭模型以非常不同的方式使用工具。
With open models, people go to Hugging Face and download the model, and then the person’s going to be like, “What tool do I want?” Maybe X.ai is my preferred search provider, but someone else might care for a different search startup.
有了开放模型，人们就会去 Hugging Face 并下载模型，然后人们会问：“我想要什么工具？”也许 X.ai 是我首选的搜索提供商，但其他人可能会关心不同的搜索初创公司。
When you release a model, it needs to be useful for multiple tools, which is really hard because you’re making a general reasoning engine, which is actually what gpt-oss-120b is good for.
当你发布一个模型时，它需要对多种工具有用，这真的很难，因为你正在制作一个通用推理引擎，而这实际上正是 gpt-oss-120b 的优点。
2:37:36 But on the closed models, you’re deeply integrating the specific tool into your experience.
但在封闭模型上，您正在将特定工具深入集成到您的体验中。
I think that open models will struggle to replicate some of the things that I like to do with closed models, where you can reference a mix of public and private information.
我认为开放模型将很难复制我喜欢用封闭模型做的一些事情，在封闭模型中你可以参考公共和私人信息的混合。
Something that I keep trying every three to six months is Codex on the web, which is just prompting a model to make an update to some GitHub repository that I have.
我每三到六个月就会尝试一次 Web 上的 Codex，它只是提示模型对我拥有的某些 GitHub 存储库进行更新。
2:38:01 That set of secure cloud environment is just so nice for just sending it off to do this thing and then come back to me.
这套安全的云环境非常适合将其发送出去做这件事，然后再回到我身边。
This will probably help define some of the local open and closed niches.
这可能有助于定义一些本地开放和封闭的利基市场。
Because there was such a rush to get tool use working, the open models were on the back foot, which is kind of inevitable.
由于急于让工具使用发挥作用，开放模型处于不利地位，这是不可避免的。
There are so many resources in these frontier labs, but it will be fun when the open models solve this because it’s going to necessitate a more flexible model that might work with this recursive idea to be an orchestrator.
这些前沿实验室中有很多资源，但是当开放模型解决这个问题时会很有趣，因为这将需要一个更灵活的模型，可以与这种递归思想一起工作，成为一个协调器。
Hopefully, necessity drives innovation there.
希望必要性能够推动那里的创新。
So, continual learning—this is a longstanding topic and an important problem.
所以，持续学习——这是一个长期存在的话题，也是一个重要的问题。
2:38:45 I think that increases in importance as the cost of training models goes up.
我认为随着训练模型成本的上升，这一点变得越来越重要。
So can you explain what continual learning is and how important it might be this year and in the coming years to make progress?
那么您能解释一下什么是持续学习以及今年和未来几年取得进步的重要性吗？
This relates a lot to this kind of SF zeitgeist of: what is AGI, Artificial General Intelligence, and what is ASI, Artificial Superintelligence?
这与这种科幻时代精神有很大关系：什么是 AGI（通用人工智能），什么是 ASI（人工超级智能）？
2:39:03 What are the language models that we have today capable of doing?
我们今天拥有的语言模型能够做什么？
I think language models can solve a lot of tasks, but a key milestone for the AI community is when AI can replace any remote worker, taking in information and solving digital tasks.
我认为语言模型可以解决很多任务，但人工智能社区的一个关键里程碑是人工智能何时可以取代任何远程工作者，接收信息并解决数字任务。
The limitation is that a language model will not learn from feedback the same way an employee does.
局限性在于语言模型不会像员工那样从反馈中学习。
If you hire an editor, they might mess up, but you will tell them, and they don’t do it again.
如果你雇用一名编辑，他们可能会搞砸，但你会告诉他们，他们就不会再这样做了。
2:39:43 But language models don’t have this ability to modify themselves and learn very quickly.
但语言模型不具备自我修改和快速学习的能力。
The idea is, if we are going to get to something that is a true, general adaptable intelligence that can go into any remote work scenario, it needs to be able to learn quickly from feedback and on-the-job learning.
我们的想法是，如果我们想要获得真正的、通用的、可适应任何远程工作场景的智能，它需要能够从反馈和在职学习中快速学习。
I’m personally more bullish on language models being able to just provide very good context.
我个人更看好语言模型能够提供非常好的上下文。
You can write extensive documents where you say, “I have all this information.
您可以编写大量文档，其中写道：“我拥有所有这些信息。
Here are all the blog posts I’ve ever written.
这是我写过的所有博客文章。
I like this type of writing; my voice is based on this.” But a lot of people don’t provide this to models.
我喜欢这样的写作方式；我的声音就是基于此。”但很多人不向模特提供这一点。
2:40:24 The agentic models are just starting.
代理模式才刚刚开始。
So it’s this kind of trade-off: do we need to update the weights of this model with this continual learning thing to make them learn fast?
所以这是一种权衡：我们是否需要通过持续学习来更新该模型的权重，以使它们学习得更快？
Or, the counterargument is we just need to provide them with more context and information, and they will have the appearance of learning fast by just having a lot of context and being very smart.
或者，反驳是我们只需要为他们提供更多的背景和信息，他们就会通过拥有大量的背景并且非常聪明而表现出学习速度很快。
So we should mention the terminology here.
所以我们应该在这里提到术语。
2:40:43 Continual learning refers to changing the weights continuously so that the model adapts and adjusts based on the new incoming information, and does so continually, rapidly, and frequently.
持续学习是指不断改变权重，使模型根据新输入的信息进行适应和调整，并且持续、快速、频繁地进行。
And then the thing you mentioned on the other side of it is generally referred to as in-context learning.
然后你在另一面提到的东西通常被称为情境学习。
As you learn stuff, there’s a huge context window.
当你学习东西时，有一个巨大的上下文窗口。
You can just keep loading it with extra information every time you prompt the system, which I think both can legitimately be seen as learning.
每次提示系统时，您都可以继续加载额外的信息，我认为这两者都可以合理地视为学习。
It’s just a different place where you’re doing the learning.
这只是您学习的不同地方。
I think, to be honest with you, continual learning—the updating of weights—we already have that in different flavors.
我认为，老实说，持续学习——权重更新——我们已经有了不同的风格。
2:41:24 I think the distinction here is: do you do that on a personalized custom model for each person, or do you do it on a global model scale?
我认为这里的区别是：你是在每个人的个性化定制模型上这样做，还是在全球模型规模上这样做？
And I think we have that already with going from GPT-5 to 5.1 and 5.2.
我认为我们已经从 GPT-5 升级到 5.1 和 5.2。
It’s maybe not immediate, but it is like a quick curated update where there was feedback by the community on things they couldn’t do.
这可能不是立竿见影的，但它就像一个快速策划的更新，社区针对他们无法做的事情提供了反馈。
They updated the weights, released the next model, and so forth.
他们更新了权重，发布了下一个模型，等等。
So it is kind of a flavor of that.
所以这是一种味道。
Another even finer-grained example is RLVR; you run it, it updates.
另一个更细粒度的例子是 RLVR；你运行它，它就会更新。
2:42:08 The problem is you can’t just do that for each person because it would be too expensive to update the weights for each person.
问题是你不能只对每个人这样做，因为更新每个人的权重成本太高。
Even at OpenAI scale, building the data centers, it would be too expensive.
即使在 OpenAI 规模上，建设数据中心也太昂贵了。
I think that is only feasible once you have something on the device where the cost is on the consumer.
我认为只有当你的设备上有一些东西并且成本由消费者承担时，这才是可行的。
Like what Apple tried to do with the Apple Intelligence models, putting them on the phone so they learn from the experience.
就像苹果试图对苹果智能模型所做的那样，将它们放在手机上，以便他们从体验中学习。
A bit of a related topic, but this kind of—maybe anthropomorphized term—memory.
有点相关的话题，但是这种——也许是拟人化的术语——记忆。
2:42:33 What are different ideas of the mechanism of how to add memory to these systems as you’re increasingly seeing?
您越来越多地看到，对于如何向这些系统添加内存的机制有哪些不同的想法？
Especially personalized memory?
特别是个性化记忆？
Right now, it’s mostly like context—stuffing things into the context and then just recalling that.
现在，它就像上下文一样——将事物填充到上下文中，然后回忆起来。
2:42:49 But again, it’s expensive because even if you cache it, you spend tokens on that.
但同样，它很昂贵，因为即使你缓存它，你也要花费代币。
And the second one is you can only do so much.
第二个是你只能做这么多。
I think it’s more like a preference or style.
我认为这更像是一种偏好或风格。
A lot of people do that when they solve math problems.
很多人在解决数学问题时都会这样做。
You can add previous knowledge, but you also give it certain preference prompts, like “do what I preferred last time.” But it doesn’t unlock new capabilities.
你可以添加以前的知识，但你也可以给它一些偏好提示，比如“做我上次喜欢做的事”。但它并没有解锁新的功能。
For that, one thing people still use is LoRA adapters.
为此，人们仍在使用的一件事是 LoRA 适配器。
2:43:32 These are basically, instead of updating the whole weight matrix, two smaller weight matrices that you have in parallel or overlays, like the delta.
这些基本上不是更新整个权重矩阵，而是并行或叠加的两个较小的权重矩阵，例如增量。
But you can do that to some extent, and then again, it is economics.
但你可以在某种程度上做到这一点，话又说回来，这是经济学。
There were also papers showing, for example, LoRA learns less but forgets less.
例如，也有论文表明 LoRA 学到的东西更少，但忘记的东西也更少。
There’s no free lunch.
天下没有免费的午餐。
If you want to learn more, you need to use more weights, but it gets more expensive.
如果你想了解更多，你需要使用更多的重量，但它会变得更昂贵。
And then if you learn more, you forget more; you have to find that Goldilocks zone.
如果你学得越多，你就会忘记更多；你必须找到那个金发姑娘区。
We haven’t really mentioned it much, but implied in this discussion is context length as well.
我们并没有真正提到太多，但本次讨论中也暗示了上下文长度。
2:44:04 Is there a lot of innovations that’s possible there?
那里有很多可能的创新吗？
I think the colloquially accepted thing is that it’s a compute and data problem.
我认为通俗地接受的事情是这是一个计算和数据问题。
2:44:13 Sometimes there are small architecture things, like attention variants.
有时会有一些小的架构问题，例如注意力变体。
We talked about hybrid attention models, which is essentially if you have what looks like a state space model within your transformer.
我们讨论了混合注意力模型，本质上是如果你的变压器中有一个看起来像状态空间模型的东西。
Those are better suited because you have to spend less compute to model the furthest along token.
这些更适合，因为您只需花费更少的计算来对最远的令牌进行建模。
But those aren’t free because they have to be accompanied by a lot of compute or the right data.
但这些并不是免费的，因为它们必须伴随大量的计算或正确的数据。
How many sequences of 100,000 tokens do you have in the world, and where do you get these?
您在世界上有多少个 100,000 个代币的序列？您从哪里获得这些？
It just ends up being pretty expensive to scale them.
扩展它们最终会变得相当昂贵。
2:44:56 So we’ve gotten pretty quickly to a million tokens of input context length.
所以我们很快就达到了一百万个令牌的输入上下文长度。
And I would expect it to keep increasing and get to 2 million or 5 million this year, but I don’t expect it to go to, like, 100 million.
我预计它会继续增长，今年会达到 200 万或 500 万，但我预计不会达到 1 亿。
That would be a true breakthrough, and I think those breakthroughs are possible.
这将是一个真正的突破，而且我认为这些突破是可能的。
I think of the continual learning thing as a research problem where there could be a breakthrough that makes transformers work way better at this and it’s cheap.
我认为持续学习是一个研究问题，可能会出现突破，使变压器在这方面工作得更好，而且价格便宜。
These things could happen with so much scientific attention.
这些事情可能会在科学界如此多的关注下发生。
Но turning the crank, it’ll be consistent increases over time.
转动曲柄，它会随着时间的推移而持续增加。
I think also looking at the extremes, there’s no free lunch.
我认为从极端情况来看，天下没有免费的午餐。
2:45:27 One extreme to make it cheap is to have, let’s say, an RNN that has a single state where you save everything from the previous stuff.
为了降低成本，一种极端的做法是使用一个 RNN，它只有一个状态，可以保存之前的所有内容。
It’s a specific fixed-size thing, so you never really grow the memory.
它是一个特定的固定大小的东西，所以你永远不会真正增加内存。
You are stuffing everything into one state, but then the longer the context gets, the more information you forget because you can’t compress everything into one state.
你把所有东西都塞进一种状态，但是上下文越长，你忘记的信息就越多，因为你无法将所有东西压缩到一种状态。
Then on the other hand, you have the transformers, which try to remember every token.
另一方面，你有变压器，它会尝试记住每个标记。
That is great if you want to look up specific information, but very expensive because you have the KV cache and the dot product that grow.
如果您想查找特定信息，这很好，但非常昂贵，因为您有 KV 缓存和不断增长的点积。
2:46:06 But then, like you said, the Mamba layers kind of have the same problem.
但正如你所说，曼巴层也有同样的问题。
Like an RNN, you try to compress everything into one state, and you’re a bit more selective there.
就像 RNN 一样，你尝试将所有内容压缩到一种状态，并且你在那里更有选择性。
I think it’s like this Goldilocks zone again with NVIDIA Nemotron 3; they found a good ratio of how many attention layers you need for the global information where everything is accessible compared to having these compressed states.
我认为 NVIDIA Nemotron 3 又像这个金发姑娘区；他们发现，与拥有这些压缩状态相比，您需要多少个注意力层来获取所有内容均可访问的全局信息，这是一个很好的比率。
I think we will scale more by finding better ratios in that Goldilocks zone between making it cheap enough to run and making it powerful enough to be useful.
我认为，我们将通过在“金发姑娘区”找到更好的比率来扩大规模，使其足够便宜以运行，并使其强大到足以有用。
2:46:43 And one more plug here: the recursive language model paper is one of the papers that tries to address the long context thing.
这里还有一个插曲：递归语言模型论文是试图解决长上下文问题的论文之一。
What they found is, essentially, instead of stuffing everything into this long context, if you break it up into multiple smaller tasks, you save memory and can actually get better accuracy than having the LLM try everything all at once.
他们发现，本质上，如果你把它分解成多个较小的任务，而不是把所有的东西都塞进这个长的上下文中，你可以节省内存，而且实际上比让法学硕士一次尝试所有的东西可以获得更好的准确性。
It’s a new paradigm; we will see if there are other flavors of that.
这是一个新的范式；我们会看看是否还有其他口味。
I think we will still make improvement on long context, but like Nathan said, the problem is for pre-training itself, we don’t have as many long-context documents as other documents.
我认为我们仍然会在长上下文上做出改进，但就像 Nathan 所说，问题在于预训练本身，我们没有像其他文档一样多的长上下文文档。
So it’s harder to study basically how LMs behave on that level.
因此，从根本上研究 LM 在该级别上的行为方式更加困难。
There are some rules of thumb where, essentially, you pre-train a language model—like OLMo, we pre-trained at an 8K context length and then extended to 32K with training.
有一些经验法则，本质上是预训练语言模型，例如 OLMo，我们以 8K 上下文长度进行预训练，然后通过训练扩展到 32K。
2:47:31 There’s a rule of thumb where doubling the training context length takes about 2X compute, and then you can normally 2 to 4X the context length again.
根据经验，将训练上下文长度加倍大约需要 2 倍的计算量，然后通常可以再次将上下文长度增加 2 到 4 倍。
I think a lot of it ends up being compute-bound at pre-training.
我认为其中很多最终都会在预训练时受到计算限制。
Everyone talks about this big increase in compute for the top labs this year, and that should reflect in some longer context windows.
每个人都在谈论今年顶级实验室的计算量的大幅增长，这应该反映在一些更长的上下文窗口中。
2:48:02 But I think on the post-training side, there’s some more interesting things.
但我认为在训练后方面，还有一些更有趣的事情。
As we have agents, the agents are going to manage this context on their own.
由于我们有代理，代理将自行管理此上下文。
Now people who use Claude Code a lot dread the compaction, which is when Claude takes its entire 100,000 tokens of work and compacts it into a bulleted list.
现在，使用 Claude Code 的人非常害怕压缩，即 Claude 获取全部 100,000 个工作令牌并将其压缩到项目符号列表中。
But what the next models will do—I’m sure people are already working on this—is the model can control when it compacts and how.
但下一个模型将要做的是——我确信人们已经在研究这个——模型可以控制何时以及如何压缩。
So you can essentially train your RL algorithm where compaction is an action,
所以你基本上可以训练你的强化学习算法，其中压缩是一个动作，
2:48:30 where it shortens the history.
它缩短了历史。
Then the problem formulation will be, “I want to keep the maximum evaluation scores while the model compacts its history to the minimum length.” Because then you have the minimum amount of tokens that you need to do this kind of compounding auto-regressive prediction.
那么问题的表述将是：“我想在模型将其历史压缩到最小长度的同时保持最大评估分数。”因为这样你就拥有了进行这种复合自回归预测所需的最少量的令牌。
There are actually pretty nice problem setups in this where these agentic models learn to use their context in a different way than just plowing forward.
实际上，这里有非常好的问题设置，这些代理模型学习以不同的方式使用它们的上下文，而不仅仅是向前推进。
One interesting recent example would be DeepSeek-V3.2, where they had a sparse attention mechanism with a very efficient, small, lightweight indexer.
最近一个有趣的例子是 DeepSeek-V3.2，其中他们有一个稀疏注意力机制和一个非常高效、小型、轻量级的索引器。
2:48:56 Instead of attending to all the tokens, it selects which tokens I actually need.
它不会关注所有令牌，而是选择我实际需要的令牌。
It almost comes back to the original idea of attention where you are selective, but attention is always on; you have maybe zero weight on some of them, but you use them all.
这几乎回到了最初的注意力概念，你是有选择性的，但注意力总是存在的；你可能对其中一些的权重为零，但你会使用它们。
But they are even more like, “Okay, let’s just mask that out or not even do that.” And even with sliding window attention in OLMo, that is also kind of like that idea.
但他们更像是，“好吧，让我们掩盖这一点，或者干脆不这样做。”即使 OLMo 中有滑动窗口注意力，这也有点像那个想法。
You have that rolling window where you keep it fixed, because you don’t need everything all the time.
你有一个固定的滚动窗口，因为你并不总是需要一切。
2:49:34 Occasionally, in some layers you might, but it’s wasteful.
有时，在某些层中你可能会这样做，但这很浪费。
But right now, I think if you use everything, you’re on the safe side; it gives you the best bang for the buck because you never miss information.
但现在，我认为如果你使用一切，你就安全了；它为您带来最大的性价比，因为您永远不会错过任何信息。
And right now, I think this year will also be the year of figuring out, like you said, how to be smarter about that.
现在，我认为今年也将是弄清楚如何在这方面变得更聪明的一年，就像你说的那样。
Right now people want to have the next state-of-the-art, and the state-of-the-art happens to be the brute force, expensive thing.
现在人们想要拥有下一个最先进的技术，而最先进的技术恰好是蛮力、昂贵的东西。
Once you have that, like you said, you want to keep that accuracy but see how we can do that cheaper now using tricks.
一旦你有了这个，就像你说的，你想保持这种准确性，但看看我们现在如何使用技巧更便宜地做到这一点。
Yeah, all this scaling thing.
是的，所有这些缩放的事情。
2:50:07 Like the reason we get the Claude 4.5 Sonnet model first is because you can train it faster and you’re not hitting these compute walls as soon.
就像我们首先获得 Claude 4.5 Sonnet 模型的原因一样，因为您可以更快地训练它，并且不会很快遇到这些计算墙。
They can just try a lot more things and get the model out faster, even though the bigger model is actually better.
他们可以尝试更多的事情并更快地得到模型，即使更大的模型实际上更好。
I think we should say that there’s a lot of exciting stuff going on in the AI space.
我认为我们应该说人工智能领域正在发生很多令人兴奋的事情。
2:50:22 My mind has recently been really focused on robotics, so today we almost entirely didn’t talk about robotics.
我最近的注意力非常集中在机器人技术上，所以今天我们几乎完全不谈论机器人技术。
There’s a lot of stuff on image generation and video generation.
关于图像生成和视频生成有很多内容。
I think it’s fair to say that the most exciting research work in terms of intensity and fervor is in the LLM space, which is why I think it’s justified for us to focus on the LLMs we’re discussing.
我认为可以公平地说，就强度和热情而言，最令人兴奋的研究工作是在法学硕士领域，这就是为什么我认为我们有理由专注于我们正在讨论的法学硕士。
But it’d be nice to bring in certain things that might be useful.
但最好引入某些可能有用的东西。
For example, world models—there’s growing excitement about that.
例如，世界模型——人们对此越来越兴奋。
Do you think there will be any use in this coming year for world models in the LLM space?
您认为明年 LLM 领域的世界模型会有什么用处吗？
2:51:08 Also with LLMs, what’s an interesting thing here is I think if we unlock more LLM capabilities, it also automatically unlocks all the other fields because it makes progress faster.
另外，对于法学硕士来说，有趣的是，我认为如果我们解锁更多法学硕士功能，它也会自动解锁所有其他领域，因为它会使进展更快。
Because, you know, a lot of researchers and engineers use LLMs for coding.
因为，你知道，很多研究人员和工程师都使用法学硕士进行编码。
So even if they work on robotics, if you optimize these LLMs that help with coding, it pays off.
因此，即使他们研究机器人技术，如果你优化这些有助于编码的法学硕士，也会得到回报。
But then yes, world models are interesting.
但是，是的，世界模型很有趣。
It’s basically where you have the model run a simulation of the world—like a little toy version of the real thing—which can unlock capabilities like data the LLM is not aware of.
基本上，你可以让模型运行对世界的模拟——就像真实事物的小玩具版本——它可以解锁法学硕士不知道的数据等功能。
It can simulate things.
它可以模拟事物。
I think LLMs happen to work well by pre-training and doing next-token prediction, but we could do this in a more sophisticated way.
我认为法学硕士恰好可以通过预训练和下一个令牌预测来很好地发挥作用，但我们可以用更复杂的方式来做到这一点。
2:52:05 There was a paper, I think by Meta, called “Coder World Models.” They basically apply the concept of world models to LLMs where, instead of just having next-token prediction and verifiable rewards checking the answer correctness, they also make sure the intermediate variables are correct.
我认为 Meta 曾发表过一篇论文，名为“Coder World Models”。他们基本上将世界模型的概念应用于法学硕士，他们不仅通过下一个令牌预测和可验证的奖励来检查答案的正确性，还确保中间变量是正确的。
The model is basically learning a code environment.
该模型基本上是学习代码环境。
I think this makes a lot of sense; it’s just expensive to do.
我认为这很有道理；只是做起来很昂贵。
But it is making things more sophisticated by modeling the whole process, not just the result, and that can add more value.
但它通过对整个过程（而不仅仅是结果）进行建模，使事情变得更加复杂，并且可以增加更多价值。
2:52:51 I remember when I was a grad student, there’s a competition called CASP where they do protein structure prediction.
我记得当我还是一名研究生时，有一个名为 CASP 的竞赛，他们进行蛋白质结构预测。
They predict the structure of a protein that is not solved yet.
他们预测了尚未解决的蛋白质结构。
In a sense, this is actually great, and I think we need something like that for LLMs also, where you do the benchmark but no one knows the solution until someone reveals it after the fact.
从某种意义上说，这实际上很棒，我认为法学硕士也需要类似的东西，你可以进行基准测试，但没有人知道解决方案，直到有人在事后揭示它。
When AlphaFold came out, it crushed this benchmark.
当 AlphaFold 出现时，它打破了这一基准。
I mean there were multiple iterations, but I remember the first one explicitly modeled the physical interactions and the physics of the molecule.
我的意思是有多次迭代，但我记得第一次明确地模拟了物理相互作用和分子的物理学。
2:53:34 Also, things like impossible angles.
还有一些不可能的角度之类的事情。
Then in the next version, I think they got rid of this and just used brute force, scaling it up.
然后在下一个版本中，我认为他们摆脱了这个，只是使用蛮力，扩大规模。
I think with LLMs, we are currently in this brute-force scaling because it just happens to work, but I do think at some point it might make sense to bring back this approach.
我认为对于法学硕士来说，我们目前正处于这种强力扩展的状态，因为它恰好起作用，但我确实认为在某些时候恢复这种方法可能是有意义的。
I think with world models, that might be actually quite cool.
我认为对于世界模型来说，这实际上可能非常酷。
And of course, for robotics, that is completely related to LLMs.
当然，对于机器人技术来说，这与法学硕士完全相关。
Yeah, and robotics is very explicit.
是的，机器人技术非常明确。
2:54:03 There’s the problem of locomotion or manipulation.
存在运动或操纵的问题。
Locomotion is much more solved, especially in the learning domain.
运动问题得到了更多解决，尤其是在学习领域。
But there’s a lot of value, just like with the initial protein folding systems…
但它有很多价值，就像最初的蛋白质折叠系统一样……
…Bringing in the traditional model-based methods.
...引入传统的基于模型的方法。
So it’s unlikely that you can just learn the manipulation or the whole-body local manipulation problem end-to-end.
因此，你不太可能端到端地学习操纵或全身局部操纵问题。
That’s the dream.
这就是梦想。
But then you realize when you look at the magic of the human hand…
但当你看到人手的魔力时，你就会意识到……
…And the complexity of the real world, you realize it’s really hard to learn this all the way through- …the way I guess AlphaFold 2 didn’t.
......现实世界的复杂性，你会意识到从头到尾学习这个真的很难 - ......我猜 AlphaFold 2 没有这样做。
I’m excited about the robotic learning space.
我对机器人学习空间感到兴奋。
2:54:40 I think it’s collectively getting supercharged by all the excitement and investment in language models generally.
我认为，人们普遍对语言模型的兴奋和投资共同推动了这一趋势。
The infrastructure for training transformers, which is a general modeling thing, is becoming world-class industrial tooling.
用于训练变压器的基础设施是一种通用的建模工具，正在成为世界一流的工业工具。
Wherever there was a limitation for robotics, it’s just way better now.
无论机器人技术有什么限制，现在都好多了。
There’s way more compute.
还有更多的计算量。
They take these language models and use them as central units where you can do interesting explorative work around something that already works.
他们采用这些语言模型并将它们用作中心单元，您可以在其中围绕已经有效的东西进行有趣的探索性工作。
And then I see it emerging as, kind of like we talked about, Hugging Face transformers and Hugging Face.
然后我看到它的出现，有点像我们谈到的，拥抱脸变形金刚和拥抱脸。
2:55:19 I think when I was at Hugging Face, I was trying to get this to happen, but it was too early.
我想当我在 Hugging Face 的时候，我就试图让这一切发生，但还为时过早。
These open robotic models on Hugging Face enable people to contribute data and fine-tune them.
Hugging Face 上的这些开放机器人模型使人们能够贡献数据并对其进行微调。
I think we’re much closer now that the investment in robotics and self-driving cars is related and enables this.
我认为现在我们已经更接近了，因为对机器人和自动驾驶汽车的投资是相关的并且能够实现这一点。
Once you get to the point where you have this sort of ecosystem, someone can download a robotics model and fine-tune it to their robot or share datasets across the world.
一旦你达到了拥有这种生态系统的地步，人们就可以下载机器人模型并将其微调到他们的机器人或在世界各地共享数据集。
There’s some work in this area like RTX from a few years ago where people are starting to do that.
这个领域有一些工作，比如几年前的 RTX，人们就开始这样做了。
But once they have this ecosystem, it’ll look very different.
但一旦他们拥有了这个生态系统，它看起来就会非常不同。
And then this whole post-ChatGPT boom is putting more resources into that, which I think is a very good area for doing research.
然后整个 ChatGPT 后的繁荣正在投入更多的资源，我认为这是一个非常好的研究领域。
This is also resulting in much better, more accurate, more realistic simulators being built, closing this sim-to-real gap in the robotic space.
这也导致了更好、更准确、更真实的模拟器的诞生，缩小了机器人领域中模拟与真实的差距。
2:56:02 But you know, you mentioned a lot of excitement and investment.
但你知道，你提到了很多兴奋和投资。
The downside of that, which happens in hype cycles—I personally believe, and most robotics people believe—is that robotics is not going to be solved on the timescale being implicitly or explicitly promised.
其缺点是，在炒作周期中发生的——我个人认为，大多数机器人技术人员也相信——是机器人技术不会在隐含或明确承诺的时间范围内得到解决。
So what happens when all these robotics companies spring up and then they don’t have a product that works?
那么，当所有这些机器人公司如雨后春笋般涌现，但他们却没有有效的产品时，会发生什么呢？
Then there’s going to be this crash of excitement, which is nerve-wracking.
然后就会出现令人兴奋的崩溃，这令人伤脑筋。
Hopefully something else will swoop in so that the continued development of some of these ideas keeps going.
希望其他东西会突然出现，以便其中一些想法能够继续发展。
I think it’s also related to the continual learning issue.
我认为这也与持续学习的问题有关。
2:56:53 The real world is so complex, whereas with LLMs, you don’t really need to have something learn for the user because there are a lot of things everyone has to do—everyone maybe wants to fix their grammar in their email or code.
现实世界是如此复杂，而对于法学硕士来说，你真的不需要为用户学习一些东西，因为每个人都必须做很多事情——每个人可能都想在他们的电子邮件或代码中修复他们的语法。
It’s more constrained, so you can prepare the model for that.
它受到更多限制，因此您可以为此准备模型。
But preparing a robot for the real world is harder.
但为现实世界准备一个机器人却更困难。
You have robotic foundation models, and you can learn things like grasping, but every house is different.
你有机器人基础模型，你可以学习诸如抓取之类的东西，但每个房子都是不同的。
It’s so different that the robot would have to learn on the job, essentially.
它是如此不同，以至于机器人本质上必须在工作中学习。
And I think that is the bottleneck right now: customizing it on the fly.
我认为这就是现在的瓶颈：即时定制。
I don’t think I can possibly understate the importance of the thing that doesn’t get talked about almost at all by robotics folks or anyone, and that is safety.
我认为我不能低估机器人技术人员或任何人几乎根本没有谈论过的事情的重要性，那就是安全。
2:57:42 All the interesting complexities we talk about regarding learning, all the failure modes and failure cases—everything we’ve been talking about with LLMs where sometimes it fails in interesting ways—all of that is fun and games in the LLM space.
我们谈论的有关学习的所有有趣的复杂性、所有的失败模式和失败案例——我们与法学硕士谈论的所有事情，有时会以有趣的方式失败——所有这些都是法学硕士领域的乐趣和游戏。
In the robotic space, in people’s homes, across millions of minutes and billions of interactions, you really are almost allowed to fail never.
在机器人领域，在人们的家中，经过数百万分钟和数十亿次的互动，你几乎可以永远不会失败。
When you have embodied systems put out there in the real world, you just have to solve so many problems you never thought you’d have to solve when you’re just thinking about the general robot learning problem.
当你在现实世界中部署具体系统时，你只需要解决许多你在考虑一般机器人学习问题时从未想过必须解决的问题。
I’m so bearish on in-home learned robots for consumer purchase.
我对消费者购买的家用学习机器人非常看空。
2:58:32 I’m very bullish on self-driving cars, and I’m very bullish for robotic automation, like Amazon distribution— …where Amazon has built whole new distribution centers designed for robots first rather than humans.
我非常看好自动驾驶汽车，也非常看好机器人自动化，比如亚马逊配送——……亚马逊首先为机器人而不是人类建立了全新的配送中心。
There’s a lot of excitement in AI circles about AI enabling automation—
人工智能圈子里对人工智能实现自动化充满了兴奋——
2:58:54 …and mass-scale manufacturing, and I do think that the path to robots doing that is more reasonable.
......以及大规模制造，我确实认为机器人这样做的路径更合理。
It’s a thing that is designed and optimized to do a repetitive task that a human could conceivably do but doesn’t want to.
它的设计和优化是为了完成人类可以做但不想做的重复性任务。
But it’s also going to take a lot longer than people probably predict.
但这也将比人们预计的时间要长得多。
I think the leap from the AI singularity to scaling up mass manufacturing in the US because we have a massive AI advantage is one that is troubled by a lot of political and other challenging problems.
我认为，由于我们拥有巨大的人工智能优势，美国从人工智能奇点到扩大大规模制造的飞跃受到许多政治和其他挑战性问题的困扰。
Let’s talk about timelines specifically: timelines to AGI or ASI.
让我们具体谈谈时间表：AGI 或 ASI 的时间表。
2:59:31 Is it fair, as a starting point, to say that nobody really agrees on the definitions of AGI and ASI?
作为一个起点，说没有人真正同意 AGI 和 ASI 的定义是否公平？
I think there’s a lot of disagreement, but I’ve been getting pushback where people say it is something that could reproduce most digital economic work.
我认为存在很多分歧，但我一直受到反对，人们说这是可以复制大多数数字经济工作的东西。
2:59:46 The remote worker is a fairly reasonable example.
远程工作者是一个相当合理的例子。
I think OpenAI’s definition is somewhat related to that—an AI that can do a certain number of economically valuable tasks—which I don’t really love as a definition, but it could be a grounding point.
我认为 OpenAI 的定义在某种程度上与此相关——一种可以完成一定数量的具有经济价值的任务的人工智能——我不太喜欢这个定义，但它可能是一个基础点。
Language models today, while immensely powerful, are not this remote worker drop-in.
如今的语言模型虽然非常强大，但并不是远程工作者的替代品。
There are things an AI could do that are way harder than remote work, like solving a…
人工智能可以做的一些事情比远程工作要困难得多，比如解决......
3:00:29 …finding an unexpected scientific discovery that you couldn’t even posit, which would be an example of something people call an artificial superintelligence problem.
......发现一个你甚至无法假设的意想不到的科学发现，这将是人们所谓的人工超级智能问题的一个例子。
Or taking in all medical records and finding linkages across certain illnesses that people didn’t know or figuring out that some common drug can treat a niche cancer.
或者查看所有医疗记录并找到人们不知道的某些疾病之间的联系，或者找出某种常见药物可以治疗某种特定的癌症。
They would say that is a superintelligence thing.
他们会说这是超级智能。
So these are natural tiers.
所以这些都是自然的层次。
My problem is that it becomes deeply entwined with the quest for meaning in AI and these religious aspects.
我的问题是，它与对人工智能和这些宗教方面意义的追求深深地交织在一起。
There are different paths you can take.
您可以采取不同的路径。
And I don’t even know if remote work is a good definition.
我什至不知道远程工作是否是一个好的定义。
3:01:06 I liked the originally titled AI2027 report.
我喜欢最初标题为 AI2027 的报告。
They focus more on code and research taste, so the target there is the superhuman coder.
他们更注重代码和研究品味，因此目标是超人编码员。
They have several milestone systems: superhuman coders, superhuman AI researcher, then superintelligent AI researcher, and then the full ASI.
他们有几个里程碑式的系统：超人编码员、超人人工智能研究员、超级智能人工智能研究员，然后是完整的 ASI。
After you develop the superhuman coder, everything else follows quickly.
当你开发出超人编码员后，其他一切都会很快发生。
The task is to have fully autonomous, automated coding, so any kind of coding you need to do in order to perform research is fully automated.
任务是拥有完全自主、自动化的编码，因此您为了进行研究而需要执行的任何类型的编码都是完全自动化的。
3:01:58 From there, humans would be doing AI research together with that system, and they will quickly be able to develop a system that actually can do the research for you.
从那时起，人类将与该系统一起进行人工智能研究，他们将很快能够开发出一个真正可以为您进行研究的系统。
That’s the idea.
这就是这个想法。
Initially, their prediction was 2027 or ’28, and now they’ve pushed it back by three to four years to 2031, mean prediction.
最初，他们的预测是 2027 年或 28 年，现在他们将预测推迟了三到四年，即 2031 年。
My prediction is probably even beyond 2031, but at least you can think concretely about how difficult it is to fully automate programming.
我的预测甚至可能会超过 2031 年，但至少你可以具体思考完全自动化编程有多困难。
Yeah, I disagree with some of their presumptions and dynamics on how it would play out, but I think they did good work in defining concrete milestones to tell a useful story.
是的，我不同意他们关于事情如何发展的一些假设和动态，但我认为他们在定义具体的里程碑来讲述一个有用的故事方面做得很好。
3:02:31 That’s why the reach of this AI 2027 document well transcended Silicon Valley—because they told a good story and did a lot of rigorous work.
这就是为什么这份《AI 2027》文件的影响力远远超出了硅谷——因为他们讲述了一个好故事并做了很多严谨的工作。
3:02:53 I think the camp that I fall into is that AI is so-called jagged, which will be excellent at some things and really bad at some things.
我认为我属于的阵营是人工智能是所谓的锯齿状，它在某些事情上表现出色，但在某些事情上却很糟糕。
I think that when they’re close to this automated software engineer, what it will be good at is traditional ML systems and front end—the model is excellent at those—but the distributed ML, the models are actually really quite bad at because there’s so little training data on doing large-scale distributed learning and things.
我认为，当他们接近这位自动化软件工程师时，他们所擅长的是传统的机器学习系统和前端——该模型在这些方面非常出色——但分布式机器学习，这些模型实际上非常糟糕，因为进行大规模分布式学习之类的训练数据太少了。
And this is something that we already see, and I think this will just get amplified.
这是我们已经看到的，而且我认为这种情况只会被放大。
And then it’s kind of messier in these trade-offs, and then there’s how you think AI research works and so on.
然后这些权衡就有点混乱了，还有你认为人工智能研究是如何运作的等等。
So you think basically a superhuman coder is almost unachievable meaning, because of the jagged nature of the thing, you’re just always going to have gaps in capabilities?
所以你认为基本上一个超人编码员几乎是无法实现的意思，因为事情的锯齿状本质，你总是会在能力上存在差距？
I think it’s assigning completeness to something where the models are kind of superhuman at some types of code, and I think that will continue.
我认为它赋予模型在某些类型的代码上具有超人能力的东西的完整性，并且我认为这种情况将会继续下去。
3:03:38 And people are creative, so they’ll utilize these incredible abilities to fill in the weaknesses of the models and move really fast.
人们富有创造力，因此他们会利用这些令人难以置信的能力来填补模型的弱点并快速行动。
There will always be, for a long time, this dance between the humans enabling this thing that the model can’t do, and the best AI researchers are the ones that can enable this superpower.
在很长一段时间内，人类之间总会存在这种舞蹈，从而实现模型无法做到的事情，而最好的人工智能研究人员就是能够实现这种超能力的人。
3:04:04 And I think those lines, compared to what we already see…
我认为这些线条与我们已经看到的相比......
I think like Claude Code for building a website, you can stand up a beautiful website in a few hours or do data analysis.
我认为就像Claude Code构建网站一样，你可以在几个小时内建立一个漂亮的网站或进行数据分析。
But the whole thing is going to keep getting better at these things, and we’ll pick up some new code skills and stuff along the way.
但整个事情将会在这些事情上不断变得更好，我们将在此过程中学到一些新的代码技能和东西。
Linking to what’s happening in big tech, this AI 2027 report leans into the singularity idea where I think research is messy and social and largely in the data in ways that AI models can’t process.
与大型科技领域正在发生的事情联系起来，这份《人工智能 2027》报告倾向于奇点理念，我认为研究是混乱的、社会性的，并且主要以人工智能模型无法处理的方式处理数据。
But what we do have today is really powerful, and these tech companies are all collectively buying into this with tens of billions of dollars of investment.
但我们今天所拥有的确实非常强大，这些科技公司都集体投入了数百亿美元的投资。
So we are going to get some much better version of ChatGPT, a much better version of Claude Code than we already have.
因此，我们将获得比我们现有的 ChatGPT 更好的版本，以及 Claude Code 的更好版本。
3:04:50 I think that it’s just hard to predict where that is going, but the bright clarity of that future is why some of the most powerful people in the world are putting so much money into this.
我认为很难预测未来的发展方向，但未来的光明前景是世界上一些最有权势的人投入大量资金的原因。
And I think it’s just kind of small differences—we don’t actually know what a better version of ChatGPT is, but also can it automate AI research?
我认为这只是一些微小的差异——我们实际上不知道 ChatGPT 的更好版本是什么，但它是否可以自动化人工智能研究？
I would say probably not, at least in this timeframe.
我想说可能不会，至少在这个时间范围内。
Big tech is going to spend $100 billion much faster than we get an automated AI researcher that enables an AI research singularity.
大型科技公司花费 1000 亿美元的速度远远快于我们聘请自动化 AI 研究人员来实现 AI 研究奇点的速度。
So you think your prediction would be, if this is even a useful milestone, more than 10 years out?
那么您认为您的预测（如果这甚至是一个有用的里程碑的话）会在 10 年后发生？
I would say less than that on the software side, but I think longer than that on things like research.
我会在软件方面说得更少，但在研究等方面我会考虑得更久。
Well, let’s just for fun try to imagine a world where all software writing is fully automated.
好吧，我们只是为了好玩，尝试想象一个所有软件编写都是完全自动化的世界。
3:05:36 Can you imagine that world?
你能想象那个世界吗？
By the end of this year, the amount of software that’ll be automated will be so high.
到今年年底，自动化的软件数量将会非常多。
3:05:46 But it’ll be things like you’re trying to train a model with RL and you need to have multiple bunches of GPUs communicating with each other.
但如果你尝试使用 RL 训练模型，你需要让多个 GPU 相互通信。
That’ll still be hard, but I think it’ll be much easier.
这仍然会很困难，但我认为会容易得多。
One of the ways to think about this, the full automation of programming, is just think of lines of useful code written—the fraction of that to the number of humans in the loop.
思考这个问题的方法之一，即编程的完全自动化，只需考虑编写的有用代码行——其中的一小部分占循环中的人员数量。
3:06:02 So presumably there’ll be, for a long time, humans in the loop of software writing.
因此，很可能在很长一段时间内，人类都会参与软件编写的循环。
It’ll just be fewer and fewer relative to the amount of code written.
相对于编写的代码量来说，它只会越来越少。
Right?
正确的？
And with the superhuman coder, I think the presumption there is the number of humans in the loop goes to zero.
对于超人编码器，我认为循环中人类数量的假设会变为零。
What does that world look like when the number of humans in the loop is in the hundreds, not in the hundreds of thousands?
当循环中的人数为数百人而不是数十万人时，那个世界会是什么样子？
I think software engineering will be driven more to system design and goals of outcomes, where I do think software is largely going to be…
我认为软件工程将更多地被驱动到系统设计和结果目标，我确实认为软件很大程度上将是……
3:06:39 I think this has been happening over the last few weeks, where people have gone from a month ago saying, “Oh yeah, agents are kind of slop,” which is a famous Karpathy quote, to the industrialization of software when anyone can just create software with their fingerprints.
我认为这种情况在过去几周一直在发生，人们从一个月前说“哦，是的，代理有点糟糕”（这是卡帕蒂的一句名言），到了软件工业化，任何人都可以用他们的指纹创建软件。
I do think we are closer to that side of things, and it takes direction and understanding how the systems work to extract the best from the language models.
我确实认为我们更接近事物的这一面，并且需要指导和理解系统如何工作才能从语言模型中提取最好的内容。
And I think it’s hard to accept the gravity of how much is going to change with software development and how many more people can do things without ever looking at the code.
我认为很难接受软件开发将发生多大变化以及有多少人可以在不查看代码的情况下完成工作的严重性。
I think what’s interesting is to think about whether these systems will be independent, in the sense that while I have no doubt that LLMs will at some point solve coding in the way calculators solve calculating, right?
我认为有趣的是思考这些系统是否是独立的，从某种意义上说，虽然我毫不怀疑法学硕士将在某种程度上以计算器解决计算的方式解决编码问题，对吗？
3:07:22 At some point, humans developed a tool that you never need a human to calculate that number for; you just type it in, and it’s an algorithm.
在某种程度上，人类开发了一种工具，你永远不需要人类来计算这个数字；你只要输入它，它就是一个算法。
I think that’s the same probably for coding.
我认为编码可能也是如此。
But the question isn’t…
但问题不是……
I think what will happen is you will just say, “Build that website,” and it will make a really good website, and then you maybe refine it.
我认为将会发生的情况是，您只需说“建立该网站”，它将成为一个非常好的网站，然后您可能会完善它。
But will it do things independently where…
但它会独立做事吗……
3:07:59 Will you still have humans asking the AI to do something?
还会有人要求人工智能做某事吗？
Like will there be a person to say, “Build that website?” Or will there be AI that just builds websites or something, or whatever?
就像有人会说，“建立那个网站吗？”或者是否会有人工智能只构建网站或其他什么？
I think talking about building websites is the—
我认为谈论建立网站是——
Too simple.
太简单了。
Yeah.
是的。
3:08:16 Sure.
当然。
It’s just that the problem with websites and the problem with the web, you know, HTML and all that kind of stuff, it’s very resilient to just— slop.
只是网站的问题和网络的问题，你知道，HTML 和所有类似的东西，它非常有弹性。
It will show you slop.
它会告诉你坡度。
It’s good at showing slop.
它擅长显示坡度。
I would rather think of safety-critical systems, like asking AI to end-to-end generate something that manages logistics— or manages cars— a fleet of cars, all that kind of stuff.
我更愿意考虑安全关键系统，比如要求人工智能端到端地生成一些东西来管理物流——或者管理汽车——车队，诸如此类的东西。
So it end-to-end generates that for you.
所以它会端到端地为你生成它。
I think a more intermediate example is take something like Slack or Microsoft Word.
我认为一个更中间的例子是 Slack 或 Microsoft Word 之类的东西。
3:08:45 I think if the organizations allow it, AI could very easily implement features end-to-end and do a fairly good job for things that you want to try.
我认为如果组织允许的话，人工智能可以很容易地实现端到端的功能，并为你想要尝试的事情做得相当好。
You want to add a new tab in Slack that you want to use, and I think AI will be able to do that pretty well.
你想在 Slack 中添加一个你想要使用的新选项卡，我认为人工智能能够很好地做到这一点。
Actually, that’s a really great example.
事实上，这是一个很好的例子。
3:09:06 How far away are we from that?
我们距离这个目标还有多远？
Like this year.
就像今年一样。
See, I don’t know.
瞧，我不知道。
3:09:11 I don’t know.
我不知道。
I guess I don’t know— how bad production codebases are, but I think that within…
我想我不知道——生产代码库有多糟糕，但我认为……
3:09:14 on the order of a few years, a lot of people are going to be pushed to be more like a designer and product manager, where you have multiple of these agents that can try things for you, and they might take one to two days to implement a feature or attempt to fix a bug.
在几年的时间里，很多人将被推向设计师和产品经理，你有多个这样的代理可以为你尝试一些东西，他们可能需要一到两天的时间来实现一个功能或尝试修复一个错误。
And you have these dashboards—which I think Slack is actually a good dashboard—where your agents will talk to you and you’ll then give feedback.
你有这些仪表板——我认为 Slack 实际上是一个很好的仪表板——你的代理会在其中与你交谈，然后你会提供反馈。
But things like, I make a website and it’s like, “Do you want to make a logo that’s passable?” I think these cohesive design things and the style is going to be very hard for models and deciding on what to add next.
但是，就像我制作了一个网站，然后“你想制作一个还过得去的徽标吗？”我认为这些有凝聚力的设计和风格对于模特来说会非常困难，很难决定下一步要添加什么。
I just…
我只是……
3:09:54 Okay.
好的。
So I hang out with a lot of programmers and some of them are a little bit on the skeptical side in general—that’s just the vibe.
所以我和很多程序员一起出去玩，他们中的一些人总体上有点怀疑——这就是氛围。
I just think there’s a lot of complexity involved in adding features to complex systems.
我只是认为向复杂系统添加功能涉及很多复杂性。
Like, if you look at the browser, Chrome.
就像，如果你看看浏览器，Chrome。
If I wanted to add a feature, if I wanted to have tabs as opposed to up top, I want them on the left side.
如果我想添加一个功能，如果我想要有选项卡而不是顶部，我希望它们位于左侧。
Interface, right?
接口，对吗？
I think we’re not…
我想我们不是……
This is not a next year thing.
这不是明年的事情。
One of the Claude releases this year, one of their tests was we give it a piece of software and leave Claude to run to recreate it entirely, and it could already almost rebuild Slack from scratch, just given the parameters of the software and left in a sandbox environment to do that.
今年的 Claude 版本之一，他们的测试之一是我们给它一个软件，然后让 Claude 运行以完全重新创建它，它几乎已经可以从头开始重建 Slack，只需给出软件的参数并留在沙箱环境中即可做到这一点。
So the from-scratch part, I like almost better.
所以从头开始的部分，我几乎更喜欢。
So it might be that the smaller and newer companies are advantaged and they’re like, “We don’t have to have the bloat and complexity, and therefore this feature exists.”
因此，规模较小、较新的公司可能具有优势，他们会说，“我们不需要臃肿和复杂，因此这个功能就存在。”
And I think this gets to the point that you mentioned that some people you talk to are skeptical, and I think that’s not because the LLM can’t do X, Y, Z.
我认为这已经到了你提到的一些与你交谈的人持怀疑态度的地步，我认为这并不是因为法学硕士不能做 X、Y、Z。
3:10:53 It’s because people don’t want it to do it this way.
这是因为人们不希望它这样做。
Some of that could be a skill issue on the human side.
其中一些可能是人类方面的技能问题。
3:11:05 Unfortunately, we have to be honest with ourselves.
不幸的是，我们必须对自己诚实。
And some of that could be an underspecification issue.
其中一些可能是规格不足的问题。
So, programming…
那么，编程...
this is like a communication type of issue in relationships and friendships.
这就像人际关系和友谊中的沟通问题。
You’re assuming the LLM somehow is supposed to read your mind.
你假设法学硕士应该以某种方式读懂你的想法。
I think this is where spec-driven design is really important.
我认为这是规范驱动设计真正重要的地方。
Like you just, using natural language, specify what you want.
就像您一样，使用自然语言，指定您想要的内容。
I think if you talk to people at the labs, they use these in their training and production code.
我认为如果你与实验室的人交谈，他们会在他们的培训和生产代码中使用这些。
3:11:32 Claude Code is built with Claude Code, and they all use these things extensively.
Claude Code 是用 Claude Code 构建的，他们都广泛使用了这些东西。
And Dario talks about how much of Claude’s code…
达里奥谈到了克劳德的代码有多少……
It’s like these people are slightly ahead in terms of the capabilities—
看来这些人的能力稍微领先一些——
3:11:49 —they have, and they probably spend on inference.
——他们有，而且他们可能会花在推理上。
They could spend 10 to 100 times as much as we’re spending, like we’re on a lowly 100 or $200 a month plan.
他们的花费可能是我们的 10 到 100 倍，就像我们每月 100 或 200 美元的低计划一样。
They truly let it rip.
他们真的让它撕裂了。
And I think that with the pace of progress that we have, a year ago we didn’t have Claude Code and we didn’t really have reasoning models.
我认为，以我们的进步速度，一年前我们还没有克劳德代码，我们也没有真正的推理模型。
The difference between sitting here today and what we can do with these models—it seems like there’s a lot of low-hanging fruit to improve them.
今天坐在这里与我们可以用这些模型做的事情之间的区别是——似乎有很多容易实现的目标可以改进它们。
The failure modes are pretty dumb.
故障模式非常愚蠢。
It’s like, “Claude, you tried to use the CLI command I don’t have installed 14 times, and then I sent you the command to run.” From a modeling perspective, that thing is pretty fixable.
就像，“Claude，你尝试使用我没有安装的 CLI 命令 14 次，然后我向你发送了运行命令。”从建模的角度来看，这个问题是可以解决的。
So, I don’t know.
所以，我不知道。
I agree with you.
我同意你的看法。
3:12:34 I’ve been becoming more and more bullish in general.
总的来说，我变得越来越乐观。
Speaking to what you’re articulating, I think it is a human skill issue.
就你所阐述的内容而言，我认为这是一个人的技能问题。
So Anthropic and other companies are leading the way in understanding how to best use the models for programming; therefore, they’re effectively using them.
因此，Anthropic 和其他公司在了解如何最好地使用模型进行编程方面处于领先地位。因此，他们正在有效地使用它们。
I think there’s a lot of programmers on the outskirts who don’t…
我认为郊区有很多程序员不……
I mean, there’s not a really good guide on how to use them.
我的意思是，没有关于如何使用它们的真正好的指南。
People are trying to figure it out exactly, but—
人们试图准确地弄清楚这一点，但是——
It might be very expensive.
它可能非常昂贵。
3:13:04 It might be that the entry point for that is $2,000 a month, which is only for tech companies and rich people.
其切入点可能是每月 2,000 美元，这只适合科技公司和富人。
That could be it.
可能就是这样。
But it might be worth it.
但这可能是值得的。
3:13:13 If the final result is a working software system, it might be worth it.
如果最终结果是一个可以工作的软件系统，那么这可能是值得的。
By the way, it’s funny how we converged from the discussion of the timeline to AGI to something more pragmatic and useful.
顺便说一句，有趣的是我们如何从对 AGI 时间线的讨论转向更务实和有用的东西。
Is there anything concrete and profound to be said about the timeline to AGI and ASI?
关于 AGI 和 ASI 的时间表，有什么具体和深刻的内容可以说吗？
Or are these discussions a bit too detached from the day-to-day?
或者这些讨论是否有点脱离日常？
There’s interesting bets.
有一些有趣的赌注。
3:13:39 There’s a lot of people trying to do Reinforcement Learning with Verifiable Rewards—RLVR—but in real scientific domains.
有很多人尝试在真正的科学领域进行可验证奖励的强化学习（RLVR）。
There are startups spending hundreds of millions of dollars in funding, and they have wet labs where they’re having language models propose hypotheses that are tested in the real world.
有些初创公司花费了数亿美元的资金，他们拥有湿实验室，在那里他们的语言模型提出了在现实世界中进行测试的假设。
I would say that they’re early, but with the pace of progress—
我想说他们还早，但随着进步的步伐——
3:14:00 —maybe they’re early by six months and they make it because they were there first, or maybe they’re early by eight years.
——也许他们早了六个月并且成功了，因为他们先到了，或者也许他们早了八年。
You don’t really know.
你真的不知道。
So I think that type of moonshot to branch this momentum into other sciences would be very transformative if AlphaFold moments happen in all sorts of other scientific domains by a startup solving this.
因此，我认为，如果一家解决这个问题的初创公司在所有其他科学领域都发生了 AlphaFold 时刻，那么将这种势头扩展到其他科学领域的登月计划将是非常具有变革性的。
I think there are startups—maybe Harmonic is one—where they’re going all in on language models plus Lean for math.
我认为有些初创公司——也许 Harmonic 就是其中之一——他们全力投入语言模型和精益数学。
I think you had another podcast guest where you talked about this recently, and it’s like we don’t know exactly what’s going to fall out of spending $100 million on that model.
我想你最近有另一位播客嘉宾谈到了这个问题，就好像我们不知道在该模型上花费 1 亿美元会产生什么结果。
3:14:41 Most of them will fail, but a couple of them might be big breakthroughs that are very different than ChatGPT or Claude Code type software experiences.
其中大多数都会失败，但其中一些可能是重大突破，与 ChatGPT 或 Claude Code 类型的软件体验截然不同。
Like a tool that’s only good for a PhD mathematician but makes them 100 times more effective.
就像一个只对数学博士有用的工具，但却使他们的效率提高了 100 倍。
I agree.
我同意。
3:14:58 I think this will happen in a lot of domains, especially those with a lot of resources like finance, legal, and pharmaceutical companies.
我认为这会发生在很多领域，特别是那些拥有大量资源的领域，如金融、法律和制药公司。
But then again, is it really AGI?
但话又说回来，这真的是AGI吗？
Because we are specializing it again.
因为我们再次专业化了。
Is it really that much different from how we had specialized algorithms back in the day?
它真的与我们过去拥有专门算法的方式有很大不同吗？
I think it’s just the same thing but way more sophisticated.
我认为这只是同一件事，但更复杂。
Is there a threshold when we call it AGI?
我们称之为AGI有门槛吗？
I think the real cool thing here is that we have foundation models that we can specialize.
我认为真正酷的是我们拥有可以专门化的基础模型。
That’s the breakthrough.
这就是突破。
3:15:34 Right now, I think we are not there yet because first, it’s too expensive, but also ChatGPT doesn’t just give away their model to customize it.
现在，我认为我们还没有做到这一点，因为首先，它太贵了，而且 ChatGPT 不只是放弃他们的模型来定制它。
I can imagine a business model where OpenAI says at some point, “Hey, Bank of America, for $100 million we will do your custom model.” I think that will be the huge economic value add.
我可以想象一种商业模式，OpenAI 在某个时候会说：“嘿，美国银行，我们将花费 1 亿美元来打造您的定制模型。”我认为这将是巨大的经济附加值。
The other thing though is, what is the differentiating factor?
但另一件事是，差异化因素是什么？
If everyone uses ChatGPT, they will all do the same thing.
如果每个人都使用 ChatGPT，他们都会做同样的事情。
Everyone is moving in lockstep, but usually companies want to have a competitive advantage.
每个人都步调一致，但通常公司都希望拥有竞争优势。
I think there is no way around using some of their private data and experimenting with specialization.
我认为没有办法使用他们的一些私人数据并尝试专业化。
It’s going to be interesting.
这会很有趣。
Given the pace of progress, it does feel like things are coming.
考虑到进展的速度，确实感觉事情即将到来。
3:16:26 I don’t think the AGI and ASI thresholds are particularly useful.
我不认为 AGI 和 ASI 阈值特别有用。
I think the real question, and this relates to the remote worker thing, is when are we going to see a big, obvious leap in economic impact?
我认为真正的问题是，这与远程工作者的问题有关，我们什么时候才能看到经济影响发生重大、明显的飞跃？
3:16:35 Because currently there’s not been an obvious leap in economic impact from LLM models, for example.
例如，因为目前 LLM 模式对经济的影响并没有出现明显的飞跃。
Aside from AGI or ASI, there’s a real question of when we are going to see a GDP jump.
除了 AGI 或 ASI 之外，还有一个真正的问题：我们何时会看到 GDP 跃升。
Jump.
跳。
Yeah, it’s like, what is the GDP made up of?
是的，就像，GDP 由什么组成？
3:17:06 A lot of it is financial services, so I don’t know what this is.
很多是金融服务，所以我不知道这是什么。
It’s just hard for me to think about the GDP bump, but I would say that software development becomes valuable in a different way when you no longer have to look at the code anymore.
我很难想象 GDP 的增长，但我想说，当你不再需要查看代码时，软件开发就会以不同的方式变得有价值。
So when it is like, Claude will make you a small business—which is essentially Claude can set up your website, your bank account, your email, and your whatever else—and you just have to express what you’re trying to put into the world.
所以，当克劳德会让你成为一家小企业时——本质上克劳德可以建立你的网站、你的银行账户、你的电子邮件和你的其他任何东西——而你只需要表达你想要向世界展示什么。
That’s not just an enterprise market, but it is hard.
这不仅仅是一个企业市场，而且很难。
I don’t know how you get people to try doing that.
我不知道你如何让人们尝试这样做。
I guess if ChatGPT can do it—people are trying ChatGPT.
我想如果 ChatGPT 能做到的话——人们正在尝试 ChatGPT。
I think it boils down to the scientific question of, “How hard is tool use to solve?” Because a lot of the stuff you’re implying, the remote work stuff, is tool use.
我认为这可以归结为一个科学问题：“使用工具来解决问题有多难？”因为你所暗示的很多东西，远程工作的东西，都是工具的使用。
3:17:49 It’s like computer use; how you have an LLM that goes out there, this agentic system, and does something in the world, and only screws up 1% of the time.
就像电脑的使用一样；你如何拥有一个法学硕士，在这个代理系统中，在世界上做一些事情，并且只有 1% 的时间搞砸。
Computer use is a good example of what labs care about and we haven’t seen a lot of progress on.
计算机的使用是实验室关心的一个很好的例子，但我们还没有看到很多进展。
Or less.
或者更少。
We saw multiple demos in 2025 of, like, Claude can use your computer, or OpenAI had operator, and they all suck.
我们在 2025 年看到了多个演示，比如 Claude 可以使用你的计算机，或者 OpenAI 有操作员，但它们都很糟糕。
3:18:12 So they’re investing money in this, and I think that’ll be a good example.
所以他们在这方面投入了资金，我认为这将是一个很好的例子。
Whereas actually, taking over the whole screen seems a lot harder than having an API that they can call in the back end.
然而实际上，接管整个屏幕似乎比拥有可以在后端调用的 API 困难得多。
Some of that is you have to then set up a different environment for them all to work in.
其中一些是你必须为他们所有人设置一个不同的工作环境。
They’re not working on your MacBook; they are individually interfacing with Google and Amazon and Slack, and they handle all these things in a very different way than humans do.
它们无法在您的 MacBook 上运行；他们单独与谷歌、亚马逊和 Slack 进行交互，并且他们处理所有这些事情的方式与人类截然不同。
So some of this might be structural blockers.
所以其中一些可能是结构性阻碍。
Also, specification-wise, I think the problem for arbitrary tasks is that you still have to specify what you want your LLM to do.
另外，就规范而言，我认为任意任务的问题在于您仍然必须指定您希望法学硕士做什么。
3:18:55 What is the environment?
环境如何？
How do you specify?
你如何指定？
You can say what the end goal is, but if it can’t solve the end goal—with LLMs, if you ask it for text, it can always clarify or do sub-steps.
你可以说出最终目标是什么，但如果它无法解决最终目标——对于法学硕士，如果你要求它提供文本，它总是可以澄清或执行子步骤。
How do you put that information into a system that, let’s say, books a travel trip for you?
您如何将这些信息输入到为您预订旅行的系统中？
You can say, “Well, you screwed up my credit card information,” but even to get it to that point, as a user, how do you guide the model before it can even attempt that?
你可以说，“好吧，你搞砸了我的信用卡信息”，但即使要达到这一点，作为用户，你如何在模型尝试这样做之前引导模型呢？
I think the interface is really hard.
我觉得这个界面真的很难。
Yeah, it has to learn a lot about you specifically.
是的，它必须特别了解你的很多信息。
3:19:36 And this goes to continual learning—about the general mistakes that are made throughout, and then mistakes that are made through you.
这需要不断学习——了解整个过程中所犯的一般错误，以及你自己所犯的错误。
All the AI interfaces are getting set up to ask humans for input.
所有人工智能界面都被设置为要求人类输入。
3:19:48 I think Claude Code we talked about a lot.
我想我们谈论了很多克劳德·科德。
It asks feedback and questions.
它询问反馈和问题。
If it doesn’t have enough specification on your plan or your desire, it starts to ask questions, “Would you rather?” We talked about Memory, which saves across chats.
如果它对你的计划或愿望没有足够的说明，它就会开始问问题，“你愿意吗？”我们讨论了内存，它可以跨聊天保存。
Its first implementation is kind of odd, where it’ll mention my dog’s name or something in a chat.
它的第一个实现有点奇怪，它会在聊天中提到我的狗的名字或其他东西。
I’m like, “You don’t need to be subtle about this.
我想，“你不需要对此表现得含糊其辞。
I don’t care.” But things are emerging, like ChatGPT has the Pulse feature.
我不在乎。”但事情正在出现，比如 ChatGPT 具有 Pulse 功能。
3:20:19 Which is like a curated couple paragraphs with links to something to look at or to talk about, and people talk about how the language models are going to ask you questions.
这就像几个精心策划的段落，其中包含一些可供查看或讨论的链接，人们会谈论语言模型将如何向您提出问题。
It’s probably going to work.
它可能会起作用。
The language model knows you had a doctor appointment or something, and it’s like, “Hey, how are you feeling after that?” Which again, goes into the territory where humans are very susceptible to this and there’s a lot of social change to come.
语言模型知道你预约了医生或其他什么，就像，“嘿，那之后你感觉怎么样？”这又进入了人类非常容易受到这种影响的领域，并且即将发生很多社会变革。
But also, they’re experimenting with having the models engage.
而且，他们正在尝试让模型参与进来。
Some people really like this Pulse feature, which processes your chats and automatically searches for information and puts it in the ChatGPT app.
有些人真的很喜欢这个 Pulse 功能，它可以处理您的聊天并自动搜索信息并将其放入 ChatGPT 应用程序中。
So there’s a lot of things coming.
所以会有很多事情发生。
I used that feature before, and I always feel bad because it does that every day, and I rarely check it out.
我以前用过这个功能，但总感觉不好，因为它每天都这样，我很少去查看。
3:20:58 How much compute is burned on something I don’t even look at, you know?
你知道在我根本不看的东西上消耗了多少计算量吗？
There’s also a lot of idle compute in the world, so don’t feel too bad.
世界上还有很多闲置计算，所以不要感觉太糟糕。
Okay.
好的。
3:21:16 Do you think new ideas might be needed?
您认为可能需要新的想法吗？
Is it possible that the path to AGI—whatever that is, however we define that—to solve computer use more generally, to solve biology and chemistry and physics, sort of the Dario definition of AGI or powerful AI?
通向 AGI 的道路（无论是什么，无论我们如何定义）是否有可能解决更普遍的计算机使用问题，解决生物学、化学和物理问题，有点像达里奥对 AGI 或强大人工智能的定义？
Do you think it’s possible that totally new ideas are needed?
您认为可能需要全新的想法吗？
Non-LLM, non-RL ideas.
非法学硕士、非强化学习的想法。
What might they look like?
它们可能是什么样子？
We’re now going into philosophy land a little bit.
我们现在要稍微进入哲学领域。
For something like a singularity to happen, I would say yes.
对于像奇点这样的事情发生，我会说是的。
3:21:50 And the new ideas could be architectures or training algorithms, which are fundamental deep learning things.
新的想法可以是架构或训练算法，它们是深度学习的基础。
But they’re, in that nature, pretty hard to predict.
但从本质上来说，它们很难预测。
But I think we won’t get very far even without those advances.
但我认为即使没有这些进步，我们也不会走得太远。
Like, we might get this software solution, but it might stop at software and not do computer use without more innovation.
比如，我们可能会得到这个软件解决方案，但如果没有更多的创新，它可能会停留在软件上，而不是使用计算机。
So I think that a lot of progress will be coming, but if you’re going to zoom out, there’s still ideas in the next 30 years that are going to look like a major scientific innovation that enabled the next chapter of this.
所以我认为将会取得很多进展，但如果你把目光放远一点，在接下来的 30 年里仍然会有一些想法看起来像是一项重大的科学创新，它们开启了下一个篇章。
And I don’t know if it comes in one year or in 15 years.
我不知道它是一年后还是15年后到来。
Yeah.
是的。
3:22:32 I wonder if the Bitter Lesson holds true for the next 100 years, and what that looks like.
我想知道这个惨痛的教训在未来 100 年是否依然有效，以及它会是什么样子。
If scaling laws are fundamental in deep learning, I think the Bitter Lesson will always apply, which is compute will become more abundant.
如果缩放定律是深度学习的基础，我认为痛苦的教训将永远适用，即计算将变得更加丰富。
3:22:37 But even within abundant compute, the ones that have a steeper scaling law slope or a better offset—like, this is a 2D plot of performance and compute—even if there’s more compute available, the ones that get 100x out of it will win.
但即使在丰富的计算范围内，那些具有更陡峭的缩放定律斜率或更好的偏移量的人（例如，这是性能和计算的二维图） - 即使有更多的可用计算，那些获得 100 倍的人也会获胜。
It might be something like literally computer clusters orbiting Earth with solar panels.
它可能就像字面上带有太阳能电池板的围绕地球运行的计算机集群。
The problem with that is heat dissipation.
这样做的问题是散热。
3:23:09 You get all the radiation from the sun and you don’t have any air to dissipate heat.
你受到太阳的所有辐射，但没有任何空气来散热。
But there is a lot of space to put clusters.
但是有很多空间可以放置簇。
There’s a lot of solar energy there and you could figure out the heat dissipation, but there is a lot of energy and there probably could be engineering will to solve the heat problem— …so there could be.
那里有大量的太阳能，你可以计算出散热量，但也有大量的能源，并且可能有工程意愿来解决热量问题——……所以可能会有。
Is it possible—and we should say that it definitely is possible—that we’re basically going to be plateauing this year?
有没有可能——我们应该说这绝对有可能——今年我们基本上会陷入停滞状态？
3:23:27 Not in terms of— …the system capabilities, but what the system capabilities actually mean for human civilization.
不是指……系统能力，而是系统能力对人类文明实际上意味着什么。
So on the coding front, really nice websites will be built.
因此，在编码方面，将会建立非常好的网站。
Very nice autocomplete.
非常好的自动完成功能。
3:23:53 Very nice way to understand code bases and maybe help debug, but really just a very nice helper on the coding front.
这是理解代码库的非常好的方法，也许有助于调试，但实际上只是编码方面的一个非常好的助手。
It can help research mathematicians do some math.
它可以帮助研究数学家做一些数学工作。
It can help you with shopping.
它可以帮助您购物。
It’s a nice helper.
这是一个好帮手。
It’s Clippy on steroids.
这是 Clippy 的类固醇。
What else?
还有什么？
It may be a good education tool and all that kind of stuff, but computer use turns out extremely difficult to solve.
它可能是一个很好的教育工具和诸如此类的东西，但事实证明计算机的使用极其难以解决。
So I’m trying to frame the cynical case in all these domains where there’s not a really huge economic impact, but realize how costly it is to train these systems at every level—both the pre-training and the inference, how costly the inference is, the reasoning, all of that.
因此，我试图在所有这些领域中构建一个愤世嫉俗的案例，这些领域不会产生真正巨大的经济影响，但要意识到在各个级别上训练这些系统的成本有多大——包括预训练和推理，推理、推理等等的成本有多大。
Is that possible?
这可能吗？
And how likely is that, do you think?
您认为这种可能性有多大？
When you look at the models, there are so many obvious things to improve, and it takes a long time to train these models and to do this art, that it’ll take us with the ideas that we have multiple years to actually saturate in terms of whatever benchmark or performance we are searching for.
当你查看这些模型时，你会发现有很多明显的事情需要改进，并且需要很长时间来训练这些模型并完成这项艺术，这需要我们花很多年的时间才能在我们正在寻找的任何基准或性能方面真正达到饱和。
3:24:47 It might serve very narrow niches.
它可能服务于非常狭窄的利基市场。
The average ChatGPT user might not get a lot of benefit out of this, but it is going to serve different populations by getting better at different things.
ChatGPT 的普通用户可能不会从中获得很多好处，但它将通过在不同的事情上做得更好来服务不同的人群。
But I think what everybody’s chasing now is a general system that’s useful to everybody.
但我认为现在每个人都在追求的是一个对每个人都有用的通用系统。
3:25:18 So, okay, if that’s not…
所以，好吧，如果不是的话……
that can plateau, right?
那可以稳定下来，对吧？
I think that dream is actually kind of dying.
我认为这个梦想实际上已经快要消亡了。
3:25:28 As you talked about with the specialized models where it’s like…
正如您谈到的专业模型，就像......
and multimodal is often…
多式联运通常是……
like, video generation is a totally different thing.
就像，视频生成是完全不同的事情。
“That dream is kind of dying” is a big statement, because I don’t know if it’s dying.
“那个梦想快要消亡了”是一个很大的说法，因为我不知道它是否正在消亡。
3:25:39 If you ask the actual Frontier Lab people, they’re still chasing it, right?
如果你问前沿实验室的实际人员，他们仍然在追逐它，对吗？
I do think they are still rushing to get the next model out, which will be much better than the previous one.
我确实认为他们仍在急于推出下一个模型，这将比上一个模型好得多。
3:25:48 “Much” is a relative term, but it will be better than the previous one.
“很多”是一个相对词，但会比前一个更好。
I can’t see them slowing down.
我看不到他们放慢脚步。
I just think the gains will be made or felt more through not only scaling the model, but now…
我只是认为，不仅通过扩展模型，还会获得或感受到更多的收益，但现在……
I feel like there’s a lot of tech debt.
我觉得有很多技术债务。
It’s like, “Well, let’s just put the better model in there, and better model, better model.” And now people are like, “Okay, let’s also at the same time improve everything around it too.”
就像，“好吧，让我们把更好的模型放在那里，更好的模型，更好的模型。”现在人们会说，“好吧，让我们同时改进它周围的一切。”
3:26:20 Like the engineering of the context and inference scaling.
就像上下文和推理缩放的工程一样。
And the big labs will still keep doing that.
大型实验室仍将继续这样做。
And now also the smaller labs will catch up to that because now they are hiring more.
现在，较小的实验室也将迎头赶上，因为他们现在正在招聘更多人。
There will be more people.
会有更多的人。
LLMs, it’s kind of like a circle.
法学硕士，它有点像一个圆圈。
They also make them more productive and it’s just like an amplifier.
它们还可以提高他们的工作效率，就像放大器一样。
I think what we can expect is amplification, but not a paradigm change.
我认为我们可以期待的是放大，但不是范式改变。
I don’t think that is true, but everything will be just amplified and amplified and amplified, and I can see that continuing for a long time.
我不认为这是真的，但一切都会被放大、放大、放大，而且我可以看到这种情况会持续很长一段时间。
Yeah.
是的。
3:26:52 I guess my statement with the dream is dying depends on exactly what you think it’s going to be doing.
我想我对梦想即将消亡的陈述取决于你认为它会做什么。
Like Claude Code is a general model that can do a lot of things, but it depends a lot on integrations and other things.
像 Claude Code 一样，它是一个通用模型，可以做很多事情，但它很大程度上取决于集成和其他事情。
I bet Claude Code could do a fairly good job of doing your email, and the hardest part is figuring out how to give it information and how to get it to be able to send your emails and stuff like this.
我敢打赌 Claude Code 可以很好地处理你的电子邮件，而最困难的部分是弄清楚如何向它提供信息以及如何让它能够发送你的电子邮件和类似的东西。
But I think it goes back to what is the “one model to rule everything” ethos, which is just like a thing in the cloud that handles your entire digital life and is way smarter than everybody.
但我认为这可以追溯到“一个模型统治一切”的精神，这就像云中的一个东西，可以处理你的整个数字生活，并且比每个人都聪明得多。
3:27:34 So it’s an interesting leap of faith to go from Claude Code becomes that—which, in some ways, there are some avenues for that—but I do think that the rhetoric of the industry is a little bit different.
因此，从 Claude Code 转变为这样的信念是一个有趣的飞跃——在某些方面，有一些途径可以实现这一点——但我确实认为行业的说法有点不同。
I think the immediate thing we will feel next as a normal person using LLMs will probably be related to something trivial, like making figures.
我认为，作为一个使用法学硕士的普通人，我们接下来感受到的直接事情可能与一些琐碎的事情有关，比如制作数字。
3:27:49 Right now LLMs are terrible at making figures.
现在法学硕士在计算数字方面很糟糕。
Is it because we are getting served the cheap models with less inference compute than behind the scenes?
是因为我们得到的廉价模型比幕后的推理计算更少吗？
Maybe with some cranks we can already get better figures, but if you ask today to draw a flowchart of X, Y, Z, it’s most of the time terrible.
也许有了一些奇才，我们已经可以得到更好的数字，但如果你今天要求画一个 X、Y、Z 的流程图，大多数时候都是很糟糕的。
And it is kind of a very simple task for a human.
对于人类来说，这是一项非常简单的任务。
I think it’s almost easier sometimes to draw something than to write something.
我认为有时画一些东西比写一些东西更容易。
Yeah, the multimodal understanding does feel like something that is odd, that it’s not better solved.
是的，多模态理解确实感觉很奇怪，它没有得到更好的解决。
I think we’re not saying one actually obvious thing that we’re not realizing, that’s a gigantic thing that’s hard to measure, which is making all of human knowledge accessible…
我认为我们并不是在说一件我们没有意识到的明显的事情，这是一件难以衡量的巨大事情，它使所有人类知识变得可用......
3:28:31 …To the entire world.
......对整个世界。
One of the things that I think is hard to articulate, but there’s just a huge difference between Google Search and an LLM.
我认为其中一件事很难说清楚，但谷歌搜索和法学硕士之间存在巨大差异。
I feel like I can basically ask an LLM anything and get an answer, and it’s doing less and less hallucination.
我觉得我基本上可以向法学硕士询问任何事情并得到答案，而且它的幻觉越来越少。
3:29:04 And that means understanding my own life, figuring out a career trajectory, figuring out how to solve the problems all around me, learning about anything through human history.
这意味着了解我自己的生活，弄清楚职业轨迹，弄清楚如何解决我周围的问题，通过人类历史了解任何事情。
I feel like nobody’s really talking about that because they just immediately take it for granted that it’s awesome.
我觉得没有人真正谈论这一点，因为他们立即认为这是理所当然的，这太棒了。
That’s why everybody’s using it—it’s because you get answers for stuff, and think about the impact of that across time.
这就是为什么每个人都在使用它——因为你可以获得一些问题的答案，并考虑随着时间的推移它的影响。
This is not just in the United States; this is all across the world.
这不仅发生在美国，也发生在美国。世界各地都是如此。
Kids throughout the world being able to learn these ideas—the impact that has across time is probably where the real GDP growth will be.
世界各地的孩子们都能够学习这些想法——随着时间的推移所产生的影响可能就是真正的 GDP 增长。
It won’t be like a leap.
这不会像一个飞跃。
3:29:51 It’ll be that that’s how we get to Mars, that’s how we build these things, that’s how we have a million new OpenAIs, all the kind of innovation that happens from there.
这就是我们到达火星的方式，这就是我们构建这些东西的方式，这就是我们拥有一百万个新的 OpenAI 的方式，所有的创新都是从那里发生的。
And that’s just this quiet force that permeates everything, right?
这就是渗透到一切事物中的安静力量，对吗？
Human knowledge.
人类知识。
I do agree with you, and in a sense it makes knowledge more accessible, but it also depends on what the topic is.
我确实同意你的观点，从某种意义上说，它使知识更容易获得，但这也取决于主题是什么。
3:30:06 For something like math, you can ask it questions and it answers, but if you want to learn a topic from scratch—we talked about this earlier—I think the sweet spot is still math textbooks where someone laid it out linearly.
对于数学之类的东西，你可以问它问题，它会回答，但如果你想从头开始学习一个主题——我们之前讨论过这一点——我认为最好的地方仍然是数学教科书，有人线性地布置它。
That is a proven strategy to learn a topic, and it makes sense if you start from zero to get information-dense text to soak it up, but then you use the LLM to make infinite exercises.
这是一个经过验证的学习主题的策略，如果你从零开始获取信息密集的文本来吸收它，那么这是有意义的，但随后你使用法学硕士进行无限的练习。
3:30:47 If you have problems in a certain area or have questions about things you are uncertain about, you ask it to generate example problems, you solve them, and then maybe you need more background knowledge and you ask it to generate that.
如果你在某个领域遇到问题，或者对不确定的事情有疑问，你可以要求它生成示例问题，你解决它们，然后也许你需要更多的背景知识，你可以要求它生成这些问题。
But it won’t give you anything that is not in the textbook.
但它不会给你任何教科书上没有的东西。
It’s just packaging it differently, if that makes sense.
如果有意义的话，它只是以不同的方式包装它。
3:31:13 But then there are things where it also adds value in a more timely sense, where there is no good alternative besides a human doing it on the fly.
但有些事情也可以更及时地增加价值，除了人类即时执行之外，没有其他好的选择。
For example, if you’re planning to go to Disneyland and you try to figure out which tickets to buy for which park when, well, there is no textbook on that.
例如，如果您打算去迪士尼乐园，并且试图弄清楚要为哪个公园购买哪些门票，但没有这方面的教科书。
There is no information-dense resource on that.
没有这方面的信息密集资源。
There’s only the sparse internet, and then there is a lot of value in the LLM.
只有稀疏的互联网，然后法学硕士有很多价值。
You just ask it.
你只要问它。
You have the constraints on traveling on these specific days, you want to go to certain places, and you ask it to figure out what you need, when and from where…
你在这些特定的日子里旅行受到限制，你想去某些地方，你要求它弄清楚你需要什么、何时何地......
…What it costs and stuff like that.
......它的成本是多少等等。
It is a very customized, on-the-fly package.
这是一个非常定制的即时包。
Personalization is essentially like—
个性化本质上就像——
3:32:02 …pulling information from the sparse internet, the non-information-dense thing where there’s no better version that exists.
...从稀疏的互联网中获取信息，这是一种非信息密集型的东西，没有更好的版本存在。
You make it from scratch almost.
你几乎是从头开始做的。
And if it does exist, it’s full of—speaking of Disney World—ad slop.
即使它确实存在，也充满了——说到迪士尼世界——广告。
3:32:12 Like any city in the world, if you ask “what are the top 10 things to do?” An LLM is just way better to ask…
就像世界上任何一个城市一样，如果你问“最值得做的十件事是什么？” LLM 更好地要求......
…Than anything on the internet.
……比互联网上的任何东西都重要。
Well, for now, that’s because they’re massively subsidized, and eventually they’re going to be paid for by ads.
好吧，就目前而言，这是因为它们得到了大量补贴，最终它们将通过广告来支付费用。
Oh my goodness.
哦，我的天啊。
It’s coming.
它来了。
No.
不。
3:32:38 I’m hoping there’s a very clear indication of what’s an ad and what’s not an ad in that context, but—
我希望在这种情况下能够非常清楚地表明什么是广告、什么不是广告，但是——
That’s something I mentioned a few years ago.
这是我几年前提到过的事情。
3:32:46 It’s like, I don’t know, if you are looking for a new running shoe, is it a coincidence that Nike maybe comes up first?
就像，我不知道，如果你正在寻找一款新的跑鞋，耐克可能会先出现，这是否是巧合？
Maybe, maybe not.
也许，也许不是。
I think there are clear laws around this.
我认为这方面有明确的法律。
You have to be clear about that, but I think that’s what everyone fears.
你必须清楚这一点，但我认为这是每个人都担心的。
It’s like the subtle message in there or something like that.
就像那里微妙的信息或类似的东西。
But also, this brings us to the topic of ads where, I think this was a thing, hopefully they try to launch in 2025 because I think they’re still not making money in that other way right now, so…
而且，这也让我们谈到了广告的话题，我认为这是一件事，希望他们尝试在 2025 年推出，因为我认为他们现在仍然没有以其他方式赚钱，所以……
…Like having actual ad spots in there.
…就像那里有实际的广告位一样。
And then the thing, though, is they couldn’t because there are alternatives without ads and people would just flock-
但问题是他们不能，因为有没有广告的替代品，人们就会蜂拥而至——
3:33:31 …to the other products.
...其他产品。
And it also is just crazy how they’re one-upping each other, spending so much money just to get the users.
他们互相比拼，花这么多钱只是为了获得用户，这也太疯狂了。
I think so.
我想是的。
3:33:41 Like some Instagram ads—I don’t use Instagram- …but I understand the appeal of paying a platform to find users who will genuinely like your product.
就像一些 Instagram 广告一样——我不使用 Instagram——……但我理解付费平台来寻找真正喜欢你的产品的用户的吸引力。
That is the best case of things like Instagram ads.
这是 Instagram 广告等最好的例子。
3:33:56 But there are also plenty of cases where advertising is very awful for incentives.
但在很多情况下，广告对于激励来说是非常糟糕的。
I think that a world where the power of AI can integrate with that positive view—like, I am a person and I have a small business and I want to make the best damn steak knives in the world and I want to sell them to somebody who needs them.
我认为人工智能的力量可以与积极的观点相结合——就像，我是一个人，我有一家小企业，我想制造世界上最好的该死的牛排刀，我想把它们卖给需要它们的人。
And if AI can make that sort of advertising work even better, that’s very good for the world, especially with digital infrastructure because that’s how the modern web has been built.
如果人工智能能让这种广告效果更好，那对世界来说非常有好处，特别是在数字基础设施方面，因为现代网络就是这样构建的。
But that’s not to say that addicting feeds so that you can show people more content is a good thing.
但这并不是说，让用户上瘾的信息流可以向人们展示更多内容是一件好事。
So, I think that’s even what OpenAI would say is they want to find a way that can make the monetization upside of ads while still giving their users agency.
所以，我认为 OpenAI 甚至会说，他们希望找到一种方法，既可以提高广告的盈利能力，同时仍为用户提供代理权。
3:34:45 And I personally would think that Google is probably going to be better at figuring out how to do this because they already have ad supply.
我个人认为谷歌可能会更好地弄清楚如何做到这一点，因为他们已经拥有广告供应。
If they figure out how to turn this demand in their Gemini app into useful ads, then they can turn it on.
如果他们弄清楚如何将 Gemini 应用程序中的这一需求转化为有用的广告，那么他们就可以将其打开。
I don’t know if I think it’s this year, but there will be experiments with it.
我不知道我是否认为是今年，但会有相关实验。
I do think what holds companies back right now is really just that the competition is not doing it.
我确实认为，目前阻碍公司前进的真正原因是竞争对手没有这样做。
3:35:06 It’s more like a reputation thing.
这更像是一个声誉的事情。
I think people are just afraid right now of ruining their reputation or losing users- …because it would make headlines if someone launched these ads.
我认为人们现在只是害怕毁掉自己的声誉或失去用户——……因为如果有人推出这些广告，就会成为头条新闻。
But-
但是——
Unless they were great, but the first ads won’t be great because it’s a hard problem that we don’t know how to solve.
除非它们很棒，但第一个广告不会很棒，因为这是一个我们不知道如何解决的难题。
Yeah, I think also the first version of that will likely be something like on X, like the timeline where you have a promoted post sometimes in between.
是的，我认为第一个版本也可能类似于 X 上的内容，就像有时在两者之间有晋升职位的时间线。
3:35:28 It’ll be something like that where it will say “promoted” or something small, and then there will be an image or something.
它会是类似的东西，上面会写着“促销”或一些小东西，然后会有一个图像或其他东西。
I think right now the problem is who makes the first move.
我认为现在的问题是谁先采取行动。
If we go 10 years out, the proposition for ads is that you will make so much money on ads by having so many users- …that you can use this to fund better R&D and- …make better models, which is why- …like YouTube is dominating the market.
如果我们展望 10 年后，广告的主张是，通过拥有如此多的用户，您将在广告上赚到很多钱 - ……您可以用它来资助更好的研发，并且 - ……制作更好的模型，这就是为什么 - ……就像 YouTube 主导市场一样。
3:35:43 Netflix is scared of YouTube.
Netflix 害怕 YouTube。
They make, I don’t know—I pay $28 a month for premium.
他们赚的钱，我不知道——我每个月支付 28 美元的保费。
They make at least $28 a month off of me and many other people, and they’re just creating such a dominant position in video.
他们每个月从我和其他许多人那里至少赚 28 美元，而他们只是在视频领域创造了如此主导地位。
So I think that’s the proposition, which is that ads can give you a sustained advantage- …in what you’re spending per user.
所以我认为这就是命题，即广告可以给你带来持续的优势——……在每个用户的支出方面。
But there’s so much money in it right now that it’s like somebody starting that flywheel- is scary because it’s a long-term bet.
但现在里面有这么多钱，就像有人启动飞轮一样——很可怕，因为这是一个长期的赌注。
Do you think there’ll be some crazy big moves this year business-wise?
您认为今年商业方面会出现一些疯狂的大动作吗？
3:36:29 Like Google or Apple acquiring Anthropic or something like this?
就像谷歌或苹果收购 Anthropic 之类的？
Dario will never sell, but we are starting to see some types of consolidation, with Groq being valued at $20 billion and Scale AI for almost 30 billion.
Dario 永远不会出售，但我们开始看到一些类型的整合，Groq 的估值为 200 亿美元，Scale AI 的估值接近 300 亿美元。
3:36:40 There are countless other deals structured in a way that is actually detrimental to the Silicon Valley ecosystem—these licensing deals where not everybody gets brought along, rather than a full acquisition that benefits the rank-and-file employees by getting their stock vested.
还有无数其他交易的结构实际上对硅谷生态系统有害——这些许可交易并不是所有人都参与其中，而不是通过授予普通员工股票来使普通员工受益的全面收购。
That’s a big issue for Silicon Valley culture to address because the startup ecosystem is the lifeblood.
这是硅谷文化需要解决的一个大问题，因为创业生态系统是生命线。
If you join a startup, even if it’s not that successful, your startup very well might get acquired at a cheap premium and you’ll get paid out for your equity.
如果你加入一家初创公司，即使它没有那么成功，你的初创公司很可能会以低廉的溢价被收购，并且你会得到你的股权报酬。
3:37:24 And these licensing deals are essentially taking the top talent a lot of the time.
这些许可交易本质上占用了顶尖人才很多时间。
I think the deal for Groq to NVIDIA is rumored to be better for the employees, but it is still this antitrust-avoiding thing.
我认为据传 Groq 与 NVIDIA 的交易对员工来说更好，但这仍然是一个避免反垄断的事情。
I think that this trend of consolidation will continue.
我认为这种整合趋势将会持续下去。
Me and many smart people I respect have been expecting consolidation to have happened sooner, but it seems like things are starting to turn.
我和我尊敬的许多聪明人一直期待整合会更早发生，但情况似乎开始出现转变。
But at the same time, you have companies raising ridiculous amounts of money for reasons that I don’t understand.
但与此同时，有些公司出于我不明白的原因筹集了巨额资金。
I’m like, “I don’t know why you’re taking that money.” So it’s maybe mixed this year, but some consolidation pressure is starting.
我说：“我不知道你为什么要拿这笔钱。”因此，今年的情况可能好坏参半，但一些整合压力正在开始。
What kind of surprising consolidation do you think we’ll see?
您认为我们会看到什么样的令人惊讶的整合？
3:38:04 You say Anthropic is a “never.” I mean, Groq is a big one—Groq with a Q, by the way.
你说人择是“从来没有”。我的意思是，Groq 是一个大产品——顺便说一句，Groq 带有 Q。
Yeah.
是的。
3:38:12 There’s just a lot of startups and there’s a very high premium on AI startups.
这里有很多初创公司，而且人工智能初创公司的溢价非常高。
So there could be a lot of $10 billion range acquisitions, which is a really big acquisition for a startup that was maybe founded a year ago.
因此，可能会有大量 100 亿美元的收购，这对于一家可能成立一年前的初创公司来说是一笔非常大的收购。
I think Manus.ai—this company based in Singapore that was founded eight months ago and then had a $2 billion exit.
我想到的是 Manus.ai，这家总部位于新加坡的公司成立于 8 个月前，然后退出了 20 亿美元。
I think there will be some other big multi-billion dollar acquisitions, like Perplexity.
我认为还会有其他一些价值数十亿美元的大型收购，比如 Perplexity。
Like Perplexity, right?
就像困惑一样，对吧？
Yeah, people rumor them to Apple.
是的，人们向苹果公司传言。
3:38:40 I think there’s a lot of pressure and liquidity in AI.
我认为人工智能有很大的压力和流动性。
There’s pressure on big companies to have outcomes, and I would guess that a big acquisition gives people leeway to then tell the next chapter of that story.
大公司面临着取得成果的压力，我猜想，大型收购会给人们留下讲述故事下一章的余地。
I mean, yeah, we’ve been talking about code.
我的意思是，是的，我们一直在谈论代码。
3:38:56 Maybe somebody acquires Cursor.
也许有人获得了 Cursor。
They’re in such a good position because they have so much user data.
他们处于如此有利的地位，因为他们拥有如此多的用户数据。
3:39:02 And we talked about continual learning and stuff; they had one of the most interesting blog posts.
我们讨论了持续学习之类的事情；他们有最有趣的博客文章之一。
They mentioned that their new Composer model was a fine-tune of one of these large Mixture of Experts models from China.
他们提到，他们的新 Composer 模型是对来自中国的大型专家混合模型之一的微调。
You can know that from gossip or because the model sometimes responds in Chinese, which none of the American models do.
你可以从八卦中知道这一点，或者因为模特有时会用中文回应，而美国模特却没有这样做。
They had a blog post where they said, “We’re updating the model weights every 90 minutes based on real-world feedback from people using it.” Which is the closest thing to real-world RL happening on a model, and it was just right there in one of their blog posts.
他们在一篇博文中表示，“我们会根据用户的真实反馈，每 90 分钟更新一次模型权重。”这是模型上发生的最接近现实世界强化学习的事情，而且就在他们的一篇博客文章中。
That’s incredible.
那真是难以置信。
—which is super cool.
——这太酷了。
And by the way, I should say I use Composer a lot because one of the benefits it has is it’s fast.
顺便说一句，我应该说我经常使用 Composer，因为它的好处之一就是速度快。
I need to try it because everybody says this.
我需要尝试一下，因为每个人都这么说。
And there’ll be some IPOs potentially.
并且可能会有一些首次公开募股。
3:39:45 You think Anthropic, OpenAI, xAI?
你认为 Anthropic、OpenAI、xAI 吗？
They can all raise so much money so easily that they don’t feel a need to…
他们都可以轻松筹集这么多钱，以至于他们觉得没有必要……
3:39:51 So long as fundraising is easy, they’re not going to IPO because public markets apply pressure.
只要融资很容易，他们就不会进行首次公开募股，因为公开市场会施加压力。
3:40:00 I think we’re seeing in China that the ecosystem’s a little different, with both MiniMax and Z.ai applying for filing IPO paperwork, which will be interesting to see how the Chinese market reacts.
我认为我们在中国看到的生态系统有点不同，MiniMax 和 Z.ai 都在申请提交 IPO 文件，看看中国市场的反应将会很有趣。
I actually would guess that it’s going to be similarly hypey to the US so long as all this is going, and not based on the realities that they’re both losing a ton of money.
事实上，我猜想，只要这一切继续下去，美国就会受到同样的炒作，而不是基于他们都损失大量资金的现实。
I wish more of the American gigantic AI startups were public because it would be very interesting to see how they’re spending their money and have more insight.
我希望更多的美国大型人工智能初创公司上市，因为了解他们如何花钱并获得更多洞察力会非常有趣。
And also just to give people access to investing in these, because I think that they’re the companies of the era.
而且只是为了让人们有机会投资这些，因为我认为它们是这个时代的公司。
And the tradition is now for so many of the big startups in the US to not go public.
现在，美国许多大型初创企业都遵循不上市的传统。
3:40:43 It’s like we’re still waiting for Stripe and their IPO, but Databricks definitely didn’t; they raised like a Series G or something.
就好像我们仍在等待 Stripe 及其 IPO，但 Databricks 绝对没有；他们像 G 轮融资之类的筹集资金。
And I just feel like it’s a kind of a weird equilibrium for the market where I would like to see these companies go public and evolve in that way that a company can.
我只是觉得这对市场来说是一种奇怪的平衡，我希望看到这些公司上市并以公司可以的方式发展。
You think 10 years from now some of the frontier model companies are still around?
您认为十年后一些前沿模型公司仍然存在吗？
3:41:01 Anthropic, OpenAI?
人类，OpenAI？
I definitely don’t see it to be a winner-takes-all unless there truly is some algorithmic secret that one of them finds that lets this flywheel.
我绝对不认为这是赢者通吃，除非他们中的一个人确实发现了一些算法秘密可以让这个飞轮发挥作用。
3:41:08 Because the development path is so similar for all of them.
因为他们的发展道路都非常相似。
Google and OpenAI have all the same products, and Anthropic’s more focused, but when you talk to people, it sounds like they’re solving a lot of the same problems.
Google 和 OpenAI 拥有相同的产品，而 Anthropic 更专注，但当你与人们交谈时，听起来他们正在解决很多相同的问题。
So I think…
所以我认为……
there’s offerings that’ll spread out.
有一些产品会散布开来。
It’s a very big cake that’s being made that people are going to take money out of.
这是一块非常大的蛋糕，人们将从中捞钱。
I don’t want to trivialize it, but OpenAI and Anthropic are primarily LLM service— —providers.
我不想轻视它，但 OpenAI 和 Anthropic 主要是 LLM 服务提供商。
3:41:36 And some of the other companies like Google and xAI, linked to X, do other stuff— —too.
其他一些公司，如谷歌和 xAI，与 X 相关，也做其他事情。
And so it’s very possible if AI becomes more commodified that the companies that are just providing LLMs will die.
因此，如果人工智能变得更加商品化，那些只提供法学硕士的公司很可能会消亡。
I think the advantage they have is they have a lot of users, and I think they will just pivot.
我认为他们的优势是拥有很多用户，而且我认为他们只会转向。
3:42:00 Like Anthropic, I think, pivoted.
我认为，就像“人择”一样，它也发生了转变。
I don’t think they originally planned to work on code, but it happened that they found, “Okay, this is a nice niche and now we are comfortable in this niche and we push on this niche.” And I can see the same thing once…
我不认为他们最初计划从事代码工作，但碰巧他们发现，“好吧，这是一个很好的利基市场，现在我们在这个利基市场中感到很舒服，并且我们正在推动这个利基市场。”我曾经可以看到同样的事情......
Let’s say hypothetically speaking, I’m not sure if it will be true, but let’s say Google takes all the market share of the general chatbot.
假设来说，我不确定这是否属实，但假设谷歌占据了通用聊天机器人的所有市场份额。
Maybe OpenAI will then be focused on some other sub-topic— —like…
也许 OpenAI 之后会专注于其他一些子主题——比如……
They have too many users to go away in the foreseeable future, I think.
我认为他们有太多的用户，在可预见的未来不会消失。
I think Google is always ready to say, “Hold my beer,” with AI mode.
我认为谷歌总是准备好用人工智能模式说“别喝啤酒了”。
I think the question is if the companies can support the valuations.
我认为问题是这些公司是否能够支持估值。
3:42:40 I’d see the AI companies being looked at in some ways like AWS, Azure, and GCP, which are all competing in the same space and all very successful businesses.
我会看到人工智能公司在某些方面受到关注，比如 AWS、Azure 和 GCP，它们都在同一领域竞争，而且都是非常成功的企业。
There’s a chance that the API market is so unprofitable that they go up and down the stack to products and hardware.
API 市场有可能非常无利可图，以至于他们在产品和硬件堆栈上上下波动。
They have so much cash that they can build power plants and build data centers, which is a durable advantage now.
他们拥有大量现金，可以建造发电厂和数据中心，这现在是一个持久的优势。
But there’s also just a reasonable outcome that these APIs are so valuable and so flexible for developers that they become the likes of something like AWS.
但也有一个合理的结果，即这些 API 对于开发人员来说非常有价值且非常灵活，以至于它们变得像 AWS 这样的东西。
But AWS and Azure are also going to have these APIs, so five or six people competing in the API market is hard.
但AWS和Azure也会有这些API，所以五六个人在API市场竞争是很困难的。
So maybe that’s why they get squeezed out.
所以也许这就是他们被排挤的原因。
You mentioned “RIP LLaMA.” Is there a path to winning for Meta?
你提到了“RIP LLaMA”。 Meta 有获胜之路吗？
I think nobody knows.
我想没人知道。
3:43:32 They’re moving a lot, so they’re signing licensing deals with Black Forest Labs, which is image generation, or Midjourney.
他们搬家很多，所以他们与黑森林实验室签署了许可协议，这是图像生成，或中途。
So I think in some ways, on the product and consumer-facing AI front, it’s too early to tell.
因此，我认为在某些方面，在产品和面向消费者的人工智能方面，现在下结论还为时过早。
I think they have some people that are excellent and very motivated being close to Zuckerberg.
我认为他们有一些非常优秀并且非常有动力与扎克伯格关系密切的人。
So I think that there’s still a story to unfold there.
所以我认为那里还有一个故事要展开。
Llama is a bit different, where Llama was the most focused expression of the organization.
Llama 有点不同，Llama 是该组织最集中的表达方式。
And I don’t see Llama being supported to that extent anymore.
我认为 Llama 不再受到这种程度的支持。
I think it was a very successful brand for them, so they still might do some part of participation in the open ecosystem or continue the Llama brand into a different service, because people know what Llama is.
我认为这对他们来说是一个非常成功的品牌，所以他们仍然可能会参与开放生态系统的一部分，或者将 Llama 品牌延续到不同的服务中，因为人们知道 Llama 是什么。
You think there’s a Llama 5?
你认为有 Llama 5 吗？
Not an open weight one.
不是一个开放重量的。
It’s interesting.
这很有趣。
3:44:26 Just to recap a bit, I mean, Llama was the pioneering open-weight model—Llama 1, 2, 3, a lot of love.
简单回顾一下，我的意思是，Llama 是开创性的开放重量模型——Llama 1、2、3，备受喜爱。
But I think then what happened, just hypothesizing or speculating, is that the leaders at Meta, like the upper executives, got very excited about Llama because they saw how popular it was in the community.
但我认为随后发生的事情（只是假设或推测）是，Meta 的领导者和高层管理人员一样，对 Llama 感到非常兴奋，因为他们看到它在社区中有多受欢迎。
And then I think the problem was trying to use the open source to make a bigger splash.
然后我认为问题在于试图利用开源来引起更大的轰动。
It felt forced, like developing these very big Llama 4 models just to be on the top of the benchmarks.
这感觉像是被迫的，就像开发这些非常大的 Llama 4 模型只是为了在基准测试中名列前茅。
3:45:09 But I don’t think the goal of Llama models is to be on top of the benchmarks beating, let’s say, ChatGPT or other models.
但我不认为 Llama 模型的目标是在基准测试中击败 ChatGPT 或其他模型。
I think the goal was to have a model that people can use, trust, modify, and understand.
我认为我们的目标是建立一个人们可以使用、信任、修改和理解的模型。
That includes having smaller models; they don’t have to be the best models.
这包括拥有更小的型号；他们不一定是最好的模特。
And what happened was just these models were—of course, the benchmarks suggest that they were better than they were because I think they had specific models trained on preferences so that they performed well on the benchmarks.
所发生的事情只是这些模型 - 当然，基准测试表明它们比实际情况更好，因为我认为它们有根据偏好进行训练的特定模型，因此它们在基准测试中表现良好。
That’s kind of this overfitting thing to force it to be the best.
这就是一种过度拟合的现象，迫使它成为最好的。
But then at the same time, they didn’t do the small models that people could use, and no one could run these big models.
但与此同时，他们没有做人们可以使用的小模型，也没有人可以运行这些大模型。
3:45:45 And then there was kind of a weird thing.
然后发生了一件奇怪的事情。
I think it’s just because people got too excited about headlines pushing the frontier.
我认为这只是因为人们对推动前沿的头条新闻太兴奋了。
I think that’s it.
我想就是这样。
And too much on the benchmark-sync side.
基准同步方面的内容太多了。
It’s too much work.
工作量太大了。
I think it imploded under internal political fighting and misaligned incentives.
我认为它是在内部政治斗争和不协调的激励措施下崩溃的。
3:45:57 The researchers want to build the best models, but there’s a layer of organization— …and management that is trying to demonstrate that they do these things.
研究人员希望建立最好的模型，但有一层组织和管理层试图证明他们做了这些事情。
There are a lot of pieces and rumors where some horrible technical decision was made, and it just seems like it got too bad where it all just crashed out.
有很多文章和谣言都做出了一些可怕的技术决定，而且看起来事情太糟糕了，一切都崩溃了。
Yeah, but we should also give huge props to Mark Zuckerberg.
是的，但我们也应该大力支持马克·扎克伯格。
3:46:24 I think it comes from Mark actually, from the top of the leadership, saying open source is important.
我认为这实际上来自马克，来自领导层的高层，他说开源很重要。
The fact that that exists means there could be a Llama 5, where they learn the lessons from the benchmark-syncing and say, “We’re going to be GPT-OSS—” “…and provide a really awesome library of open source.”
这一事实的存在意味着可能会有 Llama 5，他们从基准同步中吸取教训，并说，“我们将成为 GPT-OSS——”“……并提供一个非常出色的开源库。”
What people say is that there’s a debate between Mark and Alexander Wang, who is very bright but much more against open source.
人们说的是 Mark 和 Alexander Wang 之间存在争论，Alexander Wang 非常聪明，但更反对开源。
3:46:51 To the extent that he has a lot of influence over the AI org, it seems much less likely, because Mark brought him in for fresh leadership in directing AI.
就他对人工智能组织有很大影响力而言，这种可能性似乎要小得多，因为马克任命他为指导人工智能的新领导。
And if being open or closed is no longer the defining nature of the model, I don’t expect that to be a defining argument between Mark and Alex.
如果开放或封闭不再是模型的决定性本质，我不认为这会成为马克和亚历克斯之间的决定性争论。
They’re both very bright, but I have a hard time understanding all of it because Mark wrote this piece in July of 2024, which was probably the best blog post at the time, making the case for open source AI.
他们都很聪明，但我很难理解所有这些，因为 Mark 在 2024 年 7 月写了这篇文章，这可能是当时最好的博客文章，为开源 AI 提供了案例。
And then July 2025 came around and it was like, “We’re reevaluating our relationship with open source.” So it’s just kind of…
然后 2025 年 7 月到来了，“我们正在重新评估我们与开源的关系。”所以这只是一种……
But I think also the problem—well, we may have been a bit too harsh, and that caused some of that.
但我也认为问题是——嗯，我们可能有点太严厉了，这导致了一些问题。
3:47:42 I mean, we as open source developers or the open source community.
我的意思是，我们作为开源开发人员或开源社区。
Because even though the model was maybe not what everyone hoped for, it got a lot of backlash.
因为尽管这个模型可能不是每个人都希望的，但它还是遭到了很多反对。
I think that was a bit unfortunate because as a company, they were hoping for positive headlines.
我认为这有点不幸，因为作为一家公司，他们希望获得积极的头条新闻。
Instead of just getting no headlines or positive headlines, they got negative headlines.
他们不仅没有得到头条新闻或积极的头条新闻，反而得到了负面的头条新闻。
And then it kind of reflected badly on the company.
然后这对公司产生了不好的影响。
It’s maybe a spite reaction, almost like, “Okay, we tried to do something nice, we tried to give you something cool like an open source model, and now you are being negative about us.” So in that sense, it looks like, “Well, maybe then we’ll change our mind.” I don’t know.
这可能是一种恶意反应，几乎就像是，“好吧，我们试图做一些好事，我们试图给你一些很酷的东西，比如开源模型，但现在你对我们持负面态度。”因此，从这个意义上说，“好吧，也许我们会改变主意。”我不知道。
Yeah, that’s where the dynamics of discourse on— …X can lead us as a community astray.
是的，这就是关于……X的动态讨论可能会让我们作为一个社区误入歧途。
3:48:38 Because sometimes it feels random; people pick the things they like and don’t like.
因为有时感觉是随机的；人们会选择自己喜欢和不喜欢的东西。
I mean, you can see the same thing with Grok 4.1 and Grok Code Fast 1.0.
我的意思是，您可以在 Grok 4.1 和 Grok Code Fast 1.0 中看到同样的情况。
I don’t think, vibe-wise, people love it publicly.
我不认为，就氛围而言，人们不会公开喜欢它。
But a lot of people use it.
但很多人都在使用它。
So if you look at Reddit and X, they don’t really give it praise from the programming community— …
所以如果你看看 Reddit 和 X，他们并没有真正给予它来自编程社区的赞扬——……
but, like, they use it.
但是，就像他们使用它一样。
And the same thing with probably Llama.
骆驼可能也是同样的情况。
I don’t understand the dynamics of either positive hype or negative hype.
我不明白积极炒作或消极炒作的动态。
I don’t understand it.
我不明白。
I mean, one of the stories of 2025 is the US filling the gap of Llama, which is all the rise of these Chinese open-weight models- …
我的意思是，2025 年的故事之一是美国填补了 Llama 的空白，这都是这些中国开放重量模型的崛起——……
3:49:25 to the point where I was like, “That was the single issue I’ve spent a lot of energy on in the last five months,” which is trying to do policy work- …
以至于我当时想，“这是我在过去五个月里花了很多精力解决的唯一问题”，它正在努力做政策工作——……
to get the US to invest in this.
让美国投资于此。
So just tell me the story of Adam.
告诉我亚当的故事吧。
Adam Project is…
亚当项目是……
3:49:43 It started as me calling it the American DeepSeek Project, which doesn’t really work for DC audiences, but it’s the story of what is the most impactful thing I can do with my career.
一开始我把它称为“美国 DeepSeek 项目”，这个项目并不适合 DC 观众，但它讲述了我在职业生涯中能做的最有影响力的事情。
These Chinese open-weight models are cultivating a lot of power and there is a lot of demand for building on open models, especially in enterprises in the US that are very cagey about Chinese models.
这些中国的开放重量模型正在培育很大的力量，并且对开放模型的需求也很大，尤其是对中国模型非常谨慎的美国企业。
Looking at Perplexity, The Adam Project—American Truly Open Models—is a US-based initiative to build and host high-quality, genuinely open-weight AI models and supporting infrastructure explicitly aimed at competing with and catching up to China’s rapidly advancing open-source AI ecosystem.
看看 Perplexity，Adam 项目——美国真正的开放模型——是一项总部位于美国的倡议，旨在构建和托管高质量、真正开放的人工智能模型和支持基础设施，明确旨在与中国快速发展的开源人工智能生态系统竞争和追赶。
I think the one-sentence summary would be that—or two sentences.
我认为一句话总结就是这样——或者两句话。
3:50:25 One is a proposition that open models are going to be an engine for AI research because that is what people start with; therefore, it’s important to own them.
其中一个主张是，开放模型将成为人工智能研究的引擎，因为这是人们的起点；因此，拥有它们很重要。
And the second one is, therefore, the US should be building the best models so that the best research happens in the US and those US companies take the value from being the home of where AI research is happening.
因此，第二个问题是，美国应该建立最好的模型，以便最好的研究发生在美国，而这些美国公司可以从人工智能研究发源地中获得价值。
Without more investment in open models, we have all the plots on the website where it’s like, “Qwen, Qwen, Qwen, Qwen,” and it’s all these models that are excellent from these Chinese companies that are cultivating influence in the US and internationally.
如果没有对开放模型进行更多投资，我们网站上的所有情节都是“Qwen，Qwen，Qwen，Qwen”，而所有这些模型都是这些中国公司在美国和国际上培养影响力的优秀模型。
3:51:07 And the US is spending way more on AI.
美国在人工智能上的投入要多得多。
The ability to create open models that are half a generation or a generation beyond what the cutting edge of closed labs is costs roughly $100 million, which is a lot of money, but not compared to what these companies have.
创建比封闭实验室尖端技术高出半代或一代的开放模型的能力大约需要 1 亿美元，这是一大笔钱，但与这些公司所拥有的相比还无法相比。
Therefore, we need a centralizing force of people who want to do this.
因此，我们需要一支集中力量来做到这一点。
I think we got signed engagement from people pretty much across the full stack, including policy.
我认为我们得到了几乎整个堆栈的人们的签名参与，包括政策。
So there has been support from the administration?
那么政府有支持吗？
I don’t think anyone technically in government has signed it publicly, but I know that people that have worked in AI policy, both in the Biden and Trump administrations, are very supportive of trying to promote open-source models in the US.
我认为政府中技术上没有任何人公开签署过该协议，但我知道在拜登和特朗普政府中从事人工智能政策工作的人都非常支持在美国推广开源模式。
3:51:36 I think, for example, AI2 got a grant from the NSF for $100 million over four years, which is the biggest CS grant the NSF has ever awarded, for AI2 to attempt this, and I think it’s a starting point.
我认为，例如，AI2 在四年内从 NSF 获得了 1 亿美元的资助，这是 NSF 有史以来授予的最大的 CS 资助，对于 AI2 来说，这是一个尝试，我认为这是一个起点。
But the best results happen when there are multiple organizations building models because they can cross-pollinate ideas and build this ecosystem.
但当有多个组织构建模型时，就会出现最好的结果，因为它们可以交叉传播想法并构建这个生态系统。
It doesn’t work if it’s just Llama releasing models to the world, because Llama could go away.
如果只是 Llama 向世界发布模型，这是行不通的，因为 Llama 可能会消失。
The same thing applies for AI2; I can’t be the only one building models.
同样的情况也适用于 AI2；我不可能是唯一一个建造模型的人。
3:52:24 It becomes a lot of time spent on talking to people, whether they’re in policy…
人们花费大量时间与人们交谈，无论他们是否参与政策……
I know NVIDIA is very excited about this.
我知道 NVIDIA 对此非常兴奋。
I think Jensen Huang has been specifically talking about the urgency for this, and they’ve done a lot more in 2025, where the Nemotron 3 models are more of a focus.
我认为黄仁勋已经专门谈到了这一点的紧迫性，他们在 2025 年做了更多的工作，其中 Nemotron 3 模型更受关注。
They’ve started releasing some data along with NVIDIA’s open models and very few companies do this, especially of NVIDIA’s size.
他们已经开始随 NVIDIA 的开放模型一起发布一些数据，但很少有公司这样做，尤其是像 NVIDIA 这样的规模。
So there are signs of progress.
所以有进步的迹象。
We hear about Reflection AI where they say their two-billion-dollar fundraise is dedicated to building US open models, and their announcement tweet reads like a cultural tide starting to turn.
我们听说 Reflection AI，他们说他们的 20 亿美元筹款致力于建立美国开放模型，他们的公告推文读起来就像文化潮流开始转向。
3:53:09 I think in July was when we had four or five DeepSeek-caliber Chinese open-weight models and zero from the US.
我认为 7 月份我们有四到五个 DeepSeek 口径的中国开放重量模型，而来自美国的模型为零。
That’s the moment where I released this and was like, “I guess I have to spend energy on this because nobody else is gonna do it.” So it takes a lot of people contributing together.
那是我发布这个的那一刻，我想，“我想我必须在这上面花费精力，因为没有其他人会这样做。”所以需要很多人一起贡献。
I’m not saying the Adam Project is the only thing moving the ecosystem, but it’s people like me doing this sort of thing to get the word out.
我并不是说 Adam 项目是唯一推动生态系统发展的东西，但正是像我这样的人在做这种事情来传播信息。
Do you like the 2025 America’s AI Action Plan?
你喜欢2025年美国人工智能行动计划吗？
3:53:35 That includes open source stuff.
这包括开源的东西。
The White House AI Action Plan includes a dedicated section titled “Encourage Open-Source and Open-Weight AI,” defining such models and arguing they have unique value for innovation and startups.
白宫人工智能行动计划包括一个名为“鼓励开源和开放权重人工智能”的专门部分，定义了此类模型，并认为它们对创新和初创公司具有独特的价值。
Yeah.
是的。
3:53:52 I mean, the AI Action Plan is just a plan, but I think it’s maybe the most coherent policy document that has come out of the administration, and I hope that it largely succeeds.
我的意思是，《人工智能行动计划》只是一个计划，但我认为这可能是本届政府出台的最连贯的政策文件，我希望它能在很大程度上取得成功。
I know people that have worked on it.
我认识从事这方面工作的人。
The challenge is taking policy and making it real, and I have no idea how to do this as an AI researcher, but largely a lot of things in that were very real.
挑战在于制定政策并使其成为现实，作为一名人工智能研究人员，我不知道如何做到这一点，但很大程度上其中的很多事情都是非常真实的。
There’s a huge build-out of AI in the country, and while there are issues people hear about, from water use to whatever, we should be able to build things in this country without ruining places in the process.
这个国家大力发展人工智能，虽然人们听说了一些问题，从用水到其他什么问题，但我们应该能够在这个国家建造东西，而不会在这个过程中破坏地方。
It’s worthwhile to spend energy on.
花费精力是值得的。
3:54:35 I think that’s a role for the federal government.
我认为这是联邦政府的职责。
They set the agenda.
他们制定了议程。
And setting the agenda so that open-weight should be a first consideration is a large part of what they can do to get people thinking about it.
制定议程，让开放重量应该成为首要考虑因素，这是他们可以做的很大一部分事情，让人们思考这个问题。
Also, for education and talent, it’s very important.
此外，对于教育和人才来说，这非常重要。
3:54:49 Otherwise, if there are only closed models, how do you get the next generation of people contributing?
否则，如果只有封闭模型，如何让下一代人做出贡献？
You would only be able to learn after you joined a company, but at that point, how do you identify and hire talented people?
进入公司之后你才能学习，但到了这个时候，你如何识别和聘用人才呢？
I think open source is essential for educating the population and training the next generation of researchers.
我认为开源对于教育民众和培训下一代研究人员至关重要。
It’s the only way.
这是唯一的方法。
The way that I could’ve gotten this to go more viral was to tell a story of Chinese AI integrating with an authoritarian state, becoming ASI and taking over the world, and therefore we need our own American models.
我想让这件事传播得更广的方法是讲述一个中国人工智能与独裁国家融合、成为 ASI 并接管世界的故事，因此我们需要我们自己的美国模式。
3:55:24 But it’s very intentional why I talk about innovation and science in the US, because I think it’s both more realistic as an outcome and it’s a world that I would like to manifest.
但我在美国谈论创新和科学是非常有意的，因为我认为这既是更现实的结果，也是我想展现的世界。
I would say, though, that any open-weight model is a valuable model.
不过，我想说，任何开放权重模型都是有价值的模型。
Yeah.
是的。
3:55:55 And my argument is that we should be in a leading position.
我的观点是我们应该处于领先地位。
But I think it’s worth saying it so simply because there are still voices in the AI ecosystem that say we should consider banning the release of open models due to the safety risks.
但我认为值得这么简单地说，因为人工智能生态系统中仍然有声音说，由于安全风险，我们应该考虑禁止发布开放模型。
And I think it’s worth adding that, effectively, that’s impossible without the US having its own Great Firewall, which is known to not work that well.
我认为值得补充的是，实际上，如果美国没有自己的防火墙，这是不可能的，众所周知，防火墙的效果不太好。
The cost for training these models, whether it’s one to a hundred million dollars, is attainable to a huge amount of people in the world that want to have influence, so these models will be getting trained all over the world.
训练这些模型的成本，无论是一亿美元还是一亿美元，对于世界上很多想要产生影响力的人来说都是可以承受的，所以这些模型将在世界各地接受训练。
We want this information and these tools to flow freely across the world and into the US so that people can use them and learn from them.
我们希望这些信息和这些工具能够在世界各地自由流动并进入美国，以便人们可以使用它们并向它们学习。
3:56:47 Stopping that would be such a restructuring of our internet that it seems impossible.
阻止这种情况将是对我们互联网的一次重组，这似乎是不可能的。
Do you think maybe the big open-weight models from China are actually a good thing for US companies?
您认为来自中国的大型开放式重量模型对美国公司来说实际上是一件好事吗？
3:56:51 You mentioned earlier they are usually one generation behind in terms of what they release open source.
您之前提到过，他们在发布开源内容方面通常落后一代人。
For example, gpt-oss-120b might not be the cutting-edge model, or Gemini 3 might not be, because they want to ensure it is safe.
例如，gpt-oss-120b可能不是尖端模型，或者Gemini 3可能不是，因为他们想确保它是安全的。
But when these companies see that DeepSeek-V3.2 is really awesome and is being used with no backlash or security risk, that could encourage them to release better models.
但是，当这些公司看到 DeepSeek-V3.2 确实很棒并且使用时没有出现强烈反对或安全风险时，可能会鼓励他们发布更好的模型。
Maybe that is a very positive thing.
也许这是一件非常积极的事情。
A hundred percent.
百分之一百。
3:57:30 These Chinese companies have set things into motion that I think would potentially not have happened if they were not all releasing models.
这些中国公司已经启动了一些事情，我认为如果他们不全部发布模型，这些事情可能不会发生。
I’m almost sure that those discussions have been had by leadership.
我几乎可以肯定，这些讨论是由领导层进行的。
Is there a possible future where the dominant models, AI models in the world are all open source?
未来是否有可能世界上的主导模型、人工智能模型都是开源的？
Depends on the trajectory of progress that you predict.
取决于您预测的进展轨迹。
3:57:50 If you think saturation in progress is coming within a few years, essentially within the time where financial support is still very good, then open models will be so optimized and so much cheaper to run that they’ll win out.
如果你认为几年内就会出现饱和，基本上是在财政支持仍然非常好的时间内，那么开放模型将得到如此优化，运行成本如此之低，以至于它们将获胜。
Essentially, this goes back to open source ideas where so many more people will be putting money into optimizing the serving of these open-weight common architectures that they will become standards.
从本质上讲，这可以追溯到开源理念，更多的人将投入资金来优化这些开放权重通用架构的服务，从而使它们成为标准。
Then you could have chips dedicated to them and it’ll be way cheaper than the offerings from these closed companies that are custom.
然后你可以拥有专用于它们的芯片，它会比这些定制的封闭公司的产品便宜得多。
We should say that the AI2027 report predicts—one of the things it does from a narrative perspective is that there will be a lot of centralization.
我们应该说，AI2027 报告预测——从叙述的角度来看，它所做的一件事就是将会出现大量的中心化。
3:58:25 As the AI system gets smarter and smarter, the national security concerns will come to be, and you’ll centralize the labs, and you’ll become super secretive, and there’ll be this whole race.
随着人工智能系统变得越来越聪明，国家安全问题将会出现，你将集中实验室，你将变得超级秘密，并且将会有整个竞赛。
…from a military perspective of how you…
......从军事角度看你如何......
3:58:45 between China and the United States.
中国和美国之间。
And so all of these fun conversations we’re having about LLMs—all the generals and soldiers will come into the room and be like, “All right, we’re now in the Manhattan Project stage of this whole thing.”
因此，我们关于法学硕士的所有这些有趣的对话——所有将军和士兵都会走进房间，说：“好吧，我们现在正处于整个事情的曼哈顿计划阶段。”
I think 2025, ’26, ’27—I don’t think something like that is even remotely possible.
我认为 2025 年、26 年、27 年——我认为这样的事情根本不可能发生。
3:59:02 I mean, you can make the same argument for computers, right?
我的意思是，你可以对计算机做出同样的论点，对吧？
You can say, “Okay, computers are capable and we don’t want the general public to get them.” Or chips—even AI chips—but you see how Huawei makes chips now.
你可以说：“好吧，计算机是有能力的，但我们不希望普通公众拥有它们。”或者芯片——甚至是人工智能芯片——但你会看到华为现在是如何制造芯片的。
It took a few years, but…
这花了几年时间，但是……
and I don’t think there is a way you can contain knowledge like that.
我认为没有办法可以容纳这样的知识。
I think in this day and age, it is impossible, like the internet.
我认为在当今时代，这是不可能的，就像互联网一样。
I don’t think this is a possibility.
我认为这是不可能的。
On the Manhattan Project thing, one of my funny things looking at them is I think that a Manhattan Project-like thing for open models would actually be pretty reasonable, because it wouldn’t cost that much.
关于曼哈顿计划，我看到的一件有趣的事情是，我认为类似曼哈顿计划的开放模型实际上是相当合理的，因为它不会花费那么多。
3:59:37 But I think that that will come.
但我认为那将会到来。
It seems like culturally, the companies are changing.
从文化上看，公司似乎正在发生变化。
But I agree with Sebastian on all of the stuff that he just said.
但我同意塞巴斯蒂安刚才所说的所有内容。
It’s just like, I don’t see it happening nor being helpful.
就像，我不认为它会发生，也没有帮助。
Yeah.
是的。
3:59:58 I mean, the motivating force behind the Manhattan Project was that there was civilizational risk.
我的意思是，曼哈顿计划背后的推动力是存在文明风险。
It’s harder to motivate that for open-source models.
对于开源模型来说，激励这一点更加困难。
There’s not civilizational risk.
不存在文明风险。
On the hardware side, we mentioned NVIDIA a bunch of times.
在硬件方面，我们多次提到 NVIDIA。
4:00:10 Do you think Jensen and NVIDIA are going to keep winning?
您认为 Jensen 和 NVIDIA 会继续获胜吗？
I think they have the downside that they have to iterate a lot and manufacture a lot.
我认为他们的缺点是必须进行大量迭代和大量制造。
4:00:18 And what they’re doing—they do innovate, but I think there’s always the chance that there is someone who does something fundamentally different, who gets very lucky and then does something.
他们正在做的事情——他们确实在创新，但我认为总有可能有人做了一些根本不同的事情，他们非常幸运，然后做了一些事情。
But the problem is, I think, adoption.
但我认为问题在于采用。
You know, the moat of NVIDIA is probably not just the GPU; it’s more like the CUDA ecosystem, and that has evolved over two decades.
要知道，NVIDIA的护城河恐怕不仅仅是GPU；还有GPU。它更像是 CUDA 生态系统，并且已经发展了二十多年。
I mean, even back when I was a grad student, I was in a lab doing biophysical simulations, molecular dynamics, and we had a Tesla GPU back then just for the computations.
我的意思是，甚至当我还是一名研究生时，我就在实验室里进行生物物理模拟、分子动力学，当时我们有一个 Tesla GPU 只是用于计算。
It was fifteen years ago now.
现在已经是十五年前的事了。
4:01:01 They built this up for a long time and that’s the moat, I think.
他们花了很长时间建立了这个，我认为这就是护城河。
It’s not the chip itself.
这不是芯片本身。
Although they have the money now to iterate, build, and scale, it’s really on the compatibility.
尽管他们现在有钱来迭代、构建和扩展，但这实际上取决于兼容性。
If you’re at that scale as a company, why would you go with something risky where it’s only— …
如果你是一家如此规模的公司，为什么你会选择一些有风险的东西，因为它只是——……
a few chips that they can make per year?
他们每年可以生产多少芯片？
You go with the big one.
你就和大的一起去吧。
But then I do think with LLMs now, it will be easier to design something like CUDA.
但我确实认为现在有了法学硕士，设计像 CUDA 这样的东西会更容易。
It took 15 years because it was hard, but now that we have LLMs, we can maybe replicate CUDA.
花了 15 年时间，因为这很困难，但现在我们有了法学硕士，我们也许可以复制 CUDA。
And I wonder if there will be a separation of the training and the inference- …
我想知道训练和推理是否会分开——……
4:01:35 compute, as we stabilize a bit more and more compute is needed for inference.
计算，随着我们的稳定，推理需要越来越多的计算。
That’s supposed to be the point of the Groq acquisition.
这应该就是收购 Groq 的目的。
4:01:47 And that’s why part of what Vera Rubin is—
这就是为什么 Vera Rubin 的一部分是——
4:01:52 …
……
where they have a new chip with no high-bandwidth memory, or very little, which is one of the most expensive pieces.
他们有一个新芯片，没有高带宽内存，或者很少，这是最昂贵的芯片之一。
It’s designed for pre-fill, which is the part of inference where you essentially do a lot of matrix multiplications, and then you only need the memory when you’re doing this autoregressive generation and you have the KV cache swaps.
它是为预填充而设计的，预填充是推理的一部分，本质上是进行大量矩阵乘法，然后只有在进行自回归生成并且具有 KV 缓存交换时才需要内存。
So they have this new GPU that’s designed for that specific use case, and then the cost of ownership per flop is actually way lower.
因此，他们拥有专为特定用例设计的新 GPU，然后每次翻牌的拥有成本实际上要低得多。
But I think that NVIDIA’s fate lies in the diffusion of AI still.
但我认为NVIDIA的命运仍然在于AI的传播。
Their biggest clients are still these hyperscale companies, whether it’s Google—which obviously can make TPUs—Amazon making Trainium, or Microsoft trying to do its own things.
他们最大的客户仍然是这些超大规模公司，无论是谷歌（显然可以制造 TPU）、亚马逊制造 Trainium，还是尝试做自己的事情的微软。
4:02:36 As long as the pace of AI progress is high, NVIDIA’s platform is the most flexible and people will want that.
只要人工智能的进展速度很快，NVIDIA 的平台就是最灵活的，人们就会希望如此。
But if there’s stagnation, then with creating bespoke chips, there’s more time to do it.
但如果出现停滞，那么通过创建定制芯片，就有更多的时间来做到这一点。
It’s interesting that NVIDIA is, is quite active in trying to develop all kinds of different products.
有趣的是，NVIDIA 非常积极地尝试开发各种不同的产品。
They try to create areas of commercial value that will use a lot of GPUs.
他们试图创造需要大量 GPU 的商业价值领域。
But they keep innovating and they’re doing a lot of incredible research, so…
但他们不断创新，并且正在进行大量令人难以置信的研究，所以……
Everyone says the company’s super oriented around Jensen and how operationally plugged in he is.
每个人都说公司非常以詹森为中心，以及他在运营上的投入程度。
4:03:06 It sounds so unlike many other big companies that I’ve heard about.
这听起来与我听说过的许多其他大公司很不一样。
And so long as that’s the culture, I think that you can expect that to keep progress happening.
只要这是一种文化，我认为你就可以期待这种文化能够不断取得进展。
It’s like he’s still in the Steve Jobs era of Apple.
就好像他还处于苹果的史蒂夫·乔布斯时代。
So long as that is how it operates, I’m pretty optimistic for their situation because it is their top-order problem, and I don’t know if making these chips for the whole ecosystem is the top goal of all these other companies.
只要这就是它的运作方式，我对他们的情况非常乐观，因为这是他们的首要问题，而且我不知道为整个生态系统制造这些芯片是否是所有其他公司的首要目标。
They’ll do a good job, but it might not be as good of a job.
他们会做得很好，但可能不会那么好。
Since you mentioned Jensen, I’ve been reading a lot about history and about singular figures in history.
自从你提到詹森以来，我读了很多有关历史和历史上奇异人物的书。
4:03:43 What do you guys think about the great man view of history?
对于伟人的历史观，你们怎么看？
How important are individuals for steering the direction of history in the tech sector?
个人对于引领科技行业的历史方向有多重要？
So, you know, what’s NVIDIA without Jensen?
那么，您知道，如果没有 Jensen，NVIDIA 会是什么样子呢？
You mentioned Steve Jobs.
你提到了史蒂夫·乔布斯。
What’s Apple without Steve Jobs?
没有史蒂夫·乔布斯，苹果会是什么样子？
What’s xAI without Elon or DeepMind without Demis?
没有 Elon 的 xAI 或没有 Demis 的 DeepMind 是什么？
People make things earlier and faster, whereas scientifically, many great scientists credit being in the right place at the right time.
人们更早、更快地制造事物，而从科学角度来看，许多伟大的科学家都相信在正确的时间出现在正确的地点。
4:04:11 Eventually someone else will still have the idea.
最终其他人仍然会有这个想法。
So I think that in that way, Jensen is helping manifest this GPU revolution much faster and much more focused than it would be without having a person like him there.
所以我认为，通过这种方式，Jensen 正在帮助实现这场 GPU 革命，这比没有像他这样的人的情况下更快、更集中。
This is making the whole AI build-out faster.
这使得整个人工智能的构建速度更快。
But I do still think that eventually something like ChatGPT would have happened and a build-out like this would have happened, but it probably would not have been as fast.
但我仍然认为最终会发生像 ChatGPT 这样的事情，并且会发生像这样的扩展，但它可能不会那么快。
I think that’s the sort of flavor that is applied.
我认为这就是所应用的那种味道。
These individual people are placing bets on something.
这些个人正在对某件事下注。
4:04:55 Some get lucky, some don’t.
有些人很幸运，有些人则不然。
But if you don’t have these people at the helm, it would be more diffused.
但如果没有这些人来掌舵，事情就会更加分散。
It’s almost like investing in an ETF versus individual stocks.
这几乎就像投资 ETF 与投资个股一样。
Individual stocks might go up or down more heavily than an ETF, which is more balanced.
个股的上涨或下跌幅度可能比 ETF 更大，因为 ETF 更加平衡。
We’ll eventually get there, but I just think the focus is the thing.
我们最终会到达那里，但我认为重点是重点。
Passion and focus.
激情和专注。
Isn’t there a real case to be made that without Jensen, there’s not a reinvigoration of the deep learning revolution?
难道没有真实的案例表明，没有詹森，深度学习革命就不会重振旗鼓吗？
It could’ve been 20 years later, is the thing I would say.
我想说的是，这可能是20年后的事了。
Yeah, 20 is…
是的，20是……
Or another deep learning winter could have come…
或者另一个深度学习冬天可能会到来……
4:05:30 …If GPUs weren’t around.
……如果 GPU 不存在的话。
That could change history completely because you could think of all the other technologies that could’ve come in the meantime, and the focus of human civilization would get…
这可能会彻底改变历史，因为你可以想到同时出现的所有其他技术，而人类文明的焦点将变得……
4:05:35 Silicon Valley would be captured by different hype.
硅谷将被不同的炒作所占领。
But I do think there’s certainly an aspect where the GPU trajectory was all planned.
但我确实认为 GPU 轨迹肯定有一个方面是经过规划的。
4:05:48 But on the other end, it’s also a lot of lucky coincidences or good intuition.
但另一方面，这也有很多幸运的巧合或良好的直觉。
Like the investment into, let’s say, biophysical simulations.
就像对生物物理模拟的投资一样。
I mean, I think it started with video games and then it just happened to be good at linear algebra because video games require a lot of linear algebra.
我的意思是，我认为它是从视频游戏开始的，然后它恰好擅长线性代数，因为视频游戏需要大量的线性代数。
And then you have the biophysical simulations.
然后进行生物物理模拟。
But still, I don’t think the master plan was AI.
但我仍然不认为总体规划是人工智能。
I think it just happened to be Alex Krizhevsky.
我认为这恰好是亚历克斯·克里热夫斯基（Alex Krizhevsky）。
So someone took these GPUs and said, “Hey, let’s try to train a neural network on that.” It happened to work really well and…
于是有人拿着这些 GPU 说：“嘿，让我们尝试用它来训练神经网络。”它碰巧工作得非常好并且......
…I think it only happened because you could purchase those GPUs.
……我认为这只是因为你可以购买这些 GPU。
Gaming would’ve created a demand for faster processors if…
游戏将会产生对更快处理器的需求，如果......
4:06:30 …NVIDIA had gone out of business in the early days.
...NVIDIA 早期就已经倒闭了。
That’s what I would think.
我就是这么想的。
I think GPUs would still exist…
我认为 GPU 仍然存在......
…At the time of AlexNet and at the time of the Transformer.
……在 AlexNet 和 Transformer 时代。
It was just hard to know if it would be one company as successful or multiple smaller companies with worse chips.
只是很难知道是一家公司如此成功，还是多家芯片较差的小公司。
But I don’t think that’s a 100-year delay.
但我不认为这会延迟 100 年。
It might be a decade delay.
可能会延迟十年。
Well, it could be a one, two, three, four, five-decade delay.
嗯，这可能会延迟一、二、三、四、五年。
4:07:01 I mean, I just can’t see Intel or AMD doing what NVIDIA did.
我的意思是，我不认为英特尔或 AMD 会做 NVIDIA 所做的事情。
I don’t think it would be a company that exists.
我认为这不会是一家存在的公司。
A new company.
一家新公司。
I think it would be a different company that would rise.
我认为崛起的将是一家不同的公司。
Like Silicon Graphics or something.
比如硅谷图形公司什么的。
So yeah, some company that has died would have done it.
所以是的，一些已经倒闭的公司会这么做。
But looking at it, it seems like these singular figures, these leaders, have a huge impact on the trajectory of the world.
但看看这些独特的人物、这些领导人，似乎对世界的轨迹产生了巨大的影响。
4:07:19 Obviously, incredible teams are behind them.
显然，他们的背后有令人难以置信的团队。
But, you know, having that kind of very singular, almost dogmatic focus- …is necessary to make progress.
但是，你知道，拥有那种非常单一、近乎教条的焦点……是取得进步的必要条件。
Yeah, I mean, even with GPT, it wouldn’t exist if there wasn’t a person, Ilya, who pushed for this scaling, right?
是的，我的意思是，即使有了 GPT，如果没有一个人 Ilya 推动这种扩展，它也不会存在，对吗？
Yeah, Dario was also deeply involved in that.
是的，达里奥也深度参与其中。
4:07:47 If you read some of the histories from OpenAI, it almost seems wild thinking about how early these people were like, “We need to hook up 10,000 GPUs and take all of OpenAI’s compute and train one model.” There were a lot of people there that didn’t want to do that.
如果你读过 OpenAI 的一些历史，你会发现这些人是多么早期的想法，“我们需要连接 10,000 个 GPU，利用 OpenAI 的所有计算并训练一个模型”，这似乎是一种疯狂的想法。那里有很多人不想这样做。
Which is an insane thing to believe—to believe scaling before scaling has any indication that it’s going to materialize.
这是一件疯狂的事情——在扩展之前就相信扩展有任何迹象表明它将会实现。
4:08:02 Again, singular figures.
再次，单一数字。
Speaking of which, 100 years from now, this is presumably post-singularity, whatever the singularity is.
说到这里，100 年后，无论奇点是什么，这大概都是后奇点。
When historians look back at our time now, what technological breakthroughs would they really emphasize as the breakthroughs that led to the singularity?
当历史学家现在回顾我们的时代时，他们真正会强调哪些技术突破是导致奇点的突破？
So far we have Turing to today, which is 80 years.
到目前为止我们有图灵到今天，已经80年了。
I think it would still be computing, like the umbrella term “computing.” I don’t necessarily think that even 100 or 200 years from now it would be AI.
我认为它仍然是计算，就像总括术语“计算”一样。我并不认为 100 年或 200 年后一定会是人工智能。
4:08:36 It could still well be computers, you know?
它仍然很可能是计算机，你知道吗？
We are now taking better advantage of computers, but it’s the fact of computing.
我们现在正在更好地利用计算机，但这就是计算的事实。
It’s basically a Moore’s Law kind of discussion.
这基本上是一种摩尔定律的讨论。
4:08:53 Even the details of CUDA and GPUs won’t even be remembered, and there won’t be all this software turmoil.
甚至 CUDA 和 GPU 的细节都不会被记住，也不会有所有这些软件混乱。
It’ll be just, obviously, compute.
显然，这只是计算。
I generally agree, but is it the connectivity of the internet and compute able to be merged?
我基本上同意，但是互联网和计算的连接能够合并吗？
4:09:07 Or is it both of them?
或者两者都是？
I think the internet will probably be related to communication—it could be a phone, internet, or a satellite.
我认为互联网可能与通信有关——它可以是电话、互联网或卫星。
4:09:17 And compute is more like the scaling aspect of it.
计算更像是它的缩放方面。
It’s possible that the internet is completely forgotten.
互联网有可能被完全遗忘。
4:09:29 That the internet is wrapped into the phone networks, like communication networks.
互联网被包裹在电话网络中，就像通信网络一样。
This is just another manifestation of that, and the real breakthrough comes from just the increased compute—Moore’s Law, broadly defined.
这只是另一种表现，真正的突破来自于计算量的增加——广义上的摩尔定律。
Well, I think the connection of people is very fundamental to it.
嗯，我认为人与人之间的联系是非常重要的。
4:09:46 You want to find the best person in the world for something, they are somewhere in the world.
你想找到世界上最适合某件事的人，他们就在世界的某个地方。
Being able to have that flow of information—AIs will also rely on this.
能够拥有这种信息流——人工智能也将依赖于此。
I’ve been fixating on when I said the dream was dead about the one central model; the thing that is evolving is that people have many agents for different tasks.
我一直在关注我所说的关于单一中心模型的梦想已经破灭的说法；正在发展的事情是，人们有许多代理来执行不同的任务。
People already started doing this with different Clouds for different tasks.
人们已经开始使用不同的云来执行不同的任务。
It’s described as many AGIs in the data center where each one manages and they talk to each other.
它被描述为数据中心中有许多 AGI，每个 AGI 都进行管理并相互通信。
That is so reliant on networking and the free flow of information on top of compute.
这非常依赖网络和计算之上的信息自由流动。
But networking, especially with GPUs, is such a part of the scaling of compute.
但网络，尤其是 GPU，是计算扩展的一部分。
The GPUs and the data centers need to talk to each other.
GPU 和数据中心需要相互通信。
Do you think there’s something very specific and singular to the fact that it’s neural networks that’s seen as a breakthrough?
您认为神经网络被视为一项突破，这一事实有什么非常具体和独特的地方吗？
4:10:36 Like a genius move where you’re basically replicating, in a very crude way, the structure of the human brain, the human mind?
就像一个天才的举动，你基本上以一种非常粗糙的方式复制人脑的结构，人类的思想？
I think without the human mind, we probably wouldn’t have neural networks because it was an inspiration for them.
我认为如果没有人类思维，我们可能就不会有神经网络，因为它是它们的灵感来源。
4:10:54 But on the other end, I think it’s just so different.
但另一方面，我认为这是非常不同的。
I mean, it’s digital versus biological, so I think it will probably be more grouped as an algorithm.
我的意思是，它是数字与生物的，所以我认为它可能会更多地被归类为一种算法。
That’s massively parallelizable— —on this particular kind of compute?
在这种特殊类型的计算上，这是大规模并行化的吗？
It could have well been genetic computing, like genetic algorithms, just parallelized.
它很可能是遗传计算，就像遗传算法一样，只是并行化。
4:11:15 It just happens that this is more efficient and works better.
碰巧这样效率更高，效果更好。
And it very well could be that the neural networks, the way we architect them now, are just a small component of the system that leads to the singularity.
很可能神经网络，我们现在构建它们的方式，只是导致奇点的系统的一个小组成部分。
I think if you think of it over 100 years, society can be changed more with more compute and intelligence because of autonomy.
我认为如果你考虑 100 年后，由于自治，社会可以通过更多的计算和智能而发生更大的改变。
4:11:33 But looking at this, what are the things from the Industrial Revolution that we remember?
但看看这个，我们还记得哪些工业革命的事情呢？
We remember the engine—it is probably the equivalent of the computer in this.
我们记得引擎——它可能相当于计算机。
But there’s a lot of other physical transformations that people are aware of, like the cotton gin and all these machines that are still known—air conditioning, refrigerators— Some of these things from AI will still be known; the word “transformer” could still very well be known.
但人们也意识到了许多其他的物理变化，比如轧棉机和所有这些仍然已知的机器——空调、冰箱——其中一些来自人工智能的东西仍然是已知的； “变形金刚”这个词仍然广为人知。
I would guess that deep learning is definitely still known, but the transformer might be evolved away from in 100 years with AI researchers everywhere.
我猜想深度学习肯定仍然是众所周知的，但随着人工智能研究人员遍布各地，变压器可能会在 100 年后逐渐消失。
But I think deep learning is likely to be a term that is remembered.
但我认为深度学习很可能是一个被人们记住的术语。
And I wonder what the air conditioning and the refrigeration of the future is that AI brings.
我想知道人工智能会带来什么未来的空调和制冷。
4:12:28 If we travel forward 100 years from now, what do you think is different?
如果我们时光倒流100年，你认为会有什么不同？
How does the world look?
世界看起来如何？
First of all, do you think there’s humans?
首先，你认为世界上有人类吗？
Do you think there’s robots everywhere walking around?
你认为到处都是机器人吗？
I do think there will be specialized robots for certain tasks.
我确实认为会有专门的机器人来执行某些任务。
Humanoid form?
人形形态？
Maybe half-humanoid.
也许是半人形。
4:12:50 We’ll see.
我们会看到的。
I think for certain things, yes, there will be humanoid robots because it’s just amenable to the environment.
我认为对于某些事情，是的，将会出现人形机器人，因为它适合环境。
But for certain tasks, it might not make sense.
但对于某些任务来说，这可能没有意义。
What’s harder to imagine is how we interact with devices and what humans do with them.
更难以想象的是我们如何与设备交互以及人类如何使用它们。
I’m pretty sure it will not be the cellphone or the laptop.
我很确定它不会是手机或笔记本电脑。
Will it be implants?
会是植入物吗？
I mean, it has to be brain-computer interfaces, right?
我的意思是，它必须是脑机接口，对吧？
4:13:16 I mean, 100 years from now, it has to—given the progress we’re seeing now— —there has to be, unless there’s legitimately a complete alteration of how we interact with reality.
我的意思是，从现在起 100 年后，鉴于我们现在所看到的进步，它必须如此，除非我们与现实互动的方式发生合法的彻底改变。
On the other hand, if you think of cars, cars are older than 100 years, right?
另一方面，如果你想到汽车，汽车的历史已经超过 100 年了，对吧？
4:13:33 And it’s still the same interface.
而且还是同一个界面。
We haven’t replaced cars with something else; we just made them better.
我们没有用其他东西取代汽车；我们只是让他们变得更好。
But it’s still a steering wheel, it’s still wheels.
但它仍然是方向盘，仍然是轮子。
I think we’ll still carry around a physical brick of compute— —because people want some ability to have a private interface.
我认为我们仍然会随身携带物理计算模块——因为人们想要拥有私有接口的能力。
4:13:45 You might not engage with it as much as a phone, but having something where you could have private information that is yours as an interface between you and the rest of the internet is something I think will still exist.
你可能不会像手机那样频繁地使用它，但我认为拥有一些可以让你拥有私人信息的东西作为你和互联网其他部分之间的接口仍然存在。
It might not look like an iPhone, and it might be used a lot less, but I still expect people to carry things around.
它可能看起来不像iPhone，而且它的使用率可能会少很多，但我仍然希望人们随身携带东西。
Why do you think the smartphone is the embodiment of privacy?
为什么您认为智能手机是隐私的体现？
4:14:08 There’s a camera on it.
上面有一个摄像头。
There’s a-
有一个-
Private for you, like encrypted messages, encrypted photos; you know what your life is.
对您来说是私密的，例如加密消息、加密照片；你知道你的生活是什么。
4:14:15 I guess this is a question of how optimistic you are on brain-machine interfaces.
我想这是一个你对脑机接口有多乐观的问题。
Is all that just going to be stored in the cloud, like your whole calendar?
所有这些都会像您的整个日历一样存储在云端吗？
It’s hard to think about processing all the information that we can process visually through brain-machine interfaces presenting something like a calendar to you.
很难想象如何处理我们可以通过脑机界面直观地处理的所有信息，向您呈现日历之类的东西。
It’s hard to just think about knowing your email inbox without looking.
光想不看就知道你的电子邮件收件箱是很难的。
Like you signal to a computer and then you just know your email inbox.
就像您向计算机发出信号一样，然后您就知道您的电子邮件收件箱。
Is that something that the human brain can handle being piped into it non-visually?
人类大脑可以处理以非视觉方式输入的东西吗？
I don’t know exactly how those transformations happen.
我不知道这些转变是如何发生的。
‘Cause humans aren’t changing in 100 years.
因为人类在 100 年内都不会改变。
4:15:05 I think agency and community are things that people actually want.
我认为代理和社区是人们真正想要的东西。
A local community, yeah.
当地社区，是的。
So, like, people you are close to, being able to do things with them and being able to ascribe meaning to your life.
所以，就像你亲近的人一样，能够与他们一起做事，并能够赋予你的生活意义。
4:15:10 I don’t think that human biology is changing away from those on a timescale that we can discuss.
我不认为人类生物学正在改变我们可以讨论的时间尺度。
UBI does not solve agency.
UBI 并不能解决代理问题。
I do expect mass wealth, and I hope that it has spread so that the average life does look very different in 100 years.
我确实期望大量的财富，并且我希望财富能够扩散到一百年后，人们的平均生活确实会发生很大的变化。
But that’s still a lot to happen in 100 years.
但 100 年后仍然会发生很多事情。
If you think about countries that are early in their development process, to build all the infrastructure and have policy that shares one nation’s wealth with another is…
如果您考虑处于发展进程早期的国家，那么建设所有基础设施并制定与另一个国家分享一国财富的政策是……
I think it’s an optimistic view to see all that happening in 100 years- …while they are still independent entities and not just absorbed into some international order by force.
我认为看到这一切在 100 年内发生是一种乐观的看法——……尽管它们仍然是独立的实体，而不仅仅是通过武力被纳入某些国际秩序。
But there could be just better, more elaborate, more effective- …social support systems that help alleviate some levels of basic suffering from the world.
但可能存在更好、更复杂、更有效的……社会支持系统，有助于减轻世界上某些程度的基本痛苦。
4:16:13 With the transformation of society where a lot of jobs are lost in the short term, I think we have to really remember that each individual job that’s lost is a human being who’s suffering.
随着社会的转型，大量工作在短期内消失，我认为我们必须真正记住，每一个失去的工作都是一个正在受苦的人。
When jobs are lost at scale, it is a real tragedy.
当大规模失业时，这才是真正的悲剧。
You can make all kinds of arguments about economics or say it’s all going to be okay and good for the GDP because new jobs will be created, but fundamentally at the individual level for that human being, that’s real suffering.
你可以就经济学提出各种各样的论点，或者说这一切都会好起来，对 GDP 有利，因为会创造新的就业机会，但从根本上来说，对于人类的个人层面来说，这才是真正的痛苦。
That’s a real personal tragedy.
这是一个真正的个人悲剧。
4:16:58 And we have to not forget that as the technologies are being developed.
我们必须不要忘记，随着技术的发展。
Also, my hope for all the AI slop we’re seeing is that there will be a greater and greater premium for the fundamental aspects of the human experience that are in-person.
此外，我对我们所看到的所有人工智能的希望是，人类亲身体验的基本方面将会有越来越大的溢价。
The things that we all enjoy, like seeing each other and talking together in-person.
我们都喜欢的事情，比如见面和面对面交谈。
The next few years are definitely going to see an increased value on physical goods and events- …and even more pressure from slop.
未来几年，实物商品和活动的价值肯定会增加——……甚至来自废油的压力也会更大。
4:17:22 The slop is only starting.
斜坡才刚刚开始。
The next few years will be more and more diverse-
未来几年将会越来越多元化——
Do you think we’ll all be drow-
你认为我们都会成为卓尔吗-
…versions of slop.
…斜坡的版本。
They would be drowning in slop.
他们会被泥水淹死的。
4:17:38 Is that what-
这就是——
So I’m hoping that society drowns in slop enough to snap out of it and be like, “We can’t.
所以我希望社会淹没在泥沼中，能够振作起来，然后说：“我们不能。
4:17:40 It just doesn’t matter.
这并不重要。
We all can’t deal with it.” And then, the physical has such a higher premium on it.
我们都无法应对。”然后，实物的溢价就更高了。
Even like classic examples, I honestly think this is true, and I think we will get tired of it.
即使像经典的例子一样，我真诚地认为这是真的，而且我认为我们会厌倦它。
4:17:53 We are already kind of tired of it.
我们已经有点厌倦了。
Same with art.
艺术也一样。
I don’t think art will go away.
我不认为艺术会消失。
I mean, you have physical paintings.
我的意思是，你有实体画。
There’s more value, not just monetary value, but just more appreciation for the actual painting than a photocopy of that painting.
与那幅画的复印件相比，它具有更多的价值，不仅仅是金钱价值，而且是对实际绘画的更多欣赏。
It could be a perfect digital reprint, but there is something when you go to a museum and you look at that art and you see that real thing and you just think about, “Okay, a human.” It’s like a craft.
它可能是完美的数字重印本，但当你去博物馆观看那件艺术品时，你会看到真实的东西，你就会想，“好吧，一个人。”这就像一门工艺品。
You have an appreciation for that.
你对此很欣赏。
4:18:25 And I think the same is true for writing, for talking, for any type of experience, where it will be…
我认为对于写作、谈话、任何类型的经历来说都是如此，无论它会在哪里......
I do unfortunately think it will be like a dichotomy, like a fork where some things will be automated.
不幸的是，我确实认为这将像一个二分法，就像一个分叉，其中有些事情将被自动化。
There are not as many paintings as there used to be 200 years ago.
现在的画作已经不像200年前那么多了。
There are more photographs, more photocopies.
还有更多照片、更多复印件。
But at the same time, it won’t go away.
但与此同时，它不会消失。
There will be value in that.
这将是有价值的。
I think that the difference will just be what’s the proportion of that.
我认为区别只是其中的比例。
But personally, I have a hard time reading things where I see it’s obviously AI-generated.
但就我个人而言，我很难阅读那些明显是人工智能生成的东西。
I’m sorry, there might be really good information there, but I have a certain feeling, like, it’s not for me.
抱歉，那里可能有非常好的信息，但我有一种感觉，就像，这不适合我。
I think eventually they’ll fool you, and it’ll be on platforms that give ways of verifying or building trust.
我认为最终他们会愚弄你，而且会在提供验证或建立信任方式的平台上。
4:19:08 So you will trust that Lex is not AI-generated, having been here.
所以你会相信 Lex 不是人工智能生成的，因为它已经存在了。
So then you have trust in this- -channel.
那么你就信任这个渠道了。
But it’s harder for new people- -that don’t have that trust.
但对于没有这种信任的新人来说，这就更难了。
Well, that will get interesting because I think fundamentally it’s a solvable problem by having trust in certain outlets that they won’t do it, but it’s all going to be kind of trust-based.
嗯，这会变得很有趣，因为我认为从根本上来说，这是一个可以解决的问题，只要相信某些媒体不会这样做，但这一切都将是基于信任的。
4:19:25 There will be some systems to authorize, “Okay, this is real.
会有一些系统来授权，“好吧，这是真的。
This is not real.” There will be some telltale signs where you can obviously tell this is AI-generated and this is not.
这不是真的。”会有一些明显的迹象，你可以明显看出这是人工智能生成的，而这不是。
But some will be so good that it’s hard to tell, and then you have to trust.
但有些会好到很难说，然后你就必须相信。
And that will get interesting and a bit problematic.
这会变得很有趣，但也有点问题。
The extreme case of this is to watermark all human content.
这种情况的极端情况是对所有人类内容添加水印。
4:19:54 So all photos that we take on our own- -have some watermark until they- -are edited- -or something like this.
因此，我们自己拍摄的所有照片都有一些水印，直到它们被编辑或类似的东西。
And software can manage communications with the device manufacturer- -to maintain human editing, which is the opposite of the discussion to try to watermark AI images.
软件可以管理与设备制造商的通信，以维持人工编辑，这与尝试给人工智能图像加水印的讨论相反。
And then you can make a Google image that has a watermark and use a different Google tool to remove the watermark.
然后你可以制作一个带有水印的谷歌图像，并使用不同的谷歌工具来删除水印。
Yeah.
是的。
4:20:20 It’s going to be an arms race, basically.
基本上，这将是一场军备竞赛。
And we’ve been mostly focusing on the positive aspects of AI.
我们主要关注人工智能的积极方面。
4:20:23 I mean, all the capabilities that we’ve been talking about can be used to destabilize human civilization with even just relatively dumb AI applied at scale, and then further and further, superintelligent AI systems.
我的意思是，我们一直在谈论的所有能力都可以用来破坏人类文明的稳定，即使只是大规模应用相对愚蠢的人工智能，然后越来越多的超级智能人工智能系统。
Of course, there’s the sort of doomer take that’s important to consider a little bit as we develop these technologies.
当然，在我们开发这些技术时，需要考虑一些厄运的因素。
What gives you hope about the future of human civilization?
是什么让您对人类文明的未来充满希望？
Everything we’ve been talking about—are we going to be okay?
我们一直在谈论的一切——我们会好吗？
I think we will.
我想我们会的。
4:20:59 I’m definitely a worrier both about AI and non-AI things, but humans do tend to find a way.
我绝对是人工智能和非人工智能事物的担忧者，但人类确实倾向于找到一种方法。
I think that’s what humans are built for—to have community and find a way to figure out problems.
我认为这就是人类生来的目的——建立社区并找到解决问题的方法。
And that’s what has gotten us to this point.
这就是我们走到这一步的原因。
I think the AI opportunity and related technologies is really big.
我认为人工智能和相关技术的机会确实很大。
I think that there are big social and political problems to help everybody understand that.
我认为有一些重大的社会和政治问题可以帮助每个人理解这一点。
I think that’s what we’re staring at a lot of right now; the world is a scary place, and AI is a very uncertain thing.
我认为这就是我们现在所关注的问题；世界是一个可怕的地方，而人工智能是一个非常不确定的东西。
And it takes a lot of work that is not necessarily building things.
这需要大量的工作，但不一定是建造东西。
It’s like telling people and understanding people, things that the people building AI are historically not motivated or wanting to do.
这就像告诉人们和理解人们，构建人工智能的人们历来没有动力或不想做的事情。
4:21:50 But it is something that is probably doable.
但这是可能可行的。
It just will take longer than people want.
只是这需要的时间比人们想要的要长。
And we have to go through that long period of hard, distraught AI discussions if we want to have the lasting benefits.
如果我们想获得持久的好处，我们就必须经历漫长而艰难、令人心烦意乱的人工智能讨论。
Yeah.
是的。
4:22:04 Through that process, I’m especially excited that we get a chance to better understand ourselves at the individual level as humans and at the civilization level, and answer some of the big mysteries, like what is this whole consciousness thing going on here?
通过这个过程，我特别兴奋的是，我们有机会在人类的个体层面和文明层面更好地了解自己，并回答一些重大的谜团，比如整个意识的事情在这里发生了什么？
It seems to be truly special.
看起来确实很特别。
Like, there’s a real miracle in our mind.
就像，我们心中有一个真正的奇迹。
And AI puts a mirror to ourselves and we get to answer some of the big questions about what is this whole thing going on here.
人工智能给我们自己放了一面镜子，我们可以回答一些关于这整件事到底发生了什么的重大问题。
Well, one thing about that is also what I do think makes us very different from AI and why I don’t worry about AI taking over is, like you said, consciousness.
嗯，我确实认为这一点让我们与人工智能有很大不同，而我不担心人工智能接管的原因是，就像你说的，意识。
4:22:35 We humans, we decide what we want to do.
我们人类，我们决定我们想要做什么。
AI in its current implementation, I can’t see it changing.
就目前的人工智能实施而言，我看不出它会发生变化。
You have to tell it what to do.
你必须告诉它做什么。
And so you still have the agency.
所以你仍然拥有代理权。
It doesn’t take the agency from you because it becomes a tool.
它不会夺走你的代理权，因为它变成了一种工具。
You tell it what to do.
你告诉它做什么。
It will be more automatic than other previous tools.
它将比以前的其他工具更加自动化。
It’s certainly more powerful than a hammer, it can figure things out, but it’s still you in charge, right?
它肯定比锤子更强大，它可以解决问题，但仍然是你负责，对吗？
So the AI is not in charge, you’re in charge.
所以人工智能不负责，你负责。
You tell the AI what to do and it’s doing it for you.
你告诉人工智能做什么，它就会为你做这件事。
So in the post-singularity, post-apocalyptic war between humans and machines, you’re saying humans are worth fighting for?
那么，在后奇点、后世界末日的人类与机器之间的战争中，你是说人类值得为之而战？
100%.
100%。
4:23:27 I mean, the movie Terminator, they made in- -the ’80s, essentially, and I do think the only thing I can see going wrong is, of course, if things are explicitly programmed to do things that are harmful.
我的意思是，他们在 80 年代制作的电影《终结者》，基本上，我确实认为我能看到的唯一出错的地方当然是，如果事情被明确编程为做有害的事情。
I think actually in a Terminator type of setup, I think humans win.
我认为实际上在终结者类型的设置中，我认为人类获胜。
4:23:43 I think we’re too clever.
我觉得我们太聪明了。
It’s hard to explain how we figure it out, but we do.
很难解释我们是如何弄清楚的，但我们确实做到了。
And we’ll probably be using local LLMs, open source LLMs, to help fight the machines.
我们可能会使用本地法学硕士、开源法学硕士来帮助对抗机器。
I apologize for the ridiculousness.
我为我的荒谬道歉。
Like I said, Nathan, I’ve already been a big fan of yours for a long time.
就像我说的，内森，我长期以来一直是你的忠实粉丝。
And I’ve been a big fan of yours, Sebastian, for a long time, so it’s an honor to finally meet you.
塞巴斯蒂安，我长期以来一直是你的忠实粉丝，所以很荣幸终于见到你。
Thank you for everything you put out into the world.
感谢你为这个世界付出的一切。
Thank you for the excellent books you’re writing.
感谢您所写的优秀书籍。
Thank you for teaching us.
谢谢你教导我们。
And thank you for talking today.
感谢您今天的发言。
This was fun.
这很有趣。
Thank you for inviting us here and having this human connection, which is actually-
感谢您邀请我们来到这里并建立这种人际关系，这实际上是-
-extremely valuable- -human connection.
-极其有价值- -人际关系。
4:24:30 Thanks for listening to this conversation with Sebastian Raschka and Nathan Lambert.
感谢您收听 Sebastian Raschka 和 Nathan Lambert 的对话。
To support this podcast, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback, and so on.
为了支持这个播客，请在描述中查看我们的赞助商，您还可以在其中找到联系我、提出问题、提供反馈等的链接。
And now let me leave you with some words from Albert Einstein: “It is not that I’m so smart, but I stay with the questions much longer.” Thank you for listening, and hope to see you next time.
现在让我向你们说几句话阿尔伯特·爱因斯坦的话：“并不是我有多聪明，而是我在问题上停留的时间更长。”感谢您的收听，希望下次再见。
本集资源与出处
- 视频在 YouTube 收看
- 音频官方 MP3 直链来自 lexfridman.com 官方播客 RSS
- 官网Lex Fridman 官网本集页面
- 文稿官方 Transcript（lexfridman.com）
