OpenAI researcher on why soft skills are the future of work | Karina Nguyen

Lenny Rachitsky: Not only are you working at the cutting edge of AI and LLMs, you're actually building the cutting edge.

Lenny Rachitsky：您不仅在人工智能和法学硕士领域的前沿工作，而且实际上正在构建前沿。

Karina Nguyen: When I first came to Anthropic and I was like, "Oh my God, I really love front-end engineering." And then the reason why I switched to research is because I realized, "Oh my God, Claude is getting better at front-end.

Karina Nguyen：当我第一次来到 Anthropic 时，我想，“天哪，我真的很喜欢前端工程。”然后我转向研究的原因是因为我意识到，“天哪，Claude 在前端越来越好了。

Claude is getting better at coding.

克劳德的编码能力越来越好。

I think Claude can develop new apps."

我认为克劳德可以开发新的应用程序。”

Lenny Rachitsky: What skills do you think will be most valuable going forward for product teams, in particular?

Lenny Rachitsky：您认为哪些技能对于产品团队来说最有价值？

Karina Nguyen: Creative thinking and you kind of want to generate a bunch of ideas and filter through them and not just build the best product experience.

Karina Nguyen：创造性思维，你想要产生一堆想法并过滤它们，而不仅仅是构建最佳的产品体验。

I think it's actually really, really hard to teach the model how to be aesthetic or really good visual design or how to be extremely creative in the way they write.

我认为实际上真的很难教模特如何审美或真正优秀的视觉设计或如何在他们的写作方式中极具创造力。

Lenny Rachitsky: What do you think people most misunderstand about how models are created?

Lenny Rachitsky：您认为人们对于模型的创建方式最误解的是什么？

Karina Nguyen: When you taught the model, some of the self-knowledge of you actually don't have a physical body to operate in the physical world, the model would get extremely confused.

Karina Nguyen：当你教授模型时，你的一些自我认识实际上没有物理身体在物理世界中运行，模型会变得非常混乱。

Lenny Rachitsky: Today my guest is Karina Nguyen.

Lenny Rachitsky：今天我的客人是 Karina Nguyen。

Karina is an AI researcher at OpenAI where she helped build Canvas, tasks, the o1 chain-of-thought model and more.

Karina 是 OpenAI 的人工智能研究员，她帮助构建了 Canvas、任务、o1 思想链模型等。

Prior to OpenAI, she was at Anthropic where she led work on post-training and evaluation for the Claude 3 models, built a document upload feature with 100K context windows and so much more.

在加入 OpenAI 之前，她在 Anthropic 负责 Claude 3 模型的后期训练和评估工作，构建了具有 100K 上下文窗口的文档上传功能等等。

She was also an engineer at New York Times, was a designer at Dropbox and at Square.

她还曾担任《纽约时报》的工程师、Dropbox 和 Square 的设计师。

It's very rare to get a glimpse into how someone working on the bleeding edge of AI and LLMs operates and how they think about where things are heading.

很少有人能够了解从事人工智能和法学硕士前沿工作的人是如何运作的以及他们如何看待事情的发展方向。

Canvas

帆布

Lenny Rachitsky: In our conversation, we talk about how teams that OpenAI operate and build products, what skills she thinks you should be building as AI gets smarter, how models are created, why synthetic data will allow models to keep getting smarter and why she moved from engineering to research after realizing how good LLMs are going to be at coding.

Lenny Rachitsky：在我们的谈话中，我们讨论了 OpenAI 的团队如何运营和构建产品，她认为随着人工智能变得越来越聪明，你应该培养哪些技能，如何创建模型，为什么合成数据将让模型变得越来越聪明，以及为什么她在意识到法学硕士在编码方面有多出色后从工程转向研究。

If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube.

如果您喜欢这个播客，请不要忘记订阅并在您最喜欢的播客应用程序或 YouTube 中关注它。

It's the best way to avoid missing feature episodes and it helps the podcast tremendously.

这是避免错过专题节目的最佳方式，并且对播客有很大帮助。

With that, I bring you Karina Nguyen.

接下来，我为您带来卡琳娜·阮 (Karina Nguyen)。

Lenny Rachitsky: This episode is brought to you by Enterpret.

Lenny Rachitsky：本集由 Enterpret 为您带来。

Enterpret unifies all your customer interactions from Gong calls to Zendesk tickets to Twitter threads to app store reviews, and makes it available for analysis.

Enterpret 统一了所有客户交互，从 Kong 呼叫到 Zendesk 票证到 Twitter 线程再到应用程序商店评论，并可供分析。

It's trusted by leading product orgs like Canva, Notion, Loom, Linear, monday.com, and Strava, to bring the voice of the customer into the product development process, helping you build best-in-class products faster.

它受到 Canva、Notion、Loom、Linear、monday.com 和 Strava 等领先产品组织的信赖，可将客户的声音纳入产品开发流程，帮助您更快地构建一流的产品。

What makes Enterpret special is its ability to build and update customer-specific AI models that provide the most granular and accurate insights into your business, connect customer insights to revenue and operational data in your CRM or data warehouse to map the business impact of each customer need and prioritize confidently, and empower your entire team to easily take action on use cases like win-loss analysis, critical bug detection and identifying drivers of churn with Enterpret's AI system, Wisdom.

Enterpret 的独特之处在于它能够构建和更新特定于客户的 AI 模型，为您的业务提供最精细、最准确的洞察，将客户洞察与 CRM 或数据仓库中的收入和运营数据联系起来，以映射每个客户需求的业务影响并自信地确定优先级，并让您的整个团队能够使用 Enterpret 的 AI 系统 Wisdom 轻松对赢利分析、关键错误检测和识别客户流失驱动因素等用例采取行动。

Lenny Rachitsky: Looking to automate your feedback loops and prioritize your roadmap with confidence, like Notion, Canva and Linear?

Lenny Rachitsky：希望自动化反馈循环并自信地确定路线图的优先级，例如 Notion、Canva 和 Linear？

Visit E-N-T-E-R-P-R-E-T.com/Lenny to connect with the team and to get two free months when you sign up for an annual plan.

请访问 E-N-T-E-R-P-R-E-T.com/Lenny 与团队联系，并在注册年度计划后免费获得两个月的时间。

This is a limited time offer.

这是限时优惠。

That's Enterpret.com/Lenny.

这是 Enterpret.com/Lenny。

This episode is brought to you by Vanta.

本集由 Vanta 为您带来。

And I am very excited to have Christina Cacioppo, CEO and co-founder Vanta, joining me for this very short conversation.

我非常高兴万塔首席执行官兼联合创始人克里斯蒂娜·卡西奥波 (Christina Cacioppo) 与我一起进行这次简短的对话。

Christina Cacioppo: Great to be here.

克里斯蒂娜·卡乔波：很高兴来到这里。

Big fan of the podcast and the newsletter.

播客和时事通讯的忠实粉丝。

Lenny Rachitsky: Vanta is a longtime sponsor of the show, but for some of our newer listeners, what does Vanta do and who is it for?

Lenny Rachitsky：Vanta 是该节目的长期赞助商，但对于我们的一些新听众来说，Vanta 是做什么的？它是为谁服务的？

Christina Cacioppo: Sure.

克里斯蒂娜·卡乔波：当然。

So we started Vanta in 2018.

所以我们在 2018 年启动了 Vanta。

Focused on founders, helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications like SOC 2 or ISO 27001 today, we currently help over 9,000 companies, including some startup household names like Atlassian, Ramp and LangChain start and scale their security programs and ultimately build trust by automating compliance, centralizing GRC, and accelerating security reviews.

我们以创始人为重点，帮助他们开始构建安全计划，并通过 SOC 2 或 ISO 27001 等合规认证为所有艰苦的安全工作赢得认可，目前我们帮助超过 9,000 家公司（包括 Atlassian、Ramp 和 LangChain 等一些家喻户晓的初创公司）启动和扩展他们的安全计划，并最终通过自动化合规性、集中 GRC 和加速安全审查来建立信任。

Lenny Rachitsky: That is awesome.

莱尼·拉奇茨基：太棒了。

I know from experience that these things take a lot of time and a lot of resources and nobody wants to spend time doing this.

我从经验中知道，这些事情需要花费大量时间和资源，没有人愿意花时间来做这件事。

Christina Cacioppo: That is very much our experience, but before the company and to some extent during it.

Christina Cacioppo：这就是我们的经验，但在公司成立之前以及某种程度上在公司成立期间。

But the idea is with automation, with AI, with software, we are helping customers build trust with prospects and customers in an efficient way.

但我们的想法是通过自动化、人工智能和软件，我们正在帮助客户以有效的方式与潜在客户和客户建立信任。

And our joke, we started this compliance company so you don't have to.

我们开玩笑说，我们创办了这家合规公司，这样您就不必这么做了。

Lenny Rachitsky: We appreciate you for doing that.

Lenny Rachitsky：我们感谢您这样做。

And you have a special discount for listeners, they can get $1,000 off Vanta at Vanta.com/Lenny, that's V-A-N-T-A.com/Lenny for $1,000 off Vanta.

而且您还为听众提供特别折扣，他们可以在 Vanta.com/Lenny 上享受 Vanta 优惠 1,000 美元，即 V-A-N-T-A.com/Lenny，Vanta 优惠 1,000 美元。

Thanks for that, Christina.

谢谢你，克里斯蒂娜。

Lenny Rachitsky: Karina, thank you so much for being here.

Lenny Rachitsky：卡琳娜，非常感谢你来到这里。

Welcome to the podcast.

欢迎来到播客。

Karina Nguyen: Thank you so much, Lenny, for inviting me.

卡琳娜·阮：莱尼，非常感谢你邀请我。

Lenny Rachitsky: I'm very excited to have you here because not only are you working at the cutting edge of AI and LLMs, you're actually building the cutting edge of AI and LLMs.

Lenny Rachitsky：我很高兴您能来到这里，因为您不仅在人工智能和法学硕士的前沿工作，而且实际上正在构建人工智能和法学硕士的前沿。

You recently launched this feature, which basically...

您最近推出了此功能，基本上...

the first agent feature of OpenAI.

OpenAI 的第一个代理功能。

I also just did this survey, I don't know if you know about this.

我也刚刚做了这个调查，不知道你是否了解这一点。

I did a survey of my readers and asked them what tools do you use every day in your work and most use?

我对我的读者做了一个调查，问他们在工作中每天使用并且最常用的工具是什么？

And ChatGPT was number one, above Gmail, above Slack, above anything else.

ChatGPT 排名第一，高于 Gmail，高于 Slack，高于其他任何东西。

90% of people said they use ChatGPT regularly.

90% 的人表示他们经常使用 ChatGPT。

Lenny Rachitsky: It's absurd.

莱尼·拉奇茨基：这太荒谬了。

It wasn't around two years ago.

这不是两年前的事了。

Lenny Rachitsky: Also, we're recording this the week that OpenAI announced Stargate, which is this half trillion dollar investment in AI infrastructure.

Lenny Rachitsky：另外，我们在 OpenAI 宣布 Stargate 的那一周记录了这一点，这是对人工智能基础设施的 5 万亿美元投资。

So there's just a lot happening constantly in AI and you have a really unique glimpse into how things are working, where things are going, how work gets done.

因此，人工智能领域不断发生很多事情，你可以以独特的视角了解事物如何运作、事物进展如何以及工作如何完成。

So I have a lot of questions for you.

所以我有很多问题要问你。

I want to talk about how you operate and how you work at OpenAI, where you think things are going, what skills are going to matter more and less in the future, and also just where things are going broadly.

我想谈谈你在 OpenAI 的运作方式和工作方式，你认为事情的发展方向，哪些技能在未来会变得更加重要和不那么重要，以及事情的总体发展方向。

So how does that sound?

那么听起来怎么样？

Karina Nguyen: Sounds great.

卡琳娜·阮：听起来很棒。

Thank you so much.

太感谢了。

Yeah, I was extremely lucky to join early days Anthropic and learned a lot of things there.

是的，我非常幸运能够加入早期的 Anthropic，并在那里学到了很多东西。

And I joined OpenAI around eight months ago.

我大约八个月前加入了 OpenAI。

So, yeah, I'm excited to dive more in into-

所以，是的，我很高兴能更深入地研究——

Lenny Rachitsky: Okay, I'm going to definitely ask you about the differences between those, but I want to start more technical and just dive right in.

Lenny Rachitsky：好的，我肯定会问你它们之间的区别，但我想开始更多的技术性工作，然后直接深入探讨。

I want to talk about model training.

我想谈谈模型训练。

People always hear about models being trained, these big models, how much data takes, how long it takes, how much money toss it takes, how we're running out of data, which I want to talk about.

人们总是听说正在训练模型，这些大模型，需要多少数据，需要多长时间，需要多少钱，我们如何耗尽数据，这些都是我想谈的。

Let me just ask you this question.

让我问你这个问题。

What do you think people most misunderstand about how models are created?

您认为人们对模型创建方式最有误解的是什么？

Karina Nguyen: Model training is more an art than a science.

Karina Nguyen：模特训练与其说是一门科学，不如说是一门艺术。

And in a lot of ways we, as model trainers, think a lot about data quality.

作为模型训练者，我们在很多方面都对数据质量进行了很多思考。

It's one of the most important things in model training is like how do you ensure the highest quality data for certain interaction model behavior that you want to create?

模型训练中最重要的事情之一就是如何确保您想要创建的某些交互模型行为获得最高质量的数据？

But the way you debug models is actually very similar the way you debug software.

但调试模型的方式实际上与调试软件的方式非常相似。

So one of the things that I've learned early days at Anthropic was we've discovered especially this Claude 3 training, when you taught the model some of the soft-knowledge of, "Hey, you actually don't have a physical body to operate in the physical world." But then at the same time you had data that taught the model some of the function calls, which is like, "This is how you set the alarm."

因此，我早期在 Anthropic 学到的一件事是，我们特别发现了 Claude 3 训练，当你向模型传授一些软知识时，“嘿，你实际上没有物理身体可以在物理世界中运行。”但同时你也有数据告诉模型一些函数调用，就像“这就是你设置闹钟的方式”。

Karina Nguyen: And so the model would get extremely confused about whether it can set an alarm, but it doesn't have a body in the physical world.

Karina Nguyen：因此，模型会对它是否可以设置警报感到非常困惑，但它在物理世界中没有实体。

So it's like the model gets confused and sometimes it'll over accuse.

所以这就像模型感到困惑，有时它会过度指责。

So sometimes it says, "Look, I don't know.

所以有时它会说：“听着，我不知道。

Sorry, I cannot help you." And so there is always a balance trade off between how do you make the model to be more helpful for users, but also not being harmful in other scenarios.

抱歉，我帮不了你。”因此，如何使模型对用户更有帮助，同时又不会在其他场景中造成损害，之间始终存在平衡权衡。

And so it's always about how do you make the model more robust and operate across a variety of diverse scenarios.

因此，问题始终在于如何使模型更加稳健并在各种不同的场景中运行。

Lenny Rachitsky: That is so funny.

莱尼·拉奇茨基：这太有趣了。

I never thought about that.

我从来没有想过这个。

Most of the data that it's trained on is kind of assuming it's like a human describing the world and how they operate.

它训练的大部分数据都假设它就像一个人来描述世界及其运作方式。

It assumes there's a body and you could do things, and the model is told you don't have a body.

它假设有一个身体，你可以做一些事情，但模型被告知你没有身体。

Lenny Rachitsky: Okay.

莱尼·拉奇茨基：好的。

I want to talk a little bit about data while we're on this topic.

在我们讨论这个话题时，我想谈谈数据。

I know you have strong opinions here.

我知道你在这里有强烈的意见。

There's this meme that models are going to stop getting smarter because they're running out of data.

有这样一个梗：模型将不再变得更聪明，因为它们的数据即将耗尽。

They're trained in a large part on the internet and there's only one internet and they've already been trained on it, what more can you show them about the world?

他们大部分是在互联网上接受培训的，而只有一个互联网，他们已经接受过培训，你还能向他们展示什么关于这个世界的信息？

And there's this trend of synthetic data, this term synthetic data.

还有合成数据的趋势，这个术语是合成数据。

What is synthetic data?

什么是合成数据？

Why do you think it's important?

为什么你认为这很重要？

Do you think it's going to work?

你认为这会起作用吗？

Karina Nguyen: I think there are two questions here.

Karina Nguyen：我认为这里有两个问题。

We can unpack one at a time.

我们可以一次打开一个包装。

But people say we are hitting the data wall.

但人们说我们正在撞上数据墙。

I think people think more in the terms of pre-trained large models that are trained on the entire internet to predict the next token.

我认为人们更多地考虑预先训练的大型模型，这些模型在整个互联网上进行训练以预测下一个令牌。

But what actually the model is learning during that process is actually how do you compress the compression algorithm here?

但实际上模型在这个过程中学习的是如何压缩这里的压缩算法？

The model learns to compress a lot of knowledge and it learns how to model the world.

该模型学习压缩大量知识，并学习如何对世界进行建模。

So the next prediction of the word, like, "Teach me how to drive," basically.

因此，该词的下一个预测基本上是“教我如何开车”。

And you only have a few words that will match that, a car.

而你只有几个词可以与之匹配：汽车。

So the model actually learns about the world in itself.

所以模型实际上是在了解世界本身。

So it's like it's modeling human behavior, sometimes it's modeling...

所以这就像它在模拟人类行为，有时它是在模拟......

And when you talk to pre-trained models which are very, very large, they're actually extremely diverse and extremely creative because you can talk to almost any Reddit user through a pre-trained model.

当你与非常非常大的预训练模型交谈时，它们实际上非常多样化且极具创意，因为你可以通过预训练模型与几乎任何 Reddit 用户交谈。

Karina Nguyen: But I think what's happening right now with new paradigm of o1 series is that the scaling in post-training itself is not hitting the wall.

Karina Nguyen：但我认为现在 o1 系列的新范式所发生的情况是，训练后的扩展本身并没有遇到障碍。

And that's because basically we went from raw data sets from pre-trained models to infinite amount of tasks that you can teach the model in the post-training world via reinforcement learning.

这是因为基本上我们从预训练模型的原始数据集转变为无限量的任务，您可以通过强化学习在训练后世界中教授模型。

So any task, for example, how to search the web, how to use the computer, how to write, wow, all sorts of tasks that you trying to teach the model all the different skills.

所以任何任务，例如，如何搜索网络、如何使用计算机、如何写作，哇，你试图教模型所有不同技能的各种任务。

And that's why we're saying there's no data wall or whatever, because there will be infinite amount of tasks and that's how the model becomes extremely super intelligent.

这就是为什么我们说没有数据墙或其他什么，因为会有无限量的任务，这就是模型变得极其超级智能的原因。

And we are actually getting saturated in all benchmarks.

事实上，我们的所有基准测试都已经饱和。

Karina Nguyen: So I think the bottleneck is actually in evaluations that we don't have all the frontier, like evals like, I don't know, GPQA, which is a Google-proof question answering, PhD level intelligence.

卡琳娜·阮（Karina Nguyen）：所以我认为瓶颈实际上在于我们没有所有前沿的评估，就像我不知道的 GPQA 那样的评估，这是一个经过谷歌验证的问答系统，博士级别的智能。

The benchmark is getting to, I don't know, more than 60, 70%, which is what PhD gets.

基准是达到，我不知道，超过 60%、70%，这是博士得到的。

So it's literally hitting the wall in like evals.

所以它实际上就像评估一样碰壁。

Lenny Rachitsky: I want to follow both those threads.

Lenny Rachitsky：我想关注这两条线索。

So the first is on this idea of synthetic data.

第一个是关于合成数据的想法。

Is a simple way to understand it, that the models are generating the data that future models are trained on and you ask it to generate all these ways of doing stuff, all these tasks as you described, and then the newer models trained on this data that the previous model generated?

理解它的一种简单方法是，模型正在生成未来模型训练的数据，你要求它生成所有这些做事的方式，所有这些任务，如你所描述的，然后新的模型根据先前模型生成的数据进行训练？

Karina Nguyen: Some tasks are synthetically curated.

Karina Nguyen：有些任务是综合策划的。

So this is an active research area is how can you synthetically construct new tasks with models to learn.

因此，这是一个活跃的研究领域，即如何通过模型来综合构建新任务来学习。

Sometimes when you develop products, you get a lot of data from the product and user feedback and you can use that data too in this cross-training world.

有时，当您开发产品时，您会从产品和用户反馈中获得大量数据，并且您也可以在这个交叉培训世界中使用这些数据。

Sometimes you still want to use human data because actually some of the tasks can be really, really hard to teach.

有时您仍然想使用人类数据，因为实际上有些任务可能非常非常难以教授。

Experts only know certain knowledge about some chemicals or biological knowledge, so you actually need to tap into the experts' knowledge a lot.

专家只知道一些化学知识或生物知识，所以你实际上需要大量挖掘专家的知识。

So yeah, I think to me synthetic data training is more for product...

所以，是的，我认为对我来说，合成数据培训更适合产品......

It's a rapid model iteration for similar product outcomes.

这是类似产品结果的快速模型迭代。

And we can dive more into it, but the way we made Canvas and tasks and new product features for ChatGPT was mostly done by synthetic training.

我们可以更深入地研究它，但我们为 ChatGPT 制作 Canvas 和任务以及新产品功能的方式主要是通过综合训练完成的。

Lenny Rachitsky: Let's actually get into that.

Lenny Rachitsky：让我们真正开始讨论这个问题。

That's really interesting.

这真的很有趣。

I want to talk about evals, but let's follow that thread.

我想谈谈评估，但让我们继续讨论这个主题。

So talk about how this helped you create Canvas.

那么请谈谈这如何帮助您创建 Canvas。

Karina Nguyen: So when I first came to OpenAI, I really had this idea of, "Okay, it would be really cool for ChatGPT to actually change the visual interface but also change the way it is with people." So going from being a chatbot to more of a collaborative agent, and the collaborator is a step towards more genetic systems that become innovators ultimately.

Karina Nguyen：所以当我第一次来到 OpenAI 时，我真的有这样的想法，“好吧，如果 ChatGPT 能够真正改变视觉界面，同时也改变人们与人相处的方式，那就太酷了。”因此，从聊天机器人转变为协作代理和协作者是迈向最终成为创新者的更多遗传系统的一步。

And so the entire team of applied engineers, designers, products, research got formed in the air almost out of nothing.

因此，整个应用工程师、设计师、产品、研究团队几乎是凭空形成的。

It's just like a collection of people who just got together and we rapidly started iterating with each other.

这就像一群人刚刚聚在一起，我们很快就开始互相迭代。

Karina Nguyen: Actually Canvas is one of the...

Karina Nguyen：实际上 Canvas 是...之一

I would say the first project at OpenAI, where researchers and applying engineers started working together from the very beginning of the product development cycle.

我想说的是 OpenAI 的第一个项目，研究人员和应用工程师从产品开发周期的一开始就开始合作。

And I think there's a lot of things that we have learned on the way, but I definitely came with the mindset of, "We need to do a really rapid model situation such that it would be much easier for engineers to work with the latest model possible, but also learn from user feedback or early internal dog food.

我认为我们在这一过程中学到了很多东西，但我的心态肯定是，“我们需要做一个非常快速的模型情况，这样工程师就可以更容易地使用最新的模型，同时也可以从用户反馈或早期的内部狗粮中学习。

How do we improve the model very rapidly?"

我们如何快速改进模型？”

Karina Nguyen: And it's really hard to kind of like figure out how people...

卡琳娜·阮（Karina Nguyen）：确实很难弄清楚人们如何......

when you deploy a product, how people would be able to use it.

当您部署产品时，人们将如何使用它。

And so the way you synthetically train the model is physically figuring out what are the most core behaviors that you wanted the product feature to do.

因此，综合训练模型的方式是从物理上找出您希望产品功能执行的最核心行为是什么。

And for Canvas, for example, it came down to three main behaviors.

以 Canvas 为例，它可以归结为三个主要行为。

It was how do you trigger Canvas for prompts like, "Write me a long essay," when the user intention is mostly iterating over long documents?

当用户意图主要是迭代长文档时，如何触发 Canvas 来提示“给我写一篇长文章”？

Or, "Write me a piece of code," or when to not trigger Canvas for prompts like, "Can you tell me more about President..." I don't know, some of the general questions.

或者，“给我写一段代码”，或者何时不触发 Canvas 进行提示，例如“你能告诉我更多关于总统的信息吗……”我不知道，一些一般性问题。

So you don't want to trigger Canvas because the user intention is mostly getting answer, not necessarily iterate over the long document.

因此，您不想触发 Canvas，因为用户的意图主要是获得答案，而不一定是迭代长文档。

Karina Nguyen: The second behavior is how do we teach the model to update the document when the user asks?

Karina Nguyen：第二个行为是我们如何教导模型在用户询问时更新文档？

So one of the behaviors that we taught the model is actually have some agency and autonomy to literally go to the document and select specific sections and either delete it or edit, so highlight it and rewrite certain sections.

因此，我们教授该模型的行为之一实际上是具有一定的代理和自主权，可以从字面上转到文档并选择特定部分，然后将其删除或编辑，因此突出显示它并重写某些部分。

Sometimes the user would just say, "Change the second paragraph to be something friendlier," and we would have to teach the model to literally find the second paragraph in the document and change it to a friendly tone.

有时用户只会说，“将第二段更改为更友好的内容”，我们必须教模型从字面上找到文档中的第二段并将其更改为友好的语气。

So basically you teach both how to trigger edit itself, but also how do you teach the model to get higher quality edit for the document?

所以基本上你既教如何触发编辑本身，又如何教模型获得更高质量的文档编辑？

Karina Nguyen: In case of coding, for example, there's also the question of how good the model is of completely rewriting the document, versus having a very specific target edits.

Karina Nguyen：例如，在编码的情况下，还有一个问题是，与进行非常具体的目标编辑相比，完全重写文档的模型有多好。

So that's another layer of decision boundary within edit itself is, "Let's select the entire document and rewrite completely, or do you want to have a very targeted custom behavior." And when we first launched the model, we would bias the model towards more rewrites because we saw the quality of the rewrites were much higher.

因此，编辑本身的另一层决策边界是，“让我们选择整个文档并完全重写，或者您想要有一个非常有针对性的自定义行为。”当我们第一次启动该模型时，我们会将模型偏向于更多重写，因为我们看到重写的质量要高得多。

But over time you are shifting based on user feedback and what you're learning from iterative deployment.

但随着时间的推移，您将根据用户反馈以及您从迭代部署中学到的知识进行转变。

Karina Nguyen: Lastly, the third behavior that we taught synthetically the model is how to make comments on any document.

Karina Nguyen：最后，我们综合教授该模型的第三个行为是如何对任何文档进行评论。

So the way we used that is we would use o1 model to seem a way of user conversation, let's say like, "Write me a document about XYZ." But then we used o1 to produce the document and then we injected user prompt to be like, "Oh, make some comments, critique my piece of writing or critique this piece of writing that you just made." And then we taught the model to make comments on the document on very specific So it's also what kind of comments you want the model to make.

因此，我们使用的方式是使用 o1 模型来作为用户对话的一种方式，比如“给我写一份关于 XYZ 的文档”。但是然后我们使用 o1 生成文档，然后我们注入用户提示，“哦，发表一些评论，批评我的文章或批评你刚刚写的这篇文章。”然后我们教模型对文档做出非常具体的评论，所以这也是你希望模型做出什么样的评论。

Do they make sense or not?

它们有道理吗？

How do you teach the quality of that?

你如何教授它的质量？

And it all came down to measuring progress via very robust evals.

这一切都归结为通过非常强大的评估来衡量进展。

But, yeah, this is how you used o1 and a synthetic data generation for the training.

但是，是的，这就是您使用 o1 和合成数据生成进行训练的方式。

Lenny Rachitsky: Okay, that's so interesting.

Lenny Rachitsky：好的，这很有趣。

So you talk about this idea of teaching the model and you mentioned how it's using synthetic data to teach the model different behaviors is a simple way to think about it.

因此，您谈到了教授模型的想法，并且提到了如何使用合成数据来教授模型不同的行为，这是一种简单的思考方式。

Basically that's where you do that by showing it what success looks like using basically evals.

基本上，这就是您通过使用基本评估向其展示成功是什么样子来做到这一点的地方。

Is that the simple way to think about it?

这是简单的思考方式吗？

Like, "Here's what you doing this successfully would look like," and that teaches it, "Okay, I see this is what I should be doing "

就像，“这就是你成功做到这一点的样子”，这教会了它，“好吧，我明白这就是我应该做的”

Karina Nguyen: Yeah, great.

卡琳娜·阮： 是的，太好了。

Yeah, amazing.

是的，太棒了。

Yeah, you got it.

是的，你明白了。

Lenny Rachitsky: Okay, got it.

莱尼·拉奇茨基：好的，明白了。

I want to start unpacking what your day-to-day looks like as you're building these sort of things.

我想开始解开你在构建这些东西时的日常生活。

Is it like you sitting there talking to some version of ChatGPT, crafting these evals?

是不是就像您坐在那里与某个版本的 ChatGPT 交谈，制作这些评估？

Karina Nguyen: Sometimes I do that.

卡琳娜·阮：有时我会这样做。

Sometimes I do sit with ChatGPT.

有时我确实会和 ChatGPT 坐在一起。

Actually, I think I learned this so much from Anthropic, is people spend so much time prompting models and where quality's a really bad batch all the time, and you actually get a lot of new ideas of how do you make the model better?

事实上，我想我从 Anthropic 学到了很多东西，人们花了这么多时间来提示模型，而质量一直很差，而你实际上得到了很多关于如何使模型更好的新想法？

It's like, "This response is kind of weird.

就像是，“这个反应有点奇怪。

Why's it doing this?" And you start debugging or something, or you start figuring out new methods of how do you teach the model to respond in the different way, have better personality, let's say.

它为什么要这样做？”然后你开始调试或者其他什么，或者你开始找出新的方法来教模型如何以不同的方式做出反应，比方说，拥有更好的个性。

Karina Nguyen: So it's the same thing of how personality is made in the models with those.

卡琳娜·阮（Karina Nguyen）：因此，模特的个性是如何塑造的。

It's very similar methods.

这是非常相似的方法。

But, yes, I think my time at OpenAI have changed.

但是，是的，我认为我在 OpenAI 的时间已经改变了。

I think when I first came, I was mostly research IC work so I was like building a lot of...

我想当我第一次来的时候，我主要是研究 IC 工作，所以我喜欢建造很多......

I was running code, training models, write evals, working with PMs and designers to learn, teach them how to even think about evaluation.

我运行代码，训练模型，编写评估，与产品经理和设计师一起学习，教他们如何思考评估。

I think that was really cool experience and I think it was just like an adoption of, "How do we do this product management of AI feature for our AI models?" Yeah, but now it's mostly management and mentorship.

我认为这是非常酷的体验，我认为这就像采用“我们如何为我们的人工智能模型进行人工智能功能的产品管理？”是的，但现在主要是管理和指导。

I'm still doing IC research code up to 4:00 PM, although.

尽管如此，我仍然在研究 IC 研究代码直到下午 4:00。

But I just kind of changed.

但我只是有点改变了。

Lenny Rachitsky: All right, don't talk too much about being a manager.

Lenny Rachitsky：好吧，不要过多谈论作为经理的事情。

Lenny Rachitsky: Because everyone's in firing their managers.

Lenny Rachitsky：因为每个人都在解雇他们的经理。

"Who needs managers anymore?" That's what I hear now.

“谁还需要经理？”这就是我现在听到的。

Just kidding.

只是在开玩笑。

It's interesting that so much of your time was spent on teaching product teams how evals integrate and how important it is.

有趣的是，您花费了大量时间来教导产品团队如何集成评估以及它的重要性。

And I've heard this a few times and I haven't personally experienced it yet, so I think it's an important thread to follow is just how writing these evaluations is going to become increasingly an important part of the job of product teams, especially when they're building AI features and working with LLMs.

我已经听过几次了，但我还没有亲自经历过，所以我认为这是一个重要的线索，那就是编写这些评估将如何成为产品团队工作中越来越重要的一部分，特别是当他们正在构建人工智能功能并与法学硕士合作时。

So can you just talk a bit more about what that looks like?

那么你能多谈谈它是什么样子吗？

Is it sitting there with an Excel spreadsheet basically showing, "Here's the input, here's the output, here's how good the result was"?

它是否坐在那里，里面有一个 Excel 电子表格，基本上显示“这是输入，这是输出，这是结果有多好”？

Talk about what that actually looks like very practically.

非常实际地谈论它的实际情况。

Karina Nguyen: It certainly depends on what you're developing, but there are various types of evaluations.

Karina Nguyen：这当然取决于您正在开发的内容，但评估有多种类型。

Sometimes I do ask product managers, or there's also new roles that we have, model designers, to go through some of the user feedback maybe or think of various user conversations that should have triggered...

有时我确实会要求产品经理，或者我们还有新的角色，模型设计师，来查看一些用户反馈，或者考虑应该触发的各种用户对话......

Under these circumstances, it should trigger Canvas.

在这种情况下，应该会触发 Canvas。

And then you have this ground truth label of, "Okay with this conversation it should look trigger Canvas, under this conversation it should not trigger Canvas." And you have this very deterministic kind of eval that for decision-making behaviors is like this.

然后你就有了这样的基本事实标签：“好吧，在这个对话中，它看起来应该触发 Canvas，在这个对话下，它不应该触发 Canvas。”你有一种非常确定性的评估，对于决策行为来说就是这样。

Karina Nguyen: When we were launching tasks, for example, how do you make correct schedules is actually really hard for the model.

Karina Nguyen：例如，当我们启动任务时，如何制定正确的时间表对于模型来说实际上非常困难。

But we built out some of the deterministic evaluations that is like, "Okay, if the user says 7:00 PM, the model should say 7:00 PM." So if you can have deterministic evals whether it's pass or fail.

但我们构建了一些确定性评估，例如“好吧，如果用户说晚上 7:00，模型应该说晚上 7:00。”因此，如果您可以进行确定性评估，无论它是通过还是失败。

And the way it works is all the...

它的工作方式是所有...

Sometimes I ask product managers to just go create a double sheet, have different tabs and what's the current behavior, what's the ideal behavior and why, and some notes.

有时我会要求产品经理创建一个双表，有不同的选项卡，当前的行为是什么，理想的行为是什么，为什么，以及一些注释。

Karina Nguyen: And sometimes they usually use it with evals, sometimes we use it for training.

Karina Nguyen：有时他们通常将其与评估一起使用，有时我们将其用于培训。

Because if you give the spreadsheet to o1 model, it can probably figure out how to teach itself a good behavior.

因为如果你将电子表格提供给 o1 模型，它可能会找出如何教会自己良好的行为。

And I think there are second type of evals that is more prevalent is human evaluations.

我认为第二种更普遍的评估是人类评估。

And you can have specific trainers or you can have internal people to when you have a conversation of the prompt and then you have various completion of models, you choose the win rate.

你可以有特定的培训师，也可以有内部人员，当你进行提示对话时，然后你有各种完成模型，你可以选择胜率。

Which model is the best?

哪种型号最好？

Which model produce the highest quality comment or edit?

哪种模型可以产生最高质量的评论或编辑？

And then you can have continuous win rates.

然后你就可以获得持续的胜率。

And as you develop new models it should always win over the previous models.

当您开发新模型时，它应该始终胜过以前的模型。

So it depends on what you want to measure.

所以这取决于你想要测量什么。

Lenny Rachitsky: So interesting.

莱尼·拉奇茨基：太有趣了。

Basically what I'm hearing, and there's something I'm learning about as I talk to people, is product development might move from this, "Here's a spec PRD, let's build it together and then cool, let's review it.

基本上我所听到的，也是我在与人们交谈时学到的东西，是产品开发可能会从此转变，“这是一个规范 PRD，让我们一起构建它，然后冷静，让我们回顾一下它。

Are we happy with this?" From that to, "Hey, AI, build this thing for me and here's what correct looks like," and I'm spending all my time on what does correct look like on evals essentially.

我们对此满意吗？”从那时起，“嘿，人工智能，为我构建这个东西，这就是正确的样子”，我把所有的时间都花在了评估上正确的样子上。

Karina Nguyen: You definitely want to measure progress of your model and this is where evals is, is because you can have prompted model as a baseline already.

Karina Nguyen：您肯定想要衡量模型的进度，这就是评估的位置，因为您已经可以将提示模型作为基线。

And the most robust evals is the one where prompted baselines get the lowest score or something.

最稳健的评估是提示基线获得最低分数或其他分数的评估。

And then because then you know if you're trained a good model, then it should just hill climb on that eval all the time, while not also regressing on other intelligence evals.

然后因为你知道如果你训练了一个好的模型，那么它应该一直在该评估上爬坡，同时不会在其他智力评估上回归。

That's what I'm saying, it's more of an art than science.

这就是我所说的，与其说它是一门科学，不如说它是一门艺术。

It's like, "Okay, if you optimize the model for this behavior, you don't want to brain damage in other areas of intelligence or..." This is happening all the time in every lab, in every research team.

就像，“好吧，如果你优化这种行为的模型，你就不想在其他智力领域造成大脑损伤，或者……”这种情况在每个实验室、每个研究团队中一直在发生。

Karina Nguyen: I would say prompting is also a way to prototype new product ideas.

Karina Nguyen：我想说，提示也是制作新产品创意原型的一种方式。

Early days at Anthropic when I was working file uploads feature, I remember I was just prompting the model to just...

早期在 Anthropic 工作时，我正在研究文件上传功能，我记得我只是提示模型......

I remember we were launching a hundred key contexts.

我记得我们启动了一百个关键上下文。

I was just prototyping this in their local browser.

我只是在他们的本地浏览器中对此进行了原型设计。

I did the demo.

我做了演示。

People really, really loved it.

人们真的非常喜欢它。

And they just wanted API for file uploads or something.

他们只是想要用于文件上传或其他功能的 API。

And then that's when it clicked to me, and also one of the blog posts a long time ago, it clicked on me prompting is a new way of product development or prototyping for designers and for product managers.

然后就在那时，它点击了我，也是很久以前的一篇博客文章，它点击了我，提示这是设计师和产品经理的产品开发或原型设计的新方法。

Karina Nguyen: For example, one of the features that I want to do is have a personalized starter prompts.

Karina Nguyen：例如，我想做的功能之一是具有个性化的启动提示。

So whenever you come to Claude, it should recommend you starter prompts based on what your interests are.

因此，每当您来到 Claude 时，它​​都会根据您的兴趣向您推荐入门提示。

And so you can literally do it prompting for that.

所以你实际上可以这样做提示。

Lenny Rachitsky: Mm-hmm.

莱尼·拉奇茨基：嗯嗯。

To experiment with that.

对此进行实验。

Karina Nguyen: Another feature was generating titles for the conversations.

Karina Nguyen：另一个功能是为对话生成标题。

It's a very small micro experience but I'm really proud of.

这是一次非常小的微观体验，但我真的很自豪。

The way we did that was we took five latest conversation from the model, asked the model, "What's the style of the user?" And then for the next new conversation, the generated title will be of the same style.

我们这样做的方式是，我们从模型中获取了五个最新的对话，询问模型，“用户的风格是什么？”然后对于下一个新对话，生成的标题将具有相同的样式。

It's just like really little micro experiences like this.

就像这样的微体验一样。

Lenny Rachitsky: That's so cool.

莱尼·拉奇茨基：那太酷了。

Did you do that at Anthropic or at OpenAI?

你是在 Anthropic 还是 OpenAI 这么做的？

Lenny Rachitsky: Okay, cool.

莱尼·拉奇茨基：好的，酷。

I love the file upload feature that Claude has by the way.

顺便说一句，我喜欢克劳德的文件上传功能。

ChatGPT doesn't have that yet, is that right?

ChatGPT 还没有这个功能，对吗？

Karina Nguyen: I think the way it's implement is very different though.

Karina Nguyen：我认为它的实施方式非常不同。

Lenny Rachitsky: Okay.

莱尼·拉奇茨基：好的。

Maybe it's the PDF feature, because I use it all the time with Claude.

也许是 PDF 功能，因为我一直和 Claude 一起使用它。

Lenny Rachitsky: Somebody needs to get on that.

Lenny Rachitsky：需要有人来解决这个问题。

Main, it's wild how many features you built that I use every day and that many people use every day.

主要是，您构建了多少我每天使用的功能以及许多人每天使用的功能，这真是太疯狂了。

This prototyping point you made is really important.

你提出的这个原型点非常重要。

It's something that comes up a ton on this podcast also of how that...

这也是这个播客中大量出现的内容，也是关于如何......

is maybe the way that AI has most impacted the job of product builders recently is just prototyping instead of going from showing just like, "Here's a PRD, here's a design." PMs are more and more just, "Here's the prototype with the idea that I have," and it's working.

人工智能最近对产品构建者的工作影响最大的方式可能只是原型设计，而不是像“这是一个 PRD，这是一个设计”那样展示。产品经理越来越公正地说，“这是带有我想法的原型”，而且它正在发挥作用。

You can play with it.

你可以玩它。

Lenny Rachitsky: Yeah.

莱尼·拉奇茨基：是的。

Okay, I want to spend a little more time on how you operate.

好吧，我想多花一点时间来谈谈你是如何运作的。

So you talked about you built this in launch of this tasks feature, is that the way to describe your tasks?

所以你谈到你在启动这个任务功能时构建了这个，这是描述你的任务的方式吗？

Lenny Rachitsky: So talk about how that emerged and let's better understand just how you collaborate with product teams and how OpenAI works in that way, whatever you can share there.

Lenny Rachitsky：那么请谈谈它是如何出现的，让我们更好地了解您如何与产品团队协作以及 OpenAI 如何以这种方式工作，无论您可以在那里分享什么。

Karina Nguyen: I think Canvas and tasks are going into the bucket of projects where it's more short or medium terms.

Karina Nguyen：我认为 Canvas 和任务正在进入短期或中期项目的范畴。

And actually the way Canvas and tasks came about to be was it started with one person prototyping and creating a spec.

实际上，Canvas 和任务的产生方式是从一个人原型设计和创建规范开始的。

It's kind of like PRD.

有点像PRD。

It's like creating a spec of the behavior of the model.

这就像创建模型行为的规范。

I don't think tasks is extremely groundbreaking feature necessarily.

我认为任务不一定是极具开创性的功能。

What makes it really cool is because the models are so general...

它真正酷的是因为模型太通用了......

Model can now search, they can write sci-fi stories, they can search for stocks, they can summarize the news every day.

模特现在可以搜索，他们可以写科幻故事，他们可以搜索股票，他们可以总结每天的新闻。

Because the models are so general giving something familiar to people that notifications is very familiar, having reminders is very familiar.

因为这些模型非常普遍，给人们提供熟悉的东西，所以通知非常熟悉，所以提醒也非常熟悉。

So feeling like a form factor for the people who are very familiar, same as Canvas, Google Docs is very familiar, but then you add magical AI moment and it becomes very powerful.

所以对于非常熟悉的人来说，感觉就像是一个外形因素，就像 Canvas、Google Docs 一样，非常熟悉，但是当你添加神奇的 AI 时刻，它就会变得非常强大。

Karina Nguyen: But the way it comes usually operationally...

卡琳娜·阮（Karina Nguyen）：但它的运作方式通常是......

Yeah, size is like a prototype, literally prompted prototype of how you would want the model to behave.

是的，尺寸就像一个原型，从字面上提示了您希望模型如何表现的原型。

For tasks, for example, you need to design...

例如，对于任务，您需要设计...

Literally design thinking is like okay, well, if the user says, "Remind me to go to lunch at 8:00 AM tomorrow," what information does the model need to extract from that prompt in order to create a reminder?

从字面上看，设计思维就像好吧，如果用户说，“提醒我明天早上 8:00 去吃午饭”，模型需要从该提示中提取哪些信息才能创建提醒？

And so this is how you design a spec for a new feature, like a tool.

这就是为新功能（例如工具）设计规范的方式。

Canvas and tasks are all tools.

画布和任务都是工具。

So it's like how do you create the tool stack?

那么就像如何创建工具堆栈一样？

Karina Nguyen: And then it's mostly like developing JSON schema.

Karina Nguyen：然后这主要就像开发 JSON 模式。

It was like, "Okay, from this problem maybe the model should extract the time that the user requested." And then you think about which format do you want the time to be?

就像，“好吧，也许模型应该从这个问题中提取用户请求的时间。”然后您考虑一下您希望时间采用哪种格式？

And then how do you want the model to notify you is basically the user should give instruction to the model.

然后你希望模型如何通知你，基本上就是用户应该向模型发出指令。

And then this instruction would fire off every day or something at that particular time.

然后这条指令会在每天或某个特定时间触发。

So, for example, if you say, "Every day I want to learn know about the latest AI news," the model should rewrite into, "Okay search for the latest AI news and this task will get fired at that particular type that the user requested."

因此，举例来说，如果你说，“我每天都想了解最新的人工智能新闻”，该模型应该重写为，“好吧，搜索最新的人工智能新闻，这个任务将在用户请求的特定类型上被触发。”

Karina Nguyen: And then your design is like tool spec.

Karina Nguyen：然后你的设计就像工具规格。

Actually, I don't know.

事实上，我不知道。

I feel like sometimes it's through conversations I...

我觉得有时候通过谈话我...

Either people ask me to join the team and they're like, "Oh my god, we need researchers." Or like, "We need some support.

要么有人邀请我加入这个团队，然后他们就会说：“天哪，我们需要研究人员。”或者说，“我们需要一些支持。

We need to train the models," or sometimes.

我们需要训练模型，”或者有时。

Canvas was mostly like I just pitched the idea of...

Canvas 基本上就像我刚刚提出的想法......

It got staffed quite immediately during the break, so it's dependent on the project.

在休息期间，它很快就配备了人员，所以这取决于项目。

And then usually with staffing is mostly a product manager, model designer, actual product designer, a couple of researchers and a bunch of applied engineers.

通常，人员配置主要是产品经理、模型设计师、实际产品设计师、一些研究人员和一群应用工程师。

Depends on the complexity of a project.

取决于项目的复杂程度。

And then for tasks it took, I don't know, like two months or so to go from zero to one basically.

然后对于任务，我不知道，大概需要两个月左右的时间才能从零到一。

Karina Nguyen: For Canvas this was like four, five months, I guess, to go from zero to one.

Karina Nguyen：对于 Canvas 来说，从零到一大概需要四五个月的时间。

And then you teach product managers how to build evals and maybe how do we not only ship the better feature, but how do we think longer term?

然后你教产品经理如何构建评估，也许我们不仅如何提供更好的功能，而且如何思考更长远的问题？

What kind of cool features did you want tasks to have?

您希望任务具有哪些很酷的功能？

I think it would be nice for tasks to be a little bit more personalized.

我认为任务更加个性化会更好。

It'd be nice to have to create tasks via voice on a mobile, right?

如果能够在手机上通过语音创建任务，那就太好了，对吧？

This is how you get research roadmap right here is thinking how the feature will be developed in the future.

这就是您在这里获取研究路线图的方式，即思考该功能未来将如何开发。

Karina Nguyen: And then from there it's like you start getting data sets.

Karina Nguyen：然后从那里开始获取数据集。

With evals, you want to make sure that goes well.

对于评估，您希望确保一切顺利。

And then you need to have a trade-off between what methods you want to use.

然后您需要在要使用的方法之间进行权衡。

And the reason why I really love relying purely on synthetic data instead of collecting data from humans is because it's much more scalable, it's cheap, less than half.

我真正喜欢纯粹依赖合成数据而不是从人类那里收集数据的原因是因为它更具可扩展性，而且便宜，不到一半。

You literally sample from the model and you teach the core behaviors of the models and that will generalize to all sorts of diverse coverage.

您从模型中进行抽样，并教授模型的核心行为，这将推广到各种不同的覆盖范围。

Karina Nguyen: And when you launch the beta feature, you learn so much from the users that you can...

Karina Nguyen：当你启动测试版功能时，你从用户那里学到了很多东西，你可以......

All your synthetic sets can be shifted in the distribution and how the users behave on the product behavior.

您的所有合成集都可以在分布以及用户对产品行为的表现上发生变化。

And this is how we improve.

这就是我们改进的方式。

And this is what happened with Canvass too when we launched from beta to GA.

当我们从测试版发布到正式版时，Canvass 也发生了这种情况。

Okay.: This episode is brought to you by Loom.

好的。：这一集是由 Loom 为您带来的。

Loom lets you your screen, your camera and your voice to share video messages easily.

Loom 让您可以通过屏幕、相机和声音轻松共享视频消息。

Record a Loom and send it out with just a link to gather feedback, add context or share an update.

记录 Loom 并仅通过链接发送出去，以收集反馈、添加上下文或共享更新。

So now you can delete that novel link email that you were writing.

所以现在您可以删除您正在编写的小说链接电子邮件。

Instead, you can record your screen and share your message faster.

相反，您可以录制屏幕并更快地分享您的消息。

Loom can help you have fewer meetings and make the meetings that you do have much more productive.

Loom 可以帮助您减少会议次数，并提高会议效率。

Okay.: Meetings start with everyone on the same page and end early.

好的。：会议开始时每个人都意见一致，然后提前结束。

Problem solved, time saved.

问题解决了，时间也节省了。

We know that everyone isn't a one-take wonder when it comes to recording videos.

我们知道，在录制视频方面，每个人都不是一次性的奇迹。

So Loom comes with easy editing and AI features to help you record once and get back to the work that counts.

因此，Loom 配备了简单的编辑和 AI 功能，可帮助您录制一次并重新开始重要的工作。

Save time, align your team, stay connected and get more done with Loom.

使用 Loom 节省时间、协调团队、保持联系并完成更多工作。

Now part of Atlassian, the makers of Jira.

现在是 Jira 的制造商 Atlassian 的一部分。

Try Loom for free today at Loom.com/Lenny.

立即访问 Loom.com/Lenny 免费试用 Loom。

That's L-O-O-M.com/Lenny.

那是 L-O-O-M.com/Lenny。

Okay.: Something that I want to help people understand, and I don't even 100% understand this, is what's the simplest way to understand the job of a researcher versus say a model designer and other folks involved?

好的。：我想帮助人们理解，但我什至不能 100% 理解这一点，与模型设计师和其他相关人员相比，理解研究人员工作的最简单方法是什么？

What's the simplest way to understand what researchers do at OpenAI?

了解 OpenAI 研究人员工作的最简单方法是什么？

Karina Nguyen: So the project that I described are mostly product-oriented.

Karina Nguyen：所以我描述的项目主要是面向产品的。

Research is mostly product research.

研究主要是产品研究。

Another component of my team is actually more longer term exploratory projects.

我团队的另一个组成部分实际上是更长期的探索性项目。

And it's more about developing new methods, understanding those methods under a variety of circumstances.

更多的是开发新方法，在各种情况下理解这些方法。

So basically developing methods, you need to follow very similar recipe of building evals but it's much more sophisticated evals.

因此，基本上开发方法，您需要遵循非常相似的构建评估方法，但它是更复杂的评估。

You want to have outer distribution or if you want to measure generalization, you need to capture that.

你想要有外部分布，或者如果你想要衡量泛化，你需要捕获它。

Karina Nguyen: But it is basically more sciencey in a way where...

卡琳娜·阮（Karina Nguyen）：但从某种程度上来说，它基本上更科学……

If we talk about synthetic data, one of the hardest things about synthetic data is how do you make it more diverse?

如果我们谈论合成数据，合成数据最困难的事情之一就是如何使其更加多样化？

Diversity in synthetic data is one of the most important questions right now.

合成数据的多样性是目前最重要的问题之一。

And so it's like exploring ways to inject diversity as a general method that will work for all is one of the research explorations.

因此，就像探索注入多样性的方法一样，将其作为一种适用于所有人的通用方法是研究探索之一。

Other ones is more developing new capabilities.

其他更多的是开发新功能。

I feel like it's always about you work on this new method and you have signs of life that it's working, either you think of how do you make it more general or you think of how do you make it very useful?

我觉得这总是关于你对这种新方法的研究，并且你有它正在发挥作用的生命迹象，要么你考虑如何使其更通用，要么你考虑如何使其变得非常有用？

And this is how the longer-term projects become more medium, short-term project.

这就是长期项目变成中短期项目的方式。

Lenny Rachitsky: That makes sense.

莱尼·拉奇茨基：这是有道理的。

Essentially working on developing ways to make the model smarter, o4, o5, o6.

本质上是致力于开发让模型变得更智能的方法，o4、o5、o6。

New ways to...

新方法...

o1 was a big breakthrough, right?

o1是一个重大突破，对吗？

Lenny Rachitsky: The way it operates where it's not just, "Here's your answer," it actually thinks and takes time to think through the process of coming up with an answer.

Lenny Rachitsky：它的运作方式不仅仅是“这是你的答案”，它实际上会思考并需要时间来思考得出答案的过程。

Okay.

好的。

Lenny Rachitsky: Very helpful.

莱尼·拉奇茨基：非常有帮助。

Speaking of that, of thinking about the future, where things are going, I want to spend some time on just this insight that basically you are building the cutting edge of AI, at the very bleeding edge of where AI is going and where it is.

说到这一点，谈到对未来的思考，事情的发展方向，我想花一些时间来了解这一点，即基本上你正在构建人工智能的前沿，处于人工智能的发展方向和现状的最前沿。

And so I'm very curious to hear just your take on how you think things are going to change in the world and how people work based on where you see things are going.

因此，我非常想听听您对世界将如何变化以及人们如何根据您所看到的情况进行工作的看法。

And I know it's a broad question, but let's say in the next three years, how do you see the world changing?

我知道这是一个广泛的问题，但让我们说在未来三年内，您如何看待世界的变化？

How do you see people's way of working changing?

您如何看待人们工作方式的变化？

Karina Nguyen: It's a very humbling experience to be in both labs, I guess.

Karina Nguyen：我想，在这两个实验室工作是一次非常令人谦卑的经历。

To me when I first came to Anthropic and I was like, "Oh no, I really love front-end engineering." And then the reason why I switched to research is because I realized at that time it's like, "Oh my god, Claude is getting better at front-end.

对我来说，当我第一次来到 Anthropic 时，我想，“哦不，我真的很喜欢前端工程。”然后我转向研究的原因是因为我当时意识到，“天哪，Claude 在前端越来越好了。

Claude is getting better at coding.

克劳德的编码能力越来越好。

I think Claude can develop new apps or something and so it can develop new features for the thing that I'm working." So it was kind of like this meta realization where it's like, "Oh my god, the world is actually changing." And when we first launched 100K context at that time, obviously I'm thinking about form factors that's like file uploads were very natural, very familiar to people.

我认为 Claude 可以开发新的应用程序或其他东西，因此它可以为我正在工作的东西开发新功能。”所以这有点像元认识，“天哪，世界实际上正在改变。”当我们第一次推出 100K 上下文时，显然我正在考虑外形因素，比如文件上传对于人们来说是非常自然、非常熟悉的。

But you can imagine we could just make infinite chats in the Claude.ai app, as if it's 100K context.

但你可以想象我们可以在 Claude.ai 应用程序中进行无限聊天，就好像它是 100K 上下文一样。

Karina Nguyen: But because file uploads...

Karina Nguyen：但是因为文件上传......

It's like form follows function.

就像形式服从功能一样。

It's like the form factor, the file uploads can enable people to just literally upload anything, the books, any reports, financial and ask any task to the model.

就像外形一样，文件上传可以让人们直接上传任何东西，书籍，任何报告，财务并向模型询问任何任务。

And then I remember it was either enterprise customers, financial customers were really interested in that.

然后我记得要么是企业客户，要么是金融客户对此非常感兴趣。

It's like, "Oh wow." It's actually one of the very common tasks that people do in that setting.

就像，“哇哦。”这实际上是人们在这种环境下执行的非常常见的任务之一。

It's kind of crazy to see how some of the redundant tasks are getting automated basically by these smart models.

看到这些智能模型基本上实现了一些冗余任务的自动化，这有点疯狂。

Karina Nguyen: And they're entering the era where, I actually don't know for example sometimes if o1 gives me the correct answer or not because I'm not an expert in that field.

卡琳娜·阮（Karina Nguyen）：他们正在进入这样一个时代，我实际上有时不知道 o1 是否给了我正确的答案，因为我不是该领域的专家。

And it's like, "I don't even know how to verify the outputs of the models." It's because all my experts know they can verify this.

就像，“我什至不知道如何验证模型的输出。”这是因为我所有的专家都知道他们可以验证这一点。

So, yes, so basically there are trends that are going on.

所以，是的，基本上有一些趋势正在发生。

The first trend is the cost of reasoning and intelligence is drastically going down.

第一个趋势是推理和智力的成本急剧下降。

Karina Nguyen: I had a blog post about this.

Karina Nguyen：我有一篇关于此的博客文章。

Maybe I should update on latest benchmarks, because at that time everybody was doing one benchmark and they'd be...

也许我应该更新最新的基准测试，因为当时每个人都在做一个基准测试，他们会......

quickly saturated the benchmarks.

基准很快就饱和了。

So I'm like, "Now we need to do the same plot but with another frontier eval." But the cost of intelligence is going down because it becomes that much cheaper.

所以我想，“现在我们需要做同样的情节，但进行另一个前沿评估。”但情报的成本正在下降，因为它变得便宜得多。

Small models are becoming even smarter than large models and that's because of the distillation research.

小模型变得比大模型更聪明，这是因为蒸馏研究。

Karina Nguyen: This happened with Claude 3 Haiku.

卡琳娜·阮（Karina Nguyen）：这发生在《克劳德 3 俳句》中。

I was working with the training on the Claude 3 Haiku and I realized it was much smarter than Claude 2, which was way bigger, lots .

我正在对 Claude 3 Haiku 进行训练，我意识到它比 Claude 2 聪明得多，后者要大得多。

But the power of small models become very intelligent and fast and cheap.

但小型模型的力量变得非常智能、快速且廉价。

We are moving towards that world.

我们正在走向那个世界。

That has multiple implications, but the news is that people will have more access AI and that's really good.

这具有多重含义，但消息是人们将有更多机会接触人工智能，这真的很好。

Builders and developers will have much better access to AI, but also it means all the work that has been bottlenecked by intelligence will be unblocked.

建筑商和开发人员将能够更好地接触人工智能，但这也意味着所有因智能而受到瓶颈的工作都将被畅通无阻。

Karina Nguyen: I'm thinking about healthcare, right?

Karina Nguyen：我正在考虑医疗保健，对吗？

Instead of going to a doctor, I can ask ChatGPT or give ChatGPT a list of symptoms and ask me, "Would I have a cold, flu, something else?" I can literally get the access to doctor almost.

我可以询问 ChatGPT 或向 ChatGPT 提供一系列症状，然后问我，“我会得感冒、流感还是其他疾病吗？”，而不是去看医生。我几乎可以去看医生了。

And there's been some research studies around that.

围绕这个问题已经有一些研究。

Lenny Rachitsky: There was a New York Times story about that where they compared doctors to doctors using ChatGPT to just ChatGPT and just ChatGPT was the best of them.

Lenny Rachitsky：《纽约时报》有一篇报道，他们将使用 ChatGPT 的医生与仅使用 ChatGPT 的医生进行比较，而仅使用 ChatGPT 是其中最好的。

All doctors made it worse.

所有的医生都让情况变得更糟。

Karina Nguyen: Yeah, that's crazy.

卡琳娜·阮：是的，这太疯狂了。

Yeah.

是的。

Yeah, that's crazy, right?

是的，这太疯狂了，对吧？

Education I think I would have dreamt if I had the tool like ChatGPT when I was young and would learn so much.

教育 我想如果我年轻的时候有像 ChatGPT 这样的工具，我会梦想着能学到很多东西。

But it's like people can now learn almost anything from these models.

但人们现在几乎可以从这些模型中学到任何东西。

So they can learn new language, they can learn how to build new look apps and write anything they do want.

这样他们就可以学习新的语言，可以学习如何构建新外观的应用程序并编写他们想要的任何内容。

It's humbling to have...

令人羞愧的是拥有...

launch Canvas and bring that thing to the people, enable them to do something else that they couldn't have ever before.

推出 Canvas 并将其带给人们，使他们能够做一些以前无法做的事情。

There's something magical around this experience.

这次经历有一些神奇的地方。

Karina Nguyen: Education will have massive implications.

Karina Nguyen：教育将产生巨大影响。

I guess like scientific research, I think it's the dream of any AI research is to automate AI research.

我想就像科学研究一样，我认为任何人工智能研究的梦想都是使人工智能研究自动化。

It's kind of scary, I'd say, which makes me think that people management will stay.

我想说，这有点可怕，这让我认为人员管理将会保留下来。

It's one of the hardest thing to...

这是最难的事情之一...

Emotional intelligence with the models, creativity in itself is one of the hardest things.

情商与模型、创造力本身就是最难的事情之一。

So writers, I don't think people should be worried as much.

所以作家们，我认为人们不应该那么担心。

I think will alleviate a lot of redundant tasks for people.

我认为这将为人们减轻很多多余的任务。

Lenny Rachitsky: This is awesome.

莱尼·拉奇茨基：这太棒了。

Okay, I want to follow this thread for sure.

好吧，我肯定想关注这个话题。

And it's funny that what you described as you were an engineer at Anthropic and you're like, "Okay, Claude is going to be very good at engineering.

有趣的是，当你描述你是 Anthropic 的一名工程师时，你会说，“好吧，克劳德将会非常擅长工程。

This isn't going to be a potentially career long term, so I'm going to move into research and AI is going to need me for a long time to build it, to make it smarter."

这不会是一个潜在的长期职业，所以我将进入研究领域，人工智能将需要我很长时间来构建它，使其变得更加智能。”

Karina Nguyen: I would say we still have...

卡琳娜·阮：我想说我们还有......

I think Canvas team has still have really cool front engineers that are really people who really care about interaction, design, interacting experience.

我认为 Canvas 团队仍然拥有非常酷的前端工程师，他们是真正关心交互、设计、交互体验的人。

I don't think models are there yet I think if...

我认为模型还没有存在，我想如果......

But we can get the models to this top 1% of front-ends and things for sure.

但我们肯定可以让模型达到前端的前 1%。

Lenny Rachitsky: So what I want to move on to next along these lines is just, and this is just speculation, but what skills do you think will be most valuable going forward for product teams in particular?

Lenny Rachitsky：所以我接下来要讲的只是，这只是猜测，但是您认为哪些技能对产品团队来说最有价值？

So folks are listening and they're like, "Okay, this is scary.

所以人们在听的时候会说，“好吧，这太可怕了。

What should I be building now to help me stay ahead and not be in trouble down the road?" What skills do you think are going to be more and more important to build?

我现在应该构建什么来帮助我保持领先并且不会在未来遇到麻烦？”您认为培养哪些技能将变得越来越重要？

Karina Nguyen: Yeah, I think creative thinking.

卡琳娜·阮：是的，我认为创造性思维。

You want to generate a bunch of ideas and filter through them and not just build the best product experience.

您想要产生一堆想法并过滤它们，而不仅仅是构建最佳的产品体验。

Listening.

倾听。

You want to build something that the most general model will not replace you.

您想要构建一些最通用的模型无法取代您的东西。

And oftentimes you build something and you make it really, really good for specific set of users and actually the mode is now in your user feedback.

通常，你构建了一些东西，并且让它对特定的用户组来说非常非常好，实际上该模式现在就在你的用户反馈中。

The mode is more in whether you listen to them, whether you can rapidly iterate.

模式更多在于你是否听他们的，是否能够快速迭代。

The mode is in here.

模式就在这里。

I don't think we are yet to...

我认为我们还没有...

There are so many ideas, I think there's an abundance of ideas that you can work on.

有很多想法，我认为有很多想法可以实现。

I wouldn't be worried.

我不会担心。

I feel like in fact I just think people in AI field are like...

我觉得事实上我只是认为人工智能领域的人们就像......

I wish they were a little bit more creative and connecting the dots across the print fields or something like that to develop really cool new generation and new paradigms of interactions with this AI.

我希望他们更有创意一点，将印刷领域或类似领域的点连接起来，以开发出非常酷的新一代和与人工智能交互的新范式。

Karina Nguyen: I don't think we've cracked this problem at all.

Karina Nguyen：我认为我们根本没有解决这个问题。

A couple of years ago I was telling some people, I was like, "You want to build for the future." So it's like it doesn't necessarily matter whether the model is good or not, good right now, but you can build product ideas such that by the time the models will be really good, it'll work really well.

几年前，我告诉一些人，“你想为未来而建设。”所以，就好像模型好不好，现在好并不一定重要，但你可以构建产品创意，这样当模型真的很好时，它就会工作得非常好。

I think it just happened naturally.

我认为这只是自然发生的。

For example, at Anthropic the Claude artifacts...

例如，在 Anthropic 的克劳德文物中......

And I feel early days of Canvas was, back in 2022 before ChatGPT, writing ideas was our knowledge .

我觉得 Canvas 的早期，早在 2022 年 ChatGPT 之前，写想法就是我们的知识。

But I feel like Claude 1.3 model itself was not there to have made really extreme good high quality edits.

但我觉得 Claude 1.3 模型本身并没有做出非常好的高质量编辑。

For example, like coding.

例如，像编码。

Karina Nguyen: And I feel like I see startups like Kaeser was doing super well.

Karina Nguyen：我觉得像 Kaeser 这样的初创公司做得非常好。

And that's because they iterate so fast.

那是因为它们的迭代速度非常快。

They invent new ways of training models.

他们发明了训练模型的新方法。

They move really fast.

他们移动得非常快。

They listen to what users like, massive distributions.

他们听取用户的喜好，大规模发行。

Yeah, it's kind of cool.

是的，这有点酷。

Lenny Rachitsky: That's really helpful actually.

Lenny Rachitsky：这实际上非常有帮助。

So what I'm hearing is that soft skills essentially are going to be more and more important, powerful.

所以我听到的是，软技能本质上会变得越来越重要、越来越强大。

You just talked about management, leading people, being creative and coming up with innovative insights, listening.

您刚才谈到了管理、领导人员、发挥创造力、提出创新见解以及倾听。

There's a post I wrote that I'll link to where I try to analyze how AI will impact product management.

我写了一篇文章，我将链接到我尝试分析人工智能将如何影响产品管理的地方。

And we're actually very aligned, and my sense was the same thing, that soft skills are going to become more and more important.

我们实际上非常一致，我的感觉是一样的，软技能将变得越来越重要。

And the things that are going to be replaced is the hard skills, which is interesting because usually people value the hard skills like coding, design, writing really well.

即将被取代的是硬技能，这很有趣，因为通常人们非常重视编码、设计、写作等硬技能。

And it's interesting that AI is actually really good at that because it's taking a bunch of data, synthesizing it and writing, creating a thing, versus all these fuzzy things around of what influences, convinces people to do things and aligning and listening, like you said, creativity, anything along those lines come up as I say that.

有趣的是，人工智能实际上非常擅长这一点，因为它获取大量数据，综合数据并编写，创造一个东西，而不是所有这些模糊的东西，影响，说服人们去做事，协调和倾听，就像你说的，创造力，正如我所说的，任何类似的东西都会出现。

Karina Nguyen: I think it's actually a really, really hard to teach the model how to be aesthetic or do really good visual design or how to be extremely creative in the way they write.

卡琳娜·阮（Karina Nguyen）：我认为教模特如何具有审美性或进行真正出色的视觉设计或如何在写作方式上极具创造力实际上非常非常困难。

I still think ChatGPT kind of sucks at writing and that's because it's bottlenecked by this creative reasoning.

我仍然认为 ChatGPT 在写作方面有点糟糕，那是因为它受到这种创造性推理的瓶颈。

I think characterization is one of the most important...

我认为人物塑造是最重要的之一...

I think for a manager, I feel like...

我认为对于一个经理来说，我觉得...

Karina Nguyen: Actually, AI research progress is bottlenecked by management, research management.

Karina Nguyen：实际上，人工智能研究进展的瓶颈在于管理、研究管理。

It's because you have constrained set of compute and you need to allocate the compute to the research paths that you feel the most convinced about.

这是因为您的计算集受到限制，并且您需要将计算分配给您最确信的研究路径。

It was like you need to have a really high conviction in the research paths to put the compute, and it's more return on investment kind of situation.

这就好像你需要对研究路径有非常高的信念才能投入计算，而且这是一种投资回报率更高的情况。

It's like, "Okay, I'm thinking a lot about across all my projects, which projects are higher priority?" Prioritization and also on the lower level, "Which experiments are really important to run right now and which are not?" and cut through the line.

就像，“好吧，我在所有项目中思考了很多，哪些项目的优先级更高？”优先级以及较低级别的问题是“哪些实验对于现在运行来说确实很重要，哪些不是？”并剪断线。

So I was thinking prioritization, communication, management.

所以我在考虑优先级、沟通、管理。

People skills like empathy, understanding people, collaboration.

人际交往能力，如同理心、理解他人、协作。

Karina Nguyen: I think Canvas wouldn't be an amazing launch if it wasn't about people and I think it's a wonderful group of people.

Karina Nguyen：我认为，如果 Canvas 的推出与人无关，那么它就不会是一次令人惊叹的发布，而且我认为它是一群很棒的人。

And I get a chance to work with people like Lee Byron who's a co-creator at GraphQL and some of the best Apple designers.

我有机会与像 Lee Byron 这样的人一起工作，他是 GraphQL 的联合创始人，也是一些最优秀的 Apple 设计师。

It's so cool to see...

看到它真是太酷了...

and how do you create this collaboration between people.

以及如何在人与人之间建立这种协作。

It's just something that's still humane, I think.

我认为这仍然是人性化的。

Lenny Rachitsky: Let me just follow through a little bit.

Lenny Rachitsky：让我简单介绍一下。

I imagine people listening are like, "Okay, but once we have AGI or SGI it's like it'll do all this." There's a world where like, "Why isn't all this done?" I think it's easy to just assume all that.

我想人们听的时候会想，“好吧，但是一旦我们有了 AGI 或 SGI，它就好像能做到这一切。”有一个世界，就像“为什么这一切还没有完成？”我认为假设这一切很容易。

I'm curious this idea of creativity and listening, why you think AI isn't good at it, other than it's just very hard to train it to do this well.

我很好奇创造力和倾听的想法，为什么你认为人工智能不擅长这件事，除了很难训练它做好这件事。

Is there anything there of just why this is especially difficult for AI and LLMs to get good at?

有什么原因可以解释为什么人工智能和法学硕士特别难以擅长这一点吗？

Karina Nguyen: I think currently it's difficult for many reasons.

Karina Nguyen：我认为目前这很困难，原因有很多。

I think it's still an active research area and it's something that I think my team is working on.

我认为这仍然是一个活跃的研究领域，我认为我的团队正在研究这一领域。

It's like, "Okay, how do we teach the models to be more creative in the writing?" And so I'm thinking this new paradigm of wise that the models think more should actually lead to better writing in itself.

就像，“好吧，我们如何教模型在写作中更具创造力？”所以我认为这种新的明智范式——模型思考得更多——实际上应该导致更好的写作本身。

But when it comes down to idea generation or discriminating of what is a good visual design or not, I feel like it hasn't had learned examples from people to discriminate it very well.

但当涉及到创意生成或区分什么是好的视觉设计或不好的视觉设计时，我觉得它还没有从人们那里学到例子来很好地区分它。

I do think it's because there are not that many people who are actually really...

我认为这是因为真正真正的人并不多...

It's not accessible to models to learn from these people I guess.

我猜模型无法向这些人学习。

So I definitely think that's why it sucks.

所以我绝对认为这就是它糟糕的原因。

Lenny Rachitsky: Yeah, that makes sense.

莱尼·拉奇茨基：是的，这是有道理的。

Basically there's not enough of you yet, researchers teaching it to do these things, slash people that have incredible taste and creativity that can teach these things.

基本上还没有足够的人，研究人员教它做这些事情，削减那些具有令人难以置信的品味和创造力的人，可以教这些事情。

You could argue this will come.

你可以说这将会到来。

Lenny Rachitsky: But we don't need to keep going down that thread.

Lenny Rachitsky：但我们不需要继续沿着这条线索走下去。

Let me ask you a specific question.

让我问你一个具体问题。

In this post I wrote, I made this argument that a lot of people disagreed with that strategy is something that AI tooling will become increasingly great at and take over.

在我写的这篇文章中，我提出了这样的论点：很多人不同意人工智能工具将变得越来越擅长并接管的策略。

There's the sense that that's the thing that people will continue to be much better at and you can't offload to AI basically developing your strategy, telling you what to do to win.

有一种感觉是，人们将继续在这方面做得更好，而你不能把工作交给人工智能来制定你的策略，告诉你如何取胜。

My case is, "Isn't strategy, just take all the inputs, all the data you have available, understand the world around you and come up with a plan to win?" It feels like AI and LLM would be incredibly smart at this.

我的情况是，“战略不就是利用所有的输入、所有可用的数据，了解周围的世界并制定一个获胜的计划吗？”感觉人工智能和法学硕士在这方面会非常聪明。

What's your take?

你怎么看？

Karina Nguyen: I think so too.

卡琳娜·阮：我也这么认为。

I think again, you teach the model all sorts of tools and capabilities and reasoning and it's like when it comes down to...

我再次想到，你教给模型各种工具、功能和推理，就像归根结底......

For Canvas right now, it would be very cool for the model just aggregate all the feedback from users, summarize me the top five most painful flows on user experiences.

对于目前的 Canvas 来说，如果该模型能够汇总用户的所有反馈，那就太酷了，总结一下用户体验方面最痛苦的五个流程。

And then the model itself is very capable of thinking of knowing how it's been made, figure out how to create a dataset for itself to train on it.

然后模型本身就非常有能力思考它是如何制作的，弄清楚如何为自己创建一个数据集来对其进行训练。

And I don't think that we are far away from that self-improvement, models becoming self-improved by...

我认为我们离自我完善并不遥远，模型通过……自我完善。

Karina Nguyen: That, and the part of development, is basically self-improving.

Karina Nguyen：这以及发展的一部分基本上是自我完善。

It's kind of like its own organism or something.

它有点像它自己的有机体什么的。

Again, like strategies, it's more like data analysis and coming up with...

再说一次，就像策略一样，它更像是数据分析并提出......

I think what models are really good at is connecting the dots, I think.

我认为模型真正擅长的是连接点。

It's like if you have user feedback from this source, but you also have an internal dashboard with metrics and then you have other feedback or input and then it can create a plan for you, recommendations even.

就像您从该来源获得用户反馈一样，但您还有一个包含指标的内部仪表板，然后您有其他反馈或输入，然后它可以为您创建计划，甚至是建议。

And I think this is one of the most common use cases for ChatGPT too, is coming up with these sort of things.

我认为这也是 ChatGPT 最常见的用例之一，就是提出此类事情。

Lenny Rachitsky: That makes sense essentially a human can only comprehend so much information at once and look at so much data at once to synthesize takeaways.

Lenny Rachitsky：从本质上讲，这是有道理的，因为人类只能一次理解这么多信息，一次查看这么多数据来综合结论。

And as you said, these context windows are huge now.

正如您所说，这些上下文窗口现在非常大。

Here's all the information, what's the most important thing I should do?

这是所有信息，我应该做的最重要的事情是什么？

Karina Nguyen: Yeah, same as scientific research.

Karina Nguyen：是的，和科学研究一样。

Ideally the model would be able to suggest ideas, new ideas, or iterate on the experimental given the empirical results of the previous experiments like how do you come up with new ideas or the methods?

理想情况下，模型能够提出想法、新想法，或者根据之前实验的经验结果迭代实验，比如你是如何提出新想法或方法的？

Lenny Rachitsky: Yeah.

莱尼·拉奇茨基：是的。

Oh, man.

哦，伙计。

Okay, so just to close the loop on this conversation, this part of the thread is the skills you're suggesting people focus on building and leaning into is soft skills like creativity, managing influence, collaboration, looking for patterns.

好的，为了结束这次对话，这部分内容是您建议人们重点培养和学习的软技能，例如创造力、管理影响力、协作、寻找模式。

Is that generally where your mind is at?

这通常是你的想法吗？

Karina Nguyen: Yeah, I'm thinking a lot about how do we make organizations more effectively and I think this is mostly management, I guess.

卡琳娜·阮（Karina Nguyen）：是的，我想了很多关于如何使组织更有效的问题，我想这主要是管理。

It's like how do you organize research teams or generally teams combined...

这就像你如何组织研究团队或一般团队的组合......

Compose teams such that they will be at their maximally succeed or at the maximal performance of what can possibly...

组建团队，使他们能够取得最大的成功或发挥最大的绩效......

We can literally create the next generation of computers.

我们确实可以创造下一代计算机。

It's just the matter of conviction and the way you manage through that.

这只是信念以及你处理信念的方式的问题。

It's scaling organizations or scaling product research, I guess.

我想，这是在扩大组织规模或扩大产品研究规模。

Lenny Rachitsky: Yeah, I think you're basically building this thing and not efficiently doing it is limiting the potential of the human species right now.

Lenny Rachitsky：是的，我认为你基本上是在建造这个东西，但如果不能有效地做到这一点，现在就会限制人类的潜力。

Lenny Rachitsky: It's mismanagement within the research team in OpenAI and Anthropic and some of these other models.

Lenny Rachitsky：OpenAI 和 Anthropic 以及其他一些模型的研究团队内部管理不善。

Karina Nguyen: Yeah, it's kind of crazy to think about it.

卡琳娜·阮（Karina Nguyen）：是的，想想这件事就有点疯狂。

Lenny Rachitsky: Holy moly.

莱尼·拉奇茨基：天啊。

Okay, so speaking of Anthropic and OpenAI, you've worked at both.

好的，说到 Anthropic 和 OpenAI，您都曾在这两个领域工作过。

Very few people have worked at both companies and have seen how they operate.

很少有人在这两家公司工作过并了解他们是如何运作的。

I'm curious just what you've noticed about the differences between these two, how they operate, how they think, how they approach stuff.

我很好奇你注意到这两者之间的差异，他们如何运作，他们如何思考，他们如何处理事情。

What can you share along those lines?

您可以分享哪些内容？

Karina Nguyen: It's more similar than different.

Karina Nguyen：相似之处多于不同之处。

Obviously there was a lot of...

显然有很多...

There are some differences always comes to nuances.

细微差别总是存在一些差异。

I would say culture.

我会说文化。

I really love Anthropic and I have a lot of friends there.

我真的很喜欢 Anthropic，在那里我有很多朋友。

And I also love OpenAI and they still have a lot of friends though.

我也喜欢 OpenAI，但他们仍然有很多朋友。

So it's not about enemies.

所以这与敌人无关。

I feel like there's in AI, it's all like, "Yeah, they're competitors.

我觉得人工智能领域，就像是，“是的，他们是竞争对手。

There's enemies." It's actually like one big community of people doing the same thing.

有敌人。”它实际上就像一个大社区，人们在做同样的事情。

I would say what I've learned from Anthropic is this real care and craft towards model behavior, model craft, model training.

我想说，我从 Anthropic 学到的是对模型行为、模型工艺、模型训练的真正关心和技巧。

Karina Nguyen: And I've been thinking a lot about, "Okay, what makes Claude Claude and what makes ChatGPT ChatGPT?" And it's like I still have some sense of operational processes that leads to the outputs, to the model.

Karina Nguyen：我一直在思考，“好吧，是什么造就了 Claude Claude，又是什么造就了 ChatGPT ChatGPT？”就好像我仍然对导致输出和模型的操作流程有一定的了解。

It's the outputed model.

这是输出的模型。

And it's like the reason why Claude has so much more personality and is more like a librarian...

这就像克劳德更有个性、更像图书管理员的原因一样……

I don't know.

我不知道。

I don't know.

我不知道。

I am visualizing Claude being like a librarian at some point, very nerdy or something.

我想象克劳德在某个时候就像一个图书管理员，非常书呆子之类的。

...

...

is because I feel like it's the reflection of the creators who are making this model.

因为我觉得这是制作这个模型的创作者的反映。

And a lot of details around the character and the personality and whether the model should follow up on this question or not.

还有很多关于角色和个性的细节，以及模特是否应该跟进这个问题。

Karina Nguyen: What's the correct ethical behavior for the model in these scenarios?

Karina Nguyen：在这些场景中，模型的正确道德行为是什么？

A lot of crafts and curated datasets.

很多工艺品和精选数据集。

This is where I learned that part of art, I guess, at Anthropic.

我想，这就是我在 Anthropic 学习艺术部分的地方。

I would say Anthropic is much smaller.

我想说 Anthropic 要小得多。

When I joined it was, what, like 70 people?

当我加入时，大概有 70 个人？

When I left it was tons of people.

当我离开时，已经有很多人了。

And obviously the culture changed so much.

显然，文化发生了很大变化。

I really enjoyed being early days startup lives, and people knew each other as a family.

我真的很喜欢早期的创业生活，人们像家人一样互相了解。

But the culture shifted.

但文化发生了变化。

Karina Nguyen: I would say that I learned from Anthropic that they're much better at focusing and prioritization of...

卡琳娜·阮（Karina Nguyen）：我想说，我从 Anthropic 那里了解到，他们更擅长关注和优先考虑......

Very hardcore prioritization, I guess.

我想，这是非常核心的优先顺序。

And they need to do it.

他们需要这样做。

But I think OpenAI's much more innovative and much more risk-takers in terms of product or research.

但我认为 OpenAI 在产品或研究方面更具创新性，也更敢于冒险。

Actually, in way your full-time job can be just teaching the model how to be creative writers.

实际上，在某种程度上，你的全职工作可以只是教模特如何成为创意作家。

And it's like there's some luxury in this research freedom that comes with scale, maybe.

也许随着规模的扩大，这种研究自由度变得有些奢侈了。

I don't know.

我不知道。

I'd say I have much more creative product freedom to do almost anything, I guess, within OpenAI, evolve ChatGPT into the vision that we want.

我想说，我想在 OpenAI 内，我有更多的创意产品自由去做几乎任何事情，将 ChatGPT 发展成我们想要的愿景。

It's more probably bottoms-up, I guess.

我猜这更可能是自下而上的。

Lenny Rachitsky: Yeah, that's how I was thinking about it.

Lenny Rachitsky：是的，我就是这么想的。

It feels like OpenAI is more bottoms-up, distributed, people bubble up ideas, try stuff.

感觉 OpenAI 更加自下而上、分布式，人们不断提出想法、尝试新东西。

And that leads to more products launching, I imagine more things just kind of being tried versus more of a, "Let's just make sure everything we do is awesome and great and craft and thinking deeply about every investment."

这会导致更多产品的推出，我想更多的事情只是被尝试而不是更多，“让我们确保我们所做的一切都很棒，并且对每一项投资都进行精心设计和深入思考。”

Lenny Rachitsky: That's really interesting.

Lenny Rachitsky：这真的很有趣。

I've never heard it described this way.

我从来没有听过这样的描述。

Karina, we've covered so much ground.

卡琳娜，我们已经讨论了很多内容。

This is going to help a lot of people with so many ways of thinking about where the future's going.

这将帮助很多人以多种方式思考未来的发展方向。

Before we get to our very exciting lightning round, I'm curious if there's anything else that you think might be helpful to share or get into?

在我们开始激动人心的闪电回合之前，我很好奇您认为还有什么可以分享或参与的内容吗？

Karina Nguyen: One of my regrets, I guess, when I was early days at Anthropic was that...

Karina Nguyen：我想，当我在 Anthropic 工作初期时，我的遗憾之一就是……

I think there was some luxury of the time, because pre-ChatGPT, to actually come in with a bunch of ideas and prototype almost every day.

我认为当时有一些奢侈，因为在 ChatGPT 之前，几乎每天都会有一堆想法和原型。

And I think that we did a lot of cool ideas like Claude, and Slack was actually one of the first tool-usey products.

我认为我们做了很多像 Claude 这样很酷的想法，而 Slack 实际上是第一个使用工具的产品。

It's like Claude could operate in your workplace now.

就像克劳德现在可以在你的工作场所工作一样。

It's kind of cool because you can add Claude to summarize the thread.

这很酷，因为您可以添加 Claude 来总结线程。

So maybe you have an entire conversation with someone and then you want a summary of what happened you can ask Claude, "Summarize this."

因此，也许您与某人进行了完整的对话，然后您想要总结所发生的事情，您可以问克劳德，“总结一下。”

Karina Nguyen: Also, it was really fun to iterate on the model itself.

Karina Nguyen：此外，迭代模型本身真的很有趣。

It's like when you just talk to the model in Slack forever.

这就像你永远在 Slack 中与模型交谈一样。

It created some social element, it was kind like and this Discord, people learned so much about prompting and how to work with Claude.

它创造了一些社交元素，就像这个 Discord，人们学到了很多关于提示以及如何与克劳德合作的知识。

Actually, one of the features that was early tasks prototype is every Monday Claude would just summarize the entire channel.

实际上，早期任务原型的功能之一是每周一克劳德都会总结整个频道。

Or every Friday we'd just summarize a bunch of channels and give the news about the organization, or something.

或者每个星期五我们都会总结一堆频道并提供有关组织的新闻或其他内容。

Karina Nguyen: And it's kind of like really cool form factor.

Karina Nguyen：它的外形非常酷。

I think thinking about form factor's a really important question in AI, especially we haven't even figured out how do we create an awesome product experience with o-series models.

我认为考虑外形因素是人工智能中一个非常重要的问题，特别是我们甚至还没有弄清楚如何使用 O 系列模型创造出色的产品体验。

It's like the paradigm between synchronous real time give an answer paradigm into more asynchronous paradigm of agents working on the background.

这就像同步实时给出答案范式与在后台工作的代理的异步范式之间的范式。

But then now the question is the agents should build trust with you, right?

但现在的问题是代理商应该与你建立信任，对吧？

And trust builds over time, which is like with humans.

信任随着时间的推移而建立，就像人类一样。

And you start this collaboration which is why this collaboration model with you and the model is so important because you build trust and the model learns from your preferences so that it can become more personalized and it will start predicting the next action that you want to take on the computer or something.

你开始这种协作，这就是为什么与你和模型的这种协作模型如此重要，因为你建立了信任，并且模型从你的偏好中学习，以便它可以变得更加个性化，并且它将开始预测你想要在计算机上执行的下一个操作或其他操作。

And it's more predictive, much more...

而且它更具预测性，更多......

We went from personal computers to personal model basically here.

我们基本上是从个人电脑转向个人模型。

Lenny Rachitsky: Why is it not a thing?

Lenny Rachitsky：为什么它不是一个东西？

That seems like such an obvious feature that every LLM should have as a Slack bot version of them.

这似乎是一个如此明显的功能，每个法学硕士都应该拥有它们的 Slack 机器人版本。

Is that a thing I can help you install?

这是我可以帮你安装的东西吗？

Or is that not a thing right now?

或者说现在还没有这样的事情？

Karina Nguyen: I know that Claude and Slack was sunset in 2023 or something.

Karina Nguyen：我知道 Claude 和 Slack 是在 2023 年日落之类的。

I think it was after ChatGPT was mostly the focus on customer use cases or enterprise use cases.

我认为这是在 ChatGPT 主要关注客户用例或企业用例之后。

Karina Nguyen: I think the form factor of Claude and Slack was kind of constrained a little bit when you want to talk about new features.

Karina Nguyen：我认为当你想谈论新功能时，Claude 和 Slack 的外形因素有点受到限制。

Karina Nguyen: I know that ChatGPT had Slackbar tools.

Karina Nguyen：我知道 ChatGPT 有 Slackbar 工具。

I don't know, maybe it will come back sometime.

我不知道，也许有一天它会回来。

Lenny Rachitsky: All right, I would pay for that.

Lenny Rachitsky：好吧，我愿意付钱。

Any other memories from that time of early days?

早期还有其他记忆吗？

Because that's a really special place to have been is early days Anthropic.

因为早期的 Anthropic 确实是一个非常特别的地方。

Any other memories or stories from that time that might be interesting to share?

当时还有其他有趣的回忆或故事可以分享吗？

Karina Nguyen: I think the very first launch when we felt...

Karina Nguyen：我认为第一次发布时我们感觉......

When click from use, again, was 100K context launch is when the models could input the entire book and give you a summary of the book or something.

当点击使用时，再次，100K 上下文启动是指模型可以输入整本书并为您提供该书或其他内容的摘要。

Or the financial...

或者金融...

or catalog multi files financial reports and then give you an answer to the question, to very specific questions.

或对多文件财务报告进行分类，然后为您提供非常具体的问题的答案。

I think there was something in there that was kind like, "Oh my god, this is a really cool new capability." Not model capability, but more like the capabilities that came from the product form factor itself rather than the model capability as much.

我认为其中有一些东西就像是“天哪，这是一个非常酷的新功能。”不是模型能力，但更像是来自产品外形因素本身的能力，而不是模型能力。

Karina Nguyen: I think other prototypes that we were thinking about...

Karina Nguyen：我认为我们正在考虑的其他原型......

There's one part having a Claude workspaces and it's kind of the same idea of Claude and I would have this shared workspace and that share workspace is like a document and we can iterate on the document.

其中一部分有一个 Claude 工作区，这与 Claude 的想法相同，我会拥有这个共享工作区，并且该共享工作区就像一个文档，我们可以迭代该文档。

And I feel like sometimes the ideas, and they're locked for two years, just like in this case.

我觉得有时这些想法会被锁定两年，就像在这个例子中一样。

Lenny Rachitsky: It's interesting, there's these milestones that kind of open up our view of what is happening and where things are going.

Lenny Rachitsky：很有趣，这些里程碑让我们对正在发生的事情和事情的发展方向有了更深入的了解。

ChatGPT think was the first of just like, "Wow, this is much better than I would've thought." You talked about 100K context windows where you could upload a book and ask it questions and have it summarize.

ChatGPT 认为这是第一个，“哇，这比我想象的要好得多。”您谈到了 100K 上下文窗口，您可以在其中上传一本书并向其提问并对其进行总结。

I actually use that all the time.

事实上我一直都在用它。

When I have interview guests and they wrote a book, I sometimes don't have time to read the whole book.

当我采访嘉宾并且他们写了一本书时，我有时没有时间读整本书。

So I use it to help me understand what the most interesting parts are.

所以我用它来帮助我理解最有趣的部分是什么。

And then I actually dive into the book, just to be clear.

然后我实际上深入研究了这本书，只是为了清楚起见。

And then, I don't know, maybe voice was another one where you could talk to say ChatGPT.

然后，我不知道，也许语音是另一种你可以说 ChatGPT 的方式。

Is there any other moments there that you're like, "Wow, this is much better than I thought it was going to be?"

是否有任何其他时刻让您感到“哇，这比我想象的要好得多？”

Karina Nguyen: Yeah, I think the computer use agents, like the model operating the desktop.

Karina Nguyen：是的，我认为计算机使用代理，就像操作桌面的模型一样。

And you can essentially think of new kind of experience where the model can learn the way you browse.

从本质上讲，您可以想到一种新的体验，模型可以学习您的浏览方式。

And from that preference it can just browse as just like you.

根据该偏好，它可以像您一样浏览。

It's kind of simulated persona.

这是一种模拟人物。

And it's actually very similar to the idea of like, "Okay, maybe Sam Altman doesn't have a lot of time.

这实际上与这样的想法非常相似，“好吧，也许萨姆·奥尔特曼没有很多时间。

Maybe I want to talk to his simulation and ask..." Or, for example, I really appreciate some of the technical mentorship.

也许我想与他的模拟交谈并询问......”或者，例如，我真的很感谢一些技术指导。

Yeah, cool.

是的，很酷。

But he doesn't have a lot of time so it's like I really want to ask him this questions.

但他没有太多时间，所以我真的很想问他这个问题。

How do you respond with simulated environments like this would be really cool.

你如何应对这样的模拟环境会非常酷。

Lenny Rachitsky: That's a great place to plug Lennybot, have one of those.

Lenny Rachitsky：这是插入 Lennybot 的好地方，拥有其中之一。

It's trained on all of my podcasts and newsletters.

它接受了我所有的播客和时事通讯的训练。

Lenny Rachitsky: It sits on many models.

Lenny Rachitsky：它适用于许多型号。

I don't know which exactly they use, but it's exactly that.

我不知道他们到底用的是哪个，但确实是这样。

And it's not even me, it's all the guests that have been on the podcast and on newsletter as I wrote.

甚至不是我，而是我所写的播客和时事通讯上的所有嘉宾。

And you could just ask it, "How do I grow my product?

你可以问它：“我如何发展我的产品？

How do I develop a strategy?" And it's actually shockingly good.

我该如何制定策略？”它实际上是令人震惊的好。

Karina Nguyen: Do you feel like it reflects who you are?

Karina Nguyen：你觉得它反映了你是谁吗？

Lenny Rachitsky: The best part of it is you can talk to it.

Lenny Rachitsky：最好的部分是你可以与它交谈。

There's an ElevenLabs voice version that's trained on my voice from this podcast, and it's actually very good and people have told me they sit there for hours talking to it.

有一个 ElevenLabs 语音版本是根据我在这个播客中的声音进行训练的，它实际上非常好，人们告诉我他们坐在那里与之交谈几个小时。

Lenny Rachitsky: And somebody told it, "Interview me like I am on Lenny's podcast, ask me questions about my career." And he did a half hour podcast episode with Lennybot.

莱尼·拉奇茨基：有人告诉我，“像我在莱尼的播客上一样采访我，问我有关我职业生涯的问题。”他还与 Lennybot 一起制作了半小时的播客节目。

Lenny Rachitsky: It's incredible.

莱尼·拉奇茨基：这太不可思议了。

Future is wild.

未来是狂野的。

Karina Nguyen: Yeah.

卡琳娜·阮：是的。

I think content transformation is...

我认为内容转型是...

I would imagine sometime when you generate a sci-fi story in Canvas, you can transform this into audiobook where you have very natural content transformation of one media to another media.

我想象有时当您在 Canvas 中生成科幻故事时，您可以将其转换为有声读物，其中您可以非常自然地从一种媒体到另一种媒体的内容转换。

I think one of my earliest inspiration is one of the last episodes of Westworld where, I don't want to spoil, but where Dolores comes to her work at that time and she comes to this new workspace and she starts writing a story.

我认为我最早的灵感之一是《西部世界》的最后一集，我不想剧透，但当时多洛雷斯开始工作，她来到这个新工作区，开始写一个故事。

And then as she writes a story, a 3D, virtual reality, starts creating on the fly.

然后，当她写故事时，3D 虚拟现实开始即时创作。

So I kind of want to create that.

所以我有点想创造它。

Kind of cool.

有点酷。

Lenny Rachitsky: Wow.

莱尼·拉奇茨基：哇。

Speaking of medium, I guess I was wondering if I should go in this direct or not, but real quick.

说到媒介，我想我想知道我是否应该直接进入，但是很快。

Kevin Weill/Kevin Weill, I don't know exactly how to pronounce his last name, the CPO of OpenAI.

Kevin Weill/凯文·威尔，我不知道他的姓氏到底如何发音，OpenAI 的 CPO。

Lenny Rachitsky: Weill.

莱尼·拉奇茨基：好吧。

Okay.

好的。

Okay.

好的。

Let's just say that.

就这么说吧。

We'll go with that.

我们就这样吧。

Lenny Rachitsky: He did a panel at the Lenny and Friends Summit last year and he made this really fascinating point that chat is a really interesting interface for these tools because they're just getting smarter and smarter and smarter and smarter and smarter.

Lenny Rachitsky：他去年在 Lenny 和朋友峰会上做了一个小组讨论，他提出了一个非常有趣的观点，即聊天对于这些工具来说是一个非常有趣的界面，因为它们变得越来越聪明、越来越聪明、越来越聪明。

And chat continues to work as a paradigm to just interact with them, similar to a human.

聊天仍然是与他们互动的范例，就像人类一样。

You could talk to Albert Einstein.

你可以和阿尔伯特·爱因斯坦谈谈。

You could talk to someone not very smart and it's all conversation still.

你可以和一个不太聪明的人交谈，但一切仍然只是谈话。

And so it's a really flexible way to interact with increasingly good intelligence.

因此，这是一种与日益优秀的智能互动的非常灵活的方式。

At some point it'll not be so great, and you were talking about all these ways that you're adding additional ways to interact.

在某些时候它不会那么好，并且您正在谈论添加额外的交互方式的所有这些方式。

But it's interesting chat proved to be a really powerful layer on top of all this stuff.

但有趣的是，聊天被证明是所有这些东西之上的一个非常强大的层。

Karina Nguyen: Yeah, that's real cool.

卡琳娜·阮：是的，那真的很酷。

I feel like chat also has social element which is very humane.

我觉得聊天也有社交元素，非常人性化。

It's like, yeah, you sometimes want to get into group chat.

就像，是的，你有时想加入群聊。

And having conversations with AI is kind of like a group chat in itself, as messaging.

与人工智能对话本身有点像群聊，就像消息传递一样。

Actually, this idea of how do you build features like this?

实际上，如何构建这样的功能？

I see tasks as this general feature that will scale very nicely as the models would develop new capabilities themselves.

我认为任务是一种通用功能，随着模型本身开发新功能，它可以很好地扩展。

The models will be able to do better searches and create new...

这些模型将能够进行更好的搜索并创建新的......

come up with more creative writing on render, react apps and like HTML apps.

在渲染、React 应用程序和 HTML 应用程序等方面提出更具创意的写作。

And you can have everyday new puzzle for you, every day continue the story from the previous days.

而且你每天都可以有新的谜题，每天都继续前几天的故事。

It scales very nicely.

它的伸缩性非常好。

Lenny Rachitsky: You mentioned something as we were getting into this extra section that we ended up going down is this idea of the agents using a computer.

Lenny Rachitsky：当我们进入这个额外部分时，您提到了一些内容，我们最终将其记录下来是使用计算机的代理的想法。

I know this is actually something you are going to launch today, the day we're recording it, which will be out by the time this comes out, called Operator, can you talk about this very cool feature that people will have access to?

我知道这实际上是你今天要发布的东西，我们录制它的那天，它会在发布时发布，称为 Operator，你能谈谈人们可以使用的这个非常酷的功能吗？

Karina Nguyen: Yeah, so I unfortunately did not work on that, but I'm really, really excited about this launch.

Karina Nguyen：是的，不幸的是我没有参与其中，但我对这次发布真的非常非常兴奋。

It's basically an agent that can complete the task in its own virtual computer, in its own virtual environment.

它基本上是一个代理，可以在自己的虚拟计算机、自己的虚拟环境中完成任务。

You can do any literally task like order me a book on Amazon.

你可以做任何字面上的任务，比如在亚马逊上给我订一本书。

And then ideally the model will either follow up with you which book do you want, or know you so well that it start recommending, "Oh, here is the five books that I might recommend you to buy." And then you hit, "Yeah, help me buy." And then the model goes off into its own virtual little browser and complete the task and buy the book on the Amazon.

然后理想情况下，该模型要么会跟进您想要哪本书，要么非常了解您，以至于开始推荐，“哦，这是我可能推荐您购买的五本书。”然后你就说：“是的，帮我买。”然后模型进入自己的虚拟小浏览器并完成任务并在亚马逊上购买这本书。

And then if you give the model credentials, credit cards, obviously it comes with a lot of trust and safety, then it will just complete the thing for you.

然后，如果您提供模型凭证、信用卡，显然它具有很大的信任和安全性，那么它就会为您完成事情。

It's a virtual assistant.

这是一个虚拟助手。

Lenny Rachitsky: It's interesting how this just sounds like obviously this should happen.

Lenny Rachitsky：有趣的是，这听起来显然应该发生。

Why is this not yet a thing?

为什么这还不是事呢？

Which is also mind-blowing that we're just assuming this should exist.

我们只是假设它应该存在，这也令人兴奋。

Just some AI doing things for you on a computer we just ask it to do.

只是一些人工智能在计算机上为您做一些我们只是要求它做的事情。

Karina Nguyen: It's actually really hard.

Karina Nguyen：实际上真的很难。

And I think you're still cracking this, but feel like...

我认为你仍在解决这个问题，但感觉...

I don't know if you use Tuple like a pair programming product.

不知道大家是否像结对编程产品一样使用Tuple。

Karina Nguyen: But at Anthropic we loved pair programming, so if you used-

Karina Nguyen：但是在 Anthropic，我们喜欢结对编程，所以如果你使用-

Lenny Rachitsky: Oh yeah, Shopify uses this.

Lenny Rachitsky：哦，是的，Shopify 使用这个。

I remember it came up on a podcast episode.

我记得它出现在播客节目中。

Karina Nguyen: Oh, nice.

卡琳娜·阮：哦，很好。

Yeah, so it is a very cool product where you can just call anyone at any time and then share screen and the other person can have access to the screen or start literally operating your computer.

是的，这是一个非常酷的产品，您可以随时呼叫任何人，然后共享屏幕，其他人可以访问屏幕或开始真正操作您的计算机。

And it's very realtime...

而且非常实时...

The allegiance is very...

效忠感非常...

it's very high quality.

它的质量非常高。

And it's just like I kind of want the same.

就像我也想要一样。

I want to pair program with my model and the model should even talk to me.

我想将程序与我的模型配对，模型甚至应该与我对话。

Draw very specific section in my code and just go to tell me...

在我的代码中绘制非常具体的部分，然后告诉我......

Obviously teach me and we can have different modes.

显然教我，我们可以有不同的模式。

It's like right, this is a product right here for you.

没错，这就是适合您的产品。

I don't know.

我不知道。

Some people should build that.

有些人应该建造它。

Lenny Rachitsky: It sounds like a startup just got birthed-

Lenny Rachitsky：听起来像是一家初创公司刚刚诞生——

Lenny Rachitsky: ...

莱尼·拉奇茨基：……

from someone listening to this.

来自听这个的人。

You mentioned that it's very hard to do this agent controlling a computer as you and helping out.

您提到让这个代理像您一样控制计算机并提供帮助是非常困难的。

What makes it so hard for whatever, however much you can explain briefly?

无论你能简单解释多少，是什么让事情变得如此困难？

Karina Nguyen: Much of it is because right now the model's operating on pixels instead of language or whatnot.

Karina Nguyen：很大程度上是因为现在模型在像素上运行，而不是语言或其他什么。

Pixels is actually really, really hard.

像素实际上真的非常非常难。

The models perception, or visual perception.

模型的感知或视觉感知。

I think there's still a lot of multimodal research that's going on, but I think language scaled so much easier compared to multimodal because of that.

我认为仍然有很多多模态研究正在进行，但我认为与多模态相比，语言的扩展要容易得多。

Karina Nguyen: Another thing that I guess my team is working that is how do you derive human intent very correctly?

Karina Nguyen：我猜我的团队正在研究的另一件事是如何非常正确地得出人类意图？

It's like sometimes does the model know enough information to ask a follow-up question or to complete the task?

就像有时模型是否知道足够的信息来提出后续问题或完成任务？

You don't want an agent to go off for 10 minutes and then come back with an answer that you didn't even want.

您不希望客服人员离开 10 分钟，然后回来时给出您甚至不想要的答案。

That actually creates much more worse user experience.

这实际上会造成更糟糕的用户体验。

And this comes with teaching the model people skills.

这与教授模特的人际交往能力有关。

It's like, "What do people like?

就像，“人们喜欢什么？

Kind of like creating the mental model of the user and care about the user in order to ask certain questions.

有点像创建用户的心理模型并关心用户以提出某些问题。

Actually, that part is hard to do for the models.

事实上，这部分对于模型来说很难做到。

Lenny Rachitsky: That relates to what we talked about earlier where this kind of the soft skill, people skills piece is not where these models are strong yet.

Lenny Rachitsky：这与我们之前讨论的相关，其中这种软技能、人际交往能力并不是这些模型的强项。

Lenny Rachitsky: Okay.

莱尼·拉奇茨基：好的。

I'm going to skip the lightning round.

我要跳过闪电回合。

I want to ask just one question from the lightning round, something fun.

我只想问闪电轮中的一个问题，一个有趣的问题。

Lenny Rachitsky: Okay, so when AI replaces your job, Karina, I'm curious what you're...

Lenny Rachitsky：好吧，当人工智能取代你的工作时，卡琳娜，我很好奇你在做什么......

And it gives you a stipend, gives you a monthly stipend.

它给你津贴，每月给你津贴。

Here's your salary for the month.

这是你这个月的工资。

What would you want to do?

你想做什么？

What do you want to spend your time on?

你想把时间花在什么事情上？

What will you be doing in this future world?

在这个未来的世界里你会做什么？

Karina Nguyen: I've been thinking about this a lot times.

Karina Nguyen：我已经多次思考过这个问题。

I feel like I have a lot of jobs options.

我觉得我有很多工作选择。

I would love to be a writer, I think.

我想，我很想成为一名作家。

I think that would be super cool.

我认为那会非常酷。

You should write short stories, sci-fi stories, novels.

你应该写短篇小说、科幻故事、长篇小说。

I really like art history, so you know those conservationists in the museums who just try to preserve art paintings, but just painting through a long day?

我真的很喜欢艺术史，所以你知道博物馆里的那些保护主义者只是想保护艺术画作，但只是在漫长的一天里画画吗？

Karina Nguyen: I think that would be really cool to do.

Karina Nguyen：我认为这样做真的很酷。

Yeah.

是的。

Lenny Rachitsky: What I'm hearing is you need to Nerf these models to not get very good at writing so that you can continue...

Lenny Rachitsky：我听到的是，你需要削弱这些模型，使其不擅长写作，这样你才能继续......

Although at that point you don't need to do it from...

尽管那时您不需要从...

You don't need people to buy it, you're just doing it for fun, so it doesn't even matter if they're incredibly good at writing or art conservation.

你不需要人们购买它，你只是为了好玩而做，所以他们是否非常擅长写作或艺术保护并不重要。

Oh man, what an episode of our conversation.

天哪，我们的谈话真是一段精彩的插曲。

What a wild time we're living in.

我们生活在一个多么狂野的时代啊。

Karina, thank you so much for being here.

卡琳娜，非常感谢你来到这里。

Two final questions.

最后两个问题。

Where can folks find you online if they want to reach out and follow up on anything?

如果人们想要联系并跟进任何事情，可以在哪里找到您？

And how can listeners be useful to you?

听众如何对你有用？

Karina Nguyen: You can find me, I'm on Twitter it's KarinaNguyen.

Karina Nguyen：你可以找到我，我在 Twitter 上，我是 KarinaNguyen。

You can also shoot me an email on my website.

您也可以在我的网站上给我发电子邮件。

And my team is hiring and so I'm looking for research engineers, research scientists, as well as machine learning engineers, people who come from product engineers who want to learn model training.

我的团队正在招聘，所以我正在寻找研究工程师、研究科学家以及机器学习工程师，这些人来自产品工程师，想要学习模型训练。

I'm actually hiring for my team.

我实际上正在为我的团队招聘。

My team is called Frontier Product Research, and we train models, we develop new methods but for product oriented outcomes.

我的团队称为前沿产品研究，我们训练模型，开发新方法，但都是以产品为导向的结果。

Lenny Rachitsky: What a place to work.

Lenny Rachitsky：真是一个工作的地方啊。

Holy moly.

圣钼。

What's the best way for people to apply for these very lucrative roles?

人们申请这些利润丰厚的职位的最佳方式是什么？

Karina Nguyen: I think you can shoot me a DM on Twitter.

Karina Nguyen：我想你可以在 Twitter 上给我发私信。

Karina Nguyen: Or I'm yet to create a job description for them.

Karina Nguyen：或者我还没有为他们创建职位描述。

Lenny Rachitsky: Okay.

莱尼·拉奇茨基：好的。

This is the job description.

这是职位描述。

Karina Nguyen: Or you can apply into post training team.

Karina Nguyen：或者你可以申请加入岗位培训团队。

Yeah.

是的。

Lenny Rachitsky: Okay.

莱尼·拉奇茨基：好的。

You're going to get a flood of DMs.

你将会收到大量的私信。

I hope you're prepared.

我希望你已经做好准备。

Karina, thank you so much for being here.

卡琳娜，非常感谢你来到这里。

This was incredible.

这太不可思议了。

Lenny Rachitsky: Thank you so much for listening.

莱尼·拉奇茨基：非常感谢您的聆听。

If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

如果您发现这很有价值，您可以在 Apple Podcasts、Spotify 或您最喜欢的播客应用程序上订阅该节目。

Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast.

另外，请考虑给我们评分或留下评论，因为这确实有助于其他听众找到播客。

You can find all past episodes or learn more about the show at LennysPodcasts.com.

您可以在 LennysPodcasts.com 上找到所有过去的剧集或了解有关该节目的更多信息。

See you in the next episode.

下一集见。