'Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & Amazon'
Lenny Rachitsky: We worked on a guest post together.
Lenny Rachitsky：我们一起写了一篇客座文章。
They had this really key insight that building AI products is very different from building non-AI products.
他们有一个非常重要的见解，即构建人工智能产品与构建非人工智能产品非常不同。
Aishwarya Naresh Reganti: Most people tend to ignore the non-determinism.
Aishwarya Naresh Reganti：大多数人倾向于忽视非决定论。
You don't know how the user might behave with your product, and you also don't know how the LLM might respond to that.
你不知道用户可能会如何使用你的产品，你也不知道法学硕士会如何回应。
The second difference is the agency control trade-off.
第二个区别是机构控制权的权衡。
Every time you hand over decision-making capabilities to agentic systems, you're kind of relinquishing some amount of control on your end.
每次您将决策能力移交给代理系统时，您就放弃了一定程度的控制权。
Lenny Rachitsky: This significantly changes the way you should be building product.
Lenny Rachitsky：这极大地改变了您构建产品的方式。
Kiriti Badam: So we recommend building step-by-step.
Kiriti Badam：所以我们建议逐步构建。
When you start small, it forces you to think about what is the problem that I'm going to solve.
当你从小处开始时，它会迫使你思考我要解决的问题是什么。
In all this advancements of the AI, one easy, slippery slope is to keep thinking about complexities of the solution and forget the problem that you're trying to solve.
在人工智能的所有这些进步中，一个简单的滑坡就是不断思考解决方案的复杂性，而忘记你正在尝试解决的问题。
Aishwarya Naresh Reganti: It's not about being the first company to have an agent among your competitors.
Aishwarya Naresh Reganti：这并不是要成为竞争对手中第一家拥有代理商的公司。
It's about have you built the right flywheels in place so that you can improve over time.
关键是您是否已安装了合适的飞轮，以便您可以随着时间的推移不断改进。
Lenny Rachitsky: What kind of ways of working do you see in companies that build AI products successfully?
Lenny Rachitsky：您认为成功打造人工智能产品的公司采用了哪些工作方式？
Aishwarya Naresh Reganti: I used to work with the CEO of now Rackspace.
Aishwarya Naresh Reganti：我曾经与现在的 Rackspace 首席执行官一起工作。
He would have this block every day in the morning, which would say catching up with AI 4:00 to 6:00 AM.
他每天早上都会有这个区块，也就是说在凌晨 4:00 到 6:00 赶上 AI。
Leaders have to get back to being hands-on.
领导者必须回归亲力亲为。
You must be comfortable with the fact that your intuitions might not be right.
您必须接受这样一个事实：您的直觉可能不正确。
And you probably are the dumbest person in the room and you want to learn from everyone.
你可能是房间里最愚蠢的人，你想向每个人学习。
Lenny Rachitsky: What do you think the next year of AI is going to look like?
Lenny Rachitsky：您认为人工智能的明年会是什么样子？
Kiriti Badam: Persistence is extremely valuable.
Kiriti Badam：坚持是极其宝贵的。
Successful companies right now building in any new area, they are going through the pain of learning this, implementing this and understanding what works and what doesn't work.
现在，成功的公司在任何新领域开展业务时，都会经历学习、实施、了解哪些有效、哪些无效的痛苦。
Pain is the new moat.
痛苦是新的护城河。
Lenny Rachitsky: Today, my guests are Aishwarya Reganti and Kiriti Badam.
Lenny Rachitsky：今天，我的嘉宾是 Aishwarya Reganti 和 Kiriti Badam。
Kiriti works on Kodex at OpenAI and has spent the last decade building AI and ML infrastructure at Google and at Kumo.
Kiriti 在 OpenAI 的 Kodex 工作，并在过去十年中一直在 Google 和 Kumo 构建 AI 和 ML 基础设施。
Ash was an early AI researcher at Alexa and Microsoft and has published over 35 research papers.
Ash 是 Alexa 和 Microsoft 的早期 AI 研究员，已发表超过 35 篇研究论文。
Together, they've led and supported over 50 AI product deployments across companies like Amazon, Databricks, OpenAI, Google, and both startups and large enterprises.
他们共同领导和支持了 Amazon、Databricks、OpenAI、Google 等公司以及初创公司和大型企业的 50 多个 AI 产品部署。
Together, they also teach the number one rated AI course on Maven, where they teach product leaders all of the key lessons they've learned about building successful AI products.
他们还一起在 Maven 上教授排名第一的人工智能课程，向产品领导者传授他们在构建成功的人工智能产品方面学到的所有关键课程。
The goal of this episode is to save you and your team a lot of pain and suffering and wasted time trying to build your AI product.
本集的目标是让您和您的团队在构建人工智能产品时避免很多痛苦和痛苦，并避免浪费时间。
Whether you are already struggling to make your product work or want to avoid that struggle, this episode is for you.
无论您是已经在努力让您的产品发挥作用还是想避免这种困难，本集都适合您。
If you enjoy this podcast, don't forget to subscribe and follow to your favorite podcasting app or YouTube.
如果您喜欢此播客，请不要忘记订阅并关注您最喜欢的播客应用程序或 YouTube。
Lenny Rachitsky: It helps tremendously.
Lenny Rachitsky：这很有帮助。
And if you become an annual subscriber of my newsletter, you get a year free of a ton of incredible products, including a year free of Lovable, Replit, Bold, Gamma, NA, and Linear Dev and Posttalk, Superhuman, Descript, Whisper Flow, Perplexity, Warp, Granola, Magic Mobbin, and Stripe Atlas.
如果您成为我的时事通讯的年度订阅者，您将获得一年免费的大量令人难以置信的产品，包括一年免费的 Lovable、Replit、Bold、Gamma、NA、Linear Dev 和 Posttalk、Super human、Descript、Whisper Flow、Perplexity、Warp、Granola、Magic Mobbin 和 Stripe Atlas。
Head on over to lennysnewsletter.com and click product pass.
前往 lennysnewsletter.com 并单击产品通行证。
With that, I bring you Aishwarya, Reganti, and Kiriti Badam after a short word from our sponsors.
在我们的赞助商简短发言后，我为您带来了艾西瓦娅 (Aishwarya)、雷甘蒂 (Reganti) 和基里蒂·巴达姆 (Kiriti Badam)。
Lenny Rachitsky: This episode is brought to you by Merge.
Lenny Rachitsky：本集由 Merge 为您带来。
Product leaders hate building integrations.
产品领导者讨厌构建集成。
They're messy.
它们很乱。
They're slow to build.
它们的建造速度很慢。
They're a huge drain on your roadmap, and they're definitely not why you got into product in the first place.
它们对你的路线图来说是一个巨大的消耗，而且它们绝对不是你首先进入产品的原因。
Lucky for you, Merge is obsessed with integrations.
幸运的是，Merge 痴迷于集成。
With a single API, B2B SaaS companies embed Merge into their product and ship 220 plus customer-facing integrations in weeks, not quarters.
通过单一 API，B2B SaaS 公司将 Merge 嵌入到他们的产品中，并在几周内（而不是几个季度）交付 220 多个面向客户的集成。
Lenny Rachitsky: Think of merge like Plaid, but for everything B2B SaaS.
Lenny Rachitsky：将合并想象成 Plaid，但适用于所有 B2B SaaS。
Companies like Mistral AI, Ramp, and Drata use Merge to connect their customers as accounting, HR, ticketing, CRM, and file storage systems to power everything from automatic onboarding to AI-ready data pipelines.
Mistral AI、Ramp 和 Drata 等公司使用 Merge 将其客户作为会计、人力资源、票务、CRM 和文件存储系统连接起来，为从自动入职到人工智能就绪数据管道的一切提供支持。
Even better, Merge now supports the secure deployment of connectors to AI agents with a new product so that you can safely power AI workflows with real customer data.
更好的是，Merge 现在支持通过新产品将连接器安全部署到 AI 代理，以便您可以使用真实的客户数据安全地为 AI 工作流程提供支持。
If your product needs customer data from dozens of systems, Merge is the fastest, safest way to get it.
如果您的产品需要来自数十个系统的客户数据，合并是最快、最安全的获取方式。
Book and attend a meeting at merge.dev/lenny, and they'll send you a $50 Amazon gift card.
在 merge.dev/lenny 上预订并参加会议，他们将向您发送一张 50 美元的亚马逊礼品卡。
That's merge.dev/lenny.
那是 merge.dev/lenny。
This episode is brought to you by Strella, the customer research platform built for the AI era.
本集由为人工智能时代打造的客户研究平台 Strella 为您带来。
Here's the truth about user research.
这是关于用户研究的真相。
It's never been more important or more painful.
它从未如此重要或如此痛苦。
Teams want to understand why customers do what they do, but recruiting users running interviews and analyzing insights takes weeks.
团队希望了解客户这样做的原因，但招募用户进行访谈并分析见解需要数周时间。
Lenny Rachitsky: By the time the results are in, the moment to act has passed.
Lenny Rachitsky：当结果出来时，采取行动的时刻已经过去了。
Strella changes that.
斯特雷拉改变了这一点。
It's the first platform that uses AI to run and analyze in depth interviews automatically, bringing fast and continuous user research to every team.
它是第一个使用人工智能自动运行和分析深度访谈的平台，为每个团队带来快速、持续的用户研究。
Strella's AI moderator asks real follow-up questions, probing deeper when answers are vague, and services patterns across hundreds of conversations all in a few hours, not weeks.
Strella 的人工智能主持人会提出真正的后续问题，当答案模糊时进行更深入的探讨，并且在几个小时而不是几周内就可以在数百个对话中提供服务模式。
Product, design, and research teams at companies like Amazon and Duolingo are already using Strella for Figma prototype testing, concept validation, and customer journey research, getting insights overnight instead of waiting for the next sprint.
Amazon 和 Duolingo 等公司的产品、设计和研究团队已经在使用 Strella 进行 Figma 原型测试、概念验证和客户旅程研究，在一夜之间获得见解，而不是等待下一个冲刺。
If your team wants to understand customers at the speed you ship products, try Strella.
如果您的团队希望以您交付产品的速度了解客户，请尝试 Strella。
Run your next study at strella.io/lenny.
请访问 strella.io/lenny 进行您的下一项研究。
That's S-T-R-E-L-L-A.io/lenny.
那是 S-T-R-E-L-L-A.io/lenny。
Ash and Kiriti, thank you so much for being here and welcome to the podcast.
Ash 和 Kiriti，非常感谢你们来到这里，欢迎来到播客。
Kiriti Badam: Thank you for having us.
Kiriti Badam：谢谢您邀请我们。
Super excited for this.
对此超级兴奋。
Lenny Rachitsky: Let me set the stage for the conversation that we're going to have today.
Lenny Rachitsky：让我为我们今天要进行的对话做好准备。
So you two have built a bunch of AI products yourself.
所以你们两个自己打造了一堆人工智能产品。
You've gone deep with a lot of companies who have built AI products, have struggled to build AI products, build AI agents.
您深入了解了许多构建人工智能产品的公司，在构建人工智能产品、构建人工智能代理方面遇到了困难。
You also teach a course on building AI products successfully and you're kind of on this mission to just reduce pain and suffering and failure that you constantly see people go through when they're building AI products.
你还教授一门关于成功构建人工智能产品的课程，你的使命就是减少人们在构建人工智能产品时不断经历的痛苦和失败。
So to set a little just foundation for the conversation we're going to have, what are you seeing on the ground within companies trying to build AI products?
因此，为了为我们即将进行的对话奠定一个公正的基础，您在尝试构建人工智能产品的公司中看到了什么？
What's going well?
进展顺利吗？
What's not going well?
什么事情进展不顺利？
Aishwarya Naresh Reganti: I think 2025 has been significantly different than 2024.
Aishwarya Naresh Reganti：我认为 2025 年与 2024 年有很大不同。
One, the skepticism has significantly reduced.
第一，怀疑情绪明显减少。
There were tons of leaders last year who probably thought this would be yet another crypto wave and kind of skeptical to get started.
去年有很多领导人可能认为这将是另一场加密浪潮，并且对开始持怀疑态度。
And a lot of the use cases that I saw last year were more of slap chat on your data.
我去年看到的很多用例更多的是对数据的拍打聊天。
And that was calling themselves an AI product.
这就是称自己为人工智能产品。
And this year, a ton of companies are really rethinking their user experiences and their workflows and all of that and really understanding that you need to deconstruct and reconstruct your processes in order to build successful AI products.
今年，大量公司正在重新思考他们的用户体验和工作流程以及所有这些，并真正理解您需要解构和重建流程才能构建成功的人工智能产品。
And that's the good stuff.
这就是好东西。
The bad stuff is the execution is still all over the place.
糟糕的是，执行仍然到处都是。
Think of it.
想一想吧。
This is a three-year-old field.
这是一个三年前的领域。
There are no playbooks, there are no textbooks.
没有剧本，没有教科书。
So you really need to figure out as you go.
所以你真的需要边走边弄清楚。
And the AI lifecycle, both pre-deployment and post-deployment is very different as compared to a traditional software lifecycle.
与传统软件生命周期相比，人工智能生命周期（部署前和部署后）都非常不同。
Aishwarya Naresh Reganti: And so a lot of old contracts and handoffs between traditional roles, like say PMs and engineers and data folks has now been broken and people are really getting adapted to this new way of working together and kind of owning the same feedback loop in a way.
Aishwarya Naresh Reganti：因此，传统角色之间的许多旧合同和交接，例如产品经理、工程师和数据人员，现在已经被打破，人们正在真正适应这种新的合作方式，并在某种程度上拥有相同的反馈循环。
Because previously, I feel like PMs and engineers and all of these folks had their own feedback loops to optimize.
因为以前，我觉得产品经理和工程师以及所有这些人都有自己的反馈循环需要优化。
And now you need to be probably sitting in the same room.
现在你们可能需要坐在同一个房间里。
You're probably looking at agent traces together and deciding how your product should behave.
您可能正在一起查看代理跟踪并决定您的产品应如何表现。
So it's a tighter form of collaboration.
所以这是一种更紧密的合作形式。
So companies are still kind of figuring that out.
所以公司仍在想办法解决这个问题。
That's kind of what I see in my consulting practice this year.
这就是我今年在咨询实践中看到的情况。
Lenny Rachitsky: So let me follow that thread.
Lenny Rachitsky：所以让我继续关注这个话题。
We worked on a guest post together that came out a few months ago.
我们一起写了一篇几个月前发表的客座帖子。
And the thing that stood out to me most that stuck with me most after working on that post is this really key insight that building AI products is very different from building non-AI products.
在完成那篇文章后，最让我印象深刻的是这个非常关键的见解，即构建人工智能产品与构建非人工智能产品非常不同。
And the thing that you're big on getting across is there's two very big differences.
你很想理解的是有两个非常大的差异。
Talk about those two differences.
谈谈这两个区别。
Aishwarya Naresh Reganti: Yes.
艾西瓦娅·纳雷什·雷甘蒂：是的。
And again, I want to make sure that we drive home the right point.
再次强调，我想确保我们的方向是正确的。
There are tons of similarities of building AI systems and software systems as well, but then there are some things that kind of fundamentally change the way you build software systems versus AI systems.
构建人工智能系统和软件系统也有很多相似之处，但有些事情会从根本上改变构建软件系统与人工智能系统的方式。
And one of them that most people tend to ignore is the non-determinism.
大多数人往往忽视的其中之一是非决定论。
You're pretty much working with a non-deterministic API as compared to traditional software.
与传统软件相比，您几乎使用的是非确定性 API。
What does that mean and why does that have to affect us is in traditional software, you pretty much have a very well-mapped decision engine or workflow.
这意味着什么以及为什么这必须影响我们，在传统软件中，您几乎拥有一个非常清晰的决策引擎或工作流程。
Think of something like Booking.com.
想想像 Booking.com 这样的网站。
You have an intention that you want to make a booking in San Francisco for two nights, et cetera.
您打算在旧金山预订两晚，等等。
The product has kind of been built so that your intention can be converted into a particular action and you kind of are clicking through a bunch of buttons, options, forms, and all of that, and you finally achieve your intention.
该产品的构建是为了让你的意图可以转化为特定的行动，你只需点击一堆按钮、选项、表单等等，你最终就能实现你的意图。
Aishwarya Naresh Reganti: But now that layer in AI products has completely been replaced by a very fluid interface, which is mostly natural language, which means the user can literally come up with a ton of ways of saying or communicating their intentions.
Aishwarya Naresh Reganti：但现在人工智能产品中的这一层已经完全被非常流畅的界面所取代，界面主要是自然语言，这意味着用户可以想出很多方式来表达或传达他们的意图。
And that kind of changes a lot of things because now you don't know how your user's going to be here.
这会改变很多事情，因为现在你不知道你的用户将如何在这里。
That's on the input side.
那是在输入端。
And the output is also that you're working with a non-deterministic probabilistic API, which is your LLM.
输出还表明您正在使用非确定性概率 API，这就是您的法学硕士。
And LLMs are pretty sensitive to prompt phrasings and they're pretty much black boxes.
法学硕士对提示措辞非常敏感，而且它们几乎是黑匣子。
So you don't even know how the output surface will look like.
所以你甚至不知道输出表面会是什么样子。
So you don't know how the user might behave with your product, and you also don't know how the LLM might respond to that.
所以你不知道用户可能会如何使用你的产品，你也不知道法学硕士会如何回应。
So you're now working with an input, output, and a process.
现在您正在处理输入、输出和流程。
You don't understand all the three very well.
你对这三点都不太了解。
You're trying to anticipate behavior and build for it.
您正在尝试预测行为并为其构建。
Aishwarya Naresh Reganti: And with agentic systems, this kind of gets even harder.
Aishwarya Naresh Reganti：对于代理系统，这种情况变得更加困难。
And that's where we talk about the second difference, which is the agency control trade-off.
这就是我们讨论第二个差异的地方，即机构控制权的权衡。
What we mean by that, and I'm kind of shocked so many people don't talk about this.
我们的意思是什么，我有点震惊这么多人不谈论这个。
They're extremely obsessed with building autonomous systems, agents that can do work for you.
他们非常热衷于构建自主系统和可以为你工作的代理。
But every time you hand over decision-making capabilities or autonomy to agentic systems, you're kind of relinquishing some amount of control on your end.
但每次你将决策能力或自主权交给代理系统时，你就放弃了一定程度的控制权。
And when you do that, you want to make sure that your agent has gained your trust or it is reliable enough that you can allow it to make decisions.
当您这样做时，您需要确保您的代理已获得您的信任，或者它足够可靠，您可以允许它做出决定。
And that's where we talk about this agency controlled trade-off, which is if you give your AI agent or your AI system, whatever it is, more agency, which is the ability to make decisions, you're also losing some control and you want to make sure that the agent or the AI system has earned that ability or has built up trust over time.
这就是我们谈论代理控制权衡的地方，也就是说，如果你给你的人工智能代理或你的人工智能系统，无论它是什么，更多的代理，即做出决策的能力，你也会失去一些控制权，你希望确保代理或人工智能系统已经赢得了这种能力或随着时间的推移建立了信任。
Lenny Rachitsky: So just to summarize what you're sharing here, essentially, people have been building product, software products for a long time.
Lenny Rachitsky：总结一下您在这里分享的内容，本质上，人们长期以来一直在构建产品、软件产品。
We're now in a world where the software you're building is one, non-deterministic, can just do things differently.
我们现在所处的世界是，您正在构建的软件是一个非确定性的软件，只能以不同的方式做事。
As you said, you go to booking.com, you find a hotel, it's going to be the same experience every time.
正如你所说，你访问 Booking.com，找到一家酒店，每次都会有相同的体验。
You'll see different hotels, but it's a predictable experience.
您会看到不同的酒店，但这是一种可以预见的体验。
With AI, you can't predict that it's going to be the exact same thing, the thing that you plan it to be every time.
对于人工智能，你无法预测它会是完全相同的事情，即你每次计划的事情。
And then the other is there's this trade-off between agency and control.
另一个是代理与控制之间的权衡。
How much will the AI do for you versus how much should the person still be in charge?
人工智能会为你做多少事，而这个人还应该负责多少事？
And what I'm hearing is the big point here is this significantly changes the way you should be building product.
我听到的重点是，这极大地改变了您构建产品的方式。
And we're going to talk about the impact on how the product development lifecycle should change as a result.
我们将讨论产品开发生命周期应如何改变的影响。
Lenny Rachitsky: Is there anything else you want to add there before we get into that?
Lenny Rachitsky：在我们开始讨论之前，您还有什么要补充的吗？
Kiriti Badam: Yeah, it's definitely one of the key points that this kind of distinction needs to exist in your mind when you're starting to build.
Kiriti Badam：是的，当你开始构建时，这绝对是你脑海中需要存在这种区别的关键点之一。
For example, think about if your objective is to hike Half Dome in Yosemite.
例如，考虑一下您的目标是否是徒步优胜美地的半圆顶。
You don't start hiking it every day, but you start training yourself in minor parts and then you slowly improve and then you go to the end goal.
你不会开始每天徒步旅行，而是开始从小部分开始训练自己，然后慢慢提高，然后到达最终目标。
I feel like that's extremely similar to what you want to build AI products in the sense that when you don't start with agents with all the tools and all the context that you have in the company in day one and expect it to work or even tinker at that level.
我觉得这与你想要构建的人工智能产品非常相似，因为你不从第一天开始就使用拥有公司所有工具和所有背景的代理，并期望它能够工作，甚至在那个级别进行修补。
You need to be deliberately starting in places where there is minimal impact and more human control so that you have a good grip of what are the current capabilities and what can I do with them and then slowly lean into the more agency and lesser control.
你需要有意识地从影响最小、人为控制更多的地方开始，这样你就能很好地掌握当前的能力是什么以及我能用它们做什么，然后慢慢地倾向于更多的代理和更少的控制。
Kiriti Badam: So this gives you that confidence that, okay, I can know that, okay, this is the particular problem that I'm facing and the AI can solve this extent of it.
Kiriti Badam：所以这给了你信心，好吧，我可以知道，好吧，这是我面临的特殊问题，人工智能可以解决这个问题。
And then let me next think through what context I need to bring in, what kind of tools I need to add to this to improve the experience.
然后让我思考一下我需要引入什么背景，我需要添加什么样的工具来改善体验。
So I feel like also it's a good and a bad thing in the sense that it's good that you don't have to see the complexity of the outside world of all of this fancy AI agents force and feel like I cannot do that.
所以我觉得这既是一件好事，也是一件坏事，因为你不必看到所有这些奇特的人工智能代理力量的外部世界的复杂性，并且感觉我无法做到这一点，这是件好事。
Everyone is starting from very minimalistic structures and then evolving.
每个人都是从非常简约的结构开始，然后不断发展。
And the second part is the bad thing is that as you are trying to build this one click agents into your company, you don't have to be overwhelmed with this complexity.
第二部分是坏事，当您尝试在您的公司中构建此一键式代理时，您不必因这种复杂性而不知所措。
You can slowly graduate.
就可以慢慢毕业了。
Kiriti Badam: So that's extremely important.
Kiriti Badam：所以这非常重要。
And we see this as a repeating pattern over and over.
我们认为这是一种一遍又一遍重复的模式。
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
So let's actually follow that because that's a really important component of how you recommend people build AI stuff, AI products, AI agents, all the AI things.
因此，让我们实际遵循这一点，因为这是建议人们如何构建人工智能产品、人工智能产品、人工智能代理以及所有人工智能事物的非常重要的组成部分。
So give us an example of what you're talking about here, this idea of starting slow with agency and control and then moving up this rung.
因此，请给我们举一个例子，说明您在这里谈论的内容，即通过代理和控制缓慢开始，然后向上提升这一想法。
Kiriti Badam: Yeah.
基里蒂·巴达姆：是的。
For example, a very important or very prevalent application of AI agents is customer support.
例如，人工智能代理的一个非常重要或非常普遍的应用是客户支持。
Imagine you are a company who has a lot of customer support tickets and why even imagine OpenAI is the exact same thing when we were launching products and there was a huge spike of support volume as we launched successful products like Image or GPT-5 and things like that.
想象一下，您是一家拥有大量客户支持票的公司，为什么甚至想象 OpenAI 与我们推出产品时完全相同，并且当我们推出 Image 或 GPT-5 等成功产品时，支持量激增。
The kind of questions you get is different.
您收到的问题类型有所不同。
The kind of problems that the customers bring to you is different.
客户给你带来的问题类型是不同的。
So it's not about just dumping all the list of help center articles that you have into the AI agent.
因此，这不仅仅是将所有帮助中心文章列表转储到人工智能代理中。
You kind of understand what are the things that you can build.
你有点了解你可以建造什么东西。
And so initially the first step of it would be something like you have your support agents, the human support agents, but you will be suggesting in terms of, okay, this is what the AI thinks that is the right thing to do.
因此，最初的第一步就像你有你的支持代理，人类支持代理，但你会建议，好吧，这就是人工智能认为正确的做法。
Kiriti Badam: And then you get that feedback loop from the humans that, okay, this is actually a good suggestion for me in this particular case and this is a bad suggestion.
Kiriti Badam：然后你会从人类那里得到反馈循环，好吧，在这个特殊情况下，这对我来说实际上是一个很好的建议，而这是一个糟糕的建议。
And then you can go back and understand, okay, this is what the drawbacks are or this is where the blind spots are, and then how do I fix that?
然后你可以回过头来理解，好吧，这就是缺点或者盲点，然后我该如何解决呢？
And once you get that, you can increase the autonomy to say that, okay, I don't need to suggest to the human.
一旦你明白了这一点，你就可以增加自主权，说，好吧，我不需要向人类建议。
I'll actually show the answer directly to the customer.
我实际上会直接向客户展示答案。
And then we can actually add more complexity in terms of, okay, I was only answering questions based on health center articles, but now let me add new functionality.
然后我们实际上可以增加更多的复杂性，好吧，我只是根据健康中心文章回答问题，但现在让我添加新功能。
I can actually issue refunds to the customers.
我实际上可以向客户退款。
I can actually raise feature requests with the engineering team and all of these things.
我实际上可以向工程团队提出功能请求以及所有这些事情。
So if you start with all of this on day one, it's incredibly hard to control the complexity.
因此，如果您从第一天就开始执行所有这些操作，那么控制复杂性将非常困难。
Kiriti Badam: So we recommend building step by step and then increasing it.
Kiriti Badam：所以我们建议逐步构建，然后再增加。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
And you have a visual actually that we'll share of what this looks like.
实际上，您已经看到了我们将分享的视觉效果。
But just to kind of mirror back what you're describing, this idea of start with high control, low agency, the example you gave is the support agents just kind of giving suggestions, is not able to do anything, the user is in charge.
但只是为了反映你所描述的内容，这种从高控制、低代理开始的想法，你给出的例子是支持代理只是提供建议，不能做任何事情，用户负责。
And then as that becomes useful and you are confident it's doing the right sort of work, you give it a little more agency and you kind of pull back on the control the user has.
然后，当它变得有用并且你确信它正在做正确的工作时，你会给它更多的代理权，并且你会收回用户拥有的控制权。
And then if that's starting to go well, then you give it more agency and the user needs less control to control it.
然后，如果一切开始顺利，那么你就给它更多的代理权，用户需要更少的控制来控制它。
Awesome.
惊人的。
Aishwarya Naresh Reganti: I think the higher level idea here is with AI systems, it's all about behavior calibration.
Aishwarya Naresh Reganti：我认为这里更高层次的想法是人工智能系统，这都是关于行为校准的。
It's incredibly impossible to predict upfront how your system behaves.
预先预测系统的行为方式是极其不可能的。
Now, what do you do about it?
现在，你对此做什么？
You make sure that you don't ruin your customer experience or your end user experience.
您确保不会破坏您的客户体验或最终用户体验。
You keep that as is, but then remove the amount of control that the human has.
您保持原样，但随后消除人类拥有的控制量。
And there is no single right way of doing it.
并且没有单一正确的方法来做到这一点。
You can decide how to constrain that autonomy.
您可以决定如何限制这种自主权。
I mean, a different example of how you could constrain autonomy is pre-authorization use cases.
我的意思是，如何限制自主权的另一个例子是预授权用例。
Insurance pre-authorization is a very ripe use case for AI because clinicians spend a lot of time pre-authorizing things like blood tests, MRIs and things like that.
保险预授权是人工智能的一个非常成熟的用例，因为临床医生花费大量时间来预授权诸如血液测试、核磁共振成像之类的事情。
And there are some cases which are more of low hanging fruits.
在某些情况下，更容易实现目标。
For instance, MRIs and block tests, because as soon as you know patient's information, it's easier to approve that and AI could do that versus something like an invasive surgery, et cetera, is more high risk.
例如，核磁共振成像和块测试，因为一旦你知道患者的信息，就更容易批准这一点，而人工智能可以做到这一点，而侵入性手术等风险更高。
You don't want to be doing that autonomously.
你不想自主地这样做。
Aishwarya Naresh Reganti: So you can kind of determine which of these use cases should go through that human and the loop layer versus which of the use cases AI can conveniently handle.
Aishwarya Naresh Reganti：因此，您可以确定哪些用例应该经过人工和循环层，以及哪些用例可以由人工智能方便地处理。
And then all through this process, you're also logging what the human is doing because you want to build a flywheel that you could use in order to improve your system.
然后在整个过程中，您还记录了人类正在做的事情，因为您想构建一个可以用来改进系统的飞轮。
So you're essentially not showing the user experience, not eroding trust, at the same time logging what humans would otherwise do so that you can continuously improve your system.
因此，您本质上不会展示用户体验，不会削弱信任，同时记录人们会做的事情，以便您可以不断改进您的系统。
Lenny Rachitsky: So let me give you a few more examples of this kind of progression that you recommend.
Lenny Rachitsky：那么让我再给你举几个你推荐的这种进展的例子。
And the reason I'm spending so much time here is this is a really key part of your recommendation to help people build more successful AI products.
我在这里花费这么多时间的原因是，这是您帮助人们构建更成功的人工智能产品的建议的真正关键部分。
This idea of start slow with high control and low agency and then build up over time once you've built confidence that it's doing the right sort of work.
这种想法是缓慢地开始，具有高度的控制力和较低的代理性，然后随着时间的推移，一旦你建立了对它正在做正确的工作的信心，就会逐渐建立起来。
So a few more examples that you shared in your post that I'll just read.
我将阅读您在帖子中分享的更多示例。
So say you're building a coding assistant, V1 would be just suggest inline completion and boilerplate snippets.
假设您正在构建一个编码助手，V1 只是建议内联完成和样板片段。
V2 would be generate larger blocks like tests or refactors for humans to review.
V2 将生成更大的块，例如测试或重构，供人类审查。
And then V3 is just apply the changes and open PRs autonomously.
然后V3只是应用更改并自主打开PR。
And then another example is a marketing assistant.
另一个例子是营销助理。
So V1 would be draft emails or social copy, just like here's what I would do.
因此，V1 将是电子邮件草稿或社交副本，就像我要做的那样。
Lenny Rachitsky: V2 is build a multi-step campaign and run the campaign.
Lenny Rachitsky：V2 正在构建一个多步骤的活动并运行该活动。
And V3 is just launch it A/B tested auto-optimize campaigns across channels.
V3 刚刚推出跨渠道 A/B 测试的自动优化活动。
Awesome.
惊人的。
Yeah.
是的。
And again, just to summarize where we're at, just to give people the advice we've shared so far.
再说一遍，只是为了总结我们的处境，只是为了向人们提供我们迄今为止分享的建议。
One is just important to understand AI products are different.
重要的是要了解人工智能产品的不同之处。
They're non-deterministic.
它们是不确定的。
And you pointed out, and I forgot to actually mirror back this point, both on the input and the output.
你指出，我忘了在输入和输出上实际反映这一点。
The user experience is non-deterministic.People will see different things, different outputs, different chat conversations, different maybe UI if it's designing the UI for you.
用户体验是不确定的。人们会看到不同的东西、不同的输出、不同的聊天对话，如果是为你设计 UI，可能还会有不同的 UI。
And also the output obviously is going to be non-terministic.
而且输出显然将是非确定性的。
So that's a problem and a challenge.
所以这是一个问题和挑战。
And then-
然后——
Aishwarya Naresh Reganti: I mean, if you think of it's also the most beautiful part of AI, which is, I mean, we are all much more comfortable talking than following a bunch of buttons and all of that.
Aishwarya Naresh Reganti：我的意思是，如果你认为这也是人工智能最美丽的部分，我的意思是，我们都比遵循一堆按钮之类的更舒服地说话。
So the bar to using AI products is much lower because you can be as natural as you would be with humans, but that's also the problem, which is there are tons of ways we communicate and you want to make sure that that intent is rightly communicated and the right actions are taken because most of your systems are deterministic and you want to achieve a deterministic outcome, but with non-deterministic technology and that's where it gets a little messy.
因此，使用人工智能产品的门槛要低得多，因为你可以像与人类相处一样自然，但这也是问题所在，我们沟通的方式有很多，你想要确保正确传达意图并采取正确的行动，因为你的大多数系统都是确定性的，你想要实现确定性的结果，但使用非确定性技术，这就是它变得有点混乱的地方。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
Okay.
好的。
I love the optimistic version of why this is good.
我喜欢用乐观的方式来解释为什么这很好。
Okay.
好的。
And then the other piece is this idea of this trade-off of autonomy versus control when you're designing a thing.
然后另一部分是当你设计一个东西时，自主权与控制权的权衡。
And I imagine what you're seeing is people try to jump to the ideal, like the V3 immediately and that's when they get into trouble both.
我想你所看到的是人们试图立即跳到理想的状态，比如 V3，而这就是他们陷入困境的时候。
It's probably a lot harder to build that and it just doesn't work.
构建它可能要困难得多，而且根本行不通。
And then they're just like, "Okay, this is a failure.
然后他们就会说：“好吧，这是一次失败。
What are we even doing?"
我们到底在做什么？”
Kiriti Badam: Exactly.
基里蒂·巴达姆：没错。
I feel there's a bunch of things that you actually have to get confidence in before you get to V3.
我觉得在进入 V3 之前，你实际上必须对很多事情充满信心。
And it's easy to get overwhelmed that, oh, my AI agent is doing these things wrong in a hundred different ways and you're not going to actually tabulate all of them and fix it.
人们很容易不知所措，哦，我的人工智能代理以一百种不同的方式做错了这些事情，而你不会真正将所有这些都制成表格并修复它。
Even though you've learned how do you deal with the evaluation practices and stuff like that, if you're starting on the wrong spot, you are actually going to have a hard time correcting things from there.
即使您已经学会了如何处理评估实践和类似的事情，但如果您从错误的位置开始，实际上您将很难从那里纠正问题。
And when you start small and when you start with building a very minimalistic version with high human control and low agency, it also forces you to think about what is the problem that I'm going to solve.
当你从小处开始，当你开始构建一个具有高人力控制和低代理性的非常简约的版本时，它也会迫使你思考我要解决的问题是什么。
We use this term called problem first.
我们首先使用“问题”这个术语。
And to me, it was obvious in the sense that that I do need to think about the problem, but it's incredible how well it resonates with the people that in all this advancements of the AI that we are seeing, one easy, slippery slope is to just keep thinking about complexities of the solution and forget the problem that you're trying to solve.
对我来说，很明显，我确实需要考虑这个问题，但令人难以置信的是，在我们所看到的人工智能的所有这些进步中，它与人们产生了如此好的共鸣，一个简单的滑坡就是不断思考解决方案的复杂性，而忘记了你正在尝试解决的问题。
Kiriti Badam: So when you're trying to start at a smaller scale of autonomy, you start to really think about what is the problem that I'm trying to solve and how do I break it down into levels of autonomy that I can build later?
Kiriti Badam：因此，当你尝试从较小规模的自治开始时，你会开始真正思考我要解决的问题是什么，以及如何将其分解为以后可以构建的自治级别？
So that is incredibly useful and we keep repeating this part and over and over with everyone we talk to.
所以这非常有用，我们不断地与我们交谈的每个人重复这一部分。
Lenny Rachitsky: And there's so many other benefits to limiting autonomy because there's just danger also of the thing doing too much for you and just messing up your, I don't know, your database, sending out all these emails you never expected.
Lenny Rachitsky：限制自主权还有很多其他好处，因为这也有危险，因为它会为你做太多事情，弄乱你的，我不知道，你的数据库，发送出所有这些你从未预料到的电子邮件。
And there's like so many reasons this is a good idea.
有很多理由认为这是一个好主意。
Aishwarya Naresh Reganti: Yep.
艾西瓦娅·纳雷什·雷甘蒂：是的。
I recently read this paper from a bunch of folks at UC Berkeley.
我最近读到了加州大学伯克利分校一群人写的这篇论文。
Basically Matei Zaharia, and the folks at Databricks and it said about 74% or 75% of the enterprises that they had spoken to, their biggest problem was reliability.
基本上，Matei Zaharia 和 Databricks 的人员表示，在他们交谈过的企业中，大约 74% 或 75% 的企业最大的问题是可靠性。
And that's also why they weren't comfortable deploying products to their end users or building customer facing products because they just weren't sure or they just weren't comfortable doing that and exposing their users to a bunch of these risks.
这也是为什么他们不愿意向最终用户部署产品或构建面向客户的产品，因为他们只是不确定，或者他们只是不愿意这样做并使用户面临一堆这些风险。
And that's also why they think a lot of AI products today have to do with productivity because it's much low autonomy versus end-to-end agents that would replace workflows.
这也是为什么他们认为当今的许多人工智能产品都与生产力有关，因为与取代工作流程的端到端代理相比，它的自主性要低得多。
And yeah, I love their work otherwise as well, but I think that's very in line with what at least we are seeing at my startup as well.
是的，我也喜欢他们的工作，但我认为这与至少我们在我的初创公司所看到的非常一致。
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
Very interesting.
非常有趣。
There's an episode that'll come out before this conversation where we go deep into another problem that this avoids, which is around prompt injection and jailbreaking and just how big of a risk that is for AI products where it's essentially an unsolved and unsolvable problem potentially.
在这次对话之前，我们将深入讨论另一个可以避免的问题，即即时注入和越狱，以及对于人工智能产品来说，风险有多大，因为它本质上是一个尚未解决和无法解决的潜在问题。
I'm not going to go down that track, but that's a pretty scary conversation we had that'll be out before this conversation.
我不会沿着这条路走下去，但这是一次非常可怕的谈话，我们将在这次谈话之前进行。
Aishwarya Naresh Reganti: I think that will be a huge problem once systems go mainstream.
Aishwarya Naresh Reganti：我认为一旦系统成为主流，这将是一个大问题。
We're still so busy building AI products that we're not worried about security, but it will be such a huge problem to kind of, especially with this non-deterministic API again.
我们仍然忙于构建人工智能产品，因此我们并不担心安全性，但这将是一个巨大的问题，尤其是再次使用这种非确定性 API。
So you're kind of stuck because there are tons of instructions that you could inject within your prompt and then it's going really bad.
所以你有点陷入困境，因为你可以在提示符中注入大量指令，然后事情就会变得非常糟糕。
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
Let's actually spend a little time here because it's actually really interesting to me and no one's talking about this stuff, which is like the conversation we had is just it's pretty easy to get AI to do stuff it shouldn't do.
让我们在这里花一点时间，因为这对我来说真的很有趣，而且没有人在谈论这些东西，就像我们的对话一样，让人工智能做它不应该做的事情是很容易的。
And there's all these guardrail systems people put in place, but turns out these guardrails aren't actually very good and you can always get around them.
人们安装了所有这些护栏系统，但事实证明这些护栏实际上并不是很好，你总是可以绕过它们。
And to your point, as agents become more autonomous and robots, it gets pretty scary that you could get AI to do things you shouldn't do.
就你而言，随着代理变得更加自主和机器人，你可以让人工智能做你不应该做的事情，这变得非常可怕。
Kiriti Badam: I think this is definitely a problem, but I feel in the current spectrum of customers adopting AI, the extent to which companies can actually get advantage of AI or improve their processes or streamline the existing processes that they have, I feel it's still in the very early stage.
Kiriti Badam：我认为这绝对是一个问题，但我觉得在目前采用人工智能的客户范围内，公司实际上可以在多大程度上利用人工智能或改进其流程或简化他们现有的流程，我觉得还处于非常早期的阶段。
2025 has been an extremely busy year for AI agents and customers trying to adopt AI, but I feel the penetration is still not as much as you would actually get advantage out of it.
对于试图采用人工智能的人工智能代理和客户来说，2025 年是极其忙碌的一年，但我认为渗透率仍然没有达到你实际从中获得优势的程度。
So with the right sort of human in the loop points in here, I feel we can actually avoid a bunch of these things and focus more towards streamlining the processes.
因此，有了正确的人员参与循环点，我觉得我们实际上可以避免一堆这样的事情，并更多地关注简化流程。
And I am more on the optimist side in the sense that you need to try and adopt this before actually trying to be only for highlighting the negative aspects of what could go wrong.
我更倾向于乐观的一面，因为你需要尝试并采用这一点，然后再真正尝试只是为了强调可能出现问题的负面方面。
Kiriti Badam: So I feel like strongly that companies has this adopt this, they definitely ...
Kiriti Badam：所以我强烈认为公司已经采用了这个，他们肯定......
No company at OpenAI we talk to has never had been the case that, oh, AI cannot help me in this case.
在与我们交谈的 OpenAI 公司中，没有一家公司从未出现过这样的情况：哦，人工智能在这种情况下无法帮助我。
It has always been that, oh, there is this set of things that it can optimize for me and then let me see how I can adopt it.
一直以来，哦，有一套东西它可以为我优化，然后让我看看如何采用它。
Lenny Rachitsky: Sweet.
莱尼·拉奇茨基：太好了。
I always like the optimistic perspective.
我一向喜欢乐观的观点。
I'm excited for you to listen to this and see what you think because it's really interesting.
我很高兴你能听到这个并看看你的想法，因为它真的很有趣。
And to your point, there's a lot of things to focus on.
就您而言，有很多事情需要关注。
It's one of many things to worry about and think about.
这是需要担心和思考的众多事情之一。
Okay, let's get back on track here.
好吧，让我们回到正轨吧。
So we've shared a bunch of pro-tips and important piece of advice.
因此，我们分享了一系列专业提示和重要建议。
Let me ask, what other patterns and kind of ways of working do you see in companies that do this well and teams that build AI products successfully?
我想问一下，在这方面做得好的公司和成功构建人工智能产品的团队中，您还看到了哪些其他模式和工作方式？
And then just what are the most common pitfalls people fall into?
那么人们最常见的陷阱是什么？
So we could just maybe start with, what are other ways that companies do this well, build AI products successfully?
因此，我们可以从以下几个方面开始：公司还有哪些其他方法可以做得很好，成功构建人工智能产品？
Aishwarya Naresh Reganti: I almost think of it as like a success triangle with three dimensions that's never always technical.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：我几乎将其视为具有三个维度的成功三角形，但并不总是技术性的。
Every technology problem is a people problem first.
每个技术问题首先都是人的问题。
And with companies that we have worked with, it's these three dimensions, like great leaders, good culture and technical prowess.
对于我们合作过的公司来说，这就是这三个维度，比如伟大的领导者、良好的文化和技术实力。
With leaders itself, we work with a lot of companies for their AI transformation, training, strategy and stuff like that.
对于领导者本身，我们与许多公司合作进行人工智能转型、培训、战略等。
And I feel like a lot of companies, the leaders have built intuitions over 10 or 15 years and they're kind of highly regarded for those intuitions.
我觉得很多公司的领导者都在 10 或 15 年的时间里建立了直觉，并且他们因这些直觉而受到高度重视。
But now with AI in the picture, those intuitions will have to be relearned and leaders have to be vulnerable to do that.
但现在有了人工智能，这些直觉将不得不重新学习，而领导者必须脆弱才能做到这一点。
I used to work with the CEO of now Rackspace, Gagan.
我曾经和 Rackspace 的首席执行官 Gagan 一起工作过。
So he would have this block every day in the morning, which would say catching up with AI 4:00 to 6:00 AM, and he would not have any meetings or anything like that.
所以他每天早上都会有这个区块，也就是说早上 4:00 到 6:00 赶上 AI，他不会开任何会议或类似的事情。
Aishwarya Naresh Reganti: And that was just his time to pick up on the latest AI podcast or information and all of that.
Aishwarya Naresh Reganti：这正是他了解最新人工智能播客或信息以及所有这些的时间。
And he would have weekend vibe coding sessions and stuff like that.
他会在周末举办编码会议之类的活动。
So I think leaders have to get back to being hands-on.
因此，我认为领导者必须回归亲力亲为。
And that's not because they have to be implementing these things, but more of rebuilding their intuitions because you must be comfortable with the fact that your intuitions might not be right and you probably are the dumbest person in the room and you want to learn from everyone.
这并不是因为他们必须实施这些事情，而是更多地重建他们的直觉，因为你必须接受这样一个事实：你的直觉可能不正确，你可能是房间里最愚蠢的人，你想向每个人学习。
And that I've seen that being a very distinguishing factor of companies that build products which are successful because you're kind of bringing in that top-down approach.
我发现这是制造成功产品的公司的一个非常显着的因素，因为你引入了自上而下的方法。
It's almost always impossible for it to be bottom-up.
自下而上几乎总是不可能的。
You can't have a bunch of engineers go and get buy-in from the leader if they just don't trust in the technology or if they have misaligned expectations about the technology.
如果一群工程师不信任这项技术或者对这项技术抱有错误的期望，那么你就不可能让一群工程师去获得领导者的支持。
Aishwarya Naresh Reganti: I've heard from so many folks who are building that our leaders just don't understand the extent to which AI can solve a particular problem or they just vibe code something and assume it's easy to take it to production and you really need to understand the range of what AI can solve today so that you can guide decisions within the company.
Aishwarya Naresh Reganti：我从很多正在建设的人那里听说，我们的领导者只是不了解人工智能可以在多大程度上解决特定问题，或者他们只是编码一些东西，并认为很容易将其投入生产，而你确实需要了解人工智能今天可以解决的范围，以便你可以指导公司内部的决策。
The second one is the culture itself.
第二个是文化本身。
And again, I work with enterprises where AI is not their main thing and they need to bring in AI into their processes just because a competitor is doing it.
再说一次，我与一些企业合作，人工智能不是他们的主要业务，他们需要将人工智能引入他们的流程，只是因为竞争对手正在这样做。
And just because it does make sense because there are use cases that are very ripe.
仅仅因为它确实有意义，因为有些用例已经非常成熟。
Then along the way, I feel a lot of companies have this culture of FOMO and you will be replaced and those kind of things and people get really afraid.
一路走来，我觉得很多公司都有这种 FOMO 文化，你会被取代，这类事情和人们会变得非常害怕。
Subject matter experts are such a huge part of building AI products that work because you really need to consult them to understand how your AI is behaving or what the ideal behavior should be.
主题专家是构建有效的人工智能产品的重要组成部分，因为您确实需要咨询他们以了解您的人工智能的行为方式或理想的行为应该是什么。
Aishwarya Naresh Reganti: But then I've spoken to a bunch of companies where the subject matter experts just don't want to talk to you because they think their job is being replaced.
艾西瓦娅·纳雷什·雷甘蒂（Aishwarya Naresh Reganti）：但是我和很多公司谈过，这些公司的主题专家只是不想和你说话，因为他们认为他们的工作正在被取代。
So I mean, again, this comes from the leader itself.
所以我的意思是，这又来自于领导者本身。
You want to build a culture of empowerment, of augmenting AI into your own workflows so that you can 10X at what you're doing instead of saying that probably you'll be replaced if you don't adopt AI and stuff like that.
你想要建立一种赋权文化，将人工智能增强到你自己的工作流程中，这样你就可以把你正在做的事情提高十倍，而不是说如果你不采用人工智能之类的东西，你可能会被取代。
So that kind of an empowering culture always helps.
因此，这种赋权文化总是有帮助的。
You want to make your entire organization be in it together and make AI work for you instead of trying to guard their own jobs, et cetera.
你想让你的整个组织一起参与其中，让人工智能为你服务，而不是试图保住自己的工作等等。
And with AI, it's also true that it opens up a lot more opportunities than before.
人工智能也确实带来了比以前更多的机会。
So you could have your employees doing a lot more things than before and 10x their productivity.
因此，您可以让您的员工做比以前更多的事情，并将他们的生产力提高 10 倍。
And the third one is the technical part which we talk about.
第三个是我们谈论的技术部分。
Aishwarya Naresh Reganti: I think folks that are successful are incredibly obsessed about understanding their workflows very well and augmenting parts that could be ripe for AI versus the ones that might need human in the loop somewhere, et cetera.
Aishwarya Naresh Reganti：我认为成功的人非常痴迷于很好地理解他们的工作流程，并增强人工智能可能成熟的部分，而不是那些可能需要人类参与循环的部分，等等。
Whenever you're trying to automate some part of a workflow, it's never the case that you could use an AI agent and that will solve your problems.
每当您尝试自动化工作流程的某些部分时，您都无法使用人工智能代理来解决您的问题。
It's always, you probably have a machine learning model that's going to do some part of the job.
总是，你可能有一个机器学习模型来完成部分工作。
You have deterministic code doing some part of the job.
您有确定性代码来完成部分工作。
So you really need to be obsessed with understanding that workflow so you can choose the right tool for the problem instead of being obsessed with the technology itself.
因此，您确实需要着迷于理解该工作流程，以便为问题选择正确的工具，而不是痴迷于技术本身。
And another pattern I see is also folks really understand this idea of working with a non-deterministic API, which is your LLM.
我看到的另一种模式是人们真正理解使用非确定性 API 的想法，这就是法学硕士。
And what that means is they also understand the AI development lifecycle looks very different and they iterate pretty quickly, which is can I build something iterate quickly in a way that it doesn't ruin my customer experience at the same time gives me enough amount of data so that I can estimate behavior.
这意味着他们也了解人工智能开发生命周期看起来非常不同，并且迭代速度非常快，也就是说我可以快速构建一些东西，不会破坏我的客户体验，同时为我提供足够的数据，以便我可以估计行为。
Aishwarya Naresh Reganti: So they build that flywheel very quickly.
Aishwarya Naresh Reganti：所以他们很快就制造出了飞轮。
As of today, it's not about being the first company to have an agent among your competitors.
截至今天，这并不是要成为竞争对手中第一家拥有代理商的公司。
It's about, have you built the right flywheels in place so that you can improve over time?
问题在于，您是否已构建了合适的飞轮以便您可以随着时间的推移不断改进？
When someone comes up to me and says, "We have this one click agent, it's going to be deployed in your system." And then in two or three days, it'll start showing you significant gains.
当有人走过来对我说：“我们有这个一键代理，它将部署在您的系统中。”然后两三天后，你就会开始看到显着的收获。
I would almost be skeptical because it's just not possible.
我几乎会怀疑，因为这是不可能的。
And that's not because the models aren't there, but because enterprise data and infrastructure is very messy and you need a bit to ...
这并不是因为模型不存在，而是因为企业数据和基础设施非常混乱，您需要一些...
Even the agent needs a bit to understand how these systems work.
即使是代理也需要了解这些系统的工作原理。
There are very messy taxonomies everywhere.
到处都有非常混乱的分类法。
People tend to do things like get customer data, we want, get customer data, we do, and these kind of things.
人们倾向于做一些事情，比如获取客户数据，我们想要，获取客户数据，我们做，等等。
And all those functions exist and they're being called and basically there's a lot of tech debt that you need to deal with.
所有这些功能都存在并且正在被调用，基本上您需要处理大量的技术债务。
Aishwarya Naresh Reganti: So most of the times, if you're obsessed with the problem itself and you understand your workflows very well, you will know how to improve your agents over time instead of just slapping an agent and assuming that it'll work from day one.
Aishwarya Naresh Reganti：所以大多数时候，如果您对问题本身着迷并且非常了解自己的工作流程，您就会知道如何随着时间的推移改进您的代理，而不是仅仅打压代理并假设它从第一天起就会起作用。
I probably will go as far to say that if someone's selling you one click-agents, it's pure marketing.
我可能会说，如果有人向您出售一种点击代理，那纯粹是营销。
You don't want to buy into that.
你不想相信这一点。
I would rather go with a company that says, "We're going to build this pipeline for you," and that will learn over time and build a flywheel to improve than something that's going to work out of the box.
我宁愿选择一家说“我们将为您构建这条管道”的公司，并且会随着时间的推移学习并构建一个飞轮来改进，而不是开箱即用的东西。
To replace any critical workflow or to build something that can give you significant ROI, it easily takes four to six months of work, even if you have the best data layer and infrastructure layer.
要替换任何关键工作流程或构建可以为您带来显着投资回报率的东西，即使您拥有最好的数据层和基础设施层，也很容易需要四到六个月的工作时间。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
There's a lot there that resonates so deeply with other conversations I've been having on this podcast.
其中有很多内容与我在这个播客上进行的其他对话产生了深刻的共鸣。
One is just for a company to be successful at seeing a lot of impact from AI, the founder-CEO has to be deep into it.
一是公司要想成功地看到人工智能的巨大影响，创始人兼首席执行官必须深入研究它。
I had Dan Shipper on the podcast and they work with a bunch of companies helping them adopt AI.
我的播客中有 Dan Shipper，他们与许多公司合作，帮助他们采用人工智能。
And he said that's the number one predictor of success.
他说这是成功的首要预测因素。
Is the CEO chatting with ChatGPT, Claude, whatever, many times a day.
CEO 是否每天多次与 ChatGPT、Claude 等人聊天？
I love this example you gave with the Rackspace CEO has catch up on AI news in the morning every day.
我喜欢你举的 Rackspace 首席执行官每天早上都会关注人工智能新闻的例子。
I was imagining he'd be chatting with the chatbot versus reading news.
我想象他会与聊天机器人聊天而不是阅读新闻。
Aishwarya Naresh Reganti: With the kind of information you have as of today, you could just ...
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：根据您今天掌握的信息，您可以......
I mean, you want to choose the right channels as well because everybody has an opinion.
我的意思是，你也想选择正确的渠道，因为每个人都有自己的意见。
So whose opinion do you want to bank on?
那么你想相信谁的意见呢？
I feel like having that good quality set of people that you're listening to really makes sense.
我觉得拥有一群高素质的人来倾听你的意见确实很有意义。
So he just has a list of two or three sources that he always looks at.
所以他只有一个他经常查看的两三个来源的清单。
And then he comes back with a bunch of questions and bounces it around with a bunch of AI experts to see what they think about it.
然后他带着一堆问题回来，并与一群人工智能专家讨论，看看他们对此有何看法。
And I was part of that group, so I kind of know-
我是那个团体的一员，所以我知道——
Aishwarya Naresh Reganti: ...
艾西瓦娅·纳雷什·雷甘蒂：...
about the questions that he comes up with.
关于他提出的问题。
Aishwarya Naresh Reganti: It's pretty cool.
艾西瓦娅·纳雷什·雷甘蒂：这很酷。
I was like, "Why are you doing so much?" And then he says, "It trickles down into a bunch of decisions that we take."
我当时想，“你为什么要做这么多？”然后他说，“它会渗透到我们做出的一系列决定中。”
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
Let me talk about another topic that's very ...
让我谈谈另一个非常......
It's been a hot topic on this podcast.
这是这个播客的热门话题。
It was a hot topic on Twitter for a while, evals.
评估人员称，这曾一度成为推特上的热门话题。
A lot of people are obsessed with evals, think they're the solution to a lot of problems in AI.
很多人痴迷于评估，认为它们是人工智能中许多问题的解决方案。
A lot of people think they're overrated that you don't need evals.
很多人认为他们被高估了，不需要评估。
You can just feel the vibes and you'll be all right.
你只要感受一下这种氛围就可以了。
What's your take on evals?
您对评估有何看法？
How far does that take people in solving a lot of the problems that you talk about?
人们在解决你谈到的许多问题方面能走多远？
Kiriti Badam: In terms of what is going on in the community, I feel there's just this false dichotomy of this either evals is going to solve everything or online monitoring or production monitoring is going to solve everything.
Kiriti Badam：就社区中正在发生的事情而言，我觉得存在这种错误的二分法：要么评估将解决所有问题，要么在线监控或生产监控将解决所有问题。
And I find no reason to trust one of the extremes in the sense that I will entirely bank my application on this or like that to solve the thing.
我发现没有理由相信极端之一，因为我将完全依靠我的应用程序来解决这个问题。
So if you take a step back, think of what are evals.
因此，如果您退后一步，请想想什么是评估。
Evals are basically your trusted product thinking or your knowledge about the product that is going into this set of data sets that you're going to build in the sense that this is what matters to me.
评估基本上是您可信的产品思维或您对将要构建的这组数据集的产品的了解，因为这对我来说很重要。
This is the kind of problems that my agent should not do and let me build a list of datasets so that I'm going to do well on those.
这是我的代理不应该做的问题，让我构建一个数据集列表，以便我能够在这些问题上做得很好。
And in terms of production monitoring, what you're doing there is you're deploying your application and then you're having some sort of key metrics that actually communicate back to you on how customers are using your product.
在生产监控方面，您所做的就是部署应用程序，然后您将获得某种关键指标，这些指标实际上可以向您传达有关客户如何使用您的产品的信息。
Kiriti Badam: You could be deploying any agent and if the customer is giving a thumbs up for your interaction, you better want to know that.
Kiriti Badam：您可以部署任何代理，如果客户对您的交互表示赞赏，您最好想知道这一点。
So that is what production monitoring is going to do.
这就是生产监控要做的事情。
And this production monitoring has existed for products for a long time, just that now with the AI agents, you need to be monitoring a lot more granularity.
产品的这种生产监控已经存在很长时间了，只是现在有了人工智能代理，你需要监控更细的粒度。
It's not just the customer always giving you explicit feedback, but there is many implicit feedback that you can get.
不仅仅是客户总是给你明确的反馈，你还可以获得许多隐性的反馈。
For example, in ChatGPT, if you are liking the answer, you can actually give a thumbs up.
例如，在 ChatGPT 中，如果您喜欢答案，实际上可以竖起大拇指。
Or if you don't like the answer, sometimes customers don't give you thumbs down, but actually regenerate the answer.
或者，如果您不喜欢这个答案，有时客户不会对您表示反对，而是实际上会重新生成答案。
So that is a clear indication that the initial answer that regenerator is not meeting the customer's expectation.
因此，这清楚地表明最初的答案是再生器没有满足客户的期望。
So these are the kind of implicit signals you always need to think about.
因此，这些是您始终需要考虑的隐含信号。
Kiriti Badam: And that spectrum has been increasing in terms of production monitoring.
Kiriti Badam：在生产监控方面，这一范围一直在增加。
Now let's come back to the initial topic of like, okay, is it evals or is it production monitoring?
现在让我们回到最初的话题，好吧，是评估还是生产监控？
What does it matter?
这有什么关系呢？
So I feel, again, we go back to this problem first approach of what is it that you're trying to build.
所以我觉得，我们再次回到这个问题的第一个方法，即你正在尝试构建什么。
You're trying to build a reliable application for your customers that's not going to do a bad thing.
您正在尝试为您的客户构建一个可靠的应用程序，而这不会带来坏事。
It's always going to do the right thing.
它总是会做正确的事。
Or if it is doing a wrong thing, you're basically alerted very quickly.
或者，如果它做了错误的事情，您基本上很快就会收到警报。
So I break this down into two parts.
所以我把它分成两部分。
One is nobody goes into deploying an application without actually just testing that.
一是没有人在不实际测试的情况下就部署应用程序。
This testing could be wipes or this testing could be, "Okay, I have this 10 questions that it should not go wrong no matter what changes I make, and let me build this and let's call this an evaluation dataset." Now, let's say you build this, you deployed this, and then you figured, "Okay, now I need to understand whether it's doing the right thing or not."
这个测试可以是擦除，或者这个测试可以是，“好吧，我有这 10 个问题，无论我做出什么改变，它都不会出错，让我构建这个，我们称之为评估数据集。”现在，假设您构建了这个，部署了这个，然后您想，“好吧，现在我需要了解它是否在做正确的事情。”
Kiriti Badam: So if you're a high throughput or a high transaction customer, you cannot practically sit and evaluate all the traces.
Kiriti Badam：因此，如果您是高吞吐量或高交易客户，您实际上无法坐下来评估所有跟踪。
You need some indication to understand what are the things that I should look at.
您需要一些指示来了解我应该关注哪些内容。
And this is where production monitoring comes into the picture that you cannot predict the base in which your agent could be doing wrong, but all of these other implicit signals and explicit signals, those are going to communicate back to you what are the traces that you need to look at.
这就是生产监控出现的地方，你无法预测你的代理可能做错的基础，但所有这些其他隐式信号和显式信号，这些信号都会向你传达你需要查看的痕迹。
And that is where production monitoring helps.
这就是生产监控的用武之地。
And once you get this kind of traces, you need to examine what are the failure patterns that you're seeing in these different types of interactions.
一旦获得这种跟踪，您就需要检查在这些不同类型的交互中看到的失败模式是什么。
And is there something that I really care about that should not happen?
有什么我真正关心但不应该发生的事情吗？
And if that kind of failure modes are happening, then I need to think about building an evaluation dataset for it.
如果发生这种故障模式，那么我需要考虑为其构建一个评估数据集。
Kiriti Badam: And okay, let's say I built an evaluation dataset for my agent trying to offer refunds where explicitly I have configured it not to.
Kiriti Badam：好吧，假设我为我的代理构建了一个评估数据集，试图提供退款，但我已明确配置为不提供退款。
So I built this evaluation dataset and then I made my changes in tools or prompts or whatever, and then I deployed the second version of the product.
因此，我构建了这个评估数据集，然后对工具或提示等进行了更改，然后部署了产品的第二个版本。
Now there is no guarantee that this is the only problem that you're going to see.
现在不能保证这是您将看到的唯一问题。
You still need production monitoring to actually catch different kinds of problems that you might encounter.
您仍然需要生产监控来实际捕获可能遇到的不同类型的问题。
So I feel evals are important, production monitoring is important, but this notion of only one of them is going to solve things for you that is completely dismissible in my opinion.
所以我觉得评估很重要，生产监控也很重要，但只有其中一个的概念才能为你解决问题，而我认为这是完全可以忽略的。
Lenny Rachitsky: All right.
莱尼·拉奇茨基：好的。
A very reasonable answer.
一个非常合理的答案。
And the point here isn't, it's not just as simple as do both.
这里的重点不是，它不仅仅是两者那么简单。
It's more that there are different things to catch and one approach won't catch all the things you need to be paying attention to.
更重要的是，有不同的事情需要捕捉，而一种方法无法捕捉到您需要注意的所有事情。
Aishwarya Naresh Reganti: I want to take two steps back and kind of talk about how much weight the term evals has had to take in the second half of 2025 because you go meet a data labeling company and they tell you our experts are writing evals and then you have all of these folks saying that PMs should be writing evals, they're the new PRDs.
Aishwarya Naresh Reganti：我想退后两步，谈谈“评估”一词在 2025 年下半年的重要性，因为你去见一家数据标签公司，他们告诉你我们的专家正在编写评估，然后所有这些人都说 PM 应该编写评估，他们是新的 PRD。
And then you have folks saying that evals is pretty much everything, which is the feedback loop you're supposed to be building to improve your products.
然后有人说评估几乎就是一切，这是你应该构建的反馈循环来改进你的产品。
Now, step back as a beginner and kind of think what are evals?
现在，作为一个初学者退后一步，想想什么是评估？
Why is everyone saying evals?
为什么大家都说评估？
And these are actually different parts of the process and nobody is wrong in the sense that yes, these are evals, but when a data labeling company is telling you that our experts are writing evals, they're actually referring to error analysis or experts just leading notes on what should be right.
这些实际上是过程的不同部分，没有人是错的，是的，这些是评估，但是当数据标签公司告诉您我们的专家正在编写评估时，他们实际上指的是错误分析或专家只是对应该正确的内容进行注释。
Aishwarya Naresh Reganti: Lawyers and doctors write evals, that doesn't mean they're building LLM judges or they're building this entire feedback loop.
Aishwarya Naresh Reganti：律师和医生撰写评估，这并不意味着他们正在建立法学硕士法官或他们正在建立整个反馈循环。
And when you say that a PM should be writing evals, doesn't mean they have to write an LLM judge that's good enough for production.
当你说 PM 应该编写评估时，并不意味着他们必须编写足以用于生产的 LLM 判断。
I think there are also very prescriptive ways of doing this and plus one to KD, which is you cannot predict upfront if you need to be building an LLM judge versus you need to be using implicit signals from production monitoring, et cetera.
我认为也有非常规范的方法可以做到这一点，再加上 KD，如果你需要建立一个 LLM 法官，而不是需要使用来自生产监控等的隐式信号，你无法预先预测。
I think Martin Fowler at some point had this term called semantic diffusion back in the 2000s, which kind of means that someone comes up with a term, everybody starts butchering it with their own definitions and then you kind of lose the actual definition of it.
我认为马丁·福勒（Martin Fowler）在 2000 年代的某个时候使用了这个术语，称为语义扩散，这意味着有人提出了一个术语，每个人都开始用自己的定义来屠宰它，然后你就失去了它的实际定义。
That is what is happening to evals or agents or any word in AI as of today, everybody kind of sees a different side to it, I guess.
这就是今天的评估、代理或人工智能中的任何词所发生的情况，我想每个人都会看到它不同的一面。
Aishwarya Naresh Reganti: But if you make a bunch of practitioners sit together and ask them, "Is it important to build an actionable feedback loop for AI products?" I think all of them will agree.
Aishwarya Naresh Reganti：但是如果你让一群从业者坐在一起问他们，“为人工智能产品建立一个可操作的反馈循环重要吗？”我想他们都会同意。
Now, how you do that really depends on your application itself.
现在，如何做到这一点实际上取决于您的应用程序本身。
When you go to complex use cases, it's incredibly hard to build LLM judges because you see a lot of emerging patterns.
当您处理复杂的用例时，构建 LLM 法官非常困难，因为您会看到很多新兴模式。
If you built a judge that would test for verbosity or something like that, it turns out that you're seeing newer patterns that your LLM judge is not able to catch, and then you just end up building too many evals.
如果你建立了一个可以测试冗长或类似内容的法官，那么事实证明你会看到你的 LLM 法官无法捕捉到的新模式，然后你最终会建立太多的评估。
And at that point, it just makes sense to look at your user signals, fix them, check if you have regressed and move on instead of actually building these judges.
到那时，有意义的是查看您的用户信号，修复它们，检查您是否已经退步并继续前进，而不是实际构建这些法官。
So it all depends.
所以这一切都取决于。
I think one statement that every ML practitioner will tell you is it really depends on the context.
我认为每个机器学习从业者都会告诉你的一句话是，这确实取决于上下文。
Don't be obsessed with prescriptions they're going to change.
不要沉迷于他们将要改变的处方。
Lenny Rachitsky: That's such an important point, this idea that, especially that evals just means many things to different people now.
Lenny Rachitsky：这是非常重要的一点，尤其是评估现在对不同的人来说意味着很多事情。
It's just a term for so many things.
它只是很多事物的一个术语。
And it's complicated to just talk about evals when you see it as the stuff data labeling companies are giving you and things PMR, right?
当您将评估视为数据标签公司为您提供的东西以及 PMR 的东西时，仅仅谈论评估就很复杂，对吗？
And there's also benchmarks.
还有基准。
People call benchmarks a little bit evals.
人们将基准测试称为评估。
It's like-
这就像——
Aishwarya Naresh Reganti: I recently spoke to a client who told me, "We do evals." And I was like, "Okay, can you show me your dataset?" And said, "No, we just checked LM Arena and Artificial Analysis.
Aishwarya Naresh Reganti：我最近与一位客户交谈，他告诉我，“我们进行评估。”我当时想，“好吧，你能给我看看你的数据集吗？”并说道：“不，我们刚刚检查了 LM Arena 和人工分析。
These are independent benchmarks and we know that this model is the right one for our use case." And I'm like, "You're not doing evals.
这些是独立的基准，我们知道这个模型适合我们的用例。”我说，“你没有进行评估。
That's not evals.
这不是评估。
Those are model evals."
这些是模型评估。”
Lenny Rachitsky: But it makes sense.
Lenny Rachitsky：但这有道理。
The word, it could be used in that context.
这个词，可以在这种情况下使用。
I get why people think that, but yeah, now it's just confusing it even more.
我明白为什么人们会这么想，但是，是的，现在它只是更加令人困惑。
Lenny Rachitsky: Just one more line of questioning here that I think that's on my mind is the reason this became kind of a big debate is Cloud Code.
Lenny Rachitsky：我认为这里还有一个问题，这就是云代码成为一场大辩论的原因。
The head of Cloud Code, Boris, was like, "Nah, we don't do evals on Cloud Code.
Cloud Code 的负责人 Boris 说：“不，我们不对 Cloud Code 进行评估。
It's all vibes." What can you share, Kiriti, on Kodex and the Kodex team, how you approach evals?
这都是共鸣。” Kiriti，您可以分享一下 Kodex 和 Kodex 团队的哪些内容，以及您如何进行评估？
Kiriti Badam: So Kodex, we have this balanced approach of you need to have evals and you need to definitely listen to your customers.
Kiriti Badam：Kodex，我们有这种平衡的方法，你需要进行评估，并且你需要绝对倾听客户的意见。
And I think Alex has been on your podcast recently and he's been talking about how you're extremely focused on building the right product.
我认为亚历克斯最近一直在你的播客上，他一直在谈论你如何非常专注于构建正确的产品。
And a big part of it is basically listening to your customers.
其中很大一部分基本上是倾听客户的意见。
And coding agents are extremely unique compared to agents for other domains in the sense that these are actually built for customizability and these are built for engineers.
与其他领域的代理相比，编码代理非常独特，因为它们实际上是为了可定制性而构建的，并且是为工程师构建的。
So coding agent is not a product which is going to solve these top five workflows or top six workflows or whatever.
因此，编码代理并不是一个能够解决前五个工作流程或前六个工作流程或其他任何问题的产品。
It's meant to be customizable in multi different ways.
它可以通过多种不同的方式进行定制。
And the implication of that is that your product is going to be used in different integrations and different kinds of tools and different kinds of things.
这意味着您的产品将用于不同的集成、不同类型的工具和不同类型的事物。
So it gets really hard to build an evaluation dataset for all kinds of interactions that your customers are going to use your product for.
因此，为客户将使用您的产品进行的各种交互构建评估数据集非常困难。
Kiriti Badam: With that said, you also need to understand that, okay, if I'm going to make a change, it's at least not going to damage something that is really core to the product.
Kiriti Badam：话虽如此，你还需要明白，好吧，如果我要做出改变，至少不会损坏产品真正核心的东西。
So we have evaluations for doing that, butt the same time we take extreme care on understanding how the customers are using it.
因此，我们对此进行了评估，但同时我们非常谨慎地了解客户如何使用它。
For example, we built this code review product recently and it has been gaining extreme amount of traction.
例如，我们最近构建了这个代码审查产品，它已经获得了极大的关注。
And I feel like many, many bugs in OpenAI as well as even our external customers are getting caught with this.
我觉得 OpenAI 以及我们的外部客户都遇到了很多很多错误。
And now let's say if I'm making a model change to the code review or a different kinds of RL mechanism that I trained with it, and now if I'm going to deploy it, I definitely do want A/B test and identify whether it's actually finding the right mistakes and how are users reacting to it?
现在假设我要对代码审查或我用它训练的不同类型的 RL 机制进行模型更改，现在如果我要部署它，我肯定需要 A/B 测试并确定它是否真正找到了正确的错误以及用户对此有何反应？
And sometimes if users do get annoyed by your incorrect code regis, they go to the extent of just switching off the product.
有时，如果用户确实对不正确的代码注册感到恼火，他们会直接关闭产品。
Kiriti Badam: So those are the signals that you want to look at and make sure that your new changes are doing the right thing.
Kiriti Badam：这些是您想要查看的信号，并确保您的新更改正在做正确的事情。
And it's extremely hard for us to think of these kind of scenarios beforehand and develop evaluation data sets for it.
我们很难事先想到这些场景并为其开发评估数据集。
So I feel like there's a bit of both.
所以我觉得两者都有一点。
There's a lot of vibes and there's a lot of customer feedback and we are super active on the social media to understand if anybody's having certain types of problems and quickly fix that.
有很多共鸣，有很多客户反馈，我们在社交媒体上非常活跃，以了解是否有人遇到某些类型的问题并快速解决。
So I feel it's a ...
所以我觉得这是一个...
How do I put this?
我该怎么说呢？
It's like a domain of things that you do here.
这就像你在这里所做的事情的一个领域。
Lenny Rachitsky: That makes so much sense.
Lenny Rachitsky：这很有道理。
Okay.
好的。
What I'm hearing, Codex, pro evals, but it's not enough.
我所听到的，法典，专业评估，但这还不够。
Lenny Rachitsky: But also just watch customer behavior and feedback.
Lenny Rachitsky：但也要注意客户的行为和反馈。
And also there's some vibes just like, is this feeling good?
还有一些共鸣，比如，这种感觉好吗？
As I'm using it, generating great code that I'm excited about that I think is great.
当我使用它时，生成了很棒的代码，我对此感到很兴奋，我认为这很棒。
Kiriti Badam: I don't think if anybody's coming and seeing that I have this concrete set of evals that I can bet my life on and then I don't need to think about anything else, it's not going to work.
Kiriti Badam：我不认为如果有人来看到我有一套具体的评估，我可以赌上我的生命，然后我不需要考虑任何其他事情，这是行不通的。
And every new model that you're going to launch, we get together as a team and test different things.
对于您要推出的每一个新型号，我们都会作为一个团队聚集在一起并测试不同的东西。
Each person is concentrating on something else.
每个人都专注于其他事情。
And we have this list of hard problems that we have and we throw that to the model and see how well they're progressing.
我们列出了我们遇到的难题，然后将其扔给模型，看看它们的进展情况如何。
So it's like custom evals for each engineer, you would say, and just understand what the product is doing in its new model.
因此，您可能会说，这就像为每个工程师进行自定义评估，只需了解产品在新模型中的作用即可。
Lenny Rachitsky: If you're a founder, the hardest part of starting a company isn't having the idea, it's scaling the business without getting buried in back office work.
Lenny Rachitsky：如果你是一名创始人，创办公司最困难的部分不是有想法，而是在不埋头于后台工作的情况下扩大业务规模。
That's where Brex comes in.
这就是布雷克斯发挥作用的地方。
Brex is the intelligent finance platform for founders.
Brex是为创始人打造的智能金融平台。
With Brex, you get high limit corporate cards, easy banking, high yield treasury, plus a team of AI agents that handle manual finance tasks for you.
通过 Brex，您可以获得高限额的公司卡、便捷的银行业务、高收益的财务，以及为您处理手动财务任务的人工智能代理团队。
They'll do all the stuff that you don't want to do, like file your expenses, scour transactions for waste, and run reports all according to your rules.
他们会做所有您不想做的事情，例如归档您的费用、清理交易中的浪费以及根据您的规则运行报告。
With Brex's AI agents, you can move faster while staying in full control.
借助 Brex 的 AI 代理，您可以更快地行动，同时保持完全控制。
One in three startups in the United States already runs on Brex.
美国三分之一的初创公司已经在 Brex 上运行。
You can too at brex.com.
您也可以访问 brex.com。
Lenny Rachitsky: We've been talking for almost an hour already, and we haven't even covered your extremely powerful software development workflow for building AI products that you two developed that you teach in your course, that you basically combined all the stuff we've been talking about into a step-by-step approach to building AI products.
Lenny Rachitsky：我们已经讨论了近一个小时，我们甚至还没有介绍你们两人在课程中教授的用于构建人工智能产品的极其强大的软件开发工作流程，你们基本上将我们讨论的所有内容结合到了构建人工智能产品的逐步方法中。
You call it the continuous calibration, continuous development framework.
你称之为持续校准、持续开发框架。
Let's pull up a visual to show people what the heck we're talking about, and then just walk us through what this is, how this works, how teams can shift the way they build their AI products to this approach to help them avoid a lot of pain and suffering.
让我们拉出一个视觉效果来向人们展示我们到底在谈论什么，然后向我们介绍这是什么、它是如何工作的、团队如何将他们构建人工智能产品的方式转变为这种方法，以帮助他们避免很多痛苦和磨难。
Aishwarya Naresh Reganti: Before we go about explaining the life cycle, a quick story on why Kiriti and I came up with this is because there are tons of companies that we keep talking to that have the pressure from their competitors because they're all building agents.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：在我们开始解释生命周期之前，先简单介绍一下 Kiriti 和我提出这个想法的原因，因为我们一直在与之交谈的许多公司都承受着来自竞争对手的压力，因为他们都是建筑代理商。
We should be building agents that are entirely autonomous.
我们应该建立完全自主的代理。
And I did end up working with a few customers where we built these end-to-end agents.
我最终确实与一些客户合作，我们构建了这些端到端代理。
And turns out that because you start off at a place where you don't know how the user might interact with your system and what kind of responses or actions the AI might come up with, it's really hard to fix problems when you have this really huge workflow, which is taking four or five steps, making tons of decisions.
事实证明，因为一开始你不知道用户如何与你的系统交互，也不知道人工智能可能会提出什么样的响应或操作，所以当你有如此庞大的工作流程时，要解决问题真的很难，需要四到五个步骤，做出大量的决定。
You just end up debugging so much and then kind of hot fixing to the point where at a time we were building for a customer support use case, which is the example that we give in the newsletter as well.
您最终会进行如此多的调试，然后进行热修复，直到我们为客户支持用例构建的时候，这也是我们在时事通讯中给出的示例。
Aishwarya Naresh Reganti: And we had to shut down the product because we were doing so many hot fixes and there was no way we could count all the emerging problems that were coming up.
Aishwarya Naresh Reganti：我们不得不关闭该产品，因为我们做了太多的热修复，而且我们无法统计所有即将出现的问题。
And there's also quite some news online.
而且网上也有不少新闻。
Recently, I think Air Canada had this thing where one of their agents predicted or hallucinated a policy for a refund, which was not part of their original playbook, and they had to go by it because legal stuff.
最近，我认为加拿大航空公司发生了这样的事情，他们的一名代理人预测或幻觉了退款政策，这不是他们最初的剧本的一部分，他们不得不遵循它，因为法律问题。
And there have been a ton of really scary incidents.
并且已经发生了大量非常可怕的事件。
And that's where the idea comes from.
这就是这个想法的由来。
How can you build so that you don't lose customer trust and you don't end up, or your agent or AI system doesn't end up making decisions that are super dangerous to the company itself.
您如何构建才能不失去客户信任，并且您的代理或人工智能系统最终不会做出对公司本身超级危险的决策。
At the same time, build a flywheel so that you can improve your product as you go.
同时，构建一个飞轮，以便您可以随时改进您的产品。
And that's where we came up with this idea of continuous calibration, continuous development.
这就是我们提出持续校准、持续开发的想法的地方。
Aishwarya Naresh Reganti: The idea is pretty simple, which is we have this right side of the loop, which is continuous development, where you scope capability and curate data, essentially get a data set of what your expected inputs are and what your expected outputs should be looking at.
Aishwarya Naresh Reganti：这个想法非常简单，我们有循环的右侧，即持续开发，您可以在其中范围化能力并管理数据，本质上是获得一个数据集，其中包含您的预期输入是什么以及您的预期输出应该是什么。
This is a very good exercise before you start building any AI product because many times you figure out that a lot of the folks within the team are just not aligned on how the product should behave.
在开始构建任何人工智能产品之前，这是一个非常好的练习，因为很多时候你会发现团队中的很多人对于产品的行为方式并不一致。
And that's where your PMs can really give in a lot more information and your subject matter experts as well.
这就是您的产品经理和主题专家真正可以提供更多信息的地方。
So you have this data set that you know your AI product should be doing really well on.
所以你知道你的人工智能产品应该在这个数据集上表现得很好。
It's not comprehensive, but it lets you get started.
它并不全面，但可以让您入门。
And then you set up the application and then design the right kind of evaluation metrics.
然后您设置应用程序并设计正确的评估指标。
And I intentionally use the term evaluation metrics, although we say evals because I just want to be very specific in what it is because evaluation is a process, evaluation metrics are dimensions that you want to focus on during the process.
我故意使用评估指标这个术语，尽管我们说评估是因为我只是想非常具体地说明它是什么，因为评估是一个过程，评估指标是您在该过程中想要关注的维度。
Aishwarya Naresh Reganti: And then you go about deploying, run your evaluation metrics.
Aishwarya Naresh Reganti：然后开始部署、运行评估指标。
And the second part is the continuous calibration, which is the part where you understand what behavior you hadn't expected in the beginning, right?
第二部分是持续校准，这是您了解一开始没有预料到的行为的部分，对吧？
Because when you start the development process, you have this data set that you're optimizing for, but more often than not, you realize that that data set is not comprehensive enough because users start behaving with your systems in ways that you did not predict.
因为当您开始开发过程时，您拥有要优化的数据集，但通常情况下，您会意识到该数据集不够全面，因为用户开始以您未预测到的方式使用您的系统。
And that's where you want to do the calibration piece.
这就是您想要进行校准的地方。
I've deployed my system.
我已经部署了我的系统。
Now I see that there are patterns that I did not really expect and your evaluation metrics should give you some insight into those patterns, but sometimes you figure out that those metrics were also not enough and you probably have new error patterns that you have not thought about.
现在我发现有些模式是我没有真正期望的，并且您的评估指标应该让您对这些模式有一些了解，但有时您会发现这些指标还不够，并且您可能有您没有考虑过的新错误模式。
And that's where you analyze your behavior, spot error patterns.
这就是您分析自己的行为、发现错误模式的地方。
Aishwarya Naresh Reganti: You apply fixes for issues that you see, but you also design newer evaluation metrics to figure out that they are emerging patterns.
Aishwarya Naresh Reganti：您可以对看到的问题进行修复，但您也可以设计更新的评估指标来找出它们正在出现的模式。
And that doesn't mean you should always design evaluation metrics.
这并不意味着您应该始终设计评估指标。
There are some errors that you can just fix and not really come back to because they're very spot errors.
有些错误你可以直接修复，而不必真正返回，因为它们是非常斑点的错误。
For instance, there's a tool calling error just because your tool wasn't defined well and stuff like that.
例如，有一个工具调用错误只是因为您的工具没有很好地定义等等。
You can just fix it and move on.
您可以修复它并继续。
And this is pretty much how an AI product lifecycle would look like.
这几乎就是人工智能产品生命周期的样子。
But what we specifically also mention is while you're going through these iterations, try to think of lower agency iterations in the beginning and higher control iterations.
但我们还特别提到的是，当您经历这些迭代时，请尝试在开始时考虑较低的代理迭代和较高的控制迭代。
What that means is constrain the number of decisions your AI systems can make and make sure that they're humans in the loop and then increase that over time because you're kind of building a flywheel of behavior and you're understanding what kind of use cases are coming in or how your users are using the system.
这意味着限制你的人工智能系统可以做出的决策数量，并确保它们是人类在循环中，然后随着时间的推移而增加，因为你正在构建一个行为飞轮，并且你正在了解即将出现的用例类型或用户如何使用系统。
Aishwarya Naresh Reganti: And one example I think we give in the newsletter itself is the customer support.
艾西瓦娅·纳雷什·雷甘蒂：我认为我们在时事通讯中给出的一个例子就是客户支持。
This is a nice image that kind of shows how you can think of agency and control as two dimensions.
这是一个很好的图像，它展示了如何将代理和控制视为两个维度。
And each of your versions keep on increasing the agency or the ability of your AI system to make decisions and lower the control as you go.
你的每个版本都在不断增强人工智能系统的决策能力或能力，并在你前进的过程中降低控制力。
And one example that we give is that of the customer support agent, where you can break it down into three versions.
我们给出的一个例子是客户支持代理，您可以将其分为三个版本。
The first version is just routing, which is your agent able to classify and route a particular ticket to the right department?
第一个版本只是路由，您的代理能够将特定的工单分类并路由到正确的部门吗？
And sometimes when you read this, you probably think, is it so hard to just do routing?
有时当你读到这篇文章时，你可能会想，仅仅做路由就这么难吗？
Why can't an agent easily do that?
为什么代理人不能轻易做到这一点？
And when you go to enterprises, routing itself can be a super complex problem.
当你进入企业时，路由本身可能是一个超级复杂的问题。
Any retail company, any popular retail company that you can think of has hierarchical taxonomies.
任何零售公司，任何你能想到的流行零售公司都有分层分类法。
Aishwarya Naresh Reganti: Most of the times the taxonomies are incredibly messy.
Aishwarya Naresh Reganti：大多数时候，分类法非常混乱。
I have worked in use cases where you probably have taxonomy that says some kind of hierarchy and then that says shoes and then women's shoes and men's shoes all at the same layer where idea you should be having shoes and then women's shoes and men's shoes should be subclasses.
我曾经在用例中工作过，在这些用例中，您可能有某种层次结构的分类法，然后说鞋子，然后是女鞋和男鞋都在同一层，您应该有鞋子，然后女鞋和男鞋应该是子类。
And then you're like, okay, fine.
然后你就会说，好吧，好吧。
I could just merge that.
我可以合并它。
And you go further and you see that there's also another section on the shoes that says for women and for men, and it's just not aggregated.
再往前走，你会发现鞋子上还有另外一个部分，分别写着适合女性和男性，但它并没有汇总。
It's not fixed for some reason.
由于某种原因它没有被修复。
So if an agent kind of sees this kind of a taxonomy, what is it supposed to do?
因此，如果代理看到这种分类法，它应该做什么？
Where is it supposed to route?
它应该路由到哪里？
And a lot of the times we are not aware of these problems until you actually go about building something and understanding it.
很多时候，直到你真正开始构建一些东西并理解它之前，我们都没有意识到这些问题。
Aishwarya Naresh Reganti: And when these kind of problems, real human agents see these kind of problems, they know what to check next.
Aishwarya Naresh Reganti：当这些问题，真正的人类特工看到这些问题时，他们知道下一步要检查什么。
Maybe they realize that the node that says for women and for men that's under shoes was last updated in 2019, which means that it's just a dead node that's lying there and not being used.
也许他们意识到，鞋底下的针对女性和男性的节点最后一次更新是在 2019 年，这意味着它只是一个死节点，躺在那里并且没有被使用。
So they kind of know that, okay, we're supposed to be looking at a different node and stuff like that.
所以他们知道，好吧，我们应该考虑不同的节点和类似的东西。
And I'm not saying agents cannot understand this or models are not capable enough to understand this, but there are really weird rules within enterprises that are not documented anywhere.
我并不是说代理无法理解这一点，或者模型没有足够的能力来理解这一点，但企业内部确实存在一些奇怪的规则，这些规则没有在任何地方记录下来。
And you want to make sure that the agents have all of that context instead of just throwing the problem at that.
您希望确保代理拥有所有这些上下文，而不是仅仅将问题抛给它。
Aishwarya Naresh Reganti: Yeah.
艾西瓦娅·纳雷什·雷甘蒂：是的。
Coming back to the versions we had, routing was one where you have really high control because even if your agent routes to the wrong department, humans can take control and undo those actions.
回到我们的版本，路由是一个你拥有真正高度控制权的版本，因为即使你的代理路由到错误的部门，人们也可以控制并撤消这些操作。
And along the way, you also figure out that you probably are dealing with a ton of data issues that you need to fix and make sure that your data layer is good enough for the agent to function.
在此过程中，您还发现您可能正在处理大量需要修复的数据问题，并确保您的数据层足以让代理正常运行。
We do is what we said of a Copilot, which is now that you've figured out routing works fine after a few iterations and you've fixed all of your data issues, you could go to the next step, which is, can my agent provide suggestions based on some standard operating procedures that we have for the customer support agent?
我们所做的就是我们所说的副驾驶，现在您已经在几次迭代后发现路由工作正常，并且您已经修复了所有数据问题，您可以进入下一步，即我的代理可以根据我们为客户支持代理提供的一些标准操作程序提供建议吗？
And it could just generate a draft that the human can make changes to.
它可以生成一个人类可以进行更改的草稿。
And when you do this, you're also logging human behavior, which means that how much of this draft was used by the customer support agent or what was omitted.
当您这样做时，您还记录了人类行为，这意味着客户支持代理使用了该草稿的多少内容或遗漏了哪些内容。
So you're actually getting error analysis for free when you do this because you're literally logging everything that the user is doing that you could then build back into your flywheel.
因此，当您这样做时，您实际上是免费获得错误分析的，因为您实际上记录了用户正在执行的所有操作，然后您可以将其重新构建到飞轮中。
Aishwarya Naresh Reganti: And then we say, post that, once you've figured out that those drafts look good and most of the times maybe humans are not making too many changes, they're using these drafts as is.
Aishwarya Naresh Reganti：然后我们说，一旦你发现这些草稿看起来不错，而且大多数时候人类可能不会做出太多改变，他们就会按原样使用这些草稿。
That's when you want to go to your end-to-end resolution assistant that could draft a resolution that could solve the ticket as well.
这时，您需要去找端到端解决方案助理，他可以起草一份解决方案来解决问题。
And those are the stages of agency where you start with low agency and then you go up high.
这些是代理阶段，您从低代理开始，然后逐渐上升。
We also have this really nice table that we put together, which is what do you do at each version and what you learn that can enable you to go to the next step and what information do you get that you can feed into the loop, right?
我们还整理了这张非常好的表格，其中列出了您在每个版本中所做的事情以及您学到的可以使您进入下一步的信息，以及您获得了哪些可以输入到循环中的信息，对吗？
When you're just doing your routing, you have better quality routing data, you also know what kind of prompts you need to be building to improve the routing system.
当您刚刚进行路由时，您拥有更好质量的路由数据，您还知道需要构建什么样的提示来改进路由系统。
Aishwarya Naresh Reganti: Essentially, you're figuring out your structure for context engineering and building that flywheel that you want.
Aishwarya Naresh Reganti：本质上，您正在弄清楚上下文工程的结构并构建您想要的飞轮。
And while I go through this, I want to also be very clear that two things.
在我经历这一点的同时，我还想非常清楚两件事。
One is when you build with CCCD in mind, it doesn't mean that you've fixed the problem all for one.
一是当您在构建时考虑到 CCCD，并不意味着您已经解决了所有问题。
It's possible that you've probably gone through V3 and you see a new distribution of data that you never previously imagined, but this is just one way to lower your risk, which is you get enough information about how users behave with your system before going to a point of complete autonomy.
您可能已经经历过 V3，并且看到了以前从未想象过的新数据分布，但这只是降低风险的一种方法，即您在达到完全自治之前获得有关用户如何使用系统的足够信息。
And the second thing is you're also kind of building this implicit logging system.
第二件事是你也在构建这个隐式日志系统。
A lot of people come and tell us that, "Oh, wait, there are evals.
很多人来告诉我们，“哦，等等，有评估。
Why do you need something like this?
为什么需要这样的东西？
" The issue with just building a bunch of evaluation metrics and then having them in production is evaluation metrics catch only the errors that you're already aware of, but there can be a lot of emerging patterns that you understand only after you put things in production.
“仅仅构建一堆评估指标然后将它们投入生产的问题是评估指标只能捕获您已经意识到的错误，但是可能有很多新兴模式只有在将东西投入生产后您才能理解。
Aishwarya Naresh Reganti: So for those emerging patterns, you're kind of creating a low risk kind of a framework so that you could understand user behavior and not really be in a position where there are tons of errors and you're trying to fix all of them at once.
Aishwarya Naresh Reganti：因此，对于这些新兴模式，您需要创建一个低风险的框架，以便您可以了解用户行为，而不是真正处于存在大量错误并且您试图立即修复所有错误的位置。
And this is not the only way to do it.
这并不是唯一的方法。
There are tons of different ways.
有很多不同的方法。
You want to decide how you constrain your autonomy.
你想决定如何限制你的自主权。
It could be based on the number of actions that the agent is taking, which is what we do in this example.
它可以基于代理正在采取的操作数量，这就是我们在本例中所做的。
It could be based on topic.
它可以基于主题。
There's just some domains where it's pretty high risk to make a system completely autonomous for certain decisions, but for some other topics, it's okay to make them completely autonomous and depending on the complexity of the problem.
在某些领域，让系统完全自主地做出某些决策的风险相当高，但对于其他一些主题，可以让它们完全自主，具体取决于问题的复杂性。
And that's where you really want your product managers, your engineers and subject matter experts to align on how to build this system and continuously improve it.
这就是您真正希望您的产品经理、工程师和主题专家就如何构建该系统并不断改进它达成一致的地方。
Aishwarya Naresh Reganti: The idea is just behavior calibration and not losing user trust as you do that behavior calibration, I guess.
Aishwarya Naresh Reganti：我想，这个想法只是行为校准，而不是在进行行为校准时失去用户信任。
Lenny Rachitsky: We'll link folks to this actual post if they want to go really deep.
Lenny Rachitsky：如果人们想深入了解，我们会将他们链接到这篇实际的帖子。
You basically go through all of these steps by step, a bunch of examples.
您基本上可以通过一堆示例逐步完成所有这些步骤。
And the idea here is, as you said, that the reason, everything about what you're describing here is about making it continuous and iterative and kind of moving along this progression of higher autonomy, less control.
正如你所说，这里的想法是，你在这里描述的一切的原因是为了使其连续和迭代，并沿着更高的自主性、更少的控制的进程前进。
And this idea of even calling continuous calibration, continuous development is communicating it's this kind of iterative process.
这种甚至称为持续校准、持续开发的想法正在传达这种迭代过程。
And just to be clear, this naming is kind of ode to CI/CD, continuous integration, continuous deployment suite.
需要明确的是，这个命名是对 CI/CD、持续集成、持续部署套件的颂歌。
And the idea here is that this is the version of that for AI where instead of just integrating into unit tests and deploying constantly, it's running evals, looking at results, iterating on the metrics you're watching, figuring out where it's breaking and iterating on that.
这里的想法是，这是人工智能的版本，它不只是集成到单元测试中并不断部署，而是运行评估，查看结果，迭代您正在观察的指标，找出它的问题并迭代它。
Awesome.
惊人的。
Okay.
好的。
Lenny Rachitsky: So again, we'll point people to this post if they want to go deeper.
Lenny Rachitsky：所以，如果人们想更深入地了解，我们会引导他们阅读这篇文章。
That was a great overview.
这是一个很棒的概述。
Is there anything else before we go into different topic around this framework specifically that you think is important for people to know?
在我们围绕这个框架进入不同的主题之前，还有什么您认为人们需要了解的重要信息吗？
Aishwarya Naresh Reganti: I think one of the most common questions we get is, how do I know if I need to go to the next stage or if this is calibrated enough?
Aishwarya Naresh Reganti：我认为我们最常见的问题之一是，我如何知道我是否需要进入下一阶段或者这是否已经足够校准？
There's not really a rule book you can follow, but it's all about minimizing surprise, which means let's say you're calibrating every one or two days and you figure out that you're not seeing new data distribution patterns, your users have been pretty consistent with how they're behaving with the system.
并没有真正可以遵循的规则手册，但这都是为了最大限度地减少意外，这意味着假设您每隔一两天进行一次校准，并且发现您没有看到新的数据分布模式，您的用户与他们在系统中的行为方式非常一致。
Then the amount of information you gain is kind of very low and that's when you know you can actually go to the next stage.
然后你获得的信息量非常低，那时你知道你实际上可以进入下一阶段。
And it's all about the wipes at that point, do you know you're ready, you're not receiving any new information.
那时一切都与擦拭巾有关，您知道自己已经准备好了吗，您没有收到任何新信息。
But also it really helps to understand that sometimes there are events that could completely mess up the calibration of your system.
但它也确实有助于理解，有时有些事件可能会完全扰乱系统的校准。
An example is GPT-4o doesn't exist anymore, or it's going to be deprecated in APIs as well.
一个例子是 GPT-4o 不再存在，或者它也将在 API 中被弃用。
Aishwarya Naresh Reganti: So most companies that were using 4o should switch to 5 and 5 has very different properties.
Aishwarya Naresh Reganti：所以大多数使用 4o 的公司应该转向 5，而 5 具有非常不同的特性。
So that's where your calibration's off again.
这就是你的校准再次关闭的地方。
You want to go back and do this process again.
您想返回并再次执行此过程。
Sometimes users start behaving with systems also differently over time or user behavior evolves.
有时，随着时间的推移，或者用户行为的演变，用户对系统的行为也会发生变化。
Even with consumer products, you don't talk to ChatGPT the same way you were talking, say, two years ago, just because you know the capabilities have increased so much.
即使对于消费产品，您也不会像两年前那样与 ChatGPT 交谈，只是因为您知道功能已经增加了很多。
And also just people get excited when these systems can solve one task, they want to try it out on other tasks as well.
当这些系统可以解决一项任务时，人们会感到兴奋，他们也想在其他任务上尝试。
We built this system for underwriters at some point.
我们在某个时候为承销商构建了这个系统。
Underwriting is a painful task.
承保是一项痛苦的任务。
There are agreements that are like loan applications are like 30 or 40 pages, and the idea for this bank was to build a system that could help underwriters pick policies and information about the bank so that they could approve loans.
有些协议就像贷款申请一样长达 30 或 40 页，这家银行的想法是建立一个系统，可以帮助承销商选择有关银行的政策和信息，以便他们可以批准贷款。
Aishwarya Naresh Reganti: And for a good three or four months, everybody was pretty impressed with the system.
Aishwarya Naresh Reganti：在三四个月的时间里，每个人都对这个系统印象深刻。
We had underwriters actually report gains in terms of how much time they were spending, et cetera.
我们让承销商实际上根据他们花费的时间等来报告收益。
And first three months, we realized that they were so excited with the product that they started asking very deep questions that we never anticipated.
前三个月，我们意识到他们对这个产品非常兴奋，以至于他们开始提出我们从未预料到的非常深入的问题。
They would just throw the entire application document at the system and go, "For a case that looks like this, what did previous underwriters do?
他们只是把整个申请文件扔到系统中，然后说：“对于这样的案例，以前的承销商是怎么做的？
" And for a user, that just seems like a natural extension of what they were doing, but the building behind it should significantly change.
“对于用户来说，这似乎是他们正在做的事情的自然延伸，但其背后的建筑应该发生重大变化。
Now, you need to understand what does for a case like this mean in the context of the loan itself?
现在，您需要了解这样的案例在贷款本身的背景下意味着什么？
Is it referring to people of a particular income range or is it referring to people in a particular geo and stuff like that?
它是指特定收入范围的人还是指特定地理区域的人等等？
Aishwarya Naresh Reganti: And then you need to pick up historical documents, analyze those documents, and then tell them, "Okay, this is what it looks like," versus just saying that there's a policy X, Y, and Z, and you want to look up that policy.
Aishwarya Naresh Reganti：然后你需要拿起历史文件，分析这些文件，然后告诉他们，“好吧，这就是它看起来的样子”，而不是仅仅说有一个政策 X、Y 和 Z，并且你想要查找该政策。
So something that might seem very natural to an end user might be very hard to build as a product builder, and you see that user behavior also evolves over time, and that's when you know that you want to go back and recalibrate.
因此，对于最终用户来说看起来非常自然的东西可能很难作为产品构建者来构建，并且您会看到用户行为也会随着时间的推移而演变，这时您就知道自己想要返回并重新校准。
Lenny Rachitsky: What do you think is overhyped in the AI space right now?
Lenny Rachitsky：您认为目前人工智能领域哪些方面被过度炒作了？
And even more importantly, what do you think is under-hyped?
更重要的是，您认为哪些内容被低估了？
Kiriti Badam: As I said, super optimistic in different things that are going in AI.
Kiriti Badam：正如我所说，对人工智能领域正在发生的不同事情非常乐观。
So I wouldn't say overhyped, but I feel kind of misunderstood is the concept of multi-agents.
所以我不会说过度炒作，但我觉得多代理的概念有点被误解。
People have this notion of, "I have this incredibly complex problem.
人们有这样的想法：“我有一个极其复杂的问题。
Now I'm going to break it down into, hey, you are this agent.
现在我要把它分解为，嘿，你是这个特工。
Take care of this.
照顾好这个。
You're this agent.
你就是这个经纪人。
Take care of this." And now if I somehow connect all of these agents, they think they're the agent utopia and it's never the case that there are incredibly successful multi-agent systems that are built.
照顾好这个。”现在，如果我以某种方式连接所有这些代理，他们会认为自己是代理乌托邦，而且从来没有建立过非常成功的多代理系统。
There's no doubt about that.
毫无疑问。
But I feel a lot of it comes in terms of how are you limiting the ways in which the system can go off tracks.
但我觉得很多问题都在于如何限制系统偏离轨道的方式。
And for example, if you're building a supervisor agent and there are subagents that actually do the work for the super agent, supervisor agent, that is a very successful pattern.
例如，如果您正在构建一个主管代理，并且有一些子代理实际上为超级代理（主管代理）执行工作，那么这是一个非常成功的模式。
Kiriti Badam: But coming with this notion of I'm going to divide the responsibilities based on functionality and somehow expect all of that to work together in some sort of gossip protocol, that is extremely misunderstood that you could do that.
Kiriti Badam：但是，随着我将根据功能划分职责，并以某种方式期望所有这些在某种八卦协议中协同工作，这是非常误解的，因为你可以这样做。
I don't think current ways of building and current model capabilities are right there in terms of building those kind of applications.
我认为当前的构建方法和当前的模型功能并不适合构建此类应用程序。
I feel that is kind of misunderstood than overrated.
我觉得这与其说是被高估了，不如说是被误解了。
Underrated, I feel it's hard to probably believe, but I still feel coding agents are underrated in the sense that I feel like you can go on Twitter and you can go on Reddit and you see a lot of chatter about coding agents, but talking to an engineer in any random company, especially outside of Bay Area, you can see the amount of impact this coding agents can create and the penetration is very low.
被低估了，我觉得很难相信，但我仍然觉得编码代理被低估了，因为我觉得你可以去 Twitter，你可以去 Reddit，你会看到很多关于编码代理的讨论，但是与任何随机公司的工程师交谈，特别是在湾区之外，你可以看到这种编码代理可以产生的影响力，而且渗透率非常低。
So I feel like 2025 and 2026 is going to be an incredible year for optimizing all of these processes.
所以我觉得 2025 年和 2026 年将是优化所有这些流程的令人难以置信的一年。
Kiriti Badam: And I feel that is going to be creating a lot of value with AI.
Kiriti Badam：我觉得人工智能将会创造很多价值。
Lenny Rachitsky: That's really interesting on that first point.
Lenny Rachitsky：第一点真的很有趣。
So the idea there is you'll probably be more successful building and using an agent that is able to do its own sub-agent splitting of work versus a bunch of, say, Codex agents.
因此，你可能会更成功地构建和使用能够完成自己的子代理工作分配的代理，而不是使用一堆（例如 Codex）代理。
Will you do this task, you do that task?
你会做这个任务，你会做那个任务吗？
Kiriti Badam: You can have agents to do these things and you as a human can orchestrate it or you can have one larger agent that is going to orchestrate all of these things, but letting the agents communicate in terms of peer-to-peer kind of protocol, and then especially doing this in a customer support kind of use case is incredibly hard to control what kind of agent is replying to your customer because you need to shift your guardrails everywhere and things like that.
Kiriti Badam：你可以让代理来做这些事情，而你作为一个人可以编排它，或者你可以有一个更大的代理来编排所有这些事情，但是让代理以点对点协议进行通信，然后特别是在客户支持类型的用例中这样做，很难控制哪种类型的代理正在回复你的客户，因为你需要到处移动你的护栏等等。
Lenny Rachitsky: Yeah.
莱尼·拉奇茨基：是的。
Okay.
好的。
Great picks.
很棒的选择。
Okay.
好的。
Ash, what do you got?
阿什，你有什么？
Aishwarya Naresh Reganti: Can I say evals?
艾西瓦娅·纳雷什·雷甘蒂：我可以说评估吗？
Will I be canceled?
我会被取消吗？
Lenny Rachitsky: In which category?
Lenny Rachitsky：属于哪一类？
Which bucket do they go?
他们去哪个桶？
Lenny Rachitsky: Overrated.
莱尼·拉奇茨基：被高估了。
Okay, go for it.
好吧，去吧。
We won't let you get canceled.
我们不会让你被取消。
Aishwarya Naresh Reganti: Just kidding.
艾西瓦娅·纳雷什·雷甘蒂：开个玩笑。
I think evals are misunderstood.
我认为评估被误解了。
They are important, folks.
伙计们，他们很重要。
I'm not saying they're not important, but I think just this, I'm going to keep jumping across tools and going to pick up and learn if new tool is overrated.
我并不是说它们不重要，但我认为就是这样，我将继续跳过工具，并学习和学习新工具是否被高估。
I still am old school and feel like you would really need to be obsessed with the business problem you're trying to solve.
我仍然是守旧派，觉得你真的需要专注于你想要解决的业务问题。
AI is only a tool.
人工智能只是一个工具。
I try to think of it that way.
我试着这样想。
Of course, you need to be learning about the latest and greatest, but don't be so obsessed with just building so quickly.
当然，您需要学习最新和最好的知识，但不要沉迷于如此快速的构建。
Building is really cheap today.
今天的建筑真的很便宜。
Design is more expensive, really thinking about your product, what you're going to build.
设计成本更高，真正考虑你的产品，你要建造什么。
Is it going to really solve a pain point?
真的能解决痛点吗？
Is what is way more valuable today?
今天什么更有价值？
And it will only become more true in the near future.
而且在不久的将来它只会变得更加真实。
So really obsessing about your problem and design is underrated and just rote building is overrated, I guess.
所以我想，真正痴迷于你的问题和设计被低估了，而只是死记硬背的建筑被高估了。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
Okay.
好的。
Similar sort of question.
类似的问题。
From a product point of view, what do you think the next year of AI is going to look like?
从产品的角度来看，您认为明年的人工智能会是什么样子？
Give us a vision of where you think things are going to go by, say by the end of 2026.
请告诉我们您认为事情将如何发展的愿景，比如到 2026 年底。
Kiriti Badam: Yeah, I feel there's a lot of promise in terms of this background agents are proactive agents who is ...
Kiriti Badam：是的，我觉得这个背景特工是积极主动的特工，他们有很多希望......
They're going to basically understand your workflow even more.
他们基本上会更加了解您的工作流程。
If you think of where is AI failing to create value today, it's mainly about not understanding the context.
如果你想到今天人工智能未能创造价值的地方，主要是因为不了解背景。
And the reason that it's not understanding the context is it's not plugged into the right places where actual work is happening.
它不理解上下文的原因是它没有插入实际工作发生的正确位置。
And as you do more of this, you can give the agent more of context and then it start to see the world around you and understand what are the set of metrics that you're optimizing for or what are the kind of activities that you're trying to do.
当您做得更多时，您可以为代理提供更多背景信息，然后它开始了解您周围的世界并了解您正在优化的一组指标或您正在尝试执行的活动类型。
It is a very easy extension from there to actually gain more out of it and then let the agent prompt you back.
这是一个非常简单的扩展，可以真正从中获得更多，然后让代理提示您回来。
We already do this in terms of ChatGPT pulse, which kind of gives you this daily update of things you might care about.
我们已经在 ChatGPT 脉冲方面做到了这一点，这可以为您提供您可能关心的事情的每日更新。
Kiriti Badam: And it's very nice to actually have that jog your brain up in terms of, "Oh, this is something that I haven't thought about.
Kiriti Badam：真的很高兴让你的大脑兴奋起来，“哦，这是我没有想过的事情。
Maybe this is good." And now when you extend this to more complex tasks, like a coding agent, which says that, "Okay, I have fixed five of your linear tickets and here are the patches.
也许这很好。”现在，当您将其扩展到更复杂的任务时，例如编码代理，它会说：“好吧，我已经修复了您的五个线性票证，这是补丁。
Just to review them at the start of your day." So I feel that is going to be extremely useful.
只是为了在一天开始时回顾一下它们。”所以我觉得这将非常有用。
And I see that as a strong direction in which products are going to build in 2026.
我认为这是 2026 年产品开发的一个强有力的方向。
Lenny Rachitsky: That's so cool.
莱尼·拉奇茨基：那太酷了。
So essentially agents anticipating what you want to do and getting ahead of you and I've solved these problems for you or I think this is going to crash your site.
因此，本质上，代理会预测您想要做什么并领先于您，我已经为您解决了这些问题，否则我认为这将使您的网站崩溃。
Maybe you should fix this thing right here or I see the spike here and let's refactor our database.
也许你应该在这里修复这个问题，或者我在这里看到峰值，让我们重构我们的数据库。
Amazing.
惊人的。
What a world.
真是一个世界啊。
Okay.
好的。
Ash, what do you got?
阿什，你有什么？
Aishwarya Naresh Reganti: I'm all in for multimodal experiences in 2026.
Aishwarya Naresh Reganti：我全力支持 2026 年的多式联运体验。
I think we have done quite some progress in 2025, and not just in terms of generation, but also understanding.
我认为我们在 2025 年已经取得了相当大的进步，不仅在一代人方面，而且在理解方面。
Until now, I think LLMs have been our most commonly used modules, but as humans, we are multimodal creatures, I would say.
到目前为止，我认为法学硕士一直是我们最常用的模块，但我想说，作为人类，我们是多模式生物。
Language is probably one of our last forms of evolution.
语言可能是我们最后的进化形式之一。
As the three of us are talking, I think we're constantly getting so many signals.
当我们三个人谈话时，我想我们不断收到很多信号。
I'm like, "Oh, Lenny's nodding his head, so probably I would go in this direction or Lenny's bored, so let me stop talking." So there's a chain of thought behind your chain of thought and you're constantly altering it with language that dimension of expression is not explored as well.
我想，“哦，莱尼在点头，所以我可能会朝这个方向走，或者莱尼很无聊，所以让我停止说话。”所以你的思想链背后有一个思想链，你不断地用语言改变它，而表达的维度也没有被探索。
So if we could build better multimodal experiences that would get us closer to human-like conversation richness.
因此，如果我们能够构建更好的多模式体验，将使我们更接近人类的丰富对话。
And you will also, just given the kind of models, there's a bunch of boring tasks as well, which are ripe for AI.
而且，只要给出了模型的种类，就会出现一堆无聊的任务，而这些任务对于人工智能来说已经成熟了。
Aishwarya Naresh Reganti: If multimodal understanding gets better, there are so many handwritten documents and really messy PDFs that cannot be passed even by the best of the models as of today.
Aishwarya Naresh Reganti：如果多模态理解变得更好，就会有很多手写文档和非常混乱的 PDF，即使是当今最好的模型也无法通过。
And if it's possible, there'll be so much data that we can tap into.
如果可能的话，我们将能够利用大量数据。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
I just saw Demis from DeepMind, AI, Google, whatever they call the whole org, talking about this where he thinks that's going to be a big part of where they're going, combining the image model work, the LLM, and also their world model stuff, Genie, I think is what it's called.
我刚刚看到来自 DeepMind、AI、Google 的 Demis，无论他们如何称呼整个组织，他都在谈论这个问题，他认为这将成为他们未来发展的重要组成部分，结合了图像模型工作、法学硕士，以及他们的世界模型工作，我想这就是所谓的 Genie。
Yes.
是的。
So that's going to be a wild, wild time.
所以这将是一段非常非常疯狂的时光。
Okay.
好的。
Last question.
最后一个问题。
If someone wants to just get better at building AI products, what's just maybe one skill or maybe two skills that you think they should lean into and develop?
如果有人想更好地构建人工智能产品，您认为他们应该学习和发展的一项或两项技能是什么？
Aishwarya Naresh Reganti: I think we did cover a bunch of best practices for AI products, which is start small, try to get your iteration going well and build a flywheel and all of that.
Aishwarya Naresh Reganti：我认为我们确实涵盖了人工智能产品的一系列最佳实践，从小处开始，尝试让迭代顺利进行并构建飞轮等等。
But again, if you kind of look at it at a 10,000 feet level for anybody building today, like I was saying, implementation is going to be ridiculously cheap in the next few years.
但同样，如果你从 10,000 英尺的高度来看今天的任何建筑，就像我所说的那样，未来几年的实施成本将低得离谱。
So really nail down your design, your judgment, your taste and all of that.
所以要真正确定你的设计、你的判断、你的品味等等。
And in general, if you're building a career as well, I feel for the past few years, your former years, say the first two, three years of building your career is always focused on execution, mechanics and all of that.
总的来说，如果你也在建立自己的职业生涯，我觉得在过去的几年里，你以前的几年，比如说你职业生涯的前两三年，总是专注于执行、机制和所有这些。
And now we have AI that could help you ramp pretty quickly and post that.
现在我们有了人工智能，可以帮助您快速提升并发布内容。
I mean, after a few years, I think everybody's job becomes about your taste, your judgment and kind of what is uniquely you.
我的意思是，几年后，我认为每个人的工作都取决于你的品味、你的判断以及你的独特之处。
Aishwarya Naresh Reganti: I think nail down on that part and try to figure out how you can bring in that kind of a perspective.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：我认为要明确这一点，并尝试找出如何引入这种观点。
It doesn't have to mean that you should be significantly old, have years of experience.
这并不一定意味着您应该很老，有多年的经验。
We recently hired someone and we use this very popular app for tracking our tasks and we've been using it for years and we pay a high subscription fee for it.
我们最近雇用了一个人，我们使用这个非常受欢迎的应用程序来跟踪我们的任务，我们已经使用它很多年了，并且我们为此支付了高额订阅费。
And this guy just came with his own vibe coded app to the meeting.
这个人刚刚带着他自己的氛围编码应用程序来参加会议。
He onboarded us to all of it and he's like, "Okay, let's start using this." And I think that kind of agency and that kind of ownership to really rethink experiences is what will set people apart.
他让我们了解了这一切，他说：“好吧，让我们开始使用这个吧。”我认为这种真正重新思考体验的机构和所有权会让人们与众不同。
And I'm not being blind to the fact that vibe coded apps have high maintenance costs.
我并没有忽视这样一个事实：Vivi 编码的应用程序具有很高的维护成本。
And maybe as we scale as a company, we have to replace it or we have to think of better approaches.
也许随着公司规模的扩大，我们必须更换它，或者我们必须考虑更好的方法。
Aishwarya Naresh Reganti: But given that we are a small size company now and just ...
Aishwarya Naresh Reganti：但考虑到我们现在是一家小规模公司，而且......
I was really shocked because I never thought of it.
我真的很震惊，因为我从来没有想到过。
If you've been used to working in a certain way, you associate a cost with building.
如果您习惯于以某种方式工作，您就会将成本与构建联系起来。
And I feel like folks who grew up in this age have a much lower cost associated in their mind.
我觉得在这个时代长大的人们心目中的相关成本要低得多。
They just don't mind building something and going ahead with it.
他们只是不介意建造一些东西并继续它。
And they're also very enthusiastic to try out new tools.
他们也非常热衷于尝试新工具。
That's also probably why AI products have this retention problem because everybody's so excited about trying out these new tools and all of that.
这也可能是人工智能产品存在保留问题的原因，因为每个人都对尝试这些新工具以及所有这些感到非常兴奋。
But essentially having the agency and ownership, and I think it's also the going to be the end of the busy work era.
但本质上拥有代理权和所有权，我认为这也将是繁忙工作时代的结束。
You can't be sitting in a corner doing something that doesn't move the needle for a company.
你不能坐在角落里做一些对公司没有推动作用的事情。
You really need to be thinking about end-to-end workflows, how you can bring in more impact.
您确实需要考虑端到端的工作流程，以及如何带来更大的影响。
Aishwarya Naresh Reganti: I think all of that will be super important.
艾西瓦娅·纳雷什·雷甘蒂：我认为所有这些都非常重要。
Lenny Rachitsky: That reminds me, I just had Jason Lemkit on the podcast.
Lenny Rachitsky：这让我想起了，我刚刚在播客上看到了 Jason Lemkit。
He's very smart on sales, go to market, run Saster, and he replaced his whole sales team with agents.
他在销售方面非常聪明，他会深入市场，经营 Saster，并且用代理商取代了整个销售团队。
He had 10 salespeople and then he was 1.2 and 20 agents.
他有 10 名销售人员，然后是 1.2 名销售人员和 20 名代理商。
And one of the agents, it was just tracking everyone's updates to Salesforce and kind of updating it automatically for them based on their calls.
其中一位代理，它只是跟踪每个人对 Salesforce 的更新，并根据他们的呼叫自动为他们更新。
And one of the salespeople was like, "Okay, I quit." And it turned out he wasn't really doing anything.
其中一位销售人员说：“好吧，我辞职了。”结果发现他并没有真正做任何事情。
He was just sitting around and he's like, "Okay, this will catch me.
他只是坐在那里，然后说：“好吧，这会抓住我的。
I got to get out of here.
我必须离开这里。
So to your point about, it'll be harder to sit around and twiddle your thumbs, I think is really right.
所以就你的观点而言，坐在那里无所事事会变得更加困难，我认为这是非常正确的。
Kiriti Badam: Yeah.
基里蒂·巴达姆：是的。
I think to add on to that, I feel like persistence is also something that is extremely valuable, especially given that anybody who wants to build something, the information is at your fingertips even more than the past decade.
我想补充一点，我觉得坚持也是非常有价值的，特别是考虑到任何想要构建一些东西的人，信息比过去十年更触手可及。
You can learn anything overnight and become that sort of Ironman kind of approach.
你可以在一夜之间学到任何东西，并成为那种铁人三项的方法。
So I feel like having that persistence and going through the pain of learning this, implementing this and understanding what works and what doesn't work.
所以我觉得要有这样的毅力，经历学习、实施、理解什么有效、什么无效的痛苦。
And as you are going through this pain of developing multiple approaches and then solving the problem, I feel that is going to be the real moat as an individual.
当你经历开发多种方法然后解决问题的痛苦时，我觉得这将成为个人真正的护城河。
I like to call it pain is the new moat, but I feel that is exactly super useful to actually have this in, especially in building these AI products.
我喜欢将其称为“痛苦是新的护城河”，但我觉得将其真正纳入其中非常有用，尤其是在构建这些人工智能产品时。
Lenny Rachitsky: Say more about this.
Lenny Rachitsky：对此多说一些。
I love this concept.
我喜欢这个概念。
Pain is the new moat.
痛苦是新的护城河。
Is there more there?
还有更多吗？
Kiriti Badam: Yeah, I feel as a company, I mean, successful companies right now building in any new area, they are successful not because they're first to the market or they have this fancy feature that more customers are liking it.
Kiriti Badam：是的，我觉得作为一家公司，我的意思是，现在在任何新领域建立的成功公司，他们的成功并不是因为他们是第一个进入市场的人，或者他们拥有更多客户喜欢的奇特功能。
They went through the pain of understanding what are the set of non-negotiable things and trade them off exactly with what are the features or what are the model capabilities that they can use to solve that problem.
他们经历了理解哪些是不可协商的事情的痛苦，并将它们与可用于解决该问题的功能或模型功能进行了准确的权衡。
This is not a straightforward process.
这不是一个简单的过程。
There's no textbook to do this or there's no straightforward way or a known credit path to be here.
没有教科书可以做到这一点，或者没有直接的方法或已知的信用路径。
So a lot of this pain I was talking about is just going through this iteration of like, "Okay, let's try this and if this doesn't work, let's try this." And that kind of knowledge that you built across the organization or across your own lived experiences, I feel that pain is what translates into the moat of the company.
所以我所说的很多痛苦只是经历这样的迭代，“好吧，让我们试试这个，如果这不起作用，让我们试试这个。”我觉得你在整个组织或你自己的生活经历中积累的那种知识，我觉得痛苦可以转化为公司的护城河。
This could be a product of evals or something that you built.
这可能是评估的产品或您构建的东西。
And I feel that is going to be the game changer.
我觉得这将改变游戏规则。
Lenny Rachitsky: That is awesome.
莱尼·拉奇茨基：太棒了。
It's like turning a coal into diamond.
这就像把煤变成钻石一样。
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
I feel like we've done a great job helping people avoid some of the biggest issues people consistently run into building AI products.
我觉得我们在帮助人们避免构建人工智能产品时经常遇到的一些最大问题方面做得非常出色。
We covered so many of the pitfalls and the ways to actually do it correctly.
我们涵盖了许多陷阱以及实际正确执行此操作的方法。
Before we get to our very exciting lightning round, is there anything else that you wanted to share?
在我们开始激动人心的闪电回合之前，您还有什么想分享的吗？
Anything else you want to leave listeners with?
您还有什么想留给听众的吗？
Aishwarya Naresh Reganti: Be obsessed with your customers.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：以客户为中心。
Be obsessed with the problem.
沉迷于问题。
AI is just a tool and try to make sure that you're really understanding your workflows.
人工智能只是一个工具，请尝试确保您真正了解您的工作流程。
80% of so called AI engineers, AIPMs spend their time actually understanding their workflows very well.
80% 的所谓 AI 工程师、AIPM 都花时间真正了解他们的工作流程。
They're not building the fanciest and the most cool models or workflows around it.
他们并没有围绕它构建最奇特、最酷的模型或工作流程。
They're actually in the weeds understanding their customer's behavior and data.
他们实际上正在深入了解客户的行为和数据。
And whenever a software engineer who's never done AI before, here's the term, look at your data.
每当一个以前从未从事过人工智能的软件工程师，这就是术语，看看你的数据。
I think it's a huge revelation to them, but it's always been the case.
我认为这对他们来说是一个巨大的启示，但情况一直如此。
You need to go there, look at your data, understand your users, and that's going to be a huge differentiator.
你需要去那里，查看你的数据，了解你的用户，这将是一个巨大的差异化因素。
Lenny Rachitsky: That's a great way to close it.
Lenny Rachitsky：这是结束它的好方法。
The AI isn't the answer.
人工智能不是答案。
It's a tool to solve the problem.
它是解决问题的工具。
With that, we have reached our very exciting lightning round.
至此，我们进入了非常激动人心的闪电回合。
I've got five questions for both of you.
我有五个问题要问你们俩。
Are you ready?
你准备好了吗？
Lenny Rachitsky: All right.
莱尼·拉奇茨基：好的。
So you can both answer them.
所以你们都可以回答。
You can pick one which you want to answer.
您可以选择您想要回答的一个。
Either way, up to you.
不管怎样，由你决定。
What are two or three books you find yourself recommending most to other people?
您发现自己最向其他人推荐的两三本书是什么？
Aishwarya Naresh Reganti: For me, it's this book called When Breath Becomes Air, Lenny.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：对我来说，这本书叫《当呼吸化为空气，莱尼》。
It was written by Paul Kalanithi.
它的作者是保罗·卡拉尼蒂。
I think he was an Indian original neurosurgeon who was diagnosed with lung cancer at 31 or 32.
我认为他是一位印度原创神经外科医生，在31或32岁时被诊断出患有肺癌。
And the whole book is his memoir and just is written after he was diagnosed.
整本书是他的回忆录，是他确诊后写的。
And it's really beautiful, especially because I read it during COVID and all we ever wanted to do during COVID is stay alive.
它真的很美，特别是因为我在新冠疫情期间读到了这本书，而我们在新冠疫情期间想做的就是活下去。
There are a bunch of really nice quotes within the book as well, but I remember one of them, he was kind of arguing against a very popular quote by Socrates, which is, "The unexamined life is not worth living," or something like that, which means you really need to be thinking about your choices, you need to understand your values, your mission and all of that.
书中还有很多非常好的名言，但我记得其中有一个，他有点反对苏格拉底的一句非常流行的名言，那就是“未经审视的生活不值得过”或类似的话，这意味着你真的需要考虑你的选择，你需要了解你的价值观，你的使命等等。
And Paul says, "If the unexamined life is not worth living, was the unlived life worth examining?" Which means are you spending so much time just understanding your mission and purpose that you've forgotten to live?
保罗说：“如果未经省察的生活不值得过，那么未经省察的生活还值得省察吗？”这意味着你花了太多时间来理解你的使命和目的，以至于你忘记了生活？
Aishwarya Naresh Reganti: And I think everybody who's staying in the AI era and building and continuously going through the space of reinventing themselves need to take a pause and live for a bit, I guess.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：我想，我认为，每个留在人工智能时代、不断构建并不断重塑自己空间的人都需要暂停一下，活一段时间。
They need to stop evaling life too much.
他们需要停止过多地评价生活。
Lenny Rachitsky: I was going to say that.
莱尼·拉奇茨基：我正想说这个。
That's where my mind went.
这就是我的想法。
You got to write some evals for your life.
你必须为你的生活写一些评估。
Oh my God, we've gone too far.
天啊，我们已经走得太远了。
Kiriti Badam: I like more of science fiction books.
Kiriti Badam：我更喜欢科幻小说。
So I really like this 3 Body problem series.
所以我真的很喜欢这个三体问题系列。
It's like a three book series.
这就像一个三本书系列。
It has elements of grander than science fiction, life outside earth and how it impacts human decision making process.
它具有比科幻小说更宏大的元素，包括地球以外的生命以及它如何影响人类的决策过程。
And it also has elements of geopolitics and how much important or valuable abstract science is to human progress.
它还具有地缘政治的元素以及抽象科学对人类进步有多么重要或有价值。
And then when that gets stopped, it's not noticeable in everyday life, but it can cause devastating effects.
当这种情况停止时，它在日常生活中并不明显，但它可能会造成毁灭性的影响。
So I feel like AI helping in these areas, for example, is going to be extremely crucial.
因此，我觉得人工智能在这些领域的帮助将非常重要。
And that book is a nice example of what could happen otherwise.
那本书是一个很好的例子，说明了否则会发生什么。
Lenny Rachitsky: Completely agree.
莱尼·拉奇茨基：完全同意。
Absolutely.
绝对地。
Love.
爱。
Might be my favorite sci-fi book except, or series even, and it's three.
可能是我最喜欢的科幻小说，甚至是系列，而且是三本。
I have to read of all three, by the way.
顺便说一句，我必须阅读这三本书。
I find that it only got really good about one and a half books in.
我发现它只写了一本书半左右。
So if anyone's tried it and like, "What the heck is going on here?" Just keep reading and get to the middle of the second one and then it gets mind-blowing.
因此，如果有人尝试过并喜欢“这里到底发生了什么？”只要继续读下去，读到第二篇的中间，就会让人兴奋不已。
Lenny Rachitsky: If you love sci-fi and you're in AI, you got to read this book called A Fire Upon the Deep by Vernon Vinge.
Lenny Rachitsky：如果你喜欢科幻小说并且喜欢人工智能，那么你必须阅读 Vernon Vinge 所著的《A Fire Upon the Deep》这本书。
Check it out.
一探究竟。
It's incredible.
太不可思议了。
I saw Noah Smith on his newsletter recommend this book and there's sequels to it, but this is the one that's so incredible.
我看到诺亚·史密斯在他的时事通讯上推荐了这本书，并且还有续集，但这是一本如此令人难以置信的书。
And it's actually, it turns out it's about AGI and super intelligence and all these things, and it's just so epic.
事实上，事实证明它是关于通用人工智能和超级智能以及所有这些东西，而且它是如此史诗般。
And no one's heard of it.
没有人听说过它。
Lenny Rachitsky: There you go.
莱尼·拉奇茨基：就这样吧。
I'm giving you one back.
我还给你一个
Okay, next question.
好的，下一个问题。
What's a favorite recent movie or TV show that you've really enjoyed?
您最近最喜欢的电影或电视节目是什么？
Aishwarya Naresh Reganti: I started rewatching Silicon Valley and I think it's so true.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：我开始重新审视《硅谷》，我认为这是真的。
It's so timeless.
它是如此永恒。
Everything is repeating all over again.
一切都在重演。
Anybody who's watched it a few years ago should start rewatching it and you'll see that it's eerily similar to everything that's happening right now with the AI wave.
任何几年前看过它的人都应该开始重新观看它，你会发现它与现在人工智能浪潮中发生的一切出奇地相似。
Lenny Rachitsky: That's a good idea to rewatch it.
Lenny Rachitsky：重新观看是个好主意。
I love that their whole business was like an algorithm to compress, like a compression algorithm.
我喜欢他们的整个业务就像一个压缩算法，就像一个压缩算法。
It's like maybe a precursor to LLMs in some small way.
在某种程度上，这可能是法学硕士的前身。
No, I get it.
不，我明白了。
All right, Kiriti, what you got?
好吧，基里蒂，你得到了什么？
Kiriti Badam: I'm going to drag this and say lot a movie or a TV show, but there's this game I picked up recently called Expedition 33.
Kiriti Badam：我打算把它拖到电影或电视节目中，但我最近玩了一款叫做 Expedition 33 的游戏。
It has nothing to do with AI, but it's an incredibly well-made game in terms of the gameplay or the movie and the story and the music.
它与人工智能无关，但无论从游戏玩法、电影、故事还是音乐来看，它都是一款制作精良的游戏。
It's been amazing.
太棒了。
Lenny Rachitsky: I love that you have time to play games.
Lenny Rachitsky：我喜欢你有时间玩游戏。
That's a great sign.
这是一个很好的迹象。
I love that.
我喜欢那个。
Someone OpenAI, I'm just imagining you're ...
有人 OpenAI，我只是想象你...
There's nothing else going on except just coding and having meetings.
除了编码和开会之外，没有其他事情发生。
Kiriti Badam: Yeah, it has been incredibly hard to find time for that.
Kiriti Badam：是的，很难找到时间来做这件事。
Lenny Rachitsky: That's good.
莱尼·拉奇茨基：那很好。
That's a good sign.
这是一个好兆头。
I'm happy to hear this.
我很高兴听到这个。
Okay.
好的。
What's a favorite product that you've recently discovered that you really love?
您最近发现自己真正喜欢的最喜欢的产品是什么？
Aishwarya Naresh Reganti: For me, it's Whisper Flow.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：对我来说，它是 Whisper Flow。
I think I've been using it quite a bit and I didn't know I needed it so much.
我想我已经使用它很多次了，但我不知道我如此需要它。
The best part is it's a conceptual transcription tool, which means if you go to Codex and start using Whisper Flow, it starts identifying variables and all of that.
最好的部分是它是一个概念转录工具，这意味着如果您转到 Codex 并开始使用 Whisper Flow，它就会开始识别变量等等。
And it's so seamless in terms of transcription to instruction.
而且在转录到指令方面是如此无缝。
You could say something like, "I'm so excited today.
你可以这样说：“我今天很兴奋。
Add three exclamation marks," and it seamlessly switches.
添加三个感叹号”，然后就可以无缝切换。
It adds those three exclamation marks instead of writing add three exclamation marks.
它添加了这三个感叹号，而不是编写“添加三个感叹号”。
And I think it's pretty cool.
我认为这很酷。
If you're not using it, you should try it.
如果您没有使用它，您应该尝试一下。
Lenny Rachitsky: I'll do a plug.
Lenny Rachitsky：我会做一个插件。
Get Whisper Flow for free for an entire year for a year for free by becoming an annual subscriber of my newsletter.
成为我的时事通讯的年度订阅者，即可免费获得一整年的 Whisper Flow。
Aishwarya Naresh Reganti: That's how I got access to it, Lenny.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：这就是我获得它的方式，莱尼。
Lenny Rachitsky: There we go.
莱尼·拉奇茨基：我们开始吧。
I think I pitched this deal.
我想我促成了这笔交易。
I think people don't truly understand how incredible this is.
我认为人们并没有真正理解这是多么不可思议。
They're like, "No way this is real.
他们就像是，“这不可能是真的。
It's real." And 18 other products, lennysproductpass.com, check it out.
这是真的。”还有其他 18 个产品，lennysproductpass.com，请查看。
Moving on.
继续前进。
Kiriti.
基里蒂。
Kiriti Badam: Awesome.
基里蒂·巴达姆：太棒了。
I actually am a stickler for productivity.
事实上，我是一个注重生产力的人。
I keep experimenting new CLI tools and things which can make me faster.
我不断尝试新的 CLI 工具和可以让我更快的东西。
So I feel like a Raycast has been amazing.
所以我觉得 Raycast 非常棒。
I've discovered all this new shortcuts that you can use to open different things, type in shortcut commands and things like that.
我发现了所有这些新的快捷方式，您可以使用它们来打开不同的东西，输入快捷方式命令等等。
And Caffeinate is another thing that I've recently discovered from my teammates.
咖啡因是我最近从队友那里发现的另一件事。
It helps you prevent Mac from sleeping so you can run this really long Codex task for four or five hours locally, let it build the thing and then you can wake up and be like, "Okay, this is good.
它可以帮助你防止 Mac 休眠，这样你就可以在本地运行这个很长的 Codex 任务四五个小时，让它构建这个东西，然后你就可以醒来并说，“好吧，这很好。
I like this."
我喜欢这个。”
Lenny Rachitsky: That's hilarious, that combo.
Lenny Rachitsky：这组合太搞笑了。
Codex and Caffeinate.
法典和咖啡因。
You guys need to use it, build that yourself, an OpenAI version of that, or the Codex agent should just keep your Mac from sleeping.
你们需要使用它，自己构建它，一个 OpenAI 版本，或者 Codex 代理应该让你的 Mac 不睡觉。
That's so funny.
太有趣了。
By the way, Raycast, also part of Lenny's product pass.
对了，Raycast，也是Lenny的产品通行证的一部分。
One year for your Raycast.
Raycast 为期一年。
Amazing.
惊人的。
Yeah.
是的。
Aishwarya Naresh Reganti: Lenny didn't tell us these folks.
艾西瓦娅·纳雷什·雷甘蒂：莱尼没有告诉我们这些人。
Yes.
是的。
These are actually our favorite products.
这些实际上是我们最喜欢的产品。
Lenny Rachitsky: These are just two of 19 products.
Lenny Rachitsky：这些只是 19 种产品中的两种。
No Caffeinate though.
但不含咖啡因。
I don't know if that's even paid.
我不知道这是否是付费的。
Okay, let's keep going.
好吧，我们继续吧。
Do you have a favorite life motto that you find yourself coming back to in work or in life?
您是否有最喜欢的人生格言，并在工作或生活中反复遵循？
Aishwarya Naresh Reganti: For me, I think this is one my dad told me when I was a kid and it's always stuck, which is they told it couldn't be done, but the fool didn't know it, so he did it anyway.
艾西瓦娅·纳雷什·雷甘蒂（Aishwarya Naresh Reganti）：对我来说，我想这是我小时候父亲告诉我的，但它总是被卡住，他们告诉我这是不可能完成的，但傻瓜不知道，所以他还是这么做了。
I think be foolish enough to believe that you can do anything if you put your heart to it, especially now because you have so much data at your hand that could be pointing towards the fact that you probably will be unsuccessful.
我认为愚蠢到相信只要你用心去做任何事情，你就可以做任何事情，尤其是现在，因为你手头有如此多的数据，这些数据可能表明你可能会失败。
How many podcasts made it to more than a thousand subscribers or how many companies hit more than one million ARR?
有多少播客的订阅人数超过 1000 人，或者有多少公司的 ARR 超过 100 万？
And there's always data to show you that you won't be successful, but sometimes just be foolish and go ahead with it.
总有数据告诉你，你不会成功，但有时只是愚蠢地继续下去。
Kiriti Badam: For me, I am more of an overthinker.
Kiriti Badam：对我来说，我更像是一个过度思考者。
So I really like this quote from Steve Jobs that you can only connect the dots looking backwards.
所以我真的很喜欢史蒂夫·乔布斯的这句话，你只能回顾过去才能将点点滴滴联系起来。
So a lot of the times there are numerous choices and you don't really know the optimal one to pick, but life works in ways that you can actually see back and be like, "Oh, these are actually beautiful in terms of how our transition." So I feel like that is extremely useful in keep moving forward, keep experimenting.
所以很多时候，有无数的选择，你并不真正知道选择最佳的一个，但生活的运作方式，你可以真正回顾过去，然后说，“哦，就我们的转变而言，这些实际上是美丽的。”所以我觉得这对于不断前进、不断尝试非常有用。
Lenny Rachitsky: Final question.
莱尼·拉奇茨基：最后一个问题。
Whenever I have two guests on the podcast at once, I like to ask this question.
每当我的播客同时有两位嘉宾时，我喜欢问这个问题。
What's something that you admire about the other person?
你欣赏对方的什么地方？
Aishwarya Naresh Reganti: I think with Kiriti, he's pretty calm and very grounded and he's always been my sounding board.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：我认为 Kiriti 非常冷静且脚踏实地，他一直是我的参谋。
I can throw a ton of ideas at him and he always comes up with, he's able to anticipate the kind of issues that might land into.
我可以向他提出很多想法，他总是能想到，他能够预测可能会遇到的问题。
And he's extremely kind and lets his work speak instead of actually doing a lot of talking, I guess.
我想，他非常友善，让他的工作说话，而不是实际上做很多谈话。
But if I had to pick one, I think he's the most incredible husband.
但如果我必须选一个，我认为他是最不可思议的丈夫。
Lenny Rachitsky: Reveal.
莱尼·拉奇茨基：揭晓。
Little did people know.
人们知之甚少。
Aishwarya Naresh Reganti: We've been married for four years and been the most beautiful four years of my life.
艾西瓦娅·纳雷什·雷甘蒂 (Aishwarya Naresh Reganti)：我们结婚四年了，这是我一生中最美好的四年。
Lenny Rachitsky: Wow.
莱尼·拉奇茨基：哇。
Okay.
好的。
How do you follow that?
你怎么看？
Kiriti Badam: Yeah, it's super hard to follow that.
Kiriti Badam：是的，很难理解这一点。
I would say I am extremely privileged in terms of working with really smart people in great companies in the Silicon Valley.
我想说，能够与硅谷伟大公司中真正聪明的人一起工作，我感到非常荣幸。
And I feel the unique thing that stands with Aishwarya across like any other smart folks I've worked on is she has this really amazing knack of teaching and explaining something in a very understandable and easy to comprehend way.
我觉得艾西瓦娅与我合作过的其他聪明人一样，有一个独特之处，那就是她有一种非常惊人的技巧，能够以一种非常容易理解和容易理解的方式来教学和解释某些东西。
And that combined with persistence is super useful, especially in this fast-moving AI world that we are in the sense that there's so many new things coming up.
与坚持相结合是非常有用的，特别是在这个快速发展的人工智能世界中，我们感觉有很多新事物正在出现。
It feels overwhelming, but when I hear her talk about, this is how you make sense of this entire thing, this is where it plugs in.
这感觉让人不知所措，但当我听到她谈论时，这就是你理解整件事的方式，这就是它的意义所在。
I feel like, oh, that is so simple.
我感觉，哦，这就是这么简单。
I can also do that.
我也可以这样做。
So she empowers a lot of people by simplifying things and explaining things in the most understandable way.
因此，她通过简化事情并以最容易理解的方式解释事情，赋予了很多人力量。
Kiriti Badam: So I feel that is an incredible quality.
Kiriti Badam：所以我觉得这是一种令人难以置信的品质。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
How sweet.
多么甜蜜啊。
I got to do this all the time.
我必须一直这样做。
I need more guest to do it.
我需要更多的客人来做这件事。
That was great.
那很棒。
Okay.
好的。
Final questions.
最后的问题。
Where can folks find stuff that you're working on, find you online, share your course link, and then just how can listeners be useful to you?
人们可以在哪里找到您正在做的事情，在网上找到您，分享您的课程链接，然后听众如何对您有用？
Aishwarya Naresh Reganti: I write a lot on LinkedIn.
Aishwarya Naresh Reganti：我在 LinkedIn 上写了很多东西。
So if you want to listen to pragmatists who've been in the weeds, working on AI products and what they're seeing, you can follow my work.
因此，如果你想听听那些一直在研究人工智能产品的实用主义者的意见以及他们的所见，你可以关注我的工作。
We also have a GitHub repository with about 20K stars, and that repository is all about good resources for learning AI.
我们还有一个拥有大约 20K star 的 GitHub 存储库，该存储库都是有关学习 AI 的优质资源。
It's completely free.
它是完全免费的。
And if you like what we spoke today, we also run a super popular course.
如果您喜欢我们今天所说的内容，我们还开设了一个非常受欢迎的课程。
We leave a link to it on building enterprise AI products.
我们留下了有关构建企业人工智能产品的链接。
And the course is a lot about unlearning mindsets and following a problem-first approach instead of a tool-first or a hype-first approach.
该课程主要是关于放弃学习心态并遵循问题优先的方法，而不是工具优先或炒作优先的方法。
So you can check that out as well.
所以你也可以检查一下。
And if you don't want to do the course, we write a lot, we give out a lot of free resources, we have free sessions, so make sure you follow our work.
如果你不想学习这门课程，我们会写很多东西，我们会提供很多免费资源，我们有免费课程，所以请确保你关注我们的工作。
Kiriti Badam: Yeah, I would also add that you can also find me on LinkedIn.
Kiriti Badam：是的，我还要补充一点，您也可以在 LinkedIn 上找到我。
I don't write a lot, I guess, but I'm super all excited to just talk to any complex product that you're building.
我想我写的不多，但我非常高兴能够与您正在构建的任何复杂产品进行交谈。
And if you have thoughts on how you can use coding agents to make your life better or however the problems that you're seeing, always my DMs are open and we can have a great discussion.
如果您有关于如何使用编码代理让您的生活变得更好的想法，或者您遇到的问题，我的私信总是开放的，我们可以进行深入的讨论。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
Well, Kiriti and Ash, thank you so much for being here.
好吧，Kiriti 和 Ash，非常感谢你们来到这里。
Aishwarya Naresh Reganti: Thank you, Lenny.
艾西瓦娅·纳雷什·雷甘蒂：谢谢你，莱尼。
This was so much fun.
这太有趣了。
So much fun.
太有趣了。
Bye, everyone.: Thank you so much for listening.
再见，大家：非常感谢您的聆听。
If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.
如果您发现这很有价值，您可以在 Apple Podcasts、Spotify 或您最喜欢的播客应用程序上订阅该节目。
Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast.
另外，请考虑给我们评分或留下评论，因为这确实有助于其他听众找到播客。
You can find all past episodes or learn more about the show at lennyspodcast.com.
您可以在 lennyspodcast.com 上找到所有过去的剧集或了解有关该节目的更多信息。
See you in the next episode.
下一集见。