Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Lenny Rachitsky: To build great AI products, you need to be really good at building evals.
Lenny Rachitsky：要构建出色的人工智能产品，您需要非常擅长构建评估。
It's the highest ROI activity you can engage in.
这是您可以参与的投资回报率最高的活动。
Hamel Husain: This process is a lot of fun.
Hamel Husain：这个过程很有趣。
Everyone that does this immediately gets addicted to it.
每个这样做的人都会立即上瘾。
When you're building an AI application, you just learn a lot.
当你构建人工智能应用程序时，你会学到很多东西。
Lenny Rachitsky: What's cool about this is you don't need to do this many, many times.
Lenny Rachitsky：这很酷的一点是你不需要做很多很多次。
For most products, you do this process once and then you build on it.
对于大多数产品，您只需执行一次此过程，然后就可以在此基础上进行构建。
Shreya Shankar: The goal is not to do evals perfectly, it's to actionably improve your product.
Shreya Shankar：我们的目标不是完美地进行评估，而是切实可行地改进您的产品。
Lenny Rachitsky: I did not realize how much controversy and drama there is around evals.
Lenny Rachitsky：我没有意识到围绕评估存在多少争议和戏剧性。
There's a lot of people with very strong opinions.
有很多人有非常强烈的意见。
Shreya Shankar: People have been burned by evals in the past.
Shreya Shankar：人们过去曾被评估所困扰。
People have done evals badly, so then they didn't trust it anymore, and then they're like, "Oh, I'm anti evals."
人们的评估做得不好，所以他们不再相信它，然后他们会说，“哦，我反对评估。”
Lenny Rachitsky: What are a couple of the most common misconceptions people have with evals?
Lenny Rachitsky：人们对评估最常见的误解是什么？
Hamel Husain: The top one is, "We live in the age of AI.
Hamel Husain：最重要的是，“我们生活在人工智能时代。
Can't the AI just eval it?" But it doesn't work.
人工智能就不能评估它吗？”但这不起作用。
Lenny Rachitsky: A term that you used in your posts that I love is this idea of a benevolent dictator.
Lenny Rachitsky：我喜欢你在帖子中使用的一个术语，那就是仁慈的独裁者。
Hamel Husain: When you're doing this open coding, a lot of teams get bogged down in having a committee do this.
Hamel Husain：当你进行开放式编码时，很多团队都会陷入让委员会来做这件事的困境。
For a lot of situations, that's wholly unnecessary.
对于很多情况来说，这是完全没有必要的。
You don't want to make this process so expensive that you can't do it.
你不想让这个过程变得如此昂贵以至于你无法做到。
You can appoint one person whose taste that you trust.
您可以指定一位您信任其品味的人。
It should be the person with domain expertise.
它应该是具有领域专业知识的人。
Oftentimes, it is the product manager.
通常，这个人是产品经理。
Lenny Rachitsky: Today, my guests are Hamel Husain and Shreya Shankar.
Lenny Rachitsky：今天，我的客人是 Hamel Husain 和 Shreya Shankar。
One of the most trending topics on this podcast over the past year has been the rise of evals.
过去一年该播客最热门的话题之一是评估的兴起。
Both the chief product officers of Anthropic and OpenAI shared that evals are becoming the most important new skill for product builders.
Anthropic 和 OpenAI 的首席产品官都表示，评估正在成为产品构建者最重要的新技能。
And since then, this has been a recurring theme across many of the top AI builders I've had on.
从那时起，这一直是我接触过的许多顶级人工智能构建者中反复出现的主题。
Two years ago, I had never heard the term evals.
两年前，我从未听说过 evals 这个词。
Now it's coming up constantly.
现在它不断出现。
When was the last time that a new skill emerged that product builders had to get good at to be successful?
上一次出现产品开发者必须擅长才能成功的新技能是什么时候？
Lenny Rachitsky: Hamel and Shreya have played a major role in shifting evals from being an obscure, mysterious subject to one of the most necessary skills for AI product builders.
Lenny Rachitsky：Hamel 和 Shreya 在将评估从一个晦涩、神秘的主题转变为人工智能产品构建者最必要的技能之一方面发挥了重要作用。
They teach the definitive online course on evals, which happens to be the number one course on Maven.
他们教授有关 evals 的权威在线课程，这恰好是 Maven 上排名第一的课程。
They've now taught over 2,000 PMs and engineers across 500 companies, including large swaths of the OpenAI and Anthropic teams along with every other major AI lab.
他们现在已经为 500 家公司的 2,000 多名产品经理和工程师授课，其中包括大量 OpenAI 和 Anthropic 团队以及其他所有主要 AI 实验室。
Lenny Rachitsky: In this conversation, we do a lot of show versus tell.
Lenny Rachitsky：在这次对话中，我们做了很多表演而不是讲述。
We walk through the process of developing an effective eval, explain what the heck evals are and what they look like, address many of the major misconceptions with evals, give you the first few steps you can take to start building evals for your product, and also share just a ton of best practices that Hamel and Shreya have developed over the past few years.
我们将逐步介绍开发有效评估的过程，解释评估到底是什么以及它们是什么样子，解决对评估的许多主要误解，为您提供开始为产品构建评估的前几个步骤，并分享 Hamel 和 Shreya 在过去几年中开发的大量最佳实践。
This episode is the deepest yet most understandable primer you'll find on the world of evals.
本集是您在评估世界中找到的最深入但最容易理解的入门读物。
And honestly, it got me excited to write evals, even though I have nothing to write evals for.
老实说，写评估让我很兴奋，尽管我没有什么可写评估的。
I think you'll feel the same way as you watch this.
我想当你看到这个的时候也会有同样的感觉。
Lenny Rachitsky: If this conversation gets you excited, definitely check out Hamel and Shreya's course on Maven.
Lenny Rachitsky：如果这次谈话让您兴奋，一定要看看 Hamel 和 Shreya 的 Maven 课程。
We'll link to it in the show notes.
我们将在演出笔记中链接到它。
If you use the code LENNYSLIST when you purchase the course, you'll get 35% off the price of the course.
如果您在购买课程时使用代码 LENNYSLIST，您将获得课程价格 35% 的折扣。
With that, I bring you Hamel Husain and Shreya Shankar.
接下来，我为您带来哈梅尔·侯赛因 (Hamel Husain) 和什雷亚·香卡 (Shreya Shankar)。
Lenny Rachitsky: This episode is brought to you by Fin, the number one AI agent for customer service.
Lenny Rachitsky：本集由 Fin 为您带来，Fin 是第一大客户服务人工智能代理。
If your customer support tickets are piling up, then you need Fin.
如果您的客户支持请求堆积如山，那么您需要 Fin。
Fin is the highest-performing AI agent on the market with a 65% average resolution rate.
Fin 是市场上性能最高的 AI 代理，平均分辨率为 65%。
Fin resolves even the most complex customer queries.
Fin 甚至可以解决最复杂的客户查询。
No other AI agent performs better.
没有其他人工智能代理表现得更好。
In head-head bake-offs with competitors, Fin wins every time.
在与竞争对手的正面对决中，芬每次都获胜。
Yes, switching to a new tool can be scary, but Fin works on any help desk with no migration needed, which means you don't have to overhaul your current system or deal with delays in service for your customers.
是的，切换到新工具可能会令人恐惧，但 Fin 可以在任何帮助台上工作，无需迁移，这意味着您不必彻底检修当前系统或处理客户服务延迟的问题。
Lenny Rachitsky: And Fin is trusted by over 5,000 customer service leaders and top AI companies like Anthropic and Synthesia.
Lenny Rachitsky：Fin 受到超过 5,000 家客户服务领导者以及 Anthropic 和 Synthesia 等顶级人工智能公司的信任。
And because Fin is powered by the Fin AI engine, which is a continuously improving system that allows you to analyze, train, test, and deploy with ease, Fin can continuously improve your results too.
由于 Fin 由 Fin AI 引擎提供支持，这是一个不断改进的系统，可让您轻松分析、训练、测试和部署，因此 Fin 也可以不断改进您的结果。
So if you're ready to transform your customer service and scale your support, give Fin a try for only 99 cents per resolution.
因此，如果您准备好转变您的客户服务并扩大您的支持范围，不妨尝试一下 Fin，每个解决方案仅需 99 美分。
Plus, Fin comes with a 90-day money-back guarantee.
此外，Fin 还提供 90 天退款保证。
Find out how Fin can work for your team at fin.ai/lenny.
请访问 fin.ai/lenny，了解 Fin 如何为您的团队工作。
That's fin.ai/lenny.
那是 fin.ai/lenny。
Lenny Rachitsky: This episode is brought to you by Dscout.
Lenny Rachitsky：本集由 Dscout 为您带来。
Design teams today are expected to move fast, but also to get it right.
如今的设计团队不仅要快速行动，还要做到正确。
That's where Dscout comes in.
这就是 Dscout 发挥作用的地方。
Dscout is the all-in-one research platform built for modern product and design teams.
Dscout 是专为现代产品和设计团队构建的一体化研究平台。
Whether you're running usability tests, interviews, surveys, or in-the-wild fieldwork, Dscout makes it easy to connect with real users and get real insights fast.
无论您是进行可用性测试、访谈、调查还是野外实地工作，Dscout 都可以让您轻松与真实用户联系并快速获得真正的见解。
You can even test your Figma prototypes directly inside the platform.
您甚至可以直接在平台内测试 Figma 原型。
No juggling tools, no chasing ghost participants.
没有杂耍工具，没有追赶鬼魂的参与者。
And with the industry's most trusted panel plus AI-powered analysis, your team gets clarity and confidence to build better without slowing down.
借助业界最值得信赖的面板以及人工智能支持的分析，您的团队将获得清晰的思路和信心，从而在不放慢速度的情况下更好地构建。
So if you're ready to streamline your research, speed up decisions, and design with impact, head to dscout.com to learn more.
因此，如果您准备好简化研究、加快决策速度并进行具有影响力的设计，请前往 dscout.com 了解更多信息。
That's dscout.com.
那是 dscout.com。
The answers you need to move confidently.
您需要自信地行动的答案。
Hamel and Shreya, thank you so much for being here, and welcome to the podcast.
Hamel 和 Shreya，非常感谢你们来到这里，欢迎来到播客。
Lenny Rachitsky: I'm even more excited.
Lenny Rachitsky：我更兴奋。
Okay, so a couple years ago, I had never heard the term evals.
好吧，几年前，我从未听说过“评估”这个词。
Now it's one of the most trending topics on my podcast, essentially, that to build great AI products, you need to be really good at building evals.
现在，它是我的播客上最热门的主题之一，本质上来说，要构建出色的人工智能产品，您需要非常擅长构建评估。
Also, it turns out some of the fastest-growing companies in the world are basically building and selling and creating evals for AI labs.
此外，事实证明，世界上一些增长最快的公司基本上都是在为人工智能实验室构建、销售和创建评估。
I just had the CEO of Mercor on the podcast.
我刚刚在播客上听到了 Mercor 的首席执行官。
So there's something really big happening here.
所以这里发生了一件非常大的事情。
I want to use this conversation to basically help people understand this space deeply, but let's start with the basics.
我想通过这次对话基本上帮助人们深入了解这个领域，但让我们从基础知识开始。
Just what the heck are evals?
eval 到底是什么？
For folks that have no idea what we're talking about, give us just a quick understanding of what an eval is, and let's start with Hamel.
对于那些不知道我们在说什么的人，请让我们快速了解一下 eval 是什么，让我们从 Hamel 开始。
Hamel Husain: Sure.
哈梅尔·侯赛因：当然。
Evals is a way to systematically measure and improve an AI application, and it really doesn't have to be scary or unapproachable at all.
评估是一种系统地衡量和改进人工智能应用程序的方法，它实际上并不一定是可怕的或难以接近的。
It really is, at its core, data analytics on your LLM application and a systematic way of looking at that data, and where necessary, creating metrics around things so you can measure what's happening, and then so you can iterate and do experiments and improve.
从本质上讲，它实际上是对 LLM 申请进行数据分析，并以一种系统的方式查看该数据，并在必要时围绕事物创建指标，以便您可以衡量正在发生的情况，然后您可以迭代、进行实验和改进。
Lenny Rachitsky: So that's a really good broad way of thinking about it.
Lenny Rachitsky：所以这是一种非常好的、广泛的思考方式。
If you go one level deeper just to give people a very, even more concrete way of imagining and visualizing what we're talking about, even if you have a example to show would be even better, what's an even deeper way of understanding what an eval is?
如果你更深入一层只是为了给人们一种非常、更具体的方式来想象和可视化我们正在谈论的内容，即使你有一个例子来展示会更好，那么理解什么是评估的更深入的方式是什么？
Hamel Husain: Let's say you have a real estate assistant application and it's not working the way you want.
Hamel Husain：假设您有一个房地产助理应用程序，但它没有按照您想要的方式工作。
It's not writing emails to customers the way you want, or it's not calling the right tools, or any number of errors.
它没有按照您想要的方式向客户编写电子邮件，或者没有调用正确的工具，或者出现任何数量的错误。
And before evals, you would be left with guessing.
在评估之前，您只能猜测。
You would maybe fix a prompt and hope that you're not breaking anything else with that prompt, and you might rely on vibe checks, which is totally fine.
您可能会修复一个提示，并希望该提示不会破坏其他任何内容，并且您可能会依赖振动检查，这完全没问题。
Hamel Husain: And vibe checks are good and you should do vibe checks initially, but it can become very unmanageable very fast because as your application grows, it's really hard to rely on vibe checks.
Hamel Husain：振动检查很好，您应该首先进行振动检查，但它很快就会变得非常难以管理，因为随着应用程序的增长，真的很难依赖振动检查。
You just feel lost.
你只是感到失落。
And so evals help you create metrics that you can use to measure how your application is doing and kind of give you a way to improve your application with confidence.
因此，评估可以帮助您创建可用于衡量应用程序运行情况的指标，并为您提供一种充满信心地改进应用程序的方法。
That you have a feedback signal in which to iterate against.
您有一个可以迭代的反馈信号。
Lenny Rachitsky: So just to make very real, so imagining this real estate agent, maybe they're helping you book a listing or go see an open house.
Lenny Rachitsky：所以，为了变得非常真实，想象一下这位房地产经纪人，也许他们正在帮助您预订清单或去看开放日。
The idea here is you have this agent talking to people, it's answering questions, pointing them to things.
这里的想法是让这个代理与人们交谈，回答问题，向他们指出事情。
As a builder of that agent, how do you know if it's giving them good advice, good answers?
作为该代理的构建者，您如何知道它是否为他们提供了好的建议、好的答案？
Is it telling them things that are completely wrong?
它是否告诉他们完全错误的事情？
Lenny Rachitsky: So the idea of evals, essentially, is to build a set of tests that tell you, how often is this agent doing something wrong that you don't want it to do?
Lenny Rachitsky：所以评估的想法本质上是建立一组测试来告诉你，这个代理多久会做一些你不希望它做的错误事情？
And there's a bunch of ways you could define wrong.
有很多方法可能会导致错误的定义。
It could be just making up stuff.
这可能只是编造的东西。
It could be just answering in a really strange way.
它可能只是以一种非常奇怪的方式回答。
The way I think about evals, and tell me if this is wrong, just simply is like unit tests for code.
我思考评估的方式，并告诉我这是否错误，就像代码的单元测试一样。
You're smiling.
你在微笑。
You're like, "No, you idiot."
你会说：“不，你这个白痴。”
Shreya Shankar: No, that's not what I was thinking.
Shreya Shankar：不，我不是这么想的。
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
Okay, okay, tell me.
好吧好吧，告诉我吧。
Tell me, how does that feel as a metaphor?
告诉我，这个比喻感觉如何？
Shreya Shankar: Okay.
施瑞亚·香卡：好的。
I like what you said first, which is we had a very broad definition.
我喜欢你首先说的，那就是我们有一个非常广泛的定义。
Evals is a big spectrum of ways to measure application quality.
Evals 是衡量应用程序质量的多种方法。
Now, unit tests are one way of doing this.
现在，单元测试是实现此目的的一种方法。
Maybe there are some non-negotiable functionalities that you want your AI assistant to have, and unit tests are going to be able to check that.
也许你希望你的人工智能助手具备一些不可协商的功能，单元测试将能够检查这些功能。
Now, maybe you also, because these AI assistants are doing such open-ended tasks, you kind of also want to measure how good are they at very vague or ambiguous things like responding to new types of user requests or figuring out if there's new distributions of data like new users are coming and using your real estate agent that you didn't even know would use your product.
现在，也许你也一样，因为这些人工智能助手正在执行此类开放式任务，你还想衡量他们在非常模糊或模棱两可的事情上的表现如何，例如响应新类型的用户请求或弄清楚是否有新的数据分布，例如新用户正在到来并使用你的房地产经纪人，而你甚至不知道他们会使用你的产品。
And then all of a sudden, you think, "Oh, there's a different way you want to kind of accommodate this new group of people."
然后突然间，你想，“哦，你想用一种不同的方式来适应这群新人。”
Shreya Shankar: So evals could also be a way of looking at your data regularly to find these new cohorts of people.
Shreya Shankar：因此，评估也可以是定期查看数据以找到这些新人群的一种方式。
Evals could also be like metrics that you just want to track over time, like you want to track people saying, "Yes.
评估也可能像您只想随时间跟踪的指标，就像您想跟踪人们说“是的。
Thumbs up.
竖起大拇指。
I liked your message." You want very, very basic things that are not necessarily AI-related but can go back into this flywheel of improving your product.
我喜欢你的留言。”你想要非常非常基本的东西，这些东西不一定与人工智能相关，但可以回到改进产品的飞轮中。
So I would say, overall, unit tests are a very small part of that very big puzzle.
所以我想说，总的来说，单元测试只是这个巨大难题的一小部分。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
You guys actually brought an example of an eval just to show us exactly what the hell we're talking about.
你们实际上带来了一个评估的例子，只是为了向我们展示我们到底在说什么。
We're talking in these big ideas.
我们正在谈论这些伟大的想法。
So how about let's pull one up and show people, "Here's what an eval is."
那么，让我们拿出一个并向人们展示“这就是评估”怎么样？
Hamel Husain: Yeah, let me just set the stage for it a little bit.
Hamel Husain：是的，让我为它做一些准备。
So to echo what Shreya said, it's really important that we don't think of evals as just tests.
因此，呼应 Shreya 所说的，我们不要将评估视为只是测试，这一点非常重要。
There's a common trap that a lot of people fall into because they jump straight to the test like, "Let me write some tests," and usually that's not what you want to do.
很多人都会陷入一个常见的陷阱，因为他们直接跳到测试，比如“让我写一些测试”，而通常这不是您想要做的。
You should start with some kind of data analysis to ground what you should even test, and that's a little bit different than software engineering where you have a lot more expectations of how the system is going to work.
您应该从某种数据分析开始，为您应该测试的内容奠定基础，这与软件工程有点不同，在软件工程中，您对系统如何工作有更多的期望。
With LLMs, it's a lot more surface area.
对于法学硕士来说，它的表面积要大得多。
It's very stochastic, so you kind of have a different flavor here.
这是非常随机的，所以你在这里有一种不同的味道。
Hamel Husain: And so the example I'm going to show you today, it's actually a real estate example.
哈梅尔·侯赛因：所以我今天要向您展示的示例实际上是一个房地产示例。
It's a different kind of real estate example.
这是一个不同类型的房地产例子。
It's from a company called Nurture Boss.
它来自一家名为 Nurture Boss 的公司。
I can share my screen to show you their website just to help you understand this use case a little bit, so let me share my screen.
我可以分享我的屏幕来向您展示他们的网站，只是为了帮助您稍微了解一下这个用例，所以让我分享我的屏幕。
So this is a company that I worked with.
这是我曾经合作过的一家公司。
It's called Nurture Boss, and it is a AI assistant for property managers who are managing apartments, and it helps with various tasks such as inbound leads, customer service, booking appointments, so on and so forth.
它被称为 Nurture Boss，是管理公寓的物业经理的人工智能助手，它可以帮助完成各种任务，例如入站线索、客户服务、预约等等。
It's like all the different sort of operations you might be doing as a property manager, it helps you with that.
就像您作为物业经理可能要做的所有不同类型的操作一样，它可以帮助您。
And so you can see kind of what they do.
所以你可以看到他们做了什么。
It's a very good example because it has a lot of the complexities of a modern AI application.
这是一个非常好的例子，因为它具有现代人工智能应用程序的很多复杂性。
Hamel Husain: So there's lots of different channels that you can interact through the AI with like chat, text, voice, but also, there's tool calls, lots of tool calls for booking appointments, getting information about availability, so on and so forth.
Hamel Husain：因此，您可以通过人工智能进行多种不同的交互，例如聊天、文本、语音，而且还有工具调用，用于预约的大量工具调用、获取有关可用性的信息等等。
There's also RAG retrieval, getting information about customers and properties and things like that.
还有 RAG 检索，获取有关客户和财产等的信息。
So it's pretty fully fleshed in terms of an AI application.
所以它在人工智能应用方面非常充实。
And so they have been really generous with me in allowing me to use their data as a teaching example.
因此，他们非常慷慨地允许我使用他们的数据作为教学示例。
And so we have anonymized it, but what I'm going to walk through today is, okay, let's do the first part of how we would start to build evals for Nurture Boss.
所以我们已经将其匿名化，但我今天要讲的是，好吧，让我们完成如何开始为 Nurture Boss 构建评估的第一部分。
Why would we even want to do that?
我们为什么要这么做？
Hamel Husain: So let's go through the very beginning stage, what we call error analysis, which is, let's look at the data of their application and first start with what's going wrong.
Hamel Husain：所以让我们经历一下最开始的阶段，我们称之为错误分析，也就是说，让我们看看他们的应用程序的数据，然后首先从哪里出了问题开始。
So I'm going to jump to that next, and I'm going to open an observability tool.
所以我将跳到下一步，我将打开一个可观察性工具。
And you can use whatever you want here.
您可以在这里使用任何您想要的东西。
I just happen to have this data loaded in a tool called Braintrust, but you can load it in anything.
我只是碰巧将这些数据加载到一个名为 Braintrust 的工具中，但您可以将其加载到任何东西中。
We don't have a favorite tool or anything in the blog post that we wrote with you.
我们在与您一起撰写的博客文章中没有最喜欢的工具或任何内容。
We had the same example but in Phoenix Arize, and I think Aman, on your blog post, used Phoenix Arize as well.
我们有相同的例子，但在 Phoenix Arize，我认为 Aman 在您的博客文章中也使用了 Phoenix Arize。
And there's also LangSmith.
还有兰史密斯。
So these are kind of like different tools that you can use.
所以这些有点像您可以使用的不同工具。
Hamel Husain: So what you see here on the screen, this is logs from the application, and let me just show you how it looks.
Hamel Husain：所以您在屏幕上看到的是来自应用程序的日志，让我向您展示它的外观。
So what you see here is, and let me make it full screen, this is one particular interaction that a customer had with the Nurture Boss application, and what it is is a detailed log of everything that happened.
所以你在这里看到的是，让我把它全屏显示，这是客户与 Nurture Boss 应用程序进行的一次特殊交互，它是所发生的一切的详细日志。
So it's called a trace, and it's just the engineering term for logs of a sequence of events.
所以它被称为跟踪，它只是一系列事件日志的工程术语。
The concept of a trace has been around for a really long time, but it's especially really important when it comes to AI applications.
痕迹的概念已经存在很长时间了，但对于人工智能应用程序来说，它尤其重要。
Hamel Husain: And so we have all the different components and pieces and information that the AI needs to do its job, and we are logged all of it and we're looking at a view of that.
Hamel Husain：所以我们拥有人工智能完成其工作所需的所有不同组件、片段和信息，我们记录了所有这些，我们正在查看这些信息。
And so you see here a system prompt.
所以你在这里看到一个系统提示。
The system prompt says, "You are an AI assistant working as a leasing team member at Retreat at Acme Apartments." Remember, I said this is anonymized, so that's why the name is Acme Apartments.
系统提示：“您是一名人工智能助理，在Acme Apartments Retreat担任租赁团队成员。”请记住，我说过这是匿名的，所以这就是名称为 Acme Apartments 的原因。
"Your primary role is to respond to text messages from both current residents and prospective residents.
“您的主要职责是回复当前居民和潜在居民的短信。
Your goal is to provide accurate, helpful information," yada, yada, yada.
你的目标是提供准确、有用的信息，”yada，yada，yada。
And then there's a lot of detail around guidelines of how we want this thing to behave.
然后还有很多关于我们希望这个东西如何表现的指导方针的细节。
Lenny Rachitsky: Is this their actual system prompt, by the way, for this company?
Lenny Rachitsky：顺便问一下，这是他们针对这家公司的实际系统提示吗？
Lenny Rachitsky: That's amazing because it's rare you see a actual company product's system prompt.
Lenny Rachitsky：这太神奇了，因为你很少看到实际的公司产品的系统提示。
That's like their crown jewels a lot of times, so this is actually very cool on its own.
很多时候这就像他们皇冠上的宝石，所以这本身就非常酷。
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
Yeah, it's really cool.
是的，这真的很酷。
And you see all of these different sort of features that are different use cases, so things about tour scheduling, handling applications, guidance on how to talk to different personas, so on and so forth.
您会看到所有这些不同类型的功能都是不同的用例，例如关于旅行安排、处理应用程序、如何与不同角色交谈的指导等等。
And you can see the user just kind of jumps in here and asks, "Okay, do you have a one-bedroom with study available?
你可以看到用户跳到这里问：“好吧，你们有一间带书房的单卧室吗？
I saw it on virtual tours." And then you can see that the LLM calls some tools.
我在虚拟旅游中看到了它。”然后你可以看到LLM调用了一些工具。
It calls this get individual's information tool, and it pulls back that person's information.
它称之为获取个人信息工具，它会拉回该人的信息。
And then it gets the community's availability.
然后它获得社区的可用性。
So it's querying a database with the availability for that apartment complex.
因此，它正在查询数据库以了解该公寓大楼的可用性。
Hamel Husain: And then finally, the AI responds, "Hey, we have several one-bedroom apartments available, but none specifically listed with a study.
Hamel Husain：最后，人工智能回应，“嘿，我们有几套一居室公寓可供选择，但没有一个在研究中具体列出。
Here are a few options."
这里有一些选择。”
Hamel Husain: And then it says, "Can you let me know when one with a study is available?"
哈梅尔·侯赛因：然后它说：“当有一项研究可用时，你能让我知道吗？”
Hamel Husain: And then it says, "I currently don't have specific information on the availability of a one-bedroom apartment."
哈梅尔·侯赛因：然后它说，“我目前没有关于一卧室公寓的可用性的具体信息。”
User says, "Thank you.": And the AI says, "You're welcome.
用户说：“谢谢。”：人工智能说：“不客气。”
If you have any more questions, feel free to reach out." Now, this is an example of a trace, and we're looking at one specific data point.
如果您还有任何疑问，请随时与我们联系。”现在，这是一个跟踪示例，我们正在查看一个特定的数据点。
And so one thing that's really important to do when you're doing data analysis of your LLM application is to look at data.
因此，当您对 LLM 申请进行数据分析时，真正重要的一件事就是查看数据。
Now, you might wonder, "There's a lot of these logs.
现在，您可能想知道，“有很多这样的日志。
It's kind of messy.
有点乱。
There's a lot of things going on here.
这里发生了很多事情。
How in the hell are you supposed to look at this data?
你到底应该如何看待这些数据？
Do you want to just drown in this data?
您想淹没在这些数据中吗？
How do you even analyze this data?"
你如何分析这些数据？”
User says, "Thank you.": So it turns out there is a way to do it that is completely manageable, and it's not something that we invented.
用户说：“谢谢。”：事实证明，有一种完全可控的方法，而且这不是我们发明的。
It's been around in machine learning and data science for a really long time, and it's called error analysis.
它在机器学习和数据科学中已经存在很长时间了，它被称为错误分析。
And what you do is, the first step in conquering data like this is just to write notes.
你要做的就是，征服这样的数据的第一步就是写笔记。
Okay?
好的？
So you got to put your product hat on, which is why we're talking to you, because product people have to be in the room and they have to be involved in sort of doing this.
所以你必须戴上你的产品帽子，这就是我们与你交谈的原因，因为产品人员必须在房间里，他们必须参与其中。
Usually a developer is not suited to do this, especially if it's not a coding application.
通常开发人员不适合这样做，特别是如果它不是编码应用程序。
Lenny Rachitsky: And just to mirror back, why I think you're saying that is because this is the user experience of your product.
Lenny Rachitsky：回想一下，我认为你这么说的原因是因为这就是你产品的用户体验。
People talking to this agent is the entire product essentially, and so it makes sense for the product person to be super involved in this.
与该代理交谈的人本质上是整个产品，因此产品人员超级参与其中是有道理的。
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
So let's reflect on this conversation.
那么让我们反思一下这次谈话。
Okay, a user asked about availability.
好的，有用户询问可用性。
The AI said, "Oh, we don't really have that.
人工智能说：“哦，我们真的没有这个。
Have a nice day." Now, for a product that is helping you with lead management, is that good?
祝你有美好的一天。”现在，对于帮助您进行销售线索管理的产品来说，这好吗？
Do you feel like this is the way we want it to go?
你觉得这就是我们想要的方式吗？
Hamel Husain: Yes, not ideal, and I'm glad you said that.
哈梅尔·侯赛因：是的，并不理想，我很高兴你这么说。
A lot of people would say, "Oh, it's great.
很多人会说：“噢，太好了。
The AI did the right thing.
人工智能做了正确的事。
It looked, it said, 'We didn't have available,' and it's not available." But with your product hat on, you know that's not correct.
它看起来，它说，‘我们没有可用’，而且它不可用。”但当你戴上产品帽子后，你就会知道这是不正确的。
And so what you would do is you would just write a quick note here.
所以你要做的就是在这里写一个简短的注释。
You would say, "Okay." You might pop in here, and you can write a note.
你会说：“好吧。”你可能会来这里，你可以写一张便条。
So every observability application has ability to write notes, and you wouldn't try to figure out if something is wrong.
因此，每个可观察性应用程序都能够编写注释，并且您不会试图找出是否有问题。
In this case, it's kind of not doing the right thing, but you just write a quick note, "Should have handed off to a human."
在这种情况下，这有点没有做正确的事情，但你只需写一个简短的注释，“应该交给人类。”
Lenny Rachitsky: And as we watch this happening, it's like you mention this and you'll explain more.
Lenny Rachitsky：当我们看到这一切发生时，就好像你提到了这一点，然后你会解释更多。
You're doing this, this feels very manual and unscalable, but as you said, this is just one step of the process and there's a system to this.
你正在这样做，这感觉非常手动且不可扩展，但正如你所说，这只是该过程的一步，并且有一个系统。
That was just the first one.
那只是第一个。
Hamel Husain: Yeah, and you don't have to do it for all of your data.
Hamel Husain：是的，您不必对所有数据执行此操作。
You sample your data and just take a look, and it's surprising how much you learn when you do this.
您对数据进行采样并看一下，您会惊讶地发现这样做时您学到了很多东西。
Everyone that does this immediately gets addicted to it and they say, "This is the greatest thing that you can do when you're building an AI application." You just learn a lot and you're like, "Hmm, this is not how I want it to work.
每个这样做的人都会立即沉迷其中，他们说：“这是构建人工智能应用程序时可以做的最伟大的事情。”你只是学到了很多东西，然后你就会说，“嗯，这不是我想要的方式。
Okay." And so that's just an example.
好吧。”这只是一个例子。
Hamel Husain: So you write this note, and then we can go on to the next trace.
Hamel Husain：所以你写下这个注释，然后我们就可以继续下一个跟踪。
So this is the next trace.
这就是下一个痕迹。
I just pushed a hot key on my keyboard.
我刚刚按了键盘上的热键。
Let me go back to looking at it.
让我回去看看。
Lenny Rachitsky: And these tools make it easy to go through a bunch and add these notes quickly.
Lenny Rachitsky：这些工具可以轻松浏览一堆内容并快速添加这些注释。
Hamel Husain: Yes.
哈梅尔·侯赛因：是的。
And so this is another one.
这是另一件事。
Similar system prompt.
类似的系统提示。
We don't need to go through all of it again.
我们不需要再次经历这一切。
We'll just jump right into the user question.
我们将直接跳到用户问题。
"Okay, I've been texting you all day." Isn't that funny?
“好吧，我一整天都在给你发短信。”这不是很有趣吗？
And the user says, "Please." Okay, yeah, this one is just like an error in the application where this is a text message application, sorry, the channel through which the customer is communicating is through text message, and you're just getting really garbled.
用户说：“请。”好的，是的，这就像应用程序中的一个错误，这是一个短信应用程序，抱歉，客户通信的渠道是通过短信，而您只是得到了真正的乱码。
And you can see here that it kind of doesn't make sense.
你可以看到这里这有点没有意义。
The words are being cut off like, "In the meantime," and then the system doesn't know how to respond, because you know how people text message, they write short phrases.
这些词被切断，就像“与此同时”，然后系统不知道如何回应，因为你知道人们如何发短信，他们写简短的短语。
They split their sentence across four or five different turns.
他们将句子分成四到五个不同的轮次。
So in this case-
所以在这种情况下——
Lenny Rachitsky: Yeah, so what do you do with something like that?
Lenny Rachitsky：是的，那么你会怎么处理这样的事情呢？
Hamel Husain: Yeah, so this is a different kind of error.
Hamel Husain：是的，这是一种不同类型的错误。
Hamel Husain: This is more of, "Hey, we're not handling this interaction correctly.
Hamel Husain：这更多的是，“嘿，我们没有正确处理这种交互。
This is more of a technical problem," rather than, "Hey, the AI is not doing exactly what we want." So we would write that down too.
这更多的是一个技术问题”，而不是“嘿，人工智能并没有完全按照我们想要的方式做。”所以我们也会把它写下来。
Lenny Rachitsky: It's amazing you're catching that, too, here.
Lenny Rachitsky：令人惊奇的是你也在这里发现了这一点。
Otherwise, you'd have no idea this was happening.
否则，你不会知道这件事发生了。
Hamel Husain: Yeah, you might not know this is happening, right?
哈梅尔·侯赛因：是的，你可能不知道发生了这种事，对吧？
And so you would just say, "Okay." You would write a note like, "Oh, conversation flow is janky because of text message."
所以你只会说：“好吧。”您可以写一条注释，例如“哦，由于短信，对话流程很混乱。”
Lenny Rachitsky: And I like that, I like that you're using the word janky.
Lenny Rachitsky：我喜欢这样，我喜欢你使用“janky”这个词。
It shows you just how informal this can be at this stage.
它向您展示了现阶段这可以是多么非正式。
Hamel Husain: Yeah, it's supposed to be chill.
Hamel Husain：是的，应该很冷。
Just don't overthink it.
只是别想太多。
And there's a way to do this.
有一种方法可以做到这一点。
So the question always comes up, how do you do this?
所以问题总是出现，你如何做到这一点？
Do you try to find all the different problems in this trace?
您是否尝试在该跟踪中找到所有不同的问题？
What do you write a note about?
你写的笔记是关于什么的？
And the answer is, just write down the first thing that you see that's wrong, the most upstream error.
答案是，只需写下您看到的第一个错误，即最上游的错误。
Don't worry about all the errors, just capture the first thing that you see that's wrong, and stop, and move on.
不必担心所有错误，只需捕获您看到的第一个错误，然后停止并继续前进。
And you can get really good at this.
你可以在这方面做得非常好。
The first two or three can be very painful, but you can do a bunch of them really fast.
前两三个可能会非常痛苦，但你可以很快地完成其中几个。
Hamel Husain: So here's another one, and let's skip the system prompt again.
Hamel Husain：这是另一个，让我们再次跳过系统提示。
And the user asks, "Hey, I'm looking for a two- to three-bedroom with either one or two baths.
用户问道：“嘿，我正在寻找一间带一个或两个浴室的两到三间卧室。
Do you provide virtual tours?"
你们提供虚拟旅游吗？”
Hamel Husain: And a bunch of tools are called and it says, "Hi Sarah.
Hamel Husain：调用了一堆工具，上面写着：“嗨，莎拉。
Currently, we have three-bedroom, two-and-a-half-bathroom apartment available for $2,175.
目前，我们有三间卧室、两间半浴室的公寓，售价为 2,175 美元。
Unfortunately, we don't have any two-bedroom options at the moment.
不幸的是，我们目前没有任何两居室的选择。
We do offer virtual tours.
我们确实提供虚拟旅游。
You can schedule a tour," blah, blah.
你可以安排一次旅行，”等等，等等。
It just so happens that there is no virtual tour, right?
正好没有虚拟游览吧？
Hamel Husain: So it is hallucinating something that doesn't exist.
哈梅尔·侯赛因：所以这是一种不存在的幻觉。
Then you kind of have to bring your context as an engineer, or even product content, and say, "Hey, this is kind of weird.
然后你必须带上你作为工程师的背景，甚至产品内容，然后说：“嘿，这有点奇怪。
We shouldn't be telling a person about virtual tour when it's not offered."
当没有提供虚拟旅游时，我们不应该告诉别人。”
Hamel Husain: So you would say, "Okay, offered virtual tour," and you just write the note.
哈梅尔·侯赛因：所以你会说，“好吧，提供虚拟旅游”，然后你只需写下便条即可。
So you can see there's a diversity of different kinds of errors that we're seeing, and we're actually learning a lot about your application in a very short amount of time.
因此，您可以看到我们看到了各种不同类型的错误，并且我们实际上在很短的时间内了解了有关您的应用程序的很多信息。
Shreya Shankar: One common question that we get from people at this stage is, "Okay, I understand what's going on.
Shreya Shankar：现阶段我们从人们那里得到的一个常见问题是，“好吧，我明白发生了什么。
Can I ask an LLM to do this process for me?"
我可以请法学硕士为我完成这个过程吗？”
Shreya Shankar: And I loved Hamel's most recent example because what we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be bad product smell or not.
Shreya Shankar：我喜欢 Hamel 最近的例子，因为当我们尝试要求法学硕士进行错误分析时，我们通常会发现它只是说跟踪看起来不错，因为它没有了解某些东西是否可能是不良产品气味所需的上下文。
For example, the hallucination about scheduling the tour, right?
比如关于安排行程的幻觉吧？
I can guarantee you, I would bet money on this, if I put that into chat GPT and asked, "Is there an error?" it would say, "No, did a great job."
我可以向你保证，如果我将其放入聊天 GPT 中并询问“有错误吗？”，我会为此打赌。它会说：“不，做得很好。”
Shreya Shankar: But Hamel had the context of knowing, "Oh, we don't actually have this virtual tour functionality," right?
Shreya Shankar：但是哈梅尔知道，“哦，我们实际上没有这种虚拟旅游功能”，对吗？
So I think, in these cases, it's so important to make sure you are manually doing this yourself.
所以我认为，在这些情况下，确保您自己手动执行此操作非常重要。
And we can talk a little bit more about when to use LLMs in the process later, but number one pitfall right here is people are like, "Let me automate this with an LLM."
我们可以稍后再讨论何时在这个过程中使用法学硕士，但这里的第一个陷阱是人们会说，“让我用法学硕士来自动化这一点。”
Lenny Rachitsky: Do you think we'll get to a place where an agent can do this, where it has that context?
Lenny Rachitsky：你认为我们会达到一个代理可以做到这一点的地方吗？它有这样的背景吗？
Shreya Shankar: Oh, no. No, no, no. Sorry.
施瑞亚·香卡：哦，不。不，不，不。对不起。
There are parts of error analysis that an LLM is suited for, which we could talk about later in this podcast.
法学硕士适合错误分析的某些部分，我们可以在本播客的后面讨论。
But right now, in this stage of free form, note-taking is not the place for an LLM.
但目前，在这个自由形式的阶段，记笔记并不是法学硕士的用武之地。
Lenny Rachitsky: Got it.
莱尼·拉奇茨基：明白了。
And this is something you call open coding, this step?
这就是你所说的开放编码，这一步？
Lenny Rachitsky: Cool.
莱尼·拉奇茨基：酷。
Another term that you used in your posts that I love and that fits into this step is this idea of a benevolent dictator.
您在帖子中使用的另一个我喜欢且适合这一步的术语是仁慈的独裁者的想法。
Maybe just talk about what that is, and maybe, Shreya, cover that.
也许只是谈谈那是什么，也许，Shreya，涵盖这一点。
Shreya Shankar: Yeah, so Hamel actually came up with this term.
Shreya Shankar：是的，哈梅尔实际上想出了这个术语。
Lenny Rachitsky: Okay, maybe Hamel cover that, actually.
Lenny Rachitsky：好吧，也许哈梅尔涵盖了这一点，实际上。
Hamel Husain: No problem.
哈梅尔·侯赛因：没问题。
And we'll actually show the LLM automation in this example, because we're going to take this example, we're going to go all the way through.
我们将在这个例子中实际展示法学硕士自动化，因为我们将采取这个例子，我们将一路走完。
Hamel Husain: And so benevolent dictator is just a catchy term for the fact that when you're doing this open coding, a lot of teams get bogged down in having a committee do this.
Hamel Husain：仁慈的独裁者只是一个朗朗上口的术语，指的是当你进行开放编码时，很多团队都会陷入让委员会来做这件事的困境。
And for a lot of situations, that's wholly unnecessary.
在很多情况下，这是完全没有必要的。
People get really uncomfortable with, "Okay, we want everybody on board.
人们对“好吧，我们希望每个人都参与进来”感到非常不舒服。
We want everybody involved," so on and so forth.
我们希望每个人都参与其中，”等等。
You need to cut through the noise.
你需要消除噪音。
And a lot of organizations, if you look really deeply, especially small, medium-sized companies, you can appoint one person whose tastes that you trust.
很多组织，如果你看得很深入，特别是中小型公司，你可以任命一个你信任的品味的人。
And you can do this with a small number of people and often one person, and it's really important to make this tractable.
您可以与少数人（通常是一个人）一起完成此操作，并且使此操作易于处理非常重要。
You don't want to make this process so expensive that you can't do it.
你不想让这个过程变得如此昂贵以至于你无法做到。
You're going to lose out.
你会输的。
Hamel Husain: So that's the idea behind benevolent dictator, is, "Hey, you need to simplify this across as many dimensions as you can." Another thing that we'll talk about later is when it goes to building an LLM as a judge, you need a binary score.
哈梅尔·侯赛因：这就是仁慈的独裁者背后的想法，“嘿，你需要在尽可能多的维度上简化它。”我们稍后将讨论的另一件事是，当作为法官建立法学硕士时，您需要一个二进制分数。
You don't want to think about, "Is this like a 1, 2, 3, 4, 5?" Like, assign a score to it.
你不会想“这像 1、2、3、4、5 吗？”比如，给它打分。
You can't.
你不能。
That's going to slow it down.
这会减慢它的速度。
Lenny Rachitsky: Just to make sure this benevolent dictator point is really clear, basically, this is the person that-
莱尼·拉奇茨基：只是为了确保这个仁慈的独裁者的观点真的很清楚，基本上，这个人是-
Lenny Rachitsky: Make sure this benevolent dictator point is really clear.
Lenny Rachitsky：确保这个仁慈的独裁者的观点非常明确。
Basically, this is the person that does this note-taking, and ideally they're the expert on the stuff.
基本上，这是做笔记的人，理想情况下他们是这方面的专家。
So if it's law stuff, maybe there's a legal person that owns this, it could be a product manager.
因此，如果它是法律方面的东西，也许有一个法人拥有它，它可能是产品经理。
Give us advice on who this person should be?
请给我们建议这个人应该是谁？
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
It should be the person with domain expertise.
它应该是具有领域专业知识的人。
So in this case, it would be the person who understands the business of leasing, apartment leasing, and has context to understand if this makes sense.
所以在这种情况下，这个人应该了解租赁业务、公寓租赁，并且有背景来理解这是否有意义。
It's always a domain expert, like you said.
正如您所说，它始终是领域专家。
Okay.
好的。
For legal, it would be a law person.
对于法律来说，这将是一个法律人。
For mental health, it would be the mental health expert, whether that's a psychiatrist or someone else.
对于心理健康，这将是心理健康专家，无论是精神科医生还是其他人。
Hamel Husain: Though oftentimes, it is the product manager.
Hamel Husain：虽然很多时候是产品经理。
Lenny Rachitsky: Cool.
莱尼·拉奇茨基：酷。
So the advice here is pick that person.
所以这里的建议是选择那个人。
It may not feel so super fair that they're the one in charge and they're the dictator, but they're benevolent.
他们是负责人和独裁者，这可能感觉不太公平，但他们是仁慈的。
It's going to be okay.
一切都会好起来的。
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
It's going to be okay.
一切都会好起来的。
It's not perfection.
这不是完美的。
You're just trying to make progress and get signal quickly so you have an idea of what to work on because it can become infinitely expensive if you're not careful.
你只是想取得进展并快速获得信号，这样你就知道要做什么，因为如果你不小心，它可能会变得无限昂贵。
Lenny Rachitsky: Yeah.
莱尼·拉奇茨基：是的。
Okay, cool.
好吧，酷。
Let's go back to your examples.
让我们回到你的例子。
Hamel Husain: Yeah, no problem.
哈梅尔·侯赛因：是的，没问题。
So this is another example where we have someone saying, "Okay.
这是另一个例子，有人说：“好吧。
Do you have any specials?" And the assistant or the AI responds, "Hey, we have a 5% military discount." User responds, and it switches the subject, "Can you tell me how many floors there are?
你们有什么特价菜吗？”助理或人工智能回答说：“嘿，我们有 5% 的军事折扣。”用户做出回应，然后话题就转移了：“你能告诉我有多少层楼吗？
Do you have any one-bedrooms available or one-bedrooms on the first floor?" And the AI responds, "Yeah, okay.
请问还有一楼的单间房或者一楼的单间房吗？”人工智能回答说：“是的，好吧。
We have several one-bedroom apartments available." And then the user wants to confirm, "Any of those on the first floor and how much are the one-bedrooms?" And then also, it's a current resident, so they're also asking, "I need a maintenance request."
我们有几套一卧室公寓可供选择。”然后用户想确认一下，“一楼有没有，一居室多少钱？”而且，它是当前居民，所以他们也会问：“我需要维护请求。”
Hamel Husain: You could see the messiness of the real world in here, and the assistant just calls a tool that says transfer call, but it doesn't say anything.
Hamel Husain：在这里你可以看到现实世界的混乱，助手只是调用了一个说转接电话的工具，但它什么也没说。
It just abruptly does transfer call, so it's pretty jank, I would say.
它只是突然进行转接呼叫，所以我想说，这是相当卡顿的。
It's just not-
这只是不是——
Hamel Husain: Another kind of jank, a different kind of jank.
Hamel Husain：另一种卡顿，一种不同类型的卡顿。
So when you write the open note, you don't want to say jank, because what we want to do is we want to understand, and when we look at the notes later on, we want to understand what happened.
所以当你写打开的笔记时，你不想说jank，因为我们想做的是我们想要理解，当我们稍后查看笔记时，我们想要了解发生了什么。
Hamel Husain: So you just want to say, "Did not confirm call transfer with user." And it doesn't have to be perfect.
Hamel Husain：所以你只想说，“没有与用户确认呼叫转移。”而且它不一定是完美的。
You just have to have a general idea of what's going on.
您只需要对正在发生的事情有一个大概的了解。
Hamel Husain: So, okay.
哈梅尔·侯赛因：所以，好吧。
So let's say we do, and Shreya and I, we recommend doing at least 100 of these.
假设我们这样做了，Shreya 和我建议至少做 100 次。
The question is always, "How many of this do you do?" And so there's not a magic number.
问题始终是：“你做了多少这样的事？”所以不存在一个神奇的数字。
We say 100 just because we know that as soon as you start doing this, once you do 20 of these, you will automatically find it so useful that you will continue doing it.
我们说 100 个只是因为我们知道，一旦你开始这样做，一旦你做了其中 20 个，你就会自动发现它非常有用，以至于你会继续这样做。
Hamel Husain: So we just say 100 to mentally unblock you, so it's not intimidating.
Hamel Husain：所以我们只是说 100 来在精神上解锁你，所以这并不吓人。
It's like, "Don't worry, you're only going to do 100." And there is a term for that, so the right answer is, "Keep looking at traces until you feel like you're not learning anything new." Maybe Shreya should talk about-
就像是，“别担心，你只会做 100 次。”有一个术语来形容这一点，所以正确的答案是“不断查看痕迹，直到你觉得自己没有学到任何新东西”。也许施瑞亚应该谈谈——
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
So there's actually a term-
所以其实有一个词——
Shreya Shankar: ...
施瑞亚·香卡：...
in data analysis and qualitative analysis called theoretical saturation.
在数据分析和定性分析中称为理论饱和。
So what this means is when you do all of these processes of looking at your data, when do you stop?
那么这意味着当您执行所有这些查看数据的过程时，什么时候停止？
It's when you are theoretically saturating or you're not uncovering any new types of notes, new types of concepts, or nothing that will materially change the next part of your process.
当你在理论上已经饱和，或者你没有发现任何新类型的笔记、新类型的概念，或者没有任何东西会实质性地改变你的过程的下一部分时。
Shreya Shankar: And this kind of takes a little bit of intuition to develop, so typically, people don't really know when they've reached theoretical saturation yet.
Shreya Shankar：这种需要一点直觉来发展，所以通常情况下，人们并不真正知道他们何时达到理论饱和。
That's totally fine.
那完全没问题。
When you do two or three examples or rounds of this, you will develop the intuition.
当你做两三个例子或轮次时，你就会培养直觉。
A lot of people realize, "Oh, okay.
很多人意识到，“哦，好吧。
I only need to do 40, I only need to do 60.
我只需要做40个，我只需要做60个。
Actually, I only need to do 15." I don't know.
事实上，我只需要做15个。”我不知道。
Depends on the application and depends on how savvy you are with error analysis for sure.
取决于应用程序，也取决于您对错误分析的熟练程度。
Lenny Rachitsky: And your point about you're going to want to do a bunch.
Lenny Rachitsky：你的观点是你想要做很多事情。
I imagine it's because you're just like, "Oh, I'm discovering all these problems.
我想这是因为你就像，“哦，我发现了所有这些问题。
I got to see what else is going on here."
我得看看这里还发生了什么。”
Shreya Shankar: And promise, at some point, you're not going to discover new types of problems.
Shreya Shankar：并且保证，在某些时候，你不会发现新类型的问题。
Lenny Rachitsky: Yeah.
莱尼·拉奇茨基：是的。
Awesome.
惊人的。
So let's say you did 100 of these, what's the next step?
假设您已经完成了 100 个，下一步是什么？
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
Okay.
好的。
So you did 100 of these.
所以你做了 100 个。
Now you have all these notes.
现在你已经有了所有这些笔记。
So this is where you can start using AI to help you.
因此，您可以从这里开始使用人工智能来帮助您。
So the part where you looked at this data is important, like we discussed.
因此，正如我们所讨论的，您查看这些数据的部分很重要。
You don't want to automate this part too much.
您不想让这部分自动化太多。
Lenny Rachitsky: Humans will still have jobs.
Lenny Rachitsky：人类仍然会有工作。
This is the takeaway here.
这是这里的要点。
That's great.
那太棒了。
Lenny Rachitsky: Just reviewing traces.
Lenny Rachitsky：只是检查痕迹。
At least there's one job left for now.
至少现在还剩下一份工作。
Great.
伟大的。
Hamel Husain: So, yeah.
哈梅尔·侯赛因：所以，是的。
Exactly.
确切地。
And so, okay.
所以，好吧。
You have all these notes.
你有所有这些笔记。
Now, to turn this into something useful, you can do basic counting.
现在，要将其变成有用的东西，您可以进行基本的计数。
So basic counting is the most powerful analytical technique in data science because it's so simple and it's kind of undervalued in many cases, and so it's very approachable for people.
因此，基本计数是数据科学中最强大的分析技术，因为它非常简单，而且在很多情况下都被低估了，所以它对人们来说非常平易近人。
Hamel Husain: And so the first thing you want to do is take these notes, and you can categorize them with an LLM, and so there's a lot of different ways to do that.
Hamel Husain：所以你要做的第一件事就是记下这些笔记，你可以用法学硕士对它们进行分类，所以有很多不同的方法可以做到这一点。
Right before this podcast, I took three different coding agents or AI tools in how to categorize these notes.
就在这个播客之前，我使用了三种不同的编码代理或人工智能工具来对这些笔记进行分类。
So one is, "Okay, I uploaded into a cloud project, I uploaded a CSV of these notes, and I just exported them directly from this interface." There's a lot of different ways to do this, but I'm showing you the simple, stupid way, the most basic way of doing things.
所以一个是，“好吧，我上传到一个云项目，我上传了这些笔记的 CSV，然后我直接从这个界面导出它们。”有很多不同的方法可以做到这一点，但我向您展示一种简单、愚蠢的方法，也是最基本的做事方法。
Hamel Husain: And so I dumped the CSV in here and I said, "Please analyze the following CSV file." And I told it there's a metadata field that has a note in it, but what I said is I used the word open codes, and I said, "Hey, I have different open codes," and that's a term of art.
Hamel Husain：所以我将 CSV 转储到这里，然后说：“请分析以下 CSV 文件。”我告诉它有一个元数据字段，里面有一个注释，但我所说的是我使用了开放代码这个词，我说，“嘿，我有不同的开放代码”，这是一个艺术术语。
LLMs know what open codes are and they know what axial codes are because it is a concept that's been around for a really long time, so those words help me shortcut what I'm trying to do.
法学硕士知道什么是开放代码，他们知道什么是轴向代码，因为这是一个已经存在很长时间的概念，所以这些词帮助我简化了我想要做的事情。
Lenny Rachitsky: That's awesome.
莱尼·拉奇茨基：太棒了。
And the end of the prompt is telling it to create axial codes?
提示的最后是告诉它创建轴向代码？
Hamel Husain: Yes.
哈梅尔·侯赛因：是的。
Creating axial codes, so what it does is-
创建轴向代码，所以它的作用是-
Shreya Shankar: So maybe it's worth talking about what are axial codes or what's the point here?
Shreya Shankar：所以也许值得讨论一下什么是轴向代码或者这里有什么意义？
You have a mess of open codes, and you don't have 100 distinct problems.
你有一堆乱七八糟的开放代码，而且没有 100 个不同的问题。
Actually, many of them are repeats, but because you phrased them differently, and that you shouldn't have tried to create your taxonomy of failures as you're open coding.
实际上，其中许多都是重复的，但是因为您对它们的措辞不同，并且您不应该在开放编码时尝试创建失败分类法。
You just want to get down what's wrong and then organize, "Okay, what's the most common failure mode?"
你只想找出问题所在，然后进行组织，“好吧，最常见的故障模式是什么？”
Shreya Shankar: So the purpose, axial code basically is just a failure mode.
Shreya Shankar：因此，轴向代码的目的基本上只是一种故障模式。
It's the label or category.
这是标签或类别。
And what our goal is, is to get to this clusters of failure modes and figure out what is the most prevalent, so then you can go and run and attack that problem.
我们的目标是找到这些故障模式集群并找出最普遍的故障模式，然后您就可以开始运行并解决该问题。
Lenny Rachitsky: That is really helpful.
Lenny Rachitsky：这真的很有帮助。
Basically, just synthesizing all these-
基本上，只是综合所有这些-
Lenny Rachitsky: ...
莱尼·拉奇茨基：……
into categories and themes.
分为类别和主题。
Super cool.
超级酷。
And we'll include this prompt in our show notes for folks so they don't have to sit there and screenshot it and try to type it up themselves.
我们将在为人们提供的演示笔记中包含此提示，这样他们就不必坐在那里截屏并尝试自己输入。
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
Great idea.
好主意。
And so Claude went ahead and analyzed the CSV file and decided how to parse it, blah, blah, blah.
于是 Claude 继续分析 CSV 文件并决定如何解析它，等等，等等。
We don't need to worry about all that stuff, but it came up with a bunch of axial codes.
我们不需要担心所有这些东西，但它提出了一堆轴向代码。
Basically, axial codes are categories, like Shreya said.
基本上，轴向代码是类别，就像 Shreya 所说的那样。
So one is, okay, capability limitations, misrepresentation, process and protocol violations, human handoff issues, communication, quality.
因此，其中之一就是能力限制、歪曲事实、违反流程和协议、人员交接问题、沟通、质量。
It created these categories.
它创建了这些类别。
Hamel Husain: Now, do I like all the categories?
Hamel Husain：现在，我喜欢所有类别吗？
Not really.
并不真地。
I like some of them.
我喜欢其中一些。
It's a good first stab at it.
这是一个很好的第一次尝试。
I would probably rename it a little bit because some of them are a bit too generic.
我可能会稍微重命名它，因为其中一些有点太通用了。
Like what is capability limitation?
比如什么是能力限制？
That's a little bit too broad.
这有点太宽泛了。
It's not actionable.
这是不可行的。
I want to get a little bit more actionable with it so that if I do decide it's a problem, I know what to do with it, but we'll discuss that in a little bit.
我想让它更具可操作性，这样如果我确实认为这是一个问题，我知道该怎么处理它，但我们稍后会讨论这个问题。
So you can do this with anything, and this is the dumbest way to do it, but dumb sometimes is a good way to get started, so-
所以你可以用任何东西来做这件事，这是最愚蠢的方法，但愚蠢有时是一个很好的开始方式，所以-
Lenny Rachitsky: And this is what LLMS are really good at, taking a bunch of information and synthesizing it.
Lenny Rachitsky：这就是 LLMS 真正擅长的，获取大量信息并进行综合。
Shreya Shankar: Absolutely.
施瑞亚·香卡：当然。
Synthesizing for us to make sense of, right?
综合起来让我们理解，对吗？
Note that it's not automatically proposing fixes or anything, that's our job, but now, we can wade through this mess of open codes a lot easier.
请注意，它不会自动提出修复或任何建议，这是我们的工作，但现在，我们可以更轻松地浏览这些混乱的开放代码。
Shreya Shankar: Another thing that's interesting here in this prompt to generate the axial codes is you can be very detailed if you want, right?
Shreya Shankar：在生成轴向代码的提示中，另一件有趣的事情是，如果您愿意，您可以非常详细，对吗？
You can say, "I want each axial code to actually be some actionable failure mode," and maybe the LLM will understand that and propose it, or, "I want you to group these open codes by what stage of the user story that it's in." So this is where you can be creative or do what's best for you as a product manager or engineer working on this, and that will help you do the improvement later.
你可以说，“我希望每个轴向代码实际上都是某种可操作的故障模式”，也许法学硕士会理解这一点并提出它，或者，“我希望你按照用户故事所处的阶段对这些开放代码进行分组。”因此，作为产品经理或工程师，您可以在这里发挥创造力，或者做最适合您的事情，这将帮助您以后进行改进。
Lenny Rachitsky: So there's no definitive prompt of, "Here's the one way to do it"?
Lenny Rachitsky：所以没有明确的提示“这是唯一的方法”？
Lenny Rachitsky: You're saying you can iterate, see what works for you?
Lenny Rachitsky：你是说你可以迭代，看看什么对你有用？
Lenny Rachitsky: It's interesting the tools don't do this, or do they try and they just don't do a great job?
Lenny Rachitsky：有趣的是，这些工具并没有做到这一点，或者它们尝试过但只是没有做得很好？
Shreya Shankar: No, I don't think they do it.
Shreya Shankar：不，我不认为他们这样做。
We've been screaming from the rooftops, "Please, please-"
我们一直在屋顶上尖叫，“求你了，求你了——”
Shreya Shankar: "...
施瑞亚·香卡：“……
do this." I do think it's a little bit hard, right?
做这个。”我确实觉得有点难，对吧？
Part of this whole experience with the eval scores Hamel and I are teaching are a lot of people don't actually know this, so maybe it's that people don't know this and they don't know how to build tools for it.
哈梅尔和我正在教授的评估分数的整个体验的一部分是很多人实际上并不知道这一点，所以也许人们不知道这一点，并且他们不知道如何为其构建工具。
And hopefully, we can demystify some of this magic.
希望我们能够揭开其中一些魔力的神秘面纱。
Lenny Rachitsky: And just to double-click on this point, this is not a thing everyone does or knows.
Lenny Rachitsky：双击这一点，这不是每个人都会做或知道的事情。
This is something you two developed based on your experience doing data analysis and data science at other companies?
这是你们两人根据在其他公司进行数据分析和数据科学的经验开发的吗？
Shreya Shankar: Well, I want to caveat that we didn't invent error analysis.
Shreya Shankar：嗯，我想声明一下，我们并没有发明错误分析。
We don't actually want to invent things.
我们实际上并不想发明东西。
That's bad signal.
这信号不好啊
If somebody is coming to you with a way to do something that's entirely new and not grounded in hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that.
如果有人向你提出一种全新的方法，并且没有建立在数百年的理论和文献基础上，那么我不知道，你应该对此保持警惕。
Shreya Shankar: But what we tried to do was distill, "Okay, what are the new tools and techniques that you need to make sense of the LLM error-out analysis?" And then we created a curriculum or structured way of doing this.
Shreya Shankar：但我们试图做的是提炼，“好吧，你需要哪些新工具和技术来理解法学硕士错误分析？”然后我们创建了一个课程或结构化的方法来做到这一点。
So this is all very tailored to LLMs, but the terms open coding, axial coding, are grounded in social science.
因此，这一切都是为法学硕士量身定制的，但开放编码、轴向编码等术语都是以社会科学为基础的。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
Okay.
好的。
What's funny about you guys doing this is I just want to go do this somewhere.
你们这样做的有趣之处在于我只是想去某个地方做这件事。
I don't have any AI product to do this on, but it's just like, "Oh, this would be so fun." Just sit there and find all the problems I'm running into and categorize them and then try to fix them.
我没有任何人工智能产品可以做到这一点，但它就像，“哦，这会很有趣。”只需坐在那里，找到我遇到的所有问题并对它们进行分类，然后尝试解决它们。
Lenny Rachitsky: Hamel pulled up a video.
莱尼·拉奇茨基：哈梅尔调出了一段视频。
What do you got going on here?
你在这里发生什么事？
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
So I pulled up a video just to drive home Shreya's point.
所以我拿出了一段视频来阐明 Shreya 的观点。
We are not inventing anything, so what you see on the screen here is Andrew Ng, one of the famous machine learning researchers in the world who have taught a lot of people, frankly, machine learning.
我们并没有发明任何东西，所以你在屏幕上看到的是吴恩达（Andrew Ng），他是世界上著名的机器学习研究人员之一，坦率地说，他教过很多人机器学习。
And you can see this is an eight-year-old video, and he's talking about error analysis.
你可以看到这是一个八年前的视频，他正在谈论错误分析。
Hamel Husain: And so this is a technique that's been used to analyze stochastic systems for ages, and it's something that it was just using the same machine learning ideas and principles, just bringing them into here, because again, these are stochastic systems.
Hamel Husain：这是一种多年来一直用于分析随机系统的技术，它只是使用相同的机器学习思想和原理，只是将它们带入这里，因为同样，这些都是随机系统。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
Well, one thing, we're working on getting Andrew on the podcast, we're chatting, so that will-
嗯，有一件事，我们正在努力让安德鲁加入播客，我们正在聊天，所以会-
Lenny Rachitsky: ...
莱尼·拉奇茨基：……
be really fun.
真的很有趣。
Two, I love that my podcast episode just came out today is in your feed there, and it's standing out really well in that feed, so I'm really happy about that .
第二，我喜欢我今天刚刚发布的播客剧集出现在您的提要中，并且它在该提要中非常突出，所以我对此感到非常高兴。
Hamel Husain: Very nice.
哈梅尔·侯赛因：非常好。
Yeah.
是的。
The recommendation algorithm is quite good.
推荐算法相当不错。
Lenny Rachitsky: Yes.
莱尼·拉奇茨基：是的。
Here we go.
开始了。
Hope you click on that.
希望你点击它。
Don't screw my algorithm.
别搞砸我的算法。
Okay, cool.
好吧，酷。
So we've done some synthesis.
所以我们做了一些综合。
I know we're not going to go through the entire step.
我知道我们不会完成整个步骤。
This is you have a whole course that takes many days to learn this whole process.
这是一门完整的课程，需要很多天才能学习整个过程。
What else do you want to share about how to go about this process?
关于如何进行这个过程，您还有什么想分享的吗？
Hamel Husain: Okay.
哈梅尔·侯赛因：好的。
So you can do this through anything, and the same thing works just fine in ChatGPT, the same exact prompt.
所以你可以通过任何方式做到这一点，同样的事情在 ChatGPT 中也可以正常工作，同样的提示。
You can see it made axial codes.
可以看到它制作了轴向代码。
I really like using Julius AI.
我真的很喜欢使用 Julius AI。
It's one of my favorite tools.
这是我最喜欢的工具之一。
Hamel Husain: Julius is kind of this third-party tool that uses notebooks.
Hamel Husain：Julius 是一种使用笔记本的第三方工具。
I personally like Jupiter notebooks a lot, and so it's more of a data science thing, but a lot of product managers that are kind of learning notebooks nowadays, and it's kind of cool.
我个人非常喜欢 Jupiter 笔记本，所以它更多的是数据科学的东西，但现在很多产品经理都在学习笔记本，这很酷。
It's like a fun playground where you can write code and look at data.
它就像一个有趣的游乐场，您可以在其中编写代码并查看数据。
But we don't have to go deeply into that.
但我们不必深入探讨这一点。
Just wanted to mention, you can use a lot.
只是想提一下，你可以使用很多。
AI is really good at this.
AI在这方面确实很擅长。
Hamel Husain: So let's go to the fun part.
Hamel Husain：那么让我们来看看有趣的部分吧。
Here we go.
开始了。
So now we have these axial codes.
现在我们有了这些轴向代码。
So the first thing I like to do, I have these open codes, and I have the axial codes, let's say, that we assigned from the cloud project or the ChatGPT.
所以我喜欢做的第一件事是，我有这些开放代码，还有我们从云项目或 ChatGPT 分配的轴向代码。
And so what I do is I collect them first and I take a look, like, "Does these axial codes make sense?" And I look at the correspondence between the different axial codes and the open codes, and I go through an exercise and I say, "Hmm.
所以我要做的就是先收集它们，然后看一下，“这些轴向代码有意义吗？”我查看不同轴向代码和开放代码之间的对应关系，然后进行练习，然后说：“嗯。
Do I like these codes?
我喜欢这些代码吗？
Can I make them better?
我能让他们变得更好吗？
Can I refine them?
我可以完善它们吗？
Can I make them more specific?" Instead of being generic, I make them very specific and actionable.
我可以说得更具体一些吗？”我没有让它们变得笼统，而是非常具体且可操作。
Hamel Husain: So you see the ones that I came up with here are tour scheduling, rescheduling issues, human handoff or transfer issue, formatting error with an output, conversational flow.
Hamel Husain：所以你看，我在这里提出的问题是巡演安排、重新安排问题、人工切换或转移问题、输出格式错误、对话流程。
We saw the conversational flow issue with the text messages.
我们看到短信的对话流程问题。
Making follow-up promises not kept.
做出的后续承诺没有兑现。
Hamel Husain: And so basically, what I can do, what you can do now is you have these axial codes, and so I just collect them into a list, so this is an Excel formula.
Hamel Husain：基本上，我能做的，你现在能做的就是你有这些轴向代码，所以我只是将它们收集到一个列表中，所以这是一个 Excel 公式。
Just collect these codes into a list, and now we have a comma-separated list of these codes.
只需将这些代码收集到一个列表中，现在我们就有了这些代码的逗号分隔列表。
And then what you can simply do is you could take your notes that you have, those open codes, and you can tell an AI, and this is using Gemini and AI just for simplicity, this is, again, we're trying to keep it simple, categorize the following note into one of the following categories as always.
然后你可以简单地做的是，你可以记下你拥有的笔记，那些开放的代码，你可以告诉人工智能，这是使用双子座和人工智能只是为了简单起见，这也是，我们试图保持简单，一如既往地将以下笔记分类为以下类别之一。
Lenny Rachitsky: For folks watching, I like all these different prompts and formulas you're sharing.
Lenny Rachitsky：对于观看的人来说，我喜欢您分享的所有这些不同的提示和公式。
This is the Google Sheets AI prompt.
这是 Google Sheets AI 提示。
Hamel Husain: And so basically, what you could do is you can categorize your traces into one of the buckets, and that's what we have here.
Hamel Husain：基本上，您可以将您的痕迹分类到其中一个桶中，这就是我们这里所拥有的。
We have categorized all those problems that we encountered into one of these things.
我们将遇到的所有问题归为其中一件事。
Shreya Shankar: And this is automatic, which is very exciting.
Shreya Shankar：这是自动的，这非常令人兴奋。
I mean, the AI is doing it.
我的意思是，人工智能正在这样做。
So this also drives home the point that your open codes have to be detailed, right?
所以这也说明了一点：你的开放代码必须是详细的，对吗？
You can't just say janky because if the AI is reading janky, it's not going to be able to categorize it.
你不能只说“janky”，因为如果人工智能读取的是“janky”，它就无法对其进行分类。
Even a human wouldn't, right?
即使是人类也不会，对吗？
It would have to go and remember why you said janky, so it's important to be somewhat detailed in your open code.
它必须记住你为什么说 janky，所以在你的开放代码中详细说明是很重要的。
Lenny Rachitsky: Okay.
莱尼·拉奇茨基：好的。
So avoid the word janky.
所以避免使用“janky”这个词。
It's a good rule of thumb.
这是一个很好的经验法则。
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
Or have it with 10 other words.
或者与其他 10 个单词一起使用。
Lenny Rachitsky: Yeah, okay.
莱尼·拉奇茨基：是的，好的。
What are some of those other words that people often use that you think are not good?
人们经常使用的其他哪些词你认为不好？
Shreya Shankar: I don't think it's specific words.
Shreya Shankar：我认为这不是具体的词。
I think it's just people are not detailed enough in the open code, so it's hard to do the categorization.
我认为只是人们对开放代码的描述不够详细，所以很难进行分类。
Lenny Rachitsky: Great.
莱尼·拉奇茨基：太棒了。
And by the way, the reason you have to map them back is because, say, Claude or ChatGPT gave you suggestions and you change them and iterated on them, so you can't just go back and say, "Cool, whatever," in each bucket?
顺便说一句，你必须将它们映射回来的原因是因为，比如说，Claude 或 ChatGPT 给了你建议，你改变它们并迭代它们，所以你不能只是回去并在每个桶中说“酷，无论如何”？
Hamel Husain: That's a really good question, actually.
哈梅尔·侯赛因：实际上，这是一个非常好的问题。
It's good to iterate and think about it a little bit like, "Do I like these open codes?
迭代并思考一下是件好事，就像“我喜欢这些开放代码吗？
Do these actually make sense to me?" Just like anything that AI does, it's really good to kind of put yourself in the middle just a little bit.
这些对我来说真的有意义吗？”就像人工智能所做的任何事情一样，将自己置于中间一点真的很好。
Lenny Rachitsky: It's in the loop.
Lenny Rachitsky：它在循环中。
Still space for us.
仍然有我们的空间。
Great.
伟大的。
Shreya Shankar: One of the things that I like to do with this step if I'm trying to use AI to do this labeling, is also have a new category called none of the above.
Shreya Shankar：如果我尝试使用人工智能来进行此标记，我在这一步中喜欢做的事情之一就是有一个名为“以上都不是”的新类别。
So an AI can actually say, "None of the above," in the axial code, and that informs me, "Okay, my axial codes are not complete.
因此，人工智能实际上可以在轴向代码中说“以上都不是”，这告诉我，“好吧，我的轴向代码不完整。
Let's go look at those open codes, let's figure out what some new categories are or figure out how to reword my other axial codes."
让我们看看那些开放的代码，让我们弄清楚一些新类别是什么，或者弄清楚如何重新措辞我的其他轴向代码。”
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
And what's cool about this is you don't need to do this many, many times.
这样做的好处是您不需要多次执行此操作。
Lenny Rachitsky: For most products, you do this process once, and then you build on it, I imagine, and you just tweak it over time?
Lenny Rachitsky：对于大多数产品，你会执行一次这个过程，然后你会在它的基础上进行构建，我想，你只是随着时间的推移对其进行调整？
Shreya Shankar: Absolutely.
施瑞亚·香卡：当然。
And it gets so fast.
而且它变得如此之快。
People do this once a week, and you can do all of this in 30 minutes, and suddenly your product is so much better than if you were never aware of any of these problems.
人们每周都会这样做一次，而您可以在 30 分钟内完成所有这些工作，突然之间，您的产品比您从未意识到这些问题要好得多。
Lenny Rachitsky: Yeah.
莱尼·拉奇茨基：是的。
It's absurd to feel like you wouldn't know this is happening.
感觉你不知道这件事正在发生，这是荒谬的。
Watching this happening, I'm like, "How could you not do this to your product?"
看到这一切发生，我想，“你怎么能不对你的产品这样做呢？”
Lenny Rachitsky: Most people.
莱尼·拉奇茨基：大多数人。
Yeah.
是的。
We'll talk about that.
我们会讨论这个。
There's a whole debate around this stuff that we want to talk about.
围绕我们想要谈论的这个问题存在着一场完整的辩论。
Okay, cool.
好吧，酷。
So you have the sheet.
这样你就有了这张表。
What comes next?
接下来会发生什么？
Hamel Husain: Okay.
哈梅尔·侯赛因：好的。
So here's sort of the big unveil.
这是一个重大揭幕。
This is the magic moment right now.
现在就是神奇的时刻。
So we have all these codes that we applied, the ones that we like on our traces.
因此，我们拥有我们应用的所有这些代码，以及我们喜欢的痕迹。
Now, you can do the ta-da, you can count them.
现在，你可以做ta-da，你可以数它们。
Hamel Husain: So here's a pivot table, and we just can do pivot table on those, and we can count how many times those different things occurred.
Hamel Husain：这是一个数据透视表，我们可以对它们进行数据透视表，我们可以计算这些不同的事情发生了多少次。
So what do we find?
那么我们发现了什么？
Find on these traces that we categorized?
找到我们分类的这些痕迹吗？
We found 17 conversational flow issues.
我们发现了 17 个对话流程问题。
And I really like pivot tables because you can do cool things.
我真的很喜欢数据透视表，因为你可以做很酷的事情。
You can double-click on these.
您可以双击这些。
You can say, "Oh, okay.
你可以说：“哦，好吧。
Let me take a look at those," but that's going into an aside about pivot tables, how cool they are.
让我看一下这些，”但这涉及到数据透视表，它们有多酷。
Hamel Husain: But now, we have just a nice, rough cut of what are our problems?
哈梅尔·侯赛因：但是现在，我们对我们的问题有了一个很好、粗略的了解？
And now, we have gone from chaos to some kind of thinking around, "Oh, you know what?
现在，我们已经从混乱转向某种思考，“哦，你知道吗？
These are my biggest problems.
这些是我最大的问题。
I need to fix conversational issues, maybe these human handoff issues." It's not necessarily the count is the most important thing.
我需要解决对话问题，也许是这些人为交接问题。”计数不一定是最重要的。
It might be something that's just really bad and you want to fix that, but okay.
这可能是一些非常糟糕的事情，你想解决它，但是没关系。
Now, you have some way of looking at your problem, and now you can think about whether you need evals for some of these.
现在，您已经有了某种看待问题的方法，现在您可以考虑是否需要对其中一些问题进行评估。
Hamel Husain: So there might be some of these things that might be just dumb engineering errors that you don't need to write an eval for because it's very obvious on how to fix them.
Hamel Husain：因此，其中可能有一些事情可能只是愚蠢的工程错误，您不需要为其编写评估，因为如何修复它们非常明显。
Maybe the formatting error with output, maybe you just forgot to tell the LLM how you want it to be formatted, and you didn't even say that in the prompt.
也许是输出的格式错误，也许你只是忘记告诉LLM你希望如何格式化它，而你甚至没有在提示中说出来。
So just go ahead and fix the prompt maybe, and we can decide, "Okay, do you want to write an eval for that?" You might still want to write an eval for that because you might be able to test that with just code.
因此，只要继续修复提示，我们就可以决定，“好吧，你想为此编写一个评估吗？”您可能仍然想为此编写一个评估，因为您可能只需使用代码即可对其进行测试。
You could just test the string, does it have the right formatting potentially?
您可以只测试该字符串，它是否具有正确的格式？
Without running an LLM.
无需运行法学硕士。
Hamel Husain: So there's a cost-benefit trade-off to evals.
Hamel Husain：因此评估需要权衡成本效益。
You don't want to get carried away with it, but you want to usually ground yourself in your actual errors.
你不想被它冲昏头脑，但你通常希望自己能够正视自己的实际错误。
You don't want to skip this step.
您不想跳过这一步。
And so the reason I'm kind of spending so much time on this is this is where people get lost.
所以我花这么多时间在这上面的原因是人们会迷失方向。
They go straight into evals like, "Let me just write some tests," and that is where things go off the rails.
他们直接进行评估，比如“让我写一些测试”，这就是事情脱轨的地方。
Hamel Husain: Okay.
哈梅尔·侯赛因：好的。
So let's say we want to tackle one of these things.
假设我们想要解决其中一件事。
So for example, let's say we want to tackle this human handoff issue, and we're like, "Hmm, I'm not really sure how to fix this.
举例来说，假设我们想要解决这个人工切换问题，我们会说：“嗯，我不太确定如何解决这个问题。
That's a kind of subjective sort of judgment call on should we be handing off to a human?
这是一种主观判断，我们是否应该将其交给人类？
And I don't know immediately how to fix it.
我不知道如何立即修复它。
It's not super obvious per se.
它本身并不是非常明显。
Yeah.
是的。
I can change my prompt, but I'm not sure.
我可以更改提示，但我不确定。
I'm not 100% sure."
我不是100%确定。”
Hamel Husain: Well, that might be sort of an interesting thing for an LLM as a judge, for example.
Hamel Husain：嗯，例如，对于作为法官的法学硕士来说，这可能是一件有趣的事情。
So there's different kinds of evals.
所以有不同类型的评估。
One is code-based, which you should try to do if you can because they're cheaper.
一种是基于代码的，如果可以的话你应该尝试这样做，因为它们更便宜。
LLM as a judge is something, it's like a meta eval.
LLM作为法官是很重要的，它就像一个元评估。
You have to eval that eval to make sure the LLM that's judging is doing the right thing, which we'll talk about in a second.
你必须评估该评估，以确保进行评审的法学硕士正在做正确的事情，我们稍后会讨论这一点。
Hamel Husain: So, okay.
哈梅尔·侯赛因：所以，好吧。
LLM as a judge, that's one thing.
作为一名法官，LLM 是一回事。
Okay.
好的。
How do you build an LLM as a judge?
作为一名法官，你如何获得法学硕士学位？
Lenny Rachitsky: Before we get into that actually, just to make sure people know exactly what you're describing there, these two types of evals.
Lenny Rachitsky：在我们实际讨论之前，只是为了确保人们准确地了解您所描述的内容，这两种类型的评估。
One is you said it's code-based and one is LLM as judge.
一种是你说它是基于代码的，一种是LLM作为判断。
Maybe Shreya, just help us understand what code-based eval even is?
也许 Shreya，只是帮助我们理解什么是基于代码的 eval？
It's essentially a unit test?
它本质上是一个单元测试？
Is that a simple way to think about it?
这是一个简单的思考方式吗？
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
Maybe eval is not the right term here, but think automated evaluator.
也许 eval 在这里不是正确的术语，但请考虑自动化评估器。
So when we find these failure modes, one of the things we want is, "Okay.
因此，当我们发现这些故障模式时，我们想要的事情之一就是，“好吧。
Can we now go check the prevalence of that failure mode in an automated way without me manually labeling and doing all the coding and the grouping, and I want to run it on thousands and thousands of traces, I want to run it every week." That is, okay.
我们现在可以以自动化的方式检查该故障模式的普遍性，而无需我手动标记并进行所有编码和分组，我想在成千上万的跟踪上运行它，我想每周运行它。”也就是说，好吧。
You should probably build an automated evaluator to check for that failure mode.
您可能应该构建一个自动评估器来检查该故障模式。
Shreya Shankar: Now, when we're saying code-based versus LLM-based, we're saying, "Okay.
Shreya Shankar：现在，当我们说基于代码与基于 LLM 时，我们会说，“好吧。
So maybe I could write a Python function or a piece of code to check whether that failure mode is present in a trace or not." And that's possible to do for certain things like checking the output is JSON, or checking that it's markdown, or checking that it's short.
因此，也许我可以编写一个 Python 函数或一段代码来检查跟踪中是否存在该故障模式。”对于某些事情，例如检查输出是否为 JSON、检查其是否为 markdown、或检查其是否简短，这是可能的。
These are all things you can capture in code or you could approximately capture in code.
这些都是您可以在代码中捕获或可以在代码中大致捕获的所有内容。
Shreya Shankar: When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way.
Shreya Shankar：当我们在这里谈论LLM法官时，我们说这是一种复杂的失败模式，我们不知道如何以自动化的方式进行评估。
So maybe we will try to use an LLM to evaluate this very, very narrow, specific failure mode of handoffs.
因此，也许我们会尝试使用法学硕士来评估这种非常非常狭窄的特定故障模式的切换。
Lenny Rachitsky: So just to try to mirror back what you're describing, you want to test what your, say, agent or AI product is doing.
Lenny Rachitsky：因此，为了尝试反映您所描述的内容，您想要测试您的代理或人工智能产品正在做什么。
You ask it a question, it gets back with something.
你问它一个问题，它会返回一些东西。
Lenny Rachitsky: One way to test if it's giving you the right answer is if it's consistently doing the same thing, that you could write a code to tell you this is true or false.
Lenny Rachitsky：测试它是否给你正确答案的一种方法是它是否始终做同样的事情，你可以编写一段代码来告诉你这是对还是错。
For example, will it ever say there's a virtual tour?
例如，它会说有虚拟旅游吗？
So you could ask it.
所以你可以问一下。
Lenny Rachitsky: "Do you provide virtual tours?" It says yes or no, and then you could write code to tell you if it's correct based on that specific answer.
Lenny Rachitsky：“你们提供虚拟游览吗？”它表示是或否，然后您可以编写代码来根据该特定答案告诉您它是否正确。
Lenny Rachitsky: But if you're asking about something more complicated and it's not binary, in one world, you need a human to tell you this is correct.
Lenny Rachitsky：但是，如果你问的是更复杂的问题，并且它不是二元的，在一个世界中，你需要一个人来告诉你这是正确的。
The solution to avoid humans having to review all this every time automatically is LLMs replacing human judgment, and you'd call it an LLM as judge.
避免人类每次自动审查所有这些的解决方案是法学硕士取代人类判断，你可以将其称为法学硕士作为法官。
The LLM as being the judge if this is correct or not.
法学硕士负责判断这是否正确。
Shreya Shankar: So people always think, "Oh, this is at least as hard as my problem of creating the original agent." And it's not, because you're asking the judge to do one thing, evaluate one failure mode, so the scope of the problem is very small and the output of this LLM judge is pass or fail.
Shreya Shankar：所以人们总是想，“哦，这至少和我创建原始代理的问题一样困难。”事实并非如此，因为你要求法官做一件事，评估一种失败模式，所以问题的范围非常小，并且该LLM法官的输出是通过或失败。
So it is a very, very tightly scoped thing that LLM judges are very capable of doing very reliably.
所以这是一个范围非常非常严格的事情，法学硕士法官非常有能力非常可靠地做到这一点。
Lenny Rachitsky: And the goal here is just to have a suite of tests that run before you ship to production that tell you things are going the way you want them to?
Lenny Rachitsky：这里的目标只是在交付生产之前运行一套测试，告诉您事情正在按照您希望的方式进行？
The way your agent is interacting is correct?
您的代理交互方式是否正确？
Shreya Shankar: The beautiful thing about LLM judges, you can use them in unit tests or CI, sure, but you could also use it online for monitoring, right?
Shreya Shankar：LLM 法官的美妙之处在于，你可以在单元测试或 CI 中使用它们，当然，但你也可以在线使用它们进行监控，对吧？
I can sample 1000 traces every day, run my LLM judge, real production traces, and see what the failure rate is there.
我每天可以对 1000 个跟踪进行采样，运行我的 LLM 判断，真实的生产跟踪，看看故障率是多少。
This is not a unit test, but still now we get an extremely specific measure of application quality.
这不是单元测试，但现在我们仍然获得了应用程序质量的极其具体的衡量标准。
Lenny Rachitsky: Cool.
莱尼·拉奇茨基：酷。
That's a really great point because a lot of people just see evals for being this not-real-life thing.
这是一个非常重要的观点，因为很多人只是将评估视为不现实的事情。
It's a thing that you test before it's actually in the real world.
这是您在实际进入现实世界之前对其进行测试的东西。
And what's actually happening in the real world, you're saying you should actually do exactly that?
现实世界中实际发生了什么，你是说你实际上应该这样做？
Lenny Rachitsky: Test your real thing running in production?
Lenny Rachitsky：测试您在生产环境中运行的真实事物吗？
And it's a daily, hourly sort of thing you could be running?
这是你每天、每小时都要运行的事情吗？
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
Okay.
好的。
Hamel's got an example of an actual LLM as a judge eval here, so let's take a look.
哈梅尔在这里有一个实际的法学硕士作为法官评估的例子，所以让我们看一下。
Hamel Husain: I love how Shreya really teed it up for me, so thank you so much.
Hamel Husain：我喜欢 Shreya 为我准备的方式，所以非常感谢你。
So what we have is a LLM as a judge prompt for this one specific failure.
因此，我们拥有的是法学硕士作为针对这一特定失败的判断提示。
Like Shreya said, you would want to do one specific failure and you want to make it binary because we want to simplify things.
就像 Shreya 所说，你会想要做一个特定的失败，并且你想要将其变成二进制，因为我们想要简化事情。
We don't want, "Hey, score this on a rating of one to five.
我们不想，“嘿，以一到五的评分来评分。
How good is it?" That's just in most cases, that's a weasel way of not making a decision.
有多好？”只是在大多数情况下，这是一种不做决定的狡猾方式。
Like, "No, you need to make a decision.
就像，“不，你需要做出决定。
Is this good enough or not?
这是否足够好？
Yes or no?"
是还是不是？”
Hamel Husain: It can be painful to think about what that is, but you should absolutely do it.
Hamel Husain：思考那是什么可能会很痛苦，但你绝对应该这样做。
Otherwise, this thing becomes very untractable, and then when you report these metrics, no one knows what 3.2 versus 3.7 means, so.
不然的话，这个事情就变得非常难处理，然后当你报告这些指标的时候，没有人知道3.2和3.7意味着什么，所以。
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
We see this all the time also, and even with expert-curated content on the internet where it's like, "Oh, here's your LLM judge evaluator prompt.
我们也一直看到这种情况，甚至在互联网上有专家策划的内容，就像，“哦，这是你的法学硕士法官评估提示。
Here's a one-to-seven scale."
这是从一到七的比例。”
Shreya Shankar: And I always text Hamel like, "Oh, no. Now, we have to fight the misinformation again because we know somebody is going to try it out and then come back to us and say, 'Oh, I have 4.2 average,'" and we're going to be like, "Okay."
Shreya Shankar：我总是给 Hamel 发短信，“哦，不。现在，我们必须再次与错误信息作斗争，因为我们知道有人会尝试一下，然后回来告诉我们，‘哦，我的平均分是 4.2’”，我们会说，“好吧。”
Lenny Rachitsky: It's wild how much drama there is in the evals space.
Lenny Rachitsky：评估领域有如此多的戏剧性，真是太疯狂了。
We're going to get to that.
我们会做到这一点。
Oh, man.
哦，伙计。
Lenny Rachitsky: This episode is brought to you by Mercury.
Lenny Rachitsky：本集由 Mercury 为您带来。
I've been banking with Mercury for years, and honestly, I can't imagine banking any other way at this point.
我多年来一直使用 Mercury 进行银行业务，老实说，目前我无法想象以任何其他方式进行银行业务。
I switched from Chase, and holy moly, what a difference.
我从 Chase 转到了 Holy Moly，这有什么不同。
Sending wires, tracking spend, giving people on my team access to move money around, so freaking easy.
发送电汇、跟踪支出、让我的团队成员可以转移资金，如此简单。
Where most traditional banking websites and apps are clunky and hard to use, Mercury is meticulously designed to be an intuitive and simple experience.
大多数传统银行网站和应用程序都笨重且难以使用，而 Mercury 则经过精心设计，提供直观、简单的体验。
Lenny Rachitsky: Meticulously designed to be an intuitive and simple experience, and Mercury brings all the ways that you use money into a single product, including credit cards, invoicing, bill pay, reimbursements for your teammates and capital.
Lenny Rachitsky：Mercury 经过精心设计，旨在提供直观、简单的体验，将您使用金钱的所有方式整合到一个产品中，包括信用卡、发票、账单支付、队友报销和资本。
Whether you're a funded tech startup looking for ways to pay contractors and earn yield on your idle cash, or an agency that needs to invoice customers and keep them current, or an e-commerce brand that needs to stay on top of cash flow and access capital, Mercury can be tailored to help your business perform at its highest level.
无论您是一家资金雄厚的科技初创公司，正在寻求向承包商付款并利用闲置现金赚取收益的方式，还是一家需要向客户开具发票并保持其最新状态的机构，或者是一家需要掌控现金流并获取资金的电子商务品牌，Mercury 都可以量身定制，帮助您的企业发挥最高水平。
See what over 200,000 entrepreneurs love about Mercury.
了解超过 200,000 名企业家对 Mercury 的喜爱。
Visit mercury.com to apply online in 10 minutes.
10 分钟内访问mercury.com 进行在线申请。
Mercury is a fintech, not a bank.
Mercury 是一家金融科技公司，而不是一家银行。
Banking services provided through Mercury's FDIC insured partner banks.
通过 Mercury 的 FDIC 承保合作银行提供的银行服务。
For more details, check out the show notes.
欲了解更多详情，请查看演出笔记。
Hamel Husain: Okay, so this is your judge prompt.
哈梅尔·侯赛因：好的，这是您的法官提示。
There's no one way to do it.
没有一种方法可以做到这一点。
It's okay to use an LLM to help you create it, but again, put yourself in the loop.
使用法学硕士来帮助你创建它是可以的，但同样，让你自己参与其中。
Don't just blindly accept what the LLM does, and in all of these cases, that's what we did.
不要只是盲目地接受LLM所做的事情，在所有这些情况下，这就是我们所做的。
With the axial codes, we iterated on this.
通过轴向代码，我们对此进行了迭代。
You can use an LLM to help you create this prompt, but make sure you read it, make sure you edit it, whatever.
您可以使用法学硕士来帮助您创建此提示，但请确保您阅读它，确保您编辑它，无论如何。
This is not necessarily the perfect prompt.
这不一定是完美的提示。
This is just the stupid, keeping it very simple just to show you the idea.
这只是愚蠢的做法，保持非常简单只是为了向您展示这个想法。
It's like, "Okay, for this handoff failure," I said, "Okay, I want you to output true or false," it's a binary judge.
就像，“好吧，对于这次切换失败，”我说，“好吧，我希望你输出 true 或 false，”这是一个二元判断。
That's what we recommend.
这就是我们的建议。
Then I just go through and say, "Okay, when should you be doing a handoff?" And I just list them out.
然后我就过去说：“好吧，你什么时候应该进行交接？”我只是把它们列出来。
Hamel Husain: Okay, explicit human requests ignored or looped, some policy-mandated transfer, sensitive resident issues, tool data, unavailability, same day walk-in or tour requests.
哈梅尔·侯赛因：好的，明确的人为请求被忽略或循环，一些政策强制的转移，敏感的居民问题，工具数据，不可用，当天预约或旅游请求。
You need to talk to a human for that, so on and so forth.
为此，您需要与人交谈，等等。
The idea is, now that I know that this is a failure from my data, I'm interested in iterating on it, because I know this is actually happening all the time.
我的想法是，现在我知道这是我的数据的失败，我有兴趣对其进行迭代，因为我知道这种情况实际上一直在发生。
Like Shreya said, it would be nice to have a way not only to evaluate this on the data I have, but also on production data, just to get a sense of, what scales is this happening?
正如 Shreya 所说，如果有一种方法不仅可以根据我所拥有的数据进行评估，而且还可以根据生产数据进行评估，以便了解这种情况发生的规模，那就太好了？
Let me find more traces, let me have a way to iterate on this.
让我找到更多痕迹，让我有办法迭代这个。
We can take this prompt and I'm going to use the spreadsheet again.
我们可以接受此提示，我将再次使用电子表格。
The first step is, okay, when I'm doing this judge...
第一步是，好吧，当我做这个判断时......
I wrote the prompt.
我写了提示。
Hamel Husain: Now, a lot of people stop there and they say, "Okay, I have my judge prompt.
哈梅尔·侯赛因：现在，很多人都停了下来，他们说：“好吧，我有法官提示。
We're done.
我们完成了。
Good, let's just ship it," and the prompt says...
很好，我们就发货吧，”然后提示说……
If the judge says it's wrong, it's wrong.
如果法官说错了，那就错了。
They just accept it as the gospel, be like, "Okay, the LLM says it's wrong, it must be wrong.
他们只是把它当作福音来接受，就像，“好吧，法学硕士说这是错的，它一定是错的。
Don't do that, because that's the fastest way that you can have evals that don't match what's going on, and when people lose trust in your evals, they lose trust in you.
不要这样做，因为这是最快的方式，您的评估可能与实际情况不符，并且当人们对您的评估失去信任时，他们也会对您失去信任。
It's really important that you don't do that, so before you release your LLM as a judge, you want to make sure it's aligned to the human.
不要这样做，这一点非常重要，因此在您以法官身份授予法学硕士学位之前，您需要确保它与人类保持一致。
How do you do that?
你是怎么做到的？
You have those axial codes and you want to measure your judge against the axial code, and say like, "Hey, does it agree with me?
你有这些轴向代码，你想根据轴向代码来衡量你的判断，然后说，“嘿，它同意我吗？
My own judge, does it agree with me?" Just measure it.
我自己的判断，与我一致吗？”衡量一下就可以了。
Hamel Husain: What we have here is, okay, I say, "Assess this LLM trace." Again, I'm using just spreadsheets here, "Assess this LM trace according to these rules," and the rules are just the prompt that I just showed you.
Hamel Husain：好吧，我说的是“评估这个法学硕士踪迹。”再说一次，我在这里只使用电子表格，“根据这些规则评估这个 LM 跟踪”，这些规则只是我刚刚向您展示的提示。
I ask it, "Okay, is there a handoff error, true or false?" Then this column, let me just zoom in a bit.
我问它：“好吧，是否存在切换错误，是真是假？”那么这个专栏，我稍微放大一下。
Column H, I have, "Okay, did this error occur?" Column G is whether I thought the error occurred or not.
H 列，我有，“好的，这个错误发生了吗？” G 列是我是否认为错误发生了。
You can see-
你可以看到——
Lenny Rachitsky: You're going through manually, you do that.
Lenny Rachitsky：你要手动进行，你就这样做。
Hamel Husain: Yeah, yeah, which we already did.
哈梅尔·侯赛因：是的，是的，我们已经这样做了。
We already went through it manually.
我们已经手动完成了它。
It's not like we have to do it again, because we have that cheat code from the axial coding, we already did it.
这并不是说我们必须再做一次，因为我们有来自轴向编码的作弊代码，我们已经做到了。
You might have to go through it again if you need more data, and there's a lot of details to this on how to do this correctly.
如果您需要更多数据，您可能需要再次查看它，并且有很多关于如何正确执行此操作的详细信息。
You want to split your data and do all these things, so that you're not cheating, but I just want to show you the concept.
你想要分割你的数据并做所有这些事情，这样你就不会作弊，但我只是想向你展示这个概念。
Basically, what you can do is measure the agreement.
基本上，你能做的就是衡量一致性。
Now, one thing you should know, as a product manager, is a lot of people go straight to this agreement.
现在，作为产品经理，你应该知道的一件事是，很多人都会直接同意这个协议。
They say, "Okay, my judge agrees with the human some percentage of the time."
他们说：“好吧，我的判断在一定程度上同意人类的观点。”
Hamel Husain: Now that sounds appealing, but it's a very dangerous metric to use, because a lot of times, errors, they only happen on the long tail and they don't happen as frequently, so if you only have the error 10% of the time, then you can easily have 90% agreement by just having a judge say it passes all the time.
Hamel Husain：这听起来很吸引人，但使用起来是一个非常危险的指标，因为很多时候，错误只发生在长尾上，而且不会那么频繁地发生，所以如果你只有 10% 的时间出现错误，那么只要让法官说它一直通过，你就可以轻松获得 90% 的同意。
Does that make sense?
这有道理吗？
90% agreement look good on paper, but it might be misleading.
90% 的一致性在纸面上看起来不错，但可能会产生误导。
Lenny Rachitsky: It's rare, it's a rare error.
Lenny Rachitsky：这种情况很少见，这是一个罕见的错误。
Yeah.
是的。
Hamel Husain: As a product manager or someone, even if you're not doing this calculation yourself, if someone ever reports to you agreement, you should immediately ask, "Okay, tell me more." You need to look into it.
Hamel Husain：作为产品经理或其他人，即使您自己没有进行此计算，如果有人向您报告同意，您应该立即询问，“好吧，告诉我更多信息。”你需要调查一下。
They give you more intuition, here is like a matrix of this specific judge in the Google sheet, and this is, again, a pivot table, just keeping it dumb and simple.
它们给你更多的直觉，这里就像谷歌表中这个特定法官的矩阵，这又是一个数据透视表，只是保持它的愚蠢和简单。
"Okay, on the rows I have, what did the human think?
“好吧，对于我的行，人类是怎么想的？
What did I think?
我怎么想的？
Did it have an error, true or false?
它有错误吗，是真是假？
Then did my judge have an error, true or false?"
那么我的法官是否有错误，是真是假？”
Shreya Shankar: The intuition here is exactly what Hamel said, where you need to look at each type of error.
Shreya Shankar：这里的直觉正是 Hamel 所说的，你需要查看每种类型的错误。
When the human said false, but the judge said true, or vice versa, so those non-green diagonals here, and if they're too large, then go iterate on your prompt, make it more clear to the LLM judge, so that you can reduce that misalignment.
当人类说假，但法官说真时，反之亦然，所以这里的那些非绿色对角线，如果它们太大，那么就迭代你的提示，让LLM法官更清楚，这样你就可以减少这种错位。
You want to get to a point where most...
你想要达到一个最...
You're going to have some misalignment, that's okay.
你会有一些错位，没关系。
We talk about in our course, also how to code correct that misalignment, but in this stage, if you're a product manager and the person who's building the LLM judge eval has not done this, they're saying like, "It agrees 75% of the time, we're good." They don't have this matrix and they haven't iterated to make sure that these two types of errors have gone down to zero, then it's a bad smell.
我们在课程中讨论了如何通过编码来纠正这种不一致，但在这个阶段，如果你是产品经理，而构建 LLM 法官评估的人没有这样做，他们会说，“75% 的时间都是一致的，我们很好。”他们没有这个矩阵，也没有进行迭代以确保这两类错误已降至零，那么这就很糟糕了。
Go and ask them to go fix that.
去要求他们去解决这个问题。
Lenny Rachitsky: Awesome.
莱尼·拉奇茨基：太棒了。
That's a really good tip, what to look for when someone's doing this wrong.
这是一个非常好的提示，当有人做错时应该寻找什么。
Lenny Rachitsky: Actually, can you take us back to the LLM as judge prompt?
Lenny Rachitsky：实际上，您能带我们回到法学硕士作为法官提示吗？
I just want to highlight something really interesting here.
我只想在这里强调一些非常有趣的事情。
I've had some guests on the podcast recently who've been saying, "Evals are the new PRDs," and if you look at this, this is exactly what this is.
最近我的播客上有一些客人一直在说，“Eval 是新的 PRD”，如果你仔细看看这个，就会发现这正是这个问题。
Product managers, product teams, here's what the product should be, here's all the requirements, here's the how it should work.
产品经理、产品团队，这就是产品应该是什么样的，这是所有的要求，这是它应该如何工作。
They built a thing and then they test it.
他们建造了一个东西，然后测试它。
Manually, often.
经常手动。
What's cool about this is this is exactly that same thing, and it's running constantly.
最酷的一点是，这完全是同一件事，而且它一直在运行。
It's telling you, "Here's how this agent should respond," and it's very specific ways.
它告诉您，“这是该代理应该如何响应”，而且是非常具体的方式。
"If it's this, this, this, do that.
“如果是这个，这个，这个，就那样做。
If it's this, this, that, do that." It's exactly what I've been hearing again and again, you could see right here.
如果是这样、这样、那样，就那样做。”这正是我一次又一次听到的，你可以在这里看到。
This is the purest sense of what a product requirements document should be, is this eval judge that's telling you exactly what it should be, and it's automatic and running constantly.
这是产品需求文档最纯粹的含义，这个评估法官准确地告诉您它应该是什么，并且它是自动的并且不断运行。
Shreya Shankar: Yeah, absolutely.
Shreya Shankar：是的，绝对如此。
It's derived from our own data, so of course, it's a product manager's expectations.
它是根据我们自己的数据得出的，所以当然，这是产品经理的期望。
What I find that a lot of people miss is they just put in what their expectations are before looking at their data, but as we look at our data, we uncover more expectations that we couldn't have dreamed up in the first place, and that ends up going into this prompt.
我发现很多人忽略的是，他们只是在查看数据之前输入了自己的期望，但是当我们查看数据时，我们发现了更多我们一开始无法想象的期望，这最终进入了这个提示。
Lenny Rachitsky: That is interesting.
莱尼·拉奇茨基：这很有趣。
Your advice is not skip straight to evals and LLM as judge prompts before you build the product, still write traditional one-pagers PRDs to tell your team what we're doing, why we're doing it, what success looks like.
您的建议是，在构建产品之前，不要像法官提示一样直接跳到评估和法学硕士，而是编写传统的单页 PRD 来告诉您的团队我们在做什么、为什么要这样做以及成功是什么样子。
But then at the end, you could probably pull from that and even improve that original PRD if you're evolving the product using this process.
但最后，如果您正在使用此流程改进产品，您可能可以借鉴甚至改进原始 PRD。
Shreya Shankar: I would go even further to say you're going to improve...
Shreya Shankar：我会更进一步说你会进步......
It's going to change.
它将会改变。
You're never going to know what the failure modes are going to be upfront, and you're always going to uncover new vibes that you think that your product should have.
你永远不会预先知道失败模式是什么，而且你总是会发现你认为你的产品应该具有的新氛围。
You don't really know what you want until you see it with these LLMs, so you got to be flexible, have to look at your data, have to...
在你看到这些法学硕士之前，你并不真正知道你想要什么，所以你必须保持灵活性，必须查看你的数据，必须......
PRDs are a great abstraction for thinking about this.
PRD 是思考这个问题的一个很好的抽象。
It's not the end all, be all.
这不是全部的结束，一切都结束了。
It's going to change.
它将会改变。
Lenny Rachitsky: I love that, and Hamel's pulling up some cool research report.
Lenny Rachitsky：我喜欢这个，Hamel 正在拿出一些很酷的研究报告。
What's this about?
这是关于什么的？
Hamel Husain: This is one of the coolest research reports you can possibly read if you want to know about evals.
Hamel Husain：如果您想了解评估，这是您可以阅读的最酷的研究报告之一。
It was authored by someone named Shreya Shankar.
它的作者是 Shreya Shankar。
Hamel Husain: And her collaborators.
哈梅尔·侯赛因：还有她的合作者。
It's called "Who Validates the Validated?"
它被称为“谁验证已验证的？”
Lenny Rachitsky: That's the best name for a researcher.
Lenny Rachitsky：这是对研究员最好的称呼。
Hamel Husain: I should let Shreya talk about this.
Hamel Husain：我应该让 Shreya 谈谈这个。
I think one of the most important things to pay attention in this paper are the criteria drift, and what she found.
我认为本文中最需要注意的事情之一是标准漂移以及她的发现。
Shreya Shankar: We did this super fun study when we were doing user studies with people who were trying to write LLM judges or just validate their own LLM outputs.
Shreya Shankar：当我们与那些试图编写法学硕士法官或只是验证自己的法学硕士输出的人进行用户研究时，我们做了这项超级有趣的研究。
I think this was before evals was extremely popular, I feel like, on the internet.
我认为这是在评估在互联网上非常流行之前。
We did this project late 2023 was when we started it.
我们在 2023 年底启动了这个项目。
But then the thing that really was burning in my mind as a researcher is like, "Why is this problem so hard?
但作为一名研究人员，真正在我脑海中燃烧的事情是，“为什么这个问题这么难？
We've been having machine learning and AI for so long, it's not new, but suddenly, this time around, everything is really difficult." We just did this user study with a bunch of developers and we realized, "Okay, what's new here is that you can't figure out your rubrics upfront.
我们使用机器学习和人工智能已经很长时间了，这并不新鲜，但突然间，这一次，一切都变得非常困难。”我们刚刚与一群开发人员进行了这项用户研究，我们意识到，“好吧，这里的新内容是你无法预先弄清楚你的规则。
People's opinions of good and bad change as they review more outputs, they think of failure modes only after seeing 10 outputs they would never have dreamed of in the first place," and these are experts.
人们对好坏的看法会随着审查更多输出而改变，他们只有在看到 10 个他们一开始从未梦想过的输出后才会考虑故障模式，”这些都是专家。
These are people who have built many LLM pipelines and now agents before, and you can't ever dream up everything in the first place.
这些人之前已经建立了许多法学硕士管道，现在是代理人，你不可能一开始就梦想一切。
I think that's so key in today's world of AI development.
我认为这在当今的人工智能发展世界中非常关键。
Lenny Rachitsky: That is a really good point.
Lenny Rachitsky：这是一个非常好的观点。
That's very much reinforcing what we were just talking about and that's why I'll pull this up, is just...
这非常强化了我们刚才讨论的内容，这就是为什么我会把它拉出来，只是......
Okay-
好吧-
Lenny Rachitsky: Yeah, okay, great.
莱尼·拉奇茨基：是的，好的，太好了。
You still got to do product the same way, but now you have this really powerful tool that helps you make sure what you've built is correct.
您仍然必须以相同的方式制作产品，但现在您拥有了这个非常强大的工具，可以帮助您确保所构建的内容是正确的。
It's not going to replace the PRD process.
它不会取代 PRD 流程。
Cool.
凉爽的。
How many, say, I don't know, LLM as judge prompts, do you end up with usually say...
有多少，比如说，我不知道，法学硕士作为法官的提示，你最终通常会说......
I don't know.
我不知道。
I know, obviously, depends complexity to the product, but what's a number in your experience?
我知道，显然，这取决于产品的复杂性，但根据您的经验，这个数字是多少？
Shreya Shankar: For me, between four and seven.
Shreya Shankar：对我来说，四到七岁之间。
Shreya Shankar: It's not that many, because a lot of the failure modes, as Hamel said earlier, can be fixed by just fixing your prompt.
Shreya Shankar：并没有那么多，因为正如 Hamel 之前所说，很多故障模式只需修复提示即可修复。
You just didn't think to put it in your prompts, so now you put it in your...
您只是没有想到将其放入提示中，所以现在您将其放入...
You shouldn't do an eval like this for everything, just the pesky ones that you've described your ideal behavior in your agent prompt, but it's still failing.
您不应该对所有内容都进行这样的评估，只对您在代理提示中描述的理想行为进行评估，但它仍然失败。
Lenny Rachitsky: Got it.
莱尼·拉奇茨基：明白了。
Say you found a problem, you fixed it.
假设您发现了问题并解决了它。
In traditional software development, you'd write a unit test to make sure it doesn't happen again.
在传统的软件开发中，您需要编写单元测试以确保它不会再次发生。
Is your insight here is, "Don't even bother writing an eval around that if it's just gone"?
您的见解是：“如果它已经消失了，甚至不必费心编写评估”？
Shreya Shankar: I think you can if you want to, but the whole game here is about prioritizing.
Shreya Shankar：我认为如果你愿意的话可以，但这里的整个游戏都是关于优先级的。
You have finite resources and finite time, you can't write an eval for everything, so prioritize the ones that are the more pesky areas.
您的资源和时间都是有限的，您无法为所有内容编写评估，因此请优先考虑那些更麻烦的领域。
Lenny Rachitsky: Probably the ones that are most risky to your business if they say something like Mecha Hitler, Grok.
Lenny Rachitsky：如果他们说“Mecha Hitler”、“Grok”之类的话，可能对您的业务风险最大。
Lenny Rachitsky: Cool.
莱尼·拉奇茨基：酷。
Okay, so that's very relieving, because this prompt was a lot of work to really think through all these details.
好的，这让人非常放心，因为要真正考虑所有这些细节，这个提示需要做很多工作。
Shreya Shankar: But it's a lot of one-time cost.
Shreya Shankar：但一次性成本很高。
Right now, forever, you can run this on your application.
现在，直到永远，您都可以在您的应用程序上运行它。
Hamel Husain: Okay, data analysis is super powerful, is going to drive lots of improvements very quickly to your application.
Hamel Husain：好的，数据分析非常强大，将很快为您的应用程序带来大量改进。
We showed the most basic kind of data analysis, which is counting, which is accessible to everyone.
我们展示了最基本的数据分析，即计数，每个人都可以使用它。
You can get more sophisticated with the data analysis.
您可以通过数据分析变得更加复杂。
There's lots of different ways to sample, look at data.
有很多不同的方法可以进行采样、查看数据。
We made it look easy in a sense, but there's a lot of skills here to do to it well.
从某种意义上说，我们让它看起来很简单，但这里有很多技巧可以把它做好。
Building an intuition and a nose for how to sort through this data.
培养如何对这些数据进行排序的直觉和嗅觉。
For example, let's say I find conversational issues, this conversational flow issues.
例如，假设我发现对话问题，即对话流程问题。
Maybe if I was trying to chase down this problem further, I would think about ways to find other conversational flow issues that I didn't code.
也许如果我试图进一步追查这个问题，我会考虑如何找到我没有编码的其他对话流问题。
I would maybe dig through the data in several ways, and there's different ways to go about this.
我可能会通过多种方式挖掘数据，并且有不同的方法可以实现这一点。
It's very similar, if not almost exactly similar as traditional analytics techniques that you would do on any product.
它与您在任何产品上执行的传统分析技术非常相似，即使不是几乎完全相同。
Lenny Rachitsky: Give us just a quick sense of what comes next and then let's talk about the debate around evals and a couple more things.
Lenny Rachitsky：让我们快速了解一下接下来会发生什么，然后让我们讨论一下围绕评估和其他一些事情的争论。
Shreya Shankar: What comes next after you've built your LLM judge?
Shreya Shankar：建立法学硕士法官后接下来会发生什么？
Well, we find that people just try to use that everywhere they can, so they'll put the LLM judge in unit tests and they will build, "Here are some example traces where we saw that failure, because we labeled it.
好吧，我们发现人们只是尝试在任何可以使用的地方使用它，所以他们会将 LLM 法官放入单元测试中，然后他们会构建，“这是我们看到失败的一些示例跟踪，因为我们标记了它。
Now we're going to make those part of unit tests and make sure that, every time we push a change to our code, these tests are going to pass." They also use it for online monitoring.
现在我们将把这些作为单元测试的一部分，并确保每次我们对代码进行更改时，这些测试都会通过。”他们还将其用于在线监控。
People are making dashboards on this, and I think that's incredible.
人们正在为此制作仪表板，我认为这令人难以置信。
I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don't talk about it, because this is their moat.
我认为正在这样做的产品，他们对应用程序的性能有非常敏锐的感觉，人们不会谈论它，因为这是他们的护城河。
People are not going to go and share all of these things, because it makes sense.
人们不会去分享所有这些事情，因为这是有道理的。
If you are an email-writing assistant, and you're doing this and you're doing it well, you don't want somebody else to go and build an email-writing assistant and then get you out of business.
如果你是一名电子邮件写作助理，并且你正在做这件事并且做得很好，那么你不希望其他人去建立一个电子邮件写作助理，然后让你破产。
Shreya Shankar: I really want to stress the point that it's try to use these artifacts that you're building wherever possible online, repeatedly use them to drive improvements to your product.
Shreya Shankar：我真的想强调一点，它会尝试使用您在网上尽可能构建的这些工件，反复使用它们来推动产品的改进。
Oftentimes, Hamel and I will tell people how to do this up to this very point, and it clicks for people and then they never come back again.
通常，哈梅尔和我会告诉人们如何做到这一点，人们会觉得这很有趣，然后他们就再也不会回来了。
Either they have, I don't know, quit their jobs, they're not doing AI development anymore, or they know what to do from here on out.
我不知道，他们要么辞去了工作，不再从事人工智能开发，要么知道从现在开始该做什么。
I think it's the latter, but I think it's very powerful.
我认为是后者，但我认为它非常强大。
Lenny Rachitsky: Just watching you do this really opened my eyes to what this is and how systematic the process is.
Lenny Rachitsky：看着你这样做，我真的大开眼界，明白了这是什么以及这个过程是多么系统化。
I always imagine you just sit on a computer, "Okay, what are the things I need to make sure work correctly?" What you're showing us here is it's a very simple step-by-step based on real things that are happening in your product, how to catch them, identify them, prioritize them, and then catch them if they happen again and fix them.
我总是想象你只是坐在电脑上，“好吧，我需要什么来确保正常工作？”您在这里向我们展示的是，这是一个非常简单的步骤，基于您的产品中发生的真实情况，如何捕获它们，识别它们，确定它们的优先级，然后在它们再次发生时捕获它们并修复它们。
Shreya Shankar: Yeah, it's not magic.
Shreya Shankar：是的，这不是魔法。
Anyone can do this, you're going to have to practice the skill, like any new skill, you have to practice, but you can do it.
任何人都可以做到这一点，你必须练习这项技能，就像任何新技能一样，你必须练习，但你可以做到。
I think what's very empowering now is that product managers are doing this and can do this, and can really build very, very profitable products with this skill set.
我认为现在非常有力量的是产品经理正在这样做并且可以这样做，并且可以利用这种技能真正构建非常非常有利可图的产品。
Lenny Rachitsky: Okay, great segue to a debate that we got pulled into that was happening on X the other day.
Lenny Rachitsky：好的，接下来我们就进入了前几天在 X 上发生的一场辩论。
I did not realize how much controversy and drama there is around evals.
我没有意识到围绕评估有多少争议和戏剧性。
There's a lot of people with very strong opinions.
有很多人有非常强烈的意见。
How about Shreya?
施瑞亚呢？
Give us just a sense of the two sides of the debate around the importance and value of evals, and then give us your perspective.
请让我们了解有关评估的重要性和价值的争论的两个方面，然后告诉我们您的观点。
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
All right, I'll be a little bit placating and I say I think everyone is on the same side.
好吧，我会稍微安抚一下，我说我认为每个人都站在同一边。
I think the misconception is that people have very rigid definitions of what evals is.
我认为误解是人们对 eval 的定义非常严格。
For example, they might think that evals is just unit tests or they might think that evals is just the data analysis part and no online monitoring or no monitoring of product-specific metrics, like actually number of chats engaged in or whatnot.
例如，他们可能认为评估只是单元测试，或者他们可能认为评估只是数据分析部分，没有在线监控或没有监控特定于产品的指标，例如实际参与的聊天数量等。
I think everyone has a different mindset of evals going in, and the other thing I will say is that people have been burned by evals in the past.
我认为每个人对评估都有不同的心态，我要说的另一件事是，人们过去曾被评估所困扰。
I think people have done evals badly.
我认为人们的评估做得很糟糕。
One concrete example of this is they've tried to do an LLM judge, but it has not aligned with their expectations.
一个具体的例子是，他们尝试做一名LLM法官，但与他们的期望不符。
They only uncovered this later on and then they didn't trust it anymore, and then they're like, "I'm anti evals."
他们后来才发现这一点，然后他们不再相信它，然后他们就像，“我反对评估。”
Shreya Shankar: I 100% empathize with that, because you should be anti Likert scale LLM judge.
Shreya Shankar：我百分百理解这一点，因为你应该是反李克特量表法学硕士的法官。
I absolutely agree with you, we are anti that as well.
我完全同意你的观点，我们也反对这种做法。
A lot of the misconception stems from two things, like people having a narrow definition of evals and then people not doing it well and then getting burned and then wanting to avoid other people making that mistake.
很多误解源于两件事，比如人们对评估的定义很狭隘，然后人们做得不好，然后被烧伤，然后想要避免其他人犯同样的错误。
Then, unfortunately, X or Twitter is a medium where people are misinterpreting what everybody is saying all the time, and you just get all these strong opinions of, "Don't do evals, it's bad.
然后，不幸的是，X 或 Twitter 是一种人们总是误解每个人所说内容的媒介，你只会得到所有这些强烈的意见，“不要进行评估，这很糟糕。
We tried it, it doesn't work.
我们尝试过，它不起作用。
We're Claude Code," or whatever other famous product, "And we don't do evals." There's just so much nuance behind all of it, because a lot of these applications are standing on the shoulders of evals.
我们是 Claude Code，”或者任何其他著名的产品，“而且我们不进行评估。”所有这些背后都有很多细微差别，因为许多应用程序都站在评估的肩膀上。
Coding agents is a great example of that, Claude Code.
编码代理就是一个很好的例子，Claude Code。
They're standing on the shoulders of Claude base model...
他们站在克劳德基础模型的肩膀上......
Not base, but the fine-tuned Claude models have been evaluated on many coding benchmarks.
不是基础模型，而是经过微调的 Claude 模型，已在许多编码基准上进行了评估。
Can't argue against that.
无法反驳这一点。
Lenny Rachitsky: Just to make clear exactly what you're talking about there, one of the heads, I think maybe the head engineer of Claude Code, went on a podcast and he's like, "We don't do evals, we just vibe.
Lenny Rachitsky：为了明确你在说什么，其中一位负责人，我想可能是 Claude Code 的首席工程师，参加了一个播客，他说，“我们不进行评估，我们只是进行氛围。
We just look at vibes," and vibes meaning they just use it and feel if it's right or wrong.
我们只是看共鸣，”而共鸣意味着他们只是使用它并感觉它是对还是错。
Shreya Shankar: I think that works.
Shreya Shankar：我认为这有效。
There's two things to that, right?
这有两件事，对吧？
One is they're standing on the shoulders of the evals that their colleagues are doing for coding.
一是他们站在同事进行编码评估的肩膀上。
Lenny Rachitsky: Of the Claude foundational model.
Lenny Rachitsky：克劳德基础模型。
Shreya Shankar: Absolutely, right?
Shreya Shankar：当然，对吧？
We know that they report those numbers, because we see the benchmarks, we know who's doing well on those.
我们知道他们报告了这些数字，因为我们看到了基准，我们知道谁在这些方面做得很好。
The other thing is they are actually probably very systematic about the error analysis to some extent.
另一件事是，他们实际上在某种程度上对错误分析可能非常系统。
I bet you that they're monitoring who is using Claude, how many people are using Claude, how many traps are being created, how long these chats are.
我敢打赌，他们正在监控谁在使用克劳德、有多少人在使用克劳德、创建了多少陷阱、这些聊天持续了多长时间。
They're also probably monitoring in their internal team, they're dogfooding.
他们也可能在内部团队中进行监控，他们正在进行内部测试。
Anytime something is off, they maybe have a cue or they send it to the person developing Claude Code, and this person is implicitly doing some form of hair error analysis that Hamel talked about.
每当出现问题时，他们可能会有提示，或者将其发送给开发克劳德代码的人，而这个人正在隐式地进行哈梅尔谈到的某种形式的头发错误分析。
All of this is evals, right?
所有这些都是评估，对吧？
There's no world in which they're just being like, "I made Claude Code, I'm never looking at anything," and unfortunately, when you don't think about that or talk about that, I think that the community...
他们只是说，“我制作了克劳德代码，我从不看任何东西”，不幸的是，当你不考虑或谈论这一点时，我认为社区......
Shreya Shankar: Most of the community is beginners or people who don't know about evals and want to learn about it, and it sends the wrong message there.
Shreya Shankar：社区中的大多数人都是初学者或不了解评估但想要了解它的人，并且它在那里发出了错误的信息。
Now, I don't know what Claude Code is doing, obviously, but I would be willing to bet money that they're doing something in the form of evals.
现在，显然我不知道 Claude Code 在做什么，但我愿意打赌他们正在以评估的形式做一些事情。
Hamel Husain: We'll also say that coding agents are fundamentally very different than other AI products, because the developer is the domain expert, so you can short circuit a lot of things, and also, the developer is using it all day long, so there's a type of dogfooding and type of domain expertise that is...
Hamel Husain：我们还会说，编码代理从根本上与其他人工智能产品有很大不同，因为开发人员是领域专家，所以你可以短路很多东西，而且，开发人员整天都在使用它，所以有一种类型的狗食和领域专业知识......
You can collapse the activities, you don't need as much data, you don't need as much feedback or exploration, because you know, so your eval process should look different.
你可以折叠活动，你不需要那么多的数据，你不需要那么多的反馈或探索，因为你知道，所以你的评估过程应该看起来不同。
Lenny Rachitsky: Because you're seeing the code, you see the code it's generating.
Lenny Rachitsky：因为您看到的是代码，所以您看到的是它生成的代码。
You can tell, "This is great, this is terrible."
你可以说：“这太棒了，这太糟糕了。”
Hamel Husain: Yeah, yeah.
哈梅尔·侯赛因：是的，是的。
I think a lot of people had generalized coding agents, because coding agents are the first AI product released into the wild, and I think it's a mistake to try to generalize that at large.
我认为很多人都有通用化的编码代理，因为编码代理是第一个发布到野外的人工智能产品，我认为试图概括它是一个错误。
Shreya Shankar: The other thing is, yeah, engineers have a dogfooding personality.
Shreya Shankar：另一件事是，工程师具有自我测试的个性。
There are plenty of applications where people are trying to build AI in certain domains and they don't have dogfooding for doctors, for example, or not out there trying to get all the most incorrect advice from AI and be tolerant and receptive to that.
在很多应用程序中，人们试图在某些领域构建人工智能，但他们并没有对医生进行测试，或者没有试图从人工智能中获得所有最不正确的建议，并对此保持宽容和接受。
It's very important to keep, I think these nuanced things in mind.
我认为记住这些微妙的事情非常重要。
Lenny Rachitsky: What I'm hearing from you, Shreya, interestingly, is that if humans on the team are doing very close data analysis, error analysis, dogfooding like crazy, and essentially, they're the human evals and you're describing that as that's within the umbrella of evals.
Lenny Rachitsky：Shreya，有趣的是，我从你那里听到的是，如果团队中的人员正在进行非常密切的数据分析、错误分析、疯狂的测试，从本质上讲，他们就是人类评估，而你将其描述为评估范围内的情况。
You could do it that way if you have time and motivation to do that, or you could set these things up to be automatic.
如果您有时间和动力，您可以这样做，或者您可以将这些事情设置为自动。
Shreya Shankar: Absolutely, it's also about the skills.
Shreya Shankar：当然，这也与技能有关。
People who work at Anthropic are very, very highly skilled.
在 Anthropic 工作的人技术非常非常高。
They've been trained in data analysis or software engineering or AI, and whatnot.
他们接受过数据分析、软件工程或人工智能等方面的培训。
You can get there, anyone can get there, of course, by learning the concepts, but most people don't have that skill right now.
当然，通过学习这些概念，你可以做到这一点，任何人都可以做到这一点，但大多数人现在不具备这种技能。
Hamel Husain: Dogfooding is a dangerous one, only because a lot of people will say they're dogfooding.
Hamel Husain：内部测试是一种危险的行为，只是因为很多人会说他们正在内部测试。
They're like, "Yeah, we dogfooded," but are they, really?
他们会说，“是的，我们进行了测试”，但真的是这样吗？
A lot of people aren't really dogfooding it at that visceral level that you would need to close that feedback loop.
很多人并没有真正在内心深处进行测试，而你需要关闭反馈循环。
That's the only caveat I would add.
这是我要补充的唯一警告。
Lenny Rachitsky: There's also this, feels like, straw man argument of evals versus A-B tests.
Lenny Rachitsky：还有一个评估与 A-B 测试的稻草人争论。
Talk about your thoughts there, because that feels like a big part of this debate.
在那里谈谈你的想法，因为这感觉是这场辩论的重要组成部分。
People are having like, "Do you need evals if you have A-B tests that are testing production level metrics?"
人们会问，“如果你有 A-B 测试来测试生产级别指标，你还需要评估吗？”
Shreya Shankar: A-B tests are, again, another form of evals ,I imagine, right?
Shreya Shankar：我想，A-B 测试又是另一种形式的评估，对吗？
When you're doing an A-B test, you have two different experimental conditions and then you have a metric that quantifies the success of something, and you're comparing the metric.
当您进行 A-B 测试时，您有两种不同的实验条件，然后您有一个量化某项成功的指标，并且您正在比较该指标。
Again, an eval in our mind is systematic measurement of quality, some metric.
同样，我们心中的评估是对质量的系统测量，某种指标。
You can't really do an A-B test without the eval to compare, so maybe we just have a different weird take on it.
如果没有评估来进行比较，你就无法真正进行 A-B 测试，所以也许我们只是对此有不同的奇怪看法。
Lenny Rachitsky: Yeah, okay.
莱尼·拉奇茨基：是的，好的。
What I'm hearing is you consider A-B tests as part of the suite of evals that you do.
我听到的是，您将 A-B 测试视为您所做的评估套件的一部分。
I think when people think A-B tests, it's like we're changing something in the product, we're going to see if this improves some metric we care about.
我认为，当人们考虑 A-B 测试时，就好像我们正在改变产品中的某些内容，我们将看看这是否会改善我们关心的某些指标。
Is that enough?
够了吗？
Why do we need to test every little feature?
为什么我们需要测试每一个小功能？
If it's impacting a metric we care about as a business, we have a bunch of A-B tests that are just constantly running.
如果它影响了我们作为企业关心的指标，我们就会进行一系列不断运行的 A-B 测试。
Shreya Shankar: This is now a great point.
Shreya Shankar：现在这是一个很好的观点。
I think a lot of people prematurely do A-B tests, because they've never done any error analysis in the first place.
我认为很多人过早地进行 A-B 测试，因为他们从一开始就没有做过任何错误分析。
They just have hypothetically come up with their product requirements and they believe that, "We should test these things," but it turns out, when you get into the data, as Hamel showed, that the errors that you're seeing are not what you thought what the errors might be.
他们只是假设地提出了他们的产品要求，并且他们相信，“我们应该测试这些东西”，但事实证明，当你进入数据时，正如哈梅尔所表明的那样，你看到的错误并不是你想象的错误。
They were these weird handoff issues or, I don't know, the text message thing was strange.
这些都是奇怪的交接问题，或者，我不知道，短信的事情很奇怪。
I would say that, if you're going to do A-B tests and they're powered by actual error analysis as we've shown today, then that's great, go do it.
我想说的是，如果您要进行 A-B 测试，并且它们由我们今天展示的实际错误分析提供支持，那就太好了，那就去做吧。
But if you're just going to do them, which we find that people try to do, just want to do them based on what you hypothetically think is what is important, then I would encourage people to go and rethink that and ground your hypotheses.
但是，如果你只是想做这些事情，我们发现人们试图这样做，只是想根据你假设的认为重要的事情来做这些事情，那么我会鼓励人们去重新思考这一点并为你的假设奠定基础。
Lenny Rachitsky: Do you have thoughts on what Statsig is going to do at OpenAI?
Lenny Rachitsky：您对 Statsig 将在 OpenAI 做什么有什么想法吗？
Is there anything there that's interesting?
那里有什么有趣的事情吗？
That was a big deal, a huge acquisition.
这是一件大事，一次巨大的收购。
A- B test company people are like, "A-B test, the future." Thoughts?
A-B 测试公司的人喜欢说，“A-B 测试，未来。”想法？
Hamel Husain: Just to add to the previous question a little bit, why is there this debate, A-B testing versus evals?
Hamel Husain：稍微补充一下之前的问题，为什么会出现 A-B 测试与评估的争论？
I think, fundamentally, evals is...
我认为，从根本上来说，评估是......
People are trying to wrap their head around how to improve their applications and fundamentally need to do...
人们正在努力思考如何改进他们的应用程序，并且从根本上需要做......
Data science is useful in products.
数据科学在产品中很有用。
Looking at data, doing data analytics.
查看数据，进行数据分析。
There's many different suite of tools, and you don't need to invent anything new.
有许多不同的工具套件，您不需要发明任何新东西。
Sure, you don't need necessarily the whole breadth of data science, and it looks slightly different, just slightly, with LLMs.
当然，你不一定需要数据科学的全部内容，而且法学硕士看起来略有不同，只是略有不同。
Your tactics might be different, so really what it is is using analytic tools to understand your product.
您的策略可能会有所不同，所以实际上是使用分析工具来了解您的产品。
Now, people say the word "Evals," trying to carve out this new thing, and saying evals and then A-B testing, but if you zoom out, it's the same data science as before, and I think that's what's causing the confusion is, "Hey, we need data science thinking," and AI product is helpful to have that thinking in AI products like it is in any product is my take on that.
现在，人们说“Evals”这个词，试图开创这个新事物，先说评估，然后进行 A-B 测试，但如果你缩小范围，它和以前一样是数据科学，我认为这就是造成混乱的原因，“嘿，我们需要数据科学思维”，而人工智能产品有助于在人工智能产品中像在任何产品中那样思考，这是我的看法。
Lenny Rachitsky: That's a really good take, I think just the word "Evals" triggers people now.
Lenny Rachitsky：这是一个非常好的想法，我认为现在只要“Evals”这个词就能触发人们。
Lenny Rachitsky: If you just call it, "We're just doing error analysis, doing data science to understand where our product breaks and just setting up tests to make sure we know-"
Lenny Rachitsky：如果你只是这么称呼它，“我们只是进行错误分析，进行数据科学来了解我们的产品在哪里出现问题，并设置测试以确保我们知道——”
Shreya Shankar: That's boring, sounds boring.
Shreya Shankar：这很无聊，听起来很无聊。
No, no, no. We need a mysterious term, like "Evals," to really get the momentum going.
不，不，不。我们需要一个神秘的术语，比如“Evals”，才能真正推动这一势头。
Your question about Statsig, I think it's very exciting.
你关于 Statsig 的问题，我认为这非常令人兴奋。
To be honest, I don't know much about it, because I just imagine that they're this company that...
说实话，我对此了解不多，因为我只是想象他们是这家公司......
There's a tool that many people use, and maybe it just so happened that OpenAI acquired them.
有一个很多人都使用的工具，也许 OpenAI 恰好收购了它们。
I'm sure they've been using them in the past, I'm sure OpenAI's competitors are using Statsig as well, so maybe there is something strategic in that acquisition.
我确信他们过去一直在使用它们，我确信 OpenAI 的竞争对手也在使用 Statsig，所以这次收购可能具有战略意义。
I have no idea, I don't know anything there, but I think those are really the bigger questions for me than, "Is this fundamentally changing A-B testing or making evals more of a priority?" I think they've always been a priority, I think OpenAI has always been doing some form of them, and OpenAI has gone so far, historically speaking, as to go and look at all the Twitter sentiment and try to do some retrospective on that, and then tie that back to their products.
我不知道，我对此一无所知，但我认为这些对我来说确实是比“这是否从根本上改变 A-B 测试或使评估更加优先？”更大的问题。我认为它们一直是一个优先事项，我认为 OpenAI 一直在做某种形式的事情，从历史上看，OpenAI 已经走了这么远，去查看所有 Twitter 情绪并尝试对此进行一些回顾，然后将其与他们的产品联系起来。
Certainly, they're doing-
当然，他们正在做——
Shreya Shankar: Then, tie that back to their products.
Shreya Shankar：然后，将其与他们的产品联系起来。
Certainly, they're doing some amount of evals before they ship their new foundation models, but they're going so much beyond and being like, "Okay, let's find all the tweets that are complaining about it, all the Reddit threads that are complaining about it, and go try to figure out what's going on." It goes to show that evals are very, very important.
当然，他们在发布新的基础模型之前会进行一些评估，但他们所做的远远超出了这一范围，就像，“好吧，让我们找到所有抱怨它的推文，所有抱怨它的 Reddit 线程，然后尝试找出发生了什么。”这表明评估非常非常重要。
No one has really figured it out yet.
目前还没有人真正弄清楚这一点。
People are using all the available sources signal that they can to improve their products.
人们正在使用所有可用的资源来表明他们可以改进他们的产品。
Hamel Husain: What I'll say is I'm really hopeful that it might shift or create a focus within OpenAI, hopefully.
Hamel Husain：我想说的是，我真的希望它能够在 OpenAI 中转移或创造一个焦点。
Up until now, a lot of the big labs understandably focused on general benchmarks like MMLU score, human eval, things like that, which are very important for foundation models.
到目前为止，许多大型实验室都专注于通用基准，例如 MMLU 分数、人类评估等，这对于基础模型非常重要，这是可以理解的。
Those not very related to product specific evals, like the ones we talked about today, but handoff and stuff like that, they tend not to correlate.
那些与产品特定的评估不太相关，就像我们今天讨论的那些，但是交接之类的东西，它们往往不相关。
Shreya Shankar: Yeah, they don't correlate with math problem-solving, sorry to say.
Shreya Shankar：是的，遗憾的是，它们与解决数学问题无关。
Hamel Husain: Exactly.
哈梅尔·侯赛因：没错。
If you look at the eval products, let's say the ones up until recently that some of the big labs have, they don't have error analysis.
如果你看一下评估产品，比如说一些大型实验室直到最近才有的产品，他们没有错误分析。
They have a suite of generic tools, cosine similarity, hallucination score, whatever, and that doesn't work.
他们有一套通用工具，余弦相似度，幻觉分数，等等，但那不起作用。
It's a good first stab at it.
这是一个很好的第一次尝试。
It's okay.
没关系。
At least you're doing something, getting people, maybe it's like getting people look at data.
至少你正在做一些事情，吸引人们，也许就像让人们查看数据一样。
But eventually, what we hope to see is, okay, a bit more data science thinking in this eval process.
但最终，我们希望看到的是，在这个评估过程中，有更多的数据科学思维。
That's hopefully the tools we'll get to.
希望这就是我们能够获得的工具。
Shreya Shankar: Yeah, Pamela and I should not be the only two people on the planet that are promoting a structured way of thinking about application specific evals.
Shreya Shankar：是的，帕梅拉和我不应该是这个星球上唯一提倡以结构化方式思考应用程序特定评估的两个人。
It's mind-boggling to me.
这让我难以置信。
Why are we the only two people doing this the whole world?
为什么全世界只有我们两个人这样做？
What's wrong?
怎么了？
I hope that we're not the only people and that more people catch on.
我希望我们不是唯一的人，希望有更多的人能够接受。
Lenny Rachitsky: The fact that your course on Maven is the number one highest grossing course in Maven, clearly there's demand and interest, and there's more people I think on your side.
Lenny Rachitsky：事实上，您的 Maven 课程是 Maven 中收入最高的课程，显然存在需求和兴趣，而且我认为有更多的人站在您这边。
Interestingly, just as an example you've been sharing on Twitter that I think is informative, everyone's been saying how cloud code doesn't care about evals.
有趣的是，正如您在 Twitter 上分享的一个我认为内容丰富的示例，每个人都在说云代码如何不关心评估。
They're all about vibes, and everyone's like, and they're the best coding agent out there, so clearly, this is right.
他们都是关于共鸣的，每个人都喜欢，他们是最好的编码代理，所以很明显，这是对的。
More recently, there's all this talk about Codex, OpenAI Codex being better and everyone's switching and they're so pro evals.
最近，有很多关于 Codex 的讨论，OpenAI Codex 变得更好，每个人都在转换，而且他们非常专业。
Shreya Shankar: It gets me every time.
Shreya Shankar：每次都让我感动。
The Internet's so inconsistent.
互联网是如此的不一致。
My favorite thing was yesterday, I believe, a couple of lab mates and I were out getting dessert or something, and somebody said like, "Oh, do you like Codex or Claude better or whatever?" The other person said, "Oh, I like Claude." Then, someone else said, "But the new version of Codex is better." Then, the first person said, "Oh, but the last I checked was two days ago, so maybe my thoughts, maybe I'm not up-to-date." I was like, "Oh, my God."
我相信，我最喜欢的事情是昨天，我和几个实验室伙伴出去吃甜点什么的，有人说，“哦，你更喜欢 Codex 还是 Claude 还是其他什么？”另一个人说：“哦，我喜欢克劳德。”然后，又有人说：“但是新版的Codex更好了。”然后，第一个人说：“哦，但我最后一次检查是在两天前，所以也许是我的想法，也许我不是最新的。”我当时想，“哦，天哪。”
Lenny Rachitsky: So true, so true.
莱尼·拉奇茨基：太真实了，太真实了。
This is the world we live in.
这就是我们生活的世界。
Oh, my God.
哦，天啊。
Okay.
好的。
I want to ask about just top misconceptions people have with evals and top tips and tricks for being successful.
我想问的是人们对评估的主要误解以及成功的主要提示和技巧。
Maybe just share one or two each of each.
也许每个人只分享一两个。
Let me just start with misconceptions, and maybe I'll go to the Hamel first.
让我从误解开始，也许我会先去哈梅尔。
Just what are a couple of the most common misconceptions people have with eval still?
人们对 eval 最常见的误解是什么？
Hamel Husain: The top one is, "Hey, I can just buy a tool, plug it in, and it'll do the eval for you.
Hamel Husain：最重要的是，“嘿，我只需购买一个工具，将其插入，它就会为您进行评估。
Why do I have to worry about this?
我为什么要担心这个？
We live in the age of AI.
我们生活在人工智能时代。
Can't the AI just eval it?" That's the most common misconception, and people want that so much that people do sell it, but it doesn't work.
人工智能就不能评估它吗？”这是最常见的误解，人们非常想要它，以至于人们确实出售它，但它不起作用。
That's the first one.
这是第一个。
Lenny Rachitsky: Shoot, many humans are still great.
莱尼·拉奇茨基：天啊，很多人仍然很伟大。
I think that's great news.
我认为这是个好消息。
Hamel Husain: The second one that I see a lot is, "Hey, just not looking at the data." In my consulting, people come to me with problems all the time, and the first thing I'll say is, "Let's go look at your traces." You can see their eyes pop open and be like, "What do you mean?" I'm like, "Yeah, let's look at it right now." They're surprised that I am going to go look at individual traces, and it always 100% of the time learn a lot and figure out what the problem is.
Hamel Husain：我经常看到的第二个是，“嘿，只是不看数据。”在我的咨询中，总是有人来找我提出问题，我会说的第一句话是：“让我们看看你的痕迹。”你可以看到他们的眼睛突然睁开，问道：“你什么意思？”我说：“是的，我们现在就看一下。”他们很惊讶我会去查看各个痕迹，而且总是 100% 能学到很多东西并找出问题所在。
I think people just don't know how powerful looking at the data is like we showed on this podcast.
我认为人们只是不知道查看数据有多么强大，就像我们在这个播客中展示的那样。
Lenny Rachitsky: Is there anything else or those are the ones solve those problems.
Lenny Rachitsky：还有其他什么或者这些可以解决这些问题吗？
Shreya Shankar: Oh, those are definitely...
Shreya Shankar：哦，那些绝对是......
Then, I guess the third one I would add is, there's no one correct way to do evals.
然后，我想我要添加的第三个是，没有一种正确的方法可以进行评估。
There are many incorrect ways of doing evals, but there are also many correct ways of doing it.
进行评估有很多不正确的方法，但也有很多正确的方法。
You got to think about where you are at with your product, how much resources you have, and figure out the plan that works best for you.
您必须考虑您的产品处于什么位置，您拥有多少资源，并找出最适合您的计划。
It'll always involve some form of error analysis as we showed today, but how you operationalize those metrics is going to change based on where you're at.
正如我们今天所展示的，它总是涉及某种形式的错误分析，但是您如何操作这些指标将根据您所处的位置而变化。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
Okay.
好的。
What are a couple of just tips and tricks you want to leave people with as they start on their eval journey or just try to get better at something they're already doing?
当人们开始评估之旅或只是试图在他们已经在做的事情上做得更好时，您想给他们留下哪些提示和技巧？
Shreya Shankar: Tip number one is just don't be alarmed or don't be scared of looking at your data.
Shreya Shankar：第一条建议是不要惊慌，也不要害怕查看您的数据。
The process, we try to make it as structured as possible.
在这个过程中，我们尝试使其尽可能结构化。
There are inevitably questions that are going to come up.
不可避免地会出现一些问题。
That's totally fine.
那完全没问题。
You might feel like you're not doing it perfectly.
你可能会觉得自己做得并不完美。
That's also fine.
那也好。
The goal is not to do evals perfectly, it's to actionably improve your product.
我们的目标不是完美地进行评估，而是切实可行地改进您的产品。
We guarantee you, no matter what you do, if you're doing parts of these process, you're going to find ways of actionable improvement, and then you're going to iterate on your own process from there.
我们向您保证，无论您做什么，如果您正在执行这些流程的一部分，您都会找到可行的改进方法，然后您将从那里迭代您自己的流程。
Shreya Shankar: The other tip that I would say is, we are very pro-AI.
Shreya Shankar：我想说的另一个建议是，我们非常支持人工智能。
Use LLMs to help you organize any thoughts that you have throughout this entire process.
使用法学硕士可以帮助您整理整个过程中的任何想法。
This could be everything ranging from initial product requirements.
这可能是从最初的产品需求到一切。
Figure out how to organize them for yourself.
弄清楚如何为自己组织它们。
Figure out how to improve on that product requirements doc based on the open codes that you've created.
了解如何根据您创建的开放代码改进该产品需求文档。
Don't be afraid to use AI in ways that present information better for you.
不要害怕以更好地为您呈现信息的方式使用人工智能。
Lenny Rachitsky: Sweet, so don't be scared.
Lenny Rachitsky：亲爱的，所以不要害怕。
Use LLMs as much as you can throughout the process.
在整个过程中尽可能多地使用法学硕士。
Lenny Rachitsky: Right.
莱尼·拉奇茨基：对。
Okay, great.
好的，太好了。
There's still jobs.
工作还是有的
It's great.
这很棒。
Hamel.
哈梅尔。
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
Let me actually share my screen, because I want to show something.
让我实际分享我的屏幕，因为我想展示一些东西。
To piggyback of what Shreya said is, if you heard any phrase in this podcast, you've probably heard look at your data more than anything else.
根据 Shreya 所说，如果您在此播客中听到过任何短语，那么您可能最常听到的是“查看数据”。
It's so important that we teach that you should create your own tools to make it as easy as possible.
我们教导您应该创建自己的工具以使其尽可能简单，这一点非常重要。
I showed you some tools when we're going through the live example of how to annotate data.
当我们查看如何注释数据的实时示例时，我向您展示了一些工具。
Most of the people I work with, they realize how important this is and they vibe code their own tools, or we shouldn't say vibe code.
与我一起工作的大多数人都意识到这是多么重要，并且他们使用自己的工具进行振动代码，或者我们不应该说振动代码。
They make their own tools, and it's cheaper than ever before because you have AI that can help you.
他们制造自己的工具，而且比以往任何时候都便宜，因为你有人工智能可以帮助你。
Hamel Husain: AI is really good at creating simple web applications that can show you data, that can write to a database.
Hamel Husain：人工智能非常擅长创建简单的 Web 应用程序，可以向您显示数据，可以写入数据库。
It's very simple.
这很简单。
For the Nurture Boss use case, we wanted to remove all the friction of looking at data.
对于培养老板用例，我们希望消除查看数据的所有麻烦。
What you see here is just some screenshots of what the application that they created looks like.
您在这里看到的只是他们创建的应用程序的一些屏幕截图。
It's just, "Okay, they have the different channels, voice, email, text.
只是，“好吧，他们有不同的渠道、语音、电子邮件、文本。
They have the different threads, they hid the system prompt by default." Little quality of life improvements.
他们有不同的线程，他们默认隐藏系统提示符。”生活质量改善甚微。
Then, they actually have this axial coding part here where you can see in red the count of different errors.
然后，他们实际上在这里有这个轴向编码部分，您可以在其中以红色看到不同错误的计数。
They automated that part in a nice way and they created this within a few hours.
他们以一种很好的方式自动化了该部分，并在几个小时内创建了它。
It's really hard to have a one size fits all thing for looking at your data.
很难找到一种适合所有情况的工具来查看您的数据。
You don't have to go here immediately, but something to think about is make it as easy as possible because, again, it's the most powerful activity that you can engage in.
你不必立即去这里，但要考虑的是让它尽可能简单，因为这又是你可以参与的最强大的活动。
It's the highest ROI activity you can engage in.
这是您可以参与的投资回报率最高的活动。
With AI, yeah, just remove all the friction.
是的，有了人工智能，就可以消除所有的摩擦。
Lenny Rachitsky: That's amazing.
莱尼·拉奇茨基：太棒了。
Again, I think that ROI piece is so important.
我再次认为投资回报率非常重要。
We haven't even touched on this enough.
我们甚至还没有足够地触及这一点。
The goal here is to make your product better, which will make your business more successful.
这里的目标是让您的产品变得更好，这将使您的业务更加成功。
This isn't just a little exercise to catch bugs and things like that.
这不仅仅是捕捉错误之类的小练习。
This is the way to make AI products better because the experience is how users interact with your AI.
这是让人工智能产品变得更好的方法，因为体验是用户与人工智能交互的方式。
Hamel Husain: Absolutely.
哈梅尔·侯赛因：当然。
If any, we teach our students, "Hey, when you're doing these evals, if you see something that's wrong, just go fix it." The whole point is not to have evals, a beautiful eval suite, where you can point at it, edit it and say, Oh, look at my evals." No, just fix your application, make it better.
如果有的话，我们会教我们的学生，“嘿，当你进行这些评估时，如果你发现有问题，就去修复它。”重点不是要有评估，一个漂亮的评估套件，你可以指着它，编辑它，然后说，哦，看看我的评估。”不，只需修复你的应用程序，使其更好。
If it's obvious, do it.
如果很明显，那就去做吧。
Totally agree with you.
完全同意你的观点。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
A question I didn't ask, but this is I think something people are thinking about.
我没有问这个问题，但我认为这是人们正在思考的问题。
How long do you spend on this?
你要花多长时间在这上面？
How long does it usually take to do?
一般需要多长时间做完？
The first time
第一次
Shreya Shankar: I can answer for myself for applications that I work with.
Shreya Shankar：我可以为自己使用的应用程序负责。
Usually, I'll spend three to four days really working with whoever to do initial rounds of error analysis.
通常，我会花三到四天的时间与任何人一起进行最初几轮的错误分析。
A lot of labeling, feel like we're in a good place to create the spreadsheet that Hamel had and everyone's on-board and convinced, and even a few LLM judge evaluators.
很多标签，感觉我们是在一个很好的地方来创建哈梅尔拥有的电子表格，每个人都参与并相信，甚至还有一些法学硕士评委评估员。
But this is one-time cost.
但这是一次性成本。
Once I figured out how to integrate that in unit tests, or I have a script that automatically runs it on samples and I'll create a Cron Job to just do this every week.
一旦我弄清楚如何将其集成到单元测试中，或者我有一个可以在示例上自动运行它的脚本，我将创建一个 Cron Job 每周执行此操作。
I would say it's like, I don't know, I find myself probably spending more time looking at data because I'm just data hungry like that.
我会说，我不知道，我发现自己可能会花更多时间查看数据，因为我就是这样渴望数据。
I'm so curious.
我很好奇。
Shreya Shankar: I'm like, I've gained so much from this process and it's put me above and beyond in any of my collaborations with folks, so I want to keep doing it, but I don't have to.
Shreya Shankar：我想，我从这个过程中收获了很多，它使我在与人们的任何合作中都脱颖而出，所以我想继续这样做，但我不必这样做。
I would say maybe 30 minutes a week after that.
我想说之后每周大概 30 分钟。
Lenny Rachitsky: It's a week essentially, a week essentially upfront, and then 30 minutes to keep improving on adding to your suite?
Lenny Rachitsky：这基本上是一周，基本上是提前一周，然后用 30 分钟不断改进您的套件？
Shreya Shankar: Yeah, it's really not that much time.
Shreya Shankar：是的，时间确实不多。
I think people just get overwhelmed by how much time they spend up front and then thinking that they have to keep doing this all the time.
我认为人们只是对他们预先花费的时间感到不知所措，然后认为他们必须一直这样做。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
Is there anything else that you wanted to share or leave listeners with?
您还有什么想与听众分享或留给听众的吗？
Anything else you wanted to double down as a point before we get to a very exciting lightning round?
在我们进入激动人心的闪电回合之前，您还有什么想要加倍的吗？
Hamel Husain: I would say this process is a lot of fun, actually.
Hamel Husain：实际上，我想说这个过程很有趣。
It's like, okay, you're looking at data.
就像，好吧，你正在查看数据。
Oh, it sounds like you're annotating things.
哦，听起来你在注释东西。
Okay.
好的。
Actually, I was just looking at a client's data yesterday, the same exact process.
事实上，我昨天只是在看客户的数据，同样的过程。
It's a application that sends emails, recruiting emails to try to get candidates to apply for a job.
这是一个发送电子邮件、招聘电子邮件以尝试让候选人申请工作的应用程序。
We decided to start looking at traces.
我们决定开始寻找痕迹。
We jumped right into it.
我们直接跳进去了。
"Hey, let's look at your traces." We looked at a trace, the first thing I saw was this email that is worded, "Given your background, blah, blah, blah, blah, blah." I asked the person right away, and this is where putting your product hat on and just being critical, and this is where the fun part is.
“嘿嘿，让我们看看你的踪迹吧。”我们查看了痕迹，我看到的第一件事是这封电子邮件，其措辞是“鉴于您的背景，等等，等等，等等。”我立即询问了这个人，这就是戴上产品帽子并保持批评的地方，这就是有趣的部分。
Hamel Husain: I said, "You know what?
哈梅尔·侯赛因：我说，“你知道吗？
I hate this email.
我讨厌这封电子邮件。
Do you like the email, given your background?" When I receive a message given your background, comma, I just delete that.
考虑到您的背景，您喜欢这封电子邮件吗？”当我收到一条考虑到您的背景的消息时，我会删除它。
I'm like, "What is this, given your background with machine learning and blah blah?" I'm like, "This is a generic thing." I asked the person like, "Hey, can we do better than this?
我想，“考虑到你的机器学习背景，这是什么？”我想，“这是一个普遍的事情。”我问那个人，“嘿，我们能做得更好吗？
This sounds like generic recruiting." They're like, "Oh, yeah, maybe." Because they were proud of it, they're like, "The AI is doing the right thing, it's sending this email with the right information, with the right link, with the right name, everything." That's where the fun part is, is put your product hat on and get into, is this really good?
这听起来像是一般的招聘。”他们会说，“哦，是的，也许吧。”因为他们为此感到自豪，他们会说，“人工智能正在做正确的事情，它正在发送包含正确信息、正确链接、正确名称等一切的电子邮件。”这就是有趣的部分，就是戴上你的产品帽子并投入其中，这真的好吗？
Lenny Rachitsky: Something I want to make sure we cover before we get to a very exciting lightning round is, this is just scratching the surface of all the things you need to know to do this well.
Lenny Rachitsky：在我们进入非常令人兴奋的闪电回合之前，我想确保我们先介绍一下，这只是你需要知道的所有事情的表面，以便做好这件事。
I think this is the best primer I've ever seen on how to do this well.
我认为这是我见过的关于如何做好这件事的最好的入门读物。
Lenny Rachitsky: But I think we did it.
Lenny Rachitsky：但我认为我们做到了。
But you guys teach a course that goes much, much deeper for people that really want to get good at this and take this really seriously.
但是你们教授的课程对于那些真正想要擅长并认真对待这件事的人来说，要深入得多。
Share what else you teach in the course that we didn't cover, and what else you get as a student being part of the course you teach at Maven.
分享您在课程中教授的其他内容（我们未涵盖的内容），以及作为学生在 Maven 教授的课程中获得的其他内容。
Shreya Shankar: Yeah, I can talk about the syllabus a little bit, and then Hamel can talk about all the perks.
Shreya Shankar：是的，我可以谈谈教学大纲，然后哈梅尔可以谈谈所有的福利。
We go through a lifecycle of error analysis, then automated evaluators, then how to improve your application, how do you create that flywheel for yourself?
我们经历了错误分析的生命周期，然后是自动评估器，然后是如何改进您的应用程序，您如何为自己创建飞轮？
We also have a few special topics that we find pretty much no one has ever heard of or taught before, which is exciting.
我们还有一些特殊主题，我们发现几乎没有人听说过或以前没有教过，这很令人兴奋。
One is, how do you build your own interfaces for error analysis?
一是，如何构建自己的错误分析接口？
We go through actual interfaces that we've built and we also live code them on the spot for new data.
我们会检查我们构建的实际接口，并且还会现场对它们进行实时编码以获取新数据。
We show how we use Claude code cursor, whatever we're feeling in the moment that day to build these interfaces.
我们展示了如何使用 Claude 代码光标，无论我们当天的感受如何来构建这些界面。
Shreya Shankar: We also talk about broadly cost-optimization as well.
Shreya Shankar：我们还广泛讨论了成本优化。
A couple of people that I've worked with, they get to a point where their evals are very good, their product is very good, but it's all very expensive because they're using state-of-the-art models.
与我合作过的几个人，他们的评估非常好，他们的产品非常好，但它们都非常昂贵，因为他们使用的是最先进的模型。
How can we replace certain uses of the most expensive GPT-5, with 5-nano, 4-mini whatnot and save a lot of money, but still maintain the same quality?
我们如何才能用 5-nano、4-mini 之类的材料取代最昂贵的 GPT-5 的某些用途，从而节省大量资金，同时仍保持相同的质量？
We also give some tips for that.
我们还为此提供了一些提示。
Hamel, you're on.
哈梅尔，你来了。
We also have many perks.
我们还有很多福利。
Hamel Husain: Okay, the perks.
哈梅尔·侯赛因：好的，福利。
My favorite perk is there's 160 page book that's meticulously written, that we've created, that walks through the entire process in detail of how to do evals that supplement the course.
我最喜欢的福利是我们精心编写了 160 页的书，它详细介绍了如何进行评估的整个过程，以补充课程。
You don't have to sit there and take all these notes.
您不必坐在那里记下所有这些笔记。
We've done all the hard work for you and we have documented it in detail and organize things.
我们已经为您完成了所有艰苦的工作，并且我们已经详细记录并组织了事情。
That is really useful.
这真的很有用。
Another really interesting thing, and something that I got the idea from you, Lenny, is, okay, this is an AI course.
另一件非常有趣的事情，也是我从你那里得到的想法，莱尼，是，好吧，这是一门人工智能课程。
Education shouldn't be this thing where you are only watching lectures and doing homework assignments.
教育不应该只是看讲座和做作业。
Students should have access to an AI that also helps them.
学生应该能够使用对他们也有帮助的人工智能。
What we have done is we've, just like there's the LennyBot that you have.
我们所做的就是我们所做的，就像您拥有的 LennyBot 一样。
Hamel Husain: Yeah, lennybot.com, we have made the same thing with the same software that you're using, and we have put everything we've ever said about evals into that.
Hamel Husain：是的，lennybot.com，我们使用您正在使用的相同软件做了同样的事情，并且我们已经将我们所说的有关评估的所有内容都放入其中。
Every single lesson, every office hours, every Discord chat, any blogs, papers, anything that we've ever said publicly and within our course, we've put it in there.
每节课、每个办公时间、每一次 Discord 聊天、任何博客、论文、我们在课程中公开说过的任何内容，我们都把它放在那里。
We've tested it with a bunch of students and they've said it's helpful.
我们已经对一群学生进行了测试，他们说这很有帮助。
We're giving all students 10 months free unlimited access to that alongside the course.
我们为所有学生提供 10 个月的免费无限制访问时间和课程。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
Then, you'll charge for that later down the road?
那么，您稍后会为此付费吗？
Hamel Husain: I have no idea.
哈梅尔·侯赛因：我不知道。
I just take one month at a time.
我每次只用一个月。
I don't know where we're going with that.
我不知道我们要去哪里。
Lenny Rachitsky: Eight months and then we'll have to figure it out.
Lenny Rachitsky：八个月之后我们就得想办法了。
I was thinking this whole interview should have just been our bots talking to each other.
我在想整个采访应该只是我们的机器人互相交谈。
Shreya Shankar: That's amazing.
Shreya Shankar：太棒了。
I would watch that, only for 10 minutes then I don't know what they're talking about.
我会看，只看10分钟，然后我不知道他们在说什么。
Lenny Rachitsky: Yeah, maybe 30 seconds.
Lenny Rachitsky：是的，也许 30 秒。
Do you guys train it on the voice mode, by the way?
顺便问一下，你们在语音模式下训练它吗？
That's my favorite feature of Delphi's product.
这是德尔福产品中我最喜欢的功能。
If not, you should do that.
如果没有，你应该这样做。
Hamel Husain: Oh, I think, I can't remember, I should look at it.
Hamel Husain：哦，我想，我不记得了，我应该看看。
Lenny Rachitsky: You definitely should.
Lenny Rachitsky：你绝对应该这样做。
Now that we have this podcast episode, you could use this content to train it.
现在我们有了这个播客剧集，您可以使用此内容来训练它。
It's 11Labs powered.
它由 11Labs 提供支持。
It's so good.
太好了。
Okay, so how do they get to...
好吧，那么他们如何去...
I guess that's okay.
我想没关系。
They get to that once they become, enter your course.
一旦他们成为了，就进入你的课程。
Shreya Shankar: Yeah, sign up for the course and then you'll get a bunch of emails.
Shreya Shankar：是的，注册课程，然后你会收到一堆电子邮件。
Everything will be clear, hopefully.
希望一切都会清楚。
Shreya Shankar: We also have a Discord of all the students who have ever taken the class.
Shreya Shankar：我们还有一个关于所有参加过该课程的学生的 Discord。
That Discord is so active.
那个不和谐是如此活跃。
I can't go on vacation without getting notified on the plane.
如果没有在飞机上得到通知，我就不能去度假。
Lenny Rachitsky: Bittersweet, bittersweet.
莱尼·拉奇茨基：苦乐参半，苦乐参半。
Incredible.
难以置信。
Okay.
好的。
With that, we've reached our very exciting lightning round.
至此，我们进入了非常激动人心的闪电回合。
I've got five questions for you.
我有五个问题要问你。
Are you ready?
你准备好了吗？
Lenny Rachitsky: Let's do it.
莱尼·拉奇茨基：让我们开始吧。
Okay.
好的。
I'm going to bounce between you two.
我会在你们两个之间跳来跳去。
Share something if you want.
如果你愿意的话，可以分享一些东西。
You can pass if you want.
如果你愿意的话可以通过。
First question, Shreya, what are two or three books that you find yourself recommending most to other people?
第一个问题，Shreya，您发现自己最向其他人推荐的两三本书是什么？
Shreya Shankar: I like to recommend a fiction book because life is about more than evals.
Shreya Shankar：我喜欢推荐一本小说，因为生活不仅仅是评估。
Recently, I read Pachinko by Min Jin Lee.
最近，我读了李敏珍的弹珠机。
A really great book.
一本非常棒的书。
Then, I also am currently reading Apple in China, which the name of the author is slipping my mind, but this is more of an exposition, written by a journalist on how Apple did a lot of manufacturing processes in Asia over the last couple, several decades.
然后，我目前也在阅读《苹果在中国》，作者的名字已经记不清了，但这更多的是一篇由记者撰写的阐述，讲述苹果在过去几十年里如何在亚洲完成大量制造流程。
Very eye-opening.
非常令人大开眼界。
Hamel Husain: Yeah, I have them right here.
Hamel Husain：是的，我就在这里。
I'm a nerd.
我是个书呆子。
Okay, so I'm not as cool as Shreya is.
好吧，所以我不像Shreya那么酷。
I actually have textbooks, which are my favorite.
我实际上有教科书，这是我最喜欢的。
This one is a very classic one, Machine Learning by Mitchell.
这是非常经典的一篇，Mitchell 的《机器学习》。
Now, it's theoretical, but the thing I like about it is it really drives home the fact that Occam's razor is prevalent not only in science, but also in machine learning and AI.
现在，它是理论性的，但我喜欢它的一点是，它确实让人们明白了这样一个事实：奥卡姆剃刀不仅在科学中普遍存在，而且在机器学习和人工智能中也普遍存在。
A lot of times the simplest, and also engineering, so a lot of times the simpler approach generalizes better.
很多时候是最简单的，而且也是工程性的，所以很多时候更简单的方法概括得更好。
That's the thing I internalize deeply from that book.
这就是我从那本书中深深内化的东西。
I also really like this one.
我也非常喜欢这个。
Another textbook.
又一本教科书。
I told you I'm a nerd.
我告诉过你我是个书呆子。
This is also a very old one, and this is Norvig algorithms.
这也是一个非常古老的算法，这就是 Norvig 算法。
I really like it because it's just human ingenuity and it's lots of clever useful things in computing.
我真的很喜欢它，因为它是人类的聪明才智，并且是计算中许多聪明有用的东西。
Shreya Shankar: They're down the street, him and Berkeley.
Shreya Shankar：他和伯克利就在街上。
Lenny Rachitsky: The people that did that research?
Lenny Rachitsky：做这项研究的人？
Lenny Rachitsky: Super cool.
莱尼·拉奇茨基：超级酷。
Oh, man, nerds, I love it.
哦，伙计，书呆子，我喜欢它。
Okay, next question.
好的，下一个问题。
Favorite recent movie or TV show?
最近最喜欢的电影或电视节目？
I'll jump to Hamel first.
我先跳到哈梅尔。
Hamel Husain: Okay, so I'm a dad of two parents.
哈梅尔·侯赛因：好的，我是两个父母的父亲。
I have two parents.
我有两个父母。
Sorry, two kids.
对不起，两个孩子。
Yeah, I'm a dad of two kids, and I don't really get the time to watch any TV or movies, so I watch whatever my kids are watching.
是的，我是两个孩子的父亲，我真的没有时间看任何电视或电影，所以我就看我的孩子们在看的东西。
I've watched Frozen three times in the last week.
上周我看了三遍《冰雪奇缘》。
Lenny Rachitsky: Only three?
莱尼·拉奇茨基：只有三个？
Oh, okay.
哦，好吧。
In the last week.
在上周。
Okay.
好的。
Lenny Rachitsky: Great, Hamel.
莱尼·拉奇茨基：太好了，哈默尔。
Frozen.
冻结了。
I love it.
我喜欢它。
Okay, Shreya.
好吧，施瑞亚。
Shreya Shankar: Yeah, I don't have kids, so I can give all these amazing answers.
Shreya Shankar：是的，我没有孩子，所以我可以给出所有这些令人惊奇的答案。
Actually, so my husband and I have been watching The Wire recently.
事实上，我和我丈夫最近一直在看《火线》。
We never actually saw it growing up, so we started watching it and it's great.
我们从未真正见过它成长，所以我们开始观看它，它很棒。
Lenny Rachitsky: I feel like everyone goes through that.
Lenny Rachitsky：我觉得每个人都经历过这个。
Eventually in their life they decide, I will watch The Wire.
最终他们决定，在他们的生活中，我会看《火线》。
Shreya Shankar: I know, so we are in that right now.
Shreya Shankar：我知道，所以我们现在就处于这种情况。
Lenny Rachitsky: It's like a year of your life.
Lenny Rachitsky：这就像你生命中的一年。
It's great.
太棒了。
It's such a great show.
这是一场很棒的表演。
Oh, man.
哦，伙计。
But it's so many episodes and everyone's an hour long.
但这集太多了，每个人都一个小时。
Shreya Shankar: We get through two or three a week, so we're very slow.
Shreya Shankar：我们每周要完成两到三个任务，所以我们的速度非常慢。
Lenny Rachitsky: Worth it.
莱尼·拉奇茨基：值得。
Okay, next question.
好的，下一个问题。
Do you have a favorite product you've recently discovered that you really love?
您最近发现了您真正喜欢的最喜欢的产品吗？
We'll start with Shreya.
我们将从施瑞亚开始。
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
I really like using Cursor, honestly.
老实说，我真的很喜欢使用 Cursor。
Now, Claude Code.
现在，克劳德·科德。
I'll say why.
我会说为什么。
I'm a researcher more so than anything else.
我是一名研究员，比其他任何人都重要。
I write papers, I write code, I build systems, everything, and I find that a tool...
我写论文，写代码，构建系统，一切，我发现一个工具......
I'm so bullish on AI assisted coding because I have to wear a lot of hats all the time.
我非常看好人工智能辅助编码，因为我必须一直身兼数职。
Now, I can be more ambitious with the things that I build and write papers about, so I'm super excited about those.
现在，我可以对我构建和撰写论文的事情更加雄心勃勃，所以我对此感到非常兴奋。
Cursor was my entry point into this, but I'm starting to find myself always trying to keep up with all these AI assisted coding tools.
Cursor 是我的切入点，但我开始发现自己总是试图跟上所有这些人工智能辅助编码工具。
Hamel Husain: Yeah, I really like Claude Code and I like it because I feel like the UX is outstanding.
Hamel Husain：是的，我真的很喜欢 Claude Code，因为我觉得它的用户体验非常出色。
There's a lot of love that went into that.
其中蕴含着很多爱。
It's just really impressive as a terminal application that is that nice.
作为一个如此出色的终端应用程序，它确实令人印象深刻。
Lenny Rachitsky: Ironic that you two both love Claude Code when it's just built on vibes.
Lenny Rachitsky：讽刺的是，你们两个都喜欢 Claude Code，因为它只是建立在共鸣之上。
Shreya Shankar: I think it's false.
Shreya Shankar：我认为这是错误的。
It's not just built on vibes.
它不仅仅是建立在共鸣之上。
Lenny Rachitsky: There we go.
莱尼·拉奇茨基：我们开始吧。
Okay, two more questions.
好吧，还有两个问题。
Hamel, do you have a favorite life motto that you find yourself using in coming back to in work or in life?
哈默尔，您有没有最喜欢的人生格言，您发现自己在回到工作或生活中时会使用它？
Hamel Husain: Keep learning in.
哈梅尔·侯赛因：继续学习。
Think like a beginner.
像初学者一样思考。
Shreya Shankar: I like that.
Shreya Shankar：我喜欢这样。
For me, it's to always try to think about the other side's argument.
对我来说，就是要始终尝试思考对方的论点。
I find myself sometimes just encountering arguments on the internet, like this race to eval debates and really think, "Okay, put myself in their shoes.
我发现自己有时只是在互联网上遇到争论，就像这场评估辩论的竞赛，我真的会想，“好吧，设身处地为他们着想。
There's probably a generous take, generous interpretation." I think we're all much stronger together than if we start picking fights.
可能有一个慷慨的接受，慷慨的解释。”我认为我们在一起比我们开始挑衅要强大得多。
My vision for evals is not that Hamel and I become billionaires.
我对评估的愿景并不是哈梅尔和我成为亿万富翁。
It is that everyone can build AI products, and we're all on the same page
每个人都可以构建人工智能产品，而且我们都在同一页面上
Lenny Rachitsky: Slash everyone becomes billionaires.
莱尼·拉奇茨基：让每个人都成为亿万富翁。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
Final question.
最后一个问题。
When I have two guests on, I always like to ask this question and I'll start with Hamel.
当我有两位客人时，我总是喜欢问这个问题，我将从哈梅尔开始。
What's something about Shreya that you like most?
你最喜欢 Shreya 的哪一点？
What do you like most about Shreya?
您最喜欢 Shreya 的哪一点？
I'm going to ask her the same question in reverse.
我会反过来问她同样的问题。
Hamel Husain: Yeah.
哈梅尔·侯赛因：是的。
Shreya is one of the wisest people that I know, especially for being so young relative to me.
施雷亚是我认识的最聪明的人之一，尤其是因为她比我还年轻。
I feel like she's much wiser than I am, honestly, seriously.
老实说，我觉得她比我聪明得多。
She's very grounded and has a very even perspective on things.
她非常脚踏实地，对事物的看法也非常公正。
I'm just really impressed by that all the time.
我一直对此印象深刻。
Shreya Shankar: Yeah.
施瑞亚·香卡：是的。
My favorite thing about Hamel is his energy.
我最喜欢哈梅尔的一点是他的能量。
I don't know anybody who consistently maintains momentum and energy like Hamel does.
我不知道有谁能像哈梅尔一样始终保持动力和活力。
I often think that I would start carrying much less about evals, if not for Hamel.
我经常想，如果没有哈梅尔，我会开始减少有关评估的内容。
Everyone needs a Hamel in their life, for sure.
每个人的生活中都需要一个哈梅尔，这是肯定的。
Lenny Rachitsky: Well, we all have a Hamel in our life now.
Lenny Rachitsky：嗯，现在我们的生活中都有一个 Hamel。
This was incredible.
这太不可思议了。
This was everything I'd hoped it'd be.
这就是我所希望的一切。
I feel like this is the most interesting in-depth consumable primer on evals that I've ever seen.
我觉得这是我见过的关于评估的最有趣的深入消耗性入门读物。
I'm really thankful you two made time for this.
我真的很感谢你们两个抽出时间来做这件事。
Two final questions.
最后两个问题。
Where can folks find you?
人们在哪里可以找到你？
Where can they find the course and how can listeners be useful to you?
他们在哪里可以找到课程以及听众如何对您有用？
I'll start with Shreya.
我将从施瑞亚开始。
Shreya Shankar: Yeah, you can reach me via email.
Shreya Shankar：是的，您可以通过电子邮件联系我。
It's on my website.
它在我的网站上。
If you Google my name, that is the easiest way to get to my website.
如果你用谷歌搜索我的名字，这是访问我的网站的最简单的方法。
You can find the course if you Google AI Evals for engineers and product managers, or just AI Evals course, you'll find it.
如果你在 Google 上搜索工程师和产品经理的 AI Evals，或者只是 AI Evals 课程，你就能找到该课程。
We'll send some links hopefully after this, so it's easy.
希望在此之后我们会发送一些链接，所以这很容易。
How to be helpful?
如何提供帮助？
Two things always for me.
有两件事对我来说总是如此。
One is ask me questions when you have them.
一是当你有问题时问我。
I'll try to get to the respond as soon as I can.
我会尽快回复。
The other one is tell us your successes.
另一件是告诉我们你的成功经历。
One of the things that keeps us going is somebody tells us what they implemented or what they did, a real case study.
让我们继续前进的原因之一是有人告诉我们他们实施了什么或做了什么，这是一个真实的案例研究。
Hamel and I gets so excited from these and it really keeps us going, so please share.
哈梅尔和我对这些感到非常兴奋，这确实让我们继续前进，所以请分享。
Hamel Husain: Yeah, it's pretty easy to find me.
Hamel Husain：是的，找到我很容易。
My website is Hamel.dev.
我的网站是 Hamel.dev。
I'll give you the link.
我会给你链接。
You can find me on social media, LinkedIn, Twitter.
您可以在社交媒体、LinkedIn、Twitter 上找到我。
The thing that's most helpful is to echo what Shreya said, we would be delighted if we are not the only people teaching evals.
最有帮助的是回应 Shreya 所说的，如果我们不是唯一教授评估的人，我们会很高兴。
We would love other people teach evals.
我们希望其他人能够教授评估。
Any kind of blog posts, writing, especially that as you go through this and learn this that you want to share, we would be delighted to help re-share that or amplify that.
任何类型的博客文章、写作，尤其是当您浏览并了解您想要分享的内容时，我们将很高兴帮助重新分享或放大它。
Lenny Rachitsky: Amazing.
莱尼·拉奇茨基：太棒了。
Very generous.
非常慷慨。
Thank you two, so much for being here.
谢谢你们两位，非常感谢你们来到这里。
I really appreciate it, and you guys have a lot going on, so thank you.
我真的很感激，你们做了很多事情，所以谢谢你们。
Shreya Shankar: Thanks, Lenny, for having us and for all the compliments.
Shreya Shankar：谢谢 Lenny 邀请我们并给予我们所有的赞美。
Lenny Rachitsky: My pleasure.
莱尼·拉奇茨基：我很荣幸。
Bye everyone.
再见大家。
Thank you so much for listening.
非常感谢您的聆听。
If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.
如果您发现这很有价值，您可以在 Apple Podcasts、Spotify 或您最喜欢的播客应用程序上订阅该节目。
Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast.
另外，请考虑给我们评分或留下评论，因为这确实有助于其他听众找到播客。
You can find all past episodes or learn more about the show at Lennyspodcast.com.
您可以在 Lennyspodcast.com 上找到所有过去的剧集或了解有关该节目的更多信息。
See you in the next episode.
下一集见。