The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Ronny Kohavi: I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment.

Ronny Kohavi：我很清楚，我非常喜欢测试一切，即您所做的任何代码更改、您引入的任何功能都必须进行一些实验。

Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact.

因为我再次观察到这种令人惊讶的结果，即使是小的错误修复，即使是小的更改有时也会产生令人惊讶的、意想不到的影响。

Ronny Kohavi: And so I don't think it's possible to experiment too much.

Ronny Kohavi：所以我认为不可能进行太多实验。

You have to allocate sometimes to these high risk, high reward ideas.

有时你必须分配给这些高风险、高回报的想法。

We're going to try something that's most likely to fail.

我们将尝试一些最有可能失败的事情。

But if it does win, it's going to be a home run.

但如果它真的赢了，这将是一个本垒打。

Ronny Kohavi: And you have to be ready to understand and agree that most will fail.

Ronny Kohavi：你必须准备好理解并同意大多数人都会失败。

And it's amazing how many times I've seen people come up with new designs or a radical new idea.

令人惊讶的是，我多次看到人们提出新的设计或激进的新想法。

And they believe in it, and that's okay.

他们相信这一点，这没关系。

I'm just cautioning them all the time to say, "If you go for something big, try it out, but be ready to fail 80% of the time."

我只是一直提醒他们，“如果你想做一些大事，就尝试一下，但要做好 80% 的失败的准备。”

Lenny: Welcome to Lenny's Podcast, where I interview world-class product leaders and growth experts to learn from their hard win experiences building and growing today's most successful products.

莱尼：欢迎来到莱尼的播客，我在其中采访了世界一流的产品领导者和增长专家，从他们构建和发展当今最成功产品的来之不易的经验中学习。

Lenny: Today my guest is Ronny Kohavi.

莱尼：今天我的客人是罗尼·科哈维。

Ronny is seen by many as the world expert on A/B testing and experimentation.

Ronny 被许多人视为 A/B 测试和实验方面的世界专家。

Most recently, he was VP and technical fellow of relevance at Airbnb where he led their search experience team.

最近，他担任 Airbnb 的副总裁兼相关技术研究员，领导他们的搜索体验团队。

Prior to that, he was corporate vice president at Microsoft, where he led Microsoft Experimentation Platform team.

在此之前，他曾担任 Microsoft 公司副总裁，领导 Microsoft 实验平台团队。

Before that, he was director of data mining and personalization at Amazon.

在此之前，他曾担任亚马逊数据挖掘和个性化总监。

Lenny: He's currently a full-time advisor and instructor.

莱尼：他目前是一名全职顾问和讲师。

He's also the author of the go-to book on experimentation called Trustworthy Online Controlled Experiments.

他还是有关实验的必读书籍《值得信赖的在线控制实验》的作者。

And in our show notes, you'll find a code to get a discount on taking his live cohort-based course on Maven.

在我们的表演笔记中，您会找到一个代码，可以在参加他的 Maven 实时队列课程时获得折扣。

Lenny: In our conversation, we get super tactical about A/B testing.

Lenny：在我们的谈话中，我们对 A/B 测试进行了超级策略。

Ronny shares his advice for when you should start considering running experiments at your company, how to change your company's culture to be more experiment driven, what are signs your experiments are potentially invalid, why trust is the most important element of a successful experiment, culture, and platform.

罗尼分享了他的建议，包括您何时应该开始考虑在公司进行实验，如何改变公司的文化以使其更加实验驱动，哪些迹象表明您的实验可能无效，为什么信任是成功实验、文化和平台的最重要元素。

How to get started if you want to start running experiments at your company.

如果您想在公司开始进行实验，如何开始。

He also explains what actually is a P value and something called Twyman's law, plus some hot takes about Airbnb and experiments in general.

他还解释了 P 值的实际含义和 Twyman 定律，以及有关 Airbnb 和一般实验的一些热门观点。

This episode is for anyone who's interested in either creating an experiment driven culture at their company or just fine-tuning one that already exists.

本集适合任何有兴趣在公司创建实验驱动文化或只是对现有文化进行微调的人。

Enjoy this episode with Ronny Kohavi after a short word from our sponsors.

在我们的赞助商简短致辞后，请与 Ronny Kohavi 一起欣赏这一集。

Lenny: This episode is brought to you by Mixpanel.

Lenny：这一集是由 Mixpanel 为您带来的。

Get deep insights into what your users are doing at every stage of the funnel, at a fair price that scales at you grow.

以可随着您的成长而扩展的合理价格，深入了解您的用户在渠道每个阶段的行为。

Mixpanel gives you quick answers about your users from awareness, to acquisition, through retention.

Mixpanel 为您提供有关用户的快速答案，从认知到获取，再到保留。

And by capturing website activity, ad data, and multi-touch attribution right in Mixpanel, you can improve every aspect of the full user funnel.

通过在 Mixpanel 中捕获网站活动、广告数据和多点触控归因，您可以改进整个用户渠道的各个方面。

Powered by first party behavioral data instead of third party cookies, Mixpanel is built to be more powerful and easier to use than Google Analytics.

Mixpanel 由第一方行为数据而非第三方 cookie 提供支持，比 Google Analytics 更强大且更易于使用。

Explore plans for teams of every size and see what Mixpanel can do for you at mixpanel.com/friends/lenny.

访问 mixpanel.com/friends/lenny，探索适合各种规模团队的计划并了解 Mixpanel 可以为您做什么。

And while you're at it, they're also hiring.

当你在做这件事的时候，他们也在招聘。

So check it out at mixpanel.com/friends/lenny.

请访问 mixpanel.com/friends/lenny 进行查看。

Lenny: This episode is brought to you by Round.

莱尼：这一集是由 Round 为您带来的。

Round is the private network built by tech leaders for tech leaders.

Round 是由技术领导者为技术领导者构建的专用网络。

Round combines the best of coaching, learning, and authentic relationships to help you identify where you want to go and accelerate your path to get there, which is why their wait list tops thousands of tech execs.

Round 结合了最好的辅导、学习和真实的关系，帮助你确定你想要去的地方，并加速你到达那里的道路，这就是为什么他们的候补名单排在数千名技术高管的前面。

Round is on a mission to shape the future of technology and its impact on society.

Round 的使命是塑造技术的未来及其对社会的影响。

Leading in tech is uniquely challenging, and doing it well is easiest when surrounded by leaders who understand your day-to-day experiences.

科技领域的领导者具有独特的挑战性，而当领导者了解你的日常经历时，做好这项工作是最容易的。

When we're meeting and building relationships with the right people, we're more likely to learn, find new opportunities, be dynamic in our thinking, and achieve our goals.

当我们与合适的人会面并建立关系时，我们更有可能学习、发现新的机会、思维活跃并实现我们的目标。

Building and managing your network doesn't have to feel like networking.

建立和管理你的网络并不一定像建立网络一样。

Join Round to surround yourself with leaders from tech's most innovative companies.

加入 Round，与来自科技界最具创新性公司的领导者为伍。

Build relationships, be inspired, take action.

建立关系、受到启发、采取行动。

Visit round.tech/apply and use promo code Lenny to skip the wait list.

访问 round.tech/apply 并使用促销代码 Lenny 跳过等待名单。

That's round.tech/apply.

那是 round.tech/apply。

Lenny: Ronny, welcome to the podcast.

莱尼：罗尼，欢迎来到播客。

Lenny: So you're known by many as maybe the leading expert on A/B testing and experimentation, which I think is something every product company eventually ends up trying to do, often badly.

Lenny：很多人都认为您可能是 A/B 测试和实验方面的领先专家，我认为这是每个产品公司最终都会尝试做的事情，但往往效果不佳。

And so I'm excited to dig quite deep into the world of experimentation and A/B testing to help people run better experiments.

因此，我很高兴能够深入研究实验和 A/B 测试领域，以帮助人们进行更好的实验。

So thank you again for being here.

再次感谢您来到这里。

Ronny Kohavi: That's a great goal.

罗尼·科哈维：这是一个伟大的目标。

Thank you.

谢谢。

Lenny: Let me start with kind of a fun question.

莱尼：让我从一个有趣的问题开始。

What is maybe the most unexpected A/B tests you've run or maybe the most surprising result from an A/B test that you've run?

您运行过的最意想不到的 A/B 测试或者您运行过的 A/B 测试中最令人惊讶的结果是什么？

Ronny Kohavi: So I think the opening example that I use in my book and in my class is the most surprising public example we can talk about.

Ronny Kohavi：所以我认为我在书中和课堂上使用的开头示例是我们可以谈论的最令人惊讶的公开示例。

And this was kind of an interesting experiment.

这是一个有趣的实验。

Somebody proposed to change the way that ads were displayed on Bing, the search engine.

有人提议改变广告在搜索引擎 Bing 上的显示方式。

And he basically said, "Let's take the second line and move it, promote it to the first line so that the title line becomes larger."

他基本上是说，“让我们将第二行移动，将其提升到第一行，以便标题行变得更大。”

Ronny Kohavi: And when you think about that, and if you're going to look in my book, or in the class, there's an actual diagram of what happened, the screenshots.

罗尼·科哈维（Ronny Kohavi）：当你想到这一点时，如果你要查看我的书或课堂上的内容，就会发现所发生事件的实际图表，即屏幕截图。

But if you think about it, just realistically it looks like a meh idea.

但如果你仔细想想，实际上这看起来像是一个无聊的想法。

Why would this be such a reasonable, interesting thing to do?

为什么这是一件如此合理、有趣的事情呢？

And indeed, when we went back to the backlog, it was on the backlog for months, and languished there, and many things were rated higher.

事实上，当我们回到积压工作时，它已经积压了几个月，并在那里萎靡不振，许多事情的评级都较高。

Ronny Kohavi: But the point about this is it's trivial to implement.

Ronny Kohavi：但重点是实施起来很简单。

So if you think about return on investment, we could get the data by having some engineers spend a couple of hours implementing it.

因此，如果您考虑投资回报，我们可以通过让一些工程师花几个小时来实现它来获取数据。

Ronny Kohavi: And that's exactly what happened.

Ronny Kohavi：这正是发生的事情。

Somebody at Bing who kept seeing this in the backlog and said, "My God, we're spending too much time discussing it.

Bing 的某个人不断在积压的工作中看到这个问题，他说：“天哪，我们花了太多时间讨论它。

I could just implement it." He did.

我可以直接实施它。”他做到了。

He spent a couple of days implementing it, as is the common thing at Bing, he launched the experiment.

他花了几天时间来实施它，就像 Bing 的常见做法一样，他发起了这个实验。

Ronny Kohavi: And a funny thing happened.

Ronny Kohavi：然后发生了一件有趣的事情。

We had an alarm.

我们有警报。

Big escalation, something is wrong with the revenue metric.

大升级，收入指标出了问题。

Now this alarm fired several times in the past when there were real mistakes, where somebody would log revenue twice, or there's some data problem.

现在，当出现真正的错误时，这个警报在过去多次触发，有人会两次记录收入，或者存在一些数据问题。

But in this case, there was no bug.

但在本例中，并没有出现任何错误。

That simple idea increased revenue by about 12%.

这个简单的想法使收入增加了约 12%。

Ronny Kohavi: And this is something that just doesn't happen.

罗尼·科哈维（Ronny Kohavi）：这根本就不会发生。

We can talk later about Wyman's law, but that was the first reaction, which is, "This is too good to be true.

我们可以稍后讨论怀曼定律，但这是第一反应，“这太好了，令人难以置信。

Let's find a bug." And we did.

让我们找到一个错误。”我们做到了。

And we looked for several times, and we replicated the experiment several times, and there was nothing wrong with it.

而且我们找了好几次，也重复了好几次实验，都没有什么问题。

This thing was worth $100 million at the time when Bing was a lot smaller.

在 Bing 规模还小得多的时候，这个东西价值 1 亿美元。

Ronny Kohavi: And the key thing is it didn't hurt the user metrics.

Ronny Kohavi：关键是它没有损害用户指标。

So it's very easy to increase revenue by doing theatrics.

所以通过做戏剧来增加收入是很容易的。

Displaying more ads is a trivial way to raise revenue, but it hurts the user experience.

展示更多广告是增加收入的一种微不足道的方法，但它会损害用户体验。

And we've done the experiments to show that.

我们已经做了实验来证明这一点。

In this case, this was just a home run that improved revenue, didn't significantly hurt the guardrail metrics.

在这种情况下，这只是一个提高收入的全垒打，并没有显着损害护栏指标。

And so we were in awe of what a trivial change.

所以我们对如此微小的改变感到敬畏。

That was the biggest revenue impact to Bing in all its history.

这是必应历史上对收入影响最大的一次。

Lenny: And that was basically shifting in two lines, right?

莱尼：这基本上是在两行中移动，对吧？

Switching two lines in the search results.

切换搜索结果中的两行。

Ronny Kohavi: And this was just moving the second line to the first line.

Ronny Kohavi：这只是将第二行移动到第一行。

Now you then go and run a lot of experiments to understand what happened here.

现在，您可以进行大量实验来了解这里发生了什么。

Is it the fact that the title line has a bigger font, sometimes different color?

是不是标题行字体更大，有时颜色不同？

So we ran a whole bunch of experiments.

所以我们进行了一大堆实验。

Ronny Kohavi: And this is what usually happens.

Ronny Kohavi：这就是通常发生的情况。

We have a breakthrough.

我们有了突破。

You start to understand more about, what can we do?

你开始更多地了解，我们能做什么？

And there suddenly a shift towards, "Okay, what are other things we could do that would allow us to improve revenue?" We came up with a lot of follow on ideas that helped a lot.

然后突然转向“好吧，我们还可以做哪些其他事情来提高收入？”我们提出了很多有帮助的后续想法。

Ronny Kohavi: But to me, this was an example of a tiny change that was the best revenue generating idea in Bing's history, and we didn't rate it properly.

Ronny Kohavi：但对我来说，这是 Bing 历史上最好的创收创意的一个微小变化的例子，我们没有正确评价它。

Nobody gave this the priority that in hindsight, it deserves.

事后看来，没有人给予它应有的优先权。

And that's something that happens often.

这是经常发生的事情。

I mean, we are often humbled by how bad we are at predicting the outcome of experiments.

我的意思是，我们常常因自己在预测实验结果方面的糟糕程度而感到羞愧。

Lenny: This reminds me of a classic experiment at Airbnb while I was there, and we'll talk about Airbnb in a bit.

Lenny：这让我想起了我在 Airbnb 时的一个经典实验，我们稍后会讨论 Airbnb。

The search team just ran a small experiment of what if we were to open a new tab every time someone clicked on a search result, instead of just going straight to that listing.

搜索团队刚刚进行了一个小实验，如果我们每次有人点击搜索结果时都打开一个新选项卡，而不是直接进入该列表，会怎么样。

And that was one of the biggest wins in search-

这是搜索领域最大的胜利之一——

Ronny Kohavi: And by the way, I don't know if you know the history of this, but I tell about this in class.

Ronny Kohavi：顺便说一句，我不知道你是否知道这件事的历史，但我在课堂上讲述了这件事。

We did this experiment way back around 2008 I think.

我想我们早在 2008 年就做过这个实验。

And so this predates Airbnb.

所以这早于 Airbnb。

I remember it was heavily debated.

我记得这件事引起了激烈的争论。

Why would you open something in a new tab?

为什么要在新选项卡中打开某些内容？

The users didn't ask for it.

用户没有要求它。

It was a lot of pushback from the designers.

这遭到了设计师的强烈反对。

And we ran that experiment.

我们进行了那个实验。

And again, it was one of these highly surprising results that made it that we learned so much from it.

再说一次，正是这些非常令人惊讶的结果之一让我们从中学到了很多东西。

Ronny Kohavi: So we first did this.

Ronny Kohavi：所以我们首先这样做了。

It was done in the UK for opening Hotmail, and then we moved it to MSN, so it would open search in new tab, and all the set of experiments were highly, highly beneficial.

它是在英国完成的，用于打开 Hotmail，然后我们将其转移到 MSN，因此它会在新选项卡中打开搜索，所有这组实验都非常非常有益。

We published this.

我们发表了这个。

And I have to tell you, when I came to Airbnb, I talked to our joint friend Ricardo about this.

我必须告诉你，当我来到 Airbnb 时，我和我们共同的朋友里卡多谈过这件事。

And it was sort of done, it was very beneficial, and then it was semi forgotten, which is one of the things you learned about institutional memories.

它已经完成了，非常有益，然后它就被半遗忘了，这是你学到的关于机构记忆的事情之一。

When you have winners, make sure to address them and remember them.

当你有获奖者时，一定要向他们讲话并记住他们。

So it was at Airbnb done for a long time before I joined that listings opened in a new tab, but other things that were designed in the future were not done.

因此，在我加入之前，Airbnb 已经完成了很长一段时间，在新选项卡中打开列表，但未来设计的其他事情还没有完成。

And I reintroduced this to the team, and we saw big improvements.

我向团队重新介绍了这一点，我们看到了很大的改进。

Lenny: Shout out to Ricardo, our mutual friend who helped make this conversation happen.

莱尼：向里卡多致敬，他是我们共同的朋友，他帮助促成了这次对话。

There's this holy grail of experiments that I think people are always looking for of one hour of work and it creates this massive result.

我认为人们一直在寻找一个实验的圣杯，即一小时的工作，它创造了如此巨大的结果。

I imagine this is very rare, and don't expect this to happen.

我想这种情况非常罕见，并且不要指望这种情况会发生。

I guess in your experience, how often do you find one of these gold nuggets just lying around?

我想根据您的经验，您多久会发现这些金块就在周围？

Ronny Kohavi: Yeah.

罗尼·科哈维：是的。

So again, this is a topic that's near and dear to my heart.

再说一遍，这是一个我非常关心的话题。

Everybody wants these amazing results, and I show them in chapter one in my book, multiple of these small efforts, huge gain.

每个人都想要这些惊人的结果，我在书的第一章中展示了它们，这些小小的努力的倍增，巨大的收获。

Ronny Kohavi: But as you said, they're very rare.

Ronny Kohavi：但正如你所说，它们非常罕见。

I think most of the time, the winnings are made this inch by inch.

我认为大多数时候，奖金都是一寸一寸地赚来的。

And there's a graph that I show in my book, a real graph of how Bing ads has managed to improve the revenue per a thousand searches over time, and every month you can see a small improvement and a small improvement.

我在书中展示了一个图表，这是一个真实的图表，显示了 Bing 广告如何随着时间的推移成功提高每千次搜索的收入，每个月你都可以看到一个小的改进和一个小的改进。

Sometimes the degradation because of legal reasons or other things.

有时由于法律原因或其他原因而降级。

There were some concern that we were not marking the ads properly.

有人担心我们没有正确标记广告。

So you have to suddenly do something that you know is going to hurt revenue.

所以你必须突然做一些你知道会损害收入的事情。

But yes, I think most results are inch by inch.

但是，是的，我认为大多数结果都是一寸一寸的。

You improve small amounts, lots of them.

你可以改进少量，也可以改进很多。

I think that the best example that I can say is a couple of them that I can speak about.

我认为我能说的最好的例子就是我能谈论的几个例子。

Ronny Kohavi: One is at Bing, the relevance team, hundreds of people all working to improve bing relevance.

Ronny Kohavi：其中之一是 Bing 的相关性团队，数百人都致力于提高 bing 的相关性。

They have a metric, we'll talk about OEC, the overall evaluation criterion.

他们有一个指标，我们会谈论OEC，总体评价标准。

But they have a metric that their goal is to improve it by 2% every year.

但他们有一个指标，目标是每年提高 2%。

It's a small amount, and that 2% you can see here's a 0.1, and here's a 0.15, here's a 0.2, and then they add up to around 2% every year, which is amazing.

这是一个很小的数字，你可以看到这 2% 这里是 0.1，这里是 0.15，这里是 0.2，然后每年加起来大约 2%，这是惊人的。

Ronny Kohavi: Another example that I am allowed to speak about from Airbnb is the fact that we ran some 250 experiments in my tenure there in search relevance.

Ronny Kohavi：我可以在 Airbnb 谈论的另一个例子是，我们在我的任期内进行了大约 250 个搜索相关性实验。

And again, small improvements added up.

再一次，小的改进累积起来。

So this became overall a 6% improvement to revenue.

因此，总体收入提高了 6%。

So when you think about 6%, it's a big number, but it came out not of one idea, but many, many smaller ideas that each gave you a small gain.

因此，当你想到 6% 时，这是一个很大的数字，但它不是来自一个想法，而是来自许多很多较小的想法，每个想法都给你带来了小小的收获。

Ronny Kohavi: And in fact, again, there's another number I'm allowed to say.

Ronny Kohavi：事实上，我还可以说另一个数字。

Of these experiments, 92% failed to improve the metric that we were trying to move.

在这些实验中，92% 未能改进我们试图移动的指标。

So only 8% of our ideas actually were successful at moving the key metrics.

因此，我们的想法中只有 8% 实际上成功地推动了关键指标。

Lenny: There's so many threads I want to follow here, but let me follow this one right here.

Lenny：这里有很多话题我想关注，但让我在这里关注这一话题。

You just mentioned of 92% of experiments failed.

你刚才提到92%的实验失败了。

Is that typical in your experience seeing experiments running a lot of companies?

在您对许多公司进行实验的经历中，这是典型的吗？

What should people expect when they're running experiments?

人们在进行实验时应该期待什么？

What percentage should they expect to fail?

他们预计会失败的比例是多少？

Ronny Kohavi: Well, first of all, I published three different numbers for my career.

Ronny Kohavi：嗯，首先，我为我的职业生涯发布了三个不同的数字。

So overall at Microsoft, about 66%, two thirds of ideas fail.

总体而言，在微软，大约 66% 的想法失败了，三分之二的想法都失败了。

And don't think the 66 is accurate.

并且不要认为 66 是准确的。

It's about two thirds.

大约是三分之二。

At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was around 85%.

在 Bing，经过我们优化一段时间后，它是一个更加优化的领域，失败率约为 85%。

So it's harder to improve something that you've been optimizing for a while.

因此，改进已经优化了一段时间的东西会变得更加困难。

And then at Airbnb, this 92% number is the highest failure rate that I've observed.

在 Airbnb，92% 的数字是我观察到的最高失败率。

Ronny Kohavi: Now I've quoted other sources.

Ronny Kohavi：现在我引用了其他来源。

It's not that I worked at groups that were particularly bad, Booking, Google Ads, other companies published numbers that are around 80 to 90% failure rate of ideas.

这并不是说我工作过的团队特别糟糕，Booking、Google Ads 和其他公司公布的数据显示，想法的失败率约为 80% 到 90%。

This is where it's important of experiments.

这就是实验的重要之处。

It's important to realize that when you have a platform, it's easy to get this number.

重要的是要认识到，当您拥有一个平台时，很容易获得这个数字。

You look at how many experiments were run and how many of them launched.

您可以查看进行了多少次实验以及启动了多少次实验。

Not every experiment maps to an idea.

并非每个实验都对应一个想法。

Ronny Kohavi: So it's possible that when you have an idea, your first implementation, you start an experiment.

Ronny Kohavi：所以当你有了一个想法，第一次实施时，你可能会开始一个实验。

Boom, it's egregiously bad, because you have a bug.

繁荣，这太糟糕了，因为你有一个错误。

In fact, 10% of experiments tend to be aborted on the first date.

事实上，10% 的实验往往会在第一次约会时就流产。

Those are usually not that the idea is bad, but that there is an implementation issue or something we haven't thought about, that forces an abort.

这些通常并不是因为这个想法不好，而是因为存在实施问题或我们没有考虑到的问题，从而迫使中止。

Ronny Kohavi: You may iterate and pivot again.

Ronny Kohavi：你可以再次迭代和调整。

And ultimately, if you do two, or three, or four pivots or bug fixes, you may get to a successful launch.

最终，如果您进行两个、三个或四个关键点或错误修复，您可能会成功启动。

But those numbers of 80 to 92% failure rate are of experiments.

但 80% 到 92% 的失败率只是实验数据。

Ronny Kohavi: Very humbling.

罗尼·科哈维：非常谦卑。

I know that every group that starts to run experiments, they always start off by thinking that somehow, they're different.

我知道每个开始进行实验的小组，他们总是一开始就认为他们是不同的。

And their success rate's going to be much, much higher, and they're all humbled.

他们的成功率将会高得多，而且他们都会感到谦卑。

Lenny: You mentioned that you had this pattern of clicking a link and opening a new tab as a thing that just worked at a lot of different places.

莱尼：你提到你有这种点击链接并打开新选项卡的模式，这种模式在很多不同的地方都有效。

Lenny: Are there other versions of this?

莱尼：还有其他版本吗？

Do you do you collect a list of, "Here's things that often work when we want to move" there's some you could share.

您是否收集了一份“当我们想要搬家时经常使用的东西”的清单，其中有一些您可以分享。

I don't know if you have a list in your head.

不知道你心里有没有一个清单。

Ronny Kohavi: I can give you two resources.

Ronny Kohavi：我可以给你两个资源。

One of them is a paper that we wrote called Rules of Thumb, and what we tried to do at that time at Microsoft was to just look at thousands of experiments that run and extract some patterns.

其中之一是我们写的一篇名为“经验法则”的论文，当时我们在微软试图做的就是查看数千个运行并提取一些模式的实验。

And so that's one paper that we can then put in the notes.

这就是我们可以将其放入笔记中的一篇论文。

Ronny Kohavi: But there's another more accurate, I would say, resource that's useful that I recommend to people.

Ronny Kohavi：但是我会说，我向人们推荐了另一个更准确、有用的资源。

And it's a site called goodui.org, and goodui.org is exactly the site that tries to do what you're saying at scale.

这是一个名为goodui.org 的网站，而goodui.org 正是尝试大规模实现您所说的内容的网站。

Ronny Kohavi: So guy's name is Jacob .

Ronny Kohavi：所以这个人的名字叫雅各布。

He asks people to send them results of experiments, and he puts them into patterns.

他要求人们向他们发送实验结果，然后将它们放入模式中。

There's probably 140 patterns I think at this point.

我认为目前可能有 140 种模式。

And then for each pattern he says, "Well, who has that helped?

然后对于每一种模式，他都会说：“好吧，这对谁有帮助？

How many times and by how much?" So you have an idea of this worked, three out of five times.

多少次、多少？”所以你知道这有效，五分之三。

And it was a huge win.

这是一次巨大的胜利。

In fact, you can find that open a new window in there.

事实上，你可以在那里找到打开一个新窗口。

Lenny: I feel like you feed that into ChatGPT, and you have basically a product manager creating a roadmap tool.

Lenny：我觉得你把它输入到 ChatGPT 中，基本上就是一个产品经理在创建一个路线图工具。

Ronny Kohavi: In general, by the way, a lot of that is institutional memory, which is can you document things well enough so that the organization remembers the successes and failures, and learns from them?

Ronny Kohavi：顺便说一句，总的来说，其中很多都是机构记忆，也就是说，你能否足够好地记录事情，以便组织记住成功和失败，并从中学习？

Ronny Kohavi: I think one of the mistakes that some company makes is they launch a lot of experiments and never go back and summarize the learnings.

Ronny Kohavi：我认为一些公司犯的错误之一是他们进行了大量实验，却从不回去总结经验教训。

So I've actually put a lot of effort in this idea of institutional learning, of doing the quarterly meeting of the most surprising experiments.

因此，我实际上在机构学习的想法上投入了大量的精力，每季度召开一次最令人惊讶的实验会议。

Ronny Kohavi: By the way, surprising is another question that people often are not clear about.

Ronny Kohavi：顺便说一句，令人惊讶的是另一个人们常常不清楚的问题。

What is a surprising experiment?

什么是令人惊讶的实验？

To me, a surprising experiment is one where the estimated result beforehand and the actual result differ by a lot.

对我来说，一个令人惊讶的实验是预先估计的结果与实际结果相差很大的实验。

So that absolute value of the difference is large.

因此差异的绝对值很大。

Ronny Kohavi: Now you can expect something to be great and it's flat.

Ronny Kohavi：现在你可以期待一些很棒的东西，但它却很平淡。

Well, you learn something.

嗯，你学到了一些东西。

But if you expect something to be small and it turns out to be great, like that ad title promotion, then you've learned a lot.

但是，如果您期望某件事很小，而结果却很棒，例如广告标题促销，那么您就学到了很多东西。

Or conversely, if you expect that something will be small and it's very negative, you can learn a lot by understanding why this was so negative.

或者相反，如果你预计某件事会很小并且非常消极，那么通过理解为什么这如此消极，你可以学到很多东西。

And that's interesting.

这很有趣。

Ronny Kohavi: So we focused not just on the winners, but also surprising losers, things that people thought would be a no-brainer to run.

罗尼·科哈维：所以我们不仅关注赢家，还关注令人惊讶的输家，人们认为这是理所当然的事情。

And then for some reason, it was very negative.

然后由于某种原因，它是非常消极的。

And sometimes, it's that negative that gives you insight.

有时，正是这种负面情绪给了你洞察力。

Actually, I'm just coming up with one example that of that, that I should mention.

事实上，我只是想出一个我应该提到的例子。

Ronny Kohavi: We were running this experiment at Microsoft to improve the windows indexer, and the team was able to show on offline tests that it does much better at indexing, and they showed some relevance is higher, and all these good things.

Ronny Kohavi：我们在 Microsoft 进行了这个实验，以改进 Windows 索引器，团队能够在离线测试中表明它在索引方面做得更好，并且他们显示出一些相关性更高，以及所有这些好东西。

And then they ran it as an experiment.

然后他们将其作为实验进行。

You know what happened?

你知道发生了什么吗？

Surprising result.

令人惊讶的结果。

Indexing the relevance was actually high, but it killed a battery life.

索引相关性实际上很高，但它会缩短电池寿命。

Ronny Kohavi: So here's something that comes from left field that you didn't expect.

Ronny Kohavi：所以这是来自左外野的一些你意想不到的东西。

It was consuming a lot more CPU on laptops.

它在笔记本电脑上消耗了更多的 CPU。

It was killing the laptops.

它正在杀死笔记本电脑。

And therefore, okay, we learned something.

因此，好吧，我们学到了一些东西。

Let's document it.

让我们记录下来。

Let's remember this, so that we now take this other factor into account as we design the next iteration.

让我们记住这一点，以便我们在设计下一个迭代时考虑到这个其他因素。

Lenny: What advice do you have for people to actually remember these surprises?

莱尼：对于让人们真正记住这些惊喜，你有什么建议？

You said that a lot of it is institutional.

你说很多都是制度性的。

What do you recommend people do so that they can actually remember this when people leave, say three years later?

您建议人们做什么，以便当人们离开时（比如三年后）他们能够真正记住这一点？

Ronny Kohavi: Document it.

Ronny Kohavi：记录下来。

We had a large deck internally of these successes and failures, and we encourage people to look at them.

我们内部有很多关于这些成功和失败的资料，我们鼓励人们查看它们。

The other thing that's very beneficial is just to have your whole history of experiments and do some ability to search by keywords.

另一件非常有益的事情是拥有您的整个实验历史并具有通过关键字搜索的能力。

Ronny Kohavi: So I have an idea.

Ronny Kohavi：所以我有一个想法。

Type a few keywords and see if from the thousands of experiments that ran...

输入一些关键字，看看是否从运行的数千个实验中......

And by the way, these are very reasonable numbers.

顺便说一句，这些都是非常合理的数字。

At Microsoft, just to let you know, when I left in 2019, we were on a rate of about 20 to 25,000 experiments every year.

在微软，我只是想告诉你，当我 2019 年离开时，我们每年进行大约 20 到 25,000 次实验。

So every working, day we were starting something like 100 new treatments.

所以每个工作日我们都会开始大约 100 种新的治疗方法。

Big numbers.

大数字。

So when you're running in a group like Bing, which is running thousands and thousands of experiments, you want to be able to ask, "Has anybody did an experiment on this or this or this?" And so that searching capability is in the platform.

因此，当您在像 Bing 这样的团队中运行时，该团队正在进行成千上万的实验，您希望能够问：“有人对此进行过实验吗？”因此，搜索功能就在平台中。

Ronny Kohavi: But more than that, I think just doing the quarterly meeting of the most successful...

Ronny Kohavi：但更重要的是，我认为只是召开最成功的季度会议......

Most interesting, sorry, not just successful, most interesting experiments is very key.

最有趣，抱歉，不仅仅是成功，最有趣的实验很关键。

And that also helps the flywheel of experimentation.

这也有助于实验的飞轮。

Lenny: It's a good segue to something I wanted to touch on, which is there's often, I guess a weariness of running too many experiments and being too data-driven, and the sense that experimentation just leads you to these micro optimizations, and you don't really innovate and do big things.

莱尼：这是我想谈的一个很好的延续，我想经常会出现运行太多实验和过于数据驱动的厌倦，以及实验只会引导你进行这些微观优化的感觉，而你并没有真正创新并做大事。

What's your perspective on that?

您对此有何看法？

And then, can you be too experiment driven in your experience?

然后，你的经历是否会过于实验驱动？

Ronny Kohavi: I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment.

Ronny Kohavi：我很清楚，我非常喜欢测试一切，即您所做的任何代码更改、您引入的任何功能都必须进行一些实验。

Because again, I've observed this surprising result that even small bug fixes, even small changes can sometimes have surprising unexpected impact.

因为我再次观察到了这个令人惊讶的结果，即使是很小的错误修复，即使是很小的更改有时也会产生令人惊讶的意想不到的影响。

Ronny Kohavi: And so I don't think it's possible to experiment too much.

Ronny Kohavi：所以我认为不可能进行太多实验。

I think it is possible to focus on incremental changes because some people say, "Well, if we only tested 17 things around this," you have to think about, it's like in stock.

我认为可以专注于增量变化，因为有些人说，“好吧，如果我们只围绕这个测试 17 件事，”你必须考虑一下，这就像库存一样。

You need a portfolio.

你需要一个作品集。

You need some experiments that are incremental that move you in the direction that you know you're going to be successful over time if you just try enough.

你需要一些渐进的实验，让你朝着你知道的方向前进，只要你尝试得足够多，随着时间的推移，你就会取得成功。

But some experiments, you have to allocate sometimes to these high risk, high reward ideas.

但有些实验，你有时必须分配给这些高风险、高回报的想法。

We're going to try something that's most likely to fail, but if it does win, it's going to be a home run.

We're going to try something that's most likely to fail, but if it does win, it's going to be a home run.

Ronny Kohavi: And so you have to allocate some efforts to that, and you have to be ready to understand and agree that most will fail.

Ronny Kohavi：所以你必须为此付出一些努力，你必须准备好理解并同意大多数都会失败。

And I've amazing how many times I've seen people come up with new designs, or a radical new idea, and they believe in it, and that's okay.

我很惊讶有多少次看到人们想出新的设计，或者一个激进的新想法，并且他们相信它，这没关系。

I'm just cautioning them all the time to say, "Hey, if you go for something big, try it out, but be ready to fail 80% of the time."

我只是一直警告他们说：“嘿，如果你想做一些大事，就尝试一下，但要做好 80% 的失败的准备。”

Ronny Kohavi: And one true example, that again, I'm able to talk about because we put it in my book, is we were at Bing trying to change the landscape of search.

Ronny Kohavi：还有一个真实的例子，我可以再次谈论这个例子，因为我们把它写进了我的书中，我们在 Bing 试图改变搜索的格局。

And one of the ideas, the big ideas was we are going to integrate with social.

其中一个重要的想法是我们要与社交相融合。

So we hooked into the Twitter fire hose feed and we hooked into Facebook, and we spent 100 person years on this idea.

因此，我们连接了 Twitter 和 Facebook，我们花了 100 人年的时间来研究这个想法。

Ronny Kohavi: And it failed.

Ronny Kohavi: And it failed.

You don't see it anymore.

You don't see it anymore.

It existed for about a year and a half, and all the experiments were just negative to flat.

它存在了大约一年半的时间，所有的实验都只是阴性到持平。

And it was an attempt.

这是一种尝试。

It was fair to try it.

尝试一下是公平的。

I think it took us a little long to fail, to decide that this is a failure.

我认为我们花了很长时间才失败，才决定这是一次失败。

But at least we had the data.

But at least we had the data.

We had hundreds of experiments that we tried.

We had hundreds of experiments that we tried.

None of them were a breakthrough.

None of them were a breakthrough.

And I remember mailing Qi Lu with some statistics showing that it's time to abort, it's time to fail on this.

我记得给齐鲁寄了一些统计数据，表明是时候中止了，是时候失败了。

And he decided to continue more.

And he decided to continue more.

And it's a million dollar question.

And it's a million dollar question.

Do you continue, and then maybe the breakthrough will come next month, or do you abort?

你会继续下去，也许下个月就会出现突破，还是会中止？

And a few months later, we aborted.

And a few months later, we aborted.

Lenny: That reminds me of at Netflix, they tried a social component that also failed.

Lenny：这让我想起了 Netflix，他们尝试了一个社交组件，但也失败了。

At Airbnb, early on there was a big social attempt to make, "Here's your friends that have stayed at these Airbnbs," completely had no impact.

在 Airbnb，早期曾有过一次大规模的社交尝试，“这是你住在这些 Airbnb 的朋友”，但完全没有产生任何影响。

So maybe that's one of these learnings that we should document.

所以也许这是我们应该记录的经验之一。

Ronny Kohavi: Yeah, this is hard.

Ronny Kohavi: Yeah, this is hard.

This is hard.

这很难。

But again, that's the value of experiments, which are this oracle that gives you the data.

But again, that's the value of experiments, which are this oracle that gives you the data.

You may be excited about things.

You may be excited about things.

You may believe it's a good idea.

You may believe it's a good idea.

But ultimately, the oracle is the controlled experiment.

但最终，神谕是受控实验。

It tells you whether users are actually benefiting from it, whether you and the users, the company and the users.

它告诉你用户是否真正从中受益，是你和用户，还是公司和用户。

Lenny: There's obviously a bit of overhead and downside to running an experiment, setting all up, and analyzing the results.

Lenny：显然，运行实验、设置所有内容并分析结果会产生一些开销和缺点。

Is there anything that you ever don't think is worth A/B testing?

有什么您认为不值得进行 A/B 测试的事情吗？

Ronny Kohavi: First of all, there are some necessary ingredients to A/B testing.

Ronny Kohavi：首先，A/B 测试有一些必要的要素。

And I'll just say outright, not every domain is amenable to A/B testing.

我只想直截了当地说，并不是每个领域都适合 A/B 测试。

You can't A/B test mergers and acquisitions.

您无法对并购进行 A/B 测试。

It's something that happens once, you either acquire or you don't acquire.

这是发生过一次的事情，你要么获得，要么不获得。

Ronny Kohavi: So you do have to have some necessary ingredient.

Ronny Kohavi：所以你必须有一些必要的成分。

You need to have enough units, mostly users, in order for the statistics to work out.

您需要有足够的单位（主要是用户）才能进行统计。

So if you're too small, it may be too early to A/B test.

因此，如果规模太小，那么进行 A/B 测试可能还为时过早。

But what I find is that in software, it is so easy to run A/B testing and it is so easy to build a platform.

但我发现在软件中，运行 A/B 测试非常容易，构建平台也非常容易。

Ronny Kohavi: I don't say it's easy to build a platform.

Ronny Kohavi：我并不是说构建一个平台很容易。

But once you build a platform, the incremental cost of running an experiment should approach zero.

但是一旦你建立了一个平台，运行实验的增量成本应该接近于零。

And we got to that at Microsoft, where after a while, the cost of running experiments was so low that nobody was questioning the idea that everything should be experimented with.

我们在微软就做到了这一点，过了一段时间，进行实验的成本是如此之低，以至于没有人质疑一切都应该进行实验的想法。

Ronny Kohavi: Now, I don't think we were there at Airbnb for example.

Ronny Kohavi：现在，我不认为我们在 Airbnb 那里。

The platform at Airbnb was much less mature, and required a lot more analysts in order to interpret the results and to find issues with it.

Airbnb 的平台还不够成熟，需要更多的分析师来解释结果并发现问题。

So I do think there's this trade off.

So I do think there's this trade off.

You're willing to invest in the platform.

You're willing to invest in the platform.

It is possible to get the marginal cost to be close to zero.

有可能使边际成本接近于零。

But when you're not there, it's still expensive, and there may be reasons why not to run A/B tests.

但当你不在那里时，它仍然很昂贵，并且可能有理由不运行 A/B 测试。

Lenny: You talked about how you may be too small to run A/B tests, and this is a constant question for startups is, when should we start running A/B tests?

Lenny：您谈到公司可能太小而无法运行 A/B 测试，对于初创公司来说，这是一个持续存在的问题，我们应该什么时候开始运行 A/B 测试？

Do you have kind of a heuristic or a rule of thumb of, here's a time you should really start thinking about running an A/B test?

您是否有一种启发式或经验法则：现在您应该真正开始考虑运行 A/B 测试？

Ronny Kohavi: Yeah, a dollar question that everybody asks.

Ronny Kohavi：是的，每个人都会问一个关于美元的问题。

So actually, we'll put this in the notes, but I gave a talk last year, what I called it is practical defaults.

所以实际上，我们会将其放在注释中，但我去年做了一次演讲，我称之为实用默认值。

And one of the things I show there is that unless you have at least tens of thousands of users, the math, the statistics just don't work out for most of the metrics that you're interested in.

我在那里展示的一件事是，除非您至少拥有数万用户，否则数学、统计数据对于您感兴趣的大多数指标都不起作用。

Ronny Kohavi: In fact, I gave an actual practical number of a retail site with some conversion rate, trying to detect changes that are at least 5% beneficial, which is something that startups should focus on.

Ronny Kohavi：事实上，我给出了一个具有一定转化率的零售网站的实际实际数字，试图发现至少有 5% 有益的变化，这是初创公司应该关注的事情。

They shouldn't focus on the 1%, they should focus on the 5 and 10%.

他们不应该关注那1%，而应该关注5%和10%。

Then you need something like 200,000 users.

Then you need something like 200,000 users.

Ronny Kohavi: So start experimenting when you're in the tens of thousands of users.

Ronny Kohavi：因此，当您拥有数以万计的用户时，请开始尝试。

You'll only be able to detect large effects.

You'll only be able to detect large effects.

And then once you get to 200,000 users, then the magic starts happening.

一旦用户数量达到 200,000 名，奇迹就开始发生。

Then you can start testing a lot more.

Then you can start testing a lot more.

Then you have the ability to test everything and make sure that you're not degrading and getting value out of experimentation.

然后你就有能力测试一切并确保你不会降低实验的质量并从实验中获得价值。

So you ask for rule of thumb, 200,000 users, you're magical.

所以你要求经验法则，200,000 个用户，你太神奇了。

Below that, start building the culture, start building the platform, start integrating.

接下来，开始构建文化，开始构建平台，开始整合。

So that as you scale, you start to see the value.

这样，当你扩大规模时，你就会开始看到价值。

Lenny: Love it.

莱尼：喜欢。

Coming back to this kind of concern people have of experimentation, keeps you from innovating and taking big bets, I know you have this framework overall evaluation criterion, and I think that helps with this.

回到人们对实验的这种关注，阻止你创新和下大赌注，我知道你有这个框架的总体评估标准，我认为这对此有所帮助。

Can you talk a bit about that?

你能谈谈这个吗？

Ronny Kohavi: The OEC or the overall evaluation criterion is something that I think many people that start to dabble in A/B testing miss.

Ronny Kohavi：我认为许多开始涉足 A/B 测试的人都会忽略 OEC 或总体评估标准。

And the question is, what are you optimizing for?

问题是，你要优化什么？

And it's a much harder question that people think because it's very easy to say we're going to optimize for money, revenue.

人们认为这是一个更难的问题，因为很容易说我们将针对资金和收入进行优化。

But that's the wrong question, because you can do a lot of bad things that will improve revenue.

但这是一个错误的问题，因为你可以做很多坏事来提高收入。

So there has to be some countervailing metric that tells you, how do I improve revenue without hurting the user experience?

因此，必须有一些抵消指标来告诉您，如何在不损害用户体验的情况下提高收入？

Ronny Kohavi: So let's take a good example with search.

Ronny Kohavi：让我们举一个搜索的好例子。

You can put more ads on a page and you will make more money.

您可以在页面上放置更多广告，您会赚更多钱。

There's no doubt about it.

毫无疑问。

You will make more money in the short term.

短期内你会赚更多的钱。

The question is, what happens to the user experience, and how is that going to impact you in the long term?

问题是，用户体验会发生什么？从长远来看，这会对您产生什么影响？

Ronny Kohavi: So we've run those experiments, and we were able to map out this number of ads causes this much increase to churn, this number of ads causes this much increase to the time that users take to find a successful result.

Ronny Kohavi：所以我们已经进行了这些实验，并且我们能够绘制出如此数量的广告会导致客户流失如此之多，如此数量的广告会导致用户找到成功结果所需的时间如此之长。

And we came up with an OEC that is based on all these metrics that allows you to say, "Okay, I'm willing to take this additional money if I'm not hurting the user experience by more than this much." So there's a trade-off there.

我们提出了一个基于所有这些指标的 OEC，让您可以说：“好吧，如果我不会对用户体验造成更大的损害，我愿意接受这笔额外的钱。”所以这里需要权衡。

Ronny Kohavi: One of the nice ways to phrase this, as a constraint optimization problem.

Ronny Kohavi：将其表述为约束优化问题的好方法之一。

I want you to increase revenue, but I'm going to give you a fixed amount of average real estate that you can use.

我希望你增加收入，但我会为你提供固定数量的平均房地产供你使用。

So for one query, you can have zero ads.

因此，对于一个查询，您可以拥有零个广告。

For another query, you can have three ads.

对于另一个查询，您可以有三个广告。

For a third query, you can have wider, bigger ads.

对于第三个查询，您可以有更宽、更大的广告。

I'm just going to count the pixels that you take, the vertical pixels.

我只是要计算您拍摄的像素，即垂直像素。

And I will give you some budget.

我会给你一些预算。

And if you can under the same budget make more money, you're good to go.

如果您可以在相同的预算下赚更多的钱，那么您就可以开始了。

Ronny Kohavi: So that to me turns the problem from a badly defined, let's just make more money.

Ronny Kohavi：所以对我来说，这个问题不再是一个定义错误的问题，让我们赚更多的钱。

Any page can start plastering more ads and make more money short term, but that's not the goal.

任何页面都可以开始投放更多广告并在短期内赚更多钱，但这不是目标。

The goal is long-term growth and revenue.

目标是长期增长和收入。

Then you need to insert these other criteria, and what am I doing to the user experience?

然后你需要插入这些其他标准，我对用户体验做了什么？

One way around it is to put this constraint.

解决这个问题的一种方法是设置这个约束。

Another one is just to have these other metrics.

另一种是仅拥有这些其他指标。

Again, something that we did, to look at the user experience.

同样，我们所做的就是关注用户体验。

How long does it take the user to reach a successful click?

用户需要多长时间才能实现成功点击？

What percentage of sessions are successful?

会议成功的百分比是多少？

These are key metrics that were part of the overall evaluation criteria, that we've used.

这些是我们使用的总体评估标准的一部分的关键指标。

Ronny Kohavi: I can give you another example by the way, from the hotel industry or Airbnb that we both worked at.

Ronny Kohavi：顺便说一句，我可以给你举另一个例子，来自我们都工作过的酒店行业或 Airbnb。

You can say, "I want to improve conversion rate," but you can be smarter about it and say, "It's not just enough to convert a user to buy or to pay for a listing.

你可以说，“我想提高转化率”，但你也可以更聪明地说，“仅仅让用户转化为购买或支付列表是不够的。

I want them to be happy with it several months down the road when they actually stay there."

我希望他们几个月后真正住在那里时能够感到满意。”

Ronny Kohavi: So that could be part of your OEC to say, "What is the rating that they will give to that listing when they actually stay there?" And that causes an interesting problem, because you don't have this data now.

Ronny Kohavi：因此，您的 OEC 可能会说：“当他们实际入住该房源时，他们会给该房源评分多少？”这会导致一个有趣的问题，因为您现在没有这些数据。

You're going to have it three months from now when they actually stay.

三个月后，当他们真正留下来时，你就会得到它。

So you have to build the training set that allows you to make a prediction about whether this user, whether Lenny is going to be happy at this cheap place.

所以你必须建立训练集，让你能够预测这个用户，莱尼是否会在这个便宜的地方感到高兴。

Or whether no, I should offer him something more expensive, because Lenny likes to stay at nicer places where the water actually is hot and comes out of the faucet.

或者是否不，我应该给他一些更贵的东西，因为莱尼喜欢住在更好的地方，那里的水实际上是热的，从水龙头里出来的。

Lenny: That is true.

莱尼：确实如此。

Okay, so it sounds like the core to this approach is basically have a drag metric that makes sure you're not hurting something that's really important to the business, and then being very clear on what's the long-term metric we care most about.

好吧，听起来这种方法的核心基本上是有一个阻力指标，以确保您不会伤害对业务真正重要的东西，然后非常清楚我们最关心的长期指标是什么。

Ronny Kohavi: To me, the key word is lifetime value, which is you have to define the OEC such that it is causally predictive of the lifetime value of the user.

Ronny Kohavi：对我来说，关键词是生命周期价值，也就是说你必须定义 OEC，使其能够因果预测用户的生命周期价值。

And that's what causes you to think about things properly, which is, am I doing something that just helps me short term, or am I doing something that will help me in the long term?

这就是让你正确思考事情的原因，也就是说，我做的事情是只对我短期有帮助，还是我做的事情对我长期有帮助？

Once you put that model of lifetime value, people say, "Okay, what about retention rates?

一旦你建立了终身价值模型，人们就会说：“好吧，保留率怎么样？

We can measure that.

我们可以衡量这一点。

What about the time to achieve a task?

完成任务的时间怎么样？

We can measure that." And those are these countervailing metrics that make the OEC useful.

我们可以衡量这一点。”这些是使 OEC 发挥作用的抵消指标。

Lenny: And to understand these longer term metrics, what I'm hearing is use models, and forecast, and predictions, or would you suggest sometimes use a long-term holdout or some other approach?

莱尼：为了理解这些长期指标，我听到的是使用模型、预测和预测，或者您是否建议有时使用长期坚持或其他方法？

What do you find is the best way to see these long term?

您认为实现这些长期目标的最佳方式是什么？

Ronny Kohavi: Yeah, so there's two ways that I like to think about it.

Ronny Kohavi：是的，所以我喜欢从两种角度来思考这个问题。

One is you can run long-term experiments for the goal of learning something.

一是你可以为了学习一些东西而进行长期实验。

So I mentioned that at Bing, we did run these experiments where we increased the ads and decreased the ads, so that we will understand what happens to key metrics.

所以我提到，在 Bing，我们确实进行了这些实验，增加了广告并减少了广告，以便我们了解关键指标会发生什么。

Ronny Kohavi: The other thing is you can just build models that use some of our background knowledge or use some data science to look at historical...

Ronny Kohavi：另一件事是，你可以使用我们的一些背景知识或使用一些数据科学来构建模型来查看历史......

I'll give you another good example of this.

我会给你另一个很好的例子。

When I came to Amazon, one of the teams that reported to me was the email team that it was not the transactional emails when you buy something, you get an email.

当我来到亚马逊时，向我报告的团队之一是电子邮件团队，当你购买东西时，你会收到一封电子邮件，这不是交易电子邮件。

But it was the team that sent these recommendations.

但这些建议是由团队发出的。

"Here's a book by an author that you bought.

“这是你买的一本作者写的书。

Here's a product that we recommend." And the question is, how do we give credit to that team?

这是我们推荐的产品。”问题是，我们如何给予该团队荣誉？

Ronny Kohavi: And the initial version was, whenever a user comes from the email and purchases something on Amazon, we're going to give that email credit.

Ronny Kohavi：最初的版本是，每当用户通过电子邮件在亚马逊上购买商品时，我们都会将该电子邮件归功于该电子邮件。

Well, it turned out this had no countervailing metric.

好吧，事实证明这没有抵消指标。

The more emails you send, the more money you're going to credit the team.

您发送的电子邮件越多，您向团队提供的资金就越多。

And so that led to spam.

这导致了垃圾邮件。

Literally a really interesting problem.

从字面上看，这是一个非常有趣的问题。

The team just ramped up the number of emails that they were sending out, and claimed to make more money, and their fitness function improved.

该团队只是增加了发送的电子邮件数量，并声称可以赚更多钱，并且他们的健身功能也得到了改善。

Ronny Kohavi: So then we backed up and then we said, "Okay, we can either phrase this as a constraint satisfaction problem.

Ronny Kohavi：所以我们后退了一步，然后我们说，“好吧，我们可以将其表述为约束满足问题。

You're allowed to send user an email every X days or," which is what we ended up doing is, "Let's model the cost of spamming the users."

您可以每 X 天向用户发送一封电子邮件，或者，”这就是我们最终所做的，“让我们对向用户发送垃圾邮件的成本进行建模。”

Ronny Kohavi: What's that cost?

Ronny Kohavi: What's that cost?

Well, when they unsubscribe, we can't mail them.

好吧，当他们取消订阅时，我们就无法邮寄他们。

So we did some data science study on the side and we said, "What is the value that we're losing from an unsubscribe?" And we came up with a number, it was a few dollars.

因此，我们兼职做了一些数据科学研究，然后我们说：“取消订阅会给我们带来什么价值？”我们想出了一个数字，就是几美元。

But the point was, now we have this countervailing metric.

但关键是，现在我们有了这个抵消指标。

We say, "Here's the money that we generate from the emails.

我们说：“这是我们从电子邮件中赚到的钱。

Here's the money that we're losing on long-term value.

这是我们在长期价值上损失的钱。

What's the trade-off?" And then when we started to incorporate those formula, more than half the campaigns that were being sent were negative.

有何权衡？”然后，当我们开始合并这些公式时，所发送的活动中有一半以上都是负面的。

Ronny Kohavi: So it was a huge insight at Amazon about how to send the right campaigns.

Ronny Kohavi：因此，亚马逊对于如何发送正确的营销活动有着深刻的见解。

And this is what I like about these discoveries.

And this is what I like about these discoveries.

This fact that we integrated the unsubscribe led us to a new feature to say, "Well, let's not lose their future lifetime value through email.

我们集成了取消订阅功能，这一事实使我们获得了一项新功能：“好吧，我们不要通过电子邮件失去他们未来的终身价值。

When they unsubscribe, let's offer them by default to unsubscribe from this campaign."

当他们取消订阅时，我们默认让他们取消订阅此活动。”

Ronny Kohavi: So when you get an email, there's a new book by the author.

Ronny Kohavi：所以当你收到一封电子邮件时，里面有作者的一本新书。

The default to unsubscribe would be unsubscribe me from author emails.

默认取消订阅是取消我对作者电子邮件的订阅。

And so now, the negative, the countervailing metric is much smaller.

所以现在，负面的、抵消的指标要小得多。

And so again, this was a breakthrough in our ability to send more emails, and understand based on what users were unsubscribing from, which ones are really beneficial.

再说一次，这是我们发送更多电子邮件的能力的突破，并根据用户取消订阅的内容了解哪些邮件真正有益。

Lenny: I love the surprising results.

Lenny: I love the surprising results.

Ronny Kohavi: We all love them.

Ronny Kohavi: We all love them.

This is the humbling reality.

This is the humbling reality.

And people talk about the fact that A/B testing sometimes leads you to incremental...

人们谈论这样一个事实：A/B 测试有时会导致你渐进......

I actually think that many of these small insights lead to fundamental insights about which areas to go, some strategies we should take, some things we should develop.

事实上，我认为许多这些小见解会带来一些基本见解，比如我们应该去哪些领域、我们应该采取一些策略、我们应该开发一些东西。

Helps a lot.

有很大帮助。

Lenny: This makes me think about how every time I've done a full redesign of a product, I don't think ever, has it ever been a positive result.

莱尼：这让我思考，每次我对产品进行全面重新设计时，我都不认为它曾经取得过积极的结果。

And then the team always ends up having to claw back what they just hurt and try to figure out what they messed up.

然后团队最终总是不得不挽回他们刚刚造成的伤害，并试图找出他们搞砸的地方。

Is that your experience too?

这也是你的经历吗？

Ronny Kohavi: Absolutely, yeah.

Ronny Kohavi: Absolutely, yeah.

In fact, I've published some of these in LinkedIn posts showing a large set of big launches and redesigns that dramatically failed, and it happens very often.

事实上，我在 LinkedIn 帖子中发布了其中一些内容，展示了一系列重大的发布和重新设计，但都以失败告终，而且这种情况经常发生。

So the right way to do this is to say, "Yes, we want to do a redesign, but let's do it in steps and test on the way and adjust," so you don't need to take 17 new changes, that many of them are going to fail.

因此，正确的方法是说，“是的，我们想要重新设计，但让我们分步骤进行，并在途中进行测试和调整”，这样你就不需要进行 17 个新的更改，因为其中许多更改都会失败。

Start to move incrementally in a direction that you believe is beneficial.

开始朝着您认为有益的方向逐步前进。

Adjust on the way.

途中调整。

Lenny: The worst part of those experiences I find is it took three to six months to build it.

莱尼：我发现这些经历中最糟糕的部分是花了三到六个月的时间来构建它。

And by the time it's launched, it's just like, "We're not going to unlaunch this.

当它推出时，就像是，“我们不会取消它。

Everyone's been working in this direction.

Everyone's been working in this direction.

All the new features are assuming this is going to work," and you're basically stuck.

所有新功能都假设这会起作用，”而你基本上被困住了。

Ronny Kohavi: I mean, this is a sunk cost fallacy.

Ronny Kohavi：我的意思是，这是一个沉没成本谬误。

We invested so many years in it.

We invested so many years in it.

Let's launch this, even though it's bad for the user.

让我们启动它，尽管这对用户不利。

No, that's terrible.

不，那太可怕了。

Yeah.

是的。

Yeah.

是的。

So this is the other advantage of recognizing this humble reality that most ideas fail.

因此，这是认识到大多数想法都失败的卑微现实的另一个优势。

If you believe in that statistics that I published, then doing 17 changes together is more likely to be negative.

如果您相信我发布的统计数据，那么一起进行 17 项更改更有可能是负面的。

Do them in smaller increments, learn from, it's called OFAT one-factor-at-a-time.

以较小的增量进行，从中学习，这称为 OFAT 一次一因素。

Do one factor, learn from it, and adjust.

Do one factor, learn from it, and adjust.

Of the 17, maybe you have four good ideas.

Of the 17, maybe you have four good ideas.

Those are the ones that will launch and be positive.

这些是那些将启动并且是积极的。

Lenny: I generally agree with that, and always try to avoid a big redesign, but it's hard to avoid them completely.

Lenny：我总体上同意这一点，并且总是尽力避免大的重新设计，但很难完全避免它们。

There's often team members that are really passionate like, "We just need to rethink this whole experience.

经常有一些团队成员非常热情，比如“我们只需要重新思考整个体验。

We're not going to incrementally get there." Have you found anything effective in helping people either see this perspective, or just making a larger bet more successful?

We're not going to incrementally get there."您是否发现任何有效的方法可以帮助人们了解这种观点，或者只是让更大的赌注更加成功？

Ronny Kohavi: By the way, I'm not opposed to large redesigns.

Ronny Kohavi：顺便说一下，我并不反对大规模的重新设计。

I try to give the team the data to say, "Look, here are lots of examples where big redesigns fail." Try to decompose your redesign if you can't decompose it to one factor at a time, to a small set of factors at a time.

我尝试向团队提供数据，告诉他们：“看，这里有很多大型重新设计失败的例子。”如果您无法一次将其分解为一个因素或一次一小部分因素，请尝试分解您的重新设计。

And learn from these smaller changes what works and what doesn't.

并从这些较小的变化中学习哪些有效，哪些无效。

Ronny Kohavi: Now, it's also possible to do a complete redesign.

Ronny Kohavi：现在，也可以进行彻底的重新设计。

Just, as you said yourself, be ready to fail.

Just, as you said yourself, be ready to fail.

I mean, do you really want to work on something for six months or a year, and then run the A/B test, and realize that you've hurt revenues or other key metrics by several percentage points?

我的意思是，您真的想在某件事上工作六个月或一年，然后运行 ​​A/B 测试，并意识到您已经使收入或其他关键指标损失了几个百分点吗？

And a data-driven organization will not allow you to launch.

数据驱动的组织不会让你启动。

What are you going to write in your annual review?

你会在年度回顾中写什么？

Lenny: But nobody ever thinks it's going to fail.

莱尼：但没有人认为它会失败。

They think, "No, we got this.

They think, "No, we got this.

We've talked to so many people."

我们已经和很多人谈过了。”

Ronny Kohavi: But I think organizations that start to run experiments are humbled early on from the smaller changes.

Ronny Kohavi：但我认为开始进行实验的组织很早就会因为较小的变化而感到谦卑。

Right?

正确的？

You're right.

你说得对。

I'll tell you a funny story.

我给你讲一个有趣的故事。

When I came from Amazon to Microsoft, I joined the group, and for one reason or another, that group disbanded a month after I joined.

当我从亚马逊来到微软时，我加入了这个小组，但由于某种原因，该小组在我加入一个月后就解散了。

Ronny Kohavi: And so people came to me and said, "Look, you just joined the company.

Ronny Kohavi：所以人们来找我说：“看，你刚刚加入公司。

You're at partner level.

您处于合作伙伴级别。

You figure out how you can help Microsoft." And I said, "I'm going to build an experimentation platform," because nobody at Microsoft is running experiments.

You figure out how you can help Microsoft."我说，“我要建立一个实验平台”，因为微软没有人进行实验。

And more than 50% of ideas in Amazon that we tried failed.

我们在亚马逊尝试过的想法有超过 50% 都失败了。

And the classical response was, "We have better PMs here."

典型的回答是：“我们这里有更好的产品经理。”

Ronny Kohavi: Right?

罗尼·科哈维：对吗？

There was this complete denial that it's possible that 50% of ideas that Microsoft is implementing, in a three-year development cycle by the way.

顺便说一下，人们完全否认微软可能在三年的开发周期中实现 50% 的想法。

This is how long it took Office to release.

This is how long it took Office to release.

It was a classical every three years we release.

It was a classical every three years we release.

Ronny Kohavi: And the data came about showing that Bing was the first to truly implement experimentation at scale.

Ronny Kohavi：数据显示 Bing 是第一个真正大规模实施实验的公司。

And we shared with the rest of the companies the surprising results.

我们与其他公司分享了令人惊讶的结果。

And so when Office was...

所以当 Office...

And this was credit to Qi Lu and Satya Nadella, they were ones that says, "Ronny, you try to get Office to run experiments.

这要归功于齐鲁和萨蒂亚·纳德拉，他们说：“罗尼，你尝试让 Office 进行实验。

We'll give you the air support." And it was hard, but we did it.

我们将为你提供空中支援。”这很难，但我们做到了。

It took a while, but Office started to run experiments, and they realized that many of their ideas are failing.

Office 花了一段时间才开始进行实验，他们意识到他们的许多想法都失败了。

Lenny: You said that there's a site of a failed redesigns.

莱尼：你说有一个网站重新设计失败了。

Is that in your book or is that a site that you can point people to, to help build this case?

这是在你的书中吗？或者是你可以引导人们访问的网站，以帮助建立这个案例？

Ronny Kohavi: I teach this in my class, but I think I've posted this on LinkedIn and answered some questions.

Ronny Kohavi：我在课堂上教授这个，但我想我已经在 LinkedIn 上发布了这个并回答了一些问题。

I'm happy to put that in the notes.

我很高兴把它记在笔记中。

Lenny: Okay, cool.

莱尼：好的，酷。

We'll put that in the show notes.

我们会将其放在演出笔记中。

Because I think that's the kind of data that often helps convince a team, "Maybe we shouldn't rethink this entire onboarding flow from scratch.

因为我认为这种数据通常有助于说服团队，“也许我们不应该从头开始重新考虑整个入职流程。

Maybe we should iterate towards and learn as we go."

也许我们应该不断迭代并不断学习。”

Lenny: This episode is brought to you by Eppo.

Lenny：这一集是 Eppo 为您带来的。

Eppo is a next generation A/B testing platform built by Airbnb alums for modern growth teams.

Eppo 是 Airbnb 校友为现代增长团队打造的下一代 A/B 测试平台。

Companies like DraftKings, Zapier, ClickUp, Twitch, and Cameo rely on Eppo to power their experiments.

DraftKings、Zapier、ClickUp、Twitch 和 Cameo 等公司依靠 Eppo 来支持他们的实验。

Lenny: Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern growth team stack.

Lenny：无论您在哪里工作，进行实验都变得越来越重要，但是没有可以与现代增长团队堆栈集成的商业工具。

This leads to wasted time building internal tools or trying to run your own experiments through a clunky marketing tool.

这会导致浪费时间构建内部工具或尝试通过笨重的营销工具进行自己的实验。

Lenny: When I was at Airbnb, one of the things that I loved most about working there was our experimentation platform, where I was able to slice and dice data by device types, country, user stage.

Lenny：当我在 Airbnb 工作时，我最喜欢在那里工作的事情之一就是我们的实验平台，在这里我能够按设备类型、国家/地区、用户阶段对数据进行切片和切块。

Lenny: Eppo does all that and more, delivering results quickly, avoiding annoying prolonged analytic cycles, and helping you easily get to the root cause of any issue you discover.

Lenny：Eppo 做到了这一切，甚至更多，快速提供结果，避免恼人的长时间分析周期，并帮助您轻松找到所发现问题的根本原因。

Eppo lets you go beyond basic click through metrics, and instead use your North Star metrics like activation, retention, subscription and payments.

Eppo 可以让您超越基本的点击量指标，而是使用您的北极星指标，例如激活、保留、订阅和付款。

Eppo supports tests on the front end, on the back end, email marketing, even machine learning clients.

Eppo 支持前端、后端、电子邮件营销、甚至机器学习客户端的测试。

Check out Eppo at geteppo.com, that's geteppo.com, and 10X your experiment velocity.

在 geteppo.com 上查看 Eppo，即 geteppo.com，您的实验速度将提高 10 倍。

Lenny: Is it ever worth just going, "Let's just rethink this whole thing and just give it a shot," to break out of a local minima or local maxima essentially?

莱尼：是否值得“让我们重新思考整个事情并尝试一下”，以从本质上突破局部最小值或局部最大值？

Ronny Kohavi: Yeah.

罗尼·科哈维：是的。

So I think what you said is fair.

所以我觉得你说的很公平。

I do want to allocate some percentage of resources to big bets.

我确实想将一定比例的资源分配给大赌注。

As you said, we've been optimizing this thing to hell.

正如你所说，我们一直在优化这个东西。

Could we completely redesign it?

我们可以完全重新设计它吗？

It's a very valid idea.

这是一个非常有效的想法。

You may be able to break out of a local minima.

您也许能够突破局部最小值。

What I'm telling you is 80% of the time, you will fail.

我告诉你的是，80%的情况下你都会失败。

So be ready for that.

所以要做好准备。

What people usually expect is, "My redesign is going to work." No, you're most likely going to fail, but if you do succeed, it's a breakthrough.

人们通常期望的是“我的重新设计将会起作用”。不，你很可能会失败，但如果你成功了，那就是一个突破。

Lenny: I like this 80% rule of thumb.

Lenny：我喜欢这个 80% 的经验法则。

Is that just a simple way of thinking about it?

这只是一种简单的思考方式吗？

80% of your-

你的80%——

Ronny Kohavi: That's my rule of thumb.

Ronny Kohavi：这是我的经验法则。

And I've heard people say it's 70% or 80%.

我听人们说是 70% 或 80%。

But it's in that area where I think when you talk about how much to invest in the known versus the high risk, high reward, that's usually the right percentage that most organizations end up doing this allocation, right?

但正是在这个领域，我认为当你谈论在已知的投资与高风险、高回报的投资时，这通常是大多数组织最终进行这种分配的正确百分比，对吗？

You interviewed Shreyas.

您采访了 Shreyas。

I think he mentioned that Google is like 70% the searching ads, and it's 20% for some of the apps and new stuff, and then it's the 10% for infrastructure.

我想他提到 Google 70% 负责搜索广告，20% 负责一些应用程序和新东西，然后 10% 负责基础设施。

Lenny: And I think the most important point there is if you're not running an experiment, 70% of stuff you're shipping is hurting your business.

Lenny：我认为最重要的一点是，如果你不进行实验，你所运送的 70% 的东西都会损害你的业务。

Ronny Kohavi: Well, it's not hurting, it's flat too negative.

Ronny Kohavi：嗯，这并没有什么坏处，只是太消极了。

Some of them are flat.

其中一些是平坦的。

And by the way, flat to me, if something is not Statsig, that's a no ship, because you've just introduced more code.

顺便说一句，对我来说，如果某些东西不是 Statsig，那就不行，因为您刚刚引入了更多代码。

There is a maintenance overhead to shipping your stuff.

运输你的东西需要维护费用。

I've heard people say, "Look, we already spent all this time.

我听到人们说：“看，我们已经花了这么长时间了。

The team will be demotivated if we don't ship it." And I'm, "No, that's wrong guys.

如果我们不发货，团队就会失去动力。”我说：“不，那是错误的，伙计们。

Let's make sure that we understand that shipping this project has no value, is complicating the code base.

让我们确保我们明白，发布这个项目没有任何价值，只会使代码库变得复杂。

Maintenance costs will go up." You don't ship on flat, unless it's a legal requirement.

维护成本将会上升。”除非法律要求，否则您不能平装运输。

When legal comes along and says, "You have to do X or Y," you have to ship on flat or even negative.

当法律出现并说“你必须做 X 或 Y”时，你必须平价甚至负数发货。

And that's understandable.

这是可以理解的。

Ronny Kohavi: But again, I think that's something that a lot of people make the mistake of saying, "Legal told us we have to do this, therefore we're going to take the hits." No, legal gave you a framework that you have to work under.

罗尼·科哈维：但是，我认为很多人都会犯这样的错误：“法律告诉我们必须这样做，因此我们要承受打击。”不，法律给了你一个你必须在其下工作的框架。

Try three different things, and ship the one that hurts the least.

尝试三种不同的方法，然后选择伤害最小的一种。

Lenny: That reminds me when Airbnb launched the rebrand, even that they ran as an experiment with the entire homepage redesigned, the new logo, and all that.

莱尼：这让我想起了 Airbnb 推出品牌重塑的时候，甚至他们还进行了一次实验，重新设计了整个主页、新标志等等。

And I think there was a long-term holdout even, and I think it was positive in the end from what I remember.

我认为甚至存在长期的抵制，而且根据我的记忆，我认为最终这是积极的。

Lenny: Speaking of Airbnb, I want to chat about Airbnb briefly.

Lenny：说到Airbnb，我想简单聊聊Airbnb。

I know you're limited in what you can share, but it's interesting that Airbnb seems to be moving in this other direction where it's becoming a lot more top-down, Brian vision oriented.

我知道你能分享的内容有限，但有趣的是，Airbnb 似乎正在朝另一个方向发展，变得更加自上而下、以 Brian 愿景为导向。

And Brian's even talked about how he's less motivated to run experiments.

布莱恩甚至谈到了他进行实验的积极性降低了。

He doesn't want to run as many experiments as they used to.

他不想像以前那样进行那么多实验。

Things are going well, and so it's hard to argue with the success potentially.

事情进展顺利，所以很难否认潜在的成功。

You worked there for many years.

你在那里工作了很多年。

You ran the search team essentially.

基本上你负责搜索小组。

I guess, what was your experience like there?

我想，你在那里的经历是什么样的？

And then roughly, what's your sense of how things are going, where it's going?

然后粗略地讲，您对事情的进展、进展有何感觉？

Ronny Kohavi: Well as you know, I'm restricted from talking about Airbnb.

Ronny Kohavi：如你所知，我被禁止谈论 Airbnb。

I will say a few things that I am allowed to say.

我会说一些我被允许说的话。

One is in my team in search relevance, everything was A/B tested.

其中之一是我团队中的搜索相关性，所有内容都经过 A/B 测试。

So while Brian can focus on some of the design aspects, the people who are actually doing the neural networks and the search, everything was A/B tested to hell.

因此，虽然 Brian 可以专注于一些设计方面，但实际上从事神经网络和搜索的人，一切都经过了地狱般的 A/B 测试。

So nothing was launching without an A/B test.

因此，如果没有进行 A/B 测试，任何产品都无法启动。

We had targets around improving certain metrics, and everything was done A/B test.

我们的目标是改进某些指标，并且一切都完成了 A/B 测试。

Ronny Kohavi: Now other teams, some did, some did not.

Ronny Kohavi：现在其他球队，有些做了，有些没有。

I will say that when you say things are going well, I think we don't know the counterfactual.

我想说，当你说事情进展顺利时，我认为我们不知道反事实。

I believe that had Airbnb kept people like Greg Greeley, which was pushing for a lot more data driven, and had Airbnb run more experiments, it would've been in a better state than today.

我相信，如果 Airbnb 留住像格雷格·格里利这样的人，推动更多的数据驱动，如果 Airbnb 进行更多的实验，它的状态会比今天更好。

But it's the counterfactual.

但这是反事实的。

We don't know.

我们不知道。

Lenny: That's a really interesting perspective.

莱尼：这是一个非常有趣的观点。

Airbnb's such an interesting natural experiment of a way of doing things differently.

Airbnb 是一个非常有趣的自然实验，它以不同的方式做事。

There's de-emphasizing experiments, and also, they turned off paid ads during Covid.

他们进行了一些不那么重要的实验，而且他们在新冠疫情期间关闭了付费广告。

And I don't know where it is now, but it feels like it's become a much smaller part of the growth strategy.

我不知道它现在在哪里，但感觉它已经成为增长战略的一小部分。

Who knows if they've ramped it up to back to where it's today, but I think it's going to be a really interesting case study looking back five, 10 years from now.

谁知道他们是否已经恢复到今天的水平，但我认为从现在起五年、十年后回顾这将是一个非常有趣的案例研究。

Ronny Kohavi: It's a one-off experiment where it's hard to assign value to some of the things that Airbnb is doing.

Ronny Kohavi：这是一项一次性实验，很难为 Airbnb 正在做的一些事情赋予价值。

I personally believe it could have been a lot bigger and a lot more successful if it had run more controlled experiments.

我个人认为，如果它进行更多受控实验，它可能会更大、更成功。

But I can't speak about some of those that I ran and that showed that some of the things that were initially untested were actually negative and could be better.

但我不能谈论我运行的一些测试，这些测试表明一些最初未经测试的测试实际上是负面的，并且可以做得更好。

Lenny: All right.

莱尼：好吧。

Mysterious.

神秘。

One more question.

还有一个问题。

Airbnb, you were there during Covid, which was quite a wild time for Airbnb.

Airbnb，您在新冠疫情期间就在那里，那对 Airbnb 来说是一段非常疯狂的时期。

We had Sanchan on the podcast talking about all the craziness that went on when travel basically stopped, and there was a sense that Airbnb was done, and travel's not going to happen for years and years.

Sanchan 在播客中谈到了当旅行基本上停止时发生的所有疯狂事情，人们有一种感觉 Airbnb 已经完蛋了，旅行在很多年里都不会发生。

What's your take on experimentation in that world where you have to really move fast, make crazy decisions, and make big decisions?

在那个必须快速行动、做出疯狂决定和重大决定的世界里，你对实验有何看法？

What was it like during that time?

那段时间是什么样子的？

Ronny Kohavi: So I think actually in a state like that, it's even more important to run A/B tests, right?

Ronny Kohavi：所以我认为实际上在这样的状态下，运行 A/B 测试更加重要，对吧？

Because what you want to be able to see is if we're making this change, is it actually helping in the current environment?

因为您希望看到的是，如果我们进行此更改，它实际上对当前环境有帮助吗？

There's this idea of external generalizability.

有一种外部普遍性的想法。

Is it going to work out now during Covid?

现在在新冠疫情期间，这一切会奏效吗？

Is it going to generalize later on?

以后会普遍化吗？

These are things that you can really answer with the controlled experiments, and sometimes it means that you might have to replicate them six months down when Covid say is not as impactful as it is.

这些都是你可以通过对照实验真正回答的问题，有时这意味着当新冠病毒所说的影响力不如现在时，你可能必须在六个月后重复这些问题。

Ronny Kohavi: Saying that you have to make decisions quickly, to me, I'll point you to the success rate.

Ronny Kohavi：说你必须快速做出决定，对我来说，我会向你指出成功率。

If in peace time you're wrong two thirds to 80% of the time, why would you be subtly right in wartime, in Covid time?

如果在和平时期你三分之二到80%的时间都是错的，那为什么在战时、在新冠疫情时期你会微妙地正确呢？

Ronny Kohavi: So I don't believe in the idea that because bookings went down materially, the company should suddenly not be data driven and do things differently.

Ronny Kohavi：所以我不相信因为预订量大幅下降，公司就应该突然不再由数据驱动并以不同的方式做事。

I think if Airbnb stayed the course, did nothing, the revenue would've gone up in the same way.

我认为，如果 Airbnb 坚持到底，什么都不做，收入也会以同样的方式增长。

Ronny Kohavi: In fact, if you look at one investment, one big investment that was done at the time was online experiences, and the initial data wasn't very promising.

Ronny Kohavi：事实上，如果你看一项投资，当时所做的一项大投资是在线体验，最初的数据并不乐观。

And I think today, it's a footnote.

我认为今天，这是一个脚注。

Lenny: Yeah.

莱尼：是的。

Another case study for the history books, Airbnb experiences.

历史书上的另一个案例研究是 Airbnb 的经历。

I want to shift a little bit and talk about your book, which you mentioned a couple times.

我想稍微转移一下话题，谈谈你的书，你曾多次提到过这本书。

It's called Trustworthy Online Controlled Experiments, and I think it's basically the book on A/B testing.

这本书叫做《值得信赖的在线控制实验》，我认为它基本上是一本关于 A/B 测试的书。

Let me ask you, what surprised you most about writing this book, and putting it out, and the reaction to it?

让我问你，写这本书、出版这本书以及人们的反应最令你惊讶的是什么？

Ronny Kohavi: I was pleasantly surprised that it sold more than what we thought, more than what Cambridge predicted.

Ronny Kohavi：我很惊喜，它的销量超出了我们的想象，超出了剑桥的预测。

So when first we were approached by Cambridge after a tutorial that we did to write a book, I was like, "I don't know, this is too small of a niche area."

因此，当剑桥大学第一次在我们写书的教程结束后联系我们时，我的反应是，“我不知道，这个领域太小了。”

Ronny Kohavi: And they were saying, "So you'll be able to sell a few thousand copies and help the world." And I found my co-authors, which are great.

罗尼·科哈维：他们说，“这样你就能卖出几千本，帮助世界。”我找到了我的合著者，他们都很棒。

And we wrote a book that we thought is not statistically oriented, has fewer formulas than you normally see, and focuses on the practical aspects and on trust, which is the key.

我们写了一本我们认为不是以统计为导向的书，公式比你通常看到的要少，并且侧重于实践方面和信任，这是关键。

Ronny Kohavi: The book, as I said, was more successful.

罗尼·科哈维：正如我所说，这本书更成功。

It sold over 20,000 copies in English.

该书英文版销量超过 20,000 册。

It was translated to Chinese, Korean, Japanese, and Russian.

它被翻译成中文、韩文、日文和俄文。

And so it's great to see that we help the world become more data-driven with experimentation, and I'm happy because of that.

因此，很高兴看到我们通过实验帮助世界变得更加数据驱动，因此我很高兴。

I was pleasantly surprised.

我很惊喜。

Ronny Kohavi: By the way, all proceeds from the book are donated to charity.

Ronny Kohavi：顺便说一下，这本书的所有收益都捐给了慈善机构。

So if I'm pitching the book here, there is no financial gain for me from having more copies sold.

因此，如果我在这里推销这本书，那么卖出更多的书对我来说并没有任何经济收益。

I think we made that decision, which was a good decision.

我认为我们做出了这个决定，这是一个很好的决定。

All proceeds go with the charity.

所有收益都捐献给慈善机构。

Lenny: Amazing.

莱尼：太棒了。

I didn't know that.

我不知道。

We'll link to the book in the show notes.

我们将在节目笔记中链接到这本书。

Trust is in the title.

信任就在标题里。

You just mentioned how important trust is to experimentation.

您刚才提到信任对于实验有多重要。

A lot of people talk about, "How do I run experiments faster?" You focus a lot on trust.

很多人都在谈论“如何更快地进行实验？”你非常注重信任。

Why is trust so important in running experiments?

为什么信任在实验中如此重要？

Ronny Kohavi: So to me, the experimentation platform is the safety net, and it's an oracle.

Ronny Kohavi：所以对我来说，实验平台就是安全网，它是一个预言机。

So it serves really two purposes.

所以它实际上有两个目的。

The safety net means that if you launch something bad, you should be able to abort quickly, right?

安全网意味着如果你启动了一些不好的东西，你应该能够快速中止，对吗？

Safe deployments, safe velocity.

安全部署，安全速度。

There's some names for this.

这有一些名称。

But this is one key value that the platform can give you.

但这是该平台可以为您提供的一项关键价值。

Ronny Kohavi: The other one, which is the more standard one, is at the end of the two-week experiment, we will tell you what happened to your key metric and to many of the other surrogate, and debugging, and guardrail metrics.

Ronny Kohavi：另一个是更标准的，在为期两周的实验结束时，我们将告诉您关键指标以及许多其他替代指标、调试指标和护栏指标发生了什么情况。

Trust builds up, it's easy to lose.

信任建立起来，却很容易失去。

Ronny Kohavi: And so to me, it is very important that when you present this and say, "This is science, this is a controlled experiment, this is the resolve," you better believe that this is trustworthy.

罗尼·科哈维：所以对我来说，非常重要的是，当你提出这个并说“这是科学，这是一个受控实验，这是解决方案”时，你最好相信这是值得信赖的。

Ronny Kohavi: And so I focus on that a lot.

Ronny Kohavi：所以我非常关注这一点。

I think it allowed us to gain the organizational trust that this is really...

我认为这让我们获得了组织的信任，这确实是......

And the nice thing is when we built all this checks to make sure that the experiment is correct, if there were something wrong with it, we would stop and say, "Hey, something is wrong with the experiment."

令人高兴的是，当我们进行所有这些检查以确保实验正确时，如果出现问题，我们会停下来说：“嘿，实验出了问题。”

Ronny Kohavi: And I think that's something that some of the early implementations in other places did not do, and it was a big mistake.

Ronny Kohavi：我认为其他地方的一些早期实施没有做到这一点，这是一个很大的错误。

I've mentioned this in my book so I can mention this here.

我在我的书中提到过这一点，所以我可以在这里提到这一点。

Ronny Kohavi: Optimizely in its early days were very statistically naive.

Ronny Kohavi：Optimizely 在早期在统计上非常幼稚。

They sort of said, "Hey, we're real time.

他们有点说，“嘿，我们是实时的。

We can compute your P values in real time," and then you can stop an experiment when the P value is statistically significant.

我们可以实时计算您的 P 值”，然后当 P 值具有统计显着性时您可以停止实验。

That is a big mistake.

这是一个很大的错误。

That inflates your, what's called type one error or the false positive rate materially.

这大大增加了所谓的一类错误或误报率。

So if you think you've got a 5% type one error, or you aim for that P value less than 0.05, using real time P value monitoring to optimize the offer, you would probably have a 30% error rate.

因此，如果您认为自己的第一类错误率为 5%，或者您的目标是 P 值小于 0.05，那么使用实时 P 值监控来优化报价，您可能会遇到 30% 的错误率。

Ronny Kohavi: So what this led is that people that started using Optimizely thought that the platform was telling them they were very successful.

Ronny Kohavi：这导致开始使用 Optimizely 的人们认为该平台告诉他们他们非常成功。

But when they actually started to see, "Well it told us this is positive revenue, but I don't see this over time.

但当他们真正开始看到时，“它告诉我们这是正收入，但随着时间的推移，我看不到这一点。

By now, we should have made double the money."

到现在为止，我们应该赚双倍的钱。”

Ronny Kohavi: So their questions started to come up around the trust in the platform.

Ronny Kohavi：所以他们的问题开始围绕对平台的信任出现。

There's a very famous post that somebody wrote about how, "Optimizely almost got me fired," by a person who basically said, "Look, I came to the org.

有一篇非常著名的帖子，有人写了一篇关于“优化差点让我被解雇”的文章，作者基本上是说：“看，我来到了这个组织。

I said, 'We have all these successes.' But then I said, 'Something is wrong.'"

我说：“我们取得了所有这些成功。”但后来我说，‘出了点问题。’”

Ronny Kohavi: And he tells of how he ran an A/A test when there is no difference between the A and the B.

Ronny Kohavi：他讲述了当 A 和 B 之间没有差异时他如何进行 A/A 测试。

And Optimizely told him that it was statistically significant too many times.

Optimizely 告诉他，统计上显着的次数太多了。

Optimizely learned.

优化学习了。

Optimizely, several people pointed, I pointed this out in my Amazon review of the book that the authors wrote early on.

一些人指出，最理想的是，我在亚马逊对作者早期撰写的这本书的评论中指出了这一点。

I said, "Hey, you're not doing the statistics correctly."

我说：“嘿，你的统计数据不正确。”

Ronny Kohavi: Ramesh Johari at Stanford pointed this out, became a consultant to the company, and then they fixed it.

罗尼·科哈维（Ronny Kohavi）：斯坦福大学的拉梅什·乔哈里（Ramesh Johari）指出了这一点，并成为该公司的顾问，然后他们解决了这个问题。

But to me, that's a very good example of how to lose trust.

但对我来说，这是如何失去信任的一个很好的例子。

They lost a lot of trust in the market.

他们对市场失去了很多信任。

They lost all this trust because they built something that had very much inflated error rate.

他们失去了所有这些信任，因为他们构建的东西错误率非常高。

Lenny: That is pretty scary to think about you've been running all these experiments, and they weren't actually telling you accurate results.

莱尼：想想你已经进行了所有这些实验，而他们实际上并没有告诉你准确的结果，这真是太可怕了。

What are signs that what you're doing may not be valid if you're starting to run experiments?

如果您开始进行实验，有哪些迹象表明您所做的事情可能无效？

And then how do you avoid having that situation?

那么如何避免出现这种情况呢？

What kind of tips can you share for people trying to run experiments?

您可以为尝试进行实验的人们分享哪些技巧？

Ronny Kohavi: There's a whole chapter of that in my book, but I'll say one of the things that is the most common occurrence by far, which is a sample ratio mismatch.

Ronny Kohavi：我的书中有一整章都提到了这一点，但我要说的是迄今为止最常见的事情之一，那就是样本比率不匹配。

Now, what is a sample ratio mismatch?

现在，什么是样本比率不匹配？

Ronny Kohavi: If you design the experiment to send 50% of users to control and 50% of users to treatment, it's supposed to be a random number, or a hash function.

Ronny Kohavi：如果你设计的实验是让 50% 的用户接受控制，50% 的用户接受治疗，那么它应该是一个随机数，或者一个哈希函数。

If you get something off from 50%, it's a red flag.

如果你的成绩低于 50%，那就是一个危险信号。

Ronny Kohavi: So let's take a real example.

Ronny Kohavi：让我们举一个真实的例子。

Let's say you're running an experiment, and it's large, it's got a million users, and you've got 50.2.

假设您正在运行一个实验，该实验很大，有 100 万用户，并且您有 50.2。

So people say, "Well, I don't know.

所以人们会说：“嗯，我不知道。

It's not going to be exactly the same as 50.2.

它不会与 50.2 完全相同。

Reasonable or not?" Well, there's a formula that you can plug in.

合理不合理？”嗯，有一个公式可以代入。

I have a spreadsheet available for those that are interested, and you can tell, here's how many users are in control.

我有一个电子表格可供感兴趣的人使用，您可以看出，这里有多少用户处于控制之中。

Here's how many users have in treatment.

这是有多少用户正在接受治疗。

My design was 50/50, and it tells you the probability that this could have happened by chance.

我的设计是 50/50，它告诉你这件事偶然发生的概率。

Ronny Kohavi: Now in a case like this, you plug in the numbers, it might tell you that this should happen one in half a million experiments.

Ronny Kohavi：现在在这样的情况下，你输入数字，它可能会告诉你这应该发生在五十万分之一的实验中。

Well, unless you've run half a million experiment, very unlikely that you would get a 50.2 versus 49.8 split.

好吧，除非您进行了 50 万次实验，否则不太可能获得 50.2 与 49.8 的差异。

And therefore, something is wrong with the experiment.

因此，实验出了问题。

Ronny Kohavi: I remember when we implemented this check, we were surprised to see how many experiments suffered from this.

Ronny Kohavi：我记得当我们实施这项检查时，我们惊讶地发现有多少实验遭受了这种情况。

Right?

正确的？

And there's a paper that was published, 2018, where we share that at Microsoft, even though we'd be running experiments for a while, is around 8% of experiments that suffered from the sample ratio mismatch.

2018 年发表了一篇论文，我们在微软分享了这一点，尽管我们已经进行了一段时间的实验，但仍有大约 8% 的实验遭受了样本比例不匹配的影响。

Ronny Kohavi: And it's a big number.

Ronny Kohavi：这是一个很大的数字。

I think about this.

我想这个。

You're running 20,000 experiments a year.

您每年要进行 20,000 次实验。

So many of them, 8% of them are invalid.

这么多，8%都是无效的。

And somebody has to go down and understand, what happened here?

必须有人下去了解，这里发生了什么？

We know that we can't trust the results, but why?

我们知道我们不能相信结果，但为什么呢？

Ronny Kohavi: So over time, you begin to understand there's something wrong with the data pipeline.

Ronny Kohavi：随着时间的推移，您开始意识到数据管道存在问题。

There's something that happens with bots.

机器人会发生一些事情。

Bots are a very common factor for causing sample ratio mismatch.

机器人程序是导致样本比例不匹配的一个非常常见的因素。

So that paper that was published by my team talks about how to diagnose sample ratio mismatches.

因此，我的团队发表的那篇论文讨论了如何诊断样本比率不匹配。

Ronny Kohavi: In the last probably year and a half, it was amazing to see all these third party companies implement sample ratio mismatches, and all of them were reporting, "Oh my god, 6%, 8%, 10%." So it's sometimes fun to go back and say, how many of your results in the past were invalid before you had this sample ratio mismatched test?

Ronny Kohavi：在过去的一年半时间里，令人惊讶的是看到所有这些第三方公司都实施了样本比例不匹配，而且他们都报告说，“天哪，6%、8%、10%。”因此，有时回过头来说，在进行样本比例不匹配测试之前，过去有多少结果是无效的？

Lenny: Yeah, that's frightening.

莱尼：是啊，这太可怕了。

Is the most common reason this happens is you're assigning users in the wrong place in your code?

发生这种情况的最常见原因是您在代码中的错误位置分配了用户吗？

Ronny Kohavi: So when you say most common, I think the most common is bots.

Ronny Kohavi：所以当你说最常见时，我认为最常见的是机器人。

Somehow, they hit the controller, the treatment in different proportions.

不知怎么的，他们按着控制器，按不同的比例进行治疗。

Because you change the website, the bot may fail to parse the page, and try to hit it more often.

由于您更改了网站，机器人可能无法解析该页面，并尝试更频繁地访问该页面。

And that's a classical example.

这是一个经典的例子。

Another one is just the data pipeline.

另一种是数据管道。

Ronny Kohavi: We've had cases where we were trying to remove bad traffic under certain conditions, and it was skewed because of the control and treatment.

Ronny Kohavi：我们曾经遇到过这样的情况：我们试图在某些条件下消除不良流量，但由于控制和处理而出现了偏差。

I've seen people that start an experiment in the middle of the site on some page, but they don't realize that some campaign is pushing people from the side.

我见过有人在网站中间的某个页面上开始实验，但他们没有意识到某些活动正在从侧面推动人们。

Ronny Kohavi: So there's multiple reasons.

Ronny Kohavi：所以有多种原因。

It is surprising how often this happens.

令人惊讶的是这种情况发生的频率如此之高。

And I'll tell you a funny story, which is when we first added this test to the platform, we just put a banner say, "You have a sample ratio mismatch.

我会告诉你一个有趣的故事，当我们第一次将这个测试添加到平台时，我们只是放了一个横幅，上面写着：“你的样本比率不匹配。

Do not trust these results." And we noticed that people ignored it.

不要相信这些结果。”我们注意到人们忽视了它。

They were starting to present results that had this banner.

他们开始展示带有此横幅的结果。

Ronny Kohavi: And so we blanked out the scorecard.

Ronny Kohavi：所以我们取消了记分卡。

We put a big red, "Can't see this result.

我们放了一个大红色，“看不到这个结果。

You have a sample ratio mismatch.

您的样本比率不匹配。

Click to expose the results." And why we do we need that okay?

单击以公开结果。”为什么我们需要这个？

We need that okay button because you want to be able to debug the reasons, and sometimes the metrics help you understand why you have a sample ratio mismatch.

我们需要“确定”按钮，因为您希望能够调试原因，有时这些指标可以帮助您了解样本比率不匹配的原因。

Ronny Kohavi: So we blanked out the scorecard, we had this button, and then we started to see that people pressed the button and still presented the results of experiments with sample ratio mismatch.

Ronny Kohavi：所以我们清空了记分卡，有了这个按钮，然后我们开始看到人们按下按钮，但仍然呈现样本比例不匹配的实验结果。

And so we ended up with an amazing compromise, which is every number in the scorecard was highlighted with a red line, so that if you took a screenshot, other people could tell you how to sample ratio mismatch.

因此，我们最终达成了一个惊人的妥协，即记分卡中的每个数字都用红线突出显示，这样如果您截取了屏幕截图，其他人就可以告诉您如何对比率不匹配进行采样。

Ronny Kohavi: This is intuition.

Ronny Kohavi：这是直觉。

People just say, "Well, my was small, therefore I can still present the results." People want to see success.

人们只是说，“好吧，我的很小，所以我仍然可以展示结果。”人们希望看到成功。

I mean, this is a natural bias, and then we have to be very conscientious and fight that bias and say when something looks too good to be true, investigate.

我的意思是，这是一种自然的偏见，然后我们必须非常认真地与这种偏见作斗争，并在某些事情看起来好得令人难以置信时说，进行调查。

Lenny: Which is a great segue to something you mentioned briefly, something called Twyman's law.

莱尼：这是对你简短提到的一个很好的延续，即所谓的“特威曼定律”。

Yeah.

是的。

Can you talk about that?

你能谈谈吗？

Ronny Kohavi: Yeah.

罗尼·科哈维：是的。

So Twyman's law, the general statement is if any figure that looks interesting or different is usually wrong.

所以泰曼定律，一般来说，如果任何看起来有趣或不同的图形通常是错误的。

It was first said by this person in the UK who worked in radio media, but I'm a big fan of it.

这是由一位在英国广播媒体工作的人首先说的，但我是它的忠实粉丝。

And my main claim to people is if the result looks too good to be true, your normal movement of an experiment is under 1% and you suddenly have a 10% movement, hold the celebratory dinner.

我对人们的主要主张是，如果结果看起来好得令人难以置信，你的实验正常变化低于 1%，而你突然出现 10% 的变化，请举办庆祝晚宴。

It was just your first reaction, right?

这只是你的第一反应，对吧？

Let's take everybody to a fancy dinner, because we just improved revenue by millions of dollars.

让我们带大家去吃一顿丰盛的晚餐吧，因为我们刚刚增加了数百万美元的收入。

Hold that dinner, investigate, see, because there's a large probability that something is wrong with the result.

举行晚宴，调查一下，看看，因为结果很可能有问题。

And I will say that nine out of 10, when we call it Twyman's law, it is the case that we find some flaw in the experiment.

我想说的是，十分之九，当我们称之为泰曼定律时，我们会在实验中发现一些缺陷。

Ronny Kohavi: Now there are obviously outliers.

Ronny Kohavi：现在显然存在异常值。

That first experiment that I shared where we promoted that made long titles, that was successful.

我分享的第一个实验是我们推广的长标题的实验，结果很成功。

But that was replicated multiple times, and double and triple checked, and everything was good about it.

但这被重复了多次，并进行了两次和三次检查，一切都很好。

Many other results that were so big turn out to be false.

许多其他如此大的结果最终被证明是错误的。

So I'm a big fan of Twyman's law.

所以我非常喜欢特怀曼定律。

There's a deck, I could also give this in the note, where I shared some real examples of Twyman's law.

有一个套牌，我也可以在笔记中给出，其中我分享了一些特怀曼定律的真实例子。

Lenny: Amazing.

莱尼：太棒了。

I want to talk about rolling this out of companies and things that you run into that fail.

我想谈谈如何将其从公司中推出以及您遇到的失败的事情。

But before I get to that, I'd love for you to explain P value.

但在开始之前，我希望您能解释一下 P 值。

I know that people kind of misunderstand it, and this might be a good time to just help people understand, what is it actually telling you, P value of say 0.05?

我知道人们有点误解它，这可能是帮助人们理解的好时机，它实际上告诉你什么，比如 0.05 的 P 值？

Ronny Kohavi: I don't know if this is the right forum for explaining P values, because the definition of a P value is simple.

Ronny Kohavi：我不知道这是否是解释 P 值的正确论坛，因为 P 值的定义很简单。

What it hides is very complicated.

它隐藏的东西非常复杂。

So I'll say one thing, which is many people assign one minus P value as the probability that your treatment is better than control.

所以我要说一件事，很多人将 1 负 P 值指定为你的治疗优于对照的概率。

So you ran an experiment, you got a P value of 0.02.

所以你进行了一个实验，你得到的 P 值为 0.02。

They think there's a 98% probability that the treatment is better than the control.

他们认为治疗方法有 98% 的可能性优于对照方法。

That is wrong.

那是错误的。

So rather than defining P values, I want to caution everybody that the most common interpretation is incorrect.

因此，我不想定义 P 值，而是想提醒大家，最常见的解释是不正确的。

Ronny Kohavi: P value assumes, it's a conditional probability or an assumed probability.

Ronny Kohavi：P值假设，它是条件概率或假设概率。

It assumes that the null hypothesis is true.

它假设原假设为真。

And we're computing the probability that the data we're seeing matches the hypothesis, this null hypothesis.

我们正在计算我们看到的数据与假设（零假设）相匹配的概率。

Ronny Kohavi: In order to get the probability that most people want, we need to apply Bayes' rules and invert the probability from the probability of the data given the hypothesis, to the probability of the hypothesis given the data.

Ronny Kohavi：为了得到大多数人想要的概率，我们需要应用贝叶斯规则，将概率从给定假设的数据概率反转为给定数据的假设概率。

For that, we need an additional number, which is the probability, the prior probability that the hypothesis that you're testing is successful or not.

为此，我们需要一个额外的数字，它是概率，即您正在测试的假设成功与否的先验概率。

Ronny Kohavi: That's an unknown.

Ronny Kohavi：这是一个未知数。

What we do is we can take historical data and say, "Look, people fail two thirds of the time or 80% of the time." And we can apply that number and compute that.

我们所做的是，我们可以利用历史数据并说：“看，人们有三分之二或 80% 的时间都失败了。”我们可以应用这个数字并计算它。

We've done that in a paper that I will give in the notes, so that you can assess the number that you really want, what's called a false positive risk.

我们在一篇论文中做到了这一点，我将在注释中给出，以便您可以评估您真正想要的数字，即所谓的误报风险。

Ronny Kohavi: So I think that's something for people to internalize, that what you really want to look at is this false positive risk, which tends to be much, much higher than the 5% that people think, right?

Ronny Kohavi：所以我认为这是人们应该内化的东西，你真正想要看到的是这种误报风险，它往往比人们想象的 5% 高得多，对吗？

So I think the classical example in the Airbnb where the failure rate was very, very high, is that when you get a statistically significant result, let me actually pull the note so that I know the actual number.

所以我认为 Airbnb 中失败率非常非常高的经典例子是，当你得到统计上显着的结果时，让我实际拉出笔记，以便我知道实际数字。

If you're at Airbnb, or Airbnb search where the success rate is only 8%, if you get a statistically significant result with a P value less than 0.05, there is a 26% chance that this is a false positive result.

如果你在 Airbnb，或者 Airbnb 搜索，成功率只有 8%，如果你得到一个 P 值小于 0.05 的具有统计显着性的结果，那么有 26% 的可能性这是一个假阳性结果。

It's not 5%, it's 26%.

这不是5%，而是26%。

Ronny Kohavi: So that's the number that you should have in your mind.

Ronny Kohavi：这就是你应该记住的数字。

And that's why when I worked at Airbnb, one of the things we did is we said, "Okay, if you're less than 0.05, but above 0.01, rerun, replicate." When you replicate, you can combine the two experiments, and get a combined P value using something called Fisher's method or Stouffer's method, and that gives you the joint probability.

这就是为什么当我在 Airbnb 工作时，我们所做的一件事就是说：“好吧，如果你的值低于 0.05，但高于 0.01，请重新运行、复制。”复制时，您可以结合两个实验，并使用称为 Fisher 方法或 Stouffer 方法的方法获得组合 P 值，从而得出联合概率。

And that's usually much, much lower.

而这个数字通常要低得多。

So if you get two 0.5's or something like that, then the probability that you've got them is much, much lower.

因此，如果你得到两个 0.5 或类似的分数，那么你得到它们的概率就会低得多。

Lenny: Wow, I've never heard it described that way.

莱尼：哇，我从来没有听过这样的描述。

It makes me think about how even data scientists in our teams are always just like, "This isn't perfect.

这让我想到，即使是我们团队中的数据科学家也总是会说，“这并不完美。

We're not 100% sure this experiment is positive." But on balance, if we're launching positive experiments, we're probably doing good things.

我们不能100%确定这个实验是积极的。”但总的来说，如果我们发起积极的实验，我们可能会做一些好事。

It's okay if sometimes we're wrong.

如果有时我们错了也没关系。

Ronny Kohavi: By the way, it's true.

Ronny Kohavi：顺便说一句，这是真的。

On balance, you're probably better than 50/50, but people don't appreciate how much that 26% that I mentioned is high.

总的来说，你可能比 50/50 更好，但人们并没有意识到我提到的 26% 有多高。

And the reason that I want to be sure is that I think it leads to this idea of the learning, the institutional knowledge.

我想确定的原因是，我认为它导致了学习、制度知识的想法。

What you want to be able to say is share with the org's success.

您想要表达的是与组织分享成功。

And so you want to be really sure that you're successful.

所以你想真正确定自己是否成功。

So by lowering the P value, by forcing teams to work with the P value maybe below 0.01 and do replication on higher, then you can be much more successful, and the false positive rate will be much, much lower.

因此，通过降低 P 值，通过迫使团队在 P 值可能低于 0.01 的情况下工作并在更高的值上进行复制，那么你可以取得更大的成功，并且误报率也会低得多。

Lenny: Fascinating.

莱尼：令人着迷。

And also shows the value of keeping track of what percentage your experiments are failing historically at the company or within that specific product.

它还显示了跟踪公司或特定产品历史上实验失败的百分比的价值。

Say someone listening wants to start running experiments, say they have tens of thousands of users at this point.

假设有人想要开始运行实验，假设他们目前有数万名用户。

What would be the first couple steps you'd recommend?

您建议的前几个步骤是什么？

Ronny Kohavi: Well, so if they have somebody in the org that has previously been involved with a experiment, that's a good way to consult internally.

Ronny Kohavi：嗯，所以如果他们组织中有人以前参与过实验，这是一个内部咨询的好方法。

I think the key decision is whether you want to build or buy.

我认为关键的决定是你想建造还是购买。

There's a whole series of eight sessions that I posted on LinkedIn where I invited guest speakers to talk about this problem.

我在 LinkedIn 上发布了一系列的八场会议，邀请嘉宾演讲者讨论这个问题。

So if people are interested, they can look at what the vendors say and what agency said about build versus buy question.

因此，如果人们感兴趣，他们可以查看供应商的说法以及机构对构建与购买问题的看法。

And it's usually not a zero one, it's usually both.

它通常不是零一，通常都是两者。

You build some and you buy some, and it's a question of do you build 10% or do you build in 90%?

你建造一些，你购买一些，问题是你建造 10% 还是建造 90%？

Ronny Kohavi: I think for people starting, the third party products that are available today are pretty good.

Ronny Kohavi：我认为对于刚开始使用的人来说，目前可用的第三方产品非常好。

This wasn't the case when I started working.

当我开始工作时，情况并非如此。

So when I started running experiments at Amazon, we were building the platform because nothing existed.

因此，当我开始在亚马逊进行实验时，我们正在构建这个平台，因为什么都不存在。

Same at Microsoft.

在微软也一样。

I think today, there's enough vendors that provide good experimentation platforms that are trustworthy, that I would say not a good way to consider using one of those.

我认为今天有足够多的供应商提供值得信赖的良好实验平台，我认为这不是考虑使用其中一个的好方法。

Lenny: Say you're at a company where there's resistance to experimentation and A/B testing, whether it's a startup or a bigger company.

Lenny：假设你所在的公司对实验和 A/B 测试存在阻力，无论是初创公司还是大公司。

What have you found works in helping shift that culture, and how long does that usually take, especially at a larger company?

您发现什么可以帮助转变这种文化？这通常需要多长时间，尤其是在一家较大的公司？

Ronny Kohavi: My general experience is with Microsoft, where we went with this beach head of Bing.

Ronny Kohavi：我的一般经验是在微软，我们与 Bing 的海滩负责人一起工作。

We were running a few experiments and then we were asked to focus on Bing, and we scaled experimentation and built a platform at scale at Bing.

我们进行了一些实验，然后我们被要求专注于 Bing，我们扩大了实验范围并在 Bing 上大规模构建了一个平台。

Ronny Kohavi: Once Bing was successful and we were able to share all these surprising results, I think that many, many more people in the company were amenable.

Ronny Kohavi：一旦 Bing 取得成功并且我们能够分享所有这些令人惊讶的结果，我认为公司中有很多很多人都愿意接受。

It was also the case that helped a lot that, there's a usual cross pollination.

通常的异花授粉也给了我很大帮助。

People from Bing move out to other groups, and that helped these other groups say, "Hey, there's a better way to build software."

Bing 的人员转移到其他团队，这有助于其他团队说：“嘿，有更好的方法来构建软件。”

Ronny Kohavi: So I think if you're starting out, find a place, find a team where experimentation is easy to run.

Ronny Kohavi：所以我认为如果你刚开始，找到一个地方，找到一个易于进行实验的团队。

And by that, I mean they're launching often, right?

我的意思是他们经常发布，对吧？

Don't go with the team that launches every six months, or Office used to launch every three years.

不要选择每六个月发布一次的团队，或者过去每三年发布一次的 Office。

Go with the team that launches frequently.

与经常发布的团队合作。

They're running on sprints, they launch every week or two.

他们正在进行冲刺，每隔一两周就会启动一次。

Sometimes they launch daily.

有时他们每天都会启动。

I mean, Bing used to launch multiple times a day.

我的意思是，Bing 曾经每天启动多次。

Ronny Kohavi: And then make sure that you understand the question of the OEC.

Ronny Kohavi：然后确保您理解 OEC 的问题。

Is it clear what they're optimizing for?

他们优化的目的是否明确？

There are some groups where you can come up with a good OEC.

在某些小组中，您可以提出良好的 OEC。

Some groups are harder.

有些团体更难。

Ronny Kohavi: I remember one funny example was the microsoft.com website, which this is not MSN, this is microsoft.com, has multiple different constituencies that are trying to determine this is a support site, and this is the ability to sell software through this site, and warn you about safety and updates.

Ronny Kohavi：我记得一个有趣的例子是 microsoft.com 网站，这不是 MSN，这是 microsoft.com，有多个不同的选区试图确定这是一个支持网站，这是通过该网站销售软件并警告您有关安全和更新的能力。

It has so many goals.

它有很多目标。

I remember when the team said, "We want to run experiments," and I brought the group in and some of the managers and I said, "Do you know what you're optimizing for?"

我记得当团队说“我们想要进行实验”时，我让团队和一些经理进来，我说：“你知道你要优化什么吗？”

Ronny Kohavi: It was very funny because they surprised me.

Ronny Kohavi：这很有趣，因为他们让我感到惊讶。

They said, "Hey Ronny, we read some of your papers.

他们说：“嘿罗尼，我们读了你的一些论文。

We know there's this term called OEC.

我们知道有一个叫做 OEC 的术语。

We decided the time on site is our OEC." And I said, "Wait a minute.

我们决定现场时间就是我们的 OEC。”我说：“等一下。

Some of your main goals is support site.

您的一些主要目标是支持网站。

Is people spending more time on the support site a good thing or a bad thing?" And then half the room thought that more time is better, and half the room thought that more time is worse.

人们在支持网站上花费更多时间是好事还是坏事？”然后，一半的人认为时间越长越好，一半的人认为时间越长越糟糕。

So an OEC is bad if directionally, you can't agree on it.

因此，如果在方向上你们无法达成一致，OEC 就是不好的。

Lenny: That's a great tip.

莱尼：这是一个很好的建议。

Along these same lines, I know you're a big fan of platforms and building a platform to run experiments, versus just one-off experiments.

沿着同样的思路，我知道您非常喜欢平台并构建一个平台来运行实验，而不是一次性实验。

Can you just talk briefly about that to give people a sense of where they probably should be going with their experimentation approach?

您能否简单地谈谈这一点，让人们了解他们的实验方法可能应该走向何方？

Ronny Kohavi: Yeah, so I think the motivation is to bring the marginal cost of experiments down to zero.

Ronny Kohavi：是的，所以我认为动机是将实验的边际成本降至零。

So the more you self-service, go to a website, set up your experiment, define your targets, define the metrics that you want, right?

因此，您的自助服务越多，访问网站，设置实验，定义您的目标，定义您想要的指标，对吗？

People don't appreciate that the number of metrics starts to grow really fast if you're doing things right.

人们不会意识到，如果你做得正确，指标的数量就会开始快速增长。

At Bing, you could define 10,000 metrics that you wanted to be in your scorecard.

在 Bing，您可以定义 10,000 个想要包含在记分卡中的指标。

Big numbers.

大数字。

Ronny Kohavi: So it was so big, and people said it's computationally inefficient.

Ronny Kohavi：它太大了，人们说它的计算效率低下。

We broke them into templates so that if you were launching a UI experiment, you would get this set of 2,000.

我们将它们分解为模板，这样如果您要启动 UI 实验，您将获得这组 2,000 个模板。

If you were doing a revenue experiment, you would get this set of 2,000.

如果您正在进行收入实验，您将得到这组 2,000。

Ronny Kohavi: So the point was build a platform that can quickly allow you to set up and run an experiment, and then analyze it.

Ronny Kohavi：所以重点是构建一个平台，可以让您快速设置和运行实验，然后对其进行分析。

I think one of the things that I will say at Airbnb is the analysis was relatively weak, and so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough.

我认为我要在 Airbnb 说的一件事是分析相对较弱，因此聘请了很多数据科学家来弥补该平台做得不够的事实。

Ronny Kohavi: And this happens in other organizations too, where there's this trade-off.

Ronny Kohavi：这种情况也发生在其他组织中，需要进行权衡。

If you're building a good platform, invest in it so that more and more automation will allow people to look at the analysis, without the need to involve a data scientist.

如果您正在构建一个好的平台，请对其进行投资，以便越来越多的自动化将允许人们查看分析，而无需数据科学家的参与。

Ronny Kohavi: We published a paper.

Ronny Kohavi：我们发表了一篇论文。

Again, I'll give it in the notes with this nice matrix of six axes, and how you move from crawl, to walk, to run, to fly, and what you need to build on those six axes.

再次，我将在笔记中给出这个漂亮的六个轴矩阵，以及如何从爬行移动到步行、跑步、飞行，以及需要在这六个轴上构建什么。

So one of the things that I do sometimes when I consult is I go into the org and say, "Where do you think you are on these six axes?" And that should be the guidance for what are the things you need to do next.

因此，当我咨询时，我有时会做的一件事就是进入组织并说：“你认为你在这六个轴上处于什么位置？”这应该是您下一步需要做的事情的指导。

Lenny: This is going to be the most epic show notes episode we've had yet.

莱尼：这将是我们迄今为止最史诗般的表演笔记剧集。

Maybe a last question.

也许是最后一个问题。

We talked about how important trust is to running experiments, and how even though people talk about speed, trust ends up being most important.

我们讨论了信任对于进行实验有多么重要，以及即使人们谈论速度，信任最终也是最重要的。

Still, I want to ask you about speed.

不过，我想问你关于速度的问题。

Is there anything you recommend for helping people run experiments faster and get results more quickly that they can implement?

您有什么建议可以帮助人们更快地进行实验并更快地获得结果吗？

Ronny Kohavi: Yeah, so I'll say a couple of things.

Ronny Kohavi：是的，所以我要说几件事。

One is if your platform is good, then when the experiment finishes, you should have a scorecard soon after.

一是如果你的平台很好，那么当实验完成后，你应该很快就会有一个记分卡。

Maybe takes a day, but it shouldn't be that you have to wait a week for the data scientist.

也许需要一天的时间，但您不应该等待一周才能等到数据科学家。

To me, this is the number one way to speed up things.

对我来说，这是加快速度的第一个方法。

Ronny Kohavi: Now, in terms of using the data efficiently, there are mechanisms out there under the title of variance reduction that help you reduce the variance of metrics so that you need less users, so that you can get results faster.

Ronny Kohavi：现在，在有效使用数据方面，有一些名为方差减少的机制可以帮助您减少指标的方差，以便您需要更少的用户，从而可以更快地获得结果。

Some examples that you might think about are capping metrics.

您可能会想到的一些示例是限制指标。

So if your revenue metric is very skewed, maybe you say, "Well, if somebody purchased over $1,000, let's make that $1,000." At Airbnb, one of the key metrics for example, is nights booked.

因此，如果您的收入指标非常偏差，您可能会说：“好吧，如果有人购买了超过 1,000 美元，我们就将其定为 1,000 美元。”例如，在 Airbnb，关键指标之一是预订天数。

Ronny Kohavi: Well, it turns out that some people book tens of nights.

Ronny Kohavi：嗯，事实证明有些人预订了数十晚。

They're like an agency or something, hundreds of nights.

他们就像一个机构什么的，数百个夜晚。

You may say, "Okay, let's just cap this.

你可能会说：“好吧，我们就到此为止吧。

It's unlikely that people book more than 30 days in a given month." So that various reduction technique will allow you to get statistically significant results faster.

人们在某个月的预订天数不太可能超过 30 天。”因此，各种缩减技术将使您能够更快地获得具有统计意义的结果。

Ronny Kohavi: And a third technique is called cupid, which is an article that we published.

Ronny Kohavi：第三种技术称为丘比特，这是我们发表的一篇文章。

Again, I can give it in the notes, which uses the pre-experiment data to adjust the result.

同样，我可以在笔记中给出，它使用实验前的数据来调整结果。

And we can show that you get the result as unbiased, but with lower variance, and hence, it requires fewer users.

我们可以证明，您得到的结果是无偏的，但方差较低，因此需要的用户较少。

Lenny: Ronny, is there anything else you want to share before we get to our very exciting lightning round?

莱尼：罗尼，在我们开始激动人心的闪电回合之前，你还有什么想分享的吗？

Ronny Kohavi: No, I think we've asked a lot of good questions.

Ronny Kohavi：不，我认为我们已经提出了很多很好的问题。

Hope people enjoy this.

希望人们喜欢这个。

Lenny: Lightning round.

莱尼：闪电般的圆。

Here we go.

开始了。

I'm just going to roll right into it.

我要直接投入其中。

What are two or three books that you've recommended most to other people?

您向其他人推荐最多的两三本书是什么？

Ronny Kohavi: There's a fun book called Calling Bullshit, which despite the name, which is a little extreme, I think, for a title, it actually has a lot of amazing insights that I love.

Ronny Kohavi：有一本有趣的书叫《Calling Bullshit》，尽管它的名字有点极端，但我认为，对于一个标题来说，它实际上有很多我喜欢的令人惊叹的见解。

And it sort of embodies, in my opinion, a lot of the Twyman's law showing that things that are too extreme, your bullshit meter should go up and say, "Hey, I don't believe that." So that's my number one recommendation.

在我看来，它有点体现了很多泰曼定律，表明如果事情太极端，你的废话计应该上升并说：“嘿，我不相信这一点。”这是我的第一建议。

Ronny Kohavi: There's a slightly older book that I love called Hard Facts, Dangerous Half-Truths And Total Nonsense by the Stanford professors from the Graduate School of Business.

Ronny Kohavi：我喜欢一本稍微老一点的书，叫做《硬事实、危险的半真半假和完全胡说八道》，作者是斯坦福大学商学院的教授。

Very interesting to see many of the things that we grew up with as well understood turn out to have no justification.

很有趣的是，我们成长过程中所理解的许多事情最终证明是没有道理的。

Ronny Kohavi: So a stranger book, which I love, sort of on the verge of psychology, it's called Mistakes Were Made (But Not by Me), about all the fallacies that we fall into, and the humbling results from that.

罗尼·科哈维：所以我喜欢一本陌生的书，有点接近心理学，它的名字叫《错误是由我犯的（但不是我犯的）》，讲述了我们陷入的所有谬误，以及由此产生的令人羞愧的结果。

Lenny: The titles of these are hilarious, and there's a common theme across all these books.

莱尼：这些书的标题很搞笑，而且所有这些书都有一个共同的主题。

Next question, what is a favorite recent movie or TV show?

下一个问题，最近最喜欢的电影或电视节目是什么？

Ronny Kohavi: So I recently saw a short series called Chernobyl, the disaster.

罗尼·科哈维：我最近看了一部名为《切尔诺贝利灾难》的短剧。

I thought it was amazingly well done.

我认为这做得非常好。

Highly recommended it, based on true events.

强烈推荐，根据真实事件改编。

As usual, there's some freedom for the artistic movie.

和往常一样，艺术电影有一定的自由度。

It was kind of interesting at the end, they say, "This woman in the movie wasn't really a woman.

最后有点有趣，他们说，“电影中的这个女人并不是真正的女人。

It was a bunch of 30 data scientists." Not data scientists, 30 scientists that in real life, presented all the data to the leadership of what to do.

这是一群 30 名数据科学家。”不是数据科学家，而是现实生活中的 30 位科学家，向领导呈现了所有的数据来决定做什么。

Lenny: I remember that.

莱尼：我记得。

Fun fact, I was born in Odessa, Ukraine, which was not so far from Chernobyl.

有趣的是，我出生在乌克兰敖德萨，距离切尔诺贝利不远。

And I remember my dad told me he had to go to work.

我记得我爸爸告诉我他必须去上班。

They called him into work that day to clean some stuff off the trees.

那天他们叫他去上班，清理树上的一些东西。

I think ash from the explosion or something.

我认为是爆炸留下的灰烬之类的。

It was far away where I don't think we were exposed, but we were in the vicinity.

距离很远，我认为我们不会暴露，但我们就在附近。

That's pretty scary.

这真是太可怕了。

My wife, every time something's wrong with me, she's like, "That must be a Chernobyl thing." Okay, next question.

我的妻子，每次我出了问题，她都会说，“那一定是切尔诺贝利的事情。”好的，下一个问题。

Favorite interview question you like to ask people when you're interviewing them?

当你面试别人时，你最喜欢问他们的面试问题是什么？

Ronny Kohavi: So it depends on the interview, but when I do a technical interview, which I do less of, but one question that I love that it's amazing how many people it throws away for languages like C++, is tell me what the static qualifier does.

Ronny Kohavi：所以这取决于面试，但是当我进行技术面试时，我很少做这种面试，但我喜欢一个问题，就是告诉我静态限定符是做什么的，令人惊讶的是，它为 C++ 这样的语言抛弃了多少人。

And for multiple, you can do it for a variable, you can do it for function.

对于多个，您可以为变量执行此操作，也可以为函数执行此操作。

And it is just amazing that I would say more than 50% of people that interview for engineering job cannot get this, and get it awfully wrong.

令人惊讶的是，我想说超过 50% 的工程职位面试者无法理解这一点，而且会犯严重错误。

Lenny: Definitely the most technical answer to this question yet.

Lenny：绝对是这个问题迄今为止最具技术性的答案。

Lenny: Okay.

莱尼：好的。

What's a favorite recent product you've discovered that you love?

您最近发现的最喜欢的产品是什么？

Ronny Kohavi: Blink cameras.

Ronny Kohavi：眨眼相机。

So a Blink camera is this small camera.

所以 Blink 相机就是这种小型相机。

You stick in two AA batteries, and it lasts for about six months.

你装上两节 AA 电池，它可以持续大约六个月。

They claim up to two years.

他们声称长达两年。

My experience is usually about six months.

我的经验通常是六个月左右。

But it was just amazing to me how you can throw these things around in the yard and see things that you would never know otherwise.

但令我惊奇的是，你可以把这些东西扔在院子里，然后看到一些你永远不会知道的东西。

Some animals that go by.

一些经过的动物。

We had a skunk that we couldn't figure out how he was entering, so I threw five cameras out and I saw where he came in.

我们有一只臭鼬，我们无法弄清楚他是如何进入的，所以我扔出了五个摄像机，我看到了他进来的地方。

Ronny Kohavi: He came in under a hole on the fence that was about this high.

Ronny Kohavi：他从栅栏上的一个洞下面进来，大约有这么高。

I have a video of this thing just squishing underneath.

我有一段视频，里面有这个东西被压在下面的情况。

We never would've assumed that it came from there, from the neighbor.

我们永远不会认为它来自那里，来自邻居。

But yeah, these things have just changed.

但是，是的，这些事情刚刚发生了变化。

And when you're away on a trip, it's always nice to be able to say, "I can see my house.

当您外出旅行时，能够说：“我可以看到我的房子。

Everything's okay." At one point, we had a false alarm, and the cops came in and had this amazing video of how they're entering the house and pulling the guns out.

一切都好。”有一次，我们误报了，警察进来并拍摄了一段令人惊叹的视频，展示了他们如何进入房子并拔出枪。

Lenny: You got to share that on TikTok.

莱尼：你必须在 TikTok 上分享这一点。

That's good content.

这是好内容啊

Wow.

哇。

Okay.

好的。

Blink cameras.

眨眼相机。

We'll set those up in my house asap.

我们会尽快把它们安置在我家里。

Lenny: What is something relatively minor you've changed in the way your teams develop product, that has had a big impact on their ability to execute?

Lenny：您在团队开发产品的方式上做了哪些相对较小的改变，但对他们的执行能力产生了很大的影响？

Ronny Kohavi: I think this is something that I learned at Amazon, which is a structured narrative.

Ronny Kohavi：我认为这是我在亚马逊学到的东西，这是一种结构化的叙述。

So Amazon has some variance of this, which sometimes go by the name of a six pager or something.

所以亚马逊对此有一些变化，有时被称为六寻呼机之类的名称。

But when I was at Amazon, I still remember that email from Jeff, which is, "No more PowerPoint.

但当我在亚马逊时，我仍然记得杰夫发来的那封电子邮件，“不再有 PowerPoint。

I'm going to force you to write a narrative."

我要强迫你写一篇叙述。”

Ronny Kohavi: I took that to heart.

Ronny Kohavi：我牢记这一点。

And many of the features that the team presented instead of a PowerPoint, you start off with a structured document that tells you what you need, the questions you need to answer for your idea.

该团队提供的许多功能不是 PowerPoint，而是从一份结构化文档开始，它会告诉您需要什么，以及为实现您的想法需要回答的问题。

And then we review them as a team.

然后我们作为一个团队对他们进行审查。

Ronny Kohavi: And Amazon, these were paper-based.

Ronny Kohavi：亚马逊，这些都是纸质的。

Now it's all based on Word or Google Docs where people comment, and I think the impact of that was amazing.

现在，一切都基于人们评论的 Word 或 Google 文档，我认为其影响是惊人的。

I think the ability to give people honest feedback and have them appreciate, and have it stay after the meeting in these notes on the document, just amazing.

我认为能够向人们提供诚实的反馈并让他们欣赏，并在会议结束后将其保留在文档的这些注释中，真是太棒了。

Lenny: Final question, have you ever run an A/B test on your life, either your dating life, your family, your kids?

莱尼：最后一个问题，你有没有对你的生活进行过 A/B 测试，无论是你的约会生活、你的家庭还是你的孩子？

And if so, what did you try?

如果是这样，你尝试了什么？

Ronny Kohavi: So there aren't enough units.

Ronny Kohavi：所以没有足够的单位。

Remember I said you need 10,000 of something to run true A/B tests?

还记得我说过你需要 10,000 个东西才能运行真正的 A/B 测试吗？

I will say a couple of things.

我会说几件事。

One is I try to emphasize to my family, and friends, and everybody, this idea called the hierarchy of evidence.

一是我试图向我的家人、朋友和每个人强调这个称为证据层次的想法。

When you read something, there's a hierarchy of trust levels.

当您阅读某些内容时，会存在信任级别的层次结构。

If something is anecdotal, don't trust it.

如果某件事是轶事，请不要相信它。

If there was an experiment, it was observational.

如果有实验，那也是观察性的。

Give it some bit of trust.

给它一点信任。

As you get more up and up to a natural experiment, and controlled experiments, and multiple controlled experiments, your trust levels should go up.

当你越来越多地进行自然实验、对照实验和多次对照实验时，你的信任水平应该会上升。

So I think that that's a very important thing that a lot of people miss when they see something in the news is, where does it come from?

所以我认为这是一个非常重要的事情，很多人在新闻中看到一些东西时都忽略了，它来自哪里？

Ronny Kohavi: I have a talk that I've shared of all these observational studies that people made that were published.

Ronny Kohavi：我有一个演讲，分享了人们所做的所有已发表的观察性研究。

And then somehow, a control experiment was run later on and proved that it was directionally incorrect.

然后不知何故，后来进行了一个对照实验，证明它的方向是错误的。

So I think there's a lot to learn about this idea of the hierarchy of evidence, and share it with our family, and kids, and friends.

所以我认为关于证据层次的想法有很多东西需要学习，并与我们的家人、孩子和朋友分享。

I think there's a book that's based on this.

我认为有一本书是以此为基础的。

It's like How to Read a Book.

这就像如何阅读一本书。

Lenny: Well, Ronny, the experiment of us recording a podcast I think is 100% positive P value 0.0.

Lenny：嗯，Ronny，我们录制播客的实验我认为是 100% 正 P 值 0.0。

Thank you so much for being here.

非常感谢您来到这里。

Ronny Kohavi: Thank you so much for inviting me and for great questions.

Ronny Kohavi：非常感谢您邀请我并提出很好的问题。

Lenny: Amazing.

莱尼：太棒了。

I appreciate that.

我很欣赏这一点。

Two final questions.

最后两个问题。

Where can folks find you online if they want to reach out, and is there anything that listeners can do for you?

如果人们想要联系您，可以在哪里找到您？听众可以为您做些什么吗？

Ronny Kohavi: Finding me online is easy.

Ronny Kohavi：在网上找到我很容易。

It's LinkedIn.

这是领英。

And what can people do for me?

人们能为我做什么？

Understand the idea of control experiments as a mechanism to make the right data-driven decisions.

了解控制实验的概念，将其作为做出正确的数据驱动决策的机制。

Use science.

使用科学。

Learn more by reading my book if you want.

如果您愿意，可以通过阅读我的书了解更多信息。

Again, all proceeds go to charity.

再次强调，所有收益均捐献给慈善机构。

And if you want to learn more, there's a class that I teach every quarter on Maven.

如果您想了解更多信息，我每个季度都会开设一门有关 Maven 的课程。

We'll put in the notes how to find it, and some discount for people who managed to stay all the way to the end of this podcast.

我们将在注释中注明如何找到它，并为那些设法坚持到本播客结束的人提供一些折扣。

Lenny: Yeah, that's awesome.

莱尼：是的，那太棒了。

We'll include that at the top so people don't miss it, so there's going to be a code to get a discount on your course.

我们会将其放在顶部，这样人们就不会错过它，因此将有一个代码可以在您的课程上获得折扣。

Ronny, thank you again so much for being here.

罗尼，再次非常感谢你来到这里。

This was amazing.

这太棒了。

Lenny: Bye everyone.

莱尼：大家再见。

Thank you so much for listening.

非常感谢您的聆听。

If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

如果您发现这很有价值，您可以在 Apple Podcasts、Spotify 或您最喜欢的播客应用程序上订阅该节目。

Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast.

另外，请考虑给我们评分或留下评论，因为这确实有助于其他听众找到播客。

You can find all past episodes or learn more about the show at lennyspodcast.com.

您可以在 lennyspodcast.com 上找到所有过去的剧集或了解有关该节目的更多信息。

See you in the next episode.

下一集见。