Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor)

Ramesh Johari: Marketplaces are a little bit like a game of whac-a-mole.

Ramesh Johari：市场有点像打地鼠游戏。

One example that I came across with one of the companies I worked with that I love is our new supply side was having a pretty bad experience.

我在与我合作的一家我喜欢的公司中遇到的一个例子是，我们的新供应方经历了非常糟糕的经历。

Ramesh Johari: So what we decided to do is build some custom bespoke features that were really going to direct them to more experienced folks on the other side of the market.

Ramesh Johari：所以我们决定做的是构建一些定制功能，这些功能实际上将它们引导给市场另一边更有经验的人。

Good.

好的。

And then, yeah, lo and behold, pretty soon those metrics start to look better.

然后，是的，你瞧，很快这些指标就开始看起来更好了。

But then we're looking at it, we're like, "Wait a second.

但当我们看到它时，我们会想，“等一下。

Now the existing folks on the other side are having a worse experience," so you kind of whiplash around.

现在，另一边现有的人正在经历更糟糕的经历，”所以你有点挥霍。

You're like, "Oh, wait a second.

你会说，“哦，等一下。

We better do something about that." So we take them, we try to match them up with the more experienced folks, and now suddenly a month after that, you're like, "Wait a second," and your metrics just keep moving around.

我们最好对此做点什么。”所以我们接受他们，我们尝试将他们与更有经验的人相匹配，现在突然一个月后，你会说，“等一下，”你的指标一直在变化。

And that's because the whac-a-mole game here is ultimately, a lot of marketplace management is moving attention and inventory around.

这是因为这里的打地鼠游戏最终是，许多市场管理人员正在转移注意力和库存。

Many of the changes that are most consequential create winners and losers.

许多最重要的变化会产生赢家和输家。

And rolling with those changes is about recognizing whether the winners you've created are more important to your business than the losers you've created in the process.

伴随这些变化意味着要认识到你所创造的赢家对你的业务来说是否比你在此过程中创造的输家更重要。

Lenny: Today my guest is Ramesh Johari.

莱尼：今天我的客人是拉梅什·乔哈里。

Ramesh is a professor at Stanford University, where he does research on and teaches data science methods and practices with a specific focus on the design and operation of online marketplaces.

Ramesh 是斯坦福大学的教授，他在那里研究并教授数据科学方法和实践，特别关注在线市场的设计和运营。

He's advised and worked with some of the biggest marketplaces in the world, including Airbnb, Uber, Stripe, Bumble, Stitch Fix, Upwork, and many others.

他为世界上一些最大的市场提供咨询并与之合作，包括 Airbnb、Uber、Stripe、Bumble、Stitch Fix、Upwork 等。

And in our conversation, we get super nerdy on how to build a thriving marketplace, including where to focus your resources to fuel the marketplace flywheel growth, why data and data science is so central to building a successful marketplace, how to design a better review system.

在我们的谈话中，我们对如何建立一个繁荣的市场非常感兴趣，包括在哪里集中资源来推动市场飞轮的增长，为什么数据和数据科学对于建立一个成功的市场如此重要，如何设计一个更好的评论系统。

Why as a founder, you shouldn't think of yourself as a marketplace founder, but instead simply as a founder.

为什么作为创始人，您不应该将自己视为市场创始人，而应简单地视为创始人。

Also, how AI is going to impact data science and marketplaces, and experimentation, and so much more.

此外，人工智能将如何影响数据科学和市场、实验等等。

If you're building a marketplace business, or thinking about building a marketplace, or just curious, this episode is for you.

如果您正在建立一个市场业务，或者正在考虑建立一个市场，或者只是好奇，那么本集适合您。

With that, I bring you Ramesh Johari after a short word from our sponsors.

在我们的赞助商简短发言后，我为您带来拉梅什·乔哈里 (Ramesh Johari)。

Lenny: This episode is brought to you by Sanity.

莱尼：这一集是由理智给你带来的。

Your website is the heart of your growth engine.

您的网站是您增长引擎的核心。

For that engine to drive big results, you need to be able to move super fast, ship new content, experiment, learn and iterate.

为了使该引擎能够带来巨大的成果，您需要能够超快地行动、发布新内容、进行实验、学习和迭代。

But most content management systems just aren't built for this.

但大多数内容管理系统并不是为此而构建的。

Your content teams wrestle with rigid interfaces as they build new pages.

您的内容团队在构建新页面时会与严格的界面作斗争。

You spend endless time copying and pasting across pages and recreating content for other channels and applications.

您花费无尽的时间跨页面复制和粘贴以及为其他渠道和应用程序重新创建内容。

And their ideas for new experiments are squashed when developers can't build them within the constraints of outdated tech.

当开发人员无法在过时技术的限制下构建新实验的想法时，他们的新实验想法就会被压制。

Lenny: Forward-thinking companies like Figma, Amplitude, Loom, Riot Games, Linear, and more use Sanity to build content growth engines that scale drive innovation and accelerate customer acquisition.

Lenny：Figma、Amplitude、Loom、Riot Games、Linear 等具有前瞻性思维的公司使用 Sanity 来构建内容增长引擎，以规模化推动创新并加速客户获取。

With Sanity, your team can dream bigger and move faster.

有了 Sanity，您的团队就能梦想更大、行动更快。

As the most powerful headless CMS on the market, you can tailor editorial workflows to match your business, reuse content seamlessly across any page or channel, and bring your ideas to market without developer friction.

作为市场上最强大的无头 CMS，您可以定制编辑工作流程以匹配您的业务，跨任何页面或渠道无缝地重用内容，并将您的想法推向市场，而无需开发人员遇到麻烦。

Sanity makes life better for your whole team.

理智让整个团队的生活变得更好。

It's fast for developers to build with, intuitive for content managers, and it integrates seamlessly with the rest of your tech stack.

它对于开发人员来说构建速度很快，对于内容管理者来说很直观，并且它与您的技术堆栈的其余部分无缝集成。

Get started with Sanity's generous free plan.

开始使用 Sanity 慷慨的免费计划。

And as a Lenny's Podcast listener, you can get a boosted plan with double the monthly usage.

作为 Lenny's Podcast 听众，您可以获得每月使用量双倍的增强计划。

Head over to sanity.io/lenny to get started for free.

前往 sanity.io/lenny 免费开始。

That's sanity.io/lenny.

那是 sanity.io/lenny。

Lenny: This episode is brought to you by Hex.

莱尼：这一集是由 Hex 为您带来的。

If you're a data person, you probably have to jump between different tools to run queries, build visualizations, write Python, and send around a lot of screenshots and CSV files.

如果您是一名数据人员，您可能必须在不同的工具之间切换来运行查询、构建可视化、编写 Python 以及发送大量屏幕截图和 CSV 文件。

Hex brings everything together.

十六进制将一切结合在一起。

Its powerful Notebook UI lets you analyze data in SQL, Python, or no code in any combination, and work together with live multiplayer and version control.

其强大的 Notebook UI 允许您使用 SQL、Python 或任何组合的任何代码分析数据，并与实时多人游戏和版本控制一起使用。

And now Hex's, AI tools can generate queries and code, create visualizations, and even kickstart a whole analysis for you all from natural language prompts.

现在，Hex 的人工智能工具可以生成查询和代码、创建可视化，甚至可以根据自然语言提示为您启动整个分析。

It's like having an analytics co-pilot built right into where you're already doing your work.

这就像在您已经开展工作的地方内置了一位分析副驾驶。

Then when you're ready to share, you can use Hex's drag and drop app builder to configure beautiful reports or dashboards that anyone can use.

然后，当您准备好共享时，您可以使用 Hex 的拖放应用程序生成器来配置任何人都可以使用的精美报告或仪表板。

Join the hundreds of data teams like Notion, AllTrails, Loom, Mixpanel, and Algolia using Hex every day to make their work more impactful.

加入数百个数据团队，如 Notion、AllTrails、Loom、Mixpanel 和 Algolia，每天使用 Hex，让他们的工作更具影响力。

Sign up today at hex.tech/lenny to get a 60 day free trial of the Hex team plan.

立即在 hex.tech/lenny 注册，即可免费试用 Hex 团队计划 60 天。

That's hex.tech/lenny.

那是 hex.tech/lenny。

Lenny: Ramesh, thank you so much for being here.

莱尼：拉梅什，非常感谢你来到这里。

Welcome to the podcast.

欢迎来到播客。

Ramesh Johari: Thanks so much for having me, Lenny.

拉梅什·乔哈里：非常感谢你邀请我，莱尼。

It's great to be on.

能参加真是太好了。

Lenny: It's great to have you on.

莱尼：很高兴有你在。

A big thank you to Riley Newman for connecting us.

非常感谢莱利·纽曼为我们提供联系。

Riley was the first data scientist at Airbnb and head of data science at Airbnb, and that role is actually a really good microcosm of what we're going to be focusing on in our chat today.

Riley 是 Airbnb 的第一位数据科学家，也是 Airbnb 的数据科学主管，这个角色实际上是我们今天要讨论的内容的一个很好的缩影。

We're going to get super nerdy about marketplaces, and experimentation, and data.

我们将对市场、实验和数据变得超级书呆子般。

I know that's your jam.

我知道那是你的果酱。

Are you ready to dive in?

你准备好潜入了吗？

Ramesh Johari: I really am.

拉梅什·乔哈里：我确实是这样。

Yeah.

是的。

And I actually want to thank Riley too.

事实上我也想感谢莱利。

I got to know Riley when I was at oDesk first as a research scientist, and then I directed their data science team.

我最初在 oDesk 担任研究科学家时认识了 Riley，然后我指导了他们的数据科学团队。

This is way back in 2012, and I was looking around for people who are experts on how we think about data and marketplaces, and Riley Newman came up and so I invited him to come talk to us at oDesk, and we've stayed in touch since then.

早在 2012 年，我就在四处寻找关于如何看待数据和市场的专家，Riley Newman 出现了，所以我邀请他来 oDesk 与我们交谈，从那时起我们就一直保持联系。

Ramesh Johari: Those were early days of where this industry was, and I've had a kind of lengthy career now thinking about those kinds of problems.

Ramesh Johari：那是这个行业的早期阶段，我有一段漫长的职业生涯，现在正在思考这些问题。

So I'm pretty excited to talk about it with you.

所以我很高兴能和你谈论这个问题。

Lenny: Let's start broad and set a little foundation.

莱尼：让我们从广泛的角度开始，打下一些基础。

You have a really interesting way to describe what a marketplace business even is.

您用一种非常有趣的方式来描述市场业务是什么。

So Ramesh, what is a marketplace business, and also why is data so important and such an integral part of building a successful marketplace business?

那么 Ramesh，什么是市场业务？为什么数据如此重要，并且是建立成功的市场业务不可或缺的一部分？

Ramesh Johari: It's interesting when people sit down and think about, say Airbnb, what does Airbnb sell?

Ramesh Johari：有趣的是，当人们坐下来思考 Airbnb 时，Airbnb 卖什么？

Average person is like, "That's pretty obvious.

一般人会说：“这很明显。

Airbnb sells rooms.

Airbnb 出售房间。

I go there to book a room I want to stay at," right?

我去那里预订一个我想住的房间，”对吧？

Other people say, "What does Uber sell?

其他人说：“Uber 卖什么？

Uber sells me rides.

优步向我出售乘车服务。

I'd use Uber when I need to get a ride from somewhere to somewhere else."

当我需要从某个地方搭车到另一个地方时，我会使用 Uber。”

Ramesh Johari: And in some sense, you're not wrong.

拉梅什·乔哈里：从某种意义上说，你没有错。

I mean, you go there.

我的意思是，你去那里。

That's a platform to get these things.

这是一个获得这些东西的平台。

But that's not what the platform is selling.

但这不是该平台所销售的内容。

That's a really important distinction.

这是一个非常重要的区别。

There are people on the platform that are selling that to you.

平台上有人正在向您出售该产品。

The hosts on Airbnb are selling you listings.

Airbnb 上的房东正在向您出售房源。

The drivers on Uber are selling you rides.

优步的司机正在向您推销行程。

But Uber and Airbnb are selling you the taking away of something, which is a weird thing to think about.

但 Uber 和 Airbnb 却向你兜售“带走”的东西，这想想就很奇怪。

What they're taking away is the friction of finding a place to stay.

他们消除的是寻找住宿地点的麻烦。

They're taking away the friction of finding a driver.

他们正在消除寻找司机的麻烦。

Ramesh Johari: In economics, we call those things transaction costs.

Ramesh Johari：在经济学中，我们称之为交易成本。

When you take econ 1, you learn about markets and how supply meets demand, and we get prices out of that.

当你学习经济 1 时，你会了解市场以及供应如何满足需求，然后我们从中得出价格。

But what you don't learn until econ 201 is that markets don't always work.

但直到《econ 201》你才知道市场并不总是有效。

And one of the reasons markets don't always work is because we have what are called market failures due to the presence of these kinds of friction.

市场并不总是有效的原因之一是，由于这些摩擦的存在，我们出现了所谓的市场失灵。

So what's a market failure?

那么什么是市场失灵呢？

It's that Lenny wants to get from Palo Alto to Burlingame and he can't do it.

莱尼想从帕洛阿尔托去伯林格姆，但他做不到。

Why can't he do it?

为什么他做不到？

He doesn't have anyone to drive him.

没有人可以开车送他。

Well, why doesn't he just call someone to drive him?

那么，他为什么不叫人来开车送他呢？

Well, who's he supposed to call?

那么，他应该给谁打电话呢？

Who are those people?

那些人是谁？

Are they out there?

他们在外面吗？

Are they willing to drive him right now, right at 10:00 AM on a Friday?

他们愿意在周五上午 10:00 开车送他吗？

Are they willing to take him somewhere?

他们愿意带他去某个地方吗？

Ramesh Johari: When I want to stay somewhere when I'm traveling, a friction is, who's willing to give me their room?

Ramesh Johari：当我旅行时想住在某个地方时，一个摩擦是，谁愿意给我他们的房间？

I mean on principle, there's people who are willing to let me stay in their living room, but I don't know who they are.

我的意思是原则上有人愿意让我住在他们的客厅里，但我不知道他们是谁。

Ramesh Johari: So those are frictions, and what the marketplaces are selling you is taking the friction away.

Ramesh Johari：所以这些都是摩擦，而市场向你推销的东西就是消除摩擦。

That's what you're paying them for.

这就是你付钱给他们的原因。

And it's an important observation, because what that means is the marketplace's customers aren't just the people buying the rides, they're buying the listings.

这是一个重要的观察结果，因为这意味着市场的客户不仅仅是购买游乐设施的人，他们还购买列表。

Actually, the hosts are Airbnb's customers, and the drivers are also Uber's customers.

事实上，房东是Airbnb的客户，司机也是Uber的客户。

So both sides of the marketplace are the customers of the platform.

所以市场的双方都是平台的客户。

Both sides depend on the platform to help the platform take that friction away.

双方都依赖平台来帮助平台消除摩擦。

Because just like you want a place to stay or you want to ride, the driver is at Uber because he wants to earn money by taking people places.

因为就像你想要一个住宿的地方或者你想要乘车一样，司机选择 Uber 是因为他想通过带人去地方来赚钱。

And the host is on Airbnb because they want to earn money by selling their listing.

房东选择 Airbnb 是因为他们想通过出售房源来赚钱。

Ramesh Johari: I think this concept that we're making money by taking transaction costs away is such a fundamental idea that's misunderstood around marketplaces.

Ramesh Johari：我认为我们通过消除交易成本来赚钱的概念是一个基本概念，但在市场上却被误解了。

That when you're an entrepreneur starting a marketplace or thinking about your business model, I think you can be wildly off if you forget that that's the thing that's fundamentally your value proposition.

当你是一名创业者，创建一个市场或思考你的商业模式时，我认为如果你忘记了这才是你的价值主张的根本，你可能会陷入困境。

And then you asked about the role of data, and more broadly data science in marketplaces.

然后您询问了数据的作用，以及更广泛的数据科学在市场中的作用。

Ramesh Johari: So it's an interesting thing, right?

Ramesh Johari：所以这是一件有趣的事情，对吧？

The example I always love to give are the ancient Agoras in Greece or Trajan's Market in Rome.

我总是喜欢举的例子是希腊的古代集市或罗马的图拉真市场。

When you look at pictures of these things, what really stands out to me is the rock.

当你看这些东西的照片时，对我来说真正突出的是岩石。

I mean, these things are made of stone.

我的意思是，这些东西都是石头做的。

It's not like you were going to move a booth from one place to another place without moving a lot of rock from one place to another place.

这并不像您要将一个摊位从一个地方移动到另一个地方而不将大量岩石从一个地方移动到另一个地方。

Ramesh Johari: So you flash forward to 2023, and here we are with technology undergirding pretty much every kind of commerce now.

Ramesh Johari：那么你快进到 2023 年，现在我们的技术几乎支撑着所有类型的商业。

And it means we can architect and re-architect the marketplace kind of on the fly, and we really are doing it all the time.

这意味着我们可以动态地构建和重新构建市场，而且我们确实一直在这样做。

Ramesh Johari: And these frictions that are getting taken away, they're getting taken away because of data and data science.

Ramesh Johari：这些摩擦正在被消除，因为数据和数据科学，它们被消除了。

So I really want to highlight three pieces of this for people, which I want you to think of them as a cycle.

所以我真的想向人们强调其中的三部分，我希望你们将它们视为一个循环。

But to start with, let's just lay them out one at a time.

但首先，我们一次只列出一个。

Ramesh Johari: One of them is finding people to match with.

Ramesh Johari：其中之一就是寻找匹配的人。

So that's the problem of, "I want to stay somewhere.

这就是问题所在，“我想留在某个地方。

Who is out there, who's willing to let me stay with them on a given timeframe?" And then if I'm a host, I have a listing.

谁在那里，谁愿意让我在规定的时间内和他们呆在一起？”如果我是房东，我就会有一个列表。

Who is out there, who's willing to stay at my place when I have it available?

当我有空位时，谁愿意留在我的地方？

So that's finding matches.

这就是寻找匹配。

Ramesh Johari: Then there's making the match.

Ramesh Johari：然后就是匹配。

And so here, going back to my time at oDesk, a big problem that we dealt with there was if I've got multiple applicants to my job, who should I hire?

回到我在 oDesk 的时光，我们处理的一个大问题是，如果我的工作有多个申请人，我应该雇用谁？

Who should I interview?

我应该采访谁？

It's a common problem we face in the real world, but now it's all remote.

这是我们在现实世界中面临的常见问题，但现在一切都变得遥远了。

I don't meet these people in person.

我没有亲自见过这些人。

All I've got is this application they submitted to me.

我所拥有的只是他们提交给我的这份申请。

I need help triaging that.

我需要帮助进行分类。

Okay?

好的？

So that's helping make a match out of possible partners you can match with.

因此，这有助于从您可以匹配的可能合作伙伴中进行匹配。

Ramesh Johari: And then finally, we make matches.

Ramesh Johari：最后，我们进行匹配。

Well, what do the matches tell us, right?

好吧，比赛告诉我们什么，对吧？

I mean, if you stay somewhere in Airbnb, you learn something about the host, you learn something about the listing.

我的意思是，如果您住在 Airbnb 的某个地方，您就会了解有关房东的一些信息，您会了解有关房源的一些信息。

The host learns about you too.

楼主也了解你了。

And that's all information that the marketplace should feed back in.

这就是市场应该反馈的所有信息。

So this is where we get to rating systems and feedback systems, even passive data collection, right?

这就是我们使用评级系统和反馈系统，甚至被动数据收集的地方，对吗？

Did you leave your booking before you were supposed to leave?

您是否在出发前离开了预订？

Well, maybe that's a sign that something didn't quite work out the way you wanted to work out.

好吧，也许这表明有些事情没有完全按照你想要的方式进行。

So that's passive data collection.

这就是被动数据收集。

Did you leave five stars?

你留下五颗星了吗？

That's active data collection.

这就是主动数据收集。

Ramesh Johari: Get all this back in, and what does that do?

Ramesh Johari：把所有这些都放回去，那有什么作用？

Well, that lets us do a better job finding potential matches and make potential matches in the future.

嗯，这让我们可以更好地找到潜在的匹配对象并在未来进行潜在的匹配。

Every single thing I just said, finding potential matches, making matches, and then learning about those matches, and then cycling back again, that is the data science in marketplaces.

我刚才所说的每一件事，寻找潜在的匹配，进行匹配，然后了解这些匹配，然后再次循环回来，这就是市场中的数据科学。

Ramesh Johari: And I feel like every marketplace that you could think of in any vertical has those three problems to deal with and relies on algorithms in data science to help them solve it.

Ramesh Johari：我觉得你能想到的任何垂直领域的每个市场都需要解决这三个问题，并依赖数据科学中的算法来帮助他们解决这些问题。

And in turn, that is I think really the underpinning of taking those frictions away.

反过来，我认为这才是消除这些摩擦的真正基础。

Lenny: Many founders try to start a marketplace business, think about marketplace opportunities where they don't exist.

莱尼：许多创始人尝试开展市场业务，思考不存在的市场机会。

And there's often these recurring failures of types of marketplaces that just don't work in an area.

而且经常会出现一些在某个地区不起作用的市场类型的反复失败。

I was just writing a couple ideas down while you were chatting like cleaners, getting cleaners as a marketplace doesn't seem to work ever.

当你们像清洁工一样聊天时，我只是写下了一些想法，因为市场似乎从来没有发挥过作用。

Car wash, there's a classic failure too.

洗车，也有一个经典的失败。

Getting tasks done for you on demand as a marketplace seems to not often work.

在市场上按需为您完成任务似乎并不常见。

Lenny: So this might be too big of a question, but I'm just curious if anything comes up of when someone is starting a marketplace or thinking about starting a marketplace business, what do you find are the most common flaws in, this is probably not going to work as a marketplace?

莱尼：所以这可能是一个太大的问题，但我只是很好奇，当有人创建一个市场或考虑创建一个市场业务时，是否会出现任何问题，您发现最常见的缺陷是什么，这可能无法作为一个市场运作？

Ramesh Johari: That is such a fantastic question, and I want to preface what I say with a couple of comments.

拉梅什·乔哈里（Ramesh Johari）：这是一个非常棒的问题，我想用一些评论作为我所说的话的序言。

So one of them is that I've worked with a lot of different marketplace companies, but anything I say is pertaining to something more sensitive.

其中之一是，我曾与许多不同的市场公司合作过，但我所说的任何内容都与更敏感的事情有关。

I may not name the company over the course of the podcast.

我可能不会在播客过程中提及该公司的名称。

Ramesh Johari: But the other more important thing I want to say is that I'm a professor at Stanford, and there's a reason I'm not a successful scaled entrepreneur of marketplaces, and that's because I probably haven't unlocked the key to exactly the question you asked.

拉梅什·乔哈里：但我想说的另一件更重要的事情是，我是斯坦福大学的一名教授，我不是一个成功的规模化市场企业家是有原因的，那是因为我可能还没有完全解开你问的问题的钥匙。

But nevertheless, I have some thoughts on it.

但尽管如此，我对此还是有一些想法。

Ramesh Johari: The most important one is this.

拉梅什·乔哈里：最重要的就是这一点。

What I've found talking to people who want to start what they think is a marketplace is that they think too much about a marketplace before they're a marketplace.

在与那些想要开始他们认为的市场的人交谈时，我发现他们在成为市场之前对市场考虑得太多。

That in my view is the biggest failure mode.

在我看来，这是最大的失败模式。

Ramesh Johari: You mentioned specific things, cleaners.

Ramesh Johari：清洁工，你提到了具体的事情。

I wonder about that, right?

我想知道这一点，对吗？

Is it something about the cleaning industry?

与清洁行业有关吗？

It possibly is.

可能是这样。

I don't claim to be an expert on the microeconomics of the cleaning industry.

我并不自称是清洁行业微观经济学的专家。

But often it's not that, it's that I thought I was building a marketplace from the beginning, and that's not the way the world works.

但通常情况并非如此，而是我认为我从一开始就在建立一个市场，而世界并不是这样运作的。

So I'll give you one vignette of this that I really like, and that's UrbanSitter.

所以我会给你一个我非常喜欢的小插曲，那就是 UrbanSitter。

Ramesh Johari: So first, UrbanSitter is a babysitting marketplace.

Ramesh Johari：首先，UrbanSitter 是一个保姆市场。

We can talk about their whole life story, but I think what's most interesting is really the early days.

我们可以谈论他们的整个人生故事，但我认为最有趣的是早期的日子。

And in the early days, what I found interesting, the way I found out about them actually is that we were stuck looking for some help.

在早期，我发现有趣的是，我发现他们的方式实际上是我们一直在寻求帮助。

And I found out about this new platform where the clever thing was when you used to hire a babysitter, it's like pre Venmo days, you needed cash on hand.

我发现这个新平台的聪明之处在于，当你过去雇用保姆时，就像 Venmo 之前的时代一样，你手头需要现金。

Because when the babysitter's done at the end of the day, they're usually high school suits or something.

因为当保姆在一天结束时，他们通常会穿着高中套装什么的。

They want to get paid.

他们想要得到报酬。

They're not going to take your IOU, that you'll send them some check in the mail the next day.

他们不会拿走你的欠条，你会在第二天邮寄一些支票给他们。

Ramesh Johari: And unfortunately, you often don't have cash.

Ramesh Johari：不幸的是，你经常没有现金。

They don't take credit card.

他们不接受信用卡。

They're high school students.

他们是高中生。

That was an incredible friction to address, which is literally just we accept credit card payments for babysitting.

这是一个令人难以置信的摩擦，实际上我们接受信用卡支付保姆费用。

That's it, right?

就是这样，对吧？

Ramesh Johari: Now from there, what happened is they took advantage of Facebook networks between parents and babysitters to build trusted introductions.

Ramesh Johari：从那时起，他们利用父母和保姆之间的 Facebook 网络来建立可信的介绍。

So let's say my sitter wasn't available.

假设我的保姆不在。

I get to know sitters in the Facebook network of that sitter.

我在该保姆的 Facebook 网络中认识了该保姆。

And once they overcame that first thing to get some liquidity onto their platform, they could move towards asking, how do I solve for these frictions that I talked about earlier?

一旦他们克服了第一件事，为他们的平台获得一些流动性，他们可能会问，我如何解决我之前谈到的这些摩擦？

How do I solve for helping people find potential matches?

我如何解决帮助人们找到潜在匹配对象的问题？

How do I solve for people making those matches, right?

我该如何解决人们进行这些匹配的问题，对吗？

You can't do that when you don't have liquidity on your platform.

当你的平台上没有流动性时，你就无法做到这一点。

It's silly to tell someone, "Hey, I'm really going to help you find all those drivers out there, even though I only have three drivers on my platform." That's not a friction you're solving for.

告诉别人“嘿，我真的会帮你找到所有这些司机，尽管我的平台上只有三名司机”，这是很愚蠢的。这不是你要解决的摩擦。

Ramesh Johari: So in their example, as they evolved, they actually shifted their monetization model away from billing specifically for this friction of allowing you to pay with credit cards, instead to now billing for how you were interviewing and contacting sitters.

Ramesh Johari：因此，在他们的例子中，随着他们的发展，他们实际上改变了他们的盈利模式，从专门针对允许您使用信用卡付款的摩擦进行计费，改为现在针对您采访和联系保姆的方式进行计费。

They had a two-part plan for that.

他们为此制定了一个由两部分组成的计划。

One with a pay as you go menu, one with a more of a subscription option.

一种带有即用即付菜单，一种带有更多订阅选项。

But the key thing was either way, what you were paying for now was finding potential babysitters, not paying them with a credit card.

但关键是无论哪种方式，你现在所付出的代价都是寻找潜在的保姆，而不是用信用卡支付费用。

That wasn't the key thing anymore.

那已经不是关键了。

Ramesh Johari: So what's the moral there?

拉梅什·乔哈里：那么其中的寓意是什么？

The moral is a marketplace business never starts as a marketplace business, because what we think of as a marketplace business is something which at scale is removing the friction of the two sides finding each other.

其寓意是，市场业务永远不会以市场业务开始，因为我们所认为的市场业务是一种大规模消除双方寻找对方的摩擦的东西。

But when you start, you don't have that scale.

但当你开始的时候，你还没有那个规模。

Ramesh Johari: So when you start, you had better be thinking, "What's my value proposition in a world in which I don't have that scaled liquidity on both sides?" And that's bespoke.

Ramesh Johari：所以，当你开始时，你最好思考，“在一个双方都没有大规模流动性的世界里，我的价值主张是什么？”那是定制的。

It means different things.

这意味着不同的事情。

And in the case of oDesk, where I started, that initial thing was that remote work is a weird thing, because basically you've somehow got to know that this person who you're not next to is doing what you're asking them to do.

就我开始的 oDesk 而言，最初的事情是远程工作是一件奇怪的事情，因为基本上你必须以某种方式知道你不在旁边的这个人正在做你要求他们做的事情。

And so the initial value proposition of oDesk was to provide tools for workers to verify they were working the hours and doing the things that they said they were doing, screenshots and various kinds of tracking.

因此，oDesk 最初的价值主张是为员工提供工具，以验证他们是否按时工作并完成了他们所说的事情、屏幕截图和各种跟踪。

Ramesh Johari: And then in return for that, to be able to provide guarantees on both sides.

拉梅什·乔哈里：作为回报，能够为双方提供保证。

So now the workers could say, "Hey, I worked what I said so I should get paid." And the employers could say, "I actually see that you worked what you said.

所以现在工人们可以说：“嘿，我按照我说的做了，所以我应该得到报酬。”雇主可能会说：“我实际上看到你按照你所说的去做了。

And so I feel comfortable that I got what I paid for." That was the initial value proposition, is resolving a trust issue at a remote scale.

所以我觉得很舒服，因为我得到了我所付出的代价。”这是最初的价值主张，即远程解决信任问题。

Ramesh Johari: At that point, liquidity isn't the game.

Ramesh Johari：在这一点上，流动性不再是游戏的关键。

It's asking, what's a problem that people in this space are facing that I can deal with when I'm not a scaled marketplace?

它问的是，当我不是一个规模化的市场时，这个领域的人们面临着什么问题是我可以处理的？

So again, with the cleaning industry, I can comment on that from personal experience, but otherwise, I think that's the way I would think about it.

因此，对于清洁行业，我可以根据个人经验对此进行评论，但除此之外，我认为这就是我的思考方式。

It's almost never about building a marketplace when you're building a marketplace.

当你建立一个市场时，它几乎从来都不是建立一个市场。

Lenny: That's very similar to the advice I always give marketplace founders, is 90% of your problems are going to be non marketplace specific problems.

Lenny：这与我经常给市场创始人的建议非常相似，90% 的问题都将是非市场特定的问题。

They're going to be the same problems any startup is going to have.

这些问题与任何初创公司都会遇到的问题相同。

How do I grow?

我该如何成长？

It's going to be the same things you need to do.

这将是你需要做的同样的事情。

Ramesh Johari: So one thing you said was, "That's what you tell marketplace founders." I mean, something I've actually pressed hard on in my own way of thinking about this is that maybe we shouldn't talk about the concept of a marketplace founder.

Ramesh Johari：所以你说的一件事是，“这就是你告诉市场创始人的。”我的意思是，我实际上以自己的思考方式极力强调，也许我们不应该谈论市场创始人的概念。

Really there's founders.

确实有创始人。

And I think every entrepreneur...

我认为每个企业家...

I mean one way to think about it, right?

我的意思是一种思考方式，对吗？

It's very hard to think about a human business endeavor that has not been disrupted by the potential for transactions to take place online.

很难想象人类的商业活动没有受到在线交易潜力的干扰。

Ramesh Johari: And if that's the case, it means literally any founder is a marketplace founder.

Ramesh Johari：如果是这样的话，这意味着任何创始人都是市场创始人。

It'll be a choice they make after they grow as to whether they want to build a platform.

是否要建立一个平台，是他们成长后的选择。

I mean, to take a very hot recent example, no one in their right mind would've thought of OpenAI as a marketplace, but OpenAI is a marketplace now.

我的意思是，举一个最近非常热门的例子，任何头脑清醒的人都不会认为 OpenAI 是一个市场，但 OpenAI 现在是一个市场。

They may not want to call themselves a marketplace, but they have plugins.

他们可能不想称自己为市场，但他们有插件。

The plugins are flooding that platform.

插件充斥着该平台。

People have played with it.

人们已经玩过它了。

It's not an easy thing to find the plugin you need for what you want to do.

找到您想做的事情所需的插件并不是一件容易的事。

And that really is a two-sided thing now.

现在这确实是一个两面性的事情。

There's the plugin creators and there's the users.

有插件创建者，也有用户。

And they may believe it, they may not believe it, but they are a marketplace.

他们可能相信，也可能不相信，但他们是一个市场。

Ramesh Johari: So I think a different way to think about it is every founder is a marketplace founder.

Ramesh Johari：所以我认为换一种方式思考，每个创始人都是市场创始人。

It's going to be a choice they want to make for themselves of whether they want to become that platform.

是否想成为该平台将是他们想要为自己做出的选择。

That's I think one.

这是我认为的一个。

And two is because that's the case, I think one of the other challenges I find founders struggle with is you don't want to overcommit your future.

二是因为事实就是如此，我认为我发现创始人面临的其他挑战之一是你不想过度承诺你的未来。

And what I mean by that is that you're building up trust, and you're building up a sense of what kind of business you are in your early days.

我的意思是，你正在建立信任，并且你正在建立对自己早期业务类型的认识。

If you believe that this kind of platform future awaits you, or market platform future awaits you, there may be choices you're making early on that are tying your hands later.

如果您相信这种平台的未来在等待着您，或者市场平台的未来在等待着您，那么您早期做出的选择可能会在以后束缚您的手脚。

Ramesh Johari: A great example of this is when oDesk started, it was because the tools they were providing were for ongoing monitoring of work.

Ramesh Johari：oDesk 成立时就是一个很好的例子，因为他们提供的工具用于持续监控工作。

It's a very natural thing to say, "We will just take a constant fraction of the dollars that cross the platform." That all works well and good until after you become mature.

很自然地说，“我们只会收取跨平台美元的固定比例。”在你成熟之前，一切都很顺利。

Some of these relationships between worker and employer last a long time, and most of the value was generated now not so much because they're able to track each other, because the trust is now there, but because they found each other, because they're able to build that relationship through oDesk.

工人和雇主之间的一些关系持续了很长时间，现在产生的大部分价值并不是因为他们能够相互跟踪，因为信任现在已经存在，而是因为他们找到了彼此，因为他们能够通过 oDesk 建立这种关系。

Ramesh Johari: That meant that the longer that goes on, the less value the platform is adding into that relationship, but you're still pulling 10% of all the dollars.

Ramesh Johari：这意味着持续的时间越长，平台为这种关系增加的价值就越少，但你仍然可以获得所有美元的 10%。

So what does that lead to?

那么这会导致什么呢？

A word that most marketplace CEOs know well is disintermediation, which is where you were intermediating between the two parties, and now disintermediation means that essentially they're like, "Hey, we don't need you anymore."

大多数市场首席执行官都熟悉的一个词是脱媒，即你在两方之间进行中介，现在脱媒意味着他们基本上会说，“嘿，我们不再需要你了。”

Ramesh Johari: My favorite example is we had some stuff delivered from IKEA by a Thumbtack worker once, and my wife is like, "Oh, thanks a lot.

Ramesh Johari：我最喜欢的例子是，我们有一次由图钉工人从宜家送来一些东西，我的妻子说，“哦，非常感谢。

You're so reliable." He's like, "Hey, great.

你真是太可靠了。”他就像，“嘿，太棒了。

Here's my business card.

这是我的名片。

Ever need me again?

以后还需要我吗？

Just call the number on the back." And that was it.

拨打后面的号码就可以了。”就是这样。

Thumbtack got their one lead gen, and then we didn't need the platform anymore.

Thumbtack 获得了一位潜在客户，然后我们就不再需要这个平台了。

Ramesh Johari: And I think this issue for oDesk meant that after they merged with Elance and became Upwork, they had to think a little bit about, "Okay, what's the monetization strategy we want to use?

Ramesh Johari：我认为 oDesk 的这个问题意味着，在他们与 Elance 合并并成为 Upwork 后，他们必须思考一下，“好吧，我们想要使用的盈利策略是什么？

How do we address this issue that longer term relationships may disintermediate?

我们如何解决长期关系可能脱媒的问题？

And does that mean you need a pricing plan that actually takes that into account?" So early commitments in this case to a particular pricing scheme, particular monetization, can really tie your hands as you then realize later you actually are a platform.

这是否意味着您需要一个真正考虑到这一点的定价计划？”因此，在这种情况下，早期对特定定价方案、特定货币化的承诺确实会束缚你的手脚，因为你后来意识到你实际上是一个平台。

Lenny: I really like this message.

莱尼：我真的很喜欢这个消息。

It makes me think about Substack actually, which started as just a platform for newsletter writers.

这让我实际上想到了 Substack，它最初只是一个新闻通讯作者的平台。

And then they're like, "How do we make this more valuable?" Because they take a cut of everyone's revenue.

然后他们会问，“我们如何让这个更有价值？”因为他们从每个人的收入中抽取一部分。

And they've actually invested heavily on helping drive demand to writers, for example, me.

他们实际上投入了大量资金来帮助推动对作家（例如我）的需求。

And at this point, over 80% of my subscribers come from Substack's network.

目前，我超过 80% 的订阅者来自 Substack 的网络。

And so they've built this marketplace element exactly as you're describing, where they just found, "Here's a pain point, writers need more subscribers.

因此，他们完全按照您所描述的那样构建了这个市场元素，他们刚刚发现，“这是一个痛点，作家需要更多订阅者。

How do we help them drive subscribers?" So they figured out all these ways to create demand.

我们如何帮助他们吸引订阅者？”所以他们想出了所有这些创造需求的方法。

Ramesh Johari: That's a really positive story, where they managed to actually expand the frontier of their business by enabling that network.

Ramesh Johari：这是一个非常积极的故事，他们通过启用该网络成功地实际扩展了业务领域。

For every one of those, there's unfortunately a lot of negative stories.

不幸的是，对于其中的每一个，都有很多负面的故事。

I mean, one that I think is very painful is how eBay had a lot of challenges with its seller community as it introduced more and more fine-grained sources of fees.

我的意思是，我认为非常痛苦的一点是，随着 eBay 推出越来越细粒度的费用来源，它的卖家社区遇到了很多挑战。

Ramesh Johari: And I think a lot of that, I mean there's many, many treatises at this point written on eBay, and their history, and how they got to the point that they're at.

Ramesh Johari：我认为其中有很多，我的意思是，此时在 eBay 上写了很多很多论文，以及它们的历史，以及它们如何达到现在的水平。

But I think one kind of simple thing I do want people to think about there is that the sellers on eBay who had matured with the platform, who had grown with it, had come to develop certain expectations about what their lives on that platform would look like.

但我认为我确实希望人们思考的一件简单的事情是，eBay 上的卖家随着平台的发展而成熟，随着平台的成长，他们已经对他们在该平台上的生活产生了一定的期望。

And it's understandable, because a lot of these businesses, they had built their livelihood on that platform.

这是可以理解的，因为很多这样的企业都是在这个平台上建立自己的生计的。

That was their entire business.

这就是他们的全部事业。

Ramesh Johari: So when you now reach in and you say, "I'm going to completely change the rules of the game in which your business model operates," from the perspective of those sellers, that's a breaking of a social contract that's been developed over a very long time.

拉梅什·乔哈里：所以，当你现在介入并说，“我将彻底改变你的商业模式运作的游戏规则”时，从那些卖家的角度来看，这是对长期以来形成的社会契约的破坏。

So I love the Substack example, because that's like, "Hey, let me amplify our social contract." But I think for every one of those, there's an eBay warning sign that you can trap yourself a little bit.

所以我喜欢 Substack 的例子，因为这就像，“嘿，让我放大我们的社会契约。”但我认为对于其中的每一个，eBay 都有一个警告信号，表明您可能会陷入一些陷阱。

Lenny: Just to close a loop on this really, I think important point, a lot of people listening to this are probably, "I'm a marketplace founder.

莱尼：为了结束这个问题，我认为很重要的一点是，很多听这个的人可能会说，“我是一个市场创始人。

I'm building a marketplace," are going to hear this and be like, "Oh shit, maybe I need to rethink how I think about what I'm doing." What would be your piece of advice to people like that?

我正在建立一个市场，”听到这个之后会想，“哦，糟糕，也许我需要重新思考我对自己正在做的事情的看法。”您对这样的人有什么建议？

Is it focus on the friction point and it may be a marketplace solution, it may be a managed marketplace, it may be you own the supply?

它是否专注于摩擦点，它可能是一个市场解决方案，它可能是一个托管市场，它可能是您拥有供应？

Is that the advice, or what would your advice be to someone that's like, "I'm building a marketplace"?

这是建议吗？或者您对“我正在建立一个市场”之类的人有什么建议？

How should they reframe their thinking?

他们应该如何重构思维？

Ramesh Johari: Let's go back to kind of thinking about this concept of a marketplace of reducing friction.

Ramesh Johari：让我们回过头来思考一下减少摩擦的市场概念。

So the litmus test I like to give to someone who claims to me that they're building a marketplace business or they're a marketplace founder is do you have what I would call scaled liquidity on both sides of your platform?

因此，对于那些向我声称自己正在建立市场业务或市场创始人的人，我想进行的试金石是，您的平台两侧是否拥有我所说的规模化流动性？

What does scaled liquidity mean?

流动性规模化意味着什么？

Ramesh Johari: What it means in lay terms...

拉梅什·乔哈里：通俗地说这意味着什么......

And by the way, I am a data scientist, and I love to think about these quantitatively.

顺便说一句，我是一名数据科学家，我喜欢定量地思考这些问题。

But fundamentally, if it doesn't pass the smell test, then you don't have to keep going with the data science.

但从根本上来说，如果它没有通过嗅觉测试，那么你就不必继续研究数据科学。

The smell test is scaled liquidity asks, "Do I have a lot of buyers and a lot of sellers on my platform, or do I only have one of these two, or do I have neither?" If you don't have both, you could call yourself whatever you want to call yourself, but at this moment in time, you're not a marketplace.

流动性嗅觉测试会询问“我的平台上是否有很多买家和很多卖家，或者我只有这两者之一，还是两者都没有？”如果你两者都没有，你可以随意称呼自己，但此时此刻，你不是一个市场。

If you have one, congratulations.

如果你有的话，恭喜你。

You've won the game on one side of the market.

您已经在市场的一侧赢得了这场游戏。

And now you if you want, you have a choice point.

现在，如果你愿意的话，你有一个选择点。

You can lean into growth on the side that you're doing well with.

你可以在自己做得好的方面取得增长。

You got a ton of users, ton of buyers?

你有大量的用户、大量的买家吗？

Great.

伟大的。

Lean into it, get more buyers.

深入其中，吸引更多买家。

That's one option.

这是一种选择。

There's no shame in not being a marketplace.

不成为市场并没有什么可耻的。

Scaling a business is scaling a business.

扩展业务就是扩展业务。

If that's the way to do it, do it.

如果这是这样做的方法，那就这样做。

Ramesh Johari: If you decide you want to be a marketplace, then at that moment when you've got a lot of buyers, but not a lot of sellers, or a lot of sellers, but not a lot of buyers, the choice you're facing is, how do I take advantage of having that one side scaled to attract the other side?

Ramesh Johari：如果你决定要成为一个市场，那么当你有很多买家但没有很多卖家，或者有很多卖家但没有很多买家时，你面临的选择是，我如何利用一侧的规模来吸引另一侧？

We can talk more about that, but there's a lot of ways to kind of hack that, to think about how...

我们可以更多地讨论这个问题，但是有很多方法可以解决这个问题，思考如何......

So to take Uber as an example, they would walk into a new city, and one thing that Uber was commonly known for doing this was back in the days when really Uber Black was the only service is they just hand out coupons for free rides at events, parties, things like that, to take people home.

以 Uber 为例，他们会走进一个新城市，而 Uber 因这样做而广为人知的一件事是在 Uber Black 真正是唯一服务的时代，他们只是在活动、聚会等活动中发放免费乘车优惠券，然后带人们回家。

And that was a way of saying, "Hey, we're subsidizing the drivers in the city.

这就是说，“嘿，我们正在补贴城市里的司机。

That's our scaled side.

这就是我们缩放的一面。

Now we're going to use that subsidized driver base to attract riders."

现在我们将利用补贴的司机基础来吸引乘客。”

Ramesh Johari: So that's like, how do you get that flywheel going?

Ramesh Johari：那么，如何让飞轮运转起来呢？

And again, many people have written about how to take liquidity, scaled liquidity on one side, and use it to attract the other side.

同样，许多人都写过如何利用流动性、扩大一方面的流动性，并用它来吸引另一方。

Ramesh Johari: If you don't have either side, don't worry about it.

Ramesh Johari：如果你没有任何一方，别担心。

Don't worry about being a marketplace.

不用担心成为市场。

Worry about scaling one side.

担心一侧会缩放。

And in that world, it opens your visibility up completely into the advice of many, many startup advisors.

在那个世界里，它让你完全了解许多创业顾问的建议。

People who have advice not so much about scaling a marketplace, but about scaling a startup.

他们提供的建议与其说是关于扩大市场规模，不如说是关于扩大初创企业规模。

Ramesh Johari: And I want to say you got to let the ego go at that point.

拉梅什·乔哈里：我想说，此时你必须放下自我。

It's fine to articulate to people that your vision of the future is to be a platform or marketplace.

向人们阐明你的未来愿景是成为一个平台或市场是很好的。

As I said, virtually every business is going to have that option at some point in the modern tech enabled economy anyway.

正如我所说，无论如何，在现代科技驱动的经济中，几乎每个企业在某个时候都会有这种选择。

So you're not saying something people don't already know when you tell an advisor or an investor that.

因此，当您告诉顾问或投资者时，您并不是在说人们还不知道的事情。

But I do think you need to be humble enough at the starting point to recognize that there's no sense in talking about a marketplace if you don't have scaling on either side yet.

但我确实认为，一开始你就需要足够谦虚，认识到如果双方都还没有扩大规模，那么谈论市场就没有意义。

Lenny: And then it becomes a question of a business model, unit economics of, can I build say a DoorDash, not as a marketplace?

Lenny：然后就变成了商业模式、单位经济学的问题，我可以建立一个 DoorDash，而不是作为一个市场吗？

Can I just hire a bunch of people delivering?

我可以雇佣一群人来送货吗？

Is this even possible in a different route?

这甚至可以通过不同的路线实现吗？

Ramesh Johari: Yeah, that's a great point.

拉梅什·乔哈里：是的，这是一个很好的观点。

One of the things I think that's useful for people to think about here that you're raising, at some level, it's kind of tied up I think with that question of whether I should have employees, or contract or freelance work on one side of the marketplace.

我认为人们在这里思考的一件事是有用的，你在某种程度上提出了这一问题，我认为这与我是否应该在市场的一侧拥有员工、合同工或自由职业的问题有关。

Ramesh Johari: And that's actually a pretty old question in economics.

拉梅什·乔哈里：这实际上是经济学中一个非常古老的问题。

The way we talk about it often is a distinction between a market or a firm.

我们经常谈论它的方式是市场和公司之间的区别。

And one of the interesting puzzles in economics, Ronald Coase is a famous economist who thought about this is, "Well, if markets are so efficient, why do we need firms?

经济学中有趣的谜题之一是著名经济学家罗纳德·科斯，他思考过这个问题：“如果市场如此有效，为什么我们需要公司呢？

If markets are efficient at matching labor up with things that need to get done, why would you ever need a firm?" And that's one of the earliest recognitions that transaction costs are a real thing.

如果市场能够有效地将劳动力与需要完成的事情相匹配，那么为什么还需要一家公司呢？”这是对交易成本真实存在的最早认识之一。

And that's one of the things that firms are solving for.

这是公司正在解决的问题之一。

Ramesh Johari: And I love what you're saying because what it's recognizing is, "Hey, for your frictions, the best resolution to that might not be to have a marketplace.

Ramesh Johari：我喜欢你所说的，因为它认识到，“嘿，对于你们的摩擦，最好的解决办法可能不是建立一个市场。

It might actually be to have very tightly controlled labor." A good example of this actually, Stitch Fix, I think one of the things that's cool about Stitch Fix is the experience that people had early on with stylists at Stitch Fix.

实际上可能是对劳动力进行非常严格的控制。”实际上，Stitch Fix 就是一个很好的例子，我认为 Stitch Fix 最酷的地方之一就是人们早期在 Stitch Fix 与造型师合作的体验。

Lenny: I'm a happy customer, by the way.

莱尼：顺便说一句，我是一位满意的顾客。

I think .

我认为 。

Ramesh Johari: Yeah, I think one of the great things about that experience is it felt magical to have someone who kind of got to know you.

拉梅什·乔哈里（Ramesh Johari）：是的，我认为这次经历的最棒的事情之一就是有一个认识你的人感觉很神奇。

But that depends on a relationship that doesn't feel like a freelance relationship every single time you're going back.

但这取决于你每次回去时感觉不像自由职业关系的关系。

Ramesh Johari: Another example that I would pull out is pretty much any healthcare platform.

Ramesh Johari：我想举的另一个例子是几乎任何医疗保健平台。

So for example, for physical therapy, it'd be weird if every time you went to a physical therapy platform, you just got randomly matched to whoever happened to be available then.

例如，对于物理治疗，如果每次你去物理治疗平台时，你都会随机匹配到当时有空的人，那就很奇怪了。

So I think there's some curation that needs to happen of that relationship.

所以我认为需要对这种关系进行一些管理。

Does that mean full employee?

这是否意味着全职员工？

Maybe not.

也许不是。

But it does mean you have to think a little bit about exactly as you brought up, what's the nature of curation of your labor pool?

但这确实意味着你必须像你提出的那样仔细思考一下，你的劳动力资源管理的本质是什么？

Lenny: Awesome.

莱尼：太棒了。

Okay, so let's come back to a point you made early on around the importance of data and the power of data in actually making your marketplace a lot more efficient and work more effectively.

好的，让我们回到您之前提出的关于数据的重要性以及数据的力量实际上使您的市场更加高效、运作更加有效的观点。

So say that you have a data scientist, or a data analyst, or someone that is helping you optimize your marketplace.

假设您有一位数据科学家、数据分析师或正在帮助您优化市场的人。

Where do you often find the biggest leverage and opportunity for a data person to help you make your marketplace more effective?

您经常在哪里找到数据人员帮助您提高市场效率的最大杠杆和机会？

Ramesh Johari: This is an incredible question, right?

拉梅什·乔哈里：这是一个令人难以置信的问题，对吧？

Because I think I could answer it a number of different ways.

因为我认为我可以用多种不同的方式来回答这个问题。

One question I think that's kind of basic, it's just what should this person be doing?

我认为这是一个基本问题，就是这个人应该做什么？

And I'm going to actually evade that question a little bit.

我实际上会稍微回避这个问题。

I'm going to give some examples of what they could do, but I feel like that's one where context matters a lot.

我将举一些例子来说明他们可以做什么，但我觉得这是一个非常重要的背景。

Ramesh Johari: So as an example, at ride-sharing or grocery delivery marketplaces, pricing means actually, what do you pay for that ride?

Ramesh Johari：举个例子，在拼车或杂货配送市场，定价实际上意味着，你为这次乘车支付多少钱？

Or what do you pay for that delivery?

或者你为这次送货支付什么费用？

So that's actually the price that's set at the moment you actually place the order.

这实际上是您实际下订单时设定的价格。

Just to be clear, by the way, if you order from DoorDash, I don't mean the price of the restaurant.

顺便说一句，如果您从 DoorDash 订购，我指的不是餐厅的价格。

I mean, what do you pay to DoorDash, right?

我的意思是，你向 DoorDash 支付了什么费用，对吧？

What's that fee?

那个费用是多少？

Is there a surcharge, because it's surge or whatever?

是否有附加费，因为它是激增或其他什么？

Okay, so that's a thing, right?

好吧，那是一件事，对吧？

Ramesh Johari: But that's not really a thing in a marketplace where the platform's not setting the prices.

Ramesh Johari：但在平台不定价的市场上，这并不是什么真正的事情。

So in Airbnb, really hosts are the ones who are in charge of setting prices for their listings.

因此，在 Airbnb 中，真正的房东是负责为其房源定价的人。

Ramesh Johari: One answer to your question is, if I'm in a place like Uber, Lyft, DoorDash, I want to have good data scientists thinking about pricing.

Ramesh Johari：你的问题的一个答案是，如果我在 Uber、Lyft、DoorDash 这样的地方，我希望有优秀的数据科学家考虑定价。

Because that seems like something which should be heavily dependent on the instantaneous state of supply and demand in my marketplace.

因为这似乎应该严重依赖于我的市场的瞬时供需状态。

So that's one type of answer is, well do I need data scientists working on pricing?

因此，一种答案是，我需要数据科学家来研究定价吗？

Do I need data scientists working on search?

我需要数据科学家进行搜索吗？

Why search?

为什么要搜索？

Because maybe in my marketplace, finding the needle in the haystack is really the biggest, highest friction problem.

因为也许在我的市场中，大海捞针确实是最大、摩擦最大的问题。

So maybe I need a lot more data scientists saying about search.

所以也许我需要更多的数据科学家谈论搜索。

Ramesh Johari: That's what I'm going to evade.

Ramesh Johari：这就是我要回避的。

Okay?

好的？

I'm going to focus more on something completely different, which is just a more philosophical point about what a data scientist does.

我将更多地关注完全不同的事情，这只是关于数据科学家所做工作的更哲学的观点。

Ramesh Johari: So in a lot of companies today, especially, a main thing that you ask data scientists to do is build what's called a machine learning model.

Ramesh Johari：因此，特别是在当今的许多公司中，要求数据科学家做的一件主要事情就是建立所谓的机器学习模型。

Now, machine learning model even already can mean a lot of things to a lot of different people.

现在，机器学习模型对于很多不同的人来说已经意味着很多事情。

I'm going to focus on something very concrete.

我将重点关注一些非常具体的事情。

You're asking them to predict something.

你要求他们预测一些事情。

Ramesh Johari: When I started at oDesk, this is in 2012, one of the funny things about me is I started at oDesk because I'd had a academic career up to that point in about 10 years, just building mathematical models of things.

Ramesh Johari：当我开始在 oDesk 工作时，那是在 2012 年，关于我的一件有趣的事情是我开始在 oDesk 工作，因为我的学术生涯大约有 10 年的时间，只是建立事物的数学模型。

I was not really very much of a data scientist up to that point.

到目前为止，我还不是一个真正的数据科学家。

What I expected would happen is I'd go to industry and I'd be told, "Hey, look how important data is." And definitely my eyes were opened.

我预计会发生的情况是，当我进入行业时，我会被告知，“嘿，看看数据有多重要。”我的眼睛肯定被打开了。

Ramesh Johari: And one of the first things I was asked to think about is, well, okay, someone comes to oDesk, post a job, workers apply to that job.

Ramesh Johari：我被要求考虑的第一件事是，好吧，有人来到 oDesk，发布一份工作，工人申请该工作。

Predict which of these workers is most likely to be hired on that job.

预测这些工人中哪一位最有可能被雇用从事该工作。

That was the narrow question.

这是一个狭隘的问题。

And so why is that a good question?

那么为什么这是一个好问题呢？

Because we have a whole awesome set of tools now to solve that kind of a problem exactly.

因为我们现在拥有一整套很棒的工具来准确解决此类问题。

How do we do it?

我们该怎么做呢？

Take a lot of past data of past jobs, past applicants, past hires that were made.

获取大量关于过去的工作、过去的申请人、过去的聘用人员的数据。

Then we ask these crazy big black box algorithms, "All right, do the best job you can predicting who's going to get hired on this job with these applicants." And we use that data to test how well these algorithms are doing.

然后我们问这些疯狂的大黑匣子算法，“好吧，尽你所能预测谁会与这些申请人一起被聘用。”我们使用这些数据来测试这些算法的表现。

That's machine learning in 30 seconds basically.

这基本上就是 30 秒内的机器学习。

So we're working on this problem.

所以我们正在努力解决这个问题。

Great.

伟大的。

Ramesh Johari: And then I kind of poked my head up a little bit.

Ramesh Johari：然后我稍微抬起了头。

I go, "Why are we working?

我说：“我们为什么要工作？

What is this going to do?" Well, it turns out the reason these kinds of things are important is they get used to make decisions.

这是要做什么？”事实证明，这些事情之所以重要，是因为它们习惯于做出决定。

So what kind of decision do you make with that?

那么你会做出什么样的决定呢？

Well, one thing you do is you say, "Well, if I could predict who's most likely to be hired, then I should just rank people based on that, and that would be a good matching algorithm.

好吧，你做的一件事就是说：“好吧，如果我可以预测谁最有可能被雇用，那么我应该根据此对人员进行排名，这将是一个很好的匹配算法。

That'd be a good way to sort and triage applicants for employers when they're screening, trying to figure out who to interview, who to hire." Great.

这将是雇主在筛选申请人、试图找出面试谁、雇用谁时对申请人进行分类和分类的好方法。”伟大的。

Sounds pretty natural.

听起来很自然。

Ramesh Johari: And then you think about it a little bit, and this to me, it's such a passion project to get people to understand that this is why the humans in the loop that help us in businesses and making sense of data are so critical, is the following problem.

Ramesh Johari：然后你稍微想一下，对我来说，这是一个充满激情的项目，让人们明白，这就是为什么循环中的人员在业务中帮助我们并理解数据如此重要，这就是以下问题。

Ramesh Johari: If you think about it a little bit, you realize what that algorithm is doing, it's really just picking up on patterns in past data.

Ramesh Johari：如果你稍微思考一下，你就会意识到该算法在做什么，它实际上只是在提取过去数据中的模式。

So yeah, that's great.

是的，那太好了。

This person is likely to be hired.

这个人很可能被录用。

But what we really want is something different.

但我们真正想要的是不同的东西。

We're trying to add value by ranking people.

我们试图通过对人员进行排名来增加价值。

Ramesh Johari: So to give another example that's similar to this, when you're a marketing manager, and you've got a cracked data science team that's built a long-term value, lifetime value model for you, you're not going to get in trouble with anyone if you send your highest value promotions to the highest LTV customers, right?

Ramesh Johari：举另一个与此类似的例子，当您是一名营销经理，并且您拥有一支出色的数据科学团队为您构建了长期价值、终生价值模型时，如果您将最高价值的促销活动发送给最高 LTV 的客户，您就不会遇到任何麻烦，对吧？

Who's going to blame you for that?

谁会为此责怪你呢？

Because you're like, "This person is worth a lot, and I sent them this promotion." Say that in your monthly report, nobody's going to give you a hard time.

因为你会说，“这个人很有价值，我向他们发送了这次晋升。”假设在你的月度报告中，没有人会为难你。

Ramesh Johari: But the problem with that way of thinking is actually predicting what their lifetime value is isn't really the question.

Ramesh Johari：但这种思维方式的问题实际上是预测它们的终生价值并不是真正的问题。

The question is, how much more are they going to spend on my platform because I sent them that promotion?

问题是，因为我向他们发送了促销信息，他们会在我的平台上多花多少钱？

Ramesh Johari: That's a very different thing.

拉梅什·乔哈里：这是一个非常不同的事情。

It's a differential rather than an absolute.

这是有差别的，而不是绝对的。

I'm not interested in their absolute LTV.

我对他们的绝对生命周期价值不感兴趣。

I'm absolutely interested in the difference in their LTV because I sent them this promotion.

我对他们的 LTV 差异非常感兴趣，因为我向他们发送了此促销信息。

Ramesh Johari: And when you look at it that way, what you realize can happen is picking up on patterns because of good predictions, right?

Ramesh Johari：当你这样看时，你会意识到，由于良好的预测，可能会出现某种模式，对吗？

Finding the people that have high LTV because you predicted that is very different than making good decisions, which is about saying the difference in their LTV is going to be higher because I sent them this promotion.

找到具有高 LTV 的人，因为您预测这与做出正确的决策有很大不同，这意味着他们的 LTV 差异会更高，因为我向他们发送了这次促销活动。

Ramesh Johari: I love this example, because I taught a class here at Stanford.

Ramesh Johari：我喜欢这个例子，因为我在斯坦福大学教过一门课。

It was like an executive education class.

这就像一堂高管教育课程。

We had all the executives from a company in the room, and one of the people in the room was the chief marketing officer.

我们房间里有一家公司的所有高管，其中一个人是首席营销官。

And I just asked this question like, "Hey, okay, let's say you got this great LTV model, who would you send the promotions to?" It's like, "Definitely the highest LTV people," and there's a CMO in the room.

我只是问了这样的问题：“嘿，好吧，假设您有一个很棒的 LTV 模型，您会将促销信息发送给谁？”就像，“绝对是生命周期价值最高的人”，房间里有一位首席营销官。

And so it's a little bit of a delicate situation, like pushing back a little bit, right?

所以这是一个有点微妙的情况，就像稍微推迟一点，对吧？

Ramesh Johari: I do want to be clear, there's reputational reasons you might do that anyway.

拉梅什·乔哈里：我确实想澄清一下，无论如何，出于声誉原因，你可能会这样做。

I mean, I'm not trying to get away from that.

我的意思是，我并不想逃避这一点。

But just to make the narrow point that predicting is about picking up patterns, but making decisions, it's about thinking about these differences.

但狭隘地说，预测是关于拾取模式，但做出决策是关于思考这些差异。

Ramesh Johari: Now, why is that important?

拉梅什·乔哈里：现在，为什么这很重要？

Because we learn in high school, correlation is not causation.

因为我们在高中学习，相关性不是因果关系。

That's a phrase everybody has heard all over the place.

这是每个人都听过的一句话。

What does that have to do with this?

这和这有什么关系呢？

Well, when we teach people to build machine learning models, we're asking them to make predictions, we're asking them to find correlations.

好吧，当我们教人们构建机器学习模型时，我们要求他们做出预测，我们要求他们找到相关性。

Prediction is inherently about correlation.

预测本质上是关于相关性的。

But when we ask people to make decisions, we're asking them to think about causation.

但当我们要求人们做出决定时，我们要求他们考虑因果关系。

"If I make this decision, then will I actually increase the net value of my business?

“如果我做出这个决定，那么我真的会增加我的企业净值吗？

Will I have by sending the promotion, increased the likelihood that this person is going to spend more on my platform?"

我是否可以通过发送促销信息来增加此人在我的平台上花费更多的可能性？”

Ramesh Johari: And so the first and most important thing that I feel very strongly about in what would I get a data scientist to do is no matter who they are, even if it was that person in the weeds thinking about building this prediction model for hiring, get them to be thinking in the back of their mind always that their goal is to help the business make decisions.

Ramesh Johari：因此，在让数据科学家做什么时，我强烈感受到的第一件事也是最重要的事情是，无论他们是谁，即使是那个正在考虑构建招聘预测模型的人，也要让他们始终在内心深处思考，他们的目标是帮助企业做出决策。

And that the distinction between causation and correlation matters a lot.

因果关系和相关性之间的区别非常重要。

We can talk a lot more about how does that play out in terms of their day-to-day work.

我们可以更多地讨论这在他们的日常工作中是如何发挥作用的。

But at least at a starting point, you have to recognize that the first step is always recognition that prediction isn't the same thing as making decisions.

但至少在起点上，您必须认识到，第一步始终是认识到预测与做出决策不同。

Lenny: So the takeaway here is as a data team and as a data scientist on the team, is help the business make predictions.

Lenny：因此，作为数据团队和团队中的数据科学家，这里的要点是帮助企业做出预测。

Are there a couple more examples you could share of just what is an example of a decision that you think they often should be making and using data to help them with?

您是否可以分享更多的例子来说明您认为他们经常应该做出并使用数据来帮助他们做出决定的例子是什么？

Ramesh Johari: Maybe the right frame of reference for this, and the word that an academic would use is causal inference.

Ramesh Johari：也许这是正确的参考框架，学者会使用的词是因果推理。

So what we're changing from is machine learning to causal inference.

所以我们正在从机器学习转向因果推理。

So let's think that through in a couple of different use cases that are related to that marketplace data science flywheel I talked about earlier.

因此，让我们思考一下与我之前谈到的市场数据科学飞轮相关的几个不同用例。

Finding matches, making matches, and then learning about matches.

寻找匹配，进行匹配，然后学习匹配。

Ramesh Johari: So finding matches, like you said, a core part of that is search and recommendation, and each of those relies on rankings.

Ramesh Johari：因此，正如您所说，寻找匹配的核心部分是搜索和推荐，而其中每一个都依赖于排名。

So I want to be able to rank order.

所以我希望能够排序。

Let's say I go do a search on Airbnb.

假设我去 Airbnb 上搜索。

On a rank order, the different listings in the marketplace, right?

按照排名顺序，市场上有不同的列表，对吗？

At some level, it's true that what I'm trying to do there is I'm trying to just predict, what are you going to like the most?

在某种程度上，我确实想做的是预测，你最喜欢什么？

Ramesh Johari: But I think there's an important piece of that also, which is that I want to think a little bit about the distinction between two different ranking algorithms.

Ramesh Johari：但我认为其中还有一个重要的部分，那就是我想思考一下两种不同排名算法之间的区别。

That's the real decision that's being made.

这才是正在做出的真正决定。

Ramesh Johari: And when I think about the distinction between two different ranking algorithms, I don't want to be only comparing them in terms of how well they recreate the choices people made in the past.

Ramesh Johari：当我思考两种不同排名算法之间的区别时，我不想仅仅根据它们重新创建人们过去做出的选择的程度来比较它们。

The way I'm really going to evaluate those is in my market, does one of those lead to better matches or more matches than the other one, right?

我真正要评估这些的方式是在我的市场中，其中一种是否会比另一种带来更好的匹配或更多的匹配，对吗？

Ramesh Johari: So Airbnb as a business, what are the most obvious core metrics?

Ramesh Johari：Airbnb 作为一家企业，最明显的核心指标是什么？

It's bookings and revenue.

这是预订量和收入。

So you're going to want to ask a very basic question.

所以你会想问一个非常基本的问题。

If I use the ranking algorithm Lenny just developed last night versus the ranking algorithm Ramesh developed last week, does Lenny's ranking algorithm lead to more bookings than Ramesh's ranking algorithm?

如果我使用 Lenny 昨晚刚刚开发的排名算法与 Ramesh 上周开发的排名算法进行比较，Lenny 的排名算法是否会比 Ramesh 的排名算法带来更多的预订？

Ramesh Johari: And it's so important to put it that way starkly, because that's so different a question than, does Lenny's ranking algorithm do a better job of predicting over the last two years what bookings people made than Ramesh's ranking algorithm?

Ramesh Johari：这么说非常重要，因为这是一个非常不同的问题，Lenny 的排名算法是否能比 Ramesh 的排名算法更好地预测过去两年人们的预订情况？

So that's I think at that level.

这就是我在这个层面上的想法。

Ramesh Johari: Then we talked a little bit about ranking at the point of making a match, and I think that's where this hiring issue popped up.

Ramesh Johari：然后我们在比赛时讨论了一些排名问题，我认为这就是招聘问题出现的地方。

Because in the end, while we might have these predictive algorithms to rank who you're going to hire, that's not the important question.

因为最后，虽然我们可能有这些预测算法来对你要雇用的人进行排名，但这并不是重要的问题。

Ramesh Johari: Interestingly, the important question is actually to evaluate the quality of the match that's made.

Ramesh Johari：有趣的是，重要的问题实际上是评估所进行的比赛的质量。

And we would do that through the next step of that flywheel.

我们将通过飞轮的下一步来做到这一点。

We'd ask ourselves, what ratings did they give back to that freelancer?

我们会问自己，他们给那个自由职业者什么评价？

Do they hire that freelancer again?

他们会再次雇用那个自由职业者吗？

So you're comparing two different algorithms not through their ability to recreate the past, but their ability to make matches in the future that can be objectively evaluated to say, "Hey, I increased the value of the business.

因此，您比较两种不同的算法，不是通过它们重现过去的能力，而是通过它们在未来进行匹配的能力，可以客观地评估说，“嘿，我增加了业务价值。

I actually made better matches this way." And then rating systems, I think we could talk quite a bit about a similar phenomenon there too.

事实上，我通过这种方式取得了更好的比赛。”然后是评级系统，我想我们也可以谈论很多类似的现象。

Lenny: This episode is brought to you by Eppo.

Lenny：这一集是 Eppo 为您带来的。

Eppo is a next generation A/B testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams.

Eppo 是由 Airbnb 和 Snowflake 的校友为现代增长团队构建的下一代 A/B 测试和功能管理平台。

Companies like Twitch, Miro, ClickUp, and DraftKings rely on Eppo to power their experiments.

Twitch、Miro、ClickUp 和 DraftKings 等公司依靠 Eppo 来支持他们的实验。

Lenny: Experimentation is increasingly essential for driving growth and for understanding the performance of new features.

Lenny：实验对于推动增长和了解新功能的性能越来越重要。

And Eppo helps you increase experimentation velocity while unlocking rigorous deep analysis in a way that no other commercial tool does.

Eppo 可以帮助您提高实验速度，同时以其他商业工具无法做到的方式解锁严格的深度分析。

Lenny: When I was at Airbnb, one of the things that I loved most was our experimentation platform, where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own.

Lenny：当我在 Airbnb 时，我最喜欢的事情之一就是我们的实验平台，在这里我可以轻松地设置实验、解决问题并自行分析性能。

Eppo does all that and more, with advanced statistical methods that can help you shave weeks off experiment time, and accessible UI for diving deeper into performance, and out of the box reporting that helps you avoid annoying prolonged analytic cycles.

Eppo 做到了这一切，甚至更多，先进的统计方法可以帮助您缩短数周的实验时间，可访问的 UI 可以更深入地了解性能，开箱即用的报告可以帮助您避免烦人的长时间分析周期。

Lenny: Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the A/B testing flywheel.

Lenny：Eppo 还使您可以轻松地与团队分享实验见解，激发 A/B 测试飞轮的新想法。

Eppo powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing.

Eppo 为每个用例提供实验支持，包括产品、增长、机器学习、货币化和电子邮件营销。

Check out Eppo at geteppo.com/lenny, and 10x your experiment velocity.

请访问 geteppo.com/lenny 查看 Eppo，将实验速度提高 10 倍。

That's geteppo.com/lenny.

那是 geteppo.com/lenny。

Lenny: Yeah, I would actually love to talk about rating systems, but there's an implication in everything you're describing of running an experiment versus looking at what would've happened in the previous world made.

莱尼：是的，我实际上很想谈论评级系统，但是你所描述的一切都暗示着进行实验，而不是看看在以前的世界中会发生什么。

You've made change, run an experiment, see if it actually makes an impact on bookings and revenue.

您已经做出了改变，进行了实验，看看它是否真的对预订和收入产生了影响。

And that leads me to a question I wanted to ask, which is with experiments, there's kind of this classic challenge, and always elephant in the room of if you just run a bunch of experiments, you're kind of going to micro optimize, lead to these local maxima, and you may miss big opportunities and big unlocks if you're just extremely experiment driven.

这引出了我想问的一个问题，那就是实验，有一种经典的挑战，如果你只是进行一堆实验，你就会进行微观优化，导致这些局部最大值，如果你只是极度实验驱动，你可能会错过大机会和大解锁。

Lenny: You spend a lot of time thinking about experimentation.

莱尼：你花了很多时间思考实验。

What have you learned or what advice do you have for people to either be less worried about optimizing and missing something big, or just finding a balance with running experiments, but also creating opportunity to find a huge new opportunity?

您学到了什么，或者您有什么建议可以让人们不再担心优化和错过一些大的事情，或者只是在运行实验中找到平衡，同时也创造机会来寻找巨大的新机会？

Ramesh Johari: Yeah.

拉梅什·乔哈里：是的。

First of all, I'm really glad you broached the E word.

首先，我真的很高兴你提出了“E”这个词。

I was dancing around it, and I'm really glad that we talked about experiments.

我围着它跳舞，我真的很高兴我们谈论了实验。

Because yeah, one of the big lessons of this recent conversation we've been having is just, how could you possibly know that difference without doing something like experimenting?

因为是的，我们最近这次谈话的重要教训之一就是，如果不做实验之类的事情，你怎么可能知道这种差异？

Ramesh Johari: So yeah, I am a big believer in experiments.

Ramesh Johari：是的，我非常相信实验。

I mean, I'll just lay those cards on the table.

我的意思是，我会把这些牌放在桌子上。

I love working with businesses that think experiments are important to helping make good decisions.

我喜欢与那些认为实验对于帮助做出正确决策很重要的企业合作。

Ramesh Johari: Now all that said, I am also someone who feels pretty strongly about this exact issue that you're raising.

拉梅什·乔哈里：话虽如此，我也是一个对你提出的这个问题有强烈感受的人。

It's just, you can't experiment your way out of everything.

只是，你无法尝试一切事情。

Ramesh Johari: And one frame I like to give people is that although you might say you're an experiment driven business, some businesses will proclaim, "We literally test everything." What that kind of leaves aside a little bit is there's a lot of degrees of freedom in what it means to test everything.

Ramesh Johari：我喜欢给人们的一个框架是，尽管你可能会说你是一家实验驱动的企业，但有些企业会宣称，“我们确实测试了一切。”撇开一点不谈的是，测试一切的含义有很多自由度。

Ramesh Johari: Because ultimately, what's getting built and tested are choices that are made through the organizational structure, the data scientists, the PMs, the engineers, everybody's on...

Ramesh Johari：因为最终，构建和测试的内容是通过组织结构、数据科学家、项目经理、工程师，每个人都做出的选择......

Before we're running experiments, we're actually thinking about even what's worth experimenting, what designs are we coming with?

在进行实验之前，我们实际上会思考什么值得实验，我们要采用什么设计？

So that's one.

这就是其中之一。

Ramesh Johari: And the other big one is, how long do we run these experiments?

Ramesh Johari：另一个重要问题是，我们要进行这些实验多长时间？

Okay, that's a big choice.

好吧，这是一个很大的选择。

And what I generally believe, and I think there's a paper we can link to later that I'll point your readers to as well that...

我通常相信的是，我认为我们稍后可以链接到一篇论文，我也会向您的读者指出......

Not my paper, from some folks at Microsoft.

不是我的论文，来自微软的一些人。

Ramesh Johari: What I generally believe is we're risk averse on both these two dimensions, that what people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design.

Ramesh Johari：我通常认为，我们在这两个维度上都厌恶风险，人们决定在一个提倡一切实验的世界中进行测试的内容往往在设计上更加渐进。

Okay?

好的？

And we'll come back to that actually, answer the because in a second.

我们实际上会回到这个问题，稍后回答这个问题。

So that's one.

这就是其中之一。

And two is people tend to run experiments for a long time, and probably longer than they should.

二是人们倾向于进行很长时间的实验，而且可能比他们应该做的时间更长。

Ramesh Johari: Now, what do I mean by these two things?

拉梅什·乔哈里：现在，我说的这两件事是什么意思？

So what's interesting to me about this dynamic is experiments don't live in a vacuum.

因此，对我来说，这种动态的有趣之处在于实验并非存在于真空中。

Companies have incentives.

公司有激励措施。

And in companies that really go all in on experimentation, one of the things that gets wrapped up in that is the incentives around experiments.

在真正全力以赴进行实验的公司中，其中最重要的事情之一就是围绕实验的激励措施。

Because if you go all in on experiments, a common thing you'll see is data scientists get judged based on how many wins they had that quarter.

因为如果你全力以赴地进行实验，你会发现一个常见的现象是，数据科学家的评判标准是根据他们在该季度取得的胜利次数。

How do you get more wins?

如何获得更多胜利？

Ramesh Johari: Well, it's easier to get wins when you're being incremental.

Ramesh Johari：嗯，当你渐进时，更容易获得胜利。

And because it's important to have wins, you have to run them long enough to demonstrate that they're really wins.

因为取得胜利很重要，所以你必须运行足够长的时间来证明它们是真正的胜利。

You're less willing to cut something off in exchange for trying something riskier.

你不太愿意放弃某些事情来换取尝试更高风险的事情。

Ramesh Johari: So the big lesson of this Microsoft paper, it's called A/B Testing with what's called Fat Tails, which in lay terms just means you're running a business where there's potentially big opportunities out there if you look at the effects of the experiments that you run.

Ramesh Johari：这篇微软论文的重要教训是，它被称为 A/B 测试和所谓的“肥尾”，通俗地说，这意味着你正在经营一家企业，如果你看看你运行的实验的效果，那里有潜在的巨大机会。

There's a couple of lessons there about both trying a lot more stuff that's not all risk averse, and not necessarily running everything for so long.

这里有一些教训，关于尝试更多的东西，而不是完全规避风险，并且不一定要运行这么长时间的所有东西。

So really getting velocity up.

所以速度确实加快了。

Ramesh Johari: So you could see that there's a big incentive problem there, because the culture that says it's okay to fail big actually requires changing the terminology of wins.

拉梅什·乔哈里（Ramesh Johari）：所以你可以看到那里存在一个很大的激励问题，因为认为大失败没关系的文化实际上需要改变胜利的术语。

This is one of the things I hate most in A/B testing, I have to say.

我不得不说，这是我在 A/B 测试中最讨厌的事情之一。

I get where it comes from.

我知道它来自哪里。

Experimentation was never historically in science about winners and losers.

在科学史上，实验从来都不是关于胜利者和失败者的。

It'd be weird if it Ronald Fisher who's kind of the father of experimentation with his agriculture experiments talked about winners.

如果是农业实验之父罗纳德·费舍尔（Ronald Fisher）谈论胜利者，那会很奇怪。

I don't think that's necessarily how he talked about things.

我认为他不一定是这样谈论事情的。

Experimentation is always very hypothesis driven.

实验总是由假设驱动的。

It's about, what are you learning?

问题是，你在学什么？

Ramesh Johari: And that's really an important distinction because what it means is if I go with something big, risky, and it, "fails," meaning that doesn't win.

拉梅什·乔哈里（Ramesh Johari）：这确实是一个重要的区别，因为这意味着如果我选择一些大的、有风险的东西，而它“失败了”，意味着没有获胜。

Nevertheless, if I was being rigorous about what hypotheses that's testing about my business, I'm potentially learning a lot.

尽管如此，如果我严格对待正在测试我的业务的假设，我可能会学到很多东西。

Ramesh Johari: So a great example of this kind of thing is that there's an important feature of marketplaces is badging.

Ramesh Johari：此类事情的一个很好的例子是，市场的一个重要特征就是徽章。

So sometimes, it's really important to have badges on your top-rated profiles or whatever, when people are searching.

因此，有时，当人们进行搜索时，在您的顶级个人资料或其他内容上添加徽章非常重要。

Ramesh Johari: And without going too far into the details, a common finding with badges is that badges you think are going to be great actually turn out to be terrible.

Ramesh Johari：在不深入讨论细节的情况下，关于徽章的一个常见发现是，你认为很棒的徽章实际上很糟糕。

And one reason they're terrible is they focus too much attention on the badged folks, and pull too much attention away from the unbadged folks.

他们糟糕的原因之一是他们把太多的注意力集中在有徽章的人身上，而把太多的注意力从没有徽章的人身上转移开。

Ramesh Johari: And if we judge that only in terms of winners and losers, you throw the baby out with the bath water, you're like, "Well that badging idea was terrible.

拉梅什·乔哈里：如果我们只根据赢家和输家来判断，你把婴儿和洗澡水一起倒掉，你会说，“好吧，徽章的想法很糟糕。

So ditch that, no badges."

所以放弃这个吧，没有徽章。”

Ramesh Johari: But that's not what it's telling you.

Ramesh Johari：但这不是它告诉你的。

It's teaching you something about how inventory is being reallocated, how attention's being redirected through the badges.

它教您如何重新分配库存，如何通过徽章重新引导注意力。

And you really want to think not in terms of winning and losing, but learning.

你真正想要思考的不是输赢，而是学习。

Ramesh Johari: So learning is a win.

Ramesh Johari：所以学习是一种胜利。

And I feel that that's a cultural thing fundamentally.

我觉得这从根本上来说是一个文化问题。

It's very hard to somehow attach dollars and cents at the top to data scientists running experiments that fail, but learn.

很难以某种方式将金钱和美分放在最高层，让数据科学家进行失败但能学习的实验。

And ultimately, I think getting into that space where you experiment more, meaning you don't run all your experiments for quite as long and you accept the willingness to try experiments that are into the tails where you might fail bigger is a cultural thing.

最终，我认为进入一个你可以进行更多实验的空间，这意味着你不会将所有实验进行很长时间，并且你愿意尝试可能会失败的尾部实验，这是一种文化。

It's about saying that, "We're allowing that to be part of our social contract with our data scientists," or actually our employee contract with our data scientists, that not everything is just about how many launches you had and how many wins there were.

它是说，“我们允许它成为我们与数据科学家的社会契约的一部分”，或者实际上是我们与数据科学家的员工合同的一部分，并不是所有的事情都只是关于你有多少次发布和有多少次胜利。

Ramesh Johari: It's okay to say, "That's how I want to use experimentation," but if you're going to use it that way, then I would say don't be a, "We experiment everything," business.

Ramesh Johari：可以说，“这就是我想要使用实验的方式”，但如果你要那样使用它，那么我会说不要成为“我们尝试一切”的生意。

Because then I think you need some other way to deal with these big changes that teach the whole company a lot, but maybe can't fall into the incentives you've created for your data scientists.

因为我认为你需要一些其他方法来应对这些重大变化，这些变化可以让整个公司受益匪浅，但可能无法融入你为数据科学家创建的激励措施中。

Lenny: This badging example is, I don't know if you're referring to the Airbnb example, but I actually led the launch of Superhost at Airbnb, which is the ultimate badge on Airbnb.

Lenny：这个徽章的例子是，我不知道你是否指的是 Airbnb 的例子，但我实际上领导了 Airbnb 超级房东的推出，这是 Airbnb 上的终极徽章。

And there was a lot of concern from the data team that it would destroy the marketplace, because they've built, as you've described, this very well-crafted ranking algorithm, with just a prediction of exactly as you described, which listings that guest is most likely to book and be successful booking.

数据团队非常担心这会破坏市场，因为正如您所描述的，他们已经构建了这个精心设计的排名算法，只需准确预测您所描述的客人最有可能预订并成功预订的房源。

And then we're about to throw a badge on random listings in the results.

然后我们将在结果中的随机列表上添加一个徽章。

And so this one data scientist on our team's like, "No, we can't do this.

因此，我们团队中的一位数据科学家会说：“不，我们不能这样做。

This is insane.

这太疯狂了。

We're going to destroy it all."

我们要摧毁这一切。”

Lenny: And we still went ahead with it.

莱尼：我们仍然继续这样做。

We ran an experiment showing the badge to some people and some not, actually, it was no impact at all.

我们进行了一项实验，向一些人展示了徽章，而另一些人则没有，实际上，它根本没有影响。

Which Superhost itself had no impact at all on the business as far as we could tell initially, which is also bittersweet because it felt like, "Why did we even work on this thing?" There was a slight benefit where a host felt better, they felt more satisfied with being a host, but I went exactly through what you described, so that's pretty funny.

据我们最初所知，哪个超赞房东本身对业务没有任何影响，这也是苦乐参半，因为感觉就像是，“我们为什么还要做这件事？”有一个轻微的好处，让主持人感觉更好，他们对作为主持人感到更满意，但我完全按照你的描述进行了操作，所以这很有趣。

Ramesh Johari: Without necessarily going into the weeds on the data science of Superhost, I think there's a lot wrapped up in what you said.

Ramesh Johari：不必深入了解超级主机的数据科学，我认为您所说的内容包含了很多内容。

I guess another thing I'll say is that I'm a big believer that you don't throw your understanding of the business out the window when you process experiment results.

我想我要说的另一件事是，我坚信在处理实验结果时不要抛弃对业务的理解。

And it's partly, I guess what I mean by this is data science is really about accumulation of evidence.

我想我的部分意思是数据科学实际上是关于证据的积累。

It's never about one finding in isolation.

这绝不是孤立的一项发现。

And so another kind of trap I think is to sometimes say, "Well, I hit stat sig on my A/B test, green light.

因此，我认为另一种陷阱有时会说：“好吧，我在 A/B 测试中达到了 stat sig，绿灯亮了。

It's all go."

一切都过去了。”

Ramesh Johari: And I think you had Ronny Kohavi on your show, and he made a similar point that there are different levels of evidence, and just having an outlier A/B test that goes against everything you believe about your business doesn't mean that you somehow have controverted all your knowledge.

Ramesh Johari：我认为罗尼·科哈维（Ronny Kohavi）出现在你的节目中，他也提出了类似的观点，即存在不同程度的证据，仅仅进行一个与你对你的业务所相信的一切相悖的异常值 A/B 测试并不意味着你以某种方式反驳了你所有的知识。

And I think that's one side of it.

我认为这是一方面。

Ramesh Johari: The other thing is you can't always measure everything that's important that's needed to really develop a full sense.

Ramesh Johari：另一件事是，你无法总是衡量真正发展全面意识所需的一切重要因素。

So with Superhost, one of the things that's hard to measure is the long-term impact of Superhost.

因此，对于超赞房东来说，难以衡量的事情之一就是超赞房东的长期影响。

Because in the short run, Superhost causes a rebalancing of inventory.

因为从短期来看，超赞房东会导致库存重新平衡。

There's going to be winners and losers.

将会有赢家和输家。

Part of Superhost is actually about retaining hosts that get the badge over a longer period of time.

超级房东的一部分实际上是为了长期保留获得徽章的房东。

Recognizing that hypothesis actually says something about maybe how long the experiment needs to be run or what kinds of data analyses need to be done.

认识到这一假设实际上说明了实验可能需要运行多长时间或需要进行哪些类型的数据分析。

Ramesh Johari: And in the end, if you can't do that, you can't run it long enough, or you can't do that data analysis due to sparsity of data or lack of data to address the question, it matters what you bring to the table.

Ramesh Johari：最后，如果你做不到这一点，你就无法运行足够长的时间，或者由于数据稀疏或缺乏解决问题的数据而无法进行数据分析，那么你带来的内容就很重要。

What are your beliefs about that?

你对此有何看法？

Ramesh Johari: So what I like to tell people to do there is I like to push people to be what's called quantified rather than data-driven, which is, okay fine, some things we can't measure.

Ramesh Johari：所以我喜欢告诉人们要做的就是推动人们进行所谓的量化而不是数据驱动，好吧，有些事情我们无法衡量。

But maybe you've got a leadership team with different beliefs about what they think the retention value of Superhost is going to be, and they might be all over the place.

但也许您的领导团队对于超赞房东的保留价值持有不同的信念，而且他们可能到处都是。

Ramesh Johari: You can process your experiment results in the context of these competing beliefs.

Ramesh Johari：你可以在这些相互竞争的信念的背景下处理你的实验结果。

It's almost like a prediction market kind of a thing.

这几乎就像一个预测市场之类的东西。

And start asking, "Well okay, if this is what we believe about our business, this is what the data is telling us out of the experiment, let's put those two together and ask, is this enough for us to make the bet that we're still going to go with it?" Even though maybe that short-term test you ran was flat.

然后开始问，“好吧，如果这就是我们对我们业务的信念，这就是实验中数据告诉我们的，让我们把这两者放在一起，问一下，这是否足以让我们打赌我们仍然会继续这样做？”即使您进行的短期测试可能表现平平。

Lenny: That's actually exactly how I think of Superhost looking back.

莱尼：这实际上正是我对超赞房东的看法。

It was a great idea.

这是一个好主意。

I'm really happy.

我真的很高兴。

I can't even imagine Airbnb without that, even though there's no evidence, at least initially, that it made any impact.

我什至无法想象 Airbnb 没有它，尽管没有证据表明它产生了任何影响，至少在最初是这样。

I'm guessing they looked at it again, and maybe there's something that came out of it.

我猜他们又看了一遍，也许从中发现了什么。

But even if it had no impact, it just feels like it made the marketplace better.

但即使它没有产生任何影响，也只是感觉它让市场变得更好了。

And that was a big learning for me.

这对我来说是一个很大的学习。

It doesn't need to always drive a metric that you can measure.

它不需要总是驱动您可以测量的指标。

There's just like, this is the way it should work.

就像，这就是它应该工作的方式。

Ramesh Johari: So one of the reasons the thing you said happens is because marketplaces are a little bit like a game of whac-a-mole, okay?

Ramesh Johari：所以你所说的事情发生的原因之一是因为市场有点像打地鼠游戏，好吗？

And what I mean by that is, so narrowly in the context of Superhost, because you're redirecting attention to some hosts at the expense of...

我的意思是，在超赞房东的背景下是如此狭隘，因为你将注意力转移到某些房东身上，而牺牲了……

It's not even obvious if bookings can really go up.

预订量是否真的能增加还不清楚。

Maybe you get lucky and maybe you get a bunch more bookings.

也许你很幸运，也许你会得到更多的预订。

One reason you probably wouldn't expect that in the first place is there's only a limited number of Superhosts.

您可能一开始就没有想到的一个原因是超级房东的数量有限。

How many more bookings are they going to be absorbing because of all this extra attention?

由于这些额外的关注，他们还会吸引多少预订？

And you're taking attention away from other people.

而且你正在转移其他人的注意力。

Without doing any data analysis, my prior would've been that booking should probably go down.

如果不进行任何数据分析，我的想法是预订量可能会下降。

Ramesh Johari: And one example that I came across with one of the companies I worked with that I love is we were working together over a period of time, and in a month, we looked at some of the data and it suggested that our new supply side was having a pretty bad experience.

Ramesh Johari：我在与我合作的一家我喜欢的公司中遇到的一个例子是，我们合作了一段时间，在一个月内，我们查看了一些数据，结果表明我们的新供应方经历了非常糟糕的经历。

Say, "We got to do something about this."

说：“我们必须对此做点什么。”

Ramesh Johari: So what we decided to do is build some custom bespoke features that were really going to direct them to more experienced folks on the other side of the market.

Ramesh Johari：所以我们决定做的是构建一些定制功能，这些功能实际上将它们引导给市场另一边更有经验的人。

Good.

好的。

And then lo and behold, pretty soon those metrics start to look better.

然后你瞧，很快这些指标就开始看起来更好了。

But then we're looking at it, we're like, "Wait a second.

但当我们看到它时，我们会想，“等一下。

Now the existing folks on the other side are having a worse experience."

现在，另一边现有的人们正在经历更糟糕的经历。”

Ramesh Johari: So you kind of whiplash around.

Ramesh Johari：所以你有点胡思乱想。

You're like, "Wait a second, we better do something about that." So we take them, we try to match them up with the more experienced folks.

你会说，“等一下，我们最好做点什么。”所以我们接受他们，尝试将他们与更有经验的人相匹配。

And now suddenly a month after that you're like, "Wait a second." And your metrics just keep moving around.

一个月后，你突然会说：“等一下。”而且你的指标一直在变化。

Ramesh Johari: And that's because the whac-a-mole game here is ultimately, a lot of marketplace management is moving attention and inventory around.

Ramesh Johari：那是因为打地鼠游戏最终是，许多市场管理人员正在转移注意力和库存。

Sometimes you get lucky and you really expand the pie for everybody.

有时你很幸运，你真的为每个人都扩大了蛋糕。

But I think Servaes Tholen, who was CFO at Upwork that I got to know there and then went to Thumbtack later, he had this line when he came to visit our class that I love, which is, "You have to recognize when you run marketplaces that many of the changes that are most consequential create winners and losers.

但我认为 Servaes Tholen（我在那里认识的 Upwork 的首席财务官，后来去了 Thumbtack）在参观我们的班级时说了我喜欢的这句话，那就是：“当你经营市场时，你必须认识到，许多最重要的变化会创造赢家和输家。

And rolling with those changes is about recognizing whether the winners you've created are more important to your business view than the losers you've created in the process." And it's a hard reality, because nobody likes to articulate the idea that a feature change is hurting some of the people in your marketplace.

伴随这些变化意味着要认识到，对你的商业观点来说，你所创造的赢家是否比你在此过程中创造的输家更重要。”这是一个残酷的现实，因为没有人愿意明确表达功能更改正在伤害市场中的某些人的想法。

But because of this fundamental constraint baked into how marketplaces work, many of the things that we would choose to do and the reallocation they create can't necessarily create observed high expanding wins in the short run.

但由于市场运作方式存在这一基本限制，我们选择做的许多事情以及它们所创造的重新分配不一定能在短期内创造可观察到的高扩张性胜利。

You're often making bets that that's where you're headed, partly through the reallocation that you're doing right now.

你经常押注这就是你的前进方向，部分是通过你现在正在进行的重新分配来实现的。

Ramesh Johari: And so I think that's interesting about Superhost to me is that partly points to thinking about, what's the objective you would've defined, the metric you would've defined in the short run that captures this idea of a trade-off?

Ramesh Johari：所以我认为超赞房东对我来说很有趣的一点在于，它部分地指向思考，你会定义的目标是什么，你会在短期内定义的衡量标准来捕捉这种权衡的想法？

Lenny: That's a great way to think about it.

莱尼：这是一个很好的思考方式。

I wanted to come back to this idea you're sharing of maybe you should run experiments more quickly, not wait for stat sig, have a culture of learning versus impact.

我想回到你分享的这个想法，也许你应该更快地进行实验，而不是等待统计数据，建立一种学习与影响的文化。

In practice, it's very difficult, because people are measured by impact.

实际上，这是非常困难的，因为人们是通过影响力来衡量的。

There's performance reviews, there's promotions, there's how much impact did this team drive, are going to look at their experiment results?

有绩效评估，有晋升，这个团队带来了多大影响，要看看他们的实验结果吗？

You've worked with a lot of marketplace companies, a lot of different companies.

您曾与很多市场公司、很多不同的公司合作过。

Is there anything you've seen about something you could do to help the company shift and actually work this way, while also recognizing success, and who's doing great, who's not, which team's driving impact, who's not?

您是否看到过可以做些什么来帮助公司转变并真正以这种方式工作，同时也认识到成功，以及谁做得很好，谁做得不好，哪个团队具有推动影响力，谁没有？

Ramesh Johari: Interestingly, it's actually an active area of research for me now.

Ramesh Johari：有趣的是，这实际上是我现在的一个活跃的研究领域。

What I mean by active area of research is I care a lot about the incentives that we create for data science through how we set up reward mechanisms.

我所说的活跃研究领域是指我非常关心我们通过如何建立奖励机制为数据科学创造的激励措施。

So there's a couple things I think that could be helpful, that are maybe there may be a little bit less about...

所以我认为有几件事可能会有所帮助，也许可能会少一点……

Maybe I'm not going to directly answer the question you ask, because I think that's a hard one, right?

也许我不会直接回答你提出的问题，因为我认为这很难，对吧？

I think I recognize that measurement on impact is critical.

我想我认识到衡量影响力至关重要。

Well, let me answer that actually from the most obvious way first.

好吧，让我首先从最明显的方式回答这个问题。

I think there's a cultural issue here that's really critical.

我认为这里存在一个非常关键的文化问题。

Ramesh Johari: One of the things I often find is that my PhD students, our PhD students here often go off and get great data scientist jobs.

Ramesh Johari：我经常发现的一件事是，我的博士生，我们这里的博士生经常离开并找到很棒的数据科学家工作。

And in one sense, they're doing amazing stuff.

从某种意义上说，他们正在做一些了不起的事情。

They apply really technically sophisticated methods.

他们采用了技术上非常复杂的方法。

But when I look at the problems they're working on, they're often more at the margins of the business than they should be.

但当我审视他们正在解决的问题时，他们往往更多地处于业务的边缘，而不是应有的位置。

Ramesh Johari: And it's a cultural thing.

Ramesh Johari：这是一个文化问题。

It's basically because if you're measured narrowly on impact and that's all anyone sees around you, then it's very hard to engage with the creative aspect of business change and the strategic aspects of business change.

这基本上是因为，如果你对影响力的衡量很狭隘，而这就是你周围任何人所看到的，那么就很难参与业务变革的创造性方面和业务变革的战略方面。

Ramesh Johari: So the cultural aspect there is, I think it's partly incumbent on the leaders to expect something more of their data scientists.

Ramesh Johari：所以从文化角度来看，我认为领导者有责任对他们的数据科学家寄予更多期望。

And what I mean by expect more is that you expect them to do more than deliver narrowly defined, statistically rigorous results to you in their reports.

我所说的期望更多是指你期望他们做的不仅仅是在报告中向你提供狭隘定义的、统计上严格的结果。

You're actually expecting them to talk also about what they're learning about the business in the process.

您实际上希望他们也谈论他们在此过程中学到的有关业务的知识。

So where that's headed is there's this concept of being hypothesis driven, which is like the technical phrase.

因此，这就是假设驱动的概念，就像技术术语一样。

What does that mean?

这意味着什么？

Again, in a more lay sense.

再次，从更通俗的意义上来说。

Ramesh Johari: What it means is tests aren't going to be defined only in terms of winners and losers, that each test should also say something about what will we learn about a business flow, a funnel, preferences of the guests, preferences of the hosts.

Ramesh Johari：这意味着测试不会仅根据赢家和输家来定义，每个测试还应该说明我们将了解业务流程、漏斗、客人的偏好和主人的偏好。

What will we learn about their demand elasticity if we're changing prices around?

如果我们改变价格，我们会对他们的需求弹性了解什么？

These kinds of things.

诸如此类的事情。

So it's possible to articulate in an experiment doc, a launch doc, what are the hypotheses that are being tested?

因此，可以在实验文档、启动文档中阐明正在测试的假设是什么？

So that's one thing I would say is just culturally, setting the norms that learning is part of the discourse, and it's expected actually I think is important.

所以我想说的一件事只是在文化上，设定学习是话语的一部分的规范，而且我认为这实际上是很重要的。

Ramesh Johari: But the other thing I would say that's maybe a little bit more about programmatically, what could a data science platform team do?

Ramesh Johari：但我想说的另一件事可能更多的是关于编程，数据科学平台团队可以做什么？

A funny thing about experiments is that we throw past learning away effectively.

实验的一个有趣之处在于我们有效地抛弃了过去的学习成果。

And this is just an artifact of how we analyze experiments, that the statistical methods used typically, P-values, confidence intervals, these fall into a branch of statistics known as frequentist statistics.

这只是我们分析实验的方式的产物，通常使用的统计方法、P 值、置信区间，这些都属于统计的一个分支，称为频率统计。

And the idea behind frequentist statistics without being overly technical is just I let the data speak for itself.

频率统计背后的想法又不过度技术化，那就是我让数据自己说话。

There's no beliefs brought to the table about where that data came from.

没有人相信这些数据来自哪里。

Ramesh Johari: But if you think about this in a company, in A/B testing a company, it's a weird thing, right?

Ramesh Johari：但是如果你在一家公司中考虑这一点，在对一家公司进行 A/B 测试时，这是一件很奇怪的事情，对吧？

Because I might've run 1,000 A/B tests in the past on this exact same button, or call to action, or color, and now I am going to completely ignore that and focus only on this.

因为我过去可能已经在同一个按钮、号召性用语或颜色上运行了 1,000 次 A/B 测试，而现在我将完全忽略这些，只关注这一点。

Ramesh Johari: So there's ways to take the past into account, to build what's called a prior belief before I run an experiment, and now take the data from the experiment, connect it with the prior, to come up with a conclusion of, "Okay, in light of the past plus this experiment, what's it telling me about the future?" And that falls broadly under the category of what's called Bayesian A/B testing.

拉梅什·乔哈里：所以有一些方法可以考虑过去，在进行实验之前建立所谓的先验信念，然后从实验中获取数据，将其与先验联系起来，得出这样的结论：“好吧，根据过去加上这个实验，它告诉我关于未来的什么信息？”这大致属于贝叶斯 A/B 测试的范畴。

Ramesh Johari: So that's one of the things I think can help culturally, weirdly.

拉梅什·乔哈里：所以这是我认为可以在文化上有所帮助的事情之一，很奇怪。

It's a super technical thing, but I think it can help culturally, because what it's doing is it's now rewarding people for contributing information to that prior.

这是一个超级技术性的事情，但我认为它可以在文化上有所帮助，因为它现在正在奖励那些为之前贡献信息的人们。

And I think it then becomes possible to say, "Your experiment that failed actually moved our prior." And that's an important thing, because by doing so, you're now altering how we're going to think about this flow or this pricing plan in all future experiments.

我认为这样就可以说：“你失败的实验实际上改变了我们的先验。”这是一件重要的事情，因为通过这样做，你现在正在改变我们在未来所有实验中思考这个流程或这个定价计划的方式。

Ramesh Johari: So there's an information positive externality, positive network effect that's generated for the rest of your business if I can somehow encode what you learned into the analysis of future experiments.

Ramesh Johari：因此，如果我能以某种方式将您学到的知识编码到未来实验的分析中，就会为您的其他业务产生信息正外部性和正网络效应。

So this is one thing.

所以这是一回事。

There's strong connection between the culture and incentives of A/B testing and the ability to actually incorporate past learning into these prior beliefs.

A/B 测试的文化和激励措施与将过去的学习实际融入这些先前信念的能力之间存在着密切的联系。

Lenny: I love that you're doing research in this area.

莱尼：我喜欢你在这个领域进行研究。

We should bring you back when you've completed it and have the ultimate answer for everyone to change how they operate.

当你完成它后，我们应该把你带回来，并为每个人提供最终的答案，以改变他们的运作方式。

Ramesh Johari: Yeah, one of the great things about professors is we never complete anything and never have ultimate answers.

Ramesh Johari：是的，教授最伟大的事情之一就是我们永远不会完成任何事情，也永远不会有最终的答案。

Lenny: This touches on a really interesting concept that you shared with me around how, just learning isn't free.

莱尼：这涉及到你与我分享的一个非常有趣的概念，即学习并不是免费的。

People think that they could just learn a bunch of stuff and there's not a cost to it.

人们认为他们可以学到很多东西，而且不需要任何成本。

I'd love for you to just chat a bit about what that means.

我希望您能谈谈这意味着什么。

Ramesh Johari: Let me start with an anecdote, that I just absolutely love this anecdote.

拉梅什·乔哈里：让我从一件轶事开始，我非常喜欢这个轶事。

I use it every year in class.

我每年在课堂上都会使用它。

So I was talking to a real estate platform, and they had a marketing data science manager who's basically responsible, as many marketing managers are, for allocation of ad spend across different channels.

因此，我正在与一个房地产平台交谈，他们有一位营销数据科学经理，与许多营销经理一样，他基本上负责跨不同渠道的广告支出分配。

Ramesh Johari: And what they discovered had happened at the end of the year is in one hand, the team had done great, but the manager had held out some subset of arriving visitors, not showing them any of the innovations they were making.

Ramesh Johari：他们在年底发现的情况是，一方面，团队做得很好，但经理向部分到达的访客展示了部分内容，没有向他们展示他们正在做出的任何创新。

Ramesh Johari: Yeah, exactly.

拉梅什·乔哈里：是的，完全正确。

What's called a holdout group in experimentation.

实验中所谓的坚持小组。

And one thing about this holdout is it wasn't authorized.

这种抵制的一件事是它没有得到授权。

That's not the way things are supposed to work.

事情不是应该这样进行的。

They've got their ad spend, allocate out your ad spend, great.

他们有自己的广告支出，分配你的广告支出，太好了。

So at the end of the year, they looked at the hole out and they're like, "Wow, that cost us a couple million dollars, something in that range, and it's not a trivial amount of money.

所以到了年底，他们看着这个洞，然后想，“哇，这花了我们几百万美元，在这个范围内，而且这不是一笔小数目。

What's the deal?

这是怎么回事？

What were you thinking?" Basically.

你在想什么？”基本上。

And of course the answer was, "Well, I get that I cost you that much, but number one, now you know what my team's worth.

当然，答案是：“好吧，我知道我花了你那么多钱，但第一，现在你知道我的团队值多少钱了。

And number two, you would never have had that answer unless I'd done that on my own."

第二，除非我自己做，否则你永远不会得到这个答案。”

Ramesh Johari: Now, why is that so powerful?

拉梅什·乔哈里：现在，为什么它如此强大？

I think what I find so interesting about experiments is that when you don't know something, it seems not even a question that you would allocate some of your samples to all options, right?

我认为实验的有趣之处在于，当你不知道某件事时，将一些样本分配给所有选项似乎都不是问题，对吧？

Treatment and control.

治疗和控制。

I have two different ways of doing something.

我有两种不同的做事方式。

I don't know which one's better, so of course I'll give some samples to each.

我不知道哪个更好，所以我当然会给每个提供一些示例。

After the fact you're like, "Treatment was better.

事后你会说，“治疗效果更好了。

What the heck were we thinking?

我们到底在想什么？

Why'd we give all those samples to control?

为什么我们要提供所有这些样本来进行控制？

That doesn't make any sense now." There's this great Seinfeld clip where he mentions getting a bill at the end of a large luxurious meal, and people stare at the bill like, "We're not hungry now.

现在这没有任何意义了。”有一段很棒的《宋飞正传》片段，他提到在一顿丰盛的大餐结束后收到一张账单，人们盯着账单，好像在说：“我们现在不饿了。

Why'd we order all this food?" So it's the same thing.

我们为什么要点这么多食物？”所以这是同样的事情。

I mean, you know treatment's better now.

我的意思是，你知道现在治疗效果更好了。

Why'd you waste all those samples on control?

为什么你要把所有这些样品浪费在控制上？

Ramesh Johari: And I think that is such a powerful observation that you have to put yourself in the frame of reference of when you didn't have the answer.

拉梅什·乔哈里：我认为这是一个非常有力的观察，当你没有答案时，你必须将自己置于参考框架中。

And at that moment, what you're essentially saying to yourself is that it's worth paying to learn the answer.

在那一刻，你本质上是在对自己说，花钱去了解答案是值得的。

I think it sounds obvious the way we're saying it now, or this anecdote of the marketing manager and the holdout sounds obvious.

我认为这听起来很明显，就像我们现在所说的那样，或者营销经理和坚持不懈的人的轶事听起来很明显。

What's culturally not baked in I think is that idea.

我认为文化上没有融入这个想法。

And the reason I say it's not culturally baked in, by the way, is because of the language of winners and losers.

顺便说一句，我之所以说它没有融入文化，是因为胜利者和失败者的语言。

Because if we use that language, we're implicitly saying is that we wasted time when we ran an A/B test on loser.

因为如果我们使用这种语言，我们就含蓄地说，当我们对失败者进行 A/B 测试时，我们浪费了时间。

If I reward you for shipping winners, then what I'm really telling you is all the time that you spent testing out failures was wasted time.

如果我奖励你交付获胜者，那么我真正要告诉你的是，你花在测试失败上的所有时间都是浪费时间。

Ramesh Johari: And I think, of course, you don't want to keep data scientists around who regularly are just generating failures.

Ramesh Johari：我认为，当然，你不想让经常产生失败的数据科学家留在身边。

That's not my point.

这不是我的重点。

Ramesh Johari: But my point is there's a disconnect there.

拉梅什·乔哈里：但我的观点是，其中存在脱节。

On one hand, we can all look at the story of this marketing manager and chuckle at it.

一方面，我们都可以看看这位营销经理的故事并嘲笑它。

And yet, every day we're instantiating language and processes that are reinforcing that same theme, which is essentially trying to say to you, "If you're wasting samples on things that don't ultimately end up being a winner, then the act of doing so is a failure."

然而，我们每天都在实例化强化同一主题的语言和流程，这本质上是想对你说：“如果你在最终不会成为赢家的事情上浪费样本，那么这样做的行为就是失败的。”

Ramesh Johari: So I really feel that that idea that you have to pay to learn, again, it's a cultural thing, but it's also an education issue for businesses are populated by people of all stripes.

拉梅什·乔哈里（Ramesh Johari）：所以我真的觉得，必须付费才能学习的想法，再次强调，这是一种文化，但对于各行各业的人来说，这也是企业的教育问题。

Not everybody comes from a data science or experimentation background.

并非每个人都具有数据科学或实验背景。

And this idea that learning is costly is not natural, actually.

事实上，这种认为学习成本高昂的想法并不自然。

It's not natural as a matter of human nature.

从人性的角度来看，这是不自然的。

It's certainly not natural as a matter of running a business.

对于经营企业来说，这当然是不自然的。

Lenny: I love that example of the real estate platform where it's very viscerally, clearly cost.

莱尼：我喜欢房地产平台的例子，它的成本非常明显。

They lost because they didn't roll out experiments to this group for a long time.

他们输了，因为他们很长时间没有对这个群体进行实验。

Such a good example of this idea in action.

这是这个想法付诸实践的一个很好的例子。

Lenny: You mentioned star ratings.

莱尼：你提到了星级评定。

I know you spent a lot of time on designing rating systems.

我知道您花了很多时间来设计评级系统。

Sorry, I didn't mean to imply star ratings.

抱歉，我无意暗示星级。

That's just one implementation.

这只是一种实现。

Rating systems in general.

一般评级系统。

Lenny: So maybe just to keep it focused, say a marketplace founder is trying to decide and design how they do ratings, and reviews, and things like that.

莱尼：所以也许只是为了保持重点，假设市场创始人正在尝试决定和设计他们如何进行评级、评论和类似的事情。

What's a couple pieces of advice you'd give them for how to do this correctly?

关于如何正确地做到这一点，您会给他们一些什么建议？

And is there a model marketplace you'd point them to like, "These guys really do it really well"?

有没有一个模型市场可以让他们喜欢，“这些人真的做得很好”？

And I know it's super specific based on the marketplace, but is there one just like, "They really nailed it"?

我知道它是基于市场的超级具体的，但是有没有一个就像“他们真的做到了”？

Ramesh Johari: Oh man, that's a tough one.

拉梅什·乔哈里（Ramesh Johari）：天哪，这真是太难了。

I think I'll answer the second part first.

我想我会先回答第二部分。

I don't feel like anyone's really nailed this.

我觉得没有人真正做到了这一点。

Yeah, I think there's a lot of innovation that's happened, but I think fundamentally, we're still playing with the same kind of tools that we had when eBay and Amazon first started thinking about how to do rating systems ages ago.

是的，我认为已经发生了很多创新，但我认为从根本上来说，我们仍在使用 eBay 和亚马逊多年前首次开始考虑如何建立评级系统时所使用的同类工具。

Ramesh Johari: And part of the reason we haven't nailed it is because there's a lot of dynamics in play that lead to what's called rating inflation, where if you look at ratings over time in the marketplace...

Ramesh Johari：我们没有解决这个问题的部分原因是，有很多动态因素在起作用，导致所谓的评级膨胀，如果你看看市场上一段时间内的评级......

One of my colleagues, John Horton, who was a professor at MIT and has worked very closely with Upwork, we worked together when I was at oDesk, he was the staff economist there.

我的一位同事约翰·霍顿 (John Horton) 是麻省理工学院的教授，与 Upwork 合作非常密切，我在 oDesk 时我们就一起工作过，他是那里的专职经济学家。

He's written a couple of really nice papers with this empirical phenomenon that over time, you see the median rating inflating, let's say on marketplaces like oDesk, like Uber, like any of these.

他写了几篇非常好的论文，阐述了这种经验现象：随着时间的推移，你会看到评分中值不断膨胀，比如在 oDesk、Uber 等市场上。

Ramesh Johari: And there's a lot of reasons for this, but one of them is just that there's a reciprocity issue, which is it's effectively, from your perspective, it's kind of costless if someone says to you, "Hey, please leave me a nice rating." And if you're seeing them or you're interacting with them, most people don't want to be mean.

拉梅什·乔哈里：造成这种情况的原因有很多，但其中一个就是存在互惠问题，从你的角度来看，如果有人对你说，“嘿，请给我一个好的评价”，这是没有成本的。如果你看到他们或与他们互动，大多数人都不想表现得很刻薄。

So that happens.

所以就会发生这种情况。

Ramesh Johari: But there's another aspect of it, which is norming.

拉梅什·乔哈里：但还有另一个方面，那就是规范。

As the ratings in the marketplace go up, they get normed, so that now you're in a condition, you're like, "A four star rating.

随着市场评级的上升，它们会变得规范，所以现在你处于一种状态，你就像，“四星级评级。

I'm really screwing this person over." Whereas maybe when the marketplace started, you didn't think that.

我真是把这个人搞砸了。”然而，也许当市场开始时，你并没有这么想。

Ramesh Johari: So definitely one thing that we worked on in our research was to think about renorming, the meaning of some of these labels.

Ramesh Johari：因此，我们在研究中所做的一件事肯定是考虑重新规范，即其中一些标签的含义。

And renorming could mean something like rather than the star ratings just being poor to excellent, the top rating has actually exceeded expectations.

重新调整可能意味着星级评级不仅仅是从差到优秀，最高评级实际上超出了预期。

You could go one step further and you could say, "How did this compare to this experience you had in the past that you rated really highly?" And Airbnb had something like this in place, where they would actually ask you to compare, or ask you questions about expectations.

你可以更进一步，你可以说，“这与你过去评价很高的经历相比如何？” Airbnb 也有类似的东西，他们实际上会要求你进行比较，或者问你有关期望的问题。

Ramesh Johari: I find that that's really valuable because it's easier for people to say, "That was good but didn't exceed my expectations.

Ramesh Johari：我发现这真的很有价值，因为人们更容易说：“这很好，但没有超出我的预期。

That was good, but definitely not better than this amazing stay I had two months ago," than it is to say, "Well, I'm going to ding this person and give them four stars." So that's one issue.

这很好，但绝对不比我两个月前的这次令人惊叹的住宿更好，”而不是说，“好吧，我要批评这个人并给他们四颗星。”所以这是一个问题。

Ramesh Johari: And I think another thing I want to point out for any marketplace founder is that something you want to be really careful about is the concept of averaging and whether are the implications of averaging.

Ramesh Johari：我想我想向任何市场创始人指出的另一件事是，你需要真正小心的是平均的概念以及平均的含义。

And that's because a default for many marketplaces is to just average the ratings that people get.

这是因为许多市场的默认设置只是对人们获得的评分进行平均。

It feels very natural, right?

感觉很自然，对吧？

Lenny's got five ratings, let me average them.

莱尼有五个评分，让我对它们进行平均。

Ramesh Johari: And that actually has some pretty important distributional consequences for the marketplace.

Ramesh Johari：这实际上对市场产生了一些非常重要的分配后果。

Distributional in the sense of who wins, who loses.

分配是指谁赢谁输。

And that's because if you're averaging and you're really established on a platform, think of a restaurant on Yelp with 10,000 reviews, it's irrelevant what the next review is.

这是因为，如果你进行平均并且你确实在一个平台上建立了基础，想象一下 Yelp 上有 10,000 条评论的餐厅，那么下一条评论是什么就无关紧要了。

It doesn't matter.

没关系。

Nothing's moving it at that point.

那时没有什么能动它。

Ramesh Johari: If you're new and you break into that market, and your first review is negative, you might be completely screwed.

Ramesh Johari：如果你是新人，进入这个市场，而你的第一次评论是负面的，那么你可能会彻底完蛋。

In fact, there's some early work on eBay that showed that if your first rating's negative, that could actually immediately cause an 8% hit on your immediate expected revenue, say nothing of long-term consequences.

事实上，eBay 上的一些早期研究表明，如果您的第一次评级为负面，实际上可能会立即导致您的直接预期收入下降 8%，更不用说长期后果了。

Subsequent work has found that that's a significant indicator of potential exit from the platform, just because now it's very hard to find work.

随后的研究发现，这是可能退出该平台的一个重要指标，因为现在很难找到工作。

And some platforms do things like maybe they won't show your ratings until you've accumulated a few.

有些平台会做一些事情，比如在您积累了一些评分之前，他们可能不会显示您的评分。

Ramesh Johari: But in the end, this kind of distributional fairness aspect of averaging is pretty significant.

Ramesh Johari：但最终，平均的这种分配公平性方面非常重要。

And one of the recent papers that we've written is trying to get platforms to think a little bit about that.

我们最近写的一篇论文试图让平台对此进行一些思考。

There's ways to address that interestingly, through the same concept of a prior.

有一些方法可以通过相同的先验概念来有趣地解决这个问题。

And the prior basically says hey, if someone comes into the marketplace and instead of averaging them, I average them together with a prior belief, then maybe what that prior belief does, it says, "Yeah, you got one negative rating, but maybe you got a little bit unlucky," and maybe my prior belief is something which actually pulls your rating up a little bit and allows me to still have you alongside others in the marketplace to give you a chance at getting work, getting rides, etc.

前面的基本上是说，嘿，如果有人进入市场，我不是对他们进行平均，而是将他们与先前的信念一起平均，那么也许先前的信念会做什么，它说，“是的，你得到了一个负面评价，但也许你有点不走运，”也许我先前的信念实际上会稍微拉高你的评级，并允许我仍然让你和市场上的其他人一起，让你有机会找到工作，搭车等。

Ramesh Johari: So I believe pretty strongly in this kind of distributional fairness element of designing rating systems.

Ramesh Johari：所以我非常相信设计评级系统时的这种分配公平元素。

I think it's been understudied.

我认为这已经得到了充分研究。

And I'll say in general actually, I think rating systems are understudied, which to me is astonishing.

实际上，我想说的是，我认为评级系统尚未得到充分研究，这对我来说是令人惊讶的。

Because the biggest change from those Agoras and Trajan's Market elements of those kinds of markets, to me the biggest change is that we get to see what happened with our matches.

因为这些市场的 Agoras 和 Trajan's Market 元素的最大变化对我来说最大的变化是我们可以看到我们的比赛发生了什么。

Ramesh Johari: So as a data scientist working on marketplaces, I feel like it's incredible that more of us don't spend our time thinking about what we're learning from the matches, and what these rating systems are telling us, and what the impact of that is on who wins and who loses in these markets, kind of thinking about the social implications of these things.

Ramesh Johari：作为一名在市场上工作的数据科学家，我觉得令人难以置信的是，我们中的更多人没有花时间思考我们从比赛中学到了什么，这些评级系统告诉我们什么，以及这对这些市场中谁赢谁输的影响，以及思考这些事情的社会影响。

So that's something I'm pretty passionate about.

所以这是我非常热衷的事情。

Lenny: I also led the review system flows for a while at Airbnb, and one of the things I'm most proud of is launching what we call double-blind reviews where you don't see the other person's review until you leave your review.

Lenny：我还在 Airbnb 领导过一段时间的评论系统流程，我最自豪的一件事就是推出了我们所说的双盲评论，在你留下评论之前你看不到其他人的评论。

The intention was to create more honesty and more accurate reviews.

目的是创造更诚实、更准确的评论。

Lenny: It turned out the biggest impact was review rate went up, because people get this email, "Ramesh left you a review.

莱尼：事实证明，最大的影响是评论率上升，因为人们收到这封电子邮件，“拉梅什给你留下了评论。

If you want to see it, should leave a review." And that really increased review rate, which gave us more data.

如果你想看的话，应该留下评论。”这确实提高了评论率，为我们提供了更多数据。

And it was a really fun experiment to work on.

这是一个非常有趣的实验。

Ramesh Johari: There's a great concept in the literature on rating systems called the sound of silence, which is this idea that there's a lot of information in ratings that are not left.

Ramesh Johari：关于评级系统的文献中有一个很棒的概念，称为“沉默之声”，即评级中存在大量未留下的信息。

So Steve Tadelis, who's a professor at Berkeley, he had a really nice paper with some folks at eBay talking about what they called effective percent positive, where rather than normalizing just by the ratings, they normalized by including ratings that weren't left.

因此，伯克利大学的教授史蒂夫·塔德利斯 (Steve Tadelis) 与 eBay 的一些人发表了一篇非常好的论文，讨论了他们所谓的有效百分比积极性，他们不是仅通过评级进行标准化，而是通过包括未留下的评级来进行标准化。

And what you found was this was much more predictive of downstream performance of a seller.

您发现这更能预测卖家的下游业绩。

So there's a lot of information in that lack of a response.

因此，缺乏回应背后蕴藏着很多信息。

So it's cool that you're able to get more of that out.

因此，您能够从中获得更多内容，这很酷。

Lenny: So much easier just to not leave a review than leave a bad review.

莱尼：不留下评论比留下差评要容易得多。

Right?

正确的？

The downside to you is just much better.

对你来说，不利的一面要好得多。

Oh man, marketplaces are so fascinating.

天啊，市场真是太迷人了。

I could see why a founder would want to be a marketplace founder, because it's just such an interesting space.

我可以理解为什么创始人会想成为市场创始人，因为这是一个非常有趣的空间。

And hearing your feedback of, no, you're not a marketplace founder.

听到您的反馈，不，您不是市场创始人。

Let's think about the problem you're solving.

让我们考虑一下您要解决的问题。

And it might be a marketplace, might change people's minds.

它可能是一个市场，可能会改变人们的想法。

Also, I feel like there's a podcast episode in every topic we touched on.

另外，我觉得我们谈到的每个主题都有一个播客节目。

I know we just scratched the surface a lot of things.

我知道我们只是触及了很多事情的表面。

Lenny: I know you got to run.

莱尼：我知道你必须逃跑。

Before we get to our lightning round, is there anything else you wanted to highlight, touch on, leave people with that are maybe working on marketplaces, thinking about a marketplace?

在我们开始闪电轮之前，您还有什么想强调、提及、留给那些可能在市场上工作、正在考虑市场的人吗？

Ramesh Johari: I think one of the high level points I would make, and like you said, there's an entire podcast in this topic, is that I think people want to imagine LMs and AI driven data science automating out large parts of what it means to do data science in industry.

Ramesh Johari：我认为我要提出的一个高层次观点是，就像你说的，这个主题有一个完整的播客，我认为人们想要想象 LM 和人工智能驱动的数据科学自动化了工业中数据科学的大部分含义。

And I think that's probably the wrong perspective.

我认为这可能是错误的观点。

In some mundane sense, that's true.

从某种世俗意义上来说，确实如此。

It's easier for me to code than it used to be before.

对我来说，编码比以前更容易了。

It's easier for me to develop visualizations than it used to be.

对我来说，开发可视化比以前更容易了。

I can make dashboards faster.

我可以使仪表板更快。

So programmatically, I think it's true in some basic sense.

因此，从编程角度来说，我认为在某些基本意义上这是正确的。

Ramesh Johari: But what I believe pretty strongly, and I teach data science here, and my students are asked to use LMs and generative AI on a weekly basis on all their assignments.

Ramesh Johari：但我非常坚信这一点，我在这里教授数据科学，我的学生被要求每周在所有作业中使用 LM 和生成人工智能。

So I've got an up close and personal beat on this, but I believe very strongly actually is what AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test.

所以我对此有一个近距离和个人的看法，但我非常坚信人工智能实际上为我们所做的就是它极大地扩展了我们可以思考问题、我们可以拥有的假设、也许我们可以测试的事情的前沿。

It's just an astronomical explosion of explanations, and ideas, and principle.

这只是解释、想法和原则的天文数字爆炸。

Ramesh Johari: And I really think actually what that does is puts more pressure on the human, not less.

拉梅什·乔哈里：我真的认为这实际上给人类带来了更大的压力，而不是更少。

I think it becomes more important for humans to be in the loop in interacting with these tools to drive the funneling down process of identifying what matters, at all levels.

我认为，对于人类来说，参与与这些工具交互的循环变得更加重要，以推动在各个层面上确定重要事项的漏斗过程。

That ranges from you're carrying out a data scientific analysis, and now because you've got these tools, you can hypothesize 10 explanations, maybe 100 explanations.

范围从您正在进行数据科学分析，到现在因为您拥有这些工具，您可以假设 10 种解释，也许 100 种解释。

Which of those are you going to focus attention on?

您将重点关注其中哪一个？

What are you going to tell other people to focus their attention on?

您要告诉其他人将注意力集中在什么事情上？

To you're running experiments, used to have 10 creatives you're testing for a marketing campaign, you got 1,000 creatives, you're testing for that marketing campaign.

对于您正在运行的实验，过去您正在为营销活动测试 10 个广告素材，现在您有 1,000 个广告素材，您正在为该营销活动进行测试。

Maybe that completely changes the game of what it means to run an experiment.

也许这完全改变了进行实验的意义。

What are you actually looking for now?

你现在真正在寻找什么？

How do you evaluate that you found something that was good enough?

您如何评价您发现的东西足够好？

Ramesh Johari: And I think these questions are not getting enough attention.

拉梅什·乔哈里：我认为这些问题没有得到足够的重视。

I think people are looking for the automated tool that really cuts the human out.

我认为人们正在寻找真正摆脱人工的自动化工具。

But what I've seen so far, and again, who knows?

但到目前为止我所看到的，还有，谁知道呢？

By 2024, I might have a totally different answer for you.

到 2024 年，我可能会给你一个完全不同的答案。

I don't think so.

我不这么认为。

But at the moment, what I see is that humans have actually become far more important to the productive data science loop, not far less.

但目前，我看到的是，人类对于高效的数据科学循环实际上变得更加重要，而不是减少。

Lenny: Such an important point.

莱尼：这一点很重要。

I feel like we need to add AI corner to this podcast where we always think about, how does AI impact what we're talking about on this podcast?

我觉得我们需要在这个播客中添加人工智能角落，我们总是在思考，人工智能如何影响我们在这个播客中谈论的内容？

Ramesh Johari: Yeah, I can see that.

拉梅什·乔哈里：是的，我可以看到这一点。

I totally see that.

我完全明白这一点。

Lenny: Okay, we might start doing that.

莱尼：好的，我们可以开始这样做了。

Ramesh, with that, we've reached a very exciting lightning round.

拉梅什，至此，我们进入了非常激动人心的闪电回合。

I've got six questions for you.

我有六个问题要问你。

Let's try to knock through them so you can go teach your class.

让我们试着解决它们，这样你就可以去教课了。

Are you ready?

你准备好了吗？

Lenny: All right.

莱尼：好吧。

What are two or three books you've recommended most to other people

您向其他人推荐最多的两三本书是什么

Ramesh Johari: When it comes to books, I have one I love that I start with always, which is How to Lie with Statistics.

Ramesh Johari：谈到书籍，我总是从一本书开始，那就是《如何用统计说谎》。

It's a tiny book, Darrell Huff from 1954, which is just for anyone that likes data at any level, it's such a fun read.

这是一本小书，Darrell Huff 于 1954 年出版，适合任何级别的喜欢数据的人，读起来非常有趣。

It's a great book.

这是一本很棒的书。

Ramesh Johari: The second thing I recommend to people, and actually this is true even for people who are not expert, is David Freedman was a statistician at Berkeley who passed away in the 2000s, early 2000s.

Ramesh Johari：我向人们推荐的第二件事是，大卫·弗里德曼 (David Freedman) 是伯克利的一名统计学家，于 2000 年代、2000 年代初去世，实际上，即使对于非专家来说也是如此。

His writing was fantastic in getting us to think hard about process.

他的写作非常出色，让我们认真思考过程。

He was especially fond of what he called shoe leather statistics, where you rolled your sleeves up, you got on the ground, boots on the ground, really getting in there, really trying to understand your data.

他特别喜欢他所谓的鞋革统计，你卷起袖子，脚踏实地，靴子踩在地上，真正进入那里，真正尝试理解你的数据。

Ramesh Johari: His writing is fantastic, his explanations are fantastic.

拉梅什·乔哈里：他的写作很棒，他的解释也很棒。

He has a few different books at different levels I think people would love reading.

他有几本不同级别的书，我认为人们会喜欢阅读。

Most importantly, what I like about it is he puts such emphasis on driving evidence and understanding of your processes that generate data.

最重要的是，我喜欢的是他非常强调推动证据和理解生成数据的过程。

And I find often, data scientists don't even look at examples.

我经常发现，数据科学家甚至不看例子。

Ramesh Johari: So at oDesk, it meant are you looking at actual jobs, and what's actually going on in your product before you're trying to do data science on it?

Ramesh Johari：所以在 oDesk，这意味着在尝试对产品进行数据科学之前，您是否会关注实际的工作以及产品中实际发生的情况？

So I think that's a Freedman insight, Freedman mantra, and so his writing is great.

所以我认为这是弗里德曼的见解，弗里德曼的口头禅，所以他的写作很棒。

Ramesh Johari: The last one I was going to mention has nothing to do with data science or anything.

Ramesh Johari：我要提到的最后一个与数据科学或其他任何事情无关。

It's called Four Thousand Weeks by Oliver Burkeman.

奥利弗·伯克曼（Oliver Burkeman）的书名叫《四千周》。

I'm not a huge self-help type person, but I really like this book a lot.

我不是一个喜欢自助的人，但我真的很喜欢这本书。

I think it's a little bit stoic in its approach, like stoic philosophy.

我认为它的方法有点坚忍，就像坚忍哲学一样。

But the basic point is you're only on earth somewhere in the neighborhood of 4,000 weeks, and my wife and I have this term we call infinite Q, which is no matter what you think you get done on a given day, more stuff's going to just keep coming in.

但基本点是，你只在地球上的某个地方度过了 4,000 周左右的时间，我和我的妻子有一个术语，我们称之为无限 Q，无论你认为自己在某一天完成了什么，都会有更多的事情不断涌入。

Ramesh Johari: And he basically says that recognizing that is liberating.

拉梅什·乔哈里：他基本上说认识到这一点就是一种解放。

Because once you recognize it, it doesn't matter what you do.

因为一旦你认识到了这一点，你做什么就不再重要了。

You're always going to have too much to do.

你总是会有太多的事情要做。

There's no point in stressing out about having too much to do.

没有必要因为有太多事情要做而感到压力。

And just that small shift of mindset than puts a lot more attention on the usual thing people worry about, which is, where do I want to prioritize my time?

只是这种思维方式的微小转变，就让人们更多地关注人们通常担心的事情，即我想把时间优先安排在哪里？

So he has a great way of writing about it, some concrete rules of thumb to help manage that way of thinking.

所以他有一种很好的写作方式，一些具体的经验法则可以帮助管理这种思维方式。

And yeah, I think it's a great book.

是的，我认为这是一本很棒的书。

Lenny: What is a favorite recent movie or TV show?

莱尼：最近最喜欢的电影或电视节目是什么？

Ramesh Johari: I am a climber, and one movie that I really liked was The Alpinist.

Ramesh Johari：我是一名登山者，我真正喜欢的一部电影是《登山者》。

I know a lot of people have seen Free Solo, but for anyone that kind of likes that genre, I would recommend they watch The Alpinist.

我知道很多人都看过《Free Solo》，但对于任何喜欢这种类型的人，我建议他们观看《The Alpinist》。

Ramesh Johari: I think climbing is an interesting sport because has very much a psychological aspect of it.

Ramesh Johari：我认为攀登是一项有趣的运动，因为它在很大程度上涉及到心理方面的因素。

And I think that movie is pretty good at this meta level where you reflect a little bit on, what does it mean to make a movie about people who are obviously putting themselves into such risky situations?

我认为这部电影在这个元层面上相当不错，你可以反思一下，制作一部关于那些显然将自己置于如此危险境地的人的电影意味着什么？

So I really enjoyed that.

所以我真的很喜欢这样。

Ramesh Johari: On TV, we've been watching Only Murders in the Building, but I'm enough episodes behind right now that I probably won't say anything more, because I am trying to avoid any spoilers and I'm sure there's people out there trying to do the same.

拉梅什·乔哈里：在电视上，我们一直在看《大楼里的谋杀案》，但现在已经看了足够多的剧集了，我可能不会再说什么了，因为我试图避免任何剧透，而且我确信有人也在尝试做同样的事情。

So great show though on Hulu.

Hulu 上的表演真是太棒了。

Lenny: What's a favorite interview question that you like to ask candidates that you're hiring?

莱尼：您在招聘时最喜欢问应聘者的面试问题是什么？

Ramesh Johari: I interview people probably that are a little bit different than most of your podcast listeners.

Ramesh Johari：我采访的人可能与大多数播客听众有所不同。

But that said, there's one question I like to ask a lot, and that's if you imagine...

但话虽如此，有一个问题我很喜欢问，那就是如果你想象......

Often in our interviews in academia, whether it's grad students or faculty will ask people about their plans.

通常在我们学术界的采访中，无论是研究生还是教师都会询问人们的计划。

Ramesh Johari: And what I like to ask people is, "Okay, now imagine everything works out, all the challenges you're facing work out, all your plans work out, everything hits the top end of your vision for what this could be.

拉梅什·乔哈里：我喜欢问人们的是，“好吧，现在想象一下一切都会顺利，你面临的所有挑战都顺利解决，你所有的计划都顺利，一切都达到了你的愿景的顶端。

What do you imagine is the impact of having done that?

您认为这样做会产生什么影响？

Who's being impacted by that?

谁受到了影响？

Why is that a big deal that happened?"

为什么会发生这么大的事情？”

Ramesh Johari: And I find that's a really valuable question to ask, because first of all, many people haven't thought about that.

拉梅什·乔哈里：我发现这是一个非常有价值的问题，因为首先，很多人都没有考虑过这一点。

We're so short-term focused, we don't even think, "Boy, if everything worked out, what would be the big deal because of what I did?" Startup founders tend to be better at this than most people obviously.

我们过于关注短期目标，甚至不会想：“天啊，如果一切顺利的话，我所做的事情又有什么大不了的呢？”显然，初创公司创始人往往比大多数人更擅长这一点。

Ramesh Johari: But another reason I like it is because you will find in that conversation that their vision expands a little bit of additional spheres that are touched or impacted by what they're thinking about doing.

拉梅什·乔哈里：但我喜欢它的另一个原因是，你会发现在那次谈话中，他们的视野扩展了一些额外的领域，这些领域受到他们正在考虑做的事情的触动或影响。

So on both sides, it's kind of a revealing question, I think.

所以我认为，对于双方来说，这都是一个很有启发性的问题。

So I find that important for my line of work, but my hunch is that might be useful for some of your listeners too.

所以我发现这对我的工作很重要，但我的预感是这可能对你的一些听众也有用。

Lenny: Yeah, such a unique perspective on interviewing, versus most of the guests that I interview in tech company.

莱尼：是的，与我在科技公司采访的大多数嘉宾相比，这是一种独特的采访视角。

Ramesh Johari: Yeah, normally there's a coding question, right?

Ramesh Johari：是的，通常会有一个编码问题，对吧？

I should say I would never ask a coding question post November 2022 after we got AI to help us code.

我应该说，在 2022 年 11 月之后，在人工智能帮助我们编码之后，我永远不会问编码问题。

I think it's a superpower.

我认为这是一种超能力。

Lenny: AI corner.

莱尼：人工智能角。

What is a favorite product you've recently discovered that you really like?

您最近发现并且真正喜欢的最喜欢的产品是什么？

Ramesh Johari: I also really like cycling.

Ramesh Johari：我也非常喜欢骑自行车。

And I'm not ashamed to admit that I think that e-bikes are the greatest thing for cycling.

我并不羞于承认，我认为电动自行车是骑行中最棒的东西。

Admittedly, I'm late forties, so maybe I'm the right target demographic too.

诚然，我已经四十多岁了，所以也许我也是合适的目标人群。

But yeah, I love my e-road bike.

但是，是的，我喜欢我的电动公路自行车。

It's great, because it's not one of those with a throttle, you have to work, but it kicks in just when you're on your sixth hill and you don't want to go up the last hill anymore on the way home.

这很棒，因为它不是那些有油门的车，你必须工作，但当你在第六座山上并且你不想在回家的路上再爬最后一座山时，它就会开始工作。

So yeah, that's amazing.

是的，这太棒了。

I think that's just transformative for people that like cycling, but have busy lives.

我认为这对于那些喜欢骑自行车但生活忙碌的人来说是一种变革。

Ramesh Johari: And I think another one that my son who's 10 roped me into actually, is we were in Santa Cruz browsing at a kitchenware shop of all places, and he saw an outdoor pizza oven, a tiny portable one.

Ramesh Johari：我认为我 10 岁的儿子实际上让我参与的另一件事是，我们在圣克鲁斯的一家厨具店逛遍了，他看到了一个户外披萨烤箱，一个小型便携式烤箱。

And he just did research for two weeks and insisted we get one.

他只做了两周的研究，并坚持要求我们买一个。

Ramesh Johari: So he got one over the summer, and after we got it, he refused to eat pizza out anymore as a 10-year-old.

Ramesh Johari：所以他整个夏天都买了一个，我们买了之后，他 10 岁的时候就拒绝再出去吃披萨了。

So maybe that's the best thing I could say about the quality of pizza you can get from a home outdoor portable pizza oven.

所以也许这是我能说的关于你可以从家用户外便携式披萨烤箱获得的披萨质量的最好的说法。

Lenny: Oh my God, I'm hungry.

莱尼：天哪，我饿了。

I am going to go have to get some pizza now.

我现在要去买点披萨了。

What is a favorite life motto that you like to repeat to yourself, share with folks, find useful in your day-to-day?

您最喜欢对自己重复、与他人分享、觉得对日常生活有用的人生格言是什么？

Ramesh Johari: A lot of my work involves talking to students of all stripes.

Ramesh Johari：我的很多工作都涉及与各种类型的学生交谈。

And I guess these students go on to be data scientists, go on to be founders, and a lot of them go in the tech industry.

我想这些学生会继续成为数据科学家，继续成为创始人，其中很多人都进入了科技行业。

So maybe in that sense, that advice is relevant.

所以也许从这个意义上说，这个建议是相关的。

Ramesh Johari: My main thing I tell people is slow down.

Ramesh Johari：我告诉人们的主要事情是放慢脚步。

I think what I've found has been happening, is we're so convinced that speed is the way you're going to find the right answer, that I just don't think we slow down to develop meaningful mental models of the things we're doing.

我认为我发现正在发生的事情是，我们如此确信速度是找到正确答案的方式，我只是不认为我们放慢速度来为我们正在做的事情开发有意义的心理模型。

That's certainly true in the research projects I work on.

在我从事的研究项目中确实如此。

It's consistently true when I talk to people in business, and I ask them about their...

当我与商界人士交谈时，这始终是事实，我询问他们的...

By mental model, I just mean if you're running a marketplace, what is your model of what people care about?

我所说的心理模型是指，如果您正在运营一个市场，那么您对人们关心的内容的模型是什么？

What makes people stay versus leave?

是什么让人们留下来而不是离开？

What makes matches work versus not work?

是什么让比赛有效与无效？

All those things shape a roadmap in your mind.

所有这些事情都会在你的脑海中形成一个路线图。

And I think a lot of roadmapping, a lot of execution, paper writing in academia has all just become far more fast-paced, at the expense of deeper thinking about these kinds of structural features of the thing you're building.

我认为学术界的大量路线图、大量执行、论文写作都变得更快节奏，而代价是对你正在构建的东西的这些结构特征进行更深入的思考。

Ramesh Johari: So with my students, but also I think with people I interact with in industry, I think slowing down is actually more of a virtue than it's given credit for.

Ramesh Johari：因此，对于我的学生来说，而且我认为对于与我在行业中互动的人来说，我认为放慢速度实际上更多的是一种美德，而不是人们所认为的。

Lenny: Very similar to a motto that a recent guest shared, which I think was go slow to go fast, or stay smooth to go fast.

莱尼：与最近一位嘉宾分享的座右铭非常相似，我认为这就是要慢走才能快，或者保持平稳才能快。

Ramesh Johari: Yeah, I like that.

拉梅什·乔哈里：是的，我喜欢这样。

Maybe I'll pilfer that, when I go talk to my grad students .

当我去和我的研究生谈话时，也许我会偷走这一点。

Lenny: Final question.

莱尼：最后一个问题。

You're a professor at Stanford University, which sounds incredibly cool.

你是斯坦福大学的教授，这听起来非常酷。

What's something about being a professor at Stanford in particular or in general that would surprise people, either good or bad?

作为一名斯坦福大学的教授，有什么特别或一般的事情会让人们感到惊讶，无论是好是坏？

Ramesh Johari: Yeah.

拉梅什·乔哈里：是的。

I mean, we've had a rough ride, as everybody probably knows.

我的意思是，大家可能都知道，我们经历了一段艰难的旅程。

Stanford's been in the news for a lot of not so great reasons, I think over the last five years especially.

斯坦福大学出现在新闻中的原因有很多，但并不是什么伟大的原因，我认为尤其是在过去的五年里。

Ramesh Johari: So I don't know if this is the right kind of surprise, but I think one thing that I find really energizing at Stanford is people have never asked me for credentialing here.

拉梅什·乔哈里：所以我不知道这是否是一种适当的惊喜，但我认为在斯坦福大学真正让我充满活力的一件事是人们从未要求我提供认证。

And what I mean by that is that I came from a bunch of other good schools, and obviously I've spent time in industry with a lot of great companies.

我的意思是，我来自许多其他好学校，显然我曾在许多伟大的公司工作过。

And a kind of cultural dynamic that can often develop is, "Well, before I'm going to talk to you, I want to know something about why you're worth talking to.

通常可以形成的一种文化动力是，“好吧，在我跟你说话之前，我想知道为什么你值得和你交谈。

Give me your credentials.

给我你的凭据。

Where are you a grad student or where are you a professor?

您是哪里的研究生或教授？

Tell me about yourself first."

先跟我说说你自己吧。”

Ramesh Johari: One of the things that I found very surprising when I came here is just how that never happened at any level.

拉梅什·乔哈里：当我来到这里时，我发现非常令人惊讶的一件事是，这种情况在任何层面上都从未发生过。

Grad students tell me this all the time.

研究生一直告诉我这一点。

Go talk to someone across campus and just launch right into a conversation about how your X meets my Y, and we have something we could do together.

去和校园里的某个人交谈，直接开始对话，讨论你的 X 如何与我的 Y 相遇，我们可以一起做一些事情。

As a faculty member, it happens all the time.

作为一名教师，这样的事情时常发生。

I just had a conversation a couple days ago with someone about effectively a marketplace of experiment designs for nano fabrication here, which is totally out of left field for things I do, and yet seamless.

几天前，我刚刚与某人进行了一次对话，讨论了这里的纳米制造实验设计市场，这对于我所做的事情来说完全超出了我的范围，但却是无缝的。

Our conversation was about the substance rather than the credentialing.

我们的谈话是关于实质内容而不是资格认证。

Ramesh Johari: I really think part of the reason for that is that Stanford is sort of unique in that it doesn't have a weakness across the board.

拉梅什·乔哈里：我真的认为部分原因是斯坦福大学是独一无二的，因为它没有全面的弱点。

We have strong professional schools, law, business, medicine, strong engineering schools, strong humanities and social sciences.

我们有很强的专业学院，有法学院、商学院、医学院，有很强的工程学院，有很强的人文社会科学学院。

And then that and the weather is what I usually tell people honestly, which matters a lot.

然后，天气是我通常诚实地告诉人们的，这很重要。

People are willing to walk anywhere.

人们愿意步行去任何地方。

I think those things combine to create a culture and an environment where you don't credential everybody.

我认为这些因素结合起来创造了一种文化和环境，在这种文化和环境中，你不需要信任每个人。

Ramesh Johari: And I think that means a lot.

拉梅什·乔哈里：我认为这意义重大。

I think that's something that I haven't found elsewhere.

我认为这是我在其他地方没有发现的东西。

And if people wanted to know something about what's Stanford's like on the inside, I think that's one aspect of it that probably isn't discussed very much.

如果人们想了解斯坦福大学的内部情况，我认为这可能是人们很少讨论的一个方面。

I think that's part of what makes it really fun to be here.

我认为这就是让来到这里变得非常有趣的部分原因。

Lenny: It's also an incredibly dreamy campus, that is very joyful to walk around.

Lenny：这也是一个令人难以置信的梦幻校园，走在其中非常愉快。

That helps, I'm sure.

我确信这会有所帮助。

Ramesh, I feel like we got people's brains tingling.

拉梅什，我觉得我们让人们的大脑感到刺痛。

I think we've created new marketplace founders, and also convinced people maybe they aren't marketplace founders.

我认为我们创造了新的市场创始人，并让人们相信他们可能不是市场创始人。

So maybe we netted out zero new marketplace founders.

因此，也许我们净淘汰了零个新市场创始人。

Two final questions.

最后两个问题。

Where can folks find you online if they want to reach out?

如果人们想联系您，可以在哪里找到您？

And how can listeners be useful to you?

听众如何对你有用？

Ramesh Johari: I think the easiest way, if someone's interested more on the industrial side is probably LinkedIn.

Ramesh Johari：我认为，如果有人对工业方面更感兴趣，最简单的方法可能是 LinkedIn。

You send me a message or connect there.

您给我发消息或在那里联系。

Also, because I'm an academic, I have my own Stanford webpage, and it's pretty easy to figure out how to find me there as well.

另外，因为我是一名学者，所以我有自己的斯坦福大学网页，而且很容易弄清楚如何在那里找到我。

Ramesh Johari: And how can listeners help me?

Ramesh Johari：听众如何帮助我？

I kind of feel the most important thing that someone listening to this could do is take forward some of the messages that came out in terms of what it means to be data literate.

我觉得听这个的人可以做的最重要的事情就是传达一些关于数据素养意味着什么的信息。

And I think there's a lot you can do to educate yourself there.

我认为你可以做很多事情来进行自我教育。

Ramesh Johari: Maybe one final thought I'll share is that in the same way that AI generates a lot of ideas, AI also generates a lot of prose.

Ramesh Johari：也许我要分享的最后一个想法是，就像人工智能产生大量想法一样，人工智能也产生大量散文。

And in data science, that can actually be deadly because you're getting more explanations that sometimes maybe are extraneous.

在数据科学中，这实际上可能是致命的，因为你会得到更多有时可能是无关的解释。

Ramesh Johari: So taking that as a little vignette, I think that what the world needs is data literacy on the part of people interacting with these tools and with each other.

Ramesh Johari：因此，我认为这个世界需要的是人们与这些工具以及彼此之间交互的数据素养。

So that's the thing I care most about.

所以这是我最关心的事情。

The things I teach, the things I do research on, they're all connected to that theme.

我教的东西，我做的研究，它们都与这个主题相关。

And so that's where I'm pretty excited.

这就是我非常兴奋的地方。

I do work with companies regularly, and so if there's interesting opportunities that fall in the sphere of stuff we've discussed on the podcast, always happy to listen.

我经常与公司合作，因此，如果有有趣的机会属于我们在播客中讨论的内容，我总是很乐意倾听。

Lenny: Awesome.

莱尼：太棒了。

I think we've made a dent in helping people become a little more data literate.

我认为我们在帮助人们提高数据素养方面已经取得了一定的进展。

Ramesh, thank you so much for being here.

拉梅什，非常感谢你来到这里。

Ramesh Johari: All right.

拉梅什·乔哈里：好的。

Thank you so much, Lenny.

非常感谢你，莱尼。

Bye everyone.: Thank you so much for listening.

大家再见：非常感谢您的聆听。

If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

如果您发现这很有价值，您可以在 Apple Podcasts、Spotify 或您最喜欢的播客应用程序上订阅该节目。

Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast.

另外，请考虑给我们评分或留下评论，因为这确实有助于其他听众找到播客。

You can find all past episodes or learn more about the show at lennyspodcast.com.

您可以在 lennyspodcast.com 上找到所有过去的剧集或了解有关该节目的更多信息。

See you in the next episode.

下一集见。