第 496 期·中文精读

FFmpeg:互联网视频背后的惊人技术

FFmpeg: The Incredible Technology Behind Video on the Internet

FFmpeg 如何支撑互联网上的视频编码、转码与播放,以及这套开源工具背后的工程哲学。

官方音频00:00 / --:--
中文精读由 AI 基于原始访谈改写,非官方翻译,可能存在偏差或错误;表述以原音频与官方文稿为准。说明

dav1d 的代码仓库里,手写汇编有 24 万行,C 只有 3 万行。整个 FFmpeg 里所有编解码器加在一起,汇编是 10 万行。一个解码器的手写汇编量,是整个 FFmpeg 的两倍多。

这个解码器解的是 AV1。Netflix 上 30% 的视频、YouTube 上 50% 的视频,现在都是 AV1 编码。地球上大概有 30 亿台设备在不停地解它,而其中相当一部分设备没有硬件解码器,只能靠这几百 KB 的软件代码一个像素一个像素地算。

dav1d 是 VideoLAN 的项目,同一个组织还维护着 VLC。VLC 被下载了至少 65 亿次,核心维护者五到八个人。

这是 Lex Fridman 第 496 期播客的开场区域。两位嘉宾:Jean-Baptiste Kempf(圈内叫 JB),VideoLAN 主席,VLC 的主要维护者;Kieran Kunhya,资深编解码器工程师,FFmpeg 长期贡献者,运营着那个说话很冲的 @FFmpeg 推特账号,自己也开了一家公司叫 Open Broadcast Systems,做体育赛事直播的编码设备。整期节目里 Lex 头上一直戴着一顶 VLC 的交通锥帽子。

按下播放键之后,发生了一连串没人看见的事

Kieran 把这套流程拆成了几层。第一层是从一个地址拿到字节流,URL、文件路径、DVD 都算。第二层是解复用,也就是容器层,把混在一起的数据切成视频帧、音频帧、字幕轨。第三层是解析每一帧的头部信息,判断这个编码能不能交给 GPU 硬解。Kieran 给了一个数字:大约 45% 的文件无法用 GPU 解码,必须走软件路径。

软件路径里最底层的一步叫熵解码,把比特流里的数学编码(霍夫曼编码、算术编码)拆开。然后是帧内预测,也就是 I 帧,在空间域里做,根据周围已经解出来的像素猜当前块长什么样。猜得不可能完全准,剩下的差值叫残差,残差存在频域里,量化之后做逆变换拉回空间域,叠加上去。

JB 补了一个容易被忽略的量级:音频从原始数据到 MP3 是 10 倍压缩,视频要到 100 倍、200 倍,目标是 1000 倍。

压缩到这个程度,靠的是信息本身有冗余。Kieran 说任何视频都有大量空间和时间上的冗余,编码器的工作就是把这些冗余去掉。JB 举的例子很朴素:镜头摇过去,背景那朵云一分钟没变,那就不用每帧都传一遍同样的像素;背景是全黑的,那就告诉解码器“这块跟左上角那个像素一样”,省掉整片数据。理论上可以无限往前引用,实际上被内存和算力卡死,比如在 4K 分辨率下比较两百帧之前的像素,计算量大到不现实。

压缩不是 ZIP。ZIP 是数据进、数据出,你要逼近的是无损的极限。这里我们在主动降质,而且必须用最聪明的方式降质。(Jean-Baptiste Kempf)

视频编解码器不负责保存真相,它负责骗人的眼睛。所以它不去碰 RGB,而是转到 YUV:一个亮度通道,两个色度通道。人眼里的视锥细胞和视杆细胞分工不同,对亮度的敏感度远高于对色彩细节的敏感度。于是色度分辨率直接砍一半,文件小一半,绝大多数人看不出来。

同理,所有现代编解码器都在频域里工作,用离散余弦变换把画面切成块,这就是为什么解码出错的时候你会看到一块一块的方格。

容器和编解码器:VLC 从来不信文件后缀

Lex 问了一个很多人搞混的问题:把 MP4 和 H.264 混为一谈,算不算大错。JB 的回答是行业自己把水搅浑了。H.264 的正式名字是 MPEG-4 Part 10,而 MPEG-4 本身是个元规范,里面有 Part 2(早期的 MPEG-4 视频),有 AAC 音频,有好几种视频编解码器。所以你既可以说它是 MP4 的一部分,又不是。

Kieran 补了一句更实用的:现实中 99% 的 MP4 就是 H.264 加 AAC,其余是边缘情况。

真正有意思的是 VLC 和 FFmpeg 怎么处理文件后缀。它们不信任后缀。后缀只是一个提示,用来提高某个解复用模块的优先级。打开文件之后,程序会读文件内容本身判断这是什么格式。

我们不信任输入。VLC 最早的形态是给一个叫 VideoLAN 的流媒体方案做客户端,那时候视频走 UDP 在网络上传输,包会丢、会坏。所以整个架构从第一天起就是为“文件是坏的”这个前提设计的。(Jean-Baptiste Kempf)

这个设计选择后来变成了 VLC 的招牌能力。早年大家下盗版 AVI 的时候,元数据在文件末尾,下载到一半根本读不到,别的播放器直接罢工,VLC 说“这文件是坏的,但我还是试着解释一下”。

VideoLAN 年会上他们还办过一个比赛,看谁能做出最诡异、最糟糕的文件。有人做了一个 MKV,每一帧的分辨率、宽高比、旋转角度都不一样。还有人做了个“视频”,每一帧都是黑帧,全靠叠加在上面的逐帧动画字幕(SSA 格式)撑起整段内容。都放出来了。

Lex 提到有个文件同时是合法的 ZIP 和合法的 MP3。JB 说这种蠢文件比赛办过好几届。

那个橙色交通锥,和一万封反对邮件

VLC 官网 25% 的流量来自搜索“cone player”。JB 说很多人不知道 VLC 这个名字,他们只认那个锥子。他去印度或者中国的偏远地方,人们认得那个锥。

有一年愚人节他们开玩笑说要换 logo,改成某种毛毛虫造型,收到大约一万封邮件说不行。

JB 对这事的分析挺有意思:你要做一个视频播放器,自然的想法是在电视上放一个播放按钮,那就是 YouTube 的 logo,毫无原创性。锥子是橙色的,很刺眼,很怪。怪到能变成梗,梗能变成文化,文化能撑二十年。

开源是一份社会契约,也是一场长达十四年的追人游戏

JB 用巧克力芝士蛋糕解释开源:闭源软件是面包店给你成品,开源是给你成品、给你配方、告诉你怎么造烤箱,还告诉你允许你怎么改配方、能不能转卖。

FFmpeg 从诞生到现在,累计有两三千人贡献过代码。这个数字听着很美,Kieran 和 JB 反复强调的却是另一头:能留下来的核心维护者,FFmpeg 十到十五人,VLC 五到八人。

时间线上有一千个贡献者,最后留下来的就十个。也就是说,一个人来了并且留下来的概率是 1%。人会换工作、结婚、生孩子、出事故。你大概率不会回来。所以将来维护你代码的人是我们,你的代码必须可维护,必须优秀。(Jean-Baptiste Kempf)

这段话解释了那个社区为什么说话那么难听。它不是傲慢,是容量约束下的必然结果。

后面 JB 讲了自己主导的一次重授权,把 LibVLC 的核心从 GPL 改成 LGPL,理由有两个:一是让第三方应用能内嵌 VLC 引擎而不必开源整个产品(这是他能围绕 VLC 开咨询公司的前提),二是苹果 App Store 的条款让 GPL 应用几乎无法上架,LGPL 容易得多。iOS 和 tvOS 版本最后用的是 MPL。

改授权在法律上意味着要找到每一个贡献过代码的人,拿到同意。因为开源项目在美国版权法下是“共同作品”,版权分散在每个个体手里,而且即使后来那段代码被删掉了,只要新代码是基于它改的,版权主张依然成立。

他联系了三百五十多人。

我跑到某个人工作的地方去找他,因为他儿子写了那段代码,而他儿子已经去世了。我要向他解释开源是什么意思,解释我不是一家想白拿那五行代码的公司,解释整个社区都同意这件事。他完全不知道我是谁。那年我比现在年轻十四岁,几乎要哭了。这不是代码问题,这是人的生活。(Jean-Baptiste Kempf)

Kieran 补了一句这个社区到底有多分散:有人在叙利亚战区,电力只有一部分时间;有人富,有人穷,有人老,有人少。能让这么一群人达成一致,本身就是成就。

JB 对“我们不在乎你是谁”这句话的解释,是整期里最能说明开源文化的一段:

重要的是你的代码好不好。我们关心优秀的代码。我们不在乎你是谁,也许你是条狗,我也不在乎。我不在乎你从哪来,我要看你的代码。有人发来补丁被拒了,很不高兴,说“我可是意大利、德国、美国某大公司的工程师”。我们不在乎。我们在乎你代码的质量,因为这定义了我们的社区。(Jean-Baptiste Kempf)

那些拒绝掉的钱,和那封夹着粉末的信

Reddit 上有个流传很广的梗图,说 JB 拒绝了数千万美元,只为让 VLC 保持无广告。他自己确认了:是真的,而且不止一次。

找上门的都是些什么生意?捆绑工具栏(本质是间谍软件)、改浏览器主页、改默认搜索引擎、在 VLC 里塞广告。JB 说他不是反对赚钱,他自己开过好几家公司,现在还在开。他反对的是用不诚实的方式赚钱。

如果当时 Netflix 来找我们说想把 Netflix 放进 VLC 里,故事可能会不一样。但没有人来。来找我们的只有那些见不得光的广告公司。(Jean-Baptiste Kempf)

他给的理由很个人化:得能睡得着觉,得为自己做的事感到骄傲,如果卖了自己就背叛了所有一起干活的人。最后一笔报价被他自己形容为“下流”,对方的游说词是“有了这笔钱你可以去做新的开源项目”。他说那个心理陷阱很难躲。

Kieran 补充了时代背景:2000 年代,下载一个程序然后它偷偷装间谍软件是常态,那些条款埋在许可协议最底下的小字里,没人读。

然后是那段最沉重的。2009 或 2010 年,JB 决定不再为 PowerPC 架构维护 VLC 的新版本。当时团队四个人,VLC 1.0 即将发布。他收到一封信,里面有粉末,信里写着“你这个垃圾,你该去死,PowerPC 万岁”。那个年代正好有炭疽信件事件,他母亲吓坏了,他们去报了警。粉末是面粉。

现在回头看,我挺高兴这事发生在那个时候。它把我练出来了。我能承受很多恶意,我没问题。(Jean-Baptiste Kempf)

Lex 接着说了句很真诚的话:他这辈子可能有几千次因为 FFmpeg 而露出笑容,但一次都没机会说谢谢,直到发现有这么个推特账号。

24 万行手写汇编:为什么编译器赢不了

Kieran 解释 SIMD 的方式很朴素。标量汇编是“我有数字 10,加 5,得到 15”,一条指令一个数。SIMD 是单指令多数据,你有一整个向量,16 个不同的数,一条指令把 5 加到全部 16 个上。视频天然就是像素网格,所以这个模型契合得惊人。

FFmpeg 的做法是不用任何上层抽象,直接写汇编,寄存器自己分配。世界上的另一条路是用 intrinsics,那是长得像函数、行为接近汇编但不完全一样的东西,寄存器由编译器分配。

差距有多大?Kieran 给的数字是 10 倍到 50 倍。他发过推特,某个函数的提升是 62 倍。

这就引出了持续两年的推特骂战。软件工程界的直觉是:编译器在进步,自动向量化越来越强,你手写汇编迟早会被超越。@FFmpeg 账号这两年一直在发反例,几百个。

他们会说“你可以优化你的编译器”“自动向量化是你没搞懂”。我们试过了,试了很久。差距不是 5%、10%,是好几倍。(Jean-Baptiste Kempf)

JB 从更宏观的角度解释为什么这件事现在更重要:摩尔定律到顶了,硬件不会再有过去那种代际跃迁,但我们对算力的需求在爆炸。大家的应对是堆核,堆到 250 核。而另一种应对是把每一寸机器都榨干。

榨到什么程度?Kieran 说他们会用跟用途完全无关的指令,比如拿密码学指令去做视频处理。dav1d 更进一步,它不遵守操作系统的函数调用约定。

调用约定这件事值得展开。你写一个函数调用另一个函数,数据怎么传、寄存器怎么保存,是有行业标准的,x86 上至少有四套(Linux 32 位、Windows 32 位、Windows 64 位、Linux 64 位)。早年 Loren Merritt 写了一个很轻的抽象层叫 x86inc,让你写一遍汇编就能适配四套约定,这是巨大进步。dav1d 反过来了:既然调用都发生在自己这个二进制内部,那就自己定一套约定,不保存那些用不上的寄存器,省掉 L1/L2 上的反复读写。

JB 强调这就是为什么必须懂计算机体系结构:流水线、SIMD、ALU、I/O。他见过社区里的人为一条汇编指令该用哪条争论半天,因为在这代 CPU 上是三个周期,在另一代上不是,而这个差别对最终输出影响巨大。

Kieran 用了一个比喻形容手写汇编的美感:像开一架喷火战斗机,操纵杆通过钢缆物理连着副翼,中间没有任何东西,你和处理器之间也没有任何东西。

他点名了两个他心目中的神。Henrik Gramner,住在瑞典,对 Intel x86 汇编和每一代 CPU 上每条 SIMD 指令的周期数熟到 Intel 自己的人都说“你们问我们干什么,你们有 Henrik”。Martin Storsjö,住在芬兰,主攻 Arm 的 NEON,他的编码方式是在手机上用虚拟键盘敲汇编,一边看孩子在游乐场玩。

三个人、24 万行,和“每个周期都重要”

dav1d 立项的时候,Alliance for Open Media 那边的普遍看法是:AV1 太复杂了,必须靠硬件解码。JB 和 Ronald、Henrik、Martin 几个人决定做一个极强的软件解码器,理由是硬件普及需要时间。

结果是 3 万行 C,24 万行手写汇编。项目口号是 every cycle matters。

效果是:一到两个核就能正确解 720p。JB 说这话的时候语气里带着明显的自豪,因为这意味着大量没有 AV1 硬解的老设备也能看 AV1 视频。

Kieran 发过一条推特,列了 dav1d 的代码构成:79.9% 汇编,19.6% C,0.5% 其他。他的配文是“这就是巅峰视频编解码器该有的样子”。下面吵了两年。

Lex 问如果有个工程师坐在家里听这期节目,C++ 写了十几年,后来转 Python,现在天天用自然语言让 AI 写代码,他该从这个故事里拿走什么。

JB 的回答指向 LLM 量化:为什么有人要做 FP8、FP4,甚至微软 Phi 做到 1.5 bit,因为被内存和算力卡死。实时场景里你没法总靠换更好的硬件,只能分析代码,找出那些被调用亿万次的热点。dav1d 一天要跑几十亿小时,值得为它手写 24 万行汇编;FFmpeg 的命令行胶水层不值得。

AI 会帮我们写业务代码,这是好事。但真正没法被“氛围编程”替代的,是为硬件做的极致优化。(Jean-Baptiste Kempf)

Kieran 补充了另一面:大多数公司会觉得“C 已经够快了”,而在他的公司里,同一段 10 bit 视频格式转换,手写汇编比 C 快 62 倍。10 bit 数据在 CPU 上没法原生处理,得塞进 16 位,浪费 6 位,于是有一堆打包格式;他的公司内部有一个 5×5 到 6×6 的矩阵,把每一种格式转到其他每一种格式,全部手写汇编,每个还支持不同代 CPU。

JB 引用了 Kieran 的话并称其为艺术:这是全世界唯一一个跑在几十亿设备上、还在这么干的代码库。高频交易也重视指令级优化,但那是跑在专用硬件上的极小众场景。

AI 生成的漏洞报告,和一辆 1993 年的赛车

推特上那场风波的核心事实是这样的:Google 开始用 AI 对 FFmpeg 批量生成安全报告,用标准的 90 天披露期限,而这个期限的设计前提是有一个拿工资的团队在处理。FFmpeg 这边是志愿者。

Kieran 用了一个比喻:你家的门锁是为了防住它该防的东西,不是为了防核武库。AI 现在用规模化的方式去撬锁,撬开之后告诉你“你的锁不安全”,而真正有资源修锁的正是撬锁的人。

那些报告非常啰嗦,AI 生成的漏洞报告在冷门编解码器上几乎构成了一种拒绝服务攻击。而且所有东西都被标成高危。(Kieran Kunhya)

他举了个更荒诞的例子:另一次所谓的漏洞,是一个滤镜里整数溢出,后果是某个像素颜色错了,标了 7.5 分高危红色警告。

JB 从另一个角度切入:发现漏洞的资源和修补漏洞的资源严重不对称。风波之后 Google 做了调整,开始直接发补丁,也设立了修复奖励。

然后是微软那条。Teams 有人在 FFmpeg 的公开 bug tracker 上写“这个问题是高优先级”,理由是 Teams 有大量用户。JB 说他想谈一个长期支持合同,对方给了一次性几千美元。

他们以为开源项目是传统供应商,以为公开的 bug tracker 是第三方供应商的 Jira。它不是。大公司里其实有 OSPO,开源项目办公室,本来就该负责跟开源社区对接,但他们经常没把这件事在公司内部讲清楚。(Jean-Baptiste Kempf)

结果呢?Kieran 说捐款显著增加,但仍然不够覆盖一个全职开发者。不过在认知层面和技术层面,这两年的效果是实打实的,很多人第一次意识到 FFmpeg 有多重要。

JB 讲了另一条路径:VLC 在 Android 上有一年多没法更新,因为 Play Store 的一个 bug。最后让 Google 有人回应的方式,是发一条语气很辣的推特说“我们准备停止在 Android 上分发 VLC”,涉及一亿用户。Windows Store 上也是同一招。

不幸的是我们太小了,唯一有效的力量就是在社交网络上喊话,因为它会滚雪球,然后他们才听。(Jean-Baptiste Kempf)

他补了一个具体的委屈:VLC 大概是 Windows 上使用量前十的软件,但他不在微软的 ISV 项目里,没有对接人。Adobe、Spotify 都有,他没有。

Kieran 把整件事定性成说唱对决。隔着屏幕互相放狠话,不代表真有私人恩怨。他说 Theo(OpenBSD 的 Theo de Raadt)那一段确实过界了一点,JB 后来打了电话把火压下去。

我们说了些什么,但我们留着没删。X 是一个完美的国际说唱擂台。你说我妈怎样,不代表我真对你妈有意见。这是 WWE,大家玩得开心,不必当真。(Kieran Kunhya)

一个 16 岁的贡献者叫 Ruikai Peng。他在 git 里发现并修好了一个问题,三天搞定,没有搞 CVE 编号、没有建带 logo 的网站、没有在 DEF CON 上领奖,就是修了。Kieran 的评论是“孩子们都挺好的”。

对比的是前 FFmpeg 开发者那条被广泛转发的帖子:安全研究者发现你代码里的 bug,会给它起个可爱的名字,建个带 logo 的网站,Google 给他们百万赏金,他们去 DEF CON 领奖,然后去某个大家都穿得像《黑客帝国》的秘密派对。你修好之后,没人给你任何东西。

Rust 能取代汇编吗

Lex 直接问了。两个人的意见确实不一样。

Kieran 认为内存安全这个方向有价值,但整体气质有问题,他用了两个比喻:世界语和 Sinclair C5。后者是 Sinclair 电脑创始人做的那辆电动三轮车,当年的宣传是“所有人都会开着这个上班”。Kieran 说 Rust 给他的感觉就是这样:社区有一种自我重要感,仿佛宣布了内存安全这个目标就等于解决了问题。他承认有人在用 Rust 重写一些基础工具,但“如果只做到所需功能的 85% 到 90%,剩下那 1% 要花 99% 的时间”。他用马斯克那句话收尾:原型都是简单的。要让真的电动车上路,你得做得跟现有产品一样好或者更好,而 Rust 还没到这个阶段。

JB 的态度更具体,因为他两边都干过。他在 VLC 里加过 Rust 模块,现在的新公司 Kyber 主要就是用 Rust 写的。

Rust 是一个更好的 C++,它在乎内存,能做出别的语言目前做不到的所有权管理。它在从零开始的新项目里很棒,涉及解析文件、网络这些场景时那个内存检查和边界检查非常好用。但它跟已有代码互操作很糟。(Jean-Baptiste Kempf)

他给了一条几乎是工程师守则的话:我当工程师、管理者、创业公司 CTO 这么多年,几乎从不说重写。

理由是一个容易被忽略的不对称:写代码比读代码容易一个数量级。LLM 也印证了这点,它能写,但分析难得多。你带着最高效率写出的最好代码,六个月后自己都读不懂。所以新人接手一个复杂代码库时,读不懂那些没有文档记录的取舍和业务逻辑,第一反应往往是推倒重来,但那个判断通常是错的。

JB 接着给了一个很少见的反驳角度,专门针对他自己所在的领域。假设你把 dav1d 或 x264 的 C 部分用 Rust 重写,安全了。然后你进到汇编那一层,手写汇编可以在内存里跳到任何地方。C 部分用 Rust 重写了,汇编一进来,整个安全模型就没了。

所以我认为我们真正需要的是“安全的汇编”:编译期检查的汇编。这就是我们在 dav1d 和 x264 上做的 checkasm,在编译的时候给你的汇编插桩,检查它没有跳到任意地址。(Jean-Baptiste Kempf)

这次访谈里 checkasm 只是被提了两次名字就被带过了,Kieran 提到时说的是“一个疯狂但了不起的项目”。它是 VideoLAN 对这套体系最独特的技术贡献之一:在单元测试层面,同时跑汇编实现和 C 参考实现,逐位比对输出,确保手写的那 24 万行没有偏离语义。没有这个东西,手写汇编在工程上不可维护。

FATE:一张赌桌一样铺开的测试矩阵

Kieran 介绍 FFmpeg 的测试系统,名字挺中二:FFmpeg Automated Testing Environment,缩写 FATE。

Lex 打开 fate.ffmpeg.org 的时候看到的是这样一个界面:一排排条目按“81 分钟前”“76 分钟前”倒序排列,每一行是一个配置组合,macOS 有十几个变体因为要覆盖 iOS、tvOS、watchOS,再往下是各种架构(RISC-V、Arm、PowerPC、过去还有 DEC Alpha)、各种编译器(Apple Clang 的各个版本、GCC、Visual Studio、Intel 编译器)、各种指令集。

Kieran 管这个叫“已经不是矩阵了,是一张数据透视表”。这些机器全是志愿者提供的,排在最上面那些 Mac,就挂在他自己办公室。

它的必要性和 FFmpeg 的代码特征直接相关:C 代码写得很复杂,编译器会出错。这不是假设,是经常发生的事。编译器把正确的 C 编译成了错误的机器码,而视频帧之间有依赖关系,输出里一个极小的偏差会级联成一整片画面上可见的花屏。所以你要能在一个你没见过的编译器加指令集组合上测出这个问题。

JB 拿这个当自豪点:VLC 最新版还能跑在 Windows XP 上,也跑在 Windows 11 上;macOS 从 10.7 到最新;iOS 从 9 开始;各种 Linux、BSD、Solaris;甚至 OS/2。全世界大概还有十个 OS/2 用户,其中一个在维护 VLC。

你会发现这个围绕着 VLC 和 FFmpeg 的极小团队,支持的操作系统比微软、Google、苹果都多,而他们有无限的资源和人力。(Jean-Baptiste Kempf)

代价也真实。他说最头疼的是 iOS:为了在 iOS 9 上编译,需要把好几个版本的 Xcode 和 SDK 混起来,做一个弗兰肯斯坦式的拼装,因为苹果的编译器早就不支持了,而他们还要在 iOS 9 的 Arm32 上跑。

为什么值得?他说经常收到人说“谢谢,我的 iPad 2 还能看电影”。这背后是一个价值观选择:不强迫别人换新硬件。JB 把它和汇编那条线连起来:你本可以多优化一点,而不是让人一直买新的,而优化正在变成一门失传的手艺。

x264:一次对“数学指标”的叛变

Kieran 说 H.264 能成,一半功劳要记在一次观念叛变上。

在那之前的二十年,工业和学术界被一个指标统治:PSNR,峰值信噪比,本质是均方误差取对数。麻烦在于,最小化均方误差的最优解是把误差摊平到每一个地方,也就是让整幅画面均匀地糊一点。所以那个年代编码出来的东西,指标很好看,看起来一片模糊。

x264 的开发者是一群爱好者,为的是自己的个人视频,主要是动画。他们做了两件不一样的事。

第一件叫心理视觉率失真优化,用块的能量去做决策,把人类感知作为目标函数的一部分。第二件叫自适应量化,把比特从复杂的区域挪到不那么复杂的区域。Kieran 举的例子是草地:草地有高频成分,但整体复杂度低于别的东西,所以应该少给它比特。

用来验证这套东西的那个样本叫 ParkJoy,瑞典电视台在 HD 时代初期拍的,胶片拍摄,制作上不计成本,然后免费放出。画面是一群人在河边跑,水面有反射,树叶有光影流动,草地有纹理,运动很复杂。Kieran 说至今没有哪个公开测试序列比它更好用。

这个样本让分歧变得肉眼可见:开高 PSNR 的编码器会把画面糊掉,开心理视觉优化和自适应量化的版本明显更好看,但它的 PSNR 数字更低。

而当年这些指标被视为神圣不可侵犯。PSNR 是最重要的东西。(Kieran Kunhya)

更有意思的是当时怎么判断“好看”。没有 VMAF(那是 Netflix 后来用机器学习做的),没有专业监视器,就是开发者在自己笔记本上看。Loren Merritt 明确说过他不想在一个三万美元的屏幕上调参数,他要的是在别人家笔记本上看着好。

JB 补了两个样本。一个是《行星地球》里的鸟群:一群鸟,越往后鸟越多,最后像有几百万只。他说这是编码难度最高的东西之一,你在 YouTube 上看这段,能直接看出 YouTube 的编码有多差。另一个是动画,长时间以来动画编码得很糟,因为渐变多,会出现明显的色带,这块的优化主要也是 Loren 做的。

x264 的出身地也是 École Centrale Paris,跟 VLC 同一个地方,由 Laurent Aimar 发起,后来 Loren、Jason、Måns、Henrik、Anton 这些人接力。JB 说这个项目最神奇的地方是大部分人从没见过彼此。

而 x264 走出爱好者圈子进入工业界,靠的是 Blu-ray 时代的几个人愿意拿自己的职业生涯冒险。Kieran 提到 Chris Henderson 在华纳兄弟用 x264 做了整部《危机边缘》的蓝光套装。他说自己到现在仍然尽量避免在流媒体上看最讲究影像的电影,宁可买实体碟,因为不用买贵电视它也好看。

动画社群和这套技术的纠缠还有另一层。JB 说那时候大量动画没有商业渠道,在 Crunchyroll 之前根本看不到,所以字幕组自己翻、自己做时间轴、自己压片。这催生了一批工具,其中最有名的是 Aegisub。动画的编码难度也有特殊性:它大量在屏幕上制作,渐变极多,真人实拍里很少见这种纹理。字幕本身也复杂,日语原文要标假名注音(ruby),还要叠英文或法文翻译,多层样式在同一帧上处理。

所以字幕这个东西我们见过各种疯狂的样本。这是文化的一部分,也是因为当时没有官方渠道,没有别的办法。(Jean-Baptiste Kempf)

I 帧、P 帧、B 帧,以及那个“未来”的问题

JB 用最朴素的方式解释帧类型:I 帧是关键帧,完整的图像,像一张 JPEG

P 帧是预测帧,意思是“我要上一帧的第 5、7、42 号块”,只传增量。B 帧是双向预测帧,可以依赖前面的帧,也可以依赖后面的帧。

B 帧带来的后果是解码顺序和显示顺序不一致。要解这一帧,得先等到它依赖的那帧到达,缓冲,解出来,再回头解它。JB 说他刚入行时这件事最让他震撼:编码器要足够聪明,去决定“我要依赖未来的东西”。

Lex 感叹这整套机制每天都在平滑运行本身就是奇迹。一个编码器在某个地方产出的比特流,在地球另一端某家厂商的解码器上,逐位输出完全相同的东西。

后面 JB 提到一个 Kyber 里重度使用的技巧叫 intra-refresh:整个流里只有一个 I 帧,在开头,之后再不发送完整的 I 帧,而是随着流推进逐渐刷新画面的各个部分,把 I 帧“长”出来。这样避开了关键帧带来的码率尖峰。

Lex 提到 FFmpeg 里 H.264 的默认 GOP 大概是 250 帧,也就是好几秒才有一个完整帧,中间全靠预测链撑住,而普通人完全察觉不到。这段对话里最日常的一句话反而最说明问题,JB 说的是,你用 FFmpeg 压完一个东西,播出来还是顺的,这件事本身就像魔法。

参数、码率和那些以此谋生的人

Lex 列了一串 FFmpeg 攒下来的旋钮:降分辨率、降帧率、换编码器、恒定码率还是恒定质量(CRF、QP)、GOP 长度、B 帧数量。

JB 说了一句容易被外行忽略的行业事实:一大堆人的工作就是调这些参数。你在 YouTube、Netflix、Meta 见到的那批人,很多不写编解码器,他们做的事是为特定内容找对参数组合。电影、手机拍的 UGC、屏幕录制、要拿去二次剪辑的素材,这四类要的东西完全不同。

几千个人的岗位就是干这个的。Lex 顺着说了 YouTube 的难度:Netflix 是单向广播,YouTube 还要处理从世界各地上传的、可能只有五个观众的、要在几秒内出多分辨率版本的视频。他的原话是“YouTube 基本上是网页版的 VLC”。

JB 接了个更有意思的史实:Google 在收购 YouTube 之前做的 Google Video,其实用的就是 VLC 插件,靠 ActiveX 在 Internet Explorer 里跑一个真的 VLC。而今天是反过来的,VLC 有 WebAssembly 版本,把 VLC 和 FFmpeg 一起编译进 JavaScript 虚拟机里跑。

他还讲了流媒体的真实瓶颈在别处。自适应码率这套东西本身不复杂,下个分片超时就降档,下载时间不到一半就升档,真正难的是 CDN 拥塞和人类心理。Kieran 补了一个更细的观察:观众对视频码率波动相当宽容,对音频码率波动极其敏感。从完整 AAC 切到用频谱带复制的压缩版本,声音会突然发薄,那种上下起伏很扎耳朵。视频侧平滑得多。

我们对跳帧和音频故障的容忍度其实高得惊人。我认识很多不是视频工程师的人,看体育比赛 30 帧也无所谓,本该是 60 帧。(Kieran Kunhya)

Kyber:从“画质”转向“毫秒”

JB 的路径是逐级下探的:先用 FFmpeg 编码文件,再用 FFmpeg 和 VLC 做流媒体,然后问一个问题,能低到哪去。

答案是低到有交互反馈的场景。看视频是被动接收,遥控是另一回事,你要控制它。无人机、人形机器人、远程车、云游戏,这些场景的约束彻底变了。

你不再在乎我们刚才聊了半天的那种画质,你在乎延迟。因为在控制一辆车的时候,一毫秒是有意义的。(Jean-Baptiste Kempf)

他举 Waymo 当例子:Waymo 不工作的时候,哪怕只有 1% 的情况,背后是有人在远程接管。这正是 Kyber 在做的场景,一个端到端控制机器的 SDK 平台。

技术选择上,Kyber 只用一条连接,基于 UDP 的 QUIC 协议,因为它是为低延迟设计的,避开了 TCP 和 HTTP 的队头阻塞,默认加密。在同一条线路上跑多路流:音频、视频,还有控制指令,鼠标、键盘、手柄,同时保持同步。

同步为什么难?因为所有时钟都会漂移。一台机器人上可能有十路摄像头、GPS、各种传感器,你要训练机器人 AI 模型,这些数据必须时间对齐且长期稳定。Kyber 在服务端做了重打时间戳的机制来补偿时钟漂移。现有的方案在一路摄像头上能工作,到了五六路就复杂了。

另一个机制是前向纠错:多传百分之几的数据,这样丢包时接收端能重建,而不用等重传。要保证所有包都到达,延迟就会暴涨;不要延迟,就只能靠冗余。

目标数字是 4 毫秒玻璃到玻璃延迟。4 毫秒意味着 240 Hz,如果你有一台 1000 帧每秒的相机,理论上能实时回传。目前做到的是 Windows 到 Windows 或 Windows 到 Mac 的 7 毫秒,其中 NVIDIA 硬件编码器占约 3.5 毫秒,Intel 解码器占约 2 毫秒,编解码加起来已经 6 毫秒。要再往下,得换编码器或者换更快的实现。

Lex 问这是不是没人做到过。JB 说专用硬件可以,SDI、专业设备能做到,但他要的是在互联网上、在任何一台装着 Jetson Nano 或者 N150 小主板的机器人上做到。

未来所有的远程车都会被 AI 模型远程监视,它说“一切正常”,不正常的时候说“这里有问题”,然后有人介入。这关乎安全。当你的机器人照顾你奶奶的时候,我要确保一切正常。(Jean-Baptiste Kempf)

CES 上他们做了个演示:一辆全 3D 打印的小车,带伸缩臂,在拉斯维加斯,从法国远程控制,视频来自一个普通摄像头和一块很小的板子。他们还在做 3D 打印的机翼、帆船,后面会碰人形机器人。

商业模式是双授权:AGPL 加商业授权。你想在商业产品里用又不愿意开源整个产品,就买商业授权;小团队和爱好者可以免费用。JB 说这件事对他来说很有意思,因为视频他算做得还行,网络他还有太多要学,拥塞控制协议、实时的码率自适应。

专利地雷:为什么谷歌要自己造一个编解码器

JB 给了最简洁的代际总结:每一代大概提升 30%。MPEG-2、H.264、HEVC、H.266 一路下来,AV1 比 H.264 省 40% 到 60% 带宽。他也承认从业者会来吵架:有人说 25%,有人说 50%,因为具体到某个内容,新一代工具可能省 70%,也可能只省一点点。屏幕录制就是收益极大的那一类。

命名混乱的来源是两个标准组织:ISO 是私营的,ITU 是联合国机构,两家合作,所以每个标准有两个名字。H.264 等于 MPEG-4 Part 10 等于 AVC,HEVC 等于 MPEG-H 加 H.265,H.266 等于 VVC。

真正的分歧不在技术,在专利。

多媒体是专利雷区。全世界专利最密集的两个地方,一个是 3G、4G、5G 和射频,另一个就是多媒体。因为它非常数学化,能做出来很大的增益。(Jean-Baptiste Kempf)

他解释了专利是怎么产生的:在 MPEG 的流程里,标准做完之后,各方跑来说“这个格式我有专利”,然后通常组成一个池子,MPEG LA,向所有使用者收费。他举了个荒谬的例子:本来是方块,我改成矩形,这个想法可以申请专利。

H.264 时代的专利费还算理智。到了 HEVC,因为钱太多,往规范里塞了大量在 99.9% 场景下没用的工具,唯一目的是能挂一个专利上去。结果是授权变得几乎不可能:MPEG LA 之外还有 HEVC Advance,诺基亚还在池子外面。费用没有上限,对 YouTube 或 Netflix 这种量级可能是每年几亿美元。

这笔钱算下来,结论很清楚:一年一亿美元,不如自己造一个。所以有了 Alliance for Open Media,有 AV1,有 AV2。JB 说 VideoLAN 也在这个联盟里。

AV1、AV2 在标准化流程里有专门的专利审查环节,而 MPEG 那边专利是完全的禁忌话题,不谈。(Kieran Kunhya)

代价是需要双重创造力。既要更高效,又要确保不踩已有专利。Kieran 用了一个很好笑的例子:AV1 里有个工具叫 S-frame,本质上是一种 B 帧的变体,之所以叫这个名字,就是为了绕开专利,像“我能相信这不是黄油”那种命名。

VLC 和 FFmpeg 怎么面对这些?JB 说他们不需要面对,这是 VLC 落脚法国的原因之一:法国不承认软件专利,大部分这类专利在欧洲无效,因为它们本质上是数学专利、想法专利。他算过一笔账,如果 VLC 要为每个用户付全部专利授权费,得每人两百多欧元。

他还顺手回应了欧洲不适合创业的论调。英国早就走盎格鲁-撒克逊那套商业模式,而法国这十五年变化很大:过去创业失败会背上污名,不让你再开公司,现在这个 stigma 已经没了,市场上大多数人想的是做 startup。

情报机构、后门,和“我们会关掉它”

Lex 问有没有政府来找过要后门。JB 说有,两家。

如果我们要妥协自己的软件,我们会把它关掉。这一点是明确的。(Jean-Baptiste Kempf)

具体防护措施他说得很细,而且他自己承认听起来像偏执:没有任何代码能进 VLC 而他们不控制;编译在离线机器上做,第一步是先编译编译器;全部在从没连过网的机器上完成;签名做双重签名。原因是他们发现并且相信有非西方国家的政府机构试图往他们自己的服务器里推一个假的二进制文件,这件事让他们非常警觉。

Lex 追问万一被施压怎么办。JB 的回答是开源软件没法被杀掉:我搬到马耳他、开曼群岛,换个域名,重新开始。VLC 是一个工具,不是平台。

关于内容审查,答案是不可能。VLC 完全离线,不跟任何服务器通信,没有遥测,所以谁也不知道你在看什么。有政府问过“我想知道这个人有没有看过那类视频”,回答是不知道。

他们跟执法机构唯一的合作形式是技术支持。JB 讲了一个案子:美国警方说有个谋杀案,关键文件损坏了或者在那个版本上放不出来。他说这类支持跟普通用户没区别,他们从来没有视频内容。另一个故事是阿富汗战争期间,一个军人写邮件说新版本 VLC 在 RTSP 服务器上放不出来,而那个服务器上的电影是地面部队的士气来源。JB 改坏了对 RTSP 的支持,于是给他单独做了一个版本。

他还顺口说了 VLC 大概是被允许装在美军笔记本上的,因为开源,有人审过代码和处理方式。而整个 VideoLAN 在法律层面只有他一个人。

Lex 问他怎么不焦虑。JB 说他以前非常焦虑,现在不是。

我的思考方式是永远先想最坏情况。像下棋一样,最后的问题是:我会死吗?会还是不会。做创业公司也一样,最坏是破产,公司有生有死,这没关系。我的道德判断一直是:我会死吗?我伤害到别人了吗?如果答案都是不,那就随它去。律师会不高兴?他们能怎么样,把 VideoLAN 的钱都拿走?哇,五万块。源代码在外面,这是挡不住的。(Jean-Baptiste Kempf)

整期节目里这段最接近他自己说的“禅”。它的内核不是乐观,是把最坏情况算到底之后发现它不致命。

Kieran 在这里插了一句实务观察:在欧洲应对法院传票不至于让你破产,在美国可以。JB 说他每周都收到律师函,VideoLAN 一年花在律师费上的钱不到一万美元。

关于那次著名的 CIA 事件,JB 讲得挺平静。WikiLeaks 的 Vault 7 文件披露,CIA 做了一个定制版 VLC,二进制的其余部分都是真的,只加了一个 DLL,叫 psapi.dll,功能是读你的文档目录,加密,然后发出去。伪装得很聪明:你在看电影,两个小时不会碰电脑;高清片源让风扇轰鸣、CPU 占用拉满,这也是正常的;所以你不会察觉。

同一套手法还被用在了别处。有中国黑客针对印度用户做了同样的事,结果导致 VLC 在印度被封禁,JB 不得不在印度政府那里打官司,证明那不是 VLC。他们只是拿了那个签名正确的 DLL 去做别的事,真正的 vlc.exe 调用 libVLC,但指向的是一个假的。

这种事我们其实没什么办法阻止。(Jean-Baptiste Kempf)

他还说了另一件更让人无力的事:德国有一个假冒 VLC 的网站,被举报了十二年,谷歌知道里面是什么,但因为那个二进制文件太大、超出他们病毒分析器的处理上限,所以一直没动。那个网站用德语,在德国的搜索排名比 VideoLAN 官网还高。装上去之后三周内什么都不做,三周后才有一个作为服务安装的小程序苏醒,开始下载间谍软件和广告软件。这三周的静默期就是它躲过检测的设计。

档案:让一千年后的人还能按下播放

Kieran 说开源多媒体里最酷的社群之一是档案保存社群,主要由 Dave Rice 推动,他在纽约市立大学。这个群体预算紧张,但他们清楚 FFmpeg 是那块罗塞塔石碑,是让多媒体在一千年后还能被播放的东西。

他举了一个警示性的例子:英国当年做过一个“新末日审判书”项目,把大量内容存在 BBC Micro 微机上,十来二十年后没人有能读它的软件,得有人去逆向。

二十年就要逆向。想象一千年。

这个社群资助了 FFV1 的开发,一个无损编解码器。他们的顾虑很直接:有损压缩可能会改变素材呈现的样子,某个角落里有东西,被压掉了。所以既要不损失,又要编码够快。他们资助了 FFmpeg 里 FFV1 的 GPU 编码来提速。他们还特别在乎容错:存在磁带或者硬盘上,坏了几位要能快速恢复,不能因为丢了一位就整段 GOP 全废。

JB 说他每次跟 Dave Rice 或者英国电影学院的人聊,都能学到新东西,而他做视频二十年了。对方最懂的是色度学和颜色,还有存储介质。他提到胶片扫描仪这个话题,说光这一个题目能再讲五小时。

Kieran 说这个社群还把工作流免费教给没有档案机构的国家,在印度教小孩写 FFmpeg 命令。他把他们称为我们想达成的模式的样板社群。

一千年后我们可能淹在 AI 生成的内容垃圾里。这些东西需要被认真保存。当时的生活是什么样的?(Kieran Kunhya)

Lex 把这层意思说得更明确:二十世纪到二十一世纪是一个转折点,从数据稀缺转向内容过剩,这个转折本身值得被保存。

而现状是紧迫的。三十、四十、五十年代有大量影片已经失去了价值载体;七十、八十年代的磁带还在,但全世界的磁带读取机磁头不够用,读不完所有带子。所以他们必须做取舍,决定哪些数字化,哪些扔掉。

这是人类历史的数字记录,而他们要做的决定里有一种巨大的道德风险。(Kieran Kunhya)

Kieran 还给了 FFmpeg 会被保留的一个技术理由:它是 C 写的。C 是最接近数学和逻辑的语言,一千年后 C 大概会像拉丁语,是你要从过去学的东西,但仍然可用。

未来:气味、触觉和脑机接口

JB 给的五年十年预测很轻松,难的是更远。问题在于会不会出现全息,会不会出现别的东西。

Lex 把话题推到 Neuralink 那种程度:如果多媒体的定义变成“大脑通过脑机接口想消费的任何数据”,那编解码器长什么样。

Kieran 直接接了一句,会有 FFmpeg -i 输入格式 human brain,用来压缩神经信息。

JB 说这不是纯想象。今天已经有针对点云、体积视频的编解码器,有大量关于 RGB-D 的研究,也就是带深度信息的视频,深度编码对机器人和 3D 场景很有用。VLC 已经有 VR 和 XR 版本,Kyber 也在为算力不足的眼镜、Apple Vision、Quest 做 XR 内容的流式传输。

他给多媒体下了一个可以外推的定义:多媒体是对人类多种感官的数字表示。所以如果将来有气味传感器和气味扩散器,它就会进 FFmpeg,你的解复用器会多出一种叫“气味”的轨类型。

Kieran 顺着玩:像音频一样,左鼻孔和右鼻孔两条轨。

VLC 其实已经有触觉的插件,服务于 4D 影院那种液压座椅,有同步的数据流传输运动信息。这个模块不在正常版本的 VLC 里。

Lex 问标准怎么形成。JB 描述了一个几乎每次都重复的曲线:新领域出现,人人都做,冒出五种标准;热度降下去;然后两三家非领导者的公司推动做标准;然后我们实现标准;然后曲线结束。他举了 3D 音频的例子:六七年前满世界都是 3D 音频,安卓上有 Cardboard,有两种音频格式,全死了,现在带着真实用例回来,并且吸收了上一轮的教训。

Lex 顺口提到 JB 对杜比意见不小。JB 的回答挺克制也挺重:这曾经是一家了不起的公司,有着了不起的工程师,他们定义了声音是什么。现在主要是律师和授权业务。他们不像过去那样创新了。

火星、CERN、围场,和那些奇怪的落点

Lex 问最得意的推特。Kieran 说两条。一条是“Talk is cheap, send patches”,因为它概括了一个基本事实:东西不会从以太里冒出来,得有一个人去做。另一条是“FFmpeg, nothing is beyond our reach”,那句话来自一颗美国军用卫星的补丁。

JB 提了一句 FFmpeg 跑在火星车上。Kieran 确认了:Mars 2020 的火星车用 FFmpeg 压缩图像,NASA 为此写过论文,他们的原则是尽量用现成的商用技术。所以这个库是多行星的。

VLC 的落点更杂。JB 说他们经常看到有人在奇怪的地方用 VLC:F1 各个围场里的人在用它播直播流;欧洲空间局;SpaceX 用它监视发射。

他自己经历过的最好的一次是去 CERN。大型强子对撞机的环周长 27 公里,上面布满了各种传感器和模拟摄像头,CERN 用采集卡把模拟信号转进 VLC,再由 VLC 推到他们自己的组播网络上,让全 CERN 都能看。2010 年他和 Laurent 一起去,一小时就修好了他们的问题,大概是个当时文档没写清楚的参数。修完之后对方问“今天你想干什么”,然后带他们看了个遍,反物质、对撞机,全看了。

那是我物理学背景里最神奇的一天之一。(Jean-Baptiste Kempf)

还有那个关于代码本身的故事。JB 小时候想当工程师,想做汽车,中间在意大利佛罗伦萨住过一年,数学老师让他们用 Logo 语言画乌龟。他自己说,也许将来某天会回去做车。

Kieran 的第一门语言是微软 QBasic,在 Windows 3.1 和 95 上写 10、20、30、40 的循环,然后画了一堆草图,觉得自己能做一个足球游戏。他后来承认,从 BASIC 加几张画跳到一款真正的游戏,中间的工作量他当时完全没概念。

遗憾是一种税

Lex 问 Kieran 有没有后悔发过的推特,然后自己接了那首法国歌:Je ne regrette rien。

JB 接过来说了一段可以直接抄下来贴在显示器上的话。

不要后悔。因为后悔是对你大脑征税。从错误里学习,但不要后悔。事情已经发生了,除非你有时光机,否则后悔只会消耗你的脑子。学习,可以。后悔,不必。(Jean-Baptiste Kempf)

Lex 说这让他想起约翰尼·德普那句关于恨的话,恨是一种很贵的情绪。JB 的反应是:你是在把我比作约翰尼·德普吗,那会是你今天的第一个错误。

整期节目里,两个人都没有把开源说成一种道德姿态。他们反复讲的是一组非常具体的约束:五到八个人维护着几十亿人用的东西;来了又走的人占 99%;一个补丁的质量决定这个项目还能撑多久;一封夹着面粉的信可以来自一个因为 PowerPC 停产而愤怒的人。

最后 Lex 收尾时说,他感谢这些软件,感谢那些辛辣的推特,请永远不要停,也感谢那顶让他看起来像个巫师的帽子。Kieran 和 JB 道谢。节目用林纳斯·托瓦兹的一句话作结:大部分优秀程序员写程序,不是为了拿钱或者被公众追捧,是因为写程序本身好玩。

如果你读到这里想做点什么

这期播客里最扎人的一件事实是,地球上大部分视频流量经过的那套代码,核心维护者只有十到十五个人,而且全是志愿者。Kieran 说得很直白,捐款增加了很多,但仍然不足以覆盖一个全职开发者。如果你从这套东西里得到过任何东西,下面这些是可以立刻做的。

  • 给 FFmpeg 和 VideoLAN 捐款。Kieran 在节目里被 Lex 提到捐款时说的第一句话就是“谢谢”。金额不重要,重要的是让“这件事有人在管”变成一个可持续的事实,而不是靠几个人的热情和失眠撑着。
  • 去关注 @FFmpeg 和 @VideoLAN 这两个账号。不是为了看吵架,是因为这两个账号是目前最能迫使大公司在流程上低头的杠杆。JB 说得很清楚:他们太小了,唯一有效的力量就是社交网络上喊话,因为会滚雪球。
  • 下载任何软件时手动确认域名。VLC 在德国有一个持续了十二年、至今还在的假冒网站,搜索排名比官网还高,安装后三周才开始下载间谍软件,这个延时本身就是它躲过检测的手段。谷歌知道,没动。这个坑对所有开源软件都成立。
  • 如果你会 C,去看 Kieran 那套 asm-lessons。他明确说了门槛只有高中数学和指针,不是完整的 C。这套课按 FFmpeg 的实际做法重新组织,不逐个讲指令语法,直接上手解决真实问题。已经有 16 岁的人学完之后提了补丁。
  • 如果你是工程师,找一个你真在乎的东西去做贡献,不要为了简历。JB 的原话是做你热爱的东西,他自己是因为爱看电影才做 VLC,社区里有大量人是因动画进来的。你也别指望能留下来,统计上留下来的概率是 1%,但那 1% 决定了整套基础设施能不能继续存在。
  • 如果你在决定要不要重写一个别人写的系统:先别。写代码比读代码容易一个数量级,你觉得它烂,很可能只是因为你还没读懂那些没有文档记录的取舍。
  • 如果你在公司里跟开源项目对接,确认一下你们有没有 OSPO,以及它有没有把“这不是我们的供应商”这件事讲给业务团队听。微软 Teams 那个几千美元一次性买断长期支持的报价,根源不是恶意,是一个中层管理者从来没被告知过这套东西是怎么运转的。
  • 如果你的工作涉及长期保存数据,去看 FFV1 和档案社群的工作流。他们面对的约束比大多数工程团队都严苛:无损、可验证、单比特翻转要能局部恢复、一千年后还得有人能读。他们是怎么在这些约束下做取舍的,比任何架构课都实用。
中英对照:英文为官方 Transcript 原文,中文为逐句机器翻译(Google 神经机器翻译),仅供学习英文参考,请以英文原文为准。
Jean-Baptiste Kempf00:00

The important thing is, is your code good?

重要的是,你的代码好不好?

00:00 We care about excellent code.

我们关心优秀的代码。

We don’t care who you are.

我们不在乎你是谁。

Like maybe you’re a dog.

就像你可能是一只狗一样。

I don’t care, right?

我不在乎,对吧?

I don’t care where you come from.

我不在乎你来自哪里。

I need to look at your code.

我需要看看你的代码。

Oh, yeah, but I’m an engineer at this very large company in Italy, in Germany, in the US.

哦,是的,但我是意大利、德国和美国这家非常大的公司的工程师。

We don’t care.

我们不在乎。

We care about the quality of your code because this is what defines our community, which means that we have a lot of people who contribute who are from very different backgrounds and very introverts.

我们关心您的代码质量,因为这是我们社区的定义,这意味着我们有很多来自不同背景且非常内向的人做出贡献。

Sure.

当然。

But that’s okay, right?

但没关系,对吧?

Lex Fridman00:31

FFmpeg is probably one of the biggest CPU users in the world.

FFmpeg 可能是世界上最大的 CPU 用户之一。

00:31 Everything we’ve just said in the past couple of minutes, every sentence is someone’s lifetime’s work.

我们在过去几分钟里所说的一切,每一句话都是某人一生的工作。

There are books about every sentence.

每句话都有书。

So the level of complexity in many cases is inordinate.

因此,在许多情况下,复杂程度是过度的。

Jean-Baptiste Kempf00:45

FFmpeg has one hundred thousand lines of assembly for all the codecs.

FFmpeg 为所有编解码器提供了十万行汇编代码。

Lex Fridman00:50

For all codecs.

对于所有编解码器。

Jean-Baptiste Kempf00:50

And just this one has two hundred and forty thousand.

而光是这个就有二十四万。

00:50 Every cycle matters.

每个周期都很重要。

We are talking about probably three billion devices which are going to decode video nonstop because, for example, thirty percent of the video from Netflix are now in AV1, fifty percent of YouTube.

我们正在谈论的可能有 30 亿台设备将不间断地解码视频,因为例如 Netflix 的 30% 的视频现在采用 AV1 格式,YouTube 的 50% 的视频采用 AV1 格式。

Lex Fridman01:07

This is what peak video codecs should look like.

这就是峰值视频编解码器应该有的样子。

01:07 Seventy-nine point nine percent assembly, nineteen point six percent C, and zero point five percent other.

百分之七十九点九的装配,百分之十九点六的 C,以及百分之零点五的其他。

Jean-Baptiste Kempf01:18

And what’s incredible is with those tweets, which are factual, people go crazy.

令人难以置信的是,这些真实的推文让人们疯狂。

Lex Fridman01:25

For the last two years, they go crazy.

过去两年,他们疯狂了。

01:25 No, intrinsics is fine.

不,内在函数很好。

The compiler-

编译器-

Jean-Baptiste Kempf01:28

You can optimize your compiler.

您可以优化您的编译器。

01:28 Auto-vectorization, it’s your fault.

自动矢量化,这是你的错。

You don’t understand.

你不明白。

And we’ve tried that forever, right?

我们一直在尝试,对吧?

Lex Fridman01:35

For two years, and two years later, showing hundreds of examples of handwritten assembly.

两年来,两年后,展示了数百个手写汇编的示例。

01:35 No, no, no, you’re doing it wrong.

不不不,你做错了。

The compiler can do this.

编译器可以做到这一点。

The intelligence agencies tried to, like, say, “Can you put a backdoor in VLC?”

情报机构试图说,“你能在 VLC 中设置后门吗?”

Jean-Baptiste Kempf01:48

Yes.

是的。

01:48 Two of them.

其中两个。

Lex Fridman01:50

Well, what did you say?

嗯,你说什么?

Jean-Baptiste Kempf01:51

No.

不。

01:51 Well, I was a lot less polite.

嗯,我没那么有礼貌。

Lex Fridman01:54

Basically saying, “Hell no.”

基本上是说:“天哪,不。”

Jean-Baptiste Kempf01:56

Like, if we had to compromise our software, we would shut it down.

就像,如果我们必须破坏我们的软件,我们就会将其关闭。

01:56 This is clear.

这很清楚。

Lex Fridman02:00

Any tweets, Kieran, you regret?

Kieran,你后悔发过任何推文吗?

Kieran Kunhya02:04

Tweets I regret?

我后悔的推文?

Lex Fridman02:05

Or is it like that, how does the French song go?

或者是这样,法语歌曲怎么样?

02:05 Regret nothing.

没什么遗憾的。

Jean-Baptiste Kempf02:08

Don’t regret anything.

不要后悔任何事情。

02:08 No, it’s because regrets are attacks on your mind.

不,这是因为后悔是对你心灵的攻击。

Lex Fridman02:17

The following is a conversation all about FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya.

以下是与 Jean-Baptiste Kempf 和 Kieran Kunhya 的有关 FFmpeg 和 VLC 的对话。

02:17 FFmpeg is an open source software system that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord, and basically every platform that touches video or audio on the internet.

FFmpeg 是一个开源软件系统,是 YouTube、Netflix、Chrome、VLC、Discord 以及基本上每个涉及互联网上视频或音频的平台背后的隐形支柱。

It can decode, encode, transcode, stream, and play almost any video or audio format ever created.

它可以解码、编码、转码、流式传输和播放几乎任何创建的视频或音频格式。

To me, it is one of the most incredible software systems ever developed, and it’s all done by volunteers.

对我来说,它是有史以来开发的最令人难以置信的软件系统之一,而且这一切都是由志愿者完成的。

VLC is also a legendary piece of software.

VLC也是一款具有传奇色彩的软件。

It is an open source media player that plays basically anything you throw at it, any format, any platform, no ads, no tracking.

它是一个开源媒体播放器,基本上可以播放你扔给它的任何内容,任何格式,任何平台,没有广告,没有跟踪。

03:11 It has been downloaded over six billion times, and again, for me, it has been one of my favorite pieces of software ever, with the most legendary logo, which I, of course, had to honor in this conversation by wearing the VLC traffic cone hat the whole time.

它的下载次数已超过 60 亿次,对我来说,它再次成为我最喜欢的软件之一,拥有最具传奇色彩的徽标,当然,我必须在这次谈话中一直戴着 VLC 交通锥帽来纪念这一点。

So again, above all else, thank you to the incredible volunteer engineers who put their heart and soul into this code that has been used and loved by billions of people.

所以,最重要的是,再次感谢那些令人难以置信的志愿工程师,他们全心全意地投入到这个被数十亿人使用和喜爱的代码中。

Thank you.

谢谢。

And about the two great engineers and human beings I’m talking to in this episode, Jean-Baptiste is the president of VideoLAN and is a key figure behind VLC and FFmpeg.

关于我在本集中谈论的两位伟大的工程师和人类,Jean-Baptiste 是 VideoLAN 的总裁,也是 VLC 和 FFmpeg 背后的关键人物。

03:58 Kieran is a longtime codec engineer, FFmpeg contributor, and the man behind the now infamous FFmpeg account on Twitter/X that I recommend everybody follow for the memes and for the unapologetic celebration of open source and great low-level software engineering.

Kieran 是一位长期的编解码器工程师、FFmpeg 贡献者,也是现在臭名昭著的 Twitter/X 上 FFmpeg 帐户的幕后黑手,我建议每个人都关注该帐户,以获取模因并毫无歉意地庆祝开源和伟大的低级软件工程。

Let me also say that it’s inspiring and humbling that so much of modern civilization rests on software built by people who are not chasing fame or money, but are obsessed with the craft of engineering.

我还要说,现代文明在很大程度上依赖于那些不追逐名誉或金钱,而是痴迷于工程技术的人们构建的软件,这令人鼓舞和谦卑。

We live in a world where billions of people consume video every day without ever thinking about the invisible machinery underneath it.

我们生活在一个每天有数十亿人观看视频的世界,却从未考虑过视频背后的隐形机制。

But that machinery matters.

但那台机器很重要。

Open source infrastructure matters.

开源基础设施很重要。

04:46 It is one of the great examples of human beings quietly collaborating across borders to build something useful, durable, and elegant for the rest of us.

这是人类默默地跨国界合作,为我们其他人打造有用、耐用和优雅的东西的伟大例子之一。

And so this conversation is not just about codecs and media pipelines.

因此,这次对话不仅仅涉及编解码器和媒体管道。

It is also about the deeper spirit of engineering and generosity that makes projects like FFmpeg possible.

它还涉及更深层次的工程精神和慷慨,使得 FFmpeg 这样的项目成为可能。

Again, I can never say it enough.

再说一遍,我永远都说不够。

Thank you.

谢谢。

This is the Lex Fridman Podcast.

这是莱克斯·弗里德曼播客。

To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback, and so on.

为了支持它,请在描述中查看我们的赞助商,您还可以在其中找到联系我、提出问题、提供反馈等的链接。

And now, dear friends, here’s Jean-Baptiste Kempf and Kieran Kunhya.

现在,亲爱的朋友们,这里是让-巴蒂斯特·肯普夫和基兰·昆亚。

05:35 So the legend goes VLC can open everything.

所以传说VLC可以打开一切。

What’s the weirdest thing that you know that it can open?

你知道它能打开的最奇怪的东西是什么?

Jean-Baptiste Kempf05:43

You know, there is a ton of people who are using VLC to record VHS videos, right?

您知道,有很多人使用 VLC 录制 VHS 视频,对吗?

05:43 Like, it’s just like you plug it with a capture card and you can basically record VHS video.

就像,就像你插上采集卡一样,你基本上可以录制 VHS 视频。

Lex Fridman05:52

Well, how does that work?

嗯,那是如何运作的呢?

Jean-Baptiste Kempf05:53

Basically, it’s, you know, those type of capture cards where you can put a Peritel in or- …

基本上,你知道,它是那种类型的采集卡,你可以在其中放入 Peritel,或者-...

05:53 or RCA, and you put that, and actually VLC can play those type of cards, and there is a module which allows to control directly some of those VCR camcorders.

或 RCA,实际上 VLC 可以播放这些类型的卡,并且有一个模块可以直接控制其中一些 VCR 摄像机。

We support DVD-Audio lately, right?

我们最近支持 DVD 音频,对吗?

We spent the summer working on DVD-Audio support, and like there is no, no one’s making any DVD-Audio support.

我们整个夏天都在致力于 DVD 音频支持,但就像没有一样,没有人提供任何 DVD 音频支持。

There are custom encryption schemes.

有自定义加密方案。

Lex Fridman06:19

What about Lucasfilm?

卢卡斯影业呢?

Jean-Baptiste Kempf06:20

Oh, yeah, and there is of course all the weird codecs support, game codecs supported by FFmpeg.

哦,是的,当然还有所有奇怪的编解码器支持,FFmpeg 支持的游戏编解码器。

Lex Fridman06:25

The one Star Wars video game, the first ten-second opening sequence, someone has gone and implemented that and made sure that’s bit exact on one disc that existed at one time of one little sequence in the game.

在一款《星球大战》游戏中,第一个十秒的开场序列,有人已经实施了这一点,并确保它在游戏中一个小序列的某个时间存在的一张光盘上是准确的。

Jean-Baptiste Kempf06:36

And then funnily, there was a…

然后有趣的是,有一个……

06:36 At one VideoLAN conference, we made a competition to make the weirdest and most horrible file ever- …

在一次 VideoLAN 会议上,我们举办了一场竞赛,要求制作有史以来最奇怪、最可怕的文件——……

and see if VLC could play it.

看看VLC能不能播放。

Lex Fridman06:46

What did it end up being?

它最终变成了什么?

06:46 What’s the file?

文件是什么?

Jean-Baptiste Kempf06:48

It was an MKV file made by Derek- …

这是 Derek 制作的 MKV 文件-...

06:48 which each of the frame was changing resolution, aspect ratio, rotation, and…

每个帧都在改变分辨率、宽高比、旋转和……

and it was like-

就像-

Lex Fridman06:59

Did it work?

有效吗?

Jean-Baptiste Kempf07:00

Yes.

是的。

07:00 And there was another one where the whole video was actually animated subtitles, right?

还有另外一个,整个视频实际上都是动画字幕,对吗?

SSA, right?

SSA,对吗?

So-

所以——

Lex Fridman07:08

Yeah.

是的。

07:08 I remember that, yeah.

我记得,是的。

Jean-Baptiste Kempf07:08

…

……

each, each, this one was-

每一个,每一个,这个是——

Lex Fridman07:10

It-

它——

Jean-Baptiste Kempf07:10

And so each frame was a black frame, but on top of that there was a subtitle that was animated for each frame.

因此,每一帧都是一个黑色帧,但在其之上,每个帧都有一个动画字幕。

Lex Fridman07:16

There was a file that’s a valid ZIP and a valid MP3 at the same time or something like that, so.

有一个文件同时是有效的 ZIP 和有效的 MP3 或类似的文件,所以。

Jean-Baptiste Kempf07:20

So yeah, we made a competition of stupid files.

所以,是的,我们举办了一场愚蠢文件的竞赛。

Lex Fridman07:23

And it worked.

它奏效了。

07:23 It opened all of the stupid files.

它打开了所有愚蠢的文件。

Jean-Baptiste Kempf07:27

Yes.

是的。

Lex Fridman07:27

By the way, for people who are not familiar, I am wearing a hat.

顺便说一句,对于不熟悉的人来说,我戴着帽子。

07:27 Would it be fair to say this is the best worst logo of all time, the cone?

可以公平地说,这是有史以来最好的最糟糕的标志,圆锥体吗?

Jean-Baptiste Kempf07:36

Yeah, by far, right?

是的,到目前为止,对吧?

07:36 The logo of VLC is so iconic, right?

VLC的标志太有标志性了吧?

Like we are a team with a small number of people and the icon is known everywhere.

就像我们是一个人数不多的团队,而图标却到处都是。

I go to middle of nowhere in India or in China, people know the cone, right?

我去印度或中国的偏僻地方,人们都知道锥体,对吧?

And 25% of the website traffic that comes to our main website is cone player, right?

我们主网站 25% 的网站流量是锥体播放器,对吧?

So many people don’t know VLC, right?

很多人不知道VLC,对吧?

They know the cone player.

他们认识锥体玩家。

Lex Fridman07:59

That’s the thing they Google for is “cone player.”

他们在谷歌上搜索的就是“锥形玩家”。

Jean-Baptiste Kempf08:01

Yeah.

是的。

08:01 They go on Google and they put “cone player” and they download VLC, right?

他们在 Google 上输入“cone player”,然后下载 VLC,对吧?

So that’s iconic.

所以这是标志性的。

And once we tried to change it as a joke, right?

一旦我们试图把它变成一个笑话,对吗?

We said it was going to be a type of caterpillar construction and we said that during April 1st- …

我们说这将是一种履带式建筑,我们说在 4 月 1 日期间——……

and we had around 10,000 emails saying, “No, don’t change the logo,” and so on, right?

我们收到了大约 10,000 封电子邮件,内容是“不,不要更改徽标”等等,对吗?

So it’s so iconic, right?

所以它非常具有标志性,对吧?

It’s so distinctive, right?

太有特色了吧?

If you want to do a video player, you’re going to put a play button on a TV, right?

如果你想做一个视频播放器,你会在电视上放一个播放按钮,对吗?

And that’s a YouTube logo, right?

那是 YouTube 徽标,对吗?

It’s unoriginal.

这是非原创的。

This one is orange, right?

这个是橙色的吧?

It’s very bright and it’s weird.

它非常明亮而且很奇怪。

Lex Fridman08:37

And it’s ridiculous and it’s absurd and it’s hilarious.

这很荒谬,很荒唐,很搞笑。

08:37 It becomes meme and meme becomes culture.

它变成了模因,模因变成了文化。

Yeah.

是的。

Jean-Baptiste Kempf08:41

And you keep it and you know about it and you know that in 20 years, like you still have, going to have the cones and remember, oh yeah, that was a video player.

你保留着它,你知道它,你知道 20 年后,就像你仍然拥有的那样,你会拥有视锥细胞并记住,哦,是的,那是一个视频播放器。

Lex Fridman08:49

Yeah.

是的。

08:49 And we’ll talk about, you know, the mission of FFmpeg being a kinda the archival aspect of it.

我们将讨论 FFmpeg 的使命,即它的档案方面。

So you can think about 1,000 years from now we’ll have all these videos that only VLC can open.

所以你可以想象 1000 年后我们将拥有只有 VLC 才能打开的所有这些视频。

Humans, human civilization has already destroyed itself multiple times and the only thing that will remain is this like, you know, the cockroaches will be crawling around and it’ll be the VLC logo- …

人类,人类文明已经多次自我毁灭,唯一剩下的就是,你知道,蟑螂会爬来爬去,那就是 VLC 标志——……

with our, some of the archival footage that VLC can open.

使用我们的 VLC 可以打开的一些档案片段。

And the aliens will show up and they’ll press play and they’ll get to see it all ’cause-

外星人会出现,他们会按下播放按钮,他们会看到这一切,因为-

Jean-Baptiste Kempf09:18

Well, really, really hope so, right?

嗯,真的,真的希望如此,对吧?

09:18 But there is also so many memes where people say, “Well, I’m sure I can put a pancake inside my DVD drive and VLC will play it.” Like-

但也有很多模因让人们说:“好吧,我确信我可以将煎饼放入 DVD 驱动器中,VLC 就会播放它。”就像-

Lex Fridman09:25

Can they?

他们可以吗?

Jean-Baptiste Kempf09:26

No, we tried.

不,我们尝试过。

09:26 It doesn’t.

事实并非如此。

Lex Fridman09:27

Doesn’t

没有

Jean-Baptiste Kempf09:28

…

……

09:28 but we actually have a video of us trying that.

但我们实际上有一个我们尝试这样做的视频。

Didn’t work.

没用。

Lex Fridman09:31

A codec for physical reality, I don’t know what that would even look like.

物理现实的编解码器,我不知道它会是什么样子。

Jean-Baptiste Kempf09:34

There was a guy who did that, right?

曾经有一个人这么做过,对吗?

09:34 He printed a small cone, right?

他打印了一个小圆锥体,对吗?

Like the ones we distribute as goodies and inside he put an RFID chip which was his way of playing a movie, right?

就像我们分发的礼物一样,他在里面放了一个 RFID 芯片,这就是他播放电影的方式,对吗?

And so he- …

于是他——

put this on a RFID player and when he put that it was playing like The Last Star Wars and so on.

把它放在一个 RFID 播放器上,当他放上它时,它就会像《最后的星球大战》那样播放。

So instead of having like DVD boxes, he had like VLC cones all around and he plugged that and that was like physical objects.

因此,他没有像 DVD 盒子那样,而是像 VLC 锥体一样到处都是,他将其插入,就像物理对象一样。

Lex Fridman09:59

So the thing that we’re talking about is everything around video codecs, video encoding, video decoding, video streaming, video player client that I’m wearing on my head, the entire ecosystem enabling free media.

所以我们谈论的是围绕视频编解码器、视频编码、视频解码、视频流、我头上戴着的视频播放器客户端、支持自由媒体的整个生态系统的一切。

09:59 We’ll talk about FFmpeg, we’ll talk about VideoLAN, VLC and all the other incredible video technology that is used probably by billions of people.

我们将讨论 FFmpeg,我们将讨论 VideoLAN、VLC 以及可能被数十亿人使用的所有其他令人难以置信的视频技术。

So JB, you’re the lead developer behind the legendary VLC player.

JB,您是传奇 VLC 播放器背后的首席开发人员。

Kieran, amongst many other things, you’re lead developer behind the legendary FFmpeg handle on Twitter.

Kieran 是 Twitter 上传奇的 FFmpeg 手柄的首席开发人员。

And both of you have spicy opinions I would say.

我想说你们俩都有辛辣的意见。

So today we wanna talk about FFmpeg and VLC.

所以今天我们想谈谈 FFmpeg 和 VLC。

10:45 For context for people who are not aware—and I’m sure basically everybody listening to this have used these two technologies probably regularly without knowing it.

对于那些不知道的人来说,我确信基本上每个听这个的人都可能经常在不知情的情况下使用这两种技术。

So FFmpeg underlies basically most video on the internet including YouTube, Netflix, Chrome, Firefox, of course VLC and countless other video platforms.

因此,FFmpeg 基本上是互联网上大多数视频的基础,包括 YouTube、Netflix、Chrome、Firefox,当然还有 VLC 和无数其他视频平台。

It is estimated that over 90% of video processing workflows online and offline involve FFmpeg.

据估计,超过 90% 的在线和离线视频处理工作流程都涉及 FFmpeg。

VLC has been downloaded at least 6.5 billion times.

VLC 已被下载至少 65 亿次。

But likely that number, ’cause it’s impossible to really count the number, is much higher than that.

但这个数字很可能远高于这个数字,因为不可能真正计算出这个数字。

Virtually any operating system supports virtually any media format.

几乎任何操作系统都支持几乎任何媒体格式。

The limitation being it can’t open pancakes.

限制是它不能打开煎饼。

So can we just lay out some of the basics to, to help people understand what’s involved in all of this?

那么我们能否列出一些基础知识来帮助人们理解这一切所涉及的内容?

11:45 So when we press play on a video player like VLC, what happens?

那么,当我们在 VLC 等视频播放器上按下“播放”时,会发生什么?

How does it go from the file or the stream to the pixels on the screen and the sound on the speaker?

它是如何从文件或流传输到屏幕上的像素和扬声器上的声音的?

What are the big stages to be aware of?

有哪些大阶段需要注意?

Jean-Baptiste Kempf12:01

So there are several stages, right?

所以有几个阶段,对吗?

12:01 The first stage is to get from an address, right, which is the type of URL, to give you a byte of streams, right?

第一阶段是从一个地址获取,对吧,这是 URL 的类型,给你一个字节的流,对吧?

So this would be, for example, HTTP, file, DVD, right?

例如,这将是 HTTP、文件、DVD,对吗?

You give the path to the media, and it gives you a stream of data.

你给媒体路径,它给你数据流。

Kieran Kunhya12:19

The stream needs to be cut up by what’s known as the container, the demultiplexer or demux.

流需要被所谓的容器、解复用器或解复用器分割。

12:19 We’ll try and keep the jargon light throughout this, but it needs to go and start demarcating video and audio frames.

我们将尝试在整个过程中保持行话的简洁性,但它需要开始划分视频和音频帧。

So it just gets data from the operating system blocks at a time and needs to start cutting these frames up into compressed data.

因此它一次只是从操作系统块获取数据,并且需要开始将这些帧切割成压缩数据。

It then needs to start doing simple parsing of the video frames-

然后它需要开始对视频帧进行简单的解析 -

12:40 …

……

mainly to figure out whether that codec is GPU decodable or needs to fall back to software.

主要是为了弄清楚该编解码器是否可以 GPU 解码或需要回退到软件。

We’re very sort of used to assuming the GPU will play all of these things.

我们已经习惯于假设 GPU 能够处理所有这些事情。

There’ll be hardware acceleration.

将会有硬件加速。

I think it’s up to forty-five percent of files are not GPU decodable.

我认为高达 45% 的文件无法由 GPU 解码。

So these need to be probed.

所以这些都需要探究。

They need to be detected.

他们需要被发现。

There can be variants of a given codec, some of which are decodable on the GPU.

给定编解码器可以有多种变体,其中一些可以在 GPU 上解码。

Different vendors of GPU might have different capabilities, so those need to be detected.

不同供应商的 GPU 可能具有不同的功能,因此需要检测这些功能。

So if, if it’s GPU capable, you pass it through to the GPU black box.

因此,如果它具有 GPU 功能,您可以将其传递到 GPU 黑匣子。

So now if there’s a software fallback, that means in the beginning is, is to first do deentropy coding, so removing the mathematical coding of the bit stream.

所以现在如果有软件后备,这意味着一开始就是首先进行去熵编码,从而删除比特流的数学编码。

13:22 So this uses capabilities such as Huffman coding or arithmetic coding to actually decompress the mathematical layer of the bit stream.

因此,这使用霍夫曼编码或算术编码等功能来实际解压缩比特流的数学层。

We then need to start reading the syntax elements for intra prediction.

然后我们需要开始读取帧内预测的语法元素。

So intra prediction are like still images of the video, so your I-frames.

因此,帧内预测就像视频的静态图像,即 I 帧。

So this works and operates in the spatial domain.

所以这是在空间域中起作用和运作的。

So you do your intra prediction in spatial domain.

因此,您可以在空间域中进行帧内预测。

You have a residual because your prediction isn’t quite matching that of reality.

你有残差,因为你的预测与现实不太相符。

So you’ve made a prediction, but then there’s a little bit left, and that’s what’s known as the residual.

所以你已经做出了预测,但还剩下一点,这就是所谓的残差。

This is stored in the frequency domain, and these are quantized to decompact their space.

这存储在频域中,并且这些被量化以解压缩它们的空间。

We then need to do the inverse transform to bring them back to the spatial domain and apply these residuals.

然后,我们需要进行逆变换,将它们带回空间域并应用这些残差。

Lex Fridman14:05

So a lot of the process of the decoding is this thing is compressed.

所以解码的很多过程就是把这个东西压缩了。

14:05 And you have to predict the highest quality thing that’s supposed to go there.

你必须预测应该到达那里的最高质量的东西。

I-frame- …

I 帧-...

is the best representation you have spatially.

是您在空间上的最佳表示。

Kieran Kunhya14:18

Yes.

是的。

Lex Fridman14:19

And then there’s a lot of temporal compression that can happen depending on the codec, and then you’re predicting.

然后,根据编解码器的不同,可能会发生大量时间压缩,然后您就可以进行预测。

14:19 You’re predicting what the reality that was captured in this rawest form.

你正在预测以这种最原始的形式捕捉到的现实。

Jean-Baptiste Kempf14:29

Yeah, because what people don’t realize is that the compression on video and audio is one hundred times, right?

是的,因为人们没有意识到视频和音频的压缩是一百倍,对吧?

14:29 Like, people don’t realize how compressed we do, right?

就像,人们没有意识到我们有多压缩,对吧?

For audio, you move, you compress by, when you go from normal audio to MP3, you compress by ten times, right?

对于音频来说,你移动,你压缩,当你从普通音频到MP3时,你压缩十倍,对吧?

When you move to video, you need one hundred times, two hundred times, right?

当你转向视频时,你需要一百次、两百次,对吧?

So you need to remove all the details, but that you don’t care about because all the compressions that we do, and that’s very important, people forget about that, is to be viewed by humans, right?

所以你需要删除所有细节,但你不关心这些细节,因为我们所做的所有压缩,这非常重要,人们忘记了这一点,是为了被人类查看,对吧?

15:02 So all the codecs, either for audio, mimic basically how your ear works, right?

所以所有的编解码器,无论是音频编解码器,基本上都会模仿你的耳朵的工作方式,对吗?

And a lot of things about, like, the response on the ear and same for your eyes, right?

还有很多关于耳朵的反应和眼睛的反应,对吧?

And so, for example, on video, we don’t work on RGB, right?

例如,在视频方面,我们不使用 RGB,对吗?

Everyone expect to work in RGB.

每个人都希望在 RGB 模式下工作。

We don’t, right?

我们不这样做,对吧?

We move to YUV, which is basically one is luminance, brightness, and the other are colors.

我们转向 YUV,它基本上一个是亮度、亮度,另一个是颜色。

And this matches your eyes, where inside your eyes you have the cones and the rods, right?

这与你的眼睛相匹配,你的眼睛里面有视锥细胞和视杆细胞,对吗?

With some of them look on brightness and more on the other on colors, right?

其中一些看重亮度,另一些看重颜色,对吗?

So we need to compress a lot, and so we need to degrade.

所以我们需要大量压缩,所以我们需要降级。

But in order to degrade, we need to match the human perception, and this is why it’s so difficult.

但为了降级,我们需要符合人类的感知,这就是为什么它如此困难。

15:43 And then we need to use the maximum power, mathematical power, very complex technologies.

然后我们需要使用最大的力量、数学的力量、非常复杂的技术。

We move to the frequency domain, as Kieran said.

正如基兰所说,我们转向频域。

We do a ton of dequantizing in order to get the best compression, but it still looks good.

我们进行了大量的反量化以获得最佳压缩效果,但它看起来仍然不错。

Lex Fridman15:58

You’re trying to compress in order to maximize the highest quality thing for human perception.

你正在尝试压缩,以便最大限度地提高人类感知的最高质量。

Jean-Baptiste Kempf16:04

That is correct.

这是正确的。

16:04 And this is very important, right?

这非常重要,对吧?

Compression is not like a ZIP, right?

压缩不像 ZIP,对吧?

A ZIP, you have data in, you get data out, right?

ZIP,有数据输入,也有数据输出,对吗?

And you try with all the ZIP compression to arrive with the limit.

然后您尝试使用所有 ZIP 压缩来达到限制。

Here we are degrading the signal, right?

这里我们正在降低信号,对吗?

And so we need to degrade both the audio and the video signal in the best way possible.

因此,我们需要以尽可能最好的方式降低音频和视频信号的质量。

And we can do that, but it involves, first, a lot of theoretical knowledge about how the eye works, but a lot of mathematical change, a lot of mathematical tricks, right?

我们可以做到这一点,但它首先涉及大量有关眼睛如何工作的理论知识,但也涉及大量数学变化,大量数学技巧,对吗?

For example, when you move to RGB and you go to YUV, for example, what we do very often is that we scale down the resolution of the color compared to the brightness.

例如,当您转向 RGB 并转向 YUV 时,我们经常做的是,与亮度相比,我们会降低颜色的分辨率。

16:48 And most of the time, and just this without compression, it divides the size by two, but most people don’t see it, right?

大多数时候,在没有压缩的情况下,它会将大小除以二,但大多数人看不到它,对吗?

And so on and so on, right?

等等等等,对吧?

And then you go to very complex mathematical change.

然后你会遇到非常复杂的数学变化。

So of course Fourier transforms, which de facto are not Fourier transforms, they are like discrete cosine transforms, but that’s the same idea.

当然,傅里叶变换实际上不是傅里叶变换,它们就像离散余弦变换,但这是相同的想法。

So frequency domain we split the video by blocks, right?

那么频域我们将视频按块分割,对吧?

So that’s why when it’s wrongly decoded, you see those blocks, and badly encoded, you see those blocks, and so on, to arrive to compression states that are insanely high, right?

这就是为什么当它被错误解码时,你会看到这些块,而当它被错误编码时,你会看到这些块,等等,以达到非常高的压缩状态,对吧?

And each generation of the codec is like thirty percent less- …

每一代的编解码器都减少了百分之三十——……

for the same quality, right?

同样的质量,对吗?

And this requires amount of power, of computational power, that are huge.

这需要巨大的能量和计算能力。

Kieran Kunhya17:35

No, no, but you should, you should elaborate.

不,不,但是你应该,你应该详细说明。

17:35 It’s thirty percent better, but an order of magnitude, perhaps, perhaps even two orders of magnitude more compression power.

它好百分之三十,但压缩功率可能高一个数量级,甚至可能高两个数量级。

That’s the big difference.

这就是很大的区别。

Lex Fridman17:45

What do you mean by compression power?

压缩功率是什么意思?

Kieran Kunhya17:47

Sorry, CPU power to achieve that level of compression.

抱歉,CPU 能力无法达到该级别的压缩。

Lex Fridman17:49

Oh, yeah.

哦,是的。

17:49 So and you have to be able to leverage the CPU and sometimes GPU, like you mentioned.

因此,您必须能够利用 CPU,有时还需要利用 GPU,就像您提到的那样。

And then we should mention that a lot of this programming is done at the lowest possible- …

然后我们应该提到,很多编程都是以尽可能低的方式完成的——……

stack, whether it’s C and of course, as, as the legendary- …

堆栈,无论是 C 还是当然,如传说中的那样——……

Twitter handle re-emphasizes over and over, a lot of assembly.

Twitter 句柄一遍又一遍地重新强调,大量的组装。

Jean-Baptiste Kempf18:10

So what happens globally is that you have an address, right?

那么全球范围内发生的事情就是你有一个地址,对吗?

18:10 Which gives you with the operating system, a stream of bytes, a stream of data, right?

它为您提供了操作系统、字节流、数据流,对吗?

And this is the first step.

这是第一步。

And the second step arises with demuxing, where you’re going to separate audio, video, subtitle in type of different tracks.

第二步是解复用,您将分离不同轨道类型的音频、视频、字幕。

And then on each of those tracks, you’re going to decompress them, decode them, either audio with an audio codec, video to video codec, and subtitle to subtitle codec.

然后,在每个轨道上,您将对其进行解压缩、解码,无论是使用音频编解码器的音频、视频到视频编解码器还是字幕到字幕编解码器。

And once you’ve decompressed those type of things, you have raw images, raw, and then you’re going to talk with your graphic card in your screen and display that.

一旦你解压缩了这些类型的东西,你就会得到原始图像,原始图像,然后你将在屏幕上与你的显卡对话并显示它。

And same for the audio, you’re going to talk to your audio card, which then is going to go in analog to your audio speakers.

对于音频来说也是如此,您将与您的声卡对话,然后声卡将模拟您的音频扬声器。

Kieran Kunhya18:50

And everything we’ve just said in the past couple of minutes, every sentence is someone’s lifetime’s work.

我们在过去几分钟里所说的一切,每句话都是某人一生的工作。

18:50 There are books about- …

有一些书是关于——……

every sentence.

每一句话。

So the level of complexity in many cases is inordinate.

因此,在许多情况下,复杂程度是过度的。

You know, it’s…

你知道,这是……

Every sentence has thousands of people working on this- …

每一句话都有成千上万的人在努力——……

in industry as a whole, books written about it.

在整个行业中,有关于它的书籍。

So there’s a lot of detail, there’s a lot of subtleties, there’s a lot of both academic and practical realities, both of which matter.

因此,有很多细节,有很多微妙之处,有很多学术和实践现实,两者都很重要。

Lex Fridman19:20

We mentioned codecs, but I don’t think you mentioned containers.

我们提到了编解码器,但我认为您没有提到容器。

19:20 So what are the actual containers for some of the stuff we’re talking about?

那么我们正在讨论的一些东西的实际容器是什么?

So people are familiar with MP4, MOV, MKV.

所以人们熟悉MP4、MOV、MKV。

So anyway, what are containers versus the thing that goes inside?

那么无论如何,容器与里面的东西是什么?

Jean-Baptiste Kempf19:41

So the container is what we call also the muxer, right?

那么容器就是我们所说的复用器,对吧?

19:41 When I say demuxing, it means decontainerizing, right?

当我说解复用时,它意味着去容器化,对吗?

So actually, if you look, mux means multiplexer and demultiplexer, right?

所以实际上,如果你看一下,mux 意味着复用器和解复用器,对吧?

Mux and demux are those.

复用器和解复用器就是这些。

And same, a codec is actually coder, decoder, right?

同样,编解码器实际上是编码器、解码器,对吗?

And so containers are this collection of multiple tracks, right?

所以容器是多个轨道的集合,对吧?

So it’s a, what normal people call the file format, but it’s a bit more subtle than that.

这就是普通人所说的文件格式,但它比这更微妙。

But the most known one, of course, is MP4, but when I started, it was AVI, right?

但最广为人知的当然是 MP4,但当我开始时,它是 AVI,对吗?

AVI was the video format from-

AVI 是视频格式-

20:17 …

……

from Microsoft, and MOV, M-O-V, which became MP4, was a format from Apple.

MOV、M-O-V(后来成为 MP4)是 Apple 的一种格式。

In the open source community one of the person that is still active on VideoLAN is called Steve Lhomme and sta- started this Matroska format, which is, like, a bit more complex and, and, and more future-proof.

在开源社区中,仍然活跃在 VideoLAN 上的一个人叫 Steve Lhomme,他创建了这种 Matroska 格式,该格式有点复杂,并且更加面向未来。

And there are so many others.

还有很多其他的。

Lex Fridman20:38

So, I mean, there’s a, it’s a pretty common thing, and maybe it’ll even happen in this conversation, that people confuse container and the codec, right?

所以,我的意思是,人们混淆了容器和编解码器,这是很常见的事情,甚至可能会在这次对话中发生,对吧?

20:38 So confuse MP4 and H.264, for example.

例如,请混淆 MP4 和 H.264。

Is that a horrible violation?

这是一个可怕的违规行为吗?

Jean-Baptiste Kempf20:51

No, it’s not, because technically the name of H.264 is MPEG-4 Part 10.

不,不是,因为从技术上来说,H.264 的名称是 MPEG-4 Part 10。

20:51 Because MPEG-4 is actually a meta specification which has several things in it, right?

因为 MPEG-4 实际上是一个元规范,其中包含多种内容,对吧?

There is the Part 2.

有第二部分。

So there is, like, audio codecs, right?

那么有音频编解码器之类的东西,对吗?

AAC de facto is MP4 audio- …

AAC 实际上是 MP4 音频 - ...

something.

某事。

There is actually several video codecs, right, inside the MPEG-4 specification.

MPEG-4 规范中实际上有多种视频编解码器,对吧。

One of them is MPEG-4 Part 10, called also AVC, called also H.264.

其中之一是 MPEG-4 Part 10,也称为 AVC,也称为 H.264。

Right?

正确的?

So it’s completely the fault of the industry to make things difficult to understand.

所以让事情变得难以理解完全是行业的错。

So that’s very difficult so that people then don’t understand why sometimes you talk about MPEG-4 Part 10, where you mean H.264, and why it’s not MP4.

所以这非常困难,以至于人们不明白为什么有时你会谈论 MPEG-4 第 10 部分,其中你指的是 H.264,以及为什么它不是 MP4。

Lex Fridman21:38

So you can technically shove in all kinds of different codecs inside containers and horribly so.

因此,从技术上讲,您可以将各种不同的编解码器放入容器内,而且非常可怕。

Kieran Kunhya21:44

But broadly speaking, though, MP4 is understood to generally be H.264 plus AAC audio.

但从广义上讲,MP4 通常被理解为 H.264 加 AAC 音频。

21:44 99% of the time that’s that, and that, the, the rest are de minimis, the small effects, you know, edge effects really compared to that.

99%的时间就是这样,那个,那个,其余的都是微不足道的,小的影响,你知道,与此相比,边缘效应确实如此。

So it’s not the end of the world.

所以这不是世界末日。

There are people who do get annoyed by that.

有些人确实对此感到恼火。

But also in reality, something like VLC, just to point out, the file may say .MP4, but it may be something completely different, and that’s one of the challenges both FFmpeg and VLC have is the real world is a completely different place to a three-letter file format.

但实际上,像 VLC 这样的文件,只是指出,文件可能会显示 .MP4,但它可能是完全不同的东西,这是 FFmpeg 和 VLC 面临的挑战之一,现实世界与三字母文件格式完全不同。

Jean-Baptiste Kempf22:17

And this is very important to say, right?

这一点很重要,对吧?

22:17 Like, for example, in VLC and in FFmpeg, we discard the file format, right?

例如,在 VLC 和 FFmpeg 中,我们放弃文件格式,对吗?

We look into the file to understand what’s in it because so many people, like, they say, “Oh, it’s a video, it must be MP4,” but technically it’s an MOV or maybe it’s a MKV, right?

我们会研究该文件以了解其中的内容,因为很多人会说,“哦,这是一个视频,它一定是 MP4”,但从技术上讲,它是一个 MOV,或者可能是一个 MKV,对吧?

So we analyze in real time everything that we have, and we don’t trust- …

所以我们实时分析我们拥有的一切,但我们不相信——……

the format.

格式。

Lex Fridman22:42

So what information does the fact that it’s .MP4 give you?

那么 .MP4 的事实给了你什么信息呢?

Jean-Baptiste Kempf22:45

It helps, right?

这有帮助,对吧?

22:45 It gives you a hint, right?

它给了你一个提示,对吗?

Just like, oh, it’s finished by .MP4.

就像,哦,它是用.MP4完成的。

I will start first by opening, probing it with the MP4 container demuxer to see, well, it should be that.

我将首先打开它,用 MP4 容器解复用器探测它,看看,嗯,应该是这样。

But I don’t trust it, and if I’m lost, I say, “Okay, maybe I’m going to try it.” So it bumps the priority of the module.

但我不相信它,如果我迷路了,我会说:“好吧,也许我会尝试一下。”因此它提高了模块的优先级。

Lex Fridman23:06

So how do you get to—just to take a bit of a tangent there.

那么你如何做到这一点——只是在那里切入一点。

23:06 You know, the dumb thing is if you try the MP4, but it turns out it’s a different codec than you would have expected, most players just break there.

你知道,愚蠢的事情是,如果你尝试 MP4,但结果发现它是一个与你预期不同的编解码器,大多数播放器就在那里崩溃。

And so how do you not break?

那么如何才能不崩溃呢?

There’s just philosophically, I’m sure there’s a bunch of stumbling blocks along the way where it’s easy to just break and stop, freak out.

从哲学上讲,我确信一路上会有一堆绊脚石,很容易就停下来,吓坏了。

That’s it.

就是这样。

How does VLC not?

VLC 怎么不呢?

Jean-Baptiste Kempf23:34

This is why VLC is popular.

这就是 VLC 流行的原因。

23:34 But the reason is because actually VLC was, is just a client of a streaming solution called VideoLAN from a very long time ago, from the late ’90s.

但原因是因为实际上 VLC 只是很久以前(90 年代末)称为 VideoLAN 的流媒体解决方案的客户端。

And when you’re playing video which are on UDP, right, in network, they might be damaged, right?

当你在网络中播放 UDP 上的视频时,它们可能会损坏,对吗?

So you don’t trust your inputs, and this is very important into the security is that you don’t trust your inputs.

所以你不相信你的输入,这对安全性非常重要,因为你不相信你的输入。

So everything in VLC is prepared to work with broken files.

因此,VLC 中的所有内容都已准备好处理损坏的文件。

And it’s a philosophical idea from the beginning, and everything is engineered into that.

这从一开始就是一个哲学理念,一切都是为了这个理念而设计的。

And it’s a culture, right?

这是一种文化,对吗?

And so, for example…

所以,例如……

And VLC became very popular on that because a long time ago when people were pirating content which they do a lot less today, um-

VLC 在这方面变得非常流行,因为很久以前人们都在盗版内容,而现在他们做的事情少了很多,嗯-

Lex Fridman24:23

And none of us ever have-

我们谁也没有——

Jean-Baptiste Kempf24:24

No, of course not.

不,当然不是。

24:24 Um- …

嗯-…

the metadata to play some files like AVI is at, at the end of the file, right?

播放某些文件(如 AVI)的元数据位于文件末尾,对吗?

And when you’re downloading, you don’t have that, right?

当你下载时,你没有那个,对吧?

So VLC was just like, “Hey, this file is broken, but I’m still going to try to interpret it,” and this was very useful.

所以 VLC 就像,“嘿,这个文件坏了,但我仍然会尝试解释它,”这非常有用。

Lex Fridman24:41

We hinted at the awesomeness of the various different stages.

我们暗示了各个不同阶段的精彩之处。

24:41 We hinted at the awesomeness of codecs, the depth and the richness and the complexity of everything involved there.

我们暗示了编解码器的伟大之处、其中涉及的一切的深度、丰富性和复杂性。

What— Let’s try to define what is a video codec?

什么——让我们尝试定义什么是视频编解码器?

What’s involved there?

其中涉及什么?

What does it mean to compress something?

压缩某物是什么意思?

You already started to hint at it— …

你已经开始暗示了——……

but can, can we elaborate a little bit more?

但是我们可以详细说明一下吗?

Kieran Kunhya25:01

So there’s a huge amount of redundancy in any video, both spatial and temporal, and the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process.

因此,任何视频中都存在大量的空间和时间冗余,任何视频编解码器的重点都是删除这些冗余数据,使用数学属性作为此减少过程的一部分。

25:01 So more often than not, using several orders of magnitude more compute to compress because that’s more costly, both financially and in CPU resources— …

因此,通常会使用多几个数量级的计算量来进行压缩,因为这在经济上和 CPU 资源上都更加昂贵——……

versus the decompression.

与减压相对。

So it’s asymmetric in that respect.

所以在这方面它是不对称的。

Often the case because compression is done once, but there could be lots of viewers of another file.

通常情况下,因为压缩完成一次,但另一个文件可能有很多查看者。

25:33 So to take that information and compress it by 100x, 200x, removing redundant information and using mathematical properties to make that small, but also have properties such as error resilience.

因此,要获取该信息并将其压缩 100 倍、200 倍,删除冗余信息并使用数学属性使其变小,同时还具有错误恢复等属性。

So as JB suggested, VLC in the beginning was used to play UDP network feeds, and UDP network feeds lose packets.

因此,正如 JB 所建议的,VLC 一开始是用来播放 UDP 网络源的,而 UDP 网络源会丢包。

And so some of the design goals of a codec is also to be recoverable.

因此编解码器的一些设计目标也是可恢复的。

You need to actually be able to join a stream.

您需要真正能够加入流。

It’s not necessarily a file.

它不一定是一个文件。

You need to join, get on the decoding process, and start decoding.

您需要加入,开始解码过程,然后开始解码。

Jean-Baptiste Kempf26:06

And to give a more image to people who are not familiar, right?

并且给不熟悉的人一个更多的形象,对吧?

26:06 Like, when you’re going to see any type of movie, right?

就像,当你要看任何类型的电影时,对吧?

You’re going to see the camera is going to pan, right, and travel.

您将看到摄像机将向右平移并移动。

And you realize that, for example, all the background is the same for like a minute, right?

例如,您意识到,大约一分钟内所有背景都是相同的,对吧?

Or— …

或者——……

thirty seconds, right?

三十秒,对吗?

So you can reuse the cloud that you see on the background, you can reuse that from a frame to another, right?

因此,您可以重复使用在背景上看到的云,可以将其从一个框架重复使用到另一个框架,对吧?

And so it gets the more, the more memory you have, the more power, the more comparisons you can make, right?

所以它变得越多,你拥有的内存越多,能力就越强,你可以进行的比较就越多,对吗?

And so the more compressed you can be.

所以你可以压缩得越多。

And most of the modern codecs are basically doing that.

大多数现代编解码器基本上都是这样做的。

Lex Fridman26:44

So just to make it even more explicit.

所以只是为了让它更加明确。

26:44 So what is video?

那么什么是视频呢?

Video is a bunch of pixels off an RGB.

视频是一组 RGB 像素。

You have three values, and you have a grid of pixels, and you have, let’s say, twenty-four or thirty or sixty frames a second, and you just have all these pixels repeating and showing different stuff- …

你有三个值,你有一个像素网格,你有,比方说,每秒二十四或三十或六十帧,你只是让所有这些像素重复并显示不同的东西 - ...

thirty times a second.

每秒三十次。

And so the question, the philosophical, the technical question is, how can I compress all of that, store all of that at 100x?

所以问题、哲学、技术问题是,我怎样才能压缩所有这些,以 100 倍的速度存储所有这些?

Jean-Baptiste Kempf27:20

Yep.

是的。

27:20 Or 1,000x, right?

或者 1,000 倍,对吗?

Lex Fridman27:22

1,000x.

1,000 倍。

Jean-Baptiste Kempf27:22

The, the target is 1,000x, right?

目标是 1,000 倍,对吗?

Lex Fridman27:24

And the goal is when you say redundancy, what is redundant?

目标是当你说冗余时,什么是冗余?

27:24 Meaning stuff at best that humans wouldn’t notice if it was missing.

充其量意味着如果丢失的话人类不会注意到的东西。

Jean-Baptiste Kempf27:35

So for example, you have a picture of a cloud, right?

举例来说,你有一张云的图片,对吧?

27:35 And from the next frame, it’s still going to be the same cloud, so it’s redundant.

从下一帧开始,它仍然是同一个云,所以它是多余的。

You could just put it once and not do it, right?

你可以只放一次而不做,对吗?

Or you have a black background behind me, for example.

或者,例如,我身后有黑色背景。

The black is the same on the whole picture, right?

整个图片上的黑色都是一样的,对吧?

So you can say, “Well, you know, in this picture, take the pixels that you have on the top left and the one on the top right.

所以你可以说,“嗯,你知道,在这张照片中,取左上角和右上角的像素。

I’m not going to give the value.

我不会给出这个值。

I’m just going to tell you it’s the same at the top left.” And then you can say for frame one, reuse something from the previous frame or the previous, previous frame, and so on and so on, right?

我只是想告诉你,左上角也是一样的。”然后你可以说,对于第一帧,重复使用前一帧或前一帧中的某些内容,依此类推,对吧?

So you could…

所以你可以……

Basically, it’s unlimited, but then it’s limited in terms of memory or in terms of compute power.

基本上,它是无限的,但它在内存或计算能力方面受到限制。

28:15 Because, for example, if you need to compare pixels on two hundred frames in the past on 4K resolutions, it’s a huge amount of compute.

因为,例如,如果您需要在 4K 分辨率下比较过去 200 帧的像素,则计算量非常大。

Lex Fridman28:26

And then when you’re showing it, you have to do the decompress of all of that.

然后当你展示它时,你必须对所有这些进行解压缩。

28:26 So is it the codec, has the encoding and the decoding is a coupled process that you’re developing?

那么它是编解码器吗?编码和解码是您正在开发的耦合过程吗?

Jean-Baptiste Kempf28:37

Yes, exactly, right.

是的,完全正确。

28:37 And those are two different trade-offs, right?

这是两种不同的权衡,对吗?

Are you going to compress more?

你要压缩更多吗?

But then it might be more difficult to decode.

但这样解码可能会更困难。

Are you going to make it a codec that is more complex to encode and easier to decode?

你打算让它成为一个编码更复杂、解码更容易的编解码器吗?

Are you going to make a codec that is easier to encode because you need to be fast, but then the client side, the player is going to spend more time?

您是否打算制作一个更容易编码的编解码器,因为您需要速度快,但在客户端,玩家会花费更多时间?

That’s why you have so many different types of codecs, is that it’s not always easy.

这就是为什么你有这么多不同类型的编解码器,因为这并不总是那么容易。

And to make it even more complex, modern codecs like AV1, AV2, or VVC are actually not codecs.

更复杂的是,像 AV1、AV2 或 VVC 这样的现代编解码器实际上并不是编解码器。

They are a collection of tools, right?

它们是工具的集合,对吗?

There are multiple tools, multiple codecs in the same codec to, depending on the image, get the more compression.

有多种工具,同一编解码器中有多个编解码器,可以根据图像获得更多压缩。

Kieran Kunhya29:21

So just to elaborate, codecs like AV1, VVC have a much wide, have a wide audience.

详细说明一下,像 AV1、VVC 这样的编解码器有着广泛的受众。

29:21 It could be a screen share content, it could be video, it could be animation.

它可以是屏幕共享内容,可以是视频,也可以是动画。

All of these require different coding tools.

所有这些都需要不同的编码工具。

So what happens these days is a collection of tools are put in and called AV1 and called AV2, called VVC to allow for different use cases.

所以现在发生的事情是引入一系列工具,称为 AV1、AV2、VVC,以允许不同的用例。

So you may be on Zoom and sharing your PowerPoint, and then you need to show the audience a video.

因此,您可能在 Zoom 上分享您的 PowerPoint,然后您需要向观众展示一段视频。

That codec needs to start changing its tool set depending on the content to compress in a different way.

该编解码器需要根据要以不同方式压缩的内容开始更改其工具集。

Lex Fridman29:59

And like you said, there’s a, a bunch of incredible engineers behind each part of that, each part of the tools that make up AV1, for example.

就像你说的,每个部分背后都有一群令人难以置信的工程师,例如构成 AV1 的工具的每个部分。

Kieran Kunhya30:05

Sure.

当然。

Lex Fridman30:06

So we’ve kind of danced around it.

所以我们有点绕着它跳舞。

30:06 We talked about VLC, the logo, the hat.

我们讨论了 VLC、徽标、帽子。

Let’s talk about FFmpeg.

我们来谈谈 FFmpeg。

What, what is FFmpeg exactly?

什么,FFmpeg 到底是什么?

Jean-Baptiste Kempf30:17

FFmpeg is basically the low-level libraries for codec, so compressions and decompression, muxers and demuxers, and filters.

FFmpeg 基本上是编解码器的低级库,包括压缩和解压缩、复用器和解复用器以及过滤器。

30:17 It’s– The core is this, and then you have a several tool which allow you to create a type of pipeline to process any type of video files.

它是——核心是这个,然后你有几个工具可以让你创建一种管道来处理任何类型的视频文件。

And it’s used as a library absolutely inside everything from VLC to Chrome to your smart TVs, to basically any video that you see online you usually use FFmpeg.

它绝对可以用作从 VLC 到 Chrome 到智能电视的所有内容的库,基本上到您通常使用 FFmpeg 在线观看的任何视频。

And FFmpeg in it has all those type of tools, and sometimes depend on other libraries like x264, libvpx, and others, right?

其中 FFmpeg 拥有所有这些类型的工具,有时还依赖于其他库,如 x264、libvpx 等,对吗?

So, so it’s really now the de facto tool to process images.

所以,它现在确实是处理图像的事实上的工具。

Kieran Kunhya31:06

From a philosophical level, I think it’s incredible that your home videos, your, your grandmother’s home videos and trillion-dollar corporations effectively are on a level playing field using the same technology stack.

从哲学层面来看,我认为你的家庭视频、你、你祖母的家庭视频和价值数万亿美元的公司实际上处于使用相同技术堆栈的公平竞争环境中,这是令人难以置信的。

31:06 It’s– it wouldn’t be a surprise, you know, these big companies just have three thousand-line FFmpeg commands.

这并不奇怪,你知道,这些大公司只有三千行 FFmpeg 命令。

There are some that use the API, but there are some that just have long command lines.

有一些使用 API,但也有一些只有很长的命令行。

Lex Fridman31:31

So yeah, there’s a bunch of tools, like literally command line tool, FFmpeg, of course, FFprobe.

是的,有很多工具,例如命令行工具、FFmpeg,当然还有 FFprobe。

31:31 There’s libraries, libavcodec, libavformat, libavfilter.

有库、libavcodec、libavformat、libavfilter。

But the FFmpeg on the command line- …

但是命令行上的 FFmpeg-...

is, like, legendary because you can cut– Like, there’s so many parameters.

就像传奇一样,因为你可以剪切——比如,有很多参数。

You can customize everything to hell.

你可以定制一切。

Jean-Baptiste Kempf31:53

It’s a language.

这是一种语言。

31:53 It’s an actual language.

这是一种真实的语言。

Lex Fridman31:55

It’s an actual– yeah, you could think of it as a programming language.

这是一种实际的——是的,你可以将它视为一种编程语言。

Jean-Baptiste Kempf31:57

Yeah, of course, I’m sure.

是的,当然,我确定。

31:57 Because– So most of the people, they’re going to take FFmpeg, file in, file out, and specify the format, right?

因为——所以大多数人都会采用 FFmpeg、文件输入、文件输出并指定格式,对吧?

But you can– We’ve seen thousands of characters, and we’ve seen also, like, people doing programming generation of command lines to make FFmpeg.

但你可以——我们已经看到了数千个字符,我们也看到了人们通过编程生成命令行来制作 FFmpeg。

There is a ton of people who are using AI to generate command lines for FFmpeg because you have no idea what it is.

有很多人使用 AI 为 FFmpeg 生成命令行,因为你不知道它是什么。

But you can specify so many filters, right on command line, right?

但是您可以在命令行上指定这么多过滤器,对吗?

So FFmpeg is this collection of toolbox for multimedia processing that everyone, everyone uses.

所以FFmpeg是每个人都使用的多媒体处理工具箱的集合。

And everyone that is watching your videos are also using, right?

每个观看您视频的人也在使用,对吗?

You’re on YouTube.

您在 YouTube 上。

Well, it’s FFmpeg on the client side.

嗯,客户端是 FFmpeg。

Well, the server side, on the server side.

嗯,服务器端,在服务器端。

The client side is probably Chrome.

客户端可能是Chrome。

Well, you’re using FFmpeg also.

嗯,您也在使用 FFmpeg。

32:46 And you’re using OBS to record.

并且您正在使用 OBS 进行录制。

Well, it’s FFmpeg, right?

嗯,这是 FFmpeg,对吧?

You’re using a ton of important, like, big box, professional boxes.

您正在使用大量重要的、大盒子、专业盒子。

Well, it’s very possible that inside some part of FFmpeg is running.

嗯,很有可能 FFmpeg 的某些部分正在运行。

Lex Fridman32:57

I mean, there’s like so many, just to give people an idea, like I use FFmpeg a lot on everything.

我的意思是,有很多这样的东西,只是为了给人们一个想法,就像我在所有事情上都大量使用 FFmpeg 一样。

32:57 Just trivial stuff like take a video, add an intro video and an outro video, and fade one into the other like what is it called?

只是一些琐碎的事情,比如拍摄一个视频,添加一个介绍视频和一个结尾视频,然后将一个视频淡入另一个视频,就像它叫什么一样?

Dip to black, like where it dips and then shows the next video and does the same thing with audio.

浸入黑色,就像它浸入的地方一样,然后显示下一个视频并对音频执行相同的操作。

There’s like a cross dissolve of the audio.

就像音频的交叉溶解一样。

It quiets the audio and makes it loud again.

它使音频安静并再次变得大声。

And then there’s a bunch of stuff like showing the captions on screen card, like baking the captions in.

然后还有很多东西,比如在屏幕卡上显示字幕,比如烘焙字幕。

You can customize the font.

您可以自定义字体。

You can do all kinds of layering of audio and video.

您可以对音频和视频进行各种分层。

There’s a million things and of course, all of that works like magically with basically any codec.

有一百万种东西,当然,所有这些都可以神奇地与基本上任何编解码器一起使用。

Like anything you can shove in on the audio and the video side, it works.

就像您可以在音频和视频方面添加的任何内容一样,它确实有效。

Jean-Baptiste Kempf33:53

But it’s like if you look at, for example, you can do things that you would do with Adobe After Effects- …

但如果你看一下,你可以做一些你可以用 Adobe After Effects 做的事情——……

33:53 in command line on FFmpeg, right?

在 FFmpeg 的命令行中,对吧?

It’s, and it’s very interesting because, for example, for images, there is not such tool.

是的,而且非常有趣,因为例如对于图像来说,就没有这样的工具。

There is a few tools, but not with the breadth of FFmpeg.

有一些工具,但不及 FFmpeg 的广度。

Lex Fridman34:10

So ImageMagick has a similar kind of-

所以 ImageMagick 有类似的 -

Jean-Baptiste Kempf34:13

Yes, but you will not-

是的,但你不会-

Lex Fridman34:14

…

……

34:14 spirit, but it-

精神,但它——

Jean-Baptiste Kempf34:14

…

……

do some filters, complex filters.

做一些过滤器,复杂的过滤器。

You don’t have the equivalent of Photoshop- …

你没有相当于 Photoshop 的工具——……

in command line, right?

在命令行中,对吗?

But for video, you have FFmpeg in command line.

但对于视频,您可以在命令行中使用 FFmpeg。

Lex Fridman34:22

Yeah.

是的。

34:22 It’s incredible.

太不可思议了。

I mean, it’s like an, it’s an example of a thing when a bunch of great people get together and they get a vision, and they stick by that vision for many years, which is incredible.

我的意思是,这就像,这是一个例子,一群伟大的人聚集在一起,他们有了一个愿景,并且他们多年来坚持这个愿景,这是令人难以置信的。

Jean-Baptiste Kempf34:33

And the vision behind, and the same for VLC and FFmpeg, is that we make everything that is very complex easy to use for the normal people, for everyone.

其背后的愿景与 VLC 和 FFmpeg 相同,是我们让一切非常复杂的东西都易于普通人、每个人使用。

34:33 Right?

正确的?

Our goal is to make something that is insanely complex technically and make it easy to use, right?

我们的目标是制造技术上极其复杂的东西并使其易于使用,对吗?

And people, they use VLC, they drop a file.

而人们,他们使用 VLC,他们放置一个文件。

They don’t realize how complex the file is, but they play it.

他们没有意识到文件有多复杂,但他们会播放它。

Or people put any type of thing inside FFmpeg with complex filters, and it just works like magically, right?

或者人们将任何类型的东西放入带有复杂过滤器的 FFmpeg 中,它就会像魔法一样工作,对吧?

And people- And this is our mission, right?

还有人们——这就是我们的使命,对吧?

Make very complex things.

制作非常复杂的东西。

Kieran Kunhya35:08

We wouldn’t be here and you wouldn’t be here if this required, you know, a traditional television studio setup.

如果这需要传统的电视演播室设置,我们就不会在这里,你也不会在这里。

35:08 It’s tools like FFmpeg that democratize this.

FFmpeg 这样的工具使这一过程变得民主化。

The podcast and streaming revolution, the YouTube revolution- …

播客和流媒体革命、YouTube 革命——……

was caused.

是造成的。

You know, FFmpeg was a big player in that because it democratized this technology that was once in the nineties, for example, you needed equipment that cost hundreds of thousands of dollars to do compression.

您知道,FFmpeg 在这方面发挥了重要作用,因为它使这项九十年代曾经出现的技术变得大众化,例如,您需要花费数十万美元的设备来进行压缩。

It was the size of a car, and now everybody has that at almost an exact level playing field, and that’s something that’s so remarkable.

它只有一辆汽车那么大,现在每个人都在几乎完全公平的竞争环境中拥有它,这是非常了不起的。

Lex Fridman35:42

It gave voice to a lot of people.

它为很多人发出了声音。

35:42 And just to clarify, we say you wouldn’t be here—not the human, but the podcast.

澄清一下,我们说你不会在这里——不是人类,而是播客。

Jean-Baptiste Kempf35:49

The podcast.

播客。

35:49 Oh, sorry.

哦,抱歉。

You as a…

你作为一个…

Sorry.

抱歉。

Lex Fridman35:50

I would still…

我还是会……

35:50 VLC did not have anything to do on a biological level- …

VLC 在生物学层面上没有任何关系——……

at creating me as a human.

将我创造为人类。

Jean-Baptiste Kempf35:56

But it’s like you realize also everything moved from text to images and images to video, right?

但就像你意识到一切都从文本转移到图像,从图像转移到视频,对吗?

35:56 Look at social networks.

看看社交网络。

Video is everywhere.

视频无处不在。

It’s the most powerful medium there is, right?

它是最强大的媒介,对吗?

And when you see shorts and Reels and TikTok, right?

当你看到 Shorts、Reels 和 TikTok 时,对吗?

It’s amazingly powerful to give…

它的力量是惊人的……

Video is amazing for that, right?

视频太棒了,对吧?

But the complexity is important.

但复杂性很重要。

Lex Fridman36:20

This is what people don’t realize.

这是人们没有意识到的。

36:20 I mean, this is really—it gave power to the individual all across the world.

我的意思是,这确实给了世界各地的个人力量。

That’s real freedom.

这才是真正的自由。

And I think, I can’t believe it, but we still haven’t mentioned the actual obvious thing for people who are not familiar, which it’s open source, and there’s an open source community of users and developers behind it.

我想,我简直不敢相信,但对于不熟悉的人来说,我们仍然没有提及实际显而易见的事情,即它是开源的,并且背后有一个由用户和开发人员组成的开源社区。

So it’s really a movement.

所以这确实是一场运动。

So, like, we’ll talk in a bunch of different ways about the community behind it.

因此,我们将以多种不同的方式谈论其背后的社区。

But can you speak to the open source element?

但你能谈谈开源元素吗?

So when we say what is FFmpeg, it’s an open source project.

所以当我们说什么是 FFmpeg 时,它是一个开源项目。

Jean-Baptiste Kempf36:57

Yeah.

是的。

36:57 So FFmpeg, VLC, x264, VideoLAN, everything we do is fully open source.

所以FFmpeg、VLC、x264、VideoLAN,我们所做的一切都是完全开源的。

And for the people who don’t understand how open source is, my usual analogy is about a chocolate cheesecake, right?

对于那些不了解开源的人来说,我通常的比喻是巧克力芝士蛋糕,对吗?

Usually for you, when you want to buy your cheesecake, you go to a bakery, they give you the cheesecake.

通常对你来说,当你想买芝士蛋糕时,你会去面包店,他们会给你芝士蛋糕。

The other one way of having a cheesecake is have your grandma give you a recipe of how to make that.

另一种制作芝士蛋糕的方法是让你的奶奶给你一份制作芝士蛋糕的食谱。

When we do open source, we give you the chocolate cake, and we give you the recipe to actually remake the same cake, but at the same time tell you how to build the oven and also how you’re allowed to modify the recipe and resell it to someone else.

当我们开源时,我们给你巧克力蛋糕,给你实际重新制作同一个蛋糕的食谱,但同时告诉你如何建造烤箱,以及如何允许你修改食谱并将其转售给其他人。

37:35 And this is because software is just a very long recipe of small instructions.

这是因为软件只是一个很长的小指令配方。

Computers are not very clever.

计算机并不是很聪明。

They go very, very fast.

他们走得非常非常快。

So a normal program has tens of billions of instructions instead of the tens when you have your chocolate recipe.

因此,一个正常的程序有数百亿条指令,而不是当你有巧克力配方时有数十条指令。

So a lot of the software industry was about selling software where you just have the final cheesecake.

因此,软件行业的很多领域都是在销售软件,而你只得到了最终的芝士蛋糕。

In open source, we give you everything, and that managed to get a lot of people to work together, right?

在开源中,我们为您提供一切,这成功地让很多人一起工作,对吧?

Because then you decide that you’re going to make the best program, the best recipe for video, and you create communities.

因为那时你决定要制作最好的节目、最好的视频配方,然后创建社区。

In FFmpeg, since the beginning of FFmpeg, probably two thousand to three thousand-

在FFmpeg中,自从FFmpeg诞生以来,大概两千到三千——

Lex Fridman38:17

In the thousands, yeah

数以千计,是的

Jean-Baptiste Kempf38:17

…

……

38:17 people have contributed from the beginning, right?

人们从一开始就做出了贡献,对吗?

And then it’s exactly like the Linux kernel, right?

然后它就和 Linux 内核一模一样了,对吧?

The Linux kernel has probably ten thousand people contributing everywhere, and they get together, well, mostly online, right?

Linux 内核可能有一万人在各地做出贡献,他们聚集在一起,嗯,大部分是在线的,对吗?

So they virtually get together to create the best tool for something.

因此,他们实际上聚集在一起,为某些事情创造最好的工具。

And on FFmpeg and VLC, it’s just like, well, this codec doesn’t work, so I’m going to work on the codec, and I’m going to add the support for this file inside FFmpeg, so it will be beneficial to everyone.

在 FFmpeg 和 VLC 上,就像,嗯,这个编解码器不起作用,所以我将研究编解码器,并且我将在 FFmpeg 中添加对此文件的支持,因此这对每个人都有好处。

Because again, we work for the greater good.

因为我们再次为更大的利益而努力。

We work for everyone, and that is what open source is.

我们为每个人工作,这就是开源。

Lex Fridman38:54

And we should mention, depending on the licensing you could probably build a billion-dollar, maybe even a trillion-dollar company as a wrapper to—

我们应该提到,根据许可,你可能会建立一家价值十亿美元、甚至万亿美元的公司作为包装——

Jean-Baptiste Kempf39:07

Well, yes— …

嗯,是的——……

39:07 people do.

人们确实如此。

People do, right?

人们确实如此,对吗?

There were a lot of problems with mostly cloud providers who are basically running some open source tools in the cloud and just give you the API to access to that.

大多数云提供商都存在很多问题,他们基本上在云中运行一些开源工具,只为您提供访问该工具的 API。

And there was a lot of databases like Mongo or Elastic who changed their license in order to avoid those types of scenarios.

许多数据库(例如 Mongo 或 Elastic)更改了许可证以避免此类情况的发生。

Kieran Kunhya39:31

This is a question we get a lot in FFmpeg is, “Why don’t you do that?” And you can’t.

我们在 FFmpeg 中经常遇到的一个问题是,“你为什么不这样做?”但你不能。

39:31 We have, we have thousands of contributors, some of whom aren’t even alive anymore.

我们有成千上万的贡献者,其中一些人甚至已经不在人世了。

It would need all of their agreement to do that, and JB will go maybe a bit later and talk about how challenging that process was in VLC to do the re-licensing.

这样做需要他们所有人的同意,JB 可能稍后会讨论 VLC 中进行重新许可的过程是多么具有挑战性。

Jean-Baptiste Kempf39:49

The license is a social contract in terms of Rousseau de facto of the community.

许可证是卢梭事实上的社会的社会契约。

39:49 The community does not agree on much besides the license.

除了许可证之外,社区并没有达成太多共识。

People go around, discuss around because of the license, and that also allow those license forks, right?

人们因为许可证而到处讨论,这也允许那些许可证分叉,对吧?

Sometimes the community splits, but it’s possible because of the license and to merge back.

有时社区会分裂,但由于许可证的原因,有可能重新合并。

40:12 And we’ve seen that so many times, right?

我们已经见过很多次了,对吧?

GCC and EGCS in the past.

过去的 GCC 和 EGCS。

We have seen, for example, all the web browsers, right?

例如,我们已经见过所有的网络浏览器,对吧?

They started as web, like KHTML, which becomes WebKit and then which becomes Blink, right?

它们一开始是 Web,比如 KHTML,后来变成了 WebKit,然后又变成了 Blink,对吗?

So open source license is like the core of the community and people are coming from all around the world, very different types of religion, political borders.

因此,开源许可证就像社区的核心,人们来自世界各地,宗教类型和政治边界截然不同。

They work in the same way on a project to solve a specific problem, and the specific problem we’re working on is to make multimedia easy for everyone.

他们在项目中以相同的方式工作来解决特定问题,而我们正在解决的特定问题是让每个人都能轻松使用多媒体。

Lex Fridman40:50

Looking it up on Perplexity here, looking at the different open source licenses.

在这里查看 Perplexity,查看不同的开源许可证。

40:50 Most major open source licenses fall into two buckets: permissive, very few conditions, and copyleft, share-alike requirements for derivatives.

大多数主要的开源许可证分为两类:宽松的、很少的条件,以及 Copyleft、衍生品的类似共享要求。

Below is a brief practical summary of the main ones you’ll see in the wild.

以下是您在野外会看到的主要物种的简要实用摘要。

MIT license, BSD, ISC Apache GNU GPL, GNU AGPL.

MIT 许可证、BSD、ISC Apache GNU GPL、GNU AGPL。

Where’s LGPL?

LGPL 在哪里?

Yeah, LGPL.

是的,LGPL。

Let’s see.

让我们来看看。

There’s the Mo- the Mozilla Public License.

Mo——Mozilla 公共许可证。

There’s Eclipse Public License.

有 Eclipse 公共许可证。

It goes on.

它继续下去。

There’s a lot of variety.

有很多种。

I mean, I think the really popular ones are MIT, GPL, LGPL-

我的意思是,我认为真正流行的是 MIT、GPL、LGPL-

Jean-Baptiste Kempf41:34

Yeah.

是的。

41:34 And BSD.

还有BSD。

Lex Fridman41:36

…

……

41:36 and BSD, Apache.

以及 BSD、阿帕奇。

Sometimes you’ll see-

有时你会看到——

Jean-Baptiste Kempf41:38

Apache as well

阿帕奇也一样

Lex Fridman41:38

…

……

41:38 Apache.

阿帕奇。

Unlicense, that’s an option.

取消许可,这是一个选择。

Attempts to dedicate code to the public domain with a fallback permissive license.

尝试使用备用许可将代码奉献给公共领域。

Jean-Baptiste Kempf41:43

There are many licenses for many different things.

许多不同的事物有许多许可证。

41:43 What people don’t understand is that public domain is something that doesn’t exist worldwide, right?

人们不明白的是,公有领域在全世界范围内并不存在,对吧?

So all the open source licensing use the copyright law, right, the international copyright law, in order to give rights on how you use the software or how you modify.

因此,所有开源许可都使用版权法,对吧,国际版权法,以便授予有关您如何使用软件或如何修改软件的权利。

It’s de facto a copyright license contract that you give to the end user or to the developer.

它实际上是您提供给最终用户或开发人员的版权许可合同。

And so you have like the first ones, which are basically very permissive, MIT, BSD.

所以你有像第一个,基本上是非常宽松的,麻省理工学院,BSD。

You give the code and basically you do whatever you want, right?

你给出代码,基本上你就可以做任何你想做的事,对吧?

You take it, you want, you modify, you do what you want.

你拿走它,你想要,你修改,你做你想做的。

And this is popular for JavaScript and the type of BSD operating system.

这对于 JavaScript 和 BSD 操作系统类型来说很流行。

Lex Fridman42:29

So some of them, one of the parameters is whether they require attribution, meaning if you use the code, you have to say-

所以其中一些,参数之一是它们是否需要归属,这意味着如果你使用这些代码,你必须说-

Jean-Baptiste Kempf42:35

Yes.

是的。

42:35 So in those types of permissive licenses, some you need to say if you use it, which is called attribution, and some you don’t.

因此,在这些类型的宽松许可中,有些您需要说明是否使用它,这称为归属,有些则不需要。

And then there is the other part of license which are copyleft, where you need to give back to the community your modifications and with different strings attached.

然后许可证的另一部分是 Copyleft,您需要将您的修改并附加不同的字符串回馈给社区。

Some weak copyleft licenses, like the Mozilla Public License, to some which are a bit stronger like a GPL, or even very strong like AGPL.

一些较弱的 Copyleft 许可证(例如 Mozilla 公共许可证),以及一些较弱的 Copyleft 许可证(例如 GPL),甚至非常强大(例如 AGPL)。

So all of those are different types of licensing that depends on what your goals are and how you want to structure your community, which is why I spoke about social contract, because this is very important to understand.

因此,所有这些都是不同类型的许可,取决于您的目标是什么以及您想要如何构建社区,这就是我谈到社会契约的原因,因为理解这一点非常重要。

FFmpeg and VLC are mostly GPL or LGPL.

FFmpeg和VLC大多是GPL或LGPL。

The Linux kernel is GPL but Android is Apache.

Linux 内核是 GPL,而 Android 是 Apache。

43:27 A ton of JavaScript frameworks that are used are mostly MIT.

使用的大量 JavaScript 框架大部分都是 MIT 的。

All the BSD kernels, OpenBSD, NetBSD are of course BSD.

所有的 BSD 内核、OpenBSD、NetBSD 当然都是 BSD。

And so the…

所以……

it’s a philosophical change on how you want people to contribute back- …

这是关于你希望人们如何回馈的哲学上的改变——……

basically.

基本上。

Lex Fridman43:44

So there’s I think you talked about that you’ve moved at one point from GPL to LGPL on certain parts of the project.

我认为您谈到您曾在项目的某些部分从 GPL 转向 LGPL。

43:44 What…

什么……

Can you describe the difference between the two, and what does it take to move to, I guess, a more permissive…

你能描述一下两者之间的区别吗?我想,如何才能转向更宽松的……

So that direction is more permissive.

所以这个方向更加宽松。

LGPL is more permissive than GPL.

LGPL 比 GPL 更宽松。

Jean-Baptiste Kempf44:04

Yeah.

是的。

44:04 So you have to realize that you can always go from more permissive to less permissive, right?

所以你必须意识到你总是可以从更宽松变成更宽松,对吗?

Because of course, those licenses are basically statements, and so if you restrict, you can always restrict more, right?

当然,因为这些许可证基本上都是声明,所以如果你限制,你总是可以限制更多,对吗?

So in a GPL project, you can take MIT code, but you cannot do the opposite, right?

所以在 GPL 项目中,你可以采用 MIT 代码,但你不能做相反的事情,对吗?

Because they are more constrained to match.

因为他们更难匹配。

Indeed, in fact, I changed the core of LibVLC, which is the engine of VLC- …

确实,其实我改变了LibVLC的核心,也就是VLC的引擎-...

from GPL to LGPL.

从 GPL 到 LGPL。

And there were two reasons to do that.

这样做有两个原因。

The first one is that so people can use the VLC engine, LibVLC, into third-party applications.

第一个是人们可以将 VLC 引擎 LibVLC 用于第三方应用程序。

So a lot of applications which are playing video on your phone or on your tablet are actually VLC engine in it- …

因此,许多在手机或平板电脑上播放视频的应用程序实际上都是 VLC 引擎 - ...

which is calling FFmpeg in it.

它在其中调用 FFmpeg。

44:55 So that was one of the ways to create one of the companies I created, which is doing consulting and integration of those types of applications where you integrate VLC into third-party solutions like inside game engines or stuff like that.

因此,这是创建我创建的公司之一的方法之一,该公司正在对这些类型的应用程序进行咨询和集成,在这些应用程序中,您可以将 VLC 集成到第三方解决方案中,例如内部游戏引擎或类似的东西。

With GPL, you couldn’t do that because that means you needed to open source everything, and those are for a lot of, like, commercial companies who don’t want that.

使用 GPL,你无法做到这一点,因为这意味着你需要开源所有内容,而这些都是针对许多不希望这样做的商业公司的。

Lex Fridman45:18

So you can create a company with LGPL, you can create a company around it.

所以你可以用 LGPL 创建一家公司,你可以围绕它创建一家公司。

45:18 You can do a commercial thing.

你可以做商业性的事情。

You don’t have to open source it.

您不必将其开源。

So that’s a big, big leap.

所以这是一个很大很大的飞跃。

Kieran Kunhya45:25

So you could play video in your game.

这样您就可以在游戏中播放视频了。

45:25 The problem is I’m a game developer, and I want to play some videos- …

问题是我是一名游戏开发者,我想玩一些视频 - ...

and I don’t want to be forced to open source the entire game just to play those videos.

我不想仅仅为了玩这些视频而被迫开源整个游戏。

So that’s where the, the consulting business, the libVLC LGPL- …

这就是咨询业务 libVLC LGPL-...

allows you to do that.

允许你这样做。

The LGPL, the library GPL as it used to be known, allows you to do that.

LGPL(以前称为库 GPL)允许您做到这一点。

Jean-Baptiste Kempf45:44

And FFmpeg is exactly the same.

和FFmpeg完全一样。

45:44 It force…

它迫使...

LGPL forces you to give back what you change on this component, this- …

LGPL 强迫你回馈你在这个组件上所做的改变,这-......

library, which is why it’s library GPL.

库,这就是为什么它是 GPL 库。

And so you can use FFmpeg as LGPL into, like, any type of application, even non-open source, but you need to give back the modification you did on FFmpeg.

因此,您可以将 FFmpeg 作为 LGPL 使用到任何类型的应用程序中,甚至是非开源应用程序,但您需要返回对 FFmpeg 所做的修改。

Same on libVLC.

libVLC 上也是如此。

Lex Fridman46:06

Is it limiting from an open source perspective to go GPL?

从开源的角度来看,采用 GPL 是否会受到限制?

46:06 Because if your library, if your code is GPL, it means you’re basically discouraging companies from building a business- …

因为如果你的图书馆、如果你的代码是 GPL,这意味着你基本上是在阻止公司建立业务——……

around it, right?

围绕着它,对吗?

Is that, is that fair to say?

这样说公平吗?

Jean-Baptiste Kempf46:23

It depends on the company, but for the company whose business model requires the application to be closed source, yes, it’s limited.

这取决于公司,但对于业务模式要求应用程序闭源的公司来说,是的,它是有限的。

46:23 So that’s why, for example, I moved to LGPL.

这就是为什么我转向 LGPL。

The second reason is a bit more obscure: it’s that the terms of condition of the App Store, the Apple App Store for iOS, makes it very complex to have GPL applications on it, while it’s easier to have LGPL applications on it.

第二个原因有点模糊:App Store(iOS 版 Apple App Store)的条件条款使得在其上放置 GPL 应用程序变得非常复杂,而在其上放置 LGPL 应用程序则更容易。

So VLC on Windows and on Mac and on Linux is GPL.

因此,Windows、Mac 和 Linux 上的 VLC 都是 GPL。

The core is LGPL.

核心是LGPL。

But on iOS, the iPhone version and the Apple TV version is a type of different license called the MPL.

但在 iOS 上,iPhone 版本和 Apple TV 版本是一种不同的许可证,称为 MPL。

And yes, I went and changed the license and it was a long story.

是的,我去更改了许可证,这是一个很长的故事。

Lex Fridman47:08

Yeah.

是的。

47:08 So I think basically to change the license you have to contact all the contributors.

所以我认为基本上要更改许可证,您必须联系所有贡献者。

Jean-Baptiste Kempf47:12

Yes.

是的。

47:12 It’s very important to understand that open source projects are what we call in the US copyright law joint work, or in civil law collective works or collaborative works.

理解开源项目是我们在美国版权法中所说的联合作品,或者民法集体作品或协作作品,这一点非常重要。

It’s that you work all together in terms of the same goal, and then you create one software, which is one release.

就是大家为了同一个目标一起工作,然后创建一个软件,即一个版本。

But the copyright is kept by all the individuals.

但版权归所有人所有。

Some open source projects don’t do that.

一些开源项目不这样做。

They force copyright assignment, but this is not what we do.

他们强制版权转让,但这不是我们所做的。

We’re communities.

我们是社区。

So everyone has basically copyright on what they changed.

所以每个人基本上都对他们所改变的内容拥有版权。

And this copyright stays even if at the end your contribution was deleted because the new contribution was based on your previous one, right?

即使您的贡献最终被删除,该版权仍然保留,因为新的贡献是基于您之前的贡献,对吧?

So if you want to properly re-license, you need to find all the contributors.

因此,如果您想正确地重新许可,您需要找到所有贡献者。

47:58 And at that time, I had to contact more than three hundred and fifty people.

而当时我要接触的人有三百五十多人。

And sometimes, well, they’re just an email, right?

有时,它们只是一封电子邮件,对吗?

So it’s…

所以这是……

you need to actually track down.

你需要实际追踪。

I actually, like, travel to some place to go someone that I was like, sorry, that I’d found online to see a– to go to their job and say, “Well, you licensed that.

事实上,我喜欢去某个地方,去找一个我喜欢的人,抱歉,我在网上找到了一个人,去他们的工作地点,然后说:“好吧,你已经获得了许可。

Can you– do you want to change from GPL to LGPL?” Most of the times they don’t even care.

你能——你想从 GPL 改为 LGPL 吗?”大多数时候他们甚至不在乎。

They wanted to help VLC.

他们想帮助 VLC。

But also it brought me to very complex situation.

但它也给我带来了非常复杂的情况。

I arrived to the work of a person who was a factory worker.

我到了一个工厂工人的工作地点。

And I said, “Well, I need you to sign that,” because it was his son who died who actually wrote the code, right?

我说,“好吧,我需要你签名,”因为实际上是他去世的儿子写了代码,对吗?

48:40 So I had to explain all those types of open source meanings, and no, I was not a company trying to rip out the two lines or five lines that that guy did- …

所以我必须解释所有这些类型的开源含义,不,我不是一家试图撕掉那家伙所做的两行或五行的公司——……

but was useful, and the whole community agreed on that, and he had no idea I was a factory worker.

但很有用,整个社区都同意这一点,而且他不知道我是工厂工人。

And I was a lot younger, right?

而且我还年轻很多,对吧?

Like it was fourteen years ago, and like I was almost in tears, right?

就像十四年前一样,我几乎哭了,对吧?

It’s very difficult, right?

这很困难,对吧?

We are talking about lives of people and he explaining, and we talked about the photo of this guy, right?

我们正在谈论人们的生活,他解释了,我们谈论了这个人的照片,对吧?

So it’s important to do it right and to do it correctly.

因此,正确地做事并正确地做事很重要。

But yes, that means tracking down everything because every contribution works.

但是,是的,这意味着追踪一切,因为每项贡献都有效。

There are some projects who don’t respect that, and we do re-licensing a bit, like, aggressively.

有些项目不尊重这一点,我们会积极地重新授权。

49:23 But as I said, it destroyed the whole heart of the community because it’s– we only agree on the license, so that’s important.

但正如我所说,它摧毁了整个社区的核心,因为我们只就许可证达成一致,所以这很重要。

Kieran Kunhya49:31

I would emphasize the community is such a wide-ranging group of people.

我想强调的是,社区是一个范围广泛的人群。

49:31 There’s people in the Syrian war zone with electricity part-time.

叙利亚战区有人兼职用电。

There’s…

有……

there’s people from all walks of life- …

这里有来自各行各业的人们——……

rich, poor, young, old.

富有,贫穷,年轻,年老。

So it’s quite remarkable to get, you know, a group of people aligned on something.

因此,让一群人在某件事上达成一致是非常了不起的。

And that’s an achievement in itself.

这本身就是一项成就。

Lex Fridman49:55

Yeah.

是的。

49:55 It’s incredible.

太不可思议了。

And a lot of them are introverts, so you coming to find them and getting them and getting them to answer an email might be quite, quite difficult.

他们中的很多人都是内向的人,所以你来找到他们并让他们回复电子邮件可能非常非常困难。

Jean-Baptiste Kempf50:05

Most of us are introverts, right?

我们大多数人都是内向的人,对吧?

50:05 You need to be more precise.

你需要更精确。

You have extremely introverts, extremely, extremely introverts and introverts, right?

你有非常内向的人,非常非常内向的人,非常内向的人,对吗?

Lex Fridman50:12

Yeah.

是的。

Jean-Baptiste Kempf50:12

It’s just like a whole spectrum of different people.

这就像一系列不同的人一样。

50:12 It doesn’t matter.

没关系。

The important thing is, is your code good?

重要的是,你的代码好不好?

Is your code great?

你的代码很棒吗?

Is your technology great?

你的技术很棒吗?

We care about excellent code.

我们关心优秀的代码。

We don’t care who you are.

我们不在乎你是谁。

Sorry, it’s just like we have no idea to check.

抱歉,我们好像不知道要检查什么。

We cannot check, right?

我们无法检查,对吗?

Like, maybe you’re a dog.

就像,也许你是一只狗。

I don’t care, right?

我不在乎,对吧?

I don’t care where you come from.

我不在乎你来自哪里。

I need to look at your code.

我需要看看你的代码。

And this is important because people don’t understand that, and they come to the community and send them some patches, and they get rejected, and they don’t like that because, I mean, you’re just like, “Sorry, it’s not up to our standards.” “Oh, yeah, but I’m an engineer at this very large company in Italy, in Germany, in the US.” We don’t care.

这很重要,因为人们不理解这一点,他们来到社区并向他们发送一些补丁,但他们被拒绝了,他们不喜欢这样,因为,我的意思是,你会说,“抱歉,这不符合我们的标准。” “哦,是的,但我是意大利、德国和美国这家非常大公司的工程师。”我们不在乎。

50:54 We care about the quality of your code because this is what defines our community, which means that we have a lot of people who contribute who are from some very different backgrounds and, and, and very introverts, sure.

我们关心您的代码质量,因为这是我们社区的定义,这意味着我们有很多贡献者,他们来自不同的背景,而且,当然,非常内向。

But that’s okay, right?

但没关系,对吧?

Lex Fridman51:07

So one of the legends of the community is of course, Linus Torvalds, who created Linux and is a longtime maintainer of the Linux kernel.

所以社区的传奇人物之一当然是 Linus Torvalds,他创建了 Linux,并且是 Linux 内核的长期维护者。

51:07 As the legend goes, he can be pretty harsh on this meritocratic process of reviewing the code and saying it’s not good enough.

正如传说所言,他对这种审查代码并说它不够好的精英管理流程非常严厉。

Can you just speak to the legend of Linus Torvalds?

您能谈谈 Linus Torvalds 的传奇吗?

Jean-Baptiste Kempf51:29

Linus is one of a kind, right?

Linus 是独一无二的,对吧?

51:29 And, and I would even go and say that what he did on Git is more interesting than what he did on the Linux kernel.

而且,我什至会说他在 Git 上所做的事情比他在 Linux 内核上所做的事情更有趣。

51:39 He’s very harsh, but what people don’t see is usually when he’s harsh, it’s to people who are maintainers of part of the kernel, right?

他很严厉,但人们看不到的是,通常他的严厉是针对部分内核的维护者,对吗?

So they know him, right?

所以他们认识他,对吗?

So he’s not very harsh like that to everyone.

所以他对每个人都不是那么严厉。

The thing is, what he created in his room is basically powering every server online, right?

问题是,他在房间里创造的东西基本上为每台在线服务器提供动力,对吗?

Even at Microsoft’s cloud called Azure, I’m quite sure seventy, eighty percent of the servers are running Linux.

即使在微软的 Azure 云中,我也确信 70%、80% 的服务器都在运行 Linux。

All your Android phones are running Linux.

您所有的 Android 手机都运行 Linux。

What he did with the power of open source, sure, is amazing.

当然,他利用开源的力量所做的事情是令人惊叹的。

And yes, the quality of the Linux kernel is very high, and yes, it’s difficult, but we cannot compromise on that.

是的,Linux 内核的质量非常高,是的,这很困难,但我们不能在这一点上妥协。

We cannot compromise on quality because in the end—and you have to understand that—the core community of VLC is five people.

我们不能在质量上妥协,因为最终——你必须明白这一点——VLC 的核心社区是五个人。

52:30 The core community of FFmpeg is ten to fifteen, and we are the ones who are going to maintain your code, right?

FFmpeg 的核心社区有十到十五个,我们是维护您代码的人,对吧?

Because one thousand contributors in the timeline and just ten staying, it’s a one percent chance that someone comes and stays.

因为时间线上有 1000 个贡献者,但只有 10 个贡献者留下来,所以有人来并留下来的可能性是百分之一。

One percent.

百分之一。

So you will have change of job, change of wives, you have children, you have accidents in life.

所以你会换工作,换妻子,生孩子,生活中会有意外。

You’re going to change jobs, whatever.

无论如何,你都会换工作。

You’re not going to come back.

你不会回来的。

It’s most likely.

这是最有可能的。

So we are the ones going to maintain your code.

所以我们是维护您的代码的人。

It needs to be maintainable.

它需要可维护。

It needs to be excellent.

它必须是优秀的。

And yes, sometimes that means that you need to rework your work because it was good, but it’s not excellent, and we need excellence because we are very few to maintain something that is critical for the whole.

是的,有时这意味着你需要重新做你的工作,因为它很好,但并不出色,而我们需要卓越,因为我们很少有人维护对整体至关重要的东西。

Lex Fridman53:15

But we should also mention that there is some spiciness, some harshness to the language that’s sometimes used when you’re keeping this high bar of excellence.

但我们还应该提到,当你保持如此高的卓越标准时,有时会使用一些辛辣的语言,一些严厉的语言。

53:15 Is there something to say to that?

对此有什么要说的吗?

Jean-Baptiste Kempf53:27

It’s true, right?

这是真的,对吧?

53:27 It’s also the fact that, for example, what we’re doing is low level.

举例来说,这也是一个事实,我们所做的事情是低水平的。

It’s extremely technical.

这是非常技术性的。

You get into this community.

你进入这个社区。

The tone gets very like a type of— It’s a subculture, right?

语气变得非常像一种——这是一种亚文化,对吧?

So people who arrive from the external are basically not known to the subculture.

所以从外部来的人基本上不被亚文化所认识。

Most of those people around FFmpeg and VLC, we do VideoLAN DevDays, VDD every year.

大多数关注 FFmpeg 和 VLC 的人,我们每年都会举办 VideoLAN DevDays、VDD。

They are so fun in real life, and they love it.

他们在现实生活中非常有趣,而且他们喜欢它。

But it’s true that you’re online and sometimes, like, the tone, you don’t realize how it is.

但确实,你在网上,有时,比如语气,你并没有意识到它是怎么回事。

But that’s okay.

但没关系。

Lex Fridman54:01

It’s a culture.

这是一种文化。

54:01 I mean, you get this in the gaming culture.

我的意思是,你在游戏文化中就明白了这一点。

There’s pretty harsh, intense, the way people communicate, and it’s— everyone understands that the way you show love and respect just looks different in different communities.

人们的沟通方式非常严厉、激烈,每个人都明白,在不同的社区中,表达爱和尊重的方式看起来有所不同。

Sometimes people…

有时人们...

It depends.

这取决于。

If it’s a book club, usually people are going to be much sweeter.

如果是读书俱乐部,通常人们会更加友善。

If it’s an open source project that’s very high stakes and used by millions of people-

如果它是一个风险非常高并且被数百万人使用的开源项目——

Jean-Baptiste Kempf54:24

But it’s very not often insults that you see, for example, in the gaming, right?

但你很少会在游戏中看到侮辱,对吧?

54:24 And so Linus’ tone is a bit unusual even for the open source community.

因此,即使对于开源社区来说,Linus 的语气也有点不寻常。

It’s more like it’s more harsh on the results, saying, “No, this is not good.

更像是对结果更加严厉,说:“不,这不好。

This is crap.” Those type of things that you will see.

这是垃圾。”你会看到这些类型的东西。

Lex Fridman54:40

Try not to make it about the person, make it about the code.

尽量不要把重点放在人身上,而要放在代码上。

Jean-Baptiste Kempf54:42

Yes.

是的。

Kieran Kunhya54:43

It’s very, very matter of fact, and I think you’ve got to look at it in terms of, you know, the famous FFmpeg is developed almost entirely by volunteers, and that’s true, and you’ve got to imagine someone’s done a hard day’s work at their day job.

事实上,这是非常非常重要的,我认为你必须从以下角度来看待它,你知道,著名的 FFmpeg 几乎完全是由志愿者开发的,这是事实,你必须想象有人在他们的日常工作中完成了一天的辛苦工作。

54:43 They come home.

他们回家了。

You know, terseness might be a thing, you know, it…

你知道,简洁可能是一件事,你知道,它......

And that’s not something to take personally.

这不是针对个人的事情。

Lex Fridman55:01

You’re tired, you’re busy, but you still care about this open source stuff.

你很累,你很忙,但你仍然关心这个开源的东西。

55:01 But you may not be able to explain and handhold someone on every subtle detail.

但你可能无法向某人解释并掌握每一个微妙的细节。

Jean-Baptiste Kempf55:10

And also you have to realize that most people don’t speak English as native language.

而且你还必须意识到大多数人的母语不是英语。

55:10 And this is especially for open source projects like FFmpeg and VLC, which are mostly centered out of Europe.

这尤其适用于 FFmpeg 和 VLC 等开源项目,这些项目大多集中在欧洲以外。

Sometimes people who are from the US or just are very not happy about the tone, but most of the time it’s also like they don’t know better, right?

有时来自美国的人或只是对这种语气感到非常不满意,但大多数时候他们也不太了解,对吧?

It’s difficult.

它很难。

The language is—English is a difficult language.

语言是——英语是一门很难的语言。

There is so many subtleties and tone and so on that you don’t have, right?

有很多你所不具备的微妙之处和语气等等,对吧?

So often it’s also difficult in those type of communities about different cultures and languages.

通常,在具有不同文化和语言的社区中,这也是很困难的。

Lex Fridman55:46

So as the legend goes, JB, you repeatedly turned down millions of dollars to keep VLC open source free for everyone without ads.

因此,正如传说所说,JB,您多次拒绝了数百万美元,以保持 VLC 对所有人免费开放源代码且没有广告。

55:46 So take me through the reasoning behind that decision of leaving millions of dollars on the table.

那么请带我了解一下留下数百万美元的决定背后的原因。

Jean-Baptiste Kempf56:06

Yeah, that’s like almost a meme, right, on Reddit or-

是的,这几乎就像 Reddit 上的一个模因,对吧,或者——

Lex Fridman56:09

There literally is a meme on Reddit.

Reddit 上确实有一个模因。

Jean-Baptiste Kempf56:11

9GAG and yeah, yeah.

9GAG 是的,是的。

56:11 See, there’s-

看,有——

Lex Fridman56:14

You looking like a wizard in the, in the VLC hat on Reddit.

你看起来就像 Reddit 上戴着 VLC 帽子的巫师。

56:14 This is JB, the creator of VLC media player.

我是 JB,VLC 媒体播放器的创建者。

He refused tens of millions of dollars in order to keep VLC ads free.

为了保持 VLC 广告免费,他拒绝了数千万美元。

Thanks, Jean-Baptiste Kempf.

谢谢让-巴蒂斯特·肯普夫。

You can even summon him on Reddit.

你甚至可以在 Reddit 上召唤他。

Jean-Baptiste Kempf56:33

Yeah.

是的。

56:33 And usually if you see, right, it’s usually like people tag me, right?

通常,如果你看到,对吧,通常就像人们给我贴上标签,对吧?

And then there is me, and then like I say, “Good morning.”

然后是我,然后就像我说的,“早上好。”

Lex Fridman56:40

Good morning.

早上好。

Jean-Baptiste Kempf56:40

I got twenty-four K upvotes, which is great, right?

我得到了 24 K 个赞成票,这很棒,对吧?

56:40 My karma on Reddit is amazing, at least on that account.

我在 Reddit 上的业力是惊人的,至少在这个方面是这样。

So the question is, needs to be answered first, what is the story about VLC, right?

那么问题来了,首先需要回答的是,VLC 的故事是怎样的,对吧?

Because yes, this is true, I refuse dozens of millions of dollars, yes, several times.

因为是的,这是真的,我拒绝了数千万美元,是的,好几次。

Yes, I could be a multimillionaire and be somewhere on the beach.

是的,我可以成为一个千万富翁并在海滩上的某个地方。

But I did not do it because I thought it was not moral and it was not the right thing to do.

但我没有这样做,因为我认为这不道德,也不是正确的做法。

And this is very important for myself, is to be like, I work for the greater good, I work for people, and I don’t want—it’s not just by myself.

这对我自己来说非常重要,就是说,我为更大的利益而工作,我为人们工作,而我不希望——这不仅仅是我自己的事。

But the reason is also because I did not feel that I’m completely legitimate to do that, and let me explain you why.

但原因也是因为我觉得我这样做并不完全合法,让我向你解释一下原因。

VLC’s story is a very weird story.

VLC的故事是一个非常奇怪的故事。

57:29 In France, we have university and we have a type of top colleges, and those top of excellency schools are engineering schools, business schools, and basically lawyers and medical, right?

在法国,我们有大学,有一种顶尖的学院,那些顶尖的优秀学校是工程学院、商学院,基本上还有律师和医学院,对吗?

But they’re outside of university, and in order to enter those, you spend two years working like crazy math, physics to enter those best engineering schools.

但它们不在大学之外,为了进入这些学校,你需要花两年时间像疯狂的数学、物理一样努力,才能进入那些最好的工程学校。

One of the school is called the École Centrale Paris.

其中一所学校称为巴黎中央学校。

It has changed name since, but it was called the École Centrale Paris.

此后它已更名,但被称为巴黎中央理工学院。

And because it was Centrale, they had to move it because it was too small after the World War II and they moved it, they wanted to move it to the central of France in a place called Clermont-Ferrand.

因为它是Centrale,所以他们不得不把它搬走,因为它太小了,二战后他们搬走了它,他们想把它搬到法国中部一个叫克莱蒙费朗的地方。

And the alumni decided that this was not okay, right?

校友们认为这不好,对吧?

58:12 It is the school that Eiffel, right, the one who did the Eiffel Tower, attended to, right?

这就是埃菲尔铁塔的建造者埃菲尔就读的学校,对吗?

So they said, “No, no, we are amazing, great school.

所以他们说:“不,不,我们是很棒的、很棒的学校。

We cannot do that.” And so they bought a piece of land south of Paris very near Paris.

我们不能那样做。”于是他们在巴黎南部买了一块非常靠近巴黎的土地。

And it was a campus managed by a nonprofit of the alumnis, okay?

这是一个由校友非营利组织管理的校园,好吗?

Because of that, everything on the campus was managed by students.

正因为如此,校园里的一切都是由学生管理的。

The university did nothing, right?

大学什么也没做吧?

So radio, TV, supermarket, library, defining who was going into which rooms.

因此,广播、电视、超市、图书馆,定义了谁进入哪些房间。

Everything was managed by the students.

一切都由学生来管理。

Lex Fridman58:48

That’s amazing.

太棒了。

58:48 That’s an amazing experiment, that it all didn’t go to hell quickly.

这是一个了不起的实验,一切并没有很快就陷入困境。

It somehow flourished.

它以某种方式蓬勃发展。

Jean-Baptiste Kempf58:55

It worked great, and I learned so much in my life doing those side activities, right?

它效果很好,我在生活中通过这些副业活动学到了很多东西,对吗?

58:55 Because you’re twenty-two and you need to run your campus, else you don’t have electricity, right?

因为你二十二岁了,你需要管理你的校园,否则你就没有电,对吧?

So you care about that, right?

所以你很关心这个,对吧?

But anyway, in the ’80s they did a full experiment of deploying a network mostly sponsored by IBM and 3Com, which was a token ring network.

但无论如何,在 80 年代,他们进行了一次全面的实验,部署了一个主要由 IBM 和 3Com 赞助的网络,这是一个令牌环网络。

So token ring is something that probably almost no one knows about anymore.

所以令牌环可能几乎没有人知道了。

It’s a networking technology where you don’t have routers, right?

这是一种不需要路由器的网络技术,对吗?

59:26 Everyone is linked.

每个人都是相连的。

It’s like really a ring, and when you want to send a message, you talk to your neighbor who’s going to put the message to the next one, who’s going to put the things to the next one, in terms of ring.

这就像一个真正的戒指,当你想发送一条消息时,你会与你的邻居交谈,他会将消息传递给下一个,谁会将东西传递给下一个,就戒指而言。

The issue with token ring is, of course, is that it’s very slow because every computer on the network needs to open the message, see if it’s okay.

当然,令牌环的问题是它非常慢,因为网络上的每台计算机都需要打开消息,看看是否可以。

Is it for me?

是给我的吗?

No, it’s not, and then send it back, like a token which is traveling around the ring.

不,不是,然后将其发回,就像在环上移动的令牌一样。

In the ’80s, you’re doing some Telnet and sending mails as university.

在 80 年代,大学时你会使用 Telnet 并发送邮件。

That’s okay, right?

没关系,对吧?

But starts the ’90s, and the ’90s and start video games, and when you have high latency in video games, basically you die, right?

但从 90 年代开始,90 年代开始玩电子游戏,当电子游戏出现高延迟时,基本上你就会死掉,对吧?

1:00:08 So in nineteen ninety-four, nineteen ninety-five, around Doom and Duke Nukem coming around, they want a faster network.

因此,在十九点九十四、十九点九十五的时候,在《毁灭战士》和《毁灭公爵》即将到来的时候,他们想要一个更快的网络。

So the students go and see the university and say, “You know what?

所以学生们去参观大学并说:“你知道吗?

We want a faster network.

我们想要更快的网络。

We need to work,” and also play video games.

我们需要工作”,还需要玩电子游戏。

And the university tells them that basically, “Oh, I’m sorry, we cannot help you because you understand the campus is not ours.

大学告诉他们,基本上,“哦,对不起,我们无法帮助你,因为你知道校园不是我们的。

You manage it, so do something.

你管理它,所以做点什么。

And you should see some basically partners of this university and basically go away.” And they go, and they actually go and see the CIO of Bouygues, which is a large French company and who’s doing some TVs in France.

你应该看到这所大学的一些基本上的合作伙伴,然后基本上就走开。”他们去了,他们实际上去见了 Bouygues 的首席信息官,这是一家法国大公司,在法国生产一些电视。

And he says, “Well, you know what?

他说:“好吧,你知道吗?

1:00:50 The future of video is satellite.” Well, today we know it’s not, but at least it was a good idea.

视频的未来是卫星。”好吧,今天我们知道这不是,但至少这是一个好主意。

In nineteen ninety-five, the first satellite dish, and he says that instead of having like one satellite dish and a big decoder for each of the students, which are one thousand and five hundred, what about you build, like you put an enormous dish and only one decoder, and you send the video directly on the network.

1995 年,第一个卫星天线,他说,与其为每个学生(一千五百名)配备一个卫星天线和一个大型解码器,不如建造一个巨大的天线和一个解码器,然后直接在网络上发送视频。

And that required a very fast network.

这需要非常快的网络。

Today, it’s obvious, but at the time was, like, the first to do video streaming.

今天,这是显而易见的,但在当时,它是第一个进行视频流媒体的。

So they built this project, which was called Network 2000.

所以他们建立了这个项目,称为 Network 2000。

Of course, right, we are in the ’90s, right?

当然,对吧,我们现在是90年代,对吧?

Everything futuristic is called 2000, like-

一切未来派的东西都被称为2000,比如——

Lex Fridman1:01:30

Yeah, 2000, yeah.

是的,2000 年,是的。

Jean-Baptiste Kempf1:01:32

And so they do the Network 2000 project.

因此他们开展了 Network 2000 项目。

1:01:32 It’s completely hacked.

它完全被黑客入侵了。

It crashes after 45 seconds.

45 秒后崩溃。

That’s okay.

没关系。

The demo is 40 seconds.

演示时长 40 秒。

It leaks memory.

它会泄漏内存。

That’s okay.

没关系。

They put 64 megabytes of RAM instead of the 8 or 16 you have, and the demo should have stopped there.

他们放置了 64 MB 的 RAM,而不是您拥有的 8 或 16 MB,演示应该就此停止。

And that was the Network 2000 project by the students.

这就是学生们的 Network 2000 项目。

Lex Fridman1:01:49

What was the format of the video that they had to work with?

他们必须使用的视频格式是什么?

Jean-Baptiste Kempf1:01:52

MPEG-2 because satellite is MPEG-2 TS for transport, MPEG-2 video, and MPEG-2 audio at that time.

MPEG-2,因为卫星当时是用于传输的 MPEG-2 TS、MPEG-2 视频和 MPEG-2 音频。

1:01:52 And the project should have stopped there.

该项目应该就此停止。

Everyone was happy.

大家都很高兴。

They had, like, amazing ATM network at 155 megabits per second.

他们拥有令人惊叹的 ATM 网络,每秒 155 兆比特。

They had probably one of the best networks in Europe at that time, and they stopped the project.

他们可能拥有当时欧洲最好的网络之一,但他们停止了该项目。

Six months or a year later, two students arrive and say, “Well, you know what?

六个月或一年后,两名学生到达并说:“嗯,你知道吗?

Maybe other people care about video streamed on a local network,” and they create the VideoLAN project, VideoLAN.

也许其他人关心本地网络上的视频流”,他们创建了 VideoLAN 项目 VideoLAN。

And one of them is called Christophe Massot, that is a good friend of both Kieran and me, and they start the project.

其中一位名叫克里斯托夫·马索(Christophe Massot),他是基兰和我的好朋友,他们启动了这个项目。

It’s not even open source yet, and they spend around three years to get the school to agree to make it open source.

它甚至还没有开源,他们花了大约三年的时间才让学校同意将其开源。

1:02:38 Because the university wanted to get some– because of the IP and copyright of the students, wanted to basically monetize these MPEG-2 decoders.

因为大学想要得到一些——因为学生的 IP 和版权,想要基本上通过这些 MPEG-2 解码器货币化。

Lex Fridman1:02:47

Just to be clear, so what was the main application, streaming on a local network?

需要明确的是,主要应用程序是什么,在本地网络上进行流式传输?

Jean-Baptiste Kempf1:02:50

It was streaming on a local network.

它在本地网络上流式传输。

Lex Fridman1:02:52

By the way, that’s just, like, to state the obvious.

顺便说一句,这只是陈述显而易见的事情。

1:02:52 This is before YouTube.

这是 YouTube 之前的事。

This is before-

这是之前的——

Jean-Baptiste Kempf1:02:56

Ten years before YouTube.

比 YouTube 早十年。

1:02:56 You have a Pentium 60 or 75, right?

您有 Pentium 60 或 75,对吧?

You, the main machine was 486DX at 33 megahertz, right?

你,主机是486DX,33兆吧?

Kieran Kunhya1:03:04

Bear in mind, television was the main form of video at the time.

请记住,电视是当时视频的主要形式。

1:03:04 You could get new channels.

你可以获得新的频道。

In the ’90s, having even one new channel when you grew up with four channels, having a fifth or a sixth was a big deal, and so having this satellite service with, you know, dozens, even hundreds of channels was so groundbreaking.

在 90 年代,当你在四个频道长大的时候,即使拥有一个新频道,拥有第五个或第六个频道也是一件大事,因此拥有数十个甚至数百个频道的卫星服务是如此具有开创性。

Jean-Baptiste Kempf1:03:22

Especially because this is university where you had a ton of different nationalities, right?

尤其是因为这所大学里有很多不同国籍的人,对吧?

1:03:22 So there was a ton of people who wanted…

所以有很多人想要……

So in the end, they had, like, several dishes on different types of satellites, right?

所以最后,他们在不同类型的卫星上安装了几个天线,对吗?

Because, for example, a lot of people were coming from the Maghreb or the Middle East and they went to different types of satellites.

因为,例如,很多人来自马格里布或中东,他们去了不同类型的卫星。

Anyway, the solution worked great, and they started the VideoLAN project.

不管怎样,这个解决方案效果很好,他们开始了 VideoLAN 项目。

The VideoLAN project has several, and some are completely crazy solutions, like one how to create multicast on a unicast network, but let’s not come to that.

VideoLAN 项目有多种解决方案,其中一些是完全疯狂的解决方案,例如如何在单播网络上创建多播,但我们先不讨论这个。

It’s too complex.

太复杂了。

But the VideoLAN Client part is what became VLC.

但 VideoLAN 客户端部分变成了 VLC。

1:04:03 Actually, they basically strong-armed the university to force it to open source, because the university did not understand that.

事实上,他们基本上是用武力强迫大学开源,因为大学不明白这一点。

And in 2001, it’s still early.

2001年,现在还为时过早。

But basically, yes, the university agreed early 2001 to make it open source.

但基本上,是的,大学在 2001 年初同意将其开源。

I joined the project in 2003 because that’s when I joined the university.

我于 2003 年加入了这个项目,因为那一年我加入了大学。

So the first thing is I’m not the one who created VLC, because actually no one did, right?

所以第一件事是我不是创建 VLC 的人,因为实际上没有人创建,对吧?

Lex Fridman1:04:26

Just kind of naturally emerged from the VideoLAN project.

只是从 VideoLAN 项目中自然而然地出现的。

1:04:26 And we should mention that, like, again, you said it just, but to make it clear, VideoLAN as what it became at the time was a set of technologies around video, and the VLC, what you called the client, that’s the thing that most normies, uh-

我们应该提到这一点,就像你刚才所说的那样,但要明确的是,VideoLAN 当时是一套围绕视频的技术,而 VLC,你所说的客户端,是大多数规范的东西,呃-

Jean-Baptiste Kempf1:04:47

That is correct, and-

这是正确的,而且——

Lex Fridman1:04:48

…

……

1:04:48 think of, like, as the thing, which is, like, the thing that pops up when you click on a video and you play it.

想象一下,当你点击视频并播放它时弹出的东西。

Jean-Baptiste Kempf1:04:53

So I arrive in 2003, and then I will create the open source nonprofit organization called VideoLAN, and I took everything out of the university to create it as a nonprofit project and something sustainable.

所以我在 2003 年到达,然后我将创建一个名为 VideoLAN 的开源非营利组织,我从大学中汲取一切,将其创建为一个非营利项目和可持续发展的项目。

1:04:53 It’s, yes, it’s true that I spent more time than anyone on VLC and VideoLAN.

是的,我在 VLC 和 VideoLAN 上花费的时间确实比任何人都多。

That is sure.

这是肯定的。

But it’s a continuity of a previous project, VideoLAN, the student project, which is a continuity of the Network 2000 project, which is a continuity of that and that.

但它是之前项目 VideoLAN(学生项目)的延续,它是 Network 2000 项目的延续,也是那个项目的延续。

Lex Fridman1:05:22

I’m sure there’s moments along the way there you were thinking of, like, what is the future of this from an open source perspective?

我确信您在这个过程中有时会想到,比如,从开源的角度来看,它的未来是什么?

1:05:22 ‘Cause as, as the internet is blowing up, and there is companies…

因为,随着互联网的蓬勃发展,并且有公司......

I mean, for people who don’t remember, like, there’s companies making huge amounts of money.

我的意思是,对于那些不记得的人来说,有些公司赚了很多钱。

Jean-Baptiste Kempf1:05:39

And I can tell you that in 2005, the project should have died and I made it to continue the project.

我可以告诉你,在 2005 年,这个项目应该已经终止,而我却成功地继续了这个项目。

1:05:39 At some point, we were only two active developers.

在某些时候,我们只是两个活跃的开发人员。

And I thought it was great technology and was useful, and it will be useful, and I made that my life and my time.

我认为这是一项伟大的技术,很有用,而且将会很有用,我把它变成了我的生活和时间。

And I made that grow from a few hundreds of thousands of users, millions of users to what we have now, which is probably billions of versions of VLC around the world and used everywhere.

我让这个用户从几十万、数百万用户发展到我们现在所拥有的,全世界可能有数十亿个 VLC 版本,并在世界各地使用。

So that’s a bit the story of VLC.

这就是 VLC 的一些故事。

There is a ton of very funny stories around that.

围绕它有很多非常有趣的故事。

Many people from around the world working on it, like you said, in Syria or the middle of nowhere in India.

正如你所说,来自世界各地的许多人在叙利亚或印度的偏僻地区致力于此。

But along the way, I got several offers which were either to bundle toolbars, right?

但一路走来,我收到了几个报价,要么是捆绑工具栏,对吧?

You remember those horrible toolbars-

你还记得那些可怕的工具栏-

1:06:35 …

……

which were basically spyware, or changing your web browser or your search engine or even, like, advertisement inside VLC.

这些基本上是间谍软件,或者更改您的网络浏览器或搜索引擎,甚至是 VLC 内的广告。

And I didn’t like that, right?

我不喜欢这样,对吧?

I am– and people don’t understand that.

我是——但人们不明白这一点。

It’s not– I’m not against money, right?

不是——我不反对金钱,对吧?

I’m very happy to make money.

我很高兴能赚钱。

I created several startups and one I hope that is going to work very well.

我创建了几家初创公司,我希望其中一家能够运作良好。

It’s the fact that I believe that you need to win money ethically.

事实上,我相信你需要道德地赢钱。

There is a right way of doing that, and doing sneaky advertisement or stealing data is not the correct way, right?

有一个正确的方法,但偷偷做广告或窃取数据不是正确的方法,对吧?

For example, if Netflix arrived at some point and said, “Well, we want to put Netflix inside VLC,” probably the story would have been different, right?

例如,如果 Netflix 在某个时候说:“好吧,我们想将 Netflix 放入 VLC 中”,那么故事可能会有所不同,对吗?

But they didn’t.

但他们没有。

The only people who came to us were shady ads companies.

唯一来找我们的人是阴暗的广告公司。

1:07:22 And if I do that, right, I would have a ton of money, right?

如果我这样做,对吧,我就会有很多钱,对吧?

And then three years later, the project is gone, right?

然后三年后,这个项目就消失了,对吗?

Someone forks it and something else happens.

有人分叉了它,然后发生了其他事情。

Lex Fridman1:07:31

So it’s not even necessarily ads or any of that, it’s the shadiness of the- …

所以这甚至不一定是广告或其他任何东西,而是……的阴暗面。

1:07:31 dishonesty of the– So you had a good radar, you had a good threshold of like, “No, this compromises the spirit of what this is supposed to represent.”

不诚实——所以你有一个很好的雷达,你有一个很好的门槛,“不,这损害了它应该代表的精神。”

Jean-Baptiste Kempf1:07:46

But also it’s for me, right?

但这也是为了我,对吧?

1:07:46 I’m like very selfishly, I need to go to bed at night and be happy about what I’ve done, right?

我非常自私,晚上睡觉时我需要为自己所做的事情感到高兴,对吗?

Maybe it’s my upbringing, maybe it’s my parents’ fault or whatever, right?

也许这是我的成长经历,也许是我父母的错,或者其他什么,对吧?

But I believe there is right and wrong, right?

但我相信有对有错,对吗?

And this was the right decision at the time.

这在当时是正确的决定。

It still is.

现在仍然如此。

I want to be proud of what I’ve been doing.

我想为我所做的事情感到自豪。

And like, if I had sold out, I would have betrayed so many other people who work here.

就像,如果我卖光了,我就会背叛很多在这里工作的人。

Lex Fridman1:08:14

Yeah, well, I should say me and most of the internet thank you for that decision.

是的,好吧,我应该说我和互联网上的大多数人都感谢你的决定。

1:08:14 It’s inspiring for others I think that are pushing the open source movement forward, that it’s okay to do these kinds of huge sacrifices if you believe it’s right.

我认为这对推动开源运动向前发展的其他人来说是鼓舞人心的,如果你相信这是正确的,那么做出这些巨大的牺牲是可以的。

And I think in that case it was right and it was the reason that VLC became as successful as it was, ’cause it’s an embodiment, it’s a symbol of like, you know, freedom and what the open source community can create.

我认为在这种情况下这是正确的,这也是 VLC 取得如此成功的原因,因为它是一种体现,它是自由和开源社区可以创造的东西的象征。

Jean-Baptiste Kempf1:08:46

Yeah, and be a service for so many people around the world, and this is important.

是的,为世界各地的许多人提供服务,这很重要。

Kieran Kunhya1:08:50

We should emphasize in the 2000s it was really normal to download a program and it secretly installs some spyware.

我们应该强调,在 2000 年代,下载程序并秘密安装一些间谍软件确实很正常。

1:08:50 It was, it was buried in very faint text or in the license text box that nobody reads at the bottom- …

它是,它被埋在非常微弱的文本中或在许可证文本框中,没有人在底部阅读 - ......

“Oh, I will be installing this toolbar- …

“哦,我将安装这个工具栏 - ...

and changing all these things,” and it was very common to have to, you know, you install a program to do something at the time of any sort.

并改变所有这些事情”,你知道,在任何时间都必须安装一个程序来做某事是很常见的。

Lex Fridman1:09:11

To put yourself in the mind of a developer at that time, I think it’s very easy—to everybody listening to this, it’s very easy at that time to convince yourself to take a few thousand dollars- …

把自己放在当时的开发者心里,我觉得很容易——对每个听这个的人来说,当时很容易说服自己拿几千美元——……

1:09:11 a few thousand dollars to do it.

几千美元就可以做到。

To say no to much more money- …

对更多的钱说不-...

takes guts and takes vision.

需要勇气和远见。

Jean-Baptiste Kempf1:09:34

The last offer I had was obscene, and they say, “Yeah, but imagine with all that money you could build something new, open source,” right?

我收到的最后一个报价是淫秽的,他们说,“是的,但是想象一下,用那么多钱你可以构建一些新的开源东西,”对吗?

1:09:34 It was like the mind trick was…

就好像心灵戏法是……

it was difficult.

这很困难。

But for me it was just like, “No, this doesn’t work like that or this is not the right thing, so I don’t do it.” And again, right, it’s not that I don’t like money or whatever.

但对我来说,就像是,“不,这不行,或者这不是正确的事情,所以我不这样做。”再说一遍,对,这并不是说我不喜欢钱或其他什么。

It’s just like it wasn’t right.

就好像这是不对的。

Lex Fridman1:10:01

Well, once again, thank you from me and from the rest of the internet.

好吧,再次感谢我和互联网上的其他人。

1:10:01 Let me talk a little bit more about the open source movement, about the fact that, as you say over and over and over and over, FFmpeg is and many open source projects are built by volunteers.

让我再多谈谈开源运动,正如您一遍又一遍地说的那样,FFmpeg 以及许多开源项目都是由志愿者构建的。

So there’s a bit of drama recently, Kieran, on the interwebs, on Twitter.

Kieran,最近互联网上、推特上发生了一些戏剧性事件。

You have a spicy style on Twitter that I think articulates and celebrates all the incredible developers and development and the code, especially assembly, that’s involved in building some of these codecs and building some of this incredible technology.

你在 Twitter 上有一种辛辣的风格,我认为它阐明并庆祝了所有令人难以置信的开发人员和开发以及代码,尤其是汇编,它们涉及构建其中一些编解码器和构建一些令人难以置信的技术。

But that brings us to the…

但这让我们想到了……

a bit of a debacle that happened.

发生了一点崩溃。

Tell me the full saga of what happened with the Google security engineers.

告诉我谷歌安全工程师发生的完整故事。

Kieran Kunhya1:10:50

Just to be clear, Google are one of the biggest supporters of open source out there.

需要明确的是,谷歌是开源的最大支持者之一。

1:10:50 They have been for a long time.

他们已经存在很长时间了。

It’s just I think some things kind of went a bit overboard this time.

只是我觉得这次有些事情有点过分了。

So FFmpeg itself—and this is not like a secret, it’s on the homepage, you know—it processes untrusted data.

所以 FFmpeg 本身(这不像秘密,它在主页上,你知道)它处理不受信任的数据。

There can be security issues when you parse untrusted data.

解析不受信任的数据时可能会出现安全问题。

That’s very normal.

这很正常。

But recently what changed was Google started using AI to create security reports on an open source project, FFmpeg.

但最近发生的变化是,谷歌开始使用人工智能为开源项目 FFmpeg 创建安全报告。

Volunteers had to deal with that.

志愿者必须解决这个问题。

They did, they provided very limited funding, and they even went to the media first announcing how good their AI was before the issues could be fixed.

他们做到了,他们提供了非常有限的资金,他们甚至在问题得到解决之前先向媒体宣布他们的人工智能有多好。

Lex Fridman1:11:29

And this is in the public forum.

这是在公共论坛中。

Kieran Kunhya1:11:31

Yeah, this is all public.

是的,这都是公开的。

Lex Fridman1:11:32

So reporting an issue, using AI to find an issue in the code which is a security vulnerability, and then reporting that publicly before you’re able to fix it.

因此,报告问题,使用人工智能来查找代码中的安全漏洞问题,然后在能够修复它之前公开报告该问题。

Kieran Kunhya1:11:40

Yeah.

是的。

1:11:40 It’s announcing how good their AI is, that they provided a standard 90-day industry deadline without really understanding the nature of volunteer-driven development.

他们在没有真正理解志愿者驱动开发的本质的情况下提供了标准的 90 天行业截止日期,这宣告了他们的人工智能有多么出色。

In addition, this vulnerability was on an obscure 1990s game codec.

此外,此漏洞存在于 90 年代一个不起眼的游戏编解码器上。

The way—and let’s look at it from their standpoint to begin with.

方式——让我们首先从他们的角度来看。

Let, let’s you know-

让,让你知道——

Lex Fridman1:12:04

Yeah.

是的。

1:12:04 Can you steer me in their case?

你能指导我处理他们的情况吗?

Kieran Kunhya1:12:06

Yeah, sure.

是的,当然。

1:12:06 They have substantial resources working on the security of open source projects that, you know, are ubiquitous, and they’ve used a lot of compute to do that and very expensive and very capable security researchers to do that.

他们拥有大量资源致力于开源项目的安全性,你知道,这些项目无处不在,他们使用了大量的计算来做到这一点,并且使用了非常昂贵且非常有能力的安全研究人员来做到这一点。

And that’s their viewpoint is they are contributing by doing that.

他们的观点是,他们通过这样做做出了贡献。

But I think that’s where opinions differ.

但我认为这就是意见不同的地方。

It opened up a lot of interesting fissures, I would say.

我想说,它打开了很多有趣的裂痕。

It does seem that there’s a portion of the security community that look at themselves a bit like building architects that never have to go to site.

安全社区中确实有一部分人认为自己有点像永远不需要去现场的建筑建筑师。

You know, going to site is something that is a little bit beneath them, the actual day-to-day construction.

你知道,去现场对于他们来说有点低级,实际的日常施工。

They’re there to do their security things and it’s someone else’s problem.

他们在那里做他们的安全事情,这是别人的问题。

1:12:54 The security industry also kind of has a very aggressive tone towards things.

安全行业对事物也有一种非常激进的态度。

The language they use is extremely aggressive.

他们使用的语言极具攻击性。

They use very strong language like, “You will get popped.” So and to Joe Public, “get popped,” you know, means something quite bad.

他们使用非常强烈的语言,比如“你会被淘汰的。”因此,对于 Joe Public 来说,“被弹出”,你知道,意味着非常糟糕的事情。

For them it means to get hacked.

对他们来说,这意味着被黑客攻击。

The way I would look at it personally is a little bit like the padlock on your home.

我个人认为它有点像你家的挂锁。

Not everyone…

不是每个人…

The padlock on your home or, you know, the lock on your home is there to protect against the capabilities of what it’s there to protect.

你家的挂锁,或者你知道的,你家的锁是为了防止它所要保护的东西的能力。

It’s not there to protect nuclear secrets.

它不是为了保护核秘密。

It’s not there to protect Fort Knox.

它不是为了保护诺克斯堡。

And it could be looked at that they’re using AI at a level of scale to go and pick those locks and then say, “Hey, your lock’s not secure.”

可以看出,他们正在大规模地使用人工智能来撬锁,然后说:“嘿,你的锁不安全。”

1:13:42 You need to deal with this.” Whereas actually they’re the ones with resources to be able to fix this.

你需要处理这个问题。”而实际上他们是拥有资源来解决这个问题的人。

But that seems to not be something either they’ll contribute to in terms of patches or in terms of financially.

但他们似乎不会在补丁或经济方面做出贡献。

And the scale of AI is kind of the issue.

人工智能的规模就是一个问题。

The bug reports are very wordy.

错误报告非常冗长。

They’re very, very—it’s almost a denial of service by AI-generated bug reports on very niche codecs.

它们非常非常——这几乎是人工智能在非常小众的编解码器上生成的错误报告造成的拒绝服务。

And the other issue the security community has is everything is marked high priority.

安全社区面临的另一个问题是一切都被标记为高优先级。

You’re going to, you know, “This is the most important thing in the world, and you need to deal with this.

你会,你知道,“这是世界上最重要的事情,你需要处理它。

High, high, high, vulnerable, scary, scary, scary,” on a game codec used on one disk in 1993.

高,高,高,脆弱,可怕,可怕,可怕”,1993 年在一张磁盘上使用的游戏编解码器上。

1:14:24 And that’s where the dichotomy lies.

这就是二分法所在。

Going around telling everyone that their padlock’s not safe—well, that’s a hobby project of somebody.

到处告诉每个人他们的挂锁不安全——好吧,这是某人的业余爱好项目。

The safety of that codec is consummate to what that person thinks.

该编解码器的安全性与该人的想法完全一致。

It’s their hobby.

这是他们的爱好。

It’s good that they’re security analyzing it, but it doesn’t need a big scary warning, “This is a critical vulnerability.” You may also recently see that there was another quote-unquote vulnerability.

他们正在对其进行安全分析,这很好,但不需要一个可怕的警告,“这是一个严重的漏洞。”您最近可能还发现存在另一个引用-取消引用漏洞。

It wasn’t Google in this case, but a filter could overflow and have an integer overflow, and one of your pixels could be the wrong color.

在这种情况下,不是 Google,但过滤器可能会溢出并出现整数溢出,并且您的某个像素可能是错误的颜色。

And this was marked high, 7.5 severity in red.

这被标记为高,严重程度为 7.5,用红色表示。

1:15:05 And at some point, the security industry needs to realize you can’t keep crying wolf like this because this just leads to people, you know, the equivalent thereof of putting password stickers on their PC.

在某种程度上,安全行业需要意识到你不能一直这样喊“狼来了”,因为这只会导致人们,你知道,相当于在他们的电脑上贴上密码贴纸。

You know, you can’t just keep crying wolf every day.

你知道,你不能每天都喊狼来了。

And I appreciate, you know, that’s their modus operandi is to create as much scare and fear.

我很欣赏,你知道,他们的作案手法就是制造尽可能多的恐慌和恐惧。

But from the Google standpoint, at the end of the day, they need to contribute either financially or with patches.

但从谷歌的角度来看,归根结底,他们需要提供经济或补丁方面的贡献。

Google uses FFmpeg at a scale probably you or I couldn’t even contemplate, millions of CPU cores.

Google 使用 FFmpeg 的规模可能是你我都无法想象的,数百万个 CPU 核心。

And yes, they contribute in areas mostly regarding their own products, so VP9, AV1.

是的,他们在主要涉及自己产品的领域做出了贡献,所以 VP9、AV1。

But in a wider sense, there’s a disproportionate level of contribution.

但从更广泛的意义上来说,贡献程度不成比例。

Yes, they fund students.

是的,他们资助学生。

Yes, they fund Summer of Code.

是的,他们资助了“编程之夏”。

1:15:57 And I think…

我认为……

so Alex Strange is a former FFmpeg developer, I think posting in a personal capacity.

Alex Strange 是前 FFmpeg 开发人员,我认为以个人身份发帖。

Lex Fridman1:16:02

So he posted about security engineers on Hacker News.

于是他在 Hacker News 上发布了有关安全工程师的文章。

1:16:02 His post reads, “The problem with security reports in general is security people are rampant self-promoters”—in parentheses, Linus once called them something worse.

他的帖子写道:“总体而言,安全报告的问题在于安全人员都是猖獗的自我推销者”——在括号中,莱纳斯曾经称他们为更糟糕的事情。

“Imagine you’re a humble volunteer open source developer.

“想象一下,您是一名谦虚的志愿者开源开发人员。

If a security researcher finds a bug in your code, they’re going to make up a cute name for it, start a website with a logo.

如果安全研究人员在你的代码中发现了一个错误,他们会为它起一个可爱的名字,启动一个带有徽标的网站。

Google is going to give them a million-dollar bounty.

谷歌将给予他们百万美元的赏金。

They’re going to go to DEF CON and get a prize, and I assume go to some kind of secret security people orgy where everyone is dressed like they’re in The Matrix.

他们要去 DEF CON 并获得奖品,我猜想是去参加某种秘密安全人员狂欢,每个人都穿得像《黑客帝国》里的一样。

Nobody is going to do any of this for you when you fix it.” Uh, basically commenting on the sort of the incentives for the different people involved and misaligned.

当你修复它时,没有人会为你做任何事情。”呃,基本上是评论对不同参与人员的激励措施和不一致的情况。

Jean-Baptiste Kempf1:17:01

The problem here is the disproportion of means on discovery compared to patching it, right?

这里的问题是发现手段与修补手段相比不成比例,对吧?

1:17:01 And this is the biggest issue, right?

这是最大的问题,对吗?

And after that debacle, Google did some changes.

在那次失败之后,谷歌做了一些改变。

Kieran Kunhya1:17:14

They are now starting to send patches, which is-

他们现在开始发送补丁,即-

Jean-Baptiste Kempf1:17:17

And they also now have reward tools for fixing issues.

他们现在还拥有解决问题的奖励工具。

1:17:17 So it, it has changed a bit because of that debacle.

所以,由于那次灾难,情况发生了一些变化。

So it’s good, right?

所以这很好,对吧?

But we’ve seen—and we talk about Google—but we have seen like some other large companies saying, “Oh, you need to fix this bug because it’s critical in our product.”

但我们已经看到——我们谈论谷歌——但我们也看到像其他一些大公司一样说,“哦,你需要修复这个错误,因为它对我们的产品至关重要。”

Lex Fridman1:17:33

Can you explain the XZ fiasco?

你能解释一下XZ的惨败吗?

1:17:33 The FFmpeg tweet reads, “The XZ fiasco has shown how a dependence on unpaid volunteers can cause major problems.

FFmpeg 推文写道:“XZ 的惨败表明,对无偿志愿者的依赖会导致严重问题。

Trillion-dollar corporations expect free and urgent support from volunteers.

价值数万亿美元的公司期待志愿者的免费和紧急支持。

Microsoft Teams posted on a bug tracker full of volunteers that their issue is high priority.

Microsoft Teams 在一个由志愿者组成的错误跟踪器上发布消息称,他们的问题具有高度优先级。

After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead.

在礼貌地向微软请求一份长期维护支持合同后,他们提出一次性支付几千美元。

This is unacceptable.

这是不可接受的。

We didn’t make it up.

我们没有弥补。

This is what Microsoft Teams actually did.” And then you give the image and the details and all that kind of stuff, showing that these trillion-dollar companies are not giving much money, not giving much support.

这就是 Microsoft Teams 实际所做的。”然后你提供图像和细节以及所有类似的东西,表明这些万亿美元的公司没有提供太多资金,没有提供太多支持。

Kieran Kunhya1:18:24

They think an open source project is a traditional vendor that they have an SLA.

他们认为开源项目是他们拥有 SLA 的传统供应商。

1:18:24 They think a public bug tracker is actually, you know, a third-party vendor’s Jira where you can do all of these things.

他们认为公共错误跟踪器实际上是第三方供应商的 Jira,您可以在其中执行所有这些操作。

It’s not.

它不是。

It is there to report bugs.

它是用来报告错误的。

I think the thing that made this particularly heinous was the name-dropping of Microsoft, the name-dropping that this is a visible product.

我认为,让这件事变得特别令人发指的是微软的名字被删除,这是一个可见产品的名字被删除。

If this was just a general bug report, I think that would have made it a lot better.

如果这只是一个一般性的错误报告,我认为这会好得多。

Lex Fridman1:18:51

Yeah, so they literally said, like, “This is a big deal because a lot of people are using it in Microsoft.” I wonder what happens psychologically.

是的,所以他们的字面意思是,“这是一件大事,因为很多人都在微软使用它。”我想知道心理上会发生什么。

1:18:51 So I think what happens in these companies, maybe you can correct me, is they…

所以我认为这些公司发生了什么,也许你可以纠正我,他们是……

You’re right.

你是对的。

They just think of FFmpeg as like a vendor that Microsoft surely is paying a huge amount of money to.

他们只是认为 FFmpeg 就像微软肯定向其支付巨额资金的供应商。

They kind of assume that in their interaction, and nobody anywhere on the stack is going like, “Wait a minute.

他们有点假设在他们的互动中,堆栈中的任何地方都没有人会说,“等一下。

Shouldn’t we be giving like millions of dollars to FFmpeg?”

我们难道不应该向 FFmpeg 捐赠数百万美元吗?”

Jean-Baptiste Kempf1:19:25

And this is a very big problem in large—like we’re talking about some companies, but it’s the same everywhere, right?

从整体上来说,这是一个非常大的问题——就像我们谈论的是一些公司,但到处都是一样的,对吧?

1:19:25 A lot of those companies.

这样的公司很多。

Like the…

就像……

when we talk to that person, right, he was just like a manager on one project in Microsoft Teams, right?

当我们与那个人交谈时,对吧,他就像 Microsoft Teams 中一个项目的经理,对吧?

He had, like, never really discussed with the open source community.

他从未真正与开源社区讨论过。

He had no idea, right?

他不知道,对吧?

It was like…

就像……

but the problem is that usually there is what we call OSPOs, right?

但问题是通常有我们所说的 OSPO,对吧?

Open Source Program Offices in those type of companies, and they are the ones who are supposed to discuss with open source vendors.

这类公司中的开源项目办公室,他们是应该与开源供应商讨论的人。

1:19:58 Or open source communities.

或者开源社区。

But like they often don’t explain that correctly internally, right?

但就像他们经常无法在内部正确解释这一点一样,对吧?

And here it’s just like we are not your supplier.

在这里,就像我们不是您的供应商一样。

If you want me to be your supplier, I’m very happy, right?

如果你想让我成为你的供应商,我很高兴,对吧?

I will send you a contract and SLAs.

我将向您发送合同和 SLA。

Like I created five companies who are doing that around open source projects, so that’s okay.

就像我创建了五家围绕开源项目做这件事的公司一样,所以没关系。

Lex Fridman1:20:19

We should say that some, some of the spicy tweets that Kieran, you’re behind, and some of the debacle produced results.

我们应该说,基兰,你背后的一些辛辣的推文,以及一些崩溃产生了结果。

Jean-Baptiste Kempf1:20:28

Yes.

是的。

Lex Fridman1:20:28

Positive results.

积极的结果。

Kieran Kunhya1:20:29

Donations have increased substantially.

捐款大幅增加。

1:20:29 They’re still not enough to cover even a single full-time developer, but on both a, you know, awareness level and a technical level, there’s substantially more technical awareness and sort of awareness of the importance of FFmpeg as a result of X and what’s happened.

它们仍然不足以覆盖甚至一个全职开发人员,但在意识水平和技术水平上,由于 X 和所发生的事情,人们对 FFmpeg 的重要性有了更多的技术意识和认识。

I can say, you know, it solved its purpose.

我可以说,你知道,它解决了它的目的。

People realize the level of importance FFmpeg has.

人们意识到 FFmpeg 的重要性。

Jean-Baptiste Kempf1:20:55

And on VideoLAN it’s the same, right?

在 VideoLAN 上也是一样的,对吧?

1:20:55 Like for example, a, a very simple example.

比如a,一个非常简单的例子。

For more than a year, we couldn’t update VLC on Android because of a bug on the Play Store, on Android Play Store, right?

一年多来,我们无法在 Android 上更新 VLC,因为 Android Play 商店上的 Play 商店存在错误,对吗?

The only way we got someone to answer was to put a very spicy, as you say, tweet saying that we are going to stop distributing VLC for Android, right?

我们让某人回答的唯一方法就是发布一条非常激烈的推文,正如你所说,我们将停止分发适用于 Android 的 VLC,对吧?

And we have around 100 million people using that.

我们有大约 1 亿人在使用它。

And now then someone from Android actually came and discussed to us, right?

然后现在Android的人真的来和我们讨论了,对吧?

We had the same issue with, with Microsoft or, or like saying that we were going to stop distributing VLC on the Windows Store.

我们与 Microsoft 也遇到了同样的问题,或者说我们将停止在 Windows 应用商店上分发 VLC。

1:21:39 And unfortunately, we are so small that the only very strong power we have to solve those issues is blaming on social networks because it snowballs and now they listen to us.

不幸的是,我们是如此之小,以至于我们解决这些问题的唯一强大力量就是归咎于社交网络,因为它像滚雪球一样越滚越大,现在他们听我们的。

But so as large companies often have difficulty talking to us.

但大公司通常很难与我们交谈。

Like for example, VLC, right, is probably one of the top 10 software used on Windows.

例如,VLC(右)可能是 Windows 上使用的十大软件之一。

I am not part of Microsoft ISV programs, right?

我不属于 Microsoft ISV 计划,对吗?

I don’t have a point of contact at Microsoft, right?

我在 Microsoft 没有联系人,对吧?

While I’m sure any other software, Adobe, Spotify, has a point of contact.

虽然我确信任何其他软件,例如 Adob​​e、Spotify,都有一个联系点。

I don’t have that, right?

我没有那个,对吧?

So raising awareness works.

因此,提高认识是有效的。

It’s sometimes very spicy, lot of drama.

有时非常辛辣,充满戏剧性。

Well, X and Twitter are okay for that, but it’s efficient.

嗯,X 和 Twitter 可以做到这一点,但效率很高。

Lex Fridman1:22:30

So everybody listening to this should go follow FFmpeg on Twitter, on X, follow VideoLAN on Twitter, on X.

因此,每个收听此内容的人都应该在 Twitter 上关注 FFmpeg,在 X 上关注,在 Twitter 上关注 VideoLAN,在 X 上关注。

1:22:30 Go donate.

去捐款吧。

Donate-

捐赠-

Kieran Kunhya1:22:44

Thank you

谢谢你

Lex Fridman1:22:44

…

……

1:22:44 to FFmpeg.

到 FFmpeg。

Kieran Kunhya1:22:45

And thank you, Lex.

谢谢你,莱克斯。

1:22:45 Over the years, several years you’ve been a supporter of, you know, FFmpeg and VideoLAN on X.

多年来,您一直是 X 上 FFmpeg 和 VideoLAN 的支持者。

You know, giving us shout-outs, appreciating, you know, what we do.

你知道,给我们点赞,欣赏我们所做的事情。

Lex Fridman1:22:55

FFmpeg for life.

终生使用 FFmpeg。

Jean-Baptiste Kempf1:22:57

And for example, like Tim Sweeney, Carmack, and a few others, like very high-level people have raised also the awareness on, on our X accounts, and that helped a lot also.

例如,蒂姆·斯威尼 (Tim Sweeney)、卡马克 (Carmack) 和其他一些非常高层的人也提高了我们 X 帐户的意识,这也有很大帮助。

Kieran Kunhya1:23:09

Karpathy as well.

卡帕蒂也一样。

Jean-Baptiste Kempf1:23:09

Karpathy, yes.

卡帕西,是的。

Kieran Kunhya1:23:10

Karpathy as well, yeah.

卡帕西也一样,是的。

Lex Fridman1:23:12

Yeah.

是的。

1:23:12 I mean, also, you know, outside of the fact that so many people use it, it’s so impactful on the world, it’s also a great representation of a great open source project.

我的意思是,你也知道,除了这么多人使用它、它对世界产生如此大的影响之外,它也是一个伟大的开源项目的一个很好的代表。

Like the value of assembly and C and making sure that like you take programming seriously for real world systems.

喜欢汇编和 C 的价值,并确保您认真对待现实世界系统的编程。

Kieran Kunhya1:23:30

It’s not just that.

不仅如此。

1:23:30 We’ll talk about assembly later I’m sure, ’cause that’s its whole topic in itself, but it’s also celebrating people like Andreas Rheinhardt who do maintenance.

我确信我们稍后会讨论装配,因为这本身就是整个主题,但它也庆祝像安德烈亚斯·莱因哈特这样从事维护工作的人。

It is, I believe unpaid, as a volunteer.

我相信,作为一名志愿者,这是无偿的。

He’s doing massive refactorings.

他正在进行大规模的重构。

Andreas Rheinhardt and Anton Khirnov rewriting ffmpeg.c with threading.

Andreas Rheinhardt 和 Anton Khirnov 使用线程重写 ffmpeg.c。

Celebrating those guys, celebrating the untold labor that’s gone into this that actually doesn’t change anything from the user standpoint.

庆祝这些人,庆祝他们付出的无数努力,但从用户的角度来看,这些工作实际上并没有改变任何东西。

The files are exactly the same, but wow, the airplane has been rebuilt whilst it’s in the air.

这些文件完全相同,但是哇,飞机在空中时已经被重建了。

Lex Fridman1:24:03

Christian Garcia said, “As a teenager running this account,” referring to the FFmpeg a- …

Christian Garcia 说,“作为一个管理这个账户的青少年”,他指的是 FFmpeg a-……

1:24:03 account, and you responded, “Teenagers have written more assembly in FFmpeg than Google engineers.” But also just pointing out that there’s a lot of incredible contributors who are teenagers.

帐户,你回答说:“青少年用 FFmpeg 编写的汇编比 Google 工程师还多。”但我也只是指出,有很多令人难以置信的贡献者都是青少年。

Kieran Kunhya1:24:19

Like JB said, we don’t care who you are, where you’re from, what you do.

就像 JB 所说,我们不在乎你是谁、你来自哪里、你做什么。

1:24:19 Teenagers have written thousands of lines of assembly over the years.

多年来,青少年已经编写了数千行汇编代码。

Give a shout-out back in the days to Daniel Kang.

向丹尼尔·康致敬。

So also highlighting the work of people like Ruikai Peng.

所以还要强调像彭瑞凯这样的人的工作。

This is a 16-year-old, some of his first contributions to FFmpeg, actually doing and putting some of these quote-unquote security researchers to shame by actually finding issues and fixing them and being 16.

这是一个 16 岁的孩子,他对 FFmpeg 的第一个贡献,实际上是在 16 岁的时候发现问题并修复它们,并让这些安全研究人员感到羞愧。

There’s no barriers.

没有障碍。

There’s no barriers to you have to study at college under this person and understand these.

你在这个人的指导下上大学并理解这些是没有任何障碍的。

You can learn C, and let’s be honest, it’s from the K&R book.

你可以学习 C,老实说,它来自 K&R 书。

Learn C.

学习C。

You can learn assembly.

可以学习组装。

We’ll talk about that maybe a bit later.

我们稍后会讨论这个问题。

You can contribute to world-class technologies.

您可以为世界一流的技术做出贡献。

Jean-Baptiste Kempf1:25:06

In VLC one of the oldest contributors called Felix, he’s the one doing everything on Mac and iOS.

在 VLC 中,最古老的贡献者之一叫 Felix,他负责 Mac 和 iOS 上的所有工作。

1:25:06 He’s starting working on VLC.

他开始研究 VLC。

He was 16.

他16岁。

We had a guy called Edward Wang, who used to be a Google Summer of Code student who stayed for three years around VideoLAN.

我们有一个叫 Edward Wang 的人,他曾经是 Google Summer of Code 的学生,在 VideoLAN 工作了三年。

He was 14, right?

他当时14岁,对吧?

And part of Google Summer of Code and Google Code-in, which were programs where basically we have students or high schoolers, we wrote a ton of assembly for x264 and for VLC and for FFmpeg, right?

Google Summer of Code 和 Google Code-in 的一部分,这些项目基本上都有学生或高中生,我们为 x264、VLC 和 FFmpeg 编写了大量的程序集,对吧?

So everyone can contribute.

所以每个人都可以做出贡献。

Kieran Kunhya1:25:41

And he also did a good job because he didn’t play the alarmist CVE heist, create a CVE (which is like a public exposure of security), and do these big scary red 7.5 high priority.

而且他也做得很好,因为他没有玩危言耸听的CVE抢劫,创建CVE(这就像公开曝光安全),并做这些大吓人的红色7.5高优先级。

1:25:41 He just fixed an issue in Git after three days and just fixed it.

三天后他刚刚修复了 Git 中的一个问题,并且刚刚修复了它。

He didn’t need to go and play a big security drama about it.

他不需要去演一场关于这件事的大型安全剧。

And I think I posted, you know, “the kids are all right.” Whereas there is a portion—I’m not saying all security people do this, but there is a portion of the security community, as Alex said, that likes to hype themselves up by creating drama.

我想我发布了,你知道,“孩子们都很好。”虽然有一部分人——我并不是说所有安全人员都这样做,但正如 Alex 所说,安全社区中有一部分人喜欢通过制造戏剧性来炒作自己。

They would have happily raised, “This is a high priority CVE 8.0” or whatever on an issue that actually was in Git.

他们会很高兴地提出“这是一个高优先级的 CVE 8.0”或者任何关于 Git 中实际存在的问题。

It wasn’t even in a release, it was in development, and three days later was fixed.

它甚至还没有发布,而是正在开发中,三天后就得到了修复。

Jean-Baptiste Kempf1:26:27

Well, I just want to put a little bit of love out there, even to the bigger community.

好吧,我只是想献出一点爱,甚至是对更大的社区。

1:26:27 Much love and respect to Google engineers.

非常热爱和尊重 Google 工程师。

Like you said, they’re some of the best software engineers in the world, and they do contribute a lot— …

就像你说的,他们是世界上最好的软件工程师,他们确实做出了很多贡献……

even on the security front.

即使在安全方面也是如此。

And also, you know, I’m a big fan of Theo.

而且,你知道,我是西奥的忠实粉丝。

Much love to Theo.

非常爱西奥。

He was part of this debacle and drama a little bit.

他是这场灾难和戏剧性事件的一部分。

I think when you just zoom out on the grand arc of human history, the drama contributed positively to everybody involved.

我认为,当你缩小人类历史的宏大弧线时,这部戏剧对每个参与者都做出了积极的贡献。

Donations went up.

捐款增加了。

It brought more attention to the topic, allowed everybody to bicker in a way that ultimately got them to figure out what FFmpeg is all about.

它给这个话题带来了更多的关注,让每个人都可以争吵,最终让他们弄清楚 FFmpeg 到底是什么。

Kieran Kunhya1:27:11

So the way we looked at this is like it’s a rap battle at the end of the day, you know?

所以我们看待这件事的方式就像是一天结束时的说唱大战,你知道吗?

1:27:11 No, but it is, it is.

不,但确实如此,确实如此。

Jean-Baptiste Kempf1:27:17

It is.

这是。

1:27:17 It is.

这是。

Kieran Kunhya1:27:17

We say stuff, we say stuff- …

我们说事,我们说事——……

but we can, we can leave it on.

但我们可以,我们可以保留它。

X is a perfect place for, you know, international rap battle.

X 是国际说唱大战的完美场所。

You say stuff.

你说些东西。

I say stuff about your mama, but it doesn’t mean, you know, I have an actual personal issue with her.

我说了一些关于你妈妈的事情,但这并不意味着,你知道,我和她有真正的个人问题。

And that’s what it looks like.

这就是它看起来的样子。

The Theo situation, you know, JB can maybe expand, went a little bit too far and there was a little…

Theo 的情况,你知道,JB 也许可以扩大,有点太过分了,有一点……

But, you know, it’s just a bit of fun.

但是,你知道,这只是一点乐趣。

It’s just a bit of rap battle.

这只是一场说唱之战。

It’s a bit—it’s WWE.

有点——这是WWE。

You know, everyone’s having a bit of fun on X.

你知道,每个人都在 X 上玩得很开心。

It doesn’t need to be taken seriously.

不需要认真对待。

You know, the teenagers thing, you know, that…

你知道,青少年的事情,你知道,那......

So that guy was a Google employee saying, “Hey, you know, there are other ways to run an open source business.” You know, and there’s like, oh, man, just have a bit of fun, you know?

那个人是一名谷歌员工,他说:“嘿,你知道,还有其他方式来运营开源业务。”你知道,就像,哦,伙计,享受一点乐趣,你知道吗?

1:27:54 That’s what the point of this account is.

这就是这个帐户的意义所在。

And furthermore, if you can teach people about the ways of open source projects, assembly, et cetera, by doing that, I think there’s a lot to be offered here.

此外,如果你可以通过这样做来教人们关于开源项目、汇编等的方式,我认为这里可以提供很多东西。

It’s not dunking on people for dunking’s sake.

这不是为了扣篮而对人扣篮。

It’s showing actually the story that I think X learnt is these are not big corporate open source projects.

它实际上展示了我认为 X 了解到的故事,这些不是大型企业开源项目。

This is not Kubernetes where there’s, you know, hundreds, maybe thousands of people- …

这不是 Kubernetes,那里有数百人,甚至数千人——……

paid to develop this stuff.

付费开发这个东西。

These are just people in their basements in their spare time, and if you can address that topic in a fun and entertaining way- …

这些只是人们在业余时间呆在地下室里,如果你能以一种有趣和有趣的方式讨论这个话题——……

I think that that’s the good thing and that’s, that’s the value of X and then the reach we have.

我认为这是一件好事,这就是 X 的价值以及我们所拥有的影响力。

Jean-Baptiste Kempf1:28:28

And to be honest, right, like even at Google, Google is one entity, but so many different people, right?

说实话,就像在谷歌一样,谷歌是一个实体,但有很多不同的人,对吧?

1:28:28 And there is a ton of Google engineers we work with all the time, and even like Google from YouTube to Chrome to Chrome Media to the rest of Google, those are very different type of entities.

我们一直与大量的 Google 工程师合作,甚至像 Google 从 YouTube 到 Chrome、Chrome Media 到 Google 的其他部门一样,这些都是非常不同类型的实体。

But what we do is efficient.

但我们所做的事情是有效率的。

And, for example for Theo, right?

比如说西奥,对吗?

It went a bit too far.

有点太过分了。

I had him…

我有他……

Like I calmed everyone down.

就像我让大家平静下来一样。

I had him on the phone.

我给他打电话了。

We say, “Okay, like this goes too far,” and so on.

我们说,“好吧,好像这太过分了”,等等。

But in the end yeah, it’s a rap battle, but it’s positive for the project.

但最终,是的,这是一场说唱之战,但这对这个项目来说是积极的。

It—like the awareness we have on open source and, and I mean true open source from communities right now—is increased dramatically in the last two years, and this is useful.

它——就像我们对开源的认识一样,我的意思是现在来自社区的真正的开源——在过去两年中急剧增加,这是有用的。

Kieran Kunhya1:29:19

What do you think motivates all the incredible contributors that we’ve been talking about?

您认为是什么激励了我们一直在谈论的所有令人难以置信的贡献者?

1:29:19 Like, what’s the engine?

比如,发动机是什么?

It’s so interesting to see.

看起来很有趣。

Jean-Baptiste Kempf1:29:26

So-

所以——

Kieran Kunhya1:29:26

Like you said, they’re sitting in the basement.

就像你说的,他们坐在地下室。

1:29:26 What’s the driver?

司机是什么?

What’s the engine there?

那里的引擎是什么?

Jean-Baptiste Kempf1:29:29

There are many drivers, but weirdly the main one is that what we do in multimedia plays videos, and video is cool, right?

驱动程序有很多,但奇怪的是最主要的一个是我们在多媒体中播放视频,视频很酷,对吧?

1:29:29 And, for example, we have so many people in the community who arrive because they loved watching anime, right?

举例来说,我们社区中有很多人因为喜欢看动漫而来到这里,对吧?

And this is like the advice when people ask me, “What should I work on in open source?

这就像当人们问我“我应该在开源领域做什么?

How do I start?” And my answer is always the same: work on something you love.

我该如何开始?我的答案总是一样的:做你喜欢的事情。

1:29:56 I am working on VLC because I love movies, right?

我从事 VLC 工作是因为我喜欢电影,对吧?

And I love watching the same movies over and over, even if my wife hates me when I do that, right?

我喜欢一遍又一遍地看同样的电影,即使我这样做时我的妻子讨厌我,对吧?

But because it’s interesting, right?

但因为它很有趣,对吧?

Because it’s a topic that you like, right?

因为这是你喜欢的话题,对吗?

The first, that’s the first thing where people come to usually to VLC and FFmpeg.

首先,这是人们通常首先想到的 VLC 和 FFmpeg。

The second thing is that technically we, because we search for excellence, this is the best school ever, right?

第二件事是,从技术上讲,我们因为追求卓越,所以这是有史以来最好的学校,对吧?

This is the best school ever of programming.

这是有史以来最好的编程学校。

If you’re good in C, in FFmpeg, if you know how to write assembly, I assure you you’re going to be one of the best programmers ever, even if you’re working on writing TypeScript, because this is the most amazing thing to do.

如果你擅长 C、FFmpeg,如果你知道如何编写汇编,我向你保证你将成为有史以来最好的程序员之一,即使你正在编写 TypeScript,因为这是最令人惊奇的事情。

1:30:38 And you will, like, have to get reviews by some of the most seasoned programmers ever who are going to look at every part of your code and tell you why it’s not great.

而且你必须得到一些最经验丰富的程序员的评论,他们会检查你代码的每个部分并告诉你为什么它不好。

It’s like we are the best teachers that you’ve ever had in programming, right?

就好像我们是您在编程方面遇到过的最好的老师,对吧?

Kieran Kunhya1:30:52

Andrew Kelley started Zig.

安德鲁·凯利 (Andrew Kelley) 创办了 Zig。

1:30:52 He was an FFmpeg developer and started Zig after his FFmpeg school.

他是一名 FFmpeg 开发人员,在 FFmpeg 学校毕业后创办了 Zig。

I mean, it, it’s the place to learn so many aspects of programming in the real world, in a thing used by billions of people.

我的意思是,它是在现实世界中学习编程的许多方面的地方,在一个被数十亿人使用的东西中。

You have nowhere to hide.

你无处可藏。

You have to be open and honest about your flaws and, and how you can learn and be better.

你必须对自己的缺点以及如何学习和变得更好保持开放和诚实。

Jean-Baptiste Kempf1:31:12

And what is also interesting in multimedia is that you have 16 milliseconds to display a frame.

多媒体中另一个有趣的地方是,您有 16 毫秒的时间来显示一帧。

1:31:12 It’s not like a game engine where you can basically slow down and wait a frame.

它不像游戏引擎,你基本上可以放慢速度并等待一帧。

So it’s, you need to be good, right?

所以,你需要表现得好,对吗?

There is no choice, else you don’t have your video.

别无选择,否则你就没有视频。

And because of how codecs, if you miss a frame, you’re going to destroy the look of the video, right?

由于编解码器的原因,如果您错过了一帧,就会破坏视频的外观,对吧?

So you need to be good.

所以你需要表现得好。

You need to be perfect to have the right thing.

你需要完美才能拥有正确的东西。

But also is that it’s not just pure programming in the mathematical sense, right?

但这不仅仅是数学意义上的纯粹编程,对吧?

A lot of people don’t understand, but in order to program correctly on the open source multimedia community, you need to understand how computers work.

很多人不明白,但是为了在开源多媒体社区上正确编程,您需要了解计算机是如何工作的。

And when you write assembly, you need to understand about CPU pipelining, right?

当你编写汇编时,你需要了解CPU流水线,对吧?

1:31:59 You need to understand how SIMD works, how the ALU works, right?

您需要了解 SIMD 的工作原理、ALU 的工作原理,对吧?

You need to understand how I/O works, right?

您需要了解 I/O 的工作原理,对吗?

And this is what I think that is missing to a lot of engineers and software engineers today, is understanding what we call computer architecture.

我认为当今许多工程师和软件工程师所缺少的就是理解我们所说的计算机体系结构。

And, like, seriously, some of the debates is like, should we use this assembly call or this one?

而且,说实话,有些辩论是这样的,我们应该使用这个集会电话还是这个?

And people say, “Well, no, it’s going to be like three cycles on this type of CPU and this one,” and has massive impact on the output, right?

人们会说,“嗯,不,在这种类型的 CPU 和这个 CPU 上,这将是三个周期”,并且对输出产生巨大影响,对吧?

Kieran Kunhya1:32:27

We should expand.

我们应该扩大。

1:32:27 FFmpeg is probably one of the biggest CPU users in the world.

FFmpeg 可能是世界上最大的 CPU 用户之一。

There’s proba- it’s probably running- …

有概率——它可能正在运行——……

as we speak easily 100 mil- order of magnitude 100 million, maybe even a billion CPUs as we speak.

正如我们所说,一亿数量级,一亿,甚至十亿个CPU。

So every instruction matters.

所以每一条指令都很重要。

There’s not…

没有……

The impact, at least in terms of CPU, is massive for everything that we do.

至少在 CPU 方面,这种影响对于我们所做的一切都是巨大的。

Jean-Baptiste Kempf1:32:51

So first you come because it’s an interesting subject, then you stay because it’s excellent, and in the end you’re very proud of it because it’s on the end of everyone.

所以,首先你来是因为它是一个有趣的主题,然后你留下来是因为它很棒,最后你会为此感到非常自豪,因为它是每个人的终点。

1:32:51 Like so many people like, “Oh, I’m working for whatever consulting company and I’m doing some portal to download invoices for your PG&E.” Wow, great.

就像很多人一样,“哦,我在任何一家咨询公司工作,我正在做一些门户网站来下载 PG&E 的发票。”哇,太棒了。

Like, so many jobs are like that.

就像,很多工作都是这样。

You’re not going to tell that to your grandma.

你不会把这件事告诉你的奶奶。

But if you go to see your grandma and say, “I do this so that you can play video on your laptop,” they understand.

但如果你去看你的奶奶并说:“我这样做是为了让你可以在笔记本电脑上播放视频,”他们会理解。

And this is very important, right?

这非常重要,对吧?

Because you’re working on VLC, FFmpeg, H.264.

因为您正在研究 VLC、FFmpeg、H.264。

It’s in the end of hundreds of millions of people and you have an impact.

这是数亿人的最终目标,而你会产生影响。

And so you can be proud of yourself.

这样您就可以为自己感到自豪。

1:33:35 And so I think that in addition to doing a great resume, all those things are why people contribute.

所以我认为除了写一份出色的简历之外,所有这些都是人们做出贡献的原因。

Kieran Kunhya1:33:42

Yeah, those are side effects.

是的,这些都是副作用。

1:33:42 My favorite quote on this topic is John Collison.

关于这个话题,我最喜欢的一句话是约翰·科里森(John Collison)。

He said, “The world is a museum of passion projects.” You know, everything out there is a passion project.

他说:“世界是一座充满激情项目的博物馆。”你知道,外面的一切都是一个充满激情的项目。

And open source multimedia and open source in general, you can just do that so much faster.

而开源多媒体和一般的开源,你可以更快地做到这一点。

There’s such a faster network effect, you know?

网络效应如此之快,你知道吗?

1:33:59 I can open a cafe and that can be my passion project, but I have to get building codes, I have to build a building, I have to find a location, I have to do all the, you know, all sorts of things.

我可以开一家咖啡馆,这可能是我的热情项目,但我必须获得建筑规范,我必须建造一座建筑物,我必须找到一个位置,我必须做所有的,你知道的,各种各样的事情。

Well, in the software world, that passion project can move quickly, it can be amplified by the network effect, and that amplification can be more than the sum of the parts.

嗯,在软件世界中,充满激情的项目可以快速发展,它可以通过网络效应放大,而且这种放大可以超过各个部分的总和。

You know, you can be, you can find people interested in extremely obscure things and have a network effect and make something that is truly amazing.

你知道,你可以,你可以找到对极其晦涩的事物感兴趣的人,并产生网络效应,并做出真正令人惊叹的东西。

Lex Fridman1:34:31

And on that topic of passion projects, Tim Sweeney actually said in a reply to a tweet that was complimenting JB.

关于激情项目这个话题,蒂姆·斯威尼实际上在回复一条赞扬 JB 的推文时说道。

1:34:31 He said, quote, “Many things in the world only happen because an awesome person decides to do it.

他说,“世界上许多事情的发生都是因为一个了不起的人决定去做。

This is the case with VLC.” And that speaks to something interesting to me, that it does seem that a small number of people, sometimes one person, can create something incredible in the software world.

VLC 就是这种情况。”这对我来说很有趣,似乎少数人,有时是一个人,可以在软件世界中创造出令人难以置信的东西。

Like you said this over and over and over.

就像你一遍又一遍地说的那样。

I think JavaScript is an incredible thing created by initially a single person.

我认为 JavaScript 是一个令人难以置信的东西,最初是由一个人创建的。

Some of the programming languages like Python and C and Java, like just one person has this vision, has this design, and brings it—sometimes over a weekend is the initial spark.

一些编程语言,如 Python、C 和 Java,就像只有一个人有这样的愿景、有这样的设计并带来它——有时一个周末就能产生最初的火花。

Jean-Baptiste Kempf1:35:18

Yes, Linus built Git in two weeks.

是的,Linus 在两周内构建了 Git。

1:35:18 Wow.

哇。

Lex Fridman1:35:23

It changed the world, Git.

它改变了世界,Git。

1:35:23 I mean, it really changed the world.

我的意思是,它确实改变了世界。

Kieran Kunhya1:35:25

Linus’ passion project.

Linus 的热情项目。

1:35:25 “Hey, I’m uploading this tarball to an FTP, like deal with it.”

“嘿,我正在将这个 tarball 上传到 FTP,就像处理它一样。”

Jean-Baptiste Kempf1:35:29

But for me, it’s not just in software, right?

但对我来说,这不仅仅是软件,对吧?

1:35:29 And I believe in individuals that are going to change the world, right?

我相信个人能够改变世界,对吗?

And it’s with a good, as you said, vision, right?

正如你所说,它具有良好的愿景,对吗?

I want to do that.

我想这么做。

It is useful, it will be useful.

它是有用的,它将是有用的。

And whether it’s going to like build trains or cars or rockets or something like, I believe people who believe in themselves and have a vision can have a huge impact for humanity.

无论是建造火车、汽车、火箭还是类似的东西,我相信相信自己并有远见的人可以对人类产生巨大的影响。

Lex Fridman1:35:56

Let’s actually zoom out before we zoom back in.

在放大之前,让我们实际缩小一下。

1:35:56 We’ll just keep going up and down the stack.

我们将继续在堆栈中上下移动。

So you know, we’ve been talking back and forth VLC and FFmpeg.

所以你知道,我们一直在讨论 VLC 和 FFmpeg。

Kieran, you said that FFmpeg and VideoLAN, VLC coexist, and there’s no central point of importance.

Kieran,你说FFmpeg和VideoLAN、VLC共存,并且没有重要的中心点。

It’s a kind of what you call the binary star system.

这是一种你们所说的双星系统。

They succeed because of each other.

他们因为彼此而成功。

Can you explain the difference, how they interact?

您能解释一下它们之间的区别以及它们如何相互作用吗?

What is the-

什么是-

Kieran Kunhya1:36:24

Sure

当然可以

Lex Fridman1:36:24

…

……

1:36:24 are they competitors?

他们是竞争对手吗?

Kieran Kunhya1:36:26

I don’t, I don’t think they’re competitors.

我不,我不认为他们是竞争对手。

1:36:26 I think, I think the simple answer is, the short answer before I go into detail is: VLC is to FFmpeg as Android is to Linux.

我想,我认为简单的答案是,在详细讨论之前,简短的答案是:VLC 之于 FFmpeg 就像 Android 之于 Linux 一样。

So they depend on each other, but they, they coexist because of each other.

所以他们是互相依赖的,但是他们,他们因为彼此而共存。

So they are a binary star system is the analogy I used.

所以它们是一个双星系统,这是我使用的类比。

Lex Fridman1:36:43

By the way, I feel horrible that I just recently learned that Alpha Centauri, the closest star system to us, is a triple star system.

顺便说一句,我最近才得知距离我们最近的恒星系统半人马座阿尔法星是一个三星系统,这让我感到很可怕。

Jean-Baptiste Kempf1:36:50

And, and when you start doing the physics, it’s a nightmare, right?

而且,当你开始研究物理时,这就是一场噩梦,对吗?

1:36:50 But like-

但就像——

Kieran Kunhya1:36:55

Hence the three-body problem.

于是就有了三体问题。

1:36:55 But anyway.

但无论如何。

So a lot of FFmpeg pipelines involve the x264 project, which is a VideoLAN project.

所以很多FFmpeg管道都涉及x264项目,这是一个VideoLAN项目。

I would put a finger in the air and say 80-plus percent of those pipelines are dependent on a VideoLAN project.

我可以举起一根手指说,这些管道中 80% 以上都依赖于 VideoLAN 项目。

VLC, obviously, as we’ve discussed, a VideoLAN project, uses FFmpeg, gives it reach, exposure to weird files historically, used some donation money to fund FFmpeg development, and we’ll talk a bit maybe about some of the reverse engineering later.

显然,正如我们所讨论的,VLC 是一个 VideoLAN 项目,使用 FFmpeg,使其具有影响力,历史上接触过奇怪的文件,使用了一些捐款来资助 FFmpeg 的开发,稍后我们可能会讨论一些逆向工程。

So it’s a binary star system.

所以它是一个双星系统。

They work and feed off each other.

他们互相工作、互相依赖。

Many of the developers are shared.

许多开发人员都是共享的。

There’s no central location.

没有中心位置。

It’s a virtuous cycle working together.

这是一个共同努力的良性循环。

Lex Fridman1:37:36

And we should mention that x264 is the encoder for H.264 video standard.

值得一提的是,x264 是 H.264 视频标准的编码器。

1:37:36 So H.264 is the standard.

所以H.264是标准。

X264-

X264-

Kieran Kunhya1:37:46

Is the open source implementation of the standard

是标准的开源实现

Lex Fridman1:37:49

…

……

1:37:49 that’s used by basically everybody- …

基本上每个人都在使用 - ...

for everything.

对于一切。

It’s, that is the main driver of this.

这是主要的驱动力。

When you think of an MP4 file that has H.264 codec in it-

当您想到其中包含 H.264 编解码器的 MP4 文件时 -

Kieran Kunhya1:37:59

If it came from a software environment, like a data center or somewhere, the chances are it was created with x264.

如果它来自软件环境,例如数据中心或其他地方,那么它很可能是使用 x264 创建的。

Lex Fridman1:38:06

And that’s under the flag of VideoLAN.

这就是 VideoLAN 的旗下。

Kieran Kunhya1:38:09

That’s a VideoLAN project.

这是一个 VideoLAN 项目。

1:38:09 So in the VideoLAN graphic, it sits in the VideoLAN world.

因此,在 VideoLAN 图形中,它位于 VideoLAN 世界中。

Lex Fridman1:38:14

And VideoLAN has a— says a bunch of stuff in it.

VideoLAN 有一个——里面说了很多东西。

1:38:14 Go to the VideoLAN website, there’s a bunch of icons.

去VideoLAN网站,有一堆图标。

Jean-Baptiste Kempf1:38:21

Like if you look, there are so many libraries, right?

如果你看的话,有很多图书馆,对吗?

Lex Fridman1:38:24

libdvdcss- …

libdvdcss-...

1:38:24 libdvdnav, libdvbpsi, libVLC of course, vlc-unity, libbluray- Blu-ray.

libdvdnav、libdvbpsi、libVLC 当然还有 vlc-unity、libbluray-蓝光。

Jean-Baptiste Kempf1:38:36

Blu-ray.

蓝光。

Lex Fridman1:38:37

Yeah, there’s many more.

是的,还有更多。

Jean-Baptiste Kempf1:38:39

And there is so many more, right?

还有更多,对吧?

1:38:39 Lately, the dav1d project that we might talk about is the last project from VideoLAN.

最近,我们可能谈论的 dav1d 项目是 VideoLAN 的最后一个项目。

It’s everywhere, right?

它无处不在,对吧?

And we do, we have a libspatialaudio lately that we announced.

我们确实这样做了,我们最近宣布了一个 libspatialaudio。

We have a-

我们有一个-

Kieran Kunhya1:38:51

checkasm.

检查。

Jean-Baptiste Kempf1:38:52

checkasm-

checkasm-

Kieran Kunhya1:38:53

We’ll talk about that later.

我们稍后再讨论。

Jean-Baptiste Kempf1:38:53

…

……

1:38:53 which is like an insane project- …

这就像一个疯狂的项目——……

but amazing.

但令人惊奇。

So, and x264 is one of those VideoLAN projects.

所以,x264 就是这些 VideoLAN 项目之一。

And my opinion, for example, is that x264 was, is the most amazing encoder ever designed, and this helped the adoption of FFmpeg.

例如,我的观点是,x264 是有史以来设计的最令人惊叹的编码器,这有助于 FFmpeg 的采用。

A lot of people and large companies went through FFmpeg because they wanted to use x264, and x264 increased the popularity of FFmpeg.

很多人和大公司都通过FFmpeg因为他们想使用x264,而x264增加了FFmpeg的受欢迎程度。

But also VLC had its popularity because it played so many files that were done by FFmpeg, right?

但 VLC 也很受欢迎,因为它播放了很多由 FFmpeg 完成的文件,对吗?

So it’s many projects that are intertwined and work together.

因此,许多项目相互交织、协同工作。

Kieran Kunhya1:39:32

Yeah.

是的。

1:39:32 Unfortunately, there’s a thing on X where VLC is mentioned and there’s people, “A quick reminder that it’s FFmpeg inside doing the actual work.” And that’s like I said, it’s not, that’s not the case.

不幸的是,X 上有一件事提到了 VLC,有人“快速提醒一下,实际工作是由 FFmpeg 内部完成的”。就像我说的,不是的,不是这样的。

We work together.

我们一起工作。

Jean-Baptiste Kempf1:39:46

And to give you an idea, right?

给你一个想法,对吗?

1:39:46 When I compiled VLC for Windows, I compiled around 16 million lines of code, right?

当我编译 Windows 版 VLC 时,我编译了大约 1600 万行代码,对吧?

One million of those are inside the VLC repository, and FFmpeg in total is probably around two, right?

其中 100 万个位于 VLC 存储库中,而 FFmpeg 总数可能约为 2 个,对吧?

But so it means that so many dependencies are outside.

但这意味着有很多依赖项是外部的。

And if you also look at FFmpeg per se, FFmpeg also is integrating third-party libraries like x264, but libopus and so many others, right?

如果您也看看 FFmpeg 本身,FFmpeg 还集成了 x264、libopus 等第三方库,对吧?

So we all depend on each other.

所以我们都互相依赖。

Lex Fridman1:40:15

Yeah, that’s why I was hoping to do this episode as we are doing that just kind of joins FFmpeg and VLC- …

是的,这就是为什么我希望制作这一集,因为我们正在做的就是将 FFmpeg 和 VLC 结合起来——……

1:40:15 because it’s really two of the same, like you said, binary star system and we’re all just orbiting it.

因为它实际上是两个相同的双星系统,就像你说的,我们都只是围绕它运行。

Can we give a shout-out to some of the people along the way?

我们可以向一路上的一些人大声喊叫吗?

We didn’t really quite talk about the history of FFmpeg, so maybe can you tell me about Fabrice?

我们并没有真正谈论 FFmpeg 的历史,所以也许你能告诉我一些关于 Fabrice 的事情吗?

Can you tell me about Michael Niedermayer?

你能告诉我关于迈克尔·尼德迈尔的事吗?

Can you tell me about some of the key figures here?

您能告诉我这里的一些关键人物吗?

Kieran Kunhya1:40:46

Let’s just talk about the eras of FFmpeg, because there’s key eras and key people that made this possible.

我们只讨论 FFmpeg 的时代,因为有一些关键时代和关键人物使这成为可能。

1:40:46 Fabrice Bellard, as you mentioned, creating the concept, and then probably in the 2000 era— I would call the era, Eras Tour of FFmpeg— is the 2000 era was Michael Niedermayer.

正如您提到的,Fabrice Bellard 创造了这个概念,然后可能在 2000 年时代(我称之为 FFmpeg 时代),2000 年时代是 Michael Niedermayer。

So key things he got done was exhaustive support for DivX and Xvid at the time, and all sorts of weird variants of what’s known as MPEG-4 Part 2.

因此,他所做的关键事情是对当时的 DivX 和 Xvid 以及各种奇怪的 MPEG-4 Part 2 变体的全面支持。

So this predates the MPEG-4 Part 10 that we’re used to.

所以这早于我们习惯的 MPEG-4 Part 10。

So this was 2000 era video codecs where there were flavor after flavor of weird, weird decoders.

这是 2000 年代的视频编解码器,其中有各种奇怪的解码器。

1:41:27 At the time in the 2000s, you needed a new player to play every different type of file format.

在 2000 年代,您需要一个新的播放器来播放各种不同类型的文件格式。

So there was Windows Media Player to play Windows Media formats.

于是就有了 Windows Media Player 来播放 Windows Media 格式。

There was RealPlayer to play RealMedia formats.

有 RealPlayer 可以播放 RealMedia 格式。

And those were the other key thing in FFmpeg at the time were native decoders for those.

当时 FFmpeg 中的另一个关键是它们的本机解码器。

I actually do remember being a teenager, I must have been, figuring out there was this one player that could play, could decode these files without having separate bloated players.

我确实记得当我还是个青少年的时候,我一定是,发现有一个可以玩的玩家,可以解码这些文件,而无需单独的臃肿的玩家。

Because at the time when you downloaded RealPlayer, there was a ton of other stuff in there, a ton of ads, a ton of other things, and just having a simple library that was fast led to that.

因为当您下载 RealPlayer 时,里面有大量其他内容、大量广告、大量其他内容,而只需一个简单的库就可以快速实现这些内容。

1:42:03 And then I think 2008 was a big change because that’s when H.264 got its maturity and I think something hopefully we’ll talk about a bit more.

然后我认为 2008 年是一个巨大的变化,因为那时 H.264 变得成熟,我认为希望我们能多讨论一些事情。

This was the beginning of high definition video.

这是高清视频的开始。

So H.264 was the key decoder of that.

所以 H.264 是其中的关键解码器。

So I’d call that the late 2000s and 2010s, and that’s when the big reverse engineers came along and really did astonishing work.

所以我称之为 2000 年代末和 2010 年代,那时大型逆向工程师出现并确实做了令人惊叹的工作。

The beginning was a single player that could play Xvid, DivX, Windows Media, and RealPlayer was already a massive achievement in itself without codec packs, without weird stuff you had to download that had weird ads and weird spyware.

一开始是一个可以播放 Xvid、DivX、Windows Media 的单一播放器,而 RealPlayer 本身就已经是一项巨大的成就,无需编解码器包,无需下载带有奇怪广告和奇怪间谍软件的奇怪内容。

Jean-Baptiste Kempf1:42:43

VLC 1.0 was out in those times, 2000, 2009, 2010.

VLC 1.0 就在 2000 年、2009 年、2010 年问世。

1:42:43 And this is like where it exploded.

这就像它爆炸的地方。

Lex Fridman1:42:53

Yeah, without codec packs, it just works- …

是的,没有编解码器包,它就可以工作 - ...

1:42:53 across all these different-

跨越所有这些不同的——

Jean-Baptiste Kempf1:42:57

It, de facto, it’s just like all the codec packs are FFmpeg inside VLC, plus we have other modules for all the types of codecs.

事实上,就像所有编解码器包都是 VLC 内的 FFmpeg 一样,而且我们还有适用于所有类型编解码器的其他模块。

Kieran Kunhya1:43:03

But back at the time that wasn’t…

但当时情况并非如此……

1:43:03 is there were weird, in the 2000s, there were weird codec packs with DLLs coming from this place, DLLs coming from that-

是不是很奇怪,在 2000 年代,有一些奇怪的编解码器包,其中的 DLL 来自这个地方,DLL 来自那个地方 -

Jean-Baptiste Kempf1:43:11

With a lot of spyware.

有很多间谍软件。

Kieran Kunhya1:43:12

…

……

1:43:12 with spyware, with you know what.

间谍软件,你知道吗。

It wasn’t reliable, you didn’t know, and having a single player that was open source or single playback module/player that could do this that was open source.

它不可靠,你不知道,并且有一个开源播放器或单个播放模块/播放器可以做到这一点,这是开源的。

But I think the thing to emphasize is this task in the 2000s that Michael did was Sisyphean.

但我认为要强调的是,迈克尔在 2000 年代所做的这项任务是西西弗斯式的。

It was really, the number of edge cases are poor beyond comprehension in terms of you could have a Chinese CCTV system that did one weird variant of MPEG-4 Part 2, what’s known as MPEG-4 ASP, and that was a weird variant, and you had to fix that without breaking everybody else- …

确实,边缘情况的数量少得令人难以理解,因为你可以让一个中国闭路电视系统执行 MPEG-4 第 2 部分的一个奇怪变体,即所谓的 MPEG-4 ASP,这是一个奇怪的变体,你必须在不破坏其他人的情况下修复它 - ...

times a million.

一百万倍。

Lex Fridman1:43:45

So that’s where a lot of the reverse engineering was happening.

这就是大量逆向工程发生的地方。

Kieran Kunhya1:43:49

It started in the 2000s with the Windows Media stuff because that was- …

它始于 2000 年代的 Windows Media 东西,因为那是——……

1:43:49 proprietary.

专有的。

It started with the RealMedia, so with Benjamin Larsson.

它始于 RealMedia,也始于本杰明·拉尔森 (Benjamin Larsson)。

Jean-Baptiste Kempf1:43:56

Kostya Shishkov.

克斯特亚·希什科夫。

Kieran Kunhya1:43:57

Kostya Shishkov, that era.

克斯特亚·希什科夫,那个时代。

1:43:57 Those were the key, that was the key groundwork.

这些是关键,这是关键的基础。

And then in the 2010s was kind of the Paul Mahol, Kostya era building, doing some of the most difficult codecs.

然后在 2010 年代是 Paul Mahol、Kostya 时代的建筑,做了一些最困难的编解码器。

JB maybe can talk about GoToMeeting 4 and GoToMeeting 5, and-

JB 也许可以谈谈 GoToMeeting 4 和 GoToMeeting 5,并且-

Lex Fridman1:44:13

What’s the GoToMeeting?

什么是 GoToMeeting?

Jean-Baptiste Kempf1:44:15

So, like, let’s talk about this amazing Ukrainian guy called Kostya, who was at that time living in Germany, and who was in love with Sweden, right?

那么,让我们来谈谈这个名叫科斯特亚的乌克兰人,他当时住在德国,并且爱上了瑞典,对吧?

1:44:15 He— And the guy was the most…

他——而且这个家伙是最……

He’s like a lot of the people in the community are very clever.

他就像社区里的很多人都非常聪明。

He’s one of those who are, like, borderline geniuses, right?

他是那些边缘天才之一,对吧?

He was able to reverse engineer extremely complex codecs and he does that, and we do a bit of engineering with Kieran, but clearly not at this level.

他能够对极其复杂的编解码器进行逆向工程,他做到了,我们和 Kieran 一起做了一些工程,但显然还没有达到这个水平。

Kieran Kunhya1:44:50

No, no, yeah.

不,不,是的。

Jean-Baptiste Kempf1:44:51

He reverse engineered binary blobs, which are 20 megabytes?

他对二进制 blob 进行了逆向工程,即 20 MB?

Kieran Kunhya1:44:56

Yeah, so just for reference, one megabyte binary blob to reverse engineer is probably order of magnitude a month of work, and this guy is doing 20, 30 megabyte blobs.

是的,仅供参考,对 1 MB 二进制 blob 进行逆向工程可能需要一个月的工作量,而这个人正在处理 20、30 MB blob。

1:44:56 Maybe we’ll talk about that in a minute, about the subtleties of how you do that.

也许我们稍后会讨论这个问题,讨论如何做到这一点的微妙之处。

But this guy is doing it for very difficult and very obscure codecs.

但这家伙正在为非常困难和非常晦涩的编解码器做这件事。

Jean-Baptiste Kempf1:45:13

And did that for fun, right?

这样做是为了好玩,对吧?

1:45:13 And so GoToMeeting was a big problem with VLC because that was like the number one feature request for a long time, so I put a bounty.

因此,GoToMeeting 对 VLC 来说是一个大问题,因为这在很长一段时间内都是头号功能请求,所以我提供了赏金。

And the guy at some point said, “Okay, JB, I’m going to do it.” And in a matter of two months, and then he explained how he did it.

那个人在某个时候说:“好吧,JB,我会这么做的。”两个月后,他解释了他是如何做到的。

He was just like, “Oh, I looked at the code, like this looked like a DCT that I used to see on WMV and so on.” He did that, and the funniest part is that the code he’s written is a ton of jokes.

他只是说,“哦,我看了代码,这看起来就像我以前在 WMV 等上看到的 DCT。”他做到了,最有趣的是他写的代码充满了笑话。

And there is a ton of JB, right, my name, and Kempf and Kempf and Kostya jokes inside the code.

代码中有大量 JB,对,我的名字,以及 Kempf 和 Kempf 和 Kostya 笑话。

The code is beautiful, right?

代码很漂亮,对吧?

Lex Fridman1:45:57

So one of the things I wanna comment is I’ve gotten a chance to speak to some of the developers, some of the assembly language level people, and they all always make everything sound like it’s kinda easy.

因此,我想评论的一件事是,我有机会与一些开发人员、一些汇编语言级别的人员交谈,他们总是让一切听起来都很简单。

1:45:57 There’s a kind of humility because, maybe just the level of what’s required to do this stuff is so high that everything else seems easy, I guess is the lesson to take away from that.

有一种谦虚的感觉,因为也许做这件事所需的水平是如此之高,以至于其他一切看起来都很容易,我想这就是从中吸取的教训。

Jean-Baptiste Kempf1:46:23

So in the community, like some of the most impressive people are the ones doing reverse engineering- …

因此,在社区中,一些最令人印象深刻的人就是那些进行逆向工程的人——……

1:46:23 and the other ones doing the assembly folds, right?

其他负责组装的人会折叠,对吗?

And both of those type of people are amazing.

这两类人都很棒。

x264, for example, became amazing because of a guy called Loren Merritt- …

例如,x264 之所以变得令人惊奇,是因为一个名叫 Loren Merritt 的人——……

who is, was from University of Washington, I think.

我想他来自华盛顿大学。

Kieran Kunhya1:46:44

At the time, yeah.

当时,是啊。

Jean-Baptiste Kempf1:46:45

And who was, like, who made everything great and fast doing a ton of assembly.

谁通过大量的组装使一切都变得伟大而快速。

1:46:45 So this is like the golden era, I guess, where so many things got done.

我想这就像黄金时代,做了很多事情。

Kieran Kunhya1:46:57

So, yeah, if you look at Kostya, for example, he looked at the world as a binary specification.

所以,是的,例如,如果你看看科斯蒂亚,他将世界视为二进制规范。

1:46:57 He didn’t need documentation or anything.

他不需要文件或任何东西。

It’s, “I have a binary and I can figure all of this out.” And he regularly used the phrase “binary specification.” Ah, you know, it’s not a problem.

它是,“我有一个二进制文件,我可以弄清楚所有这些。”他经常使用“二进制规范”这个词。啊,你知道,这不是问题。

And he would go away, and he would come back, and he would do interesting stuff.

他会离开,然后回来,他会做一些有趣的事情。

Lex Fridman1:47:15

Can you actually speak to the details or add color and texture to what it takes to reverse engineer a blob?

您实际上可以谈论细节或添加颜色和纹理来逆向工程斑点吗?

Kieran Kunhya1:47:22

Yeah.

是的。

1:47:22 So let’s look at GoToMeeting, for example, is a good one because I record a meeting on GoToMeeting, for example.

例如,让我们看看 GoToMeeting 是一个很好的选择,因为我在 GoToMeeting 上记录了一次会议。

How do I play it back without needing this GoToMeeting player?

如何在不需要 GoToMeeting 播放器的情况下播放它?

There may not even be a player.

甚至可能没有玩家。

I may need to send a recording of a meeting to someone that doesn’t have a player or whatever.

我可能需要将会议录音发送给没有播放器或其他设备的人。

So first of all, there’s a ton of other stuff there.

首先,还有很多其他东西。

There’s an actual video conferencing client.

有一个实际的视频会议客户端。

You need to go and find—it may be easy, it may not be easy to find—the actual module doing the decompression.

您需要去查找(可能很容易,也可能不容易找到)执行解压缩的实际模块。

You need a way to actually dump the YUV data from the module.

您需要一种方法来实际从模块转储 YUV 数据。

So often it involves opening in a disassembler, trying to guess where the hooks are to incorporate that module and run that module natively to decode a sample file.

通常,它涉及在反汇编程序中打开,尝试猜测挂钩在哪里合并该模块并本机运行该模块以解码示例文件。

1:48:06 So figure out where this module is doing the decoding process and find a way to hook in and output the raw YUV data, ’cause you will need that- …

因此,找出该模块在哪里进行解码过程,并找到一种方法来连接并输出原始 YUV 数据,因为您将需要 - ...

as a point of comparison for when you actually do the reverse engineering, ’cause you’ll need to be bit exact or in some cases close to bit exact.

作为实际进行逆向工程时的比较点,因为您需要精确一点,或者在某些情况下接近精确一点。

And then you open up your disassembler, use a lot of intuition to go and figure out, you know, where the DCT is, where’s entropy coding.

然后你打开你的反汇编器,用大量的直觉去弄清楚,你知道,DCT 在哪里,熵编码在哪里。

There is a kind of, not a rule book, but there’s always a pattern of some sort.

有一种,不是规则手册,但总有某种模式。

For example, GoToMeeting, you know it will be a lot of screen codec tools.

例如GoToMeeting,你知道它会是很多屏幕编解码工具。

There’s also different variants, so often I think there’s, what, GoToMeeting 4, 5-

还有不同的变体,所以我经常认为有,什么,GoToMeeting 4, 5-

Jean-Baptiste Kempf1:48:46

Well, 2 or 3, 4, I think.

嗯,我想是 2 或 3、4。

Kieran Kunhya1:48:48

2, 3, 4.

2、3、4。

Lex Fridman1:48:49

So as you mentioned here, going to Perplexity, GoToMeeting uses its own proprietary codec for older s- recorded sessions historically stored in WMV files that require a special decoder to play properly on Windows.

正如您在 Perplexity 中提到的,GoToMeeting 使用自己的专有编解码器来处理历史上存储在 WMV 文件中的旧 s 录制会话,这些文件需要特殊的解码器才能在 Windows 上正常播放。

1:48:49 Without this decoder installed, Windows Media Player and some editors cannot decode the video track, so you may only hear audio or see a black screen.

如果不安装此解码器,Windows Media Player 和某些编辑器将无法解码视频轨道,因此您可能只能听到音频或看到黑屏。

Boy, do I remember that.

男孩,我还记得吗?

But this is reverse engineering that.

但这是逆向工程。

Jean-Baptiste Kempf1:49:16

This is key, right?

这是关键,对吧?

1:49:16 Because the GoToMeeting is something that not many people know anymore, right?

因为 GoToMeeting 已经没有多少人知道了,对吧?

Well, you know about Zoom and Teams and so on.

嗯,您知道 Zoom 和 Teams 等等。

But like, now let’s fast-forward 10 years, 15 years, and like this is a Gotomeeting.exe for Windows 32 bits, right?

但是,现在让我们快进 10 年、15 年,就像 Windows 32 位的 Gotomeeting.exe 一样,对吗?

Which is like, oh yeah, but I’m on Android, I’m on an iPad, I’m somewhere else, right?

这就像,哦,是的,但我在 Android 上,我在 iPad 上,我在其他地方,对吧?

How are you going to do that?

你打算怎么做?

I’m going to be on RISC-V, on Arm.

我将使用 RISC-V、Arm。

Those are blocked, but there are tons of files we need to support for the future.

这些已被阻止,但我们需要为未来支持大量文件。

And this is why those type of work are exceptionally useful for humanity.

这就是为什么这类工作对人类特别有用。

Lex Fridman1:49:51

I just have to say, though, that reverse engineering process is mind-blowing.

但我不得不说,逆向工程过程令人兴奋。

1:49:51 It’s crazy.

太疯狂了。

It’s like, it’s a kinda like, you know, I’ve been reading a lot and interview archeologists.

就像,有点像,你知道,我读了很多书并采访了考古学家。

I mean, you just have so little signal.

我的意思是,你的信号太少了。

Yes, yes, you know over time you get so much experience, you understand the structure of the original code, so you can kinda start inferring basics.

是的,是的,你知道随着时间的推移,你会获得很多经验,你了解原始代码的结构,所以你可以开始推断基础知识。

But you’re like archaeologists with a little brush trying to reconstruct the entire human civilization.

但你们就像考古学家一样,拿着小画笔试图重建整个人类文明。

Kieran Kunhya1:50:22

Kieran is too humble, but Kieran has done some reverse engineering also.

Kieran 太谦虚了,但 Kieran 也做了一些逆向工程。

Lex Fridman1:50:24

Of CineForm, yeah, at the time-

CineForm,是的,当时——

Kieran Kunhya1:50:26

CineForm, nice.

电影形式,不错。

Lex Fridman1:50:27

…

……

1:50:27 yeah, at the time before actually led to the open sourcing of that work.

是的,当时实际上导致了这项工作的开源。

So in parallel to doing the binary side, you obviously have samples.

因此,在进行二进制方面的同时,您显然还有样本。

In many cases, you don’t have many samples so you have to figure out what all the different flavors are, and you may have a So CineForm, for example, is actually a collection of different approaches and toolkits within that codec ’cause often it grows naturally.

在许多情况下,您没有很多样本,因此您必须弄清楚所有不同的风格是什么,例如,您可能有一个 So CineForm,它实际上是该编解码器中不同方法和工具包的集合,因为它通常会自然增长。

And the hard part is finding a sample that gets you kind of somewhere to start without having to implement 10 different other things.

困难的部分是找到一个示例,让您可以从某个地方开始,而不必实现 10 个不同的其他东西。

So start there.

所以从这里开始。

I think thankfully at the time I found a sample by pure chance that had a lot of flat blocks.

我想庆幸的是,当时我偶然发现了一个有很多扁平块的样本。

1:51:03 It was animation, so that really helped a lot because it wasn’t using particularly complex coding tools, et cetera, and you could kind of get somewhere and then, and then build up and build up until you figure, “Hey, here’s a few bits here.

它是动画,所以这确实有很大帮助,因为它没有使用特别复杂的编码工具等等,你可以到达某个地方然后,然后不断地构建,直到你发现,“嘿,这里有一些东西。

I missed this.

我错过了这个。

I missed this, this if branch that it does,” and go, “Oh.” So when we say samples, you mean sample videos- …

我错过了这个,如果它确实有这个分支,”然后说,“哦。”因此,当我们说样本时,您指的是样本视频 - ...

and then, and then you’re tracking, trying to infer, like, what is this codec doing- …

然后,然后你正在跟踪,试图推断,比如,这个编解码器在做什么——……

by observing the sample and then looking at what, at the machine lo-

通过观察样品,然后查看机器上的内容

Kieran Kunhya1:51:30

The machine code saying-

机器代码说-

Lex Fridman1:51:31

At the machine code.

在机器码处。

Kieran Kunhya1:51:31

…

……

1:51:31 “Ah, I have byte, this byte is six.

“啊,我有一个字节,这个字节是六。

Take this branch.” And in a different sample, oh, it’s-

拿走这根树枝吧。”在另一个样本中,哦,它是-

Lex Fridman1:51:36

That’s nuts, man.

那太疯狂了,伙计。

Kieran Kunhya1:51:37

And, and, and-

还有,还有,还有——

Lex Fridman1:51:38

That is nuts.

那是疯了。

Kieran Kunhya1:51:39

…

……

1:51:39 so you see, this is nuts.

所以你看,这太疯狂了。

Then you go to things like GoToMeeting.

然后您就可以进行 GoToMeeting 之类的活动。

It’s like-

就像-

Lex Fridman1:51:43

Mine was easy, right?

我的很简单,对吧?

Kieran Kunhya1:51:44

…

……

1:51:44 ima- imagine-

ima-想象-

Lex Fridman1:51:45

Yeah, right.

是的,对。

Kieran Kunhya1:51:45

…

……

1:51:45 two orders of magnitude more complexity.

复杂性增加两个数量级。

A guy alone somewhere in Germany doing that.

一个人独自在德国某个地方这样做。

And for a long time, you work, you’re in a black box because a decoder, for a long time, because there are so many steps from the entropy decoding, the intra prediction, the motion prediction, the IDCT, and so on.

很长一段时间,你工作,你在黑匣子里,因为解码器,很长一段时间,因为熵解码、帧内预测、运动预测、IDCT 等有很多步骤。

For a long time, you don’t see anything, right?

很长一段时间,你什么也看不见,对吗?

So you’re debugging purely in memory.

所以你纯粹在内存中进行调试。

Lex Fridman1:52:10

Debugging guesswork.

调试猜测。

Kieran Kunhya1:52:10

And you may have the buffer that the coefficients are stored in completely wrong, and so you may be going down a complete rabbit hole thinking it’s this and then, oh damn, that’s not, that’s, that’s something else, and-

而且你的缓冲区中的系数存储可能完全错误,所以你可能会陷入一个完整的兔子洞,认为是这个,然后,哦该死,那不是,那是,那是别的东西,而且-

Lex Fridman1:52:21

And you’re doing that on binaries that are tens of megabytes, millions of instructions, right?

您正在对数十兆字节、数百万条指令的二进制文件执行此操作,对吧?

Kieran Kunhya1:52:27

So you’re, you’re stepping through the debugger, like one by one, you know, instruction by instruction going, “Hey, this instruction changes this.

所以你正在单步执行调试器,就像一条一条,你知道,一条条指令地走,“嘿,这条指令改变了这一点。

1:52:27 This does this.”

这就是这样做的。”

Lex Fridman1:52:34

Pausing the program on the CPU level.

在 CPU 级别暂停程序。

1:52:34 Like it’s-

就像它是-

Kieran Kunhya1:52:36

Pausing it, yeah, on the CPU level, watching what’s going on, trying to figure out-

是的,在 CPU 级别上暂停,观察发生了什么,试图弄清楚-

Lex Fridman1:52:39

Sometimes you need to, like, be in a VM, so that you can pause the VM.

有时您需要在虚拟机中,以便可以暂停虚拟机。

Kieran Kunhya1:52:43

Yeah, pause the VM, dump the memory, ’cause there could, some of the codecs could have encryption.

是的,暂停虚拟机,转储内存,因为有些编解码器可能有加密。

1:52:43 There could be like a DRM on there.

那里可能有 DRM。

So you need to dump the memory from a virtual machine.

所以你需要从虚拟机转储内存。

Lex Fridman1:52:52

Like when I joined École Centrale Paris in 2003, Jon Lech Johansen basically broke the DVD specification and created DeCSS, showed us how he was breaking a DRM, which was MP4 FairPlay from Apple.

就像我 2003 年加入巴黎中央学校时一样,Jon Lech Johansen 基本上打破了 DVD 规范并创建了 DeCSS,向我们展示了他如何打破 DRM,即 Apple 的 MP4 FairPlay。

1:52:52 What he did on his laptop, and I was young, I was 21, was just like mind-blowing because he was basically debugging Windows inside a type of VM with ex- Like, wow.

他在他的笔记本电脑上所做的事情,我很年轻,21 岁,简直令人兴奋,因为他基本上是在一种虚拟机内调试 Windows,就像,哇。

It’s incredible.

太不可思议了。

It’s mind-blowing and inspiring.

这是令人兴奋和鼓舞人心的。

Does it get, like from your experience and from what you’ve seen in the community, does it get discouraging?

从您的经验和您在社区中看到的情况来看,它是否会令人沮丧?

Does it get-

是不是得到-

Kieran Kunhya1:53:26

People help you.

人们帮助你。

1:53:26 People send you samples.

人们给你寄样品。

People are keen.

人们很热心。

Sometimes you don’t have access to an encoder, so this is even more difficult because you just, you just ask and you have to ask for samples.

有时您无法访问编码器,所以这甚至更加困难,因为您只是,您只是要求并且必须要求样品。

I remember VideoLAN used to tweet for samples at one stage.

我记得 VideoLAN 曾一度在推特上发布样品。

“Hey, I need this obscure sample,” and-

“嘿,我需要这个晦涩的样本,”并且-

Lex Fridman1:53:42

For a long time I was, “Oh, I need this codec, and I need this codec.”

很长一段时间我都在想,“哦,我需要这个编解码器,我需要这个编解码器。”

Kieran Kunhya1:53:45

And if you were really lucky, you would find like…

如果你真的很幸运,你会发现......

1:53:45 If you were unlucky, you’d get nothing or you’d get one or two, and then they would…

如果你运气不好,你什么也得不到,或者只能得到一两个,然后他们就会……

Sometimes you’d find a goldmine.

有时你会发现一座金矿。

It’s like, “Yeah, my company has 100,000 of these files ’cause we’re dependent on it for some reason.” And so those are the, those are kind of the best because then they can test bit exactness across the huge range of coding tools.

就像,“是的,我的公司有 100,000 个这样的文件,因为我们出于某种原因依赖它。”这些是最好的,因为这样它们就可以在各种编码工具中测试位的准确性。

Lex Fridman1:54:06

Can you explain bit exactness?

你能解释一下位精确性吗?

Kieran Kunhya1:54:07

Bit exactness, so most but not all video codecs, certainly from about the 2000s onwards, have a bit exact definition, so every implementation must produce exactly the same bits, bit for bit, in exactly the same data that comes out of a decoder.

位精确性,因此大多数(但不是所有)视频编解码器(当然从 2000 年代起)都有位精确定义,因此每个实现都必须在从解码器输出的完全相同的数据中逐位生成完全相同的位。

Lex Fridman1:54:26

For like a large number of samples?

对于大量样本?

Kieran Kunhya1:54:28

For a given sample.

对于给定的样本。

1:54:28 So Lex’s implementation, JB’s implementation, and my implementation of H.264 must match bit exactly.

所以 Lex 的实现、JB 的实现和我的 H.264 实现必须完全匹配。

That wasn’t the case in the ’90s of MPEG-2, probably fair to say one of the biggest mistakes the video industry made, and I think people who were in the room in ’92—I don’t think most or both of us were in diapers, I suspect—but have acknowledged…

90 年代的 MPEG-2 情况并非如此,公平地说,这可能是视频行业犯下的最大错误之一,我认为 92 年在场的人(我怀疑我们大多数或两个人都没有穿尿布)但都承认……

I would give a shout-out to Yuriy Reznik.

我要为尤里·雷兹尼克点赞。

He’s acknowledged that was one of the big mistakes of the era.

他承认这是这个时代的重大错误之一。

Lex Fridman1:54:56

And you’re saying the encoders needed to be able to run tests and then the, the, the bit exactness—I mean, that’s a nice thing to guarantee.

你是说编码器需要能够运行测试,然后是位准确性——我的意思是,这是一个很好的保证。

1:54:56 Like there’s a parallel sort of development here on the way the, the web browser works, which is a, you know, takes HTML and displays it, and there’s no bit exactness there across the different engines.

就像这里有一种并行的开发方式一样,网络浏览器的工作方式是,你知道,它获取 HTML 并显示它,并且在不同的引擎之间没有一点精确性。

Kieran Kunhya1:55:15

I would, I would point out actually FFmpeg is unique in the sense that it’s, it has been a winner-takes-all scenario.

我想说,实际上 FFmpeg 是独一无二的,因为它是赢家通吃的场景。

1:55:15 You have…

你有……

Browsers is a good analogy because it has to parse a lot of different content and render it in a particular way, like a decoder.

浏览器是一个很好的类比,因为它必须解析许多不同的内容并以特定的方式呈现它,就像解码器一样。

But there still are multiple browser engines.

但仍然有多个浏览器引擎。

There’s Firefox’s one, there’s Chrome’s one, there’s a few Japanese ones that are pretty decent.

有 Firefox 的,有 Chrome 的,还有一些日本的,相当不错。

That’s not been the case in multimedia in general across a wide range of codecs.

在各种编解码器的多媒体领域,情况并非如此。

FFmpeg has kind of won it all, I suppose, in a sense because of, because of the fact that you can get—every new codec added is actually worth more than the value of that codec itself because it makes the whole thing better.

我想,从某种意义上说,FFmpeg 已经赢得了这一切,因为,因为您可以得到的事实 - 添加的每一个新编解码器实际上都比该编解码器本身更有价值,因为它使整个事情变得更好。

Lex Fridman1:55:52

Man, this is really cool.

伙计,这真的很酷。

1:55:52 Going to Perplexity.

走向困惑。

Yuriy Reznik is a multimedia and signal processing researcher, got his PhD in computer science from Kyiv University with over 150 papers and more than 80 granted US patents, contributor to major multimedia standards including H.264, MPEG-4, AVC-H.265 MPEG-4 ALS, G.718, and-

Yuriy Reznik 是一名多媒体和信号处理研究员,在基辅大学获得计算机科学博士学位,发表了 150 多篇论文和 80 多项授权美国专利,主要多媒体标准的贡献者,包括 H.264、MPEG-4、AVC-H.265 MPEG-4 ALS、G.718 和 -

Kieran Kunhya1:56:17

G.71 is telco stuff.

G.71 是电信公司的东西。

1:56:17 Telco.

电信公司。

Lex Fridman1:56:19

Oh.

哦。

1:56:19 And so he was more connected to companies.

因此他与公司的联系更加紧密。

Kieran Kunhya1:56:21

RealAudio, RealVideo, right?

真实音频、真实视频,对吧?

1:56:21 That was- …

那是——……

very important at that time

当时非常重要

Lex Fridman1:56:25

…

……

1:56:25 Zencoder, Brightcove, Contex.

Zencoder、Brightcove、Contex。

This, man, I need to hang out with Yuriy.

这个,伙计,我需要和尤里一起出去玩。

He’s legit.

他是合法的。

And he’s like one of the nicest person-

他就像是最好的人之一——

Kieran Kunhya1:56:33

Slack guy, yeah

懒惰的家伙,是的

Lex Fridman1:56:33

…

……

1:56:33 ever, right?

曾经,对吧?

Like for example, for my startup that I’m doing right now called Kyber, right?

例如,我现在正在做的创业公司叫 Kyber,对吧?

I met Yuriy because I met him every year at the Mile-High Video Conference, which is in Denver.

我认识尤里是因为我每年都会在丹佛举行的 Mile-High 视频会议上见到他。

And he gave me like so many good ideas and good things.

他给了我很多好主意和好东西。

He’s like really amazing person.

他真的是一个了不起的人。

Kieran Kunhya1:56:52

He tells us how great it is to be, you know, even know us.

他告诉我们,了解我们是多么伟大。

1:56:52 And then we just like, you know, you look at that and it’s—I think it’s the other way around, Yuriy.

然后我们就喜欢,你知道,你看一下,我认为情况恰恰相反,尤里。

Lex Fridman1:57:01

That reminds me of a thing that you mentioned to me about FATE testing and, like, the insanely rigorous process that’s used to test everything that’s incorporated into FFmpeg.

这让我想起您向我提到的有关 FATE 测试的一件事,以及用于测试 FFmpeg 中包含的所有内容的极其严格的流程。

1:57:01 Can you take me through the testing process?

您能带我完成测试过程吗?

Kieran Kunhya1:57:14

Yeah.

是的。

1:57:14 So FFmpeg has a system called FATE, FFmpeg Automated Testing Environment.

所以FFmpeg有一个叫做FATE的系统,FFmpeg自动化测试环境。

Because FFmpeg runs on so many different OSs and can be compiled with so many different compilers, there’s been a crazy number of configurations.

由于 FFmpeg 运行在许多不同的操作系统上,并且可以使用许多不同的编译器进行编译,因此存在大量的配置。

So you can see the absurd combination of compiler variants, operating system variants, instruction sets.

所以你可以看到编译器变体、操作系统变体、指令集的荒谬组合。

You can see at the top macOS has tons of different variants because it has iOS, it has tvOS.

您可以在顶部看到 macOS 有大量不同的变体,因为它有 iOS,有 tvOS。

Lex Fridman1:57:42

Well, I’m looking at a page fate.ffmpeg.org 81 minutes ago, 76 minutes ago, looking at the different architectures, the operating systems, the different compilers, Apple Clang version-

好吧,我正在查看 81 分钟前、76 分钟前的 fat.ffmpeg.org 页面,查看不同的架构、操作系统、不同的编译器、Apple Clang 版本 -

Kieran Kunhya1:57:57

Combinations are crazy

组合很疯狂

Lex Fridman1:57:58

…

……

1:57:58 the combination is insane.

这个组合太疯狂了。

RISC-

RISC-

Kieran Kunhya1:58:00

So these are all run by volunteers, so these are all volunteer systems.

所以这些都是志愿者运行的,所以这些都是志愿者系统。

1:58:00 The ones at the top, for example, the Macs I host in my office, for example, host all sorts of different stuff.

位于顶部的设备,例如我在办公室托管的 Mac,托管着各种不同的东西。

Other people host other things.

其他人主持其他事情。

So it’s really there to make sure…

所以它确实是为了确保……

because FFmpeg does quite complex C code, for example, you do have miscompilations.

因为 FFmpeg 执行相当复杂的 C 代码,例如,您确实会出现编译错误。

So the compiler will sometimes compile C code incorrectly.

所以编译器有时会错误地编译C代码。

For example, this happens once in a while.

例如,这种情况偶尔会发生一次。

Lex Fridman1:58:25

Oh, there’s like, there’s a log of all the compilations.

哦,就像,有一个所有编译的日志。

Kieran Kunhya1:58:28

Yeah, log of all the compilations, all the tests.

是的,所有编译、所有测试的日志。

1:58:28 I think one of the other ones will show all the tests passing.

我认为其他之一将显示所有测试都通过。

Jean-Baptiste Kempf1:58:33

If you click, you can see all the tests- …

如果单击,您可以看到所有测试 - ...

1:58:33 back.

回来。

All tests successful.

所有测试均成功。

Kieran Kunhya1:58:37

In logs test, yeah.

在日志测试中,是的。

1:58:37 So you see all those tests are passing of all the different codecs, all the different filter transformations, all the— the level of scale is quite crazy.

所以你看到所有这些测试都通过了所有不同的编解码器,所有不同的过滤器转换,所有的规模水平是相当疯狂的。

Lex Fridman1:58:50

Oh, that’s nuts.

哦,那太疯狂了。

Kieran Kunhya1:58:50

On all the combinations.

在所有组合上。

1:58:50 It’s not just a matrix at this point.

此时它不仅仅是一个矩阵。

It’s like a pivot table of different combinations.

它就像不同组合的数据透视表。

Lex Fridman1:58:56

That’s nuts.

那太疯狂了。

Kieran Kunhya1:58:57

And it’s a key part of what we do because you may be able to test something locally, you make a change, but actually that breaks GCC version 11 on Mac or something like that, and you’re able to then fix that.

这是我们工作的关键部分,因为你可能能够在本地测试某些东西,你进行更改,但实际上这会破坏 Mac 上的 GCC 版本 11 或类似的东西,然后你就可以修复它。

1:58:57 We also have miscompilation, so the C code, sometimes the compiler can have a bug in it where it creates the wrong output, and that can have quite a big effect sometimes on a video because of the way frames have dependencies.

我们还存在编译错误,因此 C 代码有时编译器可能存在错误,会创建错误的输出,并且由于帧具有依赖性,有时会对视频产生相当大的影响。

Even a small change in the output can cascade to actually quite big glitches.

即使输出的微小变化也可能导致实际上相当大的故障。

Jean-Baptiste Kempf1:59:27

You see PowerPC, you see RISC, you see ARM.

你会看到 PowerPC、RISC、ARM。

Kieran Kunhya1:59:31

There was PowerPC, there was RISC, there was weird stuff in the past like DEC Alpha.

有 PowerPC,有 RISC,过去有像 DEC Alpha 这样奇怪的东西。

1:59:31 There was-

有——

Jean-Baptiste Kempf1:59:35

You see Visual Studio, different versions of Clang or GCC.

您会看到 Visual Studio、不同版本的 Clang 或 GCC。

Kieran Kunhya1:59:37

Visual Studio, Intel compiler, Apple Clang, you name it.

Visual Studio、Intel 编译器、Apple Clang,应有尽有。

Lex Fridman1:59:40

What are some of the pain points?

有哪些痛点?

1:59:40 Like maybe do you have emotional triggers, maybe nightmares, about a particular operating system, a particular container, codec combination of—

就像你是否有情绪触发因素,也许是噩梦,关于特定的操作系统、特定的容器、编解码器组合——

Kieran Kunhya1:59:53

I mean, for me, it’s really easy because I have a day job.

我的意思是,对我来说,这真的很容易,因为我有一份日常工作。

1:59:53 My company builds…

我公司建…

The company I started builds equipment for broadcasting sports matches between TV stadiums and studios, for example.

例如,我创办的公司生产用于在电视体育场和演播室之间转播体育比赛的设备。

We have to work with 10-bit video, and 10-bit video has a set of challenges that you can’t process 10-bit data natively on a CPU.

我们必须使用 10 位视频,而 10 位视频面临一系列挑战,即您无法在 CPU 上本地处理 10 位数据。

So that means you have to stick it in 16 bits.

所以这意味着你必须将其固定为 16 位。

So that means you have six wasted bits.

所以这意味着你有六个被浪费的位。

So there’s different packing formats to actually pack the data more efficiently because when you send that over a network, you need to save that 40%.

因此,有不同的打包格式可以实际更有效地打包数据,因为当您通过网络发送数据时,您需要节省 40%。

For example, on PCI Express, you may only have bus bandwidth to do that.

例如,在 PCI Express 上,您可能只有总线带宽来执行此操作。

And so I think internally we have about…

所以我认为我们内部有……

2:00:36 Some are industry ones and some are internal to our own hardware that we build.

有些是行业的,有些是我们自己构建的硬件内部的。

We have a, I think, a 5 by 5 or 6 by 6 matrix of every single format to every single other format conversion.

我认为,我们有一个 5 x 5 或 6 x 6 矩阵,用于将每种格式转换为每种其他格式。

In fact, one of them I sent you, and they’re all written in handwritten assembly, and they all support different CPU generations.

事实上,我发给你的其中之一,都是手写汇编编写的,而且都支持不同代的CPU。

So this is really traumatic, handling all these different combinations times a million.

所以这真的很痛苦,处理所有这些不同的组合一百万次。

Jean-Baptiste Kempf2:01:02

By the way, the company you’re talking about is Open Broadcast Systems.

顺便说一句,你所说的公司是开放广播系统公司。

Kieran Kunhya2:01:04

Yeah, so no, no relation to the free OBS streaming service.

是的,所以不,与免费的 OBS 流媒体服务无关。

2:01:04 But JB and I have started companies broadly speaking around the FFmpeg VLC ethos, so that’s really low-level work.

但 JB 和我已经围绕 FFmpeg VLC 精神创立了公司,所以这确实是低级工作。

So in most companies, this wouldn’t be written in assembly.

所以在大多数公司中,这不会用汇编语言编写。

It would be accepted that C is fast.

C 的速度很快是可以接受的。

As you can see from that, C is not fast.

由此可见,C 并不快。

Lex Fridman2:01:26

So here it says 62 times faster than C.

所以这里说比C快62倍。

Kieran Kunhya2:01:31

Yeah.

是的。

2:01:31 So it’s taking the ethos of doing low-level programming, real-time programming, and using that for commercial applications, and JB and I have started companies around that, in many cases hiring developers from the open source community to use that ethos.

因此,它采用了进行低级编程、实时编程的精神,并将其用于商业应用程序,JB 和我围绕这一点创办了公司,在许多情况下,从开源社区雇佣开发人员来利用这种精神。

And so that’s a great example of some of the things we’re doing.

这是我们正在做的一些事情的一个很好的例子。

In most companies, it would be, say, “Oh, I’ll write this in C and it’s fast and we’re done,” but actually you can get a lot better.

在大多数公司中,会说,“哦,我会用 C 语言编写这个,速度很快,我们就完成了”,但实际上你可以做得更好。

Jean-Baptiste Kempf2:01:59

For me, some of the headaches we have is around some OS that are difficult to support, right?

对我来说,我们遇到的一些令人头疼的问题是一些难以支持的操作系统,对吗?

2:01:59 Because if you look at VLC and thanks to FATE and FFmpeg, we run on…

因为如果你看看 VLC 并感谢 FATE 和 FFmpeg,我们就可以运行......

The last version of VLC runs on Windows XP and still run there and runs on Windows 11.

VLC 的最新版本在 Windows XP 上运行,并且仍然在 Windows 11 上运行。

We work on macOS 10.7 to the latest macOS, whatever it is, right, 26.

我们使用 macOS 10.7 到最新的 macOS,无论它是什么,对吧,26。

We work on iOS since iOS 9; well, we are actually iOS 26, right?

我们从 iOS 9 开始就在 iOS 上工作;好吧,我们实际上是 iOS 26,对吧?

We support many types of Linuxes, BSD, Solaris.

我们支持多种类型的 Linux、BSD、Solaris。

The last version still runs on OS/2, right?

最后一个版本仍然运行在 OS/2 上,对吧?

Like there is maybe 10 users of OS/2 in the world, and one of them is maintaining VLC.

就像世界上可能有 10 个 OS/2 用户,其中之一正在维护 VLC。

2:02:50 Then you realize that this very small team around VLC and using FFmpeg codecs and all the other ones support more OSs than Microsoft or Google or Apple, and they have infinite amount of power and resources.

然后你就会意识到,这个围绕 VLC 并使用 FFmpeg 编解码器和所有其他编解码器的非常小的团队支持比 Microsoft、Google 或 Apple 更多的操作系统,并且他们拥有无限的权力和资源。

But for example, the worst is iOS.

但举个例子,最糟糕的是iOS。

In order to build on iOS 9, we need to do some very clever mixing of several version of the Xcode IDE and SDK from Apple, and do a type of Frankenstein version of that so that we can still support iOS 9, which is not supported at all by the compiler of Apple in order to still run on Arm32 on iOS 9.

为了在 iOS 9 上进行构建,我们需要对 Apple 的多个版本的 Xcode IDE 和 SDK 进行一些非常巧妙的混合,并制作一种弗兰肯斯坦版本,以便我们仍然可以支持 iOS 9,而 Apple 的编译器根本不支持 iOS 9,以便仍然在 iOS 9 上的 Arm32 上运行。

And you’ve seen on FATE that it was still supporting iOS 9, right?

你在 FATE 上看到它仍然支持 iOS 9,对吧?

So my headaches are mostly related to the support of so many OSs.

所以我的头疼大多和这么多操作系统的支持有关。

2:03:48 And it’s important because, like, we receive so many people saying, “Hey, thank you.

这很重要,因为我们收到很多人说:“嘿,谢谢。

I still have my iPad 2 to watch movies,” and it still works on iOS 9, right?

我还有 iPad 2 来看电影”,而且它仍然可以在 iOS 9 上运行,对吧?

And it’s also an impact of, like, not forcing people to buy new hardware when it works fine if you optimize it correctly.

这也是一个影响,比如,如果你正确优化它,当它工作正常时,不会强迫人们购买新硬件。

Which brings us to what we were saying about assembly.

这就引出了我们刚才所说的关于装配的内容。

It’s also fighting, like, the fact that you need to buy something new nonstop while you could optimize more, which is a lost art.

这也是一种斗争,比如,你需要不停地购买新的东西,同时你可以优化更多东西,这是一门失落的艺术。

Lex Fridman2:04:18

You gotta tell me about this lost art or this, uh- …

你得告诉我关于这门失传的艺术或者这个,呃-......

2:04:18 the carriers of the flame of assembly.

集会火焰的载体。

What is Assembly?

什么是装配?

Why is it beautiful?

为什么它是美丽的?

Why is it challenging?

为什么它具有挑战性?

How does it work?

它是如何运作的?

Kieran Kunhya2:04:34

So when you write assembly code, you write this using the instructions the actual processor is using directly.

因此,当您编写汇编代码时,您可以使用实际处理器直接使用的指令来编写它。

2:04:34 So most of the time you would write in a language, let’s take C as a good example.

所以大多数时候你会用一种语言编写,让我们以 C 为例。

The compiler would use that to create assembly language and machine code instructions for you based off your C code.

编译器将使用它根据您的 C 代码创建汇编语言和机器代码指令。

And there’s a specific flavor of assembly that we use in FFmpeg that’s called SIMD, single instruction, multiple data.

我们在 FFmpeg 中使用一种特殊的汇编风格,称为 SIMD,单指令,多数据。

So this means, for example, say I want to add five to a number in scalar assembly, so this is what’s known as you work on an individual element.

因此,这意味着,例如,我想在标量汇编中将数字加五,这就是所谓的处理单个元素。

So I want to have a number of– I have the number ten and I want to add five.

所以我想要一些——我有数字十,我想加上五个。

I use the add instruction, and I add five to ten, and I get 15.

我使用 add 指令,将 5 加到 10,得到 15。

2:05:19 With SIMD, I can have a whole vector of 16 different numbers.

借助 SIMD,我可以获得包含 16 个不同数字的完整向量。

They could all be different.

他们可能都不同。

If I want to add five to that, I can run one instruction, and that one instruction sums all 16 elements.

如果我想添加 5 个元素,我可以运行一条指令,该指令对所有 16 个元素求和。

And that, as you can imagine, lends itself very well to video.

正如您可以想象的那样,这非常适合视频。

Video is, you know, pixel grid, so I can perform operations on multiple pixels at the same time.

你知道,视频是像素网格,所以我可以同时对多个像素执行操作。

The key thing that we do differently in FFmpeg is we don’t use any abstractions or any major abstractions on top of that.

我们在 FFmpeg 中所做的不同的关键是我们不使用任何抽象或在此之上的任何主要抽象。

So there’s a part of the world that uses what’s known as intrinsics.

所以世界上有一部分使用所谓的内在函数。

So these are C functions that behave very similarly but not quite the same to writing assembly by hand.

因此,这些 C 函数的行为与手动编写汇编非常相似,但又不完全相同。

So the registers that data is stored in on the CPU, the compiler allocates those for you.

因此,CPU 上存储数据的寄存器是编译器为您分配的。

2:06:10 And so the key thing to understand when we write SIMD is we have a 10x, and not percentage, 10x to 50x speed improvement.

因此,当我们编写 SIMD 时,要理解的关键是我们的速度提高了 10 倍,而不是百分比,即 10 倍到 50 倍。

That function is 62x—

该函数是 62x——

Lex Fridman2:06:21

That’s nuts.

那太疯狂了。

Kieran Kunhya2:06:22

…

……

2:06:22 on the FFmpeg account, as you know, posts and tweets a lot about that to try and say, “Hey, we are doing this stuff.”

如您所知,在 FFmpeg 帐户上,发布了很多有关此内容的帖子和推文,试图说:“嘿,我们正在做这些事情。”

Lex Fridman2:06:29

You are a person who sees the beauty in assembly, but it’s also extremely useful for these kinds of application to actually- …

您是一个能够看到组装之美的人,但对于这些类型的应用程序来说,它实际上也非常有用......

2:06:29 significantly outperform even C, which is crazy.

甚至明显优于 C,这太疯狂了。

Jean-Baptiste Kempf2:06:40

It is necessary.

有必要。

2:06:40 Right?

正确的?

Because, like, one of the projects that we need to talk about is called dav1d, right?

因为,我们需要讨论的项目之一叫做 dav1d,对吧?

So dav1d is a decoder for the format that was done by Alliance for Open Media, which is a video decoder called AV1.

所以 dav1d 是由开放媒体联盟完成的格式的解码器,这是一个称为 AV1 的视频解码器。

Lex Fridman2:06:58

So if, for people who don’t know, we’ve been talking about H.264.

所以,对于那些不知道的人来说,我们一直在谈论 H.264。

2:06:58 AV1 is another hugely popular standard and codec that is increasingly taking over the internet.

AV1 是另一个非常流行的标准和编解码器,它越来越多地接管互联网。

Jean-Baptiste Kempf2:07:12

And when this format was launched many people said, especially even from the Alliance for Open Media, right, which is Google, Netflix, Amazon, Mozilla, say, “Well, this format is so complex, it must be done in hardware to do decoding,” right?

当这种格式推出时,很多人都说,尤其是开放媒体联盟,对吧,谷歌,Netflix,亚马逊,Mozilla,说,“嗯,这种格式太复杂了,必须用硬件来解码,”对吧?

2:07:12 And well, I arrived with a few other people, mostly Ronald, Henrik, and Martin, and we said, “We need to have an extremely good software decoder because it’s going to take time to have hardware.” And so we wrote this project, which is beyond insane.

好吧,我和其他几个人一起到达,主要是罗纳德、亨利克和马丁,我们说,“我们需要一个非常好的软件解码器,因为拥有硬件需要时间。”所以我们写了这个项目,这太疯狂了。

We are talking about 30,000 lines of C, but 240,000 lines of handwritten assembly, right?

我们谈论的是 30,000 行 C 语言,但手写汇编有 240,000 行,对吗?

Lex Fridman2:07:56

Handwritten assembly, 240,000 lines.

手写汇编,24万行。

2:07:56 That’s incredible.

那真是难以置信。

That means—I mean, some of the stuff we’re talking about is probably the biggest assembly code bases.

这意味着——我的意思是,我们正在谈论的一些东西可能是最大的汇编代码库。

Jean-Baptiste Kempf2:08:09

To give you an idea, and Kieran can correct me, but I think the FFmpeg has 100,000 lines of assembly for all the codecs.

为了给你一个想法,Kieran 可以纠正我,但我认为 FFmpeg 对于所有编解码器都有 100,000 行汇编。

Kieran Kunhya2:08:17

For all codecs.

对于所有编解码器。

2:08:17 Mm-hmm.

嗯嗯。

Jean-Baptiste Kempf2:08:17

And just this one has 240,000.

而光是这个就有24万。

It’s a VideoLAN project, of course.

当然,这是一个 VideoLAN 项目。

And it is optimized at the maximum because the motto when we’re starting the project is every cycle matters, right?

它得到了最大限度的优化,因为我们开始项目时的座右铭是每个周期都很重要,对吗?

Every cycle matters because dav1d is used in VLC and in some software AV1 playback stacks.

每个周期都很重要,因为 dav1d 用于 VLC 和某些软件 AV1 播放堆栈中。

We are talking about probably 3 billion devices which are going to decode video nonstop because, for example, 30% of the video from Netflix are now in AV1, 50% of YouTube, right?

我们谈论的是可能有 30 亿台设备将不间断地解码视频,因为例如 Netflix 的 30% 的视频现在采用 AV1,YouTube 的 50%,对吧?

So, and you often don’t have a hardware decoder because not many devices have a hardware decoder.

因此,您通常没有硬件解码器,因为没有多少设备具有硬件解码器。

And with dav1d, we realized that with one or two cores you were able to decode 720p correctly.

通过 dav1d,我们意识到使用一两个核心就可以正确解码 720p。

So it is, like, literally-

所以,从字面上看就是——

Kieran Kunhya2:09:08

Yeah, that’s dav1d.

是的,那就是 dav1d。

Jean-Baptiste Kempf2:09:08

…

……

2:09:08 incredible, right?

难以置信,对吧?

Kieran Kunhya2:09:09

That’s dav1d.

那是 dav1d。

2:09:09 Look at that, Lex.

看看那个,莱克斯。

Lex Fridman2:09:10

Yeah, so this is another spicy tweet from you.

是的,这是你的另一条辛辣推文。

2:09:10 This is what peak video codec should look like, 79.9% assembly-

这就是峰值视频编解码器应有的样子,79.9% 的汇编 -

Kieran Kunhya2:09:20

That’s almost

这差不多了

Lex Fridman2:09:20

…

……

2:09:20 19.6% C and 0.5% other.

19.6% C 和 0.5% 其他。

Jean-Baptiste Kempf2:09:25

And what’s incredible is with those tweets, which are factual, people get crazy.

令人难以置信的是,这些真实的推文让人们变得疯狂。

2:09:25 They are unhappy, right?

他们不高兴,对吗?

They say-

他们说——

Kieran Kunhya2:09:33

For a year, for the last two years they go crazy, “No, intrinsics is fine.

在过去的一年里,在过去的两年里,他们变得疯狂,“不,内在函数很好。

2:09:33 The compiler is…” Oh, they go, “I have never-“

编译器是……”哦,他们说,“我从来没有——”

Jean-Baptiste Kempf2:09:37

“You can optimize your compiler, auto-vectorization, it’s your fault, you don’t understand.” And we’ve tried that forever, right?

“你可以优化你的编译器,自动矢量化,这是你的错,你不明白。”我们一直在尝试,对吧?

Kieran Kunhya2:09:44

For two years, and two years later, showing hundreds of examples of handwritten assembly.

两年来,两年后,展示了数百个手写汇编的示例。

2:09:44 “No, no, no, you’re doing it wrong.

“不不不,你错了。

The compiler can do this.”

编译器可以做到这一点。”

Lex Fridman2:09:52

So we should actually just articulate a little clearer.

所以我们实际上应该说得更清楚一点。

2:09:52 So the intuition there from the software engineering folks, when you have code like…

所以,当你有这样的代码时,软件工程人员的直觉是……

Okay, let’s just take an example, C++.

好吧,我们就举个例子吧,C++。

There’s a compiler that’s doing a lot of the optimization.

有一个编译器正在做很多优化。

And the presumption is if you have a good enough compiler, if you continue to improve the compiler, you’re going to generate code- …

假设是如果你有一个足够好的编译器,如果你继续改进编译器,你将生成代码-......

that can perform like optimal performance.

可以达到最佳性能。

You cannot possibly beat it.

你不可能打败它。

And you’re consistently challenging that thought that if you do-

你一直在挑战这样的想法:如果你这样做——

Kieran Kunhya2:10:21

By orders of magnitude.

按数量级计算。

Lex Fridman2:10:22

…

……

2:10:22 by orders of magnitude- …

按数量级 - …

handcrafted assembly can outperform C.

手工组装的性能优于 C.

Jean-Baptiste Kempf2:10:27

The two things that they tell us is, yeah, but modern compilers have auto-vectorization, right?

他们告诉我们的两件事是,是的,但是现代编译器具有自动向量化,对吧?

2:10:27 Because SIMD that we’re doing is vectorization.

因为我们正在做的SIMD是矢量化。

And like it’s not even close, right?

好像还不太接近,对吧?

It’s not even close, right?

还差得很远,对吧?

It’s not like 5%, 10% slower.

速度慢了不到 5%、10%。

It’s multiple times slower.

它慢了好几倍。

Lex Fridman2:10:43

So can we…

那么我们可以……

2:10:43 I don’t know if you can say something philosophically, because there’s a lot of, there’s a lot of great software engineers, great engineers, great machine learning people.

我不知道你是否可以从哲学角度说一些东西,因为有很多很多伟大的软件工程师、伟大的工程师、伟大的机器学习人员。

Karpathy will listen to this and say, “What’s the intuition he’s supposed to get from this?

卡帕蒂会听完后说:“他应该从中得到什么直觉?

What are we supposed to…”

我们该怎么办……”

Kieran Kunhya2:10:57

Karpathy learnt assembly because of the tweets, by the way.

顺便说一句,卡帕蒂因为这些推文而学会了汇编。

2:10:57 I just…

我只是……

He start- He went, he’s like, “Oh, I think this is a movement.”

他开始——他走了,他说,“哦,我认为这是一场运动。”

Lex Fridman2:11:01

He’s like, “Let me figure out what’s happening here.”

他就像是,“让我弄清楚这里发生了什么。”

Kieran Kunhya2:11:02

No, no, he…

不不不,他……

2:11:02 and you know the way he documents his work and so on.

你知道他记录工作的方式等等。

Jean-Baptiste Kempf2:11:05

Philosophically, what’s important to realize is that we passed the time where hardware was going so much faster, right?

从哲学上讲,重要的是要认识到我们已经度过了硬件发展速度如此之快的时代,对吗?

2:11:05 We are at the end of Moore’s law.

我们正处于摩尔定律的尽头。

We have limitation for AI, for memory.

我们对人工智能和记忆力都有限制。

You need to go down in the stack and optimize more to get more power from what you have, because our request for power, CPU power, GPU power are exploding while the hardware is not exploding in speed, right?

你需要深入堆栈并进行更多优化,以便从你拥有的资源中获得更多功能,因为我们对电源、CPU 电源、GPU 电源的要求正在爆炸式增长,而硬件的速度却没有爆炸式增长,对吗?

So what people do is that they add more cores, right?

那么人们所做的就是添加更多核心,对吧?

But that’s basically like at some point you can add 250 cores, right?

但这基本上就像在某个时候您可以添加 250 个核心,对吧?

So what we do is to take every inch of the machine.

所以我们要做的就是充分利用机器的每一寸。

Kieran Kunhya2:11:46

Not just that, not just that.

不仅如此,不仅如此。

2:11:46 We abuse the machine.

我们滥用机器。

We go and use the machine in ways that the creator didn’t expect.

我们以创造者意想不到的方式使用机器。

Sometimes we use an instruction that’s completely unrelated to what we do.

有时我们使用的指令与我们所做的事情完全无关。

We use a cryptography instruction in video processing to do nothing related.

我们在视频处理中使用加密指令不执行任何相关操作。

Jean-Baptiste Kempf2:12:01

And one of other things that we do, for example in dav1d, which is a bit crazy, is that we don’t use the function calling convention from the operating system.

我们做的其他事情之一,例如在 dav1d 中,有点疯狂,是我们不使用操作系统的函数调用约定。

Kieran Kunhya2:12:13

We should explain that.

我们应该解释一下。

Jean-Baptiste Kempf2:12:14

That is extremely- …

那是极其——……

2:12:14 complex.

复杂。

But basically, usually when you do move from one function in code to another, there is a way to save the registry, the state of the CPU to enter another function.

但基本上,通常当您从代码中的一个函数转移到另一个函数时,有一种方法可以保存注册表、CPU 的状态以进入另一个函数。

And this is like standard.

这就像标准一样。

Kieran Kunhya2:12:28

It’s a bit complex.

这有点复杂。

2:12:28 I would simplify this a bit.

我会稍微简化一下。

So, dav1d does things to abuse the calling convention.

因此,dav1d 会做一些滥用调用约定的事情。

You could define the calling convention as I’ve written a function and I want to call another function.

您可以定义调用约定,就像我编写了一个函数并且我想调用另一个函数一样。

How is the data shared between the functions?

功能之间如何共享数据?

Because there’s a convention, what’s known as a calling convention, and what dav1d does for optimal reasons is create its own calling convention sometimes.

因为有一个约定,即所谓的调用约定,而 dav1d 出于最佳原因所做的就是有时创建自己的调用约定。

So if I wanna call Lex Fridman’s library, we’ve got to agree on a convention so that I can share data with you in the assembly language space.

因此,如果我想调用 Lex Fridman 的库,我们必须就约定达成一致,以便我可以在汇编语言空间中与您共享数据。

And one of the challenges in assembly is every operating s- well, not every operating system, but there are, well, at least four that I can think of on x86, Linux 32-bit, Windows 32-bit, Windows 64, Linux 64.

汇编的挑战之一是每个操作系统——好吧,不是每个操作系统,但我能想到的至少有四种操作系统,分别是 x86、Linux 32 位、Windows 32 位、Windows 64、Linux 64。

2:13:11 They all have their own calling conventions.

他们都有自己的调用约定。

And so one of the amazing things Loren Merritt did, who we talked about before, was create a very lightweight abstraction layer, so you could write your assembly code once and it handled all the calling convention stuff for you, which was always a problem because you had to manage four different variants.

因此,Loren Merritt 所做的一件令人惊奇的事情(我们之前谈到过)是创建一个非常轻量级的抽象层,这样您只需编写一次汇编代码,它就会为您处理所有调用约定的内容,这始终是一个问题,因为您必须管理四种不同的变体。

But dav1d takes this even further, for speed reasons it does its own calling convention within itself to bypass the kind of rules, the rules of functions and say, “Okay, actually I’m gonna call a function this way because I know it’s within my library.”

但 dav1d 更进一步,出于速度原因,它在自身内部执行自己的调用约定,以绕过这种规则,即函数规则,并说:“好吧,实际上我将以这种方式调用函数,因为我知道它在我的库中。”

Lex Fridman2:13:45

Does it have to be special to every single operating system?

它对于每个操作系统都必须是特殊的吗?

Kieran Kunhya2:13:48

Well, if it’s custom, no.

好吧,如果是习惯,那就不行。

2:13:48 But the challenge is in general, yes, and in terms of each instruction set.

但总体而言,挑战是存在的,而且是针对每个指令集的。

So the thing to also emphasize is we do this on every instruction set.

所以还要强调的是我们在每个指令集上都这样做。

So every instruction set has its own handwritten assembly, which is even more crazy.

所以每个指令集都有自己的手写汇编,这就更疯狂了。

And that matrix has got bigger in recent years because of RISC-V, because of ARM64, because of the new SVE.

近年来,由于 RISC-V、ARM64、新的 SVE,这个矩阵变得越来越大。

There’s SME.

有中小企业。

x86 has AVX-512, AVX.

x86 有 AVX-512、AVX。

So we do runtime processor detection.

所以我们进行运行时处理器检测。

We see what the machine FFmpeg is running on or dav1d’s running on is capable of, because you could be on a laptop from 2008 where this isn’t there.

我们了解运行 FFmpeg 的机器或运行 dav1d 的机器的功能,因为您可能使用的是 2008 年的笔记本电脑,但不存在此功能。

Runtime detection, we set function pointers accordingly.

运行时检测,我们相应地设置函数指针。

And then from then on, off you go.

然后从那时起,你就走吧。

Lex Fridman2:14:34

Or you could be on a machine with RISC-V.

或者您可以使用带有 RISC-V 的机器。

Jean-Baptiste Kempf2:14:36

Yes.

是的。

2:14:36 And in all that, we don’t even respect the calling convention of the operating system in order to be faster, because we know that we are going to be called from within our binary, so we can share data without saving all the registry in the common way, because that can lead to loading and saving registry on the L1 and L2 CPU and gets us faster.

总而言之,为了更快,我们甚至不尊重操作系统的调用约定,因为我们知道我们将从二进制文件中调用,因此我们可以共享数据,而无需以通用方式保存所有注册表,因为这可能导致在 L1 和 L2 CPU 上加载和保存注册表,从而使我们更快。

So that’s why I said that understanding CPU architecture, computer architecture is key.

所以这就是为什么我说理解CPU架构、计算机架构是关键。

And this is also why it’s handwritten.

这也是为什么它是手写的。

I don’t know anyone, I’ve never heard any other project than Dav1d doing that.

我不认识任何人,除了 Dav1d 之外我从未听说过任何其他项目这样做。

This is why Kieran calls it an art, right?

这就是基兰称之为艺术的原因,对吧?

It is an art.

这是一门艺术。

Kieran Kunhya2:15:14

I think in a mass world, there isn’t something on billions of devices.

我认为在大众世界中,数十亿台设备上没有任何东西。

2:15:14 I know there are some specialist industries.

我知道有一些专业行业。

I know in high-frequency trading, they take this really seriously, where they’re receiving feeds from a market, and they need to react within X number of microseconds, and so the instructions matter.

我知道在高频交易中,他们非常重视这一点,他们从市场接收反馈,他们需要在 X 微秒内做出反应,因此指令很重要。

But that’s not a mass-produced thing that’s on a billion devices.

但这并不是一个在十亿台设备上大规模生产的东西。

That’s hyper-specialized, running on hyper-specialized hardware.

这是超专业化的,在超专业化的硬件上运行。

We’re running on all hardware from-

我们在所有硬件上运行 -

Lex Fridman2:15:39

Sorry to linger on it, but, like, that’s a really counterintuitive, almost, like, revolutionary idea here, that there’s a huge amount of value to assembly.

很抱歉在这个问题上徘徊,但是,这确实是一个违反直觉的、几乎是革命性的想法,即装配具有巨大的价值。

2:15:39 Like, what are we supposed to take away from that?

比如,我们应该从中得到什么?

Like, what…

比如,什么……

You know, there’s a bunch of people listening to this, they’re basically like, sorry, for myself included, you know, I programmed for many, many years in C/C++, going up the standards of C++, fell in love with C++, even meta programming and so on, and then transitioned more and more because of machine learning about 15 years ago to Python.

你知道,有很多人在听这个,他们基本上都会说,对不起,包括我自己,你知道,我用 C/C++ 编程了很多很多年,提高了 C++ 的标准,爱上了 C++,甚至元编程等等,然后由于大约 15 年前的机器学习,越来越多地转向了 Python。

2:16:12 And so, like, for me in this Python world, JavaScript world, now vibe coding, where I’m just using natural language, sitting in my jacuzzi, drinking a drink…

所以,就像,对我来说,在这个 Python 世界、JavaScript 世界、现在的氛围编码中,我只是使用自然语言,坐在按摩浴缸里,喝一杯饮料……

and just talking to the computer, like record stops.

只是和电脑说话,就像录音停止一样。

Why is the value to go back all the way down to the low level?

为何价值一路回落至低位?

Like, what’s the intuition?

比如,直觉是什么?

Jean-Baptiste Kempf2:16:31

Because you can get more power per dollar invested, right?

因为每投入一美元你就能获得更多的力量,对吗?

2:16:31 And sometimes it’s going to be a problem that is limited by your hardware.

有时这会是一个受硬件限制的问题。

A good analogy is what you see in quantization in LLMs, right?

一个很好的类比就是你在法学硕士中看到的量化,对吧?

And people are doing, “Oh, I’m going to do that in FP8 or FP4 or some crazy things like Microsoft Phi, who did it in 1.5,” because you’re constrained by memory, because you’re constrained by the machine you can run.

人们会说,“哦,我要在 FP8 或 FP4 或一些疯狂的东西中做到这一点,比如 Microsoft Phi,他们在 1.5 中做到了这一点”,因为你受到内存的限制,因为你受到可以运行的机器的限制。

Because at some point we are doing real time, and I believe this is going to happen on AI inference also, is that at some point you need to get faster, and you cannot always get more powerful hardware, right?

因为在某些时候我们正在做实时的事情,我相信这也会发生在人工智能推理上,是不是在某些时候你需要变得更快,并且你不能总是获得更强大的硬件,对吧?

So you need to analyze code and see where, like, where is the mission critical, where are the things that are called nonstops.

所以你需要分析代码,看看哪里是关键任务,哪里是所谓不间断的东西。

2:17:21 And for example, dav1d is a good example.

例如,dav1d 就是一个很好的例子。

It’s going to be run billions of hours per day.

它每天将运行数十亿小时。

That makes sense.

这是有道理的。

It doesn’t make sense to be on the glue of FFmpeg- …

依赖 FFmpeg 是没有意义的——……

CLI.

命令行界面。

It makes sense over there.

那边说得有道理。

Lex Fridman2:17:35

Yeah, and this has to do, also we’ll talk about it more, but your new effort, your new company, Kyber, is doing that kind of thing for ultra-low latency, so the slogan being, “Every millisecond counts.” And when you actually extremely highly constrained in some dimension-

是的,这必须做,我们也会更多地讨论它,但是你的新努力,你的新公司 Kyber 正在做这种事情以实现超低延迟,所以口号是,“每一毫秒都很重要。”当你实际上在某个维度上受到极大的限制时——

Jean-Baptiste Kempf2:17:53

We are also arriving at a point where we’ve done so many great things, but the hardware is getting back to us, right?

我们也已经到达了这样一个阶段:我们已经做了很多伟大的事情,但硬件正在回到我们身边,对吗?

2:17:53 Because cost is increasing, because we need more power, and so you’re limited by either your CPU, your RAM, or your networking, and you need to optimize, and this is where value is going to be.

因为成本在增加,因为我们需要更多的功率,所以你受到 CPU、RAM 或网络的限制,你需要优化,而这就是价值所在。

Especially because, like, doing AI is going to help do the programming of, like, business, right?

特别是因为,人工智能将有助于商业编程,对吗?

And so the core thing that you will not be able to vibe code are optimization for the hardware to be as fast as is possible.

因此,您无法振动代码的核心是对硬件进行优化,使其尽可能快。

Lex Fridman2:18:31

I’d love to talk to you about who and how should learn assembly, but first, I think we need a bathroom break.

我很想和你谈谈谁应该学习组装以及如何学习组装,但首先,我认为我们需要先上个厕所。

2:18:31 Quick ten-second thank you to our sponsors.

快速感谢我们的赞助商十秒钟。

Check them out in the description.

在描述中查看它们。

It really is the best way to support this podcast.

这确实是支持这个播客的最佳方式。

Go to lexfridman.com/sponsors.

请访问 lexfridman.com/sponsors。

And now back to the episode.

现在回到这一集。

All right, and we’re back.

好吧,我们回来了。

There’s this nice repo with the assembly lessons.

有一个很好的仓库,里面有组装课程。

First of all, do you think developers should learn how to program in assembly, and how would you go about learning it?

首先,您认为开发人员是否应该学习如何使用汇编语言进行编程?您将如何学习?

What is this asm-lessons?

这是什么 asm 课程?

Kieran Kunhya2:19:08

So I personally wasn’t happy with the way assembly is taught in books and online, ’cause it’s very grammar-focused, and you don’t, in general, learn a language from learning the grammar and the structure.

所以我个人对书本和网上的汇编教学方式并不满意,因为它非常注重语法,而且一般来说,你不会通过学习语法和结构来学习语言。

2:19:08 You learn a language by asking someone what their name is, and you start from there, and you go and solve real problems that you have when you want to communicate.

你通过询问某人的名字来学习一门语言,你从那里开始,当你想要交流时,你去解决你遇到的实际问题。

You don’t learn sentence structure, and this is the interrogative and the adverb, and all the assembly books seem to be doing like that, going through every instruction, even ones that aren’t really relevant, explaining what they all do and how they…

你不学习句子结构,这是疑问句和副词,所有的汇编书似乎都是这样做的,通过每一条指令,即使是那些并不真正相关的指令,解释它们都做什么以及它们如何......

It actually doesn’t really change much.

实际上并没有太大改变。

2:19:43 So, and the other problem that we have in our community is assembly is taught sort of hand to hand, like person to person, like blacksmithing one by one.

因此,我们社区中存在的另一个问题是,装配是一种手工教学,就像一个人对一个人,就像一个一个的铁匠一样。

That’s the only logical sort of analogy, and that doesn’t really scale online.

这是唯一合乎逻辑的类比,而且在网上并不能真正推广。

It doesn’t do other things.

它不做其他事情。

So I’ve started a set of assembly lessons in the way it’s done in FFmpeg, which is a little bit different to the way assembly in general for…

因此,我以 FFmpeg 中的方式开始了一系列汇编课程,这与一般的汇编方式有点不同……

I don’t know.

我不知道。

I’m trying to think the other good big use case of assembly is in embedded devices, in really low power, cheap devices, and that’s completely different to what we’re doing here.

我试图认为组装的另一个良好的大用例是在嵌入式设备中,在真正低功耗、廉价的设备中,这与我们在这里所做的完全不同。

I think it would be good if you could highlight the requirements, which are quite simple.

我认为如果您能强调一下要求就好了,这些要求很简单。

It’s high school mathematics and C.

这是高中数学和C。

2:20:23 And actually not even C, really, really it’s pointers.

实际上甚至连 C 都不是,真的,真的是指针。

To emphasize, yes, we’ve talked about how brilliant this stuff is, but high schoolers like Daniel Kang have written assembly in FFmpeg.

需要强调的是,是的,我们已经讨论过这个东西有多么出色,但是像 Daniel Kang 这样的高中生已经用 FFmpeg 编写了汇编。

I think there’s been contributions because of these lessons.

我认为这些教训让人们做出了贡献。

So it’s really about trying to get this dying art to continue, because we’ve shown it’s possible with dav1d to produce something amazing.

所以这实际上是为了让这种垂死的艺术继续下去,因为我们已经证明了 dav1d 可以创造出令人惊叹的东西。

There’s still a lot of codecs in FFmpeg that are only maybe partially assembly optimized.

FFmpeg 中仍然有很多编解码器可能只进行了部分汇编优化。

And so it really starts with basics and continues, explains a lot of the jargon, a lot of the syntax.

因此,它确实从基础知识开始,然后继续解释了很多术语和语法。

It doesn’t really try and explain to you, you know, interrupt handlers and interrupt instructions and all of these different jump targets; actually makes this really vector focused.

它并没有真正尝试向您解释,您知道,中断处理程序和中断指令以及所有这些不同的跳转目标;实际上使这真正成为矢量焦点。

Lex Fridman2:21:08

And describes all kinds of registers, general purpose registers, vector registers; really nice examples.

并介绍了各类寄存器、通用寄存器、向量寄存器;非常好的例子。

2:21:08 Oh, this is cool.

哦,这很酷。

Kieran Kunhya2:21:18

It’s a classic, yeah, it’s a classic example of FFmpeg.

这是一个经典,是的,这是一个 FFmpeg 的经典例子。

2:21:18 But some of this assembly language is really beautiful, and I think it’s beautiful because it’s kind of like flying a Spitfire.

但其中一些汇编语言确实很漂亮,我认为它很漂亮,因为它有点像驾驶喷火战斗机。

It’s really aviation at its purest, but also pushing the aircraft beyond what the designer thought was possible.

这确实是最纯粹的航空,但也使飞机超越了设计师的想象。

So we’re abusing, for example, sometimes cryptography instructions to do certain things, and there’s a level of beauty and art where it’s really you and the processor.

因此,例如,我们有时会滥用密码学指令来执行某些操作,而实际上是您和处理器,这具有一定程度的美感和艺术性。

2:21:46 There’s nothing in between.

两者之间没有什么。

It’s you and the joystick of the cockpit, and you move that joystick, and it’s physically connected to the ailerons, and you can push that plane beyond what it can normally do, and there’s a level of, yeah, beauty and amazingness to go that.

是你和驾驶舱的操纵杆,你移动操纵杆,它与副翼物理连接,你可以推动飞机超越它通常的能力,这是一种,是的,美丽和令人惊叹的水平。

But I don’t think the sort of person-by-person assembly that is…

但我不认为那种面对面的集会是……

someone taught me, and I’ve taught multiple people, is gonna work long run just because of the particular flavor and the way that we do it.

有人教过我,而且我也教过很多人,因为特殊的味道和我们的做法,它会长期有效。

Lex Fridman2:22:16

It’s literally no, I should…

确实不行,我应该……

2:22:16 I was gonna say wizards handing it down.

我想说的是巫师把它传下来。

I realize I look like a wizard- …

我发现我看起来像个巫师——……

wearing this hat.

戴着这顶帽子。

But you’re basically just like the sages, the wise sages handing- …

但你基本上就像圣人一样,明智的圣人传递——……

down the craft.

降低工艺。

Can I ask you about LLMs?

我可以向您询问LLM吗?

Like- …

就像——……

can they help?

他们能帮忙吗?

Kieran Kunhya2:22:32

They had more of an understanding than I expected, but they are still…

他们的理解比我想象的要多,但他们仍然……

2:22:32 I’ve asked it questions, and it still goes and starts not hallucinating, but making modifications, and then I go, “Is it bit exact?” “No.” “Fix it.” And then it just goes and does the same thing, and it’s going, it…

我问过它问题,它仍然继续,开始不是产生幻觉,而是进行修改,然后我说,“它有点准确吗?” “不。” “修好它。”然后它就继续做同样的事情,然后它就……

There isn’t the corpus of information like Stack Overflow to work on.

没有像 Stack Overflow 这样的信息库可供处理。

Jean-Baptiste Kempf2:22:52

There is not enough data to train on.

没有足够的数据可供训练。

2:22:52 And this is the biggest issue.

这是最大的问题。

I started my career actually doing some assembly for Itanium, right?

我的职业生涯始于为安腾进行一些组装,对吗?

So the Itanium is a dead processor type, which was done by Intel and HP a long time ago when they wanted to do 64 bits.

所以安腾是一种死处理器类型,这是英特尔和惠普很久以前想做64位时就做的。

Well, they lost, and then we got AMD, who did it, AMD64, which became x86-64.

好吧,他们输了,然后我们得到了 AMD,AMD64 做到了,它变成了 x86-64。

But Itanium was extremely interesting in the sense that those were processors who had a ton of computing power to do floats, FMAs, which is similar to what we need now for LLMs, right?

但安腾非常有趣,因为这些处理器拥有大量计算能力来执行浮点运算、FMA,这类似于我们现在需要的法学硕士,对吧?

And you could pack three operations per line that could be loaded.

您可以在每行打包三个可加载的操作。

So basically, you had an output of basically six billion operations per second, but the memory bus only allowed 1.5, right?

所以基本上,每秒的输出基本上是 60 亿次操作,但内存总线只允许 1.5 次,对吧?

2:23:47 So your CPU was four times faster, so you had to do crazy things to pack things in memory or reuse the registers, and those type of semantics, no language could do that, right?

所以你的 CPU 速度快了四倍,所以你必须做一些疯狂的事情来将东西打包到内存中或重用寄存器,而这些类型的语义,没有语言可以做到这一点,对吗?

So like I have the Itanium programming book because Intel did amazing books, but that’s exactly what Kieran says.

就像我有安腾编程书一样,因为英特尔写了很棒的书,但这正是基兰所说的。

If you don’t know what you’re going to do, it’s impossible to read, right?

如果你不知道自己要做什么,就不可能阅读,对吧?

It’s a ton of jargon and so on.

这是大量的行话等等。

While those lessons are amazing because they are targeted to a real problem, and you can do it yourself.

虽然这些课程很棒,因为它们针对的是实际问题,而且您可以自己做。

Kieran Kunhya2:24:21

And people have.

人们已经做到了。

2:24:21 People have.

人们有。

There are patches, and they said, “Oh, I studied your lessons, and here’s my first changes.”

有补丁,他们说:“哦,我学习了你的课程,这是我的第一个更改。”

Lex Fridman2:24:26

That’s amazing.

太棒了。

Jean-Baptiste Kempf2:24:27

And part of that in the lessons is a framework called x86inc, written by Loren when he was working on x264, and it allows you to do more things about that to create a type of like not caring too much about different calling conventions.

课程中的一部分是一个名为 x86inc 的框架,由 Loren 在研究 x264 时编写,它允许您对此做更多事情,以创建一种不太关心不同调用约定的类型。

2:24:27 And we had a lot of students who gave code to x264 using that a long time ago, right?

我们有很多学生很久以前就使用它向 x264 提供了代码,对吧?

So it’s really doable, and I believe it’s necessary to understand assembly language, even if you don’t do it much, to understand what’s going on inside your computer, and that will make you a better programmer.

所以它确实是可行的,而且我相信有必要了解汇编语言,即使你做得不多,了解计算机内部发生的事情,这将使你成为一个更好的程序员。

And I assure you that because doing that, you will understand some of the architecture of the memory inside your computer, right?

我向你保证,因为这样做,你会了解计算机内部内存的一些架构,对吧?

2:25:14 Understanding registers, L1, L2, L3, RAM, SSDs, disk, and so on, which are very important because then you have a good programming culture that will make you a better programmer.

了解寄存器、L1、L2、L3、RAM、SSD、磁盘等,这些非常重要,因为这样你就有了良好的编程文化,这将使你成为更好的程序员。

Lex Fridman2:25:27

What do you think about the Rust programming language?

您对 Rust 编程语言有何看法?

2:25:27 ‘Cause that’s a bit of a meme.

因为这有点像模因。

Jean-Baptiste Kempf2:25:31

We have very different opinions with Kieran.

我们和基兰有非常不同的看法。

Kieran Kunhya2:25:33

I think it’s valuable what they’re doing in terms of memory safety as a concept.

我认为他们在内存安全这一概念方面所做的事情很有价值。

Lex Fridman2:25:37

Can it achieve some of the speed up that assembly achieves?

它能达到装配所达到的一些速度吗?

Kieran Kunhya2:25:42

Oh, not assembly by hand, no.

哦,不是手工组装,不。

2:25:42 I think that that’s a given.

我认为这是理所当然的。

C potentially, but I see it very…

C 可能,但我认为它非常......

It has a very big Esperanto vibe about it.

它有一种非常强烈的世界语氛围。

It’s like we’re gonna solve this, and we’re doing this in a particular way.

就好像我们要解决这个问题,而且我们正在以一种特定的方式来做这件事。

Lex Fridman2:25:54

Meaning it’s a bit too utopian?

这是否意味着它有点太乌托邦了?

Kieran Kunhya2:25:56

There’s a lot of focus on the self-importance rather than solving real-world problems.

人们更多地关注自我重要性,而不是解决现实世界的问题。

2:25:56 It reminds me of the Sinclair C5.

它让我想起了辛克莱C5。

Sir Clive Sinclair of Sinclair Computers built a car, and he said, “Oh, everyone will be traveling around in one of these electric cars.” And it was…

辛克莱计算机公司的克莱夫·辛克莱爵士制造了一辆汽车,他说:“哦,每个人都将乘坐其中一辆电动汽车出行。”那是……

Rust reminds me of that, where I think the community doesn’t quite understand that in order to get people to move, you have to build something that’s as good as, if not better than what you have now.

Rust 让我想起了这一点,我认为社区不太明白,为了让人们搬家,你必须构建与现在一样好的东西,甚至比现在更好的东西。

Yes, people are doing Rust rewrites, but if they only do 85, 90% of the feature set of what we need, like things like coreutils, that last 1% takes 99% of the time.

是的,人们正在做 Rust 重写,但如果他们只做我们需要的 85%、90% 的功能集,比如 coreutils 之类的东西,那么最后 1% 就会占用 99% 的时间。

2:26:39 To use Elon’s famous quote, “Prototypes are easy.” Like this kind of stuff is easy.

用 Elon 的名言来说,“原型很简单。”就像这种事情很容易。

But this, to get a real electric car, you have to make a car as good as, if not better than what we have now, and Rust isn’t in that stage yet.

但是,要获得一辆真正的电动汽车,你必须制造出一辆与我们现在拥有的汽车一样好,甚至更好的汽车,而 Rust 还没有达到那个阶段。

I think we’d– I don’t think anyone would object to seeing Rust code in FFmpeg, but it needs to work as well and support the same unit testing as everything else.

我认为我们——我认为没有人会反对在 FFmpeg 中看到 Rust 代码,但它需要同样工作并支持与其他所有东西相同的单元测试。

It needs to be flawless.

它必须是完美无缺的。

It can’t just randomly break.

它不能随意破坏。

They can’t just randomly break ABI when they want to.

他们不能随意破坏 ABI。

It needs to have, I think, more– I think it still has only one compiler implementation.

我认为它需要更多——我认为它仍然只有一种编译器实现。

So it’s got to be as good as, if not better, and saying, “Hey, here’s my utopia of memory safety,” isn’t enough, even though we probably all agree that that’s the goal.

因此,即使不是更好,也必须一样好,并且说“嘿,这是我的内存安全乌托邦”是不够的,尽管我们可能都同意这就是目标。

Jean-Baptiste Kempf2:27:24

So I’ve done a ton of Rust, and the two major topics I had was adding Rust modules inside VLC.

所以我做了很多 Rust 的工作,我的两个主要主题是在 VLC 中添加 Rust 模块。

2:27:24 One of the reasons VLC got popular and which was one of the main architectural decision, is that VLC is a very small core and a ton of modules, right?

VLC 流行的原因之一,也是主要的架构决策之一,是 VLC 是一个非常小的核心和大量的模块,对吧?

And so you can write modules in C, in C++, in Objective-C, and anything that is basically interoperable with C.

因此,您可以使用 C、C++、Objective-C 以及任何基本上可与 C 互操作的内容来编写模块。

And so we did some Rust modules, and so I have experience on that, and I wrote some of it.

所以我们做了一些 Rust 模块,所以我有这方面的经验,并且我写了其中的一些。

And also, like, my new startup called Kyber, is an open source project mainly done in Rust.

而且,就像我的新创业公司 Kyber 一样,它是一个主要用 Rust 完成的开源项目。

What Rust is extremely good in, in the sense that it’s a better C++ that cares about memory and allows you to do things about memory ownership that no one else can do so far.

Rust 非常擅长什么,从某种意义上说,它是一种更好的 C++,它关心内存,并允许您做迄今为止其他人无法做的有关内存所有权的事情。

2:28:19 However, it’s great when you start a new project from scratch, and you do everything in Rust.

然而,当您从头开始一个新项目并且所有事情都用 Rust 完成时,这非常棒。

But it’s very not good when you interop with existing part.

但是当你与现有的部分进行互操作时就非常不好了。

And some part of the Rust community believes that they need to rewrite everything, and everything will be better with Rust.

Rust 社区的一些人认为他们需要重写一切,有了 Rust 一切都会变得更好。

And the answer is like, no.

答案是,不。

Like, I’m almost always, in all my years of being engineer, manager, CTO of startup and so on, don’t rewrite, right?

就像,在我担任工程师、经理、初创公司首席技术官等这些年里,我几乎总是不重写,对吧?

Lex Fridman2:28:47

Is that– That’s the, that’s the initial instinct for a lot of people when they show up to a code base, probably before LLMs, is like…

就是这样——就是这样,这是很多人在接触代码库时的最初本能,可能是在法学硕士之前,就像……

2:28:47 probably because they don’t understand the, the, the wisdom of the way things have been done in the past.

可能是因为他们不了解过去做事方式的智慧。

They say, “Well, we need to rewrite it.” Hence why there’s a thousand JavaScript frameworks.

他们说:“好吧,我们需要重写它。”这就是为什么有上千个 JavaScript 框架的原因。

Jean-Baptiste Kempf2:29:06

But the reason is the following, and this is very important to understand.

但原因如下,理解这一点非常重要。

2:29:06 It is an order of magnitude easier to write code than read code.

编写代码比阅读代码容易一个数量级。

And you see that also with LLM.

你在法学硕士中也能看到这一点。

They can write code, but analyzing is a lot- …

他们可以编写代码,但分析却很多——……

more difficult.

更困难。

And so when you arrive to a very complex piece of code, right?

因此,当您遇到一段非常复杂的代码时,对吗?

You don’t understand it, right?

你不明白,对吧?

Because it’s so much more effort to understand the code from someone else because you don’t have the thought process.

因为你没有思考过程,所以要理解别人的代码需要付出更多的努力。

And often I joke about some languages, mostly Perl for example, which has very complex syntax.

我经常拿一些语言开玩笑,主要是 Perl,它的语法非常复杂。

And imagine I am at my maximum intellectual efficiency in programming, right?

想象一下我在编程方面处于最高智力效率,对吧?

And I write the best code ever.

我写出了有史以来最好的代码。

I will not be able to understand it myself six months later, right?

六个月后我自己也无法理解吧?

2:29:58 Because reading code is more difficult.

因为阅读代码比较困难。

So very often you arrive, you don’t understand all the wisdom, all the business logic, the reasons that were done that is maybe not documented.

很多时候,当你到达时,你并不理解所有的智慧、所有的业务逻辑、以及可能没有记录的所做的原因。

And you say, “Well, I’m going to write it.” And the thing is, no, you don’t, right?

你说:“好吧,我要写它。”问题是,不,你不知道,对吧?

Because that’s, as Kieran said, right?

因为正如基兰所说,对吧?

I’m going to rewrite coreutils in Rust.

我将用 Rust 重写 coreutils。

And then, of course, you arrive very quickly at eighty percent then ninety percent, takes a bit more time, and then you got the last ones, right?

然后,当然,你很快就达到了百分之八十,然后是百分之九十,需要更多的时间,然后你得到了最后的,对吧?

On the other side, right?

在另一边,对吗?

So for new projects, it’s great.

所以对于新项目来说,这非常棒。

Everything related to parsing files networking because of the memory checker, boundary checker, it’s amazing, and there is nothing else.

与解析文件网络相关的一切都因为内存检查器、边界检查器,太神奇了,没有别的了。

2:30:40 To answer a bit differently for us, imagine I take a piece of software like dav1d or x264, right?

为了给我们一个稍微不同的答案,想象一下我使用 dav1d 或 x264 这样的软件,对吧?

Which has a ton of runtime in assembly, right?

其中在汇编中有大量的运行时间,对吧?

I rewrite the C part in Rust, right?

我用 Rust 重写了 C 部分,对吗?

So it’s more secure.

所以更安全。

Yes.

是的。

But then you arrive into the assembly, and you can jump anywhere in the memory because we are doing handwritten assembly.

但是当你到达汇编时,你可以跳转到内存中的任何地方,因为我们正在进行手写汇编。

So even if I rewrite the C part in Rust, for security reason, you break all the security when you write handwritten assembly because we can jump anywhere.

因此,即使我在 Rust 中重写了 C 部分,出于安全原因,当您编写手写程序集时,您也会破坏所有安全性,因为我们可以跳转到任何地方。

So in my opinion, we need to do something that is secure assembly, right?

所以在我看来,我们需要做一些安全组装的事情,对吧?

2:31:22 So which is compile time, check the assembly, which is similar to the checkasm projects that we’re doing on dav1d and x264 with VideoLAN, is to start instrumenting your assembly at compile time to check that it’s not jumping anywhere in the memory.

所以这是编译时,检查程序集,这类似于我们在 dav1d 和 x264 上使用 VideoLAN 进行的 checkasm 项目,就是在编译时开始检测程序集,以检查它是否没有跳转到内存中的任何位置。

Because else you might rewrite a part of C in Rust, but if you want to have the same performances, you’re going to have inline assembly, and so you destroy your whole security model.

因为否则你可能会用 Rust 重写 C 的一部分,但如果你想拥有相同的性能,你将需要内联汇编,所以你会破坏你的整个安全模型。

So that’s a bit what I think about Rust.

这就是我对 Rust 的一些看法。

Kieran Kunhya2:31:51

No, I just wanna…

不,我只是想……

2:31:51 I would say on a personal level, I’m so in awe about assembly.

我想说,就个人而言,我对装配非常敬畏。

I actually— Once in a…

我实际上——有一次……

It never gets old, the speed improvements to show sixty-two x.

它永远不会过时,速度提高到了六十二倍。

So there are months, on a personal level, I run our internal test suite at work and just see I’m still in awe at the gains we have.

因此,在个人层面上,有几个月的时间,我在工作中运行我们的内部测试套件,发现我仍然对我们所取得的成果感到敬畏。

Lex Fridman2:32:10

Well, there’s a source of joy and happiness with programming for different reasons.

嗯,出于不同的原因,编程是快乐和幸福的源泉。

2:32:10 But I think one of the greatest happiness is in the optimization of code.

但我认为最大的幸福之一就是代码的优化。

And it sounds like you’re, like, at the cutting edge of that.

听起来你正处于这方面的最前沿。

Kieran Kunhya2:32:24

I was like, “Whoa, that was cool.”

我当时想,“哇,那太酷了。”

Jean-Baptiste Kempf2:32:25

And in the community, I want to speak about two people who are wizards of assembly, right?

在社区中,我想谈谈两个装配奇才的人,对吗?

2:32:25 The two of them are actually working living in north of Europe, Sweden and Finland.

他们两人实际上在北欧瑞典和芬兰工作生活。

And Henrik Gramner knows so much about Intel x86 assembly that when we ask questions at Intel about things, they tell, like, “Why are you asking us, Intel?

Henrik Gramner 对 Intel x86 组装非常了解,因此当我们向 Intel 询问相关问题时,他们会说:“Intel,你为什么要问我们?

You have Henrik.

你有亨里克。

Henrik knows better.” He knows all the cycles of almost all the SIMD instruction by all the CPU generation.

亨里克更清楚。”他知道所有CPU代几乎所有SIMD指令的所有周期。

“Oh, yes, this is a P4, this is a Nehalem, this is a Core 2,” et cetera.

“哦,是的,这是 P4、这是 Nehalem、这是 Core 2,”等等。

That person is, like, the best person on assembly in the world.

那个人就像是世界上最好的集会人员。

2:33:12 And he’s the nicest person that you’ve seen, like, very g- He arrives, you don’t see he’s amazing.

他是你见过的最好的人,非常好——他来了,你看不到他有多棒。

And the other one is called Martin, Martin Storsjö, and he’s– they’re doing mostly the same on Arm, right?

另一位名叫 Martin,Martin Storsjö,他——他们在 Arm 上做的事情基本相同,对吗?

So Neon, right?

那么霓虹灯,对吧?

And iPhones and Androids and so on.

还有 iPhone 和 Android 等。

And he codes in assembly on his phone, editing it with the crappy keyboard, like virtual keyboard you have while watching his kids play in the playground, right?

他在手机上用汇编语言编写代码,用蹩脚的键盘进行编辑,就像你在看他的孩子在操场上玩耍时使用的虚拟键盘一样,对吧?

Like, this is just like wizard level.

就像,这就像巫师级别一样。

So those two people are like-

那么这两个人就像——

Lex Fridman2:33:55

Yes.

是的。

2:33:55 So a part, when you’re programming assembly at that high level, a part of that is knowing the architecture that you’re programming on.

因此,当您在高级别上进行汇编编程时,其中一部分就是了解您正在编程的架构。

So x86 and-

所以 x86 和 -

Jean-Baptiste Kempf2:34:03

On Arm in particular, yes.

尤其是在 Arm 上,是的。

Lex Fridman2:34:04

…

……

2:34:04 Arm in particular.

特别是手臂。

But x86, I mean, these are complicated architectures, right?

但是 x86,我的意思是,这些都是复杂的架构,对吗?

Kieran Kunhya2:34:09

Yeah.

是的。

2:34:09 But Arm in some ways is more com…

但 Arm 在某些方面更通用……

x86 with out of order execution is not so bad.

乱序执行的 x86 还不错。

Arm, you really need to understand all the different generations of Arm processor because they’re all different.

Arm,您确实需要了解所有不同代的 Arm 处理器,因为它们都是不同的。

There’s A72, …

有A72,...

…

……

Et cetera, et cetera.

等等等等。

And there’s the Apple variant, there’s this variant, there’s that, and you need to write code that works efficiently on all of them.

还有 Apple 变体、这个变体、那个变体,您需要编写能够在所有这些变体上高效运行的代码。

x86, well, broadly speaking, you have Intel, AMD, and you have sub-variants, but generally speaking, there’s…

x86,嗯,从广义上讲,有 Intel、AMD,还有子变体,但一般来说,有……

Something fast is gonna remain fast on all of the variants, whereas in Arm it’s a completely much more complicated ballgame.

快速的东西在所有变体上都会保持快速,而在 Arm 中,这是一个完全复杂得多的球赛。

Lex Fridman2:34:43

We’re taking a nonlinear journey through history here, but we’re talking about Michael Niedermayer.

我们正在这里进行一次非线性的历史之旅,但我们谈论的是迈克尔·尼德迈尔。

2:34:43 And I wanted to ask about this.

我想问一下这个问题。

For a time there was a split in FFmpeg and Libav.

FFmpeg 和 Libav 一度出现分歧。

Jean-Baptiste Kempf2:35:00

Yes.

是的。

2:35:00 So in open source projects sometimes you disagree, right?

所以在开源项目中有时你会不同意,对吧?

Lex Fridman2:35:10

You have such a nice way of putting it, yeah.

你的表达方式真好,是的。

Jean-Baptiste Kempf2:35:12

And the good thing is because of the license, you’re allowed to basically do your own, right?

好处是因为有了许可证,你基本上可以自己做,对吧?

2:35:12 And this is normal, and this has happened all the time, right?

这很正常,而且这种情况一直在发生,对吗?

At a point there was a GCC at the time of GCC 2 and EGCS which became then GCC 3, right?

在 GCC 2 和 EGCS 的时代曾有过一个 GCC,后来成为了 GCC 3,对吗?

There is what we told KHTML with WebKit, with Blink.

这就是我们用 WebKit 和 Blink 告诉 KHTML 的内容。

It is a same process.

这是一个相同的过程。

And also, like when I want to do a new feature today in VLC, I fork, I do my thing on my own, and then I merge back to the community.

而且,就像今天我想在 VLC 中做一个新功能一样,我分叉,我自己做我的事情,然后我合并回社区。

So there was a split in the open source community on FFmpeg, which become Libav and FFmpeg.

于是FFmpeg的开源社区就出现了分裂,变成了Libav和FFmpeg。

And after a few years, well, the community merged back and people moved on.

几年后,社区重新合并,人们继续前进。

It’s a bit of drama that is normal in open source community, but forks are even…

这在开源社区中很常见,但分叉甚至……

They’re important because they change the status quo of a community.

它们很重要,因为它们改变了社区的现状。

2:36:04 Not talking about FFmpeg and Libav here, but the GCC fork made GCC a ton better because some people wanted to change the architecture fundamentally to make it faster.

这里不讨论 FFmpeg 和 Libav,但 GCC 分支使 GCC 变得更好,因为有些人想从根本上改变架构以使其更快。

And of course, it’s always question of people and so on, but in the end you realize that FFmpeg today is better than it was before the fork.

当然,这总是人的问题等等,但最终你会意识到今天的 FFmpeg 比分叉之前更好。

And now, well, we’re back all together, right?

现在,好吧,我们又在一起了,对吧?

And I spent a lot of time—and Kieran can say—in the community.

基兰可以说,我在社区中度过了很多时间。

It’s not often, to be honest, very well explained because a ton of the reasons are not very public.

老实说,这种情况并不经常得到很好的解释,因为很多原因都不是很公开。

But I think that’s, that’s normal and that’s good.

但我认为这很正常,而且很好。

Lex Fridman2:36:49

Yeah.

是的。

2:36:49 I mean, you’re making it sound really nice, but there, there is battle, there’s pretty heated battles inside open source projects.

我的意思是,你说得很好,但是,开源项目内部存在着相当激烈的战斗。

I mean, it is a very passionate community and you’re kind of in a distributed way have to define the direction of things.

我的意思是,这是一个非常热情的社区,你必须以一种分布式的方式来定义事物的方向。

2:37:01 So here looking at Perplexity, “FFmpeg and Libav split in 2011 mainly over project governance, leadership style, and development processes, not because of a fundamental technical disagreement.

因此,看看 Perplexity,“FFmpeg 和 Libav 在 2011 年主要是在项目治理、领导风格和开发流程方面分歧,而不是因为根本的技术分歧。

FFmpeg effectively absorbed Libav’s work while Libav withered and most distributions and developers moved back to FFmpeg.” Yeah, that was a weird experience ’cause, you know, I’m a Linux user, so, you know, whether it’s Ubuntu and so on, all of a sudden, I think for a little bit, Ubuntu, I feel like, am I remembering correctly, switched to Libav and-

FFmpeg 有效地吸收了 Libav 的工作,而 Libav 逐渐衰落,大多数发行版和开发人员又回到了 FFmpeg。”是的,那是一次奇怪的经历,因为,你知道,我是一个 Linux 用户,所以,你知道,无论是 Ubuntu 等等,突然间,我想了一下,Ubuntu,我感觉,我没记错的话,切换到 Libav 和 -

Jean-Baptiste Kempf2:37:39

12, 14, something like that.

12、14,类似的事情。

2:37:39 Yes.

是的。

Something like that.

类似的事情。

Lex Fridman2:37:41

And then they switched back to FFmpeg.

然后他们又切换回 FFmpeg。

Jean-Baptiste Kempf2:37:43

They switched back, yeah.

他们换回来了,是的。

Lex Fridman2:37:43

I was like, “What is happening?” So on the sort of…

我当时想,“发生了什么事?”所以关于……

2:37:43 you get to feel the ripple effects of the different internal debates that are happening.

你会感受到正在发生的不同内部辩论的连锁反应。

Kieran Kunhya2:37:53

To be fair, on Apple, when you type GCC, you get Clang.

公平地说,在 Apple 上,当您输入 GCC 时,您会得到 Clang。

2:37:53 Like they, they did something like that as well, so.

像他们一样,他们也做了类似的事情,所以。

Jean-Baptiste Kempf2:37:58

Yeah.

是的。

2:37:58 So to me it’s like the fork was like heated drama, but most of the development from Libav was merged back into FFmpeg, right?

所以对我来说,分叉就像一场激烈的戏剧,但 Libav 的大部分开发都被合并回 FFmpeg,对吧?

So de facto FFmpeg got a superset around Libav, and so that gave the user, because in the end we work for the users, a larger set of features and a ton of things that were discussed.

事实上,FFmpeg 围绕 Libav 建立了一个超集,因此为用户提供了更多的功能和大量讨论的内容,因为最终我们为用户工作。

For example, the debate on reviews, on how we push, are something that now is completely settled in FFmpeg and is following what mostly what everyone in the community agrees, right?

例如,关于评论的争论,关于我们如何推动的争论,现在已经在 FFmpeg 中完全解决了,并且遵循的是社区中每个人都同意的内容,对吧?

So de facto, everyone who was active on Libav came back and worked on FFmpeg because the disagreements were fixed, and in the end, FFmpeg is stronger than it was before, right?

所以事实上,所有活跃在 Libav 上的人都回来了,并在 FFmpeg 上工作,因为分歧得到了解决,最终 FFmpeg 比以前更强了,对吗?

And- …

还有——

I know people love drama, but-

我知道人们喜欢戏剧,但是-

Lex Fridman2:38:52

Well, my main concern, I understand, and I think looking at the long history, it’s all for the good.

嗯,我主要关心的是,我理解,而且我认为从悠久的历史来看,这都是好的。

2:38:52 But I do…

但我确实……

I am concerned because there’s so few humans that are critical to the success of open source projects that I have seen it be a psychological toll on folks and, you know, sometimes leads to burnout.

我很担心,因为对开源项目的成功至关重要的人太少了,我发现这对人们造成了心理伤害,你知道,有时会导致倦怠。

So you have these incredible people that are at the core of open source projects.

因此,这些令人难以置信的人是开源项目的核心。

There is a moment that happens ’cause, like, what is the motivation of doing it?

有时候会发生这样的时刻,因为,这样做的动机是什么?

Ultimately, it’s because you’re passionate about it and it makes you happy.

归根结底,这是因为你对它充满热情,它让你快乐。

Then at a certain point, you wake up and it’s like, “This has been a bit too much heat from the drama.” So, like, at the project level, the project continues and often flourishes.

然后在某个时刻,你醒来后会想,“这出戏有点太热了。”因此,就像在项目层面一样,项目会继续并且经常蓬勃发展。

But sometimes there’s these individual humans that are just like-

但有时有些人就像——

Jean-Baptiste Kempf2:39:44

But-

但是——

Lex Fridman2:39:44

…

……

2:39:44 I’ve had enough.

我已经受够了。

Jean-Baptiste Kempf2:39:45

Yeah, but it’s not just about forks, right?

是的,但这不仅仅是叉子的问题,对吧?

2:39:45 So it’s a very- …

所以这是一个非常…

what you are referring to is one of the most challenging and most interesting part of open source today is maintainers burnout, right?

您所指的是当今开源中最具挑战性和最有趣的部分之一是维护人员的倦怠,对吧?

And AI is a problem because of that.

因此,人工智能是一个问题。

And Daniel Stenberg, who is the maintainer of curl, who’s probably one of the best promoters of open source in the world—he’s, by the way, a member of the European Open Source Academy with me, so I’m very, like, humbled to be in the same community as him, right?

Daniel Stenberg,curl 的维护者,可能是世界上最好的开源推动者之一——顺便说一句,他和我都是欧洲开源学院的成员,所以我很荣幸能和他在同一个社区,对吧?

He’s against what he calls AI slop, right?

他反对他所说的人工智能废话,对吗?

Because it gives a ton of fake reports or- …

因为它提供了大量虚假报告或-...

bad reports, bad patches, and then a lot of maintainers have a lot of burden to maintain the software.

糟糕的报告,糟糕的补丁,然后很多维护人员就有很大的负担来维护软件。

And this is straining the mind of open source developers much more than forks.

这比分叉更让开源开发者感到紧张。

2:40:42 And for example, the XZ fiasco was because there was one guy maintaining it, and he got basically hammered by two attackers who were asking him questions nonstop at weird times at night to block him, and at some point he got fed up and said, “Okay, I can’t do that,” and gave the commit access to the attacker.

例如,XZ 惨败是因为有一个人在维护它,而他基本上被两名攻击者殴打,这两名攻击者在晚上奇怪的时间不停地问他问题以阻止他,在某些时候他厌倦了并说,“好吧,我不能这样做”,并将提交访问权限授予了攻击者。

So burnout in the open source community is something that exists, but mostly it’s about maintaining things, right?

因此,开源社区中的倦怠是存在的,但主要是与维护事物有关,对吗?

Lex Fridman2:41:11

No, for sure.

不,当然。

2:41:11 But I wonder how do we help that, ’cause those people are so important.

但我想知道我们如何帮助做到这一点,因为这些人是如此重要。

Jean-Baptiste Kempf2:41:16

Mm

毫米

Lex Fridman2:41:16

…

……

2:41:16 the human beings are so important to the core of these pro- projects.

人类对于这些支持项目的核心非常重要。

Jean-Baptiste Kempf2:41:19

So, so for example, now I am maintaining a ton of multimedia and non-multimedia libraries- …as maintainer because the maintainers got fed up, right?

举例来说,现在我正在维护大量的多媒体和非多媒体库——……作为维护者,因为维护者已经厌倦了,对吧?

2:41:19 Some on VideoLAN, some outside of VideoLAN, because it’s…

有些在 VideoLAN 上,有些在 VideoLAN 之外,因为它......

sometimes you need tough skin, right?

有时你需要坚韧的皮肤,对吗?

Because you get, like, it’s not really attacks, but “oh, this is not working, this is not working,” and you feel it personally.

因为你会觉得,这并不是真正的攻击,而是“哦,这不起作用,这不起作用”,你个人会感觉到。

And this is also why resources or the Google fiasco was a problem, right?

这也是资源或谷歌惨败成为问题的原因,对吧?

They don’t realize that in the end you have, you know, it’s like the same graph where you see, like, everything and it’s just like one random open source project that is maintaining the whole—

他们没有意识到,最终你会得到,你知道,它就像你看到的同一张图表,就像,一切,它就像一个随机的开源项目,正在维护整个——

Lex Fridman2:41:59

The Nebraska thing, yeah

内布拉斯加州的事情,是的

Jean-Baptiste Kempf2:42:00

…

……

2:42:00 internet.

互联网。

You see the one, right?

你看到那个了,对吧?

The-

该-

Lex Fridman2:42:01

Yeah, this is the meme.

是的,这就是模因。

2:42:01 I mean, it applies to, to a lot of open source projects.

我的意思是,它适用于很多开源项目。

Jean-Baptiste Kempf2:42:05

So many things, yeah.

很多事情,是的。

Lex Fridman2:42:05

But this is the all modern digital multimedia infrastructure, and then that thing at the very bottom that everything relies on is FFmpeg.

但这是所有现代数字多媒体基础设施,而一切所依赖的最底层就是 FFmpeg。

2:42:05 It’s true.

这是真的。

And then there’s usually, you know, a handful of folks that are maintaining that.

然后,你知道,通常有少数人在维护它。

Jean-Baptiste Kempf2:42:19

And FFmpeg or VLC, right, you have a community of 10, 15 core developers, are not the worst open source project.

而FFmpeg或者VLC,对吧,你有一个10、15个核心开发者的社区,都不是最差的开源项目。

2:42:19 XZ, which is even in more installations, is one person, right?

XZ,甚至更多的装置中,也是一个人,对吧?

There is one guy-

有一个人——

Lex Fridman2:42:32

LibXML is-

LibXML 是-

Jean-Baptiste Kempf2:42:34

Yeah, LibXML, right?

是的,LibXML,对吧?

2:42:34 There was a big stop.

有一个大站。

No one is maintaining- …

没有人维护——……

LibXML anymore, which is like the only library that is able to parse XML everywhere.

不再是 LibXML,它就像是唯一能够在任何地方解析 XML 的库。

Lex Fridman2:42:41

All the crazy edge cases of XML under ridiculous circumstances, and they get attacked by security researchers because there’s one other crazy edge case that they haven’t thought of, and it’s like, yeah, but the body of knowledge to actually resolve that is massive.

XML 在荒谬的情况下出现的所有疯狂的边缘情况,他们都会受到安全研究人员的攻击,因为还有另一种他们没有想到的疯狂的边缘情况,就像,是的,但是真正解决这个问题的知识体系是巨大的。

Jean-Baptiste Kempf2:42:56

There is one guy maintaining all the time zones for everyone who is in the middle of, I think, was it Nebraska or-

有一个人为每个人维护所有时区,我想,是内布拉斯加州还是——

Lex Fridman2:43:02

Yeah, it could be, yeah.

是的,可能是,是的。

Jean-Baptiste Kempf2:43:02

…

……

2:43:02 South Dakota?

南达科他州?

Like, the mental health of the open source maintainers is something that large corporations don’t care or don’t see, right?

就像,开源维护者的心理健康是大公司不关心或看不到的,对吗?

It’s just like, “Oh, yeah, I’m just doing an open source report,” and so on.

就像,“哦,是的,我只是在做一份开源报告”,等等。

Lex Fridman2:43:16

Mm.

毫米。

2:43:16 Some of it is financial, but people should definitely support open source financially- …

其中一些是财务方面的,但人们绝对应该在财务上支持开源——……

all, all across the board.

全部,全面。

But some of it is also, like, spiritual on a basic human level.

但其中一些也是人类基本层面上的精神层面的。

There’s something that happens, like, with this image of F- FFmpeg and so much of the internet depending on it, where people almost, like, talk down to the folks who are carrying these projects forward and maintaining it.

发生了一些事情,比如 F-FFmpeg 的形象以及互联网上的大部分内容都依赖于它,人们几乎喜欢居高临下地与那些推进和维护这些项目的人交谈。

Jean-Baptiste Kempf2:43:40

In the security community, they certainly did.

在安全界,他们确实做到了。

2:43:40 That was one of the things I think that argument came out is there was a portion of the security community who’s like, “No, these guys write crap code.

我认为争论的焦点之一是,安全社区中有一部分人会说:“不,这些人写的是垃圾代码。

They need to fix their crap code.” I’m like, “No, no, no, no.

他们需要修复他们的垃圾代码。”我说:“不,不,不,不。

This is a guy’s hobby project.

这是一个男人的爱好项目。

You have a security bot that’s gone and found some AI-generated stuff.

你有一个安全机器人,它已经消失并发现了一些人工智能生成的东西。

That guy didn’t write crap code.” It’s just an edge case to the 99.99999 percentile he didn’t think about because it’s his hobby project decoding Star Wars games.”

那家伙没有写过垃圾代码。”这只是他没有考虑到的 99.99999 百分位数的一个边缘情况,因为这是他解码星球大战游戏的爱好项目。”

Lex Fridman2:44:10

Forget the hobby project aspect of it.

忘记它的爱好项目方面。

2:44:10 It’s just hard work, and it’s beautiful, and it’s like the right approach there is to celebrate people- …

这只是艰苦的工作,而且很美丽,这就像庆祝人们的正确方法——……

for doing incredible, incredible work.

做出令人难以置信的、令人难以置信的工作。

It’s, it’s just incredible that humans step up- …

人类的进步真是令人难以置信——……

not getting really paid at first or maybe ever, and then they’re doing it out of the love of it, and we need to, like, human civilization runs on people like that.

一开始或可能永远都没有得到真正的报酬,然后他们出于对它的热爱而这样做,我们需要,就像,人类文明依靠这样的人来运行。

We need to celebrate them.

我们需要庆祝他们。

Jean-Baptiste Kempf2:44:36

To give you an idea, I received death threats on VideoLAN, right?

给你一个想法,我在 VideoLAN 上收到了死亡威胁,对吗?

2:44:36 And, um-

而且,嗯-

Lex Fridman2:44:40

You mentioned that to me.

你向我提到过这一点。

2:44:40 Like, what, what is, what is behind that?

就像,什么,什么,这背后是什么?

Jean-Baptiste Kempf2:44:43

So that must be, what, 2009, 2010, right?

那一定是,什么,2009 年、2010 年,对吧?

2:44:43 Um, Apple is moving from PowerPC to Core Duo, um, that probably in 2006, and by 2009 or 2010, I decide that we are not going to do new versions of VLC for PowerPC.

嗯,苹果正在从 PowerPC 转向 Core Duo,嗯,大概是在 2006 年,到 2009 年或 2010 年,我决定我们不会为 PowerPC 开发新版本的 VLC。

At that time, like VLC, we were close to the number 1.0 release.

当时,和 VLC 一样,我们已经接近 1.0 版本了。

We were four of us, right?

我们是四个人,对吧?

Like, just like, “No, this is not possible.” So I receive a death threat with some powder in it, right?

就像,就像,“不,这是不可能的。”所以我收到了死亡威胁,里面有一些粉末,对吗?

It– Remember there was some- …

它——记住有一些——……

anthrax threats-

炭疽威胁——

2:45:17 …

……

at that time, right?

那时,对吗?

And it was because I had taken the decision to not maintain the PowerPC port anymore.

这是因为我决定不再维护 PowerPC 端口。

And of course, it wasn’t anthrax, of course.

当然,这当然不是炭疽病。

It was some type of flour and so on.

这是某种面粉等等。

But I received that as a, with a letter of like, “You, you piece of shit, you should die, PowerPC forever,” and so on.

但我收到的是一封类似这样的信:“你,你这狗屎,你应该死,PowerPC永远”,等等。

And it was 2009 or 2010, right?

那是 2009 年或 2010 年,对吗?

I was young.

我还年轻。

I was just like, “Why?

我当时想,“为什么?

What did I do?”, right?

我做了什么?”,对吗?

Lex Fridman2:45:47

Yeah, that can break your spirit.

是的,这会摧毁你的精神。

2:45:47 It’s like, why-

就像,为什么-

Jean-Baptiste Kempf2:45:49

My mother freaked out, right?

我妈妈吓坏了,对吗?

2:45:49 We had to go to see the police and so on.

我们不得不去见警察等等。

And now, like, I’m going to say that I’m quite happy that this happened at that time.

现在,我想说,我很高兴当时发生了这件事。

It forged me a lot, right?

它锻炼了我很多,对吧?

I am…

我是……

I can see, I can take a lot of hate on me.

我看得出来,我可以承受很多仇恨。

I’m okay with it, right?

我觉得没问题,对吧?

Lex Fridman2:46:07

It sucks that that’s part of reality, ’cause all the people that love VLC, all the people that love FFmpeg, like me, you know, I, I legitimately hundred- probably thousands of times in my life had a smile on my face because FFmpeg made me happy, period.

这是现实的一部分,这很糟糕,因为所有热爱 VLC 的人,所有热爱 FFmpeg 的人,就像我一样,你知道,我,我一生中理所当然地有数百次——可能数千次在我的脸上微笑,因为 FFmpeg 让我快乐,就这样。

2:46:07 And how many times did I get a chance to say that?

我有多少次机会这么说?

Zero.

零。

Until I realized there’s a Twitter account, and every once in a while I’m, like, messaging it.

直到我意识到有一个 Twitter 帐户,并且每隔一段时间我都会向它发送消息。

Jean-Baptiste Kempf2:46:35

One of the things I like on the Reddit meme about me, which I don’t like this meme for a lot of reasons, but…

我喜欢 Reddit 模因上关于我的一件事,但出于很多原因我不喜欢这个模因,但是……

2:46:35 And someone says, “Oh, JB is on Reddit,” which I am, right?

有人说,“哦,JB 在 Reddit 上”,我就是这样,对吧?

And I say, and say hello, right?

我说,打个招呼,对吗?

And then I got so many people who say, “Oh, thank you for VLC.” And, like, I take pictures, and then I share that to the Signal, to IRC.

然后很多人都说:“哦,谢谢 VLC。”我会拍照,然后将其分享给 Signal、IRC。

Yes, we use IRC on different-

是的,我们在不同的平台上使用 IRC——

Lex Fridman2:46:58

I saw as a quick tangent, you mentioned IRC is like Slack for old people.

我认为这是一个快速的切线,你提到 IRC 就像老年人的 Slack。

2:46:58 So you still use IRC?

那你还在用IRC吗?

Jean-Baptiste Kempf2:47:03

Of course.

当然。

Kieran Kunhya2:47:03

Yeah.

是的。

2:47:03 I have it on my phone as well.

我的手机上也有它。

Jean-Baptiste Kempf2:47:05

Of course.

当然。

Kieran Kunhya2:47:05

Every day.

每天。

Jean-Baptiste Kempf2:47:06

Works fine.

工作正常。

Lex Fridman2:47:07

Wow.

哇。

2:47:07 It works fine, huh?

效果很好,是吧?

Jean-Baptiste Kempf2:47:08

Works fine, yes.

工作正常,是的。

Lex Fridman2:47:08

You have to power with a crank, I guess.

我猜你必须用曲柄来提供动力。

Kieran Kunhya2:47:10

No, but there’s no-

不,但是没有——

Lex Fridman2:47:12

There’s AOL.

有美国在线。

2:47:12 There’s AOL as your social media.

AOL 作为您的社交媒体。

Kieran Kunhya2:47:13

There’s no ads, there’s no tracking, there’s nothing.

没有广告,没有跟踪,什么都没有。

2:47:13 Like, it’s-

就像,它是——

Jean-Baptiste Kempf2:47:16

The biggest issue, to be honest, compared to Slack is that it doesn’t have threads.

老实说,与 Slack 相比,最大的问题是它没有线程。

2:47:16 That’s annoying.

这很烦人。

It doesn’t have emojis for reaction.

它没有用于反应的表情符号。

Sometimes it would be nice.

有时会很好。

Kieran Kunhya2:47:24

I, IRCv3 has.

我的,IRCv3有。

Jean-Baptiste Kempf2:47:25

Yes, V3, but no one does it, and you cannot edit your messages.

是的,V3,但没有人这样做,而且你无法编辑你的消息。

2:47:25 Right?

对吗?

And the rest, it works perfectly fine forever.

其余的,它永远工作得很好。

Lex Fridman2:47:31

But how do you communicate without emojis?

但是,如果没有表情符号,如何进行交流呢?

Jean-Baptiste Kempf2:47:33

Well, that’s why I said it’s for old people.

嗯,这就是为什么我说这是给老年人的。

Lex Fridman2:47:35

Old people.

老人们。

2:47:35 All right.

好吧。

Jean-Baptiste Kempf2:47:37

And we do, we do emojis with like- …

我们确实这样做了,我们用类似的方式制作表情符号......

2:47:37 you know, the colons and dash and-

你知道,冒号和破折号还有-

Lex Fridman2:47:40

Yeah, exactly.

是的,完全正确。

Jean-Baptiste Kempf2:47:41

…

……

2:47:41 parentheses, right?

括号里是吧?

So.

所以。

Lex Fridman2:47:43

Old school.

老派。

2:47:43 So anyway, you communicate on IRC.

所以无论如何,你们都是在 IRC 上交流的。

What were you even talking about?

你到底在说什么?

Jean-Baptiste Kempf2:47:46

Yeah, we are talking about death threats and, and-

是的,我们正在谈论死亡威胁,而且-

Lex Fridman2:47:48

Oh, damn.

哦,该死。

Jean-Baptiste Kempf2:47:49

…

……

2:47:49 but having people thanking you, and sometimes- I get people who send me a message and, and, “Oh, thank you for VLC.” And I always answer because I want to validate the fact that you need to thank the open source community.

但人们会感谢你,有时,我会收到一些人给我发消息,然后说:“哦,谢谢你的 VLC。”我总是回答,因为我想验证你需要感谢开源社区的事实。

Lex Fridman2:48:03

Yeah, please, everybody listening to this, celebrate, celebrate FFmpeg, celebrate VLC, celebrate all the incredible open source projects, Linux, everything.

是的,请大家听这个,庆祝,庆祝 FFmpeg,庆祝 VLC,庆祝所有令人难以置信的开源项目,Linux,一切。

2:48:03 There’s so many, there’s so many…

有这么多,有这么多……

And you know what?

你知道吗?

I mean, even outside of open source, just celebrate companies that create software that you use a lot and love.

我的意思是,即使在开源之外,也要庆祝那些创建了您经常使用和喜爱的软件的公司。

Kieran Kunhya2:48:26

Celebrate human endeavor.

庆祝人类的努力。

2:48:26 Celebrate the human effort to not just build something that’s okay- …

庆祝人类的努力,不仅仅是建造一些好的东西——……

build something that is damn good.

建造一些非常棒的东西。

Jean-Baptiste Kempf2:48:34

Yes, this is important, right?

是的,这很重要,对吧?

2:48:34 Because as we said, right, we work for technol- we do something very complex for the normal people.

因为正如我们所说,我们是为技术而工作——我们为普通人做一些非常复杂的事情。

Like, we want our excellence in tech to be useful for everyone.

就像,我们希望我们卓越的技术对每个人都有用。

And this is why, like, this is why we work, right?

这就是为什么,这就是我们工作的原因,对吧?

This is why I wake up in the morning is because I want people to use our stuff- …

这就是我早上醒来的原因是因为我希望人们使用我们的东西——……

Because it’s making everyone’s life easier.

因为它让每个人的生活变得更轻松。

Kieran Kunhya2:48:58

Want to solve hard problems.

想要解决难题。

2:48:58 Work on something interesting, work on some interesting technical challenges.

做一些有趣的事情,解决一些有趣的技术挑战。

Jean-Baptiste Kempf2:49:02

As we are engineers, we love to build things, right?

作为工程师,我们喜欢建造东西,对吗?

2:49:02 When I was young, like very early, I knew I wanted to build, to be an engineer.

当我年轻的时候,就像很早的时候,我知道我想要建造,成为一名工程师。

I wanted to do cars, right?

我想做汽车,对吗?

Maybe at some point I will go back to cars, right?

也许在某个时候我会回到汽车,对吗?

But this is like we want to build things that are cool and useful.

但这就像我们想要构建一些很酷且有用的东西。

And they need to be challenging, right?

他们需要具有挑战性,对吧?

Because you want your brain to turn on.

因为你希望你的大脑开启。

Lex Fridman2:49:21

When did the two of you first fall in love with programming, with building, with engineering?

你们两个什么时候第一次爱上编程、建筑和工程?

Jean-Baptiste Kempf2:49:27

When is the first time you programmed, Kieran?

Kieran,你第一次编程是什么时候?

Kieran Kunhya2:49:29

Microsoft QBasic.

微软QBasic。

2:49:29 As I was on Windows 3.1 and Windows 95, Microsoft QBasic.

因为我使用的是 Windows 3.1 和 Windows 95,所以使用的是 Microsoft QBasic。

Lex Fridman2:49:34

Oh, wow.

哦,哇。

2:49:34 Wow.

哇。

What’d you build?

你建造了什么?

Kieran Kunhya2:49:36

Like a multiplication, just counting loops like 10, 20, 30, 40.

就像乘法一样,只是计算 10、20、30、40 等循环。

Lex Fridman2:49:40

Nice.

不错。

Kieran Kunhya2:49:41

Then I thought I could do everything after that.

然后我想我可以做以后的一切。

2:49:41 I wanted…

我想要……

I jumped from doing that to I want to create a soccer, no, a football, soccer video game.

我从这样做跳到我想创造一个足球,不,一个足球,足球视频游戏。

And I drew all the, I drew everything out.

我把所有的都画出来了,我把所有的东西都画出来了。

I was like, “I’m gonna do it.” And I didn’t quite grasp that actually it’s a massive piece of work to jump from BASIC and drawing some pictures to a video game, but there we go.

我当时想,“我会做的。”我不太明白从 BASIC 跳到视频游戏中绘制一些图片实际上是一项艰巨的工作,但我们开始了。

Jean-Baptiste Kempf2:49:59

Yeah.

是的。

2:49:59 I think I did also Basics and then Turbo Pascal when I was, yeah, end of elementary school.

我想我在小学结束时也学过基础知识,然后是 Turbo Pascal。

But mostly the first time I actually did some serious programming was the, the first year of you call that middle school when you’re 11?

但我第一次真正真正做一些严肃的编程主要是在你 11 岁的时候,你称之为中学的第一年?

I was living in Italy for a year in Florence and it was amazing year.

我在意大利佛罗伦萨住了一年,这是很棒的一年。

And like the maths teacher told us to, to work in a programming language called Logo, where you had a turtle that was designing things- …

就像数学老师告诉我们的那样,使用一种名为 Logo 的编程语言,在那里你有一只正在设计东西的乌龟 - ......

On the screen, and you would turn left and right.

在屏幕上,你会左转和右转。

And in the end, we used that to do a very complex programming because of course you could do things.

最后,我们用它来做一个非常复杂的编程,因为你当然可以做一些事情。

And, and this changed, like, as I knew I was, I wanted to do things with computers and program.

而且,这种情况发生了变化,就像我所知道的那样,我想用计算机和程序做事。

Lex Fridman2:50:51

I don’t think we quite talked about H.264 properly.

我认为我们没有正确讨论 H.264。

2:50:51 We talked about David.

我们谈到了大卫。

Can we return-

我们可以回来吗-

Kieran Kunhya2:50:57

Sure

当然可以

Lex Fridman2:50:57

…

……

2:50:57 backtrack a little bit to H.264, this thing that powers basically all of the video on the internet?

稍微回溯到 H.264,这个东西基本上为互联网上的所有视频提供支持?

So can you tell me the story of H.264?

那么您能告诉我 H.264 的故事吗?

And Kieran, you’re actually a contributor- …

基兰,你实际上是一个贡献者——……

to H.264.

到 H.264。

Kieran Kunhya2:51:12

So, so H.264 is a video encoder for the H.264 video standard.

所以,所以H.264是H.264视频标准的视频编码器。

2:51:12 It dominates internet video, but also other areas such as Blu-ray discs.

它在互联网视频领域占据主导地位,但也在蓝光光盘等其他领域占据主导地位。

And Blu-ray discs are interesting because the people that make them really want the highest quality, and there’s some really cool high-end films that have been encoded broadcasting and all sorts of other areas.

蓝光光盘很有趣,因为制作它们的人确实想要最高的质量,而且有一些非常酷的高端电影已经被编码用于广播和各种其他领域。

H.264 was a big step change ’cause it kinda happened at the right time as well.

H.264 是一个巨大的改变,因为它也发生在正确的时间。

A lot of the development took place when HD video was coming out.

很多发展都是在高清视频问世时发生的。

Intel Core 2 and Nehalem CPUs were getting fast.

Intel Core 2 和 Nehalem CPU 的速度越来越快。

You could do real-time video.

你可以做实时视频。

But the most important thing was a key sort of focus on visual metrics.

但最重要的是对视觉指标的关键关注。

2:51:52 So industry and academia for 20 years before, was obsessed with mathematical metrics or what’s known as peak signal-to-noise ratio.

因此,20 年前,工业界和学术界一直痴迷于数学指标或所谓的峰值信噪比。

So mean squared error, logarithm of mean squared error, and that led to tons of issues because mean squared error leads to blurring because you actually want to minimize– You want to add a little bit of error to everything to reduce the mean squared error as opposed to having a big error, and that led to loads and loads of blurring.

因此,均方误差、均方误差的对数,这导致了很多问题,因为均方误差会导致模糊,因为你实际上想要最小化——你想在所有内容中添加一点误差来减少均方误差,而不是产生一个大误差,这会导致大量的模糊。

So hobbyists bucked that trend.

因此,爱好者们逆势而行。

It was for their own personal videos, mostly anime.

这是他们自己的个人视频,主要是动漫。

So there were two things they did differently, and there was a big iterative feedback loop with the community.

因此,他们做了两件事不同,并且与社区有一个很大的迭代反馈循环。

They did some stuff differently.

他们做了一些不同的事情。

2:52:29 Two big things: psychovisual rate distortion, so using block energy, trying to compensate for human perception when making decisions.

两件大事:心理视觉率失真,因此使用块能量,试图在做出决策时补偿人类的感知。

Lex Fridman2:52:38

So the psychovisual distortion, that’s the critical- …

所以心理视觉扭曲,这就是关键——……

2:52:38 thing.

事。

That’s the thing.

就是这样。

I mean, it’s kind of revolutionary, like, that we can, like, rethink.

我的意思是,这是一种革命性的,我们可以重新思考。

Don’t make it, like, this kind of theoretic thing- …

不要做这种理论上的事情——……

of compression.

的压缩。

Make it all about-

让一切都围绕——

Kieran Kunhya2:52:56

Being pleasing visually to the eye.

视觉上赏心悦目。

Lex Fridman2:52:57

Yeah, yeah.

是啊,是啊。

2:52:57 So compressing in a way that loses the least amount of information for the stuff that matters for us humans.

因此,对于对我们人类来说重要的东西,以丢失最少信息的方式进行压缩。

Kieran Kunhya2:53:04

Yes, exactly.

是的,完全正确。

2:53:04 As opposed to what industry– Some parts of industry are still obsessed by this, which is mathematical numbers that don’t look good in reality.

相对于什么行业——行业的某些部分仍然痴迷于这个,这是在现实中看起来不太好的数学数字。

And then adaptive quantization was the other big one where it was biasing bits against complex areas and redistributing them to less complex areas like grass.

自适应量化是另一项重要的技术,它将比特偏置到复杂区域,并将它们重新分配到不太复杂的区域,例如草地。

Grass has some high frequencies, but it’s kind of– it’s less complex overall compared to more complicated things.

草有一些高频,但与更复杂的事物相比,它总体上不那么复杂。

And this came around by ParkJoy.

ParkJoy 解决了这一问题。

So ParkJoy was really the canonical sample that was…

所以 ParkJoy 确实是典型的样本……

Is the running around in the park.

就是在公园里跑来跑去。

Lex Fridman2:53:37

This one.

这个。

Kieran Kunhya2:53:38

Yeah.

是的。

2:53:38 So this guy was really the- So this, this was created by Swedish television in the beginning of HD, and it was done on film, and it was no expense spared in terms of production quality, and it was given away for free.

所以这个家伙真的是,所以这个,这个是瑞典电视台在高清刚开始的时候制作的,是拍成电影的,而且在制作质量上是不遗余力的,而且是免费赠送的。

This was really– And this is the sample really that sorts the men from the boys in terms of it has so many challenges with the trees, with the water, with the grass, with the motion, with the…

这真的是——这确实是区分男人和男孩的样本,因为它对树木、水、草、动作、……有很多挑战。

I don’t think there’s, there’s still been any, any public test sequence as good as that these days.

我认为现在没有、仍然有任何公开测试序列能达到这样的效果。

Lex Fridman2:54:11

So for people who are just listening, we’re looking at a bunch of humans running- …

所以对于那些只是在听的人来说,我们看到的是一群人在奔跑——……

2:54:11 along a river, as you have the reflection, a lot of really high information textures everywhere, the leaves and the lighting playing with the leaves and all of this.

沿着河流,当你有反射时,到处都有很多非常高的信息纹理,树叶和灯光与树叶一起玩耍等等。

Kieran Kunhya2:54:27

You could show clearly that encoders with high PSNR-

您可以清楚地表明具有高 PSNR 的编码器

Lex Fridman2:54:30

Will blur everything

会模糊一切

Kieran Kunhya2:54:31

…

……

2:54:31 will blur everything, and you could see actually I could turn on psychovisual stuff, I could turn on adaptive quantization, and it would just look so much better.

会模糊一切,你可以看到实际上我可以打开心理视觉的东西,我可以打开自适应量化,它看起来会好得多。

But your metrics– And these metrics at the time were considered so holy.

但是你的指标——这些指标在当时被认为是神圣的。

These are the holy metrics that are untouchable.

这些都是不可触碰的神圣指标。

PSNR is the most important thing.

PSNR 是最重要的。

Lex Fridman2:54:48

Can you speak to how do you measure psychovisual stuff?

你能谈谈你如何衡量心理视觉的东西吗?

2:54:48 Like, how do you turn how pleasing a compression is for a human eye- …

比如,如何改变压缩对人眼的愉悦感——……

into a number?

变成一个数字?

Is that even possible?

这可能吗?

Kieran Kunhya2:54:59

That’s what, that’s what Netflix has been trying to do with VMAF.

这就是 Netflix 一直试图用 VMAF 实现的目标。

2:54:59 They said they’ve used a machine learning model.

他们说他们使用了机器学习模型。

Lex Fridman2:55:04

That’s a more recent thing.

这是最近的事了。

2:55:04 But back when x264 was being developed, that’s by eye you’re basically-

但当 x264 被开发出来时,你基本上是通过肉眼观察到的——

Kieran Kunhya2:55:10

It was by eye.

是用眼睛看的。

2:55:10 It was developers on their laptops.

这是开发人员在他们的笔记本电脑上进行的。

So it’s not like even with big companies with professional screens or anything, it’s- And that was actually one of the goals, which was I don’t– The, the developers at the time, Loren Merritt in particular, it’s, “I don’t wanna test this on a thirty thousand dollar screen.

所以,即使是拥有专业屏幕或其他东西的大公司,这也不是——这实际上是我的目标之一——当时的开发人员,尤其是洛伦·梅里特,“我不想在三万美元的屏幕上测试这个。

It’s– I want this to look good on someone’s laptop at home.”

这是——我希望它在家里某人的笔记本电脑上看起来不错。”

Lex Fridman2:55:27

Yeah.

是的。

2:55:27 Brilliant.

太棒了。

Jean-Baptiste Kempf2:55:28

There is another sample which is, um- …

还有另一个样本,嗯-...

2:55:28 A sample that is Planet Earth’s killer sample that I absolutely love.

这是我非常喜欢的地球杀手样本。

And you are going to see why, right?

你会明白为什么,对吗?

Kieran Kunhya2:55:39

Yeah, you’re going to love this.

是的,你会喜欢这个的。

Jean-Baptiste Kempf2:55:39

The- it’s a ton of birds, right, flying, and the more it goes, the more there are birds, and at the end, right, it’s almost like you have millions of birds.

那-有很多鸟,对,飞翔,飞得越多,鸟就越多,最后,对,几乎就像有数百万只鸟。

2:55:39 It’s the most complex thing ever to encode, right?

这是有史以来最复杂的编码,对吧?

And well, you’re watching it on YouTube, and you see how bad the YouTube encoding is actually, right?

好吧,当你在 YouTube 上观看它时,你会发现 YouTube 编码实际上有多糟糕,对吧?

And this is, like, phenomenal to optimize and get perfect quality in a constant bit rate.

这对于优化并以恒定的比特率获得完美的质量来说是惊人的。

There was a lot of optimization, mostly by Loren also, um- on anime, right?

有很多优化,主要是由 Loren 完成的,嗯——动漫,对吧?

For a long time, anime was very badly encoded because there was a ton of banding, right?

很长一段时间以来,动漫的编码非常糟糕,因为有大量的条带,对吗?

And so you see those issues, and there was a ton of things.

所以你会看到这些问题,有很多事情。

So x264 is, like– And today it’s still the reference to any encoder, new encoder, AV1, AV2, VVC, HEVC; everyone compares to x264.

所以 x264 就像——今天它仍然是任何编码器、新编码器、AV1、AV2、VVC、HEVC 的参考;大家都拿x264来比较。

Kieran Kunhya2:56:38

One of my favorite films, Cinema Paradiso, I know the engineer who created the Blu-ray, and he showed me the comparisons of x264 versus others, and the…

我最喜欢的电影之一,《天堂电影院》,我认识制作蓝光光盘的工程师,他向我展示了 x264 与其他电影的比较,以及……

2:56:38 it’s completely different.

这是完全不同的。

And I think a bunch, a bunch of guys in the Blu-ray world started using x264.

我认为蓝光世界中的很多人开始使用 x264。

Um, I think the big one was Chris Henderson from Warner Bros.

嗯,我认为最重要的是来自华纳兄弟的克里斯亨德森。

He did the whole Fringe box set with that.

他用它制作了整个边缘盒。

So quite, like, a thing a person on the street actually watches and wants to look good.

就好像,街上的一个人实际上在观看并且想要看起来不错的东西。

And so they kind of took a risk in their jobs doing that because they’re in a big company.

因此,他们在工作中这样做有点冒险,因为他们在一家大公司。

That big company can buy whatever they want.

那个大公司想买什么就买什么。

2:57:08 And they said, “No, no, no, I want to use this free and open source thing so that things look good for my, my customers and build the best.” And to this day, I personally still try and avoid watching the most cinematic films on streaming services and buy the physical discs because they look, they look good without even having to buy an expensive TV.

他们说,“不,不,不,我想使用这个免费和开源的东西,这样事情对我和我的客户来说看起来很好,并打造出最好的东西。”直到今天,我个人仍然尽量避免在流媒体服务上观看最具电影感的电影,而是购买实体光盘,因为它们看起来很好,甚至不需要购买昂贵的电视。

I think that’s the key thing.

我认为这是关键。

Jean-Baptiste Kempf2:57:28

And x264 is yet another example of open source project.

x264 是开源项目的另一个例子。

2:57:28 It was started by Laurent Aimar when he was at École Centrale Paris, where VLC was born.

它是由 Laurent Aimar 在巴黎中央理工学院时创立的,VLC 就是在那里诞生的。

And then you got a generation of people like Loren, like Jason, like Måns, like so many-

然后就有了像洛伦、杰森、曼斯这样的一代人——

Kieran Kunhya2:57:43

Henrik from-

亨里克来自——

Jean-Baptiste Kempf2:57:44

Henrik,

亨里克,

Kieran Kunhya2:57:45

Anton,

安东,

Jean-Baptiste Kempf2:57:45

and this is– Anton, and this is where the assembly thing that we use now on FFmpeg, David and so on, was born, right?

这就是——Anton,这就是我们现在在 FFmpeg、David 等上使用的汇编东西诞生的地方,对吧?

2:57:45 So x264 is, like, an amazing project with people who were really all over the world, and I think most of them never met each other.

因此,x264 是一个令人惊叹的项目,参与者来自世界各地,我认为他们中的大多数人从未见过面。

Lex Fridman2:57:59

But all of them, according to Kieran, or a large percentage, love anime.

但根据基兰的说法,他们所有人(或者说很大一部分人)都热爱动漫。

2:57:59 There’s several things I’ve never got into, and one of them is anime, and I need to

有几件事我从未涉足过,其中之一就是动漫,我需要

Jean-Baptiste Kempf2:58:09

I watch anime so much, especially at the time.

我经常看动漫,尤其是在那个时候。

2:58:09 Like, at the time, it was like a lot of anime content doesn’t exist commercially, right?

就像,当时很多动漫内容并不存在商业化,对吗?

We are before Crunchyroll, right?

我们在 Crunchyroll 之前,对吧?

So what happens is usually people who love anime, who take some things, some DVDs in Japan and rip them because there is no commercial offering.

因此,通常会发生这样的情况:热爱动漫的人们会在日本拿走一些东西,一些 DVD,然后将它们翻录,因为没有商业产品。

And-

还有——

2:58:33 some of the people who are, what we call fansubbers, are basically translating themselves to make subtitles, right?

有些人,我们称之为粉丝字幕组,基本上是在翻译自己来制作字幕,对吧?

And at that time, you download completely illegally.

而到时候,你就完全非法下载了。

It was the only way to do that, right?

这是唯一的方法,对吧?

And so all of that was handcrafted, and it fits the open source community, right?

所以所有这些都是手工制作的,并且它适合开源社区,对吧?

Because they needed tools to encode, to do fansubbing, right?

因为他们需要工具来编码、进行字幕组,对吧?

One of the most amazing open source projects for subtitles is called Aegisub, and it’s, it’s a subtitle…

最令人惊叹的字幕开源项目之一叫做 Aegisub,它是一个字幕......

It’s done for anime, for, for, for South Asian and Japanese languages.

它是为动漫、为、为、为南亚和日语而做的。

Kieran Kunhya2:59:06

There are weird textures in anime that I don’t think you get in real life content.

动漫中有一些奇怪的纹理,我认为你在现实生活中找不到这些纹理。

2:59:06 I think that was a key one, which was optimizing these weird textures that you get- …

我认为这是一个关键,它正在优化你得到的这些奇怪的纹理——……

because anime is not done in a normal fashion.

因为动画不是以正常方式制作的。

Jean-Baptiste Kempf2:59:17

Yeah.

是的。

2:59:17 The way you produce it is not– You, you mostly produce it, like, on screens, right?

你制作它的方式不是——你,你主要是在屏幕上制作它,对吧?

Since a bit of time, and you have all those gradients, right, in colors?

一段时间以来,你已经有了所有这些渐变,对吧,颜色?

Because they are very easy to produce digitally, very complex to produce in real life.

因为它们很容易数字化生产,但在现实生活中生产却非常复杂。

And, and the subtitles also are very complex because you need to have often the Japanese and then you need to have the diacritics, right?

而且,字幕也非常复杂,因为你需要经常使用日语,然后你需要有变音符号,对吧?

The what we call the ruby, right?

就是我们所说的红宝石,对吗?

Which is the hiragana and the katakana for the kanji.

这是汉字的平假名和片假名。

And then because of course you, so that you have the official subtitling, but you also need the English subtitles or the French subtitles because you want to learn that, right?

当然,因为你,所以你有官方字幕,但你也需要英文字幕或法文字幕,因为你想学习,对吗?

2:59:54 And there is so many things crazy on, on subtitles and we had like crazy samples on, on subtitles that we’ve seen all around.

字幕上有很多疯狂的东西,我们到处都看到了疯狂的字幕样本。

So this is an important part of the, the culture, but also because there was no official offering.

所以这是文化的重要组成部分,但也是因为没有官方提供。

There was no way of doing that.

没有办法这样做。

Lex Fridman3:00:11

Can you speak to the difference between H.264 and AV1 and then x264 and dav1d?

您能谈谈 H.264 和 AV1 以及 x264 和 dav1d 之间的区别吗?

3:00:11 This is this big step.

这是迈出的一大步。

Can you help people understand, are some of the streaming sites moving more towards that direction of AV1?

您能否帮助人们了解一些流媒体网站是否正在朝着 AV1 的方向发展?

Jean-Baptiste Kempf3:00:26

Let’s be honest, all of those codecs since MPEG-2 video are the same concepts.

老实说,自 MPEG-2 视频以来的所有编解码器都是相同的概念。

3:00:26 The same concept about inverse transform, about intra prediction, motion compensation, entropy coding, all of them.

关于逆变换、帧内预测、运动补偿、熵编码等所有概念都是相同的。

However, each generation gives you a bump between twenty-five and fifty percent more compression for the same quality.

然而,在相同质量的情况下,每一代都会给您带来百分之二十五到百分之五十的压缩率提升。

And so you had the MPEG-2, you had the DivX era, you have H.264, which was, like, changing, right?

所以你有了 MPEG-2,你有了 DivX 时代,你有了 H.264,这就像在改变,对吧?

H.264 improved so much.

H.264 改进了很多。

And then you had more, right?

然后你还有更多,对吧?

You had HEVC.

你有 HEVC。

You had VP9 at the same time of HEVC.

你在 HEVC 的同时拥有 VP9。

VP9 is a bit similar to HEVC in terms of quality compression, but it’s royalty-free.

VP9 在质量压缩方面与 HEVC 有点相似,但它是免版税的。

Because in multimedia there is ton of patents and the licensing after H.264 became out of hand, right?

因为在多媒体领域,H.264 失控后,存在大量专利和许可,对吧?

And could cost hundreds of millions of dollars per year.

每年可能花费数亿美元。

So it made no sense.

所以这没有任何意义。

3:01:29 So Google did this VP9 and the Alliance for Open Media did this new codec called AV1.

因此,Google 开发了 VP9,开放媒体联盟开发了名为 AV1 的新编解码器。

So you can imagine that AV1 saves between forty and sixty percent less bandwidth than H.264- …

因此,您可以想象 AV1 比 H.264 节省百分之四十到百分之六十的带宽……

for the same quality, visual quality.

对于相同的质量,视觉质量。

Kieran Kunhya3:01:48

At a given bitrate.

在给定的比特率下。

Jean-Baptiste Kempf3:01:49

At a given bitrate, right?

在给定的比特率下,对吗?

3:01:49 So that’s really like you increase the quality– either you set the bitrate and you increase the quality, or you set the quality and you decrease your bitrate.

因此,这实际上就像您提高质量一样 - 要么设置比特率并提高质量,要么设置质量并降低比特率。

But because now you move from SD to HD and HD to 4K and 4K to 4K HDR, like you increasing the size by like factor two, three, four, right?

但因为现在你从标清转向高清,从高清转向 4K,从 4K 转向 4K HDR,就像你将尺寸增加了两倍、三倍、四倍,对吗?

So you need to have better compression to keep it in terms of something that is manageable.

因此,您需要更好的压缩以使其保持可管理的状态。

Kieran Kunhya3:02:16

It’s more coding tools, more bigger blocks, lots more sub-partitions in each block.

它有更多的编码工具、更大的块、每个块中更多的子分区。

3:02:16 It’s just exponentially more complex.

它只是指数级地复杂。

Jean-Baptiste Kempf3:02:23

It’s more complex because the encoder needs to search more possibilities, right?

它更复杂,因为编码器需要搜索更多可能性,对吧?

3:02:23 So you, for example one of the things that is easy to understand is to predict a block, a color block to another, you have directions, right?

所以你,例如,很容易理解的一件事就是预测一个块,一个颜色块到另一个块,你有方向,对吗?

So you can go left, right, bottom, up, and then in terms of like the other quadrants, right?

所以你可以向左、向右、向下、向上,然后像其他象限一样,对吧?

What I call north, northeast, northwest, and so on, right?

我所说的北、东北、西北等等,对吗?

But that’s eight directions.

但这是八个方向。

Then you can do more direction.

然后你就可以做更多的指导。

You can do sixteen or sixty-nine or one hundred and twenty-eight, right?

你可以做十六、六十九或一百二十八,对吧?

You can– And every time your encoder is going to spend more time to see, oh, well, this block is exactly this one and those type of tools that you can bring, and the encoder needs to check which of the tools are going to compress you better.

你可以——每次你的编码器都会花更多的时间来查看,哦,好吧,这个块正是你可以带来的这个和那些类型的工具,编码器需要检查哪些工具能够更好地压缩你。

3:03:09 And so I guess that AV1 encoding is two orders of magnitude more than H.264 in terms of CPU cycles, right?

所以我猜 AV1 编码在 CPU 周期方面比 H.264 多两个数量级,对吧?

Order of magnitude, right?

数量级,对吧?

Kieran Kunhya3:03:21

Yeah.

是的。

3:03:21 And as we discussed, CPUs are not getting faster.

正如我们所讨论的,CPU 并没有变得更快。

You’re just throwing more cores at the problem.

你只是在问题上投入了更多的核心。

Jean-Baptiste Kempf3:03:25

But also it’s a fact that you encode once and you have hundreds of millions of users, right?

但事实上,只要编码一次,就有数亿用户,对吧?

3:03:25 So for example, YouTube, a very good example.

例如,YouTube,一个很好的例子。

YouTube encodes almost everything in H.264, but the popular video gets re-encoded in AV1 because it costs more, of course, to encode, but you encode once and you send that to millions, right?

YouTube 几乎用 H.264 编码所有内容,但流行的视频会用 AV1 重新编码,因为当然,编码成本更高,但编码一次,然后将其发送给数百万人,对吧?

So it’s a trade-off between encoding time and complexity- …

所以这是编码时间和复杂性之间的权衡——……

and CPU usage on the server side and on the client side.

以及服务器端和客户端的 CPU 使用情况。

Because at the end, if you’re distributing a video to hundreds of thousands of people and the size is half of the other, then it’s better.

因为最后,如果你要向数十万人分发一个视频,并且大小是另一个视频的一半,那就更好了。

It’s better for your battery, it’s better for your modem, et cetera, et cetera.

这对你的电池更好,对你的调制解调器更好,等等。

Lex Fridman3:04:06

So we can lay out, let’s say, the top five codec-container combos would be H.264 inside MP4 containers, AV1 inside MP4/WebM containers, ProRes for nonlinear editing inside MOV containers.

因此,我们可以布局,比方说,前五个编解码器容器组合将是 MP4 容器内的 H.264、MP4/WebM 容器内的 AV1、MOV 容器内用于非线性编辑的 ProRes。

3:04:06 So for people who don’t know, I guess ProRes is

所以对于那些不知道的人来说,我猜 ProRes 是

Kieran Kunhya3:04:32

It’s Apple’s codec for editing, originally for Final Cut Pro, and it’s designed to be fast to decode, fast to seek, because an editor will need to move very quickly.

它是 Apple 的用于编辑的编解码器,最初用于 Final Cut Pro,其设计目标是快速解码、快速查找,因为剪辑师需要非常快速地移动。

3:04:32 So it’s a different use case to the distribution element.

所以这是与分发元素不同的用例。

Lex Fridman3:04:45

There’s no or very minimal temporal compression in the-

没有或很少有时间压缩-

Kieran Kunhya3:04:48

There’s none, yeah.

没有,是的。

3:04:48 There’s none in ProRes.

ProRes 中没有。

So you can cut, so you can do cuts.

所以你可以切割,所以你可以切割。

Jean-Baptiste Kempf3:04:52

This is what we call intra-only codecs, right?

这就是我们所说的仅限内部编解码器,对吗?

3:04:52 So I’m going to explain quickly what is IPB frames.

所以我将快速解释什么是 IPB 框架。

Lex Fridman3:04:59

Yes, please.

是的,请。

Jean-Baptiste Kempf3:05:00

So I-frames, often key frames, are complete frames.

因此 I 帧(通常是关键帧)是完整的帧。

3:05:00 It’s like an image.

它就像一个图像。

It’s a JPEG, right?

这是 JPEG,对吗?

You have…

你有……

You can start, you see everything, right?

你可以开始了,你看到了一切,对吧?

And then the next image can be a P-frame, which is a predicted frame.

然后下一个图像可以是P帧,它是预测帧。

So you take some part of the previous image saying, “Well, I need the block five and seven and forty-two,” and you replace it, and then you just give the extra information, right?

所以你拿上一张图片的一部分说,“好吧,我需要第五块、第七块和四十二块”,然后替换它,然后你只需提供额外的信息,对吧?

But that means that in order to decode this P-frame, you need to have access to a previous I-frame, right?

但这意味着为了解码这个 P 帧,您需要访问之前的 I 帧,对吧?

And then, of course, you have more complex ones, which are B-frames, which are bi-predicted frames, which can depend on different type of frames, some in the past, some in the future.

当然,还有更复杂的帧,即 B 帧,它们是双向预测帧,可以依赖于不同类型的帧,有些是过去的,有些是未来的。

And so ProRes is an intra-only codec.

因此 ProRes 是一种仅限内部的编解码器。

For the people who can see, this is-

对于看得见的人来说,这是——

Lex Fridman3:05:54

Yeah, that’s a good one

是的,这是一个很好的

Jean-Baptiste Kempf3:05:54

…

……

3:05:54 a very good one, right?

非常好的一个,对吧?

So I-frames are complete frames.

所以 I 帧是完整的帧。

Um, P-frames basically depend only on I-frames, and B-frames can depend on in front.

嗯,P帧基本上只依赖于I帧,B帧可以依赖前面。

Lex Fridman3:06:04

And this GOP, group of pictures—I think the default for actually FFmpeg for H.264 is like two hundred and fifty frames, something like this.

这个 GOP,一组图片 - 我认为实际上 FFmpeg 对于 H.264 的默认设置大约是 250 帧,类似这样。

3:06:04 And to me, it’s just, it’s like magic, like that you could predict that you could have a complete frame every-

对我来说,这就像魔法一样,你可以预测你可以拥有一个完整的框架——

Jean-Baptiste Kempf3:06:26

Several seconds, that means

几秒钟,这意味着

Lex Fridman3:06:28

…

……

3:06:28 several seconds, and then you could still, you could have this chain of predictions you make, and the fact that you can– The fact that somebody like me can use FFmpeg to compress something and not notice that the result still plays back smoothly is, is like magic.

几秒钟,然后你仍然可以,你可以做出这一连串的预测,事实上,你可以——像我这样的人可以使用 FFmpeg 来压缩某些东西,而没有注意到结果仍然可以顺利播放,这就像魔法一样。

Jean-Baptiste Kempf3:06:43

You can even have, and we use that in tons on Kyber, is what we call intra-refresh, where basically there is no I-frames present.

您甚至可以拥有,我们在 Kyber 上大量使用它,这就是我们所说的内部刷新,基本上不存在 I 帧。

Kieran Kunhya3:06:54

You have no I…

你没有我……

3:06:54 You have one at the beginning.

一开始你就有一个。

And you never send an I-frame.

而且您永远不会发送 I 帧。

You get a-

你得到一个-

Lex Fridman3:06:57

How does that work?

这是如何运作的?

3:06:57 What is it?

它是什么?

Kieran Kunhya3:06:58

You build up an I-frame gradually across as the stream continues, so-

随着流的继续,您逐渐建立一个 I 帧,所以-

Lex Fridman3:07:02

Ah.

啊。

3:07:02 So you refresh certain parts- …

所以你刷新了某些部分——……

of the image.

的图像。

Jean-Baptiste Kempf3:07:05

But so you never have an I-frame.

但这样你就永远不会有 I 帧。

3:07:05 Like this is intra-refresh that we use, right?

就像我们使用的内部刷新一样,对吗?

Lex Fridman3:07:09

That’s even smarter.

这更聪明。

Jean-Baptiste Kempf3:07:10

But for me, for me, the biggest mind-blown when I started was the B-frames.

但对我来说,对我来说,刚开始时最让我震惊的是 B 帧。

3:07:10 B-frames means bi-predicted frames can depend on frames that are coming in the future.

B 帧意味着双向预测帧可以依赖于未来出现的帧。

That means that in order to decode this B frame, you need to wait for the next frame that is dependent- …

这意味着为了解码此 B 帧,您需要等待下一个相关帧 - ...

buffer that, decode that one, so that you can decode the B frame, right?

缓冲那个,解码那个,这样你就可以解码B帧,对吧?

So the way you decode the frame, the decoding order is not the same as the display order.

因此,解码帧的方式,解码顺序与显示顺序不同。

Right?

正确的?

That means the encoder needs to be very clever and decide that, “Well, you know, I’m going to depend on things like in the future.” So this is like-

这意味着编码器需要非常聪明并决定,“嗯,你知道,我将依赖于未来的事情。”所以这就像-

Lex Fridman3:07:51

It’s incredible

太不可思议了

Jean-Baptiste Kempf3:07:51

…

……

3:07:51 mind-blowing.

令人兴奋。

Kieran Kunhya3:07:53

The fact it works so smoothly every day is kind of miraculous in some ways.

事实上,它每天都能如此顺利地工作,从某些方面来说有点不可思议。

3:07:53 It works so…

它的工作原理是这样的……

You can have a stream that works across the world on their decoder versus one in the US versus one here of different manufacturers, and they produce bit for bit exactly the same material.

你可以拥有在世界各地的解码器上运行的流,而在美国的解码器和这里的不同制造商的解码器上,他们生产的材料一点一点完全相同。

That’s quite remarkable, and do quite complex things, and getting more and more complex and still be bit-exact.

这非常了不起,可以做相当复杂的事情,而且变得越来越复杂,但仍然很精确。

There’s a lot of work that goes into that.

这方面有很多工作要做。

Lex Fridman3:08:18

There’s a lot of knobs you can control in this whole process.

在整个过程中有很多旋钮可以控制。

3:08:18 There’s a lot of really fascinating parameters that I’ve gotten to know more and more over the years that FFmpeg gives you complete access to.

多年来,FFmpeg 为您提供了完全访问权限,我对许多非常有趣的参数了解得越来越多。

Maybe you could speak to some of them.

也许你可以和他们中的一些人谈谈。

So first of all, like obviously, we can lower the resolution, we can lower the frame rate, we can use different kinds of codecs, as we mentioned, from H.264 to AV1.

首先,很明显,我们可以降低分辨率,我们可以降低帧速率,我们可以使用不同类型的编解码器,正如我们提到的,从 H.264 到 AV1。

There’s ways to tune the trade-off between bitrate and quality, as we’ve kind of spoken to.

正如我们所说,有多种方法可以调整比特率和质量之间的权衡。

You know, you could do constant bitrate, you can do constant quality, say CRF, QP.

你知道,你可以做到恒定的比特率,你可以做到恒定的质量,比如 CRF、QP。

You can do the longer or shorter group of pictures, GOP, that we mentioned.

您可以制作我们提到的较长或较短的一组图片(GOP)。

I mean, all that kind of stuff.

我的意思是,所有这些东西。

It’s crazy.

太疯狂了。

Number of B-frames.

B 帧的数量。

Jean-Baptiste Kempf3:09:00

Yeah.

是的。

3:09:00 What is crazy is that a ton of people’s job is to optimize those parameters, right?

疯狂的是,很多人的工作就是优化这些参数,对吗?

A ton of people that you see at YouTube, at Netflix, at Meta, and so on, they’re not writing codecs.

你在 YouTube、Netflix、Meta 等网站上看到的很多人,他们并不是在编写编解码器。

They’re just like finding the right parameters for the file they have, for the format they have, right?

他们就像为他们拥有的文件、他们拥有的格式找到正确的参数,对吧?

Because like something that is for a movie or something that is user-generated content from your phone or a screen recording or something that you’re going to video edit, you don’t want the same things.

因为就像电影或用户从手机生成的内容或屏幕录制或您要进行视频编辑的内容一样,您不想要相同的东西。

And there are thousands of people whose job is just to optimize all that.

有成千上万的人的工作就是优化这一切。

Lex Fridman3:09:38

Yeah.

是的。

3:09:38 They’re wizards.

他们是巫师。

Hats off to them.

向他们致敬。

YouTube, like, to deliver—all the streaming sites actually, to deliver at scale.

YouTube 等,实际上是所有流媒体网站,都在大规模地提供服务。

And like YouTube is really magical because it’s not just doing like what Netflix does, which is one-way broadcasting type thing.

YouTube 确实很神奇,因为它不只是像 Netflix 那样做,这是一种单向广播类型的事情。

It also has to upload videos from all the places.

它还必须上传来自所有地方的视频。

So they’re also doing encoding at scale- …

所以他们也在进行大规模编码——……

for videos that are gonna be watched by like five people.

适合五个人观看的视频。

And it still has to deliver them on a moment’s notice.

而且它仍然必须在接到通知后立即交付它们。

No delay, nothing.

没有延迟,什么也没有。

I mean, very minimal latency.

我的意思是,延迟非常小。

And also serve it in all different resolutions.

并且还以所有不同的分辨率提供服务。

Like YouTube is basically the web version of VLC.

就像 YouTube 基本上就是 VLC 的网页版。

Jean-Baptiste Kempf3:10:25

Yeah.

是的。

3:10:25 Well, actually, it’s funny because, like Google Video, which was something they did before they acquired YouTube, was actually using the VLC plugin so that you could run VLC inside the web browser using the ActiveX plugin.

嗯,实际上,这很有趣,因为就像 Google Video(他们在收购 YouTube 之前所做的那样)实际上使用了 VLC 插件,这样您就可以使用 ActiveX 插件在网络浏览器中运行 VLC。

And so it worked in Internet Explorer, and you were actually running VLC inside your browser.

因此它可以在 Internet Explorer 中运行,并且您实际上是在浏览器中运行 VLC。

Which is funny because today we have the opposite, where we have VLC WebAssembly, where we compile all VLC and FFmpeg to decode, to run VLC inside the JavaScript virtual machine with WebAssembly.

这很有趣,因为今天我们有相反的情况,我们有 VLC WebAssembly,我们编译所有 VLC 和 FFmpeg 进行解码,以使用 WebAssembly 在 JavaScript 虚拟机中运行 VLC。

Lex Fridman3:11:04

Okay, there’s this legendary story that you pointed me to that it was discovered via WikiLeaks release of Vault 7 documents.

好吧,你给我提到了这个传奇故事,它是通过维基解密发布的 Vault 7 文件发现的。

3:11:04 The CIA was using a modified version of VLC to basically try and trick people, what?

中央情报局使用 VLC 的修改版本基本上是试图欺骗人们,什么?

To steal their data?

窃取他们的数据?

Jean-Baptiste Kempf3:11:23

Yes, exactly.

是的,完全正确。

Lex Fridman3:11:23

So can you explain what the heck happened?

那么你能解释一下到底发生了什么吗?

3:11:23 What…

什么……

Jean-Baptiste Kempf3:11:27

So, so this was a surprise, right?

所以,这是一个惊喜,对吧?

3:11:27 Because at some point, WikiLeaks mentioned some documents.

因为在某个时候,维基解密提到了一些文件。

There were a few ones with something related to Blu-rays and VLC, but the most interesting one was the CIA Vault 7, which, if I understand correctly, was the CIA had, like, a custom version of VLC where they had a specific plugin.

有一些与蓝光和 VLC 相关的东西,但最有趣的是 CIA Vault 7,如果我理解正确的话,中央情报局有一个定制版本的 VLC,其中有一个特定的插件。

Yeah, exactly.

是的,完全正确。

This is— Like, we had to write a press release on that.

这是——就像,我们必须就此写一篇新闻稿。

Lex Fridman3:11:53

VideoLAN wrote a press release saying the only safe source for getting VLC media player is the official VideoLAN website.

VideoLAN 撰写了一份新闻稿,称获取 VLC 媒体播放器的唯一安全来源是 VideoLAN 官方网站。

3:11:53 I mean, I suppose that’s a security vulnerability for basically any piece of open source software.

我的意思是,我认为这基本上是任何开源软件的安全漏洞。

Somebody can trick you.

有人可以欺骗你。

Jean-Baptiste Kempf3:12:09

To download in a fake website—

要在假冒网站上下载——

Lex Fridman3:12:11

Yeah

是啊

Jean-Baptiste Kempf3:12:12

…

……

3:12:12 or targeted advertisement, right?

或者有针对性的广告,对吗?

That was a targeted advertisement, to watch a specific file you need to watch with this custom version of VLC.

这是一个有针对性的广告,要观看特定文件,您需要使用此自定义版本的 VLC 观看。

And it was the normal binaries of VLC, except they added one DLL, I think it was psapi.dll-

这是VLC的普通二进制文件,除了他们添加了一个DLL,我认为它是psapi.dll-

Lex Fridman3:12:27

Yeah

是啊

Jean-Baptiste Kempf3:12:27

…

……

3:12:27 which was basically reading your document folder, encrypting that, and sending that.

这基本上是读取您的文档文件夹,对其进行加密,然后发送。

And the thing is, this is very clever, to be honest, because once you’re watching a movie, right, you’re going to do that for two hours, and you’re not going to touch your computer.

事实是,说实话,这非常聪明,因为一旦你在看电影,对吧,你会看两个小时,而且你不会碰你的电脑。

And sometimes it’s normal because it’s HD that your fans are going up and say, “Vroom,” and there is a ton of CPU usage because you’re using VLC, right?

有时这很正常,因为你的粉丝会起来说“Vroom”,因为它是高清的,而且因为你使用的是 VLC,所以 CPU 使用率很高,对吧?

That’s normal.

这很正常。

But the thing is, what you don’t see is that’s actually a powered version of VLC that is used by the CIA.

但问题是,你看不到的是,这实际上是 CIA 使用的 VLC 的动力版本。

We had exactly the same problem with Chinese hackers that were targeting Indian people, and that got VLC banned from India until I had to fight in courts in India, the Indian government, to unban VLC.

我们在针对印度人民的中国黑客方面也遇到了完全相同的问题,这导致 VLC 在印度被禁止,直到我不得不在印度法庭上与印度政府抗争,以取消 VLC 的禁令。

They didn’t use VLC.

他们没有使用 VLC。

3:13:18 They took just one DLL, because we signed the DLL correctly and they used that DLL to do another program.

他们只使用了一个 DLL,因为我们正确地对该 DLL 进行了签名,然后他们使用该 DLL 来执行另一个程序。

So you had the vlc.exe and was calling libVLC, but it was calling it into a fake one.

所以你有 vlc.exe 并调用 libVLC,但它把它调用成一个假的。

And they used that to, to target.

他们用它来瞄准。

There is not much we can do actually to block those type of hacks.

我们实际上无能为力来阻止此类黑客攻击。

Lex Fridman3:13:39

Yeah, and I think people should, for all open source software, for all software in general, people should pay attention where they download the thing.

是的,我认为人们应该,对于所有开源软件,对于所有一般软件,人们应该注意他们下载该东西的地方。

Jean-Baptiste Kempf3:13:46

Yes, because that means that they were not downloading it from our website.

是的,因为这意味着他们没有从我们的网站下载它。

Lex Fridman3:13:50

Do the search engines help you?

搜索引擎对您有帮助吗?

Jean-Baptiste Kempf3:13:52

No, they don’t.

不,他们没有。

Lex Fridman3:13:53

Just to clarify, ’cause you can, you know, to prevent threats from people manipulating SEO to get up there on the links and try to-

只是为了澄清一下,因为你知道,你可以防止那些操纵 SEO 的人通过链接并尝试-

Jean-Baptiste Kempf3:14:00

Absolutely not, right?

绝对不是,对吧?

3:14:00 We have a big issue for, like, more than ten years, is that there is a fake version of VLC in Germany that was reported for now for 12 years, and Google basically decides to not— They know what’s in it, but the binary is too big for their virus analyzer to analyze it.

十多年来,我们面临的一个大问题是,德国有一个假冒的 VLC 版本,该版本已经被报道了 12 年,而谷歌基本上决定不这样做——他们知道里面有什么,但二进制文件太大,他们的病毒分析器无法分析它。

And so while if you’re in Germany, you can go to a website that is a fake version of VLC with a custom installer, and it’s very popular in Germany because their website is in German, and Google mentioned that before VideoLAN.

因此,如果你在德国,你可以访问一个带有自定义安装程序的 VLC 假冒版本的网站,它在德国非常受欢迎,因为他们的网站是德语的,谷歌在 VideoLAN 之前就提到过这一点。

And the weirdest thing is that it doesn’t do anything on your machine for three weeks.

最奇怪的是,它在三周内没有在你的机器上执行任何操作。

3:14:38 Because that’s how they do the detection.

因为这就是他们进行检测的方式。

And after three weeks, there is a small program that is a service that installed at the same time that wakes up after three weeks, and it starts downloading spyware and adware.

三周后,有一个小程序是同时安装的服务,三周后会唤醒,并开始下载间谍软件和广告软件。

And Google knows about it.

谷歌知道这一点。

They’ve decided not to do anything.

他们决定什么也不做。

The guys used dark SEO in Germany to do that at some point.

这些人在德国使用了黑暗 SEO 来做到这一点。

And this is very damaging, right?

这是非常有害的,对吧?

Because one of the things that they are downloading is actually something that is replacing your ads inside your machine, right?

因为他们正在下载的内容之一实际上是替换您机器内的广告的内容,对吗?

Lex Fridman3:15:08

It’s actually quite surprisingly effective.

它实际上非常有效。

3:15:08 Whoever is doing it with Twitter and X.

无论是谁用 Twitter 和 X 来做这件事。

With X, I’ll get emails about, “Your X account has been hacked.” And however they phrase it, it gets me to, like, at least click on the email, not to follow the thing, and then you’re like, “Man, whatever they’re doing with the psychology to try to trick you, they’re quite good.”

使用 X,我会收到有关“您的 X 帐户已被黑客入侵”的电子邮件。不管他们怎么说,它让我至少点击电子邮件,而不是关注那些东西,然后你就会想,“伙计,无论他们用心理学做什么来欺骗你,他们都非常好。”

Jean-Baptiste Kempf3:15:32

There is a security version of VLC, right?

VLC 有安全版本,对吧?

3:15:32 You received an email saying, “Hey, there is a security version update on VLC.

您收到一封电子邮件,内容是:“嘿,VLC 有安全版本更新。

Think about updating right now because—

现在就考虑更新,因为——

Lex Fridman3:15:40

Right

右

Jean-Baptiste Kempf3:15:40

…

……

3:15:40 it can hack your computer.” You come.

它可以破解你的计算机。”你来吧。

It’s a website that looks decent, and—

这是一个看起来不错的网站,而且——

Lex Fridman3:15:45

Yeah

是啊

Jean-Baptiste Kempf3:15:45

…

……

3:15:45 and you download.

然后你就下载了。

It’s a new version of VLC.

这是 VLC 的新版本。

Great.

伟大的。

You don’t know.

你不知道。

A month later, you’re hacked.

一个月后,你被黑了。

You have no idea.

你不知道。

You’re part of a botnet.

您是僵尸网络的一部分。

Lex Fridman3:15:51

Yeah.

是的。

3:15:51 So make sure wherever you’re downloading stuff, it’s legitimate.

因此,请确保无论您在何处下载内容,它都是合法的。

I’m part of the botnet.

我是僵尸网络的一部分。

Speaking of which, so you’ve mentioned that VLC sandboxing is something you’re working on, and it’s actually something quite challenging.

说到这里,你提到VLC沙箱是你正在做的事情,它实际上是一个很有挑战性的事情。

Why is it important?

为什么它很重要?

Why is it hard?

为什么很难?

Jean-Baptiste Kempf3:16:09

So VLC is a core with around 500 plugins, right?

那么 VLC 是一个包含大约 500 个插件的核心,对吧?

3:16:09 One of them is FFmpeg, but we have, we support so many other formats.

其中之一是 FFmpeg,但我们有,我们支持许多其他格式。

We support new protocols, we support new filters, we support weird architectures.

我们支持新协议,我们支持新过滤器,我们支持奇怪的架构。

And in this release of VLC, you have modules that are going to call your drivers, right?

在这个版本的 VLC 中,您有可以调用驱动程序的模块,对吗?

3:16:31 Mostly the hardware decoders, which are going to call your Intel, your NVIDIA, your AMD driver.

主要是硬件解码器,它们将调用您的 Intel、NVIDIA、AMD 驱动程序。

And all calling FFmpeg, right?

并且都调用 FFmpeg,对吧?

And there might be a security issue.

并且可能存在安全问题。

There might be a security issue in the shader, there might be a security issue in VLC, in FFmpeg that is going to basically crash.

着色器中可能存在安全问题,VLC 中可能存在安全问题,FFmpeg 中可能存在安全问题,基本上会崩溃。

The issue is that you’re running VLC like every, every other program, like Adobe, right?

问题是您像 Adob​​e 等所有其他程序一样运行 VLC,对吧?

You’re running it on your machine, and it has access to all your documents, right?

你在你的机器上运行它,它可以访问你的所有文档,对吧?

So the idea is to be sure that you do a sandbox so that we can protect from ourselves, because inside the VLC process is running some code that is not even ours.

因此,我们的想法是确保您创建一个沙箱,以便我们可以保护自己免受侵害,因为 VLC 进程内部正在运行一些甚至不属于我们的代码。

Either it’s open source for other projects that we integrate in VLC, or it’s your GPU driver or something that is provided by someone else inside.

它要么是我们集成在 VLC 中的其他项目的开源代码,要么是您的 GPU 驱动程序或内部其他人提供的东西。

3:17:23 And so when we crash, we want to not allow people to do bad things, right?

所以当我们崩溃时,我们不想允许人们做坏事,对吧?

Because one of the common ways of hacking people is to crash a program, very often done with a web browser, very often done with PDF files, less often with multimedia, but that could happen.

因为黑客攻击的常见方法之一是使程序崩溃,通常是通过网络浏览器完成的,通常是通过 PDF 文件完成的,很少是通过多媒体完成的,但这可能会发生。

And when you crash, you launch something on the, on the machine of the person.

当你崩溃时,你会在该人的机器上启动一些东西。

Could be a ransomware, could be a botnet, right?

可能是勒索软件,也可能是僵尸网络,对吧?

So security of desktop applications is important.

因此桌面应用程序的安全性非常重要。

On mobile, it’s a bit different because most of the mobile applications are running inside their own sandbox.

在移动设备上,情况有点不同,因为大多数移动应用程序都在自己的沙箱内运行。

But for VLC, we could run it inside one sandbox, but the problem is that we need access to so many things that it’s basically we would have all the permissions, right?

但对于 VLC,我们可以在一个沙箱中运行它,但问题是我们需要访问很多东西,基本上我们需要拥有所有权限,对吗?

3:18:08 And so if you have a sandbox and you put some holes everywhere, it defeats the purpose, right?

因此,如果你有一个沙箱,并且到处都打了一些洞,那就达不到目的了,对吧?

So what we are trying to do, and we’re actually doing, is splitting VLC into several processes.

因此,我们正在尝试做的,也是我们实际上正在做的,是将 VLC 分成几个进程。

One is decoding, one is demuxing, one is filters, and all of them run into their own sandbox so that if a part of VLC crashes, like Chrome crashes on some, on some tab, right?

一个是解码,一个是解复用,一个是过滤器,所有这些都运行到自己的沙箱中,因此如果 VLC 的一部分崩溃,例如 Chrome 在某些选项卡上崩溃,对吗?

It crashes, but it does not crash the whole program.

它会崩溃,但不会使整个程序崩溃。

And this is what we’re trying to do.

这就是我们正在努力做的事情。

And it’s difficult because it’s a sandbox that needs to sustain gigabits per second-

这很困难,因为它是一个需要维持每秒千兆位的沙箱 -

3:18:45 …

……

of mem copies.

内存副本。

Now, it’s not a website which is five megabytes or 10 megabytes.

现在,它不是一个 5 兆或 10 兆字节的网站。

We’re talking about hundreds of megabits per second.

我们谈论的是每秒数百兆比特。

So this is why it is quite challenging.

所以这就是为什么它非常具有挑战性。

And this is a research topic that we are working on in order to have a multimedia player that is secure.

这是我们正在研究的一个研究课题,目的是为了拥有一个安全的多媒体播放器。

Lex Fridman3:19:03

This is all the kind of stuff you have to think about when millions of people are using.

当数百万人使用时,这些都是你必须考虑的事情。

3:19:03 You, you’ve mentioned something somewhere where, like all the different features of VLC, when you have that many people using it, somebody will use every single feature, and they will tell you about it.

你,你在某个地方提到过一些东西,比如 VLC 的所有不同功能,当你有那么多人使用它时,有人会使用每一个功能,他们会告诉你。

Jean-Baptiste Kempf3:19:20

Best feature in VLC is called the puzzle filter, right?

VLC 中最好的功能就是拼图过滤器,对吧?

3:19:20 So you click the puzzle filter, and it transforms your video into a jigsaw puzzle, right?

所以你点击拼图过滤器,它就会把你的视频变成拼图,对吗?

Lex Fridman3:19:30

Nice.

好的。

Jean-Baptiste Kempf3:19:30

And you can click and move the pieces, right?

你可以点击并移动棋子,对吗?

3:19:30 It’s very, very useful when you’re watching a French movie, right?

当你看法国电影时,这非常非常有用,对吧?

You’re bored- …

你很无聊——……

because it’s like very long things or a love triangle, right?

因为这就像很长的事情或三角恋,对吧?

We’ve seen that so many times, right?

我们已经见过很多次了,对吧?

But you need to watch it because someone, your wife or- …

但你需要观看它,因为有人,你的妻子或——……

told you to do that-

告诉你要这样做-

Lex Fridman3:19:49

To catch up

赶上

Jean-Baptiste Kempf3:19:49

…

……

3:19:49 or your boyfriend told you to do that.

或者你的男朋友告诉你这样做。

So you’re doing that, right?

所以你正在这样做,对吗?

And you can click and move the pieces around.

您可以单击并移动棋子。

It’s absolutely useless, right?

这绝对没用,对吧?

Like, who cares about that?

就像,谁关心这个?

First, it was done by a math teacher in high school in south of France to teach his students about Bézier curves, which is something that-

首先,这是法国南部一位高中数学老师教他的学生贝塞尔曲线的知识,这是——

3:20:08 …

……

everyone should know about, right?

大家应该都知道吧?

It’s very useful.

这非常有用。

But the code was clean, so it got in VLC.

但代码很干净,所以它进入了 VLC。

It was merged in 2010.

于2010年合并。

Five years later, I receive an email saying, “Hello, JB.

五年后,我收到一封电子邮件,内容是:“你好,JB。

I have a problem with VLC.

我的 VLC 有问题。

The puzzle is too simple.” And I was just like, “What?” And yes, the puzzle was in the UI maximums by 16 by 16, right?

这个谜题太简单了。”我当时就想,“什么?”是的,难题在于 UI 最大值为 16 x 16,对吧?

Only 256 pieces.

仅256件。

And he says, “I’m sorry, but in a movie I love puzzles, this is too simple,” right?

他说,“对不起,但在我喜欢拼图的电影中,这太简单了,”对吧?

So there is a comment of me, you can check it online, which is JB changing that the dimensions are 256 by 256.

所以我有一个评论,你可以上网查一下,是JB改的,尺寸是256 x 256。

Lex Fridman3:20:45

Right.

对。

Jean-Baptiste Kempf3:20:45

But my point is, so many useless features are used by a few people, right?

但我的观点是,这么多无用的功能被少数人使用,对吗?

3:20:45 There is a way to watch VLC movies in command line without any UI, right?

有一种方法可以在命令行中观看 VLC 电影而无需任何 UI,对吗?

It’s-

这是——

Lex Fridman3:20:57

I saw that.

我看到了。

3:20:57 You can do ASCII.

你可以做ASCII。

Jean-Baptiste Kempf3:20:59

ASCII art.

ASCII 艺术。

3:20:59 Is it useful?

有用吗?

Very useful.

非常有用。

Imagine you’re debugging…

想象一下您正在调试...

imagine you’re debugging a multicast network, right?

想象一下您正在调试多播网络,对吧?

You have thousands very complex, very complex networking stack, right?

您有数千个非常复杂的网络堆栈,对吧?

You can SSH to all of the routers and put VLC on it with no UI, and you’re going to see whether it’s black or it’s not black, right?

你可以通过 SSH 连接到所有路由器,然后在没有 UI 的情况下将 VLC 放在上面,你就会看到它是黑的还是不是黑的,对吧?

So you see if it’s all green or not all green, right?

所以你看看它是全绿还是不全绿,对吗?

So you can see-

所以你可以看到——

Lex Fridman3:21:22

Amazing.

太棒了。

Jean-Baptiste Kempf3:21:23

Yeah, right.

是的,对。

Lex Fridman3:21:23

This is fun.

这很有趣。

Jean-Baptiste Kempf3:21:24

People don’t realize there is so many things in VLC that are useful and they have users, because once you have hundreds of millions of users, you have people who use every feature.

人们没有意识到 VLC 中有这么多有用的东西,并且他们有用户,因为一旦你拥有数亿用户,你就会有人使用每一个功能。

Lex Fridman3:21:40

I would love to sort of zoom in and talk a little bit more about the distinction between kinda downloading a file and watching it offline versus streaming.

我很乐意放大并更多地讨论下载文件并离线观看与流式传输之间的区别。

3:21:40 So the complexities, the challenges of streaming.

所以流媒体的复杂性和挑战。

Is there something we could say about what it takes to stream files?

关于流式传输文件需要什么我们可以说一下吗?

‘Cause we’ve been talking about codecs- …

因为我们一直在谈论编解码器 - ...

and I think a lot of that implies encoding and decoding without having to communicate- …

我认为这在很大程度上意味着无需通信即可进行编码和解码——……

over the network.

通过网络。

Sure.

当然。

So can you elaborate, like, what’s required to do over network stuff?

那么您能否详细说明一下,例如通过网络需要做什么?

Jean-Baptiste Kempf3:22:16

Yeah, but it is less complex than it seems compared to everything that we’ve talked about.

是的,但与我们讨论的所有内容相比,它并不像看起来那么复杂。

3:22:16 Especially because the most complex thing is not about streaming in terms of streaming services, but it was what was done to actually broadcast through satellites.

特别是因为最复杂的事情不是流媒体服务方面的流媒体,而是通过卫星实际广播所做的事情。

Because in most of the modern broadcasting services, you can pause and you can go on.

因为在大多数现代广播服务中,您可以暂停也可以继续。

But when you’re sending live streaming, whether it’s broadcast or live for streaming services which are live, this is much more difficult because you need to encode in real time.

但是,当您发送实时流媒体时,无论是广播还是实时流媒体服务的直播,这都要困难得多,因为您需要实时编码。

When you go on a satellite, you have a specific size of the link, right?

当您连接卫星时,您有特定大小的链路,对吧?

You cannot have a burst-

你不能爆发——

3:22:58 …

……

of bandwidth even for a second, right?

甚至一秒钟的带宽,对吗?

Because you don’t have the space for that in your total file.

因为你的总文件中没有足够的空间。

However, there is different types of challenges, which are interesting challenges, but I think they are less complex than the one we’ve seen with late ’90s and early 2000s about broadcasting and streaming through satellite.

然而,存在着不同类型的挑战,这些都是有趣的挑战,但我认为它们比我们在 90 年代末和 2000 年代初看到的有关通过卫星进行广播和流媒体的挑战要简单。

Lex Fridman3:23:18

They’re different.

他们是不同的。

3:23:18 They are control systems challenges, whereas some are more mathematical.

它们是控制系统的挑战,而有些则更具数学性。

I think that’s the difference.

我认为这就是区别。

Jean-Baptiste Kempf3:23:23

In the streaming world, what you have is called what we call adaptive streaming, because the difficulty– and it’s not really a video problem, it’s mostly a CDN problem, is that you might have too many people watching the same thing at the same time, and it’s a congestion of the network, right?

在流媒体世界中,你所拥有的就是我们所说的自适应流媒体,因为困难——这不是真正的视频问题,主要是 CDN 问题,是你可能有太多人同时观看同一件事,这是网络拥塞,对吗?

3:23:23 So your player has difficulty downloading things fast enough to play them.

因此,您的播放器很难以足够快的速度下载内容来播放它们。

So what happens is that locally, the player is going to read a lower resolution- …

所以发生的情况是,在本地,玩家将读取较低的分辨率——……

of it.

它的。

But there are some very clever algorithms to do that, but most of it is quite basic, to be honest.

但是有一些非常聪明的算法可以做到这一点,但老实说,其中大多数都是非常基础的。

Lex Fridman3:23:59

Even on the buffering side, it’s pretty basic.

即使在缓冲方面,这也是非常基本的。

Jean-Baptiste Kempf3:24:02

Yeah, you start to download a segment, what we call a segment, and then you time, right?

是的,你开始下载一个片段,我们称之为片段,然后你就开始下载,对吧?

3:24:02 And if it takes more than 50% of the time to download a segment, you go down to…

如果下载某个片段需要超过 50% 的时间,那么您会...

Right.

对。

And the difficulty is more about when do you go up in bandwidth, in quality.

困难更多的是何时提高带宽和质量。

But this is not very complex to do.

但这做起来并不是很复杂。

When you encode, you’re going to encode seven resolutions, right?

当您编码时,您将编码七种分辨率,对吧?

And you’re going to give the bitrate.

你要给出比特率。

The difficulty is to have your encoder give the same bitrate, but it’s not as strict as it used to be.

困难在于让你的编码器提供相同的比特率,但它不像以前那么严格。

So-

所以——

Lex Fridman3:24:34

Probably YouTube has to figure out the human psychology side of that, like how pissed off do you get when it’s like very low bitrate and how long should it wait before it increases the bitrate even though the connection is better?

也许 YouTube 必须弄清楚其中的人类心理,比如当比特率非常低时你会感到多么生气,以及即使连接更好,它应该等待多长时间才能提高比特率?

3:24:34 Because maybe the changes in the bitrate is what, like, affects you psychologically.

因为也许比特率的变化会在心理上影响你。

Jean-Baptiste Kempf3:24:58

No, I think actually the interesting one is the audio.

不,我认为实际上最有趣的是音频。

Lex Fridman3:25:01

That’s true.

这是真的。

Jean-Baptiste Kempf3:25:01

The– you can kind of notice when they move from full fat AAC to the– there are compressed versions of AAC that use spectral band replication.

当它们从全脂 AAC 转变为使用频段复制的 AAC 压缩版本时,您可能会注意到。

3:25:01 You can kind of see it goes a bit tinny, and that up and down is very jarring.

你可以看到它有点细,而且上下非常不和谐。

The video side is a lot smoother, and there’s less notice.

视频方面更加流畅,并且更少注意到。

It’s really the audio you can definitely feel it from when it’s moved you from a different audio profile to one or the other.

当您从不同的音频配置文件切换到其中一种音频配置文件时,您确实可以感受到它。

I don’t know.

我不知道。

We’re surprisingly tolerant at skipping audio glitches.

我们对跳过音频故障的容忍度令人惊讶。

I’m surprised people I know who are not video engineers, how tolerant they are to watching sports at 30 FPS, for example, whereas it should really be 60.

例如,我认识的非视频工程师人士对以 30 FPS 观看体育比赛的容忍程度感到惊讶,而实际上应该是 60 FPS。

The world is a lot more tolerant to that, but audio people are very– There’s– It’s an immediate feedback mechanism of, “Oh, something’s changed.”

世界对此的容忍度要高得多,但音频行业的人非常——有——这是一种即时反馈机制,“哦,有些东西发生了变化。”

Lex Fridman3:25:44

If you hear a glitch, you realize it directly.

如果您听到故障,您会直接意识到。

3:25:44 I get to fully realize that, I suppose.

我想我已经完全意识到了这一点。

One of the things I’m afraid of when I listen to audio more and more, that I get to notice every single tiny detail, and that you can over-obsess when people in general are able to kinda blur their consumption.

当我越来越多地听音频时,我担心的一件事是,我会注意到每一个微小的细节,而当一般人能够有点模糊他们的消费时,你可能会过度痴迷。

They can look past certain imperfections.

他们可以忽略某些缺陷。

Jean-Baptiste Kempf3:26:08

But then when you combine like an event that is, for example, a sport event that is probably going through satellite or- …

但是,当你像一项赛事一样结合起来时,例如,一项可能通过卫星进行的体育赛事,或者……

3:26:08 somewhere else and goes to a central place for encoding, and then you need to encode this older resolution in real time.

其他地方并转到一个中心位置进行编码,然后您需要实时对这个较旧的分辨率进行编码。

You don’t have time for QA.

您没有时间进行质量检查。

You need to push that to CDNs.

您需要将其推送到 CDN。

You need to add probably DRM for protection.

您可能需要添加 DRM 进行保护。

You need to have that over a ton of different devices.

您需要在大量不同的设备上实现这一点。

Then yes, it is complex.

是的,它很复杂。

But– And also, like you’re in the web browser or in very much different devices that you use for television, where you had like a defined set-top box or cable box that you know where you control end-to-end.

但是,就像您在网络浏览器或用于电视的非常不同的设备中一样,您有一个定义的机顶盒或有线电视盒,您知道您在哪里进行端到端控制。

So it’s a challenge, but it’s less…

所以这是一个挑战,但挑战较小……

I think the networking part while you agree to have 10, 20 seconds of latency, I don’t think this is very difficult.

我认为网络部分虽然你同意有 10、20 秒的延迟,但我认为这不是很困难。

Lex Fridman3:27:05

Speaking of networking and latency, so your new effort, as we mentioned, is Kyber, which is aimed at ultra-low latency.

说到网络和延迟,正如我们提到的,你们的新努力是 Kyber,它的目标是超低延迟。

3:27:05 As you say, every millisecond counts, and you’re applying that to remote control machines like robots, drones, computers.

正如你所说,每一毫秒都很重要,你正在将其应用于机器人、无人机、计算机等远程控制机器。

Can you tell me about it?

你能告诉我一下吗?

Jean-Baptiste Kempf3:27:25

Sure.

当然。

3:27:25 If you start from where we used to be, right?

如果你从我们以前的地方开始,对吗?

You used to use FFmpeg to encode files, right?

您以前使用 FFmpeg 来编码文件,对吗?

And then we used FFmpeg and VLC to encode in streaming services, right?

然后我们使用 FFmpeg 和 VLC 在流媒体服务中进行编码,对吗?

And then you need to go lower and lower.

然后你需要越来越低。

And the question was where up to where we can, can we go?

问题是我们能去到什么地方?

3:27:46 And this question is very important because there are many use cases where you need to be fast, and it’s when you have feedback interaction, right?

这个问题非常重要,因为有很多用例需要快速,而且是在进行反馈交互的时候,对吧?

We are not just listening to something, you’re actually controlling it, right?

我们不只是在听某些东西,你实际上是在控制它,对吗?

Because– And that’s the biggest difference compared to what we’ve done so far, is that I need video to have a feedback on something that is happening live, whether it’s a drone flying, whether it’s controlling a humanoid robot from distance, whether it’s controlling a hover, whether it’s playing a video game in the cloud gaming, because this is what I did on a previous job, right?

因为——这与我们迄今为止所做的最大的区别是,我需要视频来反馈现场发生的事情,无论是无人机飞行,无论是远距离控制人形机器人,无论是控制悬停,还是在云游戏中玩视频游戏,因为这就是我之前工作中所做的,对吧?

I was CTO of a cloud gaming startup.

我是一家云游戏初创公司的首席技术官。

And this is a very interesting topic because you push to the limit the network.

这是一个非常有趣的话题,因为你将网络推向了极限。

3:28:32 You need to care not about the quality like we’ve done on video, and we’ve talked about with H.264.

您不需要像我们在视频上所做的那样关心质量,我们已经讨论过 H.264。

You care about latency, because a millisecond is meaningful when you’re controlling a car, right?

您关心延迟,因为当您控制汽车时,毫秒是有意义的,对吧?

For– Well, you’ve seen, you’ve used Waymos, right?

因为——嗯,你已经看到了,你用过 Waymos,对吧?

When Waymos don’t work, and that happens even if one percent of the time, there is someone that is basically remote controlling that.

当 Waymos 无法工作时,即使只有百分之一的情况也会发生这种情况,基本上都会有人远程控制它。

And this is exactly the stuff that we’re building.

这正是我们正在构建的东西。

It’s really an SDK platform to do end-to-end control of machines.

它实际上是一个对机器进行端到端控制的SDK平台。

Lex Fridman3:29:10

So the– this comes up quite a lot in a lot of different contexts in robotics.

因此,这在机器人技术的许多不同背景下经常出现。

3:29:10 So obviously, teleoperation, teleop is becoming more and more important including for training robots via machine learning.

显然,远程操作变得越来越重要,包括通过机器学习训练机器人。

Jean-Baptiste Kempf3:29:25

Yes.

是的。

3:29:25 And what we do is a bit different from everyone else, is that we take only one socket, one connection, which is a QUIC protocol based on UDP which is interesting because it’s done for low latency.

我们所做的与其他人有点不同,我们只采用一个套接字、一个连接,这是一种基于 UDP 的 QUIC 协议,这很有趣,因为它是为了低延迟而完成的。

It doesn’t have two of the, what we call the TCP head-of-line problem and the HTTP head-of-line problem.

它没有我们所说的 TCP 队头问题和 HTTP 队头问题这两个问题。

It’s ciphered by default, but on the same wire, we send multiple streams, like multiple tracks.

默认情况下它是加密的,但在同一条线路上,我们发送多个流,就像多个轨道一样。

We send audio, we send video, but we also send the commands, right?

我们发送音频,我们发送视频,但我们也发送命令,对吧?

Mouse, keyboard, gamepad, and so on.

鼠标、键盘、游戏手柄等。

And we do that while maintaining coherence, right?

我们在保持一致性的同时做到了这一点,对吗?

Synchronization.

同步。

Because what people don’t realize is that all the clocks actually drift.

因为人们没有意识到所有的时钟实际上都在漂移。

3:30:06 And when you’re controlling a robot, a robot is going to have, like, two cameras, five cameras, ten cameras, a ton of sensors, GPS, and so on.

当你控制机器人时,机器人将拥有两个摄像头、五个摄像头、十个摄像头、大量传感器、GPS 等等。

And if you want to train correctly your robotic AI model, you need to have all those that are in sync and coherent.

如果你想正确训练你的机器人人工智能模型,你需要让所有这些模型保持同步和连贯。

And what we’ve done, and it’s all the stuff that we learned on VLC in broadcast in real time, and MPEG-TS that Kierans know well, is that we account for clock drifting.

我们所做的,也就是我们在实时广播的 VLC 和 Kierans 熟悉的 MPEG-TS 上学到的所有东西,就是我们考虑了时钟漂移。

And so when I record a Kyber stream, a robot, I am sure that it’s going to be predictive in the way you play it back.

因此,当我录制机器人 Kyber 流时,我确信它会以你回放的方式进行预测。

And so when you’re going to do recording and training of your AI model, you need to be sure that every time you retrain based on the data, the data is going to stay coherent.

因此,当您要记录和训练人工智能模型时,您需要确保每次根据数据进行重新训练时,数据都会保持一致。

And clocks actually drift.

时钟实际上会漂移。

3:30:54 Like, the existing solution works with one camera.

就像,现有的解决方案适用于一台相机。

Once you’re going to five or six, it’s more complex.

一旦你达到五六岁,事情就会变得更加复杂。

Lex Fridman3:31:00

So you wanna make sure that the visual snapshot perfectly matches the time it actually happened.

所以你要确保视觉快照与实际发生的时间完全匹配。

Jean-Baptiste Kempf3:31:08

Exactly.

确切地。

3:31:08 And also, if you’re going to control, right, I do something on robot, I need to be sure that it is actually happening at that precise time, right?

而且,如果你要控制,对吧,我在机器人上做某事,我需要确保它确实在那个精确的时间发生,对吧?

And so we have on the server, which would be a robot, a time of, like, re-timestamping mechanism accounting for clock drift for that, right?

因此,我们在服务器(这将是一个机器人)上有一个时间,例如重新时间戳机制,可以解释时钟漂移,对吧?

So that’s one of the use case of Kyber to control robots.

这就是 Kyber 控制机器人的用例之一。

I also think, like, remote drones, remote whether it’s defense or non-defense, remote cars, remote submarines.

我还认为,远程无人机,远程无论是防御还是非防御,远程汽车,远程潜艇。

There is many places in industry or remote surgery where the expert cannot go everywhere the machine is because it’s either dangerous or it’s too costly, right?

在工业或远程手术的许多地方,专家无法到达机器所在的任何地方,因为它要么危险,要么成本太高,对吗?

So you allow people to have machines next to you, right?

所以你允许人们在你旁边拥有机器,对吧?

3:31:54 The goal of Kyber is to make distance disappear because it’s either projection of skills or projection of power, right?

Kyber 的目标是让距离消失,因为它要么是技能投射,要么是力量投射,对吗?

So imagine we are all like— you’ve seen the Meta Ray-Ban and everyone else, right?

所以想象一下我们都像——你见过 Meta Ray-Ban 和其他人,对吧?

You need to stream there, right?

你需要在那里直播,对吧?

Because you’re not going to run anything over there, right?

因为你不会在那里运行任何东西,对吗?

So you need GPU power whether it’s on a cloud, on a phone to stream that.

因此,无论是在云上还是在手机上,您都需要 GPU 能力来进行流式传输。

And so all of these use cases need to be not about extremely low latency, but real-time latency for video.

因此,所有这些用例需要的不是极低延迟,而是视频的实时延迟。

And so that means you need– we’re toying with the encoders so that the encoders encode a frame in four milliseconds.

所以这意味着你需要——我们正在研究编码器,以便编码器在四毫秒内编码一帧。

And Kieran with his company also goes under those type of latency, because you need to optimize at max the local latency, right?

Kieran 和他的公司也面临这种类型的延迟,因为您需要最大程度地优化本地延迟,对吧?

3:32:42 Because it’s the decoder, the encoder and so on.

因为它是解码器、编码器等等。

Because this time is going to be added to your networking time.

因为这个时间将被添加到你的网络时间中。

And it’s not just about low latency, it’s also about, like, reliability.

这不仅涉及低延迟,还涉及可靠性等。

We do clever things like forward error correction, right?

我们做了一些聪明的事情,比如前向纠错,对吗?

So forward error correction is you over-transmit a bit of data, right, a few percent and while over-transmit, you’re allowed to lose some packets.

因此,前向纠错就是过度传输一点数据,对吧,几个百分点,在过度传输的同时,你可以丢失一些数据包。

Because all of that is very difficult over an internet network where you’re going to do things very far away.

因为所有这些在互联网上都是非常困难的,因为你要在很远的地方做事情。

And if you check that all packets are delivered, you add a ton of latency.

如果您检查所有数据包是否已传送,则会增加大量延迟。

If you don’t want latency, what we do is that we over-transmit some data that you can reconstruct on the client side when there is things that are broken, right?

如果你不想要延迟,我们所做的就是过度传输一些数据,当出现问题时你可以在客户端重建这些数据,对吗?

3:33:34 So and a few days, weeks ago, we were doing the demo around Las Vegas for the CES about— we had a rover that is fully 3D printed.

几天前,几周前,我们在拉斯维加斯的 CES 上进行了演示——我们有一个完全 3D 打印的漫游车。

It’s very simple.

这很简单。

It’s a car, right?

这是一辆车,对吧?

It’s a small car with a telescopic arm, and it was actually controlled from France, right?

这是一辆带有伸缩臂的小车,实际上是法国控制的,对吗?

And the video was with a webcam and a very small server, right?

视频是使用网络摄像头和非常小的服务器,对吗?

A small PCB was basically running and send that to someone that is on the other side of the planet.

一个小型 PCB 基本上正在运行,并将其发送给地球另一端的人。

And so there is so many use cases.

所以有很多用例。

You can also think about having AI who are going to control many drones and so on.

你还可以考虑让人工智能来控制许多无人机等等。

And technically, we need to be amazing in video, we need to be amazing at networking, we need to care about any milliseconds in networking, in encoding time, in decoding time, and also you need to integrate very low level.

从技术上讲,我们需要在视频方面表现出色,我们需要在网络方面表现出色,我们需要关心网络、编码时间、解码时间中的任何毫秒,而且还需要集成非常低的级别。

Lex Fridman3:34:26

So sync everything together well.

因此,将所有内容很好地同步在一起。

3:34:26 But how– Like, what kind of latency can you get to?

但是如何——比如,你能达到什么样的延迟?

Like, why– When you say milliseconds, what’s the goal?

就像,为什么——当你说毫秒时,目标是什么?

Jean-Baptiste Kempf3:34:34

So my goal is four milliseconds glass-to-glass latency.

所以我的目标是玻璃到玻璃的延迟为四毫秒。

Lex Fridman3:34:38

What’s glass-to-glass mean?

玻璃对玻璃是什么意思?

Jean-Baptiste Kempf3:34:40

So it’s easy, right?

所以这很容易,对吧?

3:34:40 You have a computer which is running a program, right?

您有一台正在运行程序的计算机,对吗?

Probably a video game, and this one is actually running, right?

可能是一款电子游戏,而这个游戏实际上正在运行,对吧?

It could be– it’s an example of a robot, right?

它可能是——这是一个机器人的例子,对吧?

And you have the replicate that is- …

你得到的复制品是——……

done through the network.

通过网络完成。

And you want, if you take a one thousand hertz camera, you can take a picture, and you want that to be at four milliseconds.

你希望,如果你用一千赫兹的相机,你可以拍一张照片,你希望它能在四毫秒内完成。

Four milliseconds means two hundred and forty hertz, right?

四毫秒意味着两百四十赫兹,对吗?

Lex Fridman3:35:06

Yes.

是的。

3:35:06 Nuts.

坚果。

Jean-Baptiste Kempf3:35:07

So far we achieve seven milliseconds from a Windows to Windows or Windows to Mac.

到目前为止,我们实现了从 Windows 到 Windows 或 Windows 到 Mac 的 7 毫秒。

3:35:07 And if you look in the timing, there is around three point five milliseconds inside the NVIDIA hardware encoder and around two milliseconds on the Intel decoder, right?

如果你看一下时间,NVIDIA 硬件编码器内大约有三点五毫秒,Intel 解码器上大约有两毫秒,对吧?

So, like, the encoder plus the decoder is already six milliseconds, right?

那么,编码器加上解码器已经是六毫秒了,对吧?

So in order to go down, we need either to have some other type of codecs or some better encoder that are faster.

因此,为了降低成本,我们要么需要一些其他类型的编解码器,要么需要一些更快的更好的编码器。

But four milliseconds would be the grail.

但四毫秒将是圣杯。

Lex Fridman3:35:44

That’s pretty nuts.

这太疯狂了。

3:35:44 I love it, though.

不过我喜欢它。

I don’t think anyone’s ever achieved that, right?

我认为没有人曾经实现过这一点,对吧?

That’s fast.

那很快。

Jean-Baptiste Kempf3:35:49

You can achieve that with custom hardware- …

您可以通过定制硬件来实现这一目标 - ...

3:35:49 with SDI, with professional hardware.

配备SDI,配备专业硬件。

But I want that to work over the internet.

但我希望它能够通过互联网进行。

I want that to work with any robots where you’re going to have a small Jetson Nano in it or a N150, right?

我希望它能够与任何装有小型 Jetson Nano 或 N150 的机器人一起使用,对吗?

I want that because there is going to be millions of robots or- …

我想要这个,因为将会有数百万个机器人,或者——……

drones are just rolling robots or flying robots or swimming robots, right?

无人机只是滚动机器人、飞行机器人或游泳机器人,对吗?

It’s just you, a machine that you control.

这只是你,一台你控制的机器。

And in order…

并且按顺序...

Either you need to teleoperate them or when everything will be fully autonomous, you need to teleobserve them, right?

要么你需要远程操作它们,要么当一切都完全自主时,你需要远程观察它们,对吗?

You need to check what’s happening.

你需要检查发生了什么。

3:36:29 And in my view, in the future, like, all those remote cars will be teleobserved by an AI model, which is just going to say, “Well, everything is good.” And when it’s not good, say, “Hey, there is a problem,” and then you have an operator, right?

在我看来,在未来,所有这些远程汽车都将由人工智能模型进行远程观察,它只会说,“好吧,一切都很好。”当情况不好时,说:“嘿,有问题”,然后就有了接线员,对吧?

And this is going to be about safety, right?

这与安全有关,对吧?

When you have your humanoid taking care of your grandma or my grandma, I want to be sure that everything goes well, and I’m not in those type of horrible scenarios where the robot is dangerous.

当你让你的人形机器人照顾你的奶奶或我的奶奶时,我想确保一切顺利,而且我不会处于机器人危险的那种可怕场景中。

Or when I’m driving, I want, like, the car to stop when it should stop, and if needed, someone takes care of that, right?

或者当我开车时,我希望汽车在该停下的时候停下来,如果需要的话,有人会照顾这个,对吧?

And so there is so many cases, scenarios about real time, and so the goal of Kyber is to make real time control of machine.

所以有很多关于实时的案例、场景,所以Kyber的目标是对机器进行实时控制。

Distance disappears.

距离消失。

Lex Fridman3:37:15

It’s incredible.

太不可思议了。

3:37:15 And some of the same technology, some of the same ideas that we’re talking about is connected to what you’re doing.

我们正在谈论的一些相同的技术、一些相同的想法与你正在做的事情有关。

Jean-Baptiste Kempf3:37:22

And for me, it’s amazingly challenging, right?

对我来说,这是非常具有挑战性的,对吧?

3:37:22 Because I would say that on video I’m doing okay, but networking I have so much more to learn, right?

因为我想说,在视频方面我做得还不错,但在网络方面我还有很多东西要学,对吧?

It’s about, like, congestion protocols, bitrate adaptation in real time.

它涉及拥塞协议、实时比特率适应等。

But it’s quite funny.

但这很有趣。

And so I created this project and we have fundraised in the US, of course.

所以我创建了这个项目,当然我们已经在美国筹集了资金。

But it’s open source, right?

但它是开源的,对吧?

This is important, right?

这很重要,对吧?

Like, we’ve not said that, right?

就像,我们没有这么说过,对吧?

But everything on Kyber is open source.

但 Kyber 上的一切都是开源的。

Lex Fridman3:37:50

So how do you make money?

那么如何赚钱呢?

Jean-Baptiste Kempf3:37:51

It’s a dual license, commercial and AGPL, right?

它是双重许可证,商业许可证和 AGPL,对吧?

3:37:51 You remember what you said- …

你还记得你说过的话——……

about licenses.

关于许可证。

Basically, if you want to use Kyber in your product, you must have your full product open source.

基本上,如果您想在您的产品中使用 Kyber,您必须将完整的产品开源。

If you want to use this amazing technology but not open source, you pay the commercial license, right?

如果您想使用这项令人惊叹的技术但不开源,您需要支付商业许可证,对吗?

So the small people or the hobbyist and the very small guys who want to do that, they can use the technology.

因此,想要做到这一点的小人物或爱好者以及非常小的人,他们可以使用这项技术。

They build something that is open source and cool.

他们构建了一些开源且很酷的东西。

Lex Fridman3:38:18

That’s awesome.

太棒了。

Jean-Baptiste Kempf3:38:19

And if you’re a large company, you’re going to have the support, all the IP, the right modification, and so on.

如果您是一家大公司,您将获得支持、所有 IP、正确的修改等等。

3:38:19 So yeah, it’s really cool and also I’m building robots, and I love that, right?

是的,这真的很酷,而且我正在建造机器人,我喜欢这个,对吧?

Like we have– Like the rover we have is 3D printed.

就像我们拥有的一样——就像我们拥有的流动站是 3D 打印的一样。

We are finishing a demo where it’s an actual wing, right?

我们正在完成一个演示,它是一个真正的机翼,对吧?

Like a type of drone wings that is also fully 3D printed.

就像一种完全 3D 打印的无人机机翼。

We are trying to do a sailboat that is 3D printed.

我们正在尝试制作一艘 3D 打印的帆船。

And we’ll work on some humanoids.

我们将研究一些人形机器人。

Of course, they are not going to be very good robots, right?

当然,它们不会是很好的机器人,对吗?

It’s not our job, but we’re here for everyone to make robots.

这不是我们的工作,但我们是为每个人制造机器人而存在的。

Cool.

凉爽的。

Lex Fridman3:38:57

Ah, you’re talking to the right guy.

啊,你正在跟对的人说话。

3:38:57 I love robots.

我喜欢机器人。

There’s a bunch of them upstairs.

楼上有很多人。

And teleop is gonna be really, really important, especially as the number of robots scales across the world.

Teleop 将变得非常非常重要,尤其是随着世界各地机器人数量的增加。

So 100%.

所以100%。

Let’s talk about the future of multimedia.

让我们谈谈多媒体的未来。

FFmpeg, VLC, but some of the codecs, we didn’t really mention AV2.

FFmpeg、VLC,但是一些编解码器,我们并没有真正提到 AV2。

So can we just lay out what is AV2?

那么我们可以简单介绍一下什么是 AV2 吗?

What is the hope for it?

它的希望是什么?

What is H.265, H.266?

什么是H.265、H.266?

Jean-Baptiste Kempf3:39:26

So AV1 is this codec that is done by the Alliance for Open Media, right?

那么 AV1 就是开放媒体联盟完成的编解码器,对吧?

3:39:26 Where there is Google, Netflix, Amazon, Apple VideoLAN, where we try to make a royalty-free very good codec, right?

哪里有 Google、Netflix、Amazon、Apple VideoLAN,哪里就有我们尝试制作免版税的非常好的编解码器,对吧?

And now it’s being deployed.

现在它正在部署中。

But actually, the codec was finished in 2018, but a codec takes years to be used in wide scenarios, right?

但实际上,编解码器在2018年就完成了,但是编解码器需要几年的时间才能应用于广泛的场景,对吗?

So AV2 is the next generation of this codec.

因此 AV2 是该编解码器的下一代。

It’s 30% better, right?

好 30%,对吗?

So if you keep the same quality, you get 30% bandwidth reduction compared to AV1.

因此,如果保持相同的质量,与 AV1 相比,您的带宽会减少 30%。

Lex Fridman3:40:03

What’s the connection with the dav1d and AV2?

dav1d 和 AV2 有什么关系?

Jean-Baptiste Kempf3:40:06

We are going to do a dav1d 2, right?

我们要做一个 dav1d 2,对吗?

3:40:06 That I call Devid, because de is two in French.

我称之为戴维,因为 de 在法语中是“二”。

Lex Fridman3:40:13

Ah, well done.

啊,干得好。

Jean-Baptiste Kempf3:40:13

…

……

3:40:13 And you have to know that dav1d is an actual what we call recursive acronym, right?

你必须知道 dav1d 是我们所说的递归缩写,对吗?

Because it means D, dav1d, is an AV1 decoder, right?

因为它的意思是D,dav1d,是AV1解码器,对吧?

So

所以

Lex Fridman3:40:23

Oh, nice.

哦,不错。

3:40:23 Nice.

好的。

I didn’t even think of that.

我什至没有想到这一点。

And people should know that dav1d is spelled with a one.

人们应该知道 dav1d 拼写为 1。

Jean-Baptiste Kempf3:40:29

Yes.

是的。

3:40:29 It’s…

这是……

And so dav1d 2-

所以 dav1d 2-

Lex Fridman3:40:32

It’s gonna be spelled with a two.

它将被拼写为“2”。

3:40:32 Please tell us

请告诉我们

Jean-Baptiste Kempf3:40:33

…

……

3:40:33 is going to be D-A-V-2-D.

将是 D-A-V-2-D。

Sorry, I don’t know how you pronounce that.

抱歉,我不知道你怎么发音。

And again, we did a demo at the CES of VLC running the first demo of AV2.

我们再次在 CES 上进行了 VLC 演示,运行了 AV2 的第一个演示。

Lex Fridman3:40:44

So can you clarify to me the specification of AV2?

那么您能向我解释一下 AV2 的规格吗?

3:40:44 And then the encoding and the decoding

然后是编码和解码

Jean-Baptiste Kempf3:40:52

Sure.

当然。

3:40:52 So the specification is like the document which explains how the codec is supposed to work, right?

所以规范就像解释编解码器应该如何工作的文档,对吗?

Lex Fridman3:41:00

And that’s really AV2.

这确实是 AV2。

Jean-Baptiste Kempf3:41:02

That is AV2, like H.264.

即 AV2,如 H.264。

3:41:02 Right?

正确的?

Then you have an encoder.

然后你就有了一个编码器。

The current encoder is called AVM, and there will probably be other encoders, probably one called SVT-AV2, and those are the encoder.

目前的编码器称为AVM,可能还会有其他编码器,可能有一种称为SVT-AV2,这些就是编码器。

The same way x264 is an encoder to H.264, the same way that x265 is an encoder for the H.265 codec.

同样,x264 是 H.264 的编码器,x265 是 H.265 编解码器的编码器。

And the decoders for AV1 is dav1d.

AV1 的解码器是 dav1d。

The decoder for AV2 is dav2d.

AV2 的解码器是 dav2d。

The decoder for H.264 is ffh264 inside FFmpeg.

H.264 的解码器是 FFmpeg 内部的 ffh264。

The decoder for HEVC is ffhevc inside FFmpeg.

HEVC 的解码器是 FFmpeg 内的 ffhevc。

And there is a next generation codec from the MPEG world after H.264, H.265.

继 H.264、H.265 之后,MPEG 领域还有下一代编解码器。

There is one that is called H.266, also known as VVC.

有一种称为 H.266,也称为 VVC。

Lex Fridman3:41:59

So HEVC is H.265.

所以 HEVC 就是 H.265。

3:41:59 VVC is H.266.

VVC 是 H.266。

Why is H.266 super sexy- …

为什么 H.266 超级性感 - …

and so much better?

好多了?

Jean-Baptiste Kempf3:42:10

So the question often we have is why are there two names?

那么我们经常遇到的问题是为什么有两个名字?

3:42:10 Because most of the time it is a joint work from the ISO world and the ITU, which is the International Telecommunication Union.

因为大多数时候它是 ISO 世界和 ITU(即国际电信联盟)的联合工作。

Lex Fridman3:42:24

These are these two regulatory bodies.

这是这两个监管机构。

Kieran Kunhya3:42:26

No, one is a private entity and one is the United Nations.

不,一个是私人实体,一个是联合国。

Lex Fridman3:42:28

Which one is the private?

哪一个是私人的?

Kieran Kunhya3:42:29

ISO is private.

ISO 是私有的。

Jean-Baptiste Kempf3:42:31

In theory, H.264 is MPEG-4 Part 10, H.264/AVC.

理论上,H.264 是 MPEG-4 Part 10,H.264/AVC。

3:42:31 And this is the full name.

这是全名。

Lex Fridman3:42:43

Nice.

好的。

Kieran Kunhya3:42:44

So it’s the concatenation of the ISO name and the ITU name- …

所以它是 ISO 名称和 ITU 名称的串联 - ...

3:42:44 even though they work together.

即使他们一起工作。

This is, this is politics, historical, you know— …

这是,这是政治,历史,你知道——……

gotcha.

陷阱。

Jean-Baptiste Kempf3:42:53

And for HEVC, it’s MPEG-H, H.265, HEVC.

对于 HEVC,则是 MPEG-H、H.265、HEVC。

Lex Fridman3:42:58

Got it.

知道了。

Jean-Baptiste Kempf3:42:59

And there is H.266, which is also named VVC.

还有H.266,也称为VVC。

Lex Fridman3:43:02

Is there a high-level thing to say about the improvement of-

关于以下方面的改进,有什么高层次的话要说吗?

Jean-Baptiste Kempf3:43:05

30% each generation is the best summary.

每代30%就是最好的总结。

Lex Fridman3:43:09

This is true both for the AV- …

对于 AV- 来说都是如此……

3:43:09 codecs and the- …

编解码器和 - ...

H.264, 5, 6.

H.264、5、6。

Jean-Baptiste Kempf3:43:15

So the professionals who are listening to us are going to kill us because they say, “No, it’s 35%, 25%—

所以那些听我们讲话的专业人士会杀了我们,因为他们说,“不,是 35%、25%——

Kieran Kunhya3:43:20

“No, it’s 50, 60.”

“不,是50、60。”

Jean-Baptiste Kempf3:43:21

…

……

3:43:21 it’s 50,” blah, blah, blah.

50,”等等,等等,等等。

But globally, you need to know that HEVC is 30% better than H.264.

但在全球范围内,您需要知道 HEVC 比 H.264 好 30%。

H.266 is 30% better than H.265 because there are so many cases and so many scenarios.

H.266比H.265好30%,因为案例太多,场景太多。

For example, there are cases, especially for screen recording, where the gains are humongous because you arrive, you have the right tool that is done for that.

例如,在某些情况下,特别是对于屏幕录制,收益是巨大的,因为您到达时,您拥有为此完成的正确工具。

And so for a specific video, a new generation is going to give you 70% gain or 80% gain.

因此,对于特定视频,新一代将为您带来 70% 或 80% 的增益。

Right?

正确的?

But there used to be a ton more codecs, but now the two main families for transmission are H.264, H.265, H.266, and the other is AV1, AV2.

但过去有更多的编解码器,但现在传输的两个主要系列是 H.264、H.265、H.266,另一个是 AV1、AV2。

Lex Fridman3:44:00

And I guess the major difference would be the cost of encoding.

我想主要的区别是编码的成本。

Jean-Baptiste Kempf3:44:04

Yes, and the royalty of the patents.

是的,还有专利的版税。

3:44:04 And this is the reasons why you see the AV version of codecs, is because they try to be as royalty-free—which means no cost for the patents—as much as possible.

这就是为什么您会看到 AV 版本的编解码器的原因,因为它们试图尽可能地免版税(这意味着没有专利成本)。

Because what you need to know, and we’ve not talked about that so far, is that multimedia is what we call a patent minefield.

因为你需要知道的是,多媒体就是我们所说的专利雷区,而到目前为止我们还没有讨论过这一点。

There are two places where you have the most patents.

有两个地方拥有最多的专利。

It’s everything related to 3G, 4G, 5G, RF, and multimedia.

它涉及与 3G、4G、5G、RF 和多媒体相关的一切。

Because it’s very mathematical, and you can get great gains and so on.

因为它非常数学化,你可以获得很大的收获等等。

So Google and Meta and Netflix wanted something where it was royalty-free.

因此,谷歌、Meta 和 Netflix 想要一些免版税的东西。

There are people who said that they have patents outside, but they are fringe patents, right?

有人说,他们在外面有专利,但那是边缘专利,对吧?

So it’s mostly true that it’s patent-free.

所以它基本上是无专利的。

Kieran Kunhya3:44:53

Oh, you should extend.

哦,你应该延长。

3:44:53 Patent, patent checking was done as part of the standardization process in AV1, AV2, whereas patents are not even discussed in the MPEG world.

专利、专利检查是作为 AV1、AV2 标准化过程的一部分进行的,而专利在 MPEG 世界中甚至没有讨论。

Jean-Baptiste Kempf3:45:06

The MPEG world.

MPEG 世界。

Kieran Kunhya3:45:06

Patents are off-topic completely.

专利完全是题外话。

Lex Fridman3:45:08

Can you educate me on the patent side?

您能告诉我专利方面的知识吗?

Jean-Baptiste Kempf3:45:10

So usually, MPEG does a format, right?

通常,MPEG 有一种格式,对吧?

3:45:10 And then everyone comes around and says, “Well, I have all those patents for the format,” and they do usually a union called what’s called MPEG LA, MPEG Licensing Association.

然后每个人都会过来说,“好吧,我拥有该格式的所有专利”,他们通常会成立一个名为 MPEG LA(MPEG 许可协会)的联盟。

And you put all your patents in, and then you ask everyone who’s using this format to pay for it.

然后你把所有的专利都放进去,然后你要求每个使用这种格式的人付费。

Lex Fridman3:45:31

Wait, can you elaborate?

等等,你能详细说明一下吗?

3:45:31 What does it mean to have a patent for a codec?

拥有编解码器专利意味着什么?

Why are there many patents?

为什么有这么多专利?

Jean-Baptiste Kempf3:45:36

Imagine I’m doing something where I’m going to def— instead of doing blocks which are square, I’m going to do rectangles, right?

想象一下,我正在做一些我要定义的事情,而不是做正方形的块,我要做矩形,对吗?

Lex Fridman3:45:44

Oh, so every idea- …

哦,所以每个想法-...

3:45:44 somebody patents it.

有人为其申请了专利。

Oh, man.

哦,伙计。

Oh, man.

哦,伙计。

People and their…

人们和他们的……

How many lawyers are-

有多少律师——

Kieran Kunhya3:45:56

I mean, it pays for a lot of lawyers, right?

我的意思是,它支付了很多律师的费用,对吗?

3:45:56 Like-

就像-

Jean-Baptiste Kempf3:45:58

The biggest issue is not the following, right?

最大的问题不是下面这个吧?

3:45:58 Because at time of H.264, the patents were, let’s call it, like, sane.

因为在 H.264 时代,我们可以这么说,这些专利是健全的。

But there was so much money in that— …

但里面有那么多钱——……

that for HEVC, a lot—there were a ton of things that were pushed inside the specification, which are not useful in 99.9% of the time, but so just one could add a patent on it.

对于 HEVC 来说,有很多东西——规范中塞进了很多东西,这些东西在 99.9% 的情况下都是没有用的,但只要有人可以在上面添加一项专利。

And so it became that for HEVC licensing, there was MPEG LA plus another patent pool called HEVC Advance.

因此,对于 HEVC 许可,有 MPEG LA 以及另一个名为 HEVC Advance 的专利池。

Plus-

加上-

Kieran Kunhya3:46:31

That one

那个

Jean-Baptiste Kempf3:46:31

…

……

3:46:31 I think Nokia was outside of the patent pool.

我认为诺基亚不在专利池之内。

Kieran Kunhya3:46:34

Yeah, a few of them are outside, and some other one that’s-

是的,其中一些在外面,还有一些是——

Jean-Baptiste Kempf3:46:36

And so it was impossible to license, right?

所以不可能获得许可,对吗?

3:46:36 And I think that several months ago, HP decided that they were going to remove support from HEVC in their Windows laptops because the, the cost was increasing of those patents.

我认为几个月前,惠普决定在 Windows 笔记本电脑中取消对 HEVC 的支持,因为这些专利的成本不断增加。

And it arrived—

它来了——

Lex Fridman3:46:52

Nice

尼斯

Jean-Baptiste Kempf3:46:52

…

……

3:46:52 where a point where—And there was uncapped pay.

到了某个点——而且工资没有上限。

And so for YouTube or Netflix, we could talk about hundreds of millions of dollars of licensing for patents per year.

因此,对于 YouTube 或 Netflix,我们每年可以谈论数亿美元的专利许可。

And they said, “You know what?

他们说:“你知道吗?

At hundred million per year, you know, I could create my own codec,” and this is what they did.

每年一亿美元,你知道,我可以创建自己的编解码器,”这就是他们所做的。

And so that’s why we have the Alliance for Open Media, where we are part of, that created AV1 and creates AV2.

这就是为什么我们成立开放媒体联盟(我们是该联盟的一部分),该联盟创建了 AV1 并创建了 AV2。

We create also audio codecs.

我们还创建音频编解码器。

But yes.

但是是的。

So the main difference would be that, and because you need to work around the patents or go do some things that are not patented, a lot of things are different, right?

所以主要的区别在于,因为你需要解决专利问题或者去做一些没有专利的事情,所以很多事情都是不同的,对吗?

The basic things that were done in MPEG-2 thirty years ago are, of course, out of patents.

当然,三十年前在 MPEG-2 中完成的基本工作已经没有专利了。

But so for example, there is things like a golden frame, a S-frame, or, or different type of—

但举例来说,有一些东西,比如金色框架、S 框架,或者,或者不同类型的——

Lex Fridman3:47:46

These are all patented ideas.

这些都是专利想法。

Kieran Kunhya3:47:48

Yeah, no, it’s I can’t believe it’s not butter.

是的,不,我不敢相信这不是黄油。

3:47:48 I can’t believe it’s not a B-frame.

我不敢相信这不是 B 型框架。

It’s, I mean, I mean, it’s kind of what it is.

我的意思是,我的意思是,就是这样。

In some ways, it’s like a-

在某些方面,它就像——

Lex Fridman3:47:55

Oh, so it’s a different variant of a B-frame.

哦,所以它是 B 型框架的不同变体。

Kieran Kunhya3:47:57

Yeah, that’s to try and sidestep.

是的,那就是尝试回避。

3:47:57 Things like that.

诸如此类的事情。

Jean-Baptiste Kempf3:48:00

And so you need to have double creativity, right?

所以你需要有双重创造力,对吗?

3:48:00 Creativity in terms of being more efficient, but creativity of being sure that you don’t infringe existing patents.

创造力在于提高效率,但创造力在于确保不侵犯现有专利。

And so, for example, VVC is, has all the patents of HEVC plus new ones, right?

例如,VVC 拥有 HEVC 的所有专利以及新专利,对吗?

It’s why AV2 tries to be as royalty-free as possible.

这就是 AV2 尝试尽可能免版税的原因。

Lex Fridman3:48:20

To what degree does FFmpeg and VLC have to think about this kind of stuff?

FFmpeg 和 VLC 需要在多大程度上考虑这种事情?

Jean-Baptiste Kempf3:48:24

We don’t, and one of the reasons why VLC was in France is that France rejects software patents.

我们不这样做,VLC 出现在法国的原因之一是法国拒绝软件专利。

3:48:24 So most of those patents are illegal in France because I once made the calculus that if I had to pay all the licensing fee for VLC, I needed to pay more than two hundred euros per user, right?

所以这些专利大部分在法国都是非法的,因为我曾经算过,如果我要支付VLC的所有许可费,那么我需要为每个用户支付两百多欧元,对吧?

It’s the same in dollars.

换算成美元也是一样的。

But most of those patents are invalid in Europe because those are called—it’s basically mathematical patents or idea patents, and they are not valid in Europe.

但这些专利大多数在欧洲无效,因为这些专利基本上是数学专利或创意专利,它们在欧洲无效。

Lex Fridman3:49:00

Let me just at a high level, just out of curiosity.

让我处于一个高水平,只是出于好奇。

3:49:00 So the meme online and the interwebs on X and Twitter and so on, and my own—I have friends in Europe—this, the sense is that Europe is not friendly to entrepreneurship.

所以网上的模因以及 X 和 Twitter 上的互联网等等,还有我自己的——我在欧洲有朋友——这,感觉是欧洲对创业并不友好。

They over-regulate, there’s too much bureaucracy, and so on.

他们监管过度,官僚主义太多,等等。

Is, is there any- anything positive to say?

有什么积极的事情要说吗?

Is there hope for entrepreneurship- …

创业还有希望吗——...

in the future of Europe?

欧洲的未来?

Is Europe over from a tech perspective?

从科技角度来看,欧洲已经结束了吗?

Kieran Kunhya3:49:32

Just, just look at the two of us, right?

只是,就看看我们两个吧?

3:49:32 It’s, it’s notable that there’s two people from the European continent on this podcast talking about video.

值得注意的是,这个播客中有两个来自欧洲大陆的人正在谈论视频。

It’s fair to say the community is weighted heavily.

可以公平地说,社区的权重很大。

Jean-Baptiste Kempf3:49:42

What you probably don’t see yet is that there is a new generation of entrepreneurs in Europe and mostly in France.

您可能还没有看到的是,欧洲有新一代企业家,其中大部分在法国。

3:49:42 UK has done it since a long time because, well, it’s more—it’s more Anglo-Saxon type of business.

英国长期以来一直这样做,因为它更像是盎格鲁-撒克逊类型的企业。

But especially like what happened in France, and of course, sometimes a bit overdone with everything called French Tech, but today, most of the people who come on the market want to create startups.

但尤其像法国发生的事情,当然,有时所谓的法国科技有点过头了,但今天,大多数进入市场的人都想创建初创公司。

Fifteen years ago, it wasn’t the case.

十五年前,情况并非如此。

Everyone wanted to work on big companies because when you failed in France, for example, twenty years ago, fifteen years ago, and you destroy your company, which is normal for a startup, right?

每个人都想在大公司工作,因为当你在法国失败时,例如二十年前、十五年前,你毁掉了你的公司,这对于初创公司来说很正常,对吧?

You were not allowed to create a new company, right?

你不被允许创建一家新公司,对吗?

There was a lot of stigma.

有很多耻辱。

The stigma is gone.

耻辱消失了。

3:50:32 …

……

there is so many things happening on AI in France and so on, right?

法国在人工智能方面发生了很多事情,对吧?

So there is, sure, over-regulations.

所以,肯定存在过度监管的情况。

I know that, right?

我知道,对吧?

I’m an entrepreneur.

我是一名企业家。

But it has some good things also.

但它也有一些好处。

Lex Fridman3:50:45

I mean, is there some paralyzing aspects?

我的意思是,是否有一些令人瘫痪的方面?

3:50:45 You know, if I look at, at the case of somebody I’ve become close with, Pavel Durov, you know, he was uh, blamed directly by the French government for the kind of things his, quote, “platform” was hosting.

你知道,如果我看一下与我关系密切的人帕维尔·杜罗夫(Pavel Durov)的案例,你知道,他因为他的“平台”所主持的事情而被法国政府直接指责。

Uh, I could see the same kind of stuff basically, just as an example, VLC being blamed for the kind of videos that people are watching.

呃,我基本上可以看到同样类型的东西,只是举个例子,VLC 被指责为人们正在观看的视频类型。

Jean-Baptiste Kempf3:51:13

But they tried, right?

但他们尝试过,对吗?

3:51:13 Like we had issues.

就像我们遇到了问题一样。

Like we-

就像我们——

Lex Fridman3:51:19

I mean, is that, that’s the pressure that people worry about because if you have to think about that kind of stuff when you’re kind of just obsessed about-

我的意思是,这就是人们担心的压力,因为如果你在沉迷于某种事情时必须考虑这类事情——

Jean-Baptiste Kempf3:51:25

No, you don’t think about it- …

不,你想都没想——……

3:51:25 and that’s, that’s okay, right?

那就是,没关系,对吧?

Like-

就像-

Lex Fridman3:51:29

But what if they come in?

但如果他们进来怎么办?

3:51:29 When, what if they show up and-

如果他们什么时候出现并且——

Jean-Baptiste Kempf3:51:31

There is no office.

没有办公室。

3:51:31 VideoLAN doesn’t have an office.

VideoLAN 没有办公室。

Lex Fridman3:51:33

I mean, this is what happened with Pavel.

我的意思是,这就是发生在帕维尔身上的事情。

3:51:33 They arrested him, right?

他们逮捕了他,对吗?

So arrested him for particular videos or, or a particular content that’s being shared on the platform.

因此,因平台上分享的特定视频或特定内容而逮捕他。

Jean-Baptiste Kempf3:51:42

Sure.

当然。

3:51:42 I don’t have any platform.

我没有任何平台。

Everything is on the client side.

一切都在客户端。

Lex Fridman3:51:45

Yeah, but they’re, they can still arrest you.

是的,但是他们仍然可以逮捕你。

Jean-Baptiste Kempf3:51:47

On what ground?

依据什么?

3:51:47 I’m not sharing anything.

我不会分享任何东西。

I’m not– The content doesn’t go through my stuff.

我不是——内容不会经过我的东西。

Lex Fridman3:51:52

For sure, but it’s still lawyer fees.

当然可以,但仍然是律师费。

3:51:52 That’s the problem.

这就是问题所在。

Jean-Baptiste Kempf3:51:55

Yes, that’s correct.

是的,这是正确的。

Lex Fridman3:51:55

It’s paperwork.

这是文书工作。

3:51:55 So like, actually, if you had infinite trillions of dollars you would win easily because you’re on the right side.

事实上,如果你有无限数万亿美元,你会很容易获胜,因为你站在正确的一边。

But the thing is, there is a degree to which they suffocate you with paperwork.

但问题是,他们在一定程度上让你因文书工作而感到窒息。

That’s the downside of bureaucracy, through paperwork, through process.

这就是官僚主义的缺点,通过文书工作和流程。

You know, it’s the Kafkaesque thing.

你知道,这是卡夫卡式的事情。

Jean-Baptiste Kempf3:52:17

You have to realize that one of the good things, for example in France or most of Europe, is that the—Answering to a court order does not make you bankrupt, right?

你必须意识到,例如在法国或欧洲大部分地区,一件好事是——回应法院命令并不会让你破产,对吧?

3:52:17 It’s not like in the US, where it can actually bankrupt you, right?

这不像在美国,它实际上会让你破产,对吧?

There is—The way the law system works is that, like I receive lawyers’ letters every week, right?

有——法律制度的运作方式是这样的,就像我每周都会收到律师信一样,对吗?

And I can tell you that the cost of lawyer fees for VideoLAN is less than ten thousand dollars per year, right?

而且我可以告诉你,VideoLAN的律师费每年不到一万美元吧?

Right?

正确的?

So that’s not really scary.

所以这并不可怕。

Lex Fridman3:52:53

I mean, similar with Pavel.

我的意思是,与帕维尔类似。

3:52:53 The intelligence agencies tried to like say, “Can you put a backdoor in VLC?”

情报机构试图问,“你能在 VLC 中设置后门吗?”

Jean-Baptiste Kempf3:52:59

Yes.

是的。

3:52:59 Two of them.

其中两个。

Lex Fridman3:53:01

What, what do you say?

什么,你说什么?

Jean-Baptiste Kempf3:53:02

No.

不。

3:53:02 Well, I was a lot less polite.

嗯,我没那么有礼貌。

Lex Fridman3:53:05

I see…

我明白了……

3:53:05 Yeah, yeah.

是啊,是啊。

You’re basically saying, “Hell no.”

你基本上是在说:“天哪,不。”

Jean-Baptiste Kempf3:53:09

Like, if we had to compromise our software, we would shut it down.

就像,如果我们必须破坏我们的软件,我们就会将其关闭。

3:53:09 This is clear.

这很清楚。

Lex Fridman3:53:13

And what’s the definition of compromise?

妥协的定义是什么?

3:53:13 Like allowing a government to do a backdoor-

就像允许政府开后门一样——

Jean-Baptiste Kempf3:53:19

There is no code that gets into VLC that we don’t control, and the way we compile VLC, you would call me completely paranoid.

进入 VLC 的所有代码都是我们无法控制的,而且我们编译 VLC 的方式,你可能会说我完全是偏执狂。

3:53:19 Like, we compile on boxes that are offline, where we start by compiling the compiler.

就像,我们在离线的机器上进行编译,首先编译编译器。

We do everything offline on places that have never been connected to the internet.

我们在从未连接到互联网的地方离线完成所有事情。

We— The way we do signing, there is double signature.

我们——我们的签名方式是双重签名。

And especially because, for example, we’ve seen, and we believe it’s a governmental agency that is not from the Western world who tried to push a fake binary into our own servers and that scared us a lot.

尤其是因为,例如,我们已经看到,并且我们相信这是一个不是来自西方世界的政府机构试图将假二进制文件推送到我们自己的服务器中,这让我们非常害怕。

And VideoLAN is open source.

VideoLAN 是开源的。

How can you kill it?

你怎么能杀掉它呢?

Like, I move to where?

比如,我搬到哪里去?

I move to Malta.

我搬到马耳他。

3:54:06 I move to, I don’t know, Cayman Islands, and I change the domain name, and I start again, right?

我搬到,我不知道,开曼群岛,我更改了域名,然后我重新开始,对吗?

Like, VLC is a tool.

就像,VLC 是一个工具。

It’s a tool that is going to help people doing things.

它是一个可以帮助人们做事的工具。

We are not a platform.

我们不是一个平台。

And for patents, well, I’m sorry, but most of the patents…

至于专利,好吧,我很抱歉,但大多数专利......

Like, you shouldn’t be able to patent math and matrices.

就像,你不应该能够为数学和矩阵申请专利。

Like, this is wrong.

就像,这是错误的。

Lex Fridman3:54:32

So does VLC ever, like, censor the kind of videos it can play and not based on the content of the video?

那么,VLC 是否曾经审查过它可以播放的视频类型,而不是基于视频内容?

Jean-Baptiste Kempf3:54:39

No, never.

不,从来没有。

3:54:39 We never do that.

我们从不这样做。

Because VLC is completely offline.

因为VLC完全离线。

It doesn’t talk to any server, so we don’t know anything that you’re using the software for.

它不与任何服务器通信,因此我们不知道您使用该软件的任何用途。

Lex Fridman3:54:49

So again, there’s no government that can say, you know, like the French government come in and say, “We don’t want…

再说一遍,没有哪个政府可以像法国政府那样说:“我们不想……

3:54:49 I think anime is destructive to society.

我认为动漫对社会有破坏性。

We don’t want any anime; not allowed to be…”

我们不想要任何动画;不被允许……”

Jean-Baptiste Kempf3:55:02

No, they cannot, they cannot do that.

不,他们不能,他们不能那样做。

3:55:02 And also what they tried is to say, “Hey, I want to know if that person watched that type of video.” And the answer is like, “No idea.”

他们还试图说,“嘿,我想知道那个人是否观看了此类视频。”答案是“不知道”。

Lex Fridman3:55:11

So no on that too.

所以也没有。

3:55:11 So for surveillance, no.

所以对于监视来说,没有。

Jean-Baptiste Kempf3:55:14

No, no, because the only infrastructure we have is a downloading infrastructure.

不,不,因为我们唯一的基础设施是下载基础设施。

3:55:14 There is no telemetry in VLC, right?

VLC 中没有遥测功能,对吗?

Lex Fridman3:55:20

It would be difficult ’cause of the international nature.

由于国际性质,这将是困难的。

3:55:20 It would be difficult for you to incorporate that code because there would be someone in the UK and someone in Germany and someone in the US as part of VideoLAN who’d be able to see that.

您很难合并该代码,因为作为 VideoLAN 的一部分,英国、德国和美国的某人能够看到该代码。

It would be extremely difficult.

这将是极其困难的。

Jean-Baptiste Kempf3:55:33

The only thing that we can do, which happened, is like we had the issue— We had the case with some police in the US who said, “We have a murder case,” right?

我们唯一能做的事情就是发生了,就像我们遇到了这个问题——我们与美国的一些警察发生了一起案件,他们说,“我们有一起谋杀案,”对吧?

3:55:33 And the file is destructed or doesn’t play in that version of VLC.

并且该文件已被破坏或无法在该版本的 VLC 中播放。

Could you help us?” Right?

你能帮助我们吗?对吗?

We never have access to the video.

我们永远无法访问该视频。

It’s like a normal support, right?

就像普通的支持一样,对吧?

Lex Fridman3:55:50

Oh, it’s really about playing the file?

哦,真的是播放文件吗?

Jean-Baptiste Kempf3:55:51

Yes.

是的。

3:55:51 And, like, I remember in the middle of the Afghan War, right?

而且,就像,我记得在阿富汗战争中期,对吧?

I received an email from someone in the army, right?

我收到了一封来自军队某人的电子邮件,对吗?

I don’t remember the grades, right?

我不记得成绩了,对吗?

It was just like, “We have a big issue with the latest version of VLC because it doesn’t play correctly the file on an RTSP server that we have where there is all the movies.” And he says VLC is very important for the morale on the troop on the ground, right?

就像,“我们对最新版本的 VLC 有一个大问题,因为它无法正确播放我们拥有的包含所有电影的 RTSP 服务器上的文件。”他说 VLC 对于地面部队的士气非常重要,对吧?

Because at night I think it might be boring, right?

因为晚上我觉得可能很无聊,对吧?

So they have a collection of videos to watch or movies over there, right?

所以他们在那里有一系列视频或电影可供观看,对吗?

So and, and of course I did an update, and I broke some support of RTSP, right?

当然,我进行了更新,并且破坏了一些 RTSP 支持,对吧?

So I gave them another version just for them, right?

所以我专门为他们提供了另一个版本,对吗?

Because it was important.

因为这很重要。

3:56:31 And because VLC is completely open source, I think it is allowed on the US Army laptops, right?

而且由于 VLC 是完全开源的,我认为美国陆军笔记本电脑上允许使用它,对吗?

Because the, I guess someone in the US military actually looked at it and say, “Well, okay, this is okay,” right?

因为,我猜美国军方确实有人看了它并说,“好吧,这没关系,”对吗?

And the way we document how we process, that was okay, right?

我们记录处理过程的方式也可以,对吧?

So the only way we work with authorities is to help them doing support.

因此,我们与当局合作的唯一方式就是帮助他们提供支持。

Lex Fridman3:56:53

That’s amazing.

太棒了。

3:56:53 That’s an amazing story.

这是一个了不起的故事。

Yeah.

是的。

Jean-Baptiste Kempf3:56:56

We don’t see anything happening on how people use VLC, and this is strong.

我们没有看到人们如何使用 VLC 发生任何变化,而且这很强大。

Lex Fridman3:57:00

Do you feel the stress of this?

你感受到这样的压力吗?

3:57:00 So first of all, millions of people using it.

首先,有数百万人使用它。

Second of all, the military using it.

其次,军方使用它。

Maybe sometimes pressure from governments.

也许有时来自政府的压力。

Does that…

那是吗……

That’s a small team, right?

这是一个小团队,对吗?

Jean-Baptiste Kempf3:57:16

Yeah, but-

是的,但是-

Lex Fridman3:57:16

How big is VLC core…

VLC核心有多大...

3:57:16 how many?

有多少?

Jean-Baptiste Kempf3:57:21

Six, eight.

六、八。

3:57:21 But everything legally is only me.

但合法的一切都只有我一个人。

Everything that is legal is only me.

一切合法的东西都只有我一个人。

Lex Fridman3:57:28

You’re not stressed about this?

你对此没有压力吗?

Jean-Baptiste Kempf3:57:30

I used to stress about that a lot.

我曾经对此非常强调。

3:57:30 But the thing is, we’re doing what we can for everyone, for the greater good.

但事实是,我们正在为每个人、为了更大的利益尽我们所能。

We work so that we make some extremely complex technology easy for everyone.

我们致力于让一些极其复杂的技术对每个人来说都很容易。

We’re a tool, and every tool is going to be used for great things and for bad things, right?

我们是一个工具,每个工具都会被用来做好事和坏事,对吗?

You cannot blame a tool, I think.

我认为你不能责怪工具。

And this is, like, very important for us.

这对我们来说非常重要。

I used to be in a lot of stress.

我曾经压力很大。

I’m not anymore, right?

我不再是了,对吧?

Lex Fridman3:58:00

What’s the secret to your zen?

你的禅宗秘诀是什么?

3:58:00 I mean, over and over in the chats I’ve had with you in the conversation today about every even tense topic, you’re very zen.

我的意思是,在我今天与你的谈话中,关于每一个甚至紧张的话题,你都非常禅宗。

What’s the source of zen?

禅宗的根源是什么?

Jean-Baptiste Kempf3:58:14

I have a way of thinking about what is the worst case scenario, always, right?

我总是有办法思考最坏的情况是什么,对吗?

3:58:14 And the answer is, at the end, if I take like a chess player, right?

答案是,最后,如果我像一个国际象棋棋手一样,对吗?

In the end, am I dead?

到最后,我死了吗?

Yes or no?

是还是不是?

Right?

对吗?

And I do that nonstop, right?

我不停地这样做,对吗?

3:58:33 And that’s also how I do my startups, right?

这也是我创业的方式,对吗?

Is that I’m here to get something right.

我来这里是为了把事情做好。

What is the worst case?

最坏的情况是什么?

It goes bankrupt.

它破产了。

That’s life.

这就是生活。

A company lives, a company dies.

一个公司生,一个公司死。

That’s okay, right?

没关系,对吧?

Like, and so my moral way is always like, am I dying in the end?

就像,所以我的道德方式总是这样,我最终会死吗?

Am I hurting someone?

我伤害别人了吗?

If the answer is no, then too bad, right?

如果答案是否定的,那就太糟糕了,对吧?

Like, oh, some lawyers are going to be unhappy.

就像,哦,有些律师会不高兴。

What are they going to do?

他们要做什么?

Take all the money of VideoLAN?

把VideoLAN的钱都拿走了?

Wow.

哇。

They’re going to have 50 grand.

他们将有 50 格兰德。

Amazing, right?

太棒了,对吧?

What are they going to do with that?

他们打算用它做什么?

The code– the source code is out there.

代码——源代码就在那里。

It’s not stoppable.

这是无法阻止的。

Also because what we do is good and it’s done for everyone.

还因为我们做的事情是好的,而且是为了每个人。

Lex Fridman3:59:14

That’s beautiful.

太漂亮了。

3:59:14 Kieran, you said that there’s an active archiving preservation community?

Kieran,你说有一个活跃的归档保存社区?

I think that’s super fascinating.

我认为这非常令人着迷。

You wrote that they’re stretched in budget, but they see the extreme importance of FFmpeg as a Rosetta Stone so that multimedia can be played a thousand years from now.

您写道,他们的预算捉襟见肘,但他们看到了 FFmpeg 作为罗塞塔石碑的极端重要性,这样多媒体就可以在一千年后播放。

I mean, that’s a beautiful way to see FFmpeg and VLC as a tool for preserving visual knowledge.

我的意思是,这是将 FFmpeg 和 VLC 视为保存视觉知识的工具的一种美妙方式。

Kieran Kunhya3:59:41

Yes, that’s right.

是的,没错。

3:59:41 One of the coolest communities in open source multimedia, mainly led by someone called Dave Rice, I’ll give him a shout-out, I think from City University of New York, is the archiving community.

开源多媒体领域最酷的社区之一,主要由一个叫 Dave Rice 的人领导,我要向他致敬,我想来自纽约城市大学,是归档社区。

They’ve done so much stuff.

他们做了很多事情。

They value open source, one, because yes, they lack budgets, but two, they see the fact that archiving video is important for the world, and being able to play that is a big problem.

他们重视开源,一是因为他们缺乏预算,二是他们看到存档视频对世界很重要,而能够播放视频是一个大问题。

Famously in the UK, there was something called the New Domesday Book, and they archived lots of stuff on BBC microcomputers.

在英国,有一种叫做《新末日审判书》的书,他们在 BBC 微型计算机上存档了很多东西。

Within 10 to 15 years, no one had the right software to play that.

在 10 到 15 年内,没有人拥有合适的软件来玩这个游戏。

4:00:19 I think it was 20 years or something like that, and someone had to go and reverse engineer this, and that was like 20 years.

我认为那是 20 年左右的时间,有人必须对其进行逆向工程,那大约是 20 年。

Imagine that in a thousand years.

想象一下一千年后的情况。

I think one of the great things about FFmpeg is it’s written in C.

我认为 FFmpeg 的伟大之处之一是它是用 C 编写的。

C is the closest to mathematics you’re probably gonna get.

C 语言可能是你能学到的最接近数学的语言。

The closest to logic is-

最接近逻辑的是——

Jean-Baptiste Kempf4:00:34

Do you think in 1,000 years we’d still have C compilers?

你认为 1000 年后我们还会有 C 编译器吗?

Kieran Kunhya4:00:37

Yes.

是的。

4:00:37 We have languages that exist that haven’t changed too much.

我们现有的语言并没有发生太大变化。

We have mathematical notation that exists.

我们有存在的数学符号。

It will be like Latin.

它将像拉丁语一样。

C will be like Latin.

C 将像拉丁语一样。

It will be a thing that you learn from the past, but it will still be usable in certain contexts.

这将是你从过去学到的东西,但它在某些情况下仍然可用。

So the archiving community are really great practically.

因此,归档社区实际上非常棒。

They, again, limited funds.

他们再次限制资金。

They funded the development of the FFV1 codec, so that’s a lossless codec.

他们资助了 FFV1 编解码器的开发,因此这是一种无损编解码器。

So the archiving community is really scared about the act of compression losing things, and they have a fair point in this, you know.

所以归档社区真的很害怕压缩丢失东西的行为,你知道,他们在这方面有一个合理的观点。

4:01:08 If they compress too hard, it could change the view of the material.

如果它们压缩得太紧,可能会改变材料的外观。

There could be something slightly different here and there, so they’re really concerned about things.

各地可能会有些许不同,所以他们真的很关心事情。

They need to be not just compressed well, but lossless and be fast.

它们不仅需要压缩得很好,而且还需要无损且速度快。

And so they worked with FFmpeg to develop a whole new codec designed for fast software-based encoding.

因此,他们与 FFmpeg 合作开发了一种全新的编解码器,专为基于软件的快速编码而设计。

They’re really concerned about resilience, so if they’re storing on tapes or other hard disks, I lose some bits, I need to recover quickly.

他们真的很关心弹性,所以如果他们存储在磁带或其他硬盘上,我会丢失一些位,我需要快速恢复。

I can’t lose a whole GOP because I’ve lost a bit-

我不能失去整个共和党,因为我已经失去了一点——

4:01:39 …

……

something like that.

类似的事情。

So they’re a really great bunch of people.

所以他们是一群非常棒的人。

They funded GPU encoding in FFmpeg to make FFV1 encode faster.

他们资助 FFmpeg 中的 GPU 编码,以使 FFV1 编码速度更快。

And it’s really about preserving the world’s multimedia heritage in a way that’s usable, and there’s a lot of great teams and a lot of archival groups across the world who’ve chosen FFmpeg and FFV1 as their archiving solution.

这实际上是为了以可用的方式保存世界多媒体遗产,世界各地有很多优秀的团队和档案团体选择 FFmpeg 和 FFV1 作为他们的存档解决方案。

And they can really provide us also super specialist advice.

他们确实可以为我们提供超级专业的建议。

They can- …

他们可以——……

explain, “Ah, in the 1950s, colorimetry was done like this on this certain type of tape, and so there is this special case that you need to handle, and you’ll never get this anywhere else.”

解释说,“啊,在 20 世纪 50 年代,比色法是在某种类型的胶带上进行的,因此您需要处理这种特殊情况,而您在其他任何地方都找不到这种情况。”

Jean-Baptiste Kempf4:02:20

You see, they know things on video that we don’t.

你看,他们在视频中知道我们不知道的事情。

4:02:20 Like, every time I talk to, was it Dave-

就像,每次我谈话时,是戴夫——

Kieran Kunhya4:02:26

Dave Rice

戴夫·赖斯

Jean-Baptiste Kempf4:02:27

…

……

4:02:27 or, or the people from the British, uh-

或者,或者来自英国的人,呃-

Kieran Kunhya4:02:29

British Film, uh

英国电影呃

Jean-Baptiste Kempf4:02:30

…

……

4:02:30 Film, it’s just like every time I just learn something new, and I’ve been doing video for 20 years.

电影,就像每次我都会学到新东西一样,我做视频已经有20年了。

They have, especially on colorimetry and colors.

他们有,特别是在比色法和颜色方面。

Kieran Kunhya4:02:39

Storage, these other things.

存储,这些其他的东西。

Lex Fridman4:02:40

I mean, they have a deep, deep appreciation of the content itself, of the video itself.

我的意思是,他们对内容本身、视频本身有非常非常深刻的欣赏。

4:02:40 And like, especially when you’re thinking of lossless, they’re terrified of losing something essential- …

就像,尤其是当你想到无损时,他们害怕失去一些重要的东西——……

about the thing, and in so doing, they’re deeply understanding the thing that is to be preserved, which you sometimes might not be thinking about when you’re-

这样做,他们深刻地理解了要保存的东西,而当你-时,你有时可能不会想到这一点

Kieran Kunhya4:03:00

Absolutely.

绝对的。

Lex Fridman4:03:00

…

……

4:03:00 obsessing about the actual technology of the encoding and so on.

痴迷于编码的实际技术等等。

Jean-Baptiste Kempf4:03:03

And when you enter the rabbit hole of film scanners, right?

当您进入胶片扫描仪的兔子洞时,对吧?

4:03:03 So you take those- …

所以你拿那些-...

those things to make to digital, like, it’s like- …

那些要数字化的东西,比如,就像——……

a huge topic that, like, would take another five hours of podcast- …

一个巨大的话题,比如,需要另外五个小时的播客——……

just on that topic.

就在那个话题上。

Kieran Kunhya4:03:18

On film, and there’s a lot of film that needs to be archived.

在电影方面,有很多电影需要存档。

4:03:18 Film is degrading.

电影有辱人格。

It’s maybe not stored in the right environment.

它可能没有存储在正确的环境中。

The other thing is they can…

另一件事是他们可以……

What they also do is, because it’s open source, they give this away, their workflows, to countries who can’t afford to have archiving institutions, where archiving is done by volunteers, it’s done by other things.

他们还做的是,因为它是开源的,所以他们将其工作流程赠送给那些无力建立归档机构的国家,在那里归档是由志愿者完成的,是由其他机构完成的。

They go and teach, you know, in India, they teach children to do FFmpeg commands.

他们去教,你知道,在印度,他们教孩子们执行 FFmpeg 命令。

They’re really great.

他们真的很棒。

They’re really, they’re really the model community, the model ethos of what we’re trying to achieve.

他们真的是模范社区,是我们想要实现的目标的模范精神。

They are such a great bunch of people, so interested in participating and being part of something much bigger because they realize the work they’re doing in a thousand years is gonna tell a lot.

他们是一群很棒的人,对参与和成为更大的事情的一部分非常感兴趣,因为他们意识到他们在一千年中所做的工作将会说明很多事情。

4:03:59 You know, in a thousand years we may be drowning in AI slop.

你知道,一千年后我们可能会淹没在人工智能的废液中。

This stuff needs to be important and, you know, archived well.

这些东西必须很重要,而且要妥善存档。

What was life like?

生活是什么样的?

Lex Fridman4:04:07

Yeah, it feels like capturing the 20th century and the 21st century is essential because it feels like a transition point, where we went from scarcity of data to slop- …

是的,感觉捕捉 20 世纪和 21 世纪是至关重要的,因为它感觉像是一个过渡点,我们从数据稀缺到数据匮乏……

4:04:07 oceans of slop, and that transition point is good to archive.

海洋的污水,这个转变点很适合存档。

Kieran Kunhya4:04:24

It’s important, yeah.

这很重要,是的。

Jean-Baptiste Kempf4:04:25

But people don’t realize we are losing today a ton of films.

但人们没有意识到我们今天正在失去大量电影。

4:04:25 There is a ton of things from the ’30s, from the ’40s, and the ’50s that where there is no value-

有很多 30 年代、40 年代和 50 年代的东西没有价值——

Kieran Kunhya4:04:37

And tape.

还有胶带。

4:04:37 ’70s and ’80s, there’s tape, and there’s not enough tape heads in the world-

70 年代和 80 年代,有磁带,但世界上没有足够的磁头 -

Jean-Baptiste Kempf4:04:41

To read all the tapes.

阅读所有磁带。

Kieran Kunhya4:04:42

…

……

4:04:42 left to redo, so they have to decide what they want to archive and throw away the rest of the tapes.

需要重做,因此他们必须决定要存档的内容并扔掉其余的磁带。

There’s huge moral hazard, I guess for want of a better phrase, around this topic because this is a digital record of human history and they have to make decisions that…

我想,围绕这个话题存在巨大的道德风险,因为这是人类历史的数字记录,他们必须做出决定……

And there’s digital stewardship, I suppose, for want of…

我想,还有数字管理,因为缺乏……

I made that phrase up.

这句话是我编造的。

That’s not a real phrase.

这不是一个真正的短语。

To make sure the world can have this information in something that’s playable by everybody, not- …

为了确保世界可以在每个人都可以玩的东西中获得这些信息,而不是……

playable on some device that, well, it doesn’t exist anymore.

可以在某些设备上播放,但它已经不存在了。

Lex Fridman4:05:13

And then there’s like, realistically speaking, there’s a needle in a haystack where there’s a lot of value in archiving all that footage, and then over time finding the gems- …

然后,现实地说,就像大海捞针一样,归档所有这些镜头有很大的价值,然后随着时间的推移找到宝石——……

4:05:13 that we don’t know are there.

我们不知道那里有什么。

Kieran Kunhya4:05:25

Hey, there was something in that corner that we just didn’t- And that would’ve been compressed away because it was some little thing.

嘿,那个角落里有一些我们没有的东西——而且它会被压缩掉,因为它是一些小东西。

4:05:25 Oh, wow, there’s something there.

哦,哇,那里有东西。

Lex Fridman4:05:33

That’s it.

就是这样。

Kieran Kunhya4:05:33

And that’s…

那是……

4:05:33 They’ve made sure that it’s lossless.

他们确保它是无损的。

They can prove mathematically that it’s lossless.

他们可以从数学上证明它是无损的。

They can run different trade-offs for if there’s bit…

如果有的话,他们可以进行不同的权衡……

if they lose a bit, a single bit flips, I can make sure that I only lose a portion of a given frame.

如果他们丢失了一点,一个比特翻转,我可以确保只丢失给定帧的一部分。

We can do error recovery on previous frames.

我们可以对之前的帧进行错误恢复。

They can do all sorts of different things.

他们可以做各种不同的事情。

Lex Fridman4:05:52

Do you think VLC and FFmpeg will be here 100 years from now?

您认为 100 年后 VLC 和 FFmpeg 还会出现吗?

Jean-Baptiste Kempf4:05:57

FFmpeg, yes.

FFmpeg,是的。

Kieran Kunhya4:05:58

Yep, FFmpeg, yes.

是的,FFmpeg,是的。

Jean-Baptiste Kempf4:05:59

VLC, maybe.

VLC,也许吧。

Lex Fridman4:06:01

What’s the future of…

未来是什么……

4:06:01 Where is FFmpeg going?

FFmpeg 将走向何方?

Where is VLC going?

VLC 要去哪里?

Like in the next…

就像接下来的那样……

If you think about, like, five years, 10 years, 20 years.

如果你想想,五年、十年、二十年。

Jean-Baptiste Kempf4:06:10

Five years, 10 years is easy.

五年、十年很容易。

4:06:10 The question is after that, right?

问题是在那之后,对吗?

The question is- …

问题是——……

do we arrive at something called holograms, right?

我们能得到一种叫做全息图的东西吗?

Lex Fridman4:06:19

Yeah, so will VLC and FFmpeg expand- …

是的,VLC 和 FFmpeg 也会扩展——……

4:06:19 to whatever-

无论什么——

Jean-Baptiste Kempf4:06:26

Multimedia

多媒体

Lex Fridman4:06:27

…

……

4:06:27 multimedia- …

多媒体-...

so multimedia might become, I’m sorry for the pothead expansion of topic, but, you know, if you look at something like Neuralink with brain-computer interfaces, it’s very possible that we start to consume what multimedia means is whatever codec, whatever data that our brain wants to consume through the brain-computer interfaces.

所以多媒体可能会变成,我很抱歉主题的扩展,但是,你知道,如果你看看带有脑机接口的 Neuralink 之类的东西,我们很可能开始消费多媒体意味着什么编解码器,无论我们的大脑想要通过脑机接口消费什么数据。

That’s one.

这是一个。

Then virtual reality, of course.

当然还有虚拟现实。

Jean-Baptiste Kempf4:06:52

You will have VLC for Neuralink.

您将拥有适用于 Neuralink 的 VLC。

Kieran Kunhya4:06:55

Yep, and you’ll have FFmpeg -i input format human brain.

是的,你将拥有 FFmpeg -i 输入格式人脑。

Lex Fridman4:06:58

Yeah.

是的。

4:06:58 There’s gonna be codecs for the brain.

大脑将会有编解码器。

Kieran Kunhya4:07:01

Sure, 100%.

当然,100%。

4:07:01 Yeah, to compress neural information, yeah.

是的,压缩神经信息,是的。

Jean-Baptiste Kempf4:07:05

I mean, today there is like, there are new codecs for- …

我的意思是,今天有新的编解码器用于......

4:07:05 for example, what we call point cloud, right?

例如我们所说的点云,对吗?

Or volumetric videos, right?

或者体积视频,对吧?

There is a ton of research on what we call RGB-D, right?

关于 RGB-D 有大量的研究,对吗?

So codecs for depth that is useful for robotics and for 3D things.

因此,深度编解码器对于机器人技术和 3D 事物很有用。

There is a ton of codecs for compression of 3D elements.

有大量用于压缩 3D 元素的编解码器。

Kieran Kunhya4:07:22

Compression for astronomy.

天文学的压缩。

Jean-Baptiste Kempf4:07:23

For example, on VLC, we also have already a VR and XR version of VLC.

例如,在VLC上,我们也已经有了VLC的VR和XR版本。

4:07:23 And also on Kyber, right?

还有 Kyber,对吧?

We talk about Kyber.

我们谈论凯伯。

On Kyber, we also like do streaming of XR content on, for the glasses who cannot have enough power or inside the Apple Vision or the Quest.

在 Kyber 上,我们还喜欢为那些没有足够电量或在 Apple Vision 或 Quest 内的眼镜提供 XR 内容流。

So we already work on streaming 3D, XR, interactive, low latency.

所以我们已经致力于流媒体 3D、XR、交互式、低延迟。

There is something called volumetric video, point cloud videos, so it’s not stopping.

有一种叫做体积视频、点云视频的东西,所以它不会停止。

And yes, at some point it will manage 3D data inside VLC and FFmpeg, right?

是的,在某些时候它会管理 VLC 和 FFmpeg 内部的 3D 数据,对吧?

It’s obvious.

这是显而易见的。

Lex Fridman4:07:58

So that’s where it is moving, like the community is open.

这就是它的发展方向,就像社区是开放的一样。

Jean-Baptiste Kempf4:08:01

Not everyone in the community sees that, but as Kieran and I, we are entrepreneurs, we know where it’s going.

并不是社区中的每个人都看到了这一点,但作为基兰和我,我们是企业家,我们知道它的发展方向。

4:08:01 We see that, right?

我们看到了,对吧?

Lex Fridman4:08:10

So I suppose that there is a tension probably inside FFmpeg.

所以我想 FFmpeg 内部可能存在紧张。

4:08:10 It’s like, “Hey, listen, folks, we’re really good at doing video and audio, so like why expand?

就像,“嘿,听着,伙计们,我们真的很擅长制作视频和音频,那么为什么要扩展呢?

Like let’s do the thing we’re really good at doing.”

就像让我们做我们真正擅长做的事情一样。”

Jean-Baptiste Kempf4:08:25

In order to answer that question, we need to answer the definition of what is multimedia.

为了回答这个问题,我们需要回答什么是多媒体的定义。

4:08:25 And multimedia is a digital representation of several streams for the human senses.

多媒体是人类感官多种流的数字表示。

And we will do that, right?

我们会这么做,对吗?

So imagine there is now a way to not have a mic, but have a odor sensor- …

所以想象一下现在有一种方法可以不用麦克风,但有一个气味传感器——……

and a diffuser of odors.

和气味扩散器。

It will get into FFmpeg.

它将进入 FFmpeg。

Lex Fridman4:08:54

So your demuxer is coming up.

所以你的解复用器即将出现。

Jean-Baptiste Kempf4:08:56

Yes.

是的。

4:08:56 Yes.

是的。

Of course, your demuxer has a new track type that is basically odors, right?

当然,您的解复用器有一种新的轨道类型,基本上是气味,对吗?

And you already have-

而你已经-

Lex Fridman4:09:02

Smell, touch.

闻、摸。

Kieran Kunhya4:09:03

It’s like audio.

就像音频一样。

4:09:03 You’ll have a left and right nose track.

你会有左右鼻迹。

You have a left and right audio pair.

您有一对左右音频。

It’s easy.

这很容易。

Jean-Baptiste Kempf4:09:07

Yes, of course.

是的,当然。

Lex Fridman4:09:09

Stereo smell.

立体气味。

Kieran Kunhya4:09:09

Stereo smell, yeah.

立体声的味道,是的。

Jean-Baptiste Kempf4:09:10

So in VLC, for example, we already have a plugin for haptic.

例如,在 VLC 中,我们已经有一个触觉插件。

4:09:10 It’s mostly for what we call 4D cinemas, right?

主要是为了我们所说的 4D 影院,对吧?

You know, those ones on hydraulic—I don’t know how you say that—all the hydraulic-

你知道,那些液压的——我不知道你怎么说——所有的液压——

Kieran Kunhya4:09:21

Hydraulic arms.

液压臂。

4:09:21 Hydraulic,

液压、

Jean-Baptiste Kempf4:09:22

Arms.

武器。

4:09:22 And where everything is moving, like you have in theme parks, right?

一切都在移动,就像主题公园一样,对吧?

And there is a data feed synchronized where, which is basically transporting this information.

并且有一个同步的数据源,它基本上是在传输这些信息。

Lex Fridman4:09:32

Is there yet a standard for that?

这还有一个标准吗?

Jean-Baptiste Kempf4:09:33

There are many standards, right?

有很多标准,对吗?

Lex Fridman4:09:35

This is…

这是……

4:09:35 You make me so happy.

你让我如此高兴。

Jean-Baptiste Kempf4:09:38

And so of course, like we have a plugin which is not in the normal version of VLC-

当然,就像我们有一个 VLC 正常版本中没有的插件一样 -

Lex Fridman4:09:43

That’s good.

那很好。

Jean-Baptiste Kempf4:09:43

…

……

4:09:43 that is basically transporting those type of movements, which is physical movements, which is haptic movements, right?

这基本上是传输那些类型的运动,这是物理运动,这是触觉运动,对吧?

It is a human sense, so it will get in.

这是人类的感觉,所以它会进入。

Lex Fridman4:09:54

That’s such an exciting future.

这真是一个令人兴奋的未来。

4:09:54 Was it…

难道是……

I mean, it’s a small community of developers.

我的意思是,这是一个小型的开发人员社区。

How do you pull that off?

你如何做到这一点?

Like if you’re a contributor to FFmpeg or VLC, it feels stressful.

就像如果你是 FFmpeg 或 VLC 的贡献者一样,你会感到压力很大。

Like it, just looking on Twitter, it’s like a huge amount of work to make it work on all these different operating systems, an incredible effort.

就像它一样,只要在 Twitter 上看看,就需要做大量的工作才能使其在所有这些不同的操作系统上运行,这是一项令人难以置信的努力。

Jean-Baptiste Kempf4:10:17

No, see it in the other direction.

不,从另一个方向看。

4:10:17 We are not the contributors.

我们不是贡献者。

We are the maintainers, right?

我们是维护者,对吗?

So we maintain for everyone.

所以我们为大家维护。

Meaning that, for example, every year there is around 150 people who contribute to VLC and maybe 300 on FFmpeg, right?

这意味着,每年大约有 150 人为 VLC 做出贡献,也许有 300 人为 FFmpeg 做出贡献,对吧?

Our goal as a small team is to get all the contribution in.

作为一个小团队,我们的目标是贡献所有的力量。

So if there is more usage, there will be more contributions, and those people will do the right module, the new format, and so on.

因此,如果有更多的使用,就会有更多的贡献,那些人就会做正确的模块、新的格式等等。

We care about the architecture of VLC, the architecture of FFmpeg, right?

我们关心VLC的架构、FFmpeg的架构,对吧?

Now we’re doing things in VLC, which is spatial audio, right?

现在我们在 VLC 中做事,也就是空间音频,对吧?

We did the demo not long ago.

我们不久前做了演示。

There was changes needed on the architecture, and we did the first spatial audio module.

架构需要改变,我们做了第一个空间音频模块。

4:11:07 When it’s going to add the second one, it’s going to be easy, or the third one is going to be easy, right?

当它要添加第二个时,它会很容易,或者第三个也会很容易,对吧?

Our goal, and it’s going to be the same for others or haptic, right?

我们的目标,对于其他人或触觉来说都是一样的,对吗?

We need to work the architecture so that modules can be added to add future capabilities.

我们需要改进架构,以便可以添加模块来添加未来的功能。

So yes, we are going…

所以是的,我们要去……

We are a multimedia framework, so that’s not just audio and video.

我们是一个多媒体框架,所以不仅仅是音频和视频。

It’s everything that is timed and represents something that you can sense.

它是所有有时间的东西,代表着你能感觉到的东西。

And if it’s brainwaves, it’s going to be brainwaves.

如果是脑电波,那就是脑电波。

Kieran Kunhya4:11:36

I think that’s inevitable.

我认为这是不可避免的。

4:11:36 Sorry.

抱歉。

Lex Fridman4:11:38

I love this on so many fronts because, so FFmpeg and VLC are pushing companies and pushing the world to standardize.

我在很多方面都喜欢这一点,因为 FFmpeg 和 VLC 正在推动公司并推动世界标准化。

4:11:38 So for example, to standardize- …

例如,为了标准化——……

brainwaves, right?

脑电波,对吗?

So standardize…

所以标准化...

It would push, like I hope Neuralink comes up with a standard for multimedia via brain-computer interfaces or for robots with haptic.

它会推动,就像我希望 Neuralink 提出一个通过脑机接口进行多媒体或具有触觉的机器人的标准一样。

Jean-Baptiste Kempf4:12:05

By experience, what happens is always the same, right?

根据经验,发生的事情总是一样的,对吧?

4:12:05 You start, it’s a new topic.

你开始了,这是一个新话题。

There is like five different standards because everyone starts to do this.

大约有五个不同的标准,因为每个人都开始这样做。

The hype goes down because every time the hype goes down, then people start to say, “Well, you know what?

炒作会下降,因为每当炒作下降时,人们就会开始说:“好吧,你知道吗?

We need to do a standard.” People, because two or three companies, usually not the leader, but the two or three followers do a standard, and then we implement the standard and then it’s the end of the curve.

我们需要制定一个标准。”人们,因为两三个公司,通常不是领导者,而是两三个追随者制定标准,然后我们实施该标准,然后就是曲线的终点。

It starts to be more pepper.

胡椒开始增多。

Lex Fridman4:12:32

And then the leader’s kind of pressured into it because it is better to do a standard.

然后领导者会施加压力,因为制定标准更好。

4:12:32 Yeah.

是的。

Jean-Baptiste Kempf4:12:36

Example, 3D audio, right?

例如,3D 音频,对吗?

4:12:36 Six or seven years ago, it was everything about 3D.

六七年前,一切都是关于 3D 的。

You go, you had the cardboard on Android.

你去吧,你已经在 Android 上使用了 Cardboard。

You had two audio formats.

您有两种音频格式。

They’re all dead, right?

他们都死了,对吧?

And now it’s coming back with actual use cases, and we learn from the mistakes of the past standard.

现在它带着实际的用例回来了,我们从过去标准的错误中吸取了教训。

So it will be the same everywhere.

所以到处都是一样的。

Lex Fridman4:12:56

And not try to avoid closed?

而不是尽量避免关闭?

4:12:56 I saw somewhere you didn’t have too many nice things to say about Dolby.

我在某个地方看到您对杜比没有太多好话可说。

Jean-Baptiste Kempf4:13:03

No, I don’t.

不,我不。

Lex Fridman4:13:04

What is…

什么是……

4:13:04 can you educate me on why, where they went, what did they do bad that made you mad?

你能告诉我为什么,他们去了哪里,他们做了什么坏事让你生气了?

Jean-Baptiste Kempf4:13:13

It’s used to be an amazing company doing tons of great things with amazing engineers.

它曾经是一家了不起的公司,与出色的工程师一起做了很多伟大的事情。

4:13:13 They defined what sound was.

他们定义了声音是什么。

And now it’s mostly-

现在主要是——

Kieran Kunhya4:13:25

Lawyers.

律师。

Jean-Baptiste Kempf4:13:25

…

……

4:13:25 lawyers and licensing things.

律师和许可事宜。

Lex Fridman4:13:27

Oh, so they’re, yeah, it’s, they’re, they’re closing stuff off.

哦,所以他们,是的,他们正在关闭一些东西。

4:13:27 They’re trying to make money on licensing.

他们试图通过许可来赚钱。

Jean-Baptiste Kempf4:13:30

No, it’s just like they, they don’t innovate as much as they did- …

不,就像他们一样,他们没有像他们那样创新——……

4:13:30 and so on.

等等。

It’s a bit like I’m sorry to say, right, like HP, right?

这有点像我很遗憾地说,对吧,就像惠普一样,对吧?

Kieran Kunhya4:13:38

Very true.

非常真实。

Lex Fridman4:13:40

Oh, since we talked about Twitter a bunch in a bunch of different contexts, do you have a favorite, and least favorite, most embarrassing tweet on either VideoLAN or FFmpeg Twitters?

哦,既然我们在很多不同的环境中谈论了 Twitter,那么您在 VideoLAN 或 FFmpeg Twitter 上有最喜欢的、最不喜欢的、最尴尬的推文吗?

Kieran Kunhya4:13:53

The two, my two favorites are, “Talk is cheap, send patches.” I think that, that- …

这两个,我最喜欢的两个是,“谈话很便宜,发送补丁。”我认为,那个——……

4:13:53 embodies a lot of the stuff doesn’t get, as, as we’ve talked about, stuff doesn’t get built unless someone does it.

体现了很多东西无法得到,正如我们所讨论的,除非有人去做,否则东西不会被建造出来。

It doesn’t just appear from the ether.

它不仅仅出现在以太中。

The other one that I like is “FFmpeg, nothing is beyond our reach.” I think that comes from a US military satellite patch where I think they, they invented some kind of monitoring system that could see the whole world, and this was released.

我喜欢的另一句话是“FFmpeg,没有什么是我们力所不及的”。我认为这来自美国军事卫星补丁,我认为他们发明了某种可以看到整个世界的监控系统,并且该系统已发布。

Jean-Baptiste Kempf4:14:19

Wasn’t there something where FFmpeg was running on a rover on Mars also?

火星上的漫游车上不是也运行着 FFmpeg 吗?

Kieran Kunhya4:14:22

Yeah, so FFmpeg is used by the Mars rover, the Mars 2020 rover to compress pictures.

是的,所以火星探测器,火星2020探测器使用FFmpeg来压缩图片。

4:14:22 They really wanted—they wrote a paper about it, and they really wanted to use as much commercial off-the-shelf technology as possible.

他们真的想要——他们写了一篇关于它的论文,他们真的想要尽可能多地使用商业现成的技术。

Lex Fridman4:14:34

Oh, that’s cool.

哦,那太酷了。

Kieran Kunhya4:14:35

FFmpeg runs on Mars, so we are, we are a multi-planetary open source library.

FFmpeg 在火星上运行,所以我们是,我们是一个多行星开源库。

Jean-Baptiste Kempf4:14:41

Very often we’ve seen- …

我们经常看到——...

4:14:41 Tweets for people using VLC in weird places.

在奇怪的地方使用 VLC 的人的推文。

A lot of the people doing Formula 1 are in all the paddocks, they use VLC to play the live feed.

许多参与一级方程式比赛的人都在所有围场中,他们使用 VLC 播放实时直播。

We’ve seen the European Space Agency.

我们看到了欧洲航天局。

We’ve seen SpaceX, like, monitoring the launches with VLC, and it fills you with joy, right?

我们已经看到 SpaceX 用 VLC 监控发射,这让你充满喜悦,对吧?

So-

所以——

Lex Fridman4:15:05

I’ve seen a particle accelerator.

我见过粒子加速器。

Jean-Baptiste Kempf4:15:07

Oh, yeah, yeah.

哦,是啊,是啊。

4:15:07 We had one of the most amazing thing that I went for was to go to the CERN at the LHC because they were using VLC to monitor all the captors on the ring because the ring is 27 kilometers.

我们经历过的最令人惊奇的事情之一就是去大型强子对撞机的欧洲核子研究中心,因为他们正在使用 VLC 来监视环上的所有捕获者,因为环有 27 公里。

And so they had some analog cameras- …

所以他们有一些模拟摄像机——……

and they were using some of the capture cards to go to analog to VLC, so VLC could stream on their multicast network for the whole CERN to access to that.

他们使用一些采集卡来模拟 VLC,这样 VLC 就可以在他们的多播网络上进行流传输,以便整个 CERN 访问该网络。

And, like, I visited that in 2010 with Laurent and, like, we fixed their issue in an hour or something like that, right?

2010 年我和 Laurent 一起访问过那里,我们在一小时之内就解决了他们的问题,对吧?

Because it was some parameters maybe not well documented at that time.

因为当时有些参数可能没有很好的记录。

And he said, “Okay, for the whole day, what do you want to do?” And we visited everything.

他说:“好吧,一整天,你想做什么?”我们参观了一切。

Like…

就像……

things where with antimatter and colliders and so on.

含有反物质和对撞机等的东西。

Lex Fridman4:16:00

That’s awesome.

太棒了。

Jean-Baptiste Kempf4:16:00

And that was, like, one of the most amazing days of my physics background.

那是我物理背景中最令人惊奇的日子之一。

Lex Fridman4:16:06

Yeah, it’s used, like, everywhere.

是的,它无处不在。

4:16:06 Any tweets, Kieran, you regret?

Kieran,你后悔发过任何推文吗?

Kieran Kunhya4:16:12

Tweets I regret?

我后悔的推文?

Lex Fridman4:16:14

Or is it like that, how does the French song go?

或者是这样,法语歌曲怎么样?

4:16:14 Regret nothing.

没什么遗憾的。

Kieran Kunhya4:16:17

“Je ne regrette rien.” Yeah.

“我不后悔。”是的。

Jean-Baptiste Kempf4:16:19

That’s very important for me, right?

这对我来说非常重要,对吧?

4:16:19 Don’t regret anything.

不要后悔任何事情。

No, it’s because regrets are a tax on your mind, right?

不,这是因为后悔是你心灵的负担,对吗?

So learn from your mistakes, but don’t regret.

因此,要从错误中吸取教训,但不要后悔。

Because you’ve done it, so except if you have a, a time machine to go back in time, don’t regret, right?

因为你已经做到了,所以除非你有一台时光机可以回到过去,否则就不要后悔,对吧?

It’s going to just tax your brain.

它只会消耗你的大脑。

Learn from your mistake, sure.

当然,要从错误中吸取教训。

Don’t regret.

不要后悔。

Lex Fridman4:16:43

It’s like it reminds me, it’s beautiful.

就好像它让我想起了一样,它很美。

4:16:43 It’s a tax on your brain.

这是对你大脑的负担。

It reminds me of the Johnny Depp quote I saw where he was saying, “Hate, you know, I don’t hate.

这让我想起我看到的约翰尼·德普的话,他说:“仇恨,你知道,我不仇恨。

That’s, hate is a very expensive emotion.”

也就是说,仇恨是一种非常昂贵的情感。”

Jean-Baptiste Kempf4:16:56

Are you comparing me to Johnny Depp?

你在拿我和约翰尼·德普比较吗?

Lex Fridman4:16:58

Uh-

呃-

Jean-Baptiste Kempf4:16:58

Because that would be your first one.

因为那将是你的第一个。

Lex Fridman4:17:00

Well, gentlemen, like I said, I’m eternally grateful for the software that, you know, the two of you and the bigger community have been part of building with FFmpeg and VLC and everything else.

好吧,先生们,就像我说的,我永远感谢你们两个以及更大的社区使用 FFmpeg 和 VLC 以及其他一切构建的软件。

4:17:00 I’m eternally grateful for the spicy tweets.

我永远感激那些辛辣的推文。

Never stop.

永远不要停止。

And I’m grateful that you would talk with me today and give me this sexy hat.

我很感激你今天能和我交谈并送给我这顶性感的帽子。

I feel like a wizard.

我感觉自己像个巫师。

I feel special.

我感觉很特别。

And I feel special to get a chance to talk and celebrate the piece of software that brought me so much joy over the years.

能有机会谈论和庆祝这款多年来给我带来如此多快乐的软件,我感到很特别。

So thank you for everything, and thank you for talking today.

谢谢你们所做的一切,也谢谢你们今天的发言。

Kieran Kunhya4:17:36

Thank you for having us.

感谢您拥有我们。

Jean-Baptiste Kempf4:17:37

Thank you so much.

非常感谢。

Lex Fridman4:17:38

Thanks for listening to this conversation with Jean-Baptiste Kempf and Kieran Kunhya.

感谢您收听 Jean-Baptiste Kempf 和 Kieran Kunhya 的对话。

4:17:38 To support this podcast, please check out our sponsors in the description where you can also find links to contact me, ask questions, give feedback, and so on.

为了支持这个播客,请在描述中查看我们的赞助商,您还可以在其中找到联系我、提出问题、提供反馈等的链接。

And now let me leave you with some words from the legendary Linus Torvalds.

现在让我向您介绍传奇人物莱纳斯·托瓦兹 (Linus Torvalds) 的一些话。

“Most good programmers do programming not because they expect to get paid or get adulation by the public, but because it is fun to program.” Thank you for listening, and I hope to see you next time.

“大多数优秀的程序员进行编程并不是因为他们期望获得报酬或受到公众的奉承,而是因为编程很有趣。”感谢您的聆听,希望下次再见。

文稿来自 Lex Fridman 官网公开发布的官方 Transcript(含说话人与时间戳)。
Jean-Baptiste Kempf (00:00:00): The important thing is, is your code good? We care about excellent code. We don’t care who you are. Like maybe you’re a dog. I don’t care, right? I don’t care where you come from. I need to look at your code. Oh, yeah, but I’m an engineer at this very large company in Italy, in Germany, in the US. We don’t care. We care about the quality of your code because this is what defines our community, which means that we have a lot of people who contribute who are from very different backgrounds and very introverts. Sure. But that’s okay, right? Lex Fridman (00:00:31): FFmpeg is probably one of the biggest CPU users in the world. Everything we’ve just said in the past couple of minutes, every sentence is someone’s lifetime’s work. There are books about every sentence. So the level of complexity in many cases is inordinate. Jean-Baptiste Kempf (00:00:45): FFmpeg has one hundred thousand lines of assembly for all the codecs. Lex Fridman (00:00:50): For all codecs. Jean-Baptiste Kempf (00:00:50): And just this one has two hundred and forty thousand. Every cycle matters. We are talking about probably three billion devices which are going to decode video nonstop because, for example, thirty percent of the video from Netflix are now in AV1, fifty percent of YouTube. Lex Fridman (00:01:07): This is what peak video codecs should look like. Seventy-nine point nine percent assembly, nineteen point six percent C, and zero point five percent other. Jean-Baptiste Kempf (00:01:18): And what’s incredible is with those tweets, which are factual, people go crazy. Lex Fridman (00:01:25): For the last two years, they go crazy. No, intrinsics is fine. The compiler- Jean-Baptiste Kempf (00:01:28): You can optimize your compiler. Auto-vectorization, it’s your fault. You don’t understand. And we’ve tried that forever, right? Lex Fridman (00:01:35): For two years, and two years later, showing hundreds of examples of handwritten assembly. No, no, no, you’re doing it wrong. The compiler can do this. The intelligence agencies tried to, like, say, “Can you put a backdoor in VLC?” Jean-Baptiste Kempf (00:01:48): Yes. Two of them. Lex Fridman (00:01:50): Well, what did you say? Jean-Baptiste Kempf (00:01:51): No. Well, I was a lot less polite. Lex Fridman (00:01:54): Basically saying, “Hell no.” Jean-Baptiste Kempf (00:01:56): Like, if we had to compromise our software, we would shut it down. This is clear. Lex Fridman (00:02:00): Any tweets, Kieran, you regret? Kieran Kunhya (00:02:04): Tweets I regret? Lex Fridman (00:02:05): Or is it like that, how does the French song go? Regret nothing. Jean-Baptiste Kempf (00:02:08): Don’t regret anything. No, it’s because regrets are attacks on your mind. Lex Fridman (00:02:17): The following is a conversation all about FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya. FFmpeg is an open source software system that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord, and basically every platform that touches video or audio on the internet. It can decode, encode, transcode, stream, and play almost any video or audio format ever created. To me, it is one of the most incredible software systems ever developed, and it’s all done by volunteers. VLC is also a legendary piece of software. It is an open source media player that plays basically anything you throw at it, any format, any platform, no ads, no tracking. Lex Fridman (00:03:11): It has been downloaded over six billion times, and again, for me, it has been one of my favorite pieces of software ever, with the most legendary logo, which I, of course, had to honor in this conversation by wearing the VLC traffic cone hat the whole time. So again, above all else, thank you to the incredible volunteer engineers who put their heart and soul into this code that has been used and loved by billions of people. Thank you. And about the two great engineers and human beings I’m talking to in this episode, Jean-Baptiste is the president of VideoLAN and is a key figure behind VLC and FFmpeg. Lex Fridman (00:03:58): Kieran is a longtime codec engineer, FFmpeg contributor, and the man behind the now infamous FFmpeg account on Twitter/X that I recommend everybody follow for the memes and for the unapologetic celebration of open source and great low-level software engineering. Let me also say that it’s inspiring and humbling that so much of modern civilization rests on software built by people who are not chasing fame or money, but are obsessed with the craft of engineering. We live in a world where billions of people consume video every day without ever thinking about the invisible machinery underneath it. But that machinery matters. Open source infrastructure matters. Lex Fridman (00:04:46): It is one of the great examples of human beings quietly collaborating across borders to build something useful, durable, and elegant for the rest of us. And so this conversation is not just about codecs and media pipelines. It is also about the deeper spirit of engineering and generosity that makes projects like FFmpeg possible. Again, I can never say it enough. Thank you. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback, and so on. And now, dear friends, here’s Jean-Baptiste Kempf and Kieran Kunhya. Lex Fridman (00:05:35): So the legend goes VLC can open everything. What’s the weirdest thing that you know that it can open? Jean-Baptiste Kempf (00:05:43): You know, there is a ton of people who are using VLC to record VHS videos, right? Like, it’s just like you plug it with a capture card and you can basically record VHS video. Lex Fridman (00:05:52): Well, how does that work? Jean-Baptiste Kempf (00:05:53): Basically, it’s, you know, those type of capture cards where you can put a Peritel in or- … or RCA, and you put that, and actually VLC can play those type of cards, and there is a module which allows to control directly some of those VCR camcorders. We support DVD-Audio lately, right? We spent the summer working on DVD-Audio support, and like there is no, no one’s making any DVD-Audio support. There are custom encryption schemes. Lex Fridman (00:06:19): What about Lucasfilm? Jean-Baptiste Kempf (00:06:20): Oh, yeah, and there is of course all the weird codecs support, game codecs supported by FFmpeg. Lex Fridman (00:06:25): The one Star Wars video game, the first ten-second opening sequence, someone has gone and implemented that and made sure that’s bit exact on one disc that existed at one time of one little sequence in the game. Jean-Baptiste Kempf (00:06:36): And then funnily, there was a… At one VideoLAN conference, we made a competition to make the weirdest and most horrible file ever- … and see if VLC could play it. Lex Fridman (00:06:46): What did it end up being? What’s the file? Jean-Baptiste Kempf (00:06:48): It was an MKV file made by Derek- … which each of the frame was changing resolution, aspect ratio, rotation, and… and it was like- Lex Fridman (00:06:59): Did it work? Jean-Baptiste Kempf (00:07:00): Yes. And there was another one where the whole video was actually animated subtitles, right? SSA, right? So- Lex Fridman (00:07:08): Yeah. I remember that, yeah. Jean-Baptiste Kempf (00:07:08): … each, each, this one was- Lex Fridman (00:07:10): It- Jean-Baptiste Kempf (00:07:10): And so each frame was a black frame, but on top of that there was a subtitle that was animated for each frame. Lex Fridman (00:07:16): There was a file that’s a valid ZIP and a valid MP3 at the same time or something like that, so. Jean-Baptiste Kempf (00:07:20): So yeah, we made a competition of stupid files. Lex Fridman (00:07:23): And it worked. It opened all of the stupid files. Jean-Baptiste Kempf (00:07:27): Yes. Lex Fridman (00:07:27): By the way, for people who are not familiar, I am wearing a hat. Would it be fair to say this is the best worst logo of all time, the cone? Jean-Baptiste Kempf (00:07:36): Yeah, by far, right? The logo of VLC is so iconic, right? Like we are a team with a small number of people and the icon is known everywhere. I go to middle of nowhere in India or in China, people know the cone, right? And 25% of the website traffic that comes to our main website is cone player, right? So many people don’t know VLC, right? They know the cone player. Lex Fridman (00:07:59): That’s the thing they Google for is “cone player.” Jean-Baptiste Kempf (00:08:01): Yeah. They go on Google and they put “cone player” and they download VLC, right? So that’s iconic. And once we tried to change it as a joke, right? We said it was going to be a type of caterpillar construction and we said that during April 1st- … and we had around 10,000 emails saying, “No, don’t change the logo,” and so on, right? So it’s so iconic, right? It’s so distinctive, right? If you want to do a video player, you’re going to put a play button on a TV, right? And that’s a YouTube logo, right? It’s unoriginal. This one is orange, right? It’s very bright and it’s weird. Lex Fridman (00:08:37): And it’s ridiculous and it’s absurd and it’s hilarious. It becomes meme and meme becomes culture. Yeah. Jean-Baptiste Kempf (00:08:41): And you keep it and you know about it and you know that in 20 years, like you still have, going to have the cones and remember, oh yeah, that was a video player. Lex Fridman (00:08:49): Yeah. And we’ll talk about, you know, the mission of FFmpeg being a kinda the archival aspect of it. So you can think about 1,000 years from now we’ll have all these videos that only VLC can open. Humans, human civilization has already destroyed itself multiple times and the only thing that will remain is this like, you know, the cockroaches will be crawling around and it’ll be the VLC logo- … with our, some of the archival footage that VLC can open. And the aliens will show up and they’ll press play and they’ll get to see it all ’cause- Jean-Baptiste Kempf (00:09:18): Well, really, really hope so, right? But there is also so many memes where people say, “Well, I’m sure I can put a pancake inside my DVD drive and VLC will play it.” Like- Lex Fridman (00:09:25): Can they? Jean-Baptiste Kempf (00:09:26): No, we tried. It doesn’t. Lex Fridman (00:09:27): Doesn’t Jean-Baptiste Kempf (00:09:28): … but we actually have a video of us trying that. Didn’t work. Lex Fridman (00:09:31): A codec for physical reality, I don’t know what that would even look like. Jean-Baptiste Kempf (00:09:34): There was a guy who did that, right? He printed a small cone, right? Like the ones we distribute as goodies and inside he put an RFID chip which was his way of playing a movie, right? And so he- … put this on a RFID player and when he put that it was playing like The Last Star Wars and so on. So instead of having like DVD boxes, he had like VLC cones all around and he plugged that and that was like physical objects. Lex Fridman (00:09:59): So the thing that we’re talking about is everything around video codecs, video encoding, video decoding, video streaming, video player client that I’m wearing on my head, the entire ecosystem enabling free media. We’ll talk about FFmpeg, we’ll talk about VideoLAN, VLC and all the other incredible video technology that is used probably by billions of people. So JB, you’re the lead developer behind the legendary VLC player. Kieran, amongst many other things, you’re lead developer behind the legendary FFmpeg handle on Twitter. And both of you have spicy opinions I would say. So today we wanna talk about FFmpeg and VLC. Lex Fridman (00:10:45): For context for people who are not aware—and I’m sure basically everybody listening to this have used these two technologies probably regularly without knowing it. So FFmpeg underlies basically most video on the internet including YouTube, Netflix, Chrome, Firefox, of course VLC and countless other video platforms. It is estimated that over 90% of video processing workflows online and offline involve FFmpeg. VLC has been downloaded at least 6.5 billion times. But likely that number, ’cause it’s impossible to really count the number, is much higher than that. Virtually any operating system supports virtually any media format. The limitation being it can’t open pancakes. So can we just lay out some of the basics to, to help people understand what’s involved in all of this? Lex Fridman (00:11:45): So when we press play on a video player like VLC, what happens? How does it go from the file or the stream to the pixels on the screen and the sound on the speaker? What are the big stages to be aware of? Jean-Baptiste Kempf (00:12:01): So there are several stages, right? The first stage is to get from an address, right, which is the type of URL, to give you a byte of streams, right? So this would be, for example, HTTP, file, DVD, right? You give the path to the media, and it gives you a stream of data. Kieran Kunhya (00:12:19): The stream needs to be cut up by what’s known as the container, the demultiplexer or demux. We’ll try and keep the jargon light throughout this, but it needs to go and start demarcating video and audio frames. So it just gets data from the operating system blocks at a time and needs to start cutting these frames up into compressed data. It then needs to start doing simple parsing of the video frames- Kieran Kunhya (00:12:40): … mainly to figure out whether that codec is GPU decodable or needs to fall back to software. We’re very sort of used to assuming the GPU will play all of these things. There’ll be hardware acceleration. I think it’s up to forty-five percent of files are not GPU decodable. So these need to be probed. They need to be detected. There can be variants of a given codec, some of which are decodable on the GPU. Different vendors of GPU might have different capabilities, so those need to be detected. So if, if it’s GPU capable, you pass it through to the GPU black box. So now if there’s a software fallback, that means in the beginning is, is to first do deentropy coding, so removing the mathematical coding of the bit stream. Kieran Kunhya (00:13:22): So this uses capabilities such as Huffman coding or arithmetic coding to actually decompress the mathematical layer of the bit stream. We then need to start reading the syntax elements for intra prediction. So intra prediction are like still images of the video, so your I-frames. So this works and operates in the spatial domain. So you do your intra prediction in spatial domain. You have a residual because your prediction isn’t quite matching that of reality. So you’ve made a prediction, but then there’s a little bit left, and that’s what’s known as the residual. This is stored in the frequency domain, and these are quantized to decompact their space. We then need to do the inverse transform to bring them back to the spatial domain and apply these residuals. Lex Fridman (00:14:05): So a lot of the process of the decoding is this thing is compressed. And you have to predict the highest quality thing that’s supposed to go there. I-frame- … is the best representation you have spatially. Kieran Kunhya (00:14:18): Yes. Lex Fridman (00:14:19): And then there’s a lot of temporal compression that can happen depending on the codec, and then you’re predicting. You’re predicting what the reality that was captured in this rawest form. Jean-Baptiste Kempf (00:14:29): Yeah, because what people don’t realize is that the compression on video and audio is one hundred times, right? Like, people don’t realize how compressed we do, right? For audio, you move, you compress by, when you go from normal audio to MP3, you compress by ten times, right? When you move to video, you need one hundred times, two hundred times, right? So you need to remove all the details, but that you don’t care about because all the compressions that we do, and that’s very important, people forget about that, is to be viewed by humans, right? Jean-Baptiste Kempf (00:15:02): So all the codecs, either for audio, mimic basically how your ear works, right? And a lot of things about, like, the response on the ear and same for your eyes, right? And so, for example, on video, we don’t work on RGB, right? Everyone expect to work in RGB. We don’t, right? We move to YUV, which is basically one is luminance, brightness, and the other are colors. And this matches your eyes, where inside your eyes you have the cones and the rods, right? With some of them look on brightness and more on the other on colors, right? So we need to compress a lot, and so we need to degrade. But in order to degrade, we need to match the human perception, and this is why it’s so difficult. Jean-Baptiste Kempf (00:15:43): And then we need to use the maximum power, mathematical power, very complex technologies. We move to the frequency domain, as Kieran said. We do a ton of dequantizing in order to get the best compression, but it still looks good. Lex Fridman (00:15:58): You’re trying to compress in order to maximize the highest quality thing for human perception. Jean-Baptiste Kempf (00:16:04): That is correct. And this is very important, right? Compression is not like a ZIP, right? A ZIP, you have data in, you get data out, right? And you try with all the ZIP compression to arrive with the limit. Here we are degrading the signal, right? And so we need to degrade both the audio and the video signal in the best way possible. And we can do that, but it involves, first, a lot of theoretical knowledge about how the eye works, but a lot of mathematical change, a lot of mathematical tricks, right? For example, when you move to RGB and you go to YUV, for example, what we do very often is that we scale down the resolution of the color compared to the brightness. Jean-Baptiste Kempf (00:16:48): And most of the time, and just this without compression, it divides the size by two, but most people don’t see it, right? And so on and so on, right? And then you go to very complex mathematical change. So of course Fourier transforms, which de facto are not Fourier transforms, they are like discrete cosine transforms, but that’s the same idea. So frequency domain we split the video by blocks, right? So that’s why when it’s wrongly decoded, you see those blocks, and badly encoded, you see those blocks, and so on, to arrive to compression states that are insanely high, right? And each generation of the codec is like thirty percent less- … for the same quality, right? And this requires amount of power, of computational power, that are huge. Kieran Kunhya (00:17:35): No, no, but you should, you should elaborate. It’s thirty percent better, but an order of magnitude, perhaps, perhaps even two orders of magnitude more compression power. That’s the big difference. Lex Fridman (00:17:45): What do you mean by compression power? Kieran Kunhya (00:17:47): Sorry, CPU power to achieve that level of compression. Lex Fridman (00:17:49): Oh, yeah. So and you have to be able to leverage the CPU and sometimes GPU, like you mentioned. And then we should mention that a lot of this programming is done at the lowest possible- … stack, whether it’s C and of course, as, as the legendary- … Twitter handle re-emphasizes over and over, a lot of assembly. Jean-Baptiste Kempf (00:18:10): So what happens globally is that you have an address, right? Which gives you with the operating system, a stream of bytes, a stream of data, right? And this is the first step. And the second step arises with demuxing, where you’re going to separate audio, video, subtitle in type of different tracks. And then on each of those tracks, you’re going to decompress them, decode them, either audio with an audio codec, video to video codec, and subtitle to subtitle codec. And once you’ve decompressed those type of things, you have raw images, raw, and then you’re going to talk with your graphic card in your screen and display that. And same for the audio, you’re going to talk to your audio card, which then is going to go in analog to your audio speakers. Kieran Kunhya (00:18:50): And everything we’ve just said in the past couple of minutes, every sentence is someone’s lifetime’s work. There are books about- … every sentence. So the level of complexity in many cases is inordinate. You know, it’s… Every sentence has thousands of people working on this- … in industry as a whole, books written about it. So there’s a lot of detail, there’s a lot of subtleties, there’s a lot of both academic and practical realities, both of which matter. Lex Fridman (00:19:20): We mentioned codecs, but I don’t think you mentioned containers. So what are the actual containers for some of the stuff we’re talking about? So people are familiar with MP4, MOV, MKV. So anyway, what are containers versus the thing that goes inside? Jean-Baptiste Kempf (00:19:41): So the container is what we call also the muxer, right? When I say demuxing, it means decontainerizing, right? So actually, if you look, mux means multiplexer and demultiplexer, right? Mux and demux are those. And same, a codec is actually coder, decoder, right? And so containers are this collection of multiple tracks, right? So it’s a, what normal people call the file format, but it’s a bit more subtle than that. But the most known one, of course, is MP4, but when I started, it was AVI, right? AVI was the video format from- Jean-Baptiste Kempf (00:20:17): … from Microsoft, and MOV, M-O-V, which became MP4, was a format from Apple. In the open source community one of the person that is still active on VideoLAN is called Steve Lhomme and sta- started this Matroska format, which is, like, a bit more complex and, and, and more future-proof. And there are so many others. Lex Fridman (00:20:38): So, I mean, there’s a, it’s a pretty common thing, and maybe it’ll even happen in this conversation, that people confuse container and the codec, right? So confuse MP4 and H.264, for example. Is that a horrible violation? Jean-Baptiste Kempf (00:20:51): No, it’s not, because technically the name of H.264 is MPEG-4 Part 10. Because MPEG-4 is actually a meta specification which has several things in it, right? There is the Part 2. So there is, like, audio codecs, right? AAC de facto is MP4 audio- … something. There is actually several video codecs, right, inside the MPEG-4 specification. One of them is MPEG-4 Part 10, called also AVC, called also H.264. Right? So it’s completely the fault of the industry to make things difficult to understand. So that’s very difficult so that people then don’t understand why sometimes you talk about MPEG-4 Part 10, where you mean H.264, and why it’s not MP4. Lex Fridman (00:21:38): So you can technically shove in all kinds of different codecs inside containers and horribly so. Kieran Kunhya (00:21:44): But broadly speaking, though, MP4 is understood to generally be H.264 plus AAC audio. 99% of the time that’s that, and that, the, the rest are de minimis, the small effects, you know, edge effects really compared to that. So it’s not the end of the world. There are people who do get annoyed by that. But also in reality, something like VLC, just to point out, the file may say .MP4, but it may be something completely different, and that’s one of the challenges both FFmpeg and VLC have is the real world is a completely different place to a three-letter file format. Jean-Baptiste Kempf (00:22:17): And this is very important to say, right? Like, for example, in VLC and in FFmpeg, we discard the file format, right? We look into the file to understand what’s in it because so many people, like, they say, “Oh, it’s a video, it must be MP4,” but technically it’s an MOV or maybe it’s a MKV, right? So we analyze in real time everything that we have, and we don’t trust- … the format. Lex Fridman (00:22:42): So what information does the fact that it’s .MP4 give you? Jean-Baptiste Kempf (00:22:45): It helps, right? It gives you a hint, right? Just like, oh, it’s finished by .MP4. I will start first by opening, probing it with the MP4 container demuxer to see, well, it should be that. But I don’t trust it, and if I’m lost, I say, “Okay, maybe I’m going to try it.” So it bumps the priority of the module. Lex Fridman (00:23:06): So how do you get to—just to take a bit of a tangent there. You know, the dumb thing is if you try the MP4, but it turns out it’s a different codec than you would have expected, most players just break there. And so how do you not break? There’s just philosophically, I’m sure there’s a bunch of stumbling blocks along the way where it’s easy to just break and stop, freak out. That’s it. How does VLC not? Jean-Baptiste Kempf (00:23:34): This is why VLC is popular. But the reason is because actually VLC was, is just a client of a streaming solution called VideoLAN from a very long time ago, from the late ’90s. And when you’re playing video which are on UDP, right, in network, they might be damaged, right? So you don’t trust your inputs, and this is very important into the security is that you don’t trust your inputs. So everything in VLC is prepared to work with broken files. And it’s a philosophical idea from the beginning, and everything is engineered into that. And it’s a culture, right? And so, for example… And VLC became very popular on that because a long time ago when people were pirating content which they do a lot less today, um- Lex Fridman (00:24:23): And none of us ever have- Jean-Baptiste Kempf (00:24:24): No, of course not. Um- … the metadata to play some files like AVI is at, at the end of the file, right? And when you’re downloading, you don’t have that, right? So VLC was just like, “Hey, this file is broken, but I’m still going to try to interpret it,” and this was very useful. Lex Fridman (00:24:41): We hinted at the awesomeness of the various different stages. We hinted at the awesomeness of codecs, the depth and the richness and the complexity of everything involved there. What— Let’s try to define what is a video codec? What’s involved there? What does it mean to compress something? You already started to hint at it— … but can, can we elaborate a little bit more? Kieran Kunhya (00:25:01): So there’s a huge amount of redundancy in any video, both spatial and temporal, and the point of any video codec is to remove this redundant data, use mathematical properties as part of this reduction process. So more often than not, using several orders of magnitude more compute to compress because that’s more costly, both financially and in CPU resources— … versus the decompression. So it’s asymmetric in that respect. Often the case because compression is done once, but there could be lots of viewers of another file. Kieran Kunhya (00:25:33): So to take that information and compress it by 100x, 200x, removing redundant information and using mathematical properties to make that small, but also have properties such as error resilience. So as JB suggested, VLC in the beginning was used to play UDP network feeds, and UDP network feeds lose packets. And so some of the design goals of a codec is also to be recoverable. You need to actually be able to join a stream. It’s not necessarily a file. You need to join, get on the decoding process, and start decoding. Jean-Baptiste Kempf (00:26:06): And to give a more image to people who are not familiar, right? Like, when you’re going to see any type of movie, right? You’re going to see the camera is going to pan, right, and travel. And you realize that, for example, all the background is the same for like a minute, right? Or— … thirty seconds, right? So you can reuse the cloud that you see on the background, you can reuse that from a frame to another, right? And so it gets the more, the more memory you have, the more power, the more comparisons you can make, right? And so the more compressed you can be. And most of the modern codecs are basically doing that. Lex Fridman (00:26:44): So just to make it even more explicit. So what is video? Video is a bunch of pixels off an RGB. You have three values, and you have a grid of pixels, and you have, let’s say, twenty-four or thirty or sixty frames a second, and you just have all these pixels repeating and showing different stuff- … thirty times a second. And so the question, the philosophical, the technical question is, how can I compress all of that, store all of that at 100x? Jean-Baptiste Kempf (00:27:20): Yep. Or 1,000x, right? Lex Fridman (00:27:22): 1,000x. Jean-Baptiste Kempf (00:27:22): The, the target is 1,000x, right? Lex Fridman (00:27:24): And the goal is when you say redundancy, what is redundant? Meaning stuff at best that humans wouldn’t notice if it was missing. Jean-Baptiste Kempf (00:27:35): So for example, you have a picture of a cloud, right? And from the next frame, it’s still going to be the same cloud, so it’s redundant. You could just put it once and not do it, right? Or you have a black background behind me, for example. The black is the same on the whole picture, right? So you can say, “Well, you know, in this picture, take the pixels that you have on the top left and the one on the top right. I’m not going to give the value. I’m just going to tell you it’s the same at the top left.” And then you can say for frame one, reuse something from the previous frame or the previous, previous frame, and so on and so on, right? So you could… Basically, it’s unlimited, but then it’s limited in terms of memory or in terms of compute power. Jean-Baptiste Kempf (00:28:15): Because, for example, if you need to compare pixels on two hundred frames in the past on 4K resolutions, it’s a huge amount of compute. Lex Fridman (00:28:26): And then when you’re showing it, you have to do the decompress of all of that. So is it the codec, has the encoding and the decoding is a coupled process that you’re developing? Jean-Baptiste Kempf (00:28:37): Yes, exactly, right. And those are two different trade-offs, right? Are you going to compress more? But then it might be more difficult to decode. Are you going to make it a codec that is more complex to encode and easier to decode? Are you going to make a codec that is easier to encode because you need to be fast, but then the client side, the player is going to spend more time? That’s why you have so many different types of codecs, is that it’s not always easy. And to make it even more complex, modern codecs like AV1, AV2, or VVC are actually not codecs. They are a collection of tools, right? There are multiple tools, multiple codecs in the same codec to, depending on the image, get the more compression. Kieran Kunhya (00:29:21): So just to elaborate, codecs like AV1, VVC have a much wide, have a wide audience. It could be a screen share content, it could be video, it could be animation. All of these require different coding tools. So what happens these days is a collection of tools are put in and called AV1 and called AV2, called VVC to allow for different use cases. So you may be on Zoom and sharing your PowerPoint, and then you need to show the audience a video. That codec needs to start changing its tool set depending on the content to compress in a different way. Lex Fridman (00:29:59): And like you said, there’s a, a bunch of incredible engineers behind each part of that, each part of the tools that make up AV1, for example. Kieran Kunhya (00:30:05): Sure. Lex Fridman (00:30:06): So we’ve kind of danced around it. We talked about VLC, the logo, the hat. Let’s talk about FFmpeg. What, what is FFmpeg exactly? Jean-Baptiste Kempf (00:30:17): FFmpeg is basically the low-level libraries for codec, so compressions and decompression, muxers and demuxers, and filters. It’s– The core is this, and then you have a several tool which allow you to create a type of pipeline to process any type of video files. And it’s used as a library absolutely inside everything from VLC to Chrome to your smart TVs, to basically any video that you see online you usually use FFmpeg. And FFmpeg in it has all those type of tools, and sometimes depend on other libraries like x264, libvpx, and others, right? So, so it’s really now the de facto tool to process images. Kieran Kunhya (00:31:06): From a philosophical level, I think it’s incredible that your home videos, your, your grandmother’s home videos and trillion-dollar corporations effectively are on a level playing field using the same technology stack. It’s– it wouldn’t be a surprise, you know, these big companies just have three thousand-line FFmpeg commands. There are some that use the API, but there are some that just have long command lines. Lex Fridman (00:31:31): So yeah, there’s a bunch of tools, like literally command line tool, FFmpeg, of course, FFprobe. There’s libraries, libavcodec, libavformat, libavfilter. But the FFmpeg on the command line- … is, like, legendary because you can cut– Like, there’s so many parameters. You can customize everything to hell. Jean-Baptiste Kempf (00:31:53): It’s a language. It’s an actual language. Lex Fridman (00:31:55): It’s an actual– yeah, you could think of it as a programming language. Jean-Baptiste Kempf (00:31:57): Yeah, of course, I’m sure. Because– So most of the people, they’re going to take FFmpeg, file in, file out, and specify the format, right? But you can– We’ve seen thousands of characters, and we’ve seen also, like, people doing programming generation of command lines to make FFmpeg. There is a ton of people who are using AI to generate command lines for FFmpeg because you have no idea what it is. But you can specify so many filters, right on command line, right? So FFmpeg is this collection of toolbox for multimedia processing that everyone, everyone uses. And everyone that is watching your videos are also using, right? You’re on YouTube. Well, it’s FFmpeg on the client side. Well, the server side, on the server side. The client side is probably Chrome. Well, you’re using FFmpeg also. Jean-Baptiste Kempf (00:32:46): And you’re using OBS to record. Well, it’s FFmpeg, right? You’re using a ton of important, like, big box, professional boxes. Well, it’s very possible that inside some part of FFmpeg is running. Lex Fridman (00:32:57): I mean, there’s like so many, just to give people an idea, like I use FFmpeg a lot on everything. Just trivial stuff like take a video, add an intro video and an outro video, and fade one into the other like what is it called? Dip to black, like where it dips and then shows the next video and does the same thing with audio. There’s like a cross dissolve of the audio. It quiets the audio and makes it loud again. And then there’s a bunch of stuff like showing the captions on screen card, like baking the captions in. You can customize the font. You can do all kinds of layering of audio and video. There’s a million things and of course, all of that works like magically with basically any codec. Like anything you can shove in on the audio and the video side, it works. Jean-Baptiste Kempf (00:33:53): But it’s like if you look at, for example, you can do things that you would do with Adobe After Effects- … in command line on FFmpeg, right? It’s, and it’s very interesting because, for example, for images, there is not such tool. There is a few tools, but not with the breadth of FFmpeg. Lex Fridman (00:34:10): So ImageMagick has a similar kind of- Jean-Baptiste Kempf (00:34:13): Yes, but you will not- Lex Fridman (00:34:14): … spirit, but it- Jean-Baptiste Kempf (00:34:14): … do some filters, complex filters. You don’t have the equivalent of Photoshop- … in command line, right? But for video, you have FFmpeg in command line. Lex Fridman (00:34:22): Yeah. It’s incredible. I mean, it’s like an, it’s an example of a thing when a bunch of great people get together and they get a vision, and they stick by that vision for many years, which is incredible. Jean-Baptiste Kempf (00:34:33): And the vision behind, and the same for VLC and FFmpeg, is that we make everything that is very complex easy to use for the normal people, for everyone. Right? Our goal is to make something that is insanely complex technically and make it easy to use, right? And people, they use VLC, they drop a file. They don’t realize how complex the file is, but they play it. Or people put any type of thing inside FFmpeg with complex filters, and it just works like magically, right? And people- And this is our mission, right? Make very complex things. Kieran Kunhya (00:35:08): We wouldn’t be here and you wouldn’t be here if this required, you know, a traditional television studio setup. It’s tools like FFmpeg that democratize this. The podcast and streaming revolution, the YouTube revolution- … was caused. You know, FFmpeg was a big player in that because it democratized this technology that was once in the nineties, for example, you needed equipment that cost hundreds of thousands of dollars to do compression. It was the size of a car, and now everybody has that at almost an exact level playing field, and that’s something that’s so remarkable. Lex Fridman (00:35:42): It gave voice to a lot of people. And just to clarify, we say you wouldn’t be here—not the human, but the podcast. Jean-Baptiste Kempf (00:35:49): The podcast. Oh, sorry. You as a… Sorry. Lex Fridman (00:35:50): I would still… VLC did not have anything to do on a biological level- … at creating me as a human. Jean-Baptiste Kempf (00:35:56): But it’s like you realize also everything moved from text to images and images to video, right? Look at social networks. Video is everywhere. It’s the most powerful medium there is, right? And when you see shorts and Reels and TikTok, right? It’s amazingly powerful to give… Video is amazing for that, right? But the complexity is important. Lex Fridman (00:36:20): This is what people don’t realize. I mean, this is really—it gave power to the individual all across the world. That’s real freedom. And I think, I can’t believe it, but we still haven’t mentioned the actual obvious thing for people who are not familiar, which it’s open source, and there’s an open source community of users and developers behind it. So it’s really a movement. So, like, we’ll talk in a bunch of different ways about the community behind it. But can you speak to the open source element? So when we say what is FFmpeg, it’s an open source project. Jean-Baptiste Kempf (00:36:57): Yeah. So FFmpeg, VLC, x264, VideoLAN, everything we do is fully open source. And for the people who don’t understand how open source is, my usual analogy is about a chocolate cheesecake, right? Usually for you, when you want to buy your cheesecake, you go to a bakery, they give you the cheesecake. The other one way of having a cheesecake is have your grandma give you a recipe of how to make that. When we do open source, we give you the chocolate cake, and we give you the recipe to actually remake the same cake, but at the same time tell you how to build the oven and also how you’re allowed to modify the recipe and resell it to someone else. Jean-Baptiste Kempf (00:37:35): And this is because software is just a very long recipe of small instructions. Computers are not very clever. They go very, very fast. So a normal program has tens of billions of instructions instead of the tens when you have your chocolate recipe. So a lot of the software industry was about selling software where you just have the final cheesecake. In open source, we give you everything, and that managed to get a lot of people to work together, right? Because then you decide that you’re going to make the best program, the best recipe for video, and you create communities. In FFmpeg, since the beginning of FFmpeg, probably two thousand to three thousand- Lex Fridman (00:38:17): In the thousands, yeah Jean-Baptiste Kempf (00:38:17): … people have contributed from the beginning, right? And then it’s exactly like the Linux kernel, right? The Linux kernel has probably ten thousand people contributing everywhere, and they get together, well, mostly online, right? So they virtually get together to create the best tool for something. And on FFmpeg and VLC, it’s just like, well, this codec doesn’t work, so I’m going to work on the codec, and I’m going to add the support for this file inside FFmpeg, so it will be beneficial to everyone. Because again, we work for the greater good. We work for everyone, and that is what open source is. Lex Fridman (00:38:54): And we should mention, depending on the licensing you could probably build a billion-dollar, maybe even a trillion-dollar company as a wrapper to— Jean-Baptiste Kempf (00:39:07): Well, yes— … people do. People do, right? There were a lot of problems with mostly cloud providers who are basically running some open source tools in the cloud and just give you the API to access to that. And there was a lot of databases like Mongo or Elastic who changed their license in order to avoid those types of scenarios. Kieran Kunhya (00:39:31): This is a question we get a lot in FFmpeg is, “Why don’t you do that?” And you can’t. We have, we have thousands of contributors, some of whom aren’t even alive anymore. It would need all of their agreement to do that, and JB will go maybe a bit later and talk about how challenging that process was in VLC to do the re-licensing. Jean-Baptiste Kempf (00:39:49): The license is a social contract in terms of Rousseau de facto of the community. The community does not agree on much besides the license. People go around, discuss around because of the license, and that also allow those license forks, right? Sometimes the community splits, but it’s possible because of the license and to merge back. Jean-Baptiste Kempf (00:40:12): And we’ve seen that so many times, right? GCC and EGCS in the past. We have seen, for example, all the web browsers, right? They started as web, like KHTML, which becomes WebKit and then which becomes Blink, right? So open source license is like the core of the community and people are coming from all around the world, very different types of religion, political borders. They work in the same way on a project to solve a specific problem, and the specific problem we’re working on is to make multimedia easy for everyone. Lex Fridman (00:40:50): Looking it up on Perplexity here, looking at the different open source licenses. Most major open source licenses fall into two buckets: permissive, very few conditions, and copyleft, share-alike requirements for derivatives. Below is a brief practical summary of the main ones you’ll see in the wild. MIT license, BSD, ISC Apache GNU GPL, GNU AGPL. Where’s LGPL? Yeah, LGPL. Let’s see. There’s the Mo- the Mozilla Public License. There’s Eclipse Public License. It goes on. There’s a lot of variety. I mean, I think the really popular ones are MIT, GPL, LGPL- Jean-Baptiste Kempf (00:41:34): Yeah. And BSD. Lex Fridman (00:41:36): … and BSD, Apache. Sometimes you’ll see- Jean-Baptiste Kempf (00:41:38): Apache as well Lex Fridman (00:41:38): … Apache. Unlicense, that’s an option. Attempts to dedicate code to the public domain with a fallback permissive license. Jean-Baptiste Kempf (00:41:43): There are many licenses for many different things. What people don’t understand is that public domain is something that doesn’t exist worldwide, right? So all the open source licensing use the copyright law, right, the international copyright law, in order to give rights on how you use the software or how you modify. It’s de facto a copyright license contract that you give to the end user or to the developer. And so you have like the first ones, which are basically very permissive, MIT, BSD. You give the code and basically you do whatever you want, right? You take it, you want, you modify, you do what you want. And this is popular for JavaScript and the type of BSD operating system. Lex Fridman (00:42:29): So some of them, one of the parameters is whether they require attribution, meaning if you use the code, you have to say- Jean-Baptiste Kempf (00:42:35): Yes. So in those types of permissive licenses, some you need to say if you use it, which is called attribution, and some you don’t. And then there is the other part of license which are copyleft, where you need to give back to the community your modifications and with different strings attached. Some weak copyleft licenses, like the Mozilla Public License, to some which are a bit stronger like a GPL, or even very strong like AGPL. So all of those are different types of licensing that depends on what your goals are and how you want to structure your community, which is why I spoke about social contract, because this is very important to understand. FFmpeg and VLC are mostly GPL or LGPL. The Linux kernel is GPL but Android is Apache. Jean-Baptiste Kempf (00:43:27): A ton of JavaScript frameworks that are used are mostly MIT. All the BSD kernels, OpenBSD, NetBSD are of course BSD. And so the… it’s a philosophical change on how you want people to contribute back- … basically. Lex Fridman (00:43:44): So there’s I think you talked about that you’ve moved at one point from GPL to LGPL on certain parts of the project. What… Can you describe the difference between the two, and what does it take to move to, I guess, a more permissive… So that direction is more permissive. LGPL is more permissive than GPL. Jean-Baptiste Kempf (00:44:04): Yeah. So you have to realize that you can always go from more permissive to less permissive, right? Because of course, those licenses are basically statements, and so if you restrict, you can always restrict more, right? So in a GPL project, you can take MIT code, but you cannot do the opposite, right? Because they are more constrained to match. Indeed, in fact, I changed the core of LibVLC, which is the engine of VLC- … from GPL to LGPL. And there were two reasons to do that. The first one is that so people can use the VLC engine, LibVLC, into third-party applications. So a lot of applications which are playing video on your phone or on your tablet are actually VLC engine in it- … which is calling FFmpeg in it. Jean-Baptiste Kempf (00:44:55): So that was one of the ways to create one of the companies I created, which is doing consulting and integration of those types of applications where you integrate VLC into third-party solutions like inside game engines or stuff like that. With GPL, you couldn’t do that because that means you needed to open source everything, and those are for a lot of, like, commercial companies who don’t want that. Lex Fridman (00:45:18): So you can create a company with LGPL, you can create a company around it. You can do a commercial thing. You don’t have to open source it. So that’s a big, big leap. Kieran Kunhya (00:45:25): So you could play video in your game. The problem is I’m a game developer, and I want to play some videos- … and I don’t want to be forced to open source the entire game just to play those videos. So that’s where the, the consulting business, the libVLC LGPL- … allows you to do that. The LGPL, the library GPL as it used to be known, allows you to do that. Jean-Baptiste Kempf (00:45:44): And FFmpeg is exactly the same. It force… LGPL forces you to give back what you change on this component, this- … library, which is why it’s library GPL. And so you can use FFmpeg as LGPL into, like, any type of application, even non-open source, but you need to give back the modification you did on FFmpeg. Same on libVLC. Lex Fridman (00:46:06): Is it limiting from an open source perspective to go GPL? Because if your library, if your code is GPL, it means you’re basically discouraging companies from building a business- … around it, right? Is that, is that fair to say? Jean-Baptiste Kempf (00:46:23): It depends on the company, but for the company whose business model requires the application to be closed source, yes, it’s limited. So that’s why, for example, I moved to LGPL. The second reason is a bit more obscure: it’s that the terms of condition of the App Store, the Apple App Store for iOS, makes it very complex to have GPL applications on it, while it’s easier to have LGPL applications on it. So VLC on Windows and on Mac and on Linux is GPL. The core is LGPL. But on iOS, the iPhone version and the Apple TV version is a type of different license called the MPL. And yes, I went and changed the license and it was a long story. Lex Fridman (00:47:08): Yeah. So I think basically to change the license you have to contact all the contributors. Jean-Baptiste Kempf (00:47:12): Yes. It’s very important to understand that open source projects are what we call in the US copyright law joint work, or in civil law collective works or collaborative works. It’s that you work all together in terms of the same goal, and then you create one software, which is one release. But the copyright is kept by all the individuals. Some open source projects don’t do that. They force copyright assignment, but this is not what we do. We’re communities. So everyone has basically copyright on what they changed. And this copyright stays even if at the end your contribution was deleted because the new contribution was based on your previous one, right? So if you want to properly re-license, you need to find all the contributors. Jean-Baptiste Kempf (00:47:58): And at that time, I had to contact more than three hundred and fifty people. And sometimes, well, they’re just an email, right? So it’s… you need to actually track down. I actually, like, travel to some place to go someone that I was like, sorry, that I’d found online to see a– to go to their job and say, “Well, you licensed that. Can you– do you want to change from GPL to LGPL?” Most of the times they don’t even care. They wanted to help VLC. But also it brought me to very complex situation. I arrived to the work of a person who was a factory worker. And I said, “Well, I need you to sign that,” because it was his son who died who actually wrote the code, right? Jean-Baptiste Kempf (00:48:40): So I had to explain all those types of open source meanings, and no, I was not a company trying to rip out the two lines or five lines that that guy did- … but was useful, and the whole community agreed on that, and he had no idea I was a factory worker. And I was a lot younger, right? Like it was fourteen years ago, and like I was almost in tears, right? It’s very difficult, right? We are talking about lives of people and he explaining, and we talked about the photo of this guy, right? So it’s important to do it right and to do it correctly. But yes, that means tracking down everything because every contribution works. There are some projects who don’t respect that, and we do re-licensing a bit, like, aggressively. Jean-Baptiste Kempf (00:49:23): But as I said, it destroyed the whole heart of the community because it’s– we only agree on the license, so that’s important. Kieran Kunhya (00:49:31): I would emphasize the community is such a wide-ranging group of people. There’s people in the Syrian war zone with electricity part-time. There’s… there’s people from all walks of life- … rich, poor, young, old. So it’s quite remarkable to get, you know, a group of people aligned on something. And that’s an achievement in itself. Lex Fridman (00:49:55): Yeah. It’s incredible. And a lot of them are introverts, so you coming to find them and getting them and getting them to answer an email might be quite, quite difficult. Jean-Baptiste Kempf (00:50:05): Most of us are introverts, right? You need to be more precise. You have extremely introverts, extremely, extremely introverts and introverts, right? Lex Fridman (00:50:12): Yeah. Jean-Baptiste Kempf (00:50:12): It’s just like a whole spectrum of different people. It doesn’t matter. The important thing is, is your code good? Is your code great? Is your technology great? We care about excellent code. We don’t care who you are. Sorry, it’s just like we have no idea to check. We cannot check, right? Like, maybe you’re a dog. I don’t care, right? I don’t care where you come from. I need to look at your code. And this is important because people don’t understand that, and they come to the community and send them some patches, and they get rejected, and they don’t like that because, I mean, you’re just like, “Sorry, it’s not up to our standards.” “Oh, yeah, but I’m an engineer at this very large company in Italy, in Germany, in the US.” We don’t care. Jean-Baptiste Kempf (00:50:54): We care about the quality of your code because this is what defines our community, which means that we have a lot of people who contribute who are from some very different backgrounds and, and, and very introverts, sure. But that’s okay, right? Lex Fridman (00:51:07): So one of the legends of the community is of course, Linus Torvalds, who created Linux and is a longtime maintainer of the Linux kernel. As the legend goes, he can be pretty harsh on this meritocratic process of reviewing the code and saying it’s not good enough. Can you just speak to the legend of Linus Torvalds? Jean-Baptiste Kempf (00:51:29): Linus is one of a kind, right? And, and I would even go and say that what he did on Git is more interesting than what he did on the Linux kernel. Jean-Baptiste Kempf (00:51:39): He’s very harsh, but what people don’t see is usually when he’s harsh, it’s to people who are maintainers of part of the kernel, right? So they know him, right? So he’s not very harsh like that to everyone. The thing is, what he created in his room is basically powering every server online, right? Even at Microsoft’s cloud called Azure, I’m quite sure seventy, eighty percent of the servers are running Linux. All your Android phones are running Linux. What he did with the power of open source, sure, is amazing. And yes, the quality of the Linux kernel is very high, and yes, it’s difficult, but we cannot compromise on that. We cannot compromise on quality because in the end—and you have to understand that—the core community of VLC is five people. Jean-Baptiste Kempf (00:52:30): The core community of FFmpeg is ten to fifteen, and we are the ones who are going to maintain your code, right? Because one thousand contributors in the timeline and just ten staying, it’s a one percent chance that someone comes and stays. One percent. So you will have change of job, change of wives, you have children, you have accidents in life. You’re going to change jobs, whatever. You’re not going to come back. It’s most likely. So we are the ones going to maintain your code. It needs to be maintainable. It needs to be excellent. And yes, sometimes that means that you need to rework your work because it was good, but it’s not excellent, and we need excellence because we are very few to maintain something that is critical for the whole. Lex Fridman (00:53:15): But we should also mention that there is some spiciness, some harshness to the language that’s sometimes used when you’re keeping this high bar of excellence. Is there something to say to that? Jean-Baptiste Kempf (00:53:27): It’s true, right? It’s also the fact that, for example, what we’re doing is low level. It’s extremely technical. You get into this community. The tone gets very like a type of— It’s a subculture, right? So people who arrive from the external are basically not known to the subculture. Most of those people around FFmpeg and VLC, we do VideoLAN DevDays, VDD every year. They are so fun in real life, and they love it. But it’s true that you’re online and sometimes, like, the tone, you don’t realize how it is. But that’s okay. Lex Fridman (00:54:01): It’s a culture. I mean, you get this in the gaming culture. There’s pretty harsh, intense, the way people communicate, and it’s— everyone understands that the way you show love and respect just looks different in different communities. Sometimes people… It depends. If it’s a book club, usually people are going to be much sweeter. If it’s an open source project that’s very high stakes and used by millions of people- Jean-Baptiste Kempf (00:54:24): But it’s very not often insults that you see, for example, in the gaming, right? And so Linus’ tone is a bit unusual even for the open source community. It’s more like it’s more harsh on the results, saying, “No, this is not good. This is crap.” Those type of things that you will see. Lex Fridman (00:54:40): Try not to make it about the person, make it about the code. Jean-Baptiste Kempf (00:54:42): Yes. Kieran Kunhya (00:54:43): It’s very, very matter of fact, and I think you’ve got to look at it in terms of, you know, the famous FFmpeg is developed almost entirely by volunteers, and that’s true, and you’ve got to imagine someone’s done a hard day’s work at their day job. They come home. You know, terseness might be a thing, you know, it… And that’s not something to take personally. Lex Fridman (00:55:01): You’re tired, you’re busy, but you still care about this open source stuff. But you may not be able to explain and handhold someone on every subtle detail. Jean-Baptiste Kempf (00:55:10): And also you have to realize that most people don’t speak English as native language. And this is especially for open source projects like FFmpeg and VLC, which are mostly centered out of Europe. Sometimes people who are from the US or just are very not happy about the tone, but most of the time it’s also like they don’t know better, right? It’s difficult. The language is—English is a difficult language. There is so many subtleties and tone and so on that you don’t have, right? So often it’s also difficult in those type of communities about different cultures and languages. Lex Fridman (00:55:46): So as the legend goes, JB, you repeatedly turned down millions of dollars to keep VLC open source free for everyone without ads. So take me through the reasoning behind that decision of leaving millions of dollars on the table. Jean-Baptiste Kempf (00:56:06): Yeah, that’s like almost a meme, right, on Reddit or- Lex Fridman (00:56:09): There literally is a meme on Reddit. Jean-Baptiste Kempf (00:56:11): 9GAG and yeah, yeah. See, there’s- Lex Fridman (00:56:14): You looking like a wizard in the, in the VLC hat on Reddit. This is JB, the creator of VLC media player. He refused tens of millions of dollars in order to keep VLC ads free. Thanks, Jean-Baptiste Kempf. You can even summon him on Reddit. Jean-Baptiste Kempf (00:56:33): Yeah. And usually if you see, right, it’s usually like people tag me, right? And then there is me, and then like I say, “Good morning.” Lex Fridman (00:56:40): Good morning. Jean-Baptiste Kempf (00:56:40): I got twenty-four K upvotes, which is great, right? My karma on Reddit is amazing, at least on that account. So the question is, needs to be answered first, what is the story about VLC, right? Because yes, this is true, I refuse dozens of millions of dollars, yes, several times. Yes, I could be a multimillionaire and be somewhere on the beach. But I did not do it because I thought it was not moral and it was not the right thing to do. And this is very important for myself, is to be like, I work for the greater good, I work for people, and I don’t want—it’s not just by myself. But the reason is also because I did not feel that I’m completely legitimate to do that, and let me explain you why. VLC’s story is a very weird story. Jean-Baptiste Kempf (00:57:29): In France, we have university and we have a type of top colleges, and those top of excellency schools are engineering schools, business schools, and basically lawyers and medical, right? But they’re outside of university, and in order to enter those, you spend two years working like crazy math, physics to enter those best engineering schools. One of the school is called the École Centrale Paris. It has changed name since, but it was called the École Centrale Paris. And because it was Centrale, they had to move it because it was too small after the World War II and they moved it, they wanted to move it to the central of France in a place called Clermont-Ferrand. And the alumni decided that this was not okay, right? Jean-Baptiste Kempf (00:58:12): It is the school that Eiffel, right, the one who did the Eiffel Tower, attended to, right? So they said, “No, no, we are amazing, great school. We cannot do that.” And so they bought a piece of land south of Paris very near Paris. And it was a campus managed by a nonprofit of the alumnis, okay? Because of that, everything on the campus was managed by students. The university did nothing, right? So radio, TV, supermarket, library, defining who was going into which rooms. Everything was managed by the students. Lex Fridman (00:58:48): That’s amazing. That’s an amazing experiment, that it all didn’t go to hell quickly. It somehow flourished. Jean-Baptiste Kempf (00:58:55): It worked great, and I learned so much in my life doing those side activities, right? Because you’re twenty-two and you need to run your campus, else you don’t have electricity, right? So you care about that, right? But anyway, in the ’80s they did a full experiment of deploying a network mostly sponsored by IBM and 3Com, which was a token ring network. So token ring is something that probably almost no one knows about anymore. It’s a networking technology where you don’t have routers, right? Jean-Baptiste Kempf (00:59:26): Everyone is linked. It’s like really a ring, and when you want to send a message, you talk to your neighbor who’s going to put the message to the next one, who’s going to put the things to the next one, in terms of ring. The issue with token ring is, of course, is that it’s very slow because every computer on the network needs to open the message, see if it’s okay. Is it for me? No, it’s not, and then send it back, like a token which is traveling around the ring. In the ’80s, you’re doing some Telnet and sending mails as university. That’s okay, right? But starts the ’90s, and the ’90s and start video games, and when you have high latency in video games, basically you die, right? Jean-Baptiste Kempf (01:00:08): So in nineteen ninety-four, nineteen ninety-five, around Doom and Duke Nukem coming around, they want a faster network. So the students go and see the university and say, “You know what? We want a faster network. We need to work,” and also play video games. And the university tells them that basically, “Oh, I’m sorry, we cannot help you because you understand the campus is not ours. You manage it, so do something. And you should see some basically partners of this university and basically go away.” And they go, and they actually go and see the CIO of Bouygues, which is a large French company and who’s doing some TVs in France. And he says, “Well, you know what? Jean-Baptiste Kempf (01:00:50): The future of video is satellite.” Well, today we know it’s not, but at least it was a good idea. In nineteen ninety-five, the first satellite dish, and he says that instead of having like one satellite dish and a big decoder for each of the students, which are one thousand and five hundred, what about you build, like you put an enormous dish and only one decoder, and you send the video directly on the network. And that required a very fast network. Today, it’s obvious, but at the time was, like, the first to do video streaming. So they built this project, which was called Network 2000. Of course, right, we are in the ’90s, right? Everything futuristic is called 2000, like- Lex Fridman (01:01:30): Yeah, 2000, yeah. Jean-Baptiste Kempf (01:01:32): And so they do the Network 2000 project. It’s completely hacked. It crashes after 45 seconds. That’s okay. The demo is 40 seconds. It leaks memory. That’s okay. They put 64 megabytes of RAM instead of the 8 or 16 you have, and the demo should have stopped there. And that was the Network 2000 project by the students. Lex Fridman (01:01:49): What was the format of the video that they had to work with? Jean-Baptiste Kempf (01:01:52): MPEG-2 because satellite is MPEG-2 TS for transport, MPEG-2 video, and MPEG-2 audio at that time. And the project should have stopped there. Everyone was happy. They had, like, amazing ATM network at 155 megabits per second. They had probably one of the best networks in Europe at that time, and they stopped the project. Six months or a year later, two students arrive and say, “Well, you know what? Maybe other people care about video streamed on a local network,” and they create the VideoLAN project, VideoLAN. And one of them is called Christophe Massot, that is a good friend of both Kieran and me, and they start the project. It’s not even open source yet, and they spend around three years to get the school to agree to make it open source. Jean-Baptiste Kempf (01:02:38): Because the university wanted to get some– because of the IP and copyright of the students, wanted to basically monetize these MPEG-2 decoders. Lex Fridman (01:02:47): Just to be clear, so what was the main application, streaming on a local network? Jean-Baptiste Kempf (01:02:50): It was streaming on a local network. Lex Fridman (01:02:52): By the way, that’s just, like, to state the obvious. This is before YouTube. This is before- Jean-Baptiste Kempf (01:02:56): Ten years before YouTube. You have a Pentium 60 or 75, right? You, the main machine was 486DX at 33 megahertz, right? Kieran Kunhya (01:03:04): Bear in mind, television was the main form of video at the time. You could get new channels. In the ’90s, having even one new channel when you grew up with four channels, having a fifth or a sixth was a big deal, and so having this satellite service with, you know, dozens, even hundreds of channels was so groundbreaking. Jean-Baptiste Kempf (01:03:22): Especially because this is university where you had a ton of different nationalities, right? So there was a ton of people who wanted… So in the end, they had, like, several dishes on different types of satellites, right? Because, for example, a lot of people were coming from the Maghreb or the Middle East and they went to different types of satellites. Anyway, the solution worked great, and they started the VideoLAN project. The VideoLAN project has several, and some are completely crazy solutions, like one how to create multicast on a unicast network, but let’s not come to that. It’s too complex. But the VideoLAN Client part is what became VLC. Jean-Baptiste Kempf (01:04:03): Actually, they basically strong-armed the university to force it to open source, because the university did not understand that. And in 2001, it’s still early. But basically, yes, the university agreed early 2001 to make it open source. I joined the project in 2003 because that’s when I joined the university. So the first thing is I’m not the one who created VLC, because actually no one did, right? Lex Fridman (01:04:26): Just kind of naturally emerged from the VideoLAN project. And we should mention that, like, again, you said it just, but to make it clear, VideoLAN as what it became at the time was a set of technologies around video, and the VLC, what you called the client, that’s the thing that most normies, uh- Jean-Baptiste Kempf (01:04:47): That is correct, and- Lex Fridman (01:04:48): … think of, like, as the thing, which is, like, the thing that pops up when you click on a video and you play it. Jean-Baptiste Kempf (01:04:53): So I arrive in 2003, and then I will create the open source nonprofit organization called VideoLAN, and I took everything out of the university to create it as a nonprofit project and something sustainable. It’s, yes, it’s true that I spent more time than anyone on VLC and VideoLAN. That is sure. But it’s a continuity of a previous project, VideoLAN, the student project, which is a continuity of the Network 2000 project, which is a continuity of that and that. Lex Fridman (01:05:22): I’m sure there’s moments along the way there you were thinking of, like, what is the future of this from an open source perspective? ‘Cause as, as the internet is blowing up, and there is companies… I mean, for people who don’t remember, like, there’s companies making huge amounts of money. Jean-Baptiste Kempf (01:05:39): And I can tell you that in 2005, the project should have died and I made it to continue the project. At some point, we were only two active developers. And I thought it was great technology and was useful, and it will be useful, and I made that my life and my time. And I made that grow from a few hundreds of thousands of users, millions of users to what we have now, which is probably billions of versions of VLC around the world and used everywhere. So that’s a bit the story of VLC. There is a ton of very funny stories around that. Many people from around the world working on it, like you said, in Syria or the middle of nowhere in India. But along the way, I got several offers which were either to bundle toolbars, right? You remember those horrible toolbars- Jean-Baptiste Kempf (01:06:35): … which were basically spyware, or changing your web browser or your search engine or even, like, advertisement inside VLC. And I didn’t like that, right? I am– and people don’t understand that. It’s not– I’m not against money, right? I’m very happy to make money. I created several startups and one I hope that is going to work very well. It’s the fact that I believe that you need to win money ethically. There is a right way of doing that, and doing sneaky advertisement or stealing data is not the correct way, right? For example, if Netflix arrived at some point and said, “Well, we want to put Netflix inside VLC,” probably the story would have been different, right? But they didn’t. The only people who came to us were shady ads companies. Jean-Baptiste Kempf (01:07:22): And if I do that, right, I would have a ton of money, right? And then three years later, the project is gone, right? Someone forks it and something else happens. Lex Fridman (01:07:31): So it’s not even necessarily ads or any of that, it’s the shadiness of the- … dishonesty of the– So you had a good radar, you had a good threshold of like, “No, this compromises the spirit of what this is supposed to represent.” Jean-Baptiste Kempf (01:07:46): But also it’s for me, right? I’m like very selfishly, I need to go to bed at night and be happy about what I’ve done, right? Maybe it’s my upbringing, maybe it’s my parents’ fault or whatever, right? But I believe there is right and wrong, right? And this was the right decision at the time. It still is. I want to be proud of what I’ve been doing. And like, if I had sold out, I would have betrayed so many other people who work here. Lex Fridman (01:08:14): Yeah, well, I should say me and most of the internet thank you for that decision. It’s inspiring for others I think that are pushing the open source movement forward, that it’s okay to do these kinds of huge sacrifices if you believe it’s right. And I think in that case it was right and it was the reason that VLC became as successful as it was, ’cause it’s an embodiment, it’s a symbol of like, you know, freedom and what the open source community can create. Jean-Baptiste Kempf (01:08:46): Yeah, and be a service for so many people around the world, and this is important. Kieran Kunhya (01:08:50): We should emphasize in the 2000s it was really normal to download a program and it secretly installs some spyware. It was, it was buried in very faint text or in the license text box that nobody reads at the bottom- … “Oh, I will be installing this toolbar- … and changing all these things,” and it was very common to have to, you know, you install a program to do something at the time of any sort. Lex Fridman (01:09:11): To put yourself in the mind of a developer at that time, I think it’s very easy—to everybody listening to this, it’s very easy at that time to convince yourself to take a few thousand dollars- … a few thousand dollars to do it. To say no to much more money- … takes guts and takes vision. Jean-Baptiste Kempf (01:09:34): The last offer I had was obscene, and they say, “Yeah, but imagine with all that money you could build something new, open source,” right? It was like the mind trick was… it was difficult. But for me it was just like, “No, this doesn’t work like that or this is not the right thing, so I don’t do it.” And again, right, it’s not that I don’t like money or whatever. It’s just like it wasn’t right. Lex Fridman (01:10:01): Well, once again, thank you from me and from the rest of the internet. Let me talk a little bit more about the open source movement, about the fact that, as you say over and over and over and over, FFmpeg is and many open source projects are built by volunteers. So there’s a bit of drama recently, Kieran, on the interwebs, on Twitter. You have a spicy style on Twitter that I think articulates and celebrates all the incredible developers and development and the code, especially assembly, that’s involved in building some of these codecs and building some of this incredible technology. But that brings us to the… a bit of a debacle that happened. Tell me the full saga of what happened with the Google security engineers. Kieran Kunhya (01:10:50): Just to be clear, Google are one of the biggest supporters of open source out there. They have been for a long time. It’s just I think some things kind of went a bit overboard this time. So FFmpeg itself—and this is not like a secret, it’s on the homepage, you know—it processes untrusted data. There can be security issues when you parse untrusted data. That’s very normal. But recently what changed was Google started using AI to create security reports on an open source project, FFmpeg. Volunteers had to deal with that. They did, they provided very limited funding, and they even went to the media first announcing how good their AI was before the issues could be fixed. Lex Fridman (01:11:29): And this is in the public forum. Kieran Kunhya (01:11:31): Yeah, this is all public. Lex Fridman (01:11:32): So reporting an issue, using AI to find an issue in the code which is a security vulnerability, and then reporting that publicly before you’re able to fix it. Kieran Kunhya (01:11:40): Yeah. It’s announcing how good their AI is, that they provided a standard 90-day industry deadline without really understanding the nature of volunteer-driven development. In addition, this vulnerability was on an obscure 1990s game codec. The way—and let’s look at it from their standpoint to begin with. Let, let’s you know- Lex Fridman (01:12:04): Yeah. Can you steer me in their case? Kieran Kunhya (01:12:06): Yeah, sure. They have substantial resources working on the security of open source projects that, you know, are ubiquitous, and they’ve used a lot of compute to do that and very expensive and very capable security researchers to do that. And that’s their viewpoint is they are contributing by doing that. But I think that’s where opinions differ. It opened up a lot of interesting fissures, I would say. It does seem that there’s a portion of the security community that look at themselves a bit like building architects that never have to go to site. You know, going to site is something that is a little bit beneath them, the actual day-to-day construction. They’re there to do their security things and it’s someone else’s problem. Kieran Kunhya (01:12:54): The security industry also kind of has a very aggressive tone towards things. The language they use is extremely aggressive. They use very strong language like, “You will get popped.” So and to Joe Public, “get popped,” you know, means something quite bad. For them it means to get hacked. The way I would look at it personally is a little bit like the padlock on your home. Not everyone… The padlock on your home or, you know, the lock on your home is there to protect against the capabilities of what it’s there to protect. It’s not there to protect nuclear secrets. It’s not there to protect Fort Knox. And it could be looked at that they’re using AI at a level of scale to go and pick those locks and then say, “Hey, your lock’s not secure.” Kieran Kunhya (01:13:42): You need to deal with this.” Whereas actually they’re the ones with resources to be able to fix this. But that seems to not be something either they’ll contribute to in terms of patches or in terms of financially. And the scale of AI is kind of the issue. The bug reports are very wordy. They’re very, very—it’s almost a denial of service by AI-generated bug reports on very niche codecs. And the other issue the security community has is everything is marked high priority. You’re going to, you know, “This is the most important thing in the world, and you need to deal with this. High, high, high, vulnerable, scary, scary, scary,” on a game codec used on one disk in 1993. Kieran Kunhya (01:14:24): And that’s where the dichotomy lies. Going around telling everyone that their padlock’s not safe—well, that’s a hobby project of somebody. The safety of that codec is consummate to what that person thinks. It’s their hobby. It’s good that they’re security analyzing it, but it doesn’t need a big scary warning, “This is a critical vulnerability.” You may also recently see that there was another quote-unquote vulnerability. It wasn’t Google in this case, but a filter could overflow and have an integer overflow, and one of your pixels could be the wrong color. And this was marked high, 7.5 severity in red. Kieran Kunhya (01:15:05): And at some point, the security industry needs to realize you can’t keep crying wolf like this because this just leads to people, you know, the equivalent thereof of putting password stickers on their PC. You know, you can’t just keep crying wolf every day. And I appreciate, you know, that’s their modus operandi is to create as much scare and fear. But from the Google standpoint, at the end of the day, they need to contribute either financially or with patches. Google uses FFmpeg at a scale probably you or I couldn’t even contemplate, millions of CPU cores. And yes, they contribute in areas mostly regarding their own products, so VP9, AV1. But in a wider sense, there’s a disproportionate level of contribution. Yes, they fund students. Yes, they fund Summer of Code. Kieran Kunhya (01:15:57): And I think… so Alex Strange is a former FFmpeg developer, I think posting in a personal capacity. Lex Fridman (01:16:02): So he posted about security engineers on Hacker News. His post reads, “The problem with security reports in general is security people are rampant self-promoters”—in parentheses, Linus once called them something worse. “Imagine you’re a humble volunteer open source developer. If a security researcher finds a bug in your code, they’re going to make up a cute name for it, start a website with a logo. Google is going to give them a million-dollar bounty. They’re going to go to DEF CON and get a prize, and I assume go to some kind of secret security people orgy where everyone is dressed like they’re in The Matrix. Nobody is going to do any of this for you when you fix it.” Uh, basically commenting on the sort of the incentives for the different people involved and misaligned. Jean-Baptiste Kempf (01:17:01): The problem here is the disproportion of means on discovery compared to patching it, right? And this is the biggest issue, right? And after that debacle, Google did some changes. Kieran Kunhya (01:17:14): They are now starting to send patches, which is- Jean-Baptiste Kempf (01:17:17): And they also now have reward tools for fixing issues. So it, it has changed a bit because of that debacle. So it’s good, right? But we’ve seen—and we talk about Google—but we have seen like some other large companies saying, “Oh, you need to fix this bug because it’s critical in our product.” Lex Fridman (01:17:33): Can you explain the XZ fiasco? The FFmpeg tweet reads, “The XZ fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion-dollar corporations expect free and urgent support from volunteers. Microsoft Teams posted on a bug tracker full of volunteers that their issue is high priority. After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead. This is unacceptable. We didn’t make it up. This is what Microsoft Teams actually did.” And then you give the image and the details and all that kind of stuff, showing that these trillion-dollar companies are not giving much money, not giving much support. Kieran Kunhya (01:18:24): They think an open source project is a traditional vendor that they have an SLA. They think a public bug tracker is actually, you know, a third-party vendor’s Jira where you can do all of these things. It’s not. It is there to report bugs. I think the thing that made this particularly heinous was the name-dropping of Microsoft, the name-dropping that this is a visible product. If this was just a general bug report, I think that would have made it a lot better. Lex Fridman (01:18:51): Yeah, so they literally said, like, “This is a big deal because a lot of people are using it in Microsoft.” I wonder what happens psychologically. So I think what happens in these companies, maybe you can correct me, is they… You’re right. They just think of FFmpeg as like a vendor that Microsoft surely is paying a huge amount of money to. They kind of assume that in their interaction, and nobody anywhere on the stack is going like, “Wait a minute. Shouldn’t we be giving like millions of dollars to FFmpeg?” Jean-Baptiste Kempf (01:19:25): And this is a very big problem in large—like we’re talking about some companies, but it’s the same everywhere, right? A lot of those companies. Like the… when we talk to that person, right, he was just like a manager on one project in Microsoft Teams, right? He had, like, never really discussed with the open source community. He had no idea, right? It was like… but the problem is that usually there is what we call OSPOs, right? Open Source Program Offices in those type of companies, and they are the ones who are supposed to discuss with open source vendors. Jean-Baptiste Kempf (01:19:58): Or open source communities. But like they often don’t explain that correctly internally, right? And here it’s just like we are not your supplier. If you want me to be your supplier, I’m very happy, right? I will send you a contract and SLAs. Like I created five companies who are doing that around open source projects, so that’s okay. Lex Fridman (01:20:19): We should say that some, some of the spicy tweets that Kieran, you’re behind, and some of the debacle produced results. Jean-Baptiste Kempf (01:20:28): Yes. Lex Fridman (01:20:28): Positive results. Kieran Kunhya (01:20:29): Donations have increased substantially. They’re still not enough to cover even a single full-time developer, but on both a, you know, awareness level and a technical level, there’s substantially more technical awareness and sort of awareness of the importance of FFmpeg as a result of X and what’s happened. I can say, you know, it solved its purpose. People realize the level of importance FFmpeg has. Jean-Baptiste Kempf (01:20:55): And on VideoLAN it’s the same, right? Like for example, a, a very simple example. For more than a year, we couldn’t update VLC on Android because of a bug on the Play Store, on Android Play Store, right? The only way we got someone to answer was to put a very spicy, as you say, tweet saying that we are going to stop distributing VLC for Android, right? And we have around 100 million people using that. And now then someone from Android actually came and discussed to us, right? We had the same issue with, with Microsoft or, or like saying that we were going to stop distributing VLC on the Windows Store. Jean-Baptiste Kempf (01:21:39): And unfortunately, we are so small that the only very strong power we have to solve those issues is blaming on social networks because it snowballs and now they listen to us. But so as large companies often have difficulty talking to us. Like for example, VLC, right, is probably one of the top 10 software used on Windows. I am not part of Microsoft ISV programs, right? I don’t have a point of contact at Microsoft, right? While I’m sure any other software, Adobe, Spotify, has a point of contact. I don’t have that, right? So raising awareness works. It’s sometimes very spicy, lot of drama. Well, X and Twitter are okay for that, but it’s efficient. Lex Fridman (01:22:30): So everybody listening to this should go follow FFmpeg on Twitter, on X, follow VideoLAN on Twitter, on X. Go donate. Donate- Kieran Kunhya (01:22:44): Thank you Lex Fridman (01:22:44): … to FFmpeg. Kieran Kunhya (01:22:45): And thank you, Lex. Over the years, several years you’ve been a supporter of, you know, FFmpeg and VideoLAN on X. You know, giving us shout-outs, appreciating, you know, what we do. Lex Fridman (01:22:55): FFmpeg for life. Jean-Baptiste Kempf (01:22:57): And for example, like Tim Sweeney, Carmack, and a few others, like very high-level people have raised also the awareness on, on our X accounts, and that helped a lot also. Kieran Kunhya (01:23:09): Karpathy as well. Jean-Baptiste Kempf (01:23:09): Karpathy, yes. Kieran Kunhya (01:23:10): Karpathy as well, yeah. Lex Fridman (01:23:12): Yeah. I mean, also, you know, outside of the fact that so many people use it, it’s so impactful on the world, it’s also a great representation of a great open source project. Like the value of assembly and C and making sure that like you take programming seriously for real world systems. Kieran Kunhya (01:23:30): It’s not just that. We’ll talk about assembly later I’m sure, ’cause that’s its whole topic in itself, but it’s also celebrating people like Andreas Rheinhardt who do maintenance. It is, I believe unpaid, as a volunteer. He’s doing massive refactorings. Andreas Rheinhardt and Anton Khirnov rewriting ffmpeg.c with threading. Celebrating those guys, celebrating the untold labor that’s gone into this that actually doesn’t change anything from the user standpoint. The files are exactly the same, but wow, the airplane has been rebuilt whilst it’s in the air. Lex Fridman (01:24:03): Christian Garcia said, “As a teenager running this account,” referring to the FFmpeg a- … account, and you responded, “Teenagers have written more assembly in FFmpeg than Google engineers.” But also just pointing out that there’s a lot of incredible contributors who are teenagers. Kieran Kunhya (01:24:19): Like JB said, we don’t care who you are, where you’re from, what you do. Teenagers have written thousands of lines of assembly over the years. Give a shout-out back in the days to Daniel Kang. So also highlighting the work of people like Ruikai Peng. This is a 16-year-old, some of his first contributions to FFmpeg, actually doing and putting some of these quote-unquote security researchers to shame by actually finding issues and fixing them and being 16. There’s no barriers. There’s no barriers to you have to study at college under this person and understand these. You can learn C, and let’s be honest, it’s from the K&R book. Learn C. You can learn assembly. We’ll talk about that maybe a bit later. You can contribute to world-class technologies. Jean-Baptiste Kempf (01:25:06): In VLC one of the oldest contributors called Felix, he’s the one doing everything on Mac and iOS. He’s starting working on VLC. He was 16. We had a guy called Edward Wang, who used to be a Google Summer of Code student who stayed for three years around VideoLAN. He was 14, right? And part of Google Summer of Code and Google Code-in, which were programs where basically we have students or high schoolers, we wrote a ton of assembly for x264 and for VLC and for FFmpeg, right? So everyone can contribute. Kieran Kunhya (01:25:41): And he also did a good job because he didn’t play the alarmist CVE heist, create a CVE (which is like a public exposure of security), and do these big scary red 7.5 high priority. He just fixed an issue in Git after three days and just fixed it. He didn’t need to go and play a big security drama about it. And I think I posted, you know, “the kids are all right.” Whereas there is a portion—I’m not saying all security people do this, but there is a portion of the security community, as Alex said, that likes to hype themselves up by creating drama. They would have happily raised, “This is a high priority CVE 8.0” or whatever on an issue that actually was in Git. It wasn’t even in a release, it was in development, and three days later was fixed. Jean-Baptiste Kempf (01:26:27): Well, I just want to put a little bit of love out there, even to the bigger community. Much love and respect to Google engineers. Like you said, they’re some of the best software engineers in the world, and they do contribute a lot— … even on the security front. And also, you know, I’m a big fan of Theo. Much love to Theo. He was part of this debacle and drama a little bit. I think when you just zoom out on the grand arc of human history, the drama contributed positively to everybody involved. Donations went up. It brought more attention to the topic, allowed everybody to bicker in a way that ultimately got them to figure out what FFmpeg is all about. Kieran Kunhya (01:27:11): So the way we looked at this is like it’s a rap battle at the end of the day, you know? No, but it is, it is. Jean-Baptiste Kempf (01:27:17): It is. It is. Kieran Kunhya (01:27:17): We say stuff, we say stuff- … but we can, we can leave it on. X is a perfect place for, you know, international rap battle. You say stuff. I say stuff about your mama, but it doesn’t mean, you know, I have an actual personal issue with her. And that’s what it looks like. The Theo situation, you know, JB can maybe expand, went a little bit too far and there was a little… But, you know, it’s just a bit of fun. It’s just a bit of rap battle. It’s a bit—it’s WWE. You know, everyone’s having a bit of fun on X. It doesn’t need to be taken seriously. You know, the teenagers thing, you know, that… So that guy was a Google employee saying, “Hey, you know, there are other ways to run an open source business.” You know, and there’s like, oh, man, just have a bit of fun, you know? Kieran Kunhya (01:27:54): That’s what the point of this account is. And furthermore, if you can teach people about the ways of open source projects, assembly, et cetera, by doing that, I think there’s a lot to be offered here. It’s not dunking on people for dunking’s sake. It’s showing actually the story that I think X learnt is these are not big corporate open source projects. This is not Kubernetes where there’s, you know, hundreds, maybe thousands of people- … paid to develop this stuff. These are just people in their basements in their spare time, and if you can address that topic in a fun and entertaining way- … I think that that’s the good thing and that’s, that’s the value of X and then the reach we have. Jean-Baptiste Kempf (01:28:28): And to be honest, right, like even at Google, Google is one entity, but so many different people, right? And there is a ton of Google engineers we work with all the time, and even like Google from YouTube to Chrome to Chrome Media to the rest of Google, those are very different type of entities. But what we do is efficient. And, for example for Theo, right? It went a bit too far. I had him… Like I calmed everyone down. I had him on the phone. We say, “Okay, like this goes too far,” and so on. But in the end yeah, it’s a rap battle, but it’s positive for the project. It—like the awareness we have on open source and, and I mean true open source from communities right now—is increased dramatically in the last two years, and this is useful. Kieran Kunhya (01:29:19): What do you think motivates all the incredible contributors that we’ve been talking about? Like, what’s the engine? It’s so interesting to see. Jean-Baptiste Kempf (01:29:26): So- Kieran Kunhya (01:29:26): Like you said, they’re sitting in the basement. What’s the driver? What’s the engine there? Jean-Baptiste Kempf (01:29:29): There are many drivers, but weirdly the main one is that what we do in multimedia plays videos, and video is cool, right? And, for example, we have so many people in the community who arrive because they loved watching anime, right? And this is like the advice when people ask me, “What should I work on in open source? How do I start?” And my answer is always the same: work on something you love. Jean-Baptiste Kempf (01:29:56): I am working on VLC because I love movies, right? And I love watching the same movies over and over, even if my wife hates me when I do that, right? But because it’s interesting, right? Because it’s a topic that you like, right? The first, that’s the first thing where people come to usually to VLC and FFmpeg. The second thing is that technically we, because we search for excellence, this is the best school ever, right? This is the best school ever of programming. If you’re good in C, in FFmpeg, if you know how to write assembly, I assure you you’re going to be one of the best programmers ever, even if you’re working on writing TypeScript, because this is the most amazing thing to do. Jean-Baptiste Kempf (01:30:38): And you will, like, have to get reviews by some of the most seasoned programmers ever who are going to look at every part of your code and tell you why it’s not great. It’s like we are the best teachers that you’ve ever had in programming, right? Kieran Kunhya (01:30:52): Andrew Kelley started Zig. He was an FFmpeg developer and started Zig after his FFmpeg school. I mean, it, it’s the place to learn so many aspects of programming in the real world, in a thing used by billions of people. You have nowhere to hide. You have to be open and honest about your flaws and, and how you can learn and be better. Jean-Baptiste Kempf (01:31:12): And what is also interesting in multimedia is that you have 16 milliseconds to display a frame. It’s not like a game engine where you can basically slow down and wait a frame. So it’s, you need to be good, right? There is no choice, else you don’t have your video. And because of how codecs, if you miss a frame, you’re going to destroy the look of the video, right? So you need to be good. You need to be perfect to have the right thing. But also is that it’s not just pure programming in the mathematical sense, right? A lot of people don’t understand, but in order to program correctly on the open source multimedia community, you need to understand how computers work. And when you write assembly, you need to understand about CPU pipelining, right? Jean-Baptiste Kempf (01:31:59): You need to understand how SIMD works, how the ALU works, right? You need to understand how I/O works, right? And this is what I think that is missing to a lot of engineers and software engineers today, is understanding what we call computer architecture. And, like, seriously, some of the debates is like, should we use this assembly call or this one? And people say, “Well, no, it’s going to be like three cycles on this type of CPU and this one,” and has massive impact on the output, right? Kieran Kunhya (01:32:27): We should expand. FFmpeg is probably one of the biggest CPU users in the world. There’s proba- it’s probably running- … as we speak easily 100 mil- order of magnitude 100 million, maybe even a billion CPUs as we speak. So every instruction matters. There’s not… The impact, at least in terms of CPU, is massive for everything that we do. Jean-Baptiste Kempf (01:32:51): So first you come because it’s an interesting subject, then you stay because it’s excellent, and in the end you’re very proud of it because it’s on the end of everyone. Like so many people like, “Oh, I’m working for whatever consulting company and I’m doing some portal to download invoices for your PG&E.” Wow, great. Like, so many jobs are like that. You’re not going to tell that to your grandma. But if you go to see your grandma and say, “I do this so that you can play video on your laptop,” they understand. And this is very important, right? Because you’re working on VLC, FFmpeg, H.264. It’s in the end of hundreds of millions of people and you have an impact. And so you can be proud of yourself. Jean-Baptiste Kempf (01:33:35): And so I think that in addition to doing a great resume, all those things are why people contribute. Kieran Kunhya (01:33:42): Yeah, those are side effects. My favorite quote on this topic is John Collison. He said, “The world is a museum of passion projects.” You know, everything out there is a passion project. And open source multimedia and open source in general, you can just do that so much faster. There’s such a faster network effect, you know? Kieran Kunhya (01:33:59): I can open a cafe and that can be my passion project, but I have to get building codes, I have to build a building, I have to find a location, I have to do all the, you know, all sorts of things. Well, in the software world, that passion project can move quickly, it can be amplified by the network effect, and that amplification can be more than the sum of the parts. You know, you can be, you can find people interested in extremely obscure things and have a network effect and make something that is truly amazing. Lex Fridman (01:34:31): And on that topic of passion projects, Tim Sweeney actually said in a reply to a tweet that was complimenting JB. He said, quote, “Many things in the world only happen because an awesome person decides to do it. This is the case with VLC.” And that speaks to something interesting to me, that it does seem that a small number of people, sometimes one person, can create something incredible in the software world. Like you said this over and over and over. I think JavaScript is an incredible thing created by initially a single person. Some of the programming languages like Python and C and Java, like just one person has this vision, has this design, and brings it—sometimes over a weekend is the initial spark. Jean-Baptiste Kempf (01:35:18): Yes, Linus built Git in two weeks. Wow. Lex Fridman (01:35:23): It changed the world, Git. I mean, it really changed the world. Kieran Kunhya (01:35:25): Linus’ passion project. “Hey, I’m uploading this tarball to an FTP, like deal with it.” Jean-Baptiste Kempf (01:35:29): But for me, it’s not just in software, right? And I believe in individuals that are going to change the world, right? And it’s with a good, as you said, vision, right? I want to do that. It is useful, it will be useful. And whether it’s going to like build trains or cars or rockets or something like, I believe people who believe in themselves and have a vision can have a huge impact for humanity. Lex Fridman (01:35:56): Let’s actually zoom out before we zoom back in. We’ll just keep going up and down the stack. So you know, we’ve been talking back and forth VLC and FFmpeg. Kieran, you said that FFmpeg and VideoLAN, VLC coexist, and there’s no central point of importance. It’s a kind of what you call the binary star system. They succeed because of each other. Can you explain the difference, how they interact? What is the- Kieran Kunhya (01:36:24): Sure Lex Fridman (01:36:24): … are they competitors? Kieran Kunhya (01:36:26): I don’t, I don’t think they’re competitors. I think, I think the simple answer is, the short answer before I go into detail is: VLC is to FFmpeg as Android is to Linux. So they depend on each other, but they, they coexist because of each other. So they are a binary star system is the analogy I used. Lex Fridman (01:36:43): By the way, I feel horrible that I just recently learned that Alpha Centauri, the closest star system to us, is a triple star system. Jean-Baptiste Kempf (01:36:50): And, and when you start doing the physics, it’s a nightmare, right? But like- Kieran Kunhya (01:36:55): Hence the three-body problem. But anyway. So a lot of FFmpeg pipelines involve the x264 project, which is a VideoLAN project. I would put a finger in the air and say 80-plus percent of those pipelines are dependent on a VideoLAN project. VLC, obviously, as we’ve discussed, a VideoLAN project, uses FFmpeg, gives it reach, exposure to weird files historically, used some donation money to fund FFmpeg development, and we’ll talk a bit maybe about some of the reverse engineering later. So it’s a binary star system. They work and feed off each other. Many of the developers are shared. There’s no central location. It’s a virtuous cycle working together. Lex Fridman (01:37:36): And we should mention that x264 is the encoder for H.264 video standard. So H.264 is the standard. X264- Kieran Kunhya (01:37:46): Is the open source implementation of the standard Lex Fridman (01:37:49): … that’s used by basically everybody- … for everything. It’s, that is the main driver of this. When you think of an MP4 file that has H.264 codec in it- Kieran Kunhya (01:37:59): If it came from a software environment, like a data center or somewhere, the chances are it was created with x264. Lex Fridman (01:38:06): And that’s under the flag of VideoLAN. Kieran Kunhya (01:38:09): That’s a VideoLAN project. So in the VideoLAN graphic, it sits in the VideoLAN world. Lex Fridman (01:38:14): And VideoLAN has a— says a bunch of stuff in it. Go to the VideoLAN website, there’s a bunch of icons. Jean-Baptiste Kempf (01:38:21): Like if you look, there are so many libraries, right? Lex Fridman (01:38:24): libdvdcss- … libdvdnav, libdvbpsi, libVLC of course, vlc-unity, libbluray- Blu-ray. Jean-Baptiste Kempf (01:38:36): Blu-ray. Lex Fridman (01:38:37): Yeah, there’s many more. Jean-Baptiste Kempf (01:38:39): And there is so many more, right? Lately, the dav1d project that we might talk about is the last project from VideoLAN. It’s everywhere, right? And we do, we have a libspatialaudio lately that we announced. We have a- Kieran Kunhya (01:38:51): checkasm. Jean-Baptiste Kempf (01:38:52): checkasm- Kieran Kunhya (01:38:53): We’ll talk about that later. Jean-Baptiste Kempf (01:38:53): … which is like an insane project- … but amazing. So, and x264 is one of those VideoLAN projects. And my opinion, for example, is that x264 was, is the most amazing encoder ever designed, and this helped the adoption of FFmpeg. A lot of people and large companies went through FFmpeg because they wanted to use x264, and x264 increased the popularity of FFmpeg. But also VLC had its popularity because it played so many files that were done by FFmpeg, right? So it’s many projects that are intertwined and work together. Kieran Kunhya (01:39:32): Yeah. Unfortunately, there’s a thing on X where VLC is mentioned and there’s people, “A quick reminder that it’s FFmpeg inside doing the actual work.” And that’s like I said, it’s not, that’s not the case. We work together. Jean-Baptiste Kempf (01:39:46): And to give you an idea, right? When I compiled VLC for Windows, I compiled around 16 million lines of code, right? One million of those are inside the VLC repository, and FFmpeg in total is probably around two, right? But so it means that so many dependencies are outside. And if you also look at FFmpeg per se, FFmpeg also is integrating third-party libraries like x264, but libopus and so many others, right? So we all depend on each other. Lex Fridman (01:40:15): Yeah, that’s why I was hoping to do this episode as we are doing that just kind of joins FFmpeg and VLC- … because it’s really two of the same, like you said, binary star system and we’re all just orbiting it. Can we give a shout-out to some of the people along the way? We didn’t really quite talk about the history of FFmpeg, so maybe can you tell me about Fabrice? Can you tell me about Michael Niedermayer? Can you tell me about some of the key figures here? Kieran Kunhya (01:40:46): Let’s just talk about the eras of FFmpeg, because there’s key eras and key people that made this possible. Fabrice Bellard, as you mentioned, creating the concept, and then probably in the 2000 era— I would call the era, Eras Tour of FFmpeg— is the 2000 era was Michael Niedermayer. So key things he got done was exhaustive support for DivX and Xvid at the time, and all sorts of weird variants of what’s known as MPEG-4 Part 2. So this predates the MPEG-4 Part 10 that we’re used to. So this was 2000 era video codecs where there were flavor after flavor of weird, weird decoders. Kieran Kunhya (01:41:27): At the time in the 2000s, you needed a new player to play every different type of file format. So there was Windows Media Player to play Windows Media formats. There was RealPlayer to play RealMedia formats. And those were the other key thing in FFmpeg at the time were native decoders for those. I actually do remember being a teenager, I must have been, figuring out there was this one player that could play, could decode these files without having separate bloated players. Because at the time when you downloaded RealPlayer, there was a ton of other stuff in there, a ton of ads, a ton of other things, and just having a simple library that was fast led to that. Kieran Kunhya (01:42:03): And then I think 2008 was a big change because that’s when H.264 got its maturity and I think something hopefully we’ll talk about a bit more. This was the beginning of high definition video. So H.264 was the key decoder of that. So I’d call that the late 2000s and 2010s, and that’s when the big reverse engineers came along and really did astonishing work. The beginning was a single player that could play Xvid, DivX, Windows Media, and RealPlayer was already a massive achievement in itself without codec packs, without weird stuff you had to download that had weird ads and weird spyware. Jean-Baptiste Kempf (01:42:43): VLC 1.0 was out in those times, 2000, 2009, 2010. And this is like where it exploded. Lex Fridman (01:42:53): Yeah, without codec packs, it just works- … across all these different- Jean-Baptiste Kempf (01:42:57): It, de facto, it’s just like all the codec packs are FFmpeg inside VLC, plus we have other modules for all the types of codecs. Kieran Kunhya (01:43:03): But back at the time that wasn’t… is there were weird, in the 2000s, there were weird codec packs with DLLs coming from this place, DLLs coming from that- Jean-Baptiste Kempf (01:43:11): With a lot of spyware. Kieran Kunhya (01:43:12): … with spyware, with you know what. It wasn’t reliable, you didn’t know, and having a single player that was open source or single playback module/player that could do this that was open source. But I think the thing to emphasize is this task in the 2000s that Michael did was Sisyphean. It was really, the number of edge cases are poor beyond comprehension in terms of you could have a Chinese CCTV system that did one weird variant of MPEG-4 Part 2, what’s known as MPEG-4 ASP, and that was a weird variant, and you had to fix that without breaking everybody else- … times a million. Lex Fridman (01:43:45): So that’s where a lot of the reverse engineering was happening. Kieran Kunhya (01:43:49): It started in the 2000s with the Windows Media stuff because that was- … proprietary. It started with the RealMedia, so with Benjamin Larsson. Jean-Baptiste Kempf (01:43:56): Kostya Shishkov. Kieran Kunhya (01:43:57): Kostya Shishkov, that era. Those were the key, that was the key groundwork. And then in the 2010s was kind of the Paul Mahol, Kostya era building, doing some of the most difficult codecs. JB maybe can talk about GoToMeeting 4 and GoToMeeting 5, and- Lex Fridman (01:44:13): What’s the GoToMeeting? Jean-Baptiste Kempf (01:44:15): So, like, let’s talk about this amazing Ukrainian guy called Kostya, who was at that time living in Germany, and who was in love with Sweden, right? He— And the guy was the most… He’s like a lot of the people in the community are very clever. He’s one of those who are, like, borderline geniuses, right? He was able to reverse engineer extremely complex codecs and he does that, and we do a bit of engineering with Kieran, but clearly not at this level. Kieran Kunhya (01:44:50): No, no, yeah. Jean-Baptiste Kempf (01:44:51): He reverse engineered binary blobs, which are 20 megabytes? Kieran Kunhya (01:44:56): Yeah, so just for reference, one megabyte binary blob to reverse engineer is probably order of magnitude a month of work, and this guy is doing 20, 30 megabyte blobs. Maybe we’ll talk about that in a minute, about the subtleties of how you do that. But this guy is doing it for very difficult and very obscure codecs. Jean-Baptiste Kempf (01:45:13): And did that for fun, right? And so GoToMeeting was a big problem with VLC because that was like the number one feature request for a long time, so I put a bounty. And the guy at some point said, “Okay, JB, I’m going to do it.” And in a matter of two months, and then he explained how he did it. He was just like, “Oh, I looked at the code, like this looked like a DCT that I used to see on WMV and so on.” He did that, and the funniest part is that the code he’s written is a ton of jokes. And there is a ton of JB, right, my name, and Kempf and Kempf and Kostya jokes inside the code. The code is beautiful, right? Lex Fridman (01:45:57): So one of the things I wanna comment is I’ve gotten a chance to speak to some of the developers, some of the assembly language level people, and they all always make everything sound like it’s kinda easy. There’s a kind of humility because, maybe just the level of what’s required to do this stuff is so high that everything else seems easy, I guess is the lesson to take away from that. Jean-Baptiste Kempf (01:46:23): So in the community, like some of the most impressive people are the ones doing reverse engineering- … and the other ones doing the assembly folds, right? And both of those type of people are amazing. x264, for example, became amazing because of a guy called Loren Merritt- … who is, was from University of Washington, I think. Kieran Kunhya (01:46:44): At the time, yeah. Jean-Baptiste Kempf (01:46:45): And who was, like, who made everything great and fast doing a ton of assembly. So this is like the golden era, I guess, where so many things got done. Kieran Kunhya (01:46:57): So, yeah, if you look at Kostya, for example, he looked at the world as a binary specification. He didn’t need documentation or anything. It’s, “I have a binary and I can figure all of this out.” And he regularly used the phrase “binary specification.” Ah, you know, it’s not a problem. And he would go away, and he would come back, and he would do interesting stuff. Lex Fridman (01:47:15): Can you actually speak to the details or add color and texture to what it takes to reverse engineer a blob? Kieran Kunhya (01:47:22): Yeah. So let’s look at GoToMeeting, for example, is a good one because I record a meeting on GoToMeeting, for example. How do I play it back without needing this GoToMeeting player? There may not even be a player. I may need to send a recording of a meeting to someone that doesn’t have a player or whatever. So first of all, there’s a ton of other stuff there. There’s an actual video conferencing client. You need to go and find—it may be easy, it may not be easy to find—the actual module doing the decompression. You need a way to actually dump the YUV data from the module. So often it involves opening in a disassembler, trying to guess where the hooks are to incorporate that module and run that module natively to decode a sample file. Kieran Kunhya (01:48:06): So figure out where this module is doing the decoding process and find a way to hook in and output the raw YUV data, ’cause you will need that- … as a point of comparison for when you actually do the reverse engineering, ’cause you’ll need to be bit exact or in some cases close to bit exact. And then you open up your disassembler, use a lot of intuition to go and figure out, you know, where the DCT is, where’s entropy coding. There is a kind of, not a rule book, but there’s always a pattern of some sort. For example, GoToMeeting, you know it will be a lot of screen codec tools. There’s also different variants, so often I think there’s, what, GoToMeeting 4, 5- Jean-Baptiste Kempf (01:48:46): Well, 2 or 3, 4, I think. Kieran Kunhya (01:48:48): 2, 3, 4. Lex Fridman (01:48:49): So as you mentioned here, going to Perplexity, GoToMeeting uses its own proprietary codec for older s- recorded sessions historically stored in WMV files that require a special decoder to play properly on Windows. Without this decoder installed, Windows Media Player and some editors cannot decode the video track, so you may only hear audio or see a black screen. Boy, do I remember that. But this is reverse engineering that. Jean-Baptiste Kempf (01:49:16): This is key, right? Because the GoToMeeting is something that not many people know anymore, right? Well, you know about Zoom and Teams and so on. But like, now let’s fast-forward 10 years, 15 years, and like this is a Gotomeeting.exe for Windows 32 bits, right? Which is like, oh yeah, but I’m on Android, I’m on an iPad, I’m somewhere else, right? How are you going to do that? I’m going to be on RISC-V, on Arm. Those are blocked, but there are tons of files we need to support for the future. And this is why those type of work are exceptionally useful for humanity. Lex Fridman (01:49:51): I just have to say, though, that reverse engineering process is mind-blowing. It’s crazy. It’s like, it’s a kinda like, you know, I’ve been reading a lot and interview archeologists. I mean, you just have so little signal. Yes, yes, you know over time you get so much experience, you understand the structure of the original code, so you can kinda start inferring basics. But you’re like archaeologists with a little brush trying to reconstruct the entire human civilization. Kieran Kunhya (01:50:22): Kieran is too humble, but Kieran has done some reverse engineering also. Lex Fridman (01:50:24): Of CineForm, yeah, at the time- Kieran Kunhya (01:50:26): CineForm, nice. Lex Fridman (01:50:27): … yeah, at the time before actually led to the open sourcing of that work. So in parallel to doing the binary side, you obviously have samples. In many cases, you don’t have many samples so you have to figure out what all the different flavors are, and you may have a So CineForm, for example, is actually a collection of different approaches and toolkits within that codec ’cause often it grows naturally. And the hard part is finding a sample that gets you kind of somewhere to start without having to implement 10 different other things. So start there. I think thankfully at the time I found a sample by pure chance that had a lot of flat blocks. Lex Fridman (01:51:03): It was animation, so that really helped a lot because it wasn’t using particularly complex coding tools, et cetera, and you could kind of get somewhere and then, and then build up and build up until you figure, “Hey, here’s a few bits here. I missed this. I missed this, this if branch that it does,” and go, “Oh.” So when we say samples, you mean sample videos- … and then, and then you’re tracking, trying to infer, like, what is this codec doing- … by observing the sample and then looking at what, at the machine lo- Kieran Kunhya (01:51:30): The machine code saying- Lex Fridman (01:51:31): At the machine code. Kieran Kunhya (01:51:31): … “Ah, I have byte, this byte is six. Take this branch.” And in a different sample, oh, it’s- Lex Fridman (01:51:36): That’s nuts, man. Kieran Kunhya (01:51:37): And, and, and- Lex Fridman (01:51:38): That is nuts. Kieran Kunhya (01:51:39): … so you see, this is nuts. Then you go to things like GoToMeeting. It’s like- Lex Fridman (01:51:43): Mine was easy, right? Kieran Kunhya (01:51:44): … ima- imagine- Lex Fridman (01:51:45): Yeah, right. Kieran Kunhya (01:51:45): … two orders of magnitude more complexity. A guy alone somewhere in Germany doing that. And for a long time, you work, you’re in a black box because a decoder, for a long time, because there are so many steps from the entropy decoding, the intra prediction, the motion prediction, the IDCT, and so on. For a long time, you don’t see anything, right? So you’re debugging purely in memory. Lex Fridman (01:52:10): Debugging guesswork. Kieran Kunhya (01:52:10): And you may have the buffer that the coefficients are stored in completely wrong, and so you may be going down a complete rabbit hole thinking it’s this and then, oh damn, that’s not, that’s, that’s something else, and- Lex Fridman (01:52:21): And you’re doing that on binaries that are tens of megabytes, millions of instructions, right? Kieran Kunhya (01:52:27): So you’re, you’re stepping through the debugger, like one by one, you know, instruction by instruction going, “Hey, this instruction changes this. This does this.” Lex Fridman (01:52:34): Pausing the program on the CPU level. Like it’s- Kieran Kunhya (01:52:36): Pausing it, yeah, on the CPU level, watching what’s going on, trying to figure out- Lex Fridman (01:52:39): Sometimes you need to, like, be in a VM, so that you can pause the VM. Kieran Kunhya (01:52:43): Yeah, pause the VM, dump the memory, ’cause there could, some of the codecs could have encryption. There could be like a DRM on there. So you need to dump the memory from a virtual machine. Lex Fridman (01:52:52): Like when I joined École Centrale Paris in 2003, Jon Lech Johansen basically broke the DVD specification and created DeCSS, showed us how he was breaking a DRM, which was MP4 FairPlay from Apple. What he did on his laptop, and I was young, I was 21, was just like mind-blowing because he was basically debugging Windows inside a type of VM with ex- Like, wow. It’s incredible. It’s mind-blowing and inspiring. Does it get, like from your experience and from what you’ve seen in the community, does it get discouraging? Does it get- Kieran Kunhya (01:53:26): People help you. People send you samples. People are keen. Sometimes you don’t have access to an encoder, so this is even more difficult because you just, you just ask and you have to ask for samples. I remember VideoLAN used to tweet for samples at one stage. “Hey, I need this obscure sample,” and- Lex Fridman (01:53:42): For a long time I was, “Oh, I need this codec, and I need this codec.” Kieran Kunhya (01:53:45): And if you were really lucky, you would find like… If you were unlucky, you’d get nothing or you’d get one or two, and then they would… Sometimes you’d find a goldmine. It’s like, “Yeah, my company has 100,000 of these files ’cause we’re dependent on it for some reason.” And so those are the, those are kind of the best because then they can test bit exactness across the huge range of coding tools. Lex Fridman (01:54:06): Can you explain bit exactness? Kieran Kunhya (01:54:07): Bit exactness, so most but not all video codecs, certainly from about the 2000s onwards, have a bit exact definition, so every implementation must produce exactly the same bits, bit for bit, in exactly the same data that comes out of a decoder. Lex Fridman (01:54:26): For like a large number of samples? Kieran Kunhya (01:54:28): For a given sample. So Lex’s implementation, JB’s implementation, and my implementation of H.264 must match bit exactly. That wasn’t the case in the ’90s of MPEG-2, probably fair to say one of the biggest mistakes the video industry made, and I think people who were in the room in ’92—I don’t think most or both of us were in diapers, I suspect—but have acknowledged… I would give a shout-out to Yuriy Reznik. He’s acknowledged that was one of the big mistakes of the era. Lex Fridman (01:54:56): And you’re saying the encoders needed to be able to run tests and then the, the, the bit exactness—I mean, that’s a nice thing to guarantee. Like there’s a parallel sort of development here on the way the, the web browser works, which is a, you know, takes HTML and displays it, and there’s no bit exactness there across the different engines. Kieran Kunhya (01:55:15): I would, I would point out actually FFmpeg is unique in the sense that it’s, it has been a winner-takes-all scenario. You have… Browsers is a good analogy because it has to parse a lot of different content and render it in a particular way, like a decoder. But there still are multiple browser engines. There’s Firefox’s one, there’s Chrome’s one, there’s a few Japanese ones that are pretty decent. That’s not been the case in multimedia in general across a wide range of codecs. FFmpeg has kind of won it all, I suppose, in a sense because of, because of the fact that you can get—every new codec added is actually worth more than the value of that codec itself because it makes the whole thing better. Lex Fridman (01:55:52): Man, this is really cool. Going to Perplexity. Yuriy Reznik is a multimedia and signal processing researcher, got his PhD in computer science from Kyiv University with over 150 papers and more than 80 granted US patents, contributor to major multimedia standards including H.264, MPEG-4, AVC-H.265 MPEG-4 ALS, G.718, and- Kieran Kunhya (01:56:17): G.71 is telco stuff. Telco. Lex Fridman (01:56:19): Oh. And so he was more connected to companies. Kieran Kunhya (01:56:21): RealAudio, RealVideo, right? That was- … very important at that time Lex Fridman (01:56:25): … Zencoder, Brightcove, Contex. This, man, I need to hang out with Yuriy. He’s legit. And he’s like one of the nicest person- Kieran Kunhya (01:56:33): Slack guy, yeah Lex Fridman (01:56:33): … ever, right? Like for example, for my startup that I’m doing right now called Kyber, right? I met Yuriy because I met him every year at the Mile-High Video Conference, which is in Denver. And he gave me like so many good ideas and good things. He’s like really amazing person. Kieran Kunhya (01:56:52): He tells us how great it is to be, you know, even know us. And then we just like, you know, you look at that and it’s—I think it’s the other way around, Yuriy. Lex Fridman (01:57:01): That reminds me of a thing that you mentioned to me about FATE testing and, like, the insanely rigorous process that’s used to test everything that’s incorporated into FFmpeg. Can you take me through the testing process? Kieran Kunhya (01:57:14): Yeah. So FFmpeg has a system called FATE, FFmpeg Automated Testing Environment. Because FFmpeg runs on so many different OSs and can be compiled with so many different compilers, there’s been a crazy number of configurations. So you can see the absurd combination of compiler variants, operating system variants, instruction sets. You can see at the top macOS has tons of different variants because it has iOS, it has tvOS. Lex Fridman (01:57:42): Well, I’m looking at a page fate.ffmpeg.org 81 minutes ago, 76 minutes ago, looking at the different architectures, the operating systems, the different compilers, Apple Clang version- Kieran Kunhya (01:57:57): Combinations are crazy Lex Fridman (01:57:58): … the combination is insane. RISC- Kieran Kunhya (01:58:00): So these are all run by volunteers, so these are all volunteer systems. The ones at the top, for example, the Macs I host in my office, for example, host all sorts of different stuff. Other people host other things. So it’s really there to make sure… because FFmpeg does quite complex C code, for example, you do have miscompilations. So the compiler will sometimes compile C code incorrectly. For example, this happens once in a while. Lex Fridman (01:58:25): Oh, there’s like, there’s a log of all the compilations. Kieran Kunhya (01:58:28): Yeah, log of all the compilations, all the tests. I think one of the other ones will show all the tests passing. Jean-Baptiste Kempf (01:58:33): If you click, you can see all the tests- … back. All tests successful. Kieran Kunhya (01:58:37): In logs test, yeah. So you see all those tests are passing of all the different codecs, all the different filter transformations, all the— the level of scale is quite crazy. Lex Fridman (01:58:50): Oh, that’s nuts. Kieran Kunhya (01:58:50): On all the combinations. It’s not just a matrix at this point. It’s like a pivot table of different combinations. Lex Fridman (01:58:56): That’s nuts. Kieran Kunhya (01:58:57): And it’s a key part of what we do because you may be able to test something locally, you make a change, but actually that breaks GCC version 11 on Mac or something like that, and you’re able to then fix that. We also have miscompilation, so the C code, sometimes the compiler can have a bug in it where it creates the wrong output, and that can have quite a big effect sometimes on a video because of the way frames have dependencies. Even a small change in the output can cascade to actually quite big glitches. Jean-Baptiste Kempf (01:59:27): You see PowerPC, you see RISC, you see ARM. Kieran Kunhya (01:59:31): There was PowerPC, there was RISC, there was weird stuff in the past like DEC Alpha. There was- Jean-Baptiste Kempf (01:59:35): You see Visual Studio, different versions of Clang or GCC. Kieran Kunhya (01:59:37): Visual Studio, Intel compiler, Apple Clang, you name it. Lex Fridman (01:59:40): What are some of the pain points? Like maybe do you have emotional triggers, maybe nightmares, about a particular operating system, a particular container, codec combination of— Kieran Kunhya (01:59:53): I mean, for me, it’s really easy because I have a day job. My company builds… The company I started builds equipment for broadcasting sports matches between TV stadiums and studios, for example. We have to work with 10-bit video, and 10-bit video has a set of challenges that you can’t process 10-bit data natively on a CPU. So that means you have to stick it in 16 bits. So that means you have six wasted bits. So there’s different packing formats to actually pack the data more efficiently because when you send that over a network, you need to save that 40%. For example, on PCI Express, you may only have bus bandwidth to do that. And so I think internally we have about… Kieran Kunhya (02:00:36): Some are industry ones and some are internal to our own hardware that we build. We have a, I think, a 5 by 5 or 6 by 6 matrix of every single format to every single other format conversion. In fact, one of them I sent you, and they’re all written in handwritten assembly, and they all support different CPU generations. So this is really traumatic, handling all these different combinations times a million. Jean-Baptiste Kempf (02:01:02): By the way, the company you’re talking about is Open Broadcast Systems. Kieran Kunhya (02:01:04): Yeah, so no, no relation to the free OBS streaming service. But JB and I have started companies broadly speaking around the FFmpeg VLC ethos, so that’s really low-level work. So in most companies, this wouldn’t be written in assembly. It would be accepted that C is fast. As you can see from that, C is not fast. Lex Fridman (02:01:26): So here it says 62 times faster than C. Kieran Kunhya (02:01:31): Yeah. So it’s taking the ethos of doing low-level programming, real-time programming, and using that for commercial applications, and JB and I have started companies around that, in many cases hiring developers from the open source community to use that ethos. And so that’s a great example of some of the things we’re doing. In most companies, it would be, say, “Oh, I’ll write this in C and it’s fast and we’re done,” but actually you can get a lot better. Jean-Baptiste Kempf (02:01:59): For me, some of the headaches we have is around some OS that are difficult to support, right? Because if you look at VLC and thanks to FATE and FFmpeg, we run on… The last version of VLC runs on Windows XP and still run there and runs on Windows 11. We work on macOS 10.7 to the latest macOS, whatever it is, right, 26. We work on iOS since iOS 9; well, we are actually iOS 26, right? We support many types of Linuxes, BSD, Solaris. The last version still runs on OS/2, right? Like there is maybe 10 users of OS/2 in the world, and one of them is maintaining VLC. Jean-Baptiste Kempf (02:02:50): Then you realize that this very small team around VLC and using FFmpeg codecs and all the other ones support more OSs than Microsoft or Google or Apple, and they have infinite amount of power and resources. But for example, the worst is iOS. In order to build on iOS 9, we need to do some very clever mixing of several version of the Xcode IDE and SDK from Apple, and do a type of Frankenstein version of that so that we can still support iOS 9, which is not supported at all by the compiler of Apple in order to still run on Arm32 on iOS 9. And you’ve seen on FATE that it was still supporting iOS 9, right? So my headaches are mostly related to the support of so many OSs. Jean-Baptiste Kempf (02:03:48): And it’s important because, like, we receive so many people saying, “Hey, thank you. I still have my iPad 2 to watch movies,” and it still works on iOS 9, right? And it’s also an impact of, like, not forcing people to buy new hardware when it works fine if you optimize it correctly. Which brings us to what we were saying about assembly. It’s also fighting, like, the fact that you need to buy something new nonstop while you could optimize more, which is a lost art. Lex Fridman (02:04:18): You gotta tell me about this lost art or this, uh- … the carriers of the flame of assembly. What is Assembly? Why is it beautiful? Why is it challenging? How does it work? Kieran Kunhya (02:04:34): So when you write assembly code, you write this using the instructions the actual processor is using directly. So most of the time you would write in a language, let’s take C as a good example. The compiler would use that to create assembly language and machine code instructions for you based off your C code. And there’s a specific flavor of assembly that we use in FFmpeg that’s called SIMD, single instruction, multiple data. So this means, for example, say I want to add five to a number in scalar assembly, so this is what’s known as you work on an individual element. So I want to have a number of– I have the number ten and I want to add five. I use the add instruction, and I add five to ten, and I get 15. Kieran Kunhya (02:05:19): With SIMD, I can have a whole vector of 16 different numbers. They could all be different. If I want to add five to that, I can run one instruction, and that one instruction sums all 16 elements. And that, as you can imagine, lends itself very well to video. Video is, you know, pixel grid, so I can perform operations on multiple pixels at the same time. The key thing that we do differently in FFmpeg is we don’t use any abstractions or any major abstractions on top of that. So there’s a part of the world that uses what’s known as intrinsics. So these are C functions that behave very similarly but not quite the same to writing assembly by hand. So the registers that data is stored in on the CPU, the compiler allocates those for you. Kieran Kunhya (02:06:10): And so the key thing to understand when we write SIMD is we have a 10x, and not percentage, 10x to 50x speed improvement. That function is 62x— Lex Fridman (02:06:21): That’s nuts. Kieran Kunhya (02:06:22): … on the FFmpeg account, as you know, posts and tweets a lot about that to try and say, “Hey, we are doing this stuff.” Lex Fridman (02:06:29): You are a person who sees the beauty in assembly, but it’s also extremely useful for these kinds of application to actually- … significantly outperform even C, which is crazy. Jean-Baptiste Kempf (02:06:40): It is necessary. Right? Because, like, one of the projects that we need to talk about is called dav1d, right? So dav1d is a decoder for the format that was done by Alliance for Open Media, which is a video decoder called AV1. Lex Fridman (02:06:58): So if, for people who don’t know, we’ve been talking about H.264. AV1 is another hugely popular standard and codec that is increasingly taking over the internet. Jean-Baptiste Kempf (02:07:12): And when this format was launched many people said, especially even from the Alliance for Open Media, right, which is Google, Netflix, Amazon, Mozilla, say, “Well, this format is so complex, it must be done in hardware to do decoding,” right? And well, I arrived with a few other people, mostly Ronald, Henrik, and Martin, and we said, “We need to have an extremely good software decoder because it’s going to take time to have hardware.” And so we wrote this project, which is beyond insane. We are talking about 30,000 lines of C, but 240,000 lines of handwritten assembly, right? Lex Fridman (02:07:56): Handwritten assembly, 240,000 lines. That’s incredible. That means—I mean, some of the stuff we’re talking about is probably the biggest assembly code bases. Jean-Baptiste Kempf (02:08:09): To give you an idea, and Kieran can correct me, but I think the FFmpeg has 100,000 lines of assembly for all the codecs. Kieran Kunhya (02:08:17): For all codecs. Mm-hmm. Jean-Baptiste Kempf (02:08:17): And just this one has 240,000. It’s a VideoLAN project, of course. And it is optimized at the maximum because the motto when we’re starting the project is every cycle matters, right? Every cycle matters because dav1d is used in VLC and in some software AV1 playback stacks. We are talking about probably 3 billion devices which are going to decode video nonstop because, for example, 30% of the video from Netflix are now in AV1, 50% of YouTube, right? So, and you often don’t have a hardware decoder because not many devices have a hardware decoder. And with dav1d, we realized that with one or two cores you were able to decode 720p correctly. So it is, like, literally- Kieran Kunhya (02:09:08): Yeah, that’s dav1d. Jean-Baptiste Kempf (02:09:08): … incredible, right? Kieran Kunhya (02:09:09): That’s dav1d. Look at that, Lex. Lex Fridman (02:09:10): Yeah, so this is another spicy tweet from you. This is what peak video codec should look like, 79.9% assembly- Kieran Kunhya (02:09:20): That’s almost Lex Fridman (02:09:20): … 19.6% C and 0.5% other. Jean-Baptiste Kempf (02:09:25): And what’s incredible is with those tweets, which are factual, people get crazy. They are unhappy, right? They say- Kieran Kunhya (02:09:33): For a year, for the last two years they go crazy, “No, intrinsics is fine. The compiler is…” Oh, they go, “I have never-“ Jean-Baptiste Kempf (02:09:37): “You can optimize your compiler, auto-vectorization, it’s your fault, you don’t understand.” And we’ve tried that forever, right? Kieran Kunhya (02:09:44): For two years, and two years later, showing hundreds of examples of handwritten assembly. “No, no, no, you’re doing it wrong. The compiler can do this.” Lex Fridman (02:09:52): So we should actually just articulate a little clearer. So the intuition there from the software engineering folks, when you have code like… Okay, let’s just take an example, C++. There’s a compiler that’s doing a lot of the optimization. And the presumption is if you have a good enough compiler, if you continue to improve the compiler, you’re going to generate code- … that can perform like optimal performance. You cannot possibly beat it. And you’re consistently challenging that thought that if you do- Kieran Kunhya (02:10:21): By orders of magnitude. Lex Fridman (02:10:22): … by orders of magnitude- … handcrafted assembly can outperform C. Jean-Baptiste Kempf (02:10:27): The two things that they tell us is, yeah, but modern compilers have auto-vectorization, right? Because SIMD that we’re doing is vectorization. And like it’s not even close, right? It’s not even close, right? It’s not like 5%, 10% slower. It’s multiple times slower. Lex Fridman (02:10:43): So can we… I don’t know if you can say something philosophically, because there’s a lot of, there’s a lot of great software engineers, great engineers, great machine learning people. Karpathy will listen to this and say, “What’s the intuition he’s supposed to get from this? What are we supposed to…” Kieran Kunhya (02:10:57): Karpathy learnt assembly because of the tweets, by the way. I just… He start- He went, he’s like, “Oh, I think this is a movement.” Lex Fridman (02:11:01): He’s like, “Let me figure out what’s happening here.” Kieran Kunhya (02:11:02): No, no, he… and you know the way he documents his work and so on. Jean-Baptiste Kempf (02:11:05): Philosophically, what’s important to realize is that we passed the time where hardware was going so much faster, right? We are at the end of Moore’s law. We have limitation for AI, for memory. You need to go down in the stack and optimize more to get more power from what you have, because our request for power, CPU power, GPU power are exploding while the hardware is not exploding in speed, right? So what people do is that they add more cores, right? But that’s basically like at some point you can add 250 cores, right? So what we do is to take every inch of the machine. Kieran Kunhya (02:11:46): Not just that, not just that. We abuse the machine. We go and use the machine in ways that the creator didn’t expect. Sometimes we use an instruction that’s completely unrelated to what we do. We use a cryptography instruction in video processing to do nothing related. Jean-Baptiste Kempf (02:12:01): And one of other things that we do, for example in dav1d, which is a bit crazy, is that we don’t use the function calling convention from the operating system. Kieran Kunhya (02:12:13): We should explain that. Jean-Baptiste Kempf (02:12:14): That is extremely- … complex. But basically, usually when you do move from one function in code to another, there is a way to save the registry, the state of the CPU to enter another function. And this is like standard. Kieran Kunhya (02:12:28): It’s a bit complex. I would simplify this a bit. So, dav1d does things to abuse the calling convention. You could define the calling convention as I’ve written a function and I want to call another function. How is the data shared between the functions? Because there’s a convention, what’s known as a calling convention, and what dav1d does for optimal reasons is create its own calling convention sometimes. So if I wanna call Lex Fridman’s library, we’ve got to agree on a convention so that I can share data with you in the assembly language space. And one of the challenges in assembly is every operating s- well, not every operating system, but there are, well, at least four that I can think of on x86, Linux 32-bit, Windows 32-bit, Windows 64, Linux 64. Kieran Kunhya (02:13:11): They all have their own calling conventions. And so one of the amazing things Loren Merritt did, who we talked about before, was create a very lightweight abstraction layer, so you could write your assembly code once and it handled all the calling convention stuff for you, which was always a problem because you had to manage four different variants. But dav1d takes this even further, for speed reasons it does its own calling convention within itself to bypass the kind of rules, the rules of functions and say, “Okay, actually I’m gonna call a function this way because I know it’s within my library.” Lex Fridman (02:13:45): Does it have to be special to every single operating system? Kieran Kunhya (02:13:48): Well, if it’s custom, no. But the challenge is in general, yes, and in terms of each instruction set. So the thing to also emphasize is we do this on every instruction set. So every instruction set has its own handwritten assembly, which is even more crazy. And that matrix has got bigger in recent years because of RISC-V, because of ARM64, because of the new SVE. There’s SME. x86 has AVX-512, AVX. So we do runtime processor detection. We see what the machine FFmpeg is running on or dav1d’s running on is capable of, because you could be on a laptop from 2008 where this isn’t there. Runtime detection, we set function pointers accordingly. And then from then on, off you go. Lex Fridman (02:14:34): Or you could be on a machine with RISC-V. Jean-Baptiste Kempf (02:14:36): Yes. And in all that, we don’t even respect the calling convention of the operating system in order to be faster, because we know that we are going to be called from within our binary, so we can share data without saving all the registry in the common way, because that can lead to loading and saving registry on the L1 and L2 CPU and gets us faster. So that’s why I said that understanding CPU architecture, computer architecture is key. And this is also why it’s handwritten. I don’t know anyone, I’ve never heard any other project than Dav1d doing that. This is why Kieran calls it an art, right? It is an art. Kieran Kunhya (02:15:14): I think in a mass world, there isn’t something on billions of devices. I know there are some specialist industries. I know in high-frequency trading, they take this really seriously, where they’re receiving feeds from a market, and they need to react within X number of microseconds, and so the instructions matter. But that’s not a mass-produced thing that’s on a billion devices. That’s hyper-specialized, running on hyper-specialized hardware. We’re running on all hardware from- Lex Fridman (02:15:39): Sorry to linger on it, but, like, that’s a really counterintuitive, almost, like, revolutionary idea here, that there’s a huge amount of value to assembly. Like, what are we supposed to take away from that? Like, what… You know, there’s a bunch of people listening to this, they’re basically like, sorry, for myself included, you know, I programmed for many, many years in C/C++, going up the standards of C++, fell in love with C++, even meta programming and so on, and then transitioned more and more because of machine learning about 15 years ago to Python. Lex Fridman (02:16:12): And so, like, for me in this Python world, JavaScript world, now vibe coding, where I’m just using natural language, sitting in my jacuzzi, drinking a drink… and just talking to the computer, like record stops. Why is the value to go back all the way down to the low level? Like, what’s the intuition? Jean-Baptiste Kempf (02:16:31): Because you can get more power per dollar invested, right? And sometimes it’s going to be a problem that is limited by your hardware. A good analogy is what you see in quantization in LLMs, right? And people are doing, “Oh, I’m going to do that in FP8 or FP4 or some crazy things like Microsoft Phi, who did it in 1.5,” because you’re constrained by memory, because you’re constrained by the machine you can run. Because at some point we are doing real time, and I believe this is going to happen on AI inference also, is that at some point you need to get faster, and you cannot always get more powerful hardware, right? So you need to analyze code and see where, like, where is the mission critical, where are the things that are called nonstops. Jean-Baptiste Kempf (02:17:21): And for example, dav1d is a good example. It’s going to be run billions of hours per day. That makes sense. It doesn’t make sense to be on the glue of FFmpeg- … CLI. It makes sense over there. Lex Fridman (02:17:35): Yeah, and this has to do, also we’ll talk about it more, but your new effort, your new company, Kyber, is doing that kind of thing for ultra-low latency, so the slogan being, “Every millisecond counts.” And when you actually extremely highly constrained in some dimension- Jean-Baptiste Kempf (02:17:53): We are also arriving at a point where we’ve done so many great things, but the hardware is getting back to us, right? Because cost is increasing, because we need more power, and so you’re limited by either your CPU, your RAM, or your networking, and you need to optimize, and this is where value is going to be. Especially because, like, doing AI is going to help do the programming of, like, business, right? And so the core thing that you will not be able to vibe code are optimization for the hardware to be as fast as is possible. Lex Fridman (02:18:31): I’d love to talk to you about who and how should learn assembly, but first, I think we need a bathroom break. Quick ten-second thank you to our sponsors. Check them out in the description. It really is the best way to support this podcast. Go to lexfridman.com/sponsors. And now back to the episode. All right, and we’re back. There’s this nice repo with the assembly lessons. First of all, do you think developers should learn how to program in assembly, and how would you go about learning it? What is this asm-lessons? Kieran Kunhya (02:19:08): So I personally wasn’t happy with the way assembly is taught in books and online, ’cause it’s very grammar-focused, and you don’t, in general, learn a language from learning the grammar and the structure. You learn a language by asking someone what their name is, and you start from there, and you go and solve real problems that you have when you want to communicate. You don’t learn sentence structure, and this is the interrogative and the adverb, and all the assembly books seem to be doing like that, going through every instruction, even ones that aren’t really relevant, explaining what they all do and how they… It actually doesn’t really change much. Kieran Kunhya (02:19:43): So, and the other problem that we have in our community is assembly is taught sort of hand to hand, like person to person, like blacksmithing one by one. That’s the only logical sort of analogy, and that doesn’t really scale online. It doesn’t do other things. So I’ve started a set of assembly lessons in the way it’s done in FFmpeg, which is a little bit different to the way assembly in general for… I don’t know. I’m trying to think the other good big use case of assembly is in embedded devices, in really low power, cheap devices, and that’s completely different to what we’re doing here. I think it would be good if you could highlight the requirements, which are quite simple. It’s high school mathematics and C. Kieran Kunhya (02:20:23): And actually not even C, really, really it’s pointers. To emphasize, yes, we’ve talked about how brilliant this stuff is, but high schoolers like Daniel Kang have written assembly in FFmpeg. I think there’s been contributions because of these lessons. So it’s really about trying to get this dying art to continue, because we’ve shown it’s possible with dav1d to produce something amazing. There’s still a lot of codecs in FFmpeg that are only maybe partially assembly optimized. And so it really starts with basics and continues, explains a lot of the jargon, a lot of the syntax. It doesn’t really try and explain to you, you know, interrupt handlers and interrupt instructions and all of these different jump targets; actually makes this really vector focused. Lex Fridman (02:21:08): And describes all kinds of registers, general purpose registers, vector registers; really nice examples. Oh, this is cool. Kieran Kunhya (02:21:18): It’s a classic, yeah, it’s a classic example of FFmpeg. But some of this assembly language is really beautiful, and I think it’s beautiful because it’s kind of like flying a Spitfire. It’s really aviation at its purest, but also pushing the aircraft beyond what the designer thought was possible. So we’re abusing, for example, sometimes cryptography instructions to do certain things, and there’s a level of beauty and art where it’s really you and the processor. Kieran Kunhya (02:21:46): There’s nothing in between. It’s you and the joystick of the cockpit, and you move that joystick, and it’s physically connected to the ailerons, and you can push that plane beyond what it can normally do, and there’s a level of, yeah, beauty and amazingness to go that. But I don’t think the sort of person-by-person assembly that is… someone taught me, and I’ve taught multiple people, is gonna work long run just because of the particular flavor and the way that we do it. Lex Fridman (02:22:16): It’s literally no, I should… I was gonna say wizards handing it down. I realize I look like a wizard- … wearing this hat. But you’re basically just like the sages, the wise sages handing- … down the craft. Can I ask you about LLMs? Like- … can they help? Kieran Kunhya (02:22:32): They had more of an understanding than I expected, but they are still… I’ve asked it questions, and it still goes and starts not hallucinating, but making modifications, and then I go, “Is it bit exact?” “No.” “Fix it.” And then it just goes and does the same thing, and it’s going, it… There isn’t the corpus of information like Stack Overflow to work on. Jean-Baptiste Kempf (02:22:52): There is not enough data to train on. And this is the biggest issue. I started my career actually doing some assembly for Itanium, right? So the Itanium is a dead processor type, which was done by Intel and HP a long time ago when they wanted to do 64 bits. Well, they lost, and then we got AMD, who did it, AMD64, which became x86-64. But Itanium was extremely interesting in the sense that those were processors who had a ton of computing power to do floats, FMAs, which is similar to what we need now for LLMs, right? And you could pack three operations per line that could be loaded. So basically, you had an output of basically six billion operations per second, but the memory bus only allowed 1.5, right? Jean-Baptiste Kempf (02:23:47): So your CPU was four times faster, so you had to do crazy things to pack things in memory or reuse the registers, and those type of semantics, no language could do that, right? So like I have the Itanium programming book because Intel did amazing books, but that’s exactly what Kieran says. If you don’t know what you’re going to do, it’s impossible to read, right? It’s a ton of jargon and so on. While those lessons are amazing because they are targeted to a real problem, and you can do it yourself. Kieran Kunhya (02:24:21): And people have. People have. There are patches, and they said, “Oh, I studied your lessons, and here’s my first changes.” Lex Fridman (02:24:26): That’s amazing. Jean-Baptiste Kempf (02:24:27): And part of that in the lessons is a framework called x86inc, written by Loren when he was working on x264, and it allows you to do more things about that to create a type of like not caring too much about different calling conventions. And we had a lot of students who gave code to x264 using that a long time ago, right? So it’s really doable, and I believe it’s necessary to understand assembly language, even if you don’t do it much, to understand what’s going on inside your computer, and that will make you a better programmer. And I assure you that because doing that, you will understand some of the architecture of the memory inside your computer, right? Jean-Baptiste Kempf (02:25:14): Understanding registers, L1, L2, L3, RAM, SSDs, disk, and so on, which are very important because then you have a good programming culture that will make you a better programmer. Lex Fridman (02:25:27): What do you think about the Rust programming language? ‘Cause that’s a bit of a meme. Jean-Baptiste Kempf (02:25:31): We have very different opinions with Kieran. Kieran Kunhya (02:25:33): I think it’s valuable what they’re doing in terms of memory safety as a concept. Lex Fridman (02:25:37): Can it achieve some of the speed up that assembly achieves? Kieran Kunhya (02:25:42): Oh, not assembly by hand, no. I think that that’s a given. C potentially, but I see it very… It has a very big Esperanto vibe about it. It’s like we’re gonna solve this, and we’re doing this in a particular way. Lex Fridman (02:25:54): Meaning it’s a bit too utopian? Kieran Kunhya (02:25:56): There’s a lot of focus on the self-importance rather than solving real-world problems. It reminds me of the Sinclair C5. Sir Clive Sinclair of Sinclair Computers built a car, and he said, “Oh, everyone will be traveling around in one of these electric cars.” And it was… Rust reminds me of that, where I think the community doesn’t quite understand that in order to get people to move, you have to build something that’s as good as, if not better than what you have now. Yes, people are doing Rust rewrites, but if they only do 85, 90% of the feature set of what we need, like things like coreutils, that last 1% takes 99% of the time. Kieran Kunhya (02:26:39): To use Elon’s famous quote, “Prototypes are easy.” Like this kind of stuff is easy. But this, to get a real electric car, you have to make a car as good as, if not better than what we have now, and Rust isn’t in that stage yet. I think we’d– I don’t think anyone would object to seeing Rust code in FFmpeg, but it needs to work as well and support the same unit testing as everything else. It needs to be flawless. It can’t just randomly break. They can’t just randomly break ABI when they want to. It needs to have, I think, more– I think it still has only one compiler implementation. So it’s got to be as good as, if not better, and saying, “Hey, here’s my utopia of memory safety,” isn’t enough, even though we probably all agree that that’s the goal. Jean-Baptiste Kempf (02:27:24): So I’ve done a ton of Rust, and the two major topics I had was adding Rust modules inside VLC. One of the reasons VLC got popular and which was one of the main architectural decision, is that VLC is a very small core and a ton of modules, right? And so you can write modules in C, in C++, in Objective-C, and anything that is basically interoperable with C. And so we did some Rust modules, and so I have experience on that, and I wrote some of it. And also, like, my new startup called Kyber, is an open source project mainly done in Rust. What Rust is extremely good in, in the sense that it’s a better C++ that cares about memory and allows you to do things about memory ownership that no one else can do so far. Jean-Baptiste Kempf (02:28:19): However, it’s great when you start a new project from scratch, and you do everything in Rust. But it’s very not good when you interop with existing part. And some part of the Rust community believes that they need to rewrite everything, and everything will be better with Rust. And the answer is like, no. Like, I’m almost always, in all my years of being engineer, manager, CTO of startup and so on, don’t rewrite, right? Lex Fridman (02:28:47): Is that– That’s the, that’s the initial instinct for a lot of people when they show up to a code base, probably before LLMs, is like… probably because they don’t understand the, the, the wisdom of the way things have been done in the past. They say, “Well, we need to rewrite it.” Hence why there’s a thousand JavaScript frameworks. Jean-Baptiste Kempf (02:29:06): But the reason is the following, and this is very important to understand. It is an order of magnitude easier to write code than read code. And you see that also with LLM. They can write code, but analyzing is a lot- … more difficult. And so when you arrive to a very complex piece of code, right? You don’t understand it, right? Because it’s so much more effort to understand the code from someone else because you don’t have the thought process. And often I joke about some languages, mostly Perl for example, which has very complex syntax. And imagine I am at my maximum intellectual efficiency in programming, right? And I write the best code ever. I will not be able to understand it myself six months later, right? Jean-Baptiste Kempf (02:29:58): Because reading code is more difficult. So very often you arrive, you don’t understand all the wisdom, all the business logic, the reasons that were done that is maybe not documented. And you say, “Well, I’m going to write it.” And the thing is, no, you don’t, right? Because that’s, as Kieran said, right? I’m going to rewrite coreutils in Rust. And then, of course, you arrive very quickly at eighty percent then ninety percent, takes a bit more time, and then you got the last ones, right? On the other side, right? So for new projects, it’s great. Everything related to parsing files networking because of the memory checker, boundary checker, it’s amazing, and there is nothing else. Jean-Baptiste Kempf (02:30:40): To answer a bit differently for us, imagine I take a piece of software like dav1d or x264, right? Which has a ton of runtime in assembly, right? I rewrite the C part in Rust, right? So it’s more secure. Yes. But then you arrive into the assembly, and you can jump anywhere in the memory because we are doing handwritten assembly. So even if I rewrite the C part in Rust, for security reason, you break all the security when you write handwritten assembly because we can jump anywhere. So in my opinion, we need to do something that is secure assembly, right? Jean-Baptiste Kempf (02:31:22): So which is compile time, check the assembly, which is similar to the checkasm projects that we’re doing on dav1d and x264 with VideoLAN, is to start instrumenting your assembly at compile time to check that it’s not jumping anywhere in the memory. Because else you might rewrite a part of C in Rust, but if you want to have the same performances, you’re going to have inline assembly, and so you destroy your whole security model. So that’s a bit what I think about Rust. Kieran Kunhya (02:31:51): No, I just wanna… I would say on a personal level, I’m so in awe about assembly. I actually— Once in a… It never gets old, the speed improvements to show sixty-two x. So there are months, on a personal level, I run our internal test suite at work and just see I’m still in awe at the gains we have. Lex Fridman (02:32:10): Well, there’s a source of joy and happiness with programming for different reasons. But I think one of the greatest happiness is in the optimization of code. And it sounds like you’re, like, at the cutting edge of that. Kieran Kunhya (02:32:24): I was like, “Whoa, that was cool.” Jean-Baptiste Kempf (02:32:25): And in the community, I want to speak about two people who are wizards of assembly, right? The two of them are actually working living in north of Europe, Sweden and Finland. And Henrik Gramner knows so much about Intel x86 assembly that when we ask questions at Intel about things, they tell, like, “Why are you asking us, Intel? You have Henrik. Henrik knows better.” He knows all the cycles of almost all the SIMD instruction by all the CPU generation. “Oh, yes, this is a P4, this is a Nehalem, this is a Core 2,” et cetera. That person is, like, the best person on assembly in the world. Jean-Baptiste Kempf (02:33:12): And he’s the nicest person that you’ve seen, like, very g- He arrives, you don’t see he’s amazing. And the other one is called Martin, Martin Storsjö, and he’s– they’re doing mostly the same on Arm, right? So Neon, right? And iPhones and Androids and so on. And he codes in assembly on his phone, editing it with the crappy keyboard, like virtual keyboard you have while watching his kids play in the playground, right? Like, this is just like wizard level. So those two people are like- Lex Fridman (02:33:55): Yes. So a part, when you’re programming assembly at that high level, a part of that is knowing the architecture that you’re programming on. So x86 and- Jean-Baptiste Kempf (02:34:03): On Arm in particular, yes. Lex Fridman (02:34:04): … Arm in particular. But x86, I mean, these are complicated architectures, right? Kieran Kunhya (02:34:09): Yeah. But Arm in some ways is more com… x86 with out of order execution is not so bad. Arm, you really need to understand all the different generations of Arm processor because they’re all different. There’s A72, … … Et cetera, et cetera. And there’s the Apple variant, there’s this variant, there’s that, and you need to write code that works efficiently on all of them. x86, well, broadly speaking, you have Intel, AMD, and you have sub-variants, but generally speaking, there’s… Something fast is gonna remain fast on all of the variants, whereas in Arm it’s a completely much more complicated ballgame. Lex Fridman (02:34:43): We’re taking a nonlinear journey through history here, but we’re talking about Michael Niedermayer. And I wanted to ask about this. For a time there was a split in FFmpeg and Libav. Jean-Baptiste Kempf (02:35:00): Yes. So in open source projects sometimes you disagree, right? Lex Fridman (02:35:10): You have such a nice way of putting it, yeah. Jean-Baptiste Kempf (02:35:12): And the good thing is because of the license, you’re allowed to basically do your own, right? And this is normal, and this has happened all the time, right? At a point there was a GCC at the time of GCC 2 and EGCS which became then GCC 3, right? There is what we told KHTML with WebKit, with Blink. It is a same process. And also, like when I want to do a new feature today in VLC, I fork, I do my thing on my own, and then I merge back to the community. So there was a split in the open source community on FFmpeg, which become Libav and FFmpeg. And after a few years, well, the community merged back and people moved on. It’s a bit of drama that is normal in open source community, but forks are even… They’re important because they change the status quo of a community. Jean-Baptiste Kempf (02:36:04): Not talking about FFmpeg and Libav here, but the GCC fork made GCC a ton better because some people wanted to change the architecture fundamentally to make it faster. And of course, it’s always question of people and so on, but in the end you realize that FFmpeg today is better than it was before the fork. And now, well, we’re back all together, right? And I spent a lot of time—and Kieran can say—in the community. It’s not often, to be honest, very well explained because a ton of the reasons are not very public. But I think that’s, that’s normal and that’s good. Lex Fridman (02:36:49): Yeah. I mean, you’re making it sound really nice, but there, there is battle, there’s pretty heated battles inside open source projects. I mean, it is a very passionate community and you’re kind of in a distributed way have to define the direction of things. Lex Fridman (02:37:01): So here looking at Perplexity, “FFmpeg and Libav split in 2011 mainly over project governance, leadership style, and development processes, not because of a fundamental technical disagreement. FFmpeg effectively absorbed Libav’s work while Libav withered and most distributions and developers moved back to FFmpeg.” Yeah, that was a weird experience ’cause, you know, I’m a Linux user, so, you know, whether it’s Ubuntu and so on, all of a sudden, I think for a little bit, Ubuntu, I feel like, am I remembering correctly, switched to Libav and- Jean-Baptiste Kempf (02:37:39): 12, 14, something like that. Yes. Something like that. Lex Fridman (02:37:41): And then they switched back to FFmpeg. Jean-Baptiste Kempf (02:37:43): They switched back, yeah. Lex Fridman (02:37:43): I was like, “What is happening?” So on the sort of… you get to feel the ripple effects of the different internal debates that are happening. Kieran Kunhya (02:37:53): To be fair, on Apple, when you type GCC, you get Clang. Like they, they did something like that as well, so. Jean-Baptiste Kempf (02:37:58): Yeah. So to me it’s like the fork was like heated drama, but most of the development from Libav was merged back into FFmpeg, right? So de facto FFmpeg got a superset around Libav, and so that gave the user, because in the end we work for the users, a larger set of features and a ton of things that were discussed. For example, the debate on reviews, on how we push, are something that now is completely settled in FFmpeg and is following what mostly what everyone in the community agrees, right? So de facto, everyone who was active on Libav came back and worked on FFmpeg because the disagreements were fixed, and in the end, FFmpeg is stronger than it was before, right? And- … I know people love drama, but- Lex Fridman (02:38:52): Well, my main concern, I understand, and I think looking at the long history, it’s all for the good. But I do… I am concerned because there’s so few humans that are critical to the success of open source projects that I have seen it be a psychological toll on folks and, you know, sometimes leads to burnout. So you have these incredible people that are at the core of open source projects. There is a moment that happens ’cause, like, what is the motivation of doing it? Ultimately, it’s because you’re passionate about it and it makes you happy. Then at a certain point, you wake up and it’s like, “This has been a bit too much heat from the drama.” So, like, at the project level, the project continues and often flourishes. But sometimes there’s these individual humans that are just like- Jean-Baptiste Kempf (02:39:44): But- Lex Fridman (02:39:44): … I’ve had enough. Jean-Baptiste Kempf (02:39:45): Yeah, but it’s not just about forks, right? So it’s a very- … what you are referring to is one of the most challenging and most interesting part of open source today is maintainers burnout, right? And AI is a problem because of that. And Daniel Stenberg, who is the maintainer of curl, who’s probably one of the best promoters of open source in the world—he’s, by the way, a member of the European Open Source Academy with me, so I’m very, like, humbled to be in the same community as him, right? He’s against what he calls AI slop, right? Because it gives a ton of fake reports or- … bad reports, bad patches, and then a lot of maintainers have a lot of burden to maintain the software. And this is straining the mind of open source developers much more than forks. Jean-Baptiste Kempf (02:40:42): And for example, the XZ fiasco was because there was one guy maintaining it, and he got basically hammered by two attackers who were asking him questions nonstop at weird times at night to block him, and at some point he got fed up and said, “Okay, I can’t do that,” and gave the commit access to the attacker. So burnout in the open source community is something that exists, but mostly it’s about maintaining things, right? Lex Fridman (02:41:11): No, for sure. But I wonder how do we help that, ’cause those people are so important. Jean-Baptiste Kempf (02:41:16): Mm Lex Fridman (02:41:16): … the human beings are so important to the core of these pro- projects. Jean-Baptiste Kempf (02:41:19): So, so for example, now I am maintaining a ton of multimedia and non-multimedia libraries- …as maintainer because the maintainers got fed up, right? Some on VideoLAN, some outside of VideoLAN, because it’s… sometimes you need tough skin, right? Because you get, like, it’s not really attacks, but “oh, this is not working, this is not working,” and you feel it personally. And this is also why resources or the Google fiasco was a problem, right? They don’t realize that in the end you have, you know, it’s like the same graph where you see, like, everything and it’s just like one random open source project that is maintaining the whole— Lex Fridman (02:41:59): The Nebraska thing, yeah Jean-Baptiste Kempf (02:42:00): … internet. You see the one, right? The- Lex Fridman (02:42:01): Yeah, this is the meme. I mean, it applies to, to a lot of open source projects. Jean-Baptiste Kempf (02:42:05): So many things, yeah. Lex Fridman (02:42:05): But this is the all modern digital multimedia infrastructure, and then that thing at the very bottom that everything relies on is FFmpeg. It’s true. And then there’s usually, you know, a handful of folks that are maintaining that. Jean-Baptiste Kempf (02:42:19): And FFmpeg or VLC, right, you have a community of 10, 15 core developers, are not the worst open source project. XZ, which is even in more installations, is one person, right? There is one guy- Lex Fridman (02:42:32): LibXML is- Jean-Baptiste Kempf (02:42:34): Yeah, LibXML, right? There was a big stop. No one is maintaining- … LibXML anymore, which is like the only library that is able to parse XML everywhere. Lex Fridman (02:42:41): All the crazy edge cases of XML under ridiculous circumstances, and they get attacked by security researchers because there’s one other crazy edge case that they haven’t thought of, and it’s like, yeah, but the body of knowledge to actually resolve that is massive. Jean-Baptiste Kempf (02:42:56): There is one guy maintaining all the time zones for everyone who is in the middle of, I think, was it Nebraska or- Lex Fridman (02:43:02): Yeah, it could be, yeah. Jean-Baptiste Kempf (02:43:02): … South Dakota? Like, the mental health of the open source maintainers is something that large corporations don’t care or don’t see, right? It’s just like, “Oh, yeah, I’m just doing an open source report,” and so on. Lex Fridman (02:43:16): Mm. Some of it is financial, but people should definitely support open source financially- … all, all across the board. But some of it is also, like, spiritual on a basic human level. There’s something that happens, like, with this image of F- FFmpeg and so much of the internet depending on it, where people almost, like, talk down to the folks who are carrying these projects forward and maintaining it. Jean-Baptiste Kempf (02:43:40): In the security community, they certainly did. That was one of the things I think that argument came out is there was a portion of the security community who’s like, “No, these guys write crap code. They need to fix their crap code.” I’m like, “No, no, no, no. This is a guy’s hobby project. You have a security bot that’s gone and found some AI-generated stuff. That guy didn’t write crap code.” It’s just an edge case to the 99.99999 percentile he didn’t think about because it’s his hobby project decoding Star Wars games.” Lex Fridman (02:44:10): Forget the hobby project aspect of it. It’s just hard work, and it’s beautiful, and it’s like the right approach there is to celebrate people- … for doing incredible, incredible work. It’s, it’s just incredible that humans step up- … not getting really paid at first or maybe ever, and then they’re doing it out of the love of it, and we need to, like, human civilization runs on people like that. We need to celebrate them. Jean-Baptiste Kempf (02:44:36): To give you an idea, I received death threats on VideoLAN, right? And, um- Lex Fridman (02:44:40): You mentioned that to me. Like, what, what is, what is behind that? Jean-Baptiste Kempf (02:44:43): So that must be, what, 2009, 2010, right? Um, Apple is moving from PowerPC to Core Duo, um, that probably in 2006, and by 2009 or 2010, I decide that we are not going to do new versions of VLC for PowerPC. At that time, like VLC, we were close to the number 1.0 release. We were four of us, right? Like, just like, “No, this is not possible.” So I receive a death threat with some powder in it, right? It– Remember there was some- … anthrax threats- Jean-Baptiste Kempf (02:45:17): … at that time, right? And it was because I had taken the decision to not maintain the PowerPC port anymore. And of course, it wasn’t anthrax, of course. It was some type of flour and so on. But I received that as a, with a letter of like, “You, you piece of shit, you should die, PowerPC forever,” and so on. And it was 2009 or 2010, right? I was young. I was just like, “Why? What did I do?”, right? Lex Fridman (02:45:47): Yeah, that can break your spirit. It’s like, why- Jean-Baptiste Kempf (02:45:49): My mother freaked out, right? We had to go to see the police and so on. And now, like, I’m going to say that I’m quite happy that this happened at that time. It forged me a lot, right? I am… I can see, I can take a lot of hate on me. I’m okay with it, right? Lex Fridman (02:46:07): It sucks that that’s part of reality, ’cause all the people that love VLC, all the people that love FFmpeg, like me, you know, I, I legitimately hundred- probably thousands of times in my life had a smile on my face because FFmpeg made me happy, period. And how many times did I get a chance to say that? Zero. Until I realized there’s a Twitter account, and every once in a while I’m, like, messaging it. Jean-Baptiste Kempf (02:46:35): One of the things I like on the Reddit meme about me, which I don’t like this meme for a lot of reasons, but… And someone says, “Oh, JB is on Reddit,” which I am, right? And I say, and say hello, right? And then I got so many people who say, “Oh, thank you for VLC.” And, like, I take pictures, and then I share that to the Signal, to IRC. Yes, we use IRC on different- Lex Fridman (02:46:58): I saw as a quick tangent, you mentioned IRC is like Slack for old people. So you still use IRC? Jean-Baptiste Kempf (02:47:03): Of course. Kieran Kunhya (02:47:03): Yeah. I have it on my phone as well. Jean-Baptiste Kempf (02:47:05): Of course. Kieran Kunhya (02:47:05): Every day. Jean-Baptiste Kempf (02:47:06): Works fine. Lex Fridman (02:47:07): Wow. It works fine, huh? Jean-Baptiste Kempf (02:47:08): Works fine, yes. Lex Fridman (02:47:08): You have to power with a crank, I guess. Kieran Kunhya (02:47:10): No, but there’s no- Lex Fridman (02:47:12): There’s AOL. There’s AOL as your social media. Kieran Kunhya (02:47:13): There’s no ads, there’s no tracking, there’s nothing. Like, it’s- Jean-Baptiste Kempf (02:47:16): The biggest issue, to be honest, compared to Slack is that it doesn’t have threads. That’s annoying. It doesn’t have emojis for reaction. Sometimes it would be nice. Kieran Kunhya (02:47:24): I, IRCv3 has. Jean-Baptiste Kempf (02:47:25): Yes, V3, but no one does it, and you cannot edit your messages. Right? And the rest, it works perfectly fine forever. Lex Fridman (02:47:31): But how do you communicate without emojis? Jean-Baptiste Kempf (02:47:33): Well, that’s why I said it’s for old people. Lex Fridman (02:47:35): Old people. All right. Jean-Baptiste Kempf (02:47:37): And we do, we do emojis with like- … you know, the colons and dash and- Lex Fridman (02:47:40): Yeah, exactly. Jean-Baptiste Kempf (02:47:41): … parentheses, right? So. Lex Fridman (02:47:43): Old school. So anyway, you communicate on IRC. What were you even talking about? Jean-Baptiste Kempf (02:47:46): Yeah, we are talking about death threats and, and- Lex Fridman (02:47:48): Oh, damn. Jean-Baptiste Kempf (02:47:49): … but having people thanking you, and sometimes- I get people who send me a message and, and, “Oh, thank you for VLC.” And I always answer because I want to validate the fact that you need to thank the open source community. Lex Fridman (02:48:03): Yeah, please, everybody listening to this, celebrate, celebrate FFmpeg, celebrate VLC, celebrate all the incredible open source projects, Linux, everything. There’s so many, there’s so many… And you know what? I mean, even outside of open source, just celebrate companies that create software that you use a lot and love. Kieran Kunhya (02:48:26): Celebrate human endeavor. Celebrate the human effort to not just build something that’s okay- … build something that is damn good. Jean-Baptiste Kempf (02:48:34): Yes, this is important, right? Because as we said, right, we work for technol- we do something very complex for the normal people. Like, we want our excellence in tech to be useful for everyone. And this is why, like, this is why we work, right? This is why I wake up in the morning is because I want people to use our stuff- … Because it’s making everyone’s life easier. Kieran Kunhya (02:48:58): Want to solve hard problems. Work on something interesting, work on some interesting technical challenges. Jean-Baptiste Kempf (02:49:02): As we are engineers, we love to build things, right? When I was young, like very early, I knew I wanted to build, to be an engineer. I wanted to do cars, right? Maybe at some point I will go back to cars, right? But this is like we want to build things that are cool and useful. And they need to be challenging, right? Because you want your brain to turn on. Lex Fridman (02:49:21): When did the two of you first fall in love with programming, with building, with engineering? Jean-Baptiste Kempf (02:49:27): When is the first time you programmed, Kieran? Kieran Kunhya (02:49:29): Microsoft QBasic. As I was on Windows 3.1 and Windows 95, Microsoft QBasic. Lex Fridman (02:49:34): Oh, wow. Wow. What’d you build? Kieran Kunhya (02:49:36): Like a multiplication, just counting loops like 10, 20, 30, 40. Lex Fridman (02:49:40): Nice. Kieran Kunhya (02:49:41): Then I thought I could do everything after that. I wanted… I jumped from doing that to I want to create a soccer, no, a football, soccer video game. And I drew all the, I drew everything out. I was like, “I’m gonna do it.” And I didn’t quite grasp that actually it’s a massive piece of work to jump from BASIC and drawing some pictures to a video game, but there we go. Jean-Baptiste Kempf (02:49:59): Yeah. I think I did also Basics and then Turbo Pascal when I was, yeah, end of elementary school. But mostly the first time I actually did some serious programming was the, the first year of you call that middle school when you’re 11? I was living in Italy for a year in Florence and it was amazing year. And like the maths teacher told us to, to work in a programming language called Logo, where you had a turtle that was designing things- … On the screen, and you would turn left and right. And in the end, we used that to do a very complex programming because of course you could do things. And, and this changed, like, as I knew I was, I wanted to do things with computers and program. Lex Fridman (02:50:51): I don’t think we quite talked about H.264 properly. We talked about David. Can we return- Kieran Kunhya (02:50:57): Sure Lex Fridman (02:50:57): … backtrack a little bit to H.264, this thing that powers basically all of the video on the internet? So can you tell me the story of H.264? And Kieran, you’re actually a contributor- … to H.264. Kieran Kunhya (02:51:12): So, so H.264 is a video encoder for the H.264 video standard. It dominates internet video, but also other areas such as Blu-ray discs. And Blu-ray discs are interesting because the people that make them really want the highest quality, and there’s some really cool high-end films that have been encoded broadcasting and all sorts of other areas. H.264 was a big step change ’cause it kinda happened at the right time as well. A lot of the development took place when HD video was coming out. Intel Core 2 and Nehalem CPUs were getting fast. You could do real-time video. But the most important thing was a key sort of focus on visual metrics. Kieran Kunhya (02:51:52): So industry and academia for 20 years before, was obsessed with mathematical metrics or what’s known as peak signal-to-noise ratio. So mean squared error, logarithm of mean squared error, and that led to tons of issues because mean squared error leads to blurring because you actually want to minimize– You want to add a little bit of error to everything to reduce the mean squared error as opposed to having a big error, and that led to loads and loads of blurring. So hobbyists bucked that trend. It was for their own personal videos, mostly anime. So there were two things they did differently, and there was a big iterative feedback loop with the community. They did some stuff differently. Kieran Kunhya (02:52:29): Two big things: psychovisual rate distortion, so using block energy, trying to compensate for human perception when making decisions. Lex Fridman (02:52:38): So the psychovisual distortion, that’s the critical- … thing. That’s the thing. I mean, it’s kind of revolutionary, like, that we can, like, rethink. Don’t make it, like, this kind of theoretic thing- … of compression. Make it all about- Kieran Kunhya (02:52:56): Being pleasing visually to the eye. Lex Fridman (02:52:57): Yeah, yeah. So compressing in a way that loses the least amount of information for the stuff that matters for us humans. Kieran Kunhya (02:53:04): Yes, exactly. As opposed to what industry– Some parts of industry are still obsessed by this, which is mathematical numbers that don’t look good in reality. And then adaptive quantization was the other big one where it was biasing bits against complex areas and redistributing them to less complex areas like grass. Grass has some high frequencies, but it’s kind of– it’s less complex overall compared to more complicated things. And this came around by ParkJoy. So ParkJoy was really the canonical sample that was… Is the running around in the park. Lex Fridman (02:53:37): This one. Kieran Kunhya (02:53:38): Yeah. So this guy was really the- So this, this was created by Swedish television in the beginning of HD, and it was done on film, and it was no expense spared in terms of production quality, and it was given away for free. This was really– And this is the sample really that sorts the men from the boys in terms of it has so many challenges with the trees, with the water, with the grass, with the motion, with the… I don’t think there’s, there’s still been any, any public test sequence as good as that these days. Lex Fridman (02:54:11): So for people who are just listening, we’re looking at a bunch of humans running- … along a river, as you have the reflection, a lot of really high information textures everywhere, the leaves and the lighting playing with the leaves and all of this. Kieran Kunhya (02:54:27): You could show clearly that encoders with high PSNR- Lex Fridman (02:54:30): Will blur everything Kieran Kunhya (02:54:31): … will blur everything, and you could see actually I could turn on psychovisual stuff, I could turn on adaptive quantization, and it would just look so much better. But your metrics– And these metrics at the time were considered so holy. These are the holy metrics that are untouchable. PSNR is the most important thing. Lex Fridman (02:54:48): Can you speak to how do you measure psychovisual stuff? Like, how do you turn how pleasing a compression is for a human eye- … into a number? Is that even possible? Kieran Kunhya (02:54:59): That’s what, that’s what Netflix has been trying to do with VMAF. They said they’ve used a machine learning model. Lex Fridman (02:55:04): That’s a more recent thing. But back when x264 was being developed, that’s by eye you’re basically- Kieran Kunhya (02:55:10): It was by eye. It was developers on their laptops. So it’s not like even with big companies with professional screens or anything, it’s- And that was actually one of the goals, which was I don’t– The, the developers at the time, Loren Merritt in particular, it’s, “I don’t wanna test this on a thirty thousand dollar screen. It’s– I want this to look good on someone’s laptop at home.” Lex Fridman (02:55:27): Yeah. Brilliant. Jean-Baptiste Kempf (02:55:28): There is another sample which is, um- … A sample that is Planet Earth’s killer sample that I absolutely love. And you are going to see why, right? Kieran Kunhya (02:55:39): Yeah, you’re going to love this. Jean-Baptiste Kempf (02:55:39): The- it’s a ton of birds, right, flying, and the more it goes, the more there are birds, and at the end, right, it’s almost like you have millions of birds. It’s the most complex thing ever to encode, right? And well, you’re watching it on YouTube, and you see how bad the YouTube encoding is actually, right? And this is, like, phenomenal to optimize and get perfect quality in a constant bit rate. There was a lot of optimization, mostly by Loren also, um- on anime, right? For a long time, anime was very badly encoded because there was a ton of banding, right? And so you see those issues, and there was a ton of things. So x264 is, like– And today it’s still the reference to any encoder, new encoder, AV1, AV2, VVC, HEVC; everyone compares to x264. Kieran Kunhya (02:56:38): One of my favorite films, Cinema Paradiso, I know the engineer who created the Blu-ray, and he showed me the comparisons of x264 versus others, and the… it’s completely different. And I think a bunch, a bunch of guys in the Blu-ray world started using x264. Um, I think the big one was Chris Henderson from Warner Bros. He did the whole Fringe box set with that. So quite, like, a thing a person on the street actually watches and wants to look good. And so they kind of took a risk in their jobs doing that because they’re in a big company. That big company can buy whatever they want. Kieran Kunhya (02:57:08): And they said, “No, no, no, I want to use this free and open source thing so that things look good for my, my customers and build the best.” And to this day, I personally still try and avoid watching the most cinematic films on streaming services and buy the physical discs because they look, they look good without even having to buy an expensive TV. I think that’s the key thing. Jean-Baptiste Kempf (02:57:28): And x264 is yet another example of open source project. It was started by Laurent Aimar when he was at École Centrale Paris, where VLC was born. And then you got a generation of people like Loren, like Jason, like Måns, like so many- Kieran Kunhya (02:57:43): Henrik from- Jean-Baptiste Kempf (02:57:44): Henrik, Kieran Kunhya (02:57:45): Anton, Jean-Baptiste Kempf (02:57:45): and this is– Anton, and this is where the assembly thing that we use now on FFmpeg, David and so on, was born, right? So x264 is, like, an amazing project with people who were really all over the world, and I think most of them never met each other. Lex Fridman (02:57:59): But all of them, according to Kieran, or a large percentage, love anime. There’s several things I’ve never got into, and one of them is anime, and I need to Jean-Baptiste Kempf (02:58:09): I watch anime so much, especially at the time. Like, at the time, it was like a lot of anime content doesn’t exist commercially, right? We are before Crunchyroll, right? So what happens is usually people who love anime, who take some things, some DVDs in Japan and rip them because there is no commercial offering. And- Jean-Baptiste Kempf (02:58:33): some of the people who are, what we call fansubbers, are basically translating themselves to make subtitles, right? And at that time, you download completely illegally. It was the only way to do that, right? And so all of that was handcrafted, and it fits the open source community, right? Because they needed tools to encode, to do fansubbing, right? One of the most amazing open source projects for subtitles is called Aegisub, and it’s, it’s a subtitle… It’s done for anime, for, for, for South Asian and Japanese languages. Kieran Kunhya (02:59:06): There are weird textures in anime that I don’t think you get in real life content. I think that was a key one, which was optimizing these weird textures that you get- … because anime is not done in a normal fashion. Jean-Baptiste Kempf (02:59:17): Yeah. The way you produce it is not– You, you mostly produce it, like, on screens, right? Since a bit of time, and you have all those gradients, right, in colors? Because they are very easy to produce digitally, very complex to produce in real life. And, and the subtitles also are very complex because you need to have often the Japanese and then you need to have the diacritics, right? The what we call the ruby, right? Which is the hiragana and the katakana for the kanji. And then because of course you, so that you have the official subtitling, but you also need the English subtitles or the French subtitles because you want to learn that, right? Jean-Baptiste Kempf (02:59:54): And there is so many things crazy on, on subtitles and we had like crazy samples on, on subtitles that we’ve seen all around. So this is an important part of the, the culture, but also because there was no official offering. There was no way of doing that. Lex Fridman (03:00:11): Can you speak to the difference between H.264 and AV1 and then x264 and dav1d? This is this big step. Can you help people understand, are some of the streaming sites moving more towards that direction of AV1? Jean-Baptiste Kempf (03:00:26): Let’s be honest, all of those codecs since MPEG-2 video are the same concepts. The same concept about inverse transform, about intra prediction, motion compensation, entropy coding, all of them. However, each generation gives you a bump between twenty-five and fifty percent more compression for the same quality. And so you had the MPEG-2, you had the DivX era, you have H.264, which was, like, changing, right? H.264 improved so much. And then you had more, right? You had HEVC. You had VP9 at the same time of HEVC. VP9 is a bit similar to HEVC in terms of quality compression, but it’s royalty-free. Because in multimedia there is ton of patents and the licensing after H.264 became out of hand, right? And could cost hundreds of millions of dollars per year. So it made no sense. Jean-Baptiste Kempf (03:01:29): So Google did this VP9 and the Alliance for Open Media did this new codec called AV1. So you can imagine that AV1 saves between forty and sixty percent less bandwidth than H.264- … for the same quality, visual quality. Kieran Kunhya (03:01:48): At a given bitrate. Jean-Baptiste Kempf (03:01:49): At a given bitrate, right? So that’s really like you increase the quality– either you set the bitrate and you increase the quality, or you set the quality and you decrease your bitrate. But because now you move from SD to HD and HD to 4K and 4K to 4K HDR, like you increasing the size by like factor two, three, four, right? So you need to have better compression to keep it in terms of something that is manageable. Kieran Kunhya (03:02:16): It’s more coding tools, more bigger blocks, lots more sub-partitions in each block. It’s just exponentially more complex. Jean-Baptiste Kempf (03:02:23): It’s more complex because the encoder needs to search more possibilities, right? So you, for example one of the things that is easy to understand is to predict a block, a color block to another, you have directions, right? So you can go left, right, bottom, up, and then in terms of like the other quadrants, right? What I call north, northeast, northwest, and so on, right? But that’s eight directions. Then you can do more direction. You can do sixteen or sixty-nine or one hundred and twenty-eight, right? You can– And every time your encoder is going to spend more time to see, oh, well, this block is exactly this one and those type of tools that you can bring, and the encoder needs to check which of the tools are going to compress you better. Jean-Baptiste Kempf (03:03:09): And so I guess that AV1 encoding is two orders of magnitude more than H.264 in terms of CPU cycles, right? Order of magnitude, right? Kieran Kunhya (03:03:21): Yeah. And as we discussed, CPUs are not getting faster. You’re just throwing more cores at the problem. Jean-Baptiste Kempf (03:03:25): But also it’s a fact that you encode once and you have hundreds of millions of users, right? So for example, YouTube, a very good example. YouTube encodes almost everything in H.264, but the popular video gets re-encoded in AV1 because it costs more, of course, to encode, but you encode once and you send that to millions, right? So it’s a trade-off between encoding time and complexity- … and CPU usage on the server side and on the client side. Because at the end, if you’re distributing a video to hundreds of thousands of people and the size is half of the other, then it’s better. It’s better for your battery, it’s better for your modem, et cetera, et cetera. Lex Fridman (03:04:06): So we can lay out, let’s say, the top five codec-container combos would be H.264 inside MP4 containers, AV1 inside MP4/WebM containers, ProRes for nonlinear editing inside MOV containers. So for people who don’t know, I guess ProRes is Kieran Kunhya (03:04:32): It’s Apple’s codec for editing, originally for Final Cut Pro, and it’s designed to be fast to decode, fast to seek, because an editor will need to move very quickly. So it’s a different use case to the distribution element. Lex Fridman (03:04:45): There’s no or very minimal temporal compression in the- Kieran Kunhya (03:04:48): There’s none, yeah. There’s none in ProRes. So you can cut, so you can do cuts. Jean-Baptiste Kempf (03:04:52): This is what we call intra-only codecs, right? So I’m going to explain quickly what is IPB frames. Lex Fridman (03:04:59): Yes, please. Jean-Baptiste Kempf (03:05:00): So I-frames, often key frames, are complete frames. It’s like an image. It’s a JPEG, right? You have… You can start, you see everything, right? And then the next image can be a P-frame, which is a predicted frame. So you take some part of the previous image saying, “Well, I need the block five and seven and forty-two,” and you replace it, and then you just give the extra information, right? But that means that in order to decode this P-frame, you need to have access to a previous I-frame, right? And then, of course, you have more complex ones, which are B-frames, which are bi-predicted frames, which can depend on different type of frames, some in the past, some in the future. And so ProRes is an intra-only codec. For the people who can see, this is- Lex Fridman (03:05:54): Yeah, that’s a good one Jean-Baptiste Kempf (03:05:54): … a very good one, right? So I-frames are complete frames. Um, P-frames basically depend only on I-frames, and B-frames can depend on in front. Lex Fridman (03:06:04): And this GOP, group of pictures—I think the default for actually FFmpeg for H.264 is like two hundred and fifty frames, something like this. And to me, it’s just, it’s like magic, like that you could predict that you could have a complete frame every- Jean-Baptiste Kempf (03:06:26): Several seconds, that means Lex Fridman (03:06:28): … several seconds, and then you could still, you could have this chain of predictions you make, and the fact that you can– The fact that somebody like me can use FFmpeg to compress something and not notice that the result still plays back smoothly is, is like magic. Jean-Baptiste Kempf (03:06:43): You can even have, and we use that in tons on Kyber, is what we call intra-refresh, where basically there is no I-frames present. Kieran Kunhya (03:06:54): You have no I… You have one at the beginning. And you never send an I-frame. You get a- Lex Fridman (03:06:57): How does that work? What is it? Kieran Kunhya (03:06:58): You build up an I-frame gradually across as the stream continues, so- Lex Fridman (03:07:02): Ah. So you refresh certain parts- … of the image. Jean-Baptiste Kempf (03:07:05): But so you never have an I-frame. Like this is intra-refresh that we use, right? Lex Fridman (03:07:09): That’s even smarter. Jean-Baptiste Kempf (03:07:10): But for me, for me, the biggest mind-blown when I started was the B-frames. B-frames means bi-predicted frames can depend on frames that are coming in the future. That means that in order to decode this B frame, you need to wait for the next frame that is dependent- … buffer that, decode that one, so that you can decode the B frame, right? So the way you decode the frame, the decoding order is not the same as the display order. Right? That means the encoder needs to be very clever and decide that, “Well, you know, I’m going to depend on things like in the future.” So this is like- Lex Fridman (03:07:51): It’s incredible Jean-Baptiste Kempf (03:07:51): … mind-blowing. Kieran Kunhya (03:07:53): The fact it works so smoothly every day is kind of miraculous in some ways. It works so… You can have a stream that works across the world on their decoder versus one in the US versus one here of different manufacturers, and they produce bit for bit exactly the same material. That’s quite remarkable, and do quite complex things, and getting more and more complex and still be bit-exact. There’s a lot of work that goes into that. Lex Fridman (03:08:18): There’s a lot of knobs you can control in this whole process. There’s a lot of really fascinating parameters that I’ve gotten to know more and more over the years that FFmpeg gives you complete access to. Maybe you could speak to some of them. So first of all, like obviously, we can lower the resolution, we can lower the frame rate, we can use different kinds of codecs, as we mentioned, from H.264 to AV1. There’s ways to tune the trade-off between bitrate and quality, as we’ve kind of spoken to. You know, you could do constant bitrate, you can do constant quality, say CRF, QP. You can do the longer or shorter group of pictures, GOP, that we mentioned. I mean, all that kind of stuff. It’s crazy. Number of B-frames. Jean-Baptiste Kempf (03:09:00): Yeah. What is crazy is that a ton of people’s job is to optimize those parameters, right? A ton of people that you see at YouTube, at Netflix, at Meta, and so on, they’re not writing codecs. They’re just like finding the right parameters for the file they have, for the format they have, right? Because like something that is for a movie or something that is user-generated content from your phone or a screen recording or something that you’re going to video edit, you don’t want the same things. And there are thousands of people whose job is just to optimize all that. Lex Fridman (03:09:38): Yeah. They’re wizards. Hats off to them. YouTube, like, to deliver—all the streaming sites actually, to deliver at scale. And like YouTube is really magical because it’s not just doing like what Netflix does, which is one-way broadcasting type thing. It also has to upload videos from all the places. So they’re also doing encoding at scale- … for videos that are gonna be watched by like five people. And it still has to deliver them on a moment’s notice. No delay, nothing. I mean, very minimal latency. And also serve it in all different resolutions. Like YouTube is basically the web version of VLC. Jean-Baptiste Kempf (03:10:25): Yeah. Well, actually, it’s funny because, like Google Video, which was something they did before they acquired YouTube, was actually using the VLC plugin so that you could run VLC inside the web browser using the ActiveX plugin. And so it worked in Internet Explorer, and you were actually running VLC inside your browser. Which is funny because today we have the opposite, where we have VLC WebAssembly, where we compile all VLC and FFmpeg to decode, to run VLC inside the JavaScript virtual machine with WebAssembly. Lex Fridman (03:11:04): Okay, there’s this legendary story that you pointed me to that it was discovered via WikiLeaks release of Vault 7 documents. The CIA was using a modified version of VLC to basically try and trick people, what? To steal their data? Jean-Baptiste Kempf (03:11:23): Yes, exactly. Lex Fridman (03:11:23): So can you explain what the heck happened? What… Jean-Baptiste Kempf (03:11:27): So, so this was a surprise, right? Because at some point, WikiLeaks mentioned some documents. There were a few ones with something related to Blu-rays and VLC, but the most interesting one was the CIA Vault 7, which, if I understand correctly, was the CIA had, like, a custom version of VLC where they had a specific plugin. Yeah, exactly. This is— Like, we had to write a press release on that. Lex Fridman (03:11:53): VideoLAN wrote a press release saying the only safe source for getting VLC media player is the official VideoLAN website. I mean, I suppose that’s a security vulnerability for basically any piece of open source software. Somebody can trick you. Jean-Baptiste Kempf (03:12:09): To download in a fake website— Lex Fridman (03:12:11): Yeah Jean-Baptiste Kempf (03:12:12): … or targeted advertisement, right? That was a targeted advertisement, to watch a specific file you need to watch with this custom version of VLC. And it was the normal binaries of VLC, except they added one DLL, I think it was psapi.dll- Lex Fridman (03:12:27): Yeah Jean-Baptiste Kempf (03:12:27): … which was basically reading your document folder, encrypting that, and sending that. And the thing is, this is very clever, to be honest, because once you’re watching a movie, right, you’re going to do that for two hours, and you’re not going to touch your computer. And sometimes it’s normal because it’s HD that your fans are going up and say, “Vroom,” and there is a ton of CPU usage because you’re using VLC, right? That’s normal. But the thing is, what you don’t see is that’s actually a powered version of VLC that is used by the CIA. We had exactly the same problem with Chinese hackers that were targeting Indian people, and that got VLC banned from India until I had to fight in courts in India, the Indian government, to unban VLC. They didn’t use VLC. Jean-Baptiste Kempf (03:13:18): They took just one DLL, because we signed the DLL correctly and they used that DLL to do another program. So you had the vlc.exe and was calling libVLC, but it was calling it into a fake one. And they used that to, to target. There is not much we can do actually to block those type of hacks. Lex Fridman (03:13:39): Yeah, and I think people should, for all open source software, for all software in general, people should pay attention where they download the thing. Jean-Baptiste Kempf (03:13:46): Yes, because that means that they were not downloading it from our website. Lex Fridman (03:13:50): Do the search engines help you? Jean-Baptiste Kempf (03:13:52): No, they don’t. Lex Fridman (03:13:53): Just to clarify, ’cause you can, you know, to prevent threats from people manipulating SEO to get up there on the links and try to- Jean-Baptiste Kempf (03:14:00): Absolutely not, right? We have a big issue for, like, more than ten years, is that there is a fake version of VLC in Germany that was reported for now for 12 years, and Google basically decides to not— They know what’s in it, but the binary is too big for their virus analyzer to analyze it. And so while if you’re in Germany, you can go to a website that is a fake version of VLC with a custom installer, and it’s very popular in Germany because their website is in German, and Google mentioned that before VideoLAN. And the weirdest thing is that it doesn’t do anything on your machine for three weeks. Jean-Baptiste Kempf (03:14:38): Because that’s how they do the detection. And after three weeks, there is a small program that is a service that installed at the same time that wakes up after three weeks, and it starts downloading spyware and adware. And Google knows about it. They’ve decided not to do anything. The guys used dark SEO in Germany to do that at some point. And this is very damaging, right? Because one of the things that they are downloading is actually something that is replacing your ads inside your machine, right? Lex Fridman (03:15:08): It’s actually quite surprisingly effective. Whoever is doing it with Twitter and X. With X, I’ll get emails about, “Your X account has been hacked.” And however they phrase it, it gets me to, like, at least click on the email, not to follow the thing, and then you’re like, “Man, whatever they’re doing with the psychology to try to trick you, they’re quite good.” Jean-Baptiste Kempf (03:15:32): There is a security version of VLC, right? You received an email saying, “Hey, there is a security version update on VLC. Think about updating right now because— Lex Fridman (03:15:40): Right Jean-Baptiste Kempf (03:15:40): … it can hack your computer.” You come. It’s a website that looks decent, and— Lex Fridman (03:15:45): Yeah Jean-Baptiste Kempf (03:15:45): … and you download. It’s a new version of VLC. Great. You don’t know. A month later, you’re hacked. You have no idea. You’re part of a botnet. Lex Fridman (03:15:51): Yeah. So make sure wherever you’re downloading stuff, it’s legitimate. I’m part of the botnet. Speaking of which, so you’ve mentioned that VLC sandboxing is something you’re working on, and it’s actually something quite challenging. Why is it important? Why is it hard? Jean-Baptiste Kempf (03:16:09): So VLC is a core with around 500 plugins, right? One of them is FFmpeg, but we have, we support so many other formats. We support new protocols, we support new filters, we support weird architectures. And in this release of VLC, you have modules that are going to call your drivers, right? Jean-Baptiste Kempf (03:16:31): Mostly the hardware decoders, which are going to call your Intel, your NVIDIA, your AMD driver. And all calling FFmpeg, right? And there might be a security issue. There might be a security issue in the shader, there might be a security issue in VLC, in FFmpeg that is going to basically crash. The issue is that you’re running VLC like every, every other program, like Adobe, right? You’re running it on your machine, and it has access to all your documents, right? So the idea is to be sure that you do a sandbox so that we can protect from ourselves, because inside the VLC process is running some code that is not even ours. Either it’s open source for other projects that we integrate in VLC, or it’s your GPU driver or something that is provided by someone else inside. Jean-Baptiste Kempf (03:17:23): And so when we crash, we want to not allow people to do bad things, right? Because one of the common ways of hacking people is to crash a program, very often done with a web browser, very often done with PDF files, less often with multimedia, but that could happen. And when you crash, you launch something on the, on the machine of the person. Could be a ransomware, could be a botnet, right? So security of desktop applications is important. On mobile, it’s a bit different because most of the mobile applications are running inside their own sandbox. But for VLC, we could run it inside one sandbox, but the problem is that we need access to so many things that it’s basically we would have all the permissions, right? Jean-Baptiste Kempf (03:18:08): And so if you have a sandbox and you put some holes everywhere, it defeats the purpose, right? So what we are trying to do, and we’re actually doing, is splitting VLC into several processes. One is decoding, one is demuxing, one is filters, and all of them run into their own sandbox so that if a part of VLC crashes, like Chrome crashes on some, on some tab, right? It crashes, but it does not crash the whole program. And this is what we’re trying to do. And it’s difficult because it’s a sandbox that needs to sustain gigabits per second- Jean-Baptiste Kempf (03:18:45): … of mem copies. Now, it’s not a website which is five megabytes or 10 megabytes. We’re talking about hundreds of megabits per second. So this is why it is quite challenging. And this is a research topic that we are working on in order to have a multimedia player that is secure. Lex Fridman (03:19:03): This is all the kind of stuff you have to think about when millions of people are using. You, you’ve mentioned something somewhere where, like all the different features of VLC, when you have that many people using it, somebody will use every single feature, and they will tell you about it. Jean-Baptiste Kempf (03:19:20): Best feature in VLC is called the puzzle filter, right? So you click the puzzle filter, and it transforms your video into a jigsaw puzzle, right? Lex Fridman (03:19:30): Nice. Jean-Baptiste Kempf (03:19:30): And you can click and move the pieces, right? It’s very, very useful when you’re watching a French movie, right? You’re bored- … because it’s like very long things or a love triangle, right? We’ve seen that so many times, right? But you need to watch it because someone, your wife or- … told you to do that- Lex Fridman (03:19:49): To catch up Jean-Baptiste Kempf (03:19:49): … or your boyfriend told you to do that. So you’re doing that, right? And you can click and move the pieces around. It’s absolutely useless, right? Like, who cares about that? First, it was done by a math teacher in high school in south of France to teach his students about Bézier curves, which is something that- Jean-Baptiste Kempf (03:20:08): … everyone should know about, right? It’s very useful. But the code was clean, so it got in VLC. It was merged in 2010. Five years later, I receive an email saying, “Hello, JB. I have a problem with VLC. The puzzle is too simple.” And I was just like, “What?” And yes, the puzzle was in the UI maximums by 16 by 16, right? Only 256 pieces. And he says, “I’m sorry, but in a movie I love puzzles, this is too simple,” right? So there is a comment of me, you can check it online, which is JB changing that the dimensions are 256 by 256. Lex Fridman (03:20:45): Right. Jean-Baptiste Kempf (03:20:45): But my point is, so many useless features are used by a few people, right? There is a way to watch VLC movies in command line without any UI, right? It’s- Lex Fridman (03:20:57): I saw that. You can do ASCII. Jean-Baptiste Kempf (03:20:59): ASCII art. Is it useful? Very useful. Imagine you’re debugging… imagine you’re debugging a multicast network, right? You have thousands very complex, very complex networking stack, right? You can SSH to all of the routers and put VLC on it with no UI, and you’re going to see whether it’s black or it’s not black, right? So you see if it’s all green or not all green, right? So you can see- Lex Fridman (03:21:22): Amazing. Jean-Baptiste Kempf (03:21:23): Yeah, right. Lex Fridman (03:21:23): This is fun. Jean-Baptiste Kempf (03:21:24): People don’t realize there is so many things in VLC that are useful and they have users, because once you have hundreds of millions of users, you have people who use every feature. Lex Fridman (03:21:40): I would love to sort of zoom in and talk a little bit more about the distinction between kinda downloading a file and watching it offline versus streaming. So the complexities, the challenges of streaming. Is there something we could say about what it takes to stream files? ‘Cause we’ve been talking about codecs- … and I think a lot of that implies encoding and decoding without having to communicate- … over the network. Sure. So can you elaborate, like, what’s required to do over network stuff? Jean-Baptiste Kempf (03:22:16): Yeah, but it is less complex than it seems compared to everything that we’ve talked about. Especially because the most complex thing is not about streaming in terms of streaming services, but it was what was done to actually broadcast through satellites. Because in most of the modern broadcasting services, you can pause and you can go on. But when you’re sending live streaming, whether it’s broadcast or live for streaming services which are live, this is much more difficult because you need to encode in real time. When you go on a satellite, you have a specific size of the link, right? You cannot have a burst- Jean-Baptiste Kempf (03:22:58): … of bandwidth even for a second, right? Because you don’t have the space for that in your total file. However, there is different types of challenges, which are interesting challenges, but I think they are less complex than the one we’ve seen with late ’90s and early 2000s about broadcasting and streaming through satellite. Lex Fridman (03:23:18): They’re different. They are control systems challenges, whereas some are more mathematical. I think that’s the difference. Jean-Baptiste Kempf (03:23:23): In the streaming world, what you have is called what we call adaptive streaming, because the difficulty– and it’s not really a video problem, it’s mostly a CDN problem, is that you might have too many people watching the same thing at the same time, and it’s a congestion of the network, right? So your player has difficulty downloading things fast enough to play them. So what happens is that locally, the player is going to read a lower resolution- … of it. But there are some very clever algorithms to do that, but most of it is quite basic, to be honest. Lex Fridman (03:23:59): Even on the buffering side, it’s pretty basic. Jean-Baptiste Kempf (03:24:02): Yeah, you start to download a segment, what we call a segment, and then you time, right? And if it takes more than 50% of the time to download a segment, you go down to… Right. And the difficulty is more about when do you go up in bandwidth, in quality. But this is not very complex to do. When you encode, you’re going to encode seven resolutions, right? And you’re going to give the bitrate. The difficulty is to have your encoder give the same bitrate, but it’s not as strict as it used to be. So- Lex Fridman (03:24:34): Probably YouTube has to figure out the human psychology side of that, like how pissed off do you get when it’s like very low bitrate and how long should it wait before it increases the bitrate even though the connection is better? Because maybe the changes in the bitrate is what, like, affects you psychologically. Jean-Baptiste Kempf (03:24:58): No, I think actually the interesting one is the audio. Lex Fridman (03:25:01): That’s true. Jean-Baptiste Kempf (03:25:01): The– you can kind of notice when they move from full fat AAC to the– there are compressed versions of AAC that use spectral band replication. You can kind of see it goes a bit tinny, and that up and down is very jarring. The video side is a lot smoother, and there’s less notice. It’s really the audio you can definitely feel it from when it’s moved you from a different audio profile to one or the other. I don’t know. We’re surprisingly tolerant at skipping audio glitches. I’m surprised people I know who are not video engineers, how tolerant they are to watching sports at 30 FPS, for example, whereas it should really be 60. The world is a lot more tolerant to that, but audio people are very– There’s– It’s an immediate feedback mechanism of, “Oh, something’s changed.” Lex Fridman (03:25:44): If you hear a glitch, you realize it directly. I get to fully realize that, I suppose. One of the things I’m afraid of when I listen to audio more and more, that I get to notice every single tiny detail, and that you can over-obsess when people in general are able to kinda blur their consumption. They can look past certain imperfections. Jean-Baptiste Kempf (03:26:08): But then when you combine like an event that is, for example, a sport event that is probably going through satellite or- … somewhere else and goes to a central place for encoding, and then you need to encode this older resolution in real time. You don’t have time for QA. You need to push that to CDNs. You need to add probably DRM for protection. You need to have that over a ton of different devices. Then yes, it is complex. But– And also, like you’re in the web browser or in very much different devices that you use for television, where you had like a defined set-top box or cable box that you know where you control end-to-end. So it’s a challenge, but it’s less… I think the networking part while you agree to have 10, 20 seconds of latency, I don’t think this is very difficult. Lex Fridman (03:27:05): Speaking of networking and latency, so your new effort, as we mentioned, is Kyber, which is aimed at ultra-low latency. As you say, every millisecond counts, and you’re applying that to remote control machines like robots, drones, computers. Can you tell me about it? Jean-Baptiste Kempf (03:27:25): Sure. If you start from where we used to be, right? You used to use FFmpeg to encode files, right? And then we used FFmpeg and VLC to encode in streaming services, right? And then you need to go lower and lower. And the question was where up to where we can, can we go? Jean-Baptiste Kempf (03:27:46): And this question is very important because there are many use cases where you need to be fast, and it’s when you have feedback interaction, right? We are not just listening to something, you’re actually controlling it, right? Because– And that’s the biggest difference compared to what we’ve done so far, is that I need video to have a feedback on something that is happening live, whether it’s a drone flying, whether it’s controlling a humanoid robot from distance, whether it’s controlling a hover, whether it’s playing a video game in the cloud gaming, because this is what I did on a previous job, right? I was CTO of a cloud gaming startup. And this is a very interesting topic because you push to the limit the network. Jean-Baptiste Kempf (03:28:32): You need to care not about the quality like we’ve done on video, and we’ve talked about with H.264. You care about latency, because a millisecond is meaningful when you’re controlling a car, right? For– Well, you’ve seen, you’ve used Waymos, right? When Waymos don’t work, and that happens even if one percent of the time, there is someone that is basically remote controlling that. And this is exactly the stuff that we’re building. It’s really an SDK platform to do end-to-end control of machines. Lex Fridman (03:29:10): So the– this comes up quite a lot in a lot of different contexts in robotics. So obviously, teleoperation, teleop is becoming more and more important including for training robots via machine learning. Jean-Baptiste Kempf (03:29:25): Yes. And what we do is a bit different from everyone else, is that we take only one socket, one connection, which is a QUIC protocol based on UDP which is interesting because it’s done for low latency. It doesn’t have two of the, what we call the TCP head-of-line problem and the HTTP head-of-line problem. It’s ciphered by default, but on the same wire, we send multiple streams, like multiple tracks. We send audio, we send video, but we also send the commands, right? Mouse, keyboard, gamepad, and so on. And we do that while maintaining coherence, right? Synchronization. Because what people don’t realize is that all the clocks actually drift. Jean-Baptiste Kempf (03:30:06): And when you’re controlling a robot, a robot is going to have, like, two cameras, five cameras, ten cameras, a ton of sensors, GPS, and so on. And if you want to train correctly your robotic AI model, you need to have all those that are in sync and coherent. And what we’ve done, and it’s all the stuff that we learned on VLC in broadcast in real time, and MPEG-TS that Kierans know well, is that we account for clock drifting. And so when I record a Kyber stream, a robot, I am sure that it’s going to be predictive in the way you play it back. And so when you’re going to do recording and training of your AI model, you need to be sure that every time you retrain based on the data, the data is going to stay coherent. And clocks actually drift. Jean-Baptiste Kempf (03:30:54): Like, the existing solution works with one camera. Once you’re going to five or six, it’s more complex. Lex Fridman (03:31:00): So you wanna make sure that the visual snapshot perfectly matches the time it actually happened. Jean-Baptiste Kempf (03:31:08): Exactly. And also, if you’re going to control, right, I do something on robot, I need to be sure that it is actually happening at that precise time, right? And so we have on the server, which would be a robot, a time of, like, re-timestamping mechanism accounting for clock drift for that, right? So that’s one of the use case of Kyber to control robots. I also think, like, remote drones, remote whether it’s defense or non-defense, remote cars, remote submarines. There is many places in industry or remote surgery where the expert cannot go everywhere the machine is because it’s either dangerous or it’s too costly, right? So you allow people to have machines next to you, right? Jean-Baptiste Kempf (03:31:54): The goal of Kyber is to make distance disappear because it’s either projection of skills or projection of power, right? So imagine we are all like— you’ve seen the Meta Ray-Ban and everyone else, right? You need to stream there, right? Because you’re not going to run anything over there, right? So you need GPU power whether it’s on a cloud, on a phone to stream that. And so all of these use cases need to be not about extremely low latency, but real-time latency for video. And so that means you need– we’re toying with the encoders so that the encoders encode a frame in four milliseconds. And Kieran with his company also goes under those type of latency, because you need to optimize at max the local latency, right? Jean-Baptiste Kempf (03:32:42): Because it’s the decoder, the encoder and so on. Because this time is going to be added to your networking time. And it’s not just about low latency, it’s also about, like, reliability. We do clever things like forward error correction, right? So forward error correction is you over-transmit a bit of data, right, a few percent and while over-transmit, you’re allowed to lose some packets. Because all of that is very difficult over an internet network where you’re going to do things very far away. And if you check that all packets are delivered, you add a ton of latency. If you don’t want latency, what we do is that we over-transmit some data that you can reconstruct on the client side when there is things that are broken, right? Jean-Baptiste Kempf (03:33:34): So and a few days, weeks ago, we were doing the demo around Las Vegas for the CES about— we had a rover that is fully 3D printed. It’s very simple. It’s a car, right? It’s a small car with a telescopic arm, and it was actually controlled from France, right? And the video was with a webcam and a very small server, right? A small PCB was basically running and send that to someone that is on the other side of the planet. And so there is so many use cases. You can also think about having AI who are going to control many drones and so on. And technically, we need to be amazing in video, we need to be amazing at networking, we need to care about any milliseconds in networking, in encoding time, in decoding time, and also you need to integrate very low level. Lex Fridman (03:34:26): So sync everything together well. But how– Like, what kind of latency can you get to? Like, why– When you say milliseconds, what’s the goal? Jean-Baptiste Kempf (03:34:34): So my goal is four milliseconds glass-to-glass latency. Lex Fridman (03:34:38): What’s glass-to-glass mean? Jean-Baptiste Kempf (03:34:40): So it’s easy, right? You have a computer which is running a program, right? Probably a video game, and this one is actually running, right? It could be– it’s an example of a robot, right? And you have the replicate that is- … done through the network. And you want, if you take a one thousand hertz camera, you can take a picture, and you want that to be at four milliseconds. Four milliseconds means two hundred and forty hertz, right? Lex Fridman (03:35:06): Yes. Nuts. Jean-Baptiste Kempf (03:35:07): So far we achieve seven milliseconds from a Windows to Windows or Windows to Mac. And if you look in the timing, there is around three point five milliseconds inside the NVIDIA hardware encoder and around two milliseconds on the Intel decoder, right? So, like, the encoder plus the decoder is already six milliseconds, right? So in order to go down, we need either to have some other type of codecs or some better encoder that are faster. But four milliseconds would be the grail. Lex Fridman (03:35:44): That’s pretty nuts. I love it, though. I don’t think anyone’s ever achieved that, right? That’s fast. Jean-Baptiste Kempf (03:35:49): You can achieve that with custom hardware- … with SDI, with professional hardware. But I want that to work over the internet. I want that to work with any robots where you’re going to have a small Jetson Nano in it or a N150, right? I want that because there is going to be millions of robots or- … drones are just rolling robots or flying robots or swimming robots, right? It’s just you, a machine that you control. And in order… Either you need to teleoperate them or when everything will be fully autonomous, you need to teleobserve them, right? You need to check what’s happening. Jean-Baptiste Kempf (03:36:29): And in my view, in the future, like, all those remote cars will be teleobserved by an AI model, which is just going to say, “Well, everything is good.” And when it’s not good, say, “Hey, there is a problem,” and then you have an operator, right? And this is going to be about safety, right? When you have your humanoid taking care of your grandma or my grandma, I want to be sure that everything goes well, and I’m not in those type of horrible scenarios where the robot is dangerous. Or when I’m driving, I want, like, the car to stop when it should stop, and if needed, someone takes care of that, right? And so there is so many cases, scenarios about real time, and so the goal of Kyber is to make real time control of machine. Distance disappears. Lex Fridman (03:37:15): It’s incredible. And some of the same technology, some of the same ideas that we’re talking about is connected to what you’re doing. Jean-Baptiste Kempf (03:37:22): And for me, it’s amazingly challenging, right? Because I would say that on video I’m doing okay, but networking I have so much more to learn, right? It’s about, like, congestion protocols, bitrate adaptation in real time. But it’s quite funny. And so I created this project and we have fundraised in the US, of course. But it’s open source, right? This is important, right? Like, we’ve not said that, right? But everything on Kyber is open source. Lex Fridman (03:37:50): So how do you make money? Jean-Baptiste Kempf (03:37:51): It’s a dual license, commercial and AGPL, right? You remember what you said- … about licenses. Basically, if you want to use Kyber in your product, you must have your full product open source. If you want to use this amazing technology but not open source, you pay the commercial license, right? So the small people or the hobbyist and the very small guys who want to do that, they can use the technology. They build something that is open source and cool. Lex Fridman (03:38:18): That’s awesome. Jean-Baptiste Kempf (03:38:19): And if you’re a large company, you’re going to have the support, all the IP, the right modification, and so on. So yeah, it’s really cool and also I’m building robots, and I love that, right? Like we have– Like the rover we have is 3D printed. We are finishing a demo where it’s an actual wing, right? Like a type of drone wings that is also fully 3D printed. We are trying to do a sailboat that is 3D printed. And we’ll work on some humanoids. Of course, they are not going to be very good robots, right? It’s not our job, but we’re here for everyone to make robots. Cool. Lex Fridman (03:38:57): Ah, you’re talking to the right guy. I love robots. There’s a bunch of them upstairs. And teleop is gonna be really, really important, especially as the number of robots scales across the world. So 100%. Let’s talk about the future of multimedia. FFmpeg, VLC, but some of the codecs, we didn’t really mention AV2. So can we just lay out what is AV2? What is the hope for it? What is H.265, H.266? Jean-Baptiste Kempf (03:39:26): So AV1 is this codec that is done by the Alliance for Open Media, right? Where there is Google, Netflix, Amazon, Apple VideoLAN, where we try to make a royalty-free very good codec, right? And now it’s being deployed. But actually, the codec was finished in 2018, but a codec takes years to be used in wide scenarios, right? So AV2 is the next generation of this codec. It’s 30% better, right? So if you keep the same quality, you get 30% bandwidth reduction compared to AV1. Lex Fridman (03:40:03): What’s the connection with the dav1d and AV2? Jean-Baptiste Kempf (03:40:06): We are going to do a dav1d 2, right? That I call Devid, because de is two in French. Lex Fridman (03:40:13): Ah, well done. Jean-Baptiste Kempf (03:40:13): … And you have to know that dav1d is an actual what we call recursive acronym, right? Because it means D, dav1d, is an AV1 decoder, right? So Lex Fridman (03:40:23): Oh, nice. Nice. I didn’t even think of that. And people should know that dav1d is spelled with a one. Jean-Baptiste Kempf (03:40:29): Yes. It’s… And so dav1d 2- Lex Fridman (03:40:32): It’s gonna be spelled with a two. Please tell us Jean-Baptiste Kempf (03:40:33): … is going to be D-A-V-2-D. Sorry, I don’t know how you pronounce that. And again, we did a demo at the CES of VLC running the first demo of AV2. Lex Fridman (03:40:44): So can you clarify to me the specification of AV2? And then the encoding and the decoding Jean-Baptiste Kempf (03:40:52): Sure. So the specification is like the document which explains how the codec is supposed to work, right? Lex Fridman (03:41:00): And that’s really AV2. Jean-Baptiste Kempf (03:41:02): That is AV2, like H.264. Right? Then you have an encoder. The current encoder is called AVM, and there will probably be other encoders, probably one called SVT-AV2, and those are the encoder. The same way x264 is an encoder to H.264, the same way that x265 is an encoder for the H.265 codec. And the decoders for AV1 is dav1d. The decoder for AV2 is dav2d. The decoder for H.264 is ffh264 inside FFmpeg. The decoder for HEVC is ffhevc inside FFmpeg. And there is a next generation codec from the MPEG world after H.264, H.265. There is one that is called H.266, also known as VVC. Lex Fridman (03:41:59): So HEVC is H.265. VVC is H.266. Why is H.266 super sexy- … and so much better? Jean-Baptiste Kempf (03:42:10): So the question often we have is why are there two names? Because most of the time it is a joint work from the ISO world and the ITU, which is the International Telecommunication Union. Lex Fridman (03:42:24): These are these two regulatory bodies. Kieran Kunhya (03:42:26): No, one is a private entity and one is the United Nations. Lex Fridman (03:42:28): Which one is the private? Kieran Kunhya (03:42:29): ISO is private. Jean-Baptiste Kempf (03:42:31): In theory, H.264 is MPEG-4 Part 10, H.264/AVC. And this is the full name. Lex Fridman (03:42:43): Nice. Kieran Kunhya (03:42:44): So it’s the concatenation of the ISO name and the ITU name- … even though they work together. This is, this is politics, historical, you know— … gotcha. Jean-Baptiste Kempf (03:42:53): And for HEVC, it’s MPEG-H, H.265, HEVC. Lex Fridman (03:42:58): Got it. Jean-Baptiste Kempf (03:42:59): And there is H.266, which is also named VVC. Lex Fridman (03:43:02): Is there a high-level thing to say about the improvement of- Jean-Baptiste Kempf (03:43:05): 30% each generation is the best summary. Lex Fridman (03:43:09): This is true both for the AV- … codecs and the- … H.264, 5, 6. Jean-Baptiste Kempf (03:43:15): So the professionals who are listening to us are going to kill us because they say, “No, it’s 35%, 25%— Kieran Kunhya (03:43:20): “No, it’s 50, 60.” Jean-Baptiste Kempf (03:43:21): … it’s 50,” blah, blah, blah. But globally, you need to know that HEVC is 30% better than H.264. H.266 is 30% better than H.265 because there are so many cases and so many scenarios. For example, there are cases, especially for screen recording, where the gains are humongous because you arrive, you have the right tool that is done for that. And so for a specific video, a new generation is going to give you 70% gain or 80% gain. Right? But there used to be a ton more codecs, but now the two main families for transmission are H.264, H.265, H.266, and the other is AV1, AV2. Lex Fridman (03:44:00): And I guess the major difference would be the cost of encoding. Jean-Baptiste Kempf (03:44:04): Yes, and the royalty of the patents. And this is the reasons why you see the AV version of codecs, is because they try to be as royalty-free—which means no cost for the patents—as much as possible. Because what you need to know, and we’ve not talked about that so far, is that multimedia is what we call a patent minefield. There are two places where you have the most patents. It’s everything related to 3G, 4G, 5G, RF, and multimedia. Because it’s very mathematical, and you can get great gains and so on. So Google and Meta and Netflix wanted something where it was royalty-free. There are people who said that they have patents outside, but they are fringe patents, right? So it’s mostly true that it’s patent-free. Kieran Kunhya (03:44:53): Oh, you should extend. Patent, patent checking was done as part of the standardization process in AV1, AV2, whereas patents are not even discussed in the MPEG world. Jean-Baptiste Kempf (03:45:06): The MPEG world. Kieran Kunhya (03:45:06): Patents are off-topic completely. Lex Fridman (03:45:08): Can you educate me on the patent side? Jean-Baptiste Kempf (03:45:10): So usually, MPEG does a format, right? And then everyone comes around and says, “Well, I have all those patents for the format,” and they do usually a union called what’s called MPEG LA, MPEG Licensing Association. And you put all your patents in, and then you ask everyone who’s using this format to pay for it. Lex Fridman (03:45:31): Wait, can you elaborate? What does it mean to have a patent for a codec? Why are there many patents? Jean-Baptiste Kempf (03:45:36): Imagine I’m doing something where I’m going to def— instead of doing blocks which are square, I’m going to do rectangles, right? Lex Fridman (03:45:44): Oh, so every idea- … somebody patents it. Oh, man. Oh, man. People and their… How many lawyers are- Kieran Kunhya (03:45:56): I mean, it pays for a lot of lawyers, right? Like- Jean-Baptiste Kempf (03:45:58): The biggest issue is not the following, right? Because at time of H.264, the patents were, let’s call it, like, sane. But there was so much money in that— … that for HEVC, a lot—there were a ton of things that were pushed inside the specification, which are not useful in 99.9% of the time, but so just one could add a patent on it. And so it became that for HEVC licensing, there was MPEG LA plus another patent pool called HEVC Advance. Plus- Kieran Kunhya (03:46:31): That one Jean-Baptiste Kempf (03:46:31): … I think Nokia was outside of the patent pool. Kieran Kunhya (03:46:34): Yeah, a few of them are outside, and some other one that’s- Jean-Baptiste Kempf (03:46:36): And so it was impossible to license, right? And I think that several months ago, HP decided that they were going to remove support from HEVC in their Windows laptops because the, the cost was increasing of those patents. And it arrived— Lex Fridman (03:46:52): Nice Jean-Baptiste Kempf (03:46:52): … where a point where—And there was uncapped pay. And so for YouTube or Netflix, we could talk about hundreds of millions of dollars of licensing for patents per year. And they said, “You know what? At hundred million per year, you know, I could create my own codec,” and this is what they did. And so that’s why we have the Alliance for Open Media, where we are part of, that created AV1 and creates AV2. We create also audio codecs. But yes. So the main difference would be that, and because you need to work around the patents or go do some things that are not patented, a lot of things are different, right? The basic things that were done in MPEG-2 thirty years ago are, of course, out of patents. But so for example, there is things like a golden frame, a S-frame, or, or different type of— Lex Fridman (03:47:46): These are all patented ideas. Kieran Kunhya (03:47:48): Yeah, no, it’s I can’t believe it’s not butter. I can’t believe it’s not a B-frame. It’s, I mean, I mean, it’s kind of what it is. In some ways, it’s like a- Lex Fridman (03:47:55): Oh, so it’s a different variant of a B-frame. Kieran Kunhya (03:47:57): Yeah, that’s to try and sidestep. Things like that. Jean-Baptiste Kempf (03:48:00): And so you need to have double creativity, right? Creativity in terms of being more efficient, but creativity of being sure that you don’t infringe existing patents. And so, for example, VVC is, has all the patents of HEVC plus new ones, right? It’s why AV2 tries to be as royalty-free as possible. Lex Fridman (03:48:20): To what degree does FFmpeg and VLC have to think about this kind of stuff? Jean-Baptiste Kempf (03:48:24): We don’t, and one of the reasons why VLC was in France is that France rejects software patents. So most of those patents are illegal in France because I once made the calculus that if I had to pay all the licensing fee for VLC, I needed to pay more than two hundred euros per user, right? It’s the same in dollars. But most of those patents are invalid in Europe because those are called—it’s basically mathematical patents or idea patents, and they are not valid in Europe. Lex Fridman (03:49:00): Let me just at a high level, just out of curiosity. So the meme online and the interwebs on X and Twitter and so on, and my own—I have friends in Europe—this, the sense is that Europe is not friendly to entrepreneurship. They over-regulate, there’s too much bureaucracy, and so on. Is, is there any- anything positive to say? Is there hope for entrepreneurship- … in the future of Europe? Is Europe over from a tech perspective? Kieran Kunhya (03:49:32): Just, just look at the two of us, right? It’s, it’s notable that there’s two people from the European continent on this podcast talking about video. It’s fair to say the community is weighted heavily. Jean-Baptiste Kempf (03:49:42): What you probably don’t see yet is that there is a new generation of entrepreneurs in Europe and mostly in France. UK has done it since a long time because, well, it’s more—it’s more Anglo-Saxon type of business. But especially like what happened in France, and of course, sometimes a bit overdone with everything called French Tech, but today, most of the people who come on the market want to create startups. Fifteen years ago, it wasn’t the case. Everyone wanted to work on big companies because when you failed in France, for example, twenty years ago, fifteen years ago, and you destroy your company, which is normal for a startup, right? You were not allowed to create a new company, right? There was a lot of stigma. The stigma is gone. Jean-Baptiste Kempf (03:50:32): … there is so many things happening on AI in France and so on, right? So there is, sure, over-regulations. I know that, right? I’m an entrepreneur. But it has some good things also. Lex Fridman (03:50:45): I mean, is there some paralyzing aspects? You know, if I look at, at the case of somebody I’ve become close with, Pavel Durov, you know, he was uh, blamed directly by the French government for the kind of things his, quote, “platform” was hosting. Uh, I could see the same kind of stuff basically, just as an example, VLC being blamed for the kind of videos that people are watching. Jean-Baptiste Kempf (03:51:13): But they tried, right? Like we had issues. Like we- Lex Fridman (03:51:19): I mean, is that, that’s the pressure that people worry about because if you have to think about that kind of stuff when you’re kind of just obsessed about- Jean-Baptiste Kempf (03:51:25): No, you don’t think about it- … and that’s, that’s okay, right? Like- Lex Fridman (03:51:29): But what if they come in? When, what if they show up and- Jean-Baptiste Kempf (03:51:31): There is no office. VideoLAN doesn’t have an office. Lex Fridman (03:51:33): I mean, this is what happened with Pavel. They arrested him, right? So arrested him for particular videos or, or a particular content that’s being shared on the platform. Jean-Baptiste Kempf (03:51:42): Sure. I don’t have any platform. Everything is on the client side. Lex Fridman (03:51:45): Yeah, but they’re, they can still arrest you. Jean-Baptiste Kempf (03:51:47): On what ground? I’m not sharing anything. I’m not– The content doesn’t go through my stuff. Lex Fridman (03:51:52): For sure, but it’s still lawyer fees. That’s the problem. Jean-Baptiste Kempf (03:51:55): Yes, that’s correct. Lex Fridman (03:51:55): It’s paperwork. So like, actually, if you had infinite trillions of dollars you would win easily because you’re on the right side. But the thing is, there is a degree to which they suffocate you with paperwork. That’s the downside of bureaucracy, through paperwork, through process. You know, it’s the Kafkaesque thing. Jean-Baptiste Kempf (03:52:17): You have to realize that one of the good things, for example in France or most of Europe, is that the—Answering to a court order does not make you bankrupt, right? It’s not like in the US, where it can actually bankrupt you, right? There is—The way the law system works is that, like I receive lawyers’ letters every week, right? And I can tell you that the cost of lawyer fees for VideoLAN is less than ten thousand dollars per year, right? Right? So that’s not really scary. Lex Fridman (03:52:53): I mean, similar with Pavel. The intelligence agencies tried to like say, “Can you put a backdoor in VLC?” Jean-Baptiste Kempf (03:52:59): Yes. Two of them. Lex Fridman (03:53:01): What, what do you say? Jean-Baptiste Kempf (03:53:02): No. Well, I was a lot less polite. Lex Fridman (03:53:05): I see… Yeah, yeah. You’re basically saying, “Hell no.” Jean-Baptiste Kempf (03:53:09): Like, if we had to compromise our software, we would shut it down. This is clear. Lex Fridman (03:53:13): And what’s the definition of compromise? Like allowing a government to do a backdoor- Jean-Baptiste Kempf (03:53:19): There is no code that gets into VLC that we don’t control, and the way we compile VLC, you would call me completely paranoid. Like, we compile on boxes that are offline, where we start by compiling the compiler. We do everything offline on places that have never been connected to the internet. We— The way we do signing, there is double signature. And especially because, for example, we’ve seen, and we believe it’s a governmental agency that is not from the Western world who tried to push a fake binary into our own servers and that scared us a lot. And VideoLAN is open source. How can you kill it? Like, I move to where? I move to Malta. Jean-Baptiste Kempf (03:54:06): I move to, I don’t know, Cayman Islands, and I change the domain name, and I start again, right? Like, VLC is a tool. It’s a tool that is going to help people doing things. We are not a platform. And for patents, well, I’m sorry, but most of the patents… Like, you shouldn’t be able to patent math and matrices. Like, this is wrong. Lex Fridman (03:54:32): So does VLC ever, like, censor the kind of videos it can play and not based on the content of the video? Jean-Baptiste Kempf (03:54:39): No, never. We never do that. Because VLC is completely offline. It doesn’t talk to any server, so we don’t know anything that you’re using the software for. Lex Fridman (03:54:49): So again, there’s no government that can say, you know, like the French government come in and say, “We don’t want… I think anime is destructive to society. We don’t want any anime; not allowed to be…” Jean-Baptiste Kempf (03:55:02): No, they cannot, they cannot do that. And also what they tried is to say, “Hey, I want to know if that person watched that type of video.” And the answer is like, “No idea.” Lex Fridman (03:55:11): So no on that too. So for surveillance, no. Jean-Baptiste Kempf (03:55:14): No, no, because the only infrastructure we have is a downloading infrastructure. There is no telemetry in VLC, right? Lex Fridman (03:55:20): It would be difficult ’cause of the international nature. It would be difficult for you to incorporate that code because there would be someone in the UK and someone in Germany and someone in the US as part of VideoLAN who’d be able to see that. It would be extremely difficult. Jean-Baptiste Kempf (03:55:33): The only thing that we can do, which happened, is like we had the issue— We had the case with some police in the US who said, “We have a murder case,” right? And the file is destructed or doesn’t play in that version of VLC. Could you help us?” Right? We never have access to the video. It’s like a normal support, right? Lex Fridman (03:55:50): Oh, it’s really about playing the file? Jean-Baptiste Kempf (03:55:51): Yes. And, like, I remember in the middle of the Afghan War, right? I received an email from someone in the army, right? I don’t remember the grades, right? It was just like, “We have a big issue with the latest version of VLC because it doesn’t play correctly the file on an RTSP server that we have where there is all the movies.” And he says VLC is very important for the morale on the troop on the ground, right? Because at night I think it might be boring, right? So they have a collection of videos to watch or movies over there, right? So and, and of course I did an update, and I broke some support of RTSP, right? So I gave them another version just for them, right? Because it was important. Jean-Baptiste Kempf (03:56:31): And because VLC is completely open source, I think it is allowed on the US Army laptops, right? Because the, I guess someone in the US military actually looked at it and say, “Well, okay, this is okay,” right? And the way we document how we process, that was okay, right? So the only way we work with authorities is to help them doing support. Lex Fridman (03:56:53): That’s amazing. That’s an amazing story. Yeah. Jean-Baptiste Kempf (03:56:56): We don’t see anything happening on how people use VLC, and this is strong. Lex Fridman (03:57:00): Do you feel the stress of this? So first of all, millions of people using it. Second of all, the military using it. Maybe sometimes pressure from governments. Does that… That’s a small team, right? Jean-Baptiste Kempf (03:57:16): Yeah, but- Lex Fridman (03:57:16): How big is VLC core… how many? Jean-Baptiste Kempf (03:57:21): Six, eight. But everything legally is only me. Everything that is legal is only me. Lex Fridman (03:57:28): You’re not stressed about this? Jean-Baptiste Kempf (03:57:30): I used to stress about that a lot. But the thing is, we’re doing what we can for everyone, for the greater good. We work so that we make some extremely complex technology easy for everyone. We’re a tool, and every tool is going to be used for great things and for bad things, right? You cannot blame a tool, I think. And this is, like, very important for us. I used to be in a lot of stress. I’m not anymore, right? Lex Fridman (03:58:00): What’s the secret to your zen? I mean, over and over in the chats I’ve had with you in the conversation today about every even tense topic, you’re very zen. What’s the source of zen? Jean-Baptiste Kempf (03:58:14): I have a way of thinking about what is the worst case scenario, always, right? And the answer is, at the end, if I take like a chess player, right? In the end, am I dead? Yes or no? Right? And I do that nonstop, right? Jean-Baptiste Kempf (03:58:33): And that’s also how I do my startups, right? Is that I’m here to get something right. What is the worst case? It goes bankrupt. That’s life. A company lives, a company dies. That’s okay, right? Like, and so my moral way is always like, am I dying in the end? Am I hurting someone? If the answer is no, then too bad, right? Like, oh, some lawyers are going to be unhappy. What are they going to do? Take all the money of VideoLAN? Wow. They’re going to have 50 grand. Amazing, right? What are they going to do with that? The code– the source code is out there. It’s not stoppable. Also because what we do is good and it’s done for everyone. Lex Fridman (03:59:14): That’s beautiful. Kieran, you said that there’s an active archiving preservation community? I think that’s super fascinating. You wrote that they’re stretched in budget, but they see the extreme importance of FFmpeg as a Rosetta Stone so that multimedia can be played a thousand years from now. I mean, that’s a beautiful way to see FFmpeg and VLC as a tool for preserving visual knowledge. Kieran Kunhya (03:59:41): Yes, that’s right. One of the coolest communities in open source multimedia, mainly led by someone called Dave Rice, I’ll give him a shout-out, I think from City University of New York, is the archiving community. They’ve done so much stuff. They value open source, one, because yes, they lack budgets, but two, they see the fact that archiving video is important for the world, and being able to play that is a big problem. Famously in the UK, there was something called the New Domesday Book, and they archived lots of stuff on BBC microcomputers. Within 10 to 15 years, no one had the right software to play that. Kieran Kunhya (04:00:19): I think it was 20 years or something like that, and someone had to go and reverse engineer this, and that was like 20 years. Imagine that in a thousand years. I think one of the great things about FFmpeg is it’s written in C. C is the closest to mathematics you’re probably gonna get. The closest to logic is- Jean-Baptiste Kempf (04:00:34): Do you think in 1,000 years we’d still have C compilers? Kieran Kunhya (04:00:37): Yes. We have languages that exist that haven’t changed too much. We have mathematical notation that exists. It will be like Latin. C will be like Latin. It will be a thing that you learn from the past, but it will still be usable in certain contexts. So the archiving community are really great practically. They, again, limited funds. They funded the development of the FFV1 codec, so that’s a lossless codec. So the archiving community is really scared about the act of compression losing things, and they have a fair point in this, you know. Kieran Kunhya (04:01:08): If they compress too hard, it could change the view of the material. There could be something slightly different here and there, so they’re really concerned about things. They need to be not just compressed well, but lossless and be fast. And so they worked with FFmpeg to develop a whole new codec designed for fast software-based encoding. They’re really concerned about resilience, so if they’re storing on tapes or other hard disks, I lose some bits, I need to recover quickly. I can’t lose a whole GOP because I’ve lost a bit- Kieran Kunhya (04:01:39): … something like that. So they’re a really great bunch of people. They funded GPU encoding in FFmpeg to make FFV1 encode faster. And it’s really about preserving the world’s multimedia heritage in a way that’s usable, and there’s a lot of great teams and a lot of archival groups across the world who’ve chosen FFmpeg and FFV1 as their archiving solution. And they can really provide us also super specialist advice. They can- … explain, “Ah, in the 1950s, colorimetry was done like this on this certain type of tape, and so there is this special case that you need to handle, and you’ll never get this anywhere else.” Jean-Baptiste Kempf (04:02:20): You see, they know things on video that we don’t. Like, every time I talk to, was it Dave- Kieran Kunhya (04:02:26): Dave Rice Jean-Baptiste Kempf (04:02:27): … or, or the people from the British, uh- Kieran Kunhya (04:02:29): British Film, uh Jean-Baptiste Kempf (04:02:30): … Film, it’s just like every time I just learn something new, and I’ve been doing video for 20 years. They have, especially on colorimetry and colors. Kieran Kunhya (04:02:39): Storage, these other things. Lex Fridman (04:02:40): I mean, they have a deep, deep appreciation of the content itself, of the video itself. And like, especially when you’re thinking of lossless, they’re terrified of losing something essential- … about the thing, and in so doing, they’re deeply understanding the thing that is to be preserved, which you sometimes might not be thinking about when you’re- Kieran Kunhya (04:03:00): Absolutely. Lex Fridman (04:03:00): … obsessing about the actual technology of the encoding and so on. Jean-Baptiste Kempf (04:03:03): And when you enter the rabbit hole of film scanners, right? So you take those- … those things to make to digital, like, it’s like- … a huge topic that, like, would take another five hours of podcast- … just on that topic. Kieran Kunhya (04:03:18): On film, and there’s a lot of film that needs to be archived. Film is degrading. It’s maybe not stored in the right environment. The other thing is they can… What they also do is, because it’s open source, they give this away, their workflows, to countries who can’t afford to have archiving institutions, where archiving is done by volunteers, it’s done by other things. They go and teach, you know, in India, they teach children to do FFmpeg commands. They’re really great. They’re really, they’re really the model community, the model ethos of what we’re trying to achieve. They are such a great bunch of people, so interested in participating and being part of something much bigger because they realize the work they’re doing in a thousand years is gonna tell a lot. Kieran Kunhya (04:03:59): You know, in a thousand years we may be drowning in AI slop. This stuff needs to be important and, you know, archived well. What was life like? Lex Fridman (04:04:07): Yeah, it feels like capturing the 20th century and the 21st century is essential because it feels like a transition point, where we went from scarcity of data to slop- … oceans of slop, and that transition point is good to archive. Kieran Kunhya (04:04:24): It’s important, yeah. Jean-Baptiste Kempf (04:04:25): But people don’t realize we are losing today a ton of films. There is a ton of things from the ’30s, from the ’40s, and the ’50s that where there is no value- Kieran Kunhya (04:04:37): And tape. ’70s and ’80s, there’s tape, and there’s not enough tape heads in the world- Jean-Baptiste Kempf (04:04:41): To read all the tapes. Kieran Kunhya (04:04:42): … left to redo, so they have to decide what they want to archive and throw away the rest of the tapes. There’s huge moral hazard, I guess for want of a better phrase, around this topic because this is a digital record of human history and they have to make decisions that… And there’s digital stewardship, I suppose, for want of… I made that phrase up. That’s not a real phrase. To make sure the world can have this information in something that’s playable by everybody, not- … playable on some device that, well, it doesn’t exist anymore. Lex Fridman (04:05:13): And then there’s like, realistically speaking, there’s a needle in a haystack where there’s a lot of value in archiving all that footage, and then over time finding the gems- … that we don’t know are there. Kieran Kunhya (04:05:25): Hey, there was something in that corner that we just didn’t- And that would’ve been compressed away because it was some little thing. Oh, wow, there’s something there. Lex Fridman (04:05:33): That’s it. Kieran Kunhya (04:05:33): And that’s… They’ve made sure that it’s lossless. They can prove mathematically that it’s lossless. They can run different trade-offs for if there’s bit… if they lose a bit, a single bit flips, I can make sure that I only lose a portion of a given frame. We can do error recovery on previous frames. They can do all sorts of different things. Lex Fridman (04:05:52): Do you think VLC and FFmpeg will be here 100 years from now? Jean-Baptiste Kempf (04:05:57): FFmpeg, yes. Kieran Kunhya (04:05:58): Yep, FFmpeg, yes. Jean-Baptiste Kempf (04:05:59): VLC, maybe. Lex Fridman (04:06:01): What’s the future of… Where is FFmpeg going? Where is VLC going? Like in the next… If you think about, like, five years, 10 years, 20 years. Jean-Baptiste Kempf (04:06:10): Five years, 10 years is easy. The question is after that, right? The question is- … do we arrive at something called holograms, right? Lex Fridman (04:06:19): Yeah, so will VLC and FFmpeg expand- … to whatever- Jean-Baptiste Kempf (04:06:26): Multimedia Lex Fridman (04:06:27): … multimedia- … so multimedia might become, I’m sorry for the pothead expansion of topic, but, you know, if you look at something like Neuralink with brain-computer interfaces, it’s very possible that we start to consume what multimedia means is whatever codec, whatever data that our brain wants to consume through the brain-computer interfaces. That’s one. Then virtual reality, of course. Jean-Baptiste Kempf (04:06:52): You will have VLC for Neuralink. Kieran Kunhya (04:06:55): Yep, and you’ll have FFmpeg -i input format human brain. Lex Fridman (04:06:58): Yeah. There’s gonna be codecs for the brain. Kieran Kunhya (04:07:01): Sure, 100%. Yeah, to compress neural information, yeah. Jean-Baptiste Kempf (04:07:05): I mean, today there is like, there are new codecs for- … for example, what we call point cloud, right? Or volumetric videos, right? There is a ton of research on what we call RGB-D, right? So codecs for depth that is useful for robotics and for 3D things. There is a ton of codecs for compression of 3D elements. Kieran Kunhya (04:07:22): Compression for astronomy. Jean-Baptiste Kempf (04:07:23): For example, on VLC, we also have already a VR and XR version of VLC. And also on Kyber, right? We talk about Kyber. On Kyber, we also like do streaming of XR content on, for the glasses who cannot have enough power or inside the Apple Vision or the Quest. So we already work on streaming 3D, XR, interactive, low latency. There is something called volumetric video, point cloud videos, so it’s not stopping. And yes, at some point it will manage 3D data inside VLC and FFmpeg, right? It’s obvious. Lex Fridman (04:07:58): So that’s where it is moving, like the community is open. Jean-Baptiste Kempf (04:08:01): Not everyone in the community sees that, but as Kieran and I, we are entrepreneurs, we know where it’s going. We see that, right? Lex Fridman (04:08:10): So I suppose that there is a tension probably inside FFmpeg. It’s like, “Hey, listen, folks, we’re really good at doing video and audio, so like why expand? Like let’s do the thing we’re really good at doing.” Jean-Baptiste Kempf (04:08:25): In order to answer that question, we need to answer the definition of what is multimedia. And multimedia is a digital representation of several streams for the human senses. And we will do that, right? So imagine there is now a way to not have a mic, but have a odor sensor- … and a diffuser of odors. It will get into FFmpeg. Lex Fridman (04:08:54): So your demuxer is coming up. Jean-Baptiste Kempf (04:08:56): Yes. Yes. Of course, your demuxer has a new track type that is basically odors, right? And you already have- Lex Fridman (04:09:02): Smell, touch. Kieran Kunhya (04:09:03): It’s like audio. You’ll have a left and right nose track. You have a left and right audio pair. It’s easy. Jean-Baptiste Kempf (04:09:07): Yes, of course. Lex Fridman (04:09:09): Stereo smell. Kieran Kunhya (04:09:09): Stereo smell, yeah. Jean-Baptiste Kempf (04:09:10): So in VLC, for example, we already have a plugin for haptic. It’s mostly for what we call 4D cinemas, right? You know, those ones on hydraulic—I don’t know how you say that—all the hydraulic- Kieran Kunhya (04:09:21): Hydraulic arms. Hydraulic, Jean-Baptiste Kempf (04:09:22): Arms. And where everything is moving, like you have in theme parks, right? And there is a data feed synchronized where, which is basically transporting this information. Lex Fridman (04:09:32): Is there yet a standard for that? Jean-Baptiste Kempf (04:09:33): There are many standards, right? Lex Fridman (04:09:35): This is… You make me so happy. Jean-Baptiste Kempf (04:09:38): And so of course, like we have a plugin which is not in the normal version of VLC- Lex Fridman (04:09:43): That’s good. Jean-Baptiste Kempf (04:09:43): … that is basically transporting those type of movements, which is physical movements, which is haptic movements, right? It is a human sense, so it will get in. Lex Fridman (04:09:54): That’s such an exciting future. Was it… I mean, it’s a small community of developers. How do you pull that off? Like if you’re a contributor to FFmpeg or VLC, it feels stressful. Like it, just looking on Twitter, it’s like a huge amount of work to make it work on all these different operating systems, an incredible effort. Jean-Baptiste Kempf (04:10:17): No, see it in the other direction. We are not the contributors. We are the maintainers, right? So we maintain for everyone. Meaning that, for example, every year there is around 150 people who contribute to VLC and maybe 300 on FFmpeg, right? Our goal as a small team is to get all the contribution in. So if there is more usage, there will be more contributions, and those people will do the right module, the new format, and so on. We care about the architecture of VLC, the architecture of FFmpeg, right? Now we’re doing things in VLC, which is spatial audio, right? We did the demo not long ago. There was changes needed on the architecture, and we did the first spatial audio module. Jean-Baptiste Kempf (04:11:07): When it’s going to add the second one, it’s going to be easy, or the third one is going to be easy, right? Our goal, and it’s going to be the same for others or haptic, right? We need to work the architecture so that modules can be added to add future capabilities. So yes, we are going… We are a multimedia framework, so that’s not just audio and video. It’s everything that is timed and represents something that you can sense. And if it’s brainwaves, it’s going to be brainwaves. Kieran Kunhya (04:11:36): I think that’s inevitable. Sorry. Lex Fridman (04:11:38): I love this on so many fronts because, so FFmpeg and VLC are pushing companies and pushing the world to standardize. So for example, to standardize- … brainwaves, right? So standardize… It would push, like I hope Neuralink comes up with a standard for multimedia via brain-computer interfaces or for robots with haptic. Jean-Baptiste Kempf (04:12:05): By experience, what happens is always the same, right? You start, it’s a new topic. There is like five different standards because everyone starts to do this. The hype goes down because every time the hype goes down, then people start to say, “Well, you know what? We need to do a standard.” People, because two or three companies, usually not the leader, but the two or three followers do a standard, and then we implement the standard and then it’s the end of the curve. It starts to be more pepper. Lex Fridman (04:12:32): And then the leader’s kind of pressured into it because it is better to do a standard. Yeah. Jean-Baptiste Kempf (04:12:36): Example, 3D audio, right? Six or seven years ago, it was everything about 3D. You go, you had the cardboard on Android. You had two audio formats. They’re all dead, right? And now it’s coming back with actual use cases, and we learn from the mistakes of the past standard. So it will be the same everywhere. Lex Fridman (04:12:56): And not try to avoid closed? I saw somewhere you didn’t have too many nice things to say about Dolby. Jean-Baptiste Kempf (04:13:03): No, I don’t. Lex Fridman (04:13:04): What is… can you educate me on why, where they went, what did they do bad that made you mad? Jean-Baptiste Kempf (04:13:13): It’s used to be an amazing company doing tons of great things with amazing engineers. They defined what sound was. And now it’s mostly- Kieran Kunhya (04:13:25): Lawyers. Jean-Baptiste Kempf (04:13:25): … lawyers and licensing things. Lex Fridman (04:13:27): Oh, so they’re, yeah, it’s, they’re, they’re closing stuff off. They’re trying to make money on licensing. Jean-Baptiste Kempf (04:13:30): No, it’s just like they, they don’t innovate as much as they did- … and so on. It’s a bit like I’m sorry to say, right, like HP, right? Kieran Kunhya (04:13:38): Very true. Lex Fridman (04:13:40): Oh, since we talked about Twitter a bunch in a bunch of different contexts, do you have a favorite, and least favorite, most embarrassing tweet on either VideoLAN or FFmpeg Twitters? Kieran Kunhya (04:13:53): The two, my two favorites are, “Talk is cheap, send patches.” I think that, that- … embodies a lot of the stuff doesn’t get, as, as we’ve talked about, stuff doesn’t get built unless someone does it. It doesn’t just appear from the ether. The other one that I like is “FFmpeg, nothing is beyond our reach.” I think that comes from a US military satellite patch where I think they, they invented some kind of monitoring system that could see the whole world, and this was released. Jean-Baptiste Kempf (04:14:19): Wasn’t there something where FFmpeg was running on a rover on Mars also? Kieran Kunhya (04:14:22): Yeah, so FFmpeg is used by the Mars rover, the Mars 2020 rover to compress pictures. They really wanted—they wrote a paper about it, and they really wanted to use as much commercial off-the-shelf technology as possible. Lex Fridman (04:14:34): Oh, that’s cool. Kieran Kunhya (04:14:35): FFmpeg runs on Mars, so we are, we are a multi-planetary open source library. Jean-Baptiste Kempf (04:14:41): Very often we’ve seen- … Tweets for people using VLC in weird places. A lot of the people doing Formula 1 are in all the paddocks, they use VLC to play the live feed. We’ve seen the European Space Agency. We’ve seen SpaceX, like, monitoring the launches with VLC, and it fills you with joy, right? So- Lex Fridman (04:15:05): I’ve seen a particle accelerator. Jean-Baptiste Kempf (04:15:07): Oh, yeah, yeah. We had one of the most amazing thing that I went for was to go to the CERN at the LHC because they were using VLC to monitor all the captors on the ring because the ring is 27 kilometers. And so they had some analog cameras- … and they were using some of the capture cards to go to analog to VLC, so VLC could stream on their multicast network for the whole CERN to access to that. And, like, I visited that in 2010 with Laurent and, like, we fixed their issue in an hour or something like that, right? Because it was some parameters maybe not well documented at that time. And he said, “Okay, for the whole day, what do you want to do?” And we visited everything. Like… things where with antimatter and colliders and so on. Lex Fridman (04:16:00): That’s awesome. Jean-Baptiste Kempf (04:16:00): And that was, like, one of the most amazing days of my physics background. Lex Fridman (04:16:06): Yeah, it’s used, like, everywhere. Any tweets, Kieran, you regret? Kieran Kunhya (04:16:12): Tweets I regret? Lex Fridman (04:16:14): Or is it like that, how does the French song go? Regret nothing. Kieran Kunhya (04:16:17): “Je ne regrette rien.” Yeah. Jean-Baptiste Kempf (04:16:19): That’s very important for me, right? Don’t regret anything. No, it’s because regrets are a tax on your mind, right? So learn from your mistakes, but don’t regret. Because you’ve done it, so except if you have a, a time machine to go back in time, don’t regret, right? It’s going to just tax your brain. Learn from your mistake, sure. Don’t regret. Lex Fridman (04:16:43): It’s like it reminds me, it’s beautiful. It’s a tax on your brain. It reminds me of the Johnny Depp quote I saw where he was saying, “Hate, you know, I don’t hate. That’s, hate is a very expensive emotion.” Jean-Baptiste Kempf (04:16:56): Are you comparing me to Johnny Depp? Lex Fridman (04:16:58): Uh- Jean-Baptiste Kempf (04:16:58): Because that would be your first one. Lex Fridman (04:17:00): Well, gentlemen, like I said, I’m eternally grateful for the software that, you know, the two of you and the bigger community have been part of building with FFmpeg and VLC and everything else. I’m eternally grateful for the spicy tweets. Never stop. And I’m grateful that you would talk with me today and give me this sexy hat. I feel like a wizard. I feel special. And I feel special to get a chance to talk and celebrate the piece of software that brought me so much joy over the years. So thank you for everything, and thank you for talking today. Kieran Kunhya (04:17:36): Thank you for having us. Jean-Baptiste Kempf (04:17:37): Thank you so much. Lex Fridman (04:17:38): Thanks for listening to this conversation with Jean-Baptiste Kempf and Kieran Kunhya. To support this podcast, please check out our sponsors in the description where you can also find links to contact me, ask questions, give feedback, and so on. And now let me leave you with some words from the legendary Linus Torvalds. “Most good programmers do programming not because they expect to get paid or get adulation by the public, but because it is fun to program.” Thank you for listening, and I hope to see you next time.

本集资源与出处