System.Xml.XmlElement
April 2021’s Gwern.net newsletter is now out; previous, March 2021 ( archives ).
2021 年 4 月的 Gwern.net 时事通讯现已发布；上一篇，2021 年 3 月（档案）。
This is a collation of links and summary of major changes, overlapping with my Changelog ; brought to you by my donors on Patreon .
这是主要更改的链接和摘要的整理，与我的更改日志重叠；由我的 Patreon 捐助者带给您。
“Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks” , Lee et al 2018; “Perceiver: General Perception with Iterative Attention” , Jaegle et al 2021 (skinny Transformers applied recurrently; given reinvention, one might ask “is attention , getting too much attention?”, especially given how many Transformer tweaks don’t pan out or have antecedents, indicating a gold rush?
“Set Transformer：基于注意力的排列不变神经网络的框架”，Lee 等人，2018 年； “Perceiver: General Perception with Iterative Attention”，Jaegle et al 2021（瘦身的 Transformer 经常应用；考虑到重新发明，人们可能会问“注意力，得到了太多的关注吗？”，特别是考虑到有多少 Transformer 调整没有成功或有先例，表明淘金热？
Probably not: if the marginal return on this research direction had fallen below that of competitors, we would see those neglected directions invade Transformer topics—while we continue to see the reverse, and many applications as yet untouched by all the new approaches, suggesting that we still don’t pay enough attention)
可能不会：如果这个研究方向的边际回报低于竞争对手，我们会看到那些被忽视的方向侵入 Transformer 主题，而我们继续看到相反的情况，许多应用程序尚未受到所有新方法的影响，这表明我们仍然没有给予足够的重视）
“Z-IL: Predictive Coding Can Do Exact Backpropagation on Any Neural Network” , Salvatori et al 2021 (scaling local learning rules to ImageNet AlexNet/Resnet & ALE DRL at similar compute cost)
“Z-IL：预测编码可以在任何神经网络上进行精确的反向传播”，Salvatori 等人 2021（以相似的计算成本将本地学习规则扩展到 ImageNet AlexNet/Resnet 和 ALE DRL）
“Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates” , Smith & Topin 2017 (the lingering mystery of super-convergence, saving 50–90% compute with LRs as high as 20 (!): what is it, why does it work only sometimes, is there any connection to grokking & can it work for large models like GPT-3 given the tunneling hypothesis ?)
“超融合：使用大学习率对神经网络进行快速训练”，Smith & Topin 2017（超融合的挥之不去的谜团，在 LR 高达 20 (!) 的情况下节省 50-90% 的计算量：它是什么，为什么它只是有时起作用，与 grokking 有任何联系吗？在给定隧道假设的情况下，它可以适用于像 GPT-3 这样的大型模型吗？）
“Rip van Winkle’s Razor, a Simple New Estimate for Adaptive Data Analysis” (an unusual approach to estimating generalization—by quantifying the information-theoretic simplicity of all the powerful DL research discoveries since 2012, into ~1 kilobyte.
“Rip van Winkle 的 Razor，自适应数据分析的简单新估计”（一种不寻常的估计泛化方法——通过将 2012 年以来所有强大的 DL 研究发现的信息论简单性量化为约 1 KB。
And yet, what a kilobyte…)
然而，多大的千字节……）
“Ambigrammatic Figures” , Levin & Huang 2020 (making horrifying StyleGAN faces that can be rotated 180° by projection & then gradient-ascent towards an upside-down face)
“Ambigrammatic Figures”，Levin & Huang 2020（制作可怕的 StyleGAN 面孔，可以通过投影旋转 180°，然后梯度上升到颠倒的面孔）
Congratulations to OpenAI on 1 year of GPT-3 & OA API.
祝贺 OpenAI 推出 GPT-3 和 OA API 一周年。
Has it really only been a year?—it has truly exceeded expectations.
真的才一年吗？确实超出了预期。
Naver announces 204b-parameter Korean-language NN, “HyperCLOVA” (KO; unknown arch although apparently dense, or training-compute or benchmark/loss performance; 650b token training dataset.
Naver 宣布 204b 参数韩语神经网络“HyperCLOVA”（KO；未知架构，尽管表面上很密集，或训练计算或基准/损失性能；650b 令牌训练数据集。
Who knew Naver was even trying?
谁知道 Naver 还在尝试呢？
“And we are here as on a darkling plain / Swept with confused alarms of struggle and flight, / Where ignorant armies clash by night.”)
“我们在这里就像在一片黑暗的平原上/被挣扎和逃跑的混乱警报所席卷，/无知的军队在夜间发生冲突。”）
“PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation” , Zeng et al 2021 (Zh; Huawei’s GPT-3-200b prototype, trained on indigenous Chinese GPU+DL stack; a partial replication, due to incomplete training on ~43b tokens; the 13b-parameter model checkpoint has been released for download, and they are considering releasing the 200b-parameter model…
“PanGu-α：具有自动并行计算的大规模自回归预训练中文模型”，Zeng 等人 2021（Zh；华为的 GPT-3-200b 原型，在中国本土 GPU+DL 堆栈上进行训练；部分复制，由于对约 43b 令牌的训练不完整；13b 参数模型检查点已发布供下载，他们正在考虑发布 200b 参数型号…
Ding commentary )
丁评）
New ?(100b)-parameter Transformer models announced at Google I/O ’2021: LaMDA (EN; chatbot), MUM (multimodal multilingual search/translation/Q&A)
Google I/O '2021 上宣布的新 ?(100b) 参数 Transformer 模型：LaMDA（EN；聊天机器人）、MUM（多模式多语言搜索/翻译/问答）
“PLUG” (Zh): a 27b parameter BERT-like Chinese language model, targeting 200b next (AliBaba followup to StructBERT / PALM )
“PLUG”（Zh）：一个 27b 参数的类似 BERT 的中文语言模型，下一步目标是 200b（阿里巴巴对 StructBERT / PALM 的后续）
“CogView: Mastering Text-to-Image Generation via Transformers” , Ding et al 2021 (another Chinese DALL·E clone, post- M6 : n = 30m text-image pairs , 4b-parameter GPT, models to be released)
“CogView：通过 Transformers 掌握文本到图像生成”，Ding 等人 2021（另一个中国 DALL·E 克隆，M6 后：n = 30m 文本图像对，4b 参数 GPT，模型待发布）
“VideoGPT: Video Generation using VQ-VAE and Transformers” , Yan et al 2021; “GODIVA: G enerating O pen- D oma I n V ideos from n A tural Descriptions” , Wu et al 2021 (DALL·E for video on Howto100M: VQ-VAE + sparse attention)
“VideoGPT：使用 VQ-VAE 和 Transformers 生成视频”，Yan 等人，2021 年； “GODIVA：根据真实描述生成视频中的开放域”，Wu 等人 2021（Howto100M 上的视频的 DALL·E：VQ-VAE + 稀疏注意力）
“Efficient Large-Scale Language Model Training on GPU Clusters” , Narayanan et al 2021 (Nvidia ‘Megatron-LM’ software for scaling up to 3072 A100 GPUs; allows 1t-parameter models at 502 petaFLOP/s or 50% efficiency, cf TPU rival, GSPMD , and note Patterson et al 2021 estimates GPT-3 at ~3.5m V100 GPU-hours, so OA got ~20% efficiency?); “We expect to see multi-trillion-parameter models by next year, and 100 trillion+ parameter models by 2023” —Nvidia CEO Jensen Huang ( subtitles )
“GPU 集群上的高效大规模语言模型训练”，Narayanan 等人 2021（Nvidia ‘Megatron-LM’ 软件可扩展到 3072 个 A100 GPU；允许 1t 参数模型达到 502 petaFLOP/s 或 50% 的效率，参见 TPU 竞争对手 GSPMD，并注意 Patterson 等人 2021 年估计 GPT-3 约为 3.5m V100 GPU 小时，因此 OA 的效率约为 20%？）； “我们预计到明年将看到数万亿个参数模型，到 2023 年将看到超过 100 万亿个参数模型”——Nvidia 首席执行官黄仁勋（字幕）
Mixture-Of-Experts: BAAI’s “Wudao Wensu”: 1.75-trillion parameters & multimodal!
专家云集：BAAI 的“五道问速”：1.75 万亿参数和多模式！
( prologue )
（序言）
“Exploring Sparse Expert Models and Beyond” , Yang et al 2021 (1t-parameter hierarchical Switch Transformer trained on 480 V100 GPUs)
“探索稀疏专家模型及其他”，Yang 等人 2021（在 480 V100 GPU 上训练的 1t 参数分层 Switch Transformer）
“MuZero Unplugged: Online and Offline Reinforcement Learning by Planning with a Learned Model” , Schrittwieser et al 2021 (Reanalyze+MuZero; smooth log-scaling of Ms. Pacman reward with sample size, 107–1010, showing that DRL for arcade games parallels board games)
“MuZero Unplugged：通过学习模型规划进行在线和离线强化学习”，Schrittwieser 等人 2021（Reanalyze+MuZero；样本大小为 107-1010 的 Ms. Pacman 奖励的平滑对数缩放，表明街机游戏的 DRL 与棋盘游戏相似）
“Decision Transformer: Reinforcement Learning via Sequence Modeling” , Chen et al 2021
“决策变压器：通过序列建模强化学习”，Chen 等人 2021
“Sampled MuZero: Learning and Planning in Complex Action Spaces” , Hubert et al 2021 (MuZero for continuous domains: DM Control Suite/Real-World RL Suite); “Continuous Control for Searching and Planning with a Learned Model” , Yang et al 2020
“Sampled MuZero: Learning and Planning in Complex Action Spaces”，Hubert et al 2021（连续域的 MuZero：DM Control Suite/Real-World RL Suite）； “使用学习模型进行搜索和规划的持续控制”，Yang 等人，2020
“Muesli: Combining Improvements in Policy Optimization” , Hessel et al 2020 (catching up with original MuZero)
“Muesli：结合策略优化的改进”，Hessel 等人 2020（赶上原来的 MuZero）
“Visualizing MuZero Models” , de Vries et al 2021 (reimplementing & introspecting a MuZero)
“可视化 MuZero 模型”，de Vries 等人 2021（重新实现和反思 MuZero）
“Scaling Scaling Laws with Board Games” , Jones 2021 (AlphaZero/ Hex : highly-optimized GPU implementation enables showing smooth scaling across 6 OOM of compute—2× FLOPS = 66% victory; amortization of training → runtime tree-search, where 10× training = 15× runtime)
“利用棋盘游戏扩展扩展法则”，Jones 2021（AlphaZero/ Hex：高度优化的 GPU 实现能够在 6 OOM 计算范围内显示平滑扩展 — 2× FLOPS = 66% 胜利；训练摊销 → 运行时树搜索，其中 10× 训练 = 15× 运行时）
“Scaling Laws for Language Transfer Learning” , Christina Kim ( Hernandez et al 2021 followup: smooth scaling for En → De/Es/Zh)
“语言迁移学习的缩放法则”，Christina Kim（Hernandez 等人 2021 年后续文章：En → De/Es/Zh 的平滑缩放）
“Carbon Emissions and Large Neural Network Training” , Patterson et al 2021 (“…choice of DNN/datacenter/processor can reduce the carbon footprint up to ~100–1000×.
“碳排放和大型神经网络训练”，Patterson 等人 2021（“……选择 DNN/数据中心/处理器可以将碳足迹减少约 100–1000 倍。
These large factors make retroactive estimates difficult.”)
这些重大因素使得追溯估计变得困难。”）
“How to Train BERT with an Academic Budget” , Izsak et al 2021 ( BERT in 8 GPU-days—R&D iteration allows finding efficiency; there’s nothing so expensive as demanding research be cheap.^1^)
“如何用学术预算训练 BERT”，Izsak 等人 2021（8 个 GPU 天的 BERT - 研发迭代可以提高效率；没有什么比要求研究更便宜更昂贵的了。^1^）
Everything Is Heritable: “Precision exercise medicine: understanding exercise response variability” , Ross et al 2019 (“large individual differences in CRF response (range: −33% to +118%) have been observed across the 8 exercise training studies independent of exercise duration”—nothing in psychology, or medicine, makes sense except in the light of individual differences…)
一切都是可遗传的：“精准运动医学：了解运动反应变异性”，Ross 等人，2019 年（“在 8 项运动训练研究中，观察到 CRF 反应存在巨大的个体差异（范围：-33% 至 +118%），与运动持续时间无关”——除了个体差异之外，心理学或医学中没有任何意义……）
Recent Evolution: “Analysis of genomic DNA from medieval plague victims suggests long-term effect of Yersinia pestis on human immunity genes” , Immel et al 2021
最近的演变：“对中世纪鼠疫受害者基因组 DNA 的分析表明鼠疫耶尔森氏菌对人类免疫基因的长期影响”，Immel 等人 2021
Engineering: “China officially bans CRISPR babies, human clones and animal-human hybrids” ?
工程：“中国正式禁止 CRISPR 婴儿、克隆人以及动物与人类的杂交”？
(another blow to attempts to project fears & fantasies onto China)
（对试图将恐惧和幻想投射到中国的又一次打击）
Reflecting Sunlight: Recommendations for Solar Geoengineering Research and Research Governance , National Academies 2021 ( media )
反射阳光：太阳能地球工程研究和研究治理的建议，国家科学院 2021（媒体）
“Improving Public Sector Management at Scale?
“大规模改善公共部门管理？
Experimental Evidence on School Governance India” , Muralidharan & Singh 2020
印度学校治理的实验证据”，Muralidharan 和 Singh 2020
“Jay-Z’s 99 Problems , Verse 2: A Close Reading with 4th Amendment Guidance for Cops and Perps” , Mason 2012
“Jay-Z 的 99 个问题，第 2 节：仔细阅读第四修正案对警察和罪犯的指导”，梅森，2012 年
“Oxylipin biosynthesis reinforces cellular senescence and allows detection of senolysis” , Wiley et al 2021
“Oxylipin 生物合成增强细胞衰老并允许检测衰老作用”，Wiley 等人 2021
“If I fits I sits: A citizen science investigation into illusory contour susceptibility in domestic cats ( Felis silvestris catus )” , Smith et al 2021
“如果我适合，我就坐着：对家猫 (Felis silvestris catus) 虚幻轮廓易感性的公民科学调查”，Smith 等人，2021 年
“Cetaceans, sex and sea serpents: an analysis of the Egede accounts of a ‘most dreadful monster’ seen off the coast of Greenland in 1734” , Paxton et al 2005 (is that a legendary cryptid in your pocket, or are you just happy to see me?)
“鲸类、性和海蛇：对 1734 年在格陵兰岛海岸看到的‘最可怕的怪物’的 Egede 描述的分析”，Paxton 等人 2005（这是你口袋里的传说中的神秘生物，还是你只是很高兴见到我？）
“Building the perfect curse word: A psycholinguistic investigation of the form and meaning of taboo words” , Reilly et al 2020
“构建完美的诅咒词：禁忌词的形式和含义的心理语言学调查”，Reilly 等人 2020
“How Developers Choose Names” , Feitelson et al 2021 (“Another example concerned the function ‘arrangeFilesByName(files)’.
“开发人员如何选择名称”，Feitelson 等人 2021（“另一个例子涉及函数‘arrangeFilesByName(files)’。
When asked the return value…one suggested the number of files reordered”)
当询问返回值时......有人建议重新排序的文件数量”）
“Bringing GNU Emacs to Native Code” , Corallo et al 2020 (using libgccjit to make Emacs 2.3× to 42× faster; gccemacs has been merged into Emacs HEAD & will be available soon)
“Bringing GNU Emacs to Native Code”，Corallo 等人 2020（使用 libgccjit 使 Emacs 速度提高 2.3 倍到 42 倍；gccemacs 已合并到 Emacs HEAD 中并且即将推出）
“Hosting SQLite databases on Github Pages (or any static file hoster)” (a revolution in static website technology: eg running a query need download only 54kb of a 670MB database ; fulltext site search is just the beginning of the possibilities of this clever use of range requests )
“在 Github Pages（或任何静态文件托管服务商）上托管 SQLite 数据库”（静态网站技术的一场革命：例如，运行查询只需下载 670MB 数据库的 54kb；全文站点搜索只是巧妙使用范围请求的可能性的开始）
“ Fontemon : World’s first video game in a font!” (a Pokemon -like CYOA implemented as an OpenType font file ; play in browser or text editor—still not quite Turing-complete but definitely the most impressive thing implemented in a font so far)
“Fontemon：世界上第一个使用字体的视频游戏！” （像 Pokemon 一样的 CYOA 作为 OpenType 字体文件实现；在浏览器或文本编辑器中播放 - 仍然不完全图灵完备，但绝对是迄今为止在字体中实现的最令人印象深刻的东西）
Fontemon is by far the highlight of SIGBOVIK 2021 ; but also worth noting: “Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search” · “Deep Deterministic Policy Gradient Boosted Decision Trees” · “Lowestcase and uppestcase letters: Advances in derp learning” · “openCHEAT: Computationally Helped Error bar Approximation Tool—Kick-starting Science 4.0” · “The Newcomb-Benford Law, Applied to Binary Data: An Empirical and Theoretic Analysis” · “Inverted Code Theory: Manipulating Program Entropy” ( Tenet fans only—possibly inferior to Moravec 1991 ?) · “Build your own 8-bit busy beaver on a breadboard!”
Fontemon 是 SIGBOVIK 2021 迄今为止的亮点；但也值得注意的是：“回到原点：通过深度神经网络和树搜索在滑道和梯子中实现超人性能”·“深度确定性策略梯度提升决策树”·“最小字母和大写字母：derp学习的进展”·“openCHEAT：计算帮助误差条逼近工具——启动科学4.0”·“纽科姆-本福德定律，应用于二进制数据：实证和理论分析》·“倒排码理论：操纵程序熵”（仅限《Tenet》粉丝——可能不如 Moravec 1991？）·“在面包板上构建你自己的 8 位忙碌海狸！”
Incidentally, it’s curious that while STEM fields have entire annual issues, journals, & conferences devoted to satire ( SIGBOVIK ; Arxiv April Fools papers like Garfinkel et al 2017 ; Special Topics ; the BMJ Christmas issue ; the Ig Nobel Prizes & BAHFest ), after asking in several places, I have found no instances in the humanities.
顺便说一句，奇怪的是，虽然 STEM 领域有专门用于讽刺的年度期刊、期刊和会议（SIGBOVIK；Arxiv 愚人节论文，如 Garfinkel 等人 2017；专题；BMJ 圣诞特刊；搞笑诺贝尔奖和 BAHFest），但在询问了几个地方后，我在人文学科中没有找到这样的例子。
(I know of many entertaining papers , like Sinhababu 2008 on waifus, but no regular organized publication, with the possible exception of the annual “Latke-Hamantash Debate” .)
（我知道有很多有趣的论文，比如 waifus 上的 Sinhababu 2008，但没有定期组织的出版物，年度“Latke-Hamantash 辩论”可能是个例外。）
“The Kelly Criterion in Blackjack Sports Betting, and the Stock Market” , Thorp 2006
“二十一点体育博彩中的凯利准则和股票市场”，Thorp 2006
“The Ocean’s Hot Dog: The Development of the Fish Stick” , Kelly 2008 (out of nostalgia, I bought some fish sticks for the first time in decades; better than I remembered, even if I had no tartar handy)
《海洋热狗：鱼棒的发展》，Kelly 2008（出于怀旧，我几十年来第一次买了一些鱼棒；比我记忆中的要好，即使我手边没有塔塔）
“The Aesthetics of Smelly Art” , Shiner & Kriskovets 2007; “The Odor Value Concept in the Formal Analysis of Olfactory Art” , Kraft 2019; “Perfumery as an art form” / notes , Qualia Computing 2020 (more: manufacturing: “The Scent of the Nile: Jean-Claude Ellena creates a new perfume” ; human smell is better than you think: “Mechanisms of Scent-tracking in Humans” , Porter et al 2006 ( video ; see also “Poor Human Olfaction is a 19th Century Myth” , McGann 2017); olfactory white ; Kōdō , which unexpectedly appears in Knuth .
“臭艺术的美学”，Shiner & Kriskovets 2007； “嗅觉艺术形式分析中的气味价值概念”，卡夫2019； “香水作为一种艺术形式”/注释，Qualia 计算 2020（更多：制造：“尼罗河的气味：Jean-Claude Ellena 创造了一种新香水”；人类的气味比你想象的更好：“人类气味追踪的机制”，Porter 等人 2006 年（视频；另请参阅“人类嗅觉不佳是 19 世纪的神话”，McGann） 2017）；嗅觉白；Kōdō，意外地出现在 Knuth 中。
C.
C.
Thi Nguyen ’s description of the more bizarre & avant-garde perfumes made me curious enough to nose around & order 39 LuckyScent samplers.)
Thi Nguyen 对更奇异和前卫的香水的描述让我好奇地四处寻找并订购了 39 支 LuckyScent 样品。）