System.Xml.XmlElement
January 2021’s Gwern.net newsletter is now out; previous, December 2020 ( archives ).
2021 年 1 月的 Gwern.net 时事通讯现已发布；上一篇，2020 年 12 月（档案）。
This is a summary of the revision-history RSS feed, overlapping with my Changelog & /r/gwern; brought to you by my donors on Patreon .
这是修订历史 RSS 提要的摘要，与我的变更日志和 /r/gwern 重叠；由我的 Patreon 捐助者带给您。
“Danbooru2020: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset”
“Danbooru2020：大规模众包和标记的动漫插图数据集”
Gwern.net : +return-to-top floating button; popups : can now be disabled (use the ‘gear’ icon); final reimplementation (dynamic JS now; memoizing the recursive inlining, however clever & elegant, turns out to have painful edge-cases & still not be efficient enough—web browsers really don’t like loading hundreds of kilobytes of extra HTML)
Gwern.net : +返回顶部浮动按钮；弹出窗口：现在可以禁用（使用“齿轮”图标）；最终的重新实现（现在是动态 JS；记住递归内联，无论多么聪明和优雅，结果都是有痛苦的边缘情况并且仍然不够高效——Web 浏览器真的不喜欢加载数百千字节的额外 HTML）
Scaling up : “DALL·E: Creating Images from Text” , OpenAI (GPT-3-12.5b generating 1280 tokens → VQ-VAE pixels; generates illustration & photos); “CLIP (Contrastive Language-Image Pre-training): Connecting Text and Images” , OpenAI ( Radford et al 2021 : zero-shot image understanding via text description—useful for much more than just ranking DALL·E samples by quality)
放大：“DALL·E：从文本创建图像”，OpenAI（GPT-3-12.5b生成1280个标记→VQ-VAE像素；生成插图和照片）； “CLIP（对比语言-图像预训练）：连接文本和图像”，OpenAI（Radford et al 2021：通过文本描述进行零样本图像理解 - 其用途不仅仅是按质量对 DALL·E 样本进行排名）
Further blessings of scale : simple contrastive training on n = 400m leads to remarkable generalization & combinatorial flexibility of image generation by DALL·E, and CLIP learns to reach image classification SOTA by zero-shot on many datasets, with more human-like errors & less degradation out of samples than rivals, while costing the same to train.
规模的进一步好处：n = 400m 上的简单对比训练导致 DALL·E 图像生成的显着泛化和组合灵活性，并且 CLIP 学会在许多数据集上通过零样本达到图像分类 SOTA，与竞争对手相比，具有更多类似人类的错误和更少的样本退化，同时训练成本相同。
OpenAI released their smallest CLIP model (the “ ViT -B/32”-equivalent) and people are discovering it seems able to do just about anything without any further training—the paper notes that it does everything from “fine-grained object classification, geo-localization, action recognition in videos, and OCR”, but there’s so much more, and you can use it to generate image captions/descriptions, classify your anime images, pull a specific target image description by gradient ascent or out of another neural network such as an ImageNet BigGAN or TADNE StyleGAN2-ext (or, why not, synthesize images images embodying abstract concepts like emoji or words like “nightmare fuel” or “confusion”!), search your image datasets by embedding, find mislabeled images (eg by using “upside down” as the prompt )…
OpenAI 发布了他们最小的 CLIP 模型（相当于“ViT -B/32”），人们发现它似乎不需要任何进一步的训练就能做任何事情——论文指出，它可以做所有事情，从“细粒度对象分类、地理定位、视频中的动作识别和 OCR”，但还有更多，你可以用它来生成图像标题/描述、对动漫图像进行分类、通过梯度上升或从另一个神经网络中提取特定的目标图像描述网络，如 ImageNet BigGAN 或 TADNE StyleGAN2-ext（或者，为什么不合成体现抽象概念的图像，如表情符号或“噩梦燃料”或“混乱”等单词！），通过嵌入搜索图像数据集，查找错误标记的图像（例如，通过使用“颠倒”作为提示）...
One wonders, like GPT-3, how much better the largest CLIP (“L/14-336px”) is and how many ways of using it (or DALL·E) remain to be found?
有人想知道，像 GPT-3 一样，最大的 CLIP (“L/14-336px”) 有多好，还有多少种使用它（或 DALL·E）的方法有待找到？
And why prediction losses work so well in one place, but then contrastive elsewhere?
为什么预测损失在一个地方效果很好，但在其他地方却相反？
For perspective: there are newly-minted PhDs going on the job market who got excited about deep learning because of these new “resnet” things; undergrads who applied to grad school because BERT et al were blowing open NLP & extending neural supremacy to natural language would not yet have passed quals; and it has been only 1 academic semester since GPT-3 was announced.
就观点而言：由于这些新的“resnet”事物，一些新进入就业市场的博士生对深度学习感到兴奋；因为 BERT 等人开放 NLP 并将神经霸权扩展到自然语言而申请研究生院的本科生还没有通过资格认证；自GPT-3公布以来，仅过了1个学期。
Or to put it quantitatively, for just sequence modeling: it has been 8,478 days since LSTM RNNs were published; 3,045 days since AlexNet’s ImageNet scores were released; 1,880 days since residual networks were published in a paper; 1,330 days since “Attention Is All You Need” hit Arxiv; 844 days since BERT’s paper was published; 718 days since GPT-2 was announced; 353 days since SimCLR , and 249 days since GPT-3 was; and 27 days since CLIP/DALL·E.^1^ Spring is coming.
或者定量地说，仅就序列建模而言：LSTM RNN 发布至今已有 8,478 天；自 AlexNet 的 ImageNet 分数发布以来 3,045 天；距离残差网络在论文中发表已有 1,880 天；自《Attention Is All You Need》登陆 Arxiv 以来已有 1,330 天； BERT 论文发表 844 天；自 GPT-2 宣布以来 718 天；自 SimCLR 以来 353 天，自 GPT-3 以来 249 天；距 CLIP/DALL·E 已有 27 天了。^1^ 春天即将来临。
(Some still insist we need not worry about “overpopulation on Mars” for >18,264 more days…)
（有些人仍然坚持认为我们不必担心“火星人口过剩”超过 18,264 天……）
“Meta Pseudo Labels” , Pham et al 2020 (90% on ImageNet by pretraining a meta-learning teacher using JFT-300M on a TPUv3-2048)
“元伪标签”，Pham 等人 2020（通过在 TPUv3-2048 上使用 JFT-300M 对元学习教师进行预训练，在 ImageNet 上达到 90%）
“Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity” , Fedus et al 2021 (1.57t-parameter GShard followup; the mixture-of-experts approach, while scaling stably, starts showing its limits)
“Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”，Fedus et al 2021（1.57t 参数 GShard 后续；专家混合方法在稳定扩展的同时，开始显示出其局限性）
Scaling down : “DeiT: Training data-efficient image transformers & distillation through attention” , Touvron et al 2020 (scaling Transformer classifiers down to ImageNet+1-GPU); “BoTNet: Bottleneck Transformers for Visual Recognition” , Srinivas et al 2021/ “Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet” , Yuan et al 2021 (hybrids); “not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution” , Han et al 2020/ “VQGAN: Taming Transformers for High-Resolution Image Synthesis” , Esser et al 2020 (training >1024px Transformer GANs on just 2 GPUs)
缩小规模：“DeiT：通过注意力训练数据高效的图像转换器和蒸馏”，Touvron 等人 2020（将 Transformer 分类器缩小到 ImageNet+1-GPU）； “BoTNet：视觉识别的瓶颈变压器”，Srinivas 等人 2021/“Tokens-to-Token ViT：在 ImageNet 上从头开始训练视觉变压器”，Yuan 等人 2021（混合）； “not-so-BigGAN：使用基于小波的超分辨率在小型计算上生成高保真图像”，Han 等人 2020/“VQGAN：驯服 Transformers 用于高分辨率图像合成”，Esser 等人 2020（仅在 2 个 GPU 上训练 >1024px Transformer GAN）
Transformer supremacy in image-related tasks continues, and GANs are becoming increasingly hybridized.
Transformer 在图像相关任务中继续占据主导地位，GAN 正变得越来越混合。
Do pure-GANs have a future, now that VAEs and autoregressive models are making such inroads into both the highest-quality & lowest-compute sample generation?
既然 VAE 和自回归模型正在向最高质量和最低计算的样本生成领域取得如此大的进展，那么纯 GAN 是否还有未来呢？
To take the GAN/DRL analogy seriously, perhaps they were they ultimately a dead end, akin to trying to learn everything from rewards, and an adversarial GAN loss ought to be only the cherry on the cake of a large unsupervised/semi-supervised generative model.
认真对待 GAN/DRL 类比，也许它们最终是一个死胡同，类似于试图从奖励中学习一切，而对抗性 GAN 损失应该只是大型无监督/半监督生成模型的蛋糕上的樱桃。
“ZeRO-Offload: Democratizing Billion-Scale Model Training” , Ren et al 2021 (partial CPU training for 13b-parameter models on 1 V100 GPU, scaling to 128 GPUs)
“ZeRO-Offload：大众化十亿级模型训练”，Ren 等人 2021（在 1 V100 GPU 上对 13b 参数模型进行部分 CPU 训练，扩展到 128 个 GPU）
“Prefix-Tuning: Optimizing Continuous Prompts for Generation” , Li & Liang 2021 (could the PET & CLIP trick of averaging multiple embeddings to yield much better performance be reused for GPT-3 prompts to greatly improve prompting?
“前缀调整：优化生成的连续提示”，Li＆Liang 2021（平均多个嵌入以产生更好性能的 PET 和 CLIP 技巧是否可以重复用于 GPT-3 提示，以极大地改善提示？
The fact that the prefix-tuning, by directly optimizing the prompt embeddings, yields better performance than even single optimized text prompts, suggests so.
事实上，通过直接优化提示嵌入来进行前缀调整，甚至比单个优化的文本提示也能产生更好的性能，这一事实表明了这一点。
The user could provide 3 or 4 similar prompts, and synthesize them into a single super-prompt to better program GPT-3…)
用户可以提供 3 或 4 个类似的提示，并将它们合成为一个超级提示，以更好地编程 GPT-3...）
“Scaling down Deep Learning” , Greydanus 2020 (cute: parametric simplified-MNIST for rapid iteration on tiny NNs: experiments in lottery-ticket & meta-learning of LRs/activations)
“缩小深度学习”，Greydanus 2020（可爱：用于在微型神经网络上快速迭代的参数化简化 MNIST：彩票实验和 LR/激活的元学习）
“The neural network of the Stockfish chess engine” (very lightweight NN designed for incremental recomputation over changing board states)
“Stockfish 国际象棋引擎的神经网络”（非常轻量级的神经网络，专为不断变化的棋盘状态进行增量重新计算而设计）
OpenAI departures : Dario Amodei, Sam McCandlish, Tom Brown, Tom Henighan, Chris Olah, Jack Clark, Ben Mann, Paul Christiano et al leave—most for an unspecified new entity ( “the elves leave Middle Earth” ?)
OpenAI 离职：Dario Amodei、Sam McCandlish、Tom Brown、Tom Henighan、Chris Olah、Jack Clark、Ben Mann、Paul Christiano 等人离开 - 大多数人离开了一个未指定的新实体（“精灵离开中土世界”？）
And the rest: “2020 AI Alignment Literature Review and Charity Comparison” , Larks
其余：《2020 AI 结盟文献综述及慈善比较》，Larks
“DeBERTa: Decoding-enhanced BERT with Disentangled Attention” , He et al 2020 ( SuperGLUE falls)
“DeBERTa：具有解缠注意力的解码增强型 BERT”，He et al 2020（SuperGLUE 下降）
“UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers” , Hu et al 2021
“UPDeT：通过与 Transformers 的策略解耦实现通用多智能体强化学习”，Hu 等人，2021 年
“FERM: A Framework for Efficient Robotic Manipulation” , Zhan et al 2021 (contrastive semi-supervised learning + data augmentation for sample-efficiency)
“FERM：高效机器人操作框架”，Zhan 等人 2021（对比半监督学习 + 样本效率的数据增强）
“XMC-GAN: Cross-Modal Contrastive Learning for Text-to-Image Generation” , Zhang et al 2021
“XMC-GAN：用于文本到图像生成的跨模态对比学习”，Zhang 等人，2021 年
Everything Is Heritable: “Nurture might be nature: cautionary tales and proposed solutions” , Hart et al 2021
一切都是可遗传的：“教养可能是自然：警示故事和建议的解决方案”，Hart 等人 2021
“A genetic perspective on the association between exercise and mental health in the era of genome-wide association studies” , de Geus 2020; “Evidence for shared genetics between physical activity, sedentary behaviour and adiposity-related traits” , Schnurr et al 2020
“全基因组关联研究时代运动与心理健康之间关系的遗传学视角”，de Geus 2020； “体力活动、久坐行为和肥胖相关特征之间共有遗传的证据”，Schnurr 等人 2020
“Antidepressant Response in Major Depressive Disorder: A Genome-wide Association Study” , Pain et al 2020
“重度抑郁症的抗抑郁药反应：全基因组关联研究”，Pain 等人 2020
“Genome wide analysis of gene dosage in 24,092 individuals shows that 10,000 genes modulate cognitive ability” , Huguet et al 2020 (yep, still polygenic)
“对 24,092 个个体的基因剂量进行全基因组分析表明，有 10,000 个基因调节认知能力”，Huguet 等人 2020（是的，仍然是多基因的）
“GWAS of three molecular traits highlights core genes and pathways alongside a highly polygenic background” , Sinnott-Armstrong et al 2021
“三个分子性状的 GWAS 突出了核心基因和通路以及高度多基因背景”，Sinnott-Armstrong 等人 2021
“Genome-scale sequencing and analysis of human, wolf and bison DNA from 25,000 year-old sediment” , Gelabert et al 2021 (incredible this is possible)
“对 25,000 年沉积物中的人类、狼和野牛 DNA 进行基因组规模测序和分析”，Gelabert 等人，2021 年（难以置信，这是可能的）
“Disentangling sex differences in the shared genetic architecture of PTSD, traumatic experiences, and social support with body size and composition” , Carvalho et al 2021 ( LCV )
“通过体型和成分来理清 PTSD、创伤经历和社会支持的共同遗传结构中的性别差异”，Carvalho 等人 2021 (LCV)
Recent Evolution: “African genetic diversity and adaptation inform a precision medicine agenda” , Pereira et al 2021; “The influence of evolutionary history on human health and disease” , Benton et al 2021; “Local adaptation and archaic introgression shape global diversity at human structural variant loci” , Yan et al 2021
最近的演变：“非洲遗传多样性和适应性为精准医学议程提供了信息”，Pereira 等人，2021 年； “进化史对人类健康和疾病的影响”，Benton 等，2021； “局部适应和古老的基因渗入塑造了人类结构变异位点的全球多样性”，Yan 等人，2021
“Genome scans of dog behavior implicate a gene network underlying psychopathology in mammals, including humans” , Zapata et al 2021
“对狗行为的基因组扫描暗示了哺乳动物（包括人类）精神病理学的基因网络”，Zapata 等人 2021
“Natural Selection in Contemporary Humans is Linked to Income and Substitution Effects” , Hugh-Jones & Abdellaoui 2021
“当代人类的自然选择与收入和替代效应有关”，Hugh-Jones 和 Abdellaoui 2021
“The diversity and function of sourdough starter microbiomes” , Landis et al 2021 (crowdsourced sourdough show little trace of geographic origins?)
“酸面团发酵剂微生物组的多样性和功能”，Landis 等人 2021（众包酸面团几乎没有地理起源的痕迹？）
Engineering: “In vivo base editing rescues Hutchinson-Gilford progeria syndrome in mice” , Koblan et al 2021
工程：“体内碱基编辑拯救小鼠 Hutchinson-Gilford 早衰综合症”，Koblan 等人 2021
“From Genotype to Phenotype: polygenic prediction of complex human traits” , Raben et al 2021
“从基因型到表型：复杂人类性状的多基因预测”，Raben 等人 2021
“The Quantum Field Theory on Which the Everyday World Supervenes” , Carroll 2021 (“…we have reason to be confident that the laws of physics underlying the phenomena of everyday life are completely known” because all unknown particles/fields are constrained to being extremely rare/weak, eg by Adelberger et al 2009 )
“The Quantum Field Theory on Which the Everyday World Supervenes”，Carroll 2021（“……我们有理由相信，日常生活现象背后的物理定律是完全已知的”，因为所有未知的粒子/场都被限制为极其罕见/微弱，例如 Adelberger 等人 2009 年）
“How accurate are citations of frequently cited papers in biomedical literature?” , Pavlovic et al 2020 (includes original author’s evaluation of whether a citation of their work is correct)
“生物医学文献中经常被引用的论文的引用有多准确？” ，Pavlovic et al 2020（包括原作者对其作品的引用是否正确的评估）
“Energy-Efficient Algorithms” , Demaine et al 2016 ( reversible computing asymptotics: constant-factor stacks / arrays , ?(log n ) time/energy AVL trees , ?( n ) space sorts , & various ?(Vertex+Edge) time/space/energy graph searches )
“节能算法”，Demaine 等人 2016（可逆计算渐近：常数因子堆栈/数组，?(log n) 时间/能量 AVL 树，?( n) 空间排序，以及各种 ?(Vertex+Edge) 时间/空间/能量图搜索）
“The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis” , Smith & Winkler 2006 (regression to the mean is everywhere; another example of why Bayes & decision theory are two great flavors that go great together)
“优化器的诅咒：决策分析中的怀疑主义和决策后惊喜”，Smith & Winkler 2006（均值回归无处不在；另一个例子说明为什么贝叶斯和决策理论是两种完美结合的伟大风格）
“The Mechanisms of Cult Production: An Overview” , Xavier Marquez 2020 (see previously his blog roundup )
“邪教生产的机制：概述”，Xavier Marquez 2020（参见之前的博客综述）
“Still Alive” , Scott Alexander (announcement of SSC return as Substack newsletter ‘Astral Codex Ten’ & launching a low-cost psychiatry clinic ‘Lorien Psychiatry’)
“Still Alive”，Scott Alexander（宣布 SSC 作为 Substack 时事通讯“Astral Codex Ten”回归并推出低成本精神病学诊所“Lorien Psychiatry”）
“The Temporal Dynamics of Opportunity Costs: A Normative Account of Cognitive Fatigue and Boredom” , Agrawal et al 2020
“机会成本的时间动态：认知疲劳和无聊的规范解释”，Agrawal 等人 2020
“A unified framework for association and prediction from vertex-wise grey-matter structure” , Couvy-Duchesne et al 2020 (more morphometricity )
“顶点灰质结构关联和预测的统一框架”，Couvy-Duchesne 等人 2020（更多形态测量）
Common phenomena : “Sounds from seeing silent motion: Who hears them, and what looks loudest?” , Fassnidge & Freeman 2018 (on ‘visual ear’; previously: Saenz & Koch 2008 , Fassnidge et al 2017 )
常见现象：“看到无声运动时发出的声音：谁听到了它们，什么看起来声音最大？” ，Fassnidge & Freeman 2018（关于“视觉耳朵”；之前：Saenz & Koch 2008，Fassnidge et al 2017）
“Predicting Mental Health From Followed Accounts on Twitter” , Costelli et al 2021 ( Registered Report : who you choose to follow says a lot about you— everything is correlated )
“根据 Twitter 上关注的帐户预测心理健康”，Costelli 等人 2021（注册报告：您选择关注的人可以透露很多关于您的信息 - 一切都是相关的）
“Microbiome connections with host metabolism and habitual diet from 1,098 deeply phenotyped individuals” , Asnicar et al 2021
“来自 1,098 名深度表型个体的微生物组与宿主代谢和习惯饮食的联系”，Asnicar 等人，2021
“Universal DNA methylation age across mammalian tissues” , Lu et al 2021; “Whole-body senescent cell clearance alleviates age-related brain inflammation and cognitive impairment in mice” , Ogrodnik et al 2021
“哺乳动物组织的通用 DNA 甲基化年龄”，Lu 等人，2021 年； “全身衰老细胞清除可减轻小鼠与年龄相关的脑部炎症和认知障碍”，Ogrodnik 等人 2021
“BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data” , Kostas et al 2021 (towards brain imitation learning)
“BENDR：使用 Transformer 和对比自监督学习任务从大量脑电图数据中学习”，Kostas 等人 2021（走向大脑模仿学习）
Correction : Programming competition skills do not inversely correlate with job performance after all
更正：编程竞赛技能毕竟与工作表现并不成反比
“Baffles and Bastions: The Universal Features of Fortifications” , Keeley et al 2007
“挡板和堡垒：防御工事的普遍特征”，Keeley 等人 2007 年
"Does Advertising Actually Work?" (what could be more obvious than “advertising works”, and trivial to confirm with correlational data?
“广告真的有用吗？” （还有什么比“广告作品”更明显，而且用相关数据来确认也很简单？
Yet, the tedious saying “correlation ≠ causation” stubbornly insists on being true); “Digital Paywall Design: Implications for Content Demand and Subscriptions” , Aral & Dhillon 2020 (NYT nag-paywall caused −9.9% reading; in line with all the other results )
然而，“相关≠因果”这句乏味的话却顽固地坚持是正确的）； “数字付费专区设计：对内容需求和订阅的影响”，Aral & Dhillon 2020（《纽约时报》nag-paywall 造成 -9.9% 的阅读量；与所有其他结果一致）
“Who Gains and Who Loses from Credit Card Payments?
“谁从信用卡支付中获益，谁受损？
Theory and Calibrations” , Schuh et al 2010 (a compelling case for getting a rewards credit card if you’re a debit card user—why subsidize them so much?)
理论和校准”，Schuh 等人 2010（如果您是借记卡用户，那么获得奖励信用卡的一个令人信服的案例 - 为什么要给他们这么多补贴？）
“Squeezing the bears: cornering risk and limits on arbitrage during the ‘British bicycle mania’, 1896–1898” , Quinn 2019
“挤压空头：1896-1898 年‘英国自行车狂热’期间规避风险和套利限制”，Quinn 2019