System.Xml.XmlElement

How much of the rapid progress in AI that we’ve seen over the last few years 1 has come from data versus model improvements?

过去几年我们看到的人工智能的快速进步有多少来自数据与模型的改进？

The answer has big implications for the economics of frontier labs and the pace of future progress.

这个答案对前沿实验室的经济和未来进展的步伐具有重大影响。

We investigate this question at a relatively small scale, and for pretraining specifically, from 2019 to 2025.

我们在相对较小的范围内调查了这个问题，特别是在 2019 年至 2025 年期间进行预训练。

During each of those years, a new open model recipe was published which codified that year’s publicly known algorithmic tweaks (for example, improvements in architecture, optimizer, initializations, learning rate schedule, hyperparams, etc).

在每一年中，都会发布一个新的开放模型配方，其中编纂了当年公开的算法调整（例如，架构、优化器、初始化、学习率计划、超参数等方面的改进）。

And during each of those years, there was also a new public data corpus (produced by broader scrapes and new curation/extraction/filtering techniques).

在这些年里，还出现了一个新的公共数据语料库（通过更广泛的抓取和新的管理/提取/过滤技术产生）。

We train combinations of these year-representative model recipes and data corpuses across different scales of training compute (up to 1e19 FLOPs) 2 .

我们在不同规模的训练计算（高达 1e19 FLOPs）2 中训练这些具有年份代表性的模型配方和数据语料库的组合。

Obviously, we can’t compare these different models by their cross-entropy loss against a fixed dataset, since we’re varying the datasets they’re trained on.

显然，我们无法通过它们与固定数据集的交叉熵损失来比较这些不同的模型，因为我们正在改变它们训练的数据集。

So instead we evaluate these models on end capabilities as measured by the OLMES eval (which aggregates 10 different relatively easy benchmarks, mostly multiple choice QA).

因此，我们根据 OLMES 评估（汇总了 10 个不同的相对简单的基准，主要是多项选择 QA）衡量的最终能力来评估这些模型。

Unfortunately, evaluating end capability rather than pretraining loss adds some noise to our results, as you’ll see in the graphs below, though we try to get cleaner bounds by running multiple seeds.

不幸的是，评估最终能力而不是预训练损失会给我们的结果增加一些噪音，如下图所示，尽管我们尝试通过运行多个种子来获得更清晰的边界。

We find that from 2019 to 2025, 3.24x more compute efficiency gains have come from data improvements rather than model improvements (12.0x for data and 3.7x for models), at the 1e19 FLOPs compute budget 3 .

我们发现，从 2019 年到 2025 年，在 1e19 FLOPs 计算预算下，计算效率提高了 3.24 倍，来自数据改进，而不是模型改进（数据提高 12.0 倍，模型提高 3.7 倍） 3 。

Here is a grid which shows how much better a model we train does on the end capability we're testing it on, relative to the 2019 data + architecture baseline, at 3.16e18 FLOPs 4 .

下面的表格显示了我们训练的模型在我们测试的最终能力上的表现，相对于 2019 年数据 + 架构基线（3.16e18 FLOPs 4 ）要好多少。

We find that the gains from data and model improvements are mostly independent and don’t interact (i.e. realizing the gains from some model improvement doesn’t require a specific training datapile, or vice versa).

我们发现数据和模型改进的收益大多是独立的并且不相互作用（即实现某些模型改进的收益不需要特定的训练数据堆，反之亦然）。

88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements (using a linear model).

OLMES 分数中 88% 的方差可以通过模型和数据改进的相加效应（使用线性模型）来解释。

For context, let’s briefly summarize what changed on both the data and the model side from 2019 to 2025.

作为背景，我们简单总结一下 2019 年到 2025 年数据和模型方面的变化。

On the model side, we went from GPT-2 to OLMo-2 , including key innovations in optimizers, positional encodings, normalization, activation functions, initializations, and more 5 .

在模型方面，我们从 GPT-2 转向 OLMo-2，包括优化器、位置编码、标准化、激活函数、初始化等方面的关键创新 5 。

On the data side, we started with OpenWebText in 2019, which contained just web pages linked from Reddit with enough upvotes and then deduplicated and filtered, and thus amounted to only ~9B tokens (this was mostly what GPT-2 was trained on).

在数据方面，我们从 2019 年开始使用 OpenWebText，其中仅包含从 Reddit 链接的具有足够赞成票的网页，然后进行重复数据删除和过滤，因此总共只有约 9B 个令牌（这主要是 GPT-2 训练的内容）。

By 2025, open source data corpuses like UltraFineWeb not only are far larger (by using scrapes of the whole Internet), but also use much more sophisticated filtering (for example, by training a classifier to predict what data will empirically improve model performance).

到 2025 年，像 UltraFineWeb 这样的开源数据集不仅规模更大（通过使用整个互联网的碎片），而且还使用更复杂的过滤（例如，通过训练分类器来预测哪些数据将凭经验提高模型性能）。

A naive interpretation of our result is that most of the AI progress from 2019-2024 (the era of pretraining) was just better data engineering (extraction, curation, etc.), and that all the model work during that period was much less important.

对我们结果的一个天真的解释是，2019-2024 年（预训练时代）的大部分人工智能进展只是更好的数据工程（提取、管理等），而该时期的所有模型工作都不那么重要。

But this is probably the wrong way to think about the value of model improvements.

但这可能是思考模型改进价值的错误方式。

Their main contribution was not necessarily compute efficiency - that is, achieving the same performance with fewer FLOPs.

他们的主要贡献不一定是计算效率——也就是说，用更少的 FLOP 实现相同的性能。

Rather, it was making larger amounts of compute usable in the first place.

相反，它首先使更大量的计算变得可用。

As the number of parameters, context lengths, run duration, and clusters scale up, all kinds of things are prone to breaking (gradients explode or vanish, memory and bandwidth run out, training becomes infeasibly slow).

随着参数数量、上下文长度、运行持续时间和集群规模的扩大，各种事情都容易出现问题（梯度爆炸或消失、内存和带宽耗尽、训练变得异常缓慢）。

Much of model research has consisted of removing or pushing back these constraints to scaling .

许多模型研究都包括消除或推回这些对扩展的限制。

Many of the most important innovations such as MoEs, sparse attention variants, stability innovations (norm placements, initializations, etc.) and system / kernel-level optimizations like FlashAttention fall into this category.

许多最重要的创新，例如 MoE、稀疏注意力变体、稳定性创新（规范放置、初始化等）以及系统/内核级优化（例如 FlashAttention）都属于这一类。

The data improvements we investigated here might matter less for larger models.

我们在这里研究的数据改进对于较大的模型来说可能不太重要。

Small models (like the ones we trained) see significant gains from data quality improvements, because they don’t have that much capacity, and so you have to be really careful about what you stuff into them.

小型模型（例如我们训练的模型）可以从数据质量改进中获得显着收益，因为它们没有那么多容量，因此您必须非常小心地填充它们。

Whereas big models have so much excess capacity that maybe you just want to throw in as much stuff as you can, even if it’s mostly garbage, and the magic of stochastic gradient descent will separate out the signal from the noise.

然而大型模型有如此多的过剩容量，也许你只想投入尽可能多的东西，即使它大部分是垃圾，而随机梯度下降的魔力会将信号从噪声中分离出来。

If you choose to filter aggressively, you’ll have to do dozens of epochs, which empirically gives worse performance than just having a lower average quality but larger dataset.

如果您选择积极过滤，则必须进行数十个时期，从经验上看，这比仅具有较低平均质量但较大数据集的性能更差。

In fact, aggressive data curation is even more harmful once you take into account that frontier models are up to 100x overtrained relative to Chinchilla optimal, in order to minimize the inference compute used for RL and for deployment.

事实上，一旦考虑到前沿模型相对于 Chinchilla 最优模型的过度训练高达 100 倍，为了最大限度地减少用于 RL 和部署的推理计算，激进的数据管理就更加有害。

An analogy might be the difference between a sailboat and a container ship - the container ship doesn’t necessarily go faster, but it can lug thousands of tons of cargo (analogous to hundreds of trillions of tokens of pretraining data), and won’t be toppled by choppy waters (analogous to training stably across hundreds of thousands of GPUs).

可以类比帆船和集装箱船的区别——集装箱船不一定走得更快，但它可以拖运数千吨的货物（相当于数百万亿代币的预训练数据），并且不会被波涛汹涌的海水推倒（相当于在数十万个 GPU 上稳定训练）。

Now that we have more capacious and sturdy container ships, we don’t have to fret about exactly what we load on board - we can just fill them up with everything that’s even remotely and plausibly useful.

现在我们拥有更宽敞、更坚固的集装箱船，我们不必担心船上到底装载了什么——我们可以在船上装满所有看似有用的东西。

Whereas for the tiny flimsy sailboats of 2019, you’d have to be incredibly careful about only carrying the most valuable cargo.

而对于 2019 年的小而脆弱的帆船，你必须非常小心，只运载最有价值的货物。

But to the extent that the nature of pretraining progress is simply loading more cargo into this ship, are we running out of cargo?

但就预训练进度的本质而言，只是将更多货物装载到这艘船上，那么我们的货物是否已经用完了？

This is a question about the data wall and about how well synthetic data has helped us leap over it.

这是一个关于数据墙以及合成数据如何帮助我们跨越数据墙的问题。

Synthetic data is obviously being widely used at the labs, and we have not at all investigated whether it can effectively expand a data corpus without hurting model performance.

合成数据显然在实验室中得到了广泛的使用，但我们根本没有研究过它是否可以在不损害模型性能的情况下有效地扩展数据集。

If the gains are limited, then the main driver of pretraining progress will stall, because we’re not generating more internet, and you can only curate a fixed set of data by so much.

如果收益有限，那么预训练进度的主要驱动力将停滞不前，因为我们不会产生更多的互联网，而且你只能管理一组固定的数据。

To be clear, we have no active reason to think this.

需要明确的是，我们没有充分的理由这么认为。

But given how important data seems to be in driving pretraining progress, this seems like a crucial question to investigate.

但考虑到数据对于推动预训练进展的重要性，这似乎是一个需要研究的关键问题。

Ryan Greenblatt noted that many of the historical improvements in pretraining data corpuses look like the kind of progress that automated researchers would be able to just test empirically - for example, run ablations trained on different data and see how the model performs.

Ryan Greenblatt 指出，预训练数据语料库的许多历史改进看起来像是自动化研究人员能够凭经验进行测试的进步 - 例如，运行针对不同数据训练的消融并查看模型的表现。

So it’s totally compatible with our results that the data progress which has propelled pretraining since 2019 might speed up a lot if and when we automate AI R&D.

因此，如果我们实现人工智能研发自动化，自 2019 年以来推动预训练的数据进展可能会大大加快，这与我们的结果完全一致。

We want to clarify that whether pretraining progress in isolation will speed up or slow down is not really the most important question for overall AI progress, because so many of the gains over the last two years have come from RL.

我们想澄清的是，孤立的预训练进展是否会加速或减慢并不是整个人工智能进展的最重要问题，因为过去两年的许多成果都来自强化学习。

These are some directions of future research that we think would be really cool, and important questions to answer:

这些是我们认为非常酷的未来研究方向，也是需要回答的重要问题：

You could run this experiment at larger scales to see whether the data or model improvements are more dependent on scale (and thus far more impactful at the frontier)

您可以在更大的规模上运行此实验，以查看数据或模型的改进是否更依赖于规模（因此在前沿更具影响力）

What is the marginal value of novel high-quality data for both pre and post-training, as measured by end capabilities?

根据最终能力衡量，训练前和训练后的新颖高质量数据的边际价值是多少？

We want to know broadly how effectively synthetic data works.

我们想广泛了解合成数据的工作效率如何。

One concrete question to investigate is this: if you’ve got a small corpus of high quality data, how much better is it to magnify it via synthetic data generation relative to just training on it for multiple epochs ?

需要研究的一个具体问题是：如果您有一小部分高质量数据，那么通过合成数据生成来放大它比仅对其进行多个时期的训练要好多少？

You could figure out the implied value of data through lab spending on data brokers, environment producers, etc., relative to their spending on compute and researchers.

您可以通过实验室在数据经纪人、环境生产者等方面的支出相对于他们在计算和研究人员方面的支出来计算数据的隐含价值。

We wanted to investigate what role data has played in driving AI progress.

我们想要调查数据在推动人工智能进步方面发挥了什么作用。

There are lots of other ways one could probe this question, and some may be more clever and informative than ours.

还有很多其他方法可以探讨这个问题，有些可能比我们的方法更聪明、信息更丰富。

And even our experiment was done at an extremely small scale.

甚至我们的实验也是在极小的规模下进行的。

We definitely think it’s plausible that there is something we missed - we’re eager to hear how others would research this question, and ideally to also see their results!

我们绝对认为我们错过了一些东西是合理的——我们渴望听到其他人将如何研究这个问题，并且最好也看到他们的结果！

We pre-train these model recipes from scratch on these different data corpuses, at varying compute budgets, with multiple independent seeds 6 .

我们在这些不同的数据语料库上，以不同的计算预算，使用多个独立的种子 6 从头开始​​预训练这些模型配方。

Our compute budgets are: 1e17, 3.16e17, 1e18, 3.16e18 and 1e19 FLOPs.

我们的计算预算为：1e17、3.16e17、1e18、3.16e18 和 1e19 FLOP。

The compute accounting convention is to use nominal compute C = 6ND (N = number of non-embedding parameters, D = tokens of data).

计算计算约定是使用名义计算 C = 6ND（N = 非嵌入参数的数量，D = 数据的标记）。

At each compute budget, we vary the number of parameters (and hence number of tokens trained on), to determine the compute-optimal mix for each training recipe x corpus combination.

在每个计算预算中，我们改变参数的数量（以及因此训练的标记的数量），以确定每个训练配方 x 语料库组合的计算最佳组合。

We use held-out loss on the corpus to determine this compute-optimal point.

我们使用语料库上的保留损失来确定这个计算最佳点。

We can then obtain compute scaling curves of downstream performance of each combination, from which we can finally extract our compute multipliers.

然后，我们可以获得每个组合的下游性能的计算缩放曲线，从中我们最终可以提取计算乘数。

We enforce a shared tokenizer and context length across every run: GPT-2 BPE (tiktoken, 50,257 vocab) and T=2048, batch = 262,144 tokens.

我们在每次运行中强制执行共享标记器和上下文长度：GPT-2 BPE（tiktoken，50,257 个词汇）和 T=2048，批次 = 262,144 个标记。

The end capabilities of our training runs are highly dependent on hyperparameters.

我们训练运行的最终能力高度依赖于超参数。

Obviously, there is no way to sweep over all possible sets of hyperparams (hyperparam tuning is a fine art indeed)!

显然，没有办法扫描所有可能的超参数集（超参数调整确实是一门艺术）！

We try to control for this as much as possible, and we consider peak learning rate as the main hyperparameter of significance.

我们尝试尽可能地控制这一点，并且我们将峰值学习率视为重要的主要超参数。

Some algorithm vintages do provide specifications of what peak learning rate should be tuned to (as a function of other relevant variables such as model size, data budget, batch size, etc.).

一些算法年份确实提供了峰值学习率应调整到的规范（作为其他相关变量的函数，例如模型大小、数据预算、批量大小等）。

These serve as good priors for what we think the optimal learning rate is.

这些是我们认为最佳学习率的良好先验。

We first sweep learning rates at 5 anchor points - 3 different model sizes and 2 different D/N ratios.

我们首先扫描 5 个锚点的学习率 - 3 个不同的模型大小和 2 个不同的 D/N 比率。

We determine the optimal learning rate of these anchor points, and fit an optimal learning rate parametric form

我们确定这些锚点的最佳学习率，并拟合最佳学习率参数形式

For all the model recipes except OLMo-2, we fit a common exponent a and b , and a model-specific lr₀.

对于除 OLMo-2 之外的所有模型配方，我们拟合一个公共指数 a 和 b 以及特定于模型的 lr₀。

For OLMo-2, we use the prescribed optimal learning rate according to the model recipe.

对于 OLMo-2，我们根据模型配方使用规定的最佳学习率。

The reason we do this for OLMo-2 is that Ai2 published small-model ladders as part of the recipe which specified optimal hyperparameters at the scale we are investigating.

我们为 OLMo-2 这样做的原因是 Ai2 发布了小模型梯子作为配方的一部分，该配方指定了我们正在研究的规模的最佳超参数。

We also verify, at the compute-optimal point for 3.16e18 FLOPs, that our production learning rates are at or near optimal.

我们还验证了，在 3.16e18 FLOP 的计算最佳点，我们的生产学习率处于或接近最佳状态。

We observe generally increasing compute efficiency across time for both the model and data axes as expected.

我们观察到模型和数据轴的计算效率普遍随着时间的推移而提高，正如预期的那样。

Some outliers that we observed:

我们观察到的一些异常值：

NeoX performs worse than GPT-2 at 1e19 (although it does better across the 1e17 to 3.16e18 range).

NeoX 在 1e19 上的表现比 GPT-2 差（尽管它在 1e17 到 3.16e18 范围内表现更好）。

This might arise from noise in the OLMES evaluation.

这可能是由 OLMES 评估中的噪音引起的。

We also note that on held-out pretraining loss on the FineWeb-Edu corpus, NeoX performs better than GPT-2.

我们还注意到，在 FineWeb-Edu 语料库上保留的预训练损失方面，NeoX 的表现优于 GPT-2。

The Piles seems to do much worse than OpenWebText.

Piles 的表现似乎比 OpenWebText 差得多。

This is not surprising since the Pile’s main improvement was data corpus diversity over filtering.

这并不奇怪，因为 Pile 的主要改进是数据语料库的多样性而不是过滤。

It has a curated 22-source mixture including PubMed and arXiv papers, GitHub code, legal opinions, patents, and parliamentary proceedings.

它拥有精选的 22 个来源的混合物，包括 PubMed 和 arXiv 论文、GitHub 代码、法律意见、专利和议会程序。

The amount of cross-domain transfer to OLMES (which is English web-prose MCQ) might be minimal for many of these tokens, thus resulting in lower compute efficiency.

对于许多这些令牌来说，到 OLMES（即英语网络散文 MCQ）的跨域传输量可能很小，从而导致计算效率较低。

We note that by virtue of its larger size, we expect that the Pile should eventually be better than (the really small) OpenWebText at larger scales.

我们注意到，由于其较大的尺寸，我们预计 Pile 最终在更大的尺度上应该比（非常小的）OpenWebText 更好。

It is also worth noting that the compute multipliers for NeoX and the Pile are obtained by extrapolation, which introduces further potential error.

还值得注意的是，NeoX 和 Pile 的计算乘数是通过外推法获得的，这会引入进一步的潜在误差。

Every point on the compute scaling curves is computed from multiple independently seeded training runs.

计算缩放曲线上的每个点都是通过多次独立种子训练运行计算得出的。

The error bars there are the standard deviation of the OLMES eval over those seeds.

误差线是这些种子的 OLMES 评估的标准偏差。

Consider some given reference level of performance at some compute level for our reference model or data corpus.

考虑我们的参考模型或数据语料库在某些计算级别上的某些给定参考性能水平。

We then calculate the compute multiplier by finding the left-most point of the compute scaling curve of our candidate model or corpus that first attains that reference level of performance.

然后，我们通过找到首先达到参考性能水平的候选模型或语料库的计算缩放曲线的最左边的点来计算计算乘数。

The ratio of the compute required by the reference to the compute required by our candidate is the candidate’s compute multiplier

参考所需的计算与候选所需计算的比率是候选的计算乘数

The error bars on the compute multipliers are obtained from a parametric bootstrap of the entire estimation pipeline, and are 1 standard deviation intervals

计算乘数上的误差线是从整个估计管道的参数引导中获得的，并且是 1 个标准偏差区间

We do want to highlight that we expect the actual uncertainty in the compute multipliers of the model recipes to be higher than indicated by our error bars.

我们确实想强调，我们预计模型配方的计算乘数的实际不确定性将高于误差线所示的水平。

This is because of additional uncertainty introduced by the limited extent of hyperparameter tuning we did, and end capabilities or held-out loss is probably quite sensitive to the exact choice of peak learning rate / batch size / etc.

这是因为我们所做的有限程度的超参数调整带来了额外的不确定性，并且最终能力或保留损失可能对峰值学习率/批量大小等的准确选择非常敏感。

It is also important to note that there are many reasons why our ablations do not necessarily capture the full scope of compute efficiency gains.

同样重要的是要注意，我们的消融不一定能全面提高计算效率，原因有很多。

Indeed, from 2019 to 2025, we observe year-over-year compute efficiency gains (CEG) of 1.24x [1.19, 1.29] on the model side and 1.51x [1.45, 1.57] on the data side.

事实上，从 2019 年到 2025 年，我们观察到模型端的计算效率 (CEG) 同比增长了 1.24 倍 [1.19, 1.29]，数据端的计算效率同比增长了 1.51 倍 [1.45, 1.57]。

Measured jointly, we observe a 1.57x YoY CEG [1.49, 1.65] 7 .

联合测量后，我们观察到 CEG 同比增长 1.57 倍 [1.49, 1.65] 7 。

This is indeed much lower than Anson Ho et al.’s mean estimate of 3x YoY , for the following reasons:

这确实比 Anson Ho 等人平均估计的 3 倍同比低得多，原因如下：

Many of the gains might be scale dependent or might be especially important at longer context, and we are operating at scales too small to realize many of the gains.

许多收益可能与规模相关，或者在更长期的背景下可能特别重要，而我们的运营规模太小，无法实现许多收益。

For example, OLMo-2’s layer and QK norms, parallel attention + MLP block in NeoX

例如OLMo-2的层和QK规范，NeoX中的并行注意力+MLP块

Inference efficiency optimizations (such as LLama-3’s GQA , which is a KV cache optimization) do not show up as compute multipliers in our study.

推理效率优化（例如 LLama-3 的 GQA，这是一种 KV 缓存优化）在我们的研究中并未显示为计算乘数。

We are also not investigating tokenizer improvements.

我们也没有研究分词器的改进。

The compute multipliers we obtain are pretty sensitive to our choice of model recipe or data corpus for each year.

我们获得的计算乘数对我们每年选择的模型配方或数据语料库非常敏感。

We have chosen what we believe to be representative model recipes or data corpuses.

我们选择了我们认为具有代表性的模型配方或数据语料库。

But by no means do we exhaustively conclude that these are the best of each year.

但我们绝不能详尽地得出结论说这些是每年最好的。

We are looking at compute multipliers with respect to the OLMES benchmark (which combines 10 different relatively easy task types) rather than compute multipliers in getting to some perplexity metric.

我们正在考虑相对于 OLMES 基准（它结合了 10 种不同的相对简单的任务类型）的计算乘数，而不是计算乘数以获得某些困惑度指标。

We would also have very different looking numbers if we were looking at other benchmarks (say, coding- or problem-solving-specific ones), which would probably reward very different methods of data engineering.

如果我们查看其他基准（例如，特定于编码或解决问题的基准），我们也会得到非常不同的数字，这可能会奖励非常不同的数据工程方法。

We also want to note that we have not investigated other data-side improvements, such as collecting more high-quality data from new sources, human expert generated data, synthetic data generation methods, etc. Most of the corpuses we have investigated are curations (subsets) of the same Common Crawl, rather than expanding the available set of data.

我们还想指出的是，我们还没有研究其他数据方面的改进，例如从新来源收集更多高质量数据、人类专家生成的数据、合成数据生成方法等。我们研究的大多数语料库都是同一 Common Crawl 的精选（子集），而不是扩展可用的数据集。

This is clearly consumption of a finite stock - there is only so far we can push this lever.

这显然是有限库存的消耗——我们只能推动这个杠杆。

Here is the investigation that we did to determine how independent the gains from model recipe and data corpus are.

这是我们为了确定模型配方和数据语料库的收益的独立性而进行的调查。

We looked at the grid of OLMES scores at 3.16e18 FLOPs.

我们查看了 3.16e18 FLOP 的 OLMES 分数网格。

A linear regression of OLMES score = mean + model effect + data effect gives an R squared of 0.88, which means 88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements, with only ~12% of the variance accounted for by interaction or higher order terms, and eval noise.

OLMES 得分 = 均值 + 模型效应 + 数据效应的线性回归得出 R 平方为 0.88，这意味着 OLMES 得分中 88% 的方差可以通过模型和数据改进的加性效应来解释，只有约 12% 的方差是由交互作用或高阶项以及评估噪声造成的。

This hints that complex model-data interactions (where exploiting some model improvement is contingent on some specific data engineering, or vice versa) are relatively minor.

这暗示复杂的模型-数据交互（其中利用某些模型改进取决于某些特定的数据工程，反之亦然）相对较小。

Anson Ho et al.

安森何等人。

estimated software efficiency improvements (in pretraining) of 3x per year (95% CI: 1.5x to 64x).

估计软件效率（在预训练中）每年提高 3 倍（95% CI：1.5 倍至 64 倍）。

As Ho mentioned in this blog , “most software progress might actually be due to data quality improvements” and “from scaling up just a small handful of scale-dependent algorithmic changes”.

正如 Ho 在这篇博客中提到的，“大多数软件进步实际上可能是由于数据质量的改进”和“来自于扩展一小部分与规模相关的算法变化”。

The 2019 model recipe was GPT-2, and 2025 model recipe was OLMo-2.

2019年的模型配方是GPT-2，2025年的模型配方是OLMo-2。

The 2019 data corpus was OpenWebText, and the 2025 data corpus was UltraFineWeb.

2019年的数据语料库是OpenWebText，2025年的数据语料库是UltraFineWeb。

Our implementation of GPT3 encountered some training instabilities (gradient spikes) on the Pile.

我们的 GPT3 实现在 Pile 上遇到了一些训练不稳定（梯度尖峰）。

These include: Optimizer improvements, warmup + decay schedules, RoPE replacing learning absolute positions, RMSNorm + SwiGLU gated MLPs, Norm reordering, QK-norm, Z-loss regularization and cleaner inits.

其中包括：优化器改进、预热 + 衰减计划、RoPE 替换学习绝对位置、RMSNorm + SwiGLU 门控 MLP、Norm 重新排序、QK-norm、Z-loss 正则化和更干净的初始化。

For the compute-scaling plots we use at least 3 seeds each.

对于计算缩放图，我们每个至少使用 3 个种子。

For the 7x7 grid of combinations of model recipes and data corpus at the 3.16e18 budget, we only used 1 seed each.

对于 3.16e18 预算的模型配方和数据语料库组合的 7x7 网格，我们每个仅使用 1 个种子。

The 1.57x YoY multiplier is computed using the joint improvement from 2019 model and corpus to 2025 model and corpus, and not the product of the 1.24x model side improvement and 1.51x data side improvement.

1.57 倍同比乘数是使用 2019 年模型和语料库到 2025 年模型和语料库的联合改进计算得出的，而不是 1.24 倍模型侧改进和 1.51 倍数据侧改进的乘积。