$$ \newcommand{\bone}{\mathbf{1}} \newcommand{\bbeta}{\mathbf{\beta}} \newcommand{\bdelta}{\mathbf{\delta}} \newcommand{\bepsilon}{\mathbf{\epsilon}} \newcommand{\blambda}{\mathbf{\lambda}} \newcommand{\bomega}{\mathbf{\omega}} \newcommand{\bpi}{\mathbf{\pi}} \newcommand{\bphi}{\mathbf{\phi}} \newcommand{\bvphi}{\mathbf{\varphi}} \newcommand{\bpsi}{\mathbf{\psi}} \newcommand{\bsigma}{\mathbf{\sigma}} \newcommand{\btheta}{\mathbf{\theta}} \newcommand{\btau}{\mathbf{\tau}} \newcommand{\ba}{\mathbf{a}} \newcommand{\bb}{\mathbf{b}} \newcommand{\bc}{\mathbf{c}} \newcommand{\bd}{\mathbf{d}} \newcommand{\be}{\mathbf{e}} \newcommand{\boldf}{\mathbf{f}} \newcommand{\bg}{\mathbf{g}} \newcommand{\bh}{\mathbf{h}} \newcommand{\bi}{\mathbf{i}} \newcommand{\bj}{\mathbf{j}} \newcommand{\bk}{\mathbf{k}} \newcommand{\bell}{\mathbf{\ell}} \newcommand{\bm}{\mathbf{m}} \newcommand{\bn}{\mathbf{n}} \newcommand{\bo}{\mathbf{o}} \newcommand{\bp}{\mathbf{p}} \newcommand{\bq}{\mathbf{q}} \newcommand{\br}{\mathbf{r}} \newcommand{\bs}{\mathbf{s}} \newcommand{\bt}{\mathbf{t}} \newcommand{\bu}{\mathbf{u}} \newcommand{\bv}{\mathbf{v}} \newcommand{\bw}{\mathbf{w}} \newcommand{\bx}{\mathbf{x}} \newcommand{\by}{\mathbf{y}} \newcommand{\bz}{\mathbf{z}} \newcommand{\bA}{\mathbf{A}} \newcommand{\bB}{\mathbf{B}} \newcommand{\bC}{\mathbf{C}} \newcommand{\bD}{\mathbf{D}} \newcommand{\bE}{\mathbf{E}} \newcommand{\bF}{\mathbf{F}} \newcommand{\bG}{\mathbf{G}} \newcommand{\bH}{\mathbf{H}} \newcommand{\bI}{\mathbf{I}} \newcommand{\bJ}{\mathbf{J}} \newcommand{\bK}{\mathbf{K}} \newcommand{\bL}{\mathbf{L}} \newcommand{\bM}{\mathbf{M}} \newcommand{\bN}{\mathbf{N}} \newcommand{\bP}{\mathbf{P}} \newcommand{\bQ}{\mathbf{Q}} \newcommand{\bR}{\mathbf{R}} \newcommand{\bS}{\mathbf{S}} \newcommand{\bT}{\mathbf{T}} \newcommand{\bU}{\mathbf{U}} \newcommand{\bV}{\mathbf{V}} \newcommand{\bW}{\mathbf{W}} \newcommand{\bX}{\mathbf{X}} \newcommand{\bY}{\mathbf{Y}} \newcommand{\bZ}{\mathbf{Z}} \newcommand{\calA}{\mathcal{A}} \newcommand{\calB}{\mathcal{B}} \newcommand{\calC}{\mathcal{C}} \newcommand{\calD}{\mathcal{D}} \newcommand{\calE}{\mathcal{E}} \newcommand{\calF}{\mathcal{F}} \newcommand{\calG}{\mathcal{G}} \newcommand{\calH}{\mathcal{H}} \newcommand{\calI}{\mathcal{I}} \newcommand{\calJ}{\mathcal{J}} \newcommand{\calK}{\mathcal{K}} \newcommand{\calL}{\mathcal{L}} \newcommand{\calM}{\mathcal{M}} \newcommand{\calN}{\mathcal{N}} \newcommand{\calO}{\mathcal{O}} \newcommand{\calP}{\mathcal{P}} \newcommand{\calQ}{\mathcal{Q}} \newcommand{\calR}{\mathcal{R}} \newcommand{\calS}{\mathcal{S}} \newcommand{\calT}{\mathcal{T}} \newcommand{\calU}{\mathcal{U}} \newcommand{\calV}{\mathcal{V}} \newcommand{\calW}{\mathcal{W}} \newcommand{\calX}{\mathcal{X}} \newcommand{\calY}{\mathcal{Y}} \newcommand{\calZ}{\mathcal{Z}} \newcommand{\R}{\mathbb{R}} \newcommand{\C}{\mathbb{C}} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\F}{\mathbb{F}} \newcommand{\Q}{\mathbb{Q}} \DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min} \newcommand{\nnz}[1]{\mbox{nnz}(#1)} \newcommand{\dotprod}[2]{\langle #1, #2 \rangle} \newcommand{\ignore}[1]{} \let\Pr\relax \DeclareMathOperator*{\Pr}{\mathbf{Pr}} \newcommand{\E}{\mathbb{E}} \DeclareMathOperator*{\Ex}{\mathbf{E}} \DeclareMathOperator*{\Var}{\mathbf{Var}} \DeclareMathOperator*{\Cov}{\mathbf{Cov}} \DeclareMathOperator*{\stddev}{\mathbf{stddev}} \DeclareMathOperator*{\avg}{avg} \DeclareMathOperator{\poly}{poly} \DeclareMathOperator{\polylog}{polylog} \DeclareMathOperator{\size}{size} \DeclareMathOperator{\sgn}{sgn} \DeclareMathOperator{\dist}{dist} \DeclareMathOperator{\vol}{vol} \DeclareMathOperator{\spn}{span} \DeclareMathOperator{\supp}{supp} \DeclareMathOperator{\tr}{tr} \DeclareMathOperator{\Tr}{Tr} \DeclareMathOperator{\codim}{codim} \DeclareMathOperator{\diag}{diag} \newcommand{\PTIME}{\mathsf{P}} \newcommand{\LOGSPACE}{\mathsf{L}} \newcommand{\ZPP}{\mathsf{ZPP}} \newcommand{\RP}{\mathsf{RP}} \newcommand{\BPP}{\mathsf{BPP}} \newcommand{\P}{\mathsf{P}} \newcommand{\NP}{\mathsf{NP}} \newcommand{\TC}{\mathsf{TC}} \newcommand{\AC}{\mathsf{AC}} \newcommand{\SC}{\mathsf{SC}} \newcommand{\SZK}{\mathsf{SZK}} \newcommand{\AM}{\mathsf{AM}} \newcommand{\IP}{\mathsf{IP}} \newcommand{\PSPACE}{\mathsf{PSPACE}} \newcommand{\EXP}{\mathsf{EXP}} \newcommand{\MIP}{\mathsf{MIP}} \newcommand{\NEXP}{\mathsf{NEXP}} \newcommand{\BQP}{\mathsf{BQP}} \newcommand{\distP}{\mathsf{dist\textbf{P}}} \newcommand{\distNP}{\mathsf{dist\textbf{NP}}} \newcommand{\eps}{\epsilon} \newcommand{\lam}{\lambda} \newcommand{\dleta}{\delta} \newcommand{\simga}{\sigma} \newcommand{\vphi}{\varphi} \newcommand{\la}{\langle} \newcommand{\ra}{\rangle} \newcommand{\wt}[1]{\widetilde{#1}} \newcommand{\wh}[1]{\widehat{#1}} \newcommand{\ol}[1]{\overline{#1}} \newcommand{\ul}[1]{\underline{#1}} \newcommand{\ot}{\otimes} \newcommand{\zo}{\{0,1\}} \newcommand{\co}{:} %\newcommand{\co}{\colon} \newcommand{\bdry}{\partial} \newcommand{\grad}{\nabla} \newcommand{\transp}{^\intercal} \newcommand{\inv}{^{-1}} \newcommand{\symmdiff}{\triangle} \newcommand{\symdiff}{\symmdiff} \newcommand{\half}{\tfrac{1}{2}} \newcommand{\bbone}{\mathbbm 1} \newcommand{\Id}{\bbone} \newcommand{\SAT}{\mathsf{SAT}} \newcommand{\bcalG}{\boldsymbol{\calG}} \newcommand{\calbG}{\bcalG} \newcommand{\bcalX}{\boldsymbol{\calX}} \newcommand{\calbX}{\bcalX} \newcommand{\bcalY}{\boldsymbol{\calY}} \newcommand{\calbY}{\bcalY} \newcommand{\bcalZ}{\boldsymbol{\calZ}} \newcommand{\calbZ}{\bcalZ} $$

2020

  1. Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, and 7 more authors
    Jan 2020
    arXiv Bib PDF

    Paper Abstract

    We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

Four Important Things

1. Power Law Relationship of \(L\) With Respect To \(N, C, D\)

The paper shows empirically that language modeling loss \(L\) is a predictable function of:

  • \(N\), the number of model parameters (excluding embeddings).
  • \(D\), the size of the dataset,
  • \(C\), the amount of compute used for training,

when they are not bottlenecked by any of the other two parameters.

The paper also observed that larger models are more sample-efficient than smaller models, being able to reach the same loss with fewer samples.

In the graph below, we see that the curves corresponding to the \(10^9\) parameter model achieves a much smaller test loss compared to the \(10^3\) parameter model at all regimes:

In addition, they surprisingly found that for a fixed compute budget and no restrictions on model size or available data, the optimal loss that can be achieved is not by training a model to convergence, but rather training a very large model that significantly stops short of convergence. One’s intuition for smaller models would be that models should be trained to convergence, and this often happens in practice due to hardware constraints.

In the graph below, we see that the curves flatten out towards the end with increasing compute, which provides intuition on why it is preferable to over-parameterize the model further in order to stay within the non-flattening regime of decaying test loss:

2. Transformer Performance Depends Weakly on Shape Parameters

We parameterize the Transformer architecture by the following hyperparameters:

  • \(n_{\textrm{layer}}\), the number of Transformer block layers
  • \(d_{\textrm{model}}\), the dimension of the residual stream (i.e the dimensions output by the residual connection and layer normalization components, which is the dimension preserved between each Transformer block)
  • \(d_{\textrm{ff}}\), the dimension of the feed-forward layer
  • \(d_{\textrm{attn}}\), the dimension of the attention output
  • \(n_{\textrm{heads}}\), the number of attention heads per layer

To jolt your memory of how these different components come together again, here’s the original Transformer architecture from Vaswani et. al:

The authors showed that the performance of Transformers only depends weakly on \(n_{\textrm{layer}}\), \(n_{\textrm{heads}}\), and \(d_{\textrm{ff}}\) when the total number of parameters is held fixed. It was not elaborated what “weakly” means precisely, but I understood it to mean that the relationship is something that would be swallowed in the context of big-O analysis.

To study the relationships of performance on these hyperparameters, they always scaled \(d_{\mathrm{model}}\) accordingly to keep \(N \approx 12 n_{\mathrm{layer}} d_{\mathrm{model}}^2\) fixed. \(d\_{\mathrm{model}}\) controls the width of the Transformer, and makes sense to be the one modified since it can allow the most significant parameter. changes without changing its own value too significantly.

  1. (Right plot) For \(n_{\mathrm{heads}}\), we see that loss increases as \(n_{\mathrm{heads}}\) decreases, but this can be offset by increasing \(d_{\mathrm{model}}\), which leads to their remark in the plot.

  2. (Middle plot) For \(n_{\mathrm{layer}}\), as the number of layer decreases and \(d_{\mathrm{model}}\) increases, the loss increase goes down and then goes up again across a range of architectures with different \(N\). This means there is a sweet spot for trading off between the two parameters.

  3. (Left plot) For \(d_{\mathrm{ff}}\), as \(d_{\mathrm{ff}}\) increases and \(d_{\mathrm{model}}\) decreases, the increase in loss goes down slightly, before growing a lot more. This also hints at a sweet spot between the two parameters.

The fact that the loss only weakly depends on model shape is desirable since it makes the analysis of scaling laws more robust (minor changes in architecture should not result in major changes in the results, much like how minor tweaks to the Turing machine architecture should not change asymptotic analysis significantly).

3. Comparisons with LSTMs

It would be interesting to understand how the scaling laws of Transformers compare to the classical LSTM.

On the left plot, we see that the test loss of Transformers asymptotically beat that of LSTMs.

On the right plot (red plots are for LSTM, blue plots for Transformers), we see that while LSTM and Transformer curves have lower per-token losses at all indices as the number of parameters increases, this trend plateaus for LSTMs beyond some token index, but continues to decay for Transformers. The per-token loss should decrease as the index increases since there is more context to constrain the possibilities of the next token.

4. Generalization Among Data Distributions

The authors also verified that the scaling laws hold for different data distributions:

Most Glaring Deficiency

I thought the paper was generally very comprehensive and analyzed the scaling laws from many different dimensions. However, it could have gone a step further to distill the learnings from the results to provide concrete guidelines on how to choose \(N, C, D\) when training LLMs.

Conclusions for Future Work

Given a fixed compute budget, one can choose \(N, D\) appropriately to achieve a desired loss.

While the paper provided empirical evidence for the scaling laws, there is not yet any theoretical understanding of why this is the case, which would be an interesting line of future research.

Written 2024