Does doubling model size halve loss?

Not generally; the fitted relationship determines the change.

Are scaling laws only about parameters?

No; data and computation are also common variables.

Can a fitted law stop predicting well?

Yes, especially when conditions or scale change.

AI Glossary / Natural language and speech / Advanced

Scaling laws

Definition

Scaling laws are empirical relationships describing how a model's measured performance changes as resources such as model size, training data, or computation increase. In language modeling, they often approximate loss using power-law functions over a studied range. They support predictions and resource planning, but are not universal guarantees about every model, task, or future scale.

What it means

A scaling law summarizes a pattern observed across training runs. Instead of saying only that a larger model performed better, researchers estimate how performance varies with the resources used. The relationship can help compare candidate training configurations before committing to an expensive run.
The measured outcome matters. A law fitted to next-token prediction loss does not directly specify factual accuracy, instruction-following quality, or success on a business workflow. Those outcomes require their own evaluation.
Model size and data also interact. A large model trained on too few suitable tokens may be an inefficient use of the available computation. Research by Kaplan and colleagues and later Hoffmann and colleagues illustrates how empirical studies can produce different recommendations about resource allocation. The experimental regime and fitting choices belong with the result.

How it works

Researchers train a controlled family of models while varying resources, measure a defined loss, and fit a mathematical relationship to the observations. They then test predictions against runs not used to fit the relationship. Useful reporting includes the data distribution, model family, training procedure, resource range, and uncertainty.
An illustrative form is L(N)=E+A/N^alpha, where N is model size, E is a fitted floor, and A and alpha are positive constants. Multiplying N changes the reducible part L-E by a predictable factor under this model. It does not reduce total loss by that same factor when E is nonzero.
A two-resource model can include separate terms for parameter count and training tokens. Under a specified compute constraint, one can seek the allocation that minimizes predicted loss. Hoffmann and colleagues studied this question for dense language models; their conclusions are conditional on their training setting rather than a permanent ratio for every application.
Training computation and lifetime deployment cost are different objectives. A configuration favored by a training-only calculation may be less attractive when serving many requests, meeting a latency target, or operating under memory limits. State the optimization objective before calling a configuration optimal.

Three practical examples

A research team uses smaller training runs to estimate which larger configurations are worth testing.
A model-development group compares increasing parameter count with increasing training tokens under a fixed computation budget.
A product team checks whether an anticipated reduction in model loss translates into improved results on representative user tasks.

Think of it this way

Think of a production trend estimated from pilot runs. It can inform the next experiment, but changing the process can change the trend. An equation does not eliminate uncertainty.

Worked example

Consider a hypothetical loss model L(n)=1+4/sqrt(n), where n is model size measured in a chosen normalized unit. Assume a suitable training regime at each size. This invented curve describes no named model.
At n=1, predicted loss is 5. At n=4, it is 3. At n=16, it is 2. Each fourfold size increase halves the excess above the floor of 1: the excess moves from 4 to 2 to 1. Total loss does not halve; it moves from 5 to 3 to 2.
Doubling size multiplies the excess by 1/sqrt(2), approximately 0.7071. Thus the excess decreases by about 29.3%, not 50%. At n=2, the predicted total loss is approximately 3.8284. The distinction between total loss and excess loss prevents an exaggerated interpretation of the curve.
Now consider a separate toy allocation model L(n,d)=1+4/sqrt(n)+4/sqrt(d), with normalized parameter resource n and token resource d. Assume the compute proxy is their product and the available budget requires n×d=16. The units are simplified.
The balanced allocation n=4, d=4 predicts loss 1+2+2=5. The allocation n=16, d=1 predicts 1+1+4=6. Reversing those resources gives the same value. Choosing n=8, d=2 gives approximately 1+1.4142+2.8284=5.2426.
For positive n and d with product 16, the two resource-dependent terms have product 16/sqrt(16)=4. Their sum is at least twice the square root of that product, or 4, with equality when the terms are equal. Therefore the toy model's minimum loss is 5 at n=d=4.
This balanced answer follows from equal coefficients, equal exponents, the product constraint, and the toy units. It does not say that real parameter counts should numerically equal token counts. Different fitted coefficients, exponents, data conditions, or costs change the answer.
If a new run at n=16 measures loss 2.4 rather than the first curve's prediction of 2, the prediction error is 0.4 loss units. That observation calls for checking uncertainty and assumptions, not relabeling the measured value as the predicted one. These figures are illustrative.

Illustrative case study

Fictional example: ModelHarbor plans a larger text model using a curve fitted to clean editorial material. Its target product mostly processes short, noisy support messages. Reviewers notice that the planning report assumes the original curve transfers unchanged.
The team adds representative pilot runs and tests the actual support tasks. It compares the updated forecasts with serving requirements before selecting a configuration. This fictional scenario illustrates validation.

Common misconceptions and mistakes

A scaling law is not a physical guarantee that more resources improve every capability. A smooth trend in average loss can coexist with uneven results across tasks or examples. A fitted exponent also needs a defined resource and experimental range.
Do not confuse training tokens with unique documents, or a compute proxy with measured monetary cost. Extrapolation far beyond observed runs requires additional evidence. Finally, a recommendation that was optimal for one study need not remain optimal after changing the data, architecture, training procedure, or deployment objective.

Business use

Use scaling estimates to prioritize experiments, with assumptions and uncertainty visible. Compare predicted gains against direct task evaluations and measured costs. Record forecast errors and consider operational constraints and task-specific gains.

Check your understanding

What halves after a fourfold size increase in the first toy curve?

Answer: Excess loss above one. Which toy allocation minimizes loss at product sixteen? n=4 and d=4.

Does that imply equal real token and parameter counts?

Answer: No.

Related terms

Cross-entropy loss
Power law
compute budget
parameter count
training tokens
extrapolation
sample efficiency
model evaluation

Learn next

Fit a simple curve to controlled experiments, reserve runs for validation, and compare predictions with observations. Then change the cost objective and reassess the preferred allocation.

Sources

Kaplan and colleagues, Scaling Laws for Neural Language Models, 2020: https://arxiv.org/abs/2001.08361 Hoffmann and colleagues, Training Compute-Optimal Large Language Models, 2022: https://arxiv.org/html/2203.15556v1

Previous: Scaling Next: Scan Operation