AI Glossary / AI and machine learning / Advanced
Adam
Definition
Adam is the most widely used algorithm for training neural networks: an optimization method that adjusts each of the model's millions of internal numbers with its own automatically adapted step size, making training fast and forgiving of careless settings.
What it means
Training a neural network means nudging millions of internal values, called parameters, in the direction that reduces error. The basic method, gradient descent, uses one step size for all of them, which is like tuning a piano with a single wrench setting.
Adam, short for adaptive moment estimation, keeps two running statistics per parameter: a smoothed average of recent gradients, the direction of improvement, and a smoothed average of their sizes. From these it computes a sensible step for each parameter individually: big steps for slow-moving parameters, small careful steps for jumpy ones.
The practical effect is that training works out of the box far more often. Before Adam and its relatives, getting a network to learn could take days of fiddling with the learning rate; after them, default settings often just work, which is a large part of why deep learning spread.
Adam is not magic and not always best. Some tasks, especially fine-tuning for final accuracy, still favor older methods with careful schedules, and researchers continue to study when each wins.
For a business reader, Adam is worth knowing as the engine under nearly every AI model you use: when a vendor says the model was trained, this is the family of machinery that did the training.
How it works
At each training step, the algorithm computes the gradient, how each parameter should change to reduce error on the current batch of examples. Adam then maintains a moving average of the gradient, the momentum term, and a moving average of the squared gradient, the scale term.
The update divides the momentum by the square root of the scale: parameters with consistently large gradients get damped steps, and parameters with small, steady gradients get amplified ones. Each parameter effectively carries its own learning rate.
A bias correction repairs the early steps, when the moving averages are still warming up and would otherwise skew the first updates.
The result is an optimizer that tolerates a wide range of problems and settings, which is why it became the default in nearly every deep-learning framework and model release.
Three practical examples
- Training a support-ticket classifier: Adam converges in hours with default settings, where plain gradient descent needed manual tuning.
- Fine-tuning a language model on company documents: Adam handles the millions of parameters with per-parameter step sizes automatically.
- Experiment cycles: teams test ideas faster because the optimizer rarely needs babysitting between runs.
Think of it this way
Think of a driving instructor with a separate pedal for every wheel. Instead of one accelerator for the whole car, each wheel gets exactly the push its grip allows: more where there is traction, less where it is slipping.
The limit: the instructor optimizes the driving, not the destination. Adam finds a good setting of the model's numbers; whether the model is the right model, trained on the right data, is outside its view.
Worked example
why the default settings usually work
Suppose a small data team fine-tunes an open model to classify customer emails. With plain gradient descent, their first runs either crawl or diverge, and days pass in learning-rate experiments.
Switching to Adam with default settings, the same training converges on the first evening, because each parameter's step size adapts instead of sharing one fragile global choice. The timeline is invented; the experience is the common one reported across the field.
The team still validates the resulting model on fresh tickets, because a smoothly trained model can still be the wrong model for the task.
Adam's real gift to the project was not accuracy but iteration speed: three ideas tested per week instead of one.
Illustrative case study
A fictional startup believes its model quality problems come from the optimizer and spends a month testing exotic alternatives to Adam. Quality barely moves.
A reviewer points at the training data: half the labels in one category are wrong. Fixing the labels improves the model more in a day than the optimizer hunt did in a month.
The invented lesson: the optimizer is rarely the bottleneck anymore. When a model disappoints, inspect data, labels and task design before the training machinery.
Common misconceptions and mistakes
- Adam learns what the model cannot. It tunes parameters for the task given; it cannot fix bad data, bad labels or the wrong task.
- Newer optimizers are always better. Adam remains a strong default; alternatives win in specific regimes, not in general.
- Training success means model success. Low training error says the numbers tuned well; usefulness on fresh cases is a separate measurement.
Business use
You will almost never tune Adam yourself, but knowing its role helps you read claims: trained or fine-tuned describes this machinery, and it says nothing about data quality or task fit.
When a project struggles, spend diagnostic effort on data and task definition first; the optimizer is seldom the culprit in modern practice.
People also ask
What does Adam stand for?
Adaptive moment estimation, from its two running averages of gradient direction and size.
Is Adam a kind of AI model?
No. It is the training algorithm that adjusts a model's parameters during learning.
Why did it matter so much?
It made training reliable despite careless settings, which turned deep learning from a specialist craft into standard practice.
Check your understanding
A team spends a month swapping optimizers to fix poor model quality. What should they have inspected first?
Answer: Data and labels: modern optimizers like Adam are rarely the bottleneck, while wrong labels and bad task design cap quality regardless of training machinery.
Related terms
Learn next
AI for Business Owners helps you question model-quality claims at the right level: https://aiforbusinessandlife.com/academy/ai-for-business-owners. It is an adoption course, not a deep-learning course. [C2]
More in AI and machine learning
- Action - In AI, an action is anything an agent can do that changes its environment or its relationship to it: moving, sending, writing, booking, buying.
- Action model learning - Action model learning is the process of learning descriptions of what actions require and how they change an environment.
- Active learning - Active learning is a machine-learning approach where the system chooses which examples it wants labeled next, instead of learning from a fixed or random set.
- Adaptive boosting - Adaptive boosting, or AdaBoost, is an ensemble method that combines many simple, weak models into one strong predictor by training them in sequence, each...
- Additive regression - Additive regression is a modeling approach that builds a prediction by adding up the contributions of many simple components, each learned from the data...
- Adversarial machine learning - Adversarial machine learning studies deliberate attempts to manipulate machine-learning systems and methods for evaluating or reducing those risks.
Sources
[S11] Murphy, Probabilistic Machine Learning: An Introduction. Full URLs are in the source ledger. Verified 11 October 2026.
Previous: ADALINE Next: Adamic - Adar index