How is a hyper-parameter different from a parameter?

Parameters are learned from data during training; hyper-parameters are chosen before training and control how that learning proceeds.

What is random search?

Sampling dial combinations at random within ranges instead of testing a full grid; with many dials it usually finds good settings faster.

AI Glossary / AI and machine learning / Intermediate

Hyper-parameter

Definition

A hyper-parameter is a setting chosen before training that controls how the model learns, rather than something the model learns from the data itself: how fast, how deep, how many, how strict.

What it means

A model learns its parameters from data: the weights, the splits, the coefficients. Hyper-parameters are the dials a human sets first: the learning rate, the number of trees, the depth limit, the strength of regularization.

The distinction matters because the two are found differently. Parameters come out of training automatically; hyper-parameters are found by search and judgment, usually by trying candidates and scoring them on validation data.

Small dials, big consequences. A learning rate too high and training oscillates uselessly; a tree depth too generous and the model memorizes; the right values turn the same algorithm from useless to excellent on identical data.

Hyper-parameter tuning is its own discipline: grid searches, random searches and smarter sequential methods explore the dial-space, always judged on validation data, always at the cost of compute, and always vulnerable to overfitting the validation set itself.

For a business, hyper-parameters are where model quality is negotiated: better settings are bought with experimentation time, and the honest question is not what the best setting is but how much tuning the decision at stake justifies.

How it works

List the hyper-parameters for the chosen method and their sensible ranges from experience and documentation.

Choose a search strategy: a coarse grid for few dials, random search for many, sequential methods when each trial is expensive.

Score every candidate on validation data, never on the sealed test set, using the metric the business actually cares about.

Pick the winner, retest on untouched data, and record the chosen values and the search that found them.

Revisit the dials when data or drift changes the problem; last year's settings are a starting point, not scripture.

Three practical examples

A team tuning learning rate and tree count for a gradient boosting model across 60 validation trials before settling.
A deep learning project where one decimal place in the learning rate separates a model that trains from one that diverges.
A procurement of an autoML tool whose main job is searching hyper-parameter space faster than a junior analyst can.

Think of it this way

Think of baking with a new oven. The recipe's ingredients are the parameters, worked out by the method; the oven's temperature and timing are the hyper-parameters, set by you beforehand. The same ingredients burn or rise depending on those dials, and no amount of stirring fixes a wrong temperature.

The limit: every oven is different, and so is every dataset. Published best settings are someone else's oven; your dial positions must be found on your own data, which is why tuning is a cost line, not a lookup.

Worked example

sixty trials, one dial chart

Suppose an insurer tunes a claims model and tracks three hyper-parameters across sixty trials. The figures are illustrative.

Most trials cluster near 84 per cent validation accuracy; a narrow band of learning-rate and depth combinations reaches 89 per cent.

The winning combination is retested on sealed data and holds at 88 per cent, and the dial chart goes into the model file.

A year later, new claim types arrive, the band shifts, and a twenty-trial refresh finds the new sweet spot in a day because the old chart showed where to look.

Illustrative case study

A fictional bank bought a model from a vendor whose report showed spectacular accuracy, achieved, it later emerged, after the vendor had tuned hyper-parameters against the test set for weeks.

On the bank's fresh data the model performed no better than the incumbent, and the contract's acceptance criteria, written around test-set accuracy, offered no protection.

The invented lesson: ask where the dials were tuned. A hyper-parameter search pointed at the test set is leakage with a lab coat on.

Common misconceptions and mistakes

Hyper-parameters are learned by the model. They are set outside training; the model learns parameters under whatever settings the dials specify.
More tuning always helps. Searches overfit the validation set when pushed hard, and the marginal gain shrinks fast past the sensible band.
There is one best setting. The best settings shift with data volume, drift and the metric chosen; tuning is maintenance, not installation.

Business use

Budget tuning time explicitly: the first sensible settings get most of the value, and the last points of accuracy cost the most trials.

Keep a record of every search, dials tried and scores, so future tuning starts from evidence rather than folklore.

Score tuning candidates on validation data only, and make touching the test set during tuning a reportable incident.

Check your understanding

Why must hyper-parameter tuning use validation data rather than the test set?

Answer: Because the dials are chosen by their scores, so tuning on the test set trains on it indirectly, and the final number stops being an honest measure of live performance.

Related terms

Learn next

AI for Business Owners explains how to read a model evaluation and spot where the impressive number came from: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S24] scikit-learn documentation, Glossary of Common Terms and API Elements. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Hyper basis function network Next: Hypercube (communication pattern)