Does grid search learn which candidate to try next?

Basic grid search follows a predefined collection of combinations rather than adapting it from earlier scores.

Does it always use cross-validation?

No. Grid search is the enumeration strategy; cross-validation is one possible evaluation procedure.

Is the winning candidate globally optimal?

It is the winner under the reported scores and selection rule among evaluated candidates. That is different from global optimality over every possible setting or guaranteed best future performance.

AI Glossary / Graphs and symbolic AI / Intermediate

Grid search

Definition

Grid search is a systematic way to evaluate every combination in a predefined set of candidate parameter values. In machine learning, it commonly selects hyperparameters by comparing model configurations under a chosen validation procedure. It finds the best-scoring evaluated candidate, not necessarily the best possible setting outside the chosen grid.

What it means

Suppose a model has two adjustable settings and each has several candidate values. Grid search forms their combinations, like crossing rows and columns in a table, and evaluates each one. With more settings, the same idea extends to a higher-dimensional grid.
This entry uses the parameter-search meaning. Searching for a path through a spatial grid is a different task that can use algorithms such as breadth-first search or A*. The shared word grid should not obscure the distinction. In parameter search, the points are configurations, and the main comparison is their measured objective or validation score.

How it works

Choose candidate values, a scoring rule, and a valid evaluation design. For each combination, train or otherwise evaluate the system under comparable conditions. Aggregate scores as appropriate, select according to the stated criterion, and record the results.
When cross-validation is used, each candidate is fitted separately within each training fold and scored on its corresponding validation fold. Any learned preprocessing should also be fitted inside that procedure. A final refit can train the selected configuration on the available development data. Preserve a separate assessment strategy for estimating performance after selection rather than treating the winning tuning score as an unbiased final result.

Three practical examples

A small classification exercise compares several tree-depth and minimum-leaf settings using the same cross-validation splits.
A text-modeling project evaluates a limited set of preprocessing and model configurations together in a pipeline.
A simulation study evaluates all combinations of two operating parameters over a modest finite grid, reporting the objective at every tested point.

Think of it this way

Think of testing every recipe formed from three oven settings and two baking times. The six trials cover that chosen table completely. The analogy is limited because noisy model evaluation may require repeated or cross-validated measurements, and success at one tested combination says nothing definitive about every untested recipe between or beyond the selected values.

Worked example

Consider an illustrative model grid with maximum depth in {2, 4, 6} and minimum leaf size in {1, 5}. There are 3 × 2 = 6 configurations. With three-fold cross-validation, evaluating the grid requires 6 × 3 = 18 model fits, excluding any final refit or preprocessing-specific overhead.
Suppose the candidate (depth 4, leaf size 5) receives validation accuracies 0.80, 0.84, and 0.82. Its mean is (0.80 + 0.84 + 0.82) / 3 = 0.82. Another candidate receives 0.79, 0.80, and 0.81, giving mean 0.80. Under a rule that chooses the highest mean accuracy, the first beats the second on these measurements.
These invented scores do not establish a statistically decisive difference or a future accuracy guarantee. The winning configuration must also be compared with the other four candidates. If selected for a final refit, the total becomes 19 fits. Adding four candidate values for a third setting would expand the grid to 24 configurations and require 72 cross-validation fits before refitting.

Illustrative case study

In a fictional project, Birch Text Lab starts with a small grid and publishes the complete candidate table in its experiment notes. A reviewer discovers that a vocabulary-building step had used data from all folds before evaluation, leaking information into the validation process.
The team moves that learned step inside the pipeline and reruns the comparison. It also records the dataset split and scoring rule with the selected settings. The invented case illustrates evaluation design; it does not claim a measured improvement or reuse the illustrative scores as real findings.

Common misconceptions and mistakes

Exhaustive coverage of a grid is not exhaustive coverage of a continuous parameter space. A coarse or poorly chosen grid can miss useful regions entirely. Adding many dimensions can also make the number of combinations impractical.
The best observed score is not guaranteed to identify the configuration with the best true generalization performance. Sampling noise and repeated selection matter. Avoid tuning against a supposedly untouched test set, and do not fit learned preprocessing on validation data. Grid search itself does not enforce a sound scientific comparison unless the surrounding evaluation procedure is correctly designed.

Business use

Grid search provides a transparent, reproducible baseline when the candidate set is small enough to evaluate. It makes the explored choices explicit and can reveal interactions between settings. Its cost is easy to estimate from the number of combinations and evaluation repetitions.
Before launching a run, define the budget, scoring objective, split strategy, and selection rule. Record failed fits as well as successful scores. If the grid grows too large, consider whether fewer meaningful values or another search strategy would answer the actual question. Use final evaluation to assess the selected workflow, not to keep expanding the tuning loop.

Check your understanding

How many fits are needed for six configurations under three-fold cross-validation followed by one final refit?

Answer: Eighteen fits evaluate the candidates, and one more refits the selected configuration, giving nineteen fits in this simplified count.

Related terms

Cross-validation
Validation set
Model selection
Pipeline
Data leakage
hyperparameter
parameter grid
random search

Learn next

Build a small parameter table and calculate the fit budget before running it. Then study nested cross-validation or an independent final test procedure. Compare the meaning of a tuning score with the performance estimate reported after model selection.

Sources

Sources: scikit-learn, GridSearchCV official documentation: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

Previous: Grid sampling Next: GRIM test