Is development set another name for validation set?

Often yes, although naming conventions vary across projects.

Can a model overfit the development set without training directly on it?

Yes. Repeated selection and manual changes based on its results can adapt the system to that set.

Must every project use one fixed development split?

No. Cross-validation and other designs can serve development needs when chosen appropriately.

AI Glossary / Natural language and speech / Beginner

Development set

Definition

A development set is a collection of examples used to guide choices while building a machine-learning system. It is also commonly called a validation set or dev set. Teams use it to compare configurations, choose thresholds, or select a training checkpoint. Because it influences development decisions, its score is not an untouched final estimate of performance.

What it means

Training data are used to fit a model's learned parameters. Development data help decide which version or setup to use. A separate test set can then assess the selected system after the relevant choices have been settled.
The distinction concerns how data are used, not just the filename assigned to them. If a team repeatedly changes its system in response to test results, those results have become part of development feedback.
A development set should reflect the task and comparison the team cares about. Its examples may need to preserve groups, time ordering, or realistic input conditions. A convenient random split is not automatically appropriate for every dataset.

How it works

Define the intended evaluation setting and separate the data accordingly. Fit each candidate using the training portion, then evaluate it on development examples under a consistent metric and procedure.
Use those results to make development decisions, such as choosing regularization strength or a classification threshold. Record what was compared and avoid treating repeated experiments as if each score were an independent final assessment.
Once the system and selection procedure are fixed, evaluate on an appropriate held-out test set. When data are limited, cross-validation within the development process can provide additional comparisons, while a separate final assessment or a suitable nested evaluation design protects against optimistic selection effects.

Three practical examples

A sentiment-classification team can compare feature configurations on a development set before evaluating the selected setup on its final test set.
A speech-recognition project can use development recordings to select a checkpoint, keeping final evaluation speakers separate where the task requires speaker generalization.
A document-routing system can choose a decision threshold using development labels and then assess that fixed threshold on new held-out examples.

Think of it this way

Think of rehearsals before a final performance assessment. Rehearsal feedback helps change what you do, so success on rehearsed material is not the same evidence as success on a fresh assessment. The analogy explains feedback and selection, though machine-learning splits additionally require attention to statistical dependence, data leakage, and the population being evaluated.

Worked example

Consider an invented corpus of 10,000 labeled messages. A team allocates 7,000 for training, 1,500 for development, and 1,500 for final testing. These are illustrative counts, not a universally recommended ratio.
Two classifiers are fitted on the same training examples. Candidate A correctly classifies 1,230 development messages, giving 1,230/1,500 = 82%. Candidate B correctly classifies 1,275, giving 85%. Under the team's stated accuracy criterion, B is selected.
With the model configuration now fixed, B is evaluated on the held-out test set and correctly classifies 1,245 examples, or 83%. The difference from its development score does not by itself show a mistake: finite samples differ, and development selection can favor candidates that happen to do well on that set.
Suppose the team now changes B's features repeatedly to improve the 83% test result. That test set is no longer untouched evidence for the final revised system. The team should describe its use honestly and obtain an appropriate fresh assessment or redesign its evaluation procedure.
Also suppose messages from the same support conversation appear in different splits. Similar wording or shared context may make the evaluation easier than handling genuinely new conversations. Split design must match the intended unit of generalization, not merely produce the desired row counts.

Illustrative case study

In a fictional voice-interface project, recordings are divided randomly by audio clip. Several clips from each speaker appear in training and development, but the product must handle previously unseen speakers.
The team revises its split to separate speakers and reassesses its development choices. It records that the new evaluation answers a different and more relevant question, rather than comparing the scores as if only model quality had changed.
The case illustrates that a development set's usefulness depends on what it represents. A well-maintained boundary cannot compensate for a split that measures the wrong operating scenario.

Common misconceptions and mistakes

Do not assume development examples are used to fit weights in the same way as training examples. Their usual role is selection and feedback, although that feedback still influences the resulting system.
Do not report the best score found after many development experiments as an unbiased final result.
Do not fit preprocessing statistics on held-out data when the evaluation requires those data to remain unseen. Transformations learned from data should respect the split.
Do not assume one split percentage is always correct. Dataset size, class coverage, grouping, time, and evaluation uncertainty matter. Also, renaming a reused test set does not restore its independence.

Business use

A development set gives teams a shared basis for comparing changes before release. Establish its purpose, metric, split logic, and ownership so that results remain interpretable across experiments.
Track important subgroups and errors alongside the aggregate score. Preserve a final assessment process that reflects the decisions actually made during development. This helps prevent a project from repeatedly optimizing a familiar benchmark while remaining uncertain about behavior on genuinely new work.

Check your understanding

A team changes its model after repeatedly inspecting final-test errors. Is that test set still untouched evidence for the revised model?

Answer: No. Its information has influenced development, so the evaluation must account for that reuse.

Related terms

Learn next

Draw the path by which each dataset influences a modeling decision. Include preprocessing and manual error analysis, then check that the final evaluation remains appropriate for the intended population and selection process.

Sources

Sources: Jurafsky and Martin, Speech and Language Processing, Chapter 4, section 4.10: https://web.stanford.edu/~jurafsky/slp3/4.pdf Scikit-learn documentation, cross-validation and held-out evaluation: https://scikit-learn.org/stable/modules/cross_validation.html

Previous: Detrended correspondence analysis Next: Developmental robotics