Is NAS part of AutoML?

Yes. It focuses on neural-network architecture within the broader effort to automate machine-learning development.

Is architecture search different from hyperparameter tuning?

They overlap. Architectural choices are a specialized form of configuration, often with complex structural dependencies.

Does NAS require training thousands of complete models?

Not necessarily. Methods use different budgets and estimation shortcuts, each with its own reliability trade-offs.

AI Glossary / Deep learning / Intermediate

Neural Architecture Search

Definition

Neural architecture search, or NAS, is the use of automated methods to explore neural-network designs and select promising architectures for a task. It searches within a defined set of possibilities using an evaluation objective and a limited budget.

What it means

Training a network usually adjusts its numerical weights. Architecture search addresses design choices such as layer types, widths, connections or repeated building blocks. The two activities interact because an architecture must be trained or otherwise assessed to judge its usefulness.

NAS does not search every imaginable network. People still define the allowable components, data, objectives and constraints. A result is therefore a promising design under that setup, not proof of a universally best architecture.

The objective may include accuracy alongside latency, memory or energy-related measurements. A design suitable for a server can be unsuitable for a phone, even when both perform the same prediction task.

How it works

Three questions organize a NAS project. The search space specifies which architectures are possible. The search strategy determines which candidates to explore. The performance-estimation strategy determines how to assess them affordably.

Search strategies include random sampling, evolutionary methods, reinforcement learning and differentiable approaches. Differentiable search can relax architectural choices into continuous quantities during search before selecting a discrete design.

Full training of every candidate can be expensive. Alternatives include shorter training runs, smaller proxy tasks, predictive estimates and sharing weights among candidate subnetworks. These shortcuts reduce cost but can distort the ranking compared with independent, full training.

Promising candidates need a final evaluation under the intended conditions. Keep a test set separate from repeated search decisions, measure deployment constraints on relevant hardware and report the total search effort. A favorable result without its budget and comparison setup is difficult to interpret.

Three practical examples

A mobile application searches for an image classifier that stays within a measured response-time limit on its target device.
A research team explores alternative connections in a repeated network block while keeping the overall task and training budget fixed.
An industrial system compares architectures under both prediction-error and memory constraints before choosing a deployable model.

Think of it this way

Think of an automated workshop testing different machine layouts from an approved catalog of components. The workshop can try more combinations than a person could conveniently inspect, but it still needs a design brief and a fair test.

The analogy stops short of implying exhaustive exploration. The search may sample only a small part of the possible layouts, and a quick prototype test can rank designs differently from a full production trial.

Worked example

Selecting under a latency constraint

Suppose a team evaluates three illustrative candidates on the same validation set and target device. Model A achieves 92% accuracy at 40 milliseconds, B achieves 91.5% at 20 milliseconds, and C achieves 90% at 15 milliseconds.

If the requirement is latency no greater than 25 milliseconds, A is infeasible despite its highest accuracy. B is the most accurate of the feasible candidates. C might still be preferred if its extra speed or lower resource use matters enough to the application.

Now suppose these accuracies came from short training runs. The ordering could change after full training, so the team should confirm the shortlisted designs. This example shows constrained selection, not an actual search experiment or a statistically established difference between B and C.

Illustrative case study

A fictional translation-tool company, Alder Notes, explores architectures for an on-device feature. The search initially rewards accuracy and estimated arithmetic operations. Its selected network meets the estimate but runs slowly on the intended hardware.

Engineers measure actual latency and discover that operation types and memory movement matter to the device's execution. They revise the evaluation objective, compare a simple search baseline and reserve data for final assessment.

The project records both search cost and deployment behavior. Management can then judge whether the resulting improvement justifies the experimentation expense. The team does not attribute every improvement to the search algorithm when the search space and training procedure also changed.

Common misconceptions and mistakes

NAS removes human design decisions. Humans still shape the space, objective and evaluation process.
The winning candidate is globally optimal. A finite search provides no such general guarantee.
Fewer arithmetic operations always mean lower latency. Hardware, implementation and memory behavior affect execution time.
A sophisticated strategy necessarily beats random search. Competitive baselines and matched budgets are needed to establish that claim.

Business use

NAS can be useful when model design strongly affects a valuable task and constraints make an off-the-shelf architecture inadequate. It is less compelling when the bottleneck is poor data, an unclear objective or a deployment process unrelated to network structure.

Budget for candidate evaluation, retraining, repeated runs and engineering integration, not only the final model. Ask whether the selected architecture remains useful after software or hardware changes. A credible comparison keeps data splits and training effort consistent and makes uncertainty visible when candidate differences are small. The business outcome is a better operating trade-off, not simply a more unusual network diagram.

Check your understanding

Answer: A search reports its best accuracy after examining the test set for every candidate.

Is that test result an independent estimate of the final model's performance?

Answer: No. The test set has influenced selection. A separate final evaluation is needed to assess performance without that repeated selection bias.

Related terms

Evolutionary algorithms
Validation set
Latency
AutoML
hyperparameter optimisation
search space
evolutionary algorithm
differentiable architecture search
weight sharing
Pareto frontier

Learn next

List three architectural choices for a small network and calculate how many combinations they allow. Define one quality measure and one deployment constraint. Compare a simple random search with any more elaborate strategy using the same evaluation budget.

Sources

Elsken, Metzen and Hutter, Neural Architecture Search: A Survey: https://jmlr.org/papers/v20/18-598.html Liu, Simonyan and Yang, DARTS: Differentiable Architecture Search: https://arxiv.org/abs/1806.09055 Li and Talwalkar, Random Search and Reproducibility for Neural Architecture Search: https://proceedings.mlr.press/v115/li20c.html Tan and colleagues, MnasNet: Platform-Aware Neural Architecture Search for Mobile: https://openaccess.thecvf.com/content_CVPR_2019/html/Tan_MnasNet_Platform-Aware_Neural_Architecture_Search_for_Mobile_CVPR_2019_paper.html

Previous: Neumann boundary condition Next: Neural autoregressive density estimator