AI Glossary / Causal inference and experimentation / Intermediate
Computer experiment
Definition
A computer experiment is a planned study that runs a computational model at selected input settings and analyses its outputs. It is used to learn about a simulator's behavior, compare scenarios, estimate sensitivities or build a cheaper approximation to an expensive calculation.
What it means
The experiment generates observations by executing code. Examples include a fluid-flow simulation, a manufacturing-process model or a stochastic queue simulator. Inputs might be physical parameters, operating settings or uncertain environmental conditions.
Some computer models are deterministic: identical inputs and configuration produce the same output. Others include randomness and can produce different outputs across runs. The experimental design must account for that distinction.
A computer experiment studies the implemented model. It does not automatically establish what would happen in the physical or social system the model is intended to represent. Validation against relevant external evidence remains a separate task.
How it works
Define the question, input ranges, outputs and computational budget. Select input combinations using a design appropriate to the goal, such as broad coverage of the input space or sequential selection around a region of interest. Record software version, configuration and any random seeds.
Run the model, check failures and analyze the resulting input - output relationships. An emulator or surrogate model can approximate expensive outputs, but should be tested on runs not used for fitting. For stochastic simulations, replication helps quantify simulation variability. Assess sensitivity, numerical error and model discrepancy separately where relevant, rather than combining every source of uncertainty into one unexplained interval.
Three practical examples
- An engineering team varies geometry and flow conditions in a computational simulator to study pressure loss.
- A service planner runs a queue simulation under different staffing and arrival-rate scenarios.
- A researcher builds a surrogate model of an expensive environmental simulation to explore many candidate settings.
Think of it this way
Think of a virtual laboratory where the apparatus is a computer model. You choose settings, run trials and record responses. The analogy is useful for planning and reproducibility, but the virtual apparatus contains assumptions and numerical approximations. Repeating its results faithfully is not the same as showing that it faithfully represents the real world.
Worked example
Suppose a deterministic simulator takes temperature and feed rate as inputs, and one run costs 20 minutes. A full grid with 20 settings for each input would require 400 runs, or 8,000 minutes of serial computation.
The team instead budgets 40 carefully selected runs across the region and 10 additional runs for checking an emulator. That is 50 runs, or 1,000 minutes before considering parallel execution and overhead. The emulator then predicts outputs at additional settings cheaply.
This saves simulator calls only if the approximation is adequate for the intended question. The ten check runs assess emulator error within the sampled region; they do not validate the simulator against physical reality. The numbers are illustrative, and a space-filling design or sequential strategy must be chosen according to the actual input dimension and response behavior.
Illustrative case study
Fictional case: Cypress Warehousing builds a simulation to compare staffing schedules. The first experiment uses a single random seed for each schedule, and one schedule appears best because it happened to receive an easier arrival pattern.
The team adds replication and a deliberate scenario-comparison strategy, records seeds and checks the simulator against observed queue behavior. It separates uncertainty from random arrivals, uncertainty in estimated inputs and limitations of the model itself. The final report explains which comparisons are supported. This case illustrates experimental design for stochastic code, not a claim that any particular staffing schedule improves real operations.
Common misconceptions and mistakes
A computer experiment is not merely any activity performed on a computer. A survey delivered online is not automatically a simulation experiment in this technical sense. Deterministic runs do not gain new information from identical replication unless numerical or system behavior is being investigated.
Do not confuse emulator validation with simulator validation. A surrogate can reproduce an inaccurate simulator very well. Changing many inputs without a plan can make conclusions difficult to interpret. Extrapolation outside the tested region, failed runs and unrecorded configuration changes can also undermine the analysis.
Business use
Computer experiments can reduce the cost of exploring systems that are expensive, slow or impractical to test physically. They support design exploration and scenario comparison when a suitable computational model exists.
The business case should include simulator credibility, run cost and the decision consequences of approximation error. Maintain a reproducible run log with inputs, outputs, versions and failure reasons. Use an experiment design that serves the actual objective: global understanding, local optimization and rare-event estimation can require different strategies. Present conclusions as model-based evidence and explain what external checks are needed before relying on them operationally.
People also ask
Must the model include randomness?
No. Many classic computer experiments study deterministic codes.
Is an emulator the same as the simulator?
No. It approximates the simulator's input - output behavior and introduces its own error.
Can simulated results establish a real-world causal effect by themselves?
Only under additional justified links between the model and the real system; simulation alone does not supply those links.
Check your understanding
An emulator accurately predicts outputs from a simulator on held-out runs. Does that establish that the simulator predicts the physical system accurately?
Answer: No. It supports the emulator's fidelity to the simulator. The simulator still needs validation against suitable physical observations or other credible evidence.
Related terms
- Design of experiments
- Surrogate model
- Sensitivity analysis
- Latin hypercube sampling
- Model validation
- Simulation
- emulator
- numerical error
Learn next
Learn the difference between deterministic and stochastic simulations, then design a small input - output study with a fixed budget. Reserve runs for checking the approximation and list which uncertainties come from the simulator, its inputs and any surrogate model.
More in Causal inference and experimentation
- Citation recall - Citation recall measures how much of an answer's evaluated content is supported by the citations provided, under a defined claim or statement-level protocol.
- Cohen's kappa - Cohen's kappa is a statistic measuring agreement between two raters who assign categorical labels to the same items, adjusted for agreement expected under a...
- Collider - A collider is a variable on a causal path where two arrowheads meet, as in X→Z←Y.
- Computerized adaptive testing - Computerized adaptive testing (CAT) is assessment in which a computer selects later questions using information from a person's earlier responses.
- Conditional average treatment effect - The conditional average treatment effect, or CATE, is the average causal effect of a treatment among units with specified characteristics.
- Confounder - A confounder is a variable that, in the relevant causal structure, helps create a noncausal association between an exposure or intervention and an outcome.
Sources
Jerome Sacks, William J. Welch, Toby J. Mitchell and Henry P. Wynn, Design and Analysis of Computer Experiments: https://doi.org/10.1214/ss/1177012413 Designs for Computer Experiments: https://www.tandfonline.com/doi/abs/10.1080/00401706.1989.10488474 Primary publication excerpts consulted; examples are original. Full source and experimental-design editorial review remain pending.
Previous: Computer cluster Next: Computer vision