Must an observational study use old records?

No. Researchers can collect new data prospectively while leaving exposure assignment outside their control.

Does random sampling make it a randomized experiment?

No. Sampling people and assigning an intervention are different operations.

Does adjustment prove causation?

No. It depends on appropriate variables, adequate measurement, valid methods, and other identifying assumptions.

AI Glossary / Statistics and probability / Intermediate

Observational study

Definition

An observational study investigates people, organizations, events, or other units without the investigator assigning the exposure or intervention being studied. Researchers observe existing conditions and outcomes, sometimes prospectively and sometimes using previously collected records. Such studies can describe populations, examine associations, and support carefully qualified causal investigations. Their defining feature is the absence of investigator-controlled exposure assignment, not simply a large dataset or the use of historical information.

What it means

Suppose employees decide for themselves whether to use an AI assistant. Comparing users with nonusers is observational if the researcher does not assign that use. The groups may differ in experience, motivation, workload, or other factors before the comparison begins.

Those differences matter when interpreting results. A relationship between tool use and productivity may reflect an effect of the tool, pre-existing differences, or both. Observational evidence can be valuable, but its causal interpretation depends on design, knowledge of the setting, and assumptions that the data alone may not verify.

How it works

Define the population, exposure, outcome, time period, and question. Select an appropriate design, such as following a cohort, comparing cases and controls, or measuring a population at one point in time. Establish eligibility and measurement rules before interpreting results.

For descriptive questions, address coverage, sampling, and measurement quality. For causal questions, specify the intervention comparison that the analysis aims to approximate. Identify plausible confounders using substantive knowledge, and choose methods such as standardization, matching, weighting, or outcome modeling when their assumptions fit.

Check temporal ordering, overlap between comparison groups, missing data, and selection processes. Adjustment cannot reliably remove confounding by factors that were not measured adequately, and adjusting for the wrong variables can introduce bias. Report uncertainty and investigate sensitivity to assumptions. Separate the observed association from any additional reasoning used to interpret it as an effect.

Three practical examples

A company studies whether employees who voluntarily adopt an AI tool complete more tasks than employees who do not.
Researchers follow organizations over time to describe changes in automation use and staffing patterns.
A service team analyzes existing support records to investigate associations between request characteristics and resolution time.

Think of it this way

Think of watching runners who choose different training programs. Their finishing times may reveal useful patterns, but the runners did not begin as interchangeable groups. Experience, health, motivation, and access to coaching can influence both their program choices and their results.

Worked example

Consider a fictional dataset of 100 AI-tool users and 100 nonusers. Among users, 80 are experienced and 20 are beginners. Among nonusers, 20 are experienced and 80 are beginners. Suppose experienced employees average 90 completed tasks and beginners average 50, within both tool-use groups.

The users' overall mean is (80 × 90 + 20 × 50)/100 = 82 tasks. The nonusers' mean is (20 × 90 + 80 × 50)/100 = 58 tasks. The crude difference is 24 tasks, despite no difference within either experience group.

Standardizing both groups to a population containing equal proportions of experienced employees and beginners gives 0.5 × 90 + 0.5 × 50 = 70 tasks for each. This constructed example demonstrates how group composition can explain an association. In real data, adjustment for experience alone would not establish that all confounding had been removed.

Illustrative case study

Fictional case study: Maple Operations wants to evaluate an optional writing assistant using routine productivity records. Its first report credits the assistant with every observed difference between adopters and non-adopters. A review finds that early adopters had more training and different responsibilities.

The team defines a clearer comparison, collects relevant baseline information, and examines whether comparable employees exist in both groups. It reports adjusted associations with limitations and considers a randomized pilot for a later rollout. This fictional case illustrates improved study reasoning; it does not claim that the tool helps, harms, or has no effect.

Common misconceptions and mistakes

A common mistake is treating a large observational dataset as if size alone makes its comparisons unbiased. More records can narrow statistical intervals around a biased estimate.

Another mistake is believing that observational studies are always incapable of informing causal questions. Strong designs and justified assumptions can support useful causal inference, though those assumptions need explicit scrutiny. Conversely, adding many control variables does not automatically solve confounding. Variables measured after exposure, selection criteria, measurement errors, and missing information can complicate the analysis. Prediction accuracy also does not establish the effect of changing an exposure.

Business use

Observational studies help businesses understand adoption, customer behavior, operational performance, and patterns that cannot readily be studied in experiments. They can identify questions for subsequent testing and describe conditions across real operating environments.

AI-assisted analysis should keep design metadata visible: who entered the dataset, when exposure and outcomes were measured, and which assumptions support a comparison. Generated reports should distinguish descriptive findings, adjusted associations, and causal claims rather than silently upgrading one into another. Decision makers also need the practical size and uncertainty of the finding.

Check your understanding

Volunteers who use a learning app score higher than nonusers. What additional issue must be considered before attributing the difference to the app?

Answer: The groups may differ before app use, for example in motivation or prior knowledge. Those differences can affect both adoption and scores.

Related terms

Confounding
Cohort study
Case-control study
Causal inference
Selection bias
Standardization
Propensity scores
cross-sectional study
randomized experiment
propensity score
exchangeability

Learn next

Study causal diagrams and the distinction between sampling and treatment assignment. Then learn how specific observational designs address confounding and selection, and practice stating the assumptions required for an adjusted association to answer a causal question.

Confounding

Sources

Frank Harrell's Biostatistics Glossary defines observational studies and highlights confounding: https://hbiostat.org/glossary/

Hernán and Robins, Estimating causal effects from epidemiological data, explains exchangeability and causal interpretation in its published abstract: https://pubmed.ncbi.nlm.nih.gov/16790829/

The numerical example and case study are original; they illustrate study-design reasoning rather than a real intervention effect.

Previous: Observability (software) Next: Observed frequencies