Does internal validity mean testing inside the organization?

No. Internal refers to the credibility of the study's inference under its conditions, not the location of the test.

Is randomization always possible?

No. Other designs can be useful, but their assumptions and possible alternative explanations need careful treatment.

Can a study have strong internal validity and limited external validity?

Yes. A credible result for one setting may not generalize to different users, tasks, or operating conditions.

AI Glossary / Governance, ethics and safety / Intermediate

Internal validity

Definition

Internal validity is the extent to which a study supports a trustworthy conclusion about the relationship it tests, especially a claim that one factor caused an outcome. In AI evaluation, it asks whether the observed result can reasonably be attributed to the tested system or intervention rather than alternative explanations within the study.

What it means

Suppose employees using an AI assistant complete tasks faster than employees without it. The difference is interesting, but it does not automatically show that the assistant caused the improvement. Perhaps experienced employees chose to use it, or its users received easier tasks.

Internal validity concerns the credibility of the inference under the study's own conditions. It differs from external validity, which concerns whether findings transfer to other people, settings, or circumstances. It also differs from construct validity, which asks whether a measurement captures the concept it claims to measure.

These distinctions matter because a precise number can answer the wrong question, reflect an unfair comparison, or fail to travel beyond its original setting.

How it works

A study begins with a clear question and an explicit comparison. Investigators identify other factors that could explain the outcome, then use study design and analysis to address them. Relevant factors may include experience, task difficulty, changing workloads, different instructions, or unequal access to support.

Random assignment can help make comparison groups similar on average, but it does not guarantee that every study is valid. Participants may drop out unevenly, use tools assigned to the other group, or receive inconsistent measurements. Studies without random assignment require particularly careful justification of their comparison and assumptions.

For an AI claim, preserve details of the model, workflow, tasks, participants, and outcome measurement. Report uncertainty and deviations from the planned procedure. Internal validity is strengthened by ruling out plausible rival explanations; it is not a badge awarded merely because an experiment includes a control group.

Three practical examples

1. A company compares AI-assisted document drafting with its existing workflow while assigning comparable work and recording review effort, rather than comparing different departments with different responsibilities.

2. A training study checks whether improved performance follows the new AI tutor or simply reflects participants taking the same assessment repeatedly.

3. A customer-support pilot records a simultaneous staffing change, because faster responses cannot automatically be credited entirely to the new AI routing system.

Think of it this way

Think of testing two recipes while changing both the ingredients and the oven temperature. If one cake turns out better, you have not isolated which change produced the difference.

The analogy is limited. Human and organisational studies contain many interacting influences, and some cannot be held constant or randomly assigned. The aim is a defensible explanation of the observed relationship, with stated assumptions, rather than pretending that every business process can be reduced to a perfectly controlled kitchen test.

Worked example

A fictional team reports that AI users average eight minutes per task while nonusers average fourteen minutes. The apparent saving is six minutes. Inspecting the assignment reveals that AI users handled 80 easy tasks and 20 difficult tasks, while nonusers handled 20 easy tasks and 80 difficult tasks.

Suppose both groups average six minutes on easy tasks and sixteen minutes on difficult tasks. The AI group's average is (80 × 6 + 20 × 16) / 100 = 8 minutes. The other group's average is (20 × 6 + 80 × 16) / 100 = 14 minutes.

These illustrative numbers produce a six-minute aggregate difference despite identical performance within each task type. The difference comes entirely from the mix of tasks in this simplified example, not from a within-type AI improvement.

Before attributing any observed saving to AI, the team needs a comparison that addresses task difficulty and other relevant differences.

Illustrative case study

Fictional case: Fern Analytics introduces an assistant to a volunteer group of experienced analysts. Their reports require fewer revisions than reports from a comparison group containing many new starters. Management initially treats the result as evidence that the assistant improves quality.

The evaluation team notes that experience and volunteering may explain some of the difference. It redesigns the pilot to compare participants more fairly, defines the revision measure in advance, and tracks whether reviewers know which workflow produced each report.

This invented case offers no claimed effect size. A better design can make the evidence more informative, but its findings still apply first to the tasks and participants actually studied. Wider deployment requires additional consideration of external validity.

Common misconceptions and mistakes

A large sample does not automatically remove confounding. It may estimate a biased comparison very precisely. Statistical significance also does not establish that the tested intervention caused the result.

Internal validity is not the same as repeatability: repeating the same flawed comparison can reproduce the same misleading conclusion. Adjusting for recorded variables may help, but it does not prove that all relevant differences were measured or handled correctly. Transparent limitations are part of credible evaluation, not evidence that the study has no value.

Business use

Businesses can use internal-validity questions when reviewing productivity claims, procurement pilots, and AI safety interventions. Ask what changed, what comparison was used, and what other explanation could produce the result.

Design the evaluation before making a success claim. Keep outcome definitions stable, record implementation changes, and examine missing observations. Where a causal conclusion cannot be supported, report a descriptive association instead. This helps decision-makers understand what the evidence establishes and what remains uncertain before committing to a broader rollout.

Check your understanding

AI users handle easier cases and finish faster. Does the speed difference establish an AI benefit?

Answer: No. Case difficulty is an alternative explanation. A defensible comparison must address that difference before making a causal claim about the assistant.

Related terms

Learn next

Next, examine confounding and external validity. For an AI pilot, list three plausible explanations for the reported outcome besides the tool itself, then identify what evidence or design feature would help distinguish them.

Sources

NIST AI RMF Playbook, Measure 2.5: https://airc.nist.gov/airmf-resources/playbook/measure/ NIST GCR 26-069, A Possible Approach for Evaluating AI Standards Development: https://nvlpubs.nist.gov/nistpubs/gcr/2026/NIST.GCR.26-069.pdf Relevant validity and evaluation passages were read. The business scenarios and arithmetic illustration are original.

Previous: Internal state Next: Interoperability