AI Glossary / Governance, ethics and safety / Intermediate
Mixed methods
Definition
Mixed methods is a research approach that deliberately integrates quantitative and qualitative evidence to answer a question. It combines numerical analysis with systematic study of meanings, experiences or processes, so that the findings inform one another.
What it means
An AI evaluation may show that task completion fell after a redesign without explaining why. Interviews or observations may reveal a confusing handoff. Conversely, interviews can identify a concern whose prevalence needs broader numerical investigation.
The defining feature is integration. Adding a few quotations beside a dashboard does not automatically create a rigorous mixed-methods study. Each component needs an appropriate design, and the relationship between them should help answer the research question.
The approach is useful when a single kind of evidence leaves an important gap. It is not inherently superior for every problem, and combining two weak methods does not make their conclusions strong.
How it works
Start with a question that explains why both kinds of evidence are needed. Decide whether one component will inform the design of the other, whether both will proceed together, and where their results will be combined.
An exploratory sequence might use interviews to identify relevant dimensions before developing a survey. An explanatory sequence might use interviews to investigate an unexpected numerical result. A convergent design might compare findings collected through both approaches during a similar period.
Analyze each component with suitable methods, then examine agreement, differences and unanswered questions. Do not force the findings to tell one convenient story. A disagreement may reveal different populations, measurement problems or a process that varies by context. Report what the integration adds and which limits remain.
Three practical examples
- A product team combines task-success measurements with observed user sessions to understand where an AI interface creates confusion.
- A workplace study uses interviews to develop a survey about how staff actually review generated answers.
- An evaluation compares model-error patterns with structured feedback from affected users to investigate consequences that the benchmark does not capture.
Think of it this way
Think of understanding traffic through both vehicle counts and observations of how people cross a junction. Counts show patterns of volume, while observation may reveal how those patterns arise and affect people.
The analogy illustrates complementary evidence. It does not mean that every interview explains a numerical trend or that a few observations establish how common a behavior is.
Worked example
Explaining a difference without overstating it
In an invented usability evaluation, 80 of 100 experienced users complete an AI-assisted task, compared with 50 of 100 new users. The numerical difference is 30 percentage points.
The team then observes a purposively selected set of new-user sessions and finds that several participants misunderstand an approval icon. This supplies a plausible explanation to investigate, not a statistical estimate that a particular proportion of all new users has that problem.
The team redesigns the icon and plans a suitable follow-up evaluation. The mixed-methods contribution is the connection between a measured performance difference and a systematically investigated interaction issue. It does not establish that the icon explains the entire gap or that the redesign has already solved it.
Illustrative case study
Fictional company Alder Learning evaluates an AI study assistant. Usage logs suggest that learners often leave immediately after a generated explanation, and the team initially interprets this as successful completion.
Interviews and observed sessions reveal two different reasons for leaving: some learners understand the answer, while others give up because they cannot verify it. The researchers revise the interpretation of the log metric and design follow-up questions that distinguish these experiences.
The integrated result changes what the organization measures. It does not simply add colorful anecdotes to the original success claim or assume that the interview sample represents every learner.
Common misconceptions and mistakes
- Mixed methods means using two numerical techniques. In the usage here, qualitative and quantitative components are intentionally integrated.
- Interviews automatically explain a statistical association. Their sampling, analysis and connection to the result require justification.
- Conflicting findings indicate failure. They may expose an important limitation or contextual difference.
- A quotation proves prevalence. Qualitative evidence can illuminate experiences without estimating how common they are.
Business use
Mixed methods can help businesses evaluate AI as part of a human workflow. Numerical performance measures may miss how people interpret outputs, adapt their work or experience consequences. Qualitative evidence can identify those issues and improve the questions asked quantitatively.
Plan for the expertise and time each component requires. State how participants are selected and how the findings will be integrated before treating the approach as an evaluation deliverable. The resulting report should distinguish what is measured, what is interpreted and what remains a hypothesis for further study.
People also ask
Does every AI evaluation need mixed methods?
No. The method should fit the question and the evidence needed.
Must the two components have equal sample sizes?
No. Sampling follows each component's purpose, and numerical equality is not a measure of quality.
Can findings disagree?
Yes. Investigating disagreement is part of integration, rather than a reason to discard whichever result is less convenient.
Check your understanding
A report presents a satisfaction percentage and several interview quotations but never relates them. What is missing from a mixed-methods claim?
Answer: A clear integration process showing how the two components jointly answer the question, including how agreements and differences are interpreted.
Related terms
- Usability testing
- Qualitative research
- quantitative research
- triangulation
- research design
- stakeholder engagement
Learn next
Read research design and construct validity. Choose one AI evaluation question and state what numerical evidence could answer, what it could not answer, and how a qualitative component would address that gap.
More in Governance, ethics and safety
- Malware - Malware is software or related executable logic deliberately designed to perform harmful or unauthorized activity.
- Measurement Bias - Measurement bias is systematic distortion in how a quantity is recorded or represented.
- Membership inference attack - A membership inference attack attempts to determine whether a particular record was included in a machine-learning model's training data.
- Mode Confusion Bias - Mode confusion bias occurs when a person misunderstands a system's active operating mode and takes an action that would be appropriate in a different mode.
- Model Governance - Model governance is the set of responsibilities, policies and decision processes used to oversee models throughout their lifecycle.
- Model Interpretability - Model interpretability is the extent to which people can understand relevant aspects of a model's behavior and the meaning of its outputs.
Sources
NIH Office of Behavioral and Social Sciences Research, Best Practices for Mixed Methods Research in the Health Sciences: https://obssr.od.nih.gov/research-resources/mixed-methods-research. Fetters, Curry and Creswell, Achieving Integration in Mixed Methods Designs: https://pmc.ncbi.nlm.nih.gov/articles/PMC4097839/. These methodological sources inform the definition; the AI studies and numbers here are original illustrations.
Previous: Mixed integer linear programming Next: Mixed model designs