AI Glossary / Causal inference and experimentation / Intermediate
Computerized adaptive testing
Definition
Computerized adaptive testing (CAT) is assessment in which a computer selects later questions using information from a person's earlier responses. The aim is to measure a specified ability or trait efficiently while meeting the test's content and administration requirements.
What it means
People need not receive identical questions to be measured on a common scale. A carefully calibrated item bank can support different question sequences whose responses are interpreted through the same measurement model.
The familiar description, “correct answers lead to harder questions,” captures only part of the idea. Actual selection can consider uncertainty, content coverage, item exposure and other constraints. A question can be useful because it distinguishes plausible ability levels, not simply because it is difficult.
CAT is an assessment method, not automatically a teaching system. A learning platform may combine assessment with instruction, but selecting an informative test question and selecting a helpful lesson are different decisions. Generating questions with AI also does not establish that they are calibrated or suitable for adaptive scoring.
How it works
A typical model-based CAT begins with an initial ability estimate and a bank of items whose characteristics have been estimated. The system presents an eligible item, records the response, updates the estimate and selects another item.
Item response theory often supplies the relationship between ability and response probability. A selection rule can favor items providing information near the current estimate, subject to the test blueprint. A stopping rule determines when the test ends: for example, after a fixed length or after specified precision requirements are met within length limits.
Scoring, selection and stopping must work together. A test that exhausts its eligible bank before reaching its target should record that outcome rather than silently describe the target as achieved.
Three practical examples
- A language placement service selects questions around a learner's estimated proficiency while covering required language skills.
- A staff training program measures competence using different item sequences drawn from a calibrated assessment bank.
- A research questionnaire chooses informative items to estimate a defined latent trait with an explicit uncertainty target.
Think of it this way
Think of locating a point on a map by asking increasingly useful questions. Once the region is reasonably clear, another question about a distant region may add little.
Unlike a simple guessing game, a test answer is uncertain: a person can miss a familiar item or answer an unfamiliar one correctly. CAT therefore updates a statistical estimate rather than treating every response as a perfect yes-or-no boundary.
Worked example
Choosing between candidate questions
Consider a teaching example using a Rasch model with logistic scale factor 1. The current ability estimate is theta = 0. A candidate item's correct-response probability is p = 1/(1 + exp(b − theta)), where b is its difficulty. Its information at this estimate is p(1 − p).
For item A, b = 0. Then p = 0.5 and information is 0.25. For item B, b = 2. Then p is approximately 0.1192 and information approximately 0.1050.
If both items are eligible and information alone determines selection, A is preferred. B is harder, but less informative at the current estimate under this model. Other constraints could still make A unavailable.
After the answer, the ability estimate is recomputed using the responses and the chosen estimation method. It is not increased by an arbitrary fixed amount merely because A was answered correctly. The numbers illustrate a single selection step, not a complete validated test or a prediction about a real learner.
Illustrative case study
Fictional case: a placement service
A fictional learning center considers replacing one fixed questionnaire with adaptive placement. Its first demonstration appears efficient, but the content specialist notices that some learners receive almost no questions from one required skill area.
The team adds content requirements to the selection process and simulates different response patterns before any operational decision. It also examines learners near placement boundaries and checks whether the item bank offers useful information at the ends of the scale.
The center reports uncertainty alongside placement recommendations and provides a review route. The case illustrates why reducing question count is an incomplete success measure; the assessment must still support its intended interpretation.
Common misconceptions and mistakes
A difficult final question is not a score by itself. The response history, item characteristics and scoring model matter. Raw percentages from different adaptive sequences may not be directly comparable.
Adaptive does not mean automatically fair. Miscalibrated items, inaccessible presentation or a poorly represented group can undermine interpretation even when the selection algorithm runs correctly.
Stopping early does not prove mastery. The stopping condition might concern statistical precision, maximum length or administrative limits. The report should distinguish these reasons. Likewise, changing an item with generative AI can alter its measurement properties and requires evaluation rather than inheriting the original calibration.
Business use
For a business assessment, define what decision the score supports before choosing an algorithm. Document the item bank, calibration evidence, content blueprint, scoring model, precision requirements and stopping rules.
Monitor item exposure and the distribution of administered content. Evaluate whether the available questions support the full intended proficiency range. Keep version records so an item edit or scoring change can be traced to affected results. Efficiency should be judged alongside validity, accessibility and the consequences of incorrect placement.
People also ask
Does CAT require a large language model?
No. Adaptive testing can use established statistical measurement models without generative AI.
Will everyone answer the same number of questions?
Not necessarily. Some designs have fixed lengths; others use variable lengths with explicit stopping rules.
Can two people with the same percentage correct receive different estimates?
Yes. When they answered different calibrated items, raw percentages alone do not capture the model-based evidence.
Check your understanding
At the current estimate, eligible item X provides information 0.22 and eligible item Y provides 0.12. If the rule maximizes information and no other constraints distinguish them, which is selected?
Answer: X. This does not imply that X is the harder item or that its answer alone determines the final score.
Related terms
- Item response theory
- Rasch model
- Adaptive learning
- Measurement error
- item information
- latent trait
- experimental design
Learn next
Read item response theory, then compare a fixed-length CAT with a precision-based stopping rule. Examine how content balancing changes the choice that an information-only rule would make.
More in Causal inference and experimentation
- Cohen's kappa - Cohen's kappa is a statistic measuring agreement between two raters who assign categorical labels to the same items, adjusted for agreement expected under a...
- Collider - A collider is a variable on a causal path where two arrowheads meet, as in X→Z←Y.
- Computer experiment - A computer experiment is a planned study that runs a computational model at selected input settings and analyses its outputs.
- Conditional average treatment effect - The conditional average treatment effect, or CATE, is the average causal effect of a treatment among units with specified characteristics.
- Confounder - A confounder is a variable that, in the relevant causal structure, helps create a noncausal association between an exposure or intervention and an outcome.
- Cooperative inverse reinforcement learning - Cooperative inverse reinforcement learning (CIRL) is a framework in which a human and an artificial agent share an objective, but the agent is uncertain...
Sources
Technical source: R. J. de Ayala, The Theory and Practice of Item Response Theory, second-edition online appendices, Appendix D on computerized adaptive testing: https://cehs.unl.edu/sites/unl.edu.cehs/files/media/file/de_Ayala_Appendices_2ndEd.pdf
The numerical item-selection calculation and learning-centre case are original illustrations.
Previous: Computer vision dazzle Next: Concentration inequality