Is IRT one algorithm?

No. It is a family of models and associated estimation methods, including binary, polytomous and multidimensional approaches.

Does theta have a natural universal unit?

No. Its numerical scale depends on model and identification choices; comparing different scales requires appropriate linking.

Can a high-ability respondent miss an easy item?

Yes. IRT generally models probabilities rather than making every response deterministic.

AI Glossary / Causal inference and experimentation / Intermediate

Item response theory

Definition

Item response theory (IRT) is a family of statistical models that relates responses to individual questions or tasks to an unobserved ability or trait and to characteristics of those items. It supports measurement beyond simply counting correct answers.

What it means

A test score is evidence about something that is not directly observed, such as proficiency in a defined skill. IRT represents that proficiency as a latent variable, often written theta, and describes how it affects response probabilities.

Items need not behave identically. Some are easier, some distinguish nearby proficiency levels more sharply, and some require models allowing partial credit or ordered response categories. The chosen model must match the response format and the intended measurement interpretation.

In AI-related work, IRT can inform adaptive assessment and analysis of benchmark items. Applying the mathematics to a collection of models or people does not automatically establish that a single underlying ability explains their performance. That is a modeling question requiring evidence.

How it works

For a simple binary-response model, the item characteristic curve gives the probability of a correct response at each ability level. A two-parameter logistic form, using scale factor 1, is p = 1/(1 + exp(−a(theta − b))). Here b is difficulty and positive a controls discrimination.

Calibration estimates item characteristics from suitable response data. Scoring then estimates a respondent's latent position using the administered items and their responses. The scale requires identification conventions, and different calibrations need appropriate linking before their numbers are treated as comparable.

Common models assume responses are independent after conditioning on the relevant latent trait or traits. Shared passages, repeated cues or dependencies between tasks can challenge that assumption. Model fit and measurement interpretation must therefore be evaluated rather than inferred from a successful numerical fit alone.

Three practical examples

An adaptive assessment selects questions whose information is useful near a learner's current estimated proficiency.
A test developer investigates items that provide little discrimination in the intended assessment range.
An AI evaluation researcher examines whether a benchmark's item responses support a proposed latent performance scale.

Think of it this way

Think of weighing an object with several instruments whose sensitivity and operating ranges differ. Understanding the instruments helps interpret their readings.

A test item is not a literal physical scale, and ability is not directly observable mass. The analogy explains why item characteristics matter; it does not establish that the psychological or performance construct has been measured validly.

Worked example

Reading a probability curve

Use the two-parameter logistic model with a = 1 and b = 0. At theta = 0, the exponent is zero, giving p = 1/(1 + 1) = 0.5.

At theta = ln(3), approximately 1.0986, exp(−theta) = 1/3. The probability becomes 1/(1 + 1/3) = 0.75. This is a probability of a correct response under the model, not a statement that the person knows 75 percent of the subject.

Now keep theta = 1.0986 but increase the item's difficulty to b = 1.0986. The difference theta − b becomes zero, returning the probability to 0.5. The ability number alone cannot determine success probability without the item parameters.

If 100 independent respondents all had this same theta and answered the first item under the model, the expected correct count would be 75. The actual count could differ. Repeating the same item with one person would not automatically meet the independence and unchanged-response conditions, because memory or learning might alter the process.

Illustrative case study

Fictional case: reviewing a skills assessment

A fictional training provider has two versions of a skills test and wants to compare their results. Simply using the percentage correct would ignore differences in question difficulty.

The assessment team collects appropriate calibration evidence and considers a linked IRT scale. During review, it finds that several items share wording that reveals another item's answer. It investigates those dependencies before relying on the fitted scale.

The team also asks whether the score supports the intended placement decision, rather than treating a latent-variable estimate as a complete description of competence. Its final report identifies the model, linking method and uncertainty. This fictional process does not assume that calibration alone validates the assessment.

Common misconceptions and mistakes

Difficulty is a model parameter, not simply the percentage of one sample that answered incorrectly. Sample composition affects raw response rates.

Discrimination describes how sharply response probability changes with the latent variable in the model. It does not mean social discrimination, although fairness questions can arise when items behave differently across groups after accounting for the intended trait.

IRT does not guarantee that item parameters are invariant in every population or administration. Such claims depend on model adequacy and comparable conditions. Nor does a single score reveal every strength and weakness; a unidimensional summary may hide relevant differences.

Business use

Use IRT when an organization needs a defensible measurement model for item-level responses, especially across different question sets. Begin with a clearly defined construct and a suitable response-data collection plan.

Review parameter uncertainty, model fit and the intended score interpretation. For an AI benchmark, investigate whether performance differences reflect one dimension or several, and whether item dependence or exposure affects responses. Avoid promoting a fitted ability number into a universal intelligence measure. Explain what the scale covers, how it was calibrated and where comparisons are justified.

Check your understanding

In the stated logistic model with positive discrimination, what is the correct-response probability when theta equals b?

Answer: One half, because theta − b is zero. This particular result belongs to the stated two-parameter form; adding other parameters, such as a lower asymptote, changes the interpretation.

Related terms

Computerized adaptive testing
Rasch model
Latent variable
item characteristic curve
discrimination parameter
test information
psychometrics

Learn next

Plot item curves with different difficulties and discrimination values. Then compare how a binary model differs from a partial-credit model, and examine why dependence between items changes measurement interpretation.

Sources

Technical reference: R. J. de Ayala, The Theory and Practice of Item Response Theory, second-edition appendices, including logistic-model development and adaptive testing: https://cehs.unl.edu/sites/unl.edu.cehs/files/media/file/de_Ayala_Appendices_2ndEd.pdf

The probability calculations and training-provider case are original teaching examples.

Previous: Item matrix Next: Item stem and leaf plot