AI Glossary / Causal inference and experimentation / Intermediate
Plackett-Luce model
Definition
The Plackett - Luce model is a probability model for rankings that assigns each item a positive worth and represents an ordering as successive choices from the remaining items. At each step, an item's chance of selection is proportional to its worth.
What it means
A ranking tells us that one option was placed ahead of another, without requiring a numerical rating for every option. The Plackett - Luce model connects many such observations to a compact set of relative worth parameters.
Worth is a statistical parameter, not an intrinsic measure of quality. It reflects the fitted model and the data collection setting. A preferred answer, product or design may still fail other requirements that the ranking question did not cover.
The basic model describes strict rankings without ties. Extensions can accommodate ties and other structures, but those extensions must be specified. An unranked option is also not automatically a disliked option: it may simply have been absent from the comparison.
How it works
Suppose the available items have positive worths w1 through wk. The probability of selecting item i first is wi divided by the sum of all available worths. Remove that item, repeat the same calculation on the remaining set, and multiply the successive probabilities to obtain the probability of a complete ranking.
The worths are usually estimated from observed rankings. Multiplying every worth by the same positive constant changes no probability, so estimation needs a scale convention, such as normalizing the worths to sum to one.
The choice rule imposes a strong restriction: the ratio of selection probabilities for two available items depends on their worth ratio, not on the other alternatives. Whether this is a reasonable approximation must be assessed for the application.
Three practical examples
- A product researcher models ordered preferences among several designs shown together.
- An AI evaluation team analyses rankings of candidate answers under a defined quality question.
- A recommendation researcher uses observed orderings to estimate relative item preferences, while checking whether user groups require different models.
Think of it this way
Think of drawing labeled tickets from a weighted collection, recording a winning label and removing all tickets with that label before the next draw. More tickets make a label more likely to be selected sooner.
This is an illustration of the model's probability rule, not an explanation of human psychology. A person's choice may depend on context or similarities between options in ways that fixed ticket counts cannot represent.
Worked example
Calculating one ranking probability
Give three items worths A = 3, B = 2 and C = 1. Their first-place probabilities are 3/6, 2/6 and 1/6.
For the ranking A ahead of B ahead of C, first select A with probability 3/6. Among the remaining B and C, select B with probability 2/3. C is then the only item left, with probability 1. Multiplying gives (3/6) × (2/3) × 1 = 1/3.
For A ahead of C ahead of B, the probability is (3/6) × (1/3) = 1/6. These two mutually exclusive rankings exhaust the ways A can be first, and their probabilities sum to 1/2, matching A's first-place probability.
If only A and B are available, A's selection probability is 3/5. Adding C lowers it to 3/6, but the ratio of A's to B's first-place probability remains 3/2. The model fixes this ratio, not each option's absolute probability. Doubling all three worths gives the same results.
Illustrative case study
Fictional case: ranking answer styles
A fictional service asks participants to rank three answer styles for clarity. Its analyst fits a Plackett - Luce model and reports relative worth estimates with uncertainty.
Before recommending one style, the team checks the task wording. Participants may have ranked brevity while the product team intended completeness. It also separates options that were never shown from options that were shown and placed last.
When experienced and novice users show different patterns, the team investigates whether one pooled model is adequate. The resulting analysis is described as evidence about the specified clarity judgments, not a universal ranking of answer correctness or usefulness.
Common misconceptions and mistakes
Worths are relative. A fitted worth of 0.6 after normalization does not mean an item is 60 percent correct or delivers 60 percent more business value. In the basic model, it can describe first-place probability when the normalization is over exactly the full available set.
The model does not force the highest-worth item to win every ranking. It assigns probabilities to many possible orders.
Ties should not be broken arbitrarily without considering the effect on inference. Similarly, ranking only a shown subset differs from identifying the top items from a larger known set. Those observation processes imply different information and must be encoded accordingly.
Business use
Use the model when ranking data are meaningful and a relative-worth summary serves the decision. Record which options were available, how ties were treated and what criterion participants ranked.
Examine whether the comparison network supplies enough connections to support estimation. Disconnected or one-sided evidence can cause estimation problems; regularization may help computation but does not create observed comparisons. Check sensitivity to user groups and choice-set composition before turning fitted worths into a product policy. Retain the distinction between preference evidence and independently assessed factual or operational requirements.
People also ask
Is it related to Bradley - Terry?
Yes. With two items and no ties, the choice probability has the Bradley - Terry form wi/(wi + wj).
Can it represent ties?
The basic strict-ranking model cannot; explicit extensions can model tied rankings.
Does adding an option leave existing choice probabilities unchanged?
No. Absolute probabilities change through the denominator. The basic rule preserves the odds ratio between two existing items.
Check your understanding
Two available options have worths 4 and 1. What is the probability that the first is chosen, and does rescaling the worths to 8 and 2 change it?
Answer: The probability is 4/5 = 0.8. Rescaling gives 8/10 = 0.8, so it does not change.
Related terms
- Preference learning
- Ranking
- Dueling bandit
- Bradley–Terry model
- choice model
- independence of irrelevant alternatives
- maximum likelihood
Learn next
Compare strict rankings, tied rankings and rankings of subsets. Then test the fixed-odds assumption using examples where adding a similar alternative changes how people compare existing choices.
More in Causal inference and experimentation
- Permutation variable importances - Permutation variable importances measure how much a fitted model's performance changes when the values of an input variable are shuffled across evaluation...
- Perplexity - Perplexity measures how well a probabilistic model predicts an observed sequence, commonly a sequence of words or tokens.
- Pinball loss - Pinball loss is an asymmetric error measure used to train and evaluate quantile predictions.
- Policy evaluation - Policy evaluation is the task of estimating how much future reward an agent can expect when it follows a specified policy.
- Positive class - The positive class is the category a classification task designates as the condition or event being detected.
- Post-sample evaluation - Post-sample evaluation assesses a forecasting model using observations from after the period on which it was estimated.
Sources
Primary technical documentation: Turner, van Etten, Firth and Kosmidis, Introduction to PlackettLuce, including the standard ranking probability and extensions: https://hturner.github.io/PlackettLuce/articles/Overview.html
Historical primary publication: Plackett, The Analysis of Permutations, 1975: https://academic.oup.com/jrsssc/article/24/2/193/6953554
The three-item calculation and service case are original illustrations.
Previous: Plackett - Burman design Next: Plan-and-solve