How many committee members are required?

There is no universal best size. Even small committees can be useful, but their diversity and the problem matter more than a fixed count.

Must members use different model architectures?

No. Plausible variation can arise within one model family, including posterior sampling or other suitable constructions.

Does QBC remove the need for reliable labels?

No. Incorrect or inconsistent labels can misdirect the active-learning process, so annotation quality remains essential.

AI Glossary / Probabilistic machine learning / Intermediate

Query by committee

Definition

Query by committee is an active-learning strategy that selects examples for labeling by examining disagreement among several plausible models. The committee represents alternative explanations supported by the currently labeled data. Asking for labels where those models disagree aims to reduce uncertainty about which explanation is appropriate.

What it means

When labels are expensive, a learner can choose which unlabeled examples deserve attention next. Query by committee, or QBC, uses differences among model predictions as a signal of potential information value.
This differs from asking which example one model finds uncertain. Several models can all predict a class probability near one half because the outcome is inherently ambiguous, yet agree closely with one another. Conversely, two models can each be confident but predict opposite classes. QBC is concerned with disagreement between plausible models, although practical committees may represent that uncertainty imperfectly.

How it works

Start with a labeled set and construct several competing models. Classical formulations sample hypotheses consistent with the observed labels; probabilistic extensions sample from or approximate a posterior over models. Practical variants may use resampled training sets, different initializations, or other justified diversity mechanisms.
Evaluate candidate unlabeled examples with the committee. For classification, one measure is vote entropy: if fraction p_k of members votes for class k, calculate -sum_k p_k log(p_k), treating a zero contribution as zero. Probability-distribution disagreement and regression-output spread provide other possible measures.
Select examples according to the chosen acquisition rule, obtain their labels from an appropriate source, add them to the training set, and update the committee. Committee construction and disagreement measurement are both substantive choices. A collection of identical or jointly biased models can miss informative examples even if its agreement looks strong.

Three practical examples

Document annotation: prioritize documents on which plausible classifiers disagree, then obtain human labels under a consistent annotation guide.
Image-dataset expansion: select candidate images that expose differences among models, while checking that selected images are relevant to the deployment population.
Simulator-based regression: request costly simulator outputs where plausible regressors differ, with an acquisition rule suited to continuous predictions.

Think of it this way

Think of asking several informed reviewers where they disagree before commissioning an expensive investigation. The disagreement identifies questions that could change their views. The analogy has limits: model diversity is not human expertise, agreement does not prove correctness, and some disputed examples may be inherently ambiguous rather than informative.

Worked example

Consider an original committee of four binary classifiers. On candidate A, all four vote positive. On B, three vote positive and one negative. On C, two vote each way.
Using base-2 logarithms, A's vote entropy is zero. B's entropy is -0.75 log2(0.75) - 0.25 log2(0.25), approximately 0.8113 bits. C's entropy is -2 × 0.5 log2(0.5) = 1 bit. A simple maximum-vote-entropy policy therefore selects C for labeling.
This does not mean C has a 100-percent error probability, nor that labeling it guarantees the greatest future accuracy gain. The score measures the distribution of committee votes.
For a separate contrast, suppose four probabilistic models each assign positive probability 0.51. They all vote positive, so vote entropy is zero, despite high uncertainty in each individual prediction. If their predictions are identical, a distribution-disagreement measure is also zero. Additional labels may still matter, but committee agreement and within-model uncertainty are different signals.

Illustrative case study

Fictional case: Alder Archive needs to label a large collection of documents. Its team trains a small committee from an initial labeled set and reviews high-disagreement candidates.
Some selected documents contain damaged text, making their labels hard to determine. The team records ambiguity, checks annotation consistency, and compares QBC with random selection at the same annotation-time budget. It also reserves an independent evaluation sample, because the actively chosen training examples are not a random sample of future documents. This invented case illustrates the practical review process, not a measured reduction in labeling cost.

Common misconceptions and mistakes

QBC is not ordinary ensemble voting used only to make final predictions. The distinctive step is using disagreement to decide which labels to acquire. A committee can still be used for prediction, but that is a separate role.
Disagreement is not guaranteed information gain, and unanimous predictions can be unanimously wrong. Nor should special theoretical learning-rate results from simple noise-free models be treated as guarantees for arbitrary noisy datasets, deep networks, or changing data distributions.

Business use

QBC may be useful when unlabeled examples are plentiful and trustworthy labels consume substantial expert time. The useful outcome is better task performance for the total labeling and computation budget, not simply a high disagreement score.
Track annotation duration, disagreement patterns, class coverage, and evaluation performance across selection rounds. If many selected examples are redundant or outside the intended population, revise the acquisition process. Include committee-training cost and preserve a separate evaluation set so that an apparently efficient labeling strategy can be assessed fairly.

Check your understanding

Four committee members split two votes to two on one example. Does that prove its true label is equally likely to be either class?

Answer: No. It describes the committee's votes. Interpreting those votes as calibrated probabilities requires additional assumptions and evidence.

Related terms

Active learning
Version space
Ensemble learning
committee disagreement
vote entropy
uncertainty sampling
posterior model distribution
annotation budget

Learn next

Compute vote entropy for different committee splits and compare it with a single model's predictive entropy. Then examine how committee members are generated. Finally, evaluate active selection against a random-labeling baseline using the same budget and an independent test set.

Test set

Sources

Seung, Opper and Sompolinsky, Query by Committee, original publication abstract and record: https://collaborate.princeton.edu/en/publications/query-by-committee/ Argamon-Engelson and Dagan, Committee-Based Sample Selection for Probabilistic Classifiers, original research paper: https://arxiv.org/pdf/1106.0220 Settles, Active Learning Literature Survey, author's explanation of committee construction and disagreement measures: https://burrsettles.com/pub/settles.activelearning.pdf Argamon-Engelson and Dagan committee construction and vote-entropy discussion were independently consulted.

Previous: Quasi-stationary distribution Next: Query Engines