Is BatchBALD a training loss?

It is an acquisition criterion for choosing labeling requests, distinct from the loss used to fit the predictive model.

Does it require class labels before selection?

The candidate labels are unknown; the model evaluates their possible outcomes.

Can a redundant example still help?

Yes. The worked example shows that a partially overlapping second observation can add information.

AI Glossary / Probabilistic machine learning / Advanced

BatchBALD

Definition

BatchBALD is a Bayesian active-learning acquisition method that selects a batch of examples by estimating how much their labels jointly reveal about a model's uncertain parameters. It extends BALD, Bayesian Active Learning by Disagreement, to account for overlapping information within a group of labeling requests.

What it means

A model may be uncertain about several examples for essentially the same reason. Ranking those examples independently can count the same potential learning benefit repeatedly. BatchBALD evaluates the labels as a group, aiming to select informative requests whose combined value reflects their dependencies.
The calculation is an expectation under the model's current beliefs. It is not a guarantee that the requested labels will improve deployment accuracy or expose every weakness. Its usefulness depends partly on whether those beliefs describe the uncertainty that matters for the application.

How it works

For a candidate batch B, the acquisition score is I(Y_B; W | X_B,D): mutual information between its unknown labels Y_B and model parameters W, conditional on its inputs X_B and existing training data D.
Compute joint predictive entropy minus expected joint entropy conditional on W. Under the model's conditional-independence assumption, the latter decomposes across examples; the former generally does not. A greedy procedure adds the candidate with the largest incremental score. Practical calculations use approximate posterior samples and entropy estimates. For a batch of one, the objective reduces to BALD. Labels are requested after constructing the batch.

Three practical examples

A document team could use BatchBALD to prioritize a limited review batch from many unlabeled forms. Its evaluation would need to check whether uncertain forms represent important layout variations rather than repeated copies.
A visual inspection team could investigate whether jointly selected images improve learning compared with selecting the highest individual uncertainty scores. It would compare both procedures at the same annotation budget.
A researcher could construct a small probabilistic test case with known information values before trusting an implementation on a large neural network. Such a test checks calculations, not the quality of the eventual posterior approximation.

Think of it this way

Imagine asking several witnesses about an event. Two answers may partly repeat one another, while an apparently unpredictable answer might reveal nothing about the question you actually want resolved. BatchBALD tries to value the collection of answers in relation to uncertainty about the model. The analogy does not imply that annotators themselves are unreliable or that the method measures human credibility.

Worked example

Consider an original binary toy model with two equally plausible parameter states, W=0 and W=1. For each of candidates A and B, a label agrees with W with probability 0.90. Assume their labels are independent conditional on W. For candidate C, either label has probability 0.50 regardless of W, independently of the other labels conditional on W.
Use base-two logarithms. A's predictive label entropy is one bit. Its conditional label entropy is h(0.10), where h(p)=−p log2(p)−(1−p)log2(1−p), approximately 0.468996 bits. Its BALD score is therefore 0.531004 bits. B has the same score. C also has one bit of predictive entropy, but its conditional entropy is one bit, so its information score is zero.
For the batch A,B, label-pair probabilities for 00,01,10,11 are 0.41,0.09,0.09,0.41. Their joint entropy is approximately 1.680077 bits. Subtracting twice h(0.10) gives a BatchBALD score of 0.742086 bits.
Adding the individual scores would give 1.062009 bits, exceeding the one bit initially available about W. The joint calculation avoids that double counting. B still adds approximately 0.211081 bits after A, whereas C adds none: batch A,C scores 0.531004 bits. Thus partial redundancy does not necessarily make a second example worthless. These are exact calculations for the stipulated distribution, not measured gains from a trained classifier.

Illustrative case study

Fictional archive service Hazel Index has a fixed weekly budget for labeling scanned document types. Its researchers compare BatchBALD, individual BALD ranking and random acquisition using the same initial labels and model architecture. They keep a separate evaluation set representing incoming documents.
The team logs selection time as well as annotation effort, because sophisticated acquisition can consume substantial computation. It also examines whether selections cluster around scanning artifacts that are uncommon in production. The report distinguishes information scores from actual downstream performance. This fictional study describes a fair comparison process; it provides no claimed accuracy improvement or commercial saving.

Common misconceptions and mistakes

“Predictive uncertainty and information about parameters are identical.” Candidate C is maximally uncertain but uninformative about W in the example.
“BatchBALD simply sums BALD scores.” The joint score is designed to account for dependence among requested labels.
“Conditionally independent labels are independent after averaging over parameters.” Shared parameter uncertainty can make the predictive labels dependent.
“Greedy selection guarantees the best possible practical batch.” The theoretical approximation result concerns its mathematical objective under the relevant assumptions; approximate inference and numerical scoring introduce further limitations.
“More information about model parameters necessarily improves the business metric.” That connection requires evaluation on the actual task.

Business use

Consider BatchBALD when expert labeling is costly, multiple labels must be requested together and a credible Bayesian uncertainty approximation is available. Ask how parameter samples are obtained, how acquisition cost scales and how the procedure handles a changing candidate pool. Judge the complete process against simpler alternatives using annotation time, computing cost and held-out performance. Preserve the acquisition settings and selected example identifiers so the experiment can be reproduced.

Check your understanding

Why does C have zero information score?

Answer: its predicted label distribution is the same under both parameter states.

What is the joint A,B score?

Answer: approximately 0.742086 bits.

Why is summing A and B misleading here?

Answer: it counts overlapping information about the same one-bit parameter uncertainty more than once.

Related terms

Batch active learning
Bayesian neural network
Mutual information
Conditional independence
BALD
predictive entropy
epistemic uncertainty
acquisition function

Learn next

Calculate a joint predictive distribution by averaging conditional distributions over parameter hypotheses. Then compare its entropy with the sum of individual predictive entropies and study how posterior approximation affects acquisition.

Sources

Kirsch, A., van Amersfoort, J., and Gal, Y. BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning, NeurIPS 2019, Sections 2–3: https://papers.neurips.cc/paper/8925-batchbald-efficient-and-diverse-batch-acquisition-for-deep-bayesian-active-learning.pdf The noisy binary example and fictional business scenarios are original illustrations.

Previous: Batch size Next: Batched automation