Is an oracle always a human expert?

No. In learning terminology, it is a source of labels or information; it could be a person, service or another process.

Does proactive learning replace active learning?

It generalizes the acquisition problem by considering realistic properties of multiple sources, rather than being unrelated to active learning.

Should the cheapest source always be used?

No. Cheap but unreliable answers can reduce utility or require costly correction. The decision depends on expected value, not price alone.

AI Glossary / AI and machine learning / Advanced

Proactive learning

Definition

Proactive learning is an extension of active learning that chooses both which example to label and which information source to ask. It accounts for the possibility that label providers differ in accuracy, availability, willingness to answer and cost.

What it means

An active-learning system seeks useful labels rather than labeling data indiscriminately. Proactive learning adds a practical question: who should supply each label? A specialist may be accurate but expensive or busy; another source may be cheaper but less reliable on difficult cases.

The technical term does not simply mean an AI agent that takes initiative. It names a learning framework for acquiring information under realistic constraints. Its objective is to improve the learner's information efficiently, not to assume every response from an expert or service is correct.

How it works

The system estimates the value of labeling a candidate example and the properties of available label sources. It selects a source-example pair under a budget or utility model, receives an answer if available and updates its learning process.

The estimates themselves may be uncertain. Reliability can vary by topic, and a source may decline a question. A practical implementation needs ways to learn about source quality, reconcile conflicting responses and avoid repeatedly spending the budget on apparently promising but unanswerable queries. The acquisition policy should be evaluated alongside the predictive model it trains.

Three practical examples

1. A document-labeling project can route straightforward cases to general annotators and reserve scarce specialist time for questions where it adds the most value.

2. An image-classification study can choose among label sources whose expertise differs by object type, rather than assigning one global accuracy score to everyone.

3. A multilingual dataset team can account for both language expertise and availability when deciding whom to ask about a particular example. These are illustrative workflows, not claims that one policy is optimal in every project.

Think of it this way

Imagine managing a help desk for your own research questions. You decide which uncertainty matters most and which colleague is likely to answer it accurately at an acceptable cost in time.

The analogy highlights the joint choice. A computational learner must estimate those benefits and costs from available information; it does not automatically know who is an expert or whether an answer is correct.

Worked example

Consider a toy acquisition rule based on expected useful information per cost unit. For one example, source A is estimated to answer with probability 0.9 and be correct, conditional on answering, with probability 0.8. If a correct answer is worth ten teaching utility units and costs three units, the simplified ratio is 0.9 × 0.8 × 10 / 3 = 2.4.

Source B always answers, has estimated correctness 0.95 and costs five units. Its ratio is 1 × 0.95 × 10 / 5 = 1.9. The toy rule favors A, despite B's higher accuracy. Real utility models must also consider harm from wrong labels and the source's task-specific expertise; this is not the original paper's complete algorithm.

Illustrative case study

Fictional archive project LabelBridge has a limited annotation budget. Its team initially sends every uncertain document to the most experienced reviewer, creating a queue that slows learning.

The researchers test an acquisition policy that considers document type, reviewer expertise and availability. They discover that one general reviewer is particularly reliable on a common technical category. The policy uses that information while retaining independent quality checks. Progress is judged by validated model improvement per annotation budget, rather than the raw number of labels collected or the seniority of the person supplying them.

Common misconceptions and mistakes

The most uncertain example is not always the best next query. It may be expensive, unanswerable or uninformative for the deployment task. Likewise, the most accurate source overall may be a poor choice for one specialist question.

Do not treat source reliability estimates as permanent facts. Expertise, fatigue and task difficulty can affect responses. Another mistake is using the model's own agreement with a source as the only evidence that the source is correct. This can reinforce shared errors. Evaluate acquired-label quality and downstream performance with an appropriate independent process.

Business use

Proactive learning is useful when label acquisition has meaningful cost and several sources differ in expertise or availability. It can support more deliberate use of expert time and make annotation trade-offs explicit.

A business plan should identify the budget, the value of improved predictions and the cost of incorrect labels. Compare the acquisition policy with simpler routing rules under the same resources. Track unanswered queries, disagreements and source-specific performance. A sophisticated selection formula is not useful if its reliability estimates are poorly supported or the collected labels do not improve the intended task.

Check your understanding

Two label providers have the same average accuracy, but one is much better on the current topic. Should the system necessarily treat them as interchangeable?

Answer: No. Topic-specific reliability can matter more than the overall average. The acquisition decision should use information relevant to the particular example and the available budget.

Related terms

Active learning
Information gain
annotation
oracle
label quality
cost-sensitive learning
uncertainty sampling
human-in-the-loop learning

Learn next

List several sources of labels and describe their likely strengths, costs and availability. Compare choosing only an example with jointly choosing an example and source. Then design a small evaluation of learning improvement under a fixed annotation budget.

Sources

Donmez and Carbonell, Proactive Learning: Cost-Sensitive Active Learning with Multiple Imperfect Oracles: https://www.cs.cmu.edu/~jgc/publication/PublicationPDF/Proactive_Learning_Cost-Sensitive_Active_Learning_With_Multiple_Imperfect_Oracles.pdf. The utility calculation and LabelBridge scenario are original illustrations; they do not reproduce the paper's experimental results or full decision rule. Independent review consulted the original paper introduction and oracle-selection formulation, supporting budgeted joint selection among costly, imperfect label sources. The illustrative utility calculation was checked separately; it is not presented as the original algorithm.

Previous: Pro-drop languages Next: Proaftn