Can a labeling function refuse to label an example?

Yes. Abstaining is often preferable to issuing a weak guess outside the rule's coverage.

Does combining many rules guarantee reliable labels?

No. Correlated or systematically biased rules can reinforce the same errors.

Can the trained model learn beyond the rules?

It may generalize using input features, but that possibility needs independent evaluation.

AI Glossary / AI and machine learning / Intermediate

Weak supervision

Definition

Weak supervision trains machine-learning models using supervision that is less complete, precise or reliable than carefully verified labels for every example. Sources can include noisy rules, existing databases, coarse annotations and other imperfect signals about the intended target.

What it means

Many projects have abundant raw data but limited resources to label it accurately. Domain knowledge may already exist as keywords, patterns, lookup tables or approximate business rules. Weak supervision turns those signals into training information while recognizing that they can be wrong or incomplete.

The term covers several approaches. Programmatic weak supervision often uses labeling functions that assign a tentative label or abstain. Other settings provide labels only for groups of examples or use incomplete annotations. Weak refers to the supervision available, not necessarily the final model's quality, which must be measured independently.

How it works

A team first defines the target and creates or collects weak signals. For text classification, one rule may flag a phrase, another may consult a product list and a third may use document metadata. Their outputs can overlap, conflict or leave examples unlabeled.

A combination method estimates or otherwise accounts for source reliability and produces training labels or probabilistic targets. Systems such as Snorkel illustrate this approach. Dependence between rules matters: copying one rule five times should not count as five independent pieces of evidence.

A downstream model learns from the resulting training data and may generalize beyond the explicit rules. A separately labeled validation and test set is valuable for assessing the actual task. Assumptions used to combine weak signals and any use of labeled development data should be documented.

Three practical examples

Support-ticket routing: Rules based on product names and error codes provide tentative categories for a large ticket collection. Reviewers label a smaller evaluation set carefully.
Document extraction: An existing database supplies likely entity relationships for text examples, although the text may not actually express every database relationship.
Image classification: Labels describing an entire image collection provide coarse information when individual objects are not annotated. The learning method must match that weaker label structure.

Think of it this way

Think of several imperfect clues helping an investigator decide where to look. A clue may be useful without being conclusive, and two clues copied from the same source are not independent confirmation. Unlike this informal analogy, a weak-supervision system needs explicit rules and assumptions for converting evidence into training targets.

Worked example

Consider three invented labeling functions for billing-related tickets. Rule A looks for “invoice,” rule B looks for “payment,” and rule C checks whether a billing form was used. On one ticket, A votes billing, B abstains and C votes not billing.

A simple majority does not resolve this tie. A probabilistic label model might weigh the signals differently based on estimated reliability, but its output depends on assumptions and available evidence. It should not be presented as a verified label.

Suppose a final classifier is tested on one hundred independently reviewed tickets and gets eighty-seven correct. That 87% is the toy test accuracy; agreement with the weak rules on the training set would measure something different.

Illustrative case study

Fictional case: Helpstream has many historical requests but only a modest budget for expert annotation. Its team writes labeling functions with support staff and inspects examples where the functions disagree.

Two apparently separate rules both rely on the same product-code field, so the team records their dependence rather than treating them as independent votes. A held-out expert-labeled sample reveals missed refund requests, prompting a targeted rule revision. The project keeps evaluation labels separate from training heuristics. This scenario is fictional and claims no real performance improvement.

Common misconceptions and mistakes

Weak supervision is not a way to obtain ground truth for free. Heuristics can reproduce biases, stale policies and systematic omissions. Strong agreement among related rules can still be wrong.

Do not evaluate only against labels produced by the same weak sources used in training; that can reward imitation rather than correctness. Avoid tuning repeatedly on the final test set. Also distinguish weak supervision from unsupervised learning: weak supervision does provide some task-related guidance, even when that guidance is noisy or coarse. It can coexist with semi-supervised methods, but the terms are not interchangeable.

Business use

The approach can reduce repetitive labeling effort when subject experts can express useful rules more easily than annotate every example. Its value depends on rule coverage, source diversity, maintenance effort and the quality needed for the application.

Plan for expert review of ambiguous cases and representative evaluation data. Track versions of labeling functions alongside model versions so that changes can be traced. Compare total development cost and performance with a smaller, carefully labeled baseline; more weakly labeled examples are not automatically better.

Check your understanding

Five labeling functions are copies of the same keyword rule. Should their agreement be treated as five independent confirmations?

Answer: No. Their shared origin creates dependence and does not provide five separate sources of evidence.

Related terms

Distant supervision
Semi-supervised learning
Ground truth
Labelling function
noisy labels
probabilistic label model
data programming
label correlation

Learn next

Write three deliberately different rules for a small classification task, allowing each to abstain. Inspect their coverage, conflicts and shared information sources. Label a separate sample carefully and compare majority voting with a trained model. Record which assumptions are needed before interpreting a probabilistic weak label as useful evidence.

Trained model

Sources

Alexander Ratner and colleagues, Snorkel: Rapid Training Data Creation with Weak Supervision: https://arxiv.org/pdf/1711.10160 . Independent review read labeling-function sources and section 2.2 on probabilistic label modeling, including correlation and double-counting. The rule votes, 87% result and fictional case are original illustrations; a weak label is not represented as verified truth.

Previous: Weak learning Next: Web data integration