AI Glossary / AI foundations / Foundation
Data labeling
Definition
Data labeling is the process of attaching tags, categories, scores or other annotations to raw data such as text, images, audio or records, so that a machine learning system can learn from examples with known answers. It is not the same as collecting or cleaning data, although the three often happen together. Labels may be added by people, by software, or by a mix of both, and teams use slightly different names for the work, including annotation and tagging.
What it means
Raw data rarely comes with the answer attached. A pile of customer emails does not say which ones are complaints. A folder of invoice scans does not say which field is the total. Labeling is the step where someone, or something, writes the answer next to the example.
Those labeled examples become the teaching material for supervised learning and a common part of evaluation. If the labels are wrong, vague or inconsistent, the model learns the same confusion. For a business, this means labeling is less a technical chore and more a decision about what the organization considers correct.
How it works
A typical labeling project starts with a question the model must answer, such as: is this transaction suspicious, which product category does this item belong in, or what is the sentiment of this review. The team turns that question into a label set, a short list of allowed answers, and writes guidelines that explain each label with examples and edge cases.
Next, a sample of data is given to labelers. These may be staff who know the subject, an outside vendor, or a tool that proposes a label for a person to confirm. Good projects have several labelers mark the same items and then measure how often they agree. Where they disagree, the guidelines are usually unclear, and the team revises them and relabels.
Finally, the labeled set is usually split. One part trains the model, and another part is held back to test it on examples it has not seen. Labeling does not end there. As the business, products or customers change, new data needs new labels, and old labels may need review.
Three practical examples
- Finance team: An accounts payable group wants a model to sort scanned invoices by expense category. Staff label 2,000 past invoices with the correct general ledger category. The model learns from those labeled invoices and suggests categories for new ones, which a clerk confirms.
- Retailer: An online store wants to tag product photos by color and style so search works better. Labelers mark each photo with a color and a style from a fixed list. The team then checks a sample to confirm that two labelers pick the same tag for the same photo.
- Support desk: A support team wants to route incoming tickets to the right queue. Experienced agents label 5,000 past tickets as billing, shipping, technical or account access. The labeled tickets are used to train a classifier, and agents later correct its mistakes, which creates new labels for the next round.
Think of it this way
Think of labeling as writing the answer key on practice exams before a student studies. The student can only learn what the key says, so a key with mistakes teaches mistakes. The analogy stops working because a student can ask a teacher to explain, while a model only absorbs the patterns in the labels and cannot question them.
Worked example
A lender wants to label loan applications as likely to repay or likely to default, using 1,000 past applications for training. Three reviewers each label the same 100 applications to test the guidelines. They all agree on 82 of the 100, so the agreement rate is 82 divided by 100, which is 82 percent. The team reads the 18 disputed cases, finds that the rule for self-employed applicants was unclear, rewrites it, and relabels. The second round gives agreement on 93 of 100, or 93 percent.
Cost matters too. If one reviewer labels 40 applications per hour and the full set is 1,000 applications, one pass takes 1,000 divided by 40, which is 25 hours. If each item is labeled by 2 people so disagreements can be caught, the effort doubles to 50 hours. At an internal cost of 30 dollars per hour, that is 50 times 30, or 1,500 dollars for the labeling pass.
Illustrative case study
Fictional case: Harbor Lane Outfitters, a mid-sized online retailer, built a model to flag product reviews that mention damaged goods so the returns team could act quickly. A contractor labeled 3,000 reviews in a week. The model scored well in testing but performed poorly in production, missing many real damage complaints. When the team reviewed the labels, they found the contractor had been told to mark only reviews that used the word broken. Reviews saying cracked, dented or arrived in pieces were labeled as not damaged. The team wrote clearer guidelines with examples of each phrasing, had two internal staff relabel a sample of 500 reviews, and measured their agreement before relabeling the rest. After retraining, the model caught far more genuine damage reports, and the team scheduled a small monthly relabeling check to catch drift in how customers describe problems.
Common misconceptions and mistakes
Labeling is a one-time task that ends when the model is built. In practice, data, products and customer language change, so labels need periodic review and fresh examples.
More labeled data always beats better labeled data. A smaller set with clear, consistent labels often teaches a model more than a large set where labelers disagree.
Labels are objective facts. Many labels, such as sentiment or risk level, involve judgment, so two careful people can reasonably disagree and the guidelines must settle the rule.
Automatic labeling removes the need for people. Software can propose labels quickly, but its errors spread silently, so a human check on a sample is still needed.
Business use
To start, pick one narrow question, define a short label set, and write guidelines with real examples and edge cases. Label a small pilot batch, have more than one person label the same items, and measure agreement before scaling up. Keep a held-out set that was labeled with extra care to test the model honestly.
Measure agreement between labelers, error rates found in spot checks, and the hours spent per labeled item. Costs include staff time, vendor fees and the effort of reviewing disputes. Risks include inconsistent labels, bias in who labels and what is labeled, and exposing sensitive or personal data to outside labelers, so check contracts and privacy rules before sharing records.
People also ask
Who does data labeling in a business?
It can be internal subject experts, outside vendors, or a mix of both. Tasks that need domain knowledge, such as classifying legal or financial documents, usually need people who understand the subject.
How much labeled data do I need?
It depends on the task, the model and how varied the data is, so there is no single number. Start with a small pilot, test the results, and add labeled examples where the model is weakest.
Is data labeling needed for every AI project?
No. Some approaches, such as using a pre-trained language model with good instructions, need little or no new labeling. You still usually need some labeled examples to test whether the system works.
Check your understanding
Why did the lender in the worked example rewrite the guideline for self-employed applicants?
Answer: The 18 disputed cases showed that reviewers read that rule differently. Clearer guidelines raised agreement from 82 percent to 93 percent.
How many hours did one labeling pass take if one reviewer handles 40 applications per hour and the set is 1,000?
Answer: It took 25 hours, because 1,000 divided by 40 equals 25.
What went wrong at Harbor Lane Outfitters?
Answer: The contractor was told to mark only reviews using the word broken, so cracked or dented items were labeled as fine. The model learned that gap and missed real damage complaints.
Why have several people label the same items?
Answer: Measuring how often they agree shows whether the guidelines are clear. Low agreement points to vague rules that should be fixed before labeling at scale.
Related terms
- Training data
- Supervised learning
- Ground truth
- Annotation guidelines
- Inter-annotator agreement
- Data quality
- Evaluation set
- Human in the loop
Learn next
Try labeling 50 of your own items against a written guideline, then ask a colleague to label the same items and compare. Read next about training data and supervised learning to see how labels feed a model, and about evaluation to see how a held-out labeled set tests it.
More in AI foundations
- Context window - A context window is the limited amount of information a language model can work with in a particular generation.
- Dartmouth workshop - The Dartmouth workshop was a summer research project held at Dartmouth College in 1956 that is widely treated as the founding event of artificial...
- Data annotation - Data annotation is the process of adding labels, tags or notes to raw data such as text, images, audio, video or records, so that a machine learning system...
- Deep learning - Deep learning is a branch of machine learning that learns representations through multiple layers of computation.
- Direct preference optimization (DPO) - Direct preference optimization (DPO) is a method for adjusting a language model so its answers match human preferences, using pairs of responses where...
- Embodied AI - Embodied AI is artificial intelligence that has a physical body, such as a robot, a vehicle or a machine with sensors and moving parts, and learns or acts...
Sources
Google, Machine Learning Glossary: https://developers.google.com/machine-learning/glossary NIST, AI 100-1 Artificial Intelligence Risk Management Framework: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf ISO, ISO/IEC 22989 Artificial intelligence concepts and terminology: https://www.iso.org/standard/74296.html The sources were consulted for the standard definitions of labeled data and supervised learning and for general guidance on data quality and risk; the examples, analogy, worked example and case are original educational illustrations. Checked 5 Nov 2026.