AI Glossary / Natural language and speech / Intermediate
NER model
Definition
The NER model is a framework for assessing the accuracy of live subtitles, particularly subtitles produced through respeaking. Its name refers to Number of words, Edition errors, and Recognition errors. It combines weighted error deductions with a broader assessment of how viewers experience the subtitles. Here, NER does not mean named-entity recognition.
What it means
A respeaker listens to an event and speaks a version of its content into speech-recognition software. The resulting subtitles can contain errors introduced while the person edits the speech, errors introduced by recognition, or both.
The NER model distinguishes these causes and considers their consequences. Losing a minor detail is different from presenting a plausible statement that reverses the speaker's meaning.
It also recognizes correct editions: legitimate changes that preserve the intended information and coherence should not automatically count as errors. This makes its purpose different from simply counting every textual difference against a verbatim transcript.
How it works
An assessor compares the source speech and the subtitles, using the audiovisual context where relevant. Edition errors concern the respeaker's handling of the information; recognition errors concern what the recognition process transcribes incorrectly.
Errors receive weights according to severity: minor errors count as 0.25, standard errors as 0.5, and serious errors as 1. Correct editions receive no error penalty.
The accuracy percentage is (N − E − R) / N × 100. E and R are weighted totals, rather than unweighted counts of error events. In the original model, N counts the respoken output, including words and commands such as punctuation or speaker identification.
The assessor also considers delivery features such as delay, reading speed, placement, speaker identification, and coherence. The original authors explicitly distinguish the numerical accuracy result from the overall quality judgment.
Three practical examples
- A live lecture subtitling team separates lost ideas caused by editing from misrecognized technical words to identify different training needs.
- A subtitle reviewer distinguishes a harmless reformulation from an omission that removes the speaker's main point.
- A service team examines subtitle delay alongside accuracy because a readable, accurate line can still arrive too late to match the scene.
Think of it this way
Think of assessing a parcel delivery by checking both what arrived and how it arrived. A missing small accessory and the wrong main item deserve different attention, while punctuality is a separate part of the experience.
The analogy has limits: subtitle meaning is contextual, and severity cannot always be determined through a simple inventory. The model requires informed judgment about information and viewer understanding.
Worked example
Consider an original teaching example with N = 200 counted output units under the stated convention. An assessor identifies one serious edition error and one standard edition error, giving E = 1 + 0.5 = 1.5.
There are also two standard recognition errors and two minor recognition errors. Their weighted total is R = 0.5 + 0.5 + 0.25 + 0.25 = 1.5.
The accuracy is therefore (200 − 1.5 − 1.5) / 200 × 100 = 98.5%. There are six error events, but the weighted deduction is three units. Subtracting six directly would calculate a different measure.
Assume a separate reformulation preserves the source meaning and coherence. Classifying it as a correct edition adds no penalty. The example assumes the assessor's severity classifications; arithmetic alone cannot validate those judgments or establish that the subtitles are satisfactory overall.
Illustrative case study
In a fictional project, Cedar Events reviews subtitles from an internal presentation. The score appears strong, but one subtitle changes a proposed date into a confirmed date, and several lines arrive after the relevant slide has disappeared.
The reviewers document the meaning change with its context and examine the delay separately. They use the error categories to guide work on respeaking and recognition, while retaining the overall viewing assessment.
This invented case illustrates why a percentage should be accompanied by a reasoned account of significant errors. It reports no measured outcome from an actual organization.
Common misconceptions and mistakes
Do not confuse this NER model with the NLP task of recognizing names, organizations, or places.
Do not use source-transcript word count as the denominator without acknowledging a change from the original convention.
Do not treat every paraphrase as an error, or every omission as harmless compression.
Do not interpret 98.5% as a guarantee that 98.5% of viewers understood the content. It is a weighted accuracy calculation.
Finally, a target discussed in historical publications is not automatically a universal requirement, a current regulatory rule, or proof of complete subtitle quality.
Business use
Organizations procuring live subtitling can use the framework to make review discussions more specific. Instead of reporting only a score, reviewers can describe the balance of editing and recognition problems and identify examples that materially affect understanding.
Comparisons require consistent counting, severity decisions, and sampling. Retain the source segment, subtitle output, error annotations, and viewing observations so that another reviewer can inspect the reasoning. A numeric improvement is useful only when it corresponds to a better service for the intended audience.
People also ask
Is a verbatim subtitle always better?
No. A correct edition can preserve the relevant information while adapting the delivery, although careless editing can lose meaning.
Are all recognition errors equally serious?
No. The model weights their effects rather than assuming every incorrect word has the same consequence.
Can the score replace a complete subtitle review?
No. Timing, presentation, coherence, and the impact of particular errors remain important to the overall assessment.
Check your understanding
An assessor records four minor recognition errors and no edition errors in a sample with N = 100. What is the accuracy, assuming those classifications are correct?
Answer: R = 4 × 0.25 = 1. The accuracy is (100 − 1) / 100 × 100 = 99%. Counting four full error units would disregard the model's severity weighting.
Related terms
- Automatic Speech Recognition
- Word error rate
- live subtitling
- respeaking
- subtitle latency
- accessibility evaluation
Learn next
compare weighted semantic error assessment with word error rate. Then annotate a short subtitled segment, recording why each alteration is an error or a correct edition and separating accuracy from timing observations.
More in Natural language and speech
- Natural language - A natural language is a human language that develops through use within a community, such as Arabic, English, or American Sign Language.
- Natural language programming - Natural language programming is the use of human-language descriptions to specify computational behavior that a system turns into an executable program or...
- Natural-language user interface - A natural-language user interface lets people interact with software using human-language expressions, such as typed questions or spoken requests.
- Neural machine translation - Neural machine translation, or NMT, uses neural networks to produce a translation from one human language into another.
- Newline character - A newline character is a control character used to mark a line boundary in text.
- Noisy-or - Noisy-or is a probabilistic model in which any one of several active causes can produce an effect, but each cause may independently fail to do so.
Sources
Primary sources: Romero-Fresco and Martínez, Accuracy Rate in Live Subtitling – the NER Model, original paper: https://www.commonspaces.eu/document/255/download/ Author presentation explaining the model and severity categories: https://webs.uab.cat/act/wp-content/uploads/sites/126/2017/03/ner-3.pdf The worked example and fictional case are original illustrations, not results reported by these sources. Independent review: original counting convention, severity weights, correct editions and overall-assessment passages read.
Previous: Neocognitron Next: Nested case - control study