AI Glossary / Natural language and speech / Intermediate
Automated essay scoring
Definition
Definition: Automated essay scoring, or AES, uses computational methods to estimate scores for written essays according to an assessment framework. The system may predict an overall score, separate scores for writing traits, or scores used alongside human judgment.
What it means
What it means: An essay score is meaningful only in relation to a prompt, rubric, learner population and intended use. A model estimating writing quality is not automatically assessing factual correctness, originality or subject knowledge. Those are different claims requiring appropriate evidence.
AES commonly learns from previously scored essays or uses models configured to apply a rubric. Its outputs can support practice, review or assessment, but a plausible numerical score is not self-validating. Consistency also differs from validity: a system can apply the same mistaken shortcut repeatedly. The central question is whether the score supports the interpretation and decision users intend to make.
How it works
How it works: Developers define the scoring construct and collect appropriate essays with reference ratings. A model may use explicit linguistic features, learned text representations or rubric-guided language-model predictions. It is calibrated and evaluated on data separated from development examples.
Validation examines agreement with trained raters, errors across score levels, performance across prompts and populations, and sensitivity to irrelevant features. Human ratings themselves require quality control; they are evidence rather than infallible truth. A deployment specifies how discrepancies, unusual responses and appeals are handled. Generating feedback is a related but separate task: an accurate score does not guarantee that an accompanying explanation is correct or helpful.
Three practical examples
Three practical examples:
1. A practice-writing platform provides an estimated score and feedback to help a learner revise. It makes the rubric visible and distinguishes automated suggestions from a teacher's assessment.
2. An assessment program compares an automated estimate with an independent human rating and sends large discrepancies for review under a validated policy.
3. A course team investigates whether a scoring model transfers to a new prompt. It checks the new essays rather than assuming that strong results on the original task establish suitability for a different writing assignment.
Think of it this way
Think of it: AES resembles a measuring instrument calibrated against a defined writing standard. Calibration can make its readings useful, but only for the conditions and construct it was designed to measure. The analogy is limited because writing is complex, rubrics involve judgment and a score compresses several qualities that may not move together.
Worked example
Worked example: Consider four fictional essays with agreed reference scores 1, 2, 3 and 4 on a five-point scale. A model assigns 2, 3, 4 and 5. It preserves the ranking perfectly, so the Pearson correlation is 1, but every score is one point too high.
Exact agreement is 0/4, or 0%, and mean absolute error is (1 + 1 + 1 + 1)/4 = 1 point. If a program uses a pass threshold of 3, the reference ratings pass the last two essays, while the model passes the last three. One consequential decision changes despite perfect correlation.
This deliberately small example shows why one aggregate statistic cannot establish scoring quality. It does not say correlation is useless: it measures a different property from exact agreement or threshold decisions. A real validation also needs many representative responses, reliable reference scoring and evidence that the rubric measures the intended writing skills.
Illustrative case study
Illustrative case study: Fictional Oak Learning pilots AES for revision exercises. Its dashboard initially celebrates high correlation with teacher ratings. Reviewers find a consistent upward offset and discover that some polished essays receive favorable estimates despite weak development of the assigned argument.
The team recalibrates the scoring process, reviews rubric coverage and evaluates score explanations separately. It retains teacher review for disputed or consequential results and monitors new prompts. This fictional case illustrates measurement checks, not a claim about any named scoring product. It does not suggest that calibration alone fixes missing evidence about argument quality or fairness.
Common misconceptions and mistakes
Common misconceptions and mistakes: AES is not automatically plagiarism detection or AI-generated-text detection. Agreement with human raters does not prove that every relevant writing quality is measured. A longer essay or more complex vocabulary is not inherently better under every rubric. A fluent generated justification can be wrong even when its numerical score seems reasonable. Human scoring also has limitations, so replacing machine output with an unchecked single rating is not a complete validation strategy. The appropriate workflow depends on the assessment's purpose and consequences.
Business use
Business use: AES can support educational platforms and writing-assessment operations when its scope is validated. Define the rubric and intended decision before choosing metrics or automation levels. Evaluate prompt transfer, score calibration, subgroup patterns and unusual responses, and provide clear review procedures. Keep model versions and scoring policies traceable. For feedback products, measure whether comments are accurate and useful for revision, not merely whether users receive them quickly. Avoid presenting a general language model's confident score as an established assessment without task-specific evidence.
People also ask
People also ask: Does AES check whether an essay's claims are true?
Not necessarily. That depends on the scoring task and system. Writing-quality scoring should not be assumed to include fact-checking.
Is correlation with teachers enough?
No. Correlation can remain high despite systematic score offsets or important decision errors. Use complementary evidence.
Can automated scoring and human review be combined?
Yes. They can play different roles, but the combination, discrepancy rules and intended use still require validation.
Check your understanding
Check your understanding: A model ranks every essay correctly but adds one point to every reference score. Is it perfectly accurate for assigning grades?
Answer: No. Ranking agreement does not eliminate calibration error. The extra point can change grades or threshold decisions, as the worked example shows.
Related terms
- Construct validity
- Inter-rater agreement
- scoring rubric
- score calibration
- automated writing evaluation
- formative feedback
- quadratic weighted kappa
- human review
Learn next
compare correlation, exact agreement, mean absolute error and threshold decisions on a small score table. Then identify what each measure leaves unanswered about the rubric, learner groups and quality of feedback. Treat those unanswered questions as part of validation rather than assuming one metric resolves them.
More in Natural language and speech
- Audio mining - Definition: Audio mining is the computational search and analysis of audio collections to find useful content, events or patterns.
- Audio-visual speech recognition - Definition: Audio-visual speech recognition, or AVSR, identifies spoken words using both the acoustic signal and visual evidence of speech production...
- Authorship attribution - Authorship attribution is the task of estimating who wrote a text by comparing it with writing associated with candidate authors.
- Automatic acquisition of lexicon - Definition: Automatic acquisition of a lexicon is the computational construction or expansion of structured lexical entries from language data.
- Automatic taxonomy construction - Definition: Automatic taxonomy construction uses computational methods to build or expand a hierarchy of concepts from text or other evidence.
- Auxiliary - An auxiliary is a grammatical word that helps express distinctions such as tense, aspect, voice, or modality in a predicate.
Sources
Sources: ETS, How the e-rater Scoring Engine Works, including its writing-quality scope: https://www.ets.org/erater/how.html . Zhang, Contrasting Automated and Human Scoring of Essays, 2013, measurement and validation discussion: https://www.ets.org/Media/Research/pdf/RD_Connections_21.pdf . The score table and Oak Learning case are original. Historical programme descriptions are not asserted as current testing policies.
Previous: Automated decision-making Next: Automated negotiation