Can a task mix NORA and one-right-answer elements?

Yes. A calculation may have one correct numerical result while its explanation allows many valid forms.

Does NORA require human evaluation?

Human judgment is often useful, but some requirements can be checked automatically. The evaluation mix should match the task's constraints and subjective dimensions.

Are reference answers useless for NORA tasks?

No. They can illustrate quality and support comparison, but should not automatically be treated as the only acceptable responses.

AI Glossary / Generative AI and language / Beginner

No one right answer

Definition

No one right answer, often abbreviated NORA in generative-AI evaluation discussions, describes a prompt that can have multiple acceptable responses. A request for a useful summary, a friendly explanation, or several creative names may be satisfied in different ways. The term concerns the space of valid answers. It does not mean that every answer is acceptable or that factual accuracy, instructions, and quality standards no longer apply.

What it means

Some questions have one correct result, while others allow choices about wording, emphasis, organization, or creativity. Two summaries can cover the same essential facts using different sentences. Two explanations can both be correct while using different examples suited to the audience.

This flexibility complicates evaluation. Comparing an output with one exact reference string can reject a valid alternative. Yet judging only whether an answer sounds pleasant can overlook missing facts or unsupported claims.

NORA is a useful task-description label rather than a particular algorithm. Its practical value is to remind evaluators to define acceptable outcomes and quality dimensions instead of assuming that a single stored answer captures every valid response.

How it works

Identify which parts of the task are fixed and which allow variation. A summary may need three specific facts, accurate quantities, and a length limit while allowing different wording. A creative task may allow many ideas but still require relevance and a stated tone.

Build an evaluation rubric around those requirements. Separate objective checks, such as whether a required number is correct, from judgments such as clarity or originality. Use examples to help reviewers apply the rubric consistently, and inspect disagreements rather than hiding them inside one average.

Automatic measures or model-based judges can help with scale, but their behavior should be checked against the intended standards. Multiple references can illustrate valid responses without exhausting the answer space. Evaluation should reward satisfying the task, not merely resembling the evaluator's favorite phrasing.

Three practical examples

1. Summarization: several concise summaries may preserve the important decisions in a meeting while arranging them differently.

2. Teaching: an assistant can explain a technical concept with different accurate analogies, depending on the learner's background.

3. Creative drafting: a request for a welcoming event invitation can yield many suitable messages, provided each respects the event details and requested tone.

Think of it this way

Think of asking several cooks to prepare a satisfying meal from the same ingredients. Different dishes can meet the brief, but burnt food or omitted dietary requirements still fail it. The analogy is limited because language tasks can contain precise factual obligations as well as preferences. Multiple acceptable forms do not remove the need to verify those obligations.

Worked example

Consider this fictional evaluation prompt: “Summarize these notes in one sentence: the workshop is on Friday; it starts at 10 a.m.; registration is required.”

Candidate A says, “Registration is required for Friday's workshop, which begins at 10 a.m.” Candidate B says, “The workshop starts at 10 a.m. on Friday, and attendees must register.” Both preserve the three supplied facts in one sentence, despite different wording.

Suppose a simple rubric awards one point for each required fact and one for using one clear sentence. Both candidates receive 4/4. Candidate C says, “Friday's free workshop starts at 10 a.m.” It preserves the date and time but omits registration and invents that the workshop is free. Even if it is concise, it should fail the factual-support requirement. The invented rubric is illustrative; a real assessment should state whether unsupported facts cause an automatic failure rather than being offset by points for other qualities.

Illustrative case study

Fictional case: Acorn Learning evaluates explanations of basic statistics. Its first checker compares generated answers with one teacher-written paragraph, penalizing alternative examples that teachers consider equally useful.

The team replaces exact wording as the main criterion with a rubric covering correctness, required concepts, and suitability for beginners. It retains a few example answers to illustrate expectations and asks reviewers to flag ambiguous cases. The case shows how evaluation can accommodate legitimate variation without abandoning standards. It does not report a real educational result or establish that any particular rubric improves learning.

Common misconceptions and mistakes

NORA does not mean “no facts can be checked.” A creative explanation can still contain a wrong definition, and a flexible summary can still invent an event. Separate freedom of expression from correctness of content.

It also does not mean that all valid answers are equally useful. One may be clearer, more concise, or better suited to the user's needs. Those differences can be evaluated with explicit criteria, even when there is no unique ideal sentence.

Avoid assuming that a single model judge resolves subjectivity. Judges can favor particular styles or miss errors. Likewise, multiple references broaden coverage but cannot enumerate every acceptable response. Use disagreement analysis and representative examples to understand the limits of the evaluation.

Business use

Recognizing NORA tasks helps businesses design realistic acceptance criteria for drafting, assistance, summarization, and creative tools. It prevents useful outputs from being rejected solely because they differ from a reference's wording.

Define the nonnegotiable requirements and the dimensions where quality is a matter of degree. Collect examples from intended users and check consistency among reviewers. Report factual failures separately from style preferences where that distinction matters. The aim is a defensible judgment of usefulness, not artificial certainty that one answer is uniquely correct.

Check your understanding

A prompt asks for a two-sentence explanation of photosynthesis for beginners. Two responses use different examples but are accurate and satisfy the instructions. Must one be wrong because it differs from the reference?

Answer: No. The task allows multiple valid explanations. Assess the stated content and audience requirements rather than demanding one exact wording.

Related terms

Learn next

Choose a drafting prompt and write two substantially different acceptable responses. List the requirements both satisfy, then write one fluent but unacceptable response and explain exactly which requirement it violates.

Sources

Google, Machine Learning Glossary, no one right answer: https://developers.google.com/machine-learning/glossary#no-one-right-answer

The primary glossary supplies this generative-evaluation label and its contrast with one right answer. Rubric, examples, fictional case, and exercise are original teaching material. The label is not a claim that correctness is entirely subjective.

Previous: No-information Next: No-reference metric