Does AED always use a transformer?

No. The architectural idea is attention between an encoder and decoder; implementations can use different neural components.

Does one acoustic frame produce one word?

No. Input frames and output tokens have different lengths and are related through the model's sequence processing and attention.

Is the most probable transcript necessarily correct?

No. It is the model's preferred sequence under its learned distribution and decoding procedure, which can differ from the words in the recording.

AI Glossary / Natural language and speech / Intermediate

AED

Definition

AED stands for attention-based encoder-decoder in the speech-recognition context of this glossary. It is a model architecture that converts an input sequence, such as acoustic features, into an output sequence, such as transcript tokens. The encoder represents the input, and the decoder uses attention to consult those representations while generating the output step by step.

What it means

Speech contains many acoustic measurements for each written word. An AED does not require every input frame to correspond to exactly one output token. It learns how to use the acoustic sequence when predicting a shorter text sequence.

The acronym has other meanings outside this context, so the expanded term should accompany it on first use. Here it names an architecture family, not a specific product, a recognition-quality guarantee, or one fixed implementation. Attention-based encoder-decoders can use different neural components and training choices.

How it works

An audio front end produces a sequence of representations, often after reducing the time resolution. The encoder transforms these into contextual representations. At each output step, the decoder combines information about previously generated tokens with information obtained through attention over the encoder's output.

The decoder predicts a probability distribution over possible next tokens. A decoding procedure selects an output sequence, ending when an appropriate stopping token is generated or another configured stopping rule applies. Training commonly uses paired audio and transcripts, with a loss that rewards the correct next token given the preceding reference tokens.

The output length can differ greatly from the input length. Attention weights provide weighted access to acoustic representations; they should not automatically be treated as exact word timestamps or a complete explanation of the model's decision. Standard full-context designs also require adaptation when low-latency streaming is needed.

Three practical examples

1. Recorded interviews: An AED recognizer can create a draft transcript for review. Names, overlapping speakers, and unclear audio still require attention from the person checking it.

2. Voice notes: A system can turn spoken notes into text that a user edits before sharing. Recognizing the words is separate from deciding what action the note requests.

3. Caption preparation: Transcription can support caption production, while timing, line breaks, speaker labels, and accessibility conventions may require additional processing beyond the AED's basic token sequence.

Think of it this way

Think of a writer consulting a set of notes while composing one word at a time. The notes resemble the encoded input, and consulting different parts resembles attention. The analogy is limited: the model operates on learned numerical representations, not consciously understood notes. It can produce a plausible continuation even when that continuation is not well supported by the recording.

Worked example

Consider a simplified invented decoder with two first-token options after an audio segment: “send” with probability 0.6 and “save” with probability 0.4. Suppose the next-token probability for “file” is 0.5 after send and 0.9 after save. The end-of-sequence probability after either two-word candidate is 0.8.

The complete candidate “send file” therefore has probability 0.6 times 0.5 times 0.8 = 0.24. The candidate “save file” has probability 0.4 times 0.9 times 0.8 = 0.288. Under these supplied probabilities, save file has the larger complete-sequence probability even though send was the most likely first token.

A greedy decoder that commits to send immediately can miss the better complete candidate. A search procedure retaining both first-token alternatives can compare them later, although a finite beam is not guaranteed to find the global best sequence in general.

These probabilities are invented for arithmetic clarity and omit other vocabulary outcomes from the display. Each conditional distribution would still need to allocate its remaining probability mass to other possibilities. The example explains autoregressive decoding; it does not demonstrate which phrase was actually spoken or whether the model's probabilities are calibrated.

Illustrative case study

Fictional case: Linden Training records short tutorial videos and tests an AED-based transcription component. A reviewer notices that a familiar product name is sometimes substituted for a less common name actually spoken.

The team evaluates representative recordings, keeps the audio available during review, and records the corrections required for domain terms. It assesses transcript quality and processing time separately. The case illustrates a practical evaluation workflow and reports no invented accuracy improvement or recommendation of a particular recognizer.

Common misconceptions and mistakes

Do not equate attention with perfect alignment. A token can use information distributed across several encoder positions. Do not assume that fluent text is faithful transcription: the decoder also learns patterns in text, which can favor plausible but incorrect sequences.

AED is not synonymous with every end-to-end speech recognizer. Other architectures and losses use different mechanisms. It also does not imply that a system supports streaming, speaker identification, or translation. Those capabilities depend on the full design, training data, and inference procedure.

Business use

Understanding AED helps teams ask useful questions when integrating transcription: what audio context is needed, how output is decoded, which languages and recording conditions were evaluated, and what additional components supply timestamps or speaker labels.

Assess performance on recordings resembling the intended use. Include domain vocabulary, interruptions, background noise, and relevant speaker variation. Review consequential word substitutions separately from aggregate error counts, and measure latency under the actual audio-processing configuration rather than inferring it from the architecture name.

Check your understanding

A decoder chooses the highest-probability first token. Does that guarantee the highest-probability complete transcript?

Answer: No. Later conditional probabilities affect the product for the entire sequence. The worked example shows a lower-probability first token leading to a higher-probability complete candidate.

Related terms

Learn next

Compare greedy decoding with beam search, then contrast attention-based recognition with CTC. Keep transcription, alignment, speaker attribution, and downstream interpretation separate when describing an application.

Sources

Source: Jurafsky and Martin, Speech and Language Processing, Chapter 16, Section 16.3, read for AED terminology, architecture, decoding, and training: https://web.stanford.edu/~jurafsky/slp3/16.pdf . The supplied full-book retrieval was unavailable; this author-hosted chapter was accessible. Candidate probabilities and the fictional case are original teaching examples, not benchmark results.

Previous: Adverse impact ratio Next: Affine bilateral grid (HDRnet)

AED