Are all language models generative?

Many can generate sequences, but some modeling objectives primarily produce contextual representations or fill missing tokens and are not used as ordinary left-to-right generators.

Does a language model have to use whole words?

No. Tokens can be words, subwords, characters, bytes, or other units chosen by the system.

Does a high-probability answer have to be correct?

No. It can be a familiar or plausible continuation while making false or unsupported claims.

AI Glossary / Generative AI and language / Intermediate

Language model

Definition

A language model is a model of patterns in language, commonly expressed through probabilities over tokens or token sequences. An autoregressive language model estimates what token could come next given preceding context; other language-modeling approaches predict missing or corrupted text using different context arrangements. Language models range from small statistical systems to large neural networks. They can support generation, recognition, ranking, and representation learning, but probability of wording is not the same as truth of its claims.

What it means

Language contains regularities in word choice, order, grammar, style, and subject matter. A language model learns or estimates some of those regularities from data. Given a context, it can assign different scores to possible continuations, making some more likely than others. A system trained mainly on one kind of text may handle another kind poorly. The phrase “large language model” refers to a subset of language models; being a language model does not require a chatbot interface, enormous size, or a transformer architecture.

How it works

A statistical n-gram model uses a limited recent history to estimate the next item, usually with smoothing for sparse or unseen combinations. Neural models learn numerical representations and context-dependent computations. During training, the objective rewards predictions that fit the observed data under the selected modeling setup. During generation, a decoding procedure selects tokens from the model's scores. A complete application may add instructions, retrieval, tools, and output checks. Those surrounding components can change the user experience without being identical to the language model itself.

Three practical examples

Autocomplete: rank possible continuations of a partially written message while allowing the writer to accept, change, or ignore them.
Recognition support: help compare candidate transcriptions in speech or text recognition, alongside evidence from the input signal.
Draft generation: produce an answer or summary conditioned on a request and supplied context, followed by checks appropriate to the task.

Think of it this way

Think of a sophisticated guide to which expressions tend to fit together in particular contexts. It can suggest a plausible continuation without consulting a complete list of every possible sentence. The analogy is limited: modern language models learn complex representations, but their tendency to produce fitting language still does not make them an authoritative record of what happened in the world.

Worked example

Consider a toy autoregressive model for a two-word sentence. It assigns P(“orders” at the start) = 0.4, P(“arrive” after “orders”) = 0.5, and P(end after “orders arrive”) = 0.8. The probability of the complete sentence “orders arrive” with that ending is 0.4 times 0.5 times 0.8 = 0.16. If the model assigns P(“wait” after “orders”) = 0.3 and P(end after “orders wait”) = 0.9, the complete alternative has probability 0.4 times 0.3 times 0.9 = 0.108.
The model therefore scores the first complete sequence higher under these assumptions. That does not prove that any actual order arrived, or that “orders arrive” is the most useful response to a customer. The remaining probability at each step belongs to other possible tokens not listed in this small example. Omitting the end probability would score the prefix rather than the same complete event. In practical systems, words may be split into several tokens, so the sequence probability is calculated over the actual tokenization. Comparing raw probabilities across different lengths or tokenizers also requires care; a lower probability can reflect more factors in the product rather than worse task quality.

Illustrative case study

A fictional service team tests a language model trained on general text for drafting replies about industrial equipment. The model writes fluent responses but sometimes substitutes common meanings for specialist abbreviations. The team supplies approved reference material and evaluates the resulting application on realistic requests. It distinguishes improvements from better context retrieval from changes to the model itself. It also checks factual correctness separately from readability, because a polished draft can still contain an incorrect technical claim.

Common misconceptions and mistakes

A language model is not a database of verified answers and does not necessarily retrieve a source for each statement. Token probability is not calibrated factual confidence. Good language-model loss or perplexity does not guarantee good performance on every downstream task. A model can learn from text without having access to current information at deployment. Conversation memory, search access, and tool execution depend on the surrounding system. Generating an action description does not itself perform that action.

Business use

Select and evaluate a language model as part of a complete workflow. Identify the task, domain, language requirements, latency, cost, and evidence needs. Compare the quality of outputs on representative examples, including ambiguous and unfamiliar inputs. Keep model versions and important generation settings recorded. Use authoritative sources or deterministic operations where the task requires current facts or exact calculations, and verify that the final response accurately reflects those inputs and results.

Check your understanding

Why is the probability 0.16 in the example not evidence that an order arrived?

Answer: It is a probability assigned to a token sequence by the toy model. Whether an actual delivery occurred requires evidence about the delivery, not just a score for the sentence describing it.

Related terms

Learn next

Calculate a few short sequence probabilities from conditional token probabilities, including an end marker. Then compare an n-gram model's limited context with a neural model's learned representations. Study decoding and evaluation next, keeping language likelihood, factuality, and task success as distinct concepts that must be assessed with appropriate evidence.

Sources

Jurafsky and Martin, Speech and Language Processing, N-gram Language Models: https://web.stanford.edu/~jurafsky/slp3/3.pdf Hugging Face Transformers, Generation strategies: https://huggingface.co/docs/transformers/generation_strategies Vaswani and colleagues, Attention Is All You Need: https://arxiv.org/abs/1706.03762

Previous: Language identification Next: Language resource