Must a pattern grammar be written manually?

No. Some systems learn aspects of the structure or its probabilities, while others use manually specified rules.

Is it limited to flat images?

No. Grammatical models can describe other structured data, including shapes, sequences and hierarchical arrangements.

Does parsing recover the only possible explanation?

Usually not. Several arrangements can explain uncertain evidence, and a system may compare or retain multiple candidate parses.

AI Glossary / Computer vision and multimodal AI / Intermediate

Pattern grammar

Definition

A pattern grammar is a set of rules describing how simpler components can combine into structured patterns. In computer vision, grammatical models can represent arrangements of shapes, object parts or scene elements and support the interpretation or generation of those arrangements.

What it means

The idea borrows from language: a sentence has parts and relationships, rather than being an unordered bag of words. A visual object can likewise be described through components and their organization. A simple building grammar might describe floors containing rows of windows. More expressive models can allow alternatives, repetition and uncertainty. The term covers a family of representations rather than one universally defined algorithm. In this glossary it refers to grammatical descriptions of visual structure, not a regular-expression pattern used only to match text. The appropriate grammar depends on what the application considers a meaningful pattern.

How it works

A designer or learning process specifies elementary components, larger structures and rules for combining them. A rule may say that a façade contains a roof and a set of floors, while each floor contains a row of windows. Spatial constraints can express alignment, adjacency or relative size. A stochastic grammar assigns probabilities to alternatives, allowing common arrangements to receive different weights from rare ones.
Parsing searches for a rule-based explanation of observed evidence. Generation starts from a high-level symbol and expands rules to construct an arrangement. Visual parsing must also handle uncertain component detections, occlusion and variation. Grammar rules alone do not solve those measurement problems, and an incorrect rule system can produce a confident but unsuitable explanation.

Three practical examples

A document-layout system models a page as a heading, several sections and optional figures.
An architectural research tool describes repeated floors and windows to organize façade observations.
A design application generates families of structured shapes by selecting among permitted component arrangements.

Think of it this way

Think of a recipe for assembling a model building from named pieces. The recipe tells you which pieces can be combined and how their relationships matter. Unlike a physical kit, a visual grammar may include probabilities and tolerate uncertain observations. It also cannot guarantee that the available pieces capture every real building or explain all the pixels in a photograph.

Worked example

Consider a deliberately simple grammar: a sign contains a border followed by a message region, and the message region contains either one line or two lines of text. Assign probabilities 0.7 and 0.3 to those two layout alternatives. Before examining an image, the one-line layout is the more probable option under this model.
Now suppose the image evidence fits the two-line layout much better. A parser should combine the layout probabilities with the evidence, rather than always choosing the more common rule. If the model does not allow three lines at all, a genuine three-line sign may be misinterpreted or rejected. Adding probabilities cannot compensate for a missing structural possibility.

Illustrative case study

In a fictional archive project, a team processes historical forms whose sections usually appear in a repeated order. A visual grammar describes a title, identity fields, a table and an optional stamp area. The team uses it to organize detected regions and identify pages that do not fit the expected structure. An unusual form is sent for review rather than forced into the standard template. The example illustrates how structural expectations can support a workflow while also requiring exceptions. It does not imply that every archive collection follows the same layout or that the grammar has been empirically validated.

Common misconceptions and mistakes

A grammar is not simply a list of object labels. The rules and relationships carry much of its meaning. A grammar that can generate a shape does not necessarily identify that shape reliably in a noisy image. Probabilities attached to rules are model quantities, not automatic guarantees about real-world frequency or confidence. Another mistake is making the rules so rigid that ordinary variation appears invalid. Conversely, excessively permissive rules can explain nearly anything and lose useful discrimination. More rules are therefore not always better rules.

Business use

A grammatical approach can be useful when recurring structure matters and people need understandable explanations. Teams should compare it with simpler templates, geometric constraints or learned models on representative examples. Evaluate both successful parsing and graceful handling of unfamiliar arrangements. Track whether failures come from component detection, spatial relationships or the grammar's coverage. Maintaining the rules also has a cost when product designs, document formats or environments change. The benefit should be assessed against that maintenance burden.

Check your understanding

A grammar allows a page to contain a title and exactly two panels. What should a system consider when shown a valid page containing three panels?

Answer: The grammar may lack the necessary structure. The system should allow an exception or revise the model rather than assuming the observed page must be wrong.

Related terms

Syntactic pattern recognition
Parse tree
Stochastic grammar
visual grammar
production rule
scene parsing
part–whole hierarchy
structural representation

Learn next

Learn parse trees and production rules next. Draw two alternative descriptions of the same simple layout, then identify which observations would distinguish them. After that, explore how a stochastic grammar combines structural alternatives with image evidence.

Stochastic grammar

Sources

Zhu and Mumford, A Stochastic Grammar of Images, author publication record and manuscript link: https://dash.harvard.edu/entities/publication/73120378-84d2-6bd4-e053-0100007fdf3b VLGrammar, research on grounded grammar induction: https://arxiv.org/abs/2103.12975 Primary-source search excerpts consulted; full source and editorial review remain pending.

Previous: PATR-II Next: Pattern matching