Is a visual routine the same as an image classifier?

No. Classification assigns labels, while a routine can establish task-specific properties or relationships through a sequence of operations.

Must the sequence be hand-written?

The core idea concerns composing operations. Particular computational systems may specify, select or learn aspects of their procedures.

Does the framework prove how people see?

No. It offers a way to investigate visual computation. A successful machine procedure alone does not prove that human perception uses the same mechanism.

AI Glossary / Computer vision and multimodal AI / Intermediate

Visual routine

Definition

A visual routine is a sequence of elementary visual operations used to determine a shape property or spatial relationship. In Shimon Ullman's framework, reusable operations are combined for tasks such as deciding whether something is inside a boundary, connected to another point or positioned above another object.

What it means

Recognizing objects is only one part of interpreting an image. A system may correctly label a cup and a tray yet fail to determine whether the cup sits inside the tray's boundary. Visual routines address this kind of task-directed spatial analysis.

The central idea is compositional: a limited collection of operations can support many questions through different combinations. The same image can be examined differently depending on what needs to be established.

This framework originated in research on human vision. A software implementation does not establish that the brain uses the same procedure, nor does every visual-processing function demonstrate the theory.

How it works

A routine operates on a visual representation, such as extracted contours or regions. The task determines which items matter and which sequence of operations should be applied.

Possible operations include selecting a location, following a boundary, marking a visited place and spreading activation through a region without crossing its boundary. Intermediate results allow later steps to depend on earlier ones.

For example, a routine might select a marker, explore the region around it and test whether another marker belongs to the same region. A different routine could follow a contour to determine whether it returns to its starting point.

These procedures depend on the quality and meaning of their input representation. A break in an extracted boundary can change the answer even when a person perceives a closed shape. Handling such uncertainty is part of designing a useful implementation.

Three practical examples

A diagram-analysis prototype checks whether two highlighted locations belong to the same enclosed area, using region selection and connectivity operations.
A robotic inspection system combines feature selection and spatial tests to assess whether a part lies within a designated fixture boundary.
An educational vision experiment compares procedures for following a contour, finding its endpoint and determining whether it forms a closed loop.

Think of it this way

Think of a short set of instructions for examining a map: find the marked point, follow its road, remember the junction and check whether the destination is reachable.

The analogy is limited because visual routines operate on representations and may include operations unlike conscious map-reading. A useful computer implementation also does not establish how a human observer internally solves the same problem.

Worked example

Testing a spatial relationship

Imagine a simple diagram represented by a 7-by-7 grid. A closed rectangular boundary encloses nine interior cells. Marker A is at the center, marker B is in another interior cell and marker C is outside the rectangle. Assume boundaries are unambiguous and movement between cells uses four-neighbor connectivity.

A demonstration routine starts at A, marks its cell and repeatedly visits unmarked neighboring cells without crossing the boundary. It eventually marks all nine interior cells. Testing the marker positions then gives: B is reached; C is not reached.

For this representation, the routine supports the conclusion that A and B occupy the same enclosed region. It does not need to identify the region as a particular named object.

Now remove a boundary segment that creates a traversable connection to the exterior. The same spreading procedure can reach C. The algorithm has answered consistently with the changed representation, even if the missing segment came from a scanning error.

This invented demonstration illustrates the need for a suitable sequence and trustworthy inputs. It does not claim that human vision visits grid cells in this order.

Illustrative case study

Illustrative case study - fictional

A fictional technical-training company builds a tool for checking simplified wiring diagrams. Its first model recognizes symbols correctly but sometimes links labels to the wrong wire at crossings.

The team adds a task-specific analysis that selects a labeled endpoint, follows the represented connection and records relevant junction decisions. It tests ordinary crossings separately from junctions marked as connected. When the diagram is ambiguous, the tool requests review instead of treating every crossing as a connection.

The improvement in design is the explicit treatment of relationships and intermediate decisions. No measured accuracy gain is asserted here. The example also shows why recognizing each symbol is insufficient for answering a question about connectivity.

Common misconceptions and mistakes

A visual routine is not simply a person's daily habit of looking at something.
Object labels alone do not establish all spatial relationships.
A proposed set of elementary operations is a theoretical account, not a complete verified description of the brain.
A correct sequence can still fail when contours, regions or connectivity are represented incorrectly.
The framework does not require that every task use the same sequence or that every operation be implemented without learning.

Business use

The concept is useful when evaluating systems that must answer structured visual questions rather than merely name objects. It encourages teams to specify the relationship to be tested, the evidence used and the intermediate operations that support the answer.

For business workflows, select representative diagrams or scenes and test the actual relations that matter: containment, adjacency, alignment or connectivity. Include confusing cases such as crossings, occlusions and incomplete boundaries. Review whether failures arise in perception, representation or the decision sequence before choosing a remedy.

Check your understanding

A system recognizes every symbol in a circuit diagram. Does that guarantee it knows which symbols are connected?

Answer: No. Connectivity requires analysis of spatial relationships and the diagram's representation, beyond recognizing individual symbols.

Related terms

Spatial reasoning
Attention
Region growing
visual cognition
contour tracing
compositional reasoning

Learn next

Write a small sequence for answering whether two markers share an enclosed region. Test it on a clean boundary, a broken boundary and overlapping regions. Identify which failures require better image evidence and which require a different routine.

Sources

Sources: Ullman, Visual routines, original 1984 research record: https://doi.org/10.1016/0010-0277(84)90023-4 ; Ullman, Visual Cognition and Visual Routines, High-Level Vision, chapter 9: https://courses.csail.mit.edu/6.803/pdf/ullman9.pdf . The original abstract and relevant chapter discussion of routines, region activation and spatial analysis were reviewed.

Previous: Visual microphone Next: Visual temporal attention