AI Glossary / AI and machine learning / Advanced
Programming by example
Definition
Programming by example is an approach to creating programs in which a user supplies examples of the desired behavior and a system infers a program consistent with them. Often the examples pair an input with its intended output. The inferred program can then perform the same kind of transformation on new inputs.
What it means
Instead of describing every operation, a person shows a few results they want. For example, they might demonstrate how a reference code should be extracted from a longer description. The system searches for a reusable rule that explains these demonstrations. Several rules may fit the same examples, so matching them does not prove that the system has understood the person's full intention. The practical skill is choosing examples that expose important differences, then checking the proposed behavior beyond those examples.
How it works
A system defines a space of possible programs, often using a domain-specific language containing useful operations for the task. It searches for candidates consistent with the examples, ranks plausible alternatives, and presents results or asks for further clarification. Additional examples eliminate candidates that make the wrong prediction. Gulwani's research describes this combination of synthesis, ranking and interaction in data-wrangling systems. The generated program may be compact and deterministic even when learning helps choose it. Its reliability depends on the available operations, the examples and the evaluation process.
Three practical examples
- A sales administrator demonstrates how to extract an account code from several differently formatted reference strings, then previews the inferred extraction across the full column.
- An operations analyst marks the relevant fields in sample machine logs. A synthesis tool proposes rules for extracting comparable fields from other records.
- A small business owner shows how filenames should be standardized. Before applying the inferred transformation, they inspect files with missing dates, unusual punctuation and already standardized names.
Think of it this way
Imagine teaching someone how to sort a drawer by arranging a few sample items. They may infer your intended categories, or they may notice an accidental pattern such as color. More carefully chosen examples resolve the ambiguity. The analogy has limits: a synthesis system's possible interpretations depend on its program language and search procedure, not ordinary human understanding.
Worked example
Suppose the input “Order: AB-104 | paid” should produce “AB-104,” and “Order: CD-205 | open” should produce “CD-205.” A rule that takes six characters after the prefix fits both examples. So does a rule that takes everything between the prefix and the vertical bar, trimming spaces. Now show “Order: XYZ-1007 | paid” with the desired output “XYZ-1007.” The fixed-length rule fails, while the delimiter-based rule still works. A fourth example with no vertical bar reveals another unresolved situation. The system should request a decision or flag the record rather than silently assume the intended exception behavior.
Illustrative case study
A fictional supplier-management team receives weekly spreadsheets with identifiers embedded in descriptions. An employee demonstrates the extraction on a small sample, then reviews a preview of the remaining rows. Tests include long identifiers, blank cells and records containing two candidate codes. Ambiguous records go to a separate queue, and the original data stays available for comparison. The team measures corrections and time saved over several deliveries before relying on the process. This illustrates how useful automation comes from examples plus validation, rather than treating the first plausible generated rule as finished.
Common misconceptions and mistakes
Programming by example is not simply recording a sequence of mouse clicks; it aims to infer behavior from demonstrations or example specifications. It does not necessarily require a large language model. A perfect fit to five examples can still conceal the wrong general rule. Randomly adding many similar examples may help less than one example that distinguishes competing interpretations. Avoid testing only the rows used to create the program. Also avoid allowing a preview to overwrite valuable originals before exception handling and recovery are clear.
Business use
This approach can make repetitive transformations accessible to people who understand the data but do not write code. Good candidates have recognizable patterns, observable outputs and affordable review. Before adoption, define the acceptable error rate and the cost of silent mistakes. Retain representative test examples as the input format changes, and assign someone responsibility for maintaining the transformation. Time savings should include review and corrections. For irregular or consequential tasks, an explicit rule or a specialist implementation may provide better control.
People also ask
How many examples are needed?
There is no universal number; the ambiguity of the task and the system's search space matter.
Is the result a real program?
In program synthesis systems, yes, although the interface may show only its outputs.
Why does an extra example change earlier results?
It may cause the system to choose a different candidate program that better matches the expanded specification.
Check your understanding
Why do two equal-length sample identifiers fail to establish a variable-length extraction rule?
Answer: Both a fixed-position rule and a delimiter rule may explain them.
What should you test next?
Answer: Use examples with different lengths, missing separators and multiple possible matches. Explain why reviewing unseen inputs provides evidence that simply rechecking the demonstrations cannot provide.
Related terms
- Inductive programming
- Data wrangling
- Active learning
- Program synthesis
- domain-specific language
- test cases
Learn next
Learn how input-output specifications differ from complete logical specifications. Then explore candidate ranking and ambiguity resolution. Practice building a small collection of normal, boundary and exception examples for a transformation you understand well.
More in AI and machine learning
- Probability matching - Probability matching is a choice pattern in which alternatives are selected in proportions resembling their perceived probabilities or observed frequencies.
- Procedural memory - Procedural memory, in an AI agent, is the stored knowledge of how to do things: reusable routines, skills and rules for carrying out tasks, as distinct from...
- Product of experts - A product of experts is a probabilistic model that combines several component models by multiplying their scores for the same possible outcome and then...
- Proper generalized decomposition - Proper generalized decomposition, or PGD, is a numerical modeling approach that approximates a solution as a sum of products of simpler functions.
- Prototype methods - Prototype methods are machine-learning approaches that represent categories or patterns through selected or learned representative examples, called prototypes.
- Proximal gradient methods for learning - Proximal gradient methods are optimization algorithms that combine a gradient step for a smooth part of an objective with a proximal step for another part...
Sources
Sumit Gulwani, Programming by Examples, including data-wrangling applications and ambiguity resolution: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/12/pbe16.pdf The identifier example and business scenarios above are original illustrations. They do not describe measured results from a specific commercial implementation. Independent review consulted Gulwani's primary chapter, including sections 8 and 9 on ranking and user interaction. These support ambiguity resolution and the distinction between consistency with demonstrations and intended generalization. The extraction example was independently traced.
Previous: Program dependence graph Next: Progression planning