AI Glossary / Computer vision and multimodal AI / Advanced
Segmentation
Definition
In computer vision, segmentation divides an image or related visual representation into regions or assigns labels at a fine spatial level, commonly per pixel. The intended regions may represent foreground and background, semantic categories or individual objects. Segmentation is a task family rather than one algorithm, and the appropriate output depends on what the application needs to distinguish.
What it means
Recognizing that an image contains a tree differs from identifying which pixels belong to it. Segmentation supplies spatial detail that can support editing, counting or measurement. However, an outlined region is still an estimate under a definition. Ambiguous boundaries, occlusion and annotation conventions affect what counts as correct. A mask of visible pixels also does not necessarily describe the full hidden extent of an object. The task must specify whether it concerns visible surfaces, complete inferred objects or another kind of region.
How it works
Methods range from thresholds and region-based procedures to learned models and interactive tools. A binary output can separate a target from its background. Semantic segmentation assigns category labels, while instance segmentation distinguishes separate objects within a category. Training-based approaches learn from examples and annotations; other methods rely more directly on image properties or user guidance. Predictions may require refinement, but postprocessing can also remove valid details. Evaluation should match the output type, considering region overlap, boundaries, missed objects or other errors relevant to the intended use.
Three practical examples
- A photo-editing workflow creates a foreground mask to change a background while inspecting fine boundary details.
- A warehouse application separates visible parcels into individual masks before estimating counts and occupied image area.
- An agricultural imaging exercise labels vegetation pixels while checking how shadows and soil color affect the result.
Think of it this way
Imagine placing transparent colored sheets over a photograph to mark which areas belong to different subjects. The sheets describe where the chosen regions are, but the person placing them must decide how to handle fuzzy edges and overlapping objects. A segmentation system performs a comparable spatial labeling task. The analogy highlights that a neat overlay is not enough: the region definition and the handling of uncertain boundaries determine whether the overlay serves its purpose.
Worked example
In an original binary-mask example, a reference region contains 100 pixels and a predicted region contains 90. Their overlap is 80 pixels, so their union contains 100+90−80=110. Intersection over union is therefore 80/110, approximately 0.727. The prediction misses 20 reference pixels and includes 10 pixels outside the reference. This calculation summarizes overlap for one region under the stated reference. It does not reveal where the errors occur. Missing a narrow connection or a critical boundary can matter differently from the same number of errors spread through an unimportant area.
Illustrative case study
A fictional recycling facility tests a model that marks items on a conveyor. Large isolated items receive plausible masks, but touching objects sometimes merge into one region. The team recognizes that its counting task needs instance-level separation rather than only category coverage. It evaluates merged and missed objects separately, improves the annotation guidance and tests on representative conveyor scenes. This scenario illustrates how the business objective shapes the segmentation task. It does not imply that a strong pixel-overlap score alone establishes reliable object counts or operational performance.
Common misconceptions and mistakes
Segmentation is not the same as image classification or a bounding-box prediction. A mask offers different spatial information, but it is not automatically more accurate. Another mistake is assuming semantic labels separate every object instance. Annotation quality also matters: inconsistent boundary rules can produce inconsistent training and evaluation. High overall pixel accuracy may hide poor performance on small target regions when background dominates. Finally, converting pixel area into physical area requires appropriate scale or geometry; counting mask pixels alone does not establish a measurement in square centimeters.
Business use
Define exactly which regions must be identified and what decisions depend on them. Select evaluation measures that expose costly failures, such as merged objects, omitted small regions or inaccurate boundaries. Review performance across representative lighting, backgrounds and object sizes. If masks support measurement, validate the full measurement pipeline, including calibration and geometric assumptions. Preserve a path for uncertain cases when mistakes have meaningful consequences. Segmentation becomes useful when its output matches an operational requirement, rather than merely producing a visually convincing colored overlay.
People also ask
Does segmentation require deep learning?
No. Thresholds, region methods and interactive approaches can also produce segmentations.
Does semantic segmentation count separate objects?
Not necessarily; it assigns categories without inherently distinguishing every instance.
Is a good overlap score sufficient?
No. Boundary placement, object separation and application-specific errors may also matter.
Check your understanding
What was the example's union size?
Answer: One hundred ten pixels.
How many reference pixels were missed?
Answer: Twenty.
Why could two masks with the same overlap score differ in usefulness?
Answer: Their errors can occur in different places and affect the downstream task differently.
Related terms
- Intersection over union
- Semantic segmentation
- instance segmentation
- binary mask
- object detection
- boundary accuracy
Learn next
Draw a small reference mask and two predictions with the same overlap count but different error locations. Compute their intersection over union, then compare their suitability for counting, editing and measurement. State which additional checks each application would require before using the masks.
More in Computer vision and multimodal AI
- Scatter matrix - A scatter matrix summarizes multivariate variation through sums of outer products of centered observation vectors.
- Scene text
- Seam carving - Seam carving is a content-aware image-resizing technique that removes or inserts connected paths of pixels called seams.
- Self-similarity matrix - A self-similarity matrix compares elements of a sequence or collection with other elements of the same sequence or collection.
- Semantic gap - The semantic gap is the mismatch between information represented or extracted by a computational system and the meaning people attach to that information in...
- Separable filter - A separable spatial filter has a multidimensional kernel that can be expressed as a product of lower-dimensional kernels.
Sources
MathWorks, Get Started with Image Segmentation: https://www.mathworks.com/help/images/getting-started-with-image-segmentation.html The mask-overlap calculation, transparent-sheet analogy and recycling-facility case are original educational illustrations. Segmentation is discussed in the computer-vision sense supplied by the category, rather than customer or market segmentation.
Previous: Seed tuples Next: Selection