Why are corners often good tracking points?

They typically contain gradients in multiple directions, providing better constraints on two-dimensional displacement than a flat patch or single straight edge.

Does Lucas - Kanade always produce dense flow?

No. It is often used to track selected points, though related formulations can be applied more densely.

Does a pyramid solve occlusion?

No. Coarser scales help with displacement size, but they cannot supply a correspondence for a feature that is no longer visible.

AI Glossary / Computer vision and multimodal AI / Intermediate

Lucas--Kanade method

Definition

The Lucas - Kanade method is a gradient-based approach to image alignment. In its common optical-flow form, it estimates a small displacement by assuming that neighboring pixels share approximately the same motion and that their brightness remains sufficiently stable between images. It combines local constraints using least squares.

What it means

Tracking one pixel is ambiguous: a single brightness measurement rarely tells us exactly where it moved. A surrounding patch provides additional evidence. Lucas - Kanade asks which small displacement best explains the observed changes across that patch.

The resulting vector describes apparent movement in image coordinates. It does not by itself reveal an object's physical speed, distance, identity, or three-dimensional trajectory. Camera motion can also create image movement.

The broader alignment framework supports different motion models and implementation variants. This entry emphasizes the widely used local translation model for tracking points between frames, often implemented with an image pyramid.

How it works

Compute spatial image gradients and the change between frames. Under a small-motion approximation, each pixel supplies an equation relating those measurements to horizontal and vertical displacement. Assume a shared displacement within the selected neighborhood and solve the combined equations by least squares.

Reliable estimation requires enough local variation in different directions. A flat patch provides little information. A single straight edge constrains movement across the edge more strongly than movement along it, producing the aperture problem.

An iterative implementation updates the alignment and estimates further corrections. A pyramid begins with coarser images, where a large original displacement appears smaller, then refines the result at finer scales. It extends the useful motion range but does not guarantee recovery from occlusion, severe appearance change, or an incorrect correspondence.

Three practical examples

1. Video stabilization: A program tracks suitable background points between frames and uses their motion to estimate camera movement. It rejects points on independently moving objects before deriving a global correction.

2. Laboratory motion analysis: A researcher follows textured points on a surface through a short image sequence. Image displacements become physical measurements only after appropriate calibration and timing information are supplied.

3. Image alignment: A document-processing prototype estimates small shifts between repeated captures. It checks the residual alignment error before combining images so failed tracks do not create doubled edges.

Think of it this way

Think of sliding a transparent patch over another photograph until their local markings line up. Small adjustments are guided by how the markings differ, and several markings provide more guidance than one uniform area.

The analogy ends where visual intuition would recognize objects. Lucas - Kanade uses numerical gradients and a motion model; it does not inherently understand which markings belong to the same physical thing.

Worked example

Use the simplified optical-flow constraint Ix × u + Iy × v = −It, where Ix and Iy are spatial gradients, It is the frame-to-frame intensity change, and u and v are displacement components under the chosen unit-time convention.

Suppose one local measurement gives gradients (2, 0) and It = −2. Its equation is 2u = 2, so u = 1. Another gives (0, 3) and It = −6, yielding 3v = 6, so v = 2. A third measurement with gradients (1, 1) and It = −3 gives u + v = 3, consistent with that result.

The estimated displacement is therefore one pixel horizontally and two vertically in this idealized local example. Real neighborhoods contain noise and inconsistent equations, so least squares seeks the best combined fit.

If every gradient had zero vertical component, the equations would provide no constraint on v. Adding more identical edge measurements would not resolve that missing direction of information.

Illustrative case study

Fictional case: A training-video studio uses point tracking to stabilize a close-up demonstration. Points on a plain tabletop drift, while points on a patterned tool remain easier to follow. However, the tool itself moves independently of the camera.

The team selects background features with sufficient texture and checks track consistency before estimating stabilization. When a hand hides a feature, the system discards that track and detects replacement points instead of extending the old path without evidence.

The scenario shows why trackability, object motion, and occlusion must be considered together. It does not report a real performance result.

Common misconceptions and mistakes

Do not assume every requested point will produce a valid correspondence. Use the implementation's status information and additional checks appropriate to the task.

Do not confuse small motion at one pyramid level with arbitrarily large motion in the original image. The initial alignment, texture, and pyramid design still matter.

Do not treat optical flow as ground-truth object motion. Shadows, reflections, lighting changes, and camera movement can alter apparent motion. Finally, repeated tracking can accumulate error; a visually plausible path may drift away from the original feature over time.

Business use

Lucas - Kanade is a useful building block for tracking and registration where local appearance remains sufficiently stable. It can support practical systems without requiring a learned motion model, but its assumptions must match the capture conditions.

Evaluate the fraction of usable tracks, localization error, drift, processing time, and behavior during occlusion. Include sequences with weak texture and changing illumination. For physical measurements, document calibration and frame timing separately from tracking accuracy. A two-pixel displacement is not a speed until its spatial and temporal meaning is established.

Check your understanding

A neighborhood contains only parallel vertical edges. Why might vertical displacement be difficult to estimate?

Answer: Its image gradients mainly constrain horizontal movement across the edges, leaving movement along them poorly determined.

Related terms

Optical flow
Image registration
Least squares
Image gradient
aperture problem
image pyramid
feature tracking

Learn next

Compare tracking on a flat patch, a straight edge, and a corner. Then inspect how a pyramidal implementation reports failed points and how forward and backward tracking can expose inconsistent correspondences.

Sources

OpenCV, Optical Flow, including the Lucas–Kanade derivation and implementation: https://docs.opencv.org/4.13.0/d4/dee/tutorial_optical_flow.html

Lucas and Kanade, original publication record: https://publications.ri.cmu.edu/an-iterative-image-registration-technique-with-an-application-to-stereo-vision-darpa

The official implementation tutorial was opened and read. The original PDF opened without extractable text; this explanation relies on the readable tutorial and standard derivation. The calculations and studio case are original.

Previous: LU decomposition Next: Lucky imaging