Why not update one α at a time?

With the others fixed, the equality Σ yiαi = 0 generally fixes that variable, leaving no feasible movement.

Does SMO choose the kernel?

No. Kernel selection is a separate modeling decision.

Can two implementations produce different numerical results?

Yes. Pair-selection rules, tolerances, scaling and numerical precision can differ, although properly converged solutions should satisfy the same specified optimization conditions.

AI Glossary / Mathematics and optimization / Advanced

Sequential minimal optimization

Definition

Sequential minimal optimization, or SMO, is an algorithm for training support vector machines by repeatedly solving very small pieces of their quadratic optimization problem. In the standard binary classification formulation, it updates two dual variables at a time while keeping the others fixed. John Platt introduced SMO in 1998 to avoid repeatedly invoking a general numerical quadratic-programming solver inside SVM training.

What it means

An SVM's dual variables determine how strongly training examples contribute to its decision function. They must satisfy both individual bounds and a shared equality constraint. Those restrictions explain the word minimal: changing only one variable would generally violate the equality constraint, while changing two permits a useful feasible step. Sequential refers to performing these small updates repeatedly. SMO is a training procedure, not a different prediction model or an alternative name for the kernel function.

How it works

For a standard soft-margin binary SVM, each dual variable αi lies between zero and C, and Σ yiαi = 0, with labels yi equal to −1 or +1. Select a pair, determine the feasible interval imposed by these constraints and optimize the quadratic objective along that interval. Clip an unconstrained candidate to the allowed range when necessary, then recover the other variable from the equality. Update the decision threshold and cached prediction errors. Repeat pair selection and optimization until the optimality conditions are satisfied within tolerance. Pair-selection heuristics, kernel evaluations and caching strongly affect practical speed; degenerate subproblems require appropriate handling rather than blindly dividing by a zero curvature term.

Three practical examples

A classifier for structured tabular data can use an SMO-based SVM solver, with feature scaling, regularization and validation chosen for the task.
A kernel-methods experiment can compare how pair-selection rules affect training effort while keeping the SVM objective and data fixed.
A teaching implementation can solve a tiny two-example problem exactly, then check dual feasibility and objective improvement before extending to larger data sets.

Think of it this way

Think of balancing two connected sliders on a board. Moving one alone breaks the balance rule, but coordinated movement of two can improve the setting while preserving it. SMO repeatedly chooses which pair to adjust. The analogy captures the shared constraint, but the real step follows a quadratic objective and numerical bounds; it is not merely a visual balancing exercise.

Worked example

Consider two one-dimensional training examples: x1 = −1 with y1 = −1, and x2 = 1 with y2 = +1. Use the linear kernel and C = 1. The equality constraint −α1 + α2 = 0 requires α1 = α2 = a, with 0 ≤ a ≤ 1. The primal weight is w = Σ αiyi xi = 2a. The dual objective to maximize is W(a) = 2a − 2a². Its derivative is 2 − 4a, giving a = 0.5, which is feasible. Thus α1 = α2 = 0.5 and w = 1. Using the decision convention f(x) = wx + b, symmetry gives b = 0; both examples have signed margin yi f(xi) = 1. Starting from a = 0, the pair update raises W from zero to 0.5. If C were instead 0.2, the same unconstrained optimum would lie outside the feasible interval and the solution would be clipped to a = 0.2. Then w = 0.4, illustrating that the regularization bound changes the trained model rather than merely controlling algorithmic speed. This two-example problem is a complete pair subproblem, not a claim that one update trains a general SVM.

Illustrative case study

In a fictional project, Birch Analytics tests an SVM for classifying equipment records. The team standardizes features using training data, chooses hyperparameters within a development procedure and then checks performance on separate records. Engineers monitor solver convergence independently from predictive quality. They discover that reducing optimization tolerance increases training work without necessarily improving held-out classification, so both quantities are reported separately. The example describes evaluation decisions and does not assert a measured accuracy or deployment outcome.

Common misconceptions and mistakes

SMO does not mean updating one training example independently. Its two-variable step is tied to the standard dual equality constraint. A tiny step size is not, by itself, proof of convergence; appropriate optimality checks are needed. A valid positive-semidefinite kernel gives the usual convex SVM training problem, but an arbitrary similarity function may not. The original algorithm avoids requiring the entire dense kernel matrix in memory, yet implementations may still allocate substantial caches. Historical benchmark speedups should not be treated as universal performance guarantees on modern data and hardware.

Business use

Use a well-tested implementation when SVMs suit the problem, and document kernel choice, C, feature preprocessing and stopping tolerance. Compare training cost and predictive performance with relevant alternatives. Inspect warnings about convergence and test whether results are stable under reasonable preprocessing changes. Distinguish optimization success from calibration, fairness, robustness and generalization: solving the SVM objective accurately establishes none of those automatically. For large data sets, measure kernel-evaluation and memory costs directly rather than inferring scalability from the simplicity of each pair update.

Check your understanding

The unconstrained pair optimum is a = 0.5 but the feasible interval is [0, 0.2]. What happens in the worked quadratic example?

Answer: The constrained optimum is the endpoint a = 0.2; using 0.5 would violate the box bound.

Related terms

Support vector machine
Quadratic programming
Kernel function
dual variable
Karush–Kuhn–Tucker conditions
coordinate descent

Learn next

plot W(a) = 2a − 2a² on [0, 1] and then restrict the allowed interval to [0, 0.2]. Verify the equality constraint after every proposed pair change. Then study how a solver selects pairs when many examples violate the optimality conditions.

Sources

Sources: John C. Platt, Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines, Microsoft Research technical report MSR-TR-98-14, https://www.microsoft.com/en-us/research/uploads/prod/1998/04/sequential-minimal-optimization.pdf ; publication record, https://www.microsoft.com/en-us/research/publication/sequential-minimal-optimization-a-fast-algorithm-for-training-support-vector-machines/ . The two-point calculation and fictional classifier study are original.

Previous: Sequential importance sampling Next: Sequential model-based optimization