Does horizontal autoscaling always use CPU?

No. Supported systems can use other workload, custom or external metrics.

Can it scale a stateful service?

Sometimes, but replication, partitioning and state management must support the chosen pattern.

Does it replace node or infrastructure scaling?

No. Workload replicas and the underlying capacity are related but distinct control problems.

AI Glossary / Data engineering and architecture / Intermediate

Autoscaling, horizontal

Definition

Horizontal autoscaling automatically increases or decreases the number of running instances that share a workload. It adds instances when the scaling policy calls for more capacity and removes them when fewer are needed. The change is in the number of workers or replicas, rather than the size of each individual worker.

What it means

A service might run three application replicas during ordinary demand and six during a busy period. Requests must be distributed across those replicas so that additional capacity can help.

Autoscaling adds an automatic control process to horizontal scaling. Manually starting two more servers is horizontal scaling, but it is not itself an automatic response governed by an autoscaler. A policy may respond to measured demand, scheduled expectations or other supported signals.

More replicas do not guarantee proportional speed. A shared database, model-serving endpoint or serial task can remain the bottleneck. Some workloads also require coordination or state partitioning before they can safely run across more instances.

How it works

The autoscaler observes a chosen metric and compares it with a policy target or threshold. It calculates a desired instance count subject to limits and asks the platform to make the change. New instances must then start, become ready and receive useful work.

CPU utilization can be a suitable signal for some services, while queue depth, processing lag or request load may be better for others. The metric must relate meaningfully to capacity. Low CPU does not necessarily mean a service is healthy if it is waiting on a constrained external system.

Scale-in needs care too. Workers may need to finish tasks, checkpoint progress or drain connections before termination. Stabilization windows and scaling-rate controls can reduce repeated expansion and contraction caused by noisy measurements.

Application replicas also need somewhere to run. In Kubernetes, increasing Pod replicas and adding cluster nodes are distinct operations. A requested replica can remain pending if the cluster lacks appropriate resources, even though the workload autoscaler has made its decision correctly.

Three practical examples

A document-processing service adds queue workers during a large import and removes excess workers after the backlog subsides.
A customer portal adjusts its application replicas as request demand changes, while monitoring shared database capacity.
An AI inference service adds model-serving replicas when supported infrastructure, model-loading time and request distribution make replication useful.

Think of it this way

Think of opening more checkout lanes when a shop gets busy. Each lane remains roughly the same size, but more customers can be served simultaneously if enough staff and equipment are available.

The analogy exposes a limit: opening lanes does not fix a single failing payment network. Likewise, adding application instances cannot resolve every downstream bottleneck. New lanes also take time to prepare, just as new replicas require startup, readiness checks and sometimes large model downloads.

Worked example

A capacity estimate, not a performance promise

Suppose an invented worker service can sustainably process 40 independent tasks per minute per ready worker under a tested workload. Incoming demand rises to 210 tasks per minute. A simple planning calculation gives 210 divided by 40, or 5.25, so at least six workers are needed for nominal capacity above the incoming rate.

Six workers offer a nominal 240 tasks per minute. That leaves 30 tasks per minute to reduce an existing backlog, assuming the tested rate remains valid and no shared bottleneck appears.

If 300 tasks are already queued, that spare capacity would take about ten minutes to clear them under those simplified conditions. Starting the six workers does not make the backlog disappear immediately.

If startup takes two minutes, demand can add further backlog before the new capacity becomes productive. The policy must account for this delay and may need headroom or earlier scaling. The numbers are illustrative; actual task durations and contention require measurement.

Illustrative case study

A fictional training website scales application replicas when CPU rises. During a campaign, response times deteriorate while CPU stays modest because every replica waits for an overloaded shared database.

The team reviews request traces and database saturation before changing the scaling policy. It reduces avoidable repeated queries and evaluates the database capacity separately. It then tests how additional application replicas affect the whole request path instead of assuming a larger replica count must improve latency.

The case illustrates why an autoscaler needs both a useful signal and a scalable workload. A controller can follow its rule faithfully while the service continues missing its user-facing target.

Common misconceptions and mistakes

Horizontal autoscaling makes each machine larger. That is vertical scaling; horizontal scaling changes the number of instances.
A requested instance is immediately usable capacity. Provisioning, scheduling, startup and readiness can introduce delay.
Scaling in is just deleting idle-looking workers. Long-running work and connections may need controlled completion or transfer.

Business use

Horizontal autoscaling can align capacity with changing demand and reduce unnecessary running resources. It is most effective when work can be shared, state is handled deliberately and scale-in is safe.

Define a service target such as acceptable queue delay or response latency, then choose scaling signals that reflect it. Set minimum and maximum capacity, understand quotas and test demand spikes as well as quiet periods. A maximum protects a boundary but can also leave demand unserved; monitor that condition.

For AI workloads, include model-loading costs, accelerator availability and batching effects. Several small replicas may behave differently from one larger instance. Track cost per useful completed request alongside latency and error rates, rather than treating instance count as a success metric.

Check your understanding

Six workers each process 40 tasks per minute while 210 new tasks arrive per minute. What is the nominal backlog-reduction rate under the worked assumptions?

Answer: 30 tasks per minute, because 240 minus 210 equals 30. Startup delays and shared bottlenecks can reduce that rate.

Related terms

Replica
Autoscaling
vertical scaling
load balancing
queue depth
readiness
scale-in
node autoscaling
capacity planning

Learn next

Read autoscaling metrics and graceful shutdown. Trace one workload from incoming demand through ready workers to its shared dependencies, then test where adding a replica actually changes capacity.

Replica

Sources

Kubernetes, Horizontal Pod Autoscaling: https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/. Primary documentation of replica-based control and metrics.

Kubernetes, Node Autoscaling: https://kubernetes.io/docs/concepts/cluster-administration/node-autoscaling/. Primary distinction between workload demand and node capacity.

AWS, Scaling cooldowns: https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-scaling-cooldowns.html. Primary example of warmup and scaling timing. The queue calculation and training-site case are original illustrations.

Previous: Autoregressive image model Next: Autoscaling, vertical