Does the theorem provide a ready-made best code?

No. It establishes achievable limits and impossibility boundaries; efficient finite-length coding is a separate design problem.

Can redundancy increase useful reliability without making information rate zero?

Yes, that is one of the theorem's central conclusions below capacity.

Does a noisy channel always have positive capacity?

No. If the output carries no information about the input under the allowed conditions, capacity can be zero.

AI Glossary / Mathematics and optimization / Intermediate

Noisy-channel coding theorem

Definition

The noisy-channel coding theorem states that reliable digital communication is possible over a noisy channel at information rates below its capacity, using suitable coding and sufficiently long blocks. In a standard discrete memoryless setting, decoding error can be made arbitrarily small below that limit. Rates above capacity cannot achieve vanishing error under the same channel model and assumptions.

What it means

Noise can change transmitted symbols, yet adding carefully structured redundancy lets a receiver infer the intended message with high reliability. The surprising point is that reliability need not require the information rate to approach zero. A positive rate can remain possible as error becomes very small. Capacity is the boundary defined by the channel's statistical behavior and constraints. The theorem concerns mathematical possibility, not a promise that any particular code, device or short message will reach that boundary. It is also different from compressing a source to remove redundancy.

How it works

An encoder maps each message to a longer sequence of channel inputs. Noise produces a received sequence, and a decoder chooses the most plausible original message using the code structure. For a discrete memoryless channel, capacity is the maximum mutual information between input and output over allowed input distributions, measured in bits per channel use when logarithms use base two. Achievability arguments show that suitable families of codes exist below capacity as block length grows. A converse establishes the limitation above capacity. Practical designs must additionally address finite block lengths, decoding complexity, latency, channel estimation and changing conditions.

Three practical examples

A communication engineer compares an error-correcting code's information rate and measured error performance against the capacity of a specified channel model.
A storage system uses redundancy to recover data despite read errors, treating the storage process as a noisy communication channel.
A sensor link balances message rate, delay and reliability when sending measurements to an AI processing system. Channel coding protects transmitted symbols; it does not improve the truth or usefulness of the original measurement.

Think of it this way

Think of sending instructions using an agreed vocabulary of sufficiently distinct message patterns. Even if some parts are corrupted, the recipient may recognize the intended pattern. The analogy captures deliberate redundancy, but the theorem uses statistical channel models and long coded sequences, not merely repeating ordinary sentences or relying on a person's common sense.

Worked example

Take a binary symmetric channel that independently flips each transmitted bit with probability p = 0.1. Its capacity is C = 1 − H2(p), where H2(p) = −p log2(p) − (1 − p) log2(1 − p). Here H2(0.1) is approximately 0.4690, so C is approximately 0.5310 information bits per channel use. A code carrying 400 information bits in 1,000 transmitted bits has rate 0.4, below capacity. The theorem supports the existence of increasingly reliable code families at that rate as block lengths grow; it does not specify the error probability of this particular 1,000-bit block code. A rate of 0.7 exceeds capacity and cannot achieve vanishing decoding error under this model. If the channel supports 1,000 independent uses per second, the corresponding capacity is about 531 information bits per second, before unrelated system overhead is considered.

Illustrative case study

In a fictional project, Beacon Instruments tests a telemetry link for remote equipment. Engineers model its errors and compare several coding configurations at the required message delay. A longer code improves simulated reliability but adds latency beyond the application target, so the team evaluates another operating point. They also test whether burst errors violate the initial independent-flip model. The scenario illustrates how a theoretical limit informs design without replacing measured finite-length performance. No real deployment result is asserted.

Common misconceptions and mistakes

Below capacity does not mean every transmitted block is error-free. Arbitrarily small error is a limiting statement involving suitable coding, not a zero-error guarantee at any fixed length. Capacity is not always the channel's raw symbol rate, and one channel use need not equal one information bit. The familiar bandwidth-times-logarithm formula applies to a particular Gaussian-noise channel model with stated power constraints, not every noisy channel. Compression, encryption and channel coding have different purposes, even when a communication system uses all three.

Business use

Use capacity analysis to assess whether a communication requirement is plausible under a defensible model and to compare coding efficiency. Record units carefully: bits per use and bits per second require different contextual information. Test the complete system for error rate, delay, energy and computational cost. If the error process changes with interference or operating environment, revisit the model rather than treating one capacity number as permanent. For AI applications, distinguish communication reliability from model reliability: perfectly delivered input can still lead to an incorrect prediction.

Check your understanding

A binary symmetric channel flips bits independently with probability 0.1. Does transmitting uncoded information at one bit per channel use become reliable merely by sending a longer message?

Answer: No. One bit per use exceeds its approximately 0.531-bit capacity, and uncoded longer messages do not supply the required error-correcting structure.

Related terms

Mutual information
channel capacity
error-correcting code
code rate
binary symmetric channel
source coding

Learn next

calculate binary symmetric channel capacity at error probabilities 0, 0.1 and 0.5. Then compare a simple repetition code with a more efficient block code, distinguishing information rate, bit error and block error. Keep finite examples separate from asymptotic existence results.

Sources

Sources: Claude E. Shannon, A Mathematical Theory of Communication, sections 12–14 and Theorems 11–12, https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf ; original publication record, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x . The binary-channel calculation is an original worked application under explicitly stated assumptions.

Previous: Noise-to-signal ratio Next: Noisy channel model