Information Theory and the Shannon–Hartley Law

Information Content

Information theory quantifies the uncertainty removed when a message is received. If an event has probability $P$ , its self-information is

Thus a less probable event conveys more information. Independent events add information because $-\log_2(P_1P_2)=-\log_2P_1-\log_2P_2$ .

Entropy

For a discrete source with symbols $x_i$ and probabilities $p_i$ , the average information per source symbol is its entropy:

Entropy is maximized when all source symbols are equally likely. It gives the minimum achievable average number of bits per symbol for lossless source coding, approached by sufficiently long codes.

Channel Capacity

Channel capacity is the supremum of information rates that can be transmitted reliably through a channel. Two limits must be distinguished.

Derivation of the Noiseless Capacity

A channel band-limited to $B$ Hz can convey at most $2B$ independent pulses per second without intersymbol interference. This is the Nyquist signalling rate,

R_s=2B\quad\text{symbols/s}.

If each pulse can assume one of $M$ equally likely distinguishable levels, one symbol carries $\log_2M$ bits. Therefore

C=R_s\log_2M=\boxed{2B\log_2M}.

In a mathematically noiseless channel there is no bound on $M$ , so levels could be packed arbitrarily closely. Noise removes that freedom and leads to the Shannon–Hartley limit.

Proof Sketch of the Shannon–Hartley Law

Model one independent channel sample as

Y=X+N,

where $X$ is the transmitted sample and $N$ is independent additive white Gaussian noise. Mutual information per sample is

I(X;Y)=h(Y)-h(Y\mid X)=h(Y)-h(N),

because knowing $X$ leaves only the uncertainty due to $N$ . Among all random variables of a fixed variance, the Gaussian has maximum differential entropy. For signal power $S$ and noise power $N$ ,

h(N)=\frac12\log_2(2\pi eN),\qquad h(Y)\leq\frac12\log_2\!\bigl(2\pi e(S+N)\bigr),

with equality for a Gaussian input. Hence the greatest information per independent real sample is

\begin{aligned} C_{\mathrm{sample}} &=\frac12\log_2\!\left(\frac{S+N}{N}\right)\\ &=\frac12\log_2\!\left(1+\frac{S}{N}\right) \quad\text{bits/sample}. \end{aligned}

A channel of bandwidth $B$ supplies $2B$ independent real samples per second, so

C=2B\,C_{\mathrm{sample}} =\boxed{B\log_2\!\left(1+\frac{S}{N}\right)}.

This is a proof sketch: a rigorous argument decomposes the band-limited Gaussian channel into independent orthogonal modes and invokes the channel coding theorem. For every rate $R<C$ , sufficiently long suitable codes can make error probability arbitrarily small; finite codes are not promised zero error. Rates above $C$ cannot be made reliable.

Shannon–Hartley capacity rises logarithmically with the linear signal-to-noise ratio; every additional 3 dB does not double capacity.

Shannon–Hartley capacity rises logarithmically with the linear signal-to-noise ratio; every additional 3 dB does not double capacity.

Interpretation and Bandwidth–SNR Trade-off

Fixed numerical $S/N$ : Capacity increases linearly with bandwidth $B$ .
Fixed bandwidth: Capacity increases only logarithmically with $S/N$ .
Coding significance: Capacity is a theoretical reliability limit; practical modulation and coding schemes seek to approach it.

When signal power $S$ and the two-sided white-noise description are represented by an equivalent in-band noise density $N_0$ such that $N=N_0B$ , increasing bandwidth also admits more noise. Then

C=B\log_2\!\left(1+\frac{S}{N_0B}\right),

which still increases with $B$ , but with diminishing returns rather than linearly.

Example — Shannon–Hartley Capacity

A channel has $B=3\,kHz$ and $S/N=30\,dB$ . Find its capacity.

First convert the logarithmic ratio to a linear power ratio:

\frac{S}{N}=10^{30/10}=1000.

Therefore

\begin{aligned} C&=3000\log_2(1+1000)\\ &=3000\log_2(1001)\\ &\approx\boxed{29.9\,kbit/s}. \end{aligned}

Using $30$ directly in the capacity formula would be incorrect because the formula requires a linear power ratio.

[!NOTE]

Example — Bandwidth versus SNR

For $B=1\,MHz$ and $S/N=15\,dB$ ,

\frac{S}{N}=10^{1.5}=31.62, \qquad C=10^6\log_2(32.62) \approx\boxed{5.03\,Mbit/s}.

Doubling the linear SNR to $63.24$ gives only $C\approx6.01\,Mbit/s$ , an increase of about $0.98\,Mbit/s$ . This demonstrates the logarithmic diminishing return of increasing SNR.

[!TIP]

Exam Focus

Always convert decibels before using Shannon–Hartley: $S/N=10^{(S/N)_{\mathrm{dB}}/10}$ . Do not confuse the noiseless Nyquist formula $2B\log_2M$ with the noisy-channel formula $B\log_2(1+S/N)$ .