channel capacity in information theory | channel capacity is exactly equal to | formula

what is channel capacity in information theory | channel capacity is exactly equal to | formula theorem and unit ?
THE CHANNEL CAPACITY
            In this section, let us discuss various aspects regarding channel capacity.
9.12.1. Channel Capacity Per Symbol C_s.
The channel capacity per symbol of a discrete memoryless channel (DMC) is defined as
C_s =   I (X;Y) b/symbol                                …(9.35)
where the maximization is over all possible input probability distributions {P(x_i)} on X. Note that the channel capacity C_s is a function of only the channel transition probabilities which define the channel.
9.12.2. Channel Capacity Per Second C
            If r symbols are being transmitted per second, then the maximum rate of transmission of information per second is rC_s. This is the channel capacity per second and is denoted by C(b/s), i.e.,
C = rC_s b/s                                                      …(9.36)
9.12.3. Capacities of Special Channel
            In this subsection, let us discuss capacities of various special channel.
9.12.3.1. Lossless Channel
            For a lossless channel, H(X|Y) = 0, and
I(X;Y) = H(X)                                                                            …(9.37)
Thus, the mutual information (information transfer) is equal to the input (source) entropy, and no source information is lost in transmission.
Consequently, the channel capacity per symbol will be
C_s =   H(X)  = log₂ m                                    …(9.38)
Where m is the number of symbols in X.
9.12.3.2. Deterministic Channel
For a deterministic channel, H(Y|X) = 0 for all input distributions P(x_i), and
I(X;Y) = H(Y)                                                …(9.39)
Thus, the information transfer is equal to the output entropy. The channel capacity per symbol will be
C_s =   H(Y) log₂n                              …(9.40)
where n is the number of symbols in Y.
9.12.3.3. Noiseless Channel
            Since a noiseless channel is both lossless and deterministic, we have
I(X; Y) = H(X) = H(I’)                                               …(9.41)
and the channel capacity per symbol is
C_s = log₂m = log₂n                                             …(9.42)
9.12.3.4. Binary Symmetric Channel (BSC)
For the binary symmetric channel (BSC), the mutual information is
I(X;Y) = H(Y) + p log₂ p + (1 – p) log₂ (1 – p)            …(9.43)
and the channel capacity per symbol will be
C_s = 1 + p log₂ p + (1- p) log₂ (1 -p)                            …(9.44)
EXAMPLE 9.29. Verify the following expression:
C_s = log₂ m
            where C_s is the channel capacity of a lossless channel and m is the number of symbols in X.
Solution: For a lossless channel, we have
H(X|Y) = 0
Then, by equation (9.30), we have
I(X;Y) = H(X) – H(X|Y) = H(X)
Hence, by equations (9.35) and (9.9), we have
C_s =   I(X;Y)
or                                        C_s =   H(X)  = log₂m      Hence proved.
EXAMPLE 9.30. Verify the following expression:
                                    C_s = 1 + p log₂ p + (1 – p) log₂ (1 – p)
            where C_s is the channel capacity of a BSC (figure 9.12)
Solution: We know that the mutual information /(X: Y) of a BSC is given by
I(X;Y) = H(Y) + p log₂ p + (1 – p) log₂ (1 -p)
which is maximum when H(Y) is maximum. Since, the channel output is binary, H(Y) is maximum when each output has a probability of 0.5 and is achieved for equally likely inputs.
For this case H(Y) = 1, and the channel capacity is
equation
EXAMPLE 9.31. Find the channel capacity of the binary erasure channel of figure 9.13.
Solution: Let P(x₁) = α. Then P(x₂) = 1 – α.
diagram
FIGURE 9.13
We have                                   EQUATION
Using equation (9.17), we
[P(Y)] = [α 1 – α]
= [α(1 – p)] p (1 – α) (1 – p)] = [P(y₁) P(y₂) P(y₃)]
By using equation (9.19), we have
[P(X,Y)] =
or                                 [P(X, Y)] =
In addition, from equations (9.24) and (9.26), we can calculate
equation
= – α(1 – p) log₂ α(1 – p) – p log₂ p – (1 – α)(1 – p) log₂ [(1 – α)(1 – p)]
= (1- p)[- α log₂ α – (1 – α) log₂ (1- α)] – p log₂ p – (1 -p) log₂ (1 -p)
Also, we have                         equation
= – a(1 – p) log₂ (1 – p) – αp log₂ p – (1 – α) p log₂ p
– (1 – α)(1 -p) log₂ (1 -p)
= – p log₂ p-(1-p) log₂ (1 -p)
Thus, by equations (9.33) and (9.57), we have
    I(X; Y) = I(Y) – H(Y|X) = (1 – p)[- α log₂ α – (1 – α) log₂ (1 – α)] = (1 – p)H(X)
And by equations (9.35) and (9.58), we have
equation
9.13     ENTROPY RELATIONS FOR A CONTINUOUS CHANNEL
In a Continuous channel, an information source produces a continuous signal x(t). The set of possible signals is considered as an ensemble of waveforms generated by some ergodic random process. It is further assumed that x(t) has a finite bandwidth so that x(t) is completely characterized by its periodic sample values. Hence, at any sampling instant, the collection of possible sample value  constitutes a continuous random variable X descrbed by it probability density function f_X(x).
The average amount of information per sample value of x(t) (i.e., entropy of a continuous source) is measured by
equation
The entropy H(X) defined by equation (9.45) is known as the differential entropy of X.
Also, the average mutual information in a continuous channel is defined (by analogy with the discrete case) as
I(X; Y) = H(X) H(X|Y) = H(Y) – H(Y|X)
where                                      equation                                         …(9.46)
equation                                         …(9.47)
equation                                         …(9.48)
 
9.14 CAPACITY OF AN ADDITIVE WHITE GAUSSIAN NOISE (AWGN)   CHANNEL: SHANNON-HARTLEY LAW
In an additive white Gaussian noise (AWGN) channel, the channel output Y is given by
Y = X + n                                                        …(9.48)
where X is the channel input and n is an additive bandlimited white Gaussian noise with zero mean and variance .
The capacity C_s of an AWGN channel is given by
EQUATION
where S/N is the signal-to-noise ratio at the channel output. If the channel bandwidth B Hz is fixed, then the output y(t) is also a bandlimited signal completely characterized by its periodic sample values taken at the Nyquist rate 2B samples/s.
Then the capacity C(b/s) of the AWGN channel is given by
C = 2B x C_s = B log₂  b/s                …(9.50)
Equation (9.50) is known as the Shannon-Hartley law.
            The Shannon-Hartley law underscores the fundamental role of bandwidth and signal-to-noise ratio in communication. It also shows that we can exchange increased bandwidth for decreased signal power for a system with given capacity C.
9.15 CHANNEL CAPACITY : A DETAILED STUDY
            We know that the bandwidth and the noise power place a restriction upon the rate of information that can is expressed as be transmitted by a channel. It may be shown that in a channel which is disturbed by a white Gaussian noise, one can transmit information at a rate of C bits per second, where C is the channel capacity and is expressed as
C = Blog₂                                        …(9.51)
In this expression,                   B = channel bandwidth in Hz
S = Signal power
N = Noise power
It may be noted that the expression (equation 9.50) for channel capacity is valid for white Gaussian not However, for other types of noise, the expression is modified.
            Proof: Let us present a proof of channel capacity formula based upon the assumption that if a signal is mixed with noise, the signal amplitude can be recognized only within the root main square noise voltage. In other words, we can say that the uncertainty in recognizing the exact signal amplitude is equal to the root mean square noise voltage.
Again, let us assume that the average signal power and the noise power are S watts and N watts respectively. This means that the root mean square value of the received signal is  volts and the root mean square value of the noise volt  volts.
Now, we have to distinguish the received signal of the amplitude  volts in the presence of the noise amplitude  volts.
As a matter of fact, the input signal variation of less than  volts will not be distinguished at the receiver end.
Therefore, the number of the distinct levels that can be distinguished without error can be expressed as
M =                                               …(9.51)
Thus, equation (9.51) expresses the maximum value of M.
Now, the maximum amount of information carried by each pulse having  distinct levels is given by
I = log2   =  log₂   bits                 …(9.52)
Now, after establishing expression in equation (8.15), we can determine the channel capacity. In fact, the channel capacity is the maximum amount of information that can be transmitted per second by a channel.
If a channel can transmit a maximum of K pulses per second, then, the channel capacity C is given by
C =  log₂  bits per second             …(9.53)

Recall that for bandwidth requirements of PAM signals, it has been shown that a system of bandwidth nf_m Hz can transmit 2n f_m, independent pulses per second. Further, under these conditions, the received signal will yield the correct values of the amplitudes of the pulses but will not reproduce the details of the pulse shapes.
Now, since, we are interested only in the pulse amplitudes and not their shapes, it is concluded that a system with bandwidth B Hz can transmit a maximum of 2B pulses per second. Further, since, each pulse can carry a maximum information of  log₂   bits, if follows that a system of bandwidth B can transmit the information at a following maximum rate:
C = B log₂  bits per second                         …(9.54)
Therefore, the channel capacity C is limited by the bandwidth of the channel (or system) and noise signal.
For a noiseless channel, N = 0 and the channel capacity will be infinite. However, practically, N always finite and therefore, the channel capacity is finite.*
The expression in equation (9.54) is also known as the Hartley-Shannon law and is treated as the central theorem of information theory.
From Hartley-Shannon law, it is obvious that the bandwidth and the signal power can be exchanged for one another. To transmit the information at a given rate, we may reduce, the signal power transmitted provided that the bandwidth is increased correspondingly. In a similar manner, o increase the signal power.
As a matter of fact, the process of modulation is actually a means of effecting this exchange between the bandwidth and the signal-to-noise ratio.**
NOTE: It may be noted that the channel capacity represents the maximum amount of information that can be transmitted by a channel per second. To achieve this rate of transmission, the information has to be processed properly or coded in the most efficient manner.