User:Zantolak/Sandbox

$$X \,$$

$$M \,$$

$$M=2 \,$$

$$M=1 \,$$

$$I(X) \,$$

$$I(X)=\log(M) \,$$

$$I(X)=\log_2(M) \,$$

$$I(X)=\log_2(2) \,$$

$$I(X)=1 \,$$

$$I(X)=1 \textrm{\,bit.} \,$$

$$I(X)=\log_2(2)=1 \,$$

$$I(X)=\log_2(2)=1 \textrm{\,bit.} \,$$

$$I(X)=\log_2(1) \,$$

$$I(X)=0 \,$$

$$I(X)=0 \textrm{\,bits.} \,$$

$$I(X)=\log_2(1)=0 \,$$

$$I(X)=\log_2(1)=0 \textrm{\,bits.} \,$$

$$I(X)=\log_2(6) \,$$

$$I(X)\approx 2.585 \,$$

$$I(X)\approx 2.585 \textrm{\,bits.} \,$$

$$I(X)=\log_2(6)\approx 2.585 \,$$

$$I(X)=\log_2(6)\approx 2.585 \textrm{\,bits.} \,$$

$$\{{a_1}, {a_2}, ..., {a_M}\} \,$$

$$\{{a_1}, {a_2}\} \,$$

$$P(a_i) \,$$

$$P(a_1) \,$$

$$\{\textrm{heads}, \textrm{tails}\} \,$$

$$p_i \,$$

$$p_1 \,$$

$$\{{p_1}, {p_2}, ..., {p_M}\} \,$$

$$\{{p_1}, {p_2}\} \,$$

$$\{0.5, 0.5\} \,$$

$$I({a_i})=\log\left(\frac{1}{p_i}\right) \,$$

$$I({a_1})=\log\left(\frac{1}{p_1}\right) \,$$

$$I({a_1})=\log\left(\frac{1}{0.5}\right) \,$$

$$I({a_1})=\log_2\left(\frac{1}{0.5}\right) \,$$

$$I({a_1})=1 \textrm{\,bit.} \,$$

$$I({a_1})=\log_2\left(\frac{1}{0.5}\right)=1 \textrm{\,bit.} \,$$

$$I({a_1})=\log_2\left(\frac{1}{1}\right) \,$$

$$I({a_1})=0 \textrm{\,bits.} \,$$

$$I({a_1})=\log_2\left(\frac{1}{1}\right)=0 \textrm{\,bits.} \,$$

$$\{{a_1}, {a_2}\} \,$$

$$\{0.75, 0.25\} \,$$

$$I({a_1})=\log_2\left(\frac{1}{0.75}\right) \,$$

$$I({a_1})\approx 0.415 \textrm{\,bits.} \,$$

$$I({a_1})=\log_2\left(\frac{1}{0.75}\right)\approx 0.415 \textrm{\,bits.} \,$$

$$I({a_2})=\log_2\left(\frac{1}{0.25}\right) \,$$

$$I({a_2})=2 \textrm{\,bits.} \,$$

$$I({a_2})=\log_2\left(\frac{1}{0.25}\right)=2 \textrm{\,bits.} \,$$

$$\{0.99, 0.01\} \,$$

$$I({a_1})=\log_2\left(\frac{1}{0.99}\right) \,$$

$$I({a_1})\approx 0.014 \textrm{\,bits.} \,$$

$$I({a_1})=\log_2\left(\frac{1}{0.99}\right)\approx 0.014 \textrm{\,bits.} \,$$

$$I({a_2})=\log_2\left(\frac{1}{0.01}\right) \,$$

$$I({a_2})\approx 6.644 \textrm{\,bits.} \,$$

$$I({a_2})=\log_2\left(\frac{1}{0.01}\right)\approx 6.644 \textrm{\,bits.} \,$$

$$\sum_{i=1}^M p_i \log_2\left(\frac{1}{p_i}\right) \,$$

$$X \,$$

$$H(X) \,$$

$$H(X)=\sum_{i = 1}^M p_i \log_2\left(\frac{1}{p_i}\right) \,$$

$$p_i \log_2\left(\frac{1}{p_i}\right) \,$$

$$p_1 \log_2\left(\frac{1}{p_1}\right) \,$$

$$p_M \log_2\left(\frac{1}{p_M}\right) \,$$

$$H(X)=0.5\times\log_2\left(\frac{1}{0.5}\right)+0.5\times\log_2\left(\frac{1}{0.5}\right) \,$$

$$H(X)=0.5\times\log_2\left(\frac{1}{0.5}\right)+0.5\times\log_2\left(\frac{1}{0.5}\right)=1 \textrm{\,bit.} \,$$

$$H(X)=0.75\times\log_2\left(\frac{1}{0.75}\right)+0.25\times\log_2\left(\frac{1}{0.25}\right) \,$$

$$H(X)=0.75\times\log_2\left(\frac{1}{0.75}\right)+0.25\times\log_2\left(\frac{1}{0.25}\right)\approx 0.811 \textrm{\,bits.} \,$$

$$\{0.1, 0.1, 0.1, 0.5, 0.1, 0.1\} \,$$

$$H(X)\approx 2.161 \textrm{\,bits.} \,$$

$$I(X)\approx 2.585 \textrm{\,bits.} \,$$

$$p_1 \log_2\left(\frac{1}{p_1}\right)+...+p_M \log_2\left(\frac{1}{p_M}\right) \,$$

$$H(X)=1 \textrm{\,bit.} \,$$

$$H(X)\approx 0.811 \textrm{\,bits.} \,$$