User:K687 ypo/sandbox

Basic examples
Kullback gives the following example (Table 2.1, Example 2.1). Let P and Q be the distributions shown in the table and figure. P is the distribution on the left side of the figure, a binomial distribution with N = 2 and p = 0.4. Q is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes x = 0, 1, or 2, each with probability p = 1/3.



The KL divergence is calculated as follows. This example uses the natural log with base e, designated ln.


 * $$D_{\mathrm{KL}}(Q\|P) = \sum_i Q(i) \ln\frac{Q(i)}{P(i)}$$


 * $$ = 0.333 * \ln\frac{0.333}{0.36} + 0.333 * \ln\frac{0.333}{0.48} + 0.333 * \ln\frac{0.333}{0.16}$$


 * $$ = - 0.02565 + (-0.12155) + 0.24466 $$


 * $$ = 0.09746 $$