Taylor expansions for the moments of functions of random variables

In probability theory, it is possible to approximate the moments of a function f of a random variable X using Taylor expansions, provided that f is sufficiently differentiable and that the moments of X are finite.

First moment
Given $$\mu_X$$ and $$\sigma^2_X$$, the mean and the variance of $$X$$, respectively, a Taylor expansion of the expected value of $$f(X)$$ can be found via



\begin{align} \operatorname{E}\left[f(X)\right] & {} = \operatorname{E}\left[f\left(\mu_X + \left(X - \mu_X\right)\right)\right] \\ & {} \approx \operatorname{E}\left[f(\mu_X) + f'(\mu_X)\left(X-\mu_X\right) + \frac{1}{2}f''(\mu_X) \left(X - \mu_X\right)^2 \right] \\ & {} = f(\mu_X) + f'(\mu_X) \operatorname{E} \left[ X-\mu_X \right] + \frac{1}{2}f''(\mu_X) \operatorname{E} \left[ \left(X - \mu_X\right)^2 \right]. \end{align} $$

Since $$E[X-\mu_X]=0,$$ the second term vanishes. Also, $$E[(X-\mu_X)^2]$$ is $$\sigma_X^2$$. Therefore,


 * $$\operatorname{E}\left[f(X)\right]\approx f(\mu_X) +\frac{f''(\mu_X)}{2}\sigma_X^2$$.

It is possible to generalize this to functions of more than one variable using multivariate Taylor expansions. For example,


 * $$\operatorname{E}\left[\frac{X}{Y}\right]\approx\frac{\operatorname{E}\left[X\right]}{\operatorname{E}\left[Y\right]} -\frac{\operatorname{cov}\left[X,Y\right]}{\operatorname{E}\left[Y\right]^2}+\frac{\operatorname{E}\left[X\right]}{\operatorname{E}\left[Y\right]^3}\operatorname{var}\left[Y\right]$$

Second moment
Similarly,


 * $$\operatorname{var}\left[f(X)\right]\approx \left(f'(\operatorname{E}\left[X\right])\right)^2\operatorname{var}\left[X\right] = \left(f'(\mu_X)\right)^2\sigma^2_X -\frac{1}{4}\left(f''(\mu_X)\right)^2\sigma_X^4$$

The above is obtained using a second order approximation, following the method used in estimating the first moment. It will be a poor approximation in cases where $$f(X)$$ is highly non-linear. This is a special case of the delta method.

Indeed, we take $$\operatorname{E}\left[f(X)\right]\approx f(\mu_X) +\frac{f''(\mu_X)}{2}\sigma_X^2$$.

With $$ f(X) = g(X)^2 $$, we get $$\operatorname{E}\left[Y^2\right]$$. The variance is then computed using the formula $$\operatorname{var}\left[Y\right] = \operatorname{E}\left[Y^2\right] - \mu_Y^2$$.

An example is,


 * $$\operatorname{var}\left[\frac{X}{Y}\right]\approx\frac{\operatorname{var}\left[X\right]}{\operatorname{E}\left[Y\right]^2}-\frac{2\operatorname{E}\left[X\right]}{\operatorname{E}\left[Y\right]^3}\operatorname{cov}\left[X,Y\right]+\frac{\operatorname{E}\left[X\right]^2}{\operatorname{E}\left[Y\right]^4}\operatorname{var}\left[Y\right].$$

The second order approximation, when X follows a normal distribution, is:


 * $$\operatorname{var}\left[f(X)\right]\approx \left(f'(\operatorname{E}\left[X\right])\right)^2\operatorname{var}\left[X\right] + \frac{\left(f(\operatorname{E}\left[X\right])\right)^2}{2}\left(\operatorname{var}\left[X\right]\right)^2 = \left(f'(\mu_X)\right)^2\sigma^2_X + \frac{1}{2}\left(f(\mu_X)\right)^2\sigma_X^4 + \left(f'(\mu_X)\right)\left(f'''(\mu_X)\right)\sigma_X^4$$

First product moment
To find a second-order approximation for the covariance of functions of two random variables (with the same function applied to both), one can proceed as follows. First, note that $$\operatorname{cov}\left[f(X),f(Y)\right]=\operatorname{E}\left[f(X)f(Y)\right]-\operatorname{E}\left[f(X)\right]\operatorname{E}\left[f(Y)\right]$$. Since a second-order expansion for $$\operatorname{E}\left[f(X)\right]$$ has already been derived above, it only remains to find $$\operatorname{E}\left[f(X)f(Y)\right]$$. Treating $$f(X)f(Y)$$ as a two-variable function, the second-order Taylor expansion is as follows:



\begin{align} f(X)f(Y) & {} \approx f(\mu_X) f(\mu_Y) + (X-\mu_X) f'(\mu_X)f(\mu_Y) + (Y - \mu_Y)f(\mu_X)f'(\mu_Y) + \frac{1}{2}\left[(X-\mu_X)^2 f(\mu_X)f(\mu_Y) + 2(X-\mu_X)(Y-\mu_Y)f'(\mu_X)f'(\mu_Y) + (Y-\mu_Y)^2 f(\mu_X)f(\mu_Y) \right] \end{align} $$

Taking expectation of the above and simplifying—making use of the identities $$\operatorname{E}(X^2)=\operatorname{var}(X)+\left[\operatorname{E}(X)\right]^2$$ and $$\operatorname{E}(XY)=\operatorname{cov}(X,Y)+\left[\operatorname{E}(X)\right]\left[\operatorname{E}(Y)\right]$$—leads to $$\operatorname{E}\left[f(X)f(Y)\right]\approx f(\mu_X)f(\mu_Y)+f'(\mu_X)f'(\mu_Y)\operatorname{cov}(X,Y)+\frac{1}{2}f(\mu_X)f(\mu_Y)\operatorname{var}(X)+\frac{1}{2}f(\mu_X)f(\mu_Y)\operatorname{var}(Y)$$. Hence,



\begin{align} \operatorname{cov}\left[f(X),f(Y)\right] & {} \approx f(\mu_X)f(\mu_Y)+f'(\mu_X)f'(\mu_Y)\operatorname{cov}(X,Y)+\frac{1}{2}f(\mu_X)f(\mu_Y)\operatorname{var}(X)+\frac{1}{2}f(\mu_X)f(\mu_Y)\operatorname{var}(Y) - \left[f(\mu_X)+\frac{1}{2}f(\mu_X)\operatorname{var}(X)\right] \left[f(\mu_Y)+\frac{1}{2}f(\mu_Y)\operatorname{var}(Y) \right] \\ & {} = f'(\mu_X)f'(\mu_Y) \operatorname{cov}(X,Y) - \frac{1}{4}f(\mu_X)f(\mu_Y)\operatorname{var}(X)\operatorname{var}(Y) \end{align} $$

Random vectors
If X is a random vector, the approximations for the mean and variance of $$f(X)$$ are given by

\begin{align} \operatorname{E}(f(X)) &= f(\mu_X) + \frac{1}{2} \operatorname{trace}(H_f(\mu_X) \Sigma_X) \\ \operatorname{var}(f(X)) &= \nabla f(\mu_X)^t \Sigma_X \nabla f(\mu_X) + \frac{1}{2} \operatorname{trace} \left( H_f(\mu_X) \Sigma_X H_f(\mu_X) \Sigma_X \right). \end{align} $$ Here $$\nabla f$$ and $$H_f$$ denote the gradient and the Hessian matrix respectively, and $$ \Sigma_X $$ is the covariance matrix of X.