In practice, the computation of expected values for certain particular functions of random variables have arisen repeatedly.   These are the power functions.   Thus, for a random variable   $X$   with probability density   $p$ , the expected values of   $x$ , ${\displaystyle x^2}$ , ${\displaystyle x^3}$ , … , have routinely proven useful.   These are given special names.

Definition: The   ${\displaystyle k^{\mathrm{t}\mathrm{h}}}$   moment about the origin of random variable   $X$   with density   $p$   is given by
$${\mu }_k^{\prime }=E\left[X^k\right]=\{\begin{array}{ccc}\sum _{i=1}^{\mathrm{\infty }}{x}_i^{k}p(x_i)& & X\text{discrete}\\ & & \\ {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{x}^{k}p(x)\mathrm{d}x& & X\text{continuous}\end{array}$$
It is common to just refer to the density function when performing such computations, rather than to a random variable having the given density function.   Thus we might refer to the   ${\displaystyle k^{\mathrm{t}\mathrm{h}}}$   moment of   $p$ .

The first moment   ${\mu }_1^{\prime }$   (this is   $E[X]$ , the mean of   $X$ ) arises so frequently that it is given a special symbol: $\mu$ .   Also, the moments about the mean are usually what are really of issue (and why the wording “about the origin” was used above), and we define these by

Definition: The   ${\displaystyle k^{\mathrm{t}\mathrm{h}}}$   moment about the mean of random variable   $X$   with density   $p$   is given by
$${\mu }_k=E[(X-\mu {)}^k]=\{\begin{array}{ccc}\sum _{i=1}^{\mathrm{\infty }}(x_i-\mu {)}^{k}p(x_i)& & X\text{discrete}\\ & & \\ {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}(x-\mu {)}^{k}p(x)\mathrm{d}x& & X\text{continuous}\end{array}$$

The geometric meanings of the   ${\mu }_k$   are the same as was discussed in our description
of data.   (See chapter 1, section 3 and chapter 1, section 4 in particular.)   They describe aspects of the
way probability is spread about the mean of a density function.   The most important of these is   ${\mu }_2$ , which also occurs often enough and in an important enough role that it is given a special name and symbol.   We call it the variance of the density, and denote it by   ${\displaystyle {\sigma }^2}$ .   Its positive square root, $\sigma$ , is called the standard deviation of the density function.   We use it in place of the variance when we want our measure of concentration of probability about the mean to be in the same units as our random variable.

A trick for computing   ${\mu }_2$

The following computational trick is often useful (and will be for us) for computing   ${\mu }_2$ , and in any case gives a simple relationship between   ${\mu }_2$ , ${\mu }_2^{\prime }$ , and   $\mu$ .
$$\begin{array}{rl}{\mu }_2& =\sum _{i=1}^{\mathrm{\infty }}{(x_i-\mu )}^{2}p\left(x_i\right)\\ & \\ & =\sum _{i=1}^{\mathrm{\infty }}(x_i^2-2\mu x_i+{\mu }^2)p\left(x_i\right)\\ & \\ & =\sum _{i=1}^{\mathrm{\infty }}{x}_i^{2}p\left(x_i\right)+2\mu \sum _{i=1}^{\mathrm{\infty }}{x}_{i}p\left(x_i\right)+{\mu }^2\sum _{i=1}^{\mathrm{\infty }}p\left(x_i\right)\\ & \\ & ={\mu }_2^{\prime }-2\mu \mu +{\mu }^2\\ & \\ & ={\mu }_2^{\prime }-{\mu }^2\end{array}$$
Similar such tricks exist for the higher moments, but will not be worked out here.   The reader is invited to find a relationship between   ${\mu }_3$ , ${\mu }_3^{\prime }$ , ${\mu }_2^{\prime }$ , and   $\mu$ .

By way of illustration   –   and also a display of usefulness   –   consider finding the mean and variance of a linear combination of independent random variables.

Mean and Variance of a Linear Combination of Independent Random Variables

Let   ${\displaystyle X_1}$ , … , ${\displaystyle X_n}$   be   $n$   mutually independent random variables, with means   ${\displaystyle {\mu }_1}$ , … , ${\displaystyle {\mu }_n}$   and variances   ${\displaystyle {\sigma }_1}$ , … ,
${\displaystyle {\sigma }_n}$ .   Let
$$W=a_1{X}_1+\cdots +a_n{X}_n\text{.}$$
From linearity we have
$${\mu }_W=E[W]=\sum _{k=1}^{n}E\left[a_k{X}_k\right]=\sum _{k=1}^n{a}_k{\mu }_k\text{.}$$
Now use
$$W-{\mu }_W=a_1(X_1-{\mu }_1)+\cdots +a_n(X_n-{\mu }_n)$$
to get
$${(W-{\mu }_W)}^2=\sum _{j=1}^n\sum _{k=1}^n{a}_j{a}_k(X_j-{\mu }_j)(X_k-{\mu }_k)\text{.}$$
Applying our properties above, we have
$$E\left[{(W-{\mu }_W)}^2\right]=\sum _{j=1}^n\sum _{k=1}^n{a}_j{a}_{k}E\left[(X_j-{\mu }_j)(X_k-{\mu }_k)\right]\text{.}$$
Since the   ${\displaystyle X_j}$   and   ${\displaystyle X_k}$   are independent if   $j\ne k$ , we have
$$E\left[(X_j-{\mu }_j)(X_k-{\mu }_k)\right]=E[X_j-{\mu }_j]E[X_k-{\mu }_k](j\ne k)\text{.}$$
But   ${\displaystyle E[X_j-{\mu }_j]=0}$ , so
$$E\left[{(W-{\mu }_W)}^2\right]=\sum _{k=1}^n{a}_k^{2}E\left[{(X_k-{\mu }_k)}^2\right]\text{.}$$
Since   ${\displaystyle E\left[{(X_k-{\mu }_k)}^2\right]={\sigma }_k^2}$ , we have
$${\sigma }_W^2=\sum _{k=1}^n{a}_k^2{\sigma }_k^2\text{.}$$

While direct computation of moments of a probability distribution may be rather simple, we present here another method which has enough other uses to justify its existence and our familiarity.

Consider
$$M_X(t)=E\left[{\mathrm{e}}^{tX}\right]=\{\begin{array}{rlc}\sum _{i=1}^{\mathrm{\infty }}{\mathrm{e}}^{t{x}_i}f(x_i)& & X\text{discrete}\\ & & \\ {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{tx}f(x)\mathrm{d}x& & X\text{continuous}\end{array}$$

To see how this computation yields the moments of distribution   $X$ , we need to know that as a power series
$${\mathrm{e}}^z=1+z+\frac{z^2}{2!}+\frac{z^3}{3!}+\cdots$$
So that
$$\begin{array}{rl}{M}_X(t)& =\sum _{i=1}^{\mathrm{\infty }}(1+(t{x}_i)+\frac{(t{x}_i{)}^2}{2!}+\frac{(t{x}_i{)}^3}{3!}+\cdots )f(x_i)\\ & \\ & =\sum _{i=1}^{\mathrm{\infty }}f(x_i)+t\sum _{i=1}^{\mathrm{\infty }}{x}_{i}f(x_i)+\frac{t^2}{2!}\sum _{i=1}^{\mathrm{\infty }}{x}_i^{2}f(x_i)+\cdots & \\ & =1+t{{\mu }_1}^{\prime }+\frac{t^2}{2!}{{\mu }_2}^{\prime }+\frac{t^3}{3!}{{\mu }_3}^{\prime }+\cdots \text{.}\end{array}$$
We can now obtain the moments about the mean by
$${\mu }_k^{\prime }=\frac{{\mathrm{d}}^k\phantom{x}}{\mathrm{d}{x}^k}{M}_X(t){|}_{t=0}\text{.}$$
The utility of this will show up soon.

For now we call   ${\displaystyle M_X(t)}$   the moment generating function of   $X$ , and   ${\displaystyle M_{g(X)}(t)}$   the moment generating function of   $g(X)$ .   We will usually refer to this as MGF for short.

Two Properties of MGFs   –   How to Handle Addition of, and Multiplication by, a Constant

Theorem:   For any constant   $c$   and any function   $g(X)$   for which the MGF   ${\displaystyle M_{g(X)}}$   exists, we have
$$\begin{array}{rl}{M}_{cg}(t)& =M_g(ct)\\ & \\ M_{g+c}(t)& ={\mathrm{e}}^{ct}{M}_g(t)\text{.}\end{array}$$

How to see this

Let   $c$   be a constant, and   $g(X)$   be a function for which   ${\displaystyle M_{g(X)}}$   exists.   Then
$$M_{cg}(t)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{tcg(x)}f(x)\mathrm{d}x=M_g(ct)\text{.}$$
Secondly
$$\begin{array}{rl}{M}_{g+c}(t)& ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{t(g(x)+c)}f(x)\mathrm{d}x\\ & \\ & ={\mathrm{e}}^{ct}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{tg(x)}f(x)\mathrm{d}x\\ & \\ & ={\mathrm{e}}^{ct}{M}_g(t)\text{.}\end{array}$$

These two properties will be useful in later computations.   In particular, if a constant multiplies, or is added to, a function of a random variable, these formulae will allow us to easily compute moments and MGFs based on the MGF of our function.

A simple fact is the following:

Theorem If two random variables   $X$   and   $Y$   have same distribution, then they have the same moment generating function, i.e.   ${\displaystyle M_X(t)=M_Y(t)}$   for all   $t\in \mathbb{R}$ .

The above theorem is easy to prove, but an important fact about the moment generating function is that the above result also goes other way round.   And this is what makes the moment generating function such a powerful tool.   We will not give a proof here, but will feel free to use this fact.

Theorem If two random variables   $X$   and   $Y$   have same moment generating function, i.e.   ${\displaystyle M_X(t)=M_Y(t)}$   for all   $t\in \mathbb{R}$ , then they have the same distribution function, i.e. ${\displaystyle F_X(t)=F_Y(t)}$   for all   $t\in \mathbb{R}$ .

We will use moment generating functions, for example, to understand sample means and linear combinations of random variables   &ndash:   particularly when variables generating our samples are independent.   Here is a formula that we will put to work in the case when we wish to determine information about the random variable
$$g(X_1,\cdots ,X_n)=a_1{X}_1+\cdots +a_n{X}_n$$
when   ${\displaystyle X_1}$ , … , ${\displaystyle X_n}$   are independent.

A Useful Formula: The Moment Generating Function for a Linear Combination of Independent Random Variables

Let   ${\displaystyle W=a_1{X}_1+\cdots +a_n{X}_n}$   with   ${\displaystyle X_1}$ , … , ${\displaystyle X_n}$   independent.   The moment generating function of   $W$   is given by
$$M_W(t)=E\left[{\mathrm{e}}^{t(a_1{X}_1+\cdots +a_n{X}_n)}\right]=E[{\mathrm{e}}^{t{a}_1{X}_1}\cdots {\mathrm{e}}^{t{a}_n{X}_n}]\text{.}$$
The independence of the   ${\displaystyle X_k}$ , and hence of these exponential functions, implies that
$$M_W(t)=E\left[{\mathrm{e}}^{t{a}_1{X}_1}\right]\cdots E\left[{\mathrm{e}}^{t{a}_n{X}_n}\right]=M_{a_1{X}_1}(t)\cdots M_{a_n{X}_n}(t)\text{.}$$
That is
$$M_{a_1{X}_1+\cdots +a_n{X}_n}(t)=M_{a_1{X}_1}(t)\cdots M_{a_n{X}_n}(t)\text{.}$$