Suppose we have two discrete random variables, $X$   and   $Y$ .   We denote by   ${\displaystyle \mathrm{d}{f}_{X,Y}}$   the joint density function
$$\mathrm{d}{f}_{X,Y}(a,b)=P(X=a\cap Y=b)\text{.}$$
That is, ${\displaystyle \mathrm{d}{f}_{X,Y}(a,b)}$   records the probability that   $X=a$   and   $Y=b$   simultaneously.

If we have two continuous random variables, $X$   and   $Y$ , this definition needs to be handled a bit more delicately.   We denote by   ${\displaystyle \mathrm{d}{f}_{X,Y}}$   the joint density function
$$\mathrm{d}{f}_{X,Y}(a,b)=\underset{h\to 0}{lim}\frac{P(X\in [a-h,a+h]\cap Y\in [b-h,b+h])}{4h^2}\text{.}$$
That is, ${\displaystyle \mathrm{d}{f}_{X,Y}(a,b)}$   records the probability that   $X$   and   $Y$   take on values in a small square centered at   $(a,b)$ , scaled to recognize the size of the square.

Here are a couple of simple examples.

Discrete Example

Given six people, two of whom are left-handed and one of whom is ginger (and also happens to be a leftie), we pick one person at random, with each equally likely to be picked.   Letting   $L$   be the event that a leftie is picked, and   $G$   the event that the ginger is picked, determine the joint density function.

To be more precise, let   $L$   be the Bernoulli random variable assuming the value   $0$   if a right hander is picked, and   $1$   if a leftie is picked.   Then
$$m_L(x)=\{\begin{array}{cc}\frac{4}{6}=\frac{2}{3}& x=0\\ & \\ \frac{2}{6}=\frac{1}{3}& x=1\end{array}\text{.}$$
Similarly, we let   $G$ be the Bernoulli random variable assuming the value   $0$   if the ginger is not picked, and   $1$   if the ginger picked.   We have
$$m_G(y)=\{\begin{array}{cc}\frac{5}{6}& y=0\\ & \\ \frac{1}{6}& y=1\end{array}\text{.}$$

The given information tells us that
$$m_{L,G}(x,y)=\{\begin{array}{cc}\frac{4}{6}=\frac{2}{3}& x=y=0\\ & \\ \frac{1}{6}& x=1,y=0\\ & \\ 0& x=0,y=1\\ & \\ \frac{1}{6}& x=1,y=1\end{array}\text{.}$$

Continuous Example

We will assume that when throwing a dart at a target (dart board) on a wall, the probability that the dart hits in a certain region does not depend on the rotational angle about the center of the board.   The probability that the dart hits at a point a distance   $r$   from the center of the board is proportional to   ${\displaystyle \frac{1}{(1+r^2{)}^2}}$.   Determine the joint density function.

To be more precise, put a coordinate system on the wall, with origin at the center of the dart board.     We will let   $x$   ( $y$ ) be the left-right (up-down) position, with respect to the center of the dart-board, of the a point on the wall.   Then if   $X$   ( $Y$ ) denotes the random variable describing the left-right (up-down) position of the hitting point of the dart   –   we will suppose that the probability density function   ${\displaystyle \mathrm{d}{f}_{(X,Y)}}$   describing the hitting point of the dart is proportional to   ${\displaystyle \frac{1}{(1+x^2+y^2{)}^2}}$ .   We are to determine this proportionality.

Our compatibility condition is  
$$\int {\int }_{{\mathbb{R}}^2}\mathrm{d}{f}_{(X,Y)}=\int {\int }_{{\mathbb{R}}^2}\frac{c}{{(1+x^2+y^2)}^2}\mathrm{d}x\mathrm{d}y=c{\int }_0^{2\pi }{\int }_0^{+\mathrm{\infty }}\frac{r}{{(1+r^2)}^2}\mathrm{d}r\mathrm{d}\theta =1\text{.}$$
But
$${\int }_0^{+\mathrm{\infty }}\frac{r}{{(1+r^2)}^2}\mathrm{d}r=\frac{1}{2}\text{,}$$
so
$$c{\int }_0^{2\pi }\frac{1}{2}\mathrm{d}\theta =1\text{.}$$
Thus   $c=\frac{1}{\pi }$   and the probability density is given by
$$\mathrm{d}{f}_{X,Y}(x,y)=\frac{1}{\pi }\frac{1}{(1+x^2+y^2{)}^2}\text{.}$$

In order to examine the underlying probabilistic phenomena driving the creation of our data (if indeed there is such), we will usually be taking many measurements, making many observations, or performing repeated experiments.   Given this, we will need to understand how to discuss probability when there are multiple samples.

We will start with a somewhat general discussion and then specialize it to the case mentioned in the previous section, where we are independently sampling a probability distribution   $n$   times.

Suppose we have a process that generates   $n$   values.   If the process is discrete, we can ask for the probability that a specific collection of values   $x_1$ , $x_2$ , … , $x_n$   occurs.   Thus, if we flip a fair coin and roll a fair die with two sides numbered “1” and four sides numbered “2”, the collection of possible outcomes (with probabilities indicated) is
$$\begin{array}{cc}(H,1),P=\frac{1}{6}& (H,2),P=\frac{1}{3}\\ (T,1),P=\frac{1}{6}& (T,2),P=\frac{1}{3}\end{array}\text{.}$$
This probability is then a function   $\mathrm{d}f$   of   $x_1$ , $x_2$ , … , $x_n$   If the process is continuous, the description is a bit more subtle, and we specify a density function which is integrated to obtain the probability that a collection of values will occur in a specific set.   Thus we will have a function   ${\displaystyle \mathrm{d}f(x_1,x_2,\cdots ,x_n)}$   such that the probability that an outcome satisfies   ${\displaystyle a_1\le x_1\le b_1}$ , … , ${\displaystyle a_n\le x_n\le b_n}$   is given by
$${\int }_{a_n}^{b_n}\cdots {\int }_{a_1}^{b_1}\mathrm{d}f(x_1,x_2,\cdots ,x_n)\text{.}$$
This is called a multivariate density function, as it depends on several variables.

Such a density function   is determined by the properties that

1. ${\displaystyle p(x_1,x_2,\cdots ,x_n)\ge 0}$ , and
2. $$\{\begin{array}{rlcc}\sum _{(x_1,x_2,\cdots ,x_n)\text{ occurs}}p(x_1,x_2,\cdots ,x_n)& =1& & \text{discrete case}\\ {\int }_{a_n}^{b_n}\cdots {\int }_{a_1}^{b_1}p(x_1,x_2,\cdots ,x_n)\mathrm{d}{x}_1\cdots \mathrm{d}{x}_n& =1& & \text{continuous case}\end{array}\text{.}$$

We compute the expected value of function   ${\displaystyle g(x_1,x_2,\cdots ,x_n)}$   by
$$E[g]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\cdots {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x_1,x_2,\cdots ,x_n)\cdot p(x_1,x_2,\cdots ,x_n)\mathrm{d}{x}_1\cdots \mathrm{d}{x}_n\text{.}$$
and the corresponding moment generating function for   $g$   by
$$M_g(t)=E\left[{\mathrm{e}}^{tg}\right]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\cdots {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{tg(x_1,x_2,\cdots ,x_n)}\cdot p(x_1,x_2,\cdots ,x_n)\mathrm{d}{x}_1\cdots \mathrm{d}{x}_n\text{.}$$
As properties of moments will be used in the future of this treatment, several useful properties
of the   $E$   operator will be discussed in the next section.

Independent Random Variables

We will say that discrete random variables   $X$   and   $Y$   are independent if, for any   $x_0$ , $y_0\in \mathbb{R}$   we have
$$P(X=x_0\cap Y=y_0)=P(X=x_0)\cdot P(Y=y_0)\text{.}$$
Or, more generally so as to include continuous random variables, we add any   $x_1$ , $y_1\in \mathbb{R}$   satisfying   $x_1>x_0$   and $y_1>y_0$ , and demand that
$$P(X\in [x_0,x_1]\cap Y\in [y_0,y_1])=P(X\in [x_0,x_1])\cdot P(Y\in [y_0,y_1])\text{.}$$

Example

Consider the experiment of flipping a coin and then rolling a die.   Let   $X$   denote the number of heads that occur ( $0$   or   $1$ ), and   $Y$   denote the number from the die roll ( $1$   through   $6$ ).   Here is the table of outcomes with their probabilities.
$$\begin{array}{ccccccc}& 1& 2& 3& 4& 5& 6\\ X=0& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}\\ X=1& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}& \frac{1}{12}\end{array}$$
It is easy to see that   $P(X=1)=\frac{1}{2}$   and   $P(Y=2)=\frac{1}{6}$ .   Thus
$$P(X=1\cap Y=2)=\frac{1}{12}=\frac{1}{2}\cdot \frac{1}{6}=P(X=1)\cdot P(Y=2)\text{.}$$
This computation holds for any values which   $X$   or   $Y$   might assume.

One way to think about this is that the die is completely uninfluenced by the result of the flip.

Usually the determination of the density function of a multivariate process is rather onerous.   But this can be greatly simplified if the variables   ${\displaystyle x_1,x_2,\cdots ,x_n}$   are independent.   In this case, we have
$$p(x_1,x_2,\cdots ,x_n)=p_1\left(x_1\right)\cdot p_2\left(x_2\right)\cdot p_n\left(x_n\right)\text{.}$$
That is, the density function can be written as a product of the density functions of the individual variables.

Computation of the expected value of special functions of the form   ${\displaystyle g(x_1,x_2,\cdots ,x_n)=g_1\left(x_1\right)\cdot g_2\left(x_2\right)\cdots g_n\left(x_n\right)}$   is simplified by
$$\begin{array}{rl}E[g_1\cdots g_n]& ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\cdots {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{g}_1\left(x_1\right)\cdots g_n\left(x_n\right)\cdot p_1\left(x_1\right)\cdots p_n\left(x_n\right)\mathrm{d}{x}_1\cdots \mathrm{d}{x}_n\\ & \\ & ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{g}_1\left(x_1\right)\cdot p_1\left(x_1\right)\mathrm{d}{x}_1\cdot \cdots \cdot {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{g}_n\left(x_n\right)\cdot p_n\left(x_n\right)\mathrm{d}{x}_n\\ & \\ & =E\left[g_1\right]\cdots E\left[g_n\right]\text{.}\end{array}$$
In this case, we have that the moment-generating function for   ${\displaystyle g(x_1,x_2,\cdots ,x_n)=g_1\left(x_1\right)+\cdots +g_n\left(x_n\right)}$   simplifies to
$$M_{g_1+\cdots +g_n}(t)=E\left[{\mathrm{e}}^{t(g_1+\cdots +g_n)}\right]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{t{g}_1\left(x_1\right)}\cdot p_1\left(x_1\right)\mathrm{d}{x}_1\cdot \cdots \cdot {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\mathrm{e}}^{t{g}_n\left(x_n\right)}\cdot p_n\left(x_n\right)\mathrm{d}{x}_n=M_{g_1}(t)\cdots M_{g_n}(t)\text{.}$$

The Case of Random Sampling

As was mentioned in the last section, we will think of random sampling as repeated independent values from a given probability function.   In this case, the above discussion of independence tells us that
$$p(x_1,x_2,\cdots ,x_n)=p_1\left(x_1\right)\cdot p_2\left(x_2\right)\cdot p_n\left(x_n\right)=p_1\left(x_1\right)\cdot p_1\left(x_2\right)\cdot p_1\left(x_n\right)\text{.}$$
That is, the individual density functions are identical.