9.2 Covariance

The covariance generalizes the concept of variance for two variables. Recall that the formula for the covariance between \(\mathbf{x}\) and \(\mathbf{y}\) is:

\[ cov(\mathbf{x, y}) = \frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x}) (y_i - \bar{y}) \]

where \(\bar{x}\) is the mean value of \(\mathbf{x}\) obtained as:

\[ \bar{x} = \frac{1}{n} (x_1 + x_2 + \dots + x_n) = \frac{1}{n} \sum_{i = 1}^{n} x_i \]

and \(\bar{y}\) is the mean value of \(\mathbf{y}\):

\[ \bar{y} = \frac{1}{n} (y_1 + y_2 + \dots + y_n) = \frac{1}{n} \sum_{i = 1}^{n} y_i \]

Basically, the covariance is a statistical summary that is used to assess the linear association between pairs of variables.

Assuming that the variables are mean-centered, we can get a more compact expression of the covariance in vector notation:

\[ cov(\mathbf{x, y}) = \frac{1}{n} (\mathbf{x^\mathsf{T} y}) \]

Properties of covariance:

• the covariance is a symmetric index: \(cov(X,Y) = cov(Y,X)\)
• the covariance can take any real value (negative, null, positive)
• the covariance is linked to variances under the name of the Cauchy-Schwarz inequality: \[cov(X,Y)^2 \leq var(X) var(Y) \]