# 7 Vector Basics

Because variables can be treated as vectors in a geometric sense, I would like to briefly revisit a couple of key concepts for operating with vectors: the inner product and derived operations.

# data matrix
X <- matrix(c(150, 172, 180, 49, 77, 80), nrow = 3, ncol = 2)
rownames(X) <- c("Leia", "Luke", "Han")
colnames(X) <- c("weight", "height")

X
weight height
Leia    150     49
Luke    172     77
Han     180     80

## 7.1 Inner Product

The concept of an inner product is one of the most important matrix algebra concepts, also referred to as the dot product. The inner product is a special operation defined on two vectors $$\mathbf{x}$$ and $$\mathbf{y}$$ that, as its name indicates, allows us to multiply $$\mathbf{x}$$ and $$\mathbf{y}$$ in a certain way.

The inner product of two vectors $$\mathbf{x}$$ and $$\mathbf{y}$$—of the same size— is defined as:

$\mathbf{x \cdot y} = \sum_{i = 1}^{n} x_i y_i$

basically the inner product consists of the element-by-element product of $$\mathbf{x}$$ and $$\mathbf{y}$$, and then adding everything up. The result is not another vector but a single number, a scalar. We can also write the inner product $$\mathbf{x \cdot y}$$ in vector notation as $$\mathbf{x^\mathsf{T} y}$$ since

$\mathbf{x^\mathsf{T} y} = (x_1 \dots x_n) \begin{pmatrix} y_1 \\ \vdots \\ y_n \end{pmatrix} = \sum_{i = 1}^{n} x_i y_i$

Consider the data about Leia and Luke used in the last chapter:

     weight height
Leia    150     49
Luke    172     77

For example, the inner product of weight and height in $$\mathbf{M}$$ is

$\texttt{weight}^\mathsf{T} \hspace{1mm} \texttt{height} = (150 \times 49) + (172 \times 77) = 20594$

What does this value mean? To answer this question we need to discuss three other concepts that are directly derived from having an inner product:

1. Length of a vector

2. Angle between vectors

3. Projection of vectors

All these aspects play a very important role for multivariate methods. But not only that, we’ll see in a moment how many statistical summaries can be obtained through inner products.

## 7.2 Length

Another important usage of the inner product is that it allows us to define the length of a vector $$\mathbf{x}$$, denoted by | |, as the square root of the inner product with itself::

$\| \mathbf{x} \| = \sqrt{\mathbf{x^\mathsf{T} x}}$

which is typically known as the norm of a vector. We can calculate the length of the vector weight:

$\| \texttt{weight} \| = \sqrt{(150 \times 150) + (172 \times 172)} = 228.2192$

Likewise, the length of the vector height:

$\| \texttt{height} \| = \sqrt{(49 \times 49) + (77 \times 77)} = 91.2688$

Note that the inner product of a vector with itself is equal to its squared norm: $$\mathbf{x^\mathsf{T} x} = \| \mathbf{x} \|^2$$

## 7.3 Angle

In addition to the length of a vector, the angle between two nonzero vectors $$\mathbf{x, y}$$ can also be expressed using inner products. The angle $$\theta$$ is such that:

$cos(\theta) = \frac{\mathbf{x^\mathsf{T} y}}{\sqrt{\mathbf{x^\mathsf{T} x}} \hspace{1mm} \sqrt{\mathbf{y^\mathsf{T} y}}}$

or

$cos(\theta) = \frac{\mathbf{x^\mathsf{T} y}}{\| \mathbf{x} \| \hspace{1mm} \| \mathbf{y} \|}$

Equivalently, we can reexpress the formula of the inner product using

$\mathbf{x^\mathsf{T} y} = \| \mathbf{x} \| \hspace{1mm} \| \mathbf{y} \| \hspace{1mm} cos(\theta)$

The angle between weight and height in $$\mathbf{M}$$ is such that:

$cos(\theta) = \frac{20594}{228.2192 \times 91.2688} = 0.9887$

## 7.4 Orthogonality

Besides calculating lengths of vectors and angles between vectors, an inner product allows us to know whether two vectors are orthogonal. In a two dimensional space, orthogonality is equivalent to perpendicularity; so if two vectors are perpendicular to each other—the angle between them is a 90 degree angle—they are orthogonal. Two vectors vectors $$\mathbf{x}$$ and $$\mathbf{y}$$ are orthogonal if their inner product is zero:

$\mathbf{x^\mathsf{T} y} = 0 \iff \mathbf{x} \hspace{1mm} \bot \hspace{1mm} \mathbf{y}$

## 7.5 Projection

The last aspect I want to touch related with the inner product is the so-called projections. The idea we need to consider is the orthogonal projection of a vector $$\mathbf{y}$$ onto another vector $$\mathbf{x}$$.

The basic notion of projection requires two ingredients: two vectors $$\mathbf{x}$$ and $$\mathbf{y}$$. To obtain the projection of $$\mathbf{y}$$ onto $$\mathbf{x}$$, we need to express $$\mathbf{x}$$ in unit norm. The obtained projection $$\hat{\mathbf{y}}$$ is expressed as $$a \mathbf{x}$$. This means that a projection implies multiplying $$\mathbf{x}$$ by some number $$a$$, such that $$\hat{\mathbf{y}} = a \mathbf{x}$$ is a stretched version of $$\mathbf{x}$$. This is better appreciated in the following figure.

Having two nonzero vectors $$\mathbf{x}$$ and $$\mathbf{y}$$, we can project $$\mathbf{y}$$ on $$\mathbf{x}$$

$projection = \mathbf{x} \left( \frac{\mathbf{y^\mathsf{T} x}}{\mathbf{x^\mathsf{T} x}} \right)$

$= \mathbf{x} \left( \frac{\mathbf{y^\mathsf{T} x}}{\| \mathbf{x} \|^2} \right)$