class: center, middle, inverse, title-slide # Geometry of Matrices ## High Dimensional Data Analysis ### Anastasios Panagiotelis & Ruben Loaiza-Maya ### Lecture 9 --- class: inverse, center, middle # Why? --- # Why vectors and matrices? - Nearly all the methods we cover involve *vectors* and *matrices*.<!--D--> -- - The earlier lecture on matrices may have felt like it involved remembering many seemingly arbitrary rules.<!--D--> -- - The aim of this lecture is to understand the *geometric* interpretation of vectors which is crucial in making the connection between data, data analysis and visualisation.<!--D--> -- - Matrices are important to gain a deep understanding about dimension reduction. --- # Vectors - We will think of a 2-dimensional vector in one of three ways:<!--D--> -- + A point in 2-dimensional space<!--D--> -- + An arrow pointing to that point in 2-dimensional space<!--D--> -- + An array of 2 numbers<!--D--> -- - All of these ideas work for n-dimensional vectors where n can be 3,4 or any number -- - By convention all vectors in this lecture are column vectors. --- # Vectors as points <img src="MatrixGeometry_files/figure-html/points-1.png" style="display: block; margin: auto;" /> --- #Vectors as points <img src="MatrixGeometry_files/figure-html/pointsb-1.png" style="display: block; margin: auto;" /> --- # Vectors as arrows <img src="MatrixGeometry_files/figure-html/arrows-1.png" style="display: block; margin: auto;" /> --- # Vectors as arrows <img src="MatrixGeometry_files/figure-html/arrowsb-1.png" style="display: block; margin: auto;" /> --- # Vectors as numbers In general a vector is: `$${\mathbf{v}}=\begin{bmatrix}x\\y\end{bmatrix}$$` For our two examples. `$${\mathbf{v}}_1=\begin{bmatrix}1\\2\end{bmatrix}\quad\mbox{and}\quad{\mathbf{v}}_2=\begin{bmatrix}3\\1\end{bmatrix}$$` --- # Data as a vector - Consider two examples:<!--D--> -- + The vector `\(\bf{v}\)` corresponds to a cage where `\(x\)` measures the number of chickens and `\(y\)` measures the number of rabbits.<!--D--> -- + The vector `\(\bf{v}\)` corresponds to a country where `\(x\)` measures the trade deficit in 2016 and `\(y\)` measures trade deficit in 2017.<!--D--> -- - Notice that if we have many cages or countries and plot each of these vectors as points we get a scatterplot --- # Vectors as points <img src="MatrixGeometry_files/figure-html/unnamed-chunk-1-1.png" style="display: block; margin: auto;" /> --- # Vector Operations - There are two fundamental *operations* that we can do with vectors<!--D--> -- + Vector *addition*<!--D--> -- + Scalar *multiplication*<!--D--> -- - We will look at each of these: <!--D--> -- + Geometrically (with pictures) and <!--D--> -- + Algebraically (with numbers) --- #Vector Addition <video controls><source src="MatrixGeometry_files/figure-html/addition.ogg" /><p>Click to see</p></video> --- # The other way <video controls><source src="MatrixGeometry_files/figure-html/FALSE.ogg" /><p>Click to see</p></video> --- # Vector Addition: Geometry - The operation of adding two vectors is simply to slide one along the other.<!--D--> -- - This is called *translation*<!--D--> -- - The order of adding vectors does not matter `\(\mathbf{v}_1+\mathbf{v}_2=\mathbf{v}_2+\mathbf{v}_1\)`<!--D--> -- - This property is called *commutativity* --- # With lots of points <video controls><source src="MatrixGeometry_files/figure-html/addpts.ogg" /><p>Click to see</p></video> --- # Vector Additions: Numbers Vector addition is as simple as adding up the corresponding numbers `$$\begin{bmatrix}\color{red}{1}\\\color{black}{2}\end{bmatrix}+ \begin{bmatrix}\color{red}{3}\\\color{black}{1}\end{bmatrix}= \begin{bmatrix}\color{red}{1+3}\\\color{white}{2+1}\end{bmatrix}= \begin{bmatrix}\color{white}{4}\\\color{white}{3}\end{bmatrix}$$` --- # Vector Additions: Numbers Vector addition is as simple as adding up the corresponding numbers `$$\begin{bmatrix}\color{red}{1}\\\color{black}{2}\end{bmatrix}+ \begin{bmatrix}\color{red}{3}\\\color{black}{1}\end{bmatrix}= \begin{bmatrix}\color{red}{1+3}\\\color{white}{2+1}\end{bmatrix}= \begin{bmatrix}\color{red}{4}\\\color{white}{3}\end{bmatrix}$$` --- # Vector Additions: Numbers Vector addition is as simple as adding up the corresponding numbers `$$\begin{bmatrix}\color{black}{1}\\\color{blue}{2}\end{bmatrix}+ \begin{bmatrix}\color{black}{3}\\\color{blue}{1}\end{bmatrix}= \begin{bmatrix}\color{black}{1+3}\\\color{blue}{2+1}\end{bmatrix}= \begin{bmatrix}\color{black}{4}\\\color{white}{3}\end{bmatrix}$$` --- # Vector Additions: Numbers Vector addition is as simple as adding up the corresponding numbers `$$\begin{bmatrix}\color{black}{1}\\\color{blue}{2}\end{bmatrix}+ \begin{bmatrix}\color{black}{3}\\\color{blue}{1}\end{bmatrix}= \begin{bmatrix}\color{black}{1+3}\\\color{blue}{2+1}\end{bmatrix}= \begin{bmatrix}\color{black}{4}\\\color{blue}{3}\end{bmatrix}$$` --- # Vector Addition and Data - Vector addition can be meaningful for data.<!--D--> -- - If I *add* a cage with 1 chicken and 3 rabbits to another cage with 2 chickens and 1 rabbits what do I get?<!--D--> -- + A cage with 3 chickens and 4 rabbits. --- # Vector Addition and Data - Suppose Australia had a trade deficit of $US1bn in 2016 and $US3bn in 2017 and New Zealand had a trade deficit of $US2bn in 2016 and $US1bn in 2017.<!--D--> -- - Australia and New Zealand merge to form a currency union and the trade deficit statistics need to be updated.<!--D--> -- + The new currency union had a trade deficit of $US3bn in 2016 and $US4bn in 2017. --- # Scalar multiplication - If we add a number `\(c\)` times we call that *multiplication*<!--D--> -- - We do something similar with vectors.<!--D--> -- - If we add a vector to itself `\(c\)` times this is called *scalar multiplication*.<!--D--> -- - We are multiplying a vector by a single number and NOT by another vector.<!--D--> -- - This is very different from *matrix multiplication*. --- # Scalar multiply by 2 <video controls><source src="MatrixGeometry_files/figure-html/smultgeom.ogg" /><p>Click to see</p></video> --- #Scalar multiply by 0.5 <video controls><source src="MatrixGeometry_files/figure-html/smultgeom2.ogg" /><p>Click to see</p></video> --- #Scalar multiplication by 2 <video controls loop><source src="MatrixGeometry_files/figure-html/multpts.ogg" /><p>Click to see</p></video> --- #Scalar multiplication by 0.5 <video controls loop><source src="MatrixGeometry_files/figure-html/multpts2.ogg" /><p>Click to see</p></video> --- # Scalar multiplication by 2 `$$\begin{aligned}{\color{red}{2}}{\mathbf{v}}_1&={\color{red}{2}}\begin{bmatrix}1\\2\end{bmatrix}\\&=\begin{bmatrix}{\color{red}{2}}\times 1\\{\color{red}{2}}\times 2\end{bmatrix}\\&=\begin{bmatrix}2\\4\end{bmatrix}\end{aligned}$$` --- # Scalar multiplication by 0.5 `$$\begin{aligned}{\color{red}{0.5}}{\mathbf{v}}_2&={\color{red}{0.5}}\begin{bmatrix}3\\1\end{bmatrix}\\&=\begin{bmatrix}{\color{red}{0.5}}\times 3\\{\color{red}{0.5}}\times 1\end{bmatrix}\\&=\begin{bmatrix}1.5\\0.5\end{bmatrix}\end{aligned}$$` --- # Scalar multiplication and data - How might scalar multiplication be relevant to data?<!--D--> -- - Suppose trade deficit is measured in Chinese Yuan instead of $US where 6 Yuan = $US1.<!--D--> -- - Then we can get the trade defcit in Yuan by multiplying by the scalar 6.<!--D--> -- - Suppose Australia had a trade deficit of $US1bn in 2016 and $US3bn in 2017.<!--D--> -- + This is equivalent to a trade deficit of 6bn Yuan in 2016 and 18bn Yuan in 2017. --- # Basis vector - Every vector can be created using<!--D--> -- + Scalar multiplication + Vector addition + Two *basis* vectors<!--D--> -- - The basis vectors we use are: `$${\mathbf{b}}_1=\begin{bmatrix}1\\0\end{bmatrix}\quad\mbox{and}\quad{\mathbf{b}}_2=\begin{bmatrix}0\\1\end{bmatrix}$$` --- # How the basis works - Suppose we want the vector `\(\begin{bmatrix}x\\y\end{bmatrix}\)` - This is given by `$$x\mathbf{b}_1+y\mathbf{b}_2$$` or `$$x\begin{bmatrix}1\\0\end{bmatrix}+y\begin{bmatrix}0\\1\end{bmatrix}$$` --- # Build a vector <video controls><source src="MatrixGeometry_files/figure-html/basis.ogg" /><p>Click to see</p></video> --- class: inverse, center, middle # Matrices --- # Matrices - You have learnt about matrices before as an array of numbers.<!--D--> -- - Today think about matrices as *transforming* points in space.<!--D--> -- - By multiplying a vector by a matrix, points in space are stetched, rotated and flipped around<!--D--> -- - But what does this have to do with an array of numbers? --- # Example of a matrix - Consider the following matrix `$$\mathbf{M}=\begin{bmatrix}5 &1\\2 &4\end{bmatrix}$$` - Think of this as something that moves all the points in space to a new address. - It encodes where `\(\mathbf{b}_1\)` and `\(\mathbf{b}_2\)` get moved to. --- # Change of basis Multiplying by the matrix `\(\mathbf{M}=\begin{bmatrix}{\color{magenta} 5} &\color{green}{1}\\{\color{magenta} 2} &{\color{green}4}\end{bmatrix}\)` will: Move the vector `\(\mathbf{b}_1=\begin{bmatrix}1\\0\end{bmatrix}\)` to `\(\mathbf{b}^{*}_1=\begin{bmatrix}{\color{magenta} 5}\\{\color{magenta} 2}\end{bmatrix}\)` Move the vector `\(\mathbf{b}_2=\begin{bmatrix}0\\1\end{bmatrix}\)` to `\(\mathbf{b}^{*}_2=\begin{bmatrix}{\color{green} 1}\\{\color{green} 4}\end{bmatrix}\)` Matrix multiplication is a very different type of multiplication than scalar multiplication. --- # Change of basis <img src="MatrixGeometry_files/figure-html/basischange-1.png" style="display: block; margin: auto;" /> --- #Old Basis `$$\begin{aligned} \mathbf{v}&=x\mathbf{b}_1+y\mathbf{b}_2\\ &=x\begin{bmatrix}1\\0\end{bmatrix}+y\begin{bmatrix}0\\1\end{bmatrix}\\ &=\begin{bmatrix}x\\y\end{bmatrix} \end{aligned}$$` --- # New Basis `$$\begin{aligned} \mathbf{v}^*&=x\mathbf{b}^{*}_1+y\mathbf{b}^{*}_2\\ &=x\begin{bmatrix}5\\2\end{bmatrix}+y\begin{bmatrix}1\\4\end{bmatrix}\\ &=\begin{bmatrix}5x+y\\2x+4y\end{bmatrix} \end{aligned}$$` --- # In general Multiplying by the matrix `\(\mathbf{M}\)` moves all vectors `\(\mathbf{v}\)` to `\(\mathbf{v}^*\)` We can write this as: `$$\mathbf{v}^*=\mathbf{Mv}=\begin{bmatrix}5 &1\\2 &4\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}5x+y\\2x+4y\end{bmatrix}$$` Another way to think of it is as multiplying the rows of `\({\mathbf M}\)` by the column `\(\mathbf {v}\)` --- # Matrix Multiplication in General <video controls><source src="MatrixGeometry_files/figure-html/matmult.mp4" /><p>Click to see</p></video> --- # Matrix Multiplication and Data An simple application of matrix multiplication is to create *new* variables. - Suppose we want a matrix to transform : + The number of chickens and number of rabbits to...<!--D--> -- + The number of heads and the number of feet. --- # Answer - First basis vector is the number of heads and feet of each chicken. - Second basis vector is the number of heads and feet of each rabbit. `$$\begin{bmatrix}1&1\\2&4\end{bmatrix}$$` --- # Another one - Suppose we want a matrix to transform: + The trade deficit in 2016 and trade deficit in 2017 to...<!--D--> -- + The average trade deficit from 2016 to 2017 and the change in the trade deficit between 2016 and 2017<!--D--> -- What does this matrix look like? Try yourselves. --- # Answer - First basis vector average and change if trade deficit was $1 in 2016 and $0 in 2017. - Second basis vector average and change if trade deficit was $0 in 2016 and $1 in 2017. `$$\begin{bmatrix}0.5&0.5\\-1&1\end{bmatrix}$$` --- # Non-Square Matrices - So far we have only considered `\(2\times 2\)` matrices. - These move every point in 2D space to another point in 2D space - However we can have a matrix with 3 rows and 2 columns (or `\(3\times2\)` matrix). `$$\mathbf{M}=\begin{bmatrix}1 &2\\3 &4\\5 &6\\\end{bmatrix}$$` --- # Non-square Matrices `$$\begin{aligned}\mathbf{Mz}&=\begin{bmatrix}1 &2\\3 &4\\5 &6\\\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}\\&=x\begin{bmatrix}1\\3\\5\end{bmatrix}+y\begin{bmatrix}2\\4\\6\end{bmatrix}\\&=\begin{bmatrix}x+2y\\3x+4y\\5x+6y\\\end{bmatrix}\end{aligned}$$` --- # Non-square matrices - Matrix multiplication works in the same way<!--D--> -- - However the *input* is a 2-dimensional vector but the *output* is a 3-dimensional vector<!--D--> -- - Multiplying by a non-square matrix changes the dimension -- - The points will still lie in a two-dimensional *subspace* of three-dimensional space. --- # In factor analysis - For an example of this consider the factor model and in particular the component `\(\boldsymbol{\Lambda}\boldsymbol{f}_i\)`. -- - This takes a low-dimensional set of factors and maps them to a high-dimensional vector of data. -- - What about multiplying by a `\(2\times 3\)` matrix? --- # Dimension reduction - Multiplying by a `\(2\times 3\)` matrix takes a `\(3\)`-dimensional vector and maps it to a point in `\(2\)`-dimensional space. -- - This is an example of *dimension reduction* -- - For instance taking the first two principal component of data is the same as multiplying the data by a `\(2\times p\)` matrix --- # Matrix composition - What does it mean to multiply matrices? -- - What does `\(\mathbf{F}\mathbf{G}\)` when `\(\mathbf{G}\)` is a `\(4\times 2\)` matrix and `\(\mathbf{F}\)` a `\(3\times 4\)` matrix? -- - It is as a *composition* of transformations:<!--D--> -- + Transform a `\(2\)`-dimensional vector `\(\mathbf{v}\)` into a `\(4\)`-dimensional vector by multiplying by the `\(4\times 2\)` matrix `\(\mathbf{G}\)`<!--D--> -- + Transorm the result into a `\(3\)`-dimensional vector by multiplying by a `\(3\times 4\)` matrix `\(\mathbf{F}\)` --- # Mathematically - When reading below go from right to left: `$$\underset{{ (3\times 2)}}{\mathbf{H}} \underset{{ \leftarrow}}{\times} \underset{{ (2\times 1)}}{\mathbf{v}} \underset{=}{{ =}} \underset{{ (3\times 4)}}{\mathbf{F}} \underset{{ \leftarrow}}{\times} \underset{{ (4\times 2)}}{\mathbf{G}} \underset{{ \leftarrow}}{\times} \underset{{ (2\times 1)}}{\mathbf{v}}$$` - The matrix `\(\mathbf{H}=\mathbf{FG}\)` is a composition of doing `\(\mathbf{G}\)` and then `\(\mathbf{F}\)` --- # Matrix multiplication - What happens if you try to multiply `\(\mathbf{GFv}\)`?<!--D--> -- `$$\underset{{ (4\times 2)}}{\mathbf{G}}\underset{{ \not\leftarrow}}{\times}\underset{{ (3\times 4)}}{\mathbf{F}}\underset{{ \not\leftarrow}}{\times}\underset{{ (2\times 1)}}{\mathbf{v}}$$`<!--D--> -- - Going from right to left, `\(\mathbf{F}\)` will try to transform a 4-dimensional vector to a 3-dimensional vector and cannot take the is a 2-dimensional vector `\(\mathbf{v}\)` as its input!<!--D--> -- - Also `\(\mathbf{G}\)` transforms a 2-dimensional vector to a 4-dimensional vector so cannot take the 3-dimensional output of `\(\mathbf{F}\)` as its input. --- # Matrix Multiplication - Although it is possible to get `\(\mathbf{FG}\)` it is impossible to get `\(\mathbf{GF}\)`.<!--D--> -- - In general, for matrix multiplication `\(\mathbf{AB}\neq\mathbf{BA}\)` even for square matrices.<!--D--> -- - Matrix multiplication is *non-commutative*. --- class: inverse, center, middle # Inner products and rotations --- # Inner product - One final operation with vectors is called the *inner product*. -- - The inner product of two vectors `\(\mathbf{x}\)` and `\(\mathbf{y}\)` is defined as `\(\mathbf{x}'\mathbf{y}\)`. -- - The inner product of a vector with itself `\(\mathbf{x}'\mathbf{x}\)` is the length of the vector squared. --- # Orthogonality - An interesting case is when the inner product between two vectors is zero. - `\(\mathbf{x}'\mathbf{y}=0\)` - In this case, the vectors `\(\mathbf{x}\)` and `\(\mathbf{y}\)` are at a 90 degree angle to one another. - This is also called *orthogonality*. --- # Rotation matrix - Another important matrix is known as a *orthogonal rotation* matrix.<!--D--> -- - It spins all the points around keeping them the same distance from the origin and from each other.<!--D--> -- - The word *orthogonal* means that the new basis vectors are at right angles to one another. --- # Rotation Matrix - In matrix algebra rotation can be defined in any dimension. - It is easiest to understand in two dimensions. - An example of a rotation matrix in two dimensions is: `$$\mathbf{R}=\begin{bmatrix}0.8 & -0.6\\0.6 & 0.8\end{bmatrix}$$` --- #Rotation Matrix <video controls><source src="MatrixGeometry_files/figure-html/matmultr.mp4" /><p>Click to see</p></video> --- # Rotation matrix - The new basis vectors should have a length of 1 therefore `\(\mathbf{r}_i'\mathbf{r}_i=1\)` where `\(\mathbf{r}_i\)` are the columns of `\(\mathbf{R}\)`. -- - The new basis vectors should be at right angles `\(\mathbf{r}_i'\mathbf{r}_j=0\)` -- - A rotation matrix `\(\mathbf{R}\)` has the property `\(\mathbf{R'R}=\mathbf{I}\)`<!--D--> --- # Rotations in PCA - In PCA the weights of the principal components have two properties - `\(\mathbf{w}_i'\mathbf{w}_i=1\)` - `\(\mathbf{w}_i'\mathbf{w}_j=0\)` - All `\(p\)` principal components can be obtained by multiplying by a rotation matrix by `\(p\)`-dimensional data. - Taking principal components *rotates* the data. --- # Rotations in Factor Analysis - In factor analysis rotation was used to obtain more interpretable factors. - This was a `\(r\)`-dimensional rotation of the factors rather than the data. - Again this involved multiplying by a rotation matrix. --- # Conclusion - Understand the geometry of matrix algebra assists in understanding PCA and Factor analysis in particular. - PCA forms new variables as linear combinations of old variables. - This corresponds to matrix multiplication. - The role of *rotation* should now also be understood. - The next step is to understand the importance of matrix decompositions in the methods covered so far.