class: center, middle, inverse, title-slide .title[ # Dimension Reduction:Autoencoders ] .author[ ### Anastasios Panagiotelis ] .institute[ ### University of Sydney ] --- # Outline - Neural Network -- + An entire course could be done on neural networks, I will give only the basic idea. -- - Idea behind Autoencoder -- + Has the same input and output. + Is narrow in the middle. --- class: center, middle, inverse # Neural Networks --- # Autoencoders are Neural Nets - Many of you are probably familiar with neural networks. -- - The autoencoder is a neural network with a clever trick for dimension reduction. -- - Briefly cover neural nets for those without any background. --- # The meat grinder - Considers inputs `\(\bx\)`, a target `\(t\)` and some function `\(f()\)` such that `\(f(\bx)\)` is close to the target `\(t\)` -- - Consider a the analogy of a meat grinder used to make salami.  --- # Inside the grinder <img src="05AutoEncoder_files/figure-html/unnamed-chunk-1-1.png" style="display: block; margin: auto;" /> --- # Details - What happens at each node? - First a linear combination is taken of all inputs to a node + Made up of weights and biases (intercepts) that need to be estimated. - The linear combination is passed through a non linear function + Sigmoid function + RELU --- # Training a neural network - Consider a loss function that compares output to target + Mean square error + 0-1 loss + ... -- - Find values of weights and biases that minimise the loss function over a training sample. --- # How to do this? - Many computational tricks for finding optimal weights + Automatic differentiation + (Stochastic) Gradient Descent + Use of GPUs -- - Only a different course could do this justice. --- # Trained Neural Net <img src="05AutoEncoder_files/figure-html/unnamed-chunk-2-1.png" style="display: block; margin: auto;" /> --- class:inverse, middle, center # Auto-encoders --- # Stretching the analogy - Consider the aim is not to make salami but dimension reduction. -- - The idea is to break ingredients down into protons, electrons and neutrons. -- - A neural network can be constructed for this problem. -- - The trick is to use all variables as both input and output. --- # An autoencoder <img src="05AutoEncoder_files/figure-html/unnamed-chunk-3-1.png" style="display: block; margin: auto;" /> --- # The key idea - There is a "skinny" middle layer containing fewer nodes than input (and output) variables. -- - Everything up to this skinny part is called "the encoder" -- + Think of this as breaking down ingredients into protons, neutrons and electrons. -- - Everything past this skinny part is the "decoder" -- + Think of this as reconstructing the ingredients. --- # And that's it! - An autoencoder is simply a neural network that -- + Has the same input and output + Get's "skinny" in the middle -- - The advantages and disadvantages are the usual advantages and disadvantages of neural networks. --- # Advantages - Very flexible and non-linear -- - Can borrow ideas from neural network literature -- + Using convolutional layers for images + Using recurrent layers for time series -- - Plenty of software and efficient packages --- # Software - Can be implemented using the `dimRed` package. -- - However this merely wraps around tensorflow -- + Need to have reticulate and Python properly configured -- + Can be a bit time consuming and tricky. -- + For autoencoders probably better to use Python. --- # Disadvantages - Lots of tuning + How many layers? + Which activation function? -- - Lots of parameters + Sparsity -- - Lots of training data often needed --- class: inverse, middle, center # Questions?