Outline

• Neural Network
  • An entire course could be done on neural networks, I will give only the basic idea.
• Idea behind Autoencoder
  • Has the same input and output.
  • Is narrow in the middle.

Autoencoders are Neural Nets

• Many of you are probably familiar with neural networks.
• The autoencoder is a neural network with a clever trick for dimension reduction.
• Briefly cover neural nets for those without any background.

The meat grinder

• Considers inputs $x$, a target $t$ and some function $f\left(\right)$ such that $f\left(x\right)$ is close to the target $t$
• Consider a the analogy of a meat grinder used to make salami.

Inside the grinder

Details

• What happens at each node?
• First a linear combination is taken of all inputs to a node
  • Made up of weights and biases (intercepts) that need to be estimated.
• The linear combination is passed through a non linear function
  • Sigmoid function
  • RELU

Training a neural network

• Consider a loss function that compares output to target
  • Mean square error
  • 0-1 loss
  • ...
• Find values of weights and biases that minimise the loss function over a training sample.

How to do this?

• Many computational tricks for finding optimal weights
  • Automatic differentiation
  • (Stochastic) Gradient Descent
  • Use of GPUs
• Only a different course could do this justice.

Trained Neural Net

Stretching the analogy

• Consider the aim is not to make salami but dimension reduction.
• The idea is to break ingredients down into protons, electrons and neutrons.
• A neural network can be constructed for this problem.
• The trick is to use all variables as both input and output.

An autoencoder

The key idea

• There is a "skinny" middle layer containing fewer nodes than input (and output) variables.
• Everything up to this skinny part is called "the encoder"
  • Think of this as breaking down ingredients into protons, neutrons and electrons.
• Everything past this skinny part is the "decoder"
  • Think of this as reconstructing the ingredients.

And that's it!

• An autoencoder is simply a neural network that
  • Has the same input and output
  • Get's "skinny" in the middle
• The advantages and disadvantages are the usual advantages and disadvantages of neural networks.

Advantages

• Very flexible and non-linear
• Can borrow ideas from neural network literature
  • Using convolutional layers for images
  • Using recurrent layers for time series
• Plenty of software and efficient packages

Software

• Can be implemented using the dimRed package.
• However this merely wraps around tensorflow
  • Need to have reticulate and Python properly configured
  • Can be a bit time consuming and tricky.
  • For autoencoders probably better to use Python.

Disadvantages

• Lots of tuning
  • How many layers?
  • Which activation function?
• Lots of parameters
  • Sparsity
• Lots of training data often needed