[Keras] Autoencoders in Keras


Autoencoding is a data compression algorithm using neural networks
– Data-specific: An auto encoder trained on pictures of faces would do a rather poor job on pictures of trees.
Lossy: Decompressed outputs will be degraded compared to the original inputs.
– Learned automatically: Easy to train specialized instances of the algorithm that will perform well on a specific type of input.

You need three things to train
– Encoding function
– Decoding function
Distance function between the amount of information loss between the compressed representation of your data and the decompressed representation.

* In picture compression, it is pretty difficult to train an autoencoder that does a better job than a basic algorithm like JPEG
* You can only use them on data that is similar to what they were trained on, requires lots of training data.

For training, it fell out of fashion due to random initialization being better than them. Also, deeper network training was enabled by batch normalization and residual networks, and autoencoders were out… Now it is used for:
– Data denoising
– Dimensionality reduction for data visualization: appropriate dimensionality and sparsity constraint required (sometimes better than PCA, ex. t-SNE)


basic autoencoder
loss 0.1064, val_loss 0.1045
With this, we lose quite a bit of detail…
We were constrained by the size of the hidden layer (32).
Adding sparsity constraint (makes fewer units to fire at a given time) is good: in Keras, activity_regularizer.
sparse autoencoder
loss 0.1337, val_loss 0.1320 –> Actually this shouldn’t be bigger… I don’t know why this is.
Adding depth is effective
deep autoencoder
loss 0.1005, val_loss 0.0999
Using convolution to replace fully connected layers is effective too
convolutional autoencoder
loss 0.0976, val_loss 0.0960

Sequence inputs, you could use LSTMs
There is also Variational Autoencoders which comes just before GANs



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.