Awesome Autoencoder for Representation Learning

Last updated on October 20, 2020.

Introduction

The bottlenecked nature empowered autoencoder-based models (AEs) the ability to learn features of input data; the unsupervised and generative nature of AEs further facilitate the generalizability of the learned features, which is particularly useful in the scenario when unlabeled data is abundant whereas labeled data is scarce.

Down to a science, the future of machine learning to solve real-world tasks is likely to be generative models (to pretrain) followed by discriminative models (to predict). AEs (especially its variational families), as an important member of generative models, thus becomes crucial to study.

In this curated list of literature review, we will focus on recent (1) theories to understand the learning ability and characteristics of autoencoders, (2) models and applications exploiting autoencoders for representation learning and downstream tasks, and (3) adversarial attacks and defenses for autoencoders (we include this topic here, as it is important to reveal some crutial nature (e.g. robustness, smoothness, manifold properies etc.) of the latent space of AEs). We may also include some not so autoencoder relavant but representation learning relavant papers in this list.

The list is organized as follows:

• Survey
• Theory
• Models
• Applications (Here we are specifically interested in applications using encoder outputs for downstream task.)
• Attacks on Autoencoders
• Defense by Autoencoders
• Miscellaneous

In each section, papers are primarily organized by topic, then by conferences, lastly by chronological order. A short summary will be accompanied below the paper if necessary. 🧑🏻‍🚀 denotes important papers from my own perspective. 🐣 refers to the ones I haven't read but they look interesting, and would be added a summary below later.

Survey

Recent Advances in Autoencoder-Based Representation Learning [link]
Michael Tschannen, Olivier Bachem, Mario Lucic
3rd workshop on Bayesian Deep Learning (NeurIPS 2018)

Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey [link]
Naveed Akhtar, Ajmal Mian
IEEE Access, 2018

Theory

🧑🏻‍🚀 Disentangling Adversarial Robustness and Generalization [link]
David Stutz, Matthias Hein, Bernt Schiele
CVPR, 2019

This papers shows that for a data manifold:

1. There are generally two types of adversarial examples: off-manifold and on-manifold.
2. On-manifold adversarial examples are generalization errors, and on-manifold adversarial training improves generalization.
3. Regular robutsness and generalization are not contradicting.

Guess First to Enable Better Compression and Adversarial Robustness [link]
Sicheng Zhu, Bang An, Shiyu Niu
Information Theory and Machine Learning Workshop, NIPS, 2019

This paper shows that better compression ($I(X; Z)$) and less label information ($I(Z; Y)$) improves adversarial robustness. This two properties can be useful in designing robust AEs for downstream tasks.

🧑🏻‍🚀 Towards a Theoretical Understanding of the Robustness of Variational Autoencoders [link]
Alexander Camuto, Matthew Willetts, Stephen Roberts, Chris Holmes, Tom Rainforth
Preprint, 2020

This paper provides a general metric to evaluate the robustness of probabilitic models: $r$-robustness.

• Specifically, it shows that we are able to define a region which which any perturbation will produce a reconstruction similar to the original reconstruction.

Models

Relational Autoencoder for Feature Extraction [link]
Qinxue Meng, Daniel Catchpoole, David Skillicorn, Paul J. Kennedy
International Joint Conference on Neural Networks (ICJNN), 2017

🧑🏻‍🚀 World Models [link] [website] [talk]
David Ha, Jürgen Schmidhuber
NIPS, 2018

One of the greatest paper in NIPS. Its subtitle is: Can Agents Learn Inside of Their Own Dreams?
The World Model can be conceived as this two-stage process:

1. Learn a compressed representation of the environment in an unsupervised manner;
2. Use the learned representation to train a policy to solve downstream tasks.

Supervised Autoencoders [link]
Lei Le, Andrew Patterson, Martha White
NIPS, 2018

From Variational to Deterministic Autoencoders [link] [code]
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, Bernhard Scholkopf
ICLR, 2020

🐣 Parameterized Rate-Distortion Stochastic Encoder [link]
Quan Hoang, Trung Le, Dinh Phung
ICML, 2020

🐣 Provably robust deep generative models [link]
Filipe Condessa, Zico Kolter
Preprint, 2020

Applications

Learning Visual Feature Spaces for Robotic Manipulation with Deep Spatial Autoencoders [link]
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, Pieter Abbeel
International Conference on Robotics and Automation (ICRA), 2015

Reinforcement Learning on Robot with Variational Auto-Encoder [link]
Yiwen Chen, Chenguang Yang, Ying Feng
International Conference on Modelling, Identification and Control (ICMIC), 2019

Autoencoder-Based Transfer Learning in Brain Computer Interface [link]
Chuanqi Tan, Fuchun Sun, Bin Fang, Tao Kong, Wenchang Zhang
International Journal of Advanced Robotic Systems (IJARS), 2019

This authors propose to use autoencoder to extract EEG data; the extracted features are then used to do classification tasks. One interesting thing is that to tackle the sample scarcity in training the autoencoder, they first transfer from ImageNet.

Variational Autoencoder for Semi-supervised Text Classification [link]
Weidi Xu, Haoze Sun, Chao Deng, Ying Tan
AAAI, 2017

Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records [link]
R. Miotto, Li Li, B. Kidd, J. Dudley
Scientific Reports, 2016

This paper uses a 3-layer denoising autoencoder to learn representations for raw EHR data.

Semi-Supervised Learning of the Electronic Health Record for Phenotype Stratification [link]
Brett K. Beaulieu-Jonesab, Casey S. Green
Journal of Biomedical Informatics, 2016

This paper uses the enoising autoencoder with the random forest classifier to predict survival rates of patients.

Representation Learning with Autoencoders for Electronic Health Records: A Comparative Study [link]
Najibesadat Sadati, Milad Zafar Nezhad, Ratna Babu Chinnam, Dongxiao Zhu
Preprint, 2019

This paper gives a general framework of using autoencoder to extract features of EHR data, then build prediction models on top of them.

Deep Manifold Preserving Autoencoder for Classifying Breast Cancer Histopathological Images [link]
Yangqin Feng, Lei Zhang, Juan Mo
IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020

This paper uses a simple structure: a pretrained encoder plus a softmax classifier.

Attacks

Adversarial Images for Variational Autoencoders [link] [code]
Pedro Tabacof, Julia Tavares, Eduardo Valle
PAdversarial Training Workshop (NIPS, 2016)

Adversarial Attacks on Variational Autoencoders [link]
George Gondim-Ribeiro, Pedro Tabacof, Eduardo Valle
CoRR, 2018

Adversarial Examples for Generative Models [link]
Jernej Kos, Ian Fischer, Dawn Song
IEEE S&P Workshops, 2018

Provides 3 different schemes resulting in reconstruction change.

Constructing Unrestricted Adversarial Examples with Generative Models [link] [code]
Yang Song, Rui Shu, Nate Kushman, Stefano Ermon
NIPS, 2018

LatentPoison - Adversarial Attacks On The Latent Space [link]
Antonia Creswell, Anil A. Bharath, Biswa Sengupta
Rejected by ICLR, 2018

When Deep Fool Meets Deep Prior: Adversarial Attack on Super-Resolution Network [link]
Minghao Yin, Yongbing Zhang, Xiu Li, Shiqi Wang
Proceedings of the 26th ACM international conference on Multimedia (MM, 2018)

AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks [link]
Chun-Chen Tu, Paishun Ting, Pin-Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, Shin-Ming Cheng
AAAI, 2019

Previously, black-box attack is notorious at the large amount of query needed. The proposed black-box attack is featured by query-efficiency. It has: (1) an adaptive random gradient estimation and (2) an autoencoder accelerates the attack.

Adversarial Out-domain Examples for Generative Models [link]
D. Pasquini, M. Mingione, M. Bernaschi
IEEE European Symposium on Security and Privacy Workshops (Euro S&PW, 2019)

🐣 Physical Adversarial Attacks Against End-to-End Autoencoder Communication Systems [link]
Meysam Sadeghi, Erik G. Larsson
IEEE Communications Letters, 2019

A physical attack for communication system by well-designed perturbation signal over the channel, which is more destructive than jamming attacks.

🐣 Generalizable Adversarial Attack Using Generative Models [link]
Avishek Joey Bose, Andre Cianflone, William L. Hamilton
Preprint, 2019

It seems that the idea is that adversarial attacks can be viewed as a generative modelling problem, i.e. given an unperturbed input, generate an adversarial example. The authors achieves this by an encoder-decoder framework.

Performing Co-Membership Attacks Against Deep Generative Models [link]
Kin Sum Liu, Chaowei Xiao, Bo Li, Jie Gao
Preprint, 2019

Man-in-the-Middle Attacks against Machine Learning Classifiers via Malicious Generative Models [link]
Derui Wang, Chaoran Li, Sheng Wen, Surya Nepal, Yang Xiang
Preprint, 2019

Adversarial Attack Type I: Cheat Classifiers by Significant Changes [link]
Sanli Tang, Xiaolin Huang, Mingjian Chen, Chengjin Sun, Jie Yang
Preprint, 2019

An attack on classifiers using gradient information from the latent space of autoencoders.

🐣 Type I Attack for Generative Models [link]
Chengjin Sun, Sizhe Chen, Jia Cai, Xiaolin Huang
Preprint, 2020

One example attack on VAE by this paper is that the proposed attack can change an original image significantly to a meaningless one but their reconstruction results are similar.

🐣 Towards Feature Space Adversarial Attack [link]
Xu Q, Tao G, Cheng S, Tan L, Zhang X.
Preprint, 2020

BAAAN: Backdoor Attacks Against Autoencoder and GAN-Based Machine Learning Models [link]
Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, Yang Zhang
Preprint, 2020

An Adversarial Attack against Stacked Capsule Autoencoder [link]
Jiazhu Dai, Siwei Xiong
Preprint, 2020

🐣 Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder [link]
Alvin Chan, Yi Tay, Yew-Soon Ong, Aston Zhang
Preprint, 2020

T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack [link]
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, Bo Li
Preprint, 2020

🐣 Don't Trigger Me! A Triggerless Backdoor Attack Against Deep Neural Networks [link]
Ahmed Salem, Michael Backes, Yang Zhang
Preprint, 2020

Though irrelevant to AE, this paper discusses a type of backdoor attack without triggers which may potentically influence the applications on AE.

🐣 On Breaking Deep Generative Model-based Defenses and Beyond [link]
Yanzhi Chen, Renjie Xie, Zhanxing Zhu
ICML, 2020

Understanding Classifier Mistakes with Generative Models [link]
Laëtitia Shao, Yang Song, Stefano Ermon
Submitted to ICLR, 2021

Defenses

Deep Variational Information Bottleneck [link]
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy
ICLR, 2017

Adversarial Defense of Image Classification Using a Variational Auto-Encoder [link]
Yi Luo, Henry Pfister
Preprint, 2018

Adversarial Defense based on Structure-to-Signal Autoencoders [link]
Joachim Folz, Sebastian Palacio, Joern Hees, Damian Borth, Andreas Dengel
Preprint, 2018

The idea is similar to the above one – autoencoder can be used as a pre-processing step for extracting high-level features robust to adversarial perturbations.

Combatting Adversarial Attacks through Denoising and Dimensionality Reduction: A Cascaded Autoencoder Approach [link]
Rajeev Sahay, Rehana Mahfuz, Aly El Gamal
Preprint, 2018

🐣🧑🏻‍🚀 Are Generative Classifiers More Robust to Adversarial Attacks? [link] [code]
Yingzhen Li, John Bradshaw, Yash Sharma
Rejected by ICLR Workshop, 2018; then accepted by ICML, 2019

This paper is more on the robustness of bayes classifiers compared to deterministic classifiers.

• Notably, it applies generative modeling (variational inference) to improve original bayes classifers.
• It implies that, generative models may fascilitate gradient masking which in turn become more robust to attacks. The stochastic nature of generative models may play an important role for gradient masking.

Sufficient Conditions for Robustness to Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks [link]
Yarin Gal, Lewis Smith
Rejected by ICLR Workshop, 2019

This paper proves, under two sufficient conditions, that idealised models can have no adversarial examples.

Combatting Adversarial Attacks through Denoising and Dimensionality Reduction: A Cascaded Autoencoder Approach [link]
Rajeev Sahay, Rehana Mahfuz, Aly El Gamal
Submitted to CISS, 2019

Idea: autoencoder can be used as the preprocessing for raw inputs. The preprocessing can be done in two steps: denoising and dimension reduction (e.g. use the bottleneck feature from a denoising AE to represent the original input data). The resulting features are shown to be more robust.

🧑🏻‍🚀 Disentangled Deep Autoencoding Regularization for Robust Image Classification [link]
Zhenyu Duan, Martin Renqiang Min, Li Erran Li, Mingbo Cai, Yi Xu, Bingbing Ni
Preprint, 2019

The central idea is that: disentaglement helps adversarial robustness. The proposed defense mechanism takes disentaglement into regularization terms, and the resulted features are shown to be more robust.
One interesting, though a bit deviating thing, is that the paper mentions that, "a recent neuroscience discovery revealing that primate brain employs disentangled shape and appearance representations for object recognition". Wow.

🧑🏻‍🚀 PuVAE: A Variational Autoencoder to Purify Adversarial Examples [link]
Uiwon Hwang, Jaewoo Park, Hyemi Jang, Sungroh Yoon, Nam Ik Cho
Preprint, 2019

The idea is similar to the previous ones – obtain a more robust feature of raw inputs by autoencoding.
This paper also specifies the manifold assumptions, that the features learnt by autoencoders are projections on data manifold.

DAPAS : Denoising Autoencoder to Prevent Adversarial attack in Semantic Segmentation [link]
Seungju Cho, Tae Joon Jun, Byungsoo Oh, Daeyoung Kim Preprint, 2019 (accepted by IJCNN'20 now)

Similar idea of the above one, yet using denoise autoencoders.

🐣 Towards Model-Agnostic Adversarial Defenses using Adversarially Trained Autoencoders [link]
Pratik Vaishnavi, Kevin Eykholt, Atul Prakash, Amir Rahmati
Preprint, 2019

Mitigation of Adversarial Examples in RF Deep Classifiers Utilizing AutoEncoder Pre-training [link]
Silvija Kokalj-Filipovic, Rob Miller, Nicholas Chang, Chi Leung Lau
Preprint, 2019

A simple application using the idea that features are more robust after processed by autoencoders.

🧑🏻‍🚀 Improving VAE's Robutsness to Adversarial Attacks [link]
M Willetts, A Camuto, S Roberts, C Holmes
Preprint, 2019

This paper introduces a hierarchical VAE which can improve adverarial robustness while preserving reconstruction ability.

• This idea is based on the observation that disentangled representation improves robustness yet reducing the quality of reconsturction ability.

Certified Robustness to Adversarial Examples with Differential Privacy [link]
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, Suman Jana
Preprint, 2019

This paper provides a defense which could be done in the feature space (by adversarial smoothing).

Evaluating Robustness of Deep Image Super-Resolution Against Adversarial Attacks [link]
J. Choi, H. Zhang, J. Kim, C. Hsieh, J. Lee
ICCV, 2019

🐣 Resisting Adversarial Attacks Using Gaussian Mixture Variational Autoencoders [link] [video] [blog]
Partha Ghosh, Arpan Losalka, Michael J. Black
AAAI, 2019

This paper is also relevant to VampPrior VAE (link, slides).

Defense-VAE: A Fast and Accurate Defense Against Adversarial Attacks [link]
Xiang Li, Shihao Ji
PKDD, 2019

This paper uses VAE to purge adversarial perturbations from contaminated images, and shows this preprocessing can help defend both white-box and black-box attacks.

Bridging Adversarial Robustness and Gradient Interpretability [link]
Beomsu Kim, Junghoon Seo, Taegyun Jeon
Safe Machine Learning Worshop of ICLR, 2019

• This papers shows that adversarial training makes gradients more interpretable.
• It also shows that there is a trade-off between test accuracy and gradient interpretability.
• It then provides ways to mitigate this trade-off.

🧑🏻‍🚀 Adversarially Robust Representations with Smooth Encoders [link]
Taylan Cemgil, Sumedh Ghaisas, Krishnamurthy (Dj) Dvijotham, Pushmeet Kohli
ICLR, 2020

T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack [link]
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, Bo Li
EMNLP, 2020

Evaluating the Robustness of Defense Mechanisms based on AutoEncoder Reconstructions against Carlini-Wagner Adversarial Attacks [link]
Petru Hlihor, Riccardo Volpi, Luigi Malagò
Proceedings of the Northern Lights Deep Learning Workshop, 2020

Similar to the above one, this paper shows that reconstruction by autoencoders is an effective preprocessing approach on images to defend common adversarial attacks.

🐣 Double Backpropagation for Training Autoencoders against Adversarial Attack [link]
Chengjin Sun, Sizhe Chen, Xiaolin Huang
Preprint, 2020

This paper proposes a training procedure to enhance the robustness of AEs.

• It is based on the observation that AEs are sensitive to inputs, i.e., one can slightly modify an input but has totally different codes (👀 is that so?).
• Therefore, the authors restrict gradients from the reconstruction image to the original one, making AEs less sensitive to small perturbation.

Defending Adversarial Attacks via Semantic Feature Manipulation [link]
Shuo Wang, Tianle Chen, Surya Nepal, Carsten Rudolph, Marthie Grobler, Shangyu Chen
Preprint, 2020

ARAE: Adversarially Robust Training of Autoencoders Improves Novelty Detection [link]
Mohammadreza Salehi, Atrin Arya, Barbod Pajoum, Mohammad Otoofi, Amirreza Shaeiri, Mohammad Hossein Rohban, Hamid R. Rabiee
Preprint, 2020

🐣 Metrics and Methods for Robustness Evaluation of Neural Networks with Generative Models [link]
Igor Buzhinsky, Arseny Nerinovsky, Stavros Tripakis
Preprint, 2020

This paper provides latent space performance metrics to evaluate models' robustness.

🐣 Adversarial Examples Detection and Analysis with Layer-wise Autoencoders [link]
Bartosz Wójcik, Paweł Morawiecki, Marek Śmieja, Tomasz Krzyżek, Przemysław Spurek, Jacek Tabor
Preprint, 2020

This paper uses autoencoders to do defense. The assumption is that adversarial example do not lie on the manifold of true data. The paper then uses autoencoders to find such manifold (i.e. using its latent space to approximate the manifold).
It needs to be note that, this approach can be vulnerable, especially when the latent spaces learnt by autoencoders deviate from true data manifold.
Reminds me of adding constraints on AEs for better approximation for data manifold.

DefenseVGAE: Defending against Adversarial Attacks on Graph Data via a Variational Graph Autoencoder [link]
Ao Zhang, Jinwen Ma
Preprint, 2020

Old idea (reconstructed ones are most robust than original inputs), new applications (graphs rather than images).

🐣 ARAE: Adversarially Robust Training of Autoencoders Improves Novelty Detection [link]
Mohammadreza Salehi, Atrin Arya, Barbod Pajoum, Mohammad Otoofi, Amirreza Shaeiri, Mohammad Hossein Rohban, Hamid R. Rabiee
Preprint, 2020

Revisiting Role of Autoencoders in Adversarial Settings [link]
Byeong Cheon Kim, Jung Uk Kim, Hakmin Lee, Yong Man Ro
ICIP, 2020

A short paper discusses the existence of robustness in autoencoder models (evaluated by classification performance). The central idea seems to be a bit shallow, but still important: AEs learn less discriminative features, enabling them to be more robust.

Miscellaneous

(This section can be skipped. It is not so relevant to autoencoder and is more on adversarial attacks of feature space.)

Perturbation Analysis of Learning Algorithms: A Unifying Perspective on Generation of Adversarial Examples [link]
Emilio Rafael Balda, Arash Behboodi, Rudolf Mathar
Preprint, 2018

Robustness Analysis of Deep Neural Networks in the Presence of Adversarial Perturbations and Noisy Labels [link]
Emilio Rafael Balda Cañizares
Preprint, 2019

This paper provides an information-theoretical view on learning with noisy labels.

Protecting Against Image Translation Deepfakes by Leaking Universal Perturbations from Black-Box Neural Networks [link]
Nataniel Ruiz, Sarah Adel Bargal, Stan Sclaroff
Preprint, 2020

DAPAS : Denoising Autoencoder to Prevent Adversarial attack in Semantic Segmentation [link]
Seungju Cho, Tae Joon Jun, Byungsoo Oh, Daeyoung Kim
Preprint, 2020

🧑🏻‍🚀 Double Backpropagation for Training Autoencoders against Adversarial Attack [link]
Chengjin Sun, Sizhe Chen, Xiaolin Huang
Preprint, 2020

The paper provides a gradient smoothing method specially designed for Autoencoder models.

Randomization matters How to defend against strong adversarial attacks [link]
Rafael Pinot, Raphael Ettedgui, Geovani Rizk, Yann Chevaleyre, Jamal Atif
ICML, 2020

Understanding and Mitigating the Tradeoff between Robustness and Accuracy [link]
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, Percy Liang
ICML, 2020

Traditional undertanding is that there exists a tradeoff between robust error (i.e. perturbed worst-case ones) and standard error (i.e. unperturbed ones). This paper suggests that this tradeoff can be well mitigated, i.e. we can improve robust error and standard error at the same time.

Randomized Smoothing of All Shapes and Sizes [link]
Greg Yang, Tony Duan, J. Edward Hu, Hadi Salman, Ilya Razenshteyn, Jerry Li
ICML, 2020

Adversarial Neural Pruning with Latent Vulnerability Suppression [link]
Divyam Madaan, Jinwoo Shin, Sung Ju Hwang
ICML, 2020