While the original repository works in Pytorch 0.3.1, the code in this repository works in Pytorch >= 0.4.1.
This repository contains the code for reproducing the results, and trained ImageNet models, in the following paper:
Rethinking the Value of Network Pruning. [arXiv] [OpenReview]
Zhuang Liu*, Mingjie Sun*, Tinghui Zhou, Gao Huang, Trevor Darrell (* equal contribution).
ICLR 2019. Also Best Paper Award at NIPS 2018 Workshop on Compact Deep Neural Networks.
Several pruning methods' implementations contained in this repo can also be readily used for other research purposes.
Our paper shows that for structured pruning, training the small pruned model from scratch can almost always achieve comparable or higher level of accuracy than the model obtained from the typical "training, pruning and fine-tuning" procedure. For those pruning methods:
- Training a large, over-parameterized model is not absolutely necessary to obtain an efficient final model.
- Learned “important” weights of the large model are typically not useful for the small pruned model.
- The pruned architecture itself, rather than a set of inherited “important” weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm.
Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods.
We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization. For more details please refer to our paper.
We evaluated the following seven pruning methods.
- L1-norm based channel pruning
- ThiNet
- Regression based feature reconstruction
- Network Slimming
- Sparse Structure Selection
- Soft filter pruning
- Unstructured weight-level pruning
The first six is structured while the last one is unstructured (or sparse). For CIFAR, our code is based on pytorch-classification and network-slimming. For ImageNet, we use the official Pytorch ImageNet training code. The instructions and models are in each subfolder.
For experiments on The Lottery Ticket Hypothesis, please refer to the folder cifar/lottery-ticket.
Our experiment environment is Python 3.6 & PyTorch 0.3.1.
Feel free to discuss papers/code with us through issues/emails!
sunmj15 at gmail.com
liuzhuangthu at gmail.com
If you use our code in your research, please cite:
@inproceedings{liu2018rethinking,
title={Rethinking the Value of Network Pruning},
author={Zhuang Liu and Mingjie Sun and Tinghui Zhou and Gao Huang and Trevor Darrell},
booktitle={ICLR},
year={2019}
}