Contrastive_Representation_Distillation

Implementation of CRD (Contrastive Representation Distillation)

Quick Start

Teacher training

python main.py --option "teacher"

Student training without distillation

python main.py --option "student"

Student training with distillation

python main.py --option "distill" --teacher_model [teacher_model]

Test

python main.py --option [option] --test --prev_model [prev_model]

Result

Model	Teacher	Student (No Distill)	Studnet (Distill)
Beta	-	-	0.7	0.8	0.9	1.0	1.1
Accuracy	77.08%	76%	75.92%	75.57%	76.08%	76.27%	75.74%

Reference

1. Contrastive Representation Distillation (https://arxiv.org/abs/1910.10699)
2. CIFAR-100 Dataset (https://www.cs.toronto.edu/~kriz/cifar.html)
3. ResNet implementation (https://github.com/weiaicunzai/pytorch-cifar100/blob/master/models/resnet.py)