hw7_Architecture_Design.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw7_Architecture_Design.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Iallen520/lhy_DL_Hw/blob/master/hw7_Architecture_Design.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8odNXcMV_wI8",
        "colab_type": "text"
      },
      "source": [
        "# Homework 7 - Network Compression (Architecuture Design)\n",
        "\n",
        "> Author: Arvin Liu (b05902127@ntu.edu.tw)\n",
        "\n",
        "若有任何問題，歡迎來信至助教信箱 ntu-ml-2020spring-ta@googlegroups.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7YIYiHkT4CLp",
        "colab_type": "text"
      },
      "source": [
        "# Readme\n",
        "\n",
        "HW7的任務是模型壓縮 - Neural Network Compression。\n",
        "\n",
        "Compression有很多種門派，在這裡我們會介紹上課出現過的其中四種，分別是:\n",
        "\n",
        "* 知識蒸餾 Knowledge Distillation\n",
        "* 網路剪枝 Network Pruning\n",
        "* 用少量參數來做CNN Architecture Design\n",
        "* 參數量化 Weight Quantization\n",
        "\n",
        "在這個notebook中我們會介紹MobileNet v1的Architecture Design。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTz5r-Zy4UDf",
        "colab_type": "text"
      },
      "source": [
        "# Architecture Design\n",
        "\n",
        "## Depthwise & Pointwise Convolution\n",
        "![](https://i.imgur.com/FBgcA0s.png)\n",
        "> 藍色為上下層Channel的關係，綠色則為該Receptive Field的擴張。\n",
        "> (圖片引用自arxiv:1810.04231)\n",
        "\n",
        "(a) 就是一般的Convolution Layer，所以他的Weight連接方式會跟Fully Connected一樣，只差在原本在FC是用數字相乘後相加，Convolution Layer是圖片卷積後相加。\n",
        "\n",
        "(b) DW(Depthwise Convolution Layer)你可以想像成一張feature map各自過**一個filter**處理後，再用PW(Pointwise Convolution Layer)把所有feature map的單個pixel資訊合在一起(就是1個pixel的Fully Connected Layer)。\n",
        "\n",
        "(c) GC(Group Convolution Layer)就是把feature map分組，讓他們自己過Convolution Layer後再重新Concat起來。算是一般的Convolution和Depthwise Convolution的折衷版。**所以說，Group Convolution的Group=Input Feautures數就會是Depthwise Convolution(因為每個Channel都各自獨立)，Group=1就會是一般的Convolution(因為就等於沒有Group)。**\n",
        "\n",
        "<img src=\"https://i.imgur.com/Hqhg0Q9.png\" width=\"500px\">\n",
        "\n",
        "\n",
        "## 實作細節\n",
        "```python\n",
        "# 一般的Convolution, weight大小 = in_chs * out_chs * kernel_size^2\n",
        "nn.Conv2d(in_chs, out_chs, kernel_size, stride, padding)\n",
        "\n",
        "# Group Convolution, Group數目可以自行控制，表示要分成幾群。其中in_chs和out_chs必須要可以被groups整除。(不然沒辦法分群。)\n",
        "nn.Conv2d(in_chs, out_chs, kernel_size, stride, padding, groups=groups)\n",
        "\n",
        "# Depthwise Convolution, 輸入chs=輸出chs=Groups數目, weight大小 = in_chs * kernel_size^2\n",
        "nn.Conv2d(in_chs, out_chs=in_chs, kernel_size, stride, padding, groups=in_chs)\n",
        "\n",
        "# Pointwise Convolution, 也就是1 by 1 convolution, weight大小 = in_chs * out_chs\n",
        "nn.Conv2d(in_chs, out_chs, 1)\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SRnRXK3zQzVO",
        "colab_type": "text"
      },
      "source": [
        "# Model\n",
        "\n",
        "* training的部分請參考Network Pruning、Knowledge Distillation，或直接只用Hw3的手把手即可。\n",
        "\n",
        "> 註記: 這邊把各個Block多用一層Sequential包起來是因為Network Pruning的時候抓Layer比較方便。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nrBEYCCC7JQP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch\n",
        "\n",
        "class StudentNet(nn.Module):\n",
        "    '''\n",
        "      在這個Net裡面，我們會使用Depthwise & Pointwise Convolution Layer來疊model。\n",
        "      你會發現，將原本的Convolution Layer換成Dw & Pw後，Accuracy通常不會降很多。\n",
        "\n",
        "      另外，取名為StudentNet是因為這個Model等會要做Knowledge Distillation。\n",
        "    '''\n",
        "\n",
        "    def __init__(self, base=16, width_mult=1):\n",
        "        '''\n",
        "          Args:\n",
        "            base: 這個model一開始的ch數量，每過一層都會*2，直到base*16為止。\n",
        "            width_mult: 為了之後的Network Pruning使用，在base*8 chs的Layer上會 * width_mult代表剪枝後的ch數量。        \n",
        "        '''\n",
        "        super(StudentNet, self).__init__()\n",
        "        multiplier = [1, 2, 4, 8, 16, 16, 16, 16]\n",
        "\n",
        "        # bandwidth: 每一層Layer所使用的ch數量\n",
        "        bandwidth = [ base * m for m in multiplier]\n",
        "\n",
        "        # 我們只Pruning第三層以後的Layer\n",
        "        for i in range(3, 7):\n",
        "            bandwidth[i] = int(bandwidth[i] * width_mult)\n",
        "\n",
        "        self.cnn = nn.Sequential(\n",
        "            # 第一層我們通常不會拆解Convolution Layer。\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(3, bandwidth[0], 3, 1, 1),\n",
        "                nn.BatchNorm2d(bandwidth[0]),\n",
        "                nn.ReLU6(),\n",
        "                nn.MaxPool2d(2, 2, 0),\n",
        "            ),\n",
        "            # 接下來每一個Sequential Block都一樣，所以我們只講一個Block\n",
        "            nn.Sequential(\n",
        "                # Depthwise Convolution\n",
        "                nn.Conv2d(bandwidth[0], bandwidth[0], 3, 1, 1, groups=bandwidth[0]),\n",
        "                # Batch Normalization\n",
        "                nn.BatchNorm2d(bandwidth[0]),\n",
        "                # ReLU6 是限制Neuron最小只會到0，最大只會到6。 MobileNet系列都是使用ReLU6。\n",
        "                # 使用ReLU6的原因是因為如果數字太大，會不好壓到float16 / or further qunatization，因此才給個限制。\n",
        "                nn.ReLU6(),\n",
        "                # Pointwise Convolution\n",
        "                nn.Conv2d(bandwidth[0], bandwidth[1], 1),\n",
        "                # 過完Pointwise Convolution不需要再做ReLU，經驗上Pointwise + ReLU效果都會變差。\n",
        "                nn.MaxPool2d(2, 2, 0),\n",
        "                # 每過完一個Block就Down Sampling\n",
        "            ),\n",
        "\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(bandwidth[1], bandwidth[1], 3, 1, 1, groups=bandwidth[1]),\n",
        "                nn.BatchNorm2d(bandwidth[1]),\n",
        "                nn.ReLU6(),\n",
        "                nn.Conv2d(bandwidth[1], bandwidth[2], 1),\n",
        "                nn.MaxPool2d(2, 2, 0),\n",
        "            ),\n",
        "\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(bandwidth[2], bandwidth[2], 3, 1, 1, groups=bandwidth[2]),\n",
        "                nn.BatchNorm2d(bandwidth[2]),\n",
        "                nn.ReLU6(),\n",
        "                nn.Conv2d(bandwidth[2], bandwidth[3], 1),\n",
        "                nn.MaxPool2d(2, 2, 0),\n",
        "            ),\n",
        "\n",
        "            # 到這邊為止因為圖片已經被Down Sample很多次了，所以就不做MaxPool\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(bandwidth[3], bandwidth[3], 3, 1, 1, groups=bandwidth[3]),\n",
        "                nn.BatchNorm2d(bandwidth[3]),\n",
        "                nn.ReLU6(),\n",
        "                nn.Conv2d(bandwidth[3], bandwidth[4], 1),\n",
        "            ),\n",
        "\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(bandwidth[4], bandwidth[4], 3, 1, 1, groups=bandwidth[4]),\n",
        "                nn.BatchNorm2d(bandwidth[4]),\n",
        "                nn.ReLU6(),\n",
        "                nn.Conv2d(bandwidth[5], bandwidth[5], 1),\n",
        "            ),\n",
        "\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(bandwidth[5], bandwidth[5], 3, 1, 1, groups=bandwidth[5]),\n",
        "                nn.BatchNorm2d(bandwidth[5]),\n",
        "                nn.ReLU6(),\n",
        "                nn.Conv2d(bandwidth[6], bandwidth[6], 1),\n",
        "            ),\n",
        "\n",
        "            nn.Sequential(\n",
        "                nn.Conv2d(bandwidth[6], bandwidth[6], 3, 1, 1, groups=bandwidth[6]),\n",
        "                nn.BatchNorm2d(bandwidth[6]),\n",
        "                nn.ReLU6(),\n",
        "                nn.Conv2d(bandwidth[6], bandwidth[7], 1),\n",
        "            ),\n",
        "\n",
        "            # 這邊我們採用Global Average Pooling。\n",
        "            # 如果輸入圖片大小不一樣的話，就會因為Global Average Pooling壓成一樣的形狀，這樣子接下來做FC就不會對不起來。\n",
        "            nn.AdaptiveAvgPool2d((1, 1)),\n",
        "        )\n",
        "        self.fc = nn.Sequential(\n",
        "            # 這邊我們直接Project到11維輸出答案。\n",
        "            nn.Linear(bandwidth[7], 11),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        out = self.cnn(x)\n",
        "        out = out.view(out.size()[0], -1)\n",
        "        return self.fc(out)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zPTYk9w-B_yt",
        "colab_type": "text"
      },
      "source": [
        "# Q&A\n",
        "\n",
        "有任何問題Network Compression的問題可以寄信到b05902127@ntu.edu.tw。\n",
        "\n",
        "我有空的話會更新在這裡。"
      ]
    }
  ]
}