14_rl/05_Deep_Q_Network_application.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/machine-perception-robotics-group/MPRGDeepLearningLectureNotebook/blob/master/14_rl/05_Deep_Q_Network_application.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P9WNOydibxSw"
      },
      "source": [
        "# 目的\n",
        "強化学習におけるエージェントや環境設計(報酬， 行動, etc.)の重要さを理解する．"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WCbgZdMx8uBm"
      },
      "source": [
        "## 問題設定\n",
        "\n",
        "Deep Q-Networkを用いて，波形の振幅/周波数/波長を制御し目標のsin波にフィッティングさせるAIを作成してみます．\n",
        "波形調整する環境を自作し，エージェントや環境の振る舞い(報酬，行動など）を設計する．\n",
        "\n",
        "\n",
        "**Deep Q-Network (DQN)**\n",
        "\n",
        "> Deep Q-Network (DQN)は，GoogleのDeepmindが2016年に発表した手法で，Q学習におけるQテーブルを用いた行動価値の導出を，DCNNを用いた近似関数で代用するのが主な手法の内容です．\n",
        "Q学習では，全ての状態と行動の組み合わせについて，行動価値をQテーブルに記録している．\n",
        "そのため，状態数と行動数の組み合わせが膨大な環境に対して，膨大なメモリが必要となる問題を抱えていました．\n",
        "この問題に対して，DQNはQテーブルそのものをDCNNで代用することにより，上記問題を解決しました．\\\n",
        "また，DQNにはその他にも強化学習における学習の安定性獲得のために，Experience replay，Target Q-Network，Reward clippingなどの工夫がなされている．\n",
        "\n",
        "\n",
        "<img src=\"https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2708544/0d5711a1-803f-f142-125e-6b70f84a3c77.png\" width = 50%>\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5rQGfxWYK_4O"
      },
      "source": [
        "## 準備\n",
        "\n",
        "### Google Colaboratoryの設定確認・変更\n",
        "本チュートリアルではPyTorchを利用してニューラルネットワークの実装を確認，学習および評価を行います．\n",
        "**GPUを用いて処理を行うために，上部のメニューバーの「ランタイム」→「ランタイムのタイプを変更」からハードウェアアクセラレータをGPUにしてください．**\n",
        "\n",
        "\n",
        "### モジュールの追加インストール\n",
        "下記のプログラムを実行して，実験結果の表示に必要な追加ライブラリやモジュールをインストールする．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!apt update && apt install xvfb\n",
        "!pip -q install pyglet\n",
        "!pip -q install pyopengl\n",
        "!pip -q install pyvirtualdisplay"
      ],
      "metadata": {
        "id": "fgEasFDRTf3J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CKpJ1mtDFoDo"
      },
      "source": [
        "## モジュールのインポート\n",
        "はじめに必要なモジュールをインポートする．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import time\n",
        "import datetime\n",
        "import random\n",
        "import cv2\n",
        "import matplotlib.pyplot as plt\n",
        "import collections\n",
        "%matplotlib inline\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "import torchvision.transforms as T\n",
        "\n",
        "import warnings\n",
        "warnings.simplefilter('ignore')\n",
        "\n",
        "# GPUの確認\n",
        "use_cuda = torch.cuda.is_available()\n",
        "print('Use CUDA:', use_cuda)"
      ],
      "metadata": {
        "id": "Mt0-U8lhTh77"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2-BDmHImPZ3T"
      },
      "source": [
        "## シード値の固定"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "seed = 123\n",
        "# Python random\n",
        "random.seed(seed)\n",
        "# Numpy\n",
        "np.random.seed(seed)\n",
        "# Pytorch\n",
        "torch.manual_seed(seed)\n",
        "torch.cuda.manual_seed(seed)\n",
        "torch.backends.cudnn.deterministic = True\n",
        "torch.use_deterministic_algorithms = True"
      ],
      "metadata": {
        "id": "9jWht8tzTkzq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jo5u_p4SFoMa"
      },
      "source": [
        "## 自作強化学習環境の定義\n",
        "\n",
        "タスクの目的：波形を制御し，目標の波形へフィッティングさせること\n",
        "\n",
        "制御波形の振幅/周波数/波長を制御し，目標波形(sin波)に適合させる環境を自作する．"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D3-_NG5jk5Dd"
      },
      "source": [
        "**WAVEクラス**\n",
        "\n",
        "制御波形を定義/パラメータ更新するためのクラスです．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 波形クラス\n",
        "class WAVE():\n",
        "  def __init__(self, x):\n",
        "    \"\"\"\n",
        "    波形の初期設定. 各パラメータの上下限を設定.\n",
        "    \"\"\"\n",
        "    self.x = x # x情報\n",
        "\n",
        "    # 振幅\n",
        "    self.A = 0\n",
        "    self.range_A = [0.1, 2.0]\n",
        "    # 周期\n",
        "    self.T = 0\n",
        "    self.range_T = [0.1, 3.0]\n",
        "    # 周波数\n",
        "    self.f = 0\n",
        "    # 波長\n",
        "    self.lam = 0\n",
        "    self.range_lam = [0.1, 3.0]\n",
        "\n",
        "  def set_seed(self, n_seed):\n",
        "    \"\"\"\n",
        "    seed値の設定．\n",
        "    \"\"\"\n",
        "    random.seed(n_seed)\n",
        "    np.random.seed(n_seed)\n",
        "    return\n",
        "\n",
        "  def init_wave(self):\n",
        "    \"\"\"\n",
        "    制御波形の初期化．\n",
        "    \"\"\"\n",
        "    # 制御波形の初期パラメータ (ランダムver)\n",
        "    #self.A = round(random.uniform(self.range_A[0], self.range_A[1]), 1)\n",
        "    #self.T = round(random.uniform(self.range_T[0], self.range_T[1]), 1)\n",
        "    #self.f = 1.0/self.T\n",
        "    #self.lam = round(random.uniform(self.range_lam[0], self.range_lam[1]), 1)\n",
        "\n",
        "    # 制御波形の初期パラメータ (固定ver)\n",
        "    self.A = 2.0\n",
        "    self.T = 1.0\n",
        "    self.f = 1.0/self.T\n",
        "    self.lam = 1.0\n",
        "\n",
        "    y = self.A * np.sin(2 * np.pi * (self.f - self.x/self.lam))\n",
        "    return y.astype(float)\n",
        "\n",
        "  def checkparam(self):\n",
        "    \"\"\"\n",
        "    initで設定した各パラメータの上下限をもとに各パラメータが設定値を大幅に超えないように制限．\n",
        "    \"\"\"\n",
        "    # 振幅チェック\n",
        "    if self.A < self.range_A[0]: self.A = self.range_A[0]\n",
        "    elif self.A > self.range_A[1]: self.A = self.range_A[1]\n",
        "    # 周期チェック\n",
        "    if self.T < self.range_T[0]: self.T = self.range_T[0]\n",
        "    elif self.T > self.range_T[1]: self.T = self.range_T[1]\n",
        "    # 波長チェック\n",
        "    if self.lam < self.range_lam[0]: self.lam = self.range_lam[0]\n",
        "    elif self.lam > self.range_lam[1]: self.lam = self.range_lam[1]\n",
        "    return\n",
        "\n",
        "  def update(self, x):\n",
        "    \"\"\"\n",
        "    現パラメータにおける波形を生成\n",
        "    \"\"\"\n",
        "    self.f = 1.0/self.T\n",
        "    y = self.A * np.sin(2.0 * np.pi * (self.f - x/self.lam))\n",
        "    return y.astype(float)"
      ],
      "metadata": {
        "id": "eq_NJhEZUODQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Zbbzp6NnV27"
      },
      "source": [
        "**ENVクラス**\n",
        "\n",
        "環境を定義しているクラスです． 上記のWAVEクラスはここで使用します．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 環境クラス\n",
        "class ENV():\n",
        "  def __init__(self, episode_max_step=500, obs_range=100, obs_mode=\"num\"):\n",
        "    \"\"\"\n",
        "    環境の設定\n",
        "    episode_max_step：　1エピソード間のstep数\n",
        "    obs_range：何時刻分の観測値を返すか (観測値が数値情報時のみ)\n",
        "    obs_mode：観測値を画像(image)として返すか数値情報(num)として返すか\n",
        "    \"\"\"\n",
        "    self.obs_mode = obs_mode\n",
        "    if self.obs_mode == \"num\":\n",
        "      self.observation = np.zeros(obs_range*2) # fit_wave + target_wave\n",
        "    elif self.obs_mode == \"image\":\n",
        "      self.observation = np.zeros((1,84,84)) # image\n",
        "    self.time_step = 0\n",
        "    self.episode_max_step = episode_max_step\n",
        "\n",
        "    # 選択可能な行動リスト [振幅，周期,波長]\n",
        "    self.action_list = [\n",
        "        [0.0, 0.0, 0.0],\n",
        "        [0.0, 0.0, 0.1],\n",
        "        [0.0, 0.1, 0.0],\n",
        "        [0.0, 0.1, 0.1],\n",
        "        [0.1, 0.0, 0.0],\n",
        "        [0.1, 0.0, 0.1],\n",
        "        [0.1, 0.1, 0.0],\n",
        "        [0.1, 0.1, 0.1],\n",
        "        [0.0, 0.0, -0.1],\n",
        "        [0.0, -0.1, 0.0],\n",
        "        [0.0, -0.1, -0.1],\n",
        "        [-0.1, 0.0, 0.0],\n",
        "        [-0.1, 0.0, -0.1],\n",
        "        [-0.1, -0.1, 0.0],\n",
        "        [-0.1, -0.1, -0.1]\n",
        "      ]\n",
        "\n",
        "    self.x = np.linspace(0, 6*np.pi, obs_range) # x情報\n",
        "\n",
        "    # 目標波形：sin波\n",
        "    self.target_wave = 1.0 * np.sin(2.0 * np.pi * (0.5 - self.x/2.0))\n",
        "\n",
        "    # 制御波形の初期化\n",
        "    self.fit_wave = WAVE(self.x)\n",
        "\n",
        "\n",
        "  def action_space(self):\n",
        "    \"\"\"\n",
        "    選択可能な行動数を返す\n",
        "    \"\"\"\n",
        "    return len(self.action_list)\n",
        "\n",
        "  def observation_space(self):\n",
        "    \"\"\"\n",
        "    観測値のshapeを返す\n",
        "    \"\"\"\n",
        "    if self.obs_mode == \"num\":\n",
        "      return self.observation.shape[0]\n",
        "    elif self.obs_mode == \"image\":\n",
        "      return self.observation.shape\n",
        "    else:\n",
        "      return\n",
        "\n",
        "  def seed(self, n_seed):\n",
        "    \"\"\"\n",
        "    環境のseed値を設定\n",
        "    \"\"\"\n",
        "    self.fit_wave.set_seed(n_seed)\n",
        "    return\n",
        "\n",
        "  def random_action(self):\n",
        "    \"\"\"\n",
        "    ランダムに行動選択を実行\n",
        "    \"\"\"\n",
        "    return random.randrange(len(self.action_list))\n",
        "\n",
        "\n",
        "  def cal_reward(self, tgt_wave, fit_wave):\n",
        "    \"\"\"\n",
        "    報酬を算出 (ユークリッド距離を用いた波形の類似度)\n",
        "    \"\"\"\n",
        "    # fit_waveとtarget_waveのユークリッド距離を算出\n",
        "    dist = 0.0\n",
        "    for t in range(len(fit_wave)):\n",
        "      dist += (tgt_wave[t] - fit_wave[t])**2\n",
        "    dist = 1.0/(1.0+np.sqrt(dist))\n",
        "\n",
        "    # 類似度が0.95より大きい場合：2.0\n",
        "    # それ以外：類似度\n",
        "    if dist > 0.95:\n",
        "      reward = 2.0\n",
        "    else:\n",
        "      reward = dist\n",
        "\n",
        "    return reward\n",
        "\n",
        "  # step関数\n",
        "  def step(self, act):\n",
        "    \"\"\"\n",
        "    渡された行動をもとに環境を1step進める\n",
        "    観測情報,報酬値，終了フラグを返す\n",
        "    \"\"\"\n",
        "    self.time_step += 1\n",
        "\n",
        "    # 行動リストから行動を選択し各パラメータを更新\n",
        "    actions = self.action_list[act]\n",
        "    self.fit_wave.A += actions[0]\n",
        "    self.fit_wave.T += actions[1]\n",
        "    self.fit_wave.lam += actions[2]\n",
        "\n",
        "    # 各パラメータの上下限を超えていないか確認\n",
        "    self.fit_wave.checkparam()\n",
        "\n",
        "    # 現パラメータで波形を生成\n",
        "    self.out = self.fit_wave.update(self.x)\n",
        "\n",
        "    # 観測情報を生成 (数値モード：調整波形と目標波形の数値情報, 画像モード：各波形をグラフとして描画した画像)\n",
        "    if self.obs_mode == \"num\":\n",
        "      self.observation = np.array(np.concatenate([self.out, self.target_wave]))\n",
        "    elif self.obs_mode == \"image\":\n",
        "      self.observation, _ = self.render(mode=\"rgb_array\")\n",
        "\n",
        "    # 報酬値算出\n",
        "    reward = self.cal_reward(self.target_wave, self.out)\n",
        "\n",
        "    # 終了フラグ (step数がepisode_max_step以上となったらエピソード終了)\n",
        "    if self.time_step >= self.episode_max_step:\n",
        "      done = True\n",
        "    else:\n",
        "      done = False\n",
        "\n",
        "    return self.observation, reward, done\n",
        "\n",
        "  def reset(self):\n",
        "    \"\"\"\n",
        "    環境のリセット\n",
        "    \"\"\"\n",
        "    self.out = self.fit_wave.init_wave()\n",
        "    if self.obs_mode == \"num\":\n",
        "      obs = np.array(np.concatenate([self.out, self.target_wave]))\n",
        "    elif self.obs_mode == \"image\":\n",
        "      obs, _ = self.render(mode=\"rgb_array\")\n",
        "    self.time_step = 0\n",
        "    return obs\n",
        "\n",
        "  def render(self, mode=\"rendering\"):\n",
        "    \"\"\"\n",
        "    現環境の描画\n",
        "    modeが\"rendering\"の場合：表示のみを行う\n",
        "    modeが\"rgb_array\"の場合：描画結果のカラー画像とグレースケール画像を返す．表示は行わない\n",
        "    \"\"\"\n",
        "    x_len, y_len = 500, 500\n",
        "    canvas = (np.ones((y_len, x_len, 3)) * 255.0).astype(np.uint8)\n",
        "\n",
        "    target_pts, fit_pts = [], []\n",
        "    for idx in range(len(self.target_wave)):\n",
        "      x = int(idx/len(self.target_wave) * x_len)\n",
        "      target_y = int(((self.target_wave[idx]+self.fit_wave.range_A[1]) / (self.fit_wave.range_A[1]*2.0)) * y_len)\n",
        "      fit_y = int(((self.out[idx]+self.fit_wave.range_A[1]) / (self.fit_wave.range_A[1]*2.0)) * y_len)\n",
        "      target_pts.append((x, target_y))\n",
        "      fit_pts.append((x, fit_y))\n",
        "    target_pts, fit_pts = np.array(tuple(target_pts)), np.array(tuple(fit_pts))\n",
        "\n",
        "    cv2.polylines(canvas, [target_pts], False, (0, 0, 255), thickness=2)\n",
        "    cv2.polylines(canvas, [fit_pts], False, (255, 0, 0), thickness=2)\n",
        "\n",
        "    img_color = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB)\n",
        "    if mode == \"rgb_array\":\n",
        "      img_gray = cv2.cvtColor(canvas, cv2.COLOR_BGR2GRAY)\n",
        "      img_gray = cv2.resize(img_gray, dsize=(84,84))\n",
        "      img_gray = img_gray[np.newaxis,:,:]\n",
        "      return img_gray, img_color\n",
        "    else:\n",
        "      plt.imshow(img_color)\n",
        "      plt.show()\n",
        "      return"
      ],
      "metadata": {
        "id": "wGG-_brfUQ0E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 環境の定義\n",
        "env = ENV(obs_mode=\"num\")\n",
        "env.seed(seed)\n",
        "# 環境の初期化\n",
        "obs = env.reset()\n",
        "print('observation space:', env.observation_space())\n",
        "print('action space:', env.action_space())\n",
        "print('initial observation:', obs.shape)\n",
        "\n",
        "# 行動選択と選択した行動の入力\n",
        "action = 0\n",
        "obs, r, done = env.step(action)\n",
        "print('next observation:', obs.shape) # 環境から返ってくる次状態\n",
        "print('reward:', r)\n",
        "print('done:', done)\n",
        "env.render()"
      ],
      "metadata": {
        "id": "MKkYcuJ5USeP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mOgbuO2I1bfq"
      },
      "source": [
        "## ネットワーク構造\n",
        "ネットワークモデルを定義します．\n",
        "ここでは，環境からの観測値が数値情報の場合と，画像情報の場合，それぞれにおけるネットワークを定義しています．\n",
        "\n",
        "数値情報の場合は全結合層4層とし，画像情報の場合は畳み込み層3層と全結合層2層としています．\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 観測値が数値情報の場合\n",
        "class DQN_continuous(nn.Module):\n",
        "    def __init__(self, input_shape, n_actions):\n",
        "        super(DQN_continuous, self).__init__()\n",
        "        self.fc = nn.Sequential(\n",
        "            nn.Linear(input_shape, 128),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(128, 64),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(64, 32),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(32, n_actions)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.fc(x)"
      ],
      "metadata": {
        "id": "s3AAjD4UUruk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 観測値が画像情報の場合\n",
        "class DQN_image(nn.Module):\n",
        "    def __init__(self, input_shape, n_actions):\n",
        "        super(DQN_image, self).__init__()\n",
        "        self.conv = nn.Sequential(\n",
        "            nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(32, 64, kernel_size=4, stride=2),\n",
        "            nn.ReLU(),\n",
        "            nn.Conv2d(64, 64, kernel_size=3, stride=1),\n",
        "            nn.ReLU()\n",
        "        )\n",
        "\n",
        "        conv_out_size = self._get_conv_out(input_shape)\n",
        "        self.fc = nn.Sequential(\n",
        "            nn.Linear(conv_out_size, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, n_actions)\n",
        "        )\n",
        "\n",
        "    def _get_conv_out(self, shape):\n",
        "        o = self.conv(torch.zeros(1, *shape))\n",
        "        return int(np.prod(o.size()))\n",
        "\n",
        "    def forward(self, x):\n",
        "        conv_out = self.conv(x).view(x.size()[0], -1)\n",
        "        return self.fc(conv_out)"
      ],
      "metadata": {
        "id": "Xwf2nUm6Uszd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsX8hfGurmBt"
      },
      "source": [
        "## Deep Q-Networkにおける学習工夫の定義\n",
        "\n",
        "Deep Q-Networkでは，学習の促進と安定化の為に，いくつか工夫を施して学習を行っています．\n",
        "代表的な工夫として，Experience Replay, Target Q-Network, Reward Clippingと呼ばれる3つの工夫があります．\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vXY4bzSf3fRA"
      },
      "source": [
        "### Experience Replay"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FuV4WoqzFdFN"
      },
      "source": [
        "DQNでは，獲得した経験を直接使用し学習するのではなく，獲得した経験を一度Replay Bufferと呼ばれるBufferに格納し，学習する際にはBufferから経験をランダムに取得することにより学習を行います．\n",
        "これをExperience Replayと呼び，データの再利用を行うことで，データ効率を高め効率的な学習を行います．\n",
        "\n",
        "Bufferへは，「現在の状態，その時に選択された行動，行動によって遷移した状態（次状態），その時の報酬」の4種類の情報を1つの経験として蓄積します．\\\n",
        "まず，`Experience`という変数を定義します．\n",
        "ここでは，`state`, `action`, `reward`, `done`, `next_state`が1セットとなるようなデータ構造（辞書オブジェクト）を定義します．\\\n",
        "その後，Experience Bufferクラスを定義します．\n",
        "Experience Bufferクラスでは，経験を蓄積する`buffer`（`capacity = buffer`へ格納する経験の数）を定義します．\n",
        "append関数では，メモリへ経験を格納します．\n",
        "また，sample関数では，指定したバッチサイズ (`batch_size`) 分の経験をランダムに選択し，返す関数を定義します．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "Experience = collections.namedtuple('Experience', field_names=['state', 'action', 'reward', 'done', 'new_state'])\n",
        "\n",
        "class ExperienceBuffer:\n",
        "    def __init__(self, capacity):\n",
        "        self.buffer = collections.deque(maxlen=capacity)\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.buffer)\n",
        "\n",
        "    def append(self, experience):\n",
        "        self.buffer.append(experience)\n",
        "\n",
        "    def sample(self, batch_size):\n",
        "        indices = np.random.choice(len(self.buffer), batch_size, replace=False)\n",
        "        states, actions, rewards, dones, next_states = zip(*[self.buffer[idx] for idx in indices])\n",
        "        return np.array(states), np.array(actions), np.array(rewards, dtype=np.float32), \\\n",
        "               np.array(dones, dtype=np.uint8), np.array(next_states)"
      ],
      "metadata": {
        "id": "gmsFePrUU0m5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYQi9_LQ5UiF"
      },
      "source": [
        "### Target Q-Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uYcPPzEcFggm"
      },
      "source": [
        "DQNで誤差を算出する際，目標値として用いる行動価値関数と，現在の行動価値関数をそれぞれ別のネットワークの出力を用いて誤差を算出します．\\\n",
        "目標値として算出に用いるネットワークは，一定周期経過するまで重みを固定したネットワークとし，一定周期で現在のネットワークと同期しながら学習を行います．\n",
        "この工夫をTarget Q-Networkと呼び，学習の安定化を図ります．\n",
        "\n",
        "`target_net`を現在の`net`と同期する`sync_network`を定義します．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def sync_network():\n",
        "    tgt_net.load_state_dict(net.state_dict())"
      ],
      "metadata": {
        "id": "SFdGVHnXU15J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUL1qa38uol6"
      },
      "source": [
        "### Reward clipping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsxXQtZBFjyh"
      },
      "source": [
        "DQNでは，学習に用いる報酬値を，以下の通りにクリッピングします．\n",
        "- 報酬値が正：+1\n",
        "- 報酬値が0：0\n",
        "- 報酬値が負：-1\n",
        "\n",
        "これにより，学習における報酬の外れ値に対する過剰反応を防ぐことができます."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def clipreward(reward):\n",
        "  \"\"\"Bin reward to {+1, 0, -1} by its sign.\"\"\"\n",
        "  return np.sign(reward)"
      ],
      "metadata": {
        "id": "AwghMcz5VGVj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23ixNGvKdAB4"
      },
      "source": [
        "## エージェントの定義"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xStwDIjhFJii"
      },
      "source": [
        "エージェントが，環境に対して行動価値に沿った行動を選択し，環境から経験を取得，Experience Bufferへ獲得した経験を記録するようにします．\n",
        "\n",
        "エージェントの環境に対する動きのクラスを定義します．\\\n",
        "play_step関数は，環境に対し行動を決定する関数です．\n",
        "$\\epsilon$-greedy法を用いて，一定の割合でランダムに行動選択を行います．\n",
        "それ以外の場合は，ネットワークへ環境情報（画像）を入力し，行動を決定します．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "class Agent:\n",
        "    def __init__(self, env, exp_buffer):\n",
        "        self.env = env\n",
        "        self.exp_buffer = exp_buffer\n",
        "        self._reset()\n",
        "\n",
        "    def _reset(self):\n",
        "        self.state = env.reset()\n",
        "        self.total_reward = 0.0\n",
        "\n",
        "    def play_step(self, net=None, epsilon=0.0):\n",
        "        done_reward = None\n",
        "\n",
        "        if np.random.random() < epsilon:\n",
        "            action = env.random_action()\n",
        "        else:\n",
        "            state_a = np.array([self.state], copy=False)\n",
        "            state_v = torch.tensor(state_a).float()\n",
        "            if use_cuda:\n",
        "              state_v = state_v.cuda()\n",
        "            q_vals_v = net(state_v)\n",
        "            _, act_v = torch.max(q_vals_v, dim=1)\n",
        "            action = int(act_v.item())\n",
        "\n",
        "        # do step in the environment\n",
        "        new_state, reward, is_done = self.env.step(action)\n",
        "        # reward = clipreward(reward)\n",
        "        self.total_reward += reward\n",
        "\n",
        "        exp = Experience(self.state, action, reward, is_done, new_state)\n",
        "        self.exp_buffer.append(exp)\n",
        "        self.state = new_state\n",
        "        if is_done:\n",
        "            done_reward = self.total_reward\n",
        "            self._reset()\n",
        "        return done_reward\n",
        "\n",
        "    def play_random_step(self):\n",
        "        action = env.random_action()\n",
        "\n",
        "        # do step in the environment\n",
        "        new_state, reward, is_done = self.env.step(action)\n",
        "        # reward = clipreward(reward)\n",
        "\n",
        "        exp = Experience(self.state, action, reward, is_done, new_state)\n",
        "        self.exp_buffer.append(exp)\n",
        "        self.state = new_state\n",
        "        if is_done:\n",
        "            self._reset()\n",
        "        return"
      ],
      "metadata": {
        "id": "BQfJFMwsVLAC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJrH9M2xiPey"
      },
      "source": [
        "## TD誤差の計算\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8sQjdmdWFEmi"
      },
      "source": [
        "DQNでは，TD誤差と呼ばれる次状態の推定価値と，実際に選択した行動から得られる価値の差を用いて学習を行います．\n",
        "この時，次状態の推定価値は教師あり学習の教師と同じ役割を持ちます．\\\n",
        "DQNは，Q学習をもとにしているため，現在の行動価値関数を最適行動価値関数になるように更新を行っていきます．\n",
        "\n",
        "TD誤差の計算を行う関数を定義します．\\\n",
        "calc_loss関数では，`replay_buffer`からランダムに取得した`batch`分の経験をもとに，以下のLoss計算を行います．\n",
        "\n",
        "$$\n",
        "L_{\\theta}=\\frac{1}{2}(r+\\gamma \\max_{a'}Q_{\\theta_{i}}(s',a')-Q_{\\theta_{i}}(s,a))^{2}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def calc_loss(batch, net, tgt_net):\n",
        "    states, actions, rewards, dones, next_states = batch\n",
        "\n",
        "    states_v = torch.tensor(states).float()\n",
        "    next_states_v = torch.tensor(next_states).float()\n",
        "    actions_v = torch.tensor(actions)\n",
        "    rewards_v = torch.tensor(rewards)\n",
        "    done_mask = torch.BoolTensor(dones)\n",
        "    if use_cuda:\n",
        "      states_v = states_v.cuda()\n",
        "      next_states_v = next_states_v.cuda()\n",
        "      actions_v = actions_v.cuda()\n",
        "      rewards_v = rewards_v.cuda()\n",
        "      done_mask = done_mask.cuda()\n",
        "\n",
        "    state_action_values = net(states_v).gather(1, actions_v.unsqueeze(-1)).squeeze(-1)\n",
        "    next_state_values = tgt_net(next_states_v).max(1)[0]\n",
        "    next_state_values[done_mask] = 0.0\n",
        "    next_state_values = next_state_values.detach()\n",
        "\n",
        "    expected_state_action_values = next_state_values * GAMMA + rewards_v\n",
        "    return nn.MSELoss()(state_action_values, expected_state_action_values)"
      ],
      "metadata": {
        "id": "TqYMNwBTVNkb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KMkQ1vGVxTUQ"
      },
      "source": [
        "## 学習\n",
        "DQNを用いて学習を行います．\n",
        "\n",
        "学習にはそれなりの時間がかかります．\n",
        "そのため，今回は学習途中のモデルをロードし，途中から学習を行います．\n",
        "\n",
        "まず，環境を初期化し，経験をReplayBufferへ蓄積します．\n",
        "十分に蓄積された後，パラメータの更新を行います．\n",
        "また，`SNC_TARGET_FRAMES`で指定した回数ごとに，`target_net`のパラメータを`net`のパラメータと同じになるように同期を行います．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 学習途中モデルのダウンロード\n",
        "!wget -q http://mprg.cs.chubu.ac.jp/~itaya/share/mprg_colab/DQN_application/models.zip\n",
        "!unzip -q -o models.zip"
      ],
      "metadata": {
        "id": "4PaV5YmCVPdO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 学習時のハイパーパラメータ\n",
        "GAMMA = 0.99\n",
        "BATCH_SIZE = 32\n",
        "LEARNING_RATE = 1e-4\n",
        "SYNC_TARGET_FRAMES = 1000\n",
        "REPLAY_SIZE = 10000\n",
        "REPLAY_START_SIZE = 10000\n",
        "\n",
        "# ε-greedyの設定\n",
        "EPSILON_DECAY_LAST_FRAME = 300000\n",
        "EPSILON_START = 1.0\n",
        "EPSILON_FINAL = 0.02\n",
        "\n",
        "# 環境の観測値を数値情報とするか画像とするか　(num：数値情報， image:画像情報)\n",
        "INPUT = \"num\"\n",
        "\n",
        "#　学習済みモデルを使用するか (250,000ステップ学習したモデルをロード)\n",
        "LOAD_MODEL = \"models/wave-num-premodel.dat\"\n",
        "\n",
        "# 何フレーム(ステップ)学習するか\n",
        "num_frame = 500000"
      ],
      "metadata": {
        "id": "_oBHjyyEVnX9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "env = ENV(obs_mode=INPUT) # 環境の構築\n",
        "env.seed(seed)\n",
        "\n",
        "frame_idx = 0\n",
        "buffer = ExperienceBuffer(REPLAY_SIZE) # Bufferの構築\n",
        "agent = Agent(env, buffer) # エージェントの構築\n",
        "epsilon = EPSILON_START\n",
        "record_reward = []\n",
        "record_step = []\n",
        "\n",
        "Experience = collections.namedtuple('Experience', field_names=['state', 'action', 'reward', 'done', 'new_state'])\n",
        "if INPUT == \"num\":\n",
        "  net = DQN_continuous(env.observation_space(), env.action_space())\n",
        "  tgt_net = DQN_continuous(env.observation_space(), env.action_space())\n",
        "elif INPUT == \"image\":\n",
        "  net = DQN_image(env.observation_space(), env.action_space())\n",
        "  tgt_net = DQN_image(env.observation_space(), env.action_space())\n",
        "if use_cuda:\n",
        "  net = net.cuda()\n",
        "  tgt_net = tgt_net.cuda()\n",
        "\n",
        "# 最適化手法\n",
        "optimizer = optim.RMSprop(net.parameters(), lr=LEARNING_RATE)\n",
        "\n",
        "# 学習途中モデルのロード\n",
        "if LOAD_MODEL != \"\":\n",
        "  checkpoint = torch.load(LOAD_MODEL)\n",
        "  net.load_state_dict(checkpoint['model_state_dict'])\n",
        "  tgt_net.load_state_dict(checkpoint['model_state_dict'])\n",
        "  epsilon = checkpoint['epsilon']\n",
        "  frame_idx = checkpoint['frame_idx']\n",
        "  record_reward = checkpoint['record_reward']\n",
        "  record_step = checkpoint['record_step']\n",
        "  optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n",
        "  print(\"load model: {}\".format(LOAD_MODEL))\n",
        "else:\n",
        "  print(\"Not model loading\")\n",
        "\n",
        "\n",
        "total_rewards = []\n",
        "best_mean_reward = None\n",
        "\n",
        "# 空のBefferにある程度の経験を収集\n",
        "print(\"Collect experience....\")\n",
        "while len(buffer) < REPLAY_START_SIZE:\n",
        "  if LOAD_MODEL == \"\":\n",
        "    agent.play_random_step()\n",
        "  else:\n",
        "    agent.play_step(net, epsilon=0.0)\n",
        "agent._reset()\n",
        "print(\"Filled buffer {}\".format(len(buffer)))\n",
        "\n",
        "# エージェントの学習開始\n",
        "print(\"Leaning start\")\n",
        "ts = time.time()\n",
        "while frame_idx < num_frame:\n",
        "    frame_idx += 1\n",
        "    epsilon = max(EPSILON_FINAL, EPSILON_START - frame_idx / EPSILON_DECAY_LAST_FRAME)\n",
        "\n",
        "    reward = agent.play_step(net, epsilon) # 環境の1step\n",
        "    if reward is not None:\n",
        "        total_rewards.append(reward)\n",
        "        mean_reward = np.mean(total_rewards[-100:])\n",
        "        print(\"Frame {0}/{1}: episode {2}, reward {3:.3f}, mean reward {4:.3f}, eps {5:.2f}, time {6}\".format(frame_idx, num_frame, len(total_rewards), reward, mean_reward, epsilon, datetime.timedelta(seconds = time.time() - ts)))\n",
        "        record_reward.append(mean_reward)\n",
        "        record_step.append(frame_idx)\n",
        "\n",
        "        if best_mean_reward is None or best_mean_reward < mean_reward:\n",
        "            torch.save(\n",
        "                {'epsilon':epsilon, 'frame_idx':frame_idx, 'model_state_dict':net.state_dict(), 'optimizer_state_dict':optimizer.state_dict()},\n",
        "                \"wave-{}-best.dat\".format(INPUT))\n",
        "            if best_mean_reward is not None:\n",
        "                print(\"Best mean reward updated {0:.3f} -> {1:.3f}, model saved\".format(best_mean_reward, mean_reward))\n",
        "            best_mean_reward = mean_reward\n",
        "\n",
        "    # target networkの同期\n",
        "    if (frame_idx % SYNC_TARGET_FRAMES) == 0:\n",
        "        sync_network()\n",
        "\n",
        "    optimizer.zero_grad()\n",
        "    batch = buffer.sample(BATCH_SIZE)\n",
        "    loss_t = calc_loss(batch, net, tgt_net) # 誤差計算\n",
        "    loss_t.backward()\n",
        "    optimizer.step()"
      ],
      "metadata": {
        "id": "WTkYSJUhV4O7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSHnuplKQo14"
      },
      "source": [
        "## 学習時の平均スコアの推移\n",
        "横軸エピソード数，縦軸平均スコアとしたグラフを描画してみます．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig = plt.figure()\n",
        "plt.plot(record_step, record_reward, color=\"red\")\n",
        "plt.grid()\n",
        "plt.xlabel(\"step\")\n",
        "plt.ylabel(\"mean reward\")\n",
        "plt.savefig(\"./dqn_step_per_reward_{}.png\".format(INPUT))\n",
        "plt.show()\n",
        "plt.clf()\n",
        "plt.close()"
      ],
      "metadata": {
        "id": "gXEz4wesW8Cb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aDQWPHZcQxbF"
      },
      "source": [
        "グラフのダウンロード"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "files.download(\"./dqn_step_per_reward_{}.png\".format(INPUT))"
      ],
      "metadata": {
        "id": "xQBYRrJLW9YW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ksjONOyoxstC"
      },
      "source": [
        "## 評価\n",
        "学習したネットワーク（エージェント）を確認してみます．\n",
        "\n",
        "ここでは，`frames`に描画したフレームを順次格納します．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "frames = []\n",
        "for i in range(1):\n",
        "    state = env.reset()\n",
        "    done = False\n",
        "    t = 0\n",
        "    rewards = 0.0\n",
        "\n",
        "    while not done:\n",
        "        _, img = env.render(mode='rgb_array')\n",
        "        frames.append(img)\n",
        "        state_a = np.array([state], copy=False)\n",
        "        state_v = torch.tensor(state_a).float()\n",
        "        if use_cuda:\n",
        "          state_v = state_v.cuda()\n",
        "        q_vals_v = net(state_v)\n",
        "        _, act_v = torch.max(q_vals_v, dim=1)\n",
        "        action = int(act_v.item())\n",
        "        new_state, reward, done = env.step(action)\n",
        "        rewards += reward\n",
        "        state = new_state\n",
        "    print(\"reward: \", rewards)"
      ],
      "metadata": {
        "id": "QtK0_IvOW-sD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_5tJoQekxstF"
      },
      "source": [
        "## 描画\n",
        "\n",
        "maptlotlibを用いて，保存した動画フレームをアニメーションとして作成し，表示しています．"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.animation as animation\n",
        "from IPython.display import HTML\n",
        "\n",
        "\n",
        "# フレーム画像をアニメーションに変換\n",
        "def video_anime(imgs):\n",
        "    fig = plt.figure(figsize=(imgs[0].shape[1] / 72.0, imgs[0].shape[0] / 72.0), dpi = 72)  # 表示サイズ指定\n",
        "\n",
        "    mov = []\n",
        "    for i in range(len(imgs)):  # フレームを1枚づつmovにアペンド\n",
        "        img = plt.imshow(imgs[i], animated=True)\n",
        "        plt.axis('off')\n",
        "        mov.append([img])\n",
        "\n",
        "    # アニメーション作成\n",
        "    anime = animation.ArtistAnimation(fig, mov, interval=200, repeat_delay=0, repeat=False)\n",
        "    plt.close()\n",
        "    return anime\n",
        "\n",
        "# HTML5でアニメーションをインラインに動画表示\n",
        "HTML(video_anime(frames).to_html5_video())"
      ],
      "metadata": {
        "id": "vU1fRnddXESB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 観測情報が画像の場合 (学習済みモデルでの評価)"
      ],
      "metadata": {
        "id": "GgTz8gZEJ8Kv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "環境とネットワークの構築"
      ],
      "metadata": {
        "id": "v6a-CwfKLdbP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "INPUT = \"image\"\n",
        "LOAD_MODEL = \"models/wave-image-trainmodel.dat\"\n",
        "\n",
        "env = ENV(obs_mode=INPUT)\n",
        "env.seed(seed)\n",
        "\n",
        "net = DQN_image(env.observation_space(), env.action_space())\n",
        "if use_cuda:\n",
        "  net = net.cuda()\n",
        "\n",
        "if LOAD_MODEL != \"\":\n",
        "  checkpoint = torch.load(LOAD_MODEL)\n",
        "  net.load_state_dict(checkpoint['model_state_dict'])\n",
        "  record_reward = checkpoint['record_reward']\n",
        "  record_step = checkpoint['record_step']\n",
        "  print(\"load model: {}\".format(LOAD_MODEL))\n",
        "else:\n",
        "  print(\"Not model loading\")"
      ],
      "metadata": {
        "id": "50_hqqShXM3i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "平均スコアの推移"
      ],
      "metadata": {
        "id": "WosRx9jCL1EA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig = plt.figure()\n",
        "plt.plot(record_step, record_reward, color=\"red\")\n",
        "plt.grid()\n",
        "plt.xlabel(\"step\")\n",
        "plt.ylabel(\"mean reward\")\n",
        "plt.savefig(\"./dqn_step_per_reward_{}.png\".format(INPUT))\n",
        "plt.show()\n",
        "plt.clf()\n",
        "plt.close()"
      ],
      "metadata": {
        "id": "3jfvcvEcXOOX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "グラフのダウンロード"
      ],
      "metadata": {
        "id": "WoahuBVWhPdC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "files.download(\"./dqn_step_per_reward_{}.png\".format(INPUT))"
      ],
      "metadata": {
        "id": "gIplDmBCXQGl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "評価と描画"
      ],
      "metadata": {
        "id": "L93-T2B7L_ci"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "frames = []\n",
        "for i in range(1):\n",
        "    state = env.reset()\n",
        "    done = False\n",
        "    t = 0\n",
        "    rewards = 0.0\n",
        "\n",
        "    while not done:\n",
        "        _, img = env.render(mode='rgb_array')\n",
        "        frames.append(img)\n",
        "        state_a = np.array([state], copy=False)\n",
        "        state_v = torch.tensor(state_a).float()\n",
        "        if use_cuda:\n",
        "          state_v = state_v.cuda()\n",
        "        q_vals_v = net(state_v)\n",
        "        _, act_v = torch.max(q_vals_v, dim=1)\n",
        "        action = int(act_v.item())\n",
        "        new_state, reward, done = env.step(action)\n",
        "        rewards += reward\n",
        "        state = new_state\n",
        "    print(rewards)\n",
        "\n",
        "\n",
        "# 実行結果の表示\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.animation as animation\n",
        "from IPython.display import HTML\n",
        "\n",
        "def video_anime(imgs):\n",
        "    fig = plt.figure(figsize=(imgs[0].shape[1] / 72.0, imgs[0].shape[0] / 72.0), dpi = 72)\n",
        "    mov = []\n",
        "    for i in range(len(imgs)):\n",
        "        img = plt.imshow(imgs[i], animated=True)\n",
        "        plt.axis('off')\n",
        "        mov.append([img])\n",
        "    anime = animation.ArtistAnimation(fig, mov, interval=200, repeat_delay=0, repeat=False)\n",
        "    plt.close()\n",
        "    return anime\n",
        "\n",
        "# HTML5でアニメーションをインラインに動画表示\n",
        "HTML(video_anime(frames).to_html5_video())"
      ],
      "metadata": {
        "id": "Cp2JO9_AXRWV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kZDh4S6m2hK0"
      },
      "source": [
        "# 課題\n",
        "\n",
        "1. 環境の設定（行動リスト，報酬設計など）を変更し学習の様子を確認してみましょう\n",
        "\n",
        "  **ヒント**\n",
        "  *   各パラメータの上下限設定（WAVEクラスinit関数内）\n",
        "  *   波形の初期化方法（WAVEクラスinit_wave関数内）\n",
        "  *   行動の設定（ENVクラスinit関数内）\n",
        "  *   目標波形の設定（ENVクラスinit関数内）\n",
        "  *   報酬設計（ENVクラスcal_reward関数内）\n",
        "  *   観測情報の設計（ENVクラスstep関数内）\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}