2020.01.31 RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RNN (Recurrent Neural Network)\n",
    "\n",
    "- http://blog.naver.com/PostView.nhn?blogId=magnking&logNo=221323257045&redirect=Dlog&widgetTypeCall=true&directAccess=false\n",
    "\n",
    "[simple RNN]\n",
    "-  cnn(특징추출) - ffnn\n",
    "- rnn도 마찬가지로 ffnn해줘야함_ 모든 신경망의 기본은 ffnn\n",
    "\n",
    "- 입력 데이터 받고 => 가중치 => output : 이 통째로 cell이라한다\n",
    "\n",
    "- 여러개의 cell을 묶어서 output과 연결시킨다 \n",
    "- 이전데이터 가중치의 영향을 고려한다  > 앞단의 데이터가 모두 끝나야한다는 것 . \n",
    "    - 그래서 latent time(지연시간)  발생 \n",
    "    - 각 cell의 output이 연속족으로 최종 output까지 영향을 미쳐야하는데\n",
    "    - 중간에 영향력을 잃거나 중복적으로 영향을 미치는 경우가 있다\n",
    "    \n",
    "[Long Short Time Memory]: 장기기억 _ https://wikidocs.net/22888\n",
    "- 그래서 나온게 LSTM : 처음의 영향력을 잃지않기위해 사용\n",
    "    - STATE를 2개 준다 : CONTROL STATE , HIDDEN STATE\n",
    "    - HIDDEN STATE 가 OUTPUT: 지속적으로 다음요소에 영향을 미칠것을 결정?\n",
    "    - 하나는 다음 회로로 나가고 하나는 output으로 나감?\n",
    "    - \n",
    "- gate 3개 : forget input output\n",
    "    - forget : 앞단에서 흘러오는 ct-1 를 어떻게 제어할것인가를 결정( 망각 or keep)\n",
    "        - 현재 들어오는 데이터와 이전 state에서 들어오는 ht-1 을 고려해서 \n",
    "    - input : control선에 어떻게 변화해야 영향력을 유지할 수 있을지 제어 \n",
    "    - output : hidden\n",
    "       \n",
    "- 출력을 위해서 가중치 4개가 필요 (forget input output 과 CELL 안에서 영향을 끼치는한개(TANH)를 다 계산)\n",
    "    - 4개의 메모리공간 필요\n",
    "    - 계속 영향을 미치기는 하지만 가중치가 다름    \n",
    "    - 가중치 사이즈 계산할 떄 4배해줌\n",
    "    \n",
    "    \n",
    "[ GRU ] : LSTM이 너무 복잡해서 나온것\n",
    "\n",
    "- FORGET과 INPUT 을 하나로 묶음 :회로를 간략화 , 연산이 줄얻을음\n",
    "- 순서가 있는 데이터를 다를 수 있다 .-> 시계열 DATA : SEQUENTIAL DATA\n",
    "    - 나는 /학교에/ 간다 \n",
    "   \n",
    " [CAPTIONING]   첫번째 셀(iNput)에 이미지를 넣어주면 이미지를 captioning하는 것\n",
    " \n",
    "- ONE TO ONE :\n",
    "- ONE TO MANY : captioning 할 때 사용 \n",
    "- MANY TO MANY :번역할 때 사용 \n",
    "- MANY TO MANY :동영상 FRAME에 들어있는 OBJECT를 DETECTION 할 때\n",
    "\n",
    "[rnn 종류]_ 휴대폰 사진찍어놈\n",
    "- 홋 RNN: Simple RNN \n",
    "- Bi directional RNN: 양쪽방향으로 뽑음\n",
    "- MULTI RNN:  CELL을 여러개 쌓아서 다층으로_ 다층일수록 정확\n",
    "\n",
    "\n",
    "#input character => input layer => hidden layer => ouput layer => target character\n",
    "\n",
    "\n",
    "-CELL\n",
    "MULTI LSTM CELL\n",
    "\n",
    "- STATIC RNN :셀 수가 고정\n",
    "- DYNAMIC RNN : 시간적으로 계산하면되어서 (쭉 늘어놀 필요가 없다 ) 하나계산하고 하나계산하고 쮺쮺\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE85F488>>: AssertionError: Bad argument number for Name: 3, expecting 4\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "\n",
    "# 그라프 리셋할때 이거 써주기\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "n_inputs = 3 # 입력 데이터 사이즈\n",
    "n_neurons = 5  # 셀의 가중치 사이즈\n",
    "\n",
    "# 리셋 - 초기화(변수 생성)\n",
    "reset_graph()\n",
    "   # 아니면 reset_default_graph() 이걸 해주던가\n",
    "\n",
    "\n",
    "X0 = tf.placeholder(tf.float32, [None, n_inputs])  # 4x3\n",
    "X1 = tf.placeholder(tf.float32, [None, n_inputs])  # 4x3\n",
    "\n",
    "# [0,1,2] 사이즈가 하나의 셀로 입력  BasicRNNCell (기본셀)\n",
    "# FFNN (feed forward neural network)\n",
    "# 가중치 사이즈 => 특성을 찾아내는 것  num_units = n_neurons\n",
    "basic_cell = tf.contrib.rnn.BasicRNNCell(num_units = n_neurons)\n",
    "\n",
    "# static_rnn : rnn network : 4개의 셀이 연결되면서 메모리 확보\n",
    "output_seqs, states = tf.contrib.rnn.static_rnn(basic_cell, \n",
    "                                                [X0, X1], dtype = tf.float32)  # 2x4x3 : 배치사이즈(순차)x 셀수 x뉴런수\n",
    "\n",
    "# 출력, 다음셀로 전달되는 값 (두개를 전달)    # states값은 마지막에 전달되어지는 것   (수평을 셀을 연결)\n",
    "Y0, Y1 = output_seqs\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "X0_batch = np.array([[0,1,2], [3,4,5],[6,7,8],[9,0,1]])\n",
    "X1_batch = np.array([[9,8,7],[0,0,0],[6,5,4],[3,2,1]])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "처음 데이터 특성 :  \n",
      " [[ 0.30741334 -0.32884315 -0.6542847  -0.9385059   0.52089024]\n",
      " [ 0.99122757 -0.9542541  -0.7518079  -0.9995208   0.9820235 ]\n",
      " [ 0.9999268  -0.99783254 -0.8247353  -0.9999963   0.99947774]\n",
      " [ 0.996771   -0.68750614  0.8419969   0.9303911   0.8120684 ]] 차수: (4, 5)\n",
      "\n",
      "두번째 데이터 특성 : \n",
      " [[ 0.99998885 -0.99976057 -0.06679279 -0.9999803   0.99982214]\n",
      " [-0.65249425 -0.51520866 -0.37968954 -0.5922594  -0.08968391]\n",
      " [ 0.998624   -0.99715203 -0.03308626 -0.9991566   0.9932902 ]\n",
      " [ 0.99681675 -0.9598194   0.39660627 -0.8307606   0.79671973]] 차수: (4, 5)\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    Y0_val, Y1_val = sess.run([Y0,Y1], feed_dict = {X0:X0_batch, X1:X1_batch})\n",
    "    \n",
    "print(\"처음 데이터 특성 : \", \"\\n\",Y0_val, \"차수:\",Y0_val.shape)\n",
    "print()\n",
    "print(\"두번째 데이터 특성 :\", \"\\n\"  , Y1_val, \"차수:\",Y1_val.shape)\n",
    "\n",
    "# 4x3 => 4x5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "셀이 가지고 있는 가중치 사이즈는?    \n",
    "3x5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF0F4888>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF0F4888>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF0F4888>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF0F4888>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E7CF151B88>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E7CF151B88>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E7CF151B88>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E7CF151B88>>: AssertionError: Bad argument number for Name: 3, expecting 4\n"
     ]
    }
   ],
   "source": [
    "reset_graph()\n",
    "\n",
    "# 이미지 784 (28x28)\n",
    "n_steps = 28   # 노드 수 \n",
    "n_inputs = 28  # 노드당 인풋 사이즈\n",
    "n_neurons = 150  # 뉴런 출력\n",
    "n_outputs =10   # 확률 사이즈\n",
    "learning_rate = 0.001\n",
    "batch_size = 150\n",
    "\n",
    "# 3차원으로                     배치, 셀수, 셀당 데이터입력수\n",
    "X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y = tf.placeholder(tf.int32, [None])\n",
    "# FFNN\n",
    "basic_cell = tf.contrib.rnn.BasicRNNCell(num_units = n_neurons)  # 셀당 150개\n",
    "\n",
    "# 셀 28개\n",
    "# state 는 마지막 셀에 수평으로 전달되는 값\n",
    "# state 는 마지막 셀의 output과 동일\n",
    "# 28개의 셀이 있는데 마지막 1개의 output을 사용 => many to one\n",
    "# 감정분류 <- 분류기\n",
    "# state 의 차수 : 150x150\n",
    "# output의 차수 : latent time 지연시간을 통해서 계산된 셀의 모든 값의 결합 출력 150x28x150  (? x 배치 x 뉴런수)\n",
    "\n",
    "# outputs 은 나와서 다시 들어가는 출력값(저장) // states는 맨 마지막에 결과(출력)\n",
    "outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype = tf.float32)\n",
    "\n",
    "\n",
    "# 150개의 특징중에 10개만 추출\n",
    "# 입력차수, 출력 차수만 지정하면 자동으로 바이어스를 생성\n",
    "# 가중치 공간을 확보해서\n",
    "# 150x150 => 150x10\n",
    "# dense의 가중치 사이즈 = 150x10\n",
    "logits = tf.layers.dense(states, n_outputs)\n",
    "\n",
    "xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels = y, logits = logits) # 분류를 위한 미분이 가능한 식\n",
    "\n",
    "loss = tf.reduce_mean(xentropy) # 배치사이즈 => 평균을 통해서 loss (지역해 방지)\n",
    "\n",
    "# Adam = momentum + propgrad (경사하강법)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate)\n",
    "\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "correct = tf.nn.in_top_k(logits, y,1)  # 가장 큰값\n",
    "accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# 원래 이미지 모양\n",
    "mnist = input_data.read_data_sets(\"/tmp/data\")\n",
    "\n",
    "X_test = mnist.test.images.reshape((-1,n_steps, n_inputs))\n",
    "y_test = mnist.test.labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train accuracy:  0.93333334 Test accuracy: 0.9311\n",
      "1 Train accuracy:  0.96666664 Test accuracy: 0.9522\n",
      "2 Train accuracy:  0.97333336 Test accuracy: 0.9584\n",
      "3 Train accuracy:  0.96666664 Test accuracy: 0.9638\n",
      "4 Train accuracy:  0.97333336 Test accuracy: 0.9662\n",
      "5 Train accuracy:  0.96666664 Test accuracy: 0.9686\n",
      "6 Train accuracy:  0.96666664 Test accuracy: 0.9678\n",
      "7 Train accuracy:  0.98 Test accuracy: 0.9728\n",
      "8 Train accuracy:  0.93333334 Test accuracy: 0.9651\n",
      "9 Train accuracy:  0.96666664 Test accuracy: 0.967\n",
      "10 Train accuracy:  0.9866667 Test accuracy: 0.9754\n",
      "11 Train accuracy:  0.98 Test accuracy: 0.9668\n",
      "12 Train accuracy:  0.98 Test accuracy: 0.9711\n",
      "13 Train accuracy:  0.98 Test accuracy: 0.9766\n",
      "14 Train accuracy:  0.99333334 Test accuracy: 0.976\n",
      "15 Train accuracy:  0.98 Test accuracy: 0.9731\n",
      "16 Train accuracy:  0.9866667 Test accuracy: 0.975\n",
      "17 Train accuracy:  0.98 Test accuracy: 0.97\n",
      "18 Train accuracy:  0.98 Test accuracy: 0.9786\n",
      "19 Train accuracy:  0.96 Test accuracy: 0.9737\n",
      "20 Train accuracy:  0.9866667 Test accuracy: 0.9788\n",
      "21 Train accuracy:  0.98 Test accuracy: 0.9777\n",
      "22 Train accuracy:  0.97333336 Test accuracy: 0.9731\n",
      "23 Train accuracy:  0.99333334 Test accuracy: 0.9789\n",
      "24 Train accuracy:  0.99333334 Test accuracy: 0.9775\n",
      "25 Train accuracy:  1.0 Test accuracy: 0.9804\n",
      "26 Train accuracy:  0.98 Test accuracy: 0.9806\n",
      "27 Train accuracy:  0.9866667 Test accuracy: 0.9774\n",
      "28 Train accuracy:  0.9866667 Test accuracy: 0.9776\n",
      "29 Train accuracy:  0.99333334 Test accuracy: 0.9755\n",
      "30 Train accuracy:  0.98 Test accuracy: 0.9795\n",
      "31 Train accuracy:  1.0 Test accuracy: 0.9718\n",
      "32 Train accuracy:  0.98 Test accuracy: 0.9755\n",
      "33 Train accuracy:  0.9866667 Test accuracy: 0.9781\n",
      "34 Train accuracy:  0.9866667 Test accuracy: 0.9706\n",
      "35 Train accuracy:  1.0 Test accuracy: 0.9747\n",
      "36 Train accuracy:  1.0 Test accuracy: 0.977\n",
      "37 Train accuracy:  1.0 Test accuracy: 0.9749\n",
      "38 Train accuracy:  0.98 Test accuracy: 0.9789\n",
      "39 Train accuracy:  1.0 Test accuracy: 0.9784\n",
      "40 Train accuracy:  0.99333334 Test accuracy: 0.9809\n",
      "41 Train accuracy:  0.99333334 Test accuracy: 0.9795\n",
      "42 Train accuracy:  0.9866667 Test accuracy: 0.9743\n",
      "43 Train accuracy:  1.0 Test accuracy: 0.9782\n",
      "44 Train accuracy:  0.97333336 Test accuracy: 0.9792\n",
      "45 Train accuracy:  0.9866667 Test accuracy: 0.9798\n",
      "46 Train accuracy:  0.99333334 Test accuracy: 0.9817\n",
      "47 Train accuracy:  0.99333334 Test accuracy: 0.9762\n",
      "48 Train accuracy:  0.9866667 Test accuracy: 0.9798\n",
      "49 Train accuracy:  0.9866667 Test accuracy: 0.9754\n",
      "50 Train accuracy:  0.98 Test accuracy: 0.9698\n",
      "51 Train accuracy:  0.9866667 Test accuracy: 0.9717\n",
      "52 Train accuracy:  1.0 Test accuracy: 0.9792\n",
      "53 Train accuracy:  1.0 Test accuracy: 0.9806\n",
      "54 Train accuracy:  0.9866667 Test accuracy: 0.9775\n",
      "55 Train accuracy:  0.98 Test accuracy: 0.9706\n",
      "56 Train accuracy:  0.9866667 Test accuracy: 0.981\n",
      "57 Train accuracy:  1.0 Test accuracy: 0.9801\n",
      "58 Train accuracy:  0.9866667 Test accuracy: 0.9824\n",
      "59 Train accuracy:  0.9866667 Test accuracy: 0.9788\n",
      "60 Train accuracy:  1.0 Test accuracy: 0.9795\n",
      "61 Train accuracy:  1.0 Test accuracy: 0.9785\n",
      "62 Train accuracy:  1.0 Test accuracy: 0.9766\n",
      "63 Train accuracy:  0.98 Test accuracy: 0.9629\n",
      "64 Train accuracy:  0.99333334 Test accuracy: 0.9777\n",
      "65 Train accuracy:  1.0 Test accuracy: 0.9784\n",
      "66 Train accuracy:  1.0 Test accuracy: 0.9798\n",
      "67 Train accuracy:  0.9866667 Test accuracy: 0.977\n",
      "68 Train accuracy:  0.9866667 Test accuracy: 0.9762\n",
      "69 Train accuracy:  1.0 Test accuracy: 0.9787\n",
      "70 Train accuracy:  0.9866667 Test accuracy: 0.9793\n",
      "71 Train accuracy:  1.0 Test accuracy: 0.9778\n",
      "72 Train accuracy:  1.0 Test accuracy: 0.9797\n",
      "73 Train accuracy:  1.0 Test accuracy: 0.9815\n",
      "74 Train accuracy:  0.9866667 Test accuracy: 0.9801\n",
      "75 Train accuracy:  1.0 Test accuracy: 0.975\n",
      "76 Train accuracy:  0.9866667 Test accuracy: 0.9693\n",
      "77 Train accuracy:  1.0 Test accuracy: 0.977\n",
      "78 Train accuracy:  1.0 Test accuracy: 0.9808\n",
      "79 Train accuracy:  0.99333334 Test accuracy: 0.9811\n",
      "80 Train accuracy:  0.98 Test accuracy: 0.9807\n",
      "81 Train accuracy:  0.99333334 Test accuracy: 0.9804\n",
      "82 Train accuracy:  0.9866667 Test accuracy: 0.9784\n",
      "83 Train accuracy:  0.99333334 Test accuracy: 0.9792\n",
      "84 Train accuracy:  1.0 Test accuracy: 0.9773\n",
      "85 Train accuracy:  0.99333334 Test accuracy: 0.9815\n",
      "86 Train accuracy:  1.0 Test accuracy: 0.9805\n",
      "87 Train accuracy:  0.9866667 Test accuracy: 0.9794\n",
      "88 Train accuracy:  0.99333334 Test accuracy: 0.9798\n",
      "89 Train accuracy:  1.0 Test accuracy: 0.9805\n",
      "90 Train accuracy:  0.99333334 Test accuracy: 0.9787\n",
      "91 Train accuracy:  0.97333336 Test accuracy: 0.9729\n",
      "92 Train accuracy:  0.9866667 Test accuracy: 0.978\n",
      "93 Train accuracy:  1.0 Test accuracy: 0.9774\n",
      "94 Train accuracy:  1.0 Test accuracy: 0.9794\n",
      "95 Train accuracy:  0.98 Test accuracy: 0.9751\n",
      "96 Train accuracy:  1.0 Test accuracy: 0.9798\n",
      "97 Train accuracy:  1.0 Test accuracy: 0.9766\n",
      "98 Train accuracy:  0.98 Test accuracy: 0.9778\n",
      "99 Train accuracy:  1.0 Test accuracy: 0.9737\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 100\n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        for iteration in range(  # 60000/150  (training data)\n",
    "            mnist.train.num_examples // batch_size):  \n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            \n",
    "            # 이미지 사이즈로 생성\n",
    "            X_batch = X_batch.reshape((-1, n_steps, n_inputs))\n",
    "            sess.run(training_op, feed_dict = {X:X_batch, y:y_batch})\n",
    "        acc_train = accuracy.eval(feed_dict = {X:X_batch, y:y_batch})\n",
    "        acc_test = accuracy.eval(feed_dict = {X:X_test, y:y_test})\n",
    "        print(epoch, \"Train accuracy: \", acc_train, \"Test accuracy:\", acc_test)   \n",
    "        \n",
    "# 여기서 150개의 배치(특징으로) 10개의 labels(출력값) 을 추출하는게 목표. 150x10 곱해서 10개 뽑아내기\n",
    "\n",
    "# rnn은 시간, 순차성을 가진 특징추출\n",
    "# cnn 은 주변을 고려해서 특징추출"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi RNN Cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# BasicRNNCell : 기본데이터가 입력\n",
    "# multi rnn cell : 수직으로 레이어 구성 ( 수직으로 셀이 구성 )\n",
    "# dynamic_rnn : 모델\n",
    "\n",
    "\n",
    "reset_graph()\n",
    "n_steps = 28\n",
    "n_inputs = 28\n",
    "n_outputs = 10\n",
    "learning_rate = 0.001\n",
    "batch_size = 150\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y = tf.placeholder(tf.int32, [None])\n",
    "\n",
    "n_neurons = 100\n",
    "n_layers = 3  # 3개의 멀티 레이어 :3개의 셀을 생성\n",
    "\n",
    "layers = [tf.contrib.rnn.BasicRNNCell(num_units = n_neurons, activation = tf.nn.relu)\n",
    "         for layer in range(n_layers)]\n",
    "\n",
    "# 3개의 셀을 조합해서 MultiRNNCell (수직으로 쌓는다)\n",
    "multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dynamic_rnn: 셀로 입력되는 데이터의 개수에 맞추어서 셀을 구성  \n",
    "\n",
    "예시)   \n",
    "나는 학교에 간다.  <- (2자  3자  2자) : 사이즈가 다르다.     \n",
    "so, static_rnn 은 동일한 사이즈로 맞춘다 : 큰것 기준, 작은걸 padding해서      \n",
    "    입력사이즈 변동 = 가중치 조절 => 나가는 특징은 일치 (neural 수가 같다.)\n",
    "\n",
    "\n",
    "text를 숫자화   \n",
    "\n",
    "- embedding   \n",
    "- one-hot-encoding   \n",
    "- tfidfcount?   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7CF185088>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7CF185088>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7CF185088>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7CF185088>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9C26C8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9C26C8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9C26C8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9C26C8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE7B0208>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE7B0208>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE7B0208>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CE7B0208>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9CF288>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9CF288>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9CF288>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7CF9CF288>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING:tensorflow:Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E79575F908>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E79575F908>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E79575F908>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Dense.call of <tensorflow.python.layers.core.Dense object at 0x000001E79575F908>>: AssertionError: Bad argument number for Name: 3, expecting 4\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "Variable rnn/multi_rnn_cell/cell_0/basic_rnn_cell/kernel/Adam/ already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2005, in __init__\n    self._traceback = tf_stack.extract_stack()\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 3616, in create_op\n    op_def=op_def)\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py\", line 507, in new_func\n    return func(*args, **kwargs)\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 788, in _apply_op_helper\n    op_def=op_def)\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_state_ops.py\", line 1608, in variable_v2\n    shared_name=shared_name, name=name)\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-34-d0f6eef7fb3c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     19\u001b[0m \u001b[0moptimizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mAdamOptimizer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlearning_rate\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlearning_rate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     20\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 21\u001b[1;33m \u001b[0mtraining_op\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mminimize\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     22\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     23\u001b[0m \u001b[0mcorrect\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0min_top_k\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# 가장 큰값\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\optimizer.py\u001b[0m in \u001b[0;36mminimize\u001b[1;34m(self, loss, global_step, var_list, gate_gradients, aggregation_method, colocate_gradients_with_ops, name, grad_loss)\u001b[0m\n\u001b[0;32m    411\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    412\u001b[0m     return self.apply_gradients(grads_and_vars, global_step=global_step,\n\u001b[1;32m--> 413\u001b[1;33m                                 name=name)\n\u001b[0m\u001b[0;32m    414\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    415\u001b[0m   def compute_gradients(self, loss, var_list=None,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\optimizer.py\u001b[0m in \u001b[0;36mapply_gradients\u001b[1;34m(self, grads_and_vars, global_step, name)\u001b[0m\n\u001b[0;32m    595\u001b[0m                        ([str(v) for _, v, _ in converted_grads_and_vars],))\n\u001b[0;32m    596\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minit_scope\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 597\u001b[1;33m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_create_slots\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvar_list\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    598\u001b[0m     \u001b[0mupdate_ops\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    599\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mname_scope\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\adam.py\u001b[0m in \u001b[0;36m_create_slots\u001b[1;34m(self, var_list)\u001b[0m\n\u001b[0;32m    129\u001b[0m     \u001b[1;31m# Create slots for the first and second moments.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    130\u001b[0m     \u001b[1;32mfor\u001b[0m \u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mvar_list\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 131\u001b[1;33m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_zeros_slot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"m\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    132\u001b[0m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_zeros_slot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"v\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    133\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\optimizer.py\u001b[0m in \u001b[0;36m_zeros_slot\u001b[1;34m(self, var, slot_name, op_name)\u001b[0m\n\u001b[0;32m   1153\u001b[0m     \u001b[0mnamed_slots\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_slot_dict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mslot_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1154\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0m_var_key\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mnamed_slots\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1155\u001b[1;33m       \u001b[0mnew_slot_variable\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mslot_creator\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcreate_zeros_slot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mop_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1156\u001b[0m       self._restore_slot_variable(\n\u001b[0;32m   1157\u001b[0m           \u001b[0mslot_name\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mslot_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvariable\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mvar\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\slot_creator.py\u001b[0m in \u001b[0;36mcreate_zeros_slot\u001b[1;34m(primary, name, dtype, colocate_with_primary)\u001b[0m\n\u001b[0;32m    188\u001b[0m     return create_slot_with_initializer(\n\u001b[0;32m    189\u001b[0m         \u001b[0mprimary\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minitializer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mslot_shape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 190\u001b[1;33m         colocate_with_primary=colocate_with_primary)\n\u001b[0m\u001b[0;32m    191\u001b[0m   \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    192\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprimary\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvariables\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mVariable\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\slot_creator.py\u001b[0m in \u001b[0;36mcreate_slot_with_initializer\u001b[1;34m(primary, initializer, shape, dtype, name, colocate_with_primary)\u001b[0m\n\u001b[0;32m    162\u001b[0m       \u001b[1;32mwith\u001b[0m \u001b[0mdistribution_strategy\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextended\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcolocate_vars_with\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprimary\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    163\u001b[0m         return _create_slot_var(primary, initializer, \"\", validate_shape, shape,\n\u001b[1;32m--> 164\u001b[1;33m                                 dtype)\n\u001b[0m\u001b[0;32m    165\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    166\u001b[0m       return _create_slot_var(primary, initializer, \"\", validate_shape, shape,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\slot_creator.py\u001b[0m in \u001b[0;36m_create_slot_var\u001b[1;34m(primary, val, scope, validate_shape, shape, dtype)\u001b[0m\n\u001b[0;32m     72\u001b[0m       \u001b[0mshape\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     73\u001b[0m       \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 74\u001b[1;33m       validate_shape=validate_shape)\n\u001b[0m\u001b[0;32m     75\u001b[0m   \u001b[0mvariable_scope\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_variable_scope\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mset_partitioner\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcurrent_partitioner\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     76\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\variable_scope.py\u001b[0m in \u001b[0;36mget_variable\u001b[1;34m(name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint, synchronization, aggregation)\u001b[0m\n\u001b[0;32m   1494\u001b[0m       \u001b[0mconstraint\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mconstraint\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1495\u001b[0m       \u001b[0msynchronization\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msynchronization\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1496\u001b[1;33m       aggregation=aggregation)\n\u001b[0m\u001b[0;32m   1497\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1498\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\variable_scope.py\u001b[0m in \u001b[0;36mget_variable\u001b[1;34m(self, var_store, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint, synchronization, aggregation)\u001b[0m\n\u001b[0;32m   1237\u001b[0m           \u001b[0mconstraint\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mconstraint\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1238\u001b[0m           \u001b[0msynchronization\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msynchronization\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1239\u001b[1;33m           aggregation=aggregation)\n\u001b[0m\u001b[0;32m   1240\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1241\u001b[0m   def _get_partitioned_variable(self,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\variable_scope.py\u001b[0m in \u001b[0;36mget_variable\u001b[1;34m(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint, synchronization, aggregation)\u001b[0m\n\u001b[0;32m    560\u001b[0m           \u001b[0mconstraint\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mconstraint\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    561\u001b[0m           \u001b[0msynchronization\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msynchronization\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 562\u001b[1;33m           aggregation=aggregation)\n\u001b[0m\u001b[0;32m    563\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    564\u001b[0m   def _get_partitioned_variable(self,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\variable_scope.py\u001b[0m in \u001b[0;36m_true_getter\u001b[1;34m(name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, constraint, synchronization, aggregation)\u001b[0m\n\u001b[0;32m    512\u001b[0m           \u001b[0mconstraint\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mconstraint\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    513\u001b[0m           \u001b[0msynchronization\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msynchronization\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 514\u001b[1;33m           aggregation=aggregation)\n\u001b[0m\u001b[0;32m    515\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    516\u001b[0m     synchronization, aggregation, trainable = (\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\variable_scope.py\u001b[0m in \u001b[0;36m_get_single_variable\u001b[1;34m(self, name, shape, dtype, initializer, regularizer, partition_info, reuse, trainable, collections, caching_device, validate_shape, use_resource, constraint, synchronization, aggregation)\u001b[0m\n\u001b[0;32m    862\u001b[0m         \u001b[0mtb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mx\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mtb\u001b[0m \u001b[1;32mif\u001b[0m \u001b[1;34m\"tensorflow/python\"\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    863\u001b[0m         raise ValueError(\"%s Originally defined at:\\n\\n%s\" %\n\u001b[1;32m--> 864\u001b[1;33m                          (err_msg, \"\".join(traceback.format_list(tb))))\n\u001b[0m\u001b[0;32m    865\u001b[0m       \u001b[0mfound_var\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_vars\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    866\u001b[0m       \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mshape\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_compatible_with\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfound_var\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_shape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Variable rnn/multi_rnn_cell/cell_0/basic_rnn_cell/kernel/Adam/ already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2005, in __init__\n    self._traceback = tf_stack.extract_stack()\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 3616, in create_op\n    op_def=op_def)\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py\", line 507, in new_func\n    return func(*args, **kwargs)\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 788, in _apply_op_helper\n    op_def=op_def)\n  File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_state_ops.py\", line 1608, in variable_v2\n    shared_name=shared_name, name=name)\n"
     ]
    }
   ],
   "source": [
    "# dynamic_rnn: 셀로 입력되는 데이터의 개수에 맞추어서 셀을 구성 \n",
    "\n",
    "\n",
    "# 3층 셀이 28개가 조성 : 고정 사이즈\n",
    "outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype = tf.float32)\n",
    "\n",
    "# states 가 몇개가 발생하는가?\n",
    "# 3 x 150 x 100(neuron)  : 특징을 3번 추출(multi-layer하면)\n",
    "#                          열방향\n",
    "states_concat = tf.concat(axis = 1, values = states)  # 150 x 300( multi-layer 해서 3층이 쌓여진걸 사이즈 맞추기 )\n",
    "\n",
    "# 가중치 학습 FFNN 망\n",
    "logits = tf.layers.dense(states_concat, n_outputs)  # 가중치 :  300 x 10\n",
    "\n",
    "# 150 X 10   확률값 (: 원핫인코딩 * log( 예측값 ))\n",
    "xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels = y, logits = logits)\n",
    "\n",
    "loss = tf.reduce_mean(xentropy)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate)\n",
    "\n",
    "# MSE(최적값)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "correct = tf.nn.in_top_k(logits, y,1)  # tf.nn.in_top_k : 가장 큰값\n",
    "accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train accuracy:  0.94666666 Test accuracy: 0.9365\n",
      "1 Train accuracy:  0.96 Test accuracy: 0.9486\n",
      "2 Train accuracy:  0.96666664 Test accuracy: 0.9601\n",
      "3 Train accuracy:  0.98 Test accuracy: 0.9725\n",
      "4 Train accuracy:  0.97333336 Test accuracy: 0.9614\n",
      "5 Train accuracy:  0.99333334 Test accuracy: 0.974\n",
      "6 Train accuracy:  0.97333336 Test accuracy: 0.9776\n",
      "7 Train accuracy:  0.9866667 Test accuracy: 0.977\n",
      "8 Train accuracy:  0.99333334 Test accuracy: 0.9803\n",
      "9 Train accuracy:  0.9866667 Test accuracy: 0.9729\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        for iteration in range(  # 60000/150  (training data)\n",
    "            mnist.train.num_examples // batch_size):  \n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            \n",
    "            # 이미지 사이즈로 생성\n",
    "            X_batch = X_batch.reshape((-1, n_steps, n_inputs))\n",
    "            sess.run(training_op, feed_dict = {X:X_batch, y:y_batch})\n",
    "        acc_train = accuracy.eval(feed_dict = {X:X_batch, y:y_batch})\n",
    "        acc_test = accuracy.eval(feed_dict = {X:X_test, y:y_test})\n",
    "        print(epoch, \"Train accuracy: \", acc_train, \"Test accuracy:\", acc_test)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 다른 예시\n",
    "mnist를 RNN 으로 분류해보기     \n",
    "http://blog.naver.com/PostView.nhn?blogId=magnking&logNo=221323257045&redirect=Dlog&widgetTypeCall=true&directAccess=false\n",
    "\n",
    "(아래 코드, 위 사이트 참고)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./mnist/data/train-images-idx3-ubyte.gz\n",
      "Extracting ./mnist/data/train-labels-idx1-ubyte.gz\n",
      "Extracting ./mnist/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting ./mnist/data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7BB3A0EC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7BB3A0EC8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7BB3A0EC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7BB3A0EC8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.python.framework import ops\n",
    "ops.reset_default_graph()\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"./mnist/data/\", one_hot = True)\n",
    "learning_rate = 0.001 # 경사하강법을 사용하겠다\n",
    "total_epoch = 10\n",
    "batch_size = 128  # 6만장 중에서 임의의 128장을 뽑는다\n",
    "\n",
    "n_input = 28\n",
    "n_step = 28\n",
    "n_hidden = 128  # hidden = neuron\n",
    "n_class = 10\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_step, n_input])  # 128x28x28\n",
    "Y = tf.placeholder(tf.float32, [None, n_class])         # 128x10\n",
    "W = tf.Variable(tf.random_normal([n_hidden, n_class]))  # 128x10\n",
    "b = tf.Variable(tf.random_normal([n_class]))   # 10\n",
    "\n",
    "# 28x128 사이즈의 가중치 통과하면 -> # neural: 128특징\n",
    "cell = tf.nn.rnn_cell.BasicRNNCell(n_hidden)  \n",
    "\n",
    "# cell 수 : 28개\n",
    "outputs, states = tf.nn.dynamic_rnn(cell, X, dtype = tf.float32)\n",
    "\n",
    "# FFNN 망\n",
    "# output 사이즈 : 128 x 28 x 128   (각셀마다 뉴런이 128개/// 즉, batch(128) x step(28) x hidden(128))\n",
    "# state : 128 x 128\n",
    "outputs = tf.transpose(outputs, [1,0,2]) # output에서 0과1이 바뀌어서: 28x128x128\n",
    "\n",
    "# cell로 판단( 셀은 28개 )\n",
    "outputs = outputs[-1]  # 맨 마지막 셀: 128 x 128  \n",
    "\n",
    "model = tf.matmul(outputs, W) + b   # 128x128  , 128x10  => 128x10\n",
    "\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = model, labels = Y))\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Avg.cost =  0.636\n",
      "최적화 완료\n",
      "Epoch: 0002 Avg.cost =  0.265\n",
      "최적화 완료\n",
      "Epoch: 0003 Avg.cost =  0.197\n",
      "최적화 완료\n",
      "Epoch: 0004 Avg.cost =  0.159\n",
      "최적화 완료\n",
      "Epoch: 0005 Avg.cost =  0.152\n",
      "최적화 완료\n",
      "Epoch: 0006 Avg.cost =  0.130\n",
      "최적화 완료\n",
      "Epoch: 0007 Avg.cost =  0.122\n",
      "최적화 완료\n",
      "Epoch: 0008 Avg.cost =  0.113\n",
      "최적화 완료\n",
      "Epoch: 0009 Avg.cost =  0.106\n",
      "최적화 완료\n",
      "Epoch: 0010 Avg.cost =  0.104\n",
      "최적화 완료\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()  \n",
    "sess.run(tf.global_variables_initializer())  # 변수 초기화\n",
    "total_batch = int(mnist.train.num_examples//batch_size)  # 60000/128의 몫\n",
    "\n",
    "for epoch in range(total_epoch): # 10번\n",
    "    total_cost = 0\n",
    "    \n",
    "    for i in range(total_batch):  # 한번 돌릴때 전체 사이즈 돌린다\n",
    "        batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "        \n",
    "        # 원래 사이즈로 되돌리기\n",
    "        batch_xs = batch_xs.reshape((batch_size, n_step, n_input))\n",
    "        \n",
    "      # _ : python이 계산한 마지막 결과가 저장되는 것                     \n",
    "        _, cost_val = sess.run([optimizer, cost],   # 경사하강법(optimizer)\n",
    "                              feed_dict={X: batch_xs, Y: batch_ys})\n",
    "        total_cost += cost_val\n",
    "        \n",
    "    print('Epoch:', '%04d' %(epoch +1), \n",
    "          'Avg.cost = ', '{:.3f}'.format(total_cost/total_batch))\n",
    "    print('최적화 완료')      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "문제\n",
    "1. 테스트 데이터를 이용하여 테스트 하는 회로를 구성하시오.\n",
    "2. 3개의 셀을 갖는 multi-layer cell로 수정하시오."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E786260EC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E786260EC8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E786260EC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E786260EC8>>: AttributeError: module 'gast' has no attribute 'Num'\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "in converted code:\n    relative to C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python:\n\n    ops\\rnn_cell_impl.py:1719 call\n        cur_inp, new_state = cell(cur_inp, cur_state)\n    ops\\rnn_cell_impl.py:385 __call__\n        self, inputs, state, scope=scope, *args, **kwargs)\n    layers\\base.py:537 __call__\n        outputs = super(Layer, self).__call__(inputs, *args, **kwargs)\n    keras\\engine\\base_layer.py:591 __call__\n        self._maybe_build(inputs)\n    keras\\engine\\base_layer.py:1881 _maybe_build\n        self.build(input_shapes)\n    keras\\utils\\tf_utils.py:295 wrapper\n        output_shape = fn(instance, input_shape)\n    ops\\rnn_cell_impl.py:455 build\n        shape=[input_depth + self._num_units, self._num_units])\n    keras\\engine\\base_layer.py:1484 add_variable\n        return self.add_weight(*args, **kwargs)\n    layers\\base.py:450 add_weight\n        **kwargs)\n    keras\\engine\\base_layer.py:384 add_weight\n        aggregation=aggregation)\n    training\\tracking\\base.py:663 _add_variable_with_custom_getter\n        **kwargs_for_getter)\n    ops\\variable_scope.py:1496 get_variable\n        aggregation=aggregation)\n    ops\\variable_scope.py:1239 get_variable\n        aggregation=aggregation)\n    ops\\variable_scope.py:545 get_variable\n        return custom_getter(**custom_getter_kwargs)\n    ops\\rnn_cell_impl.py:251 _rnn_get_variable\n        variable = getter(*args, **kwargs)\n    ops\\variable_scope.py:514 _true_getter\n        aggregation=aggregation)\n    ops\\variable_scope.py:864 _get_single_variable\n        (err_msg, \"\".join(traceback.format_list(tb))))\n\n    ValueError: Variable rnn/multi_rnn_cell/cell_0/basic_rnn_cell/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n    \n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2005, in __init__\n        self._traceback = tf_stack.extract_stack()\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 3616, in create_op\n        op_def=op_def)\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py\", line 507, in new_func\n        return func(*args, **kwargs)\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 788, in _apply_op_helper\n        op_def=op_def)\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_state_ops.py\", line 1608, in variable_v2\n        shared_name=shared_name, name=name)\n    \n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-46-46ce1836b79d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[0mmulti_layer_cell\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcontrib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrnn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mMultiRNNCell\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlayers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     16\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 17\u001b[1;33m \u001b[0moutputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstates\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdynamic_rnn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmulti_layer_cell\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     18\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     19\u001b[0m \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m   \u001b[1;31m# 3개로 나오니까 변경하기\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py\u001b[0m in \u001b[0;36mnew_func\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    322\u001b[0m               \u001b[1;34m'in a future version'\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mdate\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;34m'after %s'\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mdate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    323\u001b[0m               instructions)\n\u001b[1;32m--> 324\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    325\u001b[0m     return tf_decorator.make_decorator(\n\u001b[0;32m    326\u001b[0m         \u001b[0mfunc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnew_func\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'deprecated'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\rnn.py\u001b[0m in \u001b[0;36mdynamic_rnn\u001b[1;34m(cell, inputs, sequence_length, initial_state, dtype, parallel_iterations, swap_memory, time_major, scope)\u001b[0m\n\u001b[0;32m    705\u001b[0m         \u001b[0mswap_memory\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mswap_memory\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    706\u001b[0m         \u001b[0msequence_length\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msequence_length\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 707\u001b[1;33m         dtype=dtype)\n\u001b[0m\u001b[0;32m    708\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    709\u001b[0m     \u001b[1;31m# Outputs of _dynamic_rnn_loop are always shaped [time, batch, depth].\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\rnn.py\u001b[0m in \u001b[0;36m_dynamic_rnn_loop\u001b[1;34m(cell, inputs, initial_state, parallel_iterations, swap_memory, sequence_length, dtype)\u001b[0m\n\u001b[0;32m    914\u001b[0m       \u001b[0mparallel_iterations\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mparallel_iterations\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    915\u001b[0m       \u001b[0mmaximum_iterations\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtime_steps\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 916\u001b[1;33m       swap_memory=swap_memory)\n\u001b[0m\u001b[0;32m    917\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    918\u001b[0m   \u001b[1;31m# Unpack final output if not using output tuples.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\control_flow_ops.py\u001b[0m in \u001b[0;36mwhile_loop\u001b[1;34m(cond, body, loop_vars, shape_invariants, parallel_iterations, back_prop, swap_memory, name, maximum_iterations, return_same_structure)\u001b[0m\n\u001b[0;32m   3499\u001b[0m       \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madd_to_collection\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mGraphKeys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mWHILE_CONTEXT\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mloop_context\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3500\u001b[0m     result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants,\n\u001b[1;32m-> 3501\u001b[1;33m                                     return_same_structure)\n\u001b[0m\u001b[0;32m   3502\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mmaximum_iterations\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3503\u001b[0m       \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\control_flow_ops.py\u001b[0m in \u001b[0;36mBuildLoop\u001b[1;34m(self, pred, body, loop_vars, shape_invariants, return_same_structure)\u001b[0m\n\u001b[0;32m   3010\u001b[0m       \u001b[1;32mwith\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_default_graph\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_mutation_lock\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m  \u001b[1;31m# pylint: disable=protected-access\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3011\u001b[0m         original_body_result, exit_vars = self._BuildLoop(\n\u001b[1;32m-> 3012\u001b[1;33m             pred, body, original_loop_vars, loop_vars, shape_invariants)\n\u001b[0m\u001b[0;32m   3013\u001b[0m     \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3014\u001b[0m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\control_flow_ops.py\u001b[0m in \u001b[0;36m_BuildLoop\u001b[1;34m(self, pred, body, original_loop_vars, loop_vars, shape_invariants)\u001b[0m\n\u001b[0;32m   2935\u001b[0m         expand_composites=True)\n\u001b[0;32m   2936\u001b[0m     \u001b[0mpre_summaries\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_collection\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mGraphKeys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_SUMMARY_COLLECTION\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# pylint: disable=protected-access\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2937\u001b[1;33m     \u001b[0mbody_result\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mbody\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mpacked_vars_for_body\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2938\u001b[0m     \u001b[0mpost_summaries\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_collection\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mGraphKeys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_SUMMARY_COLLECTION\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# pylint: disable=protected-access\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2939\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mnest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_sequence_or_composite\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbody_result\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\control_flow_ops.py\u001b[0m in \u001b[0;36m<lambda>\u001b[1;34m(i, lv)\u001b[0m\n\u001b[0;32m   3454\u001b[0m         cond = lambda i, lv: (  # pylint: disable=g-long-lambda\n\u001b[0;32m   3455\u001b[0m             math_ops.logical_and(i < maximum_iterations, orig_cond(*lv)))\n\u001b[1;32m-> 3456\u001b[1;33m         \u001b[0mbody\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mlambda\u001b[0m \u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlv\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mi\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0morig_body\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mlv\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   3457\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3458\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mexecuting_eagerly\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\rnn.py\u001b[0m in \u001b[0;36m_time_step\u001b[1;34m(time, output_ta_t, state)\u001b[0m\n\u001b[0;32m    882\u001b[0m           skip_conditionals=True)\n\u001b[0;32m    883\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 884\u001b[1;33m       \u001b[1;33m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnew_state\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcall_cell\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    885\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    886\u001b[0m     \u001b[1;31m# Keras cells always wrap state as list, even if it's a single tensor.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\rnn.py\u001b[0m in \u001b[0;36m<lambda>\u001b[1;34m()\u001b[0m\n\u001b[0;32m    868\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mis_keras_rnn_cell\u001b[0m \u001b[1;32mand\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mnest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_sequence\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    869\u001b[0m       \u001b[0mstate\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mstate\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 870\u001b[1;33m     \u001b[0mcall_cell\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mlambda\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mcell\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput_t\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    871\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    872\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0msequence_length\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\rnn_cell_impl.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs, state, scope)\u001b[0m\n\u001b[0;32m    246\u001b[0m         \u001b[0msetattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscope_attrname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscope\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    247\u001b[0m       \u001b[1;32mwith\u001b[0m \u001b[0mscope\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 248\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mRNNCell\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    249\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    250\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_rnn_get_variable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgetter\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\layers\\base.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs, *args, **kwargs)\u001b[0m\n\u001b[0;32m    535\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    536\u001b[0m       \u001b[1;31m# Actually call layer\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 537\u001b[1;33m       \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mLayer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    538\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    539\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mcontext\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexecuting_eagerly\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\engine\\base_layer.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs, *args, **kwargs)\u001b[0m\n\u001b[0;32m    632\u001b[0m                     \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mbase_layer_utils\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmark_as_return\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0macd\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    633\u001b[0m                 \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 634\u001b[1;33m                   \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcall_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    635\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    636\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\autograph\\impl\\api.py\u001b[0m in \u001b[0;36mwrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    147\u001b[0m       \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m  \u001b[1;31m# pylint:disable=broad-except\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    148\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'ag_error_metadata'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 149\u001b[1;33m           \u001b[1;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mag_error_metadata\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mto_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    150\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    151\u001b[0m           \u001b[1;32mraise\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: in converted code:\n    relative to C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python:\n\n    ops\\rnn_cell_impl.py:1719 call\n        cur_inp, new_state = cell(cur_inp, cur_state)\n    ops\\rnn_cell_impl.py:385 __call__\n        self, inputs, state, scope=scope, *args, **kwargs)\n    layers\\base.py:537 __call__\n        outputs = super(Layer, self).__call__(inputs, *args, **kwargs)\n    keras\\engine\\base_layer.py:591 __call__\n        self._maybe_build(inputs)\n    keras\\engine\\base_layer.py:1881 _maybe_build\n        self.build(input_shapes)\n    keras\\utils\\tf_utils.py:295 wrapper\n        output_shape = fn(instance, input_shape)\n    ops\\rnn_cell_impl.py:455 build\n        shape=[input_depth + self._num_units, self._num_units])\n    keras\\engine\\base_layer.py:1484 add_variable\n        return self.add_weight(*args, **kwargs)\n    layers\\base.py:450 add_weight\n        **kwargs)\n    keras\\engine\\base_layer.py:384 add_weight\n        aggregation=aggregation)\n    training\\tracking\\base.py:663 _add_variable_with_custom_getter\n        **kwargs_for_getter)\n    ops\\variable_scope.py:1496 get_variable\n        aggregation=aggregation)\n    ops\\variable_scope.py:1239 get_variable\n        aggregation=aggregation)\n    ops\\variable_scope.py:545 get_variable\n        return custom_getter(**custom_getter_kwargs)\n    ops\\rnn_cell_impl.py:251 _rnn_get_variable\n        variable = getter(*args, **kwargs)\n    ops\\variable_scope.py:514 _true_getter\n        aggregation=aggregation)\n    ops\\variable_scope.py:864 _get_single_variable\n        (err_msg, \"\".join(traceback.format_list(tb))))\n\n    ValueError: Variable rnn/multi_rnn_cell/cell_0/basic_rnn_cell/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n    \n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2005, in __init__\n        self._traceback = tf_stack.extract_stack()\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 3616, in create_op\n        op_def=op_def)\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\util\\deprecation.py\", line 507, in new_func\n        return func(*args, **kwargs)\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 788, in _apply_op_helper\n        op_def=op_def)\n      File \"C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_state_ops.py\", line 1608, in variable_v2\n        shared_name=shared_name, name=name)\n    \n"
     ]
    }
   ],
   "source": [
    "# 2번\n",
    "n_layers = 3\n",
    "\n",
    "# 3개의 셀로 멀티레이어를 생성해도 output은 하나의 셀일 경우와 동일\n",
    "# state에서는 3개의 레이어마다 생성\n",
    "layers = [tf.contrib.rnn.BasicRNNCell(num_units = n_hidden, activation = tf.nn.relu)\n",
    "         for layer in range(n_layers)]\n",
    "\n",
    "multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)\n",
    "\n",
    "outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype = tf.float32)\n",
    "\n",
    "outputs = tf.transpose(outputs, [1,0,2])   # 3개로 나오니까 변경하기\n",
    "\n",
    "# 1 번?\n",
    "# 세션이 아직 열려있어서 다시 열지 않아도 된다\n",
    "is_correct = tf.equal(tf.argmax(model, 1), tf.argmax(Y,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(is_corect, tf.float32))\n",
    "test_batch_size = len(mnist.test.images)\n",
    "test_xs = mnist.test.images.reshape(test_batch_size, n_step, n_input)\n",
    "test_ys = mnist.test.labels\n",
    "print(\"정확도 : \", sess.run(accuracy, feed_dict = {X:test_xs, Y:test_ys}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 데이터의 shape (2, 3, 1)\n",
      "WARNING:tensorflow:Entity <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD1AFA48>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD1AFA48>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD1AFA48>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD1AFA48>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "(2, 150)\n",
      "(2, 150)\n",
      "rnn이 출력하는 outputs의 의미 (2, 3, 150)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "values231 = np.array([\n",
    "    [[1],[2],[3]],\n",
    "    [[2],[3],[4]]\n",
    "])\n",
    "print(\"입력 데이터의 shape\", values231.shape) # 2x3x1\n",
    "\n",
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "tf_values231 =tf.constant(values231, dtype = tf.float32)\n",
    "\n",
    "# 1x100 의 가중치가 4개가 생성되어야 한다\n",
    "lstm_cell = tf.contrib.rnn.LSTMCell(num_units = 100) # 뉴론\n",
    "\n",
    "# rnn 모델은 동일 : 망과 망을 연결할때\n",
    "# 단, state의 값이 2개가 나온다. \n",
    "# latent time 지연시간\n",
    "outputs, state = tf.nn.dynamic_rnn(cell = lstm_cell, dtype = tf.float32, inputs = tf_values231)\n",
    "\n",
    "# 망과 망을 연결할때 (번역 망 : 한글, 영어) 필요\n",
    "print(state.c.shape) # 2x100\n",
    "print(state.h.shape) # 2x100\n",
    "\n",
    "print(\"rnn이 출력하는 outputs의 의미\", outputs.shape)  # 2x3x100\n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    output_run, state_run = sess.run([outputs, state])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD170BC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD170BC8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD170BC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7BD170BC8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING:tensorflow:Entity <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7B369AAC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7B369AAC8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7B369AAC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method LSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.LSTMCell object at 0x000001E7B369AAC8>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "(2, 3, 100)\n",
      "(2, 3, 100)\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "values = tf.constant(np.array([\n",
    "    [[1],[2],[3]],\n",
    "    [[2],[3],[4]]\n",
    "]), dtype = tf.float32)\n",
    "lstm_cell_fw = tf.contrib.rnn.LSTMCell(100)\n",
    "lstm_cell_bw = tf.contrib.rnn.LSTMCell(100)\n",
    "# Multi Cell : 셀에서\n",
    "# 양방향 LSTM\n",
    "(output_fw, output_bw), (output_state_fw, output_state_bw) = tf.nn.bidirectional_dynamic_rnn(\n",
    "    cell_fw=lstm_cell_fw,\n",
    "    cell_bw=lstm_cell_bw,\n",
    "    inputs=values,\n",
    "    dtype=tf.float32)\n",
    "print(output_fw.shape)\n",
    "print(output_bw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 단어를 수치화\n",
    "- embedding\n",
    "- pprint :\n",
    "문자열에 포함된 format 함수를 사용하면 일반적으로 가장 보기 좋은 출력 형태를 생성할 수 있다.\n",
    "그런데, format 함수는 전달해야 할 것이 많아서 번거로운 느낌이 들 때가 많다.\n",
    "이럴 때 조금 부족하기는 하지만, pprint 모듈을 사용하면 상당 부분 절충하는 것이 가능하다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 5, 4)\n",
      "array([[[1., 0., 0., 0.],\n",
      "        [0., 1., 0., 0.],\n",
      "        [0., 0., 1., 0.],\n",
      "        [0., 0., 1., 0.],\n",
      "        [0., 0., 0., 1.]]], dtype=float32)\n",
      "WARNING:tensorflow:Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7B3B34988>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7B3B34988>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7B3B34988>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicRNNCell object at 0x000001E7B3B34988>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "array([[[-0.25850698, -0.54317063],\n",
      "        [ 0.22713791,  0.3987561 ],\n",
      "        [ 0.6081036 , -0.5190886 ],\n",
      "        [ 0.6853291 , -0.10337783],\n",
      "        [ 0.7947474 , -0.37653482]]], dtype=float32)\n",
      "\n",
      "(1, 5, 2)\n",
      "(1, 2)\n",
      "array([[ 0.7947474 , -0.37653482]], dtype=float32)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ICT01_17\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\client\\session.py:1735: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).\n",
      "  warnings.warn('An interactive session is already active. This can '\n"
     ]
    }
   ],
   "source": [
    "# embedding\n",
    "\n",
    "from tensorflow.contrib import rnn\n",
    "import pprint\n",
    "from tensorflow.python.framework import ops\n",
    "\n",
    "ops.reset_default_graph()\n",
    "pp = pprint.PrettyPrinter(indent = 4)\n",
    "sess = tf.InteractiveSession()\n",
    "\n",
    "# 단어를 원핫인코딩: 수치화\n",
    "h = [1,0,0,0]\n",
    "e = [0,1,0,0]\n",
    "l = [0,0,1,0]\n",
    "o = [0,0,0,1]\n",
    "\n",
    "# 변수 공유를 위해 지정\n",
    "with tf.variable_scope('five_sequences') as scope:\n",
    "    hidden_size = 2\n",
    "    # 4 -> 2개의 특징으로 vector화 된다\n",
    "    \n",
    "    cell = tf.contrib.rnn.BasicRNNCell(num_units = hidden_size)\n",
    "    x_data = np.array([[h,e,l,l,o]], dtype = np.float32)\n",
    "    \n",
    "    print(x_data.shape)       # (1, 5, 4): 데이터 한개 => 3차원으로\n",
    "  #      [[[1., 0., 0., 0.],\n",
    "  #       [0., 1., 0., 0.],\n",
    "  #       [0., 0., 1., 0.],\n",
    "  #       [0., 0., 1., 0.],\n",
    "  #      [0., 0., 0., 1.]]]    3차원\n",
    "        \n",
    "    pp.pprint(x_data)\n",
    "    # cell의 개수 : 5\n",
    "    # 셀당 입력데이터는 4\n",
    "    # 셀당 가중치는 4 x 2\n",
    "    \n",
    "    outputs, states = tf.nn.dynamic_rnn(cell, x_data, dtype = tf.float32)\n",
    "        \n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    pp.pprint(outputs.eval())\n",
    "    \n",
    "    print(outputs.shape)  # 1x5x2\n",
    "    print(states.shape)   # 1x2  (데이터 한개)\n",
    "    pp.pprint(states.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 문자에 번호를 키로주기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "ops.reset_default_graph()\n",
    "\n",
    "char_arr = ['a','b','c','d','e','f','g',\n",
    "           'h','i','j','k','l','m','n',\n",
    "           'o','p','q','r','s','t','u',\n",
    "            'v','w','x','y','z']\n",
    "# num_index [0,1,2,3,..................,25]\n",
    "\n",
    "num_dic = {n:i for i, n in enumerate(char_arr)}\n",
    "# 키를 만들어줌\n",
    "# a: 0 , b :1번. (번호를 키로)\n",
    "\n",
    "dic_len = len(num_dic) # 26개\n",
    "# rule base => 자동 툴 => 가중치만 이용\n",
    "seq_data = ['word','wood','deep','dive','cold','cool','load','love','kiss','kind']\n",
    "\n",
    "\n",
    "# 10x3 (3: word에서 wor)\n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([array([], shape=(0, 26), dtype=float64),\n",
       "  array([], shape=(0, 26), dtype=float64),\n",
       "  array([], shape=(0, 26), dtype=float64),\n",
       "  array([], shape=(0, 26), dtype=float64)],\n",
       " [22, 14, 17, 3])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 26x26 행렬 (이렇게)\n",
    "# 100000000000000000000000000000000...\n",
    "# 010000000000000000000000000000000...\n",
    "# 001000000000000000000000000000000...\n",
    "\n",
    "\n",
    "# w\n",
    "# 000000000000000000000000000000010...  (22번)\n",
    "# 000000000000010000000000000000000...  (14번)\n",
    "# 00000000000000010000000000000000...   (17번)\n",
    "\n",
    "\n",
    "def make_batch(seq_data):\n",
    "    input_batch = []\n",
    "    target_batch = []\n",
    "    \n",
    "    for seq in seq_data:  # seq에서 'word' 가져오고 마지막은 제외[:-1]\n",
    "        input = [num_dic[n] for n in seq[:-1]]  # w: 22번, o:14번, r:17번, (마지막 d는 제외)\n",
    "        target = num_dic[seq[-1]]   # d :3번\n",
    "        input_batch.append(np.eye(dic_len)[input])  # 원핫인코딩\n",
    "        target_batch.append(target)\n",
    "    return input_batch, target_batch\n",
    "\n",
    "make_batch(seq_data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 학습시키기\n",
    "\n",
    "learning_rate = 0.01\n",
    "n_hidden = 128\n",
    "total_epoch = 30\n",
    "n_step = 3\n",
    "n_input = n_class = dic_len  # 26개\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_step, n_input])  # 10x3x26\n",
    "Y = tf.placeholder(tf.int32, [None]) # 10x26\n",
    "\n",
    "# FFNN\n",
    "W = tf.Variable(tf.random_normal([n_hidden, n_class]))  # 128 x 26\n",
    "b = tf.Variable(tf.random_normal([n_class]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7B2AE2148>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7B2AE2148>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING: Entity <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7B2AE2148>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method MultiRNNCell.call of <tensorflow.python.ops.rnn_cell_impl.MultiRNNCell object at 0x000001E7B2AE2148>>: AttributeError: module 'gast' has no attribute 'Num'\n",
      "WARNING:tensorflow:Entity <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6CC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6CC8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6CC8>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6CC8>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING:tensorflow:Entity <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6308>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6308>>: AssertionError: Bad argument number for Name: 3, expecting 4\n",
      "WARNING: Entity <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6308>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method BasicLSTMCell.call of <tensorflow.python.ops.rnn_cell_impl.BasicLSTMCell object at 0x000001E7B2AD6308>>: AssertionError: Bad argument number for Name: 3, expecting 4\n"
     ]
    }
   ],
   "source": [
    "cell1 = tf.nn.rnn_cell.BasicLSTMCell(n_hidden) # 특정수\n",
    "# 과적합 방지하기 위해 dropout\n",
    "cell1 = tf.nn.rnn_cell.DropoutWrapper(cell1, output_keep_prob = 0.5,seed=10)\n",
    "\n",
    "# 2레이어로 구성된 멀티 셀\n",
    "cell2 = tf.nn.rnn_cell.BasicLSTMCell(n_hidden)\n",
    "\n",
    "multi_cell = tf.nn.rnn_cell.MultiRNNCell([cell1,cell2])\n",
    "outputs, states = tf.nn.dynamic_rnn(multi_cell,X,dtype = tf.float32)\n",
    "\n",
    "# 10x3x128\n",
    "# state : 10 x 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputs = tf.transpose(outputs,[1,0,2])  # 3x10x128 맨뒤의 셀\n",
    "outputs = outputs[-1]  # 10 x128\n",
    "model = tf.matmul(outputs,W) + b  # 비선형 회귀방식\n",
    "\n",
    "# 분류\n",
    "cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=model,labels=Y))\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 cost =  3.864312\n",
      "Epoch: 0002 cost =  2.913284\n",
      "Epoch: 0003 cost =  1.485137\n",
      "Epoch: 0004 cost =  1.805050\n",
      "Epoch: 0005 cost =  0.863734\n",
      "Epoch: 0006 cost =  0.965523\n",
      "Epoch: 0007 cost =  1.035705\n",
      "Epoch: 0008 cost =  0.581941\n",
      "Epoch: 0009 cost =  0.405560\n",
      "Epoch: 0010 cost =  0.634337\n",
      "Epoch: 0011 cost =  0.320449\n",
      "Epoch: 0012 cost =  0.332297\n",
      "Epoch: 0013 cost =  0.503054\n",
      "Epoch: 0014 cost =  0.443100\n",
      "Epoch: 0015 cost =  0.157287\n",
      "Epoch: 0016 cost =  0.150054\n",
      "Epoch: 0017 cost =  0.188537\n",
      "Epoch: 0018 cost =  0.050019\n",
      "Epoch: 0019 cost =  0.105574\n",
      "Epoch: 0020 cost =  0.153275\n",
      "Epoch: 0021 cost =  0.067556\n",
      "Epoch: 0022 cost =  0.046727\n",
      "Epoch: 0023 cost =  0.162803\n",
      "Epoch: 0024 cost =  0.102455\n",
      "Epoch: 0025 cost =  0.050631\n",
      "Epoch: 0026 cost =  0.048829\n",
      "Epoch: 0027 cost =  0.027342\n",
      "Epoch: 0028 cost =  0.029229\n",
      "Epoch: 0029 cost =  0.007853\n",
      "Epoch: 0030 cost =  0.010423\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "input_batch, target_batch = make_batch(seq_data) # 단어데이터\n",
    "\n",
    "\n",
    "for epoch in range(total_epoch):\n",
    "    _, loss = sess.run([optimizer, cost],feed_dict = {X:input_batch, Y:target_batch})\n",
    "       \n",
    "    print('Epoch:', '%04d' %(epoch +1), \n",
    "          'cost = ', '{:.6f}'.format(loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = tf.cast(tf.argmax(model, 1), tf.int32)\n",
    "prediction_check = tf.equal(prediction, Y)\n",
    "accuracy = tf.reduce_mean(tf.cast(prediction_check, tf.float32))\n",
    "\n",
    "input_batch, targe_batch = make_batch(seq_data)\n",
    "predict, accuracy_val = sess.run([prediction, accuracy], feed_dict = {X:input_batch, Y:target_batch})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "문제    \n",
    "- predict 한 결과를 출력하고 accuracy를 출력하시오.   \n",
    "- 입력 단어와 예측결과를 나란히 출력해 보시오."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "예측된 결과 [ 3  3 15  4  3 11  3  4 18  3]\n",
      "Accuracy 1.0\n",
      "입력값: ['wor ', 'woo ', 'dee ', 'div ', 'col ', 'coo ', 'loa ', 'lov ', 'kis ', 'kin ']\n",
      "예측값: ['word', 'wood', 'deep', 'dive', 'cold', 'cool', 'load', 'love', 'kiss', 'kind']\n",
      "정확도: 1.0\n"
     ]
    }
   ],
   "source": [
    "# 문제\n",
    "print(\"예측된 결과\", predict)\n",
    "print(\"Accuracy\", accuracy_val)\n",
    "\n",
    "predict_words = []\n",
    "# 단어가 순서대로 입력, 순서대로 예측\n",
    "for idx, val in enumerate(seq_data):\n",
    "    last_char = char_arr[predict[idx]] # 인덱스가 단어 매핑\n",
    "    predict_words.append(val[:3]+ last_char)\n",
    "print(\"입력값:\",[w[:3] + ' ' for w in seq_data])\n",
    "print(\"예측값:\", predict_words)\n",
    "print(\"정확도:\", accuracy_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}