Notebook_Report_MO436_RL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e8e702f",
   "metadata": {},
   "source": [
    "# **MO436 - Project 1 - Pokémon RL**\n",
    "\n",
    "## **Group:**\n",
    "\n",
    "* Bruno César de Oliveira Souza - \n",
    "\n",
    "* Henrique Lima Cará de Oliveira - 091518\n",
    "\n",
    "* Leonardo de Lellis Rossi - 261900\n",
    "\n",
    "* Maurício Pereira Lopes - \n",
    "\n",
    "[Git Repository](https://github.com/leolellisr/poke_RL)\n",
    "\n",
    "[Graphs MC Control, MC Control FA, Q-Learning, Q-Learning FA, SARSA($\\lambda$) Deterministic and SARSA($\\lambda$) FA](https://app.neptune.ai/leolellisr/rl-pokeenv)\n",
    "\n",
    "[Graphs SARSA($\\lambda$) Stochastic](https://app.neptune.ai/mauricioplopes/poke-env)\n",
    "\n",
    "Images and results are showed with IFrame from IPython.display, but it needs the images stored in the /images/report folder, available in our [git repository](https://github.com/leolellisr/poke_RL). \n",
    "\n",
    "Images and Results also are presented with links from LightShot, that doesn't need files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4775d7b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74c81a3e",
   "metadata": {},
   "source": [
    "# Goal and Motivation\n",
    "\n",
    "* This project aims to employ different reinforcement learning techniques to train agents in a Pokémon battle simulator;\n",
    "\n",
    "* Our motivation is that the trainer automatically learns, by making decisions through the analysis of states and rewards related to their performance, how to win battles throughout the episodes, noticing:\n",
    "  * the different types between Pokémon;\n",
    "  * which moves cause more damage to the opponent's Pokémon;\n",
    "  * what are the possible strategies using no-damage moves;\n",
    "  * and the best times to switch Pokémon. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "301ec678",
   "metadata": {},
   "source": [
    "## **The  problem addressed**\n",
    "* [Pokémon](https://www.pokemon.com) is a popular Japanese RPG (Role Playing Game) which stands a world championship every year; \n",
    "* One single [battle](https://bulbapedia.bulbagarden.net/wiki/Pokémon_battle) of Pokémon has two players. Each player has a 6-Pokémon team; \n",
    "* Each Pokémon has:\n",
    " * 6 [stats](https://bulbapedia.bulbagarden.net/wiki/Stat) (Health Points, Attack, Defense, Special Attack, Special Defense, Speed). The first 5 are used in the damage calculation. The speed defined which Pokémon moves first in the turn.\n",
    "  * The Health Points goes from 100% (healthy) to 0% (fainted);\n",
    " * 4 possible moves (each with a limited number of uses). Each move has an accuracy, a percentage of success or fail;\n",
    " * one [ability](https://bulbapedia.bulbagarden.net/wiki/Ability) that has special effects in the field;\n",
    " * one [nature](https://bulbapedia.bulbagarden.net/wiki/Nature) that specifies which stats are higher and which are lower;\n",
    " * one [item](https://bulbapedia.bulbagarden.net/wiki/Item), that can  restore Health Points or increase the Power of an Attack.\n",
    "* The winner of the battle is the player that makes all Pokémon of the oposing team to faint (all oposing Pokémon with health points equals zero, \"last man standing\" criteria);\n",
    "* Only one Pokémon of each team can be at the battle field at the same time;\n",
    "* Every turn, each players select one action: one of the 4 moves of their active Pokémon or [switching](https://bulbapedia.bulbagarden.net/wiki/Recall) for one of other non-fainted Pokémon of their team;\n",
    "\n",
    "* Pokémon can be summarized as an analyze state (turn) -> take action sequence game. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0829799b",
   "metadata": {},
   "source": [
    "* By standard, Pokémon is a stochastic game:\n",
    " * One move can have an accuracy value less than 100%, then this move has a probability to be missed;\n",
    " * The damage moves (attacks) have the following [damage calculation](https://bulbapedia.bulbagarden.net/wiki/Damage):\n",
    "  ![Damage](https://wikimedia.org/api/rest_v1/media/math/render/svg/b8c51fed93bb9a80ae8febc13700a40b8a5da402)\n",
    "  \n",
    " where:\n",
    "  * **[Level](https://bulbapedia.bulbagarden.net/wiki/Level)** (the level of the attacking Pokémon);\n",
    "  * **A** is the effective Attack stat of the attacking Pokémon if the used move is a physical move, or the effective Special Attack stat of the attacking Pokémon if the used move is a special move;\n",
    "  * **D** is the effective Defense stat of the target if the used move is a physical move or a special move that uses the target's Defense stat, or the effective Special Defense of the target if the used move is an other special move;\n",
    "  * **[Power](https://bulbapedia.bulbagarden.net/wiki/Power)** is the effective power of the used move;\n",
    "  * **Weather** is 1.5 if a Water-type move is being used during rain or a Fire-type move during harsh sunlight, and 0.5 if a Water-type move is used during harsh sunlight or a Fire-type move during rain, and 1 otherwise.\n",
    "  * **[Critical](https://bulbapedia.bulbagarden.net/wiki/Critical_hit)** has 6.25% chance of occurs and multiplies the damage by 1.5;\n",
    "  * **random** is a random factor between 0.85 and 1.00 (inclusive):\n",
    "  * **[STAB](https://bulbapedia.bulbagarden.net/wiki/Same-type_attack_bonus)** is the same-type attack bonus. This is equal to 1.5 if the move's type matches any of the user's types, 2 if the user of the move additionally has the ability Adaptability, and 1 if otherwise;\n",
    "  * **[Type](https://bulbapedia.bulbagarden.net/wiki/Type)** is the type effectiveness. This can be 0 (ineffective); 0.25, 0.5 (not very effective); 1 (normally effective); 2, or 4 (super effective), depending on both the move's and target's types;\n",
    "  * **[Burn](https://bulbapedia.bulbagarden.net/wiki/Burn_(status_condition))** is 0.5 (from Generation III onward) if the attacker is burned, its Ability is not Guts, and the used move is a physical move (other than Facade from Generation VI onward), and 1 otherwise.\n",
    "  * **other** is 1 in most cases, and a different multiplier when specific interactions of moves, Abilities, or items take effect. In this work, this is applied just to Pokémon that has the item **Life Orb**, which multiplies the damage by 1.3.\n",
    "  \n",
    "  * **Not** used in this work (equals 1):\n",
    "   * Targets (for Battles with more than two active Pokémon in the field);\n",
    "   * Badge ( just applied in Generation II);\n",
    "   \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4dc79c",
   "metadata": {},
   "source": [
    "# **MDP formulation and discretization model** \n",
    "\n",
    "## Original (stochastic)\n",
    "\n",
    "We considered our original (stochastic) MDP as a tuple $M = (S, A, \\phi, R)$, where:\n",
    "* **S** is the whole set of possible states. One state **s $\\in$ S**  is defined at each turn with 12 battle elements concatenated, that correspond to:\n",
    " * [0] Our Active Pokémon index (0: Venusaur,  1: Pikachu, 2: Tauros, 3: Sirfetch'd, 4: Blastoise, 5: Charizard);\n",
    " * [1] Opponent Active Pokémon index (0: Eevee,  1: Vaporeon, 2: Leafeon, 3: Sylveon, 4: Jolteon, 5: Umbreon);\n",
    " * [2-5] Active Pokémon moves base power (if a move doesn't have base power, default to -1);\n",
    " * [6-9] Active Pokémon moves damage multipliers;\n",
    " * [10] Our remaining Pokémon;\n",
    " * [11] Opponent remaining Pokémon.\n",
    " \n",
    "* **A** is the whole set of possible actions. Our action space is a range [0, 8]. One action **a $\\in$ A** is one of the possible choices:\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    "\n",
    "When a selected action cannot be executed, we random select another possible action.\n",
    "\n",
    "* **$\\phi$** is a stochastic transition function that occurs from state **s** to state **s'**, by taking an action **a**. The following parameters are part of our  stochastic transition function:\n",
    " * Move's accuracy (chance of the move successfully occurs or to fail);\n",
    " * Damage calculation: The **[Critical](https://bulbapedia.bulbagarden.net/wiki/Critical_hit)** parameter (6.25% chance of occurs) and the **random** parameter, ranging from 0.85 and 1.00 (inclusive).\n",
    " * Move's effects: Some moves have [additional effects](https://bulbapedia.bulbagarden.net/wiki/Additional_effect). e.g.: Iron Head have 30% chance of flinching the target (target cannot move in the turn).\n",
    "\n",
    "* **R** is a set of rewards. A reward **r $\\in$ R** is acquired in state **s** by taking the action **a**. The rewards are calculated at the end of the turn. The value of reward **r** is defined by the sum of elements:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle.\n",
    " \n",
    "### Stochastic Team\n",
    "\n",
    "Our stochastic team, with each Pokémon, their abilities, natures, items, moves (with base power and accuracy) and possible switches are shown in [Team](https://prnt.sc/1y73c3t).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0cc896fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Team stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d6d8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Team stochastic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ddd9d48",
   "metadata": {},
   "source": [
    "## Deterministic\n",
    "\n",
    "To adapt Pokémon to a deterministic environment, we use Pokémon that cannot receive a critical hit, moves with only 100% accuracy and edit the server code to ignore the random parameter in damage calculation, removing the stochastic transition function $\\phi$ from our MDP. Therefore, now our MDP is a tuple $M = (S, A, R)$, where:\n",
    "* **S** is the whole set of possible states. One state **s $\\in$ S**  is defined at each turn with 12 battle elements concatenated, that correspond to:\n",
    " * [0] Our Active Pokémon index ;\n",
    " * [1] Opponent Active Pokémon index ;\n",
    " * [2-5] Active Pokémon moves base power (if a move doesn't have base power, default to -1);\n",
    " * [6-9] Active Pokémon moves damage multipliers;\n",
    " * [10] Our remaining Pokémon;\n",
    " * [11] Opponent remaining Pokémon.\n",
    " \n",
    "* **A** is the whole set of possible actions. Our action space is a range [0, 8] (len: 9). One action **a $\\in$ A** is one of the possible choices:\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    "\n",
    "When a selected action cannot be executed, we random select another possible action.\n",
    "\n",
    "* **R** is a set of rewards. A reward **r $\\in$ R** is acquired in state **s** by taking the action **a**. The rewards are calculated at the end of each turn. The value of reward **r** is defined by the sum of elements:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle.\n",
    " \n",
    "### Deterministic Team\n",
    "\n",
    "Our deterministic team, with each Pokémon, their abilities, natures, items, moves (with base power and accuracy) and possible switches are shown in [Team](https://prnt.sc/1ydn52l).\n",
    "\n",
    "We use on both teams only Pokémon with Battle Armor or Shell Armor abilities, which prevent critical hits from being performed. Also, we use in both teams only moves with 100% accuracy, with no chance of error, and the move haven't additional effects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5568f359",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Team deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d668>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Team deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac957ae0",
   "metadata": {},
   "source": [
    "## Search space\n",
    "\n",
    "The features that integrate our states are shown in [this figure](https://prnt.sc/1yfksfe) and below. For a single battle between two players with 6 Pokémon each, we have $1.016.064$ possible states.\n",
    "\n",
    "Since we have 9 possible actions for each Pokémon, we total $9.144.576$ possibilities for each battle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8337175c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"920\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/possible_states.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d6a0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/possible_states.png\", width=920, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "140a7aca",
   "metadata": {},
   "source": [
    "# **The environments built**\n",
    "\n",
    "The environment used is [Pokémon Showdown](https://play.pokemonshowdown.com), a [open-source](https://github.com/smogon/pokemon-showdown.git) Pokémon battle simulator.\n",
    "\n",
    "[Example](https://prnt.sc/1ydofwv) of one battle in Pokémon Showdown.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "68cf8b47",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"1000\"\n",
       "            src=\"images/report/showdownEx.jpg\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d710>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/showdownEx.jpg\", width=1000, height=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aecad467",
   "metadata": {},
   "source": [
    "To communicate our agents with Pokémon Showdown we used [poke-env](https://poke-env.readthedocs.io/en/latest/) a Python environment for interacting in pokemon showdown battles.\n",
    "\n",
    "We used separated Python classes for define the Players that are trained with each method. These classes communicates with Pokémon Showdown and implements the poke-env methods to:\n",
    "* Create battles;\n",
    "* Accept battles;\n",
    "* Send orders (actions) to Pokémon Showdown.\n",
    "\n",
    "## Original (stochastic)\n",
    "\n",
    "To speed up the battles, we hosted our own server of Pokemon Showdown in localhost. It requires Node.js v10+."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df356673",
   "metadata": {},
   "outputs": [],
   "source": [
    "git clone https://github.com/smogon/pokemon-showdown.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c33adf08",
   "metadata": {},
   "source": [
    "After clone the repository, it's needed to create a logs folder in root.\n",
    "\n",
    "To configure the server, we used the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7c34f51",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd pokemon-showdown\n",
    "npm install\n",
    "cp config/config-example.js config/config.js"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eabf3c0",
   "metadata": {},
   "source": [
    "To start the server, we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59215c46",
   "metadata": {},
   "outputs": [],
   "source": [
    "node pokemon-showdown start --no-security"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6548239f",
   "metadata": {},
   "source": [
    "## Deterministic environment\n",
    "\n",
    "To adapt our environment to a deterministic setup, we had to establish the following parameters:\n",
    "\n",
    "* We removed the random component of sim/battle.ts from the Pokémon Showdown simulator;\n",
    "\n",
    "* We use on both teams only Pokémon with Battle Armor or Shell Armor abilities, which prevent critical hits from being performed;\n",
    "\n",
    "* We used in both teams only moves with 100% accuracy, with no chance of error;\n",
    "\n",
    "* We didn't use any move with additional effect. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46e0d1c5",
   "metadata": {},
   "source": [
    "# **Characteristics of  the problem**\n",
    "\n",
    "* Both of our environments (stochastic and deterministic) are episodic. One state occurs after another;\n",
    "\n",
    "* Our terminal states are:\n",
    " * When all our Pokémon are fainted (we lose);\n",
    " * When all opponent Pokémon are fainted (we won).\n",
    "\n",
    "* As specified before, a reward **r** is calculated at the end of a turn. The value of reward **r** is defined by the sum of:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "496d533a",
   "metadata": {},
   "source": [
    "# **Monte Carlo Control**\n",
    "\n",
    "We implemented Monte Carlo Control First Visit. \n",
    "\n",
    "* We defined 2 defaultDicts, $Q(s,a)$ and $N(s,a)$, to store the value function and the number of occurences of action **a** in state **s**, respectively. Both of them are initialized to zero;\n",
    " * Each dict has the keys as the states **s**;\n",
    " * The value of a key is an array of the len of action space (len: 9);\n",
    " * Each value of an array corresponds to an action in our action space;\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s,a]$;\n",
    "\n",
    "* In the first move of a turn, we choose a random action;\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s,:])$. We tested with 3 values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$};\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4b61522",
   "metadata": {},
   "outputs": [],
   "source": [
    "def epsilon_greedy_policy(Q, n0, N):\n",
    "    def policy_fn(state):\n",
    "        dimension = Q[state].shape\n",
    "        greedy_action = np.argmax(Q[state])\n",
    "        epsilon = n0/(n0+np.sum(N[state]))\n",
    "        probabilities = np.full(dimension, epsilon / dimension[0])\n",
    "        probabilities[greedy_action] += 1 - epsilon\n",
    "        return probabilities\n",
    "    return policy_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e9e92c2",
   "metadata": {},
   "source": [
    "* In each turn, we collect our observations to define our state with the embed battle function, getting the 12 battle elements concatenated. The state is converted to a string, that is used as a key to our defaultDicts $Q(s,a)$ and $S(s,a)$;\n",
    " \n",
    "* With the state, we select an action using our policy and send it to Showdown;\n",
    "\n",
    "* We store the state and selected action to append to an episode array in the next turn with the computed reward;\n",
    "\n",
    "* At the end of the battle, $Q(s,a)$ and $N(s,a)$ are updated for the first time of each state seen in that battle:\n",
    " * $N(s,a)$ is increased by 1 for each action performed in a particular state;\n",
    " * The value of $Q(s,a)$ is summed to $\\alpha*(G(t)-Q(s,a))$;\n",
    " * The return $G(t)$ is given by $ \\sum \\limits _{i=0} ^{T-1} \\gamma^iR(t)$. We used $\\gamma = 0,75$;\n",
    " * Our policy is then updated with the new values of $Q(s,a)$ and $N(s,a)$;\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e8bee26",
   "metadata": {},
   "outputs": [],
   "source": [
    "rewards = [reward for state,action,reward in self.episode]\n",
    "states = [state for state,action,reward in self.episode]\n",
    "actions = [action for state,action,reward in self.episode]\n",
    "t_array = range(len(self.episode)+1)\n",
    "        \n",
    "# Computing Q and N\n",
    "for idx, state in enumerate(states):\n",
    "    action = actions[idx]\n",
    "    if state not in self.visited_states:\n",
    "        self.N[state][action] += 1\n",
    "        returnGt = sum([reward*pow(self.gamma, t) for reward, t in zip(rewards[idx:], t_array[:(-idx-1)])]) \n",
    "\n",
    "        # incremental update of Q value is more efficient than keeping a record of all rewards\n",
    "        # and averaging after every new reward\n",
    "        # step-size: 1./N[state][action]\n",
    "                \n",
    "        self.Q[state][action] += (1/self.N[state][action])*(returnGt-self.Q[state][action]) \n",
    "        self.visited_states.append(state)\n",
    "                \n",
    "self.visited_states = []\n",
    "self.episode = []\n",
    "# Define new policy with updated Q and N\n",
    "self.policy = self.update_epsilon_greedy_policy(self.Q, self.n0, self.N)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba7eef64",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greater base power. We trained for $10.000$ battles. At the end, we saved $Q$ and $N$ in json files to future use.\n",
    "\n",
    "* We validate our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88042d79",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c3cef588",
   "metadata": {},
   "source": [
    "## MC_Control - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-84/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### MC_Control - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control - Stochastic - Rewards (per step) - Training](https://prnt.sc/1ycoapc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9335798f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d4a8>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baea5e17",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Win percentages (per step) - Training\n",
    "[ MC_Control - Stochastic - Win percentages (per step) - Training](https://prnt.sc/1ycojai)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "48deefa8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d400>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f91c562",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1262/3333$ battles [this is $37.86\\%$ and took $743.18$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1063/3333$ battles [this is $31.89\\%$ and took $743.37$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1166/3333$ battles [this is $34.98\\%$ and took $622.97$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3026/3333$ battles [this is $90.79\\%$ and took $841.88$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3113/3333$ battles [this is $93.4\\%$ and took $825.55$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3130/3333$ battles [this is $93.91\\%$ and took $762.68$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45f5be2f",
   "metadata": {},
   "source": [
    "\n",
    "### MC_Control - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[MC_Control - Stochastic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1ycopro)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9062c6b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51d4e0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf54294",
   "metadata": {},
   "source": [
    "\n",
    "### MC_Control - Stochastic - Value Function $V^*(s)$ \n",
    "\n",
    "We defined two Value Functions to demonstrate the performance of our agents. \n",
    "\n",
    "The first one considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon_Pokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Venosaur\n",
    "\n",
    "$[10, 30]:$ Pikachu\n",
    "\n",
    "$[30, 50]:$ Taurus\n",
    "\n",
    "$[50, 70]:$ Sirfetch'd\n",
    "\n",
    "$[70, 90]:$ Blastoise\n",
    "\n",
    "$[90, 110]:$ Charizard\n",
    "\n",
    "### MC_Control - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjr8v3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5f1edbc1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51dc88>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1caec8ec",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjris0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b3fd9ef1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51dba8>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b522e4e4",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_1(s)$\n",
    "[MC_Control - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjrta3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "339a1f6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51db00>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/vfunctionQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_Index.pdf\", width=1000, height=600)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8caf2f5c",
   "metadata": {},
   "source": [
    "\n",
    "Our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### MC_Control - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjspyl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "66487bb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b51de80>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_00001_gamma_075_wining_3844_Stoc_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "168a2250",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjt0nt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a17d7050",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a160>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_0001_gamma_075_wining_407_Stoc_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4d9f7a2",
   "metadata": {},
   "source": [
    "### MC_Control - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjt8qk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0107d0c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a1d0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Stochastic/MCControlQ_20211029_n_battles_10000_N0_001_gamma_075_wining_4238_Stoc_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "054f92a2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "814b869e",
   "metadata": {},
   "source": [
    "## MC_Control - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-102/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### MC_Control - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control - Deterministic - Rewards (per step) - Training](https://prnt.sc/1yd5r9z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "97efc082",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a0b8>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31418cf4",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Win percentages (per step) - Training\n",
    "[MC_Control - Deterministic - Win percentages (per step) - Training](https://prnt.sc/1yd603w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0313e1e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a0f0>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fe1aa92",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1380/3333$ battles [this is $41.4\\%$ and took $485$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1359/3333$ battles [this is $40.77\\%$ and took $400.49$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1201/3333$ battles [this is $36.03\\%$ and took $400.07$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3052/3333$ battles [this is $91.57\\%$ and took $680.85$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3007/3333$ battles [this is $90.22\\%$ and took $825.87$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2997/3333$ battles [this is $89.92\\%$ and took $720$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a269b9f",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[MC_Control - Deterministic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yd64i7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "692080e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a2b0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "591d94be",
   "metadata": {},
   "source": [
    "### MC Control - Deterministic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon_Pokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Turtonator\n",
    "\n",
    "$[10, 30]:$ Lapras\n",
    "\n",
    "$[30, 50]:$ Armaldo\n",
    "\n",
    "$[50, 70]:$ Drapion\n",
    "\n",
    "$[70, 90]:$ Kabutops\n",
    "\n",
    "$[90, 110]:$ Falinks\n",
    "\n",
    "### MC Control  - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC Control - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjy6kh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4ff8098f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a320>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10ec5f16",
   "metadata": {},
   "source": [
    "### MC Control  - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC Control - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjyg6a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4d0e9bea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a278>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ed5d0e5",
   "metadata": {},
   "source": [
    "### MC Control  - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[MC Control - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjyrca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bfa6e2c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a390>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_Index.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c64118f",
   "metadata": {},
   "source": [
    "\n",
    "Our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### MC_Control - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjyzdd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5d3ad86e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a2e8>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_00001_gamma_075_wining_4039_Det_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c84e9cf",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjz331)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b092a859",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a748>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_0001_gamma_075_wining_3851_Det_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "842b0460",
   "metadata": {},
   "source": [
    "### MC_Control - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[MC_Control - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjz7i0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "bcc84a29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a4e0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/MCControl/Deterministic/MCControlQ_MCControlDet_20211104_n_battles_10000_N0_001_gamma_075_wining_3443_Det_noIndex_action.pdf\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc2c90d3",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "79d66dd1",
   "metadata": {},
   "source": [
    "# **Monte Carlo Control - Function Approximation**\n",
    "\n",
    "We implemented the Function Approximation with Monte Carlo Control First Visit. \n",
    "\n",
    "* We defined an array $w$ to store the weights of our approximator and a defaultDict $N(s,a)$ to store the number of occurences of action **a** in state **s**. $w$ is initialized randomly and $N(s,a)$ is initialized to zero;\n",
    " * $N(s,a)$ has the keys as the states **s**. The value of a key is an array of the len of action space (len: 9) and Each value of an array corresponds to an action in our action space;\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s,a]$;\n",
    "\n",
    "* In the first move of a turn, we choose a random action;\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s,:])$. We tested with 3 values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$};\n",
    "\n",
    "* We use the following functions to define our $\\epsilon$-greedy policy and to update the weights $w$, equivalent to: $x(s, a)$ and $\\hat{q}(s, a, w)$, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9e02235",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature vector\n",
    "def x(state, action):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.append(state, action)\n",
    "\n",
    "def q_approx_fn(state, action, w):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.dot(x(state, action), w)\n",
    "        \n",
    "def epsilon_greedy_policy(self, w, n0, N):\n",
    "    # epsilon-greedy policy\n",
    "    def policy_fn(state):\n",
    "        epsilon = n0 / (n0 + np.sum(N[str(state)]))\n",
    "        # let's get the greedy action. Ties must be broken arbitrarily\n",
    "        q_approx = np.array([q_approx_fn(state, action, w) for action in range(9)])\n",
    "        greedy_action = np.argmax(q_approx)\n",
    "        action_pick_probability = np.full(9, epsilon / 9)\n",
    "        action_pick_probability[greedy_action] += 1 - epsilon\n",
    "        return action_pick_probability\n",
    "\n",
    "    return policy_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31f77ee5",
   "metadata": {},
   "source": [
    "* In each turn, we collect our observations to define our state with the embed battle function, getting the string of 12 battle elements concatenated;\n",
    " \n",
    "* With the state, we select an action using our policy and send it to Showdown;\n",
    "\n",
    "* We store the state and selected action to append to an episode array in the next turn with the computed reward;\n",
    "\n",
    "* At the end of the battle, $w$ and $N$ are updated for the first time of each state seen in that battle:\n",
    " * $N(s,a)$ is increased by 1 for each action performed in a particular state;\n",
    " * The value of $w$ is summed to $\\alpha*\\delta*x(s,a)$;\n",
    " * $\\delta$ is calculated by $G(t) -$ q_approx_fn$(s, a, w)$;\n",
    " * The return $G(t)$ is given by $ \\sum \\limits _{i=0} ^{T-1} \\gamma^iR(t)$. We used $\\gamma = 0,75$;\n",
    " * Our policy are then updated with the new values of $w$ and $N(s,a)$;\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be49f2bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "rewards = [reward for state,action,reward in self.episode]\n",
    "states = [state for state,action,reward in self.episode]\n",
    "actions = [action for state,action,reward in self.episode]\n",
    "t_array = range(len(self.episode)+1)\n",
    "\n",
    "# Computing w and N\n",
    "for idx, state in enumerate(states):\n",
    "    action = actions[idx]\n",
    "    if state not in self.visited_states:\n",
    "        self.N[str(state)][action] += 1\n",
    "        returnGt = sum([reward*pow(self.gamma, t) for reward, t in zip(rewards[idx:], t_array[:(-idx-1)])]) \n",
    "\n",
    "        # step-size: 1./N[state][action]\n",
    "        delta = \\\n",
    "            returnGt - self.q_approx(state, action, self.w)\n",
    "            self.w += (1/self.N[str(state)][action])* delta * self.x(state, action)\n",
    "            \n",
    "        self.visited_states.append(state)\n",
    "                \n",
    "self.visited_states = []\n",
    "self.episode = []\n",
    "            \n",
    "# Define new policy with updated w and N\n",
    "self.policy = self.update_epsilon_greedy_policy(self.w, self.n0, self.N)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c15df8d",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greater base power. We trained for $10.000$ battles. At the end, we saved $w$ and $N(s,a)$ in json files to future use.\n",
    "\n",
    "* We validate our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d669753d",
   "metadata": {},
   "source": [
    "\n",
    "## MC_Control Function Approximation - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-93/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### MC_Control  Function Approximation - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control  Function Approximation - Stochastic - Rewards (per step) - Training ](https://prnt.sc/1ycovut)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "029a21d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control_FA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a710>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control_FA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1cc0e77",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Stochastic - Win percentages (per step) - Training](https://prnt.sc/1ycpfrm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4ead6720",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control_FA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a6a0>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control_FA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd24c393",
   "metadata": {},
   "source": [
    "\n",
    "### MC_Control Function Approximation - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2771/3333$ battles [this is $83.14\\%$ and took $521.1$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2796/3333$ battles [this is $83.89\\%$ and took $579.7$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2778/3333$ battles [this is $83.35\\%$ and took $807.72$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $854.3$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3303/3333$ battles [this is $99.1\\%$ and took $765.18$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3306/3333$ battles [this is $99.2\\%$ and took $654.11$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fe35059",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Stochastic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1ycpquh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4ce13254",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control_FA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a5f8>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control_FA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00cd493a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a094e172",
   "metadata": {},
   "source": [
    "## MC_Control Function Approximation - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-116/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### MC_Control Function Approximation - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[MC_Control Function Approximation - Deterministic - Rewards (per step) - Training ](https://prnt.sc/1yfmhdt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "165887ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_MC_Control_FA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a668>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_MC_Control_FA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0993a373",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Deterministic - Win percentages (per step) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Deterministic - Win percentages (per step) - Training](https://prnt.sc/1yfmlck)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "15c3ab1e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_MC_Control_FA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a518>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_MC_Control_FA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77f667cf",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2022/3333$ battles [this is $60.67\\%$ and took $456.87$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1982/3333$ battles [this is $59.47\\%$ and took $508.82$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2000/3333$ battles [this is $60\\%$ and took $561.88$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $532.47$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3306/3333$ battles [this is $99.19\\%$ and took $723.62$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3310/3333$ battles [this is $99.31\\%$ and took $539.96$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09711074",
   "metadata": {},
   "source": [
    "### MC_Control Function Approximation - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[MC_Control Function Approximation - Deterministic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yfmqa3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "91ef7925",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory MC_Control_FA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59ab00>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory MC_Control_FA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4f57ac9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "763a4283",
   "metadata": {},
   "source": [
    "# **QLearning**\n",
    "\n",
    "* We have defined 2 defaultDicts, $Q(s,a)$ and $N(s,a)$, to store the value function and the number of occurences of action **a** in state **s**, respectively. Both of them initialized to zero. Each dict has key **s** (state) and respective value **a** (action), which is an array of length 9 (size of our action space);\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s, a]$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've tested using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$}.\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9152929",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(self, state):\n",
    "    epsilon = self.n0 / (self.n0 + np.sum(self.N[state]))\n",
    "    greedy_action = np.random.choice(np.where(self.Q[state] == self.Q[state].max())[0])\n",
    "    action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "    action_pick_probability[greedy_action] += 1 - epsilon\n",
    "    return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3242ae98",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements. The concatenated elements are converted to a string that can be used as a key to index our dictionaries $Q(s,a)$ and $N(s,a)$.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha=1/N(s, a)$;\n",
    " - calculate $Q(s, a)$ as: $Q(s, a) = Q(s, a) + \\alpha * (R + \\gamma * \\max_{a'}Q(s', a') - Q(s, a))$;\n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abef51e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# observe R, S'\n",
    "reward = self.compute_reward(battle)\n",
    "next_state = self.embed_battle(battle)\n",
    "# Q-learning\n",
    "self.N[self.state][self.action] += 1\n",
    "alpha = 1.0 / self.N[self.state][self.action]\n",
    "self.Q[self.state][self.action] += \\\n",
    "    alpha * (reward + self.gamma * np.max(self.Q[next_state]) - self.Q[self.state][self.action])\n",
    "# S <- S'\n",
    "self.state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9196f90e",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greater base power. We trained for $10.000$ battles. At the end, we saved $Q$ and $N$ in json files for future use.\n",
    "\n",
    "* We validated our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eed0c0c",
   "metadata": {},
   "source": [
    "## Q-Learning - Stochastic - Results\n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-95/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### Q-Learning - Stochastic - Rewards (per step) - Training\n",
    "\n",
    "[Q-Learning - Stochastic - Rewards (per step) - Training](https://prnt.sc/1ycpxci)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3fb4e5d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearning.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59aba8>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearning.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c91de7c",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning - Stochastic - Win percentages (per step) - Training](https://prnt.sc/1ycq2uv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "60adb667",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearning.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59aa20>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearning.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1d1987",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1510/3333$ battles [this is $45.3\\%$ and took $845.64$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1156/3333$ battles [this is $34.68\\%$ and took $1038.49$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1503/3333$ battles [this is $45.09\\%$ and took $748.07$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3123/3333$ battles [this is $93.7\\%$ and took $957.94$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2859/3333$ battles [this is $85.78\\%$ and took $912.79$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3021/3333$ battles [this is $90.64\\%$ and took $850.87$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "168cc6bd",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning - Stochastic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1ycqa4f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6372fd1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory QLearning.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59aac8>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory QLearning.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a30912",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon_Pokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Venosaur\n",
    "\n",
    "$[10, 30]:$ Pikachu\n",
    "\n",
    "$[30, 50]:$ Taurus\n",
    "\n",
    "$[50, 70]:$ Sirfetch'd\n",
    "\n",
    "$[70, 90]:$ Blastoise\n",
    "\n",
    "$[90, 110]:$ Charizard\n",
    "\n",
    "### Q-Learning - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yju2kw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "78b4b97e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59ab70>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec2e48bf",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjub8p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f64275b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a898>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cac89375",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjuk44)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "db070f9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a5c0>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59e6e5c0",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### Q-Learning - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjv5mt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "9058f0fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b59a470>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_0001_0_75_43_1_json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2ae388f",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjvck8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8d2db300",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7198>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_001_0_75_32_33_json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee9c36c3",
   "metadata": {},
   "source": [
    "### Q-Learning - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjvkob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e143fd47",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7048>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/stochastic/Q_2021-11-04_10000_0_01_0_75_39_89_json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f0ca8d8",
   "metadata": {},
   "source": [
    "## Q-Learning - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-119/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "\n",
    "### Q-Learning - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning - Deterministic - Rewards (per step) - Training ](https://prnt.sc/1yk6sw1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4b09d24f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearning_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7208>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearning_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16e7a4c6",
   "metadata": {},
   "source": [
    "### Q-Learning  - Deterministic - Win percentages (per step) - Training\n",
    "[Q-Learning  - Deterministic - Win percentages (per step) - Training](https://prnt.sc/1yk6ve0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2e8fd668",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearning_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7240>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearning_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c14413",
   "metadata": {},
   "source": [
    "### Q-Learning  - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1446/3333$ battles [this is $43.38\\%$ and took $572.78$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1077/3333$ battles [this is $32.31\\%$ and took $444.55$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1461/3333$ battles [this is $43.83\\%$ and took $582.1$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2801/3333$ battles [this is $84.04\\%$ and took $698.99$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2800/3333$ battles [this is $84.01\\%$ and took $794.75$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2934/3333$ battles [this is $88.03\\%$ and took $601.89$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe29ad8",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning - Deterministic - Memory and CPU usage  (absolute time) - Training](https://prnt.sc/1yk6yeo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "b2a555ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/QLearningMemory _ CPU usage_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a72e8>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/QLearningMemory _ CPU usage_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc1ee1c8",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon_Pokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Turtonator\n",
    "\n",
    "$[10, 30]:$ Lapras\n",
    "\n",
    "$[30, 50]:$ Armaldo\n",
    "\n",
    "$[50, 70]:$ Drapion\n",
    "\n",
    "$[70, 90]:$ Kabutops\n",
    "\n",
    "$[90, 110]:$ Falinks\n",
    "\n",
    "### Q-Learning - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjw2yb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ac23c990",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a70b8>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34d5ec8",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjwb7c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "92577e49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7278>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ff6143f",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjwjse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "219ad172",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_Index.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a74a8>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_Index.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e73d4b6",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### Q-Learning - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75](https://prnt.sc/1yjwvzb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "cbad256c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7320>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_0001_0_75_41_54_json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd6c2746",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75](https://prnt.sc/1yjx4zb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "fbc48db8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7358>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_001_0_75_30_86_json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf9109d6",
   "metadata": {},
   "source": [
    "### Q-Learning - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[Q-Learning - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75](https://prnt.sc/1yjxbxe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ca3427e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7470>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/QLearning/deterministic/Q_2021-11-05_10000_0_01_0_75_37_67_json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a56cf4f",
   "metadata": {},
   "source": [
    "# **Q-Learning - Function Approximation**\n",
    "\n",
    "* To implement function approximation for Q-Learning, we have first defined array $w$, to store the weights of our linear approximator, and defaultDict $N(s,a)$, to store the number of occurences of action **a** in state **s**. $w$ is initialized randomly and $N(s,a)$ is initialized to zero. $N(s,a)$ has key **s** (state) and respective value **a** (action), which is an array of length 9 (size of our action space).\n",
    "\n",
    "* We used a step-size of $\\alpha = \\alpha_0 / N[s, a]$, where $\\alpha_0 = 0.01$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've tested using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$}.\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d9c2fc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(self, state, w):\n",
    "    epsilon = self.n0 / (self.n0 + np.sum(self.N[str(state)]))\n",
    "    # let's get the greedy action. Ties must be broken arbitrarily\n",
    "    q_approx = np.array([self.q_approx(state, action, w) for action in range(N_OUR_ACTIONS)])\n",
    "    greedy_action = np.random.choice(np.where(q_approx == q_approx.max())[0])\n",
    "    action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "    action_pick_probability[greedy_action] += 1 - epsilon\n",
    "    return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60538675",
   "metadata": {},
   "source": [
    "For the function approximation part, we defined the following helpers functions, equivalent to: $x(s, a)$, $\\hat{q}(s, a, w)$ and $\\max_{a}\\hat{q}(s, a, w)$, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbc3af7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def x(state, action):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.append(state, action)\n",
    "\n",
    "def q_approx(self, state, action, w):\n",
    "    state = np.array(state).astype(float)\n",
    "    return np.dot(self.x(state, action), w)\n",
    "\n",
    "def max_q_approx(self, state, w):\n",
    "    state = np.array(state).astype(float)\n",
    "    return max(np.array([self.q_approx(state, action, w) for action in range(N_OUR_ACTIONS)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a77daf3",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s, w)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action **a**;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha = \\alpha_0 / N[s, a]$, where $\\alpha_0 = 0.01$;\n",
    " - calculate $\\delta$ as: $\\delta = R + \\gamma * \\max_{a'}\\hat{q}(s, a', w) - \\hat{q}(s, a, w)$;\n",
    " - then we update $w$ as: $w = w + \\alpha * \\delta * x(s, a)$;\n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s, w)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee5e9c1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "if self.state is not None:\n",
    "    # observe R, S'\n",
    "    reward = self.compute_reward(battle)\n",
    "    next_state = self.embed_battle(battle)\n",
    "    # Q-learning\n",
    "    self.N[str(self.state)][self.action] += 1\n",
    "    alpha = self.alpha0 / self.N[str(self.state)][self.action]\n",
    "    delta = \\\n",
    "        reward + self.gamma * self.max_q_approx(next_state, self.w) - self.q_approx(self.state, self.action, self.w)\n",
    "    self.w += alpha * delta * self.x(self.state, self.action)\n",
    "    # S <- S'\n",
    "    self.state = next_state\n",
    "else:\n",
    "    # S first initialization\n",
    "    self.state = self.embed_battle(battle)\n",
    "\n",
    "# Choose A from S using epsilon-greedy policy\n",
    "self.action = self.pi(self.state, self.w)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18b7a867",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greater base power. We trained for $10.000$ battles. At the end, we saved $Q$ and $w$ in json files for future use.\n",
    "\n",
    "* We validated our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "792edefa",
   "metadata": {},
   "source": [
    "\n",
    "## Q-Learning  Function Approximation - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-126/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### Q-Learning  Function Approximation - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Rewards (per step) - Training ](https://prnt.sc/1yk78w0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "58f87821",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearningFA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a75f8>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearningFA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e551b244",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Win percentages (per step) - Training](https://prnt.sc/1yk7c3x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "e98b8d1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearningFA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a76a0>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearningFA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c70de88",
   "metadata": {},
   "source": [
    "\n",
    "### Q-Learning Function Approximation - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $2799/3333$ battles [this is $83.98\\%$ and took $934.83$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $2812/3333$ battles [this is $84.37\\%$ and took $664.67$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $2772/3333$ battles [this is $83.17\\%$ and took $869.2$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $854.3$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3303/3333$ battles [this is $99.1\\%$ and took $765.18$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3306/3333$ battles [this is $99.2\\%$ and took $654.11$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8434891c",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yk7e50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "fe47350f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory QLearningFA.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7748>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory QLearningFA.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11868b4c",
   "metadata": {},
   "source": [
    "## Q-Learning Function Approximation - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-127/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### Q-Learning Function Approximation - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Rewards (per step) - Training ](https://prnt.sc/1yl3lgl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "091a50c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_QLearningFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7710>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_QLearningFA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "579838bd",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Deterministic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Win percentages (per step) - Training](https://prnt.sc/1yl3w4c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "d717d277",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_QLearningFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a7940>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_QLearningFA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "247b10f1",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $1925/3333$ battles [this is $57.76\\%$ and took $701.92$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $1973/3333$ battles [this is $59.20\\%$ and took $895.72$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $1974/3333$ battles [this is $59.23\\%$ and took $508.06$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$ and $\\gamma=0.75$ won $3312/3333$ battles [this is $99.37\\%$ and took $700.87$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$ and $\\gamma=0.75$ won $3308/3333$ battles [this is $99.25\\%$ and took $914.08$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$ and $\\gamma=0.75$ won $3313/3333$ battles [this is $99.40\\%$ and took $867.76$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cfb28b5",
   "metadata": {},
   "source": [
    "### Q-Learning Function Approximation - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yl4cqi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "7888d8c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory QLearningFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x2316b5a75c0>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory QLearningFA_deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac168a6e",
   "metadata": {},
   "source": [
    "# **SARSA($\\lambda$)**\n",
    "\n",
    "* We defined 3 defaultDicts, $Q(s,a)$, $N(s,a)$ and $E(s,a)$, to store the value function, the number of occurences, and the eligibility trace of action **a** in state **s**, respectively. Both of them initialized to zero;\n",
    " * each dict has key **s** (state) and respective value **a** (action), which is an array of length 9 (size of our action space);\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s, a]$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've trained using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$} and 6 different values for $\\lambda: $ {$0, 0.2, 0.4, 0.6, 0.8, 1$}.\n",
    "\n",
    "* We've validated using the best results obtained with the 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$;\n",
    "\n",
    "* We use the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a589d832",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(self, state):\n",
    "    epsilon = self.n0 / (self.n0 + np.sum(self.N[state]))\n",
    "    # let's get the greedy action. Ties must be broken arbitrarily\n",
    "    greedy_action = np.random.choice(np.where(self.Q[state] == self.Q[state].max())[0])\n",
    "    action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "    action_pick_probability[greedy_action] += 1 - epsilon\n",
    "    return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d055ad6c",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements. The concatenated elements are converted to a string that can be used as a key to index our dictionaries $Q(s,a)$, $N(s,a)$ and $E(s,a)$.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha=1/N(s, a)$;\n",
    " - calculate $\\delta$ as: $\\delta = R + \\gamma * Q(s', a') - Q(s, a)$;\n",
    " - calculate $E(s, a)$ as: $E(s, a) = E(s, a) + 1$;\n",
    " - calculate $Q(s, a)$ as: $Q(s, a) = Q(s, a) + \\sum \\limits _{s=s_i} ^{s_n} \\alpha * \\delta * E(s, a)$ for each $s \\in S$;\n",
    " - update $E(s, a)$ as: $E(s, a) = E(s, a) * $ $\\prod \\limits _{s=s_i} ^{s_n} \\lambda * \\gamma$ for each $s \\in S$;\n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s)$.\n",
    " - Our policy are then updated with the new values of $Q(s,a)$ and $N(s,a)$;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6961dc6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "if self.state is not None:\n",
    "    # observe R, next_state and next_action\n",
    "    reward = self.compute_reward(battle)\n",
    "    next_state = self.embed_battle(battle)\n",
    "    next_action = self.choose_action(next_state)\n",
    "            \n",
    "    #alpha\n",
    "    self.N[self.state][self.action] += 1\n",
    "    alpha = 1.0 / self.N[self.state][self.action]\n",
    "            \n",
    "    ########################  Calculate Lambda SARSA\n",
    "    delta = reward + self.gamma*self.Q[next_state][next_action] - self.Q[self.state][self.action]\n",
    "    self.E[self.state][self.action] += 1\n",
    "    for s, _ in self.Q.items():\n",
    "        self.Q[s][:] += alpha * delta * self.E[s][:]\n",
    "        self.E[s][:] *= self.lambda_ * self.gamma\n",
    "            \n",
    "    # S <- S'  A <- A' \n",
    "    self.state = next_state\n",
    "    self.action = next_action\n",
    "    # Update the policy\n",
    "    self.policy = self.update_epsilon_greedy_policy(self.Q, self.n0, self.N)\n",
    "            \n",
    "else:\n",
    "    # S first initialization\n",
    "    self.state = self.embed_battle(battle)\n",
    "    # Choose action\n",
    "    self.action = self.choose_action(self.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64470c37",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greater base power. We trained for $10.000$ battles. At the end, we saved $Q(s,a)$ and $N(s,a)$ in json files for future use.\n",
    "\n",
    "* We validated our solution with the best results against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad80a6b",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) - Stochastic - Results\n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/mauricioplopes/poke-env/e/POK-41/dashboard/Compare-results-de35633f-644c-4bdd-8688-573933ef0796.\n",
    "\n",
    "### SARSA($\\lambda$) - Stochastic - Rewards (per step) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Rewards (per step) - Training](https://prnt.sc/1yoie7d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b889fc3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsa_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba26646f",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$)  - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Win percentages (per step) - Training](https://prnt.sc/1yoi9oa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "836d5401",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_sarsa_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d235841e",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1872/3333$ battles [this is $56.17\\%$ and took $468.37$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0$ won $1804/3333$ battles [this is $54.13\\%$ and took $481.42$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0$ won $1802/3333$ battles [this is $54.07\\%$ and took $498.52$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $3151/3333$ battles [this is $94.54\\%$ and took $541.27$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0$ won $2989/3333$ battles [this is $89.68\\%$ and took $664.85$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0$ won $2953/3333$ battles [this is $88.6\\%$ and took $693.19$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e82f1b80",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yoims2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d6ba3bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory sarsa_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96c8eb83",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*(s)$ \n",
    "\n",
    "We've ploted just the best results obtained with SARSA($\\lambda$).\n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon_Pokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Venosaur\n",
    "\n",
    "$[10, 30]:$ Pikachu\n",
    "\n",
    "$[30, 50]:$ Taurus\n",
    "\n",
    "$[50, 70]:$ Sirfetch'd\n",
    "\n",
    "$[70, 90]:$ Blastoise\n",
    "\n",
    "$[90, 110]:$ Charizard\n",
    "\n",
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75 - lambda:0](https://prnt.sc/1ykbfp8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ab1c094",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-03_10000_0.0001_0.75_0.00_53.62.jsonindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce76e510",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75 - lambda:0](https://prnt.sc/1ykbkmi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0b61b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-03_10000_0.001_0.75_0.00_51.02.jsonindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec8fdd94",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_1(s)$\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75 - lambda:0.2](https://prnt.sc/1ykbovc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d455f6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SarsaLambda_2021-11-04_10000_0.01_0.75_0.20_46.65.jsonindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee497ba",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75 - lambda:0](https://prnt.sc/1ykbsu1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b6aa9c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-03_10000_0.0001_0.75_0.00_53.62.json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab97ccc",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75 - lambda:0](https://prnt.sc/1ykbwai)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4df5b0ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-03_10000_0.001_0.75_0.00_51.02.json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfdd0e26",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Stochastic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Stochastic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75 - lambda:0.2](https://prnt.sc/1ykc04i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ab61b94",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/vfunction/SARSA/stochastic/SARSASarsaLambda_2021-11-04_10000_0.01_0.75_0.20_46.65.json_Stoc_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8b4bcaf",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-123/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### SARSA($\\lambda$) - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Rewards (per step) - Training ](https://prnt.sc/1yk8g1l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98df1753",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsa_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ad45f23",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$)  - Deterministic - Win percentages (per step) - Training\n",
    "[SARSA($\\lambda$)  - Deterministic - Win percentages (per step) - Training](https://prnt.sc/1yk8831)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4b71b2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_sarsa_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ab883f",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$)  - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1739/3333$ battles [this is $52.18\\%$ and took $521.7$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1804/3333$ battles [this is $54.13\\%$ and took $515.35$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0.2$ won $1749/3333$ battles [this is $52.48\\%$ and took $577.84$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $2972/3333$ battles [this is $89.17\\%$ and took $690.43$ seconds]\n",
    "\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and $\\lambda=0.2$ won $2967/3333$ battles [this is $89.02\\%$ and took $669$ seconds]\n",
    "\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and $\\lambda=0.2$  won $3008/3333$ battles [this is $90.25\\%$ and took $692.09$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb208d69",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Memory and CPU usage  (absolute time) - Training](https://prnt.sc/1yk8ka1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f573346",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory sarsa_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cc0a328",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) - Deterministic - Value Function $V^*(s)$ \n",
    "\n",
    "As previous defined, our first value function considers in its Y axis the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent. On its X axis the function uses the Index of our active Pokémon in the field multiplided by 20 in a sum with the sum of the base powers moves multiplied by their respective multipliers.\n",
    "\n",
    "$y = $ our_remaining_Pokémon - opponents_remaining_Pokémon\n",
    "\n",
    "$x = $ index_our_activePokémon_Pokémon$*20 + \\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "We observed that the sum of the base powers moves multiplied by their respective multipliers is always a value between -10 and 10. Therefore, we can divide the following ranges to each of our Pokémon in X axis:\n",
    "\n",
    "$[-10, 10]:$ Turtonator\n",
    "\n",
    "$[10, 30]:$ Lapras\n",
    "\n",
    "$[30, 50]:$ Armaldo\n",
    "\n",
    "$[50, 70]:$ Drapion\n",
    "\n",
    "$[70, 90]:$ Kabutops\n",
    "\n",
    "$[90, 110]:$ Falinks\n",
    "\n",
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.0001 - gamma: 0.75 - lambda: 0.2](https://imgur.com/rlYy5yV)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "42b818d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.jsonDetindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x195290af898>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.jsonDetindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10e3517d",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.001 - gamma: 0.75 - lambda 0.2](https://imgur.com/S8MmNW0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "84e595ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.jsonDetindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x195290af5f8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.jsonDetindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceef1cc4",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_1(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 1 - $V^*_1(s)$ - With Pokémon Index - N0=0.01 - gamma: 0.75 - lambda: 0.2](https://imgur.com/PPt78on)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e755209b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.01_0.75_0.2_44.42.jsonDetindex.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f048>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SarsaLambda_2021-11-06_10000_0.01_0.75_0.2_44.42.jsonDetindex.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "215db1ff",
   "metadata": {},
   "source": [
    "As previous defined, our second value function considers in its Y axis the best action index for the state.\n",
    "\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    " \n",
    "On its X axis the function embeds our state by multiplying the difference between the number of our remaining Pokémon and the number of remaining Pokémon of the opponent with the sum of the base powers moves multiplied by their respective multipliers. The Index of our active Pokémon in the field is not considered.\n",
    "\n",
    "$y = $ best action index\n",
    "\n",
    "$x = $ (our_remaining_Pokémon - opponents_remaining_Pokémon) $\\sum \\limits _{i=1} ^{4}$ move_base_power$[i]*$move_damage_multiplier$[i]$ \n",
    "\n",
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.0001 - gamma: 0.75 - lambda: 0.2](https://imgur.com/bUg9vH1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "599b4801",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f0f0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-05_10000_0.0001_0.75_0.2_45.13.json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3484f7b5",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.001 - gamma: 0.75 - lambda: 0.2](https://imgur.com/hGJwUyQ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "783e534f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f240>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d53e7ff8",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) - Deterministic - Value Function $V^*_2(s)$\n",
    "\n",
    "[SARSA($\\lambda$) - Deterministic - Value Function 2 - $V^*_2(s)$ - With Best Action Index - N0=0.01 - gamma: 0.75 - lambda: 0.2](https://imgur.com/qd8FLvR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7c0626e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f160>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/vfunction/SARSA/deterministic/SARSASarsaLambda_2021-11-06_10000_0.001_0.75_0.2_48.05.json_Det_noIndex_action.pdf\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62157c4f",
   "metadata": {},
   "source": [
    "# **SARSA($\\lambda$) - Function Approximation**\n",
    "\n",
    "    \n",
    "We implemented the Function Approximation with SARSA($\\lambda$). \n",
    "\n",
    "* We defined an array $w$ to store the weights of our approximator, an array $e$ to store the eligibility trace and a defaultDict $N(s,a)$ to store the number of occurences of action **a** in state **s**. $w$ is initialized randomly, $e$ and $N(s,a)$ are initialized to zero;\n",
    " * $N(s,a)$ has the keys as the states **s**. The value of a key is an array of the len of action space (len: 9) and Each value of an array corresponds to an action in our action space;\n",
    "\n",
    "* We used a step-size of $\\alpha = 1/N[s, a]$.\n",
    "\n",
    "* We used discount factor $\\gamma = 0.75$.\n",
    "\n",
    "* We used an $\\epsilon$-greedy exploration strategy with $\\epsilon = N0/(N0+N[s, :])$. We've trained using 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$ and $10^{-2}$} and 6 different values for $\\lambda: $ {$0, 0.2, 0.4, 0.6, 0.8, 1$}.\n",
    "\n",
    "* We've validated using the best results obtained with the 3 different values for $N0: $ {$10^{-4}$, $10^{-3}$;\n",
    "\n",
    "\n",
    "* We use the following functions to define our $\\epsilon$-greedy policy and to update the weights $w$, equivalent to: $x(s, a)$ and $\\hat{q}(s, a, w)$, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05b4d06c",
   "metadata": {},
   "outputs": [],
   "source": [
    "    def x(state, action):\n",
    "        state = np.array(state).astype(float)\n",
    "        return np.append(state, action)\n",
    "    \n",
    "    # q^(S, A, W)\n",
    "    def q_approx(self, state, action, w):\n",
    "        state = np.array(state).astype(float)\n",
    "        return np.dot(self.x(state, action), w)\n",
    "\n",
    "    # epsilon-greedy policy\n",
    "    #Function to choose the next action\n",
    "    def policy_fn(self, state, w):\n",
    "        epsilon = self.n0 / (self.n0 + np.sum(self.N[str(state)]))\n",
    "        # let's get the greedy action. Ties must be broken arbitrarily\n",
    "        q_approx = np.array([self.q_approx(state, action, w) for action in range(N_OUR_ACTIONS)])\n",
    "        greedy_action = np.argmax(np.where(q_approx == q_approx.max())[0])\n",
    "        action_pick_probability = np.full(N_OUR_ACTIONS, epsilon / N_OUR_ACTIONS)\n",
    "        action_pick_probability[greedy_action] += 1 - epsilon\n",
    "        return np.random.choice(ALL_OUR_ACTIONS, p=action_pick_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e268a4d8",
   "metadata": {},
   "source": [
    "* At every of our turns (steps) in a battle (episode), we call the embed battle function to collect our observations, then we define our state by concatenating 12 battle elements. The concatenated elements are converted to a string that can be used as a key to index our dictionary $N(s,a)$.\n",
    "\n",
    "* In the very first turn of a battle, we first obtain a random state by calling the embed battle function. Then, we choose our next action by following policy $\\pi$: $a = \\pi(s)$.\n",
    "\n",
    "* In each of the subsequent turns until the end of the battle, we do the following:\n",
    " - get the reward **R** and the next state **s'** from the previously executed action;\n",
    " - increase $N(s, a)$ by 1 for each action **a** executed while in state **s**;\n",
    " - calculate $\\alpha$ as: $\\alpha=0.01/N(s, a)$;\n",
    " - calculate $\\delta$ as: $\\delta = R + \\gamma *$ q_approx($s', a', w$) -  q_approx($s, a, w$);\n",
    " - calculate $e$ as: $e = (e + 1)$; \n",
    " - calculate $w$ as: $w = w + \\alpha * \\delta * e$;\n",
    " - update $e$ as: $e = e * \\lambda * \\gamma$; \n",
    " - next we update the current state as: **s** = **s'**\n",
    " - finally, we choose our next action as: $a = \\pi(s)$.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76e9890e",
   "metadata": {},
   "outputs": [],
   "source": [
    "        if self.state is not None:\n",
    "            # observe R, next_state and next_action\n",
    "            reward = self.compute_reward(battle)\n",
    "            next_state = self.embed_battle(battle)\n",
    "            next_action = self.policy_fn(next_state, self.w)\n",
    "            delta = reward\n",
    "            #alpha\n",
    "            self.N[str(self.state)][self.action] += 1\n",
    "            alpha = 0.01 / self.N[str(self.state)][self.action]\n",
    "            \n",
    "             #Accumulative\n",
    "            self.e += 1\n",
    "            delta = reward + self.gamma * self.q_approx(next_state, next_action, self.w) - self.q_approx(self.state, self.action, self.w)\n",
    "            \n",
    "            # update theta\n",
    "            self.w += alpha * delta * self.e\n",
    "            \n",
    "            #update e\n",
    "            self.e *= self.gamma * self.lambda_\n",
    "           \n",
    "            # S <- S'\n",
    "            self.state = next_state\n",
    "            # Choose action\n",
    "            self.action = self.policy_fn(self.state, self.w)\n",
    "            \n",
    "        else:\n",
    "            # S first initialization\n",
    "            self.state = self.embed_battle(battle)\n",
    "            self.action = self.policy_fn(self.state, self.w)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b738858f",
   "metadata": {},
   "source": [
    "* We trained our agent against a MaxDamagePlayer, that always selects the move with the greater base power. We trained for $10.000$ battles. At the end, we saved $w$ and $N(s,a)$ in json files to future use.\n",
    "\n",
    "* We validate our solution against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03ac4581",
   "metadata": {},
   "source": [
    "\n",
    "## SARSA($\\lambda$)  Function Approximation - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-133/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e. \n",
    "\n",
    "### SARSA($\\lambda$)  Function Approximation - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Stochastic - Rewards (per step) - Training ](https://prnt.sc/1yrh0m7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c28c336a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsaFA_stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f358>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsaFA_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34f5c2fa",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Stochastic - Win percentages (per step) - Training\n",
    "\n",
    "[Q-Learning Function Approximation - Stochastic - Win percentages (per step) - Training](https://prnt.sc/1yrh85b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b32cb403",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_sarsaFA_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31f78d41",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Stochastic - Win percentages - Validation\n",
    "\n",
    "We've showed just the better results with SARSA($\\lambda$) Function Approximation.\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=1$ won $2063/3333$ battles [this is $83.53\\%$ and took $298.65$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.6$ won $2784/3333$ battles [this is $83.2\\%$ and took $290.17$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=1$ won $2788/3333$ battles [this is $83.65\\%$ and took $307.59$ seconds]\n",
    "\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=1$ won $1914/3333$ battles [this is $57.43\\%$ and took $1288.88$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.6$ won $1912/3333$ battles [this is $57.37\\%$ and took $1272.98$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=1$ won $/3333$ battles [this is $\\%$ and took $$ seconds]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff49b6b6",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Stochastic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Stochastic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yrhea1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19ed2caf",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsaFA_stochastic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69033c6",
   "metadata": {},
   "source": [
    "## SARSA($\\lambda$) Function Approximation - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-136/dashboard/Compara-es-464d2bfc-e396-4bbd-b8bc-3cf9fe90b55e.\n",
    "\n",
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[Q-Learning Function Approximation - Deterministic - Rewards (per step) - Training ](https://prnt.sc/1yrf29u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f70d1517",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_sarsaFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f2e8>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_sarsaFA_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "427d69aa",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Win percentages (per step) - Training\n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Deterministic - Win percentages (per step) - Training](https://prnt.sc/1yrfb4b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8bdf6863",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_sarsaFA_deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f1d0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_sarsaFA_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e3d6d9",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Win percentages - Validation\n",
    "\n",
    "We've showed just the better results with SARSA($\\lambda$) Function Approximation.\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $2063/3333$ battles [this is $61.9\\%$ and took $513.66$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.4$ won $2027/3333$ battles [this is $60.82\\%$ and took $229.79$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $2030/3333$ battles [this is $60.91\\%$ and took $739$ seconds]\n",
    "\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player with $N0=0.0001$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $3308/3333$ battles [this is $98.95\\%$ and took $222.79$ seconds]\n",
    "* Player with $N0=0.001$, $\\gamma=0.75$ and  $\\lambda=0.4$ won $3302/3333$ battles [this is $98.95\\%$ and took $1296.52$ seconds]\n",
    "* Player with $N0=0.01$, $\\gamma=0.75$ and  $\\lambda=0.8$ won $3316/3333$ battles [this is $99.49\\%$ and took $222.54$ seconds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63c4444b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1c337d72",
   "metadata": {},
   "source": [
    "### SARSA($\\lambda$) Function Approximation - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[SARSA($\\lambda$) Function Approximation - Deterministic - Memory and CPU usage (absolute time) - Training](https://prnt.sc/1yrfi28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d1789c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory sarsaFA_deterministic.png\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30b4f167",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c0eb5746",
   "metadata": {},
   "source": [
    "# Related work\n",
    "\n",
    "Given the stochasticity of the Pokémon game, the environment proves to be a good alternative to validate more robust reinforcement learning systems. However, in recent years, few reinforcement learning jobs have been willing to use Pokémon battles to validate their models.\n",
    "\n",
    "Regarding tabular methods, the work of [Rill-García, R. 2018](https://ccc.inaoep.mx/~esucar/Clases-mgp/Proyectos/2018/reinforcement-learning-turn%20%281%29.pdf) can be mentioned. The author proposes the use of the Q-Learning algorithm. The author defines as states only the index of his active Pokémon and the opponent's. However, 301 Pokémon are considered to choose from, among the most used in competitive battles. Actions were also defined from 601 moves most used in competitive battles, plus one, referring to an action that cannot be performed. Rewards were set based on the Pokémon that passed out (their or the opponent's), the percentages of damage produced or taken, and based on an \"unable to act\" rating. The author used constant values of $\\alpha = 0.1$ and $\\gamma = 0.8$. The author used 10 different teams for validation and reported an average of $58\\%$ wins against a RandomPlayer (player who selects random actions at each turn).\n",
    "\n",
    "Another tabular method that can be cited is the work of [Kalose, A et. al. 2018](https://web.stanford.edu/class/aa228/reports/2018/final151.pdf). The authors also propose the use of the Q-Learning algorithm, however, it uses two different policies, an $\\epsilon-greedy$ and a Softmax. The authors define as states the Health Points of their Pokémon and the opponent's and the types of their Pokémon and the opponent's. The defined actions are the 4 possible moves of each Pokemon. The authors did not establish how the rewards were defined, or the alpha and gamma values used. The authors reported an average of $60\\%$ winnings against a RandomPlayer (player who selects random actions each turn) using $\\epsilon-greedy$ policy and $68\\%$ winnings against a RandomPlayer using Softmax policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d3876988",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"1000\"\n",
       "            src=\"images/report/comparisons_val.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1952918f630>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/comparisons_val.png\", width=1000, height=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59512049",
   "metadata": {},
   "source": [
    "# Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "722f094a",
   "metadata": {},
   "source": [
    "## Comparisons\n",
    "\n",
    "One table with performance comparisons in Validation between our methods and the methods proposed by Rill-Garcia and Kalose et. al. are showed bellow and in [this figure](https://prnt.sc/1yrouxy).\n",
    "\n",
    "The method with the best performance against both Players (MaxDamagePlayer and RandomPlayer) was Q-Learning Function Approximation in the Stochastic environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18d1bad1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "99154317",
   "metadata": {},
   "source": [
    "# Video"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "600cfd7f",
   "metadata": {},
   "source": [
    "# Contributions\n",
    "\n",
    "## Methods\n",
    "\n",
    "### Monte Carlo Control\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### Monte Carlo Control Function Approximation\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### Q-Learning\n",
    "* **Code development**: Henrique;\n",
    "* **Code review**: Bruno, Leonardo, Maurício;\n",
    "* **Experiments**: Henrique, Leonardo.\n",
    "\n",
    "### Q-Learning Function Approximation\n",
    "* **Code development**: Henrique;\n",
    "* **Code review**: Bruno, Leonardo, Maurício;\n",
    "* **Experiments**: Henrique.\n",
    "\n",
    "### SARSA($\\lambda$)\n",
    "* **Code development**: Bruno;\n",
    "* **Code review**: Leonardo, Maurício;\n",
    "* **Experiments**: Bruno, Leonardo, Maurício.\n",
    "\n",
    "### SARSA($\\lambda$) Function Approximation\n",
    "* **Code development**: Bruno;\n",
    "* **Code review**: Leonardo, Maurício;\n",
    "* **Experiments**: Bruno, Leonardo, Maurício.\n",
    "\n",
    "## Report\n",
    "* **Writting**: Henrique, Leonardo;\n",
    "* **Review**: Bruno, Maurício.\n",
    "\n",
    "## Video\n",
    "* **Editing**: Leonardo;\n",
    "* **Review**: Bruno, Henrique, Maurício.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38aa37a9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}