优化方法.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 梯度下降算法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = '3'\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "opt =  tf.keras.optimizers.SGD(learning_rate=0.1)\n",
    "\n",
    "var = tf.Variable(1.0)\n",
    "\n",
    "loss = lambda:(var**2)/2.0\n",
    "\n",
    "opt.minimize(loss, [var]).numpy()\n",
    "\n",
    "var.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 梯度下降的划分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 术语\n",
    "* Epoch: 使用训练集的全部数据进行一次完整训练，称为**一代训练**\n",
    "* Batch: 使用训练集中的一小部分样本对模型权重进行一次反向传播的参数更新，这一小部分样本称为**一批数据**\n",
    "* Iteration: 使用一个Batch数据对模型进行一次参数更新的过程，称为**一次训练**\n",
    "\n",
    "### 梯度下降的划分\n",
    "* BGD: 每次迭代需要计算每个样本上损失函数的梯度并求和\n",
    "* SGD: 每次迭代只采样一个样本，计算损失函数在这个样本上的梯度并更新参数\n",
    "* Mini-Batch: 每次迭代时随机选取一小部分训练样本来计算梯度并更新参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 前向传播\n",
    "## 反向传播"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 梯度下降的优化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 动量法（Momentum）\n",
    "动量梯度下降计算梯度的指数加权平均数，并利用该值来更新参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt= tf.keras.optimizers.SGD(lr=0.1, momentum=0.9)\n",
    "\n",
    "var = tf.Variable(1.0)\n",
    "val0 = var.value()\n",
    "\n",
    "loss = lambda:(var**2)/2.0\n",
    "\n",
    "opt.minimize(loss,[var]).numpy()\n",
    "val1 = var.value()\n",
    "\n",
    "opt.minimize(loss,[var]).numpy()\n",
    "val2 = var.value()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=1.0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.9>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.71999997>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AdaGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.9046537>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opt = tf.keras.optimizers.Adagrad(learning_rate=0.1, initial_accumulator_value=0.1, epsilon=1e-7)\n",
    "\n",
    "var = tf.Variable(1.0)\n",
    "val0 = var.value()\n",
    "\n",
    "loss = lambda:(var**2)/2.0\n",
    "\n",
    "opt.minimize(loss,[var]).numpy()\n",
    "val1 = var.value()\n",
    "\n",
    "val1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RMSprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.6837723>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opt = tf.keras.optimizers.RMSprop(lr=0.1, rho=0.9)\n",
    "\n",
    "var = tf.Variable(1.0)\n",
    "\n",
    "loss = lambda:(var**2)/2.0\n",
    "\n",
    "opt.minimize(loss,[var]).numpy()\n",
    "val1 = var.value()\n",
    "\n",
    "val1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.89999735>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opt = tf.keras.optimizers.Adam(lr=0.1)\n",
    "var = tf.Variable(1.0)\n",
    "\n",
    "loss = lambda:(var**2)/2.0\n",
    "\n",
    "opt.minimize(loss,[var]).numpy()\n",
    "val1 = var.value()\n",
    "\n",
    "val1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学习率退火\n",
    "在训练神经网络时，一般情况下学习率都会随着训练而变化\n",
    "\n",
    "原因是：在网络训练的后期，如果学习率过高，会造成loss震荡，但是如果过小，就会造成收敛缓慢的情况\n",
    "\n",
    "分段常数衰减\n",
    "\n",
    "指数衰减\n",
    "\n",
    "1/t衰减"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正则化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只有训练的时候加正则化，测试时不需要加正则化！\n",
    "L1、L2正则化，提前终止，Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "在训练时会随机丢弃一部分神经元，模型在测试阶段会使用所有神经元"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  4.]\n",
      " [ 5.  6.]\n",
      " [ 7.  8.]\n",
      " [ 9. 10.]]\n",
      "tf.Tensor(\n",
      "[[ 1.25  0.  ]\n",
      " [ 3.75  5.  ]\n",
      " [ 6.25  7.5 ]\n",
      " [ 8.75  0.  ]\n",
      " [11.25 12.5 ]], shape=(5, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "layer = tf.keras.layers.Dropout(0.2, input_shape=(2,))\n",
    "\n",
    "data = np.arange(1, 11).reshape(5, 2).astype(np.float32)\n",
    "\n",
    "print(data)\n",
    "\n",
    "outputs = layer(data, training = True)\n",
    "\n",
    "print(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 批标准化（Batch Normalization）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "233.275px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}