Lab_10_(part_3).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab 10 (part 3)",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REaV1GblZovs",
        "colab_type": "text"
      },
      "source": [
        "#Visualizing and understanding ConvNets (part 3)\n",
        "In this Lab we will be experimenting with Neural Style Transfer and two closely related sub-problems:\n",
        "\n",
        "- Image reconstruction from latent representations, as in the paper [Understanding Deep Image Representations by Inverting Them](https://arxiv.org/pdf/1412.0035.pdf)\n",
        "- Texture syntesis\n",
        "\n",
        "Homepage (refer to this page for detailed explanations): https://keras.io/examples/neural_style_transfer/\n",
        "\n",
        "Source code: https://github.com/keras-team/keras/blob/master/examples/neural_style_transfer.py\n",
        "\n",
        "**Before we start - remember to set runtime to GPU**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSIpfu6wUG1r",
        "colab_type": "text"
      },
      "source": [
        "**NOTE:** In case you have trouble running Keras/TensorFlow in Colab, try one of the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fZ01fAojUGCV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Try this\n",
        "#!pip install --upgrade tensorflow==1.8.0\n",
        "\n",
        "# ... or this\n",
        "%tensorflow_version 1.x\n",
        "\n",
        "# Check TensorFlow version\n",
        "#import tensorflow as tf\n",
        "#print(tf.__version__)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6R1GkX8tIhZ5"
      },
      "source": [
        "##Mount your Google Drive\n",
        "Only required if you want test images to your Google Drive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "cf702ab2-20fc-4d1b-9926-dbbd3750ea13",
        "id": "Dn256IGRIhPG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 128
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VrwC7acasZiO",
        "colab_type": "text"
      },
      "source": [
        "##Neural style transfer\n",
        "The loss term in Neural Style Transfer consists of\n",
        "\n",
        "- a **content term** that compares the latent representation of the generated image and the content image using features from one or more layers\n",
        "-a **style term** that compares the Gram matrix of the generated image and the style image using features from one or more layers\n",
        "- a **regularization term** that enforces local constancy in the generated image (using [total variation](https://en.wikipedia.org/wiki/Total_variation)).\n",
        "\n",
        "###Tasks\n",
        "1. Upload a content image and a style image and run them through the code above.\n",
        "\n",
        "2. The generated image is denoted `x`. Below, it is initialized with the content image, who's path is `base_image_path`. THis naturally drives the reconstruction towards the content image. However, in the general case, we want to initialize `x` with random noise. Your task is simply to initialize `x` with random noise and repeat task 1.\n",
        "\n",
        "In the sub-tasks below, use a random image for initialization:\n",
        "\n",
        "3. You can make `x` match the content of the content image by setting the style loss weight to zero. This corresponds to the method used in this paper (https://arxiv.org/pdf/1412.0035.pdf). Your task is to reconstruct the content image using different convolution layers. Compare early layers to late layers.\n",
        "\n",
        "4. Similarly, you can perform texture synthesis the setting the content weight to zero. Your task is to generate an image of strawberries!!!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uSS1grsYYuYn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import print_function\n",
        "from keras.preprocessing.image import load_img, save_img, img_to_array\n",
        "import numpy as np\n",
        "from scipy.optimize import fmin_l_bfgs_b\n",
        "import time\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from keras.applications import vgg19\n",
        "from keras import backend as K\n",
        "\n",
        "base_image_path = # path to content image\n",
        "style_reference_image_path = # path to style image\n",
        "result_prefix = 'test'\n",
        "iterations = 10\n",
        "\n",
        "# these are the weights of the different loss components\n",
        "total_variation_weight = 1.0\n",
        "style_weight = 1.0\n",
        "content_weight = 0.025\n",
        "\n",
        "# dimensions of the generated picture.\n",
        "width, height = load_img(base_image_path).size\n",
        "img_nrows = 400\n",
        "img_ncols = int(width * img_nrows / height)\n",
        "\n",
        "# util function to open, resize and format pictures into appropriate tensors\n",
        "\n",
        "def preprocess_image(image_path):\n",
        "    img = load_img(image_path, target_size=(img_nrows, img_ncols))\n",
        "    img = img_to_array(img)\n",
        "    img = np.expand_dims(img, axis=0)\n",
        "    img = vgg19.preprocess_input(img)\n",
        "    return img\n",
        "\n",
        "# util function to convert a tensor into a valid image\n",
        "\n",
        "\n",
        "def deprocess_image(x):\n",
        "    if K.image_data_format() == 'channels_first':\n",
        "        x = x.reshape((3, img_nrows, img_ncols))\n",
        "        x = x.transpose((1, 2, 0))\n",
        "    else:\n",
        "        x = x.reshape((img_nrows, img_ncols, 3))\n",
        "    # Remove zero-center by mean pixel\n",
        "    x[:, :, 0] += 103.939\n",
        "    x[:, :, 1] += 116.779\n",
        "    x[:, :, 2] += 123.68\n",
        "    # 'BGR'->'RGB'\n",
        "    x = x[:, :, ::-1]\n",
        "    x = np.clip(x, 0, 255).astype('uint8')\n",
        "    return x\n",
        "\n",
        "# get tensor representations of our images\n",
        "base_image = K.variable(preprocess_image(base_image_path))\n",
        "style_reference_image = K.variable(preprocess_image(style_reference_image_path))\n",
        "\n",
        "# this will contain our generated image\n",
        "if K.image_data_format() == 'channels_first':\n",
        "    combination_image = K.placeholder((1, 3, img_nrows, img_ncols))\n",
        "else:\n",
        "    combination_image = K.placeholder((1, img_nrows, img_ncols, 3))\n",
        "\n",
        "# combine the 3 images into a single Keras tensor\n",
        "input_tensor = K.concatenate([base_image,\n",
        "                              style_reference_image,\n",
        "                              combination_image], axis=0)\n",
        "\n",
        "# build the VGG19 network with our 3 images as input\n",
        "# the model will be loaded with pre-trained ImageNet weights\n",
        "model = vgg19.VGG19(input_tensor=input_tensor,\n",
        "                    weights='imagenet', include_top=False)\n",
        "print('Model loaded.')\n",
        "\n",
        "# get the symbolic outputs of each \"key\" layer (we gave them unique names).\n",
        "outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])\n",
        "\n",
        "# compute the neural style loss\n",
        "# first we need to define 4 util functions\n",
        "\n",
        "# the gram matrix of an image tensor (feature-wise outer product)\n",
        "\n",
        "\n",
        "def gram_matrix(x):\n",
        "    assert K.ndim(x) == 3\n",
        "    if K.image_data_format() == 'channels_first':\n",
        "        features = K.batch_flatten(x)\n",
        "    else:\n",
        "        features = K.batch_flatten(K.permute_dimensions(x, (2, 0, 1)))\n",
        "    gram = K.dot(features, K.transpose(features))\n",
        "    return gram\n",
        "\n",
        "# the \"style loss\" is designed to maintain\n",
        "# the style of the reference image in the generated image.\n",
        "# It is based on the gram matrices (which capture style) of\n",
        "# feature maps from the style reference image\n",
        "# and from the generated image\n",
        "\n",
        "def style_loss(style, combination):\n",
        "    assert K.ndim(style) == 3\n",
        "    assert K.ndim(combination) == 3\n",
        "    S = gram_matrix(style)\n",
        "    C = gram_matrix(combination)\n",
        "    channels = 3\n",
        "    size = img_nrows * img_ncols\n",
        "    return K.sum(K.square(S - C)) / (4.0 * (channels ** 2) * (size ** 2))\n",
        "\n",
        "# an auxiliary loss function\n",
        "# designed to maintain the \"content\" of the\n",
        "# base image in the generated image\n",
        "\n",
        "\n",
        "def content_loss(base, combination):\n",
        "    return K.sum(K.square(combination - base))\n",
        "\n",
        "# the 3rd loss function, total variation loss,\n",
        "# designed to keep the generated image locally coherent\n",
        "\n",
        "\n",
        "def total_variation_loss(x):\n",
        "    assert K.ndim(x) == 4\n",
        "    if K.image_data_format() == 'channels_first':\n",
        "        a = K.square(\n",
        "            x[:, :, :img_nrows - 1, :img_ncols - 1] - x[:, :, 1:, :img_ncols - 1])\n",
        "        b = K.square(\n",
        "            x[:, :, :img_nrows - 1, :img_ncols - 1] - x[:, :, :img_nrows - 1, 1:])\n",
        "    else:\n",
        "        a = K.square(\n",
        "            x[:, :img_nrows - 1, :img_ncols - 1, :] - x[:, 1:, :img_ncols - 1, :])\n",
        "        b = K.square(\n",
        "            x[:, :img_nrows - 1, :img_ncols - 1, :] - x[:, :img_nrows - 1, 1:, :])\n",
        "    return K.sum(K.pow(a + b, 1.25))\n",
        "\n",
        "\n",
        "# combine these loss functions into a single scalar\n",
        "loss = K.variable(0.0)\n",
        "layer_features = outputs_dict['block5_conv2']\n",
        "base_image_features = layer_features[0, :, :, :]\n",
        "combination_features = layer_features[2, :, :, :]\n",
        "loss = loss + content_weight * content_loss(base_image_features,\n",
        "                                            combination_features)\n",
        "\n",
        "feature_layers = ['block1_conv1', 'block2_conv1',\n",
        "                  'block3_conv1', 'block4_conv1',\n",
        "                  'block5_conv1']\n",
        "for layer_name in feature_layers:\n",
        "    layer_features = outputs_dict[layer_name]\n",
        "    style_reference_features = layer_features[1, :, :, :]\n",
        "    combination_features = layer_features[2, :, :, :]\n",
        "    sl = style_loss(style_reference_features, combination_features)\n",
        "    loss = loss + (style_weight / len(feature_layers)) * sl\n",
        "loss = loss + total_variation_weight * total_variation_loss(combination_image)\n",
        "\n",
        "# get the gradients of the generated image wrt the loss\n",
        "grads = K.gradients(loss, combination_image)\n",
        "\n",
        "outputs = [loss]\n",
        "if isinstance(grads, (list, tuple)):\n",
        "    outputs += grads\n",
        "else:\n",
        "    outputs.append(grads)\n",
        "\n",
        "f_outputs = K.function([combination_image], outputs)\n",
        "\n",
        "\n",
        "def eval_loss_and_grads(x):\n",
        "    if K.image_data_format() == 'channels_first':\n",
        "        x = x.reshape((1, 3, img_nrows, img_ncols))\n",
        "    else:\n",
        "        x = x.reshape((1, img_nrows, img_ncols, 3))\n",
        "    outs = f_outputs([x])\n",
        "    loss_value = outs[0]\n",
        "    if len(outs[1:]) == 1:\n",
        "        grad_values = outs[1].flatten().astype('float64')\n",
        "    else:\n",
        "        grad_values = np.array(outs[1:]).flatten().astype('float64')\n",
        "    return loss_value, grad_values\n",
        "\n",
        "# this Evaluator class makes it possible\n",
        "# to compute loss and gradients in one pass\n",
        "# while retrieving them via two separate functions,\n",
        "# \"loss\" and \"grads\". This is done because scipy.optimize\n",
        "# requires separate functions for loss and gradients,\n",
        "# but computing them separately would be inefficient.\n",
        "class Evaluator(object):\n",
        "\n",
        "    def __init__(self):\n",
        "        self.loss_value = None\n",
        "        self.grads_values = None\n",
        "\n",
        "    def loss(self, x):\n",
        "        assert self.loss_value is None\n",
        "        loss_value, grad_values = eval_loss_and_grads(x)\n",
        "        self.loss_value = loss_value\n",
        "        self.grad_values = grad_values\n",
        "        return self.loss_value\n",
        "\n",
        "    def grads(self, x):\n",
        "        assert self.loss_value is not None\n",
        "        grad_values = np.copy(self.grad_values)\n",
        "        self.loss_value = None\n",
        "        self.grad_values = None\n",
        "        return grad_values\n",
        "\n",
        "\n",
        "evaluator = Evaluator()\n",
        "\n",
        "# run scipy-based optimization (L-BFGS) over the pixels of the generated image\n",
        "# so as to minimize the neural style loss\n",
        "x = preprocess_image(base_image_path)\n",
        "\n",
        "img = deprocess_image(x.copy())\n",
        "plt.figure()\n",
        "plt.imshow(img)\n",
        "plt.show()\n",
        "\n",
        "for i in range(iterations):\n",
        "    print('Start of iteration', i)\n",
        "    start_time = time.time()\n",
        "    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),\n",
        "                                     fprime=evaluator.grads, maxfun=20)\n",
        "    print('Current loss value:', min_val)\n",
        "    # save current generated image\n",
        "    img = deprocess_image(x.copy())\n",
        "    plt.figure()\n",
        "    plt.imshow(img)\n",
        "    plt.show()\n",
        "    fname = result_prefix + '_at_iteration_%d.png' % i\n",
        "    save_img(fname, img)\n",
        "    end_time = time.time()\n",
        "    print('Image saved as', fname)\n",
        "    print('Iteration %d completed in %ds' % (i, end_time - start_time))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z7kFfmdJvx0",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    }
  ]
}