03_basics_of_autograd.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03_basics_of_autograd.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "av4CWQiPZvBE",
        "colab_type": "text"
      },
      "source": [
        "# Autograd concepts\n",
        "\n",
        "Heart of a Neural network is its backpropogation task. This is where the loss is fixed backwards by calculating gradient for each of the tensors within the network. Thankgod these gradients (all the partical derivaties of a function, in chain fashion for each layers in the network)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "22_d3yexd5MI",
        "colab_type": "text"
      },
      "source": [
        "**Autograd** :\n",
        "In short, autograd does this gradient calculation for us automatically and store all the partical derivates in a vector called Jacobian-Vector. To understand the working we must monitor following autograd attributes for each node: \n",
        "\n",
        "\n",
        "1.   **tensor**\n",
        "2.   **requires_grad**\n",
        "3.   **grad_fn**\n",
        "4.   **grad**\n",
        "5.   **is_leaf**   \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aVPO7NOkLh7A",
        "colab_type": "code",
        "outputId": "1816e4c1-259f-4552-88a9-a9d88efad1d9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# STEP : 1\n",
        "\n",
        "import torch\n",
        "# The autograd package provides automatic differentiation \n",
        "# for all operations on Tensors\n",
        "\n",
        "# requires_grad = True -> tracks all operations on the tensor. \n",
        "x = torch.randn(2, 3, requires_grad=True)\n",
        "y = x + 2\n",
        "\n",
        "print(x.dtype)\n",
        "print(x) # OR\n",
        "#print(x.requires_grad)\n",
        "print(y) # OR\n",
        "#print(y.grad_fn)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.float32\n",
            "tensor([[-0.9890,  1.3430, -1.3031],\n",
            "        [-0.9375, -0.1760,  0.2672]], requires_grad=True)\n",
            "tensor([[1.0110, 3.3430, 0.6969],\n",
            "        [1.0625, 1.8240, 2.2672]], grad_fn=<AddBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GxApYQwAha7P",
        "colab_type": "text"
      },
      "source": [
        "If you notice above,\n",
        "\n",
        "`require_grad` is set to `True` for tensor `x`, whereas\n",
        "tensor `y` is created due to an `add` operation and hence it has `AddBackward` grad_fn."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DJp2GYcIiWbS",
        "colab_type": "code",
        "outputId": "ae721e0a-bc12-479f-cc8b-741b3511cffc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# STEP : 2 \n",
        "\n",
        "# More attributes about tensors in action. \n",
        "print(x.shape)\n",
        "print(y.shape)\n",
        "print(x.is_leaf)\n",
        "print(y.is_leaf)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([2, 3])\n",
            "torch.Size([2, 3])\n",
            "True\n",
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w8U9AzdcjcIR",
        "colab_type": "code",
        "outputId": "9c766340-7a10-43af-eace-c8b7cc02f948",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "# STEP : 3\n",
        "\n",
        "# More operations on y\n",
        "z = y * 5\n",
        "print(z.shape)\n",
        "print(z.grad_fn)\n",
        "print(z.requires_grad)\n",
        "\n",
        "print('====== Post Mean ======')\n",
        "zz = z.mean()\n",
        "print(zz.shape)\n",
        "print(zz.grad_fn)\n",
        "print(zz.requires_grad)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([2, 3])\n",
            "<MulBackward0 object at 0x7f81ea1e0320>\n",
            "True\n",
            "====== Post Mean ======\n",
            "torch.Size([])\n",
            "<MeanBackward0 object at 0x7f82386f8da0>\n",
            "True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zP5zXt03lK1R",
        "colab_type": "code",
        "outputId": "da638432-d843-4a5f-e125-e5caf25b9516",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# STEP : 4\n",
        "\n",
        "# Let's compute the gradient with builtin backpropogation\n",
        "zz.backward()\n",
        "\n",
        "# The gradient for this tensor will be accumulated into .grad attribute.\n",
        "print(zz.grad)\n",
        "\n",
        "# zero_grad clears old gradients from the previous step (otherwise you’d just accumulate the gradients from all loss.backward() calls.\n",
        "x.grad.zero_()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0.],\n",
              "        [0., 0., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R8bCd4S8mNNX",
        "colab_type": "code",
        "outputId": "ed681a85-f5de-4328-8d37-c4d96bbe837d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 307
        }
      },
      "source": [
        "# STEP : 5 (if STEP : 4 executed, re-run STEP: 1,2,3)\n",
        "\n",
        "# But, following FAILS\n",
        "z.backward()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-8-40c0c9b0bbab>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mz\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    193\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    194\u001b[0m         \"\"\"\n\u001b[0;32m--> 195\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    196\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     91\u001b[0m         \u001b[0mgrad_tensors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     92\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 93\u001b[0;31m     \u001b[0mgrad_tensors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_grads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     94\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mretain_graph\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     95\u001b[0m         \u001b[0mretain_graph\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36m_make_grads\u001b[0;34m(outputs, grads)\u001b[0m\n\u001b[1;32m     32\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 34\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"grad can be implicitly created only for scalar outputs\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     35\u001b[0m                 \u001b[0mnew_grads\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmemory_format\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpreserve_format\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     36\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mRuntimeError\u001b[0m: grad can be implicitly created only for scalar outputs"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gUfL87r2noNu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e34e4b20-532d-4d87-cdfc-6ed86cae141d"
      },
      "source": [
        "# STEP : 6 (if STEP : 4 or 5 executed, re-run STEP: 1,2,3)\n",
        "\n",
        "# Whereas, z has requires_grad set.\n",
        "# This is because you need .grad can store scalar value and due to mean zz has become scalar, whereas\n",
        "# z has shape (2,3).\n",
        "# To solve such problems:\n",
        "v = torch.empty(2,3, dtype=torch.float32)\n",
        "# NOTE : re-run all the above blocks except block #5.\n",
        "z.backward(v)\n",
        "\n",
        "# this is important! It affects the final weights & output\n",
        "x.grad.zero_()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0.],\n",
              "        [0., 0., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TrahJufFoZ4w",
        "colab_type": "code",
        "outputId": "e9f9f438-c2d0-41fc-979b-a9f89f9dc759",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Let's check the stored gradients. \n",
        "print(x.grad)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0., 0., 0.],\n",
            "        [0., 0., 0.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fhQWOCDF3FvZ",
        "colab_type": "text"
      },
      "source": [
        "## Following three ways to control weights to update or not during training cycle. \n",
        "1. `weight.requires_grad_(False/True)`\n",
        "2. `weight.detach()`\n",
        "3. within `with` statement as `with torch.no_grad()`\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-URLES6Y2NsF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "868224d9-1f25-4c0f-8de3-fe5b936a7832"
      },
      "source": [
        "weights = torch.randn(3, requires_grad=True)\n",
        "\n",
        "for epoch in range(3):\n",
        "    model = (weights * 3).mean()\n",
        "    model.backward()\n",
        "    \n",
        "    print(weights.grad)\n",
        "    with torch.no_grad():\n",
        "        weights -= 0.001 * weights.grad\n",
        "\n",
        "    # zero_grad clears old gradients from the previous step.\n",
        "    weights.grad.zero_()\n",
        "\n",
        "print(weights)\n",
        "print(model)\n"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([1., 1., 1.])\n",
            "tensor([1., 1., 1.])\n",
            "tensor([1., 1., 1.])\n",
            "tensor([ 0.7771,  0.5906, -1.4751], requires_grad=True)\n",
            "tensor(-0.1044, grad_fn=<MeanBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}