eecs498-007/A4/pytorch_autograd_and_nn.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"accelerator":"GPU","colab":{"name":"pytorch_autograd_and_nn.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.1"},"toc":{"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"toc_cell":false,"toc_position":{},"toc_section_display":"block","toc_window_display":false},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false}},"cells":[{"cell_type":"markdown","metadata":{"id":"DDJwQPZcupab"},"source":["# EECS 498-007/598-005 Assignment 4-1: Pytorch Autograd and NN\n","\n","Before we start, please put your name and UMID in following format\n","\n",": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"]},{"cell_type":"markdown","metadata":{"id":"2KMxqLt1h2kx"},"source":["**Your Answer:**   \n","Soufian BENAMARA, #NOid"]},{"cell_type":"markdown","metadata":{"id":"aQW_w1Wzw72f","tags":["pdf-title"]},"source":["# [torch.autograd](https://pytorch.org/docs/stable/autograd.html) and [torch.nn](https://pytorch.org/docs/stable/nn.html)\n","\n","So far, we used PyTorch to accelarate computation using GPU.\n","PyTorch also provides several useful packages, which help to design deep neural networks efficiently.\n","\n","The `torch.autograd` package provides classes and functions implementing automatic differentiation of arbitrary scalar valued functions.\n","\n","This notebook assumes that you are using **PyTorch version 1.3 or above**.\n","In some of the previous versions (e.g. before 0.4), Tensors had to be wrapped in `torch.autograd.Variable` objects to enable autograd;\n","however this class has now been deprecated and merged with `torch.Tensor`.\n","In addition 1.0 also separates a Tensor's datatype from its device, and uses numpy-style factories for constructing Tensors rather than directly invoking Tensor constructors.\n","Now, to obtain gradients for a tensor via autograd from arbitrary scalar valued functions, you can simply set `requires_grad=True`.\n","\n","The `torch.nn` package defines a set of Modules, which you can think of as a neural network layer that has produces output from input and may have some trainable weights.\n","\n","You can also find the detailed [API doc](http://pytorch.org/docs/stable/index.html) here.\n","If you have other questions that are not addressed by the API docs, the [PyTorch forum](https://discuss.pytorch.org/) is a much better place to ask than StackOverflow.\n"]},{"cell_type":"markdown","metadata":{"id":"ED0jpoGyIL_B","tags":["pdf-ignore"]},"source":["# Table of Contents\n","\n","This assignment has 5 parts. You will learn PyTorch on **three different levels of abstraction**, which will help you understand it better.\n","\n","1. Part I, Preparation: As we always do, we will use CIFAR-10 dataset.\n","2. Part II, Barebones PyTorch: **Abstraction level 1**, we will work directly with the lowest-level PyTorch Tensors with autograd.\n","3. Part III, PyTorch Module API: **Abstraction level 2**, we will use `nn.Module` to define an arbitrary neural network architecture. \n","4. Part IV, PyTorch Sequential API: **Abstraction level 3**, we will use `nn.Sequential` to define a fully-connected and convolutional network very conveniently. \n","5. Part V, Residual Network: please implement your own ResNet to get a high accuracy on CIFAR-10.\n","\n","Here is a table of comparison:\n","\n","| API             | Flexibility | Convenience |\n","|-----------------|-------------|-------------|\n","| Barebone        | High        | Low         |\n","| `nn.Module`     | High        | Medium      |\n","| `nn.Sequential` | Low         | High        |"]},{"cell_type":"markdown","metadata":{"id":"7LCmGZ_3IL_V"},"source":["# Part I. Preparation"]},{"cell_type":"markdown","metadata":{"id":"ubB_0e-UAOVK"},"source":["Before getting started we need to run some boilerplate code to set up our environment. You'll need to rerun this setup code each time you start the notebook.\n","\n","First, run this cell load the [autoreload](https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html?highlight=autoreload) extension. This allows us to edit `.py` source files, and re-import them into the notebook for a seamless editing and debugging experience."]},{"cell_type":"code","metadata":{"id":"ASkY27ZtA7Is","executionInfo":{"status":"ok","timestamp":1617456253557,"user_tz":-60,"elapsed":856,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["%load_ext autoreload\n","%autoreload 2"],"execution_count":1,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"rseA2LOaXlpf"},"source":["### Google Colab Setup\n","\n","Next we need to run a few commands to set up our environment on Google Colab. If you are running this notebook on a local machine you can skip this section.\n","\n","Run the following cell to mount your Google Drive. Follow the link, sign in to your Google account (the same account you used to store this notebook!) and copy the authorization code into the text box that appears below."]},{"cell_type":"code","metadata":{"id":"qXyYCFDnXkee","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456253845,"user_tz":-60,"elapsed":1133,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"f287522e-8a92-4225-840c-75a7de5f16f0"},"source":["from google.colab import drive\n","drive.mount('/content/drive')"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"A3ITvb-mXyCs"},"source":["Now recall the path in your Google Drive where you uploaded this notebook, fill it in below. If everything is working correctly then running the folowing cell should print the filenames from the assignment:\n","\n","```\n","['eecs598', 'network_visualization.py', 'style_transfer.py',  'network_visualization.ipynb', 'a4_helper.py', 'pytorch_autograd_and_nn.py', 'pytorch_autograd_and_nn.ipynb', 'style_transfer.ipynb', 'rnn_lstm_attention_captioning.ipynb',  'rnn_lstm_attention_captioning.py']\n","```"]},{"cell_type":"code","metadata":{"id":"BMop7WhzX5GT","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456253846,"user_tz":-60,"elapsed":1125,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"43f738b9-cfce-4c03-e10b-e209cb21fe7f"},"source":["import os\n","\n","# TODO: Fill in the Google Drive path where you uploaded the assignment\n","# Example: If you create a 2020FA folder and put all the files under A1 folder, then '2020FA/A1'\n","GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = '2020FA/A4'\n","GOOGLE_DRIVE_PATH = os.path.join('drive', 'My Drive', GOOGLE_DRIVE_PATH_AFTER_MYDRIVE)\n","print(os.listdir(GOOGLE_DRIVE_PATH))"],"execution_count":3,"outputs":[{"output_type":"stream","text":["['network_visualization.py', 'a4_helper.py', 'rnn_lstm_attention_captioning.ipynb', 'style_transfer.ipynb', 'network_visualization.ipynb', 'rnn_lstm_attention_captioning.py', 'style_transfer.py', 'eecs598', '__pycache__', 'pytorch_autograd_and_nn.py', 'pytorch_autograd_and_nn.ipynb']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"t8MhujYNbSir"},"source":["Once you have successfully mounted your Google Drive and located the path to this assignment, run th following cell to allow us to import from the `.py` files of this assignment. If it works correctly, it should print the message:\n","\n","```\n","Hello from pytorch_autograd_and_nn.py!\n","```\n","\n","as well as the last edit time for the file `pytorch_autograd_and_nn.py`."]},{"cell_type":"code","metadata":{"id":"pGJoaRyybcka","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456254632,"user_tz":-60,"elapsed":1902,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"f812010f-0973-4b86-ff16-3bba44854bc0"},"source":["import sys\n","sys.path.append(GOOGLE_DRIVE_PATH)\n","\n","import time, os\n","os.environ[\"TZ\"] = \"US/Eastern\"\n","time.tzset()\n","\n","from pytorch_autograd_and_nn import *\n","from a4_helper import *\n","hello()\n","\n","py_path = os.path.join(GOOGLE_DRIVE_PATH, 'pytorch_autograd_and_nn.py')\n","py_edit_time = time.ctime(os.path.getmtime(py_path))\n","print('pytorch_autograd_and_nn.py last edited on %s' % py_edit_time)"],"execution_count":4,"outputs":[{"output_type":"stream","text":["Hello from pytorch_autograd_and_nn.py!\n","pytorch_autograd_and_nn.py last edited on Sat Apr  3 09:23:41 2021\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"MzqbYcKdz6ew"},"source":["### Load Packages\n","\n","Run some setup code for this notebook: Import some useful packages and increase the default figure size."]},{"cell_type":"code","metadata":{"id":"Q8o3FxatIL_X","tags":["pdf-ignore"],"executionInfo":{"status":"ok","timestamp":1617456254633,"user_tz":-60,"elapsed":1893,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","import torch.optim as optim\n","from eecs598.utils import reset_seed\n","from collections import OrderedDict\n","\n","# for plotting\n","import matplotlib.pyplot as plt\n","plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n","plt.rcParams['image.interpolation'] = 'nearest'\n","plt.rcParams['image.cmap'] = 'gray'\n","\n","%matplotlib inline"],"execution_count":5,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"xcPWzzO3gVE2"},"source":["We will use the GPU to accelerate our computation. Run this cell to make sure you are using a GPU.\n","\n","We will be using `torch.float = torch.float32` for data and `torch.long = torch.int64` for labels.\n","\n","Please refer to https://pytorch.org/docs/stable/tensor_attributes.html#torch-dtype for more details about data types."]},{"cell_type":"code","metadata":{"id":"blz1sXlkIL_q","tags":["pdf-ignore-input"],"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456254634,"user_tz":-60,"elapsed":1888,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"b6a0716a-693e-4e1b-b41f-7c72a8cf7b7f"},"source":["to_float= torch.float\n","to_long = torch.long\n","\n","if torch.cuda.is_available:\n","  print('Good to go!')\n","else:\n","  print('Please set GPU via Edit -> Notebook Settings.')"],"execution_count":6,"outputs":[{"output_type":"stream","text":["Good to go!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"UnprYqtnfzzE"},"source":["### Load CIFAR\n","Firstly, we will load the CIFAR-10 dataset. The utility function `load_CIFAR()` in `a4_helper` returns training, validaton and testing dataloaders for CIFAR-10 dataset. We are using [torchvision.datasets.CIFAR10](https://pytorch.org/docs/stable/torchvision/datasets.html?highlight=cifar#torchvision.datasets.CIFAR10) to download the CIFAR-10 dataset."]},{"cell_type":"code","metadata":{"id":"-XB6NUX0IL_f","tags":["pdf-ignore"],"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456257046,"user_tz":-60,"elapsed":4288,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"64f5ea2f-2caf-4068-9d99-6932817d32d3"},"source":["loader_train, loader_val, loader_test = load_CIFAR(path='./datasets/')"],"execution_count":7,"outputs":[{"output_type":"stream","text":["Files already downloaded and verified\n","Files already downloaded and verified\n","Files already downloaded and verified\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"mazLauPlIL_5"},"source":["# Part II. Barebones PyTorch\n","\n","PyTorch ships with high-level APIs to help us define model architectures conveniently, which we will cover in Part II of this tutorial. In this section, we will start with the barebone PyTorch elements to understand the autograd engine better. After this exercise, you will come to appreciate the high-level model API more.\n","\n","We will start with a simple fully-connected ReLU network with two hidden layers and no biases for CIFAR classification. \n","This implementation computes the forward pass using operations on PyTorch Tensors, and uses PyTorch autograd to compute gradients. It is important that you understand every line, because you will write a harder version after the example.\n","\n","When we create a PyTorch Tensor with `requires_grad=True`, then operations involving that Tensor will not just compute values; they will also build up a computational graph in the background, allowing us to easily backpropagate through the graph to compute gradients of some Tensors with respect to a downstream loss. Concretely, if `x` is a Tensor with `x.requires_grad == True` then after backpropagation `x.grad` will be another Tensor holding the gradient of `x` with respect to the scalar loss at the end."]},{"cell_type":"markdown","metadata":{"id":"3rPnppVkIMAB","tags":["pdf-ignore"]},"source":["### Barebones PyTorch: Two-Layer Network\n","\n","Here we define a function `two_layer_fc` which performs the forward pass of a two-layer fully-connected ReLU network on a batch of image data. After defining the forward pass we check that it doesn't crash and that it produces outputs of the right shape by running zeros through the network.\n","\n","You don't have to write any code here, but it's important that you read and understand the implementation."]},{"cell_type":"code","metadata":{"id":"D6PqRQwlIMAC","tags":["pdf-ignore-input"],"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456257048,"user_tz":-60,"elapsed":4279,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"b0b7c1f3-d728-4d58-a2be-5a5f073c34ca"},"source":["def two_layer_fc(x, params):\n","  \"\"\"\n","  A fully-connected neural networks; the architecture is:\n","  NN is fully connected -> ReLU -> fully connected layer.\n","  Note that this function only defines the forward pass; \n","  PyTorch will take care of the backward pass for us.\n","  \n","  The input to the network will be a minibatch of data, of shape\n","  (N, d1, ..., dM) where d1 * ... * dM = D. The hidden layer will have H units,\n","  and the output layer will produce scores for C classes.\n","  \n","  Inputs:\n","  - x: A PyTorch Tensor of shape (N, d1, ..., dM) giving a minibatch of\n","    input data.\n","  - params: A list [w1, w2] of PyTorch Tensors giving weights for the network;\n","    w1 has shape (H, D) and w2 has shape (C, H).\n","  \n","  Returns:\n","  - scores: A PyTorch Tensor of shape (N, C) giving classification scores for\n","    the input data x.\n","  \"\"\"\n","  # first we flatten the image\n","  x = flatten(x)  # shape: [batch_size, C x H x W]\n","  \n","  w1, b1, w2, b2 = params\n","  \n","  # Forward pass: compute predicted y using operations on Tensors. Since w1 and\n","  # w2 have requires_grad=True, operations involving these Tensors will cause\n","  # PyTorch to build a computational graph, allowing automatic computation of\n","  # gradients. Since we are no longer implementing the backward pass by hand we\n","  # don't need to keep references to intermediate values.\n","  # Note that F.linear(x, w, b) is equivalent to x.mm(w.t()) + b\n","  # For ReLU, you can also use `.clamp(min=0)`, equivalent to `F.relu()`\n","  x = F.relu(F.linear(x, w1, b1))\n","  x = F.linear(x, w2, b2)\n","  return x\n","    \n","\n","def two_layer_fc_test():\n","  hidden_layer_size = 42\n","  x = torch.zeros((64, 3, 16, 16), dtype=to_float)  # minibatch size 64, feature dimension 3*16*16\n","  w1 = torch.zeros((hidden_layer_size, 3*16*16), dtype=to_float)\n","  b1 = torch.zeros((hidden_layer_size,), dtype=to_float)\n","  w2 = torch.zeros((10, hidden_layer_size), dtype=to_float)\n","  b2 = torch.zeros((10,), dtype=to_float)\n","  scores = two_layer_fc(x, [w1, b1, w2, b2])\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","\n","two_layer_fc_test()"],"execution_count":8,"outputs":[{"output_type":"stream","text":["Output size: [64, 10]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"muCDvYEKIMAJ"},"source":["### Barebones PyTorch: Three-Layer ConvNet\n","\n","Here you will complete the implementation of the function `three_layer_convnet`, which will perform the forward pass of a three-layer convolutional network. Like above, we can immediately test our implementation by passing zeros through the network. The network should have the following architecture:\n","\n","1. A convolutional layer (with bias) with `channel_1` filters, each with shape `kernel_size_1 x kernel_size_1`, and zero-padding of two\n","2. ReLU nonlinearity\n","3. A convolutional layer (with bias) with `channel_2` filters, each with shape `kernel_size_2 x kernel_size_2`, and zero-padding of one\n","4. ReLU nonlinearity\n","5. Fully-connected layer with bias, producing scores for C classes.\n","\n","Note that we have **no softmax activation** here after our fully-connected layer: this is because PyTorch's cross entropy loss performs a softmax activation for you, and by bundling that step in makes computation more efficient.\n","\n","**HINT**: For convolutions: https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.conv2d; pay attention to the shapes of convolutional filters! You can use `print(tensor.shape)` for debugging the shapes after each intemidiate layer."]},{"cell_type":"markdown","metadata":{"id":"Z7Cg9qvTIMAR"},"source":["Implement `three_layer_convnet` and run the cell below to test it. When you run this function, scores should have shape (64, 10)."]},{"cell_type":"code","metadata":{"id":"1kEMMi4QIMAa","tags":["pdf-ignore-input"],"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456257403,"user_tz":-60,"elapsed":4624,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"143a3f9d-9712-498d-b0d5-d03ff8e5db8d"},"source":["def three_layer_convnet_test():\n","  x = torch.zeros((64, 3, 32, 32), dtype=to_float)  # minibatch size 64, image size [3, 32, 32]\n","\n","  conv_w1 = torch.zeros((6, 3, 5, 5), dtype=to_float)  # [out_channel, in_channel, kernel_H, kernel_W]\n","  conv_b1 = torch.zeros((6,))  # out_channel\n","  conv_w2 = torch.zeros((9, 6, 3, 3), dtype=to_float)  # [out_channel, in_channel, kernel_H, kernel_W]\n","  conv_b2 = torch.zeros((9,))  # out_channel\n","\n","  # you must calculate the shape of the tensor after two conv layers, before the fully-connected layer\n","  fc_w = torch.zeros((10, 9 * 32 * 32))\n","  fc_b = torch.zeros(10)\n","\n","  # YOUR_TURN: Impelement the three_layer_convnet function\n","  scores = three_layer_convnet(x, [conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b])\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","three_layer_convnet_test()"],"execution_count":9,"outputs":[{"output_type":"stream","text":["Output size: [64, 10]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"WBStmY2yIMAh"},"source":["### Barebones PyTorch: Kaiming Initialization\n","In this part, we are going to use [Kaiming initialization](https://arxiv.org/abs/1502.01852), which you already implemented in Assignment 3.\n","\n","Fortunately, PyTorch already provides a function, so we will use this:\n","[torch.nn.init.kaiming_normal_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_)\n","\n","By default, `gain = 2`, because this function assumes that ReLU activation follows.\n","This is true in the linear and convolutional layers in the models you are going to implement, except for the last fully-connected layer:\n","in principle, we should give `gain = 1` because ReLU is not applied there.\n","However, as stated in the [original paper](https://arxiv.org/abs/1502.01852), since the factor 1/2 does not matter if it just exists on one layer, we are going to keep using `gain = 2` for simplicity.\n","\n","For more details on initialization methods provided by PyTorch, see https://pytorch.org/docs/stable/nn.init.html."]},{"cell_type":"code","metadata":{"id":"_rf9JRh5IMAj","tags":["pdf-ignore-input"],"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456260510,"user_tz":-60,"elapsed":7722,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"0017f2ff-1794-4629-9708-df644d97657d"},"source":["reset_seed(0)\n","\n","# create a weight of shape [3 x 5]\n","print(nn.init.kaiming_normal_(torch.empty(3, 5, dtype=to_float, device='cuda')))\n","print(nn.init.zeros_(torch.empty(3 ,5, dtype=to_float, device='cuda')))"],"execution_count":10,"outputs":[{"output_type":"stream","text":["tensor([[-0.5848, -0.2690, -1.6721,  0.0918, -0.0764],\n","        [-0.3667, -0.3939, -0.2077, -0.6796, -0.2297],\n","        [-1.0569,  1.4328,  0.1971, -0.1165,  0.8137]], device='cuda:0')\n","tensor([[0., 0., 0., 0., 0.],\n","        [0., 0., 0., 0., 0.],\n","        [0., 0., 0., 0., 0.]], device='cuda:0')\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"akQqCObPIMAo"},"source":["### Barebones PyTorch: Check Accuracy\n","When training the model we will use the following function to check the accuracy of our model on the training or validation sets.\n","\n","When checking accuracy we don't need to compute any gradients; as a result we don't need PyTorch to build a computational graph for us when we compute scores. To prevent a graph from being built we scope our computation under a `torch.no_grad()` context manager."]},{"cell_type":"code","metadata":{"id":"l4xAUWASIMAq","tags":["pdf-ignore-input"],"executionInfo":{"status":"ok","timestamp":1617456260511,"user_tz":-60,"elapsed":7712,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["def check_accuracy_part2(loader, model_fn, params):\n","  \"\"\"\n","  Check the accuracy of a classification model.\n","  \n","  Inputs:\n","  - loader: A DataLoader for the data split we want to check\n","  - model_fn: A function that performs the forward pass of the model,\n","    with the signature scores = model_fn(x, params)\n","  - params: List of PyTorch Tensors giving parameters of the model\n","  \n","  Returns: Nothing, but prints the accuracy of the model\n","  \"\"\"\n","  split = 'val' if loader.dataset.train else 'test'\n","  print('Checking accuracy on the %s set' % split)\n","  num_correct, num_samples = 0, 0\n","  with torch.no_grad():\n","    for x, y in loader:\n","      x = x.to(device='cuda', dtype=to_float)  # move to device, e.g. GPU\n","      y = y.to(device='cuda', dtype=to_long)\n","      scores = model_fn(x, params)\n","      _, preds = scores.max(1)\n","      num_correct += (preds == y).sum()\n","      num_samples += preds.size(0)\n","    acc = float(num_correct) / num_samples\n","    print('Got %d / %d correct (%.2f%%)' % (num_correct, num_samples, 100 * acc))\n","    return acc"],"execution_count":11,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"8LvPbM4WIMAv"},"source":["### BareBones PyTorch: Training Loop\n","We can now set up a basic training loop to train our network. We will train the model using stochastic gradient descent without momentum. We will use `torch.nn.functional.cross_entropy` to compute the loss; you can [read about it here](https://pytorch.org/docs/stable/nn.html#crossentropyloss).\n","\n","The training loop takes as input the neural network function, a list of initialized parameters (`[w1, w2]` in our example), and learning rate."]},{"cell_type":"code","metadata":{"id":"WYGBD0YZIMAx","tags":["pdf-ignore-input"],"executionInfo":{"status":"ok","timestamp":1617456260512,"user_tz":-60,"elapsed":7706,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["def train_part2(model_fn, params, learning_rate):\n","  \"\"\"\n","  Train a model on CIFAR-10.\n","  \n","  Inputs:\n","  - model_fn: A Python function that performs the forward pass of the model.\n","    It should have the signature scores = model_fn(x, params) where x is a\n","    PyTorch Tensor of image data, params is a list of PyTorch Tensors giving\n","    model weights, and scores is a PyTorch Tensor of shape (N, C) giving\n","    scores for the elements in x.\n","  - params: List of PyTorch Tensors giving weights for the model\n","  - learning_rate: Python scalar giving the learning rate to use for SGD\n","  \n","  Returns: Nothing\n","  \"\"\"\n","  for t, (x, y) in enumerate(loader_train):\n","    # Move the data to the proper device (GPU or CPU)\n","    x = x.to(device='cuda', dtype=to_float)\n","    y = y.to(device='cuda', dtype=to_long)\n","\n","    # Forward pass: compute scores and loss\n","    scores = model_fn(x, params)\n","    loss = F.cross_entropy(scores, y)\n","\n","    # Backward pass: PyTorch figures out which Tensors in the computational\n","    # graph has requires_grad=True and uses backpropagation to compute the\n","    # gradient of the loss with respect to these Tensors, and stores the\n","    # gradients in the .grad attribute of each Tensor.\n","    loss.backward()\n","\n","    # Update parameters. We don't want to backpropagate through the\n","    # parameter updates, so we scope the updates under a torch.no_grad()\n","    # context manager to prevent a computational graph from being built.\n","    with torch.no_grad():\n","      for w in params:\n","        if w.requires_grad:\n","          w -= learning_rate * w.grad\n","\n","          # Manually zero the gradients after running the backward pass\n","          w.grad.zero_()\n","\n","    if t % 100 == 0 or t == len(loader_train)-1:\n","      print('Iteration %d, loss = %.4f' % (t, loss.item()))\n","      acc = check_accuracy_part2(loader_val, model_fn, params)\n","  return acc"],"execution_count":12,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"gkTNXK9cIMA6"},"source":["### BareBones PyTorch: Train a Two-Layer Network\n","Now we are ready to run the training loop. We need to explicitly allocate tensors for the fully connected weights, `w1` and `w2`. \n","\n","Each minibatch of CIFAR has 64 examples, so the tensor shape is `[64, 3, 32, 32]`. \n","\n","After flattening, `x` shape should be `[64, 3 * 32 * 32]`. This will be the size of the second dimension of `w1`. \n","The first dimension of `w1` is the hidden layer size, which will also be the second dimension of `w2`. \n","\n","Finally, the output of the network is a 10-dimensional vector that represents the probability distribution over 10 classes. \n","\n","You don't need to tune any hyperparameters but you should see accuracies around 40% after training for one epoch."]},{"cell_type":"code","metadata":{"id":"OSBSy0JTIMA8","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456270533,"user_tz":-60,"elapsed":17720,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"45993e52-7bb8-464e-c7bc-3f5c5b3c8d4a"},"source":["reset_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","hidden_layer_size = 4000\n","learning_rate = 1e-2\n","\n","w1 = nn.init.kaiming_normal_(torch.empty(hidden_layer_size, C*H*W, dtype=to_float, device='cuda'))\n","w1.requires_grad = True\n","b1 = nn.init.zeros_(torch.empty(hidden_layer_size, dtype=to_float, device='cuda'))\n","b1.requires_grad = True\n","w2 = nn.init.kaiming_normal_(torch.empty(num_classes, hidden_layer_size, dtype=to_float, device='cuda'))\n","w2.requires_grad = True\n","b2 = nn.init.zeros_(torch.empty(num_classes, dtype=to_float, device='cuda'))\n","b2.requires_grad = True\n","\n","_ = train_part2(two_layer_fc, [w1, b1, w2, b2], learning_rate)"],"execution_count":13,"outputs":[{"output_type":"stream","text":["Iteration 0, loss = 3.5134\n","Checking accuracy on the val set\n","Got 153 / 1000 correct (15.30%)\n","Iteration 100, loss = 2.8268\n","Checking accuracy on the val set\n","Got 363 / 1000 correct (36.30%)\n","Iteration 200, loss = 2.1683\n","Checking accuracy on the val set\n","Got 396 / 1000 correct (39.60%)\n","Iteration 300, loss = 1.8409\n","Checking accuracy on the val set\n","Got 419 / 1000 correct (41.90%)\n","Iteration 400, loss = 1.5695\n","Checking accuracy on the val set\n","Got 430 / 1000 correct (43.00%)\n","Iteration 500, loss = 2.1471\n","Checking accuracy on the val set\n","Got 414 / 1000 correct (41.40%)\n","Iteration 600, loss = 1.9083\n","Checking accuracy on the val set\n","Got 404 / 1000 correct (40.40%)\n","Iteration 700, loss = 1.8546\n","Checking accuracy on the val set\n","Got 422 / 1000 correct (42.20%)\n","Iteration 765, loss = 1.5771\n","Checking accuracy on the val set\n","Got 397 / 1000 correct (39.70%)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7K_n_BZPIMBB"},"source":["### BareBones PyTorch: Training a ConvNet\n","\n","Now, it's your turn to  implement `initialize_three_layer_conv_part2` function and run the following cell. \n","\n","You don't need to tune any hyperparameters, but if everything works correctly you should achieve an accuracy around 45% after one epoch."]},{"cell_type":"code","metadata":{"id":"wBRWytEzIMBC","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456280813,"user_tz":-60,"elapsed":27991,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"9639fd37-ae99-43e1-a8d9-e701f9a5fa66"},"source":["reset_seed(0)\n","learning_rate = 3e-3\n","# YOUR_TURN: Impelement the initialize_three_layer_conv_part2 function\n","params = initialize_three_layer_conv_part2(dtype=to_float, device='cuda')\n","acc_hist_part2 = train_part2(three_layer_convnet, params, learning_rate)"],"execution_count":14,"outputs":[{"output_type":"stream","text":["Iteration 0, loss = 2.6007\n","Checking accuracy on the val set\n","Got 109 / 1000 correct (10.90%)\n","Iteration 100, loss = 1.9973\n","Checking accuracy on the val set\n","Got 349 / 1000 correct (34.90%)\n","Iteration 200, loss = 1.7886\n","Checking accuracy on the val set\n","Got 391 / 1000 correct (39.10%)\n","Iteration 300, loss = 1.6608\n","Checking accuracy on the val set\n","Got 416 / 1000 correct (41.60%)\n","Iteration 400, loss = 1.5936\n","Checking accuracy on the val set\n","Got 451 / 1000 correct (45.10%)\n","Iteration 500, loss = 1.6610\n","Checking accuracy on the val set\n","Got 449 / 1000 correct (44.90%)\n","Iteration 600, loss = 1.6296\n","Checking accuracy on the val set\n","Got 468 / 1000 correct (46.80%)\n","Iteration 700, loss = 1.7863\n","Checking accuracy on the val set\n","Got 481 / 1000 correct (48.10%)\n","Iteration 765, loss = 1.2417\n","Checking accuracy on the val set\n","Got 463 / 1000 correct (46.30%)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"YXGdsYInIMBH"},"source":["# Part III. PyTorch Module API\n","\n","Barebone PyTorch requires that we track all the parameter tensors by hand. This is fine for small networks with a few tensors, but it would be extremely inconvenient and error-prone to track tens or hundreds of tensors in larger networks.\n","\n","PyTorch provides the `nn.Module` API for you to define arbitrary network architectures, while tracking every learnable parameters for you. In Part II, we implemented SGD ourselves. PyTorch also provides the `torch.optim` package that implements all the common optimizers, such as RMSProp, Adagrad, and Adam. It even supports approximate second-order methods like L-BFGS! You can refer to the [doc](http://pytorch.org/docs/master/optim.html) for the exact specifications of each optimizer.\n","\n","To use the Module API, follow the steps below:\n","\n","1. Subclass `nn.Module`. Give your network class an intuitive name like `TwoLayerFC`. \n","\n","2. In the constructor `__init__()`, define all the layers you need as class attributes. Layer objects like `nn.Linear` and `nn.Conv2d` are themselves `nn.Module` subclasses and contain learnable parameters, so that you don't have to instantiate the raw tensors yourself. `nn.Module` will track these internal parameters for you. Refer to the [doc](http://pytorch.org/docs/master/nn.html) to learn more about the dozens of builtin layers. **Warning**: don't forget to call the `super().__init__()` first!\n","\n","3. In the `forward()` method, define the *connectivity* of your network. You should use the attributes defined in `__init__` as function calls that take tensor as input and output the \"transformed\" tensor. Do *not* create any new layers with learnable parameters in `forward()`! All of them must be declared upfront in `__init__`. \n","\n","After you define your Module subclass, you can instantiate it as an object and call it just like the NN forward function in part II.\n","\n","### Module API: Two-Layer Network\n","Here is a concrete example of a 2-layer fully connected network.\n","We use `nn.init.kaiming_normal_` to initialize weights using Kaiming initialization, and `nn.init.zeros_` to initialize biases."]},{"cell_type":"code","metadata":{"id":"2Ue0_Cf1IMBJ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456280814,"user_tz":-60,"elapsed":27983,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"50da9f6f-2032-4b92-8468-67f9298265e6"},"source":["class TwoLayerFC(nn.Module):\n","  def __init__(self, input_size, hidden_size, num_classes):\n","    super().__init__()\n","    # assign layer objects to class attributes\n","    self.fc1 = nn.Linear(input_size, hidden_size)\n","    self.fc2 = nn.Linear(hidden_size, num_classes)\n","    # nn.init package contains convenient initialization methods\n","    # https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_ \n","    nn.init.kaiming_normal_(self.fc1.weight)\n","    nn.init.kaiming_normal_(self.fc2.weight)\n","    nn.init.zeros_(self.fc1.bias)\n","    nn.init.zeros_(self.fc2.bias)\n","  \n","  def forward(self, x):\n","    # forward always defines connectivity\n","    x = flatten(x)\n","    scores = self.fc2(F.relu(self.fc1(x)))\n","    return scores\n","\n","def test_TwoLayerFC():\n","  input_size = 3*16*16\n","  x = torch.zeros((64, input_size), dtype=to_float)  # minibatch size 64, feature dimension 3*16*16\n","  model = TwoLayerFC(input_size, 42, 10)\n","  scores = model(x)\n","  print('Architecture:')\n","  print(model) # printing `nn.Module` shows the architecture of the module.\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","test_TwoLayerFC()"],"execution_count":15,"outputs":[{"output_type":"stream","text":["Architecture:\n","TwoLayerFC(\n","  (fc1): Linear(in_features=768, out_features=42, bias=True)\n","  (fc2): Linear(in_features=42, out_features=10, bias=True)\n",")\n","Output size: [64, 10]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"CruYc2HJIMBP"},"source":["### Module API: Three-Layer ConvNet\n","It's your turn to implement a 3-layer ConvNet followed by a fully connected layer. \n","\n","After you implement the `ThreeLayerConvNet`, the `test_ThreeLayerConvNet` function will run your implementation; it should print `(64, 10)` for the shape of the output scores."]},{"cell_type":"code","metadata":{"id":"R58EqBTYIMBU","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456280815,"user_tz":-60,"elapsed":27975,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"44bb2a55-d7f4-4ce1-b36f-48e092e61599"},"source":["def test_ThreeLayerConvNet():\n","  x = torch.zeros((64, 3, 32, 32), dtype=to_float)  # minibatch size 64, image size [3, 32, 32]\n","  # YOUR_TURN: Impelement the functions in ThreeLayerConvNet class\n","  model = ThreeLayerConvNet(in_channel=3, channel_1=12, channel_2=8, num_classes=10)\n","  scores = model(x)\n","  print(model) # printing `nn.Module` shows the architecture of the module.\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","test_ThreeLayerConvNet()"],"execution_count":16,"outputs":[{"output_type":"stream","text":["ThreeLayerConvNet(\n","  (conv1): Conv2d(3, 12, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n","  (conv2): Conv2d(12, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","  (fc): Linear(in_features=8192, out_features=10, bias=True)\n",")\n","Output size: [64, 10]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"3efFpwV_IMBZ"},"source":["### Module API: Check Accuracy\n","Given the validation or test set, we can check the classification accuracy of a neural network. \n","\n","This version is slightly different from the one in part II. You don't manually pass in the parameters anymore."]},{"cell_type":"code","metadata":{"id":"LpgKJLVbIMBb","executionInfo":{"status":"ok","timestamp":1617456280816,"user_tz":-60,"elapsed":27967,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["def check_accuracy_part34(loader, model):\n","  if loader.dataset.train:\n","    print('Checking accuracy on validation set')\n","  else:\n","    print('Checking accuracy on test set')   \n","  num_correct = 0\n","  num_samples = 0\n","  model.eval()  # set model to evaluation mode\n","  with torch.no_grad():\n","    for x, y in loader:\n","      x = x.to(device='cuda', dtype=to_float)  # move to device, e.g. GPU\n","      y = y.to(device='cuda', dtype=to_long)\n","      scores = model(x)\n","      _, preds = scores.max(1)\n","      num_correct += (preds == y).sum()\n","      num_samples += preds.size(0)\n","    acc = float(num_correct) / num_samples\n","    print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))\n","  return acc"],"execution_count":17,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-tmOMi8SIMBj"},"source":["### Module API: Training Loop\n","We also use a slightly different training loop. Rather than updating the values of the weights ourselves, we use an Optimizer object from the `torch.optim` package, which abstract the notion of an optimization algorithm and provides implementations of most of the algorithms commonly used to optimize neural networks."]},{"cell_type":"code","metadata":{"id":"HLJjvtu1IMBm","executionInfo":{"status":"ok","timestamp":1617456280816,"user_tz":-60,"elapsed":27961,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["def adjust_learning_rate(optimizer, lrd, epoch, schedule):\n","  \"\"\"\n","  Multiply lrd to the learning rate if epoch is in schedule\n","  \n","  Inputs:\n","  - optimizer: An Optimizer object we will use to train the model\n","  - lrd: learning rate decay; a factor multiplied at scheduled epochs\n","  - epochs: the current epoch number\n","  - schedule: the list of epochs that requires learning rate update\n","  \n","  Returns: Nothing, but learning rate might be updated\n","  \"\"\"\n","  if epoch in schedule:\n","    for param_group in optimizer.param_groups:\n","      print('lr decay from {} to {}'.format(param_group['lr'], param_group['lr'] * lrd))\n","      param_group['lr'] *= lrd\n","\n","def train_part345(model, optimizer, epochs=1, learning_rate_decay=.1, schedule=[], verbose=True):\n","  \"\"\"\n","  Train a model on CIFAR-10 using the PyTorch Module API.\n","  \n","  Inputs:\n","  - model: A PyTorch Module giving the model to train.\n","  - optimizer: An Optimizer object we will use to train the model\n","  - epochs: (Optional) A Python integer giving the number of epochs to train for\n","  \n","  Returns: Nothing, but prints model accuracies during training.\n","  \"\"\"\n","  model = model.to(device='cuda')  # move the model parameters to CPU/GPU\n","  num_iters = epochs * len(loader_train)\n","  print_every = 100\n","  if verbose:\n","    num_prints = num_iters // print_every + 1\n","  else:\n","    num_prints = epochs\n","  acc_history = torch.zeros(num_prints, dtype=to_float)\n","  iter_history = torch.zeros(num_prints, dtype=to_long)\n","  for e in range(epochs):\n","    \n","    adjust_learning_rate(optimizer, learning_rate_decay, e, schedule)\n","    \n","    for t, (x, y) in enumerate(loader_train):\n","      model.train()  # put model to training mode\n","      x = x.to(device='cuda', dtype=to_float)  # move to device, e.g. GPU\n","      y = y.to(device='cuda', dtype=to_long)\n","\n","      scores = model(x)\n","      loss = F.cross_entropy(scores, y)\n","\n","      # Zero out all of the gradients for the variables which the optimizer\n","      # will update.\n","      optimizer.zero_grad()\n","\n","      # This is the backwards pass: compute the gradient of the loss with\n","      # respect to each  parameter of the model.\n","      loss.backward()\n","\n","      # Actually update the parameters of the model using the gradients\n","      # computed by the backwards pass.\n","      optimizer.step()\n","\n","      tt = t + e * len(loader_train)\n","\n","      if verbose and (tt % print_every == 0 or (e == epochs-1 and t == len(loader_train)-1)):\n","        print('Epoch %d, Iteration %d, loss = %.4f' % (e, tt, loss.item()))\n","        acc = check_accuracy_part34(loader_val, model)\n","        acc_history[tt // print_every] = acc\n","        iter_history[tt // print_every] = tt\n","        print()\n","      elif not verbose and (t == len(loader_train)-1):\n","        print('Epoch %d, Iteration %d, loss = %.4f' % (e, tt, loss.item()))\n","        acc = check_accuracy_part34(loader_val, model)\n","        acc_history[e] = acc\n","        iter_history[e] = tt\n","        print()\n","  return acc_history, iter_history"],"execution_count":18,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uFQs2WWKIMBu"},"source":["### Module API: Train a Two-Layer Network\n","Now we are ready to run the training loop. In contrast to part II, we don't explicitly allocate parameter tensors anymore.\n","\n","Simply pass the input size, hidden layer size, and number of classes (i.e. output size) to the constructor of `TwoLayerFC`. \n","\n","You also need to define an optimizer that tracks all the learnable parameters inside `TwoLayerFC`.\n","\n","You don't need to tune any hyperparameters or implement anything, but you should see model accuracies around 40% after training for one epoch."]},{"cell_type":"code","metadata":{"id":"v4Od-a6_IMBv","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456290900,"user_tz":-60,"elapsed":38038,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"f05a0bfc-e8bd-40c0-914c-84e699aada2e"},"source":["reset_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","hidden_layer_size = 4000\n","learning_rate = 1e-2\n","weight_decay = 1e-4\n","\n","model = TwoLayerFC(C*H*W, hidden_layer_size, num_classes)\n","\n","optimizer = optim.SGD(model.parameters(), lr=learning_rate,\n","                      weight_decay=weight_decay)\n","\n","_ = train_part345(model, optimizer)"],"execution_count":19,"outputs":[{"output_type":"stream","text":["Epoch 0, Iteration 0, loss = 3.3988\n","Checking accuracy on validation set\n","Got 139 / 1000 correct (13.90)\n","\n","Epoch 0, Iteration 100, loss = 2.9726\n","Checking accuracy on validation set\n","Got 328 / 1000 correct (32.80)\n","\n","Epoch 0, Iteration 200, loss = 2.1085\n","Checking accuracy on validation set\n","Got 336 / 1000 correct (33.60)\n","\n","Epoch 0, Iteration 300, loss = 2.1709\n","Checking accuracy on validation set\n","Got 432 / 1000 correct (43.20)\n","\n","Epoch 0, Iteration 400, loss = 1.9805\n","Checking accuracy on validation set\n","Got 437 / 1000 correct (43.70)\n","\n","Epoch 0, Iteration 500, loss = 1.7867\n","Checking accuracy on validation set\n","Got 447 / 1000 correct (44.70)\n","\n","Epoch 0, Iteration 600, loss = 2.1035\n","Checking accuracy on validation set\n","Got 473 / 1000 correct (47.30)\n","\n","Epoch 0, Iteration 700, loss = 1.6729\n","Checking accuracy on validation set\n","Got 472 / 1000 correct (47.20)\n","\n","Epoch 0, Iteration 765, loss = 1.6794\n","Checking accuracy on validation set\n","Got 413 / 1000 correct (41.30)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"J_35SZYHIMB4"},"source":["### Module API: Train a Three-Layer ConvNet\n","You should now use the Module API to train a three-layer ConvNet on CIFAR. This should look very similar to training the two-layer network! You don't need to tune any hyperparameters, but you should achieve above around 45% after training for one epoch.\n","\n","Implement the `initialize_three_layer_conv_part3` function, you should train the model using stochastic gradient descent without momentum, with L2 weight decay of 1e-4."]},{"cell_type":"code","metadata":{"id":"-bIRiwOJIMB6","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456300965,"user_tz":-60,"elapsed":48094,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"032a311c-cf39-4f09-c468-3a6b7243b891"},"source":["reset_seed(0)\n","# YOUR_TURN: Impelement initialize_three_layer_conv_part3\n","model, optimizer = initialize_three_layer_conv_part3()\n","acc_hist_part3, _ = train_part345(model, optimizer)"],"execution_count":20,"outputs":[{"output_type":"stream","text":["Epoch 0, Iteration 0, loss = 3.2691\n","Checking accuracy on validation set\n","Got 126 / 1000 correct (12.60)\n","\n","Epoch 0, Iteration 100, loss = 1.9332\n","Checking accuracy on validation set\n","Got 319 / 1000 correct (31.90)\n","\n","Epoch 0, Iteration 200, loss = 1.7349\n","Checking accuracy on validation set\n","Got 372 / 1000 correct (37.20)\n","\n","Epoch 0, Iteration 300, loss = 1.6909\n","Checking accuracy on validation set\n","Got 409 / 1000 correct (40.90)\n","\n","Epoch 0, Iteration 400, loss = 1.4227\n","Checking accuracy on validation set\n","Got 416 / 1000 correct (41.60)\n","\n","Epoch 0, Iteration 500, loss = 1.5768\n","Checking accuracy on validation set\n","Got 435 / 1000 correct (43.50)\n","\n","Epoch 0, Iteration 600, loss = 1.3388\n","Checking accuracy on validation set\n","Got 440 / 1000 correct (44.00)\n","\n","Epoch 0, Iteration 700, loss = 1.5642\n","Checking accuracy on validation set\n","Got 457 / 1000 correct (45.70)\n","\n","Epoch 0, Iteration 765, loss = 1.7758\n","Checking accuracy on validation set\n","Got 470 / 1000 correct (47.00)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"qeTdi4-xIMB_"},"source":["# Part IV. PyTorch Sequential API\n","\n","Part III introduced the PyTorch Module API, which allows you to define arbitrary learnable layers and their connectivity. \n","\n","For simple models like a stack of feed forward layers, you still need to go through 3 steps: subclass `nn.Module`, assign layers to class attributes in `__init__`, and call each layer one by one in `forward()`. Is there a more convenient way? \n","\n","Fortunately, PyTorch provides a container Module called `nn.Sequential`, which merges the above steps into one. It is not as flexible as `nn.Module`, because you cannot specify more complex topology than a feed-forward stack, but it's good enough for many use cases."]},{"cell_type":"markdown","metadata":{"id":"ZZq-3hjxeYr4"},"source":["### Sequential API: Two-Layer Network\n","Let's see how to rewrite our two-layer fully connected network example with `nn.Sequential`, and train it using the training loop defined above.\n","Here, let's skip weight initialization for simplicity;\n","with a more advanced optimizer than the naive SGD, the default initialization provided in `torch.nn` is good enough for shallow networks.\n","\n","Again, you don't need to tune any hyperparameters or implement anything here, but you should achieve around 40% accuracy after one epoch of training."]},{"cell_type":"code","metadata":{"id":"9smkhciWIMCC","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456311021,"user_tz":-60,"elapsed":58141,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"176fb1b3-bdcc-42de-c032-0f0a8cdd88cc"},"source":["reset_seed(0)\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","hidden_layer_size = 4000\n","learning_rate = 1e-2\n","weight_decay = 1e-4\n","momentum = 0.5\n","\n","# To give a specific name to each module, use OrderedDict.\n","model = nn.Sequential(OrderedDict([\n","  ('flatten', Flatten()),\n","  ('fc1', nn.Linear(C*H*W, hidden_layer_size)),\n","  ('relu1', nn.ReLU()),\n","  ('fc2', nn.Linear(hidden_layer_size, num_classes)),\n","]))\n","\n","print('Architecture:')\n","print(model) # printing `nn.Module` shows the architecture of the module.\n","\n","# you can use Nesterov momentum in optim.SGD\n","optimizer = optim.SGD(model.parameters(), lr=learning_rate, \n","                      weight_decay=weight_decay,\n","                      momentum=momentum, nesterov=True)\n","\n","_ = train_part345(model, optimizer)"],"execution_count":21,"outputs":[{"output_type":"stream","text":["Architecture:\n","Sequential(\n","  (flatten): Flatten()\n","  (fc1): Linear(in_features=3072, out_features=4000, bias=True)\n","  (relu1): ReLU()\n","  (fc2): Linear(in_features=4000, out_features=10, bias=True)\n",")\n","Epoch 0, Iteration 0, loss = 2.3772\n","Checking accuracy on validation set\n","Got 137 / 1000 correct (13.70)\n","\n","Epoch 0, Iteration 100, loss = 1.7474\n","Checking accuracy on validation set\n","Got 394 / 1000 correct (39.40)\n","\n","Epoch 0, Iteration 200, loss = 1.7006\n","Checking accuracy on validation set\n","Got 409 / 1000 correct (40.90)\n","\n","Epoch 0, Iteration 300, loss = 1.6135\n","Checking accuracy on validation set\n","Got 444 / 1000 correct (44.40)\n","\n","Epoch 0, Iteration 400, loss = 1.6331\n","Checking accuracy on validation set\n","Got 457 / 1000 correct (45.70)\n","\n","Epoch 0, Iteration 500, loss = 1.5453\n","Checking accuracy on validation set\n","Got 460 / 1000 correct (46.00)\n","\n","Epoch 0, Iteration 600, loss = 1.4802\n","Checking accuracy on validation set\n","Got 470 / 1000 correct (47.00)\n","\n","Epoch 0, Iteration 700, loss = 1.6623\n","Checking accuracy on validation set\n","Got 451 / 1000 correct (45.10)\n","\n","Epoch 0, Iteration 765, loss = 1.2971\n","Checking accuracy on validation set\n","Got 454 / 1000 correct (45.40)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"5YMYfFUGIMCH"},"source":["### Sequential API: Three-Layer ConvNet\n","Now, it's your turn to use `nn.Sequential` to define and train a three-layer ConvNet with the same architecture we used in Part III. \n","\n","Implement `initialize_three_layer_conv_part4` and  you should see accuracy around 50% after one epoch of training.                      \n","\t"]},{"cell_type":"code","metadata":{"id":"Q2To2-mtIMCJ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456321594,"user_tz":-60,"elapsed":68705,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"278e94c1-76f8-40bc-8690-079090da4a8d"},"source":["reset_seed(0)\n","\n","# YOUR_TURN: Impelement initialize_three_layer_conv_part4\n","model, optimizer = initialize_three_layer_conv_part4()\n","print('Architecture:')\n","print(model) # printing `nn.Module` shows the architecture of the module.\n","\n","acc_hist_part4, _ = train_part345(model, optimizer)"],"execution_count":22,"outputs":[{"output_type":"stream","text":["Architecture:\n","Sequential(\n","  (0): Conv2d(3, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n","  (1): ReLU()\n","  (2): Conv2d(32, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","  (3): ReLU()\n","  (4): Flatten()\n","  (5): Linear(in_features=16384, out_features=10, bias=True)\n",")\n","Epoch 0, Iteration 0, loss = 2.2960\n","Checking accuracy on validation set\n","Got 136 / 1000 correct (13.60)\n","\n","Epoch 0, Iteration 100, loss = 1.6666\n","Checking accuracy on validation set\n","Got 382 / 1000 correct (38.20)\n","\n","Epoch 0, Iteration 200, loss = 1.4597\n","Checking accuracy on validation set\n","Got 479 / 1000 correct (47.90)\n","\n","Epoch 0, Iteration 300, loss = 1.7122\n","Checking accuracy on validation set\n","Got 482 / 1000 correct (48.20)\n","\n","Epoch 0, Iteration 400, loss = 1.6089\n","Checking accuracy on validation set\n","Got 489 / 1000 correct (48.90)\n","\n","Epoch 0, Iteration 500, loss = 1.4318\n","Checking accuracy on validation set\n","Got 499 / 1000 correct (49.90)\n","\n","Epoch 0, Iteration 600, loss = 1.5030\n","Checking accuracy on validation set\n","Got 543 / 1000 correct (54.30)\n","\n","Epoch 0, Iteration 700, loss = 1.3412\n","Checking accuracy on validation set\n","Got 542 / 1000 correct (54.20)\n","\n","Epoch 0, Iteration 765, loss = 1.3171\n","Checking accuracy on validation set\n","Got 535 / 1000 correct (53.50)\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"CM6FuhGPIMCO"},"source":["# Part V. ResNet for CIFAR-10\n","\n","In this section, you are going to implement [ResNet](https://arxiv.org/abs/1512.03385), one of the state-of-the-art CNN architecture.\n","Specifically, you are going to implement a variation of ResNet called [PreResNet](https://arxiv.org/abs/1603.05027), which locates activation before each convolutional layer (so called pre-activation).\n","You are going to first implement a plain building block, residual block, and then bottleneck block for really deep networks.\n","Finally, you will implement your own ResNet using those blocks.\n","\n","Throughout this part, we will follow the PyTorch default weight initialization for conciseness."]},{"cell_type":"markdown","metadata":{"id":"f8pgYlKN9NLH"},"source":["## Plain block\n","\n","First, let's implement a plain block without residual connection.\n","PreResNet has a different order of layers from the previously implemented ones;\n","BatchNorm and ReLU precedes Conv.\n","The name of PreResNet comes form this pre-activation architecture.\n","Here, for downsampling, we don't introduce MaxPool layers explicitly, but use stride 2 in the first Conv layer in the block.\n","\n","Concretely, a plain block accepts a feature map of shape $C_{in} \\times H_{in} \\times W_{out}$ and produces a feature map of shape $C_{out} \\times H_{out} \\times W_{out}$. If the block performs downsampling, then $W_{out}=W_{in}/2$ and $H_{out}=H_{in}/2$; otherwise $H_{out}=H_{in}$ and $W_{out}=W_{in}$. The plain block consists of the following six layers in order:\n","\n","1. Spatial Batch normalization\n","2. ReLU\n","3. Convolutional layer with `Cout` 3x3 filters, zero-padding of 1, and stride 2 if downsampling; otherwise stride 1\n","4. Spatial Batch normalization\n","5. ReLU\n","6. Convolutional layer with `Cout` 3x3 filters, with zero-padding of 1\n","\n","Implement the `PlainBlock.__init__` function and run the following cell. You should see a message indicating that your implementation is correct.\n"]},{"cell_type":"code","metadata":{"id":"SofEF-vyAekS","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456321595,"user_tz":-60,"elapsed":68697,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"8ee8db36-ae12-4183-bb9e-2c23991b3dd1"},"source":["data = torch.zeros(2, 3, 5, 6)\n","# YOUR_TURN: Impelement PlainBlock.__init__\n","model = PlainBlock(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of PlainBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of PlainBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))\n","\n","data = torch.zeros(2, 3, 5, 6)\n","# YOUR_TURN: Impelement PlainBlock.__init__\n","model = PlainBlock(3, 10, downsample=True)\n","if list(model(data).shape) == [2, 10, 3, 3]:\n","  print('The output of PlainBlock with downsampling has a *correct* dimension!')\n","else:\n","  print('The output of PlainBlock with downsampling has an *incorrect* dimension! expected:', [2, 10, 3, 3], 'got:', list(model(data).shape))"],"execution_count":23,"outputs":[{"output_type":"stream","text":["The output of PlainBlock without downsampling has a *correct* dimension!\n","The output of PlainBlock with downsampling has a *correct* dimension!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ASV5NdNADo3F"},"source":["## Residual block\n","\n","Next, let's implement a residual block.\n","A residual block adds a residual connection to a plain block. Let $\\mathcal{F}$ be a plain block; then the residual version  $\\mathcal{R}$ of the plain block $\\mathcal{F}$ computes:\n","\n","$\\mathcal{R}(x) = \\mathcal{F}(x) + x$\n","\n","However, this implementation will only work if the output from the plain block $\\mathcal{F}(x)$ has the same shape as the input $x$. Based on the plain block that we implemented above, there are two cases where the output of the plain block can have a different shape than the input:\n","\n","1. The number of output channels $C_{out}$ is different from the number of input channels $C_{in}$\n","2. The plain block $\\mathcal{F}$ performs spatial downsampling\n","\n","To deal with these cases, we need generalize our definition of the residual block and add a *shortcut connection* $\\mathcal{G}$:\n","\n","$\\mathcal{R}(x) = \\mathcal{F}(x) + \\mathcal{G}(x)$\n","\n","There are three cases for the shortcut connection $\\mathcal{G}$:\n","\n","1. If $C_{in}=C_{out}$ and $\\mathcal{F}$ does not perform downsampling, then $\\mathcal{F}(x)$ will have the same shape as $x$, so $\\mathcal{G}$ is the identity function: $\\mathcal{G}(x) = x$\n","2. If $C_{in} \\neq C_{out}$ and $\\mathcal{F}$ does not downsample, then $\\mathcal{G}$ is a 1x1 convolution with $C_out$ filters and stride 1.\n","3. If $\\mathcal{F}$ downsamples, then $\\mathcal{G}$ is a 1x1 convolution with $C_{out}$ filters and stride 2.\n","\n","Implement the `ResidualBlock.__init__` function and run the following cell. You should see a message indicating that your implementation is correct."]},{"cell_type":"code","metadata":{"id":"TMJ3-eI3Do3M","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456321596,"user_tz":-60,"elapsed":68688,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"c4a623d2-fad7-4aab-8e6c-d2e99e69f977"},"source":["data = torch.zeros(2, 3, 5, 6)\n","# YOUR_TURN: Impelement ResidualBlock.__init__\n","model = ResidualBlock(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of ResidualBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))\n","\n","data = torch.zeros(2, 3, 5, 6)\n","# YOUR_TURN: Impelement ResidualBlock.__init__\n","model = ResidualBlock(3, 10, downsample=True)\n","if list(model(data).shape) == [2, 10, 3, 3]:\n","  print('The output of ResidualBlock with downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock with downsampling has an *incorrect* dimension! expected:', [2, 10, 3, 3], 'got:', list(model(data).shape))"],"execution_count":24,"outputs":[{"output_type":"stream","text":["The output of ResidualBlock without downsampling has a *correct* dimension!\n","The output of ResidualBlock with downsampling has a *correct* dimension!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"wrEzQeTBvNso"},"source":["## Residual stage\n","\n","So far, you implemented micro layers, which consists of several convolutional laters.\n","To efficiently build a deep neural network, we define a macro layer by repeating the micro layers.\n","\n","For your convenience, we provide the implementation in `pytorch_autograd_and_nn.py`.\n","\n","```\n","class ResNetStage(nn.Module):\n","  def __init__(self, Cin, Cout, num_blocks, downsample=True,\n","               block=ResidualBlock):\n","    super().__init__()\n","    blocks = [block(Cin, Cout, downsample)]\n","    for _ in range(num_blocks - 1):\n","      blocks.append(block(Cout, Cout))\n","    self.net = nn.Sequential(*blocks)\n","  \n","  def forward(self, x):\n","    return self.net(x)\n","```\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"I21i5J3AnbhM","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456321596,"user_tz":-60,"elapsed":68677,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"1d078e74-ca2e-40bd-e73a-9e92f8e3443f"},"source":["print('Plain block stage:')\n","print(ResNetStage(3, 4, 2, block=PlainBlock))\n","print('Residual block stage:')\n","print(ResNetStage(3, 4, 2, block=ResidualBlock))"],"execution_count":25,"outputs":[{"output_type":"stream","text":["Plain block stage:\n","ResNetStage(\n","  (net): Sequential(\n","    (0): PlainBlock(\n","      (net): Sequential(\n","        (0): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (1): ReLU()\n","        (2): Conv2d(3, 4, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n","        (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (4): ReLU()\n","        (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","      )\n","    )\n","    (1): PlainBlock(\n","      (net): Sequential(\n","        (0): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (1): ReLU()\n","        (2): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (4): ReLU()\n","        (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","      )\n","    )\n","  )\n",")\n","Residual block stage:\n","ResNetStage(\n","  (net): Sequential(\n","    (0): ResidualBlock(\n","      (block): PlainBlock(\n","        (net): Sequential(\n","          (0): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (1): ReLU()\n","          (2): Conv2d(3, 4, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n","          (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (4): ReLU()\n","          (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        )\n","      )\n","      (shortcut): Conv2d(3, 4, kernel_size=(1, 1), stride=(2, 2))\n","    )\n","    (1): ResidualBlock(\n","      (block): PlainBlock(\n","        (net): Sequential(\n","          (0): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (1): ReLU()\n","          (2): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","          (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (4): ReLU()\n","          (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        )\n","      )\n","      (shortcut): Identity()\n","    )\n","  )\n",")\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7l3-PNR9mlIb"},"source":["## Residual stem\n","\n","A \"stem\" layer is required at the beginning of the network, which increases the number of channels while keeping the other dimensions.\n","\n","For your convenience, we provide the implementation in `pytorch_autograd_and_nn.py`.\n","\n","\n","\n","```\n","class ResNetStem(nn.Module):\n","  def __init__(self, Cin=3, Cout=8):\n","    super().__init__()\n","    layers = [\n","        nn.Conv2d(Cin, Cout, kernel_size=3, padding=1, stride=1),\n","        nn.ReLU(),\n","    ]\n","    self.net = nn.Sequential(*layers)\n","    \n","  def forward(self, x):\n","    return self.net(x)\n","```\n","\n"]},{"cell_type":"code","metadata":{"id":"UGzh0oVxm2Aw","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456321597,"user_tz":-60,"elapsed":68667,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"33bb8351-2c92-438f-e212-52bcc5259c3b"},"source":["data = torch.zeros(2, 3, 5, 6)\n","model = ResNetStem(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of ResNetStem without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResNetStem without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))"],"execution_count":26,"outputs":[{"output_type":"stream","text":["The output of ResNetStem without downsampling has a *correct* dimension!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"QGzn4Gp_sYBp"},"source":["## ResNet class\n","\n","Now, it is time to design the ResNet class using the blocks you implemented above!\n","\n","For general applicability, the class will get a dictionary of the architecture specification as an input, and parse it to build a CNN.\n","\n","Here we provide a couple of examples of specification;\n","`networks` is a collection of pre-defined network specifications, where each can be called by `get_resnet(key)`, where key is the name of the network, e.g., `get_resnet('resnet32')` will return a ResNet with 32 layers.\n","\n","Each specification consists of multiple tuples which correspond to a macro block (`ResNetStage`), and the values in each tuple implies `(num_in_channels, num_out_channels, num_blocks, do_downsample)`.\n","\n","To avoid dependency on the size of the input, ResNet has an average pooling at the end of the convolutional part, such that the size of the input tensor to the linear layer is always `(batch_size, stage_args[-1][1])`.\n","You may want to add an average pooling layer (`nn.AvgPool2d`), but it requires to know the size of the input.\n","Can you relax this requirement?\n","\n","**Hint**: You can perform average pooling in `forward`.\n","\n","Implement `ResNet.__init__` and `ResNet.forward` and train it on CIFAR."]},{"cell_type":"code","metadata":{"id":"1iOOBoSgs-0X","executionInfo":{"status":"ok","timestamp":1617456321598,"user_tz":-60,"elapsed":68658,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["# example of specifications\n","networks = {\n","  'plain32': {\n","    'block': PlainBlock,\n","    'stage_args': [\n","      (8, 8, 5, False),\n","      (8, 16, 5, True),\n","      (16, 32, 5, True),\n","    ]\n","  },\n","  'resnet32': {\n","    'block': ResidualBlock,\n","    'stage_args': [\n","      (8, 8, 5, False),\n","      (8, 16, 5, True),\n","      (16, 32, 5, True),\n","    ]\n","  },\n","}\n","\n","def get_resnet(name):\n","  # YOUR_TURN: Impelement ResNet.__init__ and ResNet.forward\n","  return ResNet(**networks[name])"],"execution_count":27,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uqhp641H0P9Z"},"source":["## Train your model!\n","\n","Now let's train a few epochs of plain and residual networks with 32 layers on CIFAR.\n","You will see that deep non-residual networks don't converge well.\n","\n","**Caution: This takes a long time!**\n","\n","**Disclaimer: The performance of PreResNet-32 you will see here (~ 80%) would be lower than the best performance this model can achieve, because the convergence requires much more training.**"]},{"cell_type":"code","metadata":{"id":"hRyYVBn60A58","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456714354,"user_tz":-60,"elapsed":461406,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"8db7e69d-9ffe-4838-f844-1167a5f1151e"},"source":["names = ['plain32', 'resnet32']\n","acc_history_dict = {}\n","iter_history_dict = {}\n","for name in names:\n","  reset_seed(0)\n","  print(name, '\\n')\n","  model = get_resnet(name)\n","#   init_module(model)\n","  \n","  optimizer = optim.SGD(model.parameters(), lr=1e-2, momentum=.9, weight_decay=1e-4)\n","\n","  acc_history, iter_history = train_part345(model, optimizer, epochs=10, schedule=[6, 8], verbose=False)\n","  acc_history_dict[name] = acc_history\n","  iter_history_dict[name] = iter_history"],"execution_count":28,"outputs":[{"output_type":"stream","text":["plain32 \n","\n","Epoch 0, Iteration 765, loss = 1.4826\n","Checking accuracy on validation set\n","Got 406 / 1000 correct (40.60)\n","\n","Epoch 1, Iteration 1531, loss = 1.3971\n","Checking accuracy on validation set\n","Got 472 / 1000 correct (47.20)\n","\n","Epoch 2, Iteration 2297, loss = 1.1368\n","Checking accuracy on validation set\n","Got 528 / 1000 correct (52.80)\n","\n","Epoch 3, Iteration 3063, loss = 1.0539\n","Checking accuracy on validation set\n","Got 565 / 1000 correct (56.50)\n","\n","Epoch 4, Iteration 3829, loss = 1.4275\n","Checking accuracy on validation set\n","Got 547 / 1000 correct (54.70)\n","\n","Epoch 5, Iteration 4595, loss = 1.0718\n","Checking accuracy on validation set\n","Got 598 / 1000 correct (59.80)\n","\n","lr decay from 0.01 to 0.001\n","Epoch 6, Iteration 5361, loss = 0.8453\n","Checking accuracy on validation set\n","Got 695 / 1000 correct (69.50)\n","\n","Epoch 7, Iteration 6127, loss = 0.8736\n","Checking accuracy on validation set\n","Got 694 / 1000 correct (69.40)\n","\n","lr decay from 0.001 to 0.0001\n","Epoch 8, Iteration 6893, loss = 0.8312\n","Checking accuracy on validation set\n","Got 702 / 1000 correct (70.20)\n","\n","Epoch 9, Iteration 7659, loss = 0.8707\n","Checking accuracy on validation set\n","Got 703 / 1000 correct (70.30)\n","\n","resnet32 \n","\n","Epoch 0, Iteration 765, loss = 1.2323\n","Checking accuracy on validation set\n","Got 551 / 1000 correct (55.10)\n","\n","Epoch 1, Iteration 1531, loss = 1.1267\n","Checking accuracy on validation set\n","Got 550 / 1000 correct (55.00)\n","\n","Epoch 2, Iteration 2297, loss = 0.9909\n","Checking accuracy on validation set\n","Got 642 / 1000 correct (64.20)\n","\n","Epoch 3, Iteration 3063, loss = 0.6681\n","Checking accuracy on validation set\n","Got 702 / 1000 correct (70.20)\n","\n","Epoch 4, Iteration 3829, loss = 0.8296\n","Checking accuracy on validation set\n","Got 728 / 1000 correct (72.80)\n","\n","Epoch 5, Iteration 4595, loss = 0.6406\n","Checking accuracy on validation set\n","Got 771 / 1000 correct (77.10)\n","\n","lr decay from 0.01 to 0.001\n","Epoch 6, Iteration 5361, loss = 0.3877\n","Checking accuracy on validation set\n","Got 806 / 1000 correct (80.60)\n","\n","Epoch 7, Iteration 6127, loss = 0.5694\n","Checking accuracy on validation set\n","Got 807 / 1000 correct (80.70)\n","\n","lr decay from 0.001 to 0.0001\n","Epoch 8, Iteration 6893, loss = 0.6080\n","Checking accuracy on validation set\n","Got 818 / 1000 correct (81.80)\n","\n","Epoch 9, Iteration 7659, loss = 0.6245\n","Checking accuracy on validation set\n","Got 822 / 1000 correct (82.20)\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"-u89CIFfzWWR","colab":{"base_uri":"https://localhost:8080/","height":294},"executionInfo":{"status":"ok","timestamp":1617456714359,"user_tz":-60,"elapsed":461400,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"15b6891c-18c1-4190-c0fb-5cb424787a65"},"source":["plt.title('Val accuracies')\n","for name in names:\n","  plt.plot(iter_history_dict[name], acc_history_dict[name], '-o')\n","plt.legend(names, loc='upper left')\n","plt.xlabel('iterations')\n","plt.ylabel('accuracy')\n","plt.gcf().set_size_inches(9, 4)\n","plt.show()"],"execution_count":29,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjUAAAEWCAYAAACexWadAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xV9f3H8dcne0KAyAhbZcuSFBw4UXEiVIto3bUq1VY7UKmjSqU/W62trXtj60JFBByAq24g7CV7JWFDQgjZ9/v749yEJCQQIDc3uXk/H488cu73nHvu5xxN8uZ7vud7zDmHiIiISEMXFuwCRERERGqDQo2IiIiEBIUaERERCQkKNSIiIhISFGpEREQkJCjUiIiISEhQqBGRGjMzZ2bHB7uO2mZmz5rZ/cGuQ0SOjkKNSCNiZp+Y2bgq2i81sy1mFhGMuoLNOXerc+7Pwa5DRI6OQo1I4zIBuNrMrFL7NcDrzrniINRUK8wsPNg1iEhwKdSINC6TgRbAaaUNZtYMuBh4zcwGmtn3ZpZlZpvN7Ekzi6rJjs3sBjNbbmY5ZrbWzG6ptP5SM1tgZnvMbI2Zne9vb25mr5hZppntNrPJ/vbrzeybSvsou/xlZq+a2TNm9pGZ5QJnmdlFZjbf/xmbzOzBSu8fbGbf+Y9vk5ldX25fD5fb7mJ/rVn+7fuUW3e3mWX4j3OFmQ2pyfkRkcBTqBFpRJxzecBE4NpyzSOBH51zC4ES4LdAMnAyMAT4VQ13vw0vHDUBbgD+YWYnApjZQOA1YAyQBJwOrPe/7z9AHNALaAn84zAO6SpgPJAIfAPk+o8tCbgIGG1mw/01dAQ+Bv4NHAP0AxZU3qGZ9QdeBm7BC4DPAVPMLNrMugG3Az9xziUCQ8sdh4gEmUKNSOMzAbjczGL8r6/1t+Gcm+uc+8E5V+ycW4/3B/2MmuzUOfehc26N8/wPmMH+HqFfAC8752Y653zOuQzn3I9m1ga4ALjVObfbOVfkf29NfeCc+9a/z3zn3JfOucX+14uAN8vVfxXwqXPuTf/n7HTOHRBqgJuB55xzs5xzJc65CUABcBJe6IsGeppZpHNuvXNuzWHUKyIBpFAj0sg4574BdgDDzew4YCDwBoCZdTWzaf5Bw3uAv+D12hySmV1gZj+Y2S4zywIuLPfe9kBVf/zbA7ucc7uP8HA2VaphkJl9YWbbzSwbuLUGNVTWEfi9/9JTlv9Y2gMpzrnVwJ3Ag8A2M3vLzFKOsHYRqWUKNSKN02t4PTRXA9Odc1v97c8APwJdnHNNgD8ClQcVH8DMooH3gMeAVs65JOCjcu/dBBxXxVs3Ac3NLKmKdbl4l6VKP6N1Fdu4Sq/fAKYA7Z1zTYFna1BDVTWNd84llfuKc869CeCce8M5Nxgv/DjgrzXYp4jUAYUakcbpNeAc4Jf4Lz35JQJ7gL1m1h0YXcP9ReFdltkOFJvZBcB55da/BNxgZkPMLMzM2ppZd+fcZrxxLk+bWTMzizSz0/3vWQj0MrN+/ktlD9agjkS8np98/zieq8qtex04x8xGmlmEmbUws35V7OMF4FZ/r4+ZWbx/AHKimXUzs7P9IS4fyAN8NTxHIhJgCjUijZB/vMx3QDxez0apP+AFgRy8P+5v13B/OcBv8AYh7/bvY0q59bPxDx4GsoH/4fV0gHc7eRFeD9E2vMs7OOdWAuOAT4FVeAOBD+VXwDgzywEe8NdTWsNGvEtivwd24Q0S7lvFsaThhb0n/ceyGrjevzoaeATv8t0WvIHNY2tQl4jUAXOucu+tiIiISMOjnhoREREJCQo1IiIiEhIUakRERCQkKNSIiIhISGhwT+RNTk52nTp1CnYZIiIiUgvmzp27wzl3TG3sq8GFmk6dOpGWlhbsMkRERKQWmNmG2tqXLj+JiIhISFCoERERkZCgUCMiIiIhocGNqalKUVER6enp5OfnB7uUBicmJoZ27doRGRkZ7FJERESOSkiEmvT0dBITE+nUqRNmh3ygsPg559i5cyfp6el07tw52OWIiIgclZC4/JSfn0+LFi0UaA6TmdGiRQv1cImISNUWTYR/nAAPJnnfF0089HuCKCR6agAFmiOk8yYiIlVaNBGm/gaK8rzX2Zu81wB9RgavroMImVAjIiJSbyyaCJ+Ng+x0aNoOhjxQP4KAc1BS6AWVojwoztu/XLntk3v2B5pSRXnecdWHY6mCQk2AnXnmmTz22GOkpqZWu81NN93E7373O3r27FntNs8++yxPPfUU4eHhJCQk8Pzzz9OzZ09mzpzJPffcQ2FhIVFRUTz66KOcffbZgTgUERGpiSPp4SgpriJg7IPifO97UX41bfsqhZP8SttU0eZ8R3d82elH9/4AapShZvL8DB6dvoLMrDxSkmIZM7Qbw/u3DVo9L7744iG3ueqqq7j11lsBmDJlCr/73e/45JNPSE5OZurUqaSkpLBkyRKGDh1KRkZGoEsWEZFSzkHebtiTCTlb4OO7qu7h+OB2+OGZqkOIr+jIPjsixvuKjIPI2P1fETGQ0LpSW2zF1xXa4iDSv5+IGK/t1YshJ/PAz2za7shqrQONLtRMnp/B2EmLySsqASAjK4+xkxYDHFWwWb9+Peeffz4DBgxg3rx59OrVi9dee63CNqNHj2bOnDnk5eVx+eWX89BDDwEVe3MSEhK44447mDZtGrGxsXzwwQe0atWKJk2alO0nNze3bCxM//79y9p79epFXl4eBQUFREdHH/GxiIiIX8FeL6jkbN7/tad0eYv3Rz9ni3dJ51BKCiCuebkQUi6MHDRwlAse5d8bEQthAbzf59yHKvY4gVfHkAcC95lHKeRCzUNTl7Isc0+16+dvzKKwpGLXW15RCXe9u4g3Z2+s8j09U5rwp0t6HfKzV6xYwUsvvcSpp57KjTfeyNNPP11h/fjx42nevDklJSUMGTKERYsW0adPnwrb5ObmctJJJzF+/HjuuusuXnjhBe677z4AnnrqKR5//HEKCwv5/PPPD/j89957jxNPPFGBRkTkUIoLYe/WcmFly/6elvJtBVX8PYmMhyZtILENtD8JElt7y6Vt71zvvb+ypu3h6vcCfmi1pvRSWX0cG1SNkAs1h1I50Byq/XC0b9+eU089FYCrr76af/3rXxXWT5w4keeff57i4mI2b97MsmXLDgg1UVFRXHzxxQAMGDCAmTNnlq277bbbuO2223jjjTd4+OGHmTBhQtm6pUuXcvfddzNjxoyjPg4RkaCojcG1Ph/s27m/B6VCz0q5sJK7/cD3hkX6A0prOKY7HHvW/qCS2BoSU7zv0YlwsDtHzx3X4Ho4qtVnZL0OMZWFXKg5VI/KqY98TkZW3gHtbZNiefuWk4/qsyvfHl3+9bp163jssceYM2cOzZo14/rrr69yfpjIyMiy94WHh1NcXHzANqNGjWL06NFlr9PT0xkxYgSvvfYaxx133FEdg4g0MPX1LpvDdajBtc55vSYHBJVyl4D2bIa9W8B34O9N4o/xh5MUaDvgwKCS2AbiWtTO5ZwG2MMRKkIu1BzKmKHdKoypAYiNDGfM0G5Hve+NGzfy/fffc/LJJ/PGG28wePBgpk6dCsCePXuIj4+nadOmbN26lY8//pgzzzyzxvtetWoVXbp0AeDDDz8sW87KyuKiiy7ikUceKeslEpFG4kjusnHOu/vF+cBXsn/Z+Zd9viraym9Xk/eWVLNd5bbS7w4+vrv6wbVfPuKFlqLcA48nuqkXSpq0gc6nHRhUmrSBhFYQXsePgmlgPRyhotGFmtLBwIG4+6lbt2489dRT3HjjjfTs2ZPRo0eXhZq+ffvSv39/unfvXuEyVU09+eSTfPrpp0RGRtKsWbOyS09PPvkkq1evZty4cYwbNw6AGTNm0LJly6M+HhGp5z59sOogMOlmmPa7qoMJLhiVHrmSAmjTB7qef+DYlcTWEBUf7AqlHjHnAvc/uJmdDzwBhAMvOuceqbS+AzABSPJvc49z7qOD7TM1NdWlpaVVaFu+fDk9evSozdIP2/r167n44otZsmRJUOs4EvXh/IlIDe3bBT9+CMs+gNUzq9/upNu8SylW+hW+fzks3BsTckBb6XZWRVtN3htWTZt/uap6SrebcEn1g2t/2/B+r0rNmdlc51z1k7kdhoD11JhZOPAUcC6QDswxsynOuWXlNrsPmOice8bMegIfAZ0CVZOISIO0dzv8OM0LMuu+8npfkjp6A1YLcg7cvml7OP8vdV/n0QilwbUSNIG8/DQQWO2cWwtgZm8BlwLlQ40DSidgaQpUMctPw9CpU6cG2UsjIvVUzhZYPtULMhu+9S4dNT8WTr0Del4KbfrC4ndCJwhocK3UgkCGmrbApnKv04FBlbZ5EJhhZr8G4oFzqtqRmd0M3AzQoUOHWi9URKReyE7fH2Q2/gA4SO4Gp/3BCzKtelW8lTjUgoAG18pRCvZA4SuBV51zfzezk4H/mNkJzlV8MIVz7nngefDG1AShThGRwNi9AZZP8YJM+hyvrdUJcNYfoccwaNn94O9XEBApE8hQkwG0L/e6nb+tvF8A5wM45743sxggGdgWwLpERIJr55r9QSZzvtfWpq/Xy9LjUkg+Prj1iTRQgQw1c4AuZtYZL8yMAq6qtM1GYAjwqpn1AGKAKqZ5FBFp4LavhOUfeEFmi/e8OdoO8AbI9hgGzTsHtz6REBCwUOOcKzaz24HpeLdrv+ycW2pm44A059wU4PfAC2b2W7xBw9e7QN5jHiIWLFhAZmYmF154IQAffPAB999/P2FhYURERPDPf/6TwYMHs2DBAkaPHs2ePXsIDw/n3nvv5Yorrghy9SKNhHOwbbkXYpZ9ANuXe+3tB8HQv0CPSyBJYwRFalNAx9T455z5qFLbA+WWlwF1Pw1ugKcVd87hnCMsQE9PXbBgAWlpaWWhZsiQIQwbNgwzY9GiRYwcOZIff/yRuLg4XnvtNbp06UJmZiYDBgxg6NChJCUlBaQukUbPOa8XpjTI7FwFGHQ8BS74mxdkmqQEu0qRkBXsgcJ170imFa+B9evXM3ToUAYNGsTcuXMZOXIk06ZNo6CggBEjRvDQQw+Rm5vLyJEjSU9Pp6SkhPvvv58rrriCTp06cd111zF16lSKiop455136N69O7m5ufz6179myZIlFBUV8eCDD3LBBRfwwAMPkJeXxzfffMPYsWMr9L7k5uaWPTuqa9euZe0pKSm0bNmS7du3K9SI1CbnvHExpUFm9zpvMrlOg+GkW6H7JZDYKthVijQKoRdqPr5n//XqqqTP8abdLq/0+SJzJ1T9nta94YJHql5XzqpVq5gwYQJ79uzh3XffZfbs2TjnGDZsGF999RXbt28nJSWFDz/8EIDs7Oyy9yYnJzNv3jyefvppHnvsMV588UXGjx/P2Wefzcsvv0xWVhYDBw7knHPOYdy4caSlpfHkk0+Wvf/9999n7NixbNu2rWz/5c2ePZvCwkI98FKkNvh8kDEXlk2GZVMgeyOERUDnM2DwndD9YohPDnaVIo1O6IWaQ6kcaA7Vfhg6duzISSedxB/+8AdmzJhB//79Adi7dy+rVq3itNNO4/e//z133303F198MaeddlrZe3/6058CMGDAACZNmgR4z3CaMmUKjz32GAD5+fls3Lixys8eMWIEI0aM4KuvvuL+++/n008/LVu3efNmrrnmGiZMmBCwS2IiIc/ng02zvN6Y5VNgTwaERcJxZ8OZ90C3CyCuebCrFGnUQi/UHKpH5R8neJecKmvaHm44sIfjcMTHew9Wc84xduxYbrnllgO2mTdvHh999BH33XcfQ4YM4YEHvCFG0dHRAISHh1NcXFy2n/fee49u3So+QXzWrFnV1nD66aezdu1aduzYQXJyMnv27OGiiy5i/PjxnHTSSUd1fCKNjq8ENnznDzJTYe8WCI+G48+BIX+CrkMhVpdzReqLxvfP9iEPeNOIl1fL04oPHTqUl19+mb179wKQkZHBtm3byMzMJC4ujquvvpoxY8Ywb968Q+7n3//+N6U3hM2f781nkZiYSE7O/ue9rF69umybefPmUVBQQIsWLSgsLGTEiBFce+21XH755bV2fCIhYdFE7x85DyZ53xdN9NpLimDN5zD1DnisK0y4GOb/F9oPhMtegrvWwJVvQN8rFGhE6pnQ66k5lDqYVvy8885j+fLlnHzyyQAkJCTw3//+l9WrVzNmzBjCwsKIjIzkmWeeOeh+7r//fu6880769OmDz+ejc+fOTJs2jbPOOotHHnmEfv36MXbsWNavX89rr71GZGQksbGxvP3225gZEydO5KuvvmLnzp28+uqrALz66qv069ev1o5VpEGq6oaBD26Dua/CtmWQtxsi472emJ6XQpdzISo+qCWLyKFZQ5sWJjU11aWlpVVoW758OT169AhSRQ2fzp80OtVdhsag98+8IHP8kAN7dUWk1pnZXOdcam3sq/H11IhI45a1sZpA43fZC3VXi4jUKoUaEQl9JcWwagbMfQVWzax+u6bt6q4mEal1IRNqnHNlk85JzTW0y48ihyU7A+a95n3lZEJCazj9DxDXAj57aP+YGqj1GwZEpO6FRKiJiYlh586dtGjRQsHmMDjn2LlzJzExMcEuRaT2+Epg9aeQ9gqsmu7N+Hvc2XDh36Dr+RAe6W0X1yKgNwyISN0LiVDTrl070tPT2b5dD/g+XDExMbRrpy53CQF7Nnu3Xs+b4I2ZiW8Jp94JA66DZp0O3L7PSIUYkRATEqEmMjKSzp07B7sMEalrPh+s/dzrlVnxMbgS71EF5/0Zul0EEVHBrlBE6lBIhBoRaWT2boP5//Ge15a1wbuUdPJtMOB6aKHnm4k0Vgo1ItIw+Hyw/iuvV+bHaeArhk6neWNhelwCEdHBrlBEgkyhRkTqt9wdsOB1b7bfXWshthkMutXrlUnuEuzqRKQeUagRkfrHOVj/jTevzPKpUFIIHU6BM8dCj2EQqTv2RORACjUiUn/s2wUL3/QuMe1cBTFNIfVGGHADtOwe7OpEpJ5TqBGR4HIONv7g9cosnQwlBdBuIAx/BnoOh6i4YFcoIg2EQo2IBEfeblj4thdmtv8I0U3gxGsh9QZo1SvY1YlIA6RQIyJ1xzlIn+NdXlo6CYrzoe0AGPYknPBTiIoPdoUi0oAp1IhI4OVnw6KJ3h1MW5dAVAL0vdLrlWnTN9jViUiIUKgRkcBwDjLneb0yS96Don1egLn4n9D7cohODHaFIhJiFGpEpHYV5MDid7wws2URRMZ5IWbADdD2xGBXJyIhTKFGRGrH5oVekFn8DhTuhVa94aK/Q++RENMk2NWJSCOgUCMiNbdoInw2DrLToWk7OONuwHlhJnMeRMR6A34H3ADtUsEs2BWLSCOiUCMiNbNoIkz9DRTlea+zN8GU273lY3rABX+DPldAbFLwahSRRk2hRkQOLW83fDJ2f6ApL74l/Op79cqISNAp1IjIgfZuhw3fwobvvK+tSwBX9ba52xVoRKReUKgREcjO8AeYb7zvO1Z67ZFx0H4gnPVHmP28F2Aqa9qubmsVEamGQo1IY+Mc7F63vxdm/TeQtcFbF90EOpwE/X4OHU/15pWJiPLWNetUcUwNQGQsDHmgzg9BRKQqCjUioc45r+dl/Tf7g0xOprcutjl0PAUG3QqdToVWJ0BYeNX76TPS+17+7qchD+xvFxEJMoUakVDjK4GtS8tdTvoe9u3w1iW09kJMp1O9npjkbhAWVvN99xmpECMi9ZZCjUhDV1LkTXxXOrB34/fes5YAkjpAl3O9ANPxFGh+rAb1ikjIUqgRaWiK8r2J7tZ/6wWZTbOhKNdb16IL9By+P8QktQ9urSIidUihRqS+K8z1gsuG77wQk54GJQXeupa9oN9V3uWkDqdAYqvg1ioiEkQKNSL1TX42bJy1//bqzPngKwYL8+5GGvhLrxemw8kQ1zzY1YqI1BsKNSKBVvl5SZXvGMrdCRvL3V69dQk4H4RFek+1PuU33uWk9gP1YEiRBmLy/Awenb6CzKw8UpJiGTO0G8P7tw12WYetoR1HQEONmZ0PPAGEAy865x6ptP4fwFn+l3FAS+ecHhwjoaPK5yX9GjbN8m613vAtbP/RWxcRA+1+Aqff5V1OapsKUXHBq11Ejsjk+RmMnbSYvKISADKy8hg7aTFAvQ4ElTXE4zDnqpn6/Gh3bBYOrATOBdKBOcCVzrll1Wz/a6C/c+7Gg+03NTXVpaWl1Xa5IoHxjxO8IFOVqARvoruOp3g9MSn9ISK6busTkaNWWOwja18hu/YVsju3iNten8uufUUHbJcQHcHVJ3XE5xw+n6PEOZwDn3OU+Bw+Bz6f89b720vXHbCdK7edr4rtDrHOOfz7qrhc/rOz9xVV+XCUtkmxfHvP2bV2/sxsrnMutTb2FciemoHAaufcWgAzewu4FKgy1ABXAn8KYD0idS87vZoVBndvgHBdARapL5xz5BaWsDu3kKx9RezeV+h95Raye18RWfu87/vbvbbcwpIa7X9vQTEvf7uOMINwM8LMMIPwMG85LMwIM7xlM8LCqtmu3Doz7z3hYd5yeJgRWX67cusqbOd/HRZ24Hal6yZ8v6HK48jMquLBtvVEIH+jtgXK/xM1HRhU1YZm1hHoDHxezfqbgZsBOnToULtVigTC3u3w5f9R7UMgm7ZToJFa0dDGPBxMbR5Lic+xJ6+IXfsKvTCS64WRrH1Vt5V+LyzxVbvPJjERNIuPIikuimMSounaMpGkuCiaxUWSFB9Fc//ynW8vYFtOwQHvr+0ejkD7dPk2MqoIMClJsUGopmbqy2/VUcC7zrkq465z7nngefAuP9VlYSKHpSgfZj0DXz/u3Yp97Fmw8Qco1vOSpPZVNebh7vcWsXNvAef1al2uB8D713/Zv/rL9QCUriv917oFaXLGg43fuKB36/09J7n7e0qy9hWV9aJUaNtXSHZeEdWNrogIs7Iw0iwuio4t4ujXPolm8fvbkuIi/a+9tqaxkUSE12z27T9e2KPCsQDERoYzZmi3oztJdWzM0G4N7jgCOabmZOBB59xQ/+uxAM65/6ti2/nAbc657w61X42pkXrJOVg6CWY+CNkboev5cO6f4Ziuh777SeQInfrI51X+S/poWHWXRipcqth/uaLK7coCU+XLGhUvrYT5L5eYGbPW7qSg+MBeEqPa/k7A+yPbrFwASfKHkqrbokiKjyQxOiLg4S1UetDq4jhqc0xNIENNBN5A4SFABt5A4aucc0srbdcd+ATo7GpQjEKN1Dub5sD0P0L6bO+BkOc9DMeddej3iRylzvd8WO0f/L9d3sc/SLTcoFKfo8R5Y0e8waOUtVccfOqtO2C7KganVhiM6t+Hq8HAV5+/jtLlBZuyqj3OP5zX1d+zEkWz+HIBJS6SmMhqHsAqDUaDGCjsnCs2s9uB6Xi3dL/snFtqZuOANOfcFP+mo4C3ahJoROqV3Rvgs4dgyXuQ0AqG/Rv6/bz6p1yL1KKMrDzCw4xi34G/OtsmxTIytWE9IqO6Xqe2SbHcfnaXIFQkDVFAx9Q45z4CPqrU9kCl1w8GsgaRWpe/B755HL5/2pvl9/S74NQ7IDoh2JVJI7Fyaw7XvjSb8DAICwujsNxlm/o+5qE6DXH8htQ/9WWgsEj9V1IM81+Dz8fDvh3QZxQMud8bJyNSR+Zu2MWNr6YRFRHG5NsGs2JLTkiM3SitORSORYInYGNqAkVjaiQoVn0KM+71Zv/tcAoMHe89wkCkDn22fCu3vTGPNk1jee3GgbRvrhmnpeFrEGNqRELC1mUw4z5Y8xk06wwj/wM9LvFuERGpQ++kbeKeSYvp2aYJr9zwE5ITNPu0SGUKNSJV2bsNvvgLzJsA0Ylw3njv6dh6jIHUMeccz321lkc+/pHBxyfz7DUDSIjWr26RqugnQ6S8onz44Wlv8rziPBh4M5xxN8Q1D3Zl0gj5fI6/fLScF79ZxyV9U/j7z/oSFVGzCeBEGiOFGhHwJs9b8h58+pA3eV63C+HccZCsW0klOAqLfdz17kImL8jk+lM68cDFPQkL02VPkYNRqBHZNNs/ed4caN0bhk+FzqcHuyppxHILihn9+jy+WrmdMUO78aszjwva4wtEGhKFGmm8dq+HTx+Epe9DQmu49GnoO0qT50lQ7cot5IZX57A4PYtHftqbUQP1EF+RmlKokcYnPxu+/jv88AxYOJxxD5zya02eJ0GXvnsf1748m4zdeTx79QDO69U62CWJNCgKNdJ4lBTDvFe9u5r27YS+V3mT5zVJCXZlIqzYksN1L88mt7CY//xiEAM7a3C6yOFSqJHQ5xysmunNN7NjBXQcDEMfhpT+wa5MBIA563fxi1fnEBMZzsRbTqZHmybBLkmkQVKokdC2dSlMvxfWfgHNj4VRb3h3NmnQpdQTny7zZglumxTLBM0SLHJUFGokNOVshS/Gw/z/QHQTOP8RSP0FREQFuzKRMhPTNjF20mJ6pTThlet/QgvNEixyVBRqJLQU5cH3T8E3/4DifBh0K5w+RpPnSb3inOOZ/63hb5+s4LQuyTx79QDiNUuwyFHTT5GEBp/PP3neg7AnHbpfDOc8BMnHB7sykQp8PsfDHy7n5W/XMaxvCo9plmCRWlOjUGNmk4CXgI+dc77AliRymDb+4E2elzEXWveBEc9C59OCXZXIAQqLfYx5dyEfaJZgkYCoaU/N08ANwL/M7B3gFefcisCVJVIDu9Z5PTPLJkNiGxj+DPQZBWH6V6/UP+VnCb7r/G6MPkOzBIvUthqFGufcp8CnZtYUuNK/vAl4Afivc64ogDWKVJSXBV8/BrOeg7AIOPOPcMrtEBUf7MpEqrRzbwE3vjqHxRnZ/PWy3lzxE80SLBIINR5TY2YtgKuBa4D5wOvAYOA64MxAFCdSQUkRzH3Vmzwvbzf0+zmcfR80aRPsykSqlb57H9e+NJuMrDyeuyaVc3u2CnZJIiGrpmNq3ge6Af8BLnHObfavetvM0gJVnDRiiybCZ+MgOx2atoOew2HVdNixEjqdBkPHQ5u+wa5S5KBWbMnh2pdnkVdYolmCRepATXtq/uWc+6KqFc651FqsR8QLNFN/4xp11pIAACAASURBVN2eDZC9Cb7/N8S3hFFvQrcLNHme1HulswTHRoUz8daT6d5aswSLBFpNR1T2NLOk0hdm1szMfhWgmqSx+2zc/kBTXkQUdNdswFL/zVy2latfnEVyYjTvjT5FgUakjtQ01PzSOZdV+sI5txv4ZWBKkkYvO72a9oy6rUPkCEycs4lb/pNG99aJvHvrKbRrpsceiNSVml5+Cjczc845ADMLBzTfvNS+vCwIj4KSggPXNW1X9/WI1JBzjqe/XMOj0zVLsEiw1PQn7hO8QcHP+V/f4m8TqT27N8DrP/PucgqPgpLC/esiY2HIA8GrTeQgfD7Hnz9cxivfrufSfik8erlmCRYJhpqGmrvxgsxo/+uZwIsBqUgap/S58OYVXpC5bgrkbK5499OQB6DPyGBXKXKAwmIff3hnIVMWZnLjqZ2576IemiVYJEhqOvmeD3jG/yVSu5ZPhfd+CYmt4KqP4JiuXrtCjNRzuQXF3PrfuXy9agd3n9+dW884VrMEiwRRTeep6QL8H9ATiCltd84dG6C6pDFwDr5/EmbcD+1S4cq3ID452FWJ1EjpLMFLMvfwt8v7MDK1fbBLEmn0anr56RXgT8A/gLPwngOlC8Zy5EqK4eO7IO0lb2K9Ec9642ZEGoBNu/Zx3cv+WYKvHsA5miVYpF6oaTCJdc59BphzboNz7kHgosCVJSGtIAfeHOUFmlPvhMtfUaCRBuPHLXu47Jnv2LG3gNdvGqRAI1KP1LSnpsDMwoBVZnY7kAEkBK4sCVnZGfDGFbBtGVzyBAy4PtgVidTY7HW7+MWEOcRFhfPOrafQrXVisEsSkXJqGmruAOKA3wB/xrsEdV2gipIQtXmhF2gK9sLP34HjhwS7IpEam7F0C79+cz5tm8Xy2o0DNameSD10yFDjn2jvCufcH4C9eONpRA7Pyunwzg0Q2wx+MR1a9Qp2RSI19tbsjfzx/cX0bpfEK9f/hObxmntUpD46ZKhxzpWY2eC6KEZC1OwXvEHBrXvDlW9DkzbBrkikRsrPEnxG12N45uoTiYvSLMEi9VVNfzrnm9kU4B0gt7TROTcpIFVJaPCVeLdr//AUdL0ALn8JouKDXZVIjfh8jnHTlvHqd+sZ3i+FR3/Wl8hw3fQpUp/VNNTEADuBs8u1OUChRqpWmAuTboYfp8GgW2HoXyAsPNhVidRIYbGP37+zkKkLM/nF4M7ce6FmCRZpCGo6o7DG0UjN5Wz1HnmweSGc/1c46dZgVyRSY3sLihntnyX4ngu6c8vpmiVYpKGo6YzCr+D1zFTgnLux1iuShm3bcu+hlPt2wqg3oNsFwa5IpMZ27i3ghlfnsDRzD49e3oefaZZgkQalppefppVbjgFGAJm1X440aGu+gInXehPp3fARpPQPdkUiNbZp1z6ufXk2m7PzeP6aAQzpoUn1RBqaml5+eq/8azN7E/jmUO8zs/OBJ4Bw4EXn3CNVbDMSeBCvJ2ihc+6qmtQk9cy8/8C0OyG5G1z1NiTpX7jScCzfvIfrXp5NQbGP128axICOzYNdkogcgSO9N7EL0PJgG/jnt3kKOBdIB+aY2RTn3LJy23QBxgKnOud2m9lB9yn1kM8Hn/8ZvnkcjjsbfjYBYpoEuyqRQ5o8P4NHp68gMysPgMSYCN4dfQpdW2mWYJGGqqZjanKoOKZmC3D3Id42EFjtnFvr38dbwKXAsnLb/BJ4yjm3G8A5t62GdUt9UJQPk0fD0klw4nVw0d8hPDLYVYkc0uT5GYydtJi8opKytoJiH8sy9yjUiDRgNb38dCQ/5W2BTeVepwODKm3TFcDMvsW7RPWgc+6Tyjsys5uBmwE6dOhwBKVIrcvdCW9dCZtmwTkPwal3gO4QkQbiLx8trxBowAs1j05fwfD+bYNUlYgcrRrNJGVmI8ysabnXSWY2vBY+PwLvUtaZwJXAC2aWVHkj59zzzrlU51zqMcccUwsfK0dlx2p4cQhkLoCfvQqD71SgkQZh7obd/OLVOWzLKahyfemlKBFpmGo6puZPzrn3S18457LM7E/A5IO8JwMoP1q0nb+tvHRglnOuCFhnZivxQs6cGtYldW3Dd/DWVWDhcP00aD8w2BWJHJRzjm9X7+SpL1bz/dqdNIuLJDEmgpz84gO2TUmKDUKFIlJbajrnd1XbHSoQzQG6mFlnM4sCRgFTKm0zGa+XBjNLxrsctbaGNUldWzQRXrsU4pLhpk8VaKRe8/kcM5ZuYfjT33H1S7NYu2Mv913Ug2/uPps/X3oCsZEVZ7iOjQxnzNBuQapWRGpDTXtq0szscby7mQBuA+Ye7A3OuWIzux2Yjjde5mXn3FIzGwekOeem+NedZ2bLgBJgjHNu55EciASQc/DVo/DFeOg4GK74D8Tpllepn4pLfExbtJmnv1zNyq176dA8jr+M6M1lA9oSHeEFmdJxM6V3P6UkxTJmaDeNpxFp4My5AyYKPnAjs3jgfuAcvLugZgLjnXO5B31jAKSmprq0tLS6/tjGq7gQpt4BC9+APqNg2L8hIirYVYkcoKC4hPfmZvDs/9awcdc+urZK4Lazjuei3m2I0IMoReotM5vrnEutjX3V9O6nXOCe2vhAaUDydsPb18D6r+HMsXDG3RoQLPVObkExb87eyAtfr2XrngL6tmvKfRcN4JwerfQQSpFGpqbz1MwEfuacy/K/bga85ZwbGsjiJIh2r/ee4bRrHYx4HvpeEeyKRCrI3lfEhO/X88q369i9r4iTj23B4yP7ccpxLfQASpFGqqZjapJLAw2AZv8NcZvmwJujwFcM134AnU4NdkUiZbbnFPDiN2v57/cbyC0s4ZweLRl95vEM6Ngs2KWJSJDVNNT4zKyDc24jgJl1ooqndksIWDoZ3r8FElvDz9+F5C7BrkgEgPTd+3j+q7W8PWcTRSU+LuqTwq/OPI4ebfRYDhHx1DTU3At8Y2b/Aww4Df8MvxIinIPv/gUzH4D2g2DUGxCfHOyqRFi9bS/P/m8Nk+dnYAaXndiOW844js7J8cEuTUTqmZoOFP7EzFLxgsx8vPllNPVmqCgpho/+AHNfgV4jYPizEBkT7KqkkVuSkc3TX67m4yVbiI4I45qTO/LL047VBHkiUq2aDhS+CbgDb1bgBcBJwPfA2YErTepE/h5453pY8xkM/h2cfT+E6fZXCZ4563fx1Ber+XLFdhKjI7jtzOO54dROtEiIDnZpIlLP1fTy0x3AT4AfnHNnmVl34C+BK0vqRHY6vD4Stv8Il/wLBlwX7IpC0uT5GZrk7RCcc3y1agdPfb6a2et30SI+ijFDu3HNyR1pEqMnv4tIzdQ01OQ75/LNDDOLds79aGaaT7why1wAb1wBRfvg6nfhOHW6BcLk+RmMnbS47InQGVl5jJ20GEDBBu9RBtOXbuGpL1ezJGMPbZrG8KdLejLqJx2IjQo/9A5ERMqpaahJ9z89ezIw08x2AxsCV5YE1IpP4N0bvUcdXDMdWvUMdkUh69HpK8oCTam8ohIenb6iUYeaohIfUxZk8vSXq1mzPZdOLeL462W9GdG/HVERuvwpIkempgOFR/gXHzSzL4CmwCcBq0oCZ9Zz8Mk90LoPXPW2d+u21DrnHEsz95CRVfV4+oysPG5/Yx7dWiXSpVUiXVsl0LFFPOEhPgNuflEJ76Rt4tn/rSUjK4/urRP595X9ubB3m5A/dhEJvJr21JRxzv0vEIVIgPlKYPq9MOsZ6HYRXPYCROmW2Nq2Zvtepi7MZMrCTNZur/7RaNERYSzYlMW0RZvL2qIiwjjumAS6tkqga6tE/1cC7ZvFNfjp/vcWFPP6Dxt44et17NhbQP8OSYy7tBdnd2+p2X9FpNYcdqiRBqgwF967CVZ8BCf9Cs57GMI0XqG2ZGTlMc0fZJZm7sEMBnVuzk2Dj8XnHOM/XF7hElRsZDj/99PeDO/fltyCYlZv28vKrTms2raXFVtymLNuFx8syCzbPiYyjONbJtC1pder0611Al1aJtI2Kbbeh53duYW88t16Jny3nuy8Ik7rksyvzuzPScc2V5gRkVqnUBPqcrZ4A4K3LIILHoVBmjOxNuzYW8BHizczZUEmaRt2A9C3fRL3XdSDi/uk0Lrp/nl+EqIjqr37KT46gr7tk+jbPqnC/nPyi1i1bS+rtuawcqsXer5ds4NJ8zPKtomLCqdLy4Syy1ddWiXSrVUibZrGBD0wbN2Tz4tfr+X1WRvZV1jCeT1b8auzjqdfpeMUEalN5lzDetpBamqqS0tLq92dLpoIn43zbnFu2g6GPAB9RtbuZ9SV8seS0ApKCqG4AC5/GbqdH+zqGrTsvCKmL93C1IWZfLdmJyU+R9dWCQzrm8IlfVPo2CLwl/Oy9xWxatv+oLNqWw4rtuxlx96Csm0SoyM4vlVpz87+S1mtmkQHPOxs2rWPZ/+3hnfS0ilxjmF9Uxh95nF0bZUY0M8VkYbLzOY651JrZV+NPtQsmghTfwNF5QZ0RsZ687Y0tGBT1bEAnHUfnDEmODU1cHmFJXy6fCtTF2by5YrtFJb46NA8jkv6tmFY37Z0a10//ljvzi1k5dYcVpb17njBZ1duYdk2TWIi6FpuYLK3nMAxCUcfdlZtzeHpL9cwZWEm4WZcntqOW08/jg4t4o720EQkxCnU1Gao+ccJkL3pwPaIGDj2LG/siRlYGFi49z3M/738V4W20u3Cqmgr3c6q2Z//86rc3yHeO/lXsG/HgcfStD38dkntnbMQV1js4+tV25myMJOZy7ayr7CElonRXNwnhWH9UujbrmnQL+/U1I69BV6PTmnPzta9rNyWQ9a+orJtmsVFVgw6Lb3lqmbwrTyR4BU/acfSzD1MX7qV2Mhwfj6oA788/VhaNdFjNkSkZmoz1GhMTXZ61e3F+bAn3XvQo/N5dw85H7jS7z7w+Sq2lW3jqmjzbxcM1R2jlCnxOWat3cmUhZl8vGQL2XlFJMVFcmm/tlzStw2DOrdokLccJydEk5wQzSnH7X84qXOO7TkFFS5hrdy6lw8WZJKTX1y2XYv4qLI7sLq0SmTrnnxe+Hot+UU+wBsg/fjMVcREGL8Z0oUbTulEs/ioOj9GEZFSCjVN21XdU9O0Pdz6Te1/nnMHBp1DBqJy66oMVyXeujdHQe62qo9RDuCcY/6mLKYsyOTDxZvZnlNAfFQ45/VqzSV92zD4+GNCciI4M6NlkxhaNolhcJeKYWfLnnxWbq14CevduenkFlYfyJvFR/O7c7vWRekiIgelUDPkgarH1Ax5IDCfZwbhATrtQ8fX7bE0QM45ftySw5SFmUxdmEn67jyiIsI4q9sxDOvblrO7t2y00/ObGW2axtKmaSxndD2mrN05R2Z2Pqc+8nmV79uSnV9XJYqIHJRCTelg4FC4+ymUjqWWrd+RWxZkVm3bS3iYcerxydx5TlfO69VKD008CDOjbVIsbZNiq5whOSUpNghViYgcSAOFJWRtzs7jw0WbmbIwk0Xp2QAM7NScS/qlcOEJrascCCvVq/xwTqg4kaCIyJHQQGGRauzKLfQmxVuYyZz1u3AOerdtyh8v7M7FfVLUq3AUSoNLdRMJiogEm0KNNHg5+UXMWLqVKQsz+Wb1Dkp8juOOiefOIV25pG8bjj0mIdglhozh/dsqxIhIvaVQIw1SflEJn/+4jSkLMvl8xTYKi320TYrll6cdy7C+KfRok9hg5pIREZHaoVAj9VLlSd7GDO3GRX3a8M2qHUxdmMmMZVvZW1BMckI0Vw3swCV9UzixQ5KCjIhII6aBwlLvVDUgNTzMiA439hX5aBITwQUntGFYvxQGdW5ORHjozSUjItJYaKCwhLRHp6+oEGjAm/HXhYfxwrWpnN41meiIxjmXjIiIVE+hRuqVohJflXOhgDeO5tyereq4IhERaSgUaqReKPE5pi7M5PGZK6vdRrdji4jIwWgwggSVc44ZS7dw4RNfc+fbC4iPjuDm0zoTG1nxf83YyHDGDO0WpCpFRKQhUE+NBM13q3fwt+krWLApi87J8fz7yv5c1LsNYWFGz5SmmuRNREQOi0KN1Ln5G3fz2IwVfLt6J22axvDXy3pz2YntKtzFpEneRETkcCnUSJ1ZsSWHv89YwYxlW2keH8X9F/fk54M6EBOpO5lEROToKdRIwG3Ymcs/P13F5AUZJERF8Ltzu3Lj4M4kROt/PxERqT36qyIBs3VPPv/6bBVvz9lEeJhx8+nHcuvpx9EsPirYpYmISAhSqJFatzu3kGf/t4ZXv1tPic8xamB7fn12F1o1iQl2aSIiEsIUaqTW7C0o5uVv1vHCV2vZW1jMiH5tufOcrnRoERfs0kREpBEIaKgxs/OBJ4Bw4EXn3COV1l8PPApk+JuedM69GMiapPblF5Xw3x828PSXa9iVW8h5PVvx+/O60a11YrBLExGRRiRgocbMwoGngHOBdGCOmU1xzi2rtOnbzrnbA1WHBE5xiY9356bzxGer2Jydz+Djk/nD0G70a58U7NJERKQRCmRPzUBgtXNuLYCZvQVcClQONdLA+HyOaYs384+ZK1m3I5d+7ZP4+8/6csrxycEuTUREGrFAhpq2wKZyr9OBQVVsd5mZnQ6sBH7rnNtUeQMzuxm4GaBDhw4BKFVqwjnHFyu28ej0lSzfvIdurRJ54dpUzunREjMLdnkiItLIBXug8FTgTedcgZndAkwAzq68kXPueeB5gNTUVFe3JQrAD2t38uj0FczdsJuOLeJ4YlQ/Lu6TQniYwoyIiNQPgQw1GUD7cq/bsX9AMADOuZ3lXr4I/C2A9cgRWJyezaMzVvDVyu20ahLN+BEnMDK1PZHhehaqiIjUL4EMNXOALmbWGS/MjAKuKr+BmbVxzm32vxwGLA9gPXIYVm/L4e8zVvLxki0kxUXyxwu7c+3JnfRIAxERqbcCFmqcc8VmdjswHe+W7pedc0vNbByQ5pybAvzGzIYBxcAu4PpA1SM1s2nXPp74bBWT5qUTGxnOHUO6cNNpnUmMiQx2aSIiIgdlzjWsISqpqakuLS0t2GWEnG05+Tz9xRpen7UBM+Pakzoy+szjaJEQHezSREQkhJnZXOdcam3sK9gDhSXIsvcV8dxXa3jl2/UUlvgYmdqe3ww5njZNY4NdmoiIyGFRqGmk9hUW88q363nuf2vYk1/MsL4p/PbcrnROjg92aSIiIkdEoaaRKSgu4c1ZG3nyizXs2FvAkO4t+f153eiZ0iTYpYmIiBwVhZpGorjEx/vzM/jnp6vIyMpjUOfmPHfNiQzo2DzYpYmIiNQKhZoQ55zj4yVb+PuMFazZnkufdk155LLeDD4+WbMAi4hISFGoCTGT52fw6PQVZGbl0Tw+itjIcNKz8ji+ZQLPXn0iQ3u1VpgREZGQpFATQibPz2DspMXkFZUAsDO3EAOuHNieh4f31iMNREQkpGmu+xDyyCc/lgWaUg74auUOBRoREQl56qkJAT6f4+20TWzJzq9yfWZWXh1XJCIiUvcUahq45Zv3cO/7i5m3MYuo8DAKS3wHbJOSpIn0REQk9CnUNFD7Cov556ereOmbdTSNjeTvP+tLmMEf319S4RJUbGQ4Y4Z2C2KlIiIidUOhpgGauWwrD05ZSkZWHqN+0p67z+9Os/goAMys7O6nlKRYxgztxvD+bYNcsYiISOAp1DQgGVl5PDhlKTOXbaVbq0TevfVkUjtVnDxveP+2CjEiItIoKdQ0AEUlPl79dj3/+HQlPue454Lu/GJwZyLDdfOaiIhIKYWaem7uht3c+/5iftySw5DuLXlwWC/aN48LdlkiIiL1jkJNPZW9r4hHPvmRN2dvpE3TGJ69egBDe7XSbMAiIiLVUKipZ5xzTF6QwcPTlpOVV8RNgztz57ldSYjWfyoREZGD0V/KemTN9r3c9/4Svl+7k77tk3htxAn0Smka7LJEREQaBIWaeiC/qISnv1zDs1+uIToyjIeHn8CVAzvo0QYiIiKHQaEmyL5etZ37Jy9h/c59XNovhXsv6kHLxJhglyUiItLgKNQEybacfB6etpwpCzPpnBzPf38xiMFdkoNdloiISIOlUFPHSnyON2Zt4G/TV1BQ5OPOc7pw6xnHERMZHuzSREREGjSFmjq0JCObeycvYeGmLE49vgV/vvQEjj0mIdhliYiIhASFmjqwt6CYx2es5NXv1tE8PoonRvVjWN8UzTkjIiJSixRqAsg5x/SlW3hwyjK25uRz1cAO3DW0O03jIoNdmoiISMhRqAmQTbv28acpS/n8x230aNOEp68+kRM7NAt2WSIiIiFLoaaWFZX4ePHrdTzx2UrCzLjvoh5cf0onIvTwSRERkYBSqKlFs9ft4r7Ji1m5dS9De7XiT5f0IiUpNthliYiINAoKNbVgV24hj3y8nIlp6bRNiuXFa1M5p2erYJclIiLSqCjUHAXnHO/OTecvHy0nJ7+YW844ljuGdCEuSqdVRESkrumv7xFatTWHeycvYfa6XQzo2IzxI06ge+smwS5LRESk0VKoOUx5hSX8+/NVPP/VWuKjI3jkp70ZmdqeMD18UkREJKgUag7DFyu28cAHS9i0K4/LTmzHHy/sTouE6GCXJSIiIijU1MiW7HzGTVvKR4u3cOwx8bz5y5M4+bgWwS5LREREylGoOYgSn+O179fz9xkrKSrx8YfzuvLL048lOkIPnxQREalvFGqqsSg9iz++v5glGXs4rUsyDw8/gY4t4oNdloiIiFRDoQaYPD+DR6evIDMrj9ZNYzj+mHi+WbOT5IRonryqPxf1bqOHT4qIiNRzjT7UTJ6fwdhJi8krKgFgc3Y+m7PzOe34Fjx19QCaxOjhkyIiIg1Bo38g0aPTV5QFmvLW7tinQCMiItKABDTUmNn5ZrbCzFab2T0H2e4yM3NmlhrIeqqSmZV3WO0iIiJSPwUs1JhZOPAUcAHQE7jSzHpWsV0icAcwK1C1HEx1D5zUgyhFREQalkD21AwEVjvn1jrnCoG3gEur2O7PwF+B/ADWUq0xQ7sRG1nxFu3YyHDGDO0WjHJERETkCAUy1LQFNpV7ne5vK2NmJwLtnXMfHmxHZnazmaWZWdr27dtrtcjh/dvyfz/tTdukWAxomxTL//20N8P7tz3ke0VERKT+CNrdT2YWBjwOXH+obZ1zzwPPA6SmprrarmV4/7YKMSIiIg1cIHtqMoD25V6387eVSgROAL40s/XAScCUYAwWFhERkYYvkKFmDtDFzDqbWRQwCphSutI5l+2cS3bOdXLOdQJ+AIY559ICWJOIiIiEqICFGudcMXA7MB1YDkx0zi01s3FmNixQnysiIiKNU0DH1DjnPgI+qtT2QDXbnhnIWkRERCS0NfoZhUVERCQ0mHO1fjNRQJnZdmBDsOuoQjKwI9hF1BM6Fx6dB4/Og0fnYT+dC4/Og6ebcy6xNnbU4B5o6Zw7Jtg1VMXM0pxzunMLnYtSOg8enQePzsN+OhcenQePmdXaDUK6/CQiIiIhQaFGREREQoJCTe15PtgF1CM6Fx6dB4/Og0fnYT+dC4/Og6fWzkODGygsIiIiUhX11IiIiEhIUKgRERGRkKBQcxBm9rKZbTOzJeXampvZTDNb5f/ezN9uZvYvM1ttZovM7MRy77nOv/0qM7suGMdyNMysvZl9YWbLzGypmd3hb29U58LMYsxstpkt9J+Hh/ztnc1slv943/Y/6wwzi/a/Xu1f36ncvsb621eY2dDgHNHRMbNwM5tvZtP8rxvreVhvZovNbEHpramN7WcDwMySzOxdM/vRzJab2cmN7TyYWTf//welX3vM7M7Gdh4AzOy3/t+TS8zsTf/vz8D/jnDO6auaL+B04ERgSbm2vwH3+JfvAf7qX74Q+BgwvCeOz/K3NwfW+r838y83C/axHeZ5aAOc6F9OBFYCPRvbufAfT4J/ORKY5T++icAof/uzwGj/8q+AZ/3Lo4C3/cs9gYVANNAZWAOEB/v4juB8/A54A5jmf91Yz8N6ILlSW6P62fAfwwTgJv9yFJDUGM9DufMRDmwBOja28wC0BdYBsf7XE4Hr6+J3RNAPvr5/AZ2oGGpWAG38y22AFf7l54ArK28HXAk8V669wnYN8Qv4ADi3MZ8LIA6YBwzCmxE0wt9+MjDdvzwdONm/HOHfzoCxwNhy+yrbrqF8Ae2Az4CzgWn+42p058Ff93oODDWN6mcDaIr3R8wa83modOznAd82xvOAF2o24YWyCP/viKF18TtCl58OXyvn3Gb/8haglX+59D9iqXR/W3XtDZK/W7A/Xi9FozsX/ksuC4BtwEy8fzlkOe+p9FDxmMqO178+G2hBCJwH4J/AXYDP/7oFjfM8ADhghpnNNbOb/W2N7WejM7AdeMV/SfJFM4un8Z2H8kYBb/qXG9V5cM5lAI8BG4HNeD/zc6mD3xEKNUfBedGx0dwTb2YJwHvAnc65PeXXNZZz4Zwrcc71w+upGAh0D3JJdc7MLga2OefmBruWemKwc+5E4ALgNjM7vfzKRvKzEYF3qf4Z51x/IBfvMkuZRnIeAPCPFRkGvFN5XWM4D/4xQ5fihd0UIB44vy4+W6Hm8G01szYA/u/b/O0ZQPty27Xzt1XX3qCYWSReoHndOTfJ39wozwWAcy4L+AKvCzXJzEqfo1b+mMqO17++KbCThn8eTgWGmdl64C28S1BP0PjOA1D2r1Kcc9uA9/HCbmP72UgH0p1zs/yv38ULOY3tPJS6AJjnnNvqf93YzsM5wDrn3HbnXBEwCe/3RsB/RyjUHL4pQOlI9OvwxpeUtl/rH81+EpDt726cDpxnZs386fU8f1uDYWYGvAQsd849Xm5VozoXZnaMmSX5l2PxxhUt5//bu58Qq8owjuPfHyhO6SBE7VqMwqSQpEWGksmAESguSgQrV9miggqMEEsI3A3YpkWbIBBE2hSai7BQKcUoDRvH0ZQGgooyAkNSdbD87AAAA4tJREFUMcSeFu9zmsMwKc2df5zz+8Bh7j3/7j0P99x57vu+5zwludmYq42OQxWfjcCR/JV2AHg6R/wvAHqBE1NzFJ2LiDci4t6I6KE0sR+JiM20LA4AkuZK6q4eUz7TQ7Ts3IiIi8BPkhblrDXAOVoWh5pnGOl6gvbF4UdghaQ78/9H9XmY/O+I6R5QNJMnyofyV+AG5ZfI85R+vsPA98Ah4K5cV8C7lDEWZ4CHa/vZAgzn9Nx0H9c44rCK0lw6CAzktK5tsQAeAL7NOAwBb+X8hXmiDVOam+fk/K58PpzLF9b2tSPjcwFYO93H1kFM+hi5+ql1cchjPp3TWWBHzm/VuZHvfxnwTZ4f+ylX7bQxDnMprQzza/PaGIedwPn8rtxDuYJp0r8jXCbBzMzMGsHdT2ZmZtYITmrMzMysEZzUmJmZWSM4qTEzM7NGcFJjZmZmjeCkxszGTdKX+bdH0rMTvO83x3otM7P/4ku6zaxjkvqA1yNi/f/YZlaM1IEZa/mViJg3Ee/PzNrBLTVmNm6SruTDfuAxSQOStmbhz12STkoalPRCrt8n6ZikA5Q7jCJpfxaDPFsVhJTUD9yR+9tbf628++ouSUOSzkjaVNv355I+lHRe0t68mymS+iWdy/fy9lTGyMymzqzbr2JmdlvbqbXUZHJyOSKWS5oDHJf0Wa77ELAkIn7I51si4lKWnjgp6aOI2C7p5SjFQ0fbQLl77VLg7tzmaC57ELgf+AU4Djwq6TvgKWBxRERV6sLMmsctNWY2GZ6g1LQZAL6m3Ca+N5edqCU0AK9KOg18RSle18utrQI+iFIx/TfgC2B5bd8/R8TflHIePcBl4DrwvqQNwLWOj87MZiQnNWY2GQS8EhHLcloQEVVLzdV/VypjcR4HVkbEUkptra4OXvev2uObQDVu5xFK5ej1wMEO9m9mM5iTGjObCH8C3bXnnwIvSZoNIOm+rGI92nzgj4i4JmkxsKK27Ea1/SjHgE05buceYDW3qNwraR6luOAnwFZKt5WZNZDH1JjZRBgEbmY30m7gHUrXz6kcrPs78OQY2x0EXsxxLxcoXVCV94BBSaciYnNt/j5gJaUydgDbIuJiJkVj6QY+ltRFaUF6bXyHaGYznS/pNjMzs0Zw95OZmZk1gpMaMzMzawQnNWZmZtYITmrMzMysEZzUmJmZWSM4qTEzM7NGcFJjZmZmjfAPw90pCRM8gIAAAAAASUVORK5CYII=\n","text/plain":["<Figure size 648x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"7Z31mvHGQT3y"},"source":["## Residual bottleneck block\n","\n","A bottleneck block is often useful for better efficiency, especially when importing a model to mobile devices.\n","The residual bottleneck block is similar to the standard residual block, but the plain block part has a different architecture:\n","it consists of 3 convolutional layers, and the first two convolutional layers have a smaller number of channels.\n","\n","Here is the specification of the bottleneck block:\n","\n","1. Spatial Batch normalization\n","2. ReLU\n","3. Convolutional layer with `Cout // 4` 1x1 filters, stride 2 if downsampling; otherwise stride 1\n","4. Spatial Batch normalization\n","5. ReLU\n","6. Convolutional layer with `Cout // 4` 3x3 filters, with zero-padding of 1\n","7. Spatial Batch normalization\n","8. ReLU\n","9. Convolutional layer with `Cout` 1x1 filters\n","\n","Implement `ResidualBottleneckBlock.__init__` and don't forget to add the residual connection!"]},{"cell_type":"code","metadata":{"id":"vqETnXH5QT37","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456714360,"user_tz":-60,"elapsed":461389,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"909ef3f1-aeec-48ab-8f3e-0833c2402b25"},"source":["data = torch.zeros(2, 3, 5, 6)\n","model = ResidualBottleneckBlock(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of ResidualBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))\n","\n","data = torch.zeros(2, 3, 5, 6)\n","model = ResidualBottleneckBlock(3, 10, downsample=True)\n","if list(model(data).shape) == [2, 10, 3, 3]:\n","  print('The output of ResidualBlock with downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock with downsampling has an *incorrect* dimension! expected:', [2, 10, 3, 3], 'got:', list(model(data).shape))"],"execution_count":30,"outputs":[{"output_type":"stream","text":["The output of ResidualBlock without downsampling has a *correct* dimension!\n","The output of ResidualBlock with downsampling has a *correct* dimension!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"QnRc5hiHudMP"},"source":["By running the following script, you can check the architecture of ResNet-47 with bottlenecks.\n","\n","Caution: it is long!"]},{"cell_type":"code","metadata":{"id":"md6xmG-Aucrx","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1617456714365,"user_tz":-60,"elapsed":461384,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}},"outputId":"9b6acf46-7b91-416d-b556-f4c35395d76e"},"source":["# example of specification\n","networks.update({\n","  'resnet47': {\n","    'block': ResidualBottleneckBlock,\n","    'stage_args': [\n","      (32, 32, 5, False),\n","      (32, 64, 5, True),\n","      (64, 128, 5, True),\n","    ],\n","  },\n","})\n","\n","print(get_resnet('resnet47'))"],"execution_count":31,"outputs":[{"output_type":"stream","text":["ResNet(\n","  (cnn): Sequential(\n","    (0): ResNetStem(\n","      (net): Sequential(\n","        (0): Conv2d(3, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        (1): ReLU()\n","      )\n","    )\n","    (1): ResNetStage(\n","      (net): Sequential(\n","        (0): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (1): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (2): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (3): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (4): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","      )\n","    )\n","    (2): ResNetStage(\n","      (net): Sequential(\n","        (0): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 16, kernel_size=(1, 1), stride=(2, 2))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2))\n","        )\n","        (1): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (2): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (3): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (4): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","      )\n","    )\n","    (3): ResNetStage(\n","      (net): Sequential(\n","        (0): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 32, kernel_size=(1, 1), stride=(2, 2))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2))\n","        )\n","        (1): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (2): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (3): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","        (4): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Identity()\n","        )\n","      )\n","    )\n","  )\n","  (fc): Linear(in_features=128, out_features=10, bias=True)\n",")\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"f8jUGXtx8DI_"},"source":["# Final checks\n","Make sure you run \"Runtime -> Restart and run all...\" to double check PyTorch Autograd and NN before submitting."]},{"cell_type":"markdown","metadata":{"id":"OaEWseW9Vo5a"},"source":["# Save results\n","\n","Once all the cells are completed, save the loss history of all the parts for submission."]},{"cell_type":"code","metadata":{"id":"T-3b0miHWBqL","executionInfo":{"status":"ok","timestamp":1617456714367,"user_tz":-60,"elapsed":461384,"user":{"displayName":"Soufian Benamara","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gj7-UwGIuLvBwNdi62Q2pMRjoHrgtEm3LZkyGEWZw=s64","userId":"08596286217915945464"}}},"source":["acc_history_dict['acc_hist_part2'] = acc_hist_part2 \n","acc_history_dict['acc_hist_part3'] = acc_hist_part3 \n","acc_history_dict['acc_hist_part4'] = acc_hist_part4 \n","submission_path = os.path.join(GOOGLE_DRIVE_PATH, 'pytorch_autograd_and_nn.pkl')\n","dump_results(acc_history_dict, submission_path)"],"execution_count":32,"outputs":[]}]}